Re: Concurrency via isolated process/thread

[Eli Zaretskii <eliz@gnu.org>]

> From: Ihor Radchenko <yantar92@posteo.net>
> Cc: luangruo@yahoo.com, emacs-devel@gnu.org
> Date: Fri, 07 Jul 2023 20:01:24 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> AFAIU, Elisp memory management is conceptually single-threaded.
> >
> > Again, I don't understand why you say that and what you mean by that.
> > We just call malloc, and malloc on modern systems is thread-safe.
> 
> Does it mean that we can safely call, for example, Fcons asynchronously?

Not necessarily, because Fcons is not just about allocating memory.

I think we once again have a misunderstanding here, because when you
say "memory allocation" you mean something very different than I do,
which is a call to malloc to get more memory from the system.  It
sounds like you think that Fcons _is_ memory allocation?  But if so,
this terminology is so confusing that it is not useful in a detailed
technical discussion such as this one.  We use the term "consing" to
refer to creation of Lisp objects, which includes memory allocation,
but also other stuff.

In particular, consing modifies memory blocks (already available to
Emacs, so no "memory allocation" per se) used to keep track of live
and dead Lisp objects, and those modifications cannot be concurrently
done by more than one thread, at least in some cases.

> (I am saying the above because my understanding is limited, hoping that
> you can give some pointers when I happen to be wrong.)

I'm happy to give pointers, once I understand pointers to what.
Before I have a chance of understanding that, we need to have a common
terminology, though.

> >> >   buffer-alist
> >> 
> >> I do not see how this is a problem to lock/unlock this variable.
> >
> > So your solution to each such problem is to lock variables?  If so,
> > you will end up locking a lot of them, and how is this different from
> > using the global lock we do today with Lisp threads?
> 
> The idea is to prevent simultaneous write, which will only lock for a
> small fraction of time.

If one thread writes to a data structure, reading from it could also
need to block, or else the reader will risk getting inconsistent data.
So this is not just about simultaneous writing, it's much more
general.

> And I still fail to see where base-buffer is _changed_. Is base buffer
> ever supposed to be changed?

Another thread might change it while this thread examines it.

> >> >   buffer's undo-list
> >> 
> >> That's just a synchronization between old_buffer and
> >> old_buffer->base_buffer.
> >> I am not 100% sure why it is necessary to be done this way and manually
> >> instead of making undo-list values in indirect buffers point to base
> >> buffer.
> >
> > So you are now saying that code which worked in Emacs for decades does
> > unnecessary stuff, and should be removed or ignored?
> 
> No, I am saying that the current logic of updating the undo-list will not work
> when multiple async threads are involved. It will no longer be safe to
> assume that we can safely update undo-list right before/after switching
> current_buffer.
> 
> So, I asked if an alternative approach could be used instead.

Undo records changes in text properties and markers, and those are
different in the indirect buffers from the base buffers.  Does this
explain why we cannot simply point to the base buffer?

If this is clear, then what other approach except locking do you
suggest for that?

> >> I admit that I do not understand what the following comment is talking
> >> about:
> >> 
> >>   /* Look down buffer's list of local Lisp variables
> >>      to find and update any that forward into C variables.  */
> >
> > The C code accesses some buffer-local variables via Vfoo_bar C
> > variables.  Those need to be updated when the current buffer changes.
> 
> Now, when you explained this, it is also a big problem. Such C variables
> are a global state that needs to be kept up to date. Async will break
> the existing logic of these updates.

Exactly.

> >> >   buffer's point and begv/zv markers
> >> 
> >> AFAIU, these store the last point position and narrowing state.
> >> I do not see much problem here, except a need to lock these variables
> >> while writing them. They will not affect PT, BEGZ, and ZV in other
> >> threads, even if those operate on the same buffer now.
> >
> > Oh, yes, they will: see fetch_buffer_markers, called by
> > set_buffer_internal_2.
> 
> Do you mean that in the existing cooperative Elisp threads, if one
> thread moves the point and yields to other thread, the other thread will
> be left with point in the same position (arbitrary, from the point of
> view of this other thread)?

That's one problem, yes.  There are others.  Emacs Lisp uses point,
both explicitly and implicitly, all over the board.  It is unthinkable
that a thread will find point not in a place where it last moved it.

> >> >   buffer's marker list
> >> 
> >> May you point me where it is?
> >
> > In fetch_buffer_markers.  Again, I don't understand how you missed
> > that.
> 
> Is it buffer's marker list? I thought that you are referring to
> BUF_MARKERS, not to PT, BEGV, and ZV.

Buffer's marker list are referenced in subroutines of
record_buffer_markers.