Re: Concurrency via isolated process/thread

[Ihor Radchenko <yantar92@posteo.net>]

Eli Zaretskii <eliz@gnu.org> writes:

>> Does it mean that we can safely call, for example, Fcons asynchronously?
>
> Not necessarily, because Fcons is not just about allocating memory.
>
> I think we once again have a misunderstanding here, because when you
> say "memory allocation" you mean something very different than I do,
> which is a call to malloc to get more memory from the system.  It
> sounds like you think that Fcons _is_ memory allocation?  But if so,
> this terminology is so confusing that it is not useful in a detailed
> technical discussion such as this one.  We use the term "consing" to
> refer to creation of Lisp objects, which includes memory allocation,
> but also other stuff.

> In particular, consing modifies memory blocks (already available to
> Emacs, so no "memory allocation" per se) used to keep track of live
> and dead Lisp objects, and those modifications cannot be concurrently
> done by more than one thread, at least in some cases.

Thanks for the clarification.
I heard this term, but was unsure what exactly it refers to.

>> > So your solution to each such problem is to lock variables?  If so,
>> > you will end up locking a lot of them, and how is this different from
>> > using the global lock we do today with Lisp threads?
>> 
>> The idea is to prevent simultaneous write, which will only lock for a
>> small fraction of time.
>
> If one thread writes to a data structure, reading from it could also
> need to block, or else the reader will risk getting inconsistent data.
> So this is not just about simultaneous writing, it's much more
> general.

Sure. Of course, locking should be on write.
May you elaborate what you mean by inconsistent data?

>> And I still fail to see where base-buffer is _changed_. Is base buffer
>> ever supposed to be changed?
>
> Another thread might change it while this thread examines it.

I was able to identify a single place in C code where buffer's base
buffer is being set: in make-indirect-buffer, when the buffer is just
created. So, it is safe to assume that buffer->base_buffer remain
constant for any given live buffer. Unless I miss something.

>> No, I am saying that the current logic of updating the undo-list will not work
>> when multiple async threads are involved. It will no longer be safe to
>> assume that we can safely update undo-list right before/after switching
>> current_buffer.
>> 
>> So, I asked if an alternative approach could be used instead.
>
> Undo records changes in text properties and markers, and those are
> different in the indirect buffers from the base buffers.  Does this
> explain why we cannot simply point to the base buffer?

Are you sure? Text properties are certainly shared between indirect buffers.

bset_undo_list (old_buf->base_buffer, BVAR (old_buf, undo_list));
INLINE void
bset_undo_list (struct buffer *b, Lisp_Object val)
{
  b->undo_list_ = val;
}

The markers that are not shared are pt_marker, begv_marker, and
zv_marker. But those could probably be made attached to a given thread.

>> >>   /* Look down buffer's list of local Lisp variables
>> >>      to find and update any that forward into C variables.  */
>> >
>> > The C code accesses some buffer-local variables via Vfoo_bar C
>> > variables.  Those need to be updated when the current buffer changes.
>> 
>> Now, when you explained this, it is also a big problem. Such C variables
>> are a global state that needs to be kept up to date. Async will break
>> the existing logic of these updates.
>
> Exactly.

I now looked a bit further, and what you are talking about are the
variables defined via DEFVAR_PER_BUFFER. These global variables have the
following type:

/* Forwarding pointer to a Lisp_Object variable.
   This is allowed only in the value cell of a symbol,
   and it means that the symbol's value really lives in the
   specified variable.  */
struct Lisp_Objfwd
  {
    enum Lisp_Fwd_Type type;	/* = Lisp_Fwd_Obj */
    Lisp_Object *objvar;
  };

The code in set_buffer calls
Fsymbol_value->find_symbol_value->swap_in_symval_forwarding for every symbol
with C variable equivalent.

These calls update internal pointer to Lisp object corresponding to
variable value in current_buffer.

If my understanding is correct, it should be safe to convert them into
thread-local variables and update them within current thread when
current_buffer (already thread-local) is altered.

>> > Oh, yes, they will: see fetch_buffer_markers, called by
>> > set_buffer_internal_2.
>> 
>> Do you mean that in the existing cooperative Elisp threads, if one
>> thread moves the point and yields to other thread, the other thread will
>> be left with point in the same position (arbitrary, from the point of
>> view of this other thread)?
>
> That's one problem, yes.  There are others.  Emacs Lisp uses point,
> both explicitly and implicitly, all over the board.  It is unthinkable
> that a thread will find point not in a place where it last moved it.

It is exactly what happens with current cooperative threads, AFAIK.
Will it make sense to convert PT, ZV, and BEGV into thread-local variables?

>> Is it buffer's marker list? I thought that you are referring to
>> BUF_MARKERS, not to PT, BEGV, and ZV.
>
> Buffer's marker list are referenced in subroutines of
> record_buffer_markers.

Do you mean record_buffer_markers->set_marker_both->attach_marker->
  if (m->buffer != b)
    {
      unchain_marker (m);
      m->buffer = b;
      m->next = BUF_MARKERS (b);
      BUF_MARKERS (b) = m;
    }

But will this `if' ever trigger for PT, BEGV, and ZV?

Also, it looks reasonable to block BUF_MARKERS when we need to change
BUF_MARKERS.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>