Re: Concurrency, again

[Ken Raeburn <raeburn@raeburn.org>]

> On Oct 18, 2016, at 06:41, Eli Zaretskii <eliz@gnu.org> wrote:
> 
>> From: Ken Raeburn <raeburn@raeburn.org>
>> Date: Tue, 18 Oct 2016 06:08:49 -0400
>> Cc: tom@tromey.com,
>> emacs-devel@gnu.org
>> 
>>>> * header inclusion order requirement is weird; can generating one big header help?
>>> 
>>> I'm not sure I understand what inclusion order is being alluded to
>>> here.  Can you elaborate?
>> 
>> I think it’s the ordering and recursive inclusion involving thread.h relative to the other headers.  For example lisp.h includes thread.h which includes sysselect.h which includes lisp.h; thread.h uses struct vectorlike_header, so it has to be included in lisp.h after that structure is defined but before struct thread_state gets used; thread.h also includes regex.h which includes lisp.h.  You commented at one point, “We now have an unfortunate situation whereby lisp.h cannot be included before some of the other headers, due to this.”
> 
> This could be solved by moving parts of thread.h into lisp.h, with the
> appropriate #ifdef guards.  I think Lisp objects should be in lisp.h
> anyway, even if they are optional; anything else is confusing.

For big, specialized objects like buffers, I think the modularity can help keep things organized, but where we’ve got a structure with all the per-thread state from random parts of the program, like condition handlers and regex state, we don’t have the luxury of separating things.  But even if we move most of thread.h into lisp.h, I think there may still be mutual recursion between it and the other headers.

Perhaps if the thread state contained struct pointers instead of structures, we could forward-declare some of the types and not have to pull in the other headers in such a fashion.


> 
>>>> * one thread per terminal?
>>> 
>>> Why?
>>> 
>>>> * file notifications and such shouldn’t go through same queue as keyboard events
>>> 
>>> Why?
>> 
>> Stefan’s message: http://lists.gnu.org/archive/html/emacs-devel/2013-08/msg00755.html
> 
> I don't see that as a critical problem, perhaps because we don't yet
> realize how serious it can be.  The whole purpose of trying to merge
> the concurrency branch is to collect practical experience as to what
> should and shouldn't be in this kind of Emacs feature.  So I'd tend to
> let this be, until we find out we can't, and why.
> 
>>>> * interaction of SIGCHLD handling and threads?
>>> 
>>> Details?
>> 
>> Your message http://lists.gnu.org/archive/html/emacs-devel/2013-08/msg00738.html raised questions.  If they were ever satisfactorily answered, I overlooked it when putting together my notes….
> 
> That should be easy: since a subprocess is locked to a single thread,

by default, but if that thread exits, that lock disappears

> SIGCHLD should be delivered to that thread.  If we don't have that
> already, we should add that, it doesn't sound hard, given the
> infrastructure we already have (deliver_thread_signal etc.).

It’s not completely trivial.  Under POSIX, we get no control over which thread receives SIGCHLD due to a subprocess exiting, except if we want certain threads to block receiving the signal completely.  We can use pthread_kill to explicitly send SIGCHLD to a specific thread, if it’s not blocking the signal, but we don’t know which thread until we’ve fetched the child’s pid and looked it up in some data structure; we do that by calling waitpid() but then the kernel discards the child process’s status info, so the “correct” thread can no longer respond to the signal by making another waitpid() call to collect the status info.  We’d have to save the info the first time we call waitpid().  But doing it from within the signal handler could be tricky, because in that context we’re limited to async-signal-safe functions, and helpful routines like malloc() and pthread_mutex_lock() aren’t on the list.

On the other hand, perhaps we can create one special thread to do all the waitpid() calls and pass info to the Lisp-running threads.  If that’s all it’s doing, the non-signal-handler portion of the thread’s code can loop calling waitpid and locking mutexes and updating data structures, and the Lisp-running threads can check in their thread-state structures for messages from the child-reaping thread.  I’m not sure how safe it would be to block SIGCHLD in threads that might call into the various system UI libraries and such, but we can probably just use a simple SIGCHLD handler to let any thread wake the reaper thread, which can then do its waitpid() calls.


> 
>> It’s easy enough to disable stack overflow checking when enabling thread support.
> 
> Or add some simple code in the stack overflow handler to check if we
> are in the main thread, and if not, punt (i.e. crash).
> 
>> If only one thread is allowed into the image processing code at a time (i.e., don’t release the global lock for that code) then that’s probably fine for now, and there’s probably other state there that different threads shouldn’t be mucking around with in parallel.
> 
> Redisplay runs in the main thread anyway, right?  If so, there's no
> problem.

If some random thread calls (redisplay) or (sit-for …)?  I think it’ll run in whichever Lisp-running thread triggers it.  But, it’ll be the one holding the lock.

> 
>> The keyboard.c one is the only one I’m a bit concerned about, in part because I haven’t looked at it.
> 
> What part(s) of keyboard.c, exactly?

Anything looking at getcjmp; that means read_event_from_main_queue and read_char.  Like I said, I haven’t looked very closely; if the static storage isn’t ever used across a point where the global lock could be released to allow a thread switch, it may be fine.