Re: "Asynchronous Requests from Emacs Dynamic Modules"

[Akira Kyle <ak@akirakyle.com>]

Thanks for the feedback. 

On Sat, Oct 31, 2020 at 02:18 PM, Eli Zaretskii <eliz@gnu.org> 
wrote:

>> 1) Chris Wellon's solution in the article I linked using the 
>> SIGUSR1 signal which Emacs handles in the event queue notify 
>> the 
>> running Lisp Thread that it should call back into module 
>> functions. This idea could be expanded to allow modules to 
>> define 
>> their own events and lisp handlers that will respond to them.
>
> A terrible design, if you ask me.  Reminds me how timers worked 
> in
> Emacs long ago: we had an external program which would 
> periodically
> deliver a signal to Emacs.  We switched off of that to what we 
> have
> now, for good reasons.
>
> And please keep in mind that doing anything non-trivial from a 
> signal
> handler is inherently unsafe.

I agree signal handlers are tricky to get right and probably not 
the best option here, especially considering the information that 
can be conveyed through the signal is pretty limited.

>> 2) Stefan's and Philipp's proposal of using the process 
>> interface 
>> with file descriptors to pipes and perhaps `process-filter` to 
>> notify the Lisp Thread that it should call back into the module 
>> functions.
>
> This is the way to go, IMO.  This is how Emacs itself handles 
> async
> events, so the infrastructure for doing this, both in C and in 
> Lisp,
> already exists, and is mature.  All you need is use it.

I'll explore this option a bit then and see what I can do with 
sharing a pipe between Lisp and a dynamic module thread.

>> 3) yyoncho's proposal to use the lisp threading interface, 
>> specifically condition variables to allow dynamic modules to 
>> `condition-notify`.
>
> This is unworkable within the current design of Lisp threads.
> condition-notify releases the global lock, something that under 
> the
> current design cannot be done at an arbitrary time, because it 
> could
> cause two threads run in parallel.
>
>> I think the most natural solution would be a dynamic module
>> interface that allows grabbing and releasing Lisp's global 
>> thread
>> lock.
>
> I encourage you to study how threading works in Emacs (the code 
> is in
> thread.c).  I'm sure you will see right away why this cannot be 
> done
> without a complete redesign of how switching threads works. 
> There's
> no way of interrupting a running Lisp thread in an arbitrary 
> place and
> switching to another thread.

It seems the lisp threading functionality is still pretty beta 
quality. Have the data races that Chris Wellons mentions here [1] 
ever been addressed? I generally agree with his sentiment about 
the current lisp threading interface seems like not the right 
thing for lisp. It looks like it's mostly just exposing the 
pthread interface. What is the use case for the current lisp 
threads given the severe limitation that only will ever run at a 
time and the constraints the cooperative nature puts on when 
threads can switch?

As I've poked around a bit more I see how my previous thought that 
some module interface for grabbing and releasing the Lisp thread 
lock would probably really go against the a lot of the current way 
Emacs handles threads and the emacs_env struct. I was basing that 
thought on the way python handles this where it's relatively easy 
to write low level primitives that release the GIL by just 
surrounding it with Py_BEGIN_ALLOW_THREADS and 
Py_END_ALLOW_THREADS and it's up to you to ensure you don't call 
any interpreter functions. I still think something like this is 
possible (it would probably be completely separate from the lisp 
threading interface), but it would probably just require a lot of 
care around ensuring the integrity of the stack and controlling 
exactly when in the event loop it would be okay for a thread 
switch to happen. The more I think about it, the more it would end 
up resembling the polling that's already done for the process 
interface with `wait_reading_process_output`.

[1] https://nullprogram.com/blog/2018/05/31/