Re: Releasing the thread global_lock from the module API

[Eli Zaretskii <eliz@gnu.org>]

> From: Spencer Baugh <sbaugh@janestreet.com>
> Date: Fri, 01 Mar 2024 12:34:36 -0500
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > It isn't clear what exactly do you want added, since calls to
> > thread-yield and similar APIs (like sleep-for and
> > accept-process-output), which release and re-acquire the global lock,
> > are already possible from Lisp threads.  What exactly is missing?
> 
> Those release and re-acquire the global lock, yes, but they don't allow
> module code to run on the thread while the global lock is released.

That's because the Emacs Lisp thread machinery cannot allow that.  A
Lisp thread has full access to the Emacs global state, so we cannot
allow more than one thread to have that at the same time.

> We can't do:
> - Lisp thread A calls module_work
> - On thread A, module_work releases the global lock.
> - On thread A, module_work starts native threads X and Y,

You can do that if the native threads are started before releasing the
global lock.

>   and does Emacs-independent work across all of these threads in parallel.

What do you mean by "Emacs-independent work across all of these
threads in parallel"?  If you want to break out of the restriction of
having only one Lisp thread running at any given time, then this is
IMO impossible without redesigning and reimplementing the current
threads machinery.

> Releasing the global lock allows Lisp thread B to run while Lisp thread
> A is running the module.

As explained above, you can do that already.

> > I think this is already possible, see above.  I guess I'm missing
> > something.
> 
> I don't think it is.  How would a module release global_lock, call a C
> library function on the current thread, and then re-acquire the global
> lock?  Can you say how you would do that with the APIs we have today?

If a thread releases a global lock, it cannot run the Lisp machine
until it re-acquire the global lock.  So what you want is impossible
within the current threads framework, unless you start native
(non-Lisp) threads which cannot safely access the Emacs global state.