Re: Releasing the thread global_lock from the module API

[Spencer Baugh <sbaugh@janestreet.com>]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Spencer Baugh <sbaugh@janestreet.com>
>> Date: Fri, 01 Mar 2024 16:21:03 -0500
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> > Where would you take the data for opening the socket? doesn't that
>> > come from some Lisp program or from a Lisp variable?  And how would
>> > you know what kind of request to send? doesn't that come from Lisp as
>> > well?
>> 
>> Yes: I get those things as arguments from Lisp and then convert them
>> into the native datastructures of my library, which can be used without
>> further interacting with the Lisp machine.
>> 
>> Then I would release the lock and call into my library, which does some
>> useful work which takes a while.
>
> How is this different from starting your own native thread, then
> releasing the lock?

They are completely different things.

Let me say again what this sub-thread is about:

> If a C library provides some blocking function which does some
> complicated form of IO, a module providing bindings for that C library
> can release global_lock before calling that function, and then
> re-acquire the lock after the function returns.  Then Lisp threads
> calling this module function will not block the main Emacs thread.

So, if I am running the following program in a Lisp thread:

(while (do-emacs-things)
  (let ((input (do-emacs-things)))
    (let ((result (call-into-native-module input)))
      (do-emacs-things result))))

and call-into-native-module unlocks the global lock around the calls
into my library, then during the part of call-into-native-module which
calls into my library, the main Emacs thread will not be blocked and can
run in parallel with call-into-native-module.

>> > And what would you do with the stuff you get in response? don't
>> > you want to put this in some Lisp variable or provide as input for
>> > some Lisp program?  Etc. etc.
>> 
>> Yes: After I finish the call into my library, I would take the lock
>> again and call further Lisp functions to put the results back into the
>> Lisp machine.
>
> How is this different from doing the same when your native thread
> finishes?
>
>> >> Since it's useful for me, I'd like to write a patch which allow modules
>> >> to do this; would it be considered?
>> >
>> > Once again: what cannot you do from a native thread started by the
>> > module?  IOW, why would you need access to the global lock machinery
>> > in the first place, if all you want to do is something that is
>> > unrelated to Emacs and its global state?
>> 
>> See above: The call into my library, which takes a while, and is useful,
>> does not touch the Lisp machine.  But other code around that call does
>> touch the Lisp machine, and so needs to run with the lock.  The ability
>> to release and re-acquire the lock means my module doesn't hold the lock
>> when it doesn't need to.
>
> You didn't answer my question about doing this from a native thread.

I hope my answer above clarifies it.

> As for re-acquiring the lock: you cannot do that with how the Lisp
> threads are currently implemented.  You can only wait for the lock to
> be released, and try re-acquiring it; whether you succeed is anyone's
> guess, if there are other threads competing for the lock.

Yes, that is the normal behavior of locks.  I am well aware of it.  By
"re-acquire", I mean (in your words) "wait for the lock to be released,
and try re-acquiring it".

It is okay if my module has to wait for the main Emacs Lisp thread to
stop running before it gets the global lock again.