"Asynchronous Requests from Emacs Dynamic Modules"

"Asynchronous Requests from Emacs Dynamic Modules"

[Akira Kyle]

I'm trying to use Emacs' Dynamic Modules in an asynchronous 
setting where C module functions run in a separate thread from 
Emacs' main thread which is responsible for calling into the 
module functions. The natural problem then is how to notify Emacs 
when one wants to retrieve the result of the asynchronous 
operation from some lisp. Since the module docs explicitly say 
that module functions "may only interact with Emacs from Lisp 
interpreter threads (including the main thread) created by Emacs", 
it is not possible then for a separate module thread to interact 
with Emacs in an unprompted way through the module interface [1].

This blog post [2] has a solution to this problem by sending 
SIGUSR1 to Emacs, but that feels a bit hackish and forces one to 
do more work to manage all the different possible module threads 
and events that one may want to signal to Emacs. I suppose one 
could also conceivably also use Emacs' dbus support but that still 
feels clunky. Maybe its possible it's possible to use Emacs lisp's 
threads [2] to solve this but it's not clear to me how to go about 
doing so and I haven't found anyone that has done so.

Does anyone know of the best way to handle this? Perhaps the 
module interface should be expanded to handle this situation?

[1] 
https://www.gnu.org/software/emacs/manual/html_node/elisp/Module-Functions.html#Module-Functions
[2] 
https://www.gnu.org/software/emacs/manual/html_node/elisp/Threads.html#Threads
[3] https://nullprogram.com/blog/2017/02/14/

Re: "Asynchronous Requests from Emacs Dynamic Modules"

[Stefan Monnier]

> Does anyone know of the best way to handle this?
> Perhaps the module interface should be expanded to handle this situation?

Most likely, we need some way for a module to "raise a flag" which then
causes the main Lisp thread to get back to the module.

Maybe the best way to do that is for the module to create a ELisp
"process" object with a corresponding file handle (probably a fifo, with
one end used by the ELisp process object and the other end used by the
module), so when the module needs to run an async request, it would
write to the fifo, and then the process's `process-filter` would call
the module back to actually process the async request.


        Stefan

Re: "Asynchronous Requests from Emacs Dynamic Modules"

[Zhu Zihao]

[-- Attachment #1: Type: text/plain, Size: 1931 bytes --]


What about callbacks? You can pass a Lisp closure to module and call the
closure from module. You may also want something like "Promise" in
JavaScript to make it sweet

Emacs Lisp Promise/A+ Implementation: https://github.com/chuntaro/emacs-promise 

Akira Kyle writes:

> I'm trying to use Emacs' Dynamic Modules in an asynchronous setting where C
> module functions run in a separate thread from Emacs' main thread which is
> responsible for calling into the module functions. The natural problem then is
> how to notify Emacs when one wants to retrieve the result of the asynchronous 
> operation from some lisp. Since the module docs explicitly say that module
> functions "may only interact with Emacs from Lisp interpreter threads (including
> the main thread) created by Emacs", it is not possible then for a separate
> module thread to interact with Emacs in an unprompted way through the module
> interface [1].
>
> This blog post [2] has a solution to this problem by sending SIGUSR1 to Emacs,
> but that feels a bit hackish and forces one to do more work to manage all the
> different possible module threads and events that one may want to signal to
> Emacs. I suppose one could also conceivably also use Emacs' dbus support but
> that still feels clunky. Maybe its possible it's possible to use Emacs lisp's 
> threads [2] to solve this but it's not clear to me how to go about doing so and
> I haven't found anyone that has done so.
>
> Does anyone know of the best way to handle this? Perhaps the module interface
> should be expanded to handle this situation?
>
> [1]
> https://www.gnu.org/software/emacs/manual/html_node/elisp/Module-Functions.html#Module-Functions
> [2]
> https://www.gnu.org/software/emacs/manual/html_node/elisp/Threads.html#Threads
> [3] https://nullprogram.com/blog/2017/02/14/


-- 
Retrieve my PGP public key: https://meta.sr.ht/~citreu.pgp

Zihao

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 515 bytes --]

Re: "Asynchronous Requests from Emacs Dynamic Modules"

[Philipp Stephani]

Am Fr., 30. Okt. 2020 um 22:36 Uhr schrieb Akira Kyle <ak@akirakyle.com>:
>
> I'm trying to use Emacs' Dynamic Modules in an asynchronous
> setting where C module functions run in a separate thread from
> Emacs' main thread which is responsible for calling into the
> module functions. The natural problem then is how to notify Emacs
> when one wants to retrieve the result of the asynchronous
> operation from some lisp. Since the module docs explicitly say
> that module functions "may only interact with Emacs from Lisp
> interpreter threads (including the main thread) created by Emacs",
> it is not possible then for a separate module thread to interact
> with Emacs in an unprompted way through the module interface [1].
>
> This blog post [2] has a solution to this problem by sending
> SIGUSR1 to Emacs, but that feels a bit hackish and forces one to
> do more work to manage all the different possible module threads
> and events that one may want to signal to Emacs. I suppose one
> could also conceivably also use Emacs' dbus support but that still
> feels clunky. Maybe its possible it's possible to use Emacs lisp's
> threads [2] to solve this but it's not clear to me how to go about
> doing so and I haven't found anyone that has done so.
>
> Does anyone know of the best way to handle this? Perhaps the
> module interface should be expanded to handle this situation?

With Emacs 28, you can get a file descriptor to a pipe process and
send arbitrary data there (from arbitrary threads). You'd still need
some small protocol (e.g. JSON) to encode/decode requests, but with
that you can send arbitrary requests back asynchronously.

Re: "Asynchronous Requests from Emacs Dynamic Modules"

[Eli Zaretskii]

> From: Zhu Zihao <all_but_last@163.com>
> Date: Sat, 31 Oct 2020 11:18:43 +0800
> Cc: emacs-devel@gnu.org
> 
> What about callbacks? You can pass a Lisp closure to module and call the
> closure from module.

How will this work, if the callback is called asynchronously?  The
Lisp interpreter cannot be reentered.

I think the method described by Stefan is the only sane path towards
solving these situations.  It uses the infrastructure Emacs itself
uses for asynchronous interactions.

Re: "Asynchronous Requests from Emacs Dynamic Modules"

[yyoncho]

[-- Attachment #1: Type: text/plain, Size: 724 bytes --]

Allowing condition-notify call from non-elisp threads would solve the
problem as well.

https://debbugs.gnu.org/cgi/bugreport.cgi?bug=42028

On Sat, Oct 31, 2020 at 9:46 AM Eli Zaretskii <eliz@gnu.org> wrote:

> > From: Zhu Zihao <all_but_last@163.com>
> > Date: Sat, 31 Oct 2020 11:18:43 +0800
> > Cc: emacs-devel@gnu.org
> >
> > What about callbacks? You can pass a Lisp closure to module and call the
> > closure from module.
>
> How will this work, if the callback is called asynchronously?  The
> Lisp interpreter cannot be reentered.
>
> I think the method described by Stefan is the only sane path towards
> solving these situations.  It uses the infrastructure Emacs itself
> uses for asynchronous interactions.
>
>

[-- Attachment #2: Type: text/html, Size: 1283 bytes --]

Re: "Asynchronous Requests from Emacs Dynamic Modules"

[Eli Zaretskii]

> From: yyoncho <yyoncho@gmail.com>
> Date: Sat, 31 Oct 2020 10:02:41 +0200
> Cc: Zhu Zihao <all_but_last@163.com>, ak@akirakyle.com, 
> 	emacs-devel <emacs-devel@gnu.org>
> 
> Allowing condition-notify call from non-elisp threads would solve the problem as well. 

I don't see how that could be possible without redesigning the entire
Lisp threads machinery.

In general, no part of the Lisp interpreter -- and that includes the
Lisp thread related functions -- can ever safely run from an async
non-Lisp thread.

Re: "Asynchronous Requests from Emacs Dynamic Modules"

[yyoncho]

[-- Attachment #1: Type: text/plain, Size: 708 bytes --]

That is unfortunate. I was assuming that at least thread primitives can be
made thread-safe.

On Sat, Oct 31, 2020 at 11:14 AM Eli Zaretskii <eliz@gnu.org> wrote:

> > From: yyoncho <yyoncho@gmail.com>
> > Date: Sat, 31 Oct 2020 10:02:41 +0200
> > Cc: Zhu Zihao <all_but_last@163.com>, ak@akirakyle.com,
> >       emacs-devel <emacs-devel@gnu.org>
> >
> > Allowing condition-notify call from non-elisp threads would solve the
> problem as well.
>
> I don't see how that could be possible without redesigning the entire
> Lisp threads machinery.
>
> In general, no part of the Lisp interpreter -- and that includes the
> Lisp thread related functions -- can ever safely run from an async
> non-Lisp thread.
>

[-- Attachment #2: Type: text/html, Size: 1306 bytes --]

Re: "Asynchronous Requests from Emacs Dynamic Modules"

[Eli Zaretskii]

> From: yyoncho <yyoncho@gmail.com>
> Date: Sat, 31 Oct 2020 11:45:35 +0200
> Cc: Zhu Zihao <all_but_last@163.com>, ak@akirakyle.com, 
> 	emacs-devel <emacs-devel@gnu.org>
> 
> That is unfortunate. I was assuming that at least thread primitives can be made thread-safe. 

They are thread-safe, but they rely on strict discipline of having
one, and only one, Lisp thread active at any given time.  IOW, the
mechanism that switches threads is not "thread-safe".

Re: "Asynchronous Requests from Emacs Dynamic Modules"

[Stefan Monnier]

> With Emacs 28, you can get a file descriptor to a pipe process and
> send arbitrary data there (from arbitrary threads).

Cool, so that takes care of it.

> You'd still need some small protocol (e.g. JSON) to encode/decode
> requests, but with that you can send arbitrary requests
> back asynchronously.

I don't think you need a complex protocol: just stash somewhere (inside
the module data structures) the data you need for the async request,
then send a dummy byte to the pipe.  On the Lisp side, just install
a process-filter which calls back the module to "run any pending async
requests".


        Stefan

Re: "Asynchronous Requests from Emacs Dynamic Modules"

[Akira Kyle]

On Sat, Oct 31, 2020 at 04:43 AM, Eli Zaretskii <eliz@gnu.org> 
wrote:

>> That is unfortunate. I was assuming that at least thread 
>> primitives can be made thread-safe. 
>
> They are thread-safe, but they rely on strict discipline of 
> having
> one, and only one, Lisp thread active at any given time.  IOW, 
> the
> mechanism that switches threads is not "thread-safe".

To summarize so far, I see three different ways of tackling this:

1) Chris Wellon's solution in the article I linked using the 
SIGUSR1 signal which Emacs handles in the event queue notify the 
running Lisp Thread that it should call back into module 
functions. This idea could be expanded to allow modules to define 
their own events and lisp handlers that will respond to them.

2) Stefan's and Philipp's proposal of using the process interface 
with file descriptors to pipes and perhaps `process-filter` to 
notify the Lisp Thread that it should call back into the module 
functions.

3) yyoncho's proposal to use the lisp threading interface, 
specifically condition variables to allow dynamic modules to 
`condition-notify`.

I see advantages and disadvantages to each. I think the most 
natural solution would be a dynamic module interface that allows 
grabbing and releasing Lisp's global thread lock. I think this 
would simplify writing dynamic module functions which could 
potentially cause Emacs to hang as it wouldn't require module 
authors to think about the lisp machinery to figure out what 
module function with what state to call back into when it needs to 
get its result back into lisp land. The way such an interface is 
implemented could be with any of the above mechanisms or something 
else.

Re: "Asynchronous Requests from Emacs Dynamic Modules"

[Eli Zaretskii]

> From: Akira Kyle <ak@akirakyle.com>
> Cc: yyoncho <yyoncho@gmail.com>, all_but_last@163.com, emacs-devel@gnu.org,
>  Stefan Monnier <monnier@iro.umontreal.ca>, Philipp Stephani
>  <p.stephani2@gmail.com>
> Date: Sat, 31 Oct 2020 13:25:58 -0600
> 
> 1) Chris Wellon's solution in the article I linked using the 
> SIGUSR1 signal which Emacs handles in the event queue notify the 
> running Lisp Thread that it should call back into module 
> functions. This idea could be expanded to allow modules to define 
> their own events and lisp handlers that will respond to them.

A terrible design, if you ask me.  Reminds me how timers worked in
Emacs long ago: we had an external program which would periodically
deliver a signal to Emacs.  We switched off of that to what we have
now, for good reasons.

And please keep in mind that doing anything non-trivial from a signal
handler is inherently unsafe.

> 2) Stefan's and Philipp's proposal of using the process interface 
> with file descriptors to pipes and perhaps `process-filter` to 
> notify the Lisp Thread that it should call back into the module 
> functions.

This is the way to go, IMO.  This is how Emacs itself handles async
events, so the infrastructure for doing this, both in C and in Lisp,
already exists, and is mature.  All you need is use it.

> 3) yyoncho's proposal to use the lisp threading interface, 
> specifically condition variables to allow dynamic modules to 
> `condition-notify`.

This is unworkable within the current design of Lisp threads.
condition-notify releases the global lock, something that under the
current design cannot be done at an arbitrary time, because it could
cause two threads run in parallel.

> I think the most natural solution would be a dynamic module
> interface that allows grabbing and releasing Lisp's global thread
> lock.

I encourage you to study how threading works in Emacs (the code is in
thread.c).  I'm sure you will see right away why this cannot be done
without a complete redesign of how switching threads works.  There's
no way of interrupting a running Lisp thread in an arbitrary place and
switching to another thread.

Re: "Asynchronous Requests from Emacs Dynamic Modules"

[Akira Kyle]

Thanks for the feedback. 

On Sat, Oct 31, 2020 at 02:18 PM, Eli Zaretskii <eliz@gnu.org> 
wrote:

>> 1) Chris Wellon's solution in the article I linked using the 
>> SIGUSR1 signal which Emacs handles in the event queue notify 
>> the 
>> running Lisp Thread that it should call back into module 
>> functions. This idea could be expanded to allow modules to 
>> define 
>> their own events and lisp handlers that will respond to them.
>
> A terrible design, if you ask me.  Reminds me how timers worked 
> in
> Emacs long ago: we had an external program which would 
> periodically
> deliver a signal to Emacs.  We switched off of that to what we 
> have
> now, for good reasons.
>
> And please keep in mind that doing anything non-trivial from a 
> signal
> handler is inherently unsafe.

I agree signal handlers are tricky to get right and probably not 
the best option here, especially considering the information that 
can be conveyed through the signal is pretty limited.

>> 2) Stefan's and Philipp's proposal of using the process 
>> interface 
>> with file descriptors to pipes and perhaps `process-filter` to 
>> notify the Lisp Thread that it should call back into the module 
>> functions.
>
> This is the way to go, IMO.  This is how Emacs itself handles 
> async
> events, so the infrastructure for doing this, both in C and in 
> Lisp,
> already exists, and is mature.  All you need is use it.

I'll explore this option a bit then and see what I can do with 
sharing a pipe between Lisp and a dynamic module thread.

>> 3) yyoncho's proposal to use the lisp threading interface, 
>> specifically condition variables to allow dynamic modules to 
>> `condition-notify`.
>
> This is unworkable within the current design of Lisp threads.
> condition-notify releases the global lock, something that under 
> the
> current design cannot be done at an arbitrary time, because it 
> could
> cause two threads run in parallel.
>
>> I think the most natural solution would be a dynamic module
>> interface that allows grabbing and releasing Lisp's global 
>> thread
>> lock.
>
> I encourage you to study how threading works in Emacs (the code 
> is in
> thread.c).  I'm sure you will see right away why this cannot be 
> done
> without a complete redesign of how switching threads works. 
> There's
> no way of interrupting a running Lisp thread in an arbitrary 
> place and
> switching to another thread.

It seems the lisp threading functionality is still pretty beta 
quality. Have the data races that Chris Wellons mentions here [1] 
ever been addressed? I generally agree with his sentiment about 
the current lisp threading interface seems like not the right 
thing for lisp. It looks like it's mostly just exposing the 
pthread interface. What is the use case for the current lisp 
threads given the severe limitation that only will ever run at a 
time and the constraints the cooperative nature puts on when 
threads can switch?

As I've poked around a bit more I see how my previous thought that 
some module interface for grabbing and releasing the Lisp thread 
lock would probably really go against the a lot of the current way 
Emacs handles threads and the emacs_env struct. I was basing that 
thought on the way python handles this where it's relatively easy 
to write low level primitives that release the GIL by just 
surrounding it with Py_BEGIN_ALLOW_THREADS and 
Py_END_ALLOW_THREADS and it's up to you to ensure you don't call 
any interpreter functions. I still think something like this is 
possible (it would probably be completely separate from the lisp 
threading interface), but it would probably just require a lot of 
care around ensuring the integrity of the stack and controlling 
exactly when in the event loop it would be okay for a thread 
switch to happen. The more I think about it, the more it would end 
up resembling the polling that's already done for the process 
interface with `wait_reading_process_output`.

[1] https://nullprogram.com/blog/2018/05/31/

Re: "Asynchronous Requests from Emacs Dynamic Modules"

[Eli Zaretskii]

> From: Akira Kyle <ak@akirakyle.com>
> Cc: yyoncho@gmail.com, all_but_last@163.com, emacs-devel@gnu.org,
>  monnier@iro.umontreal.ca, p.stephani2@gmail.com
> Date: Sat, 31 Oct 2020 18:14:32 -0600
> 
> It seems the lisp threading functionality is still pretty beta 
> quality. Have the data races that Chris Wellons mentions here [1] 
> ever been addressed?

It's hard to tell, because that blog has no specifics, just a broad
accusation.  It is also quite old.

> It looks like it's mostly just exposing the pthread interface.

That's simply not true.  It _uses_ pthreads to start and manage
threads, and to implement mutexes, condvars, etc.  But if you look
closely at the level of thread.c (_not_ systhread.c), you will see
something very different from a typical pthreads application.

> What is the use case for the current lisp threads given the severe
> limitation that only will ever run at a time and the constraints the
> cooperative nature puts on when threads can switch?

The main use case is to allow you to write one or more background Lisp
programs that go about their job while a foreground command lets the
user interact normally with Emacs.

Re: "Asynchronous Requests from Emacs Dynamic Modules"

[Akira Kyle]

On Sun, Nov 01, 2020 at 11:28 AM, Eli Zaretskii <eliz@gnu.org> 
wrote:

>> It seems the lisp threading functionality is still pretty beta 
>> quality. Have the data races that Chris Wellons mentions here 
>> [1] 
>> ever been addressed?
>
> It's hard to tell, because that blog has no specifics, just a 
> broad
> accusation.  It is also quite old.
>
>> It looks like it's mostly just exposing the pthread interface.
>
> That's simply not true.  It _uses_ pthreads to start and manage
> threads, and to implement mutexes, condvars, etc.  But if you 
> look
> closely at the level of thread.c (_not_ systhread.c), you will 
> see
> something very different from a typical pthreads application.

Sorry I meant not that it directly exposes pthreads but that its 
interface is modeled off of pthreads (i.e. mutexes, condition 
variables), but lisp should really have some higher level 
interface to parallel and concurrent execution (like Python's 
multiprocessing or asyncio interface). Perhaps it can be built on 
top of the current lisp threading interface, but it seems there 
may be some limitations that make that difficult.

>> What is the use case for the current lisp threads given the 
>> severe
>> limitation that only will ever run at a time and the 
>> constraints the
>> cooperative nature puts on when threads can switch?
>
> The main use case is to allow you to write one or more 
> background Lisp
> programs that go about their job while a foreground command lets 
> the
> user interact normally with Emacs.

Right, but due to the global lock this only allows for concurrent 
execution, and due to the cooperative nature needing one to 
explicitly yield execution for threads to switch, it seems like in 
reality it would be very cumbersome to work with. Do you know of 
anyone who uses it? I'd be very interested in seeing some examples 
how to successfully use it.

I think libraries like emacs-aio [1] accomplish the same thing but 
with a much easier to use interface. By using (run-at-time 0 nil 
callback args) they accomplish a sort of "green threading" that 
doesn't actually need any sort of pthread support or the 
associated issues with ensuring atomicity with the interpreter. It 
would be great to have something like this built in.

[1] https://github.com/skeeto/emacs-aio

Re: "Asynchronous Requests from Emacs Dynamic Modules"

[Akira Kyle]

On Sat, Oct 31, 2020 at 06:49 AM, Stefan Monnier 
<monnier@iro.umontreal.ca> wrote:

>> With Emacs 28, you can get a file descriptor to a pipe process 
>> and
>> send arbitrary data there (from arbitrary threads).
>
> Cool, so that takes care of it.
>
>> You'd still need some small protocol (e.g. JSON) to 
>> encode/decode
>> requests, but with that you can send arbitrary requests
>> back asynchronously.
>
> I don't think you need a complex protocol: just stash somewhere 
> (inside
> the module data structures) the data you need for the async 
> request,
> then send a dummy byte to the pipe.  On the Lisp side, just 
> install
> a process-filter which calls back the module to "run any pending 
> async
> requests".

Thanks Philipp for pointing this out! I'm glad this was already 
thought of and added so the SIGUSR1 hack isn't necessary going 
forward. For future reference, the module function is 
`open_channel`.

Re: "Asynchronous Requests from Emacs Dynamic Modules"

[Philipp Stephani]

Am So., 1. Nov. 2020 um 21:18 Uhr schrieb Akira Kyle <ak@akirakyle.com>:
>
>
> On Sat, Oct 31, 2020 at 06:49 AM, Stefan Monnier
> <monnier@iro.umontreal.ca> wrote:
>
> >> With Emacs 28, you can get a file descriptor to a pipe process
> >> and
> >> send arbitrary data there (from arbitrary threads).
> >
> > Cool, so that takes care of it.
> >
> >> You'd still need some small protocol (e.g. JSON) to
> >> encode/decode
> >> requests, but with that you can send arbitrary requests
> >> back asynchronously.
> >
> > I don't think you need a complex protocol: just stash somewhere
> > (inside
> > the module data structures) the data you need for the async
> > request,
> > then send a dummy byte to the pipe.  On the Lisp side, just
> > install
> > a process-filter which calls back the module to "run any pending
> > async
> > requests".
>
> Thanks Philipp for pointing this out! I'm glad this was already
> thought of and added so the SIGUSR1 hack isn't necessary going
> forward. For future reference, the module function is
> `open_channel`.

I actually implemented open_channel after reading that very blogpost
and thinking "there has to be a better way" :-)
Note that even without that there are a few other options, e.g.
starting a Unix domain socket server in Emacs and then connecting to
it from the module. These are essentially equivalent in that they
queue some form of event (process input) that can be processed
asynchronously on the Emacs side.

async-await (was: Re: "Asynchronous Requests from Emacs Dynamic Modules")

[Philipp Stephani]

Am So., 1. Nov. 2020 um 21:15 Uhr schrieb Akira Kyle <ak@akirakyle.com>:

> I think libraries like emacs-aio [1] accomplish the same thing but
> with a much easier to use interface. By using (run-at-time 0 nil
> callback args) they accomplish a sort of "green threading" that
> doesn't actually need any sort of pthread support or the
> associated issues with ensuring atomicity with the interpreter. It
> would be great to have something like this built in.
>
> [1] https://github.com/skeeto/emacs-aio

I agree (independent of the threading interface and module question)
that an async-await model is really nice and powerful. It would indeed
be great to better support it in Emacs.
The aio library suffers from a few problems (due to Emacs, not due to
the library itself), which could be rectified by implementing such
support in core Emacs:
1. It makes functions virtually un-debuggable because of the necessary
CPS transformations. Either Edebug should be beefed up to understand
the CPS state machines, or the state machine generation should somehow
happen in core.
2. It makes error reporting from asynchronous functions virtually
impossible because the timer machinery catches all errors and converts
them into nondescript "error in timer" messages.
None of those are impossible to overcome, but they need a bit of
design work to fix.

Re: "Asynchronous Requests from Emacs Dynamic Modules"

[Eli Zaretskii]

> From: Akira Kyle <ak@akirakyle.com>
> Cc: yyoncho@gmail.com, all_but_last@163.com, emacs-devel@gnu.org,
>  monnier@iro.umontreal.ca, p.stephani2@gmail.com
> Date: Sun, 01 Nov 2020 13:15:43 -0700
> 
> Sorry I meant not that it directly exposes pthreads but that its 
> interface is modeled off of pthreads (i.e. mutexes, condition 
> variables), but lisp should really have some higher level 
> interface to parallel and concurrent execution (like Python's 
> multiprocessing or asyncio interface). Perhaps it can be built on 
> top of the current lisp threading interface, but it seems there 
> may be some limitations that make that difficult.

Maybe it can, but I'm not sure I see the need yet.  What exactly is
wrong with the existing primitives?  After all, the Lisp threads _are_
threads, so basic thread operations for them do make sense.

> > The main use case is to allow you to write one or more background
> > Lisp programs that go about their job while a foreground command
> > lets the user interact normally with Emacs.
> 
> Right, but due to the global lock this only allows for concurrent 
> execution, and due to the cooperative nature needing one to 
> explicitly yield execution for threads to switch, it seems like in 
> reality it would be very cumbersome to work with.

Emacs yields automatically when it waits for input, so the idea was
that it wouldn't be cumbersome.

> Do you know of anyone who uses it? I'd be very interested in seeing
> some examples how to successfully use it.

Michael Albinus did, I think you can see his work on a branch in our
Git repository.

> I think libraries like emacs-aio [1] accomplish the same thing but 
> with a much easier to use interface. By using (run-at-time 0 nil 
> callback args) they accomplish a sort of "green threading" that 
> doesn't actually need any sort of pthread support or the 
> associated issues with ensuring atomicity with the interpreter. It 
> would be great to have something like this built in.

run-at-time is our poor man's threading from years ago; Lisp threads
were supposed to make such background applications easier to
implement.

Re:Re: "Asynchronous Requests from Emacs Dynamic Modules"

[Zhu Zihao]

[-- Attachment #1: Type: text/plain, Size: 1405 bytes --]

Ok, But how to do this in Emacs version 25-27?


Emacs doesn't support named pipe access, but may be we can use `make-network-process` to open a TCP socket. Like this?


```
(let ((proc (make-network-process :name "fake-proxy-process"
                     :server t
                     :host 'local
                     :filter <filter-to-execute-callback>
                     :noquery t
                     ;; Detect port automatically
                     :service t)))
     (setq <global-proc-var> proc)
     (module-function-initialize-notify (process-contact proc :service)))
```


But this maybe too strange for user to understand why a dynamic module require a TCP port access...






At 2020-10-31 20:49:21, "Stefan Monnier" <monnier@iro.umontreal.ca> wrote:
>> With Emacs 28, you can get a file descriptor to a pipe process and
>> send arbitrary data there (from arbitrary threads).
>
>Cool, so that takes care of it.
>
>> You'd still need some small protocol (e.g. JSON) to encode/decode
>> requests, but with that you can send arbitrary requests
>> back asynchronously.
>
>I don't think you need a complex protocol: just stash somewhere (inside
>the module data structures) the data you need for the async request,
>then send a dummy byte to the pipe.  On the Lisp side, just install
>a process-filter which calls back the module to "run any pending async
>requests".
>
>
>        Stefan
>

[-- Attachment #2: Type: text/html, Size: 2385 bytes --]

Re: "Asynchronous Requests from Emacs Dynamic Modules"

[Robert Pluim]

"Zhu Zihao" <all_but_last@163.com> writes:

> Ok, But how to do this in Emacs version 25-27?
>
>
> Emacs doesn't support named pipe access, but may be we can use `make-network-process` to open a TCP socket. Like this?
>
>
> ```
> (let ((proc (make-network-process :name "fake-proxy-process"
>                      :server t
>                      :host 'local
>                      :filter <filter-to-execute-callback>
>                      :noquery t
>                      ;; Detect port automatically
>                      :service t)))
>      (setq <global-proc-var> proc)
>      (module-function-initialize-notify (process-contact proc :service)))
> ```
>
>
> But this maybe too strange for user to understand why a dynamic module require a TCP port access...

It doesnʼt: you can use unix sockets by specifying :family 'local

Robert