* "Asynchronous Requests from Emacs Dynamic Modules" @ 2020-10-30 21:35 Akira Kyle 2020-10-30 22:39 ` Stefan Monnier ` (2 more replies) 0 siblings, 3 replies; 21+ messages in thread From: Akira Kyle @ 2020-10-30 21:35 UTC (permalink / raw) To: emacs-devel I'm trying to use Emacs' Dynamic Modules in an asynchronous setting where C module functions run in a separate thread from Emacs' main thread which is responsible for calling into the module functions. The natural problem then is how to notify Emacs when one wants to retrieve the result of the asynchronous operation from some lisp. Since the module docs explicitly say that module functions "may only interact with Emacs from Lisp interpreter threads (including the main thread) created by Emacs", it is not possible then for a separate module thread to interact with Emacs in an unprompted way through the module interface [1]. This blog post [2] has a solution to this problem by sending SIGUSR1 to Emacs, but that feels a bit hackish and forces one to do more work to manage all the different possible module threads and events that one may want to signal to Emacs. I suppose one could also conceivably also use Emacs' dbus support but that still feels clunky. Maybe its possible it's possible to use Emacs lisp's threads [2] to solve this but it's not clear to me how to go about doing so and I haven't found anyone that has done so. Does anyone know of the best way to handle this? Perhaps the module interface should be expanded to handle this situation? [1] https://www.gnu.org/software/emacs/manual/html_node/elisp/Module-Functions.html#Module-Functions [2] https://www.gnu.org/software/emacs/manual/html_node/elisp/Threads.html#Threads [3] https://nullprogram.com/blog/2017/02/14/ ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: "Asynchronous Requests from Emacs Dynamic Modules" 2020-10-30 21:35 "Asynchronous Requests from Emacs Dynamic Modules" Akira Kyle @ 2020-10-30 22:39 ` Stefan Monnier 2020-10-31 3:18 ` Zhu Zihao 2020-10-31 7:36 ` Philipp Stephani 2 siblings, 0 replies; 21+ messages in thread From: Stefan Monnier @ 2020-10-30 22:39 UTC (permalink / raw) To: Akira Kyle; +Cc: emacs-devel > Does anyone know of the best way to handle this? > Perhaps the module interface should be expanded to handle this situation? Most likely, we need some way for a module to "raise a flag" which then causes the main Lisp thread to get back to the module. Maybe the best way to do that is for the module to create a ELisp "process" object with a corresponding file handle (probably a fifo, with one end used by the ELisp process object and the other end used by the module), so when the module needs to run an async request, it would write to the fifo, and then the process's `process-filter` would call the module back to actually process the async request. Stefan ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: "Asynchronous Requests from Emacs Dynamic Modules" 2020-10-30 21:35 "Asynchronous Requests from Emacs Dynamic Modules" Akira Kyle 2020-10-30 22:39 ` Stefan Monnier @ 2020-10-31 3:18 ` Zhu Zihao 2020-10-31 7:45 ` Eli Zaretskii 2020-10-31 7:36 ` Philipp Stephani 2 siblings, 1 reply; 21+ messages in thread From: Zhu Zihao @ 2020-10-31 3:18 UTC (permalink / raw) To: Akira Kyle; +Cc: emacs-devel [-- Attachment #1: Type: text/plain, Size: 1931 bytes --] What about callbacks? You can pass a Lisp closure to module and call the closure from module. You may also want something like "Promise" in JavaScript to make it sweet Emacs Lisp Promise/A+ Implementation: https://github.com/chuntaro/emacs-promise Akira Kyle writes: > I'm trying to use Emacs' Dynamic Modules in an asynchronous setting where C > module functions run in a separate thread from Emacs' main thread which is > responsible for calling into the module functions. The natural problem then is > how to notify Emacs when one wants to retrieve the result of the asynchronous > operation from some lisp. Since the module docs explicitly say that module > functions "may only interact with Emacs from Lisp interpreter threads (including > the main thread) created by Emacs", it is not possible then for a separate > module thread to interact with Emacs in an unprompted way through the module > interface [1]. > > This blog post [2] has a solution to this problem by sending SIGUSR1 to Emacs, > but that feels a bit hackish and forces one to do more work to manage all the > different possible module threads and events that one may want to signal to > Emacs. I suppose one could also conceivably also use Emacs' dbus support but > that still feels clunky. Maybe its possible it's possible to use Emacs lisp's > threads [2] to solve this but it's not clear to me how to go about doing so and > I haven't found anyone that has done so. > > Does anyone know of the best way to handle this? Perhaps the module interface > should be expanded to handle this situation? > > [1] > https://www.gnu.org/software/emacs/manual/html_node/elisp/Module-Functions.html#Module-Functions > [2] > https://www.gnu.org/software/emacs/manual/html_node/elisp/Threads.html#Threads > [3] https://nullprogram.com/blog/2017/02/14/ -- Retrieve my PGP public key: https://meta.sr.ht/~citreu.pgp Zihao [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 515 bytes --] ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: "Asynchronous Requests from Emacs Dynamic Modules" 2020-10-31 3:18 ` Zhu Zihao @ 2020-10-31 7:45 ` Eli Zaretskii 2020-10-31 8:02 ` yyoncho 0 siblings, 1 reply; 21+ messages in thread From: Eli Zaretskii @ 2020-10-31 7:45 UTC (permalink / raw) To: Zhu Zihao; +Cc: ak, emacs-devel > From: Zhu Zihao <all_but_last@163.com> > Date: Sat, 31 Oct 2020 11:18:43 +0800 > Cc: emacs-devel@gnu.org > > What about callbacks? You can pass a Lisp closure to module and call the > closure from module. How will this work, if the callback is called asynchronously? The Lisp interpreter cannot be reentered. I think the method described by Stefan is the only sane path towards solving these situations. It uses the infrastructure Emacs itself uses for asynchronous interactions. ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: "Asynchronous Requests from Emacs Dynamic Modules" 2020-10-31 7:45 ` Eli Zaretskii @ 2020-10-31 8:02 ` yyoncho 2020-10-31 9:13 ` Eli Zaretskii 0 siblings, 1 reply; 21+ messages in thread From: yyoncho @ 2020-10-31 8:02 UTC (permalink / raw) To: Eli Zaretskii; +Cc: Zhu Zihao, ak, emacs-devel [-- Attachment #1: Type: text/plain, Size: 724 bytes --] Allowing condition-notify call from non-elisp threads would solve the problem as well. https://debbugs.gnu.org/cgi/bugreport.cgi?bug=42028 On Sat, Oct 31, 2020 at 9:46 AM Eli Zaretskii <eliz@gnu.org> wrote: > > From: Zhu Zihao <all_but_last@163.com> > > Date: Sat, 31 Oct 2020 11:18:43 +0800 > > Cc: emacs-devel@gnu.org > > > > What about callbacks? You can pass a Lisp closure to module and call the > > closure from module. > > How will this work, if the callback is called asynchronously? The > Lisp interpreter cannot be reentered. > > I think the method described by Stefan is the only sane path towards > solving these situations. It uses the infrastructure Emacs itself > uses for asynchronous interactions. > > [-- Attachment #2: Type: text/html, Size: 1283 bytes --] ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: "Asynchronous Requests from Emacs Dynamic Modules" 2020-10-31 8:02 ` yyoncho @ 2020-10-31 9:13 ` Eli Zaretskii 2020-10-31 9:45 ` yyoncho 0 siblings, 1 reply; 21+ messages in thread From: Eli Zaretskii @ 2020-10-31 9:13 UTC (permalink / raw) To: yyoncho; +Cc: all_but_last, ak, emacs-devel > From: yyoncho <yyoncho@gmail.com> > Date: Sat, 31 Oct 2020 10:02:41 +0200 > Cc: Zhu Zihao <all_but_last@163.com>, ak@akirakyle.com, > emacs-devel <emacs-devel@gnu.org> > > Allowing condition-notify call from non-elisp threads would solve the problem as well. I don't see how that could be possible without redesigning the entire Lisp threads machinery. In general, no part of the Lisp interpreter -- and that includes the Lisp thread related functions -- can ever safely run from an async non-Lisp thread. ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: "Asynchronous Requests from Emacs Dynamic Modules" 2020-10-31 9:13 ` Eli Zaretskii @ 2020-10-31 9:45 ` yyoncho 2020-10-31 10:43 ` Eli Zaretskii 0 siblings, 1 reply; 21+ messages in thread From: yyoncho @ 2020-10-31 9:45 UTC (permalink / raw) To: Eli Zaretskii; +Cc: Zhu Zihao, ak, emacs-devel [-- Attachment #1: Type: text/plain, Size: 708 bytes --] That is unfortunate. I was assuming that at least thread primitives can be made thread-safe. On Sat, Oct 31, 2020 at 11:14 AM Eli Zaretskii <eliz@gnu.org> wrote: > > From: yyoncho <yyoncho@gmail.com> > > Date: Sat, 31 Oct 2020 10:02:41 +0200 > > Cc: Zhu Zihao <all_but_last@163.com>, ak@akirakyle.com, > > emacs-devel <emacs-devel@gnu.org> > > > > Allowing condition-notify call from non-elisp threads would solve the > problem as well. > > I don't see how that could be possible without redesigning the entire > Lisp threads machinery. > > In general, no part of the Lisp interpreter -- and that includes the > Lisp thread related functions -- can ever safely run from an async > non-Lisp thread. > [-- Attachment #2: Type: text/html, Size: 1306 bytes --] ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: "Asynchronous Requests from Emacs Dynamic Modules" 2020-10-31 9:45 ` yyoncho @ 2020-10-31 10:43 ` Eli Zaretskii 2020-10-31 19:25 ` Akira Kyle 0 siblings, 1 reply; 21+ messages in thread From: Eli Zaretskii @ 2020-10-31 10:43 UTC (permalink / raw) To: yyoncho; +Cc: all_but_last, ak, emacs-devel > From: yyoncho <yyoncho@gmail.com> > Date: Sat, 31 Oct 2020 11:45:35 +0200 > Cc: Zhu Zihao <all_but_last@163.com>, ak@akirakyle.com, > emacs-devel <emacs-devel@gnu.org> > > That is unfortunate. I was assuming that at least thread primitives can be made thread-safe. They are thread-safe, but they rely on strict discipline of having one, and only one, Lisp thread active at any given time. IOW, the mechanism that switches threads is not "thread-safe". ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: "Asynchronous Requests from Emacs Dynamic Modules" 2020-10-31 10:43 ` Eli Zaretskii @ 2020-10-31 19:25 ` Akira Kyle 2020-10-31 20:18 ` Eli Zaretskii 0 siblings, 1 reply; 21+ messages in thread From: Akira Kyle @ 2020-10-31 19:25 UTC (permalink / raw) To: Eli Zaretskii Cc: Philipp Stephani, emacs-devel, yyoncho, Stefan Monnier, all_but_last On Sat, Oct 31, 2020 at 04:43 AM, Eli Zaretskii <eliz@gnu.org> wrote: >> That is unfortunate. I was assuming that at least thread >> primitives can be made thread-safe. > > They are thread-safe, but they rely on strict discipline of > having > one, and only one, Lisp thread active at any given time. IOW, > the > mechanism that switches threads is not "thread-safe". To summarize so far, I see three different ways of tackling this: 1) Chris Wellon's solution in the article I linked using the SIGUSR1 signal which Emacs handles in the event queue notify the running Lisp Thread that it should call back into module functions. This idea could be expanded to allow modules to define their own events and lisp handlers that will respond to them. 2) Stefan's and Philipp's proposal of using the process interface with file descriptors to pipes and perhaps `process-filter` to notify the Lisp Thread that it should call back into the module functions. 3) yyoncho's proposal to use the lisp threading interface, specifically condition variables to allow dynamic modules to `condition-notify`. I see advantages and disadvantages to each. I think the most natural solution would be a dynamic module interface that allows grabbing and releasing Lisp's global thread lock. I think this would simplify writing dynamic module functions which could potentially cause Emacs to hang as it wouldn't require module authors to think about the lisp machinery to figure out what module function with what state to call back into when it needs to get its result back into lisp land. The way such an interface is implemented could be with any of the above mechanisms or something else. ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: "Asynchronous Requests from Emacs Dynamic Modules" 2020-10-31 19:25 ` Akira Kyle @ 2020-10-31 20:18 ` Eli Zaretskii 2020-11-01 0:14 ` Akira Kyle 0 siblings, 1 reply; 21+ messages in thread From: Eli Zaretskii @ 2020-10-31 20:18 UTC (permalink / raw) To: Akira Kyle; +Cc: p.stephani2, emacs-devel, yyoncho, monnier, all_but_last > From: Akira Kyle <ak@akirakyle.com> > Cc: yyoncho <yyoncho@gmail.com>, all_but_last@163.com, emacs-devel@gnu.org, > Stefan Monnier <monnier@iro.umontreal.ca>, Philipp Stephani > <p.stephani2@gmail.com> > Date: Sat, 31 Oct 2020 13:25:58 -0600 > > 1) Chris Wellon's solution in the article I linked using the > SIGUSR1 signal which Emacs handles in the event queue notify the > running Lisp Thread that it should call back into module > functions. This idea could be expanded to allow modules to define > their own events and lisp handlers that will respond to them. A terrible design, if you ask me. Reminds me how timers worked in Emacs long ago: we had an external program which would periodically deliver a signal to Emacs. We switched off of that to what we have now, for good reasons. And please keep in mind that doing anything non-trivial from a signal handler is inherently unsafe. > 2) Stefan's and Philipp's proposal of using the process interface > with file descriptors to pipes and perhaps `process-filter` to > notify the Lisp Thread that it should call back into the module > functions. This is the way to go, IMO. This is how Emacs itself handles async events, so the infrastructure for doing this, both in C and in Lisp, already exists, and is mature. All you need is use it. > 3) yyoncho's proposal to use the lisp threading interface, > specifically condition variables to allow dynamic modules to > `condition-notify`. This is unworkable within the current design of Lisp threads. condition-notify releases the global lock, something that under the current design cannot be done at an arbitrary time, because it could cause two threads run in parallel. > I think the most natural solution would be a dynamic module > interface that allows grabbing and releasing Lisp's global thread > lock. I encourage you to study how threading works in Emacs (the code is in thread.c). I'm sure you will see right away why this cannot be done without a complete redesign of how switching threads works. There's no way of interrupting a running Lisp thread in an arbitrary place and switching to another thread. ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: "Asynchronous Requests from Emacs Dynamic Modules" 2020-10-31 20:18 ` Eli Zaretskii @ 2020-11-01 0:14 ` Akira Kyle 2020-11-01 18:28 ` Eli Zaretskii 0 siblings, 1 reply; 21+ messages in thread From: Akira Kyle @ 2020-11-01 0:14 UTC (permalink / raw) To: Eli Zaretskii; +Cc: p.stephani2, emacs-devel, yyoncho, monnier, all_but_last Thanks for the feedback. On Sat, Oct 31, 2020 at 02:18 PM, Eli Zaretskii <eliz@gnu.org> wrote: >> 1) Chris Wellon's solution in the article I linked using the >> SIGUSR1 signal which Emacs handles in the event queue notify >> the >> running Lisp Thread that it should call back into module >> functions. This idea could be expanded to allow modules to >> define >> their own events and lisp handlers that will respond to them. > > A terrible design, if you ask me. Reminds me how timers worked > in > Emacs long ago: we had an external program which would > periodically > deliver a signal to Emacs. We switched off of that to what we > have > now, for good reasons. > > And please keep in mind that doing anything non-trivial from a > signal > handler is inherently unsafe. I agree signal handlers are tricky to get right and probably not the best option here, especially considering the information that can be conveyed through the signal is pretty limited. >> 2) Stefan's and Philipp's proposal of using the process >> interface >> with file descriptors to pipes and perhaps `process-filter` to >> notify the Lisp Thread that it should call back into the module >> functions. > > This is the way to go, IMO. This is how Emacs itself handles > async > events, so the infrastructure for doing this, both in C and in > Lisp, > already exists, and is mature. All you need is use it. I'll explore this option a bit then and see what I can do with sharing a pipe between Lisp and a dynamic module thread. >> 3) yyoncho's proposal to use the lisp threading interface, >> specifically condition variables to allow dynamic modules to >> `condition-notify`. > > This is unworkable within the current design of Lisp threads. > condition-notify releases the global lock, something that under > the > current design cannot be done at an arbitrary time, because it > could > cause two threads run in parallel. > >> I think the most natural solution would be a dynamic module >> interface that allows grabbing and releasing Lisp's global >> thread >> lock. > > I encourage you to study how threading works in Emacs (the code > is in > thread.c). I'm sure you will see right away why this cannot be > done > without a complete redesign of how switching threads works. > There's > no way of interrupting a running Lisp thread in an arbitrary > place and > switching to another thread. It seems the lisp threading functionality is still pretty beta quality. Have the data races that Chris Wellons mentions here [1] ever been addressed? I generally agree with his sentiment about the current lisp threading interface seems like not the right thing for lisp. It looks like it's mostly just exposing the pthread interface. What is the use case for the current lisp threads given the severe limitation that only will ever run at a time and the constraints the cooperative nature puts on when threads can switch? As I've poked around a bit more I see how my previous thought that some module interface for grabbing and releasing the Lisp thread lock would probably really go against the a lot of the current way Emacs handles threads and the emacs_env struct. I was basing that thought on the way python handles this where it's relatively easy to write low level primitives that release the GIL by just surrounding it with Py_BEGIN_ALLOW_THREADS and Py_END_ALLOW_THREADS and it's up to you to ensure you don't call any interpreter functions. I still think something like this is possible (it would probably be completely separate from the lisp threading interface), but it would probably just require a lot of care around ensuring the integrity of the stack and controlling exactly when in the event loop it would be okay for a thread switch to happen. The more I think about it, the more it would end up resembling the polling that's already done for the process interface with `wait_reading_process_output`. [1] https://nullprogram.com/blog/2018/05/31/ ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: "Asynchronous Requests from Emacs Dynamic Modules" 2020-11-01 0:14 ` Akira Kyle @ 2020-11-01 18:28 ` Eli Zaretskii 2020-11-01 20:15 ` Akira Kyle 0 siblings, 1 reply; 21+ messages in thread From: Eli Zaretskii @ 2020-11-01 18:28 UTC (permalink / raw) To: Akira Kyle; +Cc: p.stephani2, emacs-devel, yyoncho, monnier, all_but_last > From: Akira Kyle <ak@akirakyle.com> > Cc: yyoncho@gmail.com, all_but_last@163.com, emacs-devel@gnu.org, > monnier@iro.umontreal.ca, p.stephani2@gmail.com > Date: Sat, 31 Oct 2020 18:14:32 -0600 > > It seems the lisp threading functionality is still pretty beta > quality. Have the data races that Chris Wellons mentions here [1] > ever been addressed? It's hard to tell, because that blog has no specifics, just a broad accusation. It is also quite old. > It looks like it's mostly just exposing the pthread interface. That's simply not true. It _uses_ pthreads to start and manage threads, and to implement mutexes, condvars, etc. But if you look closely at the level of thread.c (_not_ systhread.c), you will see something very different from a typical pthreads application. > What is the use case for the current lisp threads given the severe > limitation that only will ever run at a time and the constraints the > cooperative nature puts on when threads can switch? The main use case is to allow you to write one or more background Lisp programs that go about their job while a foreground command lets the user interact normally with Emacs. ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: "Asynchronous Requests from Emacs Dynamic Modules" 2020-11-01 18:28 ` Eli Zaretskii @ 2020-11-01 20:15 ` Akira Kyle 2020-11-01 20:51 ` async-await (was: Re: "Asynchronous Requests from Emacs Dynamic Modules") Philipp Stephani 2020-11-02 15:22 ` "Asynchronous Requests from Emacs Dynamic Modules" Eli Zaretskii 0 siblings, 2 replies; 21+ messages in thread From: Akira Kyle @ 2020-11-01 20:15 UTC (permalink / raw) To: Eli Zaretskii; +Cc: p.stephani2, emacs-devel, yyoncho, monnier, all_but_last On Sun, Nov 01, 2020 at 11:28 AM, Eli Zaretskii <eliz@gnu.org> wrote: >> It seems the lisp threading functionality is still pretty beta >> quality. Have the data races that Chris Wellons mentions here >> [1] >> ever been addressed? > > It's hard to tell, because that blog has no specifics, just a > broad > accusation. It is also quite old. > >> It looks like it's mostly just exposing the pthread interface. > > That's simply not true. It _uses_ pthreads to start and manage > threads, and to implement mutexes, condvars, etc. But if you > look > closely at the level of thread.c (_not_ systhread.c), you will > see > something very different from a typical pthreads application. Sorry I meant not that it directly exposes pthreads but that its interface is modeled off of pthreads (i.e. mutexes, condition variables), but lisp should really have some higher level interface to parallel and concurrent execution (like Python's multiprocessing or asyncio interface). Perhaps it can be built on top of the current lisp threading interface, but it seems there may be some limitations that make that difficult. >> What is the use case for the current lisp threads given the >> severe >> limitation that only will ever run at a time and the >> constraints the >> cooperative nature puts on when threads can switch? > > The main use case is to allow you to write one or more > background Lisp > programs that go about their job while a foreground command lets > the > user interact normally with Emacs. Right, but due to the global lock this only allows for concurrent execution, and due to the cooperative nature needing one to explicitly yield execution for threads to switch, it seems like in reality it would be very cumbersome to work with. Do you know of anyone who uses it? I'd be very interested in seeing some examples how to successfully use it. I think libraries like emacs-aio [1] accomplish the same thing but with a much easier to use interface. By using (run-at-time 0 nil callback args) they accomplish a sort of "green threading" that doesn't actually need any sort of pthread support or the associated issues with ensuring atomicity with the interpreter. It would be great to have something like this built in. [1] https://github.com/skeeto/emacs-aio ^ permalink raw reply [flat|nested] 21+ messages in thread
* async-await (was: Re: "Asynchronous Requests from Emacs Dynamic Modules") 2020-11-01 20:15 ` Akira Kyle @ 2020-11-01 20:51 ` Philipp Stephani 2020-11-02 15:22 ` "Asynchronous Requests from Emacs Dynamic Modules" Eli Zaretskii 1 sibling, 0 replies; 21+ messages in thread From: Philipp Stephani @ 2020-11-01 20:51 UTC (permalink / raw) To: Akira Kyle Cc: Eli Zaretskii, Emacs developers, yyoncho, Stefan Monnier, all_but_last Am So., 1. Nov. 2020 um 21:15 Uhr schrieb Akira Kyle <ak@akirakyle.com>: > I think libraries like emacs-aio [1] accomplish the same thing but > with a much easier to use interface. By using (run-at-time 0 nil > callback args) they accomplish a sort of "green threading" that > doesn't actually need any sort of pthread support or the > associated issues with ensuring atomicity with the interpreter. It > would be great to have something like this built in. > > [1] https://github.com/skeeto/emacs-aio I agree (independent of the threading interface and module question) that an async-await model is really nice and powerful. It would indeed be great to better support it in Emacs. The aio library suffers from a few problems (due to Emacs, not due to the library itself), which could be rectified by implementing such support in core Emacs: 1. It makes functions virtually un-debuggable because of the necessary CPS transformations. Either Edebug should be beefed up to understand the CPS state machines, or the state machine generation should somehow happen in core. 2. It makes error reporting from asynchronous functions virtually impossible because the timer machinery catches all errors and converts them into nondescript "error in timer" messages. None of those are impossible to overcome, but they need a bit of design work to fix. ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: "Asynchronous Requests from Emacs Dynamic Modules" 2020-11-01 20:15 ` Akira Kyle 2020-11-01 20:51 ` async-await (was: Re: "Asynchronous Requests from Emacs Dynamic Modules") Philipp Stephani @ 2020-11-02 15:22 ` Eli Zaretskii 1 sibling, 0 replies; 21+ messages in thread From: Eli Zaretskii @ 2020-11-02 15:22 UTC (permalink / raw) To: Akira Kyle; +Cc: p.stephani2, emacs-devel, yyoncho, monnier, all_but_last > From: Akira Kyle <ak@akirakyle.com> > Cc: yyoncho@gmail.com, all_but_last@163.com, emacs-devel@gnu.org, > monnier@iro.umontreal.ca, p.stephani2@gmail.com > Date: Sun, 01 Nov 2020 13:15:43 -0700 > > Sorry I meant not that it directly exposes pthreads but that its > interface is modeled off of pthreads (i.e. mutexes, condition > variables), but lisp should really have some higher level > interface to parallel and concurrent execution (like Python's > multiprocessing or asyncio interface). Perhaps it can be built on > top of the current lisp threading interface, but it seems there > may be some limitations that make that difficult. Maybe it can, but I'm not sure I see the need yet. What exactly is wrong with the existing primitives? After all, the Lisp threads _are_ threads, so basic thread operations for them do make sense. > > The main use case is to allow you to write one or more background > > Lisp programs that go about their job while a foreground command > > lets the user interact normally with Emacs. > > Right, but due to the global lock this only allows for concurrent > execution, and due to the cooperative nature needing one to > explicitly yield execution for threads to switch, it seems like in > reality it would be very cumbersome to work with. Emacs yields automatically when it waits for input, so the idea was that it wouldn't be cumbersome. > Do you know of anyone who uses it? I'd be very interested in seeing > some examples how to successfully use it. Michael Albinus did, I think you can see his work on a branch in our Git repository. > I think libraries like emacs-aio [1] accomplish the same thing but > with a much easier to use interface. By using (run-at-time 0 nil > callback args) they accomplish a sort of "green threading" that > doesn't actually need any sort of pthread support or the > associated issues with ensuring atomicity with the interpreter. It > would be great to have something like this built in. run-at-time is our poor man's threading from years ago; Lisp threads were supposed to make such background applications easier to implement. ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: "Asynchronous Requests from Emacs Dynamic Modules" 2020-10-30 21:35 "Asynchronous Requests from Emacs Dynamic Modules" Akira Kyle 2020-10-30 22:39 ` Stefan Monnier 2020-10-31 3:18 ` Zhu Zihao @ 2020-10-31 7:36 ` Philipp Stephani 2020-10-31 12:49 ` Stefan Monnier 2 siblings, 1 reply; 21+ messages in thread From: Philipp Stephani @ 2020-10-31 7:36 UTC (permalink / raw) To: Akira Kyle; +Cc: Emacs developers Am Fr., 30. Okt. 2020 um 22:36 Uhr schrieb Akira Kyle <ak@akirakyle.com>: > > I'm trying to use Emacs' Dynamic Modules in an asynchronous > setting where C module functions run in a separate thread from > Emacs' main thread which is responsible for calling into the > module functions. The natural problem then is how to notify Emacs > when one wants to retrieve the result of the asynchronous > operation from some lisp. Since the module docs explicitly say > that module functions "may only interact with Emacs from Lisp > interpreter threads (including the main thread) created by Emacs", > it is not possible then for a separate module thread to interact > with Emacs in an unprompted way through the module interface [1]. > > This blog post [2] has a solution to this problem by sending > SIGUSR1 to Emacs, but that feels a bit hackish and forces one to > do more work to manage all the different possible module threads > and events that one may want to signal to Emacs. I suppose one > could also conceivably also use Emacs' dbus support but that still > feels clunky. Maybe its possible it's possible to use Emacs lisp's > threads [2] to solve this but it's not clear to me how to go about > doing so and I haven't found anyone that has done so. > > Does anyone know of the best way to handle this? Perhaps the > module interface should be expanded to handle this situation? With Emacs 28, you can get a file descriptor to a pipe process and send arbitrary data there (from arbitrary threads). You'd still need some small protocol (e.g. JSON) to encode/decode requests, but with that you can send arbitrary requests back asynchronously. ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: "Asynchronous Requests from Emacs Dynamic Modules" 2020-10-31 7:36 ` Philipp Stephani @ 2020-10-31 12:49 ` Stefan Monnier 2020-11-01 20:18 ` Akira Kyle 2020-11-20 15:54 ` Zhu Zihao 0 siblings, 2 replies; 21+ messages in thread From: Stefan Monnier @ 2020-10-31 12:49 UTC (permalink / raw) To: Philipp Stephani; +Cc: Akira Kyle, Emacs developers > With Emacs 28, you can get a file descriptor to a pipe process and > send arbitrary data there (from arbitrary threads). Cool, so that takes care of it. > You'd still need some small protocol (e.g. JSON) to encode/decode > requests, but with that you can send arbitrary requests > back asynchronously. I don't think you need a complex protocol: just stash somewhere (inside the module data structures) the data you need for the async request, then send a dummy byte to the pipe. On the Lisp side, just install a process-filter which calls back the module to "run any pending async requests". Stefan ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: "Asynchronous Requests from Emacs Dynamic Modules" 2020-10-31 12:49 ` Stefan Monnier @ 2020-11-01 20:18 ` Akira Kyle 2020-11-01 20:32 ` Philipp Stephani 2020-11-20 15:54 ` Zhu Zihao 1 sibling, 1 reply; 21+ messages in thread From: Akira Kyle @ 2020-11-01 20:18 UTC (permalink / raw) To: Stefan Monnier; +Cc: Philipp Stephani, Emacs developers On Sat, Oct 31, 2020 at 06:49 AM, Stefan Monnier <monnier@iro.umontreal.ca> wrote: >> With Emacs 28, you can get a file descriptor to a pipe process >> and >> send arbitrary data there (from arbitrary threads). > > Cool, so that takes care of it. > >> You'd still need some small protocol (e.g. JSON) to >> encode/decode >> requests, but with that you can send arbitrary requests >> back asynchronously. > > I don't think you need a complex protocol: just stash somewhere > (inside > the module data structures) the data you need for the async > request, > then send a dummy byte to the pipe. On the Lisp side, just > install > a process-filter which calls back the module to "run any pending > async > requests". Thanks Philipp for pointing this out! I'm glad this was already thought of and added so the SIGUSR1 hack isn't necessary going forward. For future reference, the module function is `open_channel`. ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: "Asynchronous Requests from Emacs Dynamic Modules" 2020-11-01 20:18 ` Akira Kyle @ 2020-11-01 20:32 ` Philipp Stephani 0 siblings, 0 replies; 21+ messages in thread From: Philipp Stephani @ 2020-11-01 20:32 UTC (permalink / raw) To: Akira Kyle; +Cc: Stefan Monnier, Emacs developers Am So., 1. Nov. 2020 um 21:18 Uhr schrieb Akira Kyle <ak@akirakyle.com>: > > > On Sat, Oct 31, 2020 at 06:49 AM, Stefan Monnier > <monnier@iro.umontreal.ca> wrote: > > >> With Emacs 28, you can get a file descriptor to a pipe process > >> and > >> send arbitrary data there (from arbitrary threads). > > > > Cool, so that takes care of it. > > > >> You'd still need some small protocol (e.g. JSON) to > >> encode/decode > >> requests, but with that you can send arbitrary requests > >> back asynchronously. > > > > I don't think you need a complex protocol: just stash somewhere > > (inside > > the module data structures) the data you need for the async > > request, > > then send a dummy byte to the pipe. On the Lisp side, just > > install > > a process-filter which calls back the module to "run any pending > > async > > requests". > > Thanks Philipp for pointing this out! I'm glad this was already > thought of and added so the SIGUSR1 hack isn't necessary going > forward. For future reference, the module function is > `open_channel`. I actually implemented open_channel after reading that very blogpost and thinking "there has to be a better way" :-) Note that even without that there are a few other options, e.g. starting a Unix domain socket server in Emacs and then connecting to it from the module. These are essentially equivalent in that they queue some form of event (process input) that can be processed asynchronously on the Emacs side. ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re:Re: "Asynchronous Requests from Emacs Dynamic Modules" 2020-10-31 12:49 ` Stefan Monnier 2020-11-01 20:18 ` Akira Kyle @ 2020-11-20 15:54 ` Zhu Zihao 2020-11-20 16:04 ` Robert Pluim 1 sibling, 1 reply; 21+ messages in thread From: Zhu Zihao @ 2020-11-20 15:54 UTC (permalink / raw) To: Stefan Monnier; +Cc: Philipp Stephani, Akira Kyle, Emacs developers [-- Attachment #1: Type: text/plain, Size: 1405 bytes --] Ok, But how to do this in Emacs version 25-27? Emacs doesn't support named pipe access, but may be we can use `make-network-process` to open a TCP socket. Like this? ``` (let ((proc (make-network-process :name "fake-proxy-process" :server t :host 'local :filter <filter-to-execute-callback> :noquery t ;; Detect port automatically :service t))) (setq <global-proc-var> proc) (module-function-initialize-notify (process-contact proc :service))) ``` But this maybe too strange for user to understand why a dynamic module require a TCP port access... At 2020-10-31 20:49:21, "Stefan Monnier" <monnier@iro.umontreal.ca> wrote: >> With Emacs 28, you can get a file descriptor to a pipe process and >> send arbitrary data there (from arbitrary threads). > >Cool, so that takes care of it. > >> You'd still need some small protocol (e.g. JSON) to encode/decode >> requests, but with that you can send arbitrary requests >> back asynchronously. > >I don't think you need a complex protocol: just stash somewhere (inside >the module data structures) the data you need for the async request, >then send a dummy byte to the pipe. On the Lisp side, just install >a process-filter which calls back the module to "run any pending async >requests". > > > Stefan > [-- Attachment #2: Type: text/html, Size: 2385 bytes --] ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: "Asynchronous Requests from Emacs Dynamic Modules" 2020-11-20 15:54 ` Zhu Zihao @ 2020-11-20 16:04 ` Robert Pluim 0 siblings, 0 replies; 21+ messages in thread From: Robert Pluim @ 2020-11-20 16:04 UTC (permalink / raw) To: Zhu Zihao; +Cc: Philipp Stephani, Stefan Monnier, Akira Kyle, Emacs developers "Zhu Zihao" <all_but_last@163.com> writes: > Ok, But how to do this in Emacs version 25-27? > > > Emacs doesn't support named pipe access, but may be we can use `make-network-process` to open a TCP socket. Like this? > > > ``` > (let ((proc (make-network-process :name "fake-proxy-process" > :server t > :host 'local > :filter <filter-to-execute-callback> > :noquery t > ;; Detect port automatically > :service t))) > (setq <global-proc-var> proc) > (module-function-initialize-notify (process-contact proc :service))) > ``` > > > But this maybe too strange for user to understand why a dynamic module require a TCP port access... It doesnʼt: you can use unix sockets by specifying :family 'local Robert ^ permalink raw reply [flat|nested] 21+ messages in thread
end of thread, other threads:[~2020-11-20 16:04 UTC | newest] Thread overview: 21+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2020-10-30 21:35 "Asynchronous Requests from Emacs Dynamic Modules" Akira Kyle 2020-10-30 22:39 ` Stefan Monnier 2020-10-31 3:18 ` Zhu Zihao 2020-10-31 7:45 ` Eli Zaretskii 2020-10-31 8:02 ` yyoncho 2020-10-31 9:13 ` Eli Zaretskii 2020-10-31 9:45 ` yyoncho 2020-10-31 10:43 ` Eli Zaretskii 2020-10-31 19:25 ` Akira Kyle 2020-10-31 20:18 ` Eli Zaretskii 2020-11-01 0:14 ` Akira Kyle 2020-11-01 18:28 ` Eli Zaretskii 2020-11-01 20:15 ` Akira Kyle 2020-11-01 20:51 ` async-await (was: Re: "Asynchronous Requests from Emacs Dynamic Modules") Philipp Stephani 2020-11-02 15:22 ` "Asynchronous Requests from Emacs Dynamic Modules" Eli Zaretskii 2020-10-31 7:36 ` Philipp Stephani 2020-10-31 12:49 ` Stefan Monnier 2020-11-01 20:18 ` Akira Kyle 2020-11-01 20:32 ` Philipp Stephani 2020-11-20 15:54 ` Zhu Zihao 2020-11-20 16:04 ` Robert Pluim
Code repositories for project(s) associated with this public inbox https://git.savannah.gnu.org/cgit/emacs.git This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).