Is there any easy way to fork in elisp? Why not?

Is there any easy way to fork in elisp? Why not?

[Rudi C]

[-- Attachment #1: Type: text/plain, Size: 1486 bytes --]

There are some scripts that I find convenient to code in elisp,  and I like
to be able to run these with little startup time.

The natural solution is to use `emacs --batch --eval` together with
`server-eval-at` to reuse an already loaded server, but this solution needs
a lot of throwaway servers already started and a load balancer. (Is there a
Jupyter kernel for elisp? I already have a prototype server (
https://github.com/NightMachinary/JupyterGarden) that starts throwaway
Jupyter kernels and load-balances requests to them.)

If the server could just fork, eval the request, return it, and die,
everything would be easy and painless.

The fork API could be:
```
(let-fork ((some_val (expensive_compute)) ...)
 (lambda()
   (message "Computed: %s" some_val)))
```
Where emacs forks for every binding, and computes the results, and finally,
asynchronously calls the supplied callback function with all of the
bindings available, similar to a closure.

I am sure there are better API designs, but my point is that this fork API
should be very easy and work with synchronous third-party code, just like a
shell (or GNU Parallel). Parallelism via multiprocessing should be easy and
painless.

The third-party packages for parallelism that I have skimmed, start a fresh
emacs instance, and do not fork, which is useless, as all the needed data
and behavior needs to be manually reloaded by the user.

PS: Please use reply-to-all, I am not currently subscribed to the list.
Thank you.

[-- Attachment #2: Type: text/html, Size: 1798 bytes --]

Re: Is there any easy way to fork in elisp? Why not?

[Zhiwei Chen]

Rudi C <rudiwillalwaysloveyou@gmail.com> writes:

> The fork API could be:
> ```
> (let-fork ((some_val (expensive_compute)) ...)  
>  (lambda()
>    (message "Computed: %s" some_val)))
> ``` 
> Where emacs forks for every binding, and computes the results, and finally, asynchronously calls the supplied callback function with all of the
> bindings available, similar to a closure.
>
> I am sure there are better API designs, but my point is that this fork API should be very easy and work with synchronous third-party code, just like a
> shell (or GNU Parallel). Parallelism via multiprocessing should be easy and painless.
>
> The third-party packages for parallelism that I have skimmed, start a fresh emacs instance, and do not fork, which is useless, as all the needed data
> and behavior needs to be manually reloaded by the user.
>
> PS: Please use reply-to-all, I am not currently subscribed to the list. Thank you.
>

Maybe `async-let' https://github.com/jwiegley/emacs-async/blob/master/async.el#L405 is what your want.

-- 
Zhiwei Chen

Re: Is there any easy way to fork in elisp? Why not?

[Stefan Monnier]

> If the server could just fork, eval the request, return it, and die,
> everything would be easy and painless.
>
> The fork API could be:
> ```
> (let-fork ((some_val (expensive_compute)) ...)
>  (lambda()
>    (message "Computed: %s" some_val)))
> ```

We should indeed add such support in core.  Currently the `async`
package offers such functionality, but I think this belongs in core.

More generally a kind of "Emacs batch server" will be useful.
Something similar to the server.el but whose purpose would be to launch
independent/fresh batch Emacs sessions efficiently.

A long time ago I suggested to add support for the `fork`
POSIX primitive which could be ideal for that, but AFAIK it's hard to
provide such a functionality in w32, so we'd want the Lisp-level API not
to be tied to the semantics of POSIX `fork`.


        Stefan

Re: Is there any easy way to fork in elisp? Why not?

[Rudi C]

[-- Attachment #1: Type: text/plain, Size: 1454 bytes --]

I am not an expert, but if adding forks for all platforms is so difficult
that it cannot be added in the foreseeable future, please consider just not
supporting the fork API on Windows. (Windows can also fallback to WSL,
though I haven't tested it. Has anyone benchmarked the WSL version versus
native Windows?)

PS: I repeat, the current third-party async packages do NOT offer much of
anything; They just start a fresh emacs, without the variables/functions of
the parent process.

On Fri, Oct 29, 2021 at 1:36 AM Stefan Monnier <monnier@iro.umontreal.ca>
wrote:

> > If the server could just fork, eval the request, return it, and die,
> > everything would be easy and painless.
> >
> > The fork API could be:
> > ```
> > (let-fork ((some_val (expensive_compute)) ...)
> >  (lambda()
> >    (message "Computed: %s" some_val)))
> > ```
>
> We should indeed add such support in core.  Currently the `async`
> package offers such functionality, but I think this belongs in core.
>
> More generally a kind of "Emacs batch server" will be useful.
> Something similar to the server.el but whose purpose would be to launch
> independent/fresh batch Emacs sessions efficiently.
>
> A long time ago I suggested to add support for the `fork`
> POSIX primitive which could be ideal for that, but AFAIK it's hard to
> provide such a functionality in w32, so we'd want the Lisp-level API not
> to be tied to the semantics of POSIX `fork`.
>
>
>         Stefan
>
>

[-- Attachment #2: Type: text/html, Size: 1908 bytes --]

Re: Is there any easy way to fork in elisp? Why not?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > I am not an expert, but if adding forks for all platforms is so difficult
  > that it cannot be added in the foreseeable future, please consider just not
  > supporting the fork API on Windows.

If we decide to support this fork feature, to leave it unimplemented
on Windows is ok as an option.  We won't reject a good feature on
account of its being unimplemented on Windows.

However, we need to think carefully about the feature, whether it's a
good idea, whether there are difficulties about it.  So we need a
clear statement of what it should do.

What exactly does the feature consist of?  I suppose it creates
another process, but what then happens to each of the two processes?

We may need to figure out what it should do in various situations.
For instance, how does each one get command input?  How does each one
display?  What happens if the Emacs job has subprocesses, or sockets?

What range of situations should it handle?  A narrower range of cases
may be easier to implement, but may be less useful.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Is there any easy way to fork in elisp? Why not?

[Philipp]

> Am 26.10.2021 um 14:44 schrieb Rudi C <rudiwillalwaysloveyou@gmail.com>:
> 
> There are some scripts that I find convenient to code in elisp,  and I like to be able to run these with little startup time. 
> 
> The natural solution is to use `emacs --batch --eval` together with `server-eval-at` to reuse an already loaded server, but this solution needs a lot of throwaway servers already started and a load balancer. (Is there a Jupyter kernel for elisp? I already have a prototype server (https://github.com/NightMachinary/JupyterGarden) that starts throwaway Jupyter kernels and load-balances requests to them.)
> 
> If the server could just fork, eval the request, return it, and die, everything would be easy and painless. 
> 
> The fork API could be:
> ```
> (let-fork ((some_val (expensive_compute)) ...)  
>  (lambda()
>    (message "Computed: %s" some_val)))
> ``` 
> Where emacs forks for every binding, and computes the results, and finally, asynchronously calls the supplied callback function with all of the bindings available, similar to a closure.
> 
> I am sure there are better API designs, but my point is that this fork API should be very easy and work with synchronous third-party code, just like a shell (or GNU Parallel). Parallelism via multiprocessing should be easy and painless.
> 
> The third-party packages for parallelism that I have skimmed, start a fresh emacs instance, and do not fork, which is useless, as all the needed data and behavior needs to be manually reloaded by the user.

If you're talking about the Unix fork system call (or similar calls such as clone), then the answer is that it's not possible to fork and do interesting work in the child from any process that might use multiple threads (which includes most "interesting" interactive programs, and definitely includes Emacs). The reason is that the other threads might be in a critical section, but since the threads are gone in the child the critical section can never left, so the child has to avoid any code that might enter any critical section, which precludes calling any function that isn't async-signal-safe, and even heap memory allocation isn't async-signal-safe. Therefore programs can generally only call execve (which removes the critical sections altogether) plus a small number of other functions to set up the child process (chdir, setsid...) after forking.

If you're talking about some more generic concept similar to the Unix fork function implemented completely in userspace (e.g. by creating a thread and either eagerly or lazily copying the entire state of the Lisp interpreter), then such a thing would be possible in principle, but face quite some difficulties. The copying is difficult to get right, and for some objects (such as processes) it's hard to define what the semantics of the copy should be. It's much easier to start with a minimal known-good interpreter state and modify only the pieces of it which are needed to execute the code in question, which is exactly what the asynchronous frameworks such as emacs-async are doing.

Another relatively common way to achieve parallelism is to share VM between multiple processes ("threads"). However, that requires carefully synchronizing every access to shared mutable state, and Emacs Lisp code generally doesn't do that, so this approach is infeasible for Emacs as well.

Re: Is there any easy way to fork in elisp? Why not?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

    > The fork API could be:
    > ```
    > (let-fork ((some_val (expensive_compute)) ...)  
    >  (lambda()
    >    (message "Computed: %s" some_val)))

I don't see why this needs to do a fork at all.
If `expensive_compute' does not set any free variables
and does not modify preexisting data structures,
it should work to define `let-fork' as equivalent to `let'.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)