[RFC] Adding threads to Eshell

[RFC] Adding threads to Eshell

[Jim Porter]

First, a bit of context. For those who don't know, Eshell has an 
"iterative evaluation" feature, where any time you run an asynchronous 
command (like an external process), it yields execution back to the rest 
of Emacs so you're not stuck waiting for a long-running command. This is 
handled by having Eshell evaluate Lisp forms step-by-step (see 
'eshell-do-eval' in lisp/eshell/esh-cmd.el). This requires a lot of code 
to handle various Lisp special forms, and the allowed forms are fairly 
limited. It also has a few bugs, likely due in part to being written 
before lexical binding.

Because of this (especially the bugs), I think it would be good to 
retire 'eshell-do-eval' and switch to something else. Initially, I filed 
bug#57635 to switch over to generator.el's machinery, but then I 
realized that Emacs threads might work even better. In addition to 
fixing the bugs (which would unlock several new features), I think 
threads would make it pretty straightforward to add job control to 
Eshell. I did a few quick tests, and found that it only took a bit of 
work to get the basics working with threads.

However, before I go too far, I wanted to check with other, more 
knowledgeable people: are Emacs threads available on all platforms? It 
looks like it requires the pthreads library (though my understanding was 
that Emacs threads aren't "real" threads, since there's no good way to 
avoid data races with globals). If we want to support asynchronous 
execution in Eshell on systems that don't have access to Emacs threads, 
that might be a problem. On the other hand, if those systems *also* lack 
support for asynchronous processes, then we're probably ok: on those 
systems, Eshell already does most execution synchronously anyway.

A second issue I noticed is that Emacs threads have their own, 
completely separate set of lexical bindings. Is it possible to tell a 
thread that I want to inherit the bindings from wherever I called 
'make-thread'? I might be able to avoid this issue, but it would be nice 
to be able to let-bind some defvars and then pass those bindings to an 
Eshell command to do its thing.

Depending on how people respond, I could go with either Emacs threads or 
some abstraction of generator.el's machinery to upgrade Eshell's 
iterative evaluation. Whichever way I go, I'm planning on making a 
feature branch once I have something ready for others to look at, since 
it's shaping up to be a pretty big change.

Re: [RFC] Adding threads to Eshell

[Stefan Monnier]

> lisp/eshell/esh-cmd.el).  This requires a lot of code to handle various Lisp
> special forms, and the allowed forms are fairly limited.  It also has a few
> bugs, likely due in part to being written before lexical binding.

I remember discussing this with John the one time we got to meet face to
face and him telling me that he didn't know the concept of
CPS-conversion back then :-)

> However, before I go too far, I wanted to check with other, more
> knowledgeable people: are Emacs threads available on all platforms?

I'll let others answer this part.  IIUC the answer is not "yes", but
like you I hope the cases where they're not supported can be simply
swept under the rug.

> A second issue I noticed is that Emacs threads have their own,
> completely separate set of lexical bindings.

Do you really mean *lexical* bindings?
If yes, then I don't understand what you mean (I don't understand how
they could be non-completely separate).

> Is it possible to tell a thread that I want to inherit the bindings
> from wherever I called 'make-thread'?

For statically scoped vars, this should already be the case because the
(lambda ....) you pass to `make-thread` will close over the surrounding
static scopes.

For dynamically scoped vars, you'd have to do it more explicitly.
Lisp Machine Lisp had an operation they called `closure` for that
purpose (they didn't have static scoping at all).
See https://hanshuebner.github.io/lmman/fd-clo.xml

I've had some fun writing an `lml.el` compatibility package which
includes support for that.  The corresponding part of the code is:

    (oclosure-define (lml-closure
                      (:predicate lml-closurep))
      bindings function)

    (defun lml--closure (bindings function)
      (oclosure-lambda (lml-closure (bindings bindings)
                                    (function function))
          (&rest args)
        (cl-progv (mapcar #'car bindings) (mapcar #'cdr bindings)
          (apply function args))))

    (defun lml-closure (varlist function)
      "Create a closure over the dynamic variables in VARLIST."
      (lml--closure (mapcar (lambda (v) (cons v (symbol-value v))) varlist)
                    function))


-- Stefan

Re: [RFC] Adding threads to Eshell

[Eli Zaretskii]

> Date: Thu, 15 Dec 2022 18:37:09 -0800
> From: Jim Porter <jporterbugs@gmail.com>
> 
> However, before I go too far, I wanted to check with other, more 
> knowledgeable people: are Emacs threads available on all platforms?

Yes.  The only known exception is MSDOS.

> It looks like it requires the pthreads library

pthreads are required only on Posix platforms; on Windows we use
Windows-specific APIs instead.  See systhread.c.

> (though my understanding was 
> that Emacs threads aren't "real" threads, since there's no good way to 
> avoid data races with globals).

No, they are real threads.  You can see them with any OS-level tool
that can examine threads.  We just let only a single one of these
threads to run at any given time (well, with the exception of very
short time windows).

> A second issue I noticed is that Emacs threads have their own, 
> completely separate set of lexical bindings.

Not lexical bindings: local bindings.  That is, if you let-bind a
variable, that binding is only seen in the thread that performed the
binding.

> Is it possible to tell a thread that I want to inherit the bindings
> from wherever I called 'make-thread'?

Global bindings are always shared.

> I might be able to avoid this issue, but it would be nice to be able
> to let-bind some defvars and then pass those bindings to an Eshell
> command to do its thing.

If that is what you want, you will have to pass those bindings as
arguments to the thread function.

Re: [RFC] Adding threads to Eshell

[Jim Porter]

On 12/16/2022 6:05 AM, Stefan Monnier wrote:
>> However, before I go too far, I wanted to check with other, more
>> knowledgeable people: are Emacs threads available on all platforms?
> 
> I'll let others answer this part.  IIUC the answer is not "yes", but
> like you I hope the cases where they're not supported can be simply
> swept under the rug.

I'm not sure it's worth the added complexity (yet?), but maybe it would 
be useful to have some generic way of executing code asynchronously 
(probably using CPS-transformations). Then you could pass this code to a 
thread if those are available, or if not, just run in the main thread 
when Emacs is idle.

>> A second issue I noticed is that Emacs threads have their own,
>> completely separate set of lexical bindings.
> 
> Do you really mean *lexical* bindings?
> If yes, then I don't understand what you mean (I don't understand how
> they could be non-completely separate).

I suppose I mean *local* bindings, really. The manual says, "The new 
thread is created with no local variable bindings in effect." That's 
something that I think would be nice to change in this case, so that a 
user could do something like:

   (let ((some-defvar t))
     (eshell-command "echo $some-defvar"))

Internally, 'eshell-command' would convert "echo $some-defvar" to some 
Lisp form, then call 'make-thread' and eval that form in the thread.

> For dynamically scoped vars, you'd have to do it more explicitly.
> Lisp Machine Lisp had an operation they called `closure` for that
> purpose (they didn't have static scoping at all).
> See https://hanshuebner.github.io/lmman/fd-clo.xml

That sounds like what I want. Thanks for the code snippet.

The only bit I'm still unsure about is if there's a good (and 
performant) way to ask, "What are all the local bindings of dynamic 
variables?" Then, I could use that to feed those bindings to the thread 
and this would work. (I'm still not sure whether this is the right thing 
to do in the first place, but I want to see what it's like. It would 
also make more of the regression tests pass, which would make it easier 
to keep developing this.)

Re: [RFC] Adding threads to Eshell

[Jim Porter]

On 12/16/2022 11:51 AM, Eli Zaretskii wrote:
>> Date: Thu, 15 Dec 2022 18:37:09 -0800
>> From: Jim Porter <jporterbugs@gmail.com>
>>
>> However, before I go too far, I wanted to check with other, more
>> knowledgeable people: are Emacs threads available on all platforms?
> 
> Yes.  The only known exception is MSDOS.

Ok, so it looks like MSDOS is also the only platform that lacks support 
for async subprocesses. In that case, I think it would be ok to require 
threads for Eshell if you want to execute commands asynchronously.

>> It looks like it requires the pthreads library
> 
> pthreads are required only on Posix platforms; on Windows we use
> Windows-specific APIs instead.  See systhread.c.

Ah ha. I had somehow gotten it into my head that Emacs used pthread 
wrappers on Win32.

> No, they are real threads.  You can see them with any OS-level tool
> that can examine threads.  We just let only a single one of these
> threads to run at any given time (well, with the exception of very
> short time windows).

Oh, ok. I don't think that will make anything harder to implement 
though, so no worries there.

>> I might be able to avoid this issue, but it would be nice to be able
>> to let-bind some defvars and then pass those bindings to an Eshell
>> command to do its thing.
> 
> If that is what you want, you will have to pass those bindings as
> arguments to the thread function.

Like I said in my message to Stefan, I'm not totally sure if this is the 
right way to go, but I want to experiment with it a bit more. It doesn't 
look too hard to pass those bindings in once I have them, but I don't 
know of a good way to get the list of all local bindings of dynamic 
variables in the first place.

Re: [RFC] Adding threads to Eshell

[Stefan Monnier]

> I suppose I mean *local* bindings, really.

I must admit that I don't know what is "a local binding".
More specifically, I believe it is a term that can mean many different
things in many different contexts.

>   (let ((some-defvar t))
>     (eshell-command "echo $some-defvar"))

If `eshell-command` is a macro which turns "echo $some-defvar" into
a chunk of ELisp with a reference to a `some-defvar` variable, then
this var could be statically scoped and this code could work as-is
simple by virtual of `lambda` capturing that statically scoped var.
If `eshell-command` is a function then the above *cannot* work with
statically scoped `some-defvar`.

For dynamically scoped vars, then you'll need to do something like what
the `lml-closure` does, but that shouldn't be hard.

> The only bit I'm still unsure about is if there's a good (and performant)
> way to ask, "What are all the local bindings of dynamic variables?" Then,

We could potentially add such a thing (except for the "performant" part,
but I think it would be fast enough anyway), but there isn't one, no,
and it's not very easy to add one (e.g. because some of the currently
active dynamic bindings can be local to specific buffers).


        Stefan

Re: [RFC] Adding threads to Eshell

[Jim Porter]

On 12/16/2022 7:40 PM, Stefan Monnier wrote:
>> I suppose I mean *local* bindings, really.
> 
> I must admit that I don't know what is "a local binding".

I guess just let-bindings (and anything else that works like 'let'). 
Maybe the manual section on threads should be updated, since "local 
variable bindings" *could* mean buffer-local variables, but I think it's 
*supposed* to mean let-bindings. (Threads have their own current buffer, 
so I'd expect that buffer-local bindings take effect for the thread's 
current buffer as usual. I haven't actually tried it out though.)

> For dynamically scoped vars, then you'll need to do something like what
> the `lml-closure` does, but that shouldn't be hard.

To preserve the existing behavior, I think all we'd need is to copy any 
let-bindings for dynamically-scoped vars. Eshell already ignores 
statically-scoped bindings (though changing that would present some 
interesting possibilities...)

>> The only bit I'm still unsure about is if there's a good (and performant)
>> way to ask, "What are all the local bindings of dynamic variables?" Then,
> 
> We could potentially add such a thing (except for the "performant" part,
> but I think it would be fast enough anyway), but there isn't one, no,
> and it's not very easy to add one (e.g. because some of the currently
> active dynamic bindings can be local to specific buffers).

Well, hopefully it wouldn't cost too much performance, or else we'd 
probably want to have users opt into it. But we can cross that bridge 
later. The lml-closure example you posted gave me enough to get a 
somewhat-clumsy solution working for the regression tests at least, so I 
can keep hacking away.