bug#57635: 29.0.50; Look into using generator.el's CPS machinery to replace 'eshell-do-eval'

bug#57635: 29.0.50; Look into using generator.el's CPS machinery to replace 'eshell-do-eval'

[Jim Porter]

In order to make long-running Eshell commands behave more-nicely, Eshell 
uses iterative evaluation of many commands so that it can return control 
back to the rest of Emacs. This has a lot of similarity to how 
generator.el works.

This would hopefully help prevent obscure bugs in Eshell, such as 
bug#54190. In that bug, Stefan mentions[1]:

> Looks like a bug somewhere in the `eshell-do-eval` machinery, yes.
> Until we find the problem (or rewrite `eshell-do-eval` on top of the CPS
> converter of `generator.el`), I think reverting this change (and adding
> a comment pointing to this bug) sounds like a great plan.

[1] https://debbugs.gnu.org/cgi/bugreport.cgi?bug=54190#17

bug#57635: 29.0.50; Look into using generator.el's CPS machinery to replace 'eshell-do-eval'

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> In order to make long-running Eshell commands behave more-nicely, Eshell
> uses iterative evaluation of many commands so that it can return control
> back to the rest of Emacs. This has a lot of similarity to how
> generator.el works.

Indeed, the idea would be to replace the commands that launch processes
with "yields" to an external loop (that just launches the next process
and then calls the generator again).


        Stefan

bug#57635: 29.0.50; Look into using generator.el's CPS machinery to replace 'eshell-do-eval'

[Jim Porter]

On 9/7/2022 5:45 AM, Stefan Monnier via Bug reports for GNU Emacs, the 
Swiss army knife of text editors wrote:
> Indeed, the idea would be to replace the commands that launch processes
> with "yields" to an external loop (that just launches the next process
> and then calls the generator again).

That part should be fairly straightforward; there's already 
'eshell-eval-command' and friends ('eshell-resume-command', 
'eshell-resume-eval', and 'eshell-do-eval'), which work pretty much like 
this already. Looking through this, I see a few main differences between 
how Eshell does things and how generator.el does things:

----------------------------------------

1) Eshell uses (throw 'eshell-defer) to yield

This is probably easy to fix, since it's only used in a couple places, 
and it looks like it's purely an implementation detail.

2) Eshell doesn't have anything like 'iter-defun'

Eshell just evaluates a Lisp form directly (you can see this by tracing 
'eshell-eval-command' and then running some external process in Eshell). 
I'm not sure how hard it would be to get generator.el to be ok with this.

3) Eshell allows replacing a form by throwing 'eshell-replace-command'

This is the trickiest part (to me, anyway). It's typically used by 
Eshell built-in commands when they want to give up and run an external 
process, though it can be used for a bunch of other things. I'm not sure 
what the best way to handle this via generator.el would be. Is there any 
straightforward way to do this?

----------------------------------------

If anyone has ideas, I'm all ears. This is pretty far down into the 
weeds of Lisp metaprogramming, so I'm just making some educated guesses 
here. I think doing this would be a big improvement for Eshell though, 
since I've already bumped up against a few cases that really look like 
they're just bugs in 'eshell-do-eval'. Moving to generator.el would 
hopefully fix those and make Eshell easier to maintain/improve.

bug#57635: 29.0.50; Look into using generator.el's CPS machinery to replace 'eshell-do-eval'

[Jim Porter]

On 9/7/2022 5:45 AM, Stefan Monnier via Bug reports for GNU Emacs, the 
Swiss army knife of text editors wrote:
>> In order to make long-running Eshell commands behave more-nicely, Eshell
>> uses iterative evaluation of many commands so that it can return control
>> back to the rest of Emacs. This has a lot of similarity to how
>> generator.el works.
> 
> Indeed, the idea would be to replace the commands that launch processes
> with "yields" to an external loop (that just launches the next process
> and then calls the generator again).

Maybe an even better solution would be to use Emacs threads. This should 
do everything that Eshell needs, plus hopefully add the ability to add 
some basic job control as well. (Well, one day...)

I hacked something up that seems to work for simple cases, and it 
shouldn't be *too* hard to get it working more generally. This will be a 
big change for Eshell, so it might make sense to create a feature branch 
when I have a little more to show. However, I can already see how this 
will eliminate the need for a number of workarounds in Eshell's command 
invocation logic.