Re: Checking if this is a Eshell bug in emacs 29: Should eshell redirect all output in esh script to stdout

[Jim Porter <jporterbugs@gmail.com>]

On 11/23/2022 10:56 AM, Milan Zimmermann wrote:
> I would like to check here before reporting an Eshell bug.
> 
> A script named 'redirect-echo.esh' with two echo commands.
> 
> Emacs 29: Sourcing it and redirecting output to a file, results in the 
> first echo string ('hello') showing in eshell, only the second 
> ('there')(presumably because it is last) showing in the output file:
[snip]
>   I believe this is a bug but I am not sure about Eshell's intended 
> features, so I wanted to check first.

This is definitely a bug. I think the best way to fix it though would be 
to first replace 'eshell-do-eval', since it's a bit limited in what Lisp 
forms it can manage correctly.

(Warning: the following is based on my memory of looking at this a few 
months ago, so I might have misremembered some details. If so, sorry 
about that.)

Currently, Eshell manages output handles in a fairly tricky way: many 
internal Eshell forms automatically try to close the handles, so to get 
around this, other Eshell forms increment the handles' refcount so that 
closing just decrements that count instead of actually closing it. 
That's a fine strategy in general, but the increment-refcount 
('eshell-protect') and decrement-refcount-and-close 
('eshell-close-handles') are called very far from each other in the 
code, so the logic gets pretty hard to follow, leading to bugs like this.

I think the best way to fix this would be to put the increment and 
decrement code in the same spot, so that you'd do something like so:

   (unwind-protect
       (progn
         (eshell-protect)
         (do-stuff))
     (eshell-close-handles))

Unfortunately, if 'do-stuff' calls a deferred command[1], Eshell ends up 
evaluating 'eshell-close-handles' immediately, instead of when 
'do-stuff' actually finishes. That means that either a) we'd need to fix 
this without using 'unwind-protect', which would be painful to get 
right, or b) we'd need to make 'unwind-protect' work as expected. (b) is 
effectively bug#57635[2], I think. (That bug suggests using 
generator.el's internals to drive Eshell's iterative evaluation.)

Bug#57635 is a pretty big change too, but I think it would greatly help 
with the overall reliability of Eshell.

[1] Basically, a command that should yield execution back to the rest of 
Emacs while waiting on some background task (e.g. a subprocess).

[2] https://debbugs.gnu.org/cgi/bugreport.cgi?bug=57635