Checking if this is a Eshell bug in emacs 29: Should eshell redirect all output in esh script to stdout

Checking if this is a Eshell bug in emacs 29: Should eshell redirect all output in esh script to stdout

[Milan Zimmermann]

[-- Attachment #1: Type: text/plain, Size: 753 bytes --]

Hi,

I would like to check here before reporting an Eshell bug.

A script named 'redirect-echo.esh' with two echo commands.

Emacs 29: Sourcing it and redirecting output to a file, results in the
first echo string ('hello') showing in eshell, only the second
('there')(presumably because it is last) showing in the output file:

tmp $ cat redirect-echo.esh
echo hello
echo there
tmp $ source redirect-echo.esh > redirect-echo.out
hello
tmp $ cat redirect-echo.out
theretmp $

As a note, the same behavior if elisp "print" is used instead of echo. Also
the same behavior if I redirect output to an Emacs buffer instead of the
file.

 I believe this is a bug but I am not sure about Eshell's intended
features, so I wanted to check first.

Thanks,

Milan

[-- Attachment #2: Type: text/html, Size: 936 bytes --]

Re: Checking if this is a Eshell bug in emacs 29: Should eshell redirect all output in esh script to stdout

[Jim Porter]

On 11/23/2022 10:56 AM, Milan Zimmermann wrote:
> I would like to check here before reporting an Eshell bug.
> 
> A script named 'redirect-echo.esh' with two echo commands.
> 
> Emacs 29: Sourcing it and redirecting output to a file, results in the 
> first echo string ('hello') showing in eshell, only the second 
> ('there')(presumably because it is last) showing in the output file:
[snip]
>   I believe this is a bug but I am not sure about Eshell's intended 
> features, so I wanted to check first.

This is definitely a bug. I think the best way to fix it though would be 
to first replace 'eshell-do-eval', since it's a bit limited in what Lisp 
forms it can manage correctly.

(Warning: the following is based on my memory of looking at this a few 
months ago, so I might have misremembered some details. If so, sorry 
about that.)

Currently, Eshell manages output handles in a fairly tricky way: many 
internal Eshell forms automatically try to close the handles, so to get 
around this, other Eshell forms increment the handles' refcount so that 
closing just decrements that count instead of actually closing it. 
That's a fine strategy in general, but the increment-refcount 
('eshell-protect') and decrement-refcount-and-close 
('eshell-close-handles') are called very far from each other in the 
code, so the logic gets pretty hard to follow, leading to bugs like this.

I think the best way to fix this would be to put the increment and 
decrement code in the same spot, so that you'd do something like so:

   (unwind-protect
       (progn
         (eshell-protect)
         (do-stuff))
     (eshell-close-handles))

Unfortunately, if 'do-stuff' calls a deferred command[1], Eshell ends up 
evaluating 'eshell-close-handles' immediately, instead of when 
'do-stuff' actually finishes. That means that either a) we'd need to fix 
this without using 'unwind-protect', which would be painful to get 
right, or b) we'd need to make 'unwind-protect' work as expected. (b) is 
effectively bug#57635[2], I think. (That bug suggests using 
generator.el's internals to drive Eshell's iterative evaluation.)

Bug#57635 is a pretty big change too, but I think it would greatly help 
with the overall reliability of Eshell.

[1] Basically, a command that should yield execution back to the rest of 
Emacs while waiting on some background task (e.g. a subprocess).

[2] https://debbugs.gnu.org/cgi/bugreport.cgi?bug=57635

Re: Checking if this is a Eshell bug in emacs 29: Should eshell redirect all output in esh script to stdout

[Milan Zimmermann]

[-- Attachment #1: Type: text/plain, Size: 3430 bytes --]

Jim,

Thanks for the explanation.

Well, it looks like I keep running against issues that would likely be gone
if https://debbugs.gnu.org/cgi/bugreport.cgi?bug=57635 is addressed.
Unfortunately I do not have much confidence in my ability to help ,as it
would take me a long time to even get to the point of my Elisp to start
attempting.

On the limited scope of this output issue: Should I create a bug out of
this, or do you want to go ahead and report it? (or something else)

Thanks,
Milan

PS: I needed the stdout and stderr redirection to behave properly to be
able to finish my Org-babel ob-eshell project I started a few days ago.
There is already an existing ob-eshell, but it does not work for me, so I
started a re-do.  (I mostly want :results output to work). I see now that
effort is on a backburner for now until the redirection works.





On Wed, Nov 23, 2022 at 3:30 PM Jim Porter <jporterbugs@gmail.com> wrote:

> On 11/23/2022 10:56 AM, Milan Zimmermann wrote:
> > I would like to check here before reporting an Eshell bug.
> >
> > A script named 'redirect-echo.esh' with two echo commands.
> >
> > Emacs 29: Sourcing it and redirecting output to a file, results in the
> > first echo string ('hello') showing in eshell, only the second
> > ('there')(presumably because it is last) showing in the output file:
> [snip]
> >   I believe this is a bug but I am not sure about Eshell's intended
> > features, so I wanted to check first.
>
> This is definitely a bug. I think the best way to fix it though would be
> to first replace 'eshell-do-eval', since it's a bit limited in what Lisp
> forms it can manage correctly.
>
> (Warning: the following is based on my memory of looking at this a few
> months ago, so I might have misremembered some details. If so, sorry
> about that.)
>
> Currently, Eshell manages output handles in a fairly tricky way: many
> internal Eshell forms automatically try to close the handles, so to get
> around this, other Eshell forms increment the handles' refcount so that
> closing just decrements that count instead of actually closing it.
> That's a fine strategy in general, but the increment-refcount
> ('eshell-protect') and decrement-refcount-and-close
> ('eshell-close-handles') are called very far from each other in the
> code, so the logic gets pretty hard to follow, leading to bugs like this.
>
> I think the best way to fix this would be to put the increment and
> decrement code in the same spot, so that you'd do something like so:
>
>    (unwind-protect
>        (progn
>          (eshell-protect)
>          (do-stuff))
>      (eshell-close-handles))
>
> Unfortunately, if 'do-stuff' calls a deferred command[1], Eshell ends up
> evaluating 'eshell-close-handles' immediately, instead of when
> 'do-stuff' actually finishes. That means that either a) we'd need to fix
> this without using 'unwind-protect', which would be painful to get
> right, or b) we'd need to make 'unwind-protect' work as expected. (b) is
> effectively bug#57635[2], I think. (That bug suggests using
> generator.el's internals to drive Eshell's iterative evaluation.)
>
> Bug#57635 is a pretty big change too, but I think it would greatly help
> with the overall reliability of Eshell.
>
> [1] Basically, a command that should yield execution back to the rest of
> Emacs while waiting on some background task (e.g. a subprocess).
>
> [2] https://debbugs.gnu.org/cgi/bugreport.cgi?bug=57635
>

[-- Attachment #2: Type: text/html, Size: 4556 bytes --]

Re: Checking if this is a Eshell bug in emacs 29: Should eshell redirect all output in esh script to stdout

[Eli Zaretskii]

> Date: Wed, 23 Nov 2022 12:30:45 -0800
> From: Jim Porter <jporterbugs@gmail.com>
> 
> On 11/23/2022 10:56 AM, Milan Zimmermann wrote:
> > I would like to check here before reporting an Eshell bug.
> > 
> > A script named 'redirect-echo.esh' with two echo commands.
> > 
> > Emacs 29: Sourcing it and redirecting output to a file, results in the 
> > first echo string ('hello') showing in eshell, only the second 
> > ('there')(presumably because it is last) showing in the output file:
> [snip]
> >   I believe this is a bug but I am not sure about Eshell's intended 
> > features, so I wanted to check first.
> 
> This is definitely a bug.

Would you two please move the discussion of this bug to the bug tracker?
Please file a bug report with the details, and continue discussing the
solution there.

TIA

Re: Checking if this is a Eshell bug in emacs 29: Should eshell redirect all output in esh script to stdout

[Jim Porter]

On 11/23/2022 10:20 PM, Eli Zaretskii wrote:
> Would you two please move the discussion of this bug to the bug tracker?
> Please file a bug report with the details, and continue discussing the
> solution there.

Agreed, I think it would be good to capture this in a bug report so it 
doesn't get lost. This might actually be the same issue as bug#59400. 
(I'll need some time to look at it in more detail to be sure though.) 
Still, it'd probably be fine to file this separately and merge if they 
do turn out to be the same.