bug#34123: A patch to fix reading EOF characters in non-interactive mode

bug#34123: A patch to fix reading EOF characters in non-interactive mode

[Shawn Presser]

[-- Attachment #1.1: Type: text/plain, Size: 1525 bytes --]

Hello,

Here is a patch to allow batch mode scripts to handle EOF characters. This
solves the following problem:

$ git clone https://github.com/shawwn/y
$ cd y
$ bin/y
> (read t)
Lisp expression: 42
42
> (read t)
Lisp expression: ^Derror: Error reading from stdin
> Error reading from stdin
$

Notice that it’s currently impossible for non-interactive scripts to
recover from EOF when reading from tty. After the user sends an EOF
character by pressing C-d, calling read-from-minibuffer will always result
in an error.

This patch solves this by clearing stdin of errors and returning nil when
an EOF character is detected.

Here is the patch:

From 685452ca4a098bbd7c1062a35064755c1d09512a Mon Sep 17 00:00:00 2001
From: Shawn Presser <shawnpresser@gmail.com>
Date: Fri, 18 Jan 2019 03:57:37 -0600
Subject: [PATCH] Fix reading EOF characters in batch mode.

* src/minibuf.c (read_minibuf_noninteractive): Handle EOF characters by
clearing stdin and returning Qnil.
---
 src/minibuf.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/src/minibuf.c b/src/minibuf.c
index 321fda1ba8..8c88a316a7 100644
--- a/src/minibuf.c
+++ b/src/minibuf.c
@@ -250,7 +250,8 @@ read_minibuf_noninteractive (Lisp_Object prompt, bool
expflag,
   else
     {
       xfree (line);
-      error ("Error reading from stdin");
+      clearerr(stdin);
+      return Qnil;
     }

   /* If Lisp form desired instead of string, parse it.  */
-- 
2.19.1

Thanks!
— shawn

[-- Attachment #1.2: Type: text/html, Size: 5814 bytes --]

[-- Attachment #2: 0001-Fix-reading-EOF-characters-in-batch-mode.patch --]
[-- Type: application/octet-stream, Size: 803 bytes --]

From 685452ca4a098bbd7c1062a35064755c1d09512a Mon Sep 17 00:00:00 2001
From: Shawn Presser <shawnpresser@gmail.com>
Date: Fri, 18 Jan 2019 03:57:37 -0600
Subject: [PATCH] Fix reading EOF characters in batch mode.

* src/minibuf.c (read_minibuf_noninteractive): Handle EOF characters by
clearing stdin and returning Qnil.
---
 src/minibuf.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/src/minibuf.c b/src/minibuf.c
index 321fda1ba8..8c88a316a7 100644
--- a/src/minibuf.c
+++ b/src/minibuf.c
@@ -250,7 +250,8 @@ read_minibuf_noninteractive (Lisp_Object prompt, bool expflag,
   else
     {
       xfree (line);
-      error ("Error reading from stdin");
+      clearerr(stdin);
+      return Qnil;
     }
 
   /* If Lisp form desired instead of string, parse it.  */
-- 
2.19.1

bug#34123: A patch to fix reading EOF characters in non-interactive mode

[Eli Zaretskii]

> From: Shawn Presser <shawnpresser@gmail.com>
> Date: Fri, 18 Jan 2019 04:35:34 -0600
> 
> $ git clone https://github.com/shawwn/y
> $ cd y
> $ bin/y
> > (read t)
> Lisp expression: 42
> 42
> > (read t)
> Lisp expression: ^Derror: Error reading from stdin
> > Error reading from stdin
> $ 
> 
> Notice that it’s currently impossible for non-interactive scripts to recover from EOF when reading from tty.

What would you want such non-interactive scripts to do after recovery?

bug#34123: A patch to fix reading EOF characters in non-interactive mode

[Shawn Presser]

[-- Attachment #1: Type: text/plain, Size: 2995 bytes --]

I use Emacs Lisp as a runtime for Arc (https://github.com/shawwn/arcmacs) and
for Lumen (https://github.com/shawwn/y). The goal is to be able to run a
Lisp REPL from the terminal.

For example, in Arc:

  $ git clone https://github.com/laarc/laarc
  $ cd laarc
  $ make
  $ bin/arc
  Use (quit) to quit, (tl) to return here after an interrupt.
  arc> 1
  1
  arc> (def my-repl ()
         (pr "> ")
         (whilet expr (read)
           (write (eval expr))
           (prn) (pr "> ")))
  *** redefining repl
  #<procedure: repl>
  arc> (my-repl)
  > 1 ; we can exit this inner REPL with control-D, without exiting the
program.
  1
  > (+ 1 2)
  3
  > ^D
  arc> "If this were emacs in batch mode, the program would have exited
with errors here."
  "If this were emacs in batch mode, the program would have exited with
errors here."
  arc> "Instead, control-D returned us to the toplevel REPL. Let's type
another control-D to quit."
  "Instead, control-D returned us to the toplevel REPL. Let's type another
control-D to quit."
  arc> ^D
  $

Please compare this with arcmacs, which runs in Emacs Lisp:

  $ git clone https://github.com/shawwn/arcmacs
  $ cd arcmacs
  $ ./y-arc
  Use (quit) to quit, (tl) to return here after an interrupt.
  arc> 1
  1
  arc> (def my-repl ()
         (pr "> ")
         (whilet expr (read)
           (write (eval expr))
           (prn) (pr "> ")))
  arc> (my-repl)
  > 1  ; if we enter control-D, every REPL exits immediately and the
program quits
  1
  > 2
  2
  > ^D
  arc>
  $ echo We didn't intend to quit yet!

A single ^D exits the entire program, not just the inner REPL. This is
because after the user enters control-D, any attempt to call
`read-from-minibuffer` will always result in emacs throwing an error
with the message "Error reading from stdin".

The only way to gracefully handle this case in a script is to trap the
internal emacs error with (ignore-errors (read-from-minibuffer ""))
and to treat a nil result as if it were EOF. But that causes every
nested REPL to exit, since every attempt to (read-from-minibuffer "")
will throw "Error reading from stdin". In other words, emacs is
always failing to read from stdin after the user sends an EOF.

This patch fixes the problem by causing (read-from-minibuffer "") to
return nil if the user types control-D.


On Fri, Jan 18, 2019 at 7:25 AM Eli Zaretskii <eliz@gnu.org> wrote:

> > From: Shawn Presser <shawnpresser@gmail.com>
> > Date: Fri, 18 Jan 2019 04:35:34 -0600
> >
> > $ git clone https://github.com/shawwn/y
> > $ cd y
> > $ bin/y
> > > (read t)
> > Lisp expression: 42
> > 42
> > > (read t)
> > Lisp expression: ^Derror: Error reading from stdin
> > > Error reading from stdin
> > $
> >
> > Notice that it’s currently impossible for non-interactive scripts to
> recover from EOF when reading from tty.
>
> What would you want such non-interactive scripts to do after recovery?
>

[-- Attachment #2: Type: text/html, Size: 6769 bytes --]

bug#34123: A patch to fix reading EOF characters in non-interactive mode

[Eli Zaretskii]

> From: Shawn Presser <shawnpresser@gmail.com>
> Date: Fri, 18 Jan 2019 08:22:31 -0600
> Cc: 34123@debbugs.gnu.org
> 
> I use Emacs Lisp as a runtime for Arc (https://github.com/shawwn/arcmacs) and for Lumen
> (https://github.com/shawwn/y). The goal is to be able to run a Lisp REPL from the terminal.

So the usage you describe, and are interested in, is not
non-interactive, as far as Emacs is concerned, right?  Or am I missing
something?

bug#34123: A patch to fix reading EOF characters in non-interactive mode

[Shawn Presser]

[-- Attachment #1: Type: text/plain, Size: 1089 bytes --]

It's non-interactive:

arc> |noninteractive|
t

(The expression |noninteractive| in arcmacs is equivalent to evaluating
'noninteractive in emacs lisp.)

The runtimes start by invoking emacs using `emacs -Q --script ...`, so it's
always noninteractive. And in general for writing shell scripts rather than
repls, it's important to be in noninteractive mode. But it's equally
important to have the ability to prompt the user from a shell script, and
for the user to be able to cancel by using ^D without causing stdin errors
on all subsequent prompts.



On Fri, Jan 18, 2019 at 10:04 AM Eli Zaretskii <eliz@gnu.org> wrote:

> > From: Shawn Presser <shawnpresser@gmail.com>
> > Date: Fri, 18 Jan 2019 08:22:31 -0600
> > Cc: 34123@debbugs.gnu.org
> >
> > I use Emacs Lisp as a runtime for Arc (https://github.com/shawwn/arcmacs)
> and for Lumen
> > (https://github.com/shawwn/y). The goal is to be able to run a Lisp
> REPL from the terminal.
>
> So the usage you describe, and are interested in, is not
> non-interactive, as far as Emacs is concerned, right?  Or am I missing
> something?
>

[-- Attachment #2: Type: text/html, Size: 1852 bytes --]

bug#34123: A patch to fix reading EOF characters in non-interactive mode

[Eli Zaretskii]

> From: Shawn Presser <shawnpresser@gmail.com>
> Date: Fri, 18 Jan 2019 10:20:54 -0600
> Cc: 34123@debbugs.gnu.org
> 
> It's non-interactive:
> 
> arc> |noninteractive|
> t
> 
> (The expression |noninteractive| in arcmacs is equivalent to evaluating 'noninteractive in emacs lisp.)
> 
> The runtimes start by invoking emacs using `emacs -Q --script ...`, so it's always noninteractive. And in
> general for writing shell scripts rather than repls, it's important to be in noninteractive mode. But it's equally
> important to have the ability to prompt the user from a shell script, and for the user to be able to cancel by
> using ^D without causing stdin errors on all subsequent prompts.

I think we are miscommunicating, since your usage involves Emacs
reading from the terminal.  That is not noninteractive in my book.
What I mean by noninteractive is when stdin is not connected to a
terminal.  Your proposed patch affects that case as well, doesn't it?

bug#34123: A patch to fix reading EOF characters in non-interactive mode

[Shawn Presser]

[-- Attachment #1: Type: text/plain, Size: 2925 bytes --]

> What I mean by noninteractive is when stdin is not connected to a
terminal.  Your proposed patch affects that case as well, doesn't it?

I don't think so, because the only case that this patch affects is the case
where emacs would signal "error reading from stdin". I don't think I've
ever seen emacs throw that error during normal emacs usage, even when
reading from stdin not connected to a terminal.

But admittedly, I am not certain in all cases. It's probably better to
phrase this as a question: Has any elisp code been written to handle "error
reading from stdin"? For example, (ignore-error ...) would mask the error
message, if it could somehow be generated during normal emacs operation.

However, even in that case, the operation of emacs would end up exactly the
same, since `nil` is now returned, which is exactly what the `(ignore-error
...)` expression would return (if such code already exists).

That said, it would be surprising if this were expected behavior, because
the only way for user code to detect and handle it would be to parse the
error message, since it's a regular 'error type rather than a type that can
be handled via condition-case. I don't think this situation has been
covered till now, because emacs users typically don't care about reading
from stdin in batch mode.

Actually, I just found a blog post from 2014 that complains about this very
case:
http://joelmccracken.github.io/entries/reading-writing-data-in-emacs-batch-via-stdin-stdout/

> When EOF is read, an error is thrown, which is useless to us. Instead of
worrying about the error, we can wrap the function call in an ignore-errors
macro, which will return nil when an EOF occurs.

They don't seem to care about being able to call read-from-minibuffer
multiple times. But if they did care, they would find it quite impossible
after the first EOF.


On Fri, Jan 18, 2019 at 10:29 AM Eli Zaretskii <eliz@gnu.org> wrote:

> > From: Shawn Presser <shawnpresser@gmail.com>
> > Date: Fri, 18 Jan 2019 10:20:54 -0600
> > Cc: 34123@debbugs.gnu.org
> >
> > It's non-interactive:
> >
> > arc> |noninteractive|
> > t
> >
> > (The expression |noninteractive| in arcmacs is equivalent to evaluating
> 'noninteractive in emacs lisp.)
> >
> > The runtimes start by invoking emacs using `emacs -Q --script ...`, so
> it's always noninteractive. And in
> > general for writing shell scripts rather than repls, it's important to
> be in noninteractive mode. But it's equally
> > important to have the ability to prompt the user from a shell script,
> and for the user to be able to cancel by
> > using ^D without causing stdin errors on all subsequent prompts.
>
> I think we are miscommunicating, since your usage involves Emacs
> reading from the terminal.  That is not noninteractive in my book.
> What I mean by noninteractive is when stdin is not connected to a
> terminal.  Your proposed patch affects that case as well, doesn't it?
>

[-- Attachment #2: Type: text/html, Size: 3843 bytes --]

bug#34123: A patch to fix reading EOF characters in non-interactive mode

[Shawn Presser]

[-- Attachment #1: Type: text/plain, Size: 1380 bytes --]

Is there another way I can address the concern here? I'm willing to put in
the work to get this in.

To put it more simply, it seems like an error that emacs doesn't call
clearerr(stdin) after reading EOF. Without that, it's impossible to ever
read from stdin after the first EOF. Should that be changed?

On Fri, Jan 18, 2019 at 10:29 AM Eli Zaretskii <eliz@gnu.org> wrote:

> > From: Shawn Presser <shawnpresser@gmail.com>
> > Date: Fri, 18 Jan 2019 10:20:54 -0600
> > Cc: 34123@debbugs.gnu.org
> >
> > It's non-interactive:
> >
> > arc> |noninteractive|
> > t
> >
> > (The expression |noninteractive| in arcmacs is equivalent to evaluating
> 'noninteractive in emacs lisp.)
> >
> > The runtimes start by invoking emacs using `emacs -Q --script ...`, so
> it's always noninteractive. And in
> > general for writing shell scripts rather than repls, it's important to
> be in noninteractive mode. But it's equally
> > important to have the ability to prompt the user from a shell script,
> and for the user to be able to cancel by
> > using ^D without causing stdin errors on all subsequent prompts.
>
> I think we are miscommunicating, since your usage involves Emacs
> reading from the terminal.  That is not noninteractive in my book.
> What I mean by noninteractive is when stdin is not connected to a
> terminal.  Your proposed patch affects that case as well, doesn't it?
>

[-- Attachment #2: Type: text/html, Size: 1942 bytes --]

bug#34123: A patch to fix reading EOF characters in non-interactive mode

[Shawn Presser]

[-- Attachment #1: Type: text/plain, Size: 1427 bytes --]

Hi, just checking in. I wanted to thank you for taking the time to reply
and ask about this bug. It's my first emacs patch and I'm not entirely sure
if I'm following the process correctly here. I'd like to try answering your
questions more succinctly.

Is there a "next step" at this point, or something that I can do to help
with emacs in general?




On Fri, Jan 18, 2019 at 10:29 AM Eli Zaretskii <eliz@gnu.org> wrote:

> > From: Shawn Presser <shawnpresser@gmail.com>
> > Date: Fri, 18 Jan 2019 10:20:54 -0600
> > Cc: 34123@debbugs.gnu.org
> >
> > It's non-interactive:
> >
> > arc> |noninteractive|
> > t
> >
> > (The expression |noninteractive| in arcmacs is equivalent to evaluating
> 'noninteractive in emacs lisp.)
> >
> > The runtimes start by invoking emacs using `emacs -Q --script ...`, so
> it's always noninteractive. And in
> > general for writing shell scripts rather than repls, it's important to
> be in noninteractive mode. But it's equally
> > important to have the ability to prompt the user from a shell script,
> and for the user to be able to cancel by
> > using ^D without causing stdin errors on all subsequent prompts.
>
> I think we are miscommunicating, since your usage involves Emacs
> reading from the terminal.  That is not noninteractive in my book.
> What I mean by noninteractive is when stdin is not connected to a
> terminal.  Your proposed patch affects that case as well, doesn't it?
>

[-- Attachment #2: Type: text/html, Size: 2111 bytes --]

bug#34123: A patch to fix reading EOF characters in non-interactive mode

[Eli Zaretskii]

> From: Shawn Presser <shawnpresser@gmail.com>
> Date: Mon, 21 Jan 2019 02:59:36 -0600
> Cc: 34123@debbugs.gnu.org
> 
> Hi, just checking in. I wanted to thank you for taking the time to reply and ask about this bug. It's my first
> emacs patch and I'm not entirely sure if I'm following the process correctly here. I'd like to try answering your
> questions more succinctly.
> 
> Is there a "next step" at this point, or something that I can do to help with emacs in general?

I'm still considering the implications of your suggestion on the other
use cases.  One possible alternative would be to add a function that
will clear the error condition, and let users/Lisp programs call it.
This way, the default response to EOF will remain intact.

Perhaps someone else will chime in with opinions and ideas.

Thanks.

bug#34123: A patch to fix reading EOF characters in non-interactive mode

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> I'm still considering the implications of your suggestion on the other
> use cases.  One possible alternative would be to add a function that
> will clear the error condition, and let users/Lisp programs call it.
> This way, the default response to EOF will remain intact.
>
> Perhaps someone else will chime in with opinions and ideas.

And then Noam answered a few months later:

Noam Postavsky <npostavs@gmail.com> writes:

> Shawn Presser <shawnpresser@gmail.com> writes:
>
>> Notice that it’s currently impossible for non-interactive scripts to
>> recover from EOF when reading from tty. After the user sends an EOF
>> character by pressing C-d, calling read-from-minibuffer will always result
>> in an error.
>
> This isn't quite true, see the attached script which catches the error
> and only exits after two EOFs.

Shawn, does Noam's solution work for you?

Noam also had this suggestion:

>>        xfree (line);
>> -      error ("Error reading from stdin");
>> +      clearerr(stdin);
>> +      return Qnil;
>
> I think it would make sense to change that error to be an end-of-file
> error, to make it easier to catch (having to examine the error message,
> as I did in my script is not so nice).

And I think that's true?  It happens here in read_minibuf_noninteractive:

  if (len || c == '\n' || c == '\r')
    {
      val = make_string (line, len);
      xfree (line);
    }
  else
    {
      xfree (line);
      error ("Error reading from stdin");
    }

So I think that error can only happen on an end-of-file?  Anybody
objecting to changing that to and end-of-file error?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#34123: A patch to fix reading EOF characters in non-interactive mode

[Lars Ingebrigtsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> And I think that's true?  It happens here in read_minibuf_noninteractive:
>
>   if (len || c == '\n' || c == '\r')
>     {
>       val = make_string (line, len);
>       xfree (line);
>     }
>   else
>     {
>       xfree (line);
>       error ("Error reading from stdin");
>     }
>
> So I think that error can only happen on an end-of-file?  Anybody
> objecting to changing that to and end-of-file error?

There were no objections, so I've now done this in Emacs 28.  (It didn't
seem NEWS-worthy, so I didn't add a NEWS item.)

Witch that change, I think Noam's approach to handling EOF characters in
non-interactive Emacsen should work fine, so I'm closing this bug
report.  If more work needs to be done here, please respond and we'll
reopen the bug report.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#34123: A patch to fix reading EOF characters in non-interactive mode

[Noam Postavsky]

[-- Attachment #1: Type: text/plain, Size: 397 bytes --]

Shawn Presser <shawnpresser@gmail.com> writes:

> Notice that it’s currently impossible for non-interactive scripts to
> recover from EOF when reading from tty. After the user sends an EOF
> character by pressing C-d, calling read-from-minibuffer will always result
> in an error.

This isn't quite true, see the attached script which catches the error
and only exits after two EOFs.


[-- Attachment #2: script demoing eof handling --]
[-- Type: text/plain, Size: 389 bytes --]

(require 'cl-lib)
(cl-loop with eofs = 0
         for i from 0
         while (< eofs 2)
         do (condition-case err
                (princ (format "line%d: %S\n" i (read-from-minibuffer "> ")))
              (error
               (if (equal err '(error "Error reading from stdin"))
                   (message "EOF%d" (cl-incf eofs))
                 (signal (car err) (cdr err))))))

[-- Attachment #3: Type: text/plain, Size: 457 bytes --]


A sample usage:

    $ emacs -Q --script ../bug-34123-read-script.el 
    > one
    line0: "one"
    > 
    EOF1
    > two
    line2: "two"
    > 
    EOF2

>        xfree (line);
> -      error ("Error reading from stdin");
> +      clearerr(stdin);
> +      return Qnil;

I think it would make sense to change that error to be an end-of-file
error, to make it easier to catch (having to examine the error message,
as I did in my script is not so nice).