bug#48940: 27.2; regression: "emacs --script /dev/stdin" parses the script incorrectly when /dev/stdin is a pipe

bug#48940: 27.2; regression: "emacs --script /dev/stdin" parses the script incorrectly when /dev/stdin is a pipe

[Bryan C. Mills via Bug reports for GNU Emacs, the Swiss army knife of text editors]

`emacs --script /dev/stdin` worked reliably for me on Emacs 24 through 26.

As of Emacs 27, it no longer works reliably — depending on the script contents,
it either fails with spurious errors or silently exits without
finishing the script.


In the following bash interaction, all three emacs invocations
*should* run the same script.
The third invocation demonstrates the bug: when /dev/stdin is a pipe from `cat`
(instead of the script.el file directly), the script is no longer interpreted.
With other scripts, I get various errors, seemingly due to `;` or `"`
characters being
dropped or ignored from the input.

```
$ cat script.el | cat /dev/stdin
(print "Hello emacs!")

$ emacs --no-init-file --no-site-file --script script.el

"Hello emacs!"

$ emacs --no-init-file --no-site-file --script /dev/stdin <script.el

"Hello emacs!"

$ cat script.el | emacs --no-init-file --no-site-file --script /dev/stdin

$ echo $?
0

$
```

This reproduces with the GNU Emacs 27.1 currently packaged on Debian Testing,
as well as the Google build of GNU Emacs 27.2 from which I produced
the report below.

(My apologies if this is a duplicate report. I tried to send it
earlier using M-x report-emacs-bug,
but the resulting email appears to have gotten stuck or rejected somewhere.)


In GNU Emacs 27.2 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.24.24,
cairo version 1.16.0)
 of 2021-03-29, modified by Debian built on kokoro-ubuntu
System Description: Debian GNU/Linux rodete

Recent messages:
For information about GNU Emacs and the GNU system, type C-h C-a.
<f6> is undefined [2 times]
Making completion list... [2 times]

Configured using:
 'configure --build x86_64-linux-gnu --build x86_64-linux-gnu
 --prefix=/usr --sharedstatedir=/var/lib --libexecdir=/usr/lib
 --localstatedir=/var/lib --infodir=/usr/share/info
 --mandir=/usr/share/man --enable-libsystemd --with-pop=yes
 --enable-locallisppath=/etc/google-emacs:/usr/local/share/google-emacs/27.2+gg3+1.20210329.053400.rc147/site-lisp:/usr/local/share/google-emacs/site-lisp:/usr/share/google-emacs/27.2+gg3+1.20210329.053400.rc147/site-lisp:/usr/share/google-emacs/site-lisp
 --with-sound=alsa --without-gconf --with-mailutils
 --program-prefix=google- --disable-silent-rules
 GOOGLE_VERSION=27.2+gg3+1.20210329.053400.rc147 --with-cairo
 --with-x=yes --with-x-toolkit=gtk3 --with-toolkit-scroll-bars
 build_alias=x86_64-linux-gnu 'CFLAGS=-g -O2
 -ffile-prefix-map=/build/google-emacs-avRD2q/google-emacs-27.2+gg3+1.20210329.053400.rc147=.
-fstack-protector-strong
 -Wformat -Werror=format-security -Wall' LDFLAGS=-Wl,-z,relro
 'CPPFLAGS=-Wdate-time -D_FORTIFY_SOURCE=2' 'OBJCFLAGS=-g -O2
 -ffile-prefix-map=/build/google-emacs-avRD2q/google-emacs-27.2+gg3+1.20210329.053400.rc147=.
-fstack-protector-strong
 -Wformat -Werror=format-security''

Configured features:
XPM JPEG TIFF GIF PNG RSVG CAIRO SOUND GPM DBUS GSETTINGS GLIB NOTIFY
INOTIFY ACL LIBSELINUX GNUTLS LIBXML2 FREETYPE HARFBUZZ M17N_FLT LIBOTF
ZLIB TOOLKIT_SCROLL_BARS GTK3 X11 XDBE XIM MODULES THREADS LIBSYSTEMD
JSON PDUMPER LCMS2 GMP

Important settings:
  value of $LC_ALL: en_US.UTF-8
  value of $LANG: en_US.UTF-8
  locale-coding-system: utf-8-unix

Major mode: Text

Minor modes in effect:
  tooltip-mode: t
  global-eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  transient-mark-mode: t

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message rmc puny dired dired-loaddefs
format-spec rfc822 mml mml-sec password-cache epa derived epg epg-config
gnus-util rmail rmail-loaddefs text-property-search time-date subr-x seq
mm-decode mm-bodies mm-encode mail-parse rfc2231 mailabbrev gmm-utils
mailheader sendmail rfc2047 rfc2045 ietf-drums mm-util mail-prsvr
mail-utils vc-git diff-mode easymenu easy-mmode cl-loaddefs cl-lib
term/tmux term/xterm xterm byte-opt gv bytecomp byte-compile cconv
tooltip eldoc electric uniquify ediff-hook vc-hooks lisp-float-type
mwheel term/x-win x-win term/common-win x-dnd tool-bar dnd fontset image
regexp-opt fringe tabulated-list replace newcomment text-mode elisp-mode
lisp-mode prog-mode register page tab-bar menu-bar rfn-eshadow isearch
timer select scroll-bar mouse jit-lock font-lock syntax facemenu
font-core term/tty-colors frame minibuffer cl-generic cham georgian
utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao korean
japanese eucjp-ms cp51932 hebrew greek romanian slovak czech european
ethiopic indian cyrillic chinese composite charscript charprop
case-table epa-hook jka-cmpr-hook help simple abbrev obarray
cl-preloaded nadvice loaddefs button faces cus-face macroexp files
text-properties overlay sha1 md5 base64 format env code-pages mule
custom widget hashtable-print-readable backquote threads dbusbind
inotify lcms2 dynamic-setting system-font-setting font-render-setting
cairo move-toolbar gtk x-toolkit x multi-tty make-network-process emacs)

Memory information:
((conses 16 53224 7924)
 (symbols 48 6747 1)
 (strings 32 17548 1920)
 (string-bytes 1 565159)
 (vectors 16 8089)
 (vector-slots 8 89351 7266)
 (floats 8 29 624)
 (intervals 56 241 3)
 (buffers 1000 13))

bug#48940: 27.2; regression: "emacs --script /dev/stdin" parses the script incorrectly when /dev/stdin is a pipe

[Eli Zaretskii]

> Date: Wed, 9 Jun 2021 16:31:36 -0400
> From:  "Bryan C. Mills" via "Bug reports for GNU Emacs,
>  the Swiss army knife of text editors" <bug-gnu-emacs@gnu.org>
> 
> `emacs --script /dev/stdin` worked reliably for me on Emacs 24 through 26.
> 
> As of Emacs 27, it no longer works reliably — depending on the script contents,
> it either fails with spurious errors or silently exits without
> finishing the script.
> 
> 
> In the following bash interaction, all three emacs invocations
> *should* run the same script.
> The third invocation demonstrates the bug: when /dev/stdin is a pipe from `cat`
> (instead of the script.el file directly), the script is no longer interpreted.
> With other scripts, I get various errors, seemingly due to `;` or `"`
> characters being
> dropped or ignored from the input.
> 
> ```
> $ cat script.el | cat /dev/stdin
> (print "Hello emacs!")
> 
> $ emacs --no-init-file --no-site-file --script script.el
> 
> "Hello emacs!"
> 
> $ emacs --no-init-file --no-site-file --script /dev/stdin <script.el
> 
> "Hello emacs!"
> 
> $ cat script.el | emacs --no-init-file --no-site-file --script /dev/stdin
> 
> $ echo $?
> 0
> 
> $
> ```
> 
> This reproduces with the GNU Emacs 27.1 currently packaged on Debian Testing,
> as well as the Google build of GNU Emacs 27.2 from which I produced
> the report below.

I see the problem, but it happens for me in Emacs 26 and Emacs 25 as
well, so I'm not sure this is new, or why it works for you in older
versions.  Maybe Debian included some local patches in those older
versions?  (My old Emacs versions were built from the unmodified
upstream sources.)

AFAICS, the problem seems to be that load-with-code-conversion calls
insert-file-contents, and the latter comes up with an empty buffer in
the problematic case.

It is notoriously hard to debug a program whose standard input was
redirected from a pipe of another program, so I couldn't see why the
above happens.  If someone could step in a debugger through
insert-file-contents in this case and describe what's going on there,
or tell how to do that when Emacs is invoked like this, maybe we
could make some progress.

Thanks.

bug#48940: 27.2; regression: "emacs --script /dev/stdin" parses the script incorrectly when /dev/stdin is a pipe

[Bryan C. Mills via Bug reports for GNU Emacs, the Swiss army knife of text editors]

I did a little digging and found that I can reproduce the issue under
a debugger by constructing a named pipe with `mkfifo pipe.el` and
feeding the script into it using `cat script.el >>pipe.el`. (Then I
can execute `run --script pipe.el` under gdb to capture a backtrace at
the problematic lseek call.)

I identified one bug in the safe_to_load_version function.
I think I've got a fix for it, but I'm not very familiar with the
emacs codebase, especially when it comes to regression testing.
(My patch against the emacs-27 branch is below.)

I tested the fix locally with a small dummy script as input and it
seems to work, but I can't be confident that there won't be deeper
bugs. You're welcome to use it as a starting point for a more robust
fix.


From c97e4b97af86023177417e49c46b19812b2f4caa Mon Sep 17 00:00:00 2001
From: "Bryan C. Mills" <bcmills@google.com>
Date: Mon, 14 Jun 2021 17:30:11 -0400
Subject: [PATCH] Fix invocation with '--script /dev/stdin'

* src/lread.c (safe_to_load_version): Check lseek errors.  (Bug#48940)
---
 src/lread.c | 19 +++++++++++++++----
 1 file changed, 15 insertions(+), 4 deletions(-)

diff --git a/src/lread.c b/src/lread.c
index 47116ad5ae..96b6695b5c 100644
--- a/src/lread.c
+++ b/src/lread.c
@@ -990,12 +990,21 @@ #define UPDATE_BEG_END_STATE(ch) \
    because of an incompatible change in the byte compiler.  */

 static int
-safe_to_load_version (int fd)
+safe_to_load_version (Lisp_Object file, int fd)
 {
+  struct stat st;
   char buf[512];
   int nbytes, i;
   int version = 1;

+  /* If the file is not regular, then we cannot safely seek it.
+     Assume that it is not safe to load as a compiled file.  */
+  if (fstat(fd, &st) == 0)
+    {
+      if (!S_ISREG (st.st_mode))
+        return 0;
+    }
+
   /* Read the first few bytes from the file, and look for a line
      specifying the byte compiler version used.  */
   nbytes = emacs_read_quit (fd, buf, sizeof buf);
@@ -1013,7 +1022,9 @@ safe_to_load_version (int fd)
  version = 0;
     }

-  lseek (fd, 0, SEEK_SET);
+  if (lseek (fd, 0, SEEK_SET) < 0)
+    report_file_error ("Seeking to start of file", file);
+
   return version;
 }

@@ -1316,7 +1327,7 @@ DEFUN ("load", Fload, Sload, 1, 5, 0,
   if (is_elc
       /* version = 1 means the file is empty, in which case we can
  treat it as not byte-compiled.  */
-      || (fd >= 0 && (version = safe_to_load_version (fd)) > 1))
+      || (fd >= 0 && (version = safe_to_load_version (file, fd)) > 1))
     /* Load .elc files directly, but not when they are
        remote and have no handler!  */
     {
@@ -1326,7 +1337,7 @@ DEFUN ("load", Fload, Sload, 1, 5, 0,
    int result;

    if (version < 0
-       && ! (version = safe_to_load_version (fd)))
+       && ! (version = safe_to_load_version (file, fd)))
      {
        safe_p = 0;
        if (!load_dangerous_libraries)
-- 
2.32.0.272.g935e593368-goog

On Sun, Jun 13, 2021 at 6:26 AM Eli Zaretskii <eliz@gnu.org> wrote:
>
> > Date: Wed, 9 Jun 2021 16:31:36 -0400
> > From:  "Bryan C. Mills" via "Bug reports for GNU Emacs,
> >  the Swiss army knife of text editors" <bug-gnu-emacs@gnu.org>
> >
> > `emacs --script /dev/stdin` worked reliably for me on Emacs 24 through 26.
> >
> > As of Emacs 27, it no longer works reliably — depending on the script contents,
> > it either fails with spurious errors or silently exits without
> > finishing the script.
> >
> >
> > In the following bash interaction, all three emacs invocations
> > *should* run the same script.
> > The third invocation demonstrates the bug: when /dev/stdin is a pipe from `cat`
> > (instead of the script.el file directly), the script is no longer interpreted.
> > With other scripts, I get various errors, seemingly due to `;` or `"`
> > characters being
> > dropped or ignored from the input.
> >
> > ```
> > $ cat script.el | cat /dev/stdin
> > (print "Hello emacs!")
> >
> > $ emacs --no-init-file --no-site-file --script script.el
> >
> > "Hello emacs!"
> >
> > $ emacs --no-init-file --no-site-file --script /dev/stdin <script.el
> >
> > "Hello emacs!"
> >
> > $ cat script.el | emacs --no-init-file --no-site-file --script /dev/stdin
> >
> > $ echo $?
> > 0
> >
> > $
> > ```
> >
> > This reproduces with the GNU Emacs 27.1 currently packaged on Debian Testing,
> > as well as the Google build of GNU Emacs 27.2 from which I produced
> > the report below.
>
> I see the problem, but it happens for me in Emacs 26 and Emacs 25 as
> well, so I'm not sure this is new, or why it works for you in older
> versions.  Maybe Debian included some local patches in those older
> versions?  (My old Emacs versions were built from the unmodified
> upstream sources.)
>
> AFAICS, the problem seems to be that load-with-code-conversion calls
> insert-file-contents, and the latter comes up with an empty buffer in
> the problematic case.
>
> It is notoriously hard to debug a program whose standard input was
> redirected from a pipe of another program, so I couldn't see why the
> above happens.  If someone could step in a debugger through
> insert-file-contents in this case and describe what's going on there,
> or tell how to do that when Emacs is invoked like this, maybe we
> could make some progress.
>
> Thanks.

bug#48940: 27.2; regression: "emacs --script /dev/stdin" parses the script incorrectly when /dev/stdin is a pipe

[Lars Ingebrigtsen]

"Bryan C. Mills" <bcmills@google.com> writes:

> I tested the fix locally with a small dummy script as input and it
> seems to work, but I can't be confident that there won't be deeper
> bugs. You're welcome to use it as a starting point for a more robust
> fix.

I tried (on Debian/bullseye with Emacs 29), but it doesn't seem to make
any difference here.  With or without the patch, I get:

larsi@xo:~/src/emacs/trunk$ emake; cat /tmp/script.el | ./src/emacs -Q --script /dev/stdin
Debugger entered--Lisp error: (file-missing "Cannot open load file" "No such file or directory" "/proc/162231/fd/pipe:[1065564]")
  load("/proc/162231/fd/pipe:[1065564]" nil t t)
  command-line-1(("-scriptload" "/dev/stdin"))
  command-line()
  normal-top-level()

Perhaps something further has changed in the meantime?  (You didn't post
an output of the error message you were getting...)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#48940: 27.2; regression: "emacs --script /dev/stdin" parses the script incorrectly when /dev/stdin is a pipe

[Bryan C. Mills via Bug reports for GNU Emacs, the Swiss army knife of text editors]

On Thu, Nov 11, 2021 at 12:09 AM Lars Ingebrigtsen <larsi@gnus.org> wrote:
>
> "Bryan C. Mills" <bcmills@google.com> writes:
>
> > I tested the fix locally with a small dummy script as input and it
> > seems to work, but I can't be confident that there won't be deeper
> > bugs. You're welcome to use it as a starting point for a more robust
> > fix.
>
> I tried (on Debian/bullseye with Emacs 29), but it doesn't seem to make
> any difference here.  With or without the patch, I get:
>
> larsi@xo:~/src/emacs/trunk$ emake; cat /tmp/script.el | ./src/emacs -Q --script /dev/stdin
> Debugger entered--Lisp error: (file-missing "Cannot open load file" "No such file or directory" "/proc/162231/fd/pipe:[1065564]")
>   load("/proc/162231/fd/pipe:[1065564]" nil t t)
>   command-line-1(("-scriptload" "/dev/stdin"))
>   command-line()
>   normal-top-level()
>
> Perhaps something further has changed in the meantime?  (You didn't post
> an output of the error message you were getting...)

I did post the complete output: the pipeline `cat script.el | emacs
--no-init-file --no-site-file --script /dev/stdin` did not produce any
output whatsoever for my reproducer. (Nothing to stdout, nothing to
stderr.)

Since the return-value from lseek was ignored prior to my patch, the
failing seek did not necessarily lead to a diagnosed error at all — it
could instead result in a spurious EOF on the next read, or a parse
error from unintentionally continuing to read the file at the
unmodified (pre-lseek) offset.

I don't know what else may have regressed between when I mailed the
patch and when you looked at it in Emacs 29, and unfortunately I don't
have the time to look into it at the moment.

bug#48940: 27.2; regression: "emacs --script /dev/stdin" parses the script incorrectly when /dev/stdin is a pipe

[Lars Ingebrigtsen]

"Bryan C. Mills" <bcmills@google.com> writes:

> I did post the complete output: the pipeline `cat script.el | emacs
> --no-init-file --no-site-file --script /dev/stdin` did not produce any
> output whatsoever for my reproducer. (Nothing to stdout, nothing to
> stderr.)

Ah, I see.

I tracked down what's causing this new problem and fixed that, and after
doing that, I can confirm that your patch fixes the problem with
/dev/stdin-as-a-pipe, so I've pushed it to Emacs 29.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no