bug#57854: 29.0.50; Different exit code in Emacs and terminal for identical process

bug#57854: 29.0.50; Different exit code in Emacs and terminal for identical process

[dalanicolai]

[-- Attachment #1: Type: text/plain, Size: 3866 bytes --]

Execute in the terminal the command pdftocio (see
https://github.com/Krasjet/pdf.tocgen#pdftocio, the command is
 easily available via PyPI <https://pypi.org/project/pdf.tocgen/>) on a pdf
file that does
not contain a Table of Contents. The command produces an error and
subsequently executing `echo $?` shows exit code 1.

Subsequently, executing the same command from Emacs on the same file,
then `call-process` returns a 0, and indeed `shell-command-to-string`
produces an empty string. However, I would expect an error like in the
terminal (with call-process returning exit code 1).

The PDF in the following link could be used for testing:
https://www.orimi.com/pdf-test.pdf


In GNU Emacs 29.0.50 (build 1, x86_64-pc-linux-gnu, GTK+ Version
 3.24.34, cairo version 1.17.6) of 2022-09-10 built on fedora
Repository revision: 4cf9c92e27d20da9453f9abe89d84bee5d776329
Repository branch: master
Windowing system distributor 'The X.Org Foundation', version 11.0.12014000
System Description: Fedora Linux 36 (Workstation Edition)

Configured using:
 'configure --with-modules --with-cairo --with-native-compilation
 --with-json'

Configured features:
ACL CAIRO DBUS FREETYPE GIF GLIB GMP GNUTLS GSETTINGS HARFBUZZ JPEG JSON
LIBSELINUX LIBXML2 MODULES NATIVE_COMP NOTIFY INOTIFY PDUMPER PNG RSVG
SECCOMP SOUND SQLITE3 THREADS TIFF TOOLKIT_SCROLL_BARS X11 XDBE XIM
XINPUT2 XPM GTK3 ZLIB

Important settings:
  value of $LANG: en_US.UTF-8
  value of $XMODIFIERS: @im=ibus
  locale-coding-system: utf-8-unix

Major mode: DocView

Minor modes in effect:
  tooltip-mode: t
  global-eldoc-mode: t
  show-paren-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  buffer-read-only: t
  line-number-mode: t
  indent-tabs-mode: t
  transient-mark-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message mailcap yank-media puny rfc822
mml mml-sec password-cache epa derived epg rfc6068 epg-config gnus-util
text-property-search mm-decode mm-bodies mm-encode mail-parse rfc2231
mailabbrev gmm-utils mailheader sendmail rfc2047 rfc2045 ietf-drums
mm-util mail-prsvr mail-utils doc-view filenotify jka-compr image-mode
exif dired dired-loaddefs time-date comp comp-cstr warnings icons subr-x
rx cl-seq cl-macs gv cl-extra help-mode cl-loaddefs cl-lib bytecomp
byte-compile cconv rmc iso-transl tooltip eldoc paren electric uniquify
ediff-hook vc-hooks lisp-float-type elisp-mode mwheel term/x-win x-win
term/common-win x-dnd tool-bar dnd fontset image regexp-opt fringe
tabulated-list replace newcomment text-mode lisp-mode prog-mode register
page tab-bar menu-bar rfn-eshadow isearch easymenu timer select
scroll-bar mouse jit-lock font-lock syntax font-core term/tty-colors
frame minibuffer nadvice seq simple cl-generic indonesian philippine
cham georgian utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao
korean japanese eucjp-ms cp51932 hebrew greek romanian slovak czech
european ethiopic indian cyrillic chinese composite emoji-zwj charscript
charprop case-table epa-hook jka-cmpr-hook help abbrev obarray oclosure
cl-preloaded button loaddefs faces cus-face macroexp files window
text-properties overlay sha1 md5 base64 format env code-pages mule
custom widget keymap hashtable-print-readable backquote threads dbusbind
inotify dynamic-setting system-font-setting font-render-setting cairo
move-toolbar gtk x-toolkit xinput2 x multi-tty make-network-process
native-compile emacs)

Memory information:
((conses 16 108025 17657)
 (symbols 48 7602 0)
 (strings 32 30705 1814)
 (string-bytes 1 939199)
 (vectors 16 23470)
 (vector-slots 8 390912 21366)
 (floats 8 71 19)
 (intervals 56 222 0)
 (buffers 1000 13))

[-- Attachment #2: Type: text/html, Size: 4376 bytes --]

bug#57854: some extra info from inspecting the `pdftocio` package

[dalanicolai]

[-- Attachment #1: Type: text/plain, Size: 1103 bytes --]

When looking into the `pdftocio` package I find the following piece of code

```
    try:
        with open_pdf(path_in) as doc:
            if toc_file.isatty() or print_toc:
                # no input from user, switch to output mode and extract the
toc
                # of pdf
                toc = read_toc(doc)
                if len(toc) == 0:
                    print("error: no table of contents found",
file=sys.stderr)
                    sys.exit(1)

                if readable:
                    print(pprint_toc(toc))
                else:
                    print(dump_toc(toc), end="")
                sys.exit(0)

            # an input is given, so switch to input mode
            toc = parse_toc(toc_file)
            write_toc(doc, toc)
```

As from using `shell-command-to-string` we find that `len(toc)` must be 0
(it returns an empty string), so then it is probably the case that python
determines that some input has been given.

So maybe, even though in the `call-process` the INFILE command is set to
`nil`, still python might
determine that some kind of input is given.

[-- Attachment #2: Type: text/html, Size: 1457 bytes --]

bug#57854: some extra info from inspecting the `pdftocio` package

[Eli Zaretskii]

When responding to a bug report, please NEVER ever change the Subject
line.

If you do that, it makes it harder to follow the discussion, and can
even start a new bug report in some cases.

Please always use the original Subject line.

bug#57854: some extra info from inspecting the `pdftocio` package

[dalanicolai]

[-- Attachment #1: Type: text/plain, Size: 342 bytes --]

Thanks! Will do that...

On Fri, 16 Sept 2022 at 12:39, Eli Zaretskii <eliz@gnu.org> wrote:

> When responding to a bug report, please NEVER ever change the Subject
> line.
>
> If you do that, it makes it harder to follow the discussion, and can
> even start a new bug report in some cases.
>
> Please always use the original Subject line.
>

[-- Attachment #2: Type: text/html, Size: 639 bytes --]

bug#57854: 29.0.50; Different exit code in Emacs and terminal for identical process

[Eli Zaretskii]

> From: dalanicolai <dalanicolai@gmail.com>
> Date: Fri, 16 Sep 2022 12:00:09 +0200
> 
> When looking into the `pdftocio` package I find the following piece of code
> 
> ```
>     try:
>         with open_pdf(path_in) as doc:
>             if toc_file.isatty() or print_toc:
>                 # no input from user, switch to output mode and extract the toc
>                 # of pdf
>                 toc = read_toc(doc)
>                 if len(toc) == 0:
>                     print("error: no table of contents found", file=sys.stderr)
>                     sys.exit(1)
> 
>                 if readable:
>                     print(pprint_toc(toc))
>                 else:
>                     print(dump_toc(toc), end="")
>                 sys.exit(0)
> 
>             # an input is given, so switch to input mode
>             toc = parse_toc(toc_file)
>             write_toc(doc, toc)
> ```

Note that the above distinguishes between TTY and non-TTY input, and
call-process works via the non-TTY case, AFAIU.  So maybe what you see
is entirely expected?

bug#57854: 29.0.50; Different exit code in Emacs and terminal for identical process

[dalanicolai]

[-- Attachment #1: Type: text/plain, Size: 1730 bytes --]

I don't understand the answer well (my knowledge about computers is very
limited),
i.e. I do not immediately understand what it means for a file to be a tty.

But also, I think the isatty() is about the TOC file, i.e. the file given
as INFILE (after the `<`)
But I am not giving any INFILE (which would make the command add the TOC to
the file given as argument),
Instead I simply provide a single filepath as argument so that the command
simply prints the TOC.

(I hope this answer makes sense, I might be misunderstanding something)

On Fri, 16 Sept 2022 at 13:00, Eli Zaretskii <eliz@gnu.org> wrote:

> > From: dalanicolai <dalanicolai@gmail.com>
> > Date: Fri, 16 Sep 2022 12:00:09 +0200
> >
> > When looking into the `pdftocio` package I find the following piece of
> code
> >
> > ```
> >     try:
> >         with open_pdf(path_in) as doc:
> >             if toc_file.isatty() or print_toc:
> >                 # no input from user, switch to output mode and extract
> the toc
> >                 # of pdf
> >                 toc = read_toc(doc)
> >                 if len(toc) == 0:
> >                     print("error: no table of contents found",
> file=sys.stderr)
> >                     sys.exit(1)
> >
> >                 if readable:
> >                     print(pprint_toc(toc))
> >                 else:
> >                     print(dump_toc(toc), end="")
> >                 sys.exit(0)
> >
> >             # an input is given, so switch to input mode
> >             toc = parse_toc(toc_file)
> >             write_toc(doc, toc)
> > ```
>
> Note that the above distinguishes between TTY and non-TTY input, and
> call-process works via the non-TTY case, AFAIU.  So maybe what you see
> is entirely expected?
>

[-- Attachment #2: Type: text/html, Size: 2500 bytes --]

bug#57854: 29.0.50; Different exit code in Emacs and terminal for identical process

[dalanicolai]

[-- Attachment #1: Type: text/plain, Size: 2376 bytes --]

B.t.w. I am using this command in `toc-mode`, which is available on MELPA.
It is a quite awesome package to add TOC to pdf/djvu files that do not
contain one
(it can optionally do OCR via tesseract). The command is used in the
function
`toc--pdftocgen-add-to-pdf`, where I use the contents of the TOC buffer as
infile,
to add it to the PDF (which is passes as an argument). Maybe this
explanation
makes even more clear what the command is used for (in any case too much
explanation will probably do no harm)

On Fri, 16 Sept 2022 at 21:29, dalanicolai <dalanicolai@gmail.com> wrote:

> I don't understand the answer well (my knowledge about computers is very
> limited),
> i.e. I do not immediately understand what it means for a file to be a tty.
>
> But also, I think the isatty() is about the TOC file, i.e. the file given
> as INFILE (after the `<`)
> But I am not giving any INFILE (which would make the command add the TOC
> to the file given as argument),
> Instead I simply provide a single filepath as argument so that the command
> simply prints the TOC.
>
> (I hope this answer makes sense, I might be misunderstanding something)
>
> On Fri, 16 Sept 2022 at 13:00, Eli Zaretskii <eliz@gnu.org> wrote:
>
>> > From: dalanicolai <dalanicolai@gmail.com>
>> > Date: Fri, 16 Sep 2022 12:00:09 +0200
>> >
>> > When looking into the `pdftocio` package I find the following piece of
>> code
>> >
>> > ```
>> >     try:
>> >         with open_pdf(path_in) as doc:
>> >             if toc_file.isatty() or print_toc:
>> >                 # no input from user, switch to output mode and extract
>> the toc
>> >                 # of pdf
>> >                 toc = read_toc(doc)
>> >                 if len(toc) == 0:
>> >                     print("error: no table of contents found",
>> file=sys.stderr)
>> >                     sys.exit(1)
>> >
>> >                 if readable:
>> >                     print(pprint_toc(toc))
>> >                 else:
>> >                     print(dump_toc(toc), end="")
>> >                 sys.exit(0)
>> >
>> >             # an input is given, so switch to input mode
>> >             toc = parse_toc(toc_file)
>> >             write_toc(doc, toc)
>> > ```
>>
>> Note that the above distinguishes between TTY and non-TTY input, and
>> call-process works via the non-TTY case, AFAIU.  So maybe what you see
>> is entirely expected?
>>
>

[-- Attachment #2: Type: text/html, Size: 3456 bytes --]

bug#57854: 29.0.50; Different exit code in Emacs and terminal for identical process

[Eli Zaretskii]

> From: dalanicolai <dalanicolai@gmail.com>
> Date: Fri, 16 Sep 2022 21:29:15 +0200
> Cc: 57854@debbugs.gnu.org
> 
> I don't understand the answer well (my knowledge about computers is very limited),
> i.e. I do not immediately understand what it means for a file to be a tty.
> 
> But also, I think the isatty() is about the TOC file, i.e. the file given as INFILE (after the `<`)
> But I am not giving any INFILE (which would make the command add the TOC to the file given as
> argument),
> Instead I simply provide a single filepath as argument so that the command simply prints the TOC.

I mentioned that aspect because it could be different between
invocation from shell prompt and from call-process.  I didn't examine
the Python code of the program more than look at the snipped you
posted.

Basically, I don't think this is an Emacs problem, because
call-process faithfully reports the exit code of the program it runs.
The reason for the different behavior is almost certainly in the
program itself or in some factor that is different between how you
invoke it from shell and how you invoked it from Lisp.

bug#57854: 29.0.50; Different exit code in Emacs and terminal for identical process

[dalanicolai]

[-- Attachment #1: Type: text/plain, Size: 2182 bytes --]

Indeed, when invoking the command from the shell prompt using /dev/null as
input
 (pdftocio ... < /dev/null), then the command does not return an error
(i.e. exit code is 0).

So, indeed there is a difference between invoking it from Emacs and
invoking it from the
shell prompt (without the /dev/null input). This might not be considered a
bug, but it is not
trivial to me, that using call-process implies sending the null-device as
input.

Is there a way to call a process from elisp, without sending the input?
Otherwise, I would
probably change this into a 'documentation bug' report, in the sense that
it would be nice
if this detail was mentioned in the docs (I think it is not currently). I
am not sure, if this is a
case of a 'bad/wrongly designed' pdftocio command (I would say that the
command might
be well designed, but just it assumes that it is being used from the shell
prompt. However,
I am no expert on unix(/posix?) 'protocols').

On Sat, 17 Sept 2022 at 08:10, Eli Zaretskii <eliz@gnu.org> wrote:

> > From: dalanicolai <dalanicolai@gmail.com>
> > Date: Fri, 16 Sep 2022 21:29:15 +0200
> > Cc: 57854@debbugs.gnu.org
> >
> > I don't understand the answer well (my knowledge about computers is very
> limited),
> > i.e. I do not immediately understand what it means for a file to be a
> tty.
> >
> > But also, I think the isatty() is about the TOC file, i.e. the file
> given as INFILE (after the `<`)
> > But I am not giving any INFILE (which would make the command add the TOC
> to the file given as
> > argument),
> > Instead I simply provide a single filepath as argument so that the
> command simply prints the TOC.
>
> I mentioned that aspect because it could be different between
> invocation from shell prompt and from call-process.  I didn't examine
> the Python code of the program more than look at the snipped you
> posted.
>
> Basically, I don't think this is an Emacs problem, because
> call-process faithfully reports the exit code of the program it runs.
> The reason for the different behavior is almost certainly in the
> program itself or in some factor that is different between how you
> invoke it from shell and how you invoked it from Lisp.
>

[-- Attachment #2: Type: text/html, Size: 2860 bytes --]

bug#57854: 29.0.50; Different exit code in Emacs and terminal for identical process

[Eli Zaretskii]

> From: dalanicolai <dalanicolai@gmail.com>
> Date: Sat, 17 Sep 2022 15:37:08 +0200
> Cc: 57854@debbugs.gnu.org
> 
> Indeed, when invoking the command from the shell prompt using /dev/null as input
>  (pdftocio ... < /dev/null), then the command does not return an error (i.e. exit code is 0).
> 
> So, indeed there is a difference between invoking it from Emacs and invoking it from the
> shell prompt (without the /dev/null input). This might not be considered a bug, but it is not
> trivial to me, that using call-process implies sending the null-device as input.
> 
> Is there a way to call a process from elisp, without sending the input?

You could use start-process instead, and then wait for the process to
finish.  It would complicate the Lisp program, though.

> Otherwise, I would
> probably change this into a 'documentation bug' report, in the sense that it would be nice
> if this detail was mentioned in the docs (I think it is not currently).

The doc string of call-process already says that:

 The program’s input comes from file INFILE (nil means ‘null-device’).

bug#57854: 29.0.50; Different exit code in Emacs and terminal for identical process

[dalanicolai]

[-- Attachment #1: Type: text/plain, Size: 1490 bytes --]

Ah yes, you are totally right (I probably looked over it because I,
previously, did not understand its meaning/consequences).

Using start-process will be fine (setting a sentinel is not too
complicated).

Thanks (again) for your help!



On Sat, 17 Sept 2022 at 15:56, Eli Zaretskii <eliz@gnu.org> wrote:

> > From: dalanicolai <dalanicolai@gmail.com>
> > Date: Sat, 17 Sep 2022 15:37:08 +0200
> > Cc: 57854@debbugs.gnu.org
> >
> > Indeed, when invoking the command from the shell prompt using /dev/null
> as input
> >  (pdftocio ... < /dev/null), then the command does not return an error
> (i.e. exit code is 0).
> >
> > So, indeed there is a difference between invoking it from Emacs and
> invoking it from the
> > shell prompt (without the /dev/null input). This might not be considered
> a bug, but it is not
> > trivial to me, that using call-process implies sending the null-device
> as input.
> >
> > Is there a way to call a process from elisp, without sending the input?
>
> You could use start-process instead, and then wait for the process to
> finish.  It would complicate the Lisp program, though.
>
> > Otherwise, I would
> > probably change this into a 'documentation bug' report, in the sense
> that it would be nice
> > if this detail was mentioned in the docs (I think it is not currently).
>
> The doc string of call-process already says that:
>
>  The program’s input comes from file INFILE (nil means ‘null-device’).
>

[-- Attachment #2: Type: text/html, Size: 2066 bytes --]

bug#57854: 29.0.50; Different exit code in Emacs and terminal for identical process

[Eli Zaretskii]

> From: dalanicolai <dalanicolai@gmail.com>
> Date: Sat, 17 Sep 2022 16:16:42 +0200
> Cc: 57854@debbugs.gnu.org
> 
> Ah yes, you are totally right (I probably looked over it because I,
> previously, did not understand its meaning/consequences).
> 
> Using start-process will be fine (setting a sentinel is not too complicated).
> 
> Thanks (again) for your help!

Glad I could help, and closing the bug.

bug#57854: Test result to previous suggestion

[dalanicolai]

[-- Attachment #1: Type: text/plain, Size: 345 bytes --]

In my previous message, I suggested that python might determine that some
input was given.

I have tested it now by letting the command print a message in case input
was determined,
and indeed, while the terminal still produces the error, using
`shell-command-to-string` now
returns the printed message. So I guess the 'hypothesis' was correct.

[-- Attachment #2: Type: text/html, Size: 435 bytes --]

bug#57854: good to know

[dalanicolai]

[-- Attachment #1: Type: text/plain, Size: 487 bytes --]

Just to make it clear the command should work as follows:

if the command is called with only 1 argument (a filepath) then the command
should print the
table of contents

however when some 'table of contents' data file is given as 'INFILE' as
follows

pdftocio [options] in.pdf < toc

then the command should add the TOC to the pdf file.

So what seems to happen is that, even when INFILE in `call-process` is nil,
still python
determines that some (empty) `TOC` file is given as input.

[-- Attachment #2: Type: text/html, Size: 683 bytes --]

bug#57854: 29.0.50; Different exit code in Emacs and terminal for identical process

[Eli Zaretskii]

[Note that I resurrected the origina Subject of your bug report.]

> From: dalanicolai <dalanicolai@gmail.com>
> Date: Fri, 16 Sep 2022 12:12:00 +0200
> 
> So what seems to happen is that, even when INFILE in `call-process` is nil, still python
> determines that some (empty) `TOC` file is given as input.

Emacs connects standard input to the null device in such a case.

bug#57854: 29.0.50; Different exit code in Emacs and terminal for identical process

[Eli Zaretskii]

> From: dalanicolai <dalanicolai@gmail.com>
> Date: Fri, 16 Sep 2022 11:39:25 +0200
> 
> Execute in the terminal the command pdftocio (see
> https://github.com/Krasjet/pdf.tocgen#pdftocio, the command is
>  easily available via PyPI) on a pdf file that does
> not contain a Table of Contents. The command produces an error and
> subsequently executing `echo $?` shows exit code 1.
> 
> Subsequently, executing the same command from Emacs on the same file,
> then `call-process` returns a 0, and indeed `shell-command-to-string`
> produces an empty string. However, I would expect an error like in the
> terminal (with call-process returning exit code 1).

Please show the Lisp code you used to invoke call-process in this
case.

Thanks.

bug#57854: 29.0.50; Different exit code in Emacs and terminal for identical process

[Eli Zaretskii]

[Please use "Reply All" to reply, to keep the bug tracker on the CC.]

> From: dalanicolai <dalanicolai@gmail.com>
> Date: Fri, 16 Sep 2022 21:20:17 +0200
> 
> The code I was using is `(call-process "pdftocio" nil nil nil buffer-file-name)` (from the buffer visiting the file)

This makes the null device the standard input of the program.  What
happens if you invoke pdftocio from the shell prompt with a similar
redirection:

  $ pdftocio ... < /dev/null

(replace "..." with whatever command-line arguments you need, probably
just the file name).  What exit code does the program return then?