bug#63078: 26.1; Sentinels are sometimes unexpectedly only invoked in script mode

bug#63078: 26.1; Sentinels are sometimes unexpectedly only invoked in script mode

[Markus Triska]

Dear all,

to reproduce this issue, please save the following C code in proc.c:

    #include <stdio.h>
    #include <stdlib.h>
    int main() {
      printf("hello");
      exit(0);
    }

and compile it, yielding the program file "./proc":

    $ gcc proc.c -o proc 

Further, please store the following Elisp definitions in proc.el, in
the same directory as the above program:

    (add-to-list 'exec-path ".")

    (defvar p-stopped nil)

    (defun my-sentinel (proc str)
      (when (string-match "^\\(?:finished\n\\|exited abnormally\\|killed\n\\)"
                          str)
        (setq p-stopped t)))

    (defun wait-for-process-end (proc)
      (while (not p-stopped)
        (message "process status is: %s" (process-status proc))
        (accept-process-output proc 0.1 nil t)))

    (defun f ()
      (with-temp-buffer
        (let ((proc (start-process "p" (current-buffer) "proc")))
          (setq p-stopped nil)
          (set-process-sentinel proc 'my-sentinel)
          (wait-for-process-end proc))
        (message "process said: %s" (buffer-string))))

The following invocation and result shows a case where the program
behaves exactly as intended: It waits for the process sentinel to
signal that no more output can arrive from the process, and shows the
output it got while the process was producing output:

    $ emacs -Q --script proc.el --eval "(f)"
    process status is: exit
    process said: hello

In contrast, the following invocation produces, in the echo area, the
message "process status is: exit", and Emacs hangs unexpectedly:

    $ emacs -Q -l proc.el --eval "(f)"

The behaviour I expect in this case is analogous to the other
invocation: I do expect the message we see, followed by the message
"process said: hello", as in the case above. However, this second
message only appears if I manually interrupt Emacs with C-g.

Regarding the background of this use case:

The goal of one of my Emacs scripts is to reliably obtain *all* output
from a process, until the process ends (which it always does), and
then end the script.

A flawed attempt to do this is to check the state of the process (via
process-status), and end the script when the the state is no longer
'run. However, process output can (and sometimes does) still arrive
even after the state is no longer equal to 'run. Apparently only the
sentinel can reliably detect that no more process output can follow.

Is there a way to make Emacs invoke the sentinel in the example above?

Thank you a lot!
Markus



In GNU Emacs 26.1 (build 3, x86_64-pc-linux-gnu, X toolkit, Xaw scroll bars)
 of 2019-04-09 built on mt-laptop
Windowing system distributor 'The X.Org Foundation', version 11.0.12201003
System Description:	Ubuntu 22.10

Configured using:
 'configure --with-tiff=no'

Configured features:
XPM JPEG GIF PNG SOUND GSETTINGS NOTIFY GNUTLS LIBXML2 FREETYPE XFT ZLIB
TOOLKIT_SCROLL_BARS LUCID X11 THREADS

Important settings:
  value of $LC_MONETARY: en_GB.UTF-8
  value of $LC_NUMERIC: en_GB.UTF-8
  value of $LC_TIME: en_GB.UTF-8
  value of $LANG: en_US.UTF-8
  value of $XMODIFIERS: @im=ibus
  locale-coding-system: utf-8-unix

bug#63078: 26.1; Sentinels are sometimes unexpectedly only invoked in script mode

[Eli Zaretskii]

> From: Markus Triska <triska@metalevel.at>
> Date: Wed, 26 Apr 2023 01:15:36 +0200
> 
> to reproduce this issue, please save the following C code in proc.c:
> 
>     #include <stdio.h>
>     #include <stdlib.h>
>     int main() {
>       printf("hello");
>       exit(0);
>     }
> 
> and compile it, yielding the program file "./proc":
> 
>     $ gcc proc.c -o proc 
> 
> Further, please store the following Elisp definitions in proc.el, in
> the same directory as the above program:
> 
>     (add-to-list 'exec-path ".")
> 
>     (defvar p-stopped nil)
> 
>     (defun my-sentinel (proc str)
>       (when (string-match "^\\(?:finished\n\\|exited abnormally\\|killed\n\\)"
>                           str)
>         (setq p-stopped t)))
> 
>     (defun wait-for-process-end (proc)
>       (while (not p-stopped)
>         (message "process status is: %s" (process-status proc))
>         (accept-process-output proc 0.1 nil t)))
> 
>     (defun f ()
>       (with-temp-buffer
>         (let ((proc (start-process "p" (current-buffer) "proc")))
>           (setq p-stopped nil)
>           (set-process-sentinel proc 'my-sentinel)
>           (wait-for-process-end proc))
>         (message "process said: %s" (buffer-string))))
> 
> The following invocation and result shows a case where the program
> behaves exactly as intended: It waits for the process sentinel to
> signal that no more output can arrive from the process, and shows the
> output it got while the process was producing output:
> 
>     $ emacs -Q --script proc.el --eval "(f)"
>     process status is: exit
>     process said: hello
> 
> In contrast, the following invocation produces, in the echo area, the
> message "process status is: exit", and Emacs hangs unexpectedly:
> 
>     $ emacs -Q -l proc.el --eval "(f)"
> 
> The behaviour I expect in this case is analogous to the other
> invocation: I do expect the message we see, followed by the message
> "process said: hello", as in the case above. However, this second
> message only appears if I manually interrupt Emacs with C-g.

I cannot reproduce the problem with Emacs 28.2: I get "process said:
hello" in the echo-area, as expected.  Did you try with the -nw
command-line switch, and if you do, does it change anything?

It could be some timing problem: you start a program that exits almost
immediately, so there could be some race condition between the program
and your code.

bug#63078: 26.1; Sentinels are sometimes unexpectedly only invoked in script mode

[Markus Triska]

Eli Zaretskii <eliz@gnu.org> writes:

> I cannot reproduce the problem with Emacs 28.2: I get "process said:
> hello" in the echo-area, as expected.  Did you try with the -nw
> command-line switch, and if you do, does it change anything?

> It could be some timing problem: you start a program that exits almost
> immediately, so there could be some race condition between the program
> and your code.

I can reproduce the problem also with Emacs 28.8, and yes, I also
believe this may be a timing issue: I have now constructed a test case
that exhibits the issue also when the "-nw" option is used. To
reproduce, please start Emacs with:

    $ emacs -Q

or with:

    $ emacs -Q -nw

and then place the forms below in a buffer, and then evaluate the buffer
contents with M-x eval-buffer RET. The bash shell is used as external
process, the sample C program I originally included is no longer needed.
>
The expected output of the forms below is an endless stream of messages
of the form:

    trying to elicit race condition at (25673 25753 287220 854000)
    process status is: run [2 times]
    process said: hi
    failed
    trying to elicit race condition at (25673 25753 289270 703000)
    process status is: run [2 times]
    process said: hi
    failed

i.e., the expected case is that *no* race condition exists, and
therefore eliciting the issue *fails*, which is the desired result as
long as the program runs.

However, after I few seconds, this expected sequence of messages stops,
and only the following message is then emitted over and over:

    process status is: exit

When this occurs, it means that the sentinel was unexpectedly not
invoked when the process stops, in contrast to all preceding cases where
the sentinel was invoked as expected when the process terminates.

Running the program in script mode yields the expected result over and
over, I cannot elicit the issue with the --script option of Emacs.

I hope this helps. Can you reproduce the issue with this example?

Thank you and all the best,
Markus


(defvar p-stopped nil)

(defun my-sentinel (proc str)
  (when (string-match "^\\(?:finished\n\\|exited abnormally\\|killed\n\\)"
                      str)
    (setq p-stopped t)))

(defun wait-for-process-end (proc)
  (while (not p-stopped)
    (message "process status is: %s" (process-status proc))
    (accept-process-output proc 0.1 nil t)))

(defun elicit-issue ()
  (with-temp-buffer
    (let ((proc (start-process "p" (current-buffer) "bash" "-c" "echo -n hi")))
      (setq p-stopped nil)
      (set-process-sentinel proc 'my-sentinel)
      (wait-for-process-end proc))
    (message "process said: %s" (buffer-string))))

(while t
  (message "trying to elicit race condition at %s" (current-time))
  (elicit-issue)
  (message "failed"))

bug#63078: 26.1; Sentinels are sometimes unexpectedly only invoked in script mode

[Andreas Schwab]

On Apr 26 2023, Markus Triska wrote:

> In contrast, the following invocation produces, in the echo area, the
> message "process status is: exit", and Emacs hangs unexpectedly:
>
>     $ emacs -Q -l proc.el --eval "(f)"

If you C-g at this point and look at *Messsages* you will see that the
message "process status is: exit" has been repeated a lot of times.  So
the issue is that the sentinal is being reentered ad infinitum,
preventing to make progress.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

bug#63078: 26.1; Sentinels are sometimes unexpectedly only invoked in script mode

[Markus Triska]

Andreas Schwab <schwab@linux-m68k.org> writes:

> If you C-g at this point and look at *Messsages* you will see that the
> message "process status is: exit" has been repeated a lot of times.  So
> the issue is that the sentinal is being reentered ad infinitum,
> preventing to make progress.

In the program I posted, this message is not emitted by the sentinel,
are you certain that the sentinel is invoked as intended in this case?

Thank you and all the best,
Markus