ECM for possible process-status/sentinel bug

ECM for possible process-status/sentinel bug

[Adam Porter]

[-- Attachment #1: Type: text/plain, Size: 2959 bytes --]

Hi,

The attached file provides an ECM for a possible bug in Emacs's process
handling.  The rub is whether a process sentinel may be called with a
STATUS string of "finished\n" while the process's `process-status' is
`open': If that is expected, then this may not indicate a bug (but
perhaps an issue that needs to be more prominently documented).  But if
it's not supposed to happen, then this may provide a means to reproduce
the behavior and troubleshoot the bug.

This ECM works by making 10 (by default) curl `pipe' processes that send
requests to httpbin.org.  When each process is finished, the process
sentinel should conclude by killing the process's buffer.

This appears to work most of the time, especially with low numbers of
processes (like 1-3).  But as the number of processes is increased,
sometimes one or more of the attempts to kill a process's buffer result
in the user's being prompted to kill the buffer due to the process still
running (an obvious problem when happening in library code that the user
should not be concerned with).  As the number of processes increases,
the number of process buffers that prompt seems to increase (e.g. with
15 processes, 1-2 prompts seems common).

A possible workaround would be to check the process's status before
attempting to kill the buffer, however that would seem to raise a
problem: according to the Elisp manual, calls to the sentinel may be
coalesced when a process's status changes quickly, so if the sentinel
simply did not kill the buffer if the process were still alive, it's
unclear whether the sentinel will be called a final time, after the
process's status is `closed'; if not, the buffer would never be killed.
Alternatively, the sentinel could loop while checking `process-status',
but that would obviously be undesirable, and would seem like working
around a bug (and it might be the wrong thing to do in general,
depending on the process's and sentinel's purposes).

To test this ECM:

1.  Evaluate this file (with lexical-binding).

2.  Eval (argh).

3.  The messages buffer should show the first line of the HTTP response
body for requests 1-10, as well as a "killing buffer..."  line for each
curl process's buffer.  There should be NO "Buffer has a running
process; kill it?" prompts.

4.  But, usually, there will be one or two of those prompts.  When there
are, you can see that the buffer's process has a (process-status) value
of "open" rather than "closed", even though the process's sentinel was
called with "finished\n" as the status argument.

So far, this has been reproduced on Emacs 26.3, 27.2 (with Eric
Abrahamsen's help), and 28.0.50 (built on 2021-07-05), on GNU/Linux.

Alternatively, if this simply demonstrates a lack of understanding on my
part of how to properly use sentinels, I'd be grateful to be corrected.

[This file has also been posted at
https://gist.github.com/alphapapa/daa36dc3388dec694602e06fe6d114b7, and
may be updated there if necessary.

[-- Attachment #2: ECM --]
[-- Type: application/emacs-lisp, Size: 5471 bytes --]

[-- Attachment #3: Type: text/plain, Size: 16 bytes --]

]

Thanks,
Adam

Re: ECM for possible process-status/sentinel bug

[Adam Porter]

Hi,

I've spent some more time trying to debug my process-related code today,
but I've ended up in the same place: when the curl process exits, and
the process's buffer has finished receiving the process's output,
sometimes the sentinel is not called a final time.

It seems to happen often when processes are called in quick succession,
like when running a series of tests with ERT, but rarely when processes
are called slowly, like when running one test at a time with ERT.  It
almost seems consistent in that, out of 23 tests, it tends to begin
failing at the same test on each run of the series, yet those same tests
can pass when run individually, which makes me wonder if it's something
to do with Emacs's internal bookkeeping around processes.

In one of the ERT tests, I was able to work around the problem by:

1.  In the sentinel, putting a flag property on the process after the
    sentinel has been called and processed the response accordingly.

2.  In the test definition, test for that flag property, and loop
    calling (accept-process-output PROCESS) until the flag is set.  Of
    course, this is not suitable for real-world usage, because it
    effectively turns an asynchronous process into a synchronous one,
    defeating the purpose of the library.

So I'm left again with the test case I posted in the first message in
this thread.  In order to avoid filing useless reports, I was hoping to
get some advice along the lines of, "Yes, that might be a bug, please
report it as one," or "No, you just misunderstand how sentinels are
intended to work."  If someone could so advise me, I'd be grateful.
Otherwise I guess I'll report a bug in a day or so.

Thanks,
Adam

Re: ECM for possible process-status/sentinel bug

[Eli Zaretskii]

> From: Adam Porter <adam@alphapapa.net>
> Date: Sun, 22 Aug 2021 14:13:40 -0500
> 
> So I'm left again with the test case I posted in the first message in
> this thread.  In order to avoid filing useless reports, I was hoping to
> get some advice along the lines of, "Yes, that might be a bug, please
> report it as one," or "No, you just misunderstand how sentinels are
> intended to work."  If someone could so advise me, I'd be grateful.
> Otherwise I guess I'll report a bug in a day or so.

Please report a bug with the simplest recipe to reproduce it.

Thanks.

Re: ECM for possible process-status/sentinel bug

[chad]

[-- Attachment #1: Type: text/plain, Size: 197 bytes --]

Shot out of left field here, but: are you running these tests on macOS? If
so, are you using the ns port from git, or perhaps the mac port?

I assume TRT is to file a bug report either way.

~Chad

[-- Attachment #2: Type: text/html, Size: 267 bytes --]

Re: ECM for possible process-status/sentinel bug

[Adam Porter]

chad <yandros@gmail.com> writes:

> Shot out of left field here, but: are you running these tests on
> macOS? If so, are you using the ns port from git, or perhaps the mac
> port?

No, as written in the first message, this is on GNU/Linux.  :)

Re: ECM for possible process-status/sentinel bug

[Adam Porter]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Adam Porter <adam@alphapapa.net>
>> Date: Sun, 22 Aug 2021 14:13:40 -0500
>> 
>> So I'm left again with the test case I posted in the first message in
>> this thread.  In order to avoid filing useless reports, I was hoping to
>> get some advice along the lines of, "Yes, that might be a bug, please
>> report it as one," or "No, you just misunderstand how sentinels are
>> intended to work."  If someone could so advise me, I'd be grateful.
>> Otherwise I guess I'll report a bug in a day or so.
>
> Please report a bug with the simplest recipe to reproduce it.

Thanks.  I filed it here:

https://debbugs.gnu.org/cgi/bugreport.cgi?bug=50166

Lars narrowed down the apparent problem quickly, which may be a
documentation issue.

However, if it is, I'm left to wonder what the right way is to capture
the STDERR of a (make-process) process while also waiting on all of the
process's associated processes to exit, have their output collected into
the designated buffers, and their sentinels run.  The manual says to use
(while (accept-process-output PROCESS)), but apparently that may return
while a process's :stderr process is still running, or something like
that...  (The fact that "process" seems to have some extra meanings in
the context--it refers not only to the program called with
`make-process'--seems to add to the difficulty of reasoning and
communicating about these issues.)