Signaling errors within process sentinels only works when DEBUG-ON-ERROR is non-nil

Signaling errors within process sentinels only works when DEBUG-ON-ERROR is non-nil

[Adam Porter]

Hi,

For a while now, in my work on plz.el[0], I've been trying to understand 
what causes an intermittent problem in that some HTTP requests sometimes 
return nil instead of the intended value.  Some of my attempts are 
mentioned in plz.el#3[1], and that resulted in my filing Emacs bug#50166[2].

Finally, today I may have had a breakthough: I noticed that synchronous 
requests that are expected to signal an error which is intended to be 
handled by a CONDITION-CASE form do not have their error handled by the 
form; instead Emacs seems to intercept the error itself, and so the 
value that the CONDITION-CASE would return is not returned, with NIL 
being returned instead.  For example, see this code:

   (condition-case err
       (plz 'get "https://httpbinnnnnn.org/get/status/404")
     (error 'foo))

This makes an HTTP request to a non-existent domain name, for which plz 
signals an error in the process sentinel.  However, when evaluating this 
form (with EVAL-EXPRESSION-DEBUG-ON-ERROR set to NIL), instead of 
evaluating to 'FOO, I get this in the *Messages* buffer:

   error in process sentinel: plz: Curl error: "plz--sentinel: Curl 
error", #s(plz-error (6 . "Couldn't resolve host. The given remote host 
was not resolved.") nil nil)
   nil

The CONDITION-CASE is apparently not allowed to handle the error, and 
NIL is returned instead of 'FOO (after the 2-second 
process-sentinel-error delay, which I also learned about recently, which 
Lars recently added a variable to control).

In re-reading the Info page `(elisp)Sentinels', I saw this:

   If an error happens during execution of a sentinel, it is caught
   automatically, so that it doesn’t stop the execution of whatever
   programs was running when the sentinel was started.  However, if
   ‘debug-on-error’ is non-‘nil’, errors are not caught.

So I tried binding DEBUG-ON-ERROR non-nil around the call to PLZ, and 
sure enough, this solved the problem:

   (let ((debug-on-error t))
     (condition-case err
         (plz 'get "https://httpbinnnnnn.org/get/status/404")
       (error 'foo)))

That evaluates to 'FOO, immediately, as expected.

So my questions:

1. Is this an acceptable way to work around this problem (of not being 
"allowed" to signal errors within a process sentinel)?

2. Are there any potential problems with this workaround?

3. Is there a better way?

4. Should Emacs be patched to change this behavior?  It seems strange to 
me that, unless a seemingly unrelated variable is bound, errors within 
sentinels can't be caught by CONDITION-CASE forms enclosing the code 
that signals the error.

FWIW, I have spent many hours of apparently wasted time trying to debug 
this, what has seemed to be a mysterious problem.  Admittedly, this has 
been documented in the manual for quite some years, and I've read that 
page many times, but it wasn't until today that I was able to recognize 
what was happening in the code and connect the behavior with that 
paragraph in the manual.  So if Emacs could be made to behave more 
"naturally" in this respect, it would probably be widely useful.

As a point of comparison, request.el, the popular HTTP library for 
Emacs, seems to work around this by catching errors in a function called 
from within the sentinel and returning a list including the error data, 
rather than allowing the signal to propagate up to application code[3]. 
This certainly works, but it seems unnatural; wouldn't it be preferable 
to allow callers to wrap the call in their own CONDITION-CASE?

Thanks,
Adam

0: https://github.com/alphapapa/plz.el
1: https://github.com/alphapapa/plz.el/issues/3
2: https://debbugs.gnu.org/cgi/bugreport.cgi?bug=50166
3: 
https://github.com/tkf/emacs-request/blob/01e338c335c07e4407239619e57361944a82cb8a/request.el#L1146

Re: Signaling errors within process sentinels only works when DEBUG-ON-ERROR is non-nil

[Eli Zaretskii]

> Date: Mon, 8 May 2023 20:07:37 -0500
> From: Adam Porter <adam@alphapapa.net>
> 
> The CONDITION-CASE is apparently not allowed to handle the error, and 
> NIL is returned instead of 'FOO (after the 2-second 
> process-sentinel-error delay, which I also learned about recently, which 
> Lars recently added a variable to control).
> 
> In re-reading the Info page `(elisp)Sentinels', I saw this:
> 
>    If an error happens during execution of a sentinel, it is caught
>    automatically, so that it doesn’t stop the execution of whatever
>    programs was running when the sentinel was started.  However, if
>    ‘debug-on-error’ is non-‘nil’, errors are not caught.
> 
> So I tried binding DEBUG-ON-ERROR non-nil around the call to PLZ, and 
> sure enough, this solved the problem:
> 
>    (let ((debug-on-error t))
>      (condition-case err
>          (plz 'get "https://httpbinnnnnn.org/get/status/404")
>        (error 'foo)))
> 
> That evaluates to 'FOO, immediately, as expected.
> 
> So my questions:
> 
> 1. Is this an acceptable way to work around this problem (of not being 
> "allowed" to signal errors within a process sentinel)?

I think it is not very clean.  debug-on-error is not supposed to be
set by Lisp programs, unless the Lisp program is itself used for
debugging something.

> 2. Are there any potential problems with this workaround?

Yes.  For starters, you will break any use case where a user or a Lisp
program wants to know about any *real* errors in your sentinel code,
because when they set debug-on-error non-nil, the error will not
propagate all the way to top level or to a signal handler installed by
higher-level callers.

> 3. Is there a better way?

Yes: don't design a sentinel that signals an error.  Why is that
necessary?  I'm guessing you did that out of some convenience, not
because there are no other ways of handling those situations.

In general, code running in the background should not signal errors it
doesn't itself catch, except when there are situations it cannot
possibly handle by itself.

> 4. Should Emacs be patched to change this behavior?  It seems strange to 
> me that, unless a seemingly unrelated variable is bound, errors within 
> sentinels can't be caught by CONDITION-CASE forms enclosing the code 
> that signals the error.

debug-on-error is not "unrelated".  We pay attention to it so as to
allow debugging of sentinels, which would otherwise be nigh impossible
to do on the Lisp level.

> FWIW, I have spent many hours of apparently wasted time trying to debug 
> this, what has seemed to be a mysterious problem.  Admittedly, this has 
> been documented in the manual for quite some years, and I've read that 
> page many times, but it wasn't until today that I was able to recognize 
> what was happening in the code and connect the behavior with that 
> paragraph in the manual.  So if Emacs could be made to behave more 
> "naturally" in this respect, it would probably be widely useful.

I think what Emacs does today is very natural: we are running
arbitrary Lisp asynchronously, which could be in the middle of an
arbitrary foreground Lisp program.  E.g., imagine that Emacs just
prompted the user for something in the minibuffer and is waiting for
the user to type a complete response to the prompt -- how can we allow
a sentinel that happens to run at that time to signal an error? that
could error out of the command which prompted the user!

It is basically the same situation as calling some Lisp hook from
redisplay: it isn't a coincidence that redisplay calls such hooks in a
similar manner, catching all errors and logging the information about
them in *Messages*.

> As a point of comparison, request.el, the popular HTTP library for 
> Emacs, seems to work around this by catching errors in a function called 
> from within the sentinel and returning a list including the error data, 
> rather than allowing the signal to propagate up to application code[3]. 
> This certainly works, but it seems unnatural; wouldn't it be preferable 
> to allow callers to wrap the call in their own CONDITION-CASE?

IMO, what request.el does is _exactly_ what a sentinel should do when
it wants to handle an error signaled by its code or by one of its
subroutines.  It is not "unnatural", it is what every sentinel (and
filter function as well) should do, because signaling errors from
asynchronous code is a very bad idea.