unwind-protect and inhibit-quit

unwind-protect and inhibit-quit

[Felix Dietrich]

Hello,

I have questions regarding “unwind-protect” and the two examples
provided in the elisp reference manual [1].


    ;; 1. Example
    (let ((buffer (get-buffer-create " *temp*")))
      (with-current-buffer buffer
        (unwind-protect
            body-form
          (kill-buffer buffer))))


    ;; 2. Example
    (let ((win nil))
      (unwind-protect
          (progn
            (setq process (ftp-setup-buffer host file))
            (if (setq win (ftp-login process host user password))
                (message "Logged in")
              (error "Ftp login failed")))
        (or win (and process (delete-process process)))))


In the last paragraph [2], the description of the second example mentions a
race condition between the assignment of the “ftp-setup-buffer”
functionʼs return value to the variable “process” and the user sending a
quit signal (C-g).  Does the first example suffer from the same issue in
itʼs assignment to the “buffer” variable?

That paragraph also mentions that “there is no easy way to fix this
bug”.  Is that statement still true or can this issue be resolved by
setting “inhibit-quit” in current Emacs versions:


    ;; 2. Example (added inhibit-quit)
    (let ((win nil))
      (unwind-protect
          (progn
            (let ((inhibit-quit t))
              (setq process (ftp-setup-buffer host file)))
            (if (setq win (ftp-login process host user password))
                (message "Logged in")
              (error "Ftp login failed")))
        (or win (and process (delete-process process)))))


The first example has the “unwind-protect” form nested inside
“with-current-buffer”; this leaves a window where the buffer is both
created and assigned to the variable “buffer” but not protected from
being left alive in case of a quit (or, in more elaborated uses, an
error).  Shouldn’t it, therefore, be the other way around:
“unwind-protect” wrapping “with-current-buffer”?  In the following
snippet I also have added “inhibit-quit”, assuming that it works like I
expect.


    ;; 1. Example (moved unwind-protect up; added inhibit-quit)
    (let (buffer)
      (unwind-protect
          (progn
            (let ((inhibit-quit t))
              (setq buffer (get-buffer-create " *temp*")))
            (with-current-buffer buffer
              body-form))
        (and buffer (kill-buffer buffer))))


There also seems to be an issue in the first example with modifying and
killing a potentially already existing buffer " *temp*", which could be
avoided, I believe, by using “generate-new-buffer”:


    ;; 1. Example (moved unwind-protect up; added inhibit-quit;
    ;;             use generate-new-buffer)
    (let (buffer)
      (unwind-protect
          (progn
            (let ((inhibit-quit t))
              (setq buffer (generate-new-buffer " *temp*")))
            (with-current-buffer buffer
              body-form))
        (and buffer (kill-buffer buffer))))


Does “generate-new-buffer” make the same guarantee as
“get-buffer-create” that it will never return nil?  At least
“generate-new-buffer” is implemented with “get-buffer-create”.


Footnotes:
[1]  <https://www.gnu.org/software/emacs/manual/html_node/elisp/Cleanups.html>
     
     (info "(elisp) Cleanups")

[2]  “This example has a small bug: if the user types C-g to quit, and the
     quit happens immediately after the function ftp-setup-buffer
     returns but before the variable process is set, the process will
     not be killed. There is no easy way to fix this bug, but at least
     it is very unlikely.”

-- 
Felix Dietrich

Re: unwind-protect and inhibit-quit

[Eli Zaretskii]

> From: Felix Dietrich <felix.dietrich@sperrhaken.name>
> Date: Thu, 15 Jul 2021 17:14:56 +0200
> 
> In the last paragraph [2], the description of the second example mentions a
> race condition between the assignment of the “ftp-setup-buffer”
> functionʼs return value to the variable “process” and the user sending a
> quit signal (C-g).  Does the first example suffer from the same issue in
> itʼs assignment to the “buffer” variable?

Yes.  But leaving an unused Emacs buffer around is a much less serious
issue than leaving a zombie process around.  We even have a command
which would kill such unused buffers from time to time -- see
midnight.el.

> That paragraph also mentions that “there is no easy way to fix this
> bug”.  Is that statement still true or can this issue be resolved by
> setting “inhibit-quit” in current Emacs versions:

You could use inhibit-quit, but that is generally a bad idea from the
UI point of view, when invoking potentially long-running functions:
you are preventing the user from interrupting that long function.  For
example, suppose the FTP command stalls for some reason.

> The first example has the “unwind-protect” form nested inside
> “with-current-buffer”; this leaves a window where the buffer is both
> created and assigned to the variable “buffer” but not protected from
> being left alive in case of a quit (or, in more elaborated uses, an
> error).  Shouldn’t it, therefore, be the other way around:
> “unwind-protect” wrapping “with-current-buffer”?

If that can be done, yes, it's better.

>         (and buffer (kill-buffer buffer))))

This should be

  (and (bufferp buffer) (kill-buffer buffer))

> There also seems to be an issue in the first example with modifying and
> killing a potentially already existing buffer " *temp*", which could be
> avoided, I believe, by using “generate-new-buffer”:

Yes.  (These are simplified examples, not industry-strength programs.)

> Does “generate-new-buffer” make the same guarantee as
> “get-buffer-create” that it will never return nil?  At least
> “generate-new-buffer” is implemented with “get-buffer-create”.

So you already have your answer, right?

Re: unwind-protect and inhibit-quit

[Thibaut Verron]

>
> > That paragraph also mentions that “there is no easy way to fix this
> > bug”.  Is that statement still true or can this issue be resolved by
> > setting “inhibit-quit” in current Emacs versions:
>
> You could use inhibit-quit, but that is generally a bad idea from the
> UI point of view, when invoking potentially long-running functions:
> you are preventing the user from interrupting that long function.  For
> example, suppose the FTP command stalls for some reason.
>

Now I'm curious too... Would something like this work?

 (let ((inhibit-quit t))
   (setq process
      (let ((inhibit-quit nil))
         (ftp-setup-buffer host file))))

Re: unwind-protect and inhibit-quit

[Eli Zaretskii]

> From: Thibaut Verron <thibaut.verron@gmail.com>
> Date: Fri, 16 Jul 2021 10:10:36 +0200
> Cc: help-gnu-emacs <help-gnu-emacs@gnu.org>
> 
> > You could use inhibit-quit, but that is generally a bad idea from the
> > UI point of view, when invoking potentially long-running functions:
> > you are preventing the user from interrupting that long function.  For
> > example, suppose the FTP command stalls for some reason.
> >
> 
> Now I'm curious too... Would something like this work?
> 
>  (let ((inhibit-quit t))
>    (setq process
>       (let ((inhibit-quit nil))
>          (ftp-setup-buffer host file))))

It might, but I still suggest to bind inhibit-quit non-nil only for
short durations of time and as little as possible.  There be dragons.

Re: unwind-protect and inhibit-quit

[Felix Dietrich]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Thibaut Verron <thibaut.verron@gmail.com>
>> Date: Fri, 16 Jul 2021 10:10:36 +0200
>> Cc: help-gnu-emacs <help-gnu-emacs@gnu.org>
>> 
>> > You could use inhibit-quit, but that is generally a bad idea from the
>> > UI point of view, when invoking potentially long-running functions:
>> > you are preventing the user from interrupting that long function.  For
>> > example, suppose the FTP command stalls for some reason.
>> >
>> 
>> Now I'm curious too... Would something like this work?
>> 
>>  (let ((inhibit-quit t))
>>    (setq process
>>       (let ((inhibit-quit nil))
>>          (ftp-setup-buffer host file))))
>
> It might, but I still suggest to bind inhibit-quit non-nil only for
> short durations of time and as little as possible.  There be dragons.

I just found the “with-local-quit” macro.  Itʼs comment says that the
quit-flag “will not be handled until the next function call”.  Doesnʼt
that imply that once the let form has a value to return, the assignment
should go through?  Admittingly, this is a lot of guessing on my side,
as I do not know Emacs internals.

Also: isnʼt your advice “to bind inhibit-quit non-nil only for short
durations” heeded in Thibautʼs suggestions?  It is non-nil more or less
only for the assignment (and a bit for the setup of the let form).

-- 
Felix Dietrich

Re: unwind-protect and inhibit-quit

[Felix Dietrich]

Felix Dietrich <felix.dietrich@sperrhaken.name> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>>> From: Thibaut Verron <thibaut.verron@gmail.com>
>>> Date: Fri, 16 Jul 2021 10:10:36 +0200
>>> Cc: help-gnu-emacs <help-gnu-emacs@gnu.org>
>>> 
>>> > You could use inhibit-quit, but that is generally a bad idea from the
>>> > UI point of view, when invoking potentially long-running functions:
>>> > you are preventing the user from interrupting that long function.  For
>>> > example, suppose the FTP command stalls for some reason.
>>> >
>>> 
>>> Now I'm curious too... Would something like this work?
>>> 
>>>  (let ((inhibit-quit t))
>>>    (setq process
>>>       (let ((inhibit-quit nil))
>>>          (ftp-setup-buffer host file))))
>>
>> It might, but I still suggest to bind inhibit-quit non-nil only for
>> short durations of time and as little as possible.  There be dragons.
>
> I just found the “with-local-quit” macro.  Itʼs comment says that the
> quit-flag “will not be handled until the next function call”.  Doesnʼt
> that imply that once the let form has a value to return, the assignment
> should go through?  Admittingly, this is a lot of guessing on my side,
> as I do not know Emacs internals.

Nevermind, the manual states (I added the ‘*’ as highlighting):
“Eventually, ‘inhibit-quit’ will become ‘nil’ again, such as when its
binding is unwound at the end of a ‘let’ form.  *At that time*, if
‘quit-flag’ is still non-‘nil’, *the requested quit happens
immediately.*”[1]  So there might be more happening at the end of a let.
I guess only the source will tell, and “there be dragons” is the
definitive warning to follow for now.

Footnotes:
[1]  (info "(elisp) Quitting")

-- 
Felix Dietrich

Re: unwind-protect and inhibit-quit

[Stefan Monnier via Users list for the GNU Emacs text editor]

> Now I'm curious too... Would something like this work?
>
>  (let ((inhibit-quit t))
>    (setq process
>       (let ((inhibit-quit nil))
>          (ftp-setup-buffer host file))))

No.  The problem is not in the `setq` itself but in the fact that
a non-local exit from `ftp-setup-buffer` (e.g. because of `C-g`) will
cause `ftp-setup-buffer` not to return the process.

We should devise a more reliable API, tho I'm not completely sure what
it should look like.  Maybe

    (let ((list-of-created-processes nil))
      (unwind-protect ...
        (mapc #'delete-process list-of-created-processes)))

Where the low-level primitives which create processes add them to
`list-of-created-processes`.

But then this gets into trouble when some unrelated code is run during
`...` (e.g. via timers or whatnot) which creates unrelated processes, so
we'd need some extra care to make sure those processes don't get added
to "the same" `list-of-created-processes`.


        Stefan

Re: unwind-protect and inhibit-quit

[Thibaut Verron]

Le ven. 16 juil. 2021 à 17:01, Stefan Monnier via Users list for the
GNU Emacs text editor <help-gnu-emacs@gnu.org> a écrit :
>
> > Now I'm curious too... Would something like this work?
> >
> >  (let ((inhibit-quit t))
> >    (setq process
> >       (let ((inhibit-quit nil))
> >          (ftp-setup-buffer host file))))
>
> No.  The problem is not in the `setq` itself but in the fact that
> a non-local exit from `ftp-setup-buffer` (e.g. because of `C-g`) will
> cause `ftp-setup-buffer` not to return the process.

Ah, thanks for the clarification.

>
> We should devise a more reliable API, tho I'm not completely sure what
> it should look like.  Maybe
>
>     (let ((list-of-created-processes nil))
>       (unwind-protect ...
>         (mapc #'delete-process list-of-created-processes)))
>
> Where the low-level primitives which create processes add them to
> `list-of-created-processes`.

But wouldn't that run into the same sort of race conditions as the
other case? Where a process would be created, but the parent function
would be killed before the process is added to the list?

If not, why not simply add an unwind-protect as tight as possible
around the form actually creating the process?

Re: unwind-protect and inhibit-quit

[Stefan Monnier via Users list for the GNU Emacs text editor]

>> We should devise a more reliable API, tho I'm not completely sure what
>> it should look like.  Maybe
>>
>>     (let ((list-of-created-processes nil))
>>       (unwind-protect ...
>>         (mapc #'delete-process list-of-created-processes)))
>>
>> Where the low-level primitives which create processes add them to
>> `list-of-created-processes`.
>
> But wouldn't that run into the same sort of race conditions as the
> other case? Where a process would be created, but the parent function
> would be killed before the process is added to the list?

No, because the process would (presumably) get added "atomically" to
`list-of-created-processes` as it is created (i.e. either it gets
created and gets added to the list, or it's not created), and so it doesn't
matter what the intermediate code does and returns.

> If not, why not simply add an unwind-protect as tight as possible
> around the form actually creating the process?

Because the above `unwind-protect` needs to be placed at the spot where
one does not need the process any more.


        Stefan

Re: unwind-protect and inhibit-quit

[Felix Dietrich]

Stefan Monnier via Users list for the GNU Emacs text editor
<help-gnu-emacs@gnu.org> writes:

>> Now I'm curious too... Would something like this work?
>>
>>  (let ((inhibit-quit t))
>>    (setq process
>>       (let ((inhibit-quit nil))
>>          (ftp-setup-buffer host file))))
>
> No.  The problem is not in the `setq` itself but in the fact that
> a non-local exit from `ftp-setup-buffer` (e.g. because of `C-g`) will
> cause `ftp-setup-buffer` not to return the process.

But at which point is the ‘quit-flag’ actually handled?  When unwinding
the function, just before returning a value?  The macro
‘with-local-quit’ states that the “quit-flag” “will not be handled until
the next function call”[1].  Could, therefore, a careful and cooperative
“ftp-setup-buffer” ensure that the process is either cleaned-up or
returned?  I am thinking of something like the following:


    (defun ftp-setup-buffer (host file)
      (let (process)
        (condition-case nil
            (progn
              (let ((inhibit-quit t))
                ;; I am assuming that ‘start-process’ does not block for a
                ;; long time.  Does it?  Maybe with a defective drive?
                ;; How about ‘start-file-process’?
                (setq process (start-process …)))
              ;; do more stuff
              process)
          (quit
           ;; I guess another quit could happen here; nest more
           ;; ‘condition-case’?  Itʼs ‘condition-case’ all the way down.
           (and (processp process)
                (kill-process process))
           (setq quit-flag t)
           ;; Propagate quit and make sure an outside handler receives it.
           ;; I do not have a good understanding of this.  I simply copied
           ;; it from the ‘with-local-quit’ macro.
           (eval '(ignore nil))))))


    (let (process)
      (unwind-protect
          (let ((inhibit-quit t))
            (setq process
                  (with-local-quit
                    (ftp-setup-buffer host file)))
            ;; do stuff
            )
        (and (processp process) (kill-process process))))


> We should devise a more reliable API, tho I'm not completely sure what
> it should look like.  Maybe
>
>     (let ((list-of-created-processes nil))
>       (unwind-protect ...
>         (mapc #'delete-process list-of-created-processes)))
>
> Where the low-level primitives which create processes add them to
> `list-of-created-processes`.

How do the low-level C functions to create processes handle quit?  If it
happens inside of them, they should be responsible for killing the
process.  But I am understanding that there is a step in between the
creation of the process object, the return from a process creation
primitive, and the assignment of the process object to a variable.

> But then this gets into trouble when some unrelated code is run during
> `...` (e.g. via timers or whatnot) which creates unrelated processes,
> so we'd need some extra care to make sure those processes don't get
> added to "the same" `list-of-created-processes`.

If I am allowed to spitball here: create another function call to
“claim” a process, and subject unclaimed processes to a similar clean-up
routine as buffers (with ‘clean-buffer-list’):


    (let (process)
      (unwind-protect
          (progn
            (setq process (ftp-setup-buffer host file))
            (claim-process process)
            ;; do stuff
            )
        (and (processp process)
             (kill-process process))))


Add an optional CLAIMED parameter to process creation primitives that
defaults to t in order to not disturb existing code.  Naturally, I have
no idea how to implement that nor the skill to figure this out.


Footnotes:
[1]  Although there is also this information in (info "(elisp)
     Quitting"):

     “Eventually, ‘inhibit-quit’ will become ‘nil’ again, such as when
     its binding is unwound at the end of a ‘let’ form.  At that time,
     if ‘quit-flag’ is still non-‘nil’, the requested quit happens
     immediately.”

-- 
Felix Dietrich

Re: unwind-protect and inhibit-quit

[Stefan Monnier via Users list for the GNU Emacs text editor]

>>>  (let ((inhibit-quit t))
>>>    (setq process
>>>       (let ((inhibit-quit nil))
>>>          (ftp-setup-buffer host file))))
>>
>> No.  The problem is not in the `setq` itself but in the fact that
>> a non-local exit from `ftp-setup-buffer` (e.g. because of `C-g`) will
>> cause `ftp-setup-buffer` not to return the process.
>
> But at which point is the ‘quit-flag’ actually handled?

Doesn't matter.  The problem can occur without any quit.  If an error is
signaled after creation of the process but before `ftp-setup-buffer`
terminates, the same problem will occur.


        Stefan

Re: unwind-protect and inhibit-quit

[Felix Dietrich]

Stefan Monnier via Users list for the GNU Emacs text editor
<help-gnu-emacs@gnu.org> writes:

>>>>  (let ((inhibit-quit t))
>>>>    (setq process
>>>>       (let ((inhibit-quit nil))
>>>>          (ftp-setup-buffer host file))))
>>>
>>> No.  The problem is not in the `setq` itself but in the fact that
>>> a non-local exit from `ftp-setup-buffer` (e.g. because of `C-g`) will
>>> cause `ftp-setup-buffer` not to return the process.
>>
>> But at which point is the ‘quit-flag’ actually handled?
>
> Doesn't matter.  The problem can occur without any quit.  If an error is
> signaled after creation of the process but before `ftp-setup-buffer`
> terminates, the same problem will occur.

But, presumably, if an error occurs in ‘ftp-setup-buffer’ it is still
‘ftp-setup-buffer’ responsibility to handle the error, clean-up the
process, and, then, maybe, propagate/rethrow the signal, no?


    (defun ftp-setup-buffer ()
      (let (process)
        (condition-case err
            (progn
              (setq process (start-process …))
              ;; do stuff that may throw an error
              process ; Could errors still happen here (besides quit)?
                      ; Can quit happen?
              )
          (some-error
           (and (processp process)
                (kill-process process))
           (signal (car err) (cdr err))))))


-- 
Felix Dietrich

Re: unwind-protect and inhibit-quit

[Eli Zaretskii]

> From: Felix Dietrich <felix.dietrich@sperrhaken.name>
> Date: Fri, 16 Jul 2021 23:30:31 +0200
> 
> But at which point is the ‘quit-flag’ actually handled?

When the C-level code of the Lisp machine thinks it's a good time to
do that.  In general, whenever the Lisp evaluation is entered,
whenever a Lisp function is called, and inside any loops that might
run for a long time or might not exit.  There's no magic with handling
quit on the C level, it mast be invoked explicitly.

> The macro ‘with-local-quit’ states that the “quit-flag” “will not be
> handled until the next function call”[1].  Could, therefore, a
> careful and cooperative “ftp-setup-buffer” ensure that the process
> is either cleaned-up or returned?

There's no way to ensure atomicity on the Lisp level, because a Lisp
program has no real control of when quit handling happens.  Only
primitives can do that (and actually do that where we think it's
necessary).

> I am thinking of something like the following:

What problem(s) this is intended to fix/handle?

Re: unwind-protect and inhibit-quit

[Felix Dietrich]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Felix Dietrich <felix.dietrich@sperrhaken.name>
>> Date: Fri, 16 Jul 2021 23:30:31 +0200
>
>> The macro ‘with-local-quit’ states that the “quit-flag” “will not be
>> handled until the next function call”[1].  Could, therefore, a
>> careful and cooperative “ftp-setup-buffer” ensure that the process
>> is either cleaned-up or returned?
>
> There's no way to ensure atomicity on the Lisp level, because a Lisp
> program has no real control of when quit handling happens.  Only
> primitives can do that (and actually do that where we think it's
> necessary).
>
> What problem(s) this is intended to fix/handle?

Still the problem of process littering and, more generally, ensuring
that certain clean up operations are run and values needed for those
clean up operations are not lost in limbo.  I admit that this problem,
that quit is signalled so timed that the return value of a function is
lost in that ethereal space between return and assignment, is an so
unlikely one that one may ignore it in practice—but it piqued my
interest.

Take an ‘ftp-setup-buffer’ function like the following:


    (defun ftp-setup-buffer ()
      (let (process)
        (condition-case err
            (progn
              (setq process (start-process …))
               process ; RETURN
              )
          (quit
           (and (processp process)
                (kill-process process))
           (signal (car err) (cdr err))))))


If you can 1. trust that ‘start-process’ will either return a process
object or, in case of a quit, clean-up whatever intermediary process it
may have created before it propagates the quit, and 2.1. that a quit
received while a RETURN is in process is either signalled inside the
called ‘ftp-setup-buffer’ at a point where in can still be handled by
the ‘condition-case’ or 2.2. only once the returned value has either
been assigned or became irrelevant in the caller because the next form
is being evaluated, then canʼt killing of the process be guaranteed?

This leaves out the issue that another quit may occur inside a handler.
It would be nice if a handler could be started with ‘inhibit-quit’
non-nil.  Let me make something up:


    (condition-case err
        (do-stuff)
      ((:inhibit-quit quit)
       (clean-stuff-up)
       (signal (car err) (cdr err))))


The caller of ‘ftp-setup-buffer’ would not have to worry anymore that
the process object he requested might be lost in transit when a quit
occurs: either ‘ftp-setup-buffer’ has returned a process object, and it
was assigned to the ‘process’ variable, or the process object was killed
already in ‘ftp-setup-buffer’.  (Of course the ‘unwind-protect’ handler
suffers also from the issue that the user may send a quit while it is
being evaluated.)


    (let (process)
      (unwind-protect
          (setq process (ftp-setup-buffer …))
        (and (processp process)
             (kill-process process))))


Alas, it is probably not as “simple” as I imagine: what have I missed?
What do I not know?

-- 
Felix Dietrich

Re: unwind-protect and inhibit-quit

[Eli Zaretskii]

> From: Felix Dietrich <felix.dietrich@sperrhaken.name>
> Date: Sat, 17 Jul 2021 17:46:52 +0200
> 
> > What problem(s) this is intended to fix/handle?
> 
> Still the problem of process littering and, more generally, ensuring
> that certain clean up operations are run and values needed for those
> clean up operations are not lost in limbo.  I admit that this problem,
> that quit is signalled so timed that the return value of a function is
> lost in that ethereal space between return and assignment, is an so
> unlikely one that one may ignore it in practice—but it piqued my
> interest.

The manual tells you there's no easy fix for that.

>     (defun ftp-setup-buffer ()
>       (let (process)
>         (condition-case err
>             (progn
>               (setq process (start-process …))
>                process ; RETURN
>               )
>           (quit
>            (and (processp process)
>                 (kill-process process))
>            (signal (car err) (cdr err))))))
> 
> 
> If you can 1. trust that ‘start-process’ will either return a process
> object or, in case of a quit, clean-up whatever intermediary process it
> may have created before it propagates the quit

This is guaranteed, AFAICT.

> and 2.1. that a quit
> received while a RETURN is in process is either signalled inside the
> called ‘ftp-setup-buffer’ at a point where in can still be handled by
> the ‘condition-case’ or 2.2. only once the returned value has either
> been assigned or became irrelevant in the caller because the next form
> is being evaluated, then canʼt killing of the process be guaranteed?

I lost you here, because this whole code is the code of
ftp-setup-buffer.  So we can only discuss what happens inside that
function.

Basically, the quit will be processed inside setq, and if the user
indeed pressed C-g, the assignment will not happen, and the value of
the variable 'process' will not be a process object.  That is the
problem for which "there's no easy way of fixing it", AFAIU.

> Alas, it is probably not as “simple” as I imagine: what have I missed?
> What do I not know?

I suggest to read the code which implements make-process to see the
details, and then the implementation of setq.  And one more thing: the
time when the user presses C-g and the time when Emacs will react to
it could be far apart.  Pressing C-g basically just sets a flag.