A question regarding sit-for (and while-no-input)

A question regarding sit-for (and while-no-input)

[João Távora]

Hi,

In a couple of my extensions (jsonrpc.el and sly), I am using a
technique for synchronously fetching completion candidates from an
external process while still maintaining high responsiveness to the
user's input.

This is very useful for working with company.el, for example.  If the
user types the first few letters of a completion, the system goes to
work immediately fetching candidates.  If the candidates don't arrive in
time, they are silently discarded.

I am running into a bug in my program, which only happens sometimes and
is hard to reproduce.  I don't know how to fix it.

I hope this simplified snippet from the function jsonrpc-request in
lisp/jsonrpc.el illustrates the problem sufficiently for someone to
provide some hint as to what might be going askew:

   (let ((tag (some-unique-symbol))
         (cancelled nil))
     (catch tag
    ...
      (lambda (...) (unless cancelled (throw tag result-or-error)))
    ...
      (cond (cancel-on-input
             (while (sit-for 30))
             (setq cancelled t)
             `(cancelled ,cancel-on-input-retval))
            (t (while t (accept-process-output nil 30)))))
    ...
   )

'cancel-on-input' is a parameter to jsonrpc-request.  If the caller
provides it as 't', it means he/she wants that call to block as long as
there is no input from the user.  A response from the server before that
happens, which takes the form of a call to anonymous lambda, will also
cause the function to return.

Most of the time, this works flawlessly, as intended.  But the behaviour
I'm witnessing is that (throw tag) sometimes happens after the (catch
tag ...) has been torn down.

What am I missing?? If the catch has been torn down then surely (setq
cancelled t) must have run, right?  Otherwise I would be seeing an error
from sit-for, which I'm not.

Thanks,
João

Re: A question regarding sit-for (and while-no-input)

[João Távora]

João Távora <joaotavora@gmail.com> writes:

> Hi,
>
> In a couple of my extensions (jsonrpc.el and sly), I am using a
> technique for synchronously fetching completion candidates from an
> external process while still maintaining high responsiveness to the
> user's input.
>
> This is very useful for working with company.el, for example.  If the
> user types the first few letters of a completion, the system goes to
> work immediately fetching candidates.  If the candidates don't arrive in
> time, they are silently discarded.
>
> I am running into a bug in my program, which only happens sometimes and
> is hard to reproduce.  I don't know how to fix it.
>
> I hope this simplified snippet from the function jsonrpc-request in
> lisp/jsonrpc.el illustrates the problem sufficiently for someone to
> provide some hint as to what might be going askew:
>
>    (let ((tag (some-unique-symbol))
>          (cancelled nil))
>      (catch tag
>     ...
>       (lambda (...) (unless cancelled (throw tag result-or-error)))
>     ...
>       (cond (cancel-on-input
>              (while (sit-for 30))
>              (setq cancelled t)
>              `(cancelled ,cancel-on-input-retval))
>             (t (while t (accept-process-output nil 30)))))
>     ...
>    )
>
> 'cancel-on-input' is a parameter to jsonrpc-request.  If the caller
> provides it as 't', it means he/she wants that call to block as long as
> there is no input from the user.  A response from the server before that
> happens, which takes the form of a call to anonymous lambda, will also
> cause the function to return.
>
> Most of the time, this works flawlessly, as intended.  But the behaviour
> I'm witnessing is that (throw tag) sometimes happens after the (catch
> tag ...) has been torn down.
>
> What am I missing?? If the catch has been torn down then surely (setq
> cancelled t) must have run, right?  Otherwise I would be seeing an error
> from sit-for, which I'm not.
>
> Thanks,
> João

A quick followup to my question.  I did some more tests and it seems an
using unwind-protect fixes the issue.

   (unwind-protect
     (while (sit-for 30))
    (setq cancelled t)
    ...)

Without it, it seems multiple "sit-for" are entered without ever exiting
properly (properly = "executing through the (setq cancelled t)
instruction").  With it, every sit-for has a corresponding proper exit.

I don't understand the need for the "unwind-protect" and my question
stands.  Is the sit-for call silently quitting or what?  I don't see any
"Quit" in my *Messages* buffer and I'm not pressing C-g at any moment.
I bore down a little bit to keyboard.c's read_char() and it does have
some mentions of quitting, but I honestly don't know if I'm reading it
correctly...

João

Re: A question regarding sit-for (and while-no-input)

[Stefan Monnier]

> A quick followup to my question.  I did some more tests and it seems an
> using unwind-protect fixes the issue.
>
>    (unwind-protect
>      (while (sit-for 30))
>     (setq cancelled t)
>     ...)

Ha, that's exactly what I was about to suggest.

> I don't understand the need for the "unwind-protect" and my question
> stands.  Is the sit-for call silently quitting or what?  I don't see any
> "Quit" in my *Messages* buffer and I'm not pressing C-g at any moment.

Maybe the code is sometimes run within a while-no-input?
Also the sit-for could exit non-locally if some other code
(timer/processfilter) does a `throw`.


        Stefan

Re: A question regarding sit-for (and while-no-input)

[João Távora]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> A quick followup to my question.  I did some more tests and it seems an
>> using unwind-protect fixes the issue.
>>
>>    (unwind-protect
>>      (while (sit-for 30))
>>     (setq cancelled t)
>>     ...)
>
> Ha, that's exactly what I was about to suggest.

Hi again Stefan & the gang,

After a month-long delay, I am back with another question around this
issue.

I recently changed the code to use while-no-input +
accept-process-output instead of sit-for, because the latter would
sometimes hang in combination with company, company-quickhelp, and C-g.
(but it's very difficult to reproduce).

Anyway everything was going fine until I noticed an unexpected slowdown.
I made a little timing macro.

  (defmacro joaot/time (&rest body)
    `(let ((start (current-time)))
       (prog1
           (progn ,@body)
         (let ((msg (format "Took %s seconds"
                            (format-time-string
                             "%S.%3N"
                             (time-subtract (current-time) start)))))
           (if current-prefix-arg
               (insert " ; " msg " and returned ")
             (message msg))))))

These are the results of pressing C-u C-x C-e SPC in quick succession
after each expression.

   (joaot/time
    (while-no-input
      (while t (accept-process-output nil 0.1)))) ; Took 00.201 seconds and returned t
    
   (joaot/time
    (while-no-input
      (while t (accept-process-output nil 30)))) ; Took 03.822 seconds and returned t 
    
   (joaot/time
    (while (sit-for 30))) ; Took 00.126 seconds and returned nil 
    
   (joaot/time
    (while (sit-for 0.1))) ; Took 00.126 seconds and returned nil

I've switched to the first alternative, since it seems to solve the
problem.  I can't explain the long delay on the second one.  Can
someone?

Thanks,
João

Re: A question regarding sit-for (and while-no-input)

[Stefan Monnier]

> These are the results of pressing C-u C-x C-e SPC in quick succession
> after each expression.
>
>    (joaot/time
>     (while-no-input
>       (while t (accept-process-output nil 0.1)))) ; Took 00.201 seconds and returned t
>     
>    (joaot/time
>     (while-no-input
>       (while t (accept-process-output nil 30)))) ; Took 03.822 seconds and returned t 

I don't know why this happens to you, but FWIW, I can't reproduce
it here (using Debian).


        Stefan

Re: A question regarding sit-for (and while-no-input)

[João Távora]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> These are the results of pressing C-u C-x C-e SPC in quick succession
>> after each expression.
>>
>>    (joaot/time
>>     (while-no-input
>>       (while t (accept-process-output nil 0.1)))) ; Took 00.201 seconds and returned t
>>     
>>    (joaot/time
>>     (while-no-input
>>       (while t (accept-process-output nil 30)))) ; Took 03.822 seconds and returned t 
>
> I don't know why this happens to you, but FWIW, I can't reproduce
> it here (using Debian).

Right....  At first I was going to add it happens just on Windows, but
then I tested it on my Ubuntu virtual server and I get the same result.
But after you reported Debian being fine I tested again at home and
indeed I can't reproduce too.

So, reproducible on Windows (Emacs master) + Ubuntu (Emacs 25).  No
problem on Debian (Emacs master).  Going to try a few more combinations,
but, in the meantime, what can be happening here and how can I debug it?

Thanks,
João