run-with-timer vs run-with-idle-timer

run-with-timer vs run-with-idle-timer

[João Távora]

Hi

I suspect this one has been asked before, but still, why does this
return after 1 second:

  (catch 'done
    (run-with-timer 1 nil (lambda () (throw 'done nil)))
    (while t (accept-process-output nil 0.1)))

And this never returns?

  (catch 'done
    (run-with-idle-timer 1 nil (lambda () (throw 'done nil)))
    (while t (accept-process-output nil 0.1)))

Is it because accept-process-output means emacs isn't really idle?
Because it also fails with sleep-for and sit-for...

I need this because this (while t..) spin is a common way to wait for
async conditions (in this case the async condition is flymake kicking
in).

João

Re: run-with-timer vs run-with-idle-timer

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Wed, 09 May 2018 18:34:41 +0100
> 
> Hi
> 
> And this never returns?
> 
>   (catch 'done
>     (run-with-idle-timer 1 nil (lambda () (throw 'done nil)))
>     (while t (accept-process-output nil 0.1)))
> 
> Is it because accept-process-output means emacs isn't really idle?

More accurately, because Emacs is never idle for more than 0.1 sec,
and you requested the timer to run after 1 sec of idleness.

Re: run-with-timer vs run-with-idle-timer

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Wed, 09 May 2018 18:34:41 +0100
> 
>   (catch 'done
>     (run-with-idle-timer 1 nil (lambda () (throw 'done nil)))
>     (while t (accept-process-output nil 0.1)))
> 
> Is it because accept-process-output means emacs isn't really idle?

Actually, yes, waiting in accept-process-output doesn't count as being
idle.  Emacs considers itself idle only when it waits for input in its
main loop.

Re: run-with-timer vs run-with-idle-timer

[João Távora]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: João Távora <joaotavora@gmail.com>
>> Date: Wed, 09 May 2018 18:34:41 +0100
>> 
>>   (catch 'done
>>     (run-with-idle-timer 1 nil (lambda () (throw 'done nil)))
>>     (while t (accept-process-output nil 0.1)))
>> 
>> Is it because accept-process-output means emacs isn't really idle?

> More accurately, because Emacs is never idle for more than 0.1 sec,
> and you requested the timer to run after 1 sec of idleness.> 
>
> ...
>
> Actually, yes, waiting in accept-process-output doesn't count as being
> idle.

Right. Ditto for sit-for and sleep-for, btw

> Emacs considers itself idle only when it waits for input in its
> main loop.

Two follow-up questions (1) is this by design? (2) do interruptions by
processes reset the idle time (I believe they should)?

Re: run-with-timer vs run-with-idle-timer

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Cc: emacs-devel@gnu.org
> Date: Wed, 09 May 2018 19:40:17 +0100
> 
> > Emacs considers itself idle only when it waits for input in its
> > main loop.
> 
> Two follow-up questions (1) is this by design?

I wasn't there when this was designed, but it certainly looks like
that, because accept-process-output calls wait_reading_process_output
in a way that avoids reading from keyboard or the rest of input
events.

> (2) do interruptions by processes reset the idle time (I believe
> they should)?

If by "interruptions" you mean when output some sub-process comes our
way, then no.  The ELisp manual explicitly says:

     Emacs becomes “idle” when it starts waiting for user input, and it
  remains idle until the user provides some input.

Note the "user input" part.  And the very next paragraph says:

     Emacs can do various things while idle: garbage collect, autosave or
  handle data from a subprocess.  But these interludes during idleness do
  not interfere with idle timers, because they do not reset the clock of
  idleness to zero.  An idle timer set for 600 seconds will run when ten
  minutes have elapsed since the last user command was finished, even if
  subprocess output has been accepted thousands of times within those ten
  minutes, and even if there have been garbage collections and autosaves.

Re: run-with-timer vs run-with-idle-timer

[João Távora]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: João Távora <joaotavora@gmail.com>
>> Cc: emacs-devel@gnu.org
>> Date: Wed, 09 May 2018 19:40:17 +0100
>> 
>> > Emacs considers itself idle only when it waits for input in its
>> > main loop.
>> 
>> Two follow-up questions (1) is this by design?
>
> I wasn't there when this was designed, but it certainly looks like
> that, because accept-process-output calls wait_reading_process_output
> in a way that avoids reading from keyboard or the rest of input
> events.

So that suggests that keyboard-idleness should be preserved while
waiting for subprocess output. But it's what you quoted next that
confuses me.

> An idle timer set for 600 seconds will run when ten minutes have
> elapsed since the last user command was finished, even if subprocess
> output has been accepted thousands of times within those ten minutes,
> and even if there have been garbage collections and autosaves.

Doesn't this contradict what you told me first?  I.e doesn't this
contradict the fact that this never returns?

  (catch 'done
    (run-with-idle-timer 600 nil (lambda () (throw 'done nil)))
    (while t (accept-process-output nil 0.1))) ; 6 thousand times

or should the manual be saying "even if subprocesses output has been
non-explictly accepted thousands of times"?

João

Re: run-with-timer vs run-with-idle-timer

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Cc: emacs-devel@gnu.org
> Date: Wed, 09 May 2018 20:15:23 +0100
> 
> > An idle timer set for 600 seconds will run when ten minutes have
> > elapsed since the last user command was finished, even if subprocess
> > output has been accepted thousands of times within those ten minutes,
> > and even if there have been garbage collections and autosaves.
> 
> Doesn't this contradict what you told me first?  I.e doesn't this
> contradict the fact that this never returns?
> 
>   (catch 'done
>     (run-with-idle-timer 600 nil (lambda () (throw 'done nil)))
>     (while t (accept-process-output nil 0.1))) ; 6 thousand times
> 
> or should the manual be saying "even if subprocesses output has been
> non-explictly accepted thousands of times"?

"Subprocess output" in that text doesn't include what
accept-process-output delivers, the text alludes to reading of
subprocess output via filter functions.

Re: run-with-timer vs run-with-idle-timer

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 1285 bytes --]

On Wed, May 9, 2018 at 8:21 PM, Eli Zaretskii <eliz@gnu.org> wrote:

> > From: João Távora <joaotavora@gmail.com>
> > Cc: emacs-devel@gnu.org
> > Date: Wed, 09 May 2018 20:15:23 +0100
> >
> > > An idle timer set for 600 seconds will run when ten minutes have
> > > elapsed since the last user command was finished, even if subprocess
> > > output has been accepted thousands of times within those ten minutes,
> > > and even if there have been garbage collections and autosaves.
> >
> > Doesn't this contradict what you told me first?  I.e doesn't this
> > contradict the fact that this never returns?
> >
> >   (catch 'done
> >     (run-with-idle-timer 600 nil (lambda () (throw 'done nil)))
> >     (while t (accept-process-output nil 0.1))) ; 6 thousand times
> >
> > or should the manual be saying "even if subprocesses output has been
> > non-explictly accepted thousands of times"?
>
> "Subprocess output" in that text doesn't include what
> accept-process-output delivers, the text alludes to reading of
> subprocess output via filter functions.
>

Hmm... There certainly seems to be more than a little room for
confusion, but I couldn't probably think of a better phrasing. Thanks,
and hoping future confused users can find this thread.

João

[-- Attachment #2: Type: text/html, Size: 1985 bytes --]

Re: run-with-timer vs run-with-idle-timer

[Davis Herring]

>> An idle timer set for 600 seconds will run when ten minutes have
>> elapsed since the last user command was finished, even if subprocess
>> output has been accepted thousands of times within those ten minutes,
>> and even if there have been garbage collections and autosaves.
> 
> Doesn't this contradict what you told me first?  I.e doesn't this
> contradict the fact that this never returns?
> 
>    (catch 'done
>      (run-with-idle-timer 600 nil (lambda () (throw 'done nil)))
>      (while t (accept-process-output nil 0.1))) ; 6 thousand times
> 
> or should the manual be saying "even if subprocesses output has been
> non-explictly accepted thousands of times"?

No, because whatever command you used to invoke that code (C-x C-e in 
the trivial case) doesn't finish unless and until the loop finishes.  So 
no time has yet "elapsed since the last user command was finished", 
regardless of anything about subprocesses.

Davis

-- 
This product is sold by volume, not by mass.  If it appears too dense or 
too sparse, it is because mass-energy conversion has occurred during 
shipping.

Re: run-with-timer vs run-with-idle-timer

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 1411 bytes --]

On Wed, May 9, 2018 at 9:00 PM, Davis Herring <herring@lanl.gov> wrote:
>>>
>>> An idle timer set for 600 seconds will run when ten minutes have
>>> elapsed since the last user command was finished, even if subprocess
>>> output has been accepted thousands of times within those ten minutes,
>>> and even if there have been garbage collections and autosaves.
>>
>>
>> Doesn't this contradict what you told me first?  I.e doesn't this
>> contradict the fact that this never returns?
>>
>>    (catch 'done
>>      (run-with-idle-timer 600 nil (lambda () (throw 'done nil)))
>>      (while t (accept-process-output nil 0.1))) ; 6 thousand times
>>
>> or should the manual be saying "even if subprocesses output has been
>> non-explictly accepted thousands of times"?
>
>
> No, because whatever command you used to invoke that code (C-x C-e in the
trivial case) doesn't finish unless and until the loop finishes.  So no
time has yet "elapsed since the last user command was finished", regardless
of anything about subprocesses.

Indeed.

(run-with-timer
 1 nil
 (lambda ()
   (catch 'done
     (run-with-idle-timer 3 nil (lambda () (throw 'done nil)))
     (while t (accept-process-output nil 0.1)))
   (message "ok!")))

Does print "ok". However this blocks typing for 3 seconds. So in a way,
it forces idleness. Gross idea anyway. Wonder what happens in the filter...

João

[-- Attachment #2: Type: text/html, Size: 2209 bytes --]

Re: run-with-timer vs run-with-idle-timer

[Thien-Thi Nguyen]

[-- Attachment #1: Type: text/plain, Size: 757 bytes --]


() João Távora <joaotavora@gmail.com>
() Wed, 09 May 2018 18:34:41 +0100

   (while t..) spin is a common way to wait for async conditions

Why not check the condition directly, i.e., s/t/CONDITION/ like:

 (while CONDITION
   (accept-process-output ...))

That is more precise, no?  Another idea, if there is some
expected traffic, is to use ‘(accept-process-output ...)’
directly as (or as part of) CONDITION, using its return value.

-- 
Thien-Thi Nguyen -----------------------------------------------
 (defun responsep (query)
   (pcase (context query)
     (`(technical ,ml) (correctp ml))
     ...))                              748E A0E8 1CB8 A748 9BFA
--------------------------------------- 6CE4 6703 2224 4C80 7502


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: run-with-timer vs run-with-idle-timer

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 1638 bytes --]

Hi Thien,

I probably should have mentioned I'm setting up an RPC protocol.

I'm using throw/catch because there are lambdas setup inside that
block (but outside the while t). Inside the lambda is the throw.
I then store these lambdas in a global CONTINUATIONS var that the
process filter looks up and calls if appropriate. The infloop breaks then.

It's much harder (and much less efficient at first sight) to do it by
polling
CONDITION, but I may be missing something. Let me know
if you want to look at a concrete example of the lambda strategy.

BTW I learned this strategy reading SLIME, I just simplified it when
lexical
binding came along. It's been working there and in other projects for
a long time.

João


On Thu, May 10, 2018 at 12:46 PM, Thien-Thi Nguyen <ttn@gnu.org> wrote:

>
> () João Távora <joaotavora@gmail.com>
> () Wed, 09 May 2018 18:34:41 +0100
>
>    (while t..) spin is a common way to wait for async conditions
>
> Why not check the condition directly, i.e., s/t/CONDITION/ like:
>
>  (while CONDITION
>    (accept-process-output ...))
>
> That is more precise, no?  Another idea, if there is some
> expected traffic, is to use ‘(accept-process-output ...)’
> directly as (or as part of) CONDITION, using its return value.
>
> --
> Thien-Thi Nguyen -----------------------------------------------
>  (defun responsep (query)
>    (pcase (context query)
>      (`(technical ,ml) (correctp ml))
>      ...))                              748E A0E8 1CB8 A748 9BFA
> --------------------------------------- 6CE4 6703 2224 4C80 7502
>
>


-- 
João Távora

[-- Attachment #2: Type: text/html, Size: 2791 bytes --]

Re: run-with-timer vs run-with-idle-timer

[Thien-Thi Nguyen]

[-- Attachment #1: Type: text/plain, Size: 2777 bytes --]


() João Távora <joaotavora@gmail.com>
() Thu, 10 May 2018 13:28:20 +0100

   I'm using throw/catch because there are lambdas setup inside
   that block (but outside the while t).

   Inside the lambda is the throw.  I then store these lambdas
   in a global CONTINUATIONS var that the process filter looks
   up and calls if appropriate.  The infloop breaks then.

If you can structure whatever ‘catch’es the ‘throw’s to mutate
state visible to CONDITION, then you can use ‘while CONDITION’.
If that state is dynamic (e.g., declared w/ ‘defvar’), you can
can reconfigure the ‘throw’ to simply mutate it directly, and
eliminate the ‘catch’ part (and still use ‘while CONDITION’).

   It's much harder (and much less efficient at first sight) to
   do it by polling CONDITION, but I may be missing something.

Hmmm.

   Let me know if you want to look at a concrete example of the
   lambda strategy.

I'd appreciate seeing a skeleton of the code, because what i
imagine it to be is probably not exactly what it is.  (I could
be misunderstanding your description grossly.)

   BTW I learned this strategy reading SLIME, I just simplified
   it when lexical binding came along.  It's been working there
   and in other projects for a long time.

I confess i've never looked deeply at SLIME innards.  Aside from
some simple toys, my only deep experience w/ async code is w/
gnugo.el -- in func ‘gnugo--q’ you can see:

 (while (process-get proc :incomplete)
   (accept-process-output proc 30))

Here, CONDITION is ‘(process-get proc :incomplete)’ and the
process filter is the one responsible for evaluating:

 (process-put proc :incomplete nil)

The state is not in a variable but in a process object property
(named ‘:incomplete’).  Its value is initialized to ‘t’ prior to
looping.  The change from ‘t’ to ‘nil’ breaks the loop.  Another
precision to note is that the ‘accept-process-output’ 1st arg is
specified.  I believe (but have never actually measured) this
helps performance slightly.

This has also worked for quite a while.

I suppose it's a matter of style.  I feel uncomfortable writing
‘while t’ + ‘throw’ (or similar) in gnugo.el because one of its
design goals (fulfilled, AFAICT, knock on wood! :-D) is to be
able to play multiple games simultaneously (i.e., run multiple
independent child processes), and greedily spinning borks that.

-- 
Thien-Thi Nguyen -----------------------------------------------
 (defun responsep (query)
   (pcase (context query)
     (`(technical ,ml) (correctp ml))
     ...))                              748E A0E8 1CB8 A748 9BFA
--------------------------------------- 6CE4 6703 2224 4C80 7502


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: run-with-timer vs run-with-idle-timer

[João Távora]

Thien-Thi Nguyen <ttn@gnu.org> writes:

> If you can structure whatever ‘catch’es the ‘throw’s to mutate
> state visible to CONDITION, then you can use ‘while CONDITION’.
> If that state is dynamic (e.g., declared w/ ‘defvar’), you can
> can reconfigure the ‘throw’ to simply mutate it directly, and
> eliminate the ‘catch’ part (and still use ‘while CONDITION’).

After thinking a bit about it, you're totally right, and it becomes much
simpler (read below).
>
> I suppose it's a matter of style.

Maybe not, maybe yours is indeed more efficient, since it amounts to
far fewer calls to accept-process-output.

> able to play multiple games simultaneously (i.e., run multiple
> independent child processes), and greedily spinning borks that.

Since from Emacs's perspective we're both blocking, you must mean that
you have less CPU available for the subprocesses, right?

> I'd appreciate seeing a skeleton of the code, because what i
> imagine it to be is probably not exactly what it is.  (I could
> be misunderstanding your description grossly.)

If you haven't yet connected the dots, this is for my new mode eglot.el
from the other thread.  Since it is very new, I think I'm just going to
apply your suggestion (diff below). It works fine and the code becomes
tighter, though it still doesn't solve the original problem that makes
the idle-timer not kick in in the meantime. (Later, I can explain more
about why I need that, and how I'm working around it).

diff --git a/eglot.el b/eglot.el
index aabaf54..d8fa8be 100644
--- a/eglot.el
+++ b/eglot.el
@@ -652,25 +652,20 @@ DEFERRED is passed to `eglot--async-request', which see."
   ;; bad idea, since that might lead to the request never having a
   ;; chance to run, because `eglot--ready-predicates'.
   (when deferred (eglot--signal-textDocument/didChange))
-  (let* ((done (make-symbol "eglot--request-catch-tag"))
-         (res
-          (catch done (eglot--async-request
-                       proc method params
-                       :success-fn (lambda (&rest args)
-                                     (throw done (if (vectorp (car args))
-                                                     (car args) args)))
-                       :error-fn (eglot--lambda
-                                     (&key code message &allow-other-keys)
-                                   (throw done
-                                          `(error ,(format "Oops: %s: %s"
-                                                           code message))))
-                       :timeout-fn (lambda ()
-                                     (throw done '(error "Timed out")))
-                       :deferred deferred)
-                 ;; now spin, baby!
-                 (while t (accept-process-output nil 0.01)))))
-    (when (and (listp res) (eq 'error (car res))) (eglot--error (cadr res)))
-    res))
+  (let (retval)
+    (eglot--async-request
+     proc method params
+     :success-fn (lambda (&rest args)
+                   (setq retval `(done ,(if (vectorp (car args))
+                                            (car args) args))))
+     :error-fn (eglot--lambda (&key code message &allow-other-keys)
+                 (setq retval `(error ,(format "Oops: %s: %s" code message))))
+     :timeout-fn (lambda ()
+                   (setq retval '(error "Timed out")))
+     :deferred deferred)
+    (while (not retval) (accept-process-output nil 30))
+    (when (eq 'error (car retval)) (eglot--error (cadr retval)))
+    (cadr retval)))
 
 (cl-defun eglot--notify (process method params)
   "Notify PROCESS of something, don't expect a reply.e"

Re: run-with-timer vs run-with-idle-timer

[João Távora]

João Távora <joaotavora@gmail.com> writes:

> Thien-Thi Nguyen <ttn@gnu.org> writes:
>
>> If you can structure whatever ‘catch’es the ‘throw’s to mutate
>> state visible to CONDITION, then you can use ‘while CONDITION’.
>> If that state is dynamic (e.g., declared w/ ‘defvar’), you can
>> can reconfigure the ‘throw’ to simply mutate it directly, and
>> eliminate the ‘catch’ part (and still use ‘while CONDITION’).
>
> After thinking a bit about it, you're totally right, and it becomes much
> simpler (read below).

> I suppose it's a matter of style.

Just to clarify, since I've just realised we're talking about two
independent things here:

1. "catch/loop/throw" vs "let/test/loop/set" is indeed a matter of style
   (but you've probably convinced me to prefer the latter).
   
2. It's the short timeout to accept-process-output that I'm supposing
   hurts performance, but it can also be lengthened using the first
   idiom. I just blindly accepted what slime.el was doing since around
   2004. I dug a bit of history (thanks M-x vc-region-history!) and a
   10sec timeout first appeared to replace a timeoutless call. Two years
   later it was (apparently buggily) replaced by a 0.01 second timeout
   when adapting for the new accept-process-output in Emacs 22. That
   value has stuck until today, probably because it works OK, too.

João

Re: run-with-timer vs run-with-idle-timer

[Thien-Thi Nguyen]

[-- Attachment #1: Type: text/plain, Size: 2793 bytes --]


() João Távora <joaotavora@gmail.com>
() Fri, 11 May 2018 11:39:16 +0100

   After thinking a bit about it, you're totally right, and it
   becomes much simpler (read below).

Cool, simpler (if it still works) is better.

   > I suppose it's a matter of style.

   Maybe not, maybe [while CONDITION] is indeed more efficient,
   since it amounts to far fewer calls to accept-process-output.

It's less efficient up until the point CONDITION is no longer
satisfied (since it has to compute/check CONDITION).  But past
that point, it's infinitely more efficient for the reason you
give.

   > able to play multiple games simultaneously (i.e., run
   > multiple independent child processes), and greedily
   > spinning borks that.

   Since from Emacs's perspective we're both blocking, you must
   mean that you have less CPU available for the subprocesses,
   right?

I don't really know what i meant, actually.  I think it was just
my ego trying to sound bigger than my memory would permit.  You
will forgive the fool curmudgeon a lapse here and there, right?
It's been a long time since i used the verb "bork"... /pleading

The truth is that ‘gnugo--q’ is synchronous by design (it says
so right in the comment about ‘:srs’, after all -- but who reads
comments, anyway?!) and does indeed block (since 1st arg to
‘accept-process-output’ is specified) until input appears.

That said, top-level (user interaction) control only passes to
‘gnugo--q’ via ‘run-at-time’, which normally fires after two
seconds, plenty of time (even on this old computer) for Emacs to
move its pointers around.  Too, because the action is captured
in a timer object, Emacs has even more of a free hand to diddle
w/ its scheduling.  To sum up, spinning is what is going on,
true, but "greedily" maybe not so much.  Really, it's more like
bowing to the four corners of the earth (once) than spinning...

   > [...] seeing a skeleton of the code [...]

   If you haven't yet connected the dots, this is for my new
   mode eglot.el from the other thread.

Ah OK.  Sounds interesting!

   Since it is very new, I think I'm just going to apply your
   suggestion (diff below).

Cool.

   It works fine and the code becomes tighter, though it still
   doesn't solve the original problem that makes the idle-timer
   not kick in in the meantime.  (Later, I can explain more
   about why I need that, and how I'm working around it).

OK.

-- 
Thien-Thi Nguyen -----------------------------------------------
 (defun responsep (query)
   (pcase (context query)
     (`(technical ,ml) (correctp ml))
     ...))                              748E A0E8 1CB8 A748 9BFA
--------------------------------------- 6CE4 6703 2224 4C80 7502


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: run-with-timer vs run-with-idle-timer

[Thien-Thi Nguyen]

[-- Attachment #1: Type: text/plain, Size: 1334 bytes --]


() João Távora <joaotavora@gmail.com>
() Fri, 11 May 2018 12:05:48 +0100

   > I suppose it's a matter of style.

   Just to clarify, since I've just realised we're talking about
   two independent things here:

   1. "catch/loop/throw" vs "let/test/loop/set" is indeed a
      matter of style (but you've probably convinced me to
      prefer the latter).

Yeah, that was my meaning.  I look forward to learning why
‘run-with-idle-timer’ is even necessary (my only experience w/
that is in zone.el func ‘zone-when-idle’).

   2. It's the short timeout to accept-process-output that I'm
      supposing hurts performance, but it can also be lengthened
      using the first idiom.

Why does a short timeout hurt performance?  My understanding is:
large timeout => more time for subproc to do its thing => bigger
and fewer chunks of input => less overhead relatively => better
overall performance.

      [timeout value gyrations]

Sounds somewhat cargo-cult.  Just like Emacs, i suppose... :-D

-- 
Thien-Thi Nguyen -----------------------------------------------
 (defun responsep (query)
   (pcase (context query)
     (`(technical ,ml) (correctp ml))
     ...))                              748E A0E8 1CB8 A748 9BFA
--------------------------------------- 6CE4 6703 2224 4C80 7502


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 197 bytes --]