Temporarily disable `timer-event-handler'

Temporarily disable `timer-event-handler'

[Alexander Shukaev]

Hi,

I've recently discovered the dark side of Emacs: that timer events run 
by `timer-event-handler' can be triggered either at `top-level' or at 
`recursive-edit' or as part of `sit-for' (see `input-pending-p') or 
`accept-process-output' or `message' (which triggers `redisplay' and 
subsequently C function `redisplay_internal' that invokes Lisp 
interpreter machine yet again and hence potentially 
`timer-event-handler') or maybe even more.

With such plethora of possibilities for `timer-event-handler' to run 
especially in a nested manner one could even observe an interesting 
peculiarity:

- Event `A' scheduled to run ASAP.
- Event `B' scheduled to run ASAP.
- The expectation is that `B' strictly runs after `A' finishes execution.
- Well, if `A' calls `message', then according to the above, `A' could 
indirectly run `timer-event-handler' (nested) which, as a result, will 
run `B' now essentially somewhere in the middle of `A'.
- Clearly, unless taken care of, this may result in side effects and 
nasty bugs, which are difficult to track down.

Another possible side effect can be related to e.g. `let'-binding, when 
during an execution of some function some `let'-binding is performed, 
but `redisplay' is being triggered and some timer event occurs under 
that `let'-binding, which unfortunately affects it in a buggy unexpected 
way.  Also difficult to understand and hunt down.

Now, first of all, do I understand correctly that the recommended way to 
temporarily prevent timer events from happening is to `let'-bind both 
`timer-list' and `timer-idle-list' (similar to how TRAMP does it e.g. in 
`tramp-accept-process-output')?  How about a stock macro for this?

Secondly, does the above explanation, pitfall examples, and recipes to 
temporarily disable timer events in the middle of Lisp execution appear 
anywhere in the documentation?  I suspect that most users are not aware 
of this complicated design.

Finally, what's the motivation behind this design?  This looks fragile 
and error-prone to run some arbitrary code in the middle of execution of 
another code without separating their "stacks" (environment scopes), and 
even then their global side effects might interfere.  Why not only allow 
timer events to run at "safer" points of execution?  E.g. no nested 
timer events, only at `top-level', never in `redisplay', also allowed 
in-between some blessed hooks executions?

RE: Temporarily disable `timer-event-handler'

[Drew Adams]

Please don't post the same question to both help-gnu-emacs and emacs-devel.  Please pick one.  Thx.

Re: Temporarily disable `timer-event-handler'

[Michael Albinus]

Alexander Shukaev <emacs@Alexander.Shukaev.name> writes:

> Hi,

Hi Alexander,

> Now, first of all, do I understand correctly that the recommended way
> to temporarily prevent timer events from happening is to `let'-bind
> both `timer-list' and `timer-idle-list' (similar to how TRAMP does it
> e.g. in `tramp-accept-process-output')?  How about a stock macro for
> this?

See also https://bugs.gnu.org/29735

Unfortunately, the discussion didn't yield a reasonable answer. Instead,
it was focussed on Tramp internals which are tangent to the initial request.

Best regards, Michael.

Re: Temporarily disable `timer-event-handler'

[Stefan Monnier]

> - Event `A' scheduled to run ASAP.
> - Event `B' scheduled to run ASAP.
> - The expectation is that `B' strictly runs after `A' finishes execution.

There's the rub: currently, there's no such guarantee.  The only
"guarantee" we provide is that the processing of event A will start
before the processing of event B.

[ Note: "event" here can be a timer firing or a process output.  ]

> - Well, if `A' calls `message', then according to the above, `A' could
>   indirectly run `timer-event-handler' (nested) which, as a result, will 
>   run `B' now essentially somewhere in the middle of `A'.

If A and B are unrelated, there's no problem.
If A and B are related, there could indeed be a problem.
And notice that event B could even have been created by the processing of A.
But there can also be cases where A does want/need B to be processed
before A ends.

[ Another related situation is when A is a repeated event and a second
  A fires before the first one is done processing.  ]

> - Clearly, unless taken care of, this may result in side effects and nasty
>  bugs, which are difficult to track down.

Yup.

> Another possible side effect can be related to e.g. `let'-binding, when
>  during an execution of some function some `let'-binding is performed, 
> but `redisplay' is being triggered and some timer event occurs under that
> `let'-binding, which unfortunately affects it in a buggy unexpected way.
> Also difficult to understand and hunt down.

Oh yes, dynamically-bound vars can make that even more interesting.
Process-filters and timer events (and jit-lock, and the debugger) should
ideally be run in their own thread (and hence not affected by
dynamic-bindings in the interrupted threads).  Existing code does
occasionally depend on the current behavior, tho (e.g. in
`jit-lock-deferred-fontify` we bind `jit-lock-defer-timer` around
a call to `redisplay`), so fixing this is not completely trivial.

> Now, first of all, do I understand correctly that the recommended way to
> temporarily prevent timer events from happening is to `let'-bind both 
> `timer-list' and `timer-idle-list' (similar to how TRAMP does it e.g. in
> `tramp-accept-process-output')?  How about a stock macro for this?

The recommended way is to find a solution that doesn't involve
preventing timers from firing, so a stack macro for it doesn't sound
right (e.g. in the case of Tramp, the discussion in bug#29735 seems to
indicate that Michael agrees that disabling timers isn't "right" but
it's just what he's using so far because he wasn't able to track down
the bug in the "right" version of the code).

> I suspect that most users are not aware of this complicated design.

I don't think the design is complicated.  It's the resulting behavior
that is (arguably because the design was too naive).

> Finally, what's the motivation behind this design?  This looks fragile and
> error-prone to run some arbitrary code in the middle of execution of another
> code without separating their "stacks" (environment scopes),

It's a direct consequence of the implementation, rather than the result
of design, AFAICT.

> Why not only allow timer events to run at "safer" points of execution?
> E.g. no nested timer events,

That's probably a good idea, yes.
Probably worth giving it a try so if/what breaks.

> never in `redisplay',

Agreed.  Feel free to consider it as a bug and hence report it when
it happens.


        Stefan

Re: Temporarily disable `timer-event-handler'

[Eli Zaretskii]

[I removed help-gnu-emacs from the CC list.]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Mon, 03 Feb 2020 09:01:47 -0500
> Cc: help-gnu-emacs@gnu.org, emacs-devel@gnu.org
> 
> > Finally, what's the motivation behind this design?  This looks fragile and
> > error-prone to run some arbitrary code in the middle of execution of another
> > code without separating their "stacks" (environment scopes),
> 
> It's a direct consequence of the implementation, rather than the result
> of design, AFAICT.

There's also the need to run a timer as close to its expected time as
possible, so waiting for any Lisp to return might delay timers too
much.

Re: Temporarily disable `timer-event-handler'

[Alexander Shukaev]

On 03/02/2020 16:44, Eli Zaretskii wrote:
> [I removed help-gnu-emacs from the CC list.]
> 
>> From: Stefan Monnier <monnier@iro.umontreal.ca>
>> Date: Mon, 03 Feb 2020 09:01:47 -0500
>> Cc: help-gnu-emacs@gnu.org, emacs-devel@gnu.org
>>
>>> Finally, what's the motivation behind this design?  This looks fragile and
>>> error-prone to run some arbitrary code in the middle of execution of another
>>> code without separating their "stacks" (environment scopes),
>>
>> It's a direct consequence of the implementation, rather than the result
>> of design, AFAICT.
> 
> There's also the need to run a timer as close to its expected time as
> possible, so waiting for any Lisp to return might delay timers too
> much.
> 

Right, the current implementation looks more like coroutines, where 
certain built-in functions yield for timers to run on occasion.  This is 
indeed likely to be more performent and even useful at times, i.e. 
example from Stefan

> But there can also be cases where A does want/need B to be processed
> before A ends.

Nonetheless, given the availability of dynamic binding, I find this 
error-prone.  Is there currently a way in Emacs Lisp to run a function 
within a separate stack, i.e. free of any "user-defined" dynamic 
bindings?  I would imagine that this may be the right solution to fire 
timer events in this manner as it then does not matter whether they run 
in the middle of other routines.

Additionally, I believe the solution to fulfill

> The expectation is that `B' strictly runs after `A' finishes execution.

is also missing.  Some sort of macro (aka `ignore-timers') which 
essentially forbids yielding during the execution of the enclosed body, 
while `let'-binding both `timer-list' and `timer-idle-list' as a naive 
implementation of such does not really scale, since any inner calls to 
`cancel-timer' or otherwise scheduling other (new) timer events would 
have bad side effects.  Otherwise, should be something simple and 
similar to `inhibit-redisplay', aka `inhibit-timers' and 
`inhibit-idle-timers'.

Re: Temporarily disable `timer-event-handler'

[Stefan Monnier]

>> But there can also be cases where A does want/need B to be processed
>> before A ends.
>
> Nonetheless, given the availability of dynamic binding, I find this
> error-prone.  Is there currently a way in Emacs Lisp to run a function 
> within a separate stack,

Yes and no.  Until "recently" the answer was clearly no, and now it's
kinda possible but not in a really nice way:
- using threads (which are not always available, so it can only be used conditionally).
- using `backtrace-eval` which will temporarily undo part of the
  currently installed dynamic bindings (as well as
  `with-current-buffer`s and `save-excursion`s, IIRC) before evaluating
  an expression.

> Additionally, I believe the solution to fulfill
>> The expectation is that `B' strictly runs after `A' finishes execution.
> is also missing.

Yup!

> Some sort of macro (aka `ignore-timers') which essentially
> forbids yielding during the execution of the enclosed body,

No, remember: Emacs is a Lisp machine, aka a kind of OS, so this would
mean that application A hogs the whole machine just because it wants to
prevent B from running.

Instead we need something more specific which lets A delay B without
delaying other, unrelated, events.


        Stefan

Re: Temporarily disable `timer-event-handler'

[Alexander Shukaev]

>> Some sort of macro (aka `ignore-timers') which essentially
>> forbids yielding during the execution of the enclosed body,
> 
> No, remember: Emacs is a Lisp machine, aka a kind of OS, so this would
> mean that application A hogs the whole machine just because it wants to
> prevent B from running.
> 
> Instead we need something more specific which lets A delay B without
> delaying other, unrelated, events.
> 

In a fine-grained scenario where one knows exactly which other timer 
(event) to delay, yes.  I was more into temporarily inhibiting all the 
timers (for whatever reason may be needed too, e.g. what Michael faces). 
  Would be cool to support both of course.

Re: Temporarily disable `timer-event-handler'

[Stefan Monnier]

>> Instead we need something more specific which lets A delay B without
>> delaying other, unrelated, events.
> In a fine-grained scenario where one knows exactly which other timer (event)
> to delay, yes.  I was more into temporarily inhibiting all the timers (for
> whatever reason may be needed too, e.g. what Michael faces).

I know you are, but this would be fundamentally wrong, just like it
would be wrong for your OS to let Emacs prevent all other applications
on your machine from running.


        Stefan

Re: Temporarily disable `timer-event-handler'

[Eli Zaretskii]

> Cc: Eli Zaretskii <eliz@gnu.org>, emacs-devel@gnu.org
> From: Alexander Shukaev <emacs@Alexander.Shukaev.name>
> Date: Wed, 5 Feb 2020 02:21:37 +0100
> 
> I was more into temporarily inhibiting all the 
> timers (for whatever reason may be needed too, e.g. what Michael faces). 
>   Would be cool to support both of course.

Micheal actually wanted to inhibit all timers but hos own, AFAIU.

And before we seriously discuss the possibility of inhibiting _all_
the timers, I'd suggest to make sure we have a clear understanding how
many features we take for granted run off timers.  Starting from
cursor-blinking and undo, for example.

Re: Temporarily disable `timer-event-handler'

[Michael Albinus]

Eli Zaretskii <eliz@gnu.org> writes:

>> I was more into temporarily inhibiting all the
>> timers (for whatever reason may be needed too, e.g. what Michael faces).
>>   Would be cool to support both of course.
>
> Micheal actually wanted to inhibit all timers but hos own, AFAIU.

More precisely, I wanted to make Tramp primitives being protected
against other Tramp primitives called from timers, process sentinels and
filters, etc. Meanwhile, I found another approach protecting Tramp
against shooting itself into the knee; I don't insist any longer for
disabling all timers for the benefit of Tramp.

Best regards, Michael.