Removing redisplay-dont-pause

Removing redisplay-dont-pause

[Gerd Möllmann]

Eli Zaretskii <eliz@gnu.org> writes:

>> branch: scratch/tty-child-frames
>> commit f62d70f52f4f6b7ed158d618bf790df21f171172
>> Author: Gerd Möllmann <gerd@gnu.org>
>> Commit: Gerd Möllmann <gerd@gnu.org>
>>
>>     Don't pause display for pending input
>>
>>     * src/dispnew.c: Remove display_completed, redisplay_dont_pause,
>>     redisplay-dont-pause was declared obsolete in Emacs 24.  Remove anything
>>     checking pending input, change function signatures accordingly, and so
>>     on.
>>
>>     * src/keyboard.c (read_char): Don't use redisplay_dont_pause.
>>     * src/minibuf.c (read_minibuf): Use new function signatures.
>>     * src/xdisp.c: Don't check display_completed. Use new API.
>>
>>     * lisp/subr.el (redisplay-dont-pause): Remove declaration.
>
> I don't think this kind of change is appropriate.  Feature branches
> should not add/remove features not directly related to the feature
> being developed on the branch.  If we want to remove
> redisplay-dont-pause from Emacs (and I'm not yet sure we do), it
> should be discussed on emacs-devel or in a dedicated bug report, not
> silently installed on the branch.

I guess not many know what this is about, so what is this about?

One feature of the old redisplay was that it stopped updating the
display when it detected that input was pending, so that it could
process that input ASAP.

I kept that feature in my redisplay. As a debugging aid, I added
redisplay-dont-pause with which I could turn this feature on and off,
because pausing the display lead to a lot subtle and difficult to debug
bugs.

ISTR that setting redisplay-dont-pause to t became sort of a secret
semi-popular hack. Hyrum's Law in action, I guess. And 13 years ago or
so, the default of redisplay-dont-pause was changed to t by Eli.

10 years ago, Stefan Monnier explained the why

  +  /* Contrary to expectations, a value of "false" can be detrimental to
  +     responsiveness since aborting a redisplay throws away some of the
  +     work already performed.  It's usually more efficient (and gives
  +     more prompt feedback to the user) to let the redisplay terminate,
  +     and just completely skip the next command's redisplay (which is
  +     done regardless of this setting if there's pending input at the
  +     beginning of the next redisplay).  */
  +  redisplay_dont_pause = true;

And finally, Stefan also

  (make-obsolete-variable 'redisplay-dont-pause nil "24.5")

and removed it from the docs.

I've removed that pausing feature in my fork and thought I'd bring that
to master via scratch/tty-child-frames, but apparently that's
controversial for some reason I don't understand, and I reverted that.
And so, here we are.

Re: Removing redisplay-dont-pause

[Eli Zaretskii]

> From: Gerd Möllmann <gerd.moellmann@gmail.com>
> Cc: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Sun, 17 Nov 2024 06:43:06 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> branch: scratch/tty-child-frames
> >> commit f62d70f52f4f6b7ed158d618bf790df21f171172
> >> Author: Gerd Möllmann <gerd@gnu.org>
> >> Commit: Gerd Möllmann <gerd@gnu.org>
> >>
> >>     Don't pause display for pending input
> >>
> >>     * src/dispnew.c: Remove display_completed, redisplay_dont_pause,
> >>     redisplay-dont-pause was declared obsolete in Emacs 24.  Remove anything
> >>     checking pending input, change function signatures accordingly, and so
> >>     on.
> >>
> >>     * src/keyboard.c (read_char): Don't use redisplay_dont_pause.
> >>     * src/minibuf.c (read_minibuf): Use new function signatures.
> >>     * src/xdisp.c: Don't check display_completed. Use new API.
> >>
> >>     * lisp/subr.el (redisplay-dont-pause): Remove declaration.
> >
> > I don't think this kind of change is appropriate.  Feature branches
> > should not add/remove features not directly related to the feature
> > being developed on the branch.  If we want to remove
> > redisplay-dont-pause from Emacs (and I'm not yet sure we do), it
> > should be discussed on emacs-devel or in a dedicated bug report, not
> > silently installed on the branch.
> 
> I guess not many know what this is about, so what is this about?
> 
> One feature of the old redisplay was that it stopped updating the
> display when it detected that input was pending, so that it could
> process that input ASAP.
> 
> I kept that feature in my redisplay. As a debugging aid, I added
> redisplay-dont-pause with which I could turn this feature on and off,
> because pausing the display lead to a lot subtle and difficult to debug
> bugs.
> 
> ISTR that setting redisplay-dont-pause to t became sort of a secret
> semi-popular hack. Hyrum's Law in action, I guess. And 13 years ago or
> so, the default of redisplay-dont-pause was changed to t by Eli.
> 
> 10 years ago, Stefan Monnier explained the why
> 
>   +  /* Contrary to expectations, a value of "false" can be detrimental to
>   +     responsiveness since aborting a redisplay throws away some of the
>   +     work already performed.  It's usually more efficient (and gives
>   +     more prompt feedback to the user) to let the redisplay terminate,
>   +     and just completely skip the next command's redisplay (which is
>   +     done regardless of this setting if there's pending input at the
>   +     beginning of the next redisplay).  */
>   +  redisplay_dont_pause = true;
> 
> And finally, Stefan also
> 
>   (make-obsolete-variable 'redisplay-dont-pause nil "24.5")
> 
> and removed it from the docs.
> 
> I've removed that pausing feature in my fork and thought I'd bring that
> to master via scratch/tty-child-frames, but apparently that's
> controversial for some reason I don't understand, and I reverted that.
> And so, here we are.

Thanks for the summary.

So what do people think about removing this variable (and the code
supporting it) from Emacs?  In particular, does anyone use that
variable in a non-default nil value?

Re: Removing redisplay-dont-pause

[Eli Zaretskii]

Ping!  Any opinions?  Or should we install Gerd's changes?

> Date: Sun, 17 Nov 2024 09:11:01 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: emacs-devel@gnu.org, monnier@iro.umontreal.ca
> 
> > From: Gerd Möllmann <gerd.moellmann@gmail.com>
> > Cc: Stefan Monnier <monnier@iro.umontreal.ca>
> > Date: Sun, 17 Nov 2024 06:43:06 +0100
> > 
> > Eli Zaretskii <eliz@gnu.org> writes:
> > 
> > >> branch: scratch/tty-child-frames
> > >> commit f62d70f52f4f6b7ed158d618bf790df21f171172
> > >> Author: Gerd Möllmann <gerd@gnu.org>
> > >> Commit: Gerd Möllmann <gerd@gnu.org>
> > >>
> > >>     Don't pause display for pending input
> > >>
> > >>     * src/dispnew.c: Remove display_completed, redisplay_dont_pause,
> > >>     redisplay-dont-pause was declared obsolete in Emacs 24.  Remove anything
> > >>     checking pending input, change function signatures accordingly, and so
> > >>     on.
> > >>
> > >>     * src/keyboard.c (read_char): Don't use redisplay_dont_pause.
> > >>     * src/minibuf.c (read_minibuf): Use new function signatures.
> > >>     * src/xdisp.c: Don't check display_completed. Use new API.
> > >>
> > >>     * lisp/subr.el (redisplay-dont-pause): Remove declaration.
> > >
> > > I don't think this kind of change is appropriate.  Feature branches
> > > should not add/remove features not directly related to the feature
> > > being developed on the branch.  If we want to remove
> > > redisplay-dont-pause from Emacs (and I'm not yet sure we do), it
> > > should be discussed on emacs-devel or in a dedicated bug report, not
> > > silently installed on the branch.
> > 
> > I guess not many know what this is about, so what is this about?
> > 
> > One feature of the old redisplay was that it stopped updating the
> > display when it detected that input was pending, so that it could
> > process that input ASAP.
> > 
> > I kept that feature in my redisplay. As a debugging aid, I added
> > redisplay-dont-pause with which I could turn this feature on and off,
> > because pausing the display lead to a lot subtle and difficult to debug
> > bugs.
> > 
> > ISTR that setting redisplay-dont-pause to t became sort of a secret
> > semi-popular hack. Hyrum's Law in action, I guess. And 13 years ago or
> > so, the default of redisplay-dont-pause was changed to t by Eli.
> > 
> > 10 years ago, Stefan Monnier explained the why
> > 
> >   +  /* Contrary to expectations, a value of "false" can be detrimental to
> >   +     responsiveness since aborting a redisplay throws away some of the
> >   +     work already performed.  It's usually more efficient (and gives
> >   +     more prompt feedback to the user) to let the redisplay terminate,
> >   +     and just completely skip the next command's redisplay (which is
> >   +     done regardless of this setting if there's pending input at the
> >   +     beginning of the next redisplay).  */
> >   +  redisplay_dont_pause = true;
> > 
> > And finally, Stefan also
> > 
> >   (make-obsolete-variable 'redisplay-dont-pause nil "24.5")
> > 
> > and removed it from the docs.
> > 
> > I've removed that pausing feature in my fork and thought I'd bring that
> > to master via scratch/tty-child-frames, but apparently that's
> > controversial for some reason I don't understand, and I reverted that.
> > And so, here we are.
> 
> Thanks for the summary.
> 
> So what do people think about removing this variable (and the code
> supporting it) from Emacs?  In particular, does anyone use that
> variable in a non-default nil value?
> 
> 

Re: Removing redisplay-dont-pause

[Stefan Monnier]

> Ping!  Any opinions?  Or should we install Gerd's changes?

I'm in favor of getting rid of `redisplay-dont-pause`, as you can guess.
🙂


        Stefan

Re: Removing redisplay-dont-pause

[Mattias Engdegård]

30 nov. 2024 kl. 10.58 skrev Eli Zaretskii <eliz@gnu.org>:

> Any opinions?  Or should we install Gerd's changes?

What's the downside, worse interactive performance on 2400 bit/s serial terminals?

Re: Removing redisplay-dont-pause

[Eli Zaretskii]

> From: Mattias Engdegård <mattias.engdegard@gmail.com>
> Date: Sun, 1 Dec 2024 10:46:39 +0100
> Cc: emacs-devel@gnu.org,
>  gerd.moellmann@gmail.com,
>  monnier@iro.umontreal.ca
> 
> 30 nov. 2024 kl. 10.58 skrev Eli Zaretskii <eliz@gnu.org>:
> 
> > Any opinions?  Or should we install Gerd's changes?
> 
> What's the downside, worse interactive performance on 2400 bit/s serial terminals?

I don't think the terminal connection speed is the main issue.  The
main issue is do we want the ability to interrupt redisplay if it
takes too long and new input is available.  We do know that redisplay
is sometimes slow.  One example is scrolling through a large buffer,
when JIT fontification takes too long, and the user already pressed
C-v.

But if no one customizes this variable to a nil value, at least in
some cases, then keeping it is not important.

Re: Removing redisplay-dont-pause

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

> But if no one customizes this variable to a nil value, at least in
> some cases, then keeping it is not important.

It is hard to customize a variable that is not documented anywhere.
The problem with redisplay hangs keeps surfacing again and again, so I
do expect that a number of people will try to use it as soon as they
learn that such variable exists.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Removing redisplay-dont-pause

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@posteo.net>
> Cc: Mattias Engdegård <mattias.engdegard@gmail.com>,
>  emacs-devel@gnu.org,
>  gerd.moellmann@gmail.com, monnier@iro.umontreal.ca
> Date: Sun, 01 Dec 2024 10:43:49 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > But if no one customizes this variable to a nil value, at least in
> > some cases, then keeping it is not important.
> 
> It is hard to customize a variable that is not documented anywhere.

??? The variable has a doc string, and shows up in apropos commands.
So "not documented anywhere" is inaccurate, to say the least.

Moreover, my question is more to those who learned about the variable
back when it was mentioned in the manual as well.  (It was deleted
from the manual in Emacs 25.)

> The problem with redisplay hangs keeps surfacing again and again, so I
> do expect that a number of people will try to use it as soon as they
> learn that such variable exists.

The hangs you have in mind cannot be helped by this variable.

Re: Removing redisplay-dont-pause

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> It is hard to customize a variable that is not documented anywhere.
>
> ??? The variable has a doc string, and shows up in apropos commands.
> So "not documented anywhere" is inaccurate, to say the least.

Right. I meant hard to find.

> Moreover, my question is more to those who learned about the variable
> back when it was mentioned in the manual as well.  (It was deleted
> from the manual in Emacs 25.)

Sorry, I missed that part.

>> The problem with redisplay hangs keeps surfacing again and again, so I
>> do expect that a number of people will try to use it as soon as they
>> learn that such variable exists.
>
> The hangs you have in mind cannot be helped by this variable.

I was replying about your example with scrolling. I also tested setting
this variable when typing in an Org buffer with many large invisible
regions, and setting the variable does help typing speed.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Removing redisplay-dont-pause

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@posteo.net>
> Cc: mattias.engdegard@gmail.com, emacs-devel@gnu.org,
>  gerd.moellmann@gmail.com, monnier@iro.umontreal.ca
> Date: Sun, 01 Dec 2024 14:49:56 +0000
> 
> >> The problem with redisplay hangs keeps surfacing again and again, so I
> >> do expect that a number of people will try to use it as soon as they
> >> learn that such variable exists.
> >
> > The hangs you have in mind cannot be helped by this variable.
> 
> I was replying about your example with scrolling. I also tested setting
> this variable when typing in an Org buffer with many large invisible
> regions, and setting the variable does help typing speed.

I imagined that.

Most of the time spent by redisplay in those cases is in the part that
isn't interrupted when this variable is nil, because it is consulted
when all the invisible regions were already scanned.  I don't
understand how it could help typing speed, unless typing in those
cases has dramatic effect on what is shown on the glass (as opposed to
what typing usually does: affects at most one or two screen lines).

Re: Removing redisplay-dont-pause

[Stefan Monnier]

>>> The problem with redisplay hangs keeps surfacing again and again, so I
>>> do expect that a number of people will try to use it as soon as they
>>> learn that such variable exists.
>>
>> The hangs you have in mind cannot be helped by this variable.
>
> I was replying about your example with scrolling. I also tested setting
> this variable when typing in an Org buffer with many large invisible
> regions, and setting the variable does help typing speed.

If you can come up with a reproducible recipe, I'd be interested to see
it, since in most cases, `redisplay-dont-pause == nil` tends to make
things worse, IME.

The way I think about it is as follows:

Assume the incoming events come every T seconds and take T1 seconds to
run, and the subsequent redisplay takes time T2+T3 (where T2 is the time
in the middle where `redisplay-dont-pause` is consulted).

Then you have the following scenarios:

- If T > T1+T2+T3, then all is well.
- If T < T1, then we plain and simply can't keep up at all, the
  redisplay is just not run at all and incoming events keep accumulating
  as we fall ever further behind the incoming events until the incoming
  events stop (at which point Emacs will only refresh its redisplay
  after it has finally caught up with the buffered input).

- If T1 < T < T1 + T2, then Emacs will keep up with the input (i.e. when
  the events stop coming, the display is refreshed more or less
  instantly) but with `redisplay-dont-pause` set to nil it will not
  refresh the display at all until the incoming events stop because
  every time it tries to redisplay it gets interrupted by some
  new input.
  If OTOH `redisplay-dont-pause` is set to t, then instead of aborting
  the redisplay, we finish it which slows us down (gets us a bit further
  behind the flow of input), but does give feedback to the user, and
  for the next few inputs we'll simply perform no redisplay at all until
  we caught up with the input, at which point we will again perform
  a redisplay, so in the end we get a redisplay after every N events
  where N = (T2 + T3) / (T - T1).
- If T1 + T2 < T < T1 + T2+ T3, the difference between the settings of
  `redisplay-dont-pause` is smaller (in both cases, the display is
  refreshed every once in a while), but there will still be fewer
  redisplays when `redisplay-dont-pause` is nil because we waste time
  starting some redisplays which we later abort.

Of course, reality is more complex (e.g. the time T2 is not a constant,
e.g. it can occasionally be proportional to the number of events since
the last redisplay), but at least in my tests where we scroll down
a buffer with a high repeat rate, the behavior matched that naïve model
closely enough.


        Stefan

Re: Removing redisplay-dont-pause

[Gerd Möllmann]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>>> The problem with redisplay hangs keeps surfacing again and again, so I
>>>> do expect that a number of people will try to use it as soon as they
>>>> learn that such variable exists.
>>>
>>> The hangs you have in mind cannot be helped by this variable.
>>
>> I was replying about your example with scrolling. I also tested setting
>> this variable when typing in an Org buffer with many large invisible
>> regions, and setting the variable does help typing speed.
>
> If you can come up with a reproducible recipe, I'd be interested to see
> it, since in most cases, `redisplay-dont-pause == nil` tends to make
> things worse, IME.
>
> The way I think about it is as follows:
>
> Assume the incoming events come every T seconds and take T1 seconds to
> run, and the subsequent redisplay takes time T2+T3 (where T2 is the time
> in the middle where `redisplay-dont-pause` is consulted).

I think you meant to say that T3 is the phase at the end where
redisplay-dont-pause is checked, the update phase that writes to the
screen. T2 would then be the phase where the glyphs are produced.

>
> Then you have the following scenarios:
>
> - If T > T1+T2+T3, then all is well.
> - If T < T1, then we plain and simply can't keep up at all, the
>   redisplay is just not run at all and incoming events keep accumulating
>   as we fall ever further behind the incoming events until the incoming
>   events stop (at which point Emacs will only refresh its redisplay
>   after it has finally caught up with the buffered input).

The direct_output_for_insert optimization was though for that, ensuring
that at least the line one types in is made visible on the screen.

This comes from a time where 1200 or 2400 baud connections existed, like
my first CompuServe connection in the 80s :-). That's ca, 300 bytes max.
for 2400 baud. And a 80x25 screen has 2000 bytes. So, T3 could be much
larger than T2, and my explanation for the fact that input was not
checked during T2: it's not where the problem was.

> - If T1 < T < T1 + T2, then Emacs will keep up with the input (i.e. when
>   the events stop coming, the display is refreshed more or less
>   instantly) but with `redisplay-dont-pause` set to nil it will not
>   refresh the display at all until the incoming events stop because
>   every time it tries to redisplay it gets interrupted by some
>   new input.
>   If OTOH `redisplay-dont-pause` is set to t, then instead of aborting
>   the redisplay, we finish it which slows us down (gets us a bit further
>   behind the flow of input), but does give feedback to the user, and
>   for the next few inputs we'll simply perform no redisplay at all until
>   we caught up with the input, at which point we will again perform
>   a redisplay, so in the end we get a redisplay after every N events
>   where N = (T2 + T3) / (T - T1).
> - If T1 + T2 < T < T1 + T2+ T3, the difference between the settings of
>   `redisplay-dont-pause` is smaller (in both cases, the display is
>   refreshed every once in a while), but there will still be fewer
>   redisplays when `redisplay-dont-pause` is nil because we waste time
>   starting some redisplays which we later abort.
>
> Of course, reality is more complex (e.g. the time T2 is not a constant,
> e.g. it can occasionally be proportional to the number of events since
> the last redisplay), but at least in my tests where we scroll down
> a buffer with a high repeat rate, the behavior matched that naïve model
> closely enough.
>
>
>         Stefan

Agree, and I'd also be interested when exactly redisplay-dont-pause nil
makes a difference today.

Re: Removing redisplay-dont-pause

[Gerd Möllmann]

Mattias Engdegård <mattias.engdegard@gmail.com> writes:

> 30 nov. 2024 kl. 10.58 skrev Eli Zaretskii <eliz@gnu.org>:
>
>> Any opinions?  Or should we install Gerd's changes?
>
> What's the downside, worse interactive performance on 2400 bit/s serial terminals?

Maybe it's worth mentioning that a part of the story of slow terminals,
a display optimization, was already removed in 25, I think:
direct_output_for_insert in dispnew.c. That tried to optimize the case
of single-char input, without being interrupted by pending input.

Good riddance, from my POV, it's all complicated enough :-).

Re: Removing redisplay-dont-pause

[Eli Zaretskii]

> From: Gerd Möllmann <gerd.moellmann@gmail.com>
> Cc: Eli Zaretskii <eliz@gnu.org>,  emacs-devel@gnu.org,
>   monnier@iro.umontreal.ca
> Date: Sun, 01 Dec 2024 16:28:42 +0100
> 
> Mattias Engdegård <mattias.engdegard@gmail.com> writes:
> 
> > 30 nov. 2024 kl. 10.58 skrev Eli Zaretskii <eliz@gnu.org>:
> >
> >> Any opinions?  Or should we install Gerd's changes?
> >
> > What's the downside, worse interactive performance on 2400 bit/s serial terminals?
> 
> Maybe it's worth mentioning that a part of the story of slow terminals,
> a display optimization, was already removed in 25, I think:
> direct_output_for_insert in dispnew.c. That tried to optimize the case
> of single-char input, without being interrupted by pending input.

It was actually removed in Emacs 24, as part of support for bidi
reordering (which is incompatible with it).