Aborting display. Is this possible?

Aborting display. Is this possible?

[Alan Mackenzie]

Hello, Emacs.

A lisp program can influence an ongoing display operation by setting
redisplay-dont-pause to nil.  When input arrives, the displaying is
immediately paused.

However, at the next opportunity, the displaying carries on where it
left off, rather than being aborted.  It would be nice to arrange for
such abortion to happen, leaving the partly completed display visible on
the user's video.

Why?  Better to handle a user auto-repeating on the PageDown key.  On my
system, auto-repeat is around 35-40 characters per second, on a window
with 65 lines.  In my favourite mode, display can not keep up.  If I set
redisplay-dont-pause to nil and hold down PageDown, what the user sees
is frozen for several seconds whilst display determines the display of
every intermediate buffer position (which isn't going to get displayed
anyway) before finally reaching point.

Would it be possible to enhance display such that incoming input would
abort the current display operation?

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Aborting display. Is this possible?

[David Kastrup]

Alan Mackenzie <acm@muc.de> writes:

> Why?  Better to handle a user auto-repeating on the PageDown key.  On
> my system, auto-repeat is around 35-40 characters per second, on a
> window with 65 lines.  In my favourite mode, display can not keep up.
> If I set redisplay-dont-pause to nil and hold down PageDown, what the
> user sees is frozen for several seconds whilst display determines the
> display of every intermediate buffer position (which isn't going to
> get displayed anyway) before finally reaching point.
>
> Would it be possible to enhance display such that incoming input would
> abort the current display operation?

How would Emacs figure out just how much "a screenful" of text is when
it is not allowed to lay out the screen?  Where is PageDown going to end
up?

One possible way to deal with this is not to read characters while
redisplay is being processed: a reasonably well-programmed input stack
will throttle autorepeat while it has unconsumed characters.

Of course, that may not be feasible depending on how Emacs does C-g
processing.

-- 
David Kastrup

Re: Aborting display. Is this possible?

[Eli Zaretskii]

> Date: Sun, 19 Oct 2014 14:17:13 +0000
> From: Alan Mackenzie <acm@muc.de>
> 
> A lisp program can influence an ongoing display operation by setting
> redisplay-dont-pause to nil.  When input arrives, the displaying is
> immediately paused.

It is not paused, it is aborted.  Emacs cannot "pause" redisplay and
then "resume" it.

> However, at the next opportunity, the displaying carries on where it
> left off, rather than being aborted.

I don't understand what you mean by "carries on where it left off".
Emacs doesn't record where redisplay "left off", and cannot resume an
interrupted redisplay cycle.  It simply starts redisplay anew on the
next opportunity.

> Why?  Better to handle a user auto-repeating on the PageDown key.  On my
> system, auto-repeat is around 35-40 characters per second, on a window
> with 65 lines.  In my favourite mode, display can not keep up.  If I set
> redisplay-dont-pause to nil and hold down PageDown, what the user sees
> is frozen for several seconds whilst display determines the display of
> every intermediate buffer position (which isn't going to get displayed
> anyway) before finally reaching point.

That's not what happens.  What does happen is that redisplay _tries_
several times to do its job, right after each PageDown keypress, each
time starting with the value of point it sees, then abandoning that
attempt because input arrives before it could finish.

Emacs never examines portions of the buffer that don't need to be on
the screen (or a few lines of text above or below that).  It always
starts with the current value of point, and works from that.  So
intermediate buffer positions will never be examined.

> Would it be possible to enhance display such that incoming input would
> abort the current display operation?

It does already.  Emacs simply doesn't know any better.

Re: Aborting display. Is this possible?

[Eli Zaretskii]

> From: David Kastrup <dak@gnu.org>
> Date: Sun, 19 Oct 2014 16:32:38 +0200
> 
> One possible way to deal with this is not to read characters while
> redisplay is being processed

We already do that, since both keyboard input and redisplay happen in
the same thread.

The problem is that auto-repeat leaves just enough time between the
repeated keystrokes for redisplay to kick in.  If auto-repeat would
send many keys in one large block, we would process them all before
attempting to redisplay.

Re: Aborting display. Is this possible?

[Alan Mackenzie]

Hi, Eli.

On Sun, Oct 19, 2014 at 05:50:52PM +0300, Eli Zaretskii wrote:
> > Date: Sun, 19 Oct 2014 14:17:13 +0000
> > From: Alan Mackenzie <acm@muc.de>

> > A lisp program can influence an ongoing display operation by setting
> > redisplay-dont-pause to nil.  When input arrives, the displaying is
> > immediately paused.

> It is not paused, it is aborted.  Emacs cannot "pause" redisplay and
> then "resume" it.

Ah.  OK.

> > However, at the next opportunity, the displaying carries on where it
> > left off, rather than being aborted.

> I don't understand what you mean by "carries on where it left off".
> Emacs doesn't record where redisplay "left off", and cannot resume an
> interrupted redisplay cycle.  It simply starts redisplay anew on the
> next opportunity.

I asked for it, it seems I've got it already.

> > Why?  Better to handle a user auto-repeating on the PageDown key.  On my
> > system, auto-repeat is around 35-40 characters per second, on a window
> > with 65 lines.  In my favourite mode, display can not keep up.  If I set
> > redisplay-dont-pause to nil and hold down PageDown, what the user sees
> > is frozen for several seconds whilst display determines the display of
> > every intermediate buffer position (which isn't going to get displayed
> > anyway) before finally reaching point.

> That's not what happens.  What does happen is that redisplay _tries_
> several times to do its job, right after each PageDown keypress, each
> time starting with the value of point it sees, then abandoning that
> attempt because input arrives before it could finish.

So if 35 PageDowns are received in a second, it starts redisplay after
each and every one.  How much time does it take before checking for
pending input?

Could my problem be that this time is short enough that that no
intermediate screen has time to display a fragment, but long enough that
there's a significant pause when the user finally takes her finger of
PageDown?

What is stopping a fragment of each intermediate screen (or of some of
them) being displayed?  Could it be that the display engine calculates an
entire screenful of glyphs before it starts displaying them on the
physical screen?

> Emacs never examines portions of the buffer that don't need to be on
> the screen (or a few lines of text above or below that).  It always
> starts with the current value of point, and works from that.  So
> intermediate buffer positions will never be examined.

> > Would it be possible to enhance display such that incoming input would
> > abort the current display operation?

> It does already.  Emacs simply doesn't know any better.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Aborting display. Is this possible?

[Eli Zaretskii]

> Date: Sun, 19 Oct 2014 15:42:55 +0000
> Cc: emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> > That's not what happens.  What does happen is that redisplay _tries_
> > several times to do its job, right after each PageDown keypress, each
> > time starting with the value of point it sees, then abandoning that
> > attempt because input arrives before it could finish.
> 
> So if 35 PageDowns are received in a second, it starts redisplay after
> each and every one.

Yes, unless by the time Emacs finishes processing one PageDown key
there's another one in the keyboard queue.

> How much time does it take before checking for pending input?

It checks for it every cycle through the command loop.  IOW,
"immediately" after the command bound to PageDown finishes, and
control is returned to the main loop.

> Could my problem be that this time is short enough that that no
> intermediate screen has time to display a fragment, but long enough that
> there's a significant pause when the user finally takes her finger of
> PageDown?

It seems like you think Emacs is able to produce only a partial redraw
of the screen; if so, you are wrong: Emacs never does that.  Redisplay
of each window (or maybe even each frame, I don't recall exactly) is
an atomic operation: it is either done completely, or not at all.
Emacs checks for pending input at strategic places, so that you never
see a partially incorrect display.

(Of course, when redisplay-dont-pause is non-nil, as it is by default,
pending input doesn't interrupt redisplay at all.  the above describes
what happens when redisplay-dont-pause is nil.)

As for the user lifting the finger, this is not the problem.
Auto-repeat still sends one key at a time, with some time interval
between the keys.  As long as that interval is long enough for Emacs
to compute the beginning of the window due to PageDown (and how long
could that be?), Emacs will attempt to redisplay before the next
repeated key arrives.

> What is stopping a fragment of each intermediate screen (or of some of
> them) being displayed?  Could it be that the display engine calculates an
> entire screenful of glyphs before it starts displaying them on the
> physical screen?

Emacs first calculates the minimum portion of the screen that needs
redrawing (checking for input several times during this part, and
abandoning redisplay if there's input and redisplay-dont-pause is
nil).  When these calculations are finished, it checks for input one
last time.  If there's no input, or if redisplay-dont-pause is
non-nil, Emacs proceeds to actually drawing the parts that changed,
and redraws all that is needed without any further input checks.
Otherwise, the redisplay cycle is aborted.

This is done for every frame at least, perhaps even for each window (I
don't remember).

Re: Aborting display. Is this possible?

[Stefan Monnier]

> What is stopping a fragment of each intermediate screen (or of some of
> them) being displayed?

Redisplay is done as follows: compute a new glyph matrix (which is
basically an internal representation of what the display should look
like), and when that matrix is computed the content is drawn on screen.
I.e. drawing does not start until the glyph matrix is fully computed.

The drawing itself is relatively quick, IIUC.


        Stefan

Re: Aborting display. Is this possible?

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Eli Zaretskii <eliz@gnu.org>,  emacs-devel@gnu.org
> Date: Sun, 19 Oct 2014 21:59:09 -0400
> 
> I.e. drawing does not start until the glyph matrix is fully computed.

Where "fully computed" might mean only some portions of it are
actually computed and "enabled for redisplay", if Emacs can determine
that the rest doesn't need any update.

Re: Aborting display. Is this possible?

[Alan Mackenzie]

Hello, Eli.

Before getting into your last post, I'll make explicit what I see, and
what I'd like to see.

On a large C Mode buffer (such as xdisp.c), hold down the PageDown key
for, say, 10 seconds.

What is seen: The screen freezes.  After releasing the key, it takes many
seconds (~10) before the current buffer position is displayed on the
screen.

What I'd like to see: The screen continuously updating with (not
necessarily contiguous) parts of the buffer, the updating being frequent
enough to give the illusion of continuous scrolling.  On releasing the
key, the screen "instantly" displaying the new buffer position.

On Sun, Oct 19, 2014 at 09:09:13PM +0300, Eli Zaretskii wrote:
> > Date: Sun, 19 Oct 2014 15:42:55 +0000
> > Cc: emacs-devel@gnu.org
> > From: Alan Mackenzie <acm@muc.de>

> > > That's not what happens.  What does happen is that redisplay _tries_
> > > several times to do its job, right after each PageDown keypress, each
> > > time starting with the value of point it sees, then abandoning that
> > > attempt because input arrives before it could finish.

> > So if 35 PageDowns are received in a second, it starts redisplay after
> > each and every one.

> Yes, unless by the time Emacs finishes processing one PageDown key
> there's another one in the keyboard queue.

I'm not so sure of this.  I think Emacs is completing the fontification
for the screenful after each PageDown, only it is not physically
displaying it.  (See below).

> > How much time does it take before checking for pending input?

[ ... ]

> It seems like you think Emacs is able to produce only a partial redraw
> of the screen; if so, you are wrong: Emacs never does that.  Redisplay
> of each window (or maybe even each frame, I don't recall exactly) is
> an atomic operation: it is either done completely, or not at all.
> Emacs checks for pending input at strategic places, so that you never
> see a partially incorrect display.

OK.  This sounds like a definite design decision.  Thinking about it, it
sounds very sensible.

[ ... ]

> Emacs first calculates the minimum portion of the screen that needs
> redrawing (checking for input several times during this part, and
> abandoning redisplay if there's input and redisplay-dont-pause is
> nil).  When these calculations are finished, it checks for input one
> last time.  If there's no input, or if redisplay-dont-pause is
> non-nil, Emacs proceeds to actually drawing the parts that changed,
> and redraws all that is needed without any further input checks.
> Otherwise, the redisplay cycle is aborted.

> This is done for every frame at least, perhaps even for each window (I
> don't remember).

I've written a little tool to scan a buffer for the `fontified' property.
If I "reset" xdisp.c then hold down PageDown for ~10s, then I find that
`fontified' has been set on _every_ buffer position from (point-min) to
(point).  This suggests a bug in either CC Mode or the command loop.

emacs -Q.  Load this file, get-fontified.el:

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
(defvar regions nil)
(defun get-fontified ()
  "Display a list of regions which have the `fontified' property set.
Retain this list in variable `regions'."
  (interactive)
  (setq regions nil)
  (let ((beg (if (get-text-property (point-min) 'fontified)
		 (point-min)
	       (next-single-property-change (point-min) 'fontified)))
	end)
    (while beg
      (setq end (or (text-property-any beg (point-max) 'fontified nil)
		    (point-max)))
      (push (cons beg end) regions)
      (setq beg (next-single-property-change end 'fontified)))
    (setq regions (nreverse regions)))
  (message "Fontified regions: %s" regions))
(global-set-key [f10] 'get-fontified)

(defun wipe-fontification ()
  "Remove all fontification by switching to fundamental mode and back to C Mode."
  (interactive)
  (fundamental-mode)
  (c-mode))
(global-set-key [f11] 'wipe-fontification)
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

At BOB xdisp.c, press F11, press and hold PageDown, release PageDown
after ~10s, then press F10.  This is what I see:

    Fontified regions: ((1 . 503768))

.  Thus it seems either the command loop or CC Mode is fontifying _every_
position which PageDown is scanning over.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Aborting display. Is this possible?

[David Kastrup]

Alan Mackenzie <acm@muc.de> writes:

> At BOB xdisp.c, press F11, press and hold PageDown, release PageDown
> after ~10s, then press F10.  This is what I see:
>
>     Fontified regions: ((1 . 503768))
>
> .  Thus it seems either the command loop or CC Mode is fontifying _every_
> position which PageDown is scanning over.

How couldn't it?  Fontification affects how much material constitutes
one page full of material.

-- 
David Kastrup

Re: Aborting display. Is this possible?

[Alan Mackenzie]

Hi, David.

On Mon, Oct 20, 2014 at 01:30:02PM +0200, David Kastrup wrote:
> Alan Mackenzie <acm@muc.de> writes:

> > At BOB xdisp.c, press F11, press and hold PageDown, release PageDown
> > after ~10s, then press F10.  This is what I see:

> >     Fontified regions: ((1 . 503768))

> > .  Thus it seems either the command loop or CC Mode is fontifying _every_
> > position which PageDown is scanning over.

> How couldn't it?  Fontification affects how much material constitutes
> one page full of material.

I suppose this is true in the fully general case, when different faces
on a GUI system can have different heights and widths.

Where in the source code, then, is this fontification being done as part
of a scrolling command?

In the usual situation in a programming mode, however, characters are
going to be identical in size.  There must be room for optimisation
somewhere.

> -- 
> David Kastrup

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Aborting display. Is this possible?

[Eli Zaretskii]

> Date: Mon, 20 Oct 2014 11:09:49 +0000
> Cc: emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> On a large C Mode buffer (such as xdisp.c), hold down the PageDown key
> for, say, 10 seconds.
> 
> What is seen: The screen freezes.  After releasing the key, it takes many
> seconds (~10) before the current buffer position is displayed on the
> screen.
> 
> What I'd like to see: The screen continuously updating with (not
> necessarily contiguous) parts of the buffer, the updating being frequent
> enough to give the illusion of continuous scrolling.  On releasing the
> key, the screen "instantly" displaying the new buffer position.

I understand the desire, and I'm not saying it isn't reasonable.  So
please don't take what's below as a rebuttal; I'm just trying to
explain how the current Emacs design works against your wishes, and
what could/should be changed for your wishes to maybe become reality.

I will probably say some (or many) things that you already know, so
please bear with me.

First, Emacs's basic design is that it processes all the available
input before it enters redisplay.  As long as there's input in the
input queue, Emacs will not call redisplay.  (There are 2 exceptions
to this rule: processing input could indirectly call the 'redisplay'
function; and messages in the echo area produced while processing
input require a kind of "partial" redisplay to show the echo area.)
Why? because processing input allows to avoid showing the user a
screen that is outdated from the user's POV, given that input more
often than not changes what should be displayed.

Another part of this puzzle is that redisplay includes fontification
of the visible portion of the buffer.  Emacs applies the font-lock
faces (and any other faces) during the 1st stage of its redisplay
cycle, when it computes the so-called "glyph matrices", which are data
structures maintained for each window that describe what the updated
screen should look like.  As David points out, it is impossible to
compute the glyph matrices without applying the faces, because faces
change the metrics of each character, and thus affect the layout,
which is what glyph matrices are all about.

The final part of the puzzle is what the scroll-up-command does when
Emacs processes a single PageDown keypress.  It computes the new
starting point for the window, which (if we neglect complications) is
simply the beginning of line N screen lines below the original window
start, where N is almost the height of the window.  (Note that I wrote
"screen lines", not "physical lines", so this is not entirely trivial
when there are continuation lines and variable-size fonts.)  Then it
determines whether this change of the window-start requires to move
point to bring it into the view, and if so, moves point as well.

So, after processing a single PageDown key, the window has a new
starting point, and point in that window's buffer has been moved.

If, after processing this single PageDown key, the input queue is
empty (as it is when you lean on the key, because Emacs finishes the
above processing in less time than the auto-repeat interval), Emacs
enters redisplay.  Redisplay computes the glyph matrix of the window
given its start point as set by scroll-up-command; as part of this
matrix computation, the display engine applies all the font-lock
properties to the text, and sets the 'fontified' property on that
text.  Then, just before it is about to redraw the window given the
new glyph matrix, it checks for input.  If there's input available,
and redisplay-dont-pause is nil, the window will not be redrawn,
i.e. this redisplay cycle will be aborted.

Then the same will happen again for the next PageDown key.  The result
is a frozen window display, because all the attempts to redisplay are
aborted due to input rate that is high enough to preempt redisplay,
but not high enough to prevent Emacs from entering redisplay after
every (or almost every) PageDown key.  To see more frequent updates
that give an illusion of scrolling, we would need a feature that
ignores the input availability, or maybe dynamically changes the
number of input events that could be ignored when redisplay falls
behind.  We don't have such a feature.

Now, what happens when you release the key?  The input queue is still
full of unprocessed PageDown keys.  As I explained above, Emacs will
drain the entire input queue before it enters redisplay; thus the long
delay you see after releasing the key.

You want Emacs to "immediately" display the new buffer position, but
this is impossible without some code that would quickly scan the input
queue, analyze what's there, and "understand" that 100 PageDown
keystrokes can be "optimized" by executing a single scroll-up-command
with a suitable argument.  IOW, we would need some input preprocessing
stage that could "optimize" input by replacing a series of commands
with a single command, and do that cheaply.  We don't have such a
feature; patches to add it will probably be welcome.

Re: Aborting display. Is this possible?

[Eli Zaretskii]

> Date: Mon, 20 Oct 2014 12:00:52 +0000
> From: Alan Mackenzie <acm@muc.de>
> Cc: emacs-devel@gnu.org
> 
> > > .  Thus it seems either the command loop or CC Mode is fontifying _every_
> > > position which PageDown is scanning over.
> >
> > How couldn't it?  Fontification affects how much material constitutes
> > one page full of material.
> 
> I suppose this is true in the fully general case, when different faces
> on a GUI system can have different heights and widths.

The display engine doesn't know when it is in the "non-general" case
that allows it to make such shortcuts, except when the frame is a TTY
frame.

> Where in the source code, then, is this fontification being done as part
> of a scrolling command?

It's not part of a scrolling command, it is part of redisplay _after_
the scrolling command finishes.  See my other message with the
detailed description.

> In the usual situation in a programming mode, however, characters are
> going to be identical in size.  There must be room for optimisation
> somewhere.

This is false, for several reasons.

First, the display engine doesn't know anything about the semantics of
the modes -- it doesn't understand how a programming mode is different
from the other kind (and frankly, I'm not sure it is different, given
the wealth of optional display features people tend to use in
programming modes).

Second, even if we would introduce a feature through which a major
mode could tell the display engine that it is a "programming mode",
you are wrong when you assume the characters are identical in size.
Font-lock faces can legitimately be bold or slanted, in which case
their size will be different.  Emacs cannot know whether this is the
case without examining the buffer positions it is about to display.

I suppose we could add a feature whereby a mode or some other Lisp
(e.g., some user-defined code) could tell the display engine that the
buffer contents are "simple" in the sense that all the characters are
of the same size, and then request the optional features that violate
that to again tell the display engine.  But we don't have that now,
and never did.

Re: Aborting display. Is this possible?

[Stefan Monnier]

> If, after processing this single PageDown key, the input queue is
> empty (as it is when you lean on the key, because Emacs finishes the
> above processing in less time than the auto-repeat interval), Emacs
> enters redisplay.

This is the case when processing the scroll-up-command takes less time
than the repeat rate.

> Now, what happens when you release the key?  The input queue is still
> full of unprocessed PageDown keys.

If processing a single scroll-up-command takes less time than the repeat
rate, the input queue should never have many unprocessed PageDown events
(that's the whole reason why we check the input queue before entering
redisplay).

So I still don't understand why we see this long delay when releasing
the key.


        Stefan

Re: Aborting display. Is this possible?

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Alan Mackenzie <acm@muc.de>,  emacs-devel@gnu.org
> Date: Mon, 20 Oct 2014 12:56:56 -0400
> 
> > If, after processing this single PageDown key, the input queue is
> > empty (as it is when you lean on the key, because Emacs finishes the
> > above processing in less time than the auto-repeat interval), Emacs
> > enters redisplay.
> 
> This is the case when processing the scroll-up-command takes less time
> than the repeat rate.

That's what I said, yes.

> > Now, what happens when you release the key?  The input queue is still
> > full of unprocessed PageDown keys.
> 
> If processing a single scroll-up-command takes less time than the repeat
> rate, the input queue should never have many unprocessed PageDown events
> (that's the whole reason why we check the input queue before entering
> redisplay).
> 
> So I still don't understand why we see this long delay when releasing
> the key.

Not every screenful needs the same time to redisplay.  There comes a
screen that needs more, and the queue is no longer empty.

Aborting display. Is this possible?

[grischka]

> Would it be possible to enhance display such that incoming input would
> abort the current display operation?

I've seen such working in 23.x.  IIRC needs setting
"redisplay-preemption-period" to some (very small?) value
plus patching some bitrott in the related C.

--- griachks

Re: Aborting display. Is this possible?

[Eli Zaretskii]

> Date: Mon, 20 Oct 2014 19:18:46 +0200
> From: grischka <grishka@gmx.de>
> Cc: emacs-devel@gnu.org
> 
> > Would it be possible to enhance display such that incoming input would
> > abort the current display operation?
> 
> I've seen such working in 23.x.  IIRC needs setting
> "redisplay-preemption-period" to some (very small?) value
> plus patching some bitrott in the related C.

redisplay-preemption-period and the related support code were removed
almost 2 years ago, see
http://lists.gnu.org/archive/html/emacs-devel/2013-01/msg00576.html.

Re: Aborting display. Is this possible?

[Eli Zaretskii]

> Date: Mon, 20 Oct 2014 20:10:07 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: acm@muc.de, emacs-devel@gnu.org
> 
> > > Now, what happens when you release the key?  The input queue is still
> > > full of unprocessed PageDown keys.
> > 
> > If processing a single scroll-up-command takes less time than the repeat
> > rate, the input queue should never have many unprocessed PageDown events
> > (that's the whole reason why we check the input queue before entering
> > redisplay).
> > 
> > So I still don't understand why we see this long delay when releasing
> > the key.
> 
> Not every screenful needs the same time to redisplay.  There comes a
> screen that needs more, and the queue is no longer empty.

In addition, there are 2 places where auto-repeated keystrokes wait:
one is the Emacs keyboard buffer, the other is the system input queue,
since when we process keys we don't call the read_socket_hook.

If you put a breakpoint in kbd_buffer_store_event and
kbd_buffer_get_event, and define breakpoint commands that display the
queue size, you will see that when you release the key, the queue size
decreases to zero, then jumps to a large number, then decreases again,
and finally the window is redisplayed.

To me this means we have a lot of unprocessed keys in both queues.
IOW, when the queue fills up because some screen took too long to
display, Emacs doesn't succeed in emptying it, as long as the key is
pressed.

Re: Aborting display. Is this possible?

[Alan Mackenzie]

Hello, Eli.

On Mon, Oct 20, 2014 at 06:12:05PM +0300, Eli Zaretskii wrote:
> > Date: Mon, 20 Oct 2014 11:09:49 +0000
> > Cc: emacs-devel@gnu.org
> > From: Alan Mackenzie <acm@muc.de

> > On a large C Mode buffer (such as xdisp.c), hold down the PageDown key
> > for, say, 10 seconds.

> > What is seen: The screen freezes.  After releasing the key, it takes many
> > seconds (~10) before the current buffer position is displayed on the
> > screen.

Incidentally, I saw this happening today in emacs-lisp-mode, on my
terminal with a 3 column follow-mode.

> > What I'd like to see: The screen continuously updating with (not
> > necessarily contiguous) parts of the buffer, the updating being frequent
> > enough to give the illusion of continuous scrolling.  On releasing the
> > key, the screen "instantly" displaying the new buffer position.

> I understand the desire, and I'm not saying it isn't reasonable.  So
> please don't take what's below as a rebuttal; I'm just trying to
> explain how the current Emacs design works against your wishes, and
> what could/should be changed for your wishes to maybe become reality.

> I will probably say some (or many) things that you already know, so
> please bear with me.

Thanks for such a clear exposition.

> First, Emacs's basic design is that it processes all the available
> input before it enters redisplay.  As long as there's input in the
> input queue, Emacs will not call redisplay.  (There are 2 exceptions
> to this rule: processing input could indirectly call the 'redisplay'
> function; and messages in the echo area produced while processing
> input require a kind of "partial" redisplay to show the echo area.)
> Why? because processing input allows to avoid showing the user a
> screen that is outdated from the user's POV, given that input more
> often than not changes what should be displayed.

Up to now, I think I've been confusing display with font-locking.
Obviously f-l has to happen when d is being done, but not all
font-locking is caused by display.

> Another part of this puzzle is that redisplay includes fontification
> of the visible portion of the buffer.  Emacs applies the font-lock
> faces (and any other faces) during the 1st stage of its redisplay
> cycle, when it computes the so-called "glyph matrices", which are data
> structures maintained for each window that describe what the updated
> screen should look like.  As David points out, it is impossible to
> compute the glyph matrices without applying the faces, because faces
> change the metrics of each character, and thus affect the layout,
> which is what glyph matrices are all about.

> The final part of the puzzle is what the scroll-up-command does when
> Emacs processes a single PageDown keypress.  It computes the new
> starting point for the window, which (if we neglect complications) is
> simply the beginning of line N screen lines below the original window
> start, where N is almost the height of the window.  (Note that I wrote
> "screen lines", not "physical lines", so this is not entirely trivial
> when there are continuation lines and variable-size fonts.)  Then it
> determines whether this change of the window-start requires to move
> point to bring it into the view, and if so, moves point as well.

On a PageDown, Emacs needs to do font-locking just to work out how far
awy the next page is, like David said.  There is a comment in
window_scroll_line_based: "Fvertical_motion enters redisplay, which can
trigger fontification...".  This is the killer: scrolling a screen of
xdisp.c is taking, on average, ~0.09s, the bulk of this time being
fontification, whereas the key events are coming in every ~0.03s.
Some way has to be found to scroll with only every third screen (at most)
getting font-locked.

If every glyph were assumed to be the same size, the calculation of the
window-start position for the current scroll operation would be _much_
faster: display would still have to account for text lines overflowing
screen lines, and invisible text, and so on, 

So, how about the following strategy: when the (new) variable
`face-instead-of-fontifying' is bound to a face, AND the input queue is
non-empty (or, perhaps, very non-empty), the appropriate subroutine of
window-scroll should deem characters without a `face' (or `fontified' ?)
property to have face `face-instead-of-fontifying'.

This should empty the input queue pretty quickly, enabling a current
buffer portion to get displayed frequently enough.

> So, after processing a single PageDown key, the window has a new
> starting point, and point in that window's buffer has been moved.

> If, after processing this single PageDown key, the input queue is
> empty (as it is when you lean on the key, because Emacs finishes the
> above processing in less time than the auto-repeat interval), Emacs
> enters redisplay.  Redisplay computes the glyph matrix of the window
> given its start point as set by scroll-up-command; as part of this
> matrix computation, the display engine applies all the font-lock
> properties to the text, and sets the 'fontified' property on that
> text.  Then, just before it is about to redraw the window given the
> new glyph matrix, it checks for input.  If there's input available,
> and redisplay-dont-pause is nil, the window will not be redrawn,
> i.e. this redisplay cycle will be aborted.

But in the default situation (redisplay-dont-pause is non-nil), once the
calculation of the glyph matrix has begun, the redrawing necessarily
takes place.  This is good.  :-)

> Then the same will happen again for the next PageDown key.  The result
> is a frozen window display, because all the attempts to redisplay are
> aborted due to input rate that is high enough to preempt redisplay,
> but not high enough to prevent Emacs from entering redisplay after
> every (or almost every) PageDown key.  To see more frequent updates
> that give an illusion of scrolling, we would need a feature that
> ignores the input availability, or maybe dynamically changes the
> number of input events that could be ignored when redisplay falls
> behind.  We don't have such a feature.

Is this really the case?  Is it not that the font-locking caused by
scrolling is slow enough, that the input queue is never empty, hence
redisplay doesn't get a look in?

> Now, what happens when you release the key?  The input queue is still
> full of unprocessed PageDown keys.  As I explained above, Emacs will
> drain the entire input queue before it enters redisplay; thus the long
> delay you see after releasing the key.

And each such PageDown key will necessitate full font-locking for its
never-to-be-displayed screen.

> You want Emacs to "immediately" display the new buffer position, but
> this is impossible without some code that would quickly scan the input
> queue, analyze what's there, and "understand" that 100 PageDown
> keystrokes can be "optimized" by executing a single scroll-up-command
> with a suitable argument.  IOW, we would need some input preprocessing
> stage that could "optimize" input by replacing a series of commands
> with a single command, and do that cheaply.  We don't have such a
> feature; patches to add it will probably be welcome.

:-).  If my analysis is right, this wouldn't help.  A single mega-scroll
would still be font-locking the entire intermediate buffer region.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Aborting display. Is this possible?

[Eli Zaretskii]

> Date: Mon, 20 Oct 2014 18:57:57 +0000
> From: Alan Mackenzie <acm@muc.de>
> Cc: emacs-devel@gnu.org
> 
> On a PageDown, Emacs needs to do font-locking just to work out how far
> awy the next page is, like David said.  There is a comment in
> window_scroll_line_based: "Fvertical_motion enters redisplay, which can
> trigger fontification...".  This is the killer: scrolling a screen of
> xdisp.c is taking, on average, ~0.09s, the bulk of this time being
> fontification, whereas the key events are coming in every ~0.03s.
> Some way has to be found to scroll with only every third screen (at most)
> getting font-locked.

window_scroll_line_based is not relevant to GUI frames, only to
text-mode frames.  And window_scroll_pixel_based does not call
vertical-motion.

Also, "enters redisplay" needs to be taken with a grain of salt here.
The reference is to display "simulation", whereby Emacs proceeds
through buffer text line after line, computing the metrics of the
characters as it goes (and invoking fontification functions if
needed), but doesn't store the results in glyph matrices and doesn't
proceed to redrawing based on that.  So it's only partial redisplay.
The full redisplay will be done later, after the scrolling command did
its part, and Emacs is back in the command loop with no available
input.

> If every glyph were assumed to be the same size, the calculation of the
> window-start position for the current scroll operation would be _much_
> faster: display would still have to account for text lines overflowing
> screen lines, and invisible text, and so on, 

Yes, but we've lost that paradise when we introduced variable-font
support in Emacs 21.

> So, how about the following strategy: when the (new) variable
> `face-instead-of-fontifying' is bound to a face, AND the input queue is
> non-empty (or, perhaps, very non-empty), the appropriate subroutine of
> window-scroll should deem characters without a `face' (or `fontified' ?)
> property to have face `face-instead-of-fontifying'.
> 
> This should empty the input queue pretty quickly, enabling a current
> buffer portion to get displayed frequently enough.

This would mean we will sometimes wind up in the wrong place after
scroll.  IOW, you are sacrificing correctness on behalf of some
(questionable for now) speedup.  I say ""questionable" because it's
not clear to me that using just one fixed face will significantly
speed up the process of painstakingly moving through text line by
line.

> But in the default situation (redisplay-dont-pause is non-nil), once the
> calculation of the glyph matrix has begun, the redrawing necessarily
> takes place.

Yes, once.  After that, if input events come in faster than Emacs can
process them, you again get a frozen outdated window.

There isn't really a good solution for when input comes in faster than
Emacs can process it.

> > Then the same will happen again for the next PageDown key.  The result
> > is a frozen window display, because all the attempts to redisplay are
> > aborted due to input rate that is high enough to preempt redisplay,
> > but not high enough to prevent Emacs from entering redisplay after
> > every (or almost every) PageDown key.  To see more frequent updates
> > that give an illusion of scrolling, we would need a feature that
> > ignores the input availability, or maybe dynamically changes the
> > number of input events that could be ignored when redisplay falls
> > behind.  We don't have such a feature.
> 
> Is this really the case?  Is it not that the font-locking caused by
> scrolling is slow enough, that the input queue is never empty, hence
> redisplay doesn't get a look in?

It eventually gets to that, because some screenfuls need more time to
redisplay.

> > Now, what happens when you release the key?  The input queue is still
> > full of unprocessed PageDown keys.  As I explained above, Emacs will
> > drain the entire input queue before it enters redisplay; thus the long
> > delay you see after releasing the key.
> 
> And each such PageDown key will necessitate full font-locking for its
> never-to-be-displayed screen.

No, not full, only partial.  See above.

> > You want Emacs to "immediately" display the new buffer position, but
> > this is impossible without some code that would quickly scan the input
> > queue, analyze what's there, and "understand" that 100 PageDown
> > keystrokes can be "optimized" by executing a single scroll-up-command
> > with a suitable argument.  IOW, we would need some input preprocessing
> > stage that could "optimize" input by replacing a series of commands
> > with a single command, and do that cheaply.  We don't have such a
> > feature; patches to add it will probably be welcome.
> 
> :-).  If my analysis is right, this wouldn't help.  A single mega-scroll
> would still be font-locking the entire intermediate buffer region.

Again, only partially so.  And please note that scrolling is
inherently at disadvantage here, because it tells Emacs to move N
screen lines up/down, so Emacs must first figure out where that lands
it.  Other commands that move through buffer text, like goto-char,
don't have that problem.

Re: Aborting display. Is this possible?

[Alan Mackenzie]

Hi, Eli.

On Mon, Oct 20, 2014 at 10:24:54PM +0300, Eli Zaretskii wrote:
> > Date: Mon, 20 Oct 2014 18:57:57 +0000
> > From: Alan Mackenzie <acm@muc.de>
> > Cc: emacs-devel@gnu.org

> > On a PageDown, Emacs needs to do font-locking just to work out how far
> > awy the next page is, like David said.  There is a comment in
> > window_scroll_line_based: "Fvertical_motion enters redisplay, which can
> > trigger fontification...".  This is the killer: scrolling a screen of
> > xdisp.c is taking, on average, ~0.09s, the bulk of this time being
> > fontification, whereas the key events are coming in every ~0.03s.
> > Some way has to be found to scroll with only every third screen (at most)
> > getting font-locked.

> window_scroll_line_based is not relevant to GUI frames, only to
> text-mode frames.  And window_scroll_pixel_based does not call
> vertical-motion.

OK.  But the problem occurs both in GUI and tty.  There'll be some C
function at a deeper level where the GUI and tty cases meet up.  The tty
case is not going to be more difficult to solve than the GUI one.  [ By
the way, it's brilliant how effectively the difference between them has
been abstracted away for elisp programming. ]

> Also, "enters redisplay" needs to be taken with a grain of salt here.
> The reference is to display "simulation", whereby Emacs proceeds
> through buffer text line after line, computing the metrics of the
> characters as it goes (and invoking fontification functions if
> needed), but doesn't store the results in glyph matrices and doesn't
> proceed to redrawing based on that.  So it's only partial redisplay.
> The full redisplay will be done later, after the scrolling command did
> its part, and Emacs is back in the command loop with no available
> input.

Yes, I've got that.  But in CC Mode, it's the fontification which takes
most of the time (see also below).

> > If every glyph were assumed to be the same size, the calculation of the
> > window-start position for the current scroll operation would be _much_
> > faster: display would still have to account for text lines overflowing
> > screen lines, and invisible text, and so on, 

> Yes, but we've lost that paradise when we introduced variable-font
> support in Emacs 21.

> > So, how about the following strategy: when the (new) variable
> > `face-instead-of-fontifying' is bound to a face, AND the input queue is
> > non-empty (or, perhaps, very non-empty), the appropriate subroutine of
> > window-scroll should deem characters without a `face' (or `fontified' ?)
> > property to have face `face-instead-of-fontifying'.

> > This should empty the input queue pretty quickly, enabling a current
> > buffer portion to get displayed frequently enough.

> This would mean we will sometimes wind up in the wrong place after
> scroll.  IOW, you are sacrificing correctness on behalf of some
> (questionable for now) speedup.

I'd posit that the absolute correctness isn't all that important after an
auto-repeating PageDown.  For an individual PageDown keystroke, the
fontification and scrolling would be done accurately as now.

> I say ""questionable" because it's not clear to me that using just one
> fixed face will significantly speed up the process of painstakingly
> moving through text line by line.

OK: some numbers, on emacs-24 -Q in both cases:

To scroll backwards on a tty with a 64 line window through xdisp.c takes
0.106s per screen (on average).  Doing the same on an already fontified
buffer takes 0.0128s per screen (on average).  It would seem that the
fontification is taking ~87% of the time.  (See the thread "Scrolling
xdisp.c.  Room for optimisation in syntax.c/CC Mode." for details of the
measurement.)

The same on a GUI (in X-Windows on a GNU system, 33 line window):
0.05268s per screen, fresh; 0.00655 per screen, starting fully fontified.
Fontification is taking a similar proportion of the total time.

> > But in the default situation (redisplay-dont-pause is non-nil), once the
> > calculation of the glyph matrix has begun, the redrawing necessarily
> > takes place.

> Yes, once.  After that, if input events come in faster than Emacs can
> process them, you again get a frozen outdated window.

> There isn't really a good solution for when input comes in faster than
> Emacs can process it.

Er, making Emacs process the input faster?

> > > Then the same will happen again for the next PageDown key.  The result
> > > is a frozen window display, because all the attempts to redisplay are
> > > aborted due to input rate that is high enough to preempt redisplay,
> > > but not high enough to prevent Emacs from entering redisplay after
> > > every (or almost every) PageDown key.  To see more frequent updates
> > > that give an illusion of scrolling, we would need a feature that
> > > ignores the input availability, or maybe dynamically changes the
> > > number of input events that could be ignored when redisplay falls
> > > behind.  We don't have such a feature.

> > Is this really the case?  Is it not that the font-locking caused by
> > scrolling is slow enough, that the input queue is never empty, hence
> > redisplay doesn't get a look in?

> It eventually gets to that, because some screenfuls need more time to
> redisplay.

I think it gets to that stage very early indeed.

> > > Now, what happens when you release the key?  The input queue is still
> > > full of unprocessed PageDown keys.  As I explained above, Emacs will
> > > drain the entire input queue before it enters redisplay; thus the long
> > > delay you see after releasing the key.

> > And each such PageDown key will necessitate full font-locking for its
> > never-to-be-displayed screen.

> No, not full, only partial.  See above.

I don't see the "above" here.  When I tested the `fontified' property
this morning, EVERY position from point-min to point had been fontified.

> > > You want Emacs to "immediately" display the new buffer position, but
> > > this is impossible without some code that would quickly scan the input
> > > queue, analyze what's there, and "understand" that 100 PageDown
> > > keystrokes can be "optimized" by executing a single scroll-up-command
> > > with a suitable argument.  IOW, we would need some input preprocessing
> > > stage that could "optimize" input by replacing a series of commands
> > > with a single command, and do that cheaply.  We don't have such a
> > > feature; patches to add it will probably be welcome.

> > :-).  If my analysis is right, this wouldn't help.  A single mega-scroll
> > would still be font-locking the entire intermediate buffer region.

> Again, only partially so.

I still don't see the "partially".  Why wouldn't everything get fontified
again (for the calculations for scrolling, not for display)?

> And please note that scrolling is inherently at disadvantage here,
> because it tells Emacs to move N screen lines up/down, so Emacs must
> first figure out where that lands it.  Other commands that move through
> buffer text, like goto-char, don't have that problem.

Yes.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Aborting display. Is this possible?

[David Kastrup]

Alan Mackenzie <acm@muc.de> writes:

> I'd posit that the absolute correctness isn't all that important after an
> auto-repeating PageDown.

That would mean that if you lean on PageDown, something catches your eye
while leafing through, you stop and press PageUp a few times in order to
locate it, it won't reappear where it caught your eye.

-- 
David Kastrup

Re: Aborting display. Is this possible?

[Alan Mackenzie]

Hello, David.

On Tue, Oct 21, 2014 at 10:09:18AM +0200, David Kastrup wrote:
> Alan Mackenzie <acm@muc.de> writes:

> > I'd posit that the absolute correctness isn't all that important after an
> > auto-repeating PageDown.

> That would mean that if you lean on PageDown, something catches your eye
> while leafing through, you stop and press PageUp a few times in order to
> locate it, it won't reappear where it caught your eye.

Yes, possibly.  That's if the "default" face used to calculate positions
and occupancy is grossly bigger or smaller than the typical face.  But
since you've stopped very soon after seeing FOO, the discrepancy
typically shouldn't be too large.  The other answer is that since in
this sort of scrolling only a small proportion of screens actually get
displayed (see below), the user will soon stop expecting to be able to
eyeball a passing function declaration.

At any rate, that situation would be an improvement on the current
arrangement, where leaning on PageDown (with font-lock enabled) leads to
a frozen screen, which stays frozen long after the key is released.

Should the region of the buffer already be fontified (or font-locking
disabled), the proposed new behaviour would be the same as the current.

I've done some more measurements on my system:

Autorepeat: delay between 1st event and 2nd event: 0.168s.
            delay between subsequent consecutive events: 0.024s.
Scrolling operation on non-fontified 65 line window with xdisp.c:
            mean time: 0.106s  standard deviation: 0.046s
Scrolling operation on already fontified 65 line window with xdisp.c:
            mean time: 0.011s  standard deviation: 0.0023s

By subtracting the above two, the mean time to fontify a screen is
0.095s, to display a fontified screen is 0.011s.

Assume that the time to calculate a scrolled position, without
fontification, is less than 0.011s.  Assume 0.010s.

So, by my new scheme, a typical timeline would be:

0.0s   Arrival of PageDown event   Event queue is empty, so start of
                                     fontification of screen 2.
0.095s	                           Fontification done; start of display.
0.106s                             Display complete.  Queue is empty.

0.192s Arrival of PageDown event   Event queue is empty, so start of
                                     fontification of screen 3.
0.216s Arrival of PageDown event   Queued (1)
0.240s Arrival of PageDown event   Queued (2)
0.264s Arrival of PageDown event   Queued (3)
0.287s                             Fontification done; start of display
0.288s Arrival of PageDown event   Queued (4)
0.298s                             Display of screen 3 complete.  Start
                                     scroll command.  Queue is
				     non-empty (3), so start calculating
				     new position for screen 4 without
				     fontification.
0.312s Arrival of PageDown event   Queued (4)				     
0.318s                             Finished calculation of screen 4's
                                     position.  Start scroll command.
				     Queue is 3; start calculating new
				     screen 5 position without
				     fontification.
0.328s                             Finished calculation for scree 5.
                                     Queue is 2. Start calc for screen
				     6.
0.336s Arrival of PageDown event   Queued (3)
0.338s                             Finished calc for screen 6.  Queue is
                                     2.  Start calc for screen 7.
0.348s                             Finished calc for screen 7.  Queue is
                                     1.  Start calc for screen 8.
0.358s                             Finished calc for screen 8.  Queue is
				     empty.  Start fontification for
				     screen 9.
				   
0.360s Arrival of PageDown event   Queued (1)
0.384s Arrival of PageDown event   Queued (2)
0.408s Arrival of PageDown event   Queued (3)
0.432s Arrival of PageDown event   Queued (4)
0.453s                             Fontification of screen 9 done.
                                     Start displaying it.
0.456s Arrival of PageDown event   Queued (5)
0.464s                             Display of screen 9 complete.  Start
                                     scroll command.  Queue is 4, start
				     calculating position for screen 10.
0.475s                             Finished calc for screen 10.  Queue
                                     is 3.  Start calc for screen 11.
0.480s Arrival of PageDown event   Queued (4)
0.486s                             Finished calc for screen 11.  Queue
                                     is 3.  Start calc for screen 12.
0.497s                             Finished calc for screen 12.  Queue
                                     is 2.  Start calc for screen 13.
0.504s Arrival of PageDown event   Queued (3)
0.508s                             Finished calc for screen 13.  Queue
                                     is 2.  Start calc for screen 14.
0.519s                             Finished calc for screen 14.  Queue
                                     is 1.  Start calc for screen 15.
0.528s Arrival of PageDown event   Queued (2)
0.530s                             Finished calc for screen 15.  Queue
                                     is 1.  Start calc for screen 16.
0.541s                             Finished calc for screen 16.  Queue
                                     is empty.  Start fontification for
				     screen 17.
...
0.636s                             Finished fontifying of screen 17.
                                     Start displaying it.
...
0.647s                             Display of screen 17 complete.

------------------------------------------------------------------------

So it would appear in a worst case scenario (xdisp.c in a 65 line
window), about 5 screens per second would be displayed, approximately
every 8 screens.

> -- David Kastrup

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Aborting display. Is this possible?

[David Kastrup]

Alan Mackenzie <acm@muc.de> writes:

> I've done some more measurements on my system:
>
> Autorepeat: delay between 1st event and 2nd event: 0.168s.
>             delay between subsequent consecutive events: 0.024s.

That seems rather excessive.  Usual I remember is 0.5/0.05, and that
already requires good aim.

-- 
David Kastrup

Re: Aborting display. Is this possible?

[Stefan Monnier]

>> I'd posit that the absolute correctness isn't all that important after an
>> auto-repeating PageDown.
> That would mean that if you lean on PageDown, something catches your eye
> while leafing through, you stop and press PageUp a few times in order to
> locate it, it won't reappear where it caught your eye.

Could be, but it's highly unlikely: if it caught your eye, it means that
redisplay is fast enough to keep up with scrolling, in which case
scrolling gets to use a properly fontified buffer for its calculation,
so Alan's suggestion should make no difference in that case.


        Stefan

Re: Aborting display. Is this possible?

[Stefan Monnier]

>> I've done some more measurements on my system:
>> Autorepeat: delay between 1st event and 2nd event: 0.168s.
>> delay between subsequent consecutive events: 0.024s.
> That seems rather excessive.  Usual I remember is 0.5/0.05, and that
> already requires good aim.

FWIW, I use "xset r 200 40", i.e. 0.2s/0.025s, so Alan's settings don't
seem outlandish to me.


        Stefan

Re: Aborting display. Is this possible?

[Eli Zaretskii]

> From: Stefan Monnier <monnier@IRO.UMontreal.CA>
> Date: Tue, 21 Oct 2014 10:01:43 -0400
> Cc: emacs-devel@gnu.org
> 
> >> I'd posit that the absolute correctness isn't all that important after an
> >> auto-repeating PageDown.
> > That would mean that if you lean on PageDown, something catches your eye
> > while leafing through, you stop and press PageUp a few times in order to
> > locate it, it won't reappear where it caught your eye.
> 
> Could be, but it's highly unlikely: if it caught your eye, it means that
> redisplay is fast enough to keep up with scrolling, in which case
> scrolling gets to use a properly fontified buffer for its calculation,
> so Alan's suggestion should make no difference in that case.

In any case, the way Alan suggests is not TRT, IMNSHO.  We shouldn't
be sacrificing display correctness for speed; AFAIR, we never did
anything like that.  What we did was find and implement optimizations
that catered to specific use cases that were deemed important enough.
I see no reason to treat this case differently.

Another data point is this: in the recent years, I've dealt with any
number of bug reports that complained about minor inaccuracies in
scrolling, mostly by fractions of line height or even by a couple of
pixels.  My interpretation of this is that our users expect accurate
display, and will not easily tolerate sloppiness.

So I suggest to turn the table and ask: how come fontification of C
sources is so expensive, and how can we make it faster?  If the Lisp
implementation cannot be sped up significantly, let's implement some
of the code in C.  Or maybe we are should request less accurate
parsing of the source, at least by default -- e.g., perhaps it is not
very important to display variables in a face that is different from,
say, data types?

IOW, if the measurements show that redisplay takes 10% of the time,
let's not try to optimize those 10%, but instead concentrate on the
90%.

Re: Aborting display. Is this possible?

[Eli Zaretskii]

> Date: Mon, 20 Oct 2014 21:08:19 +0000
> Cc: emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> > Also, "enters redisplay" needs to be taken with a grain of salt here.
> > The reference is to display "simulation", whereby Emacs proceeds
> > through buffer text line after line, computing the metrics of the
> > characters as it goes (and invoking fontification functions if
> > needed), but doesn't store the results in glyph matrices and doesn't
> > proceed to redrawing based on that.  So it's only partial redisplay.
> > The full redisplay will be done later, after the scrolling command did
> > its part, and Emacs is back in the command loop with no available
> > input.
> 
> Yes, I've got that.  But in CC Mode, it's the fontification which takes
> most of the time (see also below).

Then I suggest to boost the speed of that.  Optimizations should be
done in the part that takes the 80 or 90% of the time.

> > There isn't really a good solution for when input comes in faster than
> > Emacs can process it.
> 
> Er, making Emacs process the input faster?

It is already very fast, as your measurements suggest.  What slows us
down is fontifications.  That's where we should try to optimize.

> > > And each such PageDown key will necessitate full font-locking for its
> > > never-to-be-displayed screen.
> 
> > No, not full, only partial.  See above.
> 
> I don't see the "above" here.  When I tested the `fontified' property
> this morning, EVERY position from point-min to point had been fontified.

I thought I've already succeeded in explaining why.

"Partial" here means that the actual drawing to the glass is not
done.  Fontifications _are_ done, because without that, Emacs would
not know where in the buffer is that point N screenfuls away.

Re: Aborting display. Is this possible?

[Stefan Monnier]

> So I suggest to turn the table and ask: how come fontification of C
> sources is so expensive, and how can we make it faster?

Indeed, that'd be better.  But IIUC Alan has already been working fairly
hard to make CC-mode's parsing faster.  So my guess is that either it's
just really hard because of some details of the C language, or we'd need
a serious restructuring/rewrite and I don't think anyone is up for that.

> IOW, if the measurements show that redisplay takes 10% of the time,
> let's not try to optimize those 10%, but instead concentrate on the
> 90%.

There's some general pressure to improve font-locking by making it more
precise, with somewhat marginal improvements but fairly high
processing costs (pushed to the extreme, this involves passing the text
to an external toolchain that parses and type-checks and returns to
Emacs the precise highlighting info; see for example agda-mode).

So, maybe we're looking at a future where we'll need to provide some way
for delayed/asynchronous font-lock refinement.


        Stefan

Re: Aborting display. Is this possible?

[Michael Welsh Duggan]

Alan Mackenzie <acm@muc.de> writes:

> On Mon, Oct 20, 2014 at 10:24:54PM +0300, Eli Zaretskii wrote:
>> > Date: Mon, 20 Oct 2014 18:57:57 +0000
>> > From: Alan Mackenzie <acm@muc.de>
>> > Cc: emacs-devel@gnu.org
>> >
>> > So, how about the following strategy: when the (new) variable
>> > `face-instead-of-fontifying' is bound to a face, AND the input queue is
>> > non-empty (or, perhaps, very non-empty), the appropriate subroutine of
>> > window-scroll should deem characters without a `face' (or `fontified' ?)
>> > property to have face `face-instead-of-fontifying'.
>
>> > This should empty the input queue pretty quickly, enabling a current
>> > buffer portion to get displayed frequently enough.
>
>> This would mean we will sometimes wind up in the wrong place after
>> scroll.  IOW, you are sacrificing correctness on behalf of some
>> (questionable for now) speedup.
>
> I'd posit that the absolute correctness isn't all that important after an
> auto-repeating PageDown.  For an individual PageDown keystroke, the
> fontification and scrolling would be done accurately as now.

Please be careful.  If you were to change things such that C-v might
scroll heuristically instead of "correctly", please make sure this
behavior is controlled by a configuration value.  When I am trying to
recreate cc-mode problems in order to make bug reports, it is very
important that I can recreate the problem exactly.  It would be very
strange if C-v run manually ended up in a different location than C-v in
a macro, for example.

-- 
Michael Welsh Duggan
(mwd@cert.org)

Re: Aborting display. Is this possible?

[Alan Mackenzie]

Hi, Eli.

On Tue, Oct 21, 2014 at 06:35:23PM +0300, Eli Zaretskii wrote:
> > From: Stefan Monnier <monnier@IRO.UMontreal.CA>
> > Date: Tue, 21 Oct 2014 10:01:43 -0400
> > Cc: emacs-devel@gnu.org

[ ... ]

> In any case, the way Alan suggests is not TRT, IMNSHO.  We shouldn't
> be sacrificing display correctness for speed; AFAIR, we never did
> anything like that.  What we did was find and implement optimizations
> that catered to specific use cases that were deemed important enough.
> I see no reason to treat this case differently.

> Another data point is this: in the recent years, I've dealt with any
> number of bug reports that complained about minor inaccuracies in
> scrolling, mostly by fractions of line height or even by a couple of
> pixels.  My interpretation of this is that our users expect accurate
> display, and will not easily tolerate sloppiness.

This is a red herring, surely?  Tiny fractions, visible, of the line
height are totally orthogonal to being one or two lines out (which won't
even be visible) after a "sloppy" user action.

> So I suggest to turn the table and ask: how come fontification of C
> sources is so expensive, and how can we make it faster?

It has got slower as it has become more accurate.  Users have complained
about slight inaccuracies in fontification, too.  Fast enough would mean
speeding it up by a factor of around 5.  I don't think this is
practicable.

A couple of days ago, I even got auto-repeated PageDown to hang in Emacs
LIsp mode.  I can't reproduce that at the moment, though.  But I applied
my timing program to cc-engine.el, and its average time to scroll and
display a screen was 0.018s compared with an auto-repeat rate of 0.024s.
That's not a lot of free play.

> If the Lisp implementation cannot be sped up significantly, let's
> implement some of the code in C.  Or maybe we are should request less
> accurate parsing of the source, at least by default -- e.g., perhaps it
> is not very important to display variables in a face that is different
> from, say, data types?

You're suggesting sacrificing fontification accuracy for scrolling
accuracy?  :-)

> IOW, if the measurements show that redisplay takes 10% of the time,
> let's not try to optimize those 10%, but instead concentrate on the
> 90%.

A very common use case for programming languages is when all the
font-lock-* faces are mono-spaced of equal size.  This might even be the
overwhelming majority of use.  Yet the scrolling/display code makes no
optimisation for this.  In this very common case, it is entirely
unnecessary to fontify bits of buffer to work out how much text would fit
onto the screen we want to scroll over.

The whole point of Jit Lock was that we would only need to fontify the
bits of the buffer currently being displayed.  This was partly so that
mode maintainers could be a bit more relaxed about the speed of
font-locking.  The scenario of auto-repeat on PageDown suggests that the
mechanism wasn't fully thought out.

I still say an optimisation to the scrolling code, where an option could
specify that all faces are of the same size (or should be deemed to be
so) is called for here.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Aborting display. Is this possible?

[Eli Zaretskii]

> Date: Tue, 21 Oct 2014 17:14:03 +0000
> Cc: Stefan Monnier <monnier@IRO.UMontreal.CA>, dak@gnu.org,
>   emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> > Another data point is this: in the recent years, I've dealt with any
> > number of bug reports that complained about minor inaccuracies in
> > scrolling, mostly by fractions of line height or even by a couple of
> > pixels.  My interpretation of this is that our users expect accurate
> > display, and will not easily tolerate sloppiness.
> 
> This is a red herring, surely?

No, it's a data point.  And you already got one other opinion along
these same lines.

> Tiny fractions, visible, of the line height are totally orthogonal
> to being one or two lines out (which won't even be visible) after a
> "sloppy" user action.

I ensure you they _will_ be visible.  Our users are very sharp-eyed.

> > So I suggest to turn the table and ask: how come fontification of C
> > sources is so expensive, and how can we make it faster?
> 
> It has got slower as it has become more accurate.  Users have complained
> about slight inaccuracies in fontification, too.  Fast enough would mean
> speeding it up by a factor of around 5.  I don't think this is
> practicable.

Of course, it's practicable!  E.g., make it by default do what CC Mode
did a few years ago.  There was time when I could scroll through
xdisp.c on a much slower machine I have today, and Emacs would keep
up, you know.  Let users who want "more accurate fontification" ask
for it, and pay the price!

I'm not saying that's necessarily what we should do, but it surely is
one way.

> > If the Lisp implementation cannot be sped up significantly, let's
> > implement some of the code in C.  Or maybe we are should request less
> > accurate parsing of the source, at least by default -- e.g., perhaps it
> > is not very important to display variables in a face that is different
> > from, say, data types?
> 
> You're suggesting sacrificing fontification accuracy for scrolling
> accuracy?  :-)

That's a possibility, yes.  Fontification doesn't have to be as
accurate as the compiler.

> A very common use case for programming languages is when all the
> font-lock-* faces are mono-spaced of equal size.  This might even be the
> overwhelming majority of use.  Yet the scrolling/display code makes no
> optimisation for this.  In this very common case, it is entirely
> unnecessary to fontify bits of buffer to work out how much text would fit
> onto the screen we want to scroll over.

We've been through that already: the problem is how does the display
engine _know_ this is the case, without examining the buffer and the
text properties?  Suggest a reliable and easy to use mechanism for
that, and you've got my vote.

But assuming without any grounds that this is the situation, based on
just the mode, is a non-starter.  It will wreak havoc on many widely
used support modes.

> The whole point of Jit Lock was that we would only need to fontify the
> bits of the buffer currently being displayed.  This was partly so that
> mode maintainers could be a bit more relaxed about the speed of
> font-locking.  The scenario of auto-repeat on PageDown suggests that the
> mechanism wasn't fully thought out.

Well, think it out better, if you can.  Suggestions for algorithmic
improvements of the display engine are most welcome.

> I still say an optimisation to the scrolling code, where an option could
> specify that all faces are of the same size (or should be deemed to be
> so) is called for here.

Details, please!  What option, how it will be used, and how do we
ensure it is not abused -- these are the first issues I'd like to
hear.

And I still think you are barking up the wrong tree: the CC Mode is
slow, so it is the one that needs to be worked on.

Re: Aborting display. Is this possible?

[Stefan Monnier]

> The whole point of Jit Lock was that we would only need to fontify the
> bits of the buffer currently being displayed.  This was partly so that
> mode maintainers could be a bit more relaxed about the speed of
> font-locking.

Actually, no, the main benefit of jit-lock was so you can open a large
buffer without having to wait too long.

To reduce pressure on font-lock speed, we need something more like
jit-lock-defer-time (and lazy-lock-defer-time before it) which is disabled
by default.


        Stefan

Re: Aborting display. Is this possible?

[Alan Mackenzie]

Hi, Michael.

On Tue, Oct 21, 2014 at 01:00:46PM -0400, Michael Welsh Duggan wrote:
> Alan Mackenzie <acm@muc.de> writes:

> > I'd posit that the absolute correctness isn't all that important after an
> > auto-repeating PageDown.  For an individual PageDown keystroke, the
> > fontification and scrolling would be done accurately as now.

> Please be careful.  If you were to change things such that C-v might
> scroll heuristically instead of "correctly", please make sure this
> behavior is controlled by a configuration value.

Don't worry.  What I'm proposing would only kick in on an auto-repeat
C-v.  The test would be whether or not another event is in the event
queue, and if so, to bypass the font-locking until display happens.  This
wouldn't affect a single C-v.

> When I am trying to recreate cc-mode problems in order to make bug
> reports, it is very important that I can recreate the problem exactly.
> It would be very strange if C-v run manually ended up in a different
> location than C-v in a macro, for example.

That is an important use case indeed.  ;-)

> -- 
> Michael Welsh Duggan
> (mwd@cert.org)

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Aborting display. Is this possible?

[Alan Mackenzie]

Hello, Eli.

On Tue, Oct 21, 2014 at 09:00:03PM +0300, Eli Zaretskii wrote:
> > Date: Tue, 21 Oct 2014 17:14:03 +0000
> > Cc: Stefan Monnier <monnier@IRO.UMontreal.CA>, dak@gnu.org,
> >   emacs-devel@gnu.org
> > From: Alan Mackenzie <acm@muc.de>

> > I still say an optimisation to the scrolling code, where an option could
> > specify that all faces are of the same size (or should be deemed to be
> > so) is called for here.

> Details, please!  What option, how it will be used, and how do we
> ensure it is not abused -- these are the first issues I'd like to
> hear.

The option (haven't got a good name for it yet, so let's just call it
"Foo") would cause the display engine to bypass fontification whenever
the event queue is non-empty.  Instead, it would assume all characters
without a `face' property had the Foo property.

Some working code would be a better thing than mere words.  I'm going to
try and hack out a proof of concept.

> And I still think you are barking up the wrong tree: the CC Mode is
> slow, so it is the one that needs to be worked on.

That as well.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Aborting display. Is this possible?

[Eli Zaretskii]

> Date: Tue, 21 Oct 2014 18:38:07 +0000
> Cc: monnier@IRO.UMontreal.CA, dak@gnu.org, emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> The option (haven't got a good name for it yet, so let's just call it
> "Foo") would cause the display engine to bypass fontification whenever
> the event queue is non-empty.

Don't forget that as long as redisplay is working, we don't call
read_socket_hook, so the queue is not being filled up.  IOW, more
input might be available, just not yet known to Emacs.

> Instead, it would assume all characters without a `face' property
> had the Foo property.

Based on what?  Just on the fact that there's some input?  And who and
how will set and reset that option, and under what circumstances?

Re: Aborting display. Is this possible?

[Eli Zaretskii]

> Date: Tue, 21 Oct 2014 21:43:49 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: dak@gnu.org, monnier@IRO.UMontreal.CA, emacs-devel@gnu.org
> 
> > Instead, it would assume all characters without a `face' property
> > had the Foo property.
> 
> Based on what?  Just on the fact that there's some input?  And who and
> how will set and reset that option, and under what circumstances?

Btw, if all you want is prevent unnecessary fontification of the parts
that will not be displayed, there's a much easier way: use a command
that moves point directly, rather than indirectly by moving the
window-start.  IOW, bind C-v to a function that moves point N physical
lines, where N is computed from the window height.  That's it! no
fancy new options and redisplay changes are necessary.

Scrolling is more expensive because it tries to be accurate: it tries
to move the text by almost a full screen, with a small overlap whose
size is known in advance and customizable to users' content.  If you
are willing to accept inaccuracy, just move point instead.

Re: Aborting display. Is this possible?

[Stephen Leake]

Stefan Monnier <monnier@IRO.UMontreal.CA> writes:

> There's some general pressure to improve font-locking by making it more
> precise, with somewhat marginal improvements but fairly high
> processing costs (pushed to the extreme, this involves passing the text
> to an external toolchain that parses and type-checks and returns to
> Emacs the precise highlighting info; see for example agda-mode).

Did you mean "ada-mode" here? I don't find 'agda-mode' in 24.3.94 or
ELPA. I'm still having trouble with process communication, so I'll be
trying the new FFI mechanism next.

But your point is valid; I much prefer faster font-lock, rather than
less precise font-lock. I'm not sure my customers really agree; they
have not spoken up (other than one complaint that it's too slow on huge
files).

> So, maybe we're looking at a future where we'll need to provide some way
> for delayed/asynchronous font-lock refinement.

Ah, like a jpeg sent over a slow internet connection (at least, that's
they way I remember it; haven't run into that in years).

Perhaps it would be possible to define a subset of the full grammar,
that provides some meaningful level of font-lock, and is significantly
faster to parse. That would be a lot of work, but I may have to try that
if all else fails.

-- 
-- Stephe

Re: Aborting display. Is this possible?

[Stefan Monnier]

> Did you mean "ada-mode" here?

No, agda-mode: http://wiki.portal.chalmers.se/agda/agda.php


        Stefan