Aborting display. Is this possible?

Aborting display. Is this possible?

[Alan Mackenzie]

Hello, Emacs.

A lisp program can influence an ongoing display operation by setting
redisplay-dont-pause to nil.  When input arrives, the displaying is
immediately paused.

However, at the next opportunity, the displaying carries on where it
left off, rather than being aborted.  It would be nice to arrange for
such abortion to happen, leaving the partly completed display visible on
the user's video.

Why?  Better to handle a user auto-repeating on the PageDown key.  On my
system, auto-repeat is around 35-40 characters per second, on a window
with 65 lines.  In my favourite mode, display can not keep up.  If I set
redisplay-dont-pause to nil and hold down PageDown, what the user sees
is frozen for several seconds whilst display determines the display of
every intermediate buffer position (which isn't going to get displayed
anyway) before finally reaching point.

Would it be possible to enhance display such that incoming input would
abort the current display operation?

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Aborting display. Is this possible?

[David Kastrup]

Alan Mackenzie <acm@muc.de> writes:

> Why?  Better to handle a user auto-repeating on the PageDown key.  On
> my system, auto-repeat is around 35-40 characters per second, on a
> window with 65 lines.  In my favourite mode, display can not keep up.
> If I set redisplay-dont-pause to nil and hold down PageDown, what the
> user sees is frozen for several seconds whilst display determines the
> display of every intermediate buffer position (which isn't going to
> get displayed anyway) before finally reaching point.
>
> Would it be possible to enhance display such that incoming input would
> abort the current display operation?

How would Emacs figure out just how much "a screenful" of text is when
it is not allowed to lay out the screen?  Where is PageDown going to end
up?

One possible way to deal with this is not to read characters while
redisplay is being processed: a reasonably well-programmed input stack
will throttle autorepeat while it has unconsumed characters.

Of course, that may not be feasible depending on how Emacs does C-g
processing.

-- 
David Kastrup

Re: Aborting display. Is this possible?

[Eli Zaretskii]

> From: David Kastrup <dak@gnu.org>
> Date: Sun, 19 Oct 2014 16:32:38 +0200
> 
> One possible way to deal with this is not to read characters while
> redisplay is being processed

We already do that, since both keyboard input and redisplay happen in
the same thread.

The problem is that auto-repeat leaves just enough time between the
repeated keystrokes for redisplay to kick in.  If auto-repeat would
send many keys in one large block, we would process them all before
attempting to redisplay.

Re: Aborting display. Is this possible?

[Eli Zaretskii]

> Date: Sun, 19 Oct 2014 14:17:13 +0000
> From: Alan Mackenzie <acm@muc.de>
> 
> A lisp program can influence an ongoing display operation by setting
> redisplay-dont-pause to nil.  When input arrives, the displaying is
> immediately paused.

It is not paused, it is aborted.  Emacs cannot "pause" redisplay and
then "resume" it.

> However, at the next opportunity, the displaying carries on where it
> left off, rather than being aborted.

I don't understand what you mean by "carries on where it left off".
Emacs doesn't record where redisplay "left off", and cannot resume an
interrupted redisplay cycle.  It simply starts redisplay anew on the
next opportunity.

> Why?  Better to handle a user auto-repeating on the PageDown key.  On my
> system, auto-repeat is around 35-40 characters per second, on a window
> with 65 lines.  In my favourite mode, display can not keep up.  If I set
> redisplay-dont-pause to nil and hold down PageDown, what the user sees
> is frozen for several seconds whilst display determines the display of
> every intermediate buffer position (which isn't going to get displayed
> anyway) before finally reaching point.

That's not what happens.  What does happen is that redisplay _tries_
several times to do its job, right after each PageDown keypress, each
time starting with the value of point it sees, then abandoning that
attempt because input arrives before it could finish.

Emacs never examines portions of the buffer that don't need to be on
the screen (or a few lines of text above or below that).  It always
starts with the current value of point, and works from that.  So
intermediate buffer positions will never be examined.

> Would it be possible to enhance display such that incoming input would
> abort the current display operation?

It does already.  Emacs simply doesn't know any better.

Re: Aborting display. Is this possible?

[Alan Mackenzie]

Hi, Eli.

On Sun, Oct 19, 2014 at 05:50:52PM +0300, Eli Zaretskii wrote:
> > Date: Sun, 19 Oct 2014 14:17:13 +0000
> > From: Alan Mackenzie <acm@muc.de>

> > A lisp program can influence an ongoing display operation by setting
> > redisplay-dont-pause to nil.  When input arrives, the displaying is
> > immediately paused.

> It is not paused, it is aborted.  Emacs cannot "pause" redisplay and
> then "resume" it.

Ah.  OK.

> > However, at the next opportunity, the displaying carries on where it
> > left off, rather than being aborted.

> I don't understand what you mean by "carries on where it left off".
> Emacs doesn't record where redisplay "left off", and cannot resume an
> interrupted redisplay cycle.  It simply starts redisplay anew on the
> next opportunity.

I asked for it, it seems I've got it already.

> > Why?  Better to handle a user auto-repeating on the PageDown key.  On my
> > system, auto-repeat is around 35-40 characters per second, on a window
> > with 65 lines.  In my favourite mode, display can not keep up.  If I set
> > redisplay-dont-pause to nil and hold down PageDown, what the user sees
> > is frozen for several seconds whilst display determines the display of
> > every intermediate buffer position (which isn't going to get displayed
> > anyway) before finally reaching point.

> That's not what happens.  What does happen is that redisplay _tries_
> several times to do its job, right after each PageDown keypress, each
> time starting with the value of point it sees, then abandoning that
> attempt because input arrives before it could finish.

So if 35 PageDowns are received in a second, it starts redisplay after
each and every one.  How much time does it take before checking for
pending input?

Could my problem be that this time is short enough that that no
intermediate screen has time to display a fragment, but long enough that
there's a significant pause when the user finally takes her finger of
PageDown?

What is stopping a fragment of each intermediate screen (or of some of
them) being displayed?  Could it be that the display engine calculates an
entire screenful of glyphs before it starts displaying them on the
physical screen?

> Emacs never examines portions of the buffer that don't need to be on
> the screen (or a few lines of text above or below that).  It always
> starts with the current value of point, and works from that.  So
> intermediate buffer positions will never be examined.

> > Would it be possible to enhance display such that incoming input would
> > abort the current display operation?

> It does already.  Emacs simply doesn't know any better.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Aborting display. Is this possible?

[Eli Zaretskii]

> Date: Sun, 19 Oct 2014 15:42:55 +0000
> Cc: emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> > That's not what happens.  What does happen is that redisplay _tries_
> > several times to do its job, right after each PageDown keypress, each
> > time starting with the value of point it sees, then abandoning that
> > attempt because input arrives before it could finish.
> 
> So if 35 PageDowns are received in a second, it starts redisplay after
> each and every one.

Yes, unless by the time Emacs finishes processing one PageDown key
there's another one in the keyboard queue.

> How much time does it take before checking for pending input?

It checks for it every cycle through the command loop.  IOW,
"immediately" after the command bound to PageDown finishes, and
control is returned to the main loop.

> Could my problem be that this time is short enough that that no
> intermediate screen has time to display a fragment, but long enough that
> there's a significant pause when the user finally takes her finger of
> PageDown?

It seems like you think Emacs is able to produce only a partial redraw
of the screen; if so, you are wrong: Emacs never does that.  Redisplay
of each window (or maybe even each frame, I don't recall exactly) is
an atomic operation: it is either done completely, or not at all.
Emacs checks for pending input at strategic places, so that you never
see a partially incorrect display.

(Of course, when redisplay-dont-pause is non-nil, as it is by default,
pending input doesn't interrupt redisplay at all.  the above describes
what happens when redisplay-dont-pause is nil.)

As for the user lifting the finger, this is not the problem.
Auto-repeat still sends one key at a time, with some time interval
between the keys.  As long as that interval is long enough for Emacs
to compute the beginning of the window due to PageDown (and how long
could that be?), Emacs will attempt to redisplay before the next
repeated key arrives.

> What is stopping a fragment of each intermediate screen (or of some of
> them) being displayed?  Could it be that the display engine calculates an
> entire screenful of glyphs before it starts displaying them on the
> physical screen?

Emacs first calculates the minimum portion of the screen that needs
redrawing (checking for input several times during this part, and
abandoning redisplay if there's input and redisplay-dont-pause is
nil).  When these calculations are finished, it checks for input one
last time.  If there's no input, or if redisplay-dont-pause is
non-nil, Emacs proceeds to actually drawing the parts that changed,
and redraws all that is needed without any further input checks.
Otherwise, the redisplay cycle is aborted.

This is done for every frame at least, perhaps even for each window (I
don't remember).

Re: Aborting display. Is this possible?

[Alan Mackenzie]

Hello, Eli.

Before getting into your last post, I'll make explicit what I see, and
what I'd like to see.

On a large C Mode buffer (such as xdisp.c), hold down the PageDown key
for, say, 10 seconds.

What is seen: The screen freezes.  After releasing the key, it takes many
seconds (~10) before the current buffer position is displayed on the
screen.

What I'd like to see: The screen continuously updating with (not
necessarily contiguous) parts of the buffer, the updating being frequent
enough to give the illusion of continuous scrolling.  On releasing the
key, the screen "instantly" displaying the new buffer position.

On Sun, Oct 19, 2014 at 09:09:13PM +0300, Eli Zaretskii wrote:
> > Date: Sun, 19 Oct 2014 15:42:55 +0000
> > Cc: emacs-devel@gnu.org
> > From: Alan Mackenzie <acm@muc.de>

> > > That's not what happens.  What does happen is that redisplay _tries_
> > > several times to do its job, right after each PageDown keypress, each
> > > time starting with the value of point it sees, then abandoning that
> > > attempt because input arrives before it could finish.

> > So if 35 PageDowns are received in a second, it starts redisplay after
> > each and every one.

> Yes, unless by the time Emacs finishes processing one PageDown key
> there's another one in the keyboard queue.

I'm not so sure of this.  I think Emacs is completing the fontification
for the screenful after each PageDown, only it is not physically
displaying it.  (See below).

> > How much time does it take before checking for pending input?

[ ... ]

> It seems like you think Emacs is able to produce only a partial redraw
> of the screen; if so, you are wrong: Emacs never does that.  Redisplay
> of each window (or maybe even each frame, I don't recall exactly) is
> an atomic operation: it is either done completely, or not at all.
> Emacs checks for pending input at strategic places, so that you never
> see a partially incorrect display.

OK.  This sounds like a definite design decision.  Thinking about it, it
sounds very sensible.

[ ... ]

> Emacs first calculates the minimum portion of the screen that needs
> redrawing (checking for input several times during this part, and
> abandoning redisplay if there's input and redisplay-dont-pause is
> nil).  When these calculations are finished, it checks for input one
> last time.  If there's no input, or if redisplay-dont-pause is
> non-nil, Emacs proceeds to actually drawing the parts that changed,
> and redraws all that is needed without any further input checks.
> Otherwise, the redisplay cycle is aborted.

> This is done for every frame at least, perhaps even for each window (I
> don't remember).

I've written a little tool to scan a buffer for the `fontified' property.
If I "reset" xdisp.c then hold down PageDown for ~10s, then I find that
`fontified' has been set on _every_ buffer position from (point-min) to
(point).  This suggests a bug in either CC Mode or the command loop.

emacs -Q.  Load this file, get-fontified.el:

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
(defvar regions nil)
(defun get-fontified ()
  "Display a list of regions which have the `fontified' property set.
Retain this list in variable `regions'."
  (interactive)
  (setq regions nil)
  (let ((beg (if (get-text-property (point-min) 'fontified)
		 (point-min)
	       (next-single-property-change (point-min) 'fontified)))
	end)
    (while beg
      (setq end (or (text-property-any beg (point-max) 'fontified nil)
		    (point-max)))
      (push (cons beg end) regions)
      (setq beg (next-single-property-change end 'fontified)))
    (setq regions (nreverse regions)))
  (message "Fontified regions: %s" regions))
(global-set-key [f10] 'get-fontified)

(defun wipe-fontification ()
  "Remove all fontification by switching to fundamental mode and back to C Mode."
  (interactive)
  (fundamental-mode)
  (c-mode))
(global-set-key [f11] 'wipe-fontification)
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

At BOB xdisp.c, press F11, press and hold PageDown, release PageDown
after ~10s, then press F10.  This is what I see:

    Fontified regions: ((1 . 503768))

.  Thus it seems either the command loop or CC Mode is fontifying _every_
position which PageDown is scanning over.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Aborting display. Is this possible?

[David Kastrup]

Alan Mackenzie <acm@muc.de> writes:

> At BOB xdisp.c, press F11, press and hold PageDown, release PageDown
> after ~10s, then press F10.  This is what I see:
>
>     Fontified regions: ((1 . 503768))
>
> .  Thus it seems either the command loop or CC Mode is fontifying _every_
> position which PageDown is scanning over.

How couldn't it?  Fontification affects how much material constitutes
one page full of material.

-- 
David Kastrup

Re: Aborting display. Is this possible?

[Alan Mackenzie]

Hi, David.

On Mon, Oct 20, 2014 at 01:30:02PM +0200, David Kastrup wrote:
> Alan Mackenzie <acm@muc.de> writes:

> > At BOB xdisp.c, press F11, press and hold PageDown, release PageDown
> > after ~10s, then press F10.  This is what I see:

> >     Fontified regions: ((1 . 503768))

> > .  Thus it seems either the command loop or CC Mode is fontifying _every_
> > position which PageDown is scanning over.

> How couldn't it?  Fontification affects how much material constitutes
> one page full of material.

I suppose this is true in the fully general case, when different faces
on a GUI system can have different heights and widths.

Where in the source code, then, is this fontification being done as part
of a scrolling command?

In the usual situation in a programming mode, however, characters are
going to be identical in size.  There must be room for optimisation
somewhere.

> -- 
> David Kastrup

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Aborting display. Is this possible?

[Eli Zaretskii]

> Date: Mon, 20 Oct 2014 12:00:52 +0000
> From: Alan Mackenzie <acm@muc.de>
> Cc: emacs-devel@gnu.org
> 
> > > .  Thus it seems either the command loop or CC Mode is fontifying _every_
> > > position which PageDown is scanning over.
> >
> > How couldn't it?  Fontification affects how much material constitutes
> > one page full of material.
> 
> I suppose this is true in the fully general case, when different faces
> on a GUI system can have different heights and widths.

The display engine doesn't know when it is in the "non-general" case
that allows it to make such shortcuts, except when the frame is a TTY
frame.

> Where in the source code, then, is this fontification being done as part
> of a scrolling command?

It's not part of a scrolling command, it is part of redisplay _after_
the scrolling command finishes.  See my other message with the
detailed description.

> In the usual situation in a programming mode, however, characters are
> going to be identical in size.  There must be room for optimisation
> somewhere.

This is false, for several reasons.

First, the display engine doesn't know anything about the semantics of
the modes -- it doesn't understand how a programming mode is different
from the other kind (and frankly, I'm not sure it is different, given
the wealth of optional display features people tend to use in
programming modes).

Second, even if we would introduce a feature through which a major
mode could tell the display engine that it is a "programming mode",
you are wrong when you assume the characters are identical in size.
Font-lock faces can legitimately be bold or slanted, in which case
their size will be different.  Emacs cannot know whether this is the
case without examining the buffer positions it is about to display.

I suppose we could add a feature whereby a mode or some other Lisp
(e.g., some user-defined code) could tell the display engine that the
buffer contents are "simple" in the sense that all the characters are
of the same size, and then request the optional features that violate
that to again tell the display engine.  But we don't have that now,
and never did.

Re: Aborting display. Is this possible?

[Eli Zaretskii]

> Date: Mon, 20 Oct 2014 11:09:49 +0000
> Cc: emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> On a large C Mode buffer (such as xdisp.c), hold down the PageDown key
> for, say, 10 seconds.
> 
> What is seen: The screen freezes.  After releasing the key, it takes many
> seconds (~10) before the current buffer position is displayed on the
> screen.
> 
> What I'd like to see: The screen continuously updating with (not
> necessarily contiguous) parts of the buffer, the updating being frequent
> enough to give the illusion of continuous scrolling.  On releasing the
> key, the screen "instantly" displaying the new buffer position.

I understand the desire, and I'm not saying it isn't reasonable.  So
please don't take what's below as a rebuttal; I'm just trying to
explain how the current Emacs design works against your wishes, and
what could/should be changed for your wishes to maybe become reality.

I will probably say some (or many) things that you already know, so
please bear with me.

First, Emacs's basic design is that it processes all the available
input before it enters redisplay.  As long as there's input in the
input queue, Emacs will not call redisplay.  (There are 2 exceptions
to this rule: processing input could indirectly call the 'redisplay'
function; and messages in the echo area produced while processing
input require a kind of "partial" redisplay to show the echo area.)
Why? because processing input allows to avoid showing the user a
screen that is outdated from the user's POV, given that input more
often than not changes what should be displayed.

Another part of this puzzle is that redisplay includes fontification
of the visible portion of the buffer.  Emacs applies the font-lock
faces (and any other faces) during the 1st stage of its redisplay
cycle, when it computes the so-called "glyph matrices", which are data
structures maintained for each window that describe what the updated
screen should look like.  As David points out, it is impossible to
compute the glyph matrices without applying the faces, because faces
change the metrics of each character, and thus affect the layout,
which is what glyph matrices are all about.

The final part of the puzzle is what the scroll-up-command does when
Emacs processes a single PageDown keypress.  It computes the new
starting point for the window, which (if we neglect complications) is
simply the beginning of line N screen lines below the original window
start, where N is almost the height of the window.  (Note that I wrote
"screen lines", not "physical lines", so this is not entirely trivial
when there are continuation lines and variable-size fonts.)  Then it
determines whether this change of the window-start requires to move
point to bring it into the view, and if so, moves point as well.

So, after processing a single PageDown key, the window has a new
starting point, and point in that window's buffer has been moved.

If, after processing this single PageDown key, the input queue is
empty (as it is when you lean on the key, because Emacs finishes the
above processing in less time than the auto-repeat interval), Emacs
enters redisplay.  Redisplay computes the glyph matrix of the window
given its start point as set by scroll-up-command; as part of this
matrix computation, the display engine applies all the font-lock
properties to the text, and sets the 'fontified' property on that
text.  Then, just before it is about to redraw the window given the
new glyph matrix, it checks for input.  If there's input available,
and redisplay-dont-pause is nil, the window will not be redrawn,
i.e. this redisplay cycle will be aborted.

Then the same will happen again for the next PageDown key.  The result
is a frozen window display, because all the attempts to redisplay are
aborted due to input rate that is high enough to preempt redisplay,
but not high enough to prevent Emacs from entering redisplay after
every (or almost every) PageDown key.  To see more frequent updates
that give an illusion of scrolling, we would need a feature that
ignores the input availability, or maybe dynamically changes the
number of input events that could be ignored when redisplay falls
behind.  We don't have such a feature.

Now, what happens when you release the key?  The input queue is still
full of unprocessed PageDown keys.  As I explained above, Emacs will
drain the entire input queue before it enters redisplay; thus the long
delay you see after releasing the key.

You want Emacs to "immediately" display the new buffer position, but
this is impossible without some code that would quickly scan the input
queue, analyze what's there, and "understand" that 100 PageDown
keystrokes can be "optimized" by executing a single scroll-up-command
with a suitable argument.  IOW, we would need some input preprocessing
stage that could "optimize" input by replacing a series of commands
with a single command, and do that cheaply.  We don't have such a
feature; patches to add it will probably be welcome.

Re: Aborting display. Is this possible?

[Stefan Monnier]

> If, after processing this single PageDown key, the input queue is
> empty (as it is when you lean on the key, because Emacs finishes the
> above processing in less time than the auto-repeat interval), Emacs
> enters redisplay.

This is the case when processing the scroll-up-command takes less time
than the repeat rate.

> Now, what happens when you release the key?  The input queue is still
> full of unprocessed PageDown keys.

If processing a single scroll-up-command takes less time than the repeat
rate, the input queue should never have many unprocessed PageDown events
(that's the whole reason why we check the input queue before entering
redisplay).

So I still don't understand why we see this long delay when releasing
the key.


        Stefan

Re: Aborting display. Is this possible?

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Alan Mackenzie <acm@muc.de>,  emacs-devel@gnu.org
> Date: Mon, 20 Oct 2014 12:56:56 -0400
> 
> > If, after processing this single PageDown key, the input queue is
> > empty (as it is when you lean on the key, because Emacs finishes the
> > above processing in less time than the auto-repeat interval), Emacs
> > enters redisplay.
> 
> This is the case when processing the scroll-up-command takes less time
> than the repeat rate.

That's what I said, yes.

> > Now, what happens when you release the key?  The input queue is still
> > full of unprocessed PageDown keys.
> 
> If processing a single scroll-up-command takes less time than the repeat
> rate, the input queue should never have many unprocessed PageDown events
> (that's the whole reason why we check the input queue before entering
> redisplay).
> 
> So I still don't understand why we see this long delay when releasing
> the key.

Not every screenful needs the same time to redisplay.  There comes a
screen that needs more, and the queue is no longer empty.

Re: Aborting display. Is this possible?

[Eli Zaretskii]

> Date: Mon, 20 Oct 2014 20:10:07 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: acm@muc.de, emacs-devel@gnu.org
> 
> > > Now, what happens when you release the key?  The input queue is still
> > > full of unprocessed PageDown keys.
> > 
> > If processing a single scroll-up-command takes less time than the repeat
> > rate, the input queue should never have many unprocessed PageDown events
> > (that's the whole reason why we check the input queue before entering
> > redisplay).
> > 
> > So I still don't understand why we see this long delay when releasing
> > the key.
> 
> Not every screenful needs the same time to redisplay.  There comes a
> screen that needs more, and the queue is no longer empty.

In addition, there are 2 places where auto-repeated keystrokes wait:
one is the Emacs keyboard buffer, the other is the system input queue,
since when we process keys we don't call the read_socket_hook.

If you put a breakpoint in kbd_buffer_store_event and
kbd_buffer_get_event, and define breakpoint commands that display the
queue size, you will see that when you release the key, the queue size
decreases to zero, then jumps to a large number, then decreases again,
and finally the window is redisplayed.

To me this means we have a lot of unprocessed keys in both queues.
IOW, when the queue fills up because some screen took too long to
display, Emacs doesn't succeed in emptying it, as long as the key is
pressed.

Re: Aborting display. Is this possible?

[Alan Mackenzie]

Hello, Eli.

On Mon, Oct 20, 2014 at 06:12:05PM +0300, Eli Zaretskii wrote:
> > Date: Mon, 20 Oct 2014 11:09:49 +0000
> > Cc: emacs-devel@gnu.org
> > From: Alan Mackenzie <acm@muc.de

> > On a large C Mode buffer (such as xdisp.c), hold down the PageDown key
> > for, say, 10 seconds.

> > What is seen: The screen freezes.  After releasing the key, it takes many
> > seconds (~10) before the current buffer position is displayed on the
> > screen.

Incidentally, I saw this happening today in emacs-lisp-mode, on my
terminal with a 3 column follow-mode.

> > What I'd like to see: The screen continuously updating with (not
> > necessarily contiguous) parts of the buffer, the updating being frequent
> > enough to give the illusion of continuous scrolling.  On releasing the
> > key, the screen "instantly" displaying the new buffer position.

> I understand the desire, and I'm not saying it isn't reasonable.  So
> please don't take what's below as a rebuttal; I'm just trying to
> explain how the current Emacs design works against your wishes, and
> what could/should be changed for your wishes to maybe become reality.

> I will probably say some (or many) things that you already know, so
> please bear with me.

Thanks for such a clear exposition.

> First, Emacs's basic design is that it processes all the available
> input before it enters redisplay.  As long as there's input in the
> input queue, Emacs will not call redisplay.  (There are 2 exceptions
> to this rule: processing input could indirectly call the 'redisplay'
> function; and messages in the echo area produced while processing
> input require a kind of "partial" redisplay to show the echo area.)
> Why? because processing input allows to avoid showing the user a
> screen that is outdated from the user's POV, given that input more
> often than not changes what should be displayed.

Up to now, I think I've been confusing display with font-locking.
Obviously f-l has to happen when d is being done, but not all
font-locking is caused by display.

> Another part of this puzzle is that redisplay includes fontification
> of the visible portion of the buffer.  Emacs applies the font-lock
> faces (and any other faces) during the 1st stage of its redisplay
> cycle, when it computes the so-called "glyph matrices", which are data
> structures maintained for each window that describe what the updated
> screen should look like.  As David points out, it is impossible to
> compute the glyph matrices without applying the faces, because faces
> change the metrics of each character, and thus affect the layout,
> which is what glyph matrices are all about.

> The final part of the puzzle is what the scroll-up-command does when
> Emacs processes a single PageDown keypress.  It computes the new
> starting point for the window, which (if we neglect complications) is
> simply the beginning of line N screen lines below the original window
> start, where N is almost the height of the window.  (Note that I wrote
> "screen lines", not "physical lines", so this is not entirely trivial
> when there are continuation lines and variable-size fonts.)  Then it
> determines whether this change of the window-start requires to move
> point to bring it into the view, and if so, moves point as well.

On a PageDown, Emacs needs to do font-locking just to work out how far
awy the next page is, like David said.  There is a comment in
window_scroll_line_based: "Fvertical_motion enters redisplay, which can
trigger fontification...".  This is the killer: scrolling a screen of
xdisp.c is taking, on average, ~0.09s, the bulk of this time being
fontification, whereas the key events are coming in every ~0.03s.
Some way has to be found to scroll with only every third screen (at most)
getting font-locked.

If every glyph were assumed to be the same size, the calculation of the
window-start position for the current scroll operation would be _much_
faster: display would still have to account for text lines overflowing
screen lines, and invisible text, and so on, 

So, how about the following strategy: when the (new) variable
`face-instead-of-fontifying' is bound to a face, AND the input queue is
non-empty (or, perhaps, very non-empty), the appropriate subroutine of
window-scroll should deem characters without a `face' (or `fontified' ?)
property to have face `face-instead-of-fontifying'.

This should empty the input queue pretty quickly, enabling a current
buffer portion to get displayed frequently enough.

> So, after processing a single PageDown key, the window has a new
> starting point, and point in that window's buffer has been moved.

> If, after processing this single PageDown key, the input queue is
> empty (as it is when you lean on the key, because Emacs finishes the
> above processing in less time than the auto-repeat interval), Emacs
> enters redisplay.  Redisplay computes the glyph matrix of the window
> given its start point as set by scroll-up-command; as part of this
> matrix computation, the display engine applies all the font-lock
> properties to the text, and sets the 'fontified' property on that
> text.  Then, just before it is about to redraw the window given the
> new glyph matrix, it checks for input.  If there's input available,
> and redisplay-dont-pause is nil, the window will not be redrawn,
> i.e. this redisplay cycle will be aborted.

But in the default situation (redisplay-dont-pause is non-nil), once the
calculation of the glyph matrix has begun, the redrawing necessarily
takes place.  This is good.  :-)

> Then the same will happen again for the next PageDown key.  The result
> is a frozen window display, because all the attempts to redisplay are
> aborted due to input rate that is high enough to preempt redisplay,
> but not high enough to prevent Emacs from entering redisplay after
> every (or almost every) PageDown key.  To see more frequent updates
> that give an illusion of scrolling, we would need a feature that
> ignores the input availability, or maybe dynamically changes the
> number of input events that could be ignored when redisplay falls
> behind.  We don't have such a feature.

Is this really the case?  Is it not that the font-locking caused by
scrolling is slow enough, that the input queue is never empty, hence
redisplay doesn't get a look in?

> Now, what happens when you release the key?  The input queue is still
> full of unprocessed PageDown keys.  As I explained above, Emacs will
> drain the entire input queue before it enters redisplay; thus the long
> delay you see after releasing the key.

And each such PageDown key will necessitate full font-locking for its
never-to-be-displayed screen.

> You want Emacs to "immediately" display the new buffer position, but
> this is impossible without some code that would quickly scan the input
> queue, analyze what's there, and "understand" that 100 PageDown
> keystrokes can be "optimized" by executing a single scroll-up-command
> with a suitable argument.  IOW, we would need some input preprocessing
> stage that could "optimize" input by replacing a series of commands
> with a single command, and do that cheaply.  We don't have such a
> feature; patches to add it will probably be welcome.

:-).  If my analysis is right, this wouldn't help.  A single mega-scroll
would still be font-locking the entire intermediate buffer region.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Aborting display. Is this possible?

[Eli Zaretskii]

> Date: Mon, 20 Oct 2014 18:57:57 +0000
> From: Alan Mackenzie <acm@muc.de>
> Cc: emacs-devel@gnu.org
> 
> On a PageDown, Emacs needs to do font-locking just to work out how far
> awy the next page is, like David said.  There is a comment in
> window_scroll_line_based: "Fvertical_motion enters redisplay, which can
> trigger fontification...".  This is the killer: scrolling a screen of
> xdisp.c is taking, on average, ~0.09s, the bulk of this time being
> fontification, whereas the key events are coming in every ~0.03s.
> Some way has to be found to scroll with only every third screen (at most)
> getting font-locked.

window_scroll_line_based is not relevant to GUI frames, only to
text-mode frames.  And window_scroll_pixel_based does not call
vertical-motion.

Also, "enters redisplay" needs to be taken with a grain of salt here.
The reference is to display "simulation", whereby Emacs proceeds
through buffer text line after line, computing the metrics of the
characters as it goes (and invoking fontification functions if
needed), but doesn't store the results in glyph matrices and doesn't
proceed to redrawing based on that.  So it's only partial redisplay.
The full redisplay will be done later, after the scrolling command did
its part, and Emacs is back in the command loop with no available
input.

> If every glyph were assumed to be the same size, the calculation of the
> window-start position for the current scroll operation would be _much_
> faster: display would still have to account for text lines overflowing
> screen lines, and invisible text, and so on, 

Yes, but we've lost that paradise when we introduced variable-font
support in Emacs 21.

> So, how about the following strategy: when the (new) variable
> `face-instead-of-fontifying' is bound to a face, AND the input queue is
> non-empty (or, perhaps, very non-empty), the appropriate subroutine of
> window-scroll should deem characters without a `face' (or `fontified' ?)
> property to have face `face-instead-of-fontifying'.
> 
> This should empty the input queue pretty quickly, enabling a current
> buffer portion to get displayed frequently enough.

This would mean we will sometimes wind up in the wrong place after
scroll.  IOW, you are sacrificing correctness on behalf of some
(questionable for now) speedup.  I say ""questionable" because it's
not clear to me that using just one fixed face will significantly
speed up the process of painstakingly moving through text line by
line.

> But in the default situation (redisplay-dont-pause is non-nil), once the
> calculation of the glyph matrix has begun, the redrawing necessarily
> takes place.

Yes, once.  After that, if input events come in faster than Emacs can
process them, you again get a frozen outdated window.

There isn't really a good solution for when input comes in faster than
Emacs can process it.

> > Then the same will happen again for the next PageDown key.  The result
> > is a frozen window display, because all the attempts to redisplay are
> > aborted due to input rate that is high enough to preempt redisplay,
> > but not high enough to prevent Emacs from entering redisplay after
> > every (or almost every) PageDown key.  To see more frequent updates
> > that give an illusion of scrolling, we would need a feature that
> > ignores the input availability, or maybe dynamically changes the
> > number of input events that could be ignored when redisplay falls
> > behind.  We don't have such a feature.
> 
> Is this really the case?  Is it not that the font-locking caused by
> scrolling is slow enough, that the input queue is never empty, hence
> redisplay doesn't get a look in?

It eventually gets to that, because some screenfuls need more time to
redisplay.

> > Now, what happens when you release the key?  The input queue is still
> > full of unprocessed PageDown keys.  As I explained above, Emacs will
> > drain the entire input queue before it enters redisplay; thus the long
> > delay you see after releasing the key.
> 
> And each such PageDown key will necessitate full font-locking for its
> never-to-be-displayed screen.

No, not full, only partial.  See above.

> > You want Emacs to "immediately" display the new buffer position, but
> > this is impossible without some code that would quickly scan the input
> > queue, analyze what's there, and "understand" that 100 PageDown
> > keystrokes can be "optimized" by executing a single scroll-up-command
> > with a suitable argument.  IOW, we would need some input preprocessing
> > stage that could "optimize" input by replacing a series of commands
> > with a single command, and do that cheaply.  We don't have such a
> > feature; patches to add it will probably be welcome.
> 
> :-).  If my analysis is right, this wouldn't help.  A single mega-scroll
> would still be font-locking the entire intermediate buffer region.

Again, only partially so.  And please note that scrolling is
inherently at disadvantage here, because it tells Emacs to move N
screen lines up/down, so Emacs must first figure out where that lands
it.  Other commands that move through buffer text, like goto-char,
don't have that problem.

Re: Aborting display. Is this possible?

[Alan Mackenzie]

Hi, Eli.

On Mon, Oct 20, 2014 at 10:24:54PM +0300, Eli Zaretskii wrote:
> > Date: Mon, 20 Oct 2014 18:57:57 +0000
> > From: Alan Mackenzie <acm@muc.de>
> > Cc: emacs-devel@gnu.org

> > On a PageDown, Emacs needs to do font-locking just to work out how far
> > awy the next page is, like David said.  There is a comment in
> > window_scroll_line_based: "Fvertical_motion enters redisplay, which can
> > trigger fontification...".  This is the killer: scrolling a screen of
> > xdisp.c is taking, on average, ~0.09s, the bulk of this time being
> > fontification, whereas the key events are coming in every ~0.03s.
> > Some way has to be found to scroll with only every third screen (at most)
> > getting font-locked.

> window_scroll_line_based is not relevant to GUI frames, only to
> text-mode frames.  And window_scroll_pixel_based does not call
> vertical-motion.

OK.  But the problem occurs both in GUI and tty.  There'll be some C
function at a deeper level where the GUI and tty cases meet up.  The tty
case is not going to be more difficult to solve than the GUI one.  [ By
the way, it's brilliant how effectively the difference between them has
been abstracted away for elisp programming. ]

> Also, "enters redisplay" needs to be taken with a grain of salt here.
> The reference is to display "simulation", whereby Emacs proceeds
> through buffer text line after line, computing the metrics of the
> characters as it goes (and invoking fontification functions if
> needed), but doesn't store the results in glyph matrices and doesn't
> proceed to redrawing based on that.  So it's only partial redisplay.
> The full redisplay will be done later, after the scrolling command did
> its part, and Emacs is back in the command loop with no available
> input.

Yes, I've got that.  But in CC Mode, it's the fontification which takes
most of the time (see also below).

> > If every glyph were assumed to be the same size, the calculation of the
> > window-start position for the current scroll operation would be _much_
> > faster: display would still have to account for text lines overflowing
> > screen lines, and invisible text, and so on, 

> Yes, but we've lost that paradise when we introduced variable-font
> support in Emacs 21.

> > So, how about the following strategy: when the (new) variable
> > `face-instead-of-fontifying' is bound to a face, AND the input queue is
> > non-empty (or, perhaps, very non-empty), the appropriate subroutine of
> > window-scroll should deem characters without a `face' (or `fontified' ?)
> > property to have face `face-instead-of-fontifying'.

> > This should empty the input queue pretty quickly, enabling a current
> > buffer portion to get displayed frequently enough.

> This would mean we will sometimes wind up in the wrong place after
> scroll.  IOW, you are sacrificing correctness on behalf of some
> (questionable for now) speedup.

I'd posit that the absolute correctness isn't all that important after an
auto-repeating PageDown.  For an individual PageDown keystroke, the
fontification and scrolling would be done accurately as now.

> I say ""questionable" because it's not clear to me that using just one
> fixed face will significantly speed up the process of painstakingly
> moving through text line by line.

OK: some numbers, on emacs-24 -Q in both cases:

To scroll backwards on a tty with a 64 line window through xdisp.c takes
0.106s per screen (on average).  Doing the same on an already fontified
buffer takes 0.0128s per screen (on average).  It would seem that the
fontification is taking ~87% of the time.  (See the thread "Scrolling
xdisp.c.  Room for optimisation in syntax.c/CC Mode." for details of the
measurement.)

The same on a GUI (in X-Windows on a GNU system, 33 line window):
0.05268s per screen, fresh; 0.00655 per screen, starting fully fontified.
Fontification is taking a similar proportion of the total time.

> > But in the default situation (redisplay-dont-pause is non-nil), once the
> > calculation of the glyph matrix has begun, the redrawing necessarily
> > takes place.

> Yes, once.  After that, if input events come in faster than Emacs can
> process them, you again get a frozen outdated window.

> There isn't really a good solution for when input comes in faster than
> Emacs can process it.

Er, making Emacs process the input faster?

> > > Then the same will happen again for the next PageDown key.  The result
> > > is a frozen window display, because all the attempts to redisplay are
> > > aborted due to input rate that is high enough to preempt redisplay,
> > > but not high enough to prevent Emacs from entering redisplay after
> > > every (or almost every) PageDown key.  To see more frequent updates
> > > that give an illusion of scrolling, we would need a feature that
> > > ignores the input availability, or maybe dynamically changes the
> > > number of input events that could be ignored when redisplay falls
> > > behind.  We don't have such a feature.

> > Is this really the case?  Is it not that the font-locking caused by
> > scrolling is slow enough, that the input queue is never empty, hence
> > redisplay doesn't get a look in?

> It eventually gets to that, because some screenfuls need more time to
> redisplay.

I think it gets to that stage very early indeed.

> > > Now, what happens when you release the key?  The input queue is still
> > > full of unprocessed PageDown keys.  As I explained above, Emacs will
> > > drain the entire input queue before it enters redisplay; thus the long
> > > delay you see after releasing the key.

> > And each such PageDown key will necessitate full font-locking for its
> > never-to-be-displayed screen.

> No, not full, only partial.  See above.

I don't see the "above" here.  When I tested the `fontified' property
this morning, EVERY position from point-min to point had been fontified.

> > > You want Emacs to "immediately" display the new buffer position, but
> > > this is impossible without some code that would quickly scan the input
> > > queue, analyze what's there, and "understand" that 100 PageDown
> > > keystrokes can be "optimized" by executing a single scroll-up-command
> > > with a suitable argument.  IOW, we would need some input preprocessing
> > > stage that could "optimize" input by replacing a series of commands
> > > with a single command, and do that cheaply.  We don't have such a
> > > feature; patches to add it will probably be welcome.

> > :-).  If my analysis is right, this wouldn't help.  A single mega-scroll
> > would still be font-locking the entire intermediate buffer region.

> Again, only partially so.

I still don't see the "partially".  Why wouldn't everything get fontified
again (for the calculations for scrolling, not for display)?

> And please note that scrolling is inherently at disadvantage here,
> because it tells Emacs to move N screen lines up/down, so Emacs must
> first figure out where that lands it.  Other commands that move through
> buffer text, like goto-char, don't have that problem.

Yes.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Aborting display. Is this possible?

[David Kastrup]

Alan Mackenzie <acm@muc.de> writes:

> I'd posit that the absolute correctness isn't all that important after an
> auto-repeating PageDown.

That would mean that if you lean on PageDown, something catches your eye
while leafing through, you stop and press PageUp a few times in order to
locate it, it won't reappear where it caught your eye.

-- 
David Kastrup

Re: Aborting display. Is this possible?

[Alan Mackenzie]

Hello, David.

On Tue, Oct 21, 2014 at 10:09:18AM +0200, David Kastrup wrote:
> Alan Mackenzie <acm@muc.de> writes:

> > I'd posit that the absolute correctness isn't all that important after an
> > auto-repeating PageDown.

> That would mean that if you lean on PageDown, something catches your eye
> while leafing through, you stop and press PageUp a few times in order to
> locate it, it won't reappear where it caught your eye.

Yes, possibly.  That's if the "default" face used to calculate positions
and occupancy is grossly bigger or smaller than the typical face.  But
since you've stopped very soon after seeing FOO, the discrepancy
typically shouldn't be too large.  The other answer is that since in
this sort of scrolling only a small proportion of screens actually get
displayed (see below), the user will soon stop expecting to be able to
eyeball a passing function declaration.

At any rate, that situation would be an improvement on the current
arrangement, where leaning on PageDown (with font-lock enabled) leads to
a frozen screen, which stays frozen long after the key is released.

Should the region of the buffer already be fontified (or font-locking
disabled), the proposed new behaviour would be the same as the current.

I've done some more measurements on my system:

Autorepeat: delay between 1st event and 2nd event: 0.168s.
            delay between subsequent consecutive events: 0.024s.
Scrolling operation on non-fontified 65 line window with xdisp.c:
            mean time: 0.106s  standard deviation: 0.046s
Scrolling operation on already fontified 65 line window with xdisp.c:
            mean time: 0.011s  standard deviation: 0.0023s

By subtracting the above two, the mean time to fontify a screen is
0.095s, to display a fontified screen is 0.011s.

Assume that the time to calculate a scrolled position, without
fontification, is less than 0.011s.  Assume 0.010s.

So, by my new scheme, a typical timeline would be:

0.0s   Arrival of PageDown event   Event queue is empty, so start of
                                     fontification of screen 2.
0.095s	                           Fontification done; start of display.
0.106s                             Display complete.  Queue is empty.

0.192s Arrival of PageDown event   Event queue is empty, so start of
                                     fontification of screen 3.
0.216s Arrival of PageDown event   Queued (1)
0.240s Arrival of PageDown event   Queued (2)
0.264s Arrival of PageDown event   Queued (3)
0.287s                             Fontification done; start of display
0.288s Arrival of PageDown event   Queued (4)
0.298s                             Display of screen 3 complete.  Start
                                     scroll command.  Queue is
				     non-empty (3), so start calculating
				     new position for screen 4 without
				     fontification.
0.312s Arrival of PageDown event   Queued (4)				     
0.318s                             Finished calculation of screen 4's
                                     position.  Start scroll command.
				     Queue is 3; start calculating new
				     screen 5 position without
				     fontification.
0.328s                             Finished calculation for scree 5.
                                     Queue is 2. Start calc for screen
				     6.
0.336s Arrival of PageDown event   Queued (3)
0.338s                             Finished calc for screen 6.  Queue is
                                     2.  Start calc for screen 7.
0.348s                             Finished calc for screen 7.  Queue is
                                     1.  Start calc for screen 8.
0.358s                             Finished calc for screen 8.  Queue is
				     empty.  Start fontification for
				     screen 9.
				   
0.360s Arrival of PageDown event   Queued (1)
0.384s Arrival of PageDown event   Queued (2)
0.408s Arrival of PageDown event   Queued (3)
0.432s Arrival of PageDown event   Queued (4)
0.453s                             Fontification of screen 9 done.
                                     Start displaying it.
0.456s Arrival of PageDown event   Queued (5)
0.464s                             Display of screen 9 complete.  Start
                                     scroll command.  Queue is 4, start
				     calculating position for screen 10.
0.475s                             Finished calc for screen 10.  Queue
                                     is 3.  Start calc for screen 11.
0.480s Arrival of PageDown event   Queued (4)
0.486s                             Finished calc for screen 11.  Queue
                                     is 3.  Start calc for screen 12.
0.497s                             Finished calc for screen 12.  Queue
                                     is 2.  Start calc for screen 13.
0.504s Arrival of PageDown event   Queued (3)
0.508s                             Finished calc for screen 13.  Queue
                                     is 2.  Start calc for screen 14.
0.519s                             Finished calc for screen 14.  Queue
                                     is 1.  Start calc for screen 15.
0.528s Arrival of PageDown event   Queued (2)
0.530s                             Finished calc for screen 15.  Queue
                                     is 1.  Start calc for screen 16.
0.541s                             Finished calc for screen 16.  Queue
                                     is empty.  Start fontification for
				     screen 17.
...
0.636s                             Finished fontifying of screen 17.
                                     Start displaying it.
...
0.647s                             Display of screen 17 complete.

------------------------------------------------------------------------

So it would appear in a worst case scenario (xdisp.c in a 65 line
window), about 5 screens per second would be displayed, approximately
every 8 screens.

> -- David Kastrup

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Aborting display. Is this possible?

[David Kastrup]

Alan Mackenzie <acm@muc.de> writes:

> I've done some more measurements on my system:
>
> Autorepeat: delay between 1st event and 2nd event: 0.168s.
>             delay between subsequent consecutive events: 0.024s.

That seems rather excessive.  Usual I remember is 0.5/0.05, and that
already requires good aim.

-- 
David Kastrup

Re: Aborting display. Is this possible?

[Stefan Monnier]

>> I've done some more measurements on my system:
>> Autorepeat: delay between 1st event and 2nd event: 0.168s.
>> delay between subsequent consecutive events: 0.024s.
> That seems rather excessive.  Usual I remember is 0.5/0.05, and that
> already requires good aim.

FWIW, I use "xset r 200 40", i.e. 0.2s/0.025s, so Alan's settings don't
seem outlandish to me.


        Stefan

Re: Aborting display. Is this possible?

[Stefan Monnier]

>> I'd posit that the absolute correctness isn't all that important after an
>> auto-repeating PageDown.
> That would mean that if you lean on PageDown, something catches your eye
> while leafing through, you stop and press PageUp a few times in order to
> locate it, it won't reappear where it caught your eye.

Could be, but it's highly unlikely: if it caught your eye, it means that
redisplay is fast enough to keep up with scrolling, in which case
scrolling gets to use a properly fontified buffer for its calculation,
so Alan's suggestion should make no difference in that case.


        Stefan

Re: Aborting display. Is this possible?

[Eli Zaretskii]

> From: Stefan Monnier <monnier@IRO.UMontreal.CA>
> Date: Tue, 21 Oct 2014 10:01:43 -0400
> Cc: emacs-devel@gnu.org
> 
> >> I'd posit that the absolute correctness isn't all that important after an
> >> auto-repeating PageDown.
> > That would mean that if you lean on PageDown, something catches your eye
> > while leafing through, you stop and press PageUp a few times in order to
> > locate it, it won't reappear where it caught your eye.
> 
> Could be, but it's highly unlikely: if it caught your eye, it means that
> redisplay is fast enough to keep up with scrolling, in which case
> scrolling gets to use a properly fontified buffer for its calculation,
> so Alan's suggestion should make no difference in that case.

In any case, the way Alan suggests is not TRT, IMNSHO.  We shouldn't
be sacrificing display correctness for speed; AFAIR, we never did
anything like that.  What we did was find and implement optimizations
that catered to specific use cases that were deemed important enough.
I see no reason to treat this case differently.

Another data point is this: in the recent years, I've dealt with any
number of bug reports that complained about minor inaccuracies in
scrolling, mostly by fractions of line height or even by a couple of
pixels.  My interpretation of this is that our users expect accurate
display, and will not easily tolerate sloppiness.

So I suggest to turn the table and ask: how come fontification of C
sources is so expensive, and how can we make it faster?  If the Lisp
implementation cannot be sped up significantly, let's implement some
of the code in C.  Or maybe we are should request less accurate
parsing of the source, at least by default -- e.g., perhaps it is not
very important to display variables in a face that is different from,
say, data types?

IOW, if the measurements show that redisplay takes 10% of the time,
let's not try to optimize those 10%, but instead concentrate on the
90%.

Re: Aborting display. Is this possible?

[Stefan Monnier]

> So I suggest to turn the table and ask: how come fontification of C
> sources is so expensive, and how can we make it faster?

Indeed, that'd be better.  But IIUC Alan has already been working fairly
hard to make CC-mode's parsing faster.  So my guess is that either it's
just really hard because of some details of the C language, or we'd need
a serious restructuring/rewrite and I don't think anyone is up for that.

> IOW, if the measurements show that redisplay takes 10% of the time,
> let's not try to optimize those 10%, but instead concentrate on the
> 90%.

There's some general pressure to improve font-locking by making it more
precise, with somewhat marginal improvements but fairly high
processing costs (pushed to the extreme, this involves passing the text
to an external toolchain that parses and type-checks and returns to
Emacs the precise highlighting info; see for example agda-mode).

So, maybe we're looking at a future where we'll need to provide some way
for delayed/asynchronous font-lock refinement.


        Stefan

Re: Aborting display. Is this possible?

[Stephen Leake]

Stefan Monnier <monnier@IRO.UMontreal.CA> writes:

> There's some general pressure to improve font-locking by making it more
> precise, with somewhat marginal improvements but fairly high
> processing costs (pushed to the extreme, this involves passing the text
> to an external toolchain that parses and type-checks and returns to
> Emacs the precise highlighting info; see for example agda-mode).

Did you mean "ada-mode" here? I don't find 'agda-mode' in 24.3.94 or
ELPA. I'm still having trouble with process communication, so I'll be
trying the new FFI mechanism next.

But your point is valid; I much prefer faster font-lock, rather than
less precise font-lock. I'm not sure my customers really agree; they
have not spoken up (other than one complaint that it's too slow on huge
files).

> So, maybe we're looking at a future where we'll need to provide some way
> for delayed/asynchronous font-lock refinement.

Ah, like a jpeg sent over a slow internet connection (at least, that's
they way I remember it; haven't run into that in years).

Perhaps it would be possible to define a subset of the full grammar,
that provides some meaningful level of font-lock, and is significantly
faster to parse. That would be a lot of work, but I may have to try that
if all else fails.

-- 
-- Stephe

Re: Aborting display. Is this possible?

[Stefan Monnier]

> Did you mean "ada-mode" here?

No, agda-mode: http://wiki.portal.chalmers.se/agda/agda.php


        Stefan

Re: Aborting display. Is this possible?

[Alan Mackenzie]

Hi, Eli.

On Tue, Oct 21, 2014 at 06:35:23PM +0300, Eli Zaretskii wrote:
> > From: Stefan Monnier <monnier@IRO.UMontreal.CA>
> > Date: Tue, 21 Oct 2014 10:01:43 -0400
> > Cc: emacs-devel@gnu.org

[ ... ]

> In any case, the way Alan suggests is not TRT, IMNSHO.  We shouldn't
> be sacrificing display correctness for speed; AFAIR, we never did
> anything like that.  What we did was find and implement optimizations
> that catered to specific use cases that were deemed important enough.
> I see no reason to treat this case differently.

> Another data point is this: in the recent years, I've dealt with any
> number of bug reports that complained about minor inaccuracies in
> scrolling, mostly by fractions of line height or even by a couple of
> pixels.  My interpretation of this is that our users expect accurate
> display, and will not easily tolerate sloppiness.

This is a red herring, surely?  Tiny fractions, visible, of the line
height are totally orthogonal to being one or two lines out (which won't
even be visible) after a "sloppy" user action.

> So I suggest to turn the table and ask: how come fontification of C
> sources is so expensive, and how can we make it faster?

It has got slower as it has become more accurate.  Users have complained
about slight inaccuracies in fontification, too.  Fast enough would mean
speeding it up by a factor of around 5.  I don't think this is
practicable.

A couple of days ago, I even got auto-repeated PageDown to hang in Emacs
LIsp mode.  I can't reproduce that at the moment, though.  But I applied
my timing program to cc-engine.el, and its average time to scroll and
display a screen was 0.018s compared with an auto-repeat rate of 0.024s.
That's not a lot of free play.

> If the Lisp implementation cannot be sped up significantly, let's
> implement some of the code in C.  Or maybe we are should request less
> accurate parsing of the source, at least by default -- e.g., perhaps it
> is not very important to display variables in a face that is different
> from, say, data types?

You're suggesting sacrificing fontification accuracy for scrolling
accuracy?  :-)

> IOW, if the measurements show that redisplay takes 10% of the time,
> let's not try to optimize those 10%, but instead concentrate on the
> 90%.

A very common use case for programming languages is when all the
font-lock-* faces are mono-spaced of equal size.  This might even be the
overwhelming majority of use.  Yet the scrolling/display code makes no
optimisation for this.  In this very common case, it is entirely
unnecessary to fontify bits of buffer to work out how much text would fit
onto the screen we want to scroll over.

The whole point of Jit Lock was that we would only need to fontify the
bits of the buffer currently being displayed.  This was partly so that
mode maintainers could be a bit more relaxed about the speed of
font-locking.  The scenario of auto-repeat on PageDown suggests that the
mechanism wasn't fully thought out.

I still say an optimisation to the scrolling code, where an option could
specify that all faces are of the same size (or should be deemed to be
so) is called for here.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Aborting display. Is this possible?

[Eli Zaretskii]

> Date: Tue, 21 Oct 2014 17:14:03 +0000
> Cc: Stefan Monnier <monnier@IRO.UMontreal.CA>, dak@gnu.org,
>   emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> > Another data point is this: in the recent years, I've dealt with any
> > number of bug reports that complained about minor inaccuracies in
> > scrolling, mostly by fractions of line height or even by a couple of
> > pixels.  My interpretation of this is that our users expect accurate
> > display, and will not easily tolerate sloppiness.
> 
> This is a red herring, surely?

No, it's a data point.  And you already got one other opinion along
these same lines.

> Tiny fractions, visible, of the line height are totally orthogonal
> to being one or two lines out (which won't even be visible) after a
> "sloppy" user action.

I ensure you they _will_ be visible.  Our users are very sharp-eyed.

> > So I suggest to turn the table and ask: how come fontification of C
> > sources is so expensive, and how can we make it faster?
> 
> It has got slower as it has become more accurate.  Users have complained
> about slight inaccuracies in fontification, too.  Fast enough would mean
> speeding it up by a factor of around 5.  I don't think this is
> practicable.

Of course, it's practicable!  E.g., make it by default do what CC Mode
did a few years ago.  There was time when I could scroll through
xdisp.c on a much slower machine I have today, and Emacs would keep
up, you know.  Let users who want "more accurate fontification" ask
for it, and pay the price!

I'm not saying that's necessarily what we should do, but it surely is
one way.

> > If the Lisp implementation cannot be sped up significantly, let's
> > implement some of the code in C.  Or maybe we are should request less
> > accurate parsing of the source, at least by default -- e.g., perhaps it
> > is not very important to display variables in a face that is different
> > from, say, data types?
> 
> You're suggesting sacrificing fontification accuracy for scrolling
> accuracy?  :-)

That's a possibility, yes.  Fontification doesn't have to be as
accurate as the compiler.

> A very common use case for programming languages is when all the
> font-lock-* faces are mono-spaced of equal size.  This might even be the
> overwhelming majority of use.  Yet the scrolling/display code makes no
> optimisation for this.  In this very common case, it is entirely
> unnecessary to fontify bits of buffer to work out how much text would fit
> onto the screen we want to scroll over.

We've been through that already: the problem is how does the display
engine _know_ this is the case, without examining the buffer and the
text properties?  Suggest a reliable and easy to use mechanism for
that, and you've got my vote.

But assuming without any grounds that this is the situation, based on
just the mode, is a non-starter.  It will wreak havoc on many widely
used support modes.

> The whole point of Jit Lock was that we would only need to fontify the
> bits of the buffer currently being displayed.  This was partly so that
> mode maintainers could be a bit more relaxed about the speed of
> font-locking.  The scenario of auto-repeat on PageDown suggests that the
> mechanism wasn't fully thought out.

Well, think it out better, if you can.  Suggestions for algorithmic
improvements of the display engine are most welcome.

> I still say an optimisation to the scrolling code, where an option could
> specify that all faces are of the same size (or should be deemed to be
> so) is called for here.

Details, please!  What option, how it will be used, and how do we
ensure it is not abused -- these are the first issues I'd like to
hear.

And I still think you are barking up the wrong tree: the CC Mode is
slow, so it is the one that needs to be worked on.

Re: Aborting display. Is this possible?

[Alan Mackenzie]

Hello, Eli.

On Tue, Oct 21, 2014 at 09:00:03PM +0300, Eli Zaretskii wrote:
> > Date: Tue, 21 Oct 2014 17:14:03 +0000
> > Cc: Stefan Monnier <monnier@IRO.UMontreal.CA>, dak@gnu.org,
> >   emacs-devel@gnu.org
> > From: Alan Mackenzie <acm@muc.de>

> > I still say an optimisation to the scrolling code, where an option could
> > specify that all faces are of the same size (or should be deemed to be
> > so) is called for here.

> Details, please!  What option, how it will be used, and how do we
> ensure it is not abused -- these are the first issues I'd like to
> hear.

The option (haven't got a good name for it yet, so let's just call it
"Foo") would cause the display engine to bypass fontification whenever
the event queue is non-empty.  Instead, it would assume all characters
without a `face' property had the Foo property.

Some working code would be a better thing than mere words.  I'm going to
try and hack out a proof of concept.

> And I still think you are barking up the wrong tree: the CC Mode is
> slow, so it is the one that needs to be worked on.

That as well.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Aborting display. Is this possible?

[Eli Zaretskii]

> Date: Tue, 21 Oct 2014 18:38:07 +0000
> Cc: monnier@IRO.UMontreal.CA, dak@gnu.org, emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> The option (haven't got a good name for it yet, so let's just call it
> "Foo") would cause the display engine to bypass fontification whenever
> the event queue is non-empty.

Don't forget that as long as redisplay is working, we don't call
read_socket_hook, so the queue is not being filled up.  IOW, more
input might be available, just not yet known to Emacs.

> Instead, it would assume all characters without a `face' property
> had the Foo property.

Based on what?  Just on the fact that there's some input?  And who and
how will set and reset that option, and under what circumstances?

Re: Aborting display. Is this possible?

[Eli Zaretskii]

> Date: Tue, 21 Oct 2014 21:43:49 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: dak@gnu.org, monnier@IRO.UMontreal.CA, emacs-devel@gnu.org
> 
> > Instead, it would assume all characters without a `face' property
> > had the Foo property.
> 
> Based on what?  Just on the fact that there's some input?  And who and
> how will set and reset that option, and under what circumstances?

Btw, if all you want is prevent unnecessary fontification of the parts
that will not be displayed, there's a much easier way: use a command
that moves point directly, rather than indirectly by moving the
window-start.  IOW, bind C-v to a function that moves point N physical
lines, where N is computed from the window height.  That's it! no
fancy new options and redisplay changes are necessary.

Scrolling is more expensive because it tries to be accurate: it tries
to move the text by almost a full screen, with a small overlap whose
size is known in advance and customizable to users' content.  If you
are willing to accept inaccuracy, just move point instead.

Unfreezing the display during auto-repeated scrolling. [ Was: Aborting display. Is this possible? ]

[Alan Mackenzie]

Hello, Eli.

On Tue, Oct 21, 2014 at 06:38:07PM +0000, Alan Mackenzie wrote:
> On Tue, Oct 21, 2014 at 09:00:03PM +0300, Eli Zaretskii wrote:
> > > Date: Tue, 21 Oct 2014 17:14:03 +0000
> > > Cc: Stefan Monnier <monnier@IRO.UMontreal.CA>, dak@gnu.org,
> > >   emacs-devel@gnu.org
> > > From: Alan Mackenzie <acm@muc.de>

> > > I still say an optimisation to the scrolling code, where an option could
> > > specify that all faces are of the same size (or should be deemed to be
> > > so) is called for here.

> > Details, please!  What option, how it will be used, and how do we
> > ensure it is not abused -- these are the first issues I'd like to
> > hear.

> The option (haven't got a good name for it yet, so let's just call it
> "Foo") would cause the display engine to bypass fontification whenever
> the event queue is non-empty.  Instead, it would assume all characters
> without a `face' property had the Foo property.

> Some working code would be a better thing than mere words.  I'm going to
> try and hack out a proof of concept.

OK, I've got some working code.  It's currently working ONLY on ttys.  To
try it out, set the new variable `use-default-face-for-fast-scrolling' to
non-nil, then auto-repeat on PageDown or PageUp through a large C file
(like xdisp.c).

Here's what I've done:  Conceptually, ITs (i.e. "struct it"s, the
iterators in xdisp.c) are used for two different purposes:
(i) display;
(ii) calculation of displayed character sizes, positions etc.
I've distinguished these two cases by adding an extra boolean flag to IT,
called `not_for_display'.  The functions `init_iterator' and
`start_display' have got this as an extra boolean parameter.

Some of the ITs in window.c and xdisp.c are now designated
`not_for_display'.

`handle_fontified_prop' now checks this flag in IT.  If both the flag and
`use-default-face-for-fast-scrolling' are set, then fontification is not
done.  The existing face (usually default) is simply left there.  The
code doesn't also check for the event queue being non-empty - in
practice, this didn't work very well.

Here's the code.  Comments and criticism would be welcome.



=== modified file 'src/dispextern.h'
--- src/dispextern.h	2014-10-15 10:22:15 +0000
+++ src/dispextern.h	2014-10-26 10:32:37 +0000
@@ -2275,6 +2275,12 @@
   /* True means cursor shouldn't be displayed here.  */
   bool_bf avoid_cursor_p : 1;
 
+  /* True means it is known that IT is only for calculating screen
+     occupancy, not for actually drawing elements on a screen.  This
+     suppresses fontification when user option
+     `use-default-face-for-fast-scrolling' is set. */
+  bool_bf not_for_display : 1;
+
   /* Display table in effect or null for none.  */
   struct Lisp_Char_Table *dp;
 
@@ -3204,10 +3210,10 @@
 void mark_window_display_accurate (Lisp_Object, int);
 void redisplay_preserve_echo_area (int);
 void init_iterator (struct it *, struct window *, ptrdiff_t,
-                    ptrdiff_t, struct glyph_row *, enum face_id);
+                    ptrdiff_t, struct glyph_row *, enum face_id, bool);
 void init_iterator_to_row_start (struct it *, struct window *,
                                  struct glyph_row *);
-void start_display (struct it *, struct window *, struct text_pos);
+void start_display (struct it *, struct window *, struct text_pos, bool);
 void move_it_vertically (struct it *, int);
 void move_it_vertically_backward (struct it *, int);
 void move_it_by_lines (struct it *, ptrdiff_t);

=== modified file 'src/dispnew.c'
--- src/dispnew.c	2014-10-14 12:45:41 +0000
+++ src/dispnew.c	2014-10-22 11:15:50 +0000
@@ -5131,7 +5131,7 @@
   Fset_buffer (w->contents);
   itdata = bidi_shelve_cache ();
   CLIP_TEXT_POS_FROM_MARKER (startp, w->start);
-  start_display (&it, w, startp);
+  start_display (&it, w, startp, 0);
   x0 = *x;
 
   /* First, move to the beginning of the row corresponding to *Y.  We

=== modified file 'src/indent.c'
--- src/indent.c	2014-08-28 01:59:29 +0000
+++ src/indent.c	2014-10-22 11:17:10 +0000
@@ -2009,7 +2009,7 @@
 
       itdata = bidi_shelve_cache ();
       SET_TEXT_POS (pt, PT, PT_BYTE);
-      start_display (&it, w, pt);
+      start_display (&it, w, pt, 1);
       first_x = it.first_visible_x;
       it_start = IT_CHARPOS (it);
 

=== modified file 'src/window.c'
--- src/window.c	2014-10-04 08:20:24 +0000
+++ src/window.c	2014-10-26 11:18:59 +0000
@@ -1799,7 +1799,7 @@
       CLIP_TEXT_POS_FROM_MARKER (startp, w->start);
 
       itdata = bidi_shelve_cache ();
-      start_display (&it, w, startp);
+      start_display (&it, w, startp, 0);
       move_it_vertically (&it, window_box_height (w));
       if (it.current_y < it.last_visible_y)
 	move_it_past_eol (&it);
@@ -4923,7 +4923,7 @@
       /* Move backward half the height of the window.  Performance note:
 	 vmotion used here is about 10% faster, but would give wrong
 	 results for variable height lines.  */
-      init_iterator (&it, w, PT, PT_BYTE, NULL, DEFAULT_FACE_ID);
+      init_iterator (&it, w, PT, PT_BYTE, NULL, DEFAULT_FACE_ID, 0);
       it.current_y = it.last_visible_y;
       move_it_vertically_backward (&it, window_box_height (w) / 2);
 
@@ -4934,7 +4934,7 @@
 	 start of the line containing PT in this case.  */
       if (it.current_y <= 0)
 	{
-	  init_iterator (&it, w, PT, PT_BYTE, NULL, DEFAULT_FACE_ID);
+        init_iterator (&it, w, PT, PT_BYTE, NULL, DEFAULT_FACE_ID, 0);
 	  move_it_vertically_backward (&it, 0);
 	  it.current_y = 0;
 	}
@@ -5015,7 +5015,7 @@
 	  || !SYMBOLP (KVAR (current_kboard, Vlast_command))
 	  || NILP (Fget (KVAR (current_kboard, Vlast_command), Qscroll_command)))
 	{
-	  start_display (&it, w, start);
+        start_display (&it, w, start, 0);
 	  move_it_to (&it, PT, -1, -1, -1, MOVE_TO_POS);
 	  window_scroll_pixel_based_preserve_y = it.current_y;
 	  window_scroll_pixel_based_preserve_x = it.current_x;
@@ -5027,7 +5027,7 @@
 
   /* Move iterator it from start the specified distance forward or
      backward.  The result is the new window start.  */
-  start_display (&it, w, start);
+  start_display (&it, w, start, 0);
   if (whole)
     {
       ptrdiff_t start_pos = IT_CHARPOS (it);
@@ -5252,7 +5252,7 @@
       else if (window_scroll_pixel_based_preserve_y >= 0)
 	{
 	  SET_TEXT_POS_FROM_MARKER (start, w->start);
-	  start_display (&it, w, start);
+	  start_display (&it, w, start, 0);
 	  /* It would be wrong to subtract CURRENT_HEADER_LINE_HEIGHT
 	     here because we called start_display again and did not
 	     alter it.current_y this time.  */
@@ -5310,6 +5310,20 @@
   bool adjust_old_pointm = !NILP (Fequal (Fwindow_point (window),
 					  Fwindow_old_point (window)));
 
+#if 0
+  Lisp_Object Fhighest_fontified = intern ("highest-fontified");
+  Lisp_Object Flowest_fontified = intern ("lowest-fontified");
+  Lisp_Object Fdump_fontification_diags = intern ("dump-fontification-diags");
+  Lisp_Object pre_point = Fpoint (), post_point, pre_low_f, post_low_f, pre_high_f, post_high_f;
+
+  Lisp_Object dfd_array [7];
+  struct gcpro gcpro1, gcpro2, gcpro3, gcpro4, gcpro5, gcpro6;
+  GCPRO6 (Fhighest_fontified, Fdump_fontification_diags,
+          pre_point, post_point, pre_high_f, post_high_f);
+  pre_low_f = Ffuncall (1, (Lisp_Object []){Flowest_fontified});
+  pre_high_f = Ffuncall (0, (Lisp_Object []){Fhighest_fontified});
+#endif
+
   /* If scrolling screen-fulls, compute the number of lines to
      scroll from the window's height.  */
   if (whole)
@@ -5355,7 +5369,12 @@
   if (lose)
     {
       if (noerror)
-	return;
+          {
+#if 0
+          UNGCPRO;
+#endif
+          return;
+          }
       else
 	xsignal0 (Qbeginning_of_buffer);
     }
@@ -5436,7 +5455,12 @@
   else
     {
       if (noerror)
-	return;
+          {
+#if 0
+          UNGCPRO;
+#endif
+          return;
+          }
       else
 	xsignal0 (Qend_of_buffer);
     }
@@ -5447,6 +5471,21 @@
 		  ? make_number (BUF_PT (XBUFFER (w->contents)))
 		  : Fmarker_position (w->pointm)),
 		 w->contents);
+
+#if 0
+  post_low_f = Ffuncall (1, (Lisp_Object []){Flowest_fontified});
+  post_high_f = Ffuncall (1, (Lisp_Object []){Fhighest_fontified});
+  post_point = Fpoint ();
+  dfd_array [0] = Fdump_fontification_diags;
+  dfd_array [1] = pre_low_f;
+  dfd_array [2] = pre_high_f;
+  dfd_array [3] = pre_point;
+  dfd_array [4] = post_low_f;
+  dfd_array [5] = post_high_f;
+  dfd_array [6] = post_point;
+  Ffuncall (7, dfd_array);
+  UNGCPRO;
+#endif
 }
 
 
@@ -5694,7 +5733,7 @@
   CLIP_TEXT_POS_FROM_MARKER (start, w->start);
 
   itdata = bidi_shelve_cache ();
-  start_display (&it, w, start);
+  start_display (&it, w, start, 0);
   move_it_vertically (&it, height);
   bottom_y = line_bottom_y (&it);
   bidi_unshelve_cache (itdata, 0);
@@ -5798,7 +5837,7 @@
 	  void *itdata = bidi_shelve_cache ();
 
 	  SET_TEXT_POS (pt, PT, PT_BYTE);
-	  start_display (&it, w, pt);
+	  start_display (&it, w, pt, 0);
 	  move_it_vertically_backward (&it, window_box_height (w) / 2);
 	  charpos = IT_CHARPOS (it);
 	  bytepos = IT_BYTEPOS (it);
@@ -5816,7 +5855,7 @@
 	  iarg = - max (-iarg, this_scroll_margin);
 
 	  SET_TEXT_POS (pt, PT, PT_BYTE);
-	  start_display (&it, w, pt);
+	  start_display (&it, w, pt, 0);
 
 	  /* Be sure we have the exact height of the full line containing PT.  */
 	  move_it_by_lines (&it, 0);
@@ -5855,7 +5894,7 @@
 	    }
 
 	  /* Now find the new top line (starting position) of the window.  */
-	  start_display (&it, w, pt);
+	  start_display (&it, w, pt, 0);
 	  it.current_y = 0;
 	  move_it_vertically_backward (&it, h);
 

=== modified file 'src/xdisp.c'
--- src/xdisp.c	2014-10-24 09:58:43 +0000
+++ src/xdisp.c	2014-10-26 12:08:41 +0000
@@ -1289,7 +1289,7 @@
       set_buffer_internal_1 (XBUFFER (w->contents));
     }
   SET_TEXT_POS (pt, PT, PT_BYTE);
-  start_display (&it, w, pt);
+  start_display (&it, w, pt, 0);
   it.vpos = it.current_y = 0;
   last_height = 0;
   result = make_number (line_bottom_y (&it));
@@ -1434,7 +1434,7 @@
       = display_mode_line (w, HEADER_LINE_FACE_ID,
 			   BVAR (current_buffer, header_line_format));
 
-  start_display (&it, w, top);
+  start_display (&it, w, top, 1);
   move_it_to (&it, charpos, -1, it.last_visible_y - 1, -1,
 	      (charpos >= 0 ? MOVE_TO_POS : 0) | MOVE_TO_Y);
 
@@ -1508,7 +1508,7 @@
 		     the previous buffer position is also displayed
 		     from a display vector, we need to consume all of
 		     the glyphs from that display vector.  */
-		  start_display (&it2, w, top);
+		  start_display (&it2, w, top, 1);
 		  move_it_to (&it2, charpos - 1, -1, -1, -1, MOVE_TO_POS);
 		  /* If we didn't get to CHARPOS - 1, there's some
 		     replacing display property at that position, and
@@ -1590,7 +1590,7 @@
 		  end = XFASTINT (endpos);
 		  /* Move to the last buffer position before the
 		     display property.  */
-		  start_display (&it3, w, top);
+		  start_display (&it3, w, top, 1);
 		  if (start > CHARPOS (top))
 		    move_it_to (&it3, start - 1, -1, -1, -1, MOVE_TO_POS);
 		  /* Move forward one more line if the position before
@@ -1625,7 +1625,7 @@
 			 display string could be _after_ the display
 			 property in the logical order.  Use the
 			 smallest vertical position of these two.  */
-		      start_display (&it3, w, top);
+                    start_display (&it3, w, top, 1);
 		      move_it_to (&it3, end + 1, -1, -1, -1, MOVE_TO_POS);
 		      if (it3.current_y < top_y)
 			top_y = it3.current_y;
@@ -1633,7 +1633,7 @@
 		  /* Move from the top of the window to the beginning
 		     of the display line where the display string
 		     begins.  */
-		  start_display (&it3, w, top);
+		  start_display (&it3, w, top, 1);
 		  move_it_to (&it3, -1, 0, top_y, -1, MOVE_TO_X | MOVE_TO_Y);
 		  /* If it3_moved stays zero after the 'while' loop
 		     below, that means we already were at a newline
@@ -2795,12 +2795,18 @@
    If ROW is null and BASE_FACE_ID is equal to MODE_LINE_FACE_ID,
    MODE_LINE_INACTIVE_FACE_ID, or HEADER_LINE_FACE_ID, the iterator
    will be initialized to use the corresponding mode line glyph row of
-   the desired matrix of W.  */
+   the desired matrix of W.
+
+   NOT_FOR_DISPLAY can be set to true if it is known the iterator will
+   not be used for display.  This optimization prevents this iterator
+   from performing fontification when the user option
+   USE-DEFAULT-FACE-FOR-FAST-SCROLLING is enabled. */
 
 void
 init_iterator (struct it *it, struct window *w,
 	       ptrdiff_t charpos, ptrdiff_t bytepos,
-	       struct glyph_row *row, enum face_id base_face_id)
+	       struct glyph_row *row, enum face_id base_face_id,
+               bool not_for_display)
 {
   enum face_id remapped_base_face_id = base_face_id;
 
@@ -2840,6 +2846,7 @@
   it->current.overlay_string_index = -1;
   it->current.dpvec_index = -1;
   it->base_face_id = remapped_base_face_id;
+  it->not_for_display = not_for_display;
   it->string = Qnil;
   IT_STRING_CHARPOS (*it) = IT_STRING_BYTEPOS (*it) = -1;
   it->paragraph_embedding = L2R;
@@ -3097,13 +3104,15 @@
 /* Initialize IT for the display of window W with window start POS.  */
 
 void
-start_display (struct it *it, struct window *w, struct text_pos pos)
+start_display (struct it *it, struct window *w, struct text_pos pos,
+               bool not_for_display)
 {
   struct glyph_row *row;
   int first_vpos = WINDOW_WANTS_HEADER_LINE_P (w) ? 1 : 0;
 
   row = w->desired_matrix->rows + first_vpos;
-  init_iterator (it, w, CHARPOS (pos), BYTEPOS (pos), row, DEFAULT_FACE_ID);
+  init_iterator (it, w, CHARPOS (pos), BYTEPOS (pos), row, DEFAULT_FACE_ID,
+                 not_for_display);
   it->first_vpos = first_vpos;
 
   /* Don't reseat to previous visible line start if current start
@@ -3243,7 +3252,7 @@
      newline before the row start, such a POS will not be in a string,
      but the call to init_iterator below will move us to the
      after-string.  */
-  init_iterator (it, w, charpos, bytepos, NULL, DEFAULT_FACE_ID);
+  init_iterator (it, w, charpos, bytepos, NULL, DEFAULT_FACE_ID, 0);
 
   /* This only scans the current chunk -- it should scan all chunks.
      However, OVERLAY_STRING_CHUNK_SIZE has been increased from 3 in 21.1
@@ -3867,7 +3876,10 @@
 	  prop = Fget_char_property (pos, Qfontified, Qnil),
 	  /* Ignore the special cased nil value always present at EOB since
 	     no amount of fontifying will be able to change it.  */
-	  NILP (prop) && IT_CHARPOS (*it) < Z))
+	  NILP (prop) && IT_CHARPOS (*it) < Z)
+      && (NILP (Vuse_default_face_for_fast_scrolling)
+          || !it->not_for_display
+          /* || NILP (Finput_pending_p (Qnil)) */))
     {
       ptrdiff_t count = SPECPDL_INDEX ();
       Lisp_Object val;
@@ -9866,7 +9878,7 @@
 
   itdata = bidi_shelve_cache ();
   SET_TEXT_POS (startp, start, CHAR_TO_BYTE (start));
-  start_display (&it, w, startp);
+  start_display (&it, w, startp, 1);
 
   if (NILP (x_limit))
     x = move_it_to (&it, end, -1, max_y, -1, MOVE_TO_POS | MOVE_TO_Y);
@@ -10915,7 +10927,7 @@
 	  set_buffer_internal (XBUFFER (w->contents));
 	}
 
-      init_iterator (&it, w, BEGV, BEGV_BYTE, NULL, DEFAULT_FACE_ID);
+      init_iterator (&it, w, BEGV, BEGV_BYTE, NULL, DEFAULT_FACE_ID, 0);
 
       /* Compute the max. number of lines specified by the user.  */
       if (FLOATP (Vmax_mini_window_height))
@@ -10946,7 +10958,7 @@
       if (height > max_height)
 	{
 	  height = (max_height / unit) * unit;
-	  init_iterator (&it, w, ZV, ZV_BYTE, NULL, DEFAULT_FACE_ID);
+	  init_iterator (&it, w, ZV, ZV_BYTE, NULL, DEFAULT_FACE_ID, 0);
 	  move_it_vertically_backward (&it, height - unit);
 	  start = it.current.pos;
 	}
@@ -11626,7 +11638,7 @@
       mode_line_target = MODE_LINE_TITLE;
       title_start = MODE_LINE_NOPROP_LEN (0);
       init_iterator (&it, XWINDOW (f->selected_window), -1, -1,
-		     NULL, DEFAULT_FACE_ID);
+		     NULL, DEFAULT_FACE_ID, 0);
       display_mode_element (&it, 0, -1, -1, fmt, Qnil, 0);
       len = MODE_LINE_NOPROP_LEN (title_start);
       title = mode_line_noprop_buf + title_start;
@@ -12355,7 +12367,7 @@
 
   /* Initialize an iterator for iteration over
      F->desired_tool_bar_string in the tool-bar window of frame F.  */
-  init_iterator (&it, w, -1, -1, temp_row, TOOL_BAR_FACE_ID);
+  init_iterator (&it, w, -1, -1, temp_row, TOOL_BAR_FACE_ID, 0);
   temp_row->reversed_p = false;
   it.first_visible_x = 0;
   it.last_visible_x = WINDOW_PIXEL_WIDTH (w);
@@ -12437,7 +12449,7 @@
     return 0;
 
   /* Set up an iterator for the tool-bar window.  */
-  init_iterator (&it, w, -1, -1, w->desired_matrix->rows, TOOL_BAR_FACE_ID);
+  init_iterator (&it, w, -1, -1, w->desired_matrix->rows, TOOL_BAR_FACE_ID, 0);
   it.first_visible_x = 0;
   it.last_visible_x = WINDOW_PIXEL_WIDTH (w);
   row = it.glyph_row;
@@ -13709,7 +13721,7 @@
 
 	  /* Note that start_display will handle the case that the
 	     line starting at tlbufpos is a continuation line.  */
-	  start_display (&it, w, tlbufpos);
+	  start_display (&it, w, tlbufpos, 0);
 
 	  /* Implementation note: It this still necessary?  */
 	  if (it.current_x != this_line_start_x)
@@ -13825,7 +13837,7 @@
 	     PT may be in invisible text.  If so, we will end at the
 	     next visible position.  */
 	  init_iterator (&it, w, CHARPOS (tlbufpos), BYTEPOS (tlbufpos),
-			 NULL, DEFAULT_FACE_ID);
+			 NULL, DEFAULT_FACE_ID, 0);
 	  it.current_x = this_line_start_x;
 	  it.current_y = this_line_y;
 	  it.vpos = this_line_vpos;
@@ -15138,7 +15150,7 @@
 
       /* Compute the pixel ypos of the scroll margin, then move IT to
 	 either that ypos or PT, whichever comes first.  */
-      start_display (&it, w, startp);
+      start_display (&it, w, startp, 1);
       scroll_margin_y = it.last_visible_y - this_scroll_margin
 	- frame_line_height * extra_scroll_margin_lines;
       move_it_to (&it, PT, -1, scroll_margin_y - 1, -1,
@@ -15208,7 +15220,7 @@
       if (amount_to_scroll <= 0)
 	return SCROLLING_FAILED;
 
-      start_display (&it, w, startp);
+      start_display (&it, w, startp, 1);
       if (arg_scroll_conservatively <= scroll_limit)
 	move_it_vertically (&it, amount_to_scroll);
       else
@@ -15250,7 +15262,7 @@
 	{
 	  int y_start;
 
-	  start_display (&it, w, startp);
+	  start_display (&it, w, startp, 1);
 	  y_start = it.current_y;
 	  move_it_vertically (&it, this_scroll_margin);
 	  scroll_margin_pos = it.current.pos;
@@ -15274,7 +15286,7 @@
 	     Give up if distance is greater than scroll_max or if we
 	     didn't reach the scroll margin position.  */
 	  SET_TEXT_POS (pos, PT, PT_BYTE);
-	  start_display (&it, w, pos);
+	  start_display (&it, w, pos, 1);
 	  y0 = it.current_y;
 	  y_to_move = max (it.last_visible_y,
 			   max (scroll_max, 10 * frame_line_height));
@@ -15290,7 +15302,7 @@
 	  dy += y_offset;
 
 	  /* Compute new window start.  */
-	  start_display (&it, w, startp);
+	  start_display (&it, w, startp, 1);
 
 	  if (arg_scroll_conservatively)
 	    amount_to_scroll = max (dy, frame_line_height *
@@ -15399,7 +15411,7 @@
 	 because find_newline is fast (newline cache).  */
       row = w->desired_matrix->rows + (WINDOW_WANTS_HEADER_LINE_P (w) ? 1 : 0);
       init_iterator (&it, w, CHARPOS (start_pos), BYTEPOS (start_pos),
-		     row, DEFAULT_FACE_ID);
+		     row, DEFAULT_FACE_ID, 0);
       reseat_at_previous_visible_line_start (&it);
 
       /* If the line start is "too far" away from the window start,
@@ -15871,7 +15883,7 @@
 	}
 
       SET_TEXT_POS_FROM_MARKER (startp, w->start);
-      start_display (&it, w, startp);
+      start_display (&it, w, startp, 0);
       it.last_visible_x = INT_MAX;
       whole = move_it_to (&it, -1, INT_MAX, window_box_height (w), -1,
 			  MOVE_TO_X | MOVE_TO_Y);
@@ -16184,7 +16196,7 @@
       ptrdiff_t it_charpos;
 
       w->optional_new_start = 0;
-      start_display (&it, w, startp);
+      start_display (&it, w, startp, 0);
       move_it_to (&it, PT, 0, it.last_visible_y, -1,
 		  MOVE_TO_POS | MOVE_TO_X | MOVE_TO_Y);
       /* Record IT's position now, since line_bottom_y might change
@@ -16577,7 +16589,7 @@
     w->base_line_number = 0;
 
   /* Determine the window start relative to point.  */
-  init_iterator (&it, w, PT, PT_BYTE, NULL, DEFAULT_FACE_ID);
+  init_iterator (&it, w, PT, PT_BYTE, NULL, DEFAULT_FACE_ID, 0);
   it.current_y = it.last_visible_y;
   if (centering_position < 0)
     {
@@ -16606,7 +16618,7 @@
 	  void *it1data = NULL;
 
 	  SAVE_IT (it1, it, it1data);
-	  start_display (&it1, w, startp);
+	  start_display (&it1, w, startp, 0);
 	  move_it_vertically (&it1, margin * frame_line_height);
 	  margin_pos = IT_CHARPOS (it1);
 	  RESTORE_IT (&it, &it, it1data);
@@ -16669,7 +16681,7 @@
      containing PT in this case.  */
   if (it.current_y <= 0)
     {
-      init_iterator (&it, w, PT, PT_BYTE, NULL, DEFAULT_FACE_ID);
+      init_iterator (&it, w, PT, PT_BYTE, NULL, DEFAULT_FACE_ID, 0);
       move_it_vertically_backward (&it, 0);
       it.current_y = 0;
     }
@@ -17013,7 +17025,7 @@
   overlay_arrow_seen = 0;
 
   /* Initialize iterator and info to start at POS.  */
-  start_display (&it, w, pos);
+  start_display (&it, w, pos, 0);
   it.glyph_row->reversed_p = false;
 
   /* Display all lines of W.  */
@@ -17161,7 +17173,7 @@
 	 last_text_row is set to the last row displayed that displays
 	 text.  Note that it.vpos == 0 if or if not there is a
          header-line; it's not the same as the MATRIX_ROW_VPOS!  */
-      start_display (&it, w, new_start);
+      start_display (&it, w, new_start, 0);
       w->cursor.vpos = -1;
       last_text_row = last_reused_text_row = NULL;
 
@@ -18142,7 +18154,7 @@
     {
       /* There are no reusable lines at the start of the window.
 	 Start displaying in the first text line.  */
-      start_display (&it, w, start);
+      start_display (&it, w, start, 0);
       it.vpos = it.first_vpos;
       start_pos = it.current.pos;
     }
@@ -18932,7 +18944,7 @@
   int n_glyphs_before;
 
   set_buffer_temp (buffer);
-  init_iterator (&it, w, -1, -1, &scratch_glyph_row, DEFAULT_FACE_ID);
+  init_iterator (&it, w, -1, -1, &scratch_glyph_row, DEFAULT_FACE_ID, 0);
   scratch_glyph_row.reversed_p = false;
   it.glyph_row->used[TEXT_AREA] = 0;
   SET_TEXT_POS (it.position, 0, 0);
@@ -21228,7 +21240,7 @@
 
       /* Setup the arena.  */
       SET_TEXT_POS (pt, PT, PT_BYTE);
-      start_display (&it, w, pt);
+      start_display (&it, w, pt, 1);
 
       if (it.cmp_it.id < 0
 	  && it.method == GET_FROM_STRING
@@ -21319,7 +21331,7 @@
 	{
 	  if (pt_x > 0)
 	    {
-	      start_display (&it, w, pt);
+              start_display (&it, w, pt, 0);
 	      reseat_at_previous_visible_line_start (&it);
 	      it.current_x = it.current_y = it.hpos = 0;
 	      if (pt_vpos != 0)
@@ -21622,7 +21634,7 @@
 
 #if defined (USE_X_TOOLKIT) || defined (USE_GTK)
   eassert (!FRAME_WINDOW_P (f));
-  init_iterator (&it, w, -1, -1, f->desired_matrix->rows, MENU_FACE_ID);
+  init_iterator (&it, w, -1, -1, f->desired_matrix->rows, MENU_FACE_ID, 0);
   it.first_visible_x = 0;
   it.last_visible_x = FRAME_PIXEL_WIDTH (f);
 #elif defined (HAVE_X_WINDOWS) /* X without toolkit.  */
@@ -21633,7 +21645,7 @@
       struct window *menu_w;
       menu_w = XWINDOW (f->menu_bar_window);
       init_iterator (&it, menu_w, -1, -1, menu_w->desired_matrix->rows,
-		     MENU_FACE_ID);
+		     MENU_FACE_ID, 0);
       it.first_visible_x = 0;
       it.last_visible_x = FRAME_PIXEL_WIDTH (f);
     }
@@ -21643,7 +21655,7 @@
       /* This is a TTY frame, i.e. character hpos/vpos are used as
 	 pixel x/y.  */
       init_iterator (&it, w, -1, -1, f->desired_matrix->rows,
-		     MENU_FACE_ID);
+		     MENU_FACE_ID, 0);
       it.first_visible_x = 0;
       it.last_visible_x = FRAME_COLS (f);
     }
@@ -21759,7 +21771,7 @@
   if (y >= f->desired_matrix->nrows)
     return;
 
-  init_iterator (&it, w, -1, -1, f->desired_matrix->rows + y, MENU_FACE_ID);
+  init_iterator (&it, w, -1, -1, f->desired_matrix->rows + y, MENU_FACE_ID, 0);
   it.first_visible_x = 0;
   it.last_visible_x = FRAME_COLS (f) - 1;
   row = it.glyph_row;
@@ -21932,7 +21944,7 @@
   struct face *face;
   ptrdiff_t count = SPECPDL_INDEX ();
 
-  init_iterator (&it, w, -1, -1, NULL, face_id);
+  init_iterator (&it, w, -1, -1, NULL, face_id, 0);
   /* Don't extend on a previously drawn mode-line.
      This may happen if called from pos_visible_p.  */
   it.glyph_row->enabled_p = false;
@@ -22651,7 +22663,7 @@
   Fselect_window (window, Qt);
   set_buffer_internal_1 (XBUFFER (buffer));
 
-  init_iterator (&it, w, -1, -1, NULL, face_id);
+  init_iterator (&it, w, -1, -1, NULL, face_id, 0);
 
   if (no_props)
     {
@@ -31056,6 +31068,18 @@
 	       doc: /*  */);
   Vredisplay__mode_lines_cause
     = Fmake_vector (make_number (100), make_number (0));
+
+  DEFVAR_LISP ("use-default-face-for-fast-scrolling",
+               Vuse_default_face_for_fast_scrolling,
+               doc: /* When non-nil, accelerate repeated scrolling operations.
+This comes into play when scrolling rapidly over previously
+unfontified buffer regions.  Only those portions of the buffer which
+are actually going to be displayed get fontified.
+
+Note that this optimization can cuase the portion of the buffer displayed to
+be slightly different. */);
+
+  Vuse_default_face_for_fast_scrolling = Qnil;
 }
 
 


-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Unfreezing the display during auto-repeated scrolling. [ Was: Aborting display. Is this possible? ]

[Eli Zaretskii]

> Date: Sun, 26 Oct 2014 12:43:33 +0000
> From: Alan Mackenzie <acm@muc.de>
> Cc: monnier@IRO.UMontreal.CA, emacs-devel@gnu.org
> 
> Here's what I've done:  Conceptually, ITs (i.e. "struct it"s, the
> iterators in xdisp.c) are used for two different purposes:
> (i) display;
> (ii) calculation of displayed character sizes, positions etc.
> I've distinguished these two cases by adding an extra boolean flag to IT,
> called `not_for_display'.  The functions `init_iterator' and
> `start_display' have got this as an extra boolean parameter.
> 
> Some of the ITs in window.c and xdisp.c are now designated
> `not_for_display'.
> 
> `handle_fontified_prop' now checks this flag in IT.  If both the flag and
> `use-default-face-for-fast-scrolling' are set, then fontification is not
> done.  The existing face (usually default) is simply left there.  The
> code doesn't also check for the event queue being non-empty - in
> practice, this didn't work very well.
> 
> Here's the code.  Comments and criticism would be welcome.

I'm afraid this is a non-starter.  Even if this feature is accepted
(and I don't like it), it cannot be controlled on such a low level.
You cannot possibly know, on this low level, which code will use the
"not for display" functions, even if today they are only used in a
context where ignoring font-lock faces might be OK.  You already
marked as "not for display" line-pixel-height, which must be accurate
in the results it returns, since otherwise it will violate its
contract.  Likewise in resize_mini_window (which is definitely "for
display"), redisplay_internal (likewise), etc.  OTOH, displaying
strings (e.g., as part of mode-line display) will never bump into
font-lock faces, so setting this flag there is not useful.

So if we go this way, we are digging a hole for ourselves -- it is an
impossible maintenance burden to have functions that don't exactly do
what they say they do under some circumstances.

You should only make changes in window_scroll_* functions, and do that
only under control of the use-default-face-for-fast-scrolling
variable, so that the Lisp application that binds this variable to a
non-nil value is the sole source of this behavior, and holds all the
responsibility for binding it where it wants.  No code should
disregard font-lock faces, except when this variable is non-nil.

And I still don't understand why writing a simple "fast-scroll"
function in Lisp, which just moves point N lines forward or backward,
is not a better solution.  It's certainly easier and has exactly zero
impact on our infrastructure.

Thanks.

Re: Unfreezing the display during auto-repeated scrolling. [ Was: Aborting display. Is this possible? ]

[Alan Mackenzie]

Hello, Eli.

Thanks for such a quick reply.

On Sun, Oct 26, 2014 at 06:45:47PM +0200, Eli Zaretskii wrote:
> > Date: Sun, 26 Oct 2014 12:43:33 +0000
> > From: Alan Mackenzie <acm@muc.de>
> > Cc: monnier@IRO.UMontreal.CA, emacs-devel@gnu.org

> > Here's what I've done:  Conceptually, ITs (i.e. "struct it"s, the
> > iterators in xdisp.c) are used for two different purposes:
> > (i) display;
> > (ii) calculation of displayed character sizes, positions etc.
> > I've distinguished these two cases by adding an extra boolean flag to IT,
> > called `not_for_display'.  The functions `init_iterator' and
> > `start_display' have got this as an extra boolean parameter.

> > Some of the ITs in window.c and xdisp.c are now designated
> > `not_for_display'.

> > `handle_fontified_prop' now checks this flag in IT.  If both the flag and
> > `use-default-face-for-fast-scrolling' are set, then fontification is not
> > done.  The existing face (usually default) is simply left there.  The
> > code doesn't also check for the event queue being non-empty - in
> > practice, this didn't work very well.

> > Here's the code.  Comments and criticism would be welcome.

> I'm afraid this is a non-starter.  Even if this feature is accepted
> (and I don't like it), it cannot be controlled on such a low level.
> You cannot possibly know, on this low level, which code will use the
> "not for display" functions, even if today they are only used in a
> context where ignoring font-lock faces might be OK.

The contract of xdisp.c says explicitly that the only visible display
function is redisplay, and this has no IT parameters.  So any IT with
not_for_display set should be "safe" if it is defined outside of xdisp.c

> You already marked as "not for display" line-pixel-height, which must
> be accurate in the results it returns, since otherwise it will violate
> its contract.  Likewise in resize_mini_window (which is definitely "for
> display"), redisplay_internal (likewise), etc.  OTOH, displaying
> strings (e.g., as part of mode-line display) will never bump into
> font-lock faces, so setting this flag there is not useful.

Sorry, I'd left unnecessary not_for_display's set.  I've cleared out
xdisp.c, such that the only function which sets not_for_display is
pos_visible_p.  The rest are not needed.

The other functions which set not_for_display are
(i) window_scroll_pixel_based (window.c);
(ii) Fvertical_motion (indent.c).

> So if we go this way, we are digging a hole for ourselves -- it is an
> impossible maintenance burden to have functions that don't exactly do
> what they say they do under some circumstances.

Yes.  But the current implementation doesn't work very well.  I'm trying
to fix it, because it's irritating.  I think I've made good progress.  I
can't see any other approach to fixing it at the moment.

> You should only make changes in window_scroll_* functions, ...

I honestly can't see how I could achieve what's wanted if I restrict
myself to window.c.

> ... and do that only under control of the
> use-default-face-for-fast-scrolling variable, so that the Lisp
> application that binds this variable to a non-nil value is the sole
> source of this behavior, and holds all the responsibility for binding
> it where it wants.  No code should disregard font-lock faces, except
> when this variable is non-nil.

The new feature I've added remains inactive when
use-default-face-for-fast-scrolling is nil.  The single effect when it is
set is to inhibit fontification when an IT.not_for_display flag is also
set.

I had in mind that use-default-face-for-fast-scrolling should be a user
option rather than something to be set by a major mode.  Something that
could upset exact scrolling should be turned on by a user, rather thaan
by a major mode.

> And I still don't understand why writing a simple "fast-scroll"
> function in Lisp, which just moves point N lines forward or backward,
> is not a better solution.  It's certainly easier and has exactly zero
> impact on our infrastructure.

That's precisely how Fscroll_up and Fscroll_down work.  I'd merely be
reinventing the wheel, and I'd face all the difficulties and special
cases which other hackers have long ago overcome.

To reiterate what I want: it is for an auto-repeated PageUp or PageDown
not to freeze the screen, either whilst depressed or after being
released.  I don't see how writing a new scroll command for these keys
would help.

Anyhow, I've now got the feature working for GUI windows.  It wasn't
difficult.

Here's the updated patch:



=== modified file 'src/dispextern.h'
--- src/dispextern.h	2014-10-15 10:22:15 +0000
+++ src/dispextern.h	2014-10-26 10:32:37 +0000
@@ -2275,6 +2275,12 @@
   /* True means cursor shouldn't be displayed here.  */
   bool_bf avoid_cursor_p : 1;
 
+  /* True means it is known that IT is only for calculating screen
+     occupancy, not for actually drawing elements on a screen.  This
+     suppresses fontification when user option
+     `use-default-face-for-fast-scrolling' is set. */
+  bool_bf not_for_display : 1;
+
   /* Display table in effect or null for none.  */
   struct Lisp_Char_Table *dp;
 
@@ -3204,10 +3210,10 @@
 void mark_window_display_accurate (Lisp_Object, int);
 void redisplay_preserve_echo_area (int);
 void init_iterator (struct it *, struct window *, ptrdiff_t,
-                    ptrdiff_t, struct glyph_row *, enum face_id);
+                    ptrdiff_t, struct glyph_row *, enum face_id, bool);
 void init_iterator_to_row_start (struct it *, struct window *,
                                  struct glyph_row *);
-void start_display (struct it *, struct window *, struct text_pos);
+void start_display (struct it *, struct window *, struct text_pos, bool);
 void move_it_vertically (struct it *, int);
 void move_it_vertically_backward (struct it *, int);
 void move_it_by_lines (struct it *, ptrdiff_t);

=== modified file 'src/dispnew.c'
--- src/dispnew.c	2014-10-14 12:45:41 +0000
+++ src/dispnew.c	2014-10-22 11:15:50 +0000
@@ -5131,7 +5131,7 @@
   Fset_buffer (w->contents);
   itdata = bidi_shelve_cache ();
   CLIP_TEXT_POS_FROM_MARKER (startp, w->start);
-  start_display (&it, w, startp);
+  start_display (&it, w, startp, 0);
   x0 = *x;
 
   /* First, move to the beginning of the row corresponding to *Y.  We

=== modified file 'src/indent.c'
--- src/indent.c	2014-08-28 01:59:29 +0000
+++ src/indent.c	2014-10-22 11:17:10 +0000
@@ -2009,7 +2009,7 @@
 
       itdata = bidi_shelve_cache ();
       SET_TEXT_POS (pt, PT, PT_BYTE);
-      start_display (&it, w, pt);
+      start_display (&it, w, pt, 1);
       first_x = it.first_visible_x;
       it_start = IT_CHARPOS (it);
 

=== modified file 'src/window.c'
--- src/window.c	2014-10-04 08:20:24 +0000
+++ src/window.c	2014-10-26 15:15:07 +0000
@@ -1799,7 +1799,7 @@
       CLIP_TEXT_POS_FROM_MARKER (startp, w->start);
 
       itdata = bidi_shelve_cache ();
-      start_display (&it, w, startp);
+      start_display (&it, w, startp, 0);
       move_it_vertically (&it, window_box_height (w));
       if (it.current_y < it.last_visible_y)
 	move_it_past_eol (&it);
@@ -4923,7 +4923,7 @@
       /* Move backward half the height of the window.  Performance note:
 	 vmotion used here is about 10% faster, but would give wrong
 	 results for variable height lines.  */
-      init_iterator (&it, w, PT, PT_BYTE, NULL, DEFAULT_FACE_ID);
+      init_iterator (&it, w, PT, PT_BYTE, NULL, DEFAULT_FACE_ID, 1);
       it.current_y = it.last_visible_y;
       move_it_vertically_backward (&it, window_box_height (w) / 2);
 
@@ -4934,7 +4934,7 @@
 	 start of the line containing PT in this case.  */
       if (it.current_y <= 0)
 	{
-	  init_iterator (&it, w, PT, PT_BYTE, NULL, DEFAULT_FACE_ID);
+        init_iterator (&it, w, PT, PT_BYTE, NULL, DEFAULT_FACE_ID, 1);
 	  move_it_vertically_backward (&it, 0);
 	  it.current_y = 0;
 	}
@@ -5015,7 +5015,7 @@
 	  || !SYMBOLP (KVAR (current_kboard, Vlast_command))
 	  || NILP (Fget (KVAR (current_kboard, Vlast_command), Qscroll_command)))
 	{
-	  start_display (&it, w, start);
+          start_display (&it, w, start, 1);
 	  move_it_to (&it, PT, -1, -1, -1, MOVE_TO_POS);
 	  window_scroll_pixel_based_preserve_y = it.current_y;
 	  window_scroll_pixel_based_preserve_x = it.current_x;
@@ -5027,7 +5027,7 @@
 
   /* Move iterator it from start the specified distance forward or
      backward.  The result is the new window start.  */
-  start_display (&it, w, start);
+  start_display (&it, w, start, 1);
   if (whole)
     {
       ptrdiff_t start_pos = IT_CHARPOS (it);
@@ -5252,7 +5252,7 @@
       else if (window_scroll_pixel_based_preserve_y >= 0)
 	{
 	  SET_TEXT_POS_FROM_MARKER (start, w->start);
-	  start_display (&it, w, start);
+	  start_display (&it, w, start, 1);
 	  /* It would be wrong to subtract CURRENT_HEADER_LINE_HEIGHT
 	     here because we called start_display again and did not
 	     alter it.current_y this time.  */
@@ -5310,6 +5310,20 @@
   bool adjust_old_pointm = !NILP (Fequal (Fwindow_point (window),
 					  Fwindow_old_point (window)));
 
+#if 0
+  Lisp_Object Fhighest_fontified = intern ("highest-fontified");
+  Lisp_Object Flowest_fontified = intern ("lowest-fontified");
+  Lisp_Object Fdump_fontification_diags = intern ("dump-fontification-diags");
+  Lisp_Object pre_point = Fpoint (), post_point, pre_low_f, post_low_f, pre_high_f, post_high_f;
+
+  Lisp_Object dfd_array [7];
+  struct gcpro gcpro1, gcpro2, gcpro3, gcpro4, gcpro5, gcpro6;
+  GCPRO6 (Fhighest_fontified, Fdump_fontification_diags,
+          pre_point, post_point, pre_high_f, post_high_f);
+  pre_low_f = Ffuncall (1, (Lisp_Object []){Flowest_fontified});
+  pre_high_f = Ffuncall (0, (Lisp_Object []){Fhighest_fontified});
+#endif
+
   /* If scrolling screen-fulls, compute the number of lines to
      scroll from the window's height.  */
   if (whole)
@@ -5355,7 +5369,12 @@
   if (lose)
     {
       if (noerror)
-	return;
+          {
+#if 0
+          UNGCPRO;
+#endif
+          return;
+          }
       else
 	xsignal0 (Qbeginning_of_buffer);
     }
@@ -5436,7 +5455,12 @@
   else
     {
       if (noerror)
-	return;
+          {
+#if 0
+          UNGCPRO;
+#endif
+          return;
+          }
       else
 	xsignal0 (Qend_of_buffer);
     }
@@ -5447,6 +5471,21 @@
 		  ? make_number (BUF_PT (XBUFFER (w->contents)))
 		  : Fmarker_position (w->pointm)),
 		 w->contents);
+
+#if 0
+  post_low_f = Ffuncall (1, (Lisp_Object []){Flowest_fontified});
+  post_high_f = Ffuncall (1, (Lisp_Object []){Fhighest_fontified});
+  post_point = Fpoint ();
+  dfd_array [0] = Fdump_fontification_diags;
+  dfd_array [1] = pre_low_f;
+  dfd_array [2] = pre_high_f;
+  dfd_array [3] = pre_point;
+  dfd_array [4] = post_low_f;
+  dfd_array [5] = post_high_f;
+  dfd_array [6] = post_point;
+  Ffuncall (7, dfd_array);
+  UNGCPRO;
+#endif
 }
 
 
@@ -5694,7 +5733,7 @@
   CLIP_TEXT_POS_FROM_MARKER (start, w->start);
 
   itdata = bidi_shelve_cache ();
-  start_display (&it, w, start);
+  start_display (&it, w, start, 0);
   move_it_vertically (&it, height);
   bottom_y = line_bottom_y (&it);
   bidi_unshelve_cache (itdata, 0);
@@ -5798,7 +5837,7 @@
 	  void *itdata = bidi_shelve_cache ();
 
 	  SET_TEXT_POS (pt, PT, PT_BYTE);
-	  start_display (&it, w, pt);
+	  start_display (&it, w, pt, 0);
 	  move_it_vertically_backward (&it, window_box_height (w) / 2);
 	  charpos = IT_CHARPOS (it);
 	  bytepos = IT_BYTEPOS (it);
@@ -5816,7 +5855,7 @@
 	  iarg = - max (-iarg, this_scroll_margin);
 
 	  SET_TEXT_POS (pt, PT, PT_BYTE);
-	  start_display (&it, w, pt);
+	  start_display (&it, w, pt, 0);
 
 	  /* Be sure we have the exact height of the full line containing PT.  */
 	  move_it_by_lines (&it, 0);
@@ -5855,7 +5894,7 @@
 	    }
 
 	  /* Now find the new top line (starting position) of the window.  */
-	  start_display (&it, w, pt);
+	  start_display (&it, w, pt, 0);
 	  it.current_y = 0;
 	  move_it_vertically_backward (&it, h);
 

=== modified file 'src/xdisp.c'
--- src/xdisp.c	2014-10-24 09:58:43 +0000
+++ src/xdisp.c	2014-10-26 18:48:23 +0000
@@ -1289,7 +1289,7 @@
       set_buffer_internal_1 (XBUFFER (w->contents));
     }
   SET_TEXT_POS (pt, PT, PT_BYTE);
-  start_display (&it, w, pt);
+  start_display (&it, w, pt, 0);
   it.vpos = it.current_y = 0;
   last_height = 0;
   result = make_number (line_bottom_y (&it));
@@ -1434,7 +1434,7 @@
       = display_mode_line (w, HEADER_LINE_FACE_ID,
 			   BVAR (current_buffer, header_line_format));
 
-  start_display (&it, w, top);
+  start_display (&it, w, top, 1);
   move_it_to (&it, charpos, -1, it.last_visible_y - 1, -1,
 	      (charpos >= 0 ? MOVE_TO_POS : 0) | MOVE_TO_Y);
 
@@ -1508,7 +1508,7 @@
 		     the previous buffer position is also displayed
 		     from a display vector, we need to consume all of
 		     the glyphs from that display vector.  */
-		  start_display (&it2, w, top);
+		  start_display (&it2, w, top, 1);
 		  move_it_to (&it2, charpos - 1, -1, -1, -1, MOVE_TO_POS);
 		  /* If we didn't get to CHARPOS - 1, there's some
 		     replacing display property at that position, and
@@ -1590,7 +1590,7 @@
 		  end = XFASTINT (endpos);
 		  /* Move to the last buffer position before the
 		     display property.  */
-		  start_display (&it3, w, top);
+		  start_display (&it3, w, top, 1);
 		  if (start > CHARPOS (top))
 		    move_it_to (&it3, start - 1, -1, -1, -1, MOVE_TO_POS);
 		  /* Move forward one more line if the position before
@@ -1625,7 +1625,7 @@
 			 display string could be _after_ the display
 			 property in the logical order.  Use the
 			 smallest vertical position of these two.  */
-		      start_display (&it3, w, top);
+                    start_display (&it3, w, top, 1);
 		      move_it_to (&it3, end + 1, -1, -1, -1, MOVE_TO_POS);
 		      if (it3.current_y < top_y)
 			top_y = it3.current_y;
@@ -1633,7 +1633,7 @@
 		  /* Move from the top of the window to the beginning
 		     of the display line where the display string
 		     begins.  */
-		  start_display (&it3, w, top);
+		  start_display (&it3, w, top, 1);
 		  move_it_to (&it3, -1, 0, top_y, -1, MOVE_TO_X | MOVE_TO_Y);
 		  /* If it3_moved stays zero after the 'while' loop
 		     below, that means we already were at a newline
@@ -2795,12 +2795,18 @@
    If ROW is null and BASE_FACE_ID is equal to MODE_LINE_FACE_ID,
    MODE_LINE_INACTIVE_FACE_ID, or HEADER_LINE_FACE_ID, the iterator
    will be initialized to use the corresponding mode line glyph row of
-   the desired matrix of W.  */
+   the desired matrix of W.
+
+   NOT_FOR_DISPLAY can be set to true if it is known the iterator will
+   not be used for display.  This optimization prevents this iterator
+   from performing fontification when the user option
+   USE-DEFAULT-FACE-FOR-FAST-SCROLLING is enabled. */
 
 void
 init_iterator (struct it *it, struct window *w,
 	       ptrdiff_t charpos, ptrdiff_t bytepos,
-	       struct glyph_row *row, enum face_id base_face_id)
+	       struct glyph_row *row, enum face_id base_face_id,
+               bool not_for_display)
 {
   enum face_id remapped_base_face_id = base_face_id;
 
@@ -2840,6 +2846,7 @@
   it->current.overlay_string_index = -1;
   it->current.dpvec_index = -1;
   it->base_face_id = remapped_base_face_id;
+  it->not_for_display = not_for_display;
   it->string = Qnil;
   IT_STRING_CHARPOS (*it) = IT_STRING_BYTEPOS (*it) = -1;
   it->paragraph_embedding = L2R;
@@ -3097,13 +3104,15 @@
 /* Initialize IT for the display of window W with window start POS.  */
 
 void
-start_display (struct it *it, struct window *w, struct text_pos pos)
+start_display (struct it *it, struct window *w, struct text_pos pos,
+               bool not_for_display)
 {
   struct glyph_row *row;
   int first_vpos = WINDOW_WANTS_HEADER_LINE_P (w) ? 1 : 0;
 
   row = w->desired_matrix->rows + first_vpos;
-  init_iterator (it, w, CHARPOS (pos), BYTEPOS (pos), row, DEFAULT_FACE_ID);
+  init_iterator (it, w, CHARPOS (pos), BYTEPOS (pos), row, DEFAULT_FACE_ID,
+                 not_for_display);
   it->first_vpos = first_vpos;
 
   /* Don't reseat to previous visible line start if current start
@@ -3243,7 +3252,7 @@
      newline before the row start, such a POS will not be in a string,
      but the call to init_iterator below will move us to the
      after-string.  */
-  init_iterator (it, w, charpos, bytepos, NULL, DEFAULT_FACE_ID);
+  init_iterator (it, w, charpos, bytepos, NULL, DEFAULT_FACE_ID, 0);
 
   /* This only scans the current chunk -- it should scan all chunks.
      However, OVERLAY_STRING_CHUNK_SIZE has been increased from 3 in 21.1
@@ -3867,7 +3876,10 @@
 	  prop = Fget_char_property (pos, Qfontified, Qnil),
 	  /* Ignore the special cased nil value always present at EOB since
 	     no amount of fontifying will be able to change it.  */
-	  NILP (prop) && IT_CHARPOS (*it) < Z))
+	  NILP (prop) && IT_CHARPOS (*it) < Z)
+      && (NILP (Vuse_default_face_for_fast_scrolling)
+          || !it->not_for_display
+          /* || NILP (Finput_pending_p (Qnil)) */))
     {
       ptrdiff_t count = SPECPDL_INDEX ();
       Lisp_Object val;
@@ -9866,7 +9878,7 @@
 
   itdata = bidi_shelve_cache ();
   SET_TEXT_POS (startp, start, CHAR_TO_BYTE (start));
-  start_display (&it, w, startp);
+  start_display (&it, w, startp, 0);
 
   if (NILP (x_limit))
     x = move_it_to (&it, end, -1, max_y, -1, MOVE_TO_POS | MOVE_TO_Y);
@@ -10915,7 +10927,7 @@
 	  set_buffer_internal (XBUFFER (w->contents));
 	}
 
-      init_iterator (&it, w, BEGV, BEGV_BYTE, NULL, DEFAULT_FACE_ID);
+      init_iterator (&it, w, BEGV, BEGV_BYTE, NULL, DEFAULT_FACE_ID, 0);
 
       /* Compute the max. number of lines specified by the user.  */
       if (FLOATP (Vmax_mini_window_height))
@@ -10946,7 +10958,7 @@
       if (height > max_height)
 	{
 	  height = (max_height / unit) * unit;
-	  init_iterator (&it, w, ZV, ZV_BYTE, NULL, DEFAULT_FACE_ID);
+	  init_iterator (&it, w, ZV, ZV_BYTE, NULL, DEFAULT_FACE_ID, 0);
 	  move_it_vertically_backward (&it, height - unit);
 	  start = it.current.pos;
 	}
@@ -11626,7 +11638,7 @@
       mode_line_target = MODE_LINE_TITLE;
       title_start = MODE_LINE_NOPROP_LEN (0);
       init_iterator (&it, XWINDOW (f->selected_window), -1, -1,
-		     NULL, DEFAULT_FACE_ID);
+		     NULL, DEFAULT_FACE_ID, 0);
       display_mode_element (&it, 0, -1, -1, fmt, Qnil, 0);
       len = MODE_LINE_NOPROP_LEN (title_start);
       title = mode_line_noprop_buf + title_start;
@@ -12355,7 +12367,7 @@
 
   /* Initialize an iterator for iteration over
      F->desired_tool_bar_string in the tool-bar window of frame F.  */
-  init_iterator (&it, w, -1, -1, temp_row, TOOL_BAR_FACE_ID);
+  init_iterator (&it, w, -1, -1, temp_row, TOOL_BAR_FACE_ID, 0);
   temp_row->reversed_p = false;
   it.first_visible_x = 0;
   it.last_visible_x = WINDOW_PIXEL_WIDTH (w);
@@ -12437,7 +12449,7 @@
     return 0;
 
   /* Set up an iterator for the tool-bar window.  */
-  init_iterator (&it, w, -1, -1, w->desired_matrix->rows, TOOL_BAR_FACE_ID);
+  init_iterator (&it, w, -1, -1, w->desired_matrix->rows, TOOL_BAR_FACE_ID, 0);
   it.first_visible_x = 0;
   it.last_visible_x = WINDOW_PIXEL_WIDTH (w);
   row = it.glyph_row;
@@ -13709,7 +13721,7 @@
 
 	  /* Note that start_display will handle the case that the
 	     line starting at tlbufpos is a continuation line.  */
-	  start_display (&it, w, tlbufpos);
+	  start_display (&it, w, tlbufpos, 0);
 
 	  /* Implementation note: It this still necessary?  */
 	  if (it.current_x != this_line_start_x)
@@ -13825,7 +13837,7 @@
 	     PT may be in invisible text.  If so, we will end at the
 	     next visible position.  */
 	  init_iterator (&it, w, CHARPOS (tlbufpos), BYTEPOS (tlbufpos),
-			 NULL, DEFAULT_FACE_ID);
+			 NULL, DEFAULT_FACE_ID, 0);
 	  it.current_x = this_line_start_x;
 	  it.current_y = this_line_y;
 	  it.vpos = this_line_vpos;
@@ -15138,7 +15150,7 @@
 
       /* Compute the pixel ypos of the scroll margin, then move IT to
 	 either that ypos or PT, whichever comes first.  */
-      start_display (&it, w, startp);
+      start_display (&it, w, startp, 0);
       scroll_margin_y = it.last_visible_y - this_scroll_margin
 	- frame_line_height * extra_scroll_margin_lines;
       move_it_to (&it, PT, -1, scroll_margin_y - 1, -1,
@@ -15208,7 +15220,7 @@
       if (amount_to_scroll <= 0)
 	return SCROLLING_FAILED;
 
-      start_display (&it, w, startp);
+      start_display (&it, w, startp, 0);
       if (arg_scroll_conservatively <= scroll_limit)
 	move_it_vertically (&it, amount_to_scroll);
       else
@@ -15250,7 +15262,7 @@
 	{
 	  int y_start;
 
-	  start_display (&it, w, startp);
+	  start_display (&it, w, startp, 0);
 	  y_start = it.current_y;
 	  move_it_vertically (&it, this_scroll_margin);
 	  scroll_margin_pos = it.current.pos;
@@ -15274,7 +15286,7 @@
 	     Give up if distance is greater than scroll_max or if we
 	     didn't reach the scroll margin position.  */
 	  SET_TEXT_POS (pos, PT, PT_BYTE);
-	  start_display (&it, w, pos);
+	  start_display (&it, w, pos, 0);
 	  y0 = it.current_y;
 	  y_to_move = max (it.last_visible_y,
 			   max (scroll_max, 10 * frame_line_height));
@@ -15290,7 +15302,7 @@
 	  dy += y_offset;
 
 	  /* Compute new window start.  */
-	  start_display (&it, w, startp);
+	  start_display (&it, w, startp, 0);
 
 	  if (arg_scroll_conservatively)
 	    amount_to_scroll = max (dy, frame_line_height *
@@ -15399,7 +15411,7 @@
 	 because find_newline is fast (newline cache).  */
       row = w->desired_matrix->rows + (WINDOW_WANTS_HEADER_LINE_P (w) ? 1 : 0);
       init_iterator (&it, w, CHARPOS (start_pos), BYTEPOS (start_pos),
-		     row, DEFAULT_FACE_ID);
+		     row, DEFAULT_FACE_ID, 0);
       reseat_at_previous_visible_line_start (&it);
 
       /* If the line start is "too far" away from the window start,
@@ -15871,7 +15883,7 @@
 	}
 
       SET_TEXT_POS_FROM_MARKER (startp, w->start);
-      start_display (&it, w, startp);
+      start_display (&it, w, startp, 0);
       it.last_visible_x = INT_MAX;
       whole = move_it_to (&it, -1, INT_MAX, window_box_height (w), -1,
 			  MOVE_TO_X | MOVE_TO_Y);
@@ -16184,7 +16196,7 @@
       ptrdiff_t it_charpos;
 
       w->optional_new_start = 0;
-      start_display (&it, w, startp);
+      start_display (&it, w, startp, 0);
       move_it_to (&it, PT, 0, it.last_visible_y, -1,
 		  MOVE_TO_POS | MOVE_TO_X | MOVE_TO_Y);
       /* Record IT's position now, since line_bottom_y might change
@@ -16577,7 +16589,7 @@
     w->base_line_number = 0;
 
   /* Determine the window start relative to point.  */
-  init_iterator (&it, w, PT, PT_BYTE, NULL, DEFAULT_FACE_ID);
+  init_iterator (&it, w, PT, PT_BYTE, NULL, DEFAULT_FACE_ID, 0);
   it.current_y = it.last_visible_y;
   if (centering_position < 0)
     {
@@ -16606,7 +16618,7 @@
 	  void *it1data = NULL;
 
 	  SAVE_IT (it1, it, it1data);
-	  start_display (&it1, w, startp);
+	  start_display (&it1, w, startp, 0);
 	  move_it_vertically (&it1, margin * frame_line_height);
 	  margin_pos = IT_CHARPOS (it1);
 	  RESTORE_IT (&it, &it, it1data);
@@ -16669,7 +16681,7 @@
      containing PT in this case.  */
   if (it.current_y <= 0)
     {
-      init_iterator (&it, w, PT, PT_BYTE, NULL, DEFAULT_FACE_ID);
+      init_iterator (&it, w, PT, PT_BYTE, NULL, DEFAULT_FACE_ID, 0);
       move_it_vertically_backward (&it, 0);
       it.current_y = 0;
     }
@@ -17013,7 +17025,7 @@
   overlay_arrow_seen = 0;
 
   /* Initialize iterator and info to start at POS.  */
-  start_display (&it, w, pos);
+  start_display (&it, w, pos, 0);
   it.glyph_row->reversed_p = false;
 
   /* Display all lines of W.  */
@@ -17161,7 +17173,7 @@
 	 last_text_row is set to the last row displayed that displays
 	 text.  Note that it.vpos == 0 if or if not there is a
          header-line; it's not the same as the MATRIX_ROW_VPOS!  */
-      start_display (&it, w, new_start);
+      start_display (&it, w, new_start, 0);
       w->cursor.vpos = -1;
       last_text_row = last_reused_text_row = NULL;
 
@@ -18142,7 +18154,7 @@
     {
       /* There are no reusable lines at the start of the window.
 	 Start displaying in the first text line.  */
-      start_display (&it, w, start);
+      start_display (&it, w, start, 0);
       it.vpos = it.first_vpos;
       start_pos = it.current.pos;
     }
@@ -18932,7 +18944,7 @@
   int n_glyphs_before;
 
   set_buffer_temp (buffer);
-  init_iterator (&it, w, -1, -1, &scratch_glyph_row, DEFAULT_FACE_ID);
+  init_iterator (&it, w, -1, -1, &scratch_glyph_row, DEFAULT_FACE_ID, 0);
   scratch_glyph_row.reversed_p = false;
   it.glyph_row->used[TEXT_AREA] = 0;
   SET_TEXT_POS (it.position, 0, 0);
@@ -21228,7 +21240,7 @@
 
       /* Setup the arena.  */
       SET_TEXT_POS (pt, PT, PT_BYTE);
-      start_display (&it, w, pt);
+      start_display (&it, w, pt, 0);
 
       if (it.cmp_it.id < 0
 	  && it.method == GET_FROM_STRING
@@ -21319,7 +21331,7 @@
 	{
 	  if (pt_x > 0)
 	    {
-	      start_display (&it, w, pt);
+              start_display (&it, w, pt, 0);
 	      reseat_at_previous_visible_line_start (&it);
 	      it.current_x = it.current_y = it.hpos = 0;
 	      if (pt_vpos != 0)
@@ -21622,7 +21634,7 @@
 
 #if defined (USE_X_TOOLKIT) || defined (USE_GTK)
   eassert (!FRAME_WINDOW_P (f));
-  init_iterator (&it, w, -1, -1, f->desired_matrix->rows, MENU_FACE_ID);
+  init_iterator (&it, w, -1, -1, f->desired_matrix->rows, MENU_FACE_ID, 0);
   it.first_visible_x = 0;
   it.last_visible_x = FRAME_PIXEL_WIDTH (f);
 #elif defined (HAVE_X_WINDOWS) /* X without toolkit.  */
@@ -21633,7 +21645,7 @@
       struct window *menu_w;
       menu_w = XWINDOW (f->menu_bar_window);
       init_iterator (&it, menu_w, -1, -1, menu_w->desired_matrix->rows,
-		     MENU_FACE_ID);
+		     MENU_FACE_ID, 0);
       it.first_visible_x = 0;
       it.last_visible_x = FRAME_PIXEL_WIDTH (f);
     }
@@ -21643,7 +21655,7 @@
       /* This is a TTY frame, i.e. character hpos/vpos are used as
 	 pixel x/y.  */
       init_iterator (&it, w, -1, -1, f->desired_matrix->rows,
-		     MENU_FACE_ID);
+		     MENU_FACE_ID, 0);
       it.first_visible_x = 0;
       it.last_visible_x = FRAME_COLS (f);
     }
@@ -21759,7 +21771,7 @@
   if (y >= f->desired_matrix->nrows)
     return;
 
-  init_iterator (&it, w, -1, -1, f->desired_matrix->rows + y, MENU_FACE_ID);
+  init_iterator (&it, w, -1, -1, f->desired_matrix->rows + y, MENU_FACE_ID, 0);
   it.first_visible_x = 0;
   it.last_visible_x = FRAME_COLS (f) - 1;
   row = it.glyph_row;
@@ -21932,7 +21944,7 @@
   struct face *face;
   ptrdiff_t count = SPECPDL_INDEX ();
 
-  init_iterator (&it, w, -1, -1, NULL, face_id);
+  init_iterator (&it, w, -1, -1, NULL, face_id, 0);
   /* Don't extend on a previously drawn mode-line.
      This may happen if called from pos_visible_p.  */
   it.glyph_row->enabled_p = false;
@@ -22651,7 +22663,7 @@
   Fselect_window (window, Qt);
   set_buffer_internal_1 (XBUFFER (buffer));
 
-  init_iterator (&it, w, -1, -1, NULL, face_id);
+  init_iterator (&it, w, -1, -1, NULL, face_id, 0);
 
   if (no_props)
     {
@@ -31056,6 +31068,18 @@
 	       doc: /*  */);
   Vredisplay__mode_lines_cause
     = Fmake_vector (make_number (100), make_number (0));
+
+  DEFVAR_LISP ("use-default-face-for-fast-scrolling",
+               Vuse_default_face_for_fast_scrolling,
+               doc: /* When non-nil, accelerate repeated scrolling operations.
+This comes into play when scrolling rapidly over previously
+unfontified buffer regions.  Only those portions of the buffer which
+are actually going to be displayed get fontified.
+
+Note that this optimization can cause the portion of the buffer displayed to
+be slightly different.  */);
+
+  Vuse_default_face_for_fast_scrolling = Qnil;
 }
 
 



> Thanks.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Unfreezing the display during auto-repeated scrolling. [ Was: Aborting display. Is this possible? ]

[Eli Zaretskii]

> Date: Sun, 26 Oct 2014 20:03:13 +0000
> Cc: monnier@IRO.UMontreal.CA, emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> > You cannot possibly know, on this low level, which code will use the
> > "not for display" functions, even if today they are only used in a
> > context where ignoring font-lock faces might be OK.
> 
> The contract of xdisp.c says explicitly that the only visible display
> function is redisplay, and this has no IT parameters.  So any IT with
> not_for_display set should be "safe" if it is defined outside of xdisp.c

No, that's wrong, because subroutines of redisplay_internal call the
"not-for-display" functions in many places, and they certainly do NOT
expect that those function behave differently, in terms of pixel
metrics and coordinates, from the "real" display.  E.g., if a series
of move_it_* calls with a certain goal winds up at some pixel
coordinates within the current window, the code expects to find itself
at the same place when it actually displays the text.  If this
condition is violated (because move_it_* functions ignore font-lock
faces), then all kinds of subtle breakage will ensue.

> > You already marked as "not for display" line-pixel-height, which must
> > be accurate in the results it returns, since otherwise it will violate
> > its contract.  Likewise in resize_mini_window (which is definitely "for
> > display"), redisplay_internal (likewise), etc.  OTOH, displaying
> > strings (e.g., as part of mode-line display) will never bump into
> > font-lock faces, so setting this flag there is not useful.
> 
> Sorry, I'd left unnecessary not_for_display's set.  I've cleared out
> xdisp.c, such that the only function which sets not_for_display is
> pos_visible_p.

That's also wrong: that function is a base for many APIs, specifically
the posn-at-* functions, that many features use and depend on.  You
cannot violate their accuracy.

> The other functions which set not_for_display are
> (i) window_scroll_pixel_based (window.c);
> (ii) Fvertical_motion (indent.c).

Fvertical_motion is called in many places.  Martin worked very hard to
make window and frame redisplay be accurate to a single pixel; you are
now going to ruin all that.

> > So if we go this way, we are digging a hole for ourselves -- it is an
> > impossible maintenance burden to have functions that don't exactly do
> > what they say they do under some circumstances.
> 
> Yes.  But the current implementation doesn't work very well.  I'm trying
> to fix it, because it's irritating.  I think I've made good progress.  I
> can't see any other approach to fixing it at the moment.

I do, at least two of them, and I pointed them out.  In any case, if
there's no good solution for a problem, it certainly doesn't mean we
should compromise with a not-so-good one.

> > You should only make changes in window_scroll_* functions, ...
> 
> I honestly can't see how I could achieve what's wanted if I restrict
> myself to window.c.

Please tell what prevents you from doing that.

> > And I still don't understand why writing a simple "fast-scroll"
> > function in Lisp, which just moves point N lines forward or backward,
> > is not a better solution.  It's certainly easier and has exactly zero
> > impact on our infrastructure.
> 
> That's precisely how Fscroll_up and Fscroll_down work.

No, they don't.  If they did, you wouldn't need to disable
fontification, because moving point doesn't need it.

What Fscroll_down and Fscroll_up do is move the _window_start_, and
then figure out whether point is still visible after that.  When you
move point first, all this is not needed, especially if you don't care
that point will move slightly less or more than a screenful.

Re: Unfreezing the display during auto-repeated scrolling. [ Was: Aborting display. Is this possible? ]

[Stefan Monnier]

I'm not sure exactly what to think here.

I think it is very desirable to try and come up with ways to speed up
redisplay in cases such as auto-repeated scrolling.

It seems that Alan's patch has a fairly small impact on the C code (in
terms of its structure, not necessarily in terms of the end result),
which is good.

But even better would be something that has no impact at all on the
C code.  Of course, I'm biased: I hacked my own defer-lock.el before
Simon Marshall published his.

So, I'd be in favor of trying to solve this at the Elisp level,
i.e. inside jit-lock.  I think the result might even be better than
Alan's patch because it should let Emacs keep up with scrolling much
more easily (tho displaying un-font-locked content while scrolling).

What I suggest is to refine jit-lock-defer such that jit-lock is
deferred if there's input pending.

I understand there's an underlying problem here, which is that "input
pending" is something that's not tested reliably.  Maybe we will need to
try and fix this as well.


        Stefan

Re: Unfreezing the display during auto-repeated scrolling.

[Alan Mackenzie]

Hello, Stefan.

On Sun, Oct 26, 2014 at 04:42:38PM -0400, Stefan Monnier wrote:
> I'm not sure exactly what to think here.

> I think it is very desirable to try and come up with ways to speed up
> redisplay in cases such as auto-repeated scrolling.

> It seems that Alan's patch has a fairly small impact on the C code (in
> terms of its structure, not necessarily in terms of the end result),
> which is good.

> But even better would be something that has no impact at all on the
> C code.  Of course, I'm biased: I hacked my own defer-lock.el before
> Simon Marshall published his.

> So, I'd be in favor of trying to solve this at the Elisp level, ...

I don't think this is possible.  At least some minimal, finicky changes
will be needed where xdisp.c calls fontification functions.  (See below).

> ... i.e. inside jit-lock.  I think the result might even be better than
> Alan's patch because it should let Emacs keep up with scrolling much
> more easily (tho displaying un-font-locked content while scrolling).

The problem is, _every_ scroll operation into unfontified territory is
calling Fvertical_motion (or worse in the GUI case).  Fvertical_motion
calls start_display calls init_iterator calls reseat calls handle_stop
calls handle_fontified_prop.

At this stage, _nothing_ can stop handle_fontified_prop running
fontification-functions causing the fontification of >=
jit-lock-chunk-size (500) bytes.  Well, my patch stopped this, but
nothing else currently in the system can.

500+ bytes is about a fifth of the 2500 bytes which fit on my 65 line
screen, so fontifying them is going to take ~.017s (see other email
threads for details of these measurements).  My auto-repeat repeat rate
is every 0.024s, leaving 0.007s for other processing if Emacs is going to
keep up.  It can't; 0.007s isn't enough.

The above fontification is happening on _every_ scroll operation, even
though the fontified text is never going to appear on the screen.  The
sole purpose of this fontification is to work out how big (in pixels)
characters are.  And this is in the tty case.

The display code has been optimised for variable sized characters, and
all possibilities of optimising for the very common case of uniform
sized characters has been thrown out the window.  My patch was a way for
a user, in effect, to tell Emacs that his characters are all the same
size.

Something's got to give.

Maybe Eli's suggestion of rewriting scroll-up and scroll-down from
scratch in Elisp could work, but if I'm not mistaken, there's no
infrastructural support for window lines, etc., which doesn't involve
fontification.  This would have to be written.

If movement on a window absolutely requires fontified characters, then
some means will need to be found for the display code to tell jit-lock
HOW MUCH needs fontifying, and for that amount to be much less than 500
bytes in the non-display case.

> What I suggest is to refine jit-lock-defer such that jit-lock is
> deferred if there's input pending.

> I understand there's an underlying problem here, which is that "input
> pending" is something that's not tested reliably.  Maybe we will need to
> try and fix this as well.

As already pointed out, even one jit-lock-chunk-size fontification per
0.024s in C Mode is going to overfill the event queue, and that's with
nothing actually being drawn on the screen.

One might hold that it is unreasonable to expect Emacs to keep up with 42
PageDown's per second with a 65 line window on a mere 2.6 Ghz processor.
I disagree.

One might argue that CC Mode fontification should be speeded up.  Yes, it
should, but it's not ever going to be speeded up by an order of
magnitude.  Even in Emacs Lisp Mode, fontifying and displaying a 65 line
screen averages out at 0.018s, compared with a 0.024s repeat rate.
There's simply no spare processing capacity for frivolous
jit-lock-chunk-size'd fontifications.

Something's got to give.

>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Unfreezing the display during auto-repeated scrolling.

[Stefan Monnier]

> At this stage, _nothing_ can stop handle_fontified_prop running
> fontification-functions causing the fontification of >=
> jit-lock-chunk-size (500) bytes.  Well, my patch stopped this, but
> nothing else currently in the system can.

Running fontification-functions does not necessarily imply running
font-lock!

Please take a look at what jit-lock-defer does.


        Stefan

Re: Unfreezing the display during auto-repeated scrolling. Simpler approach.

[Alan Mackenzie]

Hello, Stefan.

On Sun, Oct 26, 2014 at 09:03:40PM -0400, Stefan Monnier wrote:
> > At this stage, _nothing_ can stop handle_fontified_prop running
> > fontification-functions causing the fontification of >=
> > jit-lock-chunk-size (500) bytes.  Well, my patch stopped this, but
> > nothing else currently in the system can.

> Running fontification-functions does not necessarily imply running
> font-lock!

Neither it does!  :-)  New idea below.

> Please take a look at what jit-lock-defer does.

It sets a non-nil fontified text-property without doing any
fontification (yet).  I think you're suggesting "binding"
jit-lock-defer over the scroll operation, thus inhibiting fontification. 

I've had a simpler idea: bind fontification-functions to nil over the
scroll operation.  This avoids many of the pitfalls of my last patch, and
can be perfectly specific to scrolling by whole screens and nothing else,
and is much simpler and less intrusive.  It works well.

Here's a patch, which does precisely that.  Comments welcome, as always:



=== modified file 'src/lisp.h'
--- src/lisp.h	2014-10-15 13:37:10 +0000
+++ src/lisp.h	2014-10-27 13:19:02 +0000
@@ -829,6 +829,7 @@
 
 /* Defined in xdisp.c.  */
 extern Lisp_Object Qimage;
+extern Lisp_Object Qfontification_functions;
 \f
 
 /* Extract a value or address from a Lisp_Object.  */

=== modified file 'src/window.c'
--- src/window.c	2014-10-04 08:20:24 +0000
+++ src/window.c	2014-10-27 13:29:51 +0000
@@ -4867,11 +4867,16 @@
 static void
 window_scroll (Lisp_Object window, EMACS_INT n, bool whole, int noerror)
 {
+  ptrdiff_t count = SPECPDL_INDEX ();
+
   immediate_quit = 1;
   n = clip_to_bounds (INT_MIN, n, INT_MAX);
 
   wset_redisplay (XWINDOW (window));
 
+  if (whole && !NILP (Vuse_default_face_for_fast_scrolling))
+    specbind (Qfontification_functions, Qnil);
+  
   /* If we must, use the pixel-based version which is much slower than
      the line-based one but can handle varying line heights.  */
   if (FRAME_WINDOW_P (XFRAME (XWINDOW (window)->frame)))
@@ -4879,6 +4884,8 @@
   else
     window_scroll_line_based (window, n, whole, noerror);
 
+  unbind_to (count, Qnil);
+
   /* Bug#15957.  */
   XWINDOW (window)->window_end_valid = 0;
   immediate_quit = 0;
@@ -7468,6 +7475,17 @@
 pixelwise even if this option is nil.  */);
   window_resize_pixelwise = 0;
 
+  DEFVAR_LISP ("use-default-face-for-fast-scrolling",
+               Vuse_default_face_for_fast_scrolling,
+               doc: /* When non-nil, accelerate repeated scrolling operations.
+This comes into play when scrolling rapidly over previously
+unfontified buffer regions.  Only those portions of the buffer which
+are actually going to be displayed get fontified.
+
+Note that this optimization can cause the portion of the buffer displayed to
+be slightly different.  */);
+  Vuse_default_face_for_fast_scrolling = Qnil;
+
   defsubr (&Sselected_window);
   defsubr (&Sminibuffer_window);
   defsubr (&Swindow_minibuffer_p);

=== modified file 'src/xdisp.c'
--- src/xdisp.c	2014-10-24 09:58:43 +0000
+++ src/xdisp.c	2014-10-27 13:21:01 +0000
@@ -347,7 +347,7 @@
 /* Holds the list (error).  */
 static Lisp_Object list_of_error;
 
-static Lisp_Object Qfontification_functions;
+Lisp_Object Qfontification_functions;
 
 static Lisp_Object Qwrap_prefix;
 static Lisp_Object Qline_prefix;



>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Unfreezing the display during auto-repeated scrolling. Simpler approach.

[Eli Zaretskii]

> Date: Mon, 27 Oct 2014 14:28:01 +0000
> Cc: Eli Zaretskii <eliz@gnu.org>, emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> > Please take a look at what jit-lock-defer does.
> 
> It sets a non-nil fontified text-property without doing any
> fontification (yet).  I think you're suggesting "binding"
> jit-lock-defer over the scroll operation, thus inhibiting fontification. 
> 
> I've had a simpler idea: bind fontification-functions to nil over the
> scroll operation.  This avoids many of the pitfalls of my last patch, and
> can be perfectly specific to scrolling by whole screens and nothing else,
> and is much simpler and less intrusive.  It works well.
> 
> Here's a patch, which does precisely that.  Comments welcome, as always:

If jit-lock-defer-time produces the desired effect, I see no need to
have yet another separate knob to achieve the same effect just during
scrolling.

Re: Unfreezing the display during auto-repeated scrolling. Simpler approach.

[David Engster]

Eli Zaretskii writes:
> If jit-lock-defer-time produces the desired effect, I see no need to
> have yet another separate knob to achieve the same effect just during
> scrolling.

I worked with a small value for jit-lock-defer-time for a while, but
found that this does not play well with buffers that capture process
output, like compilation or grep. They are not fontified until you press
a key.

-David

Re: Unfreezing the display during auto-repeated scrolling. Simpler approach.

[Eli Zaretskii]

> From: David Engster <deng@randomsample.de>
> Cc: Alan Mackenzie <acm@muc.de>,  monnier@iro.umontreal.ca,  emacs-devel@gnu.org
> Date: Mon, 27 Oct 2014 20:13:19 +0100
> 
> I worked with a small value for jit-lock-defer-time for a while, but
> found that this does not play well with buffers that capture process
> output, like compilation or grep. They are not fontified until you press
> a key.

That warrants a bug report, I would think.

Re: Unfreezing the display during auto-repeated scrolling. Simpler approach.

[David Engster]

Eli Zaretskii writes:
>> From: David Engster <deng@randomsample.de>
>> Cc: Alan Mackenzie <acm@muc.de>,  monnier@iro.umontreal.ca,  emacs-devel@gnu.org
>> Date: Mon, 27 Oct 2014 20:13:19 +0100
>> 
>> I worked with a small value for jit-lock-defer-time for a while, but
>> found that this does not play well with buffers that capture process
>> output, like compilation or grep. They are not fontified until you press
>> a key.
>
> That warrants a bug report, I would think.

OK, it's now filed as #18856.

-David

Re: Unfreezing the display during auto-repeated scrolling. Simpler approach.

[Alan Mackenzie]

Hello, Eli.

On Mon, Oct 27, 2014 at 06:51:35PM +0200, Eli Zaretskii wrote:
> > Date: Mon, 27 Oct 2014 14:28:01 +0000
> > Cc: Eli Zaretskii <eliz@gnu.org>, emacs-devel@gnu.org
> > From: Alan Mackenzie <acm@muc.de>

> > > Please take a look at what jit-lock-defer does.

> > It sets a non-nil fontified text-property without doing any
> > fontification (yet).  I think you're suggesting "binding"
> > jit-lock-defer over the scroll operation, thus inhibiting fontification. 

> > I've had a simpler idea: bind fontification-functions to nil over the
> > scroll operation.  This avoids many of the pitfalls of my last patch, and
> > can be perfectly specific to scrolling by whole screens and nothing else,
> > and is much simpler and less intrusive.  It works well.

> > Here's a patch, which does precisely that.  Comments welcome, as always:

> If jit-lock-defer-time produces the desired effect, I see no need to
> have yet another separate knob to achieve the same effect just during
> scrolling.

I've just tried out jit-lock-defer-time with 0.03s and 0.001s.  (Just as
a matter of interest, j-l-d-time changes only take effect on restarting
font-lock mode, so it's best to make changes with f-l mode disabled.)

At 0.03s, the effect on holding down PageDown was for seemingly all
screens to be displayed at they scrolled by, but unfontified.  There was
no waiting for Emacs to be ready for more keypresses after releasing the
PageDown key.  However, on single PageDowns, the fact that the text is
first displayed and only later (~0.1s) fontified could be disturbing to
some people.

At 0.001s, the screens scrolled by more stutteringly, some of them
managing to get fontified, with hints of colour visible from time to
time.  Otherwise, much like 0.03s.

Overall, using jit-lock-defer-time feels more like a workaround than a
solution, but perhaps I'm not the best person to judge.  If it were
possible, on auto-repeat, to have all screens displayed unfontified, yet
not have the 0.1s delay between delay and fontification on single
keypresses, that would be the best of all worlds.

In any case, some extra documentation on page "Scrolling" in the Emacs
manual is probably warranted.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Unfreezing the display during auto-repeated scrolling. Simpler approach.

[Stefan Monnier]

>> If jit-lock-defer-time produces the desired effect,

It doesn't, currently.  But the technique can be tweaked.

> PageDown key.  However, on single PageDowns, the fact that the text is
> first displayed and only later (~0.1s) fontified could be disturbing to
> some people.

I suggested earlier to change jit-lock-defer so that deferral only takes
place if there's pending input.  It might take care of this case.


        Stefan

Re: Unfreezing the display during auto-repeated scrolling. Simpler approach.

[Alan Mackenzie]

Hello, Stefan.

On Mon, Oct 27, 2014 at 05:38:06PM -0400, Stefan Monnier wrote:
> >> If jit-lock-defer-time produces the desired effect,

> It doesn't, currently.  But the technique can be tweaked.

> > PageDown key.  However, on single PageDowns, the fact that the text is
> > first displayed and only later (~0.1s) fontified could be disturbing to
> > some people.

> I suggested earlier to change jit-lock-defer so that deferral only takes
> place if there's pending input.  It might take care of this case.

It doesn't.  :-(

I've set my jit-lock-defer-time to 0.03s.

Then I tried adding this condition to jit-lock-function, so that it
looks like:

  (when (and jit-lock-mode (not memory-full))
    (if (or (null jit-lock-defer-timer)
            (not (input-pending-p)))      <================
        <un-deferred fontification>
      <deferred fontification>
	))

What then happens is that some fontification requests from the scrolling
code (NOT from display) become "non-deferred", and jit-lock fontifies
these chunks.  By the time it's done this, there are more PageDown
events in the input queue, so Emacs doesn't get to redisplay anything.
The screen is frozen, but becomes free as soon as the key is released
(within a second).  This is inadequate.

I've tried making the C variable redisplaying_p visible in lisp (through
the new function `redisplaying-p') and testing that in the above
condition too, so that we only do undeferred fontification when the request
comes from redisplay.  Aside from the objectionableness of never
fontifying for any reason other than redisplay, this doesn't seem to
help matters: auto-repeating PageDown rapidly scrolls all(?) buffer
regions unfontified, but there is the ~0.1s gap between displaying the
last screen and it being fontified.  This is also not good.

I don't think jit-lock's defer can be brought to do what we want.

>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Unfreezing the display during auto-repeated scrolling. Simpler approach.

[Stefan Monnier]

>> > PageDown key.  However, on single PageDowns, the fact that the text is
>> > first displayed and only later (~0.1s) fontified could be disturbing to
>> > some people.
>> I suggested earlier to change jit-lock-defer so that deferral only takes
>> place if there's pending input.  It might take care of this case.
> It doesn't.  :-(

By "this case" I meant "the case of single page-downs".  AFAICT your
experiment focused on auto-repeating page-downs, which is a different case.

> I've set my jit-lock-defer-time to 0.03s.

I think it's better to keep it above 0.1s.

> What then happens is that some fontification requests from the scrolling
> code (NOT from display) become "non-deferred", and jit-lock fontifies
> these chunks.  By the time it's done this, there are more PageDown
> events in the input queue, so Emacs doesn't get to redisplay anything.
> The screen is frozen, but becomes free as soon as the key is released
> (within a second).  This is inadequate.

OK, so it did improve the behavior in the sense that Emacs can now keep
up with the scroll commands themselves (tho not the redisplay), so you
don't have to wait many seconds when you release the key.

But indeed, we have remaining problems.  I guess the main one is:

If we font-lock during the scroll command and that font-lock takes
a while, then by the time we get to redisplay the input queue is not
empty any more, so we end up skipping redisplay, so this font-locking
was kind of wasted.  It would have been better to either not font-lock
at all, or to font-lock during redisplay rather than during the scroll
command (since that would mean that at least redisplay does happen).

Basically if we're in jit-lock and there's input pending, there's
a pretty good chance we're not doing redisplay (since redisplay would
have been skipped if there was input pending).  More specifically, the
only case where we can have input pending in jit-lock during redisplay
is when the input arrived between the time redisplay started and the
time jit-lock gets called, and usually this time is very short.
So my idea doesn't work.

> I've tried making the C variable redisplaying_p visible in lisp (through
> the new function `redisplaying-p') and testing that in the above
> condition too, so that we only do undeferred fontification when the request
> comes from redisplay.  Aside from the objectionableness of never
> fontifying for any reason other than redisplay, this doesn't seem to
> help matters: auto-repeating PageDown rapidly scrolls all(?) buffer
> regions unfontified, but there is the ~0.1s gap between displaying the
> last screen and it being fontified.  This is also not good.

Hmm... the redisplay that happens in response to the last page-down should
perform font-lock since there's no input pending.  So I don't understand
why you see the display unfontified first, followed by fontification
0.1s later.
Oh, wait, I guess it's because the previous redisplay had its font-lock
deferred so the next 500 bytes were marked as `fontified = defer', so on
the "final" redisplay, we may font-lock some additional part of the
buffer, but there's a significant part of the visible part of the buffer
that's already been marked as `fontified = defer'.

I'm beginning to like your C-level hack.


        Stefan

Re: Unfreezing the display during auto-repeated scrolling. Simpler approach.

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Tue, 28 Oct 2014 20:57:16 -0400
> Cc: Eli Zaretskii <eliz@gnu.org>, emacs-devel@gnu.org
> 
> I'm beginning to like your C-level hack.

And I still dislike it.

Re: Unfreezing the display during auto-repeated scrolling. Simpler approach.

[Alan Mackenzie]

Hello, Eli.

On Wed, Oct 29, 2014 at 04:14:12PM +0200, Eli Zaretskii wrote:
> > From: Stefan Monnier <monnier@iro.umontreal.ca>
> > Date: Tue, 28 Oct 2014 20:57:16 -0400
> > Cc: Eli Zaretskii <eliz@gnu.org>, emacs-devel@gnu.org

> > I'm beginning to like your C-level hack.

> And I still dislike it.

Just to be absolutely clear, we're talking about my _simpler_ hack, the
one that binds fontification-functions to nil during scrolling.  The
previous complicated one, that messed with "struct it"s, I think we can
all agree is dead.

What that proposed change does is simple and very restricted.  It binds
fontification-functions to nil ONLY in the two scrolling function (up
and down), ONLY when scrolling is being done by screenfulls rather than
by lines, and ONLY when the user option (precise mechanism not yet
worked out) use-default-face-for-fast-scrolling is non-nil.

By suggesting the use of jit-lock-defer, somehow, to speed up scrolling,
the principle that scrolled over buffer regions MUST be fontified has
been given up.

My attempts to get jit-lock-defer to do the Right Thing yesterday
weren't encouraging.

I'm guessing you still accept that the problem (frozen screen during and
after auto-repeated scrolling) needs to be solved.

An approach I haven't tried yet would be to use the existing scrolling
functionality for discreet key presses, changing to
nilled-out-fontification-functions when the keystrokes are coming in at
auto-repeat rates.  This might need enhancement of the command loop, or
keyboard interrupt handler, or whatever.

What do you think?

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Unfreezing the display during auto-repeated scrolling. Simpler approach.

[Eli Zaretskii]

> Date: Wed, 29 Oct 2014 14:52:11 +0000
> Cc: Stefan Monnier <monnier@iro.umontreal.ca>, emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> On Wed, Oct 29, 2014 at 04:14:12PM +0200, Eli Zaretskii wrote:
> > > From: Stefan Monnier <monnier@iro.umontreal.ca>
> > > Date: Tue, 28 Oct 2014 20:57:16 -0400
> > > Cc: Eli Zaretskii <eliz@gnu.org>, emacs-devel@gnu.org
> 
> > > I'm beginning to like your C-level hack.
> 
> > And I still dislike it.
> 
> Just to be absolutely clear, we're talking about my _simpler_ hack, the
> one that binds fontification-functions to nil during scrolling.

I didn't realize that, thanks for clarifying.

Yes, that's better.

> ONLY when the user option (precise mechanism not yet worked out)
> use-default-face-for-fast-scrolling is non-nil.

As long as CC Mode doesn't set that by default, I can live with that.

> By suggesting the use of jit-lock-defer, somehow, to speed up scrolling,
> the principle that scrolled over buffer regions MUST be fontified has
> been given up.

That wasn't my reasoning, it was the opposite: _if_ we are willing to
accept that the scrolled-over parts be unfontified, _then_ we already
have an optional feature which does precisely that.

> I'm guessing you still accept that the problem (frozen screen during and
> after auto-repeated scrolling) needs to be solved.

Yes, but (a) it is IMO not a very important use case, and (b) the best
way to solve it is where it is caused, i.e. in CC Mode.  Otherwise, we
are on a slippery slope: here is the mode that succeeds in wedging
redisplay, and we are "solving" it by disabling important parts of the
display engine.  No offense, but that's tail wagging the dog.  Next
time another mode will come up with the same problem, what do we say
to ask the author/maintainer to get their act together?

> What do you think?

Given that no one is willing to do (what I think is) TRT, I guess your
suggestion is something we can live with, as long as that variable is
off by default.

Re: Unfreezing the display during auto-repeated scrolling. Simpler approach.

[Stefan Monnier]

> Just to be absolutely clear, we're talking about my _simpler_ hack, the
> one that binds fontification-functions to nil during scrolling.  The
> previous complicated one, that messed with "struct it"s, I think we can
> all agree is dead.

Yes.

> An approach I haven't tried yet would be to use the existing scrolling
> functionality for discreet key presses, changing to
> nilled-out-fontification-functions when the keystrokes are coming in at
> auto-repeat rates.

I think the problem is how to distinguish the two.  If we can
distinguish the two, then we could probably use it in jit-lock-defer.

More specifically, your jit-lock-defer experiment showed that the
problem is that to get smooth scrolling, we need to skip fontification
during the scroll even though there's no input pending (because the
next repetition hasn't arrived yet).

Another way to look at it: if running CMD required to run the
fontification functions, then we should not skip the subsequent
redisplay even if there's input pending (this is based on the assumption
that running the fontification functions takes the lion's share of
redisplay time and that running the fontification function during CMD
means that redisplay probably won't need to run the fontification
function).  Such a heuristic might make the
jit-lock-defer-when-input-pending work better.


        Stefan

Re: Unfreezing the display during auto-repeated scrolling. Simpler approach.

[Alan Mackenzie]

Hello, Stefan.

On Wed, Oct 29, 2014 at 12:59:26PM -0400, Stefan Monnier wrote:
> > An approach I haven't tried yet would be to use the existing scrolling
> > functionality for discreet key presses, changing to
> > nilled-out-fontification-functions when the keystrokes are coming in at
> > auto-repeat rates.

> I think the problem is how to distinguish the two.  If we can
> distinguish the two, then we could probably use it in jit-lock-defer.

Every event has a timestamp, doesn't it?  If it doesn't, it certainly
ought to!  If two successive keyboard events are "the same", and arrive
within kbd-auto-repeat-threshold of eachother, then the first event
should be marked as auto-repeat (assuming it is still in the event
queue).

Then there needs to be an Fthis_event_is_auto_repeated which remembers
whether the event which became the most recent key-sequence is
auto-repeated.

Then a scrolling command (or similar) can test
this-event-is-auto-repeated to discover whether or not it's got enough
breathing space to do something lengthy (such as invoking functions which
want to fontify with fontification-functions intact).

Or something like that.

> More specifically, your jit-lock-defer experiment showed that the
> problem is that to get smooth scrolling, we need to skip fontification
> during the scroll even though there's no input pending (because the
> next repetition hasn't arrived yet).

Yes.

> Another way to look at it: if running CMD required to run the
> fontification functions, then we should not skip the subsequent
> redisplay even if there's input pending (this is based on the assumption
> that running the fontification functions takes the lion's share of
> redisplay time and that running the fontification function during CMD
> means that redisplay probably won't need to run the fontification
> function).  Such a heuristic might make the
> jit-lock-defer-when-input-pending work better.

Running display when there's input available is a fairly drastic change
to the command loop.  Also, something, somewhere will need to determine
whether the fontification called for CMD covers the same buffer region as
what is about to be redisplayed.

>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Unfreezing the display during auto-repeated scrolling. Simpler approach.

[Stefan Monnier]

>> I think the problem is how to distinguish the two.  If we can
>> distinguish the two, then we could probably use it in jit-lock-defer.
> Every event has a timestamp, doesn't it?  If it doesn't, it certainly
> ought to!  If two successive keyboard events are "the same", and arrive
> within kbd-auto-repeat-threshold of eachother, then the first event
> should be marked as auto-repeat (assuming it is still in the event
> queue).

If it's still in the queue, then input-pending-p will return non-nil
while it's processed, so jit-lock-defer-when-input-pending would already
handle it "right".

> Then there needs to be an Fthis_event_is_auto_repeated which remembers
> whether the event which became the most recent key-sequence is
> auto-repeated.

We could indeed try to recognize auto-repeat by looking at the time
since the last event, but it seems tricky to make it work well,
especially it seems tricky to correctly handle the last such event.

>> Another way to look at it: if running CMD required to run the
>> fontification functions, then we should not skip the subsequent
>> redisplay even if there's input pending (this is based on the assumption
>> that running the fontification functions takes the lion's share of
>> redisplay time and that running the fontification function during CMD
>> means that redisplay probably won't need to run the fontification
>> function).  Such a heuristic might make the
>> jit-lock-defer-when-input-pending work better.
> Running display when there's input available is a fairly drastic change
> to the command loop.

I don't think it's technically drastic.

> Also, something, somewhere will need to determine whether the
> fontification called for CMD covers the same buffer region as what is
> about to be redisplayed.

My idea was to assume that it is.  At least, it seems unlikely that the
two regions would be very far apart.

My reasoning is that skipping redisplay should not be done for long
periods of time.  The reason we want to skip redisplay is to let Emacs
perform "real work" instead, but there's a tension where the user also
wants to see the intermediate states.

Also the gain from skipping redisplay depends on the ratio between "time
to run the command" compared to "time to perform redisplay".  So if the
command took a long time, skipping redisplay provides less gain.

So I think we really should force redisplay despite pending input
every once in a while.  The question is to figure out under which
circumstances this should be done, so that those extra redisplays don't
slow Emacs substantially overall (i.e. it stays a small fraction of the
over processing time) yet they do let the user see intermediate states.


        Stefan

Re: Unfreezing the display during auto-repeated scrolling. Simpler approach.

[Alan Mackenzie]

Hello, Stefan.

On Wed, Oct 29, 2014 at 09:49:27PM -0400, Stefan Monnier wrote:

[ ... ]

> We could indeed try to recognize auto-repeat by looking at the time
> since the last event, but it seems tricky to make it work well,
> especially it seems tricky to correctly handle the last such event.

Let's just say that what I wrote last night was utter rubbish.  We cannot
handle the last event differently, because when we start handling it, the
next event has not yet failed to arrive.  My scheme from last night can
deliver nothing more than `input-pending-p' already can.

> >> Another way to look at it: if running CMD required to run the
> >> fontification functions, then we should not skip the subsequent
> >> redisplay even if there's input pending (this is based on the assumption
> >> that running the fontification functions takes the lion's share of
> >> redisplay time and that running the fontification function during CMD
> >> means that redisplay probably won't need to run the fontification
> >> function).  Such a heuristic might make the
> >> jit-lock-defer-when-input-pending work better.
> > Running display when there's input available is a fairly drastic change
> > to the command loop.

> I don't think it's technically drastic.

Who's going to decide when a "forced" display is to take place?  We
could, I suppose, set a flag in window_scroll when there's no pending
input, and for the command loop to detect this flag and use it to give
preference to display over reading keys.

But we don't know what command is invoking window_scroll.  It might
easily be a user written command for which premature display is not
wanted. 

> > Also, something, somewhere will need to determine whether the
> > fontification called for CMD covers the same buffer region as what is
> > about to be redisplayed.

> My idea was to assume that it is.  At least, it seems unlikely that the
> two regions would be very far apart.

I think that's fair enough.

> My reasoning is that skipping redisplay should not be done for long
> periods of time.  The reason we want to skip redisplay is to let Emacs
> perform "real work" instead, but there's a tension where the user also
> wants to see the intermediate states.

So maybe a timer (?0.2s) could be inserted into the command loop, such
that if Emacs has been processing commands for that long since the last
redisplay, redisplay will be done in preference to processing the next
command.

> Also the gain from skipping redisplay depends on the ratio between "time
> to run the command" compared to "time to perform redisplay".  So if the
> command took a long time, skipping redisplay provides less gain.

That's not quite so clear when it's font-locking which is taking the
time, since it happens both in command processing and display.

> So I think we really should force redisplay despite pending input
> every once in a while.  The question is to figure out under which
> circumstances this should be done, so that those extra redisplays don't
> slow Emacs substantially overall (i.e. it stays a small fraction of the
> over processing time) yet they do let the user see intermediate states.

Coming back to scrolling through xdisp.c, merely forcing a redisplay from
time to time isn't going to help the slowness of the scrolling.  Either
eliminating most of the fontification or speeding it up radically (as Eli
would prefer) is necessary.

>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Unfreezing the display during auto-repeated scrolling. Simpler approach.

[Stefan Monnier]

> Who's going to decide when a "forced" display is to take place?

That's the question, indeed.  Maybe we could simply move the
`input-pending' test so it's done before running the command.  I.e. if
there was no input pending when the command started, then don't skip
the redisplay.

For short-running commands, this will make no difference (since the
probability that input arrives right during its execution is small), but
and for long-running commands it may force redisplay even though there's
input pending, but in that case the proportion of time spent in
redisplay is smaller since the command's runtime is long.

> So maybe a timer (?0.2s) could be inserted into the command loop, such
> that if Emacs has been processing commands for that long since the last
> redisplay, redisplay will be done in preference to processing the next
> command.

That's also an option, indeed.  I don't like the sound of a preset
duration between redisplays, tho.  I'd rather measure the average
redisplay time and then use a multiple of that.

> That's not quite so clear when it's font-locking which is taking the
> time, since it happens both in command processing and display.

Yes, reality is more complex, of course.  Hard to tell what will really
work without an actual experiment.

> Coming back to scrolling through xdisp.c, merely forcing a redisplay from
> time to time isn't going to help the slowness of the scrolling.  Either
> eliminating most of the fontification or speeding it up radically (as Eli
> would prefer) is necessary.

Your experience has shown that jit-lock-defer-if-input-pending does let
Emacs keep up (in the sense that Emacs comes back to life when you
release the key), so I presume we'd use that, and combine it with
an occasional forced redisplay.


        Stefan

Re: Unfreezing the display during auto-repeated scrolling. Simpler approach.

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Eli Zaretskii <eliz@gnu.org>,  emacs-devel@gnu.org
> Date: Thu, 30 Oct 2014 23:06:09 -0400
> 
> > Who's going to decide when a "forced" display is to take place?
> 
> That's the question, indeed.  Maybe we could simply move the
> `input-pending' test so it's done before running the command.  I.e. if
> there was no input pending when the command started, then don't skip
> the redisplay.

I hope the intent is that this behavior will be optional, and by
default turned off.  If not, I'm going to object to such a change.

Remember, all this was triggered by a single use case: leaning on the
PageDown (or C-v or M-v) key.  IMO, this use case is not very
important, although others might disagree.  In any case, changing the
way redisplay works by default due to this use case makes little sense
to me.

(Apologies if I lost track of what exactly you are discussing here.)

Re: Unfreezing the display during auto-repeated scrolling. Simpler approach.

[Stefan Monnier]

>> That's the question, indeed.  Maybe we could simply move the
>> `input-pending' test so it's done before running the command.  I.e. if
>> there was no input pending when the command started, then don't skip
>> the redisplay.
> I hope the intent is that this behavior will be optional, and by
> default turned off.

The jit-lock-defer-if-input-pending will be optional (and disabled by default).

OTOH I think the above tweak to the "don't redisplay if input pending" rule
might not even need such a config var.

> If not, I'm going to object to such a change.

I'm not dead set on foisting this tweak on every user without even
letting them opt out.  Hell, I don't even know if it will do what I want
it to do.  But at least, I don't see any obvious significant downside
to it.


        Stefan

Re: Unfreezing the display during auto-repeated scrolling. Simpler approach.

[Alan Mackenzie]

Hello, Stefan and Eli.

This matter was hanging in the air when Emacs was converted to git.

Just a short summary of this change: introduce a new user option, off by
default, which suppresses fontification (by binding
fontification-functions to nil) when the user scrolls by whole screens.

This change is ready to be applied to the Emacs master branch.  Is it OK
to apply it now?

For reference here is the change:



New feature optionally to accelerate auto-repeated scrolling.

src/xdisp.c: Remove "static" from declaration of
Qfontification_functions.

src/window.c (window_scroll): bind fontification-functions to nil when
scrolling by whole screens and use-default-face-for-fast-scrolling is
non-nil.
(syms_of_window): New DEFVAR_BOOL use-default-face-for-fast-scrolling.

src/lisp.h (bool): Declare Qfontification_functions extern.

lisp/cus-start.el (all): Add use-default-face-for-fast-scrolling.

doc/emacs/display.texi (Scrolling): use-default-face-for-fast-scrolling.
Describe new variable.

etc/NEWS: Add entry for use-default-face-for-fast-scrolling.




diff --git a/doc/emacs/ChangeLog b/doc/emacs/ChangeLog
index aa9804f..409fb54 100644
--- a/doc/emacs/ChangeLog
+++ b/doc/emacs/ChangeLog
@@ -1,3 +1,8 @@
+2014-11-21  Alan Mackenzie  <acm@muc.de>
+
+	* display.texi (Scrolling): use-default-face-for-fast-scrolling.
+	Describe new variable.
+
 2014-11-21  Eli Zaretskii  <eliz@gnu.org>
 
 	* maintaining.texi (Version Control Systems): Move "@end itemize"
diff --git a/doc/emacs/display.texi b/doc/emacs/display.texi
index a5555d5..f655ed3 100644
--- a/doc/emacs/display.texi
+++ b/doc/emacs/display.texi
@@ -127,6 +127,19 @@ the mouse wheel (@pxref{Mouse Commands}); in general, it affects any
 command that has a non-@code{nil} @code{scroll-command} property.
 @xref{Property Lists,,, elisp, The Emacs Lisp Reference Manual}.
 
+@vindex use-default-face-for-fast-scrolling
+  Sometimes, particularly when you hold down keys such as @kbd{C-v}
+and @kbd{M-v}, activating keyboard auto-repeat, Emacs fails to keep up
+with the rapid rate of scrolling requested; the display doesn't update
+and Emacs can become unresponsive to input for quite a long time.  You
+can counter this sluggishness by setting the variable
+@code{use-default-face-for-fast-scrolling} to a non-@code{nil} value.
+This instructs the scrolling commands not to fontify (@pxref{Font
+Lock}) any unfontified text they scroll over, instead to assume it has
+the default face.  This can cause Emacs to scroll to somewhat wrong
+buffer positions when the faces in use are not all the same size, even
+with single (i.e. without auto-repeat) scrolling operations.
+
 @vindex scroll-up
 @vindex scroll-down
 @findex scroll-up-line
diff --git a/etc/ChangeLog b/etc/ChangeLog
index d57190f..19a542a 100644
--- a/etc/ChangeLog
+++ b/etc/ChangeLog
@@ -1,3 +1,7 @@
+2014-11-21  Alan Mackenzie  <acm@muc.de>
+
+	* NEWS: Add entry for use-default-face-for-fast-scrolling.
+
 2014-11-19  Leo Liu  <sdl.web@gmail.com>
 
 	* NEWS: Mention new macro define-advice.
diff --git a/etc/NEWS b/etc/NEWS
index 769d2c6..2d27639 100644
--- a/etc/NEWS
+++ b/etc/NEWS
@@ -113,6 +113,12 @@ this has no effect.
 *** A new text property `inhibit-read-only' can be used in read-only
 buffers to allow certain parts of the text to be writable.
 
++++
+** New variable `use-default-face-for-fast-scrolling' inhibits
+fontification during full screen scrolling operations, giving less
+hesitant operation during auto-repeat of C-v, M-v at the cost of
+possible inaccuracies in the end position.
+
 \f
 * Editing Changes in Emacs 25.1
 
diff --git a/lisp/cus-start.el b/lisp/cus-start.el
index 3b8885a..ae29c29 100644
--- a/lisp/cus-start.el
+++ b/lisp/cus-start.el
@@ -460,6 +460,7 @@ since it could result in memory overflow and make Emacs crash."
 			      :value display-buffer)
 		       (other :tag "Always (t)" :value t))
 	      "24.3")
+	     (use-default-face-for-fast-scrolling scrolling boolean "25.1")
 	     (window-resize-pixelwise windows boolean "24.4")
 	     ;; xdisp.c
 	     ;; The whitespace group is for whitespace.el.
diff --git a/src/ChangeLog b/src/ChangeLog
index b169479..f660642 100644
--- a/src/ChangeLog
+++ b/src/ChangeLog
@@ -1,3 +1,18 @@
+2014-11-21  Alan Mackenzie  <acm@muc.de>
+
+	New feature optionally to accelerate auto-repeated scrolling.
+
+	* xdisp.c: Remove "static" from declaration of
+	Qfontification_functions.
+
+	* window.c (window_scroll): bind fontification-functions to nil
+	when scrolling by whole screens and
+	use-default-face-for-fast-scrolling is non-nil.
+	(syms_of_window): New DEFVAR_BOOL
+	use-default-face-for-fast-scrolling.
+
+	* lisp.h (bool): Declare Qfontification_functions extern.
+
 2014-11-17  Paul Eggert  <eggert@cs.ucla.edu>
 
 	Improve time stamp handling, and be more consistent about it.
diff --git a/src/lisp.h b/src/lisp.h
index bdff019..7dfd1fa 100644
--- a/src/lisp.h
+++ b/src/lisp.h
@@ -829,6 +829,7 @@ extern Lisp_Object Qwindowp;
 
 /* Defined in xdisp.c.  */
 extern Lisp_Object Qimage;
+extern Lisp_Object Qfontification_functions;
 \f
 
 /* Extract a value or address from a Lisp_Object.  */
diff --git a/src/window.c b/src/window.c
index 7c2b3ca..2585d32 100644
--- a/src/window.c
+++ b/src/window.c
@@ -4869,11 +4869,16 @@ window_internal_height (struct window *w)
 static void
 window_scroll (Lisp_Object window, EMACS_INT n, bool whole, int noerror)
 {
+  ptrdiff_t count = SPECPDL_INDEX ();
+
   immediate_quit = 1;
   n = clip_to_bounds (INT_MIN, n, INT_MAX);
 
   wset_redisplay (XWINDOW (window));
 
+  if (whole && Vuse_default_face_for_fast_scrolling)
+    specbind (Qfontification_functions, Qnil);
+
   /* If we must, use the pixel-based version which is much slower than
      the line-based one but can handle varying line heights.  */
   if (FRAME_WINDOW_P (XFRAME (XWINDOW (window)->frame)))
@@ -4881,6 +4886,8 @@ window_scroll (Lisp_Object window, EMACS_INT n, bool whole, int noerror)
   else
     window_scroll_line_based (window, n, whole, noerror);
 
+  unbind_to (count, Qnil);
+
   /* Bug#15957.  */
   XWINDOW (window)->window_end_valid = 0;
   immediate_quit = 0;
@@ -7478,6 +7485,17 @@ frame's character size, at least one window may get resized
 pixelwise even if this option is nil.  */);
   window_resize_pixelwise = 0;
 
+  DEFVAR_BOOL ("use-default-face-for-fast-scrolling",
+               Vuse_default_face_for_fast_scrolling,
+               doc: /* When non-nil, accelerate repeated scrolling operations.
+This comes into play when scrolling rapidly over previously
+unfontified buffer regions.  Only those portions of the buffer which
+are actually going to be displayed get fontified.
+
+Note that this optimization can cause the portion of the buffer displayed
+after a scrolling operation to be somewhat inaccurate.  */);
+  Vuse_default_face_for_fast_scrolling = 0;
+
   defsubr (&Sselected_window);
   defsubr (&Sminibuffer_window);
   defsubr (&Swindow_minibuffer_p);
diff --git a/src/xdisp.c b/src/xdisp.c
index 1b015e7..5751693 100644
--- a/src/xdisp.c
+++ b/src/xdisp.c
@@ -347,7 +347,7 @@ Lisp_Object Qtext;
 /* Holds the list (error).  */
 static Lisp_Object list_of_error;
 
-static Lisp_Object Qfontification_functions;
+Lisp_Object Qfontification_functions;
 
 static Lisp_Object Qwrap_prefix;
 static Lisp_Object Qline_prefix;




-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Unfreezing the display during auto-repeated scrolling. Simpler approach.

[Tassilo Horn]

Alan Mackenzie <acm@muc.de> writes:

Hi Alan,

> Just a short summary of this change: introduce a new user option, off
> by default, which suppresses fontification (by binding
> fontification-functions to nil) when the user scrolls by whole
> screens.

Just as a data point: i think I'd have a use for such a feature even
with a pretty recent machine.

But I've applied your patch and run "make", fired up a new emacs and set
`use-default-face-for-fast-scrolling' to t, then found a large file and
pressed and held `C-v'.  I couldn't see any difference compared to not
having the option enabled.  All the text that's passing by is still
fontified and scrolling doesn't seem faster.

I've tried with C files and the persistency files of my working group's
graph database (which would be my primer use-case for that feature).

Bye,
Tassilo

Re: Unfreezing the display during auto-repeated scrolling. Simpler approach.

[Alan Mackenzie]

Hello, Tassilo.

On Sun, Nov 23, 2014 at 10:51:43AM +0100, Tassilo Horn wrote:
> Alan Mackenzie <acm@muc.de> writes:

> Hi Alan,

> > Just a short summary of this change: introduce a new user option, off
> > by default, which suppresses fontification (by binding
> > fontification-functions to nil) when the user scrolls by whole
> > screens.

> Just as a data point: i think I'd have a use for such a feature even
> with a pretty recent machine.

> But I've applied your patch and run "make", fired up a new emacs and set
> `use-default-face-for-fast-scrolling' to t, then found a large file and
> pressed and held `C-v'.  I couldn't see any difference compared to not
> having the option enabled.  All the text that's passing by is still
> fontified and scrolling doesn't seem faster.

The main point is, text continues to pass by.  What was happening was
that on an auto-repeated C-v, Emacs would hang (without scrolling), and
after releasing the key, it took quite a few seconds before the display
would be updated.  During this time Emacs was completely unresponsive.

With `use-default-face-for-fast-scrolling' at nil, all the text scrolled
over gets fontified, and lesser machines (such as mine) cannot keep up.
With the option set to t, only the text being displayed gets fontified.
The appearance may be of text continually scrolling, but this is an
illusion; only parts of the scrolled over text get fontified and
displayed.

To see the effect, make your window as large as possible (mine was 66
lines deep) and try with a file like .../src/xdisp.c.  Toggle
font-lock-mode off and on between each try.  If you still don't see any
difference, then your machine is powerful enough not to need the feature.

> I've tried with C files and the persistency files of my working group's
> graph database (which would be my primer use-case for that feature).

As I say, maximise your frame first.

> Bye,
> Tassilo

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Unfreezing the display during auto-repeated scrolling. Simpler approach.

[Tassilo Horn]

Alan Mackenzie <acm@muc.de> writes:

Hi Alan,

>> But I've applied your patch and run "make", fired up a new emacs and
>> set `use-default-face-for-fast-scrolling' to t, then found a large
>> file and pressed and held `C-v'.  I couldn't see any difference
>> compared to not having the option enabled.  All the text that's
>> passing by is still fontified and scrolling doesn't seem faster.
>
> The main point is, text continues to pass by.  What was happening was
> that on an auto-repeated C-v, Emacs would hang (without scrolling),
> and after releasing the key, it took quite a few seconds before the
> display would be updated.  During this time Emacs was completely
> unresponsive.
>
> With `use-default-face-for-fast-scrolling' at nil, all the text
> scrolled over gets fontified, and lesser machines (such as mine)
> cannot keep up.  With the option set to t, only the text being
> displayed gets fontified.  The appearance may be of text continually
> scrolling, but this is an illusion; only parts of the scrolled over
> text get fontified and displayed.
>
> To see the effect, make your window as large as possible (mine was 66
> lines deep) and try with a file like .../src/xdisp.c.  Toggle
> font-lock-mode off and on between each try.  If you still don't see any
> difference, then your machine is powerful enough not to need the feature.

Oh, indeed.  My machine is pretty fast at least for a laptop, but when
holding C-v the display freezes very quickly as you say.  With
`use-default-face-for-fast-scrolling' I don't get such freezes, so this
looks like a very good feature to me (2nd best after "make the display
engine faster").

But still I think the variable's name is a bit misleading.  I expected
to see the text that scrolls by black on white, i.e., using the default
face similar to having `jit-lock-defer-time' set to some non-nil value.
But instead the text that scrolls by and is visible is fontified using
the normal font-lock faces.

Bye,
Tassilo

Re: Unfreezing the display during auto-repeated scrolling. Simpler approach.

[Alan Mackenzie]

Hello, Tassilo.

On Sun, Nov 23, 2014 at 08:44:39PM +0100, Tassilo Horn wrote:
> Alan Mackenzie <acm@muc.de> writes:

> Hi Alan,

> > To see the effect, make your window as large as possible (mine was 66
> > lines deep) and try with a file like .../src/xdisp.c.  Toggle
> > font-lock-mode off and on between each try.  If you still don't see any
> > difference, then your machine is powerful enough not to need the feature.

> Oh, indeed.  My machine is pretty fast at least for a laptop, but when
> holding C-v the display freezes very quickly as you say.  With
> `use-default-face-for-fast-scrolling' I don't get such freezes, so this
> looks like a very good feature to me (2nd best after "make the display
> engine faster").

It's CC Mode's fontification, rather than the display engine, which is
slow here.  But in a large window, even Emacs Lisp Mode can only just
keep up: On my 2.6 GHz Athlon II, with a 66 line window, and the keyboard
auto-repeating every 0.024s, fontifying and displaying a screen took
~0.018s.  There's not a lot of spare time for fancier fontification.

> But still I think the variable's name is a bit misleading.  I expected
> to see the text that scrolls by black on white, i.e., using the default
> face similar to having `jit-lock-defer-time' set to some non-nil value.
> But instead the text that scrolls by and is visible is fontified using
> the normal font-lock faces.

:-).  I had quite some trouble coming up with a name for the option.
Something like
`assume-default-face-for-nondisplayed-screens-in-scrolling' would be more
accurate, but a bit of a mouthful.  If you've any suggestions for
improvement, post them!

> Bye,
> Tassilo

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Unfreezing the display during auto-repeated scrolling. Simpler approach.

[Tassilo Horn]

Alan Mackenzie <acm@muc.de> writes:

> It's CC Mode's fontification, rather than the display engine, which is
> slow here.  But in a large window, even Emacs Lisp Mode can only just
> keep up: On my 2.6 GHz Athlon II, with a 66 line window, and the
> keyboard auto-repeating every 0.024s, fontifying and displaying a
> screen took ~0.018s.  There's not a lot of spare time for fancier
> fontification.

Indeed.

>> But still I think the variable's name is a bit misleading.  I
>> expected to see the text that scrolls by black on white, i.e., using
>> the default face similar to having `jit-lock-defer-time' set to some
>> non-nil value.  But instead the text that scrolls by and is visible
>> is fontified using the normal font-lock faces.
>
> :-).  I had quite some trouble coming up with a name for the option.
> Something like
> `assume-default-face-for-nondisplayed-screens-in-scrolling' would be
> more accurate, but a bit of a mouthful.  If you've any suggestions for
> improvement, post them!

Maybe something like `font-lock-omit-nondisplayed-screens-on-scoll'?

Bye,
Tassilo

Re: Unfreezing the display during auto-repeated scrolling. Simpler approach.

[David Kastrup]

Tassilo Horn <tsdh@gnu.org> writes:

> Alan Mackenzie <acm@muc.de> writes:
>
>> It's CC Mode's fontification, rather than the display engine, which is
>> slow here.  But in a large window, even Emacs Lisp Mode can only just
>> keep up: On my 2.6 GHz Athlon II, with a 66 line window, and the
>> keyboard auto-repeating every 0.024s, fontifying and displaying a
>> screen took ~0.018s.  There's not a lot of spare time for fancier
>> fontification.
>
> Indeed.
>
>>> But still I think the variable's name is a bit misleading.  I
>>> expected to see the text that scrolls by black on white, i.e., using
>>> the default face similar to having `jit-lock-defer-time' set to some
>>> non-nil value.  But instead the text that scrolls by and is visible
>>> is fontified using the normal font-lock faces.
>>
>> :-).  I had quite some trouble coming up with a name for the option.
>> Something like
>> `assume-default-face-for-nondisplayed-screens-in-scrolling' would be
>> more accurate, but a bit of a mouthful.  If you've any suggestions for
>> improvement, post them!
>
> Maybe something like `font-lock-omit-nondisplayed-screens-on-scoll'?

font-lock-preempt-when-scrolling

-- 
David Kastrup

Re: Unfreezing the display during auto-repeated scrolling. Simpler approach.

[Alan Mackenzie]

Good evening, David.

On Mon, Nov 24, 2014 at 12:53:55PM +0100, David Kastrup wrote:
> Tassilo Horn <tsdh@gnu.org> writes:

> > Alan Mackenzie <acm@muc.de> writes:

> >> It's CC Mode's fontification, rather than the display engine, which is
> >> slow here.  But in a large window, even Emacs Lisp Mode can only just
> >> keep up: On my 2.6 GHz Athlon II, with a 66 line window, and the
> >> keyboard auto-repeating every 0.024s, fontifying and displaying a
> >> screen took ~0.018s.  There's not a lot of spare time for fancier
> >> fontification.

> > Indeed.

> >>> But still I think the variable's name is a bit misleading.  I
> >>> expected to see the text that scrolls by black on white, i.e., using
> >>> the default face similar to having `jit-lock-defer-time' set to some
> >>> non-nil value.  But instead the text that scrolls by and is visible
> >>> is fontified using the normal font-lock faces.

> >> :-).  I had quite some trouble coming up with a name for the option.
> >> Something like
> >> `assume-default-face-for-nondisplayed-screens-in-scrolling' would be
> >> more accurate, but a bit of a mouthful.  If you've any suggestions for
> >> improvement, post them!

> > Maybe something like `font-lock-omit-nondisplayed-screens-on-scoll'?

> font-lock-preempt-when-scrolling

That's brilliant!  Though I think preempt-font-lock-when-scrolling would
be better still, since the variable isn't part of the font-lock package.

> -- 
> David Kastrup

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Unfreezing the display during auto-repeated scrolling. Simpler approach.

[Stefan Monnier]

> :-).  I had quite some trouble coming up with a name for the option.
> Something like
> `assume-default-face-for-nondisplayed-screens-in-scrolling' would be
> more accurate, but a bit of a mouthful.  If you've any suggestions for
> improvement, post them!

I think a better name would hint at something like "imprecise scrolling"
rather than at "face"s since users think of faces as something that's
purely visual and (like Tassilo) would then be surprised that the option
did not disable the faces.


        Stefan

Re: Unfreezing the display during auto-repeated scrolling. Simpler approach.

[Alan Mackenzie]

Hello, Stefan.

On Mon, Nov 24, 2014 at 09:37:34AM -0500, Stefan Monnier wrote:
> > :-).  I had quite some trouble coming up with a name for the option.
> > Something like
> > `assume-default-face-for-nondisplayed-screens-in-scrolling' would be
> > more accurate, but a bit of a mouthful.  If you've any suggestions for
> > improvement, post them!

> I think a better name would hint at something like "imprecise scrolling"
> rather than at "face"s since users think of faces as something that's
> purely visual and (like Tassilo) would then be surprised that the option
> did not disable the faces.

Pleasantly surprised, I hope.  ;-)

Maybe fast-but-imprecise-scrolling for the name?

How about David K's suggestion (slightly modified)
preempt-font-lock-when-scrolling?

>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Unfreezing the display during auto-repeated scrolling. Simpler approach.

[Stefan Monnier]

> Maybe fast-but-imprecise-scrolling for the name?

Acceptable.

> How about David K's suggestion (slightly modified)
> preempt-font-lock-when-scrolling?

While it might indeed influence font-lock's detailed working under the
hood, this is mostly hidden from the user, so mentioning font-lock is
misleading, just like mentioning "face".  Of course, it also doesn't
"preempt" anything AFAICT, so that part would also be incorrect.


        Stefan

Re: Unfreezing the display during auto-repeated scrolling.

[Eli Zaretskii]

> Date: Sun, 26 Oct 2014 22:15:30 +0000
> Cc: Eli Zaretskii <eliz@gnu.org>, emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> Maybe Eli's suggestion of rewriting scroll-up and scroll-down from
> scratch in Elisp could work, but if I'm not mistaken, there's no
> infrastructural support for window lines, etc., which doesn't involve
> fontification.  This would have to be written.

Since you don't really care about accuracy here, you could simply move
in physical lines.  That's fast and doesn't need any support from the
display engine.

> If movement on a window absolutely requires fontified characters, then
> some means will need to be found for the display code to tell jit-lock
> HOW MUCH needs fontifying, and for that amount to be much less than 500
> bytes in the non-display case.

There already is such a parameter, you just need to use it.

> One might argue that CC Mode fontification should be speeded up.  Yes, it
> should, but it's not ever going to be speeded up by an order of
> magnitude.

I don't see why not.  We could rewrite that stuff in C.

Re: Unfreezing the display during auto-repeated scrolling.

[Alan Mackenzie]

Hello, Eli.

On Mon, Oct 27, 2014 at 05:36:32AM +0200, Eli Zaretskii wrote:
> > Date: Sun, 26 Oct 2014 22:15:30 +0000
> > Cc: Eli Zaretskii <eliz@gnu.org>, emacs-devel@gnu.org
> > From: Alan Mackenzie <acm@muc.de>

> > Maybe Eli's suggestion of rewriting scroll-up and scroll-down from
> > scratch in Elisp could work, but if I'm not mistaken, there's no
> > infrastructural support for window lines, etc., which doesn't involve
> > fontification.  This would have to be written.

> Since you don't really care about accuracy here, ....

That was an uncalled for gibe.  I care a great deal about accuracy here.
Perhaps we understand different things by "accuracy".

> .... you could simply move in physical lines.  That's fast and doesn't
> need any support from the display engine.

This would end up in the wrong place if there were as much as a single
continued line in the window.  And one needs to know how many lines fit
in the window.  When a user types a single PageDown/PageUp, it MUST
scroll reliably to the right place.  That involves knowing where
continuation lines are, possibly where invisible characters are.  And one
must take account of next-screen-context-lines, and the like.

One might argue for using different functions for, on the one hand, single
scroll operations and scrolling over already fontified text, and on the
other hand auto-repeated ones over non-fontified text, where the accuracy
matters much less, but that's anything but simple, and would require
hairy stuff somewhere.

If I understand correctly, your notion of accuracy in scrolling
absolutely requires that all text scrolled over must be fully fontified.
I don't see that this makes any sense at all on ttys, and in the very
common case that all characters are equally big on a GUI.

All I need for fast scrolling is some way of telling the display code
that all characters are the same size, and that fontification is thus
necessary only for display.  Why are you so opposed to such an
optimisation?

You have asserted that my patch would lead to inaccurate scrolling.  Can
you give a concrete example of that happening?

> > If movement on a window absolutely requires fontified characters, then
> > some means will need to be found for the display code to tell jit-lock
> > HOW MUCH needs fontifying, and for that amount to be much less than 500
> > bytes in the non-display case.

I think I misunderstood things last night.  The display engine actually
requires ALL text to be fontified when moving over it, doesn't it?

> There already is such a parameter, you just need to use it.

fontification-functions is called with just one parameter, the buffer
position.  Where is this other size parameter?

> > One might argue that CC Mode fontification should be speeded up.  Yes, it
> > should, but it's not ever going to be speeded up by an order of
> > magnitude.

> I don't see why not.  We could rewrite that stuff in C.

Patches welcome.  ;-)

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Unfreezing the display during auto-repeated scrolling.

[Eli Zaretskii]

> Date: Mon, 27 Oct 2014 10:05:26 +0000
> Cc: monnier@iro.umontreal.ca, emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> > Since you don't really care about accuracy here, ....
> 
> That was an uncalled for gibe.

It wasn't supposed to be a gibe, sorry if my wording somehow implied
that.  I really understood from what you wrote that you are willing to
sacrifice display accuracy in order to have speedier PgDown/PgUp when
leaning on the key.

> I care a great deal about accuracy here.  Perhaps we understand
> different things by "accuracy".

Perhaps.  Do you at least agree that when font-lock faces change the
font, not just colors, your changes will cause scrolling to sometimes
land on a line other than what would happen without your changes?
That's what I meant by "accuracy".

> > .... you could simply move in physical lines.  That's fast and doesn't
> > need any support from the display engine.
> 
> This would end up in the wrong place if there were as much as a single
> continued line in the window.

The same will happen with your changes, in some situations.

> And one needs to know how many lines fit in the window.

That's easy: take the window pixel size and divide by the default
font's height.

> When a user types a single PageDown/PageUp, it MUST scroll reliably
> to the right place.

So we can have a variant of scroll-up/down that behaves like today on
the first invocation, and thereafter switches to moving point
instead.  Would that do what you want?

> That involves knowing where continuation lines are, possibly where
> invisible characters are.  And one must take account of
> next-screen-context-lines, and the like.

Yes, and the gazillion other scroll-* options.  Did you already try
your changes with them?  Do you always succeed to scroll as little as
possible under scroll-conservatively, for example?

> One might argue for using different functions for, on the one hand, single
> scroll operations and scrolling over already fontified text, and on the
> other hand auto-repeated ones over non-fontified text, where the accuracy
> matters much less, but that's anything but simple, and would require
> hairy stuff somewhere.

??? What hairy stuff?  We have simple and easy-to-use mechanisms to
provide commands that change their behavior when invoked repeatedly.
For example, next-line does that.

> If I understand correctly, your notion of accuracy in scrolling
> absolutely requires that all text scrolled over must be fully fontified.

The display engine must somehow know the exact same metrics of each
screen line as the line will have when actually displayed.  The only
way we know how to do that is by "simulating display", which includes
fontification, yes.

> I don't see that this makes any sense at all on ttys

I think we use the same code on TTYs for the simple reason not to
introduce yet another separate branch in the code, which will then
have separate bugs etc.

> and in the very common case that all characters are equally big on a
> GUI.

This is IMO anti-thesis of the current GUI display engine.  It was
_created_ to free us from this assumption, which was considered
outdated 15 years ago.  It makes very little sense to me to go back to
this assumption nowadays.

And if you don't assume that, the hard problem is to _know_ when this
is the case.  We have no way of knowing that, except by scanning the
buffer.

> Why are you so opposed to such an optimisation?

Because I spent the last few years making the display engine as
accurate as possible, sometimes to (IMO) ridiculous levels, all
because users complained and demanded that, not because I invented
this attitude.  I have a very bad feeling about this change, given
that experience.

> You have asserted that my patch would lead to inaccurate scrolling.  Can
> you give a concrete example of that happening?

Just define several font-lock faces to use larger or smaller fonts,
and see if you succeed to keep the promises of the various scroll-*
options, like scroll-margin, scroll-step, scroll-up/down-aggressively
etc.

> I think I misunderstood things last night.  The display engine actually
> requires ALL text to be fontified when moving over it, doesn't it?

See above: it needs the metrics.  What parts of the text it needs the
metrics of, depends on the algorithm it uses to make its decisions, in
this case, where to set window-start for the next redisplay, and also
whether to move point back into the view.  In the case of scrolling
commands, I believe that yes, it does traverse all that text.  But
that is not carved in stone; if someone comes with an alternative
algorithm that needs to examine smaller portions of text, we can
implement scrolling that is less expensive.

> > There already is such a parameter, you just need to use it.
> 
> fontification-functions is called with just one parameter, the buffer
> position.  Where is this other size parameter?

It's the value of jit-lock-chunk-size.

> > > One might argue that CC Mode fontification should be speeded up.  Yes, it
> > > should, but it's not ever going to be speeded up by an order of
> > > magnitude.
> 
> > I don't see why not.  We could rewrite that stuff in C.
> 
> Patches welcome.  ;-)

Not from me, sorry.  I don't know enough about the underlying syntax
infrastructure, and don't have enough motivation to learn that.

And you still didn't reply to my suggestion to use jit-lock-defer-time.
In my testing, setting it to 0.05 sec is enough to give you what you
want, for free.  That is just another evidence that we already have
enough knobs in our arsenal to solve these problems when they annoy
someone.  E.g., I always have jit-lock-stealth turned on, so my
buffers are always fully fontified, except in the few first moments of
a session (something that happens rather rarely).  This allows me to
avoid the problem entirely, without sacrificing accuracy.

Re: Unfreezing the display during auto-repeated scrolling.

[Alan Mackenzie]

Hello, Eli.

A lot of your post has become overtaken by other things in the last few
hours, but I'll answer anyway.

On Mon, Oct 27, 2014 at 06:48:10PM +0200, Eli Zaretskii wrote:
> > Date: Mon, 27 Oct 2014 10:05:26 +0000
> > Cc: monnier@iro.umontreal.ca, emacs-devel@gnu.org
> > From: Alan Mackenzie <acm@muc.de>

> > I care a great deal about accuracy here.  Perhaps we understand
> > different things by "accuracy".

> Perhaps.  Do you at least agree that when font-lock faces change the
> font, not just colors, your changes will cause scrolling to sometimes
> land on a line other than what would happen without your changes?
> That's what I meant by "accuracy".

Yes, I do.  However, I can't persuade myself that this is anything but a
special case.  Do people actually use mixtures of faces where the font
isn't the same as the default's?  What for?

> > > .... you could simply move in physical lines.  That's fast and doesn't
> > > need any support from the display engine.

> > This would end up in the wrong place if there were as much as a single
> > continued line in the window.

> The same will happen with your changes, in some situations.

Admitted.

> > When a user types a single PageDown/PageUp, it MUST scroll reliably
> > to the right place.

> So we can have a variant of scroll-up/down that behaves like today on
> the first invocation, and thereafter switches to moving point
> instead.  Would that do what you want?

No.  Typing a sequence of deliberate scroll-ups/downs, one at a time, is
a common and reasonable thing to do, so they should work properly.  Only
when the scroll-ups/downs start coming in faster than human typing rate
should we switch to the point moving command.

> > That involves knowing where continuation lines are, possibly where
> > invisible characters are.  And one must take account of
> > next-screen-context-lines, and the like.

> Yes, and the gazillion other scroll-* options.  Did you already try
> your changes with them?  Do you always succeed to scroll as little as
> possible under scroll-conservatively, for example?

Yes, they seem fine.  (That's with the
binding-fontification-functions-to-nil patch.)

> > If I understand correctly, your notion of accuracy in scrolling
> > absolutely requires that all text scrolled over must be fully fontified.

> The display engine must somehow know the exact same metrics of each
> screen line as the line will have when actually displayed.  The only
> way we know how to do that is by "simulating display", which includes
> fontification, yes.

> > I don't see that this makes any sense at all on ttys

> I think we use the same code on TTYs for the simple reason not to
> introduce yet another separate branch in the code, which will then
> have separate bugs etc.

> > and in the very common case that all characters are equally big on a
> > GUI.

> This is IMO anti-thesis of the current GUI display engine.  It was
> _created_ to free us from this assumption, which was considered
> outdated 15 years ago.  It makes very little sense to me to go back to
> this assumption nowadays.

Having variable size fonts is great (possibly essential) for Emacs, but
they create an overhead, and that overhead is imposed on users with no
need for the feature.  It's analogous to font lock - a programmer's
editor without font locking is almost inconceivable today (unlike 20
years ago), but for those who don't want it, we don't impose the
processing overhead on them when they disable it.

> And if you don't assume that, the hard problem is to _know_ when this
> is the case.  We have no way of knowing that, except by scanning the
> buffer.

Allowing the user to decide by setting an option, a bit like we allow her
to turn font-lock mode off.

> And you still didn't reply to my suggestion to use jit-lock-defer-time.
> In my testing, setting it to 0.05 sec is enough to give you what you
> want, for free.  That is just another evidence that we already have
> enough knobs in our arsenal to solve these problems when they annoy
> someone.

If Stefan's suggestion to modify jit-lock-defer so that it only kicks in
when the event-queue is non-empty works, you could be right.

> E.g., I always have jit-lock-stealth turned on, so my buffers are
> always fully fontified, except in the few first moments of a session
> (something that happens rather rarely).  This allows me to avoid the
> problem entirely, without sacrificing accuracy.

I have default jit-lock settings, and power up my machine sometimes
several times a day.  I think customising jit-lock is just too difficult
for most users.  (I find it difficult enough that I don't bother.)

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Unfreezing the display during auto-repeated scrolling.

[Stefan Monnier]

> Yes, I do.  However, I can't persuade myself that this is anything but a
> special case.  Do people actually use mixtures of faces where the font
> isn't the same as the default's?  What for?

FWIW, I do, because I prefer the "unholy mix of fonts" than the "angry
fruit salad".  More specifically, I like my font-lock "colors" to go
only from grey to black, and then use bold/italics/serif/proportional
features to make further distinctions.

This said, I try to keep those fonts of the same pixel height (tho
I sometimes fail, as witnessed by one of my bug reports).

But even discounting these weirdos, there are also cases such as
latex-mode's use of font-lock to make some parts of the text appear in
super/subscript, or other uses that might make section titles larger.

I'm not convinced such minor inconsistencies in "pagedown+pageup" are
a serious problem, tho.  Especially if it only happens after the user
explicitly said he was willing to sacrifice a bit of something in
exchange for more responsive scrolling.


        Stefan

Re: Unfreezing the display during auto-repeated scrolling. [ Was: Aborting display. Is this possible? ]

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Eli Zaretskii <eliz@gnu.org>,  emacs-devel@gnu.org
> Date: Sun, 26 Oct 2014 16:42:38 -0400
> 
> What I suggest is to refine jit-lock-defer such that jit-lock is
> deferred if there's input pending.

What about jit-lock-defer-time?  Setting it to a time slightly larger
than the keyboard auto-repeat interval should give Alain what he
wants.

Re: Aborting display. Is this possible?

[Stefan Monnier]

> The whole point of Jit Lock was that we would only need to fontify the
> bits of the buffer currently being displayed.  This was partly so that
> mode maintainers could be a bit more relaxed about the speed of
> font-locking.

Actually, no, the main benefit of jit-lock was so you can open a large
buffer without having to wait too long.

To reduce pressure on font-lock speed, we need something more like
jit-lock-defer-time (and lazy-lock-defer-time before it) which is disabled
by default.


        Stefan

Re: Aborting display. Is this possible?

[Eli Zaretskii]

> Date: Mon, 20 Oct 2014 21:08:19 +0000
> Cc: emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> > Also, "enters redisplay" needs to be taken with a grain of salt here.
> > The reference is to display "simulation", whereby Emacs proceeds
> > through buffer text line after line, computing the metrics of the
> > characters as it goes (and invoking fontification functions if
> > needed), but doesn't store the results in glyph matrices and doesn't
> > proceed to redrawing based on that.  So it's only partial redisplay.
> > The full redisplay will be done later, after the scrolling command did
> > its part, and Emacs is back in the command loop with no available
> > input.
> 
> Yes, I've got that.  But in CC Mode, it's the fontification which takes
> most of the time (see also below).

Then I suggest to boost the speed of that.  Optimizations should be
done in the part that takes the 80 or 90% of the time.

> > There isn't really a good solution for when input comes in faster than
> > Emacs can process it.
> 
> Er, making Emacs process the input faster?

It is already very fast, as your measurements suggest.  What slows us
down is fontifications.  That's where we should try to optimize.

> > > And each such PageDown key will necessitate full font-locking for its
> > > never-to-be-displayed screen.
> 
> > No, not full, only partial.  See above.
> 
> I don't see the "above" here.  When I tested the `fontified' property
> this morning, EVERY position from point-min to point had been fontified.

I thought I've already succeeded in explaining why.

"Partial" here means that the actual drawing to the glass is not
done.  Fontifications _are_ done, because without that, Emacs would
not know where in the buffer is that point N screenfuls away.

Re: Aborting display. Is this possible?

[Michael Welsh Duggan]

Alan Mackenzie <acm@muc.de> writes:

> On Mon, Oct 20, 2014 at 10:24:54PM +0300, Eli Zaretskii wrote:
>> > Date: Mon, 20 Oct 2014 18:57:57 +0000
>> > From: Alan Mackenzie <acm@muc.de>
>> > Cc: emacs-devel@gnu.org
>> >
>> > So, how about the following strategy: when the (new) variable
>> > `face-instead-of-fontifying' is bound to a face, AND the input queue is
>> > non-empty (or, perhaps, very non-empty), the appropriate subroutine of
>> > window-scroll should deem characters without a `face' (or `fontified' ?)
>> > property to have face `face-instead-of-fontifying'.
>
>> > This should empty the input queue pretty quickly, enabling a current
>> > buffer portion to get displayed frequently enough.
>
>> This would mean we will sometimes wind up in the wrong place after
>> scroll.  IOW, you are sacrificing correctness on behalf of some
>> (questionable for now) speedup.
>
> I'd posit that the absolute correctness isn't all that important after an
> auto-repeating PageDown.  For an individual PageDown keystroke, the
> fontification and scrolling would be done accurately as now.

Please be careful.  If you were to change things such that C-v might
scroll heuristically instead of "correctly", please make sure this
behavior is controlled by a configuration value.  When I am trying to
recreate cc-mode problems in order to make bug reports, it is very
important that I can recreate the problem exactly.  It would be very
strange if C-v run manually ended up in a different location than C-v in
a macro, for example.

-- 
Michael Welsh Duggan
(mwd@cert.org)

Re: Aborting display. Is this possible?

[Alan Mackenzie]

Hi, Michael.

On Tue, Oct 21, 2014 at 01:00:46PM -0400, Michael Welsh Duggan wrote:
> Alan Mackenzie <acm@muc.de> writes:

> > I'd posit that the absolute correctness isn't all that important after an
> > auto-repeating PageDown.  For an individual PageDown keystroke, the
> > fontification and scrolling would be done accurately as now.

> Please be careful.  If you were to change things such that C-v might
> scroll heuristically instead of "correctly", please make sure this
> behavior is controlled by a configuration value.

Don't worry.  What I'm proposing would only kick in on an auto-repeat
C-v.  The test would be whether or not another event is in the event
queue, and if so, to bypass the font-locking until display happens.  This
wouldn't affect a single C-v.

> When I am trying to recreate cc-mode problems in order to make bug
> reports, it is very important that I can recreate the problem exactly.
> It would be very strange if C-v run manually ended up in a different
> location than C-v in a macro, for example.

That is an important use case indeed.  ;-)

> -- 
> Michael Welsh Duggan
> (mwd@cert.org)

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Aborting display. Is this possible?

[Stefan Monnier]

> What is stopping a fragment of each intermediate screen (or of some of
> them) being displayed?

Redisplay is done as follows: compute a new glyph matrix (which is
basically an internal representation of what the display should look
like), and when that matrix is computed the content is drawn on screen.
I.e. drawing does not start until the glyph matrix is fully computed.

The drawing itself is relatively quick, IIUC.


        Stefan

Re: Aborting display. Is this possible?

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Eli Zaretskii <eliz@gnu.org>,  emacs-devel@gnu.org
> Date: Sun, 19 Oct 2014 21:59:09 -0400
> 
> I.e. drawing does not start until the glyph matrix is fully computed.

Where "fully computed" might mean only some portions of it are
actually computed and "enabled for redisplay", if Emacs can determine
that the rest doesn't need any update.