bug#20180: Missing documentation about redisplay.

bug#20180: Missing documentation about redisplay.

[Alan Mackenzie]

Hello, Emacs.

The elisp manual doesn't contain adequate documentation about
(re)display.

The topics not covered include
1. What triggers redisplay.
2. What portions of the frame/all frames get redisplayed when a
  redisplay occurs, and how does this relate to the answer to 1..

There is a partial answer to 1. on the page "Forcing Redisplay" which
states that "Emacs normally tries to redisplay the screen whenever it
waits for input.", but this is clearly incomplete - redisplay also
happens in the absence of input (e.g. by context fontification).

One clearly needs an answer to 2. if one ever wants to cause the
redisplay of a particular part of a window or frame.

-- 
Alan Mackenzie (Nuremberg, Germany).

bug#20180: Missing documentation about redisplay.

[Eli Zaretskii]

> Date: Mon, 23 Mar 2015 16:08:50 +0000
> From: Alan Mackenzie <acm@muc.de>
> 
> The elisp manual doesn't contain adequate documentation about
> (re)display.
> 
> The topics not covered include

They include much more.  Describing the display engine to any depth is
a research project, not a bug report.  Maybe if you had more specific
questions, they could be answered.

The closest thing to what you are asking for is in the large
commentary at the beginning of xdisp.c.

> 1. What triggers redisplay.

See below: this is already covered.

> 2. What portions of the frame/all frames get redisplayed when a
>   redisplay occurs, and how does this relate to the answer to 1..
> 
> There is a partial answer to 1. on the page "Forcing Redisplay" which
> states that "Emacs normally tries to redisplay the screen whenever it
> waits for input.", but this is clearly incomplete - redisplay also
> happens in the absence of input (e.g. by context fontification).

I don't udnerstand what you are saying here: "waiting for input" and
"there's no input" is not a contradiction, on the contrary: yo uwait
for input when there's none.  And what is "context fontification"?

> One clearly needs an answer to 2. if one ever wants to cause the
> redisplay of a particular part of a window or frame.

To do that, you need to change the text of that part or the text
properties/overlays that affect how that part looks on display.  But
you most probably already know that, so I'm again not sure what the
question is.

bug#20180: Missing documentation about redisplay.

[Alan Mackenzie]

Hi, Eli.

On Mon, Mar 23, 2015 at 06:49:08PM +0200, Eli Zaretskii wrote:
> > Date: Mon, 23 Mar 2015 16:08:50 +0000
> > From: Alan Mackenzie <acm@muc.de>

> > The elisp manual doesn't contain adequate documentation about
> > (re)display.

> > The topics not covered include

> They include much more.  Describing the display engine to any depth is
> a research project, not a bug report.

:-)

> Maybe if you had more specific questions, they could be answered.

What triggered my documentation search was looking at
jit-lock-fontify-now.  There, if jit-lock suspects that properties at an
earlier location in the buffer have been changed, it arranges that after
the end of the current display cycle, some text properties at some of
these locations get set, thus triggering another redisplay.

How?  Why does this trigger a redisplay when the setting of the
properties during the previous redisplay cycle didn't?  In the new
redisplay, does the whole frame/window/minibuffer get redrawn (for
whatever value of "redraw") or just the bits where text properties were
set?

If I were to execute the command `ignore' via a key sequence, would that
trigger redisplay?  If so, how much would get redrawn?

> The closest thing to what you are asking for is in the large
> commentary at the beginning of xdisp.c.

> > 1. What triggers redisplay.

> See below: this is already covered.

> > 2. What portions of the frame/all frames get redisplayed when a
> >   redisplay occurs, and how does this relate to the answer to 1..

> > There is a partial answer to 1. on the page "Forcing Redisplay" which
> > states that "Emacs normally tries to redisplay the screen whenever it
> > waits for input.", but this is clearly incomplete - redisplay also
> > happens in the absence of input (e.g. by context fontification).

> I don't understand what you are saying here: "waiting for input" and
> "there's no input" is not a contradiction, on the contrary: you wait
> for input when there's none.  And what is "context fontification"?

I read the "whenever" as meaning "when it's waiting for input, it tries
_once_ to redisplay".  Having tried once, and 0.5s
(jit-lock-context-time) have passed, then jit-lock-context-fontify kicks
in to fontify screen lines below where a buffer change happened.  Another
redisplay then happens.  What triggers this second redisplay, given there
hasn't been any more "input"?  Or does the expiry of the timer count as
"input" here?  Or is something other than input (?a buffer change)
triggering this second redisplay?

> > One clearly needs an answer to 2. if one ever wants to cause the
> > redisplay of a particular part of a window or frame.

> To do that, you need to change the text of that part or the text
> properties/overlays that affect how that part looks on display.  But
> you most probably already know that, so I'm again not sure what the
> question is.

In jit-lock-fontify-now, the top half of a screen window has just been
redisplayed, and j-l-fontify-now is busily applying faces for the next
500-byte chunk.  In so doing, it sets faces in the part of the window
that have already been redisplayed.  This is apparently insufficient in
itself to trigger another redisplay of those already displayed window
parts.  I would like to understand why.

-- 
Alan Mackenzie (Nuremberg, Germany).

bug#20180: Missing documentation about redisplay.

[Eli Zaretskii]

> Date: Mon, 23 Mar 2015 17:55:25 +0000
> Cc: 20180@debbugs.gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> What triggered my documentation search was looking at
> jit-lock-fontify-now.

Note that digging into JIT Lock means going deep into the internals of
the display engine.  It's no accident that jit-lock.el was written by
Gerd Moellmann, who designed and implemented the current display
engine.

IOW, it's by no means ELisp manual level stuff (although I of course
agree that IWBN to have the display engine's internals described in
full, along with all the other internals).

> There, if jit-lock suspects that properties at an
> earlier location in the buffer have been changed, it arranges that after
> the end of the current display cycle, some text properties at some of
> these locations get set, thus triggering another redisplay.
> 
> How?

By changing the special text property 'fontified'.

> Why does this trigger a redisplay when the setting of the
> properties during the previous redisplay cycle didn't?

I don't understand the question.  What "setting of properties" during
which "previous redisplay cycle" do you have in mind?

> In the new redisplay, does the whole frame/window/minibuffer get
> redrawn (for whatever value of "redraw") or just the bits where text
> properties were set?

This depends on what you mean by "redraw".  A redisplay cycle has 2
phases.  In the first phase, Emacs examines the visible portion of the
buffer and tries to determine the minimal number of screen lines that
might need to be redrawn; it then prepares the corresponding portions
of the glyph matrix for those screen lines.  Normally, only the screen
lines where the 'fontified' text property changed will be considered.

In the second phase of redisplay, the prepared glyph matrix is
compared to the previous one, which describes what's on the glass, and
only the different portions are actually redrawn.  It could be that
nothing needs to be redrawn at all.

> If I were to execute the command `ignore' via a key sequence, would that
> trigger redisplay?

Yes, redisplay is triggered each time Emacs waits for more input and
has nothing else to do.

> If so, how much would get redrawn?

Nothing at all, since nothing changed.

> I read the "whenever" as meaning "when it's waiting for input, it tries
> _once_ to redisplay".  Having tried once, and 0.5s
> (jit-lock-context-time) have passed, then jit-lock-context-fontify kicks
> in to fontify screen lines below where a buffer change happened.  Another
> redisplay then happens.  What triggers this second redisplay, given there
> hasn't been any more "input"?

The JIT Lock timer.

> Or does the expiry of the timer count as "input" here?  Or is
> something other than input (?a buffer change) triggering this second
> redisplay?

Yes, timers have the same effect as input: they return to the command
loop after the timer function is run, and the command loop enters
redisplay if there's no input.

> > > One clearly needs an answer to 2. if one ever wants to cause the
> > > redisplay of a particular part of a window or frame.
> 
> > To do that, you need to change the text of that part or the text
> > properties/overlays that affect how that part looks on display.  But
> > you most probably already know that, so I'm again not sure what the
> > question is.
> 
> In jit-lock-fontify-now, the top half of a screen window has just been
> redisplayed, and j-l-fontify-now is busily applying faces for the next
> 500-byte chunk.  In so doing, it sets faces in the part of the window
> that have already been redisplayed.  This is apparently insufficient in
> itself to trigger another redisplay of those already displayed window
> parts.  I would like to understand why.

jit-lock-fontify-now can be called in 2 different ways.  One is as
part of redisplay itself, because jit-lock-function, which calls
jit-lock-fontify-now, is added to fontification-functions, which are
run by the display engine when it bumps into a buffer position that
has the 'fontified' text property whose value is nil.  When
jit-lock-fontify-now is called by this mechanism, it fontifies
portions that were not yet displayed.  After fontification-functions
return, the display engine re-examines the buffer position where they
were called, since it knows that faces could change there.  This
constitutes "another redisplay" you were looking for, without actually
triggering a full redisplay cycle.  Keep in mind that the display
engine examines buffer text one character at a time, so it is only
ever interested in the faces at the single buffer position it is
examining.

The other way jit-lock-fontify-now is called is from the Jit Stealth
or deferred Jit Lock, in which case it does trigger another redisplay,
as you expect.

HTH

bug#20180: Missing documentation about redisplay.

[Alan Mackenzie]

On Mon, Mar 23, 2015 at 08:29:03PM +0200, Eli Zaretskii wrote:
> > Date: Mon, 23 Mar 2015 17:55:25 +0000
> > Cc: 20180@debbugs.gnu.org
> > From: Alan Mackenzie <acm@muc.de>

> > What triggered my documentation search was looking at
> > jit-lock-fontify-now.

> Note that digging into JIT Lock means going deep into the internals of
> the display engine.  It's no accident that jit-lock.el was written by
> Gerd Moellmann, who designed and implemented the current display
> engine.

OK.  I thought it was Simon Marshall who wrote jit-lock.el.  Obviously
I'm mistaken.

> IOW, it's by no means ELisp manual level stuff (although I of course
> agree that IWBN to have the display engine's internals described in
> full, along with all the other internals).

I think some of what we've been discussing would be useful to put in a
new page in the elisp manual; it wouldn't have to be a very big page.

> > There, if jit-lock suspects that properties at an
> > earlier location in the buffer have been changed, it arranges that after
> > the end of the current display cycle, some text properties at some of
> > these locations get set, thus triggering another redisplay.

> > How?

> By changing the special text property 'fontified'.

> > Why does this trigger a redisplay when the setting of the
> > properties during the previous redisplay cycle didn't?

> I don't understand the question.  What "setting of properties" during
> which "previous redisplay cycle" do you have in mind?

1. Redisplay cycle 1 starts.
2. First 500-byte chunk gets fontified through fontification-functions.
3. Second 500-byte chunk fontification starts.
4.   jit-lock-fontify-now applies face properties to a few characters of
     the first 500-byte chunk at locations 496, 497, 498, 499.
5.   jit-lock-fontify-now arranges for 10. to happen at the expiry of a 0
     second timeout.  (`run-with-timer')
6. Second 500-byte chunk is completely fontified.  It gets drawn on the
   screen.
7. Redisplay cycle 1 is now complete.

10. (At expiry of 0 second timeout), jit-lock-force-redisplay applies
   text property (fontified t) to location 496, 497, 498, 499.  496..499
   already had the property (fontified t).
11. Redisplay cycle 2 starts, having been triggered by 10.
12. ???? Redisplay draws 496, .., 499 on the screen, but no others.
13. Redisplay cycle 2 is now complete.

I think I meant to ask: why does setting text property 'fontified to t at
stage 10 trigger redisplay cycle 2, whereas setting them at stage 4.5
wouldn't have done?

> > In the new redisplay, does the whole frame/window/minibuffer get
> > redrawn (for whatever value of "redraw") or just the bits where text
> > properties were set?

> This depends on what you mean by "redraw".  A redisplay cycle has 2
> phases.  In the first phase, Emacs examines the visible portion of the
> buffer and tries to determine the minimal number of screen lines that
> might need to be redrawn; it then prepares the corresponding portions
> of the glyph matrix for those screen lines.  Normally, only the screen
> lines where the 'fontified' text property changed will be considered.

In 10. above, the `fontified' text property is set to the value it
already had.  This seems to be sufficient to trigger a redisplay.

> In the second phase of redisplay, the prepared glyph matrix is
> compared to the previous one, which describes what's on the glass, and
> only the different portions are actually redrawn.  It could be that
> nothing needs to be redrawn at all.

OK.

> > If I were to execute the command `ignore' via a key sequence, would that
> > trigger redisplay?

> Yes, redisplay is triggered each time Emacs waits for more input and
> has nothing else to do.

> > If so, how much would get redrawn?

> Nothing at all, since nothing changed.

Thanks, that's good to understand.

> > I read the "whenever" as meaning "when it's waiting for input, it tries
> > _once_ to redisplay".  Having tried once, and 0.5s
> > (jit-lock-context-time) have passed, then jit-lock-context-fontify kicks
> > in to fontify screen lines below where a buffer change happened.  Another
> > redisplay then happens.  What triggers this second redisplay, given there
> > hasn't been any more "input"?

> The JIT Lock timer.

OK.

> > Or does the expiry of the timer count as "input" here?  Or is
> > something other than input (?a buffer change) triggering this second
> > redisplay?

> Yes, timers have the same effect as input: they return to the command
> loop after the timer function is run, and the command loop enters
> redisplay if there's no input.

> > > > One clearly needs an answer to 2. if one ever wants to cause the
> > > > redisplay of a particular part of a window or frame.

> > > To do that, you need to change the text of that part or the text
> > > properties/overlays that affect how that part looks on display.  But
> > > you most probably already know that, so I'm again not sure what the
> > > question is.

To summarize: to cause a redisplay which changes what you see (without
any scrolling happening) it is necessary to (i) change some text or
"significant" property/overlay on a displayed portion of a buffer; (ii)
cause an input event of any kind.

What I'm now thinking is that jit-lock-trigger-redisplay, and its setting
of text properties to the value they already have, is a complete red
herring[*] - the same effect could have been achieved by putting `ignore'
on the run-with-timer at stage 5, it merely being the expiry of the timer
triggering the redisplay cycle 2.

[*] "Red herring": anything which diverts attention from a topic or line
of enquiry.

> > In jit-lock-fontify-now, the top half of a screen window has just been
> > redisplayed, and j-l-fontify-now is busily applying faces for the next
> > 500-byte chunk.  In so doing, it sets faces in the part of the window
> > that have already been redisplayed.  This is apparently insufficient in
> > itself to trigger another redisplay of those already displayed window
> > parts.  I would like to understand why.

> jit-lock-fontify-now can be called in 2 different ways.  One is as
> part of redisplay itself, because jit-lock-function, which calls
> jit-lock-fontify-now, is added to fontification-functions, which are
> run by the display engine when it bumps into a buffer position that
> has the 'fontified' text property whose value is nil.  When
> jit-lock-fontify-now is called by this mechanism, it fontifies
> portions that were not yet displayed.  After fontification-functions
> return, the display engine re-examines the buffer position where they
> were called, since it knows that faces could change there.  This
> constitutes "another redisplay" you were looking for, without actually
> triggering a full redisplay cycle.  Keep in mind that the display
> engine examines buffer text one character at a time, so it is only
> ever interested in the faces at the single buffer position it is
> examining.

> The other way jit-lock-fontify-now is called is from the Jit Stealth
> or deferred Jit Lock, in which case it does trigger another redisplay,
> as you expect.

> HTH

Enormously!  Thanks very much.

-- 
Alan Mackenzie (Nuremberg, Germany).

bug#20180: Missing documentation about redisplay.

[Eli Zaretskii]

> Date: Mon, 23 Mar 2015 20:17:52 +0000
> Cc: 20180@debbugs.gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> > > There, if jit-lock suspects that properties at an
> > > earlier location in the buffer have been changed, it arranges that after
> > > the end of the current display cycle, some text properties at some of
> > > these locations get set, thus triggering another redisplay.
> 
> > > How?
> 
> > By changing the special text property 'fontified'.
> 
> > > Why does this trigger a redisplay when the setting of the
> > > properties during the previous redisplay cycle didn't?
> 
> > I don't understand the question.  What "setting of properties" during
> > which "previous redisplay cycle" do you have in mind?
> 
> 1. Redisplay cycle 1 starts.
> 2. First 500-byte chunk gets fontified through fontification-functions.
> 3. Second 500-byte chunk fontification starts.
> 4.   jit-lock-fontify-now applies face properties to a few characters of
>      the first 500-byte chunk at locations 496, 497, 498, 499.
> 5.   jit-lock-fontify-now arranges for 10. to happen at the expiry of a 0
>      second timeout.  (`run-with-timer')
> 6. Second 500-byte chunk is completely fontified.  It gets drawn on the
>    screen.
> 7. Redisplay cycle 1 is now complete.
> 
> 10. (At expiry of 0 second timeout), jit-lock-force-redisplay applies
>    text property (fontified t) to location 496, 497, 498, 499.  496..499
>    already had the property (fontified t).
> 11. Redisplay cycle 2 starts, having been triggered by 10.
> 12. ???? Redisplay draws 496, .., 499 on the screen, but no others.
> 13. Redisplay cycle 2 is now complete.
> 
> I think I meant to ask: why does setting text property 'fontified to t at
> stage 10 trigger redisplay cycle 2, whereas setting them at stage 4.5
> wouldn't have done?

Because stage 10 happens _after_ redisplay cycle 1 is complete.
Changing text properties after redisplay triggers another redisplay of
the buffer portion where the properties changed, because changes in
properties bump up the buffer-changed counter, and redisplay examines
that counter to determine whether anything changed since the previous
redisplay cycle.

IOW, if those properties weren't changed after redisplay finished, the
next redisplay would have decided that no changes happened, and do
nothing.

> In 10. above, the `fontified' text property is set to the value it
> already had.  This seems to be sufficient to trigger a redisplay.

Because it counts as a change regardless.

bug#20180: Missing documentation about redisplay.

[Lars Ingebrigtsen]

Reading this thread, I think the conclusion here is that the redisplay
stuff is just too low-level and finicky to be meaningfully documented in
the lispref manual.  You really have to read xdisp.c to start to
understand how that works.

And I think that's fine -- not everything can be put into the lispref
manual.  So I'm closing this bug report.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#20180: Missing documentation about redisplay.

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Reading this thread, I think the conclusion here is that the redisplay
  > stuff is just too low-level and finicky to be meaningfully documented in
  > the lispref manual.  You really have to read xdisp.c to start to
  > understand how that works.

I think that's right.  And there's no need to document this there.
The Lisp Ref manual is annoyingly big already.

Maybe we should remove all the internal C-level documentation
from there and put it somewhere else.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)