bug#38049: C mode fontification broken with reposition-window

bug#38049: C mode fontification broken with reposition-window

[Juri Linkov]

Version: 27.0.50

This is a reproducible test case:

0. emacs -Q
1. C-x C-f emacs/lib-src/emacsclient.c
2. M-: (progn (search-forward "create-frame" nil t) (reposition-window))

Then half screen displays unfontified lines.

Fontification doesn't fail in other modes, only in C mode.

This has something to do with interaction between c-font-lock
and buffer navigation in reposition-window.

bug#38049: C mode fontification broken with reposition-window

[Alan Mackenzie]

In article <mailman.552.1572813126.13325.bug-gnu-emacs@gnu.org> you wrote:
> Version: 27.0.50

> This is a reproducible test case:

> 0. emacs -Q
> 1. C-x C-f emacs/lib-src/emacsclient.c
> 2. M-: (progn (search-forward "create-frame" nil t) (reposition-window))

> Then half screen displays unfontified lines.

Yes.  This is an interesting bug.

The unfontified area seems to be one jit-lock chunk immediately below
the line containing "create-frame".

> Fontification doesn't fail in other modes, only in C mode.

> This has something to do with interaction between c-font-lock
> and buffer navigation in reposition-window.

reposition-window, as far as I've been able to make out so far, seems to
assume that point starts in the current visible window.  That is not the
case here, where window-start is still at BOB, but point is at line 116.

I'll look at this in more detail when it's not so late.

-- 
Alan Mackenzie (Nuremberg, Germany).

bug#38049: C mode fontification broken with reposition-window

[Alan Mackenzie]

Hello, Juri.

On Sun, Nov 03, 2019 at 22:28:05 +0200, Juri Linkov wrote:
> Version: 27.0.50

> This is a reproducible test case:

> 0. emacs -Q
> 1. C-x C-f emacs/lib-src/emacsclient.c
> 2. M-: (progn (search-forward "create-frame" nil t) (reposition-window))

> Then half screen displays unfontified lines.

> Fontification doesn't fail in other modes, only in C mode.

> This has something to do with interaction between c-font-lock
> and buffer navigation in reposition-window.

Indeed it does.

Briefly,
(i) reposition-window narrows to (2758 3940) in
repos-count-screen-lines.
(ii) This latter function uses vertical-motion to count the lines.
(iii) vertical-motion triggers jit-lock fontification.
(iv) This calls (eventually) c-font-lock-fontify-region.
(v) c-font-lock-fontify-region attempts to examine buffer text before
  the start of the jit-lock chunk to find syntactic context.
(vi) This is outside the visible region, so Emacs raises an exception.
(vii) The exception is caught and discarded by an unwind-protect in
  c-font-lock-fontify-region.
(viii) The jit-lock chunk remains unfontified.

As Stefan M has sometimes remarked, narrowing is often not a good idea.

It would seem undesirable for the vertical-motion in (iii) to trigger
font-locking, since it is merely trying to count lines.  Perhaps there
should be a macro `without-fontifying' which could be wrapped around
this call to vertical-motion, if there isn't such a thing anyway.

Maybe there are other calls of vertical-motion which are similarly
dangerous.

-- 
Alan Mackenzie (Nuremberg, Germany).

bug#38049: C mode fontification broken with reposition-window

[Juri Linkov]

> Briefly,
> (i) reposition-window narrows to (2758 3940) in
> repos-count-screen-lines.
> (ii) This latter function uses vertical-motion to count the lines.
> (iii) vertical-motion triggers jit-lock fontification.
> (iv) This calls (eventually) c-font-lock-fontify-region.
> (v) c-font-lock-fontify-region attempts to examine buffer text before
>   the start of the jit-lock chunk to find syntactic context.
> (vi) This is outside the visible region, so Emacs raises an exception.
> (vii) The exception is caught and discarded by an unwind-protect in
>   c-font-lock-fontify-region.
> (viii) The jit-lock chunk remains unfontified.
>
> As Stefan M has sometimes remarked, narrowing is often not a good idea.
>
> It would seem undesirable for the vertical-motion in (iii) to trigger
> font-locking, since it is merely trying to count lines.  Perhaps there
> should be a macro `without-fontifying' which could be wrapped around
> this call to vertical-motion, if there isn't such a thing anyway.

ISTR there was some macro to disable font-lock temporarily,
but I can't find it.  Maybe just let-binding font-lock-mode to nil
could help?

bug#38049: C mode fontification broken with reposition-window

[Eli Zaretskii]

> Date: Tue, 5 Nov 2019 21:07:37 +0000
> From: Alan Mackenzie <acm@muc.de>
> Cc: 38049@debbugs.gnu.org
> 
> (i) reposition-window narrows to (2758 3940) in
> repos-count-screen-lines.
> (ii) This latter function uses vertical-motion to count the lines.
> (iii) vertical-motion triggers jit-lock fontification.
> (iv) This calls (eventually) c-font-lock-fontify-region.
> (v) c-font-lock-fontify-region attempts to examine buffer text before
>   the start of the jit-lock chunk to find syntactic context.
> (vi) This is outside the visible region, so Emacs raises an exception.
> (vii) The exception is caught and discarded by an unwind-protect in
>   c-font-lock-fontify-region.
> (viii) The jit-lock chunk remains unfontified.
> 
> As Stefan M has sometimes remarked, narrowing is often not a good idea.

Narrowing is part of life.

Why can't c-font-lock-fontify-region be fixed so as not to signal an
error in these cases?  If we do, will that fix the original problem?

> It would seem undesirable for the vertical-motion in (iii) to trigger
> font-locking, since it is merely trying to count lines.

vertical-motion calls functions from the display engine, which
fontifies the text it traverses.  It doesn't only count lines, it also
counts columns in canonical character width units (i.e. actually in
pixels).  So it cannot possibly avoid fontifications because
fontification might affect how many pixels characters take on
display.  Fontifications can also affect the line count, if some
fontification code actually puts display properties on the fontified
text -- something that you rarely if ever see, but Lisp programs can
rightfully expect that.  So this idea:

> Perhaps there should be a macro `without-fontifying' which could be
> wrapped around this call to vertical-motion, if there isn't such a
> thing anyway.

cannot fly, sorry.

Let's try to fix the problem in c-font-lock-fontify-region instead,
okay?

bug#38049: C mode fontification broken with reposition-window

[Alan Mackenzie]

Hello, Eli.

On Wed, Nov 06, 2019 at 19:13:44 +0200, Eli Zaretskii wrote:
> > Date: Tue, 5 Nov 2019 21:07:37 +0000
> > From: Alan Mackenzie <acm@muc.de>
> > Cc: 38049@debbugs.gnu.org

> > (i) reposition-window narrows to (2758 3940) in
> > repos-count-screen-lines.
> > (ii) This latter function uses vertical-motion to count the lines.
> > (iii) vertical-motion triggers jit-lock fontification.
> > (iv) This calls (eventually) c-font-lock-fontify-region.
> > (v) c-font-lock-fontify-region attempts to examine buffer text before
> >   the start of the jit-lock chunk to find syntactic context.
> > (vi) This is outside the visible region, so Emacs raises an exception.
> > (vii) The exception is caught and discarded by an unwind-protect in
> >   c-font-lock-fontify-region.
> > (viii) The jit-lock chunk remains unfontified.

> > As Stefan M has sometimes remarked, narrowing is often not a good idea.

> Narrowing is part of life.

Indeed.

> Why can't c-font-lock-fontify-region be fixed so as not to signal an
> error in these cases?  If we do, will that fix the original problem?

Yes.  Thinking about it, font locking might present
c-font-lock-fontify-region a chunk in any state of narrowing.  It just
doesn't happen that way very often, though.  So CC Mode needs to widen
the buffer to make sure it has adequate access.

> > It would seem undesirable for the vertical-motion in (iii) to trigger
> > font-locking, since it is merely trying to count lines.

> vertical-motion calls functions from the display engine, which
> fontifies the text it traverses.  It doesn't only count lines, it also
> counts columns in canonical character width units (i.e. actually in
> pixels).  So it cannot possibly avoid fontifications because
> fontification might affect how many pixels characters take on
> display.  Fontifications can also affect the line count, if some
> fontification code actually puts display properties on the fontified
> text -- something that you rarely if ever see, but Lisp programs can
> rightfully expect that.  So this idea:

> > Perhaps there should be a macro `without-fontifying' which could be
> > wrapped around this call to vertical-motion, if there isn't such a
> > thing anyway.

> cannot fly, sorry.

Yes, accepted and understood.

> Let's try to fix the problem in c-font-lock-fontify-region instead,
> okay?

I'll do that.  It just needs a simple widen in a save-restriction.

-- 
Alan Mackenzie (Nuremberg, Germany).

bug#38049: C mode fontification broken with reposition-window

[Alan Mackenzie]

Hello, Juri.

On Tue, Nov 05, 2019 at 21:07:37 +0000, Alan Mackenzie wrote:
> On Sun, Nov 03, 2019 at 22:28:05 +0200, Juri Linkov wrote:
> > Version: 27.0.50

> > This is a reproducible test case:

> > 0. emacs -Q
> > 1. C-x C-f emacs/lib-src/emacsclient.c
> > 2. M-: (progn (search-forward "create-frame" nil t) (reposition-window))

> > Then half screen displays unfontified lines.

> > Fontification doesn't fail in other modes, only in C mode.

> > This has something to do with interaction between c-font-lock
> > and buffer navigation in reposition-window.

> Indeed it does.

> Briefly,
> (i) reposition-window narrows to (2758 3940) in
> repos-count-screen-lines.
> (ii) This latter function uses vertical-motion to count the lines.
> (iii) vertical-motion triggers jit-lock fontification.
> (iv) This calls (eventually) c-font-lock-fontify-region.
> (v) c-font-lock-fontify-region attempts to examine buffer text before
>   the start of the jit-lock chunk to find syntactic context.
> (vi) This is outside the visible region, so Emacs raises an exception.
> (vii) The exception is caught and discarded by an unwind-protect in
>   c-font-lock-fontify-region.
> (viii) The jit-lock chunk remains unfontified.

[ .... ]

I have fixed this bug by widening around c-font-lock-fontify-region.  I
am closing the bug with this post.

-- 
Alan Mackenzie (Nuremberg, Germany).