bug#22256: 25.0.50; multiline font-lock rules broken in C mode

[Alan Mackenzie <acm@muc.de>]

Hello, Anders.

In article <mailman.1133.1451298006.843.bug-gnu-emacs@gnu.org> you wrote:
> [-- text/plain, encoding 7bit, charset: UTF-8, 143 lines --]

> Hi!

> In C and related modes, multiline font-lock rules no longer work as
> expected. It looks like a few lines are highlighted, but not all of them.

> For example:
> Eval the following:
> (defvar my-multiline-test-keywords
>  '(("^X"
>      ("^.+$"
>       (progn (beginning-of-line)
>              (point-max))
>       nil
>       (0 'highlight)))))

> (defun my-multiline-test-add ()
>   (interactive)
>   (font-lock-add-keywords nil my-multiline-test-keywords)
>   (font-lock-flush))

> Insert the following in a new buffer:

> X START OF A HIGHLIGHTED BLOCK
> LONG LINES IN THE BLOCK LONG LINES IN THE BLOCK LONG LINES IN THE BLOCK
> LONG LINES IN THE BLOCK LONG LINES IN THE BLOCK LONG LINES IN THE BLOCK
> LONG LINES IN THE BLOCK LONG LINES IN THE BLOCK LONG LINES IN THE BLOCK
> LONG LINES IN THE BLOCK LONG LINES IN THE BLOCK LONG LINES IN THE BLOCK
> LONG LINES IN THE BLOCK LONG LINES IN THE BLOCK LONG LINES IN THE BLOCK
> LONG LINES IN THE BLOCK LONG LINES IN THE BLOCK LONG LINES IN THE BLOCK
> LONG LINES IN THE BLOCK LONG LINES IN THE BLOCK LONG LINES IN THE BLOCK
> LONG LINES IN THE BLOCK LONG LINES IN THE BLOCK LONG LINES IN THE BLOCK
> LONG LINES IN THE BLOCK LONG LINES IN THE BLOCK LONG LINES IN THE BLOCK
> LONG LINES IN THE BLOCK LONG LINES IN THE BLOCK LONG LINES IN THE BLOCK
> LONG LINES IN THE BLOCK LONG LINES IN THE BLOCK LONG LINES IN THE BLOCK
> LONG LINES IN THE BLOCK LONG LINES IN THE BLOCK LONG LINES IN THE BLOCK
> LONG LINES IN THE BLOCK LONG LINES IN THE BLOCK LONG LINES IN THE BLOCK
> LONG LINES IN THE BLOCK LONG LINES IN THE BLOCK LONG LINES IN THE BLOCK
> LONG LINES IN THE BLOCK LONG LINES IN THE BLOCK LONG LINES IN THE BLOCK
> LONG LINES IN THE BLOCK LONG LINES IN THE BLOCK LONG LINES IN THE BLOCK
> LONG LINES IN THE BLOCK LONG LINES IN THE BLOCK LONG LINES IN THE BLOCK

> Do:
> M-x c-mode RET
> M-x my-multiline-test-add RET

> I expect the entire buffer to be highlighted. However, only the first eight
> lines are highlighted. When the block is edited, different parts of the
> block is highlighted and unhighlighted.

This is caused by the jit-lock mechanism.  The first eight lines are 500
characters (jit-lock-chunk-size) rounded up to a whole number of lines.

What happens is this: when the first jit-lock-chunk (8 lines) is
fontified, the `fontified' property (the one the display engine uses) is
set only on these 8 lines.  The `face' property is then set on all the
characters of the file, as requested by the my-multiline-test-keywords
form.

Next thing, jit-lock fontifies the next chunk of ~7 lines starting where
the `fontified' property is nil.  The first thing done is to set
`fontified' on these ~7 lines, then the `face' property on them is
erased.  There is now no matching font lock pattern to apply any new
faces to these ~7 lines, since the "X" is many lines back.  The same
thing happens with the next 500 byte chunk, and so on till the end of the
buffer.

If you set `font-lock-support-mode' to nil and restart font locking, the
problem isn't apparent.  (Then set the variable back to 'jit-lock-mode.)

This is a fundamental problem with jit-lock-mode: the assumption that
text to be fontified has no non-trivial context.  This is a difficult
problem to solve in general.  CC Mode uses some ad-hoc tricks to catch,
for example, long struct declarations.

> As a contrast, when `emacs-lisp-mode' is used, the entire block is
> highlighted, and editing does not change the highlighting.

I do not see this in Emacs 24.5.  For me, even in emacs-lisp-mode, I
still see just the 8 lines being fontified.  If you could give me a
recipe (starting from emacs-24.5 -Q) to reproduce this, I'd be very
interested.

> This worked as intended in Emacs 24.5.

>     -- Anders Lindgren

> In GNU Emacs 25.0.50.60 (x86_64-apple-darwin14.5.0, NS appkit-1348.17
> Version 10.10.5 (Build 14F27))
>  of 2015-12-28
> Repository revision: e9916d8880561cc06b6cb73bafe7257b93ffbf4c
> Windowing system distributor 'Apple', version 10.3.1348
> Configured using:
>  'configure --without-dbus'

> Configured features:
> ACL ZLIB TOOLKIT_SCROLL_BARS NS

> Important settings:
>   value of $LC_CTYPE: UTF-8
>   locale-coding-system: utf-8-unix

> Major mode: C/l

> Minor modes in effect:
>   preproc-font-lock-global-mode: t
>   preproc-font-lock-mode: t
>   tooltip-mode: t
>   global-eldoc-mode: t
>   electric-indent-mode: t
>   mouse-wheel-mode: t
>   tool-bar-mode: t
>   menu-bar-mode: t
>   file-name-shadow-mode: t
>   global-font-lock-mode: t
>   font-lock-mode: t
>   blink-cursor-mode: t
>   auto-composition-mode: t
>   auto-encryption-mode: t
>   auto-compression-mode: t
>   line-number-mode: t
>   transient-mark-mode: t
>   abbrev-mode: t

[ .... ]

-- 
Alan Mackenzie (Nuremberg, Germany).