bug#22256: 25.0.50; multiline font-lock rules broken in C mode

[Anders Lindgren <andlind@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 6315 bytes --]

Hi!

I just realized that I missed one vital thing in the example I posted.
`font-lock-multiline' must be set. Also, I add code so that it works when
`font-lock-flush' isn't defined.

(defvar my-multiline-test-keywords
 '(("^X"
     ("^.+$"
      (progn (beginning-of-line)
             (point-max))
      nil
      (0 'highlight)))))

(defun my-multiline-test-add ()
  (interactive)
  (font-lock-add-keywords nil my-multiline-test-keywords)
  (setq font-lock-multiline t)
  (if (fboundp 'font-lock-flush)
      (font-lock-flush)
    (when font-lock-mode
      (with-no-warnings
        (font-lock-fontify-buffer)))))

Recipe:

emacs -Q
Eval the code above
C-x b test RET
Insert the text starting with an X (from the first mail)
M-x c-mode RET
M-x my-multiline-test-add RET

Alternatively, use emacs-lisp-mode instead of C mode.

When tested on Emacs 24.5 and Emacs 25, the only combination that doesn't
work is Emacs 25 + C mode.

There are a number of font-lock packages that utilize that this work,
including my https://github.com/Lindydancer/preproc-font-lock package.

    -- Anders

On Tue, Dec 29, 2015 at 10:06 PM, Alan Mackenzie <acm@muc.de> wrote:
>
> Hello, Anders.
>
> In article <mailman.1133.1451298006.843.bug-gnu-emacs@gnu.org> you wrote:
> > [-- text/plain, encoding 7bit, charset: UTF-8, 143 lines --]
>
> > Hi!
>
> > In C and related modes, multiline font-lock rules no longer work as
> > expected. It looks like a few lines are highlighted, but not all of
them.
>
> > For example:
> > Eval the following:
> > (defvar my-multiline-test-keywords
> >  '(("^X"
> >      ("^.+quot;
> >       (progn (beginning-of-line)
> >              (point-max))
> >       nil
> >       (0 'highlight)))))
>
> > (defun my-multiline-test-add ()
> >   (interactive)
> >   (font-lock-add-keywords nil my-multiline-test-keywords)
> >   (font-lock-flush))
>
> > Insert the following in a new buffer:
>
> > X START OF A HIGHLIGHTED BLOCK
> > LONG LINES IN THE BLOCK LONG LINES IN THE BLOCK LONG LINES IN THE BLOCK
> > LONG LINES IN THE BLOCK LONG LINES IN THE BLOCK LONG LINES IN THE BLOCK
> > LONG LINES IN THE BLOCK LONG LINES IN THE BLOCK LONG LINES IN THE BLOCK
> > LONG LINES IN THE BLOCK LONG LINES IN THE BLOCK LONG LINES IN THE BLOCK
> > LONG LINES IN THE BLOCK LONG LINES IN THE BLOCK LONG LINES IN THE BLOCK
> > LONG LINES IN THE BLOCK LONG LINES IN THE BLOCK LONG LINES IN THE BLOCK
> > LONG LINES IN THE BLOCK LONG LINES IN THE BLOCK LONG LINES IN THE BLOCK
> > LONG LINES IN THE BLOCK LONG LINES IN THE BLOCK LONG LINES IN THE BLOCK
> > LONG LINES IN THE BLOCK LONG LINES IN THE BLOCK LONG LINES IN THE BLOCK
> > LONG LINES IN THE BLOCK LONG LINES IN THE BLOCK LONG LINES IN THE BLOCK
> > LONG LINES IN THE BLOCK LONG LINES IN THE BLOCK LONG LINES IN THE BLOCK
> > LONG LINES IN THE BLOCK LONG LINES IN THE BLOCK LONG LINES IN THE BLOCK
> > LONG LINES IN THE BLOCK LONG LINES IN THE BLOCK LONG LINES IN THE BLOCK
> > LONG LINES IN THE BLOCK LONG LINES IN THE BLOCK LONG LINES IN THE BLOCK
> > LONG LINES IN THE BLOCK LONG LINES IN THE BLOCK LONG LINES IN THE BLOCK
> > LONG LINES IN THE BLOCK LONG LINES IN THE BLOCK LONG LINES IN THE BLOCK
> > LONG LINES IN THE BLOCK LONG LINES IN THE BLOCK LONG LINES IN THE BLOCK
>
> > Do:
> > M-x c-mode RET
> > M-x my-multiline-test-add RET
>
> > I expect the entire buffer to be highlighted. However, only the first
eight
> > lines are highlighted. When the block is edited, different parts of the
> > block is highlighted and unhighlighted.
>
> This is caused by the jit-lock mechanism.  The first eight lines are 500
> characters (jit-lock-chunk-size) rounded up to a whole number of lines.
>
> What happens is this: when the first jit-lock-chunk (8 lines) is
> fontified, the `fontified' property (the one the display engine uses) is
> set only on these 8 lines.  The `face' property is then set on all the
> characters of the file, as requested by the my-multiline-test-keywords
> form.
>
> Next thing, jit-lock fontifies the next chunk of ~7 lines starting where
> the `fontified' property is nil.  The first thing done is to set
> `fontified' on these ~7 lines, then the `face' property on them is
> erased.  There is now no matching font lock pattern to apply any new
> faces to these ~7 lines, since the "X" is many lines back.  The same
> thing happens with the next 500 byte chunk, and so on till the end of the
> buffer.
>
> If you set `font-lock-support-mode' to nil and restart font locking, the
> problem isn't apparent.  (Then set the variable back to 'jit-lock-mode.)
>
> This is a fundamental problem with jit-lock-mode: the assumption that
> text to be fontified has no non-trivial context.  This is a difficult
> problem to solve in general.  CC Mode uses some ad-hoc tricks to catch,
> for example, long struct declarations.
>
> > As a contrast, when `emacs-lisp-mode' is used, the entire block is
> > highlighted, and editing does not change the highlighting.
>
> I do not see this in Emacs 24.5.  For me, even in emacs-lisp-mode, I
> still see just the 8 lines being fontified.  If you could give me a
> recipe (starting from emacs-24.5 -Q) to reproduce this, I'd be very
> interested.
>
> > This worked as intended in Emacs 24.5.
>
> >     -- Anders Lindgren
>
> > In GNU Emacs 25.0.50.60 (x86_64-apple-darwin14.5.0, NS appkit-1348.17
> > Version 10.10.5 (Build 14F27))
> >  of 2015-12-28
> > Repository revision: e9916d8880561cc06b6cb73bafe7257b93ffbf4c
> > Windowing system distributor 'Apple', version 10.3.1348
> > Configured using:
> >  'configure --without-dbus'
>
> > Configured features:
> > ACL ZLIB TOOLKIT_SCROLL_BARS NS
>
> > Important settings:
> >   value of $LC_CTYPE: UTF-8
> >   locale-coding-system: utf-8-unix
>
> > Major mode: C/l
>
> > Minor modes in effect:
> >   preproc-font-lock-global-mode: t
> >   preproc-font-lock-mode: t
> >   tooltip-mode: t
> >   global-eldoc-mode: t
> >   electric-indent-mode: t
> >   mouse-wheel-mode: t
> >   tool-bar-mode: t
> >   menu-bar-mode: t
> >   file-name-shadow-mode: t
> >   global-font-lock-mode: t
> >   font-lock-mode: t
> >   blink-cursor-mode: t
> >   auto-composition-mode: t
> >   auto-encryption-mode: t
> >   auto-compression-mode: t
> >   line-number-mode: t
> >   transient-mark-mode: t
> >   abbrev-mode: t
>
> [ .... ]
>
> --
> Alan Mackenzie (Nuremberg, Germany).
>

[-- Attachment #2: Type: text/html, Size: 8087 bytes --]