Bug in font-lock-syntactic-keywords handling?

Bug in font-lock-syntactic-keywords handling?

[immerrr again]

Hi all

I've run into an interesting issue with syntactic fontification with
lua-mode: it uses font-lock-syntactic-keywords to mark up long-strings
and long-comments as advised earlier in this list [1] and there has
been a report that this approach failed on a certain file [2]. It
looks like that initial fontification indeed fails on that
"connection.lua", but after-change refontification works fine.

In a nutshell, the problem is that font-lock-syntactically-fontified
is only moved forward when font-lock-syntactically-fontified < start.
Which means that when you first propertize [s0; s1) and then
propertize [s1, s2), font-lock-syntactically-fontified will still be
at s1 which is clearly inconsistent. Let's look at the initial
fontification of the file I've referenced above that exhibits that
behaviour (messages are annotated with their respective line numbers
according to [3]):

===========
1212: unfontify-region: [1; 520]
1217: font-lock-syntactically-fontified: 0
1217: start: 1
1218: moving start to 1
1219: set font-lock-syntactically-fontified: 520
1220: font-lock-fontify-syntactic-keywords-region: [1; 520]
============
1212: unfontify-region: [520; 1082]
1217: font-lock-syntactically-fontified: 520
1217: start: 520
1220: font-lock-fontify-syntactic-keywords-region: [520; 1082]
============
1212: unfontify-region: [1082; 1583]
1217: font-lock-syntactically-fontified: 520
1217: start: 1082
1218: moving start to 520
1219: set font-lock-syntactically-fontified: 1583
1220: font-lock-fontify-syntactic-keywords-region: [520; 1583]

As you can see, for the third iteration start moves back to the
beginning of the second chunk and it tries to re-propertize it without
unfontifying which causes an bug in lua-mode's syntactic keyword
handling. I do know that I (ab-)used a deprecated variable to maintain
compatibility with Emacs23 and using `syntax-propertize-function`
should fix that for Emacs24, but I'd be grateful if anyone pointed me
to a simple fix for Emacs23.

Cheers,
immerrr

1. http://comments.gmane.org/gmane.emacs.devel/153447
2. https://github.com/immerrr/lua-mode/issues/78
3. http://git.savannah.gnu.org/cgit/emacs.git/tree/lisp/font-lock.el#n1212

Re: Bug in font-lock-syntactic-keywords handling?

[Stefan Monnier]

> As you can see, for the third iteration start moves back to the
> beginning of the second chunk and it tries to re-propertize it without
> unfontifying which causes an bug in lua-mode's syntactic keyword
> handling. I do know that I (ab-)used a deprecated variable to maintain
> compatibility with Emacs23 and using `syntax-propertize-function`
> should fix that for Emacs24, but I'd be grateful if anyone pointed me
> to a simple fix for Emacs23.

That seems like a performance bug (it re-fontifies something that's
already been fontified), indeed, but I don't see why that would cause
something to "fail".

If it really causes a failure, I suspect that the failure is really
caused by an underlying bug in the specific syntactic-keywords rules,
which merely gets triggered by this performance bug.


        Stefan

Re: Bug in font-lock-syntactic-keywords handling?

[immerrr again]

On Mon, Oct 6, 2014 at 5:14 PM, Stefan Monnier <monnier@iro.umontreal.ca> wrote:
>
> That seems like a performance bug (it re-fontifies something that's
> already been fontified), indeed, but I don't see why that would cause
> something to "fail".
>
> If it really causes a failure, I suspect that the failure is really
> caused by an underlying bug in the specific syntactic-keywords rules,
> which merely gets triggered by this performance bug.
>
>
>         Stefan

The problem is that it doesn't unfontify the already processed chunk beforehand.

It this particular situation, when I encounter a "--[[" comment
starter I need to be sure that there are no dashes before
("---[[...\n" is a single-line comment by design) and that this
starter is not inside another comment. I did this the "smart" way: I
checked (elt ppss 4) of the *second* dash, so that when there's
something like "--X-[[" with X being the point, the point is already
inside a single-line comment. That single check covers both cases,
yet, as most "smart" things do, it breaks unexpectedly.

This one breaks when "--[[" is already propertized as a comment opener
and the second dash is already a part of that comment. This causes
comment-open matcher to fail and simple two-step propertize function
[1],

    (lambda (limit) (or (try-close-comment limit) (try-open-comment limit)))

doesn't work, since there is no comment to be closed at the beginning
of the chunk, the opener is already propertized and thus skipped and
the closer is never invoked again.

Cheers,
immerrr

1. This lambda is a simplified version of
https://github.com/immerrr/lua-mode/blob/master/lua-mode.el#L868-L872

Re: Bug in font-lock-syntactic-keywords handling?

[Stefan Monnier]

> The problem is that it doesn't unfontify the already processed
> chunk beforehand.

Oh, indeed, sorry for being dense.
That's indeed a bug.  You could try to work around it with something like:

(defvar lua-font-lock-syntactic-keywords
  '(((lambda (limit)
      ;; font-lock sometimes fails to erase the `syntax-table' property before
      ;; calling us.  So let's make sure it's clean before moving on.
      (remove-text-properties (point) limit '(syntax-table))
      nil))
    (lua-match-multiline-literal-bounds
     (1 "!" nil noerror)
     (2 "|" nil noerror))))

Also, I recommend you use syntax-propertize-function when available.
It comes with various advantages (e.g. it also works when font-lock is
disabled).


        Stefan


PS: BTW, do you have any news about the copyright status of lua-mode
that would prevent including it in GNU ELPA?

Re: Bug in font-lock-syntactic-keywords handling?

[immerrr again]

On Mon, Oct 6, 2014 at 8:28 PM, Stefan Monnier <monnier@iro.umontreal.ca> wrote:
>
>
> Also, I recommend you use syntax-propertize-function when available.
> It comes with various advantages (e.g. it also works when font-lock is
> disabled).
>
>

Thanks a lot for your help, this should work. For some reason I
considered remove-text-properties an expensive operation and avoided
it, but a quick benchmark showed that syntax-ppss takes a lot more
time during syntactical fontification.

>
> PS: BTW, do you have any news about the copyright status of lua-mode
> that would prevent including it in GNU ELPA?

Sadly, no news. I haven't tried reaching those who hadn't signed the
assignment since the last time and I'm a lot more humble now in
estimates of my spare time to do actual rewriting. I will most likely
have another stab at it before Christmas.