possible bug in anchored font-lock functions

possible bug in anchored font-lock functions

[Sam Halliday]

[-- Attachment #1: Type: text/plain, Size: 2300 bytes --]

Dear all,

I think I might have found a bug in GNU Emacs but I would like to check
my understanding first, before filing a report or (ideally) fixing it.

In Search Based Fontification[1] it is possible to specify a function as a
matcher. It must obey the following contract:

> it receives one argument, the limit of the search; it should begin
> searching at point, and not search beyond the limit. It should return
> non-nil if it succeeds, and set the match data to describe the match
> that was found. Returning nil indicates failure of the search.

In addition, it is possible to extend the region to be fontified by
adding a routine to `font-lock-extend-region-functions` that in-place
updates the `font-lock-beg` or `font-lock-end` variables (these
variables are not visible in a function matcher). Let's ignore
`font-lock-multiline` property approaches, I'm not using them.

Indeed, I have confirmed that if I extend the region in a
`font-lock-extend-region-functions` then the `limit` does increase for
my function matcher!

However, if I use an an `anchored` matcher, having the form `(matcher .
anchored-highlighter)`, where the `anchored-highlighter` is a
`function`, my custom `font-lock-{beg,end}` regions are ignored and
`limit` is much reduced!

Is there something I need to do so that anchored matchers receive the
calculated regions or are they only designed to extend to the end of the
current line by default?

If I had to guess I'd say the anchored matcher is forgetting to use
`font-lock-{beg,end}` and is instead calculating a new limit or using a
cached version of the limit from before
`font-lock-extend-region-functions` ran.

I would greatly appreciate it if somebody could please point me to the
source code in GNU Emacs where the `font-lock-keywords` are called for
anchored matchers. I suspect the limit is also broken for `regexp`
matchers, not just `function`, but I have no way of printing out `limit`
in that case.

A final note, this is the first time I've written a syntax-table and
font-lock for a programming language, and I have found the experience to
be much more pleasant than I expected! The font-lock-keyword API is
lovely to work with and I've been using the rx macro to avoid writing
regexps by hand... my code reads like a simplified BNF description!


[-- Attachment #2.1: Type: text/plain, Size: 128 bytes --]


[1] https://www.gnu.org/software/emacs/manual/html_mono/elisp.html#Search_002dbased-Fontification

-- 
Best regards,
Sam

[-- Attachment #2.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 194 bytes --]

Re: possible bug in anchored font-lock functions

[Sam Halliday]

Further to this, I think I've tracked the issue to
`font-lock-fontify-anchored-keywords` which is called with a `limit`
parameter, but then `limit` is mutated very intentionally to the limit
generated by the outer matcher... so I think this is NOT a bug, but
just different behaviour to what I expected. i.e. the first pattern
resets the limit for the anchored highlighters.
On Sun, 11 Nov 2018 at 14:26, Sam Halliday <sam.halliday@gmail.com> wrote:
>
> Dear all,
>
> I think I might have found a bug in GNU Emacs but I would like to check
> my understanding first, before filing a report or (ideally) fixing it.
>
> In Search Based Fontification[1] it is possible to specify a function as a
> matcher. It must obey the following contract:
>
> > it receives one argument, the limit of the search; it should begin
> > searching at point, and not search beyond the limit. It should return
> > non-nil if it succeeds, and set the match data to describe the match
> > that was found. Returning nil indicates failure of the search.
>
> In addition, it is possible to extend the region to be fontified by
> adding a routine to `font-lock-extend-region-functions` that in-place
> updates the `font-lock-beg` or `font-lock-end` variables (these
> variables are not visible in a function matcher). Let's ignore
> `font-lock-multiline` property approaches, I'm not using them.
>
> Indeed, I have confirmed that if I extend the region in a
> `font-lock-extend-region-functions` then the `limit` does increase for
> my function matcher!
>
> However, if I use an an `anchored` matcher, having the form `(matcher .
> anchored-highlighter)`, where the `anchored-highlighter` is a
> `function`, my custom `font-lock-{beg,end}` regions are ignored and
> `limit` is much reduced!
>
> Is there something I need to do so that anchored matchers receive the
> calculated regions or are they only designed to extend to the end of the
> current line by default?
>
> If I had to guess I'd say the anchored matcher is forgetting to use
> `font-lock-{beg,end}` and is instead calculating a new limit or using a
> cached version of the limit from before
> `font-lock-extend-region-functions` ran.
>
> I would greatly appreciate it if somebody could please point me to the
> source code in GNU Emacs where the `font-lock-keywords` are called for
> anchored matchers. I suspect the limit is also broken for `regexp`
> matchers, not just `function`, but I have no way of printing out `limit`
> in that case.
>
> A final note, this is the first time I've written a syntax-table and
> font-lock for a programming language, and I have found the experience to
> be much more pleasant than I expected! The font-lock-keyword API is
> lovely to work with and I've been using the rx macro to avoid writing
> regexps by hand... my code reads like a simplified BNF description!
>
>
> [1] https://www.gnu.org/software/emacs/manual/html_mono/elisp.html#Search_002dbased-Fontification
>
> --
> Best regards,
> Sam

Re: possible bug in anchored font-lock functions

[Stefan Monnier]

> However, if I use an an `anchored` matcher, having the form `(matcher .
> anchored-highlighter)`, where the `anchored-highlighter` is a
> `function`, my custom `font-lock-{beg,end}` regions are ignored and
> `limit` is much reduced!

Of course: the anchored matcher is for use within a particular subregion.
By default, this subregion is the text between where MATCHER matched and
end-of-line.

> Further to this, I think I've tracked the issue to
> `font-lock-fontify-anchored-keywords` which is called with a `limit`
> parameter, but then `limit` is mutated very intentionally to the limit
> generated by the outer matcher... so I think this is NOT a bug, but
> just different behaviour to what I expected. i.e. the first pattern
> resets the limit for the anchored highlighters.

The docstring of font-lock-keywords explains the following:

    [...]
    The above-mentioned exception is as follows.  The limit of the
    MATCHER search defaults to the end of the line after
    PRE-MATCH-FORM is evaluated.  However, if PRE-MATCH-FORM returns
    a position greater than the position after PRE-MATCH-FORM is
    evaluated, that position is used as the limit of the search.  It
    is generally a bad idea to return a position greater than the end
    of the line, i.e., cause the MATCHER search to span lines.
    [...]

Of "bad idea" doesn't mean it can't work, but it comes with the usual
caveat for multiline matches.


        Stefan