`font-lock-extend-region-functions' vs. `font-lock-extend-after-change-region-function'

`font-lock-extend-region-functions' vs. `font-lock-extend-after-change-region-function'

[Tobias C. Rittweiler]

Hi,

what is the difference between the variables

  `font-lock-extend-region-functions' 

and 

  `font-lock-extend-after-change-region-function' ?

In particular, how do they relate? Which should be used for what?

From a major-mode writer's point of view. 


I face the following problem: Font-lock, perhaps just jit-lock (I'm
using that), calls `font-lock-fontify-region-function' with start and
end values pretty much arbitrary in the buffer.

That breaks the following:

I have a MATCHER function on `font-lock-keywords' which on some
occasions has to to match beyond the limit it's called with. In that
case it declines, but that results in font-lock skipping over that
region and going to the next. If it doesn't decline, and returns match
data for the whole thing (beyond the limit), only up to the limit will
in fact be fontified.

What should be done in this case?

I thought about specifying a function for
`font-lock-extend-region-functions' which extends the end to the end of
the current defun.

  -T.

Re: `font-lock-extend-region-functions' vs. `font-lock-extend-after-change-region-function'

[Stefan Monnier]

> what is the difference between the variables
>   `font-lock-extend-region-functions'
> and
>   `font-lock-extend-after-change-region-function' ?
> In particular, how do they relate? Which should be used for what?

Have you read the Elisp manual's discussion about font-locking
multiline elements?

The "right" answer is usually font-lock-extend-region-functions, but
it's hard to tell without knowing anything more (sometimes it's
difficult/impossible to write the necessary
font-lock-extend-region-functions).

> I face the following problem: Font-lock, perhaps just jit-lock (I'm
> using that), calls `font-lock-fontify-region-function' with start and
> end values pretty much arbitrary in the buffer.

Yes, that's normal.  That's what font-lock-extend-region-functions is for.


        Stefan

Re: `font-lock-extend-region-functions' vs. `font-lock-extend-after-change-region-function'

[Tobias C. Rittweiler]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> > what is the difference between the variables
> >   `font-lock-extend-region-functions'
> > and
> >   `font-lock-extend-after-change-region-function' ?
> > In particular, how do they relate? Which should be used for what?
>
> Have you read the Elisp manual's discussion about font-locking
> multiline elements?

No, I somehow missed that section. Of course, it's all in there! (blush)


> The "right" answer is usually font-lock-extend-region-functions, but
> it's hard to tell without knowing anything more (sometimes it's
> difficult/impossible to write the necessary
> font-lock-extend-region-functions).
>
> > I face the following problem: Font-lock, perhaps just jit-lock (I'm
> > using that), calls `font-lock-fontify-region-function' with start and
> > end values pretty much arbitrary in the buffer.
>
> Yes, that's normal.  That's what font-lock-extend-region-functions is for.

Right, works like a charm now.

Thank you very much for your prompt answer!,

  -T.

Re: `font-lock-extend-region-functions' vs. `font-lock-extend-after-change-region-function'

[Tobias C. Rittweiler]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> The "right" answer is usually font-lock-extend-region-functions, but
> it's hard to tell without knowing anything more (sometimes it's
> difficult/impossible to write the necessary
> font-lock-extend-region-functions).

Let EXTEND be a function that's pushed onto
`font-lock-extend-region-functions', and let MARKER be font-lock search
function that's specified in `font-lock-keywords'.

I get the following call chain:

  extend: (1, 523)        --> t   (1, 701)
  extend: (1, 701)        --> nil (1, 701)
  marker: pt=1 limit=701  ==> nil
  
  extend: (523, 1034)     --> t   (483, 1156)
           ^^^
  extend: (483, 1156)     --> nil (483, 1156)
  marker: pt=483 limit=1156

(The values in parentheses are font-lock-beg, and font-lock-end.)

As you see, EXTEND extends the region to be fontified from (1, 523) to
(1, 701). MARKER cannot find anything in that region.

But the new region that EXTEND is called with begins at 523, and not 701
even though we processed the region between 523 and 701 already.

This seems more inefficient than it needs to be if I'm not missing
anything.

  -T.

PS: I'm using `jit-lock' as font-lock-support-mode.

Re: `font-lock-extend-region-functions' vs. `font-lock-extend-after-change-region-function'

[Stefan Monnier]

> But the new region that EXTEND is called with begins at 523, and not 701
> even though we processed the region between 523 and 701 already.
> This seems more inefficient than it needs to be if I'm not missing
> anything.

You're not missing anything.  It's an inefficiency in jit-lock, indeed.
jit-lock asks font-lock to highlight 1-523 and font-lock has no way to
tell jit-lock that the first call already highlighted 1-701, so next
time jit-lock needs highlighting it will do 523-1034 because it doesn't
know that 523-701 was done already.

It gets worse in the other direction: if jit-lock asks font-lock to
fontify 1000-1500 and font-lock decids to highlight 500-1600, the part
between 500-1000 will not be redisplayed (i.e. the display will only
take into account the highlighting applied by font-lock next time it
gets refreshed, which usually means at the next command).


        Stefan

Re: `font-lock-extend-region-functions' vs. `font-lock-extend-after-change-region-function'

[Tobias C. Rittweiler]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> > But the new region that EXTEND is called with begins at 523, and not 701
> > even though we processed the region between 523 and 701 already.
> > This seems more inefficient than it needs to be if I'm not missing
> > anything.
>
> You're not missing anything.  It's an inefficiency in jit-lock, indeed.
> jit-lock asks font-lock to highlight 1-523 and font-lock has no way to
> tell jit-lock that the first call already highlighted 1-701, so next
> time jit-lock needs highlighting it will do 523-1034 because it doesn't
> know that 523-701 was done already.

Is this intended to be fixed? For example, the extending of the
font-lock region could be made a seperate step which
`jit-lock-fontify-now' has to run explicitly.

  -T.

Re: `font-lock-extend-region-functions' vs. `font-lock-extend-after-change-region-function'

[Stefan Monnier]

>> > But the new region that EXTEND is called with begins at 523, and not 701
>> > even though we processed the region between 523 and 701 already.
>> > This seems more inefficient than it needs to be if I'm not missing
>> > anything.
>> You're not missing anything.  It's an inefficiency in jit-lock, indeed.
>> jit-lock asks font-lock to highlight 1-523 and font-lock has no way to
>> tell jit-lock that the first call already highlighted 1-701, so next
>> time jit-lock needs highlighting it will do 523-1034 because it doesn't
>> know that 523-701 was done already.
> Is this intended to be fixed? For example, the extending of the
> font-lock region could be made a seperate step which
> `jit-lock-fontify-now' has to run explicitly.

Yes.  This problem was introduced in Emacs-22 but at the time I decided
it was better to "keep it that way for now" (the freeze was coming) and
fix it for Emacs-23, but I never got around to do it.

The way I imagine fixing it is by changing jit-lock-functions such that
the functions on that hook (e.g. font-lock-fontify-region) would return
the new boundaries they used (or nil if they kept the boundaries passed
by jit-lock).


        Stefan

Re: `font-lock-extend-region-functions' vs. `font-lock-extend-after-change-region-function'

[Samuel Bronson]

On Wed, Apr 29, 2009 at 4:26 PM, Stefan Monnier
<monnier@iro.umontreal.ca> wrote:
>> Is this intended to be fixed? For example, the extending of the
>> font-lock region could be made a seperate step which
>> `jit-lock-fontify-now' has to run explicitly.
>
> Yes.  This problem was introduced in Emacs-22 but at the time I decided
> it was better to "keep it that way for now" (the freeze was coming) and
> fix it for Emacs-23, but I never got around to do it.
>
> The way I imagine fixing it is by changing jit-lock-functions such that
> the functions on that hook (e.g. font-lock-fontify-region) would return
> the new boundaries they used (or nil if they kept the boundaries passed
> by jit-lock).

Would that degrade gracefully in the presence of old jit-lock-functions?

Re: `font-lock-extend-region-functions' vs. `font-lock-extend-after-change-region-function'

[Stefan Monnier]

>> The way I imagine fixing it is by changing jit-lock-functions such that
>> the functions on that hook (e.g. font-lock-fontify-region) would return
>> the new boundaries they used (or nil if they kept the boundaries passed
>> by jit-lock).

> Would that degrade gracefully in the presence of old jit-lock-functions?

That would be the intention, of course (I was thinking of accepting any
return value but recognizing a return value of (boundaries START
END) as indication that it uses the new calling convention).


        Stefan

Re: `font-lock-extend-region-functions' vs. `font-lock-extend-after-change-region-function'

[Tobias C. Rittweiler]

Stefan Monnier <monnier@IRO.UMontreal.CA> writes:

> The way I imagine fixing it is by changing jit-lock-functions such that
> the functions on that hook (e.g. font-lock-fontify-region) would return
> the new boundaries they used (or nil if they kept the boundaries passed
> by jit-lock).

I'd also like if the font-lock-extend-region-functions were called with
an argument counter representing the number of extend region passes so
far.

Of course that would be a backwards-incompatible API change but you
could bind a special variable representing the count.

Purpose is to guard against infinite loops.


Alternatively `font-lock-default-fontify-region' could be made smarter
in so far as it could check whether a pass actually changed
font-lock-beg and font-lock-end, and if not, would not try another pass.

(Strictly speaking the latter would also mean a slight API change, but
anyone who were bitten by this should be slapped three times by some
appropriately thick book anyway. :-))

  -T.

Re: `font-lock-extend-region-functions' vs. `font-lock-extend-after-change-region-function'

[Stefan Monnier]

> Alternatively `font-lock-default-fontify-region' could be made smarter
> in so far as it could check whether a pass actually changed
> font-lock-beg and font-lock-end, and if not, would not try
> another pass.

If a function doesn't actually change font-lock-{beg/end}, it should
return nil.

And yes, the risk of inf-loop is a problem, but I don't know how to
solve it, because counters aren't really a solution either.
Maybe I could simply impose an arbitrary limit so we can't go through
the loop more than 10 times (or N times, N being the length of
font-lock-default-fontify-region).


        Stefan

Re: `font-lock-extend-region-functions' vs. `font-lock-extend-after-change-region-function'

[Tobias C. Rittweiler]

Stefan Monnier <monnier@IRO.UMontreal.CA> writes:

> > Alternatively `font-lock-default-fontify-region' could be made
> > smarter in so far as it could check whether a pass actually changed
> > font-lock-beg and font-lock-end, and if not, would not try another
> > pass.
>
> If a function doesn't actually change font-lock-{beg/end}, it should
> return nil.

Right, my request is to test for this assertion explicitly.

  -T.

Re: `font-lock-extend-region-functions' vs. `font-lock-extend-after-change-region-function'

[Tobias C. Rittweiler]

"Tobias C. Rittweiler" <tcr@freebits.de> writes:

> Hi,
>
> what is the difference between the variables
>
>   `font-lock-extend-region-functions' 
>
> and 
>
>   `font-lock-extend-after-change-region-function' ?

Another question: The latter variable requires that its function
preverse match-data.

Looking at how the first variable is used, I don't think the same is
true for the first variable, right? (Perhaps it docstring could
explicitly say that its functions do not need to save match-data or
point.)

  -T.