bug#20146: font-lock-extend-jit-lock-region-after-change: results are discarded instead of being returned.

[Alan Mackenzie <acm@muc.de>]

Hello, Stefan.

On Fri, Mar 20, 2015 at 10:29:16PM -0400, Stefan Monnier wrote:
> >> > Perhaps we could implement the convention that when a major mode has
> >> > positively set the font-lock region's start and end points, these should
> >> > be accepted by F/J-lock, but when not, F/J-lock should be free to alter
> >> > them (as it typically does now).
> >> No the core of the API is font-lock-fontify-region and it should work
> >> with *any* bounds (i.e. if these need to be extended, it should be done
> >> by font-lock-extend-region-function).
> > However, when the bounds are set by the major mode, those bounds should
> > be respected by Font Lock.

> The major mode sets font-lock-extend-region-function and this functions'
> result should be (and is) respected by the rest of font-lock.

It is not.  font-lock-extend-region-functions (note the "s") is plural,
and all functions on it are run repeatedly until none makes a change.  So
when the major mode sets the region, this is instantly violated by the
other functions in f-l-extend-r-f.  This is what caused bug #19669, and
I'm still struggling to find a way round it.

The problem is the mixing up of Font Lock's internal functions with the
major mode's in f-l-extend-region-functions.  The major mode's function
is not given the priority it should have.

Is there any particular reason for the repeated running of the functions
on f-l-extend-region-functions?  Why are they not simply each run once?
Is there any use case where it is helpful for one of these functions to
make a second (or subsequent) change to the font-lock region?  Why don't
we change this bit to a simple `run-hook' form?

> But callers of font-lock-fontify-region (such as
> font-lock-after-change-function, or jit-lock) can choose *any* bounds
> they feel like and font-lock-fontify-region should behave correctly.

If the major mode is going to get *any* bounds rather than the ones it
has already specified by its function on f-l-extend-region-functions, it
will need to adjust those bounds again, to a syntactically acceptable
place.  This can't be the Right Thing.

> >> Jit-lock is implemented on top of that API and is hence free to use
> >> any bounds it sees fit.
> > But surely not to pass bounds to the major mode that the major mode
> > must, for a second time, adjust.  This is crazy.

> No, it is just good design to keep complexity under check.

???

> >> If you rely on more specific bounds being passed to
> >> font-lock-fontify-region, that means you have a problem on your side.
> > There is a problem, yes, but on which side it is is a matter for
> > discussion.  CC Mode has always relied on the specific bounds that it
> > provides.

> AFAIK CC-mode does not provide any bounds.  Instead it uses
> font-lock-extend-after-change-region-function which changes the part of
> the buffer that is invalidated, which is something different.

No it's not different.  The bounds CC Mode provides are those around the
region which is to be invalidated, and later refontified.  I think you're
picking nits here.

> The relationship between the two is brittle and subtle, so relying on it
> is crazy and a good way to get impenetrable code with special cases
> tacked on top of special cases with lots of unspecified assumptions that
> only hold in common cases.

What is the alternative?  CC Mode knows exactly what portion of the
buffer needs refontifying, Font Lock doesn't, and can't.  Any chance of a
robust way of communicating those region bounds to Font Lock?  

> > You yourself pointed out that CC Mode is the sole user of the
> > functionality for expanding FL regions.

> Indeed.  Others use font-lock-extend-region-function.

> font-lock-extend-after-change-region-function can be useful when
> a *change* on line N can cause a highlighting change on a line <N, or
> when it causes a highlighting change on a line >N and you don't like the
> jit-lock-context-time delay (or you want it to work correctly even if
> jit-lock(-context) is not in use).

Yes.

>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).