bug#71345: Feature: unleash font-lock's secret weapon; handle Qfontified = non-nil

bug#71345: Feature: unleash font-lock's secret weapon; handle Qfontified = non-nil

[JD Smith]

[-- Attachment #1: Type: text/plain, Size: 3730 bytes --]

	
Many emacs packages concern themselves with updating the display just-in-time, prior to final redisplay.  They may apply overlays, additional custom fontification, images or stipples, etc.  And they need this to happen before redisplay, to avoid flashing temporarily untreated text.  They may be slow to run, so cannot afford to update the entire buffer when an update is needed.  And they may update relatively rapidly based on point, edits, or external events.

Package authors who encounter this class of problem may reach for post-command-hook's, pre-redisplay-functions, window-scroll-functions, etc.   But in all cases they will face an unfortunate reality: in many cases you cannot know the window bounds accurately until after redisplay has completed.  The docs are very up front about this, but don't offer a great solution.  You can force a window bounds update with (window-end win t), but that appears to incur the full cost of redisplay, and in many cases costs 5-10x as much as the work you wanted to do in the displayed portion of the buffer itself, and does not always work.  Lots of discussion has occurred on this topic, with ideas like post-redisplay-hook considered but never implemented.

These are effectively the same problems that font-lock faces.  But font-lock (via jit-lock) has a secret weapon it alone may use: the fontified=nil handler.  As I understand it, redisplay looks for fontified=nil and calls fontification-functions (usually containing jit-lock-function) as many times as needed to achieve fontification of the about-to-be displayed window contents.   jit-lock-function adds face information to chunks of jit-lock-chunk-size (1500, by default), and gets called as many times as needed.  This is very efficient, compared to anything which can be done in elisp.

What I'm proposing is to open the use of font-lock's secret weapon for others.  You could imagine many ways to achieve this, but a simple one might be:

Only Qfontified = t indicates "clean" buffer text.
For other values (including nil), if fontification-functions is a list, call each function with the start position of "dirty" buffer text, same as normal.
If fontification-functions is instead an alist, consult the value of Qfontitified and call the correct function(s) in the list whose key is that value 
(Optional) The key value `t' indicates a function to be called with all Qfontified values, passed to the function in a second argument.

In my particular case, I've been trying for several months to achieve a scheme for position-dependent additional fontification based on treesitter scope: think font-lock, but respondent not just to the contents of the buffer, but also the location of point within it.  My particular application is drawing treesitter scope-aware indentation bars.  There are many other applications you could envision, including many discussed here (e.g. bug#22404). 

Font-lock can handle this type of things very well (by using its secret weapon).  But what I (and many packages) want to do is in addition to font-lock.  You can of course do this within font-lock: just consult treesitter scope and set fontified=nil across potentially large regions of the buffer -- regions which may change rapidly as point moves.  It will work well-enough, but it ends up doing a ton of wasted extra work as point moves around, continuously fully refontifying the underlying text, text which was perfectly well fontified already.

I have tried window-scroll-functions, post-command-hooks, jit-lock-functions, etc.   All come back to the same issue that no doubt originally motivated jit-lock's need for special handler support: you don't know what the window will contain at the right time.

[-- Attachment #2: Type: text/html, Size: 4664 bytes --]

bug#71345: Feature: unleash font-lock's secret weapon; handle Qfontified = non-nil

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> In my particular case, I've been trying for several months to achieve
> a scheme for position-dependent additional fontification based on treesitter
> scope: think font-lock, but respondent not just to the contents of the
> buffer, but also the location of point within it.  My particular application
> is drawing treesitter scope-aware indentation bars.  There are many other
> applications you could envision, including many discussed here
> (e.g. bug#22404). 

I've been meaning to add support for that to jit-lock.
Here's the design I have in mind:

From the outside, only one change, i.e. a new function

    (jit-lock-flush BEG END FONTIFICATION-FUNCTION)

where FONTIFICATION-FUNCTION is the exact same function that was passed
to `jit-lock-register` (and is thus used as a kind of identifier of the
corresponding backend).

So in your case, when the position-dependent highlighting needs to be
updated, you'd request it by calling `jit-lock-flush`.

The way it would be implemented inside `jit-lock.el` is that
`jit-lock-flush` would set `fontified` to nil over that whole region,
but before that it would scan the region for those places where
`fontified` is non-nil, and set the `jit-lock-already-fontified` property
to the list of currently active backends (minus FONTIFICATION-FUNCTION),
so as to remember that while this region does require jit-lock action
those backends don't need to be called.

Then when `jit-lock-fontify-now` is called it will skip those backends
mentioned in `jit-lock-already-fontified`.


        Stefan

bug#71345: Feature: unleash font-lock's secret weapon; handle Qfontified = non-nil

[JD Smith]

[-- Attachment #1: Type: text/plain, Size: 3425 bytes --]



> On Jun 3, 2024, at 12:56 PM, Stefan Monnier <monnier@iro.umontreal.ca> wrote:
> 
>> In my particular case, I've been trying for several months to achieve
>> a scheme for position-dependent additional fontification based on treesitter
>> scope: think font-lock, but respondent not just to the contents of the
>> buffer, but also the location of point within it.  My particular application
>> is drawing treesitter scope-aware indentation bars.  There are many other
>> applications you could envision, including many discussed here
>> (e.g. bug#22404).
> 
> I've been meaning to add support for that to jit-lock.
> Here's the design I have in mind:
> 
> From the outside, only one change, i.e. a new function
> 
>    (jit-lock-flush BEG END FONTIFICATION-FUNCTION)
> 
> where FONTIFICATION-FUNCTION is the exact same function that was passed
> to `jit-lock-register` (and is thus used as a kind of identifier of the
> corresponding backend).
> 
> So in your case, when the position-dependent highlighting needs to be
> updated, you'd request it by calling `jit-lock-flush`.
> 
> The way it would be implemented inside `jit-lock.el` is that
> `jit-lock-flush` would set `fontified` to nil over that whole region,
> but before that it would scan the region for those places where
> `fontified` is non-nil, and set the `jit-lock-already-fontified` property
> to the list of currently active backends (minus FONTIFICATION-FUNCTION),
> so as to remember that while this region does require jit-lock action
> those backends don't need to be called.

Thanks.  Very interesting approach, recycling the existing Qfontified handler.  I wonder though, if you have several different unrelated functions calling `jit-lock-flush' with different FONTIFICATION-FUNCTION's prior to jit-lock-fontify-now running on a region, won't they step on each other?  I.e it seems that you would need to look for existing jit-lock-already-fontified+fontified=nil properties over the region mentioned in jit-lock-flush and subtract the passed FONTIFICATION-FUNCTION from the various values already found there.  Of course you'd also need to handle the case where you "subtract it all the way to nil".  That starts to sound like a lot of property slinging, which might even dominate the work done.   I don't guess there is a fast hidden

 (remove-from-text-property-lists beg end property val-to-remove)

You could end also up with a quilted patchwork of different levels of the already-fontified property before you clear them, which might make jit-lock-fontify-now's job harder.  Although for my particular case I doubt this will be a problem:

jit-lock-after-change will set fontified=nil (and btw, will probably also need to clear the already-fontified property over the affected change region).
My tree-sitter based code, upon edits and position-based scope changes, will use jit-lock-flush to add already-fontified properties to the affected locations where fontified=t (if any).

I imagine that the functions may also need a way to opt-out of "deferred contextual refontification", for example if they add some other properties/overlays orthogonal to face.  Perhaps this could be done by having jit-lock-context-fontify setup the "rest of the buffer" with already-fontified = jit-lock-no-context-functions — an optional list of FONTIFICATION-FUNCTION's that don't require contextual refontification.

[-- Attachment #2: Type: text/html, Size: 4443 bytes --]

bug#71345: Feature: unleash font-lock's secret weapon; handle Qfontified = non-nil

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> Thanks.  Very interesting approach, recycling the existing Qfontified
> handler.  I wonder though, if you have several different unrelated functions
> calling `jit-lock-flush' with different FONTIFICATION-FUNCTION's prior to
> jit-lock-fontify-now running on a region, won't they step on each other?

Not if `jit-lock.el` handles it correctly, no.

> I.e it seems that you would need to look for existing
> jit-lock-already-fontified+fontified=nil properties over the region
> mentioned in jit-lock-flush and subtract the passed FONTIFICATION-FUNCTION
> from the various values already found there.

Yes, of course.

> Of course you'd also need to handle the case where you "subtract it
> all the way to nil".

I don't think you'd need to do anything special for this case.

> That starts to sound like a lot of property slinging, which might even
> dominate the work done.

Indeed, this amount of work could become significant.  It's my main
worry, but I don't have a clear feel for how serious it would be
in practice.

We could try and unify `fontified` and `jit-lock-already-fontified` by
having a `fontified-done-value` variable and making the redisplay call
jit-lock whenever `fontified` has a value that's not-eq from
`fontified-done-value`.

So jit-lock would set `fontified-done-value` to the list of backends.

> I imagine that the functions may also need a way to opt-out of "deferred
> contextual refontification", for example if they add some other
> properties/overlays orthogonal to face.

`jit-lock.el` already has that info (it's the second arg to
`jit-lock-register`), but it currently doesn't keep track of it
individually for each backend.


        Stefan

bug#71345: Feature: unleash font-lock's secret weapon; handle Qfontified = non-nil

[JD Smith]

[-- Attachment #1: Type: text/plain, Size: 3464 bytes --]


> 
>> That starts to sound like a lot of property slinging, which might even
>> dominate the work done.
> 
> Indeed, this amount of work could become significant.  It's my main
> worry, but I don't have a clear feel for how serious it would be
> in practice.

In my situation, the most likely scenario is that fontified=nil is noticed during redisplay when there is a fairly large stretch of already-fontified property having the same value.  So jit-lock-fontify-now will quickly find a nice large chunk to call my FONTIFICATION-FUNCTION=F-F with.

Since jit-lock-after-change will likely clear away already-fontified and set fontified=nil, a single additional F-F on top of jit-lock-function will probably be very well handled.  A good question is how it would scale with more functions all operating in the same region.  One idea is to rig up a test file, do some fake jit-lock-flushing on it, and check performance of just subtracting/searching/dividing the already-fontified property as you add more (fake) F-F's.   For me, jit-lock-fontify-now of a 2500 char chunk in a heavy treesitter buffer is in the 2-5ms range.  Individual F-F's could be much lighter weight.

> We could try and unify `fontified` and `jit-lock-already-fontified` by
> having a `fontified-done-value` variable and making the redisplay call
> jit-lock whenever `fontified` has a value that's not-eq from
> `fontified-done-value`.
> 
> So jit-lock would set `fontified-done-value` to the list of backends.

Nice idea.  Since redisplay just directs jit-lock to a "starting position", it would be free to update however much buffer text it wants.  To overcome the issue of "many small domains", jit-lock could cheat, for example checking if ANY chars in the next block need a particular F-F, and running it on the full block if so.  That's already implicitly how jit-lock works now if I understand correctly.  But things like `text-property-any' will be quickly defeated by the combinatorics of a large F-F set.  Also, an advantage of keeping the fontified=nil semantics is that changes to jit-lock could be back-ported to earlier Emacs versions.

So here's an idea.  You could invert the logic, and have a set of `fontified-pending' properties which jit-lock-flush adds to as it sets fontified=nil, maintaining one property symbol for each F-F (e.g. fontified-pending-N).  Then jit-lock-flush's only job is to select and apply one such property over the region, and fontify-now can use a simple and very fast `text-property-any' as it loops through the list of F-F's, and a final `remove-list-of-text-properties' to strip them all away.  For the convenience of jit-lock-after-change, setting a "single property to rule them all" (fontified-pending=t) directs jit-lock to run all the F-F's, there.  fontify-now could check for that first, then only bother to look for the individual (-N) properties if it is not set.  Is there code out there that sets fontified=nil on its own or is that an internal detail of jit-lock?

>> I imagine that the functions may also need a way to opt-out of "deferred
>> contextual refontification", for example if they add some other
>> properties/overlays orthogonal to face.
> 
> `jit-lock.el` already has that info (it's the second arg to
> `jit-lock-register`), but it currently doesn't keep track of it
> individually for each backend.

Great, so this information could be put into action using one of these approaches.


[-- Attachment #2: Type: text/html, Size: 4739 bytes --]

bug#71345: Feature: unleash font-lock's secret weapon; handle Qfontified = non-nil

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>>> That starts to sound like a lot of property slinging, which might even
>>> dominate the work done.
>> Indeed, this amount of work could become significant.  It's my main
>> worry, but I don't have a clear feel for how serious it would be
>> in practice.
> In my situation, the most likely scenario is that fontified=nil is noticed
> during redisplay when there is a fairly large stretch of already-fontified
> property having the same value.  So jit-lock-fontify-now will quickly find
> a nice large chunk to call my FONTIFICATION-FUNCTION=F-F with.
>
> Since jit-lock-after-change will likely clear away already-fontified and set
> fontified=nil, a single additional F-F on top of jit-lock-function will
> probably be very well handled.  A good question is how it would scale with
> more functions all operating in the same region.  One idea is to rig up
> a test file, do some fake jit-lock-flushing on it, and check performance of
> just subtracting/searching/dividing the already-fontified property as you
> add more (fake) F-F's.   For me, jit-lock-fontify-now of a 2500 char chunk
> in a heavy treesitter buffer is in the 2-5ms range.  Individual F-F's could
> be much lighter weight.

I must say that I can't follow you.  I suspect we're not talking about
quite the same thing.  Could you clarify what is the costs you imagine
could be significant?  What you compare it to?

You seem to be comparing "a single big jit-lock backend" vs "several
jit-lock backends", which is a completely different worry from mine.
Splitting a backend into several backends comes with many more issues
(such as the issue of fighting over which one controls which properties,
or removing internal dependencies such that none of them needs to look
at the properties set by the others, ...) but that seems largely
orthogonal to the question at hand: if you want to be able to refresh
the position-dependent highlighting separately from the rest of the
highlighting you need that position-dependent highlighting to be
independent anyway (e.g. you need to be able to remove it without
affecting the position-independent highlighting).

> But things like `text-property-any' will be quickly defeated by the
> combinatorics of a large F-F set.

`text-property-any` only tests `eq`ness so it works just as quickly with
a property made up of a million-element list as with a property made of
a boolean.

IOW, I again can't follow you.

> So here's an idea.  You could invert the logic, and have a set of
> `fontified-pending' properties which jit-lock-flush adds to as it sets
> fontified=nil,

Yes, of course, we could use the complement set.


        Stefan

bug#71345: Feature: unleash font-lock's secret weapon; handle Qfontified = non-nil

[JD Smith]

[-- Attachment #1: Type: text/plain, Size: 7056 bytes --]



> On Jun 4, 2024, at 10:15 AM, Stefan Monnier <monnier@iro.umontreal.ca> wrote:
> 
>>>> That starts to sound like a lot of property slinging, which might even
>>>> dominate the work done.
>>> Indeed, this amount of work could become significant.  It's my main
>>> worry, but I don't have a clear feel for how serious it would be
>>> in practice.
>> In my situation, the most likely scenario is that fontified=nil is noticed
>> during redisplay when there is a fairly large stretch of already-fontified
>> property having the same value.  So jit-lock-fontify-now will quickly find
>> a nice large chunk to call my FONTIFICATION-FUNCTION=F-F with.

>> Since jit-lock-after-change will likely clear away already-fontified and set
>> fontified=nil, a single additional F-F on top of jit-lock-function will
>> probably be very well handled.  A good question is how it would scale with
>> more functions all operating in the same region.  One idea is to rig up
>> a test file, do some fake jit-lock-flushing on it, and check performance of
>> just subtracting/searching/dividing the already-fontified property as you
>> add more (fake) F-F's.   For me, jit-lock-fontify-now of a 2500 char chunk
>> in a heavy treesitter buffer is in the 2-5ms range.  Individual F-F's could
>> be much lighter weight.
> 
> I must say that I can't follow you.  I suspect we're not talking about
> quite the same thing.  Could you clarify what is the costs you imagine
> could be significant?  What you compare it to?

Apologies for the lack of clarity.  Here I was revisiting the notion that "this amount of work could become significant."  I was trying to convey that the costs of i) applying the proposed jit-lock-already-fontified property (with subtraction, as in your original idea), and ii) parsing it into regions in jit-lock-fontify-now might in fact be fairly minimal, for my situation.  My situation = font-lock-fontify-region + my-special-fontify-region.  

In other words, for many cases there would in fact not be much property management work.  This leads naturally to considering more complicated cases, with several additional fontification functions all interoperating.  The property work will grow quickly (though I also outlined some ideas to keep it under control, which probably already occurred to you).

> You seem to be comparing "a single big jit-lock backend" vs "several
> jit-lock backends", which is a completely different worry from mine.

This is indeed the implicit comparison I'm making: 

the current situation of a single big backend which redoes EVERYTHING as potentially large regions are invalidated, with much of its work done unnecessarily vs. 
multiple backends used for more targeted & orthogonal updates, at the cost of additional property management in jit-lock.

As long as the additional property management costs are well below the savings you reap from not having repeated the unnecessary work, this would be a positive outcome.  The 2-5ms I mention is the cost for me of running "one large backend" over one chunk — namely font-lock-fontify-region with treesitter backing.  In my scenario of bar updates resulting from point motion, this represents purely wasted work.  So if the additional "property management" costs per chunk are, say, 100x below that, you are safely in "well worth it" territory.

> Splitting a backend into several backends comes with many more issues
> (such as the issue of fighting over which one controls which properties,
> or removing internal dependencies such that none of them needs to look
> at the properties set by the others, ...) but that seems largely
> orthogonal to the question at hand: if you want to be able to refresh
> the position-dependent highlighting separately from the rest of the
> highlighting you need that position-dependent highlighting to be
> independent anyway (e.g. you need to be able to remove it without
> affecting the position-independent highlighting).

Agreed that could be an issue.  In practice keyword-based fontification can lead to these same sorts of conflicts for non trivial FACE forms too.  So backends would need to ensure the changes they are making in the buffer are interoperable with the other likely backends (in particular font-lock).

This also raises the question of what should happen after-change.  In my view, that should wipe the slate fully clean in the changed region.  This means other backends would still need to add to font-lock-extra-managed-props any unusual properties they will apply (or do the equivalent on their own during unfontify).  And the order of backend registration would be significant, with the last one having "the final word".  Context re-fontification is a special case of this: some backends could ignore that, others would need to be re-run — something they'd have to decide by themselves. 

>> But things like `text-property-any' will be quickly defeated by the
>> combinatorics of a large F-F set.
> 
> `text-property-any` only tests `eq`ness so it works just as quickly with
> a property made up of a million-element list as with a property made of
> a boolean.
> 
> IOW, I again can't follow you.

I was referring to the number of such lists, not the speed of testing them.  Imagine a scenario as follows: 4 different backends are all operating over the same region — F (for normal font-lock), A, B, and C.  As various invalidation events occur and backends call jit-lock-flush, a given region of text may accumulate a patchwork of already-fontified lists (here assuming F always wipes the slate clean as it works, and therefore always appears on the already-fontified list):

'(F) '(F A) '(F B) '(F C) '(F A B)  '(F A C) '(F B C)  '(F A B C)

So jit-lock-fontify-now's job has gotten quite challenging, as it decides over what region to apply a particular backend, say A.  To know whether it can skip A, it must either look inside all the lists to see if there's an A, or it must look for lists `eq` to all possible combinations which contain A.  

It's possible you've already conceived of this and have a solution in mind; apologies if so.  My simple solution to this was to let the property values themselves constitute the list of already-done/pending backends.  Then it's much easier to ask "is A already fontified everywhere in this block"?

>> So here's an idea.  You could invert the logic, and have a set of
>> `fontified-pending' properties which jit-lock-flush adds to as it sets
>> fontified=nil,
> 
> Yes, of course, we could use the complement set.

The distinct idea here was to map each backend to an individual property, in place of the idea of a single property holding a list of already-done or pending backends, with the aim of significantly reducing property management costs.  That's really just an implementation detail though.  

I think your concern of backend priority and the related issue of how after-change and contextual refontification are handled is probably more important to sort out.

[-- Attachment #2: Type: text/html, Size: 24535 bytes --]

bug#71345: Feature: unleash font-lock's secret weapon; handle Qfontified = non-nil

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> In other words, for many cases there would in fact not be much
> property management work.  This leads naturally to considering more
> complicated cases, with several additional fontification functions all
> interoperating.  The property work will grow quickly (though I also
> outlined some ideas to keep it under control, which probably already
> occurred to you).

In theory, yes, but in practice it's not clear that's relevant.
Also, I think it's worth distinguishing the API from its implementation.

The question being whether the API is flexible enough for clients to
clearly express their needs while at the same time making it possible
for jit-lock to provide the functionality efficiently enough without
resorting to an oracle (even though the implementation at any given time
may suffer inefficiencies in some cases).

> As long as the additional property management costs are well below the
> savings you reap from not having repeated the unnecessary work, this would
> be a positive outcome.  The 2-5ms I mention is the cost for me of running
> "one large backend" over one chunk — namely font-lock-fontify-region with
> treesitter backing.  In my scenario of bar updates resulting from point
> motion, this represents purely wasted work.  So if the additional "property
> management" costs per chunk are, say, 100x below that, you are safely in
> "well worth it" territory.

That probably depends if your position-dependent backend does just

    (jit-lock-flush (point-min) (point-max) #'my-pd-backend)

or on the contrary if it's careful to do something like

    (jit-lock-flush (min my-pd-old-beg (my-pd-new-beg-estimate))
                    (max my-pd-old-end (my-pd-new-end-estimate))
                    #'my-pd-backend)

> Agreed that could be an issue.  In practice keyword-based fontification can
> lead to these same sorts of conflicts for non trivial FACE forms too.
> So backends would need to ensure the changes they are making in the buffer
> are interoperable with the other likely backends (in particular font-lock).

Because a given buffer can have several (window-)points,
position-dependent highlighting will ideally want to be added via
(window-specific) overlays rather than text-properties.

> This means other backends would still need to add to
> font-lock-extra-managed-props any unusual properties they will apply
> (or do the equivalent on their own during unfontify).

No: `font-lock-extra-managed-props` is a variable that belongs to
font-lock, not jit-lock.  `font-lock` is *one* backend of jit-lock.
Other backends will have to use something else (and should use other
text-properties or should use overlays (or make sure font-lock is not
used), otherwise you'll get into conflicts with font-lock).

If we want to allow backends that are not independent (e.g. your PD
highlighting that has to run after font-lock), then it make the API more
complex, since `jit-lock-flush` on the font-lock backend would have to
know to also flush the PD backend.

> '(F) '(F A) '(F B) '(F C) '(F A B)  '(F A C) '(F B C)  '(F A B C)
> So jit-lock-fontify-now's job has gotten quite challenging, as it decides
> over what region to apply a particular backend, say A.

I think the code would loop over chunks of text where the property is
`eq` (e.g. using `next-single-property-change`).  Then within each such
chunk of text it would iterate over either all backends (and skip
those mentioned in the `already-fontified` text-property) or over the
`pending` property backends.  That seems easy enough.


        Stefan

bug#71345: Feature: unleash font-lock's secret weapon; handle Qfontified = non-nil

[JD Smith]

[-- Attachment #1: Type: text/plain, Size: 4302 bytes --]



> On Jun 4, 2024, at 5:52 PM, Stefan Monnier <monnier@iro.umontreal.ca> wrote:

> Also, I think it's worth distinguishing the API from its implementation.

Fair point.  

> or on the contrary if it's careful to do something like
> 
>    (jit-lock-flush (min my-pd-old-beg (my-pd-new-beg-estimate))
>                    (max my-pd-old-end (my-pd-new-end-estimate))
>                    #'my-pd-backend)

I don't need to estimate, I have treesitter to tell me precisely!

>> Agreed that could be an issue.  In practice keyword-based fontification can
>> lead to these same sorts of conflicts for non trivial FACE forms too.
>> So backends would need to ensure the changes they are making in the buffer
>> are interoperable with the other likely backends (in particular font-lock).
> 
> Because a given buffer can have several (window-)points,
> position-dependent highlighting will ideally want to be added via
> (window-specific) overlays rather than text-properties.

In general, yes.  In my case having the scope be per-buffer not per window makes the most sense in terms of the functionality.

>> This means other backends would still need to add to
>> font-lock-extra-managed-props any unusual properties they will apply
>> (or do the equivalent on their own during unfontify).
> 
> No: `font-lock-extra-managed-props` is a variable that belongs to
> font-lock, not jit-lock.  `font-lock` is *one* backend of jit-lock.

I tend to associate font-lock and after-change in my head, but that's backwards.  Presumably after-change would clobber fontified and whatever additional properties jit-lock uses to track its backends.  So font-lock has no special status.  Perhaps I tend to think of it as special because it touches so many locations in the buffer.

> Other backends will have to use something else (and should use other
> text-properties or should use overlays (or make sure font-lock is not
> used), otherwise you'll get into conflicts with font-lock).

Sometimes those are unavoidable, if you need to add some 'face on top of what font-lock did.

> If we want to allow backends that are not independent (e.g. your PD
> highlighting that has to run after font-lock), then it make the API more
> complex, since `jit-lock-flush` on the font-lock backend would have to
> know to also flush the PD backend

I think it will be limiting to insist that all backends must be fully orthogonal (i.e. can apply in any arbitrary order), so some sort of priority or ordering system seems important.  That said, mine only has to run over font-lock because of multi-line strings/comments, and the inability to set an independent 'face property.

It's actually too bad font-lock doesn't itself use font-lock-face for much, despite what the docs say [1].  Since it sets 'face directly, it overrides any other property alias for 'face.  If it instead used 'font-lock-face (or left that to modes, and used some higher-ranked 'font-lock-priority-face), other backends could use their own 'alternate-face, and the order in char-property-alias-alist would set the priority.  Then it would be much easier to make face-using backends more orthogonal.

Regarding the priority of backends, while looking through jit-lock, I was reminded of the completion-at-point-function's API, and the ability to claim :exclusive access, halting calls to further functions on the list.  That's a strong power, but if used wisely, could be effective.

> I think the code would loop over chunks of text where the property is
> `eq` (e.g. using `next-single-property-change`).  Then within each such
> chunk of text it would iterate over either all backends (and skip
> those mentioned in the `already-fontified` text-property) or over the
> `pending` property backends.  That seems easy enough.

But those may be small regions which are inefficient to work on.  This as you say is an implementation detail so probably not important yet.

JD

[1] From `Search-based Fontification': 
If it is ‘prepend’, the face specified by FACESPEC is added to the beginning of the ‘font-lock-face’ property.  If it is ‘append’, the face is added to the end of the ‘font-lock-face’ property.
But I believe all font-lock keywords actually just set the 'face property.

[-- Attachment #2: Type: text/html, Size: 6181 bytes --]

bug#71345: Feature: unleash font-lock's secret weapon; handle Qfontified = non-nil

[Eli Zaretskii]

> Cc: 71345@debbugs.gnu.org, Dmitry Gutov <dmitry@gutov.dev>
> Date: Tue, 04 Jun 2024 17:52:31 -0400
> From:  Stefan Monnier via "Bug reports for GNU Emacs,
>  the Swiss army knife of text editors" <bug-gnu-emacs@gnu.org>
> 
> Because a given buffer can have several (window-)points,
> position-dependent highlighting will ideally want to be added via
> (window-specific) overlays rather than text-properties.

Not sure I understand how this remark is relevant to the issue
discussed here, but let me just point out that when redisplay starts
working on a window, it temporarily moves point to the window-point
position.  So position-dependent highlighting will behave in each
window according to its window-point, which I think is what's expected
here?

bug#71345: Feature: unleash font-lock's secret weapon; handle Qfontified = non-nil

[Eli Zaretskii]

> Cc: 71345@debbugs.gnu.org
> From: JD Smith <jdtsmith@gmail.com>
> Date: Tue, 4 Jun 2024 18:41:01 -0400
> 
>     (jit-lock-flush (min my-pd-old-beg (my-pd-new-beg-estimate))
>                     (max my-pd-old-end (my-pd-new-end-estimate))
>                     #'my-pd-backend)
> 
> I don't need to estimate, I have treesitter to tell me precisely!

Only if the parser successfully and correctly parses the buffer text.

>  Because a given buffer can have several (window-)points,
>  position-dependent highlighting will ideally want to be added via
>  (window-specific) overlays rather than text-properties.
> 
> In general, yes.  In my case having the scope be per-buffer not per window makes the most sense in terms of
> the functionality.

Given the feature of redisplay I just mentioned, you will actually get
a per-window functionality, at least as far as the position of point
is concerned.

bug#71345: Feature: unleash font-lock's secret weapon; handle Qfontified = non-nil

[JD Smith]

[-- Attachment #1: Type: text/plain, Size: 1031 bytes --]


>> In general, yes.  In my case having the scope be per-buffer not per window makes the most sense in terms of
>> the functionality.
> 
> Given the feature of redisplay I just mentioned, you will actually get
> a per-window functionality, at least as far as the position of point
> is concerned.

Not in my case, since I am discriminating between point in the selected window and other windows showing the same buffer, such that switching windows = changing point. Since much of the work in my mode is not position-dependent but depends on text content, updating face is the natural thing.  

Thinking more about what makes font-lock “special” as a jit-lock backend, it is exactly this: font-lock claims exclusive ownership of `face'.  Solutions I can see:
Make font-lock a good citizen by using its own alias for face. 
Give other jit-lock backends which may alter face and potentially conflict with font-lock the ability to specify to jit-lock “if you re-run font lock on a region, re-run me too after that.”

[-- Attachment #2: Type: text/html, Size: 1711 bytes --]

bug#71345: Feature: unleash font-lock's secret weapon; handle Qfontified = non-nil

[JD Smith]

> On Jun 5, 2024, at 7:24 AM, Eli Zaretskii <eliz@gnu.org> wrote:
> 
> position-dependent highlighting will behave in each
> window according to its window-point, which I think is what's expected
> here?

In general that would be true. In my case I use a timer-protected post-command hook to update the (single per buffer) containing treesitter node. 

bug#71345: Feature: unleash font-lock's secret weapon; handle Qfontified = non-nil

[Eli Zaretskii]

> From: JD Smith <jdtsmith@gmail.com>
> Cc: monnier@iro.umontreal.ca, 71345@debbugs.gnu.org
> Date: Wed, 5 Jun 2024 10:02:23 -0400
> 
>  In general, yes.  In my case having the scope be per-buffer not per window makes the most sense in
>  terms of
> 
>  the functionality.
> 
>  Given the feature of redisplay I just mentioned, you will actually get
> 
>  a per-window functionality, at least as far as the position of point
> 
>  is concerned.
> 
> Not in my case, since I am discriminating between point in the selected window and other windows showing the
> same buffer, such that switching windows = changing point. Since much of the work in my mode is not
> position-dependent but depends on text content, updating face is the natural thing.  

Sorry, I don't think I understand you.  But if the fact that redisplay
moves point to window-point doesn't bother me, we can drop this issue.

> Thinking more about what makes font-lock “special” as a jit-lock backend, it is exactly this: font-lock claims
> exclusive ownership of `face'.  Solutions I can see:
> 
> 1 Make font-lock a good citizen by using its own alias for face. 
> 2 Give other jit-lock backends which may alter face and potentially conflict with font-lock the ability to specify
>  to jit-lock “if you re-run font lock on a region, re-run me too after that.”

I don't understand.  The solution to this dilemma is well known: Lisp
programs that want to control faces without turning off font-lock mode
should set font-lock-face property, instead of the face property.  Why
do you need to find another solution?

bug#71345: Feature: unleash font-lock's secret weapon; handle Qfontified = non-nil

[JD Smith]

[-- Attachment #1: Type: text/plain, Size: 3423 bytes --]



> On Jun 5, 2024, at 10:53 AM, Eli Zaretskii <eliz@gnu.org> wrote:
> 
>> From: JD Smith <jdtsmith@gmail.com>
>> Cc: monnier@iro.umontreal.ca, 71345@debbugs.gnu.org
>> Date: Wed, 5 Jun 2024 10:02:23 -0400
>> 
>> In general, yes.  In my case having the scope be per-buffer not per window makes the most sense in
>> terms of
>> 
>> the functionality.
>> 
>> Given the feature of redisplay I just mentioned, you will actually get
>> 
>> a per-window functionality, at least as far as the position of point
>> 
>> is concerned.
>> 
>> Not in my case, since I am discriminating between point in the selected window and other windows showing the
>> same buffer, such that switching windows = changing point. Since much of the work in my mode is not
>> position-dependent but depends on text content, updating face is the natural thing.  
> 
> Sorry, I don't think I understand you.  But if the fact that redisplay
> moves point to window-point doesn't bother me, we can drop this issue.

I use treesitter to identify nodes of interest, and do that in a PCH.  So there is one buffer-local record of where that node of interest is.  Thus different windows showing the same buffer is immaterial; the window where the last PCH ran controls.

>> Thinking more about what makes font-lock “special” as a jit-lock backend, it is exactly this: font-lock claims
>> exclusive ownership of `face'.  Solutions I can see:
>> 
>> 1 Make font-lock a good citizen by using its own alias for face. 
>> 2 Give other jit-lock backends which may alter face and potentially conflict with font-lock the ability to specify
>> to jit-lock “if you re-run font lock on a region, re-run me too after that.”
> 
> I don't understand.  The solution to this dilemma is well known: Lisp
> programs that want to control faces without turning off font-lock mode
> should set font-lock-face property, instead of the face property.  Why
> do you need to find another solution?

Because such programs may need to override locations where font-lock has already applied 'face.  In contrast to the documentation[1], font-lock does not itself use font-lock-face, but only configures it for other programs to use.  The issue is that 'face, where applied, always outranks any alias to 'face placed on the same text:

>>>>>> 
;; TEST (1st disable font-lock-mode)
(push 'my-face-prop (alist-get 'face char-property-alias-alist))
(put-text-property 1 8 'face 'error)
(put-text-property 1 8 'my-face-prop 'success) ; outranked by 'face
(remove-text-properties 1 8 '(face nil)) ; now we can see my-face-prop
<<<<<<<

It would be wonderful if font-lock cleared and applied a suitable face alias (perhaps 'font-lock-face itself).  This is what I mean by making font-lock a "good citizen": let other backends be able to override the faces it sets.  Currently, if font-lock-mode is enabled, it has effectively exclusive rights to the use of 'face on the locations it matches.  The only hope then is to arrange to run after font-lock, and use the "heavy hammer" of overwriting 'face yourself.

[1] From `Search-based Fontification': 
If it is ‘prepend’, the face specified by FACESPEC is added to the beginning of the ‘font-lock-face’ property.  If it is ‘append’, the face is added to the end of the ‘font-lock-face’ property.
But I believe all font-lock keywords actually just set the 'face property.

[-- Attachment #2: Type: text/html, Size: 9685 bytes --]

bug#71345: Feature: unleash font-lock's secret weapon; handle Qfontified = non-nil

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>> Because a given buffer can have several (window-)points,
>> position-dependent highlighting will ideally want to be added via
>> (window-specific) overlays rather than text-properties.
>
> Not sure I understand how this remark is relevant to the issue
> discussed here, but let me just point out that when redisplay starts
> working on a window, it temporarily moves point to the window-point
> position.  So position-dependent highlighting will behave in each
> window according to its window-point, which I think is what's expected
> here?

But the highlighting is done "once and for all" (at least until the next
command), so if you want it to be different in different windows (to
reflect the different values of `point` in those windows) you'll need
overlays with the `window` property because the highlighting will not be
re-done in the middle of redisplay when we go from one window to another.


        Stefan

bug#71345: Feature: unleash font-lock's secret weapon; handle Qfontified = non-nil

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: jdtsmith@gmail.com,  71345@debbugs.gnu.org,  dmitry@gutov.dev
> Date: Wed, 05 Jun 2024 12:28:01 -0400
> 
> >> Because a given buffer can have several (window-)points,
> >> position-dependent highlighting will ideally want to be added via
> >> (window-specific) overlays rather than text-properties.
> >
> > Not sure I understand how this remark is relevant to the issue
> > discussed here, but let me just point out that when redisplay starts
> > working on a window, it temporarily moves point to the window-point
> > position.  So position-dependent highlighting will behave in each
> > window according to its window-point, which I think is what's expected
> > here?
> 
> But the highlighting is done "once and for all" (at least until the next
> command), so if you want it to be different in different windows (to
> reflect the different values of `point` in those windows) you'll need
> overlays with the `window` property because the highlighting will not be
> re-done in the middle of redisplay when we go from one window to another.

In that case, we are in trouble anyway, because the "once and for all"
highlighting could be (by sheer luck) be done by display code that
doesn't run as part of redisplay, but as part of something else, like
vertical-motion.

bug#71345: Feature: unleash font-lock's secret weapon; handle Qfontified = non-nil

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>> >> Because a given buffer can have several (window-)points,
>> >> position-dependent highlighting will ideally want to be added via
>> >> (window-specific) overlays rather than text-properties.
>> >
>> > Not sure I understand how this remark is relevant to the issue
>> > discussed here, but let me just point out that when redisplay starts
>> > working on a window, it temporarily moves point to the window-point
>> > position.  So position-dependent highlighting will behave in each
>> > window according to its window-point, which I think is what's expected
>> > here?
>> 
>> But the highlighting is done "once and for all" (at least until the next
>> command), so if you want it to be different in different windows (to
>> reflect the different values of `point` in those windows) you'll need
>> overlays with the `window` property because the highlighting will not be
>> re-done in the middle of redisplay when we go from one window to another.
>
> In that case, we are in trouble anyway, because the "once and for all"
> highlighting could be (by sheer luck) be done by display code that
> doesn't run as part of redisplay, but as part of something else, like
> vertical-motion.

I can't see why that would be a problem.

The highlighting code run from `jit-lock` will usually not be able to
use `point` (or `window-point`) directly.  Instead, that highlighter
will need to keep track of the windows' points elsewhere "manually" via
hooks like `post-command-hook` or `pre-redisplay-functions` and then
rely on that info when performing the highlighting.


        Stefan

bug#71345: Feature: unleash font-lock's secret weapon; handle Qfontified = non-nil

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>> 1 Make font-lock a good citizen by using its own alias for face. 
>> 2 Give other jit-lock backends which may alter face and potentially
>> conflict with font-lock the ability to specify
>>  to jit-lock “if you re-run font lock on a region, re-run me too after that.”
>
> I don't understand.  The solution to this dilemma is well known: Lisp
> programs that want to control faces without turning off font-lock mode
> should set font-lock-face property, instead of the face property.  Why
> do you need to find another solution?

Another solution to that dilemma was to use overlays.  🙂


        Stefan

bug#71345: Feature: unleash font-lock's secret weapon; handle Qfontified = non-nil

[Drew Adams via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> > The solution to this dilemma is well known: Lisp
> > programs that want to control faces without
> > turning off font-lock mode should set
> > font-lock-face property, instead of the face 
> > property.  Why do you need to find another solution?
> 
> Another solution to that dilemma was to use overlays.  🙂

Apologies, as I'm not really following this
thread.  My comment might be irrelevant to
the actual issue; if so, sorry.
___

The question of how to prevent some ad hoc
highlighting from being undone or overridden
by font-lock has always been answered by
telling users to use property `font-lock-face',
which, in effect, still gives font-lock control
but tells it to consider such highlighting as
its own.

As far back as 2007 I proposed what library
font-lock+.el offers as another approach (I
provided a patch in bug #18367): putting a
non-nil text property of `font-lock-ignore'
on text that you don't want font-lock to
fiddle with.

https://lists.gnu.org/archive/html/emacs-devel/2022-04/msg00059.html

Neither `font-lock-face' nor
`font-lock-extra-managed-props' does the
same job (see previous discussions).

They just make `font-lock' _also_ manage the
text; e.g., turning off `font-lock-mode' also
turns off highlighting with the property.

The point is not to give font-lock more
control; it's to be able to remove control
by font-lock from given text.

See https://debbugs.gnu.org/cgi/bugreport.cgi?bug=18367#38
___

2014 emacs-devel thread:

https://lists.gnu.org/archive/html/emacs-devel/2014-08/msg00540.html

https://lists.gnu.org/archive/html/emacs-devel/2014-08/msg00583.html

2007 emacs-devel thread:

https://lists.gnu.org/archive/html/emacs-devel/2007-03/msg01459.html

Bug #18367 thread:

https://debbugs.gnu.org/cgi/bugreport.cgi?bug=18367

bug#71345: Feature: unleash font-lock's secret weapon; handle Qfontified = non-nil

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: jdtsmith@gmail.com,  71345@debbugs.gnu.org,  dmitry@gutov.dev
> Date: Wed, 05 Jun 2024 12:59:08 -0400
> 
> >> But the highlighting is done "once and for all" (at least until the next
> >> command), so if you want it to be different in different windows (to
> >> reflect the different values of `point` in those windows) you'll need
> >> overlays with the `window` property because the highlighting will not be
> >> re-done in the middle of redisplay when we go from one window to another.
> >
> > In that case, we are in trouble anyway, because the "once and for all"
> > highlighting could be (by sheer luck) be done by display code that
> > doesn't run as part of redisplay, but as part of something else, like
> > vertical-motion.
> 
> I can't see why that would be a problem.
> 
> The highlighting code run from `jit-lock` will usually not be able to
> use `point` (or `window-point`) directly.  Instead, that highlighter
> will need to keep track of the windows' points elsewhere "manually" via
> hooks like `post-command-hook` or `pre-redisplay-functions` and then
> rely on that info when performing the highlighting.

You say you don't see a problem, but then describe a solution for a
problem you don't see? ;-)

Yes, if the code run from jit-lock will use some private variable
instead of point, it can get away, but such a code will need to be
written in an extremely disciplined manner, like nothing else in Emacs
(except, perhaps, the inner parts of redisplay code in C), because
what is more natural than call functions and APIs under the assumption
that point is where it's expected to be?

bug#71345: Feature: unleash font-lock's secret weapon; handle Qfontified = non-nil

[JD Smith]

> On Jun 5, 2024, at 1:52 PM, Eli Zaretskii <eliz@gnu.org> wrote:
> 
>> From: Stefan Monnier <monnier@iro.umontreal.ca>
>> Cc: jdtsmith@gmail.com,  71345@debbugs.gnu.org,  dmitry@gutov.dev
>> Date: Wed, 05 Jun 2024 12:59:08 -0400
>> 
>>>> But the highlighting is done "once and for all" (at least until the next
>>>> command), so if you want it to be different in different windows (to
>>>> reflect the different values of `point` in those windows) you'll need
>>>> overlays with the `window` property because the highlighting will not be
>>>> re-done in the middle of redisplay when we go from one window to another.
>>> 
>>> In that case, we are in trouble anyway, because the "once and for all"
>>> highlighting could be (by sheer luck) be done by display code that
>>> doesn't run as part of redisplay, but as part of something else, like
>>> vertical-motion.
>> 
>> I can't see why that would be a problem.
>> 
>> The highlighting code run from `jit-lock` will usually not be able to
>> use `point` (or `window-point`) directly.  Instead, that highlighter
>> will need to keep track of the windows' points elsewhere "manually" via
>> hooks like `post-command-hook` or `pre-redisplay-functions` and then
>> rely on that info when performing the highlighting.
> 
> You say you don't see a problem, but then describe a solution for a
> problem you don't see? ;-). Yes, if the code run from jit-lock will use some private variable
> instead of point, it can get away, 

But you must use a PCH or related hook if you want point movement to be reflected in real time (or after a short delay, for me).  Since jit-lock is never driven by changes in point, only by fontified=nil appearing somewhere in the text, I don't see how point-dependent highlighting could ever be achieved other than via a command hook saving to a private variable of some kind.  

> but such a code will need to be
> written in an extremely disciplined manner, like nothing else in Emacs
> (except, perhaps, the inner parts of redisplay code in C), because
> what is more natural than call functions and APIs under the assumption
> that point is where it's expected to be?


In practice I am doing this already. It works quite well and is not too onerous to keep track of.  You basically:

- start a timer in a PCH
- once it fires, query treesitter for the enclosing node of interest at point
- if that node has changed, save the new node boundaries and flush the union of the old and new nodes
- font-lock does the rest

The only problem I'm really facing is that last step: outside of actual text changes, font-lock is wastefully refontifying everything, when only a small amount of work was needed.  Stefan has a very good idea for a cooperative jit-lock capability, where multiple jit-lock backends can sometimes do their work independently, which could in principle solve this.

bug#71345: Feature: unleash font-lock's secret weapon; handle Qfontified = non-nil

[JD Smith]

Brief followup.  I took Stefan's concept of multiple jit-lock backends taking turns with the help of additional "marker" properties and, as a test of the idea, implemented a bespoke version, as follows:

    • I have two "backends": normal font-lock (syntactic + keyword), and my own bar drawing.

    • Rather than "already-done" I used the complementary "pending" approach.  Since I have only two backends, and one is always run, only one needs to indicate it is "pending".

    • Merely for convenience, to implement the bar-drawing backend, I wrapped the `font-lock-fontify-region-function' with a function [1] that can run (or, if indicated, inhibit) font-lock, after which it always draws indent bars.  Basically I stuffed two backends into one slot.

    • To determine when font-lock is needed for a region, I had to advise `jit-lock-context' and provide a `jit-lock-after-change-extend-region-function', both of which simply serve to mark text requested for font-locking with an `indent-bars-font-lock-pending' property.

    • The presence of that property on a region is then the sign that font-lock must be run [2].

This is clearly a non-generalizable approach, but in my testing it works very well to reduce unnecessary refontification to a minimum.  It also actually simplified and shortened the code somewhat.  I'm now confident that a multi-jit-lock capability that allows for expressing dependencies like the above would work well.

A few relevant things I learned:

    • Backends need a way to opt in or out of contextual fontification.

    • Since several backends may operate during one redisplay cycle (e.g. imagine font-lock updated by an after-change + scope altered by a PCH), backends must mark their pendency defensively, and assume nothing about other backend's status.

    • I "hard-coded" the dependency between my backends by always drawing the bars after possibly running font-lock.  A more general solution of expressing such dependencies would be preferable.

One idea for the latter: backends can indicate which text-properties they intend to alter, and, when a backend needs to be run on a region, all others which share properties with it and appear _after_ it on the `jit-lock-functions' list will also be run.  So I'd express this as:

jit-lock-functions = (font-lock-fontify-region my-backend-fontify-region)
jit-lock-function-properties = ((font-lock-fontify-region face) 
                                              (my-backend-fontify-region face my-display-alias))

Potential flaws in my approach:

    • Advising `jit-lock-context' is not ideal, but that function currently provides no hooks for backends.

    • This basically hard-codes two backends together underneath font-lock.  Whether and how it would work with other jit-lock backends remains to be seen. 

    • Maintaining my own properties to determine when to run each backend is not so bad, but having this handled automatically would be simpler.

    • Technically I don't need to refontify on context-refontification, but because font-lock and I both use 'face, I do.  I wish both of us could switch to aliases (I do use a custom 'display alias where needed).

So, to summarize, Stefan's idea of re-using fontified=nil without any changes to redisplay for multiple (in)dependent backends, managed by jit-lock using other text-properties, seems very tractable and likely to be quite useful, if the API can be clearly expressed and documented.  

Very happy to assist with this, for example by "porting" my mode to use in-development versions.

[1] https://github.com/jdtsmith/indent-bars/blob/ea0cd3de21cac3c7bbfd0a0cd8fc1cfa58469290/indent-bars.el#L1190-L1205
[2] https://github.com/jdtsmith/indent-bars/blob/ea0cd3de21cac3c7bbfd0a0cd8fc1cfa58469290/indent-bars-ts.el#L394-L402