bug#71345: Feature: unleash font-lock's secret weapon; handle Qfontified = non-nil

[JD Smith <jdtsmith@gmail.com>]

> On Jun 5, 2024, at 1:52 PM, Eli Zaretskii <eliz@gnu.org> wrote:
> 
>> From: Stefan Monnier <monnier@iro.umontreal.ca>
>> Cc: jdtsmith@gmail.com,  71345@debbugs.gnu.org,  dmitry@gutov.dev
>> Date: Wed, 05 Jun 2024 12:59:08 -0400
>> 
>>>> But the highlighting is done "once and for all" (at least until the next
>>>> command), so if you want it to be different in different windows (to
>>>> reflect the different values of `point` in those windows) you'll need
>>>> overlays with the `window` property because the highlighting will not be
>>>> re-done in the middle of redisplay when we go from one window to another.
>>> 
>>> In that case, we are in trouble anyway, because the "once and for all"
>>> highlighting could be (by sheer luck) be done by display code that
>>> doesn't run as part of redisplay, but as part of something else, like
>>> vertical-motion.
>> 
>> I can't see why that would be a problem.
>> 
>> The highlighting code run from `jit-lock` will usually not be able to
>> use `point` (or `window-point`) directly.  Instead, that highlighter
>> will need to keep track of the windows' points elsewhere "manually" via
>> hooks like `post-command-hook` or `pre-redisplay-functions` and then
>> rely on that info when performing the highlighting.
> 
> You say you don't see a problem, but then describe a solution for a
> problem you don't see? ;-). Yes, if the code run from jit-lock will use some private variable
> instead of point, it can get away, 

But you must use a PCH or related hook if you want point movement to be reflected in real time (or after a short delay, for me).  Since jit-lock is never driven by changes in point, only by fontified=nil appearing somewhere in the text, I don't see how point-dependent highlighting could ever be achieved other than via a command hook saving to a private variable of some kind.  

> but such a code will need to be
> written in an extremely disciplined manner, like nothing else in Emacs
> (except, perhaps, the inner parts of redisplay code in C), because
> what is more natural than call functions and APIs under the assumption
> that point is where it's expected to be?


In practice I am doing this already. It works quite well and is not too onerous to keep track of.  You basically:

- start a timer in a PCH
- once it fires, query treesitter for the enclosing node of interest at point
- if that node has changed, save the new node boundaries and flush the union of the old and new nodes
- font-lock does the rest

The only problem I'm really facing is that last step: outside of actual text changes, font-lock is wastefully refontifying everything, when only a small amount of work was needed.  Stefan has a very good idea for a cooperative jit-lock capability, where multiple jit-lock backends can sometimes do their work independently, which could in principle solve this.