bug#71345: Feature: unleash font-lock's secret weapon; handle Qfontified = non-nil

[Stefan Monnier via "Bug reports for GNU Emacs, the Swiss army knife of text editors" <bug-gnu-emacs@gnu.org>]

> In other words, for many cases there would in fact not be much
> property management work.  This leads naturally to considering more
> complicated cases, with several additional fontification functions all
> interoperating.  The property work will grow quickly (though I also
> outlined some ideas to keep it under control, which probably already
> occurred to you).

In theory, yes, but in practice it's not clear that's relevant.
Also, I think it's worth distinguishing the API from its implementation.

The question being whether the API is flexible enough for clients to
clearly express their needs while at the same time making it possible
for jit-lock to provide the functionality efficiently enough without
resorting to an oracle (even though the implementation at any given time
may suffer inefficiencies in some cases).

> As long as the additional property management costs are well below the
> savings you reap from not having repeated the unnecessary work, this would
> be a positive outcome.  The 2-5ms I mention is the cost for me of running
> "one large backend" over one chunk — namely font-lock-fontify-region with
> treesitter backing.  In my scenario of bar updates resulting from point
> motion, this represents purely wasted work.  So if the additional "property
> management" costs per chunk are, say, 100x below that, you are safely in
> "well worth it" territory.

That probably depends if your position-dependent backend does just

    (jit-lock-flush (point-min) (point-max) #'my-pd-backend)

or on the contrary if it's careful to do something like

    (jit-lock-flush (min my-pd-old-beg (my-pd-new-beg-estimate))
                    (max my-pd-old-end (my-pd-new-end-estimate))
                    #'my-pd-backend)

> Agreed that could be an issue.  In practice keyword-based fontification can
> lead to these same sorts of conflicts for non trivial FACE forms too.
> So backends would need to ensure the changes they are making in the buffer
> are interoperable with the other likely backends (in particular font-lock).

Because a given buffer can have several (window-)points,
position-dependent highlighting will ideally want to be added via
(window-specific) overlays rather than text-properties.

> This means other backends would still need to add to
> font-lock-extra-managed-props any unusual properties they will apply
> (or do the equivalent on their own during unfontify).

No: `font-lock-extra-managed-props` is a variable that belongs to
font-lock, not jit-lock.  `font-lock` is *one* backend of jit-lock.
Other backends will have to use something else (and should use other
text-properties or should use overlays (or make sure font-lock is not
used), otherwise you'll get into conflicts with font-lock).

If we want to allow backends that are not independent (e.g. your PD
highlighting that has to run after font-lock), then it make the API more
complex, since `jit-lock-flush` on the font-lock backend would have to
know to also flush the PD backend.

> '(F) '(F A) '(F B) '(F C) '(F A B)  '(F A C) '(F B C)  '(F A B C)
> So jit-lock-fontify-now's job has gotten quite challenging, as it decides
> over what region to apply a particular backend, say A.

I think the code would loop over chunks of text where the property is
`eq` (e.g. using `next-single-property-change`).  Then within each such
chunk of text it would iterate over either all backends (and skip
those mentioned in the `already-fontified` text-property) or over the
`pending` property backends.  That seems easy enough.


        Stefan