bug#66732: tree-sitter fontification doesn't update multi-line syntax reliably

[Dmitry Gutov <dmitry@gutov.dev>]

On 16/12/2023 19:43, Stefan Monnier wrote:
>> I think most of the time the parser notifier will send ranges of limited
>> size (as one or other piece of text gets recognized as e.g. a comment, or
>> a raw string -- the bounds of the respective node that changed).
> 
> Yes, of course, in practice it will usually work OK.
> I'm talking about what should happen ideally so that we can make an
> informed decision about the code we decide to use in the end, knowing
> its downsides.

I figured that it would be useful at least to start with the same 
solution for both font-lock and syntax, because bugs in the former are 
usually easier to notice earlier.

But you have a point that doing invalidation earlier will mean that 
font-lock won't have to refontify the whole big region (whenever that 
actually happens in practice), only the visible part.

>>> One of the places where we'd need the sprinkle would be
>>> `pre-redisplay-functions`.  Another one might be `syntax-ppss`?
>> syntax-ppss has syntax-propertize-extend-region-functions. I haven't found
>> any better suitable hooks than that.
> 
> Indeed, I don't think there's a good hook for that yet, tho maybe
> `syntax-propertize-extend-region-functions` is good enough (basically
> make it call `treesit--font-lock-notifier` but return nil), assuming we
> have set a `syntax-propertize-function`.

Solutions for syntax and font-lock will likely need to work independently:

- syntax-ppss can be called outside of font-lock (e.g. by electric-pair)
- font-lock can work without syntax-propertize (e.g. when 
syntax-propertize-function is nil), and even when it does get called, it 
can happen somewhere inside font-lock highlighting code. E.g. by the 
time we reach treesit-font-lock-fontify-region, the region to refontify 
has already been decided.