bug#66732: tree-sitter fontification doesn't update multi-line syntax reliably

[Dmitry Gutov <dmitry@gutov.dev>]

On 16/12/2023 19:11, Stefan Monnier wrote:
>> Hmm... This `put-text-property` can't be right.
>> Can you trace `font-lock-fontify-region` and
>> `font-lock-default-fontify-region` to check that they correctly return
>> the proper (extended) bounds of the region they (re)fontified?
> 
> BTW, using the extend-region-function isn't ideal: this is meant for
> things which can't be refontified separately, so if a change in the
> buffer causes notification to make changes over a large portion of the
> buffer we'll end up (re)fontified *right away* that whole large
> portion even if only a small part (or even no part at all) is displayed.

I think most of the time the parser notifier will send ranges of limited 
size (as one or other piece of text gets recognized as e.g. a comment, 
or a raw string -- the bounds of the respective node that changed).

> IOW, it'd be better to mark the changed chunks with this
> `put-text-property`, but to do it before jit&font-lock get triggered.
> 
> Basically, we'd like to call it from `after-change-functions`, but this
> can be called *many* times within a single command, so we want to delay
> it.  So we can probably just set a flag that says "there are unprocessed
> changes" and then sprinkle calls to a "lazy update" function which
> checks this flag before calling `treesit--font-lock-notifier` (or
> something similar).

I thought doing this lazily (inside font-lock-default-fontify-region), 
only when the region is set to be redisplayed, would be more economical 
than doing more work inside after-change-functions.

> One of the places where we'd need the sprinkle would be
> `pre-redisplay-functions`.  Another one might be `syntax-ppss`?

syntax-ppss has syntax-propertize-extend-region-functions. I haven't 
found any better suitable hooks than that.