bug#66732: tree-sitter fontification doesn't update multi-line syntax reliably

[Dmitry Gutov <dmitry@gutov.dev>]

On 16/12/2023 07:56, Yuan Fu wrote:

>> Hmm, let me have a closer look tomorrow. It can’t be that difficult, we must have missed something.
> 
> Please see this patch, which basically does what you did in your patch. I just started from scratch when experimenting. I didn’t do the context thing, that’s a bit confusing for me.

Yeah, it's more complex than I'd have liked but basically the "context" 
thing was to dispatch to the right invalidator: for font-lock and/or for 
syntax. Whether to return the range using the corresponding 
extend-function, or to invalidate push-style (when the notifier is 
called outside of any extend-function's).

I've tried your patch, and it seemed to have similar problems to mine. 
In particular, that the comment is not unfontified right away when the 
closing slash is removed.

Also, I think we're targeting Emacs 29 for this bug (the patch needed 
adjustments).

> I save updated ranges in a local hash table and access it in extend-region functions to extend the region. Ideally treesit should have a function that returns the updated ranges for a parser, we don’t have that yet, so I’m using the hash table to simulate it.

Okay. I wonder how other users (which care about specific parsers) would 
look.

> I found that if you don’t set text prop fontified to nil for the whole extended region, redisplay doesn’t seem to, well, redisplay the full region, even thought the new face has been correctly applied to them.

That sounds like a bug in font-lock? At the end of jit-lock-fontify-now, 
there is a call creating a timer with jit-lock-force-redisplay.

And that function ends with this:

     ;; Don't cause refontification (it's already been done), but just do
     ;; some random buffer change, so as to force redisplay.
     (put-text-property start end 'fontified t)))))

If I just change t to nil there (or to some other value, like 42), 
either of our patches starts behaving well. Perhaps Stefan could comment.