bug#66732: tree-sitter fontification doesn't update multi-line syntax reliably

[Dmitry Gutov <dmitry@gutov.dev>]

On 14/12/2023 10:29, Yuan Fu wrote:
> 
> 
>> On Dec 13, 2023, at 5:43 PM, Dmitry Gutov <dmitry@gutov.dev> wrote:
>>
>> On 13/12/2023 05:28, Yuan Fu wrote:
>>>>> c-ts-mode--emacs-set-ranges is registered as a range rule, so many tree-sitter function calls it before doing anything to make sure range is up-to-date. treesit-font-lock-fontify-region calls treesit-update-ranges at the beginning of its body, and treesit-update-ranges calls c-ts-mode--emacs-set-ranges.
>>>>
>>>> That seems to mean that any feature accessing the parse tree should call treesit-update-ranges first. Including syntax-propertize-functions and *-extend-region-functions, which we currently don't do.
>>>>
>>>> But which boundaries is it supposed to use? Should treesit--syntax-extend-region call treesit-update-ranges, when wait for the parser updates, then possibly call treesit-update-ranges again on the extended boundaries, and so on, until the parser stops sending notifications? Will it stop?
>>> It'll stop. If the ranges don't change, no reparse will happen. And only buffer content change can cause range change. Reparse itself can't. So at most you'll see reparse -> update ranges -> reparse.
>>>> Anyway, could you try my patch? Like I said, I'm not sure if the insufficient fontification I'm observing in c-ts-mode is due to the problem with the solution, or due to the other redisplay-related problems on my system.
>>> Yeah, it doesn't solve the problem in c-ts-mode regarding block comments.
>>> We might need to run (progn (force-parse) (update-ranges) (force-parse)) before jit-lock-fontify-now and sytax-ppss.
>>
>> Well... I've replaced
>>
>>   (treesit-buffer-root-node (treesit-language-at (point)))
>>
>> in treesit--syntax-extend-region with
>>
>>   (treesit-buffer-root-node (treesit-language-at (point)))
>>   (treesit-update-ranges beg end)
>>   (treesit-buffer-root-node (treesit-language-at (point)))
>>
>> and even tried commenting out the call to 'treesit-update-ranges' inside treesit-font-lock-fontify-region, but the result looks unchanged. And the region does get extended, according to my print-debugging inside font-lock-default-fontify-region.
>>
>> Could you check if you're seeing the same?
> 
> It should be
> 
> (treesit-buffer-root-node 'emacs-c)
> (treesit-update-ranges)
> (treesit-buffer-root-node ‘c)

This might be difficult for treesit--syntax-extend-region to do, since 
it's supposed to work across modes. But I suppose it could iterate 
through (mapcar #'treesit-parser-language (treesit-parser-list)).

> I tried, and it doesn’t make a difference. But I must admit I’m not very clear on what your patch tries to do. Does it try to extend the to-be-fontified region before jit-lock runs?

Yes, it extends the region to-be-fontified. If you set 
treesit--font-lock-verbose to t, you should see the appropriate messages 
"Fontifying region: x-y" in the message log where x is the position 
before the start of the comment (printed by 
treesit-font-lock-fontify-region).