bug#66732: tree-sitter fontification doesn't update multi-line syntax reliably

[Dmitry Gutov <dmitry@gutov.dev>]

On 19/12/2023 05:12, Yuan Fu wrote:
> 
>> On Dec 17, 2023, at 10:32 AM, Dmitry Gutov<dmitry@gutov.dev>  wrote:
>>
>> On 17/12/2023 03:16, Yuan Fu wrote:
>>> +(defun treesit--pre-syntax-ppss (&rest _)
>>> +  "Force reparse and consequently run all notifiers.
>>> +
>>> +Similar to font-lock, we want to update the `syntax' text
>>> +property before `syntax-ppss' starts working on the text."
>>> +  (treesit--pre-redisplay)
>>> +  nil)
>> This callback might be the best place to inform syntax-propertize about the extended bounds which need to be re-parsed, so I'm not sure about Stefan's suggestion to return nil here.
>>
>> Even calling syntax-ppss-flush-cache wouldn't be a full solution, because that wouldn't update the value of START in syntax-propertize.
> Makes sense. Then I think a treeist function that returns the latest affected regions is in order?

But latest compared to what? To the previous invocation of the parser?

What if the current caller uses this function and expects to see changes 
since time X, but some other feature already instantiated the parser 
sometime between X and NOW? The callbacks which would run now would send 
notifications compared to that other previous (unknown) invocation. 
There could be some system which keeps checkpoints that the callers 
could reference, but I'm not sure what's the best shape. And how the 
older history, not needed by any callers anymore, would be evicted (some 
weak hash map could work, but any caller that's not careful could create 
leaks...).

And without the above, we'll need a separate callback which would call 
syntax-ppss-flush-cache when the parser sends notifcations while invoked 
outside of syntax-propertize-extend-region-functions. And then its code 
will have to care about context anyway (inside s-p-e-r-f or not).