bug#66732: tree-sitter fontification doesn't update multi-line syntax reliably

[Yuan Fu <casouri@gmail.com>]

On 12/11/23 7:53 AM, Dmitry Gutov wrote:
> On 11/12/2023 06:16, Yuan Fu wrote:
>> The way we achieves this is thought parser notifiers. When 
>> tree-sitter parser reparses, it notifies us of which part of the 
>> buffer was affected by the reparse. For font-lock, we have this 
>> font-lock notifier that marks the affected buffer region as "not 
>> fontified", so redisplay will refontify those areas.
>>
>> (defun treesit--font-lock-notifier (ranges parser)
>>    "Ensures updated parts of the parse-tree are refontified.
>> RANGES is a list of (BEG . END) ranges, PARSER is the tree-sitter
>> parser notifying of the change."
>>    (with-current-buffer (treesit-parser-buffer parser)
>>      (dolist (range ranges)
>>        (when treesit--font-lock-verbose
>>          (message "Notifier received range: %s-%s"
>>                   (car range) (cdr range)))
>>        (with-silent-modifications
>>          (put-text-property (car range) (cdr range) 'fontified nil)))))
>>
>> This notifier function will be called during redisplay [1]. I suspect 
>> that because of this timing, redisplay doesn't refontify the marked 
>> region immediately. So I added a timer, I think that ensures we mark 
>> the affected region in the next command loop?
>>
>> (defun treesit--font-lock-notifier (ranges parser)
>>    "Ensures updated parts of the parse-tree are refontified.
>> RANGES is a list of (BEG . END) ranges, PARSER is the tree-sitter
>> parser notifying of the change."
>>    (with-current-buffer (treesit-parser-buffer parser)
>>      (dolist (range ranges)
>>        (when treesit--font-lock-verbose
>>          (message "Notifier received range: %s-%s"
>>                   (car range) (cdr range)))
>>        (run-with-timer
>>         0 nil
>>         (lambda ()
>>           (with-silent-modifications
>>             (put-text-property (car range) (cdr range)
>>                                'fontified nil)))))))
>>
>> This seems to work. Eli, do you see any problem using run-with-timer 
>> this way? What's the correct way to mark some region unfontified?
>>
>> [1] The chain of events if roughly: user types the last "/" -> 
>> redisplay -> fontify that character -> access parser -> parser 
>> reparses -> calls notifier.
>
> Note that it's not just font-lock. syntax-propertize has the same 
> problem (I've described it in https://debbugs.gnu.org/67262#23).
>
> And a timer wouldn't help because syntax-ppss needs to have up-to-date 
> information whenever it's called, not later.
>

I feel that font-lock and syntax-ppss have different problems and 
requires different solutions. For font-lock, we need to mark affected 
range unfontified; we just need to make sure we do that after 
jit-lock-fontify-now. For syntax-ppss, we need to force a reparse before 
entering syntax-ppss so that syntax text properties are up-to-date when 
syntax-ppss do its work.
> Here's a draft solution based on *-extend-region-functions, attached.
>
> Alas, while it works fine in python-ts-mode (for both syntax and 
> font-lock), making it behave better than python-mode, in c-ts-mode it 
> doesn't quite have the same effect: when you backspace over the 
> closing "/", the highlighting is properly updated only after you make 
> the next edit (any edit), or select another window. I'm not sure, 
> though, if it's due to my own problems with Emacs's failure to 
> redisplay (reported elsewhere), so more testing is welcome.
>
> But that might also be related to the use of 
> c-ts-mode--emacs-set-ranges: printing a backtrace calls inside 
> treesit--font-lock-notifier shows that the last notification comes 
> also during font-lock but after treesit--font-lock-extend-region, 
> inside c-ts-mode--emacs-set-ranges. I don't quite understand this 
> design where the ranges are applied inside the font-lock code.
c-ts-mode--emacs-set-ranges is registered as a range rule, so many 
tree-sitter function calls it before doing anything to make sure range 
is up-to-date. treesit-font-lock-fontify-region calls 
treesit-update-ranges at the beginning of its body, and 
treesit-update-ranges calls c-ts-mode--emacs-set-ranges.

Yuan