bug#66732: tree-sitter fontification doesn't update multi-line syntax reliably

[Yuan Fu <casouri@gmail.com>]

On 11/18/23 12:37 AM, Eli Zaretskii wrote:
> Ping!  Yuan, I'd appreciate if you chimed in on this issue.
>
>> Cc: 66732@debbugs.gnu.org, dominik@honnef.co
>> Date: Sun, 29 Oct 2023 14:22:30 +0200
>> From: Eli Zaretskii <eliz@gnu.org>
>>
>>> Date: Wed, 25 Oct 2023 02:15:30 +0300
>>> From: Dmitry Gutov <dmitry@gutov.dev>
>>>
>>> Hi Dominik,
>>>
>>> On 24/10/2023 17:22, Dominik Honnef wrote:
>>>> Steps to reproduce:
>>>>
>>>> 1. Start emacs -q
>>>> 2. Clear the scratch buffer
>>>> 3. Switch to c-ts-mode
>>>> 4. Type the following:
>>>>
>>>>       /* foo
>>>>       bar
>>>>       baz
>>>>       */
>>>>
>>>> Notice the following:
>>>>
>>>> 1. tree-sitter will not parse the buffer contents as a comment until the
>>>> closing comment marker is typed (not an issue per se.)
>>>>
>>>> 2. When you type the closing comment marker, only that line will be
>>>> fontified.
>>>>
>>>> 3. Moving to the previous line will refontify the whole comment.
>>>>
>>>> Similarly, removing the closing comment marker will not instantly
>>>> refontify the buffer, either.
>>>>
>>>> The expected behavior would be for the whole comment to be refontified
>>>> immediately after typing the closing comment marker.
>>>>
>>>> Other buffer contents often lead to even worse refontifying, where even
>>>> motion will not fix things.
>>> Can repro.
>>>
>>> Only with 'emacs -Q', though (note to others who will try).
>> Yuan, any ideas or comments?
>>
Some background: When buffer content changes, it might invalidate 
already-fontified region, as shown in the example: Typing the "*/" 
completes the block comment and should cause the comment to be 
refontified in comment-face.

The way we achieves this is thought parser notifiers. When tree-sitter 
parser reparses, it notifies us of which part of the buffer was affected 
by the reparse. For font-lock, we have this font-lock notifier that 
marks the affected buffer region as "not fontified", so redisplay will 
refontify those areas.

(defun treesit--font-lock-notifier (ranges parser)
   "Ensures updated parts of the parse-tree are refontified.
RANGES is a list of (BEG . END) ranges, PARSER is the tree-sitter
parser notifying of the change."
   (with-current-buffer (treesit-parser-buffer parser)
     (dolist (range ranges)
       (when treesit--font-lock-verbose
         (message "Notifier received range: %s-%s"
                  (car range) (cdr range)))
       (with-silent-modifications
         (put-text-property (car range) (cdr range) 'fontified nil)))))

This notifier function will be called during redisplay [1]. I suspect 
that because of this timing, redisplay doesn't refontify the marked 
region immediately. So I added a timer, I think that ensures we mark the 
affected region in the next command loop?

(defun treesit--font-lock-notifier (ranges parser)
   "Ensures updated parts of the parse-tree are refontified.
RANGES is a list of (BEG . END) ranges, PARSER is the tree-sitter
parser notifying of the change."
   (with-current-buffer (treesit-parser-buffer parser)
     (dolist (range ranges)
       (when treesit--font-lock-verbose
         (message "Notifier received range: %s-%s"
                  (car range) (cdr range)))
       (run-with-timer
        0 nil
        (lambda ()
          (with-silent-modifications
            (put-text-property (car range) (cdr range)
                               'fontified nil)))))))

This seems to work. Eli, do you see any problem using run-with-timer 
this way? What's the correct way to mark some region unfontified?

[1] The chain of events if roughly: user types the last "/" -> redisplay 
-> fontify that character -> access parser -> parser reparses -> calls 
notifier.

Yuan