bug#66732: tree-sitter fontification doesn't update multi-line syntax reliably

[Dmitry Gutov <dmitry@gutov.dev>]

[-- Attachment #1: Type: text/plain, Size: 3458 bytes --]

On 11/12/2023 06:16, Yuan Fu wrote:
> The way we achieves this is thought parser notifiers. When tree-sitter 
> parser reparses, it notifies us of which part of the buffer was affected 
> by the reparse. For font-lock, we have this font-lock notifier that 
> marks the affected buffer region as "not fontified", so redisplay will 
> refontify those areas.
> 
> (defun treesit--font-lock-notifier (ranges parser)
>    "Ensures updated parts of the parse-tree are refontified.
> RANGES is a list of (BEG . END) ranges, PARSER is the tree-sitter
> parser notifying of the change."
>    (with-current-buffer (treesit-parser-buffer parser)
>      (dolist (range ranges)
>        (when treesit--font-lock-verbose
>          (message "Notifier received range: %s-%s"
>                   (car range) (cdr range)))
>        (with-silent-modifications
>          (put-text-property (car range) (cdr range) 'fontified nil)))))
> 
> This notifier function will be called during redisplay [1]. I suspect 
> that because of this timing, redisplay doesn't refontify the marked 
> region immediately. So I added a timer, I think that ensures we mark the 
> affected region in the next command loop?
> 
> (defun treesit--font-lock-notifier (ranges parser)
>    "Ensures updated parts of the parse-tree are refontified.
> RANGES is a list of (BEG . END) ranges, PARSER is the tree-sitter
> parser notifying of the change."
>    (with-current-buffer (treesit-parser-buffer parser)
>      (dolist (range ranges)
>        (when treesit--font-lock-verbose
>          (message "Notifier received range: %s-%s"
>                   (car range) (cdr range)))
>        (run-with-timer
>         0 nil
>         (lambda ()
>           (with-silent-modifications
>             (put-text-property (car range) (cdr range)
>                                'fontified nil)))))))
> 
> This seems to work. Eli, do you see any problem using run-with-timer 
> this way? What's the correct way to mark some region unfontified?
> 
> [1] The chain of events if roughly: user types the last "/" -> redisplay 
> -> fontify that character -> access parser -> parser reparses -> calls 
> notifier.

Note that it's not just font-lock. syntax-propertize has the same 
problem (I've described it in https://debbugs.gnu.org/67262#23).

And a timer wouldn't help because syntax-ppss needs to have up-to-date 
information whenever it's called, not later.

Here's a draft solution based on *-extend-region-functions, attached.

Alas, while it works fine in python-ts-mode (for both syntax and 
font-lock), making it behave better than python-mode, in c-ts-mode it 
doesn't quite have the same effect: when you backspace over the closing 
"/", the highlighting is properly updated only after you make the next 
edit (any edit), or select another window. I'm not sure, though, if it's 
due to my own problems with Emacs's failure to redisplay (reported 
elsewhere), so more testing is welcome.

But that might also be related to the use of 
c-ts-mode--emacs-set-ranges: printing a backtrace calls inside 
treesit--font-lock-notifier shows that the last notification comes also 
during font-lock but after treesit--font-lock-extend-region, inside 
c-ts-mode--emacs-set-ranges. I don't quite understand this design where 
the ranges are applied inside the font-lock code.

[-- Attachment #2: treesit--notifier-context.diff --]
[-- Type: text/x-patch, Size: 3485 bytes --]

diff --git a/lisp/treesit.el b/lisp/treesit.el
index 962a6fc3cf8..f5d30be52bb 100644
--- a/lisp/treesit.el
+++ b/lisp/treesit.el
@@ -1075,17 +1075,56 @@ treesit-font-lock-fontify-region
                                face (treesit-node-type node))))))))))))
   `(jit-lock-bounds ,start . ,end))
 
+(defvar treesit--notifier-ranges nil)
+(defvar treesit--notifier-context 'direct)
+
 (defun treesit--font-lock-notifier (ranges parser)
   "Ensures updated parts of the parse-tree are refontified.
+And re-syntax-propertized.
 RANGES is a list of (BEG . END) ranges, PARSER is the tree-sitter
 parser notifying of the change."
   (with-current-buffer (treesit-parser-buffer parser)
-    (dolist (range ranges)
-      (when treesit--font-lock-verbose
-        (message "Notifier received range: %s-%s"
-                 (car range) (cdr range)))
-      (with-silent-modifications
-        (put-text-property (car range) (cdr range) 'fontified nil)))))
+    (unless (eq treesit--notifier-context 'direct)
+      (setq treesit--notifier-ranges (append treesit--notifier-ranges ranges)))
+    (message "notifier context %s ranges %s" treesit--notifier-context ranges)
+    (when ranges
+      (unless (eq treesit--notifier-context 'font-lock)
+        (when treesit-font-lock-settings
+          (with-silent-modifications
+            (dolist (range ranges)
+              (when treesit--font-lock-verbose
+                (message "Notifier received range: %s-%s"
+                         (car range) (cdr range)))
+              ;; (backtrace)
+              (put-text-property (car range) (cdr range) 'fontified nil)))))
+      (unless (eq treesit--notifier-context 'syntax)
+        (when syntax-propertize-function
+          (syntax-ppss-flush-cache (cl-loop for r in ranges
+                                            minimize (car r))))))))
+
+(defun treesit--syntax-extend-region (beg end &optional font-lock-p)
+  (let (treesit--notifier-ranges
+        (treesit--notifier-context (if font-lock-p 'font-lock 'syntax)))
+    (treesit-buffer-root-node (treesit-language-at (point)))
+    (when treesit--notifier-ranges
+      ;; (message "some ranges received %S while beg end %s %s" treesit--notifier-ranges beg end)
+      ;; Some updates received from `treesit_ensure_parsed'.
+      (cl-loop for range in treesit--notifier-ranges
+               if (or (<= (car range) beg (cdr range))
+                      (<= (car range) end (cdr range)))
+               return (cons (min (car range) beg)
+                            (max (cdr range) end))))))
+
+(defun treesit--font-lock-extend-region ()
+  (defvar font-lock-beg)
+  (defvar font-lock-end)
+  ;; (message "called treesit--font-lock-extend-region")
+  (let ((new (treesit--syntax-extend-region font-lock-beg font-lock-end t)))
+    (when new
+      ;; (message "eeextended to %S" new)
+      (setq font-lock-beg (car new))
+      (setq font-lock-end (cdr new))
+      t)))
 
 ;;; Indent
 
@@ -2391,7 +2430,9 @@ treesit-major-mode-setup
     (treesit-font-lock-recompute-features)
     (dolist (parser (treesit-parser-list))
       (treesit-parser-add-notifier
-       parser #'treesit--font-lock-notifier)))
+       parser #'treesit--font-lock-notifier))
+    (push #'treesit--font-lock-extend-region font-lock-extend-region-functions)
+    (push #'treesit--syntax-extend-region syntax-propertize-extend-region-functions))
   ;; Indent.
   (when treesit-simple-indent-rules
     (setq-local treesit-simple-indent-rules