bug#66732: tree-sitter fontification doesn't update multi-line syntax reliably

[Yuan Fu <casouri@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 838 bytes --]



> On Dec 16, 2023, at 3:09 PM, Dmitry Gutov <dmitry@gutov.dev> wrote:
> 
> On 16/12/2023 21:18, Yuan Fu wrote:
>>> Indeed we don't want to do it directly in `after-change-functions`, but
>>> doing it in `font-lock-default-fontify-region` is too late because the
>>> redisplay and jit-lock have already (to some extent) decided what should
>>> be refontified at that point.
>>> 
>>> The infrastructure does offer ways to make it work (to fix previous
>>> incorrect assumptions about what needed to be redisplayed/refontified),
>>> but it's best if we can avoid it.
>> I tried forcing reparse in pre-redisplay-functions, and it seems to work fine! See attached diff. The delist part can be improved a bit, but this is just a POC.
> 
> This also works.

Ok, if there’s no objections, I’ll apply this patch.

Yuan


[-- Attachment #2: pre-redisplay.patch --]
[-- Type: application/octet-stream, Size: 3118 bytes --]

From eb2e9ca49e92465c5ef6ab82c59ccd1b395481d9 Mon Sep 17 00:00:00 2001
From: Yuan Fu <casouri@gmail.com>
Date: Sat, 16 Dec 2023 17:15:04 -0800
Subject: [PATCH] Correctly refontify changed region in tree-sitter modes
 (bug#66732)

We already have treesit--font-lock-notifier that should mark changed
regions to be refontified, but it's called too late in the redsiplay &
fontification pipeline.  Here we add treesit--pre-redisplay that
forces reparse and calls notifier functions in
pre-redisplay-functions, which is early enough for the marking to take
effect.

Similarly, we force reparse in
syntax-propertize-extend-region-functions so syntax-ppss will have the
up-to-date syntax information when it scans the buffer text.

* lisp/treesit.el (treesit--pre-redisplay): New function
(treesit--pre-syntax-ppss): New function.
(treesit-major-mode-setup): Add hooks.
---
 lisp/treesit.el | 33 ++++++++++++++++++++++++++++++++-
 1 file changed, 32 insertions(+), 1 deletion(-)

diff --git a/lisp/treesit.el b/lisp/treesit.el
index 8a07f5023a9..a9a526c0062 100644
--- a/lisp/treesit.el
+++ b/lisp/treesit.el
@@ -1088,6 +1088,33 @@ treesit--font-lock-notifier
       (with-silent-modifications
         (put-text-property (car range) (cdr range) 'fontified nil)))))
 
+(defun treesit--pre-redisplay (&rest _)
+  "Force reparse and consequently run all notifiers.
+
+One of the notifiers is `treesit--font-lock-notifier', which will
+mark the region whose syntax has changed to \"need to refontify\".
+
+For example, when the user types the final slash of a C block
+comment /* xxx */, not only do we need to fontify the slash, but
+also the whole block comment, which previously wasn't fontified
+as comment due to incomplete parse tree."
+  ;; `treesit-update-ranges' will force the host language's parser to
+  ;; reparse and set correct ranges for embedded parsers.  Then
+  ;; `treesit-parser-root-node' will force those parsers to reparse.
+  (treesit-update-ranges)
+  ;; Force repase on _all_ the parsers might not be necessary, but
+  ;; this is probably the most robust way.
+  (dolist (parser (treesit-parser-list nil nil t))
+    (treesit-parser-root-node parser)))
+
+(defun treesit--pre-syntax-ppss (&rest _)
+  "Force reparse and consequently run all notifiers.
+
+Similar to font-lock, we want to update the `syntax' text
+property before `syntax-ppss' starts working on the text."
+  (treesit--pre-redisplay)
+  nil)
+
 ;;; Indent
 
 (define-error 'treesit-indent-error
@@ -2392,7 +2419,11 @@ treesit-major-mode-setup
     (treesit-font-lock-recompute-features)
     (dolist (parser (treesit-parser-list))
       (treesit-parser-add-notifier
-       parser #'treesit--font-lock-notifier)))
+       parser #'treesit--font-lock-notifier))
+    (add-hook 'pre-redisplay-functions #'treesit--pre-redisplay 0 t))
+  ;; Syntax
+  (add-hook 'syntax-propertize-extend-region-functions
+            #'treesit--pre-syntax-ppss 0 t)
   ;; Indent.
   (when treesit-simple-indent-rules
     (setq-local treesit-simple-indent-rules
-- 
2.41.0