bug#64329: 29.0.92; treesit/fill-paragraph syntax highlighting problem

[Yuan Fu <casouri@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 2292 bytes --]



> On Jun 28, 2023, at 9:46 AM, Troy Brown <brownts@troybrown.dev> wrote:
> 
> I've noticed this problem on multiple tree-sitter major modes including
> c-ts-mode, c++-ts-mode, java-ts-mode, bash-ts-mode.  I haven't tried
> others, but I suspect those might also suffer from this problem.
> 
> The issue occurs when attempting to fill the paragraph of a comment
> block.  The following comment block can be used as an example to
> reproduce the problem and happens with "emacs -Q" (assuming
> corresponding tree-sitter libraries are available).
> 
> --8<---------------cut here---------------start------------->8---
> // The quick brown fox jumps over the
> // lazy dog.
> // The quick brown fox jumps over the lazy dog.
> --8<---------------cut here---------------end--------------->8---
> 
> Switch to one of the tree-sitter modes (e.g., M-x java-ts-mode).  Move
> point to the first line of the comment block above and then execute the
> fill-paragraph command (i.e., M-q).
> 
> The text which is wrapped onto the first line of the comment block will
> be highlighted incorrectly.  The results appear as if the comment
> delimiter was removed, fontification occurred, then the text was moved
> to the first line of the comment block and never refontified with the
> comment face.

Thank you very much! It’s funny that how long this went under the radar, presumably because we always use block comment.

The culprit is the subst-char-in-region function used by the filling function. It has a branch:

if (xxx)
  {
	replace_range (pos, pos + 1, string, ...);
  }
else
  {
	for (i = 0; i < len; i++) *p++ = tostr[i];
  }

I overlooked the else branch and thought subst-char-in-region always calls replace_range. replace_range notifies tree-sitter of the change it makes; but when subst-char-in-region manually replaces the text in the else branch, those edits are not notified to tree-sitter.

Please see the attached patch. Eli, is it more preferable to add a subroutine in insdel.c that does what "for (i = 0; i < len; i++) *p++ = tostr[I];” does, plus calling treesit_record_change, and make subst-char-in-region call that subroutine? (This way editfns.c don’t need to include treesit.h and call treesit_record_change itself.)

Yuan


[-- Attachment #2: notify.patch --]
[-- Type: application/octet-stream, Size: 1461 bytes --]

From ab94e738fb0137f1296921232cbd65046c11022f Mon Sep 17 00:00:00 2001
From: Yuan Fu <casouri@gmail.com>
Date: Wed, 28 Jun 2023 14:16:52 -0700
Subject: [PATCH] Call treesit_record_change in subst-char-in-region
 (bug#64329)

* src/editfns.c (Fsubst_char_in_region): Call treesit_record_change in
the else branch.
---
 src/editfns.c | 13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/src/editfns.c b/src/editfns.c
index d02cce4aef3..0cbeefb3262 100644
--- a/src/editfns.c
+++ b/src/editfns.c
@@ -55,6 +55,11 @@ Copyright (C) 1985-2023 Free Software Foundation, Inc.
 #ifdef WINDOWSNT
 # include "w32common.h"
 #endif
+
+#ifdef HAVE_TREE_SITTER
+#include "treesit.h"
+#endif
+
 static void update_buffer_properties (ptrdiff_t, ptrdiff_t);
 static Lisp_Object styled_format (ptrdiff_t, Lisp_Object *, bool);
 
@@ -2391,6 +2396,14 @@ #define COMBINING_BOTH (COMBINING_BEFORE | COMBINING_AFTER)
 	      if (NILP (noundo))
 		record_change (pos, 1);
 	      for (i = 0; i < len; i++) *p++ = tostr[i];
+
+#ifdef HAVE_TREE_SITTER
+	      /* In the previous branch, replace_range() notifies
+                 changes to tree-sitter, but in this branch, we
+                 modified buffer content manually, so we need to
+                 notify tree-sitter manually.  */
+	      treesit_record_change (pos_byte, pos_byte + len, pos_byte + len);
+#endif
 	    }
 	  last_changed =  pos + 1;
 	}
-- 
2.33.1