bug#62333: 30.0.50; Issue with tree-sitter syntax tree during certain changes

[Yuan Fu <casouri@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 2242 bytes --]



> On Mar 22, 2023, at 6:03 PM, Dmitry Gutov <dgutov@yandex.ru> wrote:
> 
> On 21/03/2023 16:00, Wilhelm Kirschbaum wrote:
>> With the following text in a buffer with elixir-ts-mode:
>> def foo() do
>> end
>> When I delete the `)` and re-insert it, the tree-sitter syntax tree does
>> not fully recover and the `do end` part is absent when observing using
>> treesit-explore-mode.
>> Before removing the `)`:
>> (source
>> (call target: (identifier)
>>  (arguments
>>   (call target: (identifier)
>>    (arguments ( ))))
>>  (do_block do end)))
>> after re-inserting `)`:
>> (source
>> (call target: (identifier)
>>  (arguments
>>   (call target: (identifier)
>>    (arguments ( ))))))
>> I can see the same behaviour using ruby-ts-mode when I remove the
>> following expression:
>> (setq-local syntax-propertize-function #'ruby-ts--syntax-propertize)
>> The ruby syntax I tested with:
>> def foo()
>> end
>> When I add a local `syntax-propertize-function' function to elixir-ts-mode
>> it also solves the problem of not fully recovering the tree-sitter
>> syntax tree.  It does not seem to matter what the function achieves
>> thought, so am a bit confused.
> 
> I can repro both scenarios. Except with ruby-ts-mode, the difference in printed AST is the added wrapping in (program ...). But the visible result is similar: "end" not being highlighted.

The problem seems to be caused by narrowing during fontification: when parsing, tree-sitter sees that BUF_ZV_BYTE is smaller than it’s visible_end, and removed the part after the parenthesis from the parse tree. I suspect this has to do with the narrowed fontification we added for dealing with long lines. 

CC’ed Eli because I don’t really know anything about the locked narrow thing besides that it exists. In particular, how does it determine when and where to narrow?


To see it in action, apply this patch, and repeat Wilhelm’s experiment: remove the “)” and reinsert it. You should see

start: 9, oend: 10, nend: 9
start: 9, oend: 10, nend: 9
start: 9, oend: 9, nend: 10
start: 9, oend: 9, nend: 10
2.5 start: 10, oend: 18, nend: 10 —> culprit

(Make sure to turn off electric-pair-mode to reduce noise.)

Yuan


[-- Attachment #2: print.diff --]
[-- Type: application/octet-stream, Size: 2389 bytes --]

diff --git a/src/treesit.c b/src/treesit.c
index 5a4fe3e8803..9b46dcef2d1 100644
--- a/src/treesit.c
+++ b/src/treesit.c
@@ -806,6 +806,7 @@ treesit_record_change (ptrdiff_t start_byte, ptrdiff_t old_end_byte,
 	  eassert (start_offset <= old_end_offset);
 	  eassert (start_offset <= new_end_offset);
 
+	  printf("start: %ld, oend: %ld, nend: %ld\n", start_offset, old_end_offset, new_end_offset);
 	  treesit_tree_edit_1 (tree, start_offset, old_end_offset,
 			       new_end_offset);
 	  XTS_PARSER (lisp_parser)->need_reparse = true;
@@ -925,6 +926,7 @@ treesit_sync_visible_region (Lisp_Object parser)
   if (visible_beg > BUF_BEGV_BYTE (buffer))
     {
       /* Tree-sitter sees: insert at the beginning.  */
+      printf("1 start: %d, oend: %d, nend: %ld\n", 0, 0, visible_beg - BUF_BEGV_BYTE (buffer));
       treesit_tree_edit_1 (tree, 0, 0, visible_beg - BUF_BEGV_BYTE (buffer));
       visible_beg = BUF_BEGV_BYTE (buffer);
       eassert (visible_beg <= visible_end);
@@ -933,6 +935,9 @@ treesit_sync_visible_region (Lisp_Object parser)
   if (visible_end < BUF_ZV_BYTE (buffer))
     {
       /* Tree-sitter sees: insert at the end.  */
+      printf("2 start: %ld, oend: %ld, nend: %ld\n", visible_end - visible_beg,
+	     visible_end - visible_beg,
+	     BUF_ZV_BYTE (buffer) - visible_beg);
       treesit_tree_edit_1 (tree, visible_end - visible_beg,
 			   visible_end - visible_beg,
 			   BUF_ZV_BYTE (buffer) - visible_beg);
@@ -942,6 +947,9 @@ treesit_sync_visible_region (Lisp_Object parser)
   else if (visible_end > BUF_ZV_BYTE (buffer))
     {
       /* Tree-sitter sees: delete at the end.  */
+      printf("2.5 start: %ld, oend: %ld, nend: %ld\n", BUF_ZV_BYTE (buffer) - visible_beg,
+			   visible_end - visible_beg,
+			   BUF_ZV_BYTE (buffer) - visible_beg);
       treesit_tree_edit_1 (tree, BUF_ZV_BYTE (buffer) - visible_beg,
 			   visible_end - visible_beg,
 			   BUF_ZV_BYTE (buffer) - visible_beg);
@@ -952,6 +960,7 @@ treesit_sync_visible_region (Lisp_Object parser)
   if (visible_beg < BUF_BEGV_BYTE (buffer))
     {
       /* Tree-sitter sees: delete at the beginning.  */
+      printf("3 start: %d, oend: %ld, nend: %d\n", 0, BUF_BEGV_BYTE (buffer) - visible_beg, 0);
       treesit_tree_edit_1 (tree, 0, BUF_BEGV_BYTE (buffer) - visible_beg, 0);
       visible_beg = BUF_BEGV_BYTE (buffer);
       eassert (visible_beg <= visible_end);