bug#71681: 29.3.50; tree-sitter crash

bug#71681: 29.3.50; tree-sitter crash

[Juri Linkov]

Evaluating this expression causes a crash:

(progn
  (find-file (expand-file-name "src/treesit.c" installation-directory))
  (c-ts-mode)
  (font-lock-ensure 63209 63387))

in latest master, but not in latest emacs-29 (only in 5-months old emacs-29).

If this is not reproducible, I could provide more details.

libtree-sitter is at the latest version.

Thread 1 "emacs" received signal SIGSEGV, Segmentation fault.
0x00007ffff3f88f41 in ts_language_public_symbol () from /usr/local/lib/libtree-sitter.so.0
(gdb) bt
#0  0x00007ffff3f88f41 in ts_language_public_symbol () at /usr/local/lib/libtree-sitter.so.0
#1  0x00007ffff3f9fe9c in ts_query_cursor.advance () at /usr/local/lib/libtree-sitter.so.0
#2  0x00007ffff3fa117f in ts_query_cursor_next_match () at /usr/local/lib/libtree-sitter.so.0
#3  0x00005555557f0f8f in Ftreesit_query_capture (node=<optimized out>, query=<optimized out>, beg=<optimized out>, end=<optimized out>, node_only=XIL(0)) at treesit.c:3014
#4  0x00007fffec125106 in F747265657369742d2d666f6e742d6c6f636b2d666f6e746966792d726567696f6e2d31_treesit__font_lock_fontify_region_1_0 ()
#5  0x000055555575faf7 in Ffuncall (nargs=7, args=0x7fffffffcc00) at eval.c:3093
#6  0x00007fffec124e28 in F747265657369742d666f6e742d6c6f636b2d666f6e746966792d726567696f6e_treesit_font_lock_fontify_region_0 ()
#7  0x000055555575faf7 in Ffuncall (nargs=4, args=0x7fffffffccb0) at eval.c:3093
#8  0x00007fffef266534 in F666f6e742d6c6f636b2d666f6e746966792d73796e746163746963616c6c792d726567696f6e_font_lock_fontify_syntactically_region_0 ()
#9  0x000055555575faf7 in Ffuncall (nargs=4, args=0x7fffffffce10) at eval.c:3093
#10 0x00007fffef26427f in F666f6e742d6c6f636b2d64656661756c742d666f6e746966792d726567696f6e_font_lock_default_fontify_region_0 ()
#11 0x000055555575faf7 in Ffuncall (nargs=4, args=0x7fffffffceb0) at eval.c:3093
#12 0x00007fffef2630c5 in F666f6e742d6c6f636b2d666f6e746966792d726567696f6e_font_lock_fontify_region_0 ()
#13 0x00005555557a8b38 in exec_byte_code (fun=<optimized out>, args_template=<optimized out>, nargs=<optimized out>, args=<optimized out>) at lisp.h:2243
#14 0x000055555575faf7 in Ffuncall (nargs=2, args=0x7fffffffd030) at eval.c:3093
#15 0x00005555557602a0 in run_hook_wrapped_funcall (nargs=<optimized out>, args=0x7fffffffd030) at eval.c:2872
#16 0x000055555575e9fb in run_hook_with_args (nargs=2, args=0x7fffffffd030, funcall=0x555555760280 <run_hook_wrapped_funcall>) at eval.c:2953
#17 0x00007fffef236115 in F6a69742d6c6f636b2d2d72756e2d66756e6374696f6e73_jit_lock__run_functions_0 ()
#18 0x000055555575faf7 in Ffuncall (nargs=3, args=0x7fffffffd150) at eval.c:3093
#19 0x00007fffef2369e9 in F6a69742d6c6f636b2d666f6e746966792d6e6f77_jit_lock_fontify_now_0 ()
#20 0x000055555575faf7 in Ffuncall (nargs=3, args=0x7fffffffd250) at eval.c:3093
#21 0x00007fffef263482 in F666f6e742d6c6f636b2d656e73757265_font_lock_ensure_0 ()
#22 0x00005555557631da in eval_sub (form=<optimized out>) at lisp.h:2243
#23 0x0000555555763381 in Fprogn (body=<optimized out>) at eval.c:439
#24 0x0000555555763066 in eval_sub (form=<optimized out>) at lisp.h:2243
#25 0x0000555555763381 in Fprogn (body=<optimized out>) at eval.c:439
#26 0x0000555555763066 in eval_sub (form=<optimized out>) at lisp.h:2243
#27 0x0000555555764bc1 in Fprogn (body=<optimized out>) at eval.c:439
#28 Flet (args=<optimized out>) at eval.c:1109
#29 0x0000555555763066 in eval_sub (form=<optimized out>) at lisp.h:2243
#30 0x0000555555763437 in Fsetq (args=<optimized out>) at eval.c:486
#31 0x0000555555763066 in eval_sub (form=<optimized out>) at lisp.h:2243
#32 0x000055555578ce3a in readevalloop_eager_expand_eval (val=<optimized out>, macroexpand=XIL(0xadd0)) at lisp.h:1192
#33 0x0000555555794ba0 in readevalloop (readcharfun=XIL(0x7ffff02e33d5), infile0=0x0, sourcename=XIL(0), printflag=true, unibyte=<optimized out>, readfun=XIL(0x5555560bc1f5), start=make_fixnum(202), end=XIL(0x5555560bc285)) at lread.c:2538
#34 0x000055555579601a in Feval_region (start=make_fixnum(202), end=make_fixnum(328), printflag=XIL(0x30), read_function=XIL(0x5555560bc1f5)) at lisp.h:752
#35 0x00007fffefacbbf6 in F656c6973702d2d6576616c2d646566756e_elisp__eval_defun_0 ()
#36 0x000055555575faf7 in Ffuncall (nargs=1, args=0x7fffffffd9f8) at eval.c:3093
#37 0x00007fffefacbcb1 in F616e6f6e796d6f75732d6c616d626461_anonymous_lambda_38 ()
#38 0x000055555575faf7 in Ffuncall (nargs=1, args=0x7fffffffda40) at eval.c:3093
#39 0x0000555555760f09 in call0 (fn=<optimized out>) at lisp.h:3515
#40 Fhandler_bind_1 (nargs=<optimized out>, args=0x7fffffffda90) at eval.c:1478
#41 0x00007fffefacbd7a in F6576616c2d646566756e_eval_defun_0 ()
#42 0x000055555575faf7 in Ffuncall (nargs=nargs@entry=2, args=args@entry=0x7fffffffdb58) at eval.c:3093
#43 0x000055555575b4f3 in Ffuncall_interactively (nargs=2, args=0x7fffffffdb58) at callint.c:250
#44 0x000055555575faf7 in Ffuncall (nargs=nargs@entry=3, args=args@entry=0x7fffffffdb50) at eval.c:3093
#45 0x000055555575cc53 in Fcall_interactively (function=<optimized out>, record_flag=<optimized out>, keys=<optimized out>) at callint.c:789
#46 0x00007fffef9330cd in F636f6d6d616e642d65786563757465_command_execute_0 ()
#47 0x000055555575faf7 in Ffuncall (nargs=nargs@entry=2, args=args@entry=0x7fffffffde50) at eval.c:3093
#48 0x00005555556e2247 in command_loop_1 () at lisp.h:1192
#49 0x000055555575e0d7 in internal_condition_case (bfun=bfun@entry=0x5555556e1e40 <command_loop_1>, handlers=handlers@entry=XIL(0x90), hfun=hfun@entry=0x5555556d63c0 <cmd_error>) at eval.c:1613
#50 0x00005555556ce07a in command_loop_2 (handlers=handlers@entry=XIL(0x90)) at keyboard.c:1168
#51 0x000055555575e019 in internal_catch (tag=tag@entry=XIL(0x11d30), func=func@entry=0x5555556ce050 <command_loop_2>, arg=arg@entry=XIL(0x90)) at eval.c:1292
#52 0x00005555556ce016 in command_loop () at lisp.h:1192
#53 0x00005555556d5f25 in recursive_edit_1 () at keyboard.c:754
#54 0x00005555556d62d4 in Frecursive_edit () at keyboard.c:837
#55 0x00005555555aebf4 in main (argc=<optimized out>, argv=<optimized out>) at emacs.c:2629

Lisp Backtrace:
"treesit--font-lock-fontify-region-1" (0xffffcc08)
"treesit-font-lock-fontify-region" (0xffffccb8)
"font-lock-fontify-syntactically-region" (0xffffce18)
"font-lock-default-fontify-region" (0xffffceb8)
"font-lock-fontify-region" (0xedea4040)
0x5681b288 PVEC_CLOSURE
"jit-lock--run-functions" (0xffffd158)
"jit-lock-fontify-now" (0xffffd258)
"font-lock-ensure" (0xffffd2d0)
"progn" (0xffffd3a0)
"progn" (0xffffd480)
"let" (0xffffd5d0)
"setq" (0xffffd6d0)
"elisp--eval-defun" (0xffffda00)
0xf060f638 PVEC_SUBR
"eval-defun" (0xffffdb60)
"funcall-interactively" (0xffffdb58)
"command-execute" (0xffffde58)

bug#71681: 29.3.50; tree-sitter crash

[Yuan Fu]

> On Jun 20, 2024, at 9:33 AM, Juri Linkov <juri@linkov.net> wrote:
> 
> Evaluating this expression causes a crash:
> 
> (progn
>  (find-file (expand-file-name "src/treesit.c" installation-directory))
>  (c-ts-mode)
>  (font-lock-ensure 63209 63387))
> 
> in latest master, but not in latest emacs-29 (only in 5-months old emacs-29).
> 
> If this is not reproducible, I could provide more details.
> 
> libtree-sitter is at the latest version.

Hmm, I can’t reproduce with latest master and libtree-sitter. Maybe you can send me the exact commits that you used?

Here’s mine:

Emacs: 72f2b01e318
Tree-sitter: 6ec478c1

Yuan

bug#71681: 29.3.50; tree-sitter crash

[Eli Zaretskii]

> Cc: 71681@debbugs.gnu.org
> From: Yuan Fu <casouri@gmail.com>
> Date: Sat, 22 Jun 2024 16:55:38 -0700
> 
> 
> 
> > On Jun 20, 2024, at 9:33 AM, Juri Linkov <juri@linkov.net> wrote:
> > 
> > Evaluating this expression causes a crash:
> > 
> > (progn
> >  (find-file (expand-file-name "src/treesit.c" installation-directory))
> >  (c-ts-mode)
> >  (font-lock-ensure 63209 63387))
> > 
> > in latest master, but not in latest emacs-29 (only in 5-months old emacs-29).
> > 
> > If this is not reproducible, I could provide more details.
> > 
> > libtree-sitter is at the latest version.
> 
> Hmm, I can’t reproduce with latest master and libtree-sitter. Maybe you can send me the exact commits that you used?
> 
> Here’s mine:
> 
> Emacs: 72f2b01e318
> Tree-sitter: 6ec478c1

I can reproduce this with tree-sitter version 0.20.8.  Can you try
building Emacs with that version?  I know it's somewhat old, but given
the ABI breakage issue, I expect quite a few people avoid upgrading to
a later version (I didn't).

bug#71681: 29.3.50; tree-sitter crash

[Juri Linkov]

>> Evaluating this expression causes a crash:
>>
>> (progn
>>  (find-file (expand-file-name "src/treesit.c" installation-directory))
>>  (c-ts-mode)
>>  (font-lock-ensure 63209 63387))
>>
>> in latest master, but not in latest emacs-29 (only in 5-months old emacs-29).
>>
>> If this is not reproducible, I could provide more details.
>>
>> libtree-sitter is at the latest version.
>
> Hmm, I can’t reproduce with latest master and libtree-sitter.
> Maybe you can send me the exact commits that you used?
>
> Here’s mine:
>
> Emacs: 72f2b01e318
> Tree-sitter: 6ec478c1

The commit are:

Emacs: 6f2036243f2 (2024-06-23, latest master)
Tree-sitter: 3da7deed (2024-06-08, version 0.22.6)

Also fails on old commits:
Emacs: ef01b634d21 (2024-01-18, emacs-29)
Tree-sitter: 870fb877 (2022-11-16, version 0.6.3)

But doesn't fail on:
Emacs: ce85d3811da (2024-06-18, recent emacs-29)
Tree-sitter: 3da7deed (2024-06-08, version 0.22.6)

Maybe it doesn't fail on recent emacs-29 because of the fix in 20af58d3a13?
But the same fix exists in latest master as well.

bug#71681: 29.3.50; tree-sitter crash

[Juri Linkov]

>> Evaluating this expression causes a crash:
>>
>> (progn
>>  (find-file (expand-file-name "src/treesit.c" installation-directory))
>>  (c-ts-mode)
>>  (font-lock-ensure 63209 63387))
>>
>> in latest master, but not in latest emacs-29 (only in 5-months old emacs-29).
>>
>> If this is not reproducible, I could provide more details.
>>
>> libtree-sitter is at the latest version.
>
> Hmm, I can’t reproduce with latest master and libtree-sitter. Maybe you can send me the exact commits that you used?
>
> Here’s mine:
>
> Emacs: 72f2b01e318
> Tree-sitter: 6ec478c1

Probably reproducibility depends on the content of the src/treesit.c file.
Then the most reliable way to reproduce it is this:

0. emacs -Q
1. eval: (add-to-list 'major-mode-remap-alist '(c-mode . c-ts-mode))
2. C-x v L
3. in the *vc-change-log* buffer move point to the commit 20af58d3a13
4. type D
5. crash caused by diff-font-lock-syntax fontification that uses treesit

The numbers in (font-lock-ensure 63209 63387) above were extracted
from diff hunk boundaries that might be different when the file was edited.

bug#71681: 29.3.50; tree-sitter crash

[Yuan Fu]

> On Jun 23, 2024, at 10:38 AM, Juri Linkov <juri@linkov.net> wrote:
> 
>>> Evaluating this expression causes a crash:
>>> 
>>> (progn
>>> (find-file (expand-file-name "src/treesit.c" installation-directory))
>>> (c-ts-mode)
>>> (font-lock-ensure 63209 63387))
>>> 
>>> in latest master, but not in latest emacs-29 (only in 5-months old emacs-29).
>>> 
>>> If this is not reproducible, I could provide more details.
>>> 
>>> libtree-sitter is at the latest version.
>> 
>> Hmm, I can’t reproduce with latest master and libtree-sitter. Maybe you can send me the exact commits that you used?
>> 
>> Here’s mine:
>> 
>> Emacs: 72f2b01e318
>> Tree-sitter: 6ec478c1
> 
> Probably reproducibility depends on the content of the src/treesit.c file.
> Then the most reliable way to reproduce it is this:
> 
> 0. emacs -Q
> 1. eval: (add-to-list 'major-mode-remap-alist '(c-mode . c-ts-mode))
> 2. C-x v L
> 3. in the *vc-change-log* buffer move point to the commit 20af58d3a13
> 4. type D
> 5. crash caused by diff-font-lock-syntax fontification that uses treesit
> 
> The numbers in (font-lock-ensure 63209 63387) above were extracted
> from diff hunk boundaries that might be different when the file was edited.

I reproduce it once with the first set of commits you provided, but for some reason couldn’t reproduce it again. I’m sure it’s something wrong that I did. I’ll report back when I make progress. TBH it seems like something wrong with tree-sitter itself, but I’ll make sure to figure out what’s the problem exactly.

Yuan

bug#71681: 29.3.50; tree-sitter crash

[Yuan Fu]

> On Jun 24, 2024, at 12:46 AM, Yuan Fu <casouri@gmail.com> wrote:
> 
> 
> 
>> On Jun 23, 2024, at 10:38 AM, Juri Linkov <juri@linkov.net> wrote:
>> 
>>>> Evaluating this expression causes a crash:
>>>> 
>>>> (progn
>>>> (find-file (expand-file-name "src/treesit.c" installation-directory))
>>>> (c-ts-mode)
>>>> (font-lock-ensure 63209 63387))
>>>> 
>>>> in latest master, but not in latest emacs-29 (only in 5-months old emacs-29).
>>>> 
>>>> If this is not reproducible, I could provide more details.
>>>> 
>>>> libtree-sitter is at the latest version.
>>> 
>>> Hmm, I can’t reproduce with latest master and libtree-sitter. Maybe you can send me the exact commits that you used?
>>> 
>>> Here’s mine:
>>> 
>>> Emacs: 72f2b01e318
>>> Tree-sitter: 6ec478c1
>> 
>> Probably reproducibility depends on the content of the src/treesit.c file.
>> Then the most reliable way to reproduce it is this:
>> 
>> 0. emacs -Q
>> 1. eval: (add-to-list 'major-mode-remap-alist '(c-mode . c-ts-mode))
>> 2. C-x v L
>> 3. in the *vc-change-log* buffer move point to the commit 20af58d3a13
>> 4. type D
>> 5. crash caused by diff-font-lock-syntax fontification that uses treesit
>> 
>> The numbers in (font-lock-ensure 63209 63387) above were extracted
>> from diff hunk boundaries that might be different when the file was edited.
> 
> I reproduce it once with the first set of commits you provided, but for some reason couldn’t reproduce it again. I’m sure it’s something wrong that I did. I’ll report back when I make progress. TBH it seems like something wrong with tree-sitter itself, but I’ll make sure to figure out what’s the problem exactly.
> 
> Yuan

Ok, I can reproduce it now. Looking into it…

Yuan

bug#71681: 29.3.50; tree-sitter crash

[Yuan Fu]

> On Jun 25, 2024, at 11:04 PM, Yuan Fu <casouri@gmail.com> wrote:
> 
> 
> 
>> On Jun 24, 2024, at 12:46 AM, Yuan Fu <casouri@gmail.com> wrote:
>> 
>> 
>> 
>>> On Jun 23, 2024, at 10:38 AM, Juri Linkov <juri@linkov.net> wrote:
>>> 
>>>>> Evaluating this expression causes a crash:
>>>>> 
>>>>> (progn
>>>>> (find-file (expand-file-name "src/treesit.c" installation-directory))
>>>>> (c-ts-mode)
>>>>> (font-lock-ensure 63209 63387))
>>>>> 
>>>>> in latest master, but not in latest emacs-29 (only in 5-months old emacs-29).
>>>>> 
>>>>> If this is not reproducible, I could provide more details.
>>>>> 
>>>>> libtree-sitter is at the latest version.
>>>> 
>>>> Hmm, I can’t reproduce with latest master and libtree-sitter. Maybe you can send me the exact commits that you used?
>>>> 
>>>> Here’s mine:
>>>> 
>>>> Emacs: 72f2b01e318
>>>> Tree-sitter: 6ec478c1
>>> 
>>> Probably reproducibility depends on the content of the src/treesit.c file.
>>> Then the most reliable way to reproduce it is this:
>>> 
>>> 0. emacs -Q
>>> 1. eval: (add-to-list 'major-mode-remap-alist '(c-mode . c-ts-mode))
>>> 2. C-x v L
>>> 3. in the *vc-change-log* buffer move point to the commit 20af58d3a13
>>> 4. type D
>>> 5. crash caused by diff-font-lock-syntax fontification that uses treesit
>>> 
>>> The numbers in (font-lock-ensure 63209 63387) above were extracted
>>> from diff hunk boundaries that might be different when the file was edited.
>> 
>> I reproduce it once with the first set of commits you provided, but for some reason couldn’t reproduce it again. I’m sure it’s something wrong that I did. I’ll report back when I make progress. TBH it seems like something wrong with tree-sitter itself, but I’ll make sure to figure out what’s the problem exactly.
>> 
>> Yuan
> 
> Ok, I can reproduce it now. Looking into it…

Finally figured out why. It’s not tree-sitter’s problem, but ours. I reduced the crash to a signal and pushed the fix to emacs-30. Next I’ll make sure the signal is properly handled. Below quoting the commit message:

The immediate cause of the crash is that tree-sitter accessed a node's
tree, but the tree is already deleted.

What happended, I think, is this:

1. Buffer modified, parser->need_reparse set to true,
parser->timestamp incremented.
2. A node is created from the parser, this node has the old tree but
the _new_ timestamp (bad!).
3. Parser re-parses (treesit_ensure_parsed), new tree created, old
tree deleted.
4. Ftreesit_query_capture accessed the old node, and the old tree,
crash.

We shouldn't bump the parser timestamp when we set
parser->need_reparse to true; instead, we should bump the timestamp
when we actually reparsed and created a new tree.

Yuan

bug#71681: 29.3.50; tree-sitter crash

[Vincenzo Pupillo]

Today I did a git pull of the master branch.
Is it possible that this patch is the cause of this error?

Error muted by safe_call: (treesit--font-lock-mark-ranges-to-fontify ((1 . 
8867)) #<treesit-parser for php>) signaled (wrong-type-argument treesit-node-p 
#<treesit-parser for php>)
Error during redisplay: (jit-lock-function 1) signaled (wrong-type-argument 
treesit-node-p #<treesit-parser for php>)

Vincenzo

In data domenica 30 giugno 2024 01:54:39 CEST, Yuan Fu ha scritto:
> > On Jun 25, 2024, at 11:04 PM, Yuan Fu <casouri@gmail.com> wrote:
> >> On Jun 24, 2024, at 12:46 AM, Yuan Fu <casouri@gmail.com> wrote:
> >>> On Jun 23, 2024, at 10:38 AM, Juri Linkov <juri@linkov.net> wrote:
> >>>>> Evaluating this expression causes a crash:
> >>>>> 
> >>>>> (progn
> >>>>> (find-file (expand-file-name "src/treesit.c" installation-directory))
> >>>>> (c-ts-mode)
> >>>>> (font-lock-ensure 63209 63387))
> >>>>> 
> >>>>> in latest master, but not in latest emacs-29 (only in 5-months old
> >>>>> emacs-29).
> >>>>> 
> >>>>> If this is not reproducible, I could provide more details.
> >>>>> 
> >>>>> libtree-sitter is at the latest version.
> >>>> 
> >>>> Hmm, I can’t reproduce with latest master and libtree-sitter. Maybe you
> >>>> can send me the exact commits that you used?
> >>>> 
> >>>> Here’s mine:
> >>>> 
> >>>> Emacs: 72f2b01e318
> >>>> Tree-sitter: 6ec478c1
> >>> 
> >>> Probably reproducibility depends on the content of the src/treesit.c
> >>> file.
> >>> Then the most reliable way to reproduce it is this:
> >>> 
> >>> 0. emacs -Q
> >>> 1. eval: (add-to-list 'major-mode-remap-alist '(c-mode . c-ts-mode))
> >>> 2. C-x v L
> >>> 3. in the *vc-change-log* buffer move point to the commit 20af58d3a13
> >>> 4. type D
> >>> 5. crash caused by diff-font-lock-syntax fontification that uses treesit
> >>> 
> >>> The numbers in (font-lock-ensure 63209 63387) above were extracted
> >>> from diff hunk boundaries that might be different when the file was
> >>> edited.
> >> 
> >> I reproduce it once with the first set of commits you provided, but for
> >> some reason couldn’t reproduce it again. I’m sure it’s something wrong
> >> that I did. I’ll report back when I make progress. TBH it seems like
> >> something wrong with tree-sitter itself, but I’ll make sure to figure
> >> out what’s the problem exactly.
> >> 
> >> Yuan
> > 
> > Ok, I can reproduce it now. Looking into it…
> 
> Finally figured out why. It’s not tree-sitter’s problem, but ours. I reduced
> the crash to a signal and pushed the fix to emacs-30. Next I’ll make sure
> the signal is properly handled. Below quoting the commit message:
> 
> The immediate cause of the crash is that tree-sitter accessed a node's
> tree, but the tree is already deleted.
> 
> What happended, I think, is this:
> 
> 1. Buffer modified, parser->need_reparse set to true,
> parser->timestamp incremented.
> 2. A node is created from the parser, this node has the old tree but
> the _new_ timestamp (bad!).
> 3. Parser re-parses (treesit_ensure_parsed), new tree created, old
> tree deleted.
> 4. Ftreesit_query_capture accessed the old node, and the old tree,
> crash.
> 
> We shouldn't bump the parser timestamp when we set
> parser->need_reparse to true; instead, we should bump the timestamp
> when we actually reparsed and created a new tree.
> 
> Yuan

bug#71681: 29.3.50; tree-sitter crash

[Juri Linkov]

> Finally figured out why. It’s not tree-sitter’s problem, but
> ours. I reduced the crash to a signal and pushed the fix to
> emacs-30. Next I’ll make sure the signal is properly handled. Below
> quoting the commit message:
>
> The immediate cause of the crash is that tree-sitter accessed a node's
> tree, but the tree is already deleted.
>
> What happended, I think, is this:
>
> 1. Buffer modified, parser->need_reparse set to true,
> parser->timestamp incremented.
> 2. A node is created from the parser, this node has the old tree but
> the _new_ timestamp (bad!).
> 3. Parser re-parses (treesit_ensure_parsed), new tree created, old
> tree deleted.
> 4. Ftreesit_query_capture accessed the old node, and the old tree,
> crash.
>
> We shouldn't bump the parser timestamp when we set
> parser->need_reparse to true; instead, we should bump the timestamp
> when we actually reparsed and created a new tree.

Thank you very much.  I confirm there are no crashes anymore.

bug#71681: 29.3.50; tree-sitter crash

[Vincenzo Pupillo]

Today I did a git pull of the master branch.
Is it possible that this patch is the cause of this error?

Error muted by safe_call: (treesit--font-lock-mark-ranges-to-fontify ((1 . 
8867)) #<treesit-parser for php>) signaled (wrong-type-argument treesit-node-p 
#<treesit-parser for php>)
Error during redisplay: (jit-lock-function 1) signaled (wrong-type-argument 
treesit-node-p #<treesit-parser for php>)

Vincenzo

In data domenica 30 giugno 2024 01:54:39 CEST, Yuan Fu ha scritto:
> > On Jun 25, 2024, at 11:04 PM, Yuan Fu <casouri@gmail.com> wrote:
> >> On Jun 24, 2024, at 12:46 AM, Yuan Fu <casouri@gmail.com> wrote:
> >>> On Jun 23, 2024, at 10:38 AM, Juri Linkov <juri@linkov.net> wrote:
> >>>>> Evaluating this expression causes a crash:
> >>>>> 
> >>>>> (progn
> >>>>> (find-file (expand-file-name "src/treesit.c" installation-directory))
> >>>>> (c-ts-mode)
> >>>>> (font-lock-ensure 63209 63387))
> >>>>> 
> >>>>> in latest master, but not in latest emacs-29 (only in 5-months old
> >>>>> emacs-29).
> >>>>> 
> >>>>> If this is not reproducible, I could provide more details.
> >>>>> 
> >>>>> libtree-sitter is at the latest version.
> >>>> 
> >>>> Hmm, I can’t reproduce with latest master and libtree-sitter. Maybe you
> >>>> can send me the exact commits that you used?
> >>>> 
> >>>> Here’s mine:
> >>>> 
> >>>> Emacs: 72f2b01e318
> >>>> Tree-sitter: 6ec478c1
> >>> 
> >>> Probably reproducibility depends on the content of the src/treesit.c
> >>> file.
> >>> Then the most reliable way to reproduce it is this:
> >>> 
> >>> 0. emacs -Q
> >>> 1. eval: (add-to-list 'major-mode-remap-alist '(c-mode . c-ts-mode))
> >>> 2. C-x v L
> >>> 3. in the *vc-change-log* buffer move point to the commit 20af58d3a13
> >>> 4. type D
> >>> 5. crash caused by diff-font-lock-syntax fontification that uses treesit
> >>> 
> >>> The numbers in (font-lock-ensure 63209 63387) above were extracted
> >>> from diff hunk boundaries that might be different when the file was
> >>> edited.
> >> 
> >> I reproduce it once with the first set of commits you provided, but for
> >> some reason couldn’t reproduce it again. I’m sure it’s something wrong
> >> that I did. I’ll report back when I make progress. TBH it seems like
> >> something wrong with tree-sitter itself, but I’ll make sure to figure
> >> out what’s the problem exactly.
> >> 
> >> Yuan
> > 
> > Ok, I can reproduce it now. Looking into it…
> 
> Finally figured out why. It’s not tree-sitter’s problem, but ours. I reduced
> the crash to a signal and pushed the fix to emacs-30. Next I’ll make sure
> the signal is properly handled. Below quoting the commit message:
> 
> The immediate cause of the crash is that tree-sitter accessed a node's
> tree, but the tree is already deleted.
> 
> What happended, I think, is this:
> 
> 1. Buffer modified, parser->need_reparse set to true,
> parser->timestamp incremented.
> 2. A node is created from the parser, this node has the old tree but
> the _new_ timestamp (bad!).
> 3. Parser re-parses (treesit_ensure_parsed), new tree created, old
> tree deleted.
> 4. Ftreesit_query_capture accessed the old node, and the old tree,
> crash.
> 
> We shouldn't bump the parser timestamp when we set
> parser->need_reparse to true; instead, we should bump the timestamp
> when we actually reparsed and created a new tree.
> 
> Yuan

bug#71681: 29.3.50; tree-sitter crash

[Yuan Fu]

> On Jun 30, 2024, at 12:22 PM, Vincenzo Pupillo <v.pupillo@gmail.com> wrote:
> 
> Today I did a git pull of the master branch.
> Is it possible that this patch is the cause of this error?
> 
> Error muted by safe_call: (treesit--font-lock-mark-ranges-to-fontify ((1 . 
> 8867)) #<treesit-parser for php>) signaled (wrong-type-argument treesit-node-p 
> #<treesit-parser for php>)
> Error during redisplay: (jit-lock-function 1) signaled (wrong-type-argument 
> treesit-node-p #<treesit-parser for php>)
> 
> Vincenzo

Embarrassingly, yes. I made a typo and didn’t catch it. It should be fixed now (on emacs-30).

Yuan

bug#71681: 29.3.50; tree-sitter crash

[Vincenzo Pupillo]

Thank you Yuan!
Vincenzo

In data lunedì 1 luglio 2024 07:37:40 CEST, Yuan Fu ha scritto:
> 
> > On Jun 30, 2024, at 12:22 PM, Vincenzo Pupillo <v.pupillo@gmail.com> wrote:
> > 
> > Today I did a git pull of the master branch.
> > Is it possible that this patch is the cause of this error?
> > 
> > Error muted by safe_call: (treesit--font-lock-mark-ranges-to-fontify ((1 . 
> > 8867)) #<treesit-parser for php>) signaled (wrong-type-argument treesit-node-p 
> > #<treesit-parser for php>)
> > Error during redisplay: (jit-lock-function 1) signaled (wrong-type-argument 
> > treesit-node-p #<treesit-parser for php>)
> > 
> > Vincenzo
> 
> Embarrassingly, yes. I made a typo and didn’t catch it. It should be fixed now (on emacs-30).
> 
> Yuan
> 

bug#71681: 29.3.50; tree-sitter crash

[Juri Linkov]

> I reduced the crash to a signal and pushed the fix to emacs-30.
> Next I’ll make sure the signal is properly handled.

Now with the latest emacs-30 at the commit b2c966f8396
there is another problem:

0. emacs -Q
1. eval: (setq backtrace-on-redisplay-error t)
2. eval: (add-to-list 'major-mode-remap-alist '(c-mode . c-ts-mode))
3. C-x v L
4. in the *vc-change-log* buffer move point to the commit b2c966f8396
5. type D

Warning (error): Error in a redisplay Lisp hook.
See buffer *Redisplay-trace*

Error: treesit-node-outdated (#<treesit-node-outdated>)
  treesit--font-lock-fontify-region-1(#<treesit-node-outdated> #<treesit-compiled-query> 99654 99975 nil nil)
  treesit-font-lock-fontify-region(99654 99975 nil)
  font-lock-fontify-syntactically-region(99654 99975 nil)
  font-lock-default-fontify-region(99654 99974 nil)
  font-lock-fontify-region(99654 99974)
  font-lock-ensure(99654 99974)
  diff-syntax-fontify-hunk(122 539 t)
  diff-syntax-fontify(122 539)
  diff--font-lock-syntax(539)
  font-lock-fontify-keywords-region(1 539 nil)
  font-lock-default-fontify-region(1 539 nil)
  font-lock-fontify-region(1 539)
  jit-lock--run-functions(1 539)
  jit-lock-fontify-now(1 539)
  jit-lock-function(1)
  vc-diff-finish(#<buffer *vc-diff*> nil nil)
  vc-exec-after(#f(compiled-function () #<bytecode -0xcd6e6e57937525e>))
  log-view-diff-common(1 1 t)
  log-view-diff-changeset(1 1)
  funcall-interactively(log-view-diff-changeset 1 1)
  command-execute(log-view-diff-changeset)

bug#71681: 29.3.50; tree-sitter crash

[Yuan Fu]

> On Jun 30, 2024, at 11:49 PM, Juri Linkov <juri@linkov.net> wrote:
> 
>> I reduced the crash to a signal and pushed the fix to emacs-30.
>> Next I’ll make sure the signal is properly handled.
> 
> Now with the latest emacs-30 at the commit b2c966f8396
> there is another problem:
> 
> 0. emacs -Q
> 1. eval: (setq backtrace-on-redisplay-error t)
> 2. eval: (add-to-list 'major-mode-remap-alist '(c-mode . c-ts-mode))
> 3. C-x v L
> 4. in the *vc-change-log* buffer move point to the commit b2c966f8396
> 5. type D
> 
> Warning (error): Error in a redisplay Lisp hook.
> See buffer *Redisplay-trace*
> 
> Error: treesit-node-outdated (#<treesit-node-outdated>)
>  treesit--font-lock-fontify-region-1(#<treesit-node-outdated> #<treesit-compiled-query> 99654 99975 nil nil)
>  treesit-font-lock-fontify-region(99654 99975 nil)
>  font-lock-fontify-syntactically-region(99654 99975 nil)
>  font-lock-default-fontify-region(99654 99974 nil)
>  font-lock-fontify-region(99654 99974)
>  font-lock-ensure(99654 99974)
>  diff-syntax-fontify-hunk(122 539 t)
>  diff-syntax-fontify(122 539)
>  diff--font-lock-syntax(539)
>  font-lock-fontify-keywords-region(1 539 nil)
>  font-lock-default-fontify-region(1 539 nil)
>  font-lock-fontify-region(1 539)
>  jit-lock--run-functions(1 539)
>  jit-lock-fontify-now(1 539)
>  jit-lock-function(1)
>  vc-diff-finish(#<buffer *vc-diff*> nil nil)
>  vc-exec-after(#f(compiled-function () #<bytecode -0xcd6e6e57937525e>))
>  log-view-diff-common(1 1 t)
>  log-view-diff-changeset(1 1)
>  funcall-interactively(log-view-diff-changeset 1 1)
>  command-execute(log-view-diff-changeset)

Yes, that’s what meant by “reduced crash to signal”. The crash is fixed, but I need to fix the font-lock code so it can handle the signal gracefully (or don’t cause the signal from the first place). It’s not yet clear to me why does treesit-font-lock-fontify-region end up using an outdated node for query.

Yuan