bug#60650: 30.0.50; tree-sitter: parsing (and font locking) occasionally breaks during editing

bug#60650: 30.0.50; tree-sitter: parsing (and font locking) occasionally breaks during editing

[Mickey Petersen]

It is occasionally possible to put the tree-sitter parser, I am guessing, into an erroneous state w.r.t. the contents of the buffer it is charged with parsing.

This rarely presents itself, and when it does, it only seems to happen when text is deleted and inserted in short order (`delete-region' + `insert') from an elisp command.

The resulting erroneous state then breaks font locking, resulting in mis-fontified code.

I must note that the parse tree is _correct_ and not invalid. There is no easy way to 'fix' this bug, as newlines and other manual edits rarely fixes the issue. There is also no easy way to insist that tree-sitter must re-do everything. Presumably deleting and recreating the parser could work as a workaround? I do not know.

The only way to truly and accurately fix the issue is to save and re-open the file. This then forces a complete re-parse of the buffer and fontification, fixing the issue.

I have zero means of reliably of reproducing this, unfortunately: it is sporadic, intermittent, and happens somewhat infrequently.

So I'd love some ideas on how to go about debugging this.



In GNU Emacs 30.0.50 (build 6, x86_64-pc-linux-gnu, GTK+ Version
 3.24.20, cairo version 1.16.0) of 2023-01-02 built on mickey-work
Repository revision: c209802f7b3721a1b95113290934a23fee88f678
Repository branch: master
Windowing system distributor 'The X.Org Foundation', version 11.0.12013000
System Description: Ubuntu 20.04.3 LTS

Configured using:
 'configure --with-native-compilation --with-json --with-mailutils
 --without-compress-install --with-imagemagick CC=gcc-10'

bug#60650: 30.0.50; tree-sitter: parsing (and font locking) occasionally breaks during editing

[Yuan Fu]

Mickey Petersen <mickey@masteringemacs.org> writes:

> It is occasionally possible to put the tree-sitter parser, I am
> guessing, into an erroneous state w.r.t. the contents of the buffer it
> is charged with parsing.
>
> This rarely presents itself, and when it does, it only seems to happen
> when text is deleted and inserted in short order (`delete-region' +
> `insert') from an elisp command.
>
> The resulting erroneous state then breaks font locking, resulting in mis-fontified code.
>
> I must note that the parse tree is _correct_ and not invalid. There is
> no easy way to 'fix' this bug, as newlines and other manual edits
> rarely fixes the issue. There is also no easy way to insist that
> tree-sitter must re-do everything. Presumably deleting and recreating
> the parser could work as a workaround? I do not know.
>
> The only way to truly and accurately fix the issue is to save and
> re-open the file. This then forces a complete re-parse of the buffer
> and fontification, fixing the issue.

What do you mean by "correct and not invalid"? If a complete re-parse
fixes the issue, then the old parse tree must be incorrect, right? As
for mis-fontified code, is the position wrong (everything skew left or
right for a few character), or there is no fontification, or something
else? If the position is wrong, then it’s probably a bug in syncing the
parser’s position wrt the buffer content.

> I have zero means of reliably of reproducing this, unfortunately: it is sporadic, intermittent, and happens somewhat infrequently.
>
> So I'd love some ideas on how to go about debugging this.

I think the first step is to better understand what’s happening. Some
screenshots are also welcome. Thanks for your effort in advance!

Yuan

bug#60650: 30.0.50; tree-sitter: parsing (and font locking) occasionally breaks during editing

[Mickey Petersen]

Yuan Fu <casouri@gmail.com> writes:

> Mickey Petersen <mickey@masteringemacs.org> writes:
>
>> It is occasionally possible to put the tree-sitter parser, I am
>> guessing, into an erroneous state w.r.t. the contents of the buffer it
>> is charged with parsing.
>>
>> This rarely presents itself, and when it does, it only seems to happen
>> when text is deleted and inserted in short order (`delete-region' +
>> `insert') from an elisp command.
>>
>> The resulting erroneous state then breaks font locking, resulting in mis-fontified code.
>>
>> I must note that the parse tree is _correct_ and not invalid. There is
>> no easy way to 'fix' this bug, as newlines and other manual edits
>> rarely fixes the issue. There is also no easy way to insist that
>> tree-sitter must re-do everything. Presumably deleting and recreating
>> the parser could work as a workaround? I do not know.
>>
>> The only way to truly and accurately fix the issue is to save and
>> re-open the file. This then forces a complete re-parse of the buffer
>> and fontification, fixing the issue.
>
> What do you mean by "correct and not invalid"? If a complete re-parse
> fixes the issue, then the old parse tree must be incorrect, right? As
> for mis-fontified code, is the position wrong (everything skew left or
> right for a few character), or there is no fontification, or something
> else? If the position is wrong, then it’s probably a bug in syncing the
> parser’s position wrt the buffer content.

I mean that the source code is syntactically correct. Given a clean
slate, the parser would not report any erroneous nodes.

But, somehow, both the font lock and indentation remains incorrect, even when I type text that would otherwise force a reparse. The only way to fix this is to save the buffer (in a correct state), close and re-open the file. That fixes it.

That, and undoing the edited steps done by the elisp code. That snaps it back to its original state, and the parse tree is back to normal.

It's... weird.

>
>> I have zero means of reliably of reproducing this, unfortunately: it is sporadic, intermittent, and happens somewhat infrequently.
>>
>> So I'd love some ideas on how to go about debugging this.
>
> I think the first step is to better understand what’s happening. Some
> screenshots are also welcome. Thanks for your effort in advance!
>

Yep. Will do!


> Yuan

bug#60650: 30.0.50; tree-sitter: parsing (and font locking) occasionally breaks during editing

[Yuan Fu]

Mickey Petersen <mickey@masteringemacs.org> writes:

> Yuan Fu <casouri@gmail.com> writes:
>
>> Mickey Petersen <mickey@masteringemacs.org> writes:
>>
>>> It is occasionally possible to put the tree-sitter parser, I am
>>> guessing, into an erroneous state w.r.t. the contents of the buffer it
>>> is charged with parsing.
>>>
>>> This rarely presents itself, and when it does, it only seems to happen
>>> when text is deleted and inserted in short order (`delete-region' +
>>> `insert') from an elisp command.
>>>
>>> The resulting erroneous state then breaks font locking, resulting in mis-fontified code.
>>>
>>> I must note that the parse tree is _correct_ and not invalid. There is
>>> no easy way to 'fix' this bug, as newlines and other manual edits
>>> rarely fixes the issue. There is also no easy way to insist that
>>> tree-sitter must re-do everything. Presumably deleting and recreating
>>> the parser could work as a workaround? I do not know.
>>>
>>> The only way to truly and accurately fix the issue is to save and
>>> re-open the file. This then forces a complete re-parse of the buffer
>>> and fontification, fixing the issue.
>>
>> What do you mean by "correct and not invalid"? If a complete re-parse
>> fixes the issue, then the old parse tree must be incorrect, right? As
>> for mis-fontified code, is the position wrong (everything skew left or
>> right for a few character), or there is no fontification, or something
>> else? If the position is wrong, then it’s probably a bug in syncing the
>> parser’s position wrt the buffer content.
>
> I mean that the source code is syntactically correct. Given a clean
> slate, the parser would not report any erroneous nodes.

OK, so the source code is syntactically correct, but the parse tree is
not (which lead to problems in fontification and indent).  Deleting the
parser and reparsing the whole buffer from scratch produces a correct
parse tree.

> But, somehow, both the font lock and indentation remains incorrect,
> even when I type text that would otherwise force a reparse.

That I can understand. If the parse-tree becomes erroneous, further
incremental parsing on it just continues on the wrong path (GIGO). You
have to delete the parser and re-parse from scratch to fix it, as you
observed.

> The only
> way to fix this is to save the buffer (in a correct state), close and
> re-open the file. That fixes it.
>
> That, and undoing the edited steps done by the elisp code. That snaps it back to its original state, and the parse tree is back to normal.
>
> It's... weird.

Indeed, there is no "undo" in tree-sitter, the parser just parses the
edits made by Emacs’ undo. And as I mentioned above, incrementally
parsing further on an erroneous tree shouldn’t magically fix the parse
tree. So this error can be corrected by doing the reverse edit on the
parse tree, hmmm.

>>
>>> I have zero means of reliably of reproducing this, unfortunately: it is sporadic, intermittent, and happens somewhat infrequently.
>>>
>>> So I'd love some ideas on how to go about debugging this.
>>
>> I think the first step is to better understand what’s happening. Some
>> screenshots are also welcome. Thanks for your effort in advance!
>>
>
> Yep. Will do!

Thanks. Now I understands the situation better. Next time you encounter
this, please take screenshots of the wrong fontification, the parse tree
(ie, content of treesit-explore-mode), and screenshots of the correct
fontification and parse tree (since you mentioned that undo can fix it),
and the content of C-h l (view-lossage) (unless it’s sensitive ;-).

Yuan