bug#61529: 30.0.50; tree-sitter: weird off-by-one error but only in css-ts-mode(?) with `treesit-node-at'

bug#61529: 30.0.50; tree-sitter: weird off-by-one error but only in css-ts-mode(?) with `treesit-node-at'

[Mickey Petersen]

Here's a strange one.

I don't know where to point the finger here exactly, but I think
`treesit-node-at' might have a small bug in it somewhere.


Consider this `css-ts-mode' code:

    a {
      background: linear-gradient(|210deg, rgba(255,82,41,1) 0%, rgba(251,165,85,1) 54%, rgba(163,73,73,1) 100%);
    }

Let | be point.

Engage `treesit-inspect-mode' and you'll see it asserts that point is
at '('. OK, so that could easily be a glitch in that implementation,
but let's probe further.

With point at '2', then I'd expect `treesit-node-at' to yield that node. But it does not:

(cons (point) (treesit-node-at (point)))

=> (34 . #<treesit-node "(" in 34-35>)

Move back one point

(cons (1- (point)) (treesit-node-at (1- (point)))) 
 
=> (35 . #<treesit-node "(" in 34-35>)

Move *forward* one and it does, but it gives us `unit' but I'd expect
`integer_value` as per the explorer (and indeed the tree.)

(cons (1+ (point)) (treesit-node-at (1+ (point))))

=> (36 . #<treesit-node unit in 38-41>)

Again but with the TS implementation `treesit-node-on`:

(cons (point) (treesit-node-on (point) (point)))

=> (35 . #<treesit-node integer_value in 35-41>)

And now we get the right node.

bug#61529: 30.0.50; tree-sitter: weird off-by-one error but only in css-ts-mode(?) with `treesit-node-at'

[Eli Zaretskii]

> From: Mickey Petersen <mickey@masteringemacs.org>
> Date: Wed, 15 Feb 2023 08:25:53 +0000
> 
> 
> With point at '2', then I'd expect `treesit-node-at' to yield that node. But it does not:
> 
> (cons (point) (treesit-node-at (point)))
> 
> => (34 . #<treesit-node "(" in 34-35>)

The value of point is the number of the character which _follows_
point, yes?  So when the cursor is on '2', point is actually between
'(' and '2'.  Right?  What does this mean in terms of the node that
should be returned by tree-sitter?

bug#61529: 30.0.50; tree-sitter: weird off-by-one error but only in css-ts-mode(?) with `treesit-node-at'

[Mickey Petersen]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Mickey Petersen <mickey@masteringemacs.org>
>> Date: Wed, 15 Feb 2023 08:25:53 +0000
>>
>>
>> With point at '2', then I'd expect `treesit-node-at' to yield that node. But it does not:
>>
>> (cons (point) (treesit-node-at (point)))
>>
>> => (34 . #<treesit-node "(" in 34-35>)
>
> The value of point is the number of the character which _follows_
> point, yes?  So when the cursor is on '2', point is actually between
> '(' and '2'.  Right?  What does this mean in terms of the node that
> should be returned by tree-sitter?

Correct, point is between '(' and '2'. So 34-35 means it occupies
position 34-35 or [34,35). So point is outside the scope of the '('
single-char anonymous node.

Or at least it should be: the problem is that it *is* inside it in
this one weird instance and, near as I can find, only in this mode,
and then only in this place, it isn't. I suspect `treesit-node-at' has
a bug.

Consider:

    a {
      background: linear-gradient(210deg, rgba(|255,82,41,1) 0%, rgba(251,165,85,1) 54%, rgba(163,73,73,1) 100%);
    }

Note the new position of point in rgba. `treesit-node-at` with `(point)` now correctly returns

    #<treesit-node integer_value in 48-51>

Move point back one position:

    a {
      background: linear-gradient(210deg, rgba|(255,82,41,1) 0%, rgba(251,165,85,1) 54%, rgba(163,73,73,1) 100%);
    }

And now:

  (treesit-node-at (point)) => #<treesit-node "(" in 47-48>

In start contrast to the original example.

bug#61529: 30.0.50; tree-sitter: weird off-by-one error but only in css-ts-mode(?) with `treesit-node-at'

[Theodor Thornhill via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Mickey Petersen <mickey@masteringemacs.org> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>>> From: Mickey Petersen <mickey@masteringemacs.org>
>>> Date: Wed, 15 Feb 2023 08:25:53 +0000
>>>
>>>
>>> With point at '2', then I'd expect `treesit-node-at' to yield that node. But it does not:
>>>
>>> (cons (point) (treesit-node-at (point)))
>>>
>>> => (34 . #<treesit-node "(" in 34-35>)
>>
>> The value of point is the number of the character which _follows_
>> point, yes?  So when the cursor is on '2', point is actually between
>> '(' and '2'.  Right?  What does this mean in terms of the node that
>> should be returned by tree-sitter?
>
> Correct, point is between '(' and '2'. So 34-35 means it occupies
> position 34-35 or [34,35). So point is outside the scope of the '('
> single-char anonymous node.
>
> Or at least it should be: the problem is that it *is* inside it in
> this one weird instance and, near as I can find, only in this mode,
> and then only in this place, it isn't. I suspect `treesit-node-at' has
> a bug.
>

Hi, Mickey!

> Consider:
>
>     a {
>       background: linear-gradient(210deg, rgba(|255,82,41,1) 0%, rgba(251,165,85,1) 54%, rgba(163,73,73,1) 100%);
>     }
>
> Note the new position of point in rgba. `treesit-node-at` with `(point)` now correctly returns
>
>     #<treesit-node integer_value in 48-51>
>
> Move point back one position:
>
>     a {
>       background: linear-gradient(210deg, rgba|(255,82,41,1) 0%, rgba(251,165,85,1) 54%, rgba(163,73,73,1) 100%);
>     }
>
> And now:
>
>   (treesit-node-at (point)) => #<treesit-node "(" in 47-48>
>
> In start contrast to the original example.

So the docstring of treesit-node-at states:


  "Return the leaf node at position POS.

A leaf node is a node that doesn't have any child nodes.

The returned node's span covers POS: the node's beginning is before
or at POS, and the node's end is at or after POS.

If no leaf node's span covers POS (e.g., POS is on whitespace
between two leaf nodes), return the first leaf node after POS.

If there is no leaf node after POS, return the first leaf node
before POS.

Return nil if no leaf node can be returned.  If NAMED is non-nil,
only look for named nodes."

Doesn't this describe this behavior?

Theo

bug#61529: 30.0.50; tree-sitter: weird off-by-one error but only in css-ts-mode(?) with `treesit-node-at'

[Mickey Petersen]

Theodor Thornhill <theo@thornhill.no> writes:

> Mickey Petersen <mickey@masteringemacs.org> writes:
>
>> Eli Zaretskii <eliz@gnu.org> writes:
>>
>>>> From: Mickey Petersen <mickey@masteringemacs.org>
>>>> Date: Wed, 15 Feb 2023 08:25:53 +0000
>>>>
>>>>
>>>> With point at '2', then I'd expect `treesit-node-at' to yield that node. But it does not:
>>>>
>>>> (cons (point) (treesit-node-at (point)))
>>>>
>>>> => (34 . #<treesit-node "(" in 34-35>)
>>>
>>> The value of point is the number of the character which _follows_
>>> point, yes?  So when the cursor is on '2', point is actually between
>>> '(' and '2'.  Right?  What does this mean in terms of the node that
>>> should be returned by tree-sitter?
>>
>> Correct, point is between '(' and '2'. So 34-35 means it occupies
>> position 34-35 or [34,35). So point is outside the scope of the '('
>> single-char anonymous node.
>>
>> Or at least it should be: the problem is that it *is* inside it in
>> this one weird instance and, near as I can find, only in this mode,
>> and then only in this place, it isn't. I suspect `treesit-node-at' has
>> a bug.
>>
>
> Hi, Mickey!
>
Hey Theo!

>> Consider:
>>
>>     a {
>>       background: linear-gradient(210deg, rgba(|255,82,41,1) 0%, rgba(251,165,85,1) 54%, rgba(163,73,73,1) 100%);
>>     }
>>
>> Note the new position of point in rgba. `treesit-node-at` with `(point)` now correctly returns
>>
>>     #<treesit-node integer_value in 48-51>
>>
>> Move point back one position:
>>
>>     a {
>>       background: linear-gradient(210deg, rgba|(255,82,41,1) 0%, rgba(251,165,85,1) 54%, rgba(163,73,73,1) 100%);
>>     }
>>
>> And now:
>>
>>   (treesit-node-at (point)) => #<treesit-node "(" in 47-48>
>>
>> In start contrast to the original example.
>
> So the docstring of treesit-node-at states:
>
>
>   "Return the leaf node at position POS.
>
> A leaf node is a node that doesn't have any child nodes.
>
> The returned node's span covers POS: the node's beginning is before
> or at POS, and the node's end is at or after POS.
>
> If no leaf node's span covers POS (e.g., POS is on whitespace
> between two leaf nodes), return the first leaf node after POS.
>
> If there is no leaf node after POS, return the first leaf node
> before POS.
>
> Return nil if no leaf node can be returned.  If NAMED is non-nil,
> only look for named nodes."
>
> Doesn't this describe this behavior?
>

It's a good question: I suppose it's a question of wording (or
understanding) more than it necessarily being *wrong* -- it is, after
all, a custom function.

I read and interpreted it to mean that due to how node boundaries work
that "*end is at* or after POS" to mean that point is wholly contained
in the node "(" which, due to how tree-sitter determines node extents,
it technically isn't.

But I think it's fair enough if this is intentional -- I've no real
suggestions for improving its behaviour if this is intended. So if
it's working as expected, then it's safe to close the issue.

Thanks. Mickey.

bug#61529: 30.0.50; tree-sitter: weird off-by-one error but only in css-ts-mode(?) with `treesit-node-at'

[Theodor Thornhill via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Mickey Petersen <mickey@masteringemacs.org> writes:

> Theodor Thornhill <theo@thornhill.no> writes:
>
>> Mickey Petersen <mickey@masteringemacs.org> writes:
>>
>>> Eli Zaretskii <eliz@gnu.org> writes:
>>>
>>>>> From: Mickey Petersen <mickey@masteringemacs.org>
>>>>> Date: Wed, 15 Feb 2023 08:25:53 +0000
>>>>>
>>>>>
>>>>> With point at '2', then I'd expect `treesit-node-at' to yield that node. But it does not:
>>>>>
>>>>> (cons (point) (treesit-node-at (point)))
>>>>>
>>>>> => (34 . #<treesit-node "(" in 34-35>)
>>>>
>>>> The value of point is the number of the character which _follows_
>>>> point, yes?  So when the cursor is on '2', point is actually between
>>>> '(' and '2'.  Right?  What does this mean in terms of the node that
>>>> should be returned by tree-sitter?
>>>
>>> Correct, point is between '(' and '2'. So 34-35 means it occupies
>>> position 34-35 or [34,35). So point is outside the scope of the '('
>>> single-char anonymous node.
>>>
>>> Or at least it should be: the problem is that it *is* inside it in
>>> this one weird instance and, near as I can find, only in this mode,
>>> and then only in this place, it isn't. I suspect `treesit-node-at' has
>>> a bug.
>>>
>>
>> Hi, Mickey!
>>
> Hey Theo!
>
>>> Consider:
>>>
>>>     a {
>>>       background: linear-gradient(210deg, rgba(|255,82,41,1) 0%, rgba(251,165,85,1) 54%, rgba(163,73,73,1) 100%);
>>>     }
>>>
>>> Note the new position of point in rgba. `treesit-node-at` with `(point)` now correctly returns
>>>
>>>     #<treesit-node integer_value in 48-51>
>>>
>>> Move point back one position:
>>>
>>>     a {
>>>       background: linear-gradient(210deg, rgba|(255,82,41,1) 0%, rgba(251,165,85,1) 54%, rgba(163,73,73,1) 100%);
>>>     }
>>>
>>> And now:
>>>
>>>   (treesit-node-at (point)) => #<treesit-node "(" in 47-48>
>>>
>>> In start contrast to the original example.
>>
>> So the docstring of treesit-node-at states:
>>
>>
>>   "Return the leaf node at position POS.
>>
>> A leaf node is a node that doesn't have any child nodes.
>>
>> The returned node's span covers POS: the node's beginning is before
>> or at POS, and the node's end is at or after POS.
>>
>> If no leaf node's span covers POS (e.g., POS is on whitespace
>> between two leaf nodes), return the first leaf node after POS.
>>
>> If there is no leaf node after POS, return the first leaf node
>> before POS.
>>
>> Return nil if no leaf node can be returned.  If NAMED is non-nil,
>> only look for named nodes."
>>
>> Doesn't this describe this behavior?
>>
>
> It's a good question: I suppose it's a question of wording (or
> understanding) more than it necessarily being *wrong* -- it is, after
> all, a custom function.
>
> I read and interpreted it to mean that due to how node boundaries work
> that "*end is at* or after POS" to mean that point is wholly contained
> in the node "(" which, due to how tree-sitter determines node extents,
> it technically isn't.
>
> But I think it's fair enough if this is intentional -- I've no real
> suggestions for improving its behaviour if this is intended. So if
> it's working as expected, then it's safe to close the issue.
>

There is one thing here which confuses me a lot and that you might also
have some thoughts on. Consider some simple tsx:

```
const x = () => (
  <div>
    try to C-SPC C-SPC at the beginning of try after activating treesit-explore-mode
  </div>
)
```

Now you can maybe see that the jsx_text node covers a lot more than just
the line in the middle.  There are some other cases like this in some
languages, and they do trip up our semantics. May this be one similar
such case, just not concerning indentation in this case?

IOW, sometimes the parser also returns nodes including whitespace, so it
looks like we are outside a node, but we're not yet.

Theo

bug#61529: 30.0.50; tree-sitter: weird off-by-one error but only in css-ts-mode(?) with `treesit-node-at'

[Mickey Petersen]

Theodor Thornhill <theo@thornhill.no> writes:

> Mickey Petersen <mickey@masteringemacs.org> writes:
>
>> Theodor Thornhill <theo@thornhill.no> writes:
>>
>>> Mickey Petersen <mickey@masteringemacs.org> writes:
>>>
>>>> Eli Zaretskii <eliz@gnu.org> writes:
>>>>
>>>>>> From: Mickey Petersen <mickey@masteringemacs.org>
>>>>>> Date: Wed, 15 Feb 2023 08:25:53 +0000
>>>>>>
>>>>>>
>>>>>> With point at '2', then I'd expect `treesit-node-at' to yield that node. But it does not:
>>>>>>
>>>>>> (cons (point) (treesit-node-at (point)))
>>>>>>
>>>>>> => (34 . #<treesit-node "(" in 34-35>)
>>>>>
>>>>> The value of point is the number of the character which _follows_
>>>>> point, yes?  So when the cursor is on '2', point is actually between
>>>>> '(' and '2'.  Right?  What does this mean in terms of the node that
>>>>> should be returned by tree-sitter?
>>>>
>>>> Correct, point is between '(' and '2'. So 34-35 means it occupies
>>>> position 34-35 or [34,35). So point is outside the scope of the '('
>>>> single-char anonymous node.
>>>>
>>>> Or at least it should be: the problem is that it *is* inside it in
>>>> this one weird instance and, near as I can find, only in this mode,
>>>> and then only in this place, it isn't. I suspect `treesit-node-at' has
>>>> a bug.
>>>>
>>>
>>> Hi, Mickey!
>>>
>> Hey Theo!
>>
>>>> Consider:
>>>>
>>>>     a {
>>>>       background: linear-gradient(210deg, rgba(|255,82,41,1) 0%, rgba(251,165,85,1) 54%, rgba(163,73,73,1) 100%);
>>>>     }
>>>>
>>>> Note the new position of point in rgba. `treesit-node-at` with `(point)` now correctly returns
>>>>
>>>>     #<treesit-node integer_value in 48-51>
>>>>
>>>> Move point back one position:
>>>>
>>>>     a {
>>>>       background: linear-gradient(210deg, rgba|(255,82,41,1) 0%, rgba(251,165,85,1) 54%, rgba(163,73,73,1) 100%);
>>>>     }
>>>>
>>>> And now:
>>>>
>>>>   (treesit-node-at (point)) => #<treesit-node "(" in 47-48>
>>>>
>>>> In start contrast to the original example.
>>>
>>> So the docstring of treesit-node-at states:
>>>
>>>
>>>   "Return the leaf node at position POS.
>>>
>>> A leaf node is a node that doesn't have any child nodes.
>>>
>>> The returned node's span covers POS: the node's beginning is before
>>> or at POS, and the node's end is at or after POS.
>>>
>>> If no leaf node's span covers POS (e.g., POS is on whitespace
>>> between two leaf nodes), return the first leaf node after POS.
>>>
>>> If there is no leaf node after POS, return the first leaf node
>>> before POS.
>>>
>>> Return nil if no leaf node can be returned.  If NAMED is non-nil,
>>> only look for named nodes."
>>>
>>> Doesn't this describe this behavior?
>>>
>>
>> It's a good question: I suppose it's a question of wording (or
>> understanding) more than it necessarily being *wrong* -- it is, after
>> all, a custom function.
>>
>> I read and interpreted it to mean that due to how node boundaries work
>> that "*end is at* or after POS" to mean that point is wholly contained
>> in the node "(" which, due to how tree-sitter determines node extents,
>> it technically isn't.
>>
>> But I think it's fair enough if this is intentional -- I've no real
>> suggestions for improving its behaviour if this is intended. So if
>> it's working as expected, then it's safe to close the issue.
>>
>
> There is one thing here which confuses me a lot and that you might also
> have some thoughts on. Consider some simple tsx:
>
> ```
> const x = () => (
>   <div>
>     try to C-SPC C-SPC at the beginning of try after activating treesit-explore-mode
>   </div>
> )
> ```
>
> Now you can maybe see that the jsx_text node covers a lot more than just
> the line in the middle.  There are some other cases like this in some
> languages, and they do trip up our semantics. May this be one similar
> such case, just not concerning indentation in this case?
>

Yeah that's just how XML is, which is more or less what JSX is based
on. So that does not seem too surprising to me in this particular case
that the jsx_text node has extra whitespace around the text, even if
it is insignificant in SGML. (Though some xml parsers do trim the
leading and trailing element whitespace...)

> IOW, sometimes the parser also returns nodes including whitespace, so it
> looks like we are outside a node, but we're not yet.
>
> Theo

bug#61529: 30.0.50; tree-sitter: weird off-by-one error but only in css-ts-mode(?) with `treesit-node-at'

[Dmitry Gutov]

On 15/02/2023 21:01, Mickey Petersen wrote:
> I read and interpreted it to mean that due to how node boundaries work
> that "*end is at*  or after POS" to mean that point is wholly contained
> in the node "(" which, due to how tree-sitter determines node extents,
> it technically isn't.
> 
> But I think it's fair enough if this is intentional -- I've no real
> suggestions for improving its behaviour if this is intended. So if
> it's working as expected, then it's safe to close the issue.

The difference in this case is due to the tree structure:

In '210deg', (integer_value) is not a leaf node. The only leaf node it 
has inside is (unit), and that one starts 3 characters later.

In '255', however, (integer_value) *is* a leaf node. And since 
treesit-node-at looks for leaf nodes, it stops at "(" as documented.

I'd probably calls it a problem with the grammar, but it doesn't seem 
likely to be fixed, backward compatibility and all.

bug#61529: 30.0.50; tree-sitter: weird off-by-one error but only in css-ts-mode(?) with `treesit-node-at'

[Dmitry Gutov]

On 15/02/2023 20:35, Theodor Thornhill via Bug reports for GNU Emacs, 
the Swiss army knife of text editors wrote:
> So the docstring of treesit-node-at states:

Looking at the code, this change might describe it better:

diff --git a/lisp/treesit.el b/lisp/treesit.el
index 749781894b8..6e53b3d4c4a 100644
--- a/lisp/treesit.el
+++ b/lisp/treesit.el
@@ -166,10 +166,13 @@ treesit-node-at
  A leaf node is a node that doesn't have any child nodes.

  The returned node's span covers POS: the node's beginning is before
-or at POS, and the node's end is at or after POS.
+or at POS, and the node's end is after POS.

-If no leaf node's span covers POS (e.g., POS is on whitespace
-between two leaf nodes), return the first leaf node after POS.
+If no such node is found, but a leaf node ends at POS, it's
+returned.
+
+Otherwise (e.g., when POS is on whitespace between two leaf
+nodes), return the first leaf node after POS.

  If there is no leaf node after POS, return the first leaf node
  before POS.

bug#61529: 30.0.50; tree-sitter: weird off-by-one error but only in css-ts-mode(?) with `treesit-node-at'

[Eli Zaretskii]

> Date: Thu, 16 Feb 2023 23:34:58 +0200
> Cc: Eli Zaretskii <eliz@gnu.org>, 61529@debbugs.gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> +If no such node is found, but a leaf node ends at POS, it's
> +returned.

Passive tense alert!  This is better:

  If no such node exists, but there's a leaf node which ends at POS,
  return that node.

bug#61529: 30.0.50; tree-sitter: weird off-by-one error but only in css-ts-mode(?) with `treesit-node-at'

[Mickey Petersen]

Eli Zaretskii <eliz@gnu.org> writes:

>> Date: Thu, 16 Feb 2023 23:34:58 +0200
>> Cc: Eli Zaretskii <eliz@gnu.org>, 61529@debbugs.gnu.org
>> From: Dmitry Gutov <dgutov@yandex.ru>
>>
>> +If no such node is found, but a leaf node ends at POS, it's
>> +returned.
>
> Passive tense alert!  This is better:
>
>   If no such node exists, but there's a leaf node which ends at POS,
>   return that node.


Thanks, everyone. I think it's safe to say I misread the intent of the
function. It's safe to close this issue.

bug#61529: 30.0.50; tree-sitter: weird off-by-one error but only in css-ts-mode(?) with `treesit-node-at'

[Dmitry Gutov]

Version: 29.1

On 17/02/2023 08:16, Eli Zaretskii wrote:
>> Date: Thu, 16 Feb 2023 23:34:58 +0200
>> Cc: Eli Zaretskii<eliz@gnu.org>,61529@debbugs.gnu.org
>> From: Dmitry Gutov<dgutov@yandex.ru>
>>
>> +If no such node is found, but a leaf node ends at POS, it's
>> +returned.
> Passive tense alert!  This is better:
> 
>    If no such node exists, but there's a leaf node which ends at POS,
>    return that node.

Thank you, pushed. With that, I'm closing the report.