bug#59633: 29.0.50; tree-sitter-bash: Incorrect behaviour of heredocs with expansions

bug#59633: 29.0.50; tree-sitter-bash: Incorrect behaviour of heredocs with expansions

[miha--- via Bug reports for GNU Emacs, the Swiss army knife of text editors]

[-- Attachment #1: Type: text/plain, Size: 847 bytes --]

1. Visit empty buffer test in fundamental mode (C-x b test RET)
2. Insert

tee << EOF
text1 $var
text2 $(echo cmd)
text3
$var2
EOF

3. Put point before "text2"
4. M-x load-library RET treesit RET
5. M-: (treesit-node-children (treesit-node-at (point) 'bash))

    => (#<treesit-node (simple_expansion) in 18-22>
        #<treesit-node (command_substitution) in 29-40>)

   Note that this list is incorrect, "$var2" at the end of heredoc is
   missing.

6. M-: (treesit-node-first-child-for-pos (treesit-node-at (point) 'bash) (point))

   => nil

   Expected return value here is #<treesit-node (command_substitution) in 29-40>

I'm not sure if this bug is on Emacs, tree-sitter-bash or even
tree-sitter itself and I don't know how to check.

tree-sitter-bash version: 0.19.0.r19.g77cf8a7-1
tree-sitter version: 0.20.7-1

Thanks and best regards.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 861 bytes --]

bug#59633: 29.0.50; tree-sitter-bash: Incorrect behaviour of heredocs with expansions

[Yuan Fu]

miha@kamnitnik.top writes:

> 1. Visit empty buffer test in fundamental mode (C-x b test RET)
> 2. Insert
>
> tee << EOF
> text1 $var
> text2 $(echo cmd)
> text3
> $var2
> EOF
>
> 3. Put point before "text2"
> 4. M-x load-library RET treesit RET
> 5. M-: (treesit-node-children (treesit-node-at (point) 'bash))
>
>     => (#<treesit-node (simple_expansion) in 18-22>
>         #<treesit-node (command_substitution) in 29-40>)
>
>    Note that this list is incorrect, "$var2" at the end of heredoc is
>    missing.
>
> 6. M-: (treesit-node-first-child-for-pos (treesit-node-at (point) 'bash) (point))
>
>    => nil
>
>    Expected return value here is #<treesit-node (command_substitution) in 29-40>
>
> I'm not sure if this bug is on Emacs, tree-sitter-bash or even
> tree-sitter itself and I don't know how to check.
>
> tree-sitter-bash version: 0.19.0.r19.g77cf8a7-1
> tree-sitter version: 0.20.7-1
>
> Thanks and best regards.

Thanks. This is the tree produced by tree-sitter, does it match your
expectation?

(heredoc_body
(simple_expansion $ (variable_name))
(command_substitution $(
  (command
   name: (command_name (word))
   argument: (word))
  ))
(simple_expansion $ (variable_name)))

treesit-node-at gives you the _smallest_ node at point, maybe that’s why
you didn’t see expected behavior?

Yuan

bug#59633: 29.0.50; tree-sitter-bash: Incorrect behaviour of heredocs with expansions

[miha--- via Bug reports for GNU Emacs, the Swiss army knife of text editors]

[-- Attachment #1: Type: text/plain, Size: 2124 bytes --]

Yuan Fu <casouri@gmail.com> writes:

> miha@kamnitnik.top writes:
>
>> 1. Visit empty buffer test in fundamental mode (C-x b test RET)
>> 2. Insert
>>
>> tee << EOF
>> text1 $var
>> text2 $(echo cmd)
>> text3
>> $var2
>> EOF
>>
>> 3. Put point before "text2"
>> 4. M-x load-library RET treesit RET
>> 5. M-: (treesit-node-children (treesit-node-at (point) 'bash))
>>
>>     => (#<treesit-node (simple_expansion) in 18-22>
>>         #<treesit-node (command_substitution) in 29-40>)
>>
>>    Note that this list is incorrect, "$var2" at the end of heredoc is
>>    missing.
>>
>> 6. M-: (treesit-node-first-child-for-pos (treesit-node-at (point) 'bash) (point))
>>
>>    => nil
>>
>>    Expected return value here is #<treesit-node (command_substitution) in 29-40>
>>
>> I'm not sure if this bug is on Emacs, tree-sitter-bash or even
>> tree-sitter itself and I don't know how to check.
>>
>> tree-sitter-bash version: 0.19.0.r19.g77cf8a7-1
>> tree-sitter version: 0.20.7-1
>>
>> Thanks and best regards.
>
> Thanks. This is the tree produced by tree-sitter, does it match your
> expectation?
>
> (heredoc_body
> (simple_expansion $ (variable_name))
> (command_substitution $(
>   (command
>    name: (command_name (word))
>    argument: (word))
>   ))
> (simple_expansion $ (variable_name)))
>
> treesit-node-at gives you the _smallest_ node at point, maybe that’s why
> you didn’t see expected behavior?
>
> Yuan

The latest commit to tree-sitter-bash is

commit 4488aa41406547e478636a4fcfd24f5bbc3f2f74
Author: João P. L. Carvalho <jaopaulolc@gmail.com>
Date:   Sun Nov 27 20:39:41 2022 -0700

    Fix scanning of heredoc_body to allow empty bodies (#137).

Seems like it has fixed the problem described in the 5. step and the
tree produced by tree-sitter is now expected.

The problem in the 6. step is still present though. With point in front
of command_substitution, on would expect

    (treesit-node-first-child-for-pos (treesit-node-at (point) 'bash) (point))

to return the command_substitution node instead of returning nil.

Thanks.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 861 bytes --]

bug#59633: 29.0.50; tree-sitter-bash: Incorrect behaviour of heredocs with expansions

[Yuan Fu]

> On Dec 2, 2022, at 8:26 AM, miha@kamnitnik.top wrote:
> 
> Yuan Fu <casouri@gmail.com> writes:
> 
>> miha@kamnitnik.top writes:
>> 
>>> 1. Visit empty buffer test in fundamental mode (C-x b test RET)
>>> 2. Insert
>>> 
>>> tee << EOF
>>> text1 $var
>>> text2 $(echo cmd)
>>> text3
>>> $var2
>>> EOF
>>> 
>>> 3. Put point before "text2"
>>> 4. M-x load-library RET treesit RET
>>> 5. M-: (treesit-node-children (treesit-node-at (point) 'bash))
>>> 
>>>    => (#<treesit-node (simple_expansion) in 18-22>
>>>        #<treesit-node (command_substitution) in 29-40>)
>>> 
>>>   Note that this list is incorrect, "$var2" at the end of heredoc is
>>>   missing.
>>> 
>>> 6. M-: (treesit-node-first-child-for-pos (treesit-node-at (point) 'bash) (point))
>>> 
>>>   => nil
>>> 
>>>   Expected return value here is #<treesit-node (command_substitution) in 29-40>
>>> 
>>> I'm not sure if this bug is on Emacs, tree-sitter-bash or even
>>> tree-sitter itself and I don't know how to check.
>>> 
>>> tree-sitter-bash version: 0.19.0.r19.g77cf8a7-1
>>> tree-sitter version: 0.20.7-1
>>> 
>>> Thanks and best regards.
>> 
>> Thanks. This is the tree produced by tree-sitter, does it match your
>> expectation?
>> 
>> (heredoc_body
>> (simple_expansion $ (variable_name))
>> (command_substitution $(
>>  (command
>>   name: (command_name (word))
>>   argument: (word))
>>  ))
>> (simple_expansion $ (variable_name)))
>> 
>> treesit-node-at gives you the _smallest_ node at point, maybe that’s why
>> you didn’t see expected behavior?
>> 
>> Yuan
> 
> The latest commit to tree-sitter-bash is
> 
> commit 4488aa41406547e478636a4fcfd24f5bbc3f2f74
> Author: João P. L. Carvalho <jaopaulolc@gmail.com>
> Date:   Sun Nov 27 20:39:41 2022 -0700
> 
>    Fix scanning of heredoc_body to allow empty bodies (#137).
> 
> Seems like it has fixed the problem described in the 5. step and the
> tree produced by tree-sitter is now expected.
> 
> The problem in the 6. step is still present though. With point in front
> of command_substitution, on would expect
> 
>    (treesit-node-first-child-for-pos (treesit-node-at (point) 'bash) (point))
> 
> to return the command_substitution node instead of returning nil.

What’s the return value of (treesit-node-at (point)) at that point? Probably the command_substitution node, and since it doesn’t have any children, treesit-node-first-child-for-pos would return nil.

Yuan

bug#59633: 29.0.50; tree-sitter-bash: Incorrect behaviour of heredocs with expansions

[miha--- via Bug reports for GNU Emacs, the Swiss army knife of text editors]

[-- Attachment #1: Type: text/plain, Size: 1163 bytes --]

Yuan Fu <casouri@gmail.com> writes:

> What’s the return value of (treesit-node-at (point)) at that point?
> Probably the command_substitution node, and since it doesn’t have any
> children, treesit-node-first-child-for-pos would return nil.

(treesit-node-at (point) 'bash)
=> #<treesit-node (heredoc_body) in 12-56>

I forgot to say that point is before "text2", outside of the
command_substitution node.

> Yuan

tee << EOF
text1 $var
text2 $(echo cmd)
text3
$var2
EOF


Filling a test buffer with these contents and placing point before
"text2", we get

    (treesit-node-at (point) 'bash)
    => #<treesit-node (heredoc_body) in 12-56>

    (treesit-node-children (treesit-node-at (point) 'bash))
    => (#<treesit-node (simple_expansion) in 18-22>
        #<treesit-node (command_substitution) in 29-40>
        #<treesit-node (simple_expansion) in 47-52>)

This is now expected.

    (point)
    => 23

    (treesit-node-first-child-for-pos (treesit-node-at (point) 'bash) (point))
    => nil

This is not expected. One would expect this to return
command_substitution node, since it follows the position 23.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 861 bytes --]

bug#59633: 29.0.50; tree-sitter-bash: Incorrect behaviour of heredocs with expansions

[Yuan Fu]

<miha@kamnitnik.top> writes:

> Yuan Fu <casouri@gmail.com> writes:
>
>> What’s the return value of (treesit-node-at (point)) at that point?
>> Probably the command_substitution node, and since it doesn’t have any
>> children, treesit-node-first-child-for-pos would return nil.
>
> (treesit-node-at (point) 'bash)
> => #<treesit-node (heredoc_body) in 12-56>
>
> I forgot to say that point is before "text2", outside of the
> command_substitution node.
>
>> Yuan
>
> tee << EOF
> text1 $var
> text2 $(echo cmd)
> text3
> $var2
> EOF
>
>
> Filling a test buffer with these contents and placing point before
> "text2", we get
>
>     (treesit-node-at (point) 'bash)
>     => #<treesit-node (heredoc_body) in 12-56>
>
>     (treesit-node-children (treesit-node-at (point) 'bash))
>     => (#<treesit-node (simple_expansion) in 18-22>
>         #<treesit-node (command_substitution) in 29-40>
>         #<treesit-node (simple_expansion) in 47-52>)
>
> This is now expected.
>
>     (point)
>     => 23
>
>     (treesit-node-first-child-for-pos (treesit-node-at (point) 'bash) (point))
>     => nil
>
> This is not expected. One would expect this to return
> command_substitution node, since it follows the position 23.

Ah, I (finally) see what you mean. I agree that it should return the
command_substitution node. It seems that tree-sitter returns nil
whenever POS is not in a child node. I need to see what’s the problem
here since treesit-node-first-child-for-pos is just a think wrapper
around the C counterpart.

Yuan

bug#59633: 29.0.50; tree-sitter-bash: Incorrect behaviour of heredocs with expansions

[Yuan Fu]

Yuan Fu <casouri@gmail.com> writes:

> <miha@kamnitnik.top> writes:
>
>> Yuan Fu <casouri@gmail.com> writes:
>>
>>> What’s the return value of (treesit-node-at (point)) at that point?
>>> Probably the command_substitution node, and since it doesn’t have any
>>> children, treesit-node-first-child-for-pos would return nil.
>>
>> (treesit-node-at (point) 'bash)
>> => #<treesit-node (heredoc_body) in 12-56>
>>
>> I forgot to say that point is before "text2", outside of the
>> command_substitution node.
>>
>>> Yuan
>>
>> tee << EOF
>> text1 $var
>> text2 $(echo cmd)
>> text3
>> $var2
>> EOF
>>
>>
>> Filling a test buffer with these contents and placing point before
>> "text2", we get
>>
>>     (treesit-node-at (point) 'bash)
>>     => #<treesit-node (heredoc_body) in 12-56>
>>
>>     (treesit-node-children (treesit-node-at (point) 'bash))
>>     => (#<treesit-node (simple_expansion) in 18-22>
>>         #<treesit-node (command_substitution) in 29-40>
>>         #<treesit-node (simple_expansion) in 47-52>)
>>
>> This is now expected.
>>
>>     (point)
>>     => 23
>>
>>     (treesit-node-first-child-for-pos (treesit-node-at (point) 'bash) (point))
>>     => nil
>>
>> This is not expected. One would expect this to return
>> command_substitution node, since it follows the position 23.
>
> Ah, I (finally) see what you mean. I agree that it should return the
> command_substitution node. It seems that tree-sitter returns nil
> whenever POS is not in a child node. I need to see what’s the problem
> here since treesit-node-first-child-for-pos is just a think wrapper
> around the C counterpart.

Seems to be tree-sitter-bash problem, I submitted an issue at
https://github.com/tree-sitter/tree-sitter-bash/issues/139

Yuan

bug#59633: 29.0.50; tree-sitter-bash: Incorrect behaviour of heredocs with expansions

[miha--- via Bug reports for GNU Emacs, the Swiss army knife of text editors]

[-- Attachment #1: Type: text/plain, Size: 249 bytes --]

Yuan Fu <casouri@gmail.com> writes:

> Seems to be tree-sitter-bash problem, I submitted an issue at
> https://github.com/tree-sitter/tree-sitter-bash/issues/139

Thanks. As this seems to be a tree-sitter-bash problem, I'm closing this
bug.

> Yuan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 861 bytes --]