bug#66674: 30.0.50; Upstream tree-sitter and treesit disagree about fields

bug#66674: 30.0.50; Upstream tree-sitter and treesit disagree about fields

[Dominik Honnef]

Using tree-sitter's CLI as well as the publicly hosted playground
produce different parse trees than treesit in Emacs. Specifically, the
assignment of nodes to named fields differs.

Given the following C source:

    void main() {
      int x = // foo
        1+
        // comment
        2;
    }

treesit-explore-mode displays the following tree:

    (translation_unit
     (function_definition type: (primitive_type)
      declarator: 
       (function_declarator declarator: (identifier)
        parameters: (parameter_list ( )))
      body: 
       (compound_statement {
        (declaration type: (primitive_type)
         declarator: 
          (init_declarator declarator: (identifier) = value: (comment)
           (binary_expression left: (number_literal) operator: + right: (comment) (number_literal)))
         ;)
        })))

Note how in the init_declarator node, the 'value' field is a comment
node, and similarly for the 'right' field in the binary_expression node.

Running 'tree-sitter parse file.c', on the other hand, produces the
following tree:

    (translation_unit [0, 0] - [6, 0]
      (function_definition [0, 0] - [5, 1]
        type: (primitive_type [0, 0] - [0, 4])
        declarator: (function_declarator [0, 5] - [0, 11]
          declarator: (identifier [0, 5] - [0, 9])
          parameters: (parameter_list [0, 9] - [0, 11]))
        body: (compound_statement [0, 12] - [5, 1]
          (declaration [1, 2] - [4, 6]
            type: (primitive_type [1, 2] - [1, 5])
            declarator: (init_declarator [1, 6] - [4, 5]
              declarator: (identifier [1, 6] - [1, 7])
              (comment [1, 10] - [1, 16])
              value: (binary_expression [2, 4] - [4, 5]
                left: (number_literal [2, 4] - [2, 5])
                (comment [3, 4] - [3, 14])
                right: (number_literal [4, 4] - [4, 5])))))))

Here, the two comment nodes appear as unnamed nodes. IMHO the second
tree is a more useful one, as the named fields contain the semantically
important subtrees (e.g. a binary expression is made up of a left and
right subtree, not a left subtree, a right comment, and then some
unnamed subtree.)

Emacs's tree makes writing queries less convenient, as instead of being
able to refer to well-defined names, one has to rely on child indices to
account for comments.


Further mismatch arises from repeated fields and separators.

Consider the following Go source:

    package pkg
    
    var a, b, c = 1, 2, 3

treesit-explore-mode displays the following tree:

    (source_file
     (package_clause package (package_identifier))
     \n
     (var_declaration var
      (var_spec name: (identifier) name: , (identifier) value: , (identifier) =
       (expression_list (int_literal) , (int_literal) , (int_literal))))
     \n)

Here, the var_spec node has two fields named 'name' even though the
source specifies three names. Furthermore, The second 'name', as well as
'value' are set to the ',' separator between identifiers. Two of the three
identifiers aren't named.

'tree-sitter parse file.go', on the other hand, produces this more
accurate tree:

    (source_file [0, 0] - [2, 21]
      (package_clause [0, 0] - [0, 11]
        (package_identifier [0, 8] - [0, 11]))
      (var_declaration [2, 0] - [2, 21]
        (var_spec [2, 4] - [2, 21]
          name: (identifier [2, 4] - [2, 5])
          name: (identifier [2, 7] - [2, 8])
          name: (identifier [2, 10] - [2, 11])
          value: (expression_list [2, 14] - [2, 21]
            (int_literal [2, 14] - [2, 15])
            (int_literal [2, 17] - [2, 18])
            (int_literal [2, 20] - [2, 21])))))

This reproduces with 29.1 as well as 30.0.50.

bug#66674: 30.0.50; Upstream tree-sitter and treesit disagree about fields

[Eli Zaretskii]

> From: Dominik Honnef <dominik@honnef.co>
> Date: Sat, 21 Oct 2023 22:36:30 +0200
> 
> Using tree-sitter's CLI as well as the publicly hosted playground
> produce different parse trees than treesit in Emacs. Specifically, the
> assignment of nodes to named fields differs.
> 
> Given the following C source:
> 
>     void main() {
>       int x = // foo
>         1+
>         // comment
>         2;
>     }
> 
> treesit-explore-mode displays the following tree:
> 
>     (translation_unit
>      (function_definition type: (primitive_type)
>       declarator: 
>        (function_declarator declarator: (identifier)
>         parameters: (parameter_list ( )))
>       body: 
>        (compound_statement {
>         (declaration type: (primitive_type)
>          declarator: 
>           (init_declarator declarator: (identifier) = value: (comment)
>            (binary_expression left: (number_literal) operator: + right: (comment) (number_literal)))
>          ;)
>         })))
> 
> Note how in the init_declarator node, the 'value' field is a comment
> node, and similarly for the 'right' field in the binary_expression node.
> 
> Running 'tree-sitter parse file.c', on the other hand, produces the
> following tree:
> 
>     (translation_unit [0, 0] - [6, 0]
>       (function_definition [0, 0] - [5, 1]
>         type: (primitive_type [0, 0] - [0, 4])
>         declarator: (function_declarator [0, 5] - [0, 11]
>           declarator: (identifier [0, 5] - [0, 9])
>           parameters: (parameter_list [0, 9] - [0, 11]))
>         body: (compound_statement [0, 12] - [5, 1]
>           (declaration [1, 2] - [4, 6]
>             type: (primitive_type [1, 2] - [1, 5])
>             declarator: (init_declarator [1, 6] - [4, 5]
>               declarator: (identifier [1, 6] - [1, 7])
>               (comment [1, 10] - [1, 16])
>               value: (binary_expression [2, 4] - [4, 5]
>                 left: (number_literal [2, 4] - [2, 5])
>                 (comment [3, 4] - [3, 14])
>                 right: (number_literal [4, 4] - [4, 5])))))))
> 
> Here, the two comment nodes appear as unnamed nodes. IMHO the second
> tree is a more useful one, as the named fields contain the semantically
> important subtrees (e.g. a binary expression is made up of a left and
> right subtree, not a left subtree, a right comment, and then some
> unnamed subtree.)
> 
> Emacs's tree makes writing queries less convenient, as instead of being
> able to refer to well-defined names, one has to rely on child indices to
> account for comments.
> 
> 
> Further mismatch arises from repeated fields and separators.
> 
> Consider the following Go source:
> 
>     package pkg
>     
>     var a, b, c = 1, 2, 3
> 
> treesit-explore-mode displays the following tree:
> 
>     (source_file
>      (package_clause package (package_identifier))
>      \n
>      (var_declaration var
>       (var_spec name: (identifier) name: , (identifier) value: , (identifier) =
>        (expression_list (int_literal) , (int_literal) , (int_literal))))
>      \n)
> 
> Here, the var_spec node has two fields named 'name' even though the
> source specifies three names. Furthermore, The second 'name', as well as
> 'value' are set to the ',' separator between identifiers. Two of the three
> identifiers aren't named.
> 
> 'tree-sitter parse file.go', on the other hand, produces this more
> accurate tree:
> 
>     (source_file [0, 0] - [2, 21]
>       (package_clause [0, 0] - [0, 11]
>         (package_identifier [0, 8] - [0, 11]))
>       (var_declaration [2, 0] - [2, 21]
>         (var_spec [2, 4] - [2, 21]
>           name: (identifier [2, 4] - [2, 5])
>           name: (identifier [2, 7] - [2, 8])
>           name: (identifier [2, 10] - [2, 11])
>           value: (expression_list [2, 14] - [2, 21]
>             (int_literal [2, 14] - [2, 15])
>             (int_literal [2, 17] - [2, 18])
>             (int_literal [2, 20] - [2, 21])))))
> 
> This reproduces with 29.1 as well as 30.0.50.

Yuan, any comments or suggestions?

bug#66674: 30.0.50; Upstream tree-sitter and treesit disagree about fields

[Eli Zaretskii]

Ping!  Yuan, any comments?

> Cc: 66674@debbugs.gnu.org
> Date: Wed, 25 Oct 2023 16:03:10 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> 
> > From: Dominik Honnef <dominik@honnef.co>
> > Date: Sat, 21 Oct 2023 22:36:30 +0200
> > 
> > Using tree-sitter's CLI as well as the publicly hosted playground
> > produce different parse trees than treesit in Emacs. Specifically, the
> > assignment of nodes to named fields differs.
> > 
> > Given the following C source:
> > 
> >     void main() {
> >       int x = // foo
> >         1+
> >         // comment
> >         2;
> >     }
> > 
> > treesit-explore-mode displays the following tree:
> > 
> >     (translation_unit
> >      (function_definition type: (primitive_type)
> >       declarator: 
> >        (function_declarator declarator: (identifier)
> >         parameters: (parameter_list ( )))
> >       body: 
> >        (compound_statement {
> >         (declaration type: (primitive_type)
> >          declarator: 
> >           (init_declarator declarator: (identifier) = value: (comment)
> >            (binary_expression left: (number_literal) operator: + right: (comment) (number_literal)))
> >          ;)
> >         })))
> > 
> > Note how in the init_declarator node, the 'value' field is a comment
> > node, and similarly for the 'right' field in the binary_expression node.
> > 
> > Running 'tree-sitter parse file.c', on the other hand, produces the
> > following tree:
> > 
> >     (translation_unit [0, 0] - [6, 0]
> >       (function_definition [0, 0] - [5, 1]
> >         type: (primitive_type [0, 0] - [0, 4])
> >         declarator: (function_declarator [0, 5] - [0, 11]
> >           declarator: (identifier [0, 5] - [0, 9])
> >           parameters: (parameter_list [0, 9] - [0, 11]))
> >         body: (compound_statement [0, 12] - [5, 1]
> >           (declaration [1, 2] - [4, 6]
> >             type: (primitive_type [1, 2] - [1, 5])
> >             declarator: (init_declarator [1, 6] - [4, 5]
> >               declarator: (identifier [1, 6] - [1, 7])
> >               (comment [1, 10] - [1, 16])
> >               value: (binary_expression [2, 4] - [4, 5]
> >                 left: (number_literal [2, 4] - [2, 5])
> >                 (comment [3, 4] - [3, 14])
> >                 right: (number_literal [4, 4] - [4, 5])))))))
> > 
> > Here, the two comment nodes appear as unnamed nodes. IMHO the second
> > tree is a more useful one, as the named fields contain the semantically
> > important subtrees (e.g. a binary expression is made up of a left and
> > right subtree, not a left subtree, a right comment, and then some
> > unnamed subtree.)
> > 
> > Emacs's tree makes writing queries less convenient, as instead of being
> > able to refer to well-defined names, one has to rely on child indices to
> > account for comments.
> > 
> > 
> > Further mismatch arises from repeated fields and separators.
> > 
> > Consider the following Go source:
> > 
> >     package pkg
> >     
> >     var a, b, c = 1, 2, 3
> > 
> > treesit-explore-mode displays the following tree:
> > 
> >     (source_file
> >      (package_clause package (package_identifier))
> >      \n
> >      (var_declaration var
> >       (var_spec name: (identifier) name: , (identifier) value: , (identifier) =
> >        (expression_list (int_literal) , (int_literal) , (int_literal))))
> >      \n)
> > 
> > Here, the var_spec node has two fields named 'name' even though the
> > source specifies three names. Furthermore, The second 'name', as well as
> > 'value' are set to the ',' separator between identifiers. Two of the three
> > identifiers aren't named.
> > 
> > 'tree-sitter parse file.go', on the other hand, produces this more
> > accurate tree:
> > 
> >     (source_file [0, 0] - [2, 21]
> >       (package_clause [0, 0] - [0, 11]
> >         (package_identifier [0, 8] - [0, 11]))
> >       (var_declaration [2, 0] - [2, 21]
> >         (var_spec [2, 4] - [2, 21]
> >           name: (identifier [2, 4] - [2, 5])
> >           name: (identifier [2, 7] - [2, 8])
> >           name: (identifier [2, 10] - [2, 11])
> >           value: (expression_list [2, 14] - [2, 21]
> >             (int_literal [2, 14] - [2, 15])
> >             (int_literal [2, 17] - [2, 18])
> >             (int_literal [2, 20] - [2, 21])))))
> > 
> > This reproduces with 29.1 as well as 30.0.50.
> 
> Yuan, any comments or suggestions?
> 
> 
> 
> 

bug#66674: 30.0.50; Upstream tree-sitter and treesit disagree about fields

[Eli Zaretskii]

Ping! Ping!  Yuan, please chime in.

> Cc: 66674@debbugs.gnu.org, dominik@honnef.co
> Date: Sun, 19 Nov 2023 12:08:08 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> 
> Ping!  Yuan, any comments?
> 
> > Cc: 66674@debbugs.gnu.org
> > Date: Wed, 25 Oct 2023 16:03:10 +0300
> > From: Eli Zaretskii <eliz@gnu.org>
> > 
> > > From: Dominik Honnef <dominik@honnef.co>
> > > Date: Sat, 21 Oct 2023 22:36:30 +0200
> > > 
> > > Using tree-sitter's CLI as well as the publicly hosted playground
> > > produce different parse trees than treesit in Emacs. Specifically, the
> > > assignment of nodes to named fields differs.
> > > 
> > > Given the following C source:
> > > 
> > >     void main() {
> > >       int x = // foo
> > >         1+
> > >         // comment
> > >         2;
> > >     }
> > > 
> > > treesit-explore-mode displays the following tree:
> > > 
> > >     (translation_unit
> > >      (function_definition type: (primitive_type)
> > >       declarator: 
> > >        (function_declarator declarator: (identifier)
> > >         parameters: (parameter_list ( )))
> > >       body: 
> > >        (compound_statement {
> > >         (declaration type: (primitive_type)
> > >          declarator: 
> > >           (init_declarator declarator: (identifier) = value: (comment)
> > >            (binary_expression left: (number_literal) operator: + right: (comment) (number_literal)))
> > >          ;)
> > >         })))
> > > 
> > > Note how in the init_declarator node, the 'value' field is a comment
> > > node, and similarly for the 'right' field in the binary_expression node.
> > > 
> > > Running 'tree-sitter parse file.c', on the other hand, produces the
> > > following tree:
> > > 
> > >     (translation_unit [0, 0] - [6, 0]
> > >       (function_definition [0, 0] - [5, 1]
> > >         type: (primitive_type [0, 0] - [0, 4])
> > >         declarator: (function_declarator [0, 5] - [0, 11]
> > >           declarator: (identifier [0, 5] - [0, 9])
> > >           parameters: (parameter_list [0, 9] - [0, 11]))
> > >         body: (compound_statement [0, 12] - [5, 1]
> > >           (declaration [1, 2] - [4, 6]
> > >             type: (primitive_type [1, 2] - [1, 5])
> > >             declarator: (init_declarator [1, 6] - [4, 5]
> > >               declarator: (identifier [1, 6] - [1, 7])
> > >               (comment [1, 10] - [1, 16])
> > >               value: (binary_expression [2, 4] - [4, 5]
> > >                 left: (number_literal [2, 4] - [2, 5])
> > >                 (comment [3, 4] - [3, 14])
> > >                 right: (number_literal [4, 4] - [4, 5])))))))
> > > 
> > > Here, the two comment nodes appear as unnamed nodes. IMHO the second
> > > tree is a more useful one, as the named fields contain the semantically
> > > important subtrees (e.g. a binary expression is made up of a left and
> > > right subtree, not a left subtree, a right comment, and then some
> > > unnamed subtree.)
> > > 
> > > Emacs's tree makes writing queries less convenient, as instead of being
> > > able to refer to well-defined names, one has to rely on child indices to
> > > account for comments.
> > > 
> > > 
> > > Further mismatch arises from repeated fields and separators.
> > > 
> > > Consider the following Go source:
> > > 
> > >     package pkg
> > >     
> > >     var a, b, c = 1, 2, 3
> > > 
> > > treesit-explore-mode displays the following tree:
> > > 
> > >     (source_file
> > >      (package_clause package (package_identifier))
> > >      \n
> > >      (var_declaration var
> > >       (var_spec name: (identifier) name: , (identifier) value: , (identifier) =
> > >        (expression_list (int_literal) , (int_literal) , (int_literal))))
> > >      \n)
> > > 
> > > Here, the var_spec node has two fields named 'name' even though the
> > > source specifies three names. Furthermore, The second 'name', as well as
> > > 'value' are set to the ',' separator between identifiers. Two of the three
> > > identifiers aren't named.
> > > 
> > > 'tree-sitter parse file.go', on the other hand, produces this more
> > > accurate tree:
> > > 
> > >     (source_file [0, 0] - [2, 21]
> > >       (package_clause [0, 0] - [0, 11]
> > >         (package_identifier [0, 8] - [0, 11]))
> > >       (var_declaration [2, 0] - [2, 21]
> > >         (var_spec [2, 4] - [2, 21]
> > >           name: (identifier [2, 4] - [2, 5])
> > >           name: (identifier [2, 7] - [2, 8])
> > >           name: (identifier [2, 10] - [2, 11])
> > >           value: (expression_list [2, 14] - [2, 21]
> > >             (int_literal [2, 14] - [2, 15])
> > >             (int_literal [2, 17] - [2, 18])
> > >             (int_literal [2, 20] - [2, 21])))))
> > > 
> > > This reproduces with 29.1 as well as 30.0.50.
> > 
> > Yuan, any comments or suggestions?
> > 
> > 
> > 
> > 
> 
> 
> 
> 

bug#66674: 30.0.50; Upstream tree-sitter and treesit disagree about fields

[Yuan Fu]

On 11/25/23 2:03 AM, Eli Zaretskii wrote:
> Ping! Ping!  Yuan, please chime in.
>
>> Cc: 66674@debbugs.gnu.org, dominik@honnef.co
>> Date: Sun, 19 Nov 2023 12:08:08 +0200
>> From: Eli Zaretskii <eliz@gnu.org>
>>
>> Ping!  Yuan, any comments?
>>
>>> Cc: 66674@debbugs.gnu.org
>>> Date: Wed, 25 Oct 2023 16:03:10 +0300
>>> From: Eli Zaretskii <eliz@gnu.org>
>>>
>>>> From: Dominik Honnef <dominik@honnef.co>
>>>> Date: Sat, 21 Oct 2023 22:36:30 +0200
>>>>
>>>> Using tree-sitter's CLI as well as the publicly hosted playground
>>>> produce different parse trees than treesit in Emacs. Specifically, the
>>>> assignment of nodes to named fields differs.
>>>>
>>>> Given the following C source:
>>>>
>>>>      void main() {
>>>>        int x = // foo
>>>>          1+
>>>>          // comment
>>>>          2;
>>>>      }
>>>>
>>>> treesit-explore-mode displays the following tree:
>>>>
>>>>      (translation_unit
>>>>       (function_definition type: (primitive_type)
>>>>        declarator:
>>>>         (function_declarator declarator: (identifier)
>>>>          parameters: (parameter_list ( )))
>>>>        body:
>>>>         (compound_statement {
>>>>          (declaration type: (primitive_type)
>>>>           declarator:
>>>>            (init_declarator declarator: (identifier) = value: (comment)
>>>>             (binary_expression left: (number_literal) operator: + right: (comment) (number_literal)))
>>>>           ;)
>>>>          })))
>>>>
>>>> Note how in the init_declarator node, the 'value' field is a comment
>>>> node, and similarly for the 'right' field in the binary_expression node.
>>>>
>>>> Running 'tree-sitter parse file.c', on the other hand, produces the
>>>> following tree:
>>>>
>>>>      (translation_unit [0, 0] - [6, 0]
>>>>        (function_definition [0, 0] - [5, 1]
>>>>          type: (primitive_type [0, 0] - [0, 4])
>>>>          declarator: (function_declarator [0, 5] - [0, 11]
>>>>            declarator: (identifier [0, 5] - [0, 9])
>>>>            parameters: (parameter_list [0, 9] - [0, 11]))
>>>>          body: (compound_statement [0, 12] - [5, 1]
>>>>            (declaration [1, 2] - [4, 6]
>>>>              type: (primitive_type [1, 2] - [1, 5])
>>>>              declarator: (init_declarator [1, 6] - [4, 5]
>>>>                declarator: (identifier [1, 6] - [1, 7])
>>>>                (comment [1, 10] - [1, 16])
>>>>                value: (binary_expression [2, 4] - [4, 5]
>>>>                  left: (number_literal [2, 4] - [2, 5])
>>>>                  (comment [3, 4] - [3, 14])
>>>>                  right: (number_literal [4, 4] - [4, 5])))))))
>>>>
>>>> Here, the two comment nodes appear as unnamed nodes. IMHO the second
>>>> tree is a more useful one, as the named fields contain the semantically
>>>> important subtrees (e.g. a binary expression is made up of a left and
>>>> right subtree, not a left subtree, a right comment, and then some
>>>> unnamed subtree.)
>>>>
>>>> Emacs's tree makes writing queries less convenient, as instead of being
>>>> able to refer to well-defined names, one has to rely on child indices to
>>>> account for comments.
>>>>
>>>>
>>>> Further mismatch arises from repeated fields and separators.
>>>>
>>>> Consider the following Go source:
>>>>
>>>>      package pkg
>>>>      
>>>>      var a, b, c = 1, 2, 3
>>>>
>>>> treesit-explore-mode displays the following tree:
>>>>
>>>>      (source_file
>>>>       (package_clause package (package_identifier))
>>>>       \n
>>>>       (var_declaration var
>>>>        (var_spec name: (identifier) name: , (identifier) value: , (identifier) =
>>>>         (expression_list (int_literal) , (int_literal) , (int_literal))))
>>>>       \n)
>>>>
>>>> Here, the var_spec node has two fields named 'name' even though the
>>>> source specifies three names. Furthermore, The second 'name', as well as
>>>> 'value' are set to the ',' separator between identifiers. Two of the three
>>>> identifiers aren't named.
>>>>
>>>> 'tree-sitter parse file.go', on the other hand, produces this more
>>>> accurate tree:
>>>>
>>>>      (source_file [0, 0] - [2, 21]
>>>>        (package_clause [0, 0] - [0, 11]
>>>>          (package_identifier [0, 8] - [0, 11]))
>>>>        (var_declaration [2, 0] - [2, 21]
>>>>          (var_spec [2, 4] - [2, 21]
>>>>            name: (identifier [2, 4] - [2, 5])
>>>>            name: (identifier [2, 7] - [2, 8])
>>>>            name: (identifier [2, 10] - [2, 11])
>>>>            value: (expression_list [2, 14] - [2, 21]
>>>>              (int_literal [2, 14] - [2, 15])
>>>>              (int_literal [2, 17] - [2, 18])
>>>>              (int_literal [2, 20] - [2, 21])))))
>>>>
>>>> This reproduces with 29.1 as well as 30.0.50.
>>> Yuan, any comments or suggestions?

Sorry sorry sorry, another missed report. I think this is a bug in 
treesit-explore-mode, I'll work on fixing it!

Yuan

bug#66674: 30.0.50; Upstream tree-sitter and treesit disagree about fields

[Dominik Honnef]

Yuan Fu <casouri@gmail.com> writes:

> On 11/25/23 2:03 AM, Eli Zaretskii wrote:
>> Ping! Ping!  Yuan, please chime in.
>>
>>> Cc: 66674@debbugs.gnu.org, dominik@honnef.co
>>> Date: Sun, 19 Nov 2023 12:08:08 +0200
>>> From: Eli Zaretskii <eliz@gnu.org>
>>>
>>> Ping!  Yuan, any comments?
>>>
>>>> Cc: 66674@debbugs.gnu.org
>>>> Date: Wed, 25 Oct 2023 16:03:10 +0300
>>>> From: Eli Zaretskii <eliz@gnu.org>
>>>>
>>>>> From: Dominik Honnef <dominik@honnef.co>
>>>>> Date: Sat, 21 Oct 2023 22:36:30 +0200
>>>>>
>>>>> Using tree-sitter's CLI as well as the publicly hosted playground
>>>>> produce different parse trees than treesit in Emacs. Specifically, the
>>>>> assignment of nodes to named fields differs.
>>>>>
>>>>> Given the following C source:
>>>>>
>>>>>      void main() {
>>>>>        int x = // foo
>>>>>          1+
>>>>>          // comment
>>>>>          2;
>>>>>      }
>>>>>
>>>>> treesit-explore-mode displays the following tree:
>>>>>
>>>>>      (translation_unit
>>>>>       (function_definition type: (primitive_type)
>>>>>        declarator:
>>>>>         (function_declarator declarator: (identifier)
>>>>>          parameters: (parameter_list ( )))
>>>>>        body:
>>>>>         (compound_statement {
>>>>>          (declaration type: (primitive_type)
>>>>>           declarator:
>>>>>            (init_declarator declarator: (identifier) = value: (comment)
>>>>>             (binary_expression left: (number_literal) operator: + right: (comment) (number_literal)))
>>>>>           ;)
>>>>>          })))
>>>>>
>>>>> Note how in the init_declarator node, the 'value' field is a comment
>>>>> node, and similarly for the 'right' field in the binary_expression node.
>>>>>
>>>>> Running 'tree-sitter parse file.c', on the other hand, produces the
>>>>> following tree:
>>>>>
>>>>>      (translation_unit [0, 0] - [6, 0]
>>>>>        (function_definition [0, 0] - [5, 1]
>>>>>          type: (primitive_type [0, 0] - [0, 4])
>>>>>          declarator: (function_declarator [0, 5] - [0, 11]
>>>>>            declarator: (identifier [0, 5] - [0, 9])
>>>>>            parameters: (parameter_list [0, 9] - [0, 11]))
>>>>>          body: (compound_statement [0, 12] - [5, 1]
>>>>>            (declaration [1, 2] - [4, 6]
>>>>>              type: (primitive_type [1, 2] - [1, 5])
>>>>>              declarator: (init_declarator [1, 6] - [4, 5]
>>>>>                declarator: (identifier [1, 6] - [1, 7])
>>>>>                (comment [1, 10] - [1, 16])
>>>>>                value: (binary_expression [2, 4] - [4, 5]
>>>>>                  left: (number_literal [2, 4] - [2, 5])
>>>>>                  (comment [3, 4] - [3, 14])
>>>>>                  right: (number_literal [4, 4] - [4, 5])))))))
>>>>>
>>>>> Here, the two comment nodes appear as unnamed nodes. IMHO the second
>>>>> tree is a more useful one, as the named fields contain the semantically
>>>>> important subtrees (e.g. a binary expression is made up of a left and
>>>>> right subtree, not a left subtree, a right comment, and then some
>>>>> unnamed subtree.)
>>>>>
>>>>> Emacs's tree makes writing queries less convenient, as instead of being
>>>>> able to refer to well-defined names, one has to rely on child indices to
>>>>> account for comments.
>>>>>
>>>>>
>>>>> Further mismatch arises from repeated fields and separators.
>>>>>
>>>>> Consider the following Go source:
>>>>>
>>>>>      package pkg
>>>>>      
>>>>>      var a, b, c = 1, 2, 3
>>>>>
>>>>> treesit-explore-mode displays the following tree:
>>>>>
>>>>>      (source_file
>>>>>       (package_clause package (package_identifier))
>>>>>       \n
>>>>>       (var_declaration var
>>>>>        (var_spec name: (identifier) name: , (identifier) value: , (identifier) =
>>>>>         (expression_list (int_literal) , (int_literal) , (int_literal))))
>>>>>       \n)
>>>>>
>>>>> Here, the var_spec node has two fields named 'name' even though the
>>>>> source specifies three names. Furthermore, The second 'name', as well as
>>>>> 'value' are set to the ',' separator between identifiers. Two of the three
>>>>> identifiers aren't named.
>>>>>
>>>>> 'tree-sitter parse file.go', on the other hand, produces this more
>>>>> accurate tree:
>>>>>
>>>>>      (source_file [0, 0] - [2, 21]
>>>>>        (package_clause [0, 0] - [0, 11]
>>>>>          (package_identifier [0, 8] - [0, 11]))
>>>>>        (var_declaration [2, 0] - [2, 21]
>>>>>          (var_spec [2, 4] - [2, 21]
>>>>>            name: (identifier [2, 4] - [2, 5])
>>>>>            name: (identifier [2, 7] - [2, 8])
>>>>>            name: (identifier [2, 10] - [2, 11])
>>>>>            value: (expression_list [2, 14] - [2, 21]
>>>>>              (int_literal [2, 14] - [2, 15])
>>>>>              (int_literal [2, 17] - [2, 18])
>>>>>              (int_literal [2, 20] - [2, 21])))))
>>>>>
>>>>> This reproduces with 29.1 as well as 30.0.50.
>>>> Yuan, any comments or suggestions?
>
> Sorry sorry sorry, another missed report. I think this is a bug in 
> treesit-explore-mode, I'll work on fixing it!
>
> Yuan

I don't think that's the case, at least not exclusively. I used
treesit-explore-mode to debug patterns that matched in the playground
but not in Emacs. The matching behavior seemed pretty in line with what
treesit-explore-mode reported.

bug#66674: 30.0.50; Upstream tree-sitter and treesit disagree about fields

[Yuan Fu]

On 12/10/23 6:28 AM, Dominik Honnef wrote:
> Yuan Fu <casouri@gmail.com> writes:
>
>> On 11/25/23 2:03 AM, Eli Zaretskii wrote:
>>> Ping! Ping!  Yuan, please chime in.
>>>
>>>> Cc: 66674@debbugs.gnu.org, dominik@honnef.co
>>>> Date: Sun, 19 Nov 2023 12:08:08 +0200
>>>> From: Eli Zaretskii <eliz@gnu.org>
>>>>
>>>> Ping!  Yuan, any comments?
>>>>
>>>>> Cc: 66674@debbugs.gnu.org
>>>>> Date: Wed, 25 Oct 2023 16:03:10 +0300
>>>>> From: Eli Zaretskii <eliz@gnu.org>
>>>>>
>>>>>> From: Dominik Honnef <dominik@honnef.co>
>>>>>> Date: Sat, 21 Oct 2023 22:36:30 +0200
>>>>>>
>>>>>> Using tree-sitter's CLI as well as the publicly hosted playground
>>>>>> produce different parse trees than treesit in Emacs. Specifically, the
>>>>>> assignment of nodes to named fields differs.
>>>>>>
>>>>>> Given the following C source:
>>>>>>
>>>>>>       void main() {
>>>>>>         int x = // foo
>>>>>>           1+
>>>>>>           // comment
>>>>>>           2;
>>>>>>       }
>>>>>>
>>>>>> treesit-explore-mode displays the following tree:
>>>>>>
>>>>>>       (translation_unit
>>>>>>        (function_definition type: (primitive_type)
>>>>>>         declarator:
>>>>>>          (function_declarator declarator: (identifier)
>>>>>>           parameters: (parameter_list ( )))
>>>>>>         body:
>>>>>>          (compound_statement {
>>>>>>           (declaration type: (primitive_type)
>>>>>>            declarator:
>>>>>>             (init_declarator declarator: (identifier) = value: (comment)
>>>>>>              (binary_expression left: (number_literal) operator: + right: (comment) (number_literal)))
>>>>>>            ;)
>>>>>>           })))
>>>>>>
>>>>>> Note how in the init_declarator node, the 'value' field is a comment
>>>>>> node, and similarly for the 'right' field in the binary_expression node.
>>>>>>
>>>>>> Running 'tree-sitter parse file.c', on the other hand, produces the
>>>>>> following tree:
>>>>>>
>>>>>>       (translation_unit [0, 0] - [6, 0]
>>>>>>         (function_definition [0, 0] - [5, 1]
>>>>>>           type: (primitive_type [0, 0] - [0, 4])
>>>>>>           declarator: (function_declarator [0, 5] - [0, 11]
>>>>>>             declarator: (identifier [0, 5] - [0, 9])
>>>>>>             parameters: (parameter_list [0, 9] - [0, 11]))
>>>>>>           body: (compound_statement [0, 12] - [5, 1]
>>>>>>             (declaration [1, 2] - [4, 6]
>>>>>>               type: (primitive_type [1, 2] - [1, 5])
>>>>>>               declarator: (init_declarator [1, 6] - [4, 5]
>>>>>>                 declarator: (identifier [1, 6] - [1, 7])
>>>>>>                 (comment [1, 10] - [1, 16])
>>>>>>                 value: (binary_expression [2, 4] - [4, 5]
>>>>>>                   left: (number_literal [2, 4] - [2, 5])
>>>>>>                   (comment [3, 4] - [3, 14])
>>>>>>                   right: (number_literal [4, 4] - [4, 5])))))))
>>>>>>
>>>>>> Here, the two comment nodes appear as unnamed nodes. IMHO the second
>>>>>> tree is a more useful one, as the named fields contain the semantically
>>>>>> important subtrees (e.g. a binary expression is made up of a left and
>>>>>> right subtree, not a left subtree, a right comment, and then some
>>>>>> unnamed subtree.)
>>>>>>
>>>>>> Emacs's tree makes writing queries less convenient, as instead of being
>>>>>> able to refer to well-defined names, one has to rely on child indices to
>>>>>> account for comments.
>>>>>>
>>>>>>
>>>>>> Further mismatch arises from repeated fields and separators.
>>>>>>
>>>>>> Consider the following Go source:
>>>>>>
>>>>>>       package pkg
>>>>>>       
>>>>>>       var a, b, c = 1, 2, 3
>>>>>>
>>>>>> treesit-explore-mode displays the following tree:
>>>>>>
>>>>>>       (source_file
>>>>>>        (package_clause package (package_identifier))
>>>>>>        \n
>>>>>>        (var_declaration var
>>>>>>         (var_spec name: (identifier) name: , (identifier) value: , (identifier) =
>>>>>>          (expression_list (int_literal) , (int_literal) , (int_literal))))
>>>>>>        \n)
>>>>>>
>>>>>> Here, the var_spec node has two fields named 'name' even though the
>>>>>> source specifies three names. Furthermore, The second 'name', as well as
>>>>>> 'value' are set to the ',' separator between identifiers. Two of the three
>>>>>> identifiers aren't named.
>>>>>>
>>>>>> 'tree-sitter parse file.go', on the other hand, produces this more
>>>>>> accurate tree:
>>>>>>
>>>>>>       (source_file [0, 0] - [2, 21]
>>>>>>         (package_clause [0, 0] - [0, 11]
>>>>>>           (package_identifier [0, 8] - [0, 11]))
>>>>>>         (var_declaration [2, 0] - [2, 21]
>>>>>>           (var_spec [2, 4] - [2, 21]
>>>>>>             name: (identifier [2, 4] - [2, 5])
>>>>>>             name: (identifier [2, 7] - [2, 8])
>>>>>>             name: (identifier [2, 10] - [2, 11])
>>>>>>             value: (expression_list [2, 14] - [2, 21]
>>>>>>               (int_literal [2, 14] - [2, 15])
>>>>>>               (int_literal [2, 17] - [2, 18])
>>>>>>               (int_literal [2, 20] - [2, 21])))))
>>>>>>
>>>>>> This reproduces with 29.1 as well as 30.0.50.
>>>>> Yuan, any comments or suggestions?
>> Sorry sorry sorry, another missed report. I think this is a bug in
>> treesit-explore-mode, I'll work on fixing it!
>>
>> Yuan
> I don't think that's the case, at least not exclusively. I used
> treesit-explore-mode to debug patterns that matched in the playground
> but not in Emacs. The matching behavior seemed pretty in line with what
> treesit-explore-mode reported.
I do find that treesit-node-field-name are returning wrong field names, 
that's why in the first example, you see the "value" field name given to 
the comment node, rather than the binary_expression behind it. In the 
actual parse tree, "value" belongs to binary_expression. With the fixed 
I just pushed to emacs-29, the explorer parse tree for the first example 
becomes

(translation_unit
  (function_definition type: (primitive_type)
   declarator:
    (function_declarator declarator: (identifier)
     parameters: (parameter_list ( )))
   body:
    (compound_statement {
     (declaration type: (primitive_type)
      declarator:
       (init_declarator declarator: (identifier) = (comment)
        value: (binary_expression left: (number_literal) operator: +
                                  operator: (comment)
                right: (number_literal)))
      ;)
     })))

which should match the playground.

If you can find the pattern that matches in the playground but doesn't 
in Emacs, do please post it and I can see if there's anything wrong.

Yuan