bug#66674: 30.0.50; Upstream tree-sitter and treesit disagree about fields

bug#66674: 30.0.50; Upstream tree-sitter and treesit disagree about fields

[Dominik Honnef]

Using tree-sitter's CLI as well as the publicly hosted playground
produce different parse trees than treesit in Emacs. Specifically, the
assignment of nodes to named fields differs.

Given the following C source:

    void main() {
      int x = // foo
        1+
        // comment
        2;
    }

treesit-explore-mode displays the following tree:

    (translation_unit
     (function_definition type: (primitive_type)
      declarator: 
       (function_declarator declarator: (identifier)
        parameters: (parameter_list ( )))
      body: 
       (compound_statement {
        (declaration type: (primitive_type)
         declarator: 
          (init_declarator declarator: (identifier) = value: (comment)
           (binary_expression left: (number_literal) operator: + right: (comment) (number_literal)))
         ;)
        })))

Note how in the init_declarator node, the 'value' field is a comment
node, and similarly for the 'right' field in the binary_expression node.

Running 'tree-sitter parse file.c', on the other hand, produces the
following tree:

    (translation_unit [0, 0] - [6, 0]
      (function_definition [0, 0] - [5, 1]
        type: (primitive_type [0, 0] - [0, 4])
        declarator: (function_declarator [0, 5] - [0, 11]
          declarator: (identifier [0, 5] - [0, 9])
          parameters: (parameter_list [0, 9] - [0, 11]))
        body: (compound_statement [0, 12] - [5, 1]
          (declaration [1, 2] - [4, 6]
            type: (primitive_type [1, 2] - [1, 5])
            declarator: (init_declarator [1, 6] - [4, 5]
              declarator: (identifier [1, 6] - [1, 7])
              (comment [1, 10] - [1, 16])
              value: (binary_expression [2, 4] - [4, 5]
                left: (number_literal [2, 4] - [2, 5])
                (comment [3, 4] - [3, 14])
                right: (number_literal [4, 4] - [4, 5])))))))

Here, the two comment nodes appear as unnamed nodes. IMHO the second
tree is a more useful one, as the named fields contain the semantically
important subtrees (e.g. a binary expression is made up of a left and
right subtree, not a left subtree, a right comment, and then some
unnamed subtree.)

Emacs's tree makes writing queries less convenient, as instead of being
able to refer to well-defined names, one has to rely on child indices to
account for comments.


Further mismatch arises from repeated fields and separators.

Consider the following Go source:

    package pkg
    
    var a, b, c = 1, 2, 3

treesit-explore-mode displays the following tree:

    (source_file
     (package_clause package (package_identifier))
     \n
     (var_declaration var
      (var_spec name: (identifier) name: , (identifier) value: , (identifier) =
       (expression_list (int_literal) , (int_literal) , (int_literal))))
     \n)

Here, the var_spec node has two fields named 'name' even though the
source specifies three names. Furthermore, The second 'name', as well as
'value' are set to the ',' separator between identifiers. Two of the three
identifiers aren't named.

'tree-sitter parse file.go', on the other hand, produces this more
accurate tree:

    (source_file [0, 0] - [2, 21]
      (package_clause [0, 0] - [0, 11]
        (package_identifier [0, 8] - [0, 11]))
      (var_declaration [2, 0] - [2, 21]
        (var_spec [2, 4] - [2, 21]
          name: (identifier [2, 4] - [2, 5])
          name: (identifier [2, 7] - [2, 8])
          name: (identifier [2, 10] - [2, 11])
          value: (expression_list [2, 14] - [2, 21]
            (int_literal [2, 14] - [2, 15])
            (int_literal [2, 17] - [2, 18])
            (int_literal [2, 20] - [2, 21])))))

This reproduces with 29.1 as well as 30.0.50.

bug#66674: 30.0.50; Upstream tree-sitter and treesit disagree about fields

[Eli Zaretskii]

> From: Dominik Honnef <dominik@honnef.co>
> Date: Sat, 21 Oct 2023 22:36:30 +0200
> 
> Using tree-sitter's CLI as well as the publicly hosted playground
> produce different parse trees than treesit in Emacs. Specifically, the
> assignment of nodes to named fields differs.
> 
> Given the following C source:
> 
>     void main() {
>       int x = // foo
>         1+
>         // comment
>         2;
>     }
> 
> treesit-explore-mode displays the following tree:
> 
>     (translation_unit
>      (function_definition type: (primitive_type)
>       declarator: 
>        (function_declarator declarator: (identifier)
>         parameters: (parameter_list ( )))
>       body: 
>        (compound_statement {
>         (declaration type: (primitive_type)
>          declarator: 
>           (init_declarator declarator: (identifier) = value: (comment)
>            (binary_expression left: (number_literal) operator: + right: (comment) (number_literal)))
>          ;)
>         })))
> 
> Note how in the init_declarator node, the 'value' field is a comment
> node, and similarly for the 'right' field in the binary_expression node.
> 
> Running 'tree-sitter parse file.c', on the other hand, produces the
> following tree:
> 
>     (translation_unit [0, 0] - [6, 0]
>       (function_definition [0, 0] - [5, 1]
>         type: (primitive_type [0, 0] - [0, 4])
>         declarator: (function_declarator [0, 5] - [0, 11]
>           declarator: (identifier [0, 5] - [0, 9])
>           parameters: (parameter_list [0, 9] - [0, 11]))
>         body: (compound_statement [0, 12] - [5, 1]
>           (declaration [1, 2] - [4, 6]
>             type: (primitive_type [1, 2] - [1, 5])
>             declarator: (init_declarator [1, 6] - [4, 5]
>               declarator: (identifier [1, 6] - [1, 7])
>               (comment [1, 10] - [1, 16])
>               value: (binary_expression [2, 4] - [4, 5]
>                 left: (number_literal [2, 4] - [2, 5])
>                 (comment [3, 4] - [3, 14])
>                 right: (number_literal [4, 4] - [4, 5])))))))
> 
> Here, the two comment nodes appear as unnamed nodes. IMHO the second
> tree is a more useful one, as the named fields contain the semantically
> important subtrees (e.g. a binary expression is made up of a left and
> right subtree, not a left subtree, a right comment, and then some
> unnamed subtree.)
> 
> Emacs's tree makes writing queries less convenient, as instead of being
> able to refer to well-defined names, one has to rely on child indices to
> account for comments.
> 
> 
> Further mismatch arises from repeated fields and separators.
> 
> Consider the following Go source:
> 
>     package pkg
>     
>     var a, b, c = 1, 2, 3
> 
> treesit-explore-mode displays the following tree:
> 
>     (source_file
>      (package_clause package (package_identifier))
>      \n
>      (var_declaration var
>       (var_spec name: (identifier) name: , (identifier) value: , (identifier) =
>        (expression_list (int_literal) , (int_literal) , (int_literal))))
>      \n)
> 
> Here, the var_spec node has two fields named 'name' even though the
> source specifies three names. Furthermore, The second 'name', as well as
> 'value' are set to the ',' separator between identifiers. Two of the three
> identifiers aren't named.
> 
> 'tree-sitter parse file.go', on the other hand, produces this more
> accurate tree:
> 
>     (source_file [0, 0] - [2, 21]
>       (package_clause [0, 0] - [0, 11]
>         (package_identifier [0, 8] - [0, 11]))
>       (var_declaration [2, 0] - [2, 21]
>         (var_spec [2, 4] - [2, 21]
>           name: (identifier [2, 4] - [2, 5])
>           name: (identifier [2, 7] - [2, 8])
>           name: (identifier [2, 10] - [2, 11])
>           value: (expression_list [2, 14] - [2, 21]
>             (int_literal [2, 14] - [2, 15])
>             (int_literal [2, 17] - [2, 18])
>             (int_literal [2, 20] - [2, 21])))))
> 
> This reproduces with 29.1 as well as 30.0.50.

Yuan, any comments or suggestions?

bug#66674: 30.0.50; Upstream tree-sitter and treesit disagree about fields

[Eli Zaretskii]

Ping!  Yuan, any comments?

> Cc: 66674@debbugs.gnu.org
> Date: Wed, 25 Oct 2023 16:03:10 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> 
> > From: Dominik Honnef <dominik@honnef.co>
> > Date: Sat, 21 Oct 2023 22:36:30 +0200
> > 
> > Using tree-sitter's CLI as well as the publicly hosted playground
> > produce different parse trees than treesit in Emacs. Specifically, the
> > assignment of nodes to named fields differs.
> > 
> > Given the following C source:
> > 
> >     void main() {
> >       int x = // foo
> >         1+
> >         // comment
> >         2;
> >     }
> > 
> > treesit-explore-mode displays the following tree:
> > 
> >     (translation_unit
> >      (function_definition type: (primitive_type)
> >       declarator: 
> >        (function_declarator declarator: (identifier)
> >         parameters: (parameter_list ( )))
> >       body: 
> >        (compound_statement {
> >         (declaration type: (primitive_type)
> >          declarator: 
> >           (init_declarator declarator: (identifier) = value: (comment)
> >            (binary_expression left: (number_literal) operator: + right: (comment) (number_literal)))
> >          ;)
> >         })))
> > 
> > Note how in the init_declarator node, the 'value' field is a comment
> > node, and similarly for the 'right' field in the binary_expression node.
> > 
> > Running 'tree-sitter parse file.c', on the other hand, produces the
> > following tree:
> > 
> >     (translation_unit [0, 0] - [6, 0]
> >       (function_definition [0, 0] - [5, 1]
> >         type: (primitive_type [0, 0] - [0, 4])
> >         declarator: (function_declarator [0, 5] - [0, 11]
> >           declarator: (identifier [0, 5] - [0, 9])
> >           parameters: (parameter_list [0, 9] - [0, 11]))
> >         body: (compound_statement [0, 12] - [5, 1]
> >           (declaration [1, 2] - [4, 6]
> >             type: (primitive_type [1, 2] - [1, 5])
> >             declarator: (init_declarator [1, 6] - [4, 5]
> >               declarator: (identifier [1, 6] - [1, 7])
> >               (comment [1, 10] - [1, 16])
> >               value: (binary_expression [2, 4] - [4, 5]
> >                 left: (number_literal [2, 4] - [2, 5])
> >                 (comment [3, 4] - [3, 14])
> >                 right: (number_literal [4, 4] - [4, 5])))))))
> > 
> > Here, the two comment nodes appear as unnamed nodes. IMHO the second
> > tree is a more useful one, as the named fields contain the semantically
> > important subtrees (e.g. a binary expression is made up of a left and
> > right subtree, not a left subtree, a right comment, and then some
> > unnamed subtree.)
> > 
> > Emacs's tree makes writing queries less convenient, as instead of being
> > able to refer to well-defined names, one has to rely on child indices to
> > account for comments.
> > 
> > 
> > Further mismatch arises from repeated fields and separators.
> > 
> > Consider the following Go source:
> > 
> >     package pkg
> >     
> >     var a, b, c = 1, 2, 3
> > 
> > treesit-explore-mode displays the following tree:
> > 
> >     (source_file
> >      (package_clause package (package_identifier))
> >      \n
> >      (var_declaration var
> >       (var_spec name: (identifier) name: , (identifier) value: , (identifier) =
> >        (expression_list (int_literal) , (int_literal) , (int_literal))))
> >      \n)
> > 
> > Here, the var_spec node has two fields named 'name' even though the
> > source specifies three names. Furthermore, The second 'name', as well as
> > 'value' are set to the ',' separator between identifiers. Two of the three
> > identifiers aren't named.
> > 
> > 'tree-sitter parse file.go', on the other hand, produces this more
> > accurate tree:
> > 
> >     (source_file [0, 0] - [2, 21]
> >       (package_clause [0, 0] - [0, 11]
> >         (package_identifier [0, 8] - [0, 11]))
> >       (var_declaration [2, 0] - [2, 21]
> >         (var_spec [2, 4] - [2, 21]
> >           name: (identifier [2, 4] - [2, 5])
> >           name: (identifier [2, 7] - [2, 8])
> >           name: (identifier [2, 10] - [2, 11])
> >           value: (expression_list [2, 14] - [2, 21]
> >             (int_literal [2, 14] - [2, 15])
> >             (int_literal [2, 17] - [2, 18])
> >             (int_literal [2, 20] - [2, 21])))))
> > 
> > This reproduces with 29.1 as well as 30.0.50.
> 
> Yuan, any comments or suggestions?
> 
> 
> 
> 

bug#66674: 30.0.50; Upstream tree-sitter and treesit disagree about fields

[Eli Zaretskii]

Ping! Ping!  Yuan, please chime in.

> Cc: 66674@debbugs.gnu.org, dominik@honnef.co
> Date: Sun, 19 Nov 2023 12:08:08 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> 
> Ping!  Yuan, any comments?
> 
> > Cc: 66674@debbugs.gnu.org
> > Date: Wed, 25 Oct 2023 16:03:10 +0300
> > From: Eli Zaretskii <eliz@gnu.org>
> > 
> > > From: Dominik Honnef <dominik@honnef.co>
> > > Date: Sat, 21 Oct 2023 22:36:30 +0200
> > > 
> > > Using tree-sitter's CLI as well as the publicly hosted playground
> > > produce different parse trees than treesit in Emacs. Specifically, the
> > > assignment of nodes to named fields differs.
> > > 
> > > Given the following C source:
> > > 
> > >     void main() {
> > >       int x = // foo
> > >         1+
> > >         // comment
> > >         2;
> > >     }
> > > 
> > > treesit-explore-mode displays the following tree:
> > > 
> > >     (translation_unit
> > >      (function_definition type: (primitive_type)
> > >       declarator: 
> > >        (function_declarator declarator: (identifier)
> > >         parameters: (parameter_list ( )))
> > >       body: 
> > >        (compound_statement {
> > >         (declaration type: (primitive_type)
> > >          declarator: 
> > >           (init_declarator declarator: (identifier) = value: (comment)
> > >            (binary_expression left: (number_literal) operator: + right: (comment) (number_literal)))
> > >          ;)
> > >         })))
> > > 
> > > Note how in the init_declarator node, the 'value' field is a comment
> > > node, and similarly for the 'right' field in the binary_expression node.
> > > 
> > > Running 'tree-sitter parse file.c', on the other hand, produces the
> > > following tree:
> > > 
> > >     (translation_unit [0, 0] - [6, 0]
> > >       (function_definition [0, 0] - [5, 1]
> > >         type: (primitive_type [0, 0] - [0, 4])
> > >         declarator: (function_declarator [0, 5] - [0, 11]
> > >           declarator: (identifier [0, 5] - [0, 9])
> > >           parameters: (parameter_list [0, 9] - [0, 11]))
> > >         body: (compound_statement [0, 12] - [5, 1]
> > >           (declaration [1, 2] - [4, 6]
> > >             type: (primitive_type [1, 2] - [1, 5])
> > >             declarator: (init_declarator [1, 6] - [4, 5]
> > >               declarator: (identifier [1, 6] - [1, 7])
> > >               (comment [1, 10] - [1, 16])
> > >               value: (binary_expression [2, 4] - [4, 5]
> > >                 left: (number_literal [2, 4] - [2, 5])
> > >                 (comment [3, 4] - [3, 14])
> > >                 right: (number_literal [4, 4] - [4, 5])))))))
> > > 
> > > Here, the two comment nodes appear as unnamed nodes. IMHO the second
> > > tree is a more useful one, as the named fields contain the semantically
> > > important subtrees (e.g. a binary expression is made up of a left and
> > > right subtree, not a left subtree, a right comment, and then some
> > > unnamed subtree.)
> > > 
> > > Emacs's tree makes writing queries less convenient, as instead of being
> > > able to refer to well-defined names, one has to rely on child indices to
> > > account for comments.
> > > 
> > > 
> > > Further mismatch arises from repeated fields and separators.
> > > 
> > > Consider the following Go source:
> > > 
> > >     package pkg
> > >     
> > >     var a, b, c = 1, 2, 3
> > > 
> > > treesit-explore-mode displays the following tree:
> > > 
> > >     (source_file
> > >      (package_clause package (package_identifier))
> > >      \n
> > >      (var_declaration var
> > >       (var_spec name: (identifier) name: , (identifier) value: , (identifier) =
> > >        (expression_list (int_literal) , (int_literal) , (int_literal))))
> > >      \n)
> > > 
> > > Here, the var_spec node has two fields named 'name' even though the
> > > source specifies three names. Furthermore, The second 'name', as well as
> > > 'value' are set to the ',' separator between identifiers. Two of the three
> > > identifiers aren't named.
> > > 
> > > 'tree-sitter parse file.go', on the other hand, produces this more
> > > accurate tree:
> > > 
> > >     (source_file [0, 0] - [2, 21]
> > >       (package_clause [0, 0] - [0, 11]
> > >         (package_identifier [0, 8] - [0, 11]))
> > >       (var_declaration [2, 0] - [2, 21]
> > >         (var_spec [2, 4] - [2, 21]
> > >           name: (identifier [2, 4] - [2, 5])
> > >           name: (identifier [2, 7] - [2, 8])
> > >           name: (identifier [2, 10] - [2, 11])
> > >           value: (expression_list [2, 14] - [2, 21]
> > >             (int_literal [2, 14] - [2, 15])
> > >             (int_literal [2, 17] - [2, 18])
> > >             (int_literal [2, 20] - [2, 21])))))
> > > 
> > > This reproduces with 29.1 as well as 30.0.50.
> > 
> > Yuan, any comments or suggestions?
> > 
> > 
> > 
> > 
> 
> 
> 
>