bug#66674: 30.0.50; Upstream tree-sitter and treesit disagree about fields

[Yuan Fu <casouri@gmail.com>]

On 11/25/23 2:03 AM, Eli Zaretskii wrote:
> Ping! Ping!  Yuan, please chime in.
>
>> Cc: 66674@debbugs.gnu.org, dominik@honnef.co
>> Date: Sun, 19 Nov 2023 12:08:08 +0200
>> From: Eli Zaretskii <eliz@gnu.org>
>>
>> Ping!  Yuan, any comments?
>>
>>> Cc: 66674@debbugs.gnu.org
>>> Date: Wed, 25 Oct 2023 16:03:10 +0300
>>> From: Eli Zaretskii <eliz@gnu.org>
>>>
>>>> From: Dominik Honnef <dominik@honnef.co>
>>>> Date: Sat, 21 Oct 2023 22:36:30 +0200
>>>>
>>>> Using tree-sitter's CLI as well as the publicly hosted playground
>>>> produce different parse trees than treesit in Emacs. Specifically, the
>>>> assignment of nodes to named fields differs.
>>>>
>>>> Given the following C source:
>>>>
>>>>      void main() {
>>>>        int x = // foo
>>>>          1+
>>>>          // comment
>>>>          2;
>>>>      }
>>>>
>>>> treesit-explore-mode displays the following tree:
>>>>
>>>>      (translation_unit
>>>>       (function_definition type: (primitive_type)
>>>>        declarator:
>>>>         (function_declarator declarator: (identifier)
>>>>          parameters: (parameter_list ( )))
>>>>        body:
>>>>         (compound_statement {
>>>>          (declaration type: (primitive_type)
>>>>           declarator:
>>>>            (init_declarator declarator: (identifier) = value: (comment)
>>>>             (binary_expression left: (number_literal) operator: + right: (comment) (number_literal)))
>>>>           ;)
>>>>          })))
>>>>
>>>> Note how in the init_declarator node, the 'value' field is a comment
>>>> node, and similarly for the 'right' field in the binary_expression node.
>>>>
>>>> Running 'tree-sitter parse file.c', on the other hand, produces the
>>>> following tree:
>>>>
>>>>      (translation_unit [0, 0] - [6, 0]
>>>>        (function_definition [0, 0] - [5, 1]
>>>>          type: (primitive_type [0, 0] - [0, 4])
>>>>          declarator: (function_declarator [0, 5] - [0, 11]
>>>>            declarator: (identifier [0, 5] - [0, 9])
>>>>            parameters: (parameter_list [0, 9] - [0, 11]))
>>>>          body: (compound_statement [0, 12] - [5, 1]
>>>>            (declaration [1, 2] - [4, 6]
>>>>              type: (primitive_type [1, 2] - [1, 5])
>>>>              declarator: (init_declarator [1, 6] - [4, 5]
>>>>                declarator: (identifier [1, 6] - [1, 7])
>>>>                (comment [1, 10] - [1, 16])
>>>>                value: (binary_expression [2, 4] - [4, 5]
>>>>                  left: (number_literal [2, 4] - [2, 5])
>>>>                  (comment [3, 4] - [3, 14])
>>>>                  right: (number_literal [4, 4] - [4, 5])))))))
>>>>
>>>> Here, the two comment nodes appear as unnamed nodes. IMHO the second
>>>> tree is a more useful one, as the named fields contain the semantically
>>>> important subtrees (e.g. a binary expression is made up of a left and
>>>> right subtree, not a left subtree, a right comment, and then some
>>>> unnamed subtree.)
>>>>
>>>> Emacs's tree makes writing queries less convenient, as instead of being
>>>> able to refer to well-defined names, one has to rely on child indices to
>>>> account for comments.
>>>>
>>>>
>>>> Further mismatch arises from repeated fields and separators.
>>>>
>>>> Consider the following Go source:
>>>>
>>>>      package pkg
>>>>      
>>>>      var a, b, c = 1, 2, 3
>>>>
>>>> treesit-explore-mode displays the following tree:
>>>>
>>>>      (source_file
>>>>       (package_clause package (package_identifier))
>>>>       \n
>>>>       (var_declaration var
>>>>        (var_spec name: (identifier) name: , (identifier) value: , (identifier) =
>>>>         (expression_list (int_literal) , (int_literal) , (int_literal))))
>>>>       \n)
>>>>
>>>> Here, the var_spec node has two fields named 'name' even though the
>>>> source specifies three names. Furthermore, The second 'name', as well as
>>>> 'value' are set to the ',' separator between identifiers. Two of the three
>>>> identifiers aren't named.
>>>>
>>>> 'tree-sitter parse file.go', on the other hand, produces this more
>>>> accurate tree:
>>>>
>>>>      (source_file [0, 0] - [2, 21]
>>>>        (package_clause [0, 0] - [0, 11]
>>>>          (package_identifier [0, 8] - [0, 11]))
>>>>        (var_declaration [2, 0] - [2, 21]
>>>>          (var_spec [2, 4] - [2, 21]
>>>>            name: (identifier [2, 4] - [2, 5])
>>>>            name: (identifier [2, 7] - [2, 8])
>>>>            name: (identifier [2, 10] - [2, 11])
>>>>            value: (expression_list [2, 14] - [2, 21]
>>>>              (int_literal [2, 14] - [2, 15])
>>>>              (int_literal [2, 17] - [2, 18])
>>>>              (int_literal [2, 20] - [2, 21])))))
>>>>
>>>> This reproduces with 29.1 as well as 30.0.50.
>>> Yuan, any comments or suggestions?

Sorry sorry sorry, another missed report. I think this is a bug in 
treesit-explore-mode, I'll work on fixing it!

Yuan