bug#73404: 30.0.50; [forward/kill/etc]-sexp commands do not behave as expected in tree-sitter modes

[Juri Linkov <juri@linkov.net>]

> Another example on line 152 in dispnew.c:
>
>   struct redisplay_history
>   {
>     char trace[512 + 100];
>   };
>
> Move point to the [ before 512: C-M-n will signal an error.
>
> Any bracket in the file will trigger either the first or the second
> behavior.

Sorry, I didn't foresee that you would expect C-M-n and C-M-f to behave
the same way regarding list motion.  This is fixed now by syncing
treesit-forward-list with treesit-forward-sexp-list, so all your
test cases work correctly now.

>> > w32_get_internal_run_time (void)
>> > {
>> >   if (get_process_times_fn)
>> >     {
>> >       FILETIME create, exit, kernel, user;
>> >       HANDLE proc = GetCurrentProcess ();
>> >       if ((*get_process_times_fn) (proc, &create, &exit, &kernel, &user))
>> >         {
>> > 	  return ltime (total.QuadPart);
>> >         }                              ^
>> >     }
>> >
>> >   return Fcurrent_time ();
>> > }
>> >
>> > If you put point on the semi-colon marked by "^", C-M-n signals an
>> > error.  Can't we instead move to the next "list" after
>> > "Fcurrent_time"?
>> 
>> 'C-M-n' doesn't move to the next statement in c-mode,
>> since we need to support backward-compatibility.  The key
>> that moves to the next statement ignoring nested lists
>> in C modes is 'M-e'.
>
> Well, then maybe we could have the more useful behavior as an opt-in
> option?

Do you also need an option to move in Lisp from 1 to 2 in:

  (compound_statement
   (if_statement
    (compound_statement
     (return_statement 1)))
   (return_statement 2))

>> C-M-n is unable to move to the first group because
>> it's inside the node in the AST:
>> 
>>     (cast_expression "("
>>      type: (type_descriptor type: (type_identifier))
>>      ")"
>>      value: 
>>       (call_expression function: (identifier)
>>        arguments: 
>>         (argument_list "(" (identifier) ,
>>          (string_literal " (string_content) ")
>>          ")")))
>> 
>> Both the first and the 2nd parenthesized groups are inside the
>> "cast_expression" node.
>
> Sorry, I don't follow: the parentheses of the argument list are also
> inside a node, and yet we do recognize them.  What's the difference?

The problem is that ")" is not the final child of 'cast_expression'.
So C-M-f/C-M-n can't stop after ")".

All these problems stem from the imperfection of ts grammars.

It would be nice if someone will find a way to modify the grammars
after importing them to Emacs core, to be able to insert more nodes.
This will solve a whole class of problems.

Otherwise, if vendoring grammars is not feasible, an alternative
would be to define a new treesit-thing like e.g. "paren" that
will match anonymous nodes like:

  (setq-local treesit-thing-settings
              `((c
                 (paren ,(rx (or "{" "}" "[" "]" "(" ")")))

Then low-level treesit functions such as treesit-parent-until
will have to check if there a preceding sibling that matches "paren"
before using their real AST parent node.  This means maintaining
a parallel virtual tree.

>> > As for terminology: the Emacs user manual calls these "groupings
>> > delimited by parentheses (or whatever else serves as delimiters in the
>> > language you are working with)", or in short "parenthetical group".
>> > Why cannot we keep this terminology?  It sounds like it describes its
>> > subject as accurate as we can come up with.
>> 
>> So you prefer "group" over "list"?
>
> As a shorthand; the better term is "parenthesized group".

By definition a list is a parenthesized group.
The term "list" is much shorter than its synonym
"parenthesized group", so I see no reasons not to use "list".

>> Then 'forward-list' should move over "group"?
>
> I'm not sure we should rename the commands, given the legacy.  But
> the doc string should use "group" or "parenthesized group", IMO.

The doc string of 'forward-list' already uses this:

  (forward-list &optional ARG INTERACTIVE)
  Move forward across one balanced group of parentheses.