bug#73404: 30.0.50; [forward/kill/etc]-sexp commands do not behave as expected in tree-sitter modes

[Juri Linkov <juri@linkov.net>]

>>> However, deciding on what makes sense is hard, which is why I'm
>>> suggesting we try to define a "spec". It needn't really be much more
>>> than reference suggestions for mode implementors at first, then maybe
>>> some intersection of features crystallize across all modes.
>>
>> I don't think it's possible to find a spec better than Stefan already
>> defined as an AST node delimited by an opening and a closing "keyword":
>> (...), [...], if...fi, begin...end, ...
>
> IIUC, Theodor was talking more generally than just about "list" operations.
> I think "list" operations are among the clearest case (BTW, I wonder if
> someone has already tried to implement that functionality for Haskell or
> Python,

Recently I looked at python-ts-mode, but then discovered that it
completely misses any use of treesit-thing-settings unlike many
other ts-modes.  Also there is no treesit-thing-settings yet
in the new haskell-ts-mode.

> where the opening and a closing "keywords" are basically newlines).

The main problem with empty opening and closing keywords is that
nodes from different levels all end up at the same position,
and currently treesit prefers a higher-level node.  This means
that the sequence of `C-M-f C-M-b` is not idempotent anymore:
`C-M-b` might jump to the start of a higher-level node.

>> Or when looking at the commands with the 'forward-' prefix:
>>
>>   forward-sexp (C-M-f)
>>   forward-list (C-M-n)
>
> OK, we already talked about these two.  We could probably start drafting
> a kind of guideline.

Also in treesit.el, forward-sexp and forward-list
could share the same implementation.

>>   forward-symbol
>
> This one doesn't look too hard either.
>
>>   forward-sentence (M-e)
>
> This one is trickier.  You just provided a reasonably good start with
> "In languages with statements a sentence corresponds to a statement",
> but there are some non-trivial questions about language without
> statements, about use within comments, about how to handle nesting
> (e.g. statements within statements) and more generally the use of `M-e`
> from outside of statements.

I've checked that `M-e` in text-mode currently doesn't support
sentences in nested lists.  For example, `M-e` jumps inside
the list in:

  Top sentence.  (First inner sentence.  Second inner sentence.)

>>   forward-paragraph (M-})
>
> This one is also tied to filling, so it's important for it to behave
> well w.r.t to comments to allow filling actual text paragraphs
> within comments.

This provides an opportunity to move treesit-specific code
from `prog-fill-reindent-defun` to treesit.el.

> As for its behavior outside of comments ...
>
>>   forward-page (C-x ])
>>   forward-word (M-f)
>
> I think these should be oblivious to the syntax (i.e. line `forward-line`
> and `forward-char`).
>
> Your list missed the "defun" unit of navigation.

Because there is no `forward-defun` ;-)

But strange that "defun" still is defined by `treesit-defun-type-regexp`
instead of a "defun" thing in `treesit-thing-settings`.

> One other potentially useful unit of navigation that depends on the
> syntax would be "list element", which would do something like skip to
> the next/previous separator delimiting elements of the nearest enclosing
> paired delimiters.  For {...;...;...} blocks, this would behave like
> `forward-sentence`, except that when used from within the args of
> a function call like a `foo (..., ..., ...)`, it would jump
> between arguments instead of skipping all the way to the next statement.

Indeed, there is a need to have an additional command to move
between more ATS nodes, besides the list nodes navigated by 'C-M-n'.
But the problem is to find free keys for more commands.