bug#61374: 30.0.50; Wrong mark-sexp with tree-sitter

bug#61374: 30.0.50; Wrong mark-sexp with tree-sitter

[Ergus via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Hi:

Just trying tree-sitter with c++-mode is doing a wrong mark-sexp.

With this code:

{
  vector<int> myvar;
}

M-x c++-ts-mode

go to { and do C-M-SPC. The region marked goes from { up to > instead of
the corresponding }


In GNU Emacs 30.0.50 (build 7, x86_64-pc-linux-gnu, GTK+ Version
 3.24.36, cairo version 1.17.6) of 2023-02-09 built on Ergus
Repository revision: 680bc20553ebf01375ab7957b6f0be066335fd6e
Repository branch: master
System Description: Arch Linux

Configured using:
 'configure --prefix=/home/ergo/.local/ --with-mailutils --with-json
 --with-x-toolkit=gtk3 --with-xft --with-modules --with-cairo
 --with-harfbuzz --with-native-compilation
 '--program-transform-name=s/^ctags$/ctags.emacs/''

Configured features:
ACL CAIRO DBUS FREETYPE GIF GLIB GMP GNUTLS GPM GSETTINGS HARFBUZZ JPEG
JSON LCMS2 LIBSYSTEMD LIBXML2 MODULES NATIVE_COMP NOTIFY INOTIFY PDUMPER
PNG RSVG SECCOMP SOUND SQLITE3 THREADS TIFF TOOLKIT_SCROLL_BARS
TREE_SITTER WEBP X11 XDBE XIM XINPUT2 XPM GTK3 ZLIB

Important settings:
  value of $LANG: en_US.UTF-8
  locale-coding-system: utf-8-unix

Major mode: C++

Minor modes in effect:
  global-auto-revert-mode: t
  electric-pair-mode: t
  flyspell-mode: t
  company-mode: t
  flycheck-mode: t
  diff-hl-margin-local-mode: t
  diff-hl-margin-mode: t
  diff-hl-mode: t
  gtags-mode: t
  repeat-mode: t
  xterm-mouse-mode: t
  xclip-mode: t
  override-global-mode: t
  winner-mode: t
  save-place-mode: t
  delete-selection-mode: t
  savehist-mode: t
  global-display-fill-column-indicator-mode: t
  display-fill-column-indicator-mode: t
  global-display-line-numbers-mode: t
  display-line-numbers-mode: t
  which-key-mode: t
  show-paren-mode: t
  mouse-wheel-mode: t
  file-name-shadow-mode: t
  context-menu-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  size-indication-mode: t
  column-number-mode: t
  line-number-mode: t
  indent-tabs-mode: t
  transient-mark-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t

Load-path shadows:
/mnt/casa/gits/emacs_clones/gtags-mode/gtags-mode hides /home/ergo/.config/emacs/elpa/gtags-mode-1.0/gtags-mode
/home/ergo/.config/emacs/elpa/transient-20230201.1644/transient hides /home/ergo/.local/share/emacs/30.0.50/lisp/transient

Features:
(shadow sort mail-extr shortdoc help-fns radix-tree emacsbug message
mailcap yank-media puny rfc822 mml mml-sec epa derived epg rfc6068
epg-config gnus-util text-property-search time-date mm-decode mm-bodies
mm-encode mail-parse rfc2231 mailabbrev gmm-utils mailheader sendmail
rfc2047 rfc2045 ietf-drums mm-util mail-prsvr mail-utils c-ts-mode
c-ts-common treesit dabbrev cape-keyword autorevert filenotify ffap
thingatpt url-parse auth-source password-cache url-vars elec-pair
flyspell-correct flyspell ispell company-semantic company-template
company-capf company-c-headers company flycheck ansi-color json map
find-func dash pcase diff-hl-margin diff-hl-dired dired-x dired
dired-loaddefs diff-hl log-view pcvs-util vc-dir ewoc vc vc-dispatcher
diff-mode cape compat comp comp-cstr warnings icons rx gtags-mode subr-x
files-x xref project modern-cpp-font-lock cap-words superword subword
cc-mode cc-fonts cc-guess cc-menus cc-cmds cc-styles cc-align cc-engine
cc-vars cc-defs term/tmux term/xterm xterm init repeat xt-mouse xclip
edmacro kmacro use-package-bind-key bind-key simple-16-theme winner ring
saveplace delsel savehist easy-mmode display-fill-column-indicator
display-line-numbers diminish which-key cl-extra help-mode
use-package-diminish use-package-core disp-table info
dumb-jump-autoloads highlight-indent-guides-autoloads
company-lua-autoloads yasnippet-snippets-autoloads vundo-autoloads
sudo-edit-autoloads cuda-mode-autoloads nginx-mode-autoloads
crdt-autoloads company-auctex-autoloads groovy-mode-autoloads
flycheck-rust-autoloads evil-collection-autoloads annalist-autoloads
evil-autoloads goto-chg-autoloads string-inflection-autoloads
company-c-headers-autoloads protobuf-mode-autoloads lice-autoloads
lorem-ipsum-autoloads julia-mode-autoloads nasm-mode-autoloads
deadgrep-autoloads popup-autoloads company-nginx-autoloads
d-mode-autoloads i3wm-config-mode-autoloads tree-sitter-langs-autoloads
tree-sitter-autoloads tsc-autoloads ssh-config-mode-autoloads
move-dup-autoloads clang-format-autoloads esup-autoloads
dired-sidebar-autoloads gnuplot-autoloads web-completion-data-autoloads
phi-search-autoloads better-shell-autoloads fancy-compilation-autoloads
arduino-cli-mode-autoloads flycheck-julia-autoloads auctex-autoloads
tex-site which-key-autoloads multiple-cursors-autoloads
ibuffer-sidebar-autoloads dired-subtree-autoloads
dired-hacks-utils-autoloads systemd-autoloads pkgbuild-mode-autoloads
neotree-autoloads modern-cpp-font-lock-autoloads
company-reftex-autoloads magit-autoloads git-modes-autoloads
google-c-style-autoloads flymake-nasm-autoloads request-autoloads
caml-autoloads arduino-mode-autoloads ede/auto eieio-base cl-seq eieio
byte-opt bytecomp byte-compile eieio-core cl-macs gv cl-loaddefs cl-lib
sphinx-mode-autoloads f-autoloads magit-section-autoloads
diff-hl-autoloads lua-mode-autoloads gtags-mode-autoloads
mutt-mode-autoloads xclip-autoloads diminish-autoloads
imenu-list-autoloads paradox-autoloads spinner-autoloads
avy-zap-autoloads nftables-mode-autoloads s-autoloads csv-mode-autoloads
ibuffer-vc-autoloads objed-autoloads iedit-autoloads
markdown-mode-autoloads languagetool-autoloads vterm-toggle-autoloads
vterm-autoloads avy-autoloads git-timemachine-autoloads emamux-autoloads
flymake-quickdef-autoloads ibuffer-project-autoloads
haskell-mode-autoloads shell-command+-autoloads notmuch-autoloads
e2ansi-autoloads face-explorer-autoloads flycheck-autoloads
pkg-info-autoloads flx-autoloads opencl-mode-autoloads company-autoloads
ptemplate-templates-autoloads ptemplate-autoloads yasnippet-autoloads
ibuffer-tramp-autoloads debbugs-autoloads cobol-mode-autoloads
slime-autoloads cape-autoloads macrostep-autoloads git-commit-autoloads
with-editor-autoloads transient-autoloads compat-autoloads
flyspell-correct-autoloads dash-autoloads epl-autoloads vdiff-autoloads
hydra-autoloads lv-autoloads early-init rmc iso-transl tooltip cconv
eldoc paren electric uniquify ediff-hook vc-hooks lisp-float-type
elisp-mode mwheel term/x-win x-win term/common-win x-dnd tool-bar dnd
fontset image regexp-opt fringe tabulated-list replace newcomment
text-mode lisp-mode prog-mode register page tab-bar menu-bar rfn-eshadow
isearch easymenu timer select scroll-bar mouse jit-lock font-lock syntax
font-core term/tty-colors frame minibuffer nadvice seq simple cl-generic
indonesian philippine cham georgian utf-8-lang misc-lang vietnamese
tibetan thai tai-viet lao korean japanese eucjp-ms cp51932 hebrew greek
romanian slovak czech european ethiopic indian cyrillic chinese
composite emoji-zwj charscript charprop case-table epa-hook
jka-cmpr-hook help abbrev obarray oclosure cl-preloaded button loaddefs
theme-loaddefs faces cus-face macroexp files window text-properties
overlay sha1 md5 base64 format env code-pages mule custom widget keymap
hashtable-print-readable backquote threads dbusbind inotify lcms2
dynamic-setting system-font-setting font-render-setting cairo
move-toolbar gtk x-toolkit xinput2 x multi-tty make-network-process
native-compile emacs)

Memory information:
((conses 16 263724 9342)
 (symbols 48 17832 0)
 (strings 32 69077 6563)
 (string-bytes 1 2462231)
 (vectors 16 34202)
 (vector-slots 8 551420 13126)
 (floats 8 181 1366)
 (intervals 56 1110 0)
 (buffers 984 14))

bug#61374: 30.0.50; Wrong mark-sexp with tree-sitter

[Eli Zaretskii]

> Date: Thu, 09 Feb 2023 01:19:52 +0100
> From:  Ergus via "Bug reports for GNU Emacs,
>  the Swiss army knife of text editors" <bug-gnu-emacs@gnu.org>
> 
> Just trying tree-sitter with c++-mode is doing a wrong mark-sexp.
> 
> With this code:
> 
> {
>   vector<int> myvar;
> }
> 
> M-x c++-ts-mode
> 
> go to { and do C-M-SPC. The region marked goes from { up to > instead of
> the corresponding }

The problem is in forward-sexp (try C-M-f from the same place), which
C-M-SPC calls.  This problem exists only on master, where forward-sexp
was modified to call treesit-forward-sexp; on emacs-29 the behavior is
as expected.

CC'ing Yuan and Theo, who will probably find a fix in no time...

Thanks.

bug#61374: 30.0.50; Wrong mark-sexp with tree-sitter

[Theodor Thornhill via Bug reports for GNU Emacs, the Swiss army knife of text editors]

On 9 February 2023 07:40:27 CET, Eli Zaretskii <eliz@gnu.org> wrote:
>> Date: Thu, 09 Feb 2023 01:19:52 +0100
>> From:  Ergus via "Bug reports for GNU Emacs,
>>  the Swiss army knife of text editors" <bug-gnu-emacs@gnu.org>
>> 
>> Just trying tree-sitter with c++-mode is doing a wrong mark-sexp.
>> 
>> With this code:
>> 
>> {
>>   vector<int> myvar;
>> }
>> 
>> M-x c++-ts-mode
>> 
>> go to { and do C-M-SPC. The region marked goes from { up to > instead of
>> the corresponding }
>
>The problem is in forward-sexp (try C-M-f from the same place), which
>C-M-SPC calls.  This problem exists only on master, where forward-sexp
>was modified to call treesit-forward-sexp; on emacs-29 the behavior is
>as expected.
>
>CC'ing Yuan and Theo, who will probably find a fix in no time...
>
>Thanks.

I'll look at it in just a bit :)

Thanks for pinging!

Theo

bug#61374: 30.0.50; Wrong mark-sexp with tree-sitter

[Theodor Thornhill via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Theodor Thornhill <theo@thornhill.no> writes:

> On 9 February 2023 07:40:27 CET, Eli Zaretskii <eliz@gnu.org> wrote:
>>> Date: Thu, 09 Feb 2023 01:19:52 +0100
>>> From:  Ergus via "Bug reports for GNU Emacs,
>>>  the Swiss army knife of text editors" <bug-gnu-emacs@gnu.org>
>>> 
>>> Just trying tree-sitter with c++-mode is doing a wrong mark-sexp.
>>> 
>>> With this code:
>>> 
>>> {
>>>   vector<int> myvar;
>>> }
>>> 
>>> M-x c++-ts-mode
>>> 
>>> go to { and do C-M-SPC. The region marked goes from { up to > instead of
>>> the corresponding }
>>
>>The problem is in forward-sexp (try C-M-f from the same place), which
>>C-M-SPC calls.  This problem exists only on master, where forward-sexp
>>was modified to call treesit-forward-sexp; on emacs-29 the behavior is
>>as expected.
>>
>>CC'ing Yuan and Theo, who will probably find a fix in no time...
>>
>>Thanks.
>
> I'll look at it in just a bit :)
>
> Thanks for pinging!
>
> Theo


I think to remember why I decided on the current settings in
'treesit-sexp-type-regexp' - compound_statement is very frequently used
in the c/c++ grammars, and iirc that makes sexp-moving almost always
move to end of the next or current compound_statement.

try adding

```

  (setq-local treesit-sexp-type-regexp
              (regexp-opt '("preproc"
                            "declarator"
                            "qualifier"
                            "type"
                            "parameter"
                            "expression"
                            "literal"
                            "string"
                            "statement")))
```

and observe that mark-sexp and forward-sexp is ok now wrt this
bug-report, but running same commands inside of a scope may not.  I'm
not sure what the best combination of nodes for this particular regexp
is, but maybe you can give me some expectations, Ergus, and I can follow
up with some new settings?

Theo

bug#61374: 30.0.50; Wrong mark-sexp with tree-sitter

[Eli Zaretskii]

> From: Theodor Thornhill <theo@thornhill.no>
> Cc: 61374@debbugs.gnu.org
> Date: Thu, 09 Feb 2023 09:42:43 +0100
> 
> I think to remember why I decided on the current settings in
> 'treesit-sexp-type-regexp' - compound_statement is very frequently used
> in the c/c++ grammars, and iirc that makes sexp-moving almost always
> move to end of the next or current compound_statement.

Can you show some examples that illustrate these issues?  I'm not sure
I follow your line of reasoning, and thus cannot understand the
relevant considerations and decisions, and their expected effects on
behavior.

Thanks.

bug#61374: 30.0.50; Wrong mark-sexp with tree-sitter

[Theodor Thornhill via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Theodor Thornhill <theo@thornhill.no>
>> Cc: 61374@debbugs.gnu.org
>> Date: Thu, 09 Feb 2023 09:42:43 +0100
>> 
>> I think to remember why I decided on the current settings in
>> 'treesit-sexp-type-regexp' - compound_statement is very frequently used
>> in the c/c++ grammars, and iirc that makes sexp-moving almost always
>> move to end of the next or current compound_statement.
>
> Can you show some examples that illustrate these issues?  I'm not sure
> I follow your line of reasoning, and thus cannot understand the
> relevant considerations and decisions, and their expected effects on
> behavior.
>
> Thanks.


consider same code as in the first mail:

{
  vector<int> myvar;
}


If point is before the first curly, C-M-f will move to after the semi.


if "compound_statement" is added to the regexps, it will move to after
the closing curly - all good.

Now if point is at the c in 'vector', now we will also move to after the
closing curly, not the first space or after the semi.

This will happen in many places iirc.  I'm not saying it's unfixable,
just that I need to think a little about it, and some expected examples
would be nice.

Did that help?

Theo

bug#61374: 30.0.50; Wrong mark-sexp with tree-sitter

[Eli Zaretskii]

> From: Theodor Thornhill <theo@thornhill.no>
> Cc: spacibba@aol.com, casouri@gmail.com, 61374@debbugs.gnu.org
> Date: Thu, 09 Feb 2023 10:41:52 +0100
> 
> > Can you show some examples that illustrate these issues?  I'm not sure
> > I follow your line of reasoning, and thus cannot understand the
> > relevant considerations and decisions, and their expected effects on
> > behavior.
> >
> > Thanks.
> 
> 
> consider same code as in the first mail:
> 
> {
>   vector<int> myvar;
> }
> 
> 
> If point is before the first curly, C-M-f will move to after the semi.
> 
> 
> if "compound_statement" is added to the regexps, it will move to after
> the closing curly - all good.
> 
> Now if point is at the c in 'vector', now we will also move to after the
> closing curly, not the first space or after the semi.

Sounds like the treesit sexp movement doesn't have any notion of the
"level" of the sexp or something?  IOW, it doesn't know about the
"innermost" sexp at point?  If so, can we teach treesit.el about that?

Or am I missing the point?

bug#61374: 30.0.50; Wrong mark-sexp with tree-sitter

[Theodor Thornhill via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Theodor Thornhill <theo@thornhill.no>
>> Cc: spacibba@aol.com, casouri@gmail.com, 61374@debbugs.gnu.org
>> Date: Thu, 09 Feb 2023 10:41:52 +0100
>> 
>> > Can you show some examples that illustrate these issues?  I'm not sure
>> > I follow your line of reasoning, and thus cannot understand the
>> > relevant considerations and decisions, and their expected effects on
>> > behavior.
>> >
>> > Thanks.
>> 
>> 
>> consider same code as in the first mail:
>> 
>> {
>>   vector<int> myvar;
>> }
>> 
>> 
>> If point is before the first curly, C-M-f will move to after the semi.
>> 
>> 
>> if "compound_statement" is added to the regexps, it will move to after
>> the closing curly - all good.
>> 
>> Now if point is at the c in 'vector', now we will also move to after the
>> closing curly, not the first space or after the semi.
>
> Sounds like the treesit sexp movement doesn't have any notion of the
> "level" of the sexp or something?  IOW, it doesn't know about the
> "innermost" sexp at point?  If so, can we teach treesit.el about that?
>

Yes, I think we should too.  I'll look into it.

> Or am I missing the point?

Nope, don't think so!

Theo

bug#61374: 30.0.50; Wrong mark-sexp with tree-sitter

[Ergus via Bug reports for GNU Emacs, the Swiss army knife of text editors]

On Thu, Feb 09, 2023 at 12:08:49PM +0100, Theodor Thornhill wrote:
>Eli Zaretskii <eliz@gnu.org> writes:
>
>>> From: Theodor Thornhill <theo@thornhill.no>
>>> Cc: spacibba@aol.com, casouri@gmail.com, 61374@debbugs.gnu.org
>>> Date: Thu, 09 Feb 2023 10:41:52 +0100
>>>
>>> > Can you show some examples that illustrate these issues?  I'm not sure
>>> > I follow your line of reasoning, and thus cannot understand the
>>> > relevant considerations and decisions, and their expected effects on
>>> > behavior.
>>> >
>>> > Thanks.
>>>
>>>
>>> consider same code as in the first mail:
>>>
>>> {
>>>   vector<int> myvar;
>>> }
>>>
>>>
>>> If point is before the first curly, C-M-f will move to after the semi.
>>>
>>>
>>> if "compound_statement" is added to the regexps, it will move to after
>>> the closing curly - all good.
>>>
>>> Now if point is at the c in 'vector', now we will also move to after the
>>> closing curly, not the first space or after the semi.
>>
>> Sounds like the treesit sexp movement doesn't have any notion of the
>> "level" of the sexp or something?  IOW, it doesn't know about the
>> "innermost" sexp at point?  If so, can we teach treesit.el about that?
>>
>
>Yes, I think we should too.  I'll look into it.
>
Hi:

I am not aware of the details in the treesit.el implementation for
emacs, but the treesit-api already provides the ts_node_start_point and
ts_node_end_point functions which are intended for this use.

Are we relying on that?

>> Or am I missing the point?
>
>Nope, don't think so!
>
>Theo

bug#61374: 30.0.50; Wrong mark-sexp with tree-sitter

[Juri Linkov]

> This will happen in many places iirc.  I'm not saying it's unfixable,
> just that I need to think a little about it, and some expected examples
> would be nice.

Would it be possible to make sexp navigation in c-ts-mode exactly
like it was in c-mode?  It's quite frustrating when after switching
to c-ts-mode sexp commands operate on different things.

bug#61374: 30.0.50; Wrong mark-sexp with tree-sitter

[Theodor Thornhill via Bug reports for GNU Emacs, the Swiss army knife of text editors]

On 9 February 2023 18:47:29 CET, Juri Linkov <juri@linkov.net> wrote:
>> This will happen in many places iirc.  I'm not saying it's unfixable,
>> just that I need to think a little about it, and some expected examples
>> would be nice.
>
>Would it be possible to make sexp navigation in c-ts-mode exactly
>like it was in c-mode?  It's quite frustrating when after switching
>to c-ts-mode sexp commands operate on different things.

That should be the goal, yeah :)

Theo

bug#61374: 30.0.50; Wrong mark-sexp with tree-sitter

[Ergus via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Hi:

Sorry to bother. Any progress on this? 

Best,
Ergus

On Thu, Feb 09, 2023 at 08:53:03PM +0100, Theodor Thornhill wrote:
>
>
>On 9 February 2023 18:47:29 CET, Juri Linkov <juri@linkov.net> wrote:
>>> This will happen in many places iirc.  I'm not saying it's unfixable,
>>> just that I need to think a little about it, and some expected examples
>>> would be nice.
>>
>>Would it be possible to make sexp navigation in c-ts-mode exactly
>>like it was in c-mode?  It's quite frustrating when after switching
>>to c-ts-mode sexp commands operate on different things.
>
>That should be the goal, yeah :)
>
>Theo

bug#61374: 30.0.50; Wrong mark-sexp with tree-sitter

[Theodor Thornhill via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Ergus <spacibba@aol.com> writes:

> Hi:
>
> Sorry to bother. Any progress on this? 
>

No problem! No, not yet. Will try to look at it tonight. Thanks for
reminding me :)

Theo

bug#61374: 30.0.50; Wrong mark-sexp with tree-sitter

[Ergus via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Hi Theo:

I know you have multiple bug reports going on (actually some of them are
all my fault ;p indeed) But this one with sexp is a very broken one
IMHO, so I ping again ;).

Did you finally got some time to think on this?

Best,
Ergus



On Tue, Feb 14, 2023 at 11:50:56AM +0100, Theodor Thornhill wrote:
>Ergus <spacibba@aol.com> writes:
>
>> Hi:
>>
>> Sorry to bother. Any progress on this?
>>
>
>No problem! No, not yet. Will try to look at it tonight. Thanks for
>reminding me :)
>
>Theo

bug#61374: 30.0.50; Wrong mark-sexp with tree-sitter

[Theodor Thornhill via Bug reports for GNU Emacs, the Swiss army knife of text editors]

On 19 February 2023 03:15:42 CET, Ergus <spacibba@aol.com> wrote:
>Hi Theo:
>
>I know you have multiple bug reports going on (actually some of them are
>all my fault ;p indeed) But this one with sexp is a very broken one
>IMHO, so I ping again ;).
>
>Did you finally got some time to think on this?
>
>Best,
>Ergus
>
>
>
>On Tue, Feb 14, 2023 at 11:50:56AM +0100, Theodor Thornhill wrote:
>> Ergus <spacibba@aol.com> writes:
>> 
>>> Hi:
>>> 
>>> Sorry to bother. Any progress on this?
>>> 
>> 
>> No problem! No, not yet. Will try to look at it tonight. Thanks for
>> reminding me :)
>> 
>> Theo

No worries!

Yeah, I'm working on it, but it seems particularly hard with c for some reason. I haven't prioritized this lately as this code is in master, but the other fixes are on release branch.

I think I have a solution that will make the transition much easier, though. I'll see if I can make a proper attempt this evening.

Sorry :)

Theo 

bug#61374: 30.0.50; Wrong mark-sexp with tree-sitter

[Ergus via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Hi Theodor:

Just to remind... no progress on this yet?

Sorry to bother,
Ergus

On Tue, Feb 14, 2023 at 11:50:56AM +0100, Theodor Thornhill wrote:
>Ergus <spacibba@aol.com> writes:
>
>> Hi:
>>
>> Sorry to bother. Any progress on this?
>>
>
>No problem! No, not yet. Will try to look at it tonight. Thanks for
>reminding me :)
>
>Theo