bug#62302: 30.0.50; c/c++ tree sitter forward-sexp problem

bug#62302: 30.0.50; c/c++ tree sitter forward-sexp problem

[Herman, Geza]

With tree sitter, forward-sexp behaves a little strange. To reproduce:
  - copy the little example program below into a c++-ts-mode buffer
  - move the point to the opening curly brace of "main() {"
  - M-x forward-sexp

This will move the point to the space character in "int a;". But without 
tree sitter, this moves the point to the closing brace of main (I 
believe this is the correct behavior).

I noticed this problem because hideshow behaves strangely with tree sitter:
  - enable hideshow mode (M-x hs-minor-mode)
  - move the point to the same opening brace
  - M-x hs-toggle-hiding

Hideshow should hide the whole body of main, but instead it only hides 
parts of the body, and the buffer ends up like this:

int main() {... a;
     }
    int b;
}

If I set forward-sexp-function to nil, the problem goes away. I think 
this issue is somewhat related to #60894

Here's the little example program:

int main() {
     if (false) {
         int a;
     }
     int b;
}


In GNU Emacs 30.0.50 (build 1, x86_64-pc-linux-gnu, GTK+ Version
  3.24.36, cairo version 1.16.0) of 2023-03-20 built on okoska
Repository revision: 42fba8f36b19536964d6deb6a34f3fd1c02b43dd
Repository branch: master
Windowing system distributor 'The X.Org Foundation', version 11.0.12101006
System Description: Debian GNU/Linux bookworm/sid

Configured using:
  'configure --with-native-compilation=aot --without-compress-install
  --with-json --with-xinput2 --with-xwidgets --with-tree-sitter
  --with-cairo'

Configured features:
ACL CAIRO DBUS FREETYPE GIF GLIB GMP GNUTLS GPM GSETTINGS HARFBUZZ JPEG
JSON LCMS2 LIBOTF LIBSELINUX LIBSYSTEMD LIBXML2 M17N_FLT MODULES
NATIVE_COMP NOTIFY INOTIFY PDUMPER PNG SECCOMP SOUND SQLITE3 THREADS
TIFF TOOLKIT_SCROLL_BARS TREE_SITTER WEBP X11 XDBE XIM XINPUT2 XPM
XWIDGETS GTK3 ZLIB

Important settings:
   value of $LC_ALL: C.UTF-8
   value of $LANG: en_US.UTF-8
   value of $XMODIFIERS: @im=none
   locale-coding-system: utf-8-unix

Major mode: C++//

Minor modes in effect:
   tooltip-mode: t
   global-eldoc-mode: t
   show-paren-mode: t
   electric-indent-mode: t
   mouse-wheel-mode: t
   tool-bar-mode: t
   menu-bar-mode: t
   file-name-shadow-mode: t
   global-font-lock-mode: t
   font-lock-mode: t
   blink-cursor-mode: t
   line-number-mode: t
   indent-tabs-mode: t
   transient-mark-mode: t
   auto-composition-mode: t
   auto-encryption-mode: t
   auto-compression-mode: t
   hs-minor-mode: t

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message mailcap yank-media puny dired
dired-loaddefs rfc822 mml mml-sec password-cache epa derived epg rfc6068
epg-config gnus-util text-property-search time-date mm-decode mm-bodies
mm-encode mail-parse rfc2231 mailabbrev gmm-utils mailheader sendmail
rfc2047 rfc2045 ietf-drums mm-util mail-prsvr mail-utils hideshow
c-ts-mode c-ts-common treesit comp comp-cstr warnings icons subr-x rx
cl-seq cl-macs gv cl-extra help-mode bytecomp byte-compile cc-mode
cc-fonts cc-guess cc-menus cc-cmds cc-styles cc-align cc-engine cc-vars
cc-defs cl-loaddefs cl-lib rmc iso-transl tooltip cconv eldoc paren
electric uniquify ediff-hook vc-hooks lisp-float-type elisp-mode mwheel
term/x-win x-win term/common-win x-dnd tool-bar dnd fontset image
regexp-opt fringe tabulated-list replace newcomment text-mode lisp-mode
prog-mode register page tab-bar menu-bar rfn-eshadow isearch easymenu
timer select scroll-bar mouse jit-lock font-lock syntax font-core
term/tty-colors frame minibuffer nadvice seq simple cl-generic
indonesian philippine cham georgian utf-8-lang misc-lang vietnamese
tibetan thai tai-viet lao korean japanese eucjp-ms cp51932 hebrew greek
romanian slovak czech european ethiopic indian cyrillic chinese
composite emoji-zwj charscript charprop case-table epa-hook
jka-cmpr-hook help abbrev obarray oclosure cl-preloaded button loaddefs
theme-loaddefs faces cus-face macroexp files window text-properties
overlay sha1 md5 base64 format env code-pages mule custom widget keymap
hashtable-print-readable backquote threads xwidget-internal dbusbind
inotify lcms2 dynamic-setting system-font-setting font-render-setting
cairo move-toolbar gtk x-toolkit xinput2 x multi-tty
make-network-process native-compile emacs)

Memory information:
((conses 16 110613 15271)
  (symbols 48 9248 0)
  (strings 32 26736 1257)
  (string-bytes 1 928972)
  (vectors 16 18719)
  (vector-slots 8 373020 18319)
  (floats 8 32 48)
  (intervals 56 274 0)
  (buffers 984 13))

bug#62302: 30.0.50; c/c++ tree sitter forward-sexp problem

[Eli Zaretskii]

> Date: Mon, 20 Mar 2023 21:40:26 +0100
> From: "Herman, Geza" <geza.herman@gmail.com>
> 
> With tree sitter, forward-sexp behaves a little strange. To reproduce:
>   - copy the little example program below into a c++-ts-mode buffer
>   - move the point to the opening curly brace of "main() {"
>   - M-x forward-sexp
> 
> This will move the point to the space character in "int a;". But without 
> tree sitter, this moves the point to the closing brace of main (I 
> believe this is the correct behavior).
> 
> I noticed this problem because hideshow behaves strangely with tree sitter:
>   - enable hideshow mode (M-x hs-minor-mode)
>   - move the point to the same opening brace
>   - M-x hs-toggle-hiding
> 
> Hideshow should hide the whole body of main, but instead it only hides 
> parts of the body, and the buffer ends up like this:
> 
> int main() {... a;
>      }
>     int b;
> }
> 
> If I set forward-sexp-function to nil, the problem goes away. I think 
> this issue is somewhat related to #60894
> 
> Here's the little example program:
> 
> int main() {
>      if (false) {
>          int a;
>      }
>      int b;
> }

Theo, Yuan: any comments?  Can you look into fixing this?

Thanks.

bug#62302: 30.0.50; c/c++ tree sitter forward-sexp problem

[Yuan Fu]

Eli Zaretskii <eliz@gnu.org> writes:

>> Date: Mon, 20 Mar 2023 21:40:26 +0100
>> From: "Herman, Geza" <geza.herman@gmail.com>
>> 
>> With tree sitter, forward-sexp behaves a little strange. To reproduce:
>>   - copy the little example program below into a c++-ts-mode buffer
>>   - move the point to the opening curly brace of "main() {"
>>   - M-x forward-sexp
>> 
>> This will move the point to the space character in "int a;". But without 
>> tree sitter, this moves the point to the closing brace of main (I 
>> believe this is the correct behavior).
>> 
>> I noticed this problem because hideshow behaves strangely with tree sitter:
>>   - enable hideshow mode (M-x hs-minor-mode)
>>   - move the point to the same opening brace
>>   - M-x hs-toggle-hiding
>> 
>> Hideshow should hide the whole body of main, but instead it only hides 
>> parts of the body, and the buffer ends up like this:
>> 
>> int main() {... a;
>>      }
>>     int b;
>> }
>> 
>> If I set forward-sexp-function to nil, the problem goes away. I think 
>> this issue is somewhat related to #60894
>> 
>> Here's the little example program:
>> 
>> int main() {
>>      if (false) {
>>          int a;
>>      }
>>      int b;
>> }
>
> Theo, Yuan: any comments?  Can you look into fixing this?
>
> Thanks.

Missed this, sorry :-)

I modified the definition of "sexp" in c/c++-ts-mode. I think new
definition is intuitive but only time will tell. The new definition
works on the little example in this report.

Yuan

bug#62302: 30.0.50; c/c++ tree sitter forward-sexp problem

[Stefan Kangas]

Yuan Fu <casouri@gmail.com> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>>> Date: Mon, 20 Mar 2023 21:40:26 +0100
>>> From: "Herman, Geza" <geza.herman@gmail.com>
>>>
>>> With tree sitter, forward-sexp behaves a little strange. To reproduce:
>>>   - copy the little example program below into a c++-ts-mode buffer
>>>   - move the point to the opening curly brace of "main() {"
>>>   - M-x forward-sexp
>>>
>>> This will move the point to the space character in "int a;". But without
>>> tree sitter, this moves the point to the closing brace of main (I
>>> believe this is the correct behavior).
>>>
>>> I noticed this problem because hideshow behaves strangely with tree sitter:
>>>   - enable hideshow mode (M-x hs-minor-mode)
>>>   - move the point to the same opening brace
>>>   - M-x hs-toggle-hiding
>>>
>>> Hideshow should hide the whole body of main, but instead it only hides
>>> parts of the body, and the buffer ends up like this:
>>>
>>> int main() {... a;
>>>      }
>>>     int b;
>>> }
>>>
>>> If I set forward-sexp-function to nil, the problem goes away. I think
>>> this issue is somewhat related to #60894
>>>
>>> Here's the little example program:
>>>
>>> int main() {
>>>      if (false) {
>>>          int a;
>>>      }
>>>      int b;
>>> }
>>
>> Theo, Yuan: any comments?  Can you look into fixing this?
>>
>> Thanks.
>
> Missed this, sorry :-)
>
> I modified the definition of "sexp" in c/c++-ts-mode. I think new
> definition is intuitive but only time will tell. The new definition
> works on the little example in this report.
>
> Yuan

It seems like this has been fixed, so I'm closing this bug report.