bug#62238: 30.0.50; Unusual interpretation of "S-expressions" in c-ts-mode

bug#62238: 30.0.50; Unusual interpretation of "S-expressions" in c-ts-mode

[Philip Kaludercic]

X-Debbugs-CC: Theodor Thornhill <theo@thornhill.no>, Yuan Fu <casouri@gmail.com>

Take this code block, where | is the position of the point.

if (foo)|{
  bar;
  baz;
}

When I press C-M-SPC to mark a "S-expression" (knowing that C doesn't
consist of S-Expressions), then I would assume that the region from the
current point until the closing bracket would be selected.  Instead I
get the region from the point up until after the first semicolon.

When I press C-M-SPC again, I get the second one as well, but a third
press refuses to go on.  So I never get the entire block.

I do not know if this is intentional, but it also applies to movement
and killing.  This is also particularly unintuitive when matching pairs
are highlighted (in this case the brackets), which always (?) matches
the interpretation of sexp-commands.


In GNU Emacs 30.0.50 (build 1, x86_64-pc-linux-gnu, GTK+ Version
 3.24.36, cairo version 1.16.0) of 2023-03-15 built on quetzal
Repository revision: a798a29f7519244b47ffc3035fcd8bf7bafea4d5
Repository branch: master
System Description: Debian GNU/Linux bookworm/sid

Configured using:
 'configure --with-pgtk --with-imagemagick --with-native-compilation
 --with-tree-sitter --with-json CC=gcc 'CFLAGS=-O2 -march=native -pipe'
 LDFLAGS=-flto'

Configured features:
ACL CAIRO DBUS FREETYPE GIF GLIB GMP GNUTLS GPM GSETTINGS HARFBUZZ
IMAGEMAGICK JPEG JSON LCMS2 LIBOTF LIBSELINUX LIBSYSTEMD LIBXML2 MODULES
NATIVE_COMP NOTIFY INOTIFY PDUMPER PGTK PNG RSVG SECCOMP SOUND SQLITE3
THREADS TIFF TOOLKIT_SCROLL_BARS TREE_SITTER WEBP XIM GTK3 ZLIB

Important settings:
  value of $EMACSLOADPATH: 
  value of $LC_MONETARY: en_US.UTF-8
  value of $LC_NUMERIC: en_US.UTF-8
  value of $LC_TIME: en_US.UTF-8
  value of $LANG: en_US.UTF-8
  value of $XMODIFIERS: @im=ibus
  locale-coding-system: utf-8-unix

Major mode: C/*

Minor modes in effect:
  global-git-commit-mode: t
  magit-auto-revert-mode: t
  shell-dirtrack-mode: t
  jit-spell-mode: t
  flymake-mode: t
  yas-minor-mode: t
  editorconfig-mode: t
  repeat-mode: t
  display-battery-mode: t
  display-time-mode: t
  diff-hl-flydiff-mode: t
  diff-hl-mode: t
  winner-mode: t
  windmove-mode: t
  corfu-history-mode: t
  corfu-mode: t
  electric-pair-mode: t
  recentf-mode: t
  save-place-mode: t
  savehist-mode: t
  pixel-scroll-precision-mode: t
  pixel-scroll-mode: t
  xterm-mouse-mode: t
  which-function-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  eldoc-mode: t
  show-paren-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tab-bar-mode: t
  file-name-shadow-mode: t
  context-menu-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  line-number-mode: t
  indent-tabs-mode: t
  transient-mark-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t

Load-path shadows:
/home/philip/.config/emacs/elpa/transient-0.3.7/transient hides /home/philip/Source/emacs/lisp/transient

Features:
(shadow ecomplete emacsbug two-column c-ts-mode dictionary
dictionary-connection java-ts-mode ffap rst mule-util face-remap
magit-submodule magit-obsolete magit-blame magit-stash magit-reflog
magit-bisect magit-push magit-pull magit-fetch magit-clone magit-remote
magit-commit magit-sequence magit-notes magit-worktree magit-tag
magit-merge magit-branch magit-reset magit-files magit-refs magit-status
magit magit-repos magit-apply magit-wip magit-log magit-diff smerge-mode
git-commit log-edit add-log magit-core magit-autorevert autorevert
filenotify magit-margin magit-transient magit-process with-editor server
magit-mode magit-git magit-section magit-utils crm dash grep
bug-reference find-func sort smiley gnus-cite mail-extr textsec
uni-scripts idna-mapping ucs-normalize uni-confusable textsec-check
gnus-bcklg gnus-async gnus-ml disp-table nndraft nnmh utf-7 nnfolder
epa-file network-stream gnus-agent gnus-srvr gnus-score score-mode
nnvirtual gnus-msg gnus-art mm-uu mml2015 mm-view mml-smime smime gnutls
dig nntp gnus-cache gnus-sum gnus-group gnus-undo gnus-start gnus-dbus
gnus-cloud nnimap nnmail mail-source utf7 nnoo gnus-spec gnus-int
gnus-range message rfc822 mml mml-sec epa mm-decode mm-bodies mm-encode
mailabbrev gmm-utils mailheader gnus-win url-http url-auth mail-parse
rfc2231 url-gw nsm epg rfc6068 epg-config display-line-numbers quail
copyright geiser-mode geiser-xref geiser-compile geiser-guile transient
edmacro kmacro geiser-debug geiser-mit info-look geiser geiser-repl
geiser-image geiser-capf geiser-doc geiser-menu geiser-autodoc
geiser-edit etags fileloop generator xref geiser-completion geiser-eval
geiser-connection tq geiser-syntax geiser-log geiser-popup view
geiser-impl help-fns radix-tree geiser-custom geiser-base scheme vc-hg
vc-bzr vc-backup flymake-proselint writegood-mode yank-media mhtml-mode
css-mode smie eww xdg url-queue shr pixel-fill kinsoku url-file svg puny
mm-url color js c-ts-common treesit cc-mode cc-fonts cc-guess cc-menus
cc-cmds cc-styles cc-align cc-engine cc-langs cc-vars cc-bytecomp
cc-defs sgml-mode facemenu dom dired-aux gnus-dired dired-x dired
desktop frameset dired-loaddefs char-fold misearch multi-isearch vc-git
buffer-env tramp-cmds tramp-sh tramp tramp-cache time-stamp
tramp-loaddefs trampver tramp-integration files-x tramp-compat shell
pcomplete parse-time iso8601 ls-lisp jit-spell ispell checkdoc
flymake-proc flymake yasnippet-snippets yasnippet noutline outline
editorconfig editorconfig-core editorconfig-core-handle
editorconfig-fnmatch init repeat project format-spec battery dbus xml
shell-command+ thingatpt time sendmail rfc2047 rfc2045 ietf-drums gnus
nnheader gnus-util time-date mail-utils range mm-util mail-prsvr
diff-hl-flydiff diff diff-hl log-view pcvs-util vc-dir ewoc diff-mode
easy-mmode hippie-exp winner windmove corfu-history corfu compat
elec-pair recentf tree-widget saveplace savehist pixel-scroll cua-base
xt-mouse cus-edit pp wid-edit which-func imenu package-vc vc
vc-dispatcher lisp-mnt cus-load setup finder-inf
shell-command+-autoloads corfu-autoloads haskell-mode-autoloads
macrostep-autoloads flymake-proselint-autoloads focus-autoloads
buffer-env-autoloads avy-autoloads yasnippet-snippets-autoloads
magit-autoloads jit-spell-autoloads geiser-mit-autoloads
bash-completion-autoloads vc-backup-autoloads setup-autoloads
auctex-autoloads tex-site proof-general-autoloads proof-site
proof-autoloads git-commit-autoloads with-editor-autoloads
compat-autoloads inspector-autoloads geiser-guile-autoloads
geiser-autoloads transient-autoloads magit-section-autoloads
0x0-autoloads sp-tutor-autoloads diff-hl-autoloads debbugs-autoloads
package-lint-autoloads xkcd-autoloads editorconfig-autoloads
rcirc-color-autoloads yasnippet-autoloads markdown-mode-autoloads
writegood-mode-autoloads sly-autoloads go-mode-autoloads info
dash-autoloads package browse-url url url-proxy url-privacy url-expand
url-methods url-history url-cookie generate-lisp-file url-domsuf
url-util mailcap url-handlers url-parse auth-source eieio eieio-core
password-cache json map byte-opt comp comp-cstr warnings bytecomp
byte-compile rx derived pcase cl-seq cl-macs gv subr-x compile
text-property-search comint ansi-osc ansi-color ring url-vars cl-extra
help-mode icons cl-loaddefs cl-lib rmc iso-transl tooltip cconv eldoc
paren electric uniquify ediff-hook vc-hooks lisp-float-type elisp-mode
mwheel term/pgtk-win pgtk-win term/common-win pgtk-dnd tool-bar dnd
fontset image regexp-opt fringe tabulated-list replace newcomment
text-mode lisp-mode prog-mode register page tab-bar menu-bar rfn-eshadow
isearch easymenu timer select scroll-bar mouse jit-lock font-lock syntax
font-core term/tty-colors frame minibuffer nadvice seq simple cl-generic
indonesian philippine cham georgian utf-8-lang misc-lang vietnamese
tibetan thai tai-viet lao korean japanese eucjp-ms cp51932 hebrew greek
romanian slovak czech european ethiopic indian cyrillic chinese
composite emoji-zwj charscript charprop case-table epa-hook
jka-cmpr-hook help abbrev obarray oclosure cl-preloaded button loaddefs
theme-loaddefs faces cus-face macroexp files window text-properties
overlay sha1 md5 base64 format env code-pages mule custom widget keymap
hashtable-print-readable backquote threads dbusbind inotify
dynamic-setting system-font-setting font-render-setting cairo gtk pgtk
lcms2 multi-tty make-network-process native-compile emacs)

Memory information:
((conses 16 1085291 119447)
 (symbols 48 41414 1)
 (strings 32 191733 12247)
 (string-bytes 1 5822782)
 (vectors 16 109526)
 (vector-slots 8 2681949 80394)
 (floats 8 565 1531)
 (intervals 56 20198 1681)
 (buffers 984 46))

bug#62238: 30.0.50; Unusual interpretation of "S-expressions" in c-ts-mode

[Theodor Thornhill via Bug reports for GNU Emacs, the Swiss army knife of text editors]

On 17 March 2023 10:52:39 CET, Philip Kaludercic <philipk@posteo.net> wrote:
>
>X-Debbugs-CC: Theodor Thornhill <theo@thornhill.no>, Yuan Fu <casouri@gmail.com>
>
>Take this code block, where | is the position of the point.
>
>if (foo)|{
>  bar;
>  baz;
>}
>
>When I press C-M-SPC to mark a "S-expression" (knowing that C doesn't
>consist of S-Expressions), then I would assume that the region from the
>current point until the closing bracket would be selected.  Instead I
>get the region from the point up until after the first semicolon.
>
>When I press C-M-SPC again, I get the second one as well, but a third
>press refuses to go on.  So I never get the entire block.
>
>I do not know if this is intentional, but it also applies to movement
>and killing.  This is also particularly unintuitive when matching pairs
>are highlighted (in this case the brackets), which always (?) matches
>the interpretation of sexp-commands.
>

Hi and thanks for taking a look.

Yeah, this isn't really finished yet.

It's on my backlog, but I didn't have enough time to focus on Emacs last couple of weeks. Hopefully I'll get more time soon :)

Theo


>
>In GNU Emacs 30.0.50 (build 1, x86_64-pc-linux-gnu, GTK+ Version
> 3.24.36, cairo version 1.16.0) of 2023-03-15 built on quetzal
>Repository revision: a798a29f7519244b47ffc3035fcd8bf7bafea4d5
>Repository branch: master
>System Description: Debian GNU/Linux bookworm/sid
>
>Configured using:
> 'configure --with-pgtk --with-imagemagick --with-native-compilation
> --with-tree-sitter --with-json CC=gcc 'CFLAGS=-O2 -march=native -pipe'
> LDFLAGS=-flto'
>
>Configured features:
>ACL CAIRO DBUS FREETYPE GIF GLIB GMP GNUTLS GPM GSETTINGS HARFBUZZ
>IMAGEMAGICK JPEG JSON LCMS2 LIBOTF LIBSELINUX LIBSYSTEMD LIBXML2 MODULES
>NATIVE_COMP NOTIFY INOTIFY PDUMPER PGTK PNG RSVG SECCOMP SOUND SQLITE3
>THREADS TIFF TOOLKIT_SCROLL_BARS TREE_SITTER WEBP XIM GTK3 ZLIB
>
>Important settings:
>  value of $EMACSLOADPATH: 
>  value of $LC_MONETARY: en_US.UTF-8
>  value of $LC_NUMERIC: en_US.UTF-8
>  value of $LC_TIME: en_US.UTF-8
>  value of $LANG: en_US.UTF-8
>  value of $XMODIFIERS: @im=ibus
>  locale-coding-system: utf-8-unix
>
>Major mode: C/*
>
>Minor modes in effect:
>  global-git-commit-mode: t
>  magit-auto-revert-mode: t
>  shell-dirtrack-mode: t
>  jit-spell-mode: t
>  flymake-mode: t
>  yas-minor-mode: t
>  editorconfig-mode: t
>  repeat-mode: t
>  display-battery-mode: t
>  display-time-mode: t
>  diff-hl-flydiff-mode: t
>  diff-hl-mode: t
>  winner-mode: t
>  windmove-mode: t
>  corfu-history-mode: t
>  corfu-mode: t
>  electric-pair-mode: t
>  recentf-mode: t
>  save-place-mode: t
>  savehist-mode: t
>  pixel-scroll-precision-mode: t
>  pixel-scroll-mode: t
>  xterm-mouse-mode: t
>  which-function-mode: t
>  tooltip-mode: t
>  global-eldoc-mode: t
>  eldoc-mode: t
>  show-paren-mode: t
>  electric-indent-mode: t
>  mouse-wheel-mode: t
>  tab-bar-mode: t
>  file-name-shadow-mode: t
>  context-menu-mode: t
>  global-font-lock-mode: t
>  font-lock-mode: t
>  line-number-mode: t
>  indent-tabs-mode: t
>  transient-mark-mode: t
>  auto-composition-mode: t
>  auto-encryption-mode: t
>  auto-compression-mode: t
>
>Load-path shadows:
>/home/philip/.config/emacs/elpa/transient-0.3.7/transient hides /home/philip/Source/emacs/lisp/transient
>
>Features:
>(shadow ecomplete emacsbug two-column c-ts-mode dictionary
>dictionary-connection java-ts-mode ffap rst mule-util face-remap
>magit-submodule magit-obsolete magit-blame magit-stash magit-reflog
>magit-bisect magit-push magit-pull magit-fetch magit-clone magit-remote
>magit-commit magit-sequence magit-notes magit-worktree magit-tag
>magit-merge magit-branch magit-reset magit-files magit-refs magit-status
>magit magit-repos magit-apply magit-wip magit-log magit-diff smerge-mode
>git-commit log-edit add-log magit-core magit-autorevert autorevert
>filenotify magit-margin magit-transient magit-process with-editor server
>magit-mode magit-git magit-section magit-utils crm dash grep
>bug-reference find-func sort smiley gnus-cite mail-extr textsec
>uni-scripts idna-mapping ucs-normalize uni-confusable textsec-check
>gnus-bcklg gnus-async gnus-ml disp-table nndraft nnmh utf-7 nnfolder
>epa-file network-stream gnus-agent gnus-srvr gnus-score score-mode
>nnvirtual gnus-msg gnus-art mm-uu mml2015 mm-view mml-smime smime gnutls
>dig nntp gnus-cache gnus-sum gnus-group gnus-undo gnus-start gnus-dbus
>gnus-cloud nnimap nnmail mail-source utf7 nnoo gnus-spec gnus-int
>gnus-range message rfc822 mml mml-sec epa mm-decode mm-bodies mm-encode
>mailabbrev gmm-utils mailheader gnus-win url-http url-auth mail-parse
>rfc2231 url-gw nsm epg rfc6068 epg-config display-line-numbers quail
>copyright geiser-mode geiser-xref geiser-compile geiser-guile transient
>edmacro kmacro geiser-debug geiser-mit info-look geiser geiser-repl
>geiser-image geiser-capf geiser-doc geiser-menu geiser-autodoc
>geiser-edit etags fileloop generator xref geiser-completion geiser-eval
>geiser-connection tq geiser-syntax geiser-log geiser-popup view
>geiser-impl help-fns radix-tree geiser-custom geiser-base scheme vc-hg
>vc-bzr vc-backup flymake-proselint writegood-mode yank-media mhtml-mode
>css-mode smie eww xdg url-queue shr pixel-fill kinsoku url-file svg puny
>mm-url color js c-ts-common treesit cc-mode cc-fonts cc-guess cc-menus
>cc-cmds cc-styles cc-align cc-engine cc-langs cc-vars cc-bytecomp
>cc-defs sgml-mode facemenu dom dired-aux gnus-dired dired-x dired
>desktop frameset dired-loaddefs char-fold misearch multi-isearch vc-git
>buffer-env tramp-cmds tramp-sh tramp tramp-cache time-stamp
>tramp-loaddefs trampver tramp-integration files-x tramp-compat shell
>pcomplete parse-time iso8601 ls-lisp jit-spell ispell checkdoc
>flymake-proc flymake yasnippet-snippets yasnippet noutline outline
>editorconfig editorconfig-core editorconfig-core-handle
>editorconfig-fnmatch init repeat project format-spec battery dbus xml
>shell-command+ thingatpt time sendmail rfc2047 rfc2045 ietf-drums gnus
>nnheader gnus-util time-date mail-utils range mm-util mail-prsvr
>diff-hl-flydiff diff diff-hl log-view pcvs-util vc-dir ewoc diff-mode
>easy-mmode hippie-exp winner windmove corfu-history corfu compat
>elec-pair recentf tree-widget saveplace savehist pixel-scroll cua-base
>xt-mouse cus-edit pp wid-edit which-func imenu package-vc vc
>vc-dispatcher lisp-mnt cus-load setup finder-inf
>shell-command+-autoloads corfu-autoloads haskell-mode-autoloads
>macrostep-autoloads flymake-proselint-autoloads focus-autoloads
>buffer-env-autoloads avy-autoloads yasnippet-snippets-autoloads
>magit-autoloads jit-spell-autoloads geiser-mit-autoloads
>bash-completion-autoloads vc-backup-autoloads setup-autoloads
>auctex-autoloads tex-site proof-general-autoloads proof-site
>proof-autoloads git-commit-autoloads with-editor-autoloads
>compat-autoloads inspector-autoloads geiser-guile-autoloads
>geiser-autoloads transient-autoloads magit-section-autoloads
>0x0-autoloads sp-tutor-autoloads diff-hl-autoloads debbugs-autoloads
>package-lint-autoloads xkcd-autoloads editorconfig-autoloads
>rcirc-color-autoloads yasnippet-autoloads markdown-mode-autoloads
>writegood-mode-autoloads sly-autoloads go-mode-autoloads info
>dash-autoloads package browse-url url url-proxy url-privacy url-expand
>url-methods url-history url-cookie generate-lisp-file url-domsuf
>url-util mailcap url-handlers url-parse auth-source eieio eieio-core
>password-cache json map byte-opt comp comp-cstr warnings bytecomp
>byte-compile rx derived pcase cl-seq cl-macs gv subr-x compile
>text-property-search comint ansi-osc ansi-color ring url-vars cl-extra
>help-mode icons cl-loaddefs cl-lib rmc iso-transl tooltip cconv eldoc
>paren electric uniquify ediff-hook vc-hooks lisp-float-type elisp-mode
>mwheel term/pgtk-win pgtk-win term/common-win pgtk-dnd tool-bar dnd
>fontset image regexp-opt fringe tabulated-list replace newcomment
>text-mode lisp-mode prog-mode register page tab-bar menu-bar rfn-eshadow
>isearch easymenu timer select scroll-bar mouse jit-lock font-lock syntax
>font-core term/tty-colors frame minibuffer nadvice seq simple cl-generic
>indonesian philippine cham georgian utf-8-lang misc-lang vietnamese
>tibetan thai tai-viet lao korean japanese eucjp-ms cp51932 hebrew greek
>romanian slovak czech european ethiopic indian cyrillic chinese
>composite emoji-zwj charscript charprop case-table epa-hook
>jka-cmpr-hook help abbrev obarray oclosure cl-preloaded button loaddefs
>theme-loaddefs faces cus-face macroexp files window text-properties
>overlay sha1 md5 base64 format env code-pages mule custom widget keymap
>hashtable-print-readable backquote threads dbusbind inotify
>dynamic-setting system-font-setting font-render-setting cairo gtk pgtk
>lcms2 multi-tty make-network-process native-compile emacs)
>
>Memory information:
>((conses 16 1085291 119447)
> (symbols 48 41414 1)
> (strings 32 191733 12247)
> (string-bytes 1 5822782)
> (vectors 16 109526)
> (vector-slots 8 2681949 80394)
> (floats 8 565 1531)
> (intervals 56 20198 1681)
> (buffers 984 46))
>
>

bug#62238: 30.0.50; Unusual interpretation of "S-expressions" in c-ts-mode

[Yuan Fu]

> On Mar 17, 2023, at 2:52 AM, Philip Kaludercic <philipk@posteo.net> wrote:
> 
> 
> X-Debbugs-CC: Theodor Thornhill <theo@thornhill.no>, Yuan Fu <casouri@gmail.com>
> 
> Take this code block, where | is the position of the point.
> 
> if (foo)|{
>  bar;
>  baz;
> }
> 
> When I press C-M-SPC to mark a "S-expression" (knowing that C doesn't
> consist of S-Expressions), then I would assume that the region from the
> current point until the closing bracket would be selected.  Instead I
> get the region from the point up until after the first semicolon.
> 
> When I press C-M-SPC again, I get the second one as well, but a third
> press refuses to go on.  So I never get the entire block.
> 
> I do not know if this is intentional, but it also applies to movement
> and killing.  This is also particularly unintuitive when matching pairs
> are highlighted (in this case the brackets), which always (?) matches
> the interpretation of sexp-commands.

I tested this on my Emacs session and vanilla session, and both marked to the closing bracket. I believe forward-sexp should just work by the syntax table. Perhaps it’s your config or something?

Yuan

bug#62238: 30.0.50; Unusual interpretation of "S-expressions" in c-ts-mode

[Daniel Martín via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Yuan Fu <casouri@gmail.com> writes:

>
> I tested this on my Emacs session and vanilla session, and both marked
> to the closing bracket. I believe forward-sexp should just work by the
> syntax table. Perhaps it’s your config or something?
>

You need to enable c-ts-mode first, which redirects
forward-sexp-function to treesit-forward-sexp.

bug#62238: 30.0.50; Unusual interpretation of "S-expressions" in c-ts-mode

[Kévin Le Gouguec]

Yuan Fu <casouri@gmail.com> writes:

> I tested this on my Emacs session and vanilla session, and both marked to the
> closing bracket. I believe forward-sexp should just work by the syntax
> table. Perhaps it’s your config or something?

FWIW, I tested this on

* the master branch (where Philip reported the bug): I can reproduce
  Philip's results;

    2023-03-18 "; * lisp/find-dired.el (find-gnu-find-p): Doc fix."
    (95d5154feed)

* emacs-29: things work as you (Yuan) describe.

    2023-03-18 "; Fix 'make-obsolete-variable' forms" (faee8d50738)

So maybe something that will get resolved by the next merge?  Or
something that has been broken or master.

bug#62238: 30.0.50; Unusual interpretation of "S-expressions" in c-ts-mode

[Daniel Martín via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Daniel Martín <mardani29@yahoo.es> writes:

> Yuan Fu <casouri@gmail.com> writes:
>
>>
>> I tested this on my Emacs session and vanilla session, and both marked
>> to the closing bracket. I believe forward-sexp should just work by the
>> syntax table. Perhaps it’s your config or something?
>>
>
> You need to enable c-ts-mode first, which redirects
> forward-sexp-function to treesit-forward-sexp.

I see in treesit.el that we set forward-sexp-function to
treesit-forward-sexp when treesit-sexp-type-regexp is set by the major
mode.  For languages with simple grammars, like C, I think that the
current approach that uses the syntax table is simpler and less prone to
errors, because the Tree-sitter function is general and should work for
every language.  I'd suggest we don't define treesit-sexp-type-regexp in
c-ts-mode, at least for C.

For languages like TypeScript, whose grammar is more complex, perhaps
forward-sexp does not work very well and using Tree-sitter to implement
it gives better results with code that is simpler to understand.

Thanks.

bug#62238: 30.0.50; Unusual interpretation of "S-expressions" in c-ts-mode

[Theodor Thornhill via Bug reports for GNU Emacs, the Swiss army knife of text editors]

On 18 March 2023 13:11:15 CET, "Daniel Martín" <mardani29@yahoo.es> wrote:
>Daniel Martín <mardani29@yahoo.es> writes:
>
>> Yuan Fu <casouri@gmail.com> writes:
>>
>>>
>>> I tested this on my Emacs session and vanilla session, and both marked
>>> to the closing bracket. I believe forward-sexp should just work by the
>>> syntax table. Perhaps it’s your config or something?
>>>
>>
>> You need to enable c-ts-mode first, which redirects
>> forward-sexp-function to treesit-forward-sexp.
>
>I see in treesit.el that we set forward-sexp-function to
>treesit-forward-sexp when treesit-sexp-type-regexp is set by the major
>mode.  For languages with simple grammars, like C, I think that the
>current approach that uses the syntax table is simpler and less prone to
>errors, because the Tree-sitter function is general and should work for
>every language.  I'd suggest we don't define treesit-sexp-type-regexp in
>c-ts-mode, at least for C.
>
>For languages like TypeScript, whose grammar is more complex, perhaps
>forward-sexp does not work very well and using Tree-sitter to implement
>it gives better results with code that is simpler to understand.
>
>Thanks.

I tend to agree. I think it works pretty well with java for instance, but I'm struggling with the compound_statement node in the c grammar.

If too much of an annoyance, I'm ok with not defining this for c, but maybe someone that writes more c than me can have a go at defining nodes that makes sense?

Theo

bug#62238: 30.0.50; Unusual interpretation of "S-expressions" in c-ts-mode

[Eli Zaretskii]

> Cc: 62238@debbugs.gnu.org, Philip Kaludercic <philipk@posteo.net>,
>  theodor thornhill <theo@thornhill.no>
> Date: Sat, 18 Mar 2023 13:11:15 +0100
> From:  Daniel Martín via "Bug reports for GNU Emacs,
>  the Swiss army knife of text editors" <bug-gnu-emacs@gnu.org>
> 
> > You need to enable c-ts-mode first, which redirects
> > forward-sexp-function to treesit-forward-sexp.
> 
> I see in treesit.el that we set forward-sexp-function to
> treesit-forward-sexp when treesit-sexp-type-regexp is set by the major
> mode.  For languages with simple grammars, like C, I think that the
> current approach that uses the syntax table is simpler and less prone to
> errors, because the Tree-sitter function is general and should work for
> every language.  I'd suggest we don't define treesit-sexp-type-regexp in
> c-ts-mode, at least for C.

I don't understand how you came to that conclusion.  Why would we want
to use syntax tables when we have a parser at our fingertips?  And if
"the Tree-sitter function is general and should work for every
language", as you say (and I agree), why should we refrain from using
it for C?

> For languages like TypeScript, whose grammar is more complex, perhaps
> forward-sexp does not work very well and using Tree-sitter to implement
> it gives better results with code that is simpler to understand.

There's a huge advantage of using the same function for all the
supported languages, because that makes that function better, as it is
tested in many different situations.

So I don't think I agree with you here, not at all.

bug#62238: 30.0.50; Unusual interpretation of "S-expressions" in c-ts-mode

[Theodor Thornhill via Bug reports for GNU Emacs, the Swiss army knife of text editors]

On 18 March 2023 14:33:49 CET, Eli Zaretskii <eliz@gnu.org> wrote:
>> Cc: 62238@debbugs.gnu.org, Philip Kaludercic <philipk@posteo.net>,
>>  theodor thornhill <theo@thornhill.no>
>> Date: Sat, 18 Mar 2023 13:11:15 +0100
>> From:  Daniel Martín via "Bug reports for GNU Emacs,
>>  the Swiss army knife of text editors" <bug-gnu-emacs@gnu.org>
>> 
>> > You need to enable c-ts-mode first, which redirects
>> > forward-sexp-function to treesit-forward-sexp.
>> 
>> I see in treesit.el that we set forward-sexp-function to
>> treesit-forward-sexp when treesit-sexp-type-regexp is set by the major
>> mode.  For languages with simple grammars, like C, I think that the
>> current approach that uses the syntax table is simpler and less prone to
>> errors, because the Tree-sitter function is general and should work for
>> every language.  I'd suggest we don't define treesit-sexp-type-regexp in
>> c-ts-mode, at least for C.
>
>I don't understand how you came to that conclusion.  Why would we want
>to use syntax tables when we have a parser at our fingertips?  And if
>"the Tree-sitter function is general and should work for every
>language", as you say (and I agree), why should we refrain from using
>it for C?
>
>> For languages like TypeScript, whose grammar is more complex, perhaps
>> forward-sexp does not work very well and using Tree-sitter to implement
>> it gives better results with code that is simpler to understand.
>
>There's a huge advantage of using the same function for all the
>supported languages, because that makes that function better, as it is
>tested in many different situations.
>
>So I don't think I agree with you here, not at all.

Yeah but I need some more time, or at least some patches to look at :)

Theo

bug#62238: 30.0.50; Unusual interpretation of "S-expressions" in c-ts-mode

[Daniel Martín via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Eli Zaretskii <eliz@gnu.org> writes:

>> 
>> I see in treesit.el that we set forward-sexp-function to
>> treesit-forward-sexp when treesit-sexp-type-regexp is set by the major
>> mode.  For languages with simple grammars, like C, I think that the
>> current approach that uses the syntax table is simpler and less prone to
>> errors, because the Tree-sitter function is general and should work for
>> every language.  I'd suggest we don't define treesit-sexp-type-regexp in
>> c-ts-mode, at least for C.
>
> I don't understand how you came to that conclusion.  Why would we want
> to use syntax tables when we have a parser at our fingertips?  And if
> "the Tree-sitter function is general and should work for every
> language", as you say (and I agree), why should we refrain from using
> it for C?

Note that basing C-M-x on syntax tables (that is, traditional
forward-sexp) does not completely exclude the use of Tree-sitter, AFAIU.
Here's my thought process: To do its job, C-M-x needs to know about some
code structures such as symbol constituents, strings, comments, and
parenthetical groups.  If in some language or future version of C the
syntax is complex enough that getting the syntax class of a character
requires proper parsing, the Tree-sitter major modes can augment the
syntax table to make C-M-x work correctly.  See
c-ts-mode--syntax-propertize for an example of how Tree-sitter can
augment a buffer's syntax table, if needed.

>
>> For languages like TypeScript, whose grammar is more complex, perhaps
>> forward-sexp does not work very well and using Tree-sitter to implement
>> it gives better results with code that is simpler to understand.
>
> There's a huge advantage of using the same function for all the
> supported languages, because that makes that function better, as it is
> tested in many different situations.
>

I agree that using a single function for every language is great for
simplicity and maintainability but, should it handle every movement
command as well?  My main concern is that a single function
(treesit--navigate-thing) is now being used not only for every language,
but for every structural movement command.  I think that it is difficult
that a single piece of logic can handle all structure movement commands
well.  There's a good chance that the code will end up being complex and
difficult to maintain.

bug#62238: 30.0.50; Unusual interpretation of "S-expressions" in c-ts-mode

[Daniel Martín via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Daniel Martín via "Bug reports for GNU Emacs, the Swiss army knife of
text editors" <bug-gnu-emacs@gnu.org> writes:

>
> Note that basing C-M-x on syntax tables (that is, traditional
> forward-sexp) does not completely exclude the use of Tree-sitter, AFAIU.
> Here's my thought process: To do its job, C-M-x needs to know about some
> code structures such as symbol constituents, strings, comments, and
> parenthetical groups.  If in some language or future version of C the
> syntax is complex enough that getting the syntax class of a character
> requires proper parsing, the Tree-sitter major modes can augment the
> syntax table to make C-M-x work correctly.  See
> c-ts-mode--syntax-propertize for an example of how Tree-sitter can
> augment a buffer's syntax table, if needed.

Typo: Where I said C-M-x, I meant C-M-f or C-M-b.

bug#62238: 30.0.50; Unusual interpretation of "S-expressions" in c-ts-mode

[Eli Zaretskii]

> From: Daniel Martín <mardani29@yahoo.es>
> Cc: casouri@gmail.com,  62238@debbugs.gnu.org,  philipk@posteo.net,
>   theo@thornhill.no
> Date: Sat, 18 Mar 2023 17:08:25 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > I don't understand how you came to that conclusion.  Why would we want
> > to use syntax tables when we have a parser at our fingertips?  And if
> > "the Tree-sitter function is general and should work for every
> > language", as you say (and I agree), why should we refrain from using
> > it for C?
> 
> Note that basing C-M-x on syntax tables (that is, traditional
> forward-sexp) does not completely exclude the use of Tree-sitter, AFAIU.
> Here's my thought process: To do its job, C-M-x needs to know about some
> code structures such as symbol constituents, strings, comments, and
> parenthetical groups.  If in some language or future version of C the
> syntax is complex enough that getting the syntax class of a character
> requires proper parsing, the Tree-sitter major modes can augment the
> syntax table to make C-M-x work correctly.  See
> c-ts-mode--syntax-propertize for an example of how Tree-sitter can
> augment a buffer's syntax table, if needed.

We have already C mode that uses syntax tables.  I think it's useful
to try syntactic movement using results of parsing as well, and
compare the relative merits and demerits.

> > There's a huge advantage of using the same function for all the
> > supported languages, because that makes that function better, as it is
> > tested in many different situations.
> 
> I agree that using a single function for every language is great for
> simplicity and maintainability but, should it handle every movement
> command as well?  My main concern is that a single function
> (treesit--navigate-thing) is now being used not only for every language,
> but for every structural movement command.  I think that it is difficult
> that a single piece of logic can handle all structure movement commands
> well.  There's a good chance that the code will end up being complex and
> difficult to maintain.

Instead of deciding up front that it could be a problem, I suggest to
try the tree-sitter based way and see how well it works.  We can
always go back to syntax tables if we decide they will work better for
some modes.  It makes little sense to me to make such decisions
without trying the tree-sitter way and trying it well.

bug#62238: 30.0.50; Unusual interpretation of "S-expressions" in c-ts-mode

[Dmitry Gutov]

On 18/03/2023 12:31, Kévin Le Gouguec wrote:
> Yuan Fu<casouri@gmail.com>  writes:
> 
>> I tested this on my Emacs session and vanilla session, and both marked to the
>> closing bracket. I believe forward-sexp should just work by the syntax
>> table. Perhaps it’s your config or something?
> FWIW, I tested this on
> 
> * the master branch (where Philip reported the bug): I can reproduce
>    Philip's results;
> 
>      2023-03-18 "; * lisp/find-dired.el (find-gnu-find-p): Doc fix."
>      (95d5154feed)
> 
> * emacs-29: things work as you (Yuan) describe.
> 
>      2023-03-18 "; Fix 'make-obsolete-variable' forms" (faee8d50738)
> 
> So maybe something that will get resolved by the next merge?  Or
> something that has been broken or master.

Parse tree based forward-sexp was indeed only added on master.

bug#62238: 30.0.50; Unusual interpretation of "S-expressions" in c-ts-mode

[Theodor Thornhill via Bug reports for GNU Emacs, the Swiss army knife of text editors]

On 18 March 2023 18:48:10 CET, Dmitry Gutov <dgutov@yandex.ru> wrote:
>On 18/03/2023 12:31, Kévin Le Gouguec wrote:
>> Yuan Fu<casouri@gmail.com>  writes:
>> 
>>> I tested this on my Emacs session and vanilla session, and both marked to the
>>> closing bracket. I believe forward-sexp should just work by the syntax
>>> table. Perhaps it’s your config or something?
>> FWIW, I tested this on
>> 
>> * the master branch (where Philip reported the bug): I can reproduce
>>    Philip's results;
>> 
>>      2023-03-18 "; * lisp/find-dired.el (find-gnu-find-p): Doc fix."
>>      (95d5154feed)
>> 
>> * emacs-29: things work as you (Yuan) describe.
>> 
>>      2023-03-18 "; Fix 'make-obsolete-variable' forms" (faee8d50738)
>> 
>> So maybe something that will get resolved by the next merge?  Or
>> something that has been broken or master.
>
>Parse tree based forward-sexp was indeed only added on master.
>

Yeah

bug#62238: 30.0.50; Unusual interpretation of "S-expressions" in c-ts-mode

[Yuan Fu]

> On Mar 18, 2023, at 3:31 AM, Kévin Le Gouguec <kevin.legouguec@gmail.com> wrote:
> 
> Yuan Fu <casouri@gmail.com> writes:
> 
>> I tested this on my Emacs session and vanilla session, and both marked to the
>> closing bracket. I believe forward-sexp should just work by the syntax
>> table. Perhaps it’s your config or something?
> 
> FWIW, I tested this on
> 
> * the master branch (where Philip reported the bug): I can reproduce
>  Philip's results;
> 
>    2023-03-18 "; * lisp/find-dired.el (find-gnu-find-p): Doc fix."
>    (95d5154feed)
> 
> * emacs-29: things work as you (Yuan) describe.
> 
>    2023-03-18 "; Fix 'make-obsolete-variable' forms" (faee8d50738)
> 
> So maybe something that will get resolved by the next merge?  Or
> something that has been broken or master.

Duh, ok, I know what’s going on. Theo added some sexp related tree-sitter stuff to master, but I’ve been focusing on emacs-29. On emacs-29, c-ts-mode doesn’t redefine forward-sexp-function and the default function uses the syntax table, which works fine.

Yuan

bug#62238: 30.0.50; Unusual interpretation of "S-expressions" in c-ts-mode

[Juri Linkov]

>> > I don't understand how you came to that conclusion.  Why would we want
>> > to use syntax tables when we have a parser at our fingertips?  And if
>> > "the Tree-sitter function is general and should work for every
>> > language", as you say (and I agree), why should we refrain from using
>> > it for C?
>>
>> Note that basing C-M-x on syntax tables (that is, traditional
>> forward-sexp) does not completely exclude the use of Tree-sitter, AFAIU.
>> Here's my thought process: To do its job, C-M-x needs to know about some
>> code structures such as symbol constituents, strings, comments, and
>> parenthetical groups.  If in some language or future version of C the
>> syntax is complex enough that getting the syntax class of a character
>> requires proper parsing, the Tree-sitter major modes can augment the
>> syntax table to make C-M-x work correctly.  See
>> c-ts-mode--syntax-propertize for an example of how Tree-sitter can
>> augment a buffer's syntax table, if needed.
>
> We have already C mode that uses syntax tables.  I think it's useful
> to try syntactic movement using results of parsing as well, and
> compare the relative merits and demerits.

After trying to tweak treesit-sexp-type-regexp a few times
I become convinced it is not up to the task of properly handling
all sexp operations.  It seems the existing functions that
implement C-M-f (forward-sexp), C-M-u (backward-up-list), etc.
should remain in place, and the role of the tree-sitter would be
only to provide syntax information for them, i.e. just to replace
syntax tables with tree-sitter wrappers.

bug#62238: 30.0.50; Unusual interpretation of "S-expressions" in c-ts-mode

[Eli Zaretskii]

> From: Juri Linkov <juri@linkov.net>
> Cc: Daniel Martín <mardani29@yahoo.es>,
>   62238@debbugs.gnu.org,
>   casouri@gmail.com,  theo@thornhill.no,  philipk@posteo.net
> Date: Sun, 19 Mar 2023 19:28:58 +0200
> 
> After trying to tweak treesit-sexp-type-regexp a few times
> I become convinced it is not up to the task of properly handling
> all sexp operations.  It seems the existing functions that
> implement C-M-f (forward-sexp), C-M-u (backward-up-list), etc.
> should remain in place, and the role of the tree-sitter would be
> only to provide syntax information for them, i.e. just to replace
> syntax tables with tree-sitter wrappers.

It would be good if you could tell more about what you tried and what
you saw, and how you reached this quite radical conclusion.  Maybe you
are right, but without knowing the details, how can we possibly
discuss it and try to reach a common conclusion?  Up front, the
conclusion sounds almost incredible: after all, how can a compiler
recognize blocks and defuns if not by parsing the program source?  If
the compiler does it, why cannot we do the same using the parsing
products?

So please consider making the discussion more useful by telling more.
(Perhaps on emacs-devel, not here.)

bug#62238: 30.0.50; Unusual interpretation of "S-expressions" in c-ts-mode

[Juri Linkov]

>> After trying to tweak treesit-sexp-type-regexp a few times
>> I become convinced it is not up to the task of properly handling
>> all sexp operations.  It seems the existing functions that
>> implement C-M-f (forward-sexp), C-M-u (backward-up-list), etc.
>> should remain in place, and the role of the tree-sitter would be
>> only to provide syntax information for them, i.e. just to replace
>> syntax tables with tree-sitter wrappers.
>
> It would be good if you could tell more about what you tried and what
> you saw, and how you reached this quite radical conclusion.  Maybe you
> are right, but without knowing the details, how can we possibly
> discuss it and try to reach a common conclusion?  Up front, the
> conclusion sounds almost incredible: after all, how can a compiler
> recognize blocks and defuns if not by parsing the program source?  If
> the compiler does it, why cannot we do the same using the parsing
> products?

Ideally, the existing syntax tables should still be used, and
tree-sitter could provide an additional support for "implicit parens"
as mentioned in bug#62086.  "Implicit parens" are such constructs
that are not defined by syntax tables, but perceived as invisible parens.

> So please consider making the discussion more useful by telling more.
> (Perhaps on emacs-devel, not here.)

I could provide more feedback when someone will start a discussion
on emacs-devel.

bug#62238: 30.0.50; Unusual interpretation of "S-expressions" in c-ts-mode

[Juri Linkov]

> Ideally, the existing syntax tables should still be used, and
> tree-sitter could provide an additional support for "implicit parens"
> as mentioned in bug#62086.  "Implicit parens" are such constructs
> that are not defined by syntax tables, but perceived as invisible parens.
>
>> So please consider making the discussion more useful by telling more.
>> (Perhaps on emacs-devel, not here.)
>
> I could provide more feedback when someone will start a discussion
> on emacs-devel.

There is another discussion in bug#62416 that could help to complete this feature.