bug#62416: 30.0.50; Symbols skipped in the navigation in ruby-ts-mode

bug#62416: 30.0.50; Symbols skipped in the navigation in ruby-ts-mode

[Boško Ivanišević]

[-- Attachment #1: Type: text/plain, Size: 13609 bytes --]

Navigating word forward/backward skips the Ruby symbol in the ruby-ts-mode.
Adding "simple_symbol" to the treesit-sexp-type-regexp is fixing it.

  (setq-local treesit-sexp-type-regexp
              (regexp-opt '("class"
                            "module"
                            "method"
                            "argument_list"
                            "array"
                            "hash"
                            "parenthesized_statements"
                            "if"
                            "case"
                            "when"
                            "block"
                            "do_block"
                            "begin"
                            "binary"
                            "simple_symbol" ;; <-- missing
                            "assignment")))

In GNU Emacs 30.0.50 (build 1, aarch64-apple-darwin22.3.0, NS
 appkit-2299.40 Version 13.2.1 (Build 22D68)) of 2023-03-14 built on
 Bosko-M2.local
Repository revision: 3fb30c8f1338ee8429411d6d36cd8ffbeb42f3cc
Repository branch: master
Windowing system distributor 'Apple', version 10.3.2299
System Description:  macOS 13.2.1

Configured using:
 'configure --with-native-compilation --with-xwidgets --with-tree-sitter
 --with-json --with-modules
 PKG_CONFIG_PATH=/opt/homebrew/opt:/opt/homebrew/opt/gcc/lib/pkgconfig:/opt/homebrew/opt/jansson/lib/pkgconfig:/opt/homebrew/opt/libxml2/lib/pkgconfig:/opt/homebrew/opt/gnutls/lib/pkgconfig:/opt/homebrew/opt/librsvg/lib/pkgconfig:/opt/homebrew/opt/p11-kit/lib/pkgconfig:/opt/homebrew/opt/libgccjit/lib/pkgconfig:/opt/homebrew/opt/tree-sitter/lib/pkgconfig:/opt/homebrew/opt/sqlite/lib/pkgconfig'

Configured features:
ACL GLIB GNUTLS JSON LCMS2 LIBXML2 MODULES NATIVE_COMP NOTIFY KQUEUE NS
PDUMPER PNG RSVG SQLITE3 THREADS TOOLKIT_SCROLL_BARS TREE_SITTER WEBP
XIM XWIDGETS ZLIB

Important settings:
  locale-coding-system: utf-8

Major mode: Ruby

Minor modes in effect:
  global-git-commit-mode: t
  magit-auto-revert-mode: t
  csv-field-index-mode: t
  auto-revert-mode: t
  volatile-highlights-mode: t
  yas-minor-mode: t
  savehist-mode: t
  which-key-mode: t
  server-mode: t
  global-rbenv-mode: t
  save-place-mode: t
  which-function-mode: t
  global-corfu-mode: t
  corfu-mode: t
  recentf-mode: t
  global-undo-tree-mode: t
  undo-tree-mode: t
  show-smartparens-global-mode: t
  show-smartparens-mode: t
  smartparens-global-mode: t
  smartparens-mode: t
  global-diff-hl-mode: t
  diff-hl-mode: t
  marginalia-mode: t
  vertico-mode: t
  eat-eshell-visual-command-mode: t
  eat-eshell-mode: t
  shell-dirtrack-mode: t
  global-treesit-auto-mode: t
  global-dash-fontify-mode: t
  override-global-mode: t
  delete-selection-mode: t
  global-hl-line-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  show-paren-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tab-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  column-number-mode: t
  line-number-mode: t
  transient-mark-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  hs-minor-mode: t

Load-path shadows:
/Users/bosko/.config/emacs/packages/30.0.50/magit-section-20230302.2130/magit-section-autoloads
hides
/Users/bosko/.config/emacs/packages/30.0.50/magit-section-20230314.1653/magit-section-autoloads
/Users/bosko/.config/emacs/packages/30.0.50/magit-section-20230302.2130/magit-section
hides
/Users/bosko/.config/emacs/packages/30.0.50/magit-section-20230314.1653/magit-section
/Users/bosko/.config/emacs/packages/30.0.50/magit-section-20230302.2130/magit-section-pkg
hides
/Users/bosko/.config/emacs/packages/30.0.50/magit-section-20230314.1653/magit-section-pkg
/Users/bosko/.config/emacs/packages/30.0.50/transient-20230315.1520/transient
hides /Applications/Emacs.app/Contents/Resources/lisp/transient

Features:
(shadow mail-extr emacsbug ielm subword-mode-expansions cap-words
superword subword completion dos-w32 find-cmd find-dired
browse-kill-ring expand-region text-mode-expansions cc-mode-expansions
the-org-mode-expansions ruby-mode-expansions nxml-mode-expansions
js-mode-expansions web-mode-expansions html-mode-expansions
css-mode-expansions er-basic-expansions expand-region-core
expand-region-custom consult-xref typescript-ts-mode web-mode tetris
gamegrid popup avy mc-edit-lines cl-print tempo smtpmail
consult-icomplete icomplete ido tar-mode arc-mode archive-mode
loaddefs-gen lisp-mnt mm-archive network-stream url-cache url-http
url-auth url-gw nsm display-line-numbers shortdoc json-ts-mode thai-util
thai-word lao-util enriched docker-cli cal-move po
mc-hide-unmatched-lines-mode mc-mark-more mc-cycle-cursors
multiple-cursors-core rect misearch multi-isearch move-text char-fold
dabbrev git-rebase help-fns radix-tree magit-subtree magit-gitignore
magit-ediff ediff ediff-merg ediff-mult ediff-wind ediff-diff ediff-help
ediff-init ediff-util misc css-mode consult-imenu rfc1345 quail
dired-aux diff-hl-dired rng-xsd xsd-regexp rng-cmpct rng-nxml rng-valid
rng-loc rng-uri rng-parse nxml-parse rng-match rng-dt rng-util rng-pttrn
nxml-ns nxml-mode nxml-outln nxml-rap nxml-util nxml-enc xmltok
magit-patch vc-hg vc-bzr vc-src vc-sccs vc-svn vc-cvs vc-rcs
bug-reference flyspell ispell magit-extras git-timemachine embark-org
embark-consult embark ffap magit-bookmark magit-submodule magit-blame
magit-stash magit-reflog magit-bisect magit-push magit-pull magit-fetch
magit-clone magit-remote magit-commit magit-sequence magit-notes
magit-worktree magit-tag magit-merge magit-branch magit-reset
magit-files magit-refs magit-status magit magit-repos magit-apply
magit-wip magit-log magit-diff git-commit log-edit magit-core
magit-autorevert magit-margin magit-transient magit-process with-editor
magit-mode transient magit-git magit-base magit-section crm yaml-ts-mode
csv-mode sort smartparens-markdown markdown-mode color pcmpl-unix
em-unix em-term em-script em-prompt em-pred em-ls em-hist em-glob
em-extpipe em-cmpl em-dirs em-basic em-banner em-alias eshell esh-mode
esh-var esh-cmd esh-ext esh-opt esh-proc esh-io esh-arg esh-module
esh-groups esh-util cus-edit cus-start cus-load files-x diary-lib
diary-loaddefs cal-iso face-remap whitespace smerge-mode inf-ruby sql
view sh-script executable oc-basic ol-eww eww url-queue mm-url ol-rmail
ol-mhe ol-irc ol-info ol-gnus nnselect gnus-art mm-uu mml2015 mm-view
mml-smime smime dig gnus-sum shr pixel-fill kinsoku url-file svg
gnus-group gnus-undo gnus-start gnus-dbus dbus gnus-cloud nnimap nnmail
mail-source utf7 nnoo parse-time gnus-spec gnus-int gnus-range message
sendmail rfc822 mml mml-sec epa derived epg rfc6068 epg-config mm-decode
mm-bodies mm-encode mail-parse rfc2231 rfc2047 rfc2045 ietf-drums
mailabbrev gmm-utils gnus-win gnus nnheader gnus-util range mm-util
mail-prsvr ol-docview doc-view jka-compr image-mode exif ol-bibtex
bibtex iso8601 ol-bbdb ol-w3m ol-doi org-link-doi org-superstar ob-js
ob-sql ob-css ob-ruby ob-dot ob-plantuml ob-ditaa ob-shell org-crypt
org-habit org-agenda org-element org-persist xdg org-id avl-tree
org-refile smartparens-org org ob-graphql request mailheader mail-utils
graphql-mode let-alist ob-restclient restclient ob ob-tangle ob-ref
ob-lob ob-table ob-exp org-macro org-src ob-comint org-pcomplete
org-list org-footnote org-faces org-entities noutline outline
ob-emacs-lisp ob-core ob-eval org-cycle org-table ol org-fold
org-fold-core org-keys oc org-loaddefs cal-menu calendar cal-loaddefs
org-version org-compat org-macs yank-media smartparens-html sgml-mode
facemenu dom smartparens-javascript js c-ts-common smartparens-c cc-mode
cc-fonts cc-guess cc-menus cc-cmds cc-styles cc-align cc-engine cc-vars
cc-defs ruby-ts-mode smartparens-ruby ruby-mode smie time-date vc-git
hideshow eglot external-completion array jsonrpc ert debug backtrace
find-func flymake-proc flymake smartparens-elixir elixir-ts-mode
heex-ts-mode desktop frameset cursor-sensor autorevert filenotify term
disp-table ehelp add-log etags fileloop generator xref project
volatile-highlights yasnippet-snippets yasnippet savehist which-key init
server pinentry init-packages empv ibuf-macs rbenv saveplace
erlang-start which-func imenu corfu-terminal popon corfu recentf
tree-widget wid-edit undo-tree diff queue diminish smartparens-config
smartparens-text smartparens advice loadhist diff-hl smartrep log-view
pcvs-util vc-dir ewoc vc vc-dispatcher diff-mode git-modes
gitignore-mode gitconfig-mode conf-mode gitattributes-mode
all-the-icons-dired all-the-icons all-the-icons-faces data-material
data-weathericons data-octicons data-fileicons data-faicons
data-alltheicons modus-vivendi-theme modus-themes wgrep grep compile
consult-vertico consult bookmark text-property-search pp marginalia
orderless dired-subtree dired-hacks-utils vertico compat dash-docs
use-package-dash-docs gnutls puny thingatpt async format-spec xml
edmacro kmacro eat shell pcomplete comint ansi-osc ring ansi-color
exec-path-from-shell treesit-auto treesit f f-shortdoc dash s dired-x
dired dired-loaddefs use-package use-package-ensure use-package-delight
use-package-diminish use-package-bind-key bind-key easy-mmode
use-package-core finder-inf magit-autoloads pcase
docker-compose-mode-autoloads ibuffer-vc-autoloads
json-snatcher-autoloads magit-section-autoloads ibuffer-git-autoloads
inf-ruby-autoloads wgrep-autoloads eat-autoloads move-text-autoloads
yari-autoloads ibuffer-tramp-autoloads rainbow-delimiters-autoloads
smex-autoloads easy-kill-autoloads avy-autoloads yaml-mode-autoloads
markup-faces-autoloads orderless-autoloads git-timemachine-autoloads
git-commit-autoloads railscasts-theme-autoloads
browse-kill-ring-autoloads pinentry-autoloads consult-dash-autoloads
ov-autoloads csv-mode-autoloads change-inner-autoloads
ob-graphql-autoloads graphql-mode-autoloads adoc-mode-autoloads
vlf-autoloads neotree-autoloads docker-cli-autoloads
all-the-icons-dired-autoloads all-the-icons-autoloads f-autoloads
smartparens-autoloads undo-tree-autoloads queue-autoloads
flycheck-autoloads websocket-autoloads dash-docs-autoloads
popup-autoloads gist-autoloads which-key-autoloads rbenv-autoloads
vertico-autoloads textile-mode-autoloads request-autoloads
smartrep-autoloads expand-region-autoloads sass-mode-autoloads
haml-mode-autoloads embark-consult-autoloads consult-autoloads
embark-autoloads anzu-autoloads marginalia-autoloads
with-editor-autoloads gh-autoloads marshal-autoloads logito-autoloads
pcache-autoloads async-autoloads pkg-info-autoloads epl-autoloads
diminish-autoloads git-modes-autoloads treesit-auto-autoloads
typescript-mode-autoloads scss-mode-autoloads imenu-anywhere-autoloads
modus-themes-autoloads web-mode-autoloads multiple-cursors-autoloads
elisp-slime-nav-autoloads volatile-highlights-autoloads
markdown-mode-autoloads dockerfile-mode-autoloads litable-autoloads
org-superstar-autoloads corfu-terminal-autoloads popon-autoloads
corfu-autoloads ob-restclient-autoloads restclient-autoloads
discover-my-major-autoloads makey-autoloads diff-hl-autoloads
fringe-helper-autoloads julia-shell-autoloads julia-mode-autoloads
yasnippet-snippets-autoloads yasnippet-autoloads empv-autoloads
docker-autoloads transient-autoloads compat-autoloads tablist-autoloads
s-autoloads aio-autoloads impatient-mode-autoloads htmlize-autoloads
simple-httpd-autoloads dired-subtree-autoloads
dired-hacks-utils-autoloads exec-path-from-shell-autoloads ht-autoloads
info dash-autoloads package browse-url url url-proxy url-privacy
url-expand url-methods url-history url-cookie generate-lisp-file
url-domsuf url-util mailcap url-handlers url-parse auth-source eieio
eieio-core password-cache json map byte-opt url-vars comp comp-cstr
warnings icons subr-x rx cl-seq cl-macs gv cl-extra help-mode
cl-loaddefs cl-lib bytecomp byte-compile delsel hl-line init-basic rmc
iso-transl tooltip cconv eldoc paren electric uniquify ediff-hook
vc-hooks lisp-float-type elisp-mode mwheel term/ns-win ns-win
ucs-normalize mule-util term/common-win tool-bar dnd fontset image
regexp-opt fringe tabulated-list replace newcomment text-mode lisp-mode
prog-mode register page tab-bar menu-bar rfn-eshadow isearch easymenu
timer select scroll-bar mouse jit-lock font-lock syntax font-core
term/tty-colors frame minibuffer nadvice seq simple cl-generic
indonesian philippine cham georgian utf-8-lang misc-lang vietnamese
tibetan thai tai-viet lao korean japanese eucjp-ms cp51932 hebrew greek
romanian slovak czech european ethiopic indian cyrillic chinese
composite emoji-zwj charscript charprop case-table epa-hook
jka-cmpr-hook help abbrev obarray oclosure cl-preloaded button loaddefs
theme-loaddefs faces cus-face macroexp files window text-properties
overlay sha1 md5 base64 format env code-pages mule custom widget keymap
hashtable-print-readable backquote threads xwidget-internal kqueue cocoa
ns lcms2 multi-tty make-network-process native-compile emacs)

Memory information:
((conses 16 10314598 2323441)
 (symbols 48 150486 125)
 (strings 32 645485 143465)
 (string-bytes 1 1444475747)
 (vectors 16 196492)
 (vector-slots 8 4033147 2066742)
 (floats 8 2001 10400)
 (intervals 56 664485 22032)
 (buffers 984 174))

Regards,
Boško Ivanišević
http://pragdevnotes.com
Ruby on Windows Guides <http://rubyonwindowsguides.github.io>
@boskoivanisevic <https://twitter.com/boskoivanisevic/>

[-- Attachment #2: Type: text/html, Size: 14896 bytes --]

bug#62416: 30.0.50; Symbols skipped in the navigation in ruby-ts-mode

[Dmitry Gutov]

On 24/03/2023 12:24, Boško Ivanišević wrote:
> Navigating word forward/backward skips the Ruby symbol in the ruby-ts-mode.
> Adding "simple_symbol" to the treesit-sexp-type-regexp is fixing it.
> 
>    (setq-local treesit-sexp-type-regexp
>                (regexp-opt '("class"
>                              "module"
>                              "method"
>                              "argument_list"
>                              "array"
>                              "hash"
>                              "parenthesized_statements"
>                              "if"
>                              "case"
>                              "when"
>                              "block"
>                              "do_block"
>                              "begin"
>                              "binary"
>                              "simple_symbol" ;; <-- missing
>                              "assignment")))

Juri, what do you think?

In the context of your previous feedback regarding sexp navigation in 
ruby-ts-mode.

bug#62416: 30.0.50; Symbols skipped in the navigation in ruby-ts-mode

[Juri Linkov]

>> Navigating word forward/backward skips the Ruby symbol in the ruby-ts-mode.
>> Adding "simple_symbol" to the treesit-sexp-type-regexp is fixing it.
>>    (setq-local treesit-sexp-type-regexp
>>                (regexp-opt '("class"
>>                              "module"
>>                              "method"
>>                              "argument_list"
>>                              "array"
>>                              "hash"
>>                              "parenthesized_statements"
>>                              "if"
>>                              "case"
>>                              "when"
>>                              "block"
>>                              "do_block"
>>                              "begin"
>>                              "binary"
>>                              "simple_symbol" ;; <-- missing
>>                              "assignment")))
>
> Juri, what do you think?
>
> In the context of your previous feedback regarding sexp navigation in
> ruby-ts-mode.

This is fine.  But anyway I think in its current state
treesit-sexp-type-regexp is underdesigned as noted in
https://debbugs.gnu.org/cgi/bugreport.cgi?bug=62238#59
so no tweaking could fix its design flaws.  For example,

  foo = {
    a: b
  }

when point is on the left curly bracket, 'C-M-f C-M-b' doesn't
move back to the original position, etc.

bug#62416: 30.0.50; Symbols skipped in the navigation in ruby-ts-mode

[Dmitry Gutov]

On 25/03/2023 21:04, Juri Linkov wrote:
>>> Navigating word forward/backward skips the Ruby symbol in the ruby-ts-mode.
>>> Adding "simple_symbol" to the treesit-sexp-type-regexp is fixing it.
>>>     (setq-local treesit-sexp-type-regexp
>>>                 (regexp-opt '("class"
>>>                               "module"
>>>                               "method"
>>>                               "argument_list"
>>>                               "array"
>>>                               "hash"
>>>                               "parenthesized_statements"
>>>                               "if"
>>>                               "case"
>>>                               "when"
>>>                               "block"
>>>                               "do_block"
>>>                               "begin"
>>>                               "binary"
>>>                               "simple_symbol" ;; <-- missing
>>>                               "assignment")))
>>
>> Juri, what do you think?
>>
>> In the context of your previous feedback regarding sexp navigation in
>> ruby-ts-mode.
> 
> This is fine.

Good. Let's push it into master.

> But anyway I think in its current state
> treesit-sexp-type-regexp is underdesigned as noted in
> https://debbugs.gnu.org/cgi/bugreport.cgi?bug=62238#59
> so no tweaking could fix its design flaws.  For example,
> 
>    foo = {
>      a: b
>    }
> 
> when point is on the left curly bracket, 'C-M-f C-M-b' doesn't
> move back to the original position, etc.

Hm, yeah, it's a little unexpected that not only it doesn't move back 
exactly, it skips the full assignment expression in both directions.

What is the downside of dropping "binary" and "assignment" from 
treesit-sexp-type-regexp, added there not too long ago? Without 
"assignment" there, your example becomes a round-trip.

BTW, we should wrap the regexp with \`...\'. It currently does substring 
matching, that's why it also matches "identifier" (which should probably 
be added too).

bug#62416: 30.0.50; Symbols skipped in the navigation in ruby-ts-mode

[Juri Linkov]

>> But anyway I think in its current state
>> treesit-sexp-type-regexp is underdesigned as noted in
>> https://debbugs.gnu.org/cgi/bugreport.cgi?bug=62238#59
>> so no tweaking could fix its design flaws.  For example,
>>    foo = {
>>      a: b
>>    }
>> when point is on the left curly bracket, 'C-M-f C-M-b' doesn't
>> move back to the original position, etc.
>
> Hm, yeah, it's a little unexpected that not only it doesn't move back
> exactly, it skips the full assignment expression in both directions.
>
> What is the downside of dropping "binary" and "assignment" from
> treesit-sexp-type-regexp, added there not too long ago? Without
> "assignment" there, your example becomes a round-trip.

This will break other cases, e.g.

  b = %Q{This is a "string"}
  c = %w!foo
   bar
   baz!
  d = %(hello (nested) world)

when point is after "b", 'C-M-f' will move to "c" instead of the end of line.

bug#62416: 30.0.50; Symbols skipped in the navigation in ruby-ts-mode

[Dmitry Gutov]

On 27/03/2023 19:29, Juri Linkov wrote:
>>> But anyway I think in its current state
>>> treesit-sexp-type-regexp is underdesigned as noted in
>>> https://debbugs.gnu.org/cgi/bugreport.cgi?bug=62238#59
>>> so no tweaking could fix its design flaws.  For example,
>>>     foo = {
>>>       a: b
>>>     }
>>> when point is on the left curly bracket, 'C-M-f C-M-b' doesn't
>>> move back to the original position, etc.
>>
>> Hm, yeah, it's a little unexpected that not only it doesn't move back
>> exactly, it skips the full assignment expression in both directions.
>>
>> What is the downside of dropping "binary" and "assignment" from
>> treesit-sexp-type-regexp, added there not too long ago? Without
>> "assignment" there, your example becomes a round-trip.
> 
> This will break other cases, e.g.
> 
>    b = %Q{This is a "string"}
>    c = %w!foo
>     bar
>     baz!
>    d = %(hello (nested) world)
> 
> when point is after "b", 'C-M-f' will move to "c" instead of the end of line.

That is because the string literals aren't recognized as sexps yet. Try 
this:

diff --git a/lisp/progmodes/ruby-ts-mode.el b/lisp/progmodes/ruby-ts-mode.el
index d1034d467ab..c46247c747a 100644
--- a/lisp/progmodes/ruby-ts-mode.el
+++ b/lisp/progmodes/ruby-ts-mode.el
@@ -1129,8 +1129,11 @@ ruby-ts-mode
                              "block"
                              "do_block"
                              "begin"
-                            "binary"
-                            "assignment")))
+                            "integer"
+                            "simple_symbol"
+                            "string"
+                            "string_array"
+                            )))

    ;; AFAIK, Ruby can not nest methods
    (setq-local treesit-defun-prefer-top-level nil)

bug#62416: 30.0.50; Symbols skipped in the navigation in ruby-ts-mode

[Juri Linkov]

>> This will break other cases, e.g.
>>    b = %Q{This is a "string"}
>>    c = %w!foo
>>     bar
>>     baz!
>>    d = %(hello (nested) world)
>> when point is after "b", 'C-M-f' will move to "c" instead of the end of
>> line.
>
> That is because the string literals aren't recognized as sexps yet. Try
> this:
>
> @@ -1129,8 +1129,11 @@ ruby-ts-mode
>                              "block"
>                              "do_block"
>                              "begin"
> -                            "binary"
> -                            "assignment")))
> +                            "integer"
> +                            "simple_symbol"
> +                            "string"
> +                            "string_array"
> +                            )))

Thanks, this definitely is an improvement since it handles all mentioned cases.

However, there are still a lot of more things that need fixing.
When point is on the left curly bracket in

  b = %Q{This is a "string"}

'C-M-f' doesn't move to the right curly bracket.
Also double quotes inside the string are not matched by 'C-M-f'.

In

  d = %(hello (nested) world)

'C-M-f' doesn't move to the closing parens from opening parens.

Looking at test/lisp/progmodes/ruby-mode-resources/ruby.rb
for example here curly brackets are not matched with 'C-M-f C-M-b'
in string interpolation and regexps:

  "abc/#{ddf}ghi"

  /foo/xi != %r{bar}mo.tee

Do you think it is possible to handle these cases
by crafting treesit-sexp-type-regexp?

bug#62416: 30.0.50; Symbols skipped in the navigation in ruby-ts-mode

[Dmitry Gutov]

On 28/03/2023 09:31, Juri Linkov wrote:
>>> This will break other cases, e.g.
>>>     b = %Q{This is a "string"}
>>>     c = %w!foo
>>>      bar
>>>      baz!
>>>     d = %(hello (nested) world)
>>> when point is after "b", 'C-M-f' will move to "c" instead of the end of
>>> line.
>>
>> That is because the string literals aren't recognized as sexps yet. Try
>> this:
>>
>> @@ -1129,8 +1129,11 @@ ruby-ts-mode
>>                               "block"
>>                               "do_block"
>>                               "begin"
>> -                            "binary"
>> -                            "assignment")))
>> +                            "integer"
>> +                            "simple_symbol"
>> +                            "string"
>> +                            "string_array"
>> +                            )))
> 
> Thanks, this definitely is an improvement since it handles all mentioned cases.

Very good, I've pushed that change, and with that I'm closing this bug. 
Thank you both.

> However, there are still a lot of more things that need fixing.
> When point is on the left curly bracket in
> 
>    b = %Q{This is a "string"}
> 
> 'C-M-f' doesn't move to the right curly bracket.
> Also double quotes inside the string are not matched by 'C-M-f'.
> 
> In
> 
>    d = %(hello (nested) world)
> 
> 'C-M-f' doesn't move to the closing parens from opening parens.
> 
> Looking at test/lisp/progmodes/ruby-mode-resources/ruby.rb
> for example here curly brackets are not matched with 'C-M-f C-M-b'
> in string interpolation and regexps:
> 
>    "abc/#{ddf}ghi"
> 
>    /foo/xi != %r{bar}mo.tee
> 
> Do you think it is possible to handle these cases
> by crafting treesit-sexp-type-regexp?

I don't think so. tree-sitter parse tree has no information about these 
parens or their positions.

So we'd need to do this with some custom code: check the context (e.g. 
see that we are inside a string) and then move over parens and etc. It's 
not out of the question to do that in the common tree-sitter code, too 
(Ruby doesn't look very special in having strings). But whether this is 
useful enough to expend that effort, seems like a subject for a separate 
discussion (report-emacs-bug or emacs-devel -- your choice).

bug#62416: 30.0.50; Symbols skipped in the navigation in ruby-ts-mode

[Juri Linkov]

>> Looking at test/lisp/progmodes/ruby-mode-resources/ruby.rb
>> for example here curly brackets are not matched with 'C-M-f C-M-b'
>> in string interpolation and regexps:
>>    "abc/#{ddf}ghi"
>>    /foo/xi != %r{bar}mo.tee
>> Do you think it is possible to handle these cases
>> by crafting treesit-sexp-type-regexp?
>
> I don't think so. tree-sitter parse tree has no information about these
> parens or their positions.

Actually, it has information about string interpolation,
so adding "interpolation" to 'treesit-sexp-type-regexp'
allows to navigate "#{ddf}" in "abc/#{ddf}ghi".

But other paired characters in strings have no parsed information
and need examining the text in the buffer indeed.

bug#62416: 30.0.50; Symbols skipped in the navigation in ruby-ts-mode

[Dmitry Gutov]

On 03/04/2023 19:03, Juri Linkov wrote:
>>> Looking at test/lisp/progmodes/ruby-mode-resources/ruby.rb
>>> for example here curly brackets are not matched with 'C-M-f C-M-b'
>>> in string interpolation and regexps:
>>>     "abc/#{ddf}ghi"
>>>     /foo/xi != %r{bar}mo.tee
>>> Do you think it is possible to handle these cases
>>> by crafting treesit-sexp-type-regexp?
>> I don't think so. tree-sitter parse tree has no information about these
>> parens or their positions.
> Actually, it has information about string interpolation,
> so adding "interpolation" to 'treesit-sexp-type-regexp'
> allows to navigate "#{ddf}" in "abc/#{ddf}ghi".

Now added in commit bd5c1d1cbbd.

Note that it doesn't pair the curly braces either: the beginning is 
before '#'.

> But other paired characters in strings have no parsed information
> and need examining the text in the buffer indeed.

Right.

bug#62416: 30.0.50; Symbols skipped in the navigation in ruby-ts-mode

[Yuan Fu]

Juri Linkov <juri@linkov.net> writes:

>>> This will break other cases, e.g.
>>>    b = %Q{This is a "string"}
>>>    c = %w!foo
>>>     bar
>>>     baz!
>>>    d = %(hello (nested) world)
>>> when point is after "b", 'C-M-f' will move to "c" instead of the end of
>>> line.
>>
>> That is because the string literals aren't recognized as sexps yet. Try
>> this:
>>
>> @@ -1129,8 +1129,11 @@ ruby-ts-mode
>>                              "block"
>>                              "do_block"
>>                              "begin"
>> -                            "binary"
>> -                            "assignment")))
>> +                            "integer"
>> +                            "simple_symbol"
>> +                            "string"
>> +                            "string_array"
>> +                            )))
>
> Thanks, this definitely is an improvement since it handles all mentioned cases.
>
> However, there are still a lot of more things that need fixing.
> When point is on the left curly bracket in
>
>   b = %Q{This is a "string"}
>
> 'C-M-f' doesn't move to the right curly bracket.
> Also double quotes inside the string are not matched by 'C-M-f'.
>
> In
>
>   d = %(hello (nested) world)
>
> 'C-M-f' doesn't move to the closing parens from opening parens.

Have someone fixed these two cases? Because when I tried to invoke
(treesit-forward-sexp), point moved to the closing bracket/paren.

Anyway, I just wonder if there’s any fundamental shortcoming with how
treesit-beginning/end-of-thing works?

Yuan

bug#62416: 30.0.50; Symbols skipped in the navigation in ruby-ts-mode

[Dmitry Gutov]

On 30/03/2023 10:47, Yuan Fu wrote:

>> However, there are still a lot of more things that need fixing.
>> When point is on the left curly bracket in
>>
>>    b = %Q{This is a "string"}
>>
>> 'C-M-f' doesn't move to the right curly bracket.
>> Also double quotes inside the string are not matched by 'C-M-f'.
>>
>> In
>>
>>    d = %(hello (nested) world)
>>
>> 'C-M-f' doesn't move to the closing parens from opening parens.
> 
> Have someone fixed these two cases? Because when I tried to invoke
> (treesit-forward-sexp), point moved to the closing bracket/paren.

 From which position? When point is right before '{', it doesn't move in 
my testing. It does move when it was before '%'.

> Anyway, I just wonder if there’s any fundamental shortcoming with how
> treesit-beginning/end-of-thing works?

I don't know. Seems like this method is good for a lot of things, but 
some fiddly details are going to be different from the default forward-sexp.

ruby-mode's sexp navigation is also not ideal in its own way, and it's 
been useful anyway.

bug#62416: 30.0.50; Symbols skipped in the navigation in ruby-ts-mode

[Yuan Fu]

> On Mar 30, 2023, at 2:32 AM, Dmitry Gutov <dgutov@yandex.ru> wrote:
> 
> On 30/03/2023 10:47, Yuan Fu wrote:
> 
>>> However, there are still a lot of more things that need fixing.
>>> When point is on the left curly bracket in
>>> 
>>>   b = %Q{This is a "string"}
>>> 
>>> 'C-M-f' doesn't move to the right curly bracket.
>>> Also double quotes inside the string are not matched by 'C-M-f'.
>>> 
>>> In
>>> 
>>>   d = %(hello (nested) world)
>>> 
>>> 'C-M-f' doesn't move to the closing parens from opening parens.
>> Have someone fixed these two cases? Because when I tried to invoke
>> (treesit-forward-sexp), point moved to the closing bracket/paren.
> 
> From which position? When point is right before '{', it doesn't move in my testing. It does move when it was before '%'.

Ok, I see it. I’ll try to see what’s going on when I find some time. 

> 
>> Anyway, I just wonder if there’s any fundamental shortcoming with how
>> treesit-beginning/end-of-thing works?
> 
> I don't know. Seems like this method is good for a lot of things, but some fiddly details are going to be different from the default forward-sexp.
> 
> ruby-mode's sexp navigation is also not ideal in its own way, and it's been useful anyway.

One thing I noticed is that treesit-forward-sexp uses treesit-beginning/end-of-thing, which jumps out of the parent when there is no more siblings to go to. The default forward-sexp obviously doesn’t do this. Perhaps we should stay in the same level in tree-sitter-forward-sexp too. 

Yuan

bug#62416: 30.0.50; Symbols skipped in the navigation in ruby-ts-mode

[Dmitry Gutov]

On 03/04/2023 01:34, Yuan Fu wrote:
> 
>> On Mar 30, 2023, at 2:32 AM, Dmitry Gutov<dgutov@yandex.ru>  wrote:
>>
>> On 30/03/2023 10:47, Yuan Fu wrote:
>>
>>>> However, there are still a lot of more things that need fixing.
>>>> When point is on the left curly bracket in
>>>>
>>>>    b = %Q{This is a "string"}
>>>>
>>>> 'C-M-f' doesn't move to the right curly bracket.
>>>> Also double quotes inside the string are not matched by 'C-M-f'.
>>>>
>>>> In
>>>>
>>>>    d = %(hello (nested) world)
>>>>
>>>> 'C-M-f' doesn't move to the closing parens from opening parens.
>>> Have someone fixed these two cases? Because when I tried to invoke
>>> (treesit-forward-sexp), point moved to the closing bracket/paren.
>>  From which position? When point is right before '{', it doesn't move in my testing. It does move when it was before '%'.
> Ok, I see it. I’ll try to see what’s going on when I find some time.

There is nothing surprising in this behavior, given the current 
implementation: it doesn't examine the text in the buffer, just uses the 
parse tree,

And there is no trace of these parens/braces in the parse tree.

>>> Anyway, I just wonder if there’s any fundamental shortcoming with how
>>> treesit-beginning/end-of-thing works?
>> I don't know. Seems like this method is good for a lot of things, but some fiddly details are going to be different from the default forward-sexp.
>>
>> ruby-mode's sexp navigation is also not ideal in its own way, and it's been useful anyway.
> One thing I noticed is that treesit-forward-sexp uses treesit-beginning/end-of-thing, which jumps out of the parent when there is no more siblings to go to. The default forward-sexp obviously doesn’t do this. Perhaps we should stay in the same level in tree-sitter-forward-sexp too.

That's a different aspect of its behavior. One that makes 
'backward-up-list' fail to work, IIUC.