bug#74277: 29.4; rust-ts-mode doesn't highlight some function calls

bug#74277: 29.4; rust-ts-mode doesn't highlight some function calls

[acidbong--- via Bug reports for GNU Emacs, the Swiss army knife of text editors]

rust-ts-mode highlights function calls when they're used in an
assignment or as statements, but not in string formatting:


```
println!("Some line and {}", some_function(a, b))
                           # ^^^^^^^^^^^^^ this is not getting highlighted
```

This is how it looks in Emacs (treesit-level is 4 btw):
https://matrix.envs.net/_matrix/media/r0/download/envs.net/66652827fc7545e57d2eef263ee28ef6bc2ef1c01854976160883015680

This is how it looks in Neovim (with treesitter highlighting on):
https://matrix.envs.net/_matrix/media/r0/download/envs.net/382a104ee3e6e4dacabebfa7f84f4b9b7231b1171854976151936565248

-----------------------------------

In GNU Emacs 29.4 (build 1, x86_64-pc-linux-gnu, X toolkit, cairo
version 1.18.2, Xaw3d scroll bars)
Windowing system distributor 'The X.Org Foundation', version 11.0.12101013
System Description: NixOS 24.11 (Vicuna)

Configured using:
 'configure
 --prefix=/nix/store/fsglyaxfhlavaskr159b4yyjlajy4qq5-emacs-29.4
 --disable-build-details --with-modules --with-x-toolkit=lucid
 --with-cairo --with-xft --with-compress-install
 --with-toolkit-scroll-bars --with-native-compilation
 --without-imagemagick --with-mailutils --without-small-ja-dic
 --with-tree-sitter --with-xinput2 --without-xwidgets --with-dbus
 --with-selinux'

Configured features:
CAIRO DBUS FREETYPE GIF GLIB GMP GNUTLS GPM GSETTINGS HARFBUZZ JPEG JSON
LIBOTF LIBSELINUX LIBSYSTEMD LIBXML2 M17N_FLT MODULES NATIVE_COMP NOTIFY
INOTIFY PDUMPER PNG RSVG SECCOMP SOUND SQLITE3 THREADS TIFF
TOOLKIT_SCROLL_BARS TREE_SITTER WEBP X11 XAW3D XDBE XIM XINPUT2 XPM
LUCID ZLIB

Important settings:
  value of $EMACSLOADPATH: 
  value of $EMACSNATIVELOADPATH: 
  value of $LANG: ru_RU.UTF-8
  locale-coding-system: utf-8-unix

Major mode: Rust

Minor modes in effect:
  consult-eglot-embark-mode: t
  eglot-inlay-hints-mode: t
  eglot--managed-mode: t
  flymake-mode: t
  server-mode: t
  display-line-numbers-mode: t
  rainbow-mode: t
  corfu-popupinfo-mode: t
  corfu-history-mode: t
  corfu-mode: t
  doom-modeline-mode: t
  nerd-icons-completion-mode: t
  marginalia-mode: t
  vertico-mode: t
  global-treesit-auto-mode: t
  diff-hl-flydiff-mode: t
  global-diff-hl-mode: t
  diff-hl-mode: t
  ws-butler-global-mode: t
  ws-butler-mode: t
  direnv-mode: t
  global-ligature-mode: t
  ligature-mode: t
  which-key-mode: t
  meow-global-mode: t
  meow-mode: t
  meow-normal-mode: t
  meow-esc-mode: t
  override-global-mode: t
  delete-selection-mode: t
  global-auto-revert-mode: t
  recentf-mode: t
  save-place-mode: t
  savehist-mode: t
  global-hl-line-mode: t
  electric-pair-mode: t
  global-eldoc-mode: t
  eldoc-mode: t
  show-paren-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  size-indication-mode: t
  column-number-mode: t
  line-number-mode: t
  global-visual-line-mode: t
  visual-line-mode: t
  transient-mark-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t

Load-path shadows:
/nix/store/qm7zdj12cb53nm3lvyh7pmf9kzfbz613-emacs-packages-deps/share/emacs/site-lisp/site-start hides /nix/store/fsglyaxfhlavaskr159b4yyjlajy4qq5-emacs-29.4/share/emacs/site-lisp/site-start
/nix/store/qm7zdj12cb53nm3lvyh7pmf9kzfbz613-emacs-packages-deps/share/emacs/site-lisp/elpa/transient-20241104.2211/transient hides /nix/store/fsglyaxfhlavaskr159b4yyjlajy4qq5-emacs-29.4/share/emacs/29.4/lisp/transient
/nix/store/qm7zdj12cb53nm3lvyh7pmf9kzfbz613-emacs-packages-deps/share/emacs/site-lisp/elpa/seq-2.24/seq hides /nix/store/fsglyaxfhlavaskr159b4yyjlajy4qq5-emacs-29.4/share/emacs/29.4/lisp/emacs-lisp/seq

Features:
(edmacro shadow sort mail-extr emacsbug consult-eglot-embark
consult-eglot consult-eglot-embark-autoloads consult-eglot-autoloads
loaddefs-gen lisp-mnt radix-tree tar-mode arc-mode archive-mode
mm-archive message sendmail yank-media rfc822 mml mml-sec epa derived
epg rfc6068 gnus-util mailabbrev gmm-utils mailheader mm-decode
mm-bodies mm-encode epg-config mail-utils gnutls network-stream
url-cache url-http url-auth mail-parse rfc2231 rfc2047 rfc2045 mm-util
ietf-drums mail-prsvr url-gw nsm puny cus-edit cus-start cus-load
descr-text embark-consult consult bookmark eglot external-completion
jsonrpc ert debug backtrace xref flymake-proc flymake imenu vc-git
rust-ts-mode c-ts-common misearch multi-isearch dabbrev cape-char pp
mule-util face-remap embark-org org-element org-persist xdg org-id
org-refile avl-tree generator embark ffap thingatpt project pulse server
display-line-numbers rainbow-mode corfu-popupinfo corfu-history
nerd-icons-corfu corfu doom-modeline doom-modeline-segments
doom-modeline-env doom-modeline-core shrink-path f s org ob ob-tangle
ob-ref ob-lob ob-table ob-exp org-macro org-src ob-comint org-pcomplete
pcomplete org-list org-footnote org-faces org-entities time-date
noutline outline ob-emacs-lisp ob-core ob-eval org-cycle org-table ol
org-fold org-fold-core org-keys oc org-loaddefs find-func cal-menu
calendar cal-loaddefs org-version org-compat org-macs format-spec cape
nerd-icons-completion marginalia vertico orderless compat compat-30
treesit-auto treesit meson-mode smie compile text-property-search comint
ansi-osc ansi-color ring diff-hl-dired dired dired-loaddefs
diff-hl-flydiff diff diff-hl log-view pcvs-util vc-dir ewoc vc
vc-dispatcher ws-butler direnv diff-mode dash ligature nerd-icons
nerd-icons-faces nerd-icons-data nerd-icons-data-mdicon
nerd-icons-data-flicon nerd-icons-data-codicon nerd-icons-data-devicon
nerd-icons-data-sucicon nerd-icons-data-wicon nerd-icons-data-faicon
nerd-icons-data-powerline nerd-icons-data-octicon
nerd-icons-data-pomicon nerd-icons-data-ipsicon doom-themes-ext-org
doom-gruvbox-theme doom-themes doom-themes-base which-key meow
meow-tutor meow-cheatsheet meow-cheatsheet-layout meow-core meow-shims
meow-esc meow-command array meow-beacon kmacro meow-thing meow-visual
meow-keypad meow-helpers meow-util color meow-keymap meow-face meow-var
bind-key easy-mmode comp comp-cstr warnings icons cl-extra help-mode
delsel autorevert filenotify recentf tree-widget wid-edit saveplace
savehist hl-line elec-pair use-package-ensure use-package-core
finder-inf consult-project-extra-autoloads cue-sheet-mode-autoloads rx
inputrc-mode-autoloads ws-butler-autoloads cape-autoloads
corfu-autoloads diff-hl-autoloads direnv-autoloads
doom-modeline-autoloads doom-themes-autoloads embark-consult-autoloads
consult-autoloads embark-autoloads ligature-autoloads magit-autoloads
pcase magit-section-autoloads marginalia-autoloads meow-autoloads
meson-mode-autoloads nerd-icons-completion-autoloads
nerd-icons-corfu-autoloads nerd-icons-dired-autoloads
nerd-icons-autoloads nix-ts-mode-autoloads orderless-autoloads
rainbow-mode-autoloads shrink-path-autoloads f-autoloads dash-autoloads
s-autoloads transient-autoloads treesit-auto-autoloads vertico-autoloads
which-key-autoloads with-editor-autoloads info compat-autoloads package
browse-url url url-proxy url-privacy url-expand url-methods url-history
url-cookie generate-lisp-file url-domsuf url-util mailcap url-handlers
url-parse auth-source cl-seq eieio eieio-core cl-macs password-cache
json subr-x map byte-opt gv bytecomp byte-compile url-vars cl-loaddefs
cl-lib cyril-util rmc iso-transl tooltip cconv eldoc paren electric
uniquify ediff-hook vc-hooks lisp-float-type elisp-mode mwheel
term/x-win x-win term/common-win x-dnd tool-bar dnd fontset image
regexp-opt fringe tabulated-list replace newcomment text-mode lisp-mode
prog-mode register page tab-bar menu-bar rfn-eshadow isearch easymenu
timer select scroll-bar mouse jit-lock font-lock syntax font-core
term/tty-colors frame minibuffer nadvice seq simple cl-generic
indonesian philippine cham georgian utf-8-lang misc-lang vietnamese
tibetan thai tai-viet lao korean japanese eucjp-ms cp51932 hebrew greek
romanian slovak czech european ethiopic indian cyrillic chinese
composite emoji-zwj charscript charprop case-table epa-hook
jka-cmpr-hook help abbrev obarray oclosure cl-preloaded button loaddefs
theme-loaddefs faces cus-face macroexp files window text-properties
overlay sha1 md5 base64 format env code-pages mule custom widget keymap
hashtable-print-readable backquote threads dbusbind inotify
dynamic-setting system-font-setting font-render-setting cairo x-toolkit
xinput2 x multi-tty make-network-process native-compile emacs)

Memory information:
((conses 16 638926 185590)
 (symbols 48 47388 5)
 (strings 32 264344 60226)
 (string-bytes 1 7396023)
 (vectors 16 62210)
 (vector-slots 8 1620783 87746)
 (floats 8 820 424)
 (intervals 56 1600 2497)
 (buffers 984 28))

bug#74277: 29.4; rust-ts-mode doesn't highlight some function calls

[Eli Zaretskii]

> Date: Fri, 08 Nov 2024 20:24:07 +0000
> From: acidbong--- via "Bug reports for GNU Emacs,
>  the Swiss army knife of text editors" <bug-gnu-emacs@gnu.org>
> 
> rust-ts-mode highlights function calls when they're used in an
> assignment or as statements, but not in string formatting:
> 
> 
> ```
> println!("Some line and {}", some_function(a, b))
>                            # ^^^^^^^^^^^^^ this is not getting highlighted
> ```
> 
> This is how it looks in Emacs (treesit-level is 4 btw):
> https://matrix.envs.net/_matrix/media/r0/download/envs.net/66652827fc7545e57d2eef263ee28ef6bc2ef1c01854976160883015680
> 
> This is how it looks in Neovim (with treesitter highlighting on):
> https://matrix.envs.net/_matrix/media/r0/download/envs.net/382a104ee3e6e4dacabebfa7f84f4b9b7231b1171854976151936565248
> 
> -----------------------------------

Thanks.

Randy/Yuan, any suggestions or fixes?

bug#74277: 29.4; rust-ts-mode doesn't highlight some function calls

[Randy Taylor]

On Saturday, November 9th, 2024 at 03:52, Eli Zaretskii <eliz@gnu.org> wrote:

> 
> 
> > Date: Fri, 08 Nov 2024 20:24:07 +0000
> 
> > From: acidbong--- via "Bug reports for GNU Emacs,
> > the Swiss army knife of text editors" bug-gnu-emacs@gnu.org
> > 
> > rust-ts-mode highlights function calls when they're used in an
> > assignment or as statements, but not in string formatting:
> > 
> > `println!("Some line and {}", some_function(a, b)) # ^^^^^^^^^^^^^ this is not getting highlighted`
> > 
> > This is how it looks in Emacs (treesit-level is 4 btw):
> > https://matrix.envs.net/_matrix/media/r0/download/envs.net/66652827fc7545e57d2eef263ee28ef6bc2ef1c01854976160883015680
> > 
> > This is how it looks in Neovim (with treesitter highlighting on):
> > https://matrix.envs.net/_matrix/media/r0/download/envs.net/382a104ee3e6e4dacabebfa7f84f4b9b7231b1171854976151936565248
> > 
> > -----------------------------------
> 
> 
> Thanks.
> 
> Randy/Yuan, any suggestions or fixes?

This was discussed briefly awhile back, but it's basically going to
require special handling because macro invocations don't give us the
usual dead simple way of doing it.

Maybe it's easy enough to see if the identifier has parentheses
immediately following it...

I probably won't have a chance to take a crack at it anytime soon, so
if anyone wants to beat me to it, feel free.

bug#74277: 29.4; rust-ts-mode doesn't highlight some function calls

[Trevor Arjeski]

Randy Taylor <dev@rjt.dev> writes:

Hi all, randomly started looking into this...

> This was discussed briefly awhile back, but it's basically going to
> require special handling because macro invocations don't give us the
> usual dead simple way of doing it.

Is this due to lack of support for injections.scm in Emacs' treesitter
implementation?
http://tree-sitter.github.io/tree-sitter/syntax-highlighting#language-injection

Can't seem to find any traces of that in the current source.

From inspecting this line of code in Neovim, it appears that the
injection on macro_invocation is what forces parsing the macro params as
another source_file.

bug#74277: 29.4; rust-ts-mode doesn't highlight some function calls

[Acid Bong via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>Is this due to lack of support for injections.scm in Emacs' treesitter
>implementation?

This starts making sense for me: I guess that's why nix-ts-mode can't process its Bash injections.

Looks like not using injections is the real issue here.

bug#74277: 29.4; rust-ts-mode doesn't highlight some function calls

[Trevor Arjeski]

[-- Attachment #1: Type: text/plain, Size: 154 bytes --]

Trevor Arjeski <tmarjeski@gmail.com> writes:

I decided to hack around with this a little bit and found that adding
the following lines partially works:


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: rudamentary fix --]
[-- Type: text/x-patch, Size: 799 bytes --]

diff --git a/lisp/progmodes/rust-ts-mode.el b/lisp/progmodes/rust-ts-mode.el
index e52ea3b125a..508a53d9e32 100644
--- a/lisp/progmodes/rust-ts-mode.el
+++ b/lisp/progmodes/rust-ts-mode.el
@@ -517,6 +517,19 @@ rust-ts-mode
     (setq-local syntax-propertize-function
                 #'rust-ts-mode--syntax-propertize)
 
+    ;; Injected languages
+    ;; TODO: add json?
+    (setq-local treesit-range-settings
+                (treesit-range-rules
+                 :embed 'rust
+                 :host 'rust
+                 :local t
+                 '((macro_invocation
+                    (token_tree) @capture))))
+
+    ;; TODO: set this to a useful function
+    (setq-local treesit-language-at-point-function (lambda (_pos) 'rust))
+
     ;; Comments.
     (c-ts-common-comment-setup)
 

[-- Attachment #3: Type: text/plain, Size: 424 bytes --]


I'm sure there is an issue with solving it this way, I just need someone
else more experienced to confirm. It is finicky where it turns the
highlighting on and off when you make some code changes, for example
adding and removing the semi-colon after the macro invocation.

Here is some example code to test it against:

   fn foo() -> i32 {
       4
   }
   
   fn main() {
       println!("{:?}", foo());
   }
   

Thanks

bug#74277: 29.4; rust-ts-mode doesn't highlight some function calls

[Randy Taylor]

On Wednesday, November 20th, 2024 at 13:22, Trevor Arjeski <tmarjeski@gmail.com> wrote:
> 
> Trevor Arjeski tmarjeski@gmail.com writes:
> 
> 
> I decided to hack around with this a little bit and found that adding
> the following lines partially works:
> 
> 
> I'm sure there is an issue with solving it this way, I just need someone
> else more experienced to confirm. It is finicky where it turns the
> highlighting on and off when you make some code changes, for example
> adding and removing the semi-colon after the macro invocation.

Yuan would be the best to answer that.

Personally, I envisioned a custom highlight helper function like
rust-ts-mode--fontify-pattern or rust-ts-mode--fontify-scope which
seems like the simplest solution that should cover this use case, but
I'm not actually aware of all the sorts of craziness one could get up
to in a macro invocation so that might not be enough and maybe injections
are the way to go.

> 
> Here is some example code to test it against:
> 
> fn foo() -> i32 {
> 
> 4
> }
> 
> fn main() {
> println!("{:?}", foo());
> }
> 
> 
> Thanks

bug#74277: 29.4; rust-ts-mode doesn't highlight some function calls

[Yuan Fu]

> On Nov 21, 2024, at 7:30 PM, Randy Taylor <dev@rjt.dev> wrote:
> 
> On Wednesday, November 20th, 2024 at 13:22, Trevor Arjeski <tmarjeski@gmail.com> wrote:
>> 
>> Trevor Arjeski tmarjeski@gmail.com writes:
>> 
>> 
>> I decided to hack around with this a little bit and found that adding
>> the following lines partially works:
>> 
>> 
>> I'm sure there is an issue with solving it this way, I just need someone
>> else more experienced to confirm. It is finicky where it turns the
>> highlighting on and off when you make some code changes, for example
>> adding and removing the semi-colon after the macro invocation.
> 
> Yuan would be the best to answer that.

Using injection (what Emacs calls local parser, same thing) here is fine, I think. As Randy said, tree-sitter doesn’t have a good answer for macros. If nvim uses this workaround (create an injection for the macro and parses it like normal rust code), then it should be fine for us to follow suit.

> 
> Personally, I envisioned a custom highlight helper function like
> rust-ts-mode--fontify-pattern or rust-ts-mode--fontify-scope which
> seems like the simplest solution that should cover this use case, but
> I'm not actually aware of all the sorts of craziness one could get up
> to in a macro invocation so that might not be enough and maybe injections
> are the way to go.

Rust macros are not as crazy as C/C++, but definitely goes beyond normal rust code, for example, the select! macro:

#[tokio::main]
async fn main() {
    tokio::select! {
        _ = do_stuff_async() => {
            println!("do_stuff_async() completed first")
        }
        _ = more_async_work() => {
            println!("more_async_work() completed first")
        }
    };
}

Does what you envisioned work for something like this? To be fair, I don’t think the injection solution works well for this either.

Yuan

bug#74277: 29.4; rust-ts-mode doesn't highlight some function calls

[Trevor Arjeski]

Yuan Fu <casouri@gmail.com> writes:

>> On Nov 21, 2024, at 7:30 PM, Randy Taylor <dev@rjt.dev> wrote:
>> 
>> On Wednesday, November 20th, 2024 at 13:22, Trevor Arjeski <tmarjeski@gmail.com> wrote:
>>> 
>>> Trevor Arjeski tmarjeski@gmail.com writes:
>>> 
>>> 
>>> I decided to hack around with this a little bit and found that adding
>>> the following lines partially works:
>>> 
>>> 
>>> I'm sure there is an issue with solving it this way, I just need someone
>>> else more experienced to confirm. It is finicky where it turns the
>>> highlighting on and off when you make some code changes, for example
>>> adding and removing the semi-colon after the macro invocation.
>> 
>> Yuan would be the best to answer that.
>
> Using injection (what Emacs calls local parser, same thing) here is fine, I
> think. As Randy said, tree-sitter doesn’t have a good answer for macros. If nvim
> uses this workaround (create an injection for the macro and parses it like
> normal rust code), then it should be fine for us to follow suit.
>

Any ideas why my crude implementation of the injection "flickers" on and
off after certain changes?

Thanks,
Trevor

bug#74277: 29.4; rust-ts-mode doesn't highlight some function calls

[Yuan Fu]

> On Nov 23, 2024, at 11:17 PM, Trevor Arjeski <tmarjeski@gmail.com> wrote:
> 
> Yuan Fu <casouri@gmail.com> writes:
> 
>>> On Nov 21, 2024, at 7:30 PM, Randy Taylor <dev@rjt.dev> wrote:
>>> 
>>> On Wednesday, November 20th, 2024 at 13:22, Trevor Arjeski <tmarjeski@gmail.com> wrote:
>>>> 
>>>> Trevor Arjeski tmarjeski@gmail.com writes:
>>>> 
>>>> 
>>>> I decided to hack around with this a little bit and found that adding
>>>> the following lines partially works:
>>>> 
>>>> 
>>>> I'm sure there is an issue with solving it this way, I just need someone
>>>> else more experienced to confirm. It is finicky where it turns the
>>>> highlighting on and off when you make some code changes, for example
>>>> adding and removing the semi-colon after the macro invocation.
>>> 
>>> Yuan would be the best to answer that.
>> 
>> Using injection (what Emacs calls local parser, same thing) here is fine, I
>> think. As Randy said, tree-sitter doesn’t have a good answer for macros. If nvim
>> uses this workaround (create an injection for the macro and parses it like
>> normal rust code), then it should be fine for us to follow suit.
>> 
> 
> Any ideas why my crude implementation of the injection "flickers" on and
> off after certain changes?
> 
> Thanks,
> Trevor

So, after spending a night on this, it appears to be a tree-sitter bug :( I created an issue here: https://github.com/tree-sitter/tree-sitter/issues/3962

Yuan

bug#74277: 29.4; rust-ts-mode doesn't highlight some function calls

[Randy Taylor]

On Sunday, November 24th, 2024 at 00:18, Yuan Fu <casouri@gmail.com> wrote:
> 
> 
> > On Nov 21, 2024, at 7:30 PM, Randy Taylor dev@rjt.dev wrote:
> > 
> > On Wednesday, November 20th, 2024 at 13:22, Trevor Arjeski tmarjeski@gmail.com wrote:
> > 
> > > Trevor Arjeski tmarjeski@gmail.com writes:
> > > 
> > > I decided to hack around with this a little bit and found that adding
> > > the following lines partially works:
> > > 
> > > I'm sure there is an issue with solving it this way, I just need someone
> > > else more experienced to confirm. It is finicky where it turns the
> > > highlighting on and off when you make some code changes, for example
> > > adding and removing the semi-colon after the macro invocation.
> > 
> > Yuan would be the best to answer that.
> 
> 
> Using injection (what Emacs calls local parser, same thing) here is fine, I think. As Randy said, tree-sitter doesn’t have a good answer for macros. If nvim uses this workaround (create an injection for the macro and parses it like normal rust code), then it should be fine for us to follow suit.
> 
> > Personally, I envisioned a custom highlight helper function like
> > rust-ts-mode--fontify-pattern or rust-ts-mode--fontify-scope which
> > seems like the simplest solution that should cover this use case, but
> > I'm not actually aware of all the sorts of craziness one could get up
> > to in a macro invocation so that might not be enough and maybe injections
> > are the way to go.
> 
> 
> Rust macros are not as crazy as C/C++, but definitely goes beyond normal rust code, for example, the select! macro:
> 
> #[tokio::main]
> async fn main() {
> tokio::select! {
> _ = do_stuff_async() => {
> 
> println!("do_stuff_async() completed first")
> }
> _ = more_async_work() => {
> 
> println!("more_async_work() completed first")
> }
> };
> }
> 
> Does what you envisioned work for something like this? To be fair, I don’t think the injection solution works well for this either.
> 
> Yuan

Sorry for the super late reply!

The custom highlighter function(s) would definitely not be the best way for macros like that. Back when this was discussed ages ago, I remember that injection had some problems or questions surrounding it (don't remember the details), so depending on whether those can be ironed out (whatever those issues were if they even still apply), we may only be able to tackle simpler highlights with those custom highlighter functions until injection becomes more feasible.

But injection seems to be the way to go to cover all types of macros appropriately.