bug#65470: 29.1.50; js-ts-mode: regex pattern can cause incorrect parenthesis matching

bug#65470: 29.1.50; js-ts-mode: regex pattern can cause incorrect parenthesis matching

[Augustin Chéneau]

With the mode js-ts-mode, matching tokens (such as '()', '[]') can be
incorrectly paired by `show-paren-mode`. This is trivially reproducible
with this simple example:

(/foobar)/)

The first parenthesis is matched with the second one, which is inside a
regular expression pattern (between slashes), and the last one is not
paired.

The behavior should be the same as for string, the content of the regex
pattern should have no influence on the structure of the code. The first
parenthesis should match with the third one. Here, the first parenthesis
is matched with the last one:

("foobar)")

js-mode behaves correctly in both cases.



In GNU Emacs 29.1.50 (build 2, x86_64-pc-linux-gnu, GTK+ Version
  3.24.38, cairo version 1.17.8) of 2023-07-30 built on inspiron-5567
Repository revision: 0002d4f31666719c5e972ca65efffc70750db613
Repository branch: makepkg
Windowing system distributor 'The X.Org Foundation', version 11.0.12302000
System Description: Arch Linux

Configured using:
  'configure --prefix=/opt --sysconfdir=/etc --libexecdir=/opt/lib
  --localstatedir=/var --mandir=/opt/share/man --with-gameuser=:games
  --with-modules --without-libotf --without-m17n-flt --without-gconf
  --with-native-compilation=yes --with-native-compilation=aot
  --with-xinput2 --with-x-toolkit=gtk3 --without-xaw3d --with-sound=no
  --with-tree-sitter --without-gpm --without-compress-install
  '--program-transform-name=s/\([ec]tags\)/\1.emacs/'
  'CFLAGS=-march=x86-64 -mtune=generic -O2 -pipe -fno-plt -fexceptions
  -Wp,-D_FORTIFY_SOURCE=2 -Wformat -Werror=format-security
  -fstack-clash-protection -fcf-protection'
  LDFLAGS=-Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now
  PKG_CONFIG_PATH=/home/arch-aug/_opam/lib/pkgconfig
  'CXXFLAGS=-march=x86-64 -mtune=generic -O2 -pipe -fno-plt -fexceptions
  -Wp,-D_FORTIFY_SOURCE=2 -Wformat -Werror=format-security
  -fstack-clash-protection -fcf-protection -Wp,-D_GLIBCXX_ASSERTIONS''

Configured features:
ACL CAIRO DBUS FREETYPE GIF GLIB GMP GNUTLS GSETTINGS HARFBUZZ JPEG JSON
LCMS2 LIBSYSTEMD LIBXML2 MODULES NATIVE_COMP NOTIFY INOTIFY PDUMPER PNG
RSVG SECCOMP SQLITE3 THREADS TIFF TOOLKIT_SCROLL_BARS TREE_SITTER WEBP
X11 XDBE XIM XINPUT2 XPM GTK3 ZLIB

Important settings:
   value of $LANG: fr_FR.UTF-8
   locale-coding-system: utf-8-unix

Major mode: Info

Minor modes in effect:
   windmove-mode: t
   yas-global-mode: t
   yas-minor-mode: t
   delete-selection-mode: t
   global-subword-mode: t
   subword-mode: t
   save-place-mode: t
   smartparens-global-mode: t
   treemacs-filewatch-mode: t
   treemacs-follow-mode: t
   treemacs-git-mode: t
   treemacs-fringe-indicator-mode: t
   global-hl-line-mode: t
   editorconfig-mode: t
   drag-stuff-global-mode: t
   drag-stuff-mode: t
   doom-modeline-mode: t
   global-git-commit-mode: t
   magit-auto-revert-mode: t
   global-auto-revert-mode: t
   shell-dirtrack-mode: t
   server-mode: t
   adaptive-wrap-prefix-mode: t
   global-undo-tree-mode: t
   undo-tree-mode: t
   global-anzu-mode: t
   anzu-mode: t
   which-key-mode: t
   projectile-mode: t
   global-corfu-mode: t
   corfu-mode: t
   TeX-PDF-mode: t
   TeX-source-correlate-mode: t
   vertico-mode: t
   marginalia-mode: t
   override-global-mode: t
   straight-use-package-mode: t
   straight-package-neutering-mode: t
   savehist-mode: t
   global-display-line-numbers-mode: t
   display-line-numbers-mode: t
   tooltip-mode: t
   global-eldoc-mode: t
   show-paren-mode: t
   electric-indent-mode: t
   mouse-wheel-mode: t
   tool-bar-mode: t
   menu-bar-mode: t
   file-name-shadow-mode: t
   isearch-fold-quotes-mode: t
   global-font-lock-mode: t
   font-lock-mode: t
   buffer-read-only: t
   size-indication-mode: t
   column-number-mode: t
   line-number-mode: t
   global-visual-line-mode: t
   visual-line-mode: t
   indent-tabs-mode: t
   transient-mark-mode: t
   auto-composition-mode: t
   auto-encryption-mode: t
   auto-compression-mode: t

Load-path shadows:
/home/arch-aug/.config/emacs-neo/straight/build/transient/transient 
hides /opt/share/emacs/29.1.50/lisp/transient
/home/arch-aug/.config/emacs-neo/straight/build/jsonrpc/jsonrpc hides 
/opt/share/emacs/29.1.50/lisp/jsonrpc
/home/arch-aug/.config/emacs-neo/straight/build/external-completion/external-completion 
hides /opt/share/emacs/29.1.50/lisp/external-completion
/home/arch-aug/.config/emacs-neo/straight/build/use-package/use-package 
hides /opt/share/emacs/29.1.50/lisp/use-package/use-package
/home/arch-aug/.config/emacs-neo/straight/build/use-package/use-package-lint 
hides /opt/share/emacs/29.1.50/lisp/use-package/use-package-lint
/home/arch-aug/.config/emacs-neo/straight/build/bind-key/bind-key hides 
/opt/share/emacs/29.1.50/lisp/use-package/bind-key
/home/arch-aug/.config/emacs-neo/straight/build/use-package/use-package-diminish 
hides /opt/share/emacs/29.1.50/lisp/use-package/use-package-diminish
/home/arch-aug/.config/emacs-neo/straight/build/use-package/use-package-jump 
hides /opt/share/emacs/29.1.50/lisp/use-package/use-package-jump
/home/arch-aug/.config/emacs-neo/straight/build/use-package/use-package-core 
hides /opt/share/emacs/29.1.50/lisp/use-package/use-package-core
/home/arch-aug/.config/emacs-neo/straight/build/use-package/use-package-bind-key 
hides /opt/share/emacs/29.1.50/lisp/use-package/use-package-bind-key
/home/arch-aug/.config/emacs-neo/straight/build/use-package/use-package-delight 
hides /opt/share/emacs/29.1.50/lisp/use-package/use-package-delight
/home/arch-aug/.config/emacs-neo/straight/build/use-package/use-package-ensure 
hides /opt/share/emacs/29.1.50/lisp/use-package/use-package-ensure
/home/arch-aug/.config/emacs-neo/straight/build/xref/xref hides 
/opt/share/emacs/29.1.50/lisp/progmodes/xref
/home/arch-aug/.config/emacs-neo/straight/build/project/project hides 
/opt/share/emacs/29.1.50/lisp/progmodes/project
/home/arch-aug/.config/emacs-neo/straight/build/flymake/flymake hides 
/opt/share/emacs/29.1.50/lisp/progmodes/flymake
/home/arch-aug/.config/emacs-neo/straight/build/eglot/eglot hides 
/opt/share/emacs/29.1.50/lisp/progmodes/eglot
/home/arch-aug/.config/emacs-neo/straight/build/soap-client/soap-inspect 
hides /opt/share/emacs/29.1.50/lisp/net/soap-inspect
/home/arch-aug/.config/emacs-neo/straight/build/soap-client/soap-client 
hides /opt/share/emacs/29.1.50/lisp/net/soap-client
/home/arch-aug/.config/emacs-neo/straight/build/eldoc/eldoc hides 
/opt/share/emacs/29.1.50/lisp/emacs-lisp/eldoc
/home/arch-aug/.config/emacs-neo/straight/build/let-alist/let-alist 
hides /opt/share/emacs/29.1.50/lisp/emacs-lisp/let-alist

Features:
(shadow sort mail-extr emacsbug cus-start windmove magit-extras
magit-delta xterm-color dabbrev textutils cape combobulate
combobulate-yaml combobulate-css combobulate-js-ts combobulate-python
combobulate-html combobulate-ui combobulate-display combobulate-contrib
combobulate-manipulation python combobulate-navigation combobulate-misc
combobulate-interface combobulate-rules combobulate-settings tempo
misearch multi-isearch noutline outline shortdoc vertico-directory js
c-ts-common mule-util checkdoc doom-snippets doom-snippets-lib yasnippet
rainbow-delimiters ws-butler delsel cap-words superword subword
saveplace flex-mode jison-mode bison-mode cc-mode cc-fonts cc-guess
cc-menus cc-cmds cc-styles cc-align cc-engine cc-vars cc-defs
bison-mode-autoloads dired-x cmake-mode rst cmake-mode-autoloads utop
utop-minor-mode tuareg tuareg-compat tuareg-opam caml-help find-file
utop-autoloads flycheck-ocaml flycheck find-func
flycheck-ocaml-autoloads merlin-eldoc-autoloads merlin-xref merlin-cap
merlin caml-types merlin-autoloads dune dune-autoloads tuareg-autoloads
caml-autoloads pkgbuild-mode tramp tramp-loaddefs trampver
tramp-integration files-x tramp-compat parse-time iso8601 sh-script smie
executable pkgbuild-mode-autoloads geiser-guile info-look geiser-debug
geiser-repl geiser-image geiser-capf geiser-doc geiser-menu
geiser-autodoc geiser-edit etags fileloop generator geiser-completion
geiser-eval geiser-connection tq geiser-syntax scheme geiser-impl
help-fns radix-tree geiser-log geiser-popup view geiser-custom
geiser-base geiser-guile-autoloads geiser geiser-autoloads lua-mode
lua-mode-autoloads debbugs soap-client url-http url-auth url-gw nsm
rng-xsd rng-dt rng-util xsd-regexp debbugs-autoloads
soap-client-autoloads expand-region-autoloads jinx-autoloads smartparens
loadhist smartparens-autoloads builder vc-git vc-dispatcher dir-var
builder-autoloads treemacs treemacs-header-line treemacs-compatibility
treemacs-mode treemacs-bookmarks treemacs-tags xref treemacs-interface
treemacs-persistence treemacs-filewatch-mode treemacs-follow-mode
treemacs-rendering treemacs-annotations treemacs-async
treemacs-workspaces treemacs-dom treemacs-visuals
treemacs-fringe-indicator treemacs-scope pulse color treemacs-faces
treemacs-icons treemacs-themes treemacs-core-utils pfuture inline
hl-line ht treemacs-logging treemacs-customization treemacs-macros
treemacs-autoloads cfrs-autoloads posframe-autoloads ht-autoloads
hydra-autoloads lv-autoloads pfuture-autoloads ace-window-autoloads
avy-autoloads hare-mode-autoloads eglot-autoloads
external-completion-autoloads jsonrpc-autoloads flymake-autoloads
project-autoloads xref-autoloads eldoc-autoloads flycheck-autoloads
let-alist-autoloads pkg-info-autoloads epl-autoloads vertico-search
general general-autoloads combobulate-autoloads treesit editorconfig
editorconfig-core editorconfig-core-handle editorconfig-fnmatch pcase
editorconfig-autoloads ws-butler-autoloads bel-coloration
rainbow-delimiters-autoloads drag-stuff drag-stuff-autoloads
doom-modeline doom-modeline-segments doom-modeline-env
doom-modeline-core shrink-path f f-shortdoc s doom-modeline-autoloads
shrink-path-autoloads f-autoloads s-autoloads all-the-icons
all-the-icons-faces data-material data-weathericons data-octicons
data-fileicons data-faicons data-alltheicons all-the-icons-autoloads
doom-themes-ext-org doom-themes-ext-treemacs doom-themes-ext-neotree
doom-themes-ext-visual-bell face-remap doom-dracula-theme doom-themes
doom-themes-base doom-themes-autoloads magit-delta-autoloads
xterm-color-autoloads magit-bookmark magit-submodule magit-obsolete
magit-blame magit-stash magit-reflog magit-bisect magit-push magit-pull
magit-fetch magit-clone magit-remote magit-commit magit-sequence
magit-notes magit-worktree magit-tag magit-merge magit-branch
magit-reset magit-files magit-refs magit-status magit package browse-url
url url-proxy url-privacy url-expand url-methods url-history url-cookie
generate-lisp-file url-domsuf url-util url-handlers url-parse
auth-source json map url-vars magit-repos magit-apply magit-wip
magit-log which-func imenu magit-diff smerge-mode diff-mode git-commit
log-edit message sendmail mailcap yank-media puny dired dired-loaddefs
rfc822 mml mml-sec password-cache epa derived epg rfc6068 epg-config
gnus-util time-date mm-decode mm-bodies mm-encode mail-parse rfc2231
rfc2047 rfc2045 mm-util ietf-drums mail-prsvr mailabbrev mail-utils
gmm-utils mailheader pcvs-util add-log magit-core magit-autorevert
autorevert filenotify magit-margin magit-transient magit-process
with-editor shell pcomplete server magit-mode transient magit-git
magit-base magit-section format-spec eieio eieio-core dash
magit-autoloads magit-section-autoloads git-commit-autoloads
with-editor-autoloads transient-autoloads dash-autoloads consult-vertico
consult recentf tree-widget wid-edit bookmark pp consult-autoloads
adaptive-wrap adaptive-wrap-autoloads undo-tree diff queue
undo-tree-autoloads queue-autoloads anzu anzu-autoloads which-key
which-key-autoloads comp comp-cstr rx projectile lisp-mnt grep ibuf-ext
ibuffer ibuffer-loaddefs projectile-autoloads doom-snippets-autoloads
yasnippet-autoloads finder-inf cape-autoloads corfu corfu-autoloads
auctex-latexmk advice latex latex-flymake flymake-proc flymake project
byte-opt compile text-property-search comint ansi-osc ansi-color ring
warnings icons thingatpt tex-ispell tex-style auctex-latexmk-autoloads
tex dbus xml crm texmathp auctex-autoloads tex-site vertico
vertico-autoloads marginalia compat edmacro kmacro marginalia-autoloads
compat-autoloads use-package-bind-key bind-key easy-mmode orderless
orderless-autoloads use-package-ensure use-package-autoloads info
bind-key-autoloads straight-autoloads cl-seq cl-extra help-mode straight
subr-x cl-macs gv use-package-core cl-loaddefs cl-lib bytecomp
byte-compile savehist display-line-numbers elec-pair cus-load rmc
iso-transl tooltip cconv eldoc paren electric uniquify ediff-hook
vc-hooks lisp-float-type elisp-mode mwheel term/x-win x-win
term/common-win x-dnd tool-bar dnd fontset image regexp-opt fringe
tabulated-list replace newcomment text-mode lisp-mode prog-mode register
page tab-bar menu-bar rfn-eshadow isearch easymenu timer select
scroll-bar mouse jit-lock font-lock syntax font-core term/tty-colors
frame minibuffer nadvice seq simple cl-generic indonesian philippine
cham georgian utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao
korean japanese eucjp-ms cp51932 hebrew greek romanian slovak czech
european ethiopic indian cyrillic chinese composite emoji-zwj charscript
charprop case-table epa-hook jka-cmpr-hook help abbrev obarray oclosure
cl-preloaded button loaddefs theme-loaddefs faces cus-face macroexp
files window text-properties overlay sha1 md5 base64 format env
code-pages mule custom widget keymap hashtable-print-readable backquote
threads dbusbind inotify lcms2 dynamic-setting system-font-setting
font-render-setting cairo move-toolbar gtk x-toolkit xinput2 x multi-tty
make-network-process native-compile emacs)

Memory information:
((conses 16 855111 185665)
  (symbols 48 62979 2)
  (strings 32 199519 18907)
  (string-bytes 1 6782525)
  (vectors 16 90652)
  (vector-slots 8 2300969 313710)
  (floats 8 1625 1345)
  (intervals 56 20355 3201)
  (buffers 984 25))

bug#65470: 29.1.50; js-ts-mode: regex pattern can cause incorrect parenthesis matching

[Dmitry Gutov]

On 23/08/2023 12:05, Augustin Chéneau (BTuin) wrote:
> 
> With the mode js-ts-mode, matching tokens (such as '()', '[]') can be
> incorrectly paired by `show-paren-mode`. This is trivially reproducible
> with this simple example:
> 
> (/foobar)/)
> 
> The first parenthesis is matched with the second one, which is inside a
> regular expression pattern (between slashes), and the last one is not
> paired.
> 
> The behavior should be the same as for string, the content of the regex
> pattern should have no influence on the structure of the code. The first
> parenthesis should match with the third one. Here, the first parenthesis
> is matched with the last one:
> 
> ("foobar)")
> 
> js-mode behaves correctly in both cases.

Sounds like js-ts-mode also needs a syntax-propertize-function, similar 
to c-ts-mode, ruby-ts-mode and rust-ts-mode.

Others (typescript-ts-mode?) probably need it as well, at least modes 
for those languages that have dedicated regexp or heredoc syntax.

bug#65470: 29.1.50; js-ts-mode: regex pattern can cause incorrect parenthesis matching

[Eli Zaretskii]

> Date: Wed, 23 Aug 2023 16:23:13 +0300
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> On 23/08/2023 12:05, Augustin Chéneau (BTuin) wrote:
> > 
> > With the mode js-ts-mode, matching tokens (such as '()', '[]') can be
> > incorrectly paired by `show-paren-mode`. This is trivially reproducible
> > with this simple example:
> > 
> > (/foobar)/)
> > 
> > The first parenthesis is matched with the second one, which is inside a
> > regular expression pattern (between slashes), and the last one is not
> > paired.
> > 
> > The behavior should be the same as for string, the content of the regex
> > pattern should have no influence on the structure of the code. The first
> > parenthesis should match with the third one. Here, the first parenthesis
> > is matched with the last one:
> > 
> > ("foobar)")
> > 
> > js-mode behaves correctly in both cases.
> 
> Sounds like js-ts-mode also needs a syntax-propertize-function, similar 
> to c-ts-mode, ruby-ts-mode and rust-ts-mode.
> 
> Others (typescript-ts-mode?) probably need it as well, at least modes 
> for those languages that have dedicated regexp or heredoc syntax.

Would someone please add syntax-propertize-function in modes that need
it?  I think this should be done on the emacs-29 branch.

TIA

bug#65470: 29.1.50; js-ts-mode: regex pattern can cause incorrect parenthesis matching

[Augustin Chéneau]

[-- Attachment #1: Type: text/plain, Size: 197 bytes --]

I have created a patch heavily inspired from ruby-ts-mode.

It fixes the trivial examples and it works as expected in a more complex 
file.
I'm not sure if the commit message is formatted properly.

[-- Attachment #2: 0001-Apply-syntax-properties-to-regex-delimiters-in-js-ts.patch --]
[-- Type: text/x-patch, Size: 1955 bytes --]

From d7bb7a77a57026f385c4137a25199eb2a2c15768 Mon Sep 17 00:00:00 2001
From: Augustin Chéneau <btuin@mailo.com>
Date: Thu, 24 Aug 2023 21:00:47 +0200
Subject: [PATCH] Apply syntax properties to regex delimiters in js-ts-mode

Add a property to regex slashes to avoid incorrect parenthesis
pairing by using `syntax-propertize-function` (Bug#65470)
* lisp/progmodes/js.el: add a new function to apply syntax properties
---
 lisp/progmodes/js.el | 20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)

diff --git a/lisp/progmodes/js.el b/lisp/progmodes/js.el
index 9d2990e7bc9..efdddfeadf4 100644
--- a/lisp/progmodes/js.el
+++ b/lisp/progmodes/js.el
@@ -3711,6 +3711,23 @@ js--treesit-valid-imenu-entry
     ("lexical_declaration" (treesit-node-top-level node))
     (_ t)))
 
+(defvar js-ts--regex-query
+  (when (treesit-available-p)
+    (treesit-query-compile 'javascript
+                           '(((regex "/" @regex_slash))))))
+
+(defun js-ts-mode--syntax-propertize (beg end)
+  "Apply syntax properties to special characters between BEG and END.
+
+Apply syntax properties to regex delimiters (slashes)."
+  (let ((captures (treesit-query-capture 'javascript js-ts--regex-query beg end)))
+    (pcase-dolist (`(,name . ,node) captures)
+      (pcase-exhaustive name
+	('regex_slash
+	 (put-text-property (treesit-node-start node) (1+ (treesit-node-start node))
+			    'syntax-table
+			    (string-to-syntax "\"/")))))))
+
 ;;; Main Function
 
 ;;;###autoload
@@ -3850,6 +3867,9 @@ js-ts-mode
                   ( assignment constant escape-sequence jsx number
                     pattern string-interpolation)
                   ( bracket delimiter function operator property)))
+    ;; Syntax table
+    (setq-local syntax-propertize-function
+                #'js-ts-mode--syntax-propertize)
     ;; Imenu
     (setq-local treesit-simple-imenu-settings
                 `(("Function" "\\`function_declaration\\'" nil nil)
-- 
2.42.0

bug#65470: 29.1.50; js-ts-mode: regex pattern can cause incorrect parenthesis matching

[Theodor Thornhill via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Eli Zaretskii <eliz@gnu.org> writes:

>> Date: Wed, 23 Aug 2023 16:23:13 +0300
>> From: Dmitry Gutov <dmitry@gutov.dev>
>> 
>> On 23/08/2023 12:05, Augustin Chéneau (BTuin) wrote:
>> > 
>> > With the mode js-ts-mode, matching tokens (such as '()', '[]') can be
>> > incorrectly paired by `show-paren-mode`. This is trivially reproducible
>> > with this simple example:
>> > 
>> > (/foobar)/)
>> > 
>> > The first parenthesis is matched with the second one, which is inside a
>> > regular expression pattern (between slashes), and the last one is not
>> > paired.
>> > 
>> > The behavior should be the same as for string, the content of the regex
>> > pattern should have no influence on the structure of the code. The first
>> > parenthesis should match with the third one. Here, the first parenthesis
>> > is matched with the last one:
>> > 
>> > ("foobar)")
>> > 
>> > js-mode behaves correctly in both cases.
>> 
>> Sounds like js-ts-mode also needs a syntax-propertize-function, similar 
>> to c-ts-mode, ruby-ts-mode and rust-ts-mode.
>> 
>> Others (typescript-ts-mode?) probably need it as well, at least modes 
>> for those languages that have dedicated regexp or heredoc syntax.
>
> Would someone please add syntax-propertize-function in modes that need
> it?  I think this should be done on the emacs-29 branch.
>
> TIA

I'll add it to my list, but if someone will grab it that's fine, as I'm
a little short on time the next couple of weeks :-(

Theo

bug#65470: 29.1.50; js-ts-mode: regex pattern can cause incorrect parenthesis matching

[Dmitry Gutov]

[-- Attachment #1: Type: text/plain, Size: 1675 bytes --]

On 24/08/2023 22:47, Theodor Thornhill wrote:
> Eli Zaretskii<eliz@gnu.org>  writes:
> 
>>> Date: Wed, 23 Aug 2023 16:23:13 +0300
>>> From: Dmitry Gutov<dmitry@gutov.dev>
>>>
>>> On 23/08/2023 12:05, Augustin Chéneau (BTuin) wrote:
>>>> With the mode js-ts-mode, matching tokens (such as '()', '[]') can be
>>>> incorrectly paired by `show-paren-mode`. This is trivially reproducible
>>>> with this simple example:
>>>>
>>>> (/foobar)/)
>>>>
>>>> The first parenthesis is matched with the second one, which is inside a
>>>> regular expression pattern (between slashes), and the last one is not
>>>> paired.
>>>>
>>>> The behavior should be the same as for string, the content of the regex
>>>> pattern should have no influence on the structure of the code. The first
>>>> parenthesis should match with the third one. Here, the first parenthesis
>>>> is matched with the last one:
>>>>
>>>> ("foobar)")
>>>>
>>>> js-mode behaves correctly in both cases.
>>> Sounds like js-ts-mode also needs a syntax-propertize-function, similar
>>> to c-ts-mode, ruby-ts-mode and rust-ts-mode.
>>>
>>> Others (typescript-ts-mode?) probably need it as well, at least modes
>>> for those languages that have dedicated regexp or heredoc syntax.
>> Would someone please add syntax-propertize-function in modes that need
>> it?  I think this should be done on the emacs-29 branch.
>>
>> TIA
> I'll add it to my list, but if someone will grab it that's fine, as I'm
> a little short on time the next couple of weeks 🙁

This one seems to work for js-ts-mode.

typescript is a bit more fiddly (two separate modes, one with jsx and 
one without), but should be able to follow the similar pattern.

[-- Attachment #2: js-ts--syntax-propertize.diff --]
[-- Type: text/x-patch, Size: 1745 bytes --]

diff --git a/lisp/progmodes/js.el b/lisp/progmodes/js.el
index 9d2990e7bc9..23fdc57533d 100644
--- a/lisp/progmodes/js.el
+++ b/lisp/progmodes/js.el
@@ -3829,6 +3829,7 @@ js-ts-mode
                 (append "{}():;,<>/" electric-indent-chars)) ;FIXME: js2-mode adds "[]*".
     (setq-local electric-layout-rules
 	        '((?\; . after) (?\{ . after) (?\} . before)))
+    (setq-local syntax-propertize-function #'js-ts--syntax-propertize)
 
     ;; Tree-sitter setup.
     (treesit-parser-create 'javascript)
@@ -3864,6 +3865,26 @@ js-ts-mode
     (add-to-list 'auto-mode-alist
                  '("\\(\\.js[mx]\\|\\.har\\)\\'" . js-ts-mode))))
 
+(defvar js-ts--s-p-query
+  (when (treesit-available-p)
+    (treesit-query-compile 'javascript
+                           '(((regex pattern: (regex_pattern) @regexp))
+                             ((variable_declarator value: (jsx_element) @jsx))
+                             ((assignment_expression right: (jsx_element) @jsx))))))
+
+(defun js-ts--syntax-propertize (beg end)
+  (let ((captures (treesit-query-capture 'javascript js-ts--s-p-query beg end)))
+    (pcase-dolist (`(,name . ,node) captures)
+      (let ((syntax (pcase-exhaustive name
+                      ('regexp
+                       (string-to-syntax "\"/"))
+                      ('jsx
+                       (string-to-syntax "|")))))
+        (put-text-property (treesit-node-start node) (1+ (treesit-node-start node))
+                           'syntax-table syntax)
+        (put-text-property (1- (treesit-node-end node)) (treesit-node-end node)
+                           'syntax-table syntax)))))
+
 ;;;###autoload
 (define-derived-mode js-json-mode js-mode "JSON"
   (setq-local js-enabled-frameworks nil)

bug#65470: 29.1.50; js-ts-mode: regex pattern can cause incorrect parenthesis matching

[Eli Zaretskii]

> Date: Fri, 25 Aug 2023 03:18:54 +0300
> Cc: btuin@mailo.com, 65470@debbugs.gnu.org
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> >>> Others (typescript-ts-mode?) probably need it as well, at least modes
> >>> for those languages that have dedicated regexp or heredoc syntax.
> >> Would someone please add syntax-propertize-function in modes that need
> >> it?  I think this should be done on the emacs-29 branch.
> >>
> >> TIA
> > I'll add it to my list, but if someone will grab it that's fine, as I'm
> > a little short on time the next couple of weeks 🙁
> 
> This one seems to work for js-ts-mode.
> 
> typescript is a bit more fiddly (two separate modes, one with jsx and 
> one without), but should be able to follow the similar pattern.

Thanks.  If no issues are found with this within the next couple of
days, please install on the emacs-29 branch.

bug#65470: 29.1.50; js-ts-mode: regex pattern can cause incorrect parenthesis matching

[Augustin Chéneau]

[-- Attachment #1: Type: text/plain, Size: 1992 bytes --]

Le 25/08/2023 à 02:18, Dmitry Gutov a écrit :
> On 24/08/2023 22:47, Theodor Thornhill wrote:
>> Eli Zaretskii<eliz@gnu.org>  writes:
>>
>>>> Date: Wed, 23 Aug 2023 16:23:13 +0300
>>>> From: Dmitry Gutov<dmitry@gutov.dev>
>>>>
>>>> On 23/08/2023 12:05, Augustin Chéneau (BTuin) wrote:
>>>>> With the mode js-ts-mode, matching tokens (such as '()', '[]') can be
>>>>> incorrectly paired by `show-paren-mode`. This is trivially 
>>>>> reproducible
>>>>> with this simple example:
>>>>>
>>>>> (/foobar)/)
>>>>>
>>>>> The first parenthesis is matched with the second one, which is 
>>>>> inside a
>>>>> regular expression pattern (between slashes), and the last one is not
>>>>> paired.
>>>>>
>>>>> The behavior should be the same as for string, the content of the 
>>>>> regex
>>>>> pattern should have no influence on the structure of the code. The 
>>>>> first
>>>>> parenthesis should match with the third one. Here, the first 
>>>>> parenthesis
>>>>> is matched with the last one:
>>>>>
>>>>> ("foobar)")
>>>>>
>>>>> js-mode behaves correctly in both cases.
>>>> Sounds like js-ts-mode also needs a syntax-propertize-function, similar
>>>> to c-ts-mode, ruby-ts-mode and rust-ts-mode.
>>>>
>>>> Others (typescript-ts-mode?) probably need it as well, at least modes
>>>> for those languages that have dedicated regexp or heredoc syntax.
>>> Would someone please add syntax-propertize-function in modes that need
>>> it?  I think this should be done on the emacs-29 branch.
>>>
>>> TIA
>> I'll add it to my list, but if someone will grab it that's fine, as I'm
>> a little short on time the next couple of weeks 🙁
> 
> This one seems to work for js-ts-mode.
> 
> typescript is a bit more fiddly (two separate modes, one with jsx and 
> one without), but should be able to follow the similar pattern.

Thanks for the patch, but I still have some issues. I believe that the
START and END arguments of `put-text-property` are off by one. The
attached modified patch seems to work better.

[-- Attachment #2: js-ts--syntax-propertize-v2.diff --]
[-- Type: text/x-patch, Size: 1745 bytes --]

diff --git a/lisp/progmodes/js.el b/lisp/progmodes/js.el
index 9d2990e7bc9..a74f67d4466 100644
--- a/lisp/progmodes/js.el
+++ b/lisp/progmodes/js.el
@@ -3829,6 +3829,7 @@ js-ts-mode
                 (append "{}():;,<>/" electric-indent-chars)) ;FIXME: js2-mode adds "[]*".
     (setq-local electric-layout-rules
 	        '((?\; . after) (?\{ . after) (?\} . before)))
+    (setq-local syntax-propertize-function #'js-ts--syntax-propertize)
 
     ;; Tree-sitter setup.
     (treesit-parser-create 'javascript)
@@ -3864,6 +3865,26 @@ js-ts-mode
     (add-to-list 'auto-mode-alist
                  '("\\(\\.js[mx]\\|\\.har\\)\\'" . js-ts-mode))))
 
+(defvar js-ts--s-p-query
+  (when (treesit-available-p)
+    (treesit-query-compile 'javascript
+                           '(((regex pattern: (regex_pattern) @regexp))
+                             ((variable_declarator value: (jsx_element) @jsx))
+                             ((assignment_expression right: (jsx_element) @jsx))))))
+
+(defun js-ts--syntax-propertize (beg end)
+  (let ((captures (treesit-query-capture 'javascript js-ts--s-p-query beg end)))
+    (pcase-dolist (`(,name . ,node) captures)
+      (let ((syntax (pcase-exhaustive name
+                      ('regexp
+                       (string-to-syntax "\"/"))
+                      ('jsx
+                       (string-to-syntax "|")))))
+        (put-text-property (1- (treesit-node-start node)) (treesit-node-start node)
+                           'syntax-table syntax)
+        (put-text-property (treesit-node-end node) (1+ (treesit-node-end node))
+                           'syntax-table syntax)))))
+
 ;;;###autoload
 (define-derived-mode js-json-mode js-mode "JSON"
   (setq-local js-enabled-frameworks nil)

bug#65470: 29.1.50; js-ts-mode: regex pattern can cause incorrect parenthesis matching

[Dmitry Gutov]

[-- Attachment #1: Type: text/plain, Size: 908 bytes --]

Hi Augustin!

On 25/08/2023 10:17, Augustin Chéneau (BTuin) wrote:
> Thanks for the patch, but I still have some issues. I believe that the
> START and END arguments of `put-text-property` are off by one. The
> attached modified patch seems to work better.

Thanks for trying it out. Sorry about missing your first patch: 
apparently while it was the first time-wise, it was help up in moderation.

Anyway, you make a good point about the case with the regexp: since I 
matched the pattern inside, the bounds of the node are off by one 
compared to the case with the jsx node. Your patch moves those bounds, 
but then they'll be wrong for the latter case.

Here's an update which tries to handle both, but I guess if we add more 
node types later we'll just have to stop trying to handle them the same 
way (and do it closer to ruby-ts--syntax-propertize).

Also added the new return_statement case for jsx.

[-- Attachment #2: js-ts--syntax-propertize-v2.diff --]
[-- Type: text/x-patch, Size: 1840 bytes --]

diff --git a/lisp/progmodes/js.el b/lisp/progmodes/js.el
index 9d2990e7bc9..9b6da4ec2fc 100644
--- a/lisp/progmodes/js.el
+++ b/lisp/progmodes/js.el
@@ -3829,6 +3829,7 @@ js-ts-mode
                 (append "{}():;,<>/" electric-indent-chars)) ;FIXME: js2-mode adds "[]*".
     (setq-local electric-layout-rules
 	        '((?\; . after) (?\{ . after) (?\} . before)))
+    (setq-local syntax-propertize-function #'js-ts--syntax-propertize)
 
     ;; Tree-sitter setup.
     (treesit-parser-create 'javascript)
@@ -3864,6 +3865,29 @@ js-ts-mode
     (add-to-list 'auto-mode-alist
                  '("\\(\\.js[mx]\\|\\.har\\)\\'" . js-ts-mode))))
 
+(defvar js-ts--s-p-query
+  (when (treesit-available-p)
+    (treesit-query-compile 'javascript
+                           '(((regex pattern: (regex_pattern) @regexp))
+                             ((variable_declarator value: (jsx_element) @jsx))
+                             ((assignment_expression right: (jsx_element) @jsx))
+                             ((return_statement (jsx_element) @jsx))))))
+
+(defun js-ts--syntax-propertize (beg end)
+  (let ((captures (treesit-query-capture 'javascript js-ts--s-p-query beg end)))
+    (pcase-dolist (`(,name . ,node) captures)
+      (let* ((ns (treesit-node-start node))
+             (ne (treesit-node-end node))
+             (syntax (pcase-exhaustive name
+                       ('regexp
+                        (cl-decf ns)
+                        (cl-incf ne)
+                        (string-to-syntax "\"/"))
+                       ('jsx
+                        (string-to-syntax "|")))))
+        (put-text-property ns (1+ ns) 'syntax-table syntax)
+        (put-text-property (1- ne) ne 'syntax-table syntax)))))
+
 ;;;###autoload
 (define-derived-mode js-json-mode js-mode "JSON"
   (setq-local js-enabled-frameworks nil)

bug#65470: 29.1.50; js-ts-mode: regex pattern can cause incorrect parenthesis matching

[Jostein Kjønigsen]

> On 24 Aug 2023, at 21:47, Theodor Thornhill <theo@thornhill.no> wrote:
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
>> 
>> Would someone please add syntax-propertize-function in modes that need
>> it?  I think this should be done on the emacs-29 branch.
>> 
>> TIA
> 
> I'll add it to my list, but if someone will grab it that's fine, as I'm
> a little short on time the next couple of weeks :-(
> 
> Theo

I’ve just been forced over to a Mac at work, and I can’t (yet) for the life of me figure out how to build Emacs from source on this platform. It seems Emacs is not picking up libraries I’ve installed using homebrew?

If anyone can help me get past this roadblock, I might be able to get a patch going. (It’s a ARM-based Mac, if that matters.)

—
Jostein

bug#65470: 29.1.50; js-ts-mode: regex pattern can cause incorrect parenthesis matching

[Yuan Fu]

> On Aug 24, 2023, at 11:59 PM, Jostein Kjønigsen <jostein@secure.kjonigsen.net> wrote:
> 
> 
> 
>> On 24 Aug 2023, at 21:47, Theodor Thornhill <theo@thornhill.no> wrote:
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>>> 
>>> Would someone please add syntax-propertize-function in modes that need
>>> it?  I think this should be done on the emacs-29 branch.
>>> 
>>> TIA
>> 
>> I'll add it to my list, but if someone will grab it that's fine, as I'm
>> a little short on time the next couple of weeks :-(
>> 
>> Theo
> 
> I’ve just been forced over to a Mac at work, and I can’t (yet) for the life of me figure out how to build Emacs from source on this platform. It seems Emacs is not picking up libraries I’ve installed using homebrew?
> 
> If anyone can help me get past this roadblock, I might be able to get a patch going. (It’s a ARM-based Mac, if that matters.)

It’s been quite a while since I needed to setup this, so I don’t quite remember. But setting LD_LIBRARY_PATH (Homebrew’s path) and PKG_CONFIG_PATH (/usr/local/lib/pkgconfig) should help the build process and Emacs find the libraries.

Yuan

bug#65470: 29.1.50; js-ts-mode: regex pattern can cause incorrect parenthesis matching

[Jostein Kjønigsen]

> On 25 Aug 2023, at 20:28, Yuan Fu <casouri@gmail.com> wrote:
> 
> 
> 
>>> On Aug 24, 2023, at 11:59 PM, Jostein Kjønigsen <jostein@secure.kjonigsen.net> wrote:
>>> 
>>> 
>>> 
>>>> On 24 Aug 2023, at 21:47, Theodor Thornhill <theo@thornhill.no> wrote:
>>> 
>>> Eli Zaretskii <eliz@gnu.org> writes:
>>> 
>>>> 
>>>> Would someone please add syntax-propertize-function in modes that need
>>>> it?  I think this should be done on the emacs-29 branch.
>>>> 
>>>> TIA
>>> 
>>> I'll add it to my list, but if someone will grab it that's fine, as I'm
>>> a little short on time the next couple of weeks :-(
>>> 
>>> Theo
>> 
>> I’ve just been forced over to a Mac at work, and I can’t (yet) for the life of me figure out how to build Emacs from source on this platform. It seems Emacs is not picking up libraries I’ve installed using homebrew?
>> 
>> If anyone can help me get past this roadblock, I might be able to get a patch going. (It’s a ARM-based Mac, if that matters.)
> 
> It’s been quite a while since I needed to setup this, so I don’t quite remember. But setting LD_LIBRARY_PATH (Homebrew’s path) and PKG_CONFIG_PATH (/usr/local/lib/pkgconfig) should help the build process and Emacs find the libraries.
> 
> Yuan

I figured it out in the end. For anyone suffering a similar problem:

I had all packages I needed installed, but not pkg-config, leading configure not being able to locate any of them. 

Installing pkg-config too and running “make bootstrap” got me a working build. 

With that out of the way, I’ll see if I can come up with some patches for the typescript ts-modes sometime the next week. 

—
Jostein

bug#65470: 29.1.50; js-ts-mode: regex pattern can cause incorrect parenthesis matching

[Fu Yuan]

>> 
>>>> On Aug 24, 2023, at 11:59 PM, Jostein Kjønigsen <jostein@secure.kjonigsen.net> wrote:
>>>> 
>>>> 
>>>> 
>>>>> On 24 Aug 2023, at 21:47, Theodor Thornhill <theo@thornhill.no> wrote:
>>>> 
>>>> Eli Zaretskii <eliz@gnu.org> writes:
>>>> 
>>>>> 
>>>>> Would someone please add syntax-propertize-function in modes that need
>>>>> it?  I think this should be done on the emacs-29 branch.
>>>>> 
>>>>> TIA
>>>> 
>>>> I'll add it to my list, but if someone will grab it that's fine, as I'm
>>>> a little short on time the next couple of weeks :-(
>>>> 
>>>> Theo
>>> 
>>> I’ve just been forced over to a Mac at work, and I can’t (yet) for the life of me figure out how to build Emacs from source on this platform. It seems Emacs is not picking up libraries I’ve installed using homebrew?
>>> 
>>> If anyone can help me get past this roadblock, I might be able to get a patch going. (It’s a ARM-based Mac, if that matters.)
>> 
>> It’s been quite a while since I needed to setup this, so I don’t quite remember. But setting LD_LIBRARY_PATH (Homebrew’s path) and PKG_CONFIG_PATH (/usr/local/lib/pkgconfig) should help the build process and Emacs find the libraries.
>> 
>> Yuan
> 
> I figured it out in the end. For anyone suffering a similar problem:
> 
> I had all packages I needed installed, but not pkg-config, leading configure not being able to locate any of them. 
> 
> Installing pkg-config too and running “make bootstrap” got me a working build. 
> 
> With that out of the way, I’ll see if I can come up with some patches for the typescript ts-modes sometime the next week. 

Good to hear that! Exiles you are at it, could you also have a look at JSX? I think the latest tree-sitter grammar breaks some queries.

Yuan

bug#65470: 29.1.50; js-ts-mode: regex pattern can cause incorrect parenthesis matching

[Jostein Kjønigsen]

> On 26 Aug 2023, at 17:29, Fu Yuan <casouri@gmail.com> wrote:
> 
> 
>>> 
>>>>> On Aug 24, 2023, at 11:59 PM, Jostein Kjønigsen <jostein@secure.kjonigsen.net> wrote:
>>>>> 
>>>>> 
>>>>> 
>>>>>> On 24 Aug 2023, at 21:47, Theodor Thornhill <theo@thornhill.no> wrote:
>>>>> 
>>>>> Eli Zaretskii <eliz@gnu.org> writes:
>>>>> 
>>>>>> 
>>>>>> Would someone please add syntax-propertize-function in modes that need
>>>>>> it?  I think this should be done on the emacs-29 branch.
>>>>>> 
>>>>>> TIA
>>>>> 
>>>>> I'll add it to my list, but if someone will grab it that's fine, as I'm
>>>>> a little short on time the next couple of weeks :-(
>>>>> 
>>>>> Theo
>>>> 
>>>> I’ve just been forced over to a Mac at work, and I can’t (yet) for the life of me figure out how to build Emacs from source on this platform. It seems Emacs is not picking up libraries I’ve installed using homebrew?
>>>> 
>>>> If anyone can help me get past this roadblock, I might be able to get a patch going. (It’s a ARM-based Mac, if that matters.)
>>> 
>>> It’s been quite a while since I needed to setup this, so I don’t quite remember. But setting LD_LIBRARY_PATH (Homebrew’s path) and PKG_CONFIG_PATH (/usr/local/lib/pkgconfig) should help the build process and Emacs find the libraries.
>>> 
>>> Yuan
>> 
>> I figured it out in the end. For anyone suffering a similar problem:
>> 
>> I had all packages I needed installed, but not pkg-config, leading configure not being able to locate any of them. 
>> 
>> Installing pkg-config too and running “make bootstrap” got me a working build. 
>> 
>> With that out of the way, I’ll see if I can come up with some patches for the typescript ts-modes sometime the next week. 
> 
> Good to hear that! Exiles you are at it, could you also have a look at JSX? I think the latest tree-sitter grammar breaks some queries.
> 
> Yuan

I don’t really work with JSX and don’t think have a “idiomatic” code base to try it on where I would notice brokenness. 

I think this would be better handled by people more experienced with both JSX and our JSX code base than I am. 

—
Jostein

bug#65470: 29.1.50; js-ts-mode: regex pattern can cause incorrect parenthesis matching

[Dmitry Gutov]

On 27/08/2023 00:13, Jostein Kjønigsen wrote:
> I don’t really work with JSX and don’t think have a “idiomatic” code base to try it on where I would notice brokenness.
> 
> I think this would be better handled by people more experienced with both JSX and our JSX code base than I am.

Just checking that viewing some JSX code in .tsx files doesn't show 
errors in Messages might be good enough.

I tried that with javascript, didn't notice any problems so far.

bug#65470: 29.1.50; js-ts-mode: regex pattern can cause incorrect parenthesis matching

[Eli Zaretskii]

> Date: Sun, 27 Aug 2023 00:45:19 +0300
> Cc: Theodor Thornhill <theo@thornhill.no>, Eli Zaretskii <eliz@gnu.org>,
>  Jostein Kjønigsen <jostein@kjonigsen.net>, btuin@mailo.com,
>  65470@debbugs.gnu.org
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> On 27/08/2023 00:13, Jostein Kjønigsen wrote:
> > I don’t really work with JSX and don’t think have a “idiomatic” code base to try it on where I would notice brokenness.
> > 
> > I think this would be better handled by people more experienced with both JSX and our JSX code base than I am.
> 
> Just checking that viewing some JSX code in .tsx files doesn't show 
> errors in Messages might be good enough.
> 
> I tried that with javascript, didn't notice any problems so far.

I'm a bit confused by this discussion.  Should I install the patch on
the emacs-29 branch now, or are you still discussing whether it's
correct?

bug#65470: 29.1.50; js-ts-mode: regex pattern can cause incorrect parenthesis matching

[Dmitry Gutov]

On 31/08/2023 12:41, Eli Zaretskii wrote:
>> Date: Sun, 27 Aug 2023 00:45:19 +0300
>> Cc: Theodor Thornhill<theo@thornhill.no>, Eli Zaretskii<eliz@gnu.org>,
>>   Jostein Kjønigsen<jostein@kjonigsen.net>,btuin@mailo.com,
>>   65470@debbugs.gnu.org
>> From: Dmitry Gutov<dmitry@gutov.dev>
>>
>> On 27/08/2023 00:13, Jostein Kjønigsen wrote:
>>> I don’t really work with JSX and don’t think have a “idiomatic” code base to try it on where I would notice brokenness.
>>>
>>> I think this would be better handled by people more experienced with both JSX and our JSX code base than I am.
>> Just checking that viewing some JSX code in .tsx files doesn't show
>> errors in Messages might be good enough.
>>
>> I tried that with javascript, didn't notice any problems so far.
> I'm a bit confused by this discussion.  Should I install the patch on
> the emacs-29 branch now, or are you still discussing whether it's
> correct?

I think we were discussing the typescript modes here.

Personally I was waiting for some additional confirmation that my v2 for 
js-ts-mode is correct, but I guess we have a silent approval, so we can 
install it.

bug#65470: 29.1.50; js-ts-mode: regex pattern can cause incorrect parenthesis matching

[Eli Zaretskii]

> Date: Thu, 31 Aug 2023 14:15:22 +0300
> Cc: jostein@secure.kjonigsen.net, casouri@gmail.com, theo@thornhill.no,
>  jostein@kjonigsen.net, btuin@mailo.com, 65470@debbugs.gnu.org
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> On 31/08/2023 12:41, Eli Zaretskii wrote:
> > I'm a bit confused by this discussion.  Should I install the patch on
> > the emacs-29 branch now, or are you still discussing whether it's
> > correct?
> 
> I think we were discussing the typescript modes here.
> 
> Personally I was waiting for some additional confirmation that my v2 for 
> js-ts-mode is correct, but I guess we have a silent approval, so we can 
> install it.

Good to hear.  Then please install it, and thanks.

bug#65470: 29.1.50; js-ts-mode: regex pattern can cause incorrect parenthesis matching

[Dmitry Gutov]

On 31/08/2023 15:53, Eli Zaretskii wrote:
>> Date: Thu, 31 Aug 2023 14:15:22 +0300
>> Cc:jostein@secure.kjonigsen.net,casouri@gmail.com,theo@thornhill.no,
>>   jostein@kjonigsen.net,btuin@mailo.com,65470@debbugs.gnu.org
>> From: Dmitry Gutov<dmitry@gutov.dev>
>>
>> On 31/08/2023 12:41, Eli Zaretskii wrote:
>>> I'm a bit confused by this discussion.  Should I install the patch on
>>> the emacs-29 branch now, or are you still discussing whether it's
>>> correct?
>> I think we were discussing the typescript modes here.
>>
>> Personally I was waiting for some additional confirmation that my v2 for
>> js-ts-mode is correct, but I guess we have a silent approval, so we can
>> install it.
> Good to hear.  Then please install it, and thanks.

All right, installed one for js-ts-mode.

bug#65470: 29.1.50; js-ts-mode: regex pattern can cause incorrect parenthesis matching

[Jostein Kjønigsen]

[-- Attachment #1: Type: text/plain, Size: 1655 bytes --]



> On 1 Sep 2023, at 03:42, Dmitry Gutov <dmitry@gutov.dev> wrote:
> 
> On 31/08/2023 15:53, Eli Zaretskii wrote:
>>> Date: Thu, 31 Aug 2023 14:15:22 +0300
>>> Cc:jostein@secure.kjonigsen.net,casouri@gmail.com,theo@thornhill.no,
>>>  jostein@kjonigsen.net,btuin@mailo.com,65470@debbugs.gnu.org
>>> From: Dmitry Gutov<dmitry@gutov.dev>
>>> 
>>> On 31/08/2023 12:41, Eli Zaretskii wrote:
>>>> I'm a bit confused by this discussion.  Should I install the patch on
>>>> the emacs-29 branch now, or are you still discussing whether it's
>>>> correct?
>>> I think we were discussing the typescript modes here.
>>> 
>>> Personally I was waiting for some additional confirmation that my v2 for
>>> js-ts-mode is correct, but I guess we have a silent approval, so we can
>>> install it.
>> Good to hear.  Then please install it, and thanks.
> 
> All right, installed one for js-ts-mode.

Sorry for the somewhat slow response on this. Life has been busy for me as well :)

Based on the infamous “copy, paste & adapt” methodology I’ve come up with a patch for typescript-ts-mode and tsx-ts-mode which seems to work for the simple test case provided above.

However the patch for js-ts-mode has lots of … interesting stuff about jsx I have not included, for no other reason than not understanding what type of use-cases they are meant to support.

If someone can provide me some examples for the JSX use-cases, I can try to make room for TSX-variants of the same code.

The patch so far is attached. It’s intentionally -not- optimized to leave room for TSX cases which may arise, so hopefully no need to nitpick this yet.


[-- Attachment #2: 0001-typescript-ts-mode-Fix-syntax-properties-for-regexp-.patch --]
[-- Type: application/octet-stream, Size: 2178 bytes --]

From db0393034ca8f251dd1d5cedf928f0c0264d61ef Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Jostein=20Kj=C3=B8nigsen?= <jostein@kjonigsen.net>
Date: Tue, 5 Sep 2023 21:29:27 +0200
Subject: [PATCH] typescript-ts-mode: Fix syntax properties for regexp
 expressions.

---
 lisp/progmodes/typescript-ts-mode.el | 20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)

diff --git a/lisp/progmodes/typescript-ts-mode.el b/lisp/progmodes/typescript-ts-mode.el
index 441cdc1f7aa..016c62623c7 100644
--- a/lisp/progmodes/typescript-ts-mode.el
+++ b/lisp/progmodes/typescript-ts-mode.el
@@ -473,6 +473,7 @@ typescript-ts-mode
                   (keyword string escape-sequence)
                   (constant expression identifier number pattern property)
                   (function bracket delimiter)))
+    (setq-local syntax-propertize-function #'ts-ts--syntax-propertize)
 
     (treesit-major-mode-setup)))
 
@@ -529,9 +530,28 @@ tsx-ts-mode
                   (keyword string escape-sequence)
                   (constant expression identifier jsx number pattern property)
                   (function bracket delimiter)))
+    (setq-local syntax-propertize-function #'ts-ts--syntax-propertize)
 
     (treesit-major-mode-setup)))
 
+(defvar ts-ts--s-p-query
+  (when (treesit-available-p)
+    (treesit-query-compile 'typescript
+                           '(((regex pattern: (regex_pattern) @regexp))))))
+
+(defun ts-ts--syntax-propertize (beg end)
+  (let ((captures (treesit-query-capture 'typescript ts-ts--s-p-query beg end)))
+    (pcase-dolist (`(,name . ,node) captures)
+      (let* ((ns (treesit-node-start node))
+             (ne (treesit-node-end node))
+             (syntax (pcase-exhaustive name
+                       ('regexp
+                        (cl-decf ns)
+                        (cl-incf ne)
+                        (string-to-syntax "\"/")))))
+        (put-text-property ns (1+ ns) 'syntax-table syntax)
+        (put-text-property (1- ne) ne 'syntax-table syntax)))))
+
 (if (treesit-ready-p 'tsx)
     (add-to-list 'auto-mode-alist '("\\.tsx\\'" . tsx-ts-mode)))
 
-- 
2.39.2 (Apple Git-143)


[-- Attachment #3: Type: text/plain, Size: 16 bytes --]



—
Jostein

bug#65470: 29.1.50; js-ts-mode: regex pattern can cause incorrect parenthesis matching

[Eli Zaretskii]

> From: Jostein Kjønigsen <jostein@secure.kjonigsen.net>
> Date: Tue, 5 Sep 2023 21:31:25 +0200
> Cc: Eli Zaretskii <eliz@gnu.org>,
>  Yuan Fu <casouri@gmail.com>,
>  Theodor Thornhill <theo@thornhill.no>,
>  Jostein Kjønigsen <jostein@kjonigsen.net>,
>  btuin@mailo.com,
>  65470@debbugs.gnu.org
> 
> Sorry for the somewhat slow response on this. Life has been busy for me as well :)
> 
> Based on the infamous “copy, paste & adapt” methodology I’ve come up with a patch for typescript-ts-mode and tsx-ts-mode which seems to work for the simple test case provided above.
> 
> However the patch for js-ts-mode has lots of … interesting stuff about jsx I have not included, for no other reason than not understanding what type of use-cases they are meant to support.
> 
> If someone can provide me some examples for the JSX use-cases, I can try to make room for TSX-variants of the same code.
> 
> The patch so far is attached. It’s intentionally -not- optimized to leave room for TSX cases which may arise, so hopefully no need to nitpick this yet.

Thanks, Jostein.

Any objections to my installing this on the emacs-29 branch, anyone?

bug#65470: 29.1.50; js-ts-mode: regex pattern can cause incorrect parenthesis matching

[Stefan Kangas]

Eli Zaretskii <eliz@gnu.org> writes:

> Any objections to my installing this on the emacs-29 branch, anyone?

No objections here.

bug#65470: 29.1.50; js-ts-mode: regex pattern can cause incorrect parenthesis matching

[Dmitry Gutov]

On 05/09/2023 22:31, Jostein Kjønigsen wrote:
 > The patch so far is attached. It’s intentionally -not- optimized to 
leave room for TSX cases which may arise, so hopefully no need to 
nitpick this yet.

I was going to nitpick it, but then read the rest of your message ;-(

> However the patch for js-ts-mode has lots of … interesting stuff about jsx I have not included, for no other reason than not understanding what type of use-cases they are meant to support.
> 
> If someone can provide me some examples for the JSX use-cases, I can try to make room for TSX-variants of the same code.

The idea was to "enclose" every TSX in "generic string" syntax so that 
whatever unpaired characters are inside (such as (, ", ...), won't 
affect syntax-ppss status on the outside. I'm not sure how critical that 
is, but I guess some users might encounter such situations.

Examples like:

   ReactDOM.render(
     <div className="">
       <h1>Hello, Welcome to React and TypeScript ;-(</h1>
     </div>,
     // type closing paren here and see that it's matched to opener above
     document.getElementById("root")
   );

bug#65470: 29.1.50; js-ts-mode: regex pattern can cause incorrect parenthesis matching

[Jostein Kjønigsen]

[-- Attachment #1: Type: text/plain, Size: 1992 bytes --]



> On 7 Sep 2023, at 14:09, Dmitry Gutov <dmitry@gutov.dev> wrote:
> 
> On 05/09/2023 22:31, Jostein Kjønigsen wrote:
> > The patch so far is attached. It’s intentionally -not- optimized to leave room for TSX cases which may arise, so hopefully no need to nitpick this yet.
> 
> I was going to nitpick it, but then read the rest of your message ;-(
> 
>> However the patch for js-ts-mode has lots of … interesting stuff about jsx I have not included, for no other reason than not understanding what type of use-cases they are meant to support.
>> If someone can provide me some examples for the JSX use-cases, I can try to make room for TSX-variants of the same code.
> 
> The idea was to "enclose" every TSX in "generic string" syntax so that whatever unpaired characters are inside (such as (, ", ...), won't affect syntax-ppss status on the outside. I'm not sure how critical that is, but I guess some users might encounter such situations.
> 
> Examples like:
> 
>  ReactDOM.render(
>    <div className="">
>      <h1>Hello, Welcome to React and TypeScript ;-(</h1>
>    </div>,
>    // type closing paren here and see that it's matched to opener above
>    document.getElementById("root")
>  );
> 

Ok. That makes sense, and explains everything. I’ve tried implementing the same thing for tsx-ts-mode too in the attached patch. (You may start nitpicking now).

I first tried using a more naive and wide approach only trying to capture full (jsx_element) blocks without any further constraints.
What I found was that when having nested elements (which one tend to have all the time), the syntax-propertizing would cancel itself for every second nesting.

So to fix that, I’ve tried to anchor the query to top-level constructs which typically does not nest. Looking at the query for js.el, I can only assume that you were trying to too solve the same problem, Dmitry?

But without further ado… Here’s the patch. Do your worst, gentlemen :D


[-- Attachment #2: 0001-typescript-ts-mode-Fix-syntax-properties-for-regexp-.patch --]
[-- Type: application/octet-stream, Size: 2910 bytes --]

From 2f08456a4aa6a093cbda4359af31edcd5864a44c Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Jostein=20Kj=C3=B8nigsen?= <jostein@kjonigsen.net>
Date: Tue, 5 Sep 2023 21:29:27 +0200
Subject: [PATCH] typescript-ts-mode: Fix syntax properties for regexp
 expressions.

- Also handle JSX-elements in TSX-files as generic strings.
---
 lisp/progmodes/typescript-ts-mode.el | 36 ++++++++++++++++++++++++++++
 1 file changed, 36 insertions(+)

diff --git a/lisp/progmodes/typescript-ts-mode.el b/lisp/progmodes/typescript-ts-mode.el
index 441cdc1f7aa..dd61baf5e3c 100644
--- a/lisp/progmodes/typescript-ts-mode.el
+++ b/lisp/progmodes/typescript-ts-mode.el
@@ -473,6 +473,7 @@ typescript-ts-mode
                   (keyword string escape-sequence)
                   (constant expression identifier number pattern property)
                   (function bracket delimiter)))
+    (setq-local syntax-propertize-function #'ts-ts--syntax-propertize)
 
     (treesit-major-mode-setup)))
 
@@ -529,9 +530,44 @@ tsx-ts-mode
                   (keyword string escape-sequence)
                   (constant expression identifier jsx number pattern property)
                   (function bracket delimiter)))
+    (setq-local syntax-propertize-function #'tsx-ts--syntax-propertize)
 
     (treesit-major-mode-setup)))
 
+(defvar ts-ts--s-p-query
+  (when (treesit-available-p)
+    (treesit-query-compile 'typescript
+                           '(((regex pattern: (regex_pattern) @regexp))))))
+
+(defvar tsx-ts--s-p-query
+  (when (treesit-available-p)
+    (treesit-query-compile 'tsx
+                           '(((regex pattern: (regex_pattern) @regexp))
+                             ((arguments (jsx_element) @jsx))
+                             ((parenthesized_expression (jsx_element) @jsx))))))
+
+(defun ts-ts--syntax-propertize (beg end)
+  (let ((captures (treesit-query-capture 'typescript ts-ts--s-p-query beg end)))
+    (ts-ts--syntax-propertize-captures captures)))
+
+(defun tsx-ts--syntax-propertize (beg end)
+  (let ((captures (treesit-query-capture 'tsx tsx-ts--s-p-query beg end)))
+    (ts-ts--syntax-propertize-captures captures)))
+
+(defun ts-ts--syntax-propertize-captures (captures)
+  (pcase-dolist (`(,name . ,node) captures)
+    (let* ((ns (treesit-node-start node))
+           (ne (treesit-node-end node))
+           (syntax (pcase-exhaustive name
+                     ('regexp
+                      (cl-decf ns)
+                      (cl-incf ne)
+                      (string-to-syntax "\"/"))
+                     ('jsx
+                      (string-to-syntax "|")))))
+      (put-text-property ns (1+ ns) 'syntax-table syntax)
+      (put-text-property (1- ne) ne 'syntax-table syntax))))
+
 (if (treesit-ready-p 'tsx)
     (add-to-list 'auto-mode-alist '("\\.tsx\\'" . tsx-ts-mode)))
 
-- 
2.39.2 (Apple Git-143)


[-- Attachment #3: Type: text/plain, Size: 16 bytes --]



—
Jostein

bug#65470: 29.1.50; js-ts-mode: regex pattern can cause incorrect parenthesis matching

[Dmitry Gutov]

Hi Jostein!

The patch looks good overall (though I haven't tested it).

On 11/09/2023 22:37, Jostein Kjønigsen wrote:

 > So to fix that, I’ve tried to anchor the query to top-level 
constructs which typically does not nest. Looking at the query for 
js.el, I can only assume that you were trying to too solve the same 
problem, Dmitry?

That's right.

> +                             ((arguments (jsx_element) @jsx))
> +                             ((parenthesized_expression (jsx_element) @jsx))))))

These two contexts actually seem missing in my patch for js-ts-mode. :-)

OTOH, it seems like the ones that _are_ included there, are missing 
here. Don't they?

   ((variable_declarator value: (jsx_element) @jsx))

corresponds do

   let v = <div>...</div>

   ((assignment_expression right: (jsx_element) @jsx))

corresponds do

   v = <p>...<p>

(with variable declared previously), and

   ((return_statement (jsx_element) @jsx))

corresponds to

   function f() {
     return <div>...<div>;
   }

bug#65470: 29.1.50; js-ts-mode: regex pattern can cause incorrect parenthesis matching

[Jostein Kjønigsen]

[-- Attachment #1: Type: text/plain, Size: 1256 bytes --]

On 12 Sep 2023, at 00:23, Dmitry Gutov <dmitry@gutov.dev> wrote:
> 
> Hi Jostein!
> 
> The patch looks good overall (though I haven't tested it).
> 
> On 11/09/2023 22:37, Jostein Kjønigsen wrote:
> 
> > So to fix that, I’ve tried to anchor the query to top-level constructs which typically does not nest. Looking at the query for js.el, I can only assume that you were trying to too solve the same problem, Dmitry?
> 
> That's right.
> 
>> +                             ((arguments (jsx_element) @jsx))
>> +                             ((parenthesized_expression (jsx_element) @jsx))))))
> 
> These two contexts actually seem missing in my patch for js-ts-mode. :-)
> 
> OTOH, it seems like the ones that _are_ included there, are missing here. Don't they?
> 
>  ((variable_declarator value: (jsx_element) @jsx))
> 
> corresponds do
> 
>  let v = <div>...</div>
> 
>  ((assignment_expression right: (jsx_element) @jsx))
> 
> corresponds do
> 
>  v = <p>...<p>
> 
> (with variable declared previously), and
> 
>  ((return_statement (jsx_element) @jsx))
> 
> corresponds to
> 
>  function f() {
>    return <div>...<div>;
>  }

Good feedback. I’ve tested and included those as well. Attached is a new patch.


[-- Attachment #2: 0001-typescript-ts-mode-Fix-syntax-properties-for-regexp-.patch --]
[-- Type: application/octet-stream, Size: 3177 bytes --]

From 16eb97efd5442e65eebd83bce2469aaee6d11697 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Jostein=20Kj=C3=B8nigsen?= <jostein@kjonigsen.net>
Date: Tue, 5 Sep 2023 21:29:27 +0200
Subject: [PATCH] typescript-ts-mode: Fix syntax properties for regexp
 expressions.

- Also handle JSX-elements in TSX-files as generic strings.
---
 lisp/progmodes/typescript-ts-mode.el | 39 ++++++++++++++++++++++++++++
 1 file changed, 39 insertions(+)

diff --git a/lisp/progmodes/typescript-ts-mode.el b/lisp/progmodes/typescript-ts-mode.el
index 441cdc1f7aa..4bf4cf1f85f 100644
--- a/lisp/progmodes/typescript-ts-mode.el
+++ b/lisp/progmodes/typescript-ts-mode.el
@@ -473,6 +473,7 @@ typescript-ts-mode
                   (keyword string escape-sequence)
                   (constant expression identifier number pattern property)
                   (function bracket delimiter)))
+    (setq-local syntax-propertize-function #'ts-ts--syntax-propertize)
 
     (treesit-major-mode-setup)))
 
@@ -529,9 +530,47 @@ tsx-ts-mode
                   (keyword string escape-sequence)
                   (constant expression identifier jsx number pattern property)
                   (function bracket delimiter)))
+    (setq-local syntax-propertize-function #'tsx-ts--syntax-propertize)
 
     (treesit-major-mode-setup)))
 
+(defvar ts-ts--s-p-query
+  (when (treesit-available-p)
+    (treesit-query-compile 'typescript
+                           '(((regex pattern: (regex_pattern) @regexp))))))
+
+(defvar tsx-ts--s-p-query
+      (when (treesit-available-p)
+        (treesit-query-compile 'tsx
+                               '(((regex pattern: (regex_pattern) @regexp))
+                                 ((variable_declarator value: (jsx_element) @jsx))
+                                 ((assignment_expression right: (jsx_element) @jsx))
+                                 ((arguments (jsx_element) @jsx))
+                                 ((parenthesized_expression (jsx_element) @jsx))
+                                 ((return_statement (jsx_element) @jsx))))))
+
+(defun ts-ts--syntax-propertize (beg end)
+  (let ((captures (treesit-query-capture 'typescript ts-ts--s-p-query beg end)))
+    (ts-ts--syntax-propertize-captures captures)))
+
+(defun tsx-ts--syntax-propertize (beg end)
+  (let ((captures (treesit-query-capture 'tsx tsx-ts--s-p-query beg end)))
+    (ts-ts--syntax-propertize-captures captures)))
+
+(defun ts-ts--syntax-propertize-captures (captures)
+  (pcase-dolist (`(,name . ,node) captures)
+    (let* ((ns (treesit-node-start node))
+           (ne (treesit-node-end node))
+           (syntax (pcase-exhaustive name
+                     ('regexp
+                      (cl-decf ns)
+                      (cl-incf ne)
+                      (string-to-syntax "\"/"))
+                     ('jsx
+                      (string-to-syntax "|")))))
+      (put-text-property ns (1+ ns) 'syntax-table syntax)
+      (put-text-property (1- ne) ne 'syntax-table syntax))))
+
 (if (treesit-ready-p 'tsx)
     (add-to-list 'auto-mode-alist '("\\.tsx\\'" . tsx-ts-mode)))
 
-- 
2.39.2 (Apple Git-143)


[-- Attachment #3: Type: text/plain, Size: 20 bytes --]



—
Jostein

bug#65470: 29.1.50; js-ts-mode: regex pattern can cause incorrect parenthesis matching

[Dmitry Gutov]

On 12/09/2023 09:29, Jostein Kjønigsen wrote:
> Good feedback. I’ve tested and included those as well. Attached is a new patch.

Thanks! Installed. It would've been better with a commit message too 
(next time).

Speaking of the TSX difficulties, though, which Yuan hinted at recently: 
I think the current fix (tsx-ts-mode--font-lock-compatibility-bb1f97b) 
is not working. To be clear, that's unrelated to the current patch.

Basically, the query for ((member_expression) @capture) succeeds, but 
then the more complex one fails with

Structure error at: 22
(jsx_opening_element [(member_expression (identifier)) (identifier)] 
@typescript-ts-jsx-tag-face) (jsx_closing_element [(member_expression 
(identifier)) (identifier)] @typescript-ts-jsx-tag-face) 
(jsx_self_closing_element [(member_expression (identifier)) 
(identifier)] @typescript-ts-jsx-tag-face)

This patch seems to fix that and bring JSX syntax highlighting back:

diff --git a/lisp/progmodes/typescript-ts-mode.el 
b/lisp/progmodes/typescript-ts-mode.el
index 57382c9cb31..7108ff49b28 100644
--- a/lisp/progmodes/typescript-ts-mode.el
+++ b/lisp/progmodes/typescript-ts-mode.el
@@ -163,7 +163,7 @@ tsx-ts-mode--font-lock-compatibility-bb1f97b
    ;; but then raises an error if the wrong node type is used. So it is
    ;; important to check with the new node type (member_expression)
    (condition-case nil
-      (progn (treesit-query-capture language '((member_expression) 
@capture))
+      (progn (treesit-query-capture language '(jsx_opening_element 
(member_expression) @capture))
  	     '((jsx_opening_element
  		[(member_expression (identifier)) (identifier)]
  		@typescript-ts-jsx-tag-face)

bug#65470: 29.1.50; js-ts-mode: regex pattern can cause incorrect parenthesis matching

[Jostein Kjønigsen]

> On 13 Sep 2023, at 01:14, Dmitry Gutov <dmitry@gutov.dev> wrote:
> 
> On 12/09/2023 09:29, Jostein Kjønigsen wrote:
>> Good feedback. I’ve tested and included those as well. Attached is a new patch.
> 
> Thanks! Installed. It would've been better with a commit message too (next time).
> 
> Speaking of the TSX difficulties, though, which Yuan hinted at recently: I think the current fix (tsx-ts-mode--font-lock-compatibility-bb1f97b) is not working. To be clear, that's unrelated to the current patch.
> 
> Basically, the query for ((member_expression) @capture) succeeds, but then the more complex one fails with
> 
> Structure error at: 22
> (jsx_opening_element [(member_expression (identifier)) (identifier)] @typescript-ts-jsx-tag-face) (jsx_closing_element [(member_expression (identifier)) (identifier)] @typescript-ts-jsx-tag-face) (jsx_self_closing_element [(member_expression (identifier)) (identifier)] @typescript-ts-jsx-tag-face)
> 
> This patch seems to fix that and bring JSX syntax highlighting back:
> 
> diff --git a/lisp/progmodes/typescript-ts-mode.el b/lisp/progmodes/typescript-ts-mode.el
> index 57382c9cb31..7108ff49b28 100644
> --- a/lisp/progmodes/typescript-ts-mode.el
> +++ b/lisp/progmodes/typescript-ts-mode.el
> @@ -163,7 +163,7 @@ tsx-ts-mode--font-lock-compatibility-bb1f97b
>   ;; but then raises an error if the wrong node type is used. So it is
>   ;; important to check with the new node type (member_expression)
>   (condition-case nil
> -      (progn (treesit-query-capture language '((member_expression) @capture))
> +      (progn (treesit-query-capture language '(jsx_opening_element (member_expression) @capture))
> 	     '((jsx_opening_element
> 		[(member_expression (identifier)) (identifier)]
> 		@typescript-ts-jsx-tag-face)
> 

I really don’t know this part of the code very well. If you’ve identified an issue/bug for a specific TSX use-case, and found a fix for it, I have no objections to it being applied.

—
Jostein

bug#65470: 29.1.50; js-ts-mode: regex pattern can cause incorrect parenthesis matching

[Dmitry Gutov]

On 15/09/2023 15:11, Jostein Kjønigsen wrote:
>> This patch seems to fix that and bring JSX syntax highlighting back:
>>
>> diff --git a/lisp/progmodes/typescript-ts-mode.el b/lisp/progmodes/typescript-ts-mode.el
>> index 57382c9cb31..7108ff49b28 100644
>> --- a/lisp/progmodes/typescript-ts-mode.el
>> +++ b/lisp/progmodes/typescript-ts-mode.el
>> @@ -163,7 +163,7 @@ tsx-ts-mode--font-lock-compatibility-bb1f97b
>>    ;; but then raises an error if the wrong node type is used. So it is
>>    ;; important to check with the new node type (member_expression)
>>    (condition-case nil
>> -      (progn (treesit-query-capture language '((member_expression) @capture))
>> +      (progn (treesit-query-capture language '(jsx_opening_element (member_expression) @capture))
>> 	     '((jsx_opening_element
>> 		[(member_expression (identifier)) (identifier)]
>> 		@typescript-ts-jsx-tag-face)
>>
> I really don’t know this part of the code very well. If you’ve identified an issue/bug for a specific TSX use-case, and found a fix for it, I have no objections to it being applied.

No problem, pushed that to emacs-29, thanks.

Just wanted to notify in advance in case somebody had an opinion on this 
change.

bug#65470: 29.1.50; js-ts-mode: regex pattern can cause incorrect parenthesis matching

[Eli Zaretskii]

> Date: Fri, 15 Sep 2023 16:35:55 +0300
> Cc: Eli Zaretskii <eliz@gnu.org>, Yuan Fu <casouri@gmail.com>,
>  Theodor Thornhill <theo@thornhill.no>,
>  Jostein Kjønigsen <jostein@kjonigsen.net>, btuin@mailo.com,
>  65470@debbugs.gnu.org
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> On 15/09/2023 15:11, Jostein Kjønigsen wrote:
> >> This patch seems to fix that and bring JSX syntax highlighting back:
> >>
> >> diff --git a/lisp/progmodes/typescript-ts-mode.el b/lisp/progmodes/typescript-ts-mode.el
> >> index 57382c9cb31..7108ff49b28 100644
> >> --- a/lisp/progmodes/typescript-ts-mode.el
> >> +++ b/lisp/progmodes/typescript-ts-mode.el
> >> @@ -163,7 +163,7 @@ tsx-ts-mode--font-lock-compatibility-bb1f97b
> >>    ;; but then raises an error if the wrong node type is used. So it is
> >>    ;; important to check with the new node type (member_expression)
> >>    (condition-case nil
> >> -      (progn (treesit-query-capture language '((member_expression) @capture))
> >> +      (progn (treesit-query-capture language '(jsx_opening_element (member_expression) @capture))
> >> 	     '((jsx_opening_element
> >> 		[(member_expression (identifier)) (identifier)]
> >> 		@typescript-ts-jsx-tag-face)
> >>
> > I really don’t know this part of the code very well. If you’ve identified an issue/bug for a specific TSX use-case, and found a fix for it, I have no objections to it being applied.
> 
> No problem, pushed that to emacs-29, thanks.

This seems to cause:

    INFO     Scraping files for loaddefs...85%
  typescript-ts-mode:0: Warning: Not registering prefix "ts".  Affects: ("tsx-ts-mode--indent-compatibility-b893426" "tsx-ts-mode--font-lock-compatibility-bb1f97b" "tsx-ts--s-p-query" "tsx-ts--syntax-propertize" "ts-ts--s-p-query" "ts-ts--syntax-propertize" "ts-ts--syntax-propertize-captures")

Stefan, what do we usually do with such warnings?

bug#65470: 29.1.50; js-ts-mode: regex pattern can cause incorrect parenthesis matching

[Eli Zaretskii]

> Cc: casouri@gmail.com, 65470@debbugs.gnu.org, theo@thornhill.no,
>  jostein@secure.kjonigsen.net, btuin@mailo.com, jostein@kjonigsen.net
> Date: Sat, 16 Sep 2023 08:54:32 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> 
> > Date: Fri, 15 Sep 2023 16:35:55 +0300
> > Cc: Eli Zaretskii <eliz@gnu.org>, Yuan Fu <casouri@gmail.com>,
> >  Theodor Thornhill <theo@thornhill.no>,
> >  Jostein Kjønigsen <jostein@kjonigsen.net>, btuin@mailo.com,
> >  65470@debbugs.gnu.org
> > From: Dmitry Gutov <dmitry@gutov.dev>
> > 
> > On 15/09/2023 15:11, Jostein Kjønigsen wrote:
> > >> This patch seems to fix that and bring JSX syntax highlighting back:
> > >>
> > >> diff --git a/lisp/progmodes/typescript-ts-mode.el b/lisp/progmodes/typescript-ts-mode.el
> > >> index 57382c9cb31..7108ff49b28 100644
> > >> --- a/lisp/progmodes/typescript-ts-mode.el
> > >> +++ b/lisp/progmodes/typescript-ts-mode.el
> > >> @@ -163,7 +163,7 @@ tsx-ts-mode--font-lock-compatibility-bb1f97b
> > >>    ;; but then raises an error if the wrong node type is used. So it is
> > >>    ;; important to check with the new node type (member_expression)
> > >>    (condition-case nil
> > >> -      (progn (treesit-query-capture language '((member_expression) @capture))
> > >> +      (progn (treesit-query-capture language '(jsx_opening_element (member_expression) @capture))
> > >> 	     '((jsx_opening_element
> > >> 		[(member_expression (identifier)) (identifier)]
> > >> 		@typescript-ts-jsx-tag-face)
> > >>
> > > I really don’t know this part of the code very well. If you’ve identified an issue/bug for a specific TSX use-case, and found a fix for it, I have no objections to it being applied.
> > 
> > No problem, pushed that to emacs-29, thanks.
> 
> This seems to cause:
> 
>     INFO     Scraping files for loaddefs...85%
>   typescript-ts-mode:0: Warning: Not registering prefix "ts".  Affects: ("tsx-ts-mode--indent-compatibility-b893426" "tsx-ts-mode--font-lock-compatibility-bb1f97b" "tsx-ts--s-p-query" "tsx-ts--syntax-propertize" "ts-ts--s-p-query" "ts-ts--syntax-propertize" "ts-ts--syntax-propertize-captures")
> 
> Stefan, what do we usually do with such warnings?

This also causes byte-compilation warnings on a system that has no
tree-sitter installed:

  In end of data:
  progmodes/typescript-ts-mode.el:498:17: Warning: the function ‘treesit-node-end’ is not known to be defined.
  progmodes/typescript-ts-mode.el:497:17: Warning: the function ‘treesit-node-start’ is not known to be defined.
  progmodes/typescript-ts-mode.el:474:6: Warning: the function ‘treesit-query-compile’ is not known to be defined.

Could you please fix these?

bug#65470: 29.1.50; js-ts-mode: regex pattern can cause incorrect parenthesis matching

[Eli Zaretskii]

> Cc: casouri@gmail.com, 65470@debbugs.gnu.org, theo@thornhill.no,
>  jostein@secure.kjonigsen.net, monnier@iro.umontreal.ca, btuin@mailo.com
> Date: Sat, 16 Sep 2023 14:20:16 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> 
> > Cc: casouri@gmail.com, 65470@debbugs.gnu.org, theo@thornhill.no,
> >  jostein@secure.kjonigsen.net, btuin@mailo.com, jostein@kjonigsen.net
> > Date: Sat, 16 Sep 2023 08:54:32 +0300
> > From: Eli Zaretskii <eliz@gnu.org>
> > 
> > > Date: Fri, 15 Sep 2023 16:35:55 +0300
> > > Cc: Eli Zaretskii <eliz@gnu.org>, Yuan Fu <casouri@gmail.com>,
> > >  Theodor Thornhill <theo@thornhill.no>,
> > >  Jostein Kjønigsen <jostein@kjonigsen.net>, btuin@mailo.com,
> > >  65470@debbugs.gnu.org
> > > From: Dmitry Gutov <dmitry@gutov.dev>
> > > 
> > > On 15/09/2023 15:11, Jostein Kjønigsen wrote:
> > > >> This patch seems to fix that and bring JSX syntax highlighting back:
> > > >>
> > > >> diff --git a/lisp/progmodes/typescript-ts-mode.el b/lisp/progmodes/typescript-ts-mode.el
> > > >> index 57382c9cb31..7108ff49b28 100644
> > > >> --- a/lisp/progmodes/typescript-ts-mode.el
> > > >> +++ b/lisp/progmodes/typescript-ts-mode.el
> > > >> @@ -163,7 +163,7 @@ tsx-ts-mode--font-lock-compatibility-bb1f97b
> > > >>    ;; but then raises an error if the wrong node type is used. So it is
> > > >>    ;; important to check with the new node type (member_expression)
> > > >>    (condition-case nil
> > > >> -      (progn (treesit-query-capture language '((member_expression) @capture))
> > > >> +      (progn (treesit-query-capture language '(jsx_opening_element (member_expression) @capture))
> > > >> 	     '((jsx_opening_element
> > > >> 		[(member_expression (identifier)) (identifier)]
> > > >> 		@typescript-ts-jsx-tag-face)
> > > >>
> > > > I really don’t know this part of the code very well. If you’ve identified an issue/bug for a specific TSX use-case, and found a fix for it, I have no objections to it being applied.
> > > 
> > > No problem, pushed that to emacs-29, thanks.
> > 
> > This seems to cause:
> > 
> >     INFO     Scraping files for loaddefs...85%
> >   typescript-ts-mode:0: Warning: Not registering prefix "ts".  Affects: ("tsx-ts-mode--indent-compatibility-b893426" "tsx-ts-mode--font-lock-compatibility-bb1f97b" "tsx-ts--s-p-query" "tsx-ts--syntax-propertize" "ts-ts--s-p-query" "ts-ts--syntax-propertize" "ts-ts--syntax-propertize-captures")
> > 
> > Stefan, what do we usually do with such warnings?
> 
> This also causes byte-compilation warnings on a system that has no
> tree-sitter installed:
> 
>   In end of data:
>   progmodes/typescript-ts-mode.el:498:17: Warning: the function ‘treesit-node-end’ is not known to be defined.
>   progmodes/typescript-ts-mode.el:497:17: Warning: the function ‘treesit-node-start’ is not known to be defined.
>   progmodes/typescript-ts-mode.el:474:6: Warning: the function ‘treesit-query-compile’ is not known to be defined.
> 
> Could you please fix these?

I fixed this myself by simply declaring the functions the
byte-compiler was complaining about, but I'd be happier if someone
could explain these warnings, since the offending functions are not
called anywhere in the code I could see.  What did I miss?

bug#65470: 29.1.50; js-ts-mode: regex pattern can cause incorrect parenthesis matching

[Dmitry Gutov]

On 16/09/2023 14:40, Eli Zaretskii wrote:
>> This also causes byte-compilation warnings on a system that has no
>> tree-sitter installed:
>>
>>    In end of data:
>>    progmodes/typescript-ts-mode.el:498:17: Warning: the function ‘treesit-node-end’ is not known to be defined.
>>    progmodes/typescript-ts-mode.el:497:17: Warning: the function ‘treesit-node-start’ is not known to be defined.
>>    progmodes/typescript-ts-mode.el:474:6: Warning: the function ‘treesit-query-compile’ is not known to be defined.
>>
>> Could you please fix these?
> I fixed this myself by simply declaring the functions the
> byte-compiler was complaining about, but I'd be happier if someone
> could explain these warnings, since the offending functions are not
> called anywhere in the code I could see.  What did I miss?

treesit-node-start/end are called in tsx-ts--syntax-propertize-captures 
(note the rename I just performed), and treesit-query-compile is called 
in *-ts--s-p-query's init forms.

Both added in commit 1fb2fb501f3 reviewed by yours truly, so, sorry 
about that.

bug#65470: 29.1.50; js-ts-mode: regex pattern can cause incorrect parenthesis matching

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>> No problem, pushed that to emacs-29, thanks.
>
> This seems to cause:
>
>     INFO     Scraping files for loaddefs...85%
>   typescript-ts-mode:0: Warning: Not registering prefix "ts".  Affects: ("tsx-ts-mode--indent-compatibility-b893426" "tsx-ts-mode--font-lock-compatibility-bb1f97b" "tsx-ts--s-p-query" "tsx-ts--syntax-propertize" "ts-ts--s-p-query" "ts-ts--syntax-propertize" "ts-ts--syntax-propertize-captures")
>
> Stefan, what do we usually do with such warnings?

We usually ignored them?
[ They only affect the features controlled by
  `help-enable-(completion-)autoload`.  ]

A better option, is to try and use fewer namespace prefixes.
e.g. make sure all the definitions start with `ts-`, `ts-ts-`,
`tsts-`, or somesuch (except for the autoloaded ones which can break
that rule).

Maybe rename `typescript-ts-mode-*` to `ts-ts-mode-*` (and
make `typescript-ts-mode` an alias of `ts-ts-mode`).


        Stefan


diff --git a/lisp/progmodes/typescript-ts-mode.el b/lisp/progmodes/typescript-ts-mode.el
index 7c85a7b6fe9..8c50b55535b 100644
--- a/lisp/progmodes/typescript-ts-mode.el
+++ b/lisp/progmodes/typescript-ts-mode.el
@@ -87,7 +87,7 @@ tsx-ts-mode--indent-compatibility-b893426
       (progn (treesit-query-capture 'tsx '((jsx_fragment) @capture))
              `(((match "<" "jsx_fragment") parent 0)
                ((parent-is "jsx_fragment") parent typescript-ts-mode-indent-offset)))
-    (error
+    (error ;; FIXME: Use more precise error name.
      `(((match "<" "jsx_text") parent 0)
        ((parent-is "jsx_text") parent typescript-ts-mode-indent-offset)))))
 
@@ -414,8 +414,11 @@ typescript-ts-mode--sexp-nodes
   "Nodes that designate sexps in TypeScript.
 See `treesit-thing-settings' for more information.")
 
+;; FIXME: Do we really need to autoload this?
 ;;;###autoload
 (define-derived-mode typescript-ts-base-mode prog-mode "TypeScript"
+  ;; FIXME: This is the same docstring as `typescript-ts-mode',
+  ;; we should document the difference here!
   "Major mode for editing TypeScript."
   :group 'typescript
   :syntax-table typescript-ts-mode--syntax-table

bug#65470: 29.1.50; js-ts-mode: regex pattern can cause incorrect parenthesis matching

[Dmitry Gutov]

On 16/09/2023 16:59, Stefan Monnier wrote:
>>> No problem, pushed that to emacs-29, thanks.
>> This seems to cause:
>>
>>      INFO     Scraping files for loaddefs...85%
>>    typescript-ts-mode:0: Warning: Not registering prefix "ts".  Affects: ("tsx-ts-mode--indent-compatibility-b893426" "tsx-ts-mode--font-lock-compatibility-bb1f97b" "tsx-ts--s-p-query" "tsx-ts--syntax-propertize" "ts-ts--s-p-query" "ts-ts--syntax-propertize" "ts-ts--syntax-propertize-captures")
>>
>> Stefan, what do we usually do with such warnings?
> We usually ignored them?
> [ They only affect the features controlled by
>    `help-enable-(completion-)autoload`.  ]
> 
> A better option, is to try and use fewer namespace prefixes.
> e.g. make sure all the definitions start with `ts-`, `ts-ts-`,
> `tsts-`, or somesuch (except for the autoloaded ones which can break
> that rule).
> 
> Maybe rename `typescript-ts-mode-*` to `ts-ts-mode-*` (and
> make `typescript-ts-mode` an alias of `ts-ts-mode`).

Thanks. I haven't seen the warnings myself, but I sorted the new 
addition into two existing prefixes in that file: typescript-ts and tsx-ts.

> diff --git a/lisp/progmodes/typescript-ts-mode.el b/lisp/progmodes/typescript-ts-mode.el
> index 7c85a7b6fe9..8c50b55535b 100644
> --- a/lisp/progmodes/typescript-ts-mode.el
> +++ b/lisp/progmodes/typescript-ts-mode.el
> @@ -87,7 +87,7 @@ tsx-ts-mode--indent-compatibility-b893426
>         (progn (treesit-query-capture 'tsx '((jsx_fragment) @capture))
>                `(((match "<" "jsx_fragment") parent 0)
>                  ((parent-is "jsx_fragment") parent typescript-ts-mode-indent-offset)))
> -    (error
> +    (error ;; FIXME: Use more precise error name.
>        `(((match "<" "jsx_text") parent 0)
>          ((parent-is "jsx_text") parent typescript-ts-mode-indent-offset)))))
>   
> @@ -414,8 +414,11 @@ typescript-ts-mode--sexp-nodes
>     "Nodes that designate sexps in TypeScript.
>   See `treesit-thing-settings' for more information.")
>   
> +;; FIXME: Do we really need to autoload this?
>   ;;;###autoload

Don't think so. Though I'm not sure if we should change that in emacs-29.

>   (define-derived-mode typescript-ts-base-mode prog-mode "TypeScript"
> +  ;; FIXME: This is the same docstring as `typescript-ts-mode',
> +  ;; we should document the difference here!
>     "Major mode for editing TypeScript."
>     :group 'typescript
>     :syntax-table typescript-ts-mode--syntax-table
> 

Fixed two other things on the release branch, seemed safe enough.

bug#65470: 29.1.50; js-ts-mode: regex pattern can cause incorrect parenthesis matching

[Eli Zaretskii]

> Date: Sat, 16 Sep 2023 23:05:35 +0300
> Cc: jostein@secure.kjonigsen.net, casouri@gmail.com, theo@thornhill.no,
>  jostein@kjonigsen.net, btuin@mailo.com, 65470@debbugs.gnu.org
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> On 16/09/2023 16:59, Stefan Monnier wrote:
> >>> No problem, pushed that to emacs-29, thanks.
> >> This seems to cause:
> >>
> >>      INFO     Scraping files for loaddefs...85%
> >>    typescript-ts-mode:0: Warning: Not registering prefix "ts".  Affects: ("tsx-ts-mode--indent-compatibility-b893426" "tsx-ts-mode--font-lock-compatibility-bb1f97b" "tsx-ts--s-p-query" "tsx-ts--syntax-propertize" "ts-ts--s-p-query" "ts-ts--syntax-propertize" "ts-ts--syntax-propertize-captures")
> >>
> >> Stefan, what do we usually do with such warnings?
> > We usually ignored them?
> > [ They only affect the features controlled by
> >    `help-enable-(completion-)autoload`.  ]
> > 
> > A better option, is to try and use fewer namespace prefixes.
> > e.g. make sure all the definitions start with `ts-`, `ts-ts-`,
> > `tsts-`, or somesuch (except for the autoloaded ones which can break
> > that rule).
> > 
> > Maybe rename `typescript-ts-mode-*` to `ts-ts-mode-*` (and
> > make `typescript-ts-mode` an alias of `ts-ts-mode`).
> 
> Thanks. I haven't seen the warnings myself, but I sorted the new 
> addition into two existing prefixes in that file: typescript-ts and tsx-ts.

Thanks, compiles cleanly now.

bug#65470: 29.1.50; js-ts-mode: regex pattern can cause incorrect parenthesis matching

[Augustin Chéneau]

Le 31/08/2023 à 13:15, Dmitry Gutov a écrit :
> On 31/08/2023 12:41, Eli Zaretskii wrote:
>>> Date: Sun, 27 Aug 2023 00:45:19 +0300
>>> Cc: Theodor Thornhill<theo@thornhill.no>, Eli Zaretskii<eliz@gnu.org>,
>>>   Jostein Kjønigsen<jostein@kjonigsen.net>,btuin@mailo.com,
>>>   65470@debbugs.gnu.org
>>> From: Dmitry Gutov<dmitry@gutov.dev>
>>>
>>> On 27/08/2023 00:13, Jostein Kjønigsen wrote:
>>>> I don’t really work with JSX and don’t think have a “idiomatic” code 
>>>> base to try it on where I would notice brokenness.
>>>>
>>>> I think this would be better handled by people more experienced with 
>>>> both JSX and our JSX code base than I am.
>>> Just checking that viewing some JSX code in .tsx files doesn't show
>>> errors in Messages might be good enough.
>>>
>>> I tried that with javascript, didn't notice any problems so far.
>> I'm a bit confused by this discussion.  Should I install the patch on
>> the emacs-29 branch now, or are you still discussing whether it's
>> correct?
> 
> I think we were discussing the typescript modes here.
> 
> Personally I was waiting for some additional confirmation that my v2 for 
> js-ts-mode is correct, but I guess we have a silent approval, so we can 
> install it.

Whoops, sorry.  I confirm this fixed my issue with js-ts-mode.
Thank you for your patch!

bug#65470: 29.1.50; js-ts-mode: regex pattern can cause incorrect parenthesis matching

[Eli Zaretskii]

> Cc: 65470@debbugs.gnu.org
> Date: Fri, 1 Sep 2023 17:45:17 +0200
> From: Augustin Chéneau (BTuin) <btuin@mailo.com>
> 
> > Personally I was waiting for some additional confirmation that my v2 for 
> > js-ts-mode is correct, but I guess we have a silent approval, so we can 
> > install it.
> 
> Whoops, sorry.  I confirm this fixed my issue with js-ts-mode.
> Thank you for your patch!

Thanks, I'm therefore closing the bug.

bug#65470: 29.1.50; js-ts-mode: regex pattern can cause incorrect parenthesis matching

[Dmitry Gutov]

On 01/09/2023 18:45, Augustin Chéneau (BTuin) wrote:
>>>>
>>> I'm a bit confused by this discussion.  Should I install the patch on
>>> the emacs-29 branch now, or are you still discussing whether it's
>>> correct?
>>
>> I think we were discussing the typescript modes here.
>>
>> Personally I was waiting for some additional confirmation that my v2 
>> for js-ts-mode is correct, but I guess we have a silent approval, so 
>> we can install it.
> 
> Whoops, sorry.  I confirm this fixed my issue with js-ts-mode.
> Thank you for your patch!

Thanks for testing!