bug#59906: 29.0.60; typescript-ts-mode is not highlighting escape sequences

bug#59906: 29.0.60; typescript-ts-mode is not highlighting escape sequences

[Jostein Kjønigsen]

[-- Attachment #1: Type: text/plain, Size: 15037 bytes --]

In a buffer with typescript-ts-mode activated, write the following code:

const test = "quoted\"string\" contents";

The entire text, include the \" is highlighted as strings, and escape 
sequences are not handled:

This code has the following tree-sitter syntax tree:

> (lexical_declaration kind: const
>  (variable_declarator name: (identifier) value: =
>   (string " (string_fragment) (escape_sequence) (string_fragment) 
> (escape_sequence) (string_fragment) "))
>  ;)
In typescript-ts-mode we seemingly have rules to handle fontification of 
escape sequences:

>    :language language
>    :feature 'escape-sequence
>    :override t
>    '((escape_sequence) @font-lock-escape-face))
This feature is however not activated in the major-mode setup:

>     (setq-local treesit-font-lock-feature-list
>                 '((comment declaration)
>                   (keyword string)
>                   (constant expression identifier number pattern property)
>                   (bracket delimiter)))

Naively I assumed that simply adding this feature would activate this 
rule, but still \" is fontified as string.

Invalid escape sequences are not highlighted as warnings either (like 
they are in json-ts-mode).

Would anyone like to take a stab at this? :)

--
Jostein



In GNU Emacs 29.0.60 (build 1, x86_64-pc-linux-gnu, GTK+ Version
  3.24.33, cairo version 1.16.0) of 2022-12-08 built on dev-jostein
Repository revision: 8fb2afe051168b2eac2bba0ee102039d0af90a05
Repository branch: emacs-29
Windowing system distributor 'The X.Org Foundation', version 11.0.12201001
System Description: Ubuntu 22.04.1 LTS

Configured using:
  'configure --with-tree-sitter'

Configured features:
ACL CAIRO DBUS FREETYPE GIF GLIB GMP GNUTLS GPM GSETTINGS HARFBUZZ JPEG
LCMS2 LIBOTF LIBSELINUX LIBSYSTEMD LIBXML2 M17N_FLT MODULES NOTIFY
INOTIFY PDUMPER PNG RSVG SECCOMP SOUND THREADS TIFF TOOLKIT_SCROLL_BARS
TREE_SITTER X11 XDBE XIM XINPUT2 XPM GTK3 ZLIB

Important settings:
   value of $LC_MONETARY: nb_NO.UTF-8
   value of $LC_NUMERIC: nb_NO.UTF-8
   value of $LC_TIME: nb_NO.UTF-8
   value of $LANG: en_US.UTF-8
   locale-coding-system: utf-8-unix

Major mode: TypeScript

Minor modes in effect:
   global-git-commit-mode: t
   magit-auto-revert-mode: t
   lsp-diagnostics-mode: t
   lsp-headerline-breadcrumb-mode: t
   lsp-modeline-workspace-status-mode: t
   lsp-modeline-diagnostics-mode: t
   lsp-modeline-code-actions-mode: t
   electric-pair-mode: t
   lsp-completion-mode: t
   editorconfig-mode: t
   flycheck-mode: t
   which-function-mode: t
   nlinum-mode: t
   company-mode: t
   global-ede-mode: t
   dap-tooltip-mode: t
   dap-ui-many-windows-mode: t
   dap-ui-controls-mode: t
   dap-ui-mode: t
   treemacs-filewatch-mode: t
   treemacs-follow-mode: t
   treemacs-git-mode: t
   treemacs-fringe-indicator-mode: t
   dap-auto-configure-mode: t
   dap-mode: t
   global-undo-tree-mode: t
   undo-tree-mode: t
   doom-modeline-mode: t
   projectile-mode: t
   ido-yes-or-no-mode: t
   helm-mode: t
   helm-minibuffer-history-mode: t
   shell-dirtrack-mode: t
   helm--remap-mouse-mode: t
   async-bytecomp-package-mode: t
   delete-selection-mode: t
   global-auto-revert-mode: t
   server-mode: t
   global-hl-line-mode: t
   lsp-managed-mode: t
   lsp-mode: t
   yas-global-mode: t
   yas-minor-mode: t
   tooltip-mode: t
   global-eldoc-mode: t
   eldoc-mode: t
   show-paren-mode: t
   electric-indent-mode: t
   mouse-wheel-mode: t
   menu-bar-mode: t
   file-name-shadow-mode: t
   global-font-lock-mode: t
   font-lock-mode: t
   blink-cursor-mode: t
   column-number-mode: t
   line-number-mode: t
   auto-fill-function: yas--auto-fill
   transient-mark-mode: t
   auto-composition-mode: t
   auto-encryption-mode: t
   auto-compression-mode: t

Load-path shadows:
/home/jostein/.emacs.d/elpa/transient-20221202.1727/transient hides 
/home/jostein/build/emacs/lisp/transient

Features:
(dired-aux ede/dired magit-extras magit-bookmark magit-submodule
magit-obsolete magit-blame magit-stash magit-reflog magit-bisect
magit-push magit-pull magit-fetch magit-clone magit-remote magit-commit
magit-sequence magit-notes magit-worktree magit-tag magit-merge
magit-branch magit-reset magit-files magit-refs magit-status magit
magit-repos magit-apply magit-wip magit-log magit-diff smerge-mode
git-commit log-edit pcvs-util magit-core magit-autorevert magit-margin
magit-transient magit-process with-editor magit-mode transient magit-git
magit-base magit-section crm compat-27 compat-26 semantic/lex-spp
ede/emacs semantic/db semantic/util-modes semantic/util semantic
semantic/tag semantic/lex semantic/fw mode-local bug-reference help-fns
radix-tree shadow sort flyspell ispell emacsbug message yank-media
rfc822 mml mml-sec epa derived mm-decode mm-bodies mm-encode mail-parse
rfc2231 mailabbrev gmm-utils mailheader sendmail rfc2047 rfc2045
ietf-drums mail-extr helm-command helm-elisp helm-eval
typescript-ts-mode lsp-diagnostics lsp-headerline lsp-icons lsp-modeline
view elec-pair js cc-mode cc-fonts cc-guess cc-menus cc-cmds cc-styles
cc-align cc-engine cc-vars cc-defs executable misearch multi-isearch
vc-git diff-mode vc-dispatcher winner ffap disp-table tramp-archive
tramp-gvfs tramp-cache warnings time-stamp zeroconf dbus helm-bookmark
helm-net helm-adaptive helm-info treemacs-bookmarks treemacs-tags
bookmark face-remap add-log lsp-zig lsp-steep lsp-svelte lsp-sqls
lsp-ruby-syntax-tree lsp-yaml lsp-xml lsp-vimscript lsp-vhdl lsp-volar
lsp-vetur lsp-html lsp-verilog lsp-vala lsp-v lsp-typeprof lsp-ttcn3
lsp-toml lsp-terraform lsp-tex lsp-sorbet lsp-solargraph lsp-rust lsp-rf
lsp-remark lsp-racket lsp-r lsp-purescript lsp-pylsp lsp-pyls lsp-php
lsp-pls lsp-perlnavigator lsp-perl lsp-openscad lsp-ocaml lsp-magik
lsp-nix lsp-nim lsp-nginx lsp-mint lsp-marksman lsp-markdown lsp-lua
lsp-kotlin lsp-json lsp-javascript lsp-idris lsp-haxe lsp-groovy
lsp-hack lsp-graphql lsp-gleam lsp-go lsp-completion lsp-gdscript
lsp-fsharp lsp-fortran lsp-eslint lsp-erlang lsp-emmet lsp-elixir
lsp-elm lsp-dockerfile lsp-dhall lsp-d lsp-css lsp-csharp gnutls
lsp-crystal lsp-cmake lsp-clojure lsp-semantic-tokens lsp-clangd
lsp-beancount lsp-bash lsp-astro lsp-ansible lsp-angular lsp-ada
lsp-actionscript ido-completing-read+ memoize minibuf-eldef
elisp-slime-nav paredit editorconfig editorconfig-core
editorconfig-core-handle editorconfig-fnmatch realgud realgud-zshdb
realgud:zshdb-track-mode realgud:zshdb-core realgud:zshdb-init
realgud-trepan3k realgud:trepan3k-track-mode realgud:trepan3k-core
realgud:trepan3k-init realgud-trepan2 realgud:trepan2-track-mode
realgud:trepan2-core realgud:trepan2-init realgud-trepanpl
realgud:trepanpl-track-mode realgud:trepanpl-core realgud:trepanpl-init
realgud-trepanjs realgud:trepanjs-track-mode realgud:trepanjs-core
realgud:trepanjs-init realgud-lang-js realgud-trepan
realgud:trepan-track-mode realgud:trepan-core realgud:trepan-init
realgud-remake realgud:remake-track-mode realgud:remake-core
realgud:remake-init realgud-rdebug realgud-rdebug-track-mode
realgud-rdebug-core realgud-rdebug-init realgud-lang-ruby realgud-perldb
realgud:perldb-track-mode realgud:perldb-core realgud:perldb-init
realgud-lang-perl realgud-pdb realgud:pdb-track-mode realgud:pdb-core
realgud:pdb-init realgud-lang-python python treesit realgud-kshdb
realgud:kshdb-track-mode realgud:kshdb-core realgud:kshdb-init
realgud-gub realgud:gub-track-mode realgud:gub-core realgud:gub-init
realgud-gdb realgud:gdb-track-mode realgud:gdb-init realgud:gdb-core
realgud-bashdb realgud:bashdb-track-mode realgud:bashdb-core
realgud:bashdb-init realgud-lang-posix-shell realgud:run
realgud-locals-mode realgud-breakpoint-mode realgud-backtrack-mode
realgud-track-mode realgud-backtrace-mode realgud-attach
realgud-lang-java realgud-track realgud-shortkey realgud-menu
realgud-eval realgud-cmds realgud-send realgud-window realgud-utils
eshell realgud-init realgud-file realgud-core realgud-reset
realgud-buffer-helper realgud-buffer-breakpoint realgud-buffer-backtrace
realgud-locals realgud-buffer-locals realgud-buffer-command
realgud-buffer-info realgud-lochist realgud-bp realgud-bp-image-data
realgud-lang esh-mode esh-cmd esh-ext esh-opt esh-proc esh-io esh-arg
esh-module esh-groups esh-util cus-start realgud-loc
realgud-buffer-source realgud-key key realgud-follow realgud-fringe
realgud-helper loc-changes realgud-regexp realgud-custom load-relative
flycheck highlight-symbol which-func edebug debug backtrace nlinum linum
company-oddmuse company-keywords company-etags etags fileloop generator
company-gtags company-dabbrev-code company-dabbrev company-files
company-clang company-capf company-cmake company-semantic
company-template company-bbdb company-web-html company-web company-css
web-completion-data company eww url-queue shr pixel-fill kinsoku
url-file svg mm-url gnus nnheader gnus-util mail-utils range mm-util
mail-prsvr ede/speedbar ede/files ede ede/detect ede/base ede/auto
ede/source eieio-base eieio-speedbar speedbar ezimage dframe
eieio-custom cedet dap-mouse dap-ui lsp-treemacs lsp-treemacs-generic
lsp-treemacs-themes treemacs-treelib treemacs treemacs-header-line
treemacs-compatibility treemacs-mode treemacs-interface
treemacs-persistence treemacs-filewatch-mode treemacs-follow-mode
treemacs-rendering treemacs-annotations treemacs-async
treemacs-workspaces treemacs-dom treemacs-visuals
treemacs-fringe-indicator treemacs-scope pulse treemacs-faces
treemacs-icons treemacs-themes treemacs-core-utils pfuture
treemacs-logging treemacs-customization treemacs-macros gdb-mi bindat
gud bui bui-list bui-info bui-entry bui-core bui-history bui-button
bui-utils lsp-lens dap-gdb-lldb dap-netcore dap-node dap-utils dom xml
dap-pwsh lsp-pwsh dap-python dap-mode dap-tasks dap-launch lsp-docker
yaml posframe dap-overlays undo-tree diff queue doom-modeline
doom-modeline-segments doom-modeline-env doom-modeline-core shrink-path
compat compat-macs projectile lisp-mnt grep ibuf-ext ibuffer
ibuffer-loaddefs helm-imenu ob-plantuml org ob ob-tangle ob-ref ob-lob
ob-table ob-exp org-macro org-src ob-comint org-pcomplete org-list
org-footnote org-faces org-entities ob-emacs-lisp ob-core ob-eval
org-cycle org-table ol org-fold org-fold-core org-keys oc org-loaddefs
find-func cal-menu calendar cal-loaddefs org-version org-compat org-macs
ido-yes-or-no advice ido helm-mode helm-misc helm-files image-dired
image-dired-tags image-dired-external image-dired-util xdg image-mode
dired dired-loaddefs exif tramp tramp-loaddefs trampver
tramp-integration cus-edit pp cus-load files-x tramp-compat shell
pcomplete parse-time iso8601 time-date ls-lisp helm-buffers helm-occur
helm-tags helm-locate helm-grep helm-regexp format-spec helm-utils
helm-help helm-types helm helm-global-bindings helm-easymenu edmacro
kmacro helm-core easy-mmode async-bytecomp helm-source helm-multi-match
helm-lib async helm-config delsel cl-extra autorevert server hl-line
lsp-mode lsp-protocol yasnippet help-mode xref project tree-widget
wid-edit spinner pcase network-stream puny nsm markdown-mode color
thingatpt noutline outline icons lv inline imenu ht filenotify f
f-shortdoc shortdoc s ewoc epg rfc6068 epg-config dash dracula-theme
compile-eslint compile text-property-search comint ansi-osc ansi-color
ring cl finder-inf tree-sitter-indent-autoloads expand-region-autoloads
ido-yes-or-no-autoloads nodejs-repl-autoloads marmalade-client-autoloads
doom-modeline-autoloads elisp-slime-nav-autoloads
multiple-cursors-autoloads dracula-theme-autoloads
git-timemachine-autoloads crontab-mode-autoloads
highlight-symbol-autoloads ssh-config-mode-autoloads langtool-autoloads
dap-mode-autoloads lsp-treemacs-autoloads treemacs-autoloads
cfrs-autoloads posframe-autoloads gh-autoloads marshal-autoloads
logito-autoloads pcache-autoloads kv-autoloads toml-mode-autoloads
company-web-autoloads shrink-path-autoloads realgud-autoloads
realgud-recursive-autoloads load-relative-autoloads paredit-autoloads
bui-autoloads tree-sitter-langs-autoloads helpful-autoloads
elisp-refs-autoloads magit-autoloads magit-section-autoloads
git-commit-autoloads with-editor-autoloads flycheck-package-autoloads
package-lint-autoloads rust-mode-autoloads yaml-mode-autoloads
macrostep-autoloads lsp-docker-autoloads yaml-autoloads
lsp-mode-autoloads spinner-autoloads f-autoloads hydra-autoloads
nlinum-autoloads ht-autoloads loc-changes-autoloads transient-autoloads
compat-autoloads web-mode-autoloads undo-tree-autoloads queue-autoloads
ace-window-autoloads avy-autoloads bmx-mode-autoloads company-autoloads
test-simple-autoloads js2-mode-autoloads web-autoloads s-autoloads
ido-completing-read+-autoloads memoize-autoloads lv-autoloads
imenu-anywhere-autoloads helm-projectile-autoloads projectile-autoloads
helm-autoloads popup-autoloads helm-core-autoloads async-autoloads
tree-sitter-autoloads tsc-autoloads editorconfig-autoloads
json-mode-autoloads rx json-snatcher-autoloads yasnippet-autoloads
web-completion-data-autoloads cargo-autoloads markdown-mode-autoloads
pfuture-autoloads flycheck-autoloads pkg-info-autoloads epl-autoloads
info dash-autoloads package browse-url url url-proxy url-privacy
url-expand url-methods url-history url-cookie generate-lisp-file
url-domsuf url-util mailcap url-handlers url-parse auth-source cl-seq
eieio eieio-core cl-macs password-cache json subr-x map byte-opt gv
bytecomp byte-compile url-vars cl-loaddefs cl-lib rmc iso-transl tooltip
cconv eldoc paren electric uniquify ediff-hook vc-hooks lisp-float-type
elisp-mode mwheel term/x-win x-win term/common-win x-dnd tool-bar dnd
fontset image regexp-opt fringe tabulated-list replace newcomment
text-mode lisp-mode prog-mode register page tab-bar menu-bar rfn-eshadow
isearch easymenu timer select scroll-bar mouse jit-lock font-lock syntax
font-core term/tty-colors frame minibuffer nadvice seq simple cl-generic
indonesian philippine cham georgian utf-8-lang misc-lang vietnamese
tibetan thai tai-viet lao korean japanese eucjp-ms cp51932 hebrew greek
romanian slovak czech european ethiopic indian cyrillic chinese
composite emoji-zwj charscript charprop case-table epa-hook
jka-cmpr-hook help abbrev obarray oclosure cl-preloaded button loaddefs
theme-loaddefs faces cus-face macroexp files window text-properties
overlay sha1 md5 base64 format env code-pages mule custom widget keymap
hashtable-print-readable backquote threads dbusbind inotify lcms2
dynamic-setting system-font-setting font-render-setting cairo
move-toolbar gtk x-toolkit xinput2 x multi-tty make-network-process
emacs)

Memory information:
((conses 16 679481 87371)
  (symbols 48 59262 47)
  (strings 32 211608 16397)
  (string-bytes 1 6867045)
  (vectors 16 125808)
  (vector-slots 8 2378444 112756)
  (floats 8 591 891)
  (intervals 56 9416 3545)
  (buffers 992 46))

-- 
*Jostein Kjønigsen*
jostein.kjønigsen.no <https://jostein.kjønigsen.no>
jostein@kjonigsen.net - jostein@gmail.com

[-- Attachment #2: Type: text/html, Size: 18997 bytes --]

bug#59906: 29.0.60; typescript-ts-mode is not highlighting escape sequences

[Theodor Thornhill via Bug reports for GNU Emacs, the Swiss army knife of text editors]

[-- Attachment #1: Type: text/plain, Size: 1333 bytes --]

Jostein Kjønigsen <jostein@secure.kjonigsen.net> writes:

> In a buffer with typescript-ts-mode activated, write the following code:
>
> const test = "quoted\"string\" contents";
>
> The entire text, include the \" is highlighted as strings, and escape sequences are not handled:
>
> This code has the following tree-sitter syntax tree:
>
>  (lexical_declaration kind: const
>   (variable_declarator name: (identifier) value: =
>    (string " (string_fragment) (escape_sequence) (string_fragment) (escape_sequence) (string_fragment) "))
>   ;)
>
> In typescript-ts-mode we seemingly have rules to handle fontification of escape sequences: 
>
>     :language language
>     :feature 'escape-sequence
>     :override t
>     '((escape_sequence) @font-lock-escape-face))
>
> This feature is however not activated in the major-mode setup: 
>
>      (setq-local treesit-font-lock-feature-list
>                  '((comment declaration)
>                    (keyword string)
>                    (constant expression identifier number pattern property)
>                    (bracket delimiter)))
>
> Naively I assumed that simply adding this feature would activate this rule, but still \" is fontified as string.

Yeah, there was an error there.  See below patch.

Yuan, can you install this?

Theo


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Fix-escape-sequence-feature-in-typescript-ts-mode-bu.patch --]
[-- Type: text/x-diff, Size: 1592 bytes --]

From 1b0f2a181c38902804e323b064ab85cf84d88ff5 Mon Sep 17 00:00:00 2001
From: Theodor Thornhill <theo@thornhill.no>
Date: Thu, 8 Dec 2022 16:17:49 +0100
Subject: [PATCH] Fix escape-sequence feature in typescript-ts-mode (bug#59906)

* lisp/progmodes/typescript-ts-mode.el: (typescript-ts-mode,
tsx-ts-mode): Use escape-sequence feature.
---
 lisp/progmodes/typescript-ts-mode.el | 6 ++----
 1 file changed, 2 insertions(+), 4 deletions(-)

diff --git a/lisp/progmodes/typescript-ts-mode.el b/lisp/progmodes/typescript-ts-mode.el
index a56568ae78..d5ca813e34 100644
--- a/lisp/progmodes/typescript-ts-mode.el
+++ b/lisp/progmodes/typescript-ts-mode.el
@@ -362,8 +362,7 @@ typescript-ts-mode
     (setq-local treesit-font-lock-settings
                 (typescript-ts-mode--font-lock-settings 'typescript))
     (setq-local treesit-font-lock-feature-list
-                '((comment declaration)
-                  (keyword string)
+                '((comment declaration keyword string escape-sequence)
                   (constant expression identifier number pattern property)
                   (bracket delimiter)))
 
@@ -397,8 +396,7 @@ tsx-ts-mode
     (setq-local treesit-font-lock-settings
                 (typescript-ts-mode--font-lock-settings 'tsx))
     (setq-local treesit-font-lock-feature-list
-                '((comment declaration)
-                  (keyword string)
+                '((comment declaration keyword string escape-sequence)
                   (constant expression identifier jsx number pattern property)
                   (bracket delimiter)))
 
-- 
2.34.1

bug#59906: 29.0.60; typescript-ts-mode is not highlighting escape sequences

[Jostein Kjønigsen]

On 08.12.2022 16:19, Theodor Thornhill wrote:
> Jostein Kjønigsen <jostein@secure.kjonigsen.net> writes:
>
>> In a buffer with typescript-ts-mode activated, write the following code:
>>
>> const test = "quoted\"string\" contents";
>>
>> The entire text, include the \" is highlighted as strings, and escape sequences are not handled:
>>
>> This code has the following tree-sitter syntax tree:
>>
>>   (lexical_declaration kind: const
>>    (variable_declarator name: (identifier) value: =
>>     (string " (string_fragment) (escape_sequence) (string_fragment) (escape_sequence) (string_fragment) "))
>>    ;)
>>
>> In typescript-ts-mode we seemingly have rules to handle fontification of escape sequences:
>>
>>      :language language
>>      :feature 'escape-sequence
>>      :override t
>>      '((escape_sequence) @font-lock-escape-face))
>>
>> This feature is however not activated in the major-mode setup:
>>
>>       (setq-local treesit-font-lock-feature-list
>>                   '((comment declaration)
>>                     (keyword string)
>>                     (constant expression identifier number pattern property)
>>                     (bracket delimiter)))
>>
>> Naively I assumed that simply adding this feature would activate this rule, but still \" is fontified as string.
> Yeah, there was an error there.  See below patch.
>
> Yuan, can you install this?
>
> Theo
>
Yeah that works for me.

Yuan: another vote for installing right here :)

--
Jostein

bug#59906: 29.0.60; typescript-ts-mode is not highlighting escape sequences

[Yuan Fu]

Jostein Kjønigsen <jostein@secure.kjonigsen.net> writes:

> On 08.12.2022 16:19, Theodor Thornhill wrote:
>> Jostein Kjønigsen <jostein@secure.kjonigsen.net> writes:
>>
>>> In a buffer with typescript-ts-mode activated, write the following code:
>>>
>>> const test = "quoted\"string\" contents";
>>>
>>> The entire text, include the \" is highlighted as strings, and escape sequences are not handled:
>>>
>>> This code has the following tree-sitter syntax tree:
>>>
>>>   (lexical_declaration kind: const
>>>    (variable_declarator name: (identifier) value: =
>>>     (string " (string_fragment) (escape_sequence) (string_fragment) (escape_sequence) (string_fragment) "))
>>>    ;)
>>>
>>> In typescript-ts-mode we seemingly have rules to handle fontification of escape sequences:
>>>
>>>      :language language
>>>      :feature 'escape-sequence
>>>      :override t
>>>      '((escape_sequence) @font-lock-escape-face))
>>>
>>> This feature is however not activated in the major-mode setup:
>>>
>>>       (setq-local treesit-font-lock-feature-list
>>>                   '((comment declaration)
>>>                     (keyword string)
>>>                     (constant expression identifier number pattern property)
>>>                     (bracket delimiter)))
>>>
>>> Naively I assumed that simply adding this feature would activate this rule, but still \" is fontified as string.
>> Yeah, there was an error there.  See below patch.
>>
>> Yuan, can you install this?
>>
>> Theo
>>
> Yeah that works for me.
>
> Yuan: another vote for installing right here :)

Yep, applied, many thanks!

Yuan