bug#25529: 25.1.90; js-mode: Regexp literal with unbalanced brackets breaks font-lock

bug#25529: 25.1.90; js-mode: Regexp literal with unbalanced brackets breaks font-lock

[Mikhail Gusarov]

0. emacs -Q
1. Open a buffer, M-x js-mode
2. Enter the following text: /[^[]/
3. Enter any text on the following line, note it is highlighted as a
    string literal
4. Remove ^[ from the regex literal, note that the following text
    is now highlighted correctly


In GNU Emacs 25.1.90.1 (x86_64-apple-darwin15.6.0, NS appkit-1404.47 
Version 10.11.6 (Build 15G1212))
  of 2016-12-26 built on newton.malta.dottedmag.net
Repository revision: b5f1d2159db6c883f5adfe55a2cac8f450917afe
Windowing system distributor 'Apple', version 10.3.1404
Configured using:
  'configure --with-ns --with-tiff --with-jpeg --with-xpm --with-png
  --with-modules CFLAGS=-I/opt/local/include LDFLAGS=-L/opt/local/lib'

Configured features:
JPEG RSVG DBUS NOTIFY ACL GNUTLS LIBXML2 ZLIB TOOLKIT_SCROLL_BARS NS
MODULES

Important settings:
   value of $LANG: en_NO.UTF-8
   locale-coding-system: utf-8-unix

Major mode: Emacs-Lisp

Minor modes in effect:
   savehist-mode: t
   which-function-mode: t
   show-paren-mode: t
   which-key-mode: t
   diff-auto-refine-mode: t
   magit-auto-revert-mode: t
   global-git-commit-mode: t
   async-bytecomp-package-mode: t
   shell-dirtrack-mode: t
   global-flycheck-mode: t
   flycheck-mode: t
   global-auto-revert-mode: t
   rainbow-delimiters-mode: t
   paredit-mode: t
   global-eldoc-mode: t
   electric-indent-mode: t
   mouse-wheel-mode: t
   prettify-symbols-mode: t
   menu-bar-mode: t
   file-name-shadow-mode: t
   global-font-lock-mode: t
   font-lock-mode: t
   blink-cursor-mode: t
   auto-composition-mode: t
   auto-encryption-mode: t
   auto-compression-mode: t
   buffer-read-only: t
   column-number-mode: t
   line-number-mode: t
   transient-mark-mode: t

Recent messages:
Search failed. This means there is unmatched expression somewhere or we 
are at the beginning/end of file. [2 times]
Undo!
funcall-interactively: End of buffer
Saving file /Users/hola/tmp/foo.js...
Wrote /Users/hola/tmp/foo.js
Note: file is write protected
Type "q" in help window to restore its previous buffer.
uncompressing js.el.gz...done
Note: file is write protected
Unable to find location in file

Load-path shadows:
/Users/hola/.var-emacs/el-get/org-mode/lisp/ox hides 
/Applications/Emacs.app/Contents/Resources/lisp/org/ox
/Users/hola/.var-emacs/el-get/org-mode/lisp/ox-texinfo hides 
/Applications/Emacs.app/Contents/Resources/lisp/org/ox-texinfo
/Users/hola/.var-emacs/el-get/org-mode/lisp/ox-publish hides 
/Applications/Emacs.app/Contents/Resources/lisp/org/ox-publish
/Users/hola/.var-emacs/el-get/org-mode/lisp/ox-org hides 
/Applications/Emacs.app/Contents/Resources/lisp/org/ox-org
/Users/hola/.var-emacs/el-get/org-mode/lisp/ox-odt hides 
/Applications/Emacs.app/Contents/Resources/lisp/org/ox-odt
/Users/hola/.var-emacs/el-get/org-mode/lisp/ox-md hides 
/Applications/Emacs.app/Contents/Resources/lisp/org/ox-md
/Users/hola/.var-emacs/el-get/org-mode/lisp/ox-man hides 
/Applications/Emacs.app/Contents/Resources/lisp/org/ox-man
/Users/hola/.var-emacs/el-get/org-mode/lisp/ox-latex hides 
/Applications/Emacs.app/Contents/Resources/lisp/org/ox-latex
/Users/hola/.var-emacs/el-get/org-mode/lisp/ox-icalendar hides 
/Applications/Emacs.app/Contents/Resources/lisp/org/ox-icalendar
/Users/hola/.var-emacs/el-get/org-mode/lisp/ox-html hides 
/Applications/Emacs.app/Contents/Resources/lisp/org/ox-html
/Users/hola/.var-emacs/el-get/org-mode/lisp/ox-beamer hides 
/Applications/Emacs.app/Contents/Resources/lisp/org/ox-beamer
/Users/hola/.var-emacs/el-get/org-mode/lisp/ox-ascii hides 
/Applications/Emacs.app/Contents/Resources/lisp/org/ox-ascii
/Users/hola/.var-emacs/el-get/org-mode/lisp/org hides 
/Applications/Emacs.app/Contents/Resources/lisp/org/org
/Users/hola/.var-emacs/el-get/org-mode/lisp/org-w3m hides 
/Applications/Emacs.app/Contents/Resources/lisp/org/org-w3m
/Users/hola/.var-emacs/el-get/org-mode/lisp/org-version hides 
/Applications/Emacs.app/Contents/Resources/lisp/org/org-version
/Users/hola/.var-emacs/el-get/org-mode/lisp/org-timer hides 
/Applications/Emacs.app/Contents/Resources/lisp/org/org-timer
/Users/hola/.var-emacs/el-get/org-mode/lisp/org-table hides 
/Applications/Emacs.app/Contents/Resources/lisp/org/org-table
/Users/hola/.var-emacs/el-get/org-mode/lisp/org-src hides 
/Applications/Emacs.app/Contents/Resources/lisp/org/org-src
/Users/hola/.var-emacs/el-get/org-mode/lisp/org-rmail hides 
/Applications/Emacs.app/Contents/Resources/lisp/org/org-rmail
/Users/hola/.var-emacs/el-get/org-mode/lisp/org-protocol hides 
/Applications/Emacs.app/Contents/Resources/lisp/org/org-protocol
/Users/hola/.var-emacs/el-get/org-mode/lisp/org-plot hides 
/Applications/Emacs.app/Contents/Resources/lisp/org/org-plot
/Users/hola/.var-emacs/el-get/org-mode/lisp/org-pcomplete hides 
/Applications/Emacs.app/Contents/Resources/lisp/org/org-pcomplete
/Users/hola/.var-emacs/el-get/org-mode/lisp/org-mouse hides 
/Applications/Emacs.app/Contents/Resources/lisp/org/org-mouse
/Users/hola/.var-emacs/el-get/org-mode/lisp/org-mobile hides 
/Applications/Emacs.app/Contents/Resources/lisp/org/org-mobile
/Users/hola/.var-emacs/el-get/org-mode/lisp/org-mhe hides 
/Applications/Emacs.app/Contents/Resources/lisp/org/org-mhe
/Users/hola/.var-emacs/el-get/org-mode/lisp/org-macs hides 
/Applications/Emacs.app/Contents/Resources/lisp/org/org-macs
/Users/hola/.var-emacs/el-get/org-mode/lisp/org-macro hides 
/Applications/Emacs.app/Contents/Resources/lisp/org/org-macro
/Users/hola/.var-emacs/el-get/org-mode/lisp/org-loaddefs hides 
/Applications/Emacs.app/Contents/Resources/lisp/org/org-loaddefs
/Users/hola/.var-emacs/el-get/org-mode/lisp/org-list hides 
/Applications/Emacs.app/Contents/Resources/lisp/org/org-list
/Users/hola/.var-emacs/el-get/org-mode/lisp/org-irc hides 
/Applications/Emacs.app/Contents/Resources/lisp/org/org-irc
/Users/hola/.var-emacs/el-get/org-mode/lisp/org-install hides 
/Applications/Emacs.app/Contents/Resources/lisp/org/org-install
/Users/hola/.var-emacs/el-get/org-mode/lisp/org-inlinetask hides 
/Applications/Emacs.app/Contents/Resources/lisp/org/org-inlinetask
/Users/hola/.var-emacs/el-get/org-mode/lisp/org-info hides 
/Applications/Emacs.app/Contents/Resources/lisp/org/org-info
/Users/hola/.var-emacs/el-get/org-mode/lisp/org-indent hides 
/Applications/Emacs.app/Contents/Resources/lisp/org/org-indent
/Users/hola/.var-emacs/el-get/org-mode/lisp/org-id hides 
/Applications/Emacs.app/Contents/Resources/lisp/org/org-id
/Users/hola/.var-emacs/el-get/org-mode/lisp/org-habit hides 
/Applications/Emacs.app/Contents/Resources/lisp/org/org-habit
/Users/hola/.var-emacs/el-get/org-mode/lisp/org-gnus hides 
/Applications/Emacs.app/Contents/Resources/lisp/org/org-gnus
/Users/hola/.var-emacs/el-get/org-mode/lisp/org-footnote hides 
/Applications/Emacs.app/Contents/Resources/lisp/org/org-footnote
/Users/hola/.var-emacs/el-get/org-mode/lisp/org-feed hides 
/Applications/Emacs.app/Contents/Resources/lisp/org/org-feed
/Users/hola/.var-emacs/el-get/org-mode/lisp/org-faces hides 
/Applications/Emacs.app/Contents/Resources/lisp/org/org-faces
/Users/hola/.var-emacs/el-get/org-mode/lisp/org-eshell hides 
/Applications/Emacs.app/Contents/Resources/lisp/org/org-eshell
/Users/hola/.var-emacs/el-get/org-mode/lisp/org-entities hides 
/Applications/Emacs.app/Contents/Resources/lisp/org/org-entities
/Users/hola/.var-emacs/el-get/org-mode/lisp/org-element hides 
/Applications/Emacs.app/Contents/Resources/lisp/org/org-element
/Users/hola/.var-emacs/el-get/org-mode/lisp/org-docview hides 
/Applications/Emacs.app/Contents/Resources/lisp/org/org-docview
/Users/hola/.var-emacs/el-get/org-mode/lisp/org-datetree hides 
/Applications/Emacs.app/Contents/Resources/lisp/org/org-datetree
/Users/hola/.var-emacs/el-get/org-mode/lisp/org-ctags hides 
/Applications/Emacs.app/Contents/Resources/lisp/org/org-ctags
/Users/hola/.var-emacs/el-get/org-mode/lisp/org-crypt hides 
/Applications/Emacs.app/Contents/Resources/lisp/org/org-crypt
/Users/hola/.var-emacs/el-get/org-mode/lisp/org-compat hides 
/Applications/Emacs.app/Contents/Resources/lisp/org/org-compat
/Users/hola/.var-emacs/el-get/org-mode/lisp/org-colview hides 
/Applications/Emacs.app/Contents/Resources/lisp/org/org-colview
/Users/hola/.var-emacs/el-get/org-mode/lisp/org-clock hides 
/Applications/Emacs.app/Contents/Resources/lisp/org/org-clock
/Users/hola/.var-emacs/el-get/org-mode/lisp/org-capture hides 
/Applications/Emacs.app/Contents/Resources/lisp/org/org-capture
/Users/hola/.var-emacs/el-get/org-mode/lisp/org-bibtex hides 
/Applications/Emacs.app/Contents/Resources/lisp/org/org-bibtex
/Users/hola/.var-emacs/el-get/org-mode/lisp/org-bbdb hides 
/Applications/Emacs.app/Contents/Resources/lisp/org/org-bbdb
/Users/hola/.var-emacs/el-get/org-mode/lisp/org-attach hides 
/Applications/Emacs.app/Contents/Resources/lisp/org/org-attach
/Users/hola/.var-emacs/el-get/org-mode/lisp/org-archive hides 
/Applications/Emacs.app/Contents/Resources/lisp/org/org-archive
/Users/hola/.var-emacs/el-get/org-mode/lisp/org-agenda hides 
/Applications/Emacs.app/Contents/Resources/lisp/org/org-agenda
/Users/hola/.var-emacs/el-get/org-mode/lisp/ob hides 
/Applications/Emacs.app/Contents/Resources/lisp/org/ob
/Users/hola/.var-emacs/el-get/org-mode/lisp/ob-tangle hides 
/Applications/Emacs.app/Contents/Resources/lisp/org/ob-tangle
/Users/hola/.var-emacs/el-get/org-mode/lisp/ob-table hides 
/Applications/Emacs.app/Contents/Resources/lisp/org/ob-table
/Users/hola/.var-emacs/el-get/org-mode/lisp/ob-sqlite hides 
/Applications/Emacs.app/Contents/Resources/lisp/org/ob-sqlite
/Users/hola/.var-emacs/el-get/org-mode/lisp/ob-sql hides 
/Applications/Emacs.app/Contents/Resources/lisp/org/ob-sql
/Users/hola/.var-emacs/el-get/org-mode/lisp/ob-shen hides 
/Applications/Emacs.app/Contents/Resources/lisp/org/ob-shen
/Users/hola/.var-emacs/el-get/org-mode/lisp/ob-screen hides 
/Applications/Emacs.app/Contents/Resources/lisp/org/ob-screen
/Users/hola/.var-emacs/el-get/org-mode/lisp/ob-scheme hides 
/Applications/Emacs.app/Contents/Resources/lisp/org/ob-scheme
/Users/hola/.var-emacs/el-get/org-mode/lisp/ob-scala hides 
/Applications/Emacs.app/Contents/Resources/lisp/org/ob-scala
/Users/hola/.var-emacs/el-get/org-mode/lisp/ob-sass hides 
/Applications/Emacs.app/Contents/Resources/lisp/org/ob-sass
/Users/hola/.var-emacs/el-get/org-mode/lisp/ob-ruby hides 
/Applications/Emacs.app/Contents/Resources/lisp/org/ob-ruby
/Users/hola/.var-emacs/el-get/org-mode/lisp/ob-ref hides 
/Applications/Emacs.app/Contents/Resources/lisp/org/ob-ref
/Users/hola/.var-emacs/el-get/org-mode/lisp/ob-R hides 
/Applications/Emacs.app/Contents/Resources/lisp/org/ob-R
/Users/hola/.var-emacs/el-get/org-mode/lisp/ob-python hides 
/Applications/Emacs.app/Contents/Resources/lisp/org/ob-python
/Users/hola/.var-emacs/el-get/org-mode/lisp/ob-plantuml hides 
/Applications/Emacs.app/Contents/Resources/lisp/org/ob-plantuml
/Users/hola/.var-emacs/el-get/org-mode/lisp/ob-picolisp hides 
/Applications/Emacs.app/Contents/Resources/lisp/org/ob-picolisp
/Users/hola/.var-emacs/el-get/org-mode/lisp/ob-perl hides 
/Applications/Emacs.app/Contents/Resources/lisp/org/ob-perl
/Users/hola/.var-emacs/el-get/org-mode/lisp/ob-org hides 
/Applications/Emacs.app/Contents/Resources/lisp/org/ob-org
/Users/hola/.var-emacs/el-get/org-mode/lisp/ob-octave hides 
/Applications/Emacs.app/Contents/Resources/lisp/org/ob-octave
/Users/hola/.var-emacs/el-get/org-mode/lisp/ob-ocaml hides 
/Applications/Emacs.app/Contents/Resources/lisp/org/ob-ocaml
/Users/hola/.var-emacs/el-get/org-mode/lisp/ob-mscgen hides 
/Applications/Emacs.app/Contents/Resources/lisp/org/ob-mscgen
/Users/hola/.var-emacs/el-get/org-mode/lisp/ob-maxima hides 
/Applications/Emacs.app/Contents/Resources/lisp/org/ob-maxima
/Users/hola/.var-emacs/el-get/org-mode/lisp/ob-matlab hides 
/Applications/Emacs.app/Contents/Resources/lisp/org/ob-matlab
/Users/hola/.var-emacs/el-get/org-mode/lisp/ob-makefile hides 
/Applications/Emacs.app/Contents/Resources/lisp/org/ob-makefile
/Users/hola/.var-emacs/el-get/org-mode/lisp/ob-lob hides 
/Applications/Emacs.app/Contents/Resources/lisp/org/ob-lob
/Users/hola/.var-emacs/el-get/org-mode/lisp/ob-lisp hides 
/Applications/Emacs.app/Contents/Resources/lisp/org/ob-lisp
/Users/hola/.var-emacs/el-get/org-mode/lisp/ob-lilypond hides 
/Applications/Emacs.app/Contents/Resources/lisp/org/ob-lilypond
/Users/hola/.var-emacs/el-get/org-mode/lisp/ob-ledger hides 
/Applications/Emacs.app/Contents/Resources/lisp/org/ob-ledger
/Users/hola/.var-emacs/el-get/org-mode/lisp/ob-latex hides 
/Applications/Emacs.app/Contents/Resources/lisp/org/ob-latex
/Users/hola/.var-emacs/el-get/org-mode/lisp/ob-keys hides 
/Applications/Emacs.app/Contents/Resources/lisp/org/ob-keys
/Users/hola/.var-emacs/el-get/org-mode/lisp/ob-js hides 
/Applications/Emacs.app/Contents/Resources/lisp/org/ob-js
/Users/hola/.var-emacs/el-get/org-mode/lisp/ob-java hides 
/Applications/Emacs.app/Contents/Resources/lisp/org/ob-java
/Users/hola/.var-emacs/el-get/org-mode/lisp/ob-io hides 
/Applications/Emacs.app/Contents/Resources/lisp/org/ob-io
/Users/hola/.var-emacs/el-get/org-mode/lisp/ob-haskell hides 
/Applications/Emacs.app/Contents/Resources/lisp/org/ob-haskell
/Users/hola/.var-emacs/el-get/org-mode/lisp/ob-gnuplot hides 
/Applications/Emacs.app/Contents/Resources/lisp/org/ob-gnuplot
/Users/hola/.var-emacs/el-get/org-mode/lisp/ob-fortran hides 
/Applications/Emacs.app/Contents/Resources/lisp/org/ob-fortran
/Users/hola/.var-emacs/el-get/org-mode/lisp/ob-exp hides 
/Applications/Emacs.app/Contents/Resources/lisp/org/ob-exp
/Users/hola/.var-emacs/el-get/org-mode/lisp/ob-eval hides 
/Applications/Emacs.app/Contents/Resources/lisp/org/ob-eval
/Users/hola/.var-emacs/el-get/org-mode/lisp/ob-emacs-lisp hides 
/Applications/Emacs.app/Contents/Resources/lisp/org/ob-emacs-lisp
/Users/hola/.var-emacs/el-get/org-mode/lisp/ob-dot hides 
/Applications/Emacs.app/Contents/Resources/lisp/org/ob-dot
/Users/hola/.var-emacs/el-get/org-mode/lisp/ob-ditaa hides 
/Applications/Emacs.app/Contents/Resources/lisp/org/ob-ditaa
/Users/hola/.var-emacs/el-get/org-mode/lisp/ob-css hides 
/Applications/Emacs.app/Contents/Resources/lisp/org/ob-css
/Users/hola/.var-emacs/el-get/org-mode/lisp/ob-core hides 
/Applications/Emacs.app/Contents/Resources/lisp/org/ob-core
/Users/hola/.var-emacs/el-get/org-mode/lisp/ob-comint hides 
/Applications/Emacs.app/Contents/Resources/lisp/org/ob-comint
/Users/hola/.var-emacs/el-get/org-mode/lisp/ob-clojure hides 
/Applications/Emacs.app/Contents/Resources/lisp/org/ob-clojure
/Users/hola/.var-emacs/el-get/org-mode/lisp/ob-calc hides 
/Applications/Emacs.app/Contents/Resources/lisp/org/ob-calc
/Users/hola/.var-emacs/el-get/org-mode/lisp/ob-C hides 
/Applications/Emacs.app/Contents/Resources/lisp/org/ob-C
/Users/hola/.var-emacs/el-get/org-mode/lisp/ob-awk hides 
/Applications/Emacs.app/Contents/Resources/lisp/org/ob-awk
/Users/hola/.var-emacs/el-get/org-mode/lisp/ob-asymptote hides 
/Applications/Emacs.app/Contents/Resources/lisp/org/ob-asymptote
/Users/hola/.var-emacs/el-get/seq/seq hides 
/Applications/Emacs.app/Contents/Resources/lisp/emacs-lisp/seq

Features:
(shadow sort mail-extr emacsbug sendmail ibuf-ext ibuffer vc-git
log-view vc-rcs grep vc-annotate vc vc-dispatcher parse-time vc-cvs
dired-aux misearch multi-isearch jka-compr eieio-opt speedbar sb-image
ezimage dframe disp-table tramp-cmds tramp-cache rng-nxml rng-valid
rng-loc rng-uri rng-parse nxml-parse rng-match rng-dt rng-util rng-pttrn
nxml-ns nxml-mode nxml-outln nxml-rap nxml-util nxml-glyph nxml-enc
xmltok dm-x avoid woman man dm-whitespace dm-tiddlywiki tid-mode
dm-smartparens smartparens-config smartparens-python smartparens-rust
smartparens-html smartparens dm-scheme geiser-mode geiser-xref
geiser-compile geiser-debug geiser-chibi geiser-mit geiser-chez
geiser-chicken geiser-racket geiser-guile info-look info geiser-repl
geiser-image geiser-company geiser-doc geiser-menu geiser-edit
geiser-completion geiser-autodoc geiser-eval geiser-connection tq
geiser-syntax scheme geiser-log geiser-popup view geiser-impl
geiser-custom geiser-base geiser-load geiser savehist dm-rust
flycheck-rust rust-mode python which-func dm-popup popup cperl-mode
dm-parens paren dm-org-mode org-mobile org-agenda org-element avl-tree
org org-macro org-footnote org-pcomplete org-list org-faces org-entities
noutline outline org-version ob-emacs-lisp ob ob-tangle org-src ob-ref
ob-lob ob-table ob-keys ob-exp ob-comint ob-core ob-eval org-compat
org-macs cal-menu calendar cal-loaddefs org-install org-loaddefs dm-objc
dm-misc ido finder-inf which-key dm-json json-mode json-reformat
json-snatcher js sgml-mode imenu dm-go go-eldoc go-mode url url-proxy
url-privacy url-expand url-methods url-history url-cookie url-domsuf
url-util mailcap find-file ffap thingatpt url-parse url-vars etags xref
project compile whitespace dm-git magit-obsolete magit-blame magit-stash
magit-bisect magit-remote magit-commit magit-sequence magit-notes
magit-worktree magit-branch magit-files magit-refs magit-status magit
magit-repos magit-apply magit-wip magit-log magit-diff smerge-mode
diff-mode magit-core magit-autorevert magit-process magit-margin
magit-mode magit-git crm magit-section magit-popup git-commit
magit-utils log-edit message rfc822 mml mml-sec epg mm-decode mm-bodies
mm-encode mail-parse rfc2231 rfc2047 rfc2045 ietf-drums mailabbrev
mail-utils gmm-utils mailheader pcvs-util add-log with-editor
async-bytecomp async tramp-sh tramp tramp-compat auth-source eieio
eieio-core gnus-util mm-util help-fns mail-prsvr password-cache
tramp-loaddefs trampver shell pcomplete comint ansi-color ring
format-spec advice dm-elisp dm-flycheck flycheck json map find-func rx
dash server assoc easy-mmode autorevert filenotify dired-x dm-clojure
dm-prettify-symbols dm-rainbow cl-macs rainbow-delimiters dm-paredit
paredit edmacro kmacro cc-mode cc-fonts cc-guess cc-menus cc-cmds
cc-styles cc-align cc-engine cc-vars cc-defs dm-path
exec-path-from-shell dm-el-get .loaddefs el-get el-get-autoloading
el-get-list-packages el-get-dependencies el-get-build el-get-status pp
el-get-methods el-get-fossil el-get-svn el-get-pacman el-get-github-zip
el-get-github-tar el-get-http-zip el-get-http-tar el-get-hg el-get-go
el-get-git-svn el-get-fink el-get-emacswiki el-get-http el-get-notify
el-get-emacsmirror el-get-github el-get-git el-get-elpa package
epg-config seq byte-opt el-get-darcs el-get-cvs el-get-bzr el-get-brew
el-get-builtin el-get-apt-get el-get-recipes el-get-byte-compile subr-x
el-get-custom cl-seq el-get-core autoload lisp-mnt bytecomp byte-compile
cl-extra help-mode easymenu cconv cl gv cl-loaddefs pcase cl-lib dired
dm-functions dm-look time-date mule-util tooltip eldoc electric uniquify
ediff-hook vc-hooks lisp-float-type mwheel ns-win ucs-normalize
term/common-win tool-bar dnd fontset image regexp-opt fringe
tabulated-list newcomment elisp-mode lisp-mode prog-mode register page
menu-bar rfn-eshadow timer select scroll-bar mouse jit-lock font-lock
syntax facemenu font-core frame cl-generic cham georgian utf-8-lang
misc-lang vietnamese tibetan thai tai-viet lao korean japanese eucjp-ms
cp51932 hebrew greek romanian slovak czech european ethiopic indian
cyrillic chinese charscript case-table epa-hook jka-cmpr-hook help
simple abbrev minibuffer cl-preloaded nadvice loaddefs button faces
cus-face macroexp files text-properties overlay sha1 md5 base64 format
env code-pages mule custom widget hashtable-print-readable backquote
dbusbind kqueue cocoa ns multi-tty make-network-process emacs)

Memory information:
((conses 16 578417 497407)
  (symbols 48 51155 64)
  (miscs 40 520 3222)
  (strings 32 130093 345819)
  (string-bytes 1 4086693)
  (vectors 16 81276)
  (vector-slots 8 1890698 414414)
  (floats 8 610 3479)
  (intervals 56 11420 10783)
  (buffers 976 75))
	

bug#25529: diagnosis and one approach to a fix

[Tom Tromey]

I think the problem here is that js-syntax-propertize-regexp is using
parse-partial-sexp to try to parse the regexp literal.  However, this
gets confused by the construct [^[], because it thinks that the brackets
aren't matched.

I don't think there is a way to teach parse-partial-sexp that "^" is a
quote only in this one specific instance.

One possible fix here would be to change this function to do a simple
parse of the regexp literal.  I think it would only really have to
handle parsing bracket syntax and looking for the terminating "/".  The
current code also looks for balanced parens, but I don't think this is
actually needed.

Tom

bug#25529: diagnosis and one approach to a fix

[Tom Tromey]

Tom> I don't think there is a way to teach parse-partial-sexp that "^" is a
Tom> quote only in this one specific instance.

This doesn't actually make sense anyway.
The "^" is a distraction, as this is valid:

    let x = /[[]/;

Tom> One possible fix here would be to change this function to do a simple
Tom> parse of the regexp literal.  I think it would only really have to
Tom> handle parsing bracket syntax and looking for the terminating "/".  The
Tom> current code also looks for balanced parens, but I don't think this is
Tom> actually needed.

I've appended a patch implementing this idea.

Tom

diff --git a/lisp/progmodes/js.el b/lisp/progmodes/js.el
index e42e014..083cef9 100644
--- a/lisp/progmodes/js.el
+++ b/lisp/progmodes/js.el
@@ -1698,18 +1698,30 @@ js-syntax-propertize-regexp
   (let ((ppss (syntax-ppss)))
     (when (eq (nth 3 ppss) ?/)
       ;; A /.../ regexp.
-      (while
-          (when (re-search-forward "\\(?:\\=\\|[^\\]\\)\\(?:\\\\\\\\\\)*/"
-                                   end 'move)
-            (if (nth 1 (with-syntax-table
-                           js--syntax-propertize-regexp-syntax-table
-                         (let ((parse-sexp-lookup-properties nil))
-                           (parse-partial-sexp (nth 8 ppss) (point)))))
-                ;; A / within a character class is not the end of a regexp.
-                t
-              (put-text-property (1- (point)) (point)
-                                 'syntax-table (string-to-syntax "\"/"))
-              nil))))))
+      (let ((keep-going t)
+            (backslash nil)
+            (in-bracket nil))
+        (while keep-going
+          (forward-char)
+          (let ((c (char-after)))
+            (cond
+             (backslash
+              (setq backslash nil))
+             ((eq c ?\\)
+              (setq backslash t))
+             ((eq c ?\[)
+              ;; Note that inside a bracket we can see another unescaped open
+              ;; bracket.
+              (setq in-bracket t))
+             ((eq c ?\])
+              (setq in-bracket nil))
+             ((eq c ?/)
+              (unless in-bracket
+                ;; We're done.
+                (setq keep-going nil)
+                (put-text-property
+                 (point) (1+ (point))
+                 'syntax-table (string-to-syntax "\"/")))))))))))
 
 (defun js-syntax-propertize (start end)
   ;; JavaScript allows immediate regular expression objects, written /.../.

bug#25529: diagnosis and one approach to a fix

[Tom Tromey]

Tom> I've appended a patch implementing this idea.

Another way is to use a regexp, see appended.
I'll try to write a test for this soon.

Tom

diff --git a/lisp/progmodes/js.el b/lisp/progmodes/js.el
index e42e014..2b4ae8e 100644
--- a/lisp/progmodes/js.el
+++ b/lisp/progmodes/js.el
@@ -1694,22 +1694,32 @@ js--syntax-propertize-regexp-syntax-table
     (modify-syntax-entry ?\\ "\\" st)
     st))
 
+(defconst js--syntax-propertize-regexp-regexp
+  (rx
+   ;; Start of regexp.
+   "/"
+   (0+ (or
+        ;; Match characters outside of a character class.
+        (not (any ?\[ ?/ ?\\))
+        ;; Match backslash quoted characters.
+        (and "\\" not-newline)
+        ;; Match character class.
+        (and
+         "["
+         (0+ (or
+              (not (any ?\] ?\\))
+              (and "\\" not-newline)))
+         "]")))
+   (group "/")))
+
 (defun js-syntax-propertize-regexp (end)
   (let ((ppss (syntax-ppss)))
     (when (eq (nth 3 ppss) ?/)
       ;; A /.../ regexp.
-      (while
-          (when (re-search-forward "\\(?:\\=\\|[^\\]\\)\\(?:\\\\\\\\\\)*/"
-                                   end 'move)
-            (if (nth 1 (with-syntax-table
-                           js--syntax-propertize-regexp-syntax-table
-                         (let ((parse-sexp-lookup-properties nil))
-                           (parse-partial-sexp (nth 8 ppss) (point)))))
-                ;; A / within a character class is not the end of a regexp.
-                t
-              (put-text-property (1- (point)) (point)
-                                 'syntax-table (string-to-syntax "\"/"))
-              nil))))))
+      (goto-char (nth 8 ppss))
+      (when (looking-at js--syntax-propertize-regexp-regexp)
+        (put-text-property (match-beginning 1) (match-end 1)
+                           'syntax-table (string-to-syntax "\"/"))))))
 
 (defun js-syntax-propertize (start end)
   ;; JavaScript allows immediate regular expression objects, written /.../.

bug#25529: diagnosis and one approach to a fix

[Tom Tromey]

Tom> Another way is to use a regexp, see appended.
Tom> I'll try to write a test for this soon.

Now with a test.

Dmitry, I'd appreciate your comments on this.

Tom

commit 0841c586b5a933773e770579b4a9cd6f86b2dcf7
Author: Tom Tromey <tom@tromey.com>
Date:   Sun Feb 5 11:40:18 2017 -0700

    Recognize JS regexp literals more correctly
    
    * lisp/progmodes/js.el (js--syntax-propertize-regexp-regexp): New
    constant.
    (js-syntax-propertize-regexp): Use it.
    * test/lisp/progmodes/js-tests.el (js-mode-regexp-syntax-bug-25529):
    New test.

diff --git a/lisp/progmodes/js.el b/lisp/progmodes/js.el
index e42e014..145aa6f 100644
--- a/lisp/progmodes/js.el
+++ b/lisp/progmodes/js.el
@@ -1694,22 +1694,33 @@ js--syntax-propertize-regexp-syntax-table
     (modify-syntax-entry ?\\ "\\" st)
     st))
 
+(defconst js--syntax-propertize-regexp-regexp
+  (rx
+   ;; Start of regexp.
+   "/"
+   (0+ (or
+        ;; Match characters outside of a character class.
+        (not (any ?\[ ?/ ?\\))
+        ;; Match backslash quoted characters.
+        (and "\\" not-newline)
+        ;; Match character class.
+        (and
+         "["
+         (0+ (or
+              (not (any ?\] ?\\))
+              (and "\\" not-newline)))
+         "]")))
+   (group "/"))
+  "Regular expression matching the body of a JavaScript regexp literal.")
+
 (defun js-syntax-propertize-regexp (end)
   (let ((ppss (syntax-ppss)))
     (when (eq (nth 3 ppss) ?/)
       ;; A /.../ regexp.
-      (while
-          (when (re-search-forward "\\(?:\\=\\|[^\\]\\)\\(?:\\\\\\\\\\)*/"
-                                   end 'move)
-            (if (nth 1 (with-syntax-table
-                           js--syntax-propertize-regexp-syntax-table
-                         (let ((parse-sexp-lookup-properties nil))
-                           (parse-partial-sexp (nth 8 ppss) (point)))))
-                ;; A / within a character class is not the end of a regexp.
-                t
-              (put-text-property (1- (point)) (point)
-                                 'syntax-table (string-to-syntax "\"/"))
-              nil))))))
+      (goto-char (nth 8 ppss))
+      (when (looking-at js--syntax-propertize-regexp-regexp)
+        (put-text-property (match-beginning 1) (match-end 1)
+                           'syntax-table (string-to-syntax "\"/"))))))
 
 (defun js-syntax-propertize (start end)
   ;; JavaScript allows immediate regular expression objects, written /.../.
diff --git a/test/lisp/progmodes/js-tests.el b/test/lisp/progmodes/js-tests.el
index 7cb737c..d1a8db0 100644
--- a/test/lisp/progmodes/js-tests.el
+++ b/test/lisp/progmodes/js-tests.el
@@ -99,6 +99,15 @@
     (forward-line)
     (should (looking-at " \\* test"))))
 
+(ert-deftest js-mode-regexp-syntax-bug-25529 ()
+  (with-temp-buffer
+    (js-mode)
+    (insert "let x = /[^[]/;\n")
+    (save-excursion (insert "something();\n"))
+    ;; The failure mode was that the regexp literal was not
+    ;; recognized, causing the next line to be given string syntax.
+    (should-not (nth 3 (syntax-ppss)))))
+
 (provide 'js-tests)
 
 ;;; js-tests.el ends here

bug#25529: diagnosis and one approach to a fix

[Dmitry Gutov]

Hey Tom,

On 05.02.2017 20:43, Tom Tromey wrote:
> Tom> Another way is to use a regexp, see appended.
> Tom> I'll try to write a test for this soon.
> 
> Now with a test.

This is very good, thank you. Here's just one example I've come so far 
that works with the old code, but not with yours:

let x = /[/]/;

And that's probably just because the new js-syntax-propertize-regexp 
doesn't move point forward after matching (the current one leaves it 
after the closing slash), because the regexp in js-syntax-propertize is 
too primitive.

We could try to combine the regexps together, though. Then we'd move the 
syntax-ppss status check inside the rule for the opening delimiter, and 
use a simple one for the closer.

> +  "Regular expression matching the body of a JavaScript regexp literal.")

Maybe remove "the body of". The regexp matches the delimiters as well.

Thanks!

bug#25529: diagnosis and one approach to a fix

[Tom Tromey]

Dmitry> This is very good, thank you. Here's just one example I've come so far
Dmitry> that works with the old code, but not with yours:
Dmitry> let x = /[/]/;
Dmitry> And that's probably just because the new js-syntax-propertize-regexp
Dmitry> doesn't move point forward after matching

Yep, adding a goto-char at the end of the function fixed this.
Thanks for pointing that out; I'll add a test case for it.

Dmitry> We could try to combine the regexps together, though. Then we'd move
Dmitry> the syntax-ppss status check inside the rule for the opening
Dmitry> delimiter, and use a simple one for the closer.

I think I understand combining the regexps.  But what does using a simple
regexp for the closer mean?

Also, what is the purpose of the first call to
js-syntax-propertize-regexp in js-syntax-propertize?

Tom

bug#25529: diagnosis and one approach to a fix

[Dmitry Gutov]

On 06.02.2017 19:27, Tom Tromey wrote:

> Yep, adding a goto-char at the end of the function fixed this.
> Thanks for pointing that out; I'll add a test case for it.

Thanks.

> I think I understand combining the regexps.  But what does using a simple
> regexp for the closer mean?

Simple handler (not regexp), for the group corresponding to the closer 
(what's called a HIGHLIGHTn in syntax-propertize-rules docstring). The 
string "\"/", probably.

> Also, what is the purpose of the first call to
> js-syntax-propertize-regexp in js-syntax-propertize?

I'm not sure. Normally, that helps deal with multiline literals, but 
regexps in JavaScript are single-line. Maybe Stefan remembers.

Or maybe it was a result of confusion: the code deleted in 6cd18349 
includes this mistaken sentence: "XXX: Javascript can continue a regexp 
literal across lines so long as the newline is escaped with \."

bug#25529: diagnosis and one approach to a fix

[Tom Tromey]

>>>>> "Dmitry" == Dmitry Gutov <dgutov@yandex.ru> writes:

>> Also, what is the purpose of the first call to
>> js-syntax-propertize-regexp in js-syntax-propertize?

Dmitry> I'm not sure. Normally, that helps deal with multiline literals, but
Dmitry> regexps in JavaScript are single-line. Maybe Stefan remembers.

Based on discussion in the multi-mode thread, I'm thinking perhaps it
handles the case where the start of the regexp literal was syntax-ified
but not the end.

Tom

bug#25529: diagnosis and one approach to a fix

[Dmitry Gutov]

On 07.02.2017 15:07, Tom Tromey wrote:

> Dmitry> I'm not sure. Normally, that helps deal with multiline literals, but
> Dmitry> regexps in JavaScript are single-line. Maybe Stefan remembers.
> 
> Based on discussion in the multi-mode thread, I'm thinking perhaps it
> handles the case where the start of the regexp literal was syntax-ified
> but not the end.

That's the idea, but syntax-propertize runs 
syntax-propertize-extend-region-functions, which extends to the 
beginning of the line of the edited region.

So if we made an edit inside a line containing a regexp, START should be 
at the beginning of that line.

There could be some edge cases, though.

bug#25529: diagnosis and one approach to a fix

[Stefan Monnier]

>> Also, what is the purpose of the first call to
>> js-syntax-propertize-regexp in js-syntax-propertize?

It tests if the start point is inside a regexp, and if so, does what's
needed with it and moves to its end since the rest of the
js-syntax-propertize code assumes we're outside of a regexp.

> I'm not sure. Normally, that helps deal with multiline literals, but regexps
> in JavaScript are single-line. Maybe Stefan remembers.

I think I just didn't want to presume that the start point is never
inside a regexp (not sure if I was aware that Javascrit doesn't support
multiline regexps, but in any case I'd be surprised if there isn't some
Javascript implementation somewhere which does).  IOW I just reproduced
the pattern I've used in many other syntax-propertize-functions.
Maybe it's not needed, but it doesn't hurt.


        Stefan

bug#25529: diagnosis and one approach to a fix

[Tom Tromey]

>> I think I understand combining the regexps.  But what does using a simple
>> regexp for the closer mean?

Dmitry> Simple handler (not regexp), for the group corresponding to the closer
Dmitry> (what's called a HIGHLIGHTn in syntax-propertize-rules docstring). The
Dmitry> string "\"/", probably.

Ok, I looked at this a bit.  My idea was to change js-syntax-propertize
to incorporate the new regexp I wrote.

However, after looking at this a bit, I think it's actually better the
way it is.  The reason is that the existing regexp in
js-syntax-propertize ends with "[^/*]" -- to prevent recognizing
"let x = /* comment */" as if it were using a regexp literal.

While it's possible to add this to the new regexp, it's kind of ugly,
since the regexp has to duplicate much of the body of the regexp just to
exclude "*" as the first character.

Conversely I don't think there's a drawback to the current approach.

Here's the latest version of my patch.  Let me know what you think.

Tom

commit 2518a9ca4ab59dbbfedc42a023267ae78476fc2e
Author: Tom Tromey <tom@tromey.com>
Date:   Sun Feb 5 11:40:18 2017 -0700

    Recognize JS regexp literals more correctly
    
    * lisp/progmodes/js.el (js--syntax-propertize-regexp-regexp): New
    constant.
    (js-syntax-propertize-regexp): Use it.  Remove "end" argument.
    (js--syntax-propertize-regexp-syntax-table): Remove.
    (js-syntax-propertize): Update.
    * test/lisp/progmodes/js-tests.el (js-mode-regexp-syntax-bug-25529):
    New test.

diff --git a/lisp/progmodes/js.el b/lisp/progmodes/js.el
index e42e014..3d9c1b4 100644
--- a/lisp/progmodes/js.el
+++ b/lisp/progmodes/js.el
@@ -1687,34 +1687,39 @@ js--font-lock-keywords
                                    js--font-lock-keywords-3)
   "Font lock keywords for `js-mode'.  See `font-lock-keywords'.")
 
-(defconst js--syntax-propertize-regexp-syntax-table
-  (let ((st (make-char-table 'syntax-table (string-to-syntax "."))))
-    (modify-syntax-entry ?\[ "(]" st)
-    (modify-syntax-entry ?\] ")[" st)
-    (modify-syntax-entry ?\\ "\\" st)
-    st))
-
-(defun js-syntax-propertize-regexp (end)
+(defconst js--syntax-propertize-regexp-regexp
+  (rx
+   ;; Start of regexp.
+   "/"
+   (0+ (or
+        ;; Match characters outside of a character class.
+        (not (any ?\[ ?/ ?\\))
+        ;; Match backslash quoted characters.
+        (and "\\" not-newline)
+        ;; Match character class.
+        (and
+         "["
+         (0+ (or
+              (not (any ?\] ?\\))
+              (and "\\" not-newline)))
+         "]")))
+   (group "/"))
+  "Regular expression matching a JavaScript regexp literal.")
+
+(defun js-syntax-propertize-regexp ()
   (let ((ppss (syntax-ppss)))
     (when (eq (nth 3 ppss) ?/)
       ;; A /.../ regexp.
-      (while
-          (when (re-search-forward "\\(?:\\=\\|[^\\]\\)\\(?:\\\\\\\\\\)*/"
-                                   end 'move)
-            (if (nth 1 (with-syntax-table
-                           js--syntax-propertize-regexp-syntax-table
-                         (let ((parse-sexp-lookup-properties nil))
-                           (parse-partial-sexp (nth 8 ppss) (point)))))
-                ;; A / within a character class is not the end of a regexp.
-                t
-              (put-text-property (1- (point)) (point)
-                                 'syntax-table (string-to-syntax "\"/"))
-              nil))))))
+      (goto-char (nth 8 ppss))
+      (when (looking-at js--syntax-propertize-regexp-regexp)
+        (put-text-property (match-beginning 1) (match-end 1)
+                           'syntax-table (string-to-syntax "\"/"))
+        (goto-char (match-end 0))))))
 
 (defun js-syntax-propertize (start end)
   ;; JavaScript allows immediate regular expression objects, written /.../.
   (goto-char start)
-  (js-syntax-propertize-regexp end)
+  (js-syntax-propertize-regexp)
   (funcall
    (syntax-propertize-rules
     ;; Distinguish /-division from /-regexp chars (and from /-comment-starter).
@@ -1736,7 +1741,7 @@ js-syntax-propertize
                            (eval-when-compile (append "=({[,:;" '(nil))))))
            (put-text-property (match-beginning 1) (match-end 1)
                               'syntax-table (string-to-syntax "\"/"))
-           (js-syntax-propertize-regexp end)))))
+           (js-syntax-propertize-regexp)))))
     ("\\`\\(#\\)!" (1 "< b")))
    (point) end))
 
diff --git a/test/lisp/progmodes/js-tests.el b/test/lisp/progmodes/js-tests.el
index 7cb737c..d61f084 100644
--- a/test/lisp/progmodes/js-tests.el
+++ b/test/lisp/progmodes/js-tests.el
@@ -23,6 +23,7 @@
 
 (require 'ert)
 (require 'js)
+(require 'syntax)
 
 (ert-deftest js-mode-fill-bug-19399 ()
   (with-temp-buffer
@@ -99,6 +100,22 @@
     (forward-line)
     (should (looking-at " \\* test"))))
 
+(ert-deftest js-mode-regexp-syntax-bug-25529 ()
+  (dolist (regexp-contents '("[^[]"
+                             "[/]"
+                             ;; A comment with the regexp on the next
+                             ;; line.
+                             "*comment*/\n/regexp"))
+    (with-temp-buffer
+      (js-mode)
+      (insert "let x = /" regexp-contents "/;\n")
+      (save-excursion (insert "something();\n"))
+      ;; The failure mode was that the regexp literal was not
+      ;; recognized, causing the next line to be given string syntax;
+      ;; but check for comment syntax as well to prevent an
+      ;; implementation not recognizing the comment example.
+      (should-not (syntax-ppss-context (syntax-ppss))))))
+
 (provide 'js-tests)
 
 ;;; js-tests.el ends here

bug#25529: diagnosis and one approach to a fix

[Stefan Monnier]

> -(defun js-syntax-propertize-regexp (end)
> +(defun js-syntax-propertize-regexp ()

Removing the `end` argument is fundamentally wrong.  It's OK to play it
fast-and-loose and ignore that argument, but the function *should* keep
the buffer untouched after `end`.

> +        (put-text-property (match-beginning 1) (match-end 1)
                                                  ^^^^^^^^^^^^^
                                              (min end (match-end 1))

should do it.


        Stefan

bug#25529: diagnosis and one approach to a fix

[Tom Tromey]

>>>>> "Stefan" == Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> -(defun js-syntax-propertize-regexp (end)
>> +(defun js-syntax-propertize-regexp ()

Stefan> Removing the `end` argument is fundamentally wrong.  It's OK to play it
Stefan> fast-and-loose and ignore that argument, but the function *should* keep
Stefan> the buffer untouched after `end`.

I thought it was ok because the region is always extended to the end of
the line, and I think this regexp can't span lines.

>> +        (put-text-property (match-beginning 1) (match-end 1)
Stefan>                                                   ^^^^^^^^^^^^^
Stefan>                                               (min end (match-end 1))
Stefan> should do it.

If this change is still needed, should the final goto-char also be
treated this way?

Tom

bug#25529: diagnosis and one approach to a fix

[Stefan Monnier]

Stefan> Removing the `end` argument is fundamentally wrong.  It's OK to play it
Stefan> fast-and-loose and ignore that argument, but the function *should* keep
Stefan> the buffer untouched after `end`.

> I thought it was ok because the region is always extended to the end of
> the line, and I think this regexp can't span lines.

Why come up with complex semantic arguments when a simple `max` ensures
the right behavior regardless of other assumptions?

> If this change is still needed, should the final goto-char also be
> treated this way?

It's not needed there, no: the important part is to avoid modifying the
buffer after `end` (although, as I said, it's not terribly bad if you
do it: you'd only get burned by it in very unusual cases).


        Stefan

bug#25529: diagnosis and one approach to a fix

[Tom Tromey]

>> I thought it was ok because the region is always extended to the end of
>> the line, and I think this regexp can't span lines.

Stefan> Why come up with complex semantic arguments when a simple `max` ensures
Stefan> the right behavior regardless of other assumptions?

I had already done it, so inertia.

I'm making the change, but I think the fix has to look like this:

      (when (and (looking-at js--syntax-propertize-regexp-regexp)
                 ;; Don't touch text after END.
                 (<= (match-end 1) end))
        (put-text-property (match-beginning 1) (match-end 1)
                           'syntax-table (string-to-syntax "\"/"))
        (goto-char (match-end 0))))))

... the reason being, in the new regexp match #1 is just the terminating
"/", which is the only character whose syntax we want to modify here.

Let me know what you think.

Tom

bug#25529: diagnosis and one approach to a fix

[Stefan Monnier]

> I'm making the change, but I think the fix has to look like this:

>       (when (and (looking-at js--syntax-propertize-regexp-regexp)
>                  ;; Don't touch text after END.
>                  (<= (match-end 1) end))
>         (put-text-property (match-beginning 1) (match-end 1)
>                            'syntax-table (string-to-syntax "\"/"))
>         (goto-char (match-end 0))))))

> ... the reason being, in the new regexp match #1 is just the terminating
> "/", which is the only character whose syntax we want to modify here.

Ah, I had misunderstood the code.  Then yes.


        Stefan

bug#25529: diagnosis and one approach to a fix

[Tom Tromey]

Stefan> Ah, I had misunderstood the code.  Then yes.

Thanks.  I'll wait a little while in case Dmitry has comments, then I'll
check it in.

Tom

bug#25529: diagnosis and one approach to a fix

[Dmitry Gutov]

On 11.02.2017 19:13, Tom Tromey wrote:

> I'll wait a little while in case Dmitry has comments, then I'll
> check it in.

LGTM, thanks!

bug#25529: done

[Tom Tromey]

I've checked in the patch.

Tom