bug#11953: 23.4; tex-mode does not fontify \url{foo/$(bar)} correctly

bug#11953: 23.4; tex-mode does not fontify \url{foo/$(bar)} correctly

[Norman Ramsey]

   The latex-mode fontifier is not aware that the (nonstandard)
   \url{...} and \path{...} commands should have their contents treated
   as if verbatim.  Here's an example that sends fontification around
   the bend (key text on middle line):

          To fix this, remove the \url{sn9c102.ko} from where it appears in
          \url{/lib/modules/$(uname -r)}, 
          and install the appropriate \url{gspca-modules} package.

   N.B. The mode appears aware of the \url command but not the \path
   command.  The two commands are provided by the same package and they
   have similar semantics: the arguments may contain characters that are
   'special' to {\TeX}, but these characters are treated as if verbatim.



In GNU Emacs 23.4.1 (i486-pc-linux-gnu, GTK+ Version 2.24.10)
 of 2012-04-07 on biber, modified by Debian
Windowing system distributor `The X.Org Foundation', version 11.0.11201902
configured using `configure  '--build' 'i486-linux-gnu' '--build' 'i486-linux-gnu' '--prefix=/usr' '--sharedstatedir=/var/lib' '--libexecdir=/usr/lib' '--localstatedir=/var/lib' '--infodir=/usr/share/info' '--mandir=/usr/share/man' '--with-pop=yes' '--enable-locallisppath=/etc/emacs23:/etc/emacs:/usr/local/share/emacs/23.4/site-lisp:/usr/local/share/emacs/site-lisp:/usr/share/emacs/23.4/site-lisp:/usr/share/emacs/site-lisp' '--with-crt-dir=/usr/lib/i386-linux-gnu' '--with-x=yes' '--with-x-toolkit=gtk' '--with-toolkit-scroll-bars' 'build_alias=i486-linux-gnu' 'CFLAGS=-g -O2 -fstack-protector --param=ssp-buffer-size=4 -Wformat -Wformat-security -Werror=format-security -Wall -DDEBIAN -O2' 'CPPFLAGS=-D_FORTIFY_SOURCE=2''

Important settings:
  value of $LC_ALL: nil
  value of $LC_COLLATE: nil
  value of $LC_CTYPE: nil
  value of $LC_MESSAGES: nil
  value of $LC_MONETARY: nil
  value of $LC_NUMERIC: nil
  value of $LC_TIME: nil
  value of $LANG: nil
  value of $XMODIFIERS: nil
  locale-coding-system: nil
  default enable-multibyte-characters: t

Major mode: Shell-script

Minor modes in effect:
  shell-dirtrack-mode: t
  tooltip-mode: t
  mouse-wheel-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  transient-mark-mode: t

Recent input:
y C-g M-x r e p o <tab> r <tab> <return>

Recent messages:
Setting up indent for shell type pdksh
setting up indent stuff
Indentation variables are now local.
Indentation setup for shell type pdksh
Setting up indent for shell type pdksh
setting up indent stuff
Indentation variables are now local.
Indentation setup for shell type pdksh
Quit
Making completion list...

Load-path shadows:
~nr/emacs/sml-mode hides /usr/share/emacs23/site-lisp/sml-mode/sml-mode
~nr/emacs/sml-proc hides /usr/share/emacs23/site-lisp/sml-mode/sml-proc
/usr/share/emacs/site-lisp/haskell-mode/haskell-indent hides /usr/share/emacs/23.4/site-lisp/haskell-mode/haskell-indent
/usr/share/emacs/site-lisp/haskell-mode/haskell-indentation hides /usr/share/emacs/23.4/site-lisp/haskell-mode/haskell-indentation
/usr/share/emacs/site-lisp/haskell-mode/haskell-c hides /usr/share/emacs/23.4/site-lisp/haskell-mode/haskell-c
/usr/share/emacs/site-lisp/haskell-mode/haskell-hugs hides /usr/share/emacs/23.4/site-lisp/haskell-mode/haskell-hugs
/usr/share/emacs/site-lisp/haskell-mode/haskell-font-lock hides /usr/share/emacs/23.4/site-lisp/haskell-mode/haskell-font-lock
/usr/share/emacs/site-lisp/haskell-mode/haskell-mode hides /usr/share/emacs/23.4/site-lisp/haskell-mode/haskell-mode
/usr/share/emacs/site-lisp/haskell-mode/haskell-doc hides /usr/share/emacs/23.4/site-lisp/haskell-mode/haskell-doc
/usr/share/emacs/site-lisp/haskell-mode/haskell-decl-scan hides /usr/share/emacs/23.4/site-lisp/haskell-mode/haskell-decl-scan
/usr/share/emacs/site-lisp/haskell-mode/haskell-site-file hides /usr/share/emacs/23.4/site-lisp/haskell-mode/haskell-site-file
/usr/share/emacs/site-lisp/haskell-mode/haskell-simple-indent hides /usr/share/emacs/23.4/site-lisp/haskell-mode/haskell-simple-indent
/usr/share/emacs/site-lisp/haskell-mode/haskell-ghci hides /usr/share/emacs/23.4/site-lisp/haskell-mode/haskell-ghci
/usr/share/emacs/site-lisp/haskell-mode/inf-haskell hides /usr/share/emacs/23.4/site-lisp/haskell-mode/inf-haskell
/usr/share/emacs/site-lisp/haskell-mode/haskell-cabal hides /usr/share/emacs/23.4/site-lisp/haskell-mode/haskell-cabal
~nr/emacs/noweb-mode hides /usr/share/emacs/site-lisp/noweb-mode
/usr/share/emacs/23.4/site-lisp/debian-startup hides /usr/share/emacs/site-lisp/debian-startup
~nr/emacs/mwheel hides /usr/share/emacs/23.4/lisp/mwheel
~nr/emacs/longlines hides /usr/share/emacs/23.4/lisp/longlines
/usr/share/emacs23/site-lisp/dictionaries-common/ispell hides /usr/share/emacs/23.4/lisp/textmodes/ispell
/usr/share/emacs23/site-lisp/dictionaries-common/flyspell hides /usr/share/emacs/23.4/lisp/textmodes/flyspell

Features:
(shadow sort mail-extr message sendmail ecomplete rfc822 mml mml-sec
password-cache mm-decode mm-bodies mm-encode mailcap mail-parse rfc2231
rfc2047 rfc2045 qp ietf-drums mailabbrev nnheader gnus-util netrc
time-date mm-util mail-prsvr gmm-utils wid-edit mailheader canlock sha1
hex-util hashcash mail-utils emacsbug help-mode view todoo noutline
outline cc-mode cc-fonts cc-menus cc-cmds cc-styles cc-align cc-engine
cc-vars cc-defs info caml warnings sh-script executable vc-cvs lua-mode
haskell-font-lock haskell-decl-scan haskell-doc haskell-indentation
haskell-indent haskell-mode byte-opt bytecomp byte-compile pp dired
flyspell ispell latexenc tex-mode shell vc-git vc-hg darcsum easymenu
add-log rect assoc filladapt scroll-in-place noweb derived multi-mode
advice help-fns advice-preload imenu compile comint ring desktop edmacro
kmacro mmm-auto mmm-vars mmm-compat regexp-opt cl cl-19 emacs-goodies-el
emacs-goodies-custom emacs-goodies-loaddefs easy-mmode tooltip
ediff-hook vc-hooks lisp-float-type mwheel x-win x-dnd font-setting
tool-bar dnd fontset image fringe lisp-mode register page menu-bar
rfn-eshadow timer select scroll-bar mldrag mouse jit-lock font-lock
syntax facemenu font-core frame cham georgian utf-8-lang misc-lang
vietnamese tibetan thai tai-viet lao korean japanese hebrew greek
romanian slovak czech european ethiopic indian cyrillic chinese
case-table epa-hook jka-cmpr-hook help simple abbrev loaddefs button
minibuffer faces cus-face files text-properties overlay md5 base64
format env code-pages mule custom widget hashtable-print-readable
backquote make-network-process dbusbind system-font-setting
font-render-setting gtk x-toolkit x multi-tty emacs)

bug#11953: 23.4; tex-mode does not fontify \url{foo/$(bar)} correctly

[Stefan Monnier]

>    The latex-mode fontifier is not aware that the (nonstandard)
>    \url{...} and \path{...} commands should have their contents treated
>    as if verbatim.  Here's an example that sends fontification around
>    the bend (key text on middle line):

>           To fix this, remove the \url{sn9c102.ko} from where it appears in
>           \url{/lib/modules/$(uname -r)}, 
>           and install the appropriate \url{gspca-modules} package.

>    N.B. The mode appears aware of the \url command but not the \path
>    command.  The two commands are provided by the same package and they
>    have similar semantics: the arguments may contain characters that are
>    'special' to {\TeX}, but these characters are treated as if verbatim.

I installed a patch in the trunk which fixes your example.
It does not really handle \url specially but instead changes the
handling of $ to be a bit more conservative.

So \url{foo$bar$baz} will still highlight "$bar$" with the tex-math
face, for example.  But a lone $ should not cause the whole rest of the
document to be mis-fontified any more, or at least much less often.
OTOH it will probably cause some valid $...$ math uses to fail to be
highlighted with the tex-math face.


        Stefan


--- lisp/textmodes/tex-mode.el	2012-07-11 23:13:41 +0000
+++ lisp/textmodes/tex-mode.el	2012-07-17 07:59:18 +0000
@@ -476,25 +476,28 @@
 		      '("input" "include" "includeonly" "bibliography"
 			"epsfig" "psfig" "epsf" "nofiles" "usepackage"
 			"documentstyle" "documentclass" "verbatiminput"
-			"includegraphics" "includegraphics*"
-			"url" "nolinkurl")
+			"includegraphics" "includegraphics*")
 		      t))
+           (verbish (regexp-opt '("url" "nolinkurl" "path") t))
 	   ;; Miscellany.
 	   (slash "\\\\")
 	   (opt " *\\(\\[[^]]*\\] *\\)*")
 	   ;; This would allow highlighting \newcommand\CMD but requires
 	   ;; adapting subgroup numbers below.
 	   ;; (arg "\\(?:{\\(\\(?:[^{}\\]+\\|\\\\.\\|{[^}]*}\\)+\\)\\|\\\\[a-z*]+\\)"))
-	   (arg "{\\(\\(?:[^{}\\]+\\|\\\\.\\|{[^}]*}\\)+\\)"))
-      (list
-       ;; display $$ math $$
-       ;; We only mark the match between $$ and $$ because the $$ delimiters
-       ;; themselves have already been marked (along with $..$) by syntactic
-       ;; fontification.  Also this is done at the very beginning so as to
-       ;; interact with the other keywords in the same way as $...$ does.
-       (list "\\$\\$\\([^$]+\\)\\$\\$" 1 'tex-math-face)
+           (inbraces-re (lambda (re)
+                          (concat "\\(?:[^{}\\]\\|\\\\.\\|" re "\\)")))
+	   (arg (concat "{\\(" (funcall inbraces-re "{[^}]*}") "+\\)")))
+      `( ;; Highlight $$math$$ and $math$.
+        ;; This is done at the very beginning so as to interact with the other
+        ;; keywords in the same way as comments and strings.
+        (,(concat "\\$\\$?\\(?:[^$\\{}]\\|\\\\.\\|{"
+                  (funcall inbraces-re
+                           (concat "{" (funcall inbraces-re "{[^}]*}") "*}"))
+                  "*}\\)+\\$?\\$")
+         (0 tex-math-face))
        ;; Heading args.
-       (list (concat slash headings "\\*?" opt arg)
+        (,(concat slash headings "\\*?" opt arg)
 	     ;; If ARG ends up matching too much (if the {} don't match, e.g.)
 	     ;; jit-lock will do funny things: when updating the buffer
 	     ;; the re-highlighting is only done locally so it will just
@@ -507,15 +510,17 @@
 	     ;; Using `keep' works around this un-intuitive behavior as well
 	     ;; as improves the behavior in the very rare case where you do
 	     ;; have a comment in ARG.
-	     3 'font-lock-function-name-face 'keep)
-       (list (concat slash "\\(?:provide\\|\\(?:re\\)?new\\)command\\** *\\(\\\\[A-Za-z@]+\\)")
-	     1 'font-lock-function-name-face 'keep)
+         3 font-lock-function-name-face keep)
+        (,(concat slash "\\(?:provide\\|\\(?:re\\)?new\\)command\\** *\\(\\\\[A-Za-z@]+\\)")
+         1 font-lock-function-name-face keep)
        ;; Variable args.
-       (list (concat slash variables " *" arg) 2 'font-lock-variable-name-face)
+        (,(concat slash variables " *" arg) 2 font-lock-variable-name-face)
        ;; Include args.
-       (list (concat slash includes opt arg) 3 'font-lock-builtin-face)
+        (,(concat slash includes opt arg) 3 font-lock-builtin-face)
+        ;; Verbatim-like args.
+        (,(concat slash verbish opt arg) 3 'tex-verbatim)
        ;; Definitions.  I think.
-       '("^[ \t]*\\\\def *\\\\\\(\\(\\w\\|@\\)+\\)"
+        ("^[ \t]*\\\\def *\\\\\\(\\(\\w\\|@\\)+\\)"
 	 1 font-lock-function-name-face))))
   "Subdued expressions to highlight in TeX modes.")
 
@@ -629,7 +634,7 @@
 	     (1 (tex-font-lock-suscript (match-beginning 0)) append))))
   "Experimental expressions to highlight in TeX modes.")
 
-(defvar tex-font-lock-keywords tex-font-lock-keywords-1
+(defconst tex-font-lock-keywords tex-font-lock-keywords-1
   "Default expressions to highlight in TeX modes.")
 
 (defvar tex-verbatim-environments
@@ -1219,7 +1224,7 @@
   (set (make-local-variable 'font-lock-defaults)
        '((tex-font-lock-keywords tex-font-lock-keywords-1
 	  tex-font-lock-keywords-2 tex-font-lock-keywords-3)
-	 nil nil ((?$ . "\"")) nil
+	 nil nil nil nil
 	 ;; Who ever uses that anyway ???
 	 (font-lock-mark-block-function . mark-paragraph)
 	 (font-lock-syntactic-face-function