bug#19455: 24.4; sh-mode highlighting breaks with ZSH glob modifiers

bug#19455: 24.4; sh-mode highlighting breaks with ZSH glob modifiers

[Jonathan H]

[-- Attachment #1: Type: text/plain, Size: 7443 bytes --]

In ZSH globs can have optional modifiers before globs (as well as
after). Usually, the format is (#<letter here>)<glob>. The leading
pound symbol confuses sh-mode syntax highlighting, making it appear to
be a comment. As these modifiers usually appear in a conditional
expression (they are used for group capture), they also break
indentation.

I've attached a file that demonstrates this.

In GNU Emacs 24.4.1 (x86_64-unknown-linux-gnu, GTK+ Version 3.14.3)
 of 2014-10-21 on bitzer.hoetzel.info
Windowing system distributor `The Cygwin/X Project', version 11.0.11602000
Configured using:
 `configure --prefix=/usr --sysconfdir=/etc --libexecdir=/usr/lib
 --localstatedir=/var --with-x-toolkit=gtk3 --with-xft
 'CFLAGS=-march=x86-64 -mtune=generic -O2 -pipe -fstack-protector-strong
 --param=ssp-buffer-size=4' CPPFLAGS=-D_FORTIFY_SOURCE=2
 LDFLAGS=-Wl,-O1,--sort-common,--as-needed,-z,relro'

Important settings:
  value of $LANG: en_US.UTF-8
  locale-coding-system: utf-8-unix

Major mode: λ

Minor modes in effect:
  ido-vertical-mode: t
  flx-ido-mode: t
  minibuffer-depth-indicate-mode: t
  magit-auto-revert-mode: t
  diff-auto-refine-mode: t
  shell-dirtrack-mode: t
  helm-match-plugin-mode: t
  helm-occur-match-plugin-mode: t
  aggressive-indent-mode: t
  global-flycheck-mode: t
  flyspell-mode: t
  ido-ubiquitous-mode: t
  auto-indent-global-mode: t
  auto-indent-mode: t
  global-auto-complete-mode: t
  auto-complete-mode: t
  cua-mode: t
  adaptive-wrap-prefix-mode: t
  show-paren-mode: t
  rainbow-delimiters-mode: t
  smartparens-global-mode: t
  smartparens-mode: t
  linum-mode: t
  global-evil-surround-mode: t
  evil-surround-mode: t
  evil-leader-mode: t
  evil-mode: t
  global-undo-tree-mode: t
  undo-tree-mode: t
  evil-local-mode: t
  savehist-mode: t
  global-hl-line-mode: t
  delete-selection-mode: t
  tooltip-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  column-number-mode: t
  line-number-mode: t
  global-visual-line-mode: t
  visual-line-mode: t
  transient-mark-mode: t

Recent input:
<help-echo> <select-window> <help-echo> <select-window>
<help-echo> <select-window> <select-window> <select-window>
<help-echo> <down-mouse-1> <mouse-movement> <mouse-1>
M-x r e p o r t - e m a <return>

Recent messages:
Loading /home/pythonnut/.emacs.d/.mc-lists.el (source)...done
idle-require: require windmove
idle-require: require framemove
Loading easy-kill...done
idle-require: require flx-isearch
idle-require: require whole-line-or-region
Loading smex...done
idle-require: require psvn
idle-require finished
(No files need saving)

Load-path shadows:
~/.emacs.d/personal/ws-butler hides
/home/pythonnut/.emacs.d/elpa/ws-butler-20140816.1448/ws-butler
~/.emacs.d/personal/edmacro hides /usr/share/emacs/24.4/lisp/edmacro

Features:
(shadow sort mail-extr emacsbug sendmail ido-vertical-mode flx-ido
mb-depth psvn smex whole-line-or-region flx-isearch flx easy-kill
framemove multiple-cursors mc-hide-unmatched-lines-mode
mc-separate-operations rectangular-region-mode mc-mark-pop mc-mark-more
mc-cycle-cursors mc-edit-lines multiple-cursors-core magit-filenotify
icicles icicles-mode icicles-cmd2 icicles-cmd1 icicles-mcmd icicles-fn
icicles-var icicles-opt cus-theme cus-edit wid-edit hexrgb icicles-face
helm-ag helm-semantic helm-imenu imenu semantic/util-modes semantic/util
semantic semantic/tag semantic/lex semantic/fw mode-local cedet
helm-projectile helm-config async-bytecomp async helm-aliases
magit-key-mode magit view epa derived epg diff-git diff-mode autorevert
filenotify git-rebase-mode git-commit-mode server log-edit message idna
rfc822 mml mml-sec mm-decode mm-bodies mm-encode mail-parse rfc2231
rfc2047 rfc2045 ietf-drums mailabbrev mail-utils gmm-utils mailheader
pcvs-util add-log projectile ibuf-ext ibuffer pkg-info lisp-mnt epl f s
ucs-normalize helm-ring helm-elisp helm-eval eldoc helm-files
image-dired tramp tramp-compat tramp-loaddefs trampver shell dired-x
dired-aux ffap helm-buffers helm-elscreen helm-tags helm-bookmark
helm-adaptive helm-info helm-net browse-url xml url url-proxy
url-privacy url-expand url-methods url-history url-cookie url-domsuf
url-util url-parse auth-source gnus-util mm-util mail-prsvr
password-cache url-vars mailcap bookmark pp helm-locate helm-help
helm-org org org-macro org-footnote org-pcomplete pcomplete org-list
org-faces org-entities noutline outline org-version ob-emacs-lisp ob
ob-tangle ob-ref ob-lob ob-table ob-exp org-src ob-keys ob-comint
ob-core ob-eval org-compat org-macs org-loaddefs format-spec cal-menu
calendar cal-loaddefs helm-match-plugin helm-grep helm-regexp
helm-plugin grep helm-external helm-utils ls-lisp dired helm helm-source
eieio eieio-core idle-require flymake compile comint ansi-color
aggressive-indent easy-mmode names edebug flycheck find-func help-mode
rx pcase subr-x flyspell ispell load-dir config-solarized
solarized-dark-theme solarized config-intel config-minibuffer
config-icicles config-helm config-ido ido-ubiquitous warnings
auto-indent-mode auto-complete-config auto-complete popup ido config-vcs
config-auto-complete config-paste cua-base config-whitespace
adaptive-wrap config-indent config-ui paren config-rainbow-delimiters
rainbow-delimiters config-smartparens smartparens-config smartparens
dash config-undo diminish config-ace-easymotion evil-easymotion noflet
cl-indent cl-macs ace-jump-mode cl gv config-line-numbers linum-relative
linum config-scroll config-evil config-evil-textobjects
config-evil-modules evil-surround evil-leader evil evil-integration
undo-tree diff evil-maps evil-commands evil-command-window evil-types
evil-search evil-ex evil-macros evil-repeat evil-states evil-core
evil-common windmove thingatpt rect evil-digraphs evil-vars ring
config-safety config-desktop savehist config-modes key-chord hl-line
config-package finder-inf edmacro kmacro cl-loaddefs cl-lib info
easymenu package epg-config config-setq delsel cus-start cus-load
byte-opt bytecomp byte-compile cconv advice help-fns time-date tooltip
electric uniquify ediff-hook vc-hooks lisp-float-type mwheel x-win x-dnd
tool-bar dnd fontset image regexp-opt fringe tabulated-list newcomment
lisp-mode prog-mode register page menu-bar rfn-eshadow timer select
scroll-bar mouse jit-lock font-lock syntax facemenu font-core frame cham
georgian utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao
korean japanese hebrew greek romanian slovak czech european ethiopic
indian cyrillic chinese case-table epa-hook jka-cmpr-hook help simple
abbrev minibuffer nadvice loaddefs button faces cus-face macroexp files
text-properties overlay sha1 md5 base64 format env code-pages mule
custom widget hashtable-print-readable backquote make-network-process
dbusbind gfilenotify dynamic-setting system-font-setting
font-render-setting move-toolbar gtk x-toolkit x multi-tty emacs)

Memory information:
((conses 16 782052 178836)
  (symbols 48 58485 0)
  (miscs 40 120 268)
  (strings 32 135655 141330)
  (string-bytes 1 4259469)
  (vectors 16 110096)
  (vector-slots 8 2260594 96673)
  (floats 8 10670 428)
  (intervals 56 550 248)
  (buffers 960 12)
  (heap 1024 63413 7100))

[-- Attachment #2: zsh_testcase.zsh.txt --]
[-- Type: text/plain, Size: 269 bytes --]

for i in $region_highlight; do
  if [[ $param == (#b)[^0-9]##(<->)[^0-9]##(<->)(*) ]]; then
    i=("$match")
    if [[ $i[3] == *black* ]] && (($i[2] - $i[1] > 0 && $i[1] > 1)); then
      BUFFER=$BUFFER[1,$i[1]]$BUFFER[$i[2],$(($#BUFFER - 1))]
    fi
  fi
end

bug#19455: Preliminary Patch

[Jonathan H]

[-- Attachment #1.1: Type: text/plain, Size: 690 bytes --]

Hi all!

I've produced a patch that should fix this issue. As brought up in this
Emacs.SE answer <http://emacs.stackexchange.com/a/20406/2642>, the main
challenge was distinguishing the qualifiers from subshells. To do this, I
disallowed leading octothorpes wrapped in parentheses where the parentheses
are not separated from other tokens by metacharacters. For qualifiers
(which appear after a glob), the fix is trivial. For flags (which appear
before the glob), I have to rely on the fact that flags may not contain
parentheses themselves to avoid needing more sophisticated parsing.

I've tested this against a fairly comprehensive test file, which I've also
attached.

Thanks,
Jonathan

[-- Attachment #1.2: Type: text/html, Size: 796 bytes --]

[-- Attachment #2: 0001-Highlight-zsh-glob-flags-and-qualifiers-in-sh-mode.patch --]
[-- Type: application/octet-stream, Size: 1501 bytes --]

From a478edef2b55b0495128eaaad3a1a7b277062fca Mon Sep 17 00:00:00 2001
From: PythonNut <PythonNut@users.noreply.github.com>
Date: Sat, 20 Feb 2016 17:21:26 -0800
Subject: [PATCH] Highlight zsh glob flags and qualifiers in sh-mode

* lisp/progmode/sh-script.el (sh-syntax-propertize-function):
Add regexps to highlight zsh glob flags and alternate qualifiers
without mistaking them for comments.
---
 lisp/progmodes/sh-script.el | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/lisp/progmodes/sh-script.el b/lisp/progmodes/sh-script.el
index 7f89ab2..0c392f1 100644
--- a/lisp/progmodes/sh-script.el
+++ b/lisp/progmodes/sh-script.el
@@ -1184,7 +1184,13 @@ subshells can nest."
     ;; metacharacters.  The list of special chars is taken from
     ;; the single-unix spec of the shell command language (under
     ;; `quoting') but with `$' removed.
-    ("\\(?:[^|&;<>()`\\\"' \t\n]\\|\\${\\)\\(#+\\)" (1 "_"))
+    ("\\(?:[^|&;<>(`\\\"' \t\n]\\|\\${\\)\\(#+\\)" (1 "_"))
+    ;; In addition, `#' at the beginning of closed parentheses
+    ;; does not start a comment if the parentheses are not isolated
+    ;; by metacharacters, excluding [()].
+    ;; (e.g. `foo(#q/)' and `(#b)foo' in zsh)
+    ("[^|&;<>(`\\\"' \t\n](\\(#+\\)" (1 "_"))
+    ("(\\(#\\)[^)]+?)[^|&;<>)`\\\"' \t\n]" (1 "_"))
     ;; In a '...' the backslash is not escaping.
     ("\\(\\\\\\)'" (1 (sh-font-lock-backslash-quote)))
     ;; Make sure $@ and $? are correctly recognized as sexps.
-- 
2.5.0


[-- Attachment #3: sh_comment_tests.sh --]
[-- Type: application/x-sh, Size: 1105 bytes --]

bug#19455: 24.4; sh-mode highlighting breaks with ZSH glob modifiers

[Alex]

tags 19455 patch
found 19455 25.1.50
quit

I was just about to report a similar issue. My issue was with a
zsh-specific substitution, such as:

echo ${$(some_function)#*.}

The patch provided fixed that for me.

bug#19455: Preliminary Patch

[Lars Ingebrigtsen]

Jonathan H <pythonnut@gmail.com> writes:

> I've tested this against a fairly comprehensive test file, which I've
> also attached.

Thanks; I've tested the change with the test file, and things seem to
work as promised.  I've now applied the patch to the Emacs trunk.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no