shell mode syntax higlighting problem with here documents

shell mode syntax higlighting problem with here documents

[Nicholas Sushkin]

This bug report will be sent to the Free Software Foundation,
not to your local site managers!
Please write in English, because the Emacs maintainers do not have
translators to read other languages for them.

Your bug report will be posted to the bug-gnu-emacs@gnu.org mailing list,
and to the gnu.emacs.bug news group.

In GNU Emacs 21.3.1 (i386-redhat-linux-gnu, X toolkit, Xaw3d scroll bars)
 of 2005-02-04 on bugs.build.redhat.com
configured using `configure  --build=i386-redhat-linux --host=i386-redhat-linux --target=i386-redhat-linux-gnu --program-prefix= --prefix=/usr --exec-prefix=/usr --bindir=/usr/bin --sbindir=/usr/sbin --sysconfdir=/etc --datadir=/usr/share --includedir=/usr/include --libdir=/usr/lib --libexecdir=/usr/libexec --localstatedir=/var --sharedstatedir=/usr/com --mandir=/usr/share/man --infodir=/usr/share/info --with-pop --with-sound'
Important settings:
  value of $LC_ALL: nil
  value of $LC_COLLATE: nil
  value of $LC_CTYPE: nil
  value of $LC_MESSAGES: nil
  value of $LC_MONETARY: nil
  value of $LC_NUMERIC: nil
  value of $LC_TIME: nil
  value of $LANG: en_US.UTF-8
  locale-coding-system: utf-8
  default-enable-multibyte-characters: t

Please describe exactly what actions triggered the bug
and the precise symptoms of the bug:

In the following shell script, the shell script mode syntax highlighting is broken.

#!/bin/bash
cat << EOF | \
    tee -a bug.txt

This is a test case for a bug in bash shell mode text highlighting
EOF
echo "Highlighting is screwed up now"
if [ 1 = 1 ]
then
  echo "screwed up"
fi



Recent input:
<down> <down> <up> s c r e w e d SPC u p " <return> 
f i <return> C-x 4 b C-g C-x s <up> <up> <up> <up> 
<up> <up> <up> <up> <up> <up> <up> <up> <return> <S-up> 
<down> <up> # @ <S-backspace> ! / b i n / b a s h C-x 
s <down> <down> <down> <down-mouse-1> <mouse-movement> 
<mouse-movement> <mouse-movement> <mouse-movement> 
<mouse-movement> <mouse-movement> <mouse-movement> 
<mouse-movement> <mouse-movement> <mouse-movement> 
<mouse-movement> <mouse-movement> <mouse-movement> 
<mouse-movement> <mouse-movement> <mouse-movement> 
<mouse-movement> <mouse-movement> <drag-mouse-1> <down-mouse-1> 
<mouse-1> <down-mouse-1> <mouse-1> <help-echo> <help-echo> 
<help-echo> <help-echo> <help-echo> <help-echo> <help-echo> 
<help-echo> <help-echo> <help-echo> <help-echo> <help-echo> 
<help-echo> <help-echo> <help-echo> <menu-bar> <help-menu> 
<report-emacs-bug>

Recent messages:
Indentation setup for shell type bash
Mark set [3 times]
Auto-saving...done
Wrote /tmp/bug.sh
Quit
Wrote /tmp/bug.sh
line-move: Beginning of buffer [2 times]
Mark set
Wrote /tmp/bug.sh
Loading emacsbug...done

Re: shell mode syntax higlighting problem with here documents

[Stefan Monnier]

> #!/bin/bash
> cat << EOF | \
>     tee -a bug.txt

> This is a test case for a bug in bash shell mode text highlighting
> EOF

If people who hack on SH scripts with lots of heredoc thingies could try the
patch below, it would help.  It's relative to the Emacs-CVS code.
It seems "safe" and appears to do the right thing with the above example,
but I'd like to get some positive feedback before installing it,


        Stefan


--- orig/lisp/progmodes/sh-script.el
+++ mod/lisp/progmodes/sh-script.el
@@ -869,7 +869,18 @@
 (defconst sh-st-symbol (string-to-syntax "_"))
 (defconst sh-here-doc-syntax (string-to-syntax "|")) ;; generic string
 
-(defconst sh-here-doc-open-re "<<-?\\s-*\\\\?\\(\\(?:['\"][^'\"]+['\"]\\|\\sw\\)+\\).*\\(\n\\)")
+(defconst sh-escaped-line-re
+  (concat
+   "\\(?:"
+   ;; First the real regexp.
+   "\\(?:.*[^\\\n]\\)?\\(?:\\(?:\\\\\\\\\\)*\\\\\n\\(?:.*[^\\\n]\\)?\\)*"
+   ;; Then a failsafe fallback: in case the search boundary prevents us from
+   ;; finding the actual end of line.
+   "\\|.*\\)"))
+
+(defconst sh-here-doc-open-re
+  (concat "<<-?\\s-*\\\\?\\(\\(?:['\"][^'\"]+['\"]\\|\\sw\\)+\\)"
+          sh-escaped-line-re "\\(\n\\)"))
 
 (defvar sh-here-doc-markers nil)
 (make-variable-buffer-local 'sh-here-doc-markers)
@@ -883,7 +894,9 @@
          ;; A rough regexp that should find the opening <<EOF back.
 	 (sre (concat "<<\\(-?\\)\\s-*['\"\\]?"
 		      ;; Use \s| to cheaply check it's an open-heredoc.
-		      eof-re "['\"]?\\([ \t|;&)<>].*\\)?\\s|"))
+		      eof-re "['\"]?\\([ \t|;&)<>]"
+                      sh-escaped-line-re
+                      "\\)?\\s|"))
 	 ;; A regexp that will find other EOFs.
 	 (ere (concat "^" (if indented "[ \t]*") eof-re "\n"))
 	 (start (save-excursion
@@ -922,7 +935,8 @@
 START is the position of <<.
 STRING is the actual word used as delimiter (f.ex. \"EOF\").
 INDENTED is non-nil if the here document's content (and the EOF mark) can
-be indented (i.e. a <<- was used rather than just <<)."
+be indented (i.e. a <<- was used rather than just <<).
+Point is at the beginning of the next line."
   (unless (or (memq (char-before start) '(?< ?>))
 	      (sh-in-comment-or-string start))
     ;; We're looking at <<STRING, so we add "^STRING$" to the syntactic
@@ -933,6 +947,20 @@
 	(setq sh-here-doc-re
 	      (concat sh-here-doc-open-re "\\|^\\([ \t]*\\)"
 		      (regexp-opt sh-here-doc-markers t) "\\(\n\\)"))))
+    (let ((ppss (save-excursion (syntax-ppss (1- (point))))))
+      (if (nth 4 ppss)
+          ;; The \n not only starts the heredoc but also closes a comment.
+          ;; Let's close the comment just before the \n.
+          (put-text-property (1- (point)) (point) 'syntax-table '(12))) ;">"
+      (if (or (> (count-lines start (point)) 1) (nth 5 ppss))
+          ;; If the sh-escaped-line-re part of sh-here-doc-re has matched
+          ;; several lines, make sure we refontify them together.
+          ;; Furthermore, if (nth 5 ppss) is non-nil (i.e. the \n is
+          ;; escaped), it means we haven't actually found the right \n.
+          ;; Don't bother fixing it now, but place a multiline property so
+          ;; that when jit-lock-context-* refontifies the rest of the
+          ;; buffer, it also refontifies the current line with it.
+          (put-text-property start (point) 'font-lock-multiline t)))
     sh-here-doc-syntax))
 
 (defun sh-font-lock-here-doc (limit)

Re: shell mode syntax higlighting problem with here documents

[Nicholas Sushkin]

Stefan,

Merci. I'll try to test the patch within the next couple of days and let you 
know.

On Wednesday 19 October 2005 23:48, Stefan Monnier wrote: 

> If people who hack on SH scripts with lots of heredoc thingies could try
> the patch below, it would help.  It's relative to the Emacs-CVS code. It
> seems "safe" and appears to do the right thing with the above example,
> but I'd like to get some positive feedback before installing it,

-- 
Nicholas Sushkin, Senior Software Engineer
Open Finance

Re: shell mode syntax higlighting problem with here documents

[Nicholas Sushkin]

Stefan, I am trying to test in Emacs 21.3.1 your revision 1.169, but getting 
"Invalid escape character syntax" in the statement below. Is my emacs too 
old or it's really a bug? 

Thanks

(defvar sh-skeleton-pair-default-alist '((?( _ ?)) (?\))
          (?[ ?\s _ ?\s ?]) (?\])
          (?{ _ ?}) (?\}))
  "Value to use for `skeleton-pair-default-alist' in Shell-Script mode.")




On Wednesday 19 October 2005 23:48, Stefan Monnier wrote: 

> > #!/bin/bash
> > cat << EOF | \
> >     tee -a bug.txt
> >
> > This is a test case for a bug in bash shell mode text highlighting
> > EOF
>
> If people who hack on SH scripts with lots of heredoc thingies could try
> the patch below, it would help.  It's relative to the Emacs-CVS code. It
> seems "safe" and appears to do the right thing with the above example,
> but I'd like to get some positive feedback before installing it,
>
>
>         Stefan
>
>
> --- orig/lisp/progmodes/sh-script.el
> +++ mod/lisp/progmodes/sh-script.el
> @@ -869,7 +869,18 @@
>  (defconst sh-st-symbol (string-to-syntax "_"))
>  (defconst sh-here-doc-syntax (string-to-syntax "|")) ;; generic string
>
> -(defconst sh-here-doc-open-re
> "<<-?\\s-*\\\\?\\(\\(?:['\"][^'\"]+['\"]\\|\\sw\\)+\\).*\\(\n\\)")
> +(defconst sh-escaped-line-re
> +  (concat
> +   "\\(?:"
> +   ;; First the real regexp.
> +  
> "\\(?:.*[^\\\n]\\)?\\(?:\\(?:\\\\\\\\\\)*\\\\\n\\(?:.*[^\\\n]\\)?\\)*" + 
>  ;; Then a failsafe fallback: in case the search boundary prevents us
> from +   ;; finding the actual end of line.
> +   "\\|.*\\)"))
> +
> +(defconst sh-here-doc-open-re
> +  (concat "<<-?\\s-*\\\\?\\(\\(?:['\"][^'\"]+['\"]\\|\\sw\\)+\\)"
> +          sh-escaped-line-re "\\(\n\\)"))
>
>  (defvar sh-here-doc-markers nil)
>  (make-variable-buffer-local 'sh-here-doc-markers)
> @@ -883,7 +894,9 @@
>           ;; A rough regexp that should find the opening <<EOF back.
>  	 (sre (concat "<<\\(-?\\)\\s-*['\"\\]?"
>  		      ;; Use \s| to cheaply check it's an open-heredoc.
> -		      eof-re "['\"]?\\([ \t|;&)<>].*\\)?\\s|"))
> +		      eof-re "['\"]?\\([ \t|;&)<>]"
> +                      sh-escaped-line-re
> +                      "\\)?\\s|"))
>  	 ;; A regexp that will find other EOFs.
>  	 (ere (concat "^" (if indented "[ \t]*") eof-re "\n"))
>  	 (start (save-excursion
> @@ -922,7 +935,8 @@
>  START is the position of <<.
>  STRING is the actual word used as delimiter (f.ex. \"EOF\").
>  INDENTED is non-nil if the here document's content (and the EOF mark)
> can -be indented (i.e. a <<- was used rather than just <<)."
> +be indented (i.e. a <<- was used rather than just <<).
> +Point is at the beginning of the next line."
>    (unless (or (memq (char-before start) '(?< ?>))
>  	      (sh-in-comment-or-string start))
>      ;; We're looking at <<STRING, so we add "^STRING$" to the syntactic
> @@ -933,6 +947,20 @@
>  	(setq sh-here-doc-re
>  	      (concat sh-here-doc-open-re "\\|^\\([ \t]*\\)"
>  		      (regexp-opt sh-here-doc-markers t) "\\(\n\\)"))))
> +    (let ((ppss (save-excursion (syntax-ppss (1- (point))))))
> +      (if (nth 4 ppss)
> +          ;; The \n not only starts the heredoc but also closes a
> comment. +          ;; Let's close the comment just before the \n.
> +          (put-text-property (1- (point)) (point) 'syntax-table '(12)))
> ;">" +      (if (or (> (count-lines start (point)) 1) (nth 5 ppss))
> +          ;; If the sh-escaped-line-re part of sh-here-doc-re has
> matched +          ;; several lines, make sure we refontify them
> together. +          ;; Furthermore, if (nth 5 ppss) is non-nil (i.e. the
> \n is +          ;; escaped), it means we haven't actually found the
> right \n. +          ;; Don't bother fixing it now, but place a multiline
> property so +          ;; that when jit-lock-context-* refontifies the
> rest of the +          ;; buffer, it also refontifies the current line
> with it. +          (put-text-property start (point) 'font-lock-multiline
> t))) sh-here-doc-syntax))
>
>  (defun sh-font-lock-here-doc (limit)

-- 
Nicholas Sushkin, Senior Software Engineer
Open Finance, Chelsea Piers Pier 62 Suite 316, New York NY 10011
Tel +1 646 723 2790 Fax +1 646 723 2789 nsushkin@openfinance.com

Re: shell mode syntax higlighting problem with here documents

[Nick Roberts]

Nicholas Sushkin writes:
 > Stefan, I am trying to test in Emacs 21.3.1 your revision 1.169, but getting 
 > "Invalid escape character syntax" in the statement below. Is my emacs too 
 > old or it's really a bug? 

Your emacs is too old.  I guess there might be some value in testing against
an old version but there's a lot more value in testing with the current one.
If you can't build Emacs from CVS then perhaps you can download the pretest
which should be available shortly.

Nick

Re: shell mode syntax higlighting problem with here documents

[Nicholas Sushkin]

Ok, I tried the CVS version as of half an hour ago.

It seems that on the initial load, it highlights correctly. However, if I 
insert a couple of spaces at the beginning of the line that contains << 
EOF, the highlighting after the final EOF breaks. Kill and reload the file 
fixes it again.

On Thursday 20 October 2005 17:22, Nick Roberts wrote: 

> Nicholas Sushkin writes:
>  > Stefan, I am trying to test in Emacs 21.3.1 your revision 1.169, but
>  > getting "Invalid escape character syntax" in the statement below. Is
>  > my emacs too old or it's really a bug?
>
> Your emacs is too old.  I guess there might be some value in testing
> against an old version but there's a lot more value in testing with the
> current one. If you can't build Emacs from CVS then perhaps you can
> download the pretest which should be available shortly.
>
> Nick

-- 
Nicholas Sushkin, Senior Software Engineer
Open Finance

Re: shell mode syntax higlighting problem with here documents

[Stefan Monnier]

> Ok, I tried the CVS version as of half an hour ago.
> It seems that on the initial load, it highlights correctly. However, if I 
> insert a couple of spaces at the beginning of the line that contains << 
> EOF, the highlighting after the final EOF breaks. Kill and reload the file 
> fixes it again.

I've installed a slightly fixed variant of the patch I sent.
Can you try with the latest CVS code?  It seemed to work on all my tests.


        Stefan

Re: shell mode syntax higlighting problem with here documents

[Nicholas Sushkin]

[-- Attachment #1: Type: text/plain, Size: 575 bytes --]

On Thursday 20 October 2005 23:14, Stefan Monnier wrote: 

> I've installed a slightly fixed variant of the patch I sent.
> Can you try with the latest CVS code?  It seemed to work on all my tests.

I still get the same behavior. I load the test case below. It starts 
highlighted properly (snapshot 1). I insert a space before the command that 
starts the here document, highlighting breaks (snapshot 2). I remove the 
inserted space, the highlighting is still broken (snapshot 3).

Do you have the same behavior?
-- 
Nicholas Sushkin, Senior Software Engineer
Open Finance

[-- Attachment #2: emacs-shhighlight-bug-1.png --]
[-- Type: image/png, Size: 26071 bytes --]

[-- Attachment #3: emacs-shhighlight-bug-2.png --]
[-- Type: image/png, Size: 24313 bytes --]

[-- Attachment #4: emacs-shhighlight-bug-3.png --]
[-- Type: image/png, Size: 24310 bytes --]

[-- Attachment #5: Type: text/plain, Size: 142 bytes --]

_______________________________________________
Emacs-devel mailing list
Emacs-devel@gnu.org
http://lists.gnu.org/mailman/listinfo/emacs-devel

Re: shell mode syntax higlighting problem with here documents

[Stefan Monnier]

>> I've installed a slightly fixed variant of the patch I sent.
>> Can you try with the latest CVS code?  It seemed to work on all my tests.

> I still get the same behavior. I load the test case below. It starts 
> highlighted properly (snapshot 1). I insert a space before the command that 
> starts the here document, highlighting breaks (snapshot 2). I remove the 
> inserted space, the highlighting is still broken (snapshot 3).

> Do you have the same behavior?

I've just installed a patch to font-lock.el which I've been using "for ever"
but somehow forgot to install until now.  It should work better now.


        Stefan

Re: shell mode syntax higlighting problem with here documents

[Nicholas Sushkin]

On Friday 21 October 2005 13:42, Stefan Monnier wrote: 

> I've just installed a patch to font-lock.el which I've been using "for
> ever" but somehow forgot to install until now.  It should work better
> now.

It does! It seems to work right now. Thanks a lot, Stefan!

-- 
Nicholas Sushkin, Senior Software Engineer
Open Finance