bug#52496: syntax highlighting problem on emacs shell-script mode

bug#52496: syntax highlighting problem on emacs shell-script mode

[Yilkal Argaw]

I don't know exactly how to explain it but emacs comments most of a
script if the script contains too many long strings. As an example
look at the syntax highlighting of the script at
https://gitlab.com/uppercat/tlbx/-/raw/master/tlbx and compare the
syntax highlighting for it in vim and emacs.

bug#52496: syntax highlighting problem on emacs shell-script mode

[Lars Ingebrigtsen]

[-- Attachment #1: Type: text/plain, Size: 530 bytes --]

Yilkal Argaw <yilkalargawworkneh@gmail.com> writes:

> I don't know exactly how to explain it but emacs comments most of a
> script if the script contains too many long strings. As an example
> look at the syntax highlighting of the script at
> https://gitlab.com/uppercat/tlbx/-/raw/master/tlbx and compare the
> syntax highlighting for it in vim and emacs.

You don't say what Emacs version this is with, or which parts you think
Emacs is displaying wrong.  Is it the heredoc thing that's wrong, or
something else in addition?


[-- Attachment #2: Type: image/png, Size: 35653 bytes --]

[-- Attachment #3: Type: text/plain, Size: 182 bytes --]


It seems like sh-mode isn't interpreting that "@" as the end of the
heredoc.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#52496: syntax highlighting problem on emacs shell-script mode

[Andreas Schwab]

On Dez 19 2021, Lars Ingebrigtsen wrote:

> It seems like sh-mode isn't interpreting that "@" as the end of the
> heredoc.

ELISP> (let ((s " <<- @\n")) (string-match sh-here-doc-open-re s) (match-string 1 s))
"-"
ELISP> (let ((s " <<- EOF\n")) (string-match sh-here-doc-open-re s) (match-string 1 s))
"EOF"

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

bug#52496: syntax highlighting problem on emacs shell-script mode

[Lars Ingebrigtsen]

Andreas Schwab <schwab@linux-m68k.org> writes:

>> It seems like sh-mode isn't interpreting that "@" as the end of the
>> heredoc.
>
> ELISP> (let ((s " <<- @\n")) (string-match sh-here-doc-open-re s)
> ELISP> (match-string 1 s))
> "-"
> ELISP> (let ((s " <<- EOF\n")) (string-match sh-here-doc-open-re s)
> ELISP> (match-string 1 s))
> "EOF"

Right.  I haven't real the shell documentation -- is "@" a valid heredoc
... separator?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#52496: syntax highlighting problem on emacs shell-script mode

[Andreas Schwab]

On Dez 19 2021, Lars Ingebrigtsen wrote:

> Right.  I haven't real the shell documentation -- is "@" a valid heredoc
> ... separator?

Any word is.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

bug#52496: syntax highlighting problem on emacs shell-script mode

[Lars Ingebrigtsen]

Andreas Schwab <schwab@linux-m68k.org> writes:

> On Dez 19 2021, Lars Ingebrigtsen wrote:
>
>> Right.  I haven't real the shell documentation -- is "@" a valid heredoc
>> ... separator?
>
> Any word is.

But is @ word constituent?

It's not in sh-mode's word definition, anyway.  The following tweak
fixes this particular problem, but I have no idea whether it was what
what originally reported.

diff --git a/lisp/progmodes/sh-script.el b/lisp/progmodes/sh-script.el
index c6b6f83471..01bebe4a23 100644
--- a/lisp/progmodes/sh-script.el
+++ b/lisp/progmodes/sh-script.el
@@ -867,7 +867,7 @@ sh-here-doc-syntax
     "\\(?:\\(?:.*[^\\\n]\\)?\\(?:\\\\\\\\\\)*\\\\\n\\)*.*")
 
   (defconst sh-here-doc-open-re
-    (concat "[^<]<<-?\\s-*\\\\?\\(\\(?:['\"][^'\"]+['\"]\\|\\sw\\|[-/~._]\\)+\\)"
+    (concat "[^<]<<-?\\s-*\\\\?\\(\\(?:['\"][^'\"]+['\"]\\|\\sw\\|[-/~._@]\\)+\\)"
             sh-escaped-line-re "\\(\n\\)")))
 
 (defun sh--inside-noncommand-expression (pos)


-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#52496: syntax highlighting problem on emacs shell-script mode

[Andreas Schwab]

On Dez 19 2021, Lars Ingebrigtsen wrote:

> Andreas Schwab <schwab@linux-m68k.org> writes:
>
>> On Dez 19 2021, Lars Ingebrigtsen wrote:
>>
>>> Right.  I haven't real the shell documentation -- is "@" a valid heredoc
>>> ... separator?
>>
>> Any word is.
>
> But is @ word constituent?

A shell word can be almost anything.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

bug#52496: syntax highlighting problem on emacs shell-script mode

[Kévin Le Gouguec]

Andreas Schwab <schwab@linux-m68k.org> writes:

> On Dez 19 2021, Lars Ingebrigtsen wrote:
>
>> Andreas Schwab <schwab@linux-m68k.org> writes:
>>
>>> On Dez 19 2021, Lars Ingebrigtsen wrote:
>>>
>>>> Right.  I haven't real the shell documentation -- is "@" a valid heredoc
>>>> ... separator?
>>>
>>> Any word is.
>>
>> But is @ word constituent?
>
> A shell word can be almost anything.

Relevant part of (bash) Redirections:

>    The format of here-documents is:
>      [N]<<[-]WORD
>              HERE-DOCUMENT
>      DELIMITER
> 
>    No parameter and variable expansion, command substitution, arithmetic
> expansion, or filename expansion is performed on WORD.  If any part of
> WORD is quoted, the DELIMITER is the result of quote removal on WORD,

And (bash) Definitions:

> 'word'
>      A sequence of characters treated as a unit by the shell.  Words may
>      not include unquoted 'metacharacters'.

Preceded by:

> 'metacharacter'
>      A character that, when unquoted, separates words.  A metacharacter
>      is a 'space', 'tab', 'newline', or one of the following characters:
>      '|', '&', ';', '(', ')', '<', or '>'.

(AFAICT the same goes for POSIX sh[1])

So IIUC for here-documents anything is fair game except (1)
{single,double,back} quotes (2) metacharacters (3) backslashes and
comment openers (#); all of which can still be used if quoted or escaped
*on the opening line only*.


[1] https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_07_04

bug#52496: syntax highlighting problem on emacs shell-script mode

[Lars Ingebrigtsen]

Kévin Le Gouguec <kevin.legouguec@gmail.com> writes:

> So IIUC for here-documents anything is fair game except (1)
> {single,double,back} quotes (2) metacharacters (3) backslashes and
> comment openers (#); all of which can still be used if quoted or escaped
> *on the opening line only*.

Thanks.

As far as I can see, sh-mode doesn't really try too hard to be correct
here -- it accepts word constituents, but then adds back -/~._ which it
has defined as punctuation in `sh-mode-syntax-table'.  So perhaps we
should just continue doing that and add @, too, even though that's not
very satisfactory.

So I've now done that in Emacs 29.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#52496: syntax highlighting problem on emacs shell-script mode

[Yilkal Argaw]

Yes, I agree.  I think this is enough for now until a refactor becomes
necessary.

On Mon, Dec 20, 2021 at 12:46 PM Lars Ingebrigtsen <larsi@gnus.org> wrote:
>
> Kévin Le Gouguec <kevin.legouguec@gmail.com> writes:
>
> > So IIUC for here-documents anything is fair game except (1)
> > {single,double,back} quotes (2) metacharacters (3) backslashes and
> > comment openers (#); all of which can still be used if quoted or escaped
> > *on the opening line only*.
>
> Thanks.
>
> As far as I can see, sh-mode doesn't really try too hard to be correct
> here -- it accepts word constituents, but then adds back -/~._ which it
> has defined as punctuation in `sh-mode-syntax-table'.  So perhaps we
> should just continue doing that and add @, too, even though that's not
> very satisfactory.
>
> So I've now done that in Emacs 29.
>
> --
> (domestic pets only, the antidote for overdose, milk.)
>    bloggy blog: http://lars.ingebrigtsen.no

bug#52496: syntax highlighting problem on emacs shell-script mode

[Lars Ingebrigtsen]

Yilkal Argaw <yilkalargawworkneh@gmail.com> writes:

> Yes it is the heredoc part. I have tried emacs 27, 28 and 29(i.e. the
> master) and it all behaves the same.

Right.  So the question is whether we should just continue to add
characters to that regexp (safe solution) or whether to try to amend the
shell-script-mode concept of a "word", which I think is likely to break
stuff, but would be more correct...  probably...