bug#3226: here-document syntax inference and completion is broken

bug#3226: here-document syntax inference and completion is broken

[Paul Nienaber]

Steps to reproduce:

Cook yourself up a bash script and type a line of code like:
for i in $(/bin/vim <<

A default emacs install will complete with "EOF\n\nEOF" (in a more 
clever fashion so the rest of the line is undisturbed), and place the 
cursor on the new blank line.  This is
incorrect behaviour, as it breaks entry of BASH here-strings when the 
user's intent is still ambiguous.  This should only respond to things 
ending in, for example "<< " or
possibly "<<E", but certainly NOT just "<<".  Further, emacs does 
something outright wrong:  An attempt to fix the situation by inserting 
another '<' between the "<<" and "EOF" +results in ANOTHER pair of "EOF" 
being inserted in a syntactically invalid way.


~Paul Nienaber

bug#3226: here-document syntax inference and completion is broken

[Stefan Monnier]

> A default emacs install will complete with "EOF\n\nEOF" (in a more
> clever fashion so the rest of the line is undisturbed), and place the
> cursor on the new blank line.  This is incorrect behaviour, as it
> breaks entry of BASH here-strings when the user's intent is
> still ambiguous.

Thanks for your report.  Indeed, sh-mode does not know about
here-strings at all.  I guess waiting for "<<[^<]" is a good idea.
Patches welcome for Emacs-23.2.


        Stefan

bug#3226: here-document syntax inference and completion is broken

[Chong Yidong]

> Cook yourself up a bash script and type a line of code like:
> for i in $(/bin/vim <<
>
> A default emacs install will complete with "EOF\n\nEOF" (in a more
>clever fashion so the rest of the line is undisturbed), and place the
>cursor on the new blank line.  This is incorrect behaviour, as it
>breaks entry of BASH here-strings when the user's intent is still
>ambiguous.  This should only respond to things ending in, for example
>"<< " or possibly "<<E", but certainly NOT just "<<".
>
> Further, emacs does something outright wrong: An attempt to fix the
>situation by inserting another '<' between the "<<" and "EOF" +results
>in ANOTHER pair of "EOF" being inserted in a syntactically invalid way.

I fixed the second problem, but I don't see any easy way to solve the
first one.

(The difficulty of fixing the first bug is illustrative of the problems
with individual major modes rolling their own "electric" completion
facilities.  It might be nice to have a minor mode that provides a
unified framework for this kind of functionality; which may also need
built-in support.)

bug#3226: here-document syntax inference and completion is broken

[Andrew Hyatt]

Chong Yidong <cyd@stupidchicken.com> writes:

>> Cook yourself up a bash script and type a line of code like:
>> for i in $(/bin/vim <<
>>
>> A default emacs install will complete with "EOF\n\nEOF" (in a more
>>clever fashion so the rest of the line is undisturbed), and place the
>>cursor on the new blank line.  This is incorrect behaviour, as it
>>breaks entry of BASH here-strings when the user's intent is still
>>ambiguous.  This should only respond to things ending in, for example
>>"<< " or possibly "<<E", but certainly NOT just "<<".
>>
>> Further, emacs does something outright wrong: An attempt to fix the
>>situation by inserting another '<' between the "<<" and "EOF" +results
>>in ANOTHER pair of "EOF" being inserted in a syntactically invalid way.
>
> I fixed the second problem, but I don't see any easy way to solve the
> first one.
>
> (The difficulty of fixing the first bug is illustrative of the problems
> with individual major modes rolling their own "electric" completion
> facilities.  It might be nice to have a minor mode that provides a
> unified framework for this kind of functionality; which may also need
> built-in support.)

Just a note that I've reproduced the original problem in Emacs 25, so it
appears to be unfixed at present.

bug#3226: here-document syntax inference and completion is broken

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 1578 bytes --]

Andrew Hyatt <ahyatt@gmail.com> schrieb am Di., 26. Jan. 2016 um 06:16 Uhr:

> Chong Yidong <cyd@stupidchicken.com> writes:
>
> >> Cook yourself up a bash script and type a line of code like:
> >> for i in $(/bin/vim <<
> >>
> >> A default emacs install will complete with "EOF\n\nEOF" (in a more
> >>clever fashion so the rest of the line is undisturbed), and place the
> >>cursor on the new blank line.  This is incorrect behaviour, as it
> >>breaks entry of BASH here-strings when the user's intent is still
> >>ambiguous.  This should only respond to things ending in, for example
> >>"<< " or possibly "<<E", but certainly NOT just "<<".
> >>
> >> Further, emacs does something outright wrong: An attempt to fix the
> >>situation by inserting another '<' between the "<<" and "EOF" +results
> >>in ANOTHER pair of "EOF" being inserted in a syntactically invalid way.
> >
> > I fixed the second problem, but I don't see any easy way to solve the
> > first one.
> >
> > (The difficulty of fixing the first bug is illustrative of the problems
> > with individual major modes rolling their own "electric" completion
> > facilities.  It might be nice to have a minor mode that provides a
> > unified framework for this kind of functionality; which may also need
> > built-in support.)
>
> Just a note that I've reproduced the original problem in Emacs 25, so it
> appears to be unfixed at present.
>
>
>
The behavior can be disabled using

(add-hook 'sh-mode-hook (lambda () (sh-electric-here-document-mode 0)))

I think this behavior should be customizable and off by default.

[-- Attachment #2: Type: text/html, Size: 2293 bytes --]

bug#3226: here-document syntax inference and completion is broken

[Mark Oteiza]

Philipp Stephani <p.stephani2@gmail.com> writes:

> Andrew Hyatt <ahyatt@gmail.com> schrieb am Di., 26. Jan. 2016 um 06:16 Uhr:
>
>  Chong Yidong <cyd@stupidchicken.com> writes:
>
>  >> Cook yourself up a bash script and type a line of code like:
>  >> for i in $(/bin/vim <<
>  >>
>  >> A default emacs install will complete with "EOF\n\nEOF" (in a more
>  >>clever fashion so the rest of the line is undisturbed), and place the
>  >>cursor on the new blank line. This is incorrect behaviour, as it
>  >>breaks entry of BASH here-strings when the user's intent is still
>  >>ambiguous. This should only respond to things ending in, for example
>  >>"<< " or possibly "<<E", but certainly NOT just "<<".
>  >>
>  >> Further, emacs does something outright wrong: An attempt to fix the
>  >>situation by inserting another '<' between the "<<" and "EOF" +results
>  >>in ANOTHER pair of "EOF" being inserted in a syntactically invalid way.
>  >
>  > I fixed the second problem, but I don't see any easy way to solve the
>  > first one.
>  >
>  > (The difficulty of fixing the first bug is illustrative of the problems
>  > with individual major modes rolling their own "electric" completion
>  > facilities. It might be nice to have a minor mode that provides a
>  > unified framework for this kind of functionality; which may also need
>  > built-in support.)
>
>  Just a note that I've reproduced the original problem in Emacs 25, so it
>  appears to be unfixed at present.
>
> The behavior can be disabled using
>
> (add-hook 'sh-mode-hook (lambda () (sh-electric-here-document-mode 0)))
>
> I think this behavior should be customizable and off by default. 

I agree. In addition to breaking here strings, it also complicates
writing here docs not using sh-here-document-word, as well as here docs
with additional contents after <<TAG. For example, this is valid bash:

sed -f - <<EOF <(foo "$x") | bar
s/cat/dog/g
EOF

The behavior has been in place for a long time though (IIUC Stefan's
mail correctly, since emacs 23), so I am wary of applying the following:

diff --git a/lisp/progmodes/sh-script.el b/lisp/progmodes/sh-script.el
index 4fb5835..324cbf1 100644
--- a/lisp/progmodes/sh-script.el
+++ b/lisp/progmodes/sh-script.el
@@ -1228,6 +1228,7 @@ defcustom sh-set-shell-hook nil
 (defcustom sh-mode-hook nil
   "Hook run by `sh-mode'."
   :type 'hook
+  :options '(sh-electric-here-document-mode)
   :group 'sh-script)
 
 (defcustom sh-learn-basic-offset nil
@@ -1616,7 +1617,8 @@ define-derived-mode sh-mode prog-mode "Shell-script"
 \\[sh-execute-region]	 Have optional header and region be executed in a subshell.
 
 `sh-electric-here-document-mode' controls whether insertion of two
-unquoted < insert a here document.
+unquoted < insert a here document.  You can enable this behavior by
+adding to `sh-mode-hook'.
 
 If you generally program a shell different from your login shell you can
 set `sh-shell-file' accordingly.  If your shell's file name doesn't correctly
@@ -1653,7 +1655,6 @@ define-derived-mode sh-mode prog-mode "Shell-script"
   (setq-local syntax-propertize-function #'sh-syntax-propertize-function)
   (add-hook 'syntax-propertize-extend-region-functions
             #'syntax-propertize-multiline 'append 'local)
-  (sh-electric-here-document-mode 1)
   (setq-local skeleton-pair-alist '((?` _ ?`)))
   (setq-local skeleton-pair-filter-function 'sh-quoted-p)
   (setq-local skeleton-further-elements

bug#3226: here-document syntax inference and completion is broken

[Stefan Monnier]

> -  (sh-electric-here-document-mode 1)

Whether or not we enable it by default, the code needs to be changed so
that you need to type something like "<< " rather than just "<<" in
order to trigger the electric behavior.


        Stefan

bug#3226: here-document syntax inference and completion is broken

[Mark Oteiza]

On 27/03/16 at 06:43pm, Stefan Monnier wrote:
> > -  (sh-electric-here-document-mode 1)
> 
> Whether or not we enable it by default, the code needs to be changed so
> that you need to type something like "<< " rather than just "<<" in
> order to trigger the electric behavior.

Until then, I moved the mode invocation to sh-mode-hook in ac00a92.

bug#3226: here-document syntax inference and completion is broken

[Lars Ingebrigtsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> A default emacs install will complete with "EOF\n\nEOF" (in a more
>> clever fashion so the rest of the line is undisturbed), and place the
>> cursor on the new blank line.  This is incorrect behaviour, as it
>> breaks entry of BASH here-strings when the user's intent is
>> still ambiguous.
>
> Thanks for your report.  Indeed, sh-mode does not know about
> here-strings at all.  I guess waiting for "<<[^<]" is a good idea.
> Patches welcome for Emacs-23.2.

I've now made it look for a space after the << -- that feels more
natural than << and then some arbitrary character.  And as far as I can
tell, all the shells handle

foo <<EOF
EOF

and

foo << EOF
EOF

identically. 

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#3226: here-document syntax inference and completion is broken

[Paul Nienaber]

[-- Attachment #1: Type: text/plain, Size: 1403 bytes --]

While I use here-strings more than here-docs by a huge margin I would also
like to not negatively impact people using the former a lot, so in light of
that I'd like to suggest going with Stefan's suggestion, importantly
because it's also totally valid to trip on '<<-' and '<<[A-Za-z]', whereas
here-strings are always only '<<<'.  Also '<<EOF' is the overwhelmingly
more common form that I see vs '<< EOF'.

Thanks again for fixing this!

On Fri, Oct 11, 2019 at 12:38 PM Lars Ingebrigtsen <larsi@gnus.org> wrote:

> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>
> >> A default emacs install will complete with "EOF\n\nEOF" (in a more
> >> clever fashion so the rest of the line is undisturbed), and place the
> >> cursor on the new blank line.  This is incorrect behaviour, as it
> >> breaks entry of BASH here-strings when the user's intent is
> >> still ambiguous.
> >
> > Thanks for your report.  Indeed, sh-mode does not know about
> > here-strings at all.  I guess waiting for "<<[^<]" is a good idea.
> > Patches welcome for Emacs-23.2.
>
> I've now made it look for a space after the << -- that feels more
> natural than << and then some arbitrary character.  And as far as I can
> tell, all the shells handle
>
> foo <<EOF
> EOF
>
> and
>
> foo << EOF
> EOF
>
> identically.
>
> --
> (domestic pets only, the antidote for overdose, milk.)
>    bloggy blog: http://lars.ingebrigtsen.no
>

[-- Attachment #2: Type: text/html, Size: 2133 bytes --]

bug#3226: here-document syntax inference and completion is broken

[Lars Ingebrigtsen]

Paul Nienaber <phox@phox.ca> writes:

> While I use here-strings more than here-docs by a huge margin I would
> also like to not negatively impact people using the former a lot, so
> in light of that I'd like to suggest going with Stefan's suggestion,
> importantly because it's also totally valid to trip on '<<-' and
> '<<[A-Za-z]', whereas here-strings are always only '<<<'.  Also
> '<<EOF' is the overwhelmingly more common form that I see vs '<< EOF'.

The additional wrinkle here is that sh-mode will insert the
sh-here-document-word when expanding.

So if the user types <<X, what should that be expanded to?  Just
appending EOF would be bad, and replacing the X with EOF would be bad,
and expanding to

foo <<X

X

would be bad, and prompting for what to expand to would be pretty
pointless, too.

I guess is "X" is "E", then expanding to

foo <<EOF

EOF

would be nice, though.  I think I'll add that.

But I didn't see any way to make the <<[^<] thing work in any intuitive
way.  If anybody has an idea here, I'm all ears.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#3226: here-document syntax inference and completion is broken

[Paul Nienaber]

Ah yeah totally fair.

Definitely do cover '<<-' as well (=> '<<-EOF\n\nEOF') for the strip-leading-tabs here-doc behaviour :)

Thanks again.

~Paul

> On Oct 15, 2019, at 18:40, Lars Ingebrigtsen <larsi@gnus.org> wrote:
> 
> Paul Nienaber <phox@phox.ca> writes:
> 
>> While I use here-strings more than here-docs by a huge margin I would
>> also like to not negatively impact people using the former a lot, so
>> in light of that I'd like to suggest going with Stefan's suggestion,
>> importantly because it's also totally valid to trip on '<<-' and
>> '<<[A-Za-z]', whereas here-strings are always only '<<<'.  Also
>> '<<EOF' is the overwhelmingly more common form that I see vs '<< EOF'.
> 
> The additional wrinkle here is that sh-mode will insert the
> sh-here-document-word when expanding.
> 
> So if the user types <<X, what should that be expanded to?  Just
> appending EOF would be bad, and replacing the X with EOF would be bad,
> and expanding to
> 
> foo <<X
> 
> X
> 
> would be bad, and prompting for what to expand to would be pretty
> pointless, too.
> 
> I guess is "X" is "E", then expanding to
> 
> foo <<EOF
> 
> EOF
> 
> would be nice, though.  I think I'll add that.
> 
> But I didn't see any way to make the <<[^<] thing work in any intuitive
> way.  If anybody has an idea here, I'm all ears.
> 
> -- 
> (domestic pets only, the antidote for overdose, milk.)
>   bloggy blog: http://lars.ingebrigtsen.no

bug#3226: here-document syntax inference and completion is broken

[Lars Ingebrigtsen]

Paul Nienaber <phox@phox.ca> writes:

> Definitely do cover '<<-' as well (=> '<<-EOF\n\nEOF') for the
> strip-leading-tabs here-doc behaviour :)

Sure; done now.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#3226: here-document syntax inference and completion is broken

[Stefan Monnier]

> I guess is "X" is "E", then expanding to
>
> foo <<EOF
>
> EOF
>
> would be nice, though.  I think I'll add that.

More important is likely support for

    foo <<'EOF'

    EOF

But yes, that sounds good.

> But I didn't see any way to make the <<[^<] thing work in any
> intuitive way.

Same here, which is why I didn't implement it at the time (and doing it
on "<< " like you did is a pretty good compromise, I think).


        Stefan