bug#22717: 25.0.91; sh-mode indentation bug with opening parens at the beginnings of lines, and apostrophes

bug#22717: 25.0.91; sh-mode indentation bug with opening parens at the beginnings of lines, and apostrophes

[Phil Sainty]

I've come up with the following test case:

#!/bin/sh
echo "
(" # Opening parenthesis at the beginning of a line

if true; then
     # Here's an apostrophe
fi # ...and now this will be indented -- aligned with the apostrophe


If you auto-indent that, the final line gets indented to align
with the column containing the apostrophe in the preceding comment.


I've been working through a very large shell script trying to
eliminate indentation issues, and I discovered that quotes in
comments could cause problems with the indentation of later
lines of code.

Balanced apostrophes 'like so' seem to be ok in general, but
sometimes unbalanced quotes would cause subsequent issues.

I noticed the opening parenthesis bit while trying to narrow
this down to a small test case, and have subsequently confirmed
that inserting a space before all such line-starting opening
parens eliminates all of my problems with commented quotes.

My initial workaround had been to convert unbalanced commented
apostrophes into a unicode RIGHT SINGLE QUOTATION MARK, and
that resolved most of the problems I was seeing.

I found one instance in my large script where double quotes were
an issue, such that changing "emacsclient -a ''" in a comment
(including both types of quote) into just emacsclient -a '' (i.e.
removing the double quotes) resolved the issue.

I also saw occasional (but not consistent) issues with `symbol'
style quotes, and I ended up replacing the backticks with normal
apostrophes for all of those instances.

(But again, the opening paren seems to be the trigger.)

This was mostly with 24.5, but I've just checked my test case
with the current emacs-25 branch from git, and replicated the
problem.

I found one other consistent indentation issue, which I'll post
as a separate bug, as it seems like a different issue.


-Phil






In GNU Emacs 25.0.91.1 (x86_64-unknown-linux-gnu, X toolkit, Xaw3d 
scroll bars)
  of 2016-02-18 built on shodan
Repository revision: 14aec913ac3f0dd408487c0e8327403e0f239964
Windowing system distributor 'The X.Org Foundation', version 11.0.11701000
System Description:	Ubuntu 15.04

Configured using:
  'configure --prefix=/home/phil/emacs/trunk/usr/local
  --with-x-toolkit=lucid --without-sound'

Configured features:
XAW3D XPM JPEG TIFF GIF PNG RSVG IMAGEMAGICK DBUS GSETTINGS NOTIFY
GNUTLS LIBXML2 FREETYPE XFT ZLIB TOOLKIT_SCROLL_BARS LUCID X11

Important settings:
   value of $LANG: en_NZ.UTF-8
   value of $XMODIFIERS: @im=ibus
   locale-coding-system: utf-8

Major mode: Shell-script

Minor modes in effect:
   sh-electric-here-document-mode: t
   show-paren-mode: t
   minibuffer-depth-indicate-mode: t
   winner-mode: t
   global-hl-line-mode: t
   tooltip-mode: t
   global-eldoc-mode: t
   electric-indent-mode: t
   mouse-wheel-mode: t
   menu-bar-mode: t
   file-name-shadow-mode: t
   global-font-lock-mode: t
   font-lock-mode: t
   auto-composition-mode: t
   auto-encryption-mode: t
   auto-compression-mode: t
   line-number-mode: t
   transient-mark-mode: t

Recent messages:
For information about GNU Emacs and the GNU system, type C-h C-a.
GNU Emacs 25.0.91.1 (x86_64-unknown-linux-gnu, X toolkit, Xaw3d scroll 
bars) of 2016-02-18
Mark set [5 times]
Setting up indent for shell type sh
Indentation variables are now local.
Indentation setup for shell type sh
Mark activated
Indenting region...done
Undo!
funcall-interactively: End of buffer

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message dired-x dired format-spec rfc822
mml mml-sec epg epg-config gnus-util mm-decode mm-bodies mm-encode
mail-parse rfc2231 mailabbrev gmm-utils mailheader sendmail rfc2047
rfc2045 ietf-drums mm-util help-fns help-mode easymenu cl-loaddefs pcase
cl-lib mail-prsvr mail-utils sh-script smie executable paren mb-depth
winner ring hl-line time-date mule-util tooltip eldoc electric uniquify
ediff-hook vc-hooks lisp-float-type mwheel x-win term/common-win x-dnd
tool-bar dnd fontset image regexp-opt fringe tabulated-list newcomment
elisp-mode lisp-mode prog-mode register page menu-bar rfn-eshadow timer
select scroll-bar mouse jit-lock font-lock syntax facemenu font-core
frame cl-generic cham georgian utf-8-lang misc-lang vietnamese tibetan
thai tai-viet lao korean japanese eucjp-ms cp51932 hebrew greek romanian
slovak czech european ethiopic indian cyrillic chinese charscript
case-table epa-hook jka-cmpr-hook help simple abbrev minibuffer
cl-preloaded nadvice loaddefs button faces cus-face macroexp files
text-properties overlay sha1 md5 base64 format env code-pages mule
custom widget hashtable-print-readable backquote dbusbind inotify
dynamic-setting system-font-setting font-render-setting x-toolkit x
multi-tty make-network-process emacs)

Memory information:
((conses 16 93900 8317)
  (symbols 48 20374 0)
  (miscs 40 79 221)
  (strings 32 17885 4311)
  (string-bytes 1 490436)
  (vectors 16 12632)
  (vector-slots 8 440791 5202)
  (floats 8 168 90)
  (intervals 56 264 0)
  (buffers 976 11)
  (heap 1024 24743 910))

bug#22717: 25.0.91; sh-mode / forward-comment bug with opening parens at the beginnings of lines, and quotes in comments

[Phil Sainty]

On 18/02/16 02:18, Phil Sainty wrote:
> I've come up with the following test case:
> 
> #!/bin/sh
> echo "
> (" # Opening parenthesis at the beginning of a line
> 
> if true; then
>     # Here's an apostrophe
> fi # ...and now this will be indented -- aligned with the apostrophe
> 
> 
> If you auto-indent that, the final line gets indented to align
> with the column containing the apostrophe in the preceding comment.


Taking another look at this, I've narrowed it down to the behaviour
of `forward-comment':

With point at the beginning of the "fi" line, running
(forward-comment -1) moves point to the end of the preceding line,
after the word "apostrophe".

If any of the following changes are made, the (forward-comment -1)
call will move point to the "#" at the beginning of the comment:

* Remove the ' from the comment
* Add an additional ' to the comment line (i.e. a balanced quantity)
* Change the line with the opening parenthesis so that the lines does
  not begin with the paren.

I note that adding a closing paren (so they are balanced) does not
make a difference.

The issue occurs with double-quote and back-quote characters as well
as apostrophes. Any of: ' " `

Similarly, we can substitute a square or curly opening parenthesis
for the round opening paren. Any of ( [ {

I've tested this with version 24.5, 25.2rc2, and master.


-Phil


p.s. The actual indentation code is presumably irrelevant, but the
sequence of events for that was:

smie-indent-line calls smie-indent-calculate which runs
smie-indent-functions until a non-nil value is found.

In this instance it is smie-indent-keyword which returns a value,
and the key part is (let* ((parent (smie-backward-sexp token)))

token here is "fi" and calling (smie-backward-sexp "fi") manually with
point at the beginning of that line, we can observe that point moves
to the apostrophe inside the comment.  If we remove the apostrophe,
point moves to the start of the corresponding "if" instead.

smie-backward-sexp returns (t 96 "'") in the (t POS TOKEN) format.

Ultimately it is smie-next-sexp which produces the value, apparently
failing to skip comments during its processing.  The docstring says
"NEXT-TOKEN is a function of no argument that moves forward by one
token (after skipping comments if needed)". NEXT-TOKEN is compiled
elisp produced by (indirect-function smie-backward-token-function)
where smie-backward-token-function is sh-smie-sh-backward-token.

About the first thing to happen in sh-smie-sh-backward-token is
(forward-comment (- (point))) to skip backwards over as many comments
as possible.

bug#22717: 25.0.91; sh-mode indentation bug with opening parens at the beginnings of lines, and apostrophes

[Noam Postavsky]

tags 22717 fixed
close 22717 27.1
quit

Phil Sainty <psainty@orcon.net.nz> writes:

> I've come up with the following test case:
>
> #!/bin/sh
> echo "
> (" # Opening parenthesis at the beginning of a line
>
> if true; then
>     # Here's an apostrophe
> fi # ...and now this will be indented -- aligned with the apostrophe
>
>
> If you auto-indent that, the final line gets indented to align
> with the column containing the apostrophe in the preceding comment.

This is fixed in master, probably by [1: 14b9558752].

[1: 14b9558752]: 2017-12-12 23:03:00 -0500
  * src/syntax.c (find_defun_start): Use syntax-ppss
  https://git.savannah.gnu.org/cgit/emacs.git/commit/?id=14b95587520959c5b54356547a0a69932a9bb480