bug#21477: 24.4; Imenu - improper parse of shell function names

bug#21477: 24.4; Imenu - improper parse of shell function names

[Matthew Persico (BLOOMBERG/ 731 LEX)]

[-- Attachment #1: Type: text/plain, Size: 8515 bytes --]

--text follows this line--

1) Create a file with this text:
# -*- sh -*-
this_is_parsed ()
{
echo 1
}

function this_is_also_parsed
{
echo 1
}

this-is-not-parsed ()
{
echo 1
}

function this-is-also-not-parsed
{
echo 1
}

2) Load it into emacs that has imenu.

3) Look for functions in the imenu list. Only the ones with '_' show up. The ones with '-' do not.

In GNU Emacs 24.4.1 (x86_64-unknown-linux-gnu, GTK+ Version 3.8.9)
of 2015-09-01 on nylnxbldo2
Windowing system distributor `Hummingbird - Open Text', version 11.0.13830
System Description:     Red Hat Enterprise Linux Server release 5.8 (Tikanga)

Configured using:
`configure --prefix=/opt/bb --libdir=/opt/bb/lib64
-x-includes=/opt/bb/include -x-libraries=/opt/bb/lib64
--without-selinux --with-png=no --with-gif=no --without-gsettings
CFLAGS=-m64 CPPFLAGS=-I/opt/bb/include 'LDFLAGS=-m64 -L/opt/bb/lib64
-Wl,--enable-new-dtags,-R/opt/bb/lib64''

Important settings:
value of $LANG: C
locale-coding-system: nil

Major mode: Shell-script

Minor modes in effect:
sh-electric-here-document-mode: t
flyspell-mode: t
exordium-require-final-newline-mode: t
exordium-show-trailing-whitespace-mode: t
global-git-gutter-mode: t
diff-auto-refine-mode: t
global-git-commit-mode: t
shell-dirtrack-mode: t
projectile-global-mode: t
projectile-mode: t
global-auto-complete-mode: t
auto-complete-mode: t
recentf-mode: t
delete-selection-mode: t
show-paren-mode: t
global-linum-mode: t
linum-mode: t
tooltip-mode: t
electric-indent-mode: t
mouse-wheel-mode: t
menu-bar-mode: t
file-name-shadow-mode: t
global-font-lock-mode: t
font-lock-mode: t
auto-composition-mode: t
auto-encryption-mode: t
auto-compression-mode: t
column-number-mode: t
line-number-mode: t
transient-mark-mode: t

Recent input:
_ <return> <C-home> <C-home> C-s g i t - g o - f <backspace> 
_ <backspace> <backspace> _ f <backspace> C-s C-s C-g 
C-g C-g C-g C-g C-s M O P C-s C-s <C-home> <C-home> 
<C-home> <escape> C-t g o t C-_ C-_ <escape> C-r g 
i t - g o f i l e <return> <up> <left> <right> <right> 
<right> <right> <right> <backspace> _ <down> <up> <right> 
<right> _ <return> SPC SPC SPC SPC SPC SPC SPC SPC 
<C-home> <C-home> <escape> C-r g i t - g o _ f i l 
e <return> <up> <left> <down> <right> <down> <down> 
<down> <right> <right> <right> <right> <backspace> 
+ _ <backspace> <backspace> _ <return> ! <up> <up> 
<up> <up> <up> <up> <up> <up> <up> <up> <up> <up> <up> 
<up> <up> <up> <up> <up> <up> <up> <up> <up> <up> <up> 
<up> <up> <up> <up> <up> <up> <up> <up> <up> <up> <up> 
<up> <up> <up> <up> <up> <up> <up> <up> <up> <up> <up> 
<up> <up> <up> <up> <up> <up> <up> <up> <up> <up> <up> 
<up> <up> <up> <up> <up> <up> <up> <up> <up> <up> <up> 
<up> <up> <up> <up> <up> <up> <up> <up> <up> <up> <up> 
<up> <up> <up> <up> <up> <up> <up> <up> <up> <up> <up> 
<down> <down> <down> <down> <down> <up> <down> C-x 
C-z C-x C-s <help-echo> <help-echo> <help-echo> <help-echo> 
<help-echo> <help-echo> <help-echo> <help-echo> <help-echo> 
<help-echo> <help-echo> <down-mouse-1> <mouse-1> <down-mouse-5> 
<mouse-5> <double-down-mouse-5> <double-mouse-5> <triple-down-mouse-5> 
<triple-mouse-5> <triple-down-mouse-5> <triple-mouse-5> 
<down-mouse-5> <mouse-5> <double-down-mouse-5> <double-mouse-5> 
<triple-down-mouse-5> <triple-mouse-5> <down-mouse-1> 
<mouse-1> <down-mouse-3> <mouse-3> <down-mouse-3> <mouse-3> 
<down-mouse-1> <mouse-1> <help-echo> <escape> x e m 
a c s - r e p o t <backspace> r t - b u g <return> 
<backspace> <backspace> <backspace> <backspace> <backspace> 
<backspace> <backspace> <backspace> <backspace> <backspace> 
<backspace> <backspace> <backspace> <backspace> <backspace> 
<backspace> <backspace> <backspace> <backspace> r e 
p <tab> o <tab> r <tab> <return>

Recent messages:
call-interactively: End of buffer
user-error: End of defaults; no next item [2 times]
Mark set
Replaced 8 occurrences
Auto-saving...done
Saving file /home/mpersico5/personal/bin/gitfuncs...
Wrote /home/mpersico5/personal/bin/gitfuncs
Quit [2 times]
delete-backward-char: Text is read-only [3 times]
Making completion list... [2 times]

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug sendmail misearch multi-isearch
sh-script smie executable flymake disp-table flyspell ispell
tomorrow-night-bright-theme init-powerline powerline
powerline-separators color powerline-themes init-elisp page-break-lines
init-ruby init-python init-javascript js2-imenu-extras
js2-mode-expansions js2-mode js2-old-indent js-mode-expansions js json
imenu init-rtags-helm init-rtags auto-complete-c-headers rtags-ac rtags
repeat init-yasnippet yasnippet init-bde-style init-cpp cpp init-xml
init-org ob-dot ob-C cc-mode-expansions cc-mode cc-fonts cc-guess
cc-menus cc-cmds cc-styles cc-align cc-engine cc-vars cc-defs ob-python
ob-sh ob-ruby ob-perl ox-gfm ox-odt rng-loc rng-uri rng-parse rng-match
rng-dt rng-util rng-pttrn nxml-parse nxml-ns nxml-enc xmltok nxml-util
ox-beamer ox-md ox-latex ox-icalendar ox-html ox-ascii ox-publish ox
org-element org-bullets init-markdown markdown-mode init-shell
init-prog-mode init-git diminish git-gutter-fringe fringe-helper
git-gutter magit-blame magit-stash magit-bisect magit-remote
magit-commit magit-sequence magit magit-log magit-apply magit-wip
magit-diff smerge-mode diff-mode magit-core magit-process magit-popup
magit-mode help-mode magit-git crm magit-section magit-utils git-commit
log-edit message rfc822 mml mml-sec mm-decode mm-bodies mm-encode
mail-parse rfc2231 rfc2047 rfc2045 ietf-drums mailabbrev mail-utils
gmm-utils mailheader pcvs-util add-log with-editor tramp-sh server
init-dired find-dired wdired init-flb-mode init-helm-projectile
project-explorer helm-mode filenotify es-windows face-remap es-lib
es-lib-total-line es-lib-lexical es-lib-buffer-local-set-key
es-lib-text-navigate es-lib-core-functions es-lib-core-macros helm-swoop
helm-projectile helm-files rx image-dired tramp tramp-compat
tramp-loaddefs trampver shell dired-x dired-aux ffap helm-tags
helm-bookmark helm-adaptive helm-info bookmark pp helm-external helm-net
browse-url xml url url-proxy url-privacy url-expand url-methods
url-history url-cookie url-domsuf url-util url-parse auth-source
gnus-util mm-util mail-prsvr password-cache url-vars mailcap
helm-buffers helm-match-plugin helm-grep helm-regexp helm-plugin
helm-elscreen helm-utils helm-locate helm-help helm-types projectile
compile ibuf-ext ibuffer dash helm helm-source eieio eieio-core helm-lib
dired init-autocomplete auto-complete-config auto-complete edmacro
kmacro cl-macs init-ido recentf tree-widget wid-edit ido init-util
warnings popup goto-chg init-window-manager windmove init-linum nlinum
init-font-lock init-look-and-feel vlf-setup expand-region
text-mode-expansions the-org-mode-expansions er-basic-expansions
thingatpt expand-region-core expand-region-custom delsel paren cl gv
init-themes fill-column-indicator byte-opt hilinum-mode cl-loaddefs
cl-lib linum tomorrow-night-theme color-theme-tomorrow org org-macro
org-footnote org-pcomplete pcomplete org-list org-faces org-entities
noutline outline easy-mmode org-version ob-emacs-lisp ob ob-tangle
ob-ref ob-lob ob-table ob-exp org-src ob-keys ob-comint comint
ansi-color ring ob-core ob-eval org-compat org-macs org-loaddefs
format-spec find-func cal-menu calendar cal-loaddefs init-progress-bar
init-prefs init-environment init-lib bytecomp byte-compile cconv advice
help-fns info easymenu package epg-config time-date tooltip electric
uniquify ediff-hook vc-hooks lisp-float-type mwheel x-win x-dnd tool-bar
dnd fontset image regexp-opt fringe tabulated-list newcomment lisp-mode
prog-mode register page menu-bar rfn-eshadow timer select scroll-bar
mouse jit-lock font-lock syntax facemenu font-core frame cham georgian
utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao korean
japanese hebrew greek romanian slovak czech european ethiopic indian
cyrillic chinese case-table epa-hook jka-cmpr-hook help simple abbrev
minibuffer nadvice loaddefs button faces cus-face macroexp files
text-properties overlay sha1 md5 base64 format env code-pages mule
custom widget hashtable-print-readable backquote make-network-process
dbusbind gfilenotify dynamic-setting font-render-setting move-toolbar
gtk x-toolkit x multi-tty emacs)

Memory information:
((conses 16 476713 44382)
(symbols 48 55322 0)
(miscs 40 303 389)
(strings 32 138900 24315)
(string-bytes 1 4089758)
(vectors 16 53992)
(vector-slots 8 870426 22979)
(floats 8 475 265)
(intervals 56 1430 880)
(buffers 960 14)
(heap 1024 207714 1127))


[-- Attachment #2: Type: text/html, Size: 12229 bytes --]

bug#21477: 24.4; Imenu - improper parse of shell function names

[Stefan Kangas]

tags 21477 + moreinfo
thanks

"Matthew Persico (BLOOMBERG/ 731 LEX)" <mpersico5@bloomberg.net> writes:

> 1) Create a file with this text:
> # -*- sh -*-
> this_is_parsed ()
> {
> echo 1
> }
>
> function this_is_also_parsed
> {
> echo 1
> }
>
> this-is-not-parsed ()
> {
> echo 1
> }
>
> function this-is-also-not-parsed
> {
> echo 1
> }
>
> 2) Load it into emacs that has imenu.
>
> 3) Look for functions in the imenu list. Only the ones with '_' show up. The ones with '-' do not.

I can reproduce this behaviour, but is it really a bug?  Aren't the
names with '-' invalid?

The Bash Reference Manual says:

name

    A word consisting solely of letters, numbers, and underscores, and
    beginning with a letter or underscore. Names are used as shell
    variable and function names. Also referred to as an identifier.

https://www.gnu.org/savannah-checkouts/gnu/bash/manual/bash.html

Best regards,
Stefan Kangas

bug#21477: 24.4; Imenu - improper parse of shell function names

[Matthew Persico (BLOOMBERG/ 919 3RD A)]

[-- Attachment #1: Type: text/plain, Size: 1400 bytes --]

Interesting. I have function names with dashes all over and nothing complains. Not Bash 4.X, not shellcheck. If you ran that file in the post, it would not complain.

Now, you can't use dashes in variables. But functions work just fine.

I wonder if the bash manual needs an update?

From: stefan@marxist.se At: 08/12/20 00:15:40To:  Matthew Persico (BLOOMBERG/ 919 3RD A ) 
Cc:  21477@debbugs.gnu.org
Subject: Re: bug#21477: 24.4; Imenu - improper parse of shell function names

tags 21477 + moreinfo
thanks

"Matthew Persico (BLOOMBERG/ 731 LEX)" <mpersico5@bloomberg.net> writes:

> 1) Create a file with this text:
> # -*- sh -*-
> this_is_parsed ()
> {
> echo 1
> }
>
> function this_is_also_parsed
> {
> echo 1
> }
>
> this-is-not-parsed ()
> {
> echo 1
> }
>
> function this-is-also-not-parsed
> {
> echo 1
> }
>
> 2) Load it into emacs that has imenu.
>
> 3) Look for functions in the imenu list. Only the ones with '_' show up. The 
ones with '-' do not.

I can reproduce this behaviour, but is it really a bug?  Aren't the
names with '-' invalid?

The Bash Reference Manual says:

name

    A word consisting solely of letters, numbers, and underscores, and
    beginning with a letter or underscore. Names are used as shell
    variable and function names. Also referred to as an identifier.

https://www.gnu.org/savannah-checkouts/gnu/bash/manual/bash.html

Best regards,
Stefan Kangas



[-- Attachment #2: Type: text/html, Size: 3223 bytes --]

bug#21477: 24.4; Imenu - improper parse of shell function names

[Stefan Kangas]

"Matthew Persico (BLOOMBERG/ 919 3RD A)" <mpersico5@bloomberg.net>
writes:

> Interesting. I have function names with dashes all over and nothing
> complains. Not Bash 4.X, not shellcheck. If you ran that file in the
> post, it would not complain.
>
> Now, you can't use dashes in variables. But functions work just fine.
>
> I wonder if the bash manual needs an update?

Would you be willing to ask the bash maintainers about this?

Best regards,
Stefan Kangas

bug#21477: 24.4; Imenu - improper parse of shell function names

[Matthew Persico (BLOOMBERG/ 919 3RD A)]

[-- Attachment #1: Type: text/plain, Size: 739 bytes --]

Do you have a particular list to ping, or do I try bash-testers or bug-bash?


From: stefan@marxist.se At: 08/12/20 10:08:35To:  Matthew Persico (BLOOMBERG/ 919 3RD A ) 
Cc:  21477@debbugs.gnu.org
Subject: Re: bug#21477: 24.4; Imenu - improper parse of shell function names

"Matthew Persico (BLOOMBERG/ 919 3RD A)" <mpersico5@bloomberg.net>
writes:

> Interesting. I have function names with dashes all over and nothing
> complains. Not Bash 4.X, not shellcheck. If you ran that file in the
> post, it would not complain.
>
> Now, you can't use dashes in variables. But functions work just fine.
>
> I wonder if the bash manual needs an update?

Would you be willing to ask the bash maintainers about this?

Best regards,
Stefan Kangas



[-- Attachment #2: Type: text/html, Size: 2258 bytes --]

bug#21477: 24.4; Imenu - improper parse of shell function names

[Andreas Schwab]

On Aug 12 2020, Matthew Persico (BLOOMBERG/ 919 3RD A) wrote:

> I can reproduce this behaviour, but is it really a bug?  Aren't the
> names with '-' invalid?

It's a POSIX extension, enforced in POSIX mode:

  13. Function names must be valid shell 'name's.  That is, they may not
     contain characters other than letters, digits, and underscores, and
     may not start with a digit.  Declaring a function with an invalid
     name causes a fatal syntax error in non-interactive shells.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

bug#21477: 24.4; Imenu - improper parse of shell function names

[Matthew Persico (BLOOMBERG/ 919 3RD A)]

[-- Attachment #1: Type: text/plain, Size: 1799 bytes --]

I asked on bug-bash:

The bash-20191127 snapshot updated the manpage documentation for a
function definition (to align with reality). It is now defined as:

function fname [()] compound-command [redirection]

and includes the description:

When in posix mode, fname must be a valid shell name and may not be the
name of one of the POSIX special builtins. In default mode, a function
name can be any unquoted shell word that does not contain $.

For context:

word  -  A sequence of characters considered as a single unit by the
shell. Also known as a token.

name  -  A word consisting only of alphanumeric characters and
underscores, and beginning with an alphabetic character or an
underscore. Also referred to as an identifier.

--
Eli Schwartz
Bug Wrangler and Trusted User

I guess you could go full-on and change the function name match pattern to be a "word" or just add the '-' to it for now to be safe.


From: schwab@linux-m68k.org At: 08/12/20 10:54:40To:  Matthew Persico (BLOOMBERG/ 919 3RD A ) 
Cc:  stefan@marxist.se,  21477@debbugs.gnu.org
Subject: Re: bug#21477: 24.4; Imenu - improper parse of shell function names

On Aug 12 2020, Matthew Persico (BLOOMBERG/ 919 3RD A) wrote:

> I can reproduce this behaviour, but is it really a bug?  Aren't the
> names with '-' invalid?

It's a POSIX extension, enforced in POSIX mode:

  13. Function names must be valid shell 'name's.  That is, they may not
     contain characters other than letters, digits, and underscores, and
     may not start with a digit.  Declaring a function with an invalid
     name causes a fatal syntax error in non-interactive shells.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."



[-- Attachment #2: Type: text/html, Size: 3826 bytes --]

bug#21477: 24.4; Imenu - improper parse of shell function names

[Stefan Kangas]

"Matthew Persico (BLOOMBERG/ 919 3RD A)" <mpersico5@bloomberg.net>
writes:

> Do you have a particular list to ping, or do I try bash-testers or bug-bash?

I'm not sure.  Any of them should work presumably.

Best regards,
Stefan Kangas

bug#21477: 24.4; Imenu - improper parse of shell function names

[Stefan Kangas]

Andreas Schwab <schwab@linux-m68k.org> writes:

> It's a POSIX extension, enforced in POSIX mode:
>
>   13. Function names must be valid shell 'name's.  That is, they may not
>      contain characters other than letters, digits, and underscores, and
>      may not start with a digit.  Declaring a function with an invalid
>      name causes a fatal syntax error in non-interactive shells.

Right.  Well, then adding syntax highlighting here would in a way
encourage people to write non-POSIX conformant shell code, which even if
it happens to work in Bash sounds like a bad idea.

So I'm leaning towards closing this bug report as wontfix.

Any other opinions?

Best regards,
Stefan Kangas

bug#21477: 24.4; Imenu - improper parse of shell function names

[Matthew Persico (BLOOMBERG/ 919 3RD A)]

[-- Attachment #1: Type: text/plain, Size: 1405 bytes --]

I am not sure what is going on here, but I just loaded up this file and it parsed just fine.

Now, I KNOW that this didn't work when I reported it. And Stefan was able to reproduce it.

I wonder if Stefan can still reproduce?

Since I can't reproduce, I have no issue with closing. If it regresses, I'll open a new issue, making a note of the versions of various packages, which would certainly help diagnosis.

Stefan, it's been a while - can you send me a link to my original bug report?


From: stefan@marxist.se At: 08/12/20 12:00:09To:  Matthew Persico (BLOOMBERG/ 919 3RD A ) ,  schwab@linux-m68k.org
Cc:  21477@debbugs.gnu.org
Subject: Re: bug#21477: 24.4; Imenu - improper parse of shell function names

Andreas Schwab <schwab@linux-m68k.org> writes:

> It's a POSIX extension, enforced in POSIX mode:
>
>   13. Function names must be valid shell 'name's.  That is, they may not
>      contain characters other than letters, digits, and underscores, and
>      may not start with a digit.  Declaring a function with an invalid
>      name causes a fatal syntax error in non-interactive shells.

Right.  Well, then adding syntax highlighting here would in a way
encourage people to write non-POSIX conformant shell code, which even if
it happens to work in Bash sounds like a bad idea.

So I'm leaning towards closing this bug report as wontfix.

Any other opinions?

Best regards,
Stefan Kangas



[-- Attachment #2: Type: text/html, Size: 3056 bytes --]

bug#21477: 24.4; Imenu - improper parse of shell function names

[Stefan Kangas]

"Matthew Persico (BLOOMBERG/ 919 3RD A)" <mpersico5@bloomberg.net>
writes:

> I am not sure what is going on here, but I just loaded up this file
> and it parsed just fine.

Which version of Emacs are you using (M-x emacs-version), and could you
show an example of the code where you see the function name highlighted?

> Stefan, it's been a while - can you send me a link to my original bug report?

https://debbugs.gnu.org/cgi/bugreport.cgi?bug=21477

> Right.  Well, then adding syntax highlighting here would in a way
> encourage people to write non-POSIX conformant shell code, which even if
> it happens to work in Bash sounds like a bad idea.
>
> So I'm leaning towards closing this bug report as wontfix.
>
> Any other opinions?

I'm still undecided on this point, so other opinions would be welcome.

Perhaps this is something we should support optionally only, so we don't
help unsuspecting users shoot themselves in the foot.

Best regards,
Stefan Kangas

bug#21477: 24.4; Imenu - improper parse of shell function names

[Matthew Persico (BLOOMBERG/ 919 3RD A)]

[-- Attachment #1: Type: text/plain, Size: 1239 bytes --]

At a doctors appointment now. Will do so later this evening. 

Sent from Bloomberg Professional for iPhone

----- Original Message -----
From: Stefan Kangas <stefan@marxist.se>
To: MATTHEW PERSICO, schwab@linux-m68k.org
CC: 21477@debbugs.gnu.org
At: 20-Aug-2020 14:41:27


"Matthew Persico (BLOOMBERG/ 919 3RD A)" <mpersico5@bloomberg.net>
writes:

> I am not sure what is going on here, but I just loaded up this file
> and it parsed just fine.

Which version of Emacs are you using (M-x emacs-version), and could you
show an example of the code where you see the function name highlighted?

> Stefan, it's been a while - can you send me a link to my original bug report?

https://debbugs.gnu.org/cgi/bugreport.cgi?bug=21477

> Right.  Well, then adding syntax highlighting here would in a way
> encourage people to write non-POSIX conformant shell code, which even if
> it happens to work in Bash sounds like a bad idea.
>
> So I'm leaning towards closing this bug report as wontfix.
>
> Any other opinions?

I'm still undecided on this point, so other opinions would be welcome.

Perhaps this is something we should support optionally only, so we don't
help unsuspecting users shoot themselves in the foot.

Best regards,
Stefan Kangas

[-- Attachment #2: Type: text/html, Size: 2619 bytes --]