bug#66338: 30.0.50; grep-commnd set and using an old fish results in empty Copyright files

bug#66338: 30.0.50; grep-commnd set and using an old fish results in empty Copyright files

[Alex Schroeder via Bug reports for GNU Emacs, the Swiss army knife of text editors]

My directories are littered with empty “Copyright” files. I think the
problem is that I use fish version 3.0.2 (current is 3.6.1) and I have
grep-command set.

So, using fish, start Emacs -Q.

(getenv "SHELL") ;; "/usr/bin/fish"
(setq grep-command "grep --color -niH -e ")
M-x grep

You don’t have to actually use grep. Invoking grep calls
grep-compute-defaults which – if grep-command is set – calls grep-probe
which calls process-file-shell-command. The resulting command:

/usr/bin/fish -c "grep --color -niH -e ^Copyright \
/usr/local/share/emacs/30.0.50/etc/HELLO"

If I run this in the shell, it hangs. I terminate with C-d and notice
that the empty file “Copyright” was created. If I quote the regular
expression, no problem:

/usr/bin/fish -c "grep --color -niH -e '^Copyright' \
/usr/local/share/emacs/30.0.50/etc/HELLO"

(This correctly produces a line of output.)

This seems to be a feature of fish. The following command generates an
empty file called “XXX”.

echo ^XXX

In the documentation I see the following regarding the caret, in a
section about feature flags:

“stderr-nocaret was introduced in fish 3.0 (and made the default in
3.3). It makes ^ an ordinary character instead of denoting an stderr
redirection, to make dealing with quoting and such easier. Use 2>
instead. This can no longer be turned off since fish 3.5. The flag can
still be tested for compatibility, but a no-stderr-nocaret value will
simply be ignored.”
https://fishshell.com/docs/current/language.html#featureflags

So this is the problem and it’s already improved in newer versions of
fish.

In my case, however:

alex@melanobombus ~> status features
stderr-nocaret	off	3.0	^ no longer redirects stderr
qmark-noglob	off	3.0	? no longer globs

I have since changed the stderr-nocaret to “on” for my system (and as
seen above, it is going to be mandatory going forward). This seems to do
the job. No change to Emacs was required.

I’m not quite sure what to suggest, here. All I can say is that I was
very angry about these empty Copyright files appearing every now and
then all over my directories. I looked at hooks, I grepped the sources
for "Copyright" (which didn’t match "^Copyright"), I looked at packages,
I just couldn’t find it, and I also couldn’t reliably reproduce it,
since it effectively only created an empty Copyright file the first time
I called grep in an Emacs session.

Given the difficulty in finding the source of the problem, perhaps some
sort of warning message or workaround is appropriate?

Cheers
Alex



In GNU Emacs 30.0.50 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.24.5,
 cairo version 1.16.0) of 2023-10-04 built on melanobombus
Repository revision: fbfdd1e0e3e67f765c6dbf9f61b5f913e55e004e
Repository branch: master
Windowing system distributor 'The X.Org Foundation', version 11.0.12004000
System Description: PureOS

Configured features:
CAIRO DBUS FREETYPE GIF GLIB GMP GNUTLS GPM GSETTINGS HARFBUZZ JPEG JSON
LCMS2 LIBOTF LIBSELINUX LIBXML2 MODULES NOTIFY INOTIFY PDUMPER PNG RSVG
SECCOMP SOUND SQLITE3 THREADS TIFF TOOLKIT_SCROLL_BARS X11 XDBE XIM
XINPUT2 XPM GTK3 ZLIB

Important settings:
  value of $LANG: en_GB.UTF-8
  value of $XMODIFIERS: @im=ibus
  locale-coding-system: utf-8-unix

Major mode: ELisp/l

Minor modes in effect:
  emms-mode-line-mode: t
  emms-playing-time-display-mode: t
  emms-playing-time-mode: t
  edebug-mode: t
  windmove-mode: t
  winner-mode: t
  server-mode: t
  idle-highlight-mode: t
  global-git-gutter-mode: t
  override-global-mode: t
  savehist-mode: t
  desktop-save-mode: t
  display-time-mode: t
  midnight-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  eldoc-mode: t
  show-paren-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  minibuffer-regexp-mode: t
  buffer-read-only: t
  column-number-mode: t
  line-number-mode: t
  transient-mark-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t

Load-path shadows:
/home/alex/.emacs.d/elpa/transient-20230501.1034/transient hides /usr/local/share/emacs/30.0.50/lisp/transient

Features:
(shadow flyspell ispell typo ecomplete mail-extr emacsbug message
yank-media rfc822 mailabbrev gmm-utils mailheader sendmail tabify imenu
man cl-print dired-aux ibuffer ibuffer-loaddefs em-unix em-term term
disp-table shell ehelp em-script em-prompt em-pred em-ls em-hist em-glob
em-extpipe em-cmpl em-dirs em-basic em-banner em-alias esh-mode esh-var
eshell esh-cmd esh-ext esh-opt esh-proc esh-io esh-arg pcomplete
esh-module esh-groups esh-util emms-mpris dbus emms-librefm-stream
emms-librefm-scrobbler emms-playlist-limit emms-i18n emms-history
emms-score emms-stream-info emms-metaplaylist-mode emms-bookmarks
emms-cue emms-mode-line-icon emms-browser sort emms-volume
emms-volume-sndioctl emms-volume-mixerctl emms-volume-pulse
emms-volume-amixer emms-playlist-sort emms-last-played emms-player-xine
emms-player-mpd tq emms-lyrics emms-url emms-streams emms-show-all
emms-tag-editor format-spec emms-tag-tracktag emms-mark emms-mode-line
emms-cache emms-info-native bindat emms-info-exiftool emms-info-tinytag
emms-info-metaflac emms-info-opusinfo emms-info-ogginfo
emms-info-mp3info emms-playlist-mode emms-player-vlc emms-player-mpv
emms-playing-time emms-info emms-later-do emms-player-mplayer
emms-player-simple emms-source-playlist emms-source-file locate dired-x
dired dired-loaddefs emms-setup emms emms-compat files-x edebug debug
backtrace misearch multi-isearch find-func shortdoc comp comp-cstr
warnings grep help-fns radix-tree mediawiki url-cache mm-url gnus
nnheader mail-utils range wid-edit mml mml-sec epa epg rfc6068
epg-config gnus-util time-date mm-decode mm-bodies mm-encode url-http
url-auth mail-parse rfc2231 rfc2047 rfc2045 mm-util ietf-drums
mail-prsvr url-gw nsm brutalist-dark-theme brutalist-theme
brutalist-build move-text rcirc-emojis rx password-generator oddmuse
hi-lock goto-addr url-misc sgml-mode facemenu cl shr pixel-fill kinsoku
url-file puny svg xml dom windmove winner server slime-fancy
slime-indentation slime-cl-indent cl-indent slime-trace-dialog
slime-fontifying-fu slime-package-fu slime-references
slime-compiler-notes-tree slime-scratch slime-presentations bridge
slime-macrostep macrostep slime-mdot-fu slime-enclosing-context
slime-fuzzy slime-fancy-trace slime-fancy-inspector slime-c-p-c
slime-editing-commands slime-autodoc slime-repl elp slime-parse slime
derived lisp-mnt gud apropos compile text-property-search etags fileloop
generator xref project arc-mode archive-mode noutline outline icons pp
comint ansi-osc ansi-color ring hyperspec slime-autoloads mmm-mode
mmm-univ mmm-class mmm-region mmm-auto mmm-vars mmm-utils mmm-compat
advice image-file image-converter idle-highlight-mode thingatpt
git-gutter edmacro kmacro cl-extra help-mode use-package-bind-key
bind-key easy-mmode savehist desktop frameset time skeleton midnight
generic use-package-ensure use-package-core finder-inf
go-eldoc-autoloads go-mode-autoloads pcase mediawiki-autoloads
wgrep-autoloads info package browse-url url url-proxy url-privacy
url-expand url-methods url-history url-cookie generate-lisp-file
url-domsuf url-util mailcap url-handlers url-parse auth-source cl-seq
eieio eieio-core cl-macs password-cache json subr-x map byte-opt gv
bytecomp byte-compile url-vars cl-loaddefs cl-lib rmc iso-transl tooltip
cconv eldoc paren electric uniquify ediff-hook vc-hooks lisp-float-type
elisp-mode mwheel term/x-win x-win term/common-win x-dnd touch-screen
tool-bar dnd fontset image regexp-opt fringe tabulated-list replace
newcomment text-mode lisp-mode prog-mode register page tab-bar menu-bar
rfn-eshadow isearch easymenu timer select scroll-bar mouse jit-lock
font-lock syntax font-core term/tty-colors frame minibuffer nadvice seq
simple cl-generic indonesian philippine cham georgian utf-8-lang
misc-lang vietnamese tibetan thai tai-viet lao korean japanese eucjp-ms
cp51932 hebrew greek romanian slovak czech european ethiopic indian
cyrillic chinese composite emoji-zwj charscript charprop case-table
epa-hook jka-cmpr-hook help abbrev obarray oclosure cl-preloaded button
loaddefs theme-loaddefs faces cus-face macroexp files window
text-properties overlay sha1 md5 base64 format env code-pages mule
custom widget keymap hashtable-print-readable backquote threads dbusbind
inotify lcms2 dynamic-setting system-font-setting font-render-setting
cairo gtk x-toolkit xinput2 x multi-tty move-toolbar
make-network-process emacs)

Memory information:
((conses 16 510904 41722) (symbols 48 27987 1)
 (strings 32 120078 5410) (string-bytes 1 3474453) (vectors 16 50833)
 (vector-slots 8 1390521 192276) (floats 8 256 120)
 (intervals 56 2880 0) (buffers 984 21))

bug#66338: 30.0.50; grep-commnd set and using an old fish results in empty Copyright files

[Dmitry Gutov]

On 04/10/2023 13:09, Alex Schroeder via Bug reports for GNU Emacs, the 
Swiss army knife of text editors wrote:
> My directories are littered with empty “Copyright” files. I think the
> problem is that I use fish version 3.0.2 (current is 3.6.1) and I have
> grep-command set.

Perhaps the problem is that you changed shell-file-name to point to fish?

bug#66338: 30.0.50; grep-commnd set and using an old fish results in empty Copyright files

[Eli Zaretskii]

> Date: Wed, 4 Oct 2023 16:06:27 +0300
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> On 04/10/2023 13:09, Alex Schroeder via Bug reports for GNU Emacs, the 
> Swiss army knife of text editors wrote:
> > My directories are littered with empty “Copyright” files. I think the
> > problem is that I use fish version 3.0.2 (current is 3.6.1) and I have
> > grep-command set.
> 
> Perhaps the problem is that you changed shell-file-name to point to fish?

Exactly.  And that is never a good idea, because we use the semantics
of Bourne shell in these cases.

bug#66338: 30.0.50; grep-commnd set and using an old fish results in empty Copyright files

[Alex Schroeder via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Dmitry Gutov <dmitry@gutov.dev>
>> 
>> Perhaps the problem is that you changed shell-file-name to point to
>> fish?
>
> Exactly. And that is never a good idea, because we use the semantics
> of Bourne shell in these cases.

Well, it is set, but not by me. I checked using the following:

find ~/.emacs.d -name '*.el' -exec grep -H shell-file-name '{}' ';'

There are a few matches in magit, and a single match in my init.el file
where it is part of a huge custom-set-variables →
connection-local-profile-alist →
tramp-connection-local-default-shell-profile → shell-file-name →
"/bin/sh". So I don’t set it to "/usr/bin/fish" myself.

When I start emacs -Q from my terminal emulator, I get an Emacs that has
shell-file-name set to "/usr/bin/fish".

The doc string says that shell-file-name is based on the SHELL
environment variable. That variable is of course set to /usr/bin/fish.

It seems to me that if it so important that Bourne shell semantics be
used, either shell-file-name should not be set automatically, or it
should be temporarily overridden in the cases where we rely on Bourne
shell semantics, or a warning should be printed whenever such code sees
a shell name it doesn’t know to be compatible with (although the
required fixes by a user like me would seem to be many and confusing).

From my point of view, it seems that manually setting shell-file-name to
"/bin/sh" is the only realistic solution and therefore I’d say that
setting this variable from the SHELL environment variable seems like the
wrong thing to do for Emacs. Is there a scenario where this is a good
idea?

For interactive use like M-x shell we already use
explicit-shell-file-name which is based on ESHELL or shell-file-name.
Perhaps that variable should be based on ESHELL, SHELL, and only use
shell-file-name as a last resort and we no longer set shell-file-name
based on the SHELL variable.

Instead, shell-file-name should be set based on a list of known,
compatible shells available, or remain unset and all commands that rely
on it should fail gracefully and inform the user if no compatible shell
was found.

What do you think?

Cheers
Alex

bug#66338: 30.0.50; grep-commnd set and using an old fish results in empty Copyright files

[Eli Zaretskii]

> From: Alex Schroeder <alex@alexschroeder.ch>
> Cc: dmitry@gutov.dev, 66338@debbugs.gnu.org
> Date: Wed, 04 Oct 2023 20:47:35 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> From: Dmitry Gutov <dmitry@gutov.dev>
> >> 
> >> Perhaps the problem is that you changed shell-file-name to point to
> >> fish?
> >
> > Exactly. And that is never a good idea, because we use the semantics
> > of Bourne shell in these cases.
> 
> Well, it is set, but not by me. I checked using the following:
> 
> find ~/.emacs.d -name '*.el' -exec grep -H shell-file-name '{}' ';'
> 
> There are a few matches in magit, and a single match in my init.el file
> where it is part of a huge custom-set-variables →
> connection-local-profile-alist →
> tramp-connection-local-default-shell-profile → shell-file-name →
> "/bin/sh". So I don’t set it to "/usr/bin/fish" myself.
> 
> When I start emacs -Q from my terminal emulator, I get an Emacs that has
> shell-file-name set to "/usr/bin/fish".
> The doc string says that shell-file-name is based on the SHELL
> environment variable. That variable is of course set to /usr/bin/fish.

It should be only set in interactive sessions.  Please review your
system init files, and modify them so that you override the default
value of SHELL only in interactive sessions.

> It seems to me that if it so important that Bourne shell semantics be
> used, either shell-file-name should not be set automatically, or it
> should be temporarily overridden in the cases where we rely on Bourne
> shell semantics, or a warning should be printed whenever such code sees
> a shell name it doesn’t know to be compatible with (although the
> required fixes by a user like me would seem to be many and confusing).
> 
> >From my point of view, it seems that manually setting shell-file-name to
> "/bin/sh" is the only realistic solution and therefore I’d say that
> setting this variable from the SHELL environment variable seems like the
> wrong thing to do for Emacs. Is there a scenario where this is a good
> idea?
> 
> For interactive use like M-x shell we already use
> explicit-shell-file-name which is based on ESHELL or shell-file-name.
> Perhaps that variable should be based on ESHELL, SHELL, and only use
> shell-file-name as a last resort and we no longer set shell-file-name
> based on the SHELL variable.

The Emacs manual says:

  ‘SHELL’
       The name of an interpreter used to parse and execute programs run
       from inside Emacs.  This is used to initialize the variable
       ‘shell-file-name’ (*note Single Shell::).

So Emacs expects this variable to point to a shell that Emacs can use
to run programs, not your interactive shell.  If you want a different
shell to be used in interactive shell sessions, such as "M-x shell",
then you should use the variable explicit-shell-file-name to point to
that other shell (in your case, fish).

> Instead, shell-file-name should be set based on a list of known,
> compatible shells available, or remain unset and all commands that rely
> on it should fail gracefully and inform the user if no compatible shell
> was found.
> 
> What do you think?

I think we already have the facilities to configure this as needed,
see above.

bug#66338: 30.0.50; grep-commnd set and using an old fish results in empty Copyright files

[Alex Schroeder via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Eli Zaretskii <eliz@gnu.org> writes:

> The Emacs manual says:
>
>   ‘SHELL’
>        The name of an interpreter used to parse and execute programs run
>        from inside Emacs.  This is used to initialize the variable
>        ‘shell-file-name’ (*note Single Shell::).
>
> So Emacs expects this variable to point to a shell that Emacs can use
> to run programs, not your interactive shell.  If you want a different
> shell to be used in interactive shell sessions, such as "M-x shell",
> then you should use the variable explicit-shell-file-name to point to
> that other shell (in your case, fish).

I understand that this is what Emacs expects. It seems to me that other
parts of a user’s system might go against this expectation, however. The
man page for login(1) says this, for example:

       Your user and group ID will be set according to their values in
       the /etc/passwd file. The value for $HOME, $SHELL, $PATH,
       $LOGNAME, and $MAIL are set according to the appropriate fields
       in the password entry. Ulimit, umask and nice values may also be
       set according to entries in the GECOS field.

That is to say, if a users runs chsh(1) and decides to use fish, then
SHELL is set to fish, it is inherited to all processes, and Emacs
breaks.

The Emacs manual also says the following in (elisp) Security
Considerations, which is consistent with what you said above (and
unfortunate for users like me):

     Emacs has customization and other variables with similar
     considerations.  For example, if the variable ‘shell-file-name’
     specifies a shell with nonstandard behavior, an Emacs-based
     application may misbehave.

Too bad that there is not even a warning.

I’ll try to close this bug.

bug#66338: 30.0.50; grep-commnd set and using an old fish results in empty Copyright files

[Alex Schroeder via Bug reports for GNU Emacs, the Swiss army knife of text editors]

The systems appears to work as designed. It’s the user’s responsibility
to have a SHELL set that is compatible enough with the Bourne shell.

bug#66338: 30.0.50; grep-commnd set and using an old fish results in empty Copyright files

[Michael Albinus]

Alex Schroeder via "Bug reports for GNU Emacs, the Swiss army knife of
text editors" <bug-gnu-emacs@gnu.org> writes:

Hi Alex,

>> So Emacs expects this variable to point to a shell that Emacs can use
>> to run programs, not your interactive shell.  If you want a different
>> shell to be used in interactive shell sessions, such as "M-x shell",
>> then you should use the variable explicit-shell-file-name to point to
>> that other shell (in your case, fish).
>
> I understand that this is what Emacs expects. It seems to me that other
> parts of a user’s system might go against this expectation, however. The
> man page for login(1) says this, for example:
>
>        Your user and group ID will be set according to their values in
>        the /etc/passwd file. The value for $HOME, $SHELL, $PATH,
>        $LOGNAME, and $MAIL are set according to the appropriate fields
>        in the password entry. Ulimit, umask and nice values may also be
>        set according to entries in the GECOS field.
>
> That is to say, if a users runs chsh(1) and decides to use fish, then
> SHELL is set to fish, it is inherited to all processes, and Emacs
> breaks.

You could always add '(setenv "SHELL" "/bin/sh")' to your Emacs init file.

Best regards, Michael.

bug#66338: 30.0.50; grep-commnd set and using an old fish results in empty Copyright files

[Alex Schroeder via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Of course. From my point of view this is not a question of how to fix it. My main problem is that it took me so long to discover what the problem was.

-- 
Typed on a tiny keyboard. Sorry for being terse.

> On 5 Oct 2023, at 12:27, Michael Albinus <michael.albinus@gmx.de> wrote:
> 
> Alex Schroeder via "Bug reports for GNU Emacs, the Swiss army knife of
> text editors" <bug-gnu-emacs@gnu.org> writes:
> 
> Hi Alex,
> 
>>> So Emacs expects this variable to point to a shell that Emacs can use
>>> to run programs, not your interactive shell.  If you want a different
>>> shell to be used in interactive shell sessions, such as "M-x shell",
>>> then you should use the variable explicit-shell-file-name to point to
>>> that other shell (in your case, fish).
>> 
>> I understand that this is what Emacs expects. It seems to me that other
>> parts of a user’s system might go against this expectation, however. The
>> man page for login(1) says this, for example:
>> 
>>       Your user and group ID will be set according to their values in
>>       the /etc/passwd file. The value for $HOME, $SHELL, $PATH,
>>       $LOGNAME, and $MAIL are set according to the appropriate fields
>>       in the password entry. Ulimit, umask and nice values may also be
>>       set according to entries in the GECOS field.
>> 
>> That is to say, if a users runs chsh(1) and decides to use fish, then
>> SHELL is set to fish, it is inherited to all processes, and Emacs
>> breaks.
> 
> You could always add '(setenv "SHELL" "/bin/sh")' to your Emacs init file.
> 
> Best regards, Michael.