bug#66903: 29.1; shell-mode directory tracking incorrectly handles double slashes

bug#66903: 29.1; shell-mode directory tracking incorrectly handles double slashes

[Allen Li]

[-- Attachment #1: Type: text/plain, Size: 805 bytes --]

Reproduce:

(assume ~/bin exists)

1. M-x shell RET
2. cd ~//bin RET
3. M-: default-directory RET

Expected:

default-directory is ~/bin

Actual:

default-directory is /bin

It seems like the directory tracker uses Emacs's convention for double
slashes "resetting" a path to root, but most shells treat double slashes
the same as single slashes.

It looks like this should be fixed in `shell-cd` (which should also fix
`shell-process-pushd`) by implementing "custom" path handling (compared
to Emacs default path handling behavior).

In GNU Emacs 29.1 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.24.37,
 cairo version 1.16.0) of 2023-09-03, modified by Debian built on
 kokoro-ubuntu
Windowing system distributor 'The X.Org Foundation', version 11.0.12101007
System Description: Debian GNU/Linux rodete

[-- Attachment #2: Type: text/html, Size: 949 bytes --]

bug#66903: 29.1; shell-mode directory tracking incorrectly handles double slashes

[Eli Zaretskii]

severity 66903 wishlist
thanks

> From: Allen Li <darkfeline@felesatra.moe>
> Date: Thu, 2 Nov 2023 17:14:36 -0700
> 
> (assume ~/bin exists)
> 
> 1. M-x shell RET
> 2. cd ~//bin RET
> 3. M-: default-directory RET
> 
> Expected:
> 
> default-directory is ~/bin
> 
> Actual:
> 
> default-directory is /bin

This is the standard Emacs behavior: double slash makes everything
before that ignored.

> It seems like the directory tracker uses Emacs's convention for double
> slashes "resetting" a path to root, but most shells treat double slashes
> the same as single slashes.

"M-x shell" is not a shell, it is an Emacs interface to a shell.
Thus, it should be free to implement behavior not present in the
shell, if we consider it convenient for Emacs users.

> It looks like this should be fixed in `shell-cd` (which should also fix
> `shell-process-pushd`) by implementing "custom" path handling (compared
> to Emacs default path handling behavior).

This is not a bug.  I'm okay with adding a new optional feature that
would disable the above behavior, but it must be off by default.

bug#66903: 29.1; shell-mode directory tracking incorrectly handles double slashes

[Allen Li]

[-- Attachment #1: Type: text/plain, Size: 1239 bytes --]

On Fri, Nov 3, 2023 at 12:16 AM Eli Zaretskii <eliz@gnu.org> wrote:

>
> "M-x shell" is not a shell, it is an Emacs interface to a shell.
> Thus, it should be free to implement behavior not present in the
> shell, if we consider it convenient for Emacs users.
>

The behavior it is supposed to implement is that it tracks the working
directory of the shell.
Currently, it fails to do so for most shells that a user would use.
In fact, it fails to do so for the default value `/bin/bash`.


> > It looks like this should be fixed in `shell-cd` (which should also fix
> > `shell-process-pushd`) by implementing "custom" path handling (compared
> > to Emacs default path handling behavior).
>
> This is not a bug.  I'm okay with adding a new optional feature that
> would disable the above behavior, but it must be off by default.
>

This is absolutely a bug.  The feature, where shell-mode attempts to track
the working
directory of its subshell, fails to work for most shells in existence
(e.g., derivatives of POSIX sh, csh, zsh, etc.) and including the default
shell.

I agree it may be an option, but I disagree that it should be off by
default, given that the
default configured shell Bash requires it.

[-- Attachment #2: Type: text/html, Size: 1779 bytes --]

bug#66903: 29.1; shell-mode directory tracking incorrectly handles double slashes

[Eli Zaretskii]

> From: Allen Li <darkfeline@felesatra.moe>
> Date: Fri, 3 Nov 2023 00:22:26 -0700
> Cc: 66903@debbugs.gnu.org
> 
> On Fri, Nov 3, 2023 at 12:16 AM Eli Zaretskii <eliz@gnu.org> wrote:
> 
>  "M-x shell" is not a shell, it is an Emacs interface to a shell.
>  Thus, it should be free to implement behavior not present in the
>  shell, if we consider it convenient for Emacs users.
> 
> The behavior it is supposed to implement is that it tracks the working directory of the shell.
> Currently, it fails to do so for most shells that a user would use.
> In fact, it fails to do so for the default value `/bin/bash`.
>  
>  > It looks like this should be fixed in `shell-cd` (which should also fix
>  > `shell-process-pushd`) by implementing "custom" path handling (compared
>  > to Emacs default path handling behavior).
> 
>  This is not a bug.  I'm okay with adding a new optional feature that
>  would disable the above behavior, but it must be off by default.
> 
> This is absolutely a bug.  The feature, where shell-mode attempts to track the working 
> directory of its subshell, fails to work for most shells in existence 
> (e.g., derivatives of POSIX sh, csh, zsh, etc.) and including the default shell. 
> 
> I agree it may be an option, but I disagree that it should be off by default, given that the 
> default configured shell Bash requires it.

I think we disagree about what should be fixed: the value of
default-directory or the directory to which "cd ~//bin" switches.

bug#66903: 29.1; shell-mode directory tracking incorrectly handles double slashes

[Eshel Yaron via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Hi,

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Allen Li <darkfeline@felesatra.moe>
>> Date: Fri, 3 Nov 2023 00:22:26 -0700
>> Cc: 66903@debbugs.gnu.org
>>
>> On Fri, Nov 3, 2023 at 12:16 AM Eli Zaretskii <eliz@gnu.org> wrote:
>>
>>  "M-x shell" is not a shell, it is an Emacs interface to a shell.
>>  Thus, it should be free to implement behavior not present in the
>>  shell, if we consider it convenient for Emacs users.
>>
>> The behavior it is supposed to implement is that it tracks the working directory of the shell.
>> Currently, it fails to do so for most shells that a user would use.
>> In fact, it fails to do so for the default value `/bin/bash`.
>>
>>  > It looks like this should be fixed in `shell-cd` (which should also fix
>>  > `shell-process-pushd`) by implementing "custom" path handling (compared
>>  > to Emacs default path handling behavior).
>>
>>  This is not a bug.  I'm okay with adding a new optional feature that
>>  would disable the above behavior, but it must be off by default.
>>
>> This is absolutely a bug.  The feature, where shell-mode attempts to track the working
>> directory of its subshell, fails to work for most shells in existence
>> (e.g., derivatives of POSIX sh, csh, zsh, etc.) and including the default shell.
>>
>> I agree it may be an option, but I disagree that it should be off by default, given that the
>> default configured shell Bash requires it.
>
> I think we disagree about what should be fixed: the value of
> default-directory or the directory to which "cd ~//bin" switches.

IIUC what you're implying here is that "cd ~//bin" should actually
switch the shell's current directory to /bin instead of ~/bin, is that
right?

I just wanted to note that this is not really an option with M-x shell,
because it is the shell, and not Emacs, that interprets "cd" and changes
directory.  Emacs just tracks certain shell commands and tries to keep
the buffer's `default-directory` up to date with the shell's directory.


Best,

Eshel

bug#66903: 29.1; shell-mode directory tracking incorrectly handles double slashes

[Eli Zaretskii]

> From: Eshel Yaron <me@eshelyaron.com>
> Cc: Allen Li <darkfeline@felesatra.moe>,  66903@debbugs.gnu.org
> Date: Fri, 03 Nov 2023 09:35:26 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > I think we disagree about what should be fixed: the value of
> > default-directory or the directory to which "cd ~//bin" switches.
> 
> IIUC what you're implying here is that "cd ~//bin" should actually
> switch the shell's current directory to /bin instead of ~/bin, is that
> right?

Yes.

> I just wanted to note that this is not really an option with M-x shell,
> because it is the shell, and not Emacs, that interprets "cd" and changes
> directory.

You assume that shell.el must send everything the user types verbatim
to the shell?

bug#66903: 29.1; shell-mode directory tracking incorrectly handles double slashes

[Eshel Yaron via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Eshel Yaron <me@eshelyaron.com>
>> Cc: Allen Li <darkfeline@felesatra.moe>,  66903@debbugs.gnu.org
>> Date: Fri, 03 Nov 2023 09:35:26 +0100
>>
>> Eli Zaretskii <eliz@gnu.org> writes:
>>
>> > I think we disagree about what should be fixed: the value of
>> > default-directory or the directory to which "cd ~//bin" switches.
>>
>> IIUC what you're implying here is that "cd ~//bin" should actually
>> switch the shell's current directory to /bin instead of ~/bin, is that
>> right?
>
> Yes.
>
>> I just wanted to note that this is not really an option with M-x shell,
>> because it is the shell, and not Emacs, that interprets "cd" and changes
>> directory.
>
> You assume that shell.el must send everything the user types verbatim
> to the shell?

No, not "must" and not "everything", but if you think about it
intercepting "cd" and altering its argument would make `M-x shell`
incompatible with other shell interfaces, breaking valid command lines
in weird ways.  Does that make sense?

bug#66903: 29.1; shell-mode directory tracking incorrectly handles double slashes

[Eli Zaretskii]

> From: Eshel Yaron <me@eshelyaron.com>
> Cc: darkfeline@felesatra.moe,  66903@debbugs.gnu.org
> Date: Fri, 03 Nov 2023 13:35:46 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> I just wanted to note that this is not really an option with M-x shell,
> >> because it is the shell, and not Emacs, that interprets "cd" and changes
> >> directory.
> >
> > You assume that shell.el must send everything the user types verbatim
> > to the shell?
> 
> No, not "must" and not "everything", but if you think about it
> intercepting "cd" and altering its argument would make `M-x shell`
> incompatible with other shell interfaces, breaking valid command lines
> in weird ways.  Does that make sense?

It does to me.  To understand why, start Emacs in your HOME directory,
and then type this:

  M-x cd RET ~//bin RET

bug#66903: 29.1; shell-mode directory tracking incorrectly handles double slashes

[Stefan Kangas]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Eshel Yaron <me@eshelyaron.com>
>> Cc: darkfeline@felesatra.moe,  66903@debbugs.gnu.org
>> Date: Fri, 03 Nov 2023 13:35:46 +0100
>>
>> Eli Zaretskii <eliz@gnu.org> writes:
>>
>> >> I just wanted to note that this is not really an option with M-x shell,
>> >> because it is the shell, and not Emacs, that interprets "cd" and changes
>> >> directory.
>> >
>> > You assume that shell.el must send everything the user types verbatim
>> > to the shell?
>>
>> No, not "must" and not "everything", but if you think about it
>> intercepting "cd" and altering its argument would make `M-x shell`
>> incompatible with other shell interfaces, breaking valid command lines
>> in weird ways.  Does that make sense?
>
> It does to me.  To understand why, start Emacs in your HOME directory,
> and then type this:
>
>   M-x cd RET ~//bin RET

It would be highly dangerous to start mangling command lines to shells
run under `M-x shell' in incompatible ways.  Not to mention that such a
change would be backwards-incompatible.  Consider the command

    sudo rm -rf ~//usr

This is not a contrived example, BTW, but more or less exactly a command
I have run in the last year.  I don't think I used //, but double
slashes are always collapsed in POSIX, so I can't be sure.  It's not
something I look for that when cobbling together shell commands in
POSIX-compatible shells (bash and zsh here).

So I think we should not do that.

bug#66903: 29.1; shell-mode directory tracking incorrectly handles double slashes

[Visuwesh]

[வெள்ளி நவம்பர் 03, 2023] Stefan Kangas wrote:

> It would be highly dangerous to start mangling command lines to shells
> run under `M-x shell' in incompatible ways.  Not to mention that such a
> change would be backwards-incompatible.  Consider the command
>
>     sudo rm -rf ~//usr
>
> This is not a contrived example, BTW, but more or less exactly a command
> I have run in the last year.  I don't think I used //, but double
> slashes are always collapsed in POSIX, so I can't be sure.  It's not
> something I look for that when cobbling together shell commands in
> POSIX-compatible shells (bash and zsh here).
>
> So I think we should not do that.

A more common scenario I can think of is output of commands not
sanitising the filenames to change the double slashes to a single slash.
When you copy such a filename from the output to the comint prompt as
part of a command and experience a different behaviour when you send the
command, it would be annoying IMO.  I copy the filename from the output
precisely to avoid typing-and-editing, I don't want to do it more.

bug#66903: 29.1; shell-mode directory tracking incorrectly handles double slashes

[Allen Li]

[-- Attachment #1: Type: text/plain, Size: 1008 bytes --]

On Fri, Nov 3, 2023 at 12:26 AM Eli Zaretskii <eliz@gnu.org> wrote:

> I think we disagree about what should be fixed: the value of
> default-directory or the directory to which "cd ~//bin" switches.
>

Sorry, let me clarify

Currently, `cd ~//bin` will cause the shell (Bash or similar) to change to
~/bin, and Emacs/shell-mode will change `default-directory` to `/bin`.

This bug is saying that Emacs/shell-mode should change `default-directory`
to `~/bin` by default, optionally with a user option that reverts to the
current behavior in case the user has in fact explicitly configured to use
a shell that interprets `cd ~//bin` to mean `cd /bin` (although I am not
aware of any shells that do this (except eshell, which is not relevant to
this issue)).

I (mis?)interpreted your statement to mean that you think the current
behavior is not a bug and should be the default (that Emacs/shell-mode
should change `default-directory` to `/bin` even though the shell changes
to ~/bin).

[-- Attachment #2: Type: text/html, Size: 1308 bytes --]

bug#66903: 29.1; shell-mode directory tracking incorrectly handles double slashes

[Eli Zaretskii]

> From: Allen Li <darkfeline@felesatra.moe>
> Date: Fri, 3 Nov 2023 03:43:35 -0700
> Cc: 66903@debbugs.gnu.org
> 
> Currently, `cd ~//bin` will cause the shell (Bash or similar) to change to ~/bin, and Emacs/shell-mode
> will change `default-directory` to `/bin`.
> 
> This bug is saying that Emacs/shell-mode should change `default-directory` to `~/bin` by default,

I think it should change default-directory to the directory to which
the shell switches, not necessarily ~/bin.

bug#66903: 29.1; shell-mode directory tracking incorrectly handles double slashes

[Allen Li]

[-- Attachment #1: Type: text/plain, Size: 2605 bytes --]

On Fri, Nov 3, 2023 at 5:07 AM Eli Zaretskii <eliz@gnu.org> wrote:

> > From: Allen Li <darkfeline@felesatra.moe>
> > Date: Fri, 3 Nov 2023 03:43:35 -0700
> > Cc: 66903@debbugs.gnu.org
> >
> > Currently, `cd ~//bin` will cause the shell (Bash or similar) to change
> to ~/bin, and Emacs/shell-mode
> > will change `default-directory` to `/bin`.
> >
> > This bug is saying that Emacs/shell-mode should change
> `default-directory` to `~/bin` by default,
>
> I think it should change default-directory to the directory to which
> the shell switches, not necessarily ~/bin.
>

That sounds different from what you're implying in your other emails.
Using sentences here is confusing.

The current behavior is:

Case 1
input: ~//bin
shell: ~/bin
default-directory: /bin

I'm suggesting

Case 2
input: ~//bin
shell: ~/bin
default-directory: ~/bin

When you say "I think it should change default-directory to the directory
to which
the shell switches, not necessarily ~/bin.", that implies to me Case 2

But your other emails suggest that you want:

Case 3
input: ~//bin (Emacs rewrites to /bin before sending to shell)
shell: /bin
default-directory: /bin

Assuming that you want Case 3, I think that is horrible for multiple
reasons.  Firstly, a user would expect their shell to behave as in Case 2;
Case 3 is a surprising and semantically different rewrite of user input.
Furthermore, we would either have to rewrite all paths not just paths to
`cd`, which would be infeasible and error-prone, or surprise the user again
that Emacs is rewriting paths to `cd` specifically, but not, say, `rm` or
`mv` or many other commands (e.g., `mv ~//bin ~//lib`).  Furthermore, the
only shell input rewriting that Emacs currently does is for history
substitutions, and the syntax for this is identical to common shells;
therefore, users can expect that Emacs does not do any surprising semantic
rewrites of their shell input.  Case 3 would be violating this expectation.

For completeness, there is also

Case 4
input: ~//bin
shell: /bin (normal shell behavior)
default-directory: /bin

However, I am aware of zero shells in common use that behave like Case 4.
Even if we consider it, it should not be the default.

Also for completeness, there is a nuclear solution option, which is to
replace all of the cd, pushd, popd logic that shell-mode currently
implements and always run M-x dirs after every cd, pushd, popd command.
This naturally results in the behavior of Case 2, but also supports Case 4
for hypothetical shells that have said behavior.

[-- Attachment #2: Type: text/html, Size: 3581 bytes --]

bug#66903: 29.1; shell-mode directory tracking incorrectly handles double slashes

[Eli Zaretskii]

> From: Allen Li <darkfeline@felesatra.moe>
> Date: Fri, 3 Nov 2023 13:54:09 -0700
> Cc: 66903@debbugs.gnu.org
> 
> That sounds different from what you're implying in your other emails.  Using sentences here is
> confusing.

That's because your original report was confusing.