bug#46466: 28.0.50; Tramp hangs with bad prompt even when using /bin/sh

bug#46466: 28.0.50; Tramp hangs with bad prompt even when using /bin/sh

[Erik Hetzner]

Hello,

1. Create a new user: `adduser test`
2. `sudo su test` and ensure that home directory is completely empty
3. `echo PS1=badprompt > .bash_profile
4. Switch back to main user
5. Start emacs with emacs -q
6. C-x C-f /ssh:test@localhost:

Expected result: dired for test user home dir.

Actual result: tramp hangs.

This kind of issue is well documented on the internet (stackexchange,
etc.) but I haven't seen a bug for it.

As I understand it, because `tramp-default-remote-shell' is set to
/bin/sh, tramp should ignore `.bash_profile`. When I look at running
shells, all I see is /bin/sh, so it's not clear to me where/why/how
/bin/bash is being invoked.

Thank you!

best, Erik Hetzner

In GNU Emacs 28.0.50 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.24.20, cairo version 1.16.0)
 of 2021-02-10 built on lcy01-amd64-026
Repository revision: 4fba79feee58e074d112bb47467913f9aec089c7
Repository branch: feature/native-comp
Windowing system distributor 'The X.Org Foundation', version 11.0.12009000
System Description: Ubuntu 20.04.2 LTS

Configured using:
 'configure --prefix=/snap/emacs/current/usr --with-x-toolkit=gtk3
 --without-xaw3d --with-modules --with-cairo --with-nativecomp
 'CFLAGS=-isystem/build/emacs/parts/emacs/install/usr/include
 -isystem/build/emacs/parts/emacs/install/usr/include/x86_64-linux-gnu
 -O2' 'CPPFLAGS=-isystem/build/emacs/parts/emacs/install/usr/include
 -isystem/build/emacs/parts/emacs/install/usr/include/x86_64-linux-gnu'
 'LDFLAGS=-L/build/emacs/parts/emacs/install/lib
 -L/build/emacs/parts/emacs/install/usr/lib
 -L/build/emacs/parts/emacs/install/lib/x86_64-linux-gnu
 -L/build/emacs/parts/emacs/install/usr/lib/x86_64-linux-gnu''

Configured features:
ACL CAIRO DBUS FREETYPE GIF GLIB GMP GNUTLS GPM GSETTINGS HARFBUZZ JPEG
JSON LCMS2 LIBOTF LIBSELINUX LIBSYSTEMD LIBXML2 M17N_FLT MODULES
NATIVE_COMP NOTIFY INOTIFY PDUMPER PNG RSVG SOUND THREADS TIFF
TOOLKIT_SCROLL_BARS X11 XDBE XIM XPM GTK3 ZLIB

Important settings:
  value of $LC_MONETARY: en_US.UTF-8
  value of $LC_NUMERIC: en_US.UTF-8
  value of $LC_TIME: en_US.UTF-8
  value of $LANG: en_US.UTF-8
  value of $XMODIFIERS: @im=ibus
  locale-coding-system: utf-8-unix

Major mode: Lisp Interaction

Minor modes in effect:
  shell-dirtrack-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  transient-mark-mode: t

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message rmc puny dired dired-loaddefs
rfc822 mml mml-sec epa derived epg epg-config gnus-util rmail
rmail-loaddefs text-property-search mm-decode mm-bodies mm-encode
mail-parse rfc2231 mailabbrev gmm-utils mailheader sendmail rfc2047
rfc2045 ietf-drums mm-util mail-prsvr mail-utils cl-print tramp-adb
tramp-archive tramp-ftp tramp-gvfs zeroconf url-util url-parse url-vars
dbus xml thingatpt help-fns radix-tree help-mode easymenu tramp-cmds
tramp-cache tramp-sh tramp tramp-loaddefs trampver tramp-integration
files-x tramp-compat shell pcomplete comint ansi-color ring parse-time
iso8601 time-date ls-lisp format-spec auth-source cl-seq eieio
eieio-core cl-macs eieio-loaddefs cl-loaddefs cl-lib password-cache json
subr-x map seq byte-opt gv bytecomp byte-compile cconv iso-transl
tooltip eldoc electric uniquify ediff-hook vc-hooks lisp-float-type
mwheel term/x-win x-win term/common-win x-dnd tool-bar dnd fontset image
regexp-opt fringe tabulated-list replace newcomment text-mode elisp-mode
lisp-mode prog-mode register page tab-bar menu-bar rfn-eshadow isearch
timer select scroll-bar mouse jit-lock font-lock syntax facemenu
font-core term/tty-colors frame minibuffer cl-generic cham georgian
utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao korean
japanese eucjp-ms cp51932 hebrew greek romanian slovak czech european
ethiopic indian cyrillic chinese composite charscript charprop
case-table epa-hook jka-cmpr-hook help simple abbrev obarray
cl-preloaded nadvice button loaddefs faces cus-face pcase macroexp files
window text-properties overlay sha1 md5 base64 format env code-pages
mule custom widget hashtable-print-readable backquote threads dbusbind
inotify lcms2 dynamic-setting system-font-setting font-render-setting
cairo move-toolbar gtk x-toolkit x multi-tty make-network-process
nativecomp emacs)

Memory information:
((conses 16 104276 7827)
 (symbols 48 9051 0)
 (strings 32 31208 3140)
 (string-bytes 1 1013359)
 (vectors 16 16649)
 (vector-slots 8 350770 9051)
 (floats 8 37 235)
 (intervals 56 362 0)
 (buffers 984 13))

bug#46466: 28.0.50; Tramp hangs with bad prompt even when using /bin/sh

[Michael Albinus]

Erik Hetzner <egh@e6h.org> writes:

> Hello,

Hi Erik,

> 1. Create a new user: `adduser test`
> 2. `sudo su test` and ensure that home directory is completely empty
> 3. `echo PS1=badprompt > .bash_profile
> 4. Switch back to main user
> 5. Start emacs with emacs -q
> 6. C-x C-f /ssh:test@localhost:
>
> Expected result: dired for test user home dir.
>
> Actual result: tramp hangs.
>
> This kind of issue is well documented on the internet (stackexchange,
> etc.) but I haven't seen a bug for it.
>
> As I understand it, because `tramp-default-remote-shell' is set to
> /bin/sh, tramp should ignore `.bash_profile`. When I look at running
> shells, all I see is /bin/sh, so it's not clear to me where/why/how
> /bin/bash is being invoked.

What is /bin/sh, both locally and remote? In my case, running Fedora 33,
I see

--8<---------------cut here---------------start------------->8---
# ls -l /bin/sh
lrwxrwxrwx. 1 root root 4 Jul 27  2020 /bin/sh -> bash
--8<---------------cut here---------------end--------------->8---

> Thank you!
>
> best, Erik Hetzner

Best regards, Michael.

bug#46466: 28.0.50; Tramp hangs with bad prompt even when using /bin/sh

[Erik Hetzner]

Hi Michael,

On Sat, 13 Feb 2021 00:53:35 -0800,
Michael Albinus <michael.albinus@gmx.de> wrote:
> 
> […]
> 
> What is /bin/sh, both locally and remote? In my case, running Fedora 33,
> I see
> 
> --8<---------------cut here---------------start------------->8---
> # ls -l /bin/sh
> lrwxrwxrwx. 1 root root 4 Jul 27  2020 /bin/sh -> bash
> --8<---------------cut here---------------end--------------->8---

Good question, I should have mentioned that. I have tested this on
both debian and ubuntu, where /bin/sh is linked to dash

$ ls -l /bin/sh
lrwxrwxrwx 1 root root 4 Dec 10 05:23 /bin/sh -> dash

As a follow up, I suspect that in some way this is expected behavior.
I see that the *sshx* connection method specifies the shell to run
when connecting, whereas the *ssh* connection method connects without
specifying a command, and therefore starts a default remote shell.

This suggests a fix which turns out to works. The behavior is fixed,
 for me, by adding `("-o" "RemoteCommand='%l'")` to the list of
`tramp-login-args' for `tramp-methods' (see below).

It is also possible to set the PS1 value directly with this remote
command, e.g. `("-o" "RemoteCommand='PS1=\"$ \" %l'")`, which might
provide even more safety against unusual prompts.

Is this a patch that would be welcome? It would fix cases where users
had more elaborate prompts in their default shell and where /bin/sh
was a simpler shell (debian and ubuntu, at least). Unusual prompts are
more and more popular, but users to do not usually enable them for
/bin/sh. However, it might have effects that I am not aware of.

```
(add-to-list 'tramp-methods
             `("ssh"
               (tramp-login-program        "ssh")
               (tramp-login-args           (("-l" "%u") ("-p" "%p") ("%c")
				            ("-e" "none")
                                            ("-o" "RemoteCommand='%l'") ("%h")))
               (tramp-async-args           (("-q")))
               (tramp-direct-async         t)
               (tramp-remote-shell         ,tramp-default-remote-shell)
               (tramp-remote-shell-login   ("-l"))
               (tramp-remote-shell-args    ("-c"))))
```

best, Erik

bug#46466: 28.0.50; Tramp hangs with bad prompt even when using /bin/sh

[Michael Albinus]

Erik Hetzner <egh@e6h.org> writes:

> Hi Michael,

Hi Erik,

>> What is /bin/sh, both locally and remote? In my case, running Fedora 33,
>> I see
>>
>> --8<---------------cut here---------------start------------->8---
>> # ls -l /bin/sh
>> lrwxrwxrwx. 1 root root 4 Jul 27  2020 /bin/sh -> bash
>> --8<---------------cut here---------------end--------------->8---
>
> Good question, I should have mentioned that. I have tested this on
> both debian and ubuntu, where /bin/sh is linked to dash
>
> $ ls -l /bin/sh
> lrwxrwxrwx 1 root root 4 Dec 10 05:23 /bin/sh -> dash

Hmm, this doesn't count then. You have reported, that .bash_profile was
used for user "test". What is the default shell for that user?

> As a follow up, I suspect that in some way this is expected behavior.
> I see that the *sshx* connection method specifies the shell to run
> when connecting, whereas the *ssh* connection method connects without
> specifying a command, and therefore starts a default remote shell.

"sshx" is for special cases, where the default values of "ssh" don't work.

> This suggests a fix which turns out to works. The behavior is fixed,
>  for me, by adding `("-o" "RemoteCommand='%l'")` to the list of
> `tramp-login-args' for `tramp-methods' (see below).

We cannot hijack the argument "RemoteCommand" by default. There are
several reasons.

- People might have own values for that argument. For example, on my
  QNAP NAS devices it isn't possible to use ssh out-of-the-box for
  scripting, because QNAP starts the "Console Management" program, which
  is in the way. In order to fix this, I have added for all my QNAP NAS
  devices the following line in the respective .ssh/config entries:

        RemoteCommand /bin/sh -noprofile -noediting -norc -i

- If the RemoteCommand argument is set, it isn't possible any longer to
  use direct async processes in Tramp. See (info "(tramp) Remote processes")

> It is also possible to set the PS1 value directly with this remote
> command, e.g. `("-o" "RemoteCommand='PS1=\"$ \" %l'")`, which might
> provide even more safety against unusual prompts.

Maybe, but this should be better done with the SetEnv argument.

> Is this a patch that would be welcome? It would fix cases where users
> had more elaborate prompts in their default shell and where /bin/sh
> was a simpler shell (debian and ubuntu, at least). Unusual prompts are
> more and more popular, but users to do not usually enable them for
> /bin/sh. However, it might have effects that I am not aware of.

As said, the patch cannot be applied as-is. However, I'm open for all
proposals, because this is indeed a common problem users blame Tramp about.

> best, Erik

Best regards, Michael.

bug#46466: 28.0.50; Tramp hangs with bad prompt even when using /bin/sh

[Erik Hetzner]

Hi Michael,

On Sun, 14 Feb 2021 06:37:29 -0800,
Michael Albinus <michael.albinus@gmx.de> wrote:
> 
> […]
> 
> Hmm, this doesn't count then. You have reported, that .bash_profile was
> used for user "test". What is the default shell for that user?

Sorry, I also should have mentioned that! The default shell is bash,
which seems to be the underlying issue. tramp starts a bash shell
before execing /bin/sh.

> […]
> 
> We cannot hijack the argument "RemoteCommand" by default. There are
> several reasons.
> 
> - People might have own values for that argument. For example, on my
>   QNAP NAS devices it isn't possible to use ssh out-of-the-box for
>   scripting, because QNAP starts the "Console Management" program, which
>   is in the way. In order to fix this, I have added for all my QNAP NAS
>   devices the following line in the respective .ssh/config entries:
> 
>         RemoteCommand /bin/sh -noprofile -noediting -norc -i
> 
> - If the RemoteCommand argument is set, it isn't possible any longer to
>   use direct async processes in Tramp. See (info "(tramp) Remote processes")

I see, thanks!

> > It is also possible to set the PS1 value directly with this remote
> > command, e.g. `("-o" "RemoteCommand='PS1=\"$ \" %l'")`, which might
> > provide even more safety against unusual prompts.
> 
> Maybe, but this should be better done with the SetEnv argument.
> 
> > Is this a patch that would be welcome? It would fix cases where users
> > had more elaborate prompts in their default shell and where /bin/sh
> > was a simpler shell (debian and ubuntu, at least). Unusual prompts are
> > more and more popular, but users to do not usually enable them for
> > /bin/sh. However, it might have effects that I am not aware of.
> 
> As said, the patch cannot be applied as-is. However, I'm open for all
> proposals, because this is indeed a common problem users blame Tramp about.

Thank you so much for these explanations. Tramp is a great product and
I know that it has made a lot of accommodations to a heterogeneous
environment to make it work so well! I’ll keep exploring possible
solutions for this. In the end, it might just be a suggestion in the
manual or improvements to error reporting.

Thanks again for your help.

best, Erik

bug#46466: 28.0.50; Tramp hangs with bad prompt even when using /bin/sh

[Michael Albinus]

Erik Hetzner <egh@e6h.org> writes:

> Hi Michael,

Hi Erik,

> Thank you so much for these explanations. Tramp is a great product and
> I know that it has made a lot of accommodations to a heterogeneous
> environment to make it work so well! I’ll keep exploring possible
> solutions for this. In the end, it might just be a suggestion in the
> manual or improvements to error reporting.

This bug is still open. Is there something else I could do? Otherwise,
I'd like to close it.

> Thanks again for your help.
>
> best, Erik

Best regards, Michael.

bug#46466: 28.0.50; Tramp hangs with bad prompt even when using /bin/sh

[Erik Hetzner]

[-- Attachment #1: Type: text/plain, Size: 1558 bytes --]

Hi Michael,

On Mon, 14 Jun 2021 05:10:14 -0700,
Michael Albinus <michael.albinus@gmx.de> wrote:
> 
> Erik Hetzner <egh@e6h.org> writes:
> 
> > Hi Michael,
> 
> Hi Erik,
> 
> > Thank you so much for these explanations. Tramp is a great product and
> > I know that it has made a lot of accommodations to a heterogeneous
> > environment to make it work so well! I’ll keep exploring possible
> > solutions for this. In the end, it might just be a suggestion in the
> > manual or improvements to error reporting.
> 
> This bug is still open. Is there something else I could do? Otherwise,
> I'd like to close it.

Thank you for the reminder. While revisiting this issue I came across
an alternate fix: adding %l to the tramp-login-args for ssh. I suspect
there is a reason this is not a viable fix, but I’m not sure what it
is. I’ve been using it locally for a while. There is a patch below, or
here is a snippet to try locally:

(setq tramp-methods (assoc-delete-all "ssh" tramp-methods #'string=))
(add-to-list 'tramp-methods
             `("ssh"
               (tramp-login-program        "ssh")
               (tramp-login-args           (("-l" "%u") ("-p" "%p") ("%c")
        			            ("-e" "none") ("%h") ("%l")))
               (tramp-async-args           (("-q")))
               (tramp-direct-async         t)
               (tramp-remote-shell         ,tramp-default-remote-shell)
               (tramp-remote-shell-login   ("-l"))
               (tramp-remote-shell-args    ("-c")))))

best, Erik


[-- Attachment #2: 0001-Use-remote-shell-from-the-start-in-Tramp-ssh-method-.patch --]
[-- Type: text/plain, Size: 1041 bytes --]

From b20a16ae9001bf1f89ffb6767095dedef04f56b3 Mon Sep 17 00:00:00 2001
From: Erik Hetzner <egh@e6h.org>
Date: Wed, 16 Jun 2021 07:23:08 -0700
Subject: [PATCH] Use remote shell from the start in Tramp ssh method
 (bug#46466)

    * lisp/net/tramp-sh.el (tramp--with-startup): Add %l to
    tramp-login-args for ssh tramp-method.
---
 lisp/net/tramp-sh.el | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lisp/net/tramp-sh.el b/lisp/net/tramp-sh.el
index b613ad3f8e..921af6e7c3 100644
--- a/lisp/net/tramp-sh.el
+++ b/lisp/net/tramp-sh.el
@@ -224,7 +224,7 @@ tramp-scp-strict-file-name-checking
               `("ssh"
                 (tramp-login-program        "ssh")
                 (tramp-login-args           (("-l" "%u") ("-p" "%p") ("%c")
-				             ("-e" "none") ("%h")))
+				             ("-e" "none") ("%h") ("%l")))
                 (tramp-async-args           (("-q")))
                 (tramp-direct-async         t)
                 (tramp-remote-shell         ,tramp-default-remote-shell)
-- 
2.30.2

bug#46466: 28.0.50; Tramp hangs with bad prompt even when using /bin/sh

[Michael Albinus]

Erik Hetzner <egh@e6h.org> writes:

> Hi Michael,

Hi Erik,

> Thank you for the reminder. While revisiting this issue I came across
> an alternate fix: adding %l to the tramp-login-args for ssh. I suspect
> there is a reason this is not a viable fix, but I’m not sure what it
> is.

Good that it works for you. However, it can't be applied in general. If
somebody has declared RemoteCommand in ~/.ssh/config, it will
fail. Furthermore, it will fail for direct async processes.

You might keep your local setting, or you might use the "sshx" method
instead of "ssh", which sets RemoteCommand. Does this work for you?

> best, Erik

Best regards, Michael.

bug#46466: 28.0.50; Tramp hangs with bad prompt even when using /bin/sh

[Erik Hetzner]

Hi Michael,

On Sat, 19 Jun 2021 01:48:37 -0700,
Michael Albinus <michael.albinus@gmx.de> wrote:
> 
> Erik Hetzner <egh@e6h.org> writes:
> 
> > Hi Michael,
> 
> Hi Erik,
> 
> > Thank you for the reminder. While revisiting this issue I came across
> > an alternate fix: adding %l to the tramp-login-args for ssh. I suspect
> > there is a reason this is not a viable fix, but I’m not sure what it
> > is.
> 
> Good that it works for you. However, it can't be applied in general. If
> somebody has declared RemoteCommand in ~/.ssh/config, it will
> fail. Furthermore, it will fail for direct async processes.
> 
> You might keep your local setting, or you might use the "sshx" method
> instead of "ssh", which sets RemoteCommand. Does this work for you?

Thanks very much for the explanation. I couldn’t figure out if it
would be an issue with direct async.

This is a solution for me, thank you. Please do close the ticket.

I would like the documentation to be a little clearer about the fact
that the remote default shell is used in many methods (and not
/bin/sh). I will explore filing a patch for that.

Thank you for your help and for maintaining the very useful tramp!

best, Erik

bug#46466: 28.0.50; Tramp hangs with bad prompt even when using /bin/sh

[Michael Albinus]

Erik Hetzner <egh@e6h.org> writes:

> Hi Michael,

Hi Erik,
 
>> You might keep your local setting, or you might use the "sshx" method
>> instead of "ssh", which sets RemoteCommand. Does this work for you?
>
> Thanks very much for the explanation. I couldn’t figure out if it
> would be an issue with direct async.
>
> This is a solution for me, thank you. Please do close the ticket.

Done. Just being curious: Did you test the "sshx" method?

> I would like the documentation to be a little clearer about the fact
> that the remote default shell is used in many methods (and not
> /bin/sh). I will explore filing a patch for that.

Appreciated. Tramp 2.5.1 is scheduled for release end of June, if your
patch arrives next days, I'll integrate it. But there's no rush, I could
integrate it also anytime later.

> Thank you for your help and for maintaining the very useful tramp!
>
> best, Erik

Best regards, Michael.