bug#33791: 26.1; Eshell on Windows connecting to GNU/Linux machine using TRAMP and plink: env: ‘c:/home/jordan/test.sh’: No such file or directory

bug#33791: 26.1; Eshell on Windows connecting to GNU/Linux machine using TRAMP and plink: env: ‘c:/home/jordan/test.sh’: No such file or directory

[Jordan Wilson]

Hi,
I'm running Emacs 26.1 on Windows 10. I've replicated this with "emacs
-Q".

When I'm connected to my GNU/Linux machine (using TRAMP and plink) with
eshell, I can't run executables in the local directory. Doing
"./test.sh" returns "env: ‘c:/home/jordan/test.sh’: No such file or
directory".  

I also can't navigate properly using `cd', etc. It returns "No such
directory found via CDPATH environment variable".

Here's an example:

       c:/Users/Jordan $ /plink:jordan@domain.com:/home/jordan/test
       /plink:jordan@domain.com:/home/jordan/test $ ls
       test2       test.sh
       /plink:jordan@domain.com:/home/jordan/test $ ./test.sh
       env: ‘c:/home/jordan/test.sh’: No such file or directory
       /plink:jordan@domain.com:/home/jordan/test $ cd test2
       No such directory found via CDPATH environment variable
       /plink:jordan@domain.com:/home/jordan/test $ cd test2/
       No such directory found via CDPATH environment variable
       /plink:jordan@domain.com:/home/jordan/test/ $ 

Thanks

I don't have these problems trying to connect to "domain.com" this way
on my GNU/Linux install. I've tried the 26.2 pretest on Windows, and I'm
having the same problem.


In GNU Emacs 26.1 (build 1, x86_64-w64-mingw32)
 of 2018-05-29 built on TPW550S
Windowing system distributor 'Microsoft Corp.', version 10.0.17134

Configured using:
 'configure --without-compress-install --without-dbus --with-modules
 'CFLAGS= -O2 -g3''

Configured features:
XPM JPEG TIFF GIF PNG RSVG IMAGEMAGICK SOUND NOTIFY ACL GNUTLS LIBXML2
ZLIB TOOLKIT_SCROLL_BARS MODULES THREADS LCMS2

Important settings:
  value of $LANG: ENG
  locale-coding-system: cp1252

Major mode: Eshell

-- 
Jordan Wilson
    Sent from Gnus v5.13, GNU Emacs 26.1

bug#33791: 26.1; Eshell on Windows connecting to GNU/Linux machine using TRAMP and plink: env: ‘c:/home/jordan/test.sh’: No such file or directory

[Eli Zaretskii]

> From: Jordan Wilson <jordan.t.wilson@gmx.com>
> Date: Tue, 18 Dec 2018 15:00:35 +0000
> 
> I'm running Emacs 26.1 on Windows 10. I've replicated this with "emacs
> -Q".
> 
> When I'm connected to my GNU/Linux machine (using TRAMP and plink) with
> eshell, I can't run executables in the local directory. Doing
> "./test.sh" returns "env: ‘c:/home/jordan/test.sh’: No such file or
> directory".  

I cannot reproduce this, but I'm not on Windows 10.

> I also can't navigate properly using `cd', etc. It returns "No such
> directory found via CDPATH environment variable".

This I can reproduce.  The corresponding command, eshell/cd, doesn't
seem to support remote directories.  Michael, can you look into this?

> Here's an example:
> 
>        c:/Users/Jordan $ /plink:jordan@domain.com:/home/jordan/test
>        /plink:jordan@domain.com:/home/jordan/test $ ls
>        test2       test.sh
>        /plink:jordan@domain.com:/home/jordan/test $ ./test.sh
>        env: ‘c:/home/jordan/test.sh’: No such file or directory

What is in test.sh?  Did you make that script executable?

bug#33791: 26.1; Eshell on Windows connecting to GNU/Linux machine using TRAMP and plink: env: ‘c:/home/jordan/test.sh’: No such file or directory

[Michael Albinus]

Eli Zaretskii <eliz@gnu.org> writes:

>> I also can't navigate properly using `cd', etc. It returns "No such
>> directory found via CDPATH environment variable".
>
> This I can reproduce.  The corresponding command, eshell/cd, doesn't
> seem to support remote directories.  Michael, can you look into this?

I'll try. When I find a Windows machine.

Isn't this bug#24787?

Best regards, Michael.

bug#33791: 26.1; Eshell on Windows connecting to GNU/Linux machine using TRAMP and plink: env: ‘c:/home/jordan/test.sh’: No such file or directory

[Eli Zaretskii]

> From: Michael Albinus <michael.albinus@gmx.de>
> Cc: Jordan Wilson <jordan.t.wilson@gmx.com>,  33791@debbugs.gnu.org
> Date: Sat, 22 Dec 2018 11:25:53 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> I also can't navigate properly using `cd', etc. It returns "No such
> >> directory found via CDPATH environment variable".
> >
> > This I can reproduce.  The corresponding command, eshell/cd, doesn't
> > seem to support remote directories.  Michael, can you look into this?
> 
> I'll try. When I find a Windows machine.
> 
> Isn't this bug#24787?

Yes, definitely looks like it.  But I don't think the discussion of
that bug ended with any conclusions wrt how to fix it, did it?

Thanks.

bug#33791: 26.1; Eshell on Windows connecting to GNU/Linux machine using TRAMP and plink: env: ‘c:/home/jordan/test.sh’: No such file or directory

[Michael Albinus]

Eli Zaretskii <eliz@gnu.org> writes:

>> Isn't this bug#24787?
>
> Yes, definitely looks like it.  But I don't think the discussion of
> that bug ended with any conclusions wrt how to fix it, did it?

No. It is about MS Windows, and it is a minor bug. That's why I didn't
chime in. Maybe it's now time to have a look on it.

(I wish we would have somebody who cares about eshell, regularly)

> Thanks.

Best regards, Michael.

bug#33791: 26.1; Eshell on Windows connecting to GNU/Linux machine using TRAMP and plink: env: ‘c:/home/jordan/test.sh’: No such file or directory

[Eli Zaretskii]

> From: Michael Albinus <michael.albinus@gmx.de>
> Cc: jordan.t.wilson@gmx.com,  33791@debbugs.gnu.org
> Date: Sat, 22 Dec 2018 13:35:47 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> Isn't this bug#24787?
> >
> > Yes, definitely looks like it.  But I don't think the discussion of
> > that bug ended with any conclusions wrt how to fix it, did it?
> 
> No. It is about MS Windows, and it is a minor bug. That's why I didn't
> chime in. Maybe it's now time to have a look on it.

I hope so.  I'm also quite surprised that the problem is specific to
Windows.  If Eshell does handle remote directories in eshell/cd, then
the fact it doesn't work on Windows might be some silly syntax issue,
like some code somewhere expects a directory to begin with a slash or
something.

> (I wish we would have somebody who cares about eshell, regularly)

Seconded.

bug#33791: 26.1; Eshell on Windows connecting to GNU/Linux machine using TRAMP and plink: env: ‘c:/home/jordan/test.sh’: No such file or directory

[Jordan Wilson]

On 2018-12-22 (Sat) at 12:38 (+0200), Eli Zaretskii <eliz@gnu.org> wrote:

>> From: Michael Albinus <michael.albinus@gmx.de>
>> Cc: Jordan Wilson <jordan.t.wilson@gmx.com>,  33791@debbugs.gnu.org
>> Date: Sat, 22 Dec 2018 11:25:53 +0100
>> 
>> Isn't this bug#24787?
>
> Yes, definitely looks like it.  But I don't think the discussion of
> that bug ended with any conclusions wrt how to fix it, did it?

The CDPATH problem seems to be the same bug. Navigating (when in
/plink:jordan@domain.com:/home/jordan/test/ ) by

      cd test2

fails, but
      cd /plink:jordan@domain.com:/home/jordan/test/test2

works as expected, as the OP of that bug reported.

-- 
Jordan Wilson
    Sent from Gnus v5.13, GNU Emacs 26.1

bug#33791: 26.1; Eshell on Windows connecting to GNU/Linux machine using TRAMP and plink: env: ‘c:/home/jordan/test.sh’: No such file or directory

[Jordan Wilson]

Hmm...seems the bug I'm experiencing is more low-lying than eshell. I've
followed it down to C-level. eshell/cd -> cd -> locate-file ->
locate-file-internal. In my case both problems arise from "c:"
being prepended somewhere.

Evaluating:

     (locate-file-internal ".." '("./") nil (lambda (f) (message f)))

while in "/plink:jordan@domain.com:/home/jordan/test" returns
"c:/plink:jordan@domain.com:/home/jordan". Whilst

     (locate-file-internal ".."
            '("/plink:jordan@domain.com:/home/jordan/test")
             nil (lambda (f) (message f)))

correctly returns "/plink:jordan@domain.com:/home/jordan".

It seems the problem is something to do with converting from a
relative to absolute path. Eval'ing (expand-file-name "..") correctly
returns "/plink:jordan@domain.com:/home/jordan", though. Strange.

-- 
Jordan Wilson
    Sent from Gnus v5.13, GNU Emacs 26.1

bug#33791: 26.1; Eshell on Windows connecting to GNU/Linux machine using TRAMP and plink: env: ‘c:/home/jordan/test.sh’: No such file or directory

[Eli Zaretskii]

> From: Jordan Wilson <jordan.t.wilson@gmx.com>
> Cc: 33791@debbugs.gnu.org, michael.albinus@gmx.de
> Date: Sun, 23 Dec 2018 12:40:14 +0000
> 
> Hmm...seems the bug I'm experiencing is more low-lying than eshell. I've
> followed it down to C-level. eshell/cd -> cd -> locate-file ->
> locate-file-internal. In my case both problems arise from "c:"
> being prepended somewhere.
> 
> Evaluating:
> 
>      (locate-file-internal ".." '("./") nil (lambda (f) (message f)))
> 
> while in "/plink:jordan@domain.com:/home/jordan/test" returns
> "c:/plink:jordan@domain.com:/home/jordan". Whilst
> 
>      (locate-file-internal ".."
>             '("/plink:jordan@domain.com:/home/jordan/test")
>              nil (lambda (f) (message f)))
> 
> correctly returns "/plink:jordan@domain.com:/home/jordan".
> 
> It seems the problem is something to do with converting from a
> relative to absolute path. Eval'ing (expand-file-name "..") correctly
> returns "/plink:jordan@domain.com:/home/jordan", though. Strange.

Right you are, thanks.  The problem is that locate-file doesn't
support remote file names.  Does the patch below produce good results?

Michael, do you agree with this solution?  Do you think it's safe
enough to put it on the release branch (it's a regression from a few
years ago)?

diff --git a/lisp/files.el b/lisp/files.el
index eb09a7c..cfe67b4 100644
--- a/lisp/files.el
+++ b/lisp/files.el
@@ -801,9 +801,15 @@ cd
     (setq cd-path (or (parse-colon-path (getenv "CDPATH"))
                       (list "./"))))
   (cd-absolute
-   (or (locate-file dir cd-path nil
-                    (lambda (f) (and (file-directory-p f) 'dir-ok)))
-       (error "No such directory found via CDPATH environment variable"))))
+   (or
+    ;; locate-file doesn't support remote file names, so detect them
+    ;; and support them here by hand.
+    (and (file-name-absolute-p (expand-file-name dir))
+         (file-accessible-directory-p (expand-file-name dir))
+         (expand-file-name dir))
+    (locate-file dir cd-path nil
+                 (lambda (f) (and (file-directory-p f) 'dir-ok)))
+    (error "No such directory found via CDPATH environment variable"))))
 
 (defun directory-files-recursively (dir regexp &optional include-directories)
   "Return list of all files under DIR that have file names matching REGEXP.

bug#33791: 26.1; Eshell on Windows connecting to GNU/Linux machine using TRAMP and plink: env: ‘c:/home/jordan/test.sh’: No such file or directory

[Michael Albinus]

Eli Zaretskii <eliz@gnu.org> writes:

Hi Eli,

> Right you are, thanks.  The problem is that locate-file doesn't
> support remote file names.  Does the patch below produce good results?
>
> Michael, do you agree with this solution?  Do you think it's safe
> enough to put it on the release branch (it's a regression from a few
> years ago)?

Please give me some days, I hope to work on this over the next days. I
know you are busy with Emacs 26.2; if it cannot wait just go on with
your fix.

Btw, what about supporting locate-file by file name handlers? Just an
idea, and it wouldn't work in Emacs 26.

Best regards, Michael.

bug#33791: 26.1; Eshell on Windows connecting to GNU/Linux machine using TRAMP and plink: env: ‘c:/home/jordan/test.sh’: No such file or directory

[Eli Zaretskii]

> From: Michael Albinus <michael.albinus@gmx.de>
> Cc: Jordan Wilson <jordan.t.wilson@gmx.com>,  33791@debbugs.gnu.org
> Date: Sun, 23 Dec 2018 17:42:52 +0100
> 
> > Michael, do you agree with this solution?  Do you think it's safe
> > enough to put it on the release branch (it's a regression from a few
> > years ago)?
> 
> Please give me some days, I hope to work on this over the next days. I
> know you are busy with Emacs 26.2; if it cannot wait just go on with
> your fix.

It can wait, take your time.

> Btw, what about supporting locate-file by file name handlers? Just an
> idea, and it wouldn't work in Emacs 26.

That would be a good goal for the master branch, I think.

bug#33791: 26.1; Eshell on Windows connecting to GNU/Linux machine using TRAMP and plink: env: ‘c:/home/jordan/test.sh’: No such file or directory

[Michael Albinus]

Eli Zaretskii <eliz@gnu.org> writes:

Hi Eli,

>> Evaluating:
>> 
>>      (locate-file-internal ".." '("./") nil (lambda (f) (message f)))
>> 
>> while in "/plink:jordan@domain.com:/home/jordan/test" returns
>> "c:/plink:jordan@domain.com:/home/jordan". Whilst
>> 
>>      (locate-file-internal ".."
>>             '("/plink:jordan@domain.com:/home/jordan/test")
>>              nil (lambda (f) (message f)))
>> 
>> correctly returns "/plink:jordan@domain.com:/home/jordan".
>> 
>> It seems the problem is something to do with converting from a
>> relative to absolute path. Eval'ing (expand-file-name "..") correctly
>> returns "/plink:jordan@domain.com:/home/jordan", though. Strange.
>
> Right you are, thanks.  The problem is that locate-file doesn't
> support remote file names.  Does the patch below produce good results?
>
> Michael, do you agree with this solution?  Do you think it's safe
> enough to put it on the release branch (it's a regression from a few
> years ago)?

I've tried to debug it, but no chance to do it on Windows.
Debugging C sources is a no-go for me there.

> diff --git a/lisp/files.el b/lisp/files.el
> index eb09a7c..cfe67b4 100644
> --- a/lisp/files.el
> +++ b/lisp/files.el
> @@ -801,9 +801,15 @@ cd
>      (setq cd-path (or (parse-colon-path (getenv "CDPATH"))
>                        (list "./"))))
>    (cd-absolute
> -   (or (locate-file dir cd-path nil
> -                    (lambda (f) (and (file-directory-p f) 'dir-ok)))
> -       (error "No such directory found via CDPATH environment variable"))))
> +   (or
> +    ;; locate-file doesn't support remote file names, so detect them
> +    ;; and support them here by hand.
> +    (and (file-name-absolute-p (expand-file-name dir))
> +         (file-accessible-directory-p (expand-file-name dir))
> +         (expand-file-name dir))
> +    (locate-file dir cd-path nil
> +                 (lambda (f) (and (file-directory-p f) 'dir-ok)))
> +    (error "No such directory found via CDPATH environment variable"))))
>  
>  (defun directory-files-recursively (dir regexp &optional include-directories)
>    "Return list of all files under DIR that have file names matching REGEXP.

Works for me, also on Windows, but I have modified it slightly in order
to make it more explicit. I have applied

> +   (or
> +    ;; locate-file doesn't support remote file names, so detect them
> +    ;; and support them here by hand.
> +    (and (file-remote-p (expand-file-name dir))
...

It is also OK for me to push it to the emacs-26 branch.

Best regards, Michael.

bug#33791: 26.1; Eshell on Windows connecting to GNU/Linux machine using TRAMP and plink: env: ‘c:/home/jordan/test.sh’: No such file or directory

[Eli Zaretskii]

> From: Michael Albinus <michael.albinus@gmx.de>
> Cc: Jordan Wilson <jordan.t.wilson@gmx.com>,  33791@debbugs.gnu.org
> Date: Thu, 27 Dec 2018 14:33:40 +0100
> 
> Works for me, also on Windows, but I have modified it slightly in order
> to make it more explicit. I have applied
> 
> > +   (or
> > +    ;; locate-file doesn't support remote file names, so detect them
> > +    ;; and support them here by hand.
> > +    (and (file-remote-p (expand-file-name dir))
> ...
> 
> It is also OK for me to push it to the emacs-26 branch.

Thanks.

Jordan, can you test this, please?  If this works for you, we will
install this on the emacs-26 branch as well.

bug#33791: 26.1; Eshell on Windows connecting to GNU/Linux machine using TRAMP and plink: env: ‘c:/home/jordan/test.sh’: No such file or directory

[Jordan Wilson]

Hi Eli,
sorry, I've not been online because of Christmas.

On 2018-12-28 (Fri) at 10:15 (+0200), Eli Zaretskii <eliz@gnu.org> wrote:
> Jordan, can you test this, please?  If this works for you, we will
> install this on the emacs-26 branch as well.

The patch you provided fixes the "cd" problem for me. The "env:
‘c:/home/jordan/test/test.sh’: No such file or directory" problem is
still there though.

Here's what I've found about that bug: the program path that
`eshell-gather-process-output' passes to `start-file-process' is run
through `expand-file-name' to remove potential relative paths. Because
the path begins with `/', "c:" is prepended on Windows (from looking at
the C source, I gather this is the function's expected behaviour).

As a test, if I wrap the "(expand-file-name command)" in "(substring
... 2)", which in this instance removes the prepended "c:/", doing
"./test.sh" in eshell works as expected.

Any ideas?

Thanks

-- 
Jordan Wilson
    Sent from Gnus v5.13, GNU Emacs 26.1

bug#33791: 26.1; Eshell on Windows connecting to GNU/Linux machine using TRAMP and plink: env: ‘c:/home/jordan/test.sh’: No such file or directory

[Eli Zaretskii]

> From: Jordan Wilson <jordan.t.wilson@gmx.com>
> Cc: Michael Albinus <michael.albinus@gmx.de>,  33791@debbugs.gnu.org
> Date: Fri, 28 Dec 2018 17:23:35 +0000
> 
> On 2018-12-28 (Fri) at 10:15 (+0200), Eli Zaretskii <eliz@gnu.org> wrote:
> > Jordan, can you test this, please?  If this works for you, we will
> > install this on the emacs-26 branch as well.
> 
> The patch you provided fixes the "cd" problem for me.

The one I porovided or the variation proposed by Michael?  Which one
did you try?

> The "env: ‘c:/home/jordan/test/test.sh’: No such file or directory"
> problem is still there though.
> 
> Here's what I've found about that bug: the program path that
> `eshell-gather-process-output' passes to `start-file-process' is run
> through `expand-file-name' to remove potential relative paths. Because
> the path begins with `/', "c:" is prepended on Windows (from looking at
> the C source, I gather this is the function's expected behaviour).
> 
> As a test, if I wrap the "(expand-file-name command)" in "(substring
> ... 2)", which in this instance removes the prepended "c:/", doing
> "./test.sh" in eshell works as expected.
> 
> Any ideas?

Can you provide a reproducing recipe starting from "emacs -Q"?

bug#33791: 26.1; Eshell on Windows connecting to GNU/Linux machine using TRAMP and plink: env: ‘c:/home/jordan/test.sh’: No such file or directory

[Jordan Wilson]

On 2018-12-28 (Fri) at 20:48 (+0200), Eli Zaretskii <eliz@gnu.org> wrote:
> The one I porovided or the variation proposed by Michael?  Which one
> did you try?
Both work.

> Can you provide a reproducing recipe starting from "emacs -Q"?

- Have putty in $PATH (version 0.70 on my machine)
- Load Eli's/Michael's patched files.el (error appears regardless)
    (load "files.el")
- M-x eshell
- connect to GNU/Linux machine using plink:
      /plink:jordan@domain.com:/home/jordan/
- run executable in working directory
      ./test.sh
returns "env: ‘c:/home/jordan/test.sh’: No such file or directory"

-- 
Jordan Wilson
    Sent from Gnus v5.13, GNU Emacs 26.1

bug#33791: 26.1; Eshell on Windows connecting to GNU/Linux machine using TRAMP and plink: env: ‘c:/home/jordan/test.sh’: No such file or directory

[Eli Zaretskii]

> From: Jordan Wilson <jordan.t.wilson@gmx.com>
> Cc: michael.albinus@gmx.de,  33791@debbugs.gnu.org
> Date: Fri, 28 Dec 2018 19:25:59 +0000
> 
> - Have putty in $PATH (version 0.70 on my machine)
> - Load Eli's/Michael's patched files.el (error appears regardless)
>     (load "files.el")
> - M-x eshell
> - connect to GNU/Linux machine using plink:
>       /plink:jordan@domain.com:/home/jordan/
> - run executable in working directory
>       ./test.sh
> returns "env: ‘c:/home/jordan/test.sh’: No such file or directory"

Right, I see this as well.

Michael, I need your help here.  The problem is in this part of
eshell-gather-process-output:

    (cond
     ((fboundp 'start-file-process)
      (setq proc
	    (let ((process-connection-type
		   (unless (eshell-needs-pipe-p command)
		     process-connection-type))
		  (command (file-local-name command)))
	      (apply 'start-file-process
		     (file-name-nondirectory command) nil
		     ;; `start-process' can't deal with relative filenames.
		     (append (list (expand-file-name command)) args))))

The problem is that file-local-name returns a Unix-style absolute file
name /foo/bar/baz, and the following expand-file-name call then
prepends a drive letter on Windows, because there's no longer any sign
of COMMAND being a remote file, and so expand-file-name doesn't invoke
the Tramp handler.

The simplest fix is to remove the expand-file-name call.  I don't
understand why it is there, but the claim that start-process cannot
deal with relative file names is definitely false.  This code was
there since June 2000, when eshell-gather-process-output was first
written.  Do you see any reason why we'd need to call expand-file-name
here?

Btw, isn't it confusing that start-file-process needs only the "local"
part of COMMAND?  Why cannot its handler DTRT internally instead?

Thanks.

bug#33791: 26.1; Eshell on Windows connecting to GNU/Linux machine using TRAMP and plink: env: ‘c:/home/jordan/test.sh’: No such file or directory

[Michael Albinus]

Eli Zaretskii <eliz@gnu.org> writes:

Hi Eli,

> Michael, I need your help here.  The problem is in this part of
> eshell-gather-process-output:
>
>     (cond
>      ((fboundp 'start-file-process)
>       (setq proc
> 	    (let ((process-connection-type
> 		   (unless (eshell-needs-pipe-p command)
> 		     process-connection-type))
> 		  (command (file-local-name command)))
> 	      (apply 'start-file-process
> 		     (file-name-nondirectory command) nil
> 		     ;; `start-process' can't deal with relative filenames.
> 		     (append (list (expand-file-name command)) args))))
>
> The problem is that file-local-name returns a Unix-style absolute file
> name /foo/bar/baz, and the following expand-file-name call then
> prepends a drive letter on Windows, because there's no longer any sign
> of COMMAND being a remote file, and so expand-file-name doesn't invoke
> the Tramp handler.

This is fixed in the master branch already:

--8<---------------cut here---------------start------------->8---
    (cond
     ((fboundp 'start-file-process)
      (setq proc
	    (let ((process-connection-type
		   (unless (eshell-needs-pipe-p command)
		     process-connection-type))
		  ;; `start-process' can't deal with relative filenames.
		  (command (file-local-name (expand-file-name command))))
	      (apply 'start-file-process
		     (file-name-nondirectory command) nil command args)))
--8<---------------cut here---------------end--------------->8---

The respective commit is

--8<---------------cut here---------------start------------->8---
commit bca35315e16cb53415649e5c0ac2ec0cc1368679
Author: Michael Albinus <michael.albinus@gmx.de>
Date:   Thu Sep 6 12:16:00 2018 +0200

    Fix Bug#31704
    
    * lisp/eshell/esh-proc.el (eshell-gather-process-output): Do not
    let `expand-file-name' prefix remote file names with MS Windows
    volume letter.
    
    * lisp/net/tramp.el (tramp-eshell-directory-change):
    Use `path-separator' as it does eshell.  (Bug#31704)
--8<---------------cut here---------------end--------------->8---

As you see, it requires two changes, in esh-proc.el and tramp.el.

> Btw, isn't it confusing that start-file-process needs only the "local"
> part of COMMAND?  Why cannot its handler DTRT internally instead?

What do you do, if COMMAND is another remote location than
default-directory?

And more general, there could also be file names in the PROGRAM-ARGS
part of start-file-process. Who decides, which of them is a remote file
name to be stripped to the local part, and which offers remote file name
syntax to be used literally?

That's why we have decided (long ago), to not allow remote arguments for
both start-file-process and process-file.

> Thanks.

Best regards, Michael.

bug#33791: 26.1; Eshell on Windows connecting to GNU/Linux machine using TRAMP and plink: env: ‘c:/home/jordan/test.sh’: No such file or directory

[Eli Zaretskii]

> From: Michael Albinus <michael.albinus@gmx.de>
> Cc: Jordan Wilson <jordan.t.wilson@gmx.com>,  33791@debbugs.gnu.org
> Date: Sat, 29 Dec 2018 10:25:37 +0100
> 
> > The problem is that file-local-name returns a Unix-style absolute file
> > name /foo/bar/baz, and the following expand-file-name call then
> > prepends a drive letter on Windows, because there's no longer any sign
> > of COMMAND being a remote file, and so expand-file-name doesn't invoke
> > the Tramp handler.
> 
> This is fixed in the master branch already:
> 
> --8<---------------cut here---------------start------------->8---
>     (cond
>      ((fboundp 'start-file-process)
>       (setq proc
> 	    (let ((process-connection-type
> 		   (unless (eshell-needs-pipe-p command)
> 		     process-connection-type))
> 		  ;; `start-process' can't deal with relative filenames.
> 		  (command (file-local-name (expand-file-name command))))
> 	      (apply 'start-file-process
> 		     (file-name-nondirectory command) nil command args)))
> --8<---------------cut here---------------end--------------->8---
> 
> The respective commit is
> 
> --8<---------------cut here---------------start------------->8---
> commit bca35315e16cb53415649e5c0ac2ec0cc1368679
> Author: Michael Albinus <michael.albinus@gmx.de>
> Date:   Thu Sep 6 12:16:00 2018 +0200
> 
>     Fix Bug#31704
>     
>     * lisp/eshell/esh-proc.el (eshell-gather-process-output): Do not
>     let `expand-file-name' prefix remote file names with MS Windows
>     volume letter.
>     
>     * lisp/net/tramp.el (tramp-eshell-directory-change):
>     Use `path-separator' as it does eshell.  (Bug#31704)
> --8<---------------cut here---------------end--------------->8---
> 
> As you see, it requires two changes, in esh-proc.el and tramp.el.

I think both can be cherry-picked to emacs-26.

But do you know why we need expand-file-name here at all?  At the very
least, the comment about start-process should be removed, I think.

> > Btw, isn't it confusing that start-file-process needs only the "local"
> > part of COMMAND?  Why cannot its handler DTRT internally instead?
> 
> What do you do, if COMMAND is another remote location than
> default-directory?

Why is that a problem?  We always run the process in
default-directory, right?

> And more general, there could also be file names in the PROGRAM-ARGS
> part of start-file-process. Who decides, which of them is a remote file
> name to be stripped to the local part, and which offers remote file name
> syntax to be used literally?

the caller, of course.  But I'm not asking about arguments, I'm asking
about PROGRAM, and only about it.

> That's why we have decided (long ago), to not allow remote arguments for
> both start-file-process and process-file.

At the very least, this should be prominently mentioned in the
respective doc strings and in the ELisp manual.  As written now, this
is entirely undocumented.  Moreover, the part about "the local part of
default-directory" in the doc string and in the manual is confusing,
because we have no description of what that means.  The only attempt
of describing it, in file-local-name's doc string, viz.:

  It returns a file name which can be used directly as argument of
  ‘process-file’, ‘start-file-process’, or ‘shell-command’.

is IMO unsatisfactory, because it describes how results could be used,
not what they are.

bug#33791: 26.1; Eshell on Windows connecting to GNU/Linux machine using TRAMP and plink: env: ‘c:/home/jordan/test.sh’: No such file or directory

[Michael Albinus]

[-- Attachment #1: Type: text/plain, Size: 3261 bytes --]

Eli Zaretskii <eliz@gnu.org> writes:

Hi Eli & Jordan,

>> As you see, it requires two changes, in esh-proc.el and tramp.el.
>
> I think both can be cherry-picked to emacs-26.

Done. The change in tramp.el needed some massage.

Jordan, could you pls test whether the changes I have applied to the
emacs-26 branch work for you? See attachment for the changes.

> But do you know why we need expand-file-name here at all?

Well, in Emacs 26 we couldn't adapt exec-path for remote
processes. This is available since Emacs 27 only.

> At the very least, the comment about start-process should be removed,
> I think.

Done.

>> > Btw, isn't it confusing that start-file-process needs only the "local"
>> > part of COMMAND?  Why cannot its handler DTRT internally instead?
>> 
>> What do you do, if COMMAND is another remote location than
>> default-directory?
>
> Why is that a problem?  We always run the process in
> default-directory, right?

I said "another *remote* location". I don't want to see

(let ((default-directory "/ssh:user@host:directory"))
  (start-file-process "foo" buffer
  "/ssh:another-user@another-host:foo"))

Of course one could raise an error. But usually, this construct doesn't
appear as direct as in the example.

I believe using a local name for PROGRAM isn't such a burden.

>> And more general, there could also be file names in the PROGRAM-ARGS
>> part of start-file-process. Who decides, which of them is a remote file
>> name to be stripped to the local part, and which offers remote file name
>> syntax to be used literally?
>
> the caller, of course.  But I'm not asking about arguments, I'm asking
> about PROGRAM, and only about it.

See above. I don't believe it would make the life easier for callers of
`start-file-process', if we would allow a remote file name for PROGRAM. 

>> That's why we have decided (long ago), to not allow remote arguments for
>> both start-file-process and process-file.
>
> At the very least, this should be prominently mentioned in the
> respective doc strings and in the ELisp manual.  As written now, this
> is entirely undocumented.  Moreover, the part about "the local part of
> default-directory" in the doc string and in the manual is confusing,
> because we have no description of what that means.  The only attempt
> of describing it, in file-local-name's doc string, viz.:
>
>   It returns a file name which can be used directly as argument of
>   ‘process-file’, ‘start-file-process’, or ‘shell-command’.
>
> is IMO unsatisfactory, because it describes how results could be used,
> not what they are.

The docstring speaks about.

--8<---------------cut here---------------start------------->8---
PROGRAM and PROGRAM-ARGS might be file names.  They are not
objects of file name handler invocation.
--8<---------------cut here---------------end--------------->8---

The Elisp manual speaks about. (info "(elisp) Asynchronous Processes")

--8<---------------cut here---------------start------------->8---
This function does not try to invoke file name handlers for PROGRAM
or for the rest of ARGS.
--8<---------------cut here---------------end--------------->8---

Best regards, Michael.


[-- Attachment #2: Type: text/plain, Size: 1445 bytes --]

diff --git a/lisp/eshell/esh-proc.el b/lisp/eshell/esh-proc.el
index 94401c5daa..97170eb04b 100644
--- a/lisp/eshell/esh-proc.el
+++ b/lisp/eshell/esh-proc.el
@@ -279,11 +279,9 @@ eshell-gather-process-output
 	    (let ((process-connection-type
 		   (unless (eshell-needs-pipe-p command)
 		     process-connection-type))
-		  (command (file-local-name command)))
+		  (command (file-local-name (expand-file-name command))))
 	      (apply 'start-file-process
-		     (file-name-nondirectory command) nil
-		     ;; `start-process' can't deal with relative filenames.
-		     (append (list (expand-file-name command)) args))))
+		     (file-name-nondirectory command) nil command args)))
       (eshell-record-process-object proc)
       (set-process-buffer proc (current-buffer))
       (if (eshell-interactive-output-p)
diff --git a/lisp/net/tramp.el b/lisp/net/tramp.el
index 5fa9f9a44d..5302659b32 100644
--- a/lisp/net/tramp.el
+++ b/lisp/net/tramp.el
@@ -4580,10 +4580,11 @@ tramp-eshell-directory-change
 	       (or
 		;; When `tramp-own-remote-path' is in `tramp-remote-path',
 		;; the remote path is only set in the session cache.
+                ;; Use `path-separator' as it does eshell.
 		(tramp-get-connection-property
 		 (tramp-get-connection-process v) "remote-path" nil)
 		(tramp-get-connection-property v "remote-path" nil))
-	       ":"))
+	       path-separator))
 	  (getenv "PATH"))))
 
 (eval-after-load "esh-util"

bug#33791: 26.1; Eshell on Windows connecting to GNU/Linux machine using TRAMP and plink: env: ‘c:/home/jordan/test.sh’: No such file or directory

[Jordan Wilson]

Hi Michael and Eli,

On 2018-12-29 (Sat) at 12:12 (+0100), Michael Albinus <michael.albinus@gmx.de> wrote:
> Jordan, could you pls test whether the changes I have applied to the
> emacs-26 branch work for you? See attachment for the changes.

The patches fix my problem. Everything now works as expected.

Thank you to both of you for fixing this! 

-- 
Jordan Wilson
    Sent from Gnus v5.13, GNU Emacs 26.1

bug#33791: 26.1; Eshell on Windows connecting to GNU/Linux machine using TRAMP and plink: env: ‘c:/home/jordan/test.sh’: No such file or directory

[Michael Albinus]

Version: 26.2

Jordan Wilson <jordan.t.wilson@gmx.com> writes:

> Hi Michael and Eli,

Hi Jordan,

>> Jordan, could you pls test whether the changes I have applied to the
>> emacs-26 branch work for you? See attachment for the changes.
>
> The patches fix my problem. Everything now works as expected.

Thanks for the feedback. I'm closing the bug.

> Thank you to both of you for fixing this! 

Best regards, Michael.

bug#33791: 26.1; Eshell on Windows connecting to GNU/Linux machine using TRAMP and plink: env: ‘c:/home/jordan/test.sh’: No such file or directory

[Eli Zaretskii]

> From: Michael Albinus <michael.albinus@gmx.de>
> Cc: jordan.t.wilson@gmx.com,  33791@debbugs.gnu.org
> Date: Sat, 29 Dec 2018 12:12:56 +0100
> 
> >> As you see, it requires two changes, in esh-proc.el and tramp.el.
> >
> > I think both can be cherry-picked to emacs-26.
> 
> Done. The change in tramp.el needed some massage.
> 
> Jordan, could you pls test whether the changes I have applied to the
> emacs-26 branch work for you? See attachment for the changes.
> 
> > But do you know why we need expand-file-name here at all?
> 
> Well, in Emacs 26 we couldn't adapt exec-path for remote
> processes. This is available since Emacs 27 only.
> 
> > At the very least, the comment about start-process should be removed,
> > I think.
> 
> Done.

Thanks.

> > At the very least, this should be prominently mentioned in the
> > respective doc strings and in the ELisp manual.  As written now, this
> > is entirely undocumented.  Moreover, the part about "the local part of
> > default-directory" in the doc string and in the manual is confusing,
> > because we have no description of what that means.  The only attempt
> > of describing it, in file-local-name's doc string, viz.:
> >
> >   It returns a file name which can be used directly as argument of
> >   ‘process-file’, ‘start-file-process’, or ‘shell-command’.
> >
> > is IMO unsatisfactory, because it describes how results could be used,
> > not what they are.
> 
> The docstring speaks about.
> 
> --8<---------------cut here---------------start------------->8---
> PROGRAM and PROGRAM-ARGS might be file names.  They are not
> objects of file name handler invocation.
> --8<---------------cut here---------------end--------------->8---
> 
> The Elisp manual speaks about. (info "(elisp) Asynchronous Processes")
> 
> --8<---------------cut here---------------start------------->8---
> This function does not try to invoke file name handlers for PROGRAM
> or for the rest of ARGS.
> --8<---------------cut here---------------end--------------->8---

Thanks, I made that even more explicit and clear.

bug#33791: 26.1; Eshell on Windows connecting to GNU/Linux machine using TRAMP and plink: env: ‘c:/home/jordan/test.sh’: No such file or directory

[Michael Albinus]

Jordan Wilson <jordan.t.wilson@gmx.com> writes:

Hi Jordan,

>> Can you provide a reproducing recipe starting from "emacs -Q"?
>
> - Have putty in $PATH (version 0.70 on my machine)
> - Load Eli's/Michael's patched files.el (error appears regardless)
>     (load "files.el")
> - M-x eshell
> - connect to GNU/Linux machine using plink:
>       /plink:jordan@domain.com:/home/jordan/
> - run executable in working directory
>       ./test.sh
> returns "env: ‘c:/home/jordan/test.sh’: No such file or directory"

I've tried to reproduce the problem with

--8<---------------cut here---------------start------------->8---
GNU Emacs 27.0.50 (build 1, x86_64-w64-mingw32)
 of 2018-10-05
--8<---------------cut here---------------end--------------->8---

This is the Emacs version offered on <https://alpha.gnu.org/gnu/emacs/pretest/windows/emacs-27/>.

The error does not happen. Likely, this is due to the following commit:

--8<---------------cut here---------------start------------->8---
commit bbcd5787cb077f8b6c4eba5c1704ad953a298fd7
Author: Michael Albinus <michael.albinus@gmx.de>
Date:   Thu Mar 22 09:58:56 2018 +0100

    Fix commit c24c5dc4a4
    
    * lisp/net/tramp.el (tramp-handle-substitute-in-file-name): Drop volume
    letter of localname substitution.  Reported by Chris Zheng
    <chriszheng99@gmail.com>.
--8<---------------cut here---------------end--------------->8---

What happens, if you redefine tramp-handle-substitute-in-file-name after
loading Tramp?

--8<---------------cut here---------------start------------->8---

(defun tramp-handle-substitute-in-file-name (filename)
  "Like `substitute-in-file-name' for Tramp files.
\"//\" and \"/~\" substitute only in the local filename part."
  ;; Check, whether the local part is a quoted file name.
  (if (tramp-compat-file-name-quoted-p filename)
      filename
    ;; First, we must replace environment variables.
    (setq filename (tramp-replace-environment-variables filename))
    (with-parsed-tramp-file-name filename nil
      ;; We do not want to replace environment variables, again.  "//"
      ;; has a special meaning at the beginning of a file name on
      ;; Cygwin and MS-Windows, we must remove it.
      (let (process-environment)
	;; Ignore in LOCALNAME everything before "//" or "/~".
	(when (stringp localname)
	  (if (string-match "//\\(/\\|~\\)" localname)
	      (setq filename
                    (replace-regexp-in-string
                     "\\`/+" "/" (substitute-in-file-name localname)))
	    (setq filename
		  (concat (file-remote-p filename)
			  (replace-regexp-in-string
                           "\\`/+" "/"
			   ;; We must disable cygwin-mount file name
			   ;; handlers and alike.
			   (tramp-run-real-handler
			    'substitute-in-file-name (list localname))))))))
      ;; "/m:h:~" does not work for completion.  We use "/m:h:~/".
      (if (and (stringp localname) (string-equal "~" localname))
	  (concat filename "/")
	filename))))
--8<---------------cut here---------------end--------------->8---

Best regards, Michael.

bug#33791: 26.1; Eshell on Windows connecting to GNU/Linux machine using TRAMP and plink: env: ‘c:/home/jordan/test.sh’: No such file or directory

[Eli Zaretskii]

> From: Michael Albinus <michael.albinus@gmx.de>
> Cc: Jordan Wilson <jordan.t.wilson@gmx.com>,  33791@debbugs.gnu.org
> Date: Thu, 27 Dec 2018 14:33:40 +0100
> 
> I've tried to debug it, but no chance to do it on Windows.
> Debugging C sources is a no-go for me there.

Can you tell why?  I do that all the time, so maybe I could help.

> Works for me, also on Windows, but I have modified it slightly in order
> to make it more explicit. I have applied
> 
> > +   (or
> > +    ;; locate-file doesn't support remote file names, so detect them
> > +    ;; and support them here by hand.
> > +    (and (file-remote-p (expand-file-name dir))
> ...

You mean, testing file-remote-p _in_addition_ to the tests I proposed?
Is the file-name-absolute-p test needed if we test with file-remote-p?

> It is also OK for me to push it to the emacs-26 branch.

Done.

bug#33791: 26.1; Eshell on Windows connecting to GNU/Linux machine using TRAMP and plink: env: ‘c:/home/jordan/test.sh’: No such file or directory

[Michael Albinus]

Eli Zaretskii <eliz@gnu.org> writes:

>> I've tried to debug it, but no chance to do it on Windows.
>> Debugging C sources is a no-go for me there.
>
> Can you tell why?  I do that all the time, so maybe I could help.

When I can find a WIndows machine, I can install Emacs and debug Lisp
sources. Installing a C compiler, building Emacs from sources, and
debugging C sources under Windows is out of my scope, on a borrowed or
stolen machine I have under my control for a short time.

>> Works for me, also on Windows, but I have modified it slightly in order
>> to make it more explicit. I have applied
>> 
>> > +   (or
>> > +    ;; locate-file doesn't support remote file names, so detect them
>> > +    ;; and support them here by hand.
>> > +    (and (file-remote-p (expand-file-name dir))
>> ...
>
> You mean, testing file-remote-p _in_addition_ to the tests I proposed?

No, instead of file-name-absolute-p.

> Is the file-name-absolute-p test needed if we test with file-remote-p?

No.

>> It is also OK for me to push it to the emacs-26 branch.
>
> Done.

Thanks.

Best reghards, Michael.

bug#33791: 26.1; Eshell on Windows connecting to GNU/Linux machine using TRAMP and plink: env: ‘c:/home/jordan/test.sh’: No such file or directory

[Eli Zaretskii]

> From: Michael Albinus <michael.albinus@gmx.de>
> Cc: jordan.t.wilson@gmx.com,  33791@debbugs.gnu.org
> Date: Sat, 29 Dec 2018 09:38:55 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> I've tried to debug it, but no chance to do it on Windows.
> >> Debugging C sources is a no-go for me there.
> >
> > Can you tell why?  I do that all the time, so maybe I could help.
> 
> When I can find a WIndows machine, I can install Emacs and debug Lisp
> sources. Installing a C compiler, building Emacs from sources, and
> debugging C sources under Windows is out of my scope, on a borrowed or
> stolen machine I have under my control for a short time.

Ah, okay.  Did you try to make an installation with a debugger and a
compiler on a portable device, like DoK?

> >> > +   (or
> >> > +    ;; locate-file doesn't support remote file names, so detect them
> >> > +    ;; and support them here by hand.
> >> > +    (and (file-remote-p (expand-file-name dir))
> >> ...
> >
> > You mean, testing file-remote-p _in_addition_ to the tests I proposed?
> 
> No, instead of file-name-absolute-p.

Fixed, thanks.

bug#33791: 26.1; Eshell on Windows connecting to GNU/Linux machine using TRAMP and plink: env: ‘c:/home/jordan/test.sh’: No such file or directory

[Michael Albinus]

Eli Zaretskii <eliz@gnu.org> writes:

>> When I can find a WIndows machine, I can install Emacs and debug Lisp
>> sources. Installing a C compiler, building Emacs from sources, and
>> debugging C sources under Windows is out of my scope, on a borrowed or
>> stolen machine I have under my control for a short time.
>
> Ah, okay.  Did you try to make an installation with a debugger and a
> compiler on a portable device, like DoK?

No. Honestly, I'm not interested in investing too much energy working
for Windows. Sorry.

Best regards, Michael.

bug#33791: 26.1; Eshell on Windows connecting to GNU/Linux machine using TRAMP and plink: env: ‘c:/home/jordan/test.sh’: No such file or directory

[Jordan Wilson]

On 2018-12-22 (Sat) at 11:23 (+0200), Eli Zaretskii <eliz@gnu.org> wrote:
>> When I'm connected to my GNU/Linux machine (using TRAMP and plink) with
>> eshell, I can't run executables in the local directory. Doing
>> "./test.sh" returns "env: ‘c:/home/jordan/test.sh’: No such file or
>> directory".
>
> I cannot reproduce this, but I'm not on Windows 10.

I'm getting it on Windows 7, too. Both with my configuration and with
the "-Q" argument.

>> Here's an example:
>> 
>>        c:/Users/Jordan $ /plink:jordan@domain.com:/home/jordan/test
>>        /plink:jordan@domain.com:/home/jordan/test $ ls
>>        test2       test.sh
>>        /plink:jordan@domain.com:/home/jordan/test $ ./test.sh
>>        env: ‘c:/home/jordan/test.sh’: No such file or directory
>
> What is in test.sh?  Did you make that script executable?

Yes, it's an executable. It should actually read
"c:/home/jordan/test/test.sh", but I somehow omitted the "test"
directory when I reproduced it in my message.

I've replicated both issues connecting to another GNU/Linux machine.


-- 
Jordan Wilson
    Sent from Gnus v5.13, GNU Emacs 26.1