bug#34834: 26.1; Remote `eshell/mv' and `eshell/cp' on Windows: Opening output file: Invalid argument, c:/home/ ...

bug#34834: 26.1; Remote `eshell/mv' and `eshell/cp' on Windows: Opening output file: Invalid argument, c:/home/ ...

[Jordan Wilson]

Hi,
this bug is similar to my previously reported bug #33791. I'm running
Emacs 26.1 (with the patched files.el that fixed that bug[1]) on Windows
10. I've replicated this with "emacs -Q".

When using eshell connected to a GNU/Linux machine, `eshell/cp' and
`eshell/mv' (which are called in eshell with the commands "cp" and "mv")
both return an "Invalid argument" error when the source and destination
are relative paths on the remote machine.

Recipe:
- connect to GNU/Linux machine using plink:
    /plink:jordan@domain.com:/home/jordan/
- cp/mv a file between locations on the remote machine
    /plink:jordan@domain.com:/home/jordan $ cp file.txt directory/
- returns:
   Opening output file: Invalid argument, c:/plink:jordan@domain.com:/home/jordan/file.txt

I'm guessing it's a problem of not correctly handling the relative TRAMP
paths, as it works if provided the full paths for the source and
destination.

Thanks.

Footnotes: 
[1] https://debbugs.gnu.org/cgi/bugreport.cgi?bug=33791;msg=34  


In GNU Emacs 26.1 (build 1, x86_64-w64-mingw32)
 of 2018-05-29 built on TPW550S
Windowing system distributor 'Microsoft Corp.', version 10.0.17134
Recent messages:

Configured using:
 'configure --without-compress-install --without-dbus --with-modules
 'CFLAGS= -O2 -g3''

Configured features:
XPM JPEG TIFF GIF PNG RSVG IMAGEMAGICK SOUND NOTIFY ACL GNUTLS LIBXML2
ZLIB TOOLKIT_SCROLL_BARS MODULES THREADS LCMS2

Important settings:
  value of $LANG: ENG
  locale-coding-system: cp1252

-- 
Jordan Wilson
    Sent from Gnus v5.13, GNU Emacs 26.1

bug#34834: 26.1; Remote `eshell/mv' and `eshell/cp' on Windows: Opening output file: Invalid argument, c:/home/ ...

[Michael Albinus]

Jordan Wilson <jordan.t.wilson@gmx.com> writes:

> Hi,

Hi Jordan,

sorry for the late reply; as usual it takes time for me to find an MS
Windows machine for test.

> this bug is similar to my previously reported bug #33791. I'm running
> Emacs 26.1 (with the patched files.el that fixed that bug[1]) on Windows
> 10. I've replicated this with "emacs -Q".
>
> When using eshell connected to a GNU/Linux machine, `eshell/cp' and
> `eshell/mv' (which are called in eshell with the commands "cp" and "mv")
> both return an "Invalid argument" error when the source and destination
> are relative paths on the remote machine.
>
> Recipe:
> - connect to GNU/Linux machine using plink:
>     /plink:jordan@domain.com:/home/jordan/
> - cp/mv a file between locations on the remote machine
>     /plink:jordan@domain.com:/home/jordan $ cp file.txt directory/
> - returns:
>    Opening output file: Invalid argument, c:/plink:jordan@domain.com:/home/jordan/file.txt
>
> I'm guessing it's a problem of not correctly handling the relative TRAMP
> paths, as it works if provided the full paths for the source and
> destination.

I could reproduce the problem with "GNU Emacs 27.0.50 (build 1, x86_64-w64-mingw32)
 of 2019-01-17". It has nothing to do with eshell. My recipe:

--8<---------------cut here---------------start------------->8---
C-x C-f /plinkx:detlefx:/home/albinus/ ;; This is a remote GNU/Linux machine.

M-: (expand-file-name "123" "tmp/") ;; 123 is a file, tmp is a directory there.

=> "c:/plinkx:detlefx:/home/albinus/tmp/123"
--8<---------------cut here---------------end--------------->8---

I have added traces to this, with M-x trace-function-background for
expand-file-name, tramp-sh-handle-expand-file-name, and
tramp-file-name-handler. The latter function is Tramp's outmost
function. The traces look like this:

======================================================================
1 -> (expand-file-name "123" "tmp/")
| 2 -> (tramp-file-name-handler expand-file-name "tmp/" "/plinkx:detlefx:/home/albinus/")
| | 3 -> (tramp-sh-handle-expand-file-name "tmp/" "/plinkx:detlefx:/home/albinus/")
| | | 4 -> (tramp-file-name-handler file-name-as-directory "/plinkx:detlefx:/home/albinus/")
| | | 4 <- tramp-file-name-handler: "/plinkx:detlefx:/home/albinus/"
| | | 4 -> (expand-file-name "/home/albinus/tmp/")
| | | 4 <- expand-file-name: "c:/home/albinus/tmp/"
| | 3 <- tramp-sh-handle-expand-file-name: "/plinkx:detlefx:/home/albinus/tmp/"
| 2 <- tramp-file-name-handler: "/plinkx:detlefx:/home/albinus/tmp/"
1 <- expand-file-name: "c:/plinkx:detlefx:/home/albinus/tmp/123"
======================================================================

Looks, like Tramp returns the proper value "/plinkx:detlefx:/home/albinus/tmp/",
and then in its way through expand-file-name the drive letter is
added. Since this is a C function, I'm not able to debug further.

Eli, could you pls check this?

> Thanks.

Best regards, Michael.

bug#34834: 26.1; Remote `eshell/mv' and `eshell/cp' on Windows: Opening output file: Invalid argument, c:/home/ ...

[Eli Zaretskii]

> From: Michael Albinus <michael.albinus@gmx.de>
> Date: Thu, 28 Mar 2019 18:44:09 +0100
> Cc: 34834@debbugs.gnu.org
> 
> --8<---------------cut here---------------start------------->8---
> C-x C-f /plinkx:detlefx:/home/albinus/ ;; This is a remote GNU/Linux machine.
> 
> M-: (expand-file-name "123" "tmp/") ;; 123 is a file, tmp is a directory there.
> 
> => "c:/plinkx:detlefx:/home/albinus/tmp/123"
> --8<---------------cut here---------------end--------------->8---
> 
> I have added traces to this, with M-x trace-function-background for
> expand-file-name, tramp-sh-handle-expand-file-name, and
> tramp-file-name-handler. The latter function is Tramp's outmost
> function. The traces look like this:
> 
> ======================================================================
> 1 -> (expand-file-name "123" "tmp/")
> | 2 -> (tramp-file-name-handler expand-file-name "tmp/" "/plinkx:detlefx:/home/albinus/")
> | | 3 -> (tramp-sh-handle-expand-file-name "tmp/" "/plinkx:detlefx:/home/albinus/")
> | | | 4 -> (tramp-file-name-handler file-name-as-directory "/plinkx:detlefx:/home/albinus/")
> | | | 4 <- tramp-file-name-handler: "/plinkx:detlefx:/home/albinus/"
> | | | 4 -> (expand-file-name "/home/albinus/tmp/")
> | | | 4 <- expand-file-name: "c:/home/albinus/tmp/"
> | | 3 <- tramp-sh-handle-expand-file-name: "/plinkx:detlefx:/home/albinus/tmp/"
> | 2 <- tramp-file-name-handler: "/plinkx:detlefx:/home/albinus/tmp/"
> 1 <- expand-file-name: "c:/plinkx:detlefx:/home/albinus/tmp/123"
> ======================================================================
> 
> Looks, like Tramp returns the proper value "/plinkx:detlefx:/home/albinus/tmp/",
> and then in its way through expand-file-name the drive letter is
> added. Since this is a C function, I'm not able to debug further.
> 
> Eli, could you pls check this?

I don't think I understand what should I check.  In general a file
name like "/foo/bar" will have a drive letter added to it by
expand-file-name.  What makes this case special?

bug#34834: 26.1; Remote `eshell/mv' and `eshell/cp' on Windows: Opening output file: Invalid argument, c:/home/ ...

[Michael Albinus]

Eli Zaretskii <eliz@gnu.org> writes:

Hi Eli,

> I don't think I understand what should I check.  In general a file
> name like "/foo/bar" will have a drive letter added to it by
> expand-file-name.  What makes this case special?

Remote file names shouldn't get a drive letter. But this happened:

>> M-: (expand-file-name "123" "tmp/") ;; 123 is a file, tmp is a directory there.
>>
>> => "c:/plinkx:detlefx:/home/albinus/tmp/123"

And the traces I've shown indicate, that it isn't Tramp who adds the
drive letter. The rest to check are C sources, I'm unable to debug on MS
Windows.

Best regards, Michael

bug#34834: 26.1; Remote `eshell/mv' and `eshell/cp' on Windows: Opening output file: Invalid argument, c:/home/ ...

[Eli Zaretskii]

> From: Michael Albinus <michael.albinus@gmx.de>
> Cc: jordan.t.wilson@gmx.com,  34834@debbugs.gnu.org
> Date: Thu, 28 Mar 2019 18:57:49 +0100
> 
> Remote file names shouldn't get a drive letter. But this happened:

Can you describe how this is supposed to work?  AFAIR, Tramp's
file-name handler gets called, and then turns around and calls back
into expand-file-name after inhibiting file-name handlers?  Or
something like that.  How does Emacs expand remote file names
correctly on Posix platforms?  If that somehow depends on the fact
that remote file names begin with a slash, and thus look like absolute
file names, then that's not enough for Windows.

Also, is this use case somehow special?  If not, how come Tramp works
at all on Windows?

> >> M-: (expand-file-name "123" "tmp/") ;; 123 is a file, tmp is a directory there.
> >>
> >> => "c:/plinkx:detlefx:/home/albinus/tmp/123"
> 
> And the traces I've shown indicate, that it isn't Tramp who adds the
> drive letter. The rest to check are C sources, I'm unable to debug on MS
> Windows.

My problem is not to debug Emacs, my problem is where to debug.
Please help me focus my search for the culprit.

Meanwhile, the same problem happens in Emacs 26, so this is not new.

bug#34834: 26.1; Remote `eshell/mv' and `eshell/cp' on Windows: Opening output file: Invalid argument, c:/home/ ...

[Michael Albinus]

Eli Zaretskii <eliz@gnu.org> writes:

Hi Eli,

>> Remote file names shouldn't get a drive letter. But this happened:
>
> Can you describe how this is supposed to work?  AFAIR, Tramp's
> file-name handler gets called, and then turns around and calls back
> into expand-file-name after inhibiting file-name handlers?  Or
> something like that.

I have an idea what happened. See the traces I gave in my first answer:

======================================================================
1 -> (expand-file-name "123" "tmp/")
| 2 -> (tramp-file-name-handler expand-file-name "tmp/" "/plinkx:detlefx:/home/albinus/")
| | 3 -> (tramp-sh-handle-expand-file-name "tmp/" "/plinkx:detlefx:/home/albinus/")
| | | 4 -> (tramp-file-name-handler file-name-as-directory "/plinkx:detlefx:/home/albinus/")
| | | 4 <- tramp-file-name-handler: "/plinkx:detlefx:/home/albinus/"
| | | 4 -> (expand-file-name "/home/albinus/tmp/")
| | | 4 <- expand-file-name: "c:/home/albinus/tmp/"
| | 3 <- tramp-sh-handle-expand-file-name: "/plinkx:detlefx:/home/albinus/tmp/"
| 2 <- tramp-file-name-handler: "/plinkx:detlefx:/home/albinus/tmp/"
1 <- expand-file-name: "c:/plinkx:detlefx:/home/albinus/tmp/123"
======================================================================

The buffer's default-directory is remote, "/plinkx:detlefx:/home/albinus/".
expand-file-name is called

--8<---------------cut here---------------start------------->8---
(expand-file-name "123" "tmp/")
--8<---------------cut here---------------end--------------->8---

This is entry "1" in the trace output. In function Fexpand_file_name,
line 794 of fileio.c, there is the check

--8<---------------cut here---------------start------------->8---
  handler = Ffind_file_name_handler (name, Qexpand_file_name);
--8<---------------cut here---------------end--------------->8---

This returns Qnil, because name is "123". Then we have in line 823

--8<---------------cut here---------------start------------->8---
  handler = Ffind_file_name_handler (default_directory, Qexpand_file_name);
--8<---------------cut here---------------end--------------->8---

This returns Qnil again, because default_directory is "tmp/". That's all
about checking for a file name handler. Later on, line 872, there is

--8<---------------cut here---------------start------------->8---
	default_directory = Fexpand_file_name (default_directory, Qnil);
--8<---------------cut here---------------end--------------->8---

which belongs to entry "2" of the traces above. It returns the proper
answer "/plinkx:detlefx:/home/albinus/tmp/". But this is too late; it
must happen before the second call of Ffind_file_name_handler.
Fexpand_file_name continues to work, it constructs the final answer
"/plinkx:detlefx:/home/albinus/tmp/123" *plus* the drive letter (I
suppose line 1047),

As you know, I cannot test it myself any fix, because I cannot build
Emacs for Windows.

> Meanwhile, the same problem happens in Emacs 26, so this is not new.

Yes. It is not so common to use a relative dir argument for
expand-file-name, and in the non-Windows case it works properly. So it
was uncovered for years.

Best regards, Michael.

bug#34834: 26.1; Remote `eshell/mv' and `eshell/cp' on Windows: Opening output file: Invalid argument, c:/home/ ...

[Eli Zaretskii]

> From: Michael Albinus <michael.albinus@gmx.de>
> Cc: jordan.t.wilson@gmx.com,  34834@debbugs.gnu.org
> Date: Fri, 29 Mar 2019 12:16:17 +0100
> 
> > Meanwhile, the same problem happens in Emacs 26, so this is not new.
> 
> Yes. It is not so common to use a relative dir argument for
> expand-file-name, and in the non-Windows case it works properly.

IIUC, it works on Posix platforms by sheer luck, is that right?  IOW,
on Posix platforms, we also don't recognize this is a remote file name
until it's too late, and we only get away because the file name starts
with a slash, and so looks like an absolute file name on Posix
platforms.  is that correct?

And thanks for the detailed description of the traces.

bug#34834: 26.1; Remote `eshell/mv' and `eshell/cp' on Windows: Opening output file: Invalid argument, c:/home/ ...

[Michael Albinus]

Eli Zaretskii <eliz@gnu.org> writes:

Hi Eli,

> IIUC, it works on Posix platforms by sheer luck, is that right?

Yes.

> IOW, on Posix platforms, we also don't recognize this is a remote file
> name until it's too late, and we only get away because the file name
> starts with a slash, and so looks like an absolute file name on Posix
> platforms.  is that correct?

Yes.

I will extend tramp-tests.el to test expand-file-name also with a
relative DIR argument. At least when running on Windows platforms, this
shall trigger this error (and prove it is corrected, when it is
corrected).

> And thanks for the detailed description of the traces.

Best regards, Michael.

bug#34834: 26.1; Remote `eshell/mv' and `eshell/cp' on Windows: Opening output file: Invalid argument, c:/home/ ...

[Eli Zaretskii]

> From: Michael Albinus <michael.albinus@gmx.de>
> Cc: jordan.t.wilson@gmx.com,  34834@debbugs.gnu.org
> Date: Fri, 29 Mar 2019 14:24:28 +0100
> 
> > IIUC, it works on Posix platforms by sheer luck, is that right?
> 
> Yes.
> 
> > IOW, on Posix platforms, we also don't recognize this is a remote file
> > name until it's too late, and we only get away because the file name
> > starts with a slash, and so looks like an absolute file name on Posix
> > platforms.  is that correct?
> 
> Yes.

OK, thanks for confirming my conclusions.  I will look into fixing
this, but it was important for me to know that the root cause isn't
specific to Windows.

> I will extend tramp-tests.el to test expand-file-name also with a
> relative DIR argument. At least when running on Windows platforms, this
> shall trigger this error (and prove it is corrected, when it is
> corrected).

Thanks.

bug#34834: 26.1; Remote `eshell/mv' and `eshell/cp' on Windows: Opening output file: Invalid argument, c:/home/ ...

[Michael Albinus]

Eli Zaretskii <eliz@gnu.org> writes:

Hi Eli,

> OK, thanks for confirming my conclusions.  I will look into fixing
> this, but it was important for me to know that the root cause isn't
> specific to Windows.

While you are at this, do you have a chance to check bug#26911? It is
another problem, but also somewhere in Fexpand_file_name.

Best regards, Michael.

bug#34834: 26.1; Remote `eshell/mv' and `eshell/cp' on Windows: Opening output file: Invalid argument, c:/home/ ...

[Eli Zaretskii]

> From: Michael Albinus <michael.albinus@gmx.de>
> Cc: jordan.t.wilson@gmx.com,  34834@debbugs.gnu.org
> Date: Fri, 29 Mar 2019 17:02:27 +0100
> 
> While you are at this, do you have a chance to check bug#26911? It is
> another problem, but also somewhere in Fexpand_file_name.

Will do.

bug#34834: 26.1; Remote `eshell/mv' and `eshell/cp' on Windows: Opening output file: Invalid argument, c:/home/ ...

[Paul Eggert]

I fixed part of Bug#26911, and perhaps that fix has affected this bug. See:

https://bugs.gnu.org/26911#19

bug#34834: 26.1; Remote `eshell/mv' and `eshell/cp' on Windows: Opening output file: Invalid argument, c:/home/ ...

[Paul Eggert]

My recently-installed patch 
<https://git.savannah.gnu.org/cgit/emacs.git/commit/?id=14fb657ba82da346d36f05f88da26f1c5498b798> 
for Bug#26911 may have affected whether Bug#34834 still exists. However I don't 
use MS-Windows so can't easily check.

bug#34834: 26.1; Remote `eshell/mv' and `eshell/cp' on Windows: Opening output file: Invalid argument, c:/home/ ...

[Eli Zaretskii]

> Date: Fri, 29 Mar 2019 20:46:10 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: jordan.t.wilson@gmx.com, 34834@debbugs.gnu.org
> 
> > From: Michael Albinus <michael.albinus@gmx.de>
> > Cc: jordan.t.wilson@gmx.com,  34834@debbugs.gnu.org
> > Date: Fri, 29 Mar 2019 17:02:27 +0100
> > 
> > While you are at this, do you have a chance to check bug#26911? It is
> > another problem, but also somewhere in Fexpand_file_name.
> 
> Will do.

And then I didn't.

Re-reading all that discussion now, I come to the conclusion that I
need your help, Michael, to understand your analysis in more detail.

Back then, you said:

> ======================================================================
> 1 -> (expand-file-name "123" "tmp/")
> | 2 -> (tramp-file-name-handler expand-file-name "tmp/" "/plinkx:detlefx:/home/albinus/")
> | | 3 -> (tramp-sh-handle-expand-file-name "tmp/" "/plinkx:detlefx:/home/albinus/")
> | | | 4 -> (tramp-file-name-handler file-name-as-directory "/plinkx:detlefx:/home/albinus/")
> | | | 4 <- tramp-file-name-handler: "/plinkx:detlefx:/home/albinus/"
> | | | 4 -> (expand-file-name "/home/albinus/tmp/")
> | | | 4 <- expand-file-name: "c:/home/albinus/tmp/"
> | | 3 <- tramp-sh-handle-expand-file-name: "/plinkx:detlefx:/home/albinus/tmp/"
> | 2 <- tramp-file-name-handler: "/plinkx:detlefx:/home/albinus/tmp/"
> 1 <- expand-file-name: "c:/plinkx:detlefx:/home/albinus/tmp/123"
> ======================================================================
> 
> The buffer's default-directory is remote, "/plinkx:detlefx:/home/albinus/".
> expand-file-name is called
> 
> --8<---------------cut here---------------start------------->8---
> (expand-file-name "123" "tmp/")
> --8<---------------cut here---------------end--------------->8---
> 
> This is entry "1" in the trace output. In function Fexpand_file_name,
> line 794 of fileio.c, there is the check
> 
> --8<---------------cut here---------------start------------->8---
>   handler = Ffind_file_name_handler (name, Qexpand_file_name);
> --8<---------------cut here---------------end--------------->8---
> 
> This returns Qnil, because name is "123". Then we have in line 823
> 
> --8<---------------cut here---------------start------------->8---
>   handler = Ffind_file_name_handler (default_directory, Qexpand_file_name);
> --8<---------------cut here---------------end--------------->8---
> 
> This returns Qnil again, because default_directory is "tmp/". That's all
> about checking for a file name handler. Later on, line 872, there is
> 
> --8<---------------cut here---------------start------------->8---
> 	default_directory = Fexpand_file_name (default_directory, Qnil);
> --8<---------------cut here---------------end--------------->8---
> 
> which belongs to entry "2" of the traces above. It returns the proper
> answer "/plinkx:detlefx:/home/albinus/tmp/". But this is too late; it
> must happen before the second call of Ffind_file_name_handler.
> Fexpand_file_name continues to work, it constructs the final answer
> "/plinkx:detlefx:/home/albinus/tmp/123" *plus* the drive letter (I
> suppose line 1047),

Could you please elaborate on the "this is too late" part?  What
exactly should have happened "before the second call of
Ffind_file_name_handler", and how would that help to fix this problem,
in your opinion?

I'm asking because the straightforward way of fixing this seems to
notice that the buffer's default-directory is remote, and refrain from
adding the drive letter in that case.  However, your comments above
seem to hint that another, perhaps simpler and/or more elegant, way
should be possible, and I wonder what could that other way be.

TIA

bug#34834: 26.1; Remote `eshell/mv' and `eshell/cp' on Windows: Opening output file: Invalid argument, c:/home/ ...

[Eli Zaretskii]

This bug is now solved by changes discussed in bug#26911, and the
solution tested.  So I'm closing this bug.  The solution will be in
Emacs 28.