Emacs syntax for filenames to mean "absolute location on the current remote host"?

Emacs syntax for filenames to mean "absolute location on the current remote host"?

[Jim Porter]

In Emacs, is there a way of spelling a file name to mean "an absolute 
name on the current remote host"? We have a way of spelling an absolute 
name on the local host ("/foo/bar"), and an absolute name on a specific 
remote ("/method:host:/foo/bar"), but I'm not aware of a way to make a 
string that means the equivalent of:

   (concat (file-name-remote-p default-directory) "/foo/bar")

Do we have a syntax for this? If not, could we add one? Maybe something 
like "/.::/foo/bar"? That would be a remote file name with the method of 
"." - i.e. "the current method" and no hostname (it would be inferred 
from the current remote).

----------------------------------------

If you'd like to know why I'm asking, read on. But be warned: I'm still 
trying to reason through all the logic here myself, so this may be a bit 
confusing (it certainly is for me!).

This is relevant for Eshell. In Eshell, you SSH into other hosts simply 
by using "cd /ssh:user@host:~". This is very convenient, but it produces 
some odd effects: when you're on a remote host like this, the meaning of 
"/foo/bar" is ambiguous. If you pass that to a command, it could mean:

   1) "/foo/bar" on your local filesystem (when the command is an Emacs 
Lisp function)

   2) "/foo/bar" on "host" (when the command is an external process)

To make matters more confusing, some Eshell commands implemented in Lisp 
will fall back to an external process if the Lisp version can't do what 
you asked, meaning that you need to know Eshell *very* well to be sure 
what will happen in these cases.[1]

To get around this, what if Eshell had a way of unambiguously referring 
to absolute file names, whether on your local system or the current 
remote host? For the local system, we already have a syntax available: 
quoted file names (e.g. "/:/foo/bar"). However, we'd need one for the 
current remote host too (see above).

As an addendum: it would be nice to have "/foo/bar" be usable too (i.e. 
not have this weird inconsistency). However, I think we'd need to 
perform some translations in Eshell to make this work, and I think 
that'd need to be defined on a per-command basis, so there would end up 
being cases where that broke. Unambiguous syntaxes would let us avoid 
that in the general case (at the cost of more typing).

[1] There's an optional Eshell module called "Electric Slash", but it 
doesn't handle this more-complex scenario.

Re: Emacs syntax for filenames to mean "absolute location on the current remote host"?

[Michael Albinus]

Jim Porter <jporterbugs@gmail.com> writes:

Hi Jim,

> In Emacs, is there a way of spelling a file name to mean "an absolute
> name on the current remote host"? We have a way of spelling an
> absolute name on the local host ("/foo/bar"), and an absolute name on
> a specific remote ("/method:host:/foo/bar"), but I'm not aware of a
> way to make a string that means the equivalent of:
>
>   (concat (file-name-remote-p default-directory) "/foo/bar")
>
> Do we have a syntax for this? If not, could we add one? Maybe
> something like "/.::/foo/bar"? That would be a remote file name with
> the method of "." - i.e. "the current method" and no hostname (it
> would be inferred from the current remote).

Anything goes. However, if we go this way, I wouldn't mix this with
Tramp's remote file name syntax, and the respective file name
handler. We would need a new file name handler similar to
file-name-non-special, which would be activated via "/.:/foo/bar" (a
little bit shorter).

> If you'd like to know why I'm asking, read on. But be warned: I'm
> still trying to reason through all the logic here myself, so this may
> be a bit confusing (it certainly is for me!).
>
> This is relevant for Eshell. In Eshell, you SSH into other hosts
> simply by using "cd /ssh:user@host:~". This is very convenient, but it
> produces some odd effects: when you're on a remote host like this, the
> meaning of "/foo/bar" is ambiguous. If you pass that to a command, it
> could mean:
>
>   1) "/foo/bar" on your local filesystem (when the command is an Emacs
>   Lisp function)
>
>   2) "/foo/bar" on "host" (when the command is an external process)

Thinking about, it seems we need this *only* in Eshell. And then the
need isn't such obvious.

What if you declare, that in Eshell an absolute file name "/foo/bar" is
always on "host"? Then it doesn't matter, whether a user writes "cd /foo/bar"
or "*cd /foo/bar". This would cover most use cases in Eshell, and it is
simpler to type than "cd /.:/foo/bar".

In the few cases a user means the local absolute file name "/foo/bar",
she still can type "cd /:/foo/bar".

Best regards, Michael.

Re: Emacs syntax for filenames to mean "absolute location on the current remote host"?

[Eli Zaretskii]

> From: Michael Albinus <michael.albinus@gmx.de>
> Cc: emacs-devel@gnu.org
> Date: Tue, 29 Aug 2023 09:30:45 +0200
> 
> What if you declare, that in Eshell an absolute file name "/foo/bar" is
> always on "host"? Then it doesn't matter, whether a user writes "cd /foo/bar"
> or "*cd /foo/bar". This would cover most use cases in Eshell, and it is
> simpler to type than "cd /.:/foo/bar".
> 
> In the few cases a user means the local absolute file name "/foo/bar",
> she still can type "cd /:/foo/bar".

I like this suggestion, FWIW.  It's simple to remember, easy to use,
and doesn't require us to come up with a whole new slew of handlers.

Re: Emacs syntax for filenames to mean "absolute location on the current remote host"?

[Jim Porter]

On 8/29/2023 12:30 AM, Michael Albinus wrote:
> Thinking about, it seems we need this *only* in Eshell. And then the
> need isn't such obvious.

Yeah, I wasn't sure if there was something I could easily piggyback off 
of for Eshell, or if it would take a lot of effort to make an Emacs-wide 
way of doing this.

> What if you declare, that in Eshell an absolute file name "/foo/bar" is
> always on "host"? Then it doesn't matter, whether a user writes "cd /foo/bar"
> or "*cd /foo/bar". This would cover most use cases in Eshell, and it is
> simpler to type than "cd /.:/foo/bar".

My main worry is that we only want to do this for absolute file names, 
not just things that *look* like absolute file names. For example 
"/foo/" might be a regexp instead of a directory name. We could add 
special handling for commands where we know they take file names, but 
that's laborious, and we'd never cover everything.

I see a couple options:

* Eshell lets you escape characters so they have their literal meaning, 
like \$ or '$' (if you don't want to use dollar-expansion). We could do 
the same for a leading /. That is, /foo/bar (no quotes) would get 
automatically expanded to /method:host:/foo/bar when on host. '/foo/bar' 
would just remain as /foo/bar though.

* I realized an important part of the Eshell side: we have Eshell-only 
variables! I could define a variable that expands to '(file-remote-p)'. 
The only question is what to call it. For example, we could use $/ or 
$:, so a user could type $//foo/bar or $:/foo/bar. I think I like $:, 
but I'm open to other suggestions. For single-char variable names, we 
can use any symbol except: ( ) { } [ ] < > " ' @ # $ * + - _ ?

We could even do both of these, and provide a user option for 
enabling/disabling the first one. That way, with the first one enabled, 
you get something pretty close to ordinary shells. However, if you don't 
like it, you still have the latter as a more-explicit option. (The 
latter is nice too because it should be a single line of code to implement.)

> In the few cases a user means the local absolute file name "/foo/bar",
> she still can type "cd /:/foo/bar".

The one edge case I'm not quite sure what to do about is: how should we 
spell "the user's home directory on localhost"? Using "/:~" *could* work 
(in that Eshell could recognize it and do the right thing), but it's 
also the way that you spell "the file named ~" elsewhere in Emacs. 
(Eshell would spell that \~ or '~'). Maybe that's not such a big deal 
though: we can just document this corner case and hope users don't get 
*too* confused.

Re: Emacs syntax for filenames to mean "absolute location on the current remote host"?

[Michael Albinus]

Jim Porter <jporterbugs@gmail.com> writes:

Hi Jim,

>> What if you declare, that in Eshell an absolute file name "/foo/bar" is
>> always on "host"? Then it doesn't matter, whether a user writes "cd /foo/bar"
>> or "*cd /foo/bar". This would cover most use cases in Eshell, and it is
>> simpler to type than "cd /.:/foo/bar".
>
> My main worry is that we only want to do this for absolute file names,
> not just things that *look* like absolute file names. For example
> "/foo/" might be a regexp instead of a directory name. We could add
> special handling for commands where we know they take file names, but
> that's laborious, and we'd never cover everything.
>
> I see a couple options:
>
> * Eshell lets you escape characters so they have their literal
>   meaning, like \$ or '$' (if you don't want to use
>   dollar-expansion). We could do the same for a leading /. That is,
>   /foo/bar (no quotes) would get automatically expanded to
>   /method:host:/foo/bar when on host. '/foo/bar' would just remain as
>  /foo/bar though.
>
> * I realized an important part of the Eshell side: we have Eshell-only
>   variables! I could define a variable that expands to
>   '(file-remote-p)'. The only question is what to call it. For
>   example, we could use $/ or $:, so a user could type $//foo/bar or
>   $:/foo/bar. I think I like $:, but I'm open to other
>   suggestions. For single-char variable names, we can use any symbol
>  except: ( ) { } [ ] < > " ' @ # $ * + - _ ?
>
> We could even do both of these, and provide a user option for
> enabling/disabling the first one. That way, with the first one
> enabled, you get something pretty close to ordinary shells. However,
> if you don't like it, you still have the latter as a more-explicit
> option. (The latter is nice too because it should be a single line of
> code to implement.)

I would let it to you.

>> In the few cases a user means the local absolute file name "/foo/bar",
>> she still can type "cd /:/foo/bar".
>
> The one edge case I'm not quite sure what to do about is: how should
> we spell "the user's home directory on localhost"? Using "/:~" *could*
> work (in that Eshell could recognize it and do the right thing), but
> it's also the way that you spell "the file named ~" elsewhere in
> Emacs. (Eshell would spell that \~ or '~'). Maybe that's not such a
> big deal though: we can just document this corner case and hope users
> don't get *too* confused.

IIUC, "~" in a file name means always the home directory when it is the
first character (of the local part in remote file names), or it comes
after a slash like in "/~". See (info "(elisp) File Name Expansion")

--8<---------------cut here---------------start------------->8---
     If the part of FILENAME before the first slash is ‘~’, it expands
     to your home directory, which is typically specified by the value
     of the ‘HOME’ environment variable (*note (emacs)General
     Variables::).  If the part before the first slash is ‘~USER’ and if
     USER is a valid login name, it expands to USER’s home directory.
     If you do not want this expansion for a relative FILENAME that
     might begin with a literal ‘~’, you can use ‘(concat
     (file-name-as-directory directory) filename)’ instead of
     ‘(expand-file-name filename directory)’.
--8<---------------cut here---------------end--------------->8---

Best regards, Michael.

Re: Emacs syntax for filenames to mean "absolute location on the current remote host"?

[Jim Porter]

On 8/31/2023 6:58 AM, Michael Albinus wrote:
> Jim Porter <jporterbugs@gmail.com> writes:
> 
>> The one edge case I'm not quite sure what to do about is: how should
>> we spell "the user's home directory on localhost"? Using "/:~" *could*
>> work (in that Eshell could recognize it and do the right thing), but
>> it's also the way that you spell "the file named ~" elsewhere in
>> Emacs. (Eshell would spell that \~ or '~'). Maybe that's not such a
>> big deal though: we can just document this corner case and hope users
>> don't get *too* confused.
> 
> IIUC, "~" in a file name means always the home directory when it is the
> first character (of the local part in remote file names), or it comes
> after a slash like in "/~". See (info "(elisp) File Name Expansion")

The main thing I'm worried about is this: if we say that in Eshell, 
"/foo/bar" is relative to the current remote host, then the same should 
be true for "~": it would refer to your remote homedir when applicable.

Then we'd want a way to refer to your *local* homedir no matter where 
you are. If we use "/:/foo/bar" to refer to the always-local "/foo/bar", 
then we might say "/:~" means your local homedir. But the Emacs manual says:

 > ‘/:’ can also prevent ‘~’ from being treated as a special character 
for a user’s home directory.

So according to that rule, "/:~" would mean "the file named tilde in the 
current directory".

I *think* what we'd want to do is to say, "Eshell's quoted file names 
work slightly differently from quoted filenames elsewhere in Emacs. 
Since Eshell expands tildes in file names on its own, '/:~' always means 
your local home directory. If you want to refer to a file whose name is 
tilde, you can use Eshell's escape sequences (etc, etc)."

I'll try to put together a patch that does this, and then people can try 
it out and see if it makes sense.

Re: Emacs syntax for filenames to mean "absolute location on the current remote host"?

[Michael Albinus]

Jim Porter <jporterbugs@gmail.com> writes:

Hi Jim,

>>> The one edge case I'm not quite sure what to do about is: how should
>>> we spell "the user's home directory on localhost"? Using "/:~" *could*
>>> work (in that Eshell could recognize it and do the right thing), but
>>> it's also the way that you spell "the file named ~" elsewhere in
>>> Emacs. (Eshell would spell that \~ or '~'). Maybe that's not such a
>>> big deal though: we can just document this corner case and hope users
>>> don't get *too* confused.
>> IIUC, "~" in a file name means always the home directory when it is
>> the
>> first character (of the local part in remote file names), or it comes
>> after a slash like in "/~". See (info "(elisp) File Name Expansion")
>
> The main thing I'm worried about is this: if we say that in Eshell,
> "/foo/bar" is relative to the current remote host, then the same
> should be true for "~": it would refer to your remote homedir when
> applicable.
>
> Then we'd want a way to refer to your *local* homedir no matter where
> you are. If we use "/:/foo/bar" to refer to the always-local
> "/foo/bar", then we might say "/:~" means your local homedir. But the
> Emacs manual says:
>
>> ‘/:’ can also prevent ‘~’ from being treated as a special character
>   for a user’s home directory.
>
> So according to that rule, "/:~" would mean "the file named tilde in
> the current directory".

Hmm, I've never stumbled over this sentence in the manual. And it is a
little bit vague: "can also prevent".

A short test shows that it doesn't if the tilde comes directly after the quoting:

--8<---------------cut here---------------start------------->8---
M-x cd RET /net RET
M-x pwd
  => /net
M-x cd RET /:~ RET
M-x pwd
  => /home/albinus

M-x cd RET /:~user RET
M-x pwd
  => /home/user
--8<---------------cut here---------------end--------------->8---

It even works for remote file names (although there are still some quirks):

--8<---------------cut here---------------start------------->8---
C-x C-f /ssh:ford:/:~ RET
  => dired buffer of remote user's home directory
--8<---------------cut here---------------end--------------->8---

What doesn't work are file names like "/:/tmp/~hack", as said in the manual.

> I *think* what we'd want to do is to say, "Eshell's quoted file names
> work slightly differently from quoted filenames elsewhere in
> Emacs. Since Eshell expands tildes in file names on its own, '/:~'
> always means your local home directory. If you want to refer to a file
> whose name is tilde, you can use Eshell's escape sequences (etc,
> etc)."

Perhaps it is sufficient to point to the above restriction? And in some
Eshell commands, the local home directory is given simply by the absence
of an argument, like in "cd".

> I'll try to put together a patch that does this, and then people can
> try it out and see if it makes sense.

Yep.

Best regards, Michael.

Re: Emacs syntax for filenames to mean "absolute location on the current remote host"?

[Jim Porter]

On 9/1/2023 11:13 AM, Michael Albinus wrote:
> Jim Porter <jporterbugs@gmail.com> writes:
> 
>> But the Emacs manual says:
>>
>>> ‘/:’ can also prevent ‘~’ from being treated as a special character
>>    for a user’s home directory.
>>
>> So according to that rule, "/:~" would mean "the file named tilde in
>> the current directory".
> 
> Hmm, I've never stumbled over this sentence in the manual. And it is a
> little bit vague: "can also prevent".
> 
> A short test shows that it doesn't if the tilde comes directly after the quoting:
> 
> --8<---------------cut here---------------start------------->8---
> M-x cd RET /net RET
> M-x pwd
>    => /net
> M-x cd RET /:~ RET
> M-x pwd
>    => /home/albinus
> 
> M-x cd RET /:~user RET
> M-x pwd
>    => /home/user
> --8<---------------cut here---------------end--------------->8---
> 
> It even works for remote file names (although there are still some quirks):
> 
> --8<---------------cut here---------------start------------->8---
> C-x C-f /ssh:ford:/:~ RET
>    => dired buffer of remote user's home directory
> --8<---------------cut here---------------end--------------->8---
> 
> What doesn't work are file names like "/:/tmp/~hack", as said in the manual.

Huh. I never realized it worked that way (though I never actually 
*tried* this case).

I think the first thing I'll do then is to try and understand *exactly* 
how quoted file names work, and then update the manual to clarify this. 
Once I know exactly how they work, it should be easier to be sure that I 
won't mess anything up with these Eshell improvements.

Re: Emacs syntax for filenames to mean "absolute location on the current remote host"?

[Jim Porter]

On 9/1/2023 11:36 AM, Jim Porter wrote:
> I think the first thing I'll do then is to try and understand *exactly* 
> how quoted file names work, and then update the manual to clarify this. 
> Once I know exactly how they work, it should be easier to be sure that I 
> won't mess anything up with these Eshell improvements.

Hmm, it turns out the behavior of quoted file names is inconsistent 
here. "C-x C-f /:~ RET" opens dired for the user's home dir on 
GNU/Linux, but opens a buffer for a new file named tilde on MS-Windows. 
I've filed bug#65685 for this.

Re: Emacs syntax for filenames to mean "absolute location on the current remote host"?

[Michael Albinus]

Jim Porter <jporterbugs@gmail.com> writes:

Hi Jim,

>> I think the first thing I'll do then is to try and understand
>> *exactly* how quoted file names work, and then update the manual to
>> clarify this. Once I know exactly how they work, it should be easier
>> to be sure that I won't mess anything up with these Eshell
>> improvements.
>
> Hmm, it turns out the behavior of quoted file names is inconsistent
> here. "C-x C-f /:~ RET" opens dired for the user's home dir on
> GNU/Linux, but opens a buffer for a new file named tilde on
> MS-Windows. I've filed bug#65685 for this.

Will check next days. I guess nobody has taken a closer look to quoted
file names and tilde expansion yet, and there are no respective test
cases in files-tests.el.

Best regards, Michael.