bug#69085: 29.2; Tramp: Extend tramp-make-copy-program-file-name via tramp-methods

bug#69085: 29.2; Tramp: Extend tramp-make-copy-program-file-name via tramp-methods

[Sean Devlin]

Hi folks,

This is a feature request.

Tramp lets you define a method for out-of-band file copying with the
tramp-methods keys tramp-copy-program, tramp-copy-args, and so on.

Currently, there is no way to customize the format of remote file
paths. The function tramp-make-copy-program-file-name is responsible for
this job, and it is hard-coded to format files as USER@HOST:FILE.

I was working on a custom Tramp method, and I ran into problems setting
up OOB copying due to this limitation.

It would be great if there were a key you could set in tramp-methods to
override this logic. For example, maybe something like:

(tramp-copy-format-file-name my-tramp-make-copy-program-file-name)

Where my-tramp-make-copy-program-file-name is a function taking a vector
as input and returning a string of the formatted file name (similar to
tramp-make-copy-program-file-name).

Alternately, maybe a format string (or something similar to
tramp-login-args/tramp-copy-args) would be sufficient. In my case, I
just need to add a constant prefix to the remote file name, so a format
string would be fine.

Thanks!

In GNU Emacs 29.2 (build 1, aarch64-apple-darwin21.6.0, NS
appkit-2113.60 Version 12.6.6 (Build 21G646)) of 2024-01-18 built on
armbob.lan
Windowing system distributor 'Apple'
System Description:  macOS 14

Configured using:
'configure --with-ns '--enable-locallisppath=/Library/Application
Support/Emacs/${version}/site-lisp:/Library/Application
Support/Emacs/site-lisp' --with-modules 'CFLAGS=-DFD_SETSIZE=10000
-DDARWIN_UNLIMITED_SELECT' --with-x-toolkit=no'

Configured features:
ACL GLIB GMP GNUTLS JPEG JSON LIBXML2 MODULES NOTIFY KQUEUE NS PDUMPER
PNG RSVG SQLITE3 THREADS TIFF TOOLKIT_SCROLL_BARS TREE_SITTER ZLIB

Important settings:
  value of $LANG: en_US.UTF-8
  locale-coding-system: utf-8-unix

Major mode: Lisp Interaction

Minor modes in effect:
  tooltip-mode: t
  global-eldoc-mode: t
  eldoc-mode: t
  show-paren-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  line-number-mode: t
  indent-tabs-mode: t
  transient-mark-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message mailcap yank-media puny dired
dired-loaddefs rfc822 mml mml-sec password-cache epa derived epg rfc6068
epg-config gnus-util text-property-search time-date subr-x mm-decode
mm-bodies mm-encode mail-parse rfc2231 mailabbrev gmm-utils mailheader
cl-loaddefs cl-lib sendmail rfc2047 rfc2045 ietf-drums mm-util
mail-prsvr mail-utils rmc iso-transl tooltip cconv eldoc paren electric
uniquify ediff-hook vc-hooks lisp-float-type elisp-mode mwheel
term/ns-win ns-win ucs-normalize mule-util term/common-win tool-bar dnd
fontset image regexp-opt fringe tabulated-list replace newcomment
text-mode lisp-mode prog-mode register page tab-bar menu-bar rfn-eshadow
isearch easymenu timer select scroll-bar mouse jit-lock font-lock syntax
font-core term/tty-colors frame minibuffer nadvice seq simple cl-generic
indonesian philippine cham georgian utf-8-lang misc-lang vietnamese
tibetan thai tai-viet lao korean japanese eucjp-ms cp51932 hebrew greek
romanian slovak czech european ethiopic indian cyrillic chinese
composite emoji-zwj charscript charprop case-table epa-hook
jka-cmpr-hook help abbrev obarray oclosure cl-preloaded button loaddefs
theme-loaddefs faces cus-face macroexp files window text-properties
overlay sha1 md5 base64 format env code-pages mule custom widget keymap
hashtable-print-readable backquote threads kqueue cocoa ns multi-tty
make-network-process emacs)

Memory information:
((conses 16 37536 8456)
(symbols 48 5045 0)
(strings 32 13121 2410)
(string-bytes 1 398579)
(vectors 16 10368)
(vector-slots 8 162386 14885)
(floats 8 21 23)
(intervals 56 217 0)
(buffers 984 10))

bug#69085: Acknowledgement (29.2; Tramp: Extend tramp-make-copy-program-file-name via tramp-methods)

[Sean Devlin]

I was able to work around the issue by advising tramp-make-copy-program-file-name:

(defun tramp-mymethod-make-copy-program-file-name (vec)
  (when (tramp-mymethod-file-name-p vec)
    (let ((localname
           (directory-file-name (tramp-file-name-unquote-localname vec))))
      (format "myprefix:%s" localname))))

(advice-add 'tramp-make-copy-program-file-name
            :before-until #'tramp-mymethod-make-copy-program-file-name)

bug#69085: Acknowledgement (29.2; Tramp: Extend tramp-make-copy-program-file-name via tramp-methods)

[Michael Albinus via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Sean Devlin <spd@toadstyle.org> writes:

Hi Sean,

> I was able to work around the issue by advising tramp-make-copy-program-file-name:
>
> (defun tramp-mymethod-make-copy-program-file-name (vec)
>   (when (tramp-mymethod-file-name-p vec)
>     (let ((localname
>            (directory-file-name (tramp-file-name-unquote-localname vec))))
>       (format "myprefix:%s" localname))))
>
> (advice-add 'tramp-make-copy-program-file-name
>             :before-until #'tramp-mymethod-make-copy-program-file-name)

It is nice that it works for you. But as I said the other message, I'd
like to understand the problem. So pls show your custome method spec.

Best regards, Michael.

bug#69085: 29.2; Tramp: Extend tramp-make-copy-program-file-name via tramp-methods

[Michael Albinus via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Sean Devlin <spd@toadstyle.org> writes:

> Hi folks,

Hi Sean,

> This is a feature request.
>
> Tramp lets you define a method for out-of-band file copying with the
> tramp-methods keys tramp-copy-program, tramp-copy-args, and so on.
>
> Currently, there is no way to customize the format of remote file
> paths. The function tramp-make-copy-program-file-name is responsible for
> this job, and it is hard-coded to format files as USER@HOST:FILE.

No. USER@HOST:FILE is only taken if the remote file name contains a
user. Otherwise, it takes
HOST:FILE.

> I was working on a custom Tramp method, and I ran into problems setting
> up OOB copying due to this limitation.
>
> It would be great if there were a key you could set in tramp-methods to
> override this logic. For example, maybe something like:
>
> (tramp-copy-format-file-name my-tramp-make-copy-program-file-name)
>
> Where my-tramp-make-copy-program-file-name is a function taking a vector
> as input and returning a string of the formatted file name (similar to
> tramp-make-copy-program-file-name).
>
> Alternately, maybe a format string (or something similar to
> tramp-login-args/tramp-copy-args) would be sufficient. In my case, I
> just need to add a constant prefix to the remote file name, so a format
> string would be fine.

Could you pls show your custom method spec, and what
tramp-copy-format-file-name returns for it?

> Thanks!

Best regards, Michael.

bug#69085: 29.2; Tramp: Extend tramp-make-copy-program-file-name via tramp-methods

[Sean Devlin]

[-- Attachment #1: Type: text/plain, Size: 2641 bytes --]

Hi Michael,

> On Feb 12, 2024, at 4:14 PM, Michael Albinus <michael.albinus@gmx.de> wrote:
> 
> Sean Devlin <spd@toadstyle.org <mailto:spd@toadstyle.org>> writes:
> 
>> Hi folks,
> 
> Hi Sean,
> 
>> This is a feature request.
>> 
>> Tramp lets you define a method for out-of-band file copying with the
>> tramp-methods keys tramp-copy-program, tramp-copy-args, and so on.
>> 
>> Currently, there is no way to customize the format of remote file
>> paths. The function tramp-make-copy-program-file-name is responsible for
>> this job, and it is hard-coded to format files as USER@HOST:FILE.
> 
> No. USER@HOST:FILE is only taken if the remote file name contains a
> user. Otherwise, it takes
> HOST:FILE.

You’re correct, my mistake.

> 
>> I was working on a custom Tramp method, and I ran into problems setting
>> up OOB copying due to this limitation.
>> 
>> It would be great if there were a key you could set in tramp-methods to
>> override this logic. For example, maybe something like:
>> 
>> (tramp-copy-format-file-name my-tramp-make-copy-program-file-name)
>> 
>> Where my-tramp-make-copy-program-file-name is a function taking a vector
>> as input and returning a string of the formatted file name (similar to
>> tramp-make-copy-program-file-name).
>> 
>> Alternately, maybe a format string (or something similar to
>> tramp-login-args/tramp-copy-args) would be sufficient. In my case, I
>> just need to add a constant prefix to the remote file name, so a format
>> string would be fine.
> 
> Could you pls show your custom method spec, and what
> tramp-copy-format-file-name returns for it?

Tramp-copy-format-file-name was just a proposal for how this feature might
work; it doesn’t exist in my own custom method spec.

The copy parameters for my method look like so:

(defconst tramp-mymethod
  '("mymethod"
    ...

    (tramp-copy-program “my-copy-program")
    (tramp-copy-args
     ((“-id" "%h")))))

Invocations of the copy program look like:

my-copy-program -id HOST SRC-FILE DEST-FILE

The HOST is not a DNS name in this case; instead, it’s an identifier for a sort of
abstract container.

For SRC-FILE and DEST-FILE, “remote” file paths just have a fixed prefix string,
i.e. it’s not dependent on the “host” name or the user.

So all I really need is a way to format “remote” file paths like:

(format "myprefix:%s" localname)

The current hack I use to implement this is to advise tramp-make-copy-file-name
as described in my other message.

Thanks for your help!

> 
>> Thanks!
> 
> Best regards, Michael.


[-- Attachment #2: Type: text/html, Size: 13458 bytes --]

bug#69085: 29.2; Tramp: Extend tramp-make-copy-program-file-name via tramp-methods

[Michael Albinus via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Sean Devlin <spd@toadstyle.org> writes:

> Hi Michael,

Hi Sean,

> The copy parameters for my method look like so:
>
> (defconst tramp-mymethod
>   '("mymethod"
>     ...
>
>     (tramp-copy-program “my-copy-program")
>     (tramp-copy-args
>      ((“-id" "%h")))))
>
> Invocations of the copy program look like:
>
> my-copy-program -id HOST SRC-FILE DEST-FILE
>
> The HOST is not a DNS name in this case; instead, it’s an identifier
> for a sort of
> abstract container.
>
> For SRC-FILE and DEST-FILE, “remote” file paths just have a fixed
> prefix string,
> i.e. it’s not dependent on the “host” name or the user.
>
> So all I really need is a way to format “remote” file paths like:
>
> (format "myprefix:%s" localname)

Hmm. Some questions:

- What if SRC-FILE or DEST-FILE is local, and the other one is remote?
  Would it still be something like "my-copy-program -id HOST SRC-FILE DEST-FILE"?

- In case both SRC-FILE and DEST-FILE are remote, are they always files
  on the same remote host identified as HOST? Or do you need also a copy
  between two different hosts, using "mymethod"?

- And if HOST is a kind of container, would it be possible for you to
  use the "docker" or "podman" method of Tramp directly?

> Thanks for your help!

Best regards, Michael.

bug#69085: 29.2; Tramp: Extend tramp-make-copy-program-file-name via tramp-methods

[Sean Devlin]

Hi Michael,

> On Feb 13, 2024, at 7:18 AM, Michael Albinus <michael.albinus@gmx.de> wrote:
> 
> Sean Devlin <spd@toadstyle.org> writes:
> 
>> Hi Michael,
> 
> Hi Sean,
> 
>> The copy parameters for my method look like so:
>> 
>> (defconst tramp-mymethod
>>  '("mymethod"
>>    ...
>> 
>>    (tramp-copy-program “my-copy-program")
>>    (tramp-copy-args
>>     ((“-id" "%h")))))
>> 
>> Invocations of the copy program look like:
>> 
>> my-copy-program -id HOST SRC-FILE DEST-FILE
>> 
>> The HOST is not a DNS name in this case; instead, it’s an identifier
>> for a sort of
>> abstract container.
>> 
>> For SRC-FILE and DEST-FILE, “remote” file paths just have a fixed
>> prefix string,
>> i.e. it’s not dependent on the “host” name or the user.
>> 
>> So all I really need is a way to format “remote” file paths like:
>> 
>> (format "myprefix:%s" localname)
> 
> Hmm. Some questions:
> 
> - What if SRC-FILE or DEST-FILE is local, and the other one is remote?
>  Would it still be something like "my-copy-program -id HOST SRC-FILE DEST-FILE"?

This is the main supported use case. One of the files must be local, and the other
must be “remote” (i.e. it’s inside the container HOST).

The path for the local file is formatted just like a typical local file path.

For the “remote” file, it’s formatted like a local file path but with a constant prefix in
front. The prefix is just a fixed string, and it’s just there to tell the copy program that
the path is inside the container.

> 
> - In case both SRC-FILE and DEST-FILE are remote, are they always files
>  on the same remote host identified as HOST? Or do you need also a copy
>  between two different hosts, using "mymethod"?

Direct copying between two hosts/containers isn’t supported with this program. To
handle this case, you need to copy from one remote to the local system, and then
copy that file into the second remote.

Based on my experiments, it seems that Tramp handles this scenario correctly and
uses the local system as an intermediate stop.

> 
> - And if HOST is a kind of container, would it be possible for you to
>  use the "docker" or "podman" method of Tramp directly?

I’ve looked a little bit at the code supporting these methods in tramp-container.el.
They seem similar, but not exactly the same. Also, they do not support OOB
copying, at least in the version of Tramp I’m reading (i.e. the version packaged with
Emacs 29.2).

Thanks for your help!

> 
>> Thanks for your help!
> 
> Best regards, Michael.

bug#69085: 29.2; Tramp: Extend tramp-make-copy-program-file-name via tramp-methods

[Michael Albinus via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Sean Devlin <spd@toadstyle.org> writes:

> Hi Michael,

Hi Sean,
 
>> - And if HOST is a kind of container, would it be possible for you to
>>  use the "docker" or "podman" method of Tramp directly?
>
> I’ve looked a little bit at the code supporting these methods in tramp-container.el.
> They seem similar, but not exactly the same. Also, they do not support OOB
> copying, at least in the version of Tramp I’m reading (i.e. the version packaged with
> Emacs 29.2).

Could you pls elaborate verbosely, what is missing in your use case?
Perhaps we could extend tramp-container.el that it is applicable for you
as well.

(I take the time to invest it with you, because your urgent problem seems
to be mitigated by your defadvice.)

> Thanks for your help!

Best regards, Michael.

bug#69085: 29.2; Tramp: Extend tramp-make-copy-program-file-name via tramp-methods

[Sean Devlin]

[-- Attachment #1: Type: text/plain, Size: 5112 bytes --]

Hi Michael,

> On Feb 14, 2024, at 1:59 AM, Michael Albinus <michael.albinus@gmx.de> wrote:
> 
> Sean Devlin <spd@toadstyle.org> writes:
> 
>> Hi Michael,
> 
> Hi Sean,
> 
>>> - And if HOST is a kind of container, would it be possible for you to
>>> use the "docker" or "podman" method of Tramp directly?
>> 
>> I’ve looked a little bit at the code supporting these methods in tramp-container.el.
>> They seem similar, but not exactly the same. Also, they do not support OOB
>> copying, at least in the version of Tramp I’m reading (i.e. the version packaged with
>> Emacs 29.2).
> 
> Could you pls elaborate verbosely, what is missing in your use case?
> Perhaps we could extend tramp-container.el that it is applicable for you
> as well.

Sure, let’s focus on tramp-container.el. I think this will allow us to be
concrete, which will make things simpler to discuss.

Suppose we want to add OOB copying to the Docker method. Docker
has a copy command with usage like this:

Usage:  docker cp [OPTIONS] CONTAINER:SRC_PATH DEST_PATH|-
	docker cp [OPTIONS] SRC_PATH|- CONTAINER:DEST_PATH

We could try to add support to the Docker method for OOB copying by
starting Emacs -Q and evaluating these forms:

(require 'tramp-container)
(add-to-list 'tramp-methods
             `(,tramp-docker-method
               (tramp-login-program ,tramp-docker-program)
               (tramp-login-args (("exec")
                                  ("-it")
                                  ("-u" "%u")
                                  ("%h")
			          ("%l")))
	       (tramp-direct-async (,tramp-default-remote-shell "-c"))
               (tramp-remote-shell ,tramp-default-remote-shell)
               (tramp-remote-shell-login ("-l"))
               (tramp-remote-shell-args ("-i" "-c"))

	       (tramp-copy-program ,tramp-docker-program)
	       (tramp-copy-args (("cp")))
	       ))

Outside of Emacs, start a Docker container running some image (e.g.
ubuntu:latest). An example command is:

$ docker run -it ubuntu:latest bash -l

Leave this running.

Next, create a large (i.e. greater than tramp-copy-size-limit) temporary
file for copying:

$ head -c 20000 /dev/random > /tmp/bigfile

Back in Emacs, evaluate these forms (with “my_container” replaced with
the name of your container):

(copy-file "/tmp/bigfile" "/docker:my_container:/tmp/bigfile")

(copy-file "/tmp/bigfile" "/docker:root@my_container:/tmp/bigfile2”)

The first copy will succeed, but the second will fail. This is because the
connection has an explicit user, so it will try to run a command like:

docker cp /tmp/bigfile root@my_container:/tmp/bigfile2

It will fail because it will interpret “root@my_container” as the container’s
name, and it will say that there is no container by that name.

I think what is needed is some way for the tramp-docker method to specify
the format for remote paths in the OOB copy command, since
tramp-make-copy-program-file-name will include the user if it is specified
in the connection.

One method might be to add a key to tramp-methods that lets you specify
the format similar to tramp-login-args and tramp-copy-args:

(tramp-copy-remote-file-format (("%h" ":" "%f")))

Where “%h” and “%f” are the host and the file’s local name, respectively.

Like tramp-login-args, the format could be a list of lists, and sublists could
be ignored when the required properties are absent. For example, the
format for scp-like copy programs might look like this:

(tramp-copy-remote-file-format
 (("%u" "@")
  ("%h" ":" "%f")))

Where “%u” is the user. So the leading “user@“ could be dropped if the
user is unspecified.

Unlike tramp-login-args and tramp-copy-args, this would need to
concatenate the strings together without spaces.

The function tramp-make-copy-program-file-name could check for this
method parameter and, if present, use it to format the result. For example,
the cond at the end could be changed like so:

(let ((fmt (tramp-get-method-parameter vec 'tramp-copy-remote-file-format)))
  (cond
   ((tramp-get-method-parameter vec 'tramp-remote-copy-program)
    localname)
   (fmt
    ;; Format the file name using fmt.
    )
   ((tramp-string-empty-or-nil-p user) (format "%s:%s" host localname))
   (t (format "%s@%s:%s" user host localname)))

Instead of a list of lists of strings, another method would be to specify a
function that does the work. The function could take the vector as input and
return the formatted file name. The tramp-make-copy-program-file-name
function could call that function (if specified) at the same place in that cond
form. This would be more flexible than the list of lists of strings, but it also
would be more complicated for the implementer and may be overkill.

> 
> (I take the time to invest it with you, because your urgent problem seems
> to be mitigated by your defadvice.)

I appreciate your help—thank you for taking the time!

> 
>> Thanks for your help!
> 
> Best regards, Michael.


[-- Attachment #2: Type: text/html, Size: 7945 bytes --]

bug#69085: 29.2; Tramp: Extend tramp-make-copy-program-file-name via tramp-methods

[Michael Albinus via Bug reports for GNU Emacs, the Swiss army knife of text editors]

[-- Attachment #1: Type: text/plain, Size: 2291 bytes --]

Sean Devlin <spd@toadstyle.org> writes:

> Hi Michael,

Hi Sean,

> Suppose we want to add OOB copying to the Docker method. Docker
> has a copy command with usage like this:
>
> Usage:  docker cp [OPTIONS] CONTAINER:SRC_PATH DEST_PATH|-
> docker cp [OPTIONS] SRC_PATH|- CONTAINER:DEST_PATH
>
> We could try to add support to the Docker method for OOB copying by
> starting Emacs -Q and evaluating these forms:
>
> (add-to-list 'tramp-methods
>              `(,tramp-docker-method
>                (tramp-login-program ,tramp-docker-program)
>                (tramp-login-args (("exec")
>                                   ("-it")
>                                   ("-u" "%u")
>                                   ("%h")
>          ("%l")))
>       (tramp-direct-async (,tramp-default-remote-shell "-c"))
>                (tramp-remote-shell ,tramp-default-remote-shell)
>                (tramp-remote-shell-login ("-l"))
>                (tramp-remote-shell-args ("-i" "-c"))
>
>       (tramp-copy-program ,tramp-docker-program)
>       (tramp-copy-args (("cp")))
>       ))

Well, if "docker cp" doesn't need the user, why do we use the user at
all? Couldn't we get rid of it in this case? That is, don't use the
("-u" "%u") in tramp-login-args.

> I think what is needed is some way for the tramp-docker method to
> specify the format for remote paths in the OOB copy command, since
> tramp-make-copy-program-file-name will include the user if it is
> specified in the connection.
>
> One method might be to add a key to tramp-methods that lets you
> specify the format similar to tramp-login-args and tramp-copy-args:

I've tried another approach. In Tramp 2.7, I've added a new method
"dockercp" which doesn't use the user at all, and which adds
tramp-copy-program and tramp-copy-args.

Do you have a chance to install Tramp from its repository like

--8<---------------cut here---------------start------------->8---
# git clone https://git.savannah.gnu.org/git/tramp.git
--8<---------------cut here---------------end--------------->8---

Then pls modify tramp-container.el with the appended patch. After that,
you should be able to access your docker container like
/dockercp:CONTAINER:/path/to/file. And if you copy large files, "docker
cp" will be applied.

Best regards, Michael.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Type: text/x-patch, Size: 2688 bytes --]

diff --git a/lisp/tramp-container.el b/lisp/tramp-container.el
index 1f578949..caea6b47 100644
--- a/lisp/tramp-container.el
+++ b/lisp/tramp-container.el
@@ -31,6 +31,7 @@
 ;; Open a file on a running Docker container:
 ;;
 ;;     C-x C-f /docker:USER@CONTAINER:/path/to/file
+;;     C-x C-f /dockercp:CONTAINER:/path/to/file
 ;;
 ;; or Podman:
 ;;
@@ -141,6 +142,11 @@ If it is nil, the default context will be used."
 (defconst tramp-docker-method "docker"
   "Tramp method name to use to connect to Docker containers.")

+;;;###tramp-autoload
+(defconst tramp-dockercp-method "dockercp"
+  "Tramp method name to use to connect to Docker containers.
+This is for out-of-band connections.")
+
 ;;;###tramp-autoload
 (defconst tramp-podman-method "podman"
   "Tramp method name to use to connect to Podman containers.")
@@ -183,7 +189,7 @@ BODY is the backend specific code."
 (defun tramp-container--completion-function (method)
   "List running containers available for connection.
 METHOD is the Tramp method to be used for \"ps\", either
-`tramp-docker-method' or `tramp-podman-method'.
+`tramp-docker-method', `tramp-dockercp-method' or `tramp-podman-method'.

 This function is used by `tramp-set-completion-function', please
 see its function help for a description of the format."
@@ -375,6 +381,20 @@ see its function help for a description of the format."
                 (tramp-remote-shell-login ("-l"))
                 (tramp-remote-shell-args ("-i" "-c"))))

+ (add-to-list 'tramp-methods
+              `(,tramp-dockercp-method
+                (tramp-login-program ,tramp-docker-program)
+                (tramp-login-args (("exec")
+                                   ("-it")
+                                   ("%h")
+			           ("%l")))
+		(tramp-direct-async (,tramp-default-remote-shell "-c"))
+                (tramp-remote-shell ,tramp-default-remote-shell)
+                (tramp-remote-shell-login ("-l"))
+                (tramp-remote-shell-args ("-i" "-c"))
+		(tramp-copy-program ,tramp-docker-program)
+		(tramp-copy-args (("cp")))))
+
  (add-to-list 'tramp-methods
               `(,tramp-podman-method
                 (tramp-login-program ,tramp-podman-program)
@@ -431,6 +451,10 @@ see its function help for a description of the format."
   tramp-docker-method
   `((tramp-container--completion-function ,tramp-docker-method)))

+ (tramp-set-completion-function
+  tramp-dockercp-method
+  `((tramp-container--completion-function ,tramp-dockercp-method)))
+
  (tramp-set-completion-function
   tramp-podman-method
   `((tramp-container--completion-function ,tramp-podman-method)))

bug#69085: 29.2; Tramp: Extend tramp-make-copy-program-file-name via tramp-methods

[Sean Devlin]

Hi Michael,

>> I think what is needed is some way for the tramp-docker method to
>> specify the format for remote paths in the OOB copy command, since
>> tramp-make-copy-program-file-name will include the user if it is
>> specified in the connection.
>> 
>> One method might be to add a key to tramp-methods that lets you
>> specify the format similar to tramp-login-args and tramp-copy-args:
> 
> I've tried another approach. In Tramp 2.7, I've added a new method
> "dockercp" which doesn't use the user at all, and which adds
> tramp-copy-program and tramp-copy-args.
> 
> Do you have a chance to install Tramp from its repository like
> 
> --8<---------------cut here---------------start------------->8---
> # git clone https://git.savannah.gnu.org/git/tramp.git
> --8<---------------cut here---------------end--------------->8---
> 
> Then pls modify tramp-container.el with the appended patch. After that,
> you should be able to access your docker container like
> /dockercp:CONTAINER:/path/to/file. And if you copy large files, "docker
> cp" will be applied.

The patch does work as described, but it doesn’t integrate well with typical
workflows.

For example, I’m likely to start a session from Eshell or Dired browsing
around the remote. At this point, I may not realize I will need to copy files to
or from the remote. I would start with the normal “docker” method, because
I usually want to have an explicit user for my connection, so that I can run
programs as the user.

Now, suppose I do need to copy a file back to the host. I need to:

1. Check the size of the file. If it’s small, I can copy it normally using the
inline method.

2. If the file is big, I now need to switch to the other method, so I open a new
connection to the same container under the “dockercp” method.

3. I also need to find my way back to the same directory. This may be a
minor nuisance, but for example it won’t show up in the Recentf list, since
Emacs and Tramp and Recentf will not know that
/docker:user@container:/path/to/dir and /dockercp:container:/path/to/dir are
related.

4. Finally, I copy the file.

So while this approach does work for Docker containers, it adds some
ergonomic overhead for the user making it inconvenient to use.

Thanks,

Sean

bug#69085: 29.2; Tramp: Extend tramp-make-copy-program-file-name via tramp-methods

[Michael Albinus via Bug reports for GNU Emacs, the Swiss army knife of text editors]

[-- Attachment #1: Type: text/plain, Size: 610 bytes --]

Sean Devlin <spd@toadstyle.org> writes:

> Hi Michael,

Hi Sean,

> The patch does work as described, but it doesn’t integrate well with typical
> workflows.
>
> So while this approach does work for Docker containers, it adds some
> ergonomic overhead for the user making it inconvenient to use.

I see your point. Appended is a new patch, which addresses your
inconvenience. Could you pls apply it on top of Tramp 2.7 instead of the
other patch? The method "dockercp" allows now user names, which are
ignored when calling "docker cp ...".

> Thanks,
>
> Sean

Best regards, Michael.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Type: text/x-patch, Size: 8424 bytes --]

diff --git a/lisp/tramp-container.el b/lisp/tramp-container.el
index 1f578949..ff2e11b5 100644
--- a/lisp/tramp-container.el
+++ b/lisp/tramp-container.el
@@ -31,15 +31,20 @@
 ;; Open a file on a running Docker container:
 ;;
 ;;     C-x C-f /docker:USER@CONTAINER:/path/to/file
+;;     C-x C-f /dockercp:USER@CONTAINER:/path/to/file
 ;;
 ;; or Podman:
 ;;
 ;;     C-x C-f /podman:USER@CONTAINER:/path/to/file
+;;     C-x C-f /podmancp:USER@CONTAINER:/path/to/file
 ;;
 ;; Where:
 ;;     USER          is the user on the container to connect as (optional).
 ;;     CONTAINER     is the container to connect to.
 ;;
+;; "docker" and "podman" are inline methods, "dockercp" and "podmancp"
+;; are out-of-band methods.
+;;
 ;;
 ;;
 ;; Open file in a Kubernetes container:
@@ -141,10 +146,20 @@ If it is nil, the default context will be used."
 (defconst tramp-docker-method "docker"
   "Tramp method name to use to connect to Docker containers.")

+;;;###tramp-autoload
+(defconst tramp-dockercp-method "dockercp"
+  "Tramp method name to use to connect to Docker containers.
+This is for out-of-band connections.")
+
 ;;;###tramp-autoload
 (defconst tramp-podman-method "podman"
   "Tramp method name to use to connect to Podman containers.")

+;;;###tramp-autoload
+(defconst tramp-podmancp-method "podmancp"
+  "Tramp method name to use to connect to Podman containers.
+This is for out-of-band connections.")
+
 ;;;###tramp-autoload
 (defconst tramp-kubernetes-method "kubernetes"
   "Tramp method name to use to connect to Kubernetes containers.")
@@ -183,7 +198,8 @@ BODY is the backend specific code."
 (defun tramp-container--completion-function (method)
   "List running containers available for connection.
 METHOD is the Tramp method to be used for \"ps\", either
-`tramp-docker-method' or `tramp-podman-method'.
+`tramp-docker-method', `tramp-dockercp-method', `tramp-podman-method',
+or `tramp-podmancp-method'.

 This function is used by `tramp-set-completion-function', please
 see its function help for a description of the format."
@@ -375,6 +391,22 @@ see its function help for a description of the format."
                 (tramp-remote-shell-login ("-l"))
                 (tramp-remote-shell-args ("-i" "-c"))))

+ (add-to-list 'tramp-methods
+              `(,tramp-dockercp-method
+                (tramp-login-program ,tramp-docker-program)
+                (tramp-login-args (("exec")
+                                   ("-it")
+                                   ("-u" "%u")
+                                   ("%h")
+			           ("%l")))
+		(tramp-direct-async (,tramp-default-remote-shell "-c"))
+                (tramp-remote-shell ,tramp-default-remote-shell)
+                (tramp-remote-shell-login ("-l"))
+                (tramp-remote-shell-args ("-i" "-c"))
+		(tramp-copy-program ,tramp-docker-program)
+		(tramp-copy-args (("cp")))
+		(tramp-copy-file-name (("%h" ":") ("%f")))))
+
  (add-to-list 'tramp-methods
               `(,tramp-podman-method
                 (tramp-login-program ,tramp-podman-program)
@@ -388,6 +420,22 @@ see its function help for a description of the format."
                 (tramp-remote-shell-login ("-l"))
                 (tramp-remote-shell-args ("-i" "-c"))))

+ (add-to-list 'tramp-methods
+              `(,tramp-podmancp-method
+                (tramp-login-program ,tramp-podman-program)
+                (tramp-login-args (("exec")
+                                   ("-it")
+                                   ("-u" "%u")
+                                   ("%h")
+			           ("%l")))
+		(tramp-direct-async (,tramp-default-remote-shell "-c"))
+                (tramp-remote-shell ,tramp-default-remote-shell)
+                (tramp-remote-shell-login ("-l"))
+                (tramp-remote-shell-args ("-i" "-c"))
+		(tramp-copy-program ,tramp-podman-program)
+		(tramp-copy-args (("cp")))
+		(tramp-copy-file-name (("%h" ":") ("%f")))))
+
  (add-to-list 'tramp-methods
               `(,tramp-kubernetes-method
                 (tramp-login-program ,tramp-kubernetes-program)
@@ -431,10 +479,18 @@ see its function help for a description of the format."
   tramp-docker-method
   `((tramp-container--completion-function ,tramp-docker-method)))

+ (tramp-set-completion-function
+  tramp-dockercp-method
+  `((tramp-container--completion-function ,tramp-dockercp-method)))
+
  (tramp-set-completion-function
   tramp-podman-method
   `((tramp-container--completion-function ,tramp-podman-method)))

+ (tramp-set-completion-function
+  tramp-podmancp-method
+  `((tramp-container--completion-function ,tramp-podmancp-method)))
+
  (tramp-set-completion-function
   tramp-kubernetes-method
   `((tramp-kubernetes--completion-function ,tramp-kubernetes-method)))
diff --git a/lisp/tramp-sh.el b/lisp/tramp-sh.el
index 3557b3a1..b3ab5609 100644
--- a/lisp/tramp-sh.el
+++ b/lisp/tramp-sh.el
@@ -282,6 +282,7 @@ The string is used in `tramp-methods'.")
                 (tramp-copy-program         "nc")
                 ;; We use "-v" for better error tracking.
                 (tramp-copy-args            (("-w" "1") ("-v") ("%h") ("%r")))
+                (tramp-copy-file-name       (("%f")))
                 (tramp-remote-copy-program  "nc")
                 ;; We use "-p" as required for newer busyboxes.  For older
                 ;; busybox/nc versions, the value must be (("-l") ("%r")).  This
@@ -2399,10 +2400,10 @@ The method used must be an out-of-band method."
 			#'file-name-as-directory
 		      #'identity)
 		    (if v1
-			(tramp-make-copy-program-file-name v1)
+			(tramp-make-copy-file-name v1)
 		      (file-name-unquote filename)))
 	    target (if v2
-		       (tramp-make-copy-program-file-name v2)
+		       (tramp-make-copy-file-name v2)
 		     (file-name-unquote newname)))

       ;; Check for listener port.
@@ -5559,13 +5560,17 @@ raises an error."
    string
    ""))

-(defun tramp-make-copy-program-file-name (vec)
+(defun tramp-make-copy-file-name (vec)
   "Create a file name suitable for `scp', `pscp', or `nc' and workalikes."
   (let ((method (tramp-file-name-method vec))
 	(user (tramp-file-name-user vec))
 	(host (tramp-file-name-host vec))
 	(localname
 	 (directory-file-name (tramp-file-name-unquote-localname vec))))
+    ;; Set the default value of connection property "copy-file-name".
+    (unless (tramp-get-method-parameter vec 'tramp-copy-file-name)
+      (tramp-set-connection-property
+       vec "copy-file-name" '(("%u" "@") ("%h" ":") ("%f"))))
     (when (string-match-p tramp-ipv6-regexp host)
       (setq host (format "[%s]" host)))
     ;; This does not work for MS Windows scp, if there are characters
@@ -5573,11 +5578,10 @@ raises an error."
     ;; checking in scp, we use it when available.
     (unless (string-match-p (rx "ftp" eos) method)
       (setq localname (tramp-unquote-shell-quote-argument localname)))
-    (cond
-     ((tramp-get-method-parameter vec 'tramp-remote-copy-program)
-      localname)
-     ((tramp-string-empty-or-nil-p user) (format "%s:%s" host localname))
-     (t (format "%s@%s:%s" user host localname)))))
+    (string-join
+     (apply #'tramp-expand-args vec 'tramp-copy-file-name
+	    (list ?h (or host "") ?u (or user "") ?f localname))
+     "")))

 (defun tramp-method-out-of-band-p (vec size)
   "Return t if this is an out-of-band method, nil otherwise."
diff --git a/lisp/tramp.el b/lisp/tramp.el
index 2efee234..0e64b779 100644
--- a/lisp/tramp.el
+++ b/lisp/tramp.el
@@ -301,6 +301,14 @@ pair of the form (KEY VALUE).  The following KEYs are defined:
     This specifies the list of parameters to pass to the above mentioned
     program, the hints for `tramp-login-args' also apply here.

+  * `tramp-copy-file-name'
+    The remote source or destination file name for out-of-band methods.
+    You can use the \"%u\" and \"%h\" like in `tramp-login-args'.
+    Additionally, \"%f\" denotes the local file name part.  It will be
+    expanded to a string without spaces between the elements of the list.
+
+    The default value is \\='((\"%u\" \"@\") (\"%h\" \":\") (\"%f\"))
+
   * `tramp-copy-env'
      A list of environment variables and their values, which will
      be set when calling `tramp-copy-program'.

bug#69085: 29.2; Tramp: Extend tramp-make-copy-program-file-name via tramp-methods

[Sean Devlin]

Hi Michael,

> I see your point. Appended is a new patch, which addresses your
> inconvenience. Could you pls apply it on top of Tramp 2.7 instead of the
> other patch? The method "dockercp" allows now user names, which are
> ignored when calling "docker cp ...".

This works perfectly! I see no issues with this patch.

Thanks for spending so much time on this, I really appreciate your help!

Sean

bug#69085: 29.2; Tramp: Extend tramp-make-copy-program-file-name via tramp-methods

[Michael Albinus via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Version: 30.1

Sean Devlin <spd@toadstyle.org> writes:

> Hi Michael,

Hi Sean,

>> I see your point. Appended is a new patch, which addresses your
>> inconvenience. Could you pls apply it on top of Tramp 2.7 instead of the
>> other patch? The method "dockercp" allows now user names, which are
>> ignored when calling "docker cp ...".
>
> This works perfectly! I see no issues with this patch.

Thanks for the feedback.

Over the weekend, I ran Tramp regression tests. There were still some
minor annoyances with the new methods, but that's fixed now.

I've pushed an extended version of the patch to the repositories. Will
be available with Tramp 2.7.1 and Emacs 30.1.

Closing the bug.

> Thanks for spending so much time on this, I really appreciate your help!
>
> Sean

Best regards, Michael.

bug#69085: 29.2; Tramp: Extend tramp-make-copy-program-file-name via tramp-methods

[Sean Devlin]

Hi Michael,

> Thanks for the feedback.
> 
> Over the weekend, I ran Tramp regression tests. There were still some
> minor annoyances with the new methods, but that's fixed now.
> 
> I've pushed an extended version of the patch to the repositories. Will
> be available with Tramp 2.7.1 and Emacs 30.1.
> 
> Closing the bug.

Sounds good. Thanks again for your help with this.

Sean