* bug#69085: 29.2; Tramp: Extend tramp-make-copy-program-file-name via tramp-methods @ 2024-02-12 20:08 Sean Devlin [not found] ` <handler.69085.B.17077686248331.ack@debbugs.gnu.org> 2024-02-12 21:14 ` bug#69085: 29.2; Tramp: Extend tramp-make-copy-program-file-name via tramp-methods Michael Albinus via Bug reports for GNU Emacs, the Swiss army knife of text editors 0 siblings, 2 replies; 15+ messages in thread From: Sean Devlin @ 2024-02-12 20:08 UTC (permalink / raw) To: 69085 Hi folks, This is a feature request. Tramp lets you define a method for out-of-band file copying with the tramp-methods keys tramp-copy-program, tramp-copy-args, and so on. Currently, there is no way to customize the format of remote file paths. The function tramp-make-copy-program-file-name is responsible for this job, and it is hard-coded to format files as USER@HOST:FILE. I was working on a custom Tramp method, and I ran into problems setting up OOB copying due to this limitation. It would be great if there were a key you could set in tramp-methods to override this logic. For example, maybe something like: (tramp-copy-format-file-name my-tramp-make-copy-program-file-name) Where my-tramp-make-copy-program-file-name is a function taking a vector as input and returning a string of the formatted file name (similar to tramp-make-copy-program-file-name). Alternately, maybe a format string (or something similar to tramp-login-args/tramp-copy-args) would be sufficient. In my case, I just need to add a constant prefix to the remote file name, so a format string would be fine. Thanks! In GNU Emacs 29.2 (build 1, aarch64-apple-darwin21.6.0, NS appkit-2113.60 Version 12.6.6 (Build 21G646)) of 2024-01-18 built on armbob.lan Windowing system distributor 'Apple' System Description: macOS 14 Configured using: 'configure --with-ns '--enable-locallisppath=/Library/Application Support/Emacs/${version}/site-lisp:/Library/Application Support/Emacs/site-lisp' --with-modules 'CFLAGS=-DFD_SETSIZE=10000 -DDARWIN_UNLIMITED_SELECT' --with-x-toolkit=no' Configured features: ACL GLIB GMP GNUTLS JPEG JSON LIBXML2 MODULES NOTIFY KQUEUE NS PDUMPER PNG RSVG SQLITE3 THREADS TIFF TOOLKIT_SCROLL_BARS TREE_SITTER ZLIB Important settings: value of $LANG: en_US.UTF-8 locale-coding-system: utf-8-unix Major mode: Lisp Interaction Minor modes in effect: tooltip-mode: t global-eldoc-mode: t eldoc-mode: t show-paren-mode: t electric-indent-mode: t mouse-wheel-mode: t tool-bar-mode: t menu-bar-mode: t file-name-shadow-mode: t global-font-lock-mode: t font-lock-mode: t blink-cursor-mode: t line-number-mode: t indent-tabs-mode: t transient-mark-mode: t auto-composition-mode: t auto-encryption-mode: t auto-compression-mode: t Load-path shadows: None found. Features: (shadow sort mail-extr emacsbug message mailcap yank-media puny dired dired-loaddefs rfc822 mml mml-sec password-cache epa derived epg rfc6068 epg-config gnus-util text-property-search time-date subr-x mm-decode mm-bodies mm-encode mail-parse rfc2231 mailabbrev gmm-utils mailheader cl-loaddefs cl-lib sendmail rfc2047 rfc2045 ietf-drums mm-util mail-prsvr mail-utils rmc iso-transl tooltip cconv eldoc paren electric uniquify ediff-hook vc-hooks lisp-float-type elisp-mode mwheel term/ns-win ns-win ucs-normalize mule-util term/common-win tool-bar dnd fontset image regexp-opt fringe tabulated-list replace newcomment text-mode lisp-mode prog-mode register page tab-bar menu-bar rfn-eshadow isearch easymenu timer select scroll-bar mouse jit-lock font-lock syntax font-core term/tty-colors frame minibuffer nadvice seq simple cl-generic indonesian philippine cham georgian utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao korean japanese eucjp-ms cp51932 hebrew greek romanian slovak czech european ethiopic indian cyrillic chinese composite emoji-zwj charscript charprop case-table epa-hook jka-cmpr-hook help abbrev obarray oclosure cl-preloaded button loaddefs theme-loaddefs faces cus-face macroexp files window text-properties overlay sha1 md5 base64 format env code-pages mule custom widget keymap hashtable-print-readable backquote threads kqueue cocoa ns multi-tty make-network-process emacs) Memory information: ((conses 16 37536 8456) (symbols 48 5045 0) (strings 32 13121 2410) (string-bytes 1 398579) (vectors 16 10368) (vector-slots 8 162386 14885) (floats 8 21 23) (intervals 56 217 0) (buffers 984 10)) ^ permalink raw reply [flat|nested] 15+ messages in thread
[parent not found: <handler.69085.B.17077686248331.ack@debbugs.gnu.org>]
* bug#69085: Acknowledgement (29.2; Tramp: Extend tramp-make-copy-program-file-name via tramp-methods) [not found] ` <handler.69085.B.17077686248331.ack@debbugs.gnu.org> @ 2024-02-12 20:14 ` Sean Devlin 2024-02-12 21:16 ` Michael Albinus via Bug reports for GNU Emacs, the Swiss army knife of text editors 0 siblings, 1 reply; 15+ messages in thread From: Sean Devlin @ 2024-02-12 20:14 UTC (permalink / raw) To: 69085 I was able to work around the issue by advising tramp-make-copy-program-file-name: (defun tramp-mymethod-make-copy-program-file-name (vec) (when (tramp-mymethod-file-name-p vec) (let ((localname (directory-file-name (tramp-file-name-unquote-localname vec)))) (format "myprefix:%s" localname)))) (advice-add 'tramp-make-copy-program-file-name :before-until #'tramp-mymethod-make-copy-program-file-name) ^ permalink raw reply [flat|nested] 15+ messages in thread
* bug#69085: Acknowledgement (29.2; Tramp: Extend tramp-make-copy-program-file-name via tramp-methods) 2024-02-12 20:14 ` bug#69085: Acknowledgement (29.2; Tramp: Extend tramp-make-copy-program-file-name via tramp-methods) Sean Devlin @ 2024-02-12 21:16 ` Michael Albinus via Bug reports for GNU Emacs, the Swiss army knife of text editors 0 siblings, 0 replies; 15+ messages in thread From: Michael Albinus via Bug reports for GNU Emacs, the Swiss army knife of text editors @ 2024-02-12 21:16 UTC (permalink / raw) To: Sean Devlin; +Cc: 69085 Sean Devlin <spd@toadstyle.org> writes: Hi Sean, > I was able to work around the issue by advising tramp-make-copy-program-file-name: > > (defun tramp-mymethod-make-copy-program-file-name (vec) > (when (tramp-mymethod-file-name-p vec) > (let ((localname > (directory-file-name (tramp-file-name-unquote-localname vec)))) > (format "myprefix:%s" localname)))) > > (advice-add 'tramp-make-copy-program-file-name > :before-until #'tramp-mymethod-make-copy-program-file-name) It is nice that it works for you. But as I said the other message, I'd like to understand the problem. So pls show your custome method spec. Best regards, Michael. ^ permalink raw reply [flat|nested] 15+ messages in thread
* bug#69085: 29.2; Tramp: Extend tramp-make-copy-program-file-name via tramp-methods 2024-02-12 20:08 bug#69085: 29.2; Tramp: Extend tramp-make-copy-program-file-name via tramp-methods Sean Devlin [not found] ` <handler.69085.B.17077686248331.ack@debbugs.gnu.org> @ 2024-02-12 21:14 ` Michael Albinus via Bug reports for GNU Emacs, the Swiss army knife of text editors 2024-02-12 21:51 ` Sean Devlin 1 sibling, 1 reply; 15+ messages in thread From: Michael Albinus via Bug reports for GNU Emacs, the Swiss army knife of text editors @ 2024-02-12 21:14 UTC (permalink / raw) To: Sean Devlin; +Cc: 69085 Sean Devlin <spd@toadstyle.org> writes: > Hi folks, Hi Sean, > This is a feature request. > > Tramp lets you define a method for out-of-band file copying with the > tramp-methods keys tramp-copy-program, tramp-copy-args, and so on. > > Currently, there is no way to customize the format of remote file > paths. The function tramp-make-copy-program-file-name is responsible for > this job, and it is hard-coded to format files as USER@HOST:FILE. No. USER@HOST:FILE is only taken if the remote file name contains a user. Otherwise, it takes HOST:FILE. > I was working on a custom Tramp method, and I ran into problems setting > up OOB copying due to this limitation. > > It would be great if there were a key you could set in tramp-methods to > override this logic. For example, maybe something like: > > (tramp-copy-format-file-name my-tramp-make-copy-program-file-name) > > Where my-tramp-make-copy-program-file-name is a function taking a vector > as input and returning a string of the formatted file name (similar to > tramp-make-copy-program-file-name). > > Alternately, maybe a format string (or something similar to > tramp-login-args/tramp-copy-args) would be sufficient. In my case, I > just need to add a constant prefix to the remote file name, so a format > string would be fine. Could you pls show your custom method spec, and what tramp-copy-format-file-name returns for it? > Thanks! Best regards, Michael. ^ permalink raw reply [flat|nested] 15+ messages in thread
* bug#69085: 29.2; Tramp: Extend tramp-make-copy-program-file-name via tramp-methods 2024-02-12 21:14 ` bug#69085: 29.2; Tramp: Extend tramp-make-copy-program-file-name via tramp-methods Michael Albinus via Bug reports for GNU Emacs, the Swiss army knife of text editors @ 2024-02-12 21:51 ` Sean Devlin 2024-02-13 12:18 ` Michael Albinus via Bug reports for GNU Emacs, the Swiss army knife of text editors 0 siblings, 1 reply; 15+ messages in thread From: Sean Devlin @ 2024-02-12 21:51 UTC (permalink / raw) To: Michael Albinus; +Cc: 69085 [-- Attachment #1: Type: text/plain, Size: 2641 bytes --] Hi Michael, > On Feb 12, 2024, at 4:14 PM, Michael Albinus <michael.albinus@gmx.de> wrote: > > Sean Devlin <spd@toadstyle.org <mailto:spd@toadstyle.org>> writes: > >> Hi folks, > > Hi Sean, > >> This is a feature request. >> >> Tramp lets you define a method for out-of-band file copying with the >> tramp-methods keys tramp-copy-program, tramp-copy-args, and so on. >> >> Currently, there is no way to customize the format of remote file >> paths. The function tramp-make-copy-program-file-name is responsible for >> this job, and it is hard-coded to format files as USER@HOST:FILE. > > No. USER@HOST:FILE is only taken if the remote file name contains a > user. Otherwise, it takes > HOST:FILE. You’re correct, my mistake. > >> I was working on a custom Tramp method, and I ran into problems setting >> up OOB copying due to this limitation. >> >> It would be great if there were a key you could set in tramp-methods to >> override this logic. For example, maybe something like: >> >> (tramp-copy-format-file-name my-tramp-make-copy-program-file-name) >> >> Where my-tramp-make-copy-program-file-name is a function taking a vector >> as input and returning a string of the formatted file name (similar to >> tramp-make-copy-program-file-name). >> >> Alternately, maybe a format string (or something similar to >> tramp-login-args/tramp-copy-args) would be sufficient. In my case, I >> just need to add a constant prefix to the remote file name, so a format >> string would be fine. > > Could you pls show your custom method spec, and what > tramp-copy-format-file-name returns for it? Tramp-copy-format-file-name was just a proposal for how this feature might work; it doesn’t exist in my own custom method spec. The copy parameters for my method look like so: (defconst tramp-mymethod '("mymethod" ... (tramp-copy-program “my-copy-program") (tramp-copy-args ((“-id" "%h"))))) Invocations of the copy program look like: my-copy-program -id HOST SRC-FILE DEST-FILE The HOST is not a DNS name in this case; instead, it’s an identifier for a sort of abstract container. For SRC-FILE and DEST-FILE, “remote” file paths just have a fixed prefix string, i.e. it’s not dependent on the “host” name or the user. So all I really need is a way to format “remote” file paths like: (format "myprefix:%s" localname) The current hack I use to implement this is to advise tramp-make-copy-file-name as described in my other message. Thanks for your help! > >> Thanks! > > Best regards, Michael. [-- Attachment #2: Type: text/html, Size: 13458 bytes --] ^ permalink raw reply [flat|nested] 15+ messages in thread
* bug#69085: 29.2; Tramp: Extend tramp-make-copy-program-file-name via tramp-methods 2024-02-12 21:51 ` Sean Devlin @ 2024-02-13 12:18 ` Michael Albinus via Bug reports for GNU Emacs, the Swiss army knife of text editors 2024-02-13 20:18 ` Sean Devlin 0 siblings, 1 reply; 15+ messages in thread From: Michael Albinus via Bug reports for GNU Emacs, the Swiss army knife of text editors @ 2024-02-13 12:18 UTC (permalink / raw) To: Sean Devlin; +Cc: 69085 Sean Devlin <spd@toadstyle.org> writes: > Hi Michael, Hi Sean, > The copy parameters for my method look like so: > > (defconst tramp-mymethod > '("mymethod" > ... > > (tramp-copy-program “my-copy-program") > (tramp-copy-args > ((“-id" "%h"))))) > > Invocations of the copy program look like: > > my-copy-program -id HOST SRC-FILE DEST-FILE > > The HOST is not a DNS name in this case; instead, it’s an identifier > for a sort of > abstract container. > > For SRC-FILE and DEST-FILE, “remote” file paths just have a fixed > prefix string, > i.e. it’s not dependent on the “host” name or the user. > > So all I really need is a way to format “remote” file paths like: > > (format "myprefix:%s" localname) Hmm. Some questions: - What if SRC-FILE or DEST-FILE is local, and the other one is remote? Would it still be something like "my-copy-program -id HOST SRC-FILE DEST-FILE"? - In case both SRC-FILE and DEST-FILE are remote, are they always files on the same remote host identified as HOST? Or do you need also a copy between two different hosts, using "mymethod"? - And if HOST is a kind of container, would it be possible for you to use the "docker" or "podman" method of Tramp directly? > Thanks for your help! Best regards, Michael. ^ permalink raw reply [flat|nested] 15+ messages in thread
* bug#69085: 29.2; Tramp: Extend tramp-make-copy-program-file-name via tramp-methods 2024-02-13 12:18 ` Michael Albinus via Bug reports for GNU Emacs, the Swiss army knife of text editors @ 2024-02-13 20:18 ` Sean Devlin 2024-02-14 6:59 ` Michael Albinus via Bug reports for GNU Emacs, the Swiss army knife of text editors 0 siblings, 1 reply; 15+ messages in thread From: Sean Devlin @ 2024-02-13 20:18 UTC (permalink / raw) To: Michael Albinus; +Cc: 69085 Hi Michael, > On Feb 13, 2024, at 7:18 AM, Michael Albinus <michael.albinus@gmx.de> wrote: > > Sean Devlin <spd@toadstyle.org> writes: > >> Hi Michael, > > Hi Sean, > >> The copy parameters for my method look like so: >> >> (defconst tramp-mymethod >> '("mymethod" >> ... >> >> (tramp-copy-program “my-copy-program") >> (tramp-copy-args >> ((“-id" "%h"))))) >> >> Invocations of the copy program look like: >> >> my-copy-program -id HOST SRC-FILE DEST-FILE >> >> The HOST is not a DNS name in this case; instead, it’s an identifier >> for a sort of >> abstract container. >> >> For SRC-FILE and DEST-FILE, “remote” file paths just have a fixed >> prefix string, >> i.e. it’s not dependent on the “host” name or the user. >> >> So all I really need is a way to format “remote” file paths like: >> >> (format "myprefix:%s" localname) > > Hmm. Some questions: > > - What if SRC-FILE or DEST-FILE is local, and the other one is remote? > Would it still be something like "my-copy-program -id HOST SRC-FILE DEST-FILE"? This is the main supported use case. One of the files must be local, and the other must be “remote” (i.e. it’s inside the container HOST). The path for the local file is formatted just like a typical local file path. For the “remote” file, it’s formatted like a local file path but with a constant prefix in front. The prefix is just a fixed string, and it’s just there to tell the copy program that the path is inside the container. > > - In case both SRC-FILE and DEST-FILE are remote, are they always files > on the same remote host identified as HOST? Or do you need also a copy > between two different hosts, using "mymethod"? Direct copying between two hosts/containers isn’t supported with this program. To handle this case, you need to copy from one remote to the local system, and then copy that file into the second remote. Based on my experiments, it seems that Tramp handles this scenario correctly and uses the local system as an intermediate stop. > > - And if HOST is a kind of container, would it be possible for you to > use the "docker" or "podman" method of Tramp directly? I’ve looked a little bit at the code supporting these methods in tramp-container.el. They seem similar, but not exactly the same. Also, they do not support OOB copying, at least in the version of Tramp I’m reading (i.e. the version packaged with Emacs 29.2). Thanks for your help! > >> Thanks for your help! > > Best regards, Michael. ^ permalink raw reply [flat|nested] 15+ messages in thread
* bug#69085: 29.2; Tramp: Extend tramp-make-copy-program-file-name via tramp-methods 2024-02-13 20:18 ` Sean Devlin @ 2024-02-14 6:59 ` Michael Albinus via Bug reports for GNU Emacs, the Swiss army knife of text editors 2024-02-15 0:57 ` Sean Devlin 0 siblings, 1 reply; 15+ messages in thread From: Michael Albinus via Bug reports for GNU Emacs, the Swiss army knife of text editors @ 2024-02-14 6:59 UTC (permalink / raw) To: Sean Devlin; +Cc: 69085 Sean Devlin <spd@toadstyle.org> writes: > Hi Michael, Hi Sean, >> - And if HOST is a kind of container, would it be possible for you to >> use the "docker" or "podman" method of Tramp directly? > > I’ve looked a little bit at the code supporting these methods in tramp-container.el. > They seem similar, but not exactly the same. Also, they do not support OOB > copying, at least in the version of Tramp I’m reading (i.e. the version packaged with > Emacs 29.2). Could you pls elaborate verbosely, what is missing in your use case? Perhaps we could extend tramp-container.el that it is applicable for you as well. (I take the time to invest it with you, because your urgent problem seems to be mitigated by your defadvice.) > Thanks for your help! Best regards, Michael. ^ permalink raw reply [flat|nested] 15+ messages in thread
* bug#69085: 29.2; Tramp: Extend tramp-make-copy-program-file-name via tramp-methods 2024-02-14 6:59 ` Michael Albinus via Bug reports for GNU Emacs, the Swiss army knife of text editors @ 2024-02-15 0:57 ` Sean Devlin 2024-02-16 14:54 ` Michael Albinus via Bug reports for GNU Emacs, the Swiss army knife of text editors 0 siblings, 1 reply; 15+ messages in thread From: Sean Devlin @ 2024-02-15 0:57 UTC (permalink / raw) To: Michael Albinus; +Cc: 69085 [-- Attachment #1: Type: text/plain, Size: 5112 bytes --] Hi Michael, > On Feb 14, 2024, at 1:59 AM, Michael Albinus <michael.albinus@gmx.de> wrote: > > Sean Devlin <spd@toadstyle.org> writes: > >> Hi Michael, > > Hi Sean, > >>> - And if HOST is a kind of container, would it be possible for you to >>> use the "docker" or "podman" method of Tramp directly? >> >> I’ve looked a little bit at the code supporting these methods in tramp-container.el. >> They seem similar, but not exactly the same. Also, they do not support OOB >> copying, at least in the version of Tramp I’m reading (i.e. the version packaged with >> Emacs 29.2). > > Could you pls elaborate verbosely, what is missing in your use case? > Perhaps we could extend tramp-container.el that it is applicable for you > as well. Sure, let’s focus on tramp-container.el. I think this will allow us to be concrete, which will make things simpler to discuss. Suppose we want to add OOB copying to the Docker method. Docker has a copy command with usage like this: Usage: docker cp [OPTIONS] CONTAINER:SRC_PATH DEST_PATH|- docker cp [OPTIONS] SRC_PATH|- CONTAINER:DEST_PATH We could try to add support to the Docker method for OOB copying by starting Emacs -Q and evaluating these forms: (require 'tramp-container) (add-to-list 'tramp-methods `(,tramp-docker-method (tramp-login-program ,tramp-docker-program) (tramp-login-args (("exec") ("-it") ("-u" "%u") ("%h") ("%l"))) (tramp-direct-async (,tramp-default-remote-shell "-c")) (tramp-remote-shell ,tramp-default-remote-shell) (tramp-remote-shell-login ("-l")) (tramp-remote-shell-args ("-i" "-c")) (tramp-copy-program ,tramp-docker-program) (tramp-copy-args (("cp"))) )) Outside of Emacs, start a Docker container running some image (e.g. ubuntu:latest). An example command is: $ docker run -it ubuntu:latest bash -l Leave this running. Next, create a large (i.e. greater than tramp-copy-size-limit) temporary file for copying: $ head -c 20000 /dev/random > /tmp/bigfile Back in Emacs, evaluate these forms (with “my_container” replaced with the name of your container): (copy-file "/tmp/bigfile" "/docker:my_container:/tmp/bigfile") (copy-file "/tmp/bigfile" "/docker:root@my_container:/tmp/bigfile2”) The first copy will succeed, but the second will fail. This is because the connection has an explicit user, so it will try to run a command like: docker cp /tmp/bigfile root@my_container:/tmp/bigfile2 It will fail because it will interpret “root@my_container” as the container’s name, and it will say that there is no container by that name. I think what is needed is some way for the tramp-docker method to specify the format for remote paths in the OOB copy command, since tramp-make-copy-program-file-name will include the user if it is specified in the connection. One method might be to add a key to tramp-methods that lets you specify the format similar to tramp-login-args and tramp-copy-args: (tramp-copy-remote-file-format (("%h" ":" "%f"))) Where “%h” and “%f” are the host and the file’s local name, respectively. Like tramp-login-args, the format could be a list of lists, and sublists could be ignored when the required properties are absent. For example, the format for scp-like copy programs might look like this: (tramp-copy-remote-file-format (("%u" "@") ("%h" ":" "%f"))) Where “%u” is the user. So the leading “user@“ could be dropped if the user is unspecified. Unlike tramp-login-args and tramp-copy-args, this would need to concatenate the strings together without spaces. The function tramp-make-copy-program-file-name could check for this method parameter and, if present, use it to format the result. For example, the cond at the end could be changed like so: (let ((fmt (tramp-get-method-parameter vec 'tramp-copy-remote-file-format))) (cond ((tramp-get-method-parameter vec 'tramp-remote-copy-program) localname) (fmt ;; Format the file name using fmt. ) ((tramp-string-empty-or-nil-p user) (format "%s:%s" host localname)) (t (format "%s@%s:%s" user host localname))) Instead of a list of lists of strings, another method would be to specify a function that does the work. The function could take the vector as input and return the formatted file name. The tramp-make-copy-program-file-name function could call that function (if specified) at the same place in that cond form. This would be more flexible than the list of lists of strings, but it also would be more complicated for the implementer and may be overkill. > > (I take the time to invest it with you, because your urgent problem seems > to be mitigated by your defadvice.) I appreciate your help—thank you for taking the time! > >> Thanks for your help! > > Best regards, Michael. [-- Attachment #2: Type: text/html, Size: 7945 bytes --] ^ permalink raw reply [flat|nested] 15+ messages in thread
* bug#69085: 29.2; Tramp: Extend tramp-make-copy-program-file-name via tramp-methods 2024-02-15 0:57 ` Sean Devlin @ 2024-02-16 14:54 ` Michael Albinus via Bug reports for GNU Emacs, the Swiss army knife of text editors 2024-02-16 17:51 ` Sean Devlin 0 siblings, 1 reply; 15+ messages in thread From: Michael Albinus via Bug reports for GNU Emacs, the Swiss army knife of text editors @ 2024-02-16 14:54 UTC (permalink / raw) To: Sean Devlin; +Cc: 69085 [-- Attachment #1: Type: text/plain, Size: 2291 bytes --] Sean Devlin <spd@toadstyle.org> writes: > Hi Michael, Hi Sean, > Suppose we want to add OOB copying to the Docker method. Docker > has a copy command with usage like this: > > Usage: docker cp [OPTIONS] CONTAINER:SRC_PATH DEST_PATH|- > docker cp [OPTIONS] SRC_PATH|- CONTAINER:DEST_PATH > > We could try to add support to the Docker method for OOB copying by > starting Emacs -Q and evaluating these forms: > > (add-to-list 'tramp-methods > `(,tramp-docker-method > (tramp-login-program ,tramp-docker-program) > (tramp-login-args (("exec") > ("-it") > ("-u" "%u") > ("%h") > ("%l"))) > (tramp-direct-async (,tramp-default-remote-shell "-c")) > (tramp-remote-shell ,tramp-default-remote-shell) > (tramp-remote-shell-login ("-l")) > (tramp-remote-shell-args ("-i" "-c")) > > (tramp-copy-program ,tramp-docker-program) > (tramp-copy-args (("cp"))) > )) Well, if "docker cp" doesn't need the user, why do we use the user at all? Couldn't we get rid of it in this case? That is, don't use the ("-u" "%u") in tramp-login-args. > I think what is needed is some way for the tramp-docker method to > specify the format for remote paths in the OOB copy command, since > tramp-make-copy-program-file-name will include the user if it is > specified in the connection. > > One method might be to add a key to tramp-methods that lets you > specify the format similar to tramp-login-args and tramp-copy-args: I've tried another approach. In Tramp 2.7, I've added a new method "dockercp" which doesn't use the user at all, and which adds tramp-copy-program and tramp-copy-args. Do you have a chance to install Tramp from its repository like --8<---------------cut here---------------start------------->8--- # git clone https://git.savannah.gnu.org/git/tramp.git --8<---------------cut here---------------end--------------->8--- Then pls modify tramp-container.el with the appended patch. After that, you should be able to access your docker container like /dockercp:CONTAINER:/path/to/file. And if you copy large files, "docker cp" will be applied. Best regards, Michael. [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #2: Type: text/x-patch, Size: 2688 bytes --] diff --git a/lisp/tramp-container.el b/lisp/tramp-container.el index 1f578949..caea6b47 100644 --- a/lisp/tramp-container.el +++ b/lisp/tramp-container.el @@ -31,6 +31,7 @@ ;; Open a file on a running Docker container: ;; ;; C-x C-f /docker:USER@CONTAINER:/path/to/file +;; C-x C-f /dockercp:CONTAINER:/path/to/file ;; ;; or Podman: ;; @@ -141,6 +142,11 @@ If it is nil, the default context will be used." (defconst tramp-docker-method "docker" "Tramp method name to use to connect to Docker containers.") +;;;###tramp-autoload +(defconst tramp-dockercp-method "dockercp" + "Tramp method name to use to connect to Docker containers. +This is for out-of-band connections.") + ;;;###tramp-autoload (defconst tramp-podman-method "podman" "Tramp method name to use to connect to Podman containers.") @@ -183,7 +189,7 @@ BODY is the backend specific code." (defun tramp-container--completion-function (method) "List running containers available for connection. METHOD is the Tramp method to be used for \"ps\", either -`tramp-docker-method' or `tramp-podman-method'. +`tramp-docker-method', `tramp-dockercp-method' or `tramp-podman-method'. This function is used by `tramp-set-completion-function', please see its function help for a description of the format." @@ -375,6 +381,20 @@ see its function help for a description of the format." (tramp-remote-shell-login ("-l")) (tramp-remote-shell-args ("-i" "-c")))) + (add-to-list 'tramp-methods + `(,tramp-dockercp-method + (tramp-login-program ,tramp-docker-program) + (tramp-login-args (("exec") + ("-it") + ("%h") + ("%l"))) + (tramp-direct-async (,tramp-default-remote-shell "-c")) + (tramp-remote-shell ,tramp-default-remote-shell) + (tramp-remote-shell-login ("-l")) + (tramp-remote-shell-args ("-i" "-c")) + (tramp-copy-program ,tramp-docker-program) + (tramp-copy-args (("cp"))))) + (add-to-list 'tramp-methods `(,tramp-podman-method (tramp-login-program ,tramp-podman-program) @@ -431,6 +451,10 @@ see its function help for a description of the format." tramp-docker-method `((tramp-container--completion-function ,tramp-docker-method))) + (tramp-set-completion-function + tramp-dockercp-method + `((tramp-container--completion-function ,tramp-dockercp-method))) + (tramp-set-completion-function tramp-podman-method `((tramp-container--completion-function ,tramp-podman-method))) ^ permalink raw reply related [flat|nested] 15+ messages in thread
* bug#69085: 29.2; Tramp: Extend tramp-make-copy-program-file-name via tramp-methods 2024-02-16 14:54 ` Michael Albinus via Bug reports for GNU Emacs, the Swiss army knife of text editors @ 2024-02-16 17:51 ` Sean Devlin 2024-02-17 17:57 ` Michael Albinus via Bug reports for GNU Emacs, the Swiss army knife of text editors 0 siblings, 1 reply; 15+ messages in thread From: Sean Devlin @ 2024-02-16 17:51 UTC (permalink / raw) To: Michael Albinus; +Cc: 69085 Hi Michael, >> I think what is needed is some way for the tramp-docker method to >> specify the format for remote paths in the OOB copy command, since >> tramp-make-copy-program-file-name will include the user if it is >> specified in the connection. >> >> One method might be to add a key to tramp-methods that lets you >> specify the format similar to tramp-login-args and tramp-copy-args: > > I've tried another approach. In Tramp 2.7, I've added a new method > "dockercp" which doesn't use the user at all, and which adds > tramp-copy-program and tramp-copy-args. > > Do you have a chance to install Tramp from its repository like > > --8<---------------cut here---------------start------------->8--- > # git clone https://git.savannah.gnu.org/git/tramp.git > --8<---------------cut here---------------end--------------->8--- > > Then pls modify tramp-container.el with the appended patch. After that, > you should be able to access your docker container like > /dockercp:CONTAINER:/path/to/file. And if you copy large files, "docker > cp" will be applied. The patch does work as described, but it doesn’t integrate well with typical workflows. For example, I’m likely to start a session from Eshell or Dired browsing around the remote. At this point, I may not realize I will need to copy files to or from the remote. I would start with the normal “docker” method, because I usually want to have an explicit user for my connection, so that I can run programs as the user. Now, suppose I do need to copy a file back to the host. I need to: 1. Check the size of the file. If it’s small, I can copy it normally using the inline method. 2. If the file is big, I now need to switch to the other method, so I open a new connection to the same container under the “dockercp” method. 3. I also need to find my way back to the same directory. This may be a minor nuisance, but for example it won’t show up in the Recentf list, since Emacs and Tramp and Recentf will not know that /docker:user@container:/path/to/dir and /dockercp:container:/path/to/dir are related. 4. Finally, I copy the file. So while this approach does work for Docker containers, it adds some ergonomic overhead for the user making it inconvenient to use. Thanks, Sean ^ permalink raw reply [flat|nested] 15+ messages in thread
* bug#69085: 29.2; Tramp: Extend tramp-make-copy-program-file-name via tramp-methods 2024-02-16 17:51 ` Sean Devlin @ 2024-02-17 17:57 ` Michael Albinus via Bug reports for GNU Emacs, the Swiss army knife of text editors 2024-02-19 19:19 ` Sean Devlin 0 siblings, 1 reply; 15+ messages in thread From: Michael Albinus via Bug reports for GNU Emacs, the Swiss army knife of text editors @ 2024-02-17 17:57 UTC (permalink / raw) To: Sean Devlin; +Cc: 69085 [-- Attachment #1: Type: text/plain, Size: 610 bytes --] Sean Devlin <spd@toadstyle.org> writes: > Hi Michael, Hi Sean, > The patch does work as described, but it doesn’t integrate well with typical > workflows. > > So while this approach does work for Docker containers, it adds some > ergonomic overhead for the user making it inconvenient to use. I see your point. Appended is a new patch, which addresses your inconvenience. Could you pls apply it on top of Tramp 2.7 instead of the other patch? The method "dockercp" allows now user names, which are ignored when calling "docker cp ...". > Thanks, > > Sean Best regards, Michael. [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #2: Type: text/x-patch, Size: 8424 bytes --] diff --git a/lisp/tramp-container.el b/lisp/tramp-container.el index 1f578949..ff2e11b5 100644 --- a/lisp/tramp-container.el +++ b/lisp/tramp-container.el @@ -31,15 +31,20 @@ ;; Open a file on a running Docker container: ;; ;; C-x C-f /docker:USER@CONTAINER:/path/to/file +;; C-x C-f /dockercp:USER@CONTAINER:/path/to/file ;; ;; or Podman: ;; ;; C-x C-f /podman:USER@CONTAINER:/path/to/file +;; C-x C-f /podmancp:USER@CONTAINER:/path/to/file ;; ;; Where: ;; USER is the user on the container to connect as (optional). ;; CONTAINER is the container to connect to. ;; +;; "docker" and "podman" are inline methods, "dockercp" and "podmancp" +;; are out-of-band methods. +;; ;; ;; ;; Open file in a Kubernetes container: @@ -141,10 +146,20 @@ If it is nil, the default context will be used." (defconst tramp-docker-method "docker" "Tramp method name to use to connect to Docker containers.") +;;;###tramp-autoload +(defconst tramp-dockercp-method "dockercp" + "Tramp method name to use to connect to Docker containers. +This is for out-of-band connections.") + ;;;###tramp-autoload (defconst tramp-podman-method "podman" "Tramp method name to use to connect to Podman containers.") +;;;###tramp-autoload +(defconst tramp-podmancp-method "podmancp" + "Tramp method name to use to connect to Podman containers. +This is for out-of-band connections.") + ;;;###tramp-autoload (defconst tramp-kubernetes-method "kubernetes" "Tramp method name to use to connect to Kubernetes containers.") @@ -183,7 +198,8 @@ BODY is the backend specific code." (defun tramp-container--completion-function (method) "List running containers available for connection. METHOD is the Tramp method to be used for \"ps\", either -`tramp-docker-method' or `tramp-podman-method'. +`tramp-docker-method', `tramp-dockercp-method', `tramp-podman-method', +or `tramp-podmancp-method'. This function is used by `tramp-set-completion-function', please see its function help for a description of the format." @@ -375,6 +391,22 @@ see its function help for a description of the format." (tramp-remote-shell-login ("-l")) (tramp-remote-shell-args ("-i" "-c")))) + (add-to-list 'tramp-methods + `(,tramp-dockercp-method + (tramp-login-program ,tramp-docker-program) + (tramp-login-args (("exec") + ("-it") + ("-u" "%u") + ("%h") + ("%l"))) + (tramp-direct-async (,tramp-default-remote-shell "-c")) + (tramp-remote-shell ,tramp-default-remote-shell) + (tramp-remote-shell-login ("-l")) + (tramp-remote-shell-args ("-i" "-c")) + (tramp-copy-program ,tramp-docker-program) + (tramp-copy-args (("cp"))) + (tramp-copy-file-name (("%h" ":") ("%f"))))) + (add-to-list 'tramp-methods `(,tramp-podman-method (tramp-login-program ,tramp-podman-program) @@ -388,6 +420,22 @@ see its function help for a description of the format." (tramp-remote-shell-login ("-l")) (tramp-remote-shell-args ("-i" "-c")))) + (add-to-list 'tramp-methods + `(,tramp-podmancp-method + (tramp-login-program ,tramp-podman-program) + (tramp-login-args (("exec") + ("-it") + ("-u" "%u") + ("%h") + ("%l"))) + (tramp-direct-async (,tramp-default-remote-shell "-c")) + (tramp-remote-shell ,tramp-default-remote-shell) + (tramp-remote-shell-login ("-l")) + (tramp-remote-shell-args ("-i" "-c")) + (tramp-copy-program ,tramp-podman-program) + (tramp-copy-args (("cp"))) + (tramp-copy-file-name (("%h" ":") ("%f"))))) + (add-to-list 'tramp-methods `(,tramp-kubernetes-method (tramp-login-program ,tramp-kubernetes-program) @@ -431,10 +479,18 @@ see its function help for a description of the format." tramp-docker-method `((tramp-container--completion-function ,tramp-docker-method))) + (tramp-set-completion-function + tramp-dockercp-method + `((tramp-container--completion-function ,tramp-dockercp-method))) + (tramp-set-completion-function tramp-podman-method `((tramp-container--completion-function ,tramp-podman-method))) + (tramp-set-completion-function + tramp-podmancp-method + `((tramp-container--completion-function ,tramp-podmancp-method))) + (tramp-set-completion-function tramp-kubernetes-method `((tramp-kubernetes--completion-function ,tramp-kubernetes-method))) diff --git a/lisp/tramp-sh.el b/lisp/tramp-sh.el index 3557b3a1..b3ab5609 100644 --- a/lisp/tramp-sh.el +++ b/lisp/tramp-sh.el @@ -282,6 +282,7 @@ The string is used in `tramp-methods'.") (tramp-copy-program "nc") ;; We use "-v" for better error tracking. (tramp-copy-args (("-w" "1") ("-v") ("%h") ("%r"))) + (tramp-copy-file-name (("%f"))) (tramp-remote-copy-program "nc") ;; We use "-p" as required for newer busyboxes. For older ;; busybox/nc versions, the value must be (("-l") ("%r")). This @@ -2399,10 +2400,10 @@ The method used must be an out-of-band method." #'file-name-as-directory #'identity) (if v1 - (tramp-make-copy-program-file-name v1) + (tramp-make-copy-file-name v1) (file-name-unquote filename))) target (if v2 - (tramp-make-copy-program-file-name v2) + (tramp-make-copy-file-name v2) (file-name-unquote newname))) ;; Check for listener port. @@ -5559,13 +5560,17 @@ raises an error." string "")) -(defun tramp-make-copy-program-file-name (vec) +(defun tramp-make-copy-file-name (vec) "Create a file name suitable for `scp', `pscp', or `nc' and workalikes." (let ((method (tramp-file-name-method vec)) (user (tramp-file-name-user vec)) (host (tramp-file-name-host vec)) (localname (directory-file-name (tramp-file-name-unquote-localname vec)))) + ;; Set the default value of connection property "copy-file-name". + (unless (tramp-get-method-parameter vec 'tramp-copy-file-name) + (tramp-set-connection-property + vec "copy-file-name" '(("%u" "@") ("%h" ":") ("%f")))) (when (string-match-p tramp-ipv6-regexp host) (setq host (format "[%s]" host))) ;; This does not work for MS Windows scp, if there are characters @@ -5573,11 +5578,10 @@ raises an error." ;; checking in scp, we use it when available. (unless (string-match-p (rx "ftp" eos) method) (setq localname (tramp-unquote-shell-quote-argument localname))) - (cond - ((tramp-get-method-parameter vec 'tramp-remote-copy-program) - localname) - ((tramp-string-empty-or-nil-p user) (format "%s:%s" host localname)) - (t (format "%s@%s:%s" user host localname))))) + (string-join + (apply #'tramp-expand-args vec 'tramp-copy-file-name + (list ?h (or host "") ?u (or user "") ?f localname)) + ""))) (defun tramp-method-out-of-band-p (vec size) "Return t if this is an out-of-band method, nil otherwise." diff --git a/lisp/tramp.el b/lisp/tramp.el index 2efee234..0e64b779 100644 --- a/lisp/tramp.el +++ b/lisp/tramp.el @@ -301,6 +301,14 @@ pair of the form (KEY VALUE). The following KEYs are defined: This specifies the list of parameters to pass to the above mentioned program, the hints for `tramp-login-args' also apply here. + * `tramp-copy-file-name' + The remote source or destination file name for out-of-band methods. + You can use the \"%u\" and \"%h\" like in `tramp-login-args'. + Additionally, \"%f\" denotes the local file name part. It will be + expanded to a string without spaces between the elements of the list. + + The default value is \\='((\"%u\" \"@\") (\"%h\" \":\") (\"%f\")) + * `tramp-copy-env' A list of environment variables and their values, which will be set when calling `tramp-copy-program'. ^ permalink raw reply related [flat|nested] 15+ messages in thread
* bug#69085: 29.2; Tramp: Extend tramp-make-copy-program-file-name via tramp-methods 2024-02-17 17:57 ` Michael Albinus via Bug reports for GNU Emacs, the Swiss army knife of text editors @ 2024-02-19 19:19 ` Sean Devlin 2024-02-20 12:04 ` Michael Albinus via Bug reports for GNU Emacs, the Swiss army knife of text editors 0 siblings, 1 reply; 15+ messages in thread From: Sean Devlin @ 2024-02-19 19:19 UTC (permalink / raw) To: Michael Albinus; +Cc: 69085 Hi Michael, > I see your point. Appended is a new patch, which addresses your > inconvenience. Could you pls apply it on top of Tramp 2.7 instead of the > other patch? The method "dockercp" allows now user names, which are > ignored when calling "docker cp ...". This works perfectly! I see no issues with this patch. Thanks for spending so much time on this, I really appreciate your help! Sean ^ permalink raw reply [flat|nested] 15+ messages in thread
* bug#69085: 29.2; Tramp: Extend tramp-make-copy-program-file-name via tramp-methods 2024-02-19 19:19 ` Sean Devlin @ 2024-02-20 12:04 ` Michael Albinus via Bug reports for GNU Emacs, the Swiss army knife of text editors 2024-02-20 16:15 ` Sean Devlin 0 siblings, 1 reply; 15+ messages in thread From: Michael Albinus via Bug reports for GNU Emacs, the Swiss army knife of text editors @ 2024-02-20 12:04 UTC (permalink / raw) To: Sean Devlin; +Cc: 69085-done Version: 30.1 Sean Devlin <spd@toadstyle.org> writes: > Hi Michael, Hi Sean, >> I see your point. Appended is a new patch, which addresses your >> inconvenience. Could you pls apply it on top of Tramp 2.7 instead of the >> other patch? The method "dockercp" allows now user names, which are >> ignored when calling "docker cp ...". > > This works perfectly! I see no issues with this patch. Thanks for the feedback. Over the weekend, I ran Tramp regression tests. There were still some minor annoyances with the new methods, but that's fixed now. I've pushed an extended version of the patch to the repositories. Will be available with Tramp 2.7.1 and Emacs 30.1. Closing the bug. > Thanks for spending so much time on this, I really appreciate your help! > > Sean Best regards, Michael. ^ permalink raw reply [flat|nested] 15+ messages in thread
* bug#69085: 29.2; Tramp: Extend tramp-make-copy-program-file-name via tramp-methods 2024-02-20 12:04 ` Michael Albinus via Bug reports for GNU Emacs, the Swiss army knife of text editors @ 2024-02-20 16:15 ` Sean Devlin 0 siblings, 0 replies; 15+ messages in thread From: Sean Devlin @ 2024-02-20 16:15 UTC (permalink / raw) To: Michael Albinus; +Cc: 69085-done Hi Michael, > Thanks for the feedback. > > Over the weekend, I ran Tramp regression tests. There were still some > minor annoyances with the new methods, but that's fixed now. > > I've pushed an extended version of the patch to the repositories. Will > be available with Tramp 2.7.1 and Emacs 30.1. > > Closing the bug. Sounds good. Thanks again for your help with this. Sean ^ permalink raw reply [flat|nested] 15+ messages in thread
end of thread, other threads:[~2024-02-20 16:15 UTC | newest] Thread overview: 15+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2024-02-12 20:08 bug#69085: 29.2; Tramp: Extend tramp-make-copy-program-file-name via tramp-methods Sean Devlin [not found] ` <handler.69085.B.17077686248331.ack@debbugs.gnu.org> 2024-02-12 20:14 ` bug#69085: Acknowledgement (29.2; Tramp: Extend tramp-make-copy-program-file-name via tramp-methods) Sean Devlin 2024-02-12 21:16 ` Michael Albinus via Bug reports for GNU Emacs, the Swiss army knife of text editors 2024-02-12 21:14 ` bug#69085: 29.2; Tramp: Extend tramp-make-copy-program-file-name via tramp-methods Michael Albinus via Bug reports for GNU Emacs, the Swiss army knife of text editors 2024-02-12 21:51 ` Sean Devlin 2024-02-13 12:18 ` Michael Albinus via Bug reports for GNU Emacs, the Swiss army knife of text editors 2024-02-13 20:18 ` Sean Devlin 2024-02-14 6:59 ` Michael Albinus via Bug reports for GNU Emacs, the Swiss army knife of text editors 2024-02-15 0:57 ` Sean Devlin 2024-02-16 14:54 ` Michael Albinus via Bug reports for GNU Emacs, the Swiss army knife of text editors 2024-02-16 17:51 ` Sean Devlin 2024-02-17 17:57 ` Michael Albinus via Bug reports for GNU Emacs, the Swiss army knife of text editors 2024-02-19 19:19 ` Sean Devlin 2024-02-20 12:04 ` Michael Albinus via Bug reports for GNU Emacs, the Swiss army knife of text editors 2024-02-20 16:15 ` Sean Devlin
Code repositories for project(s) associated with this public inbox https://git.savannah.gnu.org/cgit/emacs.git This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).