Dired command on same host

Dired command on same host

[Manuel Giraud]

Hi,

I'm happily using dired and tramp with `dired-dwim-target' set to t. So
when I'm in this frame configuration:

+-------------+-------------+
|             |             |
| some dired  | other dired |
| on host A   | on host A   |
|             |             |
|             |             |
|             |             |
|             |             |
+-------------+-------------+

I can, for example, copy some files from one directory to another. But
when I do this, tramp copy the file over ssh to my local /tmp before
copying it back to the other directory. It can be a problem when the
file is too big. Is it possible for tramp to detect that this is the
same host and just make a simple cp on host A?

Same question for two different hosts: Is it possible to make tramp do a
scp from host A to host B directly?
-- 
Manuel Giraud

Re: Dired command on same host

[Michael Albinus]

Manuel Giraud <manuel@ledu-giraud.fr> writes:

> Hi,

Hi Manuel,

> I'm happily using dired and tramp with `dired-dwim-target' set to t. So
> when I'm in this frame configuration:
>
> +-------------+-------------+
> |             |             |
> | some dired  | other dired |
> | on host A   | on host A   |
> |             |             |
> |             |             |
> |             |             |
> |             |             |
> +-------------+-------------+
>
> I can, for example, copy some files from one directory to another. But
> when I do this, tramp copy the file over ssh to my local /tmp before
> copying it back to the other directory. It can be a problem when the
> file is too big. Is it possible for tramp to detect that this is the
> same host and just make a simple cp on host A?

Tramp tries this if possible. See the function tramp-do-copy-or-rename-file,
which calls tramp-do-copy-or-rename-file-directly. This happens in two
cases:

- Both files are on the same (remote) host.

- One file is local, and the other one matches the function
  tramp-local-host-p. Likely, the crucial point here is that either the
  local or the remote user must be root.

> Same question for two different hosts: Is it possible to make tramp do a
> scp from host A to host B directly?

Not (yet). See the TODO section at the end of tramp-sh.el:

;; * Optimize out-of-band copying when both methods are scp-like (not rsync).

Best regards, Michael.

Re: Dired command on same host

[Manuel Giraud]

Michael Albinus <michael.albinus@gmx.de> writes:

[...]

> Tramp tries this if possible. See the function tramp-do-copy-or-rename-file,
> which calls tramp-do-copy-or-rename-file-directly. This happens in two
> cases:

Hi Michael,

Sorry my bad. Maybe my two dired buffers were not `tramp-equal-remote'
(BTW, I think there is a typo in the example of the docstring of
`tramp-equal-remote':

Example:

  (tramp-equal-remote \"/ssh::/etc\" \"/-:<your host name>:/home\")
                                                            ^
                                                            |_ shouldn't
                                                               it be etc here)

>> Same question for two different hosts: Is it possible to make tramp do a
>> scp from host A to host B directly?
>
> Not (yet). See the TODO section at the end of tramp-sh.el:
>
> ;; * Optimize out-of-band copying when both methods are scp-like (not
> rsync).

Ok, i'll look that up. You said "Not (yet)": I've you already started
some work on it?

I have in my init a (simple) function that does this out-of-band scp but
I think that the most tricky parts (for something general pupose) are:
    - how host A understand host B name?
    - is host B reachable from host A?
    
… this kind of things.

Best regards,
-- 
Manuel Giraud

Re: Dired command on same host

[Michael Albinus]

Manuel Giraud <manuel@ledu-giraud.fr> writes:

> Hi Michael,

Hi Manuel,

> Sorry my bad. Maybe my two dired buffers were not `tramp-equal-remote'
> (BTW, I think there is a typo in the example of the docstring of
> `tramp-equal-remote':
>
> Example:
>
>   (tramp-equal-remote \"/ssh::/etc\" \"/-:<your host name>:/home\")
>                                                             ^
>                                                             |_ shouldn't
>                                                                it be etc here)

No. tramp-equal-remote ignores the local part of the remote file
names. That's what the docstring says above the example.

>>> Same question for two different hosts: Is it possible to make tramp do a
>>> scp from host A to host B directly?
>>
>> Not (yet). See the TODO section at the end of tramp-sh.el:
>>
>> ;; * Optimize out-of-band copying when both methods are scp-like (not
>> rsync).
>
> Ok, i'll look that up. You said "Not (yet)": I've you already started
> some work on it?

No, the comment exist for years. Usually, I follow the path of least
resistance: if there is pressure on me for a feature, I'll implement :-)

> I have in my init a (simple) function that does this out-of-band scp but
> I think that the most tricky parts (for something general pupose) are:
>     - how host A understand host B name?
>     - is host B reachable from host A?

Must be checked, yes. But this isn't a problem for Tramp. It keeps shell
sessions on both host A and host B, a simple "ssh -q -o ConnectTimeout=1
B exit 2>/dev/null" on host A would tell us whether host B is reachable
(checking the exit code, and caching the result). If not, Tramp could
fall back to the current behavior.

> Best regards,

Best regards, Michael.

Re: Dired command on same host

[Tassilo Horn]

Michael Albinus <michael.albinus@gmx.de> writes:

>> I have in my init a (simple) function that does this out-of-band scp
>> but I think that the most tricky parts (for something general pupose)
>> are:
>>     - how host A understand host B name?
>>     - is host B reachable from host A?
>
> Must be checked, yes. But this isn't a problem for Tramp. It keeps
> shell sessions on both host A and host B, a simple "ssh -q -o
> ConnectTimeout=1 B exit 2>/dev/null" on host A would tell us whether
> host B is reachable

Sorry if that question just illustrates that I don't know how Tramp
works but: is it ensured that the host B on A is the same as B on my
local machine?  Like I could have different hosts defined as B in my
local /etc/hosts and the one on A.

Bye,
Tassilo

Re: Dired command on same host

[Manuel Giraud]

Tassilo Horn <tsdh@gnu.org> writes:

> Michael Albinus <michael.albinus@gmx.de> writes:
>
>>> I have in my init a (simple) function that does this out-of-band scp
>>> but I think that the most tricky parts (for something general pupose)
>>> are:
>>>     - how host A understand host B name?
>>>     - is host B reachable from host A?
>>
>> Must be checked, yes. But this isn't a problem for Tramp. It keeps
>> shell sessions on both host A and host B, a simple "ssh -q -o
>> ConnectTimeout=1 B exit 2>/dev/null" on host A would tell us whether
>> host B is reachable
>
> Sorry if that question just illustrates that I don't know how Tramp
> works but: is it ensured that the host B on A is the same as B on my
> local machine?  Like I could have different hosts defined as B in my
> local /etc/hosts and the one on A.

AFAIU, tramp relies on ssh here. So you are right that host B on A could
be something completely different (configured by means of /etc/hosts or
ssh config). In fact, this out-of-band could even do something
*wrong*. But I think it could be useful on a «reasonable» setup.
-- 
Manuel Giraud

Re: Dired command on same host

[Tomas Hlavaty]

On Wed 05 Jan 2022 at 10:35, Manuel Giraud <manuel@ledu-giraud.fr> wrote:
> AFAIU, tramp relies on ssh here.

ssh is able to identify the host using host keys:

/etc/ssh/ssh_host_ed25519_key.pub
/etc/ssh/ssh_host_rsa_key.pub

Re: Dired command on same host

[Michael Albinus]

Tomas Hlavaty <tom@logand.com> writes:

Hi Tomas,

> ssh is able to identify the host using host keys:
>
> /etc/ssh/ssh_host_ed25519_key.pub
> /etc/ssh/ssh_host_rsa_key.pub

Thanks, these can be accessed via ssh-keyscan. Will use it in Tramp.

Best regards, Michael.

Re: Dired command on same host

[Michael Albinus]

[-- Attachment #1: Type: text/plain, Size: 1061 bytes --]

Michael Albinus <michael.albinus@gmx.de> writes:

Hi everybody,

>> ssh is able to identify the host using host keys:
>>
>> /etc/ssh/ssh_host_ed25519_key.pub
>> /etc/ssh/ssh_host_rsa_key.pub
>
> Thanks, these can be accessed via ssh-keyscan. Will use it in Tramp.

I've implemented a proof-of-concept, see appended patch to
tramp-sh.el. It is towards Emacs 29.0.50, but might also apply for Emacs
28.0.90 (for testing). After loading Tramp, there is a new user option
tramp-use-scp-direct-remote-copying which must be set to non-nil.

Direct scp copying between two remote servers shall happen then. Could
people test it? Setting tramp-verbose to 6 shall show the scp command in
the debug buffer, like in my case

--8<---------------cut here---------------start------------->8---
16:41:38.755570 tramp-do-copy-or-rename-file-out-of-band (6) # scp -p -T -R -q -r gandalf:/home/albinus/Downloads/CentOS-8.4.2105-x86_64-dvd1.iso detlef:/tmp/CentOS-8.4.2105-x86_64-dvd1.iso
--8<---------------cut here---------------end--------------->8---

Best regards, Michael.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Type: text/x-patch, Size: 14789 bytes --]

*** /tmp/ediffuxUjdM	2022-01-07 16:33:41.019095060 +0100
--- /home/albinus/src/tramp/lisp/tramp-sh.el	2022-01-07 16:26:54.686931060 +0100
***************
*** 136,141 ****
--- 136,157 ----

  The string is used in `tramp-methods'.")

+ (defcustom tramp-use-scp-direct-remote-copying nil
+   "Whether to use direct copying between two remote hosts."
+   :group 'tramp
+   :version "29.1"
+   :type 'boolean)
+
+ (defvar tramp-scp-direct-remote-copying nil
+   "Which scp direct remote copying argument to use.
+
+ It is the string \"-R\" if supported by the local scp (since
+ release 8.7), otherwise the string \"\".  If it is nil, it will
+ be auto-detected by Tramp, if
+ `tramp-use-scp-direct-remote-copying' is non-nil..
+
+ The string is used in `tramp-methods'.")
+
  ;; Initialize `tramp-methods' with the supported methods.
  ;;;###tramp-autoload
  (tramp--with-startup
***************
*** 172,178 ****
                  (tramp-remote-shell-args    ("-c"))
                  (tramp-copy-program         "scp")
                  (tramp-copy-args            (("-P" "%p") ("-p" "%k")
! 					     ("%x") ("-q") ("-r") ("%c")))
                  (tramp-copy-keep-date       t)
                  (tramp-copy-recursive       t)))
   (add-to-list 'tramp-methods
--- 188,194 ----
                  (tramp-remote-shell-args    ("-c"))
                  (tramp-copy-program         "scp")
                  (tramp-copy-args            (("-P" "%p") ("-p" "%k")
! 					     ("%x") ("%y") ("-q") ("-r") ("%c")))
                  (tramp-copy-keep-date       t)
                  (tramp-copy-recursive       t)))
   (add-to-list 'tramp-methods
***************
*** 188,194 ****
                  (tramp-remote-shell-args    ("-c"))
                  (tramp-copy-program         "scp")
                  (tramp-copy-args            (("-P" "%p") ("-p" "%k")
! 				             ("%x") ("-q") ("-r") ("%c")))
                  (tramp-copy-keep-date       t)
                  (tramp-copy-recursive       t)))
   (add-to-list 'tramp-methods
--- 204,210 ----
                  (tramp-remote-shell-args    ("-c"))
                  (tramp-copy-program         "scp")
                  (tramp-copy-args            (("-P" "%p") ("-p" "%k")
! 				             ("%x") ("%y") ("-q") ("-r") ("%c")))
                  (tramp-copy-keep-date       t)
                  (tramp-copy-recursive       t)))
   (add-to-list 'tramp-methods
***************
*** 2241,2259 ****
      (op filename newname ok-if-already-exists keep-date)
    "Invoke `scp' program to copy.
  The method used must be an out-of-band method."
!   (let* ((t1 (tramp-tramp-file-p filename))
! 	 (t2 (tramp-tramp-file-p newname))
! 	 (orig-vec (tramp-dissect-file-name (if t1 filename newname)))
  	 copy-program copy-args copy-env copy-keep-date listener spec
  	 options source target remote-copy-program remote-copy-args p)

!     (with-parsed-tramp-file-name (if t1 filename newname) nil
!       (if (and t1 t2)

! 	  ;; Both are Tramp files.  We shall optimize it when the
! 	  ;; methods for FILENAME and NEWNAME are the same.
  	  (let* ((dir-flag (file-directory-p filename))
! 		 (tmpfile (tramp-compat-make-temp-file localname dir-flag)))
  	    (if dir-flag
  		(setq tmpfile
  		      (expand-file-name
--- 2257,2277 ----
      (op filename newname ok-if-already-exists keep-date)
    "Invoke `scp' program to copy.
  The method used must be an out-of-band method."
!   (let* ((v1 (and (tramp-tramp-file-p filename)
! 		  (tramp-dissect-file-name filename)))
! 	 (v2 (and (tramp-tramp-file-p newname)
! 		  (tramp-dissect-file-name newname)))
! 	 (v (or v1 v2))
  	 copy-program copy-args copy-env copy-keep-date listener spec
  	 options source target remote-copy-program remote-copy-args p)

! ;    (with-parsed-tramp-file-name (if v1 filename newname) nil
!       (if (and v1 v2 (not (tramp-scp-direct-remote-copying-p v1 v2)))

! 	  ;; Both are Tramp files.  We cannot use direct remote copying.
  	  (let* ((dir-flag (file-directory-p filename))
! 		 (tmpfile (tramp-compat-make-temp-file
! 			   (tramp-file-name-localname v1) dir-flag)))
  	    (if dir-flag
  		(setq tmpfile
  		      (expand-file-name
***************
*** 2273,2299 ****

  	;; Check which ones of source and target are Tramp files.
  	(setq source (funcall
! 		      (if (and (string-equal method "rsync")
  			       (file-directory-p filename)
  			       (not (file-exists-p newname)))
  			  #'file-name-as-directory
  			#'identity)
! 		      (if t1
! 			  (tramp-make-copy-program-file-name v)
  			(tramp-compat-file-name-unquote filename)))
! 	      target (if t2
! 			 (tramp-make-copy-program-file-name v)
  		       (tramp-compat-file-name-unquote newname)))

  	;; Check for user.  There might be an interactive setting.
! 	(setq user (or (tramp-file-name-user v)
! 		       (tramp-get-connection-property v "login-as" nil)))

  	;; Check for listener port.
  	(when (tramp-get-method-parameter v 'tramp-remote-copy-args)
  	  (setq listener (number-to-string (+ 50000 (random 10000))))
  	  (while
! 	      (zerop (tramp-call-process v "nc" nil nil nil "-z" host listener))
  	    (setq listener (number-to-string (+ 50000 (random 10000))))))

  	;; Compose copy command.
--- 2291,2318 ----

  	;; Check which ones of source and target are Tramp files.
  	(setq source (funcall
! 		      (if (and (string-equal (tramp-file-name-method v) "rsync")
  			       (file-directory-p filename)
  			       (not (file-exists-p newname)))
  			  #'file-name-as-directory
  			#'identity)
! 		      (if v1
! 			  (tramp-make-copy-program-file-name v1)
  			(tramp-compat-file-name-unquote filename)))
! 	      target (if v2
! 			 (tramp-make-copy-program-file-name v2)
  		       (tramp-compat-file-name-unquote newname)))

  	;; Check for user.  There might be an interactive setting.
! ;	(setq user (or (tramp-file-name-user v)
! ;		       (tramp-get-connection-property v "login-as" nil)))

  	;; Check for listener port.
  	(when (tramp-get-method-parameter v 'tramp-remote-copy-args)
  	  (setq listener (number-to-string (+ 50000 (random 10000))))
  	  (while
! 	      (zerop (tramp-call-process
! 		      v "nc" nil nil nil "-z" (tramp-file-name-host v) listener))
  	    (setq listener (number-to-string (+ 50000 (random 10000))))))

  	;; Compose copy command.
***************
*** 2304,2313 ****
  		?t (tramp-get-connection-property
  		    (tramp-get-connection-process v) "temp-file" "")))
  	      spec (list
! 		    ?h (or host "") ?u (or user "") ?p (or port "")
  		    ?r listener ?c options ?k (if keep-date " " "")
                      ?n (concat "2>" (tramp-get-remote-null-device v))
! 		    ?x (tramp-scp-strict-file-name-checking v))
  	      copy-program (tramp-get-method-parameter v 'tramp-copy-program)
  	      copy-keep-date (tramp-get-method-parameter
  			      v 'tramp-copy-keep-date)
--- 2323,2335 ----
  		?t (tramp-get-connection-property
  		    (tramp-get-connection-process v) "temp-file" "")))
  	      spec (list
! 		    ?h (or (tramp-file-name-host v) "")
! 		    ?u (or (tramp-file-name-user v) "")
! 		    ?p (or (tramp-file-name-port v) "")
  		    ?r listener ?c options ?k (if keep-date " " "")
                      ?n (concat "2>" (tramp-get-remote-null-device v))
! 		    ?x (tramp-scp-strict-file-name-checking v)
! 		    ?y (tramp-scp-direct-remote-copying v))
  	      copy-program (tramp-get-method-parameter v 'tramp-copy-program)
  	      copy-keep-date (tramp-get-method-parameter
  			      v 'tramp-copy-keep-date)
***************
*** 2350,2356 ****
  		 #'identity
  		 (append
  		  (list remote-copy-program) remote-copy-args
! 		  (list (if t1 (concat "<" source) (concat ">" target)) "&"))
  		 " "))
  	  (tramp-send-command v remote-copy-program)
  	  (with-timeout
--- 2372,2378 ----
  		 #'identity
  		 (append
  		  (list remote-copy-program) remote-copy-args
! 		  (list (if v1 (concat "<" source) (concat ">" target)) "&"))
  		 " "))
  	  (tramp-send-command v remote-copy-program)
  	  (with-timeout
***************
*** 2367,2373 ****
  	  (unwind-protect
  	      ;; The default directory must be remote.
  	      (let ((default-directory
! 		      (file-name-directory (if t1 filename newname)))
  		    (process-environment (copy-sequence process-environment)))
  		;; Set the transfer process properties.
  		(tramp-set-connection-property
--- 2389,2395 ----
  	  (unwind-protect
  	      ;; The default directory must be remote.
  	      (let ((default-directory
! 		     (file-name-directory (if v1 filename newname)))
  		    (process-environment (copy-sequence process-environment)))
  		;; Set the transfer process properties.
  		(tramp-set-connection-property
***************
*** 2376,2382 ****
  		 v "process-buffer" (current-buffer))
  		(when copy-env
  		  (tramp-message
! 		   orig-vec 6 "%s=\"%s\""
  		   (car copy-env) (string-join (cdr copy-env) " "))
  		  (setenv (car copy-env) (string-join (cdr copy-env) " ")))
  		(setq
--- 2398,2404 ----
  		 v "process-buffer" (current-buffer))
  		(when copy-env
  		  (tramp-message
! 		   v 6 "%s=\"%s\""
  		   (car copy-env) (string-join (cdr copy-env) " "))
  		  (setenv (car copy-env) (string-join (cdr copy-env) " ")))
  		(setq
***************
*** 2384,2403 ****
  		 (append
  		  copy-args
  		  (if remote-copy-program
! 		      (list (if t1 (concat ">" target) (concat "<" source)))
  		    (list source target)))
  		 ;; Use an asynchronous process.  By this, password
  		 ;; can be handled.  We don't set a timeout, because
  		 ;; the copying of large files can last longer than 60
  		 ;; secs.
! 		 p (let ((default-directory tramp-compat-temporary-file-directory))
  		     (apply
  		      #'start-process
  		      (tramp-get-connection-name v)
  		      (tramp-get-connection-buffer v)
  		      copy-program copy-args)))
! 		(tramp-message orig-vec 6 "%s" (string-join (process-command p) " "))
! 		(process-put p 'vector orig-vec)
  		(process-put p 'adjust-window-size-function #'ignore)
  		(set-process-query-on-exit-flag p nil)

--- 2406,2426 ----
  		 (append
  		  copy-args
  		  (if remote-copy-program
! 		      (list (if v1 (concat ">" target) (concat "<" source)))
  		    (list source target)))
  		 ;; Use an asynchronous process.  By this, password
  		 ;; can be handled.  We don't set a timeout, because
  		 ;; the copying of large files can last longer than 60
  		 ;; secs.
! 		 p (let ((default-directory
! 			  tramp-compat-temporary-file-directory))
  		     (apply
  		      #'start-process
  		      (tramp-get-connection-name v)
  		      (tramp-get-connection-buffer v)
  		      copy-program copy-args)))
! 		(tramp-message v 6 "%s" (string-join (process-command p) " "))
! 		(process-put p 'vector v)
  		(process-put p 'adjust-window-size-function #'ignore)
  		(set-process-query-on-exit-flag p nil)

***************
*** 2434,2440 ****
        (unless (eq op 'copy)
  	(if (file-regular-p filename)
  	    (delete-file filename)
! 	  (delete-directory filename 'recursive))))))

  (defun tramp-sh-handle-make-directory (dir &optional parents)
    "Like `make-directory' for Tramp files."
--- 2457,2463 ----
        (unless (eq op 'copy)
  	(if (file-regular-p filename)
  	    (delete-file filename)
! 	  (delete-directory filename 'recursive)))));)

  (defun tramp-sh-handle-make-directory (dir &optional parents)
    "Like `make-directory' for Tramp files."
***************
*** 4824,4829 ****
--- 4847,4912 ----
  		  (setq tramp-scp-strict-file-name-checking "-T")))))))
        tramp-scp-strict-file-name-checking)))

+ (defun tramp-scp-direct-remote-copying-p (vec1 vec2)
+   "Check, whether direct remote copying between VEC1 and VEC2 is possible."
+   (and tramp-use-scp-direct-remote-copying
+        (assoc "%y" (tramp-get-method-parameter vec1 'tramp-copy-args))
+        (assoc "%y" (tramp-get-method-parameter vec2 'tramp-copy-args))
+        (with-tramp-connection-property
+ 	   (tramp-get-process vec1)
+ 	   (concat "direct-remote-copying-"
+ 		   (tramp-make-tramp-file-name vec2 'local 'hop))
+ 	 (let ((command
+ 		(if (tramp-file-name-port vec2)
+ 		    `("ssh-keyscan" ,(tramp-file-name-host vec2)
+ 		      "-p" ,(tramp-file-name-port vec2))
+ 		  `("ssh-keyscan" ,(tramp-file-name-host vec2))))
+ 	       found string)
+ 	   (with-temp-buffer
+ 	     ;; Check hostkey of VEC2, seen from VEC1.
+ 	     (tramp-send-command vec1 (mapconcat #'identity command " "))
+ 	     ;; Check hostkey of VEC2, seen locally.
+ 	     (apply
+ 	      #'tramp-call-process vec1 (car command) nil t nil (cdr command))
+ 	     (goto-char (point-min))
+ 	     (while (and (not found) (not (eobp)))
+ 	       (setq string (buffer-substring
+ 			     (line-beginning-position) (line-end-position))
+ 		     found (and (not (string-match-p "^#" string))
+ 				(with-current-buffer (tramp-get-buffer vec1)
+ 				  (goto-char (point-min))
+ 				  (search-forward string nil 'noerror))))
+ 	       (forward-line))
+ 	     ;; Result.
+ 	     found)))))
+
+ (defun tramp-scp-direct-remote-copying (vec)
+   "Return the direct remote copying argument of the local scp."
+   (cond
+    ;; No options to be computed.
+    ((null (assoc "%y" (tramp-get-method-parameter vec 'tramp-copy-args)))
+     "")
+
+    ;; There is already a value to be used.
+    ((stringp tramp-scp-direct-remote-copying)
+     tramp-scp-direct-remote-copying)
+
+    ;; Determine the options.
+    (t (setq tramp-scp-direct-remote-copying "")
+       (let ((case-fold-search t))
+ 	(ignore-errors
+ 	  (when (executable-find "scp")
+ 	    (with-tramp-progress-reporter
+ 		vec 4 "Computing direct remote copying argument"
+ 	      (with-temp-buffer
+ 		(tramp-call-process vec "scp" nil t nil "-R")
+ 		(goto-char (point-min))
+ 		(unless
+                     (search-forward-regexp
+                      "\\(illegal\\|unknown\\) option -- R" nil t)
+ 		  (setq tramp-scp-strict-file-name-checking "-R")))))))
+       tramp-scp-strict-file-name-checking)))
+
  (defun tramp-timeout-session (vec)
    "Close the connection VEC after a session timeout.
  If there is just some editing, retry it after 5 seconds."
***************
*** 5977,5985 ****
  ;;
  ;; * Use lsh instead of ssh.  (Alfred M. Szmidt)
  ;;
- ;; * Optimize out-of-band copying when both methods are scp-like (not
- ;;   rsync).
- ;;
  ;; * Keep a second connection open for out-of-band methods like scp or
  ;;   rsync.
  ;;
--- 6060,6065 ----

Re: Dired command on same host

[Manuel Giraud]

Michael Albinus <michael.albinus@gmx.de> writes:

> Michael Albinus <michael.albinus@gmx.de> writes:
>
> Hi everybody,
>
>>> ssh is able to identify the host using host keys:
>>>
>>> /etc/ssh/ssh_host_ed25519_key.pub
>>> /etc/ssh/ssh_host_rsa_key.pub
>>
>> Thanks, these can be accessed via ssh-keyscan. Will use it in Tramp.
>
> I've implemented a proof-of-concept, see appended patch to
> tramp-sh.el. It is towards Emacs 29.0.50, but might also apply for Emacs
> 28.0.90 (for testing). After loading Tramp, there is a new user option
> tramp-use-scp-direct-remote-copying which must be set to non-nil.
>
> Direct scp copying between two remote servers shall happen then. Could
> people test it? Setting tramp-verbose to 6 shall show the scp command in
> the debug buffer, like in my case

Hi Michael,

I've tested it with tramp-use-scp-direct-remote-copying to t and
tramp-verbose to 6 but it does not work for me: it still use the method
via a file in /tmp (which fills up my /tmp with a big file). Sorry,
there is not much more to say.

In the meantime, I was trying to implement a new "scp -3" method into
tramp but I'm slow to catchup with the tramp codebase :)
-- 
Manuel Giraud

Re: Dired command on same host

[Michael Albinus]

Manuel Giraud <manuel@ledu-giraud.fr> writes:

> Hi Michael,

Hi Manuel,

> I've tested it with tramp-use-scp-direct-remote-copying to t and
> tramp-verbose to 6 but it does not work for me: it still use the method
> via a file in /tmp (which fills up my /tmp with a big file). Sorry,
> there is not much more to say.

Please set tramp-verbose to 10, and rerun your test. Apply "M-x
tramp-cleanup-all-connections" first. There will be two Tramp debug
buffers; I'd like to see both.

> In the meantime, I was trying to implement a new "scp -3" method into
> tramp but I'm slow to catchup with the tramp codebase :)

What will it be good for? It copies also via your local tmp directory,
doesn't it?

Best regards, Michael.

Re: Dired command on same host

[Manuel Giraud]

Michael Albinus <michael.albinus@gmx.de> writes:

> Manuel Giraud <manuel@ledu-giraud.fr> writes:
>
>> Hi Michael,
>
> Hi Manuel,
>
>> I've tested it with tramp-use-scp-direct-remote-copying to t and
>> tramp-verbose to 6 but it does not work for me: it still use the method
>> via a file in /tmp (which fills up my /tmp with a big file). Sorry,
>> there is not much more to say.
>
> Please set tramp-verbose to 10, and rerun your test. Apply "M-x
> tramp-cleanup-all-connections" first. There will be two Tramp debug
> buffers; I'd like to see both.

Ok. I'll try this and report back.

>> In the meantime, I was trying to implement a new "scp -3" method into
>> tramp but I'm slow to catchup with the tramp codebase :)
>
> What will it be good for? It copies also via your local tmp directory,
> doesn't it?

My intent was to use scp default "-3" behaviour when source and
destination are remote hosts. Why? Because this behaviour works without
filling up my local (small) /tmp directory whereas tramp does. I don't
know how it works but it seems that a "scp hostA:file hostB:" does not
use /tmp.
-- 
Manuel Giraud

Re: Dired command on same host

[Michael Albinus]

Manuel Giraud <manuel@ledu-giraud.fr> writes:

Hi Manuel,

>> What will it be good for? It copies also via your local tmp directory,
>> doesn't it?
>
> My intent was to use scp default "-3" behaviour when source and
> destination are remote hosts. Why? Because this behaviour works without
> filling up my local (small) /tmp directory whereas tramp does. I don't
> know how it works but it seems that a "scp hostA:file hostB:" does not
> use /tmp.

Interesting. Honestly, I have no idea where the scp program buffers the
local temp data. Perhaps it is better to use this, indeed.

Best regards, Michael.

Re: Dired command on same host

[Manuel Giraud]

Michael Albinus <michael.albinus@gmx.de> writes:

> Manuel Giraud <manuel@ledu-giraud.fr> writes:
>
> Hi Manuel,
>
>>> What will it be good for? It copies also via your local tmp directory,
>>> doesn't it?
>>
>> My intent was to use scp default "-3" behaviour when source and
>> destination are remote hosts. Why? Because this behaviour works without
>> filling up my local (small) /tmp directory whereas tramp does. I don't
>> know how it works but it seems that a "scp hostA:file hostB:" does not
>> use /tmp.
>
> Interesting. Honestly, I have no idea where the scp program buffers the
> local temp data. Perhaps it is better to use this, indeed.

Ok, I might have been too bleeding edge on ssh here.

If, I understand the following code correctly:
https://github.com/openbsd/src/blob/33e609bc6fba341792696aa7159aa47556483f0e/usr.bin/ssh/scp.c#L994

It seems that if mode is SFTP, the copy is done from one SFTP connection
to another directly. But, the default to SFTP mode was activated quite
recently:
https://github.com/openbsd/src/commit/88549df51ff400f3bab7dd1ee92d4f6fd526b715#diff-b94bba0ea2aa22d09c30d4c1e41153505d94a5d9cb61fad6823a950c2cf0bb22

And according this: https://www.openssh.com/releasenotes.html, the
default is not SFTP yet in the last release of openssh. So maybe it
could be a method worth having for tramp in the future ;-)

Does a "scp hostA:fileA hostB:" creates a /tmp file on you side?
-- 
Manuel Giraud

Re: Dired command on same host

[Michael Albinus]

Manuel Giraud <manuel@ledu-giraud.fr> writes:

Hi Manuel,

> And according this: https://www.openssh.com/releasenotes.html, the
> default is not SFTP yet in the last release of openssh. So maybe it
> could be a method worth having for tramp in the future ;-)

"sftp" was a supported Tramp method in tramp-sh.el in the past. However,
it also needs an ordinary "ssh" connection. This doesn't work, if just
the sftp subsystem is enabled on the remote side, but ssh is disabled.

Therefore, Tramp's "sftp" support was moved to tramp-gvfs.el, where it
seems to work properly. However, tramp-gvfs.el has its own limitations,
it requires GVFS and D-Bus (roughly, a GNU/Linux system).

Best regards, Michael.

Re: Dired command on same host

[Manuel Giraud]

Michael Albinus <michael.albinus@gmx.de> writes:

> Manuel Giraud <manuel@ledu-giraud.fr> writes:
>
> Hi Manuel,
>
>> And according this: https://www.openssh.com/releasenotes.html, the
>> default is not SFTP yet in the last release of openssh. So maybe it
>> could be a method worth having for tramp in the future ;-)
>
> "sftp" was a supported Tramp method in tramp-sh.el in the past. However,
> it also needs an ordinary "ssh" connection. This doesn't work, if just
> the sftp subsystem is enabled on the remote side, but ssh is disabled.
>
> Therefore, Tramp's "sftp" support was moved to tramp-gvfs.el, where it
> seems to work properly. However, tramp-gvfs.el has its own limitations,
> it requires GVFS and D-Bus (roughly, a GNU/Linux system).

Hi Michael,

Ok. I was talking about the fact that, in future release of ssh, scp
will rely on the sftp protocol by default.
-- 
Manuel Giraud

Re: Dired command on same host

[Michael Albinus]

Manuel Giraud <manuel@ledu-giraud.fr> writes:

> Hi Michael,

Hi Manuel,

> Ok. I was talking about the fact that, in future release of ssh, scp
> will rely on the sftp protocol by default.

That won't be a problem, I guess the scp command and its arguments won't
change by this protocol change. We'll see.

Best regards, Michael.

Re: Dired command on same host

[Yuri Khan]

On Mon, 10 Jan 2022 at 22:02, Michael Albinus <michael.albinus@gmx.de> wrote:

> Interesting. Honestly, I have no idea where the scp program buffers the
> local temp data. Perhaps it is better to use this, indeed.

Why would it have to buffer it anywhere? I can imagine it allocating a
megabyte (or eight) of RAM, opening two network connections, then
reading from one and writing to the other.

Re: Dired command on same host

[Michael Albinus]

Yuri Khan <yuri.v.khan@gmail.com> writes:

>> Interesting. Honestly, I have no idea where the scp program buffers the
>> local temp data. Perhaps it is better to use this, indeed.
>
> Why would it have to buffer it anywhere? I can imagine it allocating a
> megabyte (or eight) of RAM, opening two network connections, then
> reading from one and writing to the other.

Perhaps. And perhaps, Tramp shall check and apply "-3" if
possible. Since OpenSSH 8.7, there seem to be more restrictions wrt
authentication. Quoting the man page:

--8<---------------cut here---------------start------------->8---
    -3      Copies between two remote hosts are transferred through the lo‐
             cal host.  Without this option the data is copied directly be‐
             tween the two remote hosts.  Note that, when using the legacy
             SCP protocol (the default), this option selects batch mode for
             the second host as scp cannot ask for passwords or passphrases
             for both hosts.  This mode is the default.
--8<---------------cut here---------------end--------------->8---

Re: Dired command on same host

[Michael Albinus]

Tassilo Horn <tsdh@gnu.org> writes:

Hi Tassilo,

>>> I have in my init a (simple) function that does this out-of-band scp
>>> but I think that the most tricky parts (for something general pupose)
>>> are:
>>>     - how host A understand host B name?
>>>     - is host B reachable from host A?
>>
>> Must be checked, yes. But this isn't a problem for Tramp. It keeps
>> shell sessions on both host A and host B, a simple "ssh -q -o
>> ConnectTimeout=1 B exit 2>/dev/null" on host A would tell us whether
>> host B is reachable
>
> Sorry if that question just illustrates that I don't know how Tramp
> works but: is it ensured that the host B on A is the same as B on my
> local machine?  Like I could have different hosts defined as B in my
> local /etc/hosts and the one on A.

Indeed, this is not guaranteed. So it doesn't make sense to run this
check on host B.

I've just applied some tests. If I run "scp hostA:/path/to/file
hostB:/another/path", scp assumes the "-3" argument by default. That
means, it copies the file first from hostA to my local host, and copies
then from my local host to hostB - like Tramp does. The difference is,
that it keeps the local transfer file in /tmp.

In order to force scp to copy directly, it needs the "-R" argument for
scp. Another restriction is, that connection from hostA to hostB must be
possible in batch mode, w/o interactive authentication.

This tells us, that we shall support direct copy between two remote
machines in Tramp as opt-in, via a user option to be configured. And the
user would be responsible that the restrictions (hostB is the same
everywhere, connection from hostA to hostB possible in batch mode) are
kept.

> Bye,
> Tassilo

Best regards, Michael.

Re: Dired command on same host

[Manuel Giraud]

Michael Albinus <michael.albinus@gmx.de> writes:

[...]

> I've just applied some tests. If I run "scp hostA:/path/to/file
> hostB:/another/path", scp assumes the "-3" argument by default. That
> means, it copies the file first from hostA to my local host, and copies
> then from my local host to hostB - like Tramp does. The difference is,
> that it keeps the local transfer file in /tmp.

I've made the same test : scp hostA:/path/to/file hostB:/another/path
and with a relatively large file (1.5 GB) and it works as expected
without filling /tmp. I'm using scp on fairly recent openbsd machine.

OTOH, the out-of-band method of tramp fills up my /tmp (which is only
256 MB). So maybe, having tramp out-of-band trying to "scp hostA:old
hostB:new" before relying on other method could be enough without going
to the trouble of out-of-band from hostA to hostB directly?
-- 
Manuel Giraud

Re: Dired command on same host

[Michael Albinus]

Manuel Giraud <manuel@ledu-giraud.fr> writes:

Hi Manuel,

>> I've just applied some tests. If I run "scp hostA:/path/to/file
>> hostB:/another/path", scp assumes the "-3" argument by default. That
>> means, it copies the file first from hostA to my local host, and copies
>> then from my local host to hostB - like Tramp does. The difference is,
>> that it keeps the local transfer file in /tmp.
>
> I've made the same test : scp hostA:/path/to/file hostB:/another/path
> and with a relatively large file (1.5 GB) and it works as expected
> without filling /tmp. I'm using scp on fairly recent openbsd machine.
>
> OTOH, the out-of-band method of tramp fills up my /tmp (which is only
> 256 MB). So maybe, having tramp out-of-band trying to "scp hostA:old
> hostB:new" before relying on other method could be enough without going
> to the trouble of out-of-band from hostA to hostB directly?

Likely, you're using an older scp version. This behavior has changed in
OpenSSH 8.7, see <https://www.openssh.com/txt/release-8.7>:

--8<---------------cut here---------------start------------->8---
 * scp(1): this release changes the behaviour of remote to remote
   copies (e.g. "scp host-a:/path host-b:") to transfer through the
   local host by default. This was previously available via the -3
   flag. This mode avoids the need to expose credentials on the
   origin hop, avoids triplicate interpretation of filenames by the
   shell (by the local system, the copy origin and the destination)
   and, in conjunction with the SFTP support for scp(1) mentioned
   below, allows use of all authentication methods to the remote
   hosts (previously, only non-interactive methods could be used).
   A -R flag has been added to select the old behaviour.
--8<---------------cut here---------------end--------------->8---

So Tramp needs even more checks in order to detect what scp supports.

Best regards, Michael.

Re: Dired command on same host

[Manuel Giraud]

Michael Albinus <michael.albinus@gmx.de> writes:

> Likely, you're using an older scp version. This behavior has changed in
> OpenSSH 8.7, see <https://www.openssh.com/txt/release-8.7>:
>
>  * scp(1): this release changes the behaviour of remote to remote
>    copies (e.g. "scp host-a:/path host-b:") to transfer through the
>    local host by default. This was previously available via the -3
>    flag. This mode avoids the need to expose credentials on the
>    origin hop, avoids triplicate interpretation of filenames by the
>    shell (by the local system, the copy origin and the destination)
>    and, in conjunction with the SFTP support for scp(1) mentioned
>    below, allows use of all authentication methods to the remote
>    hosts (previously, only non-interactive methods could be used).
>    A -R flag has been added to select the old behaviour.
>
> So Tramp needs even more checks in order to detect what scp supports.

No I'm using OpenSSH_8.8 (ssh -V) and I think the implicict '-3' is
still there. It is just that it does not seem to use a temporary file in
/tmp.
-- 
Manuel Giraud

Re: Dired command on same host

[Manuel Giraud]

Michael Albinus <michael.albinus@gmx.de> writes:

> Manuel Giraud <manuel@ledu-giraud.fr> writes:
>
>> Hi Michael,
>
> Hi Manuel,
>
>> Sorry my bad. Maybe my two dired buffers were not `tramp-equal-remote'
>> (BTW, I think there is a typo in the example of the docstring of
>> `tramp-equal-remote':
>>
>> Example:
>>
>>   (tramp-equal-remote \"/ssh::/etc\" \"/-:<your host name>:/home\")
>>                                                             ^
>>                                                             |_ shouldn't
>>                                                                it be etc here)
>
> No. tramp-equal-remote ignores the local part of the remote file
> names. That's what the docstring says above the example.

Sorry again. I should read more carefully :-|

[...]

> Must be checked, yes. But this isn't a problem for Tramp. It keeps shell
> sessions on both host A and host B, a simple "ssh -q -o ConnectTimeout=1
> B exit 2>/dev/null" on host A would tell us whether host B is reachable
> (checking the exit code, and caching the result). If not, Tramp could
> fall back to the current behavior.

To adress Tassilo remark, maybe it should make a hash of «some known»
file too to ensure that this is the correct host B. Or maybe this is too
much.
-- 
Manuel Giraud

Re: Dired command on same host

[Michael Albinus]

Manuel Giraud <manuel@ledu-giraud.fr> writes:

Hi Manual,

> To adress Tassilo remark, maybe it should make a hash of «some known»
> file too to ensure that this is the correct host B. Or maybe this is too
> much.

Well, a better check would be whether a given host name resolves to the
same IP address, taking into account also ssh's config via the "Host"
option. Don't know how hard this is.

And even this isn't bullet-proof. Due to network configuration, hostB
could resolve to different addresses on my local machine and on hostA,
although it is still the same machine.

And no, I don't want to start comparision of different machine uuids or
so.

Best regards, Michael.

Re: Dired command on same host

[Yuri Khan]

On Wed, 5 Jan 2022 at 17:41, Michael Albinus <michael.albinus@gmx.de> wrote:

> And even this isn't bullet-proof. Due to network configuration, hostB
> could resolve to different addresses on my local machine and on hostA,
> although it is still the same machine.

Additionally, the reverse is also possible: 192.168.0.1 when accessed
from local machine and 192.168.0.1 when accessed from hostA could be
two completely different hostsB.

A somewhat reliable way to detect if two jump paths resolve to the
same host would be:

1. Through one path, perform some (small) file system modification.
Like, create a uniquely named file at a known location, and write some
unique content.
2. Through the other path, observe that modification. If successful,
assume identity.

(Various TLS certificate issuers do this to check that the user who
requests a certificate for a domain name actually controls the host
that the domain name resolves to.)

Re: Dired command on same host

[Michael Albinus]

Yuri Khan <yuri.v.khan@gmail.com> writes:

Hi Yuri,

>> And even this isn't bullet-proof. Due to network configuration, hostB
>> could resolve to different addresses on my local machine and on hostA,
>> although it is still the same machine.
>
> Additionally, the reverse is also possible: 192.168.0.1 when accessed
> from local machine and 192.168.0.1 when accessed from hostA could be
> two completely different hostsB.
>
> A somewhat reliable way to detect if two jump paths resolve to the
> same host would be:
>
> 1. Through one path, perform some (small) file system modification.
> Like, create a uniquely named file at a known location, and write some
> unique content.
> 2. Through the other path, observe that modification. If successful,
> assume identity.
>
> (Various TLS certificate issuers do this to check that the user who
> requests a certificate for a domain name actually controls the host
> that the domain name resolves to.)

Might be feasible, thanks!

Best regards, Michael.

Re: Dired command on same host

[Tassilo Horn]

Michael Albinus <michael.albinus@gmx.de> writes:

>> To adress Tassilo remark, maybe it should make a hash of «some known»
>> file too to ensure that this is the correct host B. Or maybe this is
>> too much.
>
> Well, a better check would be whether a given host name resolves to
> the same IP address, taking into account also ssh's config via the
> "Host" option. Don't know how hard this is.
>
> And even this isn't bullet-proof. Due to network configuration, hostB
> could resolve to different addresses on my local machine and on hostA,
> although it is still the same machine.

That's exactly a common usage pattern for me.  I frequently connect to
customer machines that some network VPN magic makes accessible as if
they were in our local network, so hostA and hostB are 192.168.233.1 and
192.168.233.2 from my POV although in that customer's network their IPs
are different.

Bye,
Tassilo