Re: SCPC Detection Program

Re: SCPC Detection Program

[Michael Albinus]

Noah Lavine <noah549@gmail.com> writes:

> Dear Tramp Developers,

Hi,

> I recently started using Tramp and learned about the different access
> methods. As I understand it, the `scpc` method is superior to the
> `ssh` method, but can't be set as default because of old versions of
> ssh that don't support the ControlMaster argument.
>
> It seems to me that it would be better if Tramp could automatically
> detect whether the version of ssh on a computer supported the
> ControlMaster argument and use scpc if it were possible. Therefore, I
> started working on a program to detect this based on the SSH version
> string. The following program correctly detects the presence of an SSH
> version greater than 4.0 on my computer:

I am not too happy in detecting features by checking version numbers of
programs. This will work for a while, and this will generate problems
later on, likely.

Maybe it is sufficient to add a hint to the Tramp manual, that the
following setting in the ssh config file is useful, when applicable by
the underlying ssh version:

Host *
	ControlMaster auto
	ControlPath   /tmp/%r@%h:%p

> Noah Lavine

Best regards, Michael.

Re: SCPC Detection Program

[Noah Lavine]

Michael,

I see your point about version numbers, but I don't think it's as good
to add a hint to the Tramp manual, because users still have to look
through it to see that it's there and then configure Tramp. In
general, I think it's safe to assume that they will read the manual as
little as possible in order to accomplish whatever task they're trying
to accomplish.

For instance, I used Tramp for a while before I realized that there
was a way to make it stop asking for my password all the time. This is
basically my effort to make sure that this doesn't happen to other
users in the future.

Is there a commonly-used way to discover the capabilities of a system
other than version numbers? Talking to the system package manager
would also work for most (although not all) cases, and I could look
into writing a connection to PackageKit if that seemed like it could
work.

It would also be really cool if the ssh program had a way to tell us
what command-line arguments it accepted, but I don't see any way to do
that short of asking it to open up a test connection and seeing if it
gave an error message. That might actually be a reasonable way to do
this, if you defer the checking until the first use of ssh - first try
opening a connection with ControlMaster, and if that fails, try again
without it.

Noah

On Sun, Apr 11, 2010 at 12:19 PM, Michael Albinus
<michael.albinus@gmx.de> wrote:
> Noah Lavine <noah549@gmail.com> writes:
>
>> Dear Tramp Developers,
>
> Hi,
>
>> I recently started using Tramp and learned about the different access
>> methods. As I understand it, the `scpc` method is superior to the
>> `ssh` method, but can't be set as default because of old versions of
>> ssh that don't support the ControlMaster argument.
>>
>> It seems to me that it would be better if Tramp could automatically
>> detect whether the version of ssh on a computer supported the
>> ControlMaster argument and use scpc if it were possible. Therefore, I
>> started working on a program to detect this based on the SSH version
>> string. The following program correctly detects the presence of an SSH
>> version greater than 4.0 on my computer:
>
> I am not too happy in detecting features by checking version numbers of
> programs. This will work for a while, and this will generate problems
> later on, likely.
>
> Maybe it is sufficient to add a hint to the Tramp manual, that the
> following setting in the ssh config file is useful, when applicable by
> the underlying ssh version:
>
> Host *
>        ControlMaster auto
>        ControlPath   /tmp/%r@%h:%p
>
>> Noah Lavine
>
> Best regards, Michael.
>

Re: SCPC Detection Program

[Michael Albinus]

Noah Lavine <noah549@gmail.com> writes:

> Michael,

Hi Noah,

> Is there a commonly-used way to discover the capabilities of a system
> other than version numbers? Talking to the system package manager
> would also work for most (although not all) cases, and I could look
> into writing a connection to PackageKit if that seemed like it could
> work.
>
> It would also be really cool if the ssh program had a way to tell us
> what command-line arguments it accepted, but I don't see any way to do
> that short of asking it to open up a test connection and seeing if it
> gave an error message. That might actually be a reasonable way to do
> this, if you defer the checking until the first use of ssh - first try
> opening a connection with ControlMaster, and if that fails, try again
> without it.

Maybe "ssh --help", and checking for the "-M" option, might be sufficient.
Trying a "test connection" is not mature, because you don't know which
test host to connect to. "localhost" does not need to run the sshd daemon.

> Noah

Best regards, Michael.

Re: SCPC Detection Program

[Noah Lavine]

Hi Michael,

I checked with my version of ssh, and we would indeed be able to
detect this from its help version string. However, it seems like this
could have similar fragility issues as the version number idea, if the
help string changed format some time. You could solve this by only
trying scpc on help strings that are known to be good, but you could
also have a version number whitelist that would serve the same
purpose. It seems like both of these would work and would be quite
similar - is there a reason to think that one would be better than the
other?

As for the test connection, maybe a change would help. It's probably
not good to try connecting to a known-good host, but what about trying
it with the host that Tramp needs to connect to anyway? You could try
it first with the -ControlMaster option, and if ssh gave an error,
then try again without it and fall back on the other method. This
wouldn't create more ssh connections than necessary. The problem I see
is that it might require a more substantial change to the Tramp
codebase than the other methods, but I don't know, because I haven't
looked around enough to see. Do you think this method would be useful
enough to make it worth looking through the Tramp code more and maybe
making bigger changes?

Noah

On Mon, Apr 12, 2010 at 3:18 PM, Michael Albinus <michael.albinus@gmx.de> wrote:
> Noah Lavine <noah549@gmail.com> writes:
>
>> Michael,
>
> Hi Noah,
>
>> Is there a commonly-used way to discover the capabilities of a system
>> other than version numbers? Talking to the system package manager
>> would also work for most (although not all) cases, and I could look
>> into writing a connection to PackageKit if that seemed like it could
>> work.
>>
>> It would also be really cool if the ssh program had a way to tell us
>> what command-line arguments it accepted, but I don't see any way to do
>> that short of asking it to open up a test connection and seeing if it
>> gave an error message. That might actually be a reasonable way to do
>> this, if you defer the checking until the first use of ssh - first try
>> opening a connection with ControlMaster, and if that fails, try again
>> without it.
>
> Maybe "ssh --help", and checking for the "-M" option, might be sufficient.
> Trying a "test connection" is not mature, because you don't know which
> test host to connect to. "localhost" does not need to run the sshd daemon.
>
>> Noah
>
> Best regards, Michael.
>

Re: SCPC Detection Program

[Michael Albinus]

Noah Lavine <noah549@gmail.com> writes:

> Hi Michael,

Hi Noah,

> I checked with my version of ssh, and we would indeed be able to
> detect this from its help version string. However, it seems like this
> could have similar fragility issues as the version number idea, if the
> help string changed format some time. You could solve this by only
> trying scpc on help strings that are known to be good, but you could
> also have a version number whitelist that would serve the same
> purpose. It seems like both of these would work and would be quite
> similar - is there a reason to think that one would be better than the
> other?

I have no bullet-proof reason. But your version test depends on
underlying OpenSSH, and its version numbering scheme. What if other ssh
implementations will support this option as well? Therefore, the version
check seems to be more fragile to me than the "-M" option. Again, it is
just a feeling.

Another test could be the following:

$ ssh -o ControlMaster=auto wrong-host
ssh: Could not resolve hostname wrong-host: Name or service not known

This would indicate, that the option is supported. Unsupported options
return the following error message:

$ ssh -o ControlMasterr=auto wrong-host
command-line: line 0: Bad configuration option: ControlMasterr

> As for the test connection, maybe a change would help. It's probably
> not good to try connecting to a known-good host, but what about trying
> it with the host that Tramp needs to connect to anyway? You could try
> it first with the -ControlMaster option, and if ssh gave an error,
> then try again without it and fall back on the other method. This
> wouldn't create more ssh connections than necessary. The problem I see
> is that it might require a more substantial change to the Tramp
> codebase than the other methods, but I don't know, because I haven't
> looked around enough to see. Do you think this method would be useful
> enough to make it worth looking through the Tramp code more and maybe
> making bigger changes?

Yes, this would require deeper changes in Tramp's workflow, which I
would like to avoid. And it might discriminate users for whom the test
fails, because they would *always* need to spend the time for that
failed test.

> Noah

Best regards, Michael.

Re: SCPC Detection Program

[Noah Lavine]

Hi Michael,

I was thinking about it, and the help string idea does seem better
than version numbers, especially if it turns out that several
different SSH implementations use similarly-formatted help strings. I
think I will check the formatting of a few different version and help
strings before coding more.

However, I thought of an alternate possibility. What if tramp first
saves the version string of ssh, then tries a connection with
ControlMaster to see if it works, and then remembers whether this
worked or not? Then it would choose a method for future connections
based on the results of the test until the version string changed, at
which point it would experiment again.

That way it would have to run a test connection only when the user
updated their ssh installation, which seems infrequent enough to be
reasonable, and it would have the advantage that we wouldn't have to
keep any sort of whitelist of ssh programs or help string formats.

Noah

On Mon, Apr 12, 2010 at 3:55 PM, Michael Albinus <michael.albinus@gmx.de> wrote:
> Noah Lavine <noah549@gmail.com> writes:
>
>> Hi Michael,
>
> Hi Noah,
>
>> I checked with my version of ssh, and we would indeed be able to
>> detect this from its help version string. However, it seems like this
>> could have similar fragility issues as the version number idea, if the
>> help string changed format some time. You could solve this by only
>> trying scpc on help strings that are known to be good, but you could
>> also have a version number whitelist that would serve the same
>> purpose. It seems like both of these would work and would be quite
>> similar - is there a reason to think that one would be better than the
>> other?
>
> I have no bullet-proof reason. But your version test depends on
> underlying OpenSSH, and its version numbering scheme. What if other ssh
> implementations will support this option as well? Therefore, the version
> check seems to be more fragile to me than the "-M" option. Again, it is
> just a feeling.
>
> Another test could be the following:
>
> $ ssh -o ControlMaster=auto wrong-host
> ssh: Could not resolve hostname wrong-host: Name or service not known
>
> This would indicate, that the option is supported. Unsupported options
> return the following error message:
>
> $ ssh -o ControlMasterr=auto wrong-host
> command-line: line 0: Bad configuration option: ControlMasterr
>
>> As for the test connection, maybe a change would help. It's probably
>> not good to try connecting to a known-good host, but what about trying
>> it with the host that Tramp needs to connect to anyway? You could try
>> it first with the -ControlMaster option, and if ssh gave an error,
>> then try again without it and fall back on the other method. This
>> wouldn't create more ssh connections than necessary. The problem I see
>> is that it might require a more substantial change to the Tramp
>> codebase than the other methods, but I don't know, because I haven't
>> looked around enough to see. Do you think this method would be useful
>> enough to make it worth looking through the Tramp code more and maybe
>> making bigger changes?
>
> Yes, this would require deeper changes in Tramp's workflow, which I
> would like to avoid. And it might discriminate users for whom the test
> fails, because they would *always* need to spend the time for that
> failed test.
>
>> Noah
>
> Best regards, Michael.
>

Re: SCPC Detection Program

[Michael Albinus]

Noah Lavine <noah549@gmail.com> writes:

> Hi Michael,

Hi Noah,

> I was thinking about it, and the help string idea does seem better
> than version numbers, especially if it turns out that several
> different SSH implementations use similarly-formatted help strings. I
> think I will check the formatting of a few different version and help
> strings before coding more.
>
> However, I thought of an alternate possibility. What if tramp first
> saves the version string of ssh, then tries a connection with
> ControlMaster to see if it works, and then remembers whether this
> worked or not? Then it would choose a method for future connections
> based on the results of the test until the version string changed, at
> which point it would experiment again.
>
> That way it would have to run a test connection only when the user
> updated their ssh installation, which seems infrequent enough to be
> reasonable, and it would have the advantage that we wouldn't have to
> keep any sort of whitelist of ssh programs or help string formats.

I think it is not worth the hassle to let a connection fail, and to
apply book-keeping of ssh versions. Ted's proposal seems to fit best, I
believe.

> Noah

Best regards, Michael.

Re: SCPC Detection Program

[Noah Lavine]

Hi Michael,

That sounds excellent. I'll try to submit a patch soon to implement
that. Is it correct that patches like that should go to
bug-gnu-emacs@gnu.org?

Noah

On Tue, Apr 13, 2010 at 5:01 PM, Michael Albinus <michael.albinus@gmx.de> wrote:
> Noah Lavine <noah549@gmail.com> writes:
>
>> Hi Michael,
>
> Hi Noah,
>
>> I was thinking about it, and the help string idea does seem better
>> than version numbers, especially if it turns out that several
>> different SSH implementations use similarly-formatted help strings. I
>> think I will check the formatting of a few different version and help
>> strings before coding more.
>>
>> However, I thought of an alternate possibility. What if tramp first
>> saves the version string of ssh, then tries a connection with
>> ControlMaster to see if it works, and then remembers whether this
>> worked or not? Then it would choose a method for future connections
>> based on the results of the test until the version string changed, at
>> which point it would experiment again.
>>
>> That way it would have to run a test connection only when the user
>> updated their ssh installation, which seems infrequent enough to be
>> reasonable, and it would have the advantage that we wouldn't have to
>> keep any sort of whitelist of ssh programs or help string formats.
>
> I think it is not worth the hassle to let a connection fail, and to
> apply book-keeping of ssh versions. Ted's proposal seems to fit best, I
> believe.
>
>> Noah
>
> Best regards, Michael.
>

Re: SCPC Detection Program

[Michael Albinus]

Noah Lavine <noah549@gmail.com> writes:

> Hi Michael,

Hi Noah,

> That sounds excellent. I'll try to submit a patch soon to implement
> that. Is it correct that patches like that should go to
> bug-gnu-emacs@gnu.org?

It depends. In this case, it would be sufficient, if you simply post the
diff + a ChangeLog entry; I would commit it then for you.

However, the change in tramp.el shall not be longer than ~15 lines, this
is acceptable if you have not signed the FSF papers. If you intend to
contribute further on, I recommend you to contact the FSF and sign the
papers; somebody has told earlier where to find information about.

> Noah

Thanks, and best regards, Michael.

Re: SCPC Detection Program

[Noah Lavine]

Hi Michael,

I believe the following patch should do it. It's a bit less than 15
lines, but if that's still a problem, I've already written
copyright-clerk@fsf.org asking for papers to make it completely legal.

One issue is that the detection is currently fragile in the sense that
it will return nil if the response isn't formatted exactly as it
expects, even if the ssh program supports the ControlMaster argument -
but it seems to me that false positives are the thing to worry about
here, since false negatives are essentially exactly what the current
scheme gives anyway.

As for a ChangeLog entry, how about this?

    lisp/net/tramp.el: Detect ssh 'ControlMaster' argument
automatically in some cases.

Thank you very much
Noah Lavine

--- lisp/net/tramp-old.el	2010-04-08 10:40:26.000000000 -0400
+++ lisp/net/tramp.el	2010-04-15 19:45:21.000000000 -0400
@@ -653,38 +653,47 @@
 `localhost' or the name of the local host.  Another host name is
 useful only in combination with `tramp-default-proxies-alist'.")

-
+(defun tramp-detect-ssh-controlmaster ()
+  "Call ssh with invalid arguments to try to detect whether it
supports the ControlMaster argument. This function may return nil when
the argument is supported, but shouldn't return t when it isn't"
+  (with-temp-buffer
+    (let ((ssh-call (condition-case nil
+                        (call-process "ssh" nil t nil "-o" "ControlMaster")
+                      (error nil))))
+      (and ssh-call
+           (goto-char (point-min))
+           (looking-at "command-line line 0: Missing ControlMaster
argument.")))))

 (defcustom tramp-default-method
   ;; An external copy method seems to be preferred, because it is much
   ;; more performant for large files, and it hasn't too serious delays
   ;; for small files.  But it must be ensured that there aren't
   ;; permanent password queries.  Either a password agent like
-  ;; "ssh-agent" or "Pageant" shall run, or the optional password.el
-  ;; package shall be active for password caching.  "scpc" would be
-  ;; another good choice because of the "ControlMaster" option, but
-  ;; this is a more modern alternative in OpenSSH 4, which cannot be
-  ;; taken as default.
-  (cond
-   ;; PuTTY is installed.
-   ((executable-find "pscp")
-    (if	(or (fboundp 'password-read)
+  ;; "ssh-agent" or "Pageant" shall run, or the optional
+  ;; password-cache.el or auth-sources.el packages shall be active for
+  ;; password caching.
+  ;; "scpc" is chosen if we detect that the user is running OpenSSH 4.0
+  ;; or newer.
+  cond
+  ;; PuTTY is installed.
+  ((executable-find "pscp")
+   (if	(or (fboundp 'password-read)
+	    (fboundp 'auth-source-user-or-password)
 	    ;; Pageant is running.
-	    (and (fboundp 'w32-window-exists-p)
-		 (funcall (symbol-function 'w32-window-exists-p)
-			  "Pageant" "Pageant")))
-	"pscp"
-      "plink"))
-   ;; There is an ssh installation.
-   ((executable-find "scp")
-    (if	(or (fboundp 'password-read)
-	    ;; ssh-agent is running.
-	    (getenv "SSH_AUTH_SOCK")
-	    (getenv "SSH_AGENT_PID"))
-	"scp"
-      "ssh"))
-   ;; Fallback.
-   (t "ftp"))
+	    (tramp-compat-process-running-p "Pageant"))
+       "pscp"
+     "plink"))
+  ;; There is an ssh installation.
+  ((executable-find "scp")
+   (cond
+    ((tramp-detect-ssh-controlmaster) "scpc")
+    ((or (fboundp 'password-read)
+         (fboundp 'auth-source-user-or-password)
+         ;; ssh-agent is running.
+         (getenv "SSH_AUTH_SOCK")
+         (getenv "SSH_AGENT_PID")) "scp")
+    (t "ssh")))
+  ;; Fallback.
+  (t "ftp")
   "*Default method to use for transferring files.
 See `tramp-methods' for possibilities.
 Also see `tramp-default-method-alist'."


On Wed, Apr 14, 2010 at 11:31 PM, Michael Albinus
<michael.albinus@gmx.de> wrote:
> Noah Lavine <noah549@gmail.com> writes:
>
>> Hi Michael,
>
> Hi Noah,
>
>> That sounds excellent. I'll try to submit a patch soon to implement
>> that. Is it correct that patches like that should go to
>> bug-gnu-emacs@gnu.org?
>
> It depends. In this case, it would be sufficient, if you simply post the
> diff + a ChangeLog entry; I would commit it then for you.
>
> However, the change in tramp.el shall not be longer than ~15 lines, this
> is acceptable if you have not signed the FSF papers. If you intend to
> contribute further on, I recommend you to contact the FSF and sign the
> papers; somebody has told earlier where to find information about.
>
>> Noah
>
> Thanks, and best regards, Michael.
>

Re: SCPC Detection Program

[Michael Albinus]

Noah Lavine <noah549@gmail.com> writes:

> Hi Michael,

Hi Noah,

> I believe the following patch should do it. It's a bit less than 15
> lines, but if that's still a problem, I've already written
> copyright-clerk@fsf.org asking for papers to make it completely legal.

I've committed your patch to the trunk.

> Thank you very much
> Noah Lavine

Thanks, and best regards, Michael.