package.el: what does package "Checking ... " actually do?

package.el: what does package "Checking ... " actually do?

[Yuri D'Elia]

When I'm using a high-latency network connection, and I update some
packages, downloading takes a few seconds after the connection is
initiated, but package "Checking" instead takes literally minutes to
complete.

Re: package.el: what does package "Checking ... " actually do?

[Artur Malabarba]

A quick M-x rgrep shows this is probably the byte-compiler, or
checkdoc, or check-declare.

2015-07-17 12:11 GMT+01:00 Yuri D'Elia <wavexx@thregr.org>:
> When I'm using a high-latency network connection, and I update some
> packages, downloading takes a few seconds after the connection is
> initiated, but package "Checking" instead takes literally minutes to
> complete.
>
>

Re: package.el: what does package "Checking ... " actually do?

[Artur Malabarba]

[-- Attachment #1: Type: text/plain, Size: 201 bytes --]

Is it possible that these minutes are actually spent downloading a second
package, and it just happens that "Checking" was the last thing to be
messaged when the first downloaded package was compiled?

[-- Attachment #2: Type: text/html, Size: 229 bytes --]

Re: package.el: what does package "Checking ... " actually do?

[Yuri D'Elia]

On 17/07/15 13:11, Yuri D'Elia wrote:
> When I'm using a high-latency network connection, and I update some
> packages, downloading takes a few seconds after the connection is
> initiated, but package "Checking" instead takes literally minutes to
> complete.

Turns out, it's another case of:

https://debbugs.gnu.org/cgi/bugreport.cgi?bug=20015

tramp ControlMaster options (was: Re: package.el: what does package "Checking ... " actually do?)

[Yuri D'Elia]

On 18/07/15 13:22, Yuri D'Elia wrote:
> On 17/07/15 13:11, Yuri D'Elia wrote:
>> When I'm using a high-latency network connection, and I update some
>> packages, downloading takes a few seconds after the connection is
>> initiated, but package "Checking" instead takes literally minutes to
>> complete.
> 
> Turns out, it's another case of:
> 
> https://debbugs.gnu.org/cgi/bugreport.cgi?bug=20015

It's really that bad. Even bringing out the variable description of
tramp-ssh-controlmaster-options hangs.

The issue is caused by a rogue provider's DNS server responding with
NXDOMAIN but valid IN A record, which ssh still tries to connect to. And
since the host is filtered, this takes minutes to time out.

Considering that I have a global setting in my ~/.ssh/config that
enables ControlMaster/ControlPath globally, I consider this as a regression.

I tried the same probes using:

  ssh -o BatchMode=yes [various flags] 0.0.0.0

and I get comparable output, and there's no chance 0.0.0.0 would be a
valid address.

Re: tramp ControlMaster options

[Michael Albinus]

Yuri D'Elia <wavexx@thregr.org> writes:

> Considering that I have a global setting in my ~/.ssh/config that
> enables ControlMaster/ControlPath globally, I consider this as a regression.

There is `tramp-use-ssh-controlmaster-options' which you could set to
nil after loading Tramp.

Best regards, Michael.

Re: tramp ControlMaster options

[Thierry Volpiatto]

Michael Albinus <michael.albinus@gmx.de> writes:

> Yuri D'Elia <wavexx@thregr.org> writes:
>
>> Considering that I have a global setting in my ~/.ssh/config that
>> enables ControlMaster/ControlPath globally, I consider this as a regression.
>
> There is `tramp-use-ssh-controlmaster-options' which you could set to
> nil after loading Tramp.

Some people still have similar problem with helm, and I still don't know
why tramp is involved at startup, very annoying.

https://github.com/emacs-helm/helm/issues/1000

-- 
Thierry
Get my Gnupg key:
gpg --keyserver pgp.mit.edu --recv-keys 59F29997 

Re: tramp ControlMaster options

[Yuri D'Elia]

On 20/07/15 08:21, Thierry Volpiatto wrote:
> Michael Albinus <michael.albinus@gmx.de> writes:
> 
>> Yuri D'Elia <wavexx@thregr.org> writes:
>>
>>> Considering that I have a global setting in my ~/.ssh/config that
>>> enables ControlMaster/ControlPath globally, I consider this as a regression.
>>
>> There is `tramp-use-ssh-controlmaster-options' which you could set to
>> nil after loading Tramp.
> 
> Some people still have similar problem with helm, and I still don't know
> why tramp is involved at startup, very annoying.

This is because the definition of `tramp-use-ssh-controlmaster-options'
itself probes ssh in a bad way.

Any reason why we cannot use `ssh -o BatchMode=yes [various flags]
0.0.0.0' instead of using a domain name? This would greatly reduce the
issue.

Re: tramp ControlMaster options

[Michael Albinus]

Yuri D'Elia <wavexx@thregr.org> writes:

>>> There is `tramp-use-ssh-controlmaster-options' which you could set to
>>> nil after loading Tramp.
>> 
>> Some people still have similar problem with helm, and I still don't know
>> why tramp is involved at startup, very annoying.
>
> This is because the definition of `tramp-use-ssh-controlmaster-options'
> itself probes ssh in a bad way.

I don't know what you mean by this. This user option defaults simply to
t, no further test.

Likely you mean `tramp-ssh-controlmaster-options'. This is a variable,
and it will be computed only when it is needed. No Tramp ssh connection,
no probe of ssh for this.

Which Emacs version are you using?

> Any reason why we cannot use `ssh -o BatchMode=yes [various flags]
> 0.0.0.0' instead of using a domain name? This would greatly reduce the
> issue.

The function (!) `tramp-ssh-controlmaster-options' uses "localhost" for
test. This shall avoid DNS requests.

Best regards, Michael.

Re: tramp ControlMaster options

[Yuri D'Elia]

On 20/07/15 20:43, Michael Albinus wrote:
> Likely you mean `tramp-ssh-controlmaster-options'. This is a variable,
> and it will be computed only when it is needed. No Tramp ssh connection,
> no probe of ssh for this.

I hope I understand how defcustom works(?).
In emacs 24.5.1, I see this at line 310 of tramp.el:

(defcustom tramp-ssh-controlmaster-options
  ...

whose standard value is computed by calling ssh a few times.
I don't know what the ;;;###tramp-autoload cookie does though, but no
miracles I assume?

Just to confirm, this is the output of "strace -f -e trace=process"
after a (require 'tramp):

[pid  2189] vfork(Process 2255 attached
 <unfinished ...>
[pid  2255] execve("/usr/bin/ssh", ["/usr/bin/ssh", "-o",
"ControlMaster"], [/* 73 vars */] <unfinished ...>
[pid  2189] <... vfork resumed> )       = 2255
[pid  2255] <... execve resumed> )      = 0
[pid  2255] arch_prctl(ARCH_SET_FS, 0x7fcdde4cd800) = 0
...

>> Any reason why we cannot use `ssh -o BatchMode=yes [various flags]
>> 0.0.0.0' instead of using a domain name? This would greatly reduce the
>> issue.
> 
> The function (!) `tramp-ssh-controlmaster-options' uses "localhost" for
> test. This shall avoid DNS requests.

Still connects to a valid host, which might have a daemon running (in
fact, I have one on localhost for debugging).

If there's no reason to connect to something, call 0.0.0.0.

I love that tramp detects ControlMaster support in ssh, but calling
external processes on load, if really needed, should /really/ be quick.

Re: tramp ControlMaster options

[Thierry Volpiatto]

Yuri D'Elia <wavexx@thregr.org> writes:

> On 20/07/15 20:43, Michael Albinus wrote:
>> Likely you mean `tramp-ssh-controlmaster-options'. This is a variable,
>> and it will be computed only when it is needed. No Tramp ssh connection,
>> no probe of ssh for this.
>
> I hope I understand how defcustom works(?).
> In emacs 24.5.1, I see this at line 310 of tramp.el:
>
> (defcustom tramp-ssh-controlmaster-options
>   ...
>
> whose standard value is computed by calling ssh a few times.
> I don't know what the ;;;###tramp-autoload cookie does though, but no
> miracles I assume?
>
> Just to confirm, this is the output of "strace -f -e trace=process"
> after a (require 'tramp):
>
> [pid  2189] vfork(Process 2255 attached
>  <unfinished ...>
> [pid  2255] execve("/usr/bin/ssh", ["/usr/bin/ssh", "-o",
> "ControlMaster"], [/* 73 vars */] <unfinished ...>
> [pid  2189] <... vfork resumed> )       = 2255
> [pid  2255] <... execve resumed> )      = 0
> [pid  2255] arch_prctl(ARCH_SET_FS, 0x7fcdde4cd800) = 0
> ...

So IIUC tramp is calling ssh at load time when evaling its defcustom ?!!

>>> Any reason why we cannot use `ssh -o BatchMode=yes [various flags]
>>> 0.0.0.0' instead of using a domain name? This would greatly reduce the
>>> issue.
>> 
>> The function (!) `tramp-ssh-controlmaster-options' uses "localhost" for
>> test. This shall avoid DNS requests.
>
> Still connects to a valid host, which might have a daemon running (in
> fact, I have one on localhost for debugging).
>
> If there's no reason to connect to something, call 0.0.0.0.

Isn't it possible to setup the default value of
`tramp-ssh-controlmaster-options' only when tramp is involved (i.e at
first connection) instead of evaling this at load time ?

If tramp hang forever like it is the case actually user will know why
emacs is hanging.

> I love that tramp detects ControlMaster support in ssh, but calling
> external processes on load, if really needed, should /really/ be quick.

IMHO this should not happen at load time.


Thanks.

-- 
Thierry
Get my Gnupg key:
gpg --keyserver pgp.mit.edu --recv-keys 59F29997 

Re: tramp ControlMaster options

[Michael Albinus]

Thierry Volpiatto <thierry.volpiatto@gmail.com> writes:

>> In emacs 24.5.1, I see this at line 310 of tramp.el:

Here we are. What I have written is about Emacs 25, the development
branch in git.

> So IIUC tramp is calling ssh at load time when evaling its defcustom ?!!

Not any longer. This is fixed, as I have explained.

>>>> Any reason why we cannot use `ssh -o BatchMode=yes [various flags]
>>>> 0.0.0.0' instead of using a domain name? This would greatly reduce the
>>>> issue.
>>> 
>>> The function (!) `tramp-ssh-controlmaster-options' uses "localhost" for
>>> test. This shall avoid DNS requests.
>>
>> Still connects to a valid host, which might have a daemon running (in
>> fact, I have one on localhost for debugging).
>>
>> If there's no reason to connect to something, call 0.0.0.0.

I've tested. If there is a daemon running on your local host, it *also*
connects using 0.0.0.0. No difference to localhost.

--8<---------------cut here---------------start------------->8---
# ssh -o ControlPath=%C 0.0.0.0
Welcome to Ubuntu 15.04 (GNU/Linux 3.19.0-23-generic x86_64)

 * Documentation:  https://help.ubuntu.com/

0 packages can be updated.
0 updates are security updates.

*** System restart required ***
Last login: Mon Jul 20 20:41:09 2015 from localhost
--8<---------------cut here---------------end--------------->8---

That's why *current* Tramp connects to another port, hoping nobody let
run an sshd on that port.

--8<---------------cut here---------------start------------->8---
# ssh -p 23 -o ControlPath=%C localhost
ssh: connect to host localhost port 23: Connection refused
--8<---------------cut here---------------end--------------->8---

> Isn't it possible to setup the default value of
> `tramp-ssh-controlmaster-options' only when tramp is involved (i.e at
> first connection) instead of evaling this at load time ?

That's the case in recent Tramp.

>> I love that tramp detects ControlMaster support in ssh, but calling
>> external processes on load, if really needed, should /really/ be quick.

It does now.

In order to fix the problem for you, you could upgrade to the
development branch of Emacs (25.0.50). If this is not an option for you,
you might install Tramp 2.2.12, which has been released end of June.

Another Emacs 24.x release is unlikely, as Stefan has said a while ago.

> Thanks.

Best regards, Michael.

Re: tramp ControlMaster options

[Thierry Volpiatto]

Michael Albinus <michael.albinus@gmx.de> writes:

>> Isn't it possible to setup the default value of
>> `tramp-ssh-controlmaster-options' only when tramp is involved (i.e at
>> first connection) instead of evaling this at load time ?
>
> That's the case in recent Tramp.

Good, thanks.

-- 
Thierry
Get my Gnupg key:
gpg --keyserver pgp.mit.edu --recv-keys 59F29997 

Re: tramp ControlMaster options

[Yuri D'Elia]

On 21/07/15 07:53, Michael Albinus wrote:
> Thierry Volpiatto <thierry.volpiatto@gmail.com> writes:
> 
>>> In emacs 24.5.1, I see this at line 310 of tramp.el:
> 
> Here we are. What I have written is about Emacs 25, the development
> branch in git.

A-ha, sorry then.

> I've tested. If there is a daemon running on your local host, it *also*
> connects using 0.0.0.0. No difference to localhost.

Odd, but yes, does the same on my host. I'm wondering why it does that?
What about some other illegal addresses, like 0.0.0.1?

$ ssh 0.0.0.1
ssh: connect to host 0.0.0.1 port 22: Invalid argument
$ ssh -o random 0.0.0.1
command-line: line 0: Bad configuration option: random

so it looks usable and doesn't require an invalid port.

> In order to fix the problem for you, you could upgrade to the
> development branch of Emacs (25.0.50). If this is not an option for you,
> you might install Tramp 2.2.12, which has been released end of June.
> 
> Another Emacs 24.x release is unlikely, as Stefan has said a while ago.

I've set the options manually, so I'm good about this. I was just
preoccupied about the default behavior on load.

Re: tramp ControlMaster options

[Michael Albinus]

Yuri D'Elia <wavexx@thregr.org> writes:

> What about some other illegal addresses, like 0.0.0.1?
>
> $ ssh 0.0.0.1
> ssh: connect to host 0.0.0.1 port 22: Invalid argument
> $ ssh -o random 0.0.0.1
> command-line: line 0: Bad configuration option: random
>
> so it looks usable and doesn't require an invalid port.

Nice idea, thanks. I've committed a respective patch to Tramp and Emacs
repositories.

Best regards, Michael.