Long response times from elpa.gnu.org

Long response times from elpa.gnu.org

[Johan Andersson]

Hi,

I have in the last months noticed that elpa.gnu.org can be really slow and
sometimes package.el times out when fetching the archive-contents. Note
that this does not happen all the time. Most of the times it works
perfectly fine.

I am in Sweden, but I have also noticed that tests on Travis fails at the
same time. For example:
https://travis-ci.org/ecukes/ecukes/jobs/18464804From the Travis
output:

Contacting host: elpa.gnu.org:80
Failed to download `gnu' archive.

This is not specific to any Emacs version. I have the same issue in these
versions: 23.4, 24.1, 24.2, 24.3 and snapshot.

Any ideas?

Re: Long response times from elpa.gnu.org

[Stefan Monnier]

> I have in the last months noticed that elpa.gnu.org can be really slow and
> sometimes package.el times out when fetching the archive-contents. Note
> that this does not happen all the time. Most of the times it works
> perfectly fine.

I indeed saw this problem recently (a couple weeks ago), and when
I logged into elpa.gnu.org to investigate, I saw a flood of connections
in the log.  So I think it was a kind of DoS attack.  Logging in via SSH
was perfectly responsive, tho, and I'm no sysadmin.

When you see such slow response times, please go to savannah.gnu.org and
open a support request about it.


        Stefan

Re: Long response times from elpa.gnu.org

[Bob Proulx]

Stefan Monnier wrote:
> Johan Andersson wrote:
> > I have in the last months noticed that elpa.gnu.org can be really slow and
> > sometimes package.el times out when fetching the archive-contents. Note
> > that this does not happen all the time. Most of the times it works
> > perfectly fine.
> 
> I indeed saw this problem recently (a couple weeks ago), and when
> I logged into elpa.gnu.org to investigate, I saw a flood of connections
> in the log.  So I think it was a kind of DoS attack.  Logging in via SSH
> was perfectly responsive, tho, and I'm no sysadmin.
> 
> When you see such slow response times, please go to savannah.gnu.org and
> open a support request about it.

Good idea but as far as I know elpa.gnu.org is not a Savannah machine.
Therefore the Savannah Hackers won't have any ability to do anything
about it.  (Speaking as one of the Savannah Hackers.  But I keep
finding things tucked away in unknown corners.)

  https://savannah.gnu.org/maintenance/SavannahArchitecture/

Problems with elpa.gnu.org needs to be reported to the folks that
maintain elpa.gnu.org.  Or if that is unknown then to FSF sysadmin.

Bob

Re: Long response times from elpa.gnu.org

[Eli Zaretskii]

> Date: Sat, 8 Feb 2014 12:47:05 -0700
> From: Bob Proulx <bob@proulx.com>
> 
> > When you see such slow response times, please go to savannah.gnu.org and
> > open a support request about it.
> 
> Good idea but as far as I know elpa.gnu.org is not a Savannah machine.

??? Then how come I have in my elpa/.git/config this snippet:

  [remote "origin"]
	  url = git+ssh://git.savannah.gnu.org/srv/git/emacs/elpa

Re: Long response times from elpa.gnu.org

[Bob Proulx]

Johan Andersson wrote:
> Contacting host: elpa.gnu.org:80
                   ^^^^^^^^^^^^
> Failed to download `gnu' archive.

Stefan Monnier wrote:
> I indeed saw this problem recently (a couple weeks ago), and when
> I logged into elpa.gnu.org to investigate, I saw a flood of connections
                ^^^^^^^^^^^^

Eli Zaretskii wrote:
> > Bob Proulx wrote:
> > > When you see such slow response times, please go to savannah.gnu.org and
> > > open a support request about it.
> > 
> > Good idea but as far as I know elpa.gnu.org is not a Savannah machine.
> 
> ??? Then how come I have in my elpa/.git/config this snippet:
> 
>   [remote "origin"]
> 	  url = git+ssh://git.savannah.gnu.org/srv/git/emacs/elpa

Because git.savannah.gnu.org != elpa.gnu.org .  Those are different
systems.  How is elpa.gnu.org related to git.sv.gnu.org?

However if you are talking about vcs.sv.gnu.org VM (hosting
git.sv.gnu.org aka git.savannah.gnu.org) then the entire VM stack has
known performance problems.  There are at least 24 VMs hosted on one
Xen based dom0 system.  Karl and I believe that the dom0 is I/O
saturated when several of the systems are active at the same time.
The I/O saturation causes long I/O waits with little cpu usage causing
the appearance of a high load average while the cpu is idle.
Meanwhile any performance metrics observed on the VM is fake data and
they always report all okay.  The only way to really know what is
happening would be to observe the dom0 host during performance
brownouts.  If we had some visibility into what was reported by the
dom0 then we would know something.  So far we don't.  Until someone
can look at the dom0 we can't actually know anything.

The FSF admins so far are unconvinced that the dom0 is at I/O
capacity.  They think that the dom0 should be able to handle all of
the current load plus more.  And so we have the current status.  But I
don't think anything will improve the situation short of adding
additional hardware to increase the amount of data I/O capability.
Three dom0 systems instead of one would give 3x the capability.  Put
vcs onto its own hardware and I believe the problem would go away.
Note that they require that their systems run coreboot bios and
therefore I can't just send additional hardware to Boston to help.

Bob

Re: Long response times from elpa.gnu.org

[Glenn Morris]

Eli Zaretskii wrote:

>> Good idea but as far as I know elpa.gnu.org is not a Savannah machine.
>
> ??? Then how come I have in my elpa/.git/config this snippet:
>
>   [remote "origin"]
> 	  url = git+ssh://git.savannah.gnu.org/srv/git/emacs/elpa

That's irrelevant. Yes, savannah hosts the elpa repository, but the
actual package download server is a different machine, elpa.gnu.org.
It offers packages for download in much the same way as ftp.gnu.org
offers release tarfiles.

(But I thought/assumed Stefan maintained elpa.gnu.org, so I don't know
why he is referring people to Savannah about it. I hope *someone* is
keeping the system up-to-date and monitored, eg as I do for
debbugs.gnu.org.)

Re: Long response times from elpa.gnu.org

[Glenn Morris]

Glenn Morris wrote:

> (But I thought/assumed Stefan maintained elpa.gnu.org, so I don't know
> why he is referring people to Savannah about it. I hope *someone* is
> keeping the system up-to-date and monitored, eg as I do for
> debbugs.gnu.org.)

PS elpa.gnu.org is more important, since people download and run code
from it, assuming it is a trusted/secure host.

Re: Long response times from elpa.gnu.org

[Stefan Monnier]

> Good idea but as far as I know elpa.gnu.org is not a Savannah machine.
> Therefore the Savannah Hackers won't have any ability to do anything
> about it.  (Speaking as one of the Savannah Hackers.

Hmmm... I'm one of the "admin"s of elpa.gnu.org, but I don't know enough
to have any clue what I might do about a DoS, or even to really figure
out that the machine is indeed under a DoS attack.

So, I hope someone managing the "hypervisor" under which elpa.gnu.org is
running might be able to help.  Admittedly, I don't know if that's
linked to Savannah hackers or not, or who that might be.


        Stefan

Re: Long response times from elpa.gnu.org

[Bob Proulx]

Stefan Monnier wrote:
> > Good idea but as far as I know elpa.gnu.org is not a Savannah machine.
> > Therefore the Savannah Hackers won't have any ability to do anything
> > about it.  (Speaking as one of the Savannah Hackers.
> 
> Hmmm... I'm one of the "admin"s of elpa.gnu.org, but I don't know enough
> to have any clue what I might do about a DoS, or even to really figure
> out that the machine is indeed under a DoS attack.

It is a problem.  When it comes to a DoS attack the best place to
mitigate it is as far upstream in the routers as possible.  I think if
a network DoS attack is detected then it is definitely a good thing to
notify the FSF admins so that they are aware of the problem and can
help.  But even then I don't know what the best action that can be
taken specific to the gnu.org systems.  It would need someone familiar
with the network to be able to block the attacks upstream so that the
machine load can return to normal.

> So, I hope someone managing the "hypervisor" under which elpa.gnu.org is
> running might be able to help.  Admittedly, I don't know if that's
> linked to Savannah hackers or not, or who that might be.

I can run xentop and see that elpa does appear to be one of the VMs on
the "colonialone" xen dom0 server.  Along with debbugs too.  But I
have no other access to it.  For any admin help the escalation would
be to the FSF admins.

Being on the same dom0 means that potentially those are also affected
by the same issues as have been talked about for vcs.  On the VM it
just looks like long I/O wait times causing load with no cpu usage.
But the VM runs very slowly sometimes to the point that access, such
as through web browsers, times out.

When non-DoS slowdown problems such as these are encountered please do
report them to the FSF admins so that we can get things improved.  I
think the Savannah folks would appreciate knowing about it too since
we are all in the same dom0 boat.  Personally I would appreciate a
note to savannah-hackers-public or savannah-hackers-private about it.
We can then compare notes.  (About the slowdown problem.  Not about a
DoS.  I don't think we can help about a DoS.)

Bob

Re: Long response times from elpa.gnu.org

[Stefan Monnier]

> It is a problem.  When it comes to a DoS attack the best place to
> mitigate it is as far upstream in the routers as possible.

That's my understanding as well.

> Being on the same dom0 means that potentially those are also affected
> by the same issues as have been talked about for vcs.

I don't know what those issues are (haven't sen this discussion).

In the case of elpa.gnu.org, I saw the problem once (it lasted a day or
two), with the following symptoms:
- web access was slow, sometimes/often resulting in timeouts.
- ssh access was as responsive as usual (i.e. not slow).
dmesg had lots of repeated messages (including some messages about
skipping other messages, apparently some kind of throttling mechanism)
which I stupidly did not save.  IIRC those messages mentioned
IP addresses.


        Stefan