Re: plz is there a roadmap for a more resilient substitutes infrastructure?

[ludo@gnu.org (Ludovic Courtès)]

Ciao Giovanni,

Giovanni Biscuolo <g@xelera.eu> skribis:

> recently users and developers are facing hard to manage problems due to
> the maintainance of hydra.gnu.org and its proxy mirror.hydra.gnu.org [1]
> since 23 Oct 2018

We Guix developers don’t have control over the physical hardware behind
hydra.gnu.org; for this machine, we rely on the work of the FSF
sysadmins for all things hardware/networking.

Unfortunately in this case, this maintenance period was rather
unprepared: it wasn’t supposed to last a whole week, rather a few hours
or a day at most.  Most of the time it took was about copying data to a
new disk (!).  Had this been prepared, we could have arranged to keep
hydra.gnu.org up until the replacement was ready.  We Guix developers
didn’t have much visibility over what was going on though, and we just
didn’t anticipate this.

It is clear that this prolonged downtime was harmful to many users and
to the project’s reputation.


What to do from here?

Our main focus is on making berlin.guixsd.org the primary build farm of
the project.  It has the advantage that one Guix dev has physical access
to it (Ricardo); it’s also much more powerful than hydra.gnu.org and the
associated build machines.

Yet, there’s more work to do: berlin has just 1T of disk space.  Ricardo
started looking on growing it but was stuck on software issues IIRC.  I
think fixing this should be a priority, so I think we should help
Ricardo fix the software issues as much as we can.


That alone doesn’t fix the resilience issue: berlin.guixsd.org could go
down at some point for some time.

To address that, a possibility that was discussed recently on
guix-sysadmin is use bayfront.guixsd.org has a separate build farm
and/or mirror of berlin.

On top of that we could have a service like httpredir.debian.org, or
maybe even a CDN where we’d replicate substitutes, or torrents (looking
at you, Julien ;-)).


At this point, all these options are still on the table, and anyone with
expertise in these areas is very much welcome!

> given the prolonged issue, please also consider writing an *official*
> blog post explaining the current situation and steps adopted to prevent
> similar issues in the future

We set up the info-guix mailing list with that in mind (but too late for
this incident).  Posting blog posts is also a good idea; we should have
done that, with instructions on how to switch to berlin.guixsd.org.

> 1. is there a method to "replicate the whole store of an official server
> (e.g. hydra.gnu.org once healed)" so we can just "guix publish" a
> *complete* mirror? In this case a ready to use official
> mirror-config.scm could be useful

mirror.hydra.gnu.org is a simple nginx proxy to hydra.gnu.org.  You can
find its config here:

  https://git.savannah.gnu.org/cgit/guix/maintenance.git/tree/hydra/nginx/mirror.conf

In the past a few people set up their own mirrors using a similar
configuration.

> 2. is there an official mirrors directory users can look at when needed?

No.

> 3. is there a plan to build a service similar to
> http://httpredir.debian.org/? (I looked on the web but did not find any
> reference to such plan)

Like I wrote, there’s no concrete plan at this point, which means it’s
an opportunity for you and anyone else to chime in and give a hand!

Thanks,
Ludo’.