Re: Solstice infrastructure hackathon

[pukkamustard <pukkamustard@posteo.net>]

I'm in!

Ludovic Courtès <ludo@gnu.org> writes:

> Hello Guix!
>
> This week, the node behind {ci,issues,disarchive}.guix.gnu.org and
> guix.gnu.org was down twice for a few hours—nothing terrible in the end,
> but it reminded us that, even though Guix doesn’t rely on any particular
> machine, we can definitely feel the inconvenience when it’s down.
>
> We were unlucky enough that it happened days after the other build farm,
> bordeaux.guix.gnu.org, ran out of disk space and had its CI stopped,
> right before the big merge—so it doesn’t have substitutes for current
> master.
>
> While discussing this on IRC the other day, we thought that perhaps it
> was time to have an infrastructure hackathon.  How about Tuesday,
> Dec. 21st?  (Probably with a followup in January.)
>
> Here are tasks that were brought up:
>
>   • Set up a backup server for berlin.guix.gnu.org, the head node of the
>     ci.guix.gnu.org, possibly moving some services such as the web site
>     there.
>
>   • Add DNS redundancy for guix.gnu.org so it can point to one of two
>     hosts (need to figure out certbot challenges so both machines can
>     update their certificates).
>
>   • Set up status.guix.gnu.org with sysadmin status updates (possibly
>     using Prometheus?).
>
>   • Come up with a plan to add disks to the RAID array on bayfront, the
>     head node of bordeaux.guix.gnu.org.
>
>   • Work on a plan to back up the Disarchive database currently on
>     berlin.guix.
>
>   • Work on a plan to mirror nars from ci.guix and bordeaux.guix, using
>     plain rsync or <https://git.cbaines.net/guix/nar-herder/about/>.
>
>   • Have a documented procedure to set up substitute mirrors, such as
>     the one in .cn (I can’t find the URL), ideally with plain rsync
>     access.
>
> Am I forgetting something?
>
> Some of these tasks require root or physical access for the final steps,
> but most of them are about (1) coming up with a plan, and (2) adjusting
> the system configuration at
> <https://git.savannah.gnu.org/cgit/guix/maintenance.git/tree/hydra/>.
> Anyone with sysadmin experience to share and Guix System knowledge can
> join!  We’ll communicate over #guix on irc.libera.chat.
>
> Who’s in?  :-)
>
> Cheers,
> Ludo’.