From: "Ludovic Courtès" <ludo@gnu.org>
To: Guix Devel <guix-devel@gnu.org>
Cc: guix-sysadmin <guix-sysadmin@gnu.org>
Subject: Solstice infrastructure hackathon
Date: Thu, 16 Dec 2021 10:46:56 +0100 [thread overview]
Message-ID: <87v8zorien.fsf@inria.fr> (raw)
Hello Guix!
This week, the node behind {ci,issues,disarchive}.guix.gnu.org and
guix.gnu.org was down twice for a few hours—nothing terrible in the end,
but it reminded us that, even though Guix doesn’t rely on any particular
machine, we can definitely feel the inconvenience when it’s down.
We were unlucky enough that it happened days after the other build farm,
bordeaux.guix.gnu.org, ran out of disk space and had its CI stopped,
right before the big merge—so it doesn’t have substitutes for current
master.
While discussing this on IRC the other day, we thought that perhaps it
was time to have an infrastructure hackathon. How about Tuesday,
Dec. 21st? (Probably with a followup in January.)
Here are tasks that were brought up:
• Set up a backup server for berlin.guix.gnu.org, the head node of the
ci.guix.gnu.org, possibly moving some services such as the web site
there.
• Add DNS redundancy for guix.gnu.org so it can point to one of two
hosts (need to figure out certbot challenges so both machines can
update their certificates).
• Set up status.guix.gnu.org with sysadmin status updates (possibly
using Prometheus?).
• Come up with a plan to add disks to the RAID array on bayfront, the
head node of bordeaux.guix.gnu.org.
• Work on a plan to back up the Disarchive database currently on
berlin.guix.
• Work on a plan to mirror nars from ci.guix and bordeaux.guix, using
plain rsync or <https://git.cbaines.net/guix/nar-herder/about/>.
• Have a documented procedure to set up substitute mirrors, such as
the one in .cn (I can’t find the URL), ideally with plain rsync
access.
Am I forgetting something?
Some of these tasks require root or physical access for the final steps,
but most of them are about (1) coming up with a plan, and (2) adjusting
the system configuration at
<https://git.savannah.gnu.org/cgit/guix/maintenance.git/tree/hydra/>.
Anyone with sysadmin experience to share and Guix System knowledge can
join! We’ll communicate over #guix on irc.libera.chat.
Who’s in? :-)
Cheers,
Ludo’.
next reply other threads:[~2021-12-16 9:53 UTC|newest]
Thread overview: 9+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-12-16 9:46 Ludovic Courtès [this message]
2021-12-16 11:30 ` Solstice infrastructure hackathon zimoun
2021-12-17 1:57 ` Christopher Baines
2021-12-17 8:12 ` Mathieu Othacehe
2021-12-18 16:08 ` Ricardo Wurmus
2021-12-20 22:58 ` Ludovic Courtès
2021-12-18 10:57 ` pukkamustard
2021-12-22 0:44 ` Ludovic Courtès
-- strict thread matches above, loose matches on Subject: below --
2021-12-16 10:13 Blake Shaw
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: https://guix.gnu.org/
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=87v8zorien.fsf@inria.fr \
--to=ludo@gnu.org \
--cc=guix-devel@gnu.org \
--cc=guix-sysadmin@gnu.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://git.savannah.gnu.org/cgit/guix.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).