Hey!

I sent out the last update 3 weeks ago [1].

1: https://lists.gnu.org/archive/html/guix-devel/2021-11/msg00154.html

In summary, the space issue I mentioned in the previous update has
effectively been addressed. All the paused agents are now unpaused and
builds are happening again.

However, due to the time spent not building things, the backlog is
longer than usual, and the substitute availability (especially for
x86_64-linux and i686-linux) is lower than usual.

I've also noticed that bordeaux.guix.gnu.org doesn't work over IPv6, and
I want to fix that soon.

** Space issues and the nar-herder

bordeaux.guix.gnu.org wasn't building things for 2 weeks as the space on
bayfront was getting a little scarce. This week I started rolling out
the nar-herder [2], a utility I've been thinking about for a while. This
has enabled moving nars off of bayfront on to another machine which I've
confusingly named lakefront.

The lakefront machine is hosted by Hetzner in Germany, and has 6TB of
storage across 2 hard drives. When a nar is requested from bayfront, it
will check it's local storage, and serve it from there if it exists,
otherwise it will forward the request to lakefront. There might be a
slight drop in the speed you can download nars, but apart from that this
change shouldn't be visible.

The nar-herder is now busy deleting nars on bayfront which are available
on lakefront. Once it's got through the backlog, I'll enable NGinx
caching for the nars on bayfront, which should help improve
performance. I've tested downloading the largest nars (~2GB) though, and
it seems to work fine.

In addition to lakefront, I've also added a 6TB hard drive to hatysa,
the HoneyComb LX2 machine that I host. Like lakefront, it's busy
downloading the nars from bayfront. This will act as a backup in case
lakefront is lost.

In general this is an important step in being more flexible where the
nars are stored. There's still a reliance on storing pretty much all the
nars on a single machine, but which machine has this role is more
flexible now. I think this architecture also makes it easier to break
the "all nars on a single machine" restriction in the future as well.

Going forward, it would be good to have an additional full backup of the
nars that bayfront can serve things from, to provide more
redundancy. I'm hoping the nar-herder will also enable setting up
geographically distributed mirrors, which will hopefully improve
redundancy further, and maybe performance of fetching nars too.

Once I've had a chance to neaten up the code a little, I'll get a Guix
package and service written, plus I'll draft a Guix blog post about the
nar-herder.

2: https://git.cbaines.net/guix/nar-herder/about/

** Build machines and backlog

Because of the 2 weeks of not building anything, there's a significant
backlog of builds to get through, and I'm not including the new builds
from the core-updates-frozen merge here.

As for build machines, milano-guix-1 came back online today, which is
great. I believe harbourfront is still unusable through (broken hard
drive).

That means there's the following currently running build agents (by
architecture):

 - x86_64-linux + i686-linux (3 machines):
   - 4 core Intel NUC (giedi)
   - Max 16 cores for 1 concurrent build on bayfront
   - 32 cores on milano-guix-1 (slow storage though)
 - aarch64-linux + armhf-linux (2 machines)
   - 16 core HoneyComb LX2 (hatysa)
   - 4 core Overdrive (monokuma)
 - powerpc64le-linux (1 machine)
   - 64 core machine (polaris)

Ironically, I think that the most under-resourced area is x86_64-linux +
i686-linux. bayfront is restricted in what it can do since it also runs
the coordinator, and things go badly if the machine gets
overloaded. bayfront and milano-guix-1 both have hard drive storage,
which also can slow them down when building things (especially
milano-guix-1).

If we (as a project) want bordeaux.guix.gnu.org to have the capacity to
keep up, it would be good to make a plan to add capacity. I think even a
single high core count x86_64-linux machine with performant storage
would make a big difference.

** IPv6 not supported (yet)

I was slow to notice, but bordeaux.guix.gnu.org isn't available over
IPv6 yet, since bayfront doesn't seem to have IPv6 connectivity. I want
to address this, but I haven't worked out quite how to yet.

Please let me know if you have any comments or questions!

Chris