Start of 2022 update on bordeaux.guix.gnu.org (over 1,000,000 successful builds!)

Start of 2022 update on bordeaux.guix.gnu.org (over 1,000,000 successful builds!)

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 4416 bytes --]

Hey!

It's not quite been a year since bordeaux.guix.gnu.org started
operating, but I thought I'd send out an update email towards the start
of 2022 looking back in to 2021 and looking forward to the rest of this
year.

If you're missing some context, this blog post should be useful, and has
lots of useful links:

  https://guix.gnu.org/en/blog/2021/substitutes-now-also-available-from-bordeauxguixgnuorg/

The last update was back in December [1] and talked about addressing the
nar capacity, the machines building things, and IPv6 support (something
I'm glad to see).

1: https://lists.gnu.org/archive/html/guix-devel/2021-12/msg00140.html

### Looking back

First, some data.

The first build was created back on the 9th of April, 2021.

So far the build coordinator behind bordeaux.guix.gnu.org has been asked
to build nearly 2 million things (> 1,939,000).

Of those builds, around 824,100 haven't been processed, and > 1,114,900
have been processed.

Of those processed builds, > 113,338 failed, and > 1,001,567 succeeded.

From those successful builds, bordeaux.guix.gnu.org provides > 914586
nars. That's 3.6TiB of lzip compressed data (~11TiB's uncompressed).

The Guix Build Coordinator project began with two use cases in mind,
building things to provide substitutes, and building things to perform
quality assurance tasks. The substitutes use case is the relevant one
here, so I'm just focusing on that. Regarding substitutes, there were a
few different aims in mind: things like making operating substitute
servers easier, increasing hardware utilisation and improving substitute
availability.

I guess the baseline for comparison has changed since ci.guix.gnu.org no
longer uses offloading to perform most builds, but I think there's
reason to think that progress has been made and is being made on the
above aims.

Since bordeaux.guix.gnu.org came in to existence, not all that much has
changed with the Guix Build Coordinator, I think most of the changes
have been tweaks and optimisations.

### Looking forward

I think the most important area for improvement is data.guix.gnu.org
which has been struggling to keep up with new revisions lately. I think
as Guix grows, both in terms of packages and supported architectures,
this additional data has slowed the revision loading process. I've got
some ideas on improving this, and I'm hoping to work on it soon.

Something I've been working on recently is neatening up the
configuration for the machines involved, there's a bit more to do on
this. It would also be beneficial to add more machines and expand the
supported architectures. Also in terms of the operation, getting regular
automated database backups in place, and increasing the redundancy and
capacity for the nar storage would be good.

Specifically on the nar capacity issue, assuming that the retention is
kept at 100%, I think it's likely that the available storage (currently
there's a rough limit of 5.5TiB) will need expanding this year. Now that
the nar-herder exists, this should be much less stressful than the
previous work to expand the available storage (which involved writing
the nar-herder from scratch).

On the subject of the nar-herder, I think continuing to work towards
having mirrors in different geographical locations is important to
improve substitute performance for everyone. I sent out an email
recently to resume discussion of the nar-herder design [2]. I also want
to look at gathering metrics about substitute use, probably through the
nar-herder, and that's somewhat dependent on mirrors, as I want any
metrics gathered to be across several mirrors.

2: https://lists.gnu.org/archive/html/guix-devel/2022-02/msg00033.html

It would also be nice to make the inner workings of the build
coordinator behind bordeaux.guix.gnu.org more visible, so everyone can
see what builds are happening.

Finally, it would be good to work towards bordeaux.guix.gnu.org building
packages for non-master branches. This would help when preparing for
staging/core-updates merges. I think this mostly involves properly
setting up a data.qa.guix.gnu.org data service and enabling substitutes
from this on the build agents.

Please let me know if you have any comments or questions. I think there
will also be time for discussion at the upcoming Guix Days event.

Thanks,

Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]