September update on bordeaux.guix.gnu.org

September update on bordeaux.guix.gnu.org

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 4257 bytes --]

Hey,

The last update was sent out in May [1], so this update roughly covers
the last 3 months.

1: https://lists.gnu.org/archive/html/guix-devel/2022-05/msg00202.html

This hasn't been my primary focus, so some of the changes that somewhat
involve bordeaux.guix.gnu.org relate to using it for patch review, but
to keep this update short I'll mention those elsewhere.

## Numbers

Available from bordeaux.guix.gnu.org now, there's ~1.5 million nars,
which take up ~6.2TiB of space available.

Substitute availability for recent guix revisions is generally good,
although powerpc64le-linux substitute availability is lower than it
could be as there's not currently an always available machine to carry
out the builds.

## Hardware

In terms of machines, there was more work needed in the last few months
as hatysa, the HoneyComb LX2 machine building ARM things was running low
on storage and stopped booting. There's a couple of messages on
guix-sysadmin about this but it was eventually resolved.

Mixed up with this is the addition of another HoneyComb LX2 machine
(hamal), this brings the total number of machines doing ARM builds to 3
(hatysa, hamal and monokuma).

## The build coordinator

Hooks, which are used when various things happen can now run in
parallel. This is important to avoid delays when builds are submitted or
succeed.

Also, there was a bug in the agent that led to spurious build
failures. That's now been fixed.

## Mirrors

There's a few test mirror machines setup, and I asked for people to test
these to see what difference they make [2][3].

2: https://lists.gnu.org/archive/html/guix-devel/2022-05/msg00203.html
3: https://lists.gnu.org/archive/html/guix-devel/2022-06/msg00186.html

A couple of responses [4][5] have come in to the mailing list, both of
which seem to suggest that mirrors could provide a significant boost to
substitute download speed.

4: https://lists.gnu.org/archive/html/guix-devel/2022-07/msg00163.html
5: https://lists.gnu.org/archive/html/guix-devel/2022-07/msg00320.html

Those test servers have been running for a while now, and are generally
unused. I'll probably shut them down shortly to save money, and try to
send out a more concrete plan of getting mirrors in place for
bordeaux.guix.gnu.org.

## Serving fixed output files by hash

There's a separate thread about this:

  https://lists.gnu.org/archive/html/guix-devel/2022-06/msg00333.html

One thing that has changed is that the Guile fibers concurrent web
server (which is used by the nar-herder) now supports streaming
responses, which will reduce the memory usage of serving fixed output
files:

  https://github.com/wingo/fibers/pull/63

## Next steps

Building patches and non-master branches is starting to happen, and
that'll indirectly improve bordeaux.guix.gnu.org substitutes since it'll
have already built some things by the time the patches are merged.

As said above, from the limited data available, I think an argument
could be made that mirrors are worth it in terms of the reliability
benefits and potential performance improvements. I'll try to keep moving
this forward.

Operationally, I think the goal should be that it's not dependent on a
single person. Currently it's probably too dependent on me. Things like
moving the build-coordinator and nar-herder Git repositories to Savannah
and getting more of the machines owned by guix-europe are ways to
improve this.

zstd compression support has been requested, and I don't see any
significant blockers to this. I think the way forward is to add support
for cached recompression of the nar files in the nar-herder.

For build hardware, I think it remains to be seen how the addition of
the patch and non-master branch builds affect the load. As mentioned
above, having an always available powerpc64le-linux system would help
avoid a backlog of builds for that architecture.

It should also be much easier to see what the bordeaux build coordinator
is doing. I think this requires a web interface to expose the active
agents and builds.

If you're interested in working on any of this, do let me know as while
I don't have time to work on much of it myself, I should be able to make
time to help others.

Let me know if you have any comments or questions!

Thanks,

Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

Re: September update on bordeaux.guix.gnu.org

[John Kehayias]

Hi Chris,

On Sat, Sep 03, 2022 at 01:13 PM, Christopher Baines wrote:

> Hey,
>
> The last update was sent out in May [1], so this update roughly covers
> the last 3 months.
>
> 1: <https://lists.gnu.org/archive/html/guix-devel/2022-05/msg00202.html>
>
> This hasn't been my primary focus, so some of the changes that somewhat
> involve bordeaux.guix.gnu.org relate to using it for patch review, but
> to keep this update short I'll mention those elsewhere.
>

Thanks for these updates and all the work you do on these many fronts, much appreciated!

[snip]

> ## Mirrors
>
> There's a few test mirror machines setup, and I asked for people to test
> these to see what difference they make [2][3].
>
> 2: <https://lists.gnu.org/archive/html/guix-devel/2022-05/msg00203.html>
> 3: <https://lists.gnu.org/archive/html/guix-devel/2022-06/msg00186.html>
>
> A couple of responses [4][5] have come in to the mailing list, both of
> which seem to suggest that mirrors could provide a significant boost to
> substitute download speed.
>
> 4: <https://lists.gnu.org/archive/html/guix-devel/2022-07/msg00163.html>
> 5: <https://lists.gnu.org/archive/html/guix-devel/2022-07/msg00320.html>
>
> Those test servers have been running for a while now, and are generally
> unused. I'll probably shut them down shortly to save money, and try to
> send out a more concrete plan of getting mirrors in place for
> bordeaux.guix.gnu.org.
>

I wasn't sure if these mirrors were going to remain up and were usable, so I never actually configured my Guix machines to use them. I did make the main Bordeaux server first on my substitute list which has been a nice speedup.

Being based in the US, I would definitely make use of a US mirror, and I'm sure there are others that would to. As I mentioned in [4], I'm happy to help contribute funds for keeping up such a substitute server. I would be glad to help out in any other way I can as well (I'm no system or network administrator outside of my own machines and little home server, but feel free to contact me off list too).

Thanks again for your work on this!

John