Hey,

The Guix Build Coordinator has been around for a while, it's been
available in Guix for more than 6 months now. The setup I've been using
to test the Guix Build Coordinator for building things for substitutes
(guix.cbaines.net) has been running since mid 2020.

Anecdotally, the test setup I've been running (guix.cbaines.net) has for
a while now, generally had higher percentage substitute availability
compared to ci.guix.gnu.org, as reported by guix weather. I think it
might also be building more things since cross derivations are built
plus i586-gnu builds.

I think there are some benefits for using the Guix Build Coordinator to
build things for substitutes, and it would be good to work out how to
get those benefits to users of Guix generally.

More specifically, while the architecture is similar to daemon
offloading, there are some practical advantages. Since the switch to
Cuirass remote workers for ci.guix.gnu.org, the advantages of building
things across multiple machines in a coordinated manor have also become
clear. Additionally, there are things that the Guix Build Coordinator
enables, like automated retries, with the option to target specific
agents, which can help with avoiding issues with non-reproducible
failures, as well as helping to avoid issues when mixing QEMU emulated
and native builds.

Going back to discussions in 2020, I think there was a path for starting
to experiment with the Guix Build Coordinator, trying it on a few
machines in the berlin build farm, but as I say, this was last year and
before any work on implementing similar functionality in Cuirass as far
as I'm aware. Since then, I've also got an instance of the Guix Build
Coordinator running on bayfront with a couple of extra machines also
involved for performing builds, but this is just starting off, and
doesn't bring any benefit in terms of substitutes, at least not yet.

This is a small part of the wider puzzle, when I was designing the Guix
Build Coordinator, it was meant for much more than building things for
substitutes, and those things are also happening. The Guix Build
Coordinator has enabled things like building packages affected by
patches, building fixed output derivations regularly, and will hopefully
enable a whole load of quality assurance things. But my hope was also to
try and tackle building things for substitutes, such that there's not
more software to maintain, and also to inform future work on the
guix-daemon itself, like whether it would be good for it to have an
asynchronous API, and how that might work.

This is a topic I haven't got directly involved in, until now. I'm just
a volunteer, in some ways the most involved I am is that I host an idle
ARM build machine, I don't have any particular connection in the default
approach to substitutes for users, or the hardware currently
involved. However, I think some of the stuff I've been working on could
be helpful, as I say, I think the Guix Build Coordinator is a step
forward in terms of building things for substitutes, and that shows in
the substitute availability percentages.

Is there still a path to bring some of these benefits to users, and if
so, what things need doing?

Thanks,

Chris