On 2024-07-02, Ludovic Courtès wrote:
> We (Andreas, Chris, Ricardo, Romain, and myself) were having a
> discussion about what it would take to set up a build farm similar to
> what’s behind ci.guix: roughly 30 x86_64 servers, with 32-core/64-thread
> CPUs and 128 GiB of RAM.  The reason for this discussion is that we were
> thinking that we should not take our existing build farms for granted
> and be prepared for the future.
>
> The various options and back-of-the-envelope estimates we came up with
> are as follows:
>
>   1. Buying and hosting hardware:
>       250k€ for hardware
>       3k€/month (36k€/year)
>
>   2. Renting machines (e.g., on Hetzner):
>       6k€/month (72k€/year)
>
>   3. Sponsored:
>       get hardware and/or hosting sponsored (by academic institutions or
>       companies).

This may be a little wild, but what are the downsides to doing some
combination of all of the above? Maybe higher bandwidth requirements
between the various pieces of infrastructure presumably being hosted in
different locations? Maybe also a little more complexity in the overall
setup?

A mixed strategy could reduce ... the upfront cost of buying and hosting
hardware (#1), the ongoing costs of renting (#2), and dependence on the
generosity of a third party for sponsored hardware & hosting (#3).

It seems like any strategy should have some redundancy (e.g. multiple
independent build farms) so that a failure in one datacenter does not
effectively take down the whole network...

In a sense, we already have some of that, with ci.guix.gnu.org and
bordeaux.guix.gnu.org, and also the new North American build farm
... though they are not full replacements for each other.

live well,
  vagrant