Re: New blog post on the Guix Build Coordinator: Building derivations, how complicated can it be?

[Mathieu Othacehe <othacehe@gnu.org>]

Hello,

> On Sat, Apr 24, 2021 at 11:10 AM Christopher Baines <mail@cbaines.net> wrote:
>> I did think about trying to include something about Cuirass, but I don't
>> have a clear picture of it's scope or purpose, so I'm not really the
>> right person to attempt to write authoritatively about it.
>
> OK, fair enough, Matthieu, Ludo, or anyone else can shed some light
> here ?

You can learn more about Cuirass by reading its online manual[1].
The Cuirass server at https://ci.guix.gnu.org currently:

* Evaluates new Guix derivations on several branches (master,
  core-updates, ...).

* Builds those derivations on a build farm of 29 machines, some of them
  connected using a Wireguard VPN.

* Reports the build status on the Web interface, by email at
  guix-ci@gnu.org, though a Web API used by "guix pull" and "guix
  weather".
  
Maintaining and improving this whole architecture has been keeping me
busy for the last year. It involved a constant monitoring of the build
farm, upgrading the database, and rewriting Cuirass almost completely,
between other things.

The situation is now much better. Cuirass offers a nice substitutes
coverage, at least for Intel architectures, it is stable, well covered
by unit tests and documented.

We have already discussed the Cuirass vs Build coordinator situation in
the past. I haven't changed by mind on that subject. The Guix Build
Coordinator is more or less the equivalent of the Cuirass remote build
mechanism[2].

Integrating the Build coordinator in the Berlin build farm would mean
hooking in to Cuirass as an alternative to the remote build mechanism. I
don't think it would be wise because:

* It wouldn't bring any new features as far as I can tell.

* It would mean maintaining a new SQLite database.

  When Cuirass was using an SQLite database, maintaining it was a
  nightmare. I had to dive into SQLite sources, apply a wide variety of
  hacks[3] to make it scale.

  Even if the Build coordinator is updated to use a PostgreSQL database,
  that would mean having two databases, sitting next to each other,
  with a very similar content, which is a nonsense in my opinion.

* The Build coordinator has no documentation, no unit tests and a large
  code base that would drastically increase the system administrator
  burden.

It really makes me sad that we have two pieces of software that are
stepping on each other toes. The Build coordinator has a nice concept
and represents a huge amount of work. However, integrating it to Berlin
is not an option for me.

I think that the next challenges on the CI front are:

* Increase the number of ARM machines in the build farm.

* Work on substitutes mirrors.

* Find a way to make Cuirass and the Data Service work together.

and we should face those issues together, rather than having competing
software sharing the few build machines we own.

Thanks,

Mathieu

[1]: https://guix.gnu.org/cuirass/manual/html_node/index.html
[2]: https://guix.gnu.org/cuirass/manual/html_node/Build-modes.html#With-the-remote-build-mechanism_002e
[3]: https://wiki.mozilla.org/Performance/Avoid_SQLite_In_Your_Next_Firefox_Feature