Re: Bringing substitutes from the Guix Build Coordinator to users

[Christopher Baines <mail@cbaines.net>]

[-- Attachment #1: Type: text/plain, Size: 6452 bytes --]


Ludovic Courtès <ludo@gnu.org> writes:

> Thanks for your message!  It’s great to see a summary of what’s been
> cooking in the Coordinator and your vision around it.
>
> Christopher Baines <mail@cbaines.net> skribis:
>
>> More specifically, while the architecture is similar to daemon
>> offloading, there are some practical advantages. Since the switch to
>> Cuirass remote workers for ci.guix.gnu.org, the advantages of building
>> things across multiple machines in a coordinated manor have also become
>> clear. Additionally, there are things that the Guix Build Coordinator
>> enables, like automated retries, with the option to target specific
>> agents, which can help with avoiding issues with non-reproducible
>> failures, as well as helping to avoid issues when mixing QEMU emulated
>> and native builds.
>
> These are nice examples of current QA blind spots that would be nice to
> address.
>
> [...]
>
>> This is a topic I haven't got directly involved in, until now. I'm just
>> a volunteer, in some ways the most involved I am is that I host an idle
>> ARM build machine, I don't have any particular connection in the default
>> approach to substitutes for users, or the hardware currently
>> involved.
>
> It would be great to have that OverDrive plugged in behind ci.guix
> because we’re still short on CPU power for ARM.  (Not great we had
> forgotten about this one!)
>
> It could be reconfigured with a config similar to
> <https://git.savannah.gnu.org/cgit/guix/maintenance.git/tree/hydra/overdrive1.scm>,
> with a different host name and IP.  We can discuss the details in a
> separate thread and/or on guix-sysadmin, let me know!

It was/is being discussed, the relevant thread is here:
  https://lists.gnu.org/mailman/private/guix-sysadmin/2021-March/003487.html

>> However, I think some of the stuff I've been working on could be
>> helpful, as I say, I think the Guix Build Coordinator is a step
>> forward in terms of building things for substitutes, and that shows in
>> the substitute availability percentages.
>>
>> Is there still a path to bring some of these benefits to users, and if
>> so, what things need doing?
>
> My understanding is that, to build substitutes, we need more than the
> Coordinator; we also need something to perform evaluations, similar to
> the “top half” of Cuirass and the Data Service, right?  How would you
> envision setting it up?

The queue builds script bundled in with the guix-build-coordinator seems
to work fine, that's how bayfront is currently set up:

  (service guix-build-coordinator-queue-builds-service-type
         (guix-build-coordinator-queue-builds-configuration
          (systems '("x86_64-linux"))))

https://git.savannah.gnu.org/cgit/guix/maintenance.git/tree/hydra/bayfront.scm#n781

> To me, the way forward is not necessarily substitutes; it could be QA,
> as you showed that it has helpful features.  You already have access to
> bayfront and it’d be great to experiment there.  Do you have ideas on
> how to do make progress on that front?

Ideas yes, and the two things are interrelated, but my perspective at
the moment is that it'll be harder to try and get some value from the
Guix Build Coordinator in terms of quality assurance, as compared to
substitutes.

> Besides, I feel that the Data Service has a huge role to play with a
> variety of applications, including QA, but also way beyond as you showed
> in the blog post¹.  I think it’s time take advantage of it.  :-)
>
> Specifically, as far as QA is concerned, I think we should place the
> Data Service at the center of the stage.  It already does things that
> Cuirass cannot and will not do, notably: linting, challenging substitute
> servers to estimate reproducibility, and recording the history of all
> this.  Couldn’t it gather these other QA metrics mentioned above,
> possibly from a Coordinator-powered bayfront?

To me, linting and substitute comparison are less important quality
assurance things. I think the key issues are around testing packages and
things like system tests.

There's been some progress, albeit slow progress with the patch testing
setup I've been working on, but I've been viewing that as a way to test
branches as well. Given changes happen through patches and branches, I
think this makes sense, and I think the Guix Data Service has some nice
features for helping with this, like comparing two revisions and working
out what breakages have occurred.

However, with patches and branches, this is very much a continuous
integration tooling issue, much more than substitutes. Cuirass is
already building branches, and it's been gaining features previously
found in the Guix Data Service, like tracking the history of which
derivations relate to which revision, and separating out packages from
system tests so that it's easier to look at them.

As people are going to Cuirass to look at the build status for packages,
system tests and various branches, the problem is similar to that of
substitutes. It doesn't matter if the Guix Build Coordinator seems to do
some things better, when the question comes up about whether something
has built yet, or whether a branch is ready to merge, Cuirass on
ci.guix.gnu.org is where people are going, and that seems unlikely to
change (at least less likely than the setup for providing substitutes).

The Guix Data Service in the center, making it easier to do various
things by providing the data, this was the idea when it started, but the
reality recently is that strategy hasn't been paying off.

> What I’d love to see is a set of “skins” for the Data Service:
> purpose-specific interfaces to the service.  There’s already the
> reproducibility one, the one for package versions, etc.  It would be
> great to have a qa.guix.gnu.org entry point with a simplified interface
> that would only show (say) reproducibility data, lint warnings, and so
> forth.  (Then we can also think of other skins, like Guix Weekly News, a
> page dedicated to browsing package version history, a security-related
> page, etc.  But now I feel like a spoiled kid asking for yet more
> presents.  :-))

Haha, some QA entry point has come up before [1], but I haven't got
around to looking at implementing anything yet.

1: https://lists.gnu.org/archive/html/guix-devel/2021-02/msg00096.html

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]