Re: Hackathon: Mumi and Cuirass

[Christopher Baines <mail@cbaines.net>]

[-- Attachment #1: Type: text/plain, Size: 3914 bytes --]


Ricardo Wurmus <rekado@elephly.net> writes:

> Hey there,
>
> now that Guix 1.0 is out I think it’s a good time to improve our tools
> to allow us to better identify problems with package builds and to
> better respond to bug reports and patch submissions.

...

> 2) Cuirass
>
> Cuirass has a web interface: https://ci.guix.gnu.org/
>
> Unfortunately, it doesn’t yet enable us to answer questions like:
>
> - What packages are currently failing to build on aarch64?
> - What is the history of icecat builds on i686?
> - Can you show me all builds matching “rust” across all architectures?
> - What builds are blocked because of this one failing build?
>
> Implementing features that would allow us to better see the build status
> for all of our supported architectures would be very welcome.  It would
> be even better if we could also see an overview of our build nodes and
> what they are currently doing – how many build slots are still
> available?  Why is only one of the nodes doing any work?
>
> We have the freedom to integrate Cuirass more closely with the rest of
> Guix and the offloading framework, so this seems like a good target.
>
> If you find this kind of project interesting please get in touch!  You
> don’t need to know how to do any of these tasks as long as you are
> motivated to discuss this and help us find solutions together.
>
> Let’s do this!

I mentioned some things which seem related to this in this email [1],
but one of the enhancements I'm specifically interested in getting in to
Cuirass is a way to provide subscriptions to the build information. This
would enable "listening" to Cuirass, to get a near real time stream of
events, build information, things like that.

I previously tried scraping build information in to the Guix Data
Service, and you can see some data here [2], but that was slow, and I
was potentially missing builds. I think some kind of publish/subscribe
model would work out much better.

1: https://lists.gnu.org/archive/html/guix-devel/2019-04/msg00094.html
2: https://prototype-guix-data-service.cbaines.net/builds

> - What packages are currently failing to build on aarch64?
> - What is the history of icecat builds on i686?
> - Can you show me all builds matching “rust” across all architectures?
> - What builds are blocked because of this one failing build?

On the subject of questions like these, I would hope that if the Guix
Data Service had up to date build information, it wouldn't be too
difficult to add the necessary pages to make answering these questions
easy.

At the moment, as I say, there's no up to date information on builds,
and the only page I've added a form for comparing derivations between
two revisions ([3] for example).

3: https://prototype-guix-data-service.cbaines.net/compare/derivations?base_commit=094aaa1a1e81b6c5461c1a2aa8ccd9019faa97ec&target_commit=4ee55938de315ac965c12a99f5dc98f48ac2cc83

Now I'm not saying that I think it would be bad if Cuirass had a more
detailed web interface, but even if it did, I'd still be interested in
gathering up all the build information, as one thing I'm excited about
for the future is the possibility of having multiple independent build
farms.

In my mind, reproducible builds + multiple independant build farms is
the path towards trusted substitutes. From a users perspective, I'm
imaginging setting some policy where I trust substitutes if berlin, and
some other build farm agree on the output. But for a overall view of
what's going on, this is where I see the value of gathering up build
information from potentially many build farms in to one place.

Now that I've got some updated patches for Patchwork up [4], I'll
hopefully be able to make some time to work on this area over the coming
weeks.

4: https://issues.guix.info/issue/33185#38

Thanks,

Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 962 bytes --]