bug#63414: Evaluation comparison on cuirass

bug#63414: Evaluation comparison on cuirass

[Andreas Enge]

When working on a branch and deciding whether to merge it, we need a way
of comparing its status with that of the master branch. As far as I can see,
there is currently no way in cuirass to compare arbitrary evaluations and
get a list (or a dashboard) of builds that fail in one, but not the other.

Andreas

bug#63414: Evaluation comparison on cuirass

[Josselin Poiret via Bug reports for GNU Guix]

[-- Attachment #1: Type: text/plain, Size: 1212 bytes --]

Hi Andreas,

Andreas Enge <andreas@enge.fr> writes:

> When working on a branch and deciding whether to merge it, we need a way
> of comparing its status with that of the master branch. As far as I can see,
> there is currently no way in cuirass to compare arbitrary evaluations and
> get a list (or a dashboard) of builds that fail in one, but not the other.
>
> Andreas

I guess that this is one of the features that the Build Coordinator was
built for (and it is pretty damn good at this).  Maybe we could start
considering whether it makes sense to duplicate effort on Cuirass and
the Build Coordinator?  I don't know how "production-ready" the build
coordinator is, compared to Cuirass?  Maybe we could target getting the
Build Coordinator up to feature parity with Cuirass so that it may be
used on a wider scale?  If this is something we want to focus on, we
could create a team around it and set clear goals, which would probably
lessen the burden that's on Chris currently.

I understand that Cuirass is general enough to support much more than
Guix, but the coordinator is a wonderful piece of software and our
workflows might be outgrowing it.

WDYT?

-- 
Josselin Poiret

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 682 bytes --]

bug#63414: Evaluation comparison on cuirass

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 2685 bytes --]


Josselin Poiret via Bug reports for GNU Guix <bug-guix@gnu.org> writes:

> Hi Andreas,
>
> Andreas Enge <andreas@enge.fr> writes:
>
>> When working on a branch and deciding whether to merge it, we need a way
>> of comparing its status with that of the master branch. As far as I can see,
>> there is currently no way in cuirass to compare arbitrary evaluations and
>> get a list (or a dashboard) of builds that fail in one, but not the other.
>>
>> Andreas
>
> I guess that this is one of the features that the Build Coordinator was
> built for (and it is pretty damn good at this).  Maybe we could start
> considering whether it makes sense to duplicate effort on Cuirass and
> the Build Coordinator?  I don't know how "production-ready" the build
> coordinator is, compared to Cuirass?  Maybe we could target getting the
> Build Coordinator up to feature parity with Cuirass so that it may be
> used on a wider scale?  If this is something we want to focus on, we
> could create a team around it and set clear goals, which would probably
> lessen the burden that's on Chris currently.
>
> I understand that Cuirass is general enough to support much more than
> Guix, but the coordinator is a wonderful piece of software and our
> workflows might be outgrowing it.

There's some pedantic bits here to bring up. The build coordinator
doesn't have anything to do with comparing revisions (it doesn't even
really know what builds correspond to which revisions), it's just for
performing builds potentially across many machines, and doing something
useful with the results.

The data service however is meant for comparing revisions. There's a
circular relationship between the two as well, since the data service
can provide substitutes for derivations, which enables the build
coordinator to easily build them, and then report the results of those
builds back to the data service. This information about builds is
important since that can then factor in to comparisons between
revisions.

On the bit about "feature parity with Cuirass" though, this is a bit
misleading as the build coordinator exists because I wanted something
with very different design decisions to Cuirass. In terms of core
features, the build coordinator was complete back in late 2020
[1]. There's obviously lots that Cuirass does that the build coordinator
does not, but adding features without looking at the bigger picture can
be detrimental in the long term.

1: https://lists.gnu.org/archive/html/guix-devel/2020-11/msg00417.html

This is not to say there aren't things to work on in the build
coordinator. There are some ideas in the README and I'm more than happy
to try and help people get more involved.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

bug#63414: Evaluation comparison on cuirass

[Ludovic Courtès]

Hello,

Andreas Enge <andreas@enge.fr> skribis:

> When working on a branch and deciding whether to merge it, we need a way
> of comparing its status with that of the master branch. As far as I can see,
> there is currently no way in cuirass to compare arbitrary evaluations and
> get a list (or a dashboard) of builds that fail in one, but not the other.

Going back to this, I agree with Josselin that the Data Service does an
excellent job at comparing the status of different revisions; I think we
should leverage that rather than try to implement something similar in
Cuirass.

Perhaps one thing we can improve though is the information flow from
Cuirass to the Data Service.  ISTR that Christopher mentioned that the
Data Service couldn’t easily get information about substitute
availability from ci.guix, or something like that.

Is there still a problem here, Chris?  If we need a new HTTP endpoint in
Cuirass to address that, I’m happy to give a hand.

Thanks,
Ludo’.

bug#63414: Evaluation comparison on cuirass

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 2167 bytes --]


Ludovic Courtès <ludo@gnu.org> writes:

> Hello,
>
> Andreas Enge <andreas@enge.fr> skribis:
>
>> When working on a branch and deciding whether to merge it, we need a way
>> of comparing its status with that of the master branch. As far as I can see,
>> there is currently no way in cuirass to compare arbitrary evaluations and
>> get a list (or a dashboard) of builds that fail in one, but not the other.
>
> Going back to this, I agree with Josselin that the Data Service does an
> excellent job at comparing the status of different revisions; I think we
> should leverage that rather than try to implement something similar in
> Cuirass.
>
> Perhaps one thing we can improve though is the information flow from
> Cuirass to the Data Service.  ISTR that Christopher mentioned that the
> Data Service couldn’t easily get information about substitute
> availability from ci.guix, or something like that.

Substitute availability is easy to get, but there's some difficulties
around build information.

> Is there still a problem here, Chris?  If we need a new HTTP endpoint in
> Cuirass to address that, I’m happy to give a hand.

The data service used to poll ci.guix.gnu.org for builds and build
status information, but I stopped that quite a while ago after the data
got messy when the Cuirass database was deleted and recreated. Since the
data service stores and uses the build IDs from Cuirass, it's confusing
when they're reused.

Anyway, even ignoring that, I'm unsure if polling worked well. That's
why I started to look at pushing data from Cuirass to the data
serivce. I did implement that, but the code on the Cuirass side was
never used. It's that same endpoint though that the build coordinator
uses to send build information to the data service (both instances).

The other blocker to making use of Cuirass data in the data service is
making sure it's high quality, in particular that if it says a build has
failed, I at least want to know it's started to build that
derivation. We don't want things showing up on QA as problems when it's
just Cuirass being unable to start builds.

Thanks,

Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

bug#63414: Evaluation comparison on cuirass

[Ludovic Courtès]

Hi,

Christopher Baines <mail@cbaines.net> skribis:

> The data service used to poll ci.guix.gnu.org for builds and build
> status information, but I stopped that quite a while ago after the data
> got messy when the Cuirass database was deleted and recreated. Since the
> data service stores and uses the build IDs from Cuirass, it's confusing
> when they're reused.

Ah yes, that’s a problem.  Maybe it should have UUIDs or something in
addition to those database IDs; or maybe the Data Service can use, say,
derivation + ID as a unique ID for Cuirass builds?

[...]

> The other blocker to making use of Cuirass data in the data service is
> making sure it's high quality, in particular that if it says a build has
> failed, I at least want to know it's started to build that
> derivation. We don't want things showing up on QA as problems when it's
> just Cuirass being unable to start builds.

Indeed.  :-)  Well, I do hope that status = failed really means build
failure; seems we’re not completely done with the infamous “missing
.drv” bug though, and that’s erroneously reported as “failed”.

Ludo’.