Re: Ungrafting CI jobs

[Mathieu Othacehe <othacehe@gnu.org>]

Hello Leo,

I monitored a previous evaluation of the ungrafting Cuirass
specification that was more successful, with more than 17000 builds
performed in less than 24 hours, a new record! The recent evaluation is
sadly less victorious.

> The latest iteration failed en masse due to network timeouts between
> build farm nodes.

Yes, I described this issue here: https://issues.guix.gnu.org/48468 and
hopefully fixed it with https://issues.guix.gnu.org/48556.

I need to perform a reconfigure / deploy but I'm waiting to have a few
days ahead of me to fix potential issues.

> It lets us see that the Audacity build is not attempted due to the
> failure of wxwidgets. And in turn, the failure of webkitgtk and libsoup.
> However, there is no information (logs, duration) available about those
> failed builds.

When a build is cancelled ("Cancel pending builds" button), all
its dependents are automatically updated to "Failed dependency". It
looks like this is the case for https://ci.guix.gnu.org/build/515005/details.

> What can we do to clear all these "spurious" build failures and re-try
> building the branch? I already tried using the "restart all builds"
> button, but there were still a lot of spurious failures.

I think the easiest way to recover from this situation is to remove the
ungrafting specification and recreate it. It ensures that no build will
fail because of previously failing/cancelled build dependencies.

Note that when a build B is restarted in Cuirass ("Restart" button), any
other build B' depending from B, will automatically be restarted if B
becomes successful and all the other dependencies of B' are already
successful.

I used this mechanism to restart a few builds failing on the master
branch because of substitutes timeout. I hope that when the keep-alive
patchset will solve this situation.

Thanks,

Mathieu