Ungrafting CI jobs

Ungrafting CI jobs

[Leo Famulari]

We started building the wip-ungrafting branch a week ago:

https://ci.guix.gnu.org/jobset/ungrafting

Some progress has been made, but not enough. This is partially due to
<https://bugs.gnu.org/48574>, and also due to inefficient building of
expensive bootstrap paths (namely, every Rust package derivation tried
to build the entire Rust dependency graph concurrently).

I just rebased the branch and pushed again.

Re: Ungrafting CI jobs

[Leo Famulari]

On Tue, May 25, 2021 at 04:04:22PM -0400, Leo Famulari wrote:
> I just rebased the branch and pushed again.

The latest iteration failed en masse due to network timeouts between
build farm nodes.

Now that Cuirass is aware of dependencies, we have useful info as shown
here:

https://ci.guix.gnu.org/build/515005/details

It lets us see that the Audacity build is not attempted due to the
failure of wxwidgets. And in turn, the failure of webkitgtk and libsoup.
However, there is no information (logs, duration) available about those
failed builds.

I successfully built the Audacity package on the wip-ungrafting branch
"by hand" on the build farm.

What can we do to clear all these "spurious" build failures and re-try
building the branch? I already tried using the "restart all builds"
button, but there were still a lot of spurious failures.

Re: Ungrafting CI jobs

[Mathieu Othacehe]

Hello Leo,

I monitored a previous evaluation of the ungrafting Cuirass
specification that was more successful, with more than 17000 builds
performed in less than 24 hours, a new record! The recent evaluation is
sadly less victorious.

> The latest iteration failed en masse due to network timeouts between
> build farm nodes.

Yes, I described this issue here: https://issues.guix.gnu.org/48468 and
hopefully fixed it with https://issues.guix.gnu.org/48556.

I need to perform a reconfigure / deploy but I'm waiting to have a few
days ahead of me to fix potential issues.

> It lets us see that the Audacity build is not attempted due to the
> failure of wxwidgets. And in turn, the failure of webkitgtk and libsoup.
> However, there is no information (logs, duration) available about those
> failed builds.

When a build is cancelled ("Cancel pending builds" button), all
its dependents are automatically updated to "Failed dependency". It
looks like this is the case for https://ci.guix.gnu.org/build/515005/details.

> What can we do to clear all these "spurious" build failures and re-try
> building the branch? I already tried using the "restart all builds"
> button, but there were still a lot of spurious failures.

I think the easiest way to recover from this situation is to remove the
ungrafting specification and recreate it. It ensures that no build will
fail because of previously failing/cancelled build dependencies.

Note that when a build B is restarted in Cuirass ("Restart" button), any
other build B' depending from B, will automatically be restarted if B
becomes successful and all the other dependencies of B' are already
successful.

I used this mechanism to restart a few builds failing on the master
branch because of substitutes timeout. I hope that when the keep-alive
patchset will solve this situation.

Thanks,

Mathieu

Re: Ungrafting CI jobs

[Leo Famulari]

On Wed, Jun 02, 2021 at 07:53:35PM +0200, Mathieu Othacehe wrote:
> I monitored a previous evaluation of the ungrafting Cuirass
> specification that was more successful, with more than 17000 builds
> performed in less than 24 hours, a new record! The recent evaluation is
> sadly less victorious.

Yeah, it was great for a while!

> I need to perform a reconfigure / deploy but I'm waiting to have a few
> days ahead of me to fix potential issues.

Okay. Can you let me know when you do? I'll delay the ungrafting builds
until then.

Re: Ungrafting CI jobs

[Mathieu Othacehe]

Hey Leo,

> Okay. Can you let me know when you do? I'll delay the ungrafting builds
> until then.

I deployed the upgrade yesterday. I've been monitoring it for 12 hours
without any sign of substitutes timeout. I think you can proceed whenever
you want.

Thanks,

Mathieu