unofficial mirror of guix-devel@gnu.org 
 help / color / mirror / code / Atom feed
* Ungrafting CI jobs
@ 2021-05-25 20:04 Leo Famulari
  2021-06-01 16:59 ` Leo Famulari
  0 siblings, 1 reply; 5+ messages in thread
From: Leo Famulari @ 2021-05-25 20:04 UTC (permalink / raw)
  To: guix-devel

We started building the wip-ungrafting branch a week ago:

https://ci.guix.gnu.org/jobset/ungrafting

Some progress has been made, but not enough. This is partially due to
<https://bugs.gnu.org/48574>, and also due to inefficient building of
expensive bootstrap paths (namely, every Rust package derivation tried
to build the entire Rust dependency graph concurrently).

I just rebased the branch and pushed again.


^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: Ungrafting CI jobs
  2021-05-25 20:04 Ungrafting CI jobs Leo Famulari
@ 2021-06-01 16:59 ` Leo Famulari
  2021-06-02 17:53   ` Mathieu Othacehe
  0 siblings, 1 reply; 5+ messages in thread
From: Leo Famulari @ 2021-06-01 16:59 UTC (permalink / raw)
  To: guix-devel

On Tue, May 25, 2021 at 04:04:22PM -0400, Leo Famulari wrote:
> I just rebased the branch and pushed again.

The latest iteration failed en masse due to network timeouts between
build farm nodes.

Now that Cuirass is aware of dependencies, we have useful info as shown
here:

https://ci.guix.gnu.org/build/515005/details

It lets us see that the Audacity build is not attempted due to the
failure of wxwidgets. And in turn, the failure of webkitgtk and libsoup.
However, there is no information (logs, duration) available about those
failed builds.

I successfully built the Audacity package on the wip-ungrafting branch
"by hand" on the build farm.

What can we do to clear all these "spurious" build failures and re-try
building the branch? I already tried using the "restart all builds"
button, but there were still a lot of spurious failures.


^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: Ungrafting CI jobs
  2021-06-01 16:59 ` Leo Famulari
@ 2021-06-02 17:53   ` Mathieu Othacehe
  2021-06-03 20:22     ` Leo Famulari
  0 siblings, 1 reply; 5+ messages in thread
From: Mathieu Othacehe @ 2021-06-02 17:53 UTC (permalink / raw)
  To: Leo Famulari; +Cc: guix-devel


Hello Leo,

I monitored a previous evaluation of the ungrafting Cuirass
specification that was more successful, with more than 17000 builds
performed in less than 24 hours, a new record! The recent evaluation is
sadly less victorious.

> The latest iteration failed en masse due to network timeouts between
> build farm nodes.

Yes, I described this issue here: https://issues.guix.gnu.org/48468 and
hopefully fixed it with https://issues.guix.gnu.org/48556.

I need to perform a reconfigure / deploy but I'm waiting to have a few
days ahead of me to fix potential issues.

> It lets us see that the Audacity build is not attempted due to the
> failure of wxwidgets. And in turn, the failure of webkitgtk and libsoup.
> However, there is no information (logs, duration) available about those
> failed builds.

When a build is cancelled ("Cancel pending builds" button), all
its dependents are automatically updated to "Failed dependency". It
looks like this is the case for https://ci.guix.gnu.org/build/515005/details.

> What can we do to clear all these "spurious" build failures and re-try
> building the branch? I already tried using the "restart all builds"
> button, but there were still a lot of spurious failures.

I think the easiest way to recover from this situation is to remove the
ungrafting specification and recreate it. It ensures that no build will
fail because of previously failing/cancelled build dependencies.

Note that when a build B is restarted in Cuirass ("Restart" button), any
other build B' depending from B, will automatically be restarted if B
becomes successful and all the other dependencies of B' are already
successful.

I used this mechanism to restart a few builds failing on the master
branch because of substitutes timeout. I hope that when the keep-alive
patchset will solve this situation.

Thanks,

Mathieu


^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: Ungrafting CI jobs
  2021-06-02 17:53   ` Mathieu Othacehe
@ 2021-06-03 20:22     ` Leo Famulari
  2021-06-04  6:34       ` Mathieu Othacehe
  0 siblings, 1 reply; 5+ messages in thread
From: Leo Famulari @ 2021-06-03 20:22 UTC (permalink / raw)
  To: Mathieu Othacehe; +Cc: guix-devel

On Wed, Jun 02, 2021 at 07:53:35PM +0200, Mathieu Othacehe wrote:
> I monitored a previous evaluation of the ungrafting Cuirass
> specification that was more successful, with more than 17000 builds
> performed in less than 24 hours, a new record! The recent evaluation is
> sadly less victorious.

Yeah, it was great for a while!

> I need to perform a reconfigure / deploy but I'm waiting to have a few
> days ahead of me to fix potential issues.

Okay. Can you let me know when you do? I'll delay the ungrafting builds
until then.


^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: Ungrafting CI jobs
  2021-06-03 20:22     ` Leo Famulari
@ 2021-06-04  6:34       ` Mathieu Othacehe
  0 siblings, 0 replies; 5+ messages in thread
From: Mathieu Othacehe @ 2021-06-04  6:34 UTC (permalink / raw)
  To: Leo Famulari; +Cc: guix-devel


Hey Leo,

> Okay. Can you let me know when you do? I'll delay the ungrafting builds
> until then.

I deployed the upgrade yesterday. I've been monitoring it for 12 hours
without any sign of substitutes timeout. I think you can proceed whenever
you want.

Thanks,

Mathieu


^ permalink raw reply	[flat|nested] 5+ messages in thread

end of thread, other threads:[~2021-06-04  6:35 UTC | newest]

Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-05-25 20:04 Ungrafting CI jobs Leo Famulari
2021-06-01 16:59 ` Leo Famulari
2021-06-02 17:53   ` Mathieu Othacehe
2021-06-03 20:22     ` Leo Famulari
2021-06-04  6:34       ` Mathieu Othacehe

Code repositories for project(s) associated with this public inbox

	https://git.savannah.gnu.org/cgit/guix.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).