unofficial mirror of guix-devel@gnu.org 
 help / color / mirror / code / Atom feed
* Parallel guix builds can trample?
@ 2022-01-11 21:26 Phil
  2022-01-11 23:16 ` Ricardo Wurmus
  0 siblings, 1 reply; 12+ messages in thread
From: Phil @ 2022-01-11 21:26 UTC (permalink / raw)
  To: guix-devel

Hi all,

I'm seeing strange issues when running many "guix build" commands of the
same package simultaneously on the same linux account/server at the same
time.

Tracing this requires a lot of detective work, so for now I've omitted
the logs - my initial question is - am I doing something which is
obviously not going to work with Guix, or would people expect the below
use-case to work?

In my example I have ~12 clones of my private guix channel, sat in their
own directories under my user account.

I update the same single package on each channel locally with a
different git commit id and package version.  So 12 variations of the
same package, accessing different commit ids, in the same package's
source repo, and having different package versions in Guix.

I then call "guix build -K -L /path/to/each/local/clone package-name" 12
times, one for each clone, these run in parallel shell sessions.

This is all automated so each build is started within a fraction of
second - trying to reproduce this issue by hand is proving difficult,
and the issue is still sporadic in the automated system.

The logs all show that each of the ~12 channels receive a local update
with a unique and valid commit id and version for the changed package.

However what I'm seeing is that some of the builds are failing with the
error messages of other commit ids!

To be clear - the failure is not the surprise here, the builds are
getting mixed up and some are being trampled by what appears to be
either a race condition or stale state.  The surprise is that the
failures are being tied to the wrong commit ids which do not contain the
failures reported.

Use of "--keep-failed" means I can prove this unequivocally, where the
saved down /tmp/package-name.verson.drv.0 source code does not match the
source of the commit id stated within the updated package - which I can
see in the logs.  I can also show that the source code matches exactly
one of the other commit ids for one of the other clones, which was
expected to fail.

This happens sporadically, but we can reproduce the issue several times
a day.

I don't yet have 100% proof that the issue is happening inside Guix, but
I've ruled out most (but not all) other causes so far, and the
--keep-failed evidence tied with logs showing correct inputs is quite telling.

Have any other bugs ever been reported like this that people are aware
of?  Have other people ran multiple builds in Guix under one account at
the same time without issue?  Any advice on how trap the issue, given
it's hard to reproduce?

The problem as never been seen when we do each guix build in serial.

Apologies for the long and rather circumstantial e-mail!

Cheers,
Phil.


^ permalink raw reply	[flat|nested] 12+ messages in thread

end of thread, other threads:[~2022-01-18 15:13 UTC | newest]

Thread overview: 12+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-01-11 21:26 Parallel guix builds can trample? Phil
2022-01-11 23:16 ` Ricardo Wurmus
2022-01-12  7:00   ` Philip Beadling
2022-01-12  8:27     ` Ricardo Wurmus
2022-01-12  9:37       ` Phil
2022-01-17 17:23         ` Phil Beadling
2022-01-17 17:44           ` Maxime Devos
2022-01-18  9:28             ` Phil
2022-01-18  9:36               ` Maxime Devos
2022-01-18 14:59             ` Ludovic Courtès
2022-01-18 10:10       ` Phil
2022-01-18 12:53         ` Phil Beadling

Code repositories for project(s) associated with this public inbox

	https://git.savannah.gnu.org/cgit/guix.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).