Parallel guix builds can trample?

Parallel guix builds can trample?

[Phil]

Hi all,

I'm seeing strange issues when running many "guix build" commands of the
same package simultaneously on the same linux account/server at the same
time.

Tracing this requires a lot of detective work, so for now I've omitted
the logs - my initial question is - am I doing something which is
obviously not going to work with Guix, or would people expect the below
use-case to work?

In my example I have ~12 clones of my private guix channel, sat in their
own directories under my user account.

I update the same single package on each channel locally with a
different git commit id and package version.  So 12 variations of the
same package, accessing different commit ids, in the same package's
source repo, and having different package versions in Guix.

I then call "guix build -K -L /path/to/each/local/clone package-name" 12
times, one for each clone, these run in parallel shell sessions.

This is all automated so each build is started within a fraction of
second - trying to reproduce this issue by hand is proving difficult,
and the issue is still sporadic in the automated system.

The logs all show that each of the ~12 channels receive a local update
with a unique and valid commit id and version for the changed package.

However what I'm seeing is that some of the builds are failing with the
error messages of other commit ids!

To be clear - the failure is not the surprise here, the builds are
getting mixed up and some are being trampled by what appears to be
either a race condition or stale state.  The surprise is that the
failures are being tied to the wrong commit ids which do not contain the
failures reported.

Use of "--keep-failed" means I can prove this unequivocally, where the
saved down /tmp/package-name.verson.drv.0 source code does not match the
source of the commit id stated within the updated package - which I can
see in the logs.  I can also show that the source code matches exactly
one of the other commit ids for one of the other clones, which was
expected to fail.

This happens sporadically, but we can reproduce the issue several times
a day.

I don't yet have 100% proof that the issue is happening inside Guix, but
I've ruled out most (but not all) other causes so far, and the
--keep-failed evidence tied with logs showing correct inputs is quite telling.

Have any other bugs ever been reported like this that people are aware
of?  Have other people ran multiple builds in Guix under one account at
the same time without issue?  Any advice on how trap the issue, given
it's hard to reproduce?

The problem as never been seen when we do each guix build in serial.

Apologies for the long and rather circumstantial e-mail!

Cheers,
Phil.

Re: Parallel guix builds can trample?

[Ricardo Wurmus]

Phil <phil@beadling.co.uk> writes:

> I update the same single package on each channel locally with a
> different git commit id and package version.  So 12 variations of the
> same package, accessing different commit ids, in the same package's
> source repo, and having different package versions in Guix.

Do they have different source hashes too?  The hash is what uniquely
identifies the source, not the version or commit.

-- 
Ricardo

Re: Parallel guix builds can trample?

[Philip Beadling]

[-- Attachment #1: Type: text/plain, Size: 1981 bytes --]

Thanks for reply Ricardo.

The source in each one will be slightly different and although I only specify commit id in the package, each package commit id will reference a different branch.

To give some context, each commit id relates to a different Pull Request branch. After my integration source branch is updated - the system then rebuilds any open PRs with the new integration update merged on top.  This checks that the PRs remain mergable against the heat of integration.

Because some PRs sit in a known broken state, it was possible to detect the problem.

I've seen the issue very sporadically for a few months, but at the moment a large number of open and broken PRs have revealed the problem to be more frequent.

Is there a way to output source hash in the log, to be sure of this?

One idea I had today is to try to reproduce the problem outside of my CI system, just by manually recreating 12 local channels, one with a known commit reference to a break in it, and have a shell script loop over, building each in the background, then waiting for all builds to terminate, and looking at their return codes.  If they all pass apart from the known broken clone, then repeat.  It won't solve the problem but will remove any other source of the problem, and can be left running in fast succession.



________________________________
From: Ricardo Wurmus <rekado@elephly.net>
Sent: Tuesday, 11 January 2022, 23:17
To: Phil
Cc: guix-devel@gnu.org
Subject: Re: Parallel guix builds can trample?


Phil <phil@beadling.co.uk> writes:

> I update the same single package on each channel locally with a
> different git commit id and package version.  So 12 variations of the
> same package, accessing different commit ids, in the same package's
> source repo, and having different package versions in Guix.

Do they have different source hashes too?  The hash is what uniquely
identifies the source, not the version or commit.

--
Ricardo


[-- Attachment #2: Type: text/html, Size: 4354 bytes --]

Re: Parallel guix builds can trample?

[Ricardo Wurmus]

Philip Beadling <phil@beadling.co.uk> writes:

> The source in each one will be slightly different and although I only
> specify commit id in the package

How are you using Guix with this?  Do you generate Guix package
expressions?  Do you use “guix build --with-commit”?

If the hash in the “source” field corresponds to source code in the
store then it doesn’t matter what the commit id or version string says.
The hash identifies the source.

-- 
Ricardo

Re: Parallel guix builds can trample?

[Phil]

Hi - more details below.

Ricardo Wurmus writes:

>
> How are you using Guix with this?  Do you generate Guix package
> expressions?  Do you use “guix build --with-commit”?
>

The situation is like this - if we had a directory of clones of my
channel:

- pr-1
- pr-2
- pr-3
- pr-4
... and so on

Initially all the clones are taken from the master branch of my
channel and are all identical - but we change the version and commit to
match the head of each PR branch as per below.

Each clone looks like this:
- pr-1
      - my-package.scm
- pr-2
      - my-package.scm
and so on....

Each my-package.scm has a package like below - the inital packages are all
identical, but my system effectively seds the version and commit values
like the below.  These values are never committed back to master they
are used only as local channels to build each PR to test each build
still passes.

(define-public my-package
  (package
    (name "my-package")
    (version "this-is-different-for-each-pr")  ;; replace master version
    (source
      (git-checkout
        (url "ssh://same@repo:7999/same/repo.git")
        (commit "this-is-different-for-each-pr") ;; replace master version 
everything else remains the same in the package....


At this point we have lots of local channels referencing different commits, in
the same package, ready to build - so I spawn them all simultaneously -
the equivalent pseudo-shell that I will mock up today would be:

# define some sort of return code array:
RC=[]

for dir in pr-dirs
  RC[${dir}]=`guix build -K -L ${dir} my-package & 2>&1 > /tmp/${dir}.log`  # note the ampersand
wait

for rc in $RC
  if $rc.value != 0:
    report the failure of build $rc.key

What I'm seeing occasionally is that the logs and return code for say directory pr-1
and appearing in the guix build for pr-3 or pr-6 instead.

We know this becuse the code is different enough in pr-1 that it's logs
are unique across all the PRs.  We can also check the source code if the
build fails using --keep-failed to show it doesn't match the commit id
in the package used to build it.

Hopefully that makes sense?  I can post the actual shell script once
I've written the mock.

Re: Parallel guix builds can trample?

[Phil Beadling]

[-- Attachment #1: Type: text/plain, Size: 6552 bytes --]

Hi Ricardo, all,



I think we’ve worked out what the issue is, and have a proposed workaround,
and perhaps a case for solving the problem in Guix itself (depending on
what you people think!).



The issue is that despite each build being performed in its own isolated
container, these containers are fed by the same per-user cached source
directory.  In the case where *different* versions of the *same* repo are
built at once, this results in a race condition.



In our case we have one Linux account that does a lot of automated Guix
builds for us.



One example is this account watches our source control and automatically
rebuilds all outstanding Pull Requests (PRs) on a repo, after a separate
successful merge to our integration branch.  PRs are uniquely identified as
monotonically increasing PR numbers eg, PR-1, PR-2, PR-3 and so on.  Each
is a different branch on the same Repo with slightly different candidate
changes in it.  They are automatically kept up to date with the integration
branch.



To do this our watcher fires off (near) instantaneously dozens of guix
builds, each with their own local channel customized for the PR it is
building.  Doing them in parallel is important to make the system usably
responsive.



Each fired process does this:

   - Clone the channel containing the package into a local directory
   - Modify the commit id of the package to the new merged head of the PR
   - Modify the package version to some dummy version containing the PR
   number
   - Build the modified package using the local channel
   - Report the result (the build is effectively discarded; it is never
   used for anything)



What we think is happening is the following:



   - For each build that is kicked off in quick succession the local cache
   of the repo required updated by *update-cached-checkou*t
      -
      https://github.com/guix-mirror/guix/blob/9f526f5dad5f4af69d158c50369e182305147f3b/guix/git.scm#L476
      -
      https://github.com/guix-mirror/guix/blob/9f526f5dad5f4af69d158c50369e182305147f3b/guix/git.scm#L279
   - The problem with this is because each version is using the same cached
   repo --- before one has a chance to take a copy of the updated checkout,
   that checkout can be changed by a separate build process



Thus there is a race condition in this scenario.  We can provide a longer
test script to demo this if required – it’s quite straightforward to
reproduce just with a bash script, now we know what is causing it.



Our workaround has been to change XDG_CACHE_HOME for each PR build we do.
But this is a bit unsatisfactory as it effects processes beyond Guix – it
casts too wide of a net, but it does resolve the problem for the time being.



Do people think this is enough of an issue to make a switch available in
Guix to prevent sharing of cached clones?  This would be easy enough to
implement – a crude solution would be that each cache directory name would
simply be generated using a SHA of a string which includes the PID or
similar to ensure a unique name, and because it is never going to be reused
it could be deleted immediately after the build.



Whilst this is unlikely to happen at the console, as people script guix
build use-cases to fit their own problems (in particular building lots of
variations of a single piece of software) – I can see this causing a
headache?  I think at least the manual should make it clear that you cannot
build 2 packages referencing the same repo at the same time with the same
user (unless I’ve missed this bit I don’t think it’s made explicitly
clear?).  An even simpler change would be introduce a lock file that
refused the 2nd build and at least preventing the race condition happening,
and ensuring referential transparency, or simpler still just placed a
warning on stderr?



If people are amenable to adding a switch or other config option, we’d be
happy to look writing the patch?


Any thoughts/comments/advice?


Cheers!
Phil.


On Wed, 12 Jan 2022 at 09:37, Phil <phil@beadling.co.uk> wrote:

> Hi - more details below.
>
> Ricardo Wurmus writes:
>
> >
> > How are you using Guix with this?  Do you generate Guix package
> > expressions?  Do you use “guix build --with-commit”?
> >
>
> The situation is like this - if we had a directory of clones of my
> channel:
>
> - pr-1
> - pr-2
> - pr-3
> - pr-4
> ... and so on
>
> Initially all the clones are taken from the master branch of my
> channel and are all identical - but we change the version and commit to
> match the head of each PR branch as per below.
>
> Each clone looks like this:
> - pr-1
>       - my-package.scm
> - pr-2
>       - my-package.scm
> and so on....
>
> Each my-package.scm has a package like below - the inital packages are all
> identical, but my system effectively seds the version and commit values
> like the below.  These values are never committed back to master they
> are used only as local channels to build each PR to test each build
> still passes.
>
> (define-public my-package
>   (package
>     (name "my-package")
>     (version "this-is-different-for-each-pr")  ;; replace master version
>     (source
>       (git-checkout
>         (url "ssh://same@repo:7999/same/repo.git")
>         (commit "this-is-different-for-each-pr") ;; replace master version
> everything else remains the same in the package....
>
>
> At this point we have lots of local channels referencing different
> commits, in
> the same package, ready to build - so I spawn them all simultaneously -
> the equivalent pseudo-shell that I will mock up today would be:
>
> # define some sort of return code array:
> RC=[]
>
> for dir in pr-dirs
>   RC[${dir}]=`guix build -K -L ${dir} my-package & 2>&1 >
> /tmp/${dir}.log`  # note the ampersand
> wait
>
> for rc in $RC
>   if $rc.value != 0:
>     report the failure of build $rc.key
>
> What I'm seeing occasionally is that the logs and return code for say
> directory pr-1
> and appearing in the guix build for pr-3 or pr-6 instead.
>
> We know this becuse the code is different enough in pr-1 that it's logs
> are unique across all the PRs.  We can also check the source code if the
> build fails using --keep-failed to show it doesn't match the commit id
> in the package used to build it.
>
> Hopefully that makes sense?  I can post the actual shell script once
> I've written the mock.
>

[-- Attachment #2: Type: text/html, Size: 13220 bytes --]

Re: Parallel guix builds can trample?

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 795 bytes --]

Hi,

Phil Beadling schreef op ma 17-01-2022 om 17:23 [+0000]:
> For each build that is kicked off in quick succession the local cache
> of the repo required updated by update-cached-checkout
>  * https://github.com/guix-
> mirror/guix/blob/9f526f5dad5f4af69d158c50369e182305147f3b/guix/git.sc
> m#L476
>  * https://github.com/guix-
> mirror/guix/blob/9f526f5dad5f4af69d158c50369e182305147f3b/guix/git.sc
> m#L279


Maybe 'latest-repository-commit' and 'update-cached-checkout' commit
can be modified to not use 'switch-to-ref', and instead directly ask
libgit ‘what's the tree structure of commit cabba9e’ and call a
procedure like 'add-file-tree-to-store'.  That would avoid lock files,
creating separate directories for concurrent checkouts, ...

Greetings,
Maxime.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

Re: Parallel guix builds can trample?

[Phil]

Maxime Devos writes:

> Maybe 'latest-repository-commit' and 'update-cached-checkout' commit
> can be modified to not use 'switch-to-ref', and instead directly ask
> libgit ‘what's the tree structure of commit cabba9e’ and call a
> procedure like 'add-file-tree-to-store'.  That would avoid lock files,
> creating separate directories for concurrent checkouts, ...

Thanks Maxime - I'll take a look at this as an option.

One option I was thinking of is to make this SHA call dependent on the
branch as well as the URL:
https://github.com/guix-mirror/guix/blob/9f526f5dad5f4af69d158c50369e182305147f3b/guix/git.scm#L131-L133

Technically this still leaves a race condition if someone were to do
parallel builds on the same branch however, so to completely solve the
problem I think we'd have make the SHA call on repo+commit-id.

This obviously limit the utility of the cache.

To mitigate performance concerns, I think it's possible to set the
branches and depth in git such that we only clone the bare-bones only?.

Re: Parallel guix builds can trample?

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 361 bytes --]

Phil schreef op di 18-01-2022 om 09:28 [+0000]:
> To mitigate performance concerns, I think it's possible to set the
> branches and depth in git such that we only clone the bare-bones only?.

IIUC libgit2 and the Guile bindings guile-git do not currently support
shallow clones (depth, etc.).  Bare-bones should be possible though.

Greetings,
Maxime.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

Re: Parallel guix builds can trample?

[Ludovic Courtès]

Hi,

Maxime Devos <maximedevos@telenet.be> skribis:

> Phil Beadling schreef op ma 17-01-2022 om 17:23 [+0000]:
>> For each build that is kicked off in quick succession the local cache
>> of the repo required updated by update-cached-checkout
>>  * https://github.com/guix-
>> mirror/guix/blob/9f526f5dad5f4af69d158c50369e182305147f3b/guix/git.sc
>> m#L476
>>  * https://github.com/guix-
>> mirror/guix/blob/9f526f5dad5f4af69d158c50369e182305147f3b/guix/git.sc
>> m#L279
>
>
> Maybe 'latest-repository-commit' and 'update-cached-checkout' commit
> can be modified to not use 'switch-to-ref', and instead directly ask
> libgit ‘what's the tree structure of commit cabba9e’ and call a
> procedure like 'add-file-tree-to-store'.  That would avoid lock files,
> creating separate directories for concurrent checkouts, ...

That would be ideal.  In Guix ‘write-file-tree’ already supports some
sort of virtual file systems, so it should be possible to use it to feed
the contents of the Git tree to the store.

Ludo’.

Re: Parallel guix builds can trample?

[Phil]

Ricardo Wurmus writes:

> Philip Beadling <phil@beadling.co.uk> writes:
>
>> The source in each one will be slightly different and although I only
>> specify commit id in the package
>
> How are you using Guix with this?  Do you generate Guix package
> expressions?  Do you use “guix build --with-commit”?

No, and we probably should do, even in spite of this issue.  We're going
to repeat the experiment now using --with-commit, to confirm behaviour there.

Thanks for reminding me about the switch!

Re: Parallel guix builds can trample?

[Phil Beadling]

[-- Attachment #1: Type: text/plain, Size: 3415 bytes --]

Hi all,

The issue also exists when using --with-commit - see below for a refined
test that makes it trivial to demonstrate with any package where the source
is retrieved from git.

On Tue, 18 Jan 2022 at 10:10, Phil <phil@beadling.co.uk> wrote:

>
> > Philip Beadling <phil@beadling.co.uk> writes:
>


> >No, and we probably should do, even in spite of this issue.  We're going
> >to repeat the experiment now using --with-commit, to confirm behaviour
> there.
>
>
Let's try and build 2 package variations - for this experiement we don't
even have to run them at the same time:

mkdir checkout-test
cd checkout-test
mkdir cache1 cache2
XDG_CACHE_HOME=`realpath ./cache1` guix build
--with-commit=simm=*3981f60d81d3c9592d78b3e88545259713a93194
*simm
XDG_CACHE_HOME=`realpath ./cache2` guix build
--with-commit=simm=*ec1357c1fdf78a2ac1408d00c0c3e7d025ab9ade
*simm

Now look at the caches:

$ ll cache1/guix/checkouts

total 4.0K
-rw-rw-r-- 1 phil.beadling phil.beadling  10 Jan 18 12:37
last-expiry-cleanup
drwxrwxr-x 6 phil.beadling phil.beadling 162 Jan 18 12:37
lx4ipn27sk6uvu6b33kelt4tik65oianvl7b7yn5bpng7gpt4pyq
drwxrwxr-x 5 phil.beadling phil.beadling 115 Jan 18 12:37
mqqlbc4ksftu2tkbnxydosm4j6mynxsh5nmhpq3cfrofkktkhexq
drwxrwxr-x 5 phil.beadling phil.beadling 103 Jan 18 12:37
qqzuuoqxvqikivkpenqom43kzk2xr7nlntanbm6fpe7bci6kaenq
drwxrwxr-x 4 phil.beadling phil.beadling 202 Jan 18 12:37
smvbhjf46cbaxxju6fsqj7t6ujktxy5i7vhuhzpvgjt2vhid2vqq
drwxrwxr-x 5 phil.beadling phil.beadling 267 Jan 18 12:37
*xulvhb6agjrsf6x7wknutpn7wkt5i4nesk637czazvfxps357eoq*

$ ll cache2/guix/checkouts

total 4.0K
-rw-rw-r-- 1 phil.beadling phil.beadling  10 Jan 18 12:38
last-expiry-cleanup
drwxrwxr-x 6 phil.beadling phil.beadling 162 Jan 18 12:38
lx4ipn27sk6uvu6b33kelt4tik65oianvl7b7yn5bpng7gpt4pyq
drwxrwxr-x 5 phil.beadling phil.beadling 115 Jan 18 12:38
mqqlbc4ksftu2tkbnxydosm4j6mynxsh5nmhpq3cfrofkktkhexq
drwxrwxr-x 5 phil.beadling phil.beadling 103 Jan 18 12:39
qqzuuoqxvqikivkpenqom43kzk2xr7nlntanbm6fpe7bci6kaenq
drwxrwxr-x 4 phil.beadling phil.beadling 202 Jan 18 12:38
smvbhjf46cbaxxju6fsqj7t6ujktxy5i7vhuhzpvgjt2vhid2vqq
drwxrwxr-x 5 phil.beadling phil.beadling 267 Jan 18 12:38
*xulvhb6agjrsf6x7wknutpn7wkt5i4nesk637czazvfxps357eoq*

$

Despite asking for different variants of the simm package, they are both
stored under the same SHA.  If they both used the same cache directory (the
default) - they'd have overwritten each other.  Thus it is not safe to run
--with-commit or any other variation in parallel for the same user.

To prove I've matched the right SHA - I can back-out the original commit
ids provided to --with-commit:

$ cd
cache1/guix/checkouts/xulvhb6agjrsf6x7wknutpn7wkt5i4nesk637czazvfxps357eoq
&& git rev-parse HEAD && cd -

*3981f60d81d3c9592d78b3e88545259713a93194*
~/checkout-test

$ cd
cache2/guix/checkouts/xulvhb6agjrsf6x7wknutpn7wkt5i4nesk637czazvfxps357eoq
&& git rev-parse HEAD && cd -

*ec1357c1fdf78a2ac1408d00c0c3e7d025ab9ade*
~/checkout-test

$

Whilst my original setup was a bit of a edge-case, I think this refined
example demonstrates that the problem can occur with typical/reasonable use
of guix switches?  I can even think of how this could occur without the
user running in parallel - for example if a package required via nth-order
dependencies to build 2 versions of the same repo - it's possible that a
single "guix build" command could cause the issue?

Thoughts?

[-- Attachment #2: Type: text/html, Size: 4457 bytes --]