From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp10.migadu.com ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id y4X7IY/I5mF8VwAAgWs5BA (envelope-from ) for ; Tue, 18 Jan 2022 15:02:55 +0100 Received: from aspmx1.migadu.com ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp10.migadu.com with LMTPS id 2O/OGI/I5mFLdAAAG6o9tA (envelope-from ) for ; Tue, 18 Jan 2022 15:02:55 +0100 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 1714338637 for ; Tue, 18 Jan 2022 15:02:55 +0100 (CET) Received: from localhost ([::1]:59628 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1n9p4Y-0005Yz-6Z for larch@yhetil.org; Tue, 18 Jan 2022 09:02:54 -0500 Received: from eggs.gnu.org ([209.51.188.92]:33102) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1n9o09-0002oX-R9 for guix-devel@gnu.org; Tue, 18 Jan 2022 07:54:22 -0500 Received: from [2607:f8b0:4864:20::12e] (port=40531 helo=mail-il1-x12e.google.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1n9o04-00067y-6J for guix-devel@gnu.org; Tue, 18 Jan 2022 07:54:15 -0500 Received: by mail-il1-x12e.google.com with SMTP id r15so470852ilj.7 for ; Tue, 18 Jan 2022 04:54:00 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=beadling-co-uk.20210112.gappssmtp.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=PpTlFsJ4xNJSMKukfMY4p9IyJS2CBMGQR4wJhu5GenE=; b=oG/mqYZQ/e1lbMD0k+iSxn4jcmuSr+sMT+ELDS0Vur7c1i0CaSpLZV/eS2Gkz/gS0G rFs5LStJX4G3QW4BhLudzHf//IQn9nnnvxMn3LE4ECdxF/Pl4yRbM0I6B/YoRpLKaO5W qmE18hW3YcOp2sGX80uPGhhLlkZ4lJOvRXxgwMkUfn1QxyxHHd5x7DvsLiCVufTFCC8a jWMqfNLSaPFxZYN/KmNlX9OGUJCWIB77eWXkl42uLowMNIaclyj8rlkiwKV3ODTUO4R8 8Mp0iLlOiReRNmJeA7zgyzWkMAle7iL8iiUGDL6bIH6/vD8UlCor0Bv0sj11SNEUkEH/ lsBQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=PpTlFsJ4xNJSMKukfMY4p9IyJS2CBMGQR4wJhu5GenE=; b=cXRFjfT0majJmM7s7sP31sSU1lx6cFBuJXoYaItshrYgTxuijIFPrXZ3ujgXRJvLqg UT8ASfX6vndsWBzzB9eGyp92bKiIpVNqZx91cMwqfLv9rbMuZUA6TZgwy907zLUtbB+f SOGafASYcMMl61mHTAQ6hiy0cBHM0+shB04IUyMIzQtzqYHq3AwdlgfLQ4X+qXPsvkzi 1vKioqpCUjvQdLSq3NhG9m0dS5gzwhYoA92OnjdRjqyDlLy+Z0L8iR5o44NpRAEKWgx9 fnkxDdHYufS3fz1KFAqT3/2sPXtFakshOaiJn4ikIS1j4uWSS6CTTwxCFokbL1TTEUv1 Pe8Q== X-Gm-Message-State: AOAM530wriwNMrv/CUsgio6lryj6ZGKVzxq4cD+4Zh9ZM43pNZl7fOb1 lb2I/r6vdhcqC+4Z8A2ssUV9zkj143k+j43CIlDqRg== X-Google-Smtp-Source: ABdhPJycQ/DbPE5F+1PYq0bCbMaue36XBZMzWtY4sv7+PW2Kt4GbRZK5//FtGXwRhnF4S04qAccIUNIN/Zk2oc6hHrg= X-Received: by 2002:a05:6e02:1a69:: with SMTP id w9mr13605294ilv.79.1642510439376; Tue, 18 Jan 2022 04:53:59 -0800 (PST) MIME-Version: 1.0 References: <877db6ge0o.fsf@beadling.co.uk> <874k697thf.fsf@elephly.net> <87r19d5pcw.fsf@elephly.net> <87k0exfj6l.fsf@beadling.co.uk> In-Reply-To: <87k0exfj6l.fsf@beadling.co.uk> From: Phil Beadling Date: Tue, 18 Jan 2022 12:53:48 +0000 Message-ID: Subject: Re: Parallel guix builds can trample? To: Ricardo Wurmus Content-Type: multipart/alternative; boundary="000000000000477f4805d5dac3fb" X-Host-Lookup-Failed: Reverse DNS lookup failed for 2607:f8b0:4864:20::12e (failed) Received-SPF: none client-ip=2607:f8b0:4864:20::12e; envelope-from=phil@beadling.co.uk; helo=mail-il1-x12e.google.com X-Spam_score_int: -10 X-Spam_score: -1.1 X-Spam_bar: - X-Spam_report: (-1.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HTML_MESSAGE=0.001, PDS_HP_HELO_NORDNS=0.001, RCVD_IN_DNSWL_NONE=-0.0001, RDNS_NONE=0.793, SPF_HELO_NONE=0.001, SPF_NONE=0.001 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: "guix-devel@gnu.org" Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: "Guix-devel" X-Migadu-Flow: FLOW_IN X-Migadu-Country: US ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1642514575; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post:dkim-signature; bh=PpTlFsJ4xNJSMKukfMY4p9IyJS2CBMGQR4wJhu5GenE=; b=sMHutBsFFNuLWTEp1kLlU2jctpPt98iTohjx67BLDGFYCjQ5ooE4WRLBNztWDaJo5D772u veBKg6fB/0RCpP95hJAAGg1S3n0XX+MG6bGpEuQ1kvnp0cf4LMxbPy1flFqjTlfQdDDUnR cczkdFwetyWlASHpxY3VrdXZSWhu8D0PAr5vM/pJpdqAppMPrVdQjI9uzHXRf57tZl4AaJ 3v5l6CUvU8DqfNP0aMo4tZkxe20GXseZ0hu9qj1KeUvZNAPISj0KGqNI1VhgJ7CojqsBSb x7Q2HfN8hI7ZA+b6Dysz1FXMF0xWn2J0fnrQ5djfaqFw2lK1hlC/B9q4tvvETA== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1642514575; a=rsa-sha256; cv=none; b=NAQWX9MRNWR3nr57gnvJutlgyioKHr2dwl+hg2tjpigSedX+B1i4XHyYKQjnmICrveAlDH qm6HtpT05Ibw/uKSGth/9jzo6gVk6tYpL/rUAUH3o6QExRNWmMkNCCtW4S4oXiBKYmzHJC ldVFlOS20FZUWt2D7A1ctfGRLQhKmkUCysGu3QhkKBDiJoK12atoSGGxnrUteczjobtzO/ UDmqQneTXqNi2fk6EiTS/o9JL1Ql65CAgemGunxp1NYFFs9ZZAThY21p8f4o9z6GyFShvJ hptERnUNJXxCjFWLSL2T4ky2HPKwMcA9FjAeyB8MWMVGvB5d1R3KPi1tG1/3Qw== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=pass header.d=beadling-co-uk.20210112.gappssmtp.com header.s=20210112 header.b="oG/mqYZQ"; dmarc=none; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org" X-Migadu-Spam-Score: -2.82 Authentication-Results: aspmx1.migadu.com; dkim=pass header.d=beadling-co-uk.20210112.gappssmtp.com header.s=20210112 header.b="oG/mqYZQ"; dmarc=none; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org" X-Migadu-Queue-Id: 1714338637 X-Spam-Score: -2.82 X-Migadu-Scanner: scn1.migadu.com X-TUID: YnK21d/v8dQG --000000000000477f4805d5dac3fb Content-Type: text/plain; charset="UTF-8" Hi all, The issue also exists when using --with-commit - see below for a refined test that makes it trivial to demonstrate with any package where the source is retrieved from git. On Tue, 18 Jan 2022 at 10:10, Phil wrote: > > > Philip Beadling writes: > > >No, and we probably should do, even in spite of this issue. We're going > >to repeat the experiment now using --with-commit, to confirm behaviour > there. > > Let's try and build 2 package variations - for this experiement we don't even have to run them at the same time: mkdir checkout-test cd checkout-test mkdir cache1 cache2 XDG_CACHE_HOME=`realpath ./cache1` guix build --with-commit=simm=*3981f60d81d3c9592d78b3e88545259713a93194 *simm XDG_CACHE_HOME=`realpath ./cache2` guix build --with-commit=simm=*ec1357c1fdf78a2ac1408d00c0c3e7d025ab9ade *simm Now look at the caches: $ ll cache1/guix/checkouts total 4.0K -rw-rw-r-- 1 phil.beadling phil.beadling 10 Jan 18 12:37 last-expiry-cleanup drwxrwxr-x 6 phil.beadling phil.beadling 162 Jan 18 12:37 lx4ipn27sk6uvu6b33kelt4tik65oianvl7b7yn5bpng7gpt4pyq drwxrwxr-x 5 phil.beadling phil.beadling 115 Jan 18 12:37 mqqlbc4ksftu2tkbnxydosm4j6mynxsh5nmhpq3cfrofkktkhexq drwxrwxr-x 5 phil.beadling phil.beadling 103 Jan 18 12:37 qqzuuoqxvqikivkpenqom43kzk2xr7nlntanbm6fpe7bci6kaenq drwxrwxr-x 4 phil.beadling phil.beadling 202 Jan 18 12:37 smvbhjf46cbaxxju6fsqj7t6ujktxy5i7vhuhzpvgjt2vhid2vqq drwxrwxr-x 5 phil.beadling phil.beadling 267 Jan 18 12:37 *xulvhb6agjrsf6x7wknutpn7wkt5i4nesk637czazvfxps357eoq* $ ll cache2/guix/checkouts total 4.0K -rw-rw-r-- 1 phil.beadling phil.beadling 10 Jan 18 12:38 last-expiry-cleanup drwxrwxr-x 6 phil.beadling phil.beadling 162 Jan 18 12:38 lx4ipn27sk6uvu6b33kelt4tik65oianvl7b7yn5bpng7gpt4pyq drwxrwxr-x 5 phil.beadling phil.beadling 115 Jan 18 12:38 mqqlbc4ksftu2tkbnxydosm4j6mynxsh5nmhpq3cfrofkktkhexq drwxrwxr-x 5 phil.beadling phil.beadling 103 Jan 18 12:39 qqzuuoqxvqikivkpenqom43kzk2xr7nlntanbm6fpe7bci6kaenq drwxrwxr-x 4 phil.beadling phil.beadling 202 Jan 18 12:38 smvbhjf46cbaxxju6fsqj7t6ujktxy5i7vhuhzpvgjt2vhid2vqq drwxrwxr-x 5 phil.beadling phil.beadling 267 Jan 18 12:38 *xulvhb6agjrsf6x7wknutpn7wkt5i4nesk637czazvfxps357eoq* $ Despite asking for different variants of the simm package, they are both stored under the same SHA. If they both used the same cache directory (the default) - they'd have overwritten each other. Thus it is not safe to run --with-commit or any other variation in parallel for the same user. To prove I've matched the right SHA - I can back-out the original commit ids provided to --with-commit: $ cd cache1/guix/checkouts/xulvhb6agjrsf6x7wknutpn7wkt5i4nesk637czazvfxps357eoq && git rev-parse HEAD && cd - *3981f60d81d3c9592d78b3e88545259713a93194* ~/checkout-test $ cd cache2/guix/checkouts/xulvhb6agjrsf6x7wknutpn7wkt5i4nesk637czazvfxps357eoq && git rev-parse HEAD && cd - *ec1357c1fdf78a2ac1408d00c0c3e7d025ab9ade* ~/checkout-test $ Whilst my original setup was a bit of a edge-case, I think this refined example demonstrates that the problem can occur with typical/reasonable use of guix switches? I can even think of how this could occur without the user running in parallel - for example if a package required via nth-order dependencies to build 2 versions of the same repo - it's possible that a single "guix build" command could cause the issue? Thoughts? --000000000000477f4805d5dac3fb Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi all,

The issue also exist= s when using --with-commit - see below for a refined test that makes it tri= vial to demonstrate with any package where the source is retrieved from git= .

On Tue, 18 Jan 2022 at 10:10, Phil <phil@beadling.co.uk> wrote:

> Philip Beadling <phil@beadling.co.uk> writes:
=C2=A0
>No, and we probably should do, even in spite of this issue.=C2=A0 We= 9;re going
>to repeat the experiment now using --with-commit, to confirm behaviour = there.


Let's try and build 2 package vari= ations - for this experiement we don't even have to run them at the sam= e time:

mkdir checkout-test
cd checkout-= test
mkdir cache1 cache2
XDG_CACHE_HOME=3D`realpath ./c= ache1` guix build --with-commit=3Dsimm=3D3981f60d81d3c9592d78b3e88545259= 713a93194 simm
XDG_CACHE_HOME=3D`realpath ./cache2` guix buil= d --with-commit=3Dsimm=3Dec1357c1fdf78a2ac1408d00c0c3e7d025ab9ade si= mm

Now look at the caches:

$ ll cache1/guix/checkouts

total 4.0K
-rw-rw-r-- 1 phil.beadlin= g phil.beadling =C2=A010 Jan 18 12:37 last-expiry-cleanup
drwxrwxr-x 6 p= hil.beadling phil.beadling 162 Jan 18 12:37 lx4ipn27sk6uvu6b33kelt4tik65oia= nvl7b7yn5bpng7gpt4pyq
drwxrwxr-x 5 phil.beadling phil.beadling 115 Jan 1= 8 12:37 mqqlbc4ksftu2tkbnxydosm4j6mynxsh5nmhpq3cfrofkktkhexq
drwxrwxr-x = 5 phil.beadling phil.beadling 103 Jan 18 12:37 qqzuuoqxvqikivkpenqom43kzk2x= r7nlntanbm6fpe7bci6kaenq
drwxrwxr-x 4 phil.beadling phil.beadling 202 Ja= n 18 12:37 smvbhjf46cbaxxju6fsqj7t6ujktxy5i7vhuhzpvgjt2vhid2vqq
drwxrwxr= -x 5 phil.beadling phil.beadling 267 Jan 18 12:37 xulvhb6agjrsf6x7wknutp= n7wkt5i4nesk637czazvfxps357eoq

$ ll cache2/guix/checkouts=

total 4.0K
-rw-rw-r-- 1 phil.beadling phil.beadling =C2=A010 Jan= 18 12:38 last-expiry-cleanup
drwxrwxr-x 6 phil.beadling phil.beadling 1= 62 Jan 18 12:38 lx4ipn27sk6uvu6b33kelt4tik65oianvl7b7yn5bpng7gpt4pyq
drw= xrwxr-x 5 phil.beadling phil.beadling 115 Jan 18 12:38 mqqlbc4ksftu2tkbnxyd= osm4j6mynxsh5nmhpq3cfrofkktkhexq
drwxrwxr-x 5 phil.beadling phil.beadlin= g 103 Jan 18 12:39 qqzuuoqxvqikivkpenqom43kzk2xr7nlntanbm6fpe7bci6kaenq
= drwxrwxr-x 4 phil.beadling phil.beadling 202 Jan 18 12:38 smvbhjf46cbaxxju6= fsqj7t6ujktxy5i7vhuhzpvgjt2vhid2vqq
drwxrwxr-x 5 phil.beadling phil.bead= ling 267 Jan 18 12:38 xulvhb6agjrsf6x7wknutpn7wkt5i4nesk637czazvfxps357e= oq

$

Despite asking for different varia= nts of the simm package, they are both stored under the same SHA.=C2=A0 If = they both used the same cache directory (the default) - they'd have ove= rwritten each other.=C2=A0 Thus it is not safe to run --with-commit or any = other variation in parallel for the same user.

To = prove I've matched the right SHA - I can back-out the original commit i= ds provided to --with-commit:

$ cd cache1/guix= /checkouts/xulvhb6agjrsf6x7wknutpn7wkt5i4nesk637czazvfxps357eoq && = git rev-parse HEAD && cd -

3981f60d81d3c9592d78b3e8854525= 9713a93194
~/checkout-test

$ cd cache2/guix/checkouts/xulvhb6= agjrsf6x7wknutpn7wkt5i4nesk637czazvfxps357eoq && git rev-parse HEAD= && cd -

ec1357c1fdf78a2ac1408d00c0c3e7d025ab9ade
= ~/checkout-test

$

Whilst my original se= tup was a bit of a edge-case, I think this refined example demonstrates tha= t the problem can occur with typical/reasonable use of guix switches?=C2=A0= I can even think of how this could occur without the user running in paral= lel - for example if a package required via nth-order dependencies to build= 2 versions of the same repo - it's possible that a single "guix b= uild" command could cause the issue?

Thoughts= ?


--000000000000477f4805d5dac3fb--