From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp11.migadu.com ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id GJ6zMY983mH2IQAAgWs5BA (envelope-from ) for ; Wed, 12 Jan 2022 08:00:31 +0100 Received: from aspmx1.migadu.com ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp11.migadu.com with LMTPS id SEQmL4983mGVeQEA9RJhRA (envelope-from ) for ; Wed, 12 Jan 2022 08:00:31 +0100 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 392163B2A6 for ; Wed, 12 Jan 2022 08:00:31 +0100 (CET) Received: from localhost ([::1]:53632 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1n7XcT-0000SC-Al for larch@yhetil.org; Wed, 12 Jan 2022 02:00:29 -0500 Received: from eggs.gnu.org ([209.51.188.92]:60678) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1n7Xc7-0000QU-Jf for guix-devel@gnu.org; Wed, 12 Jan 2022 02:00:07 -0500 Received: from [2a00:1450:4864:20::52a] (port=38860 helo=mail-ed1-x52a.google.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1n7Xc5-0006ey-Dw for guix-devel@gnu.org; Wed, 12 Jan 2022 02:00:07 -0500 Received: by mail-ed1-x52a.google.com with SMTP id u21so5953304edd.5 for ; Tue, 11 Jan 2022 23:00:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=beadling-co-uk.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:thread-topic:thread-index:date:message-id :references:in-reply-to:accept-language:content-language :mime-version; bh=fDpRCSF3ac/It8laF4rBXvoeJLjbXrICiie3WLITVmI=; b=BPTyVkyBFNJEE/N4vb+sCyhXibISMmUPtyBdUN9oIAOA2OimgYYuN82AwtCkISYilz sCRG6pLaGolaSAPk4CsAZqt1nGA8ioRU+7we0lGQMe21eRSvbO99baGiDKn9UdsdKKDZ 6zIuVvnLhx8uTWGTNUkX1+a+mrxMPI43vp5oPD4vieGjOr84rM1QXylv+iXWgR033OuS J7sJOUsmhWgHQjTlVTLWTf1bvCVoSu3h2mZO0vHVPZA94twR4x2l9+6x6amQnY50D9qg AnBirB/V9Pvd2xBRcwMr8mvPxGApUvLFO9h1QjB6/fsVs9tp3awmAOgul2w+ChyKbKan 1Fhw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:thread-topic:thread-index :date:message-id:references:in-reply-to:accept-language :content-language:mime-version; bh=fDpRCSF3ac/It8laF4rBXvoeJLjbXrICiie3WLITVmI=; b=am9jR1Eh7DQl3tAz9cTsKCo4YWJRQnizWLKam+eBhVCZEvvbSQCwAUzH1Q6rVdMu35 2JZP1zLty3zB92uVPgoJ3L3qKkbIbYLQq9Fa/P+detmRZWYZfw6n8EkczHiTdNPrfBQf 0Fof6oOl3i+bqM8MDIa9bC4EDZBdedGFhwOJ1rSgdgTopHfluLq1sPdScMpKZ9CVir2J c6b2PK4NHApU8zWU9Yorhsr4CRqZmhiH+HETWqgODicLIM/Cmgnb65j2Ks2DY5HpaN5V mW0f4/ygz787xX0Afpw0z3drHxeITFX1YyQMpOYDsKegXFSBXHn5jhiP3emWhr/dPGKa rG0A== X-Gm-Message-State: AOAM532rPG6tqGmxiaipEmcvSq9B7ldNYjZZfzRld3cnThSconwMI7UK VC4ohXcCilv0PusZ1zA4iLJeM0tmw3W0FwRi X-Google-Smtp-Source: ABdhPJzQLinTEvZbRx/qcu5Q7s1Me3+FRz3uK8TIfpSD0q91m8TB8U55uTN/nIxnIvjRs6rIIaUmuw== X-Received: by 2002:aa7:de82:: with SMTP id j2mr7490227edv.389.1641970803266; Tue, 11 Jan 2022 23:00:03 -0800 (PST) Received: from LO0P265MB5941.GBRP265.PROD.OUTLOOK.COM ([2603:1026:c06:1070::5]) by smtp.gmail.com with ESMTPSA id d1sm4288356ejo.176.2022.01.11.23.00.02 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 11 Jan 2022 23:00:02 -0800 (PST) From: Philip Beadling To: Ricardo Wurmus Subject: Re: Parallel guix builds can trample? Thread-Topic: Parallel guix builds can trample? Thread-Index: AWZmb2U2kvYCALVeCWHMLp8KTu+Fm2ZmZnQ54A7tAGM= X-MS-Exchange-MessageSentRepresentingType: 1 Date: Wed, 12 Jan 2022 07:00:01 +0000 Message-ID: References: <877db6ge0o.fsf@beadling.co.uk> <874k697thf.fsf@elephly.net> In-Reply-To: <874k697thf.fsf@elephly.net> Accept-Language: en-GB, en-US Content-Language: en-GB X-MS-Has-Attach: X-MS-Exchange-Organization-SCL: -1 X-MS-TNEF-Correlator: X-MS-Exchange-Organization-RecordReviewCfmType: 0 Content-Type: multipart/alternative; boundary="_000_LO0P265MB594168298E99C9A2D880E4B1F9529LO0P265MB5941GBRP_" MIME-Version: 1.0 X-Host-Lookup-Failed: Reverse DNS lookup failed for 2a00:1450:4864:20::52a (failed) Received-SPF: none client-ip=2a00:1450:4864:20::52a; envelope-from=phil@beadling.co.uk; helo=mail-ed1-x52a.google.com X-Spam_score_int: -10 X-Spam_score: -1.1 X-Spam_bar: - X-Spam_report: (-1.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_NONE=-0.0001, RDNS_NONE=0.793, SPF_HELO_NONE=0.001, SPF_NONE=0.001 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: "guix-devel@gnu.org" Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: "Guix-devel" X-Migadu-Flow: FLOW_IN X-Migadu-Country: US ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1641970831; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post:dkim-signature; bh=fDpRCSF3ac/It8laF4rBXvoeJLjbXrICiie3WLITVmI=; b=YXNOBDIiQ9SYO+5szZVyHmDIhl8YMAFlnlVzgLEZW/IuU71Nzzjw0wNkC74WVyYzXn9Sos pAEUH334g8iuSM9pe59+BvsSsSphA7AwVAecRgvtnOgS0yczEy/sHmdGezQKhCFZbpJ/3K IW/vtpdzKGFrZ8gIH+B/NhWeQuTrQ56reqa+ZDX2H860K5OZOXYCHnA8FqJ1SqvIYvjfta h8wwxBdS3BzNXSTvEYVRW/HWJes/0AX674TvK4F1x44XUa3zeZPPAjE6CmgWYJdkCcNDqY UOWvzycSQv1HqSn3p5w3a+iqBNFkFuUBrToiPs71JwR9zOdzZe56H7dJ2zQWQg== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1641970831; a=rsa-sha256; cv=none; b=k5s+Xcx8wFafNpBPTs/yIP+popfQJ/plw4AJF+rplYtiCO8pH9fdRZ8q8XfG8Eg9tOPkPr tj3dBiWN2Nr12wgp3EsbTo0GQsBKcqWSy60XTtGWskOHpXdv4czZAiQbG7F3D+eQRx41Hd 3fC/FF4jNFG8BAfy1IR4M3JaUyauwtCJNc+yxcbusBgff8YVOgwP3HW+BAdqCtBvs2+YuT BMs2ew3o3CoRw/e7wHjq3jwoOzUoqaQjuI76oXOOgtrpQyWlHSzSbxc/9ZHYTtrehmVpO3 Xq0fWCuvCeLx4euQYBLckVbxmJmaCx/RfqPqRHlki8xH8BfcWnCwIqfYW/8SeA== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=pass header.d=beadling-co-uk.20210112.gappssmtp.com header.s=20210112 header.b=BPTyVkyB; dmarc=none; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org" X-Migadu-Spam-Score: -4.42 Authentication-Results: aspmx1.migadu.com; dkim=pass header.d=beadling-co-uk.20210112.gappssmtp.com header.s=20210112 header.b=BPTyVkyB; dmarc=none; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org" X-Migadu-Queue-Id: 392163B2A6 X-Spam-Score: -4.42 X-Migadu-Scanner: scn0.migadu.com X-TUID: zLZeo/Q84Kv5 --_000_LO0P265MB594168298E99C9A2D880E4B1F9529LO0P265MB5941GBRP_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Thanks for reply Ricardo. The source in each one will be slightly different and although I only speci= fy commit id in the package, each package commit id will reference a differ= ent branch. To give some context, each commit id relates to a different Pull Request br= anch. After my integration source branch is updated - the system then rebui= lds any open PRs with the new integration update merged on top. This check= s that the PRs remain mergable against the heat of integration. Because some PRs sit in a known broken state, it was possible to detect the= problem. I've seen the issue very sporadically for a few months, but at the moment a= large number of open and broken PRs have revealed the problem to be more f= requent. Is there a way to output source hash in the log, to be sure of this? One idea I had today is to try to reproduce the problem outside of my CI sy= stem, just by manually recreating 12 local channels, one with a known commi= t reference to a break in it, and have a shell script loop over, building e= ach in the background, then waiting for all builds to terminate, and lookin= g at their return codes. If they all pass apart from the known broken clon= e, then repeat. It won't solve the problem but will remove any other sourc= e of the problem, and can be left running in fast succession. ________________________________ From: Ricardo Wurmus Sent: Tuesday, 11 January 2022, 23:17 To: Phil Cc: guix-devel@gnu.org Subject: Re: Parallel guix builds can trample? Phil writes: > I update the same single package on each channel locally with a > different git commit id and package version. So 12 variations of the > same package, accessing different commit ids, in the same package's > source repo, and having different package versions in Guix. Do they have different source hashes too? The hash is what uniquely identifies the source, not the version or commit. -- Ricardo --_000_LO0P265MB594168298E99C9A2D880E4B1F9529LO0P265MB5941GBRP_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable
Thanks for reply Ricardo. 

The source in each one will be slightly different and although I only speci= fy commit id in the package, each package commit id will reference a differ= ent branch.

To give some context, each commit id relates to a different Pull Request br= anch. After my integration source branch is updated - the system then rebui= lds any open PRs with the new integration update merged on top.  This = checks that the PRs remain mergable against the heat of integration. 

Because some PRs sit in a known broken state, it was possible to detect the= problem.

I've seen the issue very sporadically for a few months, but at the moment a= large number of open and broken PRs have revealed the problem to be more f= requent.

Is there a way to output source hash in the log, to be sure of this? <= /div>

One idea I had today is to try to reproduce the problem outside of my CI sy= stem, just by manually recreating 12 local channels, one with a known commi= t reference to a break in it, and have a shell script loop over, building e= ach in the background, then waiting for all builds to terminate, and looking at their return codes.  If t= hey all pass apart from the known broken clone, then repeat.  It won't= solve the problem but will remove any other source of the problem, and can= be left running in fast succession. 



From: Ricardo Wurmus <rekado@= elephly.net>
Sent: Tuesday, 11 January 2022, 23:17
To: Phil
Cc: guix-devel@gnu.org
Subject: Re: Parallel guix builds can trample?


Phil <phil@beadling.co.uk> writes:

> I update the same single package on each channel locally with a
> different git commit id and package version.  So 12 variations of= the
> same package, accessing different commit ids, in the same package's > source repo, and having different package versions in Guix.

Do they have different source hashes too?  The hash is what uniquely identifies the source, not the version or commit.

--
Ricardo

--_000_LO0P265MB594168298E99C9A2D880E4B1F9529LO0P265MB5941GBRP_--