From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp11.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id MGtgE62h3mEJ9gAAgWs5BA (envelope-from ) for ; Wed, 12 Jan 2022 10:38:53 +0100 Received: from aspmx1.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp11.migadu.com with LMTPS id gJy0EK2h3mEPXgAA9RJhRA (envelope-from ) for ; Wed, 12 Jan 2022 10:38:53 +0100 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id A95C3465B2 for ; Wed, 12 Jan 2022 10:38:52 +0100 (CET) Received: from localhost ([::1]:48874 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1n7a5j-0006Ch-2A for larch@yhetil.org; Wed, 12 Jan 2022 04:38:51 -0500 Received: from eggs.gnu.org ([209.51.188.92]:32998) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1n7a4x-0006BV-Qf for guix-devel@gnu.org; Wed, 12 Jan 2022 04:38:04 -0500 Received: from [2a00:1450:4864:20::32c] (port=36727 helo=mail-wm1-x32c.google.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1n7a4r-00042X-Tn for guix-devel@gnu.org; Wed, 12 Jan 2022 04:38:03 -0500 Received: by mail-wm1-x32c.google.com with SMTP id n19-20020a7bc5d3000000b003466ef16375so3052132wmk.1 for ; Wed, 12 Jan 2022 01:37:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=beadling-co-uk.20210112.gappssmtp.com; s=20210112; h=references:user-agent:from:to:cc:subject:in-reply-to:message-id :date:mime-version:content-transfer-encoding; bh=OpGH2IdMGNjt7PufobCREVUCeYsx60gv/ob6dkHZdug=; b=eWDpAYVArFu084zLuOCxwzn6LfGuIeXJDoH3+HMxNFJRHf2aX5kL/CN4vBhyfnzk69 Xtt2LSzY6looUxVfRW5vNaCZkRiHSZ0k+Wx1XcQWEikwGYbk/x+K6vG7xjbmJEyYJceG 2lD2yw+78TmpIFNEBYr6mHvztwx2X/FSQkwTp+f32/avNywuEGen6kGkczCh7ck2bRdO WWjE5Hacecm7+IzvtY19mDQo1GLPNyrUkzoFsNFwoqW1wp5jA7TfMC19OiI8OeQF3gVT GOvlz5rMO3TKeUx9opF80U3O2paQYJT+rk4o80wamH99DzzuiT9UddxaAkprk5EI2XS5 ASiA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:references:user-agent:from:to:cc:subject :in-reply-to:message-id:date:mime-version:content-transfer-encoding; bh=OpGH2IdMGNjt7PufobCREVUCeYsx60gv/ob6dkHZdug=; b=UuUGuNOqWoq2uIojYMLY1BG35q8ApOXd1w8NhW/kzTmFXFfzxTLz57U5AUV5ndVlim 0z1pPLpx+8RnYiYIbwG4R0Q/L1eqPClEL+uWXrElWV+QEvBRKOtiKDVI8QSNBn28hboE eEboU2WyR/VjYzFfm8plGq83T8Ydicr/+nth8AbAYEG5xU2ENh03rpDvFkufFLh8tHPn u/iKeSraD8MxZLw0QKmsxeK4VQGqv2CM7N1rg4bFA3G9Df2c7UTcXeW5m8lyMIKuIGGh 4qHKWpPzmNfEFtbP87UzCBimT95rtKuHLw22V/SOr4JLrtCAD1l9rnlhOF+d7ES7MKFX v3nQ== X-Gm-Message-State: AOAM5318O24kbh5hJuxAVbiSIGJ943pu3GElArwqlCZLywG78oJjRqhg cITnaRSg0AXGuwinhioyiwwwU8JZ7uWHqTbM X-Google-Smtp-Source: ABdhPJw+wfbab5iuwK4wnlbZasU9nzvedUk0lkqY8WYjurMoxYQoQSMGzIhkzwXf3onZ69xuptkqKg== X-Received: by 2002:a7b:c055:: with SMTP id u21mr6072459wmc.0.1641980275753; Wed, 12 Jan 2022 01:37:55 -0800 (PST) Received: from xps13 (host-92-17-247-131.as13285.net. [92.17.247.131]) by smtp.gmail.com with ESMTPSA id m1sm1032322wrp.81.2022.01.12.01.37.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 12 Jan 2022 01:37:54 -0800 (PST) References: <877db6ge0o.fsf@beadling.co.uk> <874k697thf.fsf@elephly.net> <87r19d5pcw.fsf@elephly.net> User-agent: mu4e 1.4.15; emacs 27.2 From: Phil To: Ricardo Wurmus Subject: Re: Parallel guix builds can trample? In-reply-to: <87r19d5pcw.fsf@elephly.net> Message-ID: <871r1dguq6.fsf@beadling.co.uk> Date: Wed, 12 Jan 2022 09:37:53 +0000 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Host-Lookup-Failed: Reverse DNS lookup failed for 2a00:1450:4864:20::32c (failed) Received-SPF: none client-ip=2a00:1450:4864:20::32c; envelope-from=phil@beadling.co.uk; helo=mail-wm1-x32c.google.com X-Spam_score_int: -10 X-Spam_score: -1.1 X-Spam_bar: - X-Spam_report: (-1.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RDNS_NONE=0.793, SPF_HELO_NONE=0.001, SPF_NONE=0.001 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: "guix-devel@gnu.org" Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: "Guix-devel" X-Migadu-Flow: FLOW_IN X-Migadu-Country: US ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1641980332; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post:dkim-signature; bh=OpGH2IdMGNjt7PufobCREVUCeYsx60gv/ob6dkHZdug=; b=dtc2FyLYrI0PtrcQBCEvxKmvH63TkbSROY8w9RkNzQ3wv9HvijcTw8IW6ufisqk1umrasX 4GandiauLKgZdmesBdaCcZYrZ+JfEtB16TuhDjVP8+kIm55mjOzv6Z4Df3awoPAOdv4Zy4 pjkMVrioKXeONIQIQ8FdALWdjEum2uHQwyia2pOvIV88znJEWZTflz8mCbsh6709sEiTKl B9LK7z5G4KsVf+NhiudH1WvyqQxBt9+9xuYFeZIaxX5kLOhqzQFmQEOJdAc9sQypppOSJG lvdIu4nYwDmIaML7P7lcVJE/obf2fg04JZ4av/36cG6mGMrj+YBBQMfDeSNmcg== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1641980332; a=rsa-sha256; cv=none; b=nrdAo3DxUUJYOcJ9uhqdUIXCjdHUqIEL/inAgS60+WfHbpWVIX0/0OD8ms3x/HkO7gyCWn Wh8mJ9sUijd67XohCgSoMXEzDYh8REDyaSHyEfLWS23xiu5I3KIATuu+nV56Y6s9ATf3QJ 2LN+/IXsvIBnoZ2tP2cALIP8oj51Nl1wCHpy2+HUaudyj73dYVWtWyeLhwoCwIjkR9FqCQ qST8gqi8TgXp54N41y61uLYO7d0HzpCqOGdo9bM+HFGS+hAOTzx7+xCR5W0yiO27qkTDNI qVkduBV4SGB2N8IzXAhESQJdCZSqi9msgzLKLmW6iyzy5K7hk3jUYyXo7Syg/A== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=pass header.d=beadling-co-uk.20210112.gappssmtp.com header.s=20210112 header.b=eWDpAYVA; dmarc=none; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org" X-Migadu-Spam-Score: -2.82 Authentication-Results: aspmx1.migadu.com; dkim=pass header.d=beadling-co-uk.20210112.gappssmtp.com header.s=20210112 header.b=eWDpAYVA; dmarc=none; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org" X-Migadu-Queue-Id: A95C3465B2 X-Spam-Score: -2.82 X-Migadu-Scanner: scn1.migadu.com X-TUID: hMkCy3hgqoGV Hi - more details below. Ricardo Wurmus writes: > > How are you using Guix with this? Do you generate Guix package > expressions? Do you use =E2=80=9Cguix build --with-commit=E2=80=9D? > The situation is like this - if we had a directory of clones of my channel: - pr-1 - pr-2 - pr-3 - pr-4 ... and so on Initially all the clones are taken from the master branch of my channel and are all identical - but we change the version and commit to match the head of each PR branch as per below. Each clone looks like this: - pr-1 - my-package.scm - pr-2 - my-package.scm and so on.... Each my-package.scm has a package like below - the inital packages are all identical, but my system effectively seds the version and commit values like the below. These values are never committed back to master they are used only as local channels to build each PR to test each build still passes. (define-public my-package (package (name "my-package") (version "this-is-different-for-each-pr") ;; replace master version (source (git-checkout (url "ssh://same@repo:7999/same/repo.git") (commit "this-is-different-for-each-pr") ;; replace master version= =20 everything else remains the same in the package.... At this point we have lots of local channels referencing different commits,= in the same package, ready to build - so I spawn them all simultaneously - the equivalent pseudo-shell that I will mock up today would be: # define some sort of return code array: RC=3D[] for dir in pr-dirs RC[${dir}]=3D`guix build -K -L ${dir} my-package & 2>&1 > /tmp/${dir}.log= ` # note the ampersand wait for rc in $RC if $rc.value !=3D 0: report the failure of build $rc.key What I'm seeing occasionally is that the logs and return code for say direc= tory pr-1 and appearing in the guix build for pr-3 or pr-6 instead. We know this becuse the code is different enough in pr-1 that it's logs are unique across all the PRs. We can also check the source code if the build fails using --keep-failed to show it doesn't match the commit id in the package used to build it. Hopefully that makes sense? I can post the actual shell script once I've written the mock.