From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp12.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id eOfMGTL23WGQPAAAgWs5BA (envelope-from ) for ; Tue, 11 Jan 2022 22:27:14 +0100 Received: from aspmx1.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp12.migadu.com with LMTPS id YDJrFjL23WHqyQAAauVa8A (envelope-from ) for ; Tue, 11 Jan 2022 22:27:14 +0100 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 2E6263B36E for ; Tue, 11 Jan 2022 22:27:14 +0100 (CET) Received: from localhost ([::1]:55084 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1n7Ofh-0003Iu-CK for larch@yhetil.org; Tue, 11 Jan 2022 16:27:13 -0500 Received: from eggs.gnu.org ([209.51.188.92]:36408) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1n7OfD-0003IY-Oz for guix-devel@gnu.org; Tue, 11 Jan 2022 16:26:43 -0500 Received: from [2a00:1450:4864:20::535] (port=43941 helo=mail-ed1-x535.google.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1n7OfB-00058l-QQ for guix-devel@gnu.org; Tue, 11 Jan 2022 16:26:43 -0500 Received: by mail-ed1-x535.google.com with SMTP id m4so1482881edb.10 for ; Tue, 11 Jan 2022 13:26:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=beadling-co-uk.20210112.gappssmtp.com; s=20210112; h=user-agent:from:to:subject:message-id:date:mime-version; bh=kmUJNXnsr9TRw/jhIiiNmwIXBTERSOscE8xKSxrKp2M=; b=hZqHPx0uQ1JV6KHB6gS6RUj/wFhCVN+Scz4VIGmmRfEGcRKXTBeLjvpLexNi4P/C+Y MSc07a1oEuysjr6Ot00HWazOC9geQCQe3hTuonljvc5XDrks79QiYHQw5blbHjY6mFp0 7RXiZeQLIxsramadzTfgPJFKwGHmP7J3IeihIwsA0XzW4EE5AG1PH+7HXkL+DiiemR62 9/e1DS/CFyrxE2UzBELe7Jqd4bjQzmxwjQC3/1iycdHR4IGgnT2B5hVRLPWmWDh0HKdV J2Xm2Hg0rSjxmKwbzDz1aG/xp7Pdu2ADkG8EqN/a6oi/QRtlxvZuwyl1xWKdXonephjP eDtw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:user-agent:from:to:subject:message-id:date :mime-version; bh=kmUJNXnsr9TRw/jhIiiNmwIXBTERSOscE8xKSxrKp2M=; b=Im5yRE2YAgp3fbkFHcdWDJ1qwp6fup/Jmcfg5ETw6QmVkIihXPjtmm/QxzSHga22Gq rDIa3UXDEW+P/mdGO5K+N7RIQTxpk92H6u1bZBpyQ4jstarinDmwsLtdf3YRuJEpts3Q fe7h8eC3nFHrMAs6C4Xsb229Nk+KboBNHMlWR8qmkOSkEDsjyOT6DBdqJxfnoaz2KPbx d2pwga6zAsOz/MonBo5vhnjIndDhn/q5OZZRIDst/JsQBoU1ze1WZbMO6smy+XO02RQd XuddRo1H9RbMFzyVsiIHLqkPW2OOCbUxHRo7HvMA0rie8NTy0zBLqc+FD0EaXhwWfN7w gsdg== X-Gm-Message-State: AOAM532WPQl1aGyPpfS8QOw95wZWK7b4eGykDu6+qaLlljtgacCwFJo0 57f8BuXwL5x1WqDHUr4wZQOhzdq9wO5XMDW8 X-Google-Smtp-Source: ABdhPJy8i5ZIv3m4CY6RWxOX9uHK6k313HWnhHfX7th5sPvYj5D9FgY3oBqNX8ASpHiXy1BqJiFt6w== X-Received: by 2002:a05:6402:491:: with SMTP id k17mr6013409edv.333.1641936398663; Tue, 11 Jan 2022 13:26:38 -0800 (PST) Received: from xps13 (host-92-17-247-131.as13285.net. [92.17.247.131]) by smtp.gmail.com with ESMTPSA id e26sm3638871eje.73.2022.01.11.13.26.37 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 11 Jan 2022 13:26:37 -0800 (PST) User-agent: mu4e 1.4.15; emacs 27.2 From: Phil To: guix-devel@gnu.org Subject: Parallel guix builds can trample? Message-ID: <877db6ge0o.fsf@beadling.co.uk> Date: Tue, 11 Jan 2022 21:26:31 +0000 MIME-Version: 1.0 Content-Type: text/plain X-Host-Lookup-Failed: Reverse DNS lookup failed for 2a00:1450:4864:20::535 (failed) Received-SPF: none client-ip=2a00:1450:4864:20::535; envelope-from=phil@beadling.co.uk; helo=mail-ed1-x535.google.com X-Spam_score_int: -10 X-Spam_score: -1.1 X-Spam_bar: - X-Spam_report: (-1.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RDNS_NONE=0.793, SPF_HELO_NONE=0.001, SPF_NONE=0.001 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: "Guix-devel" X-Migadu-Flow: FLOW_IN X-Migadu-Country: US ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1641936434; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version: content-type:content-type:list-id:list-help:list-unsubscribe: list-subscribe:list-post:dkim-signature; bh=kmUJNXnsr9TRw/jhIiiNmwIXBTERSOscE8xKSxrKp2M=; b=sp/Ex68Mj6mLg7GO9ObYcVbc8UICvncmUVWwgyQFHshuFRrWklnMFXG1AhJz6I8qUwShLF 3/0rFsAEbrAdRT7viyMeVsLlmnH8PfmsaYmxMXHn0rv5lPIo5SPBuy53TTED+JerXSW86y Bqj8anfCqbrzc7J0C5JhvBSzRv6vdG7F+6CZGAakevHNUlfmSxzvfoMFm58gQkcYODCE2J dhqSmxy7RAdeJTaoJeROWrsNvmcmVS+BrsY+NmcXIJobrujRQeju4y+t8xsFKi+vrjuHqu DL9HWnaePiaH8Mp83BJ881CNmMXcYLnXsfrvm61i1tV4N2tP31R2lLRYW8ztkQ== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1641936434; a=rsa-sha256; cv=none; b=mBsSt09eOmCfLldVkCXyu8Vwc9x9zkfU8RsBos7sxSBWr4J6sln6jLgYaPDrJyKdJK9LPQ R5h7oBKMloDC5Sw/6OOS0BOlFhwvJF6jyEiauno6sYJup3+PQhPbfiNMEGuHj/4mXDNiGc gJU1iCHur4JbPnF1ECx3RSKlC4q/KaJ4YwH+qu/UZ1J7mPDEWibLPEy/r/NH3SZ0MV55U/ q5cCbDORZHtKWKIPQExGBZMARd6Ff7cRDRR5Mru7qWt4U2VjAobZdiMnMdZ9dKoBbALMQz odVAWDe7q7Ch166uZerh225yb0l5q9yytyZze1gpgwYvadYwzkARW5HbEBrk9Q== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=pass header.d=beadling-co-uk.20210112.gappssmtp.com header.s=20210112 header.b=hZqHPx0u; dmarc=none; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org" X-Migadu-Spam-Score: -3.92 Authentication-Results: aspmx1.migadu.com; dkim=pass header.d=beadling-co-uk.20210112.gappssmtp.com header.s=20210112 header.b=hZqHPx0u; dmarc=none; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org" X-Migadu-Queue-Id: 2E6263B36E X-Spam-Score: -3.92 X-Migadu-Scanner: scn0.migadu.com X-TUID: LnTnNfD/+DoG Hi all, I'm seeing strange issues when running many "guix build" commands of the same package simultaneously on the same linux account/server at the same time. Tracing this requires a lot of detective work, so for now I've omitted the logs - my initial question is - am I doing something which is obviously not going to work with Guix, or would people expect the below use-case to work? In my example I have ~12 clones of my private guix channel, sat in their own directories under my user account. I update the same single package on each channel locally with a different git commit id and package version. So 12 variations of the same package, accessing different commit ids, in the same package's source repo, and having different package versions in Guix. I then call "guix build -K -L /path/to/each/local/clone package-name" 12 times, one for each clone, these run in parallel shell sessions. This is all automated so each build is started within a fraction of second - trying to reproduce this issue by hand is proving difficult, and the issue is still sporadic in the automated system. The logs all show that each of the ~12 channels receive a local update with a unique and valid commit id and version for the changed package. However what I'm seeing is that some of the builds are failing with the error messages of other commit ids! To be clear - the failure is not the surprise here, the builds are getting mixed up and some are being trampled by what appears to be either a race condition or stale state. The surprise is that the failures are being tied to the wrong commit ids which do not contain the failures reported. Use of "--keep-failed" means I can prove this unequivocally, where the saved down /tmp/package-name.verson.drv.0 source code does not match the source of the commit id stated within the updated package - which I can see in the logs. I can also show that the source code matches exactly one of the other commit ids for one of the other clones, which was expected to fail. This happens sporadically, but we can reproduce the issue several times a day. I don't yet have 100% proof that the issue is happening inside Guix, but I've ruled out most (but not all) other causes so far, and the --keep-failed evidence tied with logs showing correct inputs is quite telling. Have any other bugs ever been reported like this that people are aware of? Have other people ran multiple builds in Guix under one account at the same time without issue? Any advice on how trap the issue, given it's hard to reproduce? The problem as never been seen when we do each guix build in serial. Apologies for the long and rather circumstantial e-mail! Cheers, Phil.