From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp0 ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id eGBVO08IB2FpnwAAgWs5BA (envelope-from ) for ; Sun, 01 Aug 2021 22:47:11 +0200 Received: from aspmx1.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp0 with LMTPS id 2L8cN08IB2FXFgAA1q6Kng (envelope-from ) for ; Sun, 01 Aug 2021 20:47:11 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 7229D151D6 for ; Sun, 1 Aug 2021 22:47:11 +0200 (CEST) Received: from localhost ([::1]:43910 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mAIMY-0004as-CV for larch@yhetil.org; Sun, 01 Aug 2021 16:47:10 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:42776) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mAILl-0004Z2-Cv; Sun, 01 Aug 2021 16:46:21 -0400 Received: from cascadia.aikidev.net ([173.255.214.101]:50378) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mAILc-0001Ka-QE; Sun, 01 Aug 2021 16:46:20 -0400 Received: from localhost (unknown [IPv6:2600:3c01:e000:21:21:21:0:100b]) (Authenticated sender: vagrant@cascadia.debian.net) by cascadia.aikidev.net (Postfix) with ESMTPSA id 986B71A9FD; Sun, 1 Aug 2021 13:46:05 -0700 (PDT) From: Vagrant Cascadian To: bug-guix@gnu.org Subject: source tarballs potentially built for each derivation In-Reply-To: <87k0ngg6zn.fsf@yucca> References: <87lf8xvrgj.fsf@yucca> <87k0ngg6zn.fsf@yucca> Date: Sun, 01 Aug 2021 13:45:41 -0700 Message-ID: <87o8ag6gi2.fsf@yucca> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha512; protocol="application/pgp-signature" Received-SPF: none client-ip=173.255.214.101; envelope-from=vagrant@debian.org; helo=cascadia.aikidev.net X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_NONE=0.001, SPF_NONE=0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: guix-devel Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: "Guix-devel" X-Migadu-Flow: FLOW_IN ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1627850831; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post; bh=aB8cHVl+dt4eUIRBHwRJy/IWkqZvhhYNTSIiB/LaOqA=; b=KRa5e4yuW6JTVTl8mUdgvhkkHv6s7LDuj75c7+03WQDF3cpLAxOgndLfyvM3JNPDybQFd7 E/6imZ5UrGuQbcGt54q5tZm97IAx3SpZTSXOH0YZI0z3t0egjWtsOmNbweluy21GNd6jCT M8kRXkEMRhPxnrDWRr/tCFGh/fOfuQDtqEUm70lBKfvjXYsZf3HbwSUc8e8bi/miXkZ5iH qB02QKKzOKHMKbiRYNEiaRRoYFQ8Exd6DLreouKqXOyA6VLORXllvrXhewaswIOltBrnrD J5SFWO5JnX6a1tLkZlOQHMipwnDFtemNXynQNTt5g/C/kO8G8my2IScBNkC1ww== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1627850831; a=rsa-sha256; cv=none; b=WA2t9MMMqlhDh1dzBi1ViKHFwAyRoZEK7uiZ766e3dmQ4gRZIZOKSJ3sSRtO25UDzVn55F Biyja7DtNfxx26MLK+//FMnQieuQb+KLXtrziL/tR2gt2VPJB0RFN7IC9BWZBhfu7EHyq0 qZ2ZObeMkOKsxzi6XxKFF4XLU09u+TlXyUjxltbxr3E1IylcjjuZnRv3J7/AiHQ0LCLDKM QaBHyopo7I1QaI+1wfKSYN5HS2QsGc/Hp1x9v7Bvbi7EB/BQPp425xwZIMoqMlnLP5gGT0 +w/mhqcVa57KEhXD7el3adF/zojEsBandf4wloVvKBTVUwGOxyBFuAe9s/zbXw== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of guix-devel-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-devel-bounces@gnu.org X-Migadu-Spam-Score: -1.52 Authentication-Results: aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of guix-devel-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-devel-bounces@gnu.org X-Migadu-Queue-Id: 7229D151D6 X-Spam-Score: -1.52 X-Migadu-Scanner: scn0.migadu.com X-TUID: /Q/9Dg0zhuNF --=-=-= Content-Type: text/plain Content-Transfer-Encoding: quoted-printable Turning this conversation into a bug, original thread around here: https://lists.gnu.org/archive/html/guix-devel/2021-05/threads.html#00427 On 2021-05-29, Vagrant Cascadian wrote: > On 2021-05-01, Leo Famulari wrote: >> On Sat, May 01, 2021 at 06:45:32PM -0700, Vagrant Cascadian wrote: >>> Pragmatically speaking, on slower platforms this is a huge resource >>> overhead. So much so that ci.guix.gnu.org *usually* times out when >>> generating the linux-libre aarch64 tarballs: >>>=20 >>> https://ci.guix.gnu.org/search?query=3Dsystem%3Aaarch64-linux+linux-l= ibre-arm64-generic >> >> Thanks for letting me know. I didn't know this was happening. >> >> The immediate solution is for me to make sure the tarballs have built >> before committing the updates. I already do this for x86_64 and I can >> start doing it for aarch64 too. > > This has definitely helped sometimes, thanks! I even saw a substitute of > linux-libre for aarch64 earlier today! :) > > Still, I'm noticing another problem with the way way the tarballs are > generated on ci.guix.gnu.org ... > > When it generates a tarball, all the various packages independently try > to recreate the source tarball; so you have at least fours jobs > ("linux-libre", "linux-libre-arm64-generic", "linux-libre-headers", > "linux-libre-bpf") all concurrently trying to build the very same > very-slow-to-build tarball on ci.guix.gnu.org. Sometimes one of them > might succeed, but the others may not, and even though one of them > succeeded, none of the failing ones retry... > > Not knowing exactly how ci.guix.gnu.org works, would it make sense to > create a tarball package instead of the ... computed origin(?) tarball, > so it could be better represented in the package dependency graph, and > the various linux-libre-* packages can wait till it is available rather > than all trying to recreate the same thing? > > That still requires the tarball generation to not time out in the first > place, but maybe it would help with the resource limitations a bit to > only build the source tarball once per architecture? This seems to still be an issue for ci.guix.gnu.org, but the linux-libre* substitutes for aarch64 seem to be available on https://bordeaux.guix.gnu.org ... $ guix weather linux-libre linux-libre-arm64-generic computing 2 package derivations for aarch64-linux... looking for 2 store items on https://ci.guix.gnu.org... https://ci.guix.gnu.org 0.0% substitutes available (0 out of 2) unknown substitute sizes 0.0 MiB on disk (uncompressed) 0.740 seconds per request (0.7 seconds in total) 1.4 requests per second 0.0% (0 out of 2) of the missing items are queued 1 queued builds aarch64-linux: 1 (100.0%) build rate: .00 builds per hour x86_64-linux: 0.00 builds per hour aarch64-linux: 0.00 builds per hour i686-linux: 0.00 builds per hour looking for 2 store items on https://bordeaux.guix.gnu.org... https://bordeaux.guix.gnu.org 100.0% substitutes available (2 out of 2) 83.9 MiB of nars (compressed) 202.2 MiB on disk (uncompressed) (continuous integration information unavailable) live well, vagrant --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iHUEARYKAB0WIQRlgHNhO/zFx+LkXUXcUY/If5cWqgUCYQcH9gAKCRDcUY/If5cW qp1LAP9eMIp0x8ygd5rht//VE9o8QBYm4cXec2LBxwOevDVwKwEAn10lZqhwXTPH 1XPirQBwEbLSI5V0P2x8DWY+PwgGIAU= =uCAx -----END PGP SIGNATURE----- --=-=-=--