From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp0.migadu.com ([2001:41d0:403:4876::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms13.migadu.com with LMTPS id 8IMEBaFuNWdRPgAAqHPOHw:P1 (envelope-from ) for ; Thu, 14 Nov 2024 03:29:37 +0000 Received: from aspmx1.migadu.com ([2001:41d0:403:4876::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp0.migadu.com with LMTPS id 8IMEBaFuNWdRPgAAqHPOHw (envelope-from ) for ; Thu, 14 Nov 2024 04:29:37 +0100 X-Envelope-To: larch@yhetil.org Authentication-Results: aspmx1.migadu.com; dkim=pass header.d=rekahsoft.ca header.s=key1 header.b=i+ApoppO; dmarc=pass (policy=quarantine) header.from=rekahsoft.ca; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org" ARC-Seal: i=1; s=key1; d=yhetil.org; t=1731554977; a=rsa-sha256; cv=none; b=Pw4aI11n2hFbtO9l0QA2bjiM64LBIVSYAXqYWagqIf2Y0EbiKSzQbvFxS/ya2CpnSQ/Mij 62SNPY2Cd1KKMu6t+qg81pqZl8t2zhn2ct8X4MW+ggDmcpp6IoXu3yl/v9IzF+t0UWvTX6 KGrEiWT7K9d+7iIHlG6g0ZWy9TF6uE05/+vdY7ZPioU2eFvqSxDLQSy9tZOsTxsayfDaQF ih51Q/COIq4KT26SxzKrJQBR/Virh5Jcws9wBqfYLCzepN5mdsY4wMkkJAR9WPbRU8PsI/ XIJjSPnZbQsD8ZxFyzqKhhXP7JkJjrVGOkL6XqpkeK2xSiNHw4L+1tM46EsTzA== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=pass header.d=rekahsoft.ca header.s=key1 header.b=i+ApoppO; dmarc=pass (policy=quarantine) header.from=rekahsoft.ca; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org" ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1731554977; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post:dkim-signature; bh=4fTayIs4aNZf5IwaItaUL0BpMo1gW0epQApDWwEh0oI=; b=C/+yFcHiPIerPjrt8DYQ+vEdh/SmgDkIV5jciBi1r6CTrpD3azqUtcU0j0bDDijR2dBv9I z+ZJO1xcOXfUxc/WNK+tUzHSdh8vs1uUMyvzKvBF7W5gQPLZhvTQcAYi8QvEn9b49kbela cOMbrOiXfd3HLQPMY2B2m+SV3KuZt/KCi2eM4X2ZHllCs+ZN1PzMMDOCaKBpWTXwn5Hdgf 2GvRW8sfIUZdw0u9cK+cgRf0cBgHWAZu7jCwRRcngDYPn9+DHp4fB8BDW39eYn07Bo/OUc fogVWpUeH8KJNKwxD0HJdr1/HJfUErsFrYiqtIomEqCfYDWk2Tgv3TMKLMhG6w== Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 62B777ACCE for ; Thu, 14 Nov 2024 04:29:36 +0100 (CET) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tBQXJ-0006X6-F3; Wed, 13 Nov 2024 22:28:49 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tBQXH-0006Wo-1s for guix-devel@gnu.org; Wed, 13 Nov 2024 22:28:47 -0500 Received: from out-174.mta0.migadu.com ([2001:41d0:1004:224b::ae]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tBQXE-0000l3-QP for guix-devel@gnu.org; Wed, 13 Nov 2024 22:28:46 -0500 X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rekahsoft.ca; s=key1; t=1731554917; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=4fTayIs4aNZf5IwaItaUL0BpMo1gW0epQApDWwEh0oI=; b=i+ApoppO401gEXuwOeHsmM05UApVuiPo5CtFYFwoUURWdFCfFz80iqzNloOCdzzUi5UncS CB+HRtMcx0s0FFxkCKjG+X9WfQ2M8qYqfQOkQR6yIxhSeJ3VcVqtLiMKZVBfmnXQJmYF1O RGYqU2KxqaHF98hC5I4pR5unZJLQCxU= From: "Collin J. Doering" To: Vagrant Cascadian Cc: guix-devel@gnu.org Subject: Re: New North American based Guix Substitute Server, cuirass.genenetwork.org Now Available In-Reply-To: <87v7x3622m.fsf@wireframe> (Vagrant Cascadian's message of "Sun, 03 Nov 2024 20:43:45 -0800") References: <871q46ytyn.fsf@rekahsoft.ca> <87v7x3622m.fsf@wireframe> Date: Wed, 13 Nov 2024 22:28:32 -0500 Message-ID: <877c96sdcv.fsf@rekahsoft.ca> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha512; protocol="application/pgp-signature" Received-SPF: pass client-ip=2001:41d0:1004:224b::ae; envelope-from=collin@rekahsoft.ca; helo=out-174.mta0.migadu.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: guix-devel-bounces+larch=yhetil.org@gnu.org X-Migadu-Country: US X-Migadu-Flow: FLOW_IN X-Migadu-Spam-Score: -5.43 X-Spam-Score: -5.43 X-Migadu-Queue-Id: 62B777ACCE X-Migadu-Scanner: mx10.migadu.com X-TUID: 1LEIVjhAV+dO --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Hi Vagrant, Just wanted to follow up with some good news! cuirass.genenetwork.org is no= w building packages and serving substitutes again. =2D-8<---------------cut here---------------start------------->8--- % guix time-machine -- weather --substitute-urls=3Dhttps://cuirass.genenetw= ork.org <..> computing 31,887 package derivations for x86_64-linux... looking for 37,018 store items on https://cuirass.genenetwork.org... https://cuirass.genenetwork.org =E2=9B=85 74.6% substitutes available (27,605 out of 37,018) at least 37,141.3 MiB of nars (compressed) 86,508.9 MiB on disk (uncompressed) 0.009 seconds per request (329.8 seconds in total) 112.2 requests per second 3.5% (326 out of 9,413) of the missing items are queued at least 1,000 queued builds x86_64-linux: 998 (99.8%) i686-linux: 2 (.2%) build rate: 23.71 builds per hour x86_64-linux: 23.71 builds per hour =2D-8<---------------cut here---------------end--------------->8--- Please let me know if you have any further issues and I apologize for any i= nconveniences caused! Further, I thought it would be valuable to provide a short post-mortem deta= iling the outage. 1. Builds started failing on cuirass.genenetwork.org 2024-09-06, starting = at evaluation 155302[1] where the evaluation fails to proceed due to a failure to bui= ld coreutils. 2. This same build succeeded on local systems, and on ci.guix.gnu.org, but= continued to fail on cuirass.genenetwork.org. No changes were made to cuirass.genenetwork= .org prior to this failure occurring. 3. Some initial investigation showed that the tests/cp/reflink-auto.sh cor= eutils test was failing, but why was unknown (it was not reproducible on other guix sys= tems, as well as other guix build farms). 4. Guix bug #74203[2] was opened on 2024-10-04 detailing the failure to bu= ild coreutils, and suggesting a patch to disable the failing test (similar to what nix fol= ks did for the same issue) 5. Time to investigate the issue thoroughly didn't come until 2024-W45, wh= ere it was discovered that the coreutils test in question was flaky on btrfs syste= ms in some situations. Though exact details need to be further investigated, using a tmpfs /tm= p causes the test to no longer fail! Further detail can be seen in Guix bug #74203[2]. 6. The discovery made in 5 allowed cuirass.genenetwork.org to be updated, = however the guix channel/s that were in use had a hash mismatch for the pango package. T= his was easily worked around by using a newer version of the guix channel. 7. However, another odd failure was hit during updating; specifically, whi= le building the guix package, its test suite would crash (not fail, but actually crash)= . An existing issue (#46413[3]) very closely captures the issue faced, however in this case= the test-suite for guix was not being run manually. I will follow up on this issue, and if= necessary, create a new one to track this specific failure. 8. In order to work around the issue building guix in 7, I opted to fall b= ack to using substitutes from ci.guix.gnu.org. This allowed me to update cuirass.gen= enetwork.org, and get it building again. 9. Upon updating, the default version of postgres was updated upstream, so= cuirass couldn't be started until its data was updated, which was quickly resolved. PS: apologies for the double message - forgot to wide reply (mu4e changed t= he reply workflow and I haven't caught up yet!) [1]: https://cuirass.genenetwork.org/eval/155302 [2]: https://issues.guix.gnu.org/74203 [3]: https://issues.guix.gnu.org/46413 On 03 Nov 2024 at 20:43, Vagrant Cascadian wrote: > On 2024-07-06, Collin J. Doering wrote: >> I am excited to announce that Guix substitutes (for x86_64) are now >> available in North America, thanks to the generous contribution of >> server hardware and infrastructure from GeneNetwork.org. > > The last evaluation that actually seems to have succeeded was from late > August: > > https://cuirass.genenetwork.org/eval/154706 > > All the evaluations since then have failed, up until around > mid-september... > > I was pretty thrilled to use it while it was working... especially as it > provided another reference point to check reproducible builds of guix! > > I imagine running a whole build farm would be pretty intensive resource. > Soooo... I am curious what the current prospects are of getting it > running again? :) > > > live well, > vagrant =2D-=20 Collin J. Doering http://rekahsoft.ca http://blog.rekahsoft.ca http://git.rekahsoft.ca --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iIoEARYKADIWIQSg4F3ACfM0j/GRGeP3fjGTl82nFgUCZzVuYBQcY29sbGluQHJl a2Foc29mdC5jYQAKCRD3fjGTl82nFrNlAP9LtAbuKrGj+jadUbcQG+n5AGcpeP/p jtZTtREcqQ1oDwEApTP+5YFQA5pKIzlFtaENB3Sn/5GxbCmIrlSomATpMQ0= =yKmJ -----END PGP SIGNATURE----- --=-=-=--