From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp12.migadu.com ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms5.migadu.com with LMTPS id gN6MAIXIFWNLwQAAbAwnHQ (envelope-from ) for ; Mon, 05 Sep 2022 11:59:33 +0200 Received: from aspmx1.migadu.com ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp12.migadu.com with LMTPS id QBRqAIXIFWOmrQAAauVa8A (envelope-from ) for ; Mon, 05 Sep 2022 11:59:33 +0200 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id AD1DE1B599 for ; Mon, 5 Sep 2022 11:59:32 +0200 (CEST) Received: from localhost ([::1]:55704 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1oV8t9-0003rf-8F for larch@yhetil.org; Mon, 05 Sep 2022 05:59:31 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:48752) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1oV8op-0007iy-Q8 for guix-devel@gnu.org; Mon, 05 Sep 2022 05:55:05 -0400 Received: from mira.cbaines.net ([2a01:7e00:e000:2f8:fd4d:b5c7:13fb:3d27]:36527) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1oV8on-0001Cr-Th for guix-devel@gnu.org; Mon, 05 Sep 2022 05:55:03 -0400 Received: from localhost (unknown [IPv6:2a02:8010:68c1:0:54d1:d5d4:280e:f699]) by mira.cbaines.net (Postfix) with ESMTPSA id 02DBE27BBE9 for ; Mon, 5 Sep 2022 10:54:58 +0100 (BST) Received: from felis (localhost [127.0.0.1]) by localhost (OpenSMTPD) with ESMTP id 8b85cb40 for ; Mon, 5 Sep 2022 09:54:57 +0000 (UTC) User-agent: mu4e 1.6.11; emacs 28.1 From: Christopher Baines To: guix-devel@gnu.org Subject: September update on bordeaux.guix.gnu.org Date: Sat, 03 Sep 2022 13:13:17 +0100 Message-ID: <878rmy4041.fsf@cbaines.net> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha512; protocol="application/pgp-signature" Received-SPF: pass client-ip=2a01:7e00:e000:2f8:fd4d:b5c7:13fb:3d27; envelope-from=mail@cbaines.net; helo=mira.cbaines.net X-Spam_score_int: -5 X-Spam_score: -0.6 X-Spam_bar: / X-Spam_report: (-0.6 / 5.0 requ) BAYES_00=-1.9, DATE_IN_PAST_24_48=1.34, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: "Guix-devel" X-Migadu-Flow: FLOW_IN X-Migadu-To: larch@yhetil.org X-Migadu-Country: US ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1662371972; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version: content-type:content-type:list-id:list-help:list-unsubscribe: list-subscribe:list-post; bh=ry85cBg0eb4du0C0IkFyM3rtTxFzzbnEoqlUCCItM3Y=; b=NhZNsZ495aNE4anKNX8TCMu2jBYI19lK4cFqdpjxrFKWkn748uzOwCM4ummcNTgXJYTnQF H6awttb7LuV/ylWxlhuXIHuUJ9IqKU7A1jwOz8eVXZac+EleUaqzgEZiCxd6bkOeqVILoL 1w4uUkRzXR0Kl62ZRnNaj+WhNADbClZx75UB7WfhZZi5jO/XR8ugPvfJ3fxtJE5JothX0n ufMK2xHTnV65u1ZcIFRw9Hp1mjmHSzyDs0+B+udUtGJuwghnaeB37CUdlGBztPZmT+6z+U 9TvyAf1HXWwutAlm7pF6tcABr5VMMw4wglTQr1AWTE4S3nTCOxzkUHcIbMzjbg== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1662371972; a=rsa-sha256; cv=none; b=euCjrVJB2CYYMucexXJvpdhiCtGmX7y5KbDKHGyrPqBRRmveCZO24z2Tzx2X2l7DxbTFS1 WNtK5QaTSgZ4LmveTEQUQ8dyxhmBGLF3vTJmYMh1GtHIRiMXrBIkWmkMqQX61ywTGO2+wR 2GFpX9Jkf81IVQ96vJdNwUj5m0viHxVKMVYnxcxC5lEL4zkW1+Iav3en76E5F9alobnpny JMgfE+rOe/3+cuk/WCsigZaHvtdHr+k+3k1fVHHypplzAOBmI5Ro+KbctjwPwQG6upeoXl UssZ6IKiRORcvRMPHSa1dlvWmUQ1zrtF39TFBrJUnd9IhRacHAxZ86yL+ovHpA== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org" X-Migadu-Spam-Score: -2.18 Authentication-Results: aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org" X-Migadu-Queue-Id: AD1DE1B599 X-Spam-Score: -2.18 X-Migadu-Scanner: scn1.migadu.com X-TUID: 4ruHbz0ytFJ0 --=-=-= Content-Type: text/plain Hey, The last update was sent out in May [1], so this update roughly covers the last 3 months. 1: https://lists.gnu.org/archive/html/guix-devel/2022-05/msg00202.html This hasn't been my primary focus, so some of the changes that somewhat involve bordeaux.guix.gnu.org relate to using it for patch review, but to keep this update short I'll mention those elsewhere. ## Numbers Available from bordeaux.guix.gnu.org now, there's ~1.5 million nars, which take up ~6.2TiB of space available. Substitute availability for recent guix revisions is generally good, although powerpc64le-linux substitute availability is lower than it could be as there's not currently an always available machine to carry out the builds. ## Hardware In terms of machines, there was more work needed in the last few months as hatysa, the HoneyComb LX2 machine building ARM things was running low on storage and stopped booting. There's a couple of messages on guix-sysadmin about this but it was eventually resolved. Mixed up with this is the addition of another HoneyComb LX2 machine (hamal), this brings the total number of machines doing ARM builds to 3 (hatysa, hamal and monokuma). ## The build coordinator Hooks, which are used when various things happen can now run in parallel. This is important to avoid delays when builds are submitted or succeed. Also, there was a bug in the agent that led to spurious build failures. That's now been fixed. ## Mirrors There's a few test mirror machines setup, and I asked for people to test these to see what difference they make [2][3]. 2: https://lists.gnu.org/archive/html/guix-devel/2022-05/msg00203.html 3: https://lists.gnu.org/archive/html/guix-devel/2022-06/msg00186.html A couple of responses [4][5] have come in to the mailing list, both of which seem to suggest that mirrors could provide a significant boost to substitute download speed. 4: https://lists.gnu.org/archive/html/guix-devel/2022-07/msg00163.html 5: https://lists.gnu.org/archive/html/guix-devel/2022-07/msg00320.html Those test servers have been running for a while now, and are generally unused. I'll probably shut them down shortly to save money, and try to send out a more concrete plan of getting mirrors in place for bordeaux.guix.gnu.org. ## Serving fixed output files by hash There's a separate thread about this: https://lists.gnu.org/archive/html/guix-devel/2022-06/msg00333.html One thing that has changed is that the Guile fibers concurrent web server (which is used by the nar-herder) now supports streaming responses, which will reduce the memory usage of serving fixed output files: https://github.com/wingo/fibers/pull/63 ## Next steps Building patches and non-master branches is starting to happen, and that'll indirectly improve bordeaux.guix.gnu.org substitutes since it'll have already built some things by the time the patches are merged. As said above, from the limited data available, I think an argument could be made that mirrors are worth it in terms of the reliability benefits and potential performance improvements. I'll try to keep moving this forward. Operationally, I think the goal should be that it's not dependent on a single person. Currently it's probably too dependent on me. Things like moving the build-coordinator and nar-herder Git repositories to Savannah and getting more of the machines owned by guix-europe are ways to improve this. zstd compression support has been requested, and I don't see any significant blockers to this. I think the way forward is to add support for cached recompression of the nar files in the nar-herder. For build hardware, I think it remains to be seen how the addition of the patch and non-master branch builds affect the load. As mentioned above, having an always available powerpc64le-linux system would help avoid a backlog of builds for that architecture. It should also be much easier to see what the bordeaux build coordinator is doing. I think this requires a web interface to expose the active agents and builds. If you're interested in working on any of this, do let me know as while I don't have time to work on much of it myself, I should be able to make time to help others. Let me know if you have any comments or questions! Thanks, Chris --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQKlBAEBCgCPFiEEPonu50WOcg2XVOCyXiijOwuE9XcFAmMVx25fFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNF ODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcRHG1haWxAY2Jh aW5lcy5uZXQACgkQXiijOwuE9Xf7tRAAqFUkvJUjABZW1ws6YrC1Pciw795+RfJO KtaX0cq66oQqvu5IaYQnHILOA287pQ0i7LHZc20V0IN7UKqsMq1vpvPCz9BkPtq5 emXtwlWAIj9iqqXIhg0iYGZz/bvAN4KUOkDXimGKc87smzGgOqY/H1+iqkewLubi kdJSXbvFjC6lAxMPwALXf0dLBBKOAHMHQT8zw2jo5SbBSxqiWPo58gYjQmo/brna k28TPGOT8NIGewC3/4ycuyWw47RGkT1UCF7K6CcgpoIcJpkhOODlUvSZm3c0uo9K X54rjKK3ojtYYdjqu+UZLa8isY7EsB3TDpMoTPK6+aiNzT/e/mMhqzgO74+tKVqT yIGH0Z2ImN7fU2gjtr3T4e5XKho6k8yOojqTnRGI+1U8WzgDNt1AhQt5SKO4ENMR KWXNOmTJ6o2nCmYtQlvFSYmV1AciA8t+csid5ujT6EyvmMMDg3gXZE7AkoNLTaJO 2n3yjGA0s14XoJFMffdPoAvijBX1tGWUGalZ2d1CzweFTQ/GWevmnCoPOqXg+oAL Zqn7ZrPiMjZWE9jp3eAWSmuyusqD7+5VKuEe2RAEwgIC3+nlm0SOGzT/mUJIaMpK 6vTTeliK4zgjv5BNEHICFOUUXksgy3yVk/C4thrviAltFSNhIknh5KNEYwvxuYlt 2ka5x9MNJ3s= =ka6h -----END PGP SIGNATURE----- --=-=-=--