From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp1 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id MHsyNnU8umHGQwEAgWs5BA (envelope-from ) for ; Wed, 15 Dec 2021 20:05:25 +0100 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp1 with LMTPS id aDG8MXU8umFuDgAAbx9fmQ (envelope-from ) for ; Wed, 15 Dec 2021 19:05:25 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 5FC3FFBF9 for ; Wed, 15 Dec 2021 20:05:25 +0100 (CET) Received: from localhost ([::1]:47204 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mxZae-0001Mv-FW for larch@yhetil.org; Wed, 15 Dec 2021 14:05:24 -0500 Received: from eggs.gnu.org ([209.51.188.92]:47446) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mxYF4-0006v3-SR for guix-devel@gnu.org; Wed, 15 Dec 2021 12:39:02 -0500 Received: from mira.cbaines.net ([212.71.252.8]:56398) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mxYF1-0007g8-Nc for guix-devel@gnu.org; Wed, 15 Dec 2021 12:39:01 -0500 Received: from localhost (unknown [IPv6:2a02:8010:68c1:0:8ac0:b4c7:f5c8:7caa]) by mira.cbaines.net (Postfix) with ESMTPSA id ECF2427BBE9 for ; Wed, 15 Dec 2021 17:38:57 +0000 (GMT) Received: from capella (localhost [127.0.0.1]) by localhost (OpenSMTPD) with ESMTP id 13a40078 for ; Wed, 15 Dec 2021 17:38:57 +0000 (UTC) User-agent: mu4e 1.6.6; emacs 27.2 From: Christopher Baines To: guix-devel@gnu.org Subject: Mid-December update on bordeaux.guix.gnu.org Date: Wed, 15 Dec 2021 16:48:21 +0000 Message-ID: <874k79zs29.fsf@cbaines.net> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha512; protocol="application/pgp-signature" Received-SPF: pass client-ip=212.71.252.8; envelope-from=mail@cbaines.net; helo=mira.cbaines.net X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: "Guix-devel" X-Migadu-Flow: FLOW_IN X-Migadu-Country: US ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1639595125; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version: content-type:content-type:list-id:list-help:list-unsubscribe: list-subscribe:list-post; bh=cz+Lkzq3kkVFQZMGdG3NiojeDO4fvkxinpFb/4qxjpk=; b=DKaQWkJrOQ19t5XFPRvst6if65f4SpmTJpsPAPtb8rbPuUkS2tUOJZZM+sTwaiG6iZBkD8 SIQlFRLM4jJF8fKEyuJ8byS+4UAuCwaMS/KpvdZWvIhwUThh8TqsXgF0p9KDzNsbDB6uDN MNHSM0tKlqzR7lIKgwHdsD3lvBdJO24pxfH1wVOCM/vqAT7QNs8IZnWr67+q3RhZew7kAp UUK6uLvgCzQsP2+oPbD7DaHSJFa/ur6pER8l7alZe/B2JlYLJc1EN8UAaNTFoDxDZWyXpw WI1Wh9wYKd4U18tiOUt3dYqYFZXxJ12FShpTvehr2AnlxBXKq8J1TzRC6m0aCA== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1639595125; a=rsa-sha256; cv=none; b=np0eP7GQyse50rT/SeLcoRmZ1MnuBKnoV7cAI4zLsrNhM8y88x3G4buNwz6Bw0bIyJl06r Wu5Wx0iuq41ptzsKfQLqBQCNG9OkFUysGaU/vv7zD5ZazPd9eYFKsdgdhkqRzDvacV6zF5 NPSLX7XgV9VJ96WtJyA+POfFle/172aNOBiaEPU5DHsSKYjpiyAqtu++4VwwAZiBYT907Y NYV3kHiDredvJPpJ564NdbDrXH9YEAF1Ybese5ACTHdKwFod2OVuj8SVpXPTxrDaxp1bSd dtyB7pwm2Fi0VJJZ4bt4zntp/7EYVZ+DR5/03wR0FA3NHmkWsQnr/4iUR2dXPQ== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org" X-Migadu-Spam-Score: -5.88 Authentication-Results: aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org" X-Migadu-Queue-Id: 5FC3FFBF9 X-Spam-Score: -5.88 X-Migadu-Scanner: scn0.migadu.com X-TUID: leJv6h6RDT36 --=-=-= Content-Type: text/plain Content-Transfer-Encoding: quoted-printable Hey! I sent out the last update 3 weeks ago [1]. 1: https://lists.gnu.org/archive/html/guix-devel/2021-11/msg00154.html In summary, the space issue I mentioned in the previous update has effectively been addressed. All the paused agents are now unpaused and builds are happening again. However, due to the time spent not building things, the backlog is longer than usual, and the substitute availability (especially for x86_64-linux and i686-linux) is lower than usual. I've also noticed that bordeaux.guix.gnu.org doesn't work over IPv6, and I want to fix that soon. ** Space issues and the nar-herder bordeaux.guix.gnu.org wasn't building things for 2 weeks as the space on bayfront was getting a little scarce. This week I started rolling out the nar-herder [2], a utility I've been thinking about for a while. This has enabled moving nars off of bayfront on to another machine which I've confusingly named lakefront. The lakefront machine is hosted by Hetzner in Germany, and has 6TB of storage across 2 hard drives. When a nar is requested from bayfront, it will check it's local storage, and serve it from there if it exists, otherwise it will forward the request to lakefront. There might be a slight drop in the speed you can download nars, but apart from that this change shouldn't be visible. The nar-herder is now busy deleting nars on bayfront which are available on lakefront. Once it's got through the backlog, I'll enable NGinx caching for the nars on bayfront, which should help improve performance. I've tested downloading the largest nars (~2GB) though, and it seems to work fine. In addition to lakefront, I've also added a 6TB hard drive to hatysa, the HoneyComb LX2 machine that I host. Like lakefront, it's busy downloading the nars from bayfront. This will act as a backup in case lakefront is lost. In general this is an important step in being more flexible where the nars are stored. There's still a reliance on storing pretty much all the nars on a single machine, but which machine has this role is more flexible now. I think this architecture also makes it easier to break the "all nars on a single machine" restriction in the future as well. Going forward, it would be good to have an additional full backup of the nars that bayfront can serve things from, to provide more redundancy. I'm hoping the nar-herder will also enable setting up geographically distributed mirrors, which will hopefully improve redundancy further, and maybe performance of fetching nars too. Once I've had a chance to neaten up the code a little, I'll get a Guix package and service written, plus I'll draft a Guix blog post about the nar-herder. 2: https://git.cbaines.net/guix/nar-herder/about/ ** Build machines and backlog Because of the 2 weeks of not building anything, there's a significant backlog of builds to get through, and I'm not including the new builds from the core-updates-frozen merge here. As for build machines, milano-guix-1 came back online today, which is great. I believe harbourfront is still unusable through (broken hard drive). That means there's the following currently running build agents (by architecture): - x86_64-linux + i686-linux (3 machines): - 4 core Intel NUC (giedi) - Max 16 cores for 1 concurrent build on bayfront - 32 cores on milano-guix-1 (slow storage though) - aarch64-linux + armhf-linux (2 machines) - 16 core HoneyComb LX2 (hatysa) - 4 core Overdrive (monokuma) - powerpc64le-linux (1 machine) - 64 core machine (polaris) Ironically, I think that the most under-resourced area is x86_64-linux + i686-linux. bayfront is restricted in what it can do since it also runs the coordinator, and things go badly if the machine gets overloaded. bayfront and milano-guix-1 both have hard drive storage, which also can slow them down when building things (especially milano-guix-1). If we (as a project) want bordeaux.guix.gnu.org to have the capacity to keep up, it would be good to make a plan to add capacity. I think even a single high core count x86_64-linux machine with performant storage would make a big difference. ** IPv6 not supported (yet) I was slow to notice, but bordeaux.guix.gnu.org isn't available over IPv6 yet, since bayfront doesn't seem to have IPv6 connectivity. I want to address this, but I haven't worked out quite how to yet. Please let me know if you have any comments or questions! Chris --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQKlBAEBCgCPFiEEPonu50WOcg2XVOCyXiijOwuE9XcFAmG6KC5fFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNF ODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcRHG1haWxAY2Jh aW5lcy5uZXQACgkQXiijOwuE9Xfwrw//SR+PB3ZpCN/FBUqkqxL9tv0B2h7Ya21J 38RosEeLdUE6vC1ne+wAelVI8PZ675Q2HcXlH+prt6gcs9DV7h7SQ7nwe2dxV2f2 k+W2LokTnjWjAW+S8AoyhAz/Z9rxUhlVC/IMjYKG7KOhSA0CV9rc6d/un5jFwbv4 CNRA9n5ZjWvzJUcEEpG6jOfqF52vcSsWAN4fbaN29xWf0nzcaIZkRl01OQ7tbmLd yzXCYYSObn1JYRhVZTUPa0jttYbfDcTQwDi9VyTwtMbj9aTrwJofkc1g9/v+HKYC gOgfRSqun93rwTPMdSnY8oZy66YAIx5JxOqoEDdLrXntF9T8v3BXJ4NlkrRmmZLS yqQMGznMPf7IIcBH2xNLl8Pi5+kvmFJwRZenHcpsCis1anC3OcKzin6gO5wqMJ/0 ultEIosHJOCpjA9wwUxEsEst4/99amS/WqrQUGpklTgwHoFFvzM4DcqhK5lMbieW 7Gw3UWtEo+4ZccaKyDK72lqXJKv5kGjE/eEPdm1taVin06fojv1gLdoERObaOKWQ fGym095b2eVGGOoLjNEMLlRw9q07YHX6bay+FfGLpth20gOihRm1++mF8aBnOKAt E0crcEBpHheoFKXXQp4bli0E2tTgljUx/oGKheZZtEogggrQFfjDkWf8V38yD76G ExkjYKkkXwE= =i580 -----END PGP SIGNATURE----- --=-=-=--