From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp2 ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id kCbuI6j+wGHRQQEAgWs5BA (envelope-from ) for ; Mon, 20 Dec 2021 23:07:36 +0100 Received: from aspmx1.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp2 with LMTPS id CGCRH6j+wGHpdwAAB5/wlQ (envelope-from ) for ; Mon, 20 Dec 2021 22:07:36 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 3911529ECC for ; Mon, 20 Dec 2021 23:07:36 +0100 (CET) Received: from localhost ([::1]:49848 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mzQoh-00024S-Ai for larch@yhetil.org; Mon, 20 Dec 2021 17:07:35 -0500 Received: from eggs.gnu.org ([209.51.188.92]:36120) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mzQoS-00024B-2C for guix-devel@gnu.org; Mon, 20 Dec 2021 17:07:20 -0500 Received: from hera.aquilenet.fr ([185.233.100.1]:35126) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mzQoL-0007Gy-Hj for guix-devel@gnu.org; Mon, 20 Dec 2021 17:07:16 -0500 Received: from localhost (localhost [127.0.0.1]) by hera.aquilenet.fr (Postfix) with ESMTP id 7B188551; Mon, 20 Dec 2021 23:07:06 +0100 (CET) X-Virus-Scanned: Debian amavisd-new at aquilenet.fr Received: from hera.aquilenet.fr ([127.0.0.1]) by localhost (hera.aquilenet.fr [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id y-d1qiMLrgUS; Mon, 20 Dec 2021 23:07:05 +0100 (CET) Received: from ribbon (91-160-117-201.subs.proxad.net [91.160.117.201]) by hera.aquilenet.fr (Postfix) with ESMTPSA id E14C2361; Mon, 20 Dec 2021 23:07:04 +0100 (CET) From: =?utf-8?Q?Ludovic_Court=C3=A8s?= To: Christopher Baines Subject: Re: Mid-December update on bordeaux.guix.gnu.org References: <874k79zs29.fsf@cbaines.net> X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: 30 Frimaire an 230 de la =?utf-8?Q?R=C3=A9volution?= X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-pc-linux-gnu Date: Mon, 20 Dec 2021 23:07:04 +0100 In-Reply-To: <874k79zs29.fsf@cbaines.net> (Christopher Baines's message of "Wed, 15 Dec 2021 16:48:21 +0000") Message-ID: <87bl1bdj7b.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spamd-Bar: / X-Rspamd-Server: hera X-Rspamd-Queue-Id: 7B188551 X-Spamd-Result: default: False [0.53 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; RCPT_COUNT_TWO(0.00)[2]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; R_MIXED_CHARSET(0.63)[subject]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; MID_RHS_MATCH_FROM(0.00)[] Received-SPF: softfail client-ip=185.233.100.1; envelope-from=ludo@gnu.org; helo=hera.aquilenet.fr X-Spam_score_int: -11 X-Spam_score: -1.2 X-Spam_bar: - X-Spam_report: (-1.2 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_PASS=-0.001, SPF_SOFTFAIL=0.665 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: guix-devel@gnu.org Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: "Guix-devel" X-Migadu-Flow: FLOW_IN X-Migadu-Country: US ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1640038056; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post; bh=Wgvw27tckIdD8Wy8VshB35SrmtARcbGSKuZLv7Fvie4=; b=LY9oeNJSDKcSGc0lElnLl3QlFKNX+gdQh53kvxcRNVPrgyvXsMdJJWXLC2JP5JRUCeB92n yVyMuV9dfn0KsqTTmK8IBoTxE8X1rr+zML1xqUvFDXtw8zb6zgOk8eheBQlE8WBanxheMC DlJ8wHSLjB97TzNiQ6PIP7blgzMcYDTjWAPSqFX6JD2UWSRcTM3I17Pq0R6h+aK0IhbsoA /r6AKcy7lOBqoEJ4+rMJoF4p7FIMol+owKHNiH74tyWiNUN7oX0ek90sMjPq/wmusJ6Joz 803TwMjcYRC/Rj+IYuESpwANDitPFEeUkofLJu0OsMRWdQmTEJPqbHWJ1cBYTg== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1640038056; a=rsa-sha256; cv=none; b=OjfHGriUNf42kJo8My/Jlw9IzhKEnuJArnRKF9395IfLXixMIf2i1bQpLjHNN5Rwx8YUyb HbD7Jz1RFlke1+oxJvgyqLUkqrffErMY31SkyTExNBZV/AxJn/bJPpvmz5fSvE/FkZVPHr 3n4gvvPV30v5fqGstwZ6xLRlFcRw0iTl0naUP3XUMtujOKvJLYOznd2RVVhWZV6Nvu6CRl cJQ7g8CBo8FYaGSoY22t2nw/gAYV4gDZT63GuOo0I/o67a70fSl85tBctsdKoA6PkvjalN XZzLI5mmXjjOwBeGooh6nMw5JpQfwv2kGnp01By/qpYhMljxRcq3E43pi1cgzQ== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=none; dmarc=pass (policy=none) header.from=gnu.org; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org" X-Migadu-Spam-Score: -4.52 Authentication-Results: aspmx1.migadu.com; dkim=none; dmarc=pass (policy=none) header.from=gnu.org; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org" X-Migadu-Queue-Id: 3911529ECC X-Spam-Score: -4.52 X-Migadu-Scanner: scn0.migadu.com X-TUID: QHr0zt/lAwr/ Hello! Christopher Baines skribis: > In summary, the space issue I mentioned in the previous update has > effectively been addressed. All the paused agents are now unpaused and > builds are happening again. Yay! > However, due to the time spent not building things, the backlog is > longer than usual, and the substitute availability (especially for > x86_64-linux and i686-linux) is lower than usual. Yeah, =E2=80=98guix weather coreutils=E2=80=99 finds nothing on bordeaux.gu= ix right now. > ** Space issues and the nar-herder > > bordeaux.guix.gnu.org wasn't building things for 2 weeks as the space on > bayfront was getting a little scarce. This week I started rolling out > the nar-herder [2], a utility I've been thinking about for a while. This > has enabled moving nars off of bayfront on to another machine which I've > confusingly named lakefront. Woow, neat! Regarding nar-herder, I think it=E2=80=99d be nice to have a solution to mirroring in Guix proper, developed similarly to other components, because it could be a fairly central tool. =E2=80=98guix publish=E2=80=99 is probably not extensible enough to support= that, but we could make it a new =E2=80=98guix mirror=E2=80=99 or =E2=80=98guix sync=E2= =80=99 or whatever command. > The lakefront machine is hosted by Hetzner in Germany, and has 6TB of > storage across 2 hard drives. When a nar is requested from bayfront, it > will check it's local storage, and serve it from there if it exists, > otherwise it will forward the request to lakefront. There might be a > slight drop in the speed you can download nars, but apart from that this > change shouldn't be visible. Excellent, thanks for acting this promptly as problems crop up! I think we should make sure this is funded by the project and that credentials are shared. As discussed before, I think it=E2=80=99s best to = keep things tidy in that respect, in the spirit of building and maintaining this collectively. > The nar-herder is now busy deleting nars on bayfront which are available > on lakefront. Once it's got through the backlog, I'll enable NGinx > caching for the nars on bayfront, which should help improve > performance. I've tested downloading the largest nars (~2GB) though, and > it seems to work fine. > > In addition to lakefront, I've also added a 6TB hard drive to hatysa, > the HoneyComb LX2 machine that I host. Like lakefront, it's busy > downloading the nars from bayfront. This will act as a backup in case > lakefront is lost. > > In general this is an important step in being more flexible where the > nars are stored. There's still a reliance on storing pretty much all the > nars on a single machine, but which machine has this role is more > flexible now. I think this architecture also makes it easier to break > the "all nars on a single machine" restriction in the future as well. That=E2=80=99s really cool. > Going forward, it would be good to have an additional full backup of the > nars that bayfront can serve things from, to provide more > redundancy. I'm hoping the nar-herder will also enable setting up > geographically distributed mirrors, which will hopefully improve > redundancy further, and maybe performance of fetching nars too. > > Once I've had a chance to neaten up the code a little, I'll get a Guix > package and service written, plus I'll draft a Guix blog post about the > nar-herder. Usually I=E2=80=99m the one asking for blog posts :-), but I=E2=80=99d real= ly like us as a project to collectively engage on the topic before we publicize this specific approach. [...] > That means there's the following currently running build agents (by > architecture): > > - x86_64-linux + i686-linux (3 machines): > - 4 core Intel NUC (giedi) > - Max 16 cores for 1 concurrent build on bayfront > - 32 cores on milano-guix-1 (slow storage though) > - aarch64-linux + armhf-linux (2 machines) > - 16 core HoneyComb LX2 (hatysa) > - 4 core Overdrive (monokuma) > - powerpc64le-linux (1 machine) > - 64 core machine (polaris) This is looking pretty nice! I=E2=80=99m also all for streamlining machine handling, both administratively (in maintenance.git) and financially. > Ironically, I think that the most under-resourced area is x86_64-linux + > i686-linux. bayfront is restricted in what it can do since it also runs > the coordinator, and things go badly if the machine gets > overloaded. bayfront and milano-guix-1 both have hard drive storage, > which also can slow them down when building things (especially > milano-guix-1). > > If we (as a project) want bordeaux.guix.gnu.org to have the capacity to > keep up, it would be good to make a plan to add capacity. I think even a > single high core count x86_64-linux machine with performant storage > would make a big difference. Yes, we should do that, we still have funds for it. > ** IPv6 not supported (yet) > > I was slow to notice, but bordeaux.guix.gnu.org isn't available over > IPv6 yet, since bayfront doesn't seem to have IPv6 connectivity. I want > to address this, but I haven't worked out quite how to yet. Should be almost done now, as discussed on IRC today. \o/ Thanks! Ludo=E2=80=99.