From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp10.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id eEA4HjL71mGxhQEAgWs5BA (envelope-from ) for ; Thu, 06 Jan 2022 15:22:42 +0100 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp10.migadu.com with LMTPS id sOj/FjL71mGGZwEAG6o9tA (envelope-from ) for ; Thu, 06 Jan 2022 15:22:42 +0100 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id C9CAB285B9 for ; Thu, 6 Jan 2022 15:22:41 +0100 (CET) Received: from localhost ([::1]:44456 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1n5Tf6-0002s8-MR for larch@yhetil.org; Thu, 06 Jan 2022 09:22:40 -0500 Received: from eggs.gnu.org ([209.51.188.92]:48028) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1n5T2c-0005ZX-2T for guix-devel@gnu.org; Thu, 06 Jan 2022 08:42:54 -0500 Received: from mira.cbaines.net ([212.71.252.8]:36238) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1n5T2Z-0006pk-DW for guix-devel@gnu.org; Thu, 06 Jan 2022 08:42:53 -0500 Received: from localhost (unknown [IPv6:2a02:8010:68c1:0:8ac0:b4c7:f5c8:7caa]) by mira.cbaines.net (Postfix) with ESMTPSA id 9C9B027BBE9; Thu, 6 Jan 2022 13:42:45 +0000 (GMT) Received: from capella (localhost [127.0.0.1]) by localhost (OpenSMTPD) with ESMTP id f6a01cda; Thu, 6 Jan 2022 13:42:44 +0000 (UTC) References: <874k79zs29.fsf@cbaines.net> <87bl1bdj7b.fsf@gnu.org> User-agent: mu4e 1.6.10; emacs 27.2 From: Christopher Baines To: Ludovic =?utf-8?Q?Court=C3=A8s?= Subject: Re: Mid-December update on bordeaux.guix.gnu.org Date: Thu, 06 Jan 2022 13:26:42 +0000 In-reply-to: <87bl1bdj7b.fsf@gnu.org> Message-ID: <87fsq1askt.fsf@cbaines.net> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha512; protocol="application/pgp-signature" Received-SPF: pass client-ip=212.71.252.8; envelope-from=mail@cbaines.net; helo=mira.cbaines.net X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: guix-devel@gnu.org Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: "Guix-devel" X-Migadu-Flow: FLOW_IN X-Migadu-Country: US ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1641478962; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post; bh=LRnApl892RAClbHyKyGTbfP+lBt7ZKVqy5h2xZ53Yu0=; b=AtTQm4Q/XM5NfRdB5iIQylomJiSVs3uwjWmR7eFWSHTgF7SGL5VNOcKBSjFCMAALRkJOA2 JS+Pgxxtt/A/mwNXfG9/lTyjyVmNitzPHKuppcSm3N7E3ACWTgtW/bkHmB1dcIEzmU/CvJ DAsIS4FiWNoUmc4adt1jEoA66LOHX2XBths3kFF8BRdVmOkfvmE/wno6dNbdUKDW3eXClq qBMS1dhl+twyfINqfqEC5j33t8X4JK9EUNgQ7KCn2jSKtRFr8gowyF8a2sggI98SZC4MiU QdxJzpm2h9N24OqJ+h2Q07gkOIPU8DEQ55bWMzKlryPoki+rlqoCAKI/l4QIUA== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1641478962; a=rsa-sha256; cv=none; b=BmmQAsFA4L7iN3vDjHa8bzVLTTzEYtgx9wv6jwV6s2IX/FWZlOVlsi0HvvbL5gcZS5syaZ I0JeIPrujrYN1516V8WXZopyWq1zWnwflD6vrCOjzdXnEufTyZsc555xhB+jQnhgPJWJ08 mO89nAeKAW7Qa1Ok/4SoMCV97jk7j9GidNIBfwKoQeRsOLHE0eawFAl5f8zvSaTIPys52t 4SUH4N5vbocfVrTeoHm68L5UuinpkJKZpxzXitft1ieos4ZjnQpvdUH7VWI8BF+/pMFahH FXOINU3R9MWgZ7v/ijtK2SKcb8UnWaPx5frdZq7HYdyQGa+uLSfmdVPlsW4lqw== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org" X-Migadu-Spam-Score: -5.20 Authentication-Results: aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org" X-Migadu-Queue-Id: C9CAB285B9 X-Spam-Score: -5.20 X-Migadu-Scanner: scn0.migadu.com X-TUID: 7i2fhWNmOVn7 --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Ludovic Court=C3=A8s writes: >> However, due to the time spent not building things, the backlog is >> longer than usual, and the substitute availability (especially for >> x86_64-linux and i686-linux) is lower than usual. > > Yeah, =E2=80=98guix weather coreutils=E2=80=99 finds nothing on bordeaux.= guix right now. I've been so slow in replying to this email that the situation has actually improved now, substitute availability for x86_64-linux is around ~70% and still slowly rising. >> ** Space issues and the nar-herder >> >> bordeaux.guix.gnu.org wasn't building things for 2 weeks as the space on >> bayfront was getting a little scarce. This week I started rolling out >> the nar-herder [2], a utility I've been thinking about for a while. This >> has enabled moving nars off of bayfront on to another machine which I've >> confusingly named lakefront. > > Woow, neat! > > Regarding nar-herder, I think it=E2=80=99d be nice to have a solution to > mirroring in Guix proper, developed similarly to other components, > because it could be a fairly central tool. > > =E2=80=98guix publish=E2=80=99 is probably not extensible enough to suppo= rt that, but we > could make it a new =E2=80=98guix mirror=E2=80=99 or =E2=80=98guix sync= =E2=80=99 or whatever command. I think having something in Guix would be good too. I still view the nar-herder as a bit wierd in terms of the collection of concerns, mixing mirroring in with moving nars around between machines and I'm hoping to add some metrics functionality soon as well. >> The lakefront machine is hosted by Hetzner in Germany, and has 6TB of >> storage across 2 hard drives. When a nar is requested from bayfront, it >> will check it's local storage, and serve it from there if it exists, >> otherwise it will forward the request to lakefront. There might be a >> slight drop in the speed you can download nars, but apart from that this >> change shouldn't be visible. > > Excellent, thanks for acting this promptly as problems crop up! > > I think we should make sure this is funded by the project and that > credentials are shared. As discussed before, I think it=E2=80=99s best t= o keep > things tidy in that respect, in the spirit of building and maintaining > this collectively. That would be good, I can start a thread on guix-sysadmin about this. >> Going forward, it would be good to have an additional full backup of the >> nars that bayfront can serve things from, to provide more >> redundancy. I'm hoping the nar-herder will also enable setting up >> geographically distributed mirrors, which will hopefully improve >> redundancy further, and maybe performance of fetching nars too. >> >> Once I've had a chance to neaten up the code a little, I'll get a Guix >> package and service written, plus I'll draft a Guix blog post about the >> nar-herder. > > Usually I=E2=80=99m the one asking for blog posts :-), but I=E2=80=99d re= ally like us as > a project to collectively engage on the topic before we publicize this > specific approach. Sure, I also still need to post patches for the Guix service, which I'll try to do soon. >> That means there's the following currently running build agents (by >> architecture): >> >> - x86_64-linux + i686-linux (3 machines): >> - 4 core Intel NUC (giedi) >> - Max 16 cores for 1 concurrent build on bayfront >> - 32 cores on milano-guix-1 (slow storage though) >> - aarch64-linux + armhf-linux (2 machines) >> - 16 core HoneyComb LX2 (hatysa) >> - 4 core Overdrive (monokuma) >> - powerpc64le-linux (1 machine) >> - 64 core machine (polaris) > > This is looking pretty nice! I=E2=80=99m also all for streamlining machi= ne > handling, both administratively (in maintenance.git) and financially. I think there was some discussion previously about guix-europe buying the HoneyComb board, which I can probably restart. It would be good to also sort out better hosting for the powerpc64le-linux machine. I'll try to put some time in to organising things in maintenance.git. >> Ironically, I think that the most under-resourced area is x86_64-linux + >> i686-linux. bayfront is restricted in what it can do since it also runs >> the coordinator, and things go badly if the machine gets >> overloaded. bayfront and milano-guix-1 both have hard drive storage, >> which also can slow them down when building things (especially >> milano-guix-1). >> >> If we (as a project) want bordeaux.guix.gnu.org to have the capacity to >> keep up, it would be good to make a plan to add capacity. I think even a >> single high core count x86_64-linux machine with performant storage >> would make a big difference. > > Yes, we should do that, we still have funds for it. Great :) >> ** IPv6 not supported (yet) >> >> I was slow to notice, but bordeaux.guix.gnu.org isn't available over >> IPv6 yet, since bayfront doesn't seem to have IPv6 connectivity. I want >> to address this, but I haven't worked out quite how to yet. > > Should be almost done now, as discussed on IRC today. \o/ Indeed, I think this is working now. Thanks, Chris --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQKlBAEBCgCPFiEEPonu50WOcg2XVOCyXiijOwuE9XcFAmHW8dJfFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNF ODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcRHG1haWxAY2Jh aW5lcy5uZXQACgkQXiijOwuE9Xfjkw/+I1WebhoUUjXAyo2pkSe6GcAKL9218V1t 9ZgrvBkNMFrqfqZH0yzmoGCDXIeWf6PFCDJgtSXrxMd9XHWjhp54NrcI+AXLM7wk qkrI1o9MiJujfWBa/GVVBmp40cvafEB78N4s6C8FKWTHEYRYjFm1YN2dPrylXfMh WF7l7H+xzY3I3zWI5tAtuQ7NObZNENQFOz6v3LZUrmpUwwHnF5nE87Fh5nYjskT/ JkScsJA/KTf/L3ZNv1AwFQiL2lK9DlKkeRw1Ju5L3KdVK+kxXftqjzUJg/OojsdD DDy+LOb7tlDUjl9vVlHBcobGtmlRElpWEa60seyGCfO14fGBOFEeimNm0B5AJqc6 ACIKj5roWvIzKKkuwCui89okicV6s1zJfUr1qkOzc+WxsoerpBGIRzauMBYssLrs gSDZNQuTMMGlP0VdWySMqfE4ZkMFBO876UsxV8CjKfWH2+DU2GYZUJ737/3eS7y0 G3myo0CUzucFAM+3i+v+geTtizCwSeYxxZCcCvlEsVXKVFgtYKnpGvSYZoA1Nenp jUa6MaQrDjUrQobUPflzPP2ZUXo42VHoy0bvHqI/nAASQ7mWAvIrKgW1VEFODfbC Ncq9ajgOJVZbGNuitCNfmgq0EZ2WXqBVydXdtK+HXFAOKJTqKZdRDibQkiwrPbIN Gpj5kN2nQ54= =wJ8r -----END PGP SIGNATURE----- --=-=-=--