From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp1 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id yMtxHR74u2FDQQEAgWs5BA (envelope-from ) for ; Fri, 17 Dec 2021 03:38:22 +0100 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp1 with LMTPS id OE4QGR74u2FdBQAAbx9fmQ (envelope-from ) for ; Fri, 17 Dec 2021 02:38:22 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 39B4C32F7C for ; Fri, 17 Dec 2021 03:38:22 +0100 (CET) Received: from localhost ([::1]:34078 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1my38X-0006N5-DL for larch@yhetil.org; Thu, 16 Dec 2021 21:38:21 -0500 Received: from eggs.gnu.org ([209.51.188.92]:35264) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1my38D-0006Kg-6Y; Thu, 16 Dec 2021 21:38:01 -0500 Received: from mira.cbaines.net ([212.71.252.8]:56406) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1my38B-0001oR-5E; Thu, 16 Dec 2021 21:38:00 -0500 Received: from localhost (unknown [IPv6:2a02:8010:68c1:0:8ac0:b4c7:f5c8:7caa]) by mira.cbaines.net (Postfix) with ESMTPSA id 38BC327BBE9; Fri, 17 Dec 2021 02:37:56 +0000 (GMT) Received: from capella (localhost [127.0.0.1]) by localhost (OpenSMTPD) with ESMTP id 5c5f97ba; Fri, 17 Dec 2021 02:37:55 +0000 (UTC) References: <87v8zorien.fsf@inria.fr> User-agent: mu4e 1.6.6; emacs 27.2 From: Christopher Baines To: Ludovic =?utf-8?Q?Court=C3=A8s?= Subject: Re: Solstice infrastructure hackathon Date: Fri, 17 Dec 2021 01:57:39 +0000 In-reply-to: <87v8zorien.fsf@inria.fr> Message-ID: <87fsqsx8fy.fsf@cbaines.net> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha512; protocol="application/pgp-signature" Received-SPF: pass client-ip=212.71.252.8; envelope-from=mail@cbaines.net; helo=mira.cbaines.net X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: guix-devel@gnu.org, guix-sysadmin Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: "Guix-devel" X-Migadu-Flow: FLOW_IN X-Migadu-Country: US ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1639708702; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post; bh=7ShwHhWTcyko0fmb4yzIyAKvWyCHMiJc3ycq98qrEAQ=; b=KqINQhlKH36Izqn1JG6AR+z/kWu903QHygf06OBZwCBm7Nu/XHnA8j+GuK28mDkRBi6JOt TJd8mRBiPgctfwcDJbYMoACLZctDwSVSow6olZxX4MLYX++K4F3yDU7yhx5bZ9bBxZE43g K+DDrA0tOUd/BcDiNn2Ti3koXYZFW4+BLo+V2Jv2dc7UGv8hz7UeSTaVZGPzfU2Q9X8fS3 XShxzGGfPE5Oidh9kHQeGQFeXheaVXUC16HuPT2Eua1GE2SvtLKxHib3Igt0u7VhbhgZ3j UhTMjVFBc20bzKiEP2JEBF5Mii5XnywsANcmAJhtBPOyeI1PPMo2jZ1XehSgqQ== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1639708702; a=rsa-sha256; cv=none; b=TnBoXxhsuP/ZVdOyxjs/ph7ASRt/8H+YAsGiiEmshporhIdWjDNmoEki1mObHIMqQxVkVl oa+DEEGUdgNFKOoYIgZ+hQ/KkkjNohJw6nXlpAX1P0OYgOcaCj7elqqKk0W6RQvSSGcWes U9BmmYZCPvFG7r7grFlsNDHIa9StQv0UXrRRjp1y0xWI08TI6djzQeMKkOI9o9TNlX14tb 0gYP4ScXJlUbklx1pOHs1Yv1FqSsptYAxjiqur3intEw7cx0JXnIPqQXWaQ9w9NZf9CXBY OeLmeZsZln3x3EguUaq4C1xI68ES9evX+4/PsniN6FntCYa+iRCqVx7dx/b75Q== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org" X-Migadu-Spam-Score: -6.09 Authentication-Results: aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org" X-Migadu-Queue-Id: 39B4C32F7C X-Spam-Score: -6.09 X-Migadu-Scanner: scn0.migadu.com X-TUID: 88cIRhw9AZra --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Ludovic Court=C3=A8s writes: > We were unlucky enough that it happened days after the other build farm, > bordeaux.guix.gnu.org, ran out of disk space and had its CI stopped, > right before the big merge=E2=80=94so it doesn=E2=80=99t have substitutes= for current > master. Builds effectively stopped on the 29th of November, which is more than a few days I'd say, although this is maybe not the biggest issue. Since the build coordinator instance behind bordeaux.guix.gnu.org wasn't building things from core-updates-frozen prior to the merge, even if builds hadn't stopped due to the space issues on bayfront, it still wouldn't have had many substitutes. As part of testing patches and branches [1], I think it would be good to get builds for things like core-updates-frozen happening, that will hopefully improve the substitute availability from bordeaux.guix.gnu.org on average. 1: https://lists.gnu.org/archive/html/guix-devel/2021-08/msg00001.html > =E2=80=A2 Add DNS redundancy for guix.gnu.org so it can point to one of= two > hosts (need to figure out certbot challenges so both machines can > update their certificates). This (in general) is something I'm interested in working out, since it'll be helpful for setting up mirrors for substitutes as well (in the case where you want the mirrors to respond to one common DNS name with working TLS). > =E2=80=A2 Come up with a plan to add disks to the RAID array on bayfron= t, the > head node of bordeaux.guix.gnu.org. The space issue on bayfront that led to builds not happening has now been effectively resolved (see [2]). There's definitely lots of tidying up to do, but I think the situation for storing the nars is much better now. 2: https://lists.gnu.org/archive/html/guix-devel/2021-12/msg00140.html That's not to say there's not something to be gained by upgrading the bayfront hardware, some SSD storage would be ideal to speed up the coordinator and builds. > =E2=80=A2 Work on a plan to mirror nars from ci.guix and bordeaux.guix,= using > plain rsync or . I'm interested in getting bordeaux.guix.gnu.org in to a state where there's less of a discrepancy in performance depending on where in the world it's accessed from. I'm assuming there is some difference in the performance, which is definitely an assumption to check, which is one part of the problem. If it turns out there are some gains to be had, the next step is investigating how this could be approached. Mirrors plus GeoIP based DNS is the approach I currently have in mind. Anyway, even if there isn't a meaningful performance difference, maybe it's worth setting up distributed mirrors for reliability. > =E2=80=A2 Have a documented procedure to set up substitute mirrors, suc= h as > the one in .cn (I can=E2=80=99t find the URL), ideally with plain rsy= nc > access. Getting the nar-herder in to a state where other people might be able to use it is definitely on my list of things to do. I'm assuming here that it's something that people might want to use, and again that's probably worth investigating. If it turns out that people just want to use rsync, it's probably worth assisting with getting that kind of setup working. > Who=E2=80=99s in? :-) Not sure how much time I'll have, but I'll try to be around :) Thanks, Chris --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQKlBAEBCgCPFiEEPonu50WOcg2XVOCyXiijOwuE9XcFAmG7+AFfFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNF ODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcRHG1haWxAY2Jh aW5lcy5uZXQACgkQXiijOwuE9XfiBw//bvOffRbtOmcWFKjPG259GYBZqOcOeWtA 89bWCVWSZFf9SbONGCgz3OQgMEb0w7eusdK2HL0WVA31n7g9QzRRrnZ4jQA7fHEE gBHO3lQeVARfkXTHbEWE5qEiyC6ftyl+aF2n32005xpB/dftA1mdskCgBBDsTAlt Vi93XylbaPCnULskFsPmB/ABuyx3wV/+jakVln+ObwegIYiseF/qf4GuMIZKrQe0 cs467MTd4YEpTE/P6wUWPox5L1XwRAHle+okOfkqNK44GW+W2WTF+KvrL6Kq9Erz +Xz4rBCZvC3zPECvaprIAvjd3Ssz/3U+WQ9qgxGITVYlUNplhOJZd09yHdcOYu1M b2VRNuZdBMenebEjdYDI+bnqo2BAFpbMcxscQ6BhTus5Q8UT/7O8lfmo3tybQ9gS 0zYv052mHJb7BQxfl+J2hsE+VuE5UYUo7zGwgnnOTRiDC8s7nPyXs6ganVHLH2xG vjLIiEb9GAYTRcoQssRJMZ5f7P8+HN1GsnezWUFM6j9sBkpH8imdXAtnQeYiGCss TD36Uv0ocbR77v5vifc3+oZvP47SxNsrMNEEKiv98r9KKuIP5rrbzuXJ+/Xlcvyz xqlhD6OeJns4zx2Let9X1fO1/TC8dcMFLNQWRJLiKa75HfTcR7Fh5QMB3X2ORlPg XBXSbH+kFW4= =5B7s -----END PGP SIGNATURE----- --=-=-=--