From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp0 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id 4AjnIOKQumEdswAAgWs5BA (envelope-from ) for ; Thu, 16 Dec 2021 02:05:38 +0100 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp0 with LMTPS id YIGiHOKQumFAZwAA1q6Kng (envelope-from ) for ; Thu, 16 Dec 2021 01:05:38 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 24F842F446 for ; Thu, 16 Dec 2021 02:05:38 +0100 (CET) Received: from localhost ([::1]:48098 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mxfDF-0004bE-Bu for larch@yhetil.org; Wed, 15 Dec 2021 20:05:37 -0500 Received: from eggs.gnu.org ([209.51.188.92]:41022) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mxfD5-0004ar-1j for guix-devel@gnu.org; Wed, 15 Dec 2021 20:05:27 -0500 Received: from mira.cbaines.net ([212.71.252.8]:56402) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mxfD2-0000ON-OE for guix-devel@gnu.org; Wed, 15 Dec 2021 20:05:26 -0500 Received: from localhost (unknown [IPv6:2a02:8010:68c1:0:8ac0:b4c7:f5c8:7caa]) by mira.cbaines.net (Postfix) with ESMTPSA id 0D7F727BBE9; Thu, 16 Dec 2021 01:05:22 +0000 (GMT) Received: from capella (localhost [127.0.0.1]) by localhost (OpenSMTPD) with ESMTP id 150c121f; Thu, 16 Dec 2021 01:05:20 +0000 (UTC) References: <874k79zs29.fsf@cbaines.net> <86r1adiiv4.fsf@gmail.com> User-agent: mu4e 1.6.6; emacs 27.2 From: Christopher Baines To: zimoun Subject: Re: Mid-December update on bordeaux.guix.gnu.org Date: Thu, 16 Dec 2021 00:20:51 +0000 In-reply-to: <86r1adiiv4.fsf@gmail.com> Message-ID: <87sfutxstt.fsf@cbaines.net> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha512; protocol="application/pgp-signature" Received-SPF: pass client-ip=212.71.252.8; envelope-from=mail@cbaines.net; helo=mira.cbaines.net X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: guix-devel@gnu.org Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: "Guix-devel" X-Migadu-Flow: FLOW_IN X-Migadu-Country: US ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1639616738; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post; bh=zLm4mgFllYVF9FMwuMyVnq9X53Ed0Jink+DRGl40uKA=; b=acCVJHozcCUgjtWL6Ibhtp+xos6uwGkojfFS9NxT5hpyGHFQv5T8VMFPyetOxNxlmwSXzz oyw+PrPX16U5sJrhsEcPJL+R2YOmnMhBNXesHh/8OHTPWK97qSdiKx3szTdiqjN7tP9nVX Nl/rVx6vNQL9ibKiwy8+X/LbB21DmTh/qVz0krZb1o8fHMkkdZhk23aPQyCayAkTtvSoRq 9sQ5SojxToJUu/U1tuXw4I6B/a5atdNo9V0wHR4GylCYAIKuxFaVtOJE4bVTdeY8mIJOu4 mjxAjPSFvb7Dv9Ei9sj2d6jwTYIwJvFF+QhCBFmYkjUcbTblSzwbM+BEt1lQ5Q== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1639616738; a=rsa-sha256; cv=none; b=PCh1GEm33G5wex1w5rX8K0plh7bW7gc0XFnMoLaerlEROh/AdUD6m4xSPKVakng79V3vYM 49mlHbrxgmk/N8Yj0M2+Ur3pQ4LaWDAc4VQCY0vZtLWRiXU8Bj7LYLjJHwW+PRe58gk5Af JBCtQYyd6y7Eo4l4QHaULSXl2sOLUiM3ep5o0YTGJU2PomB/qeGkVrIepHmxpzlEGLFUNI TgTFE2HkMP0F6LrQcfWxHUSVRrHI9FYZ3dnA+FzkkUpamPgTq3ME9Z1vELS7t/zb9suF7e iKa6A066sNZErF1fyxFILjp7VMSC0FXk2yIEGZCWDG4+WQnppNA+PoHtaZMzig== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org" X-Migadu-Spam-Score: -5.09 Authentication-Results: aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org" X-Migadu-Queue-Id: 24F842F446 X-Spam-Score: -5.09 X-Migadu-Scanner: scn0.migadu.com X-TUID: oqX+D082PayK --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable zimoun writes: > Hi Chris, > > Thanks for the update. And for the all work. :-) > > > On Wed, 15 Dec 2021 at 16:48, Christopher Baines wrote: > >> In summary, the space issue I mentioned in the previous update has >> effectively been addressed. All the paused agents are now unpaused and >> builds are happening again. > > The timing had almost been perfect. ;-) > > > Well, as discussed on Sept., one concern I have is about =E2=80=9Clong-te= rm > storage=E2=80=9D =E2=80=93 where long-term is not well-defined and storag= e either. > > Do you think that Bordeaux could run > > The Guix Build Coordinator just builds derivations. I haven't got it to build a manifest before, but that's possible I guess. I think it's unnecessary though, since I believe derivations for all origins of all packages are already being built, that happens through just asking the coordinator to build derivations for all packages, you don't need to specify "source" derivations separately. > ? Having a redundancy about all origins would avoid breakage. For > instance, because Berlin was down yesterday morning, =E2=80=9Cguix pull= =E2=80=9D was > broken because the missing =E2=80=99datefuge=E2=80=99 package =E2=80=93 d= isappeared upstream. I would hope that bordeaux.guix.gnu.org has a substitute for that, could you check the derivation against data.guix.gnu.org, and see if there's a build? Use a URL like: https://data.guix.gnu.org/gnu/store/vhj3gg00hzqfi8lazr3snb9msr4a3q6l-date= fudge_1.23.tar.xz.drv There is one issue though, bordeaux.guix.gnu.org doesn't provide content addressed files in the same way guix publish does. I hope to add that through the nar-herder, and once that's added, bordeaux.guix.gnu.org can hopefully be added to the list of content addressed mirrors: https://git.savannah.gnu.org/cgit/guix.git/tree/guix/download.scm#n368 That would mean that the bytes for a tar archive for example would be available by the sha256 hash, not just as a nar. I'm not sure to what extent this would help, but it's probably useful. >> In general this is an important step in being more flexible where the >> nars are stored. There's still a reliance on storing pretty much all the >> nars on a single machine, but which machine has this role is more >> flexible now. I think this architecture also makes it easier to break >> the "all nars on a single machine" restriction in the future as well. > > IIUC the design, if the proxy server is lost, then it is easy to replace > it. Right? I guess so, the nar-herder helps with managing the data at least which makes setting up new or replacement servers easier. > I remember discussions about CDN [2,3,4,5,6]. I do not know if it > solves the issue but from my understanding, it will improve at least > performance delivery. Well, it appears to me worth to give a try. > > > 2: > 3: > 4: > 5: > 6: Effectively this is moving towards building a CDN. With the nar-herder, you could deploy reverse proxies (or edge nodes) in various locations. Then the issue just becomes how to have users use the ones that are best for them. This might require doing some fancy stuff with GeoIP based DNS, and somehow sharing TLS certificates between the machines, but I think it's quite feasible. >> Going forward, it would be good to have an additional full backup of the >> nars that bayfront can serve things from, to provide more >> redundancy. I'm hoping the nar-herder will also enable setting up >> geographically distributed mirrors, which will hopefully improve >> redundancy further, and maybe performance of fetching nars too. > > To me, one first general question about backup coordination is to define > a window for time: > > - source: forever until the complete fallback to SWH is robust; > - all the substitutes to run =E2=80=9Cguix time-machine --commit=3D<> --= help =E2=80=9D > for any commit reachable by inferior: forever; > - package substitute: rule something. The idea I've been working with so far is simply to store everything that's built, forever. Currently, that amounts to 561,043 nars totaling ~2.5TB's. How feasible this is depends on a number of factors, but I don't have any reason to think it's not feasible yet. > Thanks for taking care about redundancy and reliance of CI. There's not a relationship to continuous integration yet, although I am hoping if the building and serving substitutes stuff stabilises, bordeaux.guix.gnu.org might be able to play a part in testing patches and branches (as discussed in [1]). 1: https://lists.gnu.org/archive/html/guix-devel/2021-08/msg00001.html Thanks for all your comments and questions! Chris --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQKlBAEBCgCPFiEEPonu50WOcg2XVOCyXiijOwuE9XcFAmG6kM5fFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNF ODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcRHG1haWxAY2Jh aW5lcy5uZXQACgkQXiijOwuE9Xcs3BAAuYayyrAj5RyHWGruOtWS7ZGIrh1xMopb QtzQf8DGC1Rh+RRd860Zd5pZ5vj8SkRXoEW1DmDVkoEzDi3NEeFoIrodT+CHXCVN waOqWrN9Hr1G+Cy/SyPFw76j2eRsRGN1IrGMzlt0kFb4I3563eGOaXdokbeIhqeg N7WXygNGV1dFgvYFNHJfEihwQtcXAZF5wqT94yMj/SppUXjfKuqvm2jK9k8pkHoM RHCgTh49fFeDeuq1XHcp93eG4cC7T7vE/+cVmfhLuRmqG5rTz0D+e0Lr0WXv9S43 0m5V+ekwEZICCLGLR8asuO30EqvCLKO/ry8Ktg5fVrqllkqR+GFRf7GGzHsU7uvB B4bsyHq8OsoGRo6Gkl+x5BfdGmR8eZZpLhG97WZyFgaW1BVKT6w25HOSOlTIPKW0 sb+4lx095yusS/xDGQLI4g0rGqoCLMxucMlJVSrwruitc7DSLcAvVdMEzs0E9SLN QnJ1AnvMjqOPs6pipIfz+QfRo6zfM3cO1Es+9d/fogkS2SQ6duEKzi8G4SU0c2KV K/xS2wMwPZ19oDOvhOBOfp2iVIVU6mga1kvZd3aF5g9WQE28Zn4IZpOGu0khb4/0 DM56r4WCJkKgvut/VT50Ex4hO1gw4tOIX6uj5QTQNmxF9FQtxEpGfhwZyavEJBzu GDi6gYKVFPo= =Y5Eu -----END PGP SIGNATURE----- --=-=-=--