From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp1 ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id iM/NJSM4u2F82gAAgWs5BA (envelope-from ) for ; Thu, 16 Dec 2021 13:59:15 +0100 Received: from aspmx1.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp1 with LMTPS id +LFcISM4u2HyXwAAbx9fmQ (envelope-from ) for ; Thu, 16 Dec 2021 12:59:15 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 2A7EEF92D for ; Thu, 16 Dec 2021 13:59:15 +0100 (CET) Received: from localhost ([::1]:39250 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mxqLp-0005pD-Qa for larch@yhetil.org; Thu, 16 Dec 2021 07:59:13 -0500 Received: from eggs.gnu.org ([209.51.188.92]:55328) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mxqLG-0005kq-Dh for guix-devel@gnu.org; Thu, 16 Dec 2021 07:58:38 -0500 Received: from mira.cbaines.net ([212.71.252.8]:56404) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mxqLE-0002IZ-4K for guix-devel@gnu.org; Thu, 16 Dec 2021 07:58:38 -0500 Received: from localhost (unknown [IPv6:2a02:8010:68c1:0:8ac0:b4c7:f5c8:7caa]) by mira.cbaines.net (Postfix) with ESMTPSA id DD60D27BBE9; Thu, 16 Dec 2021 12:58:31 +0000 (GMT) Received: from capella (localhost [127.0.0.1]) by localhost (OpenSMTPD) with ESMTP id da0f259c; Thu, 16 Dec 2021 12:58:30 +0000 (UTC) References: <874k79zs29.fsf@cbaines.net> <86r1adiiv4.fsf@gmail.com> <87sfutxstt.fsf@cbaines.net> <86k0g4izdk.fsf@gmail.com> User-agent: mu4e 1.6.6; emacs 27.2 From: Christopher Baines To: zimoun Subject: Re: Mid-December update on bordeaux.guix.gnu.org Date: Thu, 16 Dec 2021 12:48:34 +0000 In-reply-to: <86k0g4izdk.fsf@gmail.com> Message-ID: <87o85gyadn.fsf@cbaines.net> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha512; protocol="application/pgp-signature" Received-SPF: pass client-ip=212.71.252.8; envelope-from=mail@cbaines.net; helo=mira.cbaines.net X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: guix-devel@gnu.org Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: "Guix-devel" X-Migadu-Flow: FLOW_IN X-Migadu-Country: US ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1639659555; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post; bh=bjzV/N4WHxlg+nZCnIfKLAUpTEKbNKOnnRmkQjwI/PM=; b=BedBUZFw3y0ciYNRp/EZPySPM9Bju7XX/dMDb0Dj5cD4NL9jwjY68RMYo8jeuH8SRPiEDj AbzzqC/Yctgqw8ZM43zP6qXU+sxOkDDHkgeuKpsLq2wtbwHgNL6+Owo84Ti0phYAmStUV5 0wP7kia7PBONRwEAYfGbdIZ9k+wRSf3cM16L5XeG1coae8jsuTd+aa8AWD4H/uVgguc/hW dzMVzCFl3gdaqvO2WByIc+zHZSgWgi1CVqLgHgfOVWubNWVupvyEu91BG8qRNsSITPR/N1 VxiGJ+5+122v2dCBZasBMRlUNeDUznHVYdFDG4vRg9rgyW1W/wfJs2SS6hE//Q== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1639659555; a=rsa-sha256; cv=none; b=TkbjUKicLRQADzgDf3pI+G540eDYygDvi7WVL2If/DLR6o5H5x3CO3y8wDUd4i8C3PYxji SsrRV6t/00kH7cczG6nd6JCu6vobhM9cnOI6M8X4sQgmWByXXmHEKR7TIWrMHsCozJ/lPN uTuHfPwuKDFJjvNDnYFyMJYKILTkkZHIQfWEKyUbM4NjrXI1q8++OBGck48u3ImLywxciO AMOXUZj6zZcwwWfstnneDzTjviFb+HI1sCgnre33TT465vjlkyswP11saxnt15IYQdmR3b Dug/o5xgMM1VRASFxdLVLRESnKQ50OSaFpAvSjDm1T4nQ1vK5crNz+bSeu7B8w== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org" X-Migadu-Spam-Score: -5.59 Authentication-Results: aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org" X-Migadu-Queue-Id: 2A7EEF92D X-Spam-Score: -5.59 X-Migadu-Scanner: scn0.migadu.com X-TUID: Pd++QRQWqCiQ --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable zimoun writes: > Hi Chris, > > On Thu, 16 Dec 2021 at 00:20, Christopher Baines wrote: >> zimoun writes: > >>> Do you think that Bordeaux could run >>> >>> >> >> The Guix Build Coordinator just builds derivations. I haven't got it to >> build a manifest before, but that's possible I guess. > > I am not sure to understand since Cuirass also builds derivations and > the purpose of this source-manifest.scm is to let Cuirass ingest all > sources, > > > >> I think it's unnecessary though, since I believe derivations for all >> origins of all packages are already being built, that happens through >> just asking the coordinator to build derivations for all packages, you >> don't need to specify "source" derivations separately. > > Your assumption is wrong, IMHO. We have many failed examples and at the > end Ludo wrote this source-manifest.scm to be sure that all is ingested > for sure. What assumption? I believe the reason the source-manifest.scm thing is useful to use with Cuirass is that it has something to do with Cuirass copying the outputs from those builds back to where they're served from, and maybe registering GC roots as well. I could be wrong though. I'm saying that this additional thing is unnecessary when using the Guix Build Coordinator to build packages, since at least with the configuration for bordeaux.guix.gnu.org, it'll build the sources, store and serve them without any extra effort. >>> ? Having a redundancy about all origins would avoid breakage. For >>> instance, because Berlin was down yesterday morning, =E2=80=9Cguix pull= =E2=80=9D was >>> broken because the missing =E2=80=99datefuge=E2=80=99 package =E2=80=93= disappeared upstream. >> >> I would hope that bordeaux.guix.gnu.org has a substitute for that, could >> you check the derivation against data.guix.gnu.org, and see if there's a >> build? Use a URL like: >> >> https://data.guix.gnu.org/gnu/store/vhj3gg00hzqfi8lazr3snb9msr4a3q6l-d= atefudge_1.23.tar.xz.drv >> >> There is one issue though, bordeaux.guix.gnu.org doesn't provide content >> addressed files in the same way guix publish does. I hope to add that >> through the nar-herder, and once that's added, bordeaux.guix.gnu.org can >> hopefully be added to the list of content addressed mirrors: >> >> https://git.savannah.gnu.org/cgit/guix.git/tree/guix/download.scm#n368 >> >> That would mean that the bytes for a tar archive for example would be >> available by the sha256 hash, not just as a nar. I'm not sure to what >> extent this would help, but it's probably useful. > > Thanks for explaining the details. From a pragmatical point of view as > end-user, =E2=80=9Cguix pull=E2=80=9D must Just Work whatever the plumbin= g. > > For instance, some of us spent energy to =E2=80=9Cevangelize=E2=80=9D in = scientific > communities how Guix is awesome. Next morning, people give a look and > bang! Because a tiny and short outage. Obviously, =E2=80=9Cshit happens= =E2=80=9D(*) > and it is really really really sparse but too late the damage is there. > > My feeling here is that both build farms work independently instead of > coordinate the usage of resources and exploit this strength to have two. I too want to coordinate, although I think that having two independent build farms is actually good for reliability. >>> I remember discussions about CDN [2,3,4,5,6]. I do not know if it >>> solves the issue but from my understanding, it will improve at least >>> performance delivery. Well, it appears to me worth to give a try. >>> >>> 2: >>> 3: >>> 4: >>> 5: >>> 6: >> >> Effectively this is moving towards building a CDN. With the nar-herder, >> you could deploy reverse proxies (or edge nodes) in various >> locations. Then the issue just becomes how to have users use the ones >> that are best for them. This might require doing some fancy stuff with >> GeoIP based DNS, and somehow sharing TLS certificates between the >> machines, but I think it's quite feasible. > > Considering the human resource vs the money resource, it appears to me > better to invest the human energy into things that do not exist and rely, > for now, on existing solutions; even if the project has to pay money. > > For what my opinion is worth here. I'm not against paying for some CDN, although I'd prefer not to pay, or pay less. >>> To me, one first general question about backup coordination is to define >>> a window for time: >>> >>> - source: forever until the complete fallback to SWH is robust; >>> - all the substitutes to run =E2=80=9Cguix time-machine --commit=3D<> = -- help =E2=80=9D >>> for any commit reachable by inferior: forever; >>> - package substitute: rule something. >> >> The idea I've been working with so far is simply to store everything >> that's built, forever. > > For sure, anyone wants that at the end. My point is raising the > priority of the intermediary steps. > > >> Currently, that amounts to 561,043 nars totaling ~2.5TB's. >> >> How feasible this is depends on a number of factors, but I don't have >> any reason to think it's not feasible yet. > > That=E2=80=99s exactly my point! It is not about feasibility =E2=80=93 a= ll is doable > with enough time and energy =E2=80=93 but instead it is about controlling= the > factors to =E2=80=9Crobustify=E2=80=9C what the project consider highly i= mportant =E2=80=93 as > keep all sources or never break =E2=80=98guix pull=E2=80=99 =E2=80=93 wha= tever the status of > infrastructure. > > > Again, thanks for all the work. Becausse, in any case, for sure, the > situation is daily improving. :-) > > Cheers, > simon --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQKlBAEBCgCPFiEEPonu50WOcg2XVOCyXiijOwuE9XcFAmG7N/RfFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNF ODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcRHG1haWxAY2Jh aW5lcy5uZXQACgkQXiijOwuE9XfzHxAApTwAZnwhXtSCAorqCCcRhu5ai5HZw8mw 1MPTYYKq08niiVUOYpsVViRz7488hdoN9bpC4QuCy9zgaDupE3W4FWEUrJ9BCp1y UaPVT2jnD5OuIc8iub3IWN/oxr11ooqge7udrhYlSKAy/saOvbuS9aZy3vYNj5v8 4kwR6cKUFFoyImYTseu8EEV1jVcbTDge9kqW0kwhpUlFMl0FR5iSFKBTujndeZ1y gwl39jf1EBQcAD5I+NUjIRtc9bGEj+9EfROXC+OmuclX4ZyfwiA9/Vbup3eWV7cR iEJ/YWMkSbHpK/2BLwkBpV7iKd5p+9/BMkd5tWz+sR3zTzbhSVR3YAiP4lnivg/Z tAz2MIil4suo0zPddHKTa/0w4qtzloPZpNgU7fWTotzwmwSZ9pKktie/E6OvbFUu Oay4KPeTrVq0S3kOphDZTA3ghYXe9ZSQDJqhB7PIgd/P/3StwGzr28J5G7uHwLYg XNe/yyiRL7mDQZ9OSYWhV5/wzU84DmQe6YSAkLIbh6jDRiMkAYXJbK6cf9/ktTkm elwLbyWBdemWGa5kADDUgZPdokSmguSDGPc+BTipRF8wSlNQwykDdJ7hfqR0d1A0 r9IpO6hM0tjewxaCDws5ywjYdutlSzP47nol8T52eFhXVkSfnEXxPjHzE5SmKgm9 LWbfCk5Aku8= =Bw3n -----END PGP SIGNATURE----- --=-=-=--