From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp2 ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id wKCMDftJSmGXAAEAgWs5BA (envelope-from ) for ; Tue, 21 Sep 2021 23:09:15 +0200 Received: from aspmx1.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp2 with LMTPS id GOcLCftJSmGGDwAAB5/wlQ (envelope-from ) for ; Tue, 21 Sep 2021 21:09:15 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 9DA332CB81 for ; Tue, 21 Sep 2021 23:09:14 +0200 (CEST) Received: from localhost ([::1]:56626 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mSn0r-0004Iz-Og for larch@yhetil.org; Tue, 21 Sep 2021 17:09:13 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:53204) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mSn0B-0003re-MH for guix-devel@gnu.org; Tue, 21 Sep 2021 17:08:31 -0400 Received: from mira.cbaines.net ([212.71.252.8]:37390) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mSn09-0000mJ-EL for guix-devel@gnu.org; Tue, 21 Sep 2021 17:08:31 -0400 Received: from localhost (unknown [IPv6:2a02:8010:68c1:0:8ac0:b4c7:f5c8:7caa]) by mira.cbaines.net (Postfix) with ESMTPSA id BB8E227BBE1; Tue, 21 Sep 2021 22:08:27 +0100 (BST) Received: from capella (localhost [127.0.0.1]) by localhost (OpenSMTPD) with ESMTP id bfe03420; Tue, 21 Sep 2021 21:08:27 +0000 (UTC) References: <87sfy3y0nn.fsf@cbaines.net> <86zgs6ba6g.fsf@gmail.com> User-agent: mu4e 1.6.5; emacs 27.2 From: Christopher Baines To: zimoun Subject: Re: guix weather -m etc/sources-manifest.scm and CI? Date: Tue, 21 Sep 2021 21:36:33 +0100 In-reply-to: <86zgs6ba6g.fsf@gmail.com> Message-ID: <87bl4ly6uu.fsf@cbaines.net> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha512; protocol="application/pgp-signature" Received-SPF: pass client-ip=212.71.252.8; envelope-from=mail@cbaines.net; helo=mira.cbaines.net X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: guix-devel@gnu.org Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: "Guix-devel" X-Migadu-Flow: FLOW_IN ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1632258554; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post; bh=V7EtysjdkqDT/NcecfUHN+3Bk7y0eEBnPwGf3zK4/po=; b=lojL7KeZEaai3CHRtOswUBHk+BX5eqNjjJ2P9CpXp06QZuwNKMNy32QJW245mE384fmdp1 jRnB5H3vme41nmCSufZOeEpgr1xcl9J/wYNkTtkzpsyXDVKMwhWZa8jjiKqjwm2NQ4zfOs 5czvB2pOhDojhw/xVq/UHJ3uf1Uf6AHXrsDHoD1fQPKnuURvSCBoPwHf2EaJRODpLZIDG8 JeUn1GYM3FZUFr7dhvkxDEZyfUEUwkUxRvBqzbAAkBcMGojLmhKw5vZgMCRbE3Nz25TKdw ROQLltUcKg015omThqreWx1ZVc1aa+soej0gcmFL3JZ6nb+ROdQJNp4+ROrXlg== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1632258554; a=rsa-sha256; cv=none; b=LJRXd8qYURoTFe0CNLZoqB2/dgwgG69ltaNOQH/TwhQ2SsaoIps0uR/VU07MYNbw8jbs+R G9FijY4Xq6AHcM1lqgcmRlZRGH6oX5SlP0ZEnrf6Eyh8ZhMnkCdUuWRTgaKGfhL7cFpOyU SP5RokU0/xX1us4oGjWkcovwgXcf3LTcoFJvfT6/n1RWqiFYish0Gh0U5+xglQ9mqdx/pO 8Ye6TtD6ZSb4yqn9jDqkyRHC03tjuNeTDMtmxzm1urA5o7A9r6ywWZ7pbWmmDoO/KZpIGw hcT6wdikX1GO7VwrW1/BV819dG7rNZU9NLggtRBweUneeoA4lU5SrXNlu8kTeQ== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of guix-devel-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-devel-bounces@gnu.org X-Migadu-Spam-Score: -3.49 Authentication-Results: aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of guix-devel-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-devel-bounces@gnu.org X-Migadu-Queue-Id: 9DA332CB81 X-Spam-Score: -3.49 X-Migadu-Scanner: scn0.migadu.com X-TUID: 3xAF79GSr/KE --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable zimoun writes: >> As for keeping build results, everything that's ever been built for >> bordeaux.guix.gnu.org (that's only ~337739 things totalling ~1.4TiBs) is >> still around. In some ways, this is because deleting things is a bit >> more difficult, as the files aren't in the store, you can't just do a >> guix gc. >> >> However, I do want to try to never delete anything. That's going to >> require a bit of work as the local storage on the bayfront machine will >> be all used up at some point, but I do have the beginnings of a plan to >> avoid this an keep building things. > > From my understanding, the project does not have a lot of resources; > especially about storage. And my question is: do we have a =E2=80=9Ccohe= rent=E2=80=9D > plan between the machines behind Bordeaux and the ones behind Berlin? Not that I'm following at least. The machines behind bordeaux.guix.gnu.org were just ones that I could find to use when the plan of dpeloying behind ci.guix.gnu.org was looking less and less likely. > I mean, IMHO, it does not seem a correct usage of project resource to > back twice all from v1.0 to now (maybe from earlier to later). And I > also have in mind energy consumption which is becoming more and more a > precious value. > > For instance, we could imagine the both build farms build and challenge > the outputs. Then, on Berlin the identical outputs are deleted after > say 6 months and only live on Bordeaux. Bordeaux being the backup > time-machine. That=E2=80=99s only an idea to open the discussion. :-) No= t a > concrete plan. Some duplication of stored data is going to be necessary for redundancy I think, for bordeaux.guix.gnu.org I was thinking of at least trying to store all the substitute data in two different places. As you've mentioned, building the same things multiple times is also valuable for testing reproduciblity. > Moreover, Disarchive is also something that should be added; probably to > machines behind Bordeaux in the above plan. I haven't been following Disarchive too closely. I think I remember thinking it might be interesting for it to listen to data from data.guix.gnu.org, to find out about new source tarballs (which isn't really possible yet). > From my point of view, what is really missing are the derivations for > *all* the commits; for example module-import.drv, guix-33bc3fb2a.drv, > guix-command.drv, guix-module-union.drv, guix-33bc3fb2a-modules.drv, > guix-packages-modules.drv, guix-system-tests-modules.drv, > guix-packages-base-modules.drv, etc. And the backup time-machine should > systematically build them for all the reachable commits. If you're just talking about the derivations, data.guix.gnu.org does store these derivations for the commits it processes (which are just the states of the master branch, and that data is also missing for some older commits that were processed a while ago). If you're talking about substitutes for the outputs of these derivations, the situation on bordeaux.guix.gnu.org isn't great for x86_64-linux currently, since building guile@3.0.7 is blocking building these things for recent revisions [1]. The underlying reason is a problem with the testsuite I think, but it's hard to work around for some other reasons as well. Once it can be built, all the blocked builds will be attempted too. 1: https://data.guix.gnu.org/gnu/store/5z950sh1wpi59mfh53qdjdn8qnlcwp8g-gui= le-3.0.7.drv I am interested in what can be done to support using guix time-machine though, I think there's a range of things like storing substitutes, testing things better (so they don't start failing to build in the future), and helping users find the right commits to use. Chris --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQKlBAEBCgCPFiEEPonu50WOcg2XVOCyXiijOwuE9XcFAmFKSclfFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNF ODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcRHG1haWxAY2Jh aW5lcy5uZXQACgkQXiijOwuE9Xcl/w/+JpNnVOUNVN2jeXMzVY/kmC75d6c08nMK ZHwooMWXf7KjBiU/4+waZGKKO6456yY7RetOs2TRyTE18rFXi6277GNmeo7b+q2I EhL4AkYo1lh8OFA+7QHz3Nj2RNnpBMoitnNFB7cwlc4B2wVxynv5aiut9NooZQxN vkrlPxMYPtgxkvTPdnDcWF9K53CGK/bz8FzOz7zyyUz3znajNithitCkp+FQQSuP fjmCco4inNnpzEf41zHMa2AEFBMnaovQlkRc/ed9vg1Msl7kzY4OU7NJ69zOoTND vSg3JyM+uqTg0gnDsXBo6MVAUPgs9y0jCr3ZUBCgTvLzrRzQohklq9lDFcoCErbn xWqdy8YMjWxCI5+qM45FdufeLTMjASUj+XyjYMzj7p86WGkPPr6L1R/Jtd6Jsf8v bKdtOlZ10MAjfJTitiQ3Iejr+deMRzHT52VwZc4Ka8bmG01twBwF5QnEAX99OUTe b6DBXBodDA9qBmivF9yum8MuaFZ8LAaBDNKuXoC9V2k4xF2C+UT7yoGNmx46mhhW tznsk2xg7WR2jKfoFDSULaPq090/FUHc/HIK77qhADllg7ejqRY/ReQqc6ze2oqb 8TymhextHUlrdQuA2Os5XUD8l+FQ8MPsb8Y8XsV5zQ4zlGoZyitIzGBnVCDbW1pv xQJNYiJgg74= =/EfM -----END PGP SIGNATURE----- --=-=-=--