From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp1 ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id EEnuCPqaSWEIHQEAgWs5BA (envelope-from ) for ; Tue, 21 Sep 2021 10:42:34 +0200 Received: from aspmx1.migadu.com ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp1 with LMTPS id WJaPBPqaSWFYZAAAbx9fmQ (envelope-from ) for ; Tue, 21 Sep 2021 08:42:34 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 7C08F1D5A7 for ; Tue, 21 Sep 2021 10:42:33 +0200 (CEST) Received: from localhost ([::1]:38642 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mSbMG-0001si-KQ for larch@yhetil.org; Tue, 21 Sep 2021 04:42:32 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:42894) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mSbM0-0001sX-SS for guix-devel@gnu.org; Tue, 21 Sep 2021 04:42:16 -0400 Received: from mail-wr1-x42c.google.com ([2a00:1450:4864:20::42c]:37414) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1mSbLy-0002LJ-OY; Tue, 21 Sep 2021 04:42:16 -0400 Received: by mail-wr1-x42c.google.com with SMTP id t8so36834230wrq.4; Tue, 21 Sep 2021 01:42:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:in-reply-to:references:date:message-id :mime-version:content-transfer-encoding; bh=soNB90CtYWnE/jbNxSffnaLMxVqPF8+tFDu18oNInvg=; b=Rk2PeYNvwFVzWX93qnIj481ems8HFWMfALC7snAkJoNl5icj9q72gqyNKHAHQi4eid 68oMFKRbz7Mtdq+/uww7Naddg77kc0pnF5WArSF8Uv6mglctF5YaIcf77o1LvnMETXD5 Do45QbThxo1OeT0tT7cH7XMMvPjzbdbd/w7y9SRjaLFz+udAuQyUVig6c/BiWCx7rW+W N93CnktqHMdPYIkkQGZAhtwh9IdAwY32TWfPlUZXmUHoD2Q/g6ybg+jekjSvvs6f+2/D ABBh9V6benTrB5R7U/xDRIV55G/7qJJQl0FdtrT1gv1vP8KTcSVbj2IDHJA6S710/j/K imoA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:in-reply-to:references:date :message-id:mime-version:content-transfer-encoding; bh=soNB90CtYWnE/jbNxSffnaLMxVqPF8+tFDu18oNInvg=; b=XMnPL3fPBKe/z+/yY8/k7d7gCFLLZTSLbG8SceAr0zWVi0/VweKEcEYf7vHheSxNXf tSdx4UDbJVkXVqcjRlPpWUC+V6aoHtiHl40+ehBYmnLpLwkjlLcXdWl/4XY/xoTjfva3 I6OpJtz76tyTqjJxwZ9d/GypV5Eu6vmGOcdlb9n/dueBlNXHNs5hm8i2n/Kgei8ZNXqF MNX9vzs3GNeVybePRQ/gccR39lVGPpG4sz8UOv4BH3TnpgBzNpgMOUukuqrcWFKZUfye MWwesNTT1JFkiQFjCGP78C4qY/2l1SpTkrJKv7UBbZ+9Mrt6ZajGLfi46i0YKPFx1rot YKag== X-Gm-Message-State: AOAM5338tRXOjTBYGvP/B+cF0qG+gE30h2uvOwQX3hpFw7rENyB3EyGH GuHkVYsmOLzBtRGIUPpaA2Eaf6yVK7YxhA== X-Google-Smtp-Source: ABdhPJySMWBJaSXDo8slskt9aAt8qUejMBqQcpqgqZWn8KLn2H14XDiGfKwR8NhcJHsmpV0GlqxpiA== X-Received: by 2002:a1c:e915:: with SMTP id q21mr3305069wmc.180.1632213732325; Tue, 21 Sep 2021 01:42:12 -0700 (PDT) Received: from lili ([2a01:e0a:59b:9120:65d2:2476:f637:db1e]) by smtp.gmail.com with ESMTPSA id a25sm2011974wmj.34.2021.09.21.01.42.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 21 Sep 2021 01:42:11 -0700 (PDT) From: zimoun To: Christopher Baines , Mathieu Othacehe Subject: Re: guix weather -m etc/sources-manifest.scm and CI? In-Reply-To: <87sfy3y0nn.fsf@cbaines.net> References: <87sfy3y0nn.fsf@cbaines.net> Date: Tue, 21 Sep 2021 10:32:55 +0200 Message-ID: <86zgs6ba6g.fsf@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Received-SPF: pass client-ip=2a00:1450:4864:20::42c; envelope-from=zimon.toutoune@gmail.com; helo=mail-wr1-x42c.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: guix-devel@gnu.org Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: "Guix-devel" X-Migadu-Flow: FLOW_IN ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1632213753; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post:dkim-signature; bh=soNB90CtYWnE/jbNxSffnaLMxVqPF8+tFDu18oNInvg=; b=GwUvtAlj6e0rdUT3VjE6akPUcUDFnOI/tGJ7iDScsyLHFEMxulwGIVm8eJfZpf5oMRw4zw 5NxCvXDtJKaS8VUi7A/6myzXrq/7hWzMYrMs+i5+D6LkTeWYS/eLpxfmJfJXC+o8uXRclg E1Nra7DifKqTPcq2EAaAyDikaYwasmYBVCnJDgHdTcShFKWo+dW3KOAobpRIi1fUS2NoyS RwqZEggwvUDF1qGojK7WXJc6sdHYsI/TOPiSG0uGkC9uf8t1tWrJdbo2fF24YyhvL/WhgY X8vfD43mY6An5NjNFh2uH4VWTZcasQWWaoTKC7D2IY3oJbForzRpyJjynGmlYg== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1632213753; a=rsa-sha256; cv=none; b=bin3oWyGtAhnA9nE6j0omZHPeSU19DHyvaCs2fZsfoVkMxgNPr9w/HNhEC2prV+xJ82Ebf QSnGpRiHryk1TbCwN0SaTezimW4BwJqBaBsrMwbLTrN7yDE/8WF8BUJHyaUEhXkLONVXvm s8dgty6yjM59klqAWgxSof7pIf+hPAWnAryDdvl5dsiYsmkWafdDr3wB08urIwfy7JvAVz wk1p8T9dd8dlFLD9GHVG0ClYK7+KmXS6fN0Px4yCMhIYKbdfcOuq71GQiwbzw+9ZvZqObg ZmbBXdCSug1vReXKT9/F46XIpS3Dt8N1cNukvNJHMyfpVKMTTnqWHhivD/XRRA== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=Rk2PeYNv; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (aspmx1.migadu.com: domain of guix-devel-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-devel-bounces@gnu.org X-Migadu-Spam-Score: -0.59 Authentication-Results: aspmx1.migadu.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=Rk2PeYNv; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (aspmx1.migadu.com: domain of guix-devel-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-devel-bounces@gnu.org X-Migadu-Queue-Id: 7C08F1D5A7 X-Spam-Score: -0.59 X-Migadu-Scanner: scn0.migadu.com X-TUID: iNWqHj0fV3bJ Hi Chris, On Fri, 17 Sep 2021 at 10:49, Christopher Baines wrote: > zimoun writes: >> https://ci.guix.gnu.org >> 74.6% substitutes available (12,556 out of 16,831) [...] >> https://bordeaux.guix.gnu.org >> 99.8% substitutes available (16,804 out of 16,831) >> The questions are: >> >> Why ci.guix.gnu.org contains only 75%? And bordeaux almost everything? >> (I guess the missing ones on bordeaux are corner cases as icecat, linux-= libre). > > bordeaux.guix.gnu.org is hopefully only missing substitutes for things > where there's actually an issue building them. It's possible not to > guess though and instead ask guix weather what is missing. [...] > If you visit https://data.guix.gnu.org and add the store path to the > URL, you can get links to the failed builds ([1] for example). > > 1: > https://data.guix.gnu.org/gnu/store/42knh9b75m6kc30m8v247sswhdfqnn8i-xpp3= -1.1.4_src.tgz Cool! Thanks for explaining. Let try to systematically address this failures about the source. :-) >> Does it make sense to duplicate the storage of all these origins? >> >> Using extensively "guix time-machine", I note that a lot of >> derivations are missing and thus they are built locally, which is >> costly on poor machine. Could we reduce the duplication and so save >> some space in order to systematically keep these derivations? It >> would greatly ease the Guix experience for "guix time-machine" users. >> :-) > > In terms of duplication, I still think that an argument can be made that > bordeaux.guix.gnu.org is providing value, even if it's doing some of the > same stuff as ci.guix.gnu.org. My question with duplication is about =E2=80=9Clong term storage of all the sources=E2=80=9D. Not about building twice (it is easy to find justificati= on via reproducibility checker for instance :-)). Now Guix have 2 build farms so let consider it is a strength. :-) The question is how to exploit such strength by sharing the workload and have a better global scaling up (and not 2 independent scaling). For what my opinion is worth here. :-) > As for keeping build results, everything that's ever been built for > bordeaux.guix.gnu.org (that's only ~337739 things totalling ~1.4TiBs) is > still around. In some ways, this is because deleting things is a bit > more difficult, as the files aren't in the store, you can't just do a > guix gc. > > However, I do want to try to never delete anything. That's going to > require a bit of work as the local storage on the bayfront machine will > be all used up at some point, but I do have the beginnings of a plan to > avoid this an keep building things. >From my understanding, the project does not have a lot of resources; especially about storage. And my question is: do we have a =E2=80=9Ccohere= nt=E2=80=9D plan between the machines behind Bordeaux and the ones behind Berlin? I mean, IMHO, it does not seem a correct usage of project resource to back twice all from v1.0 to now (maybe from earlier to later). And I also have in mind energy consumption which is becoming more and more a precious value. For instance, we could imagine the both build farms build and challenge the outputs. Then, on Berlin the identical outputs are deleted after say 6 months and only live on Bordeaux. Bordeaux being the backup time-machine. That=E2=80=99s only an idea to open the discussion. :-) Not a concrete plan. Moreover, Disarchive is also something that should be added; probably to machines behind Bordeaux in the above plan. >From my point of view, what is really missing are the derivations for *all* the commits; for example module-import.drv, guix-33bc3fb2a.drv, guix-command.drv, guix-module-union.drv, guix-33bc3fb2a-modules.drv, guix-packages-modules.drv, guix-system-tests-modules.drv, guix-packages-base-modules.drv, etc. And the backup time-machine should systematically build them for all the reachable commits. Cheers, simon