From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp0 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id 8B8+DO/QZWGLggAAgWs5BA (envelope-from ) for ; Tue, 12 Oct 2021 20:16:15 +0200 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp0 with LMTPS id GMvsB+/QZWHCNwAA1q6Kng (envelope-from ) for ; Tue, 12 Oct 2021 18:16:15 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id B669F39FC0 for ; Tue, 12 Oct 2021 20:16:14 +0200 (CEST) Received: from localhost ([::1]:56870 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1maMJx-0002Yc-QE for larch@yhetil.org; Tue, 12 Oct 2021 14:16:13 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:37098) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1maMJh-0002Wq-Qo for guix-devel@gnu.org; Tue, 12 Oct 2021 14:15:57 -0400 Received: from mail-wr1-x434.google.com ([2a00:1450:4864:20::434]:44912) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1maMJd-0003Qe-I0 for guix-devel@gnu.org; Tue, 12 Oct 2021 14:15:56 -0400 Received: by mail-wr1-x434.google.com with SMTP id e3so36743wrc.11 for ; Tue, 12 Oct 2021 11:15:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:in-reply-to:references:date:message-id :mime-version:content-transfer-encoding; bh=obI0I5NIiPASJDmVK35tf7yMqhGmkHJeL5dvKB3F8MU=; b=pdr79jYq1Hg6e3Kqv4HbvV5lf4AUK30Ycv/zFIKfFLhfdDv/YqwnxQBniCATXvEJw8 Q61xeUg2U6tw71eOVzZKS8WpJV7g2Y4x/XtbwFDGX7uYGELhg1GElERjizgL7B4sy99S A8coo1JVnIw9ckpp3lKf7ymj5ehFTvnsF/LGIp/DwQ3wJxHDYevHMjL3t2G4E9DNsL4U SjFaWlYgjDXfaQGN64PYBf5H85bF9dCle4g02LsSaqKkYoFtwLU0kUomeNf0ablJfWCd 49kmTrDVWpvu0kdrAPh6Ock+9kFVdWeWZLaqMed1Zt6/t06j2h/w18w4OqzgvuX313lh ebhQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:in-reply-to:references:date :message-id:mime-version:content-transfer-encoding; bh=obI0I5NIiPASJDmVK35tf7yMqhGmkHJeL5dvKB3F8MU=; b=y1xc/B+P7o4mXqFN8DSJ5sonYLrzIn+7FphBe9BiAC4PklG2xu7ASZ6CyMj9Bfue/f XnFSXtDqxMwMNVH06d3vXcM30wvehizlcRbHWfLV6CUlJQkUkd5hE53RA6BW7++0+nIM 7Kwu8kHFSn0sZhHa8/SSU5BL4oIwTh2EEMlUbqkQqPMUdfX9vjKOkzPMCsvVYxhW7XQm vn0hFCSRZNfME/wotGVQT/gIG4fqdypoLPW/Uuo+f2bKWusbCG5DE5+67/3YZet0KbQM KL6txBH4OLGlghXZjjIWIlxOtXT0OwFc8jvy7VVl2g8uYIDtvB6a2pIXLbHzycQtNQrl UaVg== X-Gm-Message-State: AOAM532pqNpe980AQPtsrF2l3p7DNqiY9yH+n9q1QxOpUFn9cTyNyE1E sIU4kIw+Q2ZaKzUVpYSeP0UBJQ1/Rlzn7Q== X-Google-Smtp-Source: ABdhPJw3qI70c21J7WAynveUyZyOplpjp1LTgzcCoZ1haIlY+QQj4jix/6sADp9WxJTGN6e9sLnyDA== X-Received: by 2002:adf:df8c:: with SMTP id z12mr29596282wrl.292.1634062551831; Tue, 12 Oct 2021 11:15:51 -0700 (PDT) Received: from lili ([2a01:e0a:59b:9120:65d2:2476:f637:db1e]) by smtp.gmail.com with ESMTPSA id j14sm5780927wrw.12.2021.10.12.11.15.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 12 Oct 2021 11:15:51 -0700 (PDT) From: zimoun To: Ludovic =?utf-8?Q?Court=C3=A8s?= Subject: Re: Substitute retention In-Reply-To: <87y26ytek6.fsf_-_@inria.fr> References: <87a6tdce94.fsf@inria.fr> <87mu4iv0gc.fsf@inria.fr> <87v9c0ap22.fsf_-_@gnu.org> <87wnmsn5lz.fsf_-_@gnu.org> <87bl44vfvg.fsf_-_@gmail.com> <87o880byyz.fsf@inria.fr> <87czoay4sq.fsf@inria.fr> <86h7dmms8c.fsf@gmail.com> <87y26ytek6.fsf_-_@inria.fr> Date: Tue, 12 Oct 2021 20:06:22 +0200 Message-ID: <861r4qkti9.fsf@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Received-SPF: pass client-ip=2a00:1450:4864:20::434; envelope-from=zimon.toutoune@gmail.com; helo=mail-wr1-x434.google.com X-Spam_score_int: -15 X-Spam_score: -1.6 X-Spam_bar: - X-Spam_report: (-1.6 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, PDS_BTC_ID=0.499, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: guix-devel@gnu.org Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: "Guix-devel" X-Migadu-Flow: FLOW_IN ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1634062574; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post:dkim-signature; bh=obI0I5NIiPASJDmVK35tf7yMqhGmkHJeL5dvKB3F8MU=; b=D07E9WTJBMlKTaz6OPpH0WrNolzvRYkGYFt0OWKRtjRiMKfou/os75+TlE0jmX9zYIcZCW 1kA70aoBUx5u3ALDlrlJQgONOze8f9MmV+gtQCwSOWpgIe3XNOJbuBWOSemqY0SRpgHMd2 91C+Dor1If+INmZY8j9TvrxzBFrwBAAf1/ypaahI5kSzgpKKNrqN9Gcsm/ws+NkmS+9+xH CNt7bXXQvTuobC/z+V6orPnE6jtbOSbD7j2mvrPjMUcrZd0gJpY9v+Gt+M4YqstOUdb9M9 vbjFgLwy6HvF8hpVZkcf3ffqUz9LvBPLPmecCz3/ALe9AEe3c9dyHZgiZjZJwg== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1634062574; a=rsa-sha256; cv=none; b=IfpVjK0odiORfSfuFpJkHPzrsCbZfajrdEtTLgDj9JqjBV2V/JDfRZwhOMDfLdqX2TBNgs ZzWSj4udN0/kt39VrxjOElgJnA08GDkMAKDX1Y2dsI5TTRWZxSKuQZAjj5JH+cxgJ085SD BvcuumCHROx3+KiArwBU0XkNi4/sEmj7sNiyES1o2zaa0CxfY6/HEWjW+a9SG3iUo9nw5m RjFL/HG1dAEnh3EVe2u7YIl+yZB1a9Xxh8KY7o7cbA7vt38w4oui+sXxtuxxQR5R+CKYj1 Ll8vjZiIFv1FabOnpFfhnwiPpDti5DC163CGAkIdwYdumF0X/EKLyqiD0sYFXQ== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=pdr79jYq; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (aspmx1.migadu.com: domain of guix-devel-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-devel-bounces@gnu.org X-Migadu-Spam-Score: -3.12 Authentication-Results: aspmx1.migadu.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=pdr79jYq; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (aspmx1.migadu.com: domain of guix-devel-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-devel-bounces@gnu.org X-Migadu-Queue-Id: B669F39FC0 X-Spam-Score: -3.12 X-Migadu-Scanner: scn0.migadu.com X-TUID: R3CSoTriBJtg Hey, On Tue, 12 Oct 2021 at 18:04, Ludovic Court=C3=A8s wrote: > (Moving to guix-devel from .) I was preparing a report. You have been faster than me. :-) Two questions are raising, IIUC: 1. the =E2=80=9Cmodular=E2=80=9D derivations for all the commits 2. long-term support for substitutes >> Other said 3388/6996=3D ~50% of commits are pushed at the same time, i.e= ., >> missed by both build farms using 2 different strategies to collect the >> thing to build (fetch every 5 minutes or fetch from guix-commits). It >> is a quick back to envelope so keep that with some salt. :-) > > OK. To make it explicit of #1, I was talking about the =E2=80=9Cmodular=E2=80= =9D Guix, i.e., when running =E2=80=9Cguix pull=E2=80=9D or =E2=80=9Cguix time-machine=E2= =80=9D it leads to build the derivations module-import.drv, guix-.drv, guix-command.drv, guix-module-union.drv, guix--modules.drv, guix-packages-modules.drv, guix-system-tests-modules.drv, guix-packages-base-modules.drv, etc. On slow machines, it can be unpleasant; not to say unpractical. Even for recent commits. The recent addition of =E2=80=99channel-with-substitutes-available=E2=80=99= helps when going forward (guix pull) if the build farm does not have yet these. The issue is going backward (guix time-machine). Basically, commit 59d10c3112 is from March 14, 2020 and it takes ~29min on my slow laptop. And to compare apple to apple, let take another commit one year later from March 14, 2021, e.g., commit 7327295462. It takes ~5min on the same machine. --8<---------------cut here---------------start------------->8--- $ time guix time-machine --commit=3D59d10c3112 -- --version Updating channel 'guix' from Git repository at 'https://git.savannah.gnu.or= g/git/guix.git'... substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0% The following derivations will be built: /gnu/store/zvy89f9xb53fbqvfrm7lql8mbfrsfk1b-compute-guix-derivation.drv /gnu/store/7y80kn1bypnbm869hvcq8841mr6nqvfm-module-import-compiled.drv /gnu/store/amwvgaf45722k6jn4r39983zsgmbyp2g-module-import.drv /gnu/store/h3h0qfiw5100zkwfb919r7vn0q06ksqy-config.scm.drv /gnu/store/jkwhdilsbxb18hx6gi4i2rj0v06mfbab-module-import.drv /gnu/store/sixfy4sazai667n99pxa5h7wzzaabw79-module-import-compiled.drv [...] Computing Guix derivation for 'x86_64-linux'... WARNING: (guix build emacs= -build-system): imported module (guix build utils) overrides core binding `= delete' substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0% [...] substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0% The following derivations will be built: /gnu/store/50ymxym19h8whzg3ajcl6kdjmq3p7qrg-profile.drv /gnu/store/l0znp7g83lbylbv97nd3ahz8rnrvxfrf-guix-59d10c311.drv /gnu/store/bvxrnp8bydl3zsbcg7j8j7m0qfygdhfs-guix-command.drv /gnu/store/xmsciylxx1j7nbry5cv1lm7595a8rilr-guix-module-union.drv /gnu/store/yvkw65kv9bhvx41750dchxw98qqxv64b-guix-59d10c311-modules.drv /gnu/store/6hrn3bpvcg8571mckzcj519xf2kqn2sl-guix-packages-base-modules.d= rv /gnu/store/nwl8z8cx9pdwn0sx5i5j5mp0bkdi64mm-guix-packages-base.drv /gnu/store/0j0271nbm2l526m5xs7zpd686qqrjz7w-guix-core.drv /gnu/store/2134jzhhckh871vlscw6dmwqqhny9zxg-guix-core-source.drv /gnu/store/mhik7ggrf4z5f38nsg7g0gbijm916b98-config.scm.drv /gnu/store/53zlyv96nrrm4h5ns7nmmndj8jys38f7-guix-extra.drv /gnu/store/7dn3f27i48jp6zvlwanzk8mfy828k6cm-guix-config-modules.drv /gnu/store/6xy3yyvjqvjyrlrwk1lzs12knbvilqpy-guix-config-source.drv /gnu/store/15ihwkqzyz4r4b4rppb92qcawha6a7p7-config.scm.drv /gnu/store/cacawv4yib8pa2ajzw0kyaihgym72mww-guix-config.drv /gnu/store/baivfv20hzm799v0wvdrcfaimh4aw22a-guix-extra-modules.drv /gnu/store/cxpkd0jkxapzbmg5vfmn4fy30yd7vlhm-guix-core-modules.drv /gnu/store/g3nqbybggh7dc2qd9gkj7swfmgmiigpp-guix-packages-modules.drv /gnu/store/nq9mzr00ny3nrsldvcq9r4va4fhb26sq-guix-packages.drv /gnu/store/i9dfjh5mf3r83447x7fa75hv1hnp9myv-guix-cli-modules.drv /gnu/store/xdsld8rhrawgngv93qx6lk9cgmql908c-guix-cli.drv /gnu/store/qnzm0a1gcwnfvkii7vk93rimzzp3mcf9-guix-system.drv /gnu/store/x2g2s3rhw0bv9qdp21yghgl2sij15dkr-guix-system-modules.drv /gnu/store/zandkjnylznvdj8jfsfgssvmvs6jfyph-guix-system-tests-modules.drv /gnu/store/45bs7y1h3gx2m7qry6r621klgkmv47wl-guix-system-tests.drv /gnu/store/caj7qjxvhrksk3jrkpsxqnx4kg7mlj9d-guix-daemon.drv /gnu/store/mah24wyy6bd51c27ww45hsqnjxhcn0yx-guix-manual.drv /gnu/store/002i672yl1192x7wvhkdbih94qffmcdk-guix-translated-texinfo.drv /gnu/store/5sf9aalvic81qlm06lq3a1pwdb2b3bm0-inferior-script.scm.drv /gnu/store/k612gnziiy3hn2dnrj33w4mw84kcnynm-profile.drv 11.7 MB will be downloaded [...] real 29m14.588s user 0m56.126s sys 0m1.032s --8<---------------cut here---------------end--------------->8--- --8<---------------cut here---------------start------------->8--- $ time guix time-machine --commit=3D7327295462 -- --version Updating channel 'guix' from Git repository at 'https://git.savannah.gnu.or= g/git/guix.git'... substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0% [...] building /gnu/store/6xq8vxpl51l8b3fz6sxpyspa1w5chbk9-module-import.drv... module-import 2KiB 43KiB/s 00:0= 0 [##################] 100.0% module-import-compiled 1.5MiB 607KiB/s 00:0= 3 [##################] 100.0% module-import-compiled 1.5MiB 606KiB/s 00:0= 3 [##################] 100.0% building /gnu/store/pbv19dhrlqr2lnzphmydn4zrrdccghf2-compute-guix-derivatio= n.drv... substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0% substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0% @ substituter-started /gnu/store/fbn395nfpbp4d4fr6jsbmwcx6n10kg16-python-mi= nimal-3.8.2 substitute @ download-started /gnu/store/fbn395nfpbp4d4fr6jsbmwcx6n10kg16-python-minim= al-3.8.2 https://ci.guix.gnu.org/nar/lzip/fbn395nfpbp4d4fr6jsbmwcx6n10kg16-= python-minimal-3 [...] @ substituter-succeeded /gnu/store/fx0cdzzppd8jc09sianbq6gl1h7mxx3x-zziplib= -0.13.72 substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0% The following derivations will be built: /gnu/store/8kmb40b2r3cx5zxpcrwa73a4lkaxjd9l-profile.drv /gnu/store/la6c7m2b6izy22vv8xpyvpz1ajyq72br-profile.drv /gnu/store/nljv2wnw0wqkyk0am8722gdwah3b0cx2-guix-732729546.drv /gnu/store/bmrx03y52d8dhhcpyf9i8j4zn2fg7pip-guix-command.drv /gnu/store/pg26n125c9bmvk4lxxp9ssd9havk89wc-guix-module-union.drv /gnu/store/mp0c2ad09axrq8zwhh3ycfk4a2mrgvm2-guix-732729546-modules.drv /gnu/store/0gx5jr1vgkdm5ajfcscladhzjx2gz5l2-guix-system-modules.drv /gnu/store/1sqwx35rn2qinlzib74zfjanxzzgmza3-guix-packages-base-modules.d= rv /gnu/store/6d64jyviixsjvfjgpxv3lyzq7l35y0f3-guix-config-modules.drv /gnu/store/5v24gh1pbqy6jzyl9x52wzxa6qprwl6v-guix-config-source.drv /gnu/store/pv6rskbpg8rzq3wi92m3iw7a2524r994-config.scm.drv /gnu/store/i6ncmiw6agrlq290drkg94scn12sv4v8-guix-config.drv /gnu/store/6l3l0qsq280pwmrnb0z2dfiyc7g5ff5i-guix-extra-modules.drv /gnu/store/hdy7q3xm8jalssc6jim2fk1wqgxqbfm9-guix-system-tests-modules.drv /gnu/store/ki86v53288fnx8sih3zlfi9qnjx2lzay-guix-packages-modules.drv /gnu/store/l63wxbqaq98nfjkq2cyshb6q8rjxjd6h-guix-core-modules.drv /gnu/store/b1qky9s98cfgp88xan9pqg0k9k0rlzrm-guix-core-source.drv /gnu/store/wmfssg7yyz2hrwanash7yk8f86faghlf-guix-cli-modules.drv /gnu/store/z1xxvp4hmffrpmbl0ll5y87w5pyfma9l-guix-daemon.drv /gnu/store/ms6pkrkggd0rl4fh5hfh20gcva7ryip5-inferior-script.scm.drv 28.7 MB will be downloaded [...] real 5m56.451s user 3m52.055s sys 0m1.738s --8<---------------cut here---------------end--------------->8--- To be on the same wavelength, --8<---------------cut here---------------start------------->8--- $ git log --format=3D"%h %cd" --after=3D2021-03-14 --reverse | head -n16 [...] 2babf7d831 Sun Mar 14 19:16:55 2021 +0100 b15720182e Sun Mar 14 13:24:21 2021 -0500 207aa62e6b Sun Mar 14 13:24:21 2021 -0500 30f5381487 Sun Mar 14 13:24:21 2021 -0500 af25357b7d Sun Mar 14 13:24:21 2021 -0500 7164d2105a Sun Mar 14 13:24:21 2021 -0500 078f3288e2 Sun Mar 14 13:24:21 2021 -0500 5a31eb7d35 Sun Mar 14 13:24:21 2021 -0500 620206b680 Sun Mar 14 13:24:22 2021 -0500 b76762a9b7 Sun Mar 14 13:24:22 2021 -0500 cbfcbb79df Sun Mar 14 19:43:35 2021 +0100 --8<---------------cut here---------------end--------------->8--- and Cuirass builds only one of b15720182e, 207aa62e6b, 30f5381487, af25357b7d, 7164d2105a, 078f3288e2, 5a31eb7d35, 620206b680 or b76762a9b7. Considering the Build Coordinator, it uses guix-commits and from my understanding it reads: therefore, b15720182e would be missed but not b76762a9b7=E2=80=93which woul= d be missed by Cuirass. Cuirass and the Build Coordinator cannot each build the both commits b15720182e and b76762a9b7. Cuirass check every 5 minutes and Build Coordinator reads =E2=80=9Cstate=E2= =80=9D from guix-commits. Other said, none of them builds all these =E2=80=9Cmodular= =E2=80=9D derivations for all the commits; even for recent commits. The rough estimate is half of commits are missed by both build farms. Therefore, using =E2=80=9Cguix time-machine=E2=80=9D with a random commit a= nd one gets 1/2 probability to build something just to get the inferior =E2=80=93 aside= the TTL policy. (It is mitigated because the both build farms use different strategies and thus they do not miss the same commits. \o/) >> On that number, after 180 days (6 months), it is hard to evaluate the >> rate of the time-machine queries. And from my experience (no number to >> back), running time-machine on a commit older than this 180 days implies >> to build derivations. Or it is a lucky day. :-) > > Right. > > So what can we do to address this issue? I *think* we could use a > higher TTL on berlin, and we can try that right away (9 months to being > with?). I *think* the issue is not TTL for question #1. :-) But the issue that the both build farms do not build these =E2=80=9Cmodular=E2=80=9D derivatio= ns for all the commits. Here, I am focused on x86_64-linux which is the case of interest for such topic (scientific context), IMHO. Considering to build for every commit for all architectures is not affordable. I agree that increasing the TTL will help for question #2 about long-support of substitutes. > However, there is an upper bound anyway. To make informed decisions on > the retention policy, we should monitor storage space on berlin/bayfront > to better estimate what can be done. We have Zabbix but it=E2=80=99s not > accessible from the outside; maybe we could graph storage space > somewhere so people can grab the data and work on those estimates? Based on the size of these derivations for one commit, we could extrapolate back to envelope. Well, question #1 seems doable storage-speaking. The issue of #1 is to build these derivations for all the commits. IMHO. About #2, yeah if some data are available, I can try to make some estimates. Well, #1 seems actionable. However, #2 raises=E2=80=A6 > What if we decide that we need to provide substitutes for 2y old > commits? In that case, we need a plan to scale up. That could be > renting storage space somewhere. That=E2=80=99s largely non-technical wo= rk that > needs attention. =E2=80=A6a strong question. :-) What do =E2=80=9Cwe=E2=80=9D do for what = =E2=80=9Cwe=E2=80=9D build? Indeed, numbers are missing to make informed decisions on long-term storage of substitutes. What is Nix doing? > There are also technical tweaks that could help: distinguishing between > =E2=80=9Cimportant=E2=80=9D substitutes that we want to keep, and less im= portant > substitutes (how?); identifying =E2=80=9Cequivalence classes=E2=80=9D for= builds of a > given package; etc. The outcome is unclear and it=E2=80=99ll take time. I agree it will take time. :-) I think that having 2 build farms building in parallel is a strength. So let exploit it. :-) What one could have in mind is to challenge the outputs; if they are identical, let keep only one version =E2=80=9Csomewher= e=E2=80=9D and remove the other from the =E2=80=9Celsewhere=E2=80=9D. For instance, we (I? with help) could resume this discussion: Or maybe, for the identical outputs, one could imagine (dream? for) a cooking service for missing outputs. Well, I do not know how this is actionable. :-) Cheers, simon