* Substitute retention [not found] ` <86h7dmms8c.fsf@gmail.com> @ 2021-10-12 16:04 ` Ludovic Courtès 2021-10-12 18:06 ` zimoun 0 siblings, 1 reply; 3+ messages in thread From: Ludovic Courtès @ 2021-10-12 16:04 UTC (permalink / raw) To: zimoun; +Cc: guix-devel Hi! (Moving to guix-devel from <https://issues.guix.gnu.org/42162#43>.) zimoun <zimon.toutoune@gmail.com> skribis: >> For the record, the ‘guix publish’ config on berlin is here: >> >> https://git.savannah.gnu.org/cgit/guix/maintenance.git/tree/hydra/modules/sysadmin/services.scm#n485 >> >> If I read that correctly, nars have a TTL of 180 days (this is the time >> a nar is retained after the last time it has been requested, so it’s a >> lower bound.) [...] > Just for the record, a back to envelope computations. 180 days before > today was April 15th (M-x calendar C-u 180 C-b). It means 6996 commits > (35aaf1fe10 is my current last commit). > > git log --format="%cd" --after=2021-04-15 | wc -l > 6996 > > However, these commits are pushed by batch. Roughly, it reads: > > git log --format="%cd" --after=2021-04-15 --date=unix \ > | awk 'NR == 1{old= $1; next}{print old - $1; old = $1}' \ > | sort -n | uniq -c | grep -e "0$" | head > 1 -1542620 > 3388 0 > 14 10 > 6 20 > 5 30 > 2 40 > 4 50 > 1 60 > 2 70 > 2 80 > > (Take the ’awk’ with care, I am not sure of what I am doing. :-) And, > it is rough because timezone etc.) > > Other said 3388/6996= ~50% of commits are pushed at the same time, i.e., > missed by both build farms using 2 different strategies to collect the > thing to build (fetch every 5 minutes or fetch from guix-commits). It > is a quick back to envelope so keep that with some salt. :-) OK. > On that number, after 180 days (6 months), it is hard to evaluate the > rate of the time-machine queries. And from my experience (no number to > back), running time-machine on a commit older than this 180 days implies > to build derivations. Or it is a lucky day. :-) Right. So what can we do to address this issue? I *think* we could use a higher TTL on berlin, and we can try that right away (9 months to being with?). However, there is an upper bound anyway. To make informed decisions on the retention policy, we should monitor storage space on berlin/bayfront to better estimate what can be done. We have Zabbix but it’s not accessible from the outside; maybe we could graph storage space somewhere so people can grab the data and work on those estimates? What if we decide that we need to provide substitutes for 2y old commits? In that case, we need a plan to scale up. That could be renting storage space somewhere. That’s largely non-technical work that needs attention. There are also technical tweaks that could help: distinguishing between “important” substitutes that we want to keep, and less important substitutes (how?); identifying “equivalence classes” for builds of a given package; etc. The outcome is unclear and it’ll take time. Thoughts? Ludo’. ^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: Substitute retention 2021-10-12 16:04 ` Substitute retention Ludovic Courtès @ 2021-10-12 18:06 ` zimoun 2021-10-15 9:27 ` Ludovic Courtès 0 siblings, 1 reply; 3+ messages in thread From: zimoun @ 2021-10-12 18:06 UTC (permalink / raw) To: Ludovic Courtès; +Cc: guix-devel Hey, On Tue, 12 Oct 2021 at 18:04, Ludovic Courtès <ludovic.courtes@inria.fr> wrote: > (Moving to guix-devel from <https://issues.guix.gnu.org/42162#43>.) I was preparing a report. You have been faster than me. :-) Two questions are raising, IIUC: 1. the “modular” derivations for all the commits 2. long-term support for substitutes >> Other said 3388/6996= ~50% of commits are pushed at the same time, i.e., >> missed by both build farms using 2 different strategies to collect the >> thing to build (fetch every 5 minutes or fetch from guix-commits). It >> is a quick back to envelope so keep that with some salt. :-) > > OK. To make it explicit of #1, I was talking about the “modular” Guix, i.e., when running “guix pull” or “guix time-machine” it leads to build the derivations module-import.drv, guix-<hash>.drv, guix-command.drv, guix-module-union.drv, guix-<hash>-modules.drv, guix-packages-modules.drv, guix-system-tests-modules.drv, guix-packages-base-modules.drv, etc. On slow machines, it can be unpleasant; not to say unpractical. Even for recent commits. The recent addition of ’channel-with-substitutes-available’ helps when going forward (guix pull) if the build farm does not have yet these. The issue is going backward (guix time-machine). Basically, commit 59d10c3112 is from March 14, 2020 and it takes ~29min on my slow laptop. And to compare apple to apple, let take another commit one year later from March 14, 2021, e.g., commit 7327295462. It takes ~5min on the same machine. --8<---------------cut here---------------start------------->8--- $ time guix time-machine --commit=59d10c3112 -- --version Updating channel 'guix' from Git repository at 'https://git.savannah.gnu.org/git/guix.git'... substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0% The following derivations will be built: /gnu/store/zvy89f9xb53fbqvfrm7lql8mbfrsfk1b-compute-guix-derivation.drv /gnu/store/7y80kn1bypnbm869hvcq8841mr6nqvfm-module-import-compiled.drv /gnu/store/amwvgaf45722k6jn4r39983zsgmbyp2g-module-import.drv /gnu/store/h3h0qfiw5100zkwfb919r7vn0q06ksqy-config.scm.drv /gnu/store/jkwhdilsbxb18hx6gi4i2rj0v06mfbab-module-import.drv /gnu/store/sixfy4sazai667n99pxa5h7wzzaabw79-module-import-compiled.drv [...] Computing Guix derivation for 'x86_64-linux'... WARNING: (guix build emacs-build-system): imported module (guix build utils) overrides core binding `delete' substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0% [...] substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0% The following derivations will be built: /gnu/store/50ymxym19h8whzg3ajcl6kdjmq3p7qrg-profile.drv /gnu/store/l0znp7g83lbylbv97nd3ahz8rnrvxfrf-guix-59d10c311.drv /gnu/store/bvxrnp8bydl3zsbcg7j8j7m0qfygdhfs-guix-command.drv /gnu/store/xmsciylxx1j7nbry5cv1lm7595a8rilr-guix-module-union.drv /gnu/store/yvkw65kv9bhvx41750dchxw98qqxv64b-guix-59d10c311-modules.drv /gnu/store/6hrn3bpvcg8571mckzcj519xf2kqn2sl-guix-packages-base-modules.drv /gnu/store/nwl8z8cx9pdwn0sx5i5j5mp0bkdi64mm-guix-packages-base.drv /gnu/store/0j0271nbm2l526m5xs7zpd686qqrjz7w-guix-core.drv /gnu/store/2134jzhhckh871vlscw6dmwqqhny9zxg-guix-core-source.drv /gnu/store/mhik7ggrf4z5f38nsg7g0gbijm916b98-config.scm.drv /gnu/store/53zlyv96nrrm4h5ns7nmmndj8jys38f7-guix-extra.drv /gnu/store/7dn3f27i48jp6zvlwanzk8mfy828k6cm-guix-config-modules.drv /gnu/store/6xy3yyvjqvjyrlrwk1lzs12knbvilqpy-guix-config-source.drv /gnu/store/15ihwkqzyz4r4b4rppb92qcawha6a7p7-config.scm.drv /gnu/store/cacawv4yib8pa2ajzw0kyaihgym72mww-guix-config.drv /gnu/store/baivfv20hzm799v0wvdrcfaimh4aw22a-guix-extra-modules.drv /gnu/store/cxpkd0jkxapzbmg5vfmn4fy30yd7vlhm-guix-core-modules.drv /gnu/store/g3nqbybggh7dc2qd9gkj7swfmgmiigpp-guix-packages-modules.drv /gnu/store/nq9mzr00ny3nrsldvcq9r4va4fhb26sq-guix-packages.drv /gnu/store/i9dfjh5mf3r83447x7fa75hv1hnp9myv-guix-cli-modules.drv /gnu/store/xdsld8rhrawgngv93qx6lk9cgmql908c-guix-cli.drv /gnu/store/qnzm0a1gcwnfvkii7vk93rimzzp3mcf9-guix-system.drv /gnu/store/x2g2s3rhw0bv9qdp21yghgl2sij15dkr-guix-system-modules.drv /gnu/store/zandkjnylznvdj8jfsfgssvmvs6jfyph-guix-system-tests-modules.drv /gnu/store/45bs7y1h3gx2m7qry6r621klgkmv47wl-guix-system-tests.drv /gnu/store/caj7qjxvhrksk3jrkpsxqnx4kg7mlj9d-guix-daemon.drv /gnu/store/mah24wyy6bd51c27ww45hsqnjxhcn0yx-guix-manual.drv /gnu/store/002i672yl1192x7wvhkdbih94qffmcdk-guix-translated-texinfo.drv /gnu/store/5sf9aalvic81qlm06lq3a1pwdb2b3bm0-inferior-script.scm.drv /gnu/store/k612gnziiy3hn2dnrj33w4mw84kcnynm-profile.drv 11.7 MB will be downloaded [...] real 29m14.588s user 0m56.126s sys 0m1.032s --8<---------------cut here---------------end--------------->8--- --8<---------------cut here---------------start------------->8--- $ time guix time-machine --commit=7327295462 -- --version Updating channel 'guix' from Git repository at 'https://git.savannah.gnu.org/git/guix.git'... substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0% [...] building /gnu/store/6xq8vxpl51l8b3fz6sxpyspa1w5chbk9-module-import.drv... module-import 2KiB 43KiB/s 00:00 [##################] 100.0% module-import-compiled 1.5MiB 607KiB/s 00:03 [##################] 100.0% module-import-compiled 1.5MiB 606KiB/s 00:03 [##################] 100.0% building /gnu/store/pbv19dhrlqr2lnzphmydn4zrrdccghf2-compute-guix-derivation.drv... substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0% substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0% @ substituter-started /gnu/store/fbn395nfpbp4d4fr6jsbmwcx6n10kg16-python-minimal-3.8.2 substitute @ download-started /gnu/store/fbn395nfpbp4d4fr6jsbmwcx6n10kg16-python-minimal-3.8.2 https://ci.guix.gnu.org/nar/lzip/fbn395nfpbp4d4fr6jsbmwcx6n10kg16-python-minimal-3 [...] @ substituter-succeeded /gnu/store/fx0cdzzppd8jc09sianbq6gl1h7mxx3x-zziplib-0.13.72 substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0% The following derivations will be built: /gnu/store/8kmb40b2r3cx5zxpcrwa73a4lkaxjd9l-profile.drv /gnu/store/la6c7m2b6izy22vv8xpyvpz1ajyq72br-profile.drv /gnu/store/nljv2wnw0wqkyk0am8722gdwah3b0cx2-guix-732729546.drv /gnu/store/bmrx03y52d8dhhcpyf9i8j4zn2fg7pip-guix-command.drv /gnu/store/pg26n125c9bmvk4lxxp9ssd9havk89wc-guix-module-union.drv /gnu/store/mp0c2ad09axrq8zwhh3ycfk4a2mrgvm2-guix-732729546-modules.drv /gnu/store/0gx5jr1vgkdm5ajfcscladhzjx2gz5l2-guix-system-modules.drv /gnu/store/1sqwx35rn2qinlzib74zfjanxzzgmza3-guix-packages-base-modules.drv /gnu/store/6d64jyviixsjvfjgpxv3lyzq7l35y0f3-guix-config-modules.drv /gnu/store/5v24gh1pbqy6jzyl9x52wzxa6qprwl6v-guix-config-source.drv /gnu/store/pv6rskbpg8rzq3wi92m3iw7a2524r994-config.scm.drv /gnu/store/i6ncmiw6agrlq290drkg94scn12sv4v8-guix-config.drv /gnu/store/6l3l0qsq280pwmrnb0z2dfiyc7g5ff5i-guix-extra-modules.drv /gnu/store/hdy7q3xm8jalssc6jim2fk1wqgxqbfm9-guix-system-tests-modules.drv /gnu/store/ki86v53288fnx8sih3zlfi9qnjx2lzay-guix-packages-modules.drv /gnu/store/l63wxbqaq98nfjkq2cyshb6q8rjxjd6h-guix-core-modules.drv /gnu/store/b1qky9s98cfgp88xan9pqg0k9k0rlzrm-guix-core-source.drv /gnu/store/wmfssg7yyz2hrwanash7yk8f86faghlf-guix-cli-modules.drv /gnu/store/z1xxvp4hmffrpmbl0ll5y87w5pyfma9l-guix-daemon.drv /gnu/store/ms6pkrkggd0rl4fh5hfh20gcva7ryip5-inferior-script.scm.drv 28.7 MB will be downloaded [...] real 5m56.451s user 3m52.055s sys 0m1.738s --8<---------------cut here---------------end--------------->8--- To be on the same wavelength, --8<---------------cut here---------------start------------->8--- $ git log --format="%h %cd" --after=2021-03-14 --reverse | head -n16 [...] 2babf7d831 Sun Mar 14 19:16:55 2021 +0100 b15720182e Sun Mar 14 13:24:21 2021 -0500 207aa62e6b Sun Mar 14 13:24:21 2021 -0500 30f5381487 Sun Mar 14 13:24:21 2021 -0500 af25357b7d Sun Mar 14 13:24:21 2021 -0500 7164d2105a Sun Mar 14 13:24:21 2021 -0500 078f3288e2 Sun Mar 14 13:24:21 2021 -0500 5a31eb7d35 Sun Mar 14 13:24:21 2021 -0500 620206b680 Sun Mar 14 13:24:22 2021 -0500 b76762a9b7 Sun Mar 14 13:24:22 2021 -0500 cbfcbb79df Sun Mar 14 19:43:35 2021 +0100 --8<---------------cut here---------------end--------------->8--- and Cuirass builds only one of b15720182e, 207aa62e6b, 30f5381487, af25357b7d, 7164d2105a, 078f3288e2, 5a31eb7d35, 620206b680 or b76762a9b7. Considering the Build Coordinator, it uses guix-commits and from my understanding it reads: <https://lists.gnu.org/archive/html/guix-commits/2021-03/msg01201.html> therefore, b15720182e would be missed but not b76762a9b7–which would be missed by Cuirass. Cuirass and the Build Coordinator cannot each build the both commits b15720182e and b76762a9b7. Cuirass check every 5 minutes and Build Coordinator reads “state” from guix-commits. Other said, none of them builds all these “modular” derivations for all the commits; even for recent commits. The rough estimate is half of commits are missed by both build farms. Therefore, using “guix time-machine” with a random commit and one gets 1/2 probability to build something just to get the inferior – aside the TTL policy. (It is mitigated because the both build farms use different strategies and thus they do not miss the same commits. \o/) >> On that number, after 180 days (6 months), it is hard to evaluate the >> rate of the time-machine queries. And from my experience (no number to >> back), running time-machine on a commit older than this 180 days implies >> to build derivations. Or it is a lucky day. :-) > > Right. > > So what can we do to address this issue? I *think* we could use a > higher TTL on berlin, and we can try that right away (9 months to being > with?). I *think* the issue is not TTL for question #1. :-) But the issue that the both build farms do not build these “modular” derivations for all the commits. Here, I am focused on x86_64-linux which is the case of interest for such topic (scientific context), IMHO. Considering to build for every commit for all architectures is not affordable. I agree that increasing the TTL will help for question #2 about long-support of substitutes. > However, there is an upper bound anyway. To make informed decisions on > the retention policy, we should monitor storage space on berlin/bayfront > to better estimate what can be done. We have Zabbix but it’s not > accessible from the outside; maybe we could graph storage space > somewhere so people can grab the data and work on those estimates? Based on the size of these derivations for one commit, we could extrapolate back to envelope. Well, question #1 seems doable storage-speaking. The issue of #1 is to build these derivations for all the commits. IMHO. About #2, yeah if some data are available, I can try to make some estimates. Well, #1 seems actionable. However, #2 raises… > What if we decide that we need to provide substitutes for 2y old > commits? In that case, we need a plan to scale up. That could be > renting storage space somewhere. That’s largely non-technical work that > needs attention. …a strong question. :-) What do “we” do for what “we” build? Indeed, numbers are missing to make informed decisions on long-term storage of substitutes. What is Nix doing? > There are also technical tweaks that could help: distinguishing between > “important” substitutes that we want to keep, and less important > substitutes (how?); identifying “equivalence classes” for builds of a > given package; etc. The outcome is unclear and it’ll take time. I agree it will take time. :-) I think that having 2 build farms building in parallel is a strength. So let exploit it. :-) What one could have in mind is to challenge the outputs; if they are identical, let keep only one version “somewhere” and remove the other from the “elsewhere”. For instance, we (I? with help) could resume this discussion: <https://lists.gnu.org/archive/html/guix-devel/2020-10/msg00181.html> Or maybe, for the identical outputs, one could imagine (dream? for) a cooking service for missing outputs. Well, I do not know how this is actionable. :-) Cheers, simon ^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: Substitute retention 2021-10-12 18:06 ` zimoun @ 2021-10-15 9:27 ` Ludovic Courtès 0 siblings, 0 replies; 3+ messages in thread From: Ludovic Courtès @ 2021-10-15 9:27 UTC (permalink / raw) To: zimoun; +Cc: guix-devel Hi! zimoun <zimon.toutoune@gmail.com> skribis: >>> missed by both build farms using 2 different strategies to collect the >>> thing to build (fetch every 5 minutes or fetch from guix-commits). It >>> is a quick back to envelope so keep that with some salt. :-) >> >> OK. > > To make it explicit of #1, I was talking about the “modular” Guix, i.e., > when running “guix pull” or “guix time-machine” it leads to build the > derivations module-import.drv, guix-<hash>.drv, guix-command.drv, > guix-module-union.drv, guix-<hash>-modules.drv, > guix-packages-modules.drv, guix-system-tests-modules.drv, > guix-packages-base-modules.drv, etc. On slow machines, it can be > unpleasant; not to say unpractical. Even for recent commits. Ah I see. Yeah, this can be kinda annoying, and amplified by the fact that CI only builds at each push, not at each commit. That said, this is mitigated by the fact that one typically travels to a previously-fetched commit, which is a commit that has been built by CI rather than a commit in between two pushes. > Basically, commit 59d10c3112 is from March 14, 2020 and it takes ~29min > on my slow laptop. And to compare apple to apple, let take another > commit one year later from March 14, 2021, e.g., commit 7327295462. It > takes ~5min on the same machine. Yeah, OK. > To be on the same wavelength, > > $ git log --format="%h %cd" --after=2021-03-14 --reverse | head -n16 > [...] > 2babf7d831 Sun Mar 14 19:16:55 2021 +0100 > b15720182e Sun Mar 14 13:24:21 2021 -0500 > 207aa62e6b Sun Mar 14 13:24:21 2021 -0500 > 30f5381487 Sun Mar 14 13:24:21 2021 -0500 > af25357b7d Sun Mar 14 13:24:21 2021 -0500 > 7164d2105a Sun Mar 14 13:24:21 2021 -0500 > 078f3288e2 Sun Mar 14 13:24:21 2021 -0500 > 5a31eb7d35 Sun Mar 14 13:24:21 2021 -0500 > 620206b680 Sun Mar 14 13:24:22 2021 -0500 > b76762a9b7 Sun Mar 14 13:24:22 2021 -0500 > cbfcbb79df Sun Mar 14 19:43:35 2021 +0100 > > and Cuirass builds only one of b15720182e, 207aa62e6b, 30f5381487, > af25357b7d, 7164d2105a, 078f3288e2, 5a31eb7d35, 620206b680 or > b76762a9b7. > > Considering the Build Coordinator, it uses guix-commits and from my > understanding it reads: > > <https://lists.gnu.org/archive/html/guix-commits/2021-03/msg01201.html> > > therefore, b15720182e would be missed but not b76762a9b7–which would be > missed by Cuirass. > > Cuirass and the Build Coordinator cannot each build the both commits > b15720182e and b76762a9b7. > > Cuirass check every 5 minutes and Build Coordinator reads “state” from > guix-commits. Other said, none of them builds all these “modular” > derivations for all the commits; even for recent commits. > > The rough estimate is half of commits are missed by both build farms. > Therefore, using “guix time-machine” with a random commit and one gets > 1/2 probability to build something just to get the inferior – aside the > TTL policy. Right. Not every derivation produced by (guix self) needs to be rebuilt in between two commits, but anything that depends on *package-modules* typically has to be rebuilt. We can reduce the amount of rebuilt like I did in commit abd38dcee16f0ac71191527c38dcd3659111e2ba, but you’ll always have the big (gnu packages …) derivation. >> So what can we do to address this issue? I *think* we could use a >> higher TTL on berlin, and we can try that right away (9 months to being >> with?). > > I *think* the issue is not TTL for question #1. :-) But the issue that > the both build farms do not build these “modular” derivations for all > the commits. Here, I am focused on x86_64-linux which is the case of > interest for such topic (scientific context), IMHO. > > Considering to build for every commit for all architectures is not > affordable. > > I agree that increasing the TTL will help for question #2 about > long-support of substitutes. Understood! >> However, there is an upper bound anyway. To make informed decisions on >> the retention policy, we should monitor storage space on berlin/bayfront >> to better estimate what can be done. We have Zabbix but it’s not >> accessible from the outside; maybe we could graph storage space >> somewhere so people can grab the data and work on those estimates? > > Based on the size of these derivations for one commit, we could > extrapolate back to envelope. Well, question #1 seems doable > storage-speaking. > > The issue of #1 is to build these derivations for all the commits. > IMHO. > > About #2, yeah if some data are available, I can try to make some > estimates. > > > Well, #1 seems actionable. However, #2 raises… > >> What if we decide that we need to provide substitutes for 2y old >> commits? In that case, we need a plan to scale up. That could be >> renting storage space somewhere. That’s largely non-technical work that >> needs attention. > > …a strong question. :-) What do “we” do for what “we” build? > > Indeed, numbers are missing to make informed decisions on long-term > storage of substitutes. What is Nix doing? Nix, AFAIK, is doing like everyone else: pouring money on Amazon. Last I heard they’d retain substitutes basically indefinitely on Amazon S3 (incidentally, one motivation for them to work with Software Heritage, AIUI, is that it would allow them to store less data on the storage they pay for :-)). For the record, berlin (aka ci.guix.gnu.org; it was donated by the Max Delbrück Center, MDC, and is generously hosted by them) has a 37 TiB disk for /gnu/store and “baked” substitutes. That’s a lot. Technically though, a lot of it is used by less important substitutes such as disk images or intermediate ‘core-updates’ substitutes. In the end we seem to be filling it more quickly than you’d think! Perhaps we need a better strategy with a low TTL for, say, intermediate ‘core-updates’ substitutes (no need to keep them more than a few weeks if we know we’re doing a world rebuild right after). It cannot be done as things are though because ‘guix publish’ doesn’t distinguish between store items. Or we could restart the Amazon front-end that Chris Marusich had set up right before 1.0 was released. Or we could build our own front-end for substitute delivery as a proxy to berlin, thereby distributing the burden. Thoughts? > I think that having 2 build farms building in parallel is a strength. > So let exploit it. :-) What one could have in mind is to challenge the > outputs; if they are identical, let keep only one version “somewhere” > and remove the other from the “elsewhere”. > > For instance, we (I? with help) could resume this discussion: > > <https://lists.gnu.org/archive/html/guix-devel/2020-10/msg00181.html> I hadn’t seen this message, interesting! Note however that bordeaux.guix has a tenth of the storage space of berlin (3.6 TiB), so right now we probably can’t count on it for long-term substitute storage. > Or maybe, for the identical outputs, one could imagine (dream? for) a > cooking service for missing outputs. Well, I do not know how this is > actionable. :-) Well, if we keep .drv around, we could arrange so that ‘guix publish’ rebuilds on-demand, after all. I’m not sure how practical that would be, though. Ludo’. ^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2021-10-15 9:33 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <87a6tdce94.fsf@inria.fr>
[not found] ` <87mu4iv0gc.fsf@inria.fr>
[not found] ` <handler.42162.D42162.16105343699609.notifdone@debbugs.gnu.org>
[not found] ` <87v9c0ap22.fsf_-_@gnu.org>
[not found] ` <87wnmsn5lz.fsf_-_@gnu.org>
[not found] ` <87bl44vfvg.fsf_-_@gmail.com>
[not found] ` <87o880byyz.fsf@inria.fr>
[not found] ` <CAJ3okZ2WCpzAUgBGZ1JaJmKkEmjjpFfy8hkBD854CD9vLiDHSw@mail.gmail.com>
[not found] ` <87czoay4sq.fsf@inria.fr>
[not found] ` <86h7dmms8c.fsf@gmail.com>
2021-10-12 16:04 ` Substitute retention Ludovic Courtès
2021-10-12 18:06 ` zimoun
2021-10-15 9:27 ` Ludovic Courtès
Code repositories for project(s) associated with this public inbox https://git.savannah.gnu.org/cgit/guix.git This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).