Substitute retention

Substitute retention

[Ludovic Courtès]

Hi!

(Moving to guix-devel from <https://issues.guix.gnu.org/42162#43>.)

zimoun <zimon.toutoune@gmail.com> skribis:

>> For the record, the ‘guix publish’ config on berlin is here:
>>
>>   https://git.savannah.gnu.org/cgit/guix/maintenance.git/tree/hydra/modules/sysadmin/services.scm#n485
>>
>> If I read that correctly, nars have a TTL of 180 days (this is the time
>> a nar is retained after the last time it has been requested, so it’s a
>> lower bound.)

[...]

> Just for the record, a back to envelope computations.  180 days before
> today was April 15th (M-x calendar C-u 180 C-b).  It means 6996 commits
> (35aaf1fe10 is my current last commit).
>
>     git log --format="%cd" --after=2021-04-15 | wc -l
>     6996
>
> However, these commits are pushed by batch.  Roughly, it reads:
>
>     git log --format="%cd" --after=2021-04-15 --date=unix \
>         | awk 'NR == 1{old= $1; next}{print old - $1; old = $1}' \
>         | sort -n | uniq -c | grep -e "0$" | head
>           1 -1542620
>        3388 0
>          14 10
>           6 20
>           5 30
>           2 40
>           4 50
>           1 60
>           2 70
>           2 80
>
> (Take the ’awk’ with care, I am not sure of what I am doing. :-)  And,
> it is rough because timezone etc.)
>
> Other said 3388/6996= ~50% of commits are pushed at the same time, i.e.,
> missed by both build farms using 2 different strategies to collect the
> thing to build (fetch every 5 minutes or fetch from guix-commits).  It
> is a quick back to envelope so keep that with some salt. :-)

OK.

> On that number, after 180 days (6 months), it is hard to evaluate the
> rate of the time-machine queries.  And from my experience (no number to
> back), running time-machine on a commit older than this 180 days implies
> to build derivations.  Or it is a lucky day. :-)

Right.

So what can we do to address this issue?  I *think* we could use a
higher TTL on berlin, and we can try that right away (9 months to being
with?).

However, there is an upper bound anyway.  To make informed decisions on
the retention policy, we should monitor storage space on berlin/bayfront
to better estimate what can be done.  We have Zabbix but it’s not
accessible from the outside; maybe we could graph storage space
somewhere so people can grab the data and work on those estimates?

What if we decide that we need to provide substitutes for 2y old
commits?  In that case, we need a plan to scale up.  That could be
renting storage space somewhere.  That’s largely non-technical work that
needs attention.

There are also technical tweaks that could help: distinguishing between
“important” substitutes that we want to keep, and less important
substitutes (how?); identifying “equivalence classes” for builds of a
given package; etc.  The outcome is unclear and it’ll take time.

Thoughts?

Ludo’.

Re: Substitute retention

[zimoun]

Hey,

On Tue, 12 Oct 2021 at 18:04, Ludovic Courtès <ludovic.courtes@inria.fr> wrote:

> (Moving to guix-devel from <https://issues.guix.gnu.org/42162#43>.)

I was preparing a report.  You have been faster than me. :-)

Two questions are raising, IIUC:

 1. the “modular” derivations for all the commits
 2. long-term support for substitutes


>> Other said 3388/6996= ~50% of commits are pushed at the same time, i.e.,
>> missed by both build farms using 2 different strategies to collect the
>> thing to build (fetch every 5 minutes or fetch from guix-commits).  It
>> is a quick back to envelope so keep that with some salt. :-)
>
> OK.

To make it explicit of #1, I was talking about the “modular” Guix, i.e.,
when running “guix pull” or “guix time-machine” it leads to build the
derivations module-import.drv, guix-<hash>.drv, guix-command.drv,
guix-module-union.drv, guix-<hash>-modules.drv,
guix-packages-modules.drv, guix-system-tests-modules.drv,
guix-packages-base-modules.drv, etc.  On slow machines, it can be
unpleasant; not to say unpractical.  Even for recent commits.

The recent addition of ’channel-with-substitutes-available’ helps when
going forward (guix pull) if the build farm does not have yet these.

The issue is going backward (guix time-machine).

Basically, commit 59d10c3112 is from March 14, 2020 and it takes ~29min
on my slow laptop.  And to compare apple to apple, let take another
commit one year later from March 14, 2021, e.g., commit 7327295462.  It
takes ~5min on the same machine.

--8<---------------cut here---------------start------------->8---
$ time guix time-machine --commit=59d10c3112 -- --version
Updating channel 'guix' from Git repository at 'https://git.savannah.gnu.org/git/guix.git'...
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
The following derivations will be built:
   /gnu/store/zvy89f9xb53fbqvfrm7lql8mbfrsfk1b-compute-guix-derivation.drv
   /gnu/store/7y80kn1bypnbm869hvcq8841mr6nqvfm-module-import-compiled.drv
   /gnu/store/amwvgaf45722k6jn4r39983zsgmbyp2g-module-import.drv
   /gnu/store/h3h0qfiw5100zkwfb919r7vn0q06ksqy-config.scm.drv
   /gnu/store/jkwhdilsbxb18hx6gi4i2rj0v06mfbab-module-import.drv
   /gnu/store/sixfy4sazai667n99pxa5h7wzzaabw79-module-import-compiled.drv

[...]

Computing Guix derivation for 'x86_64-linux'...  WARNING: (guix build emacs-build-system): imported module (guix build utils) overrides core binding `delete'
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%

[...]

substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
The following derivations will be built:
   /gnu/store/50ymxym19h8whzg3ajcl6kdjmq3p7qrg-profile.drv
   /gnu/store/l0znp7g83lbylbv97nd3ahz8rnrvxfrf-guix-59d10c311.drv
   /gnu/store/bvxrnp8bydl3zsbcg7j8j7m0qfygdhfs-guix-command.drv
   /gnu/store/xmsciylxx1j7nbry5cv1lm7595a8rilr-guix-module-union.drv
   /gnu/store/yvkw65kv9bhvx41750dchxw98qqxv64b-guix-59d10c311-modules.drv
   /gnu/store/6hrn3bpvcg8571mckzcj519xf2kqn2sl-guix-packages-base-modules.drv
   /gnu/store/nwl8z8cx9pdwn0sx5i5j5mp0bkdi64mm-guix-packages-base.drv
   /gnu/store/0j0271nbm2l526m5xs7zpd686qqrjz7w-guix-core.drv
   /gnu/store/2134jzhhckh871vlscw6dmwqqhny9zxg-guix-core-source.drv
   /gnu/store/mhik7ggrf4z5f38nsg7g0gbijm916b98-config.scm.drv
   /gnu/store/53zlyv96nrrm4h5ns7nmmndj8jys38f7-guix-extra.drv
   /gnu/store/7dn3f27i48jp6zvlwanzk8mfy828k6cm-guix-config-modules.drv
   /gnu/store/6xy3yyvjqvjyrlrwk1lzs12knbvilqpy-guix-config-source.drv
   /gnu/store/15ihwkqzyz4r4b4rppb92qcawha6a7p7-config.scm.drv
   /gnu/store/cacawv4yib8pa2ajzw0kyaihgym72mww-guix-config.drv
   /gnu/store/baivfv20hzm799v0wvdrcfaimh4aw22a-guix-extra-modules.drv
   /gnu/store/cxpkd0jkxapzbmg5vfmn4fy30yd7vlhm-guix-core-modules.drv
   /gnu/store/g3nqbybggh7dc2qd9gkj7swfmgmiigpp-guix-packages-modules.drv
   /gnu/store/nq9mzr00ny3nrsldvcq9r4va4fhb26sq-guix-packages.drv
   /gnu/store/i9dfjh5mf3r83447x7fa75hv1hnp9myv-guix-cli-modules.drv
   /gnu/store/xdsld8rhrawgngv93qx6lk9cgmql908c-guix-cli.drv
   /gnu/store/qnzm0a1gcwnfvkii7vk93rimzzp3mcf9-guix-system.drv
   /gnu/store/x2g2s3rhw0bv9qdp21yghgl2sij15dkr-guix-system-modules.drv
   /gnu/store/zandkjnylznvdj8jfsfgssvmvs6jfyph-guix-system-tests-modules.drv
   /gnu/store/45bs7y1h3gx2m7qry6r621klgkmv47wl-guix-system-tests.drv
   /gnu/store/caj7qjxvhrksk3jrkpsxqnx4kg7mlj9d-guix-daemon.drv
   /gnu/store/mah24wyy6bd51c27ww45hsqnjxhcn0yx-guix-manual.drv
   /gnu/store/002i672yl1192x7wvhkdbih94qffmcdk-guix-translated-texinfo.drv
   /gnu/store/5sf9aalvic81qlm06lq3a1pwdb2b3bm0-inferior-script.scm.drv
   /gnu/store/k612gnziiy3hn2dnrj33w4mw84kcnynm-profile.drv

11.7 MB will be downloaded

[...]

real	29m14.588s
user	0m56.126s
sys	0m1.032s
--8<---------------cut here---------------end--------------->8---

--8<---------------cut here---------------start------------->8---
$ time guix time-machine --commit=7327295462 -- --version
Updating channel 'guix' from Git repository at 'https://git.savannah.gnu.org/git/guix.git'...
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%

[...]

building /gnu/store/6xq8vxpl51l8b3fz6sxpyspa1w5chbk9-module-import.drv...
 module-import  2KiB                                           43KiB/s 00:00 [##################] 100.0%

 module-import-compiled  1.5MiB                               607KiB/s 00:03 [##################] 100.0%

 module-import-compiled  1.5MiB                               606KiB/s 00:03 [##################] 100.0%

building /gnu/store/pbv19dhrlqr2lnzphmydn4zrrdccghf2-compute-guix-derivation.drv...
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
@ substituter-started /gnu/store/fbn395nfpbp4d4fr6jsbmwcx6n10kg16-python-minimal-3.8.2 substitute
@ download-started /gnu/store/fbn395nfpbp4d4fr6jsbmwcx6n10kg16-python-minimal-3.8.2 https://ci.guix.gnu.org/nar/lzip/fbn395nfpbp4d4fr6jsbmwcx6n10kg16-python-minimal-3
[...]
@ substituter-succeeded /gnu/store/fx0cdzzppd8jc09sianbq6gl1h7mxx3x-zziplib-0.13.72

substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
The following derivations will be built:
   /gnu/store/8kmb40b2r3cx5zxpcrwa73a4lkaxjd9l-profile.drv
   /gnu/store/la6c7m2b6izy22vv8xpyvpz1ajyq72br-profile.drv
   /gnu/store/nljv2wnw0wqkyk0am8722gdwah3b0cx2-guix-732729546.drv
   /gnu/store/bmrx03y52d8dhhcpyf9i8j4zn2fg7pip-guix-command.drv
   /gnu/store/pg26n125c9bmvk4lxxp9ssd9havk89wc-guix-module-union.drv
   /gnu/store/mp0c2ad09axrq8zwhh3ycfk4a2mrgvm2-guix-732729546-modules.drv
   /gnu/store/0gx5jr1vgkdm5ajfcscladhzjx2gz5l2-guix-system-modules.drv
   /gnu/store/1sqwx35rn2qinlzib74zfjanxzzgmza3-guix-packages-base-modules.drv
   /gnu/store/6d64jyviixsjvfjgpxv3lyzq7l35y0f3-guix-config-modules.drv
   /gnu/store/5v24gh1pbqy6jzyl9x52wzxa6qprwl6v-guix-config-source.drv
   /gnu/store/pv6rskbpg8rzq3wi92m3iw7a2524r994-config.scm.drv
   /gnu/store/i6ncmiw6agrlq290drkg94scn12sv4v8-guix-config.drv
   /gnu/store/6l3l0qsq280pwmrnb0z2dfiyc7g5ff5i-guix-extra-modules.drv
   /gnu/store/hdy7q3xm8jalssc6jim2fk1wqgxqbfm9-guix-system-tests-modules.drv
   /gnu/store/ki86v53288fnx8sih3zlfi9qnjx2lzay-guix-packages-modules.drv
   /gnu/store/l63wxbqaq98nfjkq2cyshb6q8rjxjd6h-guix-core-modules.drv
   /gnu/store/b1qky9s98cfgp88xan9pqg0k9k0rlzrm-guix-core-source.drv
   /gnu/store/wmfssg7yyz2hrwanash7yk8f86faghlf-guix-cli-modules.drv
   /gnu/store/z1xxvp4hmffrpmbl0ll5y87w5pyfma9l-guix-daemon.drv
   /gnu/store/ms6pkrkggd0rl4fh5hfh20gcva7ryip5-inferior-script.scm.drv

28.7 MB will be downloaded

[...]

real	5m56.451s
user	3m52.055s
sys	0m1.738s
--8<---------------cut here---------------end--------------->8---



To be on the same wavelength,

--8<---------------cut here---------------start------------->8---
$ git log --format="%h %cd" --after=2021-03-14 --reverse | head -n16
[...]
2babf7d831 Sun Mar 14 19:16:55 2021 +0100
b15720182e Sun Mar 14 13:24:21 2021 -0500
207aa62e6b Sun Mar 14 13:24:21 2021 -0500
30f5381487 Sun Mar 14 13:24:21 2021 -0500
af25357b7d Sun Mar 14 13:24:21 2021 -0500
7164d2105a Sun Mar 14 13:24:21 2021 -0500
078f3288e2 Sun Mar 14 13:24:21 2021 -0500
5a31eb7d35 Sun Mar 14 13:24:21 2021 -0500
620206b680 Sun Mar 14 13:24:22 2021 -0500
b76762a9b7 Sun Mar 14 13:24:22 2021 -0500
cbfcbb79df Sun Mar 14 19:43:35 2021 +0100
--8<---------------cut here---------------end--------------->8---

and Cuirass builds only one of b15720182e, 207aa62e6b, 30f5381487,
af25357b7d, 7164d2105a, 078f3288e2, 5a31eb7d35, 620206b680 or
b76762a9b7.

Considering the Build Coordinator, it uses guix-commits and from my
understanding it reads:

<https://lists.gnu.org/archive/html/guix-commits/2021-03/msg01201.html>

therefore, b15720182e would be missed but not b76762a9b7–which would be
missed by Cuirass.

Cuirass and the Build Coordinator cannot each build the both commits
b15720182e and b76762a9b7.

Cuirass check every 5 minutes and Build Coordinator reads “state” from
guix-commits.  Other said, none of them builds all these “modular”
derivations for all the commits; even for recent commits.

The rough estimate is half of commits are missed by both build farms.
Therefore, using “guix time-machine” with a random commit and one gets
1/2 probability to build something just to get the inferior – aside the
TTL policy.

(It is mitigated because the both build farms use different strategies
and thus they do not miss the same commits. \o/)

>> On that number, after 180 days (6 months), it is hard to evaluate the
>> rate of the time-machine queries.  And from my experience (no number to
>> back), running time-machine on a commit older than this 180 days implies
>> to build derivations.  Or it is a lucky day. :-)
>
> Right.
>
> So what can we do to address this issue?  I *think* we could use a
> higher TTL on berlin, and we can try that right away (9 months to being
> with?).

I *think* the issue is not TTL for question #1.  :-) But the issue that
the both build farms do not build these “modular” derivations for all
the commits.  Here, I am focused on x86_64-linux which is the case of
interest for such topic (scientific context), IMHO.

Considering to build for every commit for all architectures is not
affordable.

I agree that increasing the TTL will help for question #2 about
long-support of substitutes.

> However, there is an upper bound anyway.  To make informed decisions on
> the retention policy, we should monitor storage space on berlin/bayfront
> to better estimate what can be done.  We have Zabbix but it’s not
> accessible from the outside; maybe we could graph storage space
> somewhere so people can grab the data and work on those estimates?

Based on the size of these derivations for one commit, we could
extrapolate back to envelope.  Well, question #1 seems doable
storage-speaking.

The issue of #1 is to build these derivations for all the commits.
IMHO.

About #2, yeah if some data are available, I can try to make some
estimates.


Well, #1 seems actionable.  However, #2 raises…

> What if we decide that we need to provide substitutes for 2y old
> commits?  In that case, we need a plan to scale up.  That could be
> renting storage space somewhere.  That’s largely non-technical work that
> needs attention.

…a strong question. :-) What do “we” do for what “we” build?

Indeed, numbers are missing to make informed decisions on long-term
storage of substitutes.  What is Nix doing?


> There are also technical tweaks that could help: distinguishing between
> “important” substitutes that we want to keep, and less important
> substitutes (how?); identifying “equivalence classes” for builds of a
> given package; etc.  The outcome is unclear and it’ll take time.

I agree it will take time. :-)


I think that having 2 build farms building in parallel is a strength.
So let exploit it. :-) What one could have in mind is to challenge the
outputs; if they are identical, let keep only one version “somewhere”
and remove the other from the “elsewhere”.

For instance, we (I? with help) could resume this discussion:

<https://lists.gnu.org/archive/html/guix-devel/2020-10/msg00181.html>


Or maybe, for the identical outputs, one could imagine (dream? for) a
cooking service for missing outputs.  Well, I do not know how this is
actionable. :-)


Cheers,
simon

Re: Substitute retention

[Ludovic Courtès]

Hi!

zimoun <zimon.toutoune@gmail.com> skribis:

>>> missed by both build farms using 2 different strategies to collect the
>>> thing to build (fetch every 5 minutes or fetch from guix-commits).  It
>>> is a quick back to envelope so keep that with some salt. :-)
>>
>> OK.
>
> To make it explicit of #1, I was talking about the “modular” Guix, i.e.,
> when running “guix pull” or “guix time-machine” it leads to build the
> derivations module-import.drv, guix-<hash>.drv, guix-command.drv,
> guix-module-union.drv, guix-<hash>-modules.drv,
> guix-packages-modules.drv, guix-system-tests-modules.drv,
> guix-packages-base-modules.drv, etc.  On slow machines, it can be
> unpleasant; not to say unpractical.  Even for recent commits.

Ah I see.  Yeah, this can be kinda annoying, and amplified by the fact
that CI only builds at each push, not at each commit.

That said, this is mitigated by the fact that one typically travels to a
previously-fetched commit, which is a commit that has been built by CI
rather than a commit in between two pushes.

> Basically, commit 59d10c3112 is from March 14, 2020 and it takes ~29min
> on my slow laptop.  And to compare apple to apple, let take another
> commit one year later from March 14, 2021, e.g., commit 7327295462.  It
> takes ~5min on the same machine.

Yeah, OK.

> To be on the same wavelength,
>
> $ git log --format="%h %cd" --after=2021-03-14 --reverse | head -n16
> [...]
> 2babf7d831 Sun Mar 14 19:16:55 2021 +0100
> b15720182e Sun Mar 14 13:24:21 2021 -0500
> 207aa62e6b Sun Mar 14 13:24:21 2021 -0500
> 30f5381487 Sun Mar 14 13:24:21 2021 -0500
> af25357b7d Sun Mar 14 13:24:21 2021 -0500
> 7164d2105a Sun Mar 14 13:24:21 2021 -0500
> 078f3288e2 Sun Mar 14 13:24:21 2021 -0500
> 5a31eb7d35 Sun Mar 14 13:24:21 2021 -0500
> 620206b680 Sun Mar 14 13:24:22 2021 -0500
> b76762a9b7 Sun Mar 14 13:24:22 2021 -0500
> cbfcbb79df Sun Mar 14 19:43:35 2021 +0100
>
> and Cuirass builds only one of b15720182e, 207aa62e6b, 30f5381487,
> af25357b7d, 7164d2105a, 078f3288e2, 5a31eb7d35, 620206b680 or
> b76762a9b7.
>
> Considering the Build Coordinator, it uses guix-commits and from my
> understanding it reads:
>
> <https://lists.gnu.org/archive/html/guix-commits/2021-03/msg01201.html>
>
> therefore, b15720182e would be missed but not b76762a9b7–which would be
> missed by Cuirass.
>
> Cuirass and the Build Coordinator cannot each build the both commits
> b15720182e and b76762a9b7.
>
> Cuirass check every 5 minutes and Build Coordinator reads “state” from
> guix-commits.  Other said, none of them builds all these “modular”
> derivations for all the commits; even for recent commits.
>
> The rough estimate is half of commits are missed by both build farms.
> Therefore, using “guix time-machine” with a random commit and one gets
> 1/2 probability to build something just to get the inferior – aside the
> TTL policy.

Right.  Not every derivation produced by (guix self) needs to be rebuilt
in between two commits, but anything that depends on *package-modules*
typically has to be rebuilt.

We can reduce the amount of rebuilt like I did in commit
abd38dcee16f0ac71191527c38dcd3659111e2ba, but you’ll always have the big
(gnu packages …) derivation.

>> So what can we do to address this issue?  I *think* we could use a
>> higher TTL on berlin, and we can try that right away (9 months to being
>> with?).
>
> I *think* the issue is not TTL for question #1.  :-) But the issue that
> the both build farms do not build these “modular” derivations for all
> the commits.  Here, I am focused on x86_64-linux which is the case of
> interest for such topic (scientific context), IMHO.
>
> Considering to build for every commit for all architectures is not
> affordable.
>
> I agree that increasing the TTL will help for question #2 about
> long-support of substitutes.

Understood!

>> However, there is an upper bound anyway.  To make informed decisions on
>> the retention policy, we should monitor storage space on berlin/bayfront
>> to better estimate what can be done.  We have Zabbix but it’s not
>> accessible from the outside; maybe we could graph storage space
>> somewhere so people can grab the data and work on those estimates?
>
> Based on the size of these derivations for one commit, we could
> extrapolate back to envelope.  Well, question #1 seems doable
> storage-speaking.
>
> The issue of #1 is to build these derivations for all the commits.
> IMHO.
>
> About #2, yeah if some data are available, I can try to make some
> estimates.
>
>
> Well, #1 seems actionable.  However, #2 raises…
>
>> What if we decide that we need to provide substitutes for 2y old
>> commits?  In that case, we need a plan to scale up.  That could be
>> renting storage space somewhere.  That’s largely non-technical work that
>> needs attention.
>
> …a strong question. :-) What do “we” do for what “we” build?
>
> Indeed, numbers are missing to make informed decisions on long-term
> storage of substitutes.  What is Nix doing?

Nix, AFAIK, is doing like everyone else: pouring money on Amazon.  Last
I heard they’d retain substitutes basically indefinitely on Amazon S3
(incidentally, one motivation for them to work with Software Heritage,
AIUI, is that it would allow them to store less data on the storage they
pay for :-)).

For the record, berlin (aka ci.guix.gnu.org; it was donated by the Max
Delbrück Center, MDC, and is generously hosted by them) has a 37 TiB
disk for /gnu/store and “baked” substitutes.  That’s a lot.

Technically though, a lot of it is used by less important substitutes
such as disk images or intermediate ‘core-updates’ substitutes.

In the end we seem to be filling it more quickly than you’d think!

Perhaps we need a better strategy with a low TTL for, say, intermediate
‘core-updates’ substitutes (no need to keep them more than a few weeks
if we know we’re doing a world rebuild right after).  It cannot be done
as things are though because ‘guix publish’ doesn’t distinguish between
store items.

Or we could restart the Amazon front-end that Chris Marusich had set up
right before 1.0 was released.  Or we could build our own front-end for
substitute delivery as a proxy to berlin, thereby distributing the
burden.

Thoughts?

> I think that having 2 build farms building in parallel is a strength.
> So let exploit it. :-) What one could have in mind is to challenge the
> outputs; if they are identical, let keep only one version “somewhere”
> and remove the other from the “elsewhere”.
>
> For instance, we (I? with help) could resume this discussion:
>
> <https://lists.gnu.org/archive/html/guix-devel/2020-10/msg00181.html>

I hadn’t seen this message, interesting!

Note however that bordeaux.guix has a tenth of the storage space of
berlin (3.6 TiB), so right now we probably can’t count on it for
long-term substitute storage.

> Or maybe, for the identical outputs, one could imagine (dream? for) a
> cooking service for missing outputs.  Well, I do not know how this is
> actionable. :-)

Well, if we keep .drv around, we could arrange so that ‘guix publish’
rebuilds on-demand, after all.  I’m not sure how practical that would
be, though.

Ludo’.