Re: Building and caching old Guix derivations for a faster time machine

["Ludovic Courtès" <ludo@gnu.org>]

Hi,

Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:

>> I agree.  The ‘guix publish’ TTL¹ at ci.guix was increased to 180 days
>> following <https://issues.guix.gnu.org/48926> in 2021.  That’s still not
>> that much and these days and right now we have 84 TiB free at ci.guix.
>>
>> I guess we can afford increasing the TTL, probably starting with, say,
>> 300 days, and monitoring disk usage.
>>
>> WDYT?
>
> While the 84 TiB we have at our disposal is indeed lot, I'd rather we
> keep the TTL at 180 days, to keep things more manageable for backup/sync
> purposes.  Our current TTL currently yields 7 TiB of compressed NARs,
> which fits nicely into the hydra-guix-129 10 TiB slice available for
> local/simple redundancy (it's still on my TODO, missing the copy bit).
>
> I've been meaning to document an easy mirroring setup for that
> /var/cache/guix/publish directory, and having 14 TiB instead of 7 TiB
> there would hurt such setups.

Maybe we should learn from what Chris has been doing with the
Nar-Herder, too.  Ideally, the build farm front-end (‘berlin’ in this
case) would be merely a cache for recently-built artifacts, and we’d
have long-term storage elsewhere where we could keep nars for several
years.

The important thing being: we need to decouple the build farm from
(long-term) nar provision.

> Perhaps a compromise we could do is drop yet another compression format?
> We carry both Zstd and LZMA for Berlin, which I see little value in; if
> we carried only ZSTD archives we could probably continue having < 10 TiB
> of NARs for a TTL of 360 days (although having only 3.5 TiB of NARs to
> sync around for mirrors would be great too!).
>
> What do you think?

For compatibility reasons¹ and performance reasons², I would refrain
from removing lzip or zstd substitutes, at least for “current”
substitutes.

For long-term storage though, we could choose to keep lzip only (because
it compresses better).  Not something we can really do with the current
‘guix publish’ setup though.

Thoughts?

Ludo’.

¹ Zstd support was added relatively recently.  Older daemons may support
  lzip but not zstd.

² https://guix.gnu.org/en/blog/2021/getting-bytes-to-disk-more-quickly/