Re: Building and caching old Guix derivations for a faster time machine

[Maxim Cournoyer <maxim.cournoyer@gmail.com>]

Hi Simon,

Simon Tournier <zimon.toutoune@gmail.com> writes:

> Hi Maxim,
>
> On Thu, 30 Nov 2023 at 08:28, Maxim Cournoyer <maxim.cournoyer@gmail.com> wrote:
>
>> I'd like to have a single archive type as well in the future, but I'd
>> settle on Zstd, not lzip, because it's faster to compress and
>> decompress, and its compression ratio is not that different when using
>> its highest level (19).
>
> When running an inferior (past revision), some past Guile code as it was
> in this past revision is launched.  Hum, I have never checked: the
> substitution mechanism depends on present revision code (Guile and
> daemon) or on past revision?
>
> Other said, what are the requirements for the backward compatibility?
> Being able to run past Guix from a recent Guix, somehow.

We're only impacting the future, not the past, I think.  The inferior
mechanism still relies on the same daemon, as far as I know, and the
currently available gzipped nars would remain available according to
their current retention policy (6 months when unused).

>>>  1. Keep for as longer as we can all the requirements for running Guix
>>>  itself, e.g., “guix time-machine”.  Keep all the dependencies and all
>>>  the outputs of derivations.  At least, for all the ones the build farms
>>>  are already building.
>>>
>>>  2. Keep for 3-5 years all the outputs for specific Guix revision, as
>>>  v1.0, v1.1, v1.2, v1.3, v1.4.  And some few others.
>>
>> That'd be nice, but not presently doable as we can't fine tune retention
>> for a particular 'derivation' and its inputs in the Cuirass
>> configuration, unless I've missed it.
>
> That’s an implementation detail, a bug or a feature request, pick the
> one you prefer. ;-)

I'd say it's a feature request :-).

> We could imagine various paths for these next steps, IMHO.  For
> instance, we could move these outputs to some specific stores
> independent of the current ones (ci.guix and bordeaux.guix).  For
> instance, we could have “cold” storage with some cooking bakery for
> making hot again, instead of keeping all hot.  For instance, we could
> imagine etc. :-)
>
> Well, I do not have think much and I just speak loud: Cuirass (and Build
> Coordinator) are the builders, and I would not rely on them for some NAR
> “archiving“, instead maybe “we” could put some love into the tool
> nar-herder.  Somehow, extract specific NAR that the project would like
> to keep longer than the unpredictable current mechanism.

It seems the nar-herder would perhaps be well suited for this, if
someone is inclined to implement it, given it keeps each nars in a
database, which should make it fast to query for all the 'guix' packages
substitutes.  Perhaps it even has (or could have) hooks when registering
a new nars which could define what is done to it (send to another
server).

Otherwise good old 'find' could be used to rsync the 'guix' named nars
and their .narinfo metadata files to a different location, but that'd
probably be less efficient (IO-intensive) on the huge multi-terabytes
collection of nars we carry.

-- 
Thanks,
Maxim