Re: Building and caching old Guix derivations for a faster time machine

[Maxim Cournoyer <maxim.cournoyer@gmail.com>]

Hi Guillaume,

Guillaume Le Vaillant <glv@posteo.net> writes:

> Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:
>
>> Hi Simon,
>>
>> Simon Tournier <zimon.toutoune@gmail.com> writes:
>>
>>> Hi,
>>>
>>> On mer., 22 nov. 2023 at 19:27, Ludovic Courtès <ludo@gnu.org> wrote:
>>>
>>>> For long-term storage though, we could choose to keep lzip only (because
>>>> it compresses better).  Not something we can really do with the current
>>>> ‘guix publish’ setup though.
>>>
>>> It looks good to me.  For me, the priority list looks like:
>>
>> I'd like to have a single archive type as well in the future, but I'd
>> settle on Zstd, not lzip, because it's faster to compress and
>> decompress, and its compression ratio is not that different when using
>> its highest level (19).
>
> Last time I checked, zstd with max compression (zstd --ultra -22) was
> a little slower and had a little lower compression ratio than lzip with
> max compression (lzip -9).
> Zstd is however much faster for decompression.

I think when we talk about performance of NARs, we mean it in the
context of a Guix user installing them (decompressing) more than in the
context of the CI producing them, so zstd beats lzip here.

> Another thing that could be useful to consider is that lzip was designed
> for long term storage, so it has some redundancy allowing fixing/recovering
> a corrupt archive (e.g. using lziprecover) if there has been some bit
> rot in the hardware storing the file.
> Whereas as far as I know zstd will just tell you "error: bad checksum"
> and will have no way to fix the archive.

That's an interesting aspect of lzip, but in this age of CRC-check file
systems like Btrfs, we have other means on ensuring data integrity (and
recovery, assuming we have backups available).

I'm still of the opinion that carrying a single set of zstd-only NARs
makes the most sense in the long run.

-- 
Thanks,
Maxim