Re: Dropping gzip-compressed substitutes

[Maxim Cournoyer <maxim.cournoyer@gmail.com>]

Hi everyone,

Mathieu Othacehe <othacehe@gnu.org> writes:

> Hey Ludo,
>
>> As discussed on IRC, I’m skeptical about this because:
>>
>>   1. It requires the development and testing of a custom tool that’s
>>      easy to get wrong—e.g., it removes a gzipped nar for something that
>>      had nothing but gzip available, etc.
>>
>>   2. That code would have to run with privileges that give it access to
>>      the signing key on berlin.
>>
>>   3. Those 6.5 TB are an initial constant factor; growth of the storage
>>      requirements going forward probably matters more and
>>      <https://issues.guix.gnu.org/53901> will give us more flexibility
>>      on that.
>
> While those are valid points, we need to keep in mind that it is
> important that we manage to move the store to the new SSD array quite
> quickly to start GCing again.

I turns out that the store contains about 14 TiB of probably mostly
unnecessary things that just weren't garbage collected (due to the
prohibitive cost of doing so on the HDD hard drive).

Currently the new array still has 5 TiB so I think we can try to migrate
today and then work trimming the gzip substitutes to have some extra
head room.

With Ricardo, we've attempted to reboot Berlin onto a freshly 'guix
system init'ed system using the new array last Thursday.  Unfortunately
we hit some issue where mounting the root file system was apparently
occurring before all the 6 drives had the time to be visible.

This week, I'd like to try the following to see if we could get past
this:

1. Do the experiment again, now a 'rootdelay=20' kernel parameter was
added to Berlin's config.  This may well be enough.

2. In case mounting the RAID 10 Btrfs root partition still fails with
missing drive errors, try the following workaround suggested in the
#btrfs channel, which forces a 'btrfs device scan' on each device of the
array, with the following mount option:

"device=/dev/sda3,/dev/sdb3,/dev/sdc3,/dev/sdd3,/dev/sde3,/dev/sdf3"

To make it more convenient to experiment with different values for the
rootdelay or add the device option above, I'm planning to 'guix system init'
with the following patch applied: https://issues.guix.gnu.org/40998,
which allows providing 'rootflags' directly from the kernel command line
(thus by editing the GRUB config at boot).

I'll try to synchronize with Ricardo in the channel and hope they
weren't too frightened by our last experiment to not shy away from
trying again :-).

That's it for the update.  Until this is sorted out, I'd ask to not
reboot the machine, and not even reconfigure it.  I'll give you all an
update as soon as it is sorted out.

Thanks!

Maxim