Dropping gzip-compressed substitutes

Dropping gzip-compressed substitutes

[Maxim Cournoyer]

Hello Guix!

While migrating our file system to the newly installed SSD-backed
storage on Berlin, I noticed that the guix-publish cached nars use a lot
of space: 6.5 TiB for the gzip compressed nars alone, or about 30% of
the new storage capacity.

The new array capacity was recently added so that running 'guix gc' on
it would be able to complete in a reasonable amount of time (remember
that no substitutes from the Cuirass workers can be retrieved while the
big GC lock is on, which greatly hurts the performance of the CI),
allowing to keep the size of /gnu/store in check.

Its capacity was chosen based on the assumption that /gnu/store
compresses well with zstd, leading to a 1.5-2.0 factor capacity boost.
This assumption doesn't hold in case we are storing a large quantity of
already-compressed data such as the /var/cache/guix/publish compressed
nars.

From a past survey of the use of gzip substitutes [0], the result was a
small percentage of our users were still relying on them (1%).

Here's what I would suggest for a migration path:

1.  Promptly set up both a blog post and a NEWS entry announcing the
support for gzip substitutes is about to be phased out from the build
farm (1 month notice), urging users to upgrade their daemon to a version
>= 1.1.0.

2. Do the following in the maintenance repo, which should stop
advertising/generating new gzip substitutes:

--8<---------------cut here---------------start------------->8---
modified   hydra/modules/sysadmin/services.scm
@@ -580,9 +580,7 @@ to a selected directory.")
                    ;; <https://lists.gnu.org/archive/html/guix-devel/2021-01/msg00097.html>
                    ;; for the compression ratio/decompression speed
                    ;; tradeoffs.
-                   ;; TODO: Eventually, disable gzip, as discussed at
-                   ;; <https://lists.gnu.org/archive/html/guix-devel/2021-03/msg00333.html>.
-                   (compression '(("gzip" 9) ("lzip" 9) ("zstd" 19)))
+                   (compression '(("lzip" 9) ("zstd" 19)))
                    (cache-bypass-threshold cache-bypass-threshold)
                    (workers publish-workers)))
--8<---------------cut here---------------end--------------->8---
 
3. Come up with a Guile script that is able to

  a) Strip gzip-related metadata from the .narinfo guix-publish metadata
  files
  b) recompute and update their 'Signature' field.

4. Finally, 'rm -r /var/guix/publish/gzip' and free about 6.5 TiB of data.

I'd like to move promptly with this as I'm expecting it may be necessary
to fully migrate the current file system to the new array (and reap the
benefits).

Thank you,

Maxim

[0]  https://lists.gnu.org/archive/html/guix-devel/2021-03/msg00333.html

Re: Dropping gzip-compressed substitutes

[Ricardo Wurmus]

Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:

> 1.  Promptly set up both a blog post and a NEWS entry announcing the
> support for gzip substitutes is about to be phased out from the build
> farm (1 month notice), urging users to upgrade their daemon to a version
>>= 1.1.0.

Can we make the “guix” command print a warning if the daemon it talks to
only supports gzip substitutes?  I know many users who don’t read our
blog posts and who ignore the news output, but who are very keen on
getting rid of warnings that the “guix” command itself prints while they
want to do other things.

We could probably reach these people with a nagging warning that is
about as annoying as the infamous locales warning :)

-- 
Ricardo

Re: Dropping gzip-compressed substitutes

[Mathieu Othacehe]

[-- Attachment #1: Type: text/plain, Size: 1223 bytes --]


Hey Maxim,

Sound like a fine plan.

> 1.  Promptly set up both a blog post and a NEWS entry announcing the
> support for gzip substitutes is about to be phased out from the build
> farm (1 month notice), urging users to upgrade their daemon to a version
>>= 1.1.0.

I addition, we could warn users with old daemons as proposed by Ludo &
Ricardo. There's an attached patch here.

> -                   ;; TODO: Eventually, disable gzip, as discussed at
> -                   ;; <https://lists.gnu.org/archive/html/guix-devel/2021-03/msg00333.html>.
> -                   (compression '(("gzip" 9) ("lzip" 9) ("zstd" 19)))
> +                   (compression '(("lzip" 9) ("zstd" 19)))
>                     (cache-bypass-threshold cache-bypass-threshold)
>                     (workers publish-workers)))

Nice.

> 3. Come up with a Guile script that is able to
>
>   a) Strip gzip-related metadata from the .narinfo guix-publish metadata
>   files
>   b) recompute and update their 'Signature' field.
>
> 4. Finally, 'rm -r /var/guix/publish/gzip' and free about 6.5 TiB of data.

I hope the script will be able to complete within a reasonable amount of
time, but we cannot know before trying it out :).

Thanks,

Mathieu

[-- Attachment #2: 0001-store-Warn-about-daemon-deprecation.patch --]
[-- Type: text/x-patch, Size: 2173 bytes --]

From bd306a8b20f2033d67755dd332a5d33b2f6b822d Mon Sep 17 00:00:00 2001
From: Mathieu Othacehe <othacehe@gnu.org>
Date: Tue, 8 Feb 2022 14:28:56 +0100
Subject: [PATCH 1/1] store: Warn about daemon deprecation.

* guix/deprecation.scm (warn-about-old-daemon): New procedure.
* guix/store.scm (build-things): Use it to warn about old daemons.
---
 guix/deprecation.scm | 7 +++++++
 guix/store.scm       | 2 ++
 2 files changed, 9 insertions(+)

diff --git a/guix/deprecation.scm b/guix/deprecation.scm
index c66c9367f6..666e178d75 100644
--- a/guix/deprecation.scm
+++ b/guix/deprecation.scm
@@ -1,5 +1,6 @@
 ;;; GNU Guix --- Functional package management for GNU
 ;;; Copyright © 2019, 2020, 2021 Ludovic Courtès <ludo@gnu.org>
+;;; Copyright © 2021 Mathieu Othacehe <othacehe@gnu.org>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -24,6 +25,8 @@ (define-module (guix deprecation)
 
             define-deprecated/public
             define-deprecated/alias
+
+            warn-about-old-daemon
             warn-about-deprecation))
 
 ;;; Commentary:
@@ -32,6 +35,10 @@ (define-module (guix deprecation)
 ;;;
 ;;; Code:
 
+(define (warn-about-old-daemon)
+  (warning (G_ "Your Guix daemon is seriously outdated, please consider
+ updating it by following the 'Upgrading Guix' documentation section.~%")))
+
 (define* (warn-about-deprecation variable properties
                                  #:key replacement)
   (let ((location (and properties (source-properties->location properties))))
diff --git a/guix/store.scm b/guix/store.scm
index a93e9596d9..11dfc095b1 100644
--- a/guix/store.scm
+++ b/guix/store.scm
@@ -1442,6 +1442,8 @@ (define build-things
                              things)))
             (parameterize ((current-store-protocol-version
                             (store-connection-version store)))
+              (when (< (current-store-protocol-version) 355) ;0x163
+                (warn-about-old-daemon))
               (if (>= (store-connection-minor-version store) 15)
                   (build store things mode)
                   (if (= mode (build-mode normal))
-- 
2.34.0

Re: Dropping gzip-compressed substitutes

[Maxim Cournoyer]

Hi Mathieu!

Mathieu Othacehe <othacehe@gnu.org> writes:

> Hey Maxim,
>
> Sound like a fine plan.
>
>> 1.  Promptly set up both a blog post and a NEWS entry announcing the
>> support for gzip substitutes is about to be phased out from the build
>> farm (1 month notice), urging users to upgrade their daemon to a version
>>>= 1.1.0.
>
> I addition, we could warn users with old daemons as proposed by Ludo &
> Ricardo. There's an attached patch here.

Awesome, thank you!  It LGTM.  Could you apply it?

>> -                   ;; TODO: Eventually, disable gzip, as discussed at
>> -                   ;; <https://lists.gnu.org/archive/html/guix-devel/2021-03/msg00333.html>.
>> -                   (compression '(("gzip" 9) ("lzip" 9) ("zstd" 19)))
>> +                   (compression '(("lzip" 9) ("zstd" 19)))
>>                     (cache-bypass-threshold cache-bypass-threshold)
>>                     (workers publish-workers)))
>
> Nice.
>
>> 3. Come up with a Guile script that is able to
>>
>>   a) Strip gzip-related metadata from the .narinfo guix-publish metadata
>>   files
>>   b) recompute and update their 'Signature' field.
>>
>> 4. Finally, 'rm -r /var/guix/publish/gzip' and free about 6.5 TiB of data.
>
> I hope the script will be able to complete within a reasonable amount of
> time, but we cannot know before trying it out :).

The script will run on the cache data kept on the SSD array (which we
can snapshot before experimenting with). It'll probably take a while
still, so care should be taken to ensure it takes as little time as
possible (e.g. spawning many threads or processes to spread the load
across multiple CPUs).

I'll work on the blog post and news entry, and then we'll have to look
at the script.

Thank you!

Maxim

Re: Dropping gzip-compressed substitutes

[Ricardo Wurmus]

Hi Mathieu,

thanks for the patch!

> From bd306a8b20f2033d67755dd332a5d33b2f6b822d Mon Sep 17 00:00:00 2001
> From: Mathieu Othacehe <othacehe@gnu.org>
> Date: Tue, 8 Feb 2022 14:28:56 +0100
> Subject: [PATCH 1/1] store: Warn about daemon deprecation.
>
> * guix/deprecation.scm (warn-about-old-daemon): New procedure.
> * guix/store.scm (build-things): Use it to warn about old daemons.
> ---
>  guix/deprecation.scm | 7 +++++++
>  guix/store.scm       | 2 ++
>  2 files changed, 9 insertions(+)
>
> diff --git a/guix/deprecation.scm b/guix/deprecation.scm
> index c66c9367f6..666e178d75 100644
> --- a/guix/deprecation.scm
> +++ b/guix/deprecation.scm
> @@ -1,5 +1,6 @@
>  ;;; GNU Guix --- Functional package management for GNU
>  ;;; Copyright © 2019, 2020, 2021 Ludovic Courtès <ludo@gnu.org>
> +;;; Copyright © 2021 Mathieu Othacehe <othacehe@gnu.org>
>  ;;;
>  ;;; This file is part of GNU Guix.
>  ;;;
> @@ -24,6 +25,8 @@ (define-module (guix deprecation)
>  
>              define-deprecated/public
>              define-deprecated/alias
> +
> +            warn-about-old-daemon
>              warn-about-deprecation))
>  
>  ;;; Commentary:
> @@ -32,6 +35,10 @@ (define-module (guix deprecation)
>  ;;;
>  ;;; Code:
>  
> +(define (warn-about-old-daemon)
> +  (warning (G_ "Your Guix daemon is seriously outdated, please consider
> + updating it by following the 'Upgrading Guix' documentation section.~%")))
> +

s/documention section/section in the manual/

Should it also mention that staying with the old daemon means that
binaries will no longer be available?  For example:

   Your Guix daemon is no longer supported and will soon no longer be
   able to download binary substitutes.

I realize that what you proposed is a generic warning, but perhaps it’s
better to mention a direct consequence of failing to upgrade.  (I know
many people who will not upgrade unless they absolutely have to.)

> --- a/guix/store.scm
> +++ b/guix/store.scm
> @@ -1442,6 +1442,8 @@ (define build-things
>                               things)))
>              (parameterize ((current-store-protocol-version
>                              (store-connection-version store)))
> +              (when (< (current-store-protocol-version) 355) ;0x163
> +                (warn-about-old-daemon))
>                (if (>= (store-connection-minor-version store) 15)
>                    (build store things mode)
>                    (if (= mode (build-mode normal))

Maybe add a comment here to explain why this particular protocol version
is considered old.

-- 
Ricardo

Re: Dropping gzip-compressed substitutes

[Maxim Cournoyer]

Hi Ricardo and Mathieu,

Ricardo Wurmus <rekado@elephly.net> writes:

> Hi Mathieu,
>
> thanks for the patch!
>
>> From bd306a8b20f2033d67755dd332a5d33b2f6b822d Mon Sep 17 00:00:00 2001
>> From: Mathieu Othacehe <othacehe@gnu.org>
>> Date: Tue, 8 Feb 2022 14:28:56 +0100
>> Subject: [PATCH 1/1] store: Warn about daemon deprecation.
>>
>> * guix/deprecation.scm (warn-about-old-daemon): New procedure.
>> * guix/store.scm (build-things): Use it to warn about old daemons.
>> ---
>>  guix/deprecation.scm | 7 +++++++
>>  guix/store.scm       | 2 ++
>>  2 files changed, 9 insertions(+)
>>
>> diff --git a/guix/deprecation.scm b/guix/deprecation.scm
>> index c66c9367f6..666e178d75 100644
>> --- a/guix/deprecation.scm
>> +++ b/guix/deprecation.scm
>> @@ -1,5 +1,6 @@
>>  ;;; GNU Guix --- Functional package management for GNU
>>  ;;; Copyright © 2019, 2020, 2021 Ludovic Courtès <ludo@gnu.org>
>> +;;; Copyright © 2021 Mathieu Othacehe <othacehe@gnu.org>
>>  ;;;
>>  ;;; This file is part of GNU Guix.
>>  ;;;
>> @@ -24,6 +25,8 @@ (define-module (guix deprecation)
>>  
>>              define-deprecated/public
>>              define-deprecated/alias
>> +
>> +            warn-about-old-daemon
>>              warn-about-deprecation))
>>  
>>  ;;; Commentary:
>> @@ -32,6 +35,10 @@ (define-module (guix deprecation)
>>  ;;;
>>  ;;; Code:
>>  
>> +(define (warn-about-old-daemon)
>> +  (warning (G_ "Your Guix daemon is seriously outdated, please consider
>> + updating it by following the 'Upgrading Guix' documentation section.~%")))
>> +
>
> s/documention section/section in the manual/
>
> Should it also mention that staying with the old daemon means that
> binaries will no longer be available?  For example:
>
>    Your Guix daemon is no longer supported and will soon no longer be
>    able to download binary substitutes.
>
> I realize that what you proposed is a generic warning, but perhaps it’s
> better to mention a direct consequence of failing to upgrade.  (I know
> many people who will not upgrade unless they absolutely have to.)

Good idea; I modified it slightly like so:

--8<---------------cut here---------------start------------->8---
@@ -36,8 +36,9 @@ (define-module (guix deprecation)
 ;;; Code:
 
 (define (warn-about-old-daemon)
-  (warning (G_ "Your Guix daemon is seriously outdated, please consider
- updating it by following the 'Upgrading Guix' documentation section.~%")))
+  (warning (G_ "Your Guix daemon is severely outdated, and will soon cease to
+be able to download binary substitutes.  To upgrade it, refer to the
+'Upgrading Guix' section in the manual.~%")))
 
 (define* (warn-about-deprecation variable properties
                                  #:key replacement)
--8<---------------cut here---------------end--------------->8---
                                  
>> --- a/guix/store.scm
>> +++ b/guix/store.scm
>> @@ -1442,6 +1442,8 @@ (define build-things
>>                               things)))
>>              (parameterize ((current-store-protocol-version
>>                              (store-connection-version store)))
>> +              (when (< (current-store-protocol-version) 355) ;0x163
>> +                (warn-about-old-daemon))
>>                (if (>= (store-connection-minor-version store) 15)
>>                    (build store things mode)
>>                    (if (= mode (build-mode normal))
>
> Maybe add a comment here to explain why this particular protocol version
> is considered old.

I added the following:

--8<---------------cut here---------------start------------->8---
@@ -1442,7 +1442,11 @@ (define build-things
                              things)))
             (parameterize ((current-store-protocol-version
                             (store-connection-version store)))
-              (when (< (current-store-protocol-version) 355) ;0x163
+              (when (< (current-store-protocol-version) #x163)
+                ;; This corresponds to the first version bump of the daemon
+                ;; since the introduction of lzip compression support.  The
+                ;; version change happened with commit 6ef61cc4c30 on the
+                ;; 2018/10/15).
                 (warn-about-old-daemon))
               (if (>= (store-connection-minor-version store) 15)
                   (build store things mode)
--8<---------------cut here---------------end--------------->8---

And pushed with commit 96d7535b03.

Thanks for the feedback!

Maxim

Re: Dropping gzip-compressed substitutes

[Maxim Cournoyer]

Hi again,

Syncing /var/cache has finally finished, after copying 16 TiB from the
old array in about 16 days:

rsync warning: some files vanished before they could be transferred (code 24) at main.c(1330) [sender=3.2.3]

real    17019m52.264s
user    147m58.061s
sys     686m3.528s

I've now pushed d4b73e1 to guix-artwork which is a blog post about the
sunsetting of gzip substitutes, planned for next month, which should
allow to shave 6.5 TiB of that amount.

Feel free to edit it to correct any typo or other mishaps I may have
made.

With this, we have about 3 weeks to devise the script needed to sanitize
the narinfo files.

Thank you,

Maxim

Re: Dropping gzip-compressed substitutes

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 1757 bytes --]

On Tue, Feb 08, 2022 at 02:34:49PM +0100, Mathieu Othacehe wrote:
> 
> Hey Maxim,
> 
> Sound like a fine plan.
> 
> > 1.  Promptly set up both a blog post and a NEWS entry announcing the
> > support for gzip substitutes is about to be phased out from the build
> > farm (1 month notice), urging users to upgrade their daemon to a version
> >>= 1.1.0.
> 
> I addition, we could warn users with old daemons as proposed by Ludo &
> Ricardo. There's an attached patch here.
> 
> > -                   ;; TODO: Eventually, disable gzip, as discussed at
> > -                   ;; <https://lists.gnu.org/archive/html/guix-devel/2021-03/msg00333.html>.
> > -                   (compression '(("gzip" 9) ("lzip" 9) ("zstd" 19)))
> > +                   (compression '(("lzip" 9) ("zstd" 19)))
> >                     (cache-bypass-threshold cache-bypass-threshold)
> >                     (workers publish-workers)))
> 
> Nice.
> 
> > 3. Come up with a Guile script that is able to
> >
> >   a) Strip gzip-related metadata from the .narinfo guix-publish metadata
> >   files
> >   b) recompute and update their 'Signature' field.
> >
> > 4. Finally, 'rm -r /var/guix/publish/gzip' and free about 6.5 TiB of data.
> 
> I hope the script will be able to complete within a reasonable amount of
> time, but we cannot know before trying it out :).

I'm sure the internet is full of people who know if this is fastest, or
using find /var/guix/publish/gzip -delete or find ... -print0 | xargs -0
rm or something else :)


-- 
Efraim Flashner   <efraim@flashner.co.il>   רנשלפ םירפא
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Dropping gzip-compressed substitutes

[Ludovic Courtès]

Hi!

Thumbs up for the progress made!

Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:

> 3. Come up with a Guile script that is able to
>
>   a) Strip gzip-related metadata from the .narinfo guix-publish metadata
>   files
>   b) recompute and update their 'Signature' field.
>
> 4. Finally, 'rm -r /var/guix/publish/gzip' and free about 6.5 TiB of data.

As discussed on IRC, I’m skeptical about this because:

  1. It requires the development and testing of a custom tool that’s
     easy to get wrong—e.g., it removes a gzipped nar for something that
     had nothing but gzip available, etc.

  2. That code would have to run with privileges that give it access to
     the signing key on berlin.

  3. Those 6.5 TB are an initial constant factor; growth of the storage
     requirements going forward probably matters more and
     <https://issues.guix.gnu.org/53901> will give us more flexibility
     on that.

I like Chris Baines’ idea of decoupling nar distribution from nar
building.  If we want to keep nars long enough so that ‘time-machine’ is
usable, then storage requirements will keep growing.

Perhaps that means we can regularly copy nars “elsewhere” for long-term
storage, using nar-herder, rsync, or whatever.  The machine that stores
nars long-term has low requirements compared to the build farm because
we don’t need to trust it for anything other than storage.  If that
makes things easier (and financially viable), a VPS is good enough.

Thoughts?

Ludo’.

Re: Dropping gzip-compressed substitutes

[Attila Lendvai]

>     building. If we want to keep nars long enough so that ‘time-machine’ is
>     usable, then storage requirements will keep growing.
>
>     Perhaps that means we can regularly copy nars “elsewhere” for long-term
>     storage, using nar-herder, rsync, or whatever. The machine that stores
>     nars long-term has low requirements compared to the build farm because
>     we don’t need to trust it for anything other than storage. If that
>     makes things easier (and financially viable), a VPS is good enough.


if the ERIS project proceeds with a reasonable pace (and the Swarm backend,
too), then it would be a perfect opportunity to start stress testing the various
distributed storage backends.

unfortunately, for now Swarm only works through my custom channel that downloads
the release binaries, but that can be fixed, too.

--
• attila lendvai
• PGP: 963F 5D5F 45C7 DFCD 0A39
--
“And remember, where you have a concentration of power in a few hands, all too frequently men with the mentality of gangsters get control. History has proven that.”
	— Lord Acton (1834–1902)

Re: Dropping gzip-compressed substitutes

[Mathieu Othacehe]

Hey Ludo,

> As discussed on IRC, I’m skeptical about this because:
>
>   1. It requires the development and testing of a custom tool that’s
>      easy to get wrong—e.g., it removes a gzipped nar for something that
>      had nothing but gzip available, etc.
>
>   2. That code would have to run with privileges that give it access to
>      the signing key on berlin.
>
>   3. Those 6.5 TB are an initial constant factor; growth of the storage
>      requirements going forward probably matters more and
>      <https://issues.guix.gnu.org/53901> will give us more flexibility
>      on that.

While those are valid points, we need to keep in mind that it is
important that we manage to move the store to the new SSD array quite
quickly to start GCing again.

If we cannot manage to remove those gzip nars then, I see only two
alternatives:

* Host the nars on the HDD array only.

* Host the nars elsewhere, on a VPS as you are proposing.

> I like Chris Baines’ idea of decoupling nar distribution from nar
> building.  If we want to keep nars long enough so that ‘time-machine’ is
> usable, then storage requirements will keep growing.
>
> Perhaps that means we can regularly copy nars “elsewhere” for long-term
> storage, using nar-herder, rsync, or whatever.  The machine that stores
> nars long-term has low requirements compared to the build farm because
> we don’t need to trust it for anything other than storage.  If that
> makes things easier (and financially viable), a VPS is good enough.

Sure, the VPS would also allow us to have a less European-centric
hosting. I did not follow closely the development of the
nar-herder. Chris what improvements this tool would bring compared to a
rsync based approach?

Thanks,

Mathieu

Re: Dropping gzip-compressed substitutes

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 1366 bytes --]


Mathieu Othacehe <othacehe@gnu.org> writes:

>> I like Chris Baines’ idea of decoupling nar distribution from nar
>> building.  If we want to keep nars long enough so that ‘time-machine’ is
>> usable, then storage requirements will keep growing.
>>
>> Perhaps that means we can regularly copy nars “elsewhere” for long-term
>> storage, using nar-herder, rsync, or whatever.  The machine that stores
>> nars long-term has low requirements compared to the build farm because
>> we don’t need to trust it for anything other than storage.  If that
>> makes things easier (and financially viable), a VPS is good enough.
>
> Sure, the VPS would also allow us to have a less European-centric
> hosting. I did not follow closely the development of the
> nar-herder. Chris what improvements this tool would bring compared to a
> rsync based approach?

There's some discussion of this in the README [1].

1: https://git.cbaines.net/guix/nar-herder/about/

I think the short answer for the moment though is the nar-herder doesn't
do anything that you couldn't do with rsync.

I jumped straight in with Guile+SQLite rather than using rsync+files
because I think the performance for various operations will scale better
this way, and it'll lead on to more advanced functionality, like doing
GC like operations, metrics and tagging nars.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

Re: Dropping gzip-compressed substitutes

[Maxim Cournoyer]

Hi everyone,

Mathieu Othacehe <othacehe@gnu.org> writes:

> Hey Ludo,
>
>> As discussed on IRC, I’m skeptical about this because:
>>
>>   1. It requires the development and testing of a custom tool that’s
>>      easy to get wrong—e.g., it removes a gzipped nar for something that
>>      had nothing but gzip available, etc.
>>
>>   2. That code would have to run with privileges that give it access to
>>      the signing key on berlin.
>>
>>   3. Those 6.5 TB are an initial constant factor; growth of the storage
>>      requirements going forward probably matters more and
>>      <https://issues.guix.gnu.org/53901> will give us more flexibility
>>      on that.
>
> While those are valid points, we need to keep in mind that it is
> important that we manage to move the store to the new SSD array quite
> quickly to start GCing again.

I turns out that the store contains about 14 TiB of probably mostly
unnecessary things that just weren't garbage collected (due to the
prohibitive cost of doing so on the HDD hard drive).

Currently the new array still has 5 TiB so I think we can try to migrate
today and then work trimming the gzip substitutes to have some extra
head room.

With Ricardo, we've attempted to reboot Berlin onto a freshly 'guix
system init'ed system using the new array last Thursday.  Unfortunately
we hit some issue where mounting the root file system was apparently
occurring before all the 6 drives had the time to be visible.

This week, I'd like to try the following to see if we could get past
this:

1. Do the experiment again, now a 'rootdelay=20' kernel parameter was
added to Berlin's config.  This may well be enough.

2. In case mounting the RAID 10 Btrfs root partition still fails with
missing drive errors, try the following workaround suggested in the
#btrfs channel, which forces a 'btrfs device scan' on each device of the
array, with the following mount option:

"device=/dev/sda3,/dev/sdb3,/dev/sdc3,/dev/sdd3,/dev/sde3,/dev/sdf3"

To make it more convenient to experiment with different values for the
rootdelay or add the device option above, I'm planning to 'guix system init'
with the following patch applied: https://issues.guix.gnu.org/40998,
which allows providing 'rootflags' directly from the kernel command line
(thus by editing the GRUB config at boot).

I'll try to synchronize with Ricardo in the channel and hope they
weren't too frightened by our last experiment to not shy away from
trying again :-).

That's it for the update.  Until this is sorted out, I'd ask to not
reboot the machine, and not even reconfigure it.  I'll give you all an
update as soon as it is sorted out.

Thanks!

Maxim

Re: Dropping gzip-compressed substitutes

[Ricardo Wurmus]

Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:

> This week, I'd like to try the following to see if we could get past
> this:
>
> 1. Do the experiment again, now a 'rootdelay=20' kernel parameter was
> added to Berlin's config.  This may well be enough.
>
> 2. In case mounting the RAID 10 Btrfs root partition still fails with
> missing drive errors, try the following workaround suggested in the
> #btrfs channel, which forces a 'btrfs device scan' on each device of the
> array, with the following mount option:
>
> "device=/dev/sda3,/dev/sdb3,/dev/sdc3,/dev/sdd3,/dev/sde3,/dev/sdf3"
>
> To make it more convenient to experiment with different values for the
> rootdelay or add the device option above, I'm planning to 'guix system init'
> with the following patch applied: https://issues.guix.gnu.org/40998,
> which allows providing 'rootflags' directly from the kernel command line
> (thus by editing the GRUB config at boot).

Good work, Maxim!  Your persistence is admirable.

> I'll try to synchronize with Ricardo in the channel and hope they
> weren't too frightened by our last experiment to not shy away from
> trying again :-).

We can make another attempt this week.  Do we need to sync anything
before trying again?  Also: would be nice if we could do “guix system
init” without copying everything again and again whenever we try again…

My toddler has been pretty sick for the past couple of days, and I
haven’t slept enough to be using my brain without hand-holding :) I’ll
try to be around long enough in the eaerly evenings to see to it that
the server reboots fine, but if things go awfully sideways I just cannot
commit to riding the bike to the data centre.  Let’s hope it will just
work fine with your changes to the initrd.

See you later on IRC — I gotta have a nap :)

-- 
Ricardo

Re: Dropping gzip-compressed substitutes

[Gábor Boskovits]

[-- Attachment #1: Type: text/plain, Size: 2255 bytes --]

Hello,

Ricardo Wurmus <rekado@elephly.net> ezt írta (időpont: 2022. febr. 21., H
12:34):

>
> Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:
>
> > This week, I'd like to try the following to see if we could get past
> > this:
> >
> > 1. Do the experiment again, now a 'rootdelay=20' kernel parameter was
> > added to Berlin's config.  This may well be enough.
> >
> > 2. In case mounting the RAID 10 Btrfs root partition still fails with
> > missing drive errors, try the following workaround suggested in the
> > #btrfs channel, which forces a 'btrfs device scan' on each device of the
> > array, with the following mount option:
> >
> > "device=/dev/sda3,/dev/sdb3,/dev/sdc3,/dev/sdd3,/dev/sde3,/dev/sdf3"
> >
> > To make it more convenient to experiment with different values for the
> > rootdelay or add the device option above, I'm planning to 'guix system
> init'
> > with the following patch applied: https://issues.guix.gnu.org/40998,
> > which allows providing 'rootflags' directly from the kernel command line
> > (thus by editing the GRUB config at boot).
>
> Good work, Maxim!  Your persistence is admirable.
>
> > I'll try to synchronize with Ricardo in the channel and hope they
> > weren't too frightened by our last experiment to not shy away from
> > trying again :-).
>
> We can make another attempt this week.  Do we need to sync anything
> before trying again?  Also: would be nice if we could do “guix system
> init” without copying everything again and again whenever we try again…
>
> My toddler has been pretty sick for the past couple of days, and I
> haven’t slept enough to be using my brain without hand-holding :) I’ll
> try to be around long enough in the eaerly evenings to see to it that
> the server reboots fine, but if things go awfully sideways I just cannot
> commit to riding the bike to the data centre.  Let’s hope it will just
> work fine with your changes to the initrd.
>

Sorry to hear that, hope they get better soon. I there is anything I can
help, besides taking the the bike to the dc which is a bit far from here
feel free to contact me.

Regards, g_bor

>
> See you later on IRC — I gotta have a nap :)
>
> --
> Ricardo
>
>

[-- Attachment #2: Type: text/html, Size: 3174 bytes --]