unofficial mirror of guix-devel@gnu.org 
 help / color / mirror / code / Atom feed
* Dropping gzip-compressed substitutes
@ 2022-02-07 18:44 Maxim Cournoyer
  2022-02-07 20:02 ` Ricardo Wurmus
                   ` (2 more replies)
  0 siblings, 3 replies; 15+ messages in thread
From: Maxim Cournoyer @ 2022-02-07 18:44 UTC (permalink / raw)
  To: guix-devel, guix-sysadmin

Hello Guix!

While migrating our file system to the newly installed SSD-backed
storage on Berlin, I noticed that the guix-publish cached nars use a lot
of space: 6.5 TiB for the gzip compressed nars alone, or about 30% of
the new storage capacity.

The new array capacity was recently added so that running 'guix gc' on
it would be able to complete in a reasonable amount of time (remember
that no substitutes from the Cuirass workers can be retrieved while the
big GC lock is on, which greatly hurts the performance of the CI),
allowing to keep the size of /gnu/store in check.

Its capacity was chosen based on the assumption that /gnu/store
compresses well with zstd, leading to a 1.5-2.0 factor capacity boost.
This assumption doesn't hold in case we are storing a large quantity of
already-compressed data such as the /var/cache/guix/publish compressed
nars.

From a past survey of the use of gzip substitutes [0], the result was a
small percentage of our users were still relying on them (1%).

Here's what I would suggest for a migration path:

1.  Promptly set up both a blog post and a NEWS entry announcing the
support for gzip substitutes is about to be phased out from the build
farm (1 month notice), urging users to upgrade their daemon to a version
>= 1.1.0.

2. Do the following in the maintenance repo, which should stop
advertising/generating new gzip substitutes:

--8<---------------cut here---------------start------------->8---
modified   hydra/modules/sysadmin/services.scm
@@ -580,9 +580,7 @@ to a selected directory.")
                    ;; <https://lists.gnu.org/archive/html/guix-devel/2021-01/msg00097.html>
                    ;; for the compression ratio/decompression speed
                    ;; tradeoffs.
-                   ;; TODO: Eventually, disable gzip, as discussed at
-                   ;; <https://lists.gnu.org/archive/html/guix-devel/2021-03/msg00333.html>.
-                   (compression '(("gzip" 9) ("lzip" 9) ("zstd" 19)))
+                   (compression '(("lzip" 9) ("zstd" 19)))
                    (cache-bypass-threshold cache-bypass-threshold)
                    (workers publish-workers)))
--8<---------------cut here---------------end--------------->8---
 
3. Come up with a Guile script that is able to

  a) Strip gzip-related metadata from the .narinfo guix-publish metadata
  files
  b) recompute and update their 'Signature' field.

4. Finally, 'rm -r /var/guix/publish/gzip' and free about 6.5 TiB of data.

I'd like to move promptly with this as I'm expecting it may be necessary
to fully migrate the current file system to the new array (and reap the
benefits).

Thank you,

Maxim

[0]  https://lists.gnu.org/archive/html/guix-devel/2021-03/msg00333.html


^ permalink raw reply	[flat|nested] 15+ messages in thread

end of thread, other threads:[~2022-02-21 20:05 UTC | newest]

Thread overview: 15+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2022-02-07 18:44 Dropping gzip-compressed substitutes Maxim Cournoyer
2022-02-07 20:02 ` Ricardo Wurmus
2022-02-08 13:34 ` Mathieu Othacehe
2022-02-08 14:13   ` Maxim Cournoyer
2022-02-08 16:53   ` Ricardo Wurmus
2022-02-09  2:35     ` Maxim Cournoyer
2022-02-09  5:06     ` Maxim Cournoyer
2022-02-09 10:29   ` Efraim Flashner
2022-02-14 17:50 ` Ludovic Courtès
2022-02-14 20:04   ` Attila Lendvai
2022-02-15 12:20   ` Mathieu Othacehe
2022-02-15 18:29     ` Christopher Baines
2022-02-21  5:13     ` Maxim Cournoyer
2022-02-21 11:26       ` Ricardo Wurmus
2022-02-21 20:04         ` Gábor Boskovits

Code repositories for project(s) associated with this public inbox

	https://git.savannah.gnu.org/cgit/guix.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).