[bug#31015] [PATCH] packages, scripts, utils: Respect (parallel-job-count) when xz-compressing.

[bug#31015] [PATCH] packages, scripts, utils: Respect (parallel-job-count) when xz-compressing.

[Marius Bakke]

* guix/packages.scm (patch-and-repack): When invoking 'xz', tell it to use the
amount of cores specified by e.g. 'guix build -c'.
* guix/scripts/pack.scm (%compressors, bootstrap-xz): Likewise.  While at it,
use the long form '--threads=' for clarity.
* guix/utils.scm (decompressed-port, compressed-port, compressed-output-port):
Likewise.
---
 guix/packages.scm     |  3 ++-
 guix/scripts/pack.scm |  7 +++++--
 guix/utils.scm        | 18 +++++++++++++++---
 3 files changed, 22 insertions(+), 6 deletions(-)

diff --git a/guix/packages.scm b/guix/packages.scm
index ab4b6278d..cca2c6357 100644
--- a/guix/packages.scm
+++ b/guix/packages.scm
@@ -604,7 +604,8 @@ specifies modules in scope when evaluating SNIPPET."
                      ;; threaded compression (introduced in
                      ;; 5.2.0), but it ignores the extra flag.
                      (string-append "--use-compress-program="
-                                    #+xz "/bin/xz --threads=0")
+                                    #+xz "/bin/xz --threads="
+                                    (number->string (parallel-job-count)))
                      ;; avoid non-determinism in the archive
                      "--mtime=@0"
                      "--owner=root:0"
diff --git a/guix/scripts/pack.scm b/guix/scripts/pack.scm
index 488638adc..818de0f32 100644
--- a/guix/scripts/pack.scm
+++ b/guix/scripts/pack.scm
@@ -65,7 +65,9 @@
         (compressor "lzip"  ".lz"
                     #~(#+(file-append lzip "/bin/lzip") "-9"))
         (compressor "xz"    ".xz"
-                    #~(#+(file-append xz "/bin/xz") "-e -T0"))
+                    #~(#+(file-append xz "/bin/xz")
+                       (string-append "-e --threads="
+                                      (number->string (parallel-job-count)))))
         (compressor "bzip2" ".bz2"
                     #~(#+(file-append bzip2 "/bin/bzip2") "-9"))
         (compressor "none" "" #f)))
@@ -73,7 +75,8 @@
 ;; This one is only for use in this module, so don't put it in %compressors.
 (define bootstrap-xz
   (compressor "bootstrap-xz" ".xz"
-              #~(#+(file-append %bootstrap-coreutils&co "/bin/xz") "-e -T0")))
+              #~(#+(file-append %bootstrap-coreutils&co "/bin/xz")
+                 (string-append "-e --threads=" (number->string (parallel-job-count))))))
 
 (define (lookup-compressor name)
   "Return the compressor object called NAME.  Error out if it could not be
diff --git a/guix/utils.scm b/guix/utils.scm
index 92e45de61..043034e99 100644
--- a/guix/utils.scm
+++ b/guix/utils.scm
@@ -172,7 +172,11 @@ a symbol such as 'xz."
   (match compression
     ((or #f 'none) (values input '()))
     ('bzip2        (filtered-port `(,%bzip2 "-dc") input))
-    ('xz           (filtered-port `(,%xz "-dc" "-T0") input))
+    ('xz           (filtered-port
+                    `(,%xz "-dc" (string-append
+                                  "--threads="
+                                  (number->string (parallel-job-count))))
+                    input))
     ('gzip         (filtered-port `(,%gzip "-dc") input))
     (else          (error "unsupported compression scheme" compression))))
 
@@ -182,7 +186,11 @@ a symbol such as 'xz."
   (match compression
     ((or #f 'none) (values input '()))
     ('bzip2        (filtered-port `(,%bzip2 "-c") input))
-    ('xz           (filtered-port `(,%xz "-c" "-T0") input))
+    ('xz           (filtered-port
+                    `(,%xz "-c" (string-append
+                                 "--threads="
+                                 (number->string (parallel-job-count))))
+                    input))
     ('gzip         (filtered-port `(,%gzip "-c") input))
     (else          (error "unsupported compression scheme" compression))))
 
@@ -239,7 +247,11 @@ program--e.g., '(\"--fast\")."
   (match compression
     ((or #f 'none) (values output '()))
     ('bzip2        (filtered-output-port `(,%bzip2 "-c" ,@options) output))
-    ('xz           (filtered-output-port `(,%xz "-c" "-T0" ,@options) output))
+    ('xz           (filtered-output-port
+                    `(,%xz "-c" (string-append
+                                 "--threads="
+                                 (number->string (parallel-job-count)))
+                           ,@options) output))
     ('gzip         (filtered-output-port `(,%gzip "-c" ,@options) output))
     (else          (error "unsupported compression scheme" compression))))
 
-- 
2.16.3

[bug#31015] [PATCH] packages, scripts, utils: Respect (parallel-job-count) when xz-compressing.

[Ludovic Courtès]

Hello!

Marius Bakke <mbakke@fastmail.com> skribis:

> * guix/packages.scm (patch-and-repack): When invoking 'xz', tell it to use the
> amount of cores specified by e.g. 'guix build -c'.
> * guix/scripts/pack.scm (%compressors, bootstrap-xz): Likewise.  While at it,
> use the long form '--threads=' for clarity.
> * guix/utils.scm (decompressed-port, compressed-port, compressed-output-port):
> Likewise.

Wasn’t there an issue that parallel xz is non-deterministic?  I vaguely
remember something like this when we were producing with glibc tarballs
out of the Git repo.

If so, we should instead use -T1, even if that means getting worse
performance (I hear that lzip “works better” than xz, so we should give
it a try if we want both performance and good compression.)

Besides, the guix/packages.scm part would lead to a full rebuild, so I
think it’s a bit late for ‘core-updates’.

Thanks,
Ludo’.

[bug#31015] [PATCH] packages, scripts, utils: Respect (parallel-job-count) when xz-compressing.

[Ludovic Courtès]

Hello Marius,

ludo@gnu.org (Ludovic Courtès) skribis:

> Marius Bakke <mbakke@fastmail.com> skribis:
>
>> * guix/packages.scm (patch-and-repack): When invoking 'xz', tell it to use the
>> amount of cores specified by e.g. 'guix build -c'.
>> * guix/scripts/pack.scm (%compressors, bootstrap-xz): Likewise.  While at it,
>> use the long form '--threads=' for clarity.
>> * guix/utils.scm (decompressed-port, compressed-port, compressed-output-port):
>> Likewise.
>
> Wasn’t there an issue that parallel xz is non-deterministic?  I vaguely
> remember something like this when we were producing with glibc tarballs
> out of the Git repo.

That was indeed the case, so I’m closing this issue.  Let’s reopen it if
we have more info or new ideas!  :-)

Ludo’.