From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp0 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms11 with LMTPS id MGIKLMNaaV9gLQAA0tVLHw (envelope-from ) for ; Tue, 22 Sep 2020 02:00:35 +0000 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp0 with LMTPS id wJrrJ8NaaV8+BQAA1q6Kng (envelope-from ) for ; Tue, 22 Sep 2020 02:00:35 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 5D31794042B for ; Tue, 22 Sep 2020 02:00:34 +0000 (UTC) Received: from localhost ([::1]:42120 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kKXbc-0005tH-PB for larch@yhetil.org; Mon, 21 Sep 2020 22:00:32 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:39548) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kKXb9-0005t1-FF for guix-patches@gnu.org; Mon, 21 Sep 2020 22:00:04 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:45243) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1kKXb8-0001hK-D5 for guix-patches@gnu.org; Mon, 21 Sep 2020 22:00:03 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1kKXb8-0000yu-C1 for guix-patches@gnu.org; Mon, 21 Sep 2020 22:00:02 -0400 X-Loop: help-debbugs@gnu.org Subject: [bug#43516] [PATCH core-updates v2] packages: Enable multi-threaded xz compression when repacking source. Resent-From: Maxim Cournoyer Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Tue, 22 Sep 2020 02:00:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 43516 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 43516@debbugs.gnu.org Cc: Maxim Cournoyer , Leo Famulari Received: via spool by 43516-submit@debbugs.gnu.org id=B43516.16007399813708 (code B ref 43516); Tue, 22 Sep 2020 02:00:02 +0000 Received: (at 43516) by debbugs.gnu.org; 22 Sep 2020 01:59:41 +0000 Received: from localhost ([127.0.0.1]:56788 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kKXam-0000xk-MJ for submit@debbugs.gnu.org; Mon, 21 Sep 2020 21:59:41 -0400 Received: from mail-qk1-f193.google.com ([209.85.222.193]:41514) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kKXak-0000xX-PE for 43516@debbugs.gnu.org; Mon, 21 Sep 2020 21:59:39 -0400 Received: by mail-qk1-f193.google.com with SMTP id g72so17513674qke.8 for <43516@debbugs.gnu.org>; Mon, 21 Sep 2020 18:59:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=B0qDLOOTwEvsQmrN8iF0azTY+b7L+6nFdwpdi8n1TP8=; b=iZ9hSR9a2y6vYMvAfYrNBE66AyOtO/2A8BsfgeA73ITgs/3sUhPkhcIVd7/SxdpRUX tS/hZARZYPgt06mhGweltepJTHdz3YiPzY4TYzn+zzxyTpwlvbCrRAijqHnql7TjHGGS 3eoM4ryZUX0OZwkFitKQuir10TIYS95EW1PaVybFPXB7bQrW9l/ZUvSo912dG6t1saUA Lsi9WreApF+QuE0cZx0t1Qvskk/AoslpkjKJH3SEgBkxuzEaq34Ru6+4IB7p28HJyTeI MDC460QNN//EVsBdYEU+rukbi8t/Jys3OzG80H0KC2lUQSIOdM1fdS5quoSwgXAgYHqt JrYQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=B0qDLOOTwEvsQmrN8iF0azTY+b7L+6nFdwpdi8n1TP8=; b=A7pNUFApYRrx7/SsEQl1C1TLdEnyhB0TQJz/+vn/ubZ9V3Drs/Hx8heSJ5SQVqdQBd FjH5e+6/bDVkMjr2qapyrwPJQEIhqCGZBcLW0ffjVeDqDjkFkfPsd0FVSWSdLd/6sMWx auBs1RoTh7F/fdGsAqEbtE6+dV7Oe1ZmETXNOYf5tBaHMQJWdBbscRcRpz69Eq0gwg1E jYgTdg5LeM8isBJaWCpKSUOQZKgSEF12YI0yYnn0dKPxzSMiEzMmme+u1vSfSouk8w9Z GqHWjpt3Im9hTsCgkiEjTrlNI4legYw/800EJdte2HmvvhcXWxi6ivCtK/Jw0VToPzQ9 4InQ== X-Gm-Message-State: AOAM531kxwiXdvi0EftqJgPLYNLf3zcYBFZSPXPaCD8O7boB4CDpSva1 BesauFISNYITFVjhS+GEe0OVYxSbUmI= X-Google-Smtp-Source: ABdhPJz/5R5tqlRiqpIHNtkkgxKwGx2fUxsjkMQoc5GL89ILS0f6P8FIJRAnCG6nOpM/JpsyzhtKjQ== X-Received: by 2002:a05:620a:13f9:: with SMTP id h25mr2560546qkl.283.1600739972901; Mon, 21 Sep 2020 18:59:32 -0700 (PDT) Received: from localhost.localdomain (dsl-10-131-6.b2b2c.ca. [72.10.131.6]) by smtp.gmail.com with ESMTPSA id x26sm11113945qtr.78.2020.09.21.18.59.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 21 Sep 2020 18:59:32 -0700 (PDT) From: Maxim Cournoyer Date: Mon, 21 Sep 2020 22:00:03 -0400 Message-Id: <20200922020003.6954-2-maxim.cournoyer@gmail.com> X-Mailer: git-send-email 2.28.0 In-Reply-To: <20200922020003.6954-1-maxim.cournoyer@gmail.com> References: <20200919193805.GA31344@jasmine.lan> <20200922020003.6954-1-maxim.cournoyer@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=yes Content-Transfer-Encoding: 8bit X-Spam-Score: 0.0 (/) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-Spam-Score: -1.0 (-) X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+larch=yhetil.org@gnu.org Sender: "Guix-patches" X-Scanner: scn0 Authentication-Results: aspmx1.migadu.com; dkim=fail (body hash did not verify) header.d=gmail.com header.s=20161025 header.b=iZ9hSR9a; dmarc=fail reason="SPF not aligned (relaxed)" header.from=gmail.com (policy=none); spf=pass (aspmx1.migadu.com: domain of guix-patches-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-patches-bounces@gnu.org X-Spam-Score: 0.09 X-TUID: N4sCVbk7zjJb The xz compression is slow; using multiple threads/cores yields a linear performance improvement. * guix/build/utils.scm (%xz-parallel-args): New procedure. * guix/packages.scm (patch-and-repack): Specify the required above xz arguments by setting the XZ_DEFAULTS environment variable. * guix/scripts/pack.scm (%compressors, bootstrap-xz): Modify the commands Gexps so they do not need to be quoted. This allows lazily evaluating the arguments on the builder's side. Specify the required xz arguments. (self-contained-tarball): Do not quote the compressor command value. (docker-image): Likewise. * guix/utils.scm (decompressed-port, compressed-port) (compressed-output-port): Specify the required above xz arguments. --- guix/build/utils.scm | 15 ++++++++++++++- guix/packages.scm | 3 +++ guix/scripts/pack.scm | 27 ++++++++++++++++++--------- guix/utils.scm | 10 ++++++---- 4 files changed, 41 insertions(+), 14 deletions(-) diff --git a/guix/build/utils.scm b/guix/build/utils.scm index e884c26a22..ff4241d088 100644 --- a/guix/build/utils.scm +++ b/guix/build/utils.scm @@ -112,7 +112,9 @@ make-desktop-entry-file - locale-category->string)) + locale-category->string + + %xz-parallel-args)) ;;; @@ -1479,6 +1481,17 @@ returned." LC_NAME LC_NUMERIC LC_PAPER LC_TELEPHONE LC_TIME))) + +;;; +;;; Others. +;;; + +(define (%xz-parallel-args) + "The xz arguments required to enable bit-reproducible, multi-threaded +compression." + (list "--memlimit=50%" + (format #f "--threads=~a" (max 2 (parallel-job-count))))) + ;;; Local Variables: ;;; eval: (put 'call-with-output-file/atomic 'scheme-indent-function 1) ;;; eval: (put 'call-with-ascii-input-file 'scheme-indent-function 1) diff --git a/guix/packages.scm b/guix/packages.scm index 6598bd3149..865cb81929 100644 --- a/guix/packages.scm +++ b/guix/packages.scm @@ -5,6 +5,7 @@ ;;; Copyright © 2016 Alex Kost ;;; Copyright © 2017, 2019, 2020 Efraim Flashner ;;; Copyright © 2019 Marius Bakke +;;; Copyright © 2020 Maxim Cournoyer ;;; ;;; This file is part of GNU Guix. ;;; @@ -693,6 +694,8 @@ specifies modules in scope when evaluating SNIPPET." (setenv "PATH" (string-append #+xz "/bin" ":" #+decomp "/bin")) + (setenv "XZ_DEFAULTS" (string-join (%xz-parallel-args))) + ;; SOURCE may be either a directory or a tarball. (if (file-is-directory? #+source) (let* ((store (%store-directory)) diff --git a/guix/scripts/pack.scm b/guix/scripts/pack.scm index 379e6a3ac6..a0112162e3 100644 --- a/guix/scripts/pack.scm +++ b/guix/scripts/pack.scm @@ -5,6 +5,7 @@ ;;; Copyright © 2018 Chris Marusich ;;; Copyright © 2018 Efraim Flashner ;;; Copyright © 2020 Tobias Geerinckx-Rice +;;; Copyright © 2020 Maxim Cournoyer ;;; ;;; This file is part of GNU Guix. ;;; @@ -25,6 +26,7 @@ #:use-module (guix scripts) #:use-module (guix ui) #:use-module (guix gexp) + #:use-module ((guix build utils) #:select (%xz-parallel-args)) #:use-module (guix utils) #:use-module (guix store) #:use-module ((guix status) #:select (with-status-verbosity)) @@ -70,29 +72,34 @@ compressor? (name compressor-name) ;string (e.g., "gzip") (extension compressor-extension) ;string (e.g., ".lz") - (command compressor-command)) ;gexp (e.g., #~("/gnu/store/…/gzip" "-9n")) + (command compressor-command)) ;gexp (e.g., #~(list "/gnu/store/…/gzip" + ; "-9n" )) (define %compressors ;; Available compression tools. (list (compressor "gzip" ".gz" - #~(#+(file-append gzip "/bin/gzip") "-9n")) + #~(list #+(file-append gzip "/bin/gzip") "-9n")) (compressor "lzip" ".lz" - #~(#+(file-append lzip "/bin/lzip") "-9")) + #~(list #+(file-append lzip "/bin/lzip") "-9")) (compressor "xz" ".xz" - #~(#+(file-append xz "/bin/xz") "-e")) + #~(append (list #+(file-append xz "/bin/xz") + "-e") + (%xz-parallel-args))) (compressor "bzip2" ".bz2" - #~(#+(file-append bzip2 "/bin/bzip2") "-9")) + #~(list #+(file-append bzip2 "/bin/bzip2") "-9")) (compressor "zstd" ".zst" ;; The default level 3 compresses better than gzip in a ;; fraction of the time, while the highest level 19 ;; (de)compresses more slowly and worse than xz. - #~(#+(file-append zstd "/bin/zstd") "-3")) + #~(list #+(file-append zstd "/bin/zstd") "-3")) (compressor "none" "" #f))) ;; This one is only for use in this module, so don't put it in %compressors. (define bootstrap-xz (compressor "bootstrap-xz" ".xz" - #~(#+(file-append %bootstrap-coreutils&co "/bin/xz") "-e"))) + #~(append (list #+(file-append %bootstrap-coreutils&co "/bin/xz") + "-e") + (%xz-parallel-args)))) (define (lookup-compressor name) "Return the compressor object called NAME. Error out if it could not be @@ -269,7 +276,7 @@ added to the pack." #+@(if (compressor-command compressor) #~("-I" (string-join - '#+(compressor-command compressor))) + #+(compressor-command compressor))) #~()) "--format=gnu" @@ -541,11 +548,13 @@ the image." ,@(source-module-closure `((guix docker) (guix build store-copy) + (guix build utils) ;for %xz-parallel-args (guix profiles) (guix search-paths)) #:select? not-config?)) #~(begin (use-modules (guix docker) (guix build store-copy) + (guix build utils) (guix profiles) (guix search-paths) (srfi srfi-1) (srfi srfi-19) (ice-9 match)) @@ -602,7 +611,7 @@ the image." #~(list (string-append #$profile "/" #$entry-point))) #:extra-files directives - #:compressor '#+(compressor-command compressor) + #:compressor #+(compressor-command compressor) #:creation-time (make-time time-utc 0 1)))))) (gexp->derivation (string-append name ".tar" diff --git a/guix/utils.scm b/guix/utils.scm index 7cc321205e..ba896623f4 100644 --- a/guix/utils.scm +++ b/guix/utils.scm @@ -8,6 +8,7 @@ ;;; Copyright © 2017 Mathieu Othacehe ;;; Copyright © 2018, 2020 Marius Bakke ;;; Copyright © 2020 Efraim Flashner +;;; Copyright © 2020 Maxim Cournoyer ;;; ;;; This file is part of GNU Guix. ;;; @@ -37,7 +38,7 @@ #:use-module (guix memoization) #:use-module ((guix build utils) #:select (dump-port mkdir-p delete-file-recursively - call-with-temporary-output-file)) + call-with-temporary-output-file %xz-parallel-args)) #:use-module ((guix build syscalls) #:select (mkdtemp! fdatasync)) #:use-module (guix diagnostics) ;, &error-location, etc. #:use-module (ice-9 format) @@ -220,7 +221,7 @@ a symbol such as 'xz." (match compression ((or #f 'none) (values input '())) ('bzip2 (filtered-port `(,%bzip2 "-dc") input)) - ('xz (filtered-port `(,%xz "-dc") input)) + ('xz (filtered-port `(,%xz "-dc" ,@(%xz-parallel-args)) input)) ('gzip (filtered-port `(,%gzip "-dc") input)) ('lzip (values (lzip-port 'make-lzip-input-port input) '())) @@ -232,7 +233,7 @@ a symbol such as 'xz." (match compression ((or #f 'none) (values input '())) ('bzip2 (filtered-port `(,%bzip2 "-c") input)) - ('xz (filtered-port `(,%xz "-c") input)) + ('xz (filtered-port `(,%xz "-c" ,@(%xz-parallel-args)) input)) ('gzip (filtered-port `(,%gzip "-c") input)) ('lzip (values (lzip-port 'make-lzip-input-port/compressed input) '())) @@ -291,7 +292,8 @@ program--e.g., '(\"--fast\")." (match compression ((or #f 'none) (values output '())) ('bzip2 (filtered-output-port `(,%bzip2 "-c" ,@options) output)) - ('xz (filtered-output-port `(,%xz "-c" ,@options) output)) + ('xz (filtered-output-port `(,%xz "-c" ,@(%xz-parallel-args) + ,@options) output)) ('gzip (filtered-output-port `(,%gzip "-c" ,@options) output)) ('lzip (values (lzip-port 'make-lzip-output-port output) '())) -- 2.28.0