From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp1 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms11 with LMTPS id Zo3RGcVFvF49IAAA0tVLHw (envelope-from ) for ; Wed, 13 May 2020 19:08:53 +0000 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp1 with LMTPS id GNBZDtRFvF4eIQAAbx9fmQ (envelope-from ) for ; Wed, 13 May 2020 19:09:08 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 7A62D940B0C for ; Wed, 13 May 2020 19:09:05 +0000 (UTC) Received: from localhost ([::1]:42810 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jYwkc-0007Wt-FY for larch@yhetil.org; Wed, 13 May 2020 15:09:06 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:43670) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jYwkY-0007Vl-VA for guix-patches@gnu.org; Wed, 13 May 2020 15:09:02 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:48018) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1jYwkY-0004ts-Lg for guix-patches@gnu.org; Wed, 13 May 2020 15:09:02 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1jYwkY-0005Eh-FI for guix-patches@gnu.org; Wed, 13 May 2020 15:09:02 -0400 X-Loop: help-debbugs@gnu.org Subject: [bug#33643] [PATCH] gnu-build-system: Enable xz to decompress in parallel. Resent-From: Efraim Flashner Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Wed, 13 May 2020 19:09:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 33643 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Christopher Baines Cc: 33643@debbugs.gnu.org, Leo Famulari Received: via spool by 33643-submit@debbugs.gnu.org id=B33643.158939689220068 (code B ref 33643); Wed, 13 May 2020 19:09:02 +0000 Received: (at 33643) by debbugs.gnu.org; 13 May 2020 19:08:12 +0000 Received: from localhost ([127.0.0.1]:59564 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jYwjj-0005Dc-Sf for submit@debbugs.gnu.org; Wed, 13 May 2020 15:08:12 -0400 Received: from flashner.co.il ([178.62.234.194]:51382) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jYwji-0005DN-Ki for 33643@debbugs.gnu.org; Wed, 13 May 2020 15:08:11 -0400 Received: from localhost (unknown [188.120.128.132]) by flashner.co.il (Postfix) with ESMTPSA id AD11940032; Wed, 13 May 2020 19:08:04 +0000 (UTC) Date: Wed, 13 May 2020 22:07:21 +0300 From: Efraim Flashner Message-ID: <20200513190721.GH918@E5400> References: <20181206075615.4637-1-mail@cbaines.net> <87k11fofxj.fsf@cbaines.net> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="tSiBuZsJmMXpnp7T" Content-Disposition: inline In-Reply-To: <87k11fofxj.fsf@cbaines.net> X-PGP-Key-ID: 0x41AAE7DCCA3D8351 X-PGP-Key: https://flashner.co.il/~efraim/efraim_flashner.asc X-PGP-Fingerprint: A28B F40C 3E55 1372 662D 14F7 41AA E7DC CA3D 8351 X-Spam-Score: -0.0 (/) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-Spam-Score: -1.0 (-) X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+larch=yhetil.org@gnu.org Sender: "Guix-patches" X-Scanner: scn0 X-Spam-Score: 0.69 Authentication-Results: aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of guix-patches-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-patches-bounces@gnu.org X-Scan-Result: default: False [0.69 / 13.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; GENERIC_REPUTATION(0.00)[-0.5400255730261]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(0.00)[+ip4:209.51.188.0/24:c]; IP_REPUTATION_HAM(0.00)[asn: 22989(0.05), country: US(-0.00), ip: 209.51.188.17(-0.54)]; DWL_DNSWL_FAIL(0.00)[209.51.188.17:server fail]; MX_GOOD(-0.50)[cached: eggs.gnu.org]; MAILLIST(-0.20)[mailman]; SIGNED_PGP(-2.00)[]; FORGED_RECIPIENTS_MAILLIST(0.00)[]; RCVD_IN_DNSWL_FAIL(0.00)[209.51.188.17:server fail]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; ASN(0.00)[asn:22989, ipnet:209.51.188.0/24, country:US]; TAGGED_FROM(0.00)[larch=yhetil.org]; FROM_NEQ_ENVFROM(0.00)[efraim@flashner.co.il,guix-patches-bounces@gnu.org]; ARC_NA(0.00)[]; RECEIVED_SPAMHAUS_XBL(3.00)[188.120.128.132:received]; RECEIVED_SPAMHAUS_PBL(0.00)[188.120.128.132:received]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; URIBL_BLOCKED(0.00)[cbaines.net:email,flashner.co.il:email]; MIME_GOOD(-0.20)[multipart/signed,text/plain]; RCVD_TLS_LAST(0.00)[]; DMARC_NA(0.00)[flashner.co.il]; HAS_LIST_UNSUB(-0.01)[]; BAD_REP_POLICIES(0.10)[]; RWL_MAILSPIKE_POSSIBLE(0.00)[209.51.188.17:from]; MID_RHS_NOT_FQDN(0.50)[]; RCVD_COUNT_SEVEN(0.00)[8]; FORGED_SENDER_MAILLIST(0.00)[] X-TUID: wkO8TbsblgR2 --tSiBuZsJmMXpnp7T Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Wed, May 13, 2020 at 07:20:08PM +0100, Christopher Baines wrote: >=20 > Christopher Baines writes: >=20 > > It can take a little while to decompress some packages with large xz > > compressed source tar files. xz includes support for parallelism, so en= able > > this using the parallel job count for the overall derivation. > > > > * guix/build/gnu-build-system.scm (unpack): Set XZ_OPT to pass the -T o= ption > > to xz to enable it to work in parallel if appropriate. > > --- > > guix/build/gnu-build-system.scm | 6 +++++- > > 1 file changed, 5 insertions(+), 1 deletion(-) > > > > diff --git a/guix/build/gnu-build-system.scm b/guix/build/gnu-build-sys= tem.scm > > index e5f3197b0..9d11e5b1e 100644 > > --- a/guix/build/gnu-build-system.scm > > +++ b/guix/build/gnu-build-system.scm > > @@ -147,7 +147,7 @@ chance to be set." > > locale (strerror (system-error-errno args))) > > #t))) > > > > -(define* (unpack #:key source #:allow-other-keys) > > +(define* (unpack #:key source parallel-build? #:allow-other-keys) > > "Unpack SOURCE in the working directory, and change directory within= the > > source. When SOURCE is a directory, copy it in a sub-directory of the= current > > working directory." > > @@ -161,6 +161,10 @@ working directory." > > (copy-recursively source "." > > #:keep-mtime? #t)) > > (begin > > + (when parallel-build? > > + (setenv "XZ_OPT" > > + (format #f "-T~d" (parallel-job-count)))) > > + > > (if (string-suffix? ".zip" source) > > (invoke "unzip" source) > > (invoke "tar" "xvf" source)) >=20 > It's been a long long while, but now that core-updates has recently been > merged, I'd like to try and take a look at this again. >=20 > I think the consensus was that this will only help for xz compressed > files where they have been compressed in parallel. I think it's still > worth doing though, as some of the big xz files that need decompressing > have been compressed in parallel, and this will speed up the builds when > multiple cores are available. >=20 > Thanks, >=20 > Chris I thought the last time we looked into this we figured out that there was a mistake in release notes or something and that parallel decompression isn't actually supported. --=20 Efraim Flashner =D7=90=D7=A4=D7=A8=D7=99=D7=9D = =D7=A4=D7=9C=D7=A9=D7=A0=D7=A8 GPG key =3D A28B F40C 3E55 1372 662D 14F7 41AA E7DC CA3D 8351 Confidentiality cannot be guaranteed on emails sent or received unencrypted --tSiBuZsJmMXpnp7T Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAABCgAdFiEEoov0DD5VE3JmLRT3Qarn3Mo9g1EFAl68RWYACgkQQarn3Mo9 g1Gl6RAAoR1+UxXRaKzafGqbc6dT/zc1zHYdanoa3NWt5EEGiAiKBaEnU1RW1K16 4xYJYfzm4w+FChnp1+Buj6n85cdiGpZLst6mY4KV8qPgYJoZJLi7QRN+06AZ92Lo mgD0479QJhRe0iEwtRXk+pjhEt13xOaSv2TTY5L8cRKHT16GVQhzTmD+kgLBCEiS k3qxjP9hOfGFl5myNReHeFcCVb8jJpDl86MmNqvJTuja7ylEW0g92ujF9JfMs2af m+h7pEtB4GGeDSfAoGIUhLrr8hN6VmiRLcbladkabeZ3ySFdQ669qkxcefGQ11HR C5eu8Ac8b0gJ7CMjFzrXQeW76GvDrq9u+RhyhlJ3EQqa8rAkvOodUXtuh4WhFbfW CZN/qWtO4Vs8BQ4svuJ3oWLjqmu8HhunU7JYCyI09uVKL45pHjz14WBLOmUGvplr oOIvv6z4k/C3HFihrC6zt8DcjuXXpfr51VC3Z7PcleFx6JmduhSRAOfTsat6SP61 +vGtSl75m8zCOxpwgj4zmk0w9efiPaxFgTDHLtz7ZnZbNeReUvfiFcTJRNg8UMQC tH4Uj90eduNDQ50VDJrv6b5Fn+Wmu1tg1MwzcPPuyMVCO4n/0C+cO3qya7FpLZ1m orTniX4rFGc4AMotCPQNWLgjJzgpjMS3UjgWnnDZg11TyNm832A= =XTqp -----END PGP SIGNATURE----- --tSiBuZsJmMXpnp7T--