From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp1 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms11 with LMTPS id KBBjMrb1vF6UJgAA0tVLHw (envelope-from ) for ; Thu, 14 May 2020 07:39:34 +0000 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp1 with LMTPS id 2JijLLb1vF7cHQAAbx9fmQ (envelope-from ) for ; Thu, 14 May 2020 07:39:34 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 2D058940429 for ; Thu, 14 May 2020 07:39:32 +0000 (UTC) Received: from localhost ([::1]:54544 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jZ8Sr-0000UK-5V for larch@yhetil.org; Thu, 14 May 2020 03:39:33 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:46892) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jZ8SM-00007a-4X for guix-patches@gnu.org; Thu, 14 May 2020 03:39:02 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:48571) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1jZ8SL-0007VG-RJ for guix-patches@gnu.org; Thu, 14 May 2020 03:39:01 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1jZ8SL-0002ki-OX for guix-patches@gnu.org; Thu, 14 May 2020 03:39:01 -0400 Subject: bug#33643: [PATCH] gnu-build-system: Enable xz to decompress in parallel. Resent-From: Christopher Baines Original-Sender: "Debbugs-submit" Resent-To: guix-patches@gnu.org Resent-Date: Thu, 14 May 2020 07:39:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: cc-closed 33643 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Efraim Flashner Cc: 33643-done@debbugs.gnu.org Mail-Followup-To: 33643@debbugs.gnu.org, mail@cbaines.net, mail@cbaines.net Received: via spool by 33643-done@debbugs.gnu.org id=D33643.158944188210460 (code D ref 33643); Thu, 14 May 2020 07:39:01 +0000 Received: (at 33643-done) by debbugs.gnu.org; 14 May 2020 07:38:02 +0000 Received: from localhost ([127.0.0.1]:60115 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jZ8RN-0002iY-N5 for submit@debbugs.gnu.org; Thu, 14 May 2020 03:38:02 -0400 Received: from mira.cbaines.net ([212.71.252.8]:52130) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jZ8RM-0002iQ-5A for 33643-done@debbugs.gnu.org; Thu, 14 May 2020 03:38:00 -0400 Received: from localhost (unknown [46.237.173.210]) by mira.cbaines.net (Postfix) with ESMTPSA id 2A65927BBE1; Thu, 14 May 2020 08:37:59 +0100 (BST) Received: from localhost (localhost [local]) by localhost (OpenSMTPD) with ESMTPA id ccc3dc77; Thu, 14 May 2020 07:37:57 +0000 (UTC) References: <20181206075615.4637-1-mail@cbaines.net> <87k11fofxj.fsf@cbaines.net> <20200513190721.GH918@E5400> User-agent: mu4e 1.2.0; emacs 26.3 From: Christopher Baines In-reply-to: <20200513190721.GH918@E5400> Date: Thu, 14 May 2020 08:37:56 +0100 Message-ID: <87ftc3nezv.fsf@cbaines.net> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha512; protocol="application/pgp-signature" X-Spam-Score: 0.0 (/) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-Spam-Score: -1.0 (-) X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+larch=yhetil.org@gnu.org Sender: "Guix-patches" X-Scanner: scn0 X-Spam-Score: -1.61 Authentication-Results: aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of guix-patches-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-patches-bounces@gnu.org X-Scan-Result: default: False [-1.61 / 13.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; GENERIC_REPUTATION(0.00)[-0.53953935406463]; MX_INVALID(1.00)[cached]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:209.51.188.0/24:c]; IP_REPUTATION_HAM(0.00)[asn: 22989(0.05), country: US(-0.00), ip: 209.51.188.17(-0.54)]; DWL_DNSWL_BLOCKED(0.00)[209.51.188.17:from]; RCPT_COUNT_TWO(0.00)[2]; MAILLIST(-0.20)[mailman]; SIGNED_PGP(-2.00)[]; FORGED_RECIPIENTS_MAILLIST(0.00)[]; RCVD_TLS_LAST(0.00)[]; R_DKIM_NA(0.00)[]; ASN(0.00)[asn:22989, ipnet:209.51.188.0/24, country:US]; MID_RHS_MATCH_FROM(0.00)[]; TAGGED_FROM(0.00)[larch=yhetil.org]; ARC_NA(0.00)[]; FROM_NEQ_ENVFROM(0.00)[mail@cbaines.net,guix-patches-bounces@gnu.org]; FROM_HAS_DN(0.00)[]; URIBL_BLOCKED(0.00)[cbaines.net:email,flashner.co.il:email]; MIME_GOOD(-0.20)[multipart/signed,text/plain]; MIME_TRACE(0.00)[0:+,1:+,2:~]; DMARC_NA(0.00)[cbaines.net]; HAS_LIST_UNSUB(-0.01)[]; DNSWL_BLOCKED(0.00)[209.51.188.17:from]; RCVD_COUNT_SEVEN(0.00)[9]; FORGED_SENDER_MAILLIST(0.00)[] X-TUID: RRAcdtqfXEtq --=-=-= Content-Type: text/plain Efraim Flashner writes: > On Wed, May 13, 2020 at 07:20:08PM +0100, Christopher Baines wrote: >> >> Christopher Baines writes: >> >> > It can take a little while to decompress some packages with large xz >> > compressed source tar files. xz includes support for parallelism, so enable >> > this using the parallel job count for the overall derivation. >> > >> > * guix/build/gnu-build-system.scm (unpack): Set XZ_OPT to pass the -T option >> > to xz to enable it to work in parallel if appropriate. >> > --- >> > guix/build/gnu-build-system.scm | 6 +++++- >> > 1 file changed, 5 insertions(+), 1 deletion(-) >> > >> > diff --git a/guix/build/gnu-build-system.scm b/guix/build/gnu-build-system.scm >> > index e5f3197b0..9d11e5b1e 100644 >> > --- a/guix/build/gnu-build-system.scm >> > +++ b/guix/build/gnu-build-system.scm >> > @@ -147,7 +147,7 @@ chance to be set." >> > locale (strerror (system-error-errno args))) >> > #t))) >> > >> > -(define* (unpack #:key source #:allow-other-keys) >> > +(define* (unpack #:key source parallel-build? #:allow-other-keys) >> > "Unpack SOURCE in the working directory, and change directory within the >> > source. When SOURCE is a directory, copy it in a sub-directory of the current >> > working directory." >> > @@ -161,6 +161,10 @@ working directory." >> > (copy-recursively source "." >> > #:keep-mtime? #t)) >> > (begin >> > + (when parallel-build? >> > + (setenv "XZ_OPT" >> > + (format #f "-T~d" (parallel-job-count)))) >> > + >> > (if (string-suffix? ".zip" source) >> > (invoke "unzip" source) >> > (invoke "tar" "xvf" source)) >> >> It's been a long long while, but now that core-updates has recently been >> merged, I'd like to try and take a look at this again. >> >> I think the consensus was that this will only help for xz compressed >> files where they have been compressed in parallel. I think it's still >> worth doing though, as some of the big xz files that need decompressing >> have been compressed in parallel, and this will speed up the builds when >> multiple cores are available. >> >> Thanks, >> >> Chris > > I thought the last time we looked into this we figured out that there > was a mistake in release notes or something and that parallel > decompression isn't actually supported. Hmm, I had a look to see if I could find some examples of where this would apply, but I couldn't find any xz archives that we use in Guix where it's been compressed in a way that allows multithreaded decompression... I'm pretty sure I had some examples before, but maybe somethings changed in the intervening year. Anyway, if I discover this again, I'll actually make a note of where it's applicable. --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQKTBAEBCgB9FiEEPonu50WOcg2XVOCyXiijOwuE9XcFAl689VRfFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNF ODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcACgkQXiijOwuE 9XfLAw/+OaHGFflhFa8gpk0UfmZwBA7ZuZBuDEIycWyMsKnisG3suwOPZkDbTIZQ /gx5BCwZKXAEXkVCenRQURzDpTx6jzUsRGQtUo+7dO9lb1RPu+URaQYZqxtmkix5 hT0k7I1J0cI1EVYoKjf8V0pQoMKZ0fwEU6LlJqVdeVJkWJ2lO/YiY9ehjkaRfC8O twqLcxJt7E93EFzOZUKk/fSFO4U6CO1y5HfyFHOsCJpZ6Wv1KAW06Zjxl4dKXAQL oGpZ/O8KH9958wRZneyXCpaHjRFCNABQIRkDceMiAS70uFJhZ8XVb/1b/O13kEBz TMiVRE+nicpg4UKDKRI/ljfwtwvhAVSBXJ0Q4x8oq3jdvOAa1Ax7hKrqUV2BaRBD hn517EaaZschp8QjsoudgHNzf1V+bwD1eKrDuLmrIoo2ObCCmPCsJoMqdnEXZ16S yAu1cdtIoQs45P+Q0cuVywPfoI+pjTD8jlKBo5VK3HhfYWB20YFupSyQbZCRLEwW AjmfFUknkjLxlRt5h9im11GrRup2XEv0sI0A0BWm/Jo1ebSwxywgVh0a0GegcQhK gQb9z41b3hpP3yCW8QwisRlPR4Iy44sm8iRfPGpVfmgaccJfx9dVi9q+lMekt5cG bIEIDhQQo5ONW+Y38r7BhFK78TimI7yKvMrH6RxobnxZklkVblg= =eiqL -----END PGP SIGNATURE----- --=-=-=--