From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp2 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms11 with LMTPS id 0Jv+F3o7vF4kCQAA0tVLHw (envelope-from ) for ; Wed, 13 May 2020 18:24:58 +0000 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp2 with LMTPS id 2MCMDIk7vF4oFwAAB5/wlQ (envelope-from ) for ; Wed, 13 May 2020 18:25:13 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 9E05F940B0C for ; Wed, 13 May 2020 18:25:10 +0000 (UTC) Received: from localhost ([::1]:46926 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jYw47-0003Pc-Jg for larch@yhetil.org; Wed, 13 May 2020 14:25:11 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:37920) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jYw07-0004g7-0T for guix-patches@gnu.org; Wed, 13 May 2020 14:21:03 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:47969) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1jYw06-000379-L7 for guix-patches@gnu.org; Wed, 13 May 2020 14:21:02 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1jYw06-000439-Ek for guix-patches@gnu.org; Wed, 13 May 2020 14:21:02 -0400 X-Loop: help-debbugs@gnu.org Subject: [bug#33643] [PATCH] gnu-build-system: Enable xz to decompress in parallel. Resent-From: Christopher Baines Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Wed, 13 May 2020 18:21:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 33643 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 33643@debbugs.gnu.org Cc: Efraim Flashner , Leo Famulari Received: via spool by 33643-submit@debbugs.gnu.org id=B33643.158939401815492 (code B ref 33643); Wed, 13 May 2020 18:21:02 +0000 Received: (at 33643) by debbugs.gnu.org; 13 May 2020 18:20:18 +0000 Received: from localhost ([127.0.0.1]:59515 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jYvzM-00041n-Te for submit@debbugs.gnu.org; Wed, 13 May 2020 14:20:18 -0400 Received: from mira.cbaines.net ([212.71.252.8]:57690) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jYvzK-00041d-TD for 33643@debbugs.gnu.org; Wed, 13 May 2020 14:20:15 -0400 Received: from localhost (unknown [46.237.173.210]) by mira.cbaines.net (Postfix) with ESMTPSA id B87A827BBE1; Wed, 13 May 2020 19:20:13 +0100 (BST) Received: from localhost (localhost [local]) by localhost (OpenSMTPD) with ESMTPA id 2156064e; Wed, 13 May 2020 18:20:11 +0000 (UTC) References: <20181206075615.4637-1-mail@cbaines.net> User-agent: mu4e 1.2.0; emacs 26.3 From: Christopher Baines In-reply-to: <20181206075615.4637-1-mail@cbaines.net> Date: Wed, 13 May 2020 19:20:08 +0100 Message-ID: <87k11fofxj.fsf@cbaines.net> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha512; protocol="application/pgp-signature" X-Spam-Score: 0.0 (/) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-Spam-Score: -1.0 (-) X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+larch=yhetil.org@gnu.org Sender: "Guix-patches" X-Scanner: scn0 X-Spam-Score: -3.11 Authentication-Results: aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of guix-patches-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-patches-bounces@gnu.org X-Scan-Result: default: False [-3.11 / 13.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; GENERIC_REPUTATION(0.00)[-0.5400046361328]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:209.51.188.0/24:c]; IP_REPUTATION_HAM(0.00)[asn: 22989(0.05), country: US(-0.00), ip: 209.51.188.17(-0.54)]; DWL_DNSWL_FAIL(0.00)[209.51.188.17:server fail]; MX_GOOD(-0.50)[cached: eggs.gnu.org]; MAILLIST(-0.20)[mailman]; SIGNED_PGP(-2.00)[]; FORGED_RECIPIENTS_MAILLIST(0.00)[]; RCVD_IN_DNSWL_FAIL(0.00)[209.51.188.17:server fail]; RCVD_TLS_LAST(0.00)[]; R_DKIM_NA(0.00)[]; ASN(0.00)[asn:22989, ipnet:209.51.188.0/24, country:US]; MID_RHS_MATCH_FROM(0.00)[]; TAGGED_FROM(0.00)[larch=yhetil.org]; ARC_NA(0.00)[]; FROM_NEQ_ENVFROM(0.00)[mail@cbaines.net,guix-patches-bounces@gnu.org]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; URIBL_BLOCKED(0.00)[cbaines.net:email]; MIME_GOOD(-0.20)[multipart/signed,text/plain]; MIME_TRACE(0.00)[0:+,1:+,2:~]; DMARC_NA(0.00)[cbaines.net]; HAS_LIST_UNSUB(-0.01)[]; RWL_MAILSPIKE_POSSIBLE(0.00)[209.51.188.17:from]; RCVD_COUNT_SEVEN(0.00)[9]; FORGED_SENDER_MAILLIST(0.00)[] X-TUID: 5Ffd0UkTdeGv --=-=-= Content-Type: text/plain Christopher Baines writes: > It can take a little while to decompress some packages with large xz > compressed source tar files. xz includes support for parallelism, so enable > this using the parallel job count for the overall derivation. > > * guix/build/gnu-build-system.scm (unpack): Set XZ_OPT to pass the -T option > to xz to enable it to work in parallel if appropriate. > --- > guix/build/gnu-build-system.scm | 6 +++++- > 1 file changed, 5 insertions(+), 1 deletion(-) > > diff --git a/guix/build/gnu-build-system.scm b/guix/build/gnu-build-system.scm > index e5f3197b0..9d11e5b1e 100644 > --- a/guix/build/gnu-build-system.scm > +++ b/guix/build/gnu-build-system.scm > @@ -147,7 +147,7 @@ chance to be set." > locale (strerror (system-error-errno args))) > #t))) > > -(define* (unpack #:key source #:allow-other-keys) > +(define* (unpack #:key source parallel-build? #:allow-other-keys) > "Unpack SOURCE in the working directory, and change directory within the > source. When SOURCE is a directory, copy it in a sub-directory of the current > working directory." > @@ -161,6 +161,10 @@ working directory." > (copy-recursively source "." > #:keep-mtime? #t)) > (begin > + (when parallel-build? > + (setenv "XZ_OPT" > + (format #f "-T~d" (parallel-job-count)))) > + > (if (string-suffix? ".zip" source) > (invoke "unzip" source) > (invoke "tar" "xvf" source)) It's been a long long while, but now that core-updates has recently been merged, I'd like to try and take a look at this again. I think the consensus was that this will only help for xz compressed files where they have been compressed in parallel. I think it's still worth doing though, as some of the big xz files that need decompressing have been compressed in parallel, and this will speed up the builds when multiple cores are available. Thanks, Chris --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQKTBAEBCgB9FiEEPonu50WOcg2XVOCyXiijOwuE9XcFAl68OlhfFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNF ODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcACgkQXiijOwuE 9XeIIg//Ttm908GW5504QVqslK2vaUE2acDsgqHjmlsv7ykIKFThXtUp54VKQ5JT wOZl8OV+0HCpQDxF+wTONjGwDjE9eAyFMuyFbF6JKcVu5GNSgd+SbZdwcGBQAKM3 JYY990M6JZiiPwuuXsonLYYv46I8Lb46Jp8Oc7dogOsXXVrF3UZFIL+9pBFQlBco D4gebtkMMTLvTXsm5/pzL+rpRWHudvuBKf9ogBsd+hsKOCFIOLPTF1A0WE3Twrnm VKZ+j7SCqGmKfVkJL+a3Df5ln+fKHX7Yav8XGiYkfSI7LXPg3sJhq1/CDMBAy6O5 7KAFptf/msw7atgc4hazB2Aah7E2o230Gdu86HDy9Dbom0sd6sq6s7F1ICGvcv9C xt/COIetlMXDhnsbvpFL/lmh3kxDPk78sFaHkUfd4fbT4+9nHIXVL03AtWqFOYgF OiChboA8Eh8+vnaMuXwbE9o1O84+E9/o4RuaH3OS6RlDDBp62FC/4bQBhvLagSEH BD1erdFPVzJKV0amSfHExGTrQ/PL9Cov9CgEmWrKE0pPCcZWXbLc0pa7SkXR+jyE xqe1wDQDK4xeOG16sEMurZLgNbSD1ZfRPOoI/jIRbKkrmJtKZqAbAEFi8y6R+Xow QHbq776aESqqV9/x10QHE0mb4sJaoXFTfMc4zrJw9HLwQvUUSMo= =+wNy -----END PGP SIGNATURE----- --=-=-=--