From: Christopher Baines <mail@cbaines.net>
To: Efraim Flashner <efraim@flashner.co.il>
Cc: 33643-done@debbugs.gnu.org
Subject: bug#33643: [PATCH] gnu-build-system: Enable xz to decompress in parallel.
Date: Thu, 14 May 2020 08:37:56 +0100 [thread overview]
Message-ID: <87ftc3nezv.fsf@cbaines.net> (raw)
In-Reply-To: <20200513190721.GH918@E5400>
[-- Attachment #1: Type: text/plain, Size: 2839 bytes --]
Efraim Flashner <efraim@flashner.co.il> writes:
> On Wed, May 13, 2020 at 07:20:08PM +0100, Christopher Baines wrote:
>>
>> Christopher Baines <mail@cbaines.net> writes:
>>
>> > It can take a little while to decompress some packages with large xz
>> > compressed source tar files. xz includes support for parallelism, so enable
>> > this using the parallel job count for the overall derivation.
>> >
>> > * guix/build/gnu-build-system.scm (unpack): Set XZ_OPT to pass the -T option
>> > to xz to enable it to work in parallel if appropriate.
>> > ---
>> > guix/build/gnu-build-system.scm | 6 +++++-
>> > 1 file changed, 5 insertions(+), 1 deletion(-)
>> >
>> > diff --git a/guix/build/gnu-build-system.scm b/guix/build/gnu-build-system.scm
>> > index e5f3197b0..9d11e5b1e 100644
>> > --- a/guix/build/gnu-build-system.scm
>> > +++ b/guix/build/gnu-build-system.scm
>> > @@ -147,7 +147,7 @@ chance to be set."
>> > locale (strerror (system-error-errno args)))
>> > #t)))
>> >
>> > -(define* (unpack #:key source #:allow-other-keys)
>> > +(define* (unpack #:key source parallel-build? #:allow-other-keys)
>> > "Unpack SOURCE in the working directory, and change directory within the
>> > source. When SOURCE is a directory, copy it in a sub-directory of the current
>> > working directory."
>> > @@ -161,6 +161,10 @@ working directory."
>> > (copy-recursively source "."
>> > #:keep-mtime? #t))
>> > (begin
>> > + (when parallel-build?
>> > + (setenv "XZ_OPT"
>> > + (format #f "-T~d" (parallel-job-count))))
>> > +
>> > (if (string-suffix? ".zip" source)
>> > (invoke "unzip" source)
>> > (invoke "tar" "xvf" source))
>>
>> It's been a long long while, but now that core-updates has recently been
>> merged, I'd like to try and take a look at this again.
>>
>> I think the consensus was that this will only help for xz compressed
>> files where they have been compressed in parallel. I think it's still
>> worth doing though, as some of the big xz files that need decompressing
>> have been compressed in parallel, and this will speed up the builds when
>> multiple cores are available.
>>
>> Thanks,
>>
>> Chris
>
> I thought the last time we looked into this we figured out that there
> was a mistake in release notes or something and that parallel
> decompression isn't actually supported.
Hmm, I had a look to see if I could find some examples of where this
would apply, but I couldn't find any xz archives that we use in Guix
where it's been compressed in a way that allows multithreaded
decompression...
I'm pretty sure I had some examples before, but maybe somethings changed
in the intervening year.
Anyway, if I discover this again, I'll actually make a note of where
it's applicable.
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 962 bytes --]
prev parent reply other threads:[~2020-05-14 7:39 UTC|newest]
Thread overview: 11+ messages / expand[flat|nested] mbox.gz Atom feed top
2018-12-06 7:56 [bug#33643] [PATCH] gnu-build-system: Enable xz to decompress in parallel Christopher Baines
2018-12-06 8:08 ` Christopher Baines
2018-12-06 8:13 ` Leo Famulari
2018-12-06 19:38 ` Christopher Baines
2018-12-06 21:06 ` Leo Famulari
2018-12-09 14:32 ` Efraim Flashner
2018-12-10 16:24 ` Leo Famulari
2018-12-10 18:48 ` Efraim Flashner
2020-05-13 18:20 ` Christopher Baines
2020-05-13 19:07 ` Efraim Flashner
2020-05-14 7:37 ` Christopher Baines [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=87ftc3nezv.fsf@cbaines.net \
--to=mail@cbaines.net \
--cc=33643-done@debbugs.gnu.org \
--cc=efraim@flashner.co.il \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this external index
https://git.savannah.gnu.org/cgit/guix.git
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.