* bug#26316: Enable threaded compression of source tarballs
@ 2017-03-30 22:17 Leo Famulari
2017-03-31 18:15 ` Kei Kebreau
2017-04-15 22:35 ` Leo Famulari
0 siblings, 2 replies; 6+ messages in thread
From: Leo Famulari @ 2017-03-30 22:17 UTC (permalink / raw)
To: 26316
[-- Attachment #1.1: Type: text/plain, Size: 396 bytes --]
This patch enables multi-threaded xz compression in the repacking stage
of building source tarballs.
With it applied, I get a ~2.2x speedup building the source of
linux-libre with 4 threads, compared to 1 thread.
Specifically, with the upstream linux-libre tarball in the page cache,
`guix build -S linux-libre` took ~366 seconds using a single thread for
xz, and ~164 seconds using 4 threads.
[-- Attachment #1.2: 0001-packages-Enable-threaded-compression-of-source-tarba.patch --]
[-- Type: text/plain, Size: 1604 bytes --]
From 78547a09e50440c649c3d28d7691f32fdd47cc25 Mon Sep 17 00:00:00 2001
From: Leo Famulari <leo@famulari.name>
Date: Thu, 30 Mar 2017 03:10:48 -0400
Subject: [PATCH] packages: Enable threaded compression of source tarballs.
This provides a ~2x speedup when using 4 threads.
* guix/packages.scm (patch-and-repack)[build]: Invoke xz with
'--threads=0' when re-packing tarballs.
---
guix/packages.scm | 7 ++++++-
1 file changed, 6 insertions(+), 1 deletion(-)
diff --git a/guix/packages.scm b/guix/packages.scm
index 4bc4b017f..30be5bf38 100644
--- a/guix/packages.scm
+++ b/guix/packages.scm
@@ -573,7 +573,12 @@ specifies modules in scope when evaluating SNIPPET."
#:fail-on-error? #t)))))
(zero? (apply system*
(string-append #+tar "/bin/tar")
- "cvfa" #$output
+ "cvf" #$output
+ ;; The bootstrap xz does not support
+ ;; threaded compression (introduced in
+ ;; 5.2.0), but it ignores the extra flag.
+ (string-append "--use-compress-program="
+ #+xz "/bin/xz --threads=0")
;; avoid non-determinism in the archive
"--mtime=@0"
"--owner=root:0"
--
2.12.2
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply related [flat|nested] 6+ messages in thread* bug#26316: Enable threaded compression of source tarballs
2017-03-30 22:17 bug#26316: Enable threaded compression of source tarballs Leo Famulari
@ 2017-03-31 18:15 ` Kei Kebreau
2017-03-31 23:04 ` Leo Famulari
2017-04-15 22:35 ` Leo Famulari
1 sibling, 1 reply; 6+ messages in thread
From: Kei Kebreau @ 2017-03-31 18:15 UTC (permalink / raw)
To: Leo Famulari; +Cc: 26316
[-- Attachment #1: Type: text/plain, Size: 2080 bytes --]
Leo Famulari <leo@famulari.name> writes:
> This patch enables multi-threaded xz compression in the repacking stage
> of building source tarballs.
>
> With it applied, I get a ~2.2x speedup building the source of
> linux-libre with 4 threads, compared to 1 thread.
>
> Specifically, with the upstream linux-libre tarball in the page cache,
> `guix build -S linux-libre` took ~366 seconds using a single thread for
> xz, and ~164 seconds using 4 threads.
>
> From 78547a09e50440c649c3d28d7691f32fdd47cc25 Mon Sep 17 00:00:00 2001
> From: Leo Famulari <leo@famulari.name>
> Date: Thu, 30 Mar 2017 03:10:48 -0400
> Subject: [PATCH] packages: Enable threaded compression of source tarballs.
>
> This provides a ~2x speedup when using 4 threads.
>
> * guix/packages.scm (patch-and-repack)[build]: Invoke xz with
> '--threads=0' when re-packing tarballs.
> ---
> guix/packages.scm | 7 ++++++-
> 1 file changed, 6 insertions(+), 1 deletion(-)
>
> diff --git a/guix/packages.scm b/guix/packages.scm
> index 4bc4b017f..30be5bf38 100644
> --- a/guix/packages.scm
> +++ b/guix/packages.scm
> @@ -573,7 +573,12 @@ specifies modules in scope when evaluating SNIPPET."
> #:fail-on-error? #t)))))
> (zero? (apply system*
> (string-append #+tar "/bin/tar")
> - "cvfa" #$output
> + "cvf" #$output
> + ;; The bootstrap xz does not support
> + ;; threaded compression (introduced in
> + ;; 5.2.0), but it ignores the extra flag.
> + (string-append "--use-compress-program="
> + #+xz "/bin/xz --threads=0")
> ;; avoid non-determinism in the archive
> "--mtime=@0"
> "--owner=root:0"
LGTM.
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]
^ permalink raw reply [flat|nested] 6+ messages in thread
* bug#26316: Enable threaded compression of source tarballs
2017-03-31 18:15 ` Kei Kebreau
@ 2017-03-31 23:04 ` Leo Famulari
2017-03-31 23:56 ` Kei Kebreau
0 siblings, 1 reply; 6+ messages in thread
From: Leo Famulari @ 2017-03-31 23:04 UTC (permalink / raw)
To: Kei Kebreau; +Cc: 26316
[-- Attachment #1: Type: text/plain, Size: 1357 bytes --]
On Fri, Mar 31, 2017 at 02:15:36PM -0400, Kei Kebreau wrote:
> Leo Famulari <leo@famulari.name> writes:
> > Subject: [PATCH] packages: Enable threaded compression of source tarballs.
> > - "cvfa" #$output
> > + "cvf" #$output
> > + ;; The bootstrap xz does not support
> > + ;; threaded compression (introduced in
> > + ;; 5.2.0), but it ignores the extra flag.
> > + (string-append "--use-compress-program="
> > + #+xz "/bin/xz --threads=0")
One question I have is about the removal of 'cvfa'. Here is the relevant
documentation:
-a, --auto-compress
Use archive suffix to determine the compression
program.
AFAICT, we only build XZ archives, so it's okay to remove 'a' and
hard-code the use of xz. Is that correct?
> > ;; avoid non-determinism in the archive
> > "--mtime=@0"
> > "--owner=root:0"
>
> LGTM.
Thanks! This does mean rebuilding the world, so we'd need to save it for
the next core-updates cycle.
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 6+ messages in thread* bug#26316: Enable threaded compression of source tarballs
2017-03-31 23:04 ` Leo Famulari
@ 2017-03-31 23:56 ` Kei Kebreau
2017-04-01 0:15 ` Leo Famulari
0 siblings, 1 reply; 6+ messages in thread
From: Kei Kebreau @ 2017-03-31 23:56 UTC (permalink / raw)
To: Leo Famulari; +Cc: 26316
[-- Attachment #1: Type: text/plain, Size: 1900 bytes --]
Leo Famulari <leo@famulari.name> writes:
> On Fri, Mar 31, 2017 at 02:15:36PM -0400, Kei Kebreau wrote:
>> Leo Famulari <leo@famulari.name> writes:
>> > Subject: [PATCH] packages: Enable threaded compression of source tarballs.
>
>> > - "cvfa" #$output
>> > + "cvf" #$output
>> > + ;; The bootstrap xz does not support
>> > + ;; threaded compression (introduced in
>> > + ;; 5.2.0), but it ignores the extra flag.
>> > + (string-append "--use-compress-program="
>> > + #+xz "/bin/xz --threads=0")
>
> One question I have is about the removal of 'cvfa'. Here is the relevant
> documentation:
>
> -a, --auto-compress
> Use archive suffix to determine the compression
> program.
>
> AFAICT, we only build XZ archives, so it's okay to remove 'a' and
> hard-code the use of xz. Is that correct?
>
The surrounding code definitely seems to suggest that. Reading "up" the
code from the gexp->derivation call I can find hardcoded xz usage in the
tarxz-name procedure and this bit:
(mlet %store-monad ((tar -> (lookup-input "tar"))
->(xz -> (lookup-input "xz"))
(only xz here!)/ (patch -> (lookup-input "patch"))
(locales -> (lookup-input "locales"))
(decomp -> (lookup-input decompression-type))
(patches (sequence %store-monad
(map instantiate-patch patches))))
>> > ;; avoid non-determinism in
>> > the archive
>> > "--mtime=@0"
>> > "--owner=root:0"
>>
>> LGTM.
>
> Thanks! This does mean rebuilding the world, so we'd need to save it for
> the next core-updates cycle.
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]
^ permalink raw reply [flat|nested] 6+ messages in thread
* bug#26316: Enable threaded compression of source tarballs
2017-03-30 22:17 bug#26316: Enable threaded compression of source tarballs Leo Famulari
2017-03-31 18:15 ` Kei Kebreau
@ 2017-04-15 22:35 ` Leo Famulari
1 sibling, 0 replies; 6+ messages in thread
From: Leo Famulari @ 2017-04-15 22:35 UTC (permalink / raw)
To: 26316-done
[-- Attachment #1: Type: text/plain, Size: 971 bytes --]
On Thu, Mar 30, 2017 at 06:17:15PM -0400, Leo Famulari wrote:
> This patch enables multi-threaded xz compression in the repacking stage
> of building source tarballs.
>
> With it applied, I get a ~2.2x speedup building the source of
> linux-libre with 4 threads, compared to 1 thread.
>
> Specifically, with the upstream linux-libre tarball in the page cache,
> `guix build -S linux-libre` took ~366 seconds using a single thread for
> xz, and ~164 seconds using 4 threads.
> From 78547a09e50440c649c3d28d7691f32fdd47cc25 Mon Sep 17 00:00:00 2001
> From: Leo Famulari <leo@famulari.name>
> Date: Thu, 30 Mar 2017 03:10:48 -0400
> Subject: [PATCH] packages: Enable threaded compression of source tarballs.
>
> This provides a ~2x speedup when using 4 threads.
>
> * guix/packages.scm (patch-and-repack)[build]: Invoke xz with
> '--threads=0' when re-packing tarballs.
I pushed this to core-updates as
c8a3dea847bb9f87fa1876d0c6c3356d6226f121.
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2017-04-15 22:37 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2017-03-30 22:17 bug#26316: Enable threaded compression of source tarballs Leo Famulari
2017-03-31 18:15 ` Kei Kebreau
2017-03-31 23:04 ` Leo Famulari
2017-03-31 23:56 ` Kei Kebreau
2017-04-01 0:15 ` Leo Famulari
2017-04-15 22:35 ` Leo Famulari
Code repositories for project(s) associated with this public inbox
https://git.savannah.gnu.org/cgit/guix.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).