all messages for Guix-related lists mirrored at yhetil.org
 help / color / mirror / code / Atom feed
* bug#26316: Enable threaded compression of source tarballs
@ 2017-03-30 22:17 Leo Famulari
  2017-03-31 18:15 ` Kei Kebreau
  2017-04-15 22:35 ` Leo Famulari
  0 siblings, 2 replies; 6+ messages in thread
From: Leo Famulari @ 2017-03-30 22:17 UTC (permalink / raw)
  To: 26316


[-- Attachment #1.1: Type: text/plain, Size: 396 bytes --]

This patch enables multi-threaded xz compression in the repacking stage
of building source tarballs.

With it applied, I get a ~2.2x speedup building the source of
linux-libre with 4 threads, compared to 1 thread.

Specifically, with the upstream linux-libre tarball in the page cache,
`guix build -S linux-libre` took ~366 seconds using a single thread for
xz, and ~164 seconds using 4 threads.

[-- Attachment #1.2: 0001-packages-Enable-threaded-compression-of-source-tarba.patch --]
[-- Type: text/plain, Size: 1604 bytes --]

From 78547a09e50440c649c3d28d7691f32fdd47cc25 Mon Sep 17 00:00:00 2001
From: Leo Famulari <leo@famulari.name>
Date: Thu, 30 Mar 2017 03:10:48 -0400
Subject: [PATCH] packages: Enable threaded compression of source tarballs.

This provides a ~2x speedup when using 4 threads.

* guix/packages.scm (patch-and-repack)[build]: Invoke xz with
'--threads=0' when re-packing tarballs.
---
 guix/packages.scm | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/guix/packages.scm b/guix/packages.scm
index 4bc4b017f..30be5bf38 100644
--- a/guix/packages.scm
+++ b/guix/packages.scm
@@ -573,7 +573,12 @@ specifies modules in scope when evaluating SNIPPET."
                                                     #:fail-on-error? #t)))))
                         (zero? (apply system*
                                       (string-append #+tar "/bin/tar")
-                                      "cvfa" #$output
+                                      "cvf" #$output
+                                      ;; The bootstrap xz does not support
+                                      ;; threaded compression (introduced in
+                                      ;; 5.2.0), but it ignores the extra flag.
+                                      (string-append "--use-compress-program="
+                                                     #+xz "/bin/xz --threads=0")
                                       ;; avoid non-determinism in the archive
                                       "--mtime=@0"
                                       "--owner=root:0"
-- 
2.12.2


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

^ permalink raw reply related	[flat|nested] 6+ messages in thread

* bug#26316: Enable threaded compression of source tarballs
  2017-03-30 22:17 bug#26316: Enable threaded compression of source tarballs Leo Famulari
@ 2017-03-31 18:15 ` Kei Kebreau
  2017-03-31 23:04   ` Leo Famulari
  2017-04-15 22:35 ` Leo Famulari
  1 sibling, 1 reply; 6+ messages in thread
From: Kei Kebreau @ 2017-03-31 18:15 UTC (permalink / raw)
  To: Leo Famulari; +Cc: 26316

[-- Attachment #1: Type: text/plain, Size: 2080 bytes --]

Leo Famulari <leo@famulari.name> writes:

> This patch enables multi-threaded xz compression in the repacking stage
> of building source tarballs.
>
> With it applied, I get a ~2.2x speedup building the source of
> linux-libre with 4 threads, compared to 1 thread.
>
> Specifically, with the upstream linux-libre tarball in the page cache,
> `guix build -S linux-libre` took ~366 seconds using a single thread for
> xz, and ~164 seconds using 4 threads.
>
> From 78547a09e50440c649c3d28d7691f32fdd47cc25 Mon Sep 17 00:00:00 2001
> From: Leo Famulari <leo@famulari.name>
> Date: Thu, 30 Mar 2017 03:10:48 -0400
> Subject: [PATCH] packages: Enable threaded compression of source tarballs.
>
> This provides a ~2x speedup when using 4 threads.
>
> * guix/packages.scm (patch-and-repack)[build]: Invoke xz with
> '--threads=0' when re-packing tarballs.
> ---
>  guix/packages.scm | 7 ++++++-
>  1 file changed, 6 insertions(+), 1 deletion(-)
>
> diff --git a/guix/packages.scm b/guix/packages.scm
> index 4bc4b017f..30be5bf38 100644
> --- a/guix/packages.scm
> +++ b/guix/packages.scm
> @@ -573,7 +573,12 @@ specifies modules in scope when evaluating SNIPPET."
>                                                      #:fail-on-error? #t)))))
>                          (zero? (apply system*
>                                        (string-append #+tar "/bin/tar")
> -                                      "cvfa" #$output
> +                                      "cvf" #$output
> +                                      ;; The bootstrap xz does not support
> +                                      ;; threaded compression (introduced in
> +                                      ;; 5.2.0), but it ignores the extra flag.
> +                                      (string-append "--use-compress-program="
> +                                                     #+xz "/bin/xz --threads=0")
>                                        ;; avoid non-determinism in the archive
>                                        "--mtime=@0"
>                                        "--owner=root:0"

LGTM.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

^ permalink raw reply	[flat|nested] 6+ messages in thread

* bug#26316: Enable threaded compression of source tarballs
  2017-03-31 18:15 ` Kei Kebreau
@ 2017-03-31 23:04   ` Leo Famulari
  2017-03-31 23:56     ` Kei Kebreau
  0 siblings, 1 reply; 6+ messages in thread
From: Leo Famulari @ 2017-03-31 23:04 UTC (permalink / raw)
  To: Kei Kebreau; +Cc: 26316

[-- Attachment #1: Type: text/plain, Size: 1357 bytes --]

On Fri, Mar 31, 2017 at 02:15:36PM -0400, Kei Kebreau wrote:
> Leo Famulari <leo@famulari.name> writes:
> > Subject: [PATCH] packages: Enable threaded compression of source tarballs.

> > -                                      "cvfa" #$output
> > +                                      "cvf" #$output
> > +                                      ;; The bootstrap xz does not support
> > +                                      ;; threaded compression (introduced in
> > +                                      ;; 5.2.0), but it ignores the extra flag.
> > +                                      (string-append "--use-compress-program="
> > +                                                     #+xz "/bin/xz --threads=0")

One question I have is about the removal of 'cvfa'. Here is the relevant
documentation:

-a, --auto-compress
             Use archive suffix to determine the compression
	     program.

AFAICT, we only build XZ archives, so it's okay to remove 'a' and
hard-code the use of xz. Is that correct?

> >                                        ;; avoid non-determinism in the archive
> >                                        "--mtime=@0"
> >                                        "--owner=root:0"
> 
> LGTM.

Thanks! This does mean rebuilding the world, so we'd need to save it for
the next core-updates cycle.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

^ permalink raw reply	[flat|nested] 6+ messages in thread

* bug#26316: Enable threaded compression of source tarballs
  2017-03-31 23:04   ` Leo Famulari
@ 2017-03-31 23:56     ` Kei Kebreau
  2017-04-01  0:15       ` Leo Famulari
  0 siblings, 1 reply; 6+ messages in thread
From: Kei Kebreau @ 2017-03-31 23:56 UTC (permalink / raw)
  To: Leo Famulari; +Cc: 26316

[-- Attachment #1: Type: text/plain, Size: 1900 bytes --]

Leo Famulari <leo@famulari.name> writes:

> On Fri, Mar 31, 2017 at 02:15:36PM -0400, Kei Kebreau wrote:
>> Leo Famulari <leo@famulari.name> writes:
>> > Subject: [PATCH] packages: Enable threaded compression of source tarballs.
>
>> > -                                      "cvfa" #$output
>> > +                                      "cvf" #$output
>> > +                                      ;; The bootstrap xz does not support
>> > + ;; threaded compression (introduced in
>> > + ;; 5.2.0), but it ignores the extra flag.
>> > + (string-append "--use-compress-program="
>> > + #+xz "/bin/xz --threads=0")
>
> One question I have is about the removal of 'cvfa'. Here is the relevant
> documentation:
>
> -a, --auto-compress
>              Use archive suffix to determine the compression
> 	     program.
>
> AFAICT, we only build XZ archives, so it's okay to remove 'a' and
> hard-code the use of xz. Is that correct?
>

The surrounding code definitely seems to suggest that. Reading "up" the
code from the gexp->derivation call I can find hardcoded xz usage in the
tarxz-name procedure and this bit:

(mlet %store-monad ((tar ->     (lookup-input "tar"))
                  ->(xz ->      (lookup-input "xz"))
  (only xz here!)/  (patch ->   (lookup-input "patch"))
                    (locales -> (lookup-input "locales"))
                    (decomp ->  (lookup-input decompression-type))
                    (patches    (sequence %store-monad
                                          (map instantiate-patch patches))))

>> >                                        ;; avoid non-determinism in
>> > the archive
>> >                                        "--mtime=@0"
>> >                                        "--owner=root:0"
>> 
>> LGTM.
>
> Thanks! This does mean rebuilding the world, so we'd need to save it for
> the next core-updates cycle.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

^ permalink raw reply	[flat|nested] 6+ messages in thread

* bug#26316: Enable threaded compression of source tarballs
  2017-03-31 23:56     ` Kei Kebreau
@ 2017-04-01  0:15       ` Leo Famulari
  0 siblings, 0 replies; 6+ messages in thread
From: Leo Famulari @ 2017-04-01  0:15 UTC (permalink / raw)
  To: Kei Kebreau; +Cc: 26316

[-- Attachment #1: Type: text/plain, Size: 736 bytes --]

On Fri, Mar 31, 2017 at 07:56:18PM -0400, Kei Kebreau wrote:
> The surrounding code definitely seems to suggest that. Reading "up" the
> code from the gexp->derivation call I can find hardcoded xz usage in the
> tarxz-name procedure and this bit:
> 
> (mlet %store-monad ((tar ->     (lookup-input "tar"))
>                   ->(xz ->      (lookup-input "xz"))
>   (only xz here!)/  (patch ->   (lookup-input "patch"))
>                     (locales -> (lookup-input "locales"))
>                     (decomp ->  (lookup-input decompression-type))
>                     (patches    (sequence %store-monad
>                                           (map instantiate-patch patches))))

Right, that's what I was looking at.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

^ permalink raw reply	[flat|nested] 6+ messages in thread

* bug#26316: Enable threaded compression of source tarballs
  2017-03-30 22:17 bug#26316: Enable threaded compression of source tarballs Leo Famulari
  2017-03-31 18:15 ` Kei Kebreau
@ 2017-04-15 22:35 ` Leo Famulari
  1 sibling, 0 replies; 6+ messages in thread
From: Leo Famulari @ 2017-04-15 22:35 UTC (permalink / raw)
  To: 26316-done

[-- Attachment #1: Type: text/plain, Size: 971 bytes --]

On Thu, Mar 30, 2017 at 06:17:15PM -0400, Leo Famulari wrote:
> This patch enables multi-threaded xz compression in the repacking stage
> of building source tarballs.
> 
> With it applied, I get a ~2.2x speedup building the source of
> linux-libre with 4 threads, compared to 1 thread.
> 
> Specifically, with the upstream linux-libre tarball in the page cache,
> `guix build -S linux-libre` took ~366 seconds using a single thread for
> xz, and ~164 seconds using 4 threads.

> From 78547a09e50440c649c3d28d7691f32fdd47cc25 Mon Sep 17 00:00:00 2001
> From: Leo Famulari <leo@famulari.name>
> Date: Thu, 30 Mar 2017 03:10:48 -0400
> Subject: [PATCH] packages: Enable threaded compression of source tarballs.
> 
> This provides a ~2x speedup when using 4 threads.
> 
> * guix/packages.scm (patch-and-repack)[build]: Invoke xz with
> '--threads=0' when re-packing tarballs.

I pushed this to core-updates as
c8a3dea847bb9f87fa1876d0c6c3356d6226f121.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

^ permalink raw reply	[flat|nested] 6+ messages in thread

end of thread, other threads:[~2017-04-15 22:37 UTC | newest]

Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2017-03-30 22:17 bug#26316: Enable threaded compression of source tarballs Leo Famulari
2017-03-31 18:15 ` Kei Kebreau
2017-03-31 23:04   ` Leo Famulari
2017-03-31 23:56     ` Kei Kebreau
2017-04-01  0:15       ` Leo Famulari
2017-04-15 22:35 ` Leo Famulari

Code repositories for project(s) associated with this external index

	https://git.savannah.gnu.org/cgit/guix.git

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.