parallelizing more actions

parallelizing more actions

[Efraim Flashner]

[-- Attachment #1.1: Type: text/plain, Size: 1006 bytes --]

While rebuilding glibc-final on aarch64 I realized that the 'strip phase
took 235 seconds. The relevant code for 'strip from gnu-build-system is
in guix/build/gnu-build-system.scm, starting at line 340, with the
actual stripping starting at 398. When I changed 'for-each' to
'par-for-each' the time dropped from 235 seconds to 215, about an 8.5%
savings. I'm pretty sure most of that time was spent failing to strip
certain files, but it is still a savings.

I'm guessing there are other parts that can be parallelized in a similar
manner.

As for par-for-each (as many threads as cores on the machine) vs
n-par-for-each (n threads), I think it would be better to use
n-par-for-each, but I didn't quickly see a way to limit based on the
number of cores offered to each builder.

-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #1.2: parallel-strip.diff --]
[-- Type: text/plain, Size: 2392 bytes --]

diff --git a/guix/build/gnu-build-system.scm b/guix/build/gnu-build-system.scm
index e37b75140..3162ec08b 100644
--- a/guix/build/gnu-build-system.scm
+++ b/guix/build/gnu-build-system.scm
@@ -24,6 +24,7 @@
   #:use-module (ice-9 match)
   #:use-module (ice-9 regex)
   #:use-module (ice-9 format)
+  #:use-module (ice-9 threads)
   #:use-module (srfi srfi-1)
   #:use-module (srfi srfi-19)
   #:use-module (srfi srfi-26)
@@ -395,24 +396,24 @@ makefiles."
       (format #t "debugging output written to ~s using ~s~%"
               debug-output objcopy-command))
 
-    (for-each (lambda (file)
-                (and (or (elf-file? file) (ar-file? file))
-                     (or (not debug-output)
-                         (make-debug-file file))
-
-                     ;; Ensure the file is writable.
-                     (begin (make-file-writable file) #t)
-
-                     (zero? (apply system* strip-command
-                                   (append strip-flags (list file))))
-                     (or (not debug-output)
-                         (add-debug-link file))))
-              (find-files dir
-                          (lambda (file stat)
-                            ;; Ignore symlinks such as:
-                            ;; libfoo.so -> libfoo.so.0.0.
-                            (eq? 'regular (stat:type stat)))
-                          #:stat lstat)))
+    (par-for-each (lambda (file)
+                    (and (or (elf-file? file) (ar-file? file))
+                         (or (not debug-output)
+                             (make-debug-file file))
+
+                         ;; Ensure the file is writable.
+                         (begin (make-file-writable file) #t)
+
+                         (zero? (apply system* strip-command
+                                       (append strip-flags (list file))))
+                         (or (not debug-output)
+                             (add-debug-link file))))
+                  (find-files dir
+                              (lambda (file stat)
+                                ;; Ignore symlinks such as:
+                                ;; libfoo.so -> libfoo.so.0.0.
+                                (eq? 'regular (stat:type stat)))
+                              #:stat lstat)))
 
   (or (not strip-binaries?)
       (every strip-dir

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: parallelizing more actions

[Chris Marusich]

[-- Attachment #1: Type: text/plain, Size: 1244 bytes --]

Efraim Flashner <efraim@flashner.co.il> writes:

> While rebuilding glibc-final on aarch64 I realized that the 'strip
> phase took 235 seconds. The relevant code for 'strip from
> gnu-build-system is in guix/build/gnu-build-system.scm, starting at
> line 340, with the actual stripping starting at 398. When I changed
> 'for-each' to 'par-for-each' the time dropped from 235 seconds to 215,
> about an 8.5% savings. I'm pretty sure most of that time was spent
> failing to strip certain files, but it is still a savings.

Nice!

>
> I'm guessing there are other parts that can be parallelized in a
> similar manner.
>
> As for par-for-each (as many threads as cores on the machine) vs
> n-par-for-each (n threads), I think it would be better to use
> n-par-for-each, but I didn't quickly see a way to limit based on the
> number of cores offered to each builder.

I think you want parallel-job-count in (guix build utils):

(define parallel-job-count
  ;; Number of processes to be passed next to GNU Make's `-j' argument.
  (make-parameter
   (match (getenv "NIX_BUILD_CORES")              ;set by the daemon
     (#f  1)
     ("0" (current-processor-count))
     (x   (or (string->number x) 1)))))

-- 
Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: parallelizing more actions

[Ludovic Courtès]

Hi Efraim,

Efraim Flashner <efraim@flashner.co.il> skribis:

> While rebuilding glibc-final on aarch64 I realized that the 'strip phase
> took 235 seconds. The relevant code for 'strip from gnu-build-system is
> in guix/build/gnu-build-system.scm, starting at line 340, with the
> actual stripping starting at 398. When I changed 'for-each' to
> 'par-for-each' the time dropped from 235 seconds to 215, about an 8.5%
> savings. I'm pretty sure most of that time was spent failing to strip
> certain files, but it is still a savings.

Is it on a spinning hard disk or an SSD?  My guess is that most of the
time is I/O, and that parallelizing doesn’t buy us much (indeed, 8.5% is
not that much, far from a linear speedup.) 

So I’m mildly reluctant to parallelizing this particular piece of code.

Thanks,
Ludo’.

Re: parallelizing more actions

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 1200 bytes --]

On Sat, Nov 11, 2017 at 12:25:01PM +0100, Ludovic Courtès wrote:
> Hi Efraim,
> 
> Efraim Flashner <efraim@flashner.co.il> skribis:
> 
> > While rebuilding glibc-final on aarch64 I realized that the 'strip phase
> > took 235 seconds. The relevant code for 'strip from gnu-build-system is
> > in guix/build/gnu-build-system.scm, starting at line 340, with the
> > actual stripping starting at 398. When I changed 'for-each' to
> > 'par-for-each' the time dropped from 235 seconds to 215, about an 8.5%
> > savings. I'm pretty sure most of that time was spent failing to strip
> > certain files, but it is still a savings.
> 
> Is it on a spinning hard disk or an SSD?  My guess is that most of the
> time is I/O, and that parallelizing doesn’t buy us much (indeed, 8.5% is
> not that much, far from a linear speedup.) 
> 

It was on eMMC, so basically an SSD.

> So I’m mildly reluctant to parallelizing this particular piece of code.
> 
> Thanks,
> Ludo’.

-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: parallelizing more actions

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 1087 bytes --]

On Thu, Nov 09, 2017 at 11:05:59AM +0200, Efraim Flashner wrote:
> While rebuilding glibc-final on aarch64 I realized that the 'strip phase
> took 235 seconds. The relevant code for 'strip from gnu-build-system is
> in guix/build/gnu-build-system.scm, starting at line 340, with the
> actual stripping starting at 398. When I changed 'for-each' to
> 'par-for-each' the time dropped from 235 seconds to 215, about an 8.5%
> savings. I'm pretty sure most of that time was spent failing to strip
> certain files, but it is still a savings.
> 

These numbers were for glibc@2.26. I ran it against glibc@2.25 on my
x86_64 machine with a spinning harddrive and 2 cores, without the patch
it took 165.7 seconds (164.6 the second time), with the patch it dropped
to 24.3 seconds.

I'll test it with glibc@2.25 on aarch64 later when its not building
other things.


-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: parallelizing more actions

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 1741 bytes --]

On Sun, Nov 12, 2017 at 03:31:49PM +0200, Efraim Flashner wrote:
> On Thu, Nov 09, 2017 at 11:05:59AM +0200, Efraim Flashner wrote:
> > While rebuilding glibc-final on aarch64 I realized that the 'strip phase
> > took 235 seconds. The relevant code for 'strip from gnu-build-system is
> > in guix/build/gnu-build-system.scm, starting at line 340, with the
> > actual stripping starting at 398. When I changed 'for-each' to
> > 'par-for-each' the time dropped from 235 seconds to 215, about an 8.5%
> > savings. I'm pretty sure most of that time was spent failing to strip
> > certain files, but it is still a savings.
> > 
> 
> These numbers were for glibc@2.26. I ran it against glibc@2.25 on my
> x86_64 machine with a spinning harddrive and 2 cores, without the patch
> it took 165.7 seconds (164.6 the second time), with the patch it dropped
> to 24.3 seconds.
> 
> I'll test it with glibc@2.25 on aarch64 later when its not building
> other things.
> 

Looks like aarch64 could use some help somewhere. With the patch it took
221.9 seconds, without the patch it was 219.8.

so in summary, aarch64 is terrible at stripping, parallel or not, and on
x86_64 my patch saved 85% of the patching time, despite only using 2 cores.


both versions for aarch64 had many lines like:
strip: /gnu/store/gslfpqdmgbwb5ryilq33bn9vs1466fn2-glibc-2.25-debug/lib/debug//gnu/store/7iihk77hqvsqjx7dnb19237dg4xgqvis-glibc-2.25/lib/stSy721B/fgetws_chk.o: no group info for section .data.DW.ref.__gcc_personality_v0


-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: parallelizing more actions

[Ludovic Courtès]

Efraim Flashner <efraim@flashner.co.il> skribis:

> On Sun, Nov 12, 2017 at 03:31:49PM +0200, Efraim Flashner wrote:
>> On Thu, Nov 09, 2017 at 11:05:59AM +0200, Efraim Flashner wrote:
>> > While rebuilding glibc-final on aarch64 I realized that the 'strip phase
>> > took 235 seconds. The relevant code for 'strip from gnu-build-system is
>> > in guix/build/gnu-build-system.scm, starting at line 340, with the
>> > actual stripping starting at 398. When I changed 'for-each' to
>> > 'par-for-each' the time dropped from 235 seconds to 215, about an 8.5%
>> > savings. I'm pretty sure most of that time was spent failing to strip
>> > certain files, but it is still a savings.
>> > 
>> 
>> These numbers were for glibc@2.26. I ran it against glibc@2.25 on my
>> x86_64 machine with a spinning harddrive and 2 cores, without the patch
>> it took 165.7 seconds (164.6 the second time), with the patch it dropped
>> to 24.3 seconds.
>> 
>> I'll test it with glibc@2.25 on aarch64 later when its not building
>> other things.
>> 
>
> Looks like aarch64 could use some help somewhere. With the patch it took
> 221.9 seconds, without the patch it was 219.8.
>
> so in summary, aarch64 is terrible at stripping, parallel or not, and on
> x86_64 my patch saved 85% of the patching time, despite only using 2 cores.

This is weird.  I don’t see any reason why ‘strip’ would be slower on
aarch64 than elsewhere.  Are you sure it’s not mostly I/O?  It would be
nice to time it and see the real/user/system times.

Thanks,
Ludo’.