unofficial mirror of guix-devel@gnu.org 
 help / color / mirror / code / Atom feed
* parallelizing more actions
@ 2017-11-09  9:05 Efraim Flashner
  2017-11-11  1:42 ` Chris Marusich
                   ` (2 more replies)
  0 siblings, 3 replies; 7+ messages in thread
From: Efraim Flashner @ 2017-11-09  9:05 UTC (permalink / raw)
  To: guix-devel


[-- Attachment #1.1: Type: text/plain, Size: 1006 bytes --]

While rebuilding glibc-final on aarch64 I realized that the 'strip phase
took 235 seconds. The relevant code for 'strip from gnu-build-system is
in guix/build/gnu-build-system.scm, starting at line 340, with the
actual stripping starting at 398. When I changed 'for-each' to
'par-for-each' the time dropped from 235 seconds to 215, about an 8.5%
savings. I'm pretty sure most of that time was spent failing to strip
certain files, but it is still a savings.

I'm guessing there are other parts that can be parallelized in a similar
manner.

As for par-for-each (as many threads as cores on the machine) vs
n-par-for-each (n threads), I think it would be better to use
n-par-for-each, but I didn't quickly see a way to limit based on the
number of cores offered to each builder.

-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #1.2: parallel-strip.diff --]
[-- Type: text/plain, Size: 2392 bytes --]

diff --git a/guix/build/gnu-build-system.scm b/guix/build/gnu-build-system.scm
index e37b75140..3162ec08b 100644
--- a/guix/build/gnu-build-system.scm
+++ b/guix/build/gnu-build-system.scm
@@ -24,6 +24,7 @@
   #:use-module (ice-9 match)
   #:use-module (ice-9 regex)
   #:use-module (ice-9 format)
+  #:use-module (ice-9 threads)
   #:use-module (srfi srfi-1)
   #:use-module (srfi srfi-19)
   #:use-module (srfi srfi-26)
@@ -395,24 +396,24 @@ makefiles."
       (format #t "debugging output written to ~s using ~s~%"
               debug-output objcopy-command))
 
-    (for-each (lambda (file)
-                (and (or (elf-file? file) (ar-file? file))
-                     (or (not debug-output)
-                         (make-debug-file file))
-
-                     ;; Ensure the file is writable.
-                     (begin (make-file-writable file) #t)
-
-                     (zero? (apply system* strip-command
-                                   (append strip-flags (list file))))
-                     (or (not debug-output)
-                         (add-debug-link file))))
-              (find-files dir
-                          (lambda (file stat)
-                            ;; Ignore symlinks such as:
-                            ;; libfoo.so -> libfoo.so.0.0.
-                            (eq? 'regular (stat:type stat)))
-                          #:stat lstat)))
+    (par-for-each (lambda (file)
+                    (and (or (elf-file? file) (ar-file? file))
+                         (or (not debug-output)
+                             (make-debug-file file))
+
+                         ;; Ensure the file is writable.
+                         (begin (make-file-writable file) #t)
+
+                         (zero? (apply system* strip-command
+                                       (append strip-flags (list file))))
+                         (or (not debug-output)
+                             (add-debug-link file))))
+                  (find-files dir
+                              (lambda (file stat)
+                                ;; Ignore symlinks such as:
+                                ;; libfoo.so -> libfoo.so.0.0.
+                                (eq? 'regular (stat:type stat)))
+                              #:stat lstat)))
 
   (or (not strip-binaries?)
       (every strip-dir

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

^ permalink raw reply related	[flat|nested] 7+ messages in thread

end of thread, other threads:[~2017-11-16 10:06 UTC | newest]

Thread overview: 7+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2017-11-09  9:05 parallelizing more actions Efraim Flashner
2017-11-11  1:42 ` Chris Marusich
2017-11-11 11:25 ` Ludovic Courtès
2017-11-11 16:49   ` Efraim Flashner
2017-11-12 13:31 ` Efraim Flashner
2017-11-13 18:55   ` Efraim Flashner
2017-11-16 10:06     ` Ludovic Courtès

Code repositories for project(s) associated with this public inbox

	https://git.savannah.gnu.org/cgit/guix.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).