From mboxrd@z Thu Jan 1 00:00:00 1970 From: Efraim Flashner Subject: parallelizing more actions Date: Thu, 9 Nov 2017 11:05:59 +0200 Message-ID: <20171109090559.GF1815@macbook41> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="TBNym+cBXeFsS4Vs" Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:43211) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eCimm-0000gI-Lx for guix-devel@gnu.org; Thu, 09 Nov 2017 04:06:14 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1eCimg-0001uV-M1 for guix-devel@gnu.org; Thu, 09 Nov 2017 04:06:08 -0500 Received: from flashner.co.il ([178.62.234.194]:59865) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eCimg-0001u5-9F for guix-devel@gnu.org; Thu, 09 Nov 2017 04:06:02 -0500 Received: from localhost (46-117-130-79.bb.netvision.net.il [46.117.130.79]) by flashner.co.il (Postfix) with ESMTPSA id 9E36240059 for ; Thu, 9 Nov 2017 09:06:00 +0000 (UTC) Content-Disposition: inline List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+gcggd-guix-devel=m.gmane.org@gnu.org Sender: "Guix-devel" To: guix-devel@gnu.org --TBNym+cBXeFsS4Vs Content-Type: multipart/mixed; boundary="RDS4xtyBfx+7DiaI" Content-Disposition: inline --RDS4xtyBfx+7DiaI Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable While rebuilding glibc-final on aarch64 I realized that the 'strip phase took 235 seconds. The relevant code for 'strip from gnu-build-system is in guix/build/gnu-build-system.scm, starting at line 340, with the actual stripping starting at 398. When I changed 'for-each' to 'par-for-each' the time dropped from 235 seconds to 215, about an 8.5% savings. I'm pretty sure most of that time was spent failing to strip certain files, but it is still a savings. I'm guessing there are other parts that can be parallelized in a similar manner. As for par-for-each (as many threads as cores on the machine) vs n-par-for-each (n threads), I think it would be better to use n-par-for-each, but I didn't quickly see a way to limit based on the number of cores offered to each builder. --=20 Efraim Flashner =D7=90=D7=A4=D7=A8=D7=99=D7=9D = =D7=A4=D7=9C=D7=A9=D7=A0=D7=A8 GPG key =3D A28B F40C 3E55 1372 662D 14F7 41AA E7DC CA3D 8351 Confidentiality cannot be guaranteed on emails sent or received unencrypted --RDS4xtyBfx+7DiaI Content-Type: text/plain; charset=utf-8 Content-Disposition: attachment; filename="parallel-strip.diff" Content-Transfer-Encoding: quoted-printable diff --git a/guix/build/gnu-build-system.scm b/guix/build/gnu-build-system.= scm index e37b75140..3162ec08b 100644 --- a/guix/build/gnu-build-system.scm +++ b/guix/build/gnu-build-system.scm @@ -24,6 +24,7 @@ #:use-module (ice-9 match) #:use-module (ice-9 regex) #:use-module (ice-9 format) + #:use-module (ice-9 threads) #:use-module (srfi srfi-1) #:use-module (srfi srfi-19) #:use-module (srfi srfi-26) @@ -395,24 +396,24 @@ makefiles." (format #t "debugging output written to ~s using ~s~%" debug-output objcopy-command)) =20 - (for-each (lambda (file) - (and (or (elf-file? file) (ar-file? file)) - (or (not debug-output) - (make-debug-file file)) - - ;; Ensure the file is writable. - (begin (make-file-writable file) #t) - - (zero? (apply system* strip-command - (append strip-flags (list file)))) - (or (not debug-output) - (add-debug-link file)))) - (find-files dir - (lambda (file stat) - ;; Ignore symlinks such as: - ;; libfoo.so -> libfoo.so.0.0. - (eq? 'regular (stat:type stat))) - #:stat lstat))) + (par-for-each (lambda (file) + (and (or (elf-file? file) (ar-file? file)) + (or (not debug-output) + (make-debug-file file)) + + ;; Ensure the file is writable. + (begin (make-file-writable file) #t) + + (zero? (apply system* strip-command + (append strip-flags (list file)))) + (or (not debug-output) + (add-debug-link file)))) + (find-files dir + (lambda (file stat) + ;; Ignore symlinks such as: + ;; libfoo.so -> libfoo.so.0.0. + (eq? 'regular (stat:type stat))) + #:stat lstat))) =20 (or (not strip-binaries?) (every strip-dir --RDS4xtyBfx+7DiaI-- --TBNym+cBXeFsS4Vs Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAEBCgAdFiEEoov0DD5VE3JmLRT3Qarn3Mo9g1EFAloEGnAACgkQQarn3Mo9 g1H54BAAqEG67hgaslyFANXiaUdMgFQXcUt4JybX4vJJGTjqUT8oKKbtkavdPJib ALnJoSjrG0X3vE+eiledPoKH+9I/iU2FWDS7BrAq/axJPtfoXYd54ap4xSNpVS8+ PoAdcdSxc/ca3x6HQgMNced0HpgckwH2/42lWAt8QseD7XnTH9wC2wGGR5N+uW1x l6PWNdNTwEVpWhtfkQ1XfI9ftCYNZxKYz2vfz+K89J5r55ogrdvIrlIgdgf9dB15 s3+tkYWg9Gvhwvc9/6MA9BKKcZ/5IQBTCwPgojqq90Xsr1cs0h7nGna1+e6nfJ74 F0m6p16nsGN19C6ltcJWGcJUR13dieqUOytV+9XMRMk26tp1WSLI5sJDuMA6BLr7 wAcP+rUjean6iOdkC+vN4E/ihIA6gGA2qbQTmrZJ5kmIMba18k+1gVrEgB37Ygz8 RKkcwr+Na1Jd6a0oASeRvoUHXDHpiMRBBi6fC7d2iytHu8lqIqgPdCCyLoVM3sd9 i6i1aASSqfOJRGN1+q1X3sBLYcVd+QklOIGDJoeCgPTjrU1G1qW2YMQ0nFdU0Z2C /nPcJlqcXsShcSIrvFFibOSySZW0pPRGX4VC3cf0FOHjZ4956mBD7BHgaOhw9lXd zNEDIz8YZ76iKW6q93BG4sMRtgRhFNZ/LfXf7zEAKhKeWa4q1zw= =WZkk -----END PGP SIGNATURE----- --TBNym+cBXeFsS4Vs--