From mboxrd@z Thu Jan 1 00:00:00 1970 From: Maxim Cournoyer Subject: Speeding up 'make check-system' Date: Tue, 11 Feb 2020 11:51:19 -0500 Message-ID: <87a75p2hko.fsf@gmail.com> Mime-Version: 1.0 Content-Type: multipart/signed; boundary="==-=-="; micalg=pgp-sha256; protocol="application/pgp-signature" Return-path: Received: from eggs.gnu.org ([2001:470:142:3::10]:59189) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1j1Ykt-00068F-P0 for guix-devel@gnu.org; Tue, 11 Feb 2020 11:51:25 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1j1Yks-0003f3-7z for guix-devel@gnu.org; Tue, 11 Feb 2020 11:51:23 -0500 Received: from mail-qt1-x82e.google.com ([2607:f8b0:4864:20::82e]:37313) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1j1Ykr-0003ec-Vm for guix-devel@gnu.org; Tue, 11 Feb 2020 11:51:22 -0500 Received: by mail-qt1-x82e.google.com with SMTP id w47so8447461qtk.4 for ; Tue, 11 Feb 2020 08:51:21 -0800 (PST) Received: from kwak ([2607:fad8:4:6:afc9:fe0d:91fc:113b]) by smtp.gmail.com with ESMTPSA id m10sm2252157qki.74.2020.02.11.08.51.20 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 11 Feb 2020 08:51:20 -0800 (PST) List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+gcggd-guix-devel=m.gmane-mx.org@gnu.org Sender: "Guix-devel" To: guix-devel --==-=-= Content-Type: multipart/mixed; boundary="=-=-=" --=-=-= Content-Type: text/x-patch Content-Disposition: attachment; filename=0001-gnu-tests-Reduce-the-time-required-to-run-the-system.patch Content-Transfer-Encoding: quoted-printable From=2082d3266d42390b67341ed5b0a5cc880d0ba3ed77 Mon Sep 17 00:00:00 2001 From: Maxim Cournoyer Date: Sun, 17 Nov 2019 06:01:00 +0900 Subject: [PATCH] gnu: tests: Reduce the time required to run the system tes= ts. When setting the GUIX_DEV_HACKS environment variable, the Guix package used inside the instrumented VMs recycles the binaries already found in the Guix checkout of the developer instead of rebuilding Guix from scratch. This brings the time required for this component from 20+ minutes down to 2-3 minutes on an X200 machine. * gnu/packages/package-management.scm (current-guix/pre-built): New procedu= re. * build-aux/run-system-tests.scm (tests-for-channel-instance): Use it, when GUIX_DEV_HACKS is defined. =2D-- build-aux/run-system-tests.scm | 10 ++++- gnu/packages/package-management.scm | 66 +++++++++++++++++++++++++++++ 2 files changed, 75 insertions(+), 1 deletion(-) diff --git a/build-aux/run-system-tests.scm b/build-aux/run-system-tests.scm index b0cb3bd2bf..e6fda0546b 100644 =2D-- a/build-aux/run-system-tests.scm +++ b/build-aux/run-system-tests.scm @@ -58,8 +58,16 @@ instance." ;; of tests to run in the usual way: ;; ;; make check-system TESTS=3Dinstalled-os + + ;; When the GUIX_DEV_HACKS environment variable is defined, override the + ;; package returned by `current-guix' with a flavor that saves recompili= ng + ;; Guix from scratch and reuse the developer's checkout binaries. The + ;; override "builds" about 20 times faster than the regular `current-gui= x' + ;; package, which can help speed iterative development. (parameterize ((current-guix-package =2D (channel-instance->package instance))) + (if (getenv "GUIX_DEV_HACKS") + (current-guix/pre-built) + (channel-instance->package instance)))) (match (getenv "TESTS") (#f (all-system-tests)) diff --git a/gnu/packages/package-management.scm b/gnu/packages/package-man= agement.scm index 422d4f1959..bd2ed85189 100644 =2D-- a/gnu/packages/package-management.scm +++ b/gnu/packages/package-management.scm @@ -468,6 +468,72 @@ out) and returning a package that uses that as its 'so= urce'." #:recursive? #t #:select? (force select?)))))))) =20 +(define-public (current-guix/pre-built) + "Similar to `current-guix', but with a modified build procedure that +reuses the existing byte compiled artifacts to save recompilation time." + + (let* ( ;; The `current-source-directory' macro doesn't work from the RE= PL. + ;; For testing, you can replace it with a static string pointing = to + ;; your Guix checkout directory. + (repository-root (delay (canonicalize-path + (string-append (current-source-directory) + "/../..")))) + (select? (lambda (file stat) + (match (basename file) + ((or ".git" + "configure" "autom4te.cache" + "config.log" "config.status" + "stamp-1" "stamp-2" "stamp-3" "stamp-4" "stamp-= 5" + "stamp-h1" "stamp-vti" + "Makefile" "Makefile.in" ".libs" + ".deps" ".dirstamp" + "test-tmp" + ) #f) + (_ #t))))) + (package + (inherit guix) + (version (string-append (package-version guix) "+")) + (source (local-file (force repository-root) "guix-current" + #:recursive? #t + #:select? select?)) + (arguments + (substitute-keyword-arguments (package-arguments guix) + ((#:phases phases) + `(modify-phases ,phases + ;; XXX: References to tools such as 'mkdir' and 'install' are + ;; captured in Makefile.in when 'autoconf' is run. It'd be n= icer + ;; to find those at configuration time. + (delete 'copy-bootstrap-guile) + (delete 'check) + (delete 'disable-failing-tests) + (delete 'strip) ;can't strip .go files anyway + (replace 'build + (lambda _ + ;; Set the write permission bit on some files that need t= o be + ;; touched. + (chmod "nix" #o777) + (for-each (lambda (f) + (chmod f #o666)) + (cons* "guix-daemon" + (find-files "." ".*\\.(a|o)$"))) + + ;; The following prevent 'make install' from rebuilding t= he + ;; daemon and the documentation. + (invoke "make" "--touch" "info" + ;; TODO: Currently we must rebuild the daemon as = it + ;; was linked against external dependencies that + ;; depend on the provenance of the profile (or + ;; environment) that was used to build it. + + ;; If we could query the provenance of any profil= e, + ;; we could make this package inherit from the gu= ix + ;; inferior that was used to provide such + ;; dependencies. The most reliable way would + ;; probably be to record that provenance at build + ;; time (as a make target). + ;"guix-daemon" + )))))))))) + ;;; ;;; Other tools. =2D-=20 2.23.0 --=-=-= Content-Type: text/plain Content-Transfer-Encoding: quoted-printable Hello Guix! The attached patch aims to speed up iterative development with 'make check-system', by reusing most of the compiled objects from the Guix checkout you are developing from (which needs to be built anyway for 'make check' to run). Why most and not all? Because the C++ daemon refers (links) to external libraries that were captured at the time it was built, and are most likely different from those inherited from the Guix package defined in your checkout. A solution would be have the "current-guix/pre-built" package inherit from the *inferior* of the guix package that was used to build the profile in which the deamon was actually built. I'm punting on that one because there doesn't seem to be a robust way to query the provenance of a profile ('guix environment' generates a profile without said information, for example). Anyway, recompiling the daemon only vs the whole Guix .go files already cuts down the time nicely from 20+ minutes to a couple minutes or so, on a dated X200 machine, which helps when iterating on Guix code changes only. To benchmark the build of just the current-guix/pre-built package, one can use: =2D-8<---------------cut here---------------start------------->8--- time ./pre-inst-env guix build -e '((@ (gnu packages package-management) cu= rrent-guix/pre-built))' =2D-8<---------------cut here---------------end--------------->8--- To try using that package in an actual system test, you could use: =2D-8<---------------cut here---------------start------------->8--- GUIX_DEV_HACKS=3D1 make check-system TESTS=3Dbasic =2D-8<---------------cut here---------------end--------------->8--- or just 'make check-system', if you have lots of time on your hands :-) Thoughts? Maxim --=-=-=-- --==-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEEJ9WGpPiQCFQyn/CfEmDkZILmNWIFAl5C24cACgkQEmDkZILm NWLa1w//WCjb0R7P2EVYxUaowP6a3D2FgySdxe7Kh46h+KxQ1D6RFv56rier6JIB x8LO8S4MEHrm2c0vatrGzJDzWTITyJbr89WT+btX9E7zbleaWCo3tokiL5x1R3x2 YY1XUYfqgsraKoM8DtOcKItm+d9gnZo75tluw+0lFhGMWLgUOf+XmM96dZNtC+Om QT39dusVdN6EJUGwjrGCyYUkQfyYQkQFa4/zLKZ7WZfRBC+uFrHFHXv8y0VfBXQO 6K09ZCRW0uk+ahAHno4cJ+zsUk5+YiqLY0Cdg9e8X5ZS9dK8xrnzaGU3rxJ2emUf CpnH2ShtmcMrN8LBNohZV/ZB0yqjmTiBA2RuUYh8zdC4PL8bCDSPu/iYLx3RbtSE Rdta6a/buZkMqGErPKOu8acJH0NA8wKzsEejM1WOsx7Dhsk5W0ur6gPj+jlTiv+W vvp4NFcnd1FtYIDlK5jsgZvZKwrWM1sutTHDJ3t8g6P3wqnSaoVCa0Ww070vtx3M 5Jn/C9GFJofgnUAGfU2aWftl5D3esv6DyADrjqhUNc8FXjuZNhxQhryT27mTvJ36 NliyiQvFVNYteGHq1yTbCJ2V51gTVaRQVBBNE4AZGsfh3jlHWxsljXfyg0cx09mv 5fEUgfeN3yry6Caep/+K20MWu/UMg2zfZy5WnQlJxkUHysmDdLs= =di/S -----END PGP SIGNATURE----- --==-=-=--