From mboxrd@z Thu Jan 1 00:00:00 1970 From: =?utf-8?Q?Ludovic_Court=C3=A8s?= Subject: Grafts, revisited Date: Wed, 25 Mar 2020 16:23:18 +0100 Message-ID: <87k138ihll.fsf@inria.fr> Mime-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha256; protocol="application/pgp-signature" Return-path: Received: from eggs.gnu.org ([2001:470:142:3::10]:42843) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jH7sM-0007yF-2o for guix-devel@gnu.org; Wed, 25 Mar 2020 11:23:27 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:42052) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1jH7sL-0005MA-V6 for guix-devel@gnu.org; Wed, 25 Mar 2020 11:23:26 -0400 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=35392 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1jH7sJ-0006dF-L1 for guix-devel@gnu.org; Wed, 25 Mar 2020 11:23:25 -0400 List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+gcggd-guix-devel=m.gmane-mx.org@gnu.org Sender: "Guix-devel" To: guix-devel@gnu.org --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Hello Guix! I=E2=80=99ve just pushed a pretty cool hack to =E2=80=98wip-build-accumulat= or=E2=80=99. :-) Grafts are a function of the references of a package. Suppose Bash has a =E2=80=98replacement=E2=80=99 field. If a package has a build-time depen= dency on Bash, it=E2=80=99s a candidate for being grafted the Bash replacement. How= ever, if that package has no reference on Bash (i.e., =E2=80=9Cguix gc --referenc= es $(guix build the-package --no-grafts) | grep bash=E2=80=9D is empty), then = we don=E2=80=99t graft it. This reduces the number of grafts needed, and thus saves CPU time (fewer grafts derivations built) and space (fewer grafted variants in the store). Because grafting depends on build results, they constitute =E2=80=9Cdynamic dependencies=E2=80=9D, using the terminology of =E2=80=9CBuild Systems =C3= =A0 la Carte=E2=80=9D. Currently, grafts are handled in an ad-hoc fashion: we use substitute info to try and determine ahead of time the references of a build output if it=E2=80=99s unavailable locally. When substitute info is available, th= at allows us to display the build plan upfront. When substitute info is unavailable, we cannot display the build plan upfront anyway, hence the need for =E2=80=98with-build-handler=E2=80=99=C2= =B9. Additionally, when substitute info is available, we can end up looking up substitutes one by one, leading to repeated =E2=80=9Cupdating the list of substitutes= =E2=80=9D messages and related slowness=C2=B9 (this is particularly visible when /var/guix/substitute/cache is empty.) To address that, I think we need a better way to handle =E2=80=9Cdynamic dependencies=E2=80=9D. The =E2=80=98wip-build-accumulator=E2=80=99 branch = does that by taking advantage of =E2=80=98with-build-handler=E2=80=99. On that branch, grafts = no longer use substitute info. Instead, they just build missing store items and get their references. To avoid building things one by one, we install a build handler that =E2=80=9Caccumulates=E2=80=9D the list of =E2=80=98build= -things=E2=80=99 requests; eventually, we build all these things at once and resume the continuations of the =E2=80=98build-things=E2=80=99 calls. The goal here i= s to improve efficiency and to allow the UI to shows these stages in a meaningful way. Here=E2=80=99s a sample session (slightly edited for clarity) =2D-8<---------------cut here---------------start------------->8--- ludo@ribbon ~/src/guix$ rm /tmp/foo* ludo@ribbon ~/src/guix$ guix gc -D $(./pre-inst-env guix build vim-full zil= e-on-guile --no-grafts) $(./pre-inst-env guix build vim-full zile-on-guile) ludo@ribbon ~/src/guix$ ./pre-inst-env guix install zile-on-guile vim-full = -p /tmp/foo -n The following packages would be installed: zile-on-guile 2.4.14-0.fd09781 vim-full 8.2.0411 9.4 MB would be downloaded: /gnu/store/vf7w4yiax38ra7x8aqqvbnc38c0ldgpm-zile-on-guile-2.4.14-0.fd097= 81 /gnu/store/dnj9wljcck9vdwgp7dwxk00qnnk1g3c5-vim-full-8.2.0411 ludo@ribbon ~/src/guix$ ./pre-inst-env guix install zile-on-guile vim-full = -p /tmp/foo=20 The following packages will be installed: zile-on-guile 2.4.14-0.fd09781 vim-full 8.2.0411 9.4 MB will be downloaded: /gnu/store/vf7w4yiax38ra7x8aqqvbnc38c0ldgpm-zile-on-guile-2.4.14-0.fd097= 81 /gnu/store/dnj9wljcck9vdwgp7dwxk00qnnk1g3c5-vim-full-8.2.0411 downloading from https://ci.guix.gnu.org/nar/lzip/dnj9wljcck9vdwgp7dwxk00qn= nk1g3c5-vim-full-8.2.0411... vim-full-8.2.0411 8.9MiB 7= .6MiB/s 00:01 [##################] 100.0% downloading from https://ci.guix.gnu.org/nar/lzip/vf7w4yiax38ra7x8aqqvbnc38= c0ldgpm-zile-on-guile-2.4.14-0.fd09781... zile-on-guile-2.4.14-0.fd09781 140KiB 1= .8MiB/s 00:00 [##################] 100.0% The following derivation will be built: /gnu/store/d9xms78917w67xq71pqsx5x9s6dmq6d7-profile.drv The following graft will be made: /gnu/store/4n6dmg6iwjg0adpcvqygr9wgsnclswss-vim-full-8.2.0411.drv applying 8 grafts for /gnu/store/4n6dmg6iwjg0adpcvqygr9wgsnclswss-vim-full-= 8.2.0411.drv... building /gnu/store/d9xms78917w67xq71pqsx5x9s6dmq6d7-profile.drv... =2D-8<---------------cut here---------------end--------------->8--- Here, with =E2=80=98-n=E2=80=99, we see the main tasks (downloads). Withou= t =E2=80=98-n=E2=80=99, we see again these two downloads first, and then a second stage showing the graft and profile derivations. What if we remove =E2=80=98git-minimal=E2=80=99 from the store, which is it= self required to build the source of some nodes on the graph? =2D-8<---------------cut here---------------start------------->8--- ludo@ribbon ~/src/guix$ rm /tmp/foo* ludo@ribbon ~/src/guix$ guix gc -D $(./pre-inst-env guix build git-minimal = vim-full zile-on-guile --no-grafts) $(./pre-inst-env guix build git-minimal= vim-full zile-on-guile) ludo@ribbon ~/src/guix$ ./pre-inst-env guix install zile-on-guile vim-full = -p /tmp/foo -n The following packages would be installed: zile-on-guile 2.4.14-0.fd09781 vim-full 8.2.0411 4.9 MB would be downloaded: /gnu/store/wwjhip4wvhjyfj6fs7936bbsgd4yax6g-git-minimal-2.26.0 ludo@ribbon ~/src/guix$ ./pre-inst-env guix install zile-on-guile vim-full = -p /tmp/foo=20 The following packages will be installed: zile-on-guile 2.4.14-0.fd09781 vim-full 8.2.0411 4.9 MB will be downloaded: /gnu/store/wwjhip4wvhjyfj6fs7936bbsgd4yax6g-git-minimal-2.26.0 downloading from https://ci.guix.gnu.org/nar/lzip/wwjhip4wvhjyfj6fs7936bbsg= d4yax6g-git-minimal-2.26.0... git-minimal-2.26.0 4.6MiB 7= .0MiB/s 00:01 [##################] 100.0% 9.4 MB will be downloaded: /gnu/store/vf7w4yiax38ra7x8aqqvbnc38c0ldgpm-zile-on-guile-2.4.14-0.fd097= 81 /gnu/store/dnj9wljcck9vdwgp7dwxk00qnnk1g3c5-vim-full-8.2.0411 downloading from https://ci.guix.gnu.org/nar/lzip/dnj9wljcck9vdwgp7dwxk00qn= nk1g3c5-vim-full-8.2.0411... vim-full-8.2.0411 8.9MiB 7= .6MiB/s 00:01 [##################] 100.0% downloading from https://ci.guix.gnu.org/nar/lzip/vf7w4yiax38ra7x8aqqvbnc38= c0ldgpm-zile-on-guile-2.4.14-0.fd09781... zile-on-guile-2.4.14-0.fd09781 140KiB 1= .8MiB/s 00:00 [##################] 100.0% The following derivation will be built: /gnu/store/d9xms78917w67xq71pqsx5x9s6dmq6d7-profile.drv The following graft will be made: /gnu/store/4n6dmg6iwjg0adpcvqygr9wgsnclswss-vim-full-8.2.0411.drv applying 8 grafts for /gnu/store/4n6dmg6iwjg0adpcvqygr9wgsnclswss-vim-full-= 8.2.0411.drv... building /gnu/store/d9xms78917w67xq71pqsx5x9s6dmq6d7-profile.drv... =2D-8<---------------cut here---------------end--------------->8--- This time we see three phases. The dry run is a bit confusing because it only shows the first of these phases (on the branch =E2=80=98--dry-run= =E2=80=99 no longer implies =E2=80=98--no-grafts=E2=80=99, but that=E2=80=99s an optiona= l change.) You can give it a spin and report on the UX by doing: guix pull --branch=3Dwip-build-accumulator -p /tmp/test /tmp/test/bin/guix =E2=80=A6 Thoughts? The insight here is that the call graph mirrors the dependency graph. Capturing the continuation when we encounter a =E2=80=98build-things=E2=80= =99 call is akin to modeling dynamic dependencies in the graph and implementing backtracking. I haven=E2=80=99t measured the cost of all this but it=E2=80=99s not notice= able so far (and it very much improves the situation by avoiding repeated substitute fetches!). Thanks, Ludo=E2=80=99. =C2=B9 https://issues.guix.gnu.org/issue/40130 =C2=B2 https://issues.guix.gnu.org/issue/22990 --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEEPORkVYqE/cadtAz7CQsRmT2a67UFAl57d2YACgkQCQsRmT2a 67XNYg//Y16qN+3x19QFLVcA/RcBCYVUXcS7yhLzvc5WoeEi15U5iZsDkSuE0O/o c7GTzlhbk7BRIyF4SuZy/tqLADgHDRwHpOl0DzaDsaHC7iAnqxymLDDHay1CGlOJ 16zUDicdH++4j0oeM3IzBoj+EFW6Yg21ISGPYVVx+8uN/Oh3pd69oFcS+8vs+sOI WOe44rQxUrYz5HqXSfIJK4ulaqrPcPJs5iI9TPd/+XyVmzyWaGL8HyU9YoJViz+z NxXvAkP/7dxat2FuBED2rKp8I49xSE9VR+jg8eKrLO2rhiwq4WbYuqI9+xO77AzA eBA6aTgQ4SA4Vh4rghVc9ou4ZtOy9nqQew2CeOB0bh1bY8397B2bNmoW9aWNx0lk 8aJGBv6GjqnMOVyOXTqAxWHg5y4okk0siEl7/RW5caIc18/NzIvmhgl90eG/qCYC yzZ5W2rrizqlqE+TSPh+PxdNJ7AH0rdAEM++r0AV9V2eYBXXKce2+jqh+xy1ECAa 90+jaEm52u7eVR2NwUPpxBFpbI9txJVocwOi3OTiu/6NX8DXPe+ShPknAm4dZBJW e1p0SFwzONaU/RFrlEfj0y+CeMyOcHtB1F+5K/JNPNbuHsU+vWVsvyxdEUXIQksU aP/keK16A2fjbnzNS+HxQeUKnoJVRY+edyu6HknOjTCcY2dFMug= =n4Ar -----END PGP SIGNATURE----- --=-=-=--