Grafts, revisited

Grafts, revisited

[Ludovic Courtès]

[-- Attachment #1: Type: text/plain, Size: 7428 bytes --]

Hello Guix!

I’ve just pushed a pretty cool hack to ‘wip-build-accumulator’.  :-)

Grafts are a function of the references of a package.  Suppose Bash has
a ‘replacement’ field.  If a package has a build-time dependency on
Bash, it’s a candidate for being grafted the Bash replacement.  However,
if that package has no reference on Bash (i.e., “guix gc --references
$(guix build the-package --no-grafts) | grep bash” is empty), then we
don’t graft it.  This reduces the number of grafts needed, and thus
saves CPU time (fewer grafts derivations built) and space (fewer grafted
variants in the store).

Because grafting depends on build results, they constitute “dynamic
dependencies”, using the terminology of “Build Systems à la Carte”.

Currently, grafts are handled in an ad-hoc fashion: we use substitute
info to try and determine ahead of time the references of a build output
if it’s unavailable locally.  When substitute info is available, that
allows us to display the build plan upfront.

When substitute info is unavailable, we cannot display the build plan
upfront anyway, hence the need for ‘with-build-handler’¹.  Additionally,
when substitute info is available, we can end up looking up substitutes
one by one, leading to repeated “updating the list of substitutes”
messages and related slowness¹ (this is particularly visible when
/var/guix/substitute/cache is empty.)

To address that, I think we need a better way to handle “dynamic
dependencies”.  The ‘wip-build-accumulator’ branch does that by taking
advantage of ‘with-build-handler’.  On that branch, grafts no longer use
substitute info.  Instead, they just build missing store items and get
their references.  To avoid building things one by one, we install a
build handler that “accumulates” the list of ‘build-things’ requests;
eventually, we build all these things at once and resume the
continuations of the ‘build-things’ calls.  The goal here is to improve
efficiency and to allow the UI to shows these stages in a meaningful
way.  Here’s a sample session (slightly edited for clarity)

--8<---------------cut here---------------start------------->8---
ludo@ribbon ~/src/guix$ rm /tmp/foo*
ludo@ribbon ~/src/guix$ guix gc -D $(./pre-inst-env guix build vim-full zile-on-guile --no-grafts) $(./pre-inst-env guix build vim-full zile-on-guile)
ludo@ribbon ~/src/guix$ ./pre-inst-env guix install zile-on-guile vim-full -p /tmp/foo -n
The following packages would be installed:
   zile-on-guile 2.4.14-0.fd09781
   vim-full      8.2.0411

9.4 MB would be downloaded:
   /gnu/store/vf7w4yiax38ra7x8aqqvbnc38c0ldgpm-zile-on-guile-2.4.14-0.fd09781
   /gnu/store/dnj9wljcck9vdwgp7dwxk00qnnk1g3c5-vim-full-8.2.0411
ludo@ribbon ~/src/guix$ ./pre-inst-env guix install zile-on-guile vim-full -p /tmp/foo 
The following packages will be installed:
   zile-on-guile 2.4.14-0.fd09781
   vim-full      8.2.0411

9.4 MB will be downloaded:
   /gnu/store/vf7w4yiax38ra7x8aqqvbnc38c0ldgpm-zile-on-guile-2.4.14-0.fd09781
   /gnu/store/dnj9wljcck9vdwgp7dwxk00qnnk1g3c5-vim-full-8.2.0411
downloading from https://ci.guix.gnu.org/nar/lzip/dnj9wljcck9vdwgp7dwxk00qnnk1g3c5-vim-full-8.2.0411...
 vim-full-8.2.0411  8.9MiB                                                7.6MiB/s 00:01 [##################] 100.0%

downloading from https://ci.guix.gnu.org/nar/lzip/vf7w4yiax38ra7x8aqqvbnc38c0ldgpm-zile-on-guile-2.4.14-0.fd09781...
 zile-on-guile-2.4.14-0.fd09781  140KiB                                   1.8MiB/s 00:00 [##################] 100.0%

The following derivation will be built:
   /gnu/store/d9xms78917w67xq71pqsx5x9s6dmq6d7-profile.drv
The following graft will be made:
   /gnu/store/4n6dmg6iwjg0adpcvqygr9wgsnclswss-vim-full-8.2.0411.drv
applying 8 grafts for /gnu/store/4n6dmg6iwjg0adpcvqygr9wgsnclswss-vim-full-8.2.0411.drv...
building /gnu/store/d9xms78917w67xq71pqsx5x9s6dmq6d7-profile.drv...
--8<---------------cut here---------------end--------------->8---

Here, with ‘-n’, we see the main tasks (downloads).  Without ‘-n’, we
see again these two downloads first, and then a second stage showing the
graft and profile derivations.

What if we remove ‘git-minimal’ from the store, which is itself required
to build the source of some nodes on the graph?

--8<---------------cut here---------------start------------->8---
ludo@ribbon ~/src/guix$ rm /tmp/foo*
ludo@ribbon ~/src/guix$ guix gc -D $(./pre-inst-env guix build git-minimal vim-full zile-on-guile --no-grafts) $(./pre-inst-env guix build git-minimal vim-full zile-on-guile)
ludo@ribbon ~/src/guix$ ./pre-inst-env guix install zile-on-guile vim-full -p /tmp/foo -n
The following packages would be installed:
   zile-on-guile 2.4.14-0.fd09781
   vim-full      8.2.0411

4.9 MB would be downloaded:
   /gnu/store/wwjhip4wvhjyfj6fs7936bbsgd4yax6g-git-minimal-2.26.0
ludo@ribbon ~/src/guix$ ./pre-inst-env guix install zile-on-guile vim-full -p /tmp/foo 
The following packages will be installed:
   zile-on-guile 2.4.14-0.fd09781
   vim-full      8.2.0411

4.9 MB will be downloaded:
   /gnu/store/wwjhip4wvhjyfj6fs7936bbsgd4yax6g-git-minimal-2.26.0
downloading from https://ci.guix.gnu.org/nar/lzip/wwjhip4wvhjyfj6fs7936bbsgd4yax6g-git-minimal-2.26.0...
 git-minimal-2.26.0  4.6MiB                                               7.0MiB/s 00:01 [##################] 100.0%

9.4 MB will be downloaded:
   /gnu/store/vf7w4yiax38ra7x8aqqvbnc38c0ldgpm-zile-on-guile-2.4.14-0.fd09781
   /gnu/store/dnj9wljcck9vdwgp7dwxk00qnnk1g3c5-vim-full-8.2.0411
downloading from https://ci.guix.gnu.org/nar/lzip/dnj9wljcck9vdwgp7dwxk00qnnk1g3c5-vim-full-8.2.0411...
 vim-full-8.2.0411  8.9MiB                                                7.6MiB/s 00:01 [##################] 100.0%

downloading from https://ci.guix.gnu.org/nar/lzip/vf7w4yiax38ra7x8aqqvbnc38c0ldgpm-zile-on-guile-2.4.14-0.fd09781...
 zile-on-guile-2.4.14-0.fd09781  140KiB                                   1.8MiB/s 00:00 [##################] 100.0%

The following derivation will be built:
   /gnu/store/d9xms78917w67xq71pqsx5x9s6dmq6d7-profile.drv
The following graft will be made:
   /gnu/store/4n6dmg6iwjg0adpcvqygr9wgsnclswss-vim-full-8.2.0411.drv
applying 8 grafts for /gnu/store/4n6dmg6iwjg0adpcvqygr9wgsnclswss-vim-full-8.2.0411.drv...
building /gnu/store/d9xms78917w67xq71pqsx5x9s6dmq6d7-profile.drv...
--8<---------------cut here---------------end--------------->8---

This time we see three phases.  The dry run is a bit confusing because
it only shows the first of these phases (on the branch ‘--dry-run’ no
longer implies ‘--no-grafts’, but that’s an optional change.)

You can give it a spin and report on the UX by doing:

  guix pull --branch=wip-build-accumulator -p /tmp/test
  /tmp/test/bin/guix …

Thoughts?

The insight here is that the call graph mirrors the dependency graph.
Capturing the continuation when we encounter a ‘build-things’ call is
akin to modeling dynamic dependencies in the graph and implementing
backtracking.

I haven’t measured the cost of all this but it’s not noticeable so far
(and it very much improves the situation by avoiding repeated substitute
fetches!).

Thanks,
Ludo’.

¹ https://issues.guix.gnu.org/issue/40130
² https://issues.guix.gnu.org/issue/22990

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: Grafts, revisited

[Ludovic Courtès]

Hello Guix!

Ludovic Courtès <ludo@gnu.org> skribis:

> To address that, I think we need a better way to handle “dynamic
> dependencies”.  The ‘wip-build-accumulator’ branch does that by taking
> advantage of ‘with-build-handler’.  On that branch, grafts no longer use
> substitute info.  Instead, they just build missing store items and get
> their references.  To avoid building things one by one, we install a
> build handler that “accumulates” the list of ‘build-things’ requests;
> eventually, we build all these things at once and resume the
> continuations of the ‘build-things’ calls.  The goal here is to improve
> efficiency and to allow the UI to shows these stages in a meaningful
> way.  Here’s a sample session (slightly edited for clarity)

I went ahead and merged ‘wip-build-accumulator’ in master because it’s
better than the status quo.  This fixes this 3.5-year old bug:

  https://issues.guix.gnu.org/issue/22990

Woohoo!

If anything looks wrong or otherwise “weird”, please report a bug!
If you feel happier with this change, enjoy!  :-)

Ludo’.

Re: Grafts, revisited

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 83 bytes --]

This is great news, thanks a lot!

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]