bug#27284: ‘guix pull’ builds using multiple derivations

[ludo@gnu.org (Ludovic Courtès)]

Hello Guix!

News from the front!  I’ve pushed a new ‘wip-pull-multiple-derivations’
branch that keeps the same approach as previously proposed (building
Guix using multiple derivation, one for each group of modules: core,
CLI, packages, etc.), but does not attempt to reload modules in the
running Guile process, which proved to be too tricky.

You can already test with:

  guix pull --branch=wip-pull-multiple-derivations

Feedback welcome!

This patch set actually addresses a bootstrapping issue: you need Guix
to build Guix.  So far ‘guix pull’ would sidestep the bootstrapping
issue by building Guix with whatever is available in the currently
deployed Guix; for instance, it would use the ‘guile’, ‘guile-json’,
etc. packages from the current Guix.  The problem of that approach was
that it’s stateful: the result depends on what you currently have.
Sometimes what you have is too old, or lacks some package definition,
and you can’t really go forward.

The code in ‘wip-pull-multiple-derivations’ works like this:

  1. Assume we have Guile and Guix already installed, but not
     necessarily the latest versions thereof.

  2. Using the (guix …) modules that we have, build a program—a
     “trampoline”—that will use the modules of the target Guix (the
     commit we want to pull) to compute the derivation of that Guix.

  3. Run that trampoline, which returns /gnu/store/…-guix.drv.  The
     result should be the same regardless of the initial Guix because
     the trampoline uses exclusively modules from the target Guix.

  4. ‘guix pull’ builds that derivation (actually the branch does not
     modify (guix scripts pull) at all; everything is in
     build-aux/build-self.scm.)

In step #4, we should be able to get substitutes for at least some of
the derivations.

To build the trampoline in step #2, we first need to build a bunch of
modules from the target Guix.  Hopefully you don’t have to rebuild them
at each pull, but it can take a minute or so, and you may not have
substitutes for that (because this part is stateful.)

In step #3, the trampoline has compiled code for the core modules, but
it still has to interpret (gnu packages guile) and related modules,
because these are not compiled.  The program runs in ~40s on my laptop.
To make it faster, we could reduce the closure of (gnu packages guile)¹,
but that can be tricky.  Or we could optimize Guile itself; I’m sure the
compiler and/or interpreter could do better.

In that branch, you can also run:

  make as-derivation

and it will build Guix from $srcdir in the way described above.

To summarize, performance is not great, but hopefully it’ll be slightly
better than what we have now, especially with substitutes.
Correctness/reproducibility are greatly improved.

Ludo’.

¹ The closure of (gnu packages guile) contains python, haskell-check,
  music, and many other surprising things:
  <http://web.fdn.fr/~lcourtes/tmp/gnu-packages-guile.html>.