bug#27284: ‘guix pull’ builds using multiple derivations

[Nils Gillmann <ng0@n0.is>]

Awesome news!

Ludovic Courtès transcribed 3.1K bytes:
> Hello Guix!
> 
> News from the front!  I’ve pushed a new ‘wip-pull-multiple-derivations’
> branch that keeps the same approach as previously proposed (building
> Guix using multiple derivation, one for each group of modules: core,
> CLI, packages, etc.), but does not attempt to reload modules in the
> running Guile process, which proved to be too tricky.
> 
> You can already test with:
> 
>   guix pull --branch=wip-pull-multiple-derivations
>
> Feedback welcome!

I think you forgot to push it or you are on a very slow connection :)
 
> This patch set actually addresses a bootstrapping issue: you need Guix
> to build Guix.  So far ‘guix pull’ would sidestep the bootstrapping
> issue by building Guix with whatever is available in the currently
> deployed Guix; for instance, it would use the ‘guile’, ‘guile-json’,
> etc. packages from the current Guix.  The problem of that approach was
> that it’s stateful: the result depends on what you currently have.
> Sometimes what you have is too old, or lacks some package definition,
> and you can’t really go forward.
> 
> The code in ‘wip-pull-multiple-derivations’ works like this:
> 
>   1. Assume we have Guile and Guix already installed, but not
>      necessarily the latest versions thereof.
> 
>   2. Using the (guix …) modules that we have, build a program—a
>      “trampoline”—that will use the modules of the target Guix (the
>      commit we want to pull) to compute the derivation of that Guix.
> 
>   3. Run that trampoline, which returns /gnu/store/…-guix.drv.  The
>      result should be the same regardless of the initial Guix because
>      the trampoline uses exclusively modules from the target Guix.
> 
>   4. ‘guix pull’ builds that derivation (actually the branch does not
>      modify (guix scripts pull) at all; everything is in
>      build-aux/build-self.scm.)
> 
> In step #4, we should be able to get substitutes for at least some of
> the derivations.
> 
> To build the trampoline in step #2, we first need to build a bunch of
> modules from the target Guix.  Hopefully you don’t have to rebuild them
> at each pull, but it can take a minute or so, and you may not have
> substitutes for that (because this part is stateful.)
> 
> In step #3, the trampoline has compiled code for the core modules, but
> it still has to interpret (gnu packages guile) and related modules,
> because these are not compiled.  The program runs in ~40s on my laptop.
> To make it faster, we could reduce the closure of (gnu packages guile)¹,
> but that can be tricky.  Or we could optimize Guile itself; I’m sure the
> compiler and/or interpreter could do better.
> 
> In that branch, you can also run:
> 
>   make as-derivation
> 
> and it will build Guix from $srcdir in the way described above.
> 
> To summarize, performance is not great, but hopefully it’ll be slightly
> better than what we have now, especially with substitutes.
> Correctness/reproducibility are greatly improved.
> 
> Ludo’.
> 
> ¹ The closure of (gnu packages guile) contains python, haskell-check,
>   music, and many other surprising things:
>   <http://web.fdn.fr/~lcourtes/tmp/gnu-packages-guile.html>.
> 
> 
>