bug#27284: ‘guix pull’ builds using multiple derivations

[ludo@gnu.org (Ludovic Courtès)]

Hello!

ludo@gnu.org (Ludovic Courtès) skribis:

> The code in ‘wip-pull-multiple-derivations’ works like this:
>
>   1. Assume we have Guile and Guix already installed, but not
>      necessarily the latest versions thereof.
>
>   2. Using the (guix …) modules that we have, build a program—a
>      “trampoline”—that will use the modules of the target Guix (the
>      commit we want to pull) to compute the derivation of that Guix.
>
>   3. Run that trampoline, which returns /gnu/store/…-guix.drv.  The
>      result should be the same regardless of the initial Guix because
>      the trampoline uses exclusively modules from the target Guix.
>
>   4. ‘guix pull’ builds that derivation (actually the branch does not
>      modify (guix scripts pull) at all; everything is in
>      build-aux/build-self.scm.)

After quite a bit of tweaking I’ve finally pushed this patch series to
master.  So if you run ‘guix pull’ now, you’ll get the new
multiple-derivation build.

There’s still room for improvement, in particular:

  • The initial ‘compute-guix-derivation’ program depends on quite a few
    modules, so the first time you ‘guix pull’ you build a
    module-import-compiled.drv that takes a while and does not produce
    any output (I’ve made it verbose in core-updates commit
    d32922759bfeffa03ee189158ea00b1a0ddbe8c6 though).  Perhaps we could
    avoid that by using a double trampoline, which would allow us to get
    substitutes for all the non-trivial bits.

  • The “guix-packages” derivation is coarse-grain.  We could perhaps
    automatically split it in clusters of modules by analyzing the (gnu
    packages …) module graph.

  • The compiler (and interpreter) in Guile 2.2 is still not as fast as
    we’d like.  I think it should be possible to make it as fast as in
    2.0, at least when optimizations are turned off.  Andy made
    significant improvements in 2.2.3, but hopefully we can still
    improve on that.  The thread at
    <https://lists.gnu.org/archive/html/guile-devel/2017-10/msg00035.html>
    contains some info.

Normally berlin.guixsd.org will start providing substitutes for all this
but note that, as things are, it’ll compute substitutes for Guile 2.2.2
or 2.2.3, and that’s it.  hydra.gnu.org currently doesn’t, for obscure
reasons (‘hydra-eval-guile-jobs’ systematically adds “.” to the load
path, which is not what we want for build-aux/hydra/guix-modular.scm.)

This works unlocks some of the stuff in <https://bugs.gnu.org/22629>, so
I’ll try to focus on that now.

Feedback and bug reports welcome!

Ludo’.