From mboxrd@z Thu Jan 1 00:00:00 1970 From: Nils Gillmann Subject: bug#27284: =?UTF-8?Q?=E2=80=98guix_?= =?UTF-8?Q?pull=E2=80=99?= builds using multiple derivations Date: Tue, 27 Mar 2018 19:25:18 +0000 Message-ID: <20180327192518.rynrei7mr3qk3ocl@abyayala> References: <87h8tnz1dx.fsf@gnu.org> <20171211105253.9670-1-ludo@gnu.org> <20171211105253.9670-4-ludo@gnu.org> <871sjst76a.fsf@gnu.org> <87d0zpyj5d.fsf_-_@gnu.org> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:37679) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1f0uEP-0003tu-Hk for bug-guix@gnu.org; Tue, 27 Mar 2018 15:26:06 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1f0uEM-0007o6-Aw for bug-guix@gnu.org; Tue, 27 Mar 2018 15:26:05 -0400 Received: from debbugs.gnu.org ([208.118.235.43]:48358) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1f0uEM-0007nx-5X for bug-guix@gnu.org; Tue, 27 Mar 2018 15:26:02 -0400 Sender: "Debbugs-submit" Resent-Message-ID: Content-Disposition: inline In-Reply-To: <87d0zpyj5d.fsf_-_@gnu.org> List-Id: Bug reports for GNU Guix List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-guix-bounces+gcggb-bug-guix=m.gmane.org@gnu.org Sender: "bug-Guix" To: Ludovic =?UTF-8?Q?Court=C3=A8s?= Cc: 27284@debbugs.gnu.org, Maxim Cournoyer Awesome news! Ludovic Courtès transcribed 3.1K bytes: > Hello Guix! > > News from the front! I’ve pushed a new ‘wip-pull-multiple-derivations’ > branch that keeps the same approach as previously proposed (building > Guix using multiple derivation, one for each group of modules: core, > CLI, packages, etc.), but does not attempt to reload modules in the > running Guile process, which proved to be too tricky. > > You can already test with: > > guix pull --branch=wip-pull-multiple-derivations > > Feedback welcome! I think you forgot to push it or you are on a very slow connection :) > This patch set actually addresses a bootstrapping issue: you need Guix > to build Guix. So far ‘guix pull’ would sidestep the bootstrapping > issue by building Guix with whatever is available in the currently > deployed Guix; for instance, it would use the ‘guile’, ‘guile-json’, > etc. packages from the current Guix. The problem of that approach was > that it’s stateful: the result depends on what you currently have. > Sometimes what you have is too old, or lacks some package definition, > and you can’t really go forward. > > The code in ‘wip-pull-multiple-derivations’ works like this: > > 1. Assume we have Guile and Guix already installed, but not > necessarily the latest versions thereof. > > 2. Using the (guix …) modules that we have, build a program—a > “trampoline”—that will use the modules of the target Guix (the > commit we want to pull) to compute the derivation of that Guix. > > 3. Run that trampoline, which returns /gnu/store/…-guix.drv. The > result should be the same regardless of the initial Guix because > the trampoline uses exclusively modules from the target Guix. > > 4. ‘guix pull’ builds that derivation (actually the branch does not > modify (guix scripts pull) at all; everything is in > build-aux/build-self.scm.) > > In step #4, we should be able to get substitutes for at least some of > the derivations. > > To build the trampoline in step #2, we first need to build a bunch of > modules from the target Guix. Hopefully you don’t have to rebuild them > at each pull, but it can take a minute or so, and you may not have > substitutes for that (because this part is stateful.) > > In step #3, the trampoline has compiled code for the core modules, but > it still has to interpret (gnu packages guile) and related modules, > because these are not compiled. The program runs in ~40s on my laptop. > To make it faster, we could reduce the closure of (gnu packages guile)¹, > but that can be tricky. Or we could optimize Guile itself; I’m sure the > compiler and/or interpreter could do better. > > In that branch, you can also run: > > make as-derivation > > and it will build Guix from $srcdir in the way described above. > > To summarize, performance is not great, but hopefully it’ll be slightly > better than what we have now, especially with substitutes. > Correctness/reproducibility are greatly improved. > > Ludo’. > > ¹ The closure of (gnu packages guile) contains python, haskell-check, > music, and many other surprising things: > . > > >