From mboxrd@z Thu Jan 1 00:00:00 1970 From: ludo@gnu.org (Ludovic =?UTF-8?Q?Court=C3=A8s?=) Subject: bug#27284: =?UTF-8?Q?=E2=80=98guix_?= =?UTF-8?Q?pull=E2=80=99?= builds using multiple derivations Date: Tue, 27 Mar 2018 11:14:54 +0200 Message-ID: <87d0zpyj5d.fsf_-_@gnu.org> References: <87h8tnz1dx.fsf@gnu.org> <20171211105253.9670-1-ludo@gnu.org> <20171211105253.9670-4-ludo@gnu.org> <871sjst76a.fsf@gnu.org> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:58955) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1f0kh7-0000Uv-Uz for bug-guix@gnu.org; Tue, 27 Mar 2018 05:15:07 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1f0kh3-0002r7-Sr for bug-guix@gnu.org; Tue, 27 Mar 2018 05:15:05 -0400 Received: from debbugs.gnu.org ([208.118.235.43]:47285) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1f0kh3-0002qz-OV for bug-guix@gnu.org; Tue, 27 Mar 2018 05:15:01 -0400 Sender: "Debbugs-submit" Resent-Message-ID: In-Reply-To: <871sjst76a.fsf@gnu.org> ("Ludovic \=\?utf-8\?Q\?Court\=C3\=A8s\=22'\?\= \=\?utf-8\?Q\?s\?\= message of "Mon, 18 Dec 2017 15:57:49 +0100") List-Id: Bug reports for GNU Guix List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-guix-bounces+gcggb-bug-guix=m.gmane.org@gnu.org Sender: "bug-Guix" To: 27284@debbugs.gnu.org Cc: Maxim Cournoyer Hello Guix! News from the front! I=E2=80=99ve pushed a new =E2=80=98wip-pull-multiple-= derivations=E2=80=99 branch that keeps the same approach as previously proposed (building Guix using multiple derivation, one for each group of modules: core, CLI, packages, etc.), but does not attempt to reload modules in the running Guile process, which proved to be too tricky. You can already test with: guix pull --branch=3Dwip-pull-multiple-derivations Feedback welcome! This patch set actually addresses a bootstrapping issue: you need Guix to build Guix. So far =E2=80=98guix pull=E2=80=99 would sidestep the boots= trapping issue by building Guix with whatever is available in the currently deployed Guix; for instance, it would use the =E2=80=98guile=E2=80=99, =E2= =80=98guile-json=E2=80=99, etc. packages from the current Guix. The problem of that approach was that it=E2=80=99s stateful: the result depends on what you currently have. Sometimes what you have is too old, or lacks some package definition, and you can=E2=80=99t really go forward. The code in =E2=80=98wip-pull-multiple-derivations=E2=80=99 works like this: 1. Assume we have Guile and Guix already installed, but not necessarily the latest versions thereof. 2. Using the (guix =E2=80=A6) modules that we have, build a program=E2=80= =94a =E2=80=9Ctrampoline=E2=80=9D=E2=80=94that will use the modules of the = target Guix (the commit we want to pull) to compute the derivation of that Guix. 3. Run that trampoline, which returns /gnu/store/=E2=80=A6-guix.drv. The result should be the same regardless of the initial Guix because the trampoline uses exclusively modules from the target Guix. 4. =E2=80=98guix pull=E2=80=99 builds that derivation (actually the branc= h does not modify (guix scripts pull) at all; everything is in build-aux/build-self.scm.) In step #4, we should be able to get substitutes for at least some of the derivations. To build the trampoline in step #2, we first need to build a bunch of modules from the target Guix. Hopefully you don=E2=80=99t have to rebuild = them at each pull, but it can take a minute or so, and you may not have substitutes for that (because this part is stateful.) In step #3, the trampoline has compiled code for the core modules, but it still has to interpret (gnu packages guile) and related modules, because these are not compiled. The program runs in ~40s on my laptop. To make it faster, we could reduce the closure of (gnu packages guile)=C2= =B9, but that can be tricky. Or we could optimize Guile itself; I=E2=80=99m sur= e the compiler and/or interpreter could do better. In that branch, you can also run: make as-derivation and it will build Guix from $srcdir in the way described above. To summarize, performance is not great, but hopefully it=E2=80=99ll be slig= htly better than what we have now, especially with substitutes. Correctness/reproducibility are greatly improved. Ludo=E2=80=99. =C2=B9 The closure of (gnu packages guile) contains python, haskell-check, music, and many other surprising things: .