From mboxrd@z Thu Jan 1 00:00:00 1970 From: ludo@gnu.org (Ludovic =?UTF-8?Q?Court=C3=A8s?=) Subject: bug#27284: =?UTF-8?Q?=E2=80=98guix_?= =?UTF-8?Q?pull=E2=80=99?= builds using multiple derivations Date: Sun, 08 Apr 2018 18:37:07 +0200 Message-ID: <87zi2dvemk.fsf@gnu.org> References: <87h8tnz1dx.fsf@gnu.org> <20171211105253.9670-1-ludo@gnu.org> <20171211105253.9670-4-ludo@gnu.org> <871sjst76a.fsf@gnu.org> <87d0zpyj5d.fsf_-_@gnu.org> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:43123) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1f5DKR-0004Zx-Kj for bug-guix@gnu.org; Sun, 08 Apr 2018 12:38:08 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1f5DKO-0007uH-Gg for bug-guix@gnu.org; Sun, 08 Apr 2018 12:38:07 -0400 Received: from debbugs.gnu.org ([208.118.235.43]:34863) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1f5DKO-0007u8-DD for bug-guix@gnu.org; Sun, 08 Apr 2018 12:38:04 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1f5DKO-0001qp-6i for bug-guix@gnu.org; Sun, 08 Apr 2018 12:38:04 -0400 Sender: "Debbugs-submit" Resent-To: bug-guix@gnu.org Resent-Message-ID: In-Reply-To: <87d0zpyj5d.fsf_-_@gnu.org> ("Ludovic \=\?utf-8\?Q\?Court\=C3\=A8s\?\= \=\?utf-8\?Q\?\=22's\?\= message of "Tue, 27 Mar 2018 11:14:54 +0200") List-Id: Bug reports for GNU Guix List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-guix-bounces+gcggb-bug-guix=m.gmane.org@gnu.org Sender: "bug-Guix" To: 27284-done@debbugs.gnu.org Cc: Maxim Cournoyer Hello! ludo@gnu.org (Ludovic Court=C3=A8s) skribis: > The code in =E2=80=98wip-pull-multiple-derivations=E2=80=99 works like th= is: > > 1. Assume we have Guile and Guix already installed, but not > necessarily the latest versions thereof. > > 2. Using the (guix =E2=80=A6) modules that we have, build a program=E2= =80=94a > =E2=80=9Ctrampoline=E2=80=9D=E2=80=94that will use the modules of th= e target Guix (the > commit we want to pull) to compute the derivation of that Guix. > > 3. Run that trampoline, which returns /gnu/store/=E2=80=A6-guix.drv. T= he > result should be the same regardless of the initial Guix because > the trampoline uses exclusively modules from the target Guix. > > 4. =E2=80=98guix pull=E2=80=99 builds that derivation (actually the bra= nch does not > modify (guix scripts pull) at all; everything is in > build-aux/build-self.scm.) After quite a bit of tweaking I=E2=80=99ve finally pushed this patch series= to master. So if you run =E2=80=98guix pull=E2=80=99 now, you=E2=80=99ll get = the new multiple-derivation build. There=E2=80=99s still room for improvement, in particular: =E2=80=A2 The initial =E2=80=98compute-guix-derivation=E2=80=99 program d= epends on quite a few modules, so the first time you =E2=80=98guix pull=E2=80=99 you build a module-import-compiled.drv that takes a while and does not produce any output (I=E2=80=99ve made it verbose in core-updates commit d32922759bfeffa03ee189158ea00b1a0ddbe8c6 though). Perhaps we could avoid that by using a double trampoline, which would allow us to get substitutes for all the non-trivial bits. =E2=80=A2 The =E2=80=9Cguix-packages=E2=80=9D derivation is coarse-grain.= We could perhaps automatically split it in clusters of modules by analyzing the (gnu packages =E2=80=A6) module graph. =E2=80=A2 The compiler (and interpreter) in Guile 2.2 is still not as fas= t as we=E2=80=99d like. I think it should be possible to make it as fast as= in 2.0, at least when optimizations are turned off. Andy made significant improvements in 2.2.3, but hopefully we can still improve on that. The thread at contains some info. Normally berlin.guixsd.org will start providing substitutes for all this but note that, as things are, it=E2=80=99ll compute substitutes for Guile 2= .2.2 or 2.2.3, and that=E2=80=99s it. hydra.gnu.org currently doesn=E2=80=99t, = for obscure reasons (=E2=80=98hydra-eval-guile-jobs=E2=80=99 systematically adds =E2=80= =9C.=E2=80=9D to the load path, which is not what we want for build-aux/hydra/guix-modular.scm.) This works unlocks some of the stuff in , so I=E2=80=99ll try to focus on that now. Feedback and bug reports welcome! Ludo=E2=80=99.