bug#27284: Memory leak in 'guix pull' or 'make' in guix source

bug#27284: Memory leak in 'guix pull' or 'make' in guix source

[ng0]

[-- Attachment #1: Type: text/plain, Size: 616 bytes --]

This has been discussed on IRC and on guix-devel mailinglist
but so far I haven't seen a formal bug report to make this
visible.

I can't find the right threads or timestamps for this
bug, so take it as is and extend with facts.

At the moment of writing, guix pull has the minimum
requirement of 3 GB RAM.

Before this started, I was able to run Guix on as
little as 512MB RAM.

People are currently finding out about this bug
the hard way, so we should point this out. Didn't
really happen so I guess someone is working on this
already.
-- 
ng0
OpenPG: A88C8ADD129828D7EAC02E52E22F9BBFEE348588

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

bug#27284: Memory leak in 'guix pull' or 'make' in guix source

[ng0]

[-- Attachment #1: Type: text/plain, Size: 789 bytes --]

ng0 transcribed 1.6K bytes:
> This has been discussed on IRC and on guix-devel mailinglist
> but so far I haven't seen a formal bug report to make this
> visible.
> 
> I can't find the right threads or timestamps for this
> bug, so take it as is and extend with facts.
> 
> At the moment of writing, guix pull has the minimum
> requirement of 3 GB RAM.
> 
> Before this started, I was able to run Guix on as
> little as 512MB RAM.
> 
> People are currently finding out about this bug
> the hard way, so we should point this out. Didn't
> really happen so I guess someone is working on this
> already.

For reference with more details:
https://lists.gnu.org/archive/html/guile-devel/2017-05/msg00033.html


-- 
ng0
OpenPG: A88C8ADD129828D7EAC02E52E22F9BBFEE348588

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

bug#27284: Memory leak in 'guix pull' or 'make' in guix source

[Ludovic Courtès]

Hello Guix!

A heads-up to share the cognitive burden related to this topic.  :-)

So, we have two problems: compilation time, and memory consumption.  I
*think* I’ve identified one of the major causes for both in Guile,
though it’s too early to say exactly how much this will impact resource
consumption for a full Guix compilation.  See
<https://lists.gnu.org/archive/html/guile-devel/2017-09/msg00031.html>
for details.

When that is fixed, we’ll still have a performance problem: building all
of Guix will still take more time than we’d like, and it won’t get
better as we add new files.  So we need to address this.

This has been discussed informally many times, and here’s a summary of
the ideas I’m aware of:

  1. Build Guix as separate derivations: the first derivation builds the
     closure of (guix packages), the second one builds the closure of
     (guix scripts *), and finally we build (gnu *).  We could also have
     derivations for the non-Scheme parts: the daemon, the manual, etc.

     (That amounts to compartmentalizing Guix in sub-packages, which in
     a way solves the bootstrapping issue that Pjotr complained about at
     some point—the fact that to build Guix one needs Guile, Guile-JSON,
     GnuTLS, Autotools, etc.)

     The advantage is that when running ‘guix pull’ frequently, you
     won’t have to recompile all of these.  However, you’ll almost
     always have to rebuild (gnu packages *), which is the longest part.

  2. Build all of Guix like the ‘guix’ package does, and hope that we
     can get a substitute.

     Bootstrapping issue: to do that, we first need compute the
     derivation of this new ‘guix’ package.  Thus, we at least need to
     build the closure of (guix packages), which should take a minute or
     so, after which we can compute the derivation, which could take a
     couple of minutes maybe.

     The problem is that building all of Guix (including running the
     test suite) takes some time, potentially more than the interval
     between two subsequent pushes to the repo.  Thus, it’s quite likely
     that the build farm would always be lagging behind.

     We could work around that by having the build farm automatically
     tag commits for which it has successfully built Guix.  That would
     introduce delays in deploying the latest Guix to users, but maybe
     that can be short enough to be acceptable.

  2b. To address the bootstrapping issue above, we also discussed the
     possibility of setting up a “meta-derivation” service: you’d give
     it a Git commit, and it’d return the derivation of ‘guix’ for that
     commit.

     Less computation would take place on the user side, but that
     doesn’t reduce the delay mentioned above.

     It also has the downside of introducing another service without
     which using Guix is more painful.

I think that’s about it.

Thoughts?  Hacks?  :-)

Ludo’.

bug#27284: Memory leak in 'guix pull' or 'make' in guix source

[Maxim Cournoyer]

ludo@gnu.org (Ludovic Courtès) writes:

> Hello Guix!
>
> A heads-up to share the cognitive burden related to this topic.  :-)
>
> So, we have two problems: compilation time, and memory consumption.  I
> *think* I’ve identified one of the major causes for both in Guile,
> though it’s too early to say exactly how much this will impact resource
> consumption for a full Guix compilation.  See
> <https://lists.gnu.org/archive/html/guile-devel/2017-09/msg00031.html>
> for details.
>
> When that is fixed, we’ll still have a performance problem: building all
> of Guix will still take more time than we’d like, and it won’t get
> better as we add new files.  So we need to address this.
>
> This has been discussed informally many times, and here’s a summary of
> the ideas I’m aware of:
>
>   1. Build Guix as separate derivations: the first derivation builds the

[...]

>
>   2. Build all of Guix like the ‘guix’ package does, and hope that we
>      can get a substitute.

This seems like the best option to me. It strikes me as odd that Guix
isn't distributed as its own substitutable package that can be
efficiently downloaded and unpacked to the store, rather than the
current intensive build-it-all process.

>      Bootstrapping issue: to do that, we first need compute the
>      derivation of this new ‘guix’ package.  Thus, we at least need to
>      build the closure of (guix packages), which should take a minute or
>      so, after which we can compute the derivation, which could take a
>      couple of minutes maybe.
>      The problem is that building all of Guix (including running the
>      test suite) takes some time, potentially more than the interval
>      between two subsequent pushes to the repo.  Thus, it’s quite likely
>      that the build farm would always be lagging behind.

Do we really have to build it *all*, every time? There are two
optimizations which I can think of:

1. Only rebuild the files that changed.

I guess the challenge here is for the build to be deterministic. It
would be important to be able to detect when any of the inputs part of
the guix closure changed; when such a thing occurs we'd have to fully
rebuild Guix. Otherwise, for a constant closure, it should be safe to
simply 'git pull' and run 'make' to rebuild only the modified files,
which could speed up things a bit. A smarter Guile that would only
partially recompile the changed/referenced parts of massive files
instead of completely (think python.scm) would also improve things a lot
(this would also be a must when working on packages directly in the
tree! Updating the hash of a Python package shouldn't require minutes of
compilation).

2. Remove the --no-auto-compile arg from (guix scripts guix).

I'm not sure why this is desirable. Is it because otherwise the cached
compiled copies would be susceptible to ABI breakages? This could be
catch and handled by the build script, in which case it would retry with
'--fresh-auto-compile'. IIUC, these cached files would never end in the the
package to be distributed anyway, but having them would greatly speed up
computing the closure of guix.

Even with such optimizations it could still take more time to build Guix
than the rate at which commits hit master. Maybe a cron job that could
build the latest Guix every, say, 5 or 10 minutes (only when new commits
appeared since the last time it was built)? We would then distribute the
latest built Guix package when users would issue a 'guix pull'. There
could be a 'guix pull' option to force building the package from the
latest sources, for those who prefer the current behavior.

I think most users would happily trade having to entirely compile Guix
from scratch everytime they use 'guix pull' with being behind 5 minutes
worth of commits from master's tip!

Maxim

bug#27284: Memory leak in 'guix pull' or 'make' in guix source

[Ludovic Courtès]

Howdy,

Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:

> ludo@gnu.org (Ludovic Courtès) writes:

[...]

>> This has been discussed informally many times, and here’s a summary of
>> the ideas I’m aware of:
>>
>>   1. Build Guix as separate derivations: the first derivation builds the
>
> [...]
>
>>
>>   2. Build all of Guix like the ‘guix’ package does, and hope that we
>>      can get a substitute.
>
> This seems like the best option to me. It strikes me as odd that Guix
> isn't distributed as its own substitutable package that can be
> efficiently downloaded and unpacked to the store, rather than the
> current intensive build-it-all process.

Yeah.

>>      Bootstrapping issue: to do that, we first need compute the
>>      derivation of this new ‘guix’ package.  Thus, we at least need to
>>      build the closure of (guix packages), which should take a minute or
>>      so, after which we can compute the derivation, which could take a
>>      couple of minutes maybe.
>>      The problem is that building all of Guix (including running the
>>      test suite) takes some time, potentially more than the interval
>>      between two subsequent pushes to the repo.  Thus, it’s quite likely
>>      that the build farm would always be lagging behind.
>
> Do we really have to build it *all*, every time? There are two
> optimizations which I can think of:
>
> 1. Only rebuild the files that changed.

That’s more or less what #1 would give us, though not necessarily at the
finest granularity.

> 2. Remove the --no-auto-compile arg from (guix scripts guix).
>
> I'm not sure why this is desirable. Is it because otherwise the cached
> compiled copies would be susceptible to ABI breakages?

Yes, that’s one thing.

Another thing is that on the first run of ‘guix package’ following ‘guix
pull’, you’d end up compiling everything, which is not much better than
compiling them upfront.

There’s also the option of not compiling (gnu packages *) and instead
evaluating them, but currently this is too costly in terms of memory and
CPU.

> Even with such optimizations it could still take more time to build Guix
> than the rate at which commits hit master. Maybe a cron job that could
> build the latest Guix every, say, 5 or 10 minutes (only when new commits
> appeared since the last time it was built)? We would then distribute the
> latest built Guix package when users would issue a 'guix pull'. There
> could be a 'guix pull' option to force building the package from the
> latest sources, for those who prefer the current behavior.

Yes, what I was suggesting was to build Guix at each push, and
automatically tag successful builds, such that ‘guix pull’ can pull up
to the latest successfully-built tag.  (Though we’d need to be careful:
if the auto-tagging machinery stops working, we still want users to get
the latest Guix so they have security updates.)

Thanks for chiming in!

Ludo’.

bug#27284: Memory leak in 'guix pull' or 'make' in guix source

[Maxim Cournoyer]

ludo@gnu.org (Ludovic Courtès) writes:

> Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:
>
>> ludo@gnu.org (Ludovic Courtès) writes:
>
> [...]
>
>> 2. Remove the --no-auto-compile arg from (guix scripts guix).
>>
>> I'm not sure why this is desirable. Is it because otherwise the cached
>> compiled copies would be susceptible to ABI breakages?
>
> Yes, that’s one thing.
>
> Another thing is that on the first run of ‘guix package’ following ‘guix
> pull’, you’d end up compiling everything, which is not much better than
> compiling them upfront.

To be clear, my suggestion of removing --no-auto-compile was not as a
replacement to compilation, but in addition to. Thinking more about what
I wrote yesterday, it seems what I had on mind was a custom, optimized
way to build Guix that would minimize the amount of files to recompile,
ala 'git pull && make' without running make clean-go or even ./configure
(under the condition that the closure of guix hasn't changed -- when it
does, a full rebuild would be necessary to ensure determinism).

As an unrelated note, I find --no-auto-compile annoying in a development
context when working directly from the guix tree. It forces you to
constantly run 'make', otherwise it recompiles modules changed since the
last 'make' every time Guile needs them, even when they haven't changed
on disk.

> There’s also the option of not compiling (gnu packages *) and instead
> evaluating them, but currently this is too costly in terms of memory and
> CPU.

How do we instruct Guile to evaluate rather than compile? I don't think
I've ever experimented with this yet!

Maxim

bug#27284: Memory leak in 'guix pull' or 'make' in guix source

[Ludovic Courtès]

Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:

> How do we instruct Guile to evaluate rather than compile? I don't think
> I've ever experimented with this yet!

Simply by not compiling things (which includes passing
--no-auto-compile.)

Ludo’.

bug#27284: Memory leak in 'guix pull' or 'make' in guix source

[Maxim Cournoyer]

ludo@gnu.org (Ludovic Courtès) writes:

> Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:
>
>> How do we instruct Guile to evaluate rather than compile? I don't think
>> I've ever experimented with this yet!
>
> Simply by not compiling things (which includes passing
> --no-auto-compile.)
>
> Ludo’.

I see. I was under the impression that things would be "compiled" anyway
but the result kept in memory and discarded rather than cached when
using '--no-auto-compile'.

Thanks for clearing that up!

Maxim

bug#27284: Memory leak in 'guix pull' or 'make' in guix source

[Taylan Ulrich Bayırlı/Kammer]

ludo@gnu.org (Ludovic Courtès) writes:

> [...]
>
> There’s also the option of not compiling (gnu packages *) and instead
> evaluating them, but currently this is too costly in terms of memory
> and CPU.
>
> [...]

Can't we leave this to auto-compilation during normal use of guix?

Taylan

bug#27284: Memory leak in 'guix pull' or 'make' in guix source

[Ludovic Courtès]

taylanbayirli@gmail.com (Taylan Ulrich "Bayırlı/Kammer") skribis:

> ludo@gnu.org (Ludovic Courtès) writes:
>
>> [...]
>>
>> There’s also the option of not compiling (gnu packages *) and instead
>> evaluating them, but currently this is too costly in terms of memory
>> and CPU.
>>
>> [...]
>
> Can't we leave this to auto-compilation during normal use of guix?

We’d have the same problem, only at a different point in time (it might
even be worse because auto-compilation would use -O2 instead of -O0.)

Ludo’.

bug#27284: Memory leak in 'guix pull' or 'make' in guix source

[Maxim Cournoyer]

ludo@gnu.org (Ludovic Courtès) writes:

> taylanbayirli@gmail.com (Taylan Ulrich "Bayırlı/Kammer") skribis:
>
>> ludo@gnu.org (Ludovic Courtès) writes:
>>
>>> [...]
>>>
>>> There’s also the option of not compiling (gnu packages *) and instead
>>> evaluating them, but currently this is too costly in terms of memory
>>> and CPU.
>>>
>>> [...]
>>
>> Can't we leave this to auto-compilation during normal use of guix?
>
> We’d have the same problem, only at a different point in time (it might
> even be worse because auto-compilation would use -O2 instead of -O0.)

I've experimented a bit with --no-auto-compile (or lack of) and also
realized that it doesn't bring much to the table. Evaluating (rather
than auto-compiling) the modules is much faster and we need to properly
compile those anyway if we are to properly ship a guix binary
substitute.

Maxim

bug#27284: [PATCH 0/8] 'guix pull' creates several derivations

[Ludovic Courtès]

Hi there!

This patch set leads us to a ‘guix pull’ that creates several
derivations instead of just one: “guix-core” (38 files), “guix-extra”,
“guix-packages” (388 files), “guix-cli”, “guix-system”.

The idea is that we should be able to have substitutes for at least some
of them, for instance because “guix-core” doesn’t need to be rebuilt
very often.  Most of the time, users will just have to build
“guix-packages”, which is still a bit slow, but will hopefully get
better as Guile incorporates those long-awaited fixes.

The first two commits provide much needed improvements, and I’d be
willing to merge them ASAP.

The rest is the guts of the project.  The last commit is marked as
draft: it works as-is, but does not yet address one of the limitations
of ‘guix pull’, which is that dependencies (guile-json, guile-ssh, etc.)
are taken from the *current* Guix instead of being taken from the target
Guix.

The ‘reload-guix’ procedure there is here to fix that: we reload all the
Guix modules of the target checkout, and run the build procedure from
that context.  Unfortunately, it’s not fully baked yet because reloading
leads to incompatibilities: the (guix scripts pull) module remains in
the “old world” and manipulates the old <derivation> and <package>
record types, which is different from the new ones.  That should be
fixable, but requires some more time and fiddling.  Help from Guilers is
very much welcome!  :-)

This last part is not ready yet, but feedback is appreciated.

Thanks in advance.  :-)

Ludo’.

Ludovic Courtès (8):
  build: Factorize module compilation in (guix build compile).
  build: Honor make's '-j' flag.
  discovery: Move 'file-name->module-name' to (guix modules).
  gexp: Add 'file-union'.
  gexp: Add 'directory-union'.
  union: Parametrize the symlink procedure .
  gexp: 'directory-union' has a #:quiet? parameter.
  DRAFT Add (guix self) and use it when pulling.

 Makefile.am               |   2 +
 build-aux/build-self.scm  | 227 ++----------------
 build-aux/compile-all.scm | 116 +++------
 doc/guix.texi             |  28 +++
 gnu/packages.scm          |  21 +-
 gnu/services.scm          |  37 +--
 guix/build/compile.scm    | 165 +++++++++++++
 guix/build/pull.scm       | 105 +++-----
 guix/build/union.scm      |  11 +-
 guix/discovery.scm        |  11 +-
 guix/gexp.scm             |  72 ++++++
 guix/modules.scm          |  10 +
 guix/scripts/pull.scm     |  16 +-
 guix/self.scm             | 599 ++++++++++++++++++++++++++++++++++++++++++++++
 14 files changed, 1001 insertions(+), 419 deletions(-)
 create mode 100644 guix/build/compile.scm
 create mode 100644 guix/self.scm

-- 
2.14.2

bug#27284: [PATCH 1/8] build: Factorize module compilation in (guix build compile).

[Ludovic Courtès]

* guix/build/compile.scm: New file.
* Makefile.am (MODULES): Add it.
* build-aux/compile-all.scm: Use it.
(warnings, file->module, load-module-file)
(%default-optimizations, %lightweight-optimizations)
(optimization-options, compile-file*): Remove.
<top level>: Use 'compile-files'.
* guix/build/pull.scm (%default-optimizations)
(%lightweight-optimizations, optimization-options): Remove.
(build-guix): Rewrite as a call to 'compile-files'.
* guix/discovery.scm (file-name->module-name): Export.
---
 Makefile.am               |   1 +
 build-aux/compile-all.scm |  92 ++++----------------------
 guix/build/compile.scm    | 165 ++++++++++++++++++++++++++++++++++++++++++++++
 guix/build/pull.scm       | 105 ++++++++---------------------
 guix/discovery.scm        |   4 +-
 5 files changed, 209 insertions(+), 158 deletions(-)
 create mode 100644 guix/build/compile.scm

diff --git a/Makefile.am b/Makefile.am
index 2855b4efd..fd6f9729c 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -133,6 +133,7 @@ MODULES =					\
   guix/build/utils.scm				\
   guix/build/union.scm				\
   guix/build/profiles.scm			\
+  guix/build/compile.scm			\
   guix/build/pull.scm				\
   guix/build/rpath.scm				\
   guix/build/cvs.scm				\
diff --git a/build-aux/compile-all.scm b/build-aux/compile-all.scm
index fe25c5d06..2fc3102da 100644
--- a/build-aux/compile-all.scm
+++ b/build-aux/compile-all.scm
@@ -17,21 +17,12 @@
 ;;; You should have received a copy of the GNU General Public License
 ;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
 
-(use-modules (system base target)
-             (system base message)
-             (ice-9 match)
+(use-modules (ice-9 match)
              (ice-9 threads)
+             (guix build compile)
              (guix build utils))
 
-(define warnings
-  ;; FIXME: 'format' is missing because it reports "non-literal format
-  ;; strings" due to the fact that we use 'G_' instead of '_'.  We'll need
-  ;; help from Guile to solve this.
-  '(unsupported-warning unbound-variable arity-mismatch
-    macro-use-before-definition))                 ;new in 2.2
-
 (define host (getenv "host"))
-
 (define srcdir (getenv "srcdir"))
 
 (define (relative-file file)
@@ -53,62 +44,6 @@
     (or (not (file-exists? go))
         (file-mtime<? go file))))
 
-(define (file->module file)
-  (let* ((relative (relative-file file))
-         (module-path (string-drop-right relative 4)))
-    (map string->symbol
-         (string-split module-path #\/))))
-
-;;; To work around <http://bugs.gnu.org/15602> (FIXME), we want to load all
-;;; files to be compiled first.  We do this via resolve-interface so that the
-;;; top-level of each file (module) is only executed once.
-(define (load-module-file file)
-  (let ((module (file->module file)))
-    (format #t "  LOAD     ~a~%" module)
-    (resolve-interface module)))
-
-(cond-expand
-  (guile-2.2 (use-modules (language tree-il optimize)
-                          (language cps optimize)))
-  (else #f))
-
-(define %default-optimizations
-  ;; Default optimization options (equivalent to -O2 on Guile 2.2).
-  (cond-expand
-    (guile-2.2 (append (tree-il-default-optimization-options)
-                       (cps-default-optimization-options)))
-    (else '())))
-
-(define %lightweight-optimizations
-  ;; Lightweight optimizations (like -O0, but with partial evaluation).
-  (let loop ((opts %default-optimizations)
-             (result '()))
-    (match opts
-      (() (reverse result))
-      ((#:partial-eval? _ rest ...)
-       (loop rest `(#t #:partial-eval? ,@result)))
-      ((kw _ rest ...)
-       (loop rest `(#f ,kw ,@result))))))
-
-(define (optimization-options file)
-  (if (string-contains file "gnu/packages/")
-      %lightweight-optimizations                  ;build faster
-      '()))
-
-(define (compile-file* file output-mutex)
-  (let ((go (scm->go file)))
-    (with-mutex output-mutex
-      (format #t "  GUILEC   ~a~%" go)
-      (force-output))
-    (mkdir-p (dirname go))
-    (with-fluids ((*current-warning-prefix* ""))
-      (with-target host
-        (lambda ()
-          (compile-file file
-                        #:output-file go
-                        #:opts `(#:warnings ,warnings
-                                 ,@(optimization-options file))))))))
-
 ;; Install a SIGINT handler to give unwind handlers in 'compile-file' an
 ;; opportunity to run upon SIGINT and to remove temporary output files.
 (sigaction SIGINT
@@ -117,16 +52,13 @@
 
 (match (command-line)
   ((_ . files)
-   (let ((files (filter file-needs-compilation? files)))
-     (for-each load-module-file files)
-     (let ((mutex (make-mutex)))
-       ;; Make sure compilation related modules are loaded before starting to
-       ;; compile files in parallel.
-       (compile #f)
-       (par-for-each (lambda (file)
-                       (compile-file* file mutex))
-                     files)))))
-
-;;; Local Variables:
-;;; eval: (put 'with-target 'scheme-indent-function 1)
-;;; End:
+   (compile-files srcdir (getcwd)
+                  (filter file-needs-compilation? files)
+                  #:host host
+                  #:report-load (lambda (file total completed)
+                                  (when file
+                                    (format #t "  LOAD     ~a~%" file)))
+                  #:report-compilation (lambda (file total completed)
+                                         (when file
+                                           (format #t "  GUILEC   ~a~%"
+                                                   (scm->go file)))))))
diff --git a/guix/build/compile.scm b/guix/build/compile.scm
new file mode 100644
index 000000000..6f15ba578
--- /dev/null
+++ b/guix/build/compile.scm
@@ -0,0 +1,165 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2013, 2014, 2016, 2017 Ludovic Courtès <ludo@gnu.org>
+;;; Copyright © 2015 Taylan Ulrich Bayırlı/Kammer <taylanbayirli@gmail.com>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (guix build compile)
+  #:use-module (ice-9 match)
+  #:use-module (ice-9 format)
+  #:use-module (ice-9 threads)
+  #:use-module (system base target)
+  #:use-module (system base compile)
+  #:use-module (system base message)
+  #:use-module (guix discovery)
+  #:use-module (guix build utils)
+  #:export (%default-optimizations
+            %lightweight-optimizations
+            compile-files))
+
+;;; Commentary:
+;;;
+;;; Support code to compile Guile code as efficiently as possible (both with
+;;; Guile 2.0 and 2.2).
+;;;
+;;; Code:
+
+(cond-expand
+  (guile-2.2 (use-modules (language tree-il optimize)
+                          (language cps optimize)))
+  (else #f))
+
+(define %default-optimizations
+  ;; Default optimization options (equivalent to -O2 on Guile 2.2).
+  (cond-expand
+    (guile-2.2 (append (tree-il-default-optimization-options)
+                       (cps-default-optimization-options)))
+    (else '())))
+
+(define %lightweight-optimizations
+  ;; Lightweight optimizations (like -O0, but with partial evaluation).
+  (let loop ((opts %default-optimizations)
+             (result '()))
+    (match opts
+      (() (reverse result))
+      ((#:partial-eval? _ rest ...)
+       (loop rest `(#t #:partial-eval? ,@result)))
+      ((kw _ rest ...)
+       (loop rest `(#f ,kw ,@result))))))
+
+(define %warnings
+  ;; FIXME: 'format' is missing because it reports "non-literal format
+  ;; strings" due to the fact that we use 'G_' instead of '_'.  We'll need
+  ;; help from Guile to solve this.
+  '(unsupported-warning unbound-variable arity-mismatch
+    macro-use-before-definition))                 ;new in 2.2
+
+(define (optimization-options file)
+  "Return the default set of optimizations options for FILE."
+  (if (string-contains file "gnu/packages/")
+      %lightweight-optimizations                  ;build faster
+      '()))
+
+(define (scm->go file)
+  "Strip the \".scm\" suffix from FILE, and append \".go\"."
+  (string-append (string-drop-right file 4) ".go"))
+
+(define* (load-files directory files
+                     #:key
+                     (report-load (const #f))
+                     (debug-port (%make-void-port "w")))
+  "Load FILES, a list of relative file names, from DIRECTORY."
+  (define total
+    (length files))
+
+  (let loop ((files files)
+             (completed 0))
+    (match files
+      (()
+       (unless (zero? total)
+         (report-load #f total completed))
+       *unspecified*)
+      ((file files ...)
+       (report-load file total completed)
+       (format debug-port "~%loading '~a'...~%" file)
+
+       (parameterize ((current-warning-port debug-port))
+         (resolve-interface (file-name->module-name file)))
+
+       (loop files (+ 1 completed))))))
+
+(define-syntax-rule (with-augmented-search-path path item body ...)
+  "Within the dynamic extent of BODY, augment PATH by adding ITEM to the
+front."
+  (let ((initial-value path))
+    (dynamic-wind
+      (lambda ()
+        (set! path (cons item path)))
+      (lambda ()
+        body ...)
+      (lambda ()
+        (set! path initial-value)))))
+
+(define* (compile-files source-directory build-directory files
+                        #:key
+                        (host %host-type)
+                        (workers (current-processor-count))
+                        (optimization-options optimization-options)
+                        (warning-options `(#:warnings ,%warnings))
+                        (report-load (const #f))
+                        (report-compilation (const #f))
+                        (debug-port (%make-void-port "w")))
+  "Compile FILES, a list of source files taken from SOURCE-DIRECTORY, to
+BUILD-DIRECTORY, using up to WORKERS parallel workers.  The resulting object
+files are for HOST, a GNU triplet such as \"x86_64-linux-gnu\"."
+  (define progress-lock (make-mutex))
+  (define total (length files))
+  (define completed 0)
+
+  (define (build file)
+    (with-mutex progress-lock
+      (report-compilation file total completed))
+    (with-fluids ((*current-warning-prefix* ""))
+      (with-target host
+        (lambda ()
+          (compile-file file
+                        #:output-file (string-append build-directory "/"
+                                                     (scm->go file))
+                        #:opts (append warning-options
+                                       (optimization-options file))))))
+    (with-mutex progress-lock
+      (set! completed (+ 1 completed))))
+
+  (with-augmented-search-path %load-path source-directory
+    (with-augmented-search-path %load-compiled-path build-directory
+      ;; FIXME: To work around <https://bugs.gnu.org/15602>, we first load all
+      ;; of FILES.
+      (load-files source-directory files
+                  #:report-load report-load
+                  #:debug-port debug-port)
+
+      ;; Make sure compilation related modules are loaded before starting to
+      ;; compile files in parallel.
+      (compile #f)
+
+      (n-par-for-each workers build files)
+      (unless (zero? total)
+        (report-compilation #f total total)))))
+
+;;; Local Variables:
+;;; eval: (put 'with-augmented-search-path 'scheme-indent-function 2)
+;;; eval: (put 'with-target 'scheme-indent-function 1)
+;;; End:
diff --git a/guix/build/pull.scm b/guix/build/pull.scm
index 1ae35ab38..6f7aa2786 100644
--- a/guix/build/pull.scm
+++ b/guix/build/pull.scm
@@ -20,11 +20,10 @@
 (define-module (guix build pull)
   #:use-module (guix modules)
   #:use-module (guix build utils)
-  #:use-module (system base compile)
+  #:use-module (guix build compile)
   #:use-module (ice-9 ftw)
   #:use-module (ice-9 match)
   #:use-module (ice-9 format)
-  #:use-module (ice-9 threads)
   #:use-module (srfi srfi-1)
   #:use-module (srfi srfi-11)
   #:use-module (srfi srfi-26)
@@ -63,34 +62,6 @@ available, false otherwise."
                      (string-prefix? gnu  b))
                 (string<? a b))))))
 
-(cond-expand
-  (guile-2.2 (use-modules (language tree-il optimize)
-                          (language cps optimize)))
-  (else #f))
-
-(define %default-optimizations
-  ;; Default optimization options (equivalent to -O2 on Guile 2.2).
-  (cond-expand
-    (guile-2.2 (append (tree-il-default-optimization-options)
-                       (cps-default-optimization-options)))
-    (else '())))
-
-(define %lightweight-optimizations
-  ;; Lightweight optimizations (like -O0, but with partial evaluation).
-  (let loop ((opts %default-optimizations)
-             (result '()))
-    (match opts
-      (() (reverse result))
-      ((#:partial-eval? _ rest ...)
-       (loop rest `(#t #:partial-eval? ,@result)))
-      ((kw _ rest ...)
-       (loop rest `(#f ,kw ,@result))))))
-
-(define (optimization-options file)
-  (if (string-contains file "gnu/packages/")
-      %lightweight-optimizations                  ;build faster
-      '()))
-
 \f
 (define* (build-guix out source
                      #:key
@@ -148,53 +119,33 @@ containing the source code.  Write any debugging output to DEBUG-PORT."
     (set! %load-path (cons out %load-path))
     (set! %load-compiled-path (cons out %load-compiled-path))
 
-    ;; Compile the .scm files.  Load all the files before compiling them to
-    ;; work around <http://bugs.gnu.org/15602> (FIXME).
-    ;; Filter out files depending on Guile-SSH when Guile-SSH is missing.
-    (let* ((files (filter has-all-its-dependencies?
-                          (all-scheme-files out)))
-           (total (length files)))
-      (let loop ((files files)
-                 (completed 0))
-        (match files
-          (() *unspecified*)
-          ((file . files)
-           (display #\cr log-port)
-           (format log-port "loading...\t~5,1f% of ~d files" ;FIXME: i18n
-                   (* 100. (/ completed total)) total)
-           (force-output log-port)
-           (format debug-port "~%loading '~a'...~%" file)
-           ;; Turn "<out>/foo/bar.scm" into (foo bar).
-           (let* ((relative-file (string-drop file (+ (string-length out) 1)))
-                  (module-path (string-drop-right relative-file 4))
-                  (module-name (map string->symbol
-                                    (string-split module-path #\/))))
-             (parameterize ((current-warning-port debug-port))
-               (resolve-interface module-name)))
-           (loop files (+ 1 completed)))))
-      (newline)
-      (let ((mutex (make-mutex))
-            (completed 0))
-        ;; Make sure compilation related modules are loaded before starting to
-        ;; compile files in parallel.
-        (compile #f)
-        (n-par-for-each
-         (parallel-job-count)
-         (lambda (file)
-           (with-mutex mutex
-             (display #\cr log-port)
-             (format log-port "compiling...\t~5,1f% of ~d files" ;FIXME: i18n
-                     (* 100. (/ completed total)) total)
-             (force-output log-port)
-             (format debug-port "~%compiling '~a'...~%" file))
-           (let ((go (string-append (string-drop-right file 4) ".go")))
-             (parameterize ((current-warning-port (%make-void-port "w")))
-               (compile-file file
-                             #:output-file go
-                             #:opts (optimization-options file))))
-           (with-mutex mutex
-             (set! completed (+ 1 completed))))
-         files))))
+    ;; Compile the .scm files.  Filter out files depending on Guile-SSH when
+    ;; Guile-SSH is missing.
+    (let ((files (filter has-all-its-dependencies?
+                         (all-scheme-files out))))
+      (compile-files out out files
+
+                     #:workers (parallel-job-count)
+
+                     ;; Disable warnings.
+                     #:warning-options '()
+
+                     #:report-load
+                     (lambda (file total completed)
+                       (display #\cr log-port)
+                       (format log-port
+                               "loading...\t~5,1f% of ~d files" ;FIXME: i18n
+                               (* 100. (/ completed total)) total)
+                       (force-output log-port)
+                       (format debug-port "~%loading '~a'...~%" file))
+
+                     #:report-compilation
+                     (lambda (file total completed)
+                       (display #\cr log-port)
+                       (format log-port "compiling...\t~5,1f% of ~d files" ;FIXME: i18n
+                               (* 100. (/ completed total)) total)
+                       (force-output log-port)
+                       (format debug-port "~%compiling '~a'...~%" file)))))
 
   (newline)
   #t)
diff --git a/guix/discovery.scm b/guix/discovery.scm
index 2741725b9..c861614b8 100644
--- a/guix/discovery.scm
+++ b/guix/discovery.scm
@@ -24,7 +24,9 @@
   #:use-module (ice-9 match)
   #:use-module (ice-9 vlist)
   #:use-module (ice-9 ftw)
-  #:export (scheme-modules
+  #:export (file-name->module-name
+
+            scheme-modules
             fold-modules
             all-modules
             fold-module-public-variables))
-- 
2.14.2

bug#27284: [PATCH 1/8] build: Factorize module compilation in (guix build compile).

[Maxim Cournoyer]

Hello,

Ludovic Courtès <ludo@gnu.org> writes:

> * guix/build/compile.scm: New file.
> * Makefile.am (MODULES): Add it.
> * build-aux/compile-all.scm: Use it.
> (warnings, file->module, load-module-file)
> (%default-optimizations, %lightweight-optimizations)
> (optimization-options, compile-file*): Remove.
> <top level>: Use 'compile-files'.
> * guix/build/pull.scm (%default-optimizations)
> (%lightweight-optimizations, optimization-options): Remove.
> (build-guix): Rewrite as a call to 'compile-files'.
> * guix/discovery.scm (file-name->module-name): Export.
> ---
>  Makefile.am               |   1 +
>  build-aux/compile-all.scm |  92 ++++----------------------
>  guix/build/compile.scm    | 165 ++++++++++++++++++++++++++++++++++++++++++++++
>  guix/build/pull.scm       | 105 ++++++++---------------------
>  guix/discovery.scm        |   4 +-
>  5 files changed, 209 insertions(+), 158 deletions(-)
>  create mode 100644 guix/build/compile.scm

[...]

This and the other patches in this series look fine to me. Maybe the
change log items could be a bit more precise about what got moved rather
than removed, but given the amount of refactoring this might be too
verbose to be worth it?

Maxim

bug#27284: [PATCH 1/8] build: Factorize module compilation in (guix build compile).

[Ludovic Courtès]

Hi Maxim,

Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:

> Ludovic Courtès <ludo@gnu.org> writes:
>
>> * guix/build/compile.scm: New file.
>> * Makefile.am (MODULES): Add it.
>> * build-aux/compile-all.scm: Use it.
>> (warnings, file->module, load-module-file)
>> (%default-optimizations, %lightweight-optimizations)
>> (optimization-options, compile-file*): Remove.
>> <top level>: Use 'compile-files'.
>> * guix/build/pull.scm (%default-optimizations)
>> (%lightweight-optimizations, optimization-options): Remove.
>> (build-guix): Rewrite as a call to 'compile-files'.
>> * guix/discovery.scm (file-name->module-name): Export.
>> ---
>>  Makefile.am               |   1 +
>>  build-aux/compile-all.scm |  92 ++++----------------------
>>  guix/build/compile.scm    | 165 ++++++++++++++++++++++++++++++++++++++++++++++
>>  guix/build/pull.scm       | 105 ++++++++---------------------
>>  guix/discovery.scm        |   4 +-
>>  5 files changed, 209 insertions(+), 158 deletions(-)
>>  create mode 100644 guix/build/compile.scm
>
> [...]
>
> This and the other patches in this series look fine to me.

Awesome, thanks for looking.

> Maybe the change log items could be a bit more precise about what got
> moved rather than removed, but given the amount of refactoring this
> might be too verbose to be worth it?

Well, yes to both.  :-)

Speaking of the “make -j” patch, on IRC Eric Bavier suggested that it
didn’t work as advertised because $MAKEFLAGS didn’t contain the “-j”
flag; I just checked and it does contain it here (with GNU make 4.2.1).
So no problem after all?

Thanks,
Ludo’.

bug#27284: [PATCH 1/8] build: Factorize module compilation in (guix build compile).

[Eric Bavier]

On Sun, 22 Oct 2017 18:50:57 -0700
ludo@gnu.org (Ludovic Courtès) wrote:

> Hi Maxim,
> 
> Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:
> 
> > Ludovic Courtès <ludo@gnu.org> writes:
> >  
> >> * guix/build/compile.scm: New file.
> >> * Makefile.am (MODULES): Add it.
> >> * build-aux/compile-all.scm: Use it.
> >> (warnings, file->module, load-module-file)
> >> (%default-optimizations, %lightweight-optimizations)
> >> (optimization-options, compile-file*): Remove.
> >> <top level>: Use 'compile-files'.
> >> * guix/build/pull.scm (%default-optimizations)
> >> (%lightweight-optimizations, optimization-options): Remove.
> >> (build-guix): Rewrite as a call to 'compile-files'.
> >> * guix/discovery.scm (file-name->module-name): Export.
> >> ---
> >>  Makefile.am               |   1 +
> >>  build-aux/compile-all.scm |  92 ++++----------------------
> >>  guix/build/compile.scm    | 165 ++++++++++++++++++++++++++++++++++++++++++++++
> >>  guix/build/pull.scm       | 105 ++++++++---------------------
> >>  guix/discovery.scm        |   4 +-
> >>  5 files changed, 209 insertions(+), 158 deletions(-)
> >>  create mode 100644 guix/build/compile.scm  
> >
> > [...]
> >
> > This and the other patches in this series look fine to me.  
> 
> Awesome, thanks for looking.
> 
[...]
> Speaking of the “make -j” patch, on IRC Eric Bavier suggested that it
> didn’t work as advertised because $MAKEFLAGS didn’t contain the “-j”
> flag; I just checked and it does contain it here (with GNU make 4.2.1).
> So no problem after all?

The behavior changed in version 4.2.  From the Changelog:

2016-04-04  Paul Smith  <psmith@gnu.org>

        Preserve the real value of -jN in MAKEFLAGS using jobserver.
        Previously if the jobserver was active, MAKEFLAGS would contain only
        the -j option but not the number (not -j5 or whatever) so users
        could not discover that value.  Allow that value to be provided in
        MAKEFLAGS without error but still give warnings if -jN is provided
        on the command line if the jobserver is already activated.

        * NEWS: Discuss the new behavior.
        * os.h, posixos.c, w32/w32os.c: Return success/failure from
        jobserver_setup() and jobserver_parse_auth().
        * main.c (main): Separate the command line storage of job slots (now
        in arg_job_slots) from the control storage (in job_slots).  Make a
        distinction between -jN flags read from MAKEFLAGS and those seen
        on the command line: for the latter if the jobserver is enabled then
        warn and disable it, as before.
        * tests/scripts/features/jobserver: Add new testing.


And from NEWS for the 4.2 release:

* The amount of parallelism can be determined by querying MAKEFLAGS, even when
  the job server is enabled (previously MAKEFLAGS would always contain only
  "-j", with no number, when job server was enabled).


It Would Be Nice if the functionality worked for older versions of make
that people might have on their systems.  Using the jobserver directly
would require quite a bit of work for the current patch set, but I
wonder if there is another way to determine the -jN parameter for
make<4.2 that we could use.  Maybe simply polling the jobserver fds at
the start?

`~Eric

bug#27284: [PATCH 1/8] build: Factorize module compilation in (guix build compile).

[Ludovic Courtès]

Heya,

Eric Bavier <ericbavier@centurylink.net> skribis:

> And from NEWS for the 4.2 release:
>
> * The amount of parallelism can be determined by querying MAKEFLAGS, even when
>   the job server is enabled (previously MAKEFLAGS would always contain only
>   "-j", with no number, when job server was enabled).

Thanks for digging!  I didn’t expect such things to ever change.

> It Would Be Nice if the functionality worked for older versions of make
> that people might have on their systems.

With the patch I posted, Scheme compilation would always use one thread
per core, which is what it currently does in ‘master’.

That’s in argument in favor of the lazy in me: it’s an improvement for
people using a reasonably recent system, and it’s not a regression for
the others!  :-)

WDYT?

> Using the jobserver directly would require quite a bit of work for the
> current patch set, but I wonder if there is another way to determine
> the -jN parameter for make<4.2 that we could use.  Maybe simply
> polling the jobserver fds at the start?

AIUI the job server does not reveal how many jobs are allowed.  It
merely grants you an execution token.

Or did you have something else in mind?

Thanks,
Ludo’.

bug#27284: [PATCH 1/8] build: Factorize module compilation in (guix build compile).

[Eric Bavier]

On Sun, 22 Oct 2017 19:51:14 -0700
ludo@gnu.org (Ludovic Courtès) wrote:

> Heya,
> 
> Eric Bavier <ericbavier@centurylink.net> skribis:
> 
> > And from NEWS for the 4.2 release:
> >
> > * The amount of parallelism can be determined by querying MAKEFLAGS, even when
> >   the job server is enabled (previously MAKEFLAGS would always contain only
> >   "-j", with no number, when job server was enabled).  
> 
> Thanks for digging!  I didn’t expect such things to ever change.
> 
> > It Would Be Nice if the functionality worked for older versions of make
> > that people might have on their systems.  
> 
> With the patch I posted, Scheme compilation would always use one thread
> per core, which is what it currently does in ‘master’.
> 
> That’s in argument in favor of the lazy in me: it’s an improvement for
> people using a reasonably recent system, and it’s not a regression for
> the others!  :-)
> 
> WDYT?

Indeed.  Understood like that, it makes sense.

> 
> > Using the jobserver directly would require quite a bit of work for the
> > current patch set, but I wonder if there is another way to determine
> > the -jN parameter for make<4.2 that we could use.  Maybe simply
> > polling the jobserver fds at the start?  
> 
> AIUI the job server does not reveal how many jobs are allowed.  It
> merely grants you an execution token.
> 
> Or did you have something else in mind?

The idea (hardely tested) would be to read tokens from the input fd
until it blocks, do the scheme compiles with however many tokens were
read, then write them back out.  Crude, I guess, and probably error
prone; compile-all.scm could be invoked when make has job tokens tied
up building the daemon source...

Anyhow, the current patch works well for me with a recent make.  I'm
content leaving further improvements to a future hypothetical hacker. :)

`~Eric

bug#27284: [PATCH 1/8] build: Factorize module compilation in (guix build compile).

[Ludovic Courtès]

[-- Attachment #1: Type: text/plain, Size: 501 bytes --]

Eric Bavier <ericbavier@centurylink.net> skribis:

> On Sun, 22 Oct 2017 19:51:14 -0700
> ludo@gnu.org (Ludovic Courtès) wrote:

[...]

>> > It Would Be Nice if the functionality worked for older versions of make
>> > that people might have on their systems.  
>> 
>> With the patch I posted, Scheme compilation would always use one thread
>> per core, which is what it currently does in ‘master’.

Oops, that was not quite true: it would use one thread.

I fixed it like this:


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Type: text/x-patch, Size: 2368 bytes --]

diff --git a/build-aux/compile-all.scm b/build-aux/compile-all.scm
index 4aa4ac9b9..c7ca5a6f6 100644
--- a/build-aux/compile-all.scm
+++ b/build-aux/compile-all.scm
@@ -19,6 +19,7 @@
 
 (use-modules (ice-9 match)
              (ice-9 threads)
+             (srfi srfi-1)
              (guix build compile)
              (guix build utils))
 
@@ -50,24 +51,32 @@ to 'make'."
   (match flags
     (#f (current-processor-count))
     (flags
-     (let loop ((flags (string-tokenize flags)))
-       (match flags
-         (()
-          1)
-         (("-j" (= string->number count) _ ...)
-          (if (integer? count)
-              count
-              (current-processor-count)))
-         ((head tail ...)
-          (if (string-prefix? "-j" head)
-              (match (string-drop head 2)
-                (""
-                 (current-processor-count))
-                ((= string->number count)
-                 (if (integer? count)
-                     count
-                     (current-processor-count))))
-              (loop tail))))))))
+     (let ((initial-flags (string-tokenize flags)))
+       (let loop ((flags initial-flags))
+         (match flags
+           (()
+            ;; Note: GNU make prior to version 4.2 would hide "-j" flags from
+            ;; $MAKEFLAGS.  Thus, check for a "--jobserver" flag here and
+            ;; assume we're using all cores if specified.
+            (if (any (lambda (flag)
+                       (string-prefix? "--jobserver" flag))
+                     initial-flags)
+                (current-processor-count)         ;GNU make < 4.2
+                1))                               ;sequential make
+           (("-j" (= string->number count) _ ...)
+            (if (integer? count)
+                count
+                (current-processor-count)))
+           ((head tail ...)
+            (if (string-prefix? "-j" head)
+                (match (string-drop head 2)
+                  (""
+                   (current-processor-count))
+                  ((= string->number count)
+                   (if (integer? count)
+                       count
+                       (current-processor-count))))
+                (loop tail)))))))))
 
 ;; Install a SIGINT handler to give unwind handlers in 'compile-file' an
 ;; opportunity to run upon SIGINT and to remove temporary output files.

[-- Attachment #3: Type: text/plain, Size: 1182 bytes --]


>> > Using the jobserver directly would require quite a bit of work for the
>> > current patch set, but I wonder if there is another way to determine
>> > the -jN parameter for make<4.2 that we could use.  Maybe simply
>> > polling the jobserver fds at the start?  
>> 
>> AIUI the job server does not reveal how many jobs are allowed.  It
>> merely grants you an execution token.
>> 
>> Or did you have something else in mind?
>
> The idea (hardely tested) would be to read tokens from the input fd
> until it blocks, do the scheme compiles with however many tokens were
> read, then write them back out.  Crude, I guess, and probably error
> prone; compile-all.scm could be invoked when make has job tokens tied
> up building the daemon source...
>
> Anyhow, the current patch works well for me with a recent make.  I'm
> content leaving further improvements to a future hypothetical hacker. :)

Sounds reasonable.  Let’s make sure the hypothetical hacker has enough
on their plate.  :-)

I’ve pushed this and the easy parts of this patch series, with commit
d298c815e638581d466222f3a883b280f019b368 as the tip.

Thanks for the review!

Ludo’.

bug#27284: [PATCH 2/8] build: Honor make's '-j' flag.

[Ludovic Courtès]

* build-aux/compile-all.scm (parallel-job-count): New procedure.
<top level>: Pass it to 'compile-files' as #:workers.
---
 build-aux/compile-all.scm | 26 ++++++++++++++++++++++++++
 1 file changed, 26 insertions(+)

diff --git a/build-aux/compile-all.scm b/build-aux/compile-all.scm
index 2fc3102da..4aa4ac9b9 100644
--- a/build-aux/compile-all.scm
+++ b/build-aux/compile-all.scm
@@ -44,6 +44,31 @@
     (or (not (file-exists? go))
         (file-mtime<? go file))))
 
+(define* (parallel-job-count #:optional (flags (getenv "MAKEFLAGS")))
+  "Return the number of parallel jobs as determined by FLAGS, the flags passed
+to 'make'."
+  (match flags
+    (#f (current-processor-count))
+    (flags
+     (let loop ((flags (string-tokenize flags)))
+       (match flags
+         (()
+          1)
+         (("-j" (= string->number count) _ ...)
+          (if (integer? count)
+              count
+              (current-processor-count)))
+         ((head tail ...)
+          (if (string-prefix? "-j" head)
+              (match (string-drop head 2)
+                (""
+                 (current-processor-count))
+                ((= string->number count)
+                 (if (integer? count)
+                     count
+                     (current-processor-count))))
+              (loop tail))))))))
+
 ;; Install a SIGINT handler to give unwind handlers in 'compile-file' an
 ;; opportunity to run upon SIGINT and to remove temporary output files.
 (sigaction SIGINT
@@ -54,6 +79,7 @@
   ((_ . files)
    (compile-files srcdir (getcwd)
                   (filter file-needs-compilation? files)
+                  #:workers (parallel-job-count)
                   #:host host
                   #:report-load (lambda (file total completed)
                                   (when file
-- 
2.14.2

bug#27284: [PATCH 3/8] discovery: Move 'file-name->module-name' to (guix modules).

[Ludovic Courtès]

* guix/discovery.scm (file-name->module-name): Move to...
* guix/modules.scm (file-name->module-name): ... here.
* guix/build/compile.scm: Use (guix modules) instead of (guix discovery).
---
 guix/build/compile.scm |  2 +-
 guix/discovery.scm     | 12 ++----------
 guix/modules.scm       | 10 ++++++++++
 3 files changed, 13 insertions(+), 11 deletions(-)

diff --git a/guix/build/compile.scm b/guix/build/compile.scm
index 6f15ba578..ea0c36fa3 100644
--- a/guix/build/compile.scm
+++ b/guix/build/compile.scm
@@ -24,7 +24,7 @@
   #:use-module (system base target)
   #:use-module (system base compile)
   #:use-module (system base message)
-  #:use-module (guix discovery)
+  #:use-module (guix modules)
   #:use-module (guix build utils)
   #:export (%default-optimizations
             %lightweight-optimizations
diff --git a/guix/discovery.scm b/guix/discovery.scm
index c861614b8..7b5757902 100644
--- a/guix/discovery.scm
+++ b/guix/discovery.scm
@@ -18,15 +18,14 @@
 
 (define-module (guix discovery)
   #:use-module (guix ui)
+  #:use-module (guix modules)
   #:use-module (guix combinators)
   #:use-module (guix build syscalls)
   #:use-module (srfi srfi-1)
   #:use-module (ice-9 match)
   #:use-module (ice-9 vlist)
   #:use-module (ice-9 ftw)
-  #:export (file-name->module-name
-
-            scheme-modules
+  #:export (scheme-modules
             fold-modules
             all-modules
             fold-module-public-variables))
@@ -90,13 +89,6 @@ DIRECTORY is not accessible."
                                directory (strerror errno)))
                     '())))))
 
-(define file-name->module-name
-  (let ((not-slash (char-set-complement (char-set #\/))))
-    (lambda (file)
-      "Return the module name (a list of symbols) corresponding to FILE."
-      (map string->symbol
-           (string-tokenize (string-drop-right file 4) not-slash)))))
-
 (define* (scheme-modules directory #:optional sub-directory)
   "Return the list of Scheme modules available under DIRECTORY.
 Optionally, narrow the search to SUB-DIRECTORY."
diff --git a/guix/modules.scm b/guix/modules.scm
index 19a4acd76..6c602eda4 100644
--- a/guix/modules.scm
+++ b/guix/modules.scm
@@ -26,6 +26,9 @@
   #:export (missing-dependency-error?
             missing-dependency-module
 
+            file-name->module-name
+            module-name->file-name
+
             source-module-closure
             live-module-closure
             guix-module-name?))
@@ -93,6 +96,13 @@ depends on."
           (_
            '()))))))
 
+(define file-name->module-name
+  (let ((not-slash (char-set-complement (char-set #\/))))
+    (lambda (file)
+      "Return the module name (a list of symbols) corresponding to FILE."
+      (map string->symbol
+           (string-tokenize (string-drop-right file 4) not-slash)))))
+
 (define (module-name->file-name module)
   "Return the file name for MODULE."
   (string-append (string-join (map symbol->string module) "/")
-- 
2.14.2

bug#27284: [PATCH 4/8] gexp: Add 'file-union'.

[Ludovic Courtès]

* gnu/services.scm (file-union): Move to...
* guix/gexp.scm (file-union): ... here.  New procedure.
* doc/guix.texi (G-Expressions): Document it.
---
 doc/guix.texi    | 17 +++++++++++++++++
 gnu/services.scm | 20 --------------------
 guix/gexp.scm    | 32 ++++++++++++++++++++++++++++++++
 3 files changed, 49 insertions(+), 20 deletions(-)

diff --git a/doc/guix.texi b/doc/guix.texi
index b7f4f88f9..1de3494da 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -4990,6 +4990,23 @@ as in:
 This is the declarative counterpart of @code{text-file*}.
 @end deffn
 
+@deffn {Scheme Procedure} file-union @var{name} @var{files}
+Return a @code{<computed-file>} that builds a directory containing all of @var{files}.
+Each item in @var{files} must be a two-element list where the first element is the
+file name to use in the new directory, and the second element is a gexp
+denoting the target file.  Here's an example:
+
+@example
+(file-union "etc"
+            `(("hosts" ,(plain-file "hosts"
+                                    "127.0.0.1 localhost"))
+              ("bashrc" ,(plain-file "bashrc"
+                                     "alias ls='ls --color'"))))
+@end example
+
+This yields an @code{etc} directory containing these two files.
+@end deffn
+
 @deffn {Scheme Procedure} file-append @var{obj} @var{suffix} @dots{}
 Return a file-like object that expands to the concatenation of @var{obj}
 and @var{suffix}, where @var{obj} is a lowerable object and each
diff --git a/gnu/services.scm b/gnu/services.scm
index 0bd362085..bc866eafe 100644
--- a/gnu/services.scm
+++ b/gnu/services.scm
@@ -97,7 +97,6 @@
             %activation-service
             etc-service
 
-            file-union                        ;XXX: for lack of a better place
             directory-union))
 
 ;;; Comment:
@@ -388,25 +387,6 @@ boot."
                  (list (service-extension boot-service-type
                                           cleanup-gexp)))))
 
-(define* (file-union name files)                  ;FIXME: Factorize.
-  "Return a <computed-file> that builds a directory containing all of FILES.
-Each item in FILES must be a list where the first element is the file name to
-use in the new directory, and the second element is a gexp denoting the target
-file."
-  (computed-file name
-                 #~(begin
-                     (mkdir #$output)
-                     (chdir #$output)
-                     #$@(map (match-lambda
-                               ((target source)
-                                #~(begin
-                                    ;; Stat the source to abort early if it
-                                    ;; does not exist.
-                                    (stat #$source)
-
-                                    (symlink #$source #$target))))
-                             files))))
-
 (define (directory-union name things)
   "Return a directory that is the union of THINGS."
   (match things
diff --git a/guix/gexp.scm b/guix/gexp.scm
index 2622c5cb6..9835599bb 100644
--- a/guix/gexp.scm
+++ b/guix/gexp.scm
@@ -78,6 +78,7 @@
             gexp->script
             text-file*
             mixed-text-file
+            file-union
             imported-files
             imported-modules
             compiled-modules
@@ -1171,6 +1172,37 @@ This is the declarative counterpart of 'text-file*'."
 
   (computed-file name build))
 
+(define (file-union name files)
+  "Return a <computed-file> that builds a directory containing all of FILES.
+Each item in FILES must be a two-element list where the first element is the
+file name to use in the new directory, and the second element is a gexp
+denoting the target file.  Here's an example:
+
+  (file-union \"etc\"
+              `((\"hosts\" ,(plain-file \"hosts\"
+                                        \"127.0.0.1 localhost\"))
+                (\"bashrc\" ,(plain-file \"bashrc\"
+                                         \"alias ls='ls --color'\"))))
+
+This yields an 'etc' directory containing these two files."
+  (computed-file name
+                 (gexp
+                  (begin
+                    (mkdir (ungexp output))
+                    (chdir (ungexp output))
+                    (ungexp-splicing
+                     (map (match-lambda
+                            ((target source)
+                             (gexp
+                              (begin
+                                ;; Stat the source to abort early if it does
+                                ;; not exist.
+                                (stat (ungexp source))
+
+                                (symlink (ungexp source)
+                                         (ungexp target))))))
+                          files))))))
+
 \f
 ;;;
 ;;; Syntactic sugar.
-- 
2.14.2

bug#27284: [PATCH 5/8] gexp: Add 'directory-union'.

[Ludovic Courtès]

* gnu/services.scm (directory-union): Move to...
* guix/gexp.scm (directory-union): ... here.  New procedure.
* doc/guix.texi (G-Expressions): Document it.
---
 doc/guix.texi    | 11 +++++++++++
 gnu/services.scm | 17 +----------------
 guix/gexp.scm    | 20 ++++++++++++++++++++
 3 files changed, 32 insertions(+), 16 deletions(-)

diff --git a/doc/guix.texi b/doc/guix.texi
index 1de3494da..55a252014 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -5007,6 +5007,17 @@ denoting the target file.  Here's an example:
 This yields an @code{etc} directory containing these two files.
 @end deffn
 
+@deffn {Scheme Procedure} directory-union @var{name} @var{things}
+Return a directory that is the union of @var{things}, where @var{things} is a list of
+file-like objects denoting directories.  For example:
+
+@example
+(directory-union "guile+emacs" (list guile emacs))
+@end example
+
+yields a directory that is the union of the @code{guile} and @code{emacs} packages.
+@end deffn
+
 @deffn {Scheme Procedure} file-append @var{obj} @var{suffix} @dots{}
 Return a file-like object that expands to the concatenation of @var{obj}
 and @var{suffix}, where @var{obj} is a lowerable object and each
diff --git a/gnu/services.scm b/gnu/services.scm
index bc866eafe..50be28a38 100644
--- a/gnu/services.scm
+++ b/gnu/services.scm
@@ -95,9 +95,7 @@
 
             %boot-service
             %activation-service
-            etc-service
-
-            directory-union))
+            etc-service))
 
 ;;; Comment:
 ;;;
@@ -387,19 +385,6 @@ boot."
                  (list (service-extension boot-service-type
                                           cleanup-gexp)))))
 
-(define (directory-union name things)
-  "Return a directory that is the union of THINGS."
-  (match things
-    ((one)
-     ;; Only one thing; return it.
-     one)
-    (_
-     (computed-file name
-                    (with-imported-modules '((guix build union))
-                      #~(begin
-                          (use-modules (guix build union))
-                          (union-build #$output '#$things)))))))
-
 (define* (activation-service->script service)
   "Return as a monadic value the activation script for SERVICE, a service of
 ACTIVATION-SCRIPT-TYPE."
diff --git a/guix/gexp.scm b/guix/gexp.scm
index 9835599bb..b9525603e 100644
--- a/guix/gexp.scm
+++ b/guix/gexp.scm
@@ -79,6 +79,7 @@
             text-file*
             mixed-text-file
             file-union
+            directory-union
             imported-files
             imported-modules
             compiled-modules
@@ -1203,6 +1204,25 @@ This yields an 'etc' directory containing these two files."
                                          (ungexp target))))))
                           files))))))
 
+(define (directory-union name things)
+  "Return a directory that is the union of THINGS, where THINGS is a list of
+file-like objects denoting directories.  For example:
+
+  (directory-union \"guile+emacs\" (list guile emacs))
+
+yields a directory that is the union of the 'guile' and 'emacs' packages."
+  (match things
+    ((one)
+     ;; Only one thing; return it.
+     one)
+    (_
+     (computed-file name
+                    (with-imported-modules '((guix build union))
+                      (gexp (begin
+                              (use-modules (guix build union))
+                              (union-build (ungexp output)
+                                           '(ungexp things)))))))))
+
 \f
 ;;;
 ;;; Syntactic sugar.
-- 
2.14.2

bug#27284: [PATCH 6/8] union: Parametrize the symlink procedure .

[Ludovic Courtès]

* guix/gexp.scm (directory-union): Add #:hard-links and honor it.
* guix/build/union.scm (union-build): Add #:symlink parameter.
---
 guix/build/union.scm | 11 ++++++-----
 guix/gexp.scm        | 19 ++++++++++++++++---
 2 files changed, 22 insertions(+), 8 deletions(-)

diff --git a/guix/build/union.scm b/guix/build/union.scm
index 18167fa3e..256123c56 100644
--- a/guix/build/union.scm
+++ b/guix/build/union.scm
@@ -1,5 +1,5 @@
 ;;; GNU Guix --- Functional package management for GNU
-;;; Copyright © 2012, 2013, 2014, 2016 Ludovic Courtès <ludo@gnu.org>
+;;; Copyright © 2012, 2013, 2014, 2016, 2017 Ludovic Courtès <ludo@gnu.org>
 ;;; Copyright © 2014 Mark H Weaver <mhw@netris.org>
 ;;; Copyright © 2017 Huang Ying <huang.ying.caritas@gmail.com>
 ;;;
@@ -78,11 +78,12 @@ identical, #f otherwise."
 
 (define* (union-build output inputs
                       #:key (log-port (current-error-port))
-                      (create-all-directories? #f))
+                      (create-all-directories? #f)
+                      (symlink symlink))
   "Build in the OUTPUT directory a symlink tree that is the union of all the
-INPUTS.  As a special case, if CREATE-ALL-DIRECTORIES?, creates the
-subdirectories in the output directory to make sure the caller can modify them
-later."
+INPUTS, using SYMLINK to create symlinks.  As a special case, if
+CREATE-ALL-DIRECTORIES?, creates the subdirectories in the output directory to
+make sure the caller can modify them later."
 
   (define (symlink* input output)
     (format log-port "`~a' ~~> `~a'~%" input output)
diff --git a/guix/gexp.scm b/guix/gexp.scm
index b9525603e..e8ac3dcdc 100644
--- a/guix/gexp.scm
+++ b/guix/gexp.scm
@@ -1204,13 +1204,24 @@ This yields an 'etc' directory containing these two files."
                                          (ungexp target))))))
                           files))))))
 
-(define (directory-union name things)
+(define* (directory-union name things
+                          #:key (copy? #f))
   "Return a directory that is the union of THINGS, where THINGS is a list of
 file-like objects denoting directories.  For example:
 
   (directory-union \"guile+emacs\" (list guile emacs))
 
-yields a directory that is the union of the 'guile' and 'emacs' packages."
+yields a directory that is the union of the 'guile' and 'emacs' packages.
+
+When COPY? is true, copy files instead of creating symlinks."
+  (define symlink
+    (if copy?
+        (gexp (lambda (old new)
+                (if (file-is-directory? old)
+                    (symlink old new)
+                    (copy-file old new))))
+        (gexp symlink)))
+
   (match things
     ((one)
      ;; Only one thing; return it.
@@ -1221,7 +1232,9 @@ yields a directory that is the union of the 'guile' and 'emacs' packages."
                       (gexp (begin
                               (use-modules (guix build union))
                               (union-build (ungexp output)
-                                           '(ungexp things)))))))))
+                                           '(ungexp things)
+
+                                           #:symlink (ungexp symlink)))))))))
 
 \f
 ;;;
-- 
2.14.2

bug#27284: [PATCH 7/8] gexp: 'directory-union' has a #:quiet? parameter.

[Ludovic Courtès]

* guix/gexp.scm (directory-union): Add #:quiet? and honor it.
---
 guix/gexp.scm | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)

diff --git a/guix/gexp.scm b/guix/gexp.scm
index e8ac3dcdc..3781a1e6e 100644
--- a/guix/gexp.scm
+++ b/guix/gexp.scm
@@ -1205,7 +1205,7 @@ This yields an 'etc' directory containing these two files."
                           files))))))
 
 (define* (directory-union name things
-                          #:key (copy? #f))
+                          #:key (copy? #f) (quiet? #f))
   "Return a directory that is the union of THINGS, where THINGS is a list of
 file-like objects denoting directories.  For example:
 
@@ -1213,7 +1213,8 @@ file-like objects denoting directories.  For example:
 
 yields a directory that is the union of the 'guile' and 'emacs' packages.
 
-When COPY? is true, copy files instead of creating symlinks."
+When HARD-LINKS? is true, create hard links instead of symlinks.  When QUIET?
+is true, the derivation will not print anything."
   (define symlink
     (if copy?
         (gexp (lambda (old new)
@@ -1222,6 +1223,11 @@ When COPY? is true, copy files instead of creating symlinks."
                     (copy-file old new))))
         (gexp symlink)))
 
+  (define log-port
+    (if quiet?
+        (gexp (%make-void-port "w"))
+        (gexp (current-error-port))))
+
   (match things
     ((one)
      ;; Only one thing; return it.
@@ -1234,6 +1240,7 @@ When COPY? is true, copy files instead of creating symlinks."
                               (union-build (ungexp output)
                                            '(ungexp things)
 
+                                           #:log-port (ungexp log-port)
                                            #:symlink (ungexp symlink)))))))))
 
 \f
-- 
2.14.2

bug#27284: [PATCH 8/8] DRAFT Add (guix self) and use it when pulling.

[Ludovic Courtès]

DRAFT: Module reloading doesn't work; needs more testing.

Partly addresses <https://bugs.gnu.org/27284>.

* guix/self.scm: New file.
* Makefile.am (MODULES): Add it.
* build-aux/build-self.scm (libgcrypt, zlib, gzip, bzip2, xz)
(false-if-wrong-guile, package-for-current-guile, guile-json)
(guile-ssh, guile-git, guile-bytestructures): Remove.
(build): Rewrite to simply delegate to 'compiled-guix'.
* gnu/packages.scm (%distro-root-directory): Rewrite to try difference
directories.
* guix/discovery.scm (guix): Export 'scheme-files'.
* guix/scripts/pull.scm (guix-pull): Reopen a connection to the daemon
after the call to 'build-and-install'.
---
 Makefile.am              |   1 +
 build-aux/build-self.scm | 227 ++----------------
 gnu/packages.scm         |  21 +-
 guix/discovery.scm       |   3 +-
 guix/scripts/pull.scm    |  16 +-
 guix/self.scm            | 599 +++++++++++++++++++++++++++++++++++++++++++++++
 6 files changed, 652 insertions(+), 215 deletions(-)
 create mode 100644 guix/self.scm

diff --git a/Makefile.am b/Makefile.am
index fd6f9729c..bb12e9905 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -66,6 +66,7 @@ MODULES =					\
   guix/derivations.scm				\
   guix/grafts.scm				\
   guix/gnu-maintenance.scm			\
+  guix/self.scm					\
   guix/upstream.scm				\
   guix/licenses.scm				\
   guix/git.scm					\
diff --git a/build-aux/build-self.scm b/build-aux/build-self.scm
index 4933e0271..bb9771332 100644
--- a/build-aux/build-self.scm
+++ b/build-aux/build-self.scm
@@ -17,9 +17,6 @@
 ;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
 
 (define-module (build-self)
-  #:use-module (gnu)
-  #:use-module (guix)
-  #:use-module (guix config)
   #:use-module (srfi srfi-1)
   #:use-module (srfi srfi-19)
   #:use-module (ice-9 match)
@@ -39,98 +36,25 @@
 ;;;
 ;;; Code:
 
-\f
-;; The dependencies.  Don't refer explicitly to the variables because they
-;; could be renamed or shuffled around in modules over time.  Conversely,
-;; 'find-best-packages-by-name' is expected to always have the same semantics.
+;; Use our very own Guix modules.
+(eval-when (compile load eval)
 
-(define libgcrypt
-  (first (find-best-packages-by-name "libgcrypt" #f)))
+  ;; Ignore any available .go, and force recompilation.  This is because our
+  ;; checkout in the store has mtime set to the epoch, and thus .go files look
+  ;; newer, even though they may not correspond.
+  (set! %fresh-auto-compile #t)
 
-(define zlib
-  (first (find-best-packages-by-name "zlib" #f)))
-
-(define gzip
-  (first (find-best-packages-by-name "gzip" #f)))
-
-(define bzip2
-  (first (find-best-packages-by-name "bzip2" #f)))
-
-(define xz
-  (first (find-best-packages-by-name "xz" #f)))
-
-(define (false-if-wrong-guile package)
-  "Return #f if PACKAGE depends on the \"wrong\" major version of Guile (e.g.,
-2.0 instead of 2.2), otherwise return PACKAGE."
-  (let ((guile (any (match-lambda
-                      ((label (? package? dep) _ ...)
-                       (and (string=? (package-name dep) "guile")
-                            dep)))
-                    (package-direct-inputs package))))
-    (and (or (not guile)
-             (string-prefix? (effective-version)
-                             (package-version guile)))
-         package)))
-
-(define (package-for-current-guile . names)
-  "Return the package with one of the given NAMES that depends on the current
-Guile major version (2.0 or 2.2), or #f if none of the packages matches."
-  (let loop ((names names))
-    (match names
-      (()
-       #f)
-      ((name rest ...)
-       (match (find-best-packages-by-name name #f)
-         (()
-          (loop rest))
-         ((first _ ...)
-          (or (false-if-wrong-guile first)
-              (loop rest))))))))
-
-(define guile-json
-  (package-for-current-guile "guile-json"
-                             "guile2.2-json"
-                             "guile2.0-json"))
-
-(define guile-ssh
-  (package-for-current-guile "guile-ssh"
-                             "guile2.2-ssh"
-                             "guile2.0-ssh"))
-
-(define guile-git
-  (package-for-current-guile "guile-git"
-                             "guile2.0-git"))
-
-(define guile-bytestructures
-  (package-for-current-guile "guile-bytestructures"
-                             "guile2.0-bytestructures"))
-\f
-;; The actual build procedure.
-
-(define (top-source-directory)
-  "Return the name of the top-level directory of this source tree."
   (and=> (assoc-ref (current-source-location) 'filename)
          (lambda (file)
-           (string-append (dirname file) "/.."))))
-
+           (let ((dir (string-append (dirname file) "/../..")))
+             (set! %load-path (cons dir %load-path))))))
 
 (define (date-version-string)
   "Return the current date and hour in UTC timezone, for use as a poor
 person's version identifier."
-  ;; XXX: Replace with a Git commit id.
+  ;; XXX: Last resort when the Git commit id is missing.
   (date->string (current-date 0) "~Y~m~d.~H"))
 
-(define (guile-for-build)
-  "Return a derivation for Guile 2.0 or 2.2, whichever matches the currently
-running Guile."
-  (package->derivation (cond-expand
-                         (guile-2.2
-                          (canonical-package
-                           (specification->package "guile@2.2")))
-                         (else
-                          (canonical-package
-                           (specification->package "guile@2.0"))))))
-
 ;; The procedure below is our return value.
 (define* (build source
                 #:key verbose? (version (date-version-string))
@@ -138,130 +62,19 @@ running Guile."
                 #:rest rest)
   "Return a derivation that unpacks SOURCE into STORE and compiles Scheme
 files."
-  ;; The '%xxxdir' variables were added to (guix config) in July 2016 so we
-  ;; cannot assume that they are defined.  Try to guess their value when
-  ;; they're undefined (XXX: we get an incorrect guess when environment
-  ;; variables such as 'NIX_STATE_DIR' are defined!).
-  (define storedir
-    (if (defined? '%storedir) %storedir %store-directory))
-  (define localstatedir
-    (if (defined? '%localstatedir) %localstatedir (dirname %state-directory)))
-  (define sysconfdir
-    (if (defined? '%sysconfdir) %sysconfdir (dirname %config-directory)))
-  (define sbindir
-    (if (defined? '%sbindir) %sbindir (dirname %guix-register-program)))
-
-  (define builder
-    #~(begin
-        (use-modules (guix build pull))
-
-        (letrec-syntax ((maybe-load-path
-                         (syntax-rules ()
-                           ((_ item rest ...)
-                            (let ((tail (maybe-load-path rest ...)))
-                              (if (string? item)
-                                  (cons (string-append item
-                                                       "/share/guile/site/"
-                                                       #$(effective-version))
-                                        tail)
-                                  tail)))
-                           ((_)
-                            '()))))
-          (set! %load-path
-                (append
-                 (maybe-load-path #$guile-json #$guile-ssh
-                                  #$guile-git #$guile-bytestructures)
-                 %load-path)))
-
-        (letrec-syntax ((maybe-load-compiled-path
-                         (syntax-rules ()
-                           ((_ item rest ...)
-                            (let ((tail (maybe-load-compiled-path rest ...)))
-                              (if (string? item)
-                                  (cons (string-append item
-                                                       "/lib/guile/"
-                                                       #$(effective-version)
-                                                       "/site-ccache")
-                                        tail)
-                                  tail)))
-                           ((_)
-                            '()))))
-          (set! %load-compiled-path
-                (append
-                 (maybe-load-compiled-path #$guile-json #$guile-ssh
-                                           #$guile-git #$guile-bytestructures)
-                 %load-compiled-path)))
-
-        ;; XXX: The 'guile-ssh' package prior to Guix commit 92b7258 was
-        ;; broken: libguile-ssh could not be found.  Work around that.
-        ;; FIXME: We want Guile-SSH 0.10.2 or later anyway.
-        #$(if (string-prefix? "0.9." (package-version guile-ssh))
-              #~(setenv "LTDL_LIBRARY_PATH" (string-append #$guile-ssh "/lib"))
-              #t)
-
-        (build-guix #$output #$source
-
-                    #:system #$%system
-                    #:storedir #$storedir
-                    #:localstatedir #$localstatedir
-                    #:sysconfdir #$sysconfdir
-                    #:sbindir #$sbindir
-
-                    #:package-name #$%guix-package-name
-                    #:package-version #$version
-                    #:bug-report-address #$%guix-bug-report-address
-                    #:home-page-url #$%guix-home-page-url
-
-                    #:libgcrypt #$libgcrypt
-                    #:zlib #$zlib
-                    #:gzip #$gzip
-                    #:bzip2 #$bzip2
-                    #:xz #$xz
-
-                    ;; XXX: This is not perfect, enabling VERBOSE? means
-                    ;; building a different derivation.
-                    #:debug-port (if #$verbose?
-                                     (current-error-port)
-                                     (%make-void-port "w")))))
-
-  (unless guile-git
-    ;; XXX: Guix before February 2017 lacks a 'guile-git' package altogether.
-    ;; If we try to upgrade anyway, the logic in (guix scripts pull) will not
-    ;; build (guix git), which will leave us with an unusable 'guix pull'.  To
-    ;; avoid that, fail early.
-    (format (current-error-port)
-            "\
-Your installation is too old and lacks a '~a' package.
-Please upgrade to an intermediate version first, for instance with:
-
-  guix pull --url=https://git.savannah.gnu.org/cgit/guix.git/snapshot/v0.13.0.tar.gz
-\n"
-            (match (effective-version)
-              ("2.0" "guile2.0-git")
-              (_     "guile-git")))
-    (exit 1))
-
-  (mlet %store-monad ((guile (guile-for-build)))
-    (gexp->derivation "guix-latest" builder
-                      #:modules '((guix build pull)
-                                  (guix build utils)
-
-                                  ;; Closure of (guix modules).
-                                  (guix modules)
-                                  (guix memoization)
-                                  (guix sets))
-
-                      ;; Arrange so that our own (guix build …) modules are
-                      ;; used.
-                      #:module-path (list (top-source-directory))
-
-                      #:guile-for-build guile)))
+  ;; FIXME: Closures in (guix scripts pull) have already cached old values of
+  ;; (guix …), so they keep referring to the "old" world, with different
+  ;; record types than the one we get after reloading (because record types
+  ;; are "generative".)
+  ;; (let ((reload-guix (module-ref (resolve-interface '(guix self))
+  ;;                                'reload-guix)))
+  ;;   (reload-guix))                                ;cross fingers!
+
+  (let ((guix-derivation (module-ref (resolve-interface '(guix self))
+                                     'guix-derivation)))
+    (guix-derivation source version)))
 
 ;; This file is loaded by 'guix pull'; return it the build procedure.
 build
 
-;; Local Variables:
-;; eval: (put 'with-load-path 'scheme-indent-function 1)
-;; End:
-
 ;;; build-self.scm ends here
diff --git a/gnu/packages.scm b/gnu/packages.scm
index b4ac6661c..7ca6c3093 100644
--- a/gnu/packages.scm
+++ b/gnu/packages.scm
@@ -110,8 +110,25 @@ for system '~a'")
                         file-name system)))))))
 
 (define %distro-root-directory
-  ;; Absolute file name of the module hierarchy.
-  (dirname (search-path %load-path "guix.scm")))
+  ;; Absolute file name of the module hierarchy.  Since (gnu packages …) might
+  ;; live in a directory different from (guix), try to get the best match.
+  (letrec-syntax ((dirname* (syntax-rules ()
+                              ((_ file)
+                               (dirname file))
+                              ((_ file head tail ...)
+                               (dirname (dirname* file tail ...)))))
+                  (try      (syntax-rules ()
+                              ((_ (file things ...) rest ...)
+                               (match (search-path %load-path file)
+                                 (#f
+                                  (try rest ...))
+                                 (absolute
+                                  (dirname* absolute things ...))))
+                              ((_)
+                               #f))))
+    (try ("gnu/packages/base.scm" gnu/ packages/)
+         ("gnu/packages.scm"      gnu/)
+         ("guix.scm"))))
 
 (define %package-module-path
   ;; Search path for package modules.  Each item must be either a directory
diff --git a/guix/discovery.scm b/guix/discovery.scm
index 7b5757902..8ffcf7cd9 100644
--- a/guix/discovery.scm
+++ b/guix/discovery.scm
@@ -25,7 +25,8 @@
   #:use-module (ice-9 match)
   #:use-module (ice-9 vlist)
   #:use-module (ice-9 ftw)
-  #:export (scheme-modules
+  #:export (scheme-files
+            scheme-modules
             fold-modules
             all-modules
             fold-module-public-variables))
diff --git a/guix/scripts/pull.scm b/guix/scripts/pull.scm
index 240019800..587338c6d 100644
--- a/guix/scripts/pull.scm
+++ b/guix/scripts/pull.scm
@@ -282,10 +282,16 @@ certificates~%"))
                                 (if (assoc-ref opts 'bootstrap?)
                                     %bootstrap-guile
                                     (canonical-package guile-2.0)))))
-                 (run-with-store store
-                   (build-and-install checkout (config-directory)
-                                      #:commit commit
-                                      #:verbose?
-                                      (assoc-ref opts 'verbose?))))))))))))
+                 ;; FIXME: Reopen a connection to work around incompatible
+                 ;; generative record types for the connection, <derivation>,
+                 ;; etc.
+                 (let ((mvalue
+                        (build-and-install checkout (config-directory)
+                                           #:commit commit
+                                           #:verbose?
+                                           (assoc-ref opts 'verbose?))))
+                   (let ((store ((module-ref (resolve-module '(guix store))
+                                             'open-connection))))
+                     (run-with-store store mvalue))))))))))))
 
 ;;; pull.scm ends here
diff --git a/guix/self.scm b/guix/self.scm
new file mode 100644
index 000000000..0e867db3f
--- /dev/null
+++ b/guix/self.scm
@@ -0,0 +1,599 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2017 Ludovic Courtès <ludo@gnu.org>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (guix self)
+  #:use-module (guix config)
+  #:use-module (guix modules)
+  #:use-module (guix gexp)
+  #:use-module (guix store)
+  #:use-module (guix monads)
+  #:use-module (guix discovery)
+  #:use-module (guix packages)
+  #:use-module (guix sets)
+  #:use-module (guix build utils)
+  #:use-module (gnu packages)
+  #:use-module (srfi srfi-1)
+  #:use-module (srfi srfi-9)
+  #:use-module (ice-9 match)
+  #:export (compiled-guix
+            guix-derivation
+            reload-guix))
+
+\f
+;;;
+;;; Dependency handling.
+;;;
+
+(define* (false-if-wrong-guile package
+                               #:optional (guile-version (effective-version)))
+  "Return #f if PACKAGE depends on the \"wrong\" major version of Guile (e.g.,
+2.0 instead of 2.2), otherwise return PACKAGE."
+  (let ((guile (any (match-lambda
+                      ((label (? package? dep) _ ...)
+                       (and (string=? (package-name dep) "guile")
+                            dep)))
+                    (package-direct-inputs package))))
+    (and (or (not guile)
+             (string-prefix? guile-version
+                             (package-version guile)))
+         package)))
+
+(define (package-for-guile guile-version . names)
+  "Return the package with one of the given NAMES that depends on
+GUILE-VERSION (\"2.0\" or \"2.2\"), or #f if none of the packages matches."
+  (let loop ((names names))
+    (match names
+      (()
+       #f)
+      ((name rest ...)
+       (match (specification->package name)
+         (#f
+          (loop rest))
+         ((? package? package)
+          (or (false-if-wrong-guile package)
+              (loop rest))))))))
+
+\f
+;;;
+;;; Derivations.
+;;;
+
+;; Node in a DAG of build tasks.  Each node maps to a derivation, but it's
+;; easier to express things this way.
+(define-record-type <node>
+  (node name modules source dependencies compiled)
+  node?
+  (name          node-name)                       ;string
+  (modules       node-modules)                    ;list of module names
+  (source        node-source)                     ;list of source files
+  (dependencies  node-dependencies)               ;list of nodes
+  (compiled      node-compiled))                  ;node -> lowerable object
+
+(define (node-fold proc init nodes)
+  (let loop ((nodes nodes)
+             (visited (setq))
+             (result init))
+    (match nodes
+      (() result)
+      ((head tail ...)
+       (if (set-contains? visited head)
+           (loop tail visited result)
+           (loop tail (set-insert head visited)
+                 (proc head result)))))))
+
+(define (node-modules/recursive nodes)
+  (node-fold (lambda (node modules)
+               (append (node-modules node) modules))
+             '()
+             nodes))
+
+(define* (closure modules #:optional (except '()))
+  (source-module-closure modules
+                         #:select?
+                         (match-lambda
+                           (('guix 'config)
+                            #f)
+                           ((and module
+                                 (or ('guix _ ...) ('gnu _ ...)))
+                            (not (member module except)))
+                           (rest #f))))
+
+(define module->import
+  ;; Return a file-name/file-like object pair for the specified module and
+  ;; suitable for 'imported-files'.
+  (match-lambda
+    ((module '=> thing)
+     (let ((file (module-name->file-name module)))
+       (list file thing)))
+    (module
+        (let ((file (module-name->file-name module)))
+          (list file
+                (local-file (search-path %load-path file)))))))
+
+(define* (scheme-node name modules #:optional (dependencies '())
+                      #:key (extra-modules '()) (extra-files '())
+                      (extensions '())
+                      parallel?)
+  "Return a node that builds the given Scheme MODULES, and depends on
+DEPENDENCIES (a list of nodes).  EXTRA-MODULES is a list of additional modules
+added to the source, and EXTRA-FILES is a list of additional files.
+EXTENSIONS is a set of full-blow Guile packages (e.g., 'guile-json') that must
+be present in the search path."
+  (let* ((modules (append extra-modules
+                          (closure modules
+                                   (node-modules/recursive dependencies))))
+         (module-files (map module->import modules))
+         (source (imported-files (string-append name "-source")
+                                 (append module-files extra-files))))
+    (node name modules source dependencies
+          (compiled-modules name source modules
+                            (map node-source dependencies)
+                            (map node-compiled dependencies)
+                            #:extensions extensions
+                            #:parallel? parallel?))))
+
+(define (file-imports directory sub-directory pred)
+  "List all the files matching PRED under DIRECTORY/SUB-DIRECTORY.  Return a
+list of file-name/file-like object suitable as input to 'imported-files'."
+  (map (lambda (file)
+         (list (string-drop file (+ 1 (string-length directory)))
+               (local-file file #:recursive? #t)))
+       (find-files (string-append directory "/" sub-directory) pred)))
+
+(define (scheme-modules* directory sub-directory)
+  "Return the list of module names found under SUB-DIRECTORY in DIRECTORY."
+  (let ((prefix (string-length directory)))
+    (map (lambda (file)
+           (file-name->module-name (string-drop file prefix)))
+         (scheme-files (string-append directory "/" sub-directory)))))
+
+(define* (compiled-guix source #:key (version %guix-version)
+                        (guile-version (effective-version))
+                        (libgcrypt (specification->package "libgcrypt"))
+                        (zlib (specification->package "zlib"))
+                        (gzip (specification->package "gzip"))
+                        (bzip2 (specification->package "bzip2"))
+                        (xz (specification->package "xz")))
+  "Return a file-like objects that contains a compiled Guix."
+  (define guile-json
+    (package-for-guile guile-version
+                       "guile-json"
+                       "guile2.2-json"
+                       "guile2.0-json"))
+
+  (define guile-ssh
+    (package-for-guile guile-version
+                       "guile-ssh"
+                       "guile2.2-ssh"
+                       "guile2.0-ssh"))
+
+  (define guile-git
+    (package-for-guile guile-version
+                       "guile-git"
+                       "guile2.0-git"))
+
+
+  (define dependencies
+    (match (append-map (lambda (package)
+                         (cons (list "x" package)
+                               (package-transitive-inputs package)))
+                       (list guile-git guile-json guile-ssh))
+      (((labels packages _ ...) ...)
+       packages)))
+
+  (define *core-modules*
+    (scheme-node "guix-core"
+                 '((guix)
+                   (guix monad-repl)
+                   (guix packages)
+                   (guix download)
+                   (guix discovery)
+                   (guix profiles)
+                   (guix build-system gnu)
+                   (guix build profiles)
+                   (guix build gnu-build-system))
+
+                 ;; Provide a dummy (guix config) with the default version
+                 ;; number, storedir, etc.  This is so that "guix-core" is the
+                 ;; same across all installations and doesn't need to be
+                 ;; rebuilt when the version changes, which in turns means we
+                 ;; can have substitutes for it.
+                 #:extra-modules
+                 `(((guix config)
+                    => ,(make-config.scm #:libgcrypt
+                                         (specification->package "libgcrypt"))))))
+
+  (define *extra-modules*
+    (scheme-node "guix-extra"
+                 (filter-map (match-lambda
+                               (('guix 'scripts _ ..1) #f)
+                               (name name))
+                             (scheme-modules* source "guix"))
+                 (list *core-modules*)
+                 #:extensions dependencies))
+
+  (define *package-modules*
+    (scheme-node "guix-packages"
+                 `((gnu packages)
+                   ,@(scheme-modules* source "gnu/packages"))
+                 (list *core-modules* *extra-modules*)
+                 #:extra-files                    ;all the non-Scheme files
+                 (file-imports source "gnu/packages"
+                               (lambda (file stat)
+                                 (and (eq? 'regular (stat:type stat))
+                                      (not (string-suffix? ".scm" file))
+                                      (not (string-suffix? ".go" file))
+                                      (not (string-prefix? ".#" file))
+                                      (not (string-suffix? "~" file)))))))
+
+  (define *system-modules*
+    (scheme-node "guix-system"
+                 `((gnu system)
+                   (gnu services)
+                   ,@(scheme-modules source "gnu/system")
+                   ,@(scheme-modules source "gnu/services"))
+                 (list *package-modules* *extra-modules* *core-modules*)))
+
+  (define *cli-modules*
+    (scheme-node "guix-cli"
+                 (scheme-modules* source "/guix/scripts")
+                 (list *core-modules* *extra-modules* *package-modules*
+                       *system-modules*)
+                 #:extensions dependencies))
+
+  (define *config*
+    (scheme-node "guix-config"
+                 `(((guix config)
+                    => ,(make-config.scm #:libgcrypt libgcrypt
+                                         #:zlib zlib
+                                         #:gzip gzip
+                                         #:bzip2 bzip2
+                                         #:xz xz
+                                         #:package-name
+                                         %guix-package-name
+                                         #:package-version
+                                         version
+                                         #:bug-report-address
+                                         %guix-bug-report-address
+                                         #:home-page-url
+                                         %guix-home-page-url)))))
+
+  (directory-union (string-append "guix-" %guix-version)
+                   (append-map (lambda (node)
+                                 (list (node-source node)
+                                       (node-compiled node)))
+                               (list *config*
+                                     *cli-modules*
+                                     *system-modules*
+                                     *package-modules*
+                                     *extra-modules*
+                                     *core-modules*))
+
+                   ;; When we do (add-to-store "utils.scm"), "utils.scm" must
+                   ;; be a regular file, not a symlink.  Thus, arrange so that
+                   ;; regular files appear as regular files in the final
+                   ;; output.
+                   #:copy? #t
+                   #:quiet? #t))
+
+\f
+;;;
+;;; (guix config) generation.
+;;;
+
+(define %dependency-variables
+  ;; (guix config) variables corresponding to dependencies.
+  '(%libgcrypt %libz %xz %gzip %bzip2 %nix-instantiate))
+
+(define %config-variables
+  ;; (guix config) variables corresponding to Guix configuration (storedir,
+  ;; localstatedir, etc.)
+  (filter pair?
+          (module-map (lambda (name var)
+                        (and (not (memq name %dependency-variables))
+                             (cons name (variable-ref var))))
+                      (resolve-interface '(guix config)))))
+
+(define* (make-config.scm #:key libgcrypt zlib gzip xz bzip2
+                          (package-name "GNU Guix")
+                          (package-version "0")
+                          (bug-report-address "bug-guix@gnu.org")
+                          (home-page-url "https://gnu.org/s/guix"))
+
+  ;; Hack so that Geiser is not confused.
+  (define defmod 'define-module)
+
+  (scheme-file "config.scm"
+               #~(begin
+                   (#$defmod (guix config)
+                     #:export (%guix-package-name
+                               %guix-version
+                               %guix-bug-report-address
+                               %guix-home-page-url
+                               %libgcrypt
+                               %libz
+                               %gzip
+                               %bzip2
+                               %xz
+                               %nix-instantiate))
+
+                   ;; XXX: Work around <http://bugs.gnu.org/15602>.
+                   (eval-when (expand load eval)
+                     #$@(map (match-lambda
+                               ((name . value)
+                                #~(define-public #$name #$value)))
+                             %config-variables)
+
+                     (define %guix-package-name #$package-name)
+                     (define %guix-version #$package-version)
+                     (define %guix-bug-report-address #$bug-report-address)
+                     (define %guix-home-page-url #$home-page-url)
+
+                     (define %gzip
+                       #+(and gzip (file-append gzip "/bin/gzip")))
+                     (define %bzip2
+                       #+(and bzip2 (file-append bzip2 "/bin/bzip2")))
+                     (define %xz
+                       #+(and xz (file-append xz "/bin/xz")))
+
+                     (define %libgcrypt
+                       #+(and libgcrypt
+                              (file-append libgcrypt "/lib/libgcrypt")))
+                     (define %libz
+                       #+(and zlib
+                              (file-append zlib "/lib/libz")))
+
+                     (define %nix-instantiate     ;for (guix import snix)
+                       "nix-instantiate")))))
+
+
+\f
+;;;
+;;; Building.
+;;;
+
+(define (imported-files name files)
+  ;; This is a non-monadic, simplified version of 'imported-files' from (guix
+  ;; gexp).
+  (define build
+    (with-imported-modules (source-module-closure
+                            '((guix build utils)))
+      #~(begin
+          (use-modules (ice-9 match)
+                       (guix build utils))
+
+          (mkdir (ungexp output)) (chdir (ungexp output))
+          (for-each (match-lambda
+                      ((final-path store-path)
+                       (mkdir-p (dirname final-path))
+
+                       ;; Note: We need regular files to be regular files, not
+                       ;; symlinks, as this makes a difference for
+                       ;; 'add-to-store'.
+                       (copy-file store-path final-path)))
+                    '#$files))))
+
+  (computed-file name build))
+
+(define* (compiled-modules name module-tree modules
+                           #:optional
+                           (dependencies '())
+                           (dependencies-compiled '())
+                           #:key
+                           (extensions '())       ;full-blown Guile packages
+                           parallel?)
+  ;; This is a non-monadic, enhanced version of 'compiled-file' from (guix
+  ;; gexp).
+  (define build
+    (with-imported-modules (source-module-closure
+                            '((guix build compile)
+                              (guix build utils)))
+      #~(begin
+          (use-modules (srfi srfi-26)
+                       (ice-9 match)
+                       (ice-9 format)
+                       (ice-9 threads)
+                       (guix build compile)
+                       (guix build utils))
+
+          (define (regular? file)
+            (not (member file '("." ".."))))
+
+          (define (process-file file output)
+            (let* ((base   (string-drop-right file 4)) ;.scm
+                   (output (string-append output "/" base
+                                          ".go")))
+              (compile-file file
+                            #:output-file output
+                            #:opts (optimization-options file))))
+
+          (define (report-load file total completed)
+            (display #\cr)
+            (format #t
+                    "loading...\t~5,1f% of ~d files" ;FIXME: i18n
+                    (* 100. (/ completed total)) total)
+            (force-output))
+
+          (define (report-compilation file total completed)
+            (display #\cr)
+            (format #t "compiling...\t~5,1f% of ~d files" ;FIXME: i18n
+                    (* 100. (/ completed total)) total)
+            (force-output))
+
+          (define (process-directory directory output)
+            (let ((files  (find-files directory "\\.scm$"))
+                  (prefix (+ 1 (string-length directory))))
+              ;; Hide compilation warnings.
+              (parameterize ((current-warning-port (%make-void-port "w")))
+                (compile-files directory #$output
+                               (map (cut string-drop <> prefix) files)
+                               #:workers (parallel-job-count)
+                               #:report-load report-load
+                               #:report-compilation report-compilation))))
+
+          (debug-disable 'warn-deprecated)
+          (setvbuf (current-output-port) _IONBF)
+          (setvbuf (current-error-port) _IONBF)
+
+          (set! %load-path (cons #+module-tree %load-path))
+          (set! %load-path
+            (append '#+dependencies
+                    (map (lambda (extension)
+                           (string-append extension "/share/guile/site/"
+                                          (effective-version)))
+                         '#+extensions)
+                    %load-path))
+
+          (set! %load-compiled-path
+            (append '#+dependencies-compiled
+                    (map (lambda (extension)
+                           (string-append extension "/lib/guile/"
+                                          (effective-version)
+                                          "/site-ccache"))
+                         '#+extensions)
+                    %load-compiled-path))
+
+          ;; Load the compiler modules upfront.
+          (compile #f)
+
+          (mkdir #$output)
+          (chdir #+module-tree)
+          (process-directory "." #$output))))
+
+  (computed-file name build
+                 #:options '()))
+
+\f
+;;;
+;;; Live patching.
+;;;
+
+(define (recursive-submodules module)
+  "Return the list of submodules of MODULE."
+  (let loop ((module module)
+             (result '()))
+    (let ((submodules (hash-map->list (lambda (name module)
+                                        module)
+                                      (module-submodules module))))
+      (fold loop (append submodules result) submodules))))
+
+(define (remove-submodule! module names)
+  (let loop ((module module)
+             (names names))
+    (match names
+      (() #t)
+      ((head tail ...)
+       (match (nested-ref-module module tail)
+         (#f #t)
+         ((? module? submodule)
+          (hashq-remove! (module-submodules module) head)
+          (loop submodule tail)))))))
+
+(define (unload-module-tree! module)
+  (define (strip-prefix prefix lst)
+    (let loop ((prefix prefix)
+               (lst lst))
+      (match prefix
+        (()
+         lst)
+        ((_ prefix ...)
+         (match lst
+           ((_ lst ...)
+            (loop prefix lst)))))))
+
+  (let ((submodules (hash-map->list (lambda (name module)
+                                      module)
+                                    (module-submodules module))))
+    (let loop ((root module)
+               (submodules submodules))
+      (match submodules
+        (()
+         #t)
+        ((head tail ...)
+         (unload-module-tree! head)
+         (remove-submodule! root
+                            (strip-prefix (module-name root)
+                                          (module-name head)))
+
+         (match (module-name head)
+           ((parents ... leaf)
+            ;; Remove MODULE from the AUTOLOADS-DONE list.  Note: We don't use
+            ;; 'module-filename' because it could be an absolute file name.
+            (set-autoloaded! (string-join (map symbol->string parents)
+                                          "/" 'suffix)
+                             (symbol->string leaf) #f)))
+         (loop root tail))))))
+
+(define* (reload-guix #:optional (log-port (current-error-port)))
+  "Reload all the Guix and GNU modules currently loaded."
+  (let* ((guix (resolve-module '(guix) #f #:ensure #f))
+         (gnu  (resolve-module '(gnu) #f #:ensure #f))
+         (guix-submodules (recursive-submodules guix))
+         (gnu-submodules  (recursive-submodules gnu)))
+    (define (reload module)
+      (match (module-filename module)
+        (#f #f)
+        ((? string? file)
+         (resolve-module (module-name module)))))
+
+    ;; Ignore any available .go, and force recompilation.  This is because our
+    ;; checkout in the store has mtime set to the epoch, and thus .go files look
+    ;; newer, even though they may not correspond.
+    (set! %fresh-auto-compile #t)
+
+    ;; First, we need to nuke all the (guix) and (gnu) submodules so we don't
+    ;; end up with a mixture of old and new modules when we reload (which
+    ;; wouldn't work, because we'd have two different <package> record types,
+    ;; for instance.)
+    (format log-port "Unloading current Guix...~%")
+    (unload-module-tree! gnu)
+    (unload-module-tree! guix)
+
+    (format log-port "Loading new Guix...~%")
+    (for-each reload (append guix-submodules (list guix)))
+    (for-each reload (append gnu-submodules (list gnu)))
+    (format log-port "New Guix modules successfully loaded.~%")))
+
+\f
+;;;
+;;; Building.
+;;;
+
+(define* (guile-for-build #:optional (version (effective-version)))
+  "Return a package for Guile VERSION."
+  (define canonical-package                       ;soft reference
+    (module-ref (resolve-interface '(gnu packages base))
+                'canonical-package))
+
+  (match version
+    ("2.2"
+     (canonical-package
+      (specification->package "guile@2.2")))
+    ("2.0"
+     (canonical-package
+      (specification->package "guile@2.0")))))
+
+(define* (guix-derivation source version
+                          #:optional (guile-version (effective-version)))
+  "Return, as a monadic value, the derivation to build the Guix from SOURCE
+for GUILE-VERSION.  Use VERSION as the version string."
+  (mbegin %store-monad
+    (set-guile-for-build (guile-for-build guile-version))
+    (lower-object (compiled-guix source
+                                 #:version version
+                                 #:guile-version guile-version))))
-- 
2.14.2

bug#27284: [PATCH 8/8] DRAFT Add (guix self) and use it when pulling.

[Maxim Cournoyer]

Hi Ludovic!

I've read through this draft, and FWIW it looks OK to me! I've proof
read the comments and identified a few typos which you'll find
below. Great work!

Ludovic Courtès <ludo@gnu.org> writes:

> DRAFT: Module reloading doesn't work; needs more testing.
>
> Partly addresses <https://bugs.gnu.org/27284>.
>
> * guix/self.scm: New file.
> * Makefile.am (MODULES): Add it.
> * build-aux/build-self.scm (libgcrypt, zlib, gzip, bzip2, xz)
> (false-if-wrong-guile, package-for-current-guile, guile-json)
> (guile-ssh, guile-git, guile-bytestructures): Remove.
> (build): Rewrite to simply delegate to 'compiled-guix'.
> * gnu/packages.scm (%distro-root-directory): Rewrite to try difference

s/difference/different/ ?

[...]

> +(define* (scheme-node name modules #:optional (dependencies '())
> +                      #:key (extra-modules '()) (extra-files '())
> +                      (extensions '())
> +                      parallel?)
> +  "Return a node that builds the given Scheme MODULES, and depends on
> +DEPENDENCIES (a list of nodes).  EXTRA-MODULES is a list of additional modules
> +added to the source, and EXTRA-FILES is a list of additional files.
> +EXTENSIONS is a set of full-blow Guile packages (e.g., 'guile-json')
> that must

s/full-blow/full-blown/ ?

> +be present in the search path."
> +  (let* ((modules (append extra-modules
> +                          (closure modules
> +                                   (node-modules/recursive dependencies))))
> +         (module-files (map module->import modules))
> +         (source (imported-files (string-append name "-source")
> +                                 (append module-files extra-files))))
> +    (node name modules source dependencies
> +          (compiled-modules name source modules
> +                            (map node-source dependencies)
> +                            (map node-compiled dependencies)
> +                            #:extensions extensions
> +                            #:parallel? parallel?))))
> +
> +(define (file-imports directory sub-directory pred)
> +  "List all the files matching PRED under DIRECTORY/SUB-DIRECTORY.  Return a
> +list of file-name/file-like object suitable as input to
> 'imported-files'."

s/object/objects/, s/input/inputs/

> +  (map (lambda (file)
> +         (list (string-drop file (+ 1 (string-length directory)))
> +               (local-file file #:recursive? #t)))
> +       (find-files (string-append directory "/" sub-directory) pred)))
> +
> +(define (scheme-modules* directory sub-directory)
> +  "Return the list of module names found under SUB-DIRECTORY in DIRECTORY."
> +  (let ((prefix (string-length directory)))
> +    (map (lambda (file)
> +           (file-name->module-name (string-drop file prefix)))
> +         (scheme-files (string-append directory "/" sub-directory)))))
> +
> +(define* (compiled-guix source #:key (version %guix-version)
> +                        (guile-version (effective-version))
> +                        (libgcrypt (specification->package "libgcrypt"))
> +                        (zlib (specification->package "zlib"))
> +                        (gzip (specification->package "gzip"))
> +                        (bzip2 (specification->package "bzip2"))
> +                        (xz (specification->package "xz")))
> +  "Return a file-like objects that contains a compiled Guix."

s/objects/object/ ;)

> +  (define guile-json
> +    (package-for-guile guile-version
> +                       "guile-json"
> +                       "guile2.2-json"
> +                       "guile2.0-json"))
> +
> +  (define guile-ssh
> +    (package-for-guile guile-version
> +                       "guile-ssh"
> +                       "guile2.2-ssh"
> +                       "guile2.0-ssh"))
> +
> +  (define guile-git
> +    (package-for-guile guile-version
> +                       "guile-git"
> +                       "guile2.0-git"))
> +
> +
> +  (define dependencies
> +    (match (append-map (lambda (package)
> +                         (cons (list "x" package)
> +                               (package-transitive-inputs package)))
> +                       (list guile-git guile-json guile-ssh))
> +      (((labels packages _ ...) ...)
> +       packages)))
> +
> +  (define *core-modules*
> +    (scheme-node "guix-core"
> +                 '((guix)
> +                   (guix monad-repl)
> +                   (guix packages)
> +                   (guix download)
> +                   (guix discovery)
> +                   (guix profiles)
> +                   (guix build-system gnu)
> +                   (guix build profiles)
> +                   (guix build gnu-build-system))

Sorting the modules list lexicographically would be neat, here and in
various places. If this seems reasonable, perhaps we could have this
stylistic convention in our guidelines?

> +
> +                 ;; Provide a dummy (guix config) with the default version
> +                 ;; number, storedir, etc.  This is so that "guix-core" is the
> +                 ;; same across all installations and doesn't need to be
> +                 ;; rebuilt when the version changes, which in turns means we
> +                 ;; can have substitutes for it.

s/in turns/in turn/

[...]

> +          (define (regular? file)
> +            (not (member file '("." ".."))))
> +
> +          (define (process-file file output)
> +            (let* ((base   (string-drop-right file 4)) ;.scm
> +                   (output (string-append output "/" base
> +                                          ".go")))
> +              (compile-file file
> +                            #:output-file output
> +                            #:opts (optimization-options file))))
> +
> +          (define (report-load file total completed)
> +            (display #\cr)
> +            (format #t
> +                    "loading...\t~5,1f% of ~d files" ;FIXME: i18n
> +                    (* 100. (/ completed total)) total)
> +            (force-output))
> +
> +          (define (report-compilation file total completed)
> +            (display #\cr)
> +            (format #t "compiling...\t~5,1f% of ~d files" ;FIXME: i18n
> +                    (* 100. (/ completed total)) total)
> +            (force-output))
> +
> +          (define (process-directory directory output)
> +            (let ((files  (find-files directory "\\.scm$"))
> +                  (prefix (+ 1 (string-length directory))))
> +              ;; Hide compilation warnings.

Should this be configurable? Hidden warnings don't have much chance to
get addressed :)

Maxim

bug#27284: [PATCH 8/8] DRAFT Add (guix self) and use it when pulling.

[Ludovic Courtès]

[-- Attachment #1: Type: text/plain, Size: 2120 bytes --]

Hi Maxim,

Thanks for reviewing this!

I’ve fixed the many typos that you noticed.  Apart from that, I’ve also
addressed the “parallel world” issue that module reloading creates: the
trick is to use ‘eval’ (evil!).  Anyway, it seems to do the job.  New
patches attached!

So at this point, I think it needs more testing mostly.  To do that, you
need to create a branch with those patches, say ‘wip-pull-derivations’,
and then from there you can run:

  ./pre-inst-env guix pull --url=$PWD --branch=wip-pull-derivations

Some other comments:

Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:

>> +  (define *core-modules*
>> +    (scheme-node "guix-core"
>> +                 '((guix)
>> +                   (guix monad-repl)
>> +                   (guix packages)
>> +                   (guix download)
>> +                   (guix discovery)
>> +                   (guix profiles)
>> +                   (guix build-system gnu)
>> +                   (guix build profiles)
>> +                   (guix build gnu-build-system))
>
> Sorting the modules list lexicographically would be neat, here and in
> various places. If this seems reasonable, perhaps we could have this
> stylistic convention in our guidelines?

In this case I wanted to have them sorted “logically”, almost
topologically, and I prefer it that way.

For (gnu packages …) modules, I agree it makes sense to sort
alphabetically though, and yes, we could write it down somewhere in the
manual.

>> +          (define (process-directory directory output)
>> +            (let ((files  (find-files directory "\\.scm$"))
>> +                  (prefix (+ 1 (string-length directory))))
>> +              ;; Hide compilation warnings.
>
> Should this be configurable? Hidden warnings don't have much chance to
> get addressed :)

The idea is that when you type ‘make’ you still get warnings, but you
don’t get them when you run ‘guix pull’, as is already the case (on the
grounds that ‘make’ is for developers and ‘guix pull’ for users.)

Thanks,
Ludo’.


[-- Attachment #2: 0001-union-Parametrize-the-symlink-procedure.patch --]
[-- Type: text/x-patch, Size: 3507 bytes --]

From 4cfcfaf1d7564b49eeb267f3d56167891d06705c Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Ludovic=20Court=C3=A8s?= <ludo@gnu.org>
Date: Thu, 19 Oct 2017 16:07:34 +0200
Subject: [PATCH 1/4] union: Parametrize the symlink procedure .

* guix/gexp.scm (directory-union): Add #:hard-links and honor it.
* guix/build/union.scm (union-build): Add #:symlink parameter.
---
 guix/build/union.scm | 11 ++++++-----
 guix/gexp.scm        | 19 ++++++++++++++++---
 2 files changed, 22 insertions(+), 8 deletions(-)

diff --git a/guix/build/union.scm b/guix/build/union.scm
index 18167fa3e..256123c56 100644
--- a/guix/build/union.scm
+++ b/guix/build/union.scm
@@ -1,5 +1,5 @@
 ;;; GNU Guix --- Functional package management for GNU
-;;; Copyright © 2012, 2013, 2014, 2016 Ludovic Courtès <ludo@gnu.org>
+;;; Copyright © 2012, 2013, 2014, 2016, 2017 Ludovic Courtès <ludo@gnu.org>
 ;;; Copyright © 2014 Mark H Weaver <mhw@netris.org>
 ;;; Copyright © 2017 Huang Ying <huang.ying.caritas@gmail.com>
 ;;;
@@ -78,11 +78,12 @@ identical, #f otherwise."
 
 (define* (union-build output inputs
                       #:key (log-port (current-error-port))
-                      (create-all-directories? #f))
+                      (create-all-directories? #f)
+                      (symlink symlink))
   "Build in the OUTPUT directory a symlink tree that is the union of all the
-INPUTS.  As a special case, if CREATE-ALL-DIRECTORIES?, creates the
-subdirectories in the output directory to make sure the caller can modify them
-later."
+INPUTS, using SYMLINK to create symlinks.  As a special case, if
+CREATE-ALL-DIRECTORIES?, creates the subdirectories in the output directory to
+make sure the caller can modify them later."
 
   (define (symlink* input output)
     (format log-port "`~a' ~~> `~a'~%" input output)
diff --git a/guix/gexp.scm b/guix/gexp.scm
index b9525603e..e8ac3dcdc 100644
--- a/guix/gexp.scm
+++ b/guix/gexp.scm
@@ -1204,13 +1204,24 @@ This yields an 'etc' directory containing these two files."
                                          (ungexp target))))))
                           files))))))
 
-(define (directory-union name things)
+(define* (directory-union name things
+                          #:key (copy? #f))
   "Return a directory that is the union of THINGS, where THINGS is a list of
 file-like objects denoting directories.  For example:
 
   (directory-union \"guile+emacs\" (list guile emacs))
 
-yields a directory that is the union of the 'guile' and 'emacs' packages."
+yields a directory that is the union of the 'guile' and 'emacs' packages.
+
+When COPY? is true, copy files instead of creating symlinks."
+  (define symlink
+    (if copy?
+        (gexp (lambda (old new)
+                (if (file-is-directory? old)
+                    (symlink old new)
+                    (copy-file old new))))
+        (gexp symlink)))
+
   (match things
     ((one)
      ;; Only one thing; return it.
@@ -1221,7 +1232,9 @@ yields a directory that is the union of the 'guile' and 'emacs' packages."
                       (gexp (begin
                               (use-modules (guix build union))
                               (union-build (ungexp output)
-                                           '(ungexp things)))))))))
+                                           '(ungexp things)
+
+                                           #:symlink (ungexp symlink)))))))))
 
 \f
 ;;;
-- 
2.14.2


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #3: 0002-gexp-directory-union-has-a-quiet-parameter.patch --]
[-- Type: text/x-patch, Size: 2029 bytes --]

From 57a950b9e9721d02b420df582a968ee1e9c7229e Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Ludovic=20Court=C3=A8s?= <ludo@gnu.org>
Date: Thu, 19 Oct 2017 16:10:18 +0200
Subject: [PATCH 2/4] gexp: 'directory-union' has a #:quiet? parameter.

* guix/gexp.scm (directory-union): Add #:quiet? and honor it.
---
 guix/gexp.scm | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)

diff --git a/guix/gexp.scm b/guix/gexp.scm
index e8ac3dcdc..3781a1e6e 100644
--- a/guix/gexp.scm
+++ b/guix/gexp.scm
@@ -1205,7 +1205,7 @@ This yields an 'etc' directory containing these two files."
                           files))))))
 
 (define* (directory-union name things
-                          #:key (copy? #f))
+                          #:key (copy? #f) (quiet? #f))
   "Return a directory that is the union of THINGS, where THINGS is a list of
 file-like objects denoting directories.  For example:
 
@@ -1213,7 +1213,8 @@ file-like objects denoting directories.  For example:
 
 yields a directory that is the union of the 'guile' and 'emacs' packages.
 
-When COPY? is true, copy files instead of creating symlinks."
+When HARD-LINKS? is true, create hard links instead of symlinks.  When QUIET?
+is true, the derivation will not print anything."
   (define symlink
     (if copy?
         (gexp (lambda (old new)
@@ -1222,6 +1223,11 @@ When COPY? is true, copy files instead of creating symlinks."
                     (copy-file old new))))
         (gexp symlink)))
 
+  (define log-port
+    (if quiet?
+        (gexp (%make-void-port "w"))
+        (gexp (current-error-port))))
+
   (match things
     ((one)
      ;; Only one thing; return it.
@@ -1234,6 +1240,7 @@ When COPY? is true, copy files instead of creating symlinks."
                               (union-build (ungexp output)
                                            '(ungexp things)
 
+                                           #:log-port (ungexp log-port)
                                            #:symlink (ungexp symlink)))))))))
 
 \f
-- 
2.14.2


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #4: 0003-pull-Trim-import-list.patch --]
[-- Type: text/x-patch, Size: 1212 bytes --]

From 12b687d92b8102e833fcbc7878eeb5f63aba73fb Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Ludovic=20Court=C3=A8s?= <ludo@gnu.org>
Date: Fri, 27 Oct 2017 15:26:19 -0700
Subject: [PATCH 3/4] pull: Trim import list.

* guix/scripts/pull.scm: Remove useless imports.
---
 guix/scripts/pull.scm | 6 ------
 1 file changed, 6 deletions(-)

diff --git a/guix/scripts/pull.scm b/guix/scripts/pull.scm
index 240019800..3e95bd511 100644
--- a/guix/scripts/pull.scm
+++ b/guix/scripts/pull.scm
@@ -25,7 +25,6 @@
   #:use-module (guix config)
   #:use-module (guix packages)
   #:use-module (guix derivations)
-  #:use-module (guix download)
   #:use-module (guix gexp)
   #:use-module (guix grafts)
   #:use-module (guix monads)
@@ -39,14 +38,9 @@
   #:use-module ((gnu packages bootstrap)
                 #:select (%bootstrap-guile))
   #:use-module ((gnu packages certs) #:select (le-certs))
-  #:use-module (gnu packages compression)
-  #:use-module (gnu packages gnupg)
   #:use-module (srfi srfi-1)
   #:use-module (srfi srfi-11)
-  #:use-module (srfi srfi-34)
-  #:use-module (srfi srfi-35)
   #:use-module (srfi srfi-37)
-  #:use-module (ice-9 ftw)
   #:use-module (ice-9 match)
   #:export (guix-pull))
 
-- 
2.14.2


[-- Attachment #5: 0004-DRAFT-Add-guix-self-and-use-it-when-pulling.patch --]
[-- Type: text/x-patch, Size: 44830 bytes --]

From 0b262d35956befbbc670cc1b8ed275510c6b069e Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Ludovic=20Court=C3=A8s?= <ludo@gnu.org>
Date: Mon, 16 Oct 2017 10:41:37 +0200
Subject: [PATCH 4/4] DRAFT Add (guix self) and use it when pulling.

DRAFT: Needs more testing.

Partly addresses <https://bugs.gnu.org/27284>.

* guix/self.scm: New file.
* Makefile.am (MODULES): Add it.
* build-aux/build-self.scm (libgcrypt, zlib, gzip, bzip2, xz)
(false-if-wrong-guile, package-for-current-guile, guile-json)
(guile-ssh, guile-git, guile-bytestructures): Remove.
(build): Rewrite to simply delegate to 'compiled-guix'.
* gnu/packages.scm (%distro-root-directory): Rewrite to try different
directories.
* guix/discovery.scm (guix): Export 'scheme-files'.
* guix/scripts/pull.scm (build-and-install): Split into...
(install-latest): ... this.  New procedure.  And...
(build-and-install): ... this, which now takes a monadic value argument.
(guix-pull): Call 'add-indirect-root'.  Call 'build-from-source' and
pass the result to 'build-and-install'.
---
 Makefile.am              |   1 +
 build-aux/build-self.scm | 228 ++----------------
 gnu/packages.scm         |  21 +-
 guix/discovery.scm       |   3 +-
 guix/scripts/pull.scm    |  89 ++++---
 guix/self.scm            | 599 +++++++++++++++++++++++++++++++++++++++++++++++
 6 files changed, 698 insertions(+), 243 deletions(-)
 create mode 100644 guix/self.scm

diff --git a/Makefile.am b/Makefile.am
index fd6f9729c..bb12e9905 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -66,6 +66,7 @@ MODULES =					\
   guix/derivations.scm				\
   guix/grafts.scm				\
   guix/gnu-maintenance.scm			\
+  guix/self.scm					\
   guix/upstream.scm				\
   guix/licenses.scm				\
   guix/git.scm					\
diff --git a/build-aux/build-self.scm b/build-aux/build-self.scm
index ed8ff5f4c..039d35acd 100644
--- a/build-aux/build-self.scm
+++ b/build-aux/build-self.scm
@@ -17,9 +17,6 @@
 ;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
 
 (define-module (build-self)
-  #:use-module (gnu)
-  #:use-module (guix)
-  #:use-module (guix config)
   #:use-module (srfi srfi-1)
   #:use-module (srfi srfi-19)
   #:use-module (ice-9 match)
@@ -39,98 +36,25 @@
 ;;;
 ;;; Code:
 
-\f
-;; The dependencies.  Don't refer explicitly to the variables because they
-;; could be renamed or shuffled around in modules over time.  Conversely,
-;; 'find-best-packages-by-name' is expected to always have the same semantics.
+;; Use our very own Guix modules.
+(eval-when (compile load eval)
 
-(define libgcrypt
-  (first (find-best-packages-by-name "libgcrypt" #f)))
+  ;; Ignore any available .go, and force recompilation.  This is because our
+  ;; checkout in the store has mtime set to the epoch, and thus .go files look
+  ;; newer, even though they may not correspond.
+  (set! %fresh-auto-compile #t)
 
-(define zlib
-  (first (find-best-packages-by-name "zlib" #f)))
-
-(define gzip
-  (first (find-best-packages-by-name "gzip" #f)))
-
-(define bzip2
-  (first (find-best-packages-by-name "bzip2" #f)))
-
-(define xz
-  (first (find-best-packages-by-name "xz" #f)))
-
-(define (false-if-wrong-guile package)
-  "Return #f if PACKAGE depends on the \"wrong\" major version of Guile (e.g.,
-2.0 instead of 2.2), otherwise return PACKAGE."
-  (let ((guile (any (match-lambda
-                      ((label (? package? dep) _ ...)
-                       (and (string=? (package-name dep) "guile")
-                            dep)))
-                    (package-direct-inputs package))))
-    (and (or (not guile)
-             (string-prefix? (effective-version)
-                             (package-version guile)))
-         package)))
-
-(define (package-for-current-guile . names)
-  "Return the package with one of the given NAMES that depends on the current
-Guile major version (2.0 or 2.2), or #f if none of the packages matches."
-  (let loop ((names names))
-    (match names
-      (()
-       #f)
-      ((name rest ...)
-       (match (find-best-packages-by-name name #f)
-         (()
-          (loop rest))
-         ((first _ ...)
-          (or (false-if-wrong-guile first)
-              (loop rest))))))))
-
-(define guile-json
-  (package-for-current-guile "guile-json"
-                             "guile2.2-json"
-                             "guile2.0-json"))
-
-(define guile-ssh
-  (package-for-current-guile "guile-ssh"
-                             "guile2.2-ssh"
-                             "guile2.0-ssh"))
-
-(define guile-git
-  (package-for-current-guile "guile-git"
-                             "guile2.0-git"))
-
-(define guile-bytestructures
-  (package-for-current-guile "guile-bytestructures"
-                             "guile2.0-bytestructures"))
-\f
-;; The actual build procedure.
-
-(define (top-source-directory)
-  "Return the name of the top-level directory of this source tree."
   (and=> (assoc-ref (current-source-location) 'filename)
          (lambda (file)
-           (string-append (dirname file) "/.."))))
-
+           (let ((dir (string-append (dirname file) "/../..")))
+             (set! %load-path (cons dir %load-path))))))
 
 (define (date-version-string)
   "Return the current date and hour in UTC timezone, for use as a poor
 person's version identifier."
-  ;; XXX: Replace with a Git commit id.
+  ;; XXX: Last resort when the Git commit id is missing.
   (date->string (current-date 0) "~Y~m~d.~H"))
 
-(define (guile-for-build)
-  "Return a derivation for Guile 2.0 or 2.2, whichever matches the currently
-running Guile."
-  (package->derivation (cond-expand
-                         (guile-2.2
-                          (canonical-package
-                           (specification->package "guile@2.2")))
-                         (else
-                          (canonical-package
-                           (specification->package "guile@2.0"))))))
-
 ;; The procedure below is our return value.
 (define* (build source
                 #:key verbose? (version (date-version-string))
@@ -138,131 +62,19 @@ running Guile."
                 #:rest rest)
   "Return a derivation that unpacks SOURCE into STORE and compiles Scheme
 files."
-  ;; The '%xxxdir' variables were added to (guix config) in July 2016 so we
-  ;; cannot assume that they are defined.  Try to guess their value when
-  ;; they're undefined (XXX: we get an incorrect guess when environment
-  ;; variables such as 'NIX_STATE_DIR' are defined!).
-  (define storedir
-    (if (defined? '%storedir) %storedir %store-directory))
-  (define localstatedir
-    (if (defined? '%localstatedir) %localstatedir (dirname %state-directory)))
-  (define sysconfdir
-    (if (defined? '%sysconfdir) %sysconfdir (dirname %config-directory)))
-  (define sbindir
-    (if (defined? '%sbindir) %sbindir (dirname %guix-register-program)))
-
-  (define builder
-    #~(begin
-        (use-modules (guix build pull))
-
-        (letrec-syntax ((maybe-load-path
-                         (syntax-rules ()
-                           ((_ item rest ...)
-                            (let ((tail (maybe-load-path rest ...)))
-                              (if (string? item)
-                                  (cons (string-append item
-                                                       "/share/guile/site/"
-                                                       #$(effective-version))
-                                        tail)
-                                  tail)))
-                           ((_)
-                            '()))))
-          (set! %load-path
-                (append
-                 (maybe-load-path #$guile-json #$guile-ssh
-                                  #$guile-git #$guile-bytestructures)
-                 %load-path)))
-
-        (letrec-syntax ((maybe-load-compiled-path
-                         (syntax-rules ()
-                           ((_ item rest ...)
-                            (let ((tail (maybe-load-compiled-path rest ...)))
-                              (if (string? item)
-                                  (cons (string-append item
-                                                       "/lib/guile/"
-                                                       #$(effective-version)
-                                                       "/site-ccache")
-                                        tail)
-                                  tail)))
-                           ((_)
-                            '()))))
-          (set! %load-compiled-path
-                (append
-                 (maybe-load-compiled-path #$guile-json #$guile-ssh
-                                           #$guile-git #$guile-bytestructures)
-                 %load-compiled-path)))
-
-        ;; XXX: The 'guile-ssh' package prior to Guix commit 92b7258 was
-        ;; broken: libguile-ssh could not be found.  Work around that.
-        ;; FIXME: We want Guile-SSH 0.10.2 or later anyway.
-        #$(if (string-prefix? "0.9." (package-version guile-ssh))
-              #~(setenv "LTDL_LIBRARY_PATH" (string-append #$guile-ssh "/lib"))
-              #t)
-
-        (build-guix #$output #$source
-
-                    #:system #$%system
-                    #:storedir #$storedir
-                    #:localstatedir #$localstatedir
-                    #:sysconfdir #$sysconfdir
-                    #:sbindir #$sbindir
-
-                    #:package-name #$%guix-package-name
-                    #:package-version #$version
-                    #:bug-report-address #$%guix-bug-report-address
-                    #:home-page-url #$%guix-home-page-url
-
-                    #:libgcrypt #$libgcrypt
-                    #:zlib #$zlib
-                    #:gzip #$gzip
-                    #:bzip2 #$bzip2
-                    #:xz #$xz
-
-                    ;; XXX: This is not perfect, enabling VERBOSE? means
-                    ;; building a different derivation.
-                    #:debug-port (if #$verbose?
-                                     (current-error-port)
-                                     (%make-void-port "w")))))
-
-  (unless guile-git
-    ;; XXX: Guix before February 2017 lacks a 'guile-git' package altogether.
-    ;; If we try to upgrade anyway, the logic in (guix scripts pull) will not
-    ;; build (guix git), which will leave us with an unusable 'guix pull'.  To
-    ;; avoid that, fail early.
-    (format (current-error-port)
-            "\
-Your installation is too old and lacks a '~a' package.
-Please upgrade to an intermediate version first, for instance with:
-
-  guix pull --url=https://git.savannah.gnu.org/cgit/guix.git/snapshot/v0.13.0.tar.gz
-\n"
-            (match (effective-version)
-              ("2.0" "guile2.0-git")
-              (_     "guile-git")))
-    (exit 1))
-
-  (mlet %store-monad ((guile (guile-for-build)))
-    (gexp->derivation "guix-latest" builder
-                      #:modules '((guix build pull)
-                                  (guix build utils)
-                                  (guix build compile)
-
-                                  ;; Closure of (guix modules).
-                                  (guix modules)
-                                  (guix memoization)
-                                  (guix sets))
-
-                      ;; Arrange so that our own (guix build …) modules are
-                      ;; used.
-                      #:module-path (list (top-source-directory))
-
-                      #:guile-for-build guile)))
+  ;; FIXME: Closures in (guix scripts pull) have already cached old values of
+  ;; (guix …), so they keep referring to the "old" world, with different
+  ;; record types than the one we get after reloading (because record types
+  ;; are "generative".)
+  (let ((reload-guix (module-ref (resolve-interface '(guix self))
+                                 'reload-guix)))
+    (reload-guix))                                ;cross fingers!
+
+  (let ((guix-derivation (module-ref (resolve-interface '(guix self))
+                                     'guix-derivation)))
+    (guix-derivation source version)))
 
 ;; This file is loaded by 'guix pull'; return it the build procedure.
 build
 
-;; Local Variables:
-;; eval: (put 'with-load-path 'scheme-indent-function 1)
-;; End:
-
 ;;; build-self.scm ends here
diff --git a/gnu/packages.scm b/gnu/packages.scm
index b4ac6661c..7ca6c3093 100644
--- a/gnu/packages.scm
+++ b/gnu/packages.scm
@@ -110,8 +110,25 @@ for system '~a'")
                         file-name system)))))))
 
 (define %distro-root-directory
-  ;; Absolute file name of the module hierarchy.
-  (dirname (search-path %load-path "guix.scm")))
+  ;; Absolute file name of the module hierarchy.  Since (gnu packages …) might
+  ;; live in a directory different from (guix), try to get the best match.
+  (letrec-syntax ((dirname* (syntax-rules ()
+                              ((_ file)
+                               (dirname file))
+                              ((_ file head tail ...)
+                               (dirname (dirname* file tail ...)))))
+                  (try      (syntax-rules ()
+                              ((_ (file things ...) rest ...)
+                               (match (search-path %load-path file)
+                                 (#f
+                                  (try rest ...))
+                                 (absolute
+                                  (dirname* absolute things ...))))
+                              ((_)
+                               #f))))
+    (try ("gnu/packages/base.scm" gnu/ packages/)
+         ("gnu/packages.scm"      gnu/)
+         ("guix.scm"))))
 
 (define %package-module-path
   ;; Search path for package modules.  Each item must be either a directory
diff --git a/guix/discovery.scm b/guix/discovery.scm
index 7b5757902..8ffcf7cd9 100644
--- a/guix/discovery.scm
+++ b/guix/discovery.scm
@@ -25,7 +25,8 @@
   #:use-module (ice-9 match)
   #:use-module (ice-9 vlist)
   #:use-module (ice-9 ftw)
-  #:export (scheme-modules
+  #:export (scheme-files
+            scheme-modules
             fold-modules
             all-modules
             fold-module-public-variables))
diff --git a/guix/scripts/pull.scm b/guix/scripts/pull.scm
index 3e95bd511..e212a5b05 100644
--- a/guix/scripts/pull.scm
+++ b/guix/scripts/pull.scm
@@ -171,33 +171,48 @@ contained therein.  Use COMMIT as the version string."
     ;; tree.
     (build source #:verbose? verbose? #:version commit)))
 
-(define* (build-and-install source config-dir
-                            #:key verbose? commit)
-  "Build the tool from SOURCE, and install it in CONFIG-DIR."
-  (mlet* %store-monad ((source        (build-from-source source
-                                                         #:commit commit
-                                                         #:verbose? verbose?))
-                       (source-dir -> (derivation->output-path source))
-                       (to-do?        (what-to-build (list source)))
-                       (built?        (built-derivations (list source))))
-    ;; Always update the 'latest' symlink, regardless of whether SOURCE was
-    ;; already built or not.
-    (if built?
-        (mlet* %store-monad
-            ((latest -> (string-append config-dir "/latest"))
-             (done      (indirect-root-added latest)))
-          (if (and (file-exists? latest)
-                   (string=? (readlink latest) source-dir))
-              (begin
-                (display (G_ "Guix already up to date\n"))
-                (return #t))
-              (begin
-                (switch-symlinks latest source-dir)
-                (format #t
-                        (G_ "updated ~a successfully deployed under `~a'~%")
-                        %guix-package-name latest)
-                (return #t))))
-        (leave (G_ "failed to update Guix, check the build log~%")))))
+(define* (install-latest source-dir config-dir)
+  "Make SOURCE-DIR, a store file name, the latest Guix in CONFIG-DIR."
+  (let ((latest (string-append config-dir "/latest")))
+    (if (and (file-exists? latest)
+             (string=? (readlink latest) source-dir))
+        (begin
+          (display (G_ "Guix already up to date\n"))
+          #t)
+        (begin
+          (switch-symlinks latest source-dir)
+          (format #t
+                  (G_ "updated ~a successfully deployed under `~a'~%")
+                  %guix-package-name latest)
+          #t))))
+
+(define (build-and-install mdrv)
+  "Bind MDRV, a monadic value for a derivation, build it, and finally install
+it as the latest Guix."
+  (define do-it
+    ;; Weirdness follows!  Before we were called, the Guix modules have
+    ;; probably been reloaded, leading to a "parallel universe" with disjoint
+    ;; record types.  However, procedures in this file have already cached the
+    ;; module relative to which they lookup global bindings (see
+    ;; 'toplevel-box' documentation), so they're stuck in the old world.  To
+    ;; work around that, evaluate our procedure in the context of the "new"
+    ;; (guix scripts pull) module--which has access to the new <derivation>
+    ;; record, and so on.
+    (eval '(lambda (mdrv cont)
+             ;; Reopen a connection to the daemon so that we have a record
+             ;; with the new type.
+             (with-store store
+               (run-with-store store
+                 (mlet %store-monad ((drv mdrv))
+                   (mbegin %store-monad
+                     (what-to-build (list drv))
+                     (built-derivations (list drv))
+                     (return (cont (derivation->output-path drv))))))))
+          (resolve-module '(guix scripts pull)))) ;the new module
+
+  (do-it mdrv
+         (lambda (result)
+           (install-latest result (config-directory)))))
 
 (define (honor-lets-encrypt-certificates! store)
   "Tell Guile-Git to use the Let's Encrypt certificates."
@@ -258,6 +273,10 @@ certificates~%"))
              (when (use-le-certs? url)
                (honor-lets-encrypt-certificates! store))
 
+             ;; Ensure the 'latest' symlink is registered as a GC root.
+             (add-indirect-root store
+                                (string-append (config-directory) "/latest"))
+
              (format (current-error-port)
                      (G_ "Updating from Git repository at '~a'...~%")
                      url)
@@ -276,10 +295,16 @@ certificates~%"))
                                 (if (assoc-ref opts 'bootstrap?)
                                     %bootstrap-guile
                                     (canonical-package guile-2.0)))))
-                 (run-with-store store
-                   (build-and-install checkout (config-directory)
-                                      #:commit commit
-                                      #:verbose?
-                                      (assoc-ref opts 'verbose?))))))))))))
+
+                 ;; 'build-from-source' may cause a reload of the Guix
+                 ;; modules.  This leads to a parallel world: its record types
+                 ;; are disjoint from those we've seen until now (because we
+                 ;; use "generative" record types), and so on.  Thus, special
+                 ;; care must be taken once we have return from that call.
+                 (build-and-install
+                  (build-from-source checkout
+                                     #:commit commit
+                                     #:verbose?
+                                     (assoc-ref opts 'verbose?))))))))))))
 
 ;;; pull.scm ends here
diff --git a/guix/self.scm b/guix/self.scm
new file mode 100644
index 000000000..cd7f2410d
--- /dev/null
+++ b/guix/self.scm
@@ -0,0 +1,599 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2017 Ludovic Courtès <ludo@gnu.org>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (guix self)
+  #:use-module (guix config)
+  #:use-module (guix modules)
+  #:use-module (guix gexp)
+  #:use-module (guix store)
+  #:use-module (guix monads)
+  #:use-module (guix discovery)
+  #:use-module (guix packages)
+  #:use-module (guix sets)
+  #:use-module (guix build utils)
+  #:use-module (gnu packages)
+  #:use-module (srfi srfi-1)
+  #:use-module (srfi srfi-9)
+  #:use-module (ice-9 match)
+  #:export (compiled-guix
+            guix-derivation
+            reload-guix))
+
+\f
+;;;
+;;; Dependency handling.
+;;;
+
+(define* (false-if-wrong-guile package
+                               #:optional (guile-version (effective-version)))
+  "Return #f if PACKAGE depends on the \"wrong\" major version of Guile (e.g.,
+2.0 instead of 2.2), otherwise return PACKAGE."
+  (let ((guile (any (match-lambda
+                      ((label (? package? dep) _ ...)
+                       (and (string=? (package-name dep) "guile")
+                            dep)))
+                    (package-direct-inputs package))))
+    (and (or (not guile)
+             (string-prefix? guile-version
+                             (package-version guile)))
+         package)))
+
+(define (package-for-guile guile-version . names)
+  "Return the package with one of the given NAMES that depends on
+GUILE-VERSION (\"2.0\" or \"2.2\"), or #f if none of the packages matches."
+  (let loop ((names names))
+    (match names
+      (()
+       #f)
+      ((name rest ...)
+       (match (specification->package name)
+         (#f
+          (loop rest))
+         ((? package? package)
+          (or (false-if-wrong-guile package)
+              (loop rest))))))))
+
+\f
+;;;
+;;; Derivations.
+;;;
+
+;; Node in a DAG of build tasks.  Each node maps to a derivation, but it's
+;; easier to express things this way.
+(define-record-type <node>
+  (node name modules source dependencies compiled)
+  node?
+  (name          node-name)                       ;string
+  (modules       node-modules)                    ;list of module names
+  (source        node-source)                     ;list of source files
+  (dependencies  node-dependencies)               ;list of nodes
+  (compiled      node-compiled))                  ;node -> lowerable object
+
+(define (node-fold proc init nodes)
+  (let loop ((nodes nodes)
+             (visited (setq))
+             (result init))
+    (match nodes
+      (() result)
+      ((head tail ...)
+       (if (set-contains? visited head)
+           (loop tail visited result)
+           (loop tail (set-insert head visited)
+                 (proc head result)))))))
+
+(define (node-modules/recursive nodes)
+  (node-fold (lambda (node modules)
+               (append (node-modules node) modules))
+             '()
+             nodes))
+
+(define* (closure modules #:optional (except '()))
+  (source-module-closure modules
+                         #:select?
+                         (match-lambda
+                           (('guix 'config)
+                            #f)
+                           ((and module
+                                 (or ('guix _ ...) ('gnu _ ...)))
+                            (not (member module except)))
+                           (rest #f))))
+
+(define module->import
+  ;; Return a file-name/file-like object pair for the specified module and
+  ;; suitable for 'imported-files'.
+  (match-lambda
+    ((module '=> thing)
+     (let ((file (module-name->file-name module)))
+       (list file thing)))
+    (module
+        (let ((file (module-name->file-name module)))
+          (list file
+                (local-file (search-path %load-path file)))))))
+
+(define* (scheme-node name modules #:optional (dependencies '())
+                      #:key (extra-modules '()) (extra-files '())
+                      (extensions '())
+                      parallel?)
+  "Return a node that builds the given Scheme MODULES, and depends on
+DEPENDENCIES (a list of nodes).  EXTRA-MODULES is a list of additional modules
+added to the source, and EXTRA-FILES is a list of additional files.
+EXTENSIONS is a set of full-blown Guile packages (e.g., 'guile-json') that
+must be present in the search path."
+  (let* ((modules (append extra-modules
+                          (closure modules
+                                   (node-modules/recursive dependencies))))
+         (module-files (map module->import modules))
+         (source (imported-files (string-append name "-source")
+                                 (append module-files extra-files))))
+    (node name modules source dependencies
+          (compiled-modules name source modules
+                            (map node-source dependencies)
+                            (map node-compiled dependencies)
+                            #:extensions extensions
+                            #:parallel? parallel?))))
+
+(define (file-imports directory sub-directory pred)
+  "List all the files matching PRED under DIRECTORY/SUB-DIRECTORY.  Return a
+list of file-name/file-like objects suitable as inputs to 'imported-files'."
+  (map (lambda (file)
+         (list (string-drop file (+ 1 (string-length directory)))
+               (local-file file #:recursive? #t)))
+       (find-files (string-append directory "/" sub-directory) pred)))
+
+(define (scheme-modules* directory sub-directory)
+  "Return the list of module names found under SUB-DIRECTORY in DIRECTORY."
+  (let ((prefix (string-length directory)))
+    (map (lambda (file)
+           (file-name->module-name (string-drop file prefix)))
+         (scheme-files (string-append directory "/" sub-directory)))))
+
+(define* (compiled-guix source #:key (version %guix-version)
+                        (guile-version (effective-version))
+                        (libgcrypt (specification->package "libgcrypt"))
+                        (zlib (specification->package "zlib"))
+                        (gzip (specification->package "gzip"))
+                        (bzip2 (specification->package "bzip2"))
+                        (xz (specification->package "xz")))
+  "Return a file-like object that contains a compiled Guix."
+  (define guile-json
+    (package-for-guile guile-version
+                       "guile-json"
+                       "guile2.2-json"
+                       "guile2.0-json"))
+
+  (define guile-ssh
+    (package-for-guile guile-version
+                       "guile-ssh"
+                       "guile2.2-ssh"
+                       "guile2.0-ssh"))
+
+  (define guile-git
+    (package-for-guile guile-version
+                       "guile-git"
+                       "guile2.0-git"))
+
+
+  (define dependencies
+    (match (append-map (lambda (package)
+                         (cons (list "x" package)
+                               (package-transitive-inputs package)))
+                       (list guile-git guile-json guile-ssh))
+      (((labels packages _ ...) ...)
+       packages)))
+
+  (define *core-modules*
+    (scheme-node "guix-core"
+                 '((guix)
+                   (guix monad-repl)
+                   (guix packages)
+                   (guix download)
+                   (guix discovery)
+                   (guix profiles)
+                   (guix build-system gnu)
+                   (guix build profiles)
+                   (guix build gnu-build-system))
+
+                 ;; Provide a dummy (guix config) with the default version
+                 ;; number, storedir, etc.  This is so that "guix-core" is the
+                 ;; same across all installations and doesn't need to be
+                 ;; rebuilt when the version changes, which in turn means we
+                 ;; can have substitutes for it.
+                 #:extra-modules
+                 `(((guix config)
+                    => ,(make-config.scm #:libgcrypt
+                                         (specification->package "libgcrypt"))))))
+
+  (define *extra-modules*
+    (scheme-node "guix-extra"
+                 (filter-map (match-lambda
+                               (('guix 'scripts _ ..1) #f)
+                               (name name))
+                             (scheme-modules* source "guix"))
+                 (list *core-modules*)
+                 #:extensions dependencies))
+
+  (define *package-modules*
+    (scheme-node "guix-packages"
+                 `((gnu packages)
+                   ,@(scheme-modules* source "gnu/packages"))
+                 (list *core-modules* *extra-modules*)
+                 #:extra-files                    ;all the non-Scheme files
+                 (file-imports source "gnu/packages"
+                               (lambda (file stat)
+                                 (and (eq? 'regular (stat:type stat))
+                                      (not (string-suffix? ".scm" file))
+                                      (not (string-suffix? ".go" file))
+                                      (not (string-prefix? ".#" file))
+                                      (not (string-suffix? "~" file)))))))
+
+  (define *system-modules*
+    (scheme-node "guix-system"
+                 `((gnu system)
+                   (gnu services)
+                   ,@(scheme-modules source "gnu/system")
+                   ,@(scheme-modules source "gnu/services"))
+                 (list *package-modules* *extra-modules* *core-modules*)))
+
+  (define *cli-modules*
+    (scheme-node "guix-cli"
+                 (scheme-modules* source "/guix/scripts")
+                 (list *core-modules* *extra-modules* *package-modules*
+                       *system-modules*)
+                 #:extensions dependencies))
+
+  (define *config*
+    (scheme-node "guix-config"
+                 `(((guix config)
+                    => ,(make-config.scm #:libgcrypt libgcrypt
+                                         #:zlib zlib
+                                         #:gzip gzip
+                                         #:bzip2 bzip2
+                                         #:xz xz
+                                         #:package-name
+                                         %guix-package-name
+                                         #:package-version
+                                         version
+                                         #:bug-report-address
+                                         %guix-bug-report-address
+                                         #:home-page-url
+                                         %guix-home-page-url)))))
+
+  (directory-union (string-append "guix-" %guix-version)
+                   (append-map (lambda (node)
+                                 (list (node-source node)
+                                       (node-compiled node)))
+                               (list *config*
+                                     *cli-modules*
+                                     *system-modules*
+                                     *package-modules*
+                                     *extra-modules*
+                                     *core-modules*))
+
+                   ;; When we do (add-to-store "utils.scm"), "utils.scm" must
+                   ;; be a regular file, not a symlink.  Thus, arrange so that
+                   ;; regular files appear as regular files in the final
+                   ;; output.
+                   #:copy? #t
+                   #:quiet? #t))
+
+\f
+;;;
+;;; (guix config) generation.
+;;;
+
+(define %dependency-variables
+  ;; (guix config) variables corresponding to dependencies.
+  '(%libgcrypt %libz %xz %gzip %bzip2 %nix-instantiate))
+
+(define %config-variables
+  ;; (guix config) variables corresponding to Guix configuration (storedir,
+  ;; localstatedir, etc.)
+  (filter pair?
+          (module-map (lambda (name var)
+                        (and (not (memq name %dependency-variables))
+                             (cons name (variable-ref var))))
+                      (resolve-interface '(guix config)))))
+
+(define* (make-config.scm #:key libgcrypt zlib gzip xz bzip2
+                          (package-name "GNU Guix")
+                          (package-version "0")
+                          (bug-report-address "bug-guix@gnu.org")
+                          (home-page-url "https://gnu.org/s/guix"))
+
+  ;; Hack so that Geiser is not confused.
+  (define defmod 'define-module)
+
+  (scheme-file "config.scm"
+               #~(begin
+                   (#$defmod (guix config)
+                     #:export (%guix-package-name
+                               %guix-version
+                               %guix-bug-report-address
+                               %guix-home-page-url
+                               %libgcrypt
+                               %libz
+                               %gzip
+                               %bzip2
+                               %xz
+                               %nix-instantiate))
+
+                   ;; XXX: Work around <http://bugs.gnu.org/15602>.
+                   (eval-when (expand load eval)
+                     #$@(map (match-lambda
+                               ((name . value)
+                                #~(define-public #$name #$value)))
+                             %config-variables)
+
+                     (define %guix-package-name #$package-name)
+                     (define %guix-version #$package-version)
+                     (define %guix-bug-report-address #$bug-report-address)
+                     (define %guix-home-page-url #$home-page-url)
+
+                     (define %gzip
+                       #+(and gzip (file-append gzip "/bin/gzip")))
+                     (define %bzip2
+                       #+(and bzip2 (file-append bzip2 "/bin/bzip2")))
+                     (define %xz
+                       #+(and xz (file-append xz "/bin/xz")))
+
+                     (define %libgcrypt
+                       #+(and libgcrypt
+                              (file-append libgcrypt "/lib/libgcrypt")))
+                     (define %libz
+                       #+(and zlib
+                              (file-append zlib "/lib/libz")))
+
+                     (define %nix-instantiate     ;for (guix import snix)
+                       "nix-instantiate")))))
+
+
+\f
+;;;
+;;; Building.
+;;;
+
+(define (imported-files name files)
+  ;; This is a non-monadic, simplified version of 'imported-files' from (guix
+  ;; gexp).
+  (define build
+    (with-imported-modules (source-module-closure
+                            '((guix build utils)))
+      #~(begin
+          (use-modules (ice-9 match)
+                       (guix build utils))
+
+          (mkdir (ungexp output)) (chdir (ungexp output))
+          (for-each (match-lambda
+                      ((final-path store-path)
+                       (mkdir-p (dirname final-path))
+
+                       ;; Note: We need regular files to be regular files, not
+                       ;; symlinks, as this makes a difference for
+                       ;; 'add-to-store'.
+                       (copy-file store-path final-path)))
+                    '#$files))))
+
+  (computed-file name build))
+
+(define* (compiled-modules name module-tree modules
+                           #:optional
+                           (dependencies '())
+                           (dependencies-compiled '())
+                           #:key
+                           (extensions '())       ;full-blown Guile packages
+                           parallel?)
+  ;; This is a non-monadic, enhanced version of 'compiled-file' from (guix
+  ;; gexp).
+  (define build
+    (with-imported-modules (source-module-closure
+                            '((guix build compile)
+                              (guix build utils)))
+      #~(begin
+          (use-modules (srfi srfi-26)
+                       (ice-9 match)
+                       (ice-9 format)
+                       (ice-9 threads)
+                       (guix build compile)
+                       (guix build utils))
+
+          (define (regular? file)
+            (not (member file '("." ".."))))
+
+          (define (process-file file output)
+            (let* ((base   (string-drop-right file 4)) ;.scm
+                   (output (string-append output "/" base
+                                          ".go")))
+              (compile-file file
+                            #:output-file output
+                            #:opts (optimization-options file))))
+
+          (define (report-load file total completed)
+            (display #\cr)
+            (format #t
+                    "loading...\t~5,1f% of ~d files" ;FIXME: i18n
+                    (* 100. (/ completed total)) total)
+            (force-output))
+
+          (define (report-compilation file total completed)
+            (display #\cr)
+            (format #t "compiling...\t~5,1f% of ~d files" ;FIXME: i18n
+                    (* 100. (/ completed total)) total)
+            (force-output))
+
+          (define (process-directory directory output)
+            (let ((files  (find-files directory "\\.scm$"))
+                  (prefix (+ 1 (string-length directory))))
+              ;; Hide compilation warnings.
+              (parameterize ((current-warning-port (%make-void-port "w")))
+                (compile-files directory #$output
+                               (map (cut string-drop <> prefix) files)
+                               #:workers (parallel-job-count)
+                               #:report-load report-load
+                               #:report-compilation report-compilation))))
+
+          (debug-disable 'warn-deprecated)
+          (setvbuf (current-output-port) _IONBF)
+          (setvbuf (current-error-port) _IONBF)
+
+          (set! %load-path (cons #+module-tree %load-path))
+          (set! %load-path
+            (append '#+dependencies
+                    (map (lambda (extension)
+                           (string-append extension "/share/guile/site/"
+                                          (effective-version)))
+                         '#+extensions)
+                    %load-path))
+
+          (set! %load-compiled-path
+            (append '#+dependencies-compiled
+                    (map (lambda (extension)
+                           (string-append extension "/lib/guile/"
+                                          (effective-version)
+                                          "/site-ccache"))
+                         '#+extensions)
+                    %load-compiled-path))
+
+          ;; Load the compiler modules upfront.
+          (compile #f)
+
+          (mkdir #$output)
+          (chdir #+module-tree)
+          (process-directory "." #$output))))
+
+  (computed-file name build
+                 #:options '()))
+
+\f
+;;;
+;;; Live patching.
+;;;
+
+(define (recursive-submodules module)
+  "Return the list of submodules of MODULE."
+  (let loop ((module module)
+             (result '()))
+    (let ((submodules (hash-map->list (lambda (name module)
+                                        module)
+                                      (module-submodules module))))
+      (fold loop (append submodules result) submodules))))
+
+(define (remove-submodule! module names)
+  (let loop ((module module)
+             (names names))
+    (match names
+      (() #t)
+      ((head tail ...)
+       (match (nested-ref-module module tail)
+         (#f #t)
+         ((? module? submodule)
+          (hashq-remove! (module-submodules module) head)
+          (loop submodule tail)))))))
+
+(define (unload-module-tree! module)
+  (define (strip-prefix prefix lst)
+    (let loop ((prefix prefix)
+               (lst lst))
+      (match prefix
+        (()
+         lst)
+        ((_ prefix ...)
+         (match lst
+           ((_ lst ...)
+            (loop prefix lst)))))))
+
+  (let ((submodules (hash-map->list (lambda (name module)
+                                      module)
+                                    (module-submodules module))))
+    (let loop ((root module)
+               (submodules submodules))
+      (match submodules
+        (()
+         #t)
+        ((head tail ...)
+         (unload-module-tree! head)
+         (remove-submodule! root
+                            (strip-prefix (module-name root)
+                                          (module-name head)))
+
+         (match (module-name head)
+           ((parents ... leaf)
+            ;; Remove MODULE from the AUTOLOADS-DONE list.  Note: We don't use
+            ;; 'module-filename' because it could be an absolute file name.
+            (set-autoloaded! (string-join (map symbol->string parents)
+                                          "/" 'suffix)
+                             (symbol->string leaf) #f)))
+         (loop root tail))))))
+
+(define* (reload-guix #:optional (log-port (current-error-port)))
+  "Reload all the Guix and GNU modules currently loaded."
+  (let* ((guix (resolve-module '(guix) #f #:ensure #f))
+         (gnu  (resolve-module '(gnu) #f #:ensure #f))
+         (guix-submodules (recursive-submodules guix))
+         (gnu-submodules  (recursive-submodules gnu)))
+    (define (reload module)
+      (match (module-filename module)
+        (#f #f)
+        ((? string? file)
+         (resolve-module (module-name module)))))
+
+    ;; Ignore any available .go, and force recompilation.  This is because our
+    ;; checkout in the store has mtime set to the epoch, and thus .go files look
+    ;; newer, even though they may not correspond.
+    (set! %fresh-auto-compile #t)
+
+    ;; First, we need to nuke all the (guix) and (gnu) submodules so we don't
+    ;; end up with a mixture of old and new modules when we reload (which
+    ;; wouldn't work, because we'd have two different <package> record types,
+    ;; for instance.)
+    (format log-port "Unloading current Guix...~%")
+    (unload-module-tree! gnu)
+    (unload-module-tree! guix)
+
+    (format log-port "Loading new Guix...~%")
+    (for-each reload (append guix-submodules (list guix)))
+    (for-each reload (append gnu-submodules (list gnu)))
+    (format log-port "New Guix modules successfully loaded.~%")))
+
+\f
+;;;
+;;; Building.
+;;;
+
+(define* (guile-for-build #:optional (version (effective-version)))
+  "Return a package for Guile VERSION."
+  (define canonical-package                       ;soft reference
+    (module-ref (resolve-interface '(gnu packages base))
+                'canonical-package))
+
+  (match version
+    ("2.2"
+     (canonical-package
+      (specification->package "guile@2.2")))
+    ("2.0"
+     (canonical-package
+      (specification->package "guile@2.0")))))
+
+(define* (guix-derivation source version
+                          #:optional (guile-version (effective-version)))
+  "Return, as a monadic value, the derivation to build the Guix from SOURCE
+for GUILE-VERSION.  Use VERSION as the version string."
+  (mbegin %store-monad
+    (set-guile-for-build (guile-for-build guile-version))
+    (lower-object (compiled-guix source
+                                 #:version version
+                                 #:guile-version guile-version))))
-- 
2.14.2

bug#27284: [PATCH 0/8] 'guix pull' creates several derivations

[Ludovic Courtès]

Hello!

Ludovic Courtès <ludo@gnu.org> skribis:

> This patch set leads us to a ‘guix pull’ that creates several
> derivations instead of just one: “guix-core” (38 files), “guix-extra”,
> “guix-packages” (388 files), “guix-cli”, “guix-system”.
>
> The idea is that we should be able to have substitutes for at least some
> of them, for instance because “guix-core” doesn’t need to be rebuilt
> very often.  Most of the time, users will just have to build
> “guix-packages”, which is still a bit slow, but will hopefully get
> better as Guile incorporates those long-awaited fixes.

I’ve just pushed this, let me know how ‘guix pull’ works for you!

In addition to this, I’ve created a Hydra job to build this new
“modular Guix”:

  https://hydra.gnu.org/jobset/guix/modular

If everything goes well, this should give us substitutes for at least
“guix-core” and “guix-extra”.  We have yet to see how long it takes to
evaluate it on hydra.gnu.org and adjust the evaluation frequency
accordingly.

> The ‘reload-guix’ procedure there is here to fix that: we reload all the
> Guix modules of the target checkout, and run the build procedure from
> that context.  Unfortunately, it’s not fully baked yet because reloading
> leads to incompatibilities: the (guix scripts pull) module remains in
> the “old world” and manipulates the old <derivation> and <package>
> record types, which is different from the new ones.  That should be
> fixable, but requires some more time and fiddling.  Help from Guilers is
> very much welcome!  :-)

This part is currently commented out, it needs more work.  The main
issue is performance: we end up evaluating or building a lot of modules,
notably package modules, and again we hit the performance issues of the
compiler in Guile 2.2.2.

Ludo’.

bug#27284: [PATCH 0/8] 'guix pull' creates several derivations

[Ludovic Courtès]

ludo@gnu.org (Ludovic Courtès) skribis:

> Ludovic Courtès <ludo@gnu.org> skribis:
>
>> This patch set leads us to a ‘guix pull’ that creates several
>> derivations instead of just one: “guix-core” (38 files), “guix-extra”,
>> “guix-packages” (388 files), “guix-cli”, “guix-system”.
>>
>> The idea is that we should be able to have substitutes for at least some
>> of them, for instance because “guix-core” doesn’t need to be rebuilt
>> very often.  Most of the time, users will just have to build
>> “guix-packages”, which is still a bit slow, but will hopefully get
>> better as Guile incorporates those long-awaited fixes.
>
> I’ve just pushed this, let me know how ‘guix pull’ works for you!

On IRC Marius reported that it doesn’t work, so I reverted (a very
short-lived commit!).

The issue is that we can’t really get away with module reloading, I’m
afraid.

Ludo’.

bug#27284: [PATCH 0/4] 'guix pull' reloads modules, second try

[Ludovic Courtès]

Hello!

I’ve pushed the following patches to ‘wip-pull-reload’ so you can test
from whatever snapshot you’re currently running:

  guix pull --branch=wip-pull-reload

As a reminder, this patch set does two things:

  1. Build Guix as several derivations, with the eventual goal of having
     substitutes made available by
     <https://hydra.gnu.org/jobset/guix/modular>.  Typically only the
     derivation corresponding to package modules will have to be built,
     which is already a lot, but better than rebuilding everything every
     time.

  2. Load the target Guix modules.  What this means is that the
     derivations above will depend on, say, the latest guile-git and
     guile-ssh package.

     Conversely, we’re currently using dependencies from the current
     Guix, which is a problem, for instance if one of the dependencies
     is broken as was the case in
     <https://debbugs.gnu.org/cgi/bugreport.cgi?bug=28988>.

#2 is tricky because it means we first have to re-compile and
re-evaluate a whole bunch of modules.  The approach that seems to work
OK is to compile just the core modules and to evaluate the others.
That’s still taking ~20 seconds or so on my laptop.

Thanks,
Ludo’.

Ludovic Courtès (4):
  gnu: Fix ambiguous 'zip' reference.
  gexp: 'computed-file' has a new #:guile parameter.
  Add (guix self) and use it when pulling.
  pull: Reload modules before doing anything else.

 Makefile.am              |   1 +
 build-aux/build-self.scm | 289 +++++----------------
 gnu/packages.scm         |  21 +-
 gnu/packages/photo.scm   |   4 +-
 guix/discovery.scm       |   3 +-
 guix/gexp.scm            |  16 +-
 guix/scripts/pull.scm    |  91 ++++---
 guix/self.scm            | 652 +++++++++++++++++++++++++++++++++++++++++++++++
 po/guix/POTFILES.in      |   1 +
 9 files changed, 804 insertions(+), 274 deletions(-)
 create mode 100644 guix/self.scm

-- 
2.15.1

bug#27284: [PATCH 1/4] gnu: Fix ambiguous 'zip' reference.

[Ludovic Courtès]

* gnu/packages/photo.scm: Hide 'zip' from (srfi srfi-1).
---
 gnu/packages/photo.scm | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/gnu/packages/photo.scm b/gnu/packages/photo.scm
index 666058c9d..7e9313f82 100644
--- a/gnu/packages/photo.scm
+++ b/gnu/packages/photo.scm
@@ -1,5 +1,5 @@
 ;;; GNU Guix --- Functional package management for GNU
-;;; Copyright © 2014, 2015 Ludovic Courtès <ludo@gnu.org>
+;;; Copyright © 2014, 2015, 2017 Ludovic Courtès <ludo@gnu.org>
 ;;; Copyright © 2015 Ricardo Wurmus <rekado@elephly.net>
 ;;; Copyright © 2015, 2017 Andreas Enge <andreas@enge.fr>
 ;;; Copyright © 2016, 2017 Efraim Flashner <efraim@flashner.co.il>
@@ -59,7 +59,7 @@
   #:use-module (gnu packages xfig)
   #:use-module (gnu packages xorg)
   #:use-module (gnu packages xml)
-  #:use-module (srfi srfi-1)
+  #:use-module ((srfi srfi-1) #:hide (zip))
   #:use-module (srfi srfi-26))
 
 (define-public libraw
-- 
2.15.1

bug#27284: [PATCH 2/4] gexp: 'computed-file' has a new #:guile parameter.

[Ludovic Courtès]

* guix/gexp.scm (<computed-file>)[guile]: New field.
(computed-file): Add #:guile.
(computed-file-compiler): Honor 'guile'.
---
 guix/gexp.scm | 16 +++++++++++-----
 1 file changed, 11 insertions(+), 5 deletions(-)

diff --git a/guix/gexp.scm b/guix/gexp.scm
index 1929947d9..f005c4d29 100644
--- a/guix/gexp.scm
+++ b/guix/gexp.scm
@@ -343,28 +343,34 @@ This is the declarative counterpart of 'text-file'."
      (text-file name content references))))
 
 (define-record-type <computed-file>
-  (%computed-file name gexp options)
+  (%computed-file name gexp guile options)
   computed-file?
   (name       computed-file-name)                 ;string
   (gexp       computed-file-gexp)                 ;gexp
+  (guile      computed-file-guile)                ;<package>
   (options    computed-file-options))             ;list of arguments
 
 (define* (computed-file name gexp
-                        #:key (options '(#:local-build? #t)))
+                        #:key guile (options '(#:local-build? #t)))
   "Return an object representing the store item NAME, a file or directory
 computed by GEXP.  OPTIONS is a list of additional arguments to pass
 to 'gexp->derivation'.
 
 This is the declarative counterpart of 'gexp->derivation'."
-  (%computed-file name gexp options))
+  (%computed-file name gexp guile options))
 
 (define-gexp-compiler (computed-file-compiler (file <computed-file>)
                                               system target)
   ;; Compile FILE by returning a derivation whose build expression is its
   ;; gexp.
   (match file
-    (($ <computed-file> name gexp options)
-     (apply gexp->derivation name gexp options))))
+    (($ <computed-file> name gexp guile options)
+     (if guile
+         (mlet %store-monad ((guile (lower-object guile system
+                                                  #:target target)))
+           (apply gexp->derivation name gexp #:guile-for-build guile
+                  options))
+         (apply gexp->derivation name gexp options)))))
 
 (define-record-type <program-file>
   (%program-file name gexp guile)
-- 
2.15.1

bug#27284: [PATCH 3/4] Add (guix self) and use it when pulling.

[Ludovic Courtès]

This mitigates <https://bugs.gnu.org/27284>.

* guix/self.scm: New file.
* Makefile.am (MODULES): Add it.
* po/guix/POTFILES.in: Add it.
* build-aux/build-self.scm (libgcrypt, zlib, gzip, bzip2, xz)
(false-if-wrong-guile, package-for-current-guile, guile-json)
(guile-ssh, guile-git, guile-bytestructures, matching-guile-2.2): Remove.
(build): Rewrite to simply delegate to 'compiled-guix'.
* gnu/packages.scm (%distro-root-directory): Rewrite to try different
directories.
* guix/discovery.scm (guix): Export 'scheme-files'.
* guix/scripts/pull.scm (build-and-install): Split into...
(install-latest): ... this.  New procedure.  And...
(build-and-install): ... this, which now takes a monadic value argument.
(indirect-root-added): Remove.
(guix-pull): Call 'add-indirect-root'.  Call 'build-from-source' and
pass the result to 'build-and-install'.
---
 Makefile.am              |   1 +
 build-aux/build-self.scm | 287 ++++-----------------
 gnu/packages.scm         |  21 +-
 guix/discovery.scm       |   3 +-
 guix/scripts/pull.scm    |  91 ++++---
 guix/self.scm            | 652 +++++++++++++++++++++++++++++++++++++++++++++++
 po/guix/POTFILES.in      |   1 +
 7 files changed, 788 insertions(+), 268 deletions(-)
 create mode 100644 guix/self.scm

diff --git a/Makefile.am b/Makefile.am
index ddbf7a798..c0774d4f3 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -67,6 +67,7 @@ MODULES =					\
   guix/derivations.scm				\
   guix/grafts.scm				\
   guix/gnu-maintenance.scm			\
+  guix/self.scm					\
   guix/upstream.scm				\
   guix/licenses.scm				\
   guix/git.scm					\
diff --git a/build-aux/build-self.scm b/build-aux/build-self.scm
index 641597883..d9d926367 100644
--- a/build-aux/build-self.scm
+++ b/build-aux/build-self.scm
@@ -17,11 +17,9 @@
 ;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
 
 (define-module (build-self)
-  #:use-module (gnu)
-  #:use-module (guix)
-  #:use-module (guix config)
   #:use-module (srfi srfi-1)
   #:use-module (srfi srfi-19)
+  #:use-module (srfi srfi-26)
   #:use-module (ice-9 match)
   #:export (build))
 
@@ -31,122 +29,51 @@
 ;;; argument: the source tree to build.  It returns a derivation that
 ;;; builds it.
 ;;;
-;;; This file uses modules provided by the already-installed Guix.  Those
-;;; modules may be arbitrarily old compared to the version we want to
-;;; build.  Because of that, it must rely on the smallest set of features
-;;; that are likely to be provided by the (guix) and (gnu) modules, and by
-;;; Guile itself, forever and ever.
-;;;
 ;;; Code:
 
-\f
-;; The dependencies.  Don't refer explicitly to the variables because they
-;; could be renamed or shuffled around in modules over time.  Conversely,
-;; 'find-best-packages-by-name' is expected to always have the same semantics.
-
-(define libgcrypt
-  (first (find-best-packages-by-name "libgcrypt" #f)))
-
-(define zlib
-  (first (find-best-packages-by-name "zlib" #f)))
-
-(define gzip
-  (first (find-best-packages-by-name "gzip" #f)))
-
-(define bzip2
-  (first (find-best-packages-by-name "bzip2" #f)))
-
-(define xz
-  (first (find-best-packages-by-name "xz" #f)))
-
-(define (false-if-wrong-guile package)
-  "Return #f if PACKAGE depends on the \"wrong\" major version of Guile (e.g.,
-2.0 instead of 2.2), otherwise return PACKAGE."
-  (let ((guile (any (match-lambda
-                      ((label (? package? dep) _ ...)
-                       (and (string=? (package-name dep) "guile")
-                            dep)))
-                    (package-direct-inputs package))))
-    (and (or (not guile)
-             (string-prefix? (effective-version)
-                             (package-version guile)))
-         package)))
-
-(define (package-for-current-guile . names)
-  "Return the package with one of the given NAMES that depends on the current
-Guile major version (2.0 or 2.2), or #f if none of the packages matches."
-  (let loop ((names names))
-    (match names
-      (()
-       #f)
-      ((name rest ...)
-       (match (find-best-packages-by-name name #f)
-         (()
-          (loop rest))
-         ((first _ ...)
-          (or (false-if-wrong-guile first)
-              (loop rest))))))))
-
-(define guile-json
-  (package-for-current-guile "guile-json"
-                             "guile2.2-json"
-                             "guile2.0-json"))
-
-(define guile-ssh
-  (package-for-current-guile "guile-ssh"
-                             "guile2.2-ssh"
-                             "guile2.0-ssh"))
-
-(define guile-git
-  (package-for-current-guile "guile-git"
-                             "guile2.0-git"))
-
-(define guile-bytestructures
-  (package-for-current-guile "guile-bytestructures"
-                             "guile2.0-bytestructures"))
-\f
-;; The actual build procedure.
-
-(define (top-source-directory)
-  "Return the name of the top-level directory of this source tree."
+;; Use our very own Guix modules.
+(eval-when (compile load eval)
   (and=> (assoc-ref (current-source-location) 'filename)
          (lambda (file)
-           (string-append (dirname file) "/.."))))
-
+           (let ((dir (string-append (dirname file) "/..")))
+             (set! %load-path (cons dir %load-path))))))
 
 (define (date-version-string)
   "Return the current date and hour in UTC timezone, for use as a poor
 person's version identifier."
-  ;; XXX: Replace with a Git commit id.
+  ;; XXX: Last resort when the Git commit id is missing.
   (date->string (current-date 0) "~Y~m~d.~H"))
 
-(define (matching-guile-2.2)
-  "Return a Guile 2.2 with the same version as the current one or immediately
-older than then current one.  This is so that we do not build ABI-incompatible
-objects.  See <https://bugs.gnu.org/29570>."
-  (let loop ((packages (find-packages-by-name "guile" "2.2"))
-             (best     #f))
-    (match packages
-      (()
-       best)
-      ((head tail ...)
-       (if (string=? (package-version head) (version))
-           head
-           (if best
-               (if (version>? (package-version head) (version))
-                   (loop tail best)
-                   (loop tail head))
-               (loop tail head)))))))
+(define-syntax parameterize*
+  (syntax-rules ()
+    "Like 'parameterize' but for regular variables (!)."
+    ((_ ((var value) rest ...) body ...)
+     (let ((old var)
+           (new value))
+       (dynamic-wind
+         (lambda ()
+           (set! var new))
+         (lambda ()
+           (parameterize* (rest ...) body ...))
+         (lambda ()
+           (set! var old)))))
+    ((_ () body ...)
+     (begin body ...))))
 
-(define (guile-for-build)
-  "Return a derivation for Guile 2.0 or 2.2, whichever matches the currently
-running Guile."
-  (package->derivation (cond-expand
-                         (guile-2.2
-                          (canonical-package (matching-guile-2.2)))
-                         (else
-                          (canonical-package
-                           (specification->package "guile@2.0"))))))
+(define (pure-load-compiled-path)
+  "Return %LOAD-COMPILED-PATH minus the directories containing .go files from
+Guix."
+  (define (purify path)
+    (fold-right delete path
+                (filter-map (lambda (file)
+                              (and=> (search-path path file) dirname))
+                            '("guix.go" "gnu.go"))))
+
+  (let loop ((path %load-compiled-path))
+    (let ((next (purify path)))
+      (if (equal? next path)
+          path
+          (loop next)))))
 
 ;; The procedure below is our return value.
 (define* (build source
@@ -155,131 +82,29 @@ running Guile."
                 #:rest rest)
   "Return a derivation that unpacks SOURCE into STORE and compiles Scheme
 files."
-  ;; The '%xxxdir' variables were added to (guix config) in July 2016 so we
-  ;; cannot assume that they are defined.  Try to guess their value when
-  ;; they're undefined (XXX: we get an incorrect guess when environment
-  ;; variables such as 'NIX_STATE_DIR' are defined!).
-  (define storedir
-    (if (defined? '%storedir) %storedir %store-directory))
-  (define localstatedir
-    (if (defined? '%localstatedir) %localstatedir (dirname %state-directory)))
-  (define sysconfdir
-    (if (defined? '%sysconfdir) %sysconfdir (dirname %config-directory)))
-  (define sbindir
-    (if (defined? '%sbindir) %sbindir (dirname %guix-register-program)))
-
-  (define builder
-    #~(begin
-        (use-modules (guix build pull))
-
-        (letrec-syntax ((maybe-load-path
-                         (syntax-rules ()
-                           ((_ item rest ...)
-                            (let ((tail (maybe-load-path rest ...)))
-                              (if (string? item)
-                                  (cons (string-append item
-                                                       "/share/guile/site/"
-                                                       #$(effective-version))
-                                        tail)
-                                  tail)))
-                           ((_)
-                            '()))))
-          (set! %load-path
-                (append
-                 (maybe-load-path #$guile-json #$guile-ssh
-                                  #$guile-git #$guile-bytestructures)
-                 %load-path)))
-
-        (letrec-syntax ((maybe-load-compiled-path
-                         (syntax-rules ()
-                           ((_ item rest ...)
-                            (let ((tail (maybe-load-compiled-path rest ...)))
-                              (if (string? item)
-                                  (cons (string-append item
-                                                       "/lib/guile/"
-                                                       #$(effective-version)
-                                                       "/site-ccache")
-                                        tail)
-                                  tail)))
-                           ((_)
-                            '()))))
-          (set! %load-compiled-path
-                (append
-                 (maybe-load-compiled-path #$guile-json #$guile-ssh
-                                           #$guile-git #$guile-bytestructures)
-                 %load-compiled-path)))
-
-        ;; XXX: The 'guile-ssh' package prior to Guix commit 92b7258 was
-        ;; broken: libguile-ssh could not be found.  Work around that.
-        ;; FIXME: We want Guile-SSH 0.10.2 or later anyway.
-        #$(if (string-prefix? "0.9." (package-version guile-ssh))
-              #~(setenv "LTDL_LIBRARY_PATH" (string-append #$guile-ssh "/lib"))
-              #t)
-
-        (build-guix #$output #$source
-
-                    #:system #$%system
-                    #:storedir #$storedir
-                    #:localstatedir #$localstatedir
-                    #:sysconfdir #$sysconfdir
-                    #:sbindir #$sbindir
-
-                    #:package-name #$%guix-package-name
-                    #:package-version #$version
-                    #:bug-report-address #$%guix-bug-report-address
-                    #:home-page-url #$%guix-home-page-url
-
-                    #:libgcrypt #$libgcrypt
-                    #:zlib #$zlib
-                    #:gzip #$gzip
-                    #:bzip2 #$bzip2
-                    #:xz #$xz
-
-                    ;; XXX: This is not perfect, enabling VERBOSE? means
-                    ;; building a different derivation.
-                    #:debug-port (if #$verbose?
-                                     (current-error-port)
-                                     (%make-void-port "w")))))
-
-  (unless guile-git
-    ;; XXX: Guix before February 2017 lacks a 'guile-git' package altogether.
-    ;; If we try to upgrade anyway, the logic in (guix scripts pull) will not
-    ;; build (guix git), which will leave us with an unusable 'guix pull'.  To
-    ;; avoid that, fail early.
-    (format (current-error-port)
-            "\
-Your installation is too old and lacks a '~a' package.
-Please upgrade to an intermediate version first, for instance with:
-
-  guix pull --url=https://git.savannah.gnu.org/cgit/guix.git/snapshot/v0.13.0.tar.gz
-\n"
-            (match (effective-version)
-              ("2.0" "guile2.0-git")
-              (_     "guile-git")))
-    (exit 1))
-
-  (mlet %store-monad ((guile (guile-for-build)))
-    (gexp->derivation "guix-latest" builder
-                      #:modules '((guix build pull)
-                                  (guix build utils)
-                                  (guix build compile)
-
-                                  ;; Closure of (guix modules).
-                                  (guix modules)
-                                  (guix memoization)
-                                  (guix sets))
-
-                      ;; Arrange so that our own (guix build …) modules are
-                      ;; used.
-                      #:module-path (list (top-source-directory))
-
-                      #:guile-for-build guile)))
+  ;; Start by jumping into the target Guix so that we have access to the
+  ;; latest packages and APIs.
+  ;;
+  ;; Our checkout in the store has mtime set to the epoch, and thus .go
+  ;; files look newer, even though they may not correspond.
+  (parameterize* ((%load-should-auto-compile #f)
+                  (%fresh-auto-compile #f)
+
+                  ;; Work around <https://bugs.gnu.org/29226>.
+                  (%load-compiled-path (pure-load-compiled-path)))
+    ;; FIXME: This is currently too expensive notably because it involves
+    ;; compiling a number of the big package files such as perl.scm, which
+    ;; takes lots of time and memory as of Guile 2.2.2.
+    ;;
+    ;; (let ((reload-guix (module-ref (resolve-interface '(guix self))
+    ;;                                'reload-guix)))
+    ;;   (reload-guix))           ;cross fingers!
+
+    (let ((guix-derivation (module-ref (resolve-interface '(guix self))
+                                       'guix-derivation)))
+      (guix-derivation source version))))
 
 ;; This file is loaded by 'guix pull'; return it the build procedure.
 build
 
-;; Local Variables:
-;; eval: (put 'with-load-path 'scheme-indent-function 1)
-;; End:
-
 ;;; build-self.scm ends here
diff --git a/gnu/packages.scm b/gnu/packages.scm
index 97e6cb347..44a56dfde 100644
--- a/gnu/packages.scm
+++ b/gnu/packages.scm
@@ -110,8 +110,25 @@ for system '~a'")
                         file-name system)))))))
 
 (define %distro-root-directory
-  ;; Absolute file name of the module hierarchy.
-  (dirname (search-path %load-path "guix.scm")))
+  ;; Absolute file name of the module hierarchy.  Since (gnu packages …) might
+  ;; live in a directory different from (guix), try to get the best match.
+  (letrec-syntax ((dirname* (syntax-rules ()
+                              ((_ file)
+                               (dirname file))
+                              ((_ file head tail ...)
+                               (dirname (dirname* file tail ...)))))
+                  (try      (syntax-rules ()
+                              ((_ (file things ...) rest ...)
+                               (match (search-path %load-path file)
+                                 (#f
+                                  (try rest ...))
+                                 (absolute
+                                  (dirname* absolute things ...))))
+                              ((_)
+                               #f))))
+    (try ("gnu/packages/base.scm" gnu/ packages/)
+         ("gnu/packages.scm"      gnu/)
+         ("guix.scm"))))
 
 (define %package-module-path
   ;; Search path for package modules.  Each item must be either a directory
diff --git a/guix/discovery.scm b/guix/discovery.scm
index 7b5757902..8ffcf7cd9 100644
--- a/guix/discovery.scm
+++ b/guix/discovery.scm
@@ -25,7 +25,8 @@
   #:use-module (ice-9 match)
   #:use-module (ice-9 vlist)
   #:use-module (ice-9 ftw)
-  #:export (scheme-modules
+  #:export (scheme-files
+            scheme-modules
             fold-modules
             all-modules
             fold-module-public-variables))
diff --git a/guix/scripts/pull.scm b/guix/scripts/pull.scm
index 64c2196e0..313d87344 100644
--- a/guix/scripts/pull.scm
+++ b/guix/scripts/pull.scm
@@ -150,8 +150,6 @@ Download and deploy the latest version of Guix.\n"))
 
 (define what-to-build
   (store-lift show-what-to-build))
-(define indirect-root-added
-  (store-lift add-indirect-root))
 
 (define %self-build-file
   ;; The file containing code to build Guix.  This serves the same purpose as
@@ -172,33 +170,48 @@ contained therein.  Use COMMIT as the version string."
     ;; tree.
     (build source #:verbose? verbose? #:version commit)))
 
-(define* (build-and-install source config-dir
-                            #:key verbose? commit)
-  "Build the tool from SOURCE, and install it in CONFIG-DIR."
-  (mlet* %store-monad ((source        (build-from-source source
-                                                         #:commit commit
-                                                         #:verbose? verbose?))
-                       (source-dir -> (derivation->output-path source))
-                       (to-do?        (what-to-build (list source)))
-                       (built?        (built-derivations (list source))))
-    ;; Always update the 'latest' symlink, regardless of whether SOURCE was
-    ;; already built or not.
-    (if built?
-        (mlet* %store-monad
-            ((latest -> (string-append config-dir "/latest"))
-             (done      (indirect-root-added latest)))
-          (if (and (file-exists? latest)
-                   (string=? (readlink latest) source-dir))
-              (begin
-                (display (G_ "Guix already up to date\n"))
-                (return #t))
-              (begin
-                (switch-symlinks latest source-dir)
-                (format #t
-                        (G_ "updated ~a successfully deployed under `~a'~%")
-                        %guix-package-name latest)
-                (return #t))))
-        (leave (G_ "failed to update Guix, check the build log~%")))))
+(define* (install-latest source-dir config-dir)
+  "Make SOURCE-DIR, a store file name, the latest Guix in CONFIG-DIR."
+  (let ((latest (string-append config-dir "/latest")))
+    (if (and (file-exists? latest)
+             (string=? (readlink latest) source-dir))
+        (begin
+          (display (G_ "Guix already up to date\n"))
+          #t)
+        (begin
+          (switch-symlinks latest source-dir)
+          (format #t
+                  (G_ "updated ~a successfully deployed under `~a'~%")
+                  %guix-package-name latest)
+          #t))))
+
+(define (build-and-install mdrv)
+  "Bind MDRV, a monadic value for a derivation, build it, and finally install
+it as the latest Guix."
+  (define do-it
+    ;; Weirdness follows!  Before we were called, the Guix modules have
+    ;; probably been reloaded, leading to a "parallel universe" with disjoint
+    ;; record types.  However, procedures in this file have already cached the
+    ;; module relative to which they lookup global bindings (see
+    ;; 'toplevel-box' documentation), so they're stuck in the old world.  To
+    ;; work around that, evaluate our procedure in the context of the "new"
+    ;; (guix scripts pull) module--which has access to the new <derivation>
+    ;; record, and so on.
+    (eval '(lambda (mdrv cont)
+             ;; Reopen a connection to the daemon so that we have a record
+             ;; with the new type.
+             (with-store store
+               (run-with-store store
+                 (mlet %store-monad ((drv mdrv))
+                   (mbegin %store-monad
+                     (what-to-build (list drv))
+                     (built-derivations (list drv))
+                     (return (cont (derivation->output-path drv))))))))
+          (resolve-module '(guix scripts pull)))) ;the new module
+
+  (do-it mdrv
+         (lambda (result)
+           (install-latest result (config-directory)))))
 
 (define (honor-lets-encrypt-certificates! store)
   "Tell Guile-Git to use the Let's Encrypt certificates."
@@ -259,6 +272,10 @@ certificates~%"))
              (when (use-le-certs? url)
                (honor-lets-encrypt-certificates! store))
 
+             ;; Ensure the 'latest' symlink is registered as a GC root.
+             (add-indirect-root store
+                                (string-append (config-directory) "/latest"))
+
              (format (current-error-port)
                      (G_ "Updating from Git repository at '~a'...~%")
                      url)
@@ -277,10 +294,16 @@ certificates~%"))
                                 (if (assoc-ref opts 'bootstrap?)
                                     %bootstrap-guile
                                     (canonical-package guile-2.2)))))
-                 (run-with-store store
-                   (build-and-install checkout (config-directory)
-                                      #:commit commit
-                                      #:verbose?
-                                      (assoc-ref opts 'verbose?))))))))))))
+
+                 ;; 'build-from-source' may cause a reload of the Guix
+                 ;; modules.  This leads to a parallel world: its record types
+                 ;; are disjoint from those we've seen until now (because we
+                 ;; use "generative" record types), and so on.  Thus, special
+                 ;; care must be taken once we have return from that call.
+                 (build-and-install
+                  (build-from-source checkout
+                                     #:commit commit
+                                     #:verbose?
+                                     (assoc-ref opts 'verbose?))))))))))))
 
 ;;; pull.scm ends here
diff --git a/guix/self.scm b/guix/self.scm
new file mode 100644
index 000000000..2f7f83b59
--- /dev/null
+++ b/guix/self.scm
@@ -0,0 +1,652 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2017 Ludovic Courtès <ludo@gnu.org>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (guix self)
+  #:use-module (guix config)
+  #:use-module (guix i18n)
+  #:use-module (guix modules)
+  #:use-module (guix gexp)
+  #:use-module (guix store)
+  #:use-module (guix monads)
+  #:use-module (guix discovery)
+  #:use-module (guix packages)
+  #:use-module (guix sets)
+  #:use-module (guix utils)
+  #:use-module (guix modules)
+  #:use-module (guix build utils)
+  #:use-module ((guix build compile) #:select (%lightweight-optimizations))
+  #:use-module (gnu packages)
+  #:use-module (srfi srfi-1)
+  #:use-module (srfi srfi-9)
+  #:use-module (ice-9 match)
+  #:export (compiled-guix
+            guix-derivation
+            reload-guix))
+
+\f
+;;;
+;;; Dependency handling.
+;;;
+
+(define* (false-if-wrong-guile package
+                               #:optional (guile-version (effective-version)))
+  "Return #f if PACKAGE depends on the \"wrong\" major version of Guile (e.g.,
+2.0 instead of 2.2), otherwise return PACKAGE."
+  (let ((guile (any (match-lambda
+                      ((label (? package? dep) _ ...)
+                       (and (string=? (package-name dep) "guile")
+                            dep)))
+                    (package-direct-inputs package))))
+    (and (or (not guile)
+             (string-prefix? guile-version
+                             (package-version guile)))
+         package)))
+
+(define (package-for-guile guile-version . names)
+  "Return the package with one of the given NAMES that depends on
+GUILE-VERSION (\"2.0\" or \"2.2\"), or #f if none of the packages matches."
+  (let loop ((names names))
+    (match names
+      (()
+       #f)
+      ((name rest ...)
+       (match (specification->package name)
+         (#f
+          (loop rest))
+         ((? package? package)
+          (or (false-if-wrong-guile package)
+              (loop rest))))))))
+
+\f
+;;;
+;;; Derivations.
+;;;
+
+;; Node in a DAG of build tasks.  Each node maps to a derivation, but it's
+;; easier to express things this way.
+(define-record-type <node>
+  (node name modules source dependencies compiled)
+  node?
+  (name          node-name)                       ;string
+  (modules       node-modules)                    ;list of module names
+  (source        node-source)                     ;list of source files
+  (dependencies  node-dependencies)               ;list of nodes
+  (compiled      node-compiled))                  ;node -> lowerable object
+
+(define (node-fold proc init nodes)
+  (let loop ((nodes nodes)
+             (visited (setq))
+             (result init))
+    (match nodes
+      (() result)
+      ((head tail ...)
+       (if (set-contains? visited head)
+           (loop tail visited result)
+           (loop tail (set-insert head visited)
+                 (proc head result)))))))
+
+(define (node-modules/recursive nodes)
+  (node-fold (lambda (node modules)
+               (append (node-modules node) modules))
+             '()
+             nodes))
+
+(define* (closure modules #:optional (except '()))
+  (source-module-closure modules
+                         #:select?
+                         (match-lambda
+                           (('guix 'config)
+                            #f)
+                           ((and module
+                                 (or ('guix _ ...) ('gnu _ ...)))
+                            (not (member module except)))
+                           (rest #f))))
+
+(define module->import
+  ;; Return a file-name/file-like object pair for the specified module and
+  ;; suitable for 'imported-files'.
+  (match-lambda
+    ((module '=> thing)
+     (let ((file (module-name->file-name module)))
+       (list file thing)))
+    (module
+        (let ((file (module-name->file-name module)))
+          (list file
+                (local-file (search-path %load-path file)))))))
+
+(define* (scheme-node name modules #:optional (dependencies '())
+                      #:key (extra-modules '()) (extra-files '())
+                      (extensions '())
+                      parallel? guile-for-build)
+  "Return a node that builds the given Scheme MODULES, and depends on
+DEPENDENCIES (a list of nodes).  EXTRA-MODULES is a list of additional modules
+added to the source, and EXTRA-FILES is a list of additional files.
+EXTENSIONS is a set of full-blown Guile packages (e.g., 'guile-json') that
+must be present in the search path."
+  (let* ((modules (append extra-modules
+                          (closure modules
+                                   (node-modules/recursive dependencies))))
+         (module-files (map module->import modules))
+         (source (imported-files (string-append name "-source")
+                                 (append module-files extra-files))))
+    (node name modules source dependencies
+          (compiled-modules name source modules
+                            (map node-source dependencies)
+                            (map node-compiled dependencies)
+                            #:extensions extensions
+                            #:parallel? parallel?
+                            #:guile-for-build guile-for-build))))
+
+(define (file-imports directory sub-directory pred)
+  "List all the files matching PRED under DIRECTORY/SUB-DIRECTORY.  Return a
+list of file-name/file-like objects suitable as inputs to 'imported-files'."
+  (map (lambda (file)
+         (list (string-drop file (+ 1 (string-length directory)))
+               (local-file file #:recursive? #t)))
+       (find-files (string-append directory "/" sub-directory) pred)))
+
+(define (scheme-modules* directory sub-directory)
+  "Return the list of module names found under SUB-DIRECTORY in DIRECTORY."
+  (let ((prefix (string-length directory)))
+    (map (lambda (file)
+           (file-name->module-name (string-drop file prefix)))
+         (scheme-files (string-append directory "/" sub-directory)))))
+
+(define* (compiled-guix source #:key (version %guix-version)
+                        (guile-version (effective-version))
+                        (guile-for-build (guile-for-build guile-version))
+                        (libgcrypt (specification->package "libgcrypt"))
+                        (zlib (specification->package "zlib"))
+                        (gzip (specification->package "gzip"))
+                        (bzip2 (specification->package "bzip2"))
+                        (xz (specification->package "xz")))
+  "Return a file-like object that contains a compiled Guix."
+  (define guile-json
+    (package-for-guile guile-version
+                       "guile-json"
+                       "guile2.2-json"
+                       "guile2.0-json"))
+
+  (define guile-ssh
+    (package-for-guile guile-version
+                       "guile-ssh"
+                       "guile2.2-ssh"
+                       "guile2.0-ssh"))
+
+  (define guile-git
+    (package-for-guile guile-version
+                       "guile-git"
+                       "guile2.0-git"))
+
+
+  (define dependencies
+    (match (append-map (lambda (package)
+                         (cons (list "x" package)
+                               (package-transitive-inputs package)))
+                       (list guile-git guile-json guile-ssh))
+      (((labels packages _ ...) ...)
+       packages)))
+
+  (define *core-modules*
+    (scheme-node "guix-core"
+                 '((guix)
+                   (guix monad-repl)
+                   (guix packages)
+                   (guix download)
+                   (guix discovery)
+                   (guix profiles)
+                   (guix build-system gnu)
+                   (guix build-system trivial)
+                   (guix build profiles)
+                   (guix build gnu-build-system))
+
+                 ;; Provide a dummy (guix config) with the default version
+                 ;; number, storedir, etc.  This is so that "guix-core" is the
+                 ;; same across all installations and doesn't need to be
+                 ;; rebuilt when the version changes, which in turn means we
+                 ;; can have substitutes for it.
+                 #:extra-modules
+                 `(((guix config)
+                    => ,(make-config.scm #:libgcrypt
+                                         (specification->package
+                                          "libgcrypt"))))
+
+                 #:guile-for-build guile-for-build))
+
+  (define *extra-modules*
+    (scheme-node "guix-extra"
+                 (filter-map (match-lambda
+                               (('guix 'scripts _ ..1) #f)
+                               (name name))
+                             (scheme-modules* source "guix"))
+                 (list *core-modules*)
+                 #:extensions dependencies
+                 #:guile-for-build guile-for-build))
+
+  (define *package-modules*
+    (scheme-node "guix-packages"
+                 `((gnu packages)
+                   ,@(scheme-modules* source "gnu/packages"))
+                 (list *core-modules* *extra-modules*)
+                 #:extra-files                    ;all the non-Scheme files
+                 (file-imports source "gnu/packages"
+                               (lambda (file stat)
+                                 (and (eq? 'regular (stat:type stat))
+                                      (not (string-suffix? ".scm" file))
+                                      (not (string-suffix? ".go" file))
+                                      (not (string-prefix? ".#" file))
+                                      (not (string-suffix? "~" file)))))
+                 #:guile-for-build guile-for-build))
+
+  (define *system-modules*
+    (scheme-node "guix-system"
+                 `((gnu system)
+                   (gnu services)
+                   ,@(scheme-modules* source "gnu/system")
+                   ,@(scheme-modules* source "gnu/services"))
+                 (list *package-modules* *extra-modules* *core-modules*)
+                 #:extra-files
+                 (file-imports source "gnu/system/examples" (const #t))
+                 #:guile-for-build
+                 guile-for-build))
+
+  (define *cli-modules*
+    (scheme-node "guix-cli"
+                 (scheme-modules* source "/guix/scripts")
+                 (list *core-modules* *extra-modules* *package-modules*
+                       *system-modules*)
+                 #:extensions dependencies
+                 #:guile-for-build guile-for-build))
+
+  (define *config*
+    (scheme-node "guix-config"
+                 '()
+                 #:extra-modules
+                 `(((guix config)
+                    => ,(make-config.scm #:libgcrypt libgcrypt
+                                         #:zlib zlib
+                                         #:gzip gzip
+                                         #:bzip2 bzip2
+                                         #:xz xz
+                                         #:package-name
+                                         %guix-package-name
+                                         #:package-version
+                                         version
+                                         #:bug-report-address
+                                         %guix-bug-report-address
+                                         #:home-page-url
+                                         %guix-home-page-url)))
+                 #:guile-for-build guile-for-build))
+
+  (directory-union (string-append "guix-" version)
+                   (append-map (lambda (node)
+                                 (list (node-source node)
+                                       (node-compiled node)))
+
+                               ;; Note: *CONFIG* comes first so that it
+                               ;; overrides the (guix config) module that
+                               ;; comes with *CORE-MODULES*.
+                               (list *config*
+                                     *cli-modules*
+                                     *system-modules*
+                                     *package-modules*
+                                     *extra-modules*
+                                     *core-modules*))
+
+                   ;; When we do (add-to-store "utils.scm"), "utils.scm" must
+                   ;; be a regular file, not a symlink.  Thus, arrange so that
+                   ;; regular files appear as regular files in the final
+                   ;; output.
+                   #:copy? #t
+                   #:quiet? #t))
+
+\f
+;;;
+;;; (guix config) generation.
+;;;
+
+(define %dependency-variables
+  ;; (guix config) variables corresponding to dependencies.
+  '(%libgcrypt %libz %xz %gzip %bzip2 %nix-instantiate))
+
+(define %persona-variables
+  ;; (guix config) variables that define Guix's persona.
+  '(%guix-package-name
+    %guix-version
+    %guix-bug-report-address
+    %guix-home-page-url))
+
+(define %config-variables
+  ;; (guix config) variables corresponding to Guix configuration (storedir,
+  ;; localstatedir, etc.)
+  (sort (filter pair?
+                (module-map (lambda (name var)
+                              (and (not (memq name %dependency-variables))
+                                   (not (memq name %persona-variables))
+                                   (cons name (variable-ref var))))
+                            (resolve-interface '(guix config))))
+        (lambda (name+value1 name+value2)
+          (string<? (symbol->string (car name+value1))
+                    (symbol->string (car name+value2))))))
+
+(define* (make-config.scm #:key libgcrypt zlib gzip xz bzip2
+                          (package-name "GNU Guix")
+                          (package-version "0")
+                          (bug-report-address "bug-guix@gnu.org")
+                          (home-page-url "https://gnu.org/s/guix"))
+
+  ;; Hack so that Geiser is not confused.
+  (define defmod 'define-module)
+
+  (scheme-file "config.scm"
+               #~(begin
+                   (#$defmod (guix config)
+                     #:export (%guix-package-name
+                               %guix-version
+                               %guix-bug-report-address
+                               %guix-home-page-url
+                               %libgcrypt
+                               %libz
+                               %gzip
+                               %bzip2
+                               %xz
+                               %nix-instantiate))
+
+                   ;; XXX: Work around <http://bugs.gnu.org/15602>.
+                   (eval-when (expand load eval)
+                     #$@(map (match-lambda
+                               ((name . value)
+                                #~(define-public #$name #$value)))
+                             %config-variables)
+
+                     (define %guix-package-name #$package-name)
+                     (define %guix-version #$package-version)
+                     (define %guix-bug-report-address #$bug-report-address)
+                     (define %guix-home-page-url #$home-page-url)
+
+                     (define %gzip
+                       #+(and gzip (file-append gzip "/bin/gzip")))
+                     (define %bzip2
+                       #+(and bzip2 (file-append bzip2 "/bin/bzip2")))
+                     (define %xz
+                       #+(and xz (file-append xz "/bin/xz")))
+
+                     (define %libgcrypt
+                       #+(and libgcrypt
+                              (file-append libgcrypt "/lib/libgcrypt")))
+                     (define %libz
+                       #+(and zlib
+                              (file-append zlib "/lib/libz")))
+
+                     (define %nix-instantiate     ;for (guix import snix)
+                       "nix-instantiate")))))
+
+
+\f
+;;;
+;;; Building.
+;;;
+
+(define (imported-files name files)
+  ;; This is a non-monadic, simplified version of 'imported-files' from (guix
+  ;; gexp).
+  (define build
+    (with-imported-modules (source-module-closure
+                            '((guix build utils)))
+      #~(begin
+          (use-modules (ice-9 match)
+                       (guix build utils))
+
+          (mkdir (ungexp output)) (chdir (ungexp output))
+          (for-each (match-lambda
+                      ((final-path store-path)
+                       (mkdir-p (dirname final-path))
+
+                       ;; Note: We need regular files to be regular files, not
+                       ;; symlinks, as this makes a difference for
+                       ;; 'add-to-store'.
+                       (copy-file store-path final-path)))
+                    '#$files))))
+
+  (computed-file name build))
+
+(define* (compiled-modules name module-tree modules
+                           #:optional
+                           (dependencies '())
+                           (dependencies-compiled '())
+                           #:key
+                           (extensions '())       ;full-blown Guile packages
+                           parallel?
+                           guile-for-build)
+  ;; This is a non-monadic, enhanced version of 'compiled-file' from (guix
+  ;; gexp).
+  (define build
+    (with-imported-modules (source-module-closure
+                            '((guix build compile)
+                              (guix build utils)))
+      #~(begin
+          (use-modules (srfi srfi-26)
+                       (ice-9 match)
+                       (ice-9 format)
+                       (ice-9 threads)
+                       (guix build compile)
+                       (guix build utils))
+
+          (define (regular? file)
+            (not (member file '("." ".."))))
+
+          (define (report-load file total completed)
+            (display #\cr)
+            (format #t
+                    "loading...\t~5,1f% of ~d files" ;FIXME: i18n
+                    (* 100. (/ completed total)) total)
+            (force-output))
+
+          (define (report-compilation file total completed)
+            (display #\cr)
+            (format #t "compiling...\t~5,1f% of ~d files" ;FIXME: i18n
+                    (* 100. (/ completed total)) total)
+            (force-output))
+
+          (define (process-directory directory output)
+            (let ((files  (find-files directory "\\.scm$"))
+                  (prefix (+ 1 (string-length directory))))
+              ;; Hide compilation warnings.
+              (parameterize ((current-warning-port (%make-void-port "w")))
+                (compile-files directory #$output
+                               (map (cut string-drop <> prefix) files)
+                               #:workers (parallel-job-count)
+                               #:report-load report-load
+                               #:report-compilation report-compilation))))
+
+          (setvbuf (current-output-port) _IONBF)
+          (setvbuf (current-error-port) _IONBF)
+
+          (set! %load-path (cons #+module-tree %load-path))
+          (set! %load-path
+            (append '#+dependencies
+                    (map (lambda (extension)
+                           (string-append extension "/share/guile/site/"
+                                          (effective-version)))
+                         '#+extensions)
+                    %load-path))
+
+          (set! %load-compiled-path
+            (append '#+dependencies-compiled
+                    (map (lambda (extension)
+                           (string-append extension "/lib/guile/"
+                                          (effective-version)
+                                          "/site-ccache"))
+                         '#+extensions)
+                    %load-compiled-path))
+
+          ;; Load the compiler modules upfront.
+          (compile #f)
+
+          (mkdir #$output)
+          (chdir #+module-tree)
+          (process-directory "." #$output))))
+
+  (computed-file name build
+                 #:guile guile-for-build
+                 #:options
+                 `(#:local-build? #f              ;allow substitutes
+
+                   ;; Don't annoy people about _IONBF deprecation.
+                   #:env-vars (("GUILE_WARN_DEPRECATED" . "no")))))
+
+\f
+;;;
+;;; Live patching.
+;;;
+
+(define (recursive-submodules module)
+  "Return the list of submodules of MODULE."
+  (let loop ((module module)
+             (result '()))
+    (let ((submodules (hash-map->list (lambda (name module)
+                                        module)
+                                      (module-submodules module))))
+      (fold loop (append submodules result) submodules))))
+
+(define (remove-submodule! module names)
+  (let loop ((module module)
+             (names names))
+    (match names
+      (() #t)
+      ((head tail ...)
+       (match (nested-ref-module module tail)
+         (#f #t)
+         ((? module? submodule)
+          (hashq-remove! (module-submodules module) head)
+          (loop submodule tail)))))))
+
+(define (unload-module-tree! module)
+  (define (strip-prefix prefix lst)
+    (let loop ((prefix prefix)
+               (lst lst))
+      (match prefix
+        (()
+         lst)
+        ((_ prefix ...)
+         (match lst
+           ((_ lst ...)
+            (loop prefix lst)))))))
+
+  (let ((submodules (hash-map->list (lambda (name module)
+                                      module)
+                                    (module-submodules module))))
+    (let loop ((root module)
+               (submodules submodules))
+      (match submodules
+        (()
+         #t)
+        ((head tail ...)
+         (unload-module-tree! head)
+         (remove-submodule! root
+                            (strip-prefix (module-name root)
+                                          (module-name head)))
+
+         (match (module-name head)
+           ((parents ... leaf)
+            ;; Remove MODULE from the AUTOLOADS-DONE list.  Note: We don't use
+            ;; 'module-filename' because it could be an absolute file name.
+            (set-autoloaded! (string-join (map symbol->string parents)
+                                          "/" 'suffix)
+                             (symbol->string leaf) #f)))
+         (loop root tail))))))
+
+(define* (reload-guix #:optional (log-port (current-error-port)))
+  "Reload all the Guix and GNU modules currently loaded."
+
+  ;; Caveat! Until we've reloaded all the guix and gnu modules, we cannot rely
+  ;; on bindings coming from guix/gnu modules, except for (guix self), because
+  ;; the other modules might be outdated.
+
+  (let* ((guix (resolve-module '(guix) #f #f #:ensure #f))
+         (gnu  (resolve-module '(gnu) #f #f #:ensure #f))
+         (guix-submodules (recursive-submodules guix))
+         (gnu-submodules  (recursive-submodules gnu)))
+    (define (reload module)
+      ;; Reload MODULE, unless it's already been reloaded indirectly in the
+      ;; meantime.
+      (unless (resolve-module (module-name module) #f #f #:ensure #f)
+        (match (module-filename module)
+          (#f #f)
+          ((? string? file)
+           (primitive-load (search-path %load-path file))))))
+
+    ;; First, we need to nuke all the (guix) and (gnu) submodules so we don't
+    ;; end up with a mixture of old and new modules when we reload (which
+    ;; wouldn't work, because we'd have two different <package> record types,
+    ;; for instance.)
+    (format log-port (G_ "Unloading current modules...~%"))
+    (unload-module-tree! gnu)
+    (unload-module-tree! guix)
+
+    ;; When reloading, we compile the core modules and interpret the rest.
+    ;; This appears to be the most efficient approach.
+    (format log-port (G_ "Loading new modules...~%"))
+    (set! %auto-compilation-options %lightweight-optimizations)
+    (reload (resolve-module '(guix packages)))
+
+    (set! %load-should-auto-compile #f)
+    (for-each reload (append guix-submodules (list guix)))
+    (for-each reload (append gnu-submodules (list gnu)))
+    (format log-port (G_ "New modules successfully loaded.~%"))))
+
+\f
+;;;
+;;; Building.
+;;;
+
+(define (guile-for-build version)
+  "Return a derivation for Guile 2.0 or 2.2, whichever matches the currently
+running Guile."
+  (define canonical-package                       ;soft reference
+    (module-ref (resolve-interface '(gnu packages base))
+                'canonical-package))
+
+  (match version
+    ("2.2"
+     ;; Use the same 2.2 variant for everyone so that people can get
+     ;; substitutes, and also to avoid ABI incompatibilities (see
+     ;; <https://bugs.gnu.org/29570>.)
+     (canonical-package (module-ref (resolve-interface '(gnu packages guile))
+                                    'guile-2.2/fixed)))
+    ("2.0"
+     (canonical-package (specification->package "guile@2.0")))))
+
+(define* (guix-derivation source version
+                          #:optional (guile-version (effective-version)))
+  "Return, as a monadic value, the derivation to build the Guix from SOURCE
+for GUILE-VERSION.  Use VERSION as the version string."
+  (define max-version-length 9)
+
+  (define (shorten version)
+    ;; TODO: VERSION is a commit id, but we'd rather use something like what
+    ;; 'git describe' provides.
+    (if (> (string-length version) max-version-length)
+        (string-take version max-version-length)
+        version))
+
+  (mbegin %store-monad
+    (set-guile-for-build (guile-for-build guile-version))
+    (lower-object (compiled-guix source
+                                 #:version (shorten version)
+                                 #:guile-version guile-version))))
diff --git a/po/guix/POTFILES.in b/po/guix/POTFILES.in
index 6510b99e8..01ba5638b 100644
--- a/po/guix/POTFILES.in
+++ b/po/guix/POTFILES.in
@@ -33,6 +33,7 @@ guix/scripts/weather.scm
 guix/gnu-maintenance.scm
 guix/scripts/container.scm
 guix/scripts/container/exec.scm
+guix/self.scm
 guix/upstream.scm
 guix/ui.scm
 guix/http-client.scm
-- 
2.15.1

bug#27284: [PATCH 3/4] Add (guix self) and use it when pulling.

[Ludovic Courtès]

Hello,

As Marius reported on IRC, pulling this branch from current master fails
along these lines:

--8<---------------cut here---------------start------------->8---
$ guix pull --url=$PWD --branch=wip-pull-reload
Updating from Git repository at '/home/ludo/src/guix'...
Building from Git commit 0f34a0f4bc3f7c5483c0c9b09b1e4bf00b85271d...
Unloading current modules...
Loading new modules...
New modules successfully loaded.
Backtrace:
In ice-9/boot-9.scm:
    837:9 19 (catch _ _ #<procedure 7f2a395c6270 at guix/ui.scm:632:6 (key proc format-string format-args . res…> …)
    837:9 18 (catch _ _ #<procedure 7f2a361d3aa0 at guix/scripts/pull.scm:244:4 (key err)> _)
In guix/scripts/pull.scm:
   280:17 17 (_)
In guix/store.scm:
  1443:24 16 (run-with-store _ _ #:guile-for-build _ #:system _ #:target _)
In guix/scripts/pull.scm:
    178:2 15 (_ _)
In unknown file:
          14 (_ #<procedure d86ee60 at ice-9/eval.scm:330:13 ()> #<procedure d86ee40 at ice-9/eval.scm:336:13 (…> …)
In guix/packages.scm:
  1207:17 13 (_ #<build-daemon 256.97 2bd6f50>)
   834:14 12 (cache! #<weak-table 0/113> #<package guile@2.2.2 /gnu/store/3pmjcni27k2kx103l2v56ivcpqg95nvb-guix…> …)
In unknown file:
          11 (_ #<procedure thunk ()> #<procedure list _> #<undefined>)
In guix/packages.scm:
  1154:22 10 (thunk)
  1087:25  9 (bag->derivation #<build-daemon 256.97 2bd6f50> #<<bag> name: "guile-2.2.2" system: "x86_64-linux"…> …)
In srfi/srfi-1.scm:
   592:17  8 (map1 (("source" #<origin "mirror://gnu/guile/guile-2.2.2.tar.xz" dsi2iymx7mnn5osp2yvcl36pgyq4…>) …))
In ice-9/boot-9.scm:
    837:9  7 (catch srfi-34 #<procedure 1b033780 at guix/packages.scm:898:5 ()> #<procedure 1ae27de0 at guix/pa…> …)
In guix/packages.scm:
   903:18  6 (_)
In guix/store.scm:
  1443:24  5 (run-with-store _ _ #:guile-for-build _ #:system _ #:target _)
In guix/packages.scm:
   1255:5  4 (_ _)
In unknown file:
           3 (_ #<procedure 1ae284e0 at ice-9/eval.scm:330:13 ()> #<procedure 1ae284c0 at ice-9/eval.scm:336:13…> …)
In ice-9/eval.scm:
    159:9  2 (_ #(#(#<directory (guix download) 12d05dc0> #<weak-table 0/31>) #<build-daemon 256.97 2bd6f50>))
   293:34  1 (_ #(#(#(#<directory (guix download) 12d05dc0> #<weak-table 0/31>) #<build-daemon 256.97 2bd6f…>) #))
In guix/store.scm:
  1164:17  0 (_ #<build-daemon 256.97 2bd6f50>)

guix/store.scm:1164:17: guix/store.scm:1164:17: In procedure nix-server-major-version: Wrong type argument: #<build-daemon 256.97 2bd6f50>

Some deprecated features have been used.  Set the environment
variable GUILE_WARN_DEPRECATED to "detailed" and rerun the
program to get more information.  Set it to "no" to suppress
this message.
--8<---------------cut here---------------end--------------->8---

This comes from the fact that current master doesn’t protect against
module reloads like this branch does:

> +(define (build-and-install mdrv)
> +  "Bind MDRV, a monadic value for a derivation, build it, and finally install
> +it as the latest Guix."
> +  (define do-it
> +    ;; Weirdness follows!  Before we were called, the Guix modules have
> +    ;; probably been reloaded, leading to a "parallel universe" with disjoint
> +    ;; record types.  However, procedures in this file have already cached the
> +    ;; module relative to which they lookup global bindings (see
> +    ;; 'toplevel-box' documentation), so they're stuck in the old world.  To
> +    ;; work around that, evaluate our procedure in the context of the "new"
> +    ;; (guix scripts pull) module--which has access to the new <derivation>
> +    ;; record, and so on.
> +    (eval '(lambda (mdrv cont)
> +             ;; Reopen a connection to the daemon so that we have a record
> +             ;; with the new type.
> +             (with-store store
> +               (run-with-store store
> +                 (mlet %store-monad ((drv mdrv))
> +                   (mbegin %store-monad
> +                     (what-to-build (list drv))
> +                     (built-derivations (list drv))
> +                     (return (cont (derivation->output-path drv))))))))
> +          (resolve-module '(guix scripts pull)))) ;the new module
> +
> +  (do-it mdrv
> +         (lambda (result)
> +           (install-latest result (config-directory)))))

I’m thinking that perhaps a middle ground would be to skip the reload
thing when we detect that the calling ‘guix’ is not prepared for module
reloads.

Ludo’.

bug#27284: ‘guix pull’ builds using multiple derivations

[Ludovic Courtès]

Hello Guix!

News from the front!  I’ve pushed a new ‘wip-pull-multiple-derivations’
branch that keeps the same approach as previously proposed (building
Guix using multiple derivation, one for each group of modules: core,
CLI, packages, etc.), but does not attempt to reload modules in the
running Guile process, which proved to be too tricky.

You can already test with:

  guix pull --branch=wip-pull-multiple-derivations

Feedback welcome!

This patch set actually addresses a bootstrapping issue: you need Guix
to build Guix.  So far ‘guix pull’ would sidestep the bootstrapping
issue by building Guix with whatever is available in the currently
deployed Guix; for instance, it would use the ‘guile’, ‘guile-json’,
etc. packages from the current Guix.  The problem of that approach was
that it’s stateful: the result depends on what you currently have.
Sometimes what you have is too old, or lacks some package definition,
and you can’t really go forward.

The code in ‘wip-pull-multiple-derivations’ works like this:

  1. Assume we have Guile and Guix already installed, but not
     necessarily the latest versions thereof.

  2. Using the (guix …) modules that we have, build a program—a
     “trampoline”—that will use the modules of the target Guix (the
     commit we want to pull) to compute the derivation of that Guix.

  3. Run that trampoline, which returns /gnu/store/…-guix.drv.  The
     result should be the same regardless of the initial Guix because
     the trampoline uses exclusively modules from the target Guix.

  4. ‘guix pull’ builds that derivation (actually the branch does not
     modify (guix scripts pull) at all; everything is in
     build-aux/build-self.scm.)

In step #4, we should be able to get substitutes for at least some of
the derivations.

To build the trampoline in step #2, we first need to build a bunch of
modules from the target Guix.  Hopefully you don’t have to rebuild them
at each pull, but it can take a minute or so, and you may not have
substitutes for that (because this part is stateful.)

In step #3, the trampoline has compiled code for the core modules, but
it still has to interpret (gnu packages guile) and related modules,
because these are not compiled.  The program runs in ~40s on my laptop.
To make it faster, we could reduce the closure of (gnu packages guile)¹,
but that can be tricky.  Or we could optimize Guile itself; I’m sure the
compiler and/or interpreter could do better.

In that branch, you can also run:

  make as-derivation

and it will build Guix from $srcdir in the way described above.

To summarize, performance is not great, but hopefully it’ll be slightly
better than what we have now, especially with substitutes.
Correctness/reproducibility are greatly improved.

Ludo’.

¹ The closure of (gnu packages guile) contains python, haskell-check,
  music, and many other surprising things:
  <http://web.fdn.fr/~lcourtes/tmp/gnu-packages-guile.html>.

bug#27284: ‘guix pull’ builds using multiple derivations

[Ludovic Courtès]

ludo@gnu.org (Ludovic Courtès) skribis:

> ¹ The closure of (gnu packages guile) contains python, haskell-check,
>   music, and many other surprising things:
>   <http://web.fdn.fr/~lcourtes/tmp/gnu-packages-guile.html>.

I committed the bits to generate this graph here:

  https://git.savannah.gnu.org/cgit/guix.git/commit/?id=b06a70e05dc6252a3ecb28db5898de7ebc110973

Ludo’.

bug#27284: ‘guix pull’ builds using multiple derivations

[Nils Gillmann]

Awesome news!

Ludovic Courtès transcribed 3.1K bytes:
> Hello Guix!
> 
> News from the front!  I’ve pushed a new ‘wip-pull-multiple-derivations’
> branch that keeps the same approach as previously proposed (building
> Guix using multiple derivation, one for each group of modules: core,
> CLI, packages, etc.), but does not attempt to reload modules in the
> running Guile process, which proved to be too tricky.
> 
> You can already test with:
> 
>   guix pull --branch=wip-pull-multiple-derivations
>
> Feedback welcome!

I think you forgot to push it or you are on a very slow connection :)
 
> This patch set actually addresses a bootstrapping issue: you need Guix
> to build Guix.  So far ‘guix pull’ would sidestep the bootstrapping
> issue by building Guix with whatever is available in the currently
> deployed Guix; for instance, it would use the ‘guile’, ‘guile-json’,
> etc. packages from the current Guix.  The problem of that approach was
> that it’s stateful: the result depends on what you currently have.
> Sometimes what you have is too old, or lacks some package definition,
> and you can’t really go forward.
> 
> The code in ‘wip-pull-multiple-derivations’ works like this:
> 
>   1. Assume we have Guile and Guix already installed, but not
>      necessarily the latest versions thereof.
> 
>   2. Using the (guix …) modules that we have, build a program—a
>      “trampoline”—that will use the modules of the target Guix (the
>      commit we want to pull) to compute the derivation of that Guix.
> 
>   3. Run that trampoline, which returns /gnu/store/…-guix.drv.  The
>      result should be the same regardless of the initial Guix because
>      the trampoline uses exclusively modules from the target Guix.
> 
>   4. ‘guix pull’ builds that derivation (actually the branch does not
>      modify (guix scripts pull) at all; everything is in
>      build-aux/build-self.scm.)
> 
> In step #4, we should be able to get substitutes for at least some of
> the derivations.
> 
> To build the trampoline in step #2, we first need to build a bunch of
> modules from the target Guix.  Hopefully you don’t have to rebuild them
> at each pull, but it can take a minute or so, and you may not have
> substitutes for that (because this part is stateful.)
> 
> In step #3, the trampoline has compiled code for the core modules, but
> it still has to interpret (gnu packages guile) and related modules,
> because these are not compiled.  The program runs in ~40s on my laptop.
> To make it faster, we could reduce the closure of (gnu packages guile)¹,
> but that can be tricky.  Or we could optimize Guile itself; I’m sure the
> compiler and/or interpreter could do better.
> 
> In that branch, you can also run:
> 
>   make as-derivation
> 
> and it will build Guix from $srcdir in the way described above.
> 
> To summarize, performance is not great, but hopefully it’ll be slightly
> better than what we have now, especially with substitutes.
> Correctness/reproducibility are greatly improved.
> 
> Ludo’.
> 
> ¹ The closure of (gnu packages guile) contains python, haskell-check,
>   music, and many other surprising things:
>   <http://web.fdn.fr/~lcourtes/tmp/gnu-packages-guile.html>.
> 
> 
> 

bug#27284: ‘guix pull’ builds using multiple derivations

[Ludovic Courtès]

Nils Gillmann <ng0@n0.is> skribis:

>> You can already test with:
>> 
>>   guix pull --branch=wip-pull-multiple-derivations
>>
>> Feedback welcome!
>
> I think you forgot to push it or you are on a very slow connection :)

Oops, I had deleted the branch to push a new version, except that
pushing didn’t happen.  Should be okay now, thanks for the heads-up!

Ludo’.

bug#27284: ‘guix pull’ builds using multiple derivations

[Ludovic Courtès]

Hello!

ludo@gnu.org (Ludovic Courtès) skribis:

> The code in ‘wip-pull-multiple-derivations’ works like this:
>
>   1. Assume we have Guile and Guix already installed, but not
>      necessarily the latest versions thereof.
>
>   2. Using the (guix …) modules that we have, build a program—a
>      “trampoline”—that will use the modules of the target Guix (the
>      commit we want to pull) to compute the derivation of that Guix.
>
>   3. Run that trampoline, which returns /gnu/store/…-guix.drv.  The
>      result should be the same regardless of the initial Guix because
>      the trampoline uses exclusively modules from the target Guix.
>
>   4. ‘guix pull’ builds that derivation (actually the branch does not
>      modify (guix scripts pull) at all; everything is in
>      build-aux/build-self.scm.)

After quite a bit of tweaking I’ve finally pushed this patch series to
master.  So if you run ‘guix pull’ now, you’ll get the new
multiple-derivation build.

There’s still room for improvement, in particular:

  • The initial ‘compute-guix-derivation’ program depends on quite a few
    modules, so the first time you ‘guix pull’ you build a
    module-import-compiled.drv that takes a while and does not produce
    any output (I’ve made it verbose in core-updates commit
    d32922759bfeffa03ee189158ea00b1a0ddbe8c6 though).  Perhaps we could
    avoid that by using a double trampoline, which would allow us to get
    substitutes for all the non-trivial bits.

  • The “guix-packages” derivation is coarse-grain.  We could perhaps
    automatically split it in clusters of modules by analyzing the (gnu
    packages …) module graph.

  • The compiler (and interpreter) in Guile 2.2 is still not as fast as
    we’d like.  I think it should be possible to make it as fast as in
    2.0, at least when optimizations are turned off.  Andy made
    significant improvements in 2.2.3, but hopefully we can still
    improve on that.  The thread at
    <https://lists.gnu.org/archive/html/guile-devel/2017-10/msg00035.html>
    contains some info.

Normally berlin.guixsd.org will start providing substitutes for all this
but note that, as things are, it’ll compute substitutes for Guile 2.2.2
or 2.2.3, and that’s it.  hydra.gnu.org currently doesn’t, for obscure
reasons (‘hydra-eval-guile-jobs’ systematically adds “.” to the load
path, which is not what we want for build-aux/hydra/guix-modular.scm.)

This works unlocks some of the stuff in <https://bugs.gnu.org/22629>, so
I’ll try to focus on that now.

Feedback and bug reports welcome!

Ludo’.

bug#27284: ‘guix pull’ builds using multiple derivations

[Ricardo Wurmus]

Hi Ludo,

> After quite a bit of tweaking I’ve finally pushed this patch series to
> master.  So if you run ‘guix pull’ now, you’ll get the new
> multiple-derivation build.

This is really great!  I’m very happy that this is finally in the master
branch.  Thank you!

As discussed on IRC, there is a problem when running “guix pull” with an
older version of Guix that is still using Guile 2.0, which seems to be
adequately fixed by your patch:

--8<---------------cut here---------------start------------->8---
diff --git a/guix/self.scm b/guix/self.scm
index c9e4a4250..c19b0f6fb 100644
--- a/guix/self.scm
+++ b/guix/self.scm
@@ -88,7 +88,11 @@ GUILE-VERSION (\"2.0\" or \"2.2\"), or #f if none of the packages matches."
       ("bzip2"      (ref '(gnu packages compression) 'bzip2))
       ("xz"         (ref '(gnu packages compression) 'xz))
       ("guix"       (ref '(gnu packages package-management)
-                         'guix-register)))))
+                         'guix-register))
+      ("guile2.0-json" (ref '(gnu packages guile) 'guile2.0-json))
+      ("guile2.0-ssh"  (ref '(gnu packages ssh) 'guile2.0-ssh))
+      ("guile2.0-git"  (ref '(gnu packages guile) 'guile2.0-git))
+      (_               #f))))                     ;no such package

 \f
;;;
--8<---------------cut here---------------end--------------->8---

A remaining problem for me was that “(guix docker)” could not be built
because “(json)” was supposedly not available.

None of this seems to be a problem when using Guile 2.2.

>   • The “guix-packages” derivation is coarse-grain.  We could perhaps
>     automatically split it in clusters of modules by analyzing the (gnu
>     packages …) module graph.

I think we should do this anyway and try to split up large modules and
reduce inter-module dependencies.

> This works unlocks some of the stuff in <https://bugs.gnu.org/22629>, so
> I’ll try to focus on that now.

Exciting!

--
Ricardo

bug#27284: ‘guix pull’ broken on Guile 2.0

[Ludovic Courtès]

Hello,

Ricardo Wurmus <rekado@elephly.net> skribis:

> This is really great!  I’m very happy that this is finally in the master
> branch.  Thank you!

It’s still not as good as we’d like obviously, but if you’re lucky, you
get pull in 5 minutes now.

> As discussed on IRC, there is a problem when running “guix pull” with an
> older version of Guix that is still using Guile 2.0, which seems to be
> adequately fixed by your patch:

I’ve pushed this as commit e69dd8443ad2b8620c3a3db874dc50e06b0d43d0.

Unfortunately I’m now stuck with this:

--8<---------------cut here---------------start------------->8---
building path(s) `/gnu/store/81dmaf8c1wfzn3lifxsfxn827bgdz8y6-guix-extra'
loading...	  0.0% of 106 filesBacktrace:
In unknown file:
   ?: 19 [primitive-load-path "guix/packages" ...]
In guix/packages.scm:
  23: 18 [#<procedure 1067ac0 ()>]
In ice-9/boot-9.scm:
2987: 17 [define-module* (guix packages) #:filename ...]
2962: 16 [resolve-imports (((guix utils)) ((guix records)) ((guix store)) ...)]
2900: 15 [resolve-interface (guix utils) #:select ...]
2825: 14 [#<procedure 84b000 at ice-9/boot-9.scm:2813:4 (name #:optional autoload version #:key ensure)> # ...]
3101: 13 [try-module-autoload (guix utils) #f]
2412: 12 [save-module-excursion #<procedure 106ffc0 at ice-9/boot-9.scm:3102:17 ()>]
3121: 11 [#<procedure 106ffc0 at ice-9/boot-9.scm:3102:17 ()>]
In unknown file:
   ?: 10 [primitive-load-path "guix/utils" ...]
In guix/utils.scm:
  26: 9 [#<procedure 1080740 ()>]
In ice-9/boot-9.scm:
2987: 8 [define-module* (guix utils) #:filename ...]
2962: 7 [resolve-imports (((guix config)) ((srfi srfi-1)) ((srfi srfi-9)) ...)]
2900: 6 [resolve-interface (guix config) #:select ...]
2825: 5 [#<procedure 84b000 at ice-9/boot-9.scm:2813:4 (name #:optional autoload version #:key ensure)> # ...]
3101: 4 [try-module-autoload (guix config) #f]
2412: 3 [save-module-excursion #<procedure 1086900 at ice-9/boot-9.scm:3102:17 ()>]
3121: 2 [#<procedure 1086900 at ice-9/boot-9.scm:3102:17 ()>]
In unknown file:
   ?: 1 [primitive-load-path "guix/config" ...]
In guix/config.scm:
   1: 0 [#<procedure 10856a0 ()>]

guix/config.scm:1:726: In procedure #<procedure 10856a0 ()>:
guix/config.scm:1:726: In procedure module-lookup: Unbound variable: getenv
builder for `/gnu/store/0g3zg87w1dlqv368az3q3a1p57nd4f91-guix-extra.drv' failed with exit code 1
cannot build derivation `/gnu/store/gk74rl0k8hj79ckcf6ncg8ikgv8yyawq-guix-20180410.21.drv': 1 dependencies couldn't be built
--8<---------------cut here---------------end--------------->8---

Indeed, on closer inspection, (guix config) is miscompiled and looks for
‘getenv’ and ‘string-append’ in a nonexistent anonymous module (the
#{ g12345}# thing below) instead of (guile):

--8<---------------cut here---------------start------------->8---
$ guix environment --ad-hoc guile@2.0 -- guild disassemble /gnu/store/vhb0pps5lnsm4k2f1fg3050x85v8n7xz-guix-core/guix/config.go

[...]

 975    (load-string "bug-guix@gnu.org");; "bug-guix@gnu.org"
 995    (vector-set)                    
 996    (dup)                           
 997    (make-int8 60)                  ;; 60
 999    (load-string "https://gnu.org/s/guix");; "https://gnu.org/s/guix"
1025    (vector-set)                    
1026    (dup)                           
1027    (make-int8 61)                  ;; 61
1029    (load-symbol " g59791")         ;; #{ g59791}#
1040    (vector-set)                    
1041    (dup)                           
1042    (make-int8 62)                  ;; 62
1044    (object-ref 61)                 
1046    (list 0 1)                      ;; 1 element
1049    (vector-set)                    
1050    (dup)                           
1051    (make-int8 63)                  ;; 63
1053    (load-symbol "getenv")          ;; getenv
1063    (vector-set)                    
1064    (dup)                           
1065    (make-int8 64)                  ;; 64
1067    (object-ref 62)                 
1069    (object-ref 63)                 
1071    (make-false)                    
1072    (list 0 3)                      ;; 3 elements
1075    (vector-set)                    
1076    (dup)                           
1077    (make-int8 65)                  ;; 65
1079    (object-ref 62)                 
1081    (object-ref 63)                 
1083    (make-false)                    
1084    (list 0 3)                      ;; 3 elements
1087    (vector-set)                    
1088    (dup)                           
1089    (make-int8 66)                  ;; 66
1091    (load-string "GUIX_REGISTER")   ;; "GUIX_REGISTER"
1108    (vector-set)                    
1109    (dup)                           
1110    (make-int8 67)                  ;; 67
1112    (load-symbol "string-append")   ;; string-append
1129    (vector-set)                    

[...]

--8<---------------cut here---------------end--------------->8---

Ludo’.

bug#31117: bug#27284: ‘guix pull’ broken on Guile 2.0

[Ludovic Courtès]

Hello,

ludo@gnu.org (Ludovic Courtès) skribis:

> Indeed, on closer inspection, (guix config) is miscompiled and looks for
> ‘getenv’ and ‘string-append’ in a nonexistent anonymous module (the
> #{ g12345}# thing below) instead of (guile):

It turns out the ‘define-module’ form has to be spliced in 2.0.

Fixed in commit eb72cdf087fe51d85e0c1514ec8e669047b5d6e1.

Thanks Ricardo & Alex,
Ludo’.

bug#27284: ‘guix pull’ broken on Guile 2.0

[Ricardo Wurmus]

Ludovic Courtès <ludo@gnu.org> writes:

>> Indeed, on closer inspection, (guix config) is miscompiled and looks for
>> ‘getenv’ and ‘string-append’ in a nonexistent anonymous module (the
>> #{g12345}# thing below) instead of (guile):
>
> It turns out the ‘define-module’ form has to be spliced in 2.0.
>
> Fixed in commit eb72cdf087fe51d85e0c1514ec8e669047b5d6e1.

Thank you.  I tested this and it works fine.

-- 
Ricardo

bug#27284: [PATCH 4/4] pull: Reload modules before doing anything else.

[Ludovic Courtès]

* build-aux/build-self.scm (build): Uncomment call to 'reload-guix'.
Set '%fresh-auto-compile'
---
 build-aux/build-self.scm | 22 +++++++++++++---------
 1 file changed, 13 insertions(+), 9 deletions(-)

diff --git a/build-aux/build-self.scm b/build-aux/build-self.scm
index d9d926367..04185e0a5 100644
--- a/build-aux/build-self.scm
+++ b/build-aux/build-self.scm
@@ -21,6 +21,7 @@
   #:use-module (srfi srfi-19)
   #:use-module (srfi srfi-26)
   #:use-module (ice-9 match)
+  #:use-module (system base compile)
   #:export (build))
 
 ;;; Commentary:
@@ -82,23 +83,26 @@ Guix."
                 #:rest rest)
   "Return a derivation that unpacks SOURCE into STORE and compiles Scheme
 files."
+
+  ;; Pre-load the compiler modules so we don't end up rebuilding them due to
+  ;; %FRESH-AUTO-COMPILE.
+  (compile #t)
+
   ;; Start by jumping into the target Guix so that we have access to the
   ;; latest packages and APIs.
   ;;
   ;; Our checkout in the store has mtime set to the epoch, and thus .go
   ;; files look newer, even though they may not correspond.
-  (parameterize* ((%load-should-auto-compile #f)
-                  (%fresh-auto-compile #f)
+  (parameterize* ((%load-should-auto-compile #t)
+                  (%fresh-auto-compile #t)
 
                   ;; Work around <https://bugs.gnu.org/29226>.
                   (%load-compiled-path (pure-load-compiled-path)))
-    ;; FIXME: This is currently too expensive notably because it involves
-    ;; compiling a number of the big package files such as perl.scm, which
-    ;; takes lots of time and memory as of Guile 2.2.2.
-    ;;
-    ;; (let ((reload-guix (module-ref (resolve-interface '(guix self))
-    ;;                                'reload-guix)))
-    ;;   (reload-guix))           ;cross fingers!
+    ;; Hide auto-compilation messages.
+    (parameterize ((current-warning-port (%make-void-port "w")))
+      (let ((reload-guix (module-ref (resolve-interface '(guix self))
+                                     'reload-guix)))
+        (reload-guix)))                           ;cross fingers!
 
     (let ((guix-derivation (module-ref (resolve-interface '(guix self))
                                        'guix-derivation)))
-- 
2.15.1

bug#27284: Memory leak in 'guix pull' or 'make' in guix source

[Ludovic Courtès]

Heya,

ludo@gnu.org (Ludovic Courtès) skribis:

> So, we have two problems: compilation time, and memory consumption.  I
> *think* I’ve identified one of the major causes for both in Guile,
> though it’s too early to say exactly how much this will impact resource
> consumption for a full Guix compilation.  See
> <https://lists.gnu.org/archive/html/guile-devel/2017-09/msg00031.html>
> for details.

The latest news is that (1) the weak-table fix in Guile¹ helps in
general but does not significantly reduce memory consumption when
compiling gnu/packages/python.scm (which requires ~1.5G on x86_64), and
(2) we’ve identified the compiler’s slot allocator as a major cause of
memory consumption and CPU cost when compiling large files like
python.scm:

  https://lists.gnu.org/archive/html/guile-devel/2017-10/msg00035.html

Andy may work on a simpler slot allocator for -O0 and/or large functions
that doesn’t exhibit this pathological behavior.

In the meantime, our best workaround to reduce memory consumption is… to
split large files into smaller ones.  Per M-x guix-locations, the
candidates are:

  gnu/packages/python.scm                                   986
  gnu/packages/perl.scm                                     401
  gnu/packages/haskell.scm                                  348
  gnu/packages/bioinformatics.scm                           274
  gnu/packages/statistics.scm                               222
  gnu/packages/emacs.scm                                    213
  gnu/packages/xorg.scm                                     196
  gnu/packages/web.scm                                      188
  gnu/packages/gnome.scm                                    162
  gnu/packages/ruby.scm                                     159
  gnu/packages/java.scm                                     149
  gnu/packages/ocaml.scm                                    126
  gnu/packages/tex.scm                                      107
  gnu/packages/linux.scm                                    106

I think we could focus on the first two or three files.  FTR, compiling
bioinformatics.scm peaks at ~500 MiB resident on x86_64.

Ricardo, WDYT?

If we do this, do we split arbitrarily?  Like the second half of
python.scm goes to python-cont.scm (provided there are no cross
top-level references)?  Or do you have a better idea?

Ludo’.

¹ https://lists.gnu.org/archive/html/guile-devel/2017-10/msg00051.html

bug#27284: Memory leak in 'guix pull' or 'make' in guix source

[Ricardo Wurmus]

Hi Ludo,

> In the meantime, our best workaround to reduce memory consumption is… to
> split large files into smaller ones.  Per M-x guix-locations, the
> candidates are:
>
>   gnu/packages/python.scm                                   986
>   gnu/packages/perl.scm                                     401
>   gnu/packages/haskell.scm                                  348

[…]

> I think we could focus on the first two or three files.  FTR, compiling
> bioinformatics.scm peaks at ~500 MiB resident on x86_64.
>
> Ricardo, WDYT?

I was hoping we could avoid this, but whatever: let’s do this :)

> If we do this, do we split arbitrarily?  Like the second half of
> python.scm goes to python-cont.scm (provided there are no cross
> top-level references)?  Or do you have a better idea?

Ultimately, I’d like to move packages to locations where they could
permanently live, but that would probably take longer.

Would it make sense to move all the python2-* packages to a new module?
This would make the split rather simple and users wouldn’t have to
remember which of their packages ended up in which half of the modules.
It also means that we probably won’t have to mess with the copyright
headers.

For perl.scm I have no good ideas.  Let’s split it up at an arbitrary
point.

For haskell.scm I’d begin by moving the following packages away:

- check.scm: ghc-tasty*, ghc-quickcheck*, ghc-test*, ghc-hunit*, hspec*,
  ghc-hspec*, …

- web.scm: ghc-tagsoup, ghc-cookie, ghc-http*, ghc-wai*, ghc-json,
  ghc-warp*, ghc-multipart, ghc-aeson*

- crypto.scm: ghc-tf-random, ghc-digest, ghc-cryptonite, ghc-x509*,
  ghc-asn1*, ghc-pem, ghc-cryptohash*, ghc-entropy, ghc-crypto-api*,
  ghc-puremd5

- tls.scm: ghc-tls

Maybe that’s enough already.

Does that seem like a good idea?  I could prepare patches for splitting
up haskell.scm.

--
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net

bug#27284: Memory leak in 'guix pull' or 'make' in guix source

[Ludovic Courtès]

Hi!

Ricardo Wurmus <rekado@elephly.net> skribis:

>> In the meantime, our best workaround to reduce memory consumption is… to
>> split large files into smaller ones.  Per M-x guix-locations, the
>> candidates are:
>>
>>   gnu/packages/python.scm                                   986
>>   gnu/packages/perl.scm                                     401
>>   gnu/packages/haskell.scm                                  348
>
> […]
>
>> I think we could focus on the first two or three files.  FTR, compiling
>> bioinformatics.scm peaks at ~500 MiB resident on x86_64.
>>
>> Ricardo, WDYT?
>
> I was hoping we could avoid this, but whatever: let’s do this :)

Yeah, me too.  The problem we have is that Guix is hardly releasable in
its current state because on systems with 1 GiB of memory you can’t
upgrade, and I think that’s unacceptable.

So what are the options?  If we get a bug-fix for Guile’s compiler
today, does it help?  If we graft it then we can deliver it without
having to wait for a Guile release, which helps a bit?

I think it’s all about time: we could wait (and hack!) some more, and
solve the root problem.  This is the best long-term course of action,
but at the same time it delays the Guix release.

>> If we do this, do we split arbitrarily?  Like the second half of
>> python.scm goes to python-cont.scm (provided there are no cross
>> top-level references)?  Or do you have a better idea?
>
> Ultimately, I’d like to move packages to locations where they could
> permanently live, but that would probably take longer.
>
> Would it make sense to move all the python2-* packages to a new module?

I’m not sure we can do this, because that may lead to top-level
references across these two modules, which is not OK.

> This would make the split rather simple and users wouldn’t have to
> remember which of their packages ended up in which half of the modules.
> It also means that we probably won’t have to mess with the copyright
> headers.
>
> For perl.scm I have no good ideas.  Let’s split it up at an arbitrary
> point.
>
> For haskell.scm I’d begin by moving the following packages away:
>
> - check.scm: ghc-tasty*, ghc-quickcheck*, ghc-test*, ghc-hunit*, hspec*,
>   ghc-hspec*, …
>
> - web.scm: ghc-tagsoup, ghc-cookie, ghc-http*, ghc-wai*, ghc-json,
>   ghc-warp*, ghc-multipart, ghc-aeson*
>
> - crypto.scm: ghc-tf-random, ghc-digest, ghc-cryptonite, ghc-x509*,
>   ghc-asn1*, ghc-pem, ghc-cryptohash*, ghc-entropy, ghc-crypto-api*,
>   ghc-puremd5
>
> - tls.scm: ghc-tls
>
> Maybe that’s enough already.
>
> Does that seem like a good idea?

It does.  Actually, we could do similarly for Perl and Python:
python-web, python-check, python-crypto, etc.

WDYT?

Thanks,
Ludo’.

bug#27284: Memory leak in 'guix pull' or 'make' in guix source

[Ricardo Wurmus]

Ludovic Courtès <ludo@gnu.org> writes:

>> I was hoping we could avoid this, but whatever: let’s do this :)
>
> Yeah, me too.  The problem we have is that Guix is hardly releasable in
> its current state because on systems with 1GiB of memory you can’t
> upgrade, and I think that’s unacceptable.

Yes, I’m feeling the pain with my i686 netbook.

> So what are the options?  If we get a bug-fix for Guile’s compiler
> today, does it help?  If we graft it then we can deliver it without
> having to wait for a Guile release, which helps a bit?
>
> I think it’s all about time: we could wait (and hack!) some more, and
> solve the root problem.  This is the best long-term course of action,
> but at the same time it delays the Guix release.

The way I see it, having very large modules like (gnu packages python)
is not desirable anyway.  So we won’t get around distributing package
definitions.  If we can get a fix for Guile soon that’s great, but we
should begin moving packages independent of that.

>> For haskell.scm I’d begin by moving the following packages away:
>>
>> - check.scm: ghc-tasty*, ghc-quickcheck*, ghc-test*, ghc-hunit*, hspec*,
>>   ghc-hspec*, …
>>
>> - web.scm: ghc-tagsoup, ghc-cookie, ghc-http*, ghc-wai*, ghc-json,
>>   ghc-warp*, ghc-multipart, ghc-aeson*
>>
>> - crypto.scm: ghc-tf-random, ghc-digest, ghc-cryptonite, ghc-x509*,
>>   ghc-asn1*, ghc-pem, ghc-cryptohash*, ghc-entropy, ghc-crypto-api*,
>>   ghc-puremd5
>>
>> - tls.scm: ghc-tls
>>
>> Maybe that’s enough already.
>>
>> Does that seem like a good idea?
>
> It does.

Okay, I’ll get started with splitting haskell.scm.  I’ll prepare one
patch for each target module.

> Actually, we could do similarly for Perl and Python:
> python-web, python-check, python-crypto, etc.

Okay.  I’ll leave that task for someone else, because I don’t think I
can do more than haskell.scm today.

--
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net

bug#27284: Memory leak in 'guix pull' or 'make' in guix source

[Ricardo Wurmus]

Ricardo Wurmus <rekado@elephly.net> writes:

> Okay, I’ll get started with splitting haskell.scm.  I’ll prepare one
> patch for each target module.

I’ve moved packages from haskell.scm to haskell-check.scm,
haskell-web.scm, haskell-crypto.scm, and tls.scm.  I hope haskell.scm is
small enough now.

It’s all in master already.

-- 
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net

bug#27284: Memory leak in 'guix pull' or 'make' in guix source

[Ludovic Courtès]

Morning!

Ricardo Wurmus <rekado@elephly.net> skribis:

> Ludovic Courtès <ludo@gnu.org> writes:

[...]

>> So what are the options?  If we get a bug-fix for Guile’s compiler
>> today, does it help?  If we graft it then we can deliver it without
>> having to wait for a Guile release, which helps a bit?
>>
>> I think it’s all about time: we could wait (and hack!) some more, and
>> solve the root problem.  This is the best long-term course of action,
>> but at the same time it delays the Guix release.
>
> The way I see it, having very large modules like (gnu packages python)
> is not desirable anyway.  So we won’t get around distributing package
> definitions.  If we can get a fix for Guile soon that’s great, but we
> should begin moving packages independent of that.

Right, that makes sense.

>> Actually, we could do similarly for Perl and Python:
>> python-web, python-check, python-crypto, etc.
>
> Okay.  I’ll leave that task for someone else, because I don’t think I
> can do more than haskell.scm today.

I’ve moved 129 packages from python.scm to python-web.scm:

  https://git.savannah.gnu.org/cgit/guix.git/commit/?id=1b2f753d9c73a5431ec1f5510449c480debf80ce

There are still 863 left in python.scm, and it would be good to be at
400–500 max.  I’ll see if I can do python-crypto or python-check today.

Thank you for your work on haskell.scm and perl.scm!

Ludo’.