bug#22629: Towards a new 'guix pull'

bug#22629: Towards a new 'guix pull'

[Ludovic Courtès]

Hello!

Here’s a series of improvements that I think we should make in ‘guix
pull’:

  • Use Git instead of downloading a whole snapshot every time.  The Git
    checkout would be kept in ~/.cache/guix/pull/checkouts, say.

    A related question is whether to use Git itself, which is pretty big
    per ‘guix size’, or to use some libgit2 bindings such as
    <https://git.dthompson.us/guile-git.git> (the closure of libgit2 is
    435 MiB; that of Git is 761 MiB.)

  • Build & install not only Scheme code, but also locales and the Info
    manual.

  • Have a “channel” mechanism, similar to ‘nix-channel’, that would
    allow users to have several Guix variants available in parallel
    instead of just “latest”.  Could work like this:

      guix channel add latest git://git.sv.gnu.org/guix.git master
      guix channel add stable git://git.sv.gnu.org/guix.git stable
      guix channel pull latest
      guix channel set latest
      # here i see the latest versions of everything
      guix channel set stable
      # and here everything is old but super stable ;-)

All 3 items can be done separately, I think.

Any takers?  :-)

Ludo’.

PS: I do not mention the issue of authenticating code here, which is
    obviously very important and deserves to be treated separately.
    Related to that is the question of making sure that what you think
    is the latest version really is the latest version.  We need someone
    to sign certificates saying what the latest commit ID of a repo is.
    See the “The Update Framework” paper!

bug#22629: Towards a new 'guix pull'

[Pjotr Prins]

I wrote up a 'white paper' on a new implementation of Guix channels.
The easy on the eyes version you can read here:

    https://github.com/pjotrp/guix-notes/blob/master/proposals/channel.org

* On Channels

Related to debbugs https://debbugs.gnu.org/cgi/bugreport.cgi?bug=22629 and https://debbugs.gnu.org/cgi/bugreport.cgi?bug=22883#103

** Updating package definitions through channels

*** Introduction

The main purpose of creating Guix channels is to allow users to use
different versions of the Guix package tree that differ from the
master branch, i.e., have people share packages with each other that
are not in the GNU Guix distribution itself.

Within Guix there are good reasons to handle such cases. For example,
to support an older package, such as, ruby 1.8.7 (currently in the
master tree, but could be removed), and to have different editions of
combinations of packages. This versioning we currently achieve by
explicitely working from git checkouts of trees - not trivial for
everyone. Channels will just make versioning easier and accessible for
everyone.

Outside Guix, channels can support also different combinations of
packages, e.g., such as for easily deploying different versions of the
GeneNetwork webservice (development/staging/testing/production
branches). Channels will also make it possible to switch testing off
in certain packages, for example, or to distribute data packages that
do not belong or are not wanted in GNU Guix.

In short, channels allow for easy specialization of deployments.

Also, channels and 'guix publish' distributed services can go hand in
hand in providing such specializations.

*** Usage

Proposed syntax for adding a channel

:  guix channel add channel-name URI branch

Examples

:  guix channel add guix-stable git://git.sv.gnu.org/guix.git stable
:  guix channel add guix-master git://git.sv.gnu.org/guix.git master
:  guix channel add ruby-1.8.7 git:URI/my-ruby-repo.git ruby-1.8.7
:  guix channel add genenetwork-development git:URI/genenetwork-staging.git development
:  guix channel add genenetwork-staging git:URI/genenetwork-staging.git staging

To keep things simple I propose a channel simply reflects a fixed
version of a git tree of a Guix package list. I.e., the exact *state*
of the combined list of packages is what we support in a channel. The
git checkout can be stored in something like

:  ~/.guix-channels/ruby-1.8.7/...

There will only be one directory for one channel. Channels are not
shared between users so they can go somewhere in $HOME.

To get to a certain state, or update a channel, we should allow for
tree updating based on a git SHA value, a tag or just the HEAD. Many
channels will go by HEAD, but it is unwanted because a tree may be in
an unstable and even unusable state. Especially for Guix itself we
should not be using HEAD - the current default with a 'guix pull'.
Ideally 'guix pull' should be tried by core developers before giving
it to everyone. I think we can figure out the last tagged release and
have that as a default rather than HEAD. When someone wants HEAD it
should be explicit. So to get the actual HEAD the syntax can be:

:  guix channel update guix-master origin/HEAD

To get to a specific SHA value on origin

:  guix channel update guix-master 9cf265449f88f89c900e510fe29baba26f30159b

To get a named tagged release

:  guix channel update guix-master v0.12.0-8

To get the latest tagged release (default behaviour)

:  guix channel update guix-master

To switch channels for use

:  guix channel set guix-master

When a channel gets updated I think we ought to check it out fully
from scratch by default. Reason being that we can't deal with
conflicts and maybe changed upstream trees - git also allows rewriting
history and that can be nasty. If some person feels confident, we
could still add a switch

:  guix channel update guix-master --pull

or

:  guix channel update guix-master --checkout v0.12.0-8

which would do what you expect it to do.

Obviously all this behaviour reflects the way git behaves. I think it
is fine because if we start adding other transports (say https,
mercurial or darcs) it would be clear from the channel URI what it was
and we can change the commands.

** Channel metadata

The channel should describe its purpose and set certain prerequisites,
such as a description and the version of guile it requires. This can
be added as a channel-meta.scm file in the root of the source tree in
git. It could also point to one or more 'guix publish' servers.

** Compiling the package tree

So far, we have only discussed the package tree. To compile it we need
guile. One thing the current 'guix pull' does is compile guile to
compile the tree. This behaviour does not make use of substitute
servers, takes time and is somewhat error prone. I think what we
should do is reuse a guile already on the system or install it as a
binary using a substitute server and then compile the packages.

** Updating guix and guix-daemon

Now we come to updating guix and guix-daemon itself. I think that this
functionality does not belong in channel. Not least because not all
users have the privileges to restart the daemon.

To update guix essentially one will add a relevant channel, update the
channel to point and install guix from that package tree using 'guix
package'.

It can be that the currently running guix tools, for some reason, can
not support a package tree. To notify that, I suggest we use that same
meta data in the channel tree. The metadata can tell what version of
guix and guix-daemon should be used (say >=0.11.0) and advise the user
and system administrator.

** Guix pull

Where channels take the role of giving state to the package tree, a
new 'guix pull' could have the task of *upgrading* guix and the guix
daemon to latest. As stated above the package tree should not move to
^HEAD, but instead to the latest tagged release (which can happen more
often then we are doing today - say on a weekly basis). Starting from
an approved Guix version 'guix pull' could build/download the guix
package itself and suggest updating the daemon to latest.

** Security and trusted channels

Guix has a security mechanism for its substitute servers, i.e., you
need to install a key to trust a 'guix publish' server. For channels
using git the model is to use git trees which is open to security
concerns. To introduce the concept of trusted channels we can use the
same 'guix publish' servers which can also provide the binary
substitutes for a channel. In fact, I think secure channels should be
the default, and we should be able to install packages using such a
server, after switching to a channel, with

:  guix channel set genenetwork-staging
:  guix package -i genenetwork2 --substitute-urls=https://guix.genenetwork.org

which would validate the URL of the channel (a git URI) with the
substitute server, i.e., guix, by default, will only install channels
that are known by the publish/substitute server.

Even so, we have to allow for convenience and the fact that not
everyone wants to run a 'guix publish' server. To install an insecure
channel we should allow

:  guix channel set genenetwork-staging
:  guix package -i genenetwork2 --force-untrusted

which bypasses the 'trusted' mechanism of asking permission from the
substitute server.

To take the trusted model further, the 'guix publish' server could
also serve SHA values of the channel git trees. I.e., if a user requests

:  guix channel update guix-master v0.12.0-8

the server could say that tagged release v0.12.0-8 can be checked out
and the SHA values should be such and such. This would give full
security for guix-master checkouts.

bug#22629: Towards a new 'guix pull'

[Leo Famulari]

[-- Attachment #1: Type: text/plain, Size: 1749 bytes --]

On Wed, Feb 22, 2017 at 07:57:06AM +0000, Pjotr Prins wrote:
> To keep things simple I propose a channel simply reflects a fixed
> version of a git tree of a Guix package list. I.e., the exact *state*
> of the combined list of packages is what we support in a channel. The
> git checkout can be stored in something like

You shared *a lot* of ideas following this paragraph, but I think that
from here we can start thinking about how to incrementally evolve from
the current `guix pull` to something better.

Currently, you can `guix pull` any commit of GNU Guix with `guix pull
--url` [0]. This feature depends on the Git server implementation's HTTP
interface, but I bet it's a common feature, and it can already be used
with other Git repositories besides our Savannah repo. This achieves the
goal of channels, "easy specialization of deployments". Well, "easy" is
a matter of degrees ;)

I was thinking we could start by keeping `guix pull` the same from the
user's point of view, but change it to support Git in addition to
downloading a tarball. We already have code for this in (guix
git-download) which uses the Git command-line tool. [1]

The next step would be to keep some metadata about which channel the
last pull came from, so that subsequent pulls would point to it.

What do people think?

I haven't wrapped my head around all the proposed changes. I think it
will be best to keep the goal of "easy specialization of deployments" in
mind and work towards it incrementally.

[0]
https://git.savannah.gnu.org/cgit/guix.git/commit/?id=8a9cffb202414b20081910115ba76402924bdcdd

[1] In the long run, we should ditch the Git CLI and use something like
guile-git:
https://gitlab.com/amirouche/guile-git

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

bug#28471: guix pull doesn't update the user manual

[Maxim Cournoyer]

guix pull should build and install the latest documentation of Guix to
the user profile so that `info guix' always displays the current
documentation.

This seems to confuse Guix users every now and then, myself included,
and the work around of cloning a full copy of the guix sources just to
build the manual is not user friendly nor trivial.

bug#22629: Towards a new 'guix pull'

[Ludovic Courtès]

Hello Guix!

ludo@gnu.org (Ludovic Courtès) skribis:

> Here’s a series of improvements that I think we should make in ‘guix
> pull’:
>
>   • Use Git instead of downloading a whole snapshot every time.  The Git
>     checkout would be kept in ~/.cache/guix/pull/checkouts, say.

That was done a while back.   :-)

>   • Build & install not only Scheme code, but also locales and the Info
>     manual.

… and also the C++ bits.

With the work done in <https://bugs.gnu.org/27284>, we can build all of
Guix and its dependencies when doing ‘guix pull’, which brings us closer
to what the ‘guix’ package provides.

Thus we can at least partly address the issue described in
<https://debbugs.gnu.org/cgi/bugreport.cgi?bug=22629#10>, i.e., have
‘guix pull’ install not just a bunch of .go files but also guile-git,
guile-ssh, gnutls, and Guile itself.

To do that, we need a “paradigm shift” (sounds cute, no? :-)).
Currently, ‘scripts/guix’ fiddles with its %load-path and adds
~/.config/guix/latest if it exists.  The new ‘guix pull’ should provide
not just .go files but also a ‘guix’ command referring to the latest
Guile, Guile-Git, etc.

To achieve this, I think the new approach would be to have:

  export PATH=$HOME/.config/guix/current/bin:$PATH
  export INFOPATH=$HOME/.config/guix/current/share/info:$INFOPATH
  …

‘guix pull’ would update ~/.config/guix/current, which would contain the
‘guix’ command, all the .go files, locales, the Info manual, and ideally
‘guix-daemon’ as well.

~/.config/guix/current could be a profile, or something that resembles a
profile.

For a while ‘guix pull’ will have to keep updating ~/.config/guix/latest
as well, for users running an old ‘guix’ command.

Thoughts?

Ludo’.

bug#22629: Towards a new 'guix pull'

[Nils Gillmann]

Ludovic Courtès transcribed 2.0K bytes:
> Hello Guix!
> 
> ludo@gnu.org (Ludovic Courtès) skribis:
> 
> > Here’s a series of improvements that I think we should make in ‘guix
> > pull’:
> >
> >   • Use Git instead of downloading a whole snapshot every time.  The Git
> >     checkout would be kept in ~/.cache/guix/pull/checkouts, say.
> 
> That was done a while back.   :-)
> 
> >   • Build & install not only Scheme code, but also locales and the Info
> >     manual.
> 
> … and also the C++ bits.
> 
> With the work done in <https://bugs.gnu.org/27284>, we can build all of
> Guix and its dependencies when doing ‘guix pull’, which brings us closer
> to what the ‘guix’ package provides.
> 
> Thus we can at least partly address the issue described in
> <https://debbugs.gnu.org/cgi/bugreport.cgi?bug=22629#10>, i.e., have
> ‘guix pull’ install not just a bunch of .go files but also guile-git,
> guile-ssh, gnutls, and Guile itself.
> 
> To do that, we need a “paradigm shift” (sounds cute, no? :-)).
> Currently, ‘scripts/guix’ fiddles with its %load-path and adds
> ~/.config/guix/latest if it exists.  The new ‘guix pull’ should provide
> not just .go files but also a ‘guix’ command referring to the latest
> Guile, Guile-Git, etc.
> 
> To achieve this, I think the new approach would be to have:
> 
>   export PATH=$HOME/.config/guix/current/bin:$PATH
>   export INFOPATH=$HOME/.config/guix/current/share/info:$INFOPATH
>   …
> 
> ‘guix pull’ would update ~/.config/guix/current, which would contain the
> ‘guix’ command, all the .go files, locales, the Info manual, and ideally
> ‘guix-daemon’ as well.
> 
> ~/.config/guix/current could be a profile, or something that resembles a
> profile.
> 
> For a while ‘guix pull’ will have to keep updating ~/.config/guix/latest
> as well, for users running an old ‘guix’ command.
> 
> Thoughts?

First thought: Woo! Good progress!
On your propossed steps, sounds good to me.

> Ludo’.
> 
> 
> 

bug#22629: [PATCH 0/4] 'guix pull' produces a self-contained Guix

[Ludovic Courtès]

Hello Guix!

Here is the “new” ‘guix pull’ that we discussed notably in this thread:

  https://bugs.gnu.org/22629

The major difference is that instead of just building a bunch of modules
and putting them under ~/.config/guix/latest, it now produces a
standalone package (with bin/guix, share/info/guix.info, etc.) and puts
it in a profile under ~/.config/guix/current.  Quoth the manual:

     The result of running ‘guix pull’ is a “profile” available under
  ‘~/.config/guix/current’ containing the latest Guix.  Thus, make sure to
  add it to the beginning of your search path so that you use the latest
  version, and similarly for the Info manual (*note Documentation::):

       export PATH="$HOME/.config/guix/current/bin:$PATH"
       export INFOPATH="$HOME/.config/guix/current/share/info:$INFOPATH"

     This ‘~/.config/guix/current’ profile works like any other profile
  created by ‘guix package’ (*note Invoking guix package::).  That is, you
  can list generations, roll back to the previous generation—i.e., the
  previous Guix—and so on:

       $ guix package -p ~/.config/guix/current -l
       Generation 1	May 25 2018 10:06:41
         guix	221951a	out	/gnu/store/i4dfk7vw5k112s49jrhl6hwsfnh6wr7l-guix-221951af4

       Generation 2	May 27 2018 19:07:47
        + guix	2fbae00	out	/gnu/store/44cv9hyvxg34xf5kblf5dz57hc52y4bm-guix-2fbae006f
        - guix	221951a	out	/gnu/store/i4dfk7vw5k112s49jrhl6hwsfnh6wr7l-guix-221951af4

       Generation 3	May 30 2018 16:11:39	(current)
        + guix	a076f19	out	/gnu/store/332czkicwwg6lc3x4aqbw5q2mq12s7fj-guix-a076f1990
        - guix	2fbae00	out	/gnu/store/44cv9hyvxg34xf5kblf5dz57hc52y4bm-guix-2fbae006f
       $ guix package -p ~/.config/guix/current --roll-back
       switched from generation 3 to 2

There are two requirements it fulfills in terms of compatibility:

  1. The modified ‘build-aux/build-self.scm’ still does the right thing
     when evaluated by an “old” Guix—that is, it produces a bunch of
     modules for use in ~/.config/guix/latest as before.

  2. The modified ‘guix pull’ produces ~/.config/guix/current even when
     invoked on a commit of a past Guix.  That is, it automatically
     produces a ‘guix’ command using the modules returned by the old
     ‘build-self.scm’.

There are various improvements we can make from there.  For example,
using “manifest entry properties” as proposed in
<https://bugs.gnu.org/31442>, we can attach meta-data (commit ID, repo
URL, etc.) in each manifest entry that ‘guix pull’ populates; then we
can arrange for ‘guix pull --list-generations’ (say) to display that
information.

We could add ‘guix pull’ options for convenient: ‘--roll-back’,
‘--profile’, etc.

Going forward, additional “channels” could be presented as entries in
the ~/.config/guix/current manifest.

Caveats:

  1. The ~/.config/guix/current profile really lives there.  That is,
     unlike ~/.guix-profile, it’s not in /var/guix/profiles/per-user.
     That could be an issue for cluster setups where home directories
     are not scanned by the Guix GC.  Cluster folks, please tell me!

  2. The translated Info manual is not built.  Julien: could you turn
     the big ‘xref_command’ in a script or something that we can more
     easily reuse in (guix self)?

  3. C++ code is not built.  I wonder which will come first: getting rid
     of the C++ code, or building it?  :-)

Feedback welcome!

Ludo’.

Ludovic Courtès (4):
  self: Produce a complete package with the 'guix' command.
  pull: Install the new Guix in a profile.
  self: Compute and use locale data.
  self: Build the Info manual.

 build-aux/build-self.scm            |  19 +-
 build-aux/compile-as-derivation.scm |   2 +-
 doc/guix.texi                       |  38 +++-
 guix/scripts/pull.scm               |  79 +++++---
 guix/self.scm                       | 283 ++++++++++++++++++++++++----
 scripts/guix.in                     |  14 +-
 6 files changed, 351 insertions(+), 84 deletions(-)

-- 
2.17.0

bug#22629: [PATCH 1/4] self: Produce a complete package with the 'guix' command.

[Ludovic Courtès]

* guix/self.scm (guix-command): New procedure.
(compiled-guix): Add #:pull-version parameter.
[command, package]: New variables.
Honor PULL-VERSION.
(guix-derivation): Add #:pull-version and pass it to 'compiled-guix'.
* build-aux/build-self.scm (build-program): Add #:pull-version
parameter.  Pass it to 'guix-derivation'.
(build): Add #:pull-version and pass it to 'build-program'.
* build-aux/compile-as-derivation.scm: Pass #:pull-version to BUILD.
---
 build-aux/build-self.scm            |  19 +++-
 build-aux/compile-as-derivation.scm |   2 +-
 guix/self.scm                       | 148 +++++++++++++++++++++-------
 3 files changed, 129 insertions(+), 40 deletions(-)

diff --git a/build-aux/build-self.scm b/build-aux/build-self.scm
index bccb7a959..5898b6515 100644
--- a/build-aux/build-self.scm
+++ b/build-aux/build-self.scm
@@ -184,7 +184,8 @@ person's version identifier."
   (date->string (current-date 0) "~Y~m~d.~H"))
 
 (define* (build-program source version
-                        #:optional (guile-version (effective-version)))
+                        #:optional (guile-version (effective-version))
+                        #:key (pull-version 0))
   "Return a program that computes the derivation to build Guix from SOURCE."
   (define select?
     ;; Select every module but (guix config) and non-Guix modules.
@@ -253,11 +254,14 @@ person's version identifier."
                               (spin system)))
 
                            (display
-                            (derivation-file-name
+                            (and=>
                              (run-with-store store
                                (guix-derivation #$source #$version
-                                                #$guile-version)
-                               #:system system)))))))
+                                                #$guile-version
+                                                #:pull-version
+                                                #$pull-version)
+                               #:system system)
+                             derivation-file-name))))))
                   #:module-path (list source))))
 
 ;; The procedure below is our return value.
@@ -266,13 +270,15 @@ person's version identifier."
                 (guile-version (match ((@ (guile) version))
                                  ("2.2.2" "2.2.2")
                                  (_       (effective-version))))
+                (pull-version 0)
                 #:allow-other-keys
                 #:rest rest)
   "Return a derivation that unpacks SOURCE into STORE and compiles Scheme
 files."
   ;; Build the build program and then use it as a trampoline to build from
   ;; SOURCE.
-  (mlet %store-monad ((build  (build-program source version guile-version))
+  (mlet %store-monad ((build  (build-program source version guile-version
+                                             #:pull-version pull-version))
                       (system (if system (return system) (current-system))))
     (mbegin %store-monad
       (show-what-to-build* (list build))
@@ -292,6 +298,9 @@ files."
              (return (newline (current-output-port)))
              ((store-lift add-temp-root) drv)
              (return (read-derivation-from-file drv))))
+          ("#f"
+           ;; Unsupported PULL-VERSION.
+           (return #f))
           ((? string? str)
            (error "invalid build result" (list build str))))))))
 
diff --git a/build-aux/compile-as-derivation.scm b/build-aux/compile-as-derivation.scm
index afb134a92..2a45e71bb 100644
--- a/build-aux/compile-as-derivation.scm
+++ b/build-aux/compile-as-derivation.scm
@@ -43,7 +43,7 @@
            (mlet* %store-monad ((source (interned-file source "guix-source"
                                                        #:select? git?
                                                        #:recursive? #t))
-                                (drv    (build source)))
+                                (drv    (build source #:pull-version 1)))
              (mbegin %store-monad
                (show-what-to-build* (list drv))
                (built-derivations (list drv))
diff --git a/guix/self.scm b/guix/self.scm
index 4378a3dee..f0641c342 100644
--- a/guix/self.scm
+++ b/guix/self.scm
@@ -34,6 +34,7 @@
   #:use-module (srfi srfi-9)
   #:use-module (ice-9 match)
   #:export (make-config.scm
+            whole-package                     ;for internal use in 'guix pull'
             compiled-guix
             guix-derivation
             reload-guix))
@@ -189,7 +190,66 @@ list of file-name/file-like objects suitable as inputs to 'imported-files'."
            (file-name->module-name (string-drop file prefix)))
          (scheme-files (string-append directory "/" sub-directory)))))
 
+(define* (guix-command modules #:key (dependencies '())
+                       (guile-version (effective-version)))
+  "Return the 'guix' command such that it adds MODULES and DEPENDENCIES in its
+load path."
+  (program-file "guix-command"
+                #~(begin
+                    (set! %load-path
+                      (append '#$(map (lambda (package)
+                                        (file-append package
+                                                     "/share/guile/site/"
+                                                     guile-version))
+                                      dependencies)
+                              %load-path))
+
+                    (set! %load-compiled-path
+                      (append '#$(map (lambda (package)
+                                        (file-append package "/lib/guile/"
+                                                     guile-version
+                                                     "/site-ccache"))
+                                      dependencies)
+                              %load-compiled-path))
+
+                    (set! %load-path (cons #$modules %load-path))
+                    (set! %load-compiled-path
+                      (cons #$modules %load-compiled-path))
+
+                    (let ((guix-main (module-ref (resolve-interface '(guix ui))
+                                                 'guix-main)))
+                      ;; TODO: Compute locale data.
+                      ;; (bindtextdomain "guix" "@localedir@")
+                      ;; (bindtextdomain "guix-packages" "@localedir@")
+
+                      ;; XXX: It would be more convenient to change it to:
+                      ;;   (exit (apply guix-main (command-line)))
+                      (apply guix-main (command-line))))))
+
+(define* (whole-package name modules dependencies
+                        #:key (guile-version (effective-version)))
+  "Return the whole Guix package NAME that uses MODULES, a derivation of all
+the modules, and DEPENDENCIES, a list of packages depended on."
+  (let ((command (guix-command modules
+                               #:dependencies dependencies
+                               #:guile-version guile-version)))
+    ;; TODO: Move compiled modules to 'lib/guile' instead of 'share/guile'.
+    (computed-file name
+                   (with-imported-modules '((guix build utils))
+                     #~(begin
+                         (use-modules (guix build utils))
+                         (mkdir-p (string-append #$output "/bin"))
+                         (symlink #$command
+                                  (string-append #$output "/bin/guix"))
+
+                         (let ((modules (string-append #$output
+                                                       "/share/guile/site/"
+                                                       (effective-version))))
+                           (mkdir-p (dirname modules))
+                           (symlink #$modules modules)))))))
+
 (define* (compiled-guix source #:key (version %guix-version)
+                        (pull-version 1)
                         (name (string-append "guix-" version))
                         (guile-version (effective-version))
                         (guile-for-build (guile-for-build guile-version))
@@ -215,7 +275,6 @@ list of file-name/file-like objects suitable as inputs to 'imported-files'."
                        "guile-git"
                        "guile2.0-git"))
 
-
   (define dependencies
     (match (append-map (lambda (package)
                          (cons (list "x" package)
@@ -332,32 +391,46 @@ list of file-name/file-like objects suitable as inputs to 'imported-files'."
                                          %guix-home-page-url)))
                  #:guile-for-build guile-for-build))
 
-  (directory-union name
-                   (append-map (lambda (node)
-                                 (list (node-source node)
-                                       (node-compiled node)))
+  (define built-modules
+    (directory-union (string-append name "-modules")
+                     (append-map (lambda (node)
+                                   (list (node-source node)
+                                         (node-compiled node)))
 
-                               ;; Note: *CONFIG* comes first so that it
-                               ;; overrides the (guix config) module that
-                               ;; comes with *CORE-MODULES*.
-                               (list *config*
-                                     *cli-modules*
-                                     *system-modules*
-                                     *package-modules*
-                                     *core-package-modules*
-                                     *extra-modules*
-                                     *core-modules*))
+                                 ;; Note: *CONFIG* comes first so that it
+                                 ;; overrides the (guix config) module that
+                                 ;; comes with *CORE-MODULES*.
+                                 (list *config*
+                                       *cli-modules*
+                                       *system-modules*
+                                       *package-modules*
+                                       *core-package-modules*
+                                       *extra-modules*
+                                       *core-modules*))
 
-                   ;; Silently choose the first entry upon collision so that
-                   ;; we choose *CONFIG*.
-                   #:resolve-collision 'first
+                     ;; Silently choose the first entry upon collision so that
+                     ;; we choose *CONFIG*.
+                     #:resolve-collision 'first
 
-                   ;; When we do (add-to-store "utils.scm"), "utils.scm" must
-                   ;; be a regular file, not a symlink.  Thus, arrange so that
-                   ;; regular files appear as regular files in the final
-                   ;; output.
-                   #:copy? #t
-                   #:quiet? #t))
+                     ;; When we do (add-to-store "utils.scm"), "utils.scm" must
+                     ;; be a regular file, not a symlink.  Thus, arrange so that
+                     ;; regular files appear as regular files in the final
+                     ;; output.
+                     #:copy? #t
+                     #:quiet? #t))
+
+  ;; Version 0 of 'guix pull' meant we'd just return Scheme modules.
+  ;; Version 1 is when we return the full package.
+  (cond ((= 1 pull-version)
+         ;; The whole package, with a standard file hierarchy.
+         (whole-package name built-modules dependencies
+                        #:guile-version guile-version))
+        ((= 0 pull-version)
+         ;; Legacy 'guix pull': just return the compiled modules.
+         built-modules)
+        (else
+         ;; Unsupported 'guix pull' version.
+         #f)))
 
 \f
 ;;;
@@ -601,9 +674,12 @@ running Guile."
                  'guile-2.0))))
 
 (define* (guix-derivation source version
-                          #:optional (guile-version (effective-version)))
+                          #:optional (guile-version (effective-version))
+                          #:key (pull-version 0))
   "Return, as a monadic value, the derivation to build the Guix from SOURCE
-for GUILE-VERSION.  Use VERSION as the version string."
+for GUILE-VERSION.  Use VERSION as the version string.  PULL-VERSION specifies
+the version of the 'guix pull' protocol.  Return #f if this PULL-VERSION value
+is not supported."
   (define (shorten version)
     (if (and (string-every char-set:hex-digit version)
              (> (string-length version) 9))
@@ -615,11 +691,15 @@ for GUILE-VERSION.  Use VERSION as the version string."
 
   (mbegin %store-monad
     (set-guile-for-build guile)
-    (lower-object (compiled-guix source
-                                 #:version version
-                                 #:name (string-append "guix-"
-                                                       (shorten version))
-                                 #:guile-version (match guile-version
-                                                   ("2.2.2" "2.2")
-                                                   (version version))
-                                 #:guile-for-build guile))))
+    (let ((guix (compiled-guix source
+                               #:version version
+                               #:name (string-append "guix-"
+                                                     (shorten version))
+                               #:pull-version pull-version
+                               #:guile-version (match guile-version
+                                                 ("2.2.2" "2.2")
+                                                 (version version))
+                               #:guile-for-build guile)))
+      (if guix
+          (lower-object guix)
+          (return #f)))))
-- 
2.17.0

bug#22629: [PATCH 2/4] pull: Install the new Guix in a profile.

[Ludovic Courtès]

* guix/scripts/pull.scm (%pull-version): New variable.
(build-from-source): Pass #:pull-version to BUILD.
(whole-package-for-legacy, derivation->manifest-entry): New procedure.
(build-and-install): Rewrite in terms of 'build-and-use-profile'.
* scripts/guix.in (augment-load-paths!): Remove use of
~/.config/guix/latest.
* doc/guix.texi (Invoking guix pull): Document it.
---
 doc/guix.texi         | 38 ++++++++++++++++++---
 guix/scripts/pull.scm | 79 +++++++++++++++++++++++++++++--------------
 scripts/guix.in       | 14 +-------
 3 files changed, 87 insertions(+), 44 deletions(-)

diff --git a/doc/guix.texi b/doc/guix.texi
index 09749b15e..92e322787 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -2742,11 +2742,39 @@ Any user can update their Guix copy using @command{guix pull}, and the
 effect is limited to the user who run @command{guix pull}.  For
 instance, when user @code{root} runs @command{guix pull}, this has no
 effect on the version of Guix that user @code{alice} sees, and vice
-versa@footnote{Under the hood, @command{guix pull} updates the
-@file{~/.config/guix/latest} symbolic link to point to the latest Guix,
-and the @command{guix} command loads code from there.  Currently, the
-only way to roll back an invocation of @command{guix pull} is to
-manually update this symlink to point to the previous Guix.}.
+versa.
+
+The result of running @command{guix pull} is a @dfn{profile} available
+under @file{~/.config/guix/current} containing the latest Guix.  Thus,
+make sure to add it to the beginning of your search path so that you use
+the latest version, and similarly for the Info manual
+(@pxref{Documentation}):
+
+@example
+export PATH="$HOME/.config/guix/current/bin:$PATH"
+export INFOPATH="$HOME/.config/guix/current/share/info:$INFOPATH"
+@end example
+
+This @code{~/.config/guix/current} profile works like any other profile
+created by @command{guix package} (@pxref{Invoking guix package}).  That
+is, you can list generations, roll back to the previous
+generation---i.e., the previous Guix---and so on:
+
+@example
+$ guix package -p ~/.config/guix/current -l
+Generation 1	May 25 2018 10:06:41
+  guix	221951a	out	/gnu/store/i4dfk7vw5k112s49jrhl6hwsfnh6wr7l-guix-221951af4
+
+Generation 2	May 27 2018 19:07:47
+ + guix	2fbae00	out	/gnu/store/44cv9hyvxg34xf5kblf5dz57hc52y4bm-guix-2fbae006f
+ - guix	221951a	out	/gnu/store/i4dfk7vw5k112s49jrhl6hwsfnh6wr7l-guix-221951af4
+
+Generation 3	May 30 2018 16:11:39	(current)
+ + guix	a076f19	out	/gnu/store/332czkicwwg6lc3x4aqbw5q2mq12s7fj-guix-a076f1990
+ - guix	2fbae00	out	/gnu/store/44cv9hyvxg34xf5kblf5dz57hc52y4bm-guix-2fbae006f
+$ guix package -p ~/.config/guix/current --roll-back
+switched from generation 3 to 2
+@end example
 
 The @command{guix pull} command is usually invoked with no arguments,
 but it supports the following options:
diff --git a/guix/scripts/pull.scm b/guix/scripts/pull.scm
index 64c2196e0..c5ceebccb 100644
--- a/guix/scripts/pull.scm
+++ b/guix/scripts/pull.scm
@@ -1,5 +1,5 @@
 ;;; GNU Guix --- Functional package management for GNU
-;;; Copyright © 2013, 2014, 2015, 2017 Ludovic Courtès <ludo@gnu.org>
+;;; Copyright © 2013, 2014, 2015, 2017, 2018 Ludovic Courtès <ludo@gnu.org>
 ;;; Copyright © 2017 Marius Bakke <mbakke@fastmail.com>
 ;;;
 ;;; This file is part of GNU Guix.
@@ -25,10 +25,15 @@
   #:use-module (guix config)
   #:use-module (guix packages)
   #:use-module (guix derivations)
+  #:use-module (guix profiles)
   #:use-module (guix gexp)
   #:use-module (guix grafts)
   #:use-module (guix monads)
   #:use-module (guix scripts build)
+  #:autoload   (guix self) (whole-package)
+  #:autoload   (gnu packages ssh) (guile-ssh)
+  #:autoload   (gnu packages tls) (gnutls)
+  #:use-module ((guix scripts package) #:select (build-and-use-profile))
   #:use-module ((guix build utils)
                 #:select (with-directory-excursion delete-file-recursively))
   #:use-module ((guix build download)
@@ -158,6 +163,12 @@ Download and deploy the latest version of Guix.\n"))
   ;; a makefile, and, similarly, is intended to always keep this name.
   "build-aux/build-self.scm")
 
+(define %pull-version
+  ;; This is the version of the 'guix pull' protocol.  It specifies what's
+  ;; expected from %SELF-BUILD-FILE.  The initial version ("0") was when we'd
+  ;; place a set of compiled Guile modules in ~/.config/guix/latest.
+  1)
+
 (define* (build-from-source source
                             #:key verbose? commit)
   "Return a derivation to build Guix from SOURCE, using the self-build script
@@ -170,35 +181,51 @@ contained therein.  Use COMMIT as the version string."
          (build  (primitive-load script)))
     ;; BUILD must be a monadic procedure of at least one argument: the source
     ;; tree.
-    (build source #:verbose? verbose? #:version commit)))
+    ;;
+    ;; Note: BUILD can return #f if it does not support %PULL-VERSION.  In the
+    ;; future we'll fall back to a previous version of the protocol when that
+    ;; happens.
+    (build source #:verbose? verbose? #:version commit
+           #:pull-version %pull-version)))
+
+(define (whole-package-for-legacy name modules)
+  "Return a full-blown Guix package for MODULES, a derivation that builds Guix
+modules in the old ~/.config/guix/latest style."
+  (whole-package name modules
+
+                 ;; In the "old style", %SELF-BUILD-FILE would simply return a
+                 ;; derivation that builds modules.  We have to infer what the
+                 ;; dependencies of these modules were.
+                 (list guile-json guile-git guile-bytestructures
+                       guile-ssh gnutls)))
+
+(define (derivation->manifest-entry drv commit)
+  "Return a manifest entry for DRV, which represents Guix at COMMIT."
+  (mbegin %store-monad
+    (what-to-build (list drv))
+    (built-derivations (list drv))
+    (let ((out (derivation->output-path drv)))
+      (return (manifest-entry
+                (name "guix")
+                (version (string-take commit 7))
+                (item (if (file-exists? (string-append out "/bin/guix"))
+                          drv
+                          (whole-package-for-legacy (string-append name "-"
+                                                                   version)
+                                                    drv))))))))
 
 (define* (build-and-install source config-dir
                             #:key verbose? commit)
   "Build the tool from SOURCE, and install it in CONFIG-DIR."
-  (mlet* %store-monad ((source        (build-from-source source
-                                                         #:commit commit
-                                                         #:verbose? verbose?))
-                       (source-dir -> (derivation->output-path source))
-                       (to-do?        (what-to-build (list source)))
-                       (built?        (built-derivations (list source))))
-    ;; Always update the 'latest' symlink, regardless of whether SOURCE was
-    ;; already built or not.
-    (if built?
-        (mlet* %store-monad
-            ((latest -> (string-append config-dir "/latest"))
-             (done      (indirect-root-added latest)))
-          (if (and (file-exists? latest)
-                   (string=? (readlink latest) source-dir))
-              (begin
-                (display (G_ "Guix already up to date\n"))
-                (return #t))
-              (begin
-                (switch-symlinks latest source-dir)
-                (format #t
-                        (G_ "updated ~a successfully deployed under `~a'~%")
-                        %guix-package-name latest)
-                (return #t))))
-        (leave (G_ "failed to update Guix, check the build log~%")))))
+  (define update-profile
+    (store-lift build-and-use-profile))
+
+  (mlet* %store-monad ((drv   (build-from-source source
+                                                 #:commit commit
+                                                 #:verbose? verbose?))
+                       (entry (derivation->manifest-entry drv commit)))
+    (update-profile (string-append config-dir "/current")
+                    (manifest (list entry)))))
 
 (define (honor-lets-encrypt-certificates! store)
   "Tell Guile-Git to use the Let's Encrypt certificates."
diff --git a/scripts/guix.in b/scripts/guix.in
index d1c12eae5..0a3ab1f64 100644
--- a/scripts/guix.in
+++ b/scripts/guix.in
@@ -23,25 +23,13 @@
 ;; IMPORTANT: We must avoid loading any modules from Guix here,
 ;; because we need to adjust the guile load paths first.
 ;; It's okay to import modules from core Guile though.
-(use-modules (srfi srfi-26))
 
 (define-syntax-rule (push! elt v) (set! v (cons elt v)))
 
 (define (augment-load-paths!)
   ;; Add installed modules to load-path.
   (push! "@guilemoduledir@" %load-path)
-  (push! "@guileobjectdir@" %load-compiled-path)
-
-  ;; Add modules fetched by 'guix pull' to load-path.
-  (let ((updates-dir (and=> (or (getenv "XDG_CONFIG_HOME")
-                                (and=> (getenv "HOME")
-                                       (cut string-append <> "/.config")))
-                            (cut string-append <> "/guix/latest"))))
-    (when (and=> updates-dir file-exists?)
-      ;; XXX: Currently 'guix pull' puts both .scm and .go files in
-      ;; UPDATES-DIR.
-      (push! updates-dir %load-path)
-      (push! updates-dir %load-compiled-path))))
+  (push! "@guileobjectdir@" %load-compiled-path))
 
 (define* (main #:optional (args (command-line)))
   (unless (getenv "GUIX_UNINSTALLED")
-- 
2.17.0

bug#22629: [PATCH 3/4] self: Compute and use locale data.

[Ludovic Courtès]

* guix/self.scm (locale-data): New procedure.
(guix-command): Add SOURCE parameter.  Call 'locale-data' when SOURCE is
true and use it in staged 'bindtextdomain' calls.
(whole-package): Add #:command and honor it.
(compiled-guix): Pass #:command to 'whole-package'.
---
 guix/self.scm | 101 ++++++++++++++++++++++++++++++++++++++------------
 1 file changed, 77 insertions(+), 24 deletions(-)

diff --git a/guix/self.scm b/guix/self.scm
index f0641c342..92e69ffd5 100644
--- a/guix/self.scm
+++ b/guix/self.scm
@@ -190,7 +190,47 @@ list of file-name/file-like objects suitable as inputs to 'imported-files'."
            (file-name->module-name (string-drop file prefix)))
          (scheme-files (string-append directory "/" sub-directory)))))
 
-(define* (guix-command modules #:key (dependencies '())
+(define* (locale-data source domain
+                      #:optional (sub-directory domain))
+  "Return the locale data from 'po/SUB-DIRECTORY' in SOURCE, corresponding to
+DOMAIN, a gettext domain."
+  (define gettext
+    (module-ref (resolve-interface '(gnu packages gettext))
+                'gettext-minimal))
+
+  (define build
+    (with-imported-modules '((guix build utils))
+      #~(begin
+          (use-modules (guix build utils)
+                       (srfi srfi-26)
+                       (ice-9 match) (ice-9 ftw))
+
+          (define po-directory
+            (string-append #$source "/po/" #$sub-directory))
+
+          (define (compile language)
+            (let ((gmo (string-append #$output "/" language "/LC_MESSAGES/"
+                                      #$domain ".mo")))
+              (mkdir-p (dirname gmo))
+              (invoke #+(file-append gettext "/bin/msgfmt")
+                      "-c" "--statistics" "--verbose"
+                      "-o" gmo
+                      (string-append po-directory "/" language ".po"))))
+
+          (define (linguas)
+            ;; Return the list of languages.  Note: don't read 'LINGUAS'
+            ;; because it contains things like 'en@boldquot' that do not have
+            ;; a corresponding .po file.
+            (map (cut basename <> ".po")
+                 (scandir po-directory
+                          (cut string-suffix? ".po" <>))))
+
+          (for-each compile (linguas)))))
+
+  (computed-file (string-append "guix-locale-" domain)
+                 build))
+
+(define* (guix-command modules #:key source (dependencies '())
                        (guile-version (effective-version)))
   "Return the 'guix' command such that it adds MODULES and DEPENDENCIES in its
 load path."
@@ -218,35 +258,43 @@ load path."
 
                     (let ((guix-main (module-ref (resolve-interface '(guix ui))
                                                  'guix-main)))
-                      ;; TODO: Compute locale data.
-                      ;; (bindtextdomain "guix" "@localedir@")
-                      ;; (bindtextdomain "guix-packages" "@localedir@")
+                      #$(if source
+                            #~(begin
+                                (bindtextdomain "guix"
+                                                #$(locale-data source "guix"))
+                                (bindtextdomain "guix-packages"
+                                                #$(locale-data source
+                                                               "guix-packages"
+                                                               "packages")))
+                            #t)
 
                       ;; XXX: It would be more convenient to change it to:
                       ;;   (exit (apply guix-main (command-line)))
                       (apply guix-main (command-line))))))
 
 (define* (whole-package name modules dependencies
-                        #:key (guile-version (effective-version)))
+                        #:key
+                        (guile-version (effective-version))
+                        (command (guix-command modules
+                                               #:dependencies dependencies
+                                               #:guile-version guile-version)))
   "Return the whole Guix package NAME that uses MODULES, a derivation of all
-the modules, and DEPENDENCIES, a list of packages depended on."
-  (let ((command (guix-command modules
-                               #:dependencies dependencies
-                               #:guile-version guile-version)))
-    ;; TODO: Move compiled modules to 'lib/guile' instead of 'share/guile'.
-    (computed-file name
-                   (with-imported-modules '((guix build utils))
-                     #~(begin
-                         (use-modules (guix build utils))
-                         (mkdir-p (string-append #$output "/bin"))
-                         (symlink #$command
-                                  (string-append #$output "/bin/guix"))
+the modules, and DEPENDENCIES, a list of packages depended on.  COMMAND is the
+'guix' program to use."
+  ;; TODO: Move compiled modules to 'lib/guile' instead of 'share/guile'.
+  (computed-file name
+                 (with-imported-modules '((guix build utils))
+                   #~(begin
+                       (use-modules (guix build utils))
+                       (mkdir-p (string-append #$output "/bin"))
+                       (symlink #$command
+                                (string-append #$output "/bin/guix"))
 
-                         (let ((modules (string-append #$output
-                                                       "/share/guile/site/"
-                                                       (effective-version))))
-                           (mkdir-p (dirname modules))
-                           (symlink #$modules modules)))))))
+                       (let ((modules (string-append #$output
+                                                     "/share/guile/site/"
+                                                     (effective-version))))
+                         (mkdir-p (dirname modules))
+                         (symlink #$modules modules))))))
 
 (define* (compiled-guix source #:key (version %guix-version)
                         (pull-version 1)
@@ -423,8 +471,13 @@ the modules, and DEPENDENCIES, a list of packages depended on."
   ;; Version 1 is when we return the full package.
   (cond ((= 1 pull-version)
          ;; The whole package, with a standard file hierarchy.
-         (whole-package name built-modules dependencies
-                        #:guile-version guile-version))
+         (let ((command (guix-command built-modules
+                                      #:source source
+                                      #:dependencies dependencies
+                                      #:guile-version guile-version)))
+           (whole-package name built-modules dependencies
+                          #:command command
+                          #:guile-version guile-version)))
         ((= 0 pull-version)
          ;; Legacy 'guix pull': just return the compiled modules.
          built-modules)
-- 
2.17.0

bug#22629: [PATCH 4/4] self: Build the Info manual.

[Ludovic Courtès]

* guix/self.scm (info-manual): New procedure.
(whole-package): Add #:info and honor it.
(compiled-guix): Pass #:info.
---
 guix/self.scm | 88 +++++++++++++++++++++++++++++++++++++++++++++++++--
 1 file changed, 85 insertions(+), 3 deletions(-)

diff --git a/guix/self.scm b/guix/self.scm
index 92e69ffd5..8ed1e1073 100644
--- a/guix/self.scm
+++ b/guix/self.scm
@@ -230,6 +230,81 @@ DOMAIN, a gettext domain."
   (computed-file (string-append "guix-locale-" domain)
                  build))
 
+(define (info-manual source)
+  "Return the Info manual built from SOURCE."
+  (define texinfo
+    (module-ref (resolve-interface '(gnu packages texinfo))
+                'texinfo))
+
+  (define graphviz
+    (module-ref (resolve-interface '(gnu packages graphviz))
+                'graphviz))
+
+  (define build
+    ;; TODO: Don't depend on all of SOURCE to avoid rebuilds.
+    (with-imported-modules '((guix build utils))
+      #~(begin
+          (use-modules (guix build utils))
+
+          (mkdir #$output)
+
+          ;; Create 'version.texi'.
+          ;; XXX: Can we use a more meaningful version string yet one that
+          ;; doesn't change at each commit?
+          (call-with-output-file "version.texi"
+            (lambda (port)
+              (let ((version "(Git snapshot)"))
+                (format port "
+@set UPDATED 1 January 1970
+@set UPDATED-MONTH January 1970
+@set EDITION ~a
+@set VERSION ~a\n" version version))))
+
+          ;; Copy configuration templates that the manual includes.
+          (for-each (lambda (template)
+                      (copy-file template
+                                 (string-append
+                                  "os-config-"
+                                  (basename template ".tmpl")
+                                  ".texi")))
+                    (find-files #$source "\\.tmpl$"))
+
+          ;; Build graphs.
+          (mkdir-p (string-append #$output "/images"))
+          (for-each (lambda (dot-file)
+                      (invoke #+(file-append graphviz "/bin/dot")
+                              "-Tpng" "-Gratio=.9" "-Gnodesep=.005"
+                              "-Granksep=.00005" "-Nfontsize=9"
+                              "-Nheight=.1" "-Nwidth=.1"
+                              "-o" (string-append #$output "/images/"
+                                                  (basename dot-file ".dot")
+                                                  ".png")
+                              dot-file))
+                    (find-files (string-append #$source "/doc/images")
+                                "\\.dot$"))
+
+          ;; Copy other PNGs.
+          (for-each (lambda (png-file)
+                      (install-file png-file
+                                    (string-append #$output "/images")))
+                    (find-files (string-append #$source "/doc/images")
+                                "\\.png$"))
+
+          ;; Finally build the manual.  Copy it the Texinfo files to $PWD and
+          ;; add a symlink to the 'images' directory so that 'makeinfo' can
+          ;; see those images and produce image references in the Info output.
+          (copy-recursively (string-append #$source "/doc") "."
+                            #:log (%make-void-port "w"))
+          (delete-file-recursively "images")
+          (symlink (string-append #$output "/images") "images")
+          (invoke #+(file-append texinfo "/bin/makeinfo")
+                  "guix.texi"
+                  "-I" (string-append #$source "/doc")
+                  "-I" "."
+                  "-o" (string-append #$output "/guix.info")))))
+
+  (computed-file "guix-manual" build))
+
 (define* (guix-command modules #:key source (dependencies '())
                        (guile-version (effective-version)))
   "Return the 'guix' command such that it adds MODULES and DEPENDENCIES in its
@@ -275,12 +350,13 @@ load path."
 (define* (whole-package name modules dependencies
                         #:key
                         (guile-version (effective-version))
+                        info
                         (command (guix-command modules
                                                #:dependencies dependencies
                                                #:guile-version guile-version)))
   "Return the whole Guix package NAME that uses MODULES, a derivation of all
 the modules, and DEPENDENCIES, a list of packages depended on.  COMMAND is the
-'guix' program to use."
+'guix' program to use; INFO is the Info manual."
   ;; TODO: Move compiled modules to 'lib/guile' instead of 'share/guile'.
   (computed-file name
                  (with-imported-modules '((guix build utils))
@@ -292,9 +368,14 @@ the modules, and DEPENDENCIES, a list of packages depended on.  COMMAND is the
 
                        (let ((modules (string-append #$output
                                                      "/share/guile/site/"
-                                                     (effective-version))))
+                                                     (effective-version)))
+                             (info    #$info))
                          (mkdir-p (dirname modules))
-                         (symlink #$modules modules))))))
+                         (symlink #$modules modules)
+                         (when info
+                           (symlink #$info
+                                    (string-append #$output
+                                                   "/share/info"))))))))
 
 (define* (compiled-guix source #:key (version %guix-version)
                         (pull-version 1)
@@ -477,6 +558,7 @@ the modules, and DEPENDENCIES, a list of packages depended on.  COMMAND is the
                                       #:guile-version guile-version)))
            (whole-package name built-modules dependencies
                           #:command command
+                          #:info (info-manual source)
                           #:guile-version guile-version)))
         ((= 0 pull-version)
          ;; Legacy 'guix pull': just return the compiled modules.
-- 
2.17.0

bug#22629: [PATCH 0/4] 'guix pull' produces a self-contained Guix

[Thompson, David]

Hey Ludo,

On Thu, May 31, 2018 at 10:43 AM, Ludovic Courtès <ludo@gnu.org> wrote:
> Hello Guix!
>
> Here is the “new” ‘guix pull’ that we discussed notably in this thread:
>
>   https://bugs.gnu.org/22629
>
> The major difference is that instead of just building a bunch of modules
> and putting them under ~/.config/guix/latest, it now produces a
> standalone package (with bin/guix, share/info/guix.info, etc.) and puts
> it in a profile under ~/.config/guix/current.

Just dropping by to say this is a really awesome improvement to the
user experience! Thanks for working on it! Excited to try it out once
it gets merged.

- Dave

bug#22629: [PATCH 0/4] 'guix pull' produces a self-contained Guix

[Konrad Hinsen]

Hi Ludo,

> The major difference is that instead of just building a bunch of modules
> and putting them under ~/.config/guix/latest, it now produces a
> standalone package (with bin/guix, share/info/guix.info, etc.) and puts
> it in a profile under ~/.config/guix/current.  Quoth the manual:

This sounds like a very nice improvement, thanks!

Just wondering: does this mean that I could substitute Guix from my local
source tree simply by doing

   ./pre-inst-env guix package -p ~/.config/guix/current -i guix

Or, in fact, not use ~/.config/guix/current at all and just install
"guix" into my main profile?

Konrad.

bug#22629: [PATCH 0/4] 'guix pull' produces a self-contained Guix

[Ricardo Wurmus]

> Here is the “new” ‘guix pull’ that we discussed notably in this thread:
>
>   https://bugs.gnu.org/22629
>
> The major difference is that instead of just building a bunch of modules
> and putting them under ~/.config/guix/latest, it now produces a
> standalone package (with bin/guix, share/info/guix.info, etc.) and puts
> it in a profile under ~/.config/guix/current.  […]

This is beautiful!  Thank you so much for taking the time to implement
this.

>
>      The result of running ‘guix pull’ is a “profile” available under
>   ‘~/.config/guix/current’ containing the latest Guix.  Thus, make sure to
>   add it to the beginning of your search path so that you use the latest
>   version, and similarly for the Info manual (*note Documentation::):
>
>        export PATH="$HOME/.config/guix/current/bin:$PATH"
>        export INFOPATH="$HOME/.config/guix/current/share/info:$INFOPATH"

As a profile it will have its very own “etc/profile” file.  I suppose
that doesn’t include INFOPATH, though, because it only contains an Info
manual but not an Info reader.  It would be extra nice if we could
simplify this initial setup even more.

>      This ‘~/.config/guix/current’ profile works like any other profile
>   created by ‘guix package’ (*note Invoking guix package::).  That is, you
>   can list generations, roll back to the previous generation—i.e., the
>   previous Guix—and so on:
>
>        $ guix package -p ~/.config/guix/current -l
>        Generation 1	May 25 2018 10:06:41
>          guix	221951a	out	/gnu/store/i4dfk7vw5k112s49jrhl6hwsfnh6wr7l-guix-221951af4
>
>        Generation 2	May 27 2018 19:07:47
>         + guix	2fbae00	out	/gnu/store/44cv9hyvxg34xf5kblf5dz57hc52y4bm-guix-2fbae006f
>         - guix	221951a	out	/gnu/store/i4dfk7vw5k112s49jrhl6hwsfnh6wr7l-guix-221951af4
>
>        Generation 3	May 30 2018 16:11:39	(current)
>         + guix	a076f19	out	/gnu/store/332czkicwwg6lc3x4aqbw5q2mq12s7fj-guix-a076f1990
>         - guix	2fbae00	out	/gnu/store/44cv9hyvxg34xf5kblf5dz57hc52y4bm-guix-2fbae006f
>        $ guix package -p ~/.config/guix/current --roll-back
>        switched from generation 3 to 2

This also means that you could remove the “guix” package and install
“hello” instead.  If a user did that they would lose their variant of
guix and they’d fall back to whichever version is installed on the
system (if any).  They could still roll back manually by changing the
symlink.

They could also think that installing the “guix” package into that
profile would be a good idea — but then they would end up with a
slightly older version of Guix.  (This is already possible, of course,
but if we have a separate profile that’s intended just for Guix but with
generic properties, this could become confusing.)

I’m just thinking out loud about how users could get into trouble :)

> There are two requirements it fulfills in terms of compatibility:
>
>   1. The modified ‘build-aux/build-self.scm’ still does the right thing
>      when evaluated by an “old” Guix—that is, it produces a bunch of
>      modules for use in ~/.config/guix/latest as before.

Excellent!  Thanks for thinking about this case.

>   2. The modified ‘guix pull’ produces ~/.config/guix/current even when
>      invoked on a commit of a past Guix.  That is, it automatically
>      produces a ‘guix’ command using the modules returned by the old
>      ‘build-self.scm’.
>
> There are various improvements we can make from there.  For example,
> using “manifest entry properties” as proposed in
> <https://bugs.gnu.org/31442>, we can attach meta-data (commit ID, repo
> URL, etc.) in each manifest entry that ‘guix pull’ populates; then we
> can arrange for ‘guix pull --list-generations’ (say) to display that
> information.
>
> We could add ‘guix pull’ options for convenient: ‘--roll-back’,
> ‘--profile’, etc.
>
> Going forward, additional “channels” could be presented as entries in
> the ~/.config/guix/current manifest.
>
> Caveats:
>
>   1. The ~/.config/guix/current profile really lives there.  That is,
>      unlike ~/.guix-profile, it’s not in /var/guix/profiles/per-user.
>      That could be an issue for cluster setups where home directories
>      are not scanned by the Guix GC.  Cluster folks, please tell me!

Is it impossible to store it in localstatedir?

In practice, cluster installations don’t really run “guix gc” all that
ofter (if ever), so it may not be a problem.

>   3. C++ code is not built.  I wonder which will come first: getting rid
>      of the C++ code, or building it?  :-)

I’d say that this is a feature ;)

--
Ricardo

bug#22629: [PATCH 0/4] 'guix pull' produces a self-contained Guix

[Ludovic Courtès]

Hello!

Ricardo Wurmus <rekado@elephly.net> skribis:

>>      The result of running ‘guix pull’ is a “profile” available under
>>   ‘~/.config/guix/current’ containing the latest Guix.  Thus, make sure to
>>   add it to the beginning of your search path so that you use the latest
>>   version, and similarly for the Info manual (*note Documentation::):
>>
>>        export PATH="$HOME/.config/guix/current/bin:$PATH"
>>        export INFOPATH="$HOME/.config/guix/current/share/info:$INFOPATH"
>
> As a profile it will have its very own “etc/profile” file.  I suppose
> that doesn’t include INFOPATH, though, because it only contains an Info
> manual but not an Info reader.  It would be extra nice if we could
> simplify this initial setup even more.

I agree.  On GuixSD of course we can add the right default.  On foreign
distros, I’m not sure how we can simplify things though.

At least, when you run the new ‘guix pull’ for the first time, you get a
note that you should define PATH.

>>      This ‘~/.config/guix/current’ profile works like any other profile
>>   created by ‘guix package’ (*note Invoking guix package::).  That is, you
>>   can list generations, roll back to the previous generation—i.e., the
>>   previous Guix—and so on:

[...]

> This also means that you could remove the “guix” package and install
> “hello” instead.  If a user did that they would lose their variant of
> guix and they’d fall back to whichever version is installed on the
> system (if any).  They could still roll back manually by changing the
> symlink.
>
> They could also think that installing the “guix” package into that
> profile would be a good idea — but then they would end up with a
> slightly older version of Guix.  (This is already possible, of course,
> but if we have a separate profile that’s intended just for Guix but with
> generic properties, this could become confusing.)
>
> I’m just thinking out loud about how users could get into trouble :)

Yeah well, we can’t really prevent them from shooting themselves in the
foot.  :-)

Now, we could hide the fact that people can use “guix package -p” on
that thing by simply providing ‘--list-generations’, ‘--roll-back’,
etc. directly in ‘guix pull’.

I think it’ll be a good idea to add a custom ‘--list-generations’ in the
future anyway, because it could display more useful information (commit
IDs, number of available packages, etc.)

>>   1. The ~/.config/guix/current profile really lives there.  That is,
>>      unlike ~/.guix-profile, it’s not in /var/guix/profiles/per-user.
>>      That could be an issue for cluster setups where home directories
>>      are not scanned by the Guix GC.  Cluster folks, please tell me!
>
> Is it impossible to store it in localstatedir?

It wouldn’t be hard, but I wanted to get things done quickly.  :-)

Thanks for your feedback!

Ludo’.

bug#22629: [PATCH 0/4] 'guix pull' produces a self-contained Guix

[Ludovic Courtès]

Heya David,

"Thompson, David" <dthompson2@worcester.edu> skribis:

> On Thu, May 31, 2018 at 10:43 AM, Ludovic Courtès <ludo@gnu.org> wrote:
>> Hello Guix!
>>
>> Here is the “new” ‘guix pull’ that we discussed notably in this thread:
>>
>>   https://bugs.gnu.org/22629
>>
>> The major difference is that instead of just building a bunch of modules
>> and putting them under ~/.config/guix/latest, it now produces a
>> standalone package (with bin/guix, share/info/guix.info, etc.) and puts
>> it in a profile under ~/.config/guix/current.
>
> Just dropping by to say this is a really awesome improvement to the
> user experience! Thanks for working on it! Excited to try it out once
> it gets merged.

Thanks for your support!  We’ll still have to make the whole build
process faster but yeah, that’s one important step towards 1.0!

Ludo’.

bug#22629: [PATCH 0/4] 'guix pull' produces a self-contained Guix

[Pjotr Prins]

On Fri, Jun 01, 2018 at 02:13:10PM +0200, Ludovic Courtès wrote:
> Thanks for your support!  We’ll still have to make the whole build
> process faster but yeah, that’s one important step towards 1.0!

But we do get substitutes in principle, right?

Pj.

bug#22629: [PATCH 0/4] 'guix pull' produces a self-contained Guix

[Ludovic Courtès]

Pjotr Prins <pjotr.public12@thebird.nl> skribis:

> On Fri, Jun 01, 2018 at 02:13:10PM +0200, Ludovic Courtès wrote:
>> Thanks for your support!  We’ll still have to make the whole build
>> process faster but yeah, that’s one important step towards 1.0!
>
> But we do get substitutes in principle, right?

Yes, but only for berlin.guixsd.org at the moment (because of obscure
issues with Guile support in Hydra that I’m not really willing to fix):

  https://debbugs.gnu.org/cgi/bugreport.cgi?bug=27284#147

Ludo’.

bug#22629: [PATCH 0/4] 'guix pull' produces a self-contained Guix

[Ludovic Courtès]

Hello!

Konrad Hinsen <konrad.hinsen@fastmail.net> skribis:

> Just wondering: does this mean that I could substitute Guix from my local
> source tree simply by doing
>
>    ./pre-inst-env guix package -p ~/.config/guix/current -i guix

That would just install the snapshot that the ‘guix’ package refers to
(it’s defined in (gnu packages package-management)).)

> Or, in fact, not use ~/.config/guix/current at all and just install
> "guix" into my main profile?

If we add a ‘-p’ option to ‘guix pull’, that’s something you could do.

Now, installing Guix in your main profile is probably not a good idea.
Since Guix it what allows you to manage the profile in question, it’s
probably better to keep it separate.  That way, ‘guix’ is managed
independently of upgrades, installs, etc. that you make in your profile;
rolling back your main profile doesn’t roll back Guix, and so on.

HTH,
Ludo’.

bug#22629: [PATCH 0/4] 'guix pull' produces a self-contained Guix

[Pjotr Prins]

On Sun, Jun 03, 2018 at 10:29:43PM +0200, Ludovic Courtès wrote:
> Pjotr Prins <pjotr.public12@thebird.nl> skribis:
> 
> > On Fri, Jun 01, 2018 at 02:13:10PM +0200, Ludovic Courtès wrote:
> >> Thanks for your support!  We’ll still have to make the whole build
> >> process faster but yeah, that’s one important step towards 1.0!
> >
> > But we do get substitutes in principle, right?
> 
> Yes, but only for berlin.guixsd.org at the moment (because of obscure
> issues with Guile support in Hydra that I’m not really willing to fix):

That is great news Ludo. It also means that channels are around the
corner because we can use a substituted version of Guix as the basis.

That is unbelievably cool.

Pj.

bug#22629: [PATCH 0/4] 'guix pull' produces a self-contained Guix

[Konrad Hinsen]

Hi Ludo,

>> Just wondering: does this mean that I could substitute Guix from my local
>> source tree simply by doing
>>
>>    ./pre-inst-env guix package -p ~/.config/guix/current -i guix
>
> That would just install the snapshot that the ‘guix’ package refers to
> (it’s defined in (gnu packages package-management)).)

Fine, so if I run update-guix-package.scm and then do the install, I get
what I expect, right?

I am looking for a reasonably straightforward way to have everything in
my Guix universe (the guix command line tool, emacs-guix, guile, geiser,
...) consistently use my personal version of the package definitions.

> Now, installing Guix in your main profile is probably not a good idea.
> Since Guix it what allows you to manage the profile in question, it’s
> probably better to keep it separate.  That way, ‘guix’ is managed

Yes, that makes sense.

Konrad.

bug#22629: [PATCH 0/4] 'guix pull' produces a self-contained Guix

[Fis Trivial]

Hi, first of all, thanks for the improvement. It's really exciting know
progress in guix.

But I have a few questions around this change. Just curiosity. :)

> Hello Guix!
>
> Here is the “new” ‘guix pull’ that we discussed notably in this thread:
>
>   https://bugs.gnu.org/22629
>
> The major difference is that instead of just building a bunch of modules
> and putting them under ~/.config/guix/latest, it now produces a
> standalone package (with bin/guix, share/info/guix.info, etc.) and puts
> it in a profile under ~/.config/guix/current.  Quoth the manual:
>
>      The result of running ‘guix pull’ is a “profile” available under
>   ‘~/.config/guix/current’ containing the latest Guix.  Thus, make sure to
>   add it to the beginning of your search path so that you use the latest
>   version, and similarly for the Info manual (*note Documentation::):
>
>        export PATH="$HOME/.config/guix/current/bin:$PATH"
>        export INFOPATH="$HOME/.config/guix/current/share/info:$INFOPATH"
>

This sounds like something could be done by guix itself, choosing the
right profile path by $HOME. Since guix has absolute control about this
piece of information?

>      This ‘~/.config/guix/current’ profile works like any other profile
>   created by ‘guix package’ (*note Invoking guix package::).  That is, you
>   can list generations, roll back to the previous generation—i.e., the
>   previous Guix—and so on:
>
>        $ guix package -p ~/.config/guix/current -l
>        Generation 1	May 25 2018 10:06:41
>          guix	221951a	out	/gnu/store/i4dfk7vw5k112s49jrhl6hwsfnh6wr7l-guix-221951af4
>
>        Generation 2	May 27 2018 19:07:47
>         + guix	2fbae00	out	/gnu/store/44cv9hyvxg34xf5kblf5dz57hc52y4bm-guix-2fbae006f
>         - guix	221951a	out	/gnu/store/i4dfk7vw5k112s49jrhl6hwsfnh6wr7l-guix-221951af4
>
>        Generation 3	May 30 2018 16:11:39	(current)
>         + guix	a076f19	out	/gnu/store/332czkicwwg6lc3x4aqbw5q2mq12s7fj-guix-a076f1990
>         - guix	2fbae00	out	/gnu/store/44cv9hyvxg34xf5kblf5dz57hc52y4bm-guix-2fbae006f
>        $ guix package -p ~/.config/guix/current --roll-back
>        switched from generation 3 to 2
>
>> There are two requirements it fulfills in terms of compatibility:
>
>   1. The modified ‘build-aux/build-self.scm’ still does the right thing
>      when evaluated by an “old” Guix—that is, it produces a bunch of
>      modules for use in ~/.config/guix/latest as before.
>
>   2. The modified ‘guix pull’ produces ~/.config/guix/current even when
>      invoked on a commit of a past Guix.  That is, it automatically
>      produces a ‘guix’ command using the modules returned by the old
>      ‘build-self.scm’.
>
>
> We could add ‘guix pull’ options for convenient: ‘--roll-back’,
> ‘--profile’, etc.
>

Nice~

> Going forward, additional “channels” could be presented as entries in
> the ~/.config/guix/current manifest.
>
> Caveats:
>
>   1. The ~/.config/guix/current profile really lives there.  That is,
>      unlike ~/.guix-profile, it’s not in /var/guix/profiles/per-user.
>      That could be an issue for cluster setups where home directories
>      are not scanned by the Guix GC.  Cluster folks, please tell me!

What does that mean? We have a guix directory under $HOME/.config,
inside there's a symlink to /gnu/store/...-guix-<commit>. Does "really
lives there" mean the new profile is not a symlink but a concrete
directory or hard link?
>
>   2. The translated Info manual is not built.  Julien: could you turn
>      the big ‘xref_command’ in a script or something that we can more
>      easily reuse in (guix self)?
>
>   3. C++ code is not built.  I wonder which will come first: getting rid
>      of the C++ code, or building it?  :-)
>
In the future world, how do we update guix daemon? Is't still running
guix pull && guix package -u under root user?

bug#22629: [PATCH 0/4] 'guix pull' produces a self-contained Guix

[Ludovic Courtès]

Hello Konrad,

Konrad Hinsen <konrad.hinsen@fastmail.net> skribis:

>>> Just wondering: does this mean that I could substitute Guix from my local
>>> source tree simply by doing
>>>
>>>    ./pre-inst-env guix package -p ~/.config/guix/current -i guix
>>
>> That would just install the snapshot that the ‘guix’ package refers to
>> (it’s defined in (gnu packages package-management)).)
>
> Fine, so if I run update-guix-package.scm and then do the install, I get
> what I expect, right?

Yes (even with current ‘master’), but it’s quite heavyweight since you
end up recompiling all of Guix.

> I am looking for a reasonably straightforward way to have everything in
> my Guix universe (the guix command line tool, emacs-guix, guile, geiser,
> ...) consistently use my personal version of the package definitions.

Would ‘guix package --manifest’ suffice?  Perhaps along with ‘guix pull
--commit=XXX’?

In the not-too-distant future I think we can add some sort of manifest
support for ‘guix pull’ such that you can store the list of channel URLs
and commit IDs in one file and instantiate that.

Ludo’.

bug#22629: [PATCH 0/4] 'guix pull' produces a self-contained Guix

[Ludovic Courtès]

Hello!

Fis Trivial <ybbs.daans@hotmail.com> skribis:

>>      The result of running ‘guix pull’ is a “profile” available under
>>   ‘~/.config/guix/current’ containing the latest Guix.  Thus, make sure to
>>   add it to the beginning of your search path so that you use the latest
>>   version, and similarly for the Info manual (*note Documentation::):
>>
>>        export PATH="$HOME/.config/guix/current/bin:$PATH"
>>        export INFOPATH="$HOME/.config/guix/current/share/info:$INFOPATH"
>>
>
> This sounds like something could be done by guix itself, choosing the
> right profile path by $HOME. Since guix has absolute control about this
> piece of information?

I think it would be bad for Guix to modify your ~/.bashrc directly, so
it’s better to let users do that.

GuixSD will have the right PATH and INFOPATH by default, though.

>> Caveats:
>>
>>   1. The ~/.config/guix/current profile really lives there.  That is,
>>      unlike ~/.guix-profile, it’s not in /var/guix/profiles/per-user.
>>      That could be an issue for cluster setups where home directories
>>      are not scanned by the Guix GC.  Cluster folks, please tell me!
>
> What does that mean? We have a guix directory under $HOME/.config,
> inside there's a symlink to /gnu/store/...-guix-<commit>. Does "really
> lives there" mean the new profile is not a symlink but a concrete
> directory or hard link?

Please see how ~/.guix-profile and
/var/guix/profiles/per-user/$USER/guix-profile work together.  Basically
generations show up in /var/guix/profiles, whereas ~/.guix-profile is a
fixed symlink.

>>   3. C++ code is not built.  I wonder which will come first: getting rid
>>      of the C++ code, or building it?  :-)
>>
> In the future world, how do we update guix daemon? Is't still running
> guix pull && guix package -u under root user?

In the future world, ‘guix pull’ updates everything: client-side and
daemon.  Currently it’s still client-side only.

HTH!

Ludo’.

bug#22629: [PATCH 0/4] 'guix pull' produces a self-contained Guix

[Konrad Hinsen]

Hi Ludo,

>> Fine, so if I run update-guix-package.scm and then do the install, I get
>> what I expect, right?
>
> Yes (even with current ‘master’), but it’s quite heavyweight since you
> end up recompiling all of Guix.

Not great, but doable from time to time.

>> I am looking for a reasonably straightforward way to have everything in
>> my Guix universe (the guix command line tool, emacs-guix, guile, geiser,
>> ...) consistently use my personal version of the package definitions.
>
> Would ‘guix package --manifest’ suffice?  Perhaps along with ‘guix pull
> --commit=XXX’?

'guix pull --commit=XXX' would be of use if it works with an URL
pointing to a local repository (haven't tried yet). I'd also
have to point GUILE_LOAD_PATH to wherever 'guix pull' stores the source
code tree, which looks doable as well if that's a stable location.

> In the not-too-distant future I think we can add some sort of manifest
> support for ‘guix pull’ such that you can store the list of channel URLs
> and commit IDs in one file and instantiate that.

That sounds really nice :-)

Thanks for all the good stuff BTW!

Konrad.

bug#22629: [PATCH 0/4] 'guix pull' produces a self-contained Guix

[Ludovic Courtès]

Konrad Hinsen <konrad.hinsen@fastmail.net> skribis:

>>> I am looking for a reasonably straightforward way to have everything in
>>> my Guix universe (the guix command line tool, emacs-guix, guile, geiser,
>>> ...) consistently use my personal version of the package definitions.
>>
>> Would ‘guix package --manifest’ suffice?  Perhaps along with ‘guix pull
>> --commit=XXX’?
>
> 'guix pull --commit=XXX' would be of use if it works with an URL
> pointing to a local repository (haven't tried yet). I'd also
> have to point GUILE_LOAD_PATH to wherever 'guix pull' stores the source
> code tree, which looks doable as well if that's a stable location.

You can already run “guix pull --commit=X --url=$PWD” or similar, and
modules currently end up ~/.config/guix/latest.

Cheers,
Ludo’.

bug#22629: [PATCH 0/4] 'guix pull' produces a self-contained Guix

[Fis Trivial]

>
>>>      The result of running ‘guix pull’ is a “profile” available under
>>>   ‘~/.config/guix/current’ containing the latest Guix.  Thus, make sure to
>>>   add it to the beginning of your search path so that you use the latest
>>>   version, and similarly for the Info manual (*note Documentation::):
>>>
>>>        export PATH="$HOME/.config/guix/current/bin:$PATH"
>>>        export INFOPATH="$HOME/.config/guix/current/share/info:$INFOPATH"
>>>
>>
>> This sounds like something could be done by guix itself, choosing the
>> right profile path by $HOME. Since guix has absolute control about this
>> piece of information?
>
> I think it would be bad for Guix to modify your ~/.bashrc directly, so
> it’s better to let users do that.
>

Well, I mean a wrapper script looks similar to this in /usr/bin or
/usr/local/bin:

      #!/path/to/store/bash
      # A wrapper named guix lives in /usr/bin/ or /usr/local/bin
      exec $HOME/.config/guix/current/bin/guix "$@"

I might be wrong, please forgive my poor knowledge.

> GuixSD will have the right PATH and INFOPATH by default, though.
>
>>> Caveats:
>>>
>>>   1. The ~/.config/guix/current profile really lives there.  That is,
>>>      unlike ~/.guix-profile, it’s not in /var/guix/profiles/per-user.
>>>      That could be an issue for cluster setups where home directories
>>>      are not scanned by the Guix GC.  Cluster folks, please tell me!
>>
>> What does that mean? We have a guix directory under $HOME/.config,
>> inside there's a symlink to /gnu/store/...-guix-<commit>. Does "really
>> lives there" mean the new profile is not a symlink but a concrete
>> directory or hard link?
>
> Please see how ~/.guix-profile and
> /var/guix/profiles/per-user/$USER/guix-profile work together.  Basically
> generations show up in /var/guix/profiles, whereas ~/.guix-profile is a
> fixed symlink.
>

Thanks for the explanation, but running $ls shows:

lrwxrwxrwx. 1 fis fis 44 Mar 28 16:41 .guix-profile -> /var/guix/profiles/per-user/fis/guix-profile

You are talking about the same directory
"/var/guix/profiles/per-user/$USER/guix-profile" and "~/.guix-profile".

That's fine, I am waiting for the new feature to land home in my
device anyway. I will inspect more closely then.

>>>   3. C++ code is not built.  I wonder which will come first: getting rid
>>>      of the C++ code, or building it?  :-)
>>>
>> In the future world, how do we update guix daemon? Is't still running
>> guix pull && guix package -u under root user?
>
> In the future world, ‘guix pull’ updates everything: client-side and
> daemon.  Currently it’s still client-side only.
>

That sounds really nice! Thanks!

> HTH!
>

Interesting, you never run out of acronym :). If it means "hope this
help", sure, thanks for the reply.

> Ludo’.

bug#22629: [PATCH 0/4] 'guix pull' produces a self-contained Guix

[Ludovic Courtès]

Hello Guix!

Ludovic Courtès <ludo@gnu.org> skribis:

> Here is the “new” ‘guix pull’ that we discussed notably in this thread:
>
>   https://bugs.gnu.org/22629
>
> The major difference is that instead of just building a bunch of modules
> and putting them under ~/.config/guix/latest, it now produces a
> standalone package (with bin/guix, share/info/guix.info, etc.) and puts
> it in a profile under ~/.config/guix/current.  Quoth the manual:

I have just pushed the new ‘guix pull’, with a few improvements compared
to the patches I had sent:

  • Both guix.info and all of guix.LANG.info are now built.

  • The derivation that builds locale data depends only on the po/
    directory, so it won’t be rebuilt every time.  Likewise for the
    derivation that builds the manual.

  • Meta-data about what was pulled is kept in manifest entries, using
    the ‘properties’ discussed in <https://bugs.gnu.org/31442>.
    Currently the UI doesn’t use it but you can see that info in
    ~/.config/guix/current/manifest.

Let me know how it goes!

Ludo’.

bug#22629: bug#28471: closed (Re: bug#22629: [PATCH 0/4] 'guix pull' produces a self-contained Guix)

[Ludovic Courtès]

Hello!

>   • Meta-data about what was pulled is kept in manifest entries, using
>     the ‘properties’ discussed in <https://bugs.gnu.org/31442>.
>     Currently the UI doesn’t use it but you can see that info in
>     ~/.config/guix/current/manifest.

Commit e2f8be0664609223369f01290b69b44196783ab3 adds ‘guix pull -l’,
which produces output like this:

--8<---------------cut here---------------start------------->8---
Generation 1	Jun 10 2018 00:18:18
  guix 65956ad
    repository URL: https://git.savannah.gnu.org/git/guix.git
    branch: origin/master
    commit: 65956ad3526ba09e1f7a40722c96c6ef7c0936fe

Generation 2	Jun 11 2018 11:02:49
  guix e0cc7f6
    repository URL: https://git.savannah.gnu.org/git/guix.git
    branch: origin/master
    commit: e0cc7f669bec22c37481dd03a7941c7d11a64f1d

Generation 3	Jun 13 2018 23:31:07	(current)
  guix 844cc1c
    repository URL: https://git.savannah.gnu.org/git/guix.git
    branch: origin/master
    commit: 844cc1c8f394f03b404c5bb3aee086922373490c
--8<---------------cut here---------------end--------------->8---

In the future it could also display information such as the number of
packages available, the number of added/upgraded packages, and so on.

Ludo’.

bug#22629: [PATCH 0/4] 'guix pull' produces a self-contained Guix

[Chris Marusich]

[-- Attachment #1: Type: text/plain, Size: 1149 bytes --]

Hi Ludo,

Ludovic Courtès <ludo@gnu.org> writes:

> Here is the “new” ‘guix pull’ that we discussed notably in this thread:
>
>   https://bugs.gnu.org/22629

Although I've been enjoying the new guix pull for a while now, I just
today found the time to take a peek at these code changes.  It's always
educational to see how you write your code!

I was surprised that "guix pull" doesn't build the guix package via the
usual mechanisms (e.g., the way it would be built if I ran "guix build
guix").  The new "guix pull" code builds a profile, so it seems like we
could put packages in there (e.g., a guix package that inherits from the
original but replaces the origin with a Git checkout).  However, instead
of re-using the build logic encapsulated in the guix package, it looks
like we build Guix piece by piece using custom build logic in (guix
self).  Why do we do that?

These improvements are great!  I'm just curious about the motivation for
the custom build logic.  I'm guessing there is some sort of complication
to simply using the guix package that is non-obvious, but I can't see
what it is.

-- 
Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

bug#22629: [PATCH 0/4] 'guix pull' produces a self-contained Guix

[Ludovic Courtès]

Hello Chris,

Thanks for your feedback!

Chris Marusich <cmmarusich@gmail.com> skribis:

> I was surprised that "guix pull" doesn't build the guix package via the
> usual mechanisms (e.g., the way it would be built if I ran "guix build
> guix").  The new "guix pull" code builds a profile, so it seems like we
> could put packages in there (e.g., a guix package that inherits from the
> original but replaces the origin with a Git checkout).  However, instead
> of re-using the build logic encapsulated in the guix package, it looks
> like we build Guix piece by piece using custom build logic in (guix
> self).  Why do we do that?

There’s a couple of problems that we’re trying to fix, which were
discussed at <https://debbugs.gnu.org/cgi/bugreport.cgi?bug=27284#133>:

  1. Bootstrapping: how do we built a new Guix from an already-installed,
     older Guix?

  2. Efficiency: how do we avoid recompiling every since Scheme file at
     each revision?

The answer to #2 is, of course, to have a finer-grain view on what needs
to be built.  We have makefiles that have view at the file level, and
here we essentially want to do what makefiles do, but we also want to
introduce dependencies (our makefiles do not declare dependencies among
Scheme files, which sometimes lead to those infamous ABI issues.)

We could have defined package objects to describe each of the various
subparts of Guix: a “guix-core” package, a “guix-packages” package, etc.
But it’s not a good fit: these aren’t really packages, there’s no UI to
access them as such anyway, and we’re closer to the abstraction level of
a makefile than to the abstraction level of a package.

So I chose to define a custom <node> type in (guix self).  Each node
represents a group of files, in particular Scheme files that need to be
compiled; nodes can have edges (the ‘dependencies’ field).  On top of
that there are helper procedures to create and manipulate nodes.  For
instance, ‘scheme-nodes’ creates a node that builds the closure of a
list of .scm files, minus the files already built by its dependencies.

The end result is that the full Guix consists of half a dozen of Scheme
derivations.  Derivations at the root of the DAG (e.g., “guix-core”)
hopefully change quite infrequently, and as such we don’t have to
rebuild everything at each revision.

As I wrote before in the issue above, there’s room for improvement.  In
particular we could split nodes further.

I hope this clarifies things!

Ludo’.

bug#22629: Channels!

[Ludovic Courtès]

Hi Guix!

ludo@gnu.org (Ludovic Courtès) skribis:

>   • Have a “channel” mechanism, similar to ‘nix-channel’, that would
>     allow users to have several Guix variants available in parallel
>     instead of just “latest”.  Could work like this:
>
>       guix channel add latest git://git.sv.gnu.org/guix.git master
>       guix channel add stable git://git.sv.gnu.org/guix.git stable
>       guix channel pull latest
>       guix channel set latest
>       # here i see the latest versions of everything
>       guix channel set stable
>       # and here everything is old but super stable ;-)

The patches that follow implement this last bit, though in a slightly
different way.  Users would now have the option to provide
~/.config/guix/channels.scm along these lines:

  (cons (channel
         (name 'guix-hpc)
         (url "https://gitlab.inria.fr/guix-hpc/guix-hpc.git")
         (branch "origin/master"))
        %default-channels)

‘guix pull’ then pulls both from ‘%default-channels’, a one-element list
containing our Savannah repo, and from that ‘guix-hpc’ repo.  It builds
both Guix itself and the modules in ‘guix-hpc’, with a dependency on the
just-built Guix.  The ~/.config/guix/current profile then contains two
entries:

--8<---------------cut here---------------start------------->8---
$ guix pull -l
[…]
Generacio 18	Aug 27 2018 09:54:40
  guix 1ca5499
    repository URL: https://git.savannah.gnu.org/git/guix.git
    branch: origin/master
    commit: 1ca54999db34b0197e6c7fa3f0a852fc0d193e70
  10 new packages: avogadro, emacs-git-annex, find-circ, guile-aa-tree, lepton-eda, r-a3,
    r-abcanalysis, r-manipulatewidget, r-miniui, r-slam
  43 packages upgraded: byobu@5.127, cpupower@4.18.5, darcs@2.14.1, drumkv1@0.9.2, duplicity@0.7.18,
    emacs-recent-addresses@0.1-1.afbbfdc, font-gnu-unifont@11.0.02, freefall@4.18.5, gdb-arm-none-eabi@8.1.1,
    gdb@8.1.1, ghc-integer-logarithms@1.0.2.1, ghc-optparse-applicative@0.14.2.0, ghc-quickcheck-instances@0.3.18,
    ghc-scientific@0.3.6.2, ghc-smallcheck@1.1.5, ghc-zlib@0.6.2, guile-lib@0.2.6.1, guile2.0-lib@0.2.6.1,
    guile2.2-lib@0.2.6.1, keepassxc@2.3.4, libgig@4.1.0, libx11@1.6.6, linux-libre-arm-generic@4.18.5,
    linux-libre-arm-omap2plus@4.18.5, linux-libre@4.18.5, ncmpc@0.29, openssh@7.8p1, padthv1@0.9.2,
    parallel@20180822, perf@4.18.5, qjackctl@0.5.3, r-car@3.0-1, r-catools@1.17.1.1, r-devtools@1.13.6,
    r-digest@0.6.16, r-dplyr@0.7.6, r-rcpp@0.12.18, r-synchronicity@1.3.5, r-trimcluster@0.1-2.1, samplv1@0.9.2,
    synthv1@0.9.2, tuxguitar@1.5.2, yubico-piv-tool@1.6.1
Generacio 19	Aug 27 2018 16:20:48
  guix d894ab8
    repository URL: file:///home/ludo/src/guix
    branch: origin/wip-channels
    commit: d894ab8e9bfabcefa6c49d9ba2e834dd5a73a300
  guix-hpc dd3df5e
    repository URL: file:///home/ludo/src/guix-hpc
    branch: origin/wip-channel-test
    commit: dd3df5e2c8818760a8fc0bd699e55d3b69fef2bb
  11 new packages: chameleon, eztrace, fxt, litl, maphys, pastix, simgrid, starpu, starpu,
    starpu-fxt, starpu-simgrid
  4 packages upgraded: capstone@3.0.5-rc2, emacs-racket-mode@0.0.2-2.1b78827,
    python-capstone@3.0.5-rc2, python2-capstone@3.0.5-rc2
--8<---------------cut here---------------end--------------->8---

GUIX_PACKAGE_PATH is no longer needed (though it’s still supported):
~/.config/guix/current/bin/guix automatically finds packages defined in
the ‘guix-hpc’ channel.

The only missing bit for this patch series is documentation for channels
(including a prominent disclaimer about API compatibility).  I’ll write
it if people agree with the overall approach.


Implementation
~~~~~~~~~~~~~~

(guix channels) implements the “channel” and “channel instance”
abstractions.  It provides tools to get the latest instance of channel
and to build a list of instances.  There must always be a channel called
‘guix’, which all other instances depend on.

(guix scripts pull) now does little more than implementing the CLI,
while (guix channels) does all the heavy lifting.


Limitations
~~~~~~~~~~~

Currently third-party channels are expected to provide nothing but
package modules.  Those package modules should live at the root of the
repository; for example, the modules/ directory at
https://gitlab.inria.fr/guix-hpc/guix-hpc doesn’t work, you have to
“git mv modules/inria .” for things to work.

Both limitations could be lifted by honoring a special meta-data file in
repositories.  But I think that can come later.

Third-party channels currently all implicitly depend on the ‘guix’
channel, but it’s not possible to express dependencies among channels.
It may be useful, but it can probably come later, too.

If a channel fails to build, the user gets a backtrace and it’s up to
them to figure out what to do.  May need to be improved.  :-)

Not really a limitation, but (guix describe) relies on argv[0] to find
channel meta-data.  I’m not a fan of that but it works in practice.
More importantly, it’s the only way I could think of to retrieve
“silent” meta-data (repo URL, commit, etc.) as well as channel
directories where packages should be searched for.


Future work
~~~~~~~~~~~

With (guix channels) we’ll be able to implement things like “open an
inferior corresponding to commit X”.  In turn, that means we can support
commit IDs in manifests, and then “guix package -m manifest.scm” will
open an inferior for the requested commit ID and instantiate the thing.

With (guix describe) we can, of course, implement a ‘guix describe’
command.  We can also improve provenance tracking, for instance by
storing as a manifest entry property the channel instance an installed
package comes from.


Thoughts?  Ricardo?

Thanks in advance for your feedback!

Ludo’.

bug#22629: [PATCH 1/3] discovery: Add 'scheme-modules*'.

[Ludovic Courtès]

* guix/self.scm (scheme-modules*): Move to...
* guix/discovery.scm (scheme-modules*): ... here.  New procedure.  Make
'sub-directory' an optional parameter.
---
 guix/discovery.scm | 13 ++++++++++++-
 guix/self.scm      |  7 -------
 2 files changed, 12 insertions(+), 8 deletions(-)

diff --git a/guix/discovery.scm b/guix/discovery.scm
index 2b627d108..3fc6e2c9e 100644
--- a/guix/discovery.scm
+++ b/guix/discovery.scm
@@ -1,5 +1,5 @@
 ;;; GNU Guix --- Functional package management for GNU
-;;; Copyright © 2012, 2013, 2014, 2015, 2016, 2017 Ludovic Courtès <ludo@gnu.org>
+;;; Copyright © 2012, 2013, 2014, 2015, 2016, 2017, 2018 Ludovic Courtès <ludo@gnu.org>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -27,6 +27,7 @@
   #:use-module (ice-9 ftw)
   #:export (scheme-files
             scheme-modules
+            scheme-modules*
             fold-modules
             all-modules
             fold-module-public-variables))
@@ -115,6 +116,16 @@ name and the exception key and arguments."
                                 (string-append directory "/" sub-directory)
                                 directory))))
 
+(define* (scheme-modules* directory #:optional sub-directory)
+  "Return the list of module names found under SUB-DIRECTORY in DIRECTORY.
+This is a source-only variant that does not try to load files."
+  (let ((prefix (string-length directory)))
+    (map (lambda (file)
+           (file-name->module-name (string-drop file prefix)))
+         (scheme-files (if sub-directory
+                           (string-append directory "/" sub-directory)
+                           directory)))))
+
 (define* (fold-modules proc init path #:key (warn (const #f)))
   "Fold over all the Scheme modules present in PATH, a list of directories.
 Call (PROC MODULE RESULT) for each module that is found."
diff --git a/guix/self.scm b/guix/self.scm
index 90649db17..81f9b0cfd 100644
--- a/guix/self.scm
+++ b/guix/self.scm
@@ -206,13 +206,6 @@ list of file-name/file-like objects suitable as inputs to 'imported-files'."
                (local-file file #:recursive? #t)))
        (find-files (string-append directory "/" sub-directory) pred)))
 
-(define (scheme-modules* directory sub-directory)
-  "Return the list of module names found under SUB-DIRECTORY in DIRECTORY."
-  (let ((prefix (string-length directory)))
-    (map (lambda (file)
-           (file-name->module-name (string-drop file prefix)))
-         (scheme-files (string-append directory "/" sub-directory)))))
-
 (define* (sub-directory item sub-directory)
   "Return SUB-DIRECTORY within ITEM, which may be a file name or a file-like
 object."
-- 
2.18.0

bug#22629: [PATCH 2/3] Add (guix describe) and use it to initialize '%package-search-path'.

[Ludovic Courtès]

* guix/describe.scm: New file.
* Makefile.am (MODULES): Add it.
* gnu/packages.scm (%package-module-path): Honor 'package-path-entries'.
---
 Makefile.am       |  1 +
 gnu/packages.scm  | 22 ++++++++------
 guix/describe.scm | 73 +++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 88 insertions(+), 8 deletions(-)
 create mode 100644 guix/describe.scm

diff --git a/Makefile.am b/Makefile.am
index 324674a60..b6efd6d62 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -86,6 +86,7 @@ MODULES =					\
   guix/derivations.scm				\
   guix/grafts.scm				\
   guix/inferior.scm				\
+  guix/describe.scm				\
   guix/gnu-maintenance.scm			\
   guix/self.scm					\
   guix/upstream.scm				\
diff --git a/gnu/packages.scm b/gnu/packages.scm
index 7b954769e..e690d459f 100644
--- a/gnu/packages.scm
+++ b/gnu/packages.scm
@@ -30,6 +30,7 @@
                 #:select ((package-name->name+version
                            . hyphen-separated-name->name+version)))
   #:autoload   (guix profiles) (packages->manifest)
+  #:use-module (guix describe)
   #:use-module (ice-9 vlist)
   #:use-module (ice-9 match)
   #:use-module (srfi srfi-1)
@@ -136,16 +137,21 @@ for system '~a'")
   ;; to narrow the search.
   (let* ((not-colon   (char-set-complement (char-set #\:)))
          (environment (string-tokenize (or (getenv "GUIX_PACKAGE_PATH") "")
-                                       not-colon)))
-    ;; Automatically add items from $GUIX_PACKAGE_PATH to Guile's search path.
-    (for-each (lambda (directory)
-                (set! %load-path (cons directory %load-path))
-                (set! %load-compiled-path
-                      (cons directory %load-compiled-path)))
-              environment)
+                                       not-colon))
+         (channels    (package-path-entries)))
+    ;; Automatically add channels and items from $GUIX_PACKAGE_PATH to Guile's
+    ;; search path.  For historical reasons, $GUIX_PACKAGE_PATH goes to the
+    ;; front; channels go to the back so that they don't override Guix' own
+    ;; modules.
+    (set! %load-path
+      (append environment %load-path channels))
+    (set! %load-compiled-path
+      (append environment %load-compiled-path channels))
 
     (make-parameter
-     (append environment `((,%distro-root-directory . "gnu/packages"))))))
+     (append environment
+             `((,%distro-root-directory . "gnu/packages"))
+             channels))))
 
 (define %patch-path
   ;; Define it after '%package-module-path' so that '%load-path' contains user
diff --git a/guix/describe.scm b/guix/describe.scm
new file mode 100644
index 000000000..3122a762f
--- /dev/null
+++ b/guix/describe.scm
@@ -0,0 +1,73 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2018 Ludovic Courtès <ludo@gnu.org>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (guix describe)
+  #:use-module (guix memoization)
+  #:use-module (guix profiles)
+  #:use-module (srfi srfi-1)
+  #:use-module (ice-9 match)
+  #:export (package-path-entries))
+
+;;; Commentary:
+;;;
+;;; This module provides supporting code to allow a Guix instance to find, at
+;;; run time, which profile it's in (profiles created by 'guix pull').  That
+;;; allows it to read meta-information about itself (e.g., repository URL and
+;;; commit ID) and to find other channels available in the same profile.  It's
+;;; a bit like ELPA's pkg-info.el.
+;;;
+;;; Code:
+
+(define current-profile
+  (mlambda ()
+    "Return the profile (created by 'guix pull') the calling process lives in,
+or #f if this is not applicable."
+    (match (command-line)
+      ((program . _)
+       (and (string-suffix? "/bin/guix" program)
+            ;; Note: We want to do _lexical dot-dot resolution_.  Using ".."
+            ;; for real would instead take us into the /gnu/store directory
+            ;; that ~/.config/guix/current/bin points to, whereas we want to
+            ;; obtain ~/.config/guix/current.
+            (let ((candidate (dirname (dirname program))))
+              (and (file-exists? (string-append candidate "/manifest"))
+                   candidate)))))))
+
+(define current-profile-entries
+  (mlambda ()
+    "Return the list of entries in the 'guix pull' profile the calling process
+lives in, or #f if this is not applicable."
+    (match (current-profile)
+      (#f '())
+      (profile
+       (let ((manifest (profile-manifest profile)))
+         (manifest-entries manifest))))))
+
+(define package-path-entries
+  (mlambda ()
+    "Return a list of package path entries to be added to the package search
+path.  These entries are taken from the 'guix pull' profile the calling
+process lives in, when applicable."
+    ;; Filter out Guix itself.
+    (filter-map (lambda (entry)
+                  (and (not (string=? (manifest-entry-name entry)
+                                      "guix"))
+                       (string-append (manifest-entry-item entry)
+                                      "/share/guile/site/"
+                                      (effective-version))))
+                (current-profile-entries))))
-- 
2.18.0

bug#22629: [PATCH 3/3] DRAFT Add (guix channels) and use it in (guix scripts pull).

[Ludovic Courtès]

DRAFT: Missing documentation for ~/.config/guix/channels.scm.

* guix/channels.scm: New file.
* Makefile.am (MODULES): Add it.
* guix/scripts/pull.scm: Use it.
(%self-build-file, %pull-version, build-from-source)
(whole-package-for-legacy, derivation->manifest-entry): Remove.  These
now exist in a similar formin (guix channels).
(build-and-install): Change 'source' to 'instances'.  Remove #:url,
 #:branch, and #:commit.  Rewrite using 'channel-instances->manifest'.
(channel-list): New procedure.
(guix-pull): Parameterize %REPOSITORY-CACHE-DIRECTORY.  Call
'honor-lets-encrypt-certificates!' unconditionally.  Load
~/.config/guix/channels.scm.  Rewrite to use (guix channels).
[use-le-certs?]: Remove.
* po/guix/POTFILES.in: Add (guix channels).
---
 Makefile.am           |   1 +
 guix/channels.scm     | 292 ++++++++++++++++++++++++++++++++++++++++++
 guix/scripts/pull.scm | 198 +++++++++++-----------------
 po/guix/POTFILES.in   |   1 +
 4 files changed, 372 insertions(+), 120 deletions(-)
 create mode 100644 guix/channels.scm

diff --git a/Makefile.am b/Makefile.am
index b6efd6d62..af6870cf6 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -87,6 +87,7 @@ MODULES =					\
   guix/grafts.scm				\
   guix/inferior.scm				\
   guix/describe.scm				\
+  guix/channels.scm				\
   guix/gnu-maintenance.scm			\
   guix/self.scm					\
   guix/upstream.scm				\
diff --git a/guix/channels.scm b/guix/channels.scm
new file mode 100644
index 000000000..ec3e05eaf
--- /dev/null
+++ b/guix/channels.scm
@@ -0,0 +1,292 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2018 Ludovic Courtès <ludo@gnu.org>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (guix channels)
+  #:use-module (guix git)
+  #:use-module (guix records)
+  #:use-module (guix gexp)
+  #:use-module (guix discovery)
+  #:use-module (guix monads)
+  #:use-module (guix profiles)
+  #:use-module (guix derivations)
+  #:use-module (guix store)
+  #:use-module (guix i18n)
+  #:use-module (srfi srfi-1)
+  #:use-module (srfi srfi-9)
+  #:use-module (srfi srfi-11)
+  #:autoload   (guix self) (whole-package)
+  #:use-module (ice-9 match)
+  #:export (channel
+            channel?
+            channel-name
+            channel-url
+            channel-branch
+            channel-commit
+            channel-location
+
+            %default-channels
+
+            channel-instance?
+            channel-instance-channel
+            channel-instance-commit
+            channel-instance-checkout
+
+            latest-channel-instances
+            channel-instance-derivations
+            latest-channel-derivations
+            channel-instances->manifest))
+
+;;; Commentary:
+;;;
+;;; This module implements "channels."  A channel is usually a source of
+;;; package definitions.  There's a special channel, the 'guix' channel, that
+;;; provides all of Guix, including its commands and its documentation.
+;;; User-defined channels are expected to typically provide a bunch of .scm
+;;; files meant to be added to the '%package-search-path'.
+;;;
+;;; This module provides tools to fetch and update channels from a Git
+;;; repository and to build them.
+;;;
+;;; Code:
+
+(define-record-type* <channel> channel make-channel
+  channel?
+  (name      channel-name)
+  (url       channel-url)
+  (branch    channel-branch (default "master"))
+  (commit    channel-commit (default #f))
+  (location  channel-location
+             (default (current-source-location)) (innate)))
+;; TODO: Add a way to express dependencies among channels.
+
+(define %default-channels
+  ;; Default list of channels.
+  (list (channel
+         (name 'guix)
+         (branch "origin/master")
+         (url "https://git.savannah.gnu.org/git/guix.git"))))
+
+(define (guix-channel? channel)
+  "Return true if CHANNEL is the 'guix' channel."
+  (eq? 'guix (channel-name channel)))
+
+(define-record-type <channel-instance>
+  (channel-instance channel commit checkout)
+  channel-instance?
+  (channel   channel-instance-channel)
+  (commit    channel-instance-commit)
+  (checkout  channel-instance-checkout))
+
+(define (channel-reference channel)
+  "Return the \"reference\" for CHANNEL, an sexp suitable for
+'latest-repository-commit'."
+  (match (channel-commit channel)
+    (#f      `(branch . ,(channel-branch channel)))
+    (commit  `(commit . ,(channel-commit channel)))))
+
+(define (latest-channel-instances store channels)
+  "Return a list of channel instances corresponding to the latest checkouts of
+CHANNELS."
+  (map (lambda (channel)
+         (format (current-error-port)
+                 (G_ "Updating channel '~a' from Git repository at '~a'...~%")
+                 (channel-name channel)
+                 (channel-url channel))
+         (let-values (((checkout commit)
+                       (latest-repository-commit store (channel-url channel)
+                                                 #:ref (channel-reference
+                                                        channel))))
+           (channel-instance channel commit checkout)))
+       channels))
+
+(define %self-build-file
+  ;; The file containing code to build Guix.  This serves the same purpose as
+  ;; a makefile, and, similarly, is intended to always keep this name.
+  "build-aux/build-self.scm")
+
+(define %pull-version
+  ;; This is the version of the 'guix pull' protocol.  It specifies what's
+  ;; expected from %SELF-BUILD-FILE.  The initial version ("0") was when we'd
+  ;; place a set of compiled Guile modules in ~/.config/guix/latest.
+  1)
+
+(define (standard-module-derivation name source dependencies)
+  "Return a derivation that builds the Scheme modules in SOURCE and that
+depend on DEPENDENCIES, a list of lowerable objects.  The assumption is that
+SOURCE contains package modules to be added to '%package-module-path'."
+  (define modules
+    (scheme-modules* source))
+
+  ;; FIXME: We should load, say SOURCE/.guix-channel.scm, which would allow
+  ;; channel publishers to specify things such as the sub-directory where .scm
+  ;; files live, files to exclude from the channel, preferred substitute URLs,
+  ;; etc.
+  (mlet* %store-monad ((compiled
+                        (compiled-modules modules
+                                          #:name name
+                                          #:module-path (list source)
+                                          #:extensions dependencies)))
+
+    (gexp->derivation name
+                      (with-extensions dependencies
+                        (with-imported-modules '((guix build utils))
+                          #~(begin
+                              (use-modules (guix build utils))
+
+                              (let ((go  (string-append #$output "/lib/guile/"
+                                                        (effective-version)
+                                                        "/site-ccache"))
+                                    (scm (string-append #$output
+                                                        "/share/guile/site/"
+                                                        (effective-version))))
+                                (mkdir-p (dirname go))
+                                (symlink #$compiled go)
+                                (mkdir-p (dirname scm))
+                                (symlink #$source scm))))))))
+
+(define* (build-from-source name source
+                            #:key verbose? commit
+                            (dependencies '()))
+  "Return a derivation to build Guix from SOURCE, using the self-build script
+contained therein.  Use COMMIT as the version string."
+  ;; Running the self-build script makes it easier to update the build
+  ;; procedure: the self-build script of the Guix-to-be-installed contains the
+  ;; right dependencies, build procedure, etc., which the Guix-in-use may not
+  ;; be know.
+  (define script
+    (string-append source "/" %self-build-file))
+
+  (if (file-exists? script)
+      (let ((build (save-module-excursion
+                    (lambda ()
+                      (primitive-load script)))))
+        ;; BUILD must be a monadic procedure of at least one argument: the
+        ;; source tree.
+        ;;
+        ;; Note: BUILD can return #f if it does not support %PULL-VERSION.  In
+        ;; the future we'll fall back to a previous version of the protocol
+        ;; when that happens.
+        (build source #:verbose? verbose? #:version commit
+               #:pull-version %pull-version))
+
+      ;; Build a set of modules that extend Guix using the standard method.
+      (standard-module-derivation name source dependencies)))
+
+(define* (build-channel-instance instance #:optional (dependencies '()))
+  "Return, as a monadic value, the derivation for INSTANCE, a channel
+instance.  DEPENDENCIES is a list of extensions providing Guile modules that
+INSTANCE depends on."
+  (build-from-source (symbol->string
+                      (channel-name (channel-instance-channel instance)))
+                     (channel-instance-checkout instance)
+                     #:commit (channel-instance-commit instance)
+                     #:dependencies dependencies))
+
+(define (channel-instance-derivations instances)
+  "Return the list of derivations to build INSTANCES, in the same order as
+INSTANCES."
+  (define core-instance
+    ;; The 'guix' channel is treated specially: it's an implicit dependency of
+    ;; all the other channels.
+    (find (lambda (instance)
+            (guix-channel? (channel-instance-channel instance)))
+          instances))
+
+  (mlet %store-monad ((core (build-channel-instance core-instance)))
+    (mapm %store-monad
+          (lambda (instance)
+            (if (eq? instance core-instance)
+                (return core)
+                (build-channel-instance instance
+                                        (list core))))
+          instances)))
+
+(define latest-channel-derivations
+  (let ((latest-channel-instances (store-lift latest-channel-instances)))
+    (lambda (channels)
+      "Return, as a monadic value, the list of derivations for the latest
+instances of CHANNELS."
+      (mlet %store-monad ((instances (latest-channel-instances channels)))
+        (channel-instance-derivations instances)))))
+
+(define (whole-package-for-legacy name modules)
+  "Return a full-blown Guix package for MODULES, a derivation that builds Guix
+modules in the old ~/.config/guix/latest style."
+  (define packages
+    (resolve-interface '(gnu packages guile)))
+
+  (letrec-syntax ((list (syntax-rules (->)
+                          ((_)
+                           '())
+                          ((_ (module -> variable) rest ...)
+                           (cons (module-ref (resolve-interface
+                                              '(gnu packages module))
+                                             'variable)
+                                 (list rest ...)))
+                          ((_ variable rest ...)
+                           (cons (module-ref packages 'variable)
+                                 (list rest ...))))))
+    (whole-package name modules
+
+                   ;; In the "old style", %SELF-BUILD-FILE would simply return a
+                   ;; derivation that builds modules.  We have to infer what the
+                   ;; dependencies of these modules were.
+                   (list guile-json guile-git guile-bytestructures
+                         (ssh -> guile-ssh) (tls -> gnutls)))))
+
+(define (old-style-guix? drv)
+  "Return true if DRV corresponds to a ~/.config/guix/latest style of
+derivation."
+  ;; Here we rely on a gross historical fact: that derivations produced by the
+  ;; "old style" (before commit 8a0d9bc8a3f153159d9e239a151c0fa98f1e12d8,
+  ;; dated May 30, 2018) did not depend on "guix-command.drv".
+  (not (find (lambda (input)
+               (string-suffix? "-guix-command.drv"
+                               (derivation-input-path input)))
+             (derivation-inputs drv))))
+
+(define (channel-instances->manifest instances)
+  "Return a profile manifest with entries for all of INSTANCES, a list of
+channel instances."
+  (define instance->entry
+    (match-lambda
+      ((instance drv)
+       (let ((commit  (channel-instance-commit instance))
+             (channel (channel-instance-channel instance)))
+         (with-monad %store-monad
+           (return (manifest-entry
+                     (name (symbol->string (channel-name channel)))
+                     (version (string-take commit 7))
+                     (item (if (guix-channel? channel)
+                               (if (old-style-guix? drv)
+                                   (whole-package-for-legacy
+                                    (string-append name "-" version)
+                                    drv)
+                                   drv)
+                               drv))
+                     (properties
+                      `((source (repository
+                                 (version 0)
+                                 (url ,(channel-url channel))
+                                 (branch ,(channel-branch channel))
+                                 (commit ,commit))))))))))))
+
+  (mlet* %store-monad ((derivations (channel-instance-derivations instances))
+                       (entries     (mapm %store-monad instance->entry
+                                          (zip instances derivations))))
+    (return (manifest entries))))
diff --git a/guix/scripts/pull.scm b/guix/scripts/pull.scm
index ee68c21a4..d7931506b 100644
--- a/guix/scripts/pull.scm
+++ b/guix/scripts/pull.scm
@@ -30,26 +30,19 @@
   #:use-module (guix grafts)
   #:use-module (guix memoization)
   #:use-module (guix monads)
+  #:use-module (guix channels)
   #:autoload   (guix inferior) (open-inferior)
   #:use-module (guix scripts build)
-  #:autoload   (guix self) (whole-package)
   #:use-module (guix git)
   #:use-module (git)
   #:use-module (gnu packages)
-  #:autoload   (gnu packages ssh) (guile-ssh)
-  #:autoload   (gnu packages tls) (gnutls)
   #:use-module ((guix scripts package) #:select (build-and-use-profile))
-  #:use-module ((guix build utils)
-                #:select (with-directory-excursion delete-file-recursively))
-  #:use-module ((guix build download)
-                #:select (%x509-certificate-directory))
   #:use-module (gnu packages base)
   #:use-module (gnu packages guile)
   #:use-module ((gnu packages bootstrap)
                 #:select (%bootstrap-guile))
   #:use-module ((gnu packages certs) #:select (le-certs))
   #:use-module (srfi srfi-1)
-  #:use-module (srfi srfi-11)
   #:use-module (srfi srfi-26)
   #:use-module (srfi srfi-35)
   #:use-module (srfi srfi-37)
@@ -57,9 +50,6 @@
   #:use-module (ice-9 vlist)
   #:export (guix-pull))
 
-(define %repository-url
-  (or (getenv "GUIX_PULL_URL") "https://git.savannah.gnu.org/git/guix.git"))
-
 \f
 ;;;
 ;;; Command-line options.
@@ -67,9 +57,7 @@
 
 (define %default-options
   ;; Alist of default option values.
-  `((repository-url . ,%repository-url)
-    (ref . (branch . "origin/master"))
-    (system . ,(%current-system))
+  `((system . ,(%current-system))
     (substitutes? . #t)
     (build-hook? . #t)
     (graft? . #t)
@@ -142,70 +130,6 @@ Download and deploy the latest version of Guix.\n"))
 (define indirect-root-added
   (store-lift add-indirect-root))
 
-(define %self-build-file
-  ;; The file containing code to build Guix.  This serves the same purpose as
-  ;; a makefile, and, similarly, is intended to always keep this name.
-  "build-aux/build-self.scm")
-
-(define %pull-version
-  ;; This is the version of the 'guix pull' protocol.  It specifies what's
-  ;; expected from %SELF-BUILD-FILE.  The initial version ("0") was when we'd
-  ;; place a set of compiled Guile modules in ~/.config/guix/latest.
-  1)
-
-(define* (build-from-source source
-                            #:key verbose? commit)
-  "Return a derivation to build Guix from SOURCE, using the self-build script
-contained therein.  Use COMMIT as the version string."
-  ;; Running the self-build script makes it easier to update the build
-  ;; procedure: the self-build script of the Guix-to-be-installed contains the
-  ;; right dependencies, build procedure, etc., which the Guix-in-use may not
-  ;; be know.
-  (let* ((script (string-append source "/" %self-build-file))
-         (build  (primitive-load script)))
-    ;; BUILD must be a monadic procedure of at least one argument: the source
-    ;; tree.
-    ;;
-    ;; Note: BUILD can return #f if it does not support %PULL-VERSION.  In the
-    ;; future we'll fall back to a previous version of the protocol when that
-    ;; happens.
-    (build source #:verbose? verbose? #:version commit
-           #:pull-version %pull-version)))
-
-(define (whole-package-for-legacy name modules)
-  "Return a full-blown Guix package for MODULES, a derivation that builds Guix
-modules in the old ~/.config/guix/latest style."
-  (whole-package name modules
-
-                 ;; In the "old style", %SELF-BUILD-FILE would simply return a
-                 ;; derivation that builds modules.  We have to infer what the
-                 ;; dependencies of these modules were.
-                 (list guile-json guile-git guile-bytestructures
-                       guile-ssh gnutls)))
-
-(define* (derivation->manifest-entry drv
-                                     #:key url branch commit)
-  "Return a manifest entry for DRV, which represents Guix at COMMIT.  Record
-URL, BRANCH, and COMMIT as a property in the manifest entry."
-  (mbegin %store-monad
-    (what-to-build (list drv))
-    (built-derivations (list drv))
-    (let ((out (derivation->output-path drv)))
-      (return (manifest-entry
-                (name "guix")
-                (version (string-take commit 7))
-                (item (if (file-exists? (string-append out "/bin/guix"))
-                          drv
-                          (whole-package-for-legacy (string-append name "-"
-                                                                   version)
-                                                    drv)))
-                (properties
-                 `((source (repository
-                            (version 0)
-                            (url ,url)
-                            (branch ,branch)
-                            (commit ,commit))))))))))
-
 (define (display-profile-news profile)
   "Display what's up in PROFILE--new packages, and all that."
   (match (memv (generation-number profile)
@@ -223,8 +147,8 @@ URL, BRANCH, and COMMIT as a property in the manifest entry."
                                       #:heading (G_ "New in this revision:\n"))))
     (_ #t)))
 
-(define* (build-and-install source config-dir
-                            #:key verbose? url branch commit)
+(define* (build-and-install instances config-dir
+                            #:key verbose?)
   "Build the tool from SOURCE, and install it in CONFIG-DIR."
   (define update-profile
     (store-lift build-and-use-profile))
@@ -232,15 +156,9 @@ URL, BRANCH, and COMMIT as a property in the manifest entry."
   (define profile
     (string-append config-dir "/current"))
 
-  (mlet* %store-monad ((drv   (build-from-source source
-                                                 #:commit commit
-                                                 #:verbose? verbose?))
-                       (entry (derivation->manifest-entry drv
-                                                          #:url url
-                                                          #:branch branch
-                                                          #:commit commit)))
+  (mlet %store-monad ((manifest (channel-instances->manifest instances)))
     (mbegin %store-monad
-      (update-profile profile (manifest (list entry)))
+      (update-profile profile manifest)
       (return (display-profile-news profile)))))
 
 (define (honor-lets-encrypt-certificates! store)
@@ -426,45 +344,91 @@ and ALIST2 differ, display HEADING upfront."
                ((numbers ...)
                 (list-generations profile numbers)))))))))
 
+(define (channel-list file opts)
+  "Return the list of channels to use.  If FILE exists, channels are read from
+there; otherwise %DEFAULT-CHANNELS is used.  Apply channel transformations
+specified in OPTS (resulting from '--url', '--commit', or '--branch'), if
+any."
+  (define channels
+    (if (file-exists? file)
+        (load* file (make-user-module '((guix channels))))
+        %default-channels))
+
+  (define (environment-variable)
+    (match (getenv "GUIX_PULL_URL")
+      (#f #f)
+      (url
+       (warning (G_ "The 'GUIX_PULL_URL' environment variable is deprecated.
+Use '~/.config/guix/channels.scm' instead."))
+       url)))
+
+  (let ((ref (assoc-ref opts 'ref))
+        (url (or (assoc-ref opts 'url)
+                 (environment-variable))))
+    (if (or ref url)
+        (match channels
+          ((one)
+           ;; When there's only one channel, apply '--url', '--commit', and
+           ;; '--branch' to this specific channel.
+           (let ((url (or url (channel-url one))))
+             (list (match ref
+                     (('commit . commit)
+                      (channel (inherit one)
+                               (url url) (commit commit) (branch #f)))
+                     (('branch . branch)
+                      (channel (inherit one)
+                               (url url) (commit #f) (branch branch)))
+                     (#f
+                      (channel (inherit one) (url url)))))))
+          (_
+           ;; Otherwise bail out.
+           (leave
+            (G_ "'--url', '--commit', and '--branch' are not applicable~%"))))
+        channels)))
+
 \f
 (define (guix-pull . args)
-  (define (use-le-certs? url)
-    (string-prefix? "https://git.savannah.gnu.org/" url))
-
   (with-error-handling
     (with-git-error-handling
-     (let* ((opts  (parse-command-line args %options
-                                       (list %default-options)))
-            (url   (assoc-ref opts 'repository-url))
-            (ref   (assoc-ref opts 'ref))
-            (cache (string-append (cache-directory) "/pull")))
+     (let* ((opts         (parse-command-line args %options
+                                              (list %default-options)))
+            (cache        (string-append (cache-directory) "/pull"))
+            (channel-file (string-append (config-directory) "/channels.scm"))
+            (channels     (channel-list channel-file opts)))
+
        (cond ((assoc-ref opts 'query)
               (process-query opts))
              ((assoc-ref opts 'dry-run?)
               #t)                                 ;XXX: not very useful
              (else
               (with-store store
-                (parameterize ((%graft? (assoc-ref opts 'graft?)))
+                (parameterize ((%graft? (assoc-ref opts 'graft?))
+                               (%repository-cache-directory cache))
                   (set-build-options-from-command-line store opts)
 
-                  ;; For reproducibility, always refer to the LE certificates
-                  ;; when we know we're talking to Savannah.
-                  (when (use-le-certs? url)
-                    (honor-lets-encrypt-certificates! store))
-
-                  (format (current-error-port)
-                          (G_ "Updating from Git repository at '~a'...~%")
-                          url)
-
-                  (let-values (((checkout commit)
-                                (latest-repository-commit store url
-                                                          #:ref ref
-                                                          #:cache-directory
-                                                          cache)))
+                  ;; When certificates are already installed, use them.
+                  ;; Otherwise, use the Let's Encrypt certificates, which we
+                  ;; know Savannah uses.
+                  (let ((certs (or (getenv "SSL_CERT_DIR") "/etc/ssl/certs")))
+                    (unless (file-exists? certs)
+                      (honor-lets-encrypt-certificates! store)))
 
+                  (let ((instances (latest-channel-instances store channels)))
                     (format (current-error-port)
-                            (G_ "Building from Git commit ~a...~%")
-                            commit)
+                            (N_ "Building from this channel:~%"
+                                "Building from these channels:~%"
+                                (length instances)))
+                    (for-each (lambda (instance)
+                                (let ((channel
+                                       (channel-instance-channel instance)))
+                                  (format (current-error-port)
+                                          "  ~10a~a\t~a~%"
+                                          (channel-name channel)
+                                          (channel-url channel)
+                                          (string-take
+                                           (channel-instance-commit instance)
+                                           7))))
+                              instances)
                     (parameterize ((%guile-for-build
                                     (package-derivation
                                      store
@@ -472,13 +436,7 @@ and ALIST2 differ, display HEADING upfront."
                                          %bootstrap-guile
                                          (canonical-package guile-2.2)))))
                       (run-with-store store
-                        (build-and-install checkout (config-directory)
-                                           #:url url
-                                           #:branch (match ref
-                                                      (('branch . branch)
-                                                       branch)
-                                                      (_ #f))
-                                           #:commit commit
+                        (build-and-install instances (config-directory)
                                            #:verbose?
                                            (assoc-ref opts 'verbose?)))))))))))))
 
diff --git a/po/guix/POTFILES.in b/po/guix/POTFILES.in
index d11f408d4..7f881355e 100644
--- a/po/guix/POTFILES.in
+++ b/po/guix/POTFILES.in
@@ -38,4 +38,5 @@ guix/upstream.scm
 guix/ui.scm
 guix/http-client.scm
 guix/nar.scm
+guix/channels.scm
 nix/nix-daemon/guix-daemon.cc
-- 
2.18.0

bug#22629: Channels!

[Pjotr Prins]

Ludo, I am so ready to try this! 

GUIX_PACKAGE_PATH is killing me though it got a lot better after
latest guix pull developments.

Tell me what to run.

Pj.

bug#22629: Channels!

[Mark H Weaver]

Hi Ludovic,

ludo@gnu.org (Ludovic Courtès) writes:
> Currently third-party channels are expected to provide nothing but
> package modules.

I'd like to say again that I have grave concerns that this could be the
death-knell for long-term innovation in Guix.  It's likely that whenever
a change is proposed that will break these third-party channels, there
will be resistance, and efforts to preserve backward compatibility.

Even things as seemingly innocuous as moving a package from one module
to another will impact these third-party channels, not to mention
changing our internal APIs or making fundamental changes to the way
packages are specified.

Part of why I'm so interested in Guix is because it currently has nearly
unconstrained potential to grow into something far more beautiful and
elegant than it is today.

I fear that with the introduction of channels, that potential will be
drastically curtailed, and that we're essentially trading our future
potential for what will in practice, most likely, be primarily used to
facilitate the use of non-free software on Guix.

When I start to see signs of resistance to changes for the sake of
third-party channels, then I'll know I was right to be fearful, and
Guix will become far less interesting to me.

       Mark

bug#22629: Channels!

[Ludovic Courtès]

Hi Mark,

Mark H Weaver <mhw@netris.org> skribis:

> I'd like to say again that I have grave concerns that this could be the
> death-knell for long-term innovation in Guix.  It's likely that whenever
> a change is proposed that will break these third-party channels, there
> will be resistance, and efforts to preserve backward compatibility.

It is a risk and I acknowledge that.

In the 4 years of existence of GUIX_PACKAGE_PATH, we’ve never
encountered this issue.  To what extent would that change?  I don’t know.

I want to preserve the existing incentives to work in Guix proper, or to
work closely with the project when integration in Guix proper makes
little sense—for WIP packages, custom package variants, or
domain-specific packages like those I mentioned in my message.  I do not
see ourselves advertising channels as the primary way to add packages.

BTW, Andy’s Potluck offered a solution to the compatibility issue that’s
still on the table: <https://bugs.gnu.org/26645>.

> I fear that with the introduction of channels, that potential will be
> drastically curtailed, and that we're essentially trading our future
> potential for what will in practice, most likely, be primarily used to
> facilitate the use of non-free software on Guix.

Non-free software will be one use case.  There are other use cases
though, as mentioned above; these are the ones I’m interested in.

It’s a dilemma.  I’d like to think that we can have our cake and eat it
too; that we can give users this flexibility (which is “built in” since
it’s just a matter of setting the load path) without jeopardizing
development of Guix itself, and without creating the conditions for the
proliferation of non-free package channels.  Whether it works this way
would depend not on the technical details but on the group’s behavior.

Thoughts?

Thanks,
Ludo’.

bug#22629: Channels!

[Konrad Hinsen]

Hi Mark,

> I'd like to say again that I have grave concerns that this could be the > death-knell for long-term innovation in Guix.  It's likely that 
whenever> a change is proposed that will break these third-party 
channels, there> will be resistance, and efforts to preserve backward 
compatibility.
I understand your point, but the problem you mention is, in my opinion, 
not so much due to channels but due to different priorities of different 
users. Which means that it will come up even without channels as the 
Guix user base grows.

Look at the wider Linux world: there are people who want to live on the 
bleeding edge and run Arch Linux, and there are others who value 
stability and run CentOS. Today's Guix is more on the bleeding edge 
side. My understanding of your commment is that you would like to make 
sure it stays there. But that also means severely limiting Guix' 
potential user base.

I see channels as an opportunity to have Guix "dialects" addressing 
different needs and yet remain interoperable, although I am the first to 
admit that I have no clear idea of how this would work out in practice, 
more from the social than the technical point of view. But the goal 
looks very attractive. Looking at my own use of Guix, I am happy with 
its bleeding edge approach for the software I use for research, but I'd 
much prefer a slower pace and more stability for stuff like Emacs and 
TeX that are "boring infrastructure" for me.

> Even things as seemingly innocuous as moving a package from one module
> to another will impact these third-party channels, not to mention
> changing our internal APIs or making fundamental changes to the way
> packages are specified.

So... could we reduce the dependence of package specifications on such 
things? Package definitions use a small DSL that could be versioned, 
allowing change while maintaining compatibility. Module dependencies are 
more annoying, but do we need them? Package definitions are grouped into 
modules mostly for convenience. All packages have globally unique names, 
so could we use those to specify inputs?

Konrad.

bug#22629: Channels!

[Alex Sassmannshausen]

Mark H Weaver writes:

> Hi Ludovic,
>
> ludo@gnu.org (Ludovic Courtès) writes:
>> Currently third-party channels are expected to provide nothing but
>> package modules.
>
> I'd like to say again that I have grave concerns that this could be the
> death-knell for long-term innovation in Guix.  It's likely that whenever
> a change is proposed that will break these third-party channels, there
> will be resistance, and efforts to preserve backward compatibility.

I understand your concerns and want to acknowledge those.

My primary interest in channels at the moment comes from believing that
having a "stable" channel would be incredibly useful to increase
adoption rate of Guix.  And for me.

Currently upgrading my system involves doing a guix pull, then, over the
course of a few days, doing guix package -u and bailing out if I start
building a large program.

After this I do guix system build, and bail out if a large program
starts building.

In either case, if an upgrade broke a dependency then I'm kind of stuck
at the old versions of my profile.

Finally, when I've upgrade profile and system, I immediately run guix
pull to prepare for the next cycle.

I consider myself pretty capable, and I find this process stressful — I
certainly cannot envisage most of my currently interested friends going
through this process…

But like I say, this is not to discount your concerns, it is merely to
add to the list of reasons why channels might be important.

Best wishes,

Alex


> Even things as seemingly innocuous as moving a package from one module
> to another will impact these third-party channels, not to mention
> changing our internal APIs or making fundamental changes to the way
> packages are specified.
>
> Part of why I'm so interested in Guix is because it currently has nearly
> unconstrained potential to grow into something far more beautiful and
> elegant than it is today.
>
> I fear that with the introduction of channels, that potential will be
> drastically curtailed, and that we're essentially trading our future
> potential for what will in practice, most likely, be primarily used to
> facilitate the use of non-free software on Guix.
>
> When I start to see signs of resistance to changes for the sake of
> third-party channels, then I'll know I was right to be fearful, and
> Guix will become far less interesting to me.
>
>        Mark

bug#22629: Channels!

[Ludovic Courtès]

Hi Konrad,

Konrad Hinsen <konrad.hinsen@fastmail.net> skribis:

> Look at the wider Linux world: there are people who want to live on
> the bleeding edge and run Arch Linux, and there are others who value
> stability and run CentOS. Today's Guix is more on the bleeding edge
> side. My understanding of your commment is that you would like to make
> sure it stays there. But that also means severely limiting Guix'
> potential user base.

Mark’s concern is not about whether packages are the latest version,
etc.  It’s about the constraints that could result from widespread
development of channels outside Guix proper: technically all of Guix is
just a library, so widespread development of external channels could
force us Guix developers to keep the API stable.  This, in turn, would
limit our ability to make significant changes to Guix.

> I see channels as an opportunity to have Guix "dialects" addressing
> different needs and yet remain interoperable, although I am the first
> to admit that I have no clear idea of how this would work out in
> practice, more from the social than the technical point of view. But
> the goal looks very attractive. Looking at my own use of Guix, I am
> happy with its bleeding edge approach for the software I use for
> research, but I'd much prefer a slower pace and more stability for
> stuff like Emacs and TeX that are "boring infrastructure" for me.

“Dialect” sounds a bit strong, but I agree that such uses could be
interesting and beneficial, to users and to Guix.

>> Even things as seemingly innocuous as moving a package from one module
>> to another will impact these third-party channels, not to mention
>> changing our internal APIs or making fundamental changes to the way
>> packages are specified.
>
> So... could we reduce the dependence of package specifications on such
> things? Package definitions use a small DSL that could be versioned,
> allowing change while maintaining compatibility. Module dependencies
> are more annoying, but do we need them? Package definitions are
> grouped into modules mostly for convenience. All packages have
> globally unique names, so could we use those to specify inputs?

This is exactly that kind of issue Mark is concerned with: currently we
don’t have to worry at all about this, and it’s great to have that
freedom.

I’d rather not build fancy mechanisms just for the sake of external
channels, and I certainly don’t want to commit to API stability.  We
won’t break the API every day intentionally either, but my point is that
support for external channels will be “best effort” (as it already is
with GUIX_PACKAGE_PATH).  I think it’s useful and practical nonetheless.

Ludo’.

bug#22629: Channels!

[Konrad Hinsen]

Hi Ludo,

> Mark’s concern is not about whether packages are the latest version,
> etc.  It’s about the constraints that could result from widespread
> development of channels outside Guix proper: technically all of Guix is

That's how I understood it as well. If/when Guix becomes somebody else's
dependency, then there will be pressure on stability in Guix itself.

My point is that this will happen anyway if Guix is adopted more widely.
Every manifest file, personal or shared as part of a software package
("guix.scm"), relies on the same technical details as a
channel. Introducing channels only makes the issue more visible.

And this is really the same issue as with the stability of the packages
themselves, Guix being a kind of superpackage. Most people want agility
for the software layer they are most concerned with, and stability for
all layers below it. For Mark (and certainly others here), Guix happens
to be the layer they are most concerned with.

> I’d rather not build fancy mechanisms just for the sake of external
> channels, and I certainly don’t want to commit to API stability.  We

At this point, certainly not. But I agree with Mark that, if channels
"take off", there will be pressure in that direction.

Konrad.

bug#22629: Channels not needed for a stable branch (was: Channels!)

[Mark H Weaver]

Hi,

Alex Sassmannshausen <alex@pompo.co> writes:
> My primary interest in channels at the moment comes from believing that
> having a "stable" channel would be incredibly useful to increase
> adoption rate of Guix.  And for me.

Konrad Hinsen <konrad.hinsen@fastmail.net> writes:
> Look at the wider Linux world: there are people who want to live on
> the bleeding edge and run Arch Linux, and there are others who value
> stability and run CentOS. Today's Guix is more on the bleeding edge
> side. My understanding of your commment is that you would like to make
> sure it stays there. But that also means severely limiting Guix'
> potential user base.

Both of you seem to have reached the conclusion that third-party
channels are a prerequisite for having a 'stable' branch.  I disagree.

I agree with both of you that a 'stable' branch of Guix would be
tremendously useful.  I've often wanted it myself, and I still do.

My point is that I want to keep our APIs internal and unfrozen for the
same reason that Linux, the kernel project, does.  Linux refuses to
support out-of-tree drivers and modules, and thereby retains its freedom
to change their internal APIs.  Often they change how things work
internally and this entails doing massive find-replace on every driver
in the tree.  This has been a crucially important factor in their
long-term success.

Does this stop the Linux developers from offering stable branches, or
third-parties from maintaining older stable versions of Linux in their
own Git repositories?  Clearly not.  It's been done all along.

We should persue a similar model.  The crucial thing is to always keep
the package modules together with the rest of Guix.  If you want to
clone Guix to a third-party repository, I have no problem with that.
Just keep it all together, and maintain the entire tree as an undivided
whole.

"guix pull" already supports the ability to fetch from an arbitrary
branch, or even from an arbitrary upstream source, so we already have
what we need to start a 'stable' branch, or for a third-party to do that
if they wanted to.

     Regards,
       Mark

bug#22629: Channels not needed for a stable branch (was: Channels!)

[Ricardo Wurmus]

Hi Mark,

> I'd like to say again that I have grave concerns that this could be the
> death-knell for long-term innovation in Guix.  It's likely that whenever
> a change is proposed that will break these third-party channels, there
> will be resistance, and efforts to preserve backward compatibility.

GUIX_PACKAGE_PATH already had that same problem (and did not provide a
solution for it).  With channels we can at least add more information
about a collection of modules, e.g. what version of Guix it is known to
work with.  So channels really flesh out the feature provided by
GUIX_PACKAGE_PATH, elevating it from a simple environment variable to
one that can take additional context into account.

I think that’s a worthwhile step to take.

I agree with your sentiment that a mechanism based on a simple
environment variable does not instill confidence, whereas a special
mechanism like channels could signal to users that it’s a feature that
provides some guarantees.  But I disagree with your assertion that this
would be “a death-knell to innovation”.  That seems like an exaggeration
to me.

> My point is that I want to keep our APIs internal and unfrozen for the
> same reason that Linux, the kernel project, does.  Linux refuses to
> support out-of-tree drivers and modules, and thereby retains its freedom
> to change their internal APIs.  Often they change how things work
> internally and this entails doing massive find-replace on every driver
> in the tree.  This has been a crucially important factor in their
> long-term success.
[…]
> We should persue a similar model.  The crucial thing is to always keep
> the package modules together with the rest of Guix.

I agree.  That is and remains our recommendation.  I still want most
packages to end up in Guix proper.  There are collections of packages
for which this does not make sense, though, and I think that it is
better to have a more formal mechanism that can also be used to describe
the limits of compatibility than just a simple environment variable.

I also agree with you that we don’t need channels for providing a stable
branch.  The biggest obstacle to providing a stable branch is not
technical, but it requires people maintaining it.

--
Ricardo

bug#22629: Channels!

[Ludovic Courtès]

Hello Konrad,

Konrad Hinsen <konrad.hinsen@fastmail.net> skribis:

>> Mark’s concern is not about whether packages are the latest version,
>> etc.  It’s about the constraints that could result from widespread
>> development of channels outside Guix proper: technically all of Guix is
>
> That's how I understood it as well. If/when Guix becomes somebody else's
> dependency, then there will be pressure on stability in Guix itself.
>
> My point is that this will happen anyway if Guix is adopted more widely.
> Every manifest file, personal or shared as part of a software package
> ("guix.scm"), relies on the same technical details as a
> channel. Introducing channels only makes the issue more visible.

True.  Manifests can rely on fewer details of the API than a channel
though, particularly if they use ‘specifications->manifests’.

(BTW code in channels could use ‘specification->package’ as well to
increase decoupling a bit.)

> And this is really the same issue as with the stability of the packages
> themselves, Guix being a kind of superpackage. Most people want agility
> for the software layer they are most concerned with, and stability for
> all layers below it. For Mark (and certainly others here), Guix happens
> to be the layer they are most concerned with.

Yeah.

Thanks,
Ludo’.

bug#22629: Channels not needed for a stable branch

[Ludovic Courtès]

Hello Guix,

Mark H Weaver <mhw@netris.org> skribis:

> Both of you seem to have reached the conclusion that third-party
> channels are a prerequisite for having a 'stable' branch.  I disagree.

Same here.  We could already be doing that (I’m skeptical about the
feasibility, maintainability, and relevance of a “stable” branch in the
sense of Debian stable, but that’s another story.)

> I agree with both of you that a 'stable' branch of Guix would be
> tremendously useful.  I've often wanted it myself, and I still do.
>
> My point is that I want to keep our APIs internal and unfrozen for the
> same reason that Linux, the kernel project, does.  Linux refuses to
> support out-of-tree drivers and modules, and thereby retains its freedom
> to change their internal APIs.  Often they change how things work
> internally and this entails doing massive find-replace on every driver
> in the tree.  This has been a crucially important factor in their
> long-term success.

I had this example in mind too: the kernel technically supports
out-of-tree modules, but kernel developers have always resisted pressure
to freeze APIs.

Because this policy has been stated upfront very clearly and has
remained unchanged, out-of-tree module developers know that that the
compatibility burden is on them.  There’s flexibility, along with a
strong incentive to get things in the mainline kernel.

This is the outcome I’d like to achieve: give users some welcome
flexibility, but make it clear that it’s best-effort.

Ludo’.

bug#22629: Channels not needed for a stable branch (was: Channels!)

[Konrad Hinsen]

Hi Ricardo,

> I also agree with you that we don’t need channels for providing a stable
> branch.  The biggest obstacle to providing a stable branch is not
> technical, but it requires people maintaining it.

Look at this from the opposite end: if you were interested in
maintaining a stable software distribution, would you choose a quickly
evolving package manager such as Guix as the basis? I'd say no, and I am
speaking from experience because I did actually maintain stable software
installations for a couple of years. You want to concentrate on critical
bug fixes and avoid anything else that could perturb the stability of
your system.

For me this whole discussion isn't about technicalities. Channels are
just a convenience layer on top of what can already be done with
GUIX_PACKAGE_PATH and manifest files. The real issue is how such
features are advertised: as a way to bolt on your own developments,
following Guix evolution closely, or as a way to provide decoupled
autonomous branches such as stable software.

So I agree with Ludo that a communication along the lines of

> I had this example in mind too: the kernel technically supports
> out-of-tree modules, but kernel developers have always resisted pressure
> to freeze APIs.
>
> Because this policy has been stated upfront very clearly and has
> remained unchanged, out-of-tree module developers know that that the
> compatibility burden is on them.  There’s flexibility, along with a
> strong incentive to get things in the mainline kernel.

should be sufficient to avoid the issue that Mark has raised here.
But it also discourages people interested in stable software
installations from using or contributing to Guix.

Konrad.

bug#22629: Channels not needed for a stable branch

[Mark H Weaver]

Konrad Hinsen <konrad.hinsen@fastmail.net> writes:

>> I also agree with you that we don’t need channels for providing a stable
>> branch.  The biggest obstacle to providing a stable branch is not
>> technical, but it requires people maintaining it.
>
> Look at this from the opposite end: if you were interested in
> maintaining a stable software distribution, would you choose a quickly
> evolving package manager such as Guix as the basis? I'd say no, and I am
> speaking from experience because I did actually maintain stable software
> installations for a couple of years. You want to concentrate on critical
> bug fixes and avoid anything else that could perturb the stability of
> your system.

I'm not sure what you're trying to argue above.  To me, it looks like an
argument in favor of my position, namely that a stable version of Guix
should include _all_ of Guix, not just the packages.

If you want to maintain a stable distribution, why would you want to
combine stable package descriptions with quickly evolving infrastructure
that makes up the rest of Guix?

If you want stability, wouldn't it be better to keep a stable branch of
_all_ of Guix, so that both the package descriptions _and_ the
infrastructure upon which those packages depend are kept stable?

This is exactly what a 'stable' branch in a git repository would
provide, and moreover "guix pull" already has everything that's needed
to support this.  All that's needed are people to maintain such a
branch.

If we again compare this to Linux (the kernel project) and their refusal
to support out-of-tree drivers: I'm arguing in favor of the Linux
approach, where stable kernels are based on git branches that include a
_all_ of Linux.  Your position, transposed to Linux, would seem to be in
favor of third-parties supporting stable drivers, which would then be
combined with the latest version of the rest of the kernel.

Am I misunderstanding your argument?  Can you please clarify?

       Mark

bug#22629: Channels not needed for a stable branch

[Konrad Hinsen]

Hi Mark,

> I'm not sure what you're trying to argue above.  To me, it looks like an
> argument in favor of my position, namely that a stable version of Guix
> should include _all_ of Guix, not just the packages.

All, probably not, some, probably yes. What I am arguing is that the
productive coexistence of a stable version with the bleeding-edge
version requires agreement on a stable foundation. Where exactly the
borderline lies between this foundation and what is built on top of it
is not a question I am sufficiently qualified to answer.

The minimal stable foundation would have to include the file system
layout of profiles, to make sure that users can mix packages from both
versions safely. It would also be highly desirable to share the store,
whose layout would then have to be part of the foundation as well.

Moreover, I suspect it would be preferable or even necessary to have
only one daemon running - if that's true, then the daemon's
communication protocol would have be part of the foundation as well.

Without a common foundation, a stable version would have to be a
completely autonomous fork, which should then probably adopt a different
name as well. I don't think this is desirable, in particular for GuixSD
which would lose most of its interest if it required multiple package
managers.

Konrad.

bug#22629: “Stable” branch

[Ludovic Courtès]

Hi Konrad,

Konrad Hinsen <konrad.hinsen@fastmail.net> skribis:

> The minimal stable foundation would have to include the file system
> layout of profiles, to make sure that users can mix packages from both
> versions safely. It would also be highly desirable to share the store,
> whose layout would then have to be part of the foundation as well.
>
> Moreover, I suspect it would be preferable or even necessary to have
> only one daemon running - if that's true, then the daemon's
> communication protocol would have be part of the foundation as well.
>
> Without a common foundation, a stable version would have to be a
> completely autonomous fork, which should then probably adopt a different
> name as well. I don't think this is desirable, in particular for GuixSD
> which would lose most of its interest if it required multiple package
> managers.

These are all things that very rarely, if ever, changed over the last 5
years.  I expect the change rate to remain the same.  :-)

You seem to be arguing of a “stable” branch in the sense that the Guix
tools (the CLI in particular) wouldn’t change much, is that correct?

I’m asking because there are several ways to define “stable.”  Initially
I thought what you had in mind was like the “stable” branch in Debian,
meaning that packages only get security updates.  To me that’s a
different thing.

Ludo’.

bug#22629: “Stable” branch

[Alex Sassmannshausen]

Ludovic Courtès writes:

> Hi Konrad,
>
> Konrad Hinsen <konrad.hinsen@fastmail.net> skribis:
>
>> The minimal stable foundation would have to include the file system
>> layout of profiles, to make sure that users can mix packages from both
>> versions safely. It would also be highly desirable to share the store,
>> whose layout would then have to be part of the foundation as well.
>>
>> Moreover, I suspect it would be preferable or even necessary to have
>> only one daemon running - if that's true, then the daemon's
>> communication protocol would have be part of the foundation as well.
>>
>> Without a common foundation, a stable version would have to be a
>> completely autonomous fork, which should then probably adopt a different
>> name as well. I don't think this is desirable, in particular for GuixSD
>> which would lose most of its interest if it required multiple package
>> managers.
>
> These are all things that very rarely, if ever, changed over the last 5
> years.  I expect the change rate to remain the same.  :-)
>
> You seem to be arguing of a “stable” branch in the sense that the Guix
> tools (the CLI in particular) wouldn’t change much, is that correct?
>
> I’m asking because there are several ways to define “stable.”  Initially
> I thought what you had in mind was like the “stable” branch in Debian,
> meaning that packages only get security updates.  To me that’s a
> different thing.

I don't know if this is what Konrad desires, but from my perspective, a
desirable part of the definition of stable would be a that the build
farms have produced a set of binaries/substitutes for a given Guix
revision that is "good enough".

Where good enough might mean something like "no more build failures than
the previous iteration", or "no more build failures on common desktop
applications" or other some exciting metric (the exact definition is not
really important right now).

The main concern here would be that we try to avoid end users from
having unexpected build failures or even missing substitutes after doing
a guix pull.

Alex

bug#22629: “Stable” branch

[Konrad Hinsen]

Hi Ludo and Alex,


ludo@gnu.org (Ludovic Courtès) writes:

> These are all things that very rarely, if ever, changed over the last 5
> years.  I expect the change rate to remain the same.  :-)

That's reassuring!

> You seem to be arguing of a “stable” branch in the sense that the Guix
> tools (the CLI in particular) wouldn’t change much, is that correct?
>
> I’m asking because there are several ways to define “stable.”  Initially
> I thought what you had in mind was like the “stable” branch in Debian,
> meaning that packages only get security updates.  To me that’s a
> different thing.

What I have in mind is whatever it takes to build a stable software
system. That includes stable ingredients (packages) but also stable
glue, meaning the package definitions and the build system that produces
the binaries from them.  Stability of the Guix CLI is much less relevant
from my point of view.


Alex Sassmannshausen <alex@pompo.co> writes:

> I don't know if this is what Konrad desires, but from my perspective, a
> desirable part of the definition of stable would be a that the build
> farms have produced a set of binaries/substitutes for a given Guix
> revision that is "good enough".

That's another very practically relevant notion of stability. Mine goes
beyond that though. For example, I'd require all packages to build and
pass tests at all times.

Konrad.

bug#22629: Channels!

[Ludovic Courtès]

ludo@gnu.org (Ludovic Courtès) skribis:

> The patches that follow implement this last bit, though in a slightly
> different way.  Users would now have the option to provide
> ~/.config/guix/channels.scm along these lines:
>
>   (cons (channel
>          (name 'guix-hpc)
>          (url "https://gitlab.inria.fr/guix-hpc/guix-hpc.git")
>          (branch "origin/master"))
>         %default-channels)

What follows is version 2 of the patches, which I’d like to push within
a day or two.  The main changes are:

  1. Added documentation for channels and ~/.config/guix/channels.scm.
     In particular I added a warning about compatibility that summarizes
     what we discussed.  Please let me know what you think.

  2. Added -C/--channels option to ‘guix pull’ to specify a channel
     file.

  3. Fixed ‘update-NEWS.scm’, which assumed that the last item of
     (%package-module-path) is Guix’s own set of modules.  We also find
     this pattern in guix-artwork.git/website and probably in other
     places, so we’ll have to adjust them.

Ludo’.

bug#22629: [PATCH v2 1/3] discovery: Add 'scheme-modules*'.

[Ludovic Courtès]

* guix/self.scm (scheme-modules*): Move to...
* guix/discovery.scm (scheme-modules*): ... here.  New procedure.  Make
'sub-directory' an optional parameter.
---
 guix/discovery.scm | 13 ++++++++++++-
 guix/self.scm      |  7 -------
 2 files changed, 12 insertions(+), 8 deletions(-)

diff --git a/guix/discovery.scm b/guix/discovery.scm
index 2b627d108..3fc6e2c9e 100644
--- a/guix/discovery.scm
+++ b/guix/discovery.scm
@@ -1,5 +1,5 @@
 ;;; GNU Guix --- Functional package management for GNU
-;;; Copyright © 2012, 2013, 2014, 2015, 2016, 2017 Ludovic Courtès <ludo@gnu.org>
+;;; Copyright © 2012, 2013, 2014, 2015, 2016, 2017, 2018 Ludovic Courtès <ludo@gnu.org>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -27,6 +27,7 @@
   #:use-module (ice-9 ftw)
   #:export (scheme-files
             scheme-modules
+            scheme-modules*
             fold-modules
             all-modules
             fold-module-public-variables))
@@ -115,6 +116,16 @@ name and the exception key and arguments."
                                 (string-append directory "/" sub-directory)
                                 directory))))
 
+(define* (scheme-modules* directory #:optional sub-directory)
+  "Return the list of module names found under SUB-DIRECTORY in DIRECTORY.
+This is a source-only variant that does not try to load files."
+  (let ((prefix (string-length directory)))
+    (map (lambda (file)
+           (file-name->module-name (string-drop file prefix)))
+         (scheme-files (if sub-directory
+                           (string-append directory "/" sub-directory)
+                           directory)))))
+
 (define* (fold-modules proc init path #:key (warn (const #f)))
   "Fold over all the Scheme modules present in PATH, a list of directories.
 Call (PROC MODULE RESULT) for each module that is found."
diff --git a/guix/self.scm b/guix/self.scm
index 90649db17..81f9b0cfd 100644
--- a/guix/self.scm
+++ b/guix/self.scm
@@ -206,13 +206,6 @@ list of file-name/file-like objects suitable as inputs to 'imported-files'."
                (local-file file #:recursive? #t)))
        (find-files (string-append directory "/" sub-directory) pred)))
 
-(define (scheme-modules* directory sub-directory)
-  "Return the list of module names found under SUB-DIRECTORY in DIRECTORY."
-  (let ((prefix (string-length directory)))
-    (map (lambda (file)
-           (file-name->module-name (string-drop file prefix)))
-         (scheme-files (string-append directory "/" sub-directory)))))
-
 (define* (sub-directory item sub-directory)
   "Return SUB-DIRECTORY within ITEM, which may be a file name or a file-like
 object."
-- 
2.18.0

bug#22629: [PATCH v2 2/3] Add (guix describe) and use it to initialize '%package-search-path'.

[Ludovic Courtès]

* guix/describe.scm: New file.
* Makefile.am (MODULES): Add it.
* gnu/packages.scm (%default-package-module-path): New variable.
(%package-module-path): Honor 'package-path-entries'.
* build-aux/update-NEWS.scm (main): Use %DEFAULT-PACKAGE-MODULE-PATH
instead of (last (%package-module-path)).
---
 Makefile.am               |  1 +
 build-aux/update-NEWS.scm |  9 +++--
 gnu/packages.scm          | 27 ++++++++++-----
 guix/describe.scm         | 73 +++++++++++++++++++++++++++++++++++++++
 4 files changed, 97 insertions(+), 13 deletions(-)
 create mode 100644 guix/describe.scm

diff --git a/Makefile.am b/Makefile.am
index 324674a60..b6efd6d62 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -86,6 +86,7 @@ MODULES =					\
   guix/derivations.scm				\
   guix/grafts.scm				\
   guix/inferior.scm				\
+  guix/describe.scm				\
   guix/gnu-maintenance.scm			\
   guix/self.scm					\
   guix/upstream.scm				\
diff --git a/build-aux/update-NEWS.scm b/build-aux/update-NEWS.scm
index 2e8f68c9a..a9dffef1d 100644
--- a/build-aux/update-NEWS.scm
+++ b/build-aux/update-NEWS.scm
@@ -1,5 +1,5 @@
 ;;; GNU Guix --- Functional package management for GNU
-;;; Copyright © 2017 Ludovic Courtès <ludo@gnu.org>
+;;; Copyright © 2017, 2018 Ludovic Courtès <ludo@gnu.org>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -128,11 +128,10 @@ paragraph."
 (define (main . args)
   (match args
     ((news-file data-directory)
-     ;; Don't browse things listed in the user's $GUIX_PACKAGE_PATH.  Here we
-     ;; assume that the last item in (%package-module-path) is the distro
-     ;; directory.
+     ;; Don't browse things listed in the user's $GUIX_PACKAGE_PATH and
+     ;; in external channels.
      (parameterize ((%package-module-path
-                     (list (last (%package-module-path)))))
+                     %default-package-module-path))
        (define (package-file version)
          (string-append data-directory "/packages-"
                         version ".txt"))
diff --git a/gnu/packages.scm b/gnu/packages.scm
index 7b954769e..532297239 100644
--- a/gnu/packages.scm
+++ b/gnu/packages.scm
@@ -30,6 +30,7 @@
                 #:select ((package-name->name+version
                            . hyphen-separated-name->name+version)))
   #:autoload   (guix profiles) (packages->manifest)
+  #:use-module (guix describe)
   #:use-module (ice-9 vlist)
   #:use-module (ice-9 match)
   #:use-module (srfi srfi-1)
@@ -46,6 +47,7 @@
             %auxiliary-files-path
             %bootstrap-binaries-path
             %package-module-path
+            %default-package-module-path
 
             fold-packages
 
@@ -130,22 +132,31 @@ for system '~a'")
          ("gnu/packages.scm"      gnu/)
          ("guix.scm"))))
 
+(define %default-package-module-path
+  ;; Default search path for package modules.
+  `((,%distro-root-directory . "gnu/packages")))
+
 (define %package-module-path
   ;; Search path for package modules.  Each item must be either a directory
   ;; name or a pair whose car is a directory and whose cdr is a sub-directory
   ;; to narrow the search.
   (let* ((not-colon   (char-set-complement (char-set #\:)))
          (environment (string-tokenize (or (getenv "GUIX_PACKAGE_PATH") "")
-                                       not-colon)))
-    ;; Automatically add items from $GUIX_PACKAGE_PATH to Guile's search path.
-    (for-each (lambda (directory)
-                (set! %load-path (cons directory %load-path))
-                (set! %load-compiled-path
-                      (cons directory %load-compiled-path)))
-              environment)
+                                       not-colon))
+         (channels    (package-path-entries)))
+    ;; Automatically add channels and items from $GUIX_PACKAGE_PATH to Guile's
+    ;; search path.  For historical reasons, $GUIX_PACKAGE_PATH goes to the
+    ;; front; channels go to the back so that they don't override Guix' own
+    ;; modules.
+    (set! %load-path
+      (append environment %load-path channels))
+    (set! %load-compiled-path
+      (append environment %load-compiled-path channels))
 
     (make-parameter
-     (append environment `((,%distro-root-directory . "gnu/packages"))))))
+     (append environment
+             %default-package-module-path
+             channels))))
 
 (define %patch-path
   ;; Define it after '%package-module-path' so that '%load-path' contains user
diff --git a/guix/describe.scm b/guix/describe.scm
new file mode 100644
index 000000000..3122a762f
--- /dev/null
+++ b/guix/describe.scm
@@ -0,0 +1,73 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2018 Ludovic Courtès <ludo@gnu.org>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (guix describe)
+  #:use-module (guix memoization)
+  #:use-module (guix profiles)
+  #:use-module (srfi srfi-1)
+  #:use-module (ice-9 match)
+  #:export (package-path-entries))
+
+;;; Commentary:
+;;;
+;;; This module provides supporting code to allow a Guix instance to find, at
+;;; run time, which profile it's in (profiles created by 'guix pull').  That
+;;; allows it to read meta-information about itself (e.g., repository URL and
+;;; commit ID) and to find other channels available in the same profile.  It's
+;;; a bit like ELPA's pkg-info.el.
+;;;
+;;; Code:
+
+(define current-profile
+  (mlambda ()
+    "Return the profile (created by 'guix pull') the calling process lives in,
+or #f if this is not applicable."
+    (match (command-line)
+      ((program . _)
+       (and (string-suffix? "/bin/guix" program)
+            ;; Note: We want to do _lexical dot-dot resolution_.  Using ".."
+            ;; for real would instead take us into the /gnu/store directory
+            ;; that ~/.config/guix/current/bin points to, whereas we want to
+            ;; obtain ~/.config/guix/current.
+            (let ((candidate (dirname (dirname program))))
+              (and (file-exists? (string-append candidate "/manifest"))
+                   candidate)))))))
+
+(define current-profile-entries
+  (mlambda ()
+    "Return the list of entries in the 'guix pull' profile the calling process
+lives in, or #f if this is not applicable."
+    (match (current-profile)
+      (#f '())
+      (profile
+       (let ((manifest (profile-manifest profile)))
+         (manifest-entries manifest))))))
+
+(define package-path-entries
+  (mlambda ()
+    "Return a list of package path entries to be added to the package search
+path.  These entries are taken from the 'guix pull' profile the calling
+process lives in, when applicable."
+    ;; Filter out Guix itself.
+    (filter-map (lambda (entry)
+                  (and (not (string=? (manifest-entry-name entry)
+                                      "guix"))
+                       (string-append (manifest-entry-item entry)
+                                      "/share/guile/site/"
+                                      (effective-version))))
+                (current-profile-entries))))
-- 
2.18.0

bug#22629: [PATCH v2 3/3] Add (guix channels) and use it in (guix scripts pull).

[Ludovic Courtès]

* guix/channels.scm: New file.
* Makefile.am (MODULES): Add it.
* guix/scripts/pull.scm: Use it.
(%default-options): Remove 'repository-url' and 'ref'.
(show-help, %options): Add '--channels'.
(%self-build-file, %pull-version, build-from-source)
(whole-package-for-legacy, derivation->manifest-entry): Remove.  These
now exist in a similar form in (guix channels).
(build-and-install): Change 'source' to 'instances'.  Remove #:url,
 #:branch, and #:commit.  Rewrite using 'channel-instances->manifest'.
(channel-list): New procedure.
(guix-pull): Parameterize %REPOSITORY-CACHE-DIRECTORY.  Call
'honor-lets-encrypt-certificates!' unconditionally.  Load
~/.config/guix/channels.scm.  Rewrite to use (guix channels).
[use-le-certs?]: Remove.
* po/guix/POTFILES.in: Add (guix channels).
* doc/guix.texi (Invoking guix pull): Group the description of '--url',
'--commit', and '--branch'.  Remove mention of 'GUIX_PULL_URL'.  Add
references to "Channels".  Document '--channels'.
(Channels): New node.
(Defining Packages): Link to "Channels" instead of "Package Modules".
(Invoking guix edit): Link to "Package Modules" instead of "Defining
Packages".
(Package Modules): Document both GUIX_PACKAGE_PATH and channels.
---
 Makefile.am           |   1 +
 doc/guix.texi         | 271 ++++++++++++++++++++++++++++++++-------
 guix/channels.scm     | 292 ++++++++++++++++++++++++++++++++++++++++++
 guix/scripts/pull.scm | 218 ++++++++++++++-----------------
 po/guix/POTFILES.in   |   1 +
 5 files changed, 619 insertions(+), 164 deletions(-)
 create mode 100644 guix/channels.scm

diff --git a/Makefile.am b/Makefile.am
index b6efd6d62..af6870cf6 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -87,6 +87,7 @@ MODULES =					\
   guix/grafts.scm				\
   guix/inferior.scm				\
   guix/describe.scm				\
+  guix/channels.scm				\
   guix/gnu-maintenance.scm			\
   guix/self.scm					\
   guix/upstream.scm				\
diff --git a/doc/guix.texi b/doc/guix.texi
index cd0e74a2d..76d8626d7 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -146,17 +146,18 @@ Package Management
 * Packages with Multiple Outputs::  Single source package, multiple outputs.
 * Invoking guix gc::            Running the garbage collector.
 * Invoking guix pull::          Fetching the latest Guix and distribution.
+* Channels::                    Customizing the package collection.
 * Invoking guix pack::          Creating software bundles.
 * Invoking guix archive::       Exporting and importing store files.
 
 Substitutes
 
-* Official Substitute Server::      One particular source of substitutes.
-* Substitute Server Authorization:: How to enable or disable substitutes.
-* Substitute Authentication::       How Guix verifies substitutes.
-* Proxy Settings::                  How to get substitutes via proxy.
-* Substitution Failure::            What happens when substitution fails.
-* On Trusting Binaries::            How can you trust that binary blob?
+* Official Substitute Server::  One particular source of substitutes.
+* Substitute Server Authorization::  How to enable or disable substitutes.
+* Substitute Authentication::   How Guix verifies substitutes.
+* Proxy Settings::              How to get substitutes via proxy.
+* Substitution Failure::        What happens when substitution fails.
+* On Trusting Binaries::        How can you trust that binary blob?
 
 Programming Interface
 
@@ -202,7 +203,7 @@ GNU Distribution
 
 * System Installation::         Installing the whole operating system.
 * System Configuration::        Configuring the operating system.
-* Documentation::                Browsing software user manuals.
+* Documentation::               Browsing software user manuals.
 * Installing Debugging Files::  Feeding the debugger.
 * Security Updates::            Deploying security fixes quickly.
 * Package Modules::             Packages from the programmer's viewpoint.
@@ -264,7 +265,7 @@ Services
 * Audio Services::              The MPD.
 * Virtualization Services::     Virtualization services.
 * Version Control Services::    Providing remote access to Git repositories.
-* Game Services::                Game servers.
+* Game Services::               Game servers.
 * Miscellaneous Services::      Other services.
 
 Defining Services
@@ -1694,6 +1695,7 @@ guix package -i emacs-guix
 * Packages with Multiple Outputs::  Single source package, multiple outputs.
 * Invoking guix gc::            Running the garbage collector.
 * Invoking guix pull::          Fetching the latest Guix and distribution.
+* Channels::                    Customizing the package collection.
 * Invoking guix pack::          Creating software bundles.
 * Invoking guix archive::       Exporting and importing store files.
 @end menu
@@ -2276,12 +2278,12 @@ pre-built package binaries, but source tarballs, for instance, which
 also result from derivation builds, can be available as substitutes.
 
 @menu
-* Official Substitute Server::      One particular source of substitutes.
-* Substitute Server Authorization:: How to enable or disable substitutes.
-* Substitute Authentication::       How Guix verifies substitutes.
-* Proxy Settings::                  How to get substitutes via proxy.
-* Substitution Failure::            What happens when substitution fails.
-* On Trusting Binaries::            How can you trust that binary blob?
+* Official Substitute Server::  One particular source of substitutes.
+* Substitute Server Authorization::  How to enable or disable substitutes.
+* Substitute Authentication::   How Guix verifies substitutes.
+* Proxy Settings::              How to get substitutes via proxy.
+* Substitution Failure::        What happens when substitution fails.
+* On Trusting Binaries::        How can you trust that binary blob?
 @end menu
 
 @node Official Substitute Server
@@ -2746,7 +2748,8 @@ the distribution currently available on your local machine.  To update
 that distribution, along with the Guix tools, you must run @command{guix
 pull}: the command downloads the latest Guix source code and package
 descriptions, and deploys it.  Source code is downloaded from a
-@uref{https://git-scm.com, Git} repository.
+@uref{https://git-scm.com, Git} repository, by default the official
+GNU@tie{}Guix repository, though this can be customized.
 
 On completion, @command{guix package} will use packages and package
 versions from this just-retrieved copy of Guix.  Not only that, but all
@@ -2821,20 +2824,23 @@ but it supports the following options:
 Produce verbose output, writing build logs to the standard error output.
 
 @item --url=@var{url}
-Download Guix from the Git repository at @var{url}.
+@itemx --commit=@var{commit}
+@itemx --branch=@var{branch}
+Download code from the specified @var{url}, at the given @var{commit} (a valid
+Git commit ID represented as a hexadecimal string), or @var{branch}.
 
-@vindex GUIX_PULL_URL
-By default, the source is taken from its canonical Git repository at
-@code{gnu.org}, for the stable branch of Guix.  To use a different source,
-set the @code{GUIX_PULL_URL} environment variable.
+@cindex @file{channels.scm}, configuration file
+@cindex configuration file for channels
+These options are provided for convenience, but you can also specify your
+configuration in the @file{~/.config/guix/channels.scm} file or using the
+@option{--channels} option (see below).
 
-@item --commit=@var{commit}
-Deploy @var{commit}, a valid Git commit ID represented as a hexadecimal
-string.
-
-@item --branch=@var{branch}
-Deploy the tip of @var{branch}, the name of a Git branch available on
-the repository at @var{url}.
+@item --channels=@var{file}
+@itemx -C @var{file}
+Read the list of channels from @var{file} instead of
+@file{~/.config/guix/channels.scm}.  @var{file} must contain Scheme code that
+evaluates to a list of channel objects.  @xref{Channels}, for more
+information.
 
 @item --list-generations[=@var{pattern}]
 @itemx -l [@var{pattern}]
@@ -2848,9 +2854,178 @@ Use the bootstrap Guile to build the latest Guix.  This option is only
 useful to Guix developers.
 @end table
 
+The @dfn{channel} mechanism allows you to instruct @command{guix pull} which
+repository and branch to pull from, as well as @emph{additional} repositories
+containing package modules that should be deployed.  @xref{Channels}, for more
+information.
+
 In addition, @command{guix pull} supports all the common build options
 (@pxref{Common Build Options}).
 
+@node Channels
+@section Channels
+
+@cindex @file{channels.scm}, configuration file
+@cindex configuration file for channels
+@cindex @command{guix pull}, configuration file
+@cindex configuration of @command{guix pull}
+Guix and its package collection are updated by running @command{guix pull}
+(@pxref{Invoking guix pull}).  By default @command{guix pull} downloads and
+deploys Guix itself from the official GNU@tie{}Guix repository.  This can be
+customized by defining @dfn{channels} in the
+@file{~/.config/guix/channels.scm} file.  A channel specifies a URL and branch
+of a Git repository to be deployed, and @command{guix pull} can be instructed
+to pull from one or more channels.  In other words, channels can be used to
+@emph{customize} and to @emph{extend} Guix, as we will see below.
+
+@subsection Using a Custom Guix Channel
+
+The channel called @code{guix} specifies where Guix itself---its command-line
+tools as well as its package collection---should be downloaded.  For instance,
+suppose you want to update from your own copy of the Guix repository at
+@code{example.org}, and specifically the @code{super-hacks} branch, you can
+write in @code{~/.config/guix/channels.scm} this specification:
+
+@lisp
+;; Tell 'guix pull' to use my own repo.
+(list (channel
+        (name 'guix)
+        (url "https://example.org/my-guix.git")
+        (branch "super-hacks")))
+@end lisp
+
+@noindent
+From there on, @command{guix pull} will fetch code from the @code{super-hacks}
+branch of the repository at @code{example.org}.
+
+@subsection Specifying Additional Channels
+
+@cindex extending the package collection (channels)
+@cindex personal packages (channels)
+@cindex channels, for personal packages
+You can also specify @emph{additional channels} to pull from.  Let's say you
+have a bunch of custom package variants or personal packages that you think
+would make little sense to contribute to the Guix project, but would like to
+have these packages transparently available to you at the command line.  You
+would first write modules containing those package definitions (@pxref{Package
+Modules}), maintain them in a Git repository, and then you and anyone else can
+use it as an additional channel to get packages from.  Neat, no?
+
+@c What follows stems from discussions at
+@c <https://debbugs.gnu.org/cgi/bugreport.cgi?bug=22629#134> as well as
+@c earlier discussions on guix-devel@gnu.org.
+@quotation Warning
+Before you, dear user, shout---``woow this is @emph{soooo coool}!''---and
+publish your personal channel to the world, we would like to share a few words
+of caution:
+
+@itemize
+@item
+Before publishing a channel, please consider contributing your package
+definitions to Guix proper (@pxref{Contributing}).  Guix as a project is open
+to free software of all sorts, and packages in Guix proper are readily
+available to all Guix users and benefit from the project's quality assurance
+process.
+
+@item
+When you maintain package definitions outside Guix, we, Guix developers,
+consider that @emph{the compatibility burden is on you}.  Remember that
+package modules and package definitions are just Scheme code that uses various
+programming interfaces (APIs).  We want to remain free to change these APIs to
+keep improving Guix, possibly in ways that break your channel.  We never
+change APIs gratuitously, but we will @emph{not} commit to freezing APIs
+either.
+
+@item
+Corollary: if you're using an external channel and that channel breaks, please
+@emph{report the issue to the channel authors}, not to the Guix project.
+@end itemize
+
+You've been warned!  Having said this, we believe external channels are a
+practical way to exert your freedom to augment Guix' package collection and to
+share your improvements, which are basic tenets of
+@uref{https://www.gnu.org/philosophy/free-sw.html, free software}.
+@end quotation
+
+Once you have a Git repository containing your own package modules, you can
+write @code{~/.config/guix/channels.scm} to instruct @command{guix pull} to
+pull from your personal channel @emph{in addition} to the default Guix
+channel(s):
+
+@vindex %default-channels
+@lisp
+;; Add my personal packages to those Guix provides.
+(cons (channel
+        (name 'my-personal-packages)
+        (url "https://example.org/personal-packages.git"))
+      %default-channels)
+@end lisp
+
+@noindent
+Note that the snippet above is (as always!) Scheme code; we use @code{cons} to
+add a channel the list of channels that the variable @code{%default-channels}
+is bound to (@pxref{Pairs, @code{cons} and lists,, guile, GNU Guile Reference
+Manual}).  With this file in place, @command{guix pull} builds not only Guix
+but also the package modules from your own repository.  The result in
+@file{~/.config/guix/current} is the union of Guix with your own package
+modules:
+
+@example
+$ guix pull --list-generations
+@dots{}
+Generation 19	Aug 27 2018 16:20:48
+  guix d894ab8
+    repository URL: https://git.savannah.gnu.org/git/guix.git
+    branch: master
+    commit: d894ab8e9bfabcefa6c49d9ba2e834dd5a73a300
+  my-personal-packages dd3df5e
+    repository URL: https://example.org/personal-packages.git
+    branch: master
+    commit: dd3df5e2c8818760a8fc0bd699e55d3b69fef2bb
+  11 new packages: my-gimp, my-emacs-with-cool-features, @dots{}
+  4 packages upgraded: emacs-racket-mode@@0.0.2-2.1b78827, @dots{}
+@end example
+
+@noindent
+The output of @command{guix pull} above shows that Generation@tie{}19 includes
+both Guix and packages from the @code{my-personal-packages} channel.  Among
+the new and upgraded packages that are listed, some like @code{my-gimp} and
+@code{my-emacs-with-cool-features} might come from
+@code{my-personal-packages}, while others come from the Guix default channel.
+
+@subsection Replicating Guix
+
+@cindex pinning, channels
+@cindex replicating Guix
+@cindex reproducibility, of Guix
+The @command{guix pull --list-generations} output above shows precisely which
+commits were used to build this instance of Guix.  We can thus replicate it,
+say, on another machine, by providing a channel specification in
+@file{~/.config/guix/channels.scm} that is ``pinned'' to these commits:
+
+@lisp
+;; Deploy specific commits of my channels of interest.
+(list (channel
+       (name 'guix)
+       (url "https://git.savannah.gnu.org/git/guix.git")
+       (commit "d894ab8e9bfabcefa6c49d9ba2e834dd5a73a300"))
+      (channel
+       (name 'my-personal-packages)
+       (url "https://example.org/personal-packages.git")
+       (branch "dd3df5e2c8818760a8fc0bd699e55d3b69fef2bb")))
+@end lisp
+
+At this point the two machines run the @emph{exact same Guix}, with access to
+the @emph{exact same packages}.  The output of @command{guix build gimp} on
+one machine will be exactly the same, bit for bit, as the output of the same
+command on the other machine.  It also means both machines have access to all
+the source code of Guix and, transitively, to all the source code of every
+package it defines.
+
+This gives you super powers, allowing you to track the provenance of binary
+artifacts with very fine grain, and to reproduce software environments at
+will---some sort of ``meta reproducibility'' capabilities, if you will.
+
 @node Invoking guix pack
 @section Invoking @command{guix pack}
 
@@ -3431,9 +3606,9 @@ more information on how to test package definitions, and
 @ref{Invoking guix lint}, for information on how to check a definition
 for style conformance.
 @vindex GUIX_PACKAGE_PATH
-Lastly, @pxref{Package Modules}, for information
+Lastly, @pxref{Channels}, for information
 on how to extend the distribution by adding your own package definitions
-to @code{GUIX_PACKAGE_PATH}.
+in a ``channel''.
 
 Finally, updating the package definition to a new upstream version
 can be partly automated by the @command{guix refresh} command
@@ -6255,8 +6430,8 @@ and that of Vim.
 
 If you are using a Guix Git checkout (@pxref{Building from Git}), or
 have created your own packages on @code{GUIX_PACKAGE_PATH}
-(@pxref{Defining Packages}), you will be able to edit the package
-recipes. Otherwise, you will be able to examine the read-only recipes
+(@pxref{Package Modules}), you will be able to edit the package
+recipes.  In other cases, you will be able to examine the read-only recipes
 for packages currently in the store.
 
 
@@ -8363,7 +8538,7 @@ For information on porting to other architectures or kernels,
 @menu
 * System Installation::         Installing the whole operating system.
 * System Configuration::        Configuring the operating system.
-* Documentation::                Browsing software user manuals.
+* Documentation::               Browsing software user manuals.
 * Installing Debugging Files::  Feeding the debugger.
 * Security Updates::            Deploying security fixes quickly.
 * Package Modules::             Packages from the programmer's viewpoint.
@@ -8402,7 +8577,7 @@ available.
 @menu
 * Limitations::                 What you can expect.
 * Hardware Considerations::     Supported hardware.
-* USB Stick and DVD Installation:: Preparing the installation medium.
+* USB Stick and DVD Installation::  Preparing the installation medium.
 * Preparing for Installation::  Networking, partitioning, etc.
 * Proceeding with the Installation::  The real thing.
 * Installing GuixSD in a VM::   GuixSD playground.
@@ -10083,7 +10258,7 @@ declaration.
 * Audio Services::              The MPD.
 * Virtualization Services::     Virtualization services.
 * Version Control Services::    Providing remote access to Git repositories.
-* Game Services::                Game servers.
+* Game Services::               Game servers.
 * Miscellaneous Services::      Other services.
 @end menu
 
@@ -22520,16 +22695,24 @@ name and module name must match.  For instance, the @code{(my-packages
 emacs)} module must be stored in a @file{my-packages/emacs.scm} file
 relative to the load path specified with @option{--load-path} or
 @code{GUIX_PACKAGE_PATH}.  @xref{Modules and the File System,,,
-guile, GNU Guile Reference Manual}, for details.}.  These package definitions
-will not be visible by default.  Users can invoke commands such as
-@command{guix package} and @command{guix build} with the
-@code{-e} option so that they know where to find the package.  Better
-yet, they can use the
-@code{-L} option of these commands to make those modules visible
-(@pxref{Invoking guix build, @code{--load-path}}), or define the
-@code{GUIX_PACKAGE_PATH} environment variable.  This environment
-variable makes it easy to extend or customize the distribution and is
-honored by all the user interfaces.
+guile, GNU Guile Reference Manual}, for details.}.  There are two ways to make
+these package definitions visible to the user interfaces:
+
+@enumerate
+@item
+By adding the directory containing your package modules to the search path
+with the @code{-L} flag of @command{guix package} and other commands
+(@pxref{Common Build Options}), or by setting the @code{GUIX_PACKAGE_PATH}
+environment variable described below.
+
+@item
+By defining a @dfn{channel} and configuring @command{guix pull} so that it
+pulls from it.  A channel is essentially a Git repository containing package
+modules.  @xref{Channels}, for more information on how to define and use
+channels.
+@end enumerate
+
+@code{GUIX_PACKAGE_PATH} works similarly to other search path variables:
 
 @defvr {Environment Variable} GUIX_PACKAGE_PATH
 This is a colon-separated list of directories to search for additional
diff --git a/guix/channels.scm b/guix/channels.scm
new file mode 100644
index 000000000..ec3e05eaf
--- /dev/null
+++ b/guix/channels.scm
@@ -0,0 +1,292 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2018 Ludovic Courtès <ludo@gnu.org>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (guix channels)
+  #:use-module (guix git)
+  #:use-module (guix records)
+  #:use-module (guix gexp)
+  #:use-module (guix discovery)
+  #:use-module (guix monads)
+  #:use-module (guix profiles)
+  #:use-module (guix derivations)
+  #:use-module (guix store)
+  #:use-module (guix i18n)
+  #:use-module (srfi srfi-1)
+  #:use-module (srfi srfi-9)
+  #:use-module (srfi srfi-11)
+  #:autoload   (guix self) (whole-package)
+  #:use-module (ice-9 match)
+  #:export (channel
+            channel?
+            channel-name
+            channel-url
+            channel-branch
+            channel-commit
+            channel-location
+
+            %default-channels
+
+            channel-instance?
+            channel-instance-channel
+            channel-instance-commit
+            channel-instance-checkout
+
+            latest-channel-instances
+            channel-instance-derivations
+            latest-channel-derivations
+            channel-instances->manifest))
+
+;;; Commentary:
+;;;
+;;; This module implements "channels."  A channel is usually a source of
+;;; package definitions.  There's a special channel, the 'guix' channel, that
+;;; provides all of Guix, including its commands and its documentation.
+;;; User-defined channels are expected to typically provide a bunch of .scm
+;;; files meant to be added to the '%package-search-path'.
+;;;
+;;; This module provides tools to fetch and update channels from a Git
+;;; repository and to build them.
+;;;
+;;; Code:
+
+(define-record-type* <channel> channel make-channel
+  channel?
+  (name      channel-name)
+  (url       channel-url)
+  (branch    channel-branch (default "master"))
+  (commit    channel-commit (default #f))
+  (location  channel-location
+             (default (current-source-location)) (innate)))
+;; TODO: Add a way to express dependencies among channels.
+
+(define %default-channels
+  ;; Default list of channels.
+  (list (channel
+         (name 'guix)
+         (branch "origin/master")
+         (url "https://git.savannah.gnu.org/git/guix.git"))))
+
+(define (guix-channel? channel)
+  "Return true if CHANNEL is the 'guix' channel."
+  (eq? 'guix (channel-name channel)))
+
+(define-record-type <channel-instance>
+  (channel-instance channel commit checkout)
+  channel-instance?
+  (channel   channel-instance-channel)
+  (commit    channel-instance-commit)
+  (checkout  channel-instance-checkout))
+
+(define (channel-reference channel)
+  "Return the \"reference\" for CHANNEL, an sexp suitable for
+'latest-repository-commit'."
+  (match (channel-commit channel)
+    (#f      `(branch . ,(channel-branch channel)))
+    (commit  `(commit . ,(channel-commit channel)))))
+
+(define (latest-channel-instances store channels)
+  "Return a list of channel instances corresponding to the latest checkouts of
+CHANNELS."
+  (map (lambda (channel)
+         (format (current-error-port)
+                 (G_ "Updating channel '~a' from Git repository at '~a'...~%")
+                 (channel-name channel)
+                 (channel-url channel))
+         (let-values (((checkout commit)
+                       (latest-repository-commit store (channel-url channel)
+                                                 #:ref (channel-reference
+                                                        channel))))
+           (channel-instance channel commit checkout)))
+       channels))
+
+(define %self-build-file
+  ;; The file containing code to build Guix.  This serves the same purpose as
+  ;; a makefile, and, similarly, is intended to always keep this name.
+  "build-aux/build-self.scm")
+
+(define %pull-version
+  ;; This is the version of the 'guix pull' protocol.  It specifies what's
+  ;; expected from %SELF-BUILD-FILE.  The initial version ("0") was when we'd
+  ;; place a set of compiled Guile modules in ~/.config/guix/latest.
+  1)
+
+(define (standard-module-derivation name source dependencies)
+  "Return a derivation that builds the Scheme modules in SOURCE and that
+depend on DEPENDENCIES, a list of lowerable objects.  The assumption is that
+SOURCE contains package modules to be added to '%package-module-path'."
+  (define modules
+    (scheme-modules* source))
+
+  ;; FIXME: We should load, say SOURCE/.guix-channel.scm, which would allow
+  ;; channel publishers to specify things such as the sub-directory where .scm
+  ;; files live, files to exclude from the channel, preferred substitute URLs,
+  ;; etc.
+  (mlet* %store-monad ((compiled
+                        (compiled-modules modules
+                                          #:name name
+                                          #:module-path (list source)
+                                          #:extensions dependencies)))
+
+    (gexp->derivation name
+                      (with-extensions dependencies
+                        (with-imported-modules '((guix build utils))
+                          #~(begin
+                              (use-modules (guix build utils))
+
+                              (let ((go  (string-append #$output "/lib/guile/"
+                                                        (effective-version)
+                                                        "/site-ccache"))
+                                    (scm (string-append #$output
+                                                        "/share/guile/site/"
+                                                        (effective-version))))
+                                (mkdir-p (dirname go))
+                                (symlink #$compiled go)
+                                (mkdir-p (dirname scm))
+                                (symlink #$source scm))))))))
+
+(define* (build-from-source name source
+                            #:key verbose? commit
+                            (dependencies '()))
+  "Return a derivation to build Guix from SOURCE, using the self-build script
+contained therein.  Use COMMIT as the version string."
+  ;; Running the self-build script makes it easier to update the build
+  ;; procedure: the self-build script of the Guix-to-be-installed contains the
+  ;; right dependencies, build procedure, etc., which the Guix-in-use may not
+  ;; be know.
+  (define script
+    (string-append source "/" %self-build-file))
+
+  (if (file-exists? script)
+      (let ((build (save-module-excursion
+                    (lambda ()
+                      (primitive-load script)))))
+        ;; BUILD must be a monadic procedure of at least one argument: the
+        ;; source tree.
+        ;;
+        ;; Note: BUILD can return #f if it does not support %PULL-VERSION.  In
+        ;; the future we'll fall back to a previous version of the protocol
+        ;; when that happens.
+        (build source #:verbose? verbose? #:version commit
+               #:pull-version %pull-version))
+
+      ;; Build a set of modules that extend Guix using the standard method.
+      (standard-module-derivation name source dependencies)))
+
+(define* (build-channel-instance instance #:optional (dependencies '()))
+  "Return, as a monadic value, the derivation for INSTANCE, a channel
+instance.  DEPENDENCIES is a list of extensions providing Guile modules that
+INSTANCE depends on."
+  (build-from-source (symbol->string
+                      (channel-name (channel-instance-channel instance)))
+                     (channel-instance-checkout instance)
+                     #:commit (channel-instance-commit instance)
+                     #:dependencies dependencies))
+
+(define (channel-instance-derivations instances)
+  "Return the list of derivations to build INSTANCES, in the same order as
+INSTANCES."
+  (define core-instance
+    ;; The 'guix' channel is treated specially: it's an implicit dependency of
+    ;; all the other channels.
+    (find (lambda (instance)
+            (guix-channel? (channel-instance-channel instance)))
+          instances))
+
+  (mlet %store-monad ((core (build-channel-instance core-instance)))
+    (mapm %store-monad
+          (lambda (instance)
+            (if (eq? instance core-instance)
+                (return core)
+                (build-channel-instance instance
+                                        (list core))))
+          instances)))
+
+(define latest-channel-derivations
+  (let ((latest-channel-instances (store-lift latest-channel-instances)))
+    (lambda (channels)
+      "Return, as a monadic value, the list of derivations for the latest
+instances of CHANNELS."
+      (mlet %store-monad ((instances (latest-channel-instances channels)))
+        (channel-instance-derivations instances)))))
+
+(define (whole-package-for-legacy name modules)
+  "Return a full-blown Guix package for MODULES, a derivation that builds Guix
+modules in the old ~/.config/guix/latest style."
+  (define packages
+    (resolve-interface '(gnu packages guile)))
+
+  (letrec-syntax ((list (syntax-rules (->)
+                          ((_)
+                           '())
+                          ((_ (module -> variable) rest ...)
+                           (cons (module-ref (resolve-interface
+                                              '(gnu packages module))
+                                             'variable)
+                                 (list rest ...)))
+                          ((_ variable rest ...)
+                           (cons (module-ref packages 'variable)
+                                 (list rest ...))))))
+    (whole-package name modules
+
+                   ;; In the "old style", %SELF-BUILD-FILE would simply return a
+                   ;; derivation that builds modules.  We have to infer what the
+                   ;; dependencies of these modules were.
+                   (list guile-json guile-git guile-bytestructures
+                         (ssh -> guile-ssh) (tls -> gnutls)))))
+
+(define (old-style-guix? drv)
+  "Return true if DRV corresponds to a ~/.config/guix/latest style of
+derivation."
+  ;; Here we rely on a gross historical fact: that derivations produced by the
+  ;; "old style" (before commit 8a0d9bc8a3f153159d9e239a151c0fa98f1e12d8,
+  ;; dated May 30, 2018) did not depend on "guix-command.drv".
+  (not (find (lambda (input)
+               (string-suffix? "-guix-command.drv"
+                               (derivation-input-path input)))
+             (derivation-inputs drv))))
+
+(define (channel-instances->manifest instances)
+  "Return a profile manifest with entries for all of INSTANCES, a list of
+channel instances."
+  (define instance->entry
+    (match-lambda
+      ((instance drv)
+       (let ((commit  (channel-instance-commit instance))
+             (channel (channel-instance-channel instance)))
+         (with-monad %store-monad
+           (return (manifest-entry
+                     (name (symbol->string (channel-name channel)))
+                     (version (string-take commit 7))
+                     (item (if (guix-channel? channel)
+                               (if (old-style-guix? drv)
+                                   (whole-package-for-legacy
+                                    (string-append name "-" version)
+                                    drv)
+                                   drv)
+                               drv))
+                     (properties
+                      `((source (repository
+                                 (version 0)
+                                 (url ,(channel-url channel))
+                                 (branch ,(channel-branch channel))
+                                 (commit ,commit))))))))))))
+
+  (mlet* %store-monad ((derivations (channel-instance-derivations instances))
+                       (entries     (mapm %store-monad instance->entry
+                                          (zip instances derivations))))
+    (return (manifest entries))))
diff --git a/guix/scripts/pull.scm b/guix/scripts/pull.scm
index ee68c21a4..8400ad278 100644
--- a/guix/scripts/pull.scm
+++ b/guix/scripts/pull.scm
@@ -30,26 +30,19 @@
   #:use-module (guix grafts)
   #:use-module (guix memoization)
   #:use-module (guix monads)
+  #:use-module (guix channels)
   #:autoload   (guix inferior) (open-inferior)
   #:use-module (guix scripts build)
-  #:autoload   (guix self) (whole-package)
   #:use-module (guix git)
   #:use-module (git)
   #:use-module (gnu packages)
-  #:autoload   (gnu packages ssh) (guile-ssh)
-  #:autoload   (gnu packages tls) (gnutls)
   #:use-module ((guix scripts package) #:select (build-and-use-profile))
-  #:use-module ((guix build utils)
-                #:select (with-directory-excursion delete-file-recursively))
-  #:use-module ((guix build download)
-                #:select (%x509-certificate-directory))
   #:use-module (gnu packages base)
   #:use-module (gnu packages guile)
   #:use-module ((gnu packages bootstrap)
                 #:select (%bootstrap-guile))
   #:use-module ((gnu packages certs) #:select (le-certs))
   #:use-module (srfi srfi-1)
-  #:use-module (srfi srfi-11)
   #:use-module (srfi srfi-26)
   #:use-module (srfi srfi-35)
   #:use-module (srfi srfi-37)
@@ -57,9 +50,6 @@
   #:use-module (ice-9 vlist)
   #:export (guix-pull))
 
-(define %repository-url
-  (or (getenv "GUIX_PULL_URL") "https://git.savannah.gnu.org/git/guix.git"))
-
 \f
 ;;;
 ;;; Command-line options.
@@ -67,9 +57,7 @@
 
 (define %default-options
   ;; Alist of default option values.
-  `((repository-url . ,%repository-url)
-    (ref . (branch . "origin/master"))
-    (system . ,(%current-system))
+  `((system . ,(%current-system))
     (substitutes? . #t)
     (build-hook? . #t)
     (graft? . #t)
@@ -80,6 +68,8 @@
 Download and deploy the latest version of Guix.\n"))
   (display (G_ "
       --verbose          produce verbose output"))
+  (display (G_ "
+  -C, --channels=FILE    deploy the channels defined in FILE"))
   (display (G_ "
       --url=URL          download from the Git repository at URL"))
   (display (G_ "
@@ -105,6 +95,9 @@ Download and deploy the latest version of Guix.\n"))
   (cons* (option '("verbose") #f #f
                  (lambda (opt name arg result)
                    (alist-cons 'verbose? #t result)))
+         (option '(#\C "channels") #t #f
+                 (lambda (opt name arg result)
+                   (alist-cons 'channel-file arg result)))
          (option '(#\l "list-generations") #f #t
                  (lambda (opt name arg result)
                    (cons `(query list-generations ,(or arg ""))
@@ -142,70 +135,6 @@ Download and deploy the latest version of Guix.\n"))
 (define indirect-root-added
   (store-lift add-indirect-root))
 
-(define %self-build-file
-  ;; The file containing code to build Guix.  This serves the same purpose as
-  ;; a makefile, and, similarly, is intended to always keep this name.
-  "build-aux/build-self.scm")
-
-(define %pull-version
-  ;; This is the version of the 'guix pull' protocol.  It specifies what's
-  ;; expected from %SELF-BUILD-FILE.  The initial version ("0") was when we'd
-  ;; place a set of compiled Guile modules in ~/.config/guix/latest.
-  1)
-
-(define* (build-from-source source
-                            #:key verbose? commit)
-  "Return a derivation to build Guix from SOURCE, using the self-build script
-contained therein.  Use COMMIT as the version string."
-  ;; Running the self-build script makes it easier to update the build
-  ;; procedure: the self-build script of the Guix-to-be-installed contains the
-  ;; right dependencies, build procedure, etc., which the Guix-in-use may not
-  ;; be know.
-  (let* ((script (string-append source "/" %self-build-file))
-         (build  (primitive-load script)))
-    ;; BUILD must be a monadic procedure of at least one argument: the source
-    ;; tree.
-    ;;
-    ;; Note: BUILD can return #f if it does not support %PULL-VERSION.  In the
-    ;; future we'll fall back to a previous version of the protocol when that
-    ;; happens.
-    (build source #:verbose? verbose? #:version commit
-           #:pull-version %pull-version)))
-
-(define (whole-package-for-legacy name modules)
-  "Return a full-blown Guix package for MODULES, a derivation that builds Guix
-modules in the old ~/.config/guix/latest style."
-  (whole-package name modules
-
-                 ;; In the "old style", %SELF-BUILD-FILE would simply return a
-                 ;; derivation that builds modules.  We have to infer what the
-                 ;; dependencies of these modules were.
-                 (list guile-json guile-git guile-bytestructures
-                       guile-ssh gnutls)))
-
-(define* (derivation->manifest-entry drv
-                                     #:key url branch commit)
-  "Return a manifest entry for DRV, which represents Guix at COMMIT.  Record
-URL, BRANCH, and COMMIT as a property in the manifest entry."
-  (mbegin %store-monad
-    (what-to-build (list drv))
-    (built-derivations (list drv))
-    (let ((out (derivation->output-path drv)))
-      (return (manifest-entry
-                (name "guix")
-                (version (string-take commit 7))
-                (item (if (file-exists? (string-append out "/bin/guix"))
-                          drv
-                          (whole-package-for-legacy (string-append name "-"
-                                                                   version)
-                                                    drv)))
-                (properties
-                 `((source (repository
-                            (version 0)
-                            (url ,url)
-                            (branch ,branch)
-                            (commit ,commit))))))))))
-
 (define (display-profile-news profile)
   "Display what's up in PROFILE--new packages, and all that."
   (match (memv (generation-number profile)
@@ -223,8 +152,8 @@ URL, BRANCH, and COMMIT as a property in the manifest entry."
                                       #:heading (G_ "New in this revision:\n"))))
     (_ #t)))
 
-(define* (build-and-install source config-dir
-                            #:key verbose? url branch commit)
+(define* (build-and-install instances config-dir
+                            #:key verbose?)
   "Build the tool from SOURCE, and install it in CONFIG-DIR."
   (define update-profile
     (store-lift build-and-use-profile))
@@ -232,15 +161,9 @@ URL, BRANCH, and COMMIT as a property in the manifest entry."
   (define profile
     (string-append config-dir "/current"))
 
-  (mlet* %store-monad ((drv   (build-from-source source
-                                                 #:commit commit
-                                                 #:verbose? verbose?))
-                       (entry (derivation->manifest-entry drv
-                                                          #:url url
-                                                          #:branch branch
-                                                          #:commit commit)))
+  (mlet %store-monad ((manifest (channel-instances->manifest instances)))
     (mbegin %store-monad
-      (update-profile profile (manifest (list entry)))
+      (update-profile profile manifest)
       (return (display-profile-news profile)))))
 
 (define (honor-lets-encrypt-certificates! store)
@@ -426,45 +349,106 @@ and ALIST2 differ, display HEADING upfront."
                ((numbers ...)
                 (list-generations profile numbers)))))))))
 
+(define (channel-list opts)
+  "Return the list of channels to use.  If OPTS specify a channel file,
+channels are read from there; otherwise, if ~/.config/guix/channels.scm
+exists, read it; otherwise %DEFAULT-CHANNELS is used.  Apply channel
+transformations specified in OPTS (resulting from '--url', '--commit', or
+'--branch'), if any."
+  (define file
+    (assoc-ref opts 'channel-file))
+
+  (define default-file
+    (string-append (config-directory) "/channels.scm"))
+
+  (define (load-channels file)
+    (let ((result (load* file (make-user-module '((guix channels))))))
+      (if (and (list? result) (every channel? result))
+          result
+          (leave (G_ "'~a' did not return a list of channels~%") file))))
+
+  (define channels
+    (cond (file
+           (load-channels file))
+          ((file-exists? default-file)
+           (load-channels default-file))
+          (else
+           %default-channels)))
+
+  (define (environment-variable)
+    (match (getenv "GUIX_PULL_URL")
+      (#f #f)
+      (url
+       (warning (G_ "The 'GUIX_PULL_URL' environment variable is deprecated.
+Use '~/.config/guix/channels.scm' instead."))
+       url)))
+
+  (let ((ref (assoc-ref opts 'ref))
+        (url (or (assoc-ref opts 'url)
+                 (environment-variable))))
+    (if (or ref url)
+        (match channels
+          ((one)
+           ;; When there's only one channel, apply '--url', '--commit', and
+           ;; '--branch' to this specific channel.
+           (let ((url (or url (channel-url one))))
+             (list (match ref
+                     (('commit . commit)
+                      (channel (inherit one)
+                               (url url) (commit commit) (branch #f)))
+                     (('branch . branch)
+                      (channel (inherit one)
+                               (url url) (commit #f) (branch branch)))
+                     (#f
+                      (channel (inherit one) (url url)))))))
+          (_
+           ;; Otherwise bail out.
+           (leave
+            (G_ "'--url', '--commit', and '--branch' are not applicable~%"))))
+        channels)))
+
 \f
 (define (guix-pull . args)
-  (define (use-le-certs? url)
-    (string-prefix? "https://git.savannah.gnu.org/" url))
-
   (with-error-handling
     (with-git-error-handling
-     (let* ((opts  (parse-command-line args %options
-                                       (list %default-options)))
-            (url   (assoc-ref opts 'repository-url))
-            (ref   (assoc-ref opts 'ref))
-            (cache (string-append (cache-directory) "/pull")))
+     (let* ((opts     (parse-command-line args %options
+                                          (list %default-options)))
+            (cache    (string-append (cache-directory) "/pull"))
+            (channels (channel-list opts)))
+
        (cond ((assoc-ref opts 'query)
               (process-query opts))
              ((assoc-ref opts 'dry-run?)
               #t)                                 ;XXX: not very useful
              (else
               (with-store store
-                (parameterize ((%graft? (assoc-ref opts 'graft?)))
+                (parameterize ((%graft? (assoc-ref opts 'graft?))
+                               (%repository-cache-directory cache))
                   (set-build-options-from-command-line store opts)
 
-                  ;; For reproducibility, always refer to the LE certificates
-                  ;; when we know we're talking to Savannah.
-                  (when (use-le-certs? url)
-                    (honor-lets-encrypt-certificates! store))
-
-                  (format (current-error-port)
-                          (G_ "Updating from Git repository at '~a'...~%")
-                          url)
-
-                  (let-values (((checkout commit)
-                                (latest-repository-commit store url
-                                                          #:ref ref
-                                                          #:cache-directory
-                                                          cache)))
+                  ;; When certificates are already installed, use them.
+                  ;; Otherwise, use the Let's Encrypt certificates, which we
+                  ;; know Savannah uses.
+                  (let ((certs (or (getenv "SSL_CERT_DIR") "/etc/ssl/certs")))
+                    (unless (file-exists? certs)
+                      (honor-lets-encrypt-certificates! store)))
 
+                  (let ((instances (latest-channel-instances store channels)))
                     (format (current-error-port)
-                            (G_ "Building from Git commit ~a...~%")
-                            commit)
+                            (N_ "Building from this channel:~%"
+                                "Building from these channels:~%"
+                                (length instances)))
+                    (for-each (lambda (instance)
+                                (let ((channel
+                                       (channel-instance-channel instance)))
+                                  (format (current-error-port)
+                                          "  ~10a~a\t~a~%"
+                                          (channel-name channel)
+                                          (channel-url channel)
+                                          (string-take
+                                           (channel-instance-commit instance)
+                                           7))))
+                              instances)
                     (parameterize ((%guile-for-build
                                     (package-derivation
                                      store
@@ -472,13 +456,7 @@ and ALIST2 differ, display HEADING upfront."
                                          %bootstrap-guile
                                          (canonical-package guile-2.2)))))
                       (run-with-store store
-                        (build-and-install checkout (config-directory)
-                                           #:url url
-                                           #:branch (match ref
-                                                      (('branch . branch)
-                                                       branch)
-                                                      (_ #f))
-                                           #:commit commit
+                        (build-and-install instances (config-directory)
                                            #:verbose?
                                            (assoc-ref opts 'verbose?)))))))))))))
 
diff --git a/po/guix/POTFILES.in b/po/guix/POTFILES.in
index d11f408d4..7f881355e 100644
--- a/po/guix/POTFILES.in
+++ b/po/guix/POTFILES.in
@@ -38,4 +38,5 @@ guix/upstream.scm
 guix/ui.scm
 guix/http-client.scm
 guix/nar.scm
+guix/channels.scm
 nix/nix-daemon/guix-daemon.cc
-- 
2.18.0

bug#22629: “Stable” branch

[Ludovic Courtès]

Hi Alex,

(Cc’ing <https://bugs.gnu.org/32022> and <https://bugs.gnu.org/26608>,
which are related.)

Alex Sassmannshausen <alex@pompo.co> skribis:

> I don't know if this is what Konrad desires, but from my perspective, a
> desirable part of the definition of stable would be a that the build
> farms have produced a set of binaries/substitutes for a given Guix
> revision that is "good enough".

I just had a bright idea (yes!): this can be addressed by writing
something like this in ~/.config/guix/channels.scm:

  (map latest-commit-with-substitutes-available
       %default-channels)

The hypothetical ‘latest-commit-with-substitutes-available’ would use
(git) and (guix ci) to find the latest commit for which substitutes of
interest are available, and would return:

  (channel
    ;; …
    (commit "cabbag3"))   ;the ideal commit

This has to be done with great care to prevent a downgrade attack and to
make sure the user doesn’t miss out on security updates, but maybe we
could provide a procedure that makes reasonable choices.

Food for thought…

Ludo’.

bug#22629: “Stable” branch

[Konrad Hinsen]

Hi Ludo,

> I just had a bright idea (yes!): this can be addressed by writing
> something like this in ~/.config/guix/channels.scm:
>
>   (map latest-commit-with-substitutes-available
>        %default-channels)
>
> The hypothetical ‘latest-commit-with-substitutes-available’ would use
> (git) and (guix ci) to find the latest commit for which substitutes of
> interest are available, and would return:

I really like that idea, but it's a pity to limit it to channels.
Two scenarii I'd like to see covered are:

 1) Find the latest commit with all substitutes required by a given
    manifest.

 2) Find the latest commit with all substitutes required for updating a
    given profile.

This is in fact only one problem with two user interfaces.

Konrad.

bug#26608: bug#22629: “Stable” branch

[Ludovic Courtès]

Hi Konrad,

Konrad Hinsen <konrad.hinsen@fastmail.net> skribis:

>> I just had a bright idea (yes!): this can be addressed by writing
>> something like this in ~/.config/guix/channels.scm:
>>
>>   (map latest-commit-with-substitutes-available
>>        %default-channels)
>>
>> The hypothetical ‘latest-commit-with-substitutes-available’ would use
>> (git) and (guix ci) to find the latest commit for which substitutes of
>> interest are available, and would return:
>
> I really like that idea, but it's a pity to limit it to channels.

What do you mean by “limit it to channels”?  ‘%default-channels’ is an
alias for the official Guix channel (IOW, Guix itself.)

> Two scenarii I'd like to see covered are:
>
>  1) Find the latest commit with all substitutes required by a given
>     manifest.
>
>  2) Find the latest commit with all substitutes required for updating a
>     given profile.
>
> This is in fact only one problem with two user interfaces.

Yes, we could do that, and even maybe more sophisticated things (e.g.,
looking at the commit log to determine whether security fixes are
available, and adjusting the strategy accordingly.)

What I find interesting is that we can provide the tools to support such
policies, and then users can choose or implement the policy they want
directly in ~/.config/guix/channels.scm.

Ludo’.

bug#32022: bug#22629: “Stable” branch

[Konrad Hinsen]

Hi Ludo,

> What do you mean by “limit it to channels”?  ‘%default-channels’ is an
> alias for the official Guix channel (IOW, Guix itself.)

Fine, but I rarely care about all of Guix, or all of any other channel.
I care about the small subset of packages that I actually use.

Better yet, with a per-manifest/profile approach, I could put my most
critical packages in a special profile and get updates for them more
quickly, while still working only with substitutes.

BTW, just out of curiosity: for how many commits in Guix history all
packages could be built successfully? Is that the rule of the exception?

> Yes, we could do that, and even maybe more sophisticated things (e.g.,
> looking at the commit log to determine whether security fixes are
> available, and adjusting the strategy accordingly.)

Nice!

> What I find interesting is that we can provide the tools to support such
> policies, and then users can choose or implement the policy they want
> directly in ~/.config/guix/channels.scm.

I agree, it's nice to give people the tools they need to implement their
own policy.

Konrad.

bug#26608: bug#22629: “Stable” branch

[Jan Nieuwenhuizen]

Ludovic Courtès writes:

> I just had a bright idea (yes!): this can be addressed by writing
> something like this in ~/.config/guix/channels.scm:
>
>   (map latest-commit-with-substitutes-available
>        %default-channels)

This is a nice idea and it makes me remember that it would be useful to
provide a way to avoid installing something that is cricitally broken,
like Debian's apt-listbugs package/facility
(https://packages.debian.org/sid/apt-listbugs).

janneke

-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com

bug#32022: bug#22629: “Stable” branch

[Ricardo Wurmus]

Ludovic Courtès <ludo@gnu.org> writes:

> I just had a bright idea (yes!): this can be addressed by writing
> something like this in ~/.config/guix/channels.scm:
>
>   (map latest-commit-with-substitutes-available
>        %default-channels)
>
> The hypothetical ‘latest-commit-with-substitutes-available’ would use
> (git) and (guix ci) to find the latest commit for which substitutes of
> interest are available, and would return:
>
>   (channel
>     ;; …
>     (commit "cabbag3"))   ;the ideal commit
>
> This has to be done with great care to prevent a downgrade attack and to
> make sure the user doesn’t miss out on security updates, but maybe we
> could provide a procedure that makes reasonable choices.

This is a great idea.  Any kind of fetch policy could be implemented
with this, including one that considers the contents of a manifest.

This is another of these instances where having a general purpose
programming language underpinning it all really pays off.

--
Ricardo

bug#22629: [PATCH v2 2/3] Add (guix describe) and use it to initialize '%package-search-path'.

[Ricardo Wurmus]

Hi Ludo,

> * guix/describe.scm: New file.
> * Makefile.am (MODULES): Add it.
> * gnu/packages.scm (%default-package-module-path): New variable.
> (%package-module-path): Honor 'package-path-entries'.
> * build-aux/update-NEWS.scm (main): Use %DEFAULT-PACKAGE-MODULE-PATH
> instead of (last (%package-module-path)).
> ---
[…]
> +(define %default-package-module-path
> +  ;; Default search path for package modules.
> +  `((,%distro-root-directory . "gnu/packages")))
> +
>  (define %package-module-path
>    ;; Search path for package modules.  Each item must be either a directory
>    ;; name or a pair whose car is a directory and whose cdr is a sub-directory
>    ;; to narrow the search.
>    (let* ((not-colon   (char-set-complement (char-set #\:)))
>           (environment (string-tokenize (or (getenv "GUIX_PACKAGE_PATH") "")
> -                                       not-colon)))
> -    ;; Automatically add items from $GUIX_PACKAGE_PATH to Guile's search path.
> -    (for-each (lambda (directory)
> -                (set! %load-path (cons directory %load-path))
> -                (set! %load-compiled-path
> -                      (cons directory %load-compiled-path)))
> -              environment)
> +                                       not-colon))
> +         (channels    (package-path-entries)))
> +    ;; Automatically add channels and items from $GUIX_PACKAGE_PATH to Guile's
> +    ;; search path.  For historical reasons, $GUIX_PACKAGE_PATH goes to the
> +    ;; front; channels go to the back so that they don't override Guix' own
> +    ;; modules.
> +    (set! %load-path
> +      (append environment %load-path channels))
> +    (set! %load-compiled-path
> +      (append environment %load-compiled-path channels))
>
>      (make-parameter
> -     (append environment `((,%distro-root-directory . "gnu/packages"))))))
> +     (append environment
> +             %default-package-module-path
> +             channels))))

I’m not sure I understand the reason to add channels to the end of the
search path.  Could it not be desirable in some use-cases to override
certain Guix modules?  Should the order be made explicit in the channel
to avoid having to accomodate “historical reasons” in the future? :)

Putting them at the end probably ensures that Guix itself won’t break
and can thus be rolled back.

> diff --git a/guix/describe.scm b/guix/describe.scm
> new file mode 100644
> index 000000000..3122a762f
> --- /dev/null
> +++ b/guix/describe.scm
[…]
> +(define current-profile
> +  (mlambda ()
> +    "Return the profile (created by 'guix pull') the calling process lives in,
> +or #f if this is not applicable."
> +    (match (command-line)
> +      ((program . _)
> +       (and (string-suffix? "/bin/guix" program)
> +            ;; Note: We want to do _lexical dot-dot resolution_.  Using ".."
> +            ;; for real would instead take us into the /gnu/store directory
> +            ;; that ~/.config/guix/current/bin points to, whereas we want to
> +            ;; obtain ~/.config/guix/current.
> +            (let ((candidate (dirname (dirname program))))
> +              (and (file-exists? (string-append candidate "/manifest"))
> +                   candidate)))))))

I don’t know… there’s something about this file system traversal that
doesn’t sit right with me.  I’m not sure about (command-line) — when
…/bin/guix is executed by a wrapper, will the wrapper be the “program”
that we match against or the target?  (This is a concern for wrappers
that set up site-wide default channels or a remote daemon, for example.)

--
Ricardo

bug#32022: bug#22629: “Stable” branch

[Ludovic Courtès]

Hello,

Konrad Hinsen <konrad.hinsen@fastmail.net> skribis:

>> What do you mean by “limit it to channels”?  ‘%default-channels’ is an
>> alias for the official Guix channel (IOW, Guix itself.)
>
> Fine, but I rarely care about all of Guix, or all of any other channel.
> I care about the small subset of packages that I actually use.
>
> Better yet, with a per-manifest/profile approach, I could put my most
> critical packages in a special profile and get updates for them more
> quickly, while still working only with substitutes.

Sure!  The hypothetical procedure I gave can perform arbitrary checks;
it could be passed a manifest.

> BTW, just out of curiosity: for how many commits in Guix history all
> packages could be built successfully? Is that the rule of the exception?

We never have 100% of successful builds.  Of course we do our best to
keep the failure rate low, but sometimes there are unpopular packages
that remain broken for some time, or there are packages for which we
forgot to exclude some systems via ‘supported-systems’, and of course
there’s unintended breakage.

Ludo’.

bug#22629: [PATCH v2 2/3] Add (guix describe) and use it to initialize '%package-search-path'.

[Ludovic Courtès]

Hi Ricardo,

Ricardo Wurmus <rekado@elephly.net> skribis:

>>  (define %package-module-path
>>    ;; Search path for package modules.  Each item must be either a directory
>>    ;; name or a pair whose car is a directory and whose cdr is a sub-directory
>>    ;; to narrow the search.
>>    (let* ((not-colon   (char-set-complement (char-set #\:)))
>>           (environment (string-tokenize (or (getenv "GUIX_PACKAGE_PATH") "")
>> -                                       not-colon)))
>> -    ;; Automatically add items from $GUIX_PACKAGE_PATH to Guile's search path.
>> -    (for-each (lambda (directory)
>> -                (set! %load-path (cons directory %load-path))
>> -                (set! %load-compiled-path
>> -                      (cons directory %load-compiled-path)))
>> -              environment)
>> +                                       not-colon))
>> +         (channels    (package-path-entries)))
>> +    ;; Automatically add channels and items from $GUIX_PACKAGE_PATH to Guile's
>> +    ;; search path.  For historical reasons, $GUIX_PACKAGE_PATH goes to the
>> +    ;; front; channels go to the back so that they don't override Guix' own
>> +    ;; modules.
>> +    (set! %load-path
>> +      (append environment %load-path channels))
>> +    (set! %load-compiled-path
>> +      (append environment %load-compiled-path channels))
>>
>>      (make-parameter
>> -     (append environment `((,%distro-root-directory . "gnu/packages"))))))
>> +     (append environment
>> +             %default-package-module-path
>> +             channels))))
>
> I’m not sure I understand the reason to add channels to the end of the
> search path.  Could it not be desirable in some use-cases to override
> certain Guix modules?  Should the order be made explicit in the channel
> to avoid having to accomodate “historical reasons” in the future? :)

For %load-path and %load-compiled-path, I thought it may be safer to
always keep Guix in front of the rest.  It means that channels cannot
override, say, (guix scripts package) or (guix derivations) or (gnu
packages base).  That’s mostly to be on the safe side, and because I
think this is not “the right way” to customize things; it’d be just too
brittle.

Regarding %package-module-path itself, whether channels come first or
not doesn’t actually make much of a difference at this point since
‘fold-packages’ traverses everything anyway.  Maybe in the future
‘fold-packages’ could make some distinction though.  Dunno.

>> diff --git a/guix/describe.scm b/guix/describe.scm
>> new file mode 100644
>> index 000000000..3122a762f
>> --- /dev/null
>> +++ b/guix/describe.scm
> […]
>> +(define current-profile
>> +  (mlambda ()
>> +    "Return the profile (created by 'guix pull') the calling process lives in,
>> +or #f if this is not applicable."
>> +    (match (command-line)
>> +      ((program . _)
>> +       (and (string-suffix? "/bin/guix" program)
>> +            ;; Note: We want to do _lexical dot-dot resolution_.  Using ".."
>> +            ;; for real would instead take us into the /gnu/store directory
>> +            ;; that ~/.config/guix/current/bin points to, whereas we want to
>> +            ;; obtain ~/.config/guix/current.
>> +            (let ((candidate (dirname (dirname program))))
>> +              (and (file-exists? (string-append candidate "/manifest"))
>> +                   candidate)))))))
>
> I don’t know… there’s something about this file system traversal that
> doesn’t sit right with me.  I’m not sure about (command-line) — when
> …/bin/guix is executed by a wrapper, will the wrapper be the “program”
> that we match against or the target?  (This is a concern for wrappers
> that set up site-wide default channels or a remote daemon, for example.)

The ‘guix’ command is a script starting with:

  #!/gnu/store/…-guile-2.2.4/bin/guile --no-auto-compile

‘guile’ receives the ‘guix’ file name as its argv[1].  Since the ‘guix’
file name was passed as the first argument to ‘execve’, it is
necessarily valid (either it’s relative to $PWD or, in the likely case
where ‘guix’ was searched for in $PATH, it’s an absolute file name.)

In addition (ice-9 command-line) arranges to make the first non-hyphen
argument the first element of what ‘command-line’ returns.

If you have a wrapper that execs ‘guix’ or whatever, it’ll still work.
Of course, the trick doesn’t work if you do things like:

  guile -L ~/.config/guix/current/share/guile/site/2.2

but I think that’s OK.

(I hadn’t realized the trick about execve’s first argument becoming a
valid file name in the interpreter’s argv[1].  Handy!)

Ludo’.

bug#22629: [PATCH v2 2/3] Add (guix describe) and use it to initialize '%package-search-path'.

[Ricardo Wurmus]

Ludovic Courtès <ludo@gnu.org> writes:

> Regarding %package-module-path itself, whether channels come first or
> not doesn’t actually make much of a difference at this point since
> ‘fold-packages’ traverses everything anyway.  Maybe in the future
> ‘fold-packages’ could make some distinction though.  Dunno.

Ah, I see.  Thanks for clarifying.

>> I don’t know… there’s something about this file system traversal that
>> doesn’t sit right with me.  I’m not sure about (command-line) — when
>> …/bin/guix is executed by a wrapper, will the wrapper be the “program”
>> that we match against or the target?  (This is a concern for wrappers
>> that set up site-wide default channels or a remote daemon, for example.)
>
> The ‘guix’ command is a script starting with:
>
>   #!/gnu/store/…-guile-2.2.4/bin/guile --no-auto-compile
>
> ‘guile’ receives the ‘guix’ file name as its argv[1].  Since the ‘guix’
> file name was passed as the first argument to ‘execve’, it is
> necessarily valid (either it’s relative to $PWD or, in the likely case
> where ‘guix’ was searched for in $PATH, it’s an absolute file name.)
>
> In addition (ice-9 command-line) arranges to make the first non-hyphen
> argument the first element of what ‘command-line’ returns.
>
> If you have a wrapper that execs ‘guix’ or whatever, it’ll still work.

That’s good.

> Of course, the trick doesn’t work if you do things like:
>
>   guile -L ~/.config/guix/current/share/guile/site/2.2
>
> but I think that’s OK.

Yeah, I wouldn’t worry about that.

Thanks!

-- 
Ricardo

bug#22629: Channels!

[Ludovic Courtès]

Hello,

ludo@gnu.org (Ludovic Courtès) skribis:

> ludo@gnu.org (Ludovic Courtès) skribis:
>
>> The patches that follow implement this last bit, though in a slightly
>> different way.  Users would now have the option to provide
>> ~/.config/guix/channels.scm along these lines:
>>
>>   (cons (channel
>>          (name 'guix-hpc)
>>          (url "https://gitlab.inria.fr/guix-hpc/guix-hpc.git")
>>          (branch "origin/master"))
>>         %default-channels)
>
> What follows is version 2 of the patches, which I’d like to push within
> a day or two.  The main changes are:

I fixed a typo that would lead ‘--url’ to be ignored and pushed.
Please report any problems you may have!

I’m closing this 2.5-year old issue now.  :-)

Ludo’.

bug#32022: bug#22629: “Stable” branch

[Alex Sassmannshausen]

Hi Ludo,

Ludovic Courtès writes:

> Hi Alex,
>
> (Cc’ing <https://bugs.gnu.org/32022> and <https://bugs.gnu.org/26608>,
> which are related.)
>
> Alex Sassmannshausen <alex@pompo.co> skribis:
>
>> I don't know if this is what Konrad desires, but from my perspective, a
>> desirable part of the definition of stable would be a that the build
>> farms have produced a set of binaries/substitutes for a given Guix
>> revision that is "good enough".
>
> I just had a bright idea (yes!): this can be addressed by writing
> something like this in ~/.config/guix/channels.scm:
>
>   (map latest-commit-with-substitutes-available
>        %default-channels)
>
> The hypothetical ‘latest-commit-with-substitutes-available’ would use
> (git) and (guix ci) to find the latest commit for which substitutes of
> interest are available, and would return:
>
>   (channel
>     ;; …
>     (commit "cabbag3"))   ;the ideal commit

This sounds incredibly interesting — and it is testament once again to
the power of Guix that this kind of solution could be feasible!

Thinking this through in my head somewhat, I had the following thoughts:
- This procedure is invoked client side, where the channel is defined
- That means the git searching is done client side, on every invocation
of guix (I guess this might be cacheable?)
- So the downside vis-a-vis a maintained "stable branch" would be a
price in performance as experienced by the end user
- The upside of course would be automatic curation of a stable branch
that saves a ton of volunteer effort and work

I have no idea what the performance cost would be.  I guess you would
use "guix weather" to turn the set of requested packages into a manifest
which can then be checked with it.

So the cost would be one of the following scenarios:
Option a)
- fetch set of packages in a given commit
- query guix weather for 100% substitutes
- iterate until a match
- then perform the appropriate guix pull

Option b)
- perform a guix pull to the latest commit
- query guix weather for 100% substitutes
- until success, step back one step at a time through guix pull

(because of the cost of guix pull this seems unfeasible)

Option c)
Implement some form of substitute cache set querying on build farms, as
part of guix weather, so the 100% match is done on the build farm
instead of the client.

Dunno.  There may be some things that already exist in Guix land that
I'm missing.

It's a super exciting approach for sure.

> This has to be done with great care to prevent a downgrade attack and to
> make sure the user doesn’t miss out on security updates, but maybe we
> could provide a procedure that makes reasonable choices.

Right — so at the very least it would have to prevent us going "back in
time" from the guix pull commit we are currently at.

The question of security updates is tricky at the moment already — I
would hazard a guess that many people bail out of upgrading when they
can't get substitutes for their entire profile / system right now, which
means they are not getting security upgrades for package (a) when a
substitute for (b) fails.

Thanks for your thoughts — super intriguing!

Alex

bug#22629: “Stable” branch

[Ludovic Courtès]

Hi Alex,

Alex Sassmannshausen <alex@pompo.co> skribis:

> Ludovic Courtès writes:

[...]

>> I just had a bright idea (yes!): this can be addressed by writing
>> something like this in ~/.config/guix/channels.scm:
>>
>>   (map latest-commit-with-substitutes-available
>>        %default-channels)
>>
>> The hypothetical ‘latest-commit-with-substitutes-available’ would use
>> (git) and (guix ci) to find the latest commit for which substitutes of
>> interest are available, and would return:
>>
>>   (channel
>>     ;; …
>>     (commit "cabbag3"))   ;the ideal commit
>
> This sounds incredibly interesting — and it is testament once again to
> the power of Guix that this kind of solution could be feasible!

Just to be clear: I don’t think this would be a substitute for a
“stable” branch; rather, I view as a way to have user-defined policies
such as “pull up to the latest commit for which there’s a substitute for
IceCat.”

> Thinking this through in my head somewhat, I had the following thoughts:
> - This procedure is invoked client side, where the channel is defined
> - That means the git searching is done client side, on every invocation
> of guix (I guess this might be cacheable?)

On every invocation of ‘guix pull’ only.

> I have no idea what the performance cost would be.  I guess you would
> use "guix weather" to turn the set of requested packages into a manifest
> which can then be checked with it.

As I imagine it, the cost would be a few HTTP queries to the Cuirass
API.  I should try to come up with an example to better explain what I
had in mind!

> The question of security updates is tricky at the moment already — I
> would hazard a guess that many people bail out of upgrading when they
> can't get substitutes for their entire profile / system right now, which
> means they are not getting security upgrades for package (a) when a
> substitute for (b) fails.

That’s probably true, and I agree it’s problematic.

What I typically do is “guix pull && guix package -n -u”.  Then I look
at things that would be built; if, say, LibreOffice is among them, I
wait for a little while and try again later, until I can get enough
substitutes.  That usually works okay, but it fails if it turns out that
one of the dependencies fails to build: substitutes never become
available in that case.

Ludo’.

bug#22629: “Stable” branch

[Ludovic Courtès]

[-- Attachment #1: Type: text/plain, Size: 1014 bytes --]

Alex Sassmannshausen <alex@pompo.co> skribis:

> Ludovic Courtès writes:

[...]

>> I just had a bright idea (yes!): this can be addressed by writing
>> something like this in ~/.config/guix/channels.scm:
>>
>>   (map latest-commit-with-substitutes-available
>>        %default-channels)
>>
>> The hypothetical ‘latest-commit-with-substitutes-available’ would use
>> (git) and (guix ci) to find the latest commit for which substitutes of
>> interest are available, and would return:
>>
>>   (channel
>>     ;; …
>>     (commit "cabbag3"))   ;the ideal commit

The code below is an illustration of that.  If you install it as
~/.config/guix/channels.scm, ‘guix pull’ will pull the latest commit
that was fully built on berlin.guixsd.org (see
<https://berlin.guixsd.org/jobset/guix-modular-master>), meaning that
substitutes for Guix itself should be available, unless ‘guix publish’
hasn’t “baked” them yet.

It takes two GETs and ~1s to do that here.

Ludo’.


[-- Attachment #2: the channels config --]
[-- Type: text/plain, Size: 1894 bytes --]

(use-modules (guix http-client)
             (json)
             (srfi srfi-1)
             (ice-9 match))

(define (latest-evaluations jobset)
  "Return the latest evaluations of JOBSET."
  (filter (lambda (json)
            (string=? (hash-ref json "specification") jobset))
          (json->scm
           (http-fetch
            "https://berlin.guixsd.org/api/evaluations?nr=30"))))

(define (evaluation-complete? number)
  "Return true if evaluation NUMBER completed and all its builds were
successful."
  (let ((builds (json->scm
                 (http-fetch
                  (string-append
                   "https://berlin.guixsd.org/api/latestbuilds?nr=30&evaluation="
                   (number->string number))))))
    (every (lambda (build)
             ;; Zero means build success.
             (= (hash-ref build "buildstatus") 0))
           builds)))

(define (latest-commit-successfully-built)
  "Return the latest commit for which substitutes are (potentially)
available."
  (let* ((evaluations (latest-evaluations "guix-modular-master"))
         (candidates  (filter-map (lambda (json)
                                    (match (hash-ref json "checkouts")
                                      ((checkout)
                                       (cons (hash-ref json "id")
                                             (hash-ref checkout "commit")))
                                      (_ #f)))
                                  evaluations)))
    (any (match-lambda
            ((evaluation . commit)
             (and (evaluation-complete? evaluation)
                  commit)))
          candidates)))

;; Pull the latest commit fully built on berlin.guixsd.org.
;; WARNING: This could downgrade your system!
(list (channel
       (name 'guix)
       (url "https://git.savannah.gnu.org/git/guix.git")
       (commit (pk 'commit (latest-commit-successfully-built)))))

bug#26608: bug#22629: “Stable” branch

[Alex Sassmannshausen]

Ludovic Courtès writes:

> Hi Alex,
>
> Alex Sassmannshausen <alex@pompo.co> skribis:
>
>> Ludovic Courtès writes:
>
> [...]
>
>>> I just had a bright idea (yes!): this can be addressed by writing
>>> something like this in ~/.config/guix/channels.scm:
>>>
>>>   (map latest-commit-with-substitutes-available
>>>        %default-channels)
>>>
>>> The hypothetical ‘latest-commit-with-substitutes-available’ would use
>>> (git) and (guix ci) to find the latest commit for which substitutes of
>>> interest are available, and would return:
>>>
>>>   (channel
>>>     ;; …
>>>     (commit "cabbag3"))   ;the ideal commit
>>
>> This sounds incredibly interesting — and it is testament once again to
>> the power of Guix that this kind of solution could be feasible!
>
> Just to be clear: I don’t think this would be a substitute for a
> “stable” branch; rather, I view as a way to have user-defined policies
> such as “pull up to the latest commit for which there’s a substitute for
> IceCat.”

Ah, I understand now.

So the example you provided is a user-defined policy to install the
latest version of Guix that is downloadable using substitutes (if guix
publish has published those already).

As you say, in a similar vein, the end user could for themselves define
a policy that searches for a commit containing a specific successful
build, or a set of specific successful builds.

>> Thinking this through in my head somewhat, I had the following thoughts:
>> - This procedure is invoked client side, where the channel is defined
>> - That means the git searching is done client side, on every invocation
>> of guix (I guess this might be cacheable?)
>
> On every invocation of ‘guix pull’ only.

That makes sense, and is way better than I feared :-)

>> I have no idea what the performance cost would be.  I guess you would
>> use "guix weather" to turn the set of requested packages into a manifest
>> which can then be checked with it.
>
> As I imagine it, the cost would be a few HTTP queries to the Cuirass
> API.  I should try to come up with an example to better explain what I
> had in mind!

Your example helps visualize this, thanks.

Your example depends on there being a jobset that comprises the set of
packages you are interested in testing.

I imagine it is possible to do the same for an individual package / job.

The situation would be different if the end user wanted to perform a
similar operation for an arbitrary set of packages on their end.

It would probably involve something like this (probably naive):

(define (latest-commit-successfully-built-pkg pkg)
  "Return the latest commit for the pkg for which substitutes are
(potentially) available."
  ;;                  Like your version, but magically performs query
  ;;                  for pkg, not the guix-modular-master evaluation
  (let* ((evaluations (latest-evaluations pkg))
         (candidates  (filter-map (lambda (json)
                                    (match (hash-ref json "checkouts")
                                      ((checkout)
                                       (cons (hash-ref json "id")
                                             (hash-ref checkout "commit")))
                                      (_ #f)))
                                  evaluations)))
    (map (match-lambda
            ((evaluation . commit)
             (and (evaluation-complete? evaluation)
                  commit)))
          candidates)))

(any (match-lambda
        ((evaluation . commit) commit)
     (apply lset-intersection equal?
     ;;   Like latest-commit-successfully-built, but takes an
     ;;   individual package name for which we return the
     ;;   commit
     (map latest-commit-successfully-built-pkg
          %set-of-packages))))

Obviously the larger the set, the more requests are required, and the
lower the chance of a commit being available / a downgrade occuring

>> The question of security updates is tricky at the moment already — I
>> would hazard a guess that many people bail out of upgrading when they
>> can't get substitutes for their entire profile / system right now, which
>> means they are not getting security upgrades for package (a) when a
>> substitute for (b) fails.
>
> That’s probably true, and I agree it’s problematic.
>
> What I typically do is “guix pull && guix package -n -u”.  Then I look
> at things that would be built; if, say, LibreOffice is among them, I
> wait for a little while and try again later, until I can get enough
> substitutes.  That usually works okay, but it fails if it turns out that
> one of the dependencies fails to build: substitutes never become
> available in that case.

Interesting.  Do you think this kind of thing might be useful to have in
the Guix manual? Like, in a section about a "typical" desktop end-user
might manage their system day to day?

Alex

bug#26608: bug#22629: “Stable” branch

[Ludovic Courtès]

Hi Alex,

Alex Sassmannshausen <alex@pompo.co> skribis:

> So the example you provided is a user-defined policy to install the
> latest version of Guix that is downloadable using substitutes (if guix
> publish has published those already).
>
> As you say, in a similar vein, the end user could for themselves define
> a policy that searches for a commit containing a specific successful
> build, or a set of specific successful builds.

Exactly.

>> As I imagine it, the cost would be a few HTTP queries to the Cuirass
>> API.  I should try to come up with an example to better explain what I
>> had in mind!
>
> Your example helps visualize this, thanks.
>
> Your example depends on there being a jobset that comprises the set of
> packages you are interested in testing.

Yes, and it’s hacky in that the substitute server and jobset names are
hard-coded, but you get the idea.

> I imagine it is possible to do the same for an individual package / job.

Yes.

> The situation would be different if the end user wanted to perform a
> similar operation for an arbitrary set of packages on their end.

It would be quite similar: you would query the set of builds of an
evaluation of the “guix-modular” jobset and check whether the packages
of interest were built.

>> What I typically do is “guix pull && guix package -n -u”.  Then I look
>> at things that would be built; if, say, LibreOffice is among them, I
>> wait for a little while and try again later, until I can get enough
>> substitutes.  That usually works okay, but it fails if it turns out that
>> one of the dependencies fails to build: substitutes never become
>> available in that case.
>
> Interesting.  Do you think this kind of thing might be useful to have in
> the Guix manual? Like, in a section about a "typical" desktop end-user
> might manage their system day to day?

It would make sense to have such a section I guess.  However, before
teaching users how to work around deficiencies of our infrastructure our
processes ;-), I’d like us to improve them much as possible.  I’m sure
we have room for improvement for instance in Cuirass.

Thanks,
Ludo’.

bug#26608: bug#22629: “Stable” branch

[zimoun]

Hi,

On Fri, 31 Aug 2018 at 00:02, ludo@gnu.org (Ludovic Courtès) wrote:

> (Cc’ing <https://bugs.gnu.org/32022> and <https://bugs.gnu.org/26608>,
> which are related.)
>
> Alex Sassmannshausen <alex@pompo.co> skribis:
>
>> I don't know if this is what Konrad desires, but from my perspective, a
>> desirable part of the definition of stable would be a that the build
>> farms have produced a set of binaries/substitutes for a given Guix
>> revision that is "good enough".
>
> I just had a bright idea (yes!): this can be addressed by writing
> something like this in ~/.config/guix/channels.scm:
>
>   (map latest-commit-with-substitutes-available
>        %default-channels)
>
> The hypothetical ‘latest-commit-with-substitutes-available’ would use
> (git) and (guix ci) to find the latest commit for which substitutes of
> interest are available, and would return:
>
>   (channel
>     ;; …
>     (commit "cabbag3"))   ;the ideal commit
>
> This has to be done with great care to prevent a downgrade attack and to
> make sure the user doesn’t miss out on security updates, but maybe we
> could provide a procedure that makes reasonable choices.

From my understanding, this bug can be close because:

<https://guix.gnu.org/manual/devel/en/guix.html#Channels-with-Substitutes>
<https://guix.gnu.org/manual/devel/en/guix.html#Channel-Authentication>

WDYT?

All the best,
simon