Reproducible profiles

Reproducible profiles

[David Thompson]

[-- Attachment #1: Type: text/plain, Size: 960 bytes --]

Hey folks,

Lately I've been wanting to version control the list of packages that I
install in my user profile so that I can sync it amongst many machines.
So, I took a stab at adding a new '--apply' option to 'guix package'
that reads in a package list from a Scheme file and creates a new
generation of the profile with only those packages are installed.
Here's an example configuration:

    (use-modules (gnu))
    (use-package-modules base less guile emacs admin ruby mail pumpio man)
    
    (list ruby
          coreutils
          less
          man-db
          notmuch
          guile-2.0
          emacs
          dmd
          offlineimap
          pumpa)

Below is a naive patch that does the job, but is unideal because it
doesn't do some nice things like display the diff between generations
before building.  I'm looking for some guidance to make this option mesh
better with the rest of the 'guix package' utility.  Any help is
appreciated.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-package-Add-apply-option.patch --]
[-- Type: text/x-diff, Size: 7705 bytes --]

From b5348fb46fc5b6167099ed817aad8587bfbad20a Mon Sep 17 00:00:00 2001
From: David Thompson <dthompson2@worcester.edu>
Date: Thu, 14 May 2015 21:11:57 -0400
Subject: [PATCH] package: Add --apply option.

---
 guix/scripts/package.scm | 104 +++++++++++++++++++++++++++--------------------
 1 file changed, 60 insertions(+), 44 deletions(-)

diff --git a/guix/scripts/package.scm b/guix/scripts/package.scm
index 15f3e13..bb76fc3 100644
--- a/guix/scripts/package.scm
+++ b/guix/scripts/package.scm
@@ -426,6 +426,9 @@ Install, remove, or upgrade PACKAGES in a single transaction.\n"))
   (display (_ "
   -u, --upgrade[=REGEXP] upgrade all the installed packages matching REGEXP"))
   (display (_ "
+      --apply=FILE       create a new generation with only the packages listed
+                         in FILE installed"))
+  (display (_ "
       --do-not-upgrade[=REGEXP] do not upgrade any packages matching REGEXP"))
   (display (_ "
       --roll-back        roll back to the previous generation"))
@@ -517,6 +520,10 @@ Install, remove, or upgrade PACKAGES in a single transaction.\n"))
                  (lambda (opt name arg result arg-handler)
                    (values (alist-cons 'roll-back? #t result)
                            #f)))
+         (option '("apply") #t #f
+                 (lambda (opt name arg result arg-handler)
+                   (values (alist-cons 'apply (load arg) result)
+                           arg-handler)))
          (option '(#\l "list-generations") #f #t
                  (lambda (opt name arg result arg-handler)
                    (values (cons `(query list-generations ,(or arg ""))
@@ -783,6 +790,50 @@ more information.~%"))
     (define dry-run? (assoc-ref opts 'dry-run?))
     (define profile  (assoc-ref opts 'profile))
 
+    (define (build-and-use-profile manifest)
+      (let* ((bootstrap?  (assoc-ref opts 'bootstrap?)))
+
+        (when (equal? profile %current-profile)
+          (ensure-default-profile))
+
+        (let* ((prof-drv (run-with-store (%store)
+                           (profile-derivation
+                            manifest
+                            #:hooks (if bootstrap?
+                                        '()
+                                        %default-profile-hooks))))
+               (prof     (derivation->output-path prof-drv)))
+          (show-what-to-build (%store) (list prof-drv)
+                              #:use-substitutes?
+                              (assoc-ref opts 'substitutes?)
+                              #:dry-run? dry-run?)
+
+          (cond
+           (dry-run? #t)
+           ((and (file-exists? profile)
+                 (and=> (readlink* profile) (cut string=? prof <>)))
+            (format (current-error-port) (_ "nothing to be done~%")))
+           (else
+            (let* ((number (generation-number profile))
+
+                   ;; Always use NUMBER + 1 for the new profile,
+                   ;; possibly overwriting a "previous future
+                   ;; generation".
+                   (name   (generation-file-name profile
+                                                 (+ 1 number))))
+              (and (build-derivations (%store) (list prof-drv))
+                   (let* ((entries (manifest-entries manifest))
+                          (count   (length entries)))
+                     (switch-symlinks name prof)
+                     (switch-symlinks profile name)
+                     (unless (string=? profile %current-profile)
+                       (register-gc-root (%store) name))
+                     (format #t (N_ "~a package in profile~%"
+                                    "~a packages in profile~%"
+                                    count)
+                             count)
+                     (display-search-paths entries profile)))))))))
+
     ;; First roll back if asked to.
     (cond ((and (assoc-ref opts 'roll-back?)
                 (not dry-run?))
@@ -817,60 +868,25 @@ more information.~%"))
                (alist-delete 'delete-generations opts)))
              (_ #f))
             opts))
+          ((and (assoc-ref opts 'apply)
+                (not dry-run?))
+           (let* ((packages   (assoc-ref opts 'apply))
+                  (manifest   (make-manifest
+                               (map package->manifest-entry packages))))
+             (build-and-use-profile manifest)))
           (else
            (let* ((manifest    (profile-manifest profile))
                   (install     (options->installable opts manifest))
                   (remove      (options->removable opts manifest))
-                  (bootstrap?  (assoc-ref opts 'bootstrap?))
                   (transaction (manifest-transaction (install install)
                                                      (remove remove)))
                   (new         (manifest-perform-transaction
                                 manifest transaction)))
 
-             (when (equal? profile %current-profile)
-               (ensure-default-profile))
-
              (unless (and (null? install) (null? remove))
-               (let* ((prof-drv (run-with-store (%store)
-                                  (profile-derivation
-                                   new
-                                   #:hooks (if bootstrap?
-                                               '()
-                                               %default-profile-hooks))))
-                      (prof     (derivation->output-path prof-drv)))
-                 (show-manifest-transaction (%store) manifest transaction
-                                            #:dry-run? dry-run?)
-                 (show-what-to-build (%store) (list prof-drv)
-                                     #:use-substitutes?
-                                     (assoc-ref opts 'substitutes?)
-                                     #:dry-run? dry-run?)
-
-                 (cond
-                  (dry-run? #t)
-                  ((and (file-exists? profile)
-                        (and=> (readlink* profile) (cut string=? prof <>)))
-                   (format (current-error-port) (_ "nothing to be done~%")))
-                  (else
-                   (let* ((number (generation-number profile))
-
-                          ;; Always use NUMBER + 1 for the new profile,
-                          ;; possibly overwriting a "previous future
-                          ;; generation".
-                          (name   (generation-file-name profile
-                                                        (+ 1 number))))
-                     (and (build-derivations (%store) (list prof-drv))
-                          (let* ((entries (manifest-entries new))
-                                 (count   (length entries)))
-                            (switch-symlinks name prof)
-                            (switch-symlinks profile name)
-                            (unless (string=? profile %current-profile)
-                              (register-gc-root (%store) name))
-                            (format #t (N_ "~a package in profile~%"
-                                           "~a packages in profile~%"
-                                           count)
-                                    count)
-                            (display-search-paths entries
-                                                  profile))))))))))))
+               (show-manifest-transaction (%store) manifest transaction
+                                          #:dry-run? dry-run?)
+               (build-and-use-profile new))))))
 
   (define (process-query opts)
     ;; Process any query specified by OPTS.  Return #t when a query was
-- 
2.1.4


[-- Attachment #3: Type: text/plain, Size: 145 bytes --]


Thanks!

-- 
David Thompson
Web Developer - Free Software Foundation - http://fsf.org
GPG Key: 0FF1D807
Support the FSF: https://fsf.org/donate

Re: Reproducible profiles

[Ludovic Courtès]

David Thompson <dthompson2@worcester.edu> skribis:

> Lately I've been wanting to version control the list of packages that I
> install in my user profile so that I can sync it amongst many machines.
> So, I took a stab at adding a new '--apply' option to 'guix package'
> that reads in a package list from a Scheme file and creates a new
> generation of the profile with only those packages are installed.
> Here's an example configuration:
>
>     (use-modules (gnu))
>     (use-package-modules base less guile emacs admin ruby mail pumpio man)
>     
>     (list ruby
>           coreutils
>           less
>           man-db
>           notmuch
>           guile-2.0
>           emacs
>           dmd
>           offlineimap
>           pumpa)

Yes, that sounds very useful.

As usual though, there’s the issue of multiple-output packages.  The
above snippet is nice, but doesn’t allow you to specify a particular
output of a package.

What about instead requiring people to return a manifest:

  (use-modules (guix profiles))
  (use-package-modules base emacs guile)

  (manifest (cons (package->manifest-entry gcc-toolchain "debug")
                  (map package->entry
                       (list gcc-toolchain emacs guile-2.0))))

That means we’ll have to document (guix profiles).

It’s more verbose than what you suggest, though.  If you insist ;-), we
could allow a list of packages instead of a manifest, though I’d prefer
not to do that.

WDYT?

> Below is a naive patch that does the job, but is unideal because it
> doesn't do some nice things like display the diff between generations
> before building.

For that you would need a procedure to infer the manifest transaction:

  (manifests->transaction m1 m2)
  ;; returns a <manifest-transaction>

and then that could be passed to ‘show-manifest-transaction’.

However, I’m not sure it’s very useful.  Perhaps it would be enough to
write “installing new manifest from foo.scm with 42 entries.”
WDYT?

> I'm looking for some guidance to make this option mesh better with the
> rest of the 'guix package' utility.  Any help is appreciated.

Overall it looks OK to me!

[...]

> +         (option '("apply") #t #f
> +                 (lambda (opt name arg result arg-handler)
> +                   (values (alist-cons 'apply (load arg) result)
> +                           arg-handler)))

It would be better to delay loading until after arguments have been
parsed, as in ‘guix system’.  The procedure to load the file should be
similar to ‘read-operating-system’.

We’ll need documentation and tests, too.  :-)

Thank you!

Ludo’.

Re: Reproducible profiles

[宋文武]

Ludovic Courtès <ludo@gnu.org> writes:

> David Thompson <dthompson2@worcester.edu> skribis:
>
>> Lately I've been wanting to version control the list of packages that I
>> install in my user profile so that I can sync it amongst many machines.
>> So, I took a stab at adding a new '--apply' option to 'guix package'
>> that reads in a package list from a Scheme file and creates a new
>> generation of the profile with only those packages are installed.
>> Here's an example configuration:
>>
>>     (use-modules (gnu))
>>     (use-package-modules base less guile emacs admin ruby mail pumpio man)
>>     
>>     (list ruby
>>           coreutils
>>           less
>>           man-db
>>           notmuch
>>           guile-2.0
>>           emacs
>>           dmd
>>           offlineimap
>>           pumpa)
>
> Yes, that sounds very useful.
>
> As usual though, there’s the issue of multiple-output packages.  The
> above snippet is nice, but doesn’t allow you to specify a particular
> output of a package.
>
> What about instead requiring people to return a manifest:
>
>   (use-modules (guix profiles))
>   (use-package-modules base emacs guile)
>
>   (manifest (cons (package->manifest-entry gcc-toolchain "debug")
>                   (map package->entry
>                        (list gcc-toolchain emacs guile-2.0))))
>
> That means we’ll have to document (guix profiles).
>
> It’s more verbose than what you suggest, though.  If you insist ;-), we
> could allow a list of packages instead of a manifest, though I’d prefer
> not to do that.
>
> WDYT?
>
+1 for return 'manifest', in a same way as 'operating-system'.
And how about use specification instead of package, so I can:

  (manifest
    (map package-specification->manifest-entry
         '("emacs"
           "font-adobe-source-han-sans:cn")))

Re: Reproducible profiles

[Ludovic Courtès]

宋文武 <iyzsong@gmail.com> skribis:

> And how about use specification instead of package, so I can:
>
>   (manifest
>     (map package-specification->manifest-entry
>          '("emacs"
>            "font-adobe-source-han-sans:cn")))

Sure, with:

  (use-modules (gnu) (guix profiles))

  (define package-specification->manifest-entry
    (compose package->manifest-entry
             specification->package))

The advantage of using package names like this is that it’s less
sensitive to module changes; the disadvantage is that it is not as
precise, particularly if there are several packages matching a given
name.

Ludo’.

Re: Reproducible profiles

[David Thompson]

Ludovic Courtès <ludo@gnu.org> writes:

> David Thompson <dthompson2@worcester.edu> skribis:
>
>> Lately I've been wanting to version control the list of packages that I
>> install in my user profile so that I can sync it amongst many machines.
>> So, I took a stab at adding a new '--apply' option to 'guix package'
>> that reads in a package list from a Scheme file and creates a new
>> generation of the profile with only those packages are installed.
>> Here's an example configuration:
>>
>>     (use-modules (gnu))
>>     (use-package-modules base less guile emacs admin ruby mail pumpio man)
>>     
>>     (list ruby
>>           coreutils
>>           less
>>           man-db
>>           notmuch
>>           guile-2.0
>>           emacs
>>           dmd
>>           offlineimap
>>           pumpa)
>
> Yes, that sounds very useful.
>
> As usual though, there’s the issue of multiple-output packages.  The
> above snippet is nice, but doesn’t allow you to specify a particular
> output of a package.

I left it out of my example, but I figured one could specify the output
like so:

    (list ruby coreutils `(,glib "bin"))

> What about instead requiring people to return a manifest:
>
>   (use-modules (guix profiles))
>   (use-package-modules base emacs guile)
>
>   (manifest (cons (package->manifest-entry gcc-toolchain "debug")
>                   (map package->entry
>                        (list gcc-toolchain emacs guile-2.0))))
>
> That means we’ll have to document (guix profiles).
>
> It’s more verbose than what you suggest, though.  If you insist ;-), we
> could allow a list of packages instead of a manifest, though I’d prefer
> not to do that.

Expecting a manifest always sounds good.  How about adding a convenience
procedure for the (map package->entry ...) pattern since I think it will
be the most common thing users will want to do?

    (packages->manifest (list guile-2.0 guile-opengl guile-sdl))

It could even support using other outputs like in that other example I
gave:

    (packages->manifest
      (list guile-2.0
            guile-opengl
            guile-sdl
            `(,gcc-toolchain "debug"))

>> Below is a naive patch that does the job, but is unideal because it
>> doesn't do some nice things like display the diff between generations
>> before building.
>
> For that you would need a procedure to infer the manifest transaction:
>
>   (manifests->transaction m1 m2)
>   ;; returns a <manifest-transaction>
>
> and then that could be passed to ‘show-manifest-transaction’.
>
> However, I’m not sure it’s very useful.  Perhaps it would be enough to
> write “installing new manifest from foo.scm with 42 entries.”
> WDYT?

Okay, I'll do that instead.  I would be interested in seeing what has
changed when I apply a new manifest, but it's probably not worth the
effort right now.

>> +         (option '("apply") #t #f
>> +                 (lambda (opt name arg result arg-handler)
>> +                   (values (alist-cons 'apply (load arg) result)
>> +                           arg-handler)))
>
> It would be better to delay loading until after arguments have been
> parsed, as in ‘guix system’.  The procedure to load the file should be
> similar to ‘read-operating-system’.

Sure.  The use of 'load' was a quick hack for this prototype. :)

> We’ll need documentation and tests, too.  :-)

Naturally.

Thanks, now I have enough direction to do another iteration. :)

-- 
David Thompson
GPG Key: 0FF1D807

Re: Reproducible profiles

[David Thompson]

宋文武 <iyzsong@gmail.com> writes:

> +1 for return 'manifest', in a same way as 'operating-system'.
> And how about use specification instead of package, so I can:
>
>   (manifest
>     (map package-specification->manifest-entry
>          '("emacs"
>            "font-adobe-source-han-sans:cn")))

I don't like that idea because the text specification is just a syntax
for shell users.  In Scheme, we can use the package objects directly
rather than encode them as strings.

-- 
David Thompson
GPG Key: 0FF1D807

Re: Reproducible profiles

[Ludovic Courtès]

David Thompson <dthompson2@worcester.edu> skribis:

> Ludovic Courtès <ludo@gnu.org> writes:

[...]

>> What about instead requiring people to return a manifest:
>>
>>   (use-modules (guix profiles))
>>   (use-package-modules base emacs guile)
>>
>>   (manifest (cons (package->manifest-entry gcc-toolchain "debug")
>>                   (map package->entry
>>                        (list gcc-toolchain emacs guile-2.0))))
>>
>> That means we’ll have to document (guix profiles).
>>
>> It’s more verbose than what you suggest, though.  If you insist ;-), we
>> could allow a list of packages instead of a manifest, though I’d prefer
>> not to do that.
>
> Expecting a manifest always sounds good.  How about adding a convenience
> procedure for the (map package->entry ...) pattern since I think it will
> be the most common thing users will want to do?
>
>     (packages->manifest (list guile-2.0 guile-opengl guile-sdl))
>
> It could even support using other outputs like in that other example I
> gave:
>
>     (packages->manifest
>       (list guile-2.0
>             guile-opengl
>             guile-sdl
>             `(,gcc-toolchain "debug"))

Sounds good to me.

(FWIW I’m not fond of the `(,gcc-toolchain "debug") notation that we
also use in packages, but it has the advantage of being concise.)

>>> Below is a naive patch that does the job, but is unideal because it
>>> doesn't do some nice things like display the diff between generations
>>> before building.
>>
>> For that you would need a procedure to infer the manifest transaction:
>>
>>   (manifests->transaction m1 m2)
>>   ;; returns a <manifest-transaction>
>>
>> and then that could be passed to ‘show-manifest-transaction’.
>>
>> However, I’m not sure it’s very useful.  Perhaps it would be enough to
>> write “installing new manifest from foo.scm with 42 entries.”
>> WDYT?
>
> Okay, I'll do that instead.  I would be interested in seeing what has
> changed when I apply a new manifest, but it's probably not worth the
> effort right now.

Yeah.  And note that this can always be done from the wonderful guix.el
(info "(guix) Emacs List Buffer").

Thanks,
Ludo’.

Re: Reproducible profiles

[David Thompson]

Ludovic Courtès <ludo@gnu.org> writes:

> (FWIW I’m not fond of the `(,gcc-toolchain "debug") notation that we
> also use in packages, but it has the advantage of being concise.)

Do you have plans to introduce a better notation?  G-exps?

-- 
David Thompson
GPG Key: 0FF1D807

Re: Reproducible profiles

[David Thompson]

Ludovic Courtès <ludo@gnu.org> writes:

> We’ll need documentation and tests, too.  :-)

Regarding tests, where should I add them?  I can see that
tests/guix-package.sh uses the '-n' flag to avoid building things.
Simply running 'guix package -n --apply=test-manifest.scm' and getting a
0 exit value would suffice as a basic integration test, I think.
Anything else that you would recommend?

-- 
David Thompson
GPG Key: 0FF1D807

Syntax for package inputs

[Ludovic Courtès]

David Thompson <dthompson2@worcester.edu> skribis:

> Ludovic Courtès <ludo@gnu.org> writes:
>
>> (FWIW I’m not fond of the `(,gcc-toolchain "debug") notation that we
>> also use in packages, but it has the advantage of being concise.)
>
> Do you have plans to introduce a better notation?  G-exps?

No concrete plan yet.  A remote possibility might be to indeed rely more
on gexps and maybe get rid of input labels, like:

  (define foo
    (package
      ;; ...
      (inputs (list guile-2.0 gtk+ (gexp-input glib "bin")))))

but this introduces other challenges in particular wrt. package
customizations.  So, we’ll see.

However, for new APIs, I find it usually better to use the above gexp
style rather than the current package-input style.

Ludo’.

Re: Reproducible profiles

[Ludovic Courtès]

David Thompson <dthompson2@worcester.edu> skribis:

> Ludovic Courtès <ludo@gnu.org> writes:
>
>> We’ll need documentation and tests, too.  :-)
>
> Regarding tests, where should I add them?  I can see that
> tests/guix-package.sh uses the '-n' flag to avoid building things.
> Simply running 'guix package -n --apply=test-manifest.scm' and getting a
> 0 exit value would suffice as a basic integration test, I think.
> Anything else that you would recommend?

As discussed on IRC, this sounds good to me.  Probably in
guix-package.sh, which is for tests that do not require network access,
or otherwise in guix-package-net.sh.

Ludo’.

Re: Reproducible profiles

[David Thompson]

[-- Attachment #1: Type: text/plain, Size: 274 bytes --]

Below is a new patch set taking into account the feedback received thus
far.  The (guix profiles) module still needs to be documented in the
manual, but there's quite a lot of procedures and variables to account
for.  Would anyone be intertested in helping with this part?


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-ui-Factorize-user-provided-Scheme-file-loading.patch --]
[-- Type: text/x-diff, Size: 3064 bytes --]

From d506ad1d8824cc694364be502acddb25b76d0020 Mon Sep 17 00:00:00 2001
From: David Thompson <dthompson2@worcester.edu>
Date: Mon, 18 May 2015 07:49:44 -0400
Subject: [PATCH 1/3] ui: Factorize user-provided Scheme file loading.

* guix/ui.scm (make-user-module, read-scheme-file): New procedures.
* guix/scripts/system.scm (%user-module): Define in terms of
  'make-user-module'.
  (read-operating-system): Define in terms of 'read-scheme-file'.
---
 guix/scripts/system.scm | 22 ++++------------------
 guix/ui.scm             | 24 ++++++++++++++++++++++++
 2 files changed, 28 insertions(+), 18 deletions(-)

diff --git a/guix/scripts/system.scm b/guix/scripts/system.scm
index 1838e89..2d7c5d1 100644
--- a/guix/scripts/system.scm
+++ b/guix/scripts/system.scm
@@ -48,28 +48,14 @@
 
 (define %user-module
   ;; Module in which the machine description file is loaded.
-  (let ((module (make-fresh-user-module)))
-    (for-each (lambda (iface)
-                (module-use! module (resolve-interface iface)))
-              '((gnu system)
-                (gnu services)
-                (gnu system shadow)))
-    module))
+  (make-user-module '((gnu system)
+                      (gnu services)
+                      (gnu system shadow))))
 
 (define (read-operating-system file)
   "Read the operating-system declaration from FILE and return it."
-  ;; TODO: Factorize.
-  (catch #t
-    (lambda ()
-      ;; Avoid ABI incompatibility with the <operating-system> record.
-      (set! %fresh-auto-compile #t)
+  (read-scheme-file file %user-module))
 
-      (save-module-excursion
-       (lambda ()
-         (set-current-module %user-module)
-         (primitive-load file))))
-    (lambda args
-      (report-load-error file args))))
 
 \f
 ;;;
diff --git a/guix/ui.scm b/guix/ui.scm
index 911e5ee..5a76cf4 100644
--- a/guix/ui.scm
+++ b/guix/ui.scm
@@ -48,6 +48,8 @@
             P_
             report-error
             leave
+            make-user-module
+            read-scheme-file
             report-load-error
             warn-about-load-error
             show-version-and-exit
@@ -133,6 +135,28 @@ messages."
     (report-error args ...)
     (exit 1)))
 
+(define (make-user-module modules)
+  "Return a new user module with the additional MODULES loaded."
+  ;; Module in which the machine description file is loaded.
+  (let ((module (make-fresh-user-module)))
+    (for-each (lambda (iface)
+                (module-use! module (resolve-interface iface)))
+              modules)
+    module))
+
+(define (read-scheme-file file user-module)
+  "Read the user provided Scheme source code FILE."
+  (catch #t
+    (lambda ()
+      (set! %fresh-auto-compile #t)
+
+      (save-module-excursion
+       (lambda ()
+         (set-current-module user-module)
+         (primitive-load file))))
+    (lambda args
+      (report-load-error file args))))
+
 (define (report-load-error file args)
   "Report the failure to load FILE, a user-provided Scheme file, and exit.
 ARGS is the list of arguments received by the 'throw' handler."
-- 
2.1.4


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #3: 0002-profiles-Add-packages-manifest-procedure.patch --]
[-- Type: text/x-diff, Size: 1333 bytes --]

From 5665da9934726ce0a8c4ed358b7f606d917c300a Mon Sep 17 00:00:00 2001
From: David Thompson <dthompson2@worcester.edu>
Date: Mon, 18 May 2015 07:51:56 -0400
Subject: [PATCH 2/3] profiles: Add 'packages->manifest' procedure.

* guix/profiles.scm (packages->manifest): New procedure.
---
 guix/profiles.scm | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/guix/profiles.scm b/guix/profiles.scm
index 11d9bf0..cbc8a9a 100644
--- a/guix/profiles.scm
+++ b/guix/profiles.scm
@@ -80,6 +80,7 @@
 
             profile-manifest
             package->manifest-entry
+            packages->manifest
             %default-profile-hooks
             profile-derivation
             generation-number
@@ -172,6 +173,16 @@ omitted or #f, use the first output of PACKAGE."
      (dependencies (delete-duplicates deps))
      (search-paths (package-native-search-paths package)))))
 
+(define (packages->manifest packages)
+  "Convert PACKAGES into a manifest containing entries for all of them."
+  (manifest
+   (map (match-lambda
+         ((package output)
+          (package->manifest-entry package output))
+         (package
+           (package->manifest-entry package)))
+        packages)))
+
 (define (manifest->gexp manifest)
   "Return a representation of MANIFEST as a gexp."
   (define (entry->gexp entry)
-- 
2.1.4


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #4: 0003-package-Add-manifest-option.patch --]
[-- Type: text/x-diff, Size: 9529 bytes --]

From 3be657353bfebc33dc9733b820165699ac07b43d Mon Sep 17 00:00:00 2001
From: David Thompson <dthompson2@worcester.edu>
Date: Thu, 14 May 2015 21:11:57 -0400
Subject: [PATCH 3/3] package: Add --manifest option.

* guix/scripts/package.scm (show-help): Add help text.
  (%options): Add manifest option.
  (guix-package): Add manifest option handler.
* doc/guix.texi ("Invoking guix package"): Document it.
* tests/guix-package.sh: Add test.
---
 doc/guix.texi            |  17 ++++++++
 guix/scripts/package.scm | 107 ++++++++++++++++++++++++++++-------------------
 tests/guix-package.sh    |  10 +++++
 3 files changed, 90 insertions(+), 44 deletions(-)

diff --git a/doc/guix.texi b/doc/guix.texi
index 049292d..ca5f82d 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -1057,6 +1057,23 @@ substring ``emacs'':
 $ guix package --upgrade . --do-not-upgrade emacs
 @end example
 
+@item --manifest=@var{file}
+@itemx -m @var{file}
+Create a new @dfn{generation} of the profile from the manifest object
+contained in @var{file}, a Scheme source code file.
+
+A manifest file may look like this:
+
+@example
+(use-package-modules guile emacs gcc)
+
+(packages->manifest
+ (list guile-2.0
+       emacs
+       ;; Use a specific package output.
+       (list gcc "debug")))
+@end example
+
 @item --roll-back
 Roll back to the previous @dfn{generation} of the profile---i.e., undo
 the last transaction.
diff --git a/guix/scripts/package.scm b/guix/scripts/package.scm
index 15f3e13..f2ca663 100644
--- a/guix/scripts/package.scm
+++ b/guix/scripts/package.scm
@@ -426,6 +426,9 @@ Install, remove, or upgrade PACKAGES in a single transaction.\n"))
   (display (_ "
   -u, --upgrade[=REGEXP] upgrade all the installed packages matching REGEXP"))
   (display (_ "
+  -m, --manifest=FILE    create a new profile generation with the manifest
+                         contained within FILE."))
+  (display (_ "
       --do-not-upgrade[=REGEXP] do not upgrade any packages matching REGEXP"))
   (display (_ "
       --roll-back        roll back to the previous generation"))
@@ -517,6 +520,10 @@ Install, remove, or upgrade PACKAGES in a single transaction.\n"))
                  (lambda (opt name arg result arg-handler)
                    (values (alist-cons 'roll-back? #t result)
                            #f)))
+         (option '(#\m "manifest") #t #f
+                 (lambda (opt name arg result arg-handler)
+                   (values (alist-cons 'manifest arg result)
+                           arg-handler)))
          (option '(#\l "list-generations") #f #t
                  (lambda (opt name arg result arg-handler)
                    (values (cons `(query list-generations ,(or arg ""))
@@ -783,6 +790,50 @@ more information.~%"))
     (define dry-run? (assoc-ref opts 'dry-run?))
     (define profile  (assoc-ref opts 'profile))
 
+    (define (build-and-use-profile manifest)
+      (let* ((bootstrap?  (assoc-ref opts 'bootstrap?)))
+
+        (when (equal? profile %current-profile)
+          (ensure-default-profile))
+
+        (let* ((prof-drv (run-with-store (%store)
+                           (profile-derivation
+                            manifest
+                            #:hooks (if bootstrap?
+                                        '()
+                                        %default-profile-hooks))))
+               (prof     (derivation->output-path prof-drv)))
+          (show-what-to-build (%store) (list prof-drv)
+                              #:use-substitutes?
+                              (assoc-ref opts 'substitutes?)
+                              #:dry-run? dry-run?)
+
+          (cond
+           (dry-run? #t)
+           ((and (file-exists? profile)
+                 (and=> (readlink* profile) (cut string=? prof <>)))
+            (format (current-error-port) (_ "nothing to be done~%")))
+           (else
+            (let* ((number (generation-number profile))
+
+                   ;; Always use NUMBER + 1 for the new profile,
+                   ;; possibly overwriting a "previous future
+                   ;; generation".
+                   (name   (generation-file-name profile
+                                                 (+ 1 number))))
+              (and (build-derivations (%store) (list prof-drv))
+                   (let* ((entries (manifest-entries manifest))
+                          (count   (length entries)))
+                     (switch-symlinks name prof)
+                     (switch-symlinks profile name)
+                     (unless (string=? profile %current-profile)
+                       (register-gc-root (%store) name))
+                     (format #t (N_ "~a package in profile~%"
+                                    "~a packages in profile~%"
+                                    count)
+                             count)
+                     (display-search-paths entries profile)))))))))
+
     ;; First roll back if asked to.
     (cond ((and (assoc-ref opts 'roll-back?)
                 (not dry-run?))
@@ -817,60 +868,28 @@ more information.~%"))
                (alist-delete 'delete-generations opts)))
              (_ #f))
             opts))
+          ((and (assoc-ref opts 'manifest)
+                (not dry-run?))
+           (let* ((file-name (assoc-ref opts 'manifest))
+                  (user-module (make-user-module '((guix profiles)
+                                                   (gnu))))
+                  (manifest (read-scheme-file file-name user-module)))
+             (format #t (_ "installing new manifest from ~a with ~d entries.~%")
+                     file-name (length (manifest-entries manifest)))
+             (build-and-use-profile manifest)))
           (else
            (let* ((manifest    (profile-manifest profile))
                   (install     (options->installable opts manifest))
                   (remove      (options->removable opts manifest))
-                  (bootstrap?  (assoc-ref opts 'bootstrap?))
                   (transaction (manifest-transaction (install install)
                                                      (remove remove)))
                   (new         (manifest-perform-transaction
                                 manifest transaction)))
 
-             (when (equal? profile %current-profile)
-               (ensure-default-profile))
-
              (unless (and (null? install) (null? remove))
-               (let* ((prof-drv (run-with-store (%store)
-                                  (profile-derivation
-                                   new
-                                   #:hooks (if bootstrap?
-                                               '()
-                                               %default-profile-hooks))))
-                      (prof     (derivation->output-path prof-drv)))
-                 (show-manifest-transaction (%store) manifest transaction
-                                            #:dry-run? dry-run?)
-                 (show-what-to-build (%store) (list prof-drv)
-                                     #:use-substitutes?
-                                     (assoc-ref opts 'substitutes?)
-                                     #:dry-run? dry-run?)
-
-                 (cond
-                  (dry-run? #t)
-                  ((and (file-exists? profile)
-                        (and=> (readlink* profile) (cut string=? prof <>)))
-                   (format (current-error-port) (_ "nothing to be done~%")))
-                  (else
-                   (let* ((number (generation-number profile))
-
-                          ;; Always use NUMBER + 1 for the new profile,
-                          ;; possibly overwriting a "previous future
-                          ;; generation".
-                          (name   (generation-file-name profile
-                                                        (+ 1 number))))
-                     (and (build-derivations (%store) (list prof-drv))
-                          (let* ((entries (manifest-entries new))
-                                 (count   (length entries)))
-                            (switch-symlinks name prof)
-                            (switch-symlinks profile name)
-                            (unless (string=? profile %current-profile)
-                              (register-gc-root (%store) name))
-                            (format #t (N_ "~a package in profile~%"
-                                           "~a packages in profile~%"
-                                           count)
-                                    count)
-                            (display-search-paths entries
-                                                  profile))))))))))))
+               (show-manifest-transaction (%store) manifest transaction
+                                          #:dry-run? dry-run?)
+               (build-and-use-profile new))))))
 
   (define (process-query opts)
     ;; Process any query specified by OPTS.  Return #t when a query was
diff --git a/tests/guix-package.sh b/tests/guix-package.sh
index a732110..4591333 100644
--- a/tests/guix-package.sh
+++ b/tests/guix-package.sh
@@ -237,3 +237,13 @@ export GUIX_BUILD_OPTIONS
 available2="`guix package -A | sort`"
 test "$available2" = "$available"
 guix package -I
+
+unset GUIX_BUILD_OPTIONS
+
+# Applying a manifest file
+cat > "$module_dir/manifest.scm"<<EOF
+(use-package-modules bootstrap)
+
+(packages->manifest (list %bootstrap-guile))
+EOF
+guix package --bootstrap -m "$module_dir/manifest.scm"
-- 
2.1.4


[-- Attachment #5: Type: text/plain, Size: 38 bytes --]


-- 
David Thompson
GPG Key: 0FF1D807

Re: Reproducible profiles

[Ludovic Courtès]

David Thompson <dthompson2@worcester.edu> skribis:

> Below is a new patch set taking into account the feedback received thus
> far.  The (guix profiles) module still needs to be documented in the
> manual, but there's quite a lot of procedures and variables to account
> for.  Would anyone be intertested in helping with this part?

I can help with that, but it’s even better if someone else does it.  :-)

> From d506ad1d8824cc694364be502acddb25b76d0020 Mon Sep 17 00:00:00 2001
> From: David Thompson <dthompson2@worcester.edu>
> Date: Mon, 18 May 2015 07:49:44 -0400
> Subject: [PATCH 1/3] ui: Factorize user-provided Scheme file loading.
>
> * guix/ui.scm (make-user-module, read-scheme-file): New procedures.
> * guix/scripts/system.scm (%user-module): Define in terms of
>   'make-user-module'.
>   (read-operating-system): Define in terms of 'read-scheme-file'.

[...]

> +(define (read-scheme-file file user-module)
> +  "Read the user provided Scheme source code FILE."

What about calling it ‘load*’ and s/Read/Load/ in the docstring?
It’s really a variant of ‘load’ rather than a variant of ‘read’.

If that’s fine with you, OK to push with this change.

> From 5665da9934726ce0a8c4ed358b7f606d917c300a Mon Sep 17 00:00:00 2001
> From: David Thompson <dthompson2@worcester.edu>
> Date: Mon, 18 May 2015 07:51:56 -0400
> Subject: [PATCH 2/3] profiles: Add 'packages->manifest' procedure.
>
> * guix/profiles.scm (packages->manifest): New procedure.

[...]

> +(define (packages->manifest packages)
> +  "Convert PACKAGES into a manifest containing entries for all of them."

What about something like:

  Return a list of manifest entries, one for each item listed in
  PACKAGES.  Elements of PACKAGES can be either package objects or
  package/string tuples denoting a specific output of a package.

OK to push with something along these lines.

> From 3be657353bfebc33dc9733b820165699ac07b43d Mon Sep 17 00:00:00 2001
> From: David Thompson <dthompson2@worcester.edu>
> Date: Thu, 14 May 2015 21:11:57 -0400
> Subject: [PATCH 3/3] package: Add --manifest option.
>
> * guix/scripts/package.scm (show-help): Add help text.
>   (%options): Add manifest option.
>   (guix-package): Add manifest option handler.
> * doc/guix.texi ("Invoking guix package"): Document it.
> * tests/guix-package.sh: Add test.


[...]

> +@item --manifest=@var{file}
> +@itemx -m @var{file}
> +Create a new @dfn{generation} of the profile from the manifest object
> +contained in @var{file}, a Scheme source code file.

s/contained in.*/returned by the Scheme code in @var{file}./

> +A manifest file may look like this:
> +
> +@example
> +(use-package-modules guile emacs gcc)
> +
> +(packages->manifest
> + (list guile-2.0
> +       emacs
> +       ;; Use a specific package output.
> +       (list gcc "debug")))
> +@end example

Maybe s/gcc/guile-2.0/, which might better illustrate the use case.

> +  -m, --manifest=FILE    create a new profile generation with the manifest
> +                         contained within FILE."))

s/contained within/from/

Also no period at the end.

> +# Applying a manifest file
> +cat > "$module_dir/manifest.scm"<<EOF
> +(use-package-modules bootstrap)
> +
> +(packages->manifest (list %bootstrap-guile))
> +EOF
> +guix package --bootstrap -m "$module_dir/manifest.scm"

Maybe just add something like:

  guix package -I | grep guile
  test `guix package -I | wc -l` -eq 1

OK with these changes!  I think it’s going to be nice to be able to use
this declarative approach for user profiles, have the file under VC, etc.

Thanks!

Ludo’.

Re: Reproducible profiles

[David Thompson]

Ludovic Courtès <ludo@gnu.org> writes:

> David Thompson <dthompson2@worcester.edu> skribis:
>
>> Below is a new patch set taking into account the feedback received thus
>> far.  The (guix profiles) module still needs to be documented in the
>> manual, but there's quite a lot of procedures and variables to account
>> for.  Would anyone be intertested in helping with this part?
>
> I can help with that, but it’s even better if someone else does it.
> :-)

Okay. :)

>> From d506ad1d8824cc694364be502acddb25b76d0020 Mon Sep 17 00:00:00 2001
>> From: David Thompson <dthompson2@worcester.edu>
>> Date: Mon, 18 May 2015 07:49:44 -0400
>> Subject: [PATCH 1/3] ui: Factorize user-provided Scheme file loading.
>>
>> * guix/ui.scm (make-user-module, read-scheme-file): New procedures.
>> * guix/scripts/system.scm (%user-module): Define in terms of
>>   'make-user-module'.
>>   (read-operating-system): Define in terms of 'read-scheme-file'.
>
> [...]
>
>> +(define (read-scheme-file file user-module)
>> +  "Read the user provided Scheme source code FILE."
>
> What about calling it ‘load*’ and s/Read/Load/ in the docstring?
> It’s really a variant of ‘load’ rather than a variant of ‘read’.
>
> If that’s fine with you, OK to push with this change.

Done.

>> From 5665da9934726ce0a8c4ed358b7f606d917c300a Mon Sep 17 00:00:00 2001
>> From: David Thompson <dthompson2@worcester.edu>
>> Date: Mon, 18 May 2015 07:51:56 -0400
>> Subject: [PATCH 2/3] profiles: Add 'packages->manifest' procedure.
>>
>> * guix/profiles.scm (packages->manifest): New procedure.
>
> [...]
>
>> +(define (packages->manifest packages)
>> +  "Convert PACKAGES into a manifest containing entries for all of them."
>
> What about something like:
>
>   Return a list of manifest entries, one for each item listed in
>   PACKAGES.  Elements of PACKAGES can be either package objects or
>   package/string tuples denoting a specific output of a package.
>
> OK to push with something along these lines.

Done.

>> From 3be657353bfebc33dc9733b820165699ac07b43d Mon Sep 17 00:00:00 2001
>> From: David Thompson <dthompson2@worcester.edu>
>> Date: Thu, 14 May 2015 21:11:57 -0400
>> Subject: [PATCH 3/3] package: Add --manifest option.
>>
>> * guix/scripts/package.scm (show-help): Add help text.
>>   (%options): Add manifest option.
>>   (guix-package): Add manifest option handler.
>> * doc/guix.texi ("Invoking guix package"): Document it.
>> * tests/guix-package.sh: Add test.
>
>
> [...]
>
>> +@item --manifest=@var{file}
>> +@itemx -m @var{file}
>> +Create a new @dfn{generation} of the profile from the manifest object
>> +contained in @var{file}, a Scheme source code file.
>
> s/contained in.*/returned by the Scheme code in @var{file}./

Done.

>> +A manifest file may look like this:
>> +
>> +@example
>> +(use-package-modules guile emacs gcc)
>> +
>> +(packages->manifest
>> + (list guile-2.0
>> +       emacs
>> +       ;; Use a specific package output.
>> +       (list gcc "debug")))
>> +@end example
>
> Maybe s/gcc/guile-2.0/, which might better illustrate the use case.
>
>> +  -m, --manifest=FILE    create a new profile generation with the manifest
>> +                         contained within FILE."))
>
> s/contained within/from/
>
> Also no period at the end.

Done.

>> +# Applying a manifest file
>> +cat > "$module_dir/manifest.scm"<<EOF
>> +(use-package-modules bootstrap)
>> +
>> +(packages->manifest (list %bootstrap-guile))
>> +EOF
>> +guix package --bootstrap -m "$module_dir/manifest.scm"
>
> Maybe just add something like:
>
>   guix package -I | grep guile
>   test `guix package -I | wc -l` -eq 1

Done.

> OK with these changes!  I think it’s going to be nice to be able to
> use this declarative approach for user profiles, have the file under
> VC, etc.

Me too!  Pushed all 3 patches!

-- 
David Thompson
GPG Key: 0FF1D807

Re: Reproducible profiles

[宋文武]

David Thompson <dthompson2@worcester.edu> writes:

> 宋文武 <iyzsong@gmail.com> writes:
>
>> +1 for return 'manifest', in a same way as 'operating-system'.
>> And how about use specification instead of package, so I can:
>>
>>   (manifest
>>     (map package-specification->manifest-entry
>>          '("emacs"
>>            "font-adobe-source-han-sans:cn")))
>
> I don't like that idea because the text specification is just a syntax
> for shell users.  In Scheme, we can use the package objects directly
> rather than encode them as strings.
Well, by this way, I don't need to figure out which modules packages
belong to.

It works great, thanks!