[bug#38301] [PATCH 0/4] Allow for multiple '--manifest' options

[bug#38301] [PATCH 0/4] Allow for multiple '--manifest' options

[Ludovic Courtès]

Hello Guix!

As discussed on guix-devel, this patch series ensures that one can
pass multiple ‘--manifest’ options to ‘guix package’, ‘guix pack’,
and ‘guix environment’ (which already supported it actually!).

Feedback welcome!

Ludo’.

Ludovic Courtès (4):
  environment: Document that '--manifest' can be repeated.
  profiles: Add 'concatenate-manifests'.
  package: Allow multiple '--manifest' options.
  pack: Allow multiple '--manifest' options.

 doc/guix.texi             |  9 ++++---
 guix/profiles.scm         |  5 ++++
 guix/scripts/pack.scm     | 17 +++++++++----
 guix/scripts/package.scm  | 50 ++++++++++++++++++++-------------------
 tests/guix-environment.sh | 23 ++++++++++++++++++
 tests/guix-pack.sh        | 11 +++++++++
 tests/guix-package.sh     | 13 ++++++++++
 tests/profiles.scm        |  5 ++++
 8 files changed, 101 insertions(+), 32 deletions(-)

-- 
2.24.0

[bug#38301] [PATCH 1/4] environment: Document that '--manifest' can be repeated.

[Ludovic Courtès]

* tests/guix-environment.sh: Test 'guix environment' with two '-m' options.
* doc/guix.texi (Invoking guix environment): Explain that '-m' can be
passed multiple times.
---
 doc/guix.texi             |  3 ++-
 tests/guix-environment.sh | 23 +++++++++++++++++++++++
 2 files changed, 25 insertions(+), 1 deletion(-)

diff --git a/doc/guix.texi b/doc/guix.texi
index a7ac6ff464..443d5bd71f 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -4802,7 +4802,8 @@ As an example, @var{file} might contain a definition like this
 @item --manifest=@var{file}
 @itemx -m @var{file}
 Create an environment for the packages contained in the manifest object
-returned by the Scheme code in @var{file}.
+returned by the Scheme code in @var{file}.  This option can be repeated
+several times, in which case the manifests are concatenated.
 
 This is similar to the same-named option in @command{guix package}
 (@pxref{profile-manifest, @option{--manifest}}) and uses the same
diff --git a/tests/guix-environment.sh b/tests/guix-environment.sh
index fb1c1a022d..2faf38df06 100644
--- a/tests/guix-environment.sh
+++ b/tests/guix-environment.sh
@@ -84,6 +84,29 @@ echo "(use-modules (guix profiles) (gnu packages bootstrap))
 guix environment --bootstrap --manifest=$tmpdir/manifest.scm --pure \
      -- "$SHELL" -c 'test -f "$GUIX_ENVIRONMENT/bin/guile"'
 
+# Make sure '--manifest' can be specified multiple times.
+cat > "$tmpdir/manifest2.scm" <<EOF
+(use-modules (guix) (guix profiles)
+             (guix build-system trivial)
+             (gnu packages bootstrap))
+
+(packages->manifest
+ (list (package
+         (inherit %bootstrap-guile)
+         (name "eliug")
+         (build-system trivial-build-system)
+         (arguments
+          (quasiquote
+           (#:guile ,%bootstrap-guile
+            #:builder
+            (begin
+              (mkdir %output)
+              (mkdir (string-append %output "/eliug")))))))))
+EOF
+guix environment --bootstrap -m "$tmpdir/manifest.scm" \
+     -m "$tmpdir/manifest2.scm" --pure \
+     -- "$SHELL" -c 'test -f "$GUIX_ENVIRONMENT/bin/guile" && test -d "$GUIX_ENVIRONMENT/eliug"'
+
 # Make sure '-r' works as expected.
 rm -f "$gcroot"
 expected="`guix environment --bootstrap --ad-hoc guile-bootstrap \
-- 
2.24.0

[bug#38301] [PATCH 2/4] profiles: Add 'concatenate-manifests'.

[Ludovic Courtès]

* guix/profiles.scm (concatenate-manifests): New procedure.
* tests/profiles.scm ("concatenate-manifests"): New test.
---
 guix/profiles.scm  | 5 +++++
 tests/profiles.scm | 5 +++++
 2 files changed, 10 insertions(+)

diff --git a/guix/profiles.scm b/guix/profiles.scm
index cd3b21e390..f5e5cc33d6 100644
--- a/guix/profiles.scm
+++ b/guix/profiles.scm
@@ -92,6 +92,7 @@
             manifest-pattern-version
             manifest-pattern-output
 
+            concatenate-manifests
             manifest-remove
             manifest-add
             manifest-lookup
@@ -515,6 +516,10 @@ procedure is here for backward-compatibility and will eventually vanish."
   "Return the packages listed in MANIFEST."
   (sexp->manifest (read port)))
 
+(define (concatenate-manifests lst)
+  "Concatenate the manifests listed in LST and return the resulting manifest."
+  (manifest (append-map manifest-entries lst)))
+
 (define (entry-predicate pattern)
   "Return a procedure that returns #t when passed a manifest entry that
 matches NAME/OUTPUT/VERSION.  OUTPUT and VERSION may be #f, in which case they
diff --git a/tests/profiles.scm b/tests/profiles.scm
index a4e28672b5..21c912a532 100644
--- a/tests/profiles.scm
+++ b/tests/profiles.scm
@@ -113,6 +113,11 @@
       (manifest-matching-entries m (list p))
       #f)))
 
+(test-equal "concatenate-manifests"
+  (manifest (list guile-2.0.9 glibc))
+  (concatenate-manifests (list (manifest (list guile-2.0.9))
+                               (manifest (list glibc)))))
+
 (test-assert "manifest-remove"
   (let* ((m0 (manifest (list guile-2.0.9 guile-2.0.9:debug)))
          (m1 (manifest-remove m0
-- 
2.24.0

[bug#38301] [PATCH 3/4] package: Allow multiple '--manifest' options.

[Ludovic Courtès]

* guix/scripts/package.scm (manifest-action): Remove.
(%actions): Remove it.
(load-manifest): New procedure.
(process-actions): Handle 'manifest' options.  Define 'files' from
'manifest' options.  Define 'manifest' based on FILES.  Define 'trans'
to represent the final transaction.
* tests/guix-package.sh: Test it.
* doc/guix.texi (Invoking guix package): Mention
---
 doc/guix.texi            |  3 ++-
 guix/scripts/package.scm | 50 +++++++++++++++++++++-------------------
 tests/guix-package.sh    | 13 +++++++++++
 3 files changed, 41 insertions(+), 25 deletions(-)

diff --git a/doc/guix.texi b/doc/guix.texi
index 443d5bd71f..9de1dbd6d5 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -2830,7 +2830,8 @@ $ guix package --upgrade . --do-not-upgrade emacs
 @cindex profile declaration
 @cindex profile manifest
 Create a new generation of the profile from the manifest object
-returned by the Scheme code in @var{file}.
+returned by the Scheme code in @var{file}.  This option can be repeated
+several times, in which case the manifests are concatenated.
 
 This allows you to @emph{declare} the profile's contents rather than
 constructing it through a sequence of @code{--install} and similar
diff --git a/guix/scripts/package.scm b/guix/scripts/package.scm
index bcd03a1df9..eb578f7642 100644
--- a/guix/scripts/package.scm
+++ b/guix/scripts/package.scm
@@ -832,32 +832,17 @@ processed, #f otherwise."
   (unless dry-run?
     (delete-matching-generations store profile pattern)))
 
-(define* (manifest-action store profile file opts
-                          #:key dry-run?)
-  "Change PROFILE to contain the packages specified in FILE."
-  (let* ((user-module  (make-user-module '((guix profiles) (gnu))))
-         (manifest     (load* file user-module))
-         (bootstrap?   (assoc-ref opts 'bootstrap?))
-         (substitutes? (assoc-ref opts 'substitutes?))
-         (allow-collisions? (assoc-ref opts 'allow-collisions?)))
-    (if dry-run?
-        (format #t (G_ "would install new manifest from '~a' with ~d entries~%")
-                file (length (manifest-entries manifest)))
-        (format #t (G_ "installing new manifest from '~a' with ~d entries~%")
-                file (length (manifest-entries manifest))))
-    (build-and-use-profile store profile manifest
-                           #:allow-collisions? allow-collisions?
-                           #:bootstrap? bootstrap?
-                           #:use-substitutes? substitutes?
-                           #:dry-run? dry-run?)))
+(define (load-manifest file)
+  "Load the user-profile manifest (Scheme code) from FILE and return it."
+  (let ((user-module (make-user-module '((guix profiles) (gnu)))))
+    (load* file user-module)))
 
 (define %actions
   ;; List of actions that may be processed.  The car of each pair is the
   ;; action's symbol in the option list; the cdr is the action's procedure.
   `((roll-back? . ,roll-back-action)
     (switch-generation . ,switch-generation-action)
-    (delete-generations . ,delete-generations-action)
-    (manifest . ,manifest-action)))
+    (delete-generations . ,delete-generations-action)))
 
 (define (process-actions store opts)
   "Process any install/remove/upgrade action from OPTS."
@@ -896,7 +881,13 @@ processed, #f otherwise."
               opts)
 
     ;; Then, process normal package removal/installation/upgrade.
-    (let* ((manifest (profile-manifest profile))
+    (let* ((files    (filter-map (match-lambda
+                                   (('manifest . file) file)
+                                   (_ #f))
+                                 opts))
+           (manifest (match files
+                       (() (profile-manifest profile))
+                       (_  (concatenate-manifests (map load-manifest files)))))
            (step1    (options->removable opts manifest
                                          (manifest-transaction)))
            (step2    (options->installable opts manifest step1))
@@ -904,12 +895,23 @@ processed, #f otherwise."
                       (inherit step2)
                       (install (map transform-entry
                                     (manifest-transaction-install step2)))))
-           (new      (manifest-perform-transaction manifest step3)))
+           (new      (manifest-perform-transaction manifest step3))
+           (trans    (if (null? files)
+                         step3
+                         (fold manifest-transaction-install-entry
+                               step3
+                               (manifest-entries manifest)))))
 
       (warn-about-old-distro)
 
-      (unless (manifest-transaction-null? step3)
-        (show-manifest-transaction store manifest step3
+      (unless (manifest-transaction-null? trans)
+        ;; When '--manifest' is used, display information about TRANS as if we
+        ;; were starting from an empty profile.
+        (show-manifest-transaction store
+                                   (if (null? files)
+                                       manifest
+                                       (make-manifest '()))
+                                   trans
                                    #:dry-run? dry-run?)
         (build-and-use-profile store profile new
                                #:allow-collisions? allow-collisions?
diff --git a/tests/guix-package.sh b/tests/guix-package.sh
index 7ad0699380..6d081d58be 100644
--- a/tests/guix-package.sh
+++ b/tests/guix-package.sh
@@ -394,6 +394,19 @@ guix package -I | grep guile
 test `guix package -I | wc -l` -eq 1
 guix package --rollback --bootstrap
 
+# Applying two manifests.
+cat > "$module_dir/manifest2.scm"<<EOF
+(use-modules (gnu packages bootstrap) (guix))
+(define p (package (inherit %bootstrap-guile) (name "eliug")))
+(packages->manifest (list p))
+EOF
+guix package --bootstrap \
+     -m "$module_dir/manifest.scm" -m "$module_dir/manifest2.scm"
+guix package -I | grep guile
+guix package -I | grep eliug
+test `guix package -I | wc -l` -eq 2
+guix package --rollback --bootstrap
+
 # Applying a manifest file with inferior packages.
 cat > "$module_dir/manifest.scm"<<EOF
 (use-modules (guix inferior))
-- 
2.24.0

[bug#38301] [PATCH 4/4] pack: Allow multiple '--manifest' options.

[Ludovic Courtès]

* guix/scripts/pack.scm (guix-pack): Collect 'manifest' options, and
concatenate the resulting manifests.
* tests/guix-pack.sh: Test it.
* doc/guix.texi (Invoking guix pack): Document it.
---
 doc/guix.texi         |  3 ++-
 guix/scripts/pack.scm | 17 ++++++++++++-----
 tests/guix-pack.sh    | 11 +++++++++++
 3 files changed, 25 insertions(+), 6 deletions(-)

diff --git a/doc/guix.texi b/doc/guix.texi
index 9de1dbd6d5..e702bc3055 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -5178,7 +5178,8 @@ build} (@pxref{Additional Build Options, @code{--expression} in
 @item --manifest=@var{file}
 @itemx -m @var{file}
 Use the packages contained in the manifest object returned by the Scheme
-code in @var{file}.
+code in @var{file}.  This option can be repeated several times, in which
+case the manifests are concatenated.
 
 This has a similar purpose as the same-named option in @command{guix
 package} (@pxref{profile-manifest, @option{--manifest}}) and uses the
diff --git a/guix/scripts/pack.scm b/guix/scripts/pack.scm
index 89b3e389fc..c8a52374bd 100644
--- a/guix/scripts/pack.scm
+++ b/guix/scripts/pack.scm
@@ -965,7 +965,10 @@ Create a bundle of PACKAGE.\n"))
                                   (list (transform store package) "out")))
                                (reverse
                                 (filter-map maybe-package-argument opts))))
-           (manifest-file (assoc-ref opts 'manifest)))
+           (manifests     (filter-map (match-lambda
+                                        (('manifest . file) file)
+                                        (_ #f))
+                                      opts)))
       (define properties
         (if (assoc-ref opts 'save-provenance?)
             (lambda (package)
@@ -979,11 +982,15 @@ Create a bundle of PACKAGE.\n"))
             (const '())))
 
       (cond
-       ((and manifest-file (not (null? packages)))
+       ((and (not (null? manifests)) (not (null? packages)))
         (leave (G_ "both a manifest and a package list were given~%")))
-       (manifest-file
-        (let ((user-module (make-user-module '((guix profiles) (gnu)))))
-          (load* manifest-file user-module)))
+       ((not (null? manifests))
+        (concatenate-manifests
+         (map (lambda (file)
+                (let ((user-module (make-user-module
+                                    '((guix profiles) (gnu)))))
+                  (load* file user-module)))
+              manifests)))
        (else
         (manifest
          (map (match-lambda
diff --git a/tests/guix-pack.sh b/tests/guix-pack.sh
index cf4e4ca4f9..7a0f3400c3 100644
--- a/tests/guix-pack.sh
+++ b/tests/guix-pack.sh
@@ -109,3 +109,14 @@ drv1="`guix pack -n guile 2>&1 | grep pack.*\.drv`"
 drv2="`guix pack -n --with-source=guile=$test_directory guile 2>&1 | grep pack.*\.drv`"
 test -n "$drv1"
 test "$drv1" != "$drv2"
+
+# Try '--manifest' options.
+cat > "$test_directory/manifest1.scm" <<EOF
+(specifications->manifest '("guile"))
+EOF
+cat > "$test_directory/manifest2.scm" <<EOF
+(specifications->manifest '("emacs"))
+EOF
+drv="`guix pack -nd -m "$test_directory/manifest1.scm" -m "$test_directory/manifest2.scm"`"
+guix gc -R "$drv" | grep `guix build guile -nd`
+guix gc -R "$drv" | grep `guix build emacs -nd`
-- 
2.24.0

[bug#38301] [PATCH 4/4] pack: Allow multiple '--manifest' options.

[brettg]

On 20.11.2019 14:44, Ludovic Courtès wrote:
> * guix/scripts/pack.scm (guix-pack): Collect 'manifest' options, and
> concatenate the resulting manifests.
> * tests/guix-pack.sh: Test it.
> * doc/guix.texi (Invoking guix pack): Document it.
> ---
>  doc/guix.texi         |  3 ++-
>  guix/scripts/pack.scm | 17 ++++++++++++-----
>  tests/guix-pack.sh    | 11 +++++++++++
>  3 files changed, 25 insertions(+), 6 deletions(-)
> 
> diff --git a/doc/guix.texi b/doc/guix.texi
> index 9de1dbd6d5..e702bc3055 100644
> --- a/doc/guix.texi
> +++ b/doc/guix.texi
> @@ -5178,7 +5178,8 @@ build} (@pxref{Additional Build Options,
> @code{--expression} in
>  @item --manifest=@var{file}
>  @itemx -m @var{file}
>  Use the packages contained in the manifest object returned by the 
> Scheme
> -code in @var{file}.
> +code in @var{file}.  This option can be repeated several times, in 
> which
> +case the manifests are concatenated.
> 
>  This has a similar purpose as the same-named option in @command{guix
>  package} (@pxref{profile-manifest, @option{--manifest}}) and uses the
> diff --git a/guix/scripts/pack.scm b/guix/scripts/pack.scm
> index 89b3e389fc..c8a52374bd 100644
> --- a/guix/scripts/pack.scm
> +++ b/guix/scripts/pack.scm
> @@ -965,7 +965,10 @@ Create a bundle of PACKAGE.\n"))
>                                    (list (transform store package) 
> "out")))
>                                 (reverse
>                                  (filter-map maybe-package-argument 
> opts))))
> -           (manifest-file (assoc-ref opts 'manifest)))
> +           (manifests     (filter-map (match-lambda
> +                                        (('manifest . file) file)
> +                                        (_ #f))
> +                                      opts)))
>        (define properties
>          (if (assoc-ref opts 'save-provenance?)
>              (lambda (package)
> @@ -979,11 +982,15 @@ Create a bundle of PACKAGE.\n"))
>              (const '())))
> 
>        (cond
> -       ((and manifest-file (not (null? packages)))
> +       ((and (not (null? manifests)) (not (null? packages)))
>          (leave (G_ "both a manifest and a package list were 
> given~%")))
> -       (manifest-file
> -        (let ((user-module (make-user-module '((guix profiles) 
> (gnu)))))
> -          (load* manifest-file user-module)))
> +       ((not (null? manifests))
> +        (concatenate-manifests
> +         (map (lambda (file)
> +                (let ((user-module (make-user-module
> +                                    '((guix profiles) (gnu)))))
> +                  (load* file user-module)))
> +              manifests)))
>         (else
>          (manifest
>           (map (match-lambda
> diff --git a/tests/guix-pack.sh b/tests/guix-pack.sh
> index cf4e4ca4f9..7a0f3400c3 100644
> --- a/tests/guix-pack.sh
> +++ b/tests/guix-pack.sh
> @@ -109,3 +109,14 @@ drv1="`guix pack -n guile 2>&1 | grep 
> pack.*\.drv`"
>  drv2="`guix pack -n --with-source=guile=$test_directory guile 2>&1 |
> grep pack.*\.drv`"
>  test -n "$drv1"
>  test "$drv1" != "$drv2"
> +
> +# Try '--manifest' options.
> +cat > "$test_directory/manifest1.scm" <<EOF
> +(specifications->manifest '("guile"))
> +EOF
> +cat > "$test_directory/manifest2.scm" <<EOF
> +(specifications->manifest '("emacs"))
> +EOF
> +drv="`guix pack -nd -m "$test_directory/manifest1.scm" -m
> "$test_directory/manifest2.scm"`"
> +guix gc -R "$drv" | grep `guix build guile -nd`
> +guix gc -R "$drv" | grep `guix build emacs -nd`

At last! This feature is going to be available.

These patches LGTM!

Brett

[bug#38301] [PATCH 0/4] Allow for multiple '--manifest' options

[Konrad Hinsen]

Hi Ludo,

> As discussed on guix-devel, this patch series ensures that one can
> pass multiple ‘--manifest’ options to ‘guix package’, ‘guix pack’,
> and ‘guix environment’ (which already supported it actually!).
>
> Feedback welcome!

Looks good. And very nice to have!

Thanks,
  Konrad.

bug#38301: [PATCH 4/4] pack: Allow multiple '--manifest' options.

[Ludovic Courtès]

Hi,

brettg@posteo.net skribis:

> At last! This feature is going to be available.
>
> These patches LGTM!

Konrad Hinsen <konrad.hinsen@fastmail.net> skribis:

> Looks good. And very nice to have!

Pushed!

  ca541f9ce6 pack: Allow multiple '--manifest' options.
  bf9206d8ed package: Allow multiple '--manifest' options.
  ce30a0eb7e profiles: Add 'concatenate-manifests'.
  d76df98fa5 environment: Document that '--manifest' can be repeated.

Thanks for your feedback,
Ludo’.

[bug#38301] [PATCH 0/4] Allow for multiple '--manifest' options

[zimoun]

Hi Ludo,

On Wed, 20 Nov 2019 at 17:34, Konrad Hinsen <konrad.hinsen@fastmail.net> wrote:
>
> Hi Ludo,
>
> > As discussed on guix-devel, this patch series ensures that one can
> > pass multiple ‘--manifest’ options to ‘guix package’, ‘guix pack’,
> > and ‘guix environment’ (which already supported it actually!).
> >
> > Feedback welcome!

Neat!

The test guix-pack fails but because I am trying to use "git worktree"
so maybe I am doing something wrong.
I am investigating in tests-suite.log.


Cheers,
simon