[bug#54393] [PATCH 0/2] Add 'guix manifest' to "translate" commands to manifests

[bug#54393] [PATCH 0/2] Add 'guix manifest' to "translate" commands to manifests

[Ludovic Courtès]

Hello Guix!

Writing manifests can be intimidating: in part because it’s not
well documented, and in part because it’s always hard to start from
a blank page.

We have ‘guix package --export-manifest’ to help with that, but I
realized that colleagues of mine often use ‘guix shell’ more than
‘guix package’, and ‘--export-manifest’ is awkward to use:

  $ guix shell a b c d --transform=whatever=else
  [env]$ guix package --export-manifest -p $GUIX_ENVIRONMENT

I figured that instead of documenting this, we might as well provide
a command that does it in one go:

  guix manifest a b c d --transform=whatever=else > manifest.scm

Voilà!

Thoughts?

Next up: adding a “Writing Manifests” section.

Ludo’.

Ludovic Courtès (2):
  packages: Add 'package-unique-version-prefix'.
  Add 'guix manifest'.

 Makefile.am               |   2 +
 doc/guix.texi             | 146 +++++++++++++++++++++++++++++++-
 gnu/packages.scm          |  21 +++++
 guix/scripts/manifest.scm | 174 ++++++++++++++++++++++++++++++++++++++
 guix/scripts/package.scm  |  20 +----
 po/guix/POTFILES.in       |   1 +
 tests/guix-manifest.sh    |  76 +++++++++++++++++
 tests/packages.scm        |  13 +++
 8 files changed, 434 insertions(+), 19 deletions(-)
 create mode 100644 guix/scripts/manifest.scm
 create mode 100644 tests/guix-manifest.sh


base-commit: 857e5ab22292fe7c2ba5b9855c0829a2ef942d38
-- 
2.34.0

[bug#54393] [PATCH 1/2] packages: Add 'package-unique-version-prefix'.

[Ludovic Courtès]

* gnu/packages.scm (package-unique-version-prefix): New procedure.
* guix/scripts/package.scm (manifest-entry-version-prefix): Use it.
* tests/packages.scm ("package-unique-version-prefix, gcc@8")
("package-unique-version-prefix, grep"): New tests.
---
 gnu/packages.scm         | 21 +++++++++++++++++++++
 guix/scripts/package.scm | 20 ++------------------
 tests/packages.scm       | 13 +++++++++++++
 3 files changed, 36 insertions(+), 18 deletions(-)

diff --git a/gnu/packages.scm b/gnu/packages.scm
index 65ab7a7c1e..2ba838fd0a 100644
--- a/gnu/packages.scm
+++ b/gnu/packages.scm
@@ -66,6 +66,8 @@ (define-module (gnu packages)
             specification->location
             specifications->manifest
 
+            package-unique-version-prefix
+
             generate-package-cache))
 
 ;;; Commentary:
@@ -559,3 +561,22 @@ (define (specifications->manifest specs)
   ;; fiddle with multiple-value returns.
   (packages->manifest
    (map (compose list specification->package+output) specs)))
+
+(define (package-unique-version-prefix name version)
+  "Search among all the versions of package NAME that are available, and
+return the shortest unambiguous version prefix to designate VERSION.  If only
+one version of the package is available, return the empty string."
+  (match (map package-version (find-packages-by-name name))
+    ((_)
+     ;; A single version of NAME is available, so do not specify the version
+     ;; number, even if the available version doesn't match VERSION.
+     "")
+    (versions
+     ;; If VERSION is the latest version, don't specify any version.
+     ;; Otherwise return the shortest unique version prefix.  Note that this
+     ;; is based on the currently available packages so the result may vary
+     ;; over time.
+     (if (every (cut version>? version <>)
+                (delete version versions))
+         ""
+         (version-unique-prefix version versions)))))
diff --git a/guix/scripts/package.scm b/guix/scripts/package.scm
index 9699c70c6d..22ee8a2485 100644
--- a/guix/scripts/package.scm
+++ b/guix/scripts/package.scm
@@ -334,24 +334,8 @@ (define (manifest-entry-version-prefix entry)
   "Search among all the versions of ENTRY's package that are available, and
 return the shortest unambiguous version prefix for this package.  If only one
 version of ENTRY's package is available, return the empty string."
-  (let ((name (manifest-entry-name entry)))
-    (match (map package-version (find-packages-by-name name))
-      ((_)
-       ;; A single version of NAME is available, so do not specify the
-       ;; version number, even if the available version doesn't match ENTRY.
-       "")
-      (versions
-       ;; If ENTRY uses the latest version, don't specify any version.
-       ;; Otherwise return the shortest unique version prefix.  Note that
-       ;; this is based on the currently available packages, which could
-       ;; differ from the packages available in the revision that was used
-       ;; to build MANIFEST.
-       (let ((current (manifest-entry-version entry)))
-         (if (every (cut version>? current <>)
-                    (delete current versions))
-             ""
-             (version-unique-prefix (manifest-entry-version entry)
-                                    versions)))))))
+  (package-unique-version-prefix (manifest-entry-name entry)
+                                 (manifest-entry-version entry)))
 
 (define* (export-manifest manifest
                           #:optional (port (current-output-port)))
diff --git a/tests/packages.scm b/tests/packages.scm
index 02bdba5f98..b228c9fc3b 100644
--- a/tests/packages.scm
+++ b/tests/packages.scm
@@ -1923,6 +1923,19 @@ (define (list->set* lst)
   (package-location (specification->package "guile@2"))
   (specification->location "guile@2"))
 
+(test-equal "package-unique-version-prefix, gcc@8"
+  "8"
+  (let ((gcc (specification->package "gcc-toolchain@8")))
+    (package-unique-version-prefix (package-name gcc)
+                                   (package-version gcc))))
+
+(test-equal "package-unique-version-prefix, grep"
+  ""
+  (let ((grep (specification->package "grep")))
+    (package-unique-version-prefix (package-name grep)
+                                   (package-version grep))))
+
+
 (test-eq "this-package-input, exists"
   hello
   (package-arguments
-- 
2.34.0

[bug#54393] [PATCH 2/2] Add 'guix manifest'.

[Ludovic Courtès]

From: Ludovic Courtès <ludovic.courtes@inria.fr>

* guix/scripts/manifest.scm: New file.
* po/guix/POTFILES.in: Add it.
* tests/guix-manifest.sh: New file.
* Makefile.am (MODULES, SH_TESTS): Add them.
* doc/guix.texi (Invoking guix manifest): New section.
(Invoking guix package): Refer to it.
(Invoking guix shell): Likewise.
(Invoking guix environment): Likewise.
(Invoking guix pack): Likewise.
---
 Makefile.am               |   2 +
 doc/guix.texi             | 146 +++++++++++++++++++++++++++++++-
 guix/scripts/manifest.scm | 174 ++++++++++++++++++++++++++++++++++++++
 po/guix/POTFILES.in       |   1 +
 tests/guix-manifest.sh    |  76 +++++++++++++++++
 5 files changed, 398 insertions(+), 1 deletion(-)
 create mode 100644 guix/scripts/manifest.scm
 create mode 100644 tests/guix-manifest.sh

diff --git a/Makefile.am b/Makefile.am
index 7402c89b62..40b6c75e23 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -318,6 +318,7 @@ MODULES =					\
   guix/scripts/import/pypi.scm			\
   guix/scripts/import/stackage.scm		\
   guix/scripts/import/texlive.scm  		\
+  guix/scripts/manifest.scm			\
   guix/scripts/environment.scm			\
   guix/scripts/shell.scm			\
   guix/scripts/publish.scm			\
@@ -568,6 +569,7 @@ SH_TESTS =					\
   tests/guix-environment.sh			\
   tests/guix-environment-container.sh		\
   tests/guix-shell.sh				\
+  tests/guix-manifest.sh			\
   tests/guix-graph.sh				\
   tests/guix-describe.sh			\
   tests/guix-repl.sh     			\
diff --git a/doc/guix.texi b/doc/guix.texi
index dbe281ead7..4dc3c8b1fc 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -233,6 +233,7 @@ Package Management
 * Invoking guix package::       Package installation, removal, etc.
 * Substitutes::                 Downloading pre-built binaries.
 * Packages with Multiple Outputs::  Single source package, multiple outputs.
+* Invoking guix manifest::      Producing environment declarations.
 * Invoking guix gc::            Running the garbage collector.
 * Invoking guix pull::          Fetching the latest Guix and distribution.
 * Invoking guix time-machine::  Running an older revision of Guix.
@@ -3042,6 +3043,7 @@ guix install emacs-guix
 * Invoking guix package::       Package installation, removal, etc.
 * Substitutes::                 Downloading pre-built binaries.
 * Packages with Multiple Outputs::  Single source package, multiple outputs.
+* Invoking guix manifest::      Producing environment declarations.
 * Invoking guix gc::            Running the garbage collector.
 * Invoking guix pull::          Fetching the latest Guix and distribution.
 * Invoking guix time-machine::  Running an older revision of Guix.
@@ -3412,7 +3414,9 @@ The example above gives you all the software required to develop Emacs,
 similar to what @command{guix environment emacs} provides.
 
 @xref{export-manifest, @option{--export-manifest}}, to learn how to
-obtain a manifest file from an existing profile.
+obtain a manifest file from an existing profile, and @pxref{Invoking
+guix manifest} on how to generate a manifest file from a list of package
+specs and command-line options.
 
 @item --roll-back
 @cindex rolling back
@@ -4159,6 +4163,137 @@ Files}).  The outputs of a packages are listed in the third column of
 the output of @command{guix package --list-available} (@pxref{Invoking
 guix package}).
 
+@node Invoking guix manifest
+@section Invoking @command{guix manifest}
+
+@cindex manifest, generating
+The @command{guix manifest} command outputs a @dfn{manifest}
+corresponding to the packages and options specified on the command line.
+Manifests are code snippets that @emph{declare} the set of packages you
+want to deploy---you can view them as a more expressive form of what you
+pass on the command line to @command{guix install}, @command{guix
+shell}, etc.  All these commands accept a @option{--manifest} (or
+@option{-m}) option that allows you to pass them a manifest.  For
+non-trivial package sets and customizations, you'll find that using a
+manifest rather than a long command line is often more convenient.
+
+But how do you go from that long command line you're familiar with to
+that appealing but possibly intimidating ``manifest'' thing?  The
+@command{guix manifest} command is your companion on this journey: it
+essentially ``translates'' command-line arguments into manifests.
+
+Let's look at a few examples.  What's a manifest corresponding to a
+basic list of package specifications?  We can figure out by running,
+say:
+
+@example
+guix manifest coreutils grep sed
+@end example
+
+@noindent
+... which outputs this manifest:
+
+@lisp
+(specifications->manifest
+  (list "coreutils" "grep" "sed"))
+@end lisp
+
+The manifest constructs a list containing the three @dfn{package specs}
+and passes it to the @code{specifications->manifest} procedure, which
+returns a manifest corresponding of the three designated packages.
+
+@quotation Note
+Manifests are @emph{symbolic}: they refer to packages by their
+specification (name and optionally version), which denote different
+packages over time---@code{coreutils} above might refer to version 8.32
+today and to 9.0 six months from now.
+
+To ``pin'' a package set to a specific revision, you will additionally
+need a @dfn{channel file} as produced by @command{guix describe -f
+channels} (@pxref{Invoking guix describe}).
+@end quotation
+
+That one was easy, but @command{guix manifest} can also handle more
+complex cases.  For example, consider the manifest for the development
+environment of Guile, with the addition of Git:
+
+@example
+guix manifest -D guile git
+@end example
+
+@noindent
+This gives us:
+
+@example
+(concatenate-manifests
+  (list (specifications->manifest (list "git"))
+        (package->development-manifest
+          (specification->package "guile"))))
+@end example
+
+The @command{guix manifest} command also takes care package
+transformation options, producing a manifest that faithfully reapplies
+them (@pxref{Package Transformation Options}):
+
+@example
+guix manifest intel-mpi-benchmarks --with-input=openmpi=mpich
+@end example
+
+@noindent
+... yields:
+
+@lisp
+(use-modules (guix transformations))
+
+(define transform1
+  (options->transformation
+    '((with-input . "openmpi=mpich"))))
+
+(packages->manifest
+  (list (transform1
+          (specification->package "intel-mpi-benchmarks"))))
+@end lisp
+
+Convenient, no?  You can take the output of @command{guix manifest}
+as-is, store it in a file, and enjoy it.  But you can also view it as
+raw material that you can modify to refine the expression of the
+environment you want to deploy.
+
+The general syntax is:
+
+@example
+guix manifest [@var{options}] @var{spec}@dots{}
+@end example
+
+@noindent
+... where each @var{spec} denotes a package and (optionally) its output,
+such as @code{emacs}, @code{gcc-toolchain@@8}, or
+@code{git:send-email}.  The available options are:
+
+@table @option
+@item --development
+@itemx -D
+Consider the environment needed to @emph{develop} the following package
+rather than the package itself.
+
+For instance, to obtain a manifest representing the environment to
+develop Elixir (and not Elixir itself), with the addition of Nano, run:
+
+@example
+guix manifest -D elixir nano
+@end example
+
+In this example, @option{-D} affects @code{elixir}, not @code{nano}.
+
+@item --manifest=@var{file}
+@itemx -m @var{file}
+Read the manifest in @var{file} and combine it with other options to
+produce the resulting manifest.
+@end table
+
+Additionally, @command{guix manifest} understands all the package
+transformation options (@pxref{Package Transformation Options}).
+
 
 @node Invoking guix gc
 @section Invoking @command{guix gc}
@@ -5847,6 +5982,9 @@ This is similar to the same-named option in @command{guix package}
 (@pxref{profile-manifest, @option{--manifest}}) and uses the same
 manifest files.
 
+@xref{Invoking guix manifest}, for information on how to ``convert''
+command-line options into a manifest.
+
 @item --profile=@var{profile}
 @itemx -p @var{profile}
 Create an environment containing the packages installed in @var{profile}. 
@@ -6234,6 +6372,9 @@ This is similar to the same-named option in @command{guix package}
 (@pxref{profile-manifest, @option{--manifest}}) and uses the same
 manifest files.
 
+@xref{Invoking guix manifest}, for information on how to ``convert''
+command-line options into a manifest.
+
 @item --ad-hoc
 Include all specified packages in the resulting environment, as if an
 @i{ad hoc} package were defined with them as inputs.  This option is
@@ -6692,6 +6833,9 @@ for use on machines that do not have Guix installed.  Note that you can
 specify @emph{either} a manifest file @emph{or} a list of packages,
 but not both.
 
+@xref{Invoking guix manifest}, for information on how to ``convert''
+command-line options into a manifest.
+
 @item --system=@var{system}
 @itemx -s @var{system}
 Attempt to build for @var{system}---e.g., @code{i686-linux}---instead of
diff --git a/guix/scripts/manifest.scm b/guix/scripts/manifest.scm
new file mode 100644
index 0000000000..fea5d130c3
--- /dev/null
+++ b/guix/scripts/manifest.scm
@@ -0,0 +1,174 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2022 Ludovic Courtès <ludo@gnu.org>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (guix scripts manifest)
+  #:use-module (guix ui)
+  #:use-module ((guix diagnostics) #:select (location))
+  #:use-module (guix scripts environment)
+  #:use-module (guix transformations)
+  #:use-module (guix scripts)
+  #:use-module (guix packages)
+  #:use-module (guix profiles)
+  #:autoload   (gnu packages) (specifications->manifest
+                               specification->package
+                               package-unique-version-prefix)
+  #:use-module (srfi srfi-1)
+  #:use-module (srfi srfi-26)
+  #:use-module (srfi srfi-37)
+  #:use-module (srfi srfi-71)
+  #:use-module (ice-9 match)
+  #:autoload   (ice-9 pretty-print) (pretty-print)
+  #:export (guix-manifest))
+
+(define (show-help)
+  (display (G_ "Usage: guix manifest [OPTION] SPECS...
+Print a manifest corresponding to the given package SPECS.\n"))
+  (newline)
+
+  (display (G_ "
+  -D, --development      include the development inputs of the next package"))
+  (display (G_ "
+  -m, --manifest=FILE    create environment with the manifest from FILE"))
+
+  (newline)
+  (show-transformation-options-help)
+  (newline)
+  (display (G_ "
+  -h, --help             display this help and exit"))
+  (display (G_ "
+  -V, --version          display version information and exit"))
+  (newline)
+  (show-bug-report-information))
+
+(define %options
+  ;; Specification of the command-line options.
+  (cons* (option '(#\h "help") #f #f
+                 (lambda args
+                   (show-help)
+                   (exit 0)))
+         (option '(#\V "version") #f #f
+                 (lambda args
+                   (show-version-and-exit "guix manifest")))
+
+         (option '(#\D "development") #f #f
+                 (lambda (opt name arg result)
+                   (alist-cons 'development? #t result)))
+         (option '(#\m "manifest") #t #f
+                 (lambda (opt name arg result)
+                   (alist-cons 'manifest arg result)))
+
+         %transformation-options))
+
+(define %default-options
+  ;; Default option alist.
+  '())
+
+(define (load-manifest file)                      ;TODO: factorize
+  "Load the user-profile manifest (Scheme code) from FILE and return it."
+  (let ((user-module (make-user-module '((guix profiles) (gnu)))))
+    (load* file user-module)))
+
+(define (manifest-entry-version-prefix entry)
+  "Search among all the versions of ENTRY's package that are available, and
+return the shortest unambiguous version prefix for this package."
+  (package-unique-version-prefix (manifest-entry-name entry)
+                                 (manifest-entry-version entry)))
+
+(define (manifest->code* manifest extra-manifests)
+  "Like 'manifest->code', but insert a 'concatenate-manifests' call that
+concatenates MANIFESTS, a list of expressions."
+  (if (null? (manifest-entries manifest))
+      (match extra-manifests
+        ((one) one)
+        (lst   `(concatenate-manifests ,@extra-manifests)))
+      (match (manifest->code manifest
+                             #:entry-package-version
+                             manifest-entry-version-prefix)
+        (('begin exp ... last)
+         `(begin
+            ,@exp
+            ,(match extra-manifests
+               (() last)
+               (_  `(concatenate-manifests
+                     (list ,last ,@extra-manifests)))))))))
+
+\f
+(define-command (guix-manifest . args)
+  (category development)
+  (synopsis "turn command-line arguments into a manifest")
+
+  (define (manifest-lift proc)
+    (lambda (entry)
+      (match (manifest-entry-item entry)
+        ((? package? p)
+         (manifest-entry
+           (inherit (package->manifest-entry (proc p)))
+           (output (manifest-entry-output entry))))
+        (_
+         entry))))
+
+  (define (handle-argument arg result)
+    (if (assoc-ref result 'development?)
+        (alist-cons 'development-inputs arg
+                    (alist-delete 'development? result))
+        (alist-cons 'argument arg result)))
+
+  (with-error-handling
+    (let* ((opts (parse-command-line args %options (list %default-options)
+                                     #:build-options? #f
+                                     #:argument-handler handle-argument))
+           (transform (options->transformation opts))
+           (specs     (reverse
+                       (filter-map (match-lambda
+                                     (('argument . spec) spec)
+                                     (_ #f))
+                                   opts)))
+           (extras    (reverse
+                       (filter-map (match-lambda
+                                     (('development-inputs . spec)
+                                      ;; Make sure SPEC is valid.
+                                      (specification->package spec)
+
+                                      ;; XXX: This is an approximation:
+                                      ;; transformation options are not
+                                      ;; applied.
+                                      `(package->development-manifest
+                                        (specification->package ,spec)))
+                                     (_ #f))
+                                   opts)))
+           (manifest  (concatenate-manifests
+                       (cons (map-manifest-entries
+                              (manifest-lift transform)
+                              (specifications->manifest specs))
+                             (filter-map (match-lambda
+                                           (('manifest . file)
+                                            (load-manifest file))
+                                           (_ #f))
+                                         opts)))))
+      (display (G_ "\
+;; What follows is a \"manifest\" equivalent to the command line you gave.
+;; You can store it in a file that you may then pass to any 'guix' command
+;; that accepts a '--manifest' (or '-m') option.\n"))
+      (match (manifest->code* manifest extras)
+        (('begin exp ...)
+         (for-each (lambda (exp)
+                     (newline)
+                     (pretty-print exp))
+                   exp))
+        (exp
+         (pretty-print exp))))))
diff --git a/po/guix/POTFILES.in b/po/guix/POTFILES.in
index d97ba8c209..5b8eadf884 100644
--- a/po/guix/POTFILES.in
+++ b/po/guix/POTFILES.in
@@ -99,6 +99,7 @@ guix/scripts/weather.scm
 guix/scripts/describe.scm
 guix/scripts/processes.scm
 guix/scripts/deploy.scm
+guix/scripts/manifest.scm
 guix/gexp.scm
 guix/gnu-maintenance.scm
 guix/scripts/container.scm
diff --git a/tests/guix-manifest.sh b/tests/guix-manifest.sh
new file mode 100644
index 0000000000..de82815ba0
--- /dev/null
+++ b/tests/guix-manifest.sh
@@ -0,0 +1,76 @@
+# GNU Guix --- Functional package management for GNU
+# Copyright © 2022 Ludovic Courtès <ludo@gnu.org>
+#
+# This file is part of GNU Guix.
+#
+# GNU Guix is free software; you can redistribute it and/or modify it
+# under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or (at
+# your option) any later version.
+#
+# GNU Guix is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
+
+#
+# Test 'guix manifest'.
+#
+
+guix manifest --version
+
+tmpdir="t-guix-manifest-$$"
+trap 'rm -r "$tmpdir"' EXIT
+mkdir "$tmpdir"
+
+manifest="$tmpdir/manifest.scm"
+
+# Basics.
+guix manifest guile-bootstrap > "$manifest"
+test "$(guix build -m "$manifest")" = "$(guix build guile-bootstrap)"
+
+guix shell -m "$manifest" --bootstrap -- \
+     "$SHELL" -c 'guix package --export-manifest -p "$GUIX_ENVIRONMENT"' > \
+     "$manifest.second"
+for m in "$manifest" "$manifest.second"
+do
+    grep -v '^;' < "$m" > "$m.new" # filter out comments
+    mv "$m.new" "$m"
+done
+
+cat "$manifest"
+cat "$manifest.second"
+
+cmp "$manifest" "$manifest.second"
+
+# Package transformation option.
+guix manifest guile guix --with-latest=guile-json > "$manifest"
+grep 'options->transformation' "$manifest"
+grep '(with-latest . "guile-json")' "$manifest"
+
+# Development manifest.
+guix manifest -D guile git > "$manifest"
+grep 'package->development-manifest' "$manifest"
+grep '"guile"' "$manifest"
+guix build -m "$manifest" -d | \
+    grep "$(guix build -e '(@@ (gnu packages commencement) gcc-final)' -d)"
+guix build -m "$manifest" -d | \
+    grep "$(guix build git -d)"
+
+# Test various combinations to make sure generated code uses interfaces
+# correctly.
+for options in					\
+    "coreutils grep sed"			\
+    "gsl openblas gcc-toolchain --tune"		\
+    "guile -m $manifest.previous"		\
+    "git:send-email gdb guile:debug"		\
+    "git -D coreutils"
+do
+    guix manifest $options > "$manifest"
+    cat "$manifest"
+    guix shell -m "$manifest" -n
+    mv "$manifest" "$manifest.previous"
+done
-- 
2.34.0

[bug#54393] [PATCH 0/2] Add 'guix manifest' to "translate" commands to manifests

[Liliana Marie Prikler]

Hi,

Am Montag, dem 14.03.2022 um 22:50 +0100 schrieb Ludovic Courtès:
> Hello Guix!
> 
> Writing manifests can be intimidating: in part because it’s not
> well documented, and in part because it’s always hard to start from
> a blank page.
> 
> We have ‘guix package --export-manifest’ to help with that, but I
> realized that colleagues of mine often use ‘guix shell’ more than
> ‘guix package’, and ‘--export-manifest’ is awkward to use:
> 
>   $ guix shell a b c d --transform=whatever=else
>   [env]$ guix package --export-manifest -p $GUIX_ENVIRONMENT
> 
> I figured that instead of documenting this, we might as well provide
> a command that does it in one go:
> 
>   guix manifest a b c d --transform=whatever=else > manifest.scm
> 
> Voilà!
> 
> Thoughts?
The current implementation is missing the most important feature, which
would be ‘guix manifest --profile $GUIX_PROFILE’ :)
I think if we are to introduce a new command, we should make it handle
existing use cases as well.  If people then prefer ‘guix manifest’ over
‘guix package --export-manifest’, we can deprecate the old command in
favor of the new one.

Furthermore, one issue I have with both the existing and proposed
approaches is that both mandate the specification style, whereas for
operating system configs we typically prefer the variable style of
referencing a package.  Adding an option to choose between the two
would be much appreciated from my side.

Cheers

[bug#54393] [PATCH 0/2] Add 'guix manifest' to "translate" commands to manifests

[Ludovic Courtès]

Hi,

Liliana Marie Prikler <liliana.prikler@ist.tugraz.at> skribis:

> The current implementation is missing the most important feature, which
> would be ‘guix manifest --profile $GUIX_PROFILE’ :)

True, we could add ‘-p’ and make it equivalent to ‘guix package
--export-manifest’.

I think lack of ‘-p’ is not a showstopper though since ‘guix package
--export-manifest’ remains available.

[...]

> Furthermore, one issue I have with both the existing and proposed
> approaches is that both mandate the specification style,

The reasons for the specification style are:

  1. That it’s easier to implement reliably, since you just need to rely
     on package specs.

  2. The resulting code is future-proof: if packages are moved from one
     module to another, or if their variables are renamed, it still
     works.

  3. It’s a one-to-one mapping compared to the command line, which
     hopefully makes it clearer.

However, ‘manifest->code’ existed before this patch series, so I think
it’s beyond the scope of this review.

Thanks,
Ludo’.

[bug#54393] [PATCH 0/2] Add 'guix manifest' to "translate" commands to manifests

[Liliana Marie Prikler]

Am Dienstag, dem 15.03.2022 um 10:27 +0100 schrieb Ludovic Courtès:
> Hi,
> 
> Liliana Marie Prikler <liliana.prikler@ist.tugraz.at> skribis:
> 
> > The current implementation is missing the most important feature,
> > which would be ‘guix manifest --profile $GUIX_PROFILE’ :)
> 
> True, we could add ‘-p’ and make it equivalent to ‘guix package
> --export-manifest’.
> 
> I think lack of ‘-p’ is not a showstopper though since ‘guix package
> --export-manifest’ remains available.
True, but implementing it as part of ‘guix manifest’ would open up a
way to easily declare a manifest for “merged” profiles.  These could be
applied in integration testing, e.g. you could merge multiple profiles
together and watch for intended and unintended side effects, for
instance regarding search paths.

> > Furthermore, one issue I have with both the existing and proposed
> > approaches is that both mandate the specification style,
> 
> The reasons for the specification style are:
> 
>   1. That it’s easier to implement reliably, since you just need to 
> rely on package specs.
> 
>   2. The resulting code is future-proof: if packages are moved from
>      one module to another, or if their variables are renamed, it 
>      still works.
> 
>   3. It’s a one-to-one mapping compared to the command line, which
>      hopefully makes it clearer.
> 
> However, ‘manifest->code’ existed before this patch series, so I
> think it’s beyond the scope of this review.
It's true that it's out of scope, but I don't want this to be
forgotten.  I can see the rationale of specification style for user
profiles, but for version-controlled package development manifests,
there are different incentives in play.  For example, you might want to
make sure that a package uses gcc-toolchain from (gnu packages
commencement) and not (google monorepo cc).

Cheers

[bug#54393] [PATCH 0/2] Add 'guix manifest' to "translate" commands to manifests

[Ludovic Courtès]

Liliana Marie Prikler <liliana.prikler@ist.tugraz.at> skribis:

> Am Dienstag, dem 15.03.2022 um 10:27 +0100 schrieb Ludovic Courtès:
>> Hi,
>> 
>> Liliana Marie Prikler <liliana.prikler@ist.tugraz.at> skribis:
>> 
>> > The current implementation is missing the most important feature,
>> > which would be ‘guix manifest --profile $GUIX_PROFILE’ :)
>> 
>> True, we could add ‘-p’ and make it equivalent to ‘guix package
>> --export-manifest’.
>> 
>> I think lack of ‘-p’ is not a showstopper though since ‘guix package
>> --export-manifest’ remains available.
> True, but [...]

What I meant is that I agree this is a useful addition, though one that
could possibly come after.

Thanks,
Ludo’.

[bug#54393] [PATCH 0/2] Add 'guix manifest' to "translate" commands to manifests

[zimoun]

Hi Ludo,

On Mon, 14 Mar 2022 at 22:53, Ludovic Courtès <ludo@gnu.org> wrote:

>   $ guix shell a b c d --transform=whatever=else
>   [env]$ guix package --export-manifest -p $GUIX_ENVIRONMENT
>
> I figured that instead of documenting this, we might as well provide
> a command that does it in one go:
>
>   guix manifest a b c d --transform=whatever=else > manifest.scm

From my opinion, a complete subcommand for only one use case does not
seem the right thing.  Instead, IMHO, it would be better to have an
extension for that.

Because once we introduce a subcommand, it is hard to change (see
'guix environment') and since Guix provides this great extension
mechanism, I think it is the path for this kind of niche use cases:
start with an extension and depending on the "popularity" and interest
then upstream.

Well, if you think this feature is great to have in Guix proper,
personally I would find more coherent:

    guix package --create-manifest a b c --transform=whatever=else >
manifest.scm


Cheers,
simon

[bug#54393] [PATCH 0/2] Add 'guix manifest' to "translate" commands to manifests

[Ludovic Courtès]

Hi zimoun,

zimoun <zimon.toutoune@gmail.com> skribis:

> On Mon, 14 Mar 2022 at 22:53, Ludovic Courtès <ludo@gnu.org> wrote:
>
>>   $ guix shell a b c d --transform=whatever=else
>>   [env]$ guix package --export-manifest -p $GUIX_ENVIRONMENT
>>
>> I figured that instead of documenting this, we might as well provide
>> a command that does it in one go:
>>
>>   guix manifest a b c d --transform=whatever=else > manifest.scm
>
>>From my opinion, a complete subcommand for only one use case does not
> seem the right thing.  Instead, IMHO, it would be better to have an
> extension for that.
>
> Because once we introduce a subcommand, it is hard to change (see
> 'guix environment') and since Guix provides this great extension
> mechanism, I think it is the path for this kind of niche use cases:
> start with an extension and depending on the "popularity" and interest
> then upstream.
>
> Well, if you think this feature is great to have in Guix proper,
> personally I would find more coherent:
>
>     guix package --create-manifest a b c --transform=whatever=else >
> manifest.scm

OTOH, sub-commands as just as cheap (and as expensive!) as command-line
options.  That is, one way or another, we’ll have to maintain the thing,
whether it’s called ‘--create-manifest’ or ‘manifest’.

To me, the argument in favor of the sub-command is that that it’s more
discoverable, clearer, and easier to use.  A user who has a ‘guix shell’
command line can replace ‘shell’ by ‘manifest’ and get their manifest.

Thinking about it, another option would be to add an ‘--export-manifest’
option to ‘guix shell’ instead.

Thoughts?

Ludo’.

[bug#54393] [PATCH 0/2] Add 'guix manifest' to "translate" commands to manifests

[zimoun]

Hi,

On Tue, 15 Mar 2022 at 10:23, Ludovic Courtès <ludo@gnu.org> wrote:

> OTOH, sub-commands as just as cheap (and as expensive!) as command-line
> options.  That is, one way or another, we’ll have to maintain the thing,
> whether it’s called ‘--create-manifest’ or ‘manifest’.

I agree...

> To me, the argument in favor of the sub-command is that that it’s more
> discoverable, clearer, and easier to use.  A user who has a ‘guix shell’
> command line can replace ‘shell’ by ‘manifest’ and get their manifest.

...but I am not convinced. :-)  For instance, about discoverability,
assuming "guix install / remove" would not be an alias of "guix
package", then I would go va "guix help" then "guix install --help"
then probably via "guix help" again, then "guix remove --help" etc.
An extreme counter-example to your "clearer" point about subcommands
is Git: I do not consider such many subcommands more discoverable
because it gives the impression of a scattered mess.  Subcommands
provide a thematic hierarchy.  For sure, it is a balance. :-)  Because
Guix is doing many many things, it is hard for people to understand,
IMHO, therefore a sub-command hierarchy helps for catching the Guix
features... I digress. :-)

Well, my point is, for example, the logic of "guix system" then "guix
system <action> <--options>" appears to me easier to grasp and
discover.  Idem with "guix import".  I think "guix vm" or "guix image"
would not ease the discoverability.  What is unfortunate with "guix
package" is that <action> and <--option> are both using dash-dash.
Another story. :-)

I am not convinced that promoting a niche use-case would be a good
idea.  For what my opinion is worth.  Well, I trust your experience if
you think it is better.

> Thinking about it, another option would be to add an ‘--export-manifest’
> option to ‘guix shell’ instead.

Thinking about it, it would be the most coherent from my point of
view.  And idem for --export-channels, no?


Cheers,
simon

[bug#54393] [PATCH 0/2] Add 'guix manifest' to "translate" commands to manifests

[Ludovic Courtès]

Hi,

zimoun <zimon.toutoune@gmail.com> skribis:

> Well, my point is, for example, the logic of "guix system" then "guix
> system <action> <--options>" appears to me easier to grasp and
> discover.  Idem with "guix import".  I think "guix vm" or "guix image"
> would not ease the discoverability.  What is unfortunate with "guix
> package" is that <action> and <--option> are both using dash-dash.
> Another story. :-)

I agree, ‘guix system’ and ‘guix import’ are pretty logical, consistent,
and discoverable; ‘guix package’ less so…

> I am not convinced that promoting a niche use-case would be a good
> idea.  For what my opinion is worth.  Well, I trust your experience if
> you think it is better.

There have been proposals in the past to follow a similar three-level
command scheme, such as ‘guix profile install’, etc.  While this would
be more “logical” in terms of categorization, it’s also less intuitive,
requires more typing, and overall may well be less ergonomic than
shorthands like ‘guix install’ or ‘guix edit’.

For ‘guix install’, we were guided by the rule of least surprise.  But
it’s true that in the case of ‘guix manifest’, it’s less obvious because
other package managers for instance don’t have such a command.

>> Thinking about it, another option would be to add an ‘--export-manifest’
>> option to ‘guix shell’ instead.
>
> Thinking about it, it would be the most coherent from my point of
> view.  And idem for --export-channels, no?

For ‘guix shell’ that would be strictly equivalent to ‘guix describe’.
I’m not convinced having ‘--export-channels’ would bring anything.

Thanks for your feedback!

Ludo’.

[bug#54393] [PATCH v2 1/3] packages: Add 'package-unique-version-prefix'.

[Ludovic Courtès]

* gnu/packages.scm (package-unique-version-prefix): New procedure.
* guix/scripts/package.scm (manifest-entry-version-prefix): Use it.
* tests/packages.scm ("package-unique-version-prefix, gcc@8")
("package-unique-version-prefix, grep"): New tests.
---
 gnu/packages.scm         | 21 +++++++++++++++++++++
 guix/scripts/package.scm | 20 ++------------------
 tests/packages.scm       | 13 +++++++++++++
 3 files changed, 36 insertions(+), 18 deletions(-)

diff --git a/gnu/packages.scm b/gnu/packages.scm
index 65ab7a7c1e..2ba838fd0a 100644
--- a/gnu/packages.scm
+++ b/gnu/packages.scm
@@ -66,6 +66,8 @@ (define-module (gnu packages)
             specification->location
             specifications->manifest
 
+            package-unique-version-prefix
+
             generate-package-cache))
 
 ;;; Commentary:
@@ -559,3 +561,22 @@ (define (specifications->manifest specs)
   ;; fiddle with multiple-value returns.
   (packages->manifest
    (map (compose list specification->package+output) specs)))
+
+(define (package-unique-version-prefix name version)
+  "Search among all the versions of package NAME that are available, and
+return the shortest unambiguous version prefix to designate VERSION.  If only
+one version of the package is available, return the empty string."
+  (match (map package-version (find-packages-by-name name))
+    ((_)
+     ;; A single version of NAME is available, so do not specify the version
+     ;; number, even if the available version doesn't match VERSION.
+     "")
+    (versions
+     ;; If VERSION is the latest version, don't specify any version.
+     ;; Otherwise return the shortest unique version prefix.  Note that this
+     ;; is based on the currently available packages so the result may vary
+     ;; over time.
+     (if (every (cut version>? version <>)
+                (delete version versions))
+         ""
+         (version-unique-prefix version versions)))))
diff --git a/guix/scripts/package.scm b/guix/scripts/package.scm
index 9699c70c6d..22ee8a2485 100644
--- a/guix/scripts/package.scm
+++ b/guix/scripts/package.scm
@@ -334,24 +334,8 @@ (define (manifest-entry-version-prefix entry)
   "Search among all the versions of ENTRY's package that are available, and
 return the shortest unambiguous version prefix for this package.  If only one
 version of ENTRY's package is available, return the empty string."
-  (let ((name (manifest-entry-name entry)))
-    (match (map package-version (find-packages-by-name name))
-      ((_)
-       ;; A single version of NAME is available, so do not specify the
-       ;; version number, even if the available version doesn't match ENTRY.
-       "")
-      (versions
-       ;; If ENTRY uses the latest version, don't specify any version.
-       ;; Otherwise return the shortest unique version prefix.  Note that
-       ;; this is based on the currently available packages, which could
-       ;; differ from the packages available in the revision that was used
-       ;; to build MANIFEST.
-       (let ((current (manifest-entry-version entry)))
-         (if (every (cut version>? current <>)
-                    (delete current versions))
-             ""
-             (version-unique-prefix (manifest-entry-version entry)
-                                    versions)))))))
+  (package-unique-version-prefix (manifest-entry-name entry)
+                                 (manifest-entry-version entry)))
 
 (define* (export-manifest manifest
                           #:optional (port (current-output-port)))
diff --git a/tests/packages.scm b/tests/packages.scm
index 710eace6dc..6cbc34ba0b 100644
--- a/tests/packages.scm
+++ b/tests/packages.scm
@@ -1923,6 +1923,19 @@ (define (list->set* lst)
   (package-location (specification->package "guile@2"))
   (specification->location "guile@2"))
 
+(test-equal "package-unique-version-prefix, gcc@8"
+  "8"
+  (let ((gcc (specification->package "gcc-toolchain@8")))
+    (package-unique-version-prefix (package-name gcc)
+                                   (package-version gcc))))
+
+(test-equal "package-unique-version-prefix, grep"
+  ""
+  (let ((grep (specification->package "grep")))
+    (package-unique-version-prefix (package-name grep)
+                                   (package-version grep))))
+
+
 (test-eq "this-package-input, exists"
   hello
   (package-arguments
-- 
2.34.0

[bug#54393] [PATCH v2 2/3] environment: Export 'load-manifest'.

[Ludovic Courtès]

* guix/scripts/environment.scm (load-manifest): New procedure.
(options/resolve-packages): Use it.
---
 guix/scripts/environment.scm | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)

diff --git a/guix/scripts/environment.scm b/guix/scripts/environment.scm
index ec071402f4..07b54cd89b 100644
--- a/guix/scripts/environment.scm
+++ b/guix/scripts/environment.scm
@@ -56,6 +56,7 @@ (define-module (guix scripts environment)
   #:use-module (srfi srfi-37)
   #:use-module (srfi srfi-98)
   #:export (assert-container-features
+            load-manifest
             guix-environment
             guix-environment*
             show-environment-options-help
@@ -285,6 +286,11 @@ (define same-key? (cut eq? key <>))
             (_ memo)))
         '() alist))
 
+(define (load-manifest file)                      ;TODO: factorize
+  "Load the user-profile manifest (Scheme code) from FILE and return it."
+  (let ((user-module (make-user-module '((guix profiles) (gnu)))))
+    (load* file user-module)))
+
 (define (options/resolve-packages store opts)
   "Return OPTS with package specification strings replaced by manifest entries
 for the corresponding packages."
@@ -331,8 +337,7 @@ (define (packages->outputs packages mode)
                    (let ((module (make-user-module '())))
                      (packages->outputs (load* file module) mode)))
                   (('manifest . file)
-                   (let ((module (make-user-module '((guix profiles) (gnu)))))
-                     (manifest-entries (load* file module))))
+                   (manifest-entries (load-manifest file)))
                   (_ '()))
                 opts)
     manifest-entry=?)))
-- 
2.34.0

[bug#54393] [PATCH v2 3/3] shell: Add '--export-manifest'.

[Ludovic Courtès]

* guix/scripts/shell.scm (show-help, %options): Add '--export-manifest'.
(manifest-entry-version-prefix, manifest->code*)
(export-manifest): New procedures.
(guix-shell): Honor '--export-manifest'.
* tests/guix-shell-export-manifest.sh: New file.
* Makefile.am (SH_TESTS): Add it.
* doc/guix.texi (Invoking guix shell): Document '--export-manifest'.
(Invoking guix environment): Link to it.
(Invoking guix pack): Likewise.
---
 Makefile.am                         |   1 +
 doc/guix.texi                       |  57 +++++++++++++++
 guix/scripts/shell.scm              | 109 +++++++++++++++++++++++++++-
 tests/guix-shell-export-manifest.sh |  84 +++++++++++++++++++++
 4 files changed, 248 insertions(+), 3 deletions(-)
 create mode 100644 tests/guix-shell-export-manifest.sh

diff --git a/Makefile.am b/Makefile.am
index 91243f2cad..20ab7dd5f0 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -572,6 +572,7 @@ SH_TESTS =					\
   tests/guix-environment.sh			\
   tests/guix-environment-container.sh		\
   tests/guix-shell.sh				\
+  tests/guix-shell-export-manifest.sh		\
   tests/guix-graph.sh				\
   tests/guix-describe.sh			\
   tests/guix-repl.sh     			\
diff --git a/doc/guix.texi b/doc/guix.texi
index e8ef4286be..85a91019be 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -5848,6 +5848,55 @@ This is similar to the same-named option in @command{guix package}
 (@pxref{profile-manifest, @option{--manifest}}) and uses the same
 manifest files.
 
+See @option{--export-manifest} below on how to obtain a first manifest.
+
+@cindex manifest, exporting
+@anchor{shell-export-manifest}
+@item --export-manifest
+Write to standard output a manifest suitable for @option{--manifest}
+corresponding to given command-line options.
+
+This is a way to ``convert'' command-line arguments into a manifest.
+For example, imagine you are tired of typing long lines and would like
+to get a manifest equivalent to this command line:
+
+@example
+guix shell -D guile git emacs emacs-geiser emacs-geiser-guile
+@end example
+
+Just add @option{--export-manifest} to the command line above:
+
+@example
+guix shell --export-manifest \
+  -D guile git emacs emacs-geiser emacs-geiser-guile
+@end example
+
+@noindent
+... and you get a manifest along these lines:
+
+@lisp
+(concatenate-manifests
+  (list (specifications->manifest
+          (list "git"
+                "emacs"
+                "emacs-geiser"
+                "emacs-geiser-guile"))
+        (package->development-manifest
+          (specification->package "guile"))))
+@end lisp
+
+You can store it into a file, say @file{manifest.scm}, and from there
+pass it to @command{guix shell} or indeed pretty much any @command{guix}
+command:
+
+@example
+guix shell -m manifest.scm
+@end example
+
+Voilà, you've converted a long command line into a manifest!  That
+conversion process honors package transformation options (@pxref{Package
+Transformation Options}) so it should be lossless.
+
 @item --profile=@var{profile}
 @itemx -p @var{profile}
 Create an environment containing the packages installed in @var{profile}. 
@@ -6235,6 +6284,10 @@ This is similar to the same-named option in @command{guix package}
 (@pxref{profile-manifest, @option{--manifest}}) and uses the same
 manifest files.
 
+@xref{shell-export-manifest, @command{guix shell --export-manifest}},
+for information on how to ``convert'' command-line options into a
+manifest.
+
 @item --ad-hoc
 Include all specified packages in the resulting environment, as if an
 @i{ad hoc} package were defined with them as inputs.  This option is
@@ -6693,6 +6746,10 @@ for use on machines that do not have Guix installed.  Note that you can
 specify @emph{either} a manifest file @emph{or} a list of packages,
 but not both.
 
+@xref{shell-export-manifest, @command{guix shell --export-manifest}},
+for information on how to ``convert'' command-line options into a
+manifest.
+
 @item --system=@var{system}
 @itemx -s @var{system}
 Attempt to build for @var{system}---e.g., @code{i686-linux}---instead of
diff --git a/guix/scripts/shell.scm b/guix/scripts/shell.scm
index 1eab05d737..5e7f454f4c 100644
--- a/guix/scripts/shell.scm
+++ b/guix/scripts/shell.scm
@@ -21,7 +21,8 @@ (define-module (guix scripts shell)
   #:use-module ((guix diagnostics) #:select (location))
   #:use-module (guix scripts environment)
   #:autoload   (guix scripts build) (show-build-options-help)
-  #:autoload   (guix transformations) (transformation-option-key?
+  #:autoload   (guix transformations) (options->transformation
+                                       transformation-option-key?
                                        show-transformation-options-help)
   #:use-module (guix scripts)
   #:use-module (guix packages)
@@ -41,7 +42,12 @@ (define-module (guix scripts shell)
   #:use-module ((guix build utils) #:select (mkdir-p))
   #:use-module (guix cache)
   #:use-module ((ice-9 ftw) #:select (scandir))
-  #:autoload   (gnu packages) (cache-is-authoritative?)
+  #:autoload   (ice-9 pretty-print) (pretty-print)
+  #:autoload   (gnu packages) (cache-is-authoritative?
+                               package-unique-version-prefix
+                               specification->package
+                               specification->package+output
+                               specifications->manifest)
   #:export (guix-shell))
 
 (define (show-help)
@@ -55,10 +61,13 @@ (define (show-help)
   -D, --development      include the development inputs of the next package"))
   (display (G_ "
   -f, --file=FILE        add to the environment the package FILE evaluates to"))
+
   (display (G_ "
   -q                     inhibit loading of 'guix.scm' and 'manifest.scm'"))
   (display (G_ "
       --rebuild-cache    rebuild cached environment, if any"))
+  (display (G_ "
+      --export-manifest  print a manifest for the given options"))
 
   (show-environment-options-help)
   (newline)
@@ -112,6 +121,10 @@ (define %options
                         ;; 'wrapped-option'.
                         (alist-delete 'ad-hoc? result)))
 
+              (option '("export-manifest") #f #f
+                      (lambda (opt name arg result)
+                        (alist-cons 'export-manifest? #t result)))
+
               ;; For consistency with 'guix package', support '-f' rather than
               ;; '-l' like 'guix environment' does.
               (option '(#\f "file") #t #f
@@ -380,6 +393,94 @@ (define (key->file key)
        (loop rest system file specs))
       ((_ . rest) (loop rest system file specs)))))
 
+\f
+;;;
+;;; Exporting a manifest.
+;;;
+
+(define (manifest-entry-version-prefix entry)
+  "Search among all the versions of ENTRY's package that are available, and
+return the shortest unambiguous version prefix for this package."
+  (package-unique-version-prefix (manifest-entry-name entry)
+                                 (manifest-entry-version entry)))
+
+(define (manifest->code* manifest extra-manifests)
+  "Like 'manifest->code', but insert a 'concatenate-manifests' call that
+concatenates MANIFESTS, a list of expressions."
+  (if (null? (manifest-entries manifest))
+      (match extra-manifests
+        ((one) one)
+        (lst   `(concatenate-manifests ,@extra-manifests)))
+      (match (manifest->code manifest
+                             #:entry-package-version
+                             manifest-entry-version-prefix)
+        (('begin exp ... last)
+         `(begin
+            ,@exp
+            ,(match extra-manifests
+               (() last)
+               (_  `(concatenate-manifests
+                     (list ,last ,@extra-manifests)))))))))
+
+(define (export-manifest opts port)
+  "Write to PORT a manifest corresponding to OPTS."
+  (define (manifest-lift proc)
+    (lambda (entry)
+      (match (manifest-entry-item entry)
+        ((? package? p)
+         (manifest-entry
+           (inherit (package->manifest-entry (proc p)))
+           (output (manifest-entry-output entry))))
+        (_
+         entry))))
+
+  (define (validated-spec spec)
+    ;; Return SPEC if it's validate package spec.
+    (specification->package+output spec)
+    spec)
+
+  (let* ((transform (options->transformation opts))
+         (specs     (reverse
+                     (filter-map (match-lambda
+                                   (('package 'ad-hoc-package spec)
+                                    (validated-spec spec))
+                                   (_ #f))
+                                 opts)))
+         (extras    (reverse
+                     (filter-map (match-lambda
+                                   (('package 'package spec)
+                                    ;; Make sure SPEC is valid.
+                                    (specification->package spec)
+
+                                    ;; XXX: This is an approximation:
+                                    ;; transformation options are not applied.
+                                    `(package->development-manifest
+                                      (specification->package ,spec)))
+                                   (_ #f))
+                                 opts)))
+         (manifest  (concatenate-manifests
+                     (cons (map-manifest-entries
+                            (manifest-lift transform)
+                            (specifications->manifest specs))
+                           (filter-map (match-lambda
+                                         (('manifest . file)
+                                          (load-manifest file))
+                                         (_ #f))
+                                       opts)))))
+    (display (G_ "\
+;; What follows is a \"manifest\" equivalent to the command line you gave.
+;; You can store it in a file that you may then pass to any 'guix' command
+;; that accepts a '--manifest' (or '-m') option.\n")
+             port)
+    (match (manifest->code* manifest extras)
+      (('begin exp ...)
+       (for-each (lambda (exp)
+                   (newline port)
+                   (pretty-print exp port))
+                 exp))
+      (exp
+       (pretty-print exp port)))))
+
 \f
 ;;;
 ;;; One-time hints.
@@ -445,4 +546,6 @@ (define interactive?
                 cache-entries
                 #:entry-expiration entry-expiration)))
 
-  (guix-environment* opts))
+  (if (assoc-ref opts 'export-manifest?)
+      (export-manifest opts (current-output-port))
+      (guix-environment* opts)))
diff --git a/tests/guix-shell-export-manifest.sh b/tests/guix-shell-export-manifest.sh
new file mode 100644
index 0000000000..cbb90f04bf
--- /dev/null
+++ b/tests/guix-shell-export-manifest.sh
@@ -0,0 +1,84 @@
+# GNU Guix --- Functional package management for GNU
+# Copyright © 2022 Ludovic Courtès <ludo@gnu.org>
+#
+# This file is part of GNU Guix.
+#
+# GNU Guix is free software; you can redistribute it and/or modify it
+# under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or (at
+# your option) any later version.
+#
+# GNU Guix is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
+
+#
+# Test 'guix shell --export-manifest'.
+#
+
+guix shell --version
+
+tmpdir="t-guix-manifest-$$"
+trap 'rm -r "$tmpdir"' EXIT
+mkdir "$tmpdir"
+
+manifest="$tmpdir/manifest.scm"
+
+# Basics.
+guix shell --export-manifest guile-bootstrap > "$manifest"
+test "$(guix build -m "$manifest")" = "$(guix build guile-bootstrap)"
+
+guix shell -m "$manifest" --bootstrap -- \
+     "$SHELL" -c 'guix package --export-manifest -p "$GUIX_ENVIRONMENT"' > \
+     "$manifest.second"
+for m in "$manifest" "$manifest.second"
+do
+    grep -v '^;' < "$m" > "$m.new" # filter out comments
+    mv "$m.new" "$m"
+done
+
+cat "$manifest"
+cat "$manifest.second"
+
+cmp "$manifest" "$manifest.second"
+
+# Combining manifests.
+guix shell --export-manifest -m "$manifest" gash gash-utils \
+     > "$manifest.second"
+guix build -m "$manifest.second" -d | \
+    grep "$(guix build guile-bootstrap -d)"
+guix build -m "$manifest.second" -d | \
+    grep "$(guix build gash -d)"
+
+# Package transformation option.
+guix shell --export-manifest guile guix --with-latest=guile-json > "$manifest"
+grep 'options->transformation' "$manifest"
+grep '(with-latest . "guile-json")' "$manifest"
+
+# Development manifest.
+guix shell --export-manifest -D guile git > "$manifest"
+grep 'package->development-manifest' "$manifest"
+grep '"guile"' "$manifest"
+guix build -m "$manifest" -d | \
+    grep "$(guix build -e '(@@ (gnu packages commencement) gcc-final)' -d)"
+guix build -m "$manifest" -d | \
+    grep "$(guix build git -d)"
+
+# Test various combinations to make sure generated code uses interfaces
+# correctly.
+for options in					\
+    "coreutils grep sed"			\
+    "gsl openblas gcc-toolchain --tune"		\
+    "guile -m $manifest.previous"		\
+    "git:send-email gdb guile:debug"		\
+    "git -D coreutils"
+do
+    guix shell --export-manifest $options > "$manifest"
+    cat "$manifest"
+    guix shell -m "$manifest" -n
+    mv "$manifest" "$manifest.previous"
+done
-- 
2.34.0

[bug#54393] [PATCH 0/2] Add 'guix manifest' to "translate" commands to manifests

[Maxim Cournoyer]

Hello,

Ludovic Courtès <ludo@gnu.org> writes:

> * guix/scripts/shell.scm (show-help, %options): Add '--export-manifest'.
> (manifest-entry-version-prefix, manifest->code*)
> (export-manifest): New procedures.
> (guix-shell): Honor '--export-manifest'.
> * tests/guix-shell-export-manifest.sh: New file.
> * Makefile.am (SH_TESTS): Add it.
> * doc/guix.texi (Invoking guix shell): Document '--export-manifest'.
> (Invoking guix environment): Link to it.
> (Invoking guix pack): Likewise.

[...]

> +\f
> +;;;
> +;;; Exporting a manifest.
> +;;;
> +
> +(define (manifest-entry-version-prefix entry)
> +  "Search among all the versions of ENTRY's package that are available, and
> +return the shortest unambiguous version prefix for this package."
> +  (package-unique-version-prefix (manifest-entry-name entry)
> +                                 (manifest-entry-version entry)))
> +
> +(define (manifest->code* manifest extra-manifests)
> +  "Like 'manifest->code', but insert a 'concatenate-manifests' call that
> +concatenates MANIFESTS, a list of expressions."
> +  (if (null? (manifest-entries manifest))
> +      (match extra-manifests
> +        ((one) one)
> +        (lst   `(concatenate-manifests ,@extra-manifests)))
> +      (match (manifest->code manifest
> +                             #:entry-package-version
> +                             manifest-entry-version-prefix)
> +        (('begin exp ... last)
> +         `(begin
> +            ,@exp
> +            ,(match extra-manifests
> +               (() last)
> +               (_  `(concatenate-manifests
> +                     (list ,last ,@extra-manifests)))))))))

Should an "else" clause be added here with a more useful error message
that the default 'no match for x' or similar?  If that'd be totally
unexpected and a bug, then it's fine as-is.

> +(define (export-manifest opts port)
> +  "Write to PORT a manifest corresponding to OPTS."
> +  (define (manifest-lift proc)
> +    (lambda (entry)
> +      (match (manifest-entry-item entry)
> +        ((? package? p)
> +         (manifest-entry
> +           (inherit (package->manifest-entry (proc p)))
> +           (output (manifest-entry-output entry))))
> +        (_
> +         entry))))
> +
> +  (define (validated-spec spec)
> +    ;; Return SPEC if it's validate package spec.

As this is an action (proc), perhaps it should be named "validate-spec".
The comment doc should also be worded as "if SPEC is a valid package
spec" or similar.

The rest LGTM.

Thank you for addressing the suggestion to reuse an existing sub-command
to try to keep things neatly organized instead of extending the already
large set of them :-).

Maxim

bug#54393: [PATCH 0/2] Add 'guix manifest' to "translate" commands to manifests

[Ludovic Courtès]

Hi!

Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:


[...]

>> +(define (manifest->code* manifest extra-manifests)
>> +  "Like 'manifest->code', but insert a 'concatenate-manifests' call that
>> +concatenates MANIFESTS, a list of expressions."
>> +  (if (null? (manifest-entries manifest))
>> +      (match extra-manifests
>> +        ((one) one)
>> +        (lst   `(concatenate-manifests ,@extra-manifests)))
>> +      (match (manifest->code manifest
>> +                             #:entry-package-version
>> +                             manifest-entry-version-prefix)
>> +        (('begin exp ... last)
>> +         `(begin
>> +            ,@exp
>> +            ,(match extra-manifests
>> +               (() last)
>> +               (_  `(concatenate-manifests
>> +                     (list ,last ,@extra-manifests)))))))))
>
> Should an "else" clause be added here with a more useful error message
> that the default 'no match for x' or similar?  If that'd be totally
> unexpected and a bug, then it's fine as-is.

Yes, it would be a bug.

>> +  (define (validated-spec spec)
>> +    ;; Return SPEC if it's validate package spec.
>
> As this is an action (proc), perhaps it should be named "validate-spec".
> The comment doc should also be worded as "if SPEC is a valid package
> spec" or similar.

I fixed the comment but kept the name ‘validated-spec’, which is an
attempt at suggesting that it’s returns a meaningful value (whereas
‘validate-spec’ sounds like it’s called for effect and returns
*unspecified*).

> Thank you for addressing the suggestion to reuse an existing sub-command
> to try to keep things neatly organized instead of extending the already
> large set of them :-).

Heheh, thanks!

Pushed as b1e7e64f351fa03a66ce1f9776f9ba84cf2c6294 together with a news
entry.

Ludo’.

[bug#54393] [PATCH 0/2] Add 'guix manifest' to "translate" commands to manifests

[zimoun]

Hi,

On Mon, 4 Apr 2022 at 23:17, Ludovic Courtès <ludo@gnu.org> wrote:

> > Thank you for addressing the suggestion to reuse an existing sub-command
> > to try to keep things neatly organized instead of extending the already
> > large set of them :-).

As discussed on IRC [1], it could nice in the future to have a warning
message when some options are incompatible.  For instance,

    guix package --install hello --export-channels
    guix shell --container --export-manifest

etc.  And it would probably be an easy first contribution, or listed
as such. :-)


1: <https://logs.guix.gnu.org/guix-hpc/2022-04-04.log#140437>

> Pushed as b1e7e64f351fa03a66ce1f9776f9ba84cf2c6294 together with a news
> entry.

Cool!  Thanks.


Cheers,
simon

[bug#54393] [PATCH 0/2] Add 'guix manifest' to "translate" commands to manifests

[Ludovic Courtès]

Hi,

zimoun <zimon.toutoune@gmail.com> skribis:

> As discussed on IRC [1], it could nice in the future to have a warning
> message when some options are incompatible.  For instance,
>
>     guix package --install hello --export-channels
>     guix shell --container --export-manifest
>
> etc.  And it would probably be an easy first contribution, or listed
> as such. :-)

Agreed!

Ludo’.

[bug#54393] [PATCH v2 0/3] Add '--export-manifest' to 'guix shell'

[Ludovic Courtès]

Hello!

Here’s a v2 of the patch: instead of introducing a new command,
this version adds an ‘--export-manifest’ option to ‘guix shell’,
as suggested by zimoun and others.  The behavior is the same as
that of the previously-proposed ‘guix manifest’ command.

Thoughts?

Ludo’.

Ludovic Courtès (3):
  packages: Add 'package-unique-version-prefix'.
  environment: Export 'load-manifest'.
  shell: Add '--export-manifest'.

 Makefile.am                         |   1 +
 doc/guix.texi                       |  57 +++++++++++++++
 gnu/packages.scm                    |  21 ++++++
 guix/scripts/environment.scm        |   9 ++-
 guix/scripts/package.scm            |  20 +----
 guix/scripts/shell.scm              | 109 +++++++++++++++++++++++++++-
 tests/guix-shell-export-manifest.sh |  84 +++++++++++++++++++++
 tests/packages.scm                  |  13 ++++
 8 files changed, 291 insertions(+), 23 deletions(-)
 create mode 100644 tests/guix-shell-export-manifest.sh


base-commit: 53b04339fe521f486d3017930a419d5ca8a6cffd
-- 
2.34.0

[bug#54393] [PATCH 0/2] Add 'guix manifest' to "translate" commands to manifests

[Greg Hogan]

[-- Attachment #1: Type: text/plain, Size: 1473 bytes --]

On Mon, Mar 14, 2022 at 5:53 PM Ludovic Courtès <ludo@gnu.org> wrote:

> Next up: adding a “Writing Manifests” section.
>

Ludo',

I would find this quite helpful. The current "basic setup with manifests"
only documents the trivial case. It would be quite nice to document best
practices for custom UTF-8 profiles, system-dependent inclusion (should we
silently skip packages on unsupported systems?), and updating package
versions, inputs, and/or toolchain. And for the latter, what are the limits
of the manifest as compared with cloning, modifying, building, and
executing with "./pre-inst-env guix ..." (which loses reproducibility)?

Some background on that last thought: as Guix grows it becomes
increasingly difficult to upgrade popular dependencies. I looked at
upgrading 'fmt' and there was one dependent package that could not be
upgraded, though not a package I needed. I'd like to be able to define an
upgraded version of 'fmt' or 'boost' in my manifest and have it apply to
the entire installation (profile or pack) rather than needing to modify
specific packages with a with-input. Same for the toolchain, I think it
would be nice to simply say "create this installation using gcc-toolchain"
(the user default @11 rather than the system default @10) without needing
any package transformations. And could one specify global build flags such
as "-march=native"? Can one reach down into and modify commencement
packages?

Greg

[-- Attachment #2: Type: text/html, Size: 1866 bytes --]

[bug#54393] [PATCH 0/2] Add 'guix manifest' to "translate" commands to manifests

[Ludovic Courtès]

Hi,

Greg Hogan <code@greghogan.com> skribis:

> I would find this quite helpful. The current "basic setup with manifests"
> only documents the trivial case. It would be quite nice to document best
> practices for custom UTF-8 profiles, system-dependent inclusion (should we
> silently skip packages on unsupported systems?), and updating package
> versions, inputs, and/or toolchain.

OK.

> And for the latter, what are the limits of the manifest as compared
> with cloning, modifying, building, and executing with "./pre-inst-env
> guix ..." (which loses reproducibility)?
>
> Some background on that last thought: as Guix grows it becomes
> increasingly difficult to upgrade popular dependencies. I looked at
> upgrading 'fmt' and there was one dependent package that could not be
> upgraded, though not a package I needed. I'd like to be able to define an
> upgraded version of 'fmt' or 'boost' in my manifest and have it apply to
> the entire installation (profile or pack) rather than needing to modify
> specific packages with a with-input. Same for the toolchain, I think it
> would be nice to simply say "create this installation using gcc-toolchain"
> (the user default @11 rather than the system default @10) without needing
> any package transformations. And could one specify global build flags such
> as "-march=native"? Can one reach down into and modify commencement
> packages?

This—switching compiler tool chains and tuning for a CPU—is available as
package transformations:

  https://guix.gnu.org/manual/devel/en/html_node/Package-Transformation-Options.html

You can take advantage of those in your manifest using
‘option->transformations’:

  https://guix.gnu.org/manual/devel/en/html_node/Defining-Package-Variants.html#index-options_002d_003etransformation

The tentative ‘guix manifest’ and ‘guix package --export-manifest’
create that code snippet for you, as in this example:

  https://issues.guix.gnu.org/54393#2-lineno134

That’s the kind of thing I’d like to show in the “Writing Manifests”
section.

Tuning and switching tool chains will always happen through package
transformations.  In Guix, things have to be specified explicitly;
there’s no way you can stealthily change the meaning of the dependency
graph without changing it—unlike other tools in this area.  But that’s a
strength!  Now, we have to make it so that such changes can be done
conveniently, and package transformation options were introduced in this
spirit.

I hope that makes sense!

Ludo’.