[bug#50960] [PATCH 00/10] Add 'guix shell' to subsume 'guix environment'

[bug#50960] [PATCH 00/10] Add 'guix shell' to subsume 'guix environment'

[Ludovic Courtès]

Hello Guix!

Here comes ‘guix shell’, a proposed replacement for ‘guix environment’!
‘guix environment’ would stay around though, at least for some time,
probably for a long time.

The differences to ‘guix environment’ are:

  1. ‘--ad-hoc’ is the default.

       ‘guix shell hello’ ≍ ‘guix environment --ad-hoc hello’
       ‘guix shell -D hello git’ ≍ ‘guix environment hello --ad-hoc git’

  2. ‘guix shell’, without arguments, loads ‘guix.scm’ or ‘manifest.scm’
     from the current directory or one of its ancestors.

  3. ‘--load’/‘-l’ is not ‘-f’/‘--install-from-file’ for consistency with
     ‘guix package’.

  4. ‘guix shell’ without arguments maintains a cache, such that, the
     second time you run it, it runs in ~0.1s (it does not even need to
     connect to the daemon).

     If you run ‘guix pull’ and run again ‘guix shell’, it recomputes
     the environment, as is currently the case with ‘guix environment’.

Here’s a summary of previous proposals:

  - Dave Thompson: https://lists.gnu.org/archive/html/guix-devel/2017-08/msg00300.html
    - [X] --ad-hoc is the default
    - [X] caching
    - [X] behavior with no arguments
    - [ ] --load accepts <environment>
    - [ ] Shepherd services
    - [ ] 'guix environment --update' to explicitly update
  - make --ad-hoc the default: https://issues.guix.gnu.org/38529
    - [X] https://issues.guix.gnu.org/38529#17: proposal for a new subcommand
          deprecation of ‘guix environment’

I think <environment> records and Shepherd services could come later.
As for ‘--update’, I prefer the behavior implemented here because it’s
stateless and thus more predictable.

Thoughts?  Are there other changes people would like to see?

If there’s rough consensus I can work on v2 with documentation.  Please
let’s keep the discussion focused.  :-)

As for deprecation, I think there’s no rush.  I imagine there could be
several phases, like: initially we only mention deprecation in the manual,
later on ‘guix environment’ starts emitting a warning, and later (I guess
at least two years later, probably more) we ask ourselves whether to
remove ‘guix environment’.  At this point keeping it doesn’t cost us much.

Thanks,
Ludo’.

Ludovic Courtès (10):
  packages: Add 'package-development-inputs'.
  profiles: Add 'package->development-manifest'.
  DRAFT Add 'guix shell'.
  DRAFT shell: By default load the local 'guix.scm' or 'manifest.scm'
    file.
  environment: Add tests for '--profile'.
  environment: Skip derivation computation when '--profile' is used.
  environment: Do not connect to the daemon when '--profile' is used.
  environment: Autoload some modules.
  cache: Gracefully handle non-existent cache.
  shell: Maintain a profile cache.

 Makefile.am                         |   2 +
 doc/guix.texi                       |  52 ++++++
 guix/cache.scm                      |  10 +-
 guix/packages.scm                   |  10 ++
 guix/profiles.scm                   |  19 ++
 guix/scripts/environment.scm        | 260 +++++++++++++++-------------
 guix/scripts/shell.scm              | 254 +++++++++++++++++++++++++++
 po/guix/POTFILES.in                 |   1 +
 tests/guix-environment-container.sh |   8 +
 tests/guix-environment.sh           |   7 +
 tests/guix-shell.sh                 |  70 ++++++++
 tests/packages.scm                  |  14 ++
 tests/profiles.scm                  |   7 +
 13 files changed, 594 insertions(+), 120 deletions(-)
 create mode 100644 guix/scripts/shell.scm
 create mode 100644 tests/guix-shell.sh

-- 
2.33.0

[bug#50960] [PATCH 01/10] packages: Add 'package-development-inputs'.

[Ludovic Courtès]

* guix/packages.scm (package-development-inputs): New procedure.
* guix/scripts/environment.scm (package-environment-inputs): Use it.
* tests/packages.scm ("package-development-inputs")
("package-development-inputs, cross-compilation"): New tests.
* doc/guix.texi (package Reference): Document it.
---
 doc/guix.texi                | 41 ++++++++++++++++++++++++++++++++++++
 guix/packages.scm            | 10 +++++++++
 guix/scripts/environment.scm |  2 +-
 tests/packages.scm           | 14 ++++++++++++
 4 files changed, 66 insertions(+), 1 deletion(-)

diff --git a/doc/guix.texi b/doc/guix.texi
index a72a726b54..49399e792b 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -6847,6 +6847,47 @@ cross-compiling:
 It is an error to refer to @code{this-package} outside a package definition.
 @end deffn
 
+@cindex development inputs, of a package
+@cindex implicit inputs, of a package
+Sometimes you will want to obtain the list of inputs needed to
+@emph{develop} a package---all the inputs that are visible when the
+package is compiled.  This is what the @code{package-development-inputs}
+procedure returns.
+
+@deffn {Scheme Procedure} package-development-inputs @var{package} @
+   [@var{system}] [#:target #f]
+Return the list of inputs required by @var{package} for development
+purposes on @var{system}.  When @var{target} is true, return the inputs
+needed to cross-compile @var{package} from @var{system} to
+@var{triplet}, where @var{triplet} is a triplet such as
+@code{"aarch64-linux-gnu"}.
+
+Note that the result includes both explicit inputs and implicit
+inputs---inputs automatically added by the build system (@pxref{Build
+Systems}).  Let us take the @code{hello} package to illustrate that:
+
+@lisp
+(use-modules (gnu packages base) (guix packages))
+
+hello
+@result{} #<package hello@@2.10 gnu/packages/base.scm:79 7f585d4f6790>
+
+(package-direct-inputs hello)
+@result{} ()
+
+(package-development-inputs hello)
+@result{} (("source" @dots{}) ("tar" #<package tar@@1.32 @dots{}>) @dots{})
+@end lisp
+
+In this example, @code{package-direct-inputs} returns the empty list,
+because @code{hello} has zero explicit dependencies.  Conversely,
+@code{package-development-inputs} includes inputs implicitly added by
+@code{gnu-build-system} that are required to build @code{hello}: tar,
+gzip, GCC, libc, Bash, and more.  To visualize it, @command{guix graph
+hello} would show you explicit inputs, whereas @command{guix graph -t
+bag hello} would include implicit inputs (@pxref{Invoking guix graph}).
+@end deffn
+
 Because packages are regular Scheme objects that capture a complete
 dependency graph and associated build procedures, it is often useful to
 write procedures that take a package and return a modified version
diff --git a/guix/packages.scm b/guix/packages.scm
index 8c3a0b0b7b..43e0130793 100644
--- a/guix/packages.scm
+++ b/guix/packages.scm
@@ -153,6 +153,7 @@
             bag-transitive-host-inputs
             bag-transitive-build-inputs
             bag-transitive-target-inputs
+            package-development-inputs
             package-closure
 
             default-guile
@@ -1070,6 +1071,15 @@ dependencies are known to build on SYSTEM."
                  (%current-system (bag-system bag)))
     (transitive-inputs (bag-target-inputs bag))))
 
+(define* (package-development-inputs package
+                                     #:optional (system (%current-system))
+                                     #:key target)
+  "Return the list of inputs required by PACKAGE for development purposes on
+SYSTEM.  When TARGET is true, return the inputs needed to cross-compile
+PACKAGE from SYSTEM to TRIPLET, where TRIPLET is a triplet such as
+\"aarch64-linux-gnu\"."
+  (bag-transitive-inputs (package->bag package system target)))
+
 (define* (package-closure packages #:key (system (%current-system)))
   "Return the closure of PACKAGES on SYSTEM--i.e., PACKAGES and the list of
 packages they depend on, recursively."
diff --git a/guix/scripts/environment.scm b/guix/scripts/environment.scm
index 6958bd6238..d555969b27 100644
--- a/guix/scripts/environment.scm
+++ b/guix/scripts/environment.scm
@@ -82,7 +82,7 @@ package."
 packages for PACKAGE."
   ;; Remove non-package inputs such as origin records.
   (filter-map input->manifest-entry
-              (bag-transitive-inputs (package->bag package))))
+              (package-development-inputs package system)))
 
 (define (show-help)
   (display (G_ "Usage: guix environment [OPTION]... PACKAGE... [-- COMMAND...]
diff --git a/tests/packages.scm b/tests/packages.scm
index 3756877270..266b5aeb7a 100644
--- a/tests/packages.scm
+++ b/tests/packages.scm
@@ -353,6 +353,20 @@
           (package-transitive-supported-systems d)
           (package-transitive-supported-systems e))))
 
+(test-assert "package-development-inputs"
+  ;; Note: Due to propagated inputs, 'package-development-inputs' returns a
+  ;; couple more inputs, such as 'linux-libre-headers'.
+  (lset<= equal?
+          `(("source" ,(package-source hello)) ,@(standard-packages))
+          (package-development-inputs hello)))
+
+(test-assert "package-development-inputs, cross-compilation"
+  (lset<= equal?
+          `(("source" ,(package-source hello))
+            ,@(standard-cross-packages "mips64el-linux-gnu" 'host)
+            ,@(standard-cross-packages "mips64el-linux-gnu" 'target))
+          (package-development-inputs hello #:target "mips64el-linux-gnu")))
+
 (test-assert "package-closure"
   (let-syntax ((dummy-package/no-implicit
                 (syntax-rules ()
-- 
2.33.0

[bug#50960] [PATCH 02/10] profiles: Add 'package->development-manifest'.

[Ludovic Courtès]

From: Ludovic Courtès <ludovic.courtes@inria.fr>

* guix/profiles.scm (package->development-manifest): New procedure.
* guix/scripts/environment.scm (input->manifest-entry)
(package-environment-inputs): Remove.
* guix/scripts/environment.scm (options/resolve-packages): Use
'package->development-manifest' instead of 'package-environment-inputs'.
* tests/profiles.scm ("package->development-manifest"): New test.
---
 doc/guix.texi                | 11 +++++++++++
 guix/profiles.scm            | 19 +++++++++++++++++++
 guix/scripts/environment.scm | 27 +++++----------------------
 tests/profiles.scm           |  7 +++++++
 4 files changed, 42 insertions(+), 22 deletions(-)

diff --git a/doc/guix.texi b/doc/guix.texi
index 49399e792b..bc3f5a537b 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -3340,6 +3340,17 @@ objects, like this:
  '("emacs" "guile@@2.2" "guile@@2.2:debug"))
 @end lisp
 
+@findex package->development-manifest
+You might also want to create a manifest for all the dependencies of a
+package, rather than the package itself:
+
+@lisp
+(package->development-manifest (specification->package "emacs"))
+@end lisp
+
+The example above gives you all the software required to develop Emacs,
+similar to what @command{guix environment emacs} provides.
+
 @xref{export-manifest, @option{--export-manifest}}, to learn how to
 obtain a manifest file from an existing profile.
 
diff --git a/guix/profiles.scm b/guix/profiles.scm
index 2486f91d09..9f30349c69 100644
--- a/guix/profiles.scm
+++ b/guix/profiles.scm
@@ -124,6 +124,7 @@
 
             profile-manifest
             package->manifest-entry
+            package->development-manifest
             packages->manifest
             ca-certificate-bundle
             %default-profile-hooks
@@ -400,6 +401,24 @@ file name."
                      (properties properties))))
     entry))
 
+(define* (package->development-manifest package
+                                        #:optional
+                                        (system (%current-system))
+                                        #:key target)
+  "Return a manifest for the \"development inputs\" of PACKAGE for SYSTEM,
+optionally when cross-compiling to TARGET.  Development inputs include both
+explicit and implicit inputs of PACKAGE."
+  (manifest
+   (filter-map (match-lambda
+                 ((label (? package? package))
+                  (package->manifest-entry package))
+                 ((label (? package? package) output)
+                  (package->manifest-entry package output))
+                 ;; TODO: Support <inferior-package>.
+                 (_
+                  #f))
+               (package-development-inputs package system #:target target))))
+
 (define (packages->manifest packages)
   "Return a list of manifest entries, one for each item listed in PACKAGES.
 Elements of PACKAGES can be either package objects or package/string tuples
diff --git a/guix/scripts/environment.scm b/guix/scripts/environment.scm
index d555969b27..54f48a7482 100644
--- a/guix/scripts/environment.scm
+++ b/guix/scripts/environment.scm
@@ -66,24 +66,6 @@ do not augment existing environment variables with additional search paths."
                (newline)))
             (profile-search-paths profile manifest)))
 
-(define (input->manifest-entry input)
-  "Return a manifest entry for INPUT, or #f if INPUT does not correspond to a
-package."
-  (match input
-    ((_ (? package? package))
-     (package->manifest-entry package))
-    ((_ (? package? package) output)
-     (package->manifest-entry package output))
-    (_
-     #f)))
-
-(define (package-environment-inputs package)
-  "Return a list of manifest entries corresponding to the transitive input
-packages for PACKAGE."
-  ;; Remove non-package inputs such as origin records.
-  (filter-map input->manifest-entry
-              (package-development-inputs package system)))
-
 (define (show-help)
   (display (G_ "Usage: guix environment [OPTION]... PACKAGE... [-- COMMAND...]
 Build an environment that includes the dependencies of PACKAGE and execute
@@ -297,11 +279,11 @@ for the corresponding packages."
       ((? package? package)
        (if (eq? mode 'ad-hoc-package)
            (list (package->manifest-entry* package))
-           (package-environment-inputs package)))
+           (manifest-entries (package->development-manifest package))))
       (((? package? package) (? string? output))
        (if (eq? mode 'ad-hoc-package)
            (list (package->manifest-entry* package output))
-           (package-environment-inputs package)))
+           (manifest-entries (package->development-manifest package))))
       ((lst ...)
        (append-map (cut packages->outputs <> mode) lst))))
 
@@ -313,8 +295,9 @@ for the corresponding packages."
                                  (specification->package+output spec)))
                      (list (package->manifest-entry* package output))))
                   (('package 'package (? string? spec))
-                   (package-environment-inputs
-                    (transform (specification->package+output spec))))
+                   (manifest-entries
+                    (package->development-manifest
+                     (transform (specification->package+output spec)))))
                   (('expression mode str)
                    ;; Add all the outputs of the package STR evaluates to.
                    (packages->outputs (read/eval str) mode))
diff --git a/tests/profiles.scm b/tests/profiles.scm
index 06a0387221..cac5b73347 100644
--- a/tests/profiles.scm
+++ b/tests/profiles.scm
@@ -265,6 +265,13 @@
            (manifest-transaction-removal-candidate? guile-2.0.9 t)
            (null? install) (null? downgrade) (null? upgrade)))))
 
+(test-assert "package->development-manifest"
+  (let ((manifest (package->development-manifest packages:hello)))
+    (every (lambda (name)
+             (manifest-installed? manifest
+                                  (manifest-pattern (name name))))
+           '("gcc" "binutils" "glibc" "coreutils" "grep" "sed"))))
+
 (test-assertm "profile-derivation"
   (mlet* %store-monad
       ((entry ->   (package->manifest-entry %bootstrap-guile))
-- 
2.33.0

[bug#50960] [PATCH 03/10] DRAFT Add 'guix shell'.

[Ludovic Courtès]

From: Ludovic Courtès <ludovic.courtes@inria.fr>

DRAFT: Add doc.  Print deprecation warning for 'guix environment'?

* guix/scripts/shell.scm, tests/guix-shell.sh: New files.
* Makefile.am (MODULES): Add 'shell.scm'.
(SH_TESTS): Add 'tests/guix-shell.sh'.
* guix/scripts/environment.scm (show-environment-options-help): New
procedure.
(show-help): Use it.
(guix-environment*): New procedure.
(guix-environment): Use it.
* po/guix/POTFILES.in: Add it.
---
 Makefile.am                  |   2 +
 guix/scripts/environment.scm |  52 +++++++++-----
 guix/scripts/shell.scm       | 136 +++++++++++++++++++++++++++++++++++
 po/guix/POTFILES.in          |   1 +
 tests/guix-shell.sh          |  54 ++++++++++++++
 5 files changed, 228 insertions(+), 17 deletions(-)
 create mode 100644 guix/scripts/shell.scm
 create mode 100644 tests/guix-shell.sh

diff --git a/Makefile.am b/Makefile.am
index b66789fa0b..c28c8799ec 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -315,6 +315,7 @@ MODULES =					\
   guix/scripts/import/stackage.scm		\
   guix/scripts/import/texlive.scm  		\
   guix/scripts/environment.scm			\
+  guix/scripts/shell.scm			\
   guix/scripts/publish.scm			\
   guix/scripts/edit.scm				\
   guix/scripts/size.scm				\
@@ -550,6 +551,7 @@ SH_TESTS =					\
   tests/guix-authenticate.sh			\
   tests/guix-environment.sh			\
   tests/guix-environment-container.sh		\
+  tests/guix-shell.sh				\
   tests/guix-graph.sh				\
   tests/guix-describe.sh			\
   tests/guix-repl.sh     			\
diff --git a/guix/scripts/environment.scm b/guix/scripts/environment.scm
index 54f48a7482..77956fc018 100644
--- a/guix/scripts/environment.scm
+++ b/guix/scripts/environment.scm
@@ -50,7 +50,11 @@
   #:use-module (srfi srfi-37)
   #:use-module (srfi srfi-98)
   #:export (assert-container-features
-            guix-environment))
+            guix-environment
+            guix-environment*
+            show-environment-options-help
+            (%options . %environment-options)
+            (%default-options . %environment-default-options)))
 
 (define %default-shell
   (or (getenv "SHELL") "/bin/sh"))
@@ -66,23 +70,16 @@ do not augment existing environment variables with additional search paths."
                (newline)))
             (profile-search-paths profile manifest)))
 
-(define (show-help)
-  (display (G_ "Usage: guix environment [OPTION]... PACKAGE... [-- COMMAND...]
-Build an environment that includes the dependencies of PACKAGE and execute
-COMMAND or an interactive shell in that environment.\n"))
+(define (show-environment-options-help)
+  "Print help about options shared between 'guix environment' and 'guix
+shell'."
   (display (G_ "
   -e, --expression=EXPR  create environment for the package that EXPR
                          evaluates to"))
   (display (G_ "
-  -l, --load=FILE        create environment for the package that the code within
-                         FILE evaluates to"))
-  (display (G_ "
   -m, --manifest=FILE    create environment with the manifest from FILE"))
   (display (G_ "
   -p, --profile=PATH     create environment from profile at PATH"))
-  (display (G_ "
-      --ad-hoc           include all specified packages in the environment instead
-                         of only their inputs"))
   (display (G_ "
       --pure             unset existing environment variables"))
   (display (G_ "
@@ -118,7 +115,24 @@ COMMAND or an interactive shell in that environment.\n"))
   (display (G_ "
   -v, --verbosity=LEVEL  use the given verbosity LEVEL"))
   (display (G_ "
-      --bootstrap        use bootstrap binaries to build the environment"))
+      --bootstrap        use bootstrap binaries to build the environment")))
+
+(define (show-help)
+  (display (G_ "Usage: guix environment [OPTION]... PACKAGE... [-- COMMAND...]
+Build an environment that includes the dependencies of PACKAGE and execute
+COMMAND or an interactive shell in that environment.\n"))
+  (warning (G_ "This command is deprecated in favor of 'guix shell'.\n"))
+  (newline)
+
+  ;; These two options are left out in 'guix shell'.
+  (display (G_ "
+  -l, --load=FILE        create environment for the package that the code within
+                         FILE evaluates to"))
+  (display (G_ "
+      --ad-hoc           include all specified packages in the environment instead
+                         of only their inputs"))
+
+  (show-environment-options-help)
   (newline)
   (show-build-options-help)
   (newline)
@@ -649,11 +663,15 @@ message if any test fails."
 
 (define-command (guix-environment . args)
   (category development)
-  (synopsis "spawn one-off software environments")
+  (synopsis "spawn one-off software environments (deprecated)")
 
+  (guix-environment* (parse-args args)))
+
+(define (guix-environment* opts)
+  "Run the 'guix environment' command on OPTS, an alist resulting for
+command-line option processing with 'parse-command-line'."
   (with-error-handling
-    (let* ((opts       (parse-args args))
-           (pure?      (assoc-ref opts 'pure))
+    (let* ((pure?      (assoc-ref opts 'pure))
            (container? (assoc-ref opts 'container?))
            (link-prof? (assoc-ref opts 'link-profile?))
            (network?   (assoc-ref opts 'network?))
@@ -724,8 +742,8 @@ message if any test fails."
                                      (prof-drv   (manifest->derivation
                                                   manifest system bootstrap?))
                                      (profile -> (if profile
-                                                   (readlink* profile)
-                                                   (derivation->output-path prof-drv)))
+                                                     (readlink* profile)
+                                                     (derivation->output-path prof-drv)))
                                      (gc-root -> (assoc-ref opts 'gc-root)))
 
                   ;; First build the inputs.  This is necessary even for
diff --git a/guix/scripts/shell.scm b/guix/scripts/shell.scm
new file mode 100644
index 0000000000..6a4b7a5092
--- /dev/null
+++ b/guix/scripts/shell.scm
@@ -0,0 +1,136 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2021 Ludovic Courtès <ludo@gnu.org>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (guix scripts shell)
+  #:use-module (guix ui)
+  #:use-module (guix scripts environment)
+  #:autoload   (guix scripts build) (show-build-options-help)
+  #:autoload   (guix transformations) (show-transformation-options-help)
+  #:use-module (guix scripts)
+  #:use-module (srfi srfi-1)
+  #:use-module (srfi srfi-26)
+  #:use-module (srfi srfi-37)
+  #:use-module (srfi srfi-71)
+  #:use-module (ice-9 match)
+  #:export (guix-shell))
+
+(define (show-help)
+  (display (G_ "Usage: guix shell [OPTION] PACKAGES... [-- COMMAND...]
+Build an environment that includes PACKAGES and execute COMMAND or an
+interactive shell in that environment.\n"))
+  (newline)
+
+  ;; These two options differ from 'guix environment'.
+  (display (G_ "
+  -D, --development      include the development inputs of the next package"))
+  (display (G_ "
+  -f, --install-from-file=FILE
+                         install the package that the code within FILE
+                         evaluates to"))
+
+  (show-environment-options-help)
+  (newline)
+  (show-build-options-help)
+  (newline)
+  (show-transformation-options-help)
+  (newline)
+  (display (G_ "
+  -h, --help             display this help and exit"))
+  (display (G_ "
+  -V, --version          display version information and exit"))
+  (newline)
+  (show-bug-report-information))
+
+(define (tag-package-arg opts arg)
+  "Return a two-element list with the form (TAG ARG) that tags ARG with either
+'ad-hoc' in OPTS has the 'ad-hoc?' key set to #t, or 'inputs' otherwise."
+  (if (assoc-ref opts 'ad-hoc?)
+      `(ad-hoc-package ,arg)
+      `(package ,arg)))
+
+(define (ensure-ad-hoc alist)
+  (if (assq-ref alist 'ad-hoc?)
+      alist
+      `((ad-hoc? . #t) ,@alist)))
+
+(define (wrapped-option opt)
+  "Wrap OPT, a SRFI-37 option, such that its processor always adds the
+'ad-hoc?' flag to the resulting alist."
+  (option (option-names opt)
+          (option-required-arg? opt)
+          (option-optional-arg? opt)
+          (compose ensure-ad-hoc (option-processor opt))))
+
+(define %options
+  ;; Specification of the command-line options.
+  (let ((to-remove '("ad-hoc" "inherit" "load" "help" "version")))
+    (append
+        (list (option '(#\h "help") #f #f
+                      (lambda args
+                        (show-help)
+                        (exit 0)))
+              (option '(#\V "version") #f #f
+                      (lambda args
+                        (show-version-and-exit "guix shell")))
+
+              (option '(#\D "development") #f #f
+                      (lambda (opt name arg result)
+                        ;; Temporarily remove the 'ad-hoc?' flag from result.
+                        ;; The next option will put it back thanks to
+                        ;; 'wrapped-option'.
+                        (alist-delete 'ad-hoc? result)))
+
+              ;; For consistency with 'guix package', support '-f' rather than
+              ;; '-l' like 'guix environment' does.
+              (option '(#\f "install-from-file") #t #f
+                      (lambda (opt name arg result)
+                        (alist-cons 'load (tag-package-arg result arg)
+                                    result))))
+        (filter-map (lambda (opt)
+                      (and (not (any (lambda (name)
+                                       (member name to-remove))
+                                     (option-names opt)))
+                           (wrapped-option opt)))
+                    %environment-options))))
+
+(define %default-options
+  `((ad-hoc? . #t)                                ;always true
+    ,@%environment-default-options))
+
+(define (parse-args args)
+  "Parse the list of command line arguments ARGS."
+  (define (handle-argument arg result)
+    (alist-cons 'package (tag-package-arg result arg)
+                (ensure-ad-hoc result)))
+
+  ;; The '--' token is used to separate the command to run from the rest of
+  ;; the operands.
+  (let ((args command (break (cut string=? "--" <>) args)))
+    (let ((opts (parse-command-line args %options (list %default-options)
+                                    #:argument-handler handle-argument)))
+      (match command
+        (() opts)
+        (("--") opts)
+        (("--" command ...) (alist-cons 'exec command opts))))))
+
+\f
+(define-command (guix-shell . args)
+  (category development)
+  (synopsis "spawn one-off software environments")
+
+  (guix-environment* (parse-args args)))
diff --git a/po/guix/POTFILES.in b/po/guix/POTFILES.in
index f5b76bf582..f8abeb2d38 100644
--- a/po/guix/POTFILES.in
+++ b/po/guix/POTFILES.in
@@ -99,6 +99,7 @@ guix/derivations.scm
 guix/scripts/archive.scm
 guix/scripts/build.scm
 guix/scripts/environment.scm
+guix/scripts/shell.scm
 guix/scripts/time-machine.scm
 guix/scripts/import/cpan.scm
 guix/scripts/import/crate.scm
diff --git a/tests/guix-shell.sh b/tests/guix-shell.sh
new file mode 100644
index 0000000000..f08637f7ff
--- /dev/null
+++ b/tests/guix-shell.sh
@@ -0,0 +1,54 @@
+# GNU Guix --- Functional package management for GNU
+# Copyright © 2021 Ludovic Courtès <ludo@gnu.org>
+#
+# This file is part of GNU Guix.
+#
+# GNU Guix is free software; you can redistribute it and/or modify it
+# under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or (at
+# your option) any later version.
+#
+# GNU Guix is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
+
+#
+# Test the 'guix shell' alias.
+#
+
+guix shell --version
+
+tmpdir="t-guix-shell-$$"
+trap 'rm -r "$tmpdir"' EXIT
+mkdir "$tmpdir"
+
+guix shell --bootstrap --pure guile-bootstrap -- guile --version
+
+# '--ad-hoc' is a thing of the past.
+! guix shell --ad-hoc guile-bootstrap
+
+if guile -c '(getaddrinfo "www.gnu.org" "80" AI_NUMERICSERV)' 2> /dev/null
+then
+    # Compute the build environment for the initial GNU Make.
+    guix shell --bootstrap --no-substitutes --search-paths --pure \
+         -D -e '(@ (guix tests) gnu-make-for-tests)' > "$tmpdir/a"
+
+    # Make sure bootstrap binaries are in the profile.
+    profile=`grep "^export PATH" "$tmpdir/a" | sed -r 's|^.*="(.*)/bin"|\1|'`
+
+    # Make sure the bootstrap binaries are all listed where they belong.
+    grep -E "^export PATH=\"$profile/bin\""         "$tmpdir/a"
+    grep -E "^export CPATH=\"$profile/include\""    "$tmpdir/a"
+    grep -E "^export LIBRARY_PATH=\"$profile/lib\"" "$tmpdir/a"
+    for dep in bootstrap-binaries-0 gcc-bootstrap-0 glibc-bootstrap-0
+    do
+	guix gc --references "$profile" | grep "$dep"
+    done
+
+    # 'make-boot0' itself must not be listed.
+    ! guix gc --references "$profile" | grep make-boot0
+fi
-- 
2.33.0

[bug#50960] [PATCH 04/10] DRAFT shell: By default load the local 'guix.scm' or 'manifest.scm' file.

[Ludovic Courtès]

DRAFT: Add doc.

* guix/scripts/shell.scm (parse-args): Add call to 'auto-detect-manifest'.
(find-file-in-parent-directories, auto-detect-manifest): New procedures.
* tests/guix-shell.sh: Add test.
---
 guix/scripts/shell.scm | 44 ++++++++++++++++++++++++++++++++++++++++--
 tests/guix-shell.sh    | 16 +++++++++++++++
 2 files changed, 58 insertions(+), 2 deletions(-)

diff --git a/guix/scripts/shell.scm b/guix/scripts/shell.scm
index 6a4b7a5092..2f15befbd3 100644
--- a/guix/scripts/shell.scm
+++ b/guix/scripts/shell.scm
@@ -22,6 +22,8 @@
   #:autoload   (guix scripts build) (show-build-options-help)
   #:autoload   (guix transformations) (show-transformation-options-help)
   #:use-module (guix scripts)
+  #:use-module (guix packages)
+  #:use-module (guix profiles)
   #:use-module (srfi srfi-1)
   #:use-module (srfi srfi-26)
   #:use-module (srfi srfi-37)
@@ -121,13 +123,51 @@ interactive shell in that environment.\n"))
   ;; The '--' token is used to separate the command to run from the rest of
   ;; the operands.
   (let ((args command (break (cut string=? "--" <>) args)))
-    (let ((opts (parse-command-line args %options (list %default-options)
-                                    #:argument-handler handle-argument)))
+    (let ((opts (auto-detect-manifest
+                 (parse-command-line args %options (list %default-options)
+                                     #:argument-handler handle-argument))))
       (match command
         (() opts)
         (("--") opts)
         (("--" command ...) (alist-cons 'exec command opts))))))
 
+(define (find-file-in-parent-directories candidates)
+  "Find one of CANDIDATES in the current directory or one of its ancestors."
+  (let loop ((directory (getcwd)))
+    (and (= (stat:uid (stat directory)) (getuid))
+         (or (any (lambda (candidate)
+                    (let ((candidate (string-append directory "/" candidate)))
+                      (and (file-exists? candidate) candidate)))
+                  candidates)
+             (loop (string-append directory "/..")))))) ;Unix ".." resolution
+
+(define (auto-detect-manifest opts)
+  "If OPTS do not specify packages or a manifest, load a \"guix.scm\" or
+\"manifest.scm\" file from the current directory or one of its ancestors.
+Return the modified OPTS."
+  (define (options-contain-payload? opts)
+    (match opts
+      (() #f)
+      ((('package . _) . _) #t)
+      ((('load . _) . _) #t)
+      ((('manifest . _) . _) #t)
+      ((('expression . _) . _) #t)
+      ((_ . rest) (options-contain-payload? rest))))
+
+  (if (options-contain-payload? opts)
+      opts
+      (match (find-file-in-parent-directories '("guix.scm" "manifest.scm"))
+        (#f
+         (warning (G_ "no packages specified; creating an empty environment~%"))
+         opts)
+        (file
+         (info (G_ "loading environment from '~a'...~%") file)
+         (match (basename file)
+           ("guix.scm"
+            (alist-cons 'load `(package ,file) opts))
+           ("manifest.scm"
+            (alist-cons 'manifest file opts)))))))
+
 \f
 (define-command (guix-shell . args)
   (category development)
diff --git a/tests/guix-shell.sh b/tests/guix-shell.sh
index f08637f7ff..498c1c5515 100644
--- a/tests/guix-shell.sh
+++ b/tests/guix-shell.sh
@@ -31,6 +31,16 @@ guix shell --bootstrap --pure guile-bootstrap -- guile --version
 # '--ad-hoc' is a thing of the past.
 ! guix shell --ad-hoc guile-bootstrap
 
+# Honoring the local 'manifest.scm' file.
+cat > "$tmpdir/manifest.scm" <<EOF
+(specifications->manifest '("guile-bootstrap"))
+EOF
+profile1="$(cd "$tmpdir"; guix shell --bootstrap -- "$SHELL" -c 'echo $GUIX_ENVIRONMENT')"
+profile2="$(guix shell --bootstrap guile-bootstrap -- "$SHELL" -c 'echo $GUIX_ENVIRONMENT')"
+test -n "$profile1"
+test "$profile1" = "$profile2"
+rm "$tmpdir/manifest.scm"
+
 if guile -c '(getaddrinfo "www.gnu.org" "80" AI_NUMERICSERV)' 2> /dev/null
 then
     # Compute the build environment for the initial GNU Make.
@@ -51,4 +61,10 @@ then
 
     # 'make-boot0' itself must not be listed.
     ! guix gc --references "$profile" | grep make-boot0
+
+    # Honoring the local 'guix.scm' file.
+    echo '(@ (guix tests) gnu-make-for-tests)' > "$tmpdir/guix.scm"
+    (cd "$tmpdir"; guix shell --bootstrap --search-paths --pure > "b")
+    cmp "$tmpdir/a" "$tmpdir/b"
+    rm "$tmpdir/guix.scm"
 fi
-- 
2.33.0

[bug#50960] [PATCH 04/10] DRAFT shell: By default load the local 'guix.scm' or 'manifest.scm' file.

[Liliana Marie Prikler]

Hi,

Am Samstag, den 02.10.2021, 12:22 +0200 schrieb Ludovic Courtès:
> [...]
> +(define (auto-detect-manifest opts)
> +  "If OPTS do not specify packages or a manifest, load a
> \"guix.scm\" or
> +\"manifest.scm\" file from the current directory or one of its
> ancestors.
> +Return the modified OPTS."
> +  (define (options-contain-payload? opts)
> +    (match opts
> +      (() #f)
> +      ((('package . _) . _) #t)
> +      ((('load . _) . _) #t)
> +      ((('manifest . _) . _) #t)
> +      ((('expression . _) . _) #t)
> +      ((_ . rest) (options-contain-payload? rest))))
> +
> +  (if (options-contain-payload? opts)
> +      opts
> +      (match (find-file-in-parent-directories '("guix.scm"
> "manifest.scm"))
> +        (#f
> +         (warning (G_ "no packages specified; creating an empty
> environment~%"))
> +         opts)
> +        (file
> +         (info (G_ "loading environment from '~a'...~%") file)
> +         (match (basename file)
> +           ("guix.scm"
> +            (alist-cons 'load `(package ,file) opts))
> +           ("manifest.scm"
> +            (alist-cons 'manifest file opts)))))))
> [...]
What would happen on the top-level of the Guix source tree or deep
inside the tree of a guile package that deals with manifests, that
aren't necessarily related to Guix?  I think we should try searching
for something less ambiguous first (".guix-shell/manifest" perhaps?)
and maybe also provide further options after manifest.scm (e.g. build-
aux/guix.scm or etc/guix.scm)

WDYT?

[bug#50960] [PATCH 00/10] Add 'guix shell' to subsume 'guix environment'

[Ludovic Courtès]

Hi,

Liliana Marie Prikler <liliana.prikler@gmail.com> skribis:

> Am Samstag, den 02.10.2021, 12:22 +0200 schrieb Ludovic Courtès:
>> [...]
>> +(define (auto-detect-manifest opts)
>> +  "If OPTS do not specify packages or a manifest, load a
>> \"guix.scm\" or
>> +\"manifest.scm\" file from the current directory or one of its
>> ancestors.
>> +Return the modified OPTS."
>> +  (define (options-contain-payload? opts)
>> +    (match opts
>> +      (() #f)
>> +      ((('package . _) . _) #t)
>> +      ((('load . _) . _) #t)
>> +      ((('manifest . _) . _) #t)
>> +      ((('expression . _) . _) #t)
>> +      ((_ . rest) (options-contain-payload? rest))))
>> +
>> +  (if (options-contain-payload? opts)
>> +      opts
>> +      (match (find-file-in-parent-directories '("guix.scm"
>> "manifest.scm"))
>> +        (#f
>> +         (warning (G_ "no packages specified; creating an empty
>> environment~%"))
>> +         opts)
>> +        (file
>> +         (info (G_ "loading environment from '~a'...~%") file)
>> +         (match (basename file)
>> +           ("guix.scm"
>> +            (alist-cons 'load `(package ,file) opts))
>> +           ("manifest.scm"
>> +            (alist-cons 'manifest file opts)))))))
>> [...]
> What would happen on the top-level of the Guix source tree or deep
> inside the tree of a guile package that deals with manifests, that
> aren't necessarily related to Guix?

You mean a directory that contains a file named ‘guix.scm’ or
‘manifest.scm’ but that happens to do something completely unrelated?

We can never rule this out, but I’d say it’s unlikely (these two
conventions are rather well established) and it’s up to the user to pay
attention.

WDYT?

Thanks,
Ludo’.

[bug#50960] [PATCH 00/10] Add 'guix shell' to subsume 'guix environment'

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 2274 bytes --]

Ludovic Courtès schreef op za 02-10-2021 om 15:43 [+0200]:
> Hi,
> 
> Liliana Marie Prikler <liliana.prikler@gmail.com> skribis:
> 
> > Am Samstag, den 02.10.2021, 12:22 +0200 schrieb Ludovic Courtès:
> > > [...]
> > > +(define (auto-detect-manifest opts)
> > > +  "If OPTS do not specify packages or a manifest, load a
> > > \"guix.scm\" or
> > > +\"manifest.scm\" file from the current directory or one of its
> > > ancestors.
> > > +Return the modified OPTS."
> > > +  (define (options-contain-payload? opts)
> > > +    (match opts
> > > +      (() #f)
> > > +      ((('package . _) . _) #t)
> > > +      ((('load . _) . _) #t)
> > > +      ((('manifest . _) . _) #t)
> > > +      ((('expression . _) . _) #t)
> > > +      ((_ . rest) (options-contain-payload? rest))))
> > > +
> > > +  (if (options-contain-payload? opts)
> > > +      opts
> > > +      (match (find-file-in-parent-directories '("guix.scm"
> > > "manifest.scm"))
> > > +        (#f
> > > +         (warning (G_ "no packages specified; creating an empty
> > > environment~%"))
> > > +         opts)
> > > +        (file
> > > +         (info (G_ "loading environment from '~a'...~%") file)
> > > +         (match (basename file)
> > > +           ("guix.scm"
> > > +            (alist-cons 'load `(package ,file) opts))
> > > +           ("manifest.scm"
> > > +            (alist-cons 'manifest file opts)))))))
> > > [...]
> > What would happen on the top-level of the Guix source tree or deep
> > inside the tree of a guile package that deals with manifests, that
> > aren't necessarily related to Guix?
> 
> You mean a directory that contains a file named ‘guix.scm’ or
> ‘manifest.scm’ but that happens to do something completely unrelated?
> 
> We can never rule this out, but I’d say it’s unlikely (these two
> conventions are rather well established) and it’s up to the user to pay
> attention.
> 
> WDYT?

Guix itself doesn't follow this convention: the guix source tree has an unrelated
"guix.scm" file, that doesn't evaluate to a package.  I'd expect that running
"guix shell" within the guix source tree would be equivalent to
"guix environment guix", but currently this doesn't seem to be the case.

Greetings,
Maxime.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

[bug#50960] [PATCH 00/10] Add 'guix shell' to subsume 'guix environment'

[Ludovic Courtès]

Maxime Devos <maximedevos@telenet.be> skribis:

> Guix itself doesn't follow this convention: the guix source tree has an unrelated
> "guix.scm" file, that doesn't evaluate to a package.

Indeed, but that’s because Guix predates the convention.  :-)

> I'd expect that running "guix shell" within the guix source tree would
> be equivalent to "guix environment guix", but currently this doesn't
> seem to be the case.

Right, it would fail poorly in this particular case.

Ludo’.

[bug#50960] [PATCH 04/10] DRAFT shell: By default load the local 'guix.scm' or 'manifest.scm' file.

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 1932 bytes --]

Ludovic Courtès schreef op za 02-10-2021 om 12:22 [+0200]:
> +(define (find-file-in-parent-directories candidates)
> +  "Find one of CANDIDATES in the current directory or one of its ancestors."
> +  (let loop ((directory (getcwd)))
> +    (and (= (stat:uid (stat directory)) (getuid))
> +         (or (any (lambda (candidate)
> +                    (let ((candidate (string-append directory "/" candidate)))
> +                      (and (file-exists? candidate) candidate)))
> +                  candidates)
> +             (loop (string-append directory "/..")))))) ;Unix ".." resolution

I do not recommend this.  What would happen if someone creates a temporary directory
"/tmp/stuff" do things in to throw away later (setting permissions appropriately),
tries to create a guix.scm in that directory but misspells it as, say, guix.sm, and runs
"guix shell" from within /tmp/stuff?  Then find-file-in-parent-directories would
load /tmp/guix.scm (possibly created by a local attacker, assuming a multi-user system),
-- if it weren't for the (= (stat:uid (stat directory)) (getuid)).

Because of the (= (stat:uid ...) (getuid)), this attack method is not possible.
However, it causes other issues.  Now it isn't possible for two users (that trust
each other), to set up a directory writable by both (e.g. with ACLs, or by making
the directory group-writable and placing the two users in the same group), for
working together, with a guix.scm usable by both.

These can be two users on the same machine, or remotely via something like NFS,
or a single person having multiple user accounts used for different purposes.

(I once created multiple user accounts on Debian: one regular purpose, one for reading
and games, and one for school, and made the ‘for-reading’ and ‘school’ home directory
readable by the ‘regular-purpose’ account.  It was occasionally useful.)

Greetings,
Maxime.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

[bug#50960] [PATCH 04/10] DRAFT shell: By default load the local 'guix.scm' or 'manifest.scm' file.

[Ludovic Courtès]

Hi Maxime,

Maxime Devos <maximedevos@telenet.be> skribis:

> Ludovic Courtès schreef op za 02-10-2021 om 12:22 [+0200]:
>> +(define (find-file-in-parent-directories candidates)
>> +  "Find one of CANDIDATES in the current directory or one of its ancestors."
>> +  (let loop ((directory (getcwd)))
>> +    (and (= (stat:uid (stat directory)) (getuid))
>> +         (or (any (lambda (candidate)
>> +                    (let ((candidate (string-append directory "/" candidate)))
>> +                      (and (file-exists? candidate) candidate)))
>> +                  candidates)
>> +             (loop (string-append directory "/..")))))) ;Unix ".." resolution
>
> I do not recommend this.  What would happen if someone creates a temporary directory
> "/tmp/stuff" do things in to throw away later (setting permissions appropriately),
> tries to create a guix.scm in that directory but misspells it as, say, guix.sm, and runs
> "guix shell" from within /tmp/stuff?  Then find-file-in-parent-directories would
> load /tmp/guix.scm (possibly created by a local attacker, assuming a multi-user system),
> -- if it weren't for the (= (stat:uid (stat directory)) (getuid)).
>
> Because of the (= (stat:uid ...) (getuid)), this attack method is not possible.

Right.  :-)

In libgit2, ‘find_repo’ (called by ‘git_repository_discover’) stops at
device boundaries, which is wise.  But it doesn’t stop when the parent
has a different owner (!).

Unlike the code above, it does lexical “..” resolution after first
calling realpath(3) on the directory name; not sure what to think about
this.  (The code of Git itself is harder to read for me.)

> However, it causes other issues.  Now it isn't possible for two users (that trust
> each other), to set up a directory writable by both (e.g. with ACLs, or by making
> the directory group-writable and placing the two users in the same group), for
> working together, with a guix.scm usable by both.
>
> These can be two users on the same machine, or remotely via something like NFS,
> or a single person having multiple user accounts used for different purposes.

Well, sure, but that’s a very uncommon scenario, isn’t it?

I was actually hesitant about this find-in-parent behavior.  I find it
convenient that ‘git’ does that, for instance, so I thought it might be
nice as well.

Thoughts?

Ludo’.

[bug#50960] [PATCH 04/10] DRAFT shell: By default load the local 'guix.scm' or 'manifest.scm' file.

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 3276 bytes --]

Ludovic Courtès schreef op za 02-10-2021 om 12:22 [+0200]:
> DRAFT: Add doc.
> 
> * guix/scripts/shell.scm (parse-args): Add call to 'auto-detect-manifest'.
> (find-file-in-parent-directories, auto-detect-manifest): New procedures.
> * tests/guix-shell.sh: Add test.
> ---
>  guix/scripts/shell.scm | 44 ++++++++++++++++++++++++++++++++++++++++--
>  tests/guix-shell.sh    | 16 +++++++++++++++
>  2 files changed, 58 insertions(+), 2 deletions(-)
> 
> diff --git a/guix/scripts/shell.scm b/guix/scripts/shell.scm
> index 6a4b7a5092..2f15befbd3 100644
> --- a/guix/scripts/shell.scm
> +++ b/guix/scripts/shell.scm
> @@ -22,6 +22,8 @@
>    #:autoload   (guix scripts build) (show-build-options-help)
>    #:autoload   (guix transformations) (show-transformation-options-help)
>    #:use-module (guix scripts)
> +  #:use-module (guix packages)
> +  #:use-module (guix profiles)
>    #:use-module (srfi srfi-1)
>    #:use-module (srfi srfi-26)
>    #:use-module (srfi srfi-37)
> @@ -121,13 +123,51 @@ interactive shell in that environment.\n"))
>    ;; The '--' token is used to separate the command to run from the rest of
>    ;; the operands.
>    (let ((args command (break (cut string=? "--" <>) args)))
> -    (let ((opts (parse-command-line args %options (list %default-options)
> -                                    #:argument-handler handle-argument)))
> +    (let ((opts (auto-detect-manifest
> +                 (parse-command-line args %options (list %default-options)
> +                                     #:argument-handler handle-argument))))
>        (match command
>          (() opts)
>          (("--") opts)
>          (("--" command ...) (alist-cons 'exec command opts))))))
>  
> +(define (find-file-in-parent-directories candidates)
> +  "Find one of CANDIDATES in the current directory or one of its ancestors."
> +  (let loop ((directory (getcwd)))
> +    (and (= (stat:uid (stat directory)) (getuid))
> +         (or (any (lambda (candidate)
> +                    (let ((candidate (string-append directory "/" candidate)))
> +                      (and (file-exists? candidate) candidate)))
> +                  candidates)
> +             (loop (string-append directory "/..")))))) ;Unix ".." resolution
> +
> +(define (auto-detect-manifest opts)
> +  "If OPTS do not specify packages or a manifest, load a \"guix.scm\" or
> +\"manifest.scm\" file from the current directory or one of its ancestors.
> +Return the modified OPTS."
> +  (define (options-contain-payload? opts)
> +    (match opts
> +      (() #f)
> +      ((('package . _) . _) #t)
> +      ((('load . _) . _) #t)
> +      ((('manifest . _) . _) #t)
> +      ((('expression . _) . _) #t)
> +      ((_ . rest) (options-contain-payload? rest))))
> +
> +  (if (options-contain-payload? opts)
> +      opts
> +      (match (find-file-in-parent-directories '("guix.scm" "manifest.scm"))
> +        (#f
> +         (warning (G_ "no packages specified; creating an empty environment~%"))
> +         opts)
> +        (file
> +         (info (G_ "loading environment from '~a'...~%") file)

Could we have nice ‘curly quotes’ here:

  (info (G_ "loading environment from ‘~a’...~%") file)

Greetings,
Maxime.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

[bug#50960] [PATCH 05/10] environment: Add tests for '--profile'.

[Ludovic Courtès]

This is a followup to a643deac2de81755a1843a3b41dd53857678bebc.

* tests/guix-environment-container.sh, tests/guix-environment.sh: Add
tests for '--profile'.
---
 tests/guix-environment-container.sh | 8 ++++++++
 tests/guix-environment.sh           | 7 +++++++
 2 files changed, 15 insertions(+)

diff --git a/tests/guix-environment-container.sh b/tests/guix-environment-container.sh
index f2d15c8d0c..2e238c501d 100644
--- a/tests/guix-environment-container.sh
+++ b/tests/guix-environment-container.sh
@@ -44,6 +44,14 @@ else
     test $? = 42
 fi
 
+# Try '--root' and '--profile'.
+root="$tmpdir/root"
+guix environment -C --ad-hoc --bootstrap guile-bootstrap -r "$root" -- guile --version
+guix environment -C -p "$root" --bootstrap -- guile --version
+path1=$(guix environment -C -p "$root" --bootstrap -- guile -c '(display (getenv "PATH"))')
+path2=$(guix environment -C --ad-hoc --bootstrap guile-bootstrap  -- guile -c '(display (getenv "PATH"))')
+test "$path1" = "$path2"
+
 # Make sure "localhost" resolves.
 guix environment --container --ad-hoc --bootstrap guile-bootstrap \
      -- guile -c '(exit (pair? (getaddrinfo "localhost" "80")))'
diff --git a/tests/guix-environment.sh b/tests/guix-environment.sh
index afadcbe195..f4fc2e39ed 100644
--- a/tests/guix-environment.sh
+++ b/tests/guix-environment.sh
@@ -119,6 +119,13 @@ test `readlink "$gcroot"` = "$expected"
 guix environment --bootstrap -r "$gcroot" --ad-hoc guile-bootstrap \
      -- guile -c 1
 test `readlink "$gcroot"` = "$expected"
+
+# Make sure '-p' works as expected.
+test $(guix environment -p "$gcroot" -- "$SHELL" -c 'echo $GUIX_ENVIRONMENT') = "$expected"
+paths1="$(guix environment -p "$gcroot" --search-paths)"
+paths2="$(guix environment --bootstrap --ad-hoc guile-bootstrap --search-paths)"
+test "$paths1" = "$paths2"
+
 rm "$gcroot"
 
 # Try '-r' with a relative file name.
-- 
2.33.0

[bug#50960] [PATCH 06/10] environment: Skip derivation computation when '--profile' is used.

[Ludovic Courtès]

* guix/scripts/environment.scm (guix-environment*): Bypass calls to
'package-derivation' and to 'manifest->derivation' when PROFILE is
true.
---
 guix/scripts/environment.scm | 23 +++++++++++++----------
 1 file changed, 13 insertions(+), 10 deletions(-)

diff --git a/guix/scripts/environment.scm b/guix/scripts/environment.scm
index 77956fc018..32f376fdd2 100644
--- a/guix/scripts/environment.scm
+++ b/guix/scripts/environment.scm
@@ -729,18 +729,21 @@ command-line option processing with 'parse-command-line'."
             ;; Use the bootstrap Guile when requested.
             (parameterize ((%graft? (assoc-ref opts 'graft?))
                            (%guile-for-build
-                            (package-derivation
-                             store
-                             (if bootstrap?
-                                 %bootstrap-guile
-                                 (default-guile)))))
+                            (and (or container? (not profile))
+                                 (package-derivation
+                                  store
+                                  (if bootstrap?
+                                      %bootstrap-guile
+                                      (default-guile))))))
               (run-with-store store
                 ;; Containers need a Bourne shell at /bin/sh.
                 (mlet* %store-monad ((bash       (environment-bash container?
                                                                    bootstrap?
                                                                    system))
-                                     (prof-drv   (manifest->derivation
-                                                  manifest system bootstrap?))
+                                     (prof-drv   (if profile
+                                                     (return #f)
+                                                     (manifest->derivation
+                                                      manifest system bootstrap?)))
                                      (profile -> (if profile
                                                      (readlink* profile)
                                                      (derivation->output-path prof-drv)))
@@ -750,9 +753,9 @@ command-line option processing with 'parse-command-line'."
                   ;; --search-paths.  Additionally, we might need to build bash for
                   ;; a container.
                   (mbegin %store-monad
-                    (built-derivations (if (derivation? bash)
-                                           (list prof-drv bash)
-                                           (list prof-drv)))
+                    (built-derivations (append
+                                           (if prof-drv (list prof-drv) '())
+                                           (if (derivation? bash) (list bash) '())))
                     (mwhen gc-root
                       (register-gc-root profile gc-root))
 
-- 
2.33.0

[bug#50960] [PATCH 06/10] environment: Skip derivation computation when '--profile' is used.

[Liliana Marie Prikler]

Am Samstag, den 02.10.2021, 12:22 +0200 schrieb Ludovic Courtès:
> * guix/scripts/environment.scm (guix-environment*): Bypass calls to
> 'package-derivation' and to 'manifest->derivation' when PROFILE is
> true.
This only affects `guix shell' and not `guix environment', right?  If
not, does the outward behaviour of `guix environment' stay the same
considering this patch and 07/10?  There might be people relying on the
way `guix environment' *currently* works, who would need to be informed
about that change.

Then again, if the following holds
>   If you run ‘guix pull’ and run again ‘guix shell’, it recomputes
>   the environment, as is currently the case with ‘guix environment’.
then the behaviour of guix environment should also be consistent with
what it did before, but with the added cache of guix shell.  Am I
reading this correctly?

Regards,
Liliana

[bug#50960] [PATCH 00/10] Add 'guix shell' to subsume 'guix environment'

[Ludovic Courtès]

Liliana Marie Prikler <liliana.prikler@gmail.com> skribis:

> Am Samstag, den 02.10.2021, 12:22 +0200 schrieb Ludovic Courtès:
>> * guix/scripts/environment.scm (guix-environment*): Bypass calls to
>> 'package-derivation' and to 'manifest->derivation' when PROFILE is
>> true.
> This only affects `guix shell' and not `guix environment', right?

No, it affects ‘guix environment’ (it’s in environment.scm).  It’s an
optimization of ‘guix environment -p’, but its observable behavior is
unchanged; it’s just faster.

> Then again, if the following holds
>>   If you run ‘guix pull’ and run again ‘guix shell’, it recomputes
>>   the environment, as is currently the case with ‘guix environment’.
> then the behaviour of guix environment should also be consistent with
> what it did before, but with the added cache of guix shell.  Am I
> reading this correctly?

The cache itself is only in ‘guix shell’, in the no-argument case:

  https://issues.guix.gnu.org/50960#9

Ludo’.

[bug#50960] [PATCH 07/10] environment: Do not connect to the daemon when '--profile' is used.

[Ludovic Courtès]

This further speeds up the 'guix environment -p PROFILE' case.

* guix/scripts/environment.scm (guix-environment*)[store-needed?]: New
variable.
[with-store/maybe]: New macro.
Use it instead of 'with-store', and remove 'with-build-handler' form.
---
 guix/scripts/environment.scm | 169 +++++++++++++++++++----------------
 1 file changed, 93 insertions(+), 76 deletions(-)

diff --git a/guix/scripts/environment.scm b/guix/scripts/environment.scm
index 32f376fdd2..e23d52df39 100644
--- a/guix/scripts/environment.scm
+++ b/guix/scripts/environment.scm
@@ -691,6 +691,26 @@ command-line option processing with 'parse-command-line'."
            (mappings   (pick-all opts 'file-system-mapping))
            (white-list (pick-all opts 'inherit-regexp)))
 
+      (define store-needed?
+        ;; Whether connecting to the daemon is needed.
+        (or container? (not profile)))
+
+      (define-syntax-rule (with-store/maybe store exp ...)
+        ;; Evaluate EXP... with STORE bound to a connection, unless
+        ;; STORE-NEEDED? is false, in which case STORE is bound to #f.
+        (let ((proc (lambda (store) exp ...)))
+          (if store-needed?
+              (with-store s
+                (set-build-options-from-command-line s opts)
+                (with-build-handler (build-notifier #:use-substitutes?
+                                                    (assoc-ref opts 'substitutes?)
+                                                    #:verbosity
+                                                    (assoc-ref opts 'verbosity)
+                                                    #:dry-run?
+                                                    (assoc-ref opts 'dry-run?))
+                  (proc s)))
+              (proc #f))))
+
       (when container? (assert-container-features))
 
       (when (and (not container?) link-prof?)
@@ -701,88 +721,85 @@ command-line option processing with 'parse-command-line'."
         (leave (G_ "--no-cwd cannot be used without --container~%")))
 
 
-      (with-store store
-        (with-build-handler (build-notifier #:use-substitutes?
-                                            (assoc-ref opts 'substitutes?)
-                                            #:verbosity
-                                            (assoc-ref opts 'verbosity)
-                                            #:dry-run?
-                                            (assoc-ref opts 'dry-run?))
-          (with-status-verbosity (assoc-ref opts 'verbosity)
-            (define manifest-from-opts
-              (options/resolve-packages store opts))
+      (with-store/maybe store
+        (with-status-verbosity (assoc-ref opts 'verbosity)
+          (define manifest-from-opts
+            (options/resolve-packages store opts))
 
-            (define manifest
-              (if profile
-                  (profile-manifest profile)
-                  manifest-from-opts))
+          (define manifest
+            (if profile
+                (profile-manifest profile)
+                manifest-from-opts))
 
-            (when (and profile
-                       (> (length (manifest-entries manifest-from-opts)) 0))
-              (leave (G_ "'--profile' cannot be used with package options~%")))
+          (when (and profile
+                     (> (length (manifest-entries manifest-from-opts)) 0))
+            (leave (G_ "'--profile' cannot be used with package options~%")))
 
-            (when (null? (manifest-entries manifest))
-              (warning (G_ "no packages specified; creating an empty environment~%")))
+          (when (null? (manifest-entries manifest))
+            (warning (G_ "no packages specified; creating an empty environment~%")))
 
-            (set-build-options-from-command-line store opts)
+          ;; Use the bootstrap Guile when requested.
+          (parameterize ((%graft? (assoc-ref opts 'graft?))
+                         (%guile-for-build
+                          (and store-needed?
+                               (package-derivation
+                                store
+                                (if bootstrap?
+                                    %bootstrap-guile
+                                    (default-guile))))))
+            (run-with-store store
+              ;; Containers need a Bourne shell at /bin/sh.
+              (mlet* %store-monad ((bash       (environment-bash container?
+                                                                 bootstrap?
+                                                                 system))
+                                   (prof-drv   (if profile
+                                                   (return #f)
+                                                   (manifest->derivation
+                                                    manifest system bootstrap?)))
+                                   (profile -> (if profile
+                                                   (readlink* profile)
+                                                   (derivation->output-path prof-drv)))
+                                   (gc-root -> (assoc-ref opts 'gc-root)))
 
-            ;; Use the bootstrap Guile when requested.
-            (parameterize ((%graft? (assoc-ref opts 'graft?))
-                           (%guile-for-build
-                            (and (or container? (not profile))
-                                 (package-derivation
-                                  store
-                                  (if bootstrap?
-                                      %bootstrap-guile
-                                      (default-guile))))))
-              (run-with-store store
-                ;; Containers need a Bourne shell at /bin/sh.
-                (mlet* %store-monad ((bash       (environment-bash container?
-                                                                   bootstrap?
-                                                                   system))
-                                     (prof-drv   (if profile
-                                                     (return #f)
-                                                     (manifest->derivation
-                                                      manifest system bootstrap?)))
-                                     (profile -> (if profile
-                                                     (readlink* profile)
-                                                     (derivation->output-path prof-drv)))
-                                     (gc-root -> (assoc-ref opts 'gc-root)))
-
-                  ;; First build the inputs.  This is necessary even for
-                  ;; --search-paths.  Additionally, we might need to build bash for
-                  ;; a container.
-                  (mbegin %store-monad
+                ;; First build the inputs.  This is necessary even for
+                ;; --search-paths.  Additionally, we might need to build bash for
+                ;; a container.
+                (mbegin %store-monad
+                  (mwhen store-needed?
                     (built-derivations (append
                                            (if prof-drv (list prof-drv) '())
-                                           (if (derivation? bash) (list bash) '())))
-                    (mwhen gc-root
-                      (register-gc-root profile gc-root))
+                                           (if (derivation? bash) (list bash) '()))))
+                  (mwhen gc-root
+                    (register-gc-root profile gc-root))
 
-                    (cond
-                     ((assoc-ref opts 'search-paths)
-                      (show-search-paths profile manifest #:pure? pure?)
-                      (return #t))
-                     (container?
-                      (let ((bash-binary
-                             (if bootstrap?
-                                 (derivation->output-path bash)
-                                 (string-append (derivation->output-path bash)
-                                                "/bin/sh"))))
-                        (launch-environment/container #:command command
-                                                      #:bash bash-binary
-                                                      #:user user
-                                                      #:user-mappings mappings
-                                                      #:profile profile
-                                                      #:manifest manifest
-                                                      #:white-list white-list
-                                                      #:link-profile? link-prof?
-                                                      #:network? network?
-                                                      #:map-cwd? (not no-cwd?))))
+                  (cond
+                   ((assoc-ref opts 'search-paths)
+                    (show-search-paths profile manifest #:pure? pure?)
+                    (return #t))
+                   (container?
+                    (let ((bash-binary
+                           (if bootstrap?
+                               (derivation->output-path bash)
+                               (string-append (derivation->output-path bash)
+                                              "/bin/sh"))))
+                      (launch-environment/container #:command command
+                                                    #:bash bash-binary
+                                                    #:user user
+                                                    #:user-mappings mappings
+                                                    #:profile profile
+                                                    #:manifest manifest
+                                                    #:white-list white-list
+                                                    #:link-profile? link-prof?
+                                                    #:network? network?
+                                                    #:map-cwd? (not no-cwd?))))
 
-                     (else
-                      (return
-                       (exit/status
-                        (launch-environment/fork command profile manifest
-                                                 #:white-list white-list
-                                                 #:pure? pure?)))))))))))))))
+                   (else
+                    (return
+                     (exit/status
+                      (launch-environment/fork command profile manifest
+                                               #:white-list white-list
+                                               #:pure? pure?))))))))))))))
+
+;;; Local Variables:
+;;; (put 'with-store/maybe 'scheme-indent-function 1)
+;;; End:
-- 
2.33.0

[bug#50960] [PATCH 08/10] environment: Autoload some modules.

[Ludovic Courtès]

This further speeds up the 'guix environment -p PROFILE' case.

* guix/scripts/environment.scm: Autoload a bunch of modules.
---
 guix/scripts/environment.scm | 19 +++++++++++--------
 1 file changed, 11 insertions(+), 8 deletions(-)

diff --git a/guix/scripts/environment.scm b/guix/scripts/environment.scm
index e23d52df39..05a43659da 100644
--- a/guix/scripts/environment.scm
+++ b/guix/scripts/environment.scm
@@ -34,15 +34,18 @@
   #:use-module (guix scripts)
   #:use-module (guix scripts build)
   #:use-module (guix transformations)
-  #:use-module (gnu build linux-container)
-  #:use-module (gnu build accounts)
-  #:use-module ((guix build syscalls) #:select (set-network-interface-up))
-  #:use-module (gnu system linux-container)
+  #:autoload   (gnu build linux-container) (call-with-container %namespaces
+                                            user-namespace-supported?
+                                            unprivileged-user-namespace-supported?
+                                            setgroups-supported?)
+  #:autoload   (gnu build accounts) (password-entry group-entry
+                                     password-entry-name password-entry-directory
+                                     write-passwd write-group)
+  #:autoload   (guix build syscalls) (set-network-interface-up)
   #:use-module (gnu system file-systems)
-  #:use-module (gnu packages)
-  #:use-module (gnu packages bash)
-  #:use-module ((gnu packages bootstrap)
-                #:select (bootstrap-executable %bootstrap-guile))
+  #:autoload   (gnu packages) (specification->package+output)
+  #:autoload   (gnu packages bash) (bash)
+  #:autoload   (gnu packages bootstrap) (bootstrap-executable %bootstrap-guile)
   #:use-module (ice-9 match)
   #:use-module (srfi srfi-1)
   #:use-module (srfi srfi-11)
-- 
2.33.0

[bug#50960] [PATCH 09/10] cache: Gracefully handle non-existent cache.

[Ludovic Courtès]

* guix/cache.scm (maybe-remove-expired-cache-entries): Ignore ENOENT
when writing EXPIRY-FILE.
---
 guix/cache.scm | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)

diff --git a/guix/cache.scm b/guix/cache.scm
index 0401a9d428..51009809bd 100644
--- a/guix/cache.scm
+++ b/guix/cache.scm
@@ -101,7 +101,13 @@ CLEANUP-PERIOD denotes the minimum time between two cache cleanups."
                                   #:now now
                                   #:entry-expiration entry-expiration
                                   #:delete-entry delete-entry)
-    (call-with-output-file expiry-file
-      (cute write (time-second now) <>))))
+    (catch 'system-error
+      (lambda ()
+        (call-with-output-file expiry-file
+          (cute write (time-second now) <>)))
+      (lambda args
+        ;; ENOENT means CACHE does not exist.
+        (unless (= ENOENT (system-error-errno args))
+          (apply throw args))))))
 
 ;;; cache.scm ends here
-- 
2.33.0

[bug#50960] [PATCH 09/10] cache: Gracefully handle non-existent cache.

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 1248 bytes --]

Ludovic Courtès schreef op za 02-10-2021 om 12:22 [+0200]:
> * guix/cache.scm (maybe-remove-expired-cache-entries): Ignore ENOENT
> when writing EXPIRY-FILE.
> ---
>  guix/cache.scm | 10 ++++++++--
>  1 file changed, 8 insertions(+), 2 deletions(-)
> 
> diff --git a/guix/cache.scm b/guix/cache.scm
> index 0401a9d428..51009809bd 100644
> --- a/guix/cache.scm
> +++ b/guix/cache.scm
> @@ -101,7 +101,13 @@ CLEANUP-PERIOD denotes the minimum time between two cache cleanups."
>                                    #:now now
>                                    #:entry-expiration entry-expiration
>                                    #:delete-entry delete-entry)
> -    (call-with-output-file expiry-file
> -      (cute write (time-second now) <>))))
> +    (catch 'system-error
> +      (lambda ()
> +        (call-with-output-file expiry-file
> +          (cute write (time-second now) <>)))
> +      (lambda args
> +        ;; ENOENT means CACHE does not exist.
> +        (unless (= ENOENT (system-error-errno args))

And EROFS perhaps, such that "guix shell" works even if the root file system was
remounted read-only for some reason and all the necessary derivations have already
been built.

Greetings,
Maxime.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

[bug#50960] [PATCH 10/10] shell: Maintain a profile cache.

[Ludovic Courtès]

With this change, running "guix shell" (no arguments) is equivalent to:

  guix environment -r ~/.cache/guix/profiles/some-root -l guix.scm

This is the cache miss.  On cache hit, it's equivalent to:

  guix environment -p ~/.cache/guix/profiles/some-root

... which can run in 0.1s.

* guix/scripts/shell.scm (auto-detect-manifest): Looked for a cached GC
root to the profile and use it.
(%profile-cache-directory): New variable.
(profile-cache-key, profile-cached-gc-root): New procedures.
(guix-shell)[cache-entries, entry-expiration]: New procedures.
Add call to 'maybe-remove-expired-cache-entries'.
---
 guix/scripts/shell.scm | 90 +++++++++++++++++++++++++++++++++++++++---
 1 file changed, 84 insertions(+), 6 deletions(-)

diff --git a/guix/scripts/shell.scm b/guix/scripts/shell.scm
index 2f15befbd3..7c116cc770 100644
--- a/guix/scripts/shell.scm
+++ b/guix/scripts/shell.scm
@@ -29,6 +29,15 @@
   #:use-module (srfi srfi-37)
   #:use-module (srfi srfi-71)
   #:use-module (ice-9 match)
+  #:autoload   (guix base32) (bytevector->base32-string)
+  #:autoload   (rnrs bytevectors) (string->utf8)
+  #:autoload   (guix utils) (cache-directory)
+  #:autoload   (guix describe) (current-channels)
+  #:autoload   (guix channels) (channel-commit)
+  #:autoload   (gcrypt hash) (sha256)
+  #:use-module ((guix build utils) #:select (mkdir-p))
+  #:use-module (guix cache)
+  #:use-module ((ice-9 ftw) #:select (scandir))
   #:export (guix-shell))
 
 (define (show-help)
@@ -161,16 +170,85 @@ Return the modified OPTS."
          (warning (G_ "no packages specified; creating an empty environment~%"))
          opts)
         (file
+         ;; Load environment from FILE; if possible, use/maintain a GC root to
+         ;; the corresponding profile in cache.
          (info (G_ "loading environment from '~a'...~%") file)
-         (match (basename file)
-           ("guix.scm"
-            (alist-cons 'load `(package ,file) opts))
-           ("manifest.scm"
-            (alist-cons 'manifest file opts)))))))
+         (let* ((root (profile-cached-gc-root file))
+                (stat (and root (false-if-exception (lstat root)))))
+           (if (and stat
+                    (<= (stat:mtime ((@ (guile) stat) file))
+                        (stat:mtime stat)))
+               (let ((now (current-time)))
+                 ;; Update the atime on ROOT to reflect usage.
+                 (utime root
+                        now (stat:mtime stat)
+                        0 (stat:mtimensec stat)
+                        AT_SYMLINK_NOFOLLOW)
+                 (alist-cons 'profile root opts)) ;load right away
+               (let ((opts (match (basename file)
+                             ("guix.scm"
+                              (alist-cons 'load `(package ,file) opts))
+                             ("manifest.scm"
+                              (alist-cons 'manifest file opts)))))
+                 (if (and root (not (assq-ref opts 'gc-root)))
+                     (begin
+                       (if stat
+                           (delete-file root)
+                           (mkdir-p (dirname root)))
+                       (alist-cons 'gc-root root opts))
+                     opts))))))))
+
+\f
+;;;
+;;; Profile cache.
+;;;
+
+(define %profile-cache-directory
+  ;; Directory where profiles created by 'guix shell' alone (without extra
+  ;; options) are cached.
+  (make-parameter (string-append (cache-directory #:ensure? #f)
+                                 "/profiles")))
+
+(define (profile-cache-key file)
+  "Return the cache key for the profile corresponding to FILE, a 'guix.scm' or
+'manifest.scm' file, or #f if we lack channel information."
+  (match (current-channels)
+    (() #f)
+    (((= channel-commit commits) ...)
+     (let ((stat (stat file)))
+       (bytevector->base32-string
+        (sha256 (string->utf8
+                 (string-append (string-join commits) ":"
+                                (basename file) ":"
+                                (number->string (stat:dev stat)) ":"
+                                (number->string (stat:ino stat))))))))))
+
+(define (profile-cached-gc-root file)
+  "Return the cached GC root for FILE, a 'guix.scm' or 'manifest.scm' file, or
+#f if we lack information to cache it."
+  (match (profile-cache-key file)
+    (#f  #f)
+    (key (string-append (%profile-cache-directory) "/" key))))
 
 \f
 (define-command (guix-shell . args)
   (category development)
   (synopsis "spawn one-off software environments")
 
-  (guix-environment* (parse-args args)))
+  (define (cache-entries directory)
+    (filter-map (match-lambda
+                  ((or "." "..") #f)
+                  (file (string-append directory "/" file)))
+                (or (scandir directory) '())))
+
+  (define* (entry-expiration file)
+    ;; Return the time at which FILE, a cached profile, is considered expired.
+    (match (false-if-exception (lstat file))
+      (#f 0)                       ;FILE may have been deleted in the meantime
+      (st (+ (stat:atime st) (* 60 60 24 7)))))
+
+  (let ((result (guix-environment* (parse-args args))))
+    (maybe-remove-expired-cache-entries (%profile-cache-directory)
+                                        cache-entries
+                                        #:entry-expiration entry-expiration)
+    result))
-- 
2.33.0

[bug#50960] [PATCH 10/10] shell: Maintain a profile cache.

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 1301 bytes --]

Ludovic Courtès schreef op za 02-10-2021 om 12:22 [+0200]:
> With this change, running "guix shell" (no arguments) is equivalent to:
> 
>   guix environment -r ~/.cache/guix/profiles/some-root -l guix.scm
> 
> This is the cache miss.  On cache hit, it's equivalent to:
> 
>   guix environment -p ~/.cache/guix/profiles/some-root
> 

What if guix.scm is something like

;; Load custom package definitions
(include "a-package.scm")
(include "b-package.scm")
(include "c-package.scm")
(list a-package b-package c-package bash ...)

and a-package.scm, b-package.scm or c-package.scm is modified?
Then the cached profile should be rebuild, no?

I use something like that for my operating system definition
(though with -L and use-modules).  It would be nice if this worked
for "guix shell" as well.

So I think the cache should also check if these dependencies have been modified.
To keep track of the dependencies, something like the ‘compile-all,compile:
Keep track of dependencies of compiled modules.’ patch from
<https://issues.guix.gnu.org/50384> could be used, though the 'include', 'load',
'include-from-path' and maybe 'use-modules' (if something like "guix shell -Lextra-modules-directory ..."
is done) macros would need to be replaced.

Greetings,
Maxime.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

[bug#50960] [PATCH 10/10] shell: Maintain a profile cache.

[Ludovic Courtès]

Maxime Devos <maximedevos@telenet.be> skribis:

> Ludovic Courtès schreef op za 02-10-2021 om 12:22 [+0200]:
>> With this change, running "guix shell" (no arguments) is equivalent to:
>> 
>>   guix environment -r ~/.cache/guix/profiles/some-root -l guix.scm
>> 
>> This is the cache miss.  On cache hit, it's equivalent to:
>> 
>>   guix environment -p ~/.cache/guix/profiles/some-root
>> 
>
> What if guix.scm is something like
>
> ;; Load custom package definitions
> (include "a-package.scm")
> (include "b-package.scm")
> (include "c-package.scm")
> (list a-package b-package c-package bash ...)
>
> and a-package.scm, b-package.scm or c-package.scm is modified?
> Then the cached profile should be rebuild, no?

It should, but it won’t; that’s a weakness of this mtime-based caching
strategy.

I’m not sure how to address it.  We’d need a cache key that’s more
precise than inode/mtime, yet cheap to compute.

Perhaps we need to live with that limitation, document it, and provide a
flag to force a rebuild?

> So I think the cache should also check if these dependencies have been modified.
> To keep track of the dependencies, something like the ‘compile-all,compile:
> Keep track of dependencies of compiled modules.’ patch from
> <https://issues.guix.gnu.org/50384> could be used, though the 'include', 'load',
> 'include-from-path' and maybe 'use-modules' (if something like "guix shell -Lextra-modules-directory ..."
> is done) macros would need to be replaced.

Problem is that any attempt to keep track of dependencies is always an
approximation because macros can do anything—I can have my own macro
that reads files at expansion time.  So I’m inclined to not even try.

Thanks,
Ludo’.

[bug#50960] [PATCH 10/10] shell: Maintain a profile cache.

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 2663 bytes --]

Ludovic Courtès schreef op za 02-10-2021 om 16:12 [+0200]:
[reordered]
> > So I think the cache should also check if these dependencies have been modified.
> > To keep track of the dependencies, something like the ‘compile-all,compile:
> > Keep track of dependencies of compiled modules.’ patch from
> > <https://issues.guix.gnu.org/50384> could be used, though the 'include', 'load',
> > 'include-from-path' and maybe 'use-modules' (if something like "guix shell -Lextra-modules-directory ..."
> > is done) macros would need to be replaced.
> 
> Problem is that any attempt to keep track of dependencies is always an
> approximation because macros can do anything—I can have my own macro
> that reads files at expansion time.  So I’m inclined to not even try.

I expect most people use 'include', 'load', 'include-from-path' or 'load-from-path'
instead of writing their own macro reading files at expansion time, and if they do
write their own macro, I expect it would be implemented in terms of the former anyway,
so I expect replacing these macros and 'use-modules' to be sufficient, especially
if the flag proposed below is available as ‘escape hatch’ for when the dependency
tracking is insufficient.

This is not merely an expansion-time problem, files loaded at 'load' or 'eval' time
are important as well.  E.g., consider the case where the manifest is something like

  (list
    (package
      (inherit stuff)
      (source (local-file "stuff" #:recursive? #t)))
    other-packages ...)

then all files inside 'stuff' are important as well.  I don't see how dependencies
could be tracked here without an excessive amount of 'stat' calls, maybe guix should
ignore these dependencies (possibly with a warning, and a reference to the manual
documenting which dependencies are tracked and which are not?). 

Ludovic Courtès schreef op za 02-10-2021 om 16:12 [+0200]:
> I’m not sure how to address it.  We’d need a cache key that’s more
> precise than inode/mtime, yet cheap to compute.

The mtime of the file and the mtime of every dependency (ignoring modules from
Guile, Guix and channels to reduce the number of calls to 'stat'). 

> Perhaps we need to live with that limitation, document it, and provide a
> flag to force a rebuild?

A documented flag to always consider the cache stale seems good, though I think
at least the dependencies made with the common macros and procedures 'include',
'load', 'include-from-path', 'load-from-path', 'use-modules' and non-recursive
'local-file' could be tracked, though this could be left as a TODO for later
I suppose.

Greetings,
Maxime.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

[bug#50960] [PATCH 10/10] shell: Maintain a profile cache.

[Ludovic Courtès]

Maxime Devos <maximedevos@telenet.be> skribis:

> Ludovic Courtès schreef op za 02-10-2021 om 16:12 [+0200]:
> [reordered]
>> > So I think the cache should also check if these dependencies have been modified.
>> > To keep track of the dependencies, something like the ‘compile-all,compile:
>> > Keep track of dependencies of compiled modules.’ patch from
>> > <https://issues.guix.gnu.org/50384> could be used, though the 'include', 'load',
>> > 'include-from-path' and maybe 'use-modules' (if something like "guix shell -Lextra-modules-directory ..."
>> > is done) macros would need to be replaced.
>> 
>> Problem is that any attempt to keep track of dependencies is always an
>> approximation because macros can do anything—I can have my own macro
>> that reads files at expansion time.  So I’m inclined to not even try.
>
> I expect most people use 'include', 'load', 'include-from-path' or 'load-from-path'
> instead of writing their own macro reading files at expansion time, and if they do
> write their own macro, I expect it would be implemented in terms of the former anyway,
> so I expect replacing these macros and 'use-modules' to be sufficient, especially
> if the flag proposed below is available as ‘escape hatch’ for when the dependency
> tracking is insufficient.
>
> This is not merely an expansion-time problem, files loaded at 'load' or 'eval' time
> are important as well.  E.g., consider the case where the manifest is something like
>
>   (list
>     (package
>       (inherit stuff)
>       (source (local-file "stuff" #:recursive? #t)))
>     other-packages ...)

In a ‘guix.scm’ file, ‘source’ doesn’t matter.  It does matter in a
manifest, but that sounds a bit less common.

> then all files inside 'stuff' are important as well.  I don't see how dependencies
> could be tracked here without an excessive amount of 'stat' calls, maybe guix should
> ignore these dependencies (possibly with a warning, and a reference to the manual
> documenting which dependencies are tracked and which are not?). 

[...]

> A documented flag to always consider the cache stale seems good, though I think
> at least the dependencies made with the common macros and procedures 'include',
> 'load', 'include-from-path', 'load-from-path', 'use-modules' and non-recursive
> 'local-file' could be tracked, though this could be left as a TODO for later
> I suppose.

Tracking those uses reliably is impossible: there could be same-named
bindings that do other things, there could be custom macros, there could
be “dynamic arguments” (whose value is not known statically), etc.  You
have to expand + evaluate the code to get better results, and even then,
there might be different paths in the code so you can’t be sure you got
it right.

We could get an approximation for common uses by recognizing special
forms as you suggest.  But it’s just that, an approximation.

In such situations, I err on the side of not even trying.  The added
complexity for a flaky result doesn’t pay off to me.  I prefer to be
upfront, document limitations, and let users handle them as they see
fit.

WDYT?

A similar problem occurs with system provenance tracking, which saves
‘configuration.scm’ but leaves it up to the user to preserve additional
files if needed (info "(guix) Service Reference").

Thanks,
Ludo’.

[bug#50960] [PATCH 10/10] shell: Maintain a profile cache.

[zimoun]

Hi,

On Mon, 4 Oct 2021 at 10:52, Ludovic Courtès <ludo@gnu.org> wrote:

> In such situations, I err on the side of not even trying.  The added
> complexity for a flaky result doesn’t pay off to me.  I prefer to be
> upfront, document limitations, and let users handle them as they see
> fit.

I agree.  Just a flag for forcing seems be worth here.  For this
dependencies use-case or simply for debugging.

Cheers,
simon

[bug#50960] [PATCH 10/10] shell: Maintain a profile cache.

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 4429 bytes --]

Ludovic Courtès schreef op ma 04-10-2021 om 10:19 [+0200]:
> > A documented flag to always consider the cache stale seems good, though I think
> > at least the dependencies made with the common macros and procedures 'include',
> > 'load', 'include-from-path', 'load-from-path', 'use-modules' and non-recursive
> > 'local-file' could be tracked, though this could be left as a TODO for later
> > I suppose.
> 
> Tracking those uses reliably is impossible: there could be same-named
> bindings that do other things, there could be custom macros, there could
> be “dynamic arguments” (whose value is not known statically), etc.  You
> have to expand + evaluate the code to get better results, and even then,
> there might be different paths in the code so you can’t be sure you got
> it right.

I think there's a miscommunication here.  From what I'm reading, what you have
in mind is that, to determine the dependency information, "guix shell" would
open "guix.scm", read it with the procedure 'read' and sniff for 'load',
'include-from-path', 'load-from-path', 'use-modules' and 'local-file' form
-- something like 'module-file-dependencies' and 'extract-dependencies', but
more general.

However, my idea is to replace these macros, such that, when "guix shell"
loads "guix.scm" or "manifest.scm", these macros inform "guix shell"
that "guix.scm" or "manifest.scm" depend on certain files referred to
by 'load', 'include-from-path', etc. forms, using a mechanism like the
'notice-dependency' defined in <https://issues.guix.gnu.org/50384>.

Then, when "guix shell" puts the resulting profile in the cache,
it includes the generated list of files.  And when "guix shell" finds
an entry in the cache, it will check if the files in this list (and guix.scm
or manifest.scm of course) have been modified.  If some are (or the forcing
flag is set), guix.scm needs to be loaded and a new profile needs to be
generated.  If they are all unchanged, guix.scm will _not_ be read: the cached
profile can be reused.

It's not 100% reliable (e.g. the list of packages in the manifest could
depend on the phase of the moon if (current-time) is used) (is that what
you mean by ‘different paths’ and ‘dynamic arguments’?), but it should
cover the vast majority of cases.

I don't know a non-artifical situation where ‘custom macros’ are a problem
-- do you know an example in the wild where this dependency tracking scheme
would fail to work?

> We could get an approximation for common uses by recognizing special
> forms as you suggest.  But it’s just that, an approximation.

It's an approximation, sure, but it seems to be a quite accurate approximation
to me.  And it's not really recognising special forms that I'm suggesting,
but rather modifying the macros behind these forms to inform "guix shell"
of what the dependencies are.

> In such situations, I err on the side of not even trying.  The added
> complexity for a flaky result doesn’t pay off to me.  I prefer to be
> upfront, document limitations, and let users handle them as they see
> fit.

About complexity: there's some extra code, sure, but it doesn't seem complex
to me.  To track dependencies in <https://issues.guix.gnu.org/50384>,
I only needed to add 'notice-dependency' and some other code to (guix build compile),
and some code to build-aux/compile-all.scm to save/load the dependency information
to/from the builddir and to also check the mtime of dependencies.

Does it still seem flaky to you, after my explanation on how the dependency
information would be determined?

Being upfront, documenting limitations, and having a ‘force rebuild flag’
(is that what you mean by ‘letting users handle them as they see fit’?)
(possibly also documenting 'notice-dependency') is not incompatible with
the automatic dependency tracking.

More abstractly, this seems more like a ‘Perfect is the enemy of the good’ situation.
While I would prefer 'perfect' above 'good', and the automatic dependency tracking
certainly isn't 'perfect', it does seem 'good' to me, and perfection appears to
be impossible and there's the ‘force rebuild flag’ as an escape hatch, so I believe
'good-but-not-perfect' to be acceptable here, as long as it is documented in the manual
that there are some limitation on what dependencies are automatically tracked.

Greetings,
Maxime.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

[bug#50960] [PATCH 10/10] shell: Maintain a profile cache.

[Ludovic Courtès]

Hi Maxime,

Maxime Devos <maximedevos@telenet.be> skribis:

> Ludovic Courtès schreef op ma 04-10-2021 om 10:19 [+0200]:
>> > A documented flag to always consider the cache stale seems good, though I think
>> > at least the dependencies made with the common macros and procedures 'include',
>> > 'load', 'include-from-path', 'load-from-path', 'use-modules' and non-recursive
>> > 'local-file' could be tracked, though this could be left as a TODO for later
>> > I suppose.
>> 
>> Tracking those uses reliably is impossible: there could be same-named
>> bindings that do other things, there could be custom macros, there could
>> be “dynamic arguments” (whose value is not known statically), etc.  You
>> have to expand + evaluate the code to get better results, and even then,
>> there might be different paths in the code so you can’t be sure you got
>> it right.
>
> I think there's a miscommunication here.  From what I'm reading, what you have
> in mind is that, to determine the dependency information, "guix shell" would
> open "guix.scm", read it with the procedure 'read' and sniff for 'load',
> 'include-from-path', 'load-from-path', 'use-modules' and 'local-file' form
> -- something like 'module-file-dependencies' and 'extract-dependencies', but
> more general.

Yes, that’s roughly what I thought you had in mind, sorry!

> However, my idea is to replace these macros, such that, when "guix shell"
> loads "guix.scm" or "manifest.scm", these macros inform "guix shell"
> that "guix.scm" or "manifest.scm" depend on certain files referred to
> by 'load', 'include-from-path', etc. forms, using a mechanism like the
> 'notice-dependency' defined in <https://issues.guix.gnu.org/50384>.
>
> Then, when "guix shell" puts the resulting profile in the cache,
> it includes the generated list of files.  And when "guix shell" finds
> an entry in the cache, it will check if the files in this list (and guix.scm
> or manifest.scm of course) have been modified.  If some are (or the forcing
> flag is set), guix.scm needs to be loaded and a new profile needs to be
> generated.  If they are all unchanged, guix.scm will _not_ be read: the cached
> profile can be reused.
>
> It's not 100% reliable (e.g. the list of packages in the manifest could
> depend on the phase of the moon if (current-time) is used) (is that what
> you mean by ‘different paths’ and ‘dynamic arguments’?), but it should
> cover the vast majority of cases.
>
> I don't know a non-artifical situation where ‘custom macros’ are a problem
> -- do you know an example in the wild where this dependency tracking scheme
> would fail to work?

I think we have to look at the cost and at the benefits.  The cost:
either we spread ‘notice-dependency’ calls all over the place, or we
interpose on ‘open-file’ altogether, assuming that even works.  The
benefit: we’d have an 80% solution.  There would still be edge cases not
handled correctly; a realistic example is a ‘guix.scm’ whose execution
differs based on the presence of a file or environment variable.  People
would have to know about this pitfall and pass the right flag.

[...]

>> In such situations, I err on the side of not even trying.  The added
>> complexity for a flaky result doesn’t pay off to me.  I prefer to be
>> upfront, document limitations, and let users handle them as they see
>> fit.
>
> About complexity: there's some extra code, sure, but it doesn't seem complex
> to me.  To track dependencies in <https://issues.guix.gnu.org/50384>,
> I only needed to add 'notice-dependency' and some other code to (guix build compile),
> and some code to build-aux/compile-all.scm to save/load the dependency information
> to/from the builddir and to also check the mtime of dependencies.
>
> Does it still seem flaky to you, after my explanation on how the dependency
> information would be determined?
>
> Being upfront, documenting limitations, and having a ‘force rebuild flag’
> (is that what you mean by ‘letting users handle them as they see fit’?)
> (possibly also documenting 'notice-dependency') is not incompatible with
> the automatic dependency tracking.
>
> More abstractly, this seems more like a ‘Perfect is the enemy of the good’ situation.
> While I would prefer 'perfect' above 'good', and the automatic dependency tracking
> certainly isn't 'perfect', it does seem 'good' to me, and perfection appears to
> be impossible and there's the ‘force rebuild flag’ as an escape hatch, so I believe
> 'good-but-not-perfect' to be acceptable here, as long as it is documented in the manual
> that there are some limitation on what dependencies are automatically tracked.

Yeah, I can relate to that sentiment, even if I’m all too often on the
“perfectionist” side of things.  ;-)

I guess I could live with an 80% dependency tracking solution.  However,
my criteria for it would be that (1) it should be orthogonal (e.g., not
requiring explicit calls to ‘notice-dependency’ in many places), and (2)
it should be self-contained and reasonably small.  Perhaps having
‘load*’ or similar instrument ‘open-file’ or similar could help achieve
that?

I would rather not block ‘guix shell’ on that though.

WDYT?

Thanks for taking the time to clarify what you had in mind!

Ludo’.

[bug#50960] [PATCH 10/10] shell: Maintain a profile cache.

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 1209 bytes --]

Ludovic Courtès schreef op za 02-10-2021 om 12:22 [+0200]:
> +(define (profile-cache-key file)
> +  "Return the cache key for the profile corresponding to FILE, a 'guix.scm' or
> +'manifest.scm' file, or #f if we lack channel information."
> +  (match (current-channels)
> +    (() #f)
> +    (((= channel-commit commits) ...)
> +     (let ((stat (stat file)))
> +       (bytevector->base32-string
> +        (sha256 (string->utf8
> +                 (string-append (string-join commits) ":"
> +                                (basename file) ":"
> +                                (number->string (stat:dev stat)) ":"
> +                                (number->string (stat:ino stat))))))))))

Why only use the 'basename' of a file name instead of the full name?
(Consider the case where a user has multiple "guix.scm" or "manifest.scm".)
This turns out to be unproblematic, because stat:dev and stat:ino is included
as well, though including (a part of) the file name is superfluous because
stat:dev and stat:ino are included.

Could you document the rationale for including the file name, and why only
the basename is included instead of the full file name?

Greetings,
Maxime.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

[bug#50960] [PATCH 10/10] shell: Maintain a profile cache.

[Ludovic Courtès]

Maxime Devos <maximedevos@telenet.be> skribis:

> Ludovic Courtès schreef op za 02-10-2021 om 12:22 [+0200]:
>> +(define (profile-cache-key file)
>> +  "Return the cache key for the profile corresponding to FILE, a 'guix.scm' or
>> +'manifest.scm' file, or #f if we lack channel information."
>> +  (match (current-channels)
>> +    (() #f)
>> +    (((= channel-commit commits) ...)
>> +     (let ((stat (stat file)))
>> +       (bytevector->base32-string
>> +        (sha256 (string->utf8
>> +                 (string-append (string-join commits) ":"
>> +                                (basename file) ":"
>> +                                (number->string (stat:dev stat)) ":"
>> +                                (number->string (stat:ino stat))))))))))
>
> Why only use the 'basename' of a file name instead of the full name?
> (Consider the case where a user has multiple "guix.scm" or "manifest.scm".)
> This turns out to be unproblematic, because stat:dev and stat:ino is included
> as well, though including (a part of) the file name is superfluous because
> stat:dev and stat:ino are included.
>
> Could you document the rationale for including the file name, and why only
> the basename is included instead of the full file name?

Actually it’s probably not useful to include the file (base)name.  I
think initially I thought about distinguishing between guix.scm and
manifest.scm, since they return different kinds of objects, but dev/ino
is probably enough.  WDYT?

Ludo’.

[bug#50960] [PATCH 10/10] shell: Maintain a profile cache.

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 1334 bytes --]

Ludovic Courtès schreef op za 02-10-2021 om 16:14 [+0200]:
> > Why only use the 'basename' of a file name instead of the full name?
> > (Consider the case where a user has multiple "guix.scm" or "manifest.scm".)
> > This turns out to be unproblematic, because stat:dev and stat:ino is included
> > as well, though including (a part of) the file name is superfluous because
> > stat:dev and stat:ino are included.
> > 
> > Could you document the rationale for including the file name, and why only
> > the basename is included instead of the full file name?
> 
> Actually it’s probably not useful to include the file (base)name.  I
> think initially I thought about distinguishing between guix.scm and
> manifest.scm, since they return different kinds of objects, but dev/ino
> is probably enough.  WDYT?

Looking at https://lwn.net/Articles/866582/, it appears when BTRFS and NFS are
combined, it is possible for two dev/ino pairs to be the same for different files.
It appears to be considered a bug but non-trivial to fix.  Thus, dev/ino apparently
is not always sufficient, so it may be reasonable to include the file name.

I would include the full file name, because the basename is often rather generic
("guix.scm", "manifest.scm"), though this depends on the habits of the user.

Greetings,
Maxime.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

[bug#50960] [PATCH 10/10] shell: Maintain a profile cache.

[Ludovic Courtès]

Maxime Devos <maximedevos@telenet.be> skribis:

> Ludovic Courtès schreef op za 02-10-2021 om 16:14 [+0200]:
>> > Why only use the 'basename' of a file name instead of the full name?
>> > (Consider the case where a user has multiple "guix.scm" or "manifest.scm".)
>> > This turns out to be unproblematic, because stat:dev and stat:ino is included
>> > as well, though including (a part of) the file name is superfluous because
>> > stat:dev and stat:ino are included.
>> > 
>> > Could you document the rationale for including the file name, and why only
>> > the basename is included instead of the full file name?
>> 
>> Actually it’s probably not useful to include the file (base)name.  I
>> think initially I thought about distinguishing between guix.scm and
>> manifest.scm, since they return different kinds of objects, but dev/ino
>> is probably enough.  WDYT?
>
> Looking at https://lwn.net/Articles/866582/, it appears when BTRFS and NFS are
> combined, it is possible for two dev/ino pairs to be the same for different files.
> It appears to be considered a bug but non-trivial to fix.  Thus, dev/ino apparently
> is not always sufficient, so it may be reasonable to include the file name.
>
> I would include the full file name, because the basename is often rather generic
> ("guix.scm", "manifest.scm"), though this depends on the habits of the user.

OK, that makes sense to me, let’s do that.

Ludo’.

[bug#50960] [PATCH 00/10] Add 'guix shell' to subsume 'guix environment'

[Jelle Licht]

Ludovic Courtès <ludo@gnu.org> writes:

> Hello Guix!
>
> Here comes ‘guix shell’, a proposed replacement for ‘guix environment’!
Suddenly thousands(/dozens?) of shell scripts fear for their continued
existence :-)

>   2. ‘guix shell’, without arguments, loads ‘guix.scm’ or ‘manifest.scm’
>      from the current directory or one of its ancestors.
<snip>
>   4. ‘guix shell’ without arguments maintains a cache, such that, the
>      second time you run it, it runs in ~0.1s (it does not even need to
>      connect to the daemon).
>
>      If you run ‘guix pull’ and run again ‘guix shell’, it recomputes
>      the environment, as is currently the case with ‘guix environment’.
<snip>
> Thoughts?  Are there other changes people would like to see?
>
> If there’s rough consensus I can work on v2 with documentation.  Please
> let’s keep the discussion focused.  :-)

At the risk of doing exactly not that; since it will(/might) already
provide some automagic conveniences, perhaps it makes sense to
additionally load a guix-channels.scm.

I understand that there would be some duplication of functionality
w.r.t. ‘guix time-machine -C guix-channels.scm -- shell ...’, but if we
already go the DWIW-route, why not go all the way? I think this should
only apply when running ‘guix shell’ without arguments, if that was
unclear.

Love the proposal
- Jelle

[bug#50960] [PATCH 00/10] Add 'guix shell' to subsume 'guix environment'

[Ludovic Courtès]

Hi!

Jelle Licht <jlicht@fsfe.org> skribis:

> At the risk of doing exactly not that; since it will(/might) already
> provide some automagic conveniences, perhaps it makes sense to
> additionally load a guix-channels.scm.
>
> I understand that there would be some duplication of functionality
> w.r.t. ‘guix time-machine -C guix-channels.scm -- shell ...’, but if we
> already go the DWIW-route, why not go all the way? I think this should
> only apply when running ‘guix shell’ without arguments, if that was
> unclear.

Oh, it’s tempting yeah, but…  (1) only ‘time-machine’ and ‘pull’ deal
with channels currently, so it’d feel kind of weird to take care of that
here, and (2) pulling a specific channel is resource-intensive as you
know, much more than anything else, so I’d rather not have that happen
automatically.

But yeah, I agree that it could be useful.  Maybe a first step we could
make is have ‘time-machine’ load ‘channels.scm’, such that those who
want it can type:

  guix time-machine -- shell

?

Ludo’.

[bug#50960] [PATCH 00/10] Add 'guix shell' to subsume 'guix environment'

[pelzflorian (Florian Pelz)]

guix shell is very interesting, but:

On Sat, Oct 02, 2021 at 12:21:16PM +0200, Ludovic Courtès wrote:
>   2. ‘guix shell’, without arguments, loads ‘guix.scm’ or ‘manifest.scm’
>      from the current directory or one of its ancestors.

This however is concerning.  Users will not expect guix to execute
arbitrary code.  Maybe print a suggestion to maybe --file the file
instead.

Maybe I should not have such expectations but IMHO Guix is not like
`haunt build`, which is expected to load haunt.scm.

Regards,
Florian

[bug#50960] [PATCH 00/10] Add 'guix shell' to subsume 'guix environment'

[Ludovic Courtès]

Hi,

"pelzflorian (Florian Pelz)" <pelzflorian@pelzflorian.de> skribis:

> On Sat, Oct 02, 2021 at 12:21:16PM +0200, Ludovic Courtès wrote:
>>   2. ‘guix shell’, without arguments, loads ‘guix.scm’ or ‘manifest.scm’
>>      from the current directory or one of its ancestors.
>
> This however is concerning.  Users will not expect guix to execute
> arbitrary code.  Maybe print a suggestion to maybe --file the file
> instead.
>
> Maybe I should not have such expectations but IMHO Guix is not like
> `haunt build`, which is expected to load haunt.scm.

I think it’s fine as long as, as in the case of ‘haunt build’ or ‘make’
or ‘git’, it’s properly documented.  Also, ‘guix shell’ unconditionally
writes a message.

Ludo’.

[bug#50960] [PATCH 00/10] Add 'guix shell' to subsume 'guix environment'

[pelzflorian (Florian Pelz)]

On Sat, Oct 02, 2021 at 03:40:00PM +0200, Ludovic Courtès wrote:
> "pelzflorian (Florian Pelz)" <pelzflorian@pelzflorian.de> skribis:
> > On Sat, Oct 02, 2021 at 12:21:16PM +0200, Ludovic Courtès wrote:
> >> 2. ‘guix shell’, without arguments, loads ‘guix.scm’ or ‘manifest.scm’
> >>    from the current directory or one of its ancestors.
> > This however is concerning.  Users will not expect guix to execute
> > arbitrary code.  Maybe print a suggestion to maybe --file the file
> > instead.
> I think it’s fine as long as, as in the case of ‘haunt build’ or ‘make’
> or ‘git’, it’s properly documented.  Also, ‘guix shell’ unconditionally
> writes a message.

Let’s say I have downloaded undesirable code to a file
/home/florian/Downloads/guix.scm and am hacking on source code in
/home/florian/Downloads/something/ where I run `guix shell`, but
/home/florian/Downloads/something/ does not in fact contain a
guix.scm file.  Now I’d have accidentally run the other guix.scm.

Also `make` is typically used without arguments, but a novice `guix
shell` user might know `guix shell program-a program-b` but is
surprised when running `guix shell` without arguments in an untrusted
directory.

But yes, git hooks are dangerous too.

Regards,
Florian

[bug#50960] [PATCH 00/10] Add 'guix shell' to subsume 'guix environment'

[Ludovic Courtès]

Hi,

"pelzflorian (Florian Pelz)" <pelzflorian@pelzflorian.de> skribis:

> On Sat, Oct 02, 2021 at 03:40:00PM +0200, Ludovic Courtès wrote:
>> "pelzflorian (Florian Pelz)" <pelzflorian@pelzflorian.de> skribis:
>> > On Sat, Oct 02, 2021 at 12:21:16PM +0200, Ludovic Courtès wrote:
>> >> 2. ‘guix shell’, without arguments, loads ‘guix.scm’ or ‘manifest.scm’
>> >>    from the current directory or one of its ancestors.
>> > This however is concerning.  Users will not expect guix to execute
>> > arbitrary code.  Maybe print a suggestion to maybe --file the file
>> > instead.
>> I think it’s fine as long as, as in the case of ‘haunt build’ or ‘make’
>> or ‘git’, it’s properly documented.  Also, ‘guix shell’ unconditionally
>> writes a message.
>
> Let’s say I have downloaded undesirable code to a file
> /home/florian/Downloads/guix.scm and am hacking on source code in
> /home/florian/Downloads/something/ where I run `guix shell`, but
> /home/florian/Downloads/something/ does not in fact contain a
> guix.scm file.  Now I’d have accidentally run the other guix.scm.

Sure, but it’s all under your control; it’s not very different from
someone knowingly running “guix build -f guix.scm” on an untrusted file,
is it?

> Also `make` is typically used without arguments, but a novice `guix
> shell` user might know `guix shell program-a program-b` but is
> surprised when running `guix shell` without arguments in an untrusted
> directory.

We have the advantage that ‘guix shell’ is a new command, so we can
document it from the start as behaving this way without arguments.

WDYT?

Thanks,
Ludo’.

[bug#50960] [PATCH 00/10] Add 'guix shell' to subsume 'guix environment'

[pelzflorian (Florian Pelz)]

On Mon, Oct 04, 2021 at 10:22:54AM +0200, Ludovic Courtès wrote:
> "pelzflorian (Florian Pelz)" <pelzflorian@pelzflorian.de> skribis:
> > Let’s say I have downloaded undesirable code to a file
> > /home/florian/Downloads/guix.scm and am hacking on source code in
> > /home/florian/Downloads/something/ where I run `guix shell`, but
> > /home/florian/Downloads/something/ does not in fact contain a
> > guix.scm file.  Now I’d have accidentally run the other guix.scm.
> 
> Sure, but it’s all under your control; it’s not very different from
> someone knowingly running “guix build -f guix.scm” on an untrusted file,
> is it?

What I meant is that I may wrongly expect a guix.scm file in
/home/florian/Downloads/something/, but it is not there, so things go
awry.

`guix shell` loading files by default would mean one would have to pay
attention to what one is doing, unlike `guix environment`.  For
example, not save unrelated (not even malicious) code by the name
guix.scm, and not run guix commands without inspecting what they’d do.
This I don’t like.



> We have the advantage that ‘guix shell’ is a new command, so we can
> document it from the start as behaving this way without arguments.

Many people don’t read manuals.

The probability of an accident is low, but it feels not robust.
I can live with either (and am very happy you and others keep
improving Guix), I just don’t think loading by default is a good idea.

Regards,
Florian

[bug#50960] [PATCH 00/10] Add 'guix shell' to subsume 'guix environment'

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 3077 bytes --]

Ludovic Courtès schreef op ma 04-10-2021 om 10:22 [+0200]:
> Hi,
> 
> "pelzflorian (Florian Pelz)" <pelzflorian@pelzflorian.de> skribis:
> 
> > On Sat, Oct 02, 2021 at 03:40:00PM +0200, Ludovic Courtès wrote:
> > > "pelzflorian (Florian Pelz)" <pelzflorian@pelzflorian.de> skribis:
> > > > On Sat, Oct 02, 2021 at 12:21:16PM +0200, Ludovic Courtès wrote:
> > > > > 2. ‘guix shell’, without arguments, loads ‘guix.scm’ or ‘manifest.scm’
> > > > >    from the current directory or one of its ancestors.
> > > > This however is concerning.  Users will not expect guix to execute
> > > > arbitrary code.  Maybe print a suggestion to maybe --file the file
> > > > instead.
> > > I think it’s fine as long as, as in the case of ‘haunt build’ or ‘make’
> > > or ‘git’, it’s properly documented.  Also, ‘guix shell’ unconditionally
> > > writes a message.
> > 
> > Let’s say I have downloaded undesirable code to a file
> > /home/florian/Downloads/guix.scm and am hacking on source code in
> > /home/florian/Downloads/something/ where I run `guix shell`, but
> > /home/florian/Downloads/something/ does not in fact contain a
> > guix.scm file.  Now I’d have accidentally run the other guix.scm.
> 
> Sure, but it’s all under your control; it’s not very different from
> someone knowingly running “guix build -f guix.scm” on an untrusted file,
> is it?

Consider the following situation:

  1. I browse the web and find some rando's website.  It has a link to a "guix.scm" to download.
  2. I'd like to know how people are using guix, so I tell IceCat to download it.
     IceCat downloads it to ~/Downloads/guix.scm.
  3. I forget about the guix.scm and didn't look at it.
  4. I download some tarball, verify it (with gpg or something), unpack it,
     and run "guix shell" without arguments from within the directory
     (e.g. ~/Downloads/some-source-code).
  5. It turns out the tarball didn't actually have a guix.scm, so the
     ~/Downloads/guix.scm from the rando is loaded.
  6. It turns out the rando's guix.scm uploads my secret keys, passwords,
     all e-mails, installs a keylogger ... Oops!

> > Also `make` is typically used without arguments, but a novice `guix
> > shell` user might know `guix shell program-a program-b` but is
> > surprised when running `guix shell` without arguments in an untrusted
> > directory.
> 
> We have the advantage that ‘guix shell’ is a new command, so we can
> document it from the start as behaving this way without arguments.

Sure, this behaviour can be documented, but it's very easy to forget a piece
of documentation, especially if the behaviour is inconsistent between "guix environment"
and "guix shell", and an attacker only needs an attack to function once.

I'd prefer not be constantly kept on my toes, so if "guix shell" will automatically
load guix.scm in the current directory or parent directories, I think I'll
keep using "guix environment" to avoid any opportunities for fatal mistakes.

Greetings,
Maximes.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

[bug#50960] [PATCH 00/10] Add 'guix shell' to subsume 'guix environment'

[Christine Lemmer-Webber]

Ludovic Courtès <ludo@gnu.org> writes:

> Hello Guix!
>
> Here comes ‘guix shell’, a proposed replacement for ‘guix environment’!
> ‘guix environment’ would stay around though, at least for some time,
> probably for a long time.
>
> The differences to ‘guix environment’ are:
>
>   1. ‘--ad-hoc’ is the default.
>
>        ‘guix shell hello’ ≍ ‘guix environment --ad-hoc hello’
>        ‘guix shell -D hello git’ ≍ ‘guix environment hello --ad-hoc git’
>
>   2. ‘guix shell’, without arguments, loads ‘guix.scm’ or ‘manifest.scm’
>      from the current directory or one of its ancestors.
>
>   3. ‘--load’/‘-l’ is not ‘-f’/‘--install-from-file’ for consistency with
>      ‘guix package’.
>
>   4. ‘guix shell’ without arguments maintains a cache, such that, the
>      second time you run it, it runs in ~0.1s (it does not even need to
>      connect to the daemon).
>
>      If you run ‘guix pull’ and run again ‘guix shell’, it recomputes
>      the environment, as is currently the case with ‘guix environment’.
>
> Here’s a summary of previous proposals:
>
>   - Dave Thompson: https://lists.gnu.org/archive/html/guix-devel/2017-08/msg00300.html
>     - [X] --ad-hoc is the default
>     - [X] caching
>     - [X] behavior with no arguments
>     - [ ] --load accepts <environment>
>     - [ ] Shepherd services
>     - [ ] 'guix environment --update' to explicitly update
>   - make --ad-hoc the default: https://issues.guix.gnu.org/38529
>     - [X] https://issues.guix.gnu.org/38529#17: proposal for a new subcommand
>           deprecation of ‘guix environment’
>
> I think <environment> records and Shepherd services could come later.
> As for ‘--update’, I prefer the behavior implemented here because it’s
> stateless and thus more predictable.
>
> Thoughts?  Are there other changes people would like to see?
>
> If there’s rough consensus I can work on v2 with documentation.  Please
> let’s keep the discussion focused.  :-)

You have my massive thumbs up.  Exciting stuff.

> As for deprecation, I think there’s no rush.  I imagine there could be
> several phases, like: initially we only mention deprecation in the manual,
> later on ‘guix environment’ starts emitting a warning, and later (I guess
> at least two years later, probably more) we ask ourselves whether to
> remove ‘guix environment’.  At this point keeping it doesn’t cost us much.
>
> Thanks,
> Ludo’.
>
> Ludovic Courtès (10):
>   packages: Add 'package-development-inputs'.
>   profiles: Add 'package->development-manifest'.
>   DRAFT Add 'guix shell'.
>   DRAFT shell: By default load the local 'guix.scm' or 'manifest.scm'
>     file.
>   environment: Add tests for '--profile'.
>   environment: Skip derivation computation when '--profile' is used.
>   environment: Do not connect to the daemon when '--profile' is used.
>   environment: Autoload some modules.
>   cache: Gracefully handle non-existent cache.
>   shell: Maintain a profile cache.
>
>  Makefile.am                         |   2 +
>  doc/guix.texi                       |  52 ++++++
>  guix/cache.scm                      |  10 +-
>  guix/packages.scm                   |  10 ++
>  guix/profiles.scm                   |  19 ++
>  guix/scripts/environment.scm        | 260 +++++++++++++++-------------
>  guix/scripts/shell.scm              | 254 +++++++++++++++++++++++++++
>  po/guix/POTFILES.in                 |   1 +
>  tests/guix-environment-container.sh |   8 +
>  tests/guix-environment.sh           |   7 +
>  tests/guix-shell.sh                 |  70 ++++++++
>  tests/packages.scm                  |  14 ++
>  tests/profiles.scm                  |   7 +
>  13 files changed, 594 insertions(+), 120 deletions(-)
>  create mode 100644 guix/scripts/shell.scm
>  create mode 100644 tests/guix-shell.sh

[bug#50960] ‘guix shell’ shebangs

[Ludovic Courtès]

Ludovic Courtès <ludo@gnu.org> skribis:

> The differences to ‘guix environment’ are:
>
>   1. ‘--ad-hoc’ is the default.
>
>        ‘guix shell hello’ ≍ ‘guix environment --ad-hoc hello’
>        ‘guix shell -D hello git’ ≍ ‘guix environment hello --ad-hoc git’
>
>   2. ‘guix shell’, without arguments, loads ‘guix.scm’ or ‘manifest.scm’
>      from the current directory or one of its ancestors.
>
>   3. ‘--load’/‘-l’ is not ‘-f’/‘--install-from-file’ for consistency with
>      ‘guix package’.
>
>   4. ‘guix shell’ without arguments maintains a cache, such that, the
>      second time you run it, it runs in ~0.1s (it does not even need to
>      connect to the daemon).
>
>      If you run ‘guix pull’ and run again ‘guix shell’, it recomputes
>      the environment, as is currently the case with ‘guix environment’.

Oh, I had another goal in mind, which was to make it easy to use ‘guix
shell’ in shebangs.

Problem is, since it’s called ‘guix shell’ and not ‘guix-shell’, we have
to use ‘/usr/bin/env -S’ anyway as the shebang so it splits arguments.
And with ‘-S’, we can already do pretty much everything we want.  Here’s
a shell script:

--8<---------------cut here---------------start------------->8---
$ cat t.sh
#!/usr/bin/env -S guix shell hello bash -- sh
type -P hello
hello
--8<---------------cut here---------------end--------------->8---

Should we simply document it or should we do something more?

For the record, ‘nix-shell’ has shebang support where it can interpret
“special” lines as arguments:

  https://nixos.org/manual/nix/stable/#sec-nix-shell

Ludo’.

[bug#50960] ‘guix shell’ shebangs

[Katherine Cox-Buday]

Ludovic Courtès <ludo@gnu.org> writes:

> Oh, I had another goal in mind, which was to make it easy to use ‘guix
> shell’ in shebangs.
>
> Problem is, since it’s called ‘guix shell’ and not ‘guix-shell’, we have
> to use ‘/usr/bin/env -S’ anyway as the shebang so it splits arguments.
> And with ‘-S’, we can already do pretty much everything we want.  Here’s
> a shell script:
>
> --8<---------------cut here---------------start------------->8---
> $ cat t.sh
> #!/usr/bin/env -S guix shell hello bash -- sh
> type -P hello
> hello
> --8<---------------cut here---------------end--------------->8---

I was already very excited about this patch series, and now I see this!

I wanted to point out that I don't think Guix's shebang substitution phases know how to deal with ~env -S~, and I wish they would (I recently ran into this). I can't really think of any use-cases for a Guix package which when built contains a ~guix shell~ script, but I wanted to raise the possibility in case someone else can.

-- 
Katherine

[bug#50960] [PATCH 00/10] Add 'guix shell' to subsume 'guix environment'

[Vagrant Cascadian]

[-- Attachment #1: Type: text/plain, Size: 795 bytes --]

On 2021-10-02, Ludovic Courtès wrote:
> Here comes ‘guix shell’, a proposed replacement for ‘guix environment’!

Yay!

> ‘guix environment’ would stay around though, at least for some time,
> probably for a long time.
>
> The differences to ‘guix environment’ are:
...
>   2. ‘guix shell’, without arguments, loads ‘guix.scm’ or ‘manifest.scm’
>      from the current directory or one of its ancestors.

This sounds a little scary to me, just implicitly importing whatever
happens to be lying around doesn't sound very guixy...

Wouldn't it be better to:

  guix shell guix.scm

or

  guix shell ./guix.scm

or

  guix shell --some-argument guix.scm


Or maybe I'm not understanding the idea all that well...


live well,
  vagrant

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

[bug#50960] [PATCH 00/10] Add 'guix shell' to subsume 'guix environment'

[Nicolò Balzarotti]

Hi!

Vagrant Cascadian <vagrant@debian.org> writes:

> On 2021-10-02, Ludovic Courtès wrote:
>> Here comes ‘guix shell’, a proposed replacement for ‘guix environment’!
>
> Yay!
>
>> ‘guix environment’ would stay around though, at least for some time,
>> probably for a long time.
>>
>> The differences to ‘guix environment’ are:
> ...
>>   2. ‘guix shell’, without arguments, loads ‘guix.scm’ or ‘manifest.scm’
>>      from the current directory or one of its ancestors.
>
> This sounds a little scary to me, just implicitly importing whatever
> happens to be lying around doesn't sound very guixy...
> [...]

What about doing something like what direnv[fn:1] does?

Quoting the website:
"direnv checks for the existence of a .envrc file in the current and
parent directories. If the file exists (and is authorized), it is loaded
into a bash sub-shell and all exported variables are then captured by
direnv and then made available to the current shell."

The difference between direnv and the current approach is that if the
file has never been "authorized", before it being imported you need to
run a command (direnv allow) to authorize it.  There's the
~/.config/direnv/allow dir which stores files named with the hash of the
content of the config, and whose content is just the path of the file
(don't know why this is needed).

This allows for automatic environment ({manifest,guix}.scm) file
selection AND a it's a bit more secure (it won't run arbitrary code
residing anywhere in the directory structure).

Except for this, I'd love to see guix shell merged, it will be a major
improvement over guix environment for my use cases.


Thanks!
Nicolò

[fn:1] direnv.net

[bug#50960] [PATCH 00/10] Add 'guix shell' to subsume 'guix environment'

[Ludovic Courtès]

Hello!

Vagrant Cascadian <vagrant@debian.org> skribis:

> On 2021-10-02, Ludovic Courtès wrote:

[...]

>>   2. ‘guix shell’, without arguments, loads ‘guix.scm’ or ‘manifest.scm’
>>      from the current directory or one of its ancestors.
>
> This sounds a little scary to me, just implicitly importing whatever
> happens to be lying around doesn't sound very guixy...

Right, it would be the first command that does that.

I became quite convinced that conventions and, thus, implicit arguments
can occasionally improve usability.  We use tools that operate this way
daily: ‘make’, ‘git’, etc.  Dave nicely argued about it:

  https://lists.gnu.org/archive/html/guix-devel/2017-08/msg00300.html

But yeah, it is a departure from what we’ve done so far, so we should
take the time to think through it.

Thanks,
Ludo’.

[bug#50960] [PATCH 00/10] Add 'guix shell' to subsume 'guix environment'

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 1824 bytes --]

Ludovic Courtès schreef op ma 04-10-2021 om 10:34 [+0200]:
> Hello!
> 
> Vagrant Cascadian <vagrant@debian.org> skribis:
> 
> > On 2021-10-02, Ludovic Courtès wrote:
> 
> [...]
> 
> > >   2. ‘guix shell’, without arguments, loads ‘guix.scm’ or ‘manifest.scm’
> > >      from the current directory or one of its ancestors.
> > 
> > This sounds a little scary to me, just implicitly importing whatever
> > happens to be lying around doesn't sound very guixy...
> 
> Right, it would be the first command that does that.
> 
> I became quite convinced that conventions and, thus, implicit arguments
> can occasionally improve usability.  We use tools that operate this way
> daily: ‘make’, ‘git’, etc.  Dave nicely argued about it:

'git' doesn't run binaries in the repository, unless configured otherwise
(in .git/config I think).  ‘make’ and ‘bundle’ are verbs and are for building
source code, which needs to be checked for backdoors anyway, so those programs
implicitely reading code from the current directory seems acceptable.

"guix sh" seems to be useful outside software development.
E.g. I sometimes do
"guix environment --pure --ad-hoc minetest various-minetest-mods-... -- minetest",
which would become
"guix shell --pure minetest various-minetest-mods-... -- minetest".
I could very easily accidentally press the enter key after typing "shell"
(I write from personal experience), and this could easily happen from within,
say, a ~/Downloads directory with an untrusted guix.scm (e.g. downloaded from
some rando's site to look at later).

Conventions are nice, but loading arbitrary code from the current directory
by default is an exploit waiting to happen.  This situation seem like including "."
in PATH by default to me.

Greetings,
Maxime

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

[bug#50960] [PATCH 00/10] Add 'guix shell' to subsume 'guix environment'

[zimoun]

Hi,

All looks good to me; from the surface, I have not looked at the
details.


On Sat, 02 Oct 2021 at 12:21, Ludovic Courtès <ludo@gnu.org> wrote:

>   2. ‘guix shell’, without arguments, loads ‘guix.scm’ or ‘manifest.scm’
>      from the current directory or one of its ancestors.

Personally, I do not like this default behaviour because explicit is
better than implicit and in the face of ambiguity, refuse the temptation
to guess; as any good Zen says. :-)

Idem with the proposal ’channels.scm’ and time-machine.


> As for deprecation, I think there’s no rush.  I imagine there could be
> several phases, like: initially we only mention deprecation in the manual,
> later on ‘guix environment’ starts emitting a warning, and later (I guess
> at least two years later, probably more) we ask ourselves whether to
> remove ‘guix environment’.  At this point keeping it doesn’t cost us much.

Concretely, I propose this plan:

 - v1.4: mention deprecation in the manual and remove from “guix help”
 - v1.5: emit a warning
 - v1.6: remove the command

Well, I do not see why the removal should be an issue, because there is
“guix time-machine”.  To me, the real issue is to let people knowing
such change.

Moreover, I am doubtful that the exact same command-line from v1.0 is
producing the same result with v1.3 for instance.  Not because of Guix
but because of the underlying packages.

For example, <http://issues.guix.gnu.org/47097>.


All the best,
simon

[bug#50960] [PATCH 00/10] Add 'guix shell' to subsume 'guix environment'

[Ludovic Courtès]

Hi,

zimoun <zimon.toutoune@gmail.com> skribis:

> On Sat, 02 Oct 2021 at 12:21, Ludovic Courtès <ludo@gnu.org> wrote:
>
>>   2. ‘guix shell’, without arguments, loads ‘guix.scm’ or ‘manifest.scm’
>>      from the current directory or one of its ancestors.
>
> Personally, I do not like this default behaviour because explicit is
> better than implicit and in the face of ambiguity, refuse the temptation
> to guess; as any good Zen says. :-)

It should be clear from the code that I generally prefer explicit over
implicit :-), but I think Dave has a point when talking about
conventions for this kind of developer tool.

>> As for deprecation, I think there’s no rush.  I imagine there could be
>> several phases, like: initially we only mention deprecation in the manual,
>> later on ‘guix environment’ starts emitting a warning, and later (I guess
>> at least two years later, probably more) we ask ourselves whether to
>> remove ‘guix environment’.  At this point keeping it doesn’t cost us much.
>
> Concretely, I propose this plan:
>
>  - v1.4: mention deprecation in the manual and remove from “guix help”
>  - v1.5: emit a warning
>  - v1.6: remove the command

Could be like this.  I guess we could also slow down the plan if we
observe that ‘guix environment’ sticks around longer than we thought.

> Well, I do not see why the removal should be an issue, because there is
> “guix time-machine”.  To me, the real issue is to let people knowing
> such change.

As discussed in <https://issues.guix.gnu.org/38529#17>, removal is an
issue because of existing scripts but also because of learning material
around: MOOCs, articles, tutorials, etc.  These won’t be updated
overnight, and we owe our users stability.

Ludo’.

[bug#50960] [PATCH 00/10] Add 'guix shell' to subsume 'guix environment'

[zimoun]

Hi,

On Mon, 04 Oct 2021 at 10:39, Ludovic Courtès <ludo@gnu.org> wrote:

> It should be clear from the code that I generally prefer explicit over
> implicit :-), but I think Dave has a point when talking about
> conventions for this kind of developer tool.

Well, if you speak about this thread [1], the point is others are doing
so and it is an usability improvement. :-)

I am fine since somehow it is already the case with ’make’ or all the
config files or many of us seem to simplify their workflow using direnv
or etc.  so yes it is probably handy to have conventions and automatic
load by default. :-)

1: <https://lists.gnu.org/archive/html/guix-devel/2017-08/msg00300.html>


>>> As for deprecation, I think there’s no rush.  I imagine there could be
>>> several phases, like: initially we only mention deprecation in the manual,
>>> later on ‘guix environment’ starts emitting a warning, and later (I guess
>>> at least two years later, probably more) we ask ourselves whether to
>>> remove ‘guix environment’.  At this point keeping it doesn’t cost us much.
>>
>> Concretely, I propose this plan:
>>
>>  - v1.4: mention deprecation in the manual and remove from “guix help”
>>  - v1.5: emit a warning
>>  - v1.6: remove the command
>
> Could be like this.  I guess we could also slow down the plan if we
> observe that ‘guix environment’ sticks around longer than we thought.

Bah, let reconsider this when preparing v1.6. ;-)  At the current
release rate, probably 2024. :-)

From my point of view, we should clearly document at v1.4 that this
command will be removed soon or later.


>> Well, I do not see why the removal should be an issue, because there is
>> “guix time-machine”.  To me, the real issue is to let people knowing
>> such change.
>
> As discussed in <https://issues.guix.gnu.org/38529#17>, removal is an
> issue because of existing scripts but also because of learning material
> around: MOOCs, articles, tutorials, etc.  These won’t be updated
> overnight, and we owe our users stability.

For sure, I agree «we owe our users stability».  That’s I translate into
«the real issue is to let people knowing such change». :-)

As I am trying to explain, this scenario,

  guix environment <options> <packages>
  … wait several months or years …
  guix pull
  guix environment <options> <packages>

will break because <packages>.  Therefore, one will have to fix these
very same scripts.  Even us, we are not able to maintain working scripts
for different points in time.  The example using “guix environment” is
broken for the manual released at v1.3, as pointed here [2].

However, “guix environment” will still work using “guix time-machine”.
Nothing is really broken, IMHO.

About learning materials, we could remove the command “guix environment”
but keep a section in the manual explaining how to switch to the new
“guix shell”.

Well, let discuss all that when preparing v1.5 or v1.6.  Something as
one or two years from now. ;-)  Which i

2: <http://issues.guix.gnu.org/47097>


Cheers,
simon

[bug#50960] [PATCH 00/10] Add 'guix shell' to subsume 'guix environment'

[Ludovic Courtès]

zimoun <zimon.toutoune@gmail.com> skribis:

> On Mon, 04 Oct 2021 at 10:39, Ludovic Courtès <ludo@gnu.org> wrote:

[...]

>> As discussed in <https://issues.guix.gnu.org/38529#17>, removal is an
>> issue because of existing scripts but also because of learning material
>> around: MOOCs, articles, tutorials, etc.  These won’t be updated
>> overnight, and we owe our users stability.
>
> For sure, I agree «we owe our users stability».  That’s I translate into
> «the real issue is to let people knowing such change». :-)

No no.  The problem is that there *will* be material out there
explaining Guix in terms of ‘guix environment’, people *will* stumble on
it, and it’s important to keep it working long enough so people can
update their material.

Ludo’.

[bug#50960] [PATCH 00/10] Add 'guix shell' to subsume 'guix environment'

[zimoun]

On Mon, 4 Oct 2021 at 14:23, Ludovic Courtès <ludo@gnu.org> wrote:
> zimoun <zimon.toutoune@gmail.com> skribis:

> > For sure, I agree «we owe our users stability».  That’s I translate into
> > «the real issue is to let people knowing such change». :-)
>
> No no.  The problem is that there *will* be material out there
> explaining Guix in terms of ‘guix environment’, people *will* stumble on
> it, and it’s important to keep it working long enough so people can
> update their material.

We are saying the same, no?

Except that for me "people" and "material" are vague so I propose to add
a section to deal with this vagueness at removal time (maybe 2023 or
2024?) explaining how to switch from "guix enviornment" to "guix shell".
Therefore, people will stumble on this not-updated material, then they
will be able to run it with "guix shell" using the proposed section.
One could image:

  $ guix enviornment <old> <example> <how> <to> -- <use> <it>
  error: environment had be removed by commit: xxxxx.
  hint: Try `guix time-machine --commit=yyyyy -- environment <old> <example> <how> <to> -- <use> <it>'
  Please read section "guix environment" in the manual.

where yyyy is that last commit right before the removal.  Or something
along this idea.


Well, we agree to speak about that when preparing v1.5 or v1.6 and we
are far from that. ;-)

Cheers,
simon

[bug#50960] [PATCH 00/10] Add 'guix shell' to subsume 'guix environment'

[Leo Famulari]

On Sat, Oct 02, 2021 at 12:21:16PM +0200, Ludovic Courtès wrote:
> Here comes ‘guix shell’, a proposed replacement for ‘guix environment’!
> ‘guix environment’ would stay around though, at least for some time,
> probably for a long time.

Very nice!

>        ‘guix shell -D hello git’ ≍ ‘guix environment hello --ad-hoc git’

Is my understanding correct, that arguments to `guix environment` are
positional, whereas with `guix shell` arguments are not positional?

[bug#50960] [PATCH 00/10] Add 'guix shell' to subsume 'guix environment'

[Ludovic Courtès]

Hi,

Leo Famulari <leo@famulari.name> skribis:

> On Sat, Oct 02, 2021 at 12:21:16PM +0200, Ludovic Courtès wrote:

[...]

>>        ‘guix shell -D hello git’ ≍ ‘guix environment hello --ad-hoc git’
>
> Is my understanding correct, that arguments to `guix environment` are
> positional, whereas with `guix shell` arguments are not positional?

No, it’s the same.  However, ‘-D’ here affects only the immediately
following package, whereas ‘--ad-hoc’ would affect every package that
follows.

Ludo’.

[bug#50960] [PATCH 00/10] Add 'guix shell' to subsume 'guix environment'

[Leo Famulari]

On Fri, Oct 08, 2021 at 09:43:19AM +0200, Ludovic Courtès wrote:
> No, it’s the same.  However, ‘-D’ here affects only the immediately
> following package, whereas ‘--ad-hoc’ would affect every package that
> follows.

I must not know the correct terminology; that's what I was trying to
ask.

Can we keep the old behavior?  Or do we think this new way of doing it
is better?

[bug#50960] [PATCH 00/10] Add 'guix shell' to subsume 'guix environment'

[Ludovic Courtès]

Hi,

Leo Famulari <leo@famulari.name> skribis:

> On Fri, Oct 08, 2021 at 09:43:19AM +0200, Ludovic Courtès wrote:
>> No, it’s the same.  However, ‘-D’ here affects only the immediately
>> following package, whereas ‘--ad-hoc’ would affect every package that
>> follows.
>
> I must not know the correct terminology; that's what I was trying to
> ask.
>
> Can we keep the old behavior?  Or do we think this new way of doing it
> is better?

What do you mean by “old behavior”?

I think there’s large consensus that ‘--ad-hoc’ should be the default,
and that’s the main motivation behind this patch series.

As for ‘-D’: we could arrange so that it has effect on all the following
packages.  However, if we did that, we’d need to reintroduce ‘--ad-hoc’
to cancel the effect of ‘-D’.

More importantly, I think having ‘-D’ affect only the immediately
following package is more consistent with typical usage—I don’t think
I’ve ever run ‘guix environment foo bar baz’ to get the merged
development environments of all three packages.

Thanks,
Ludo’.

[bug#50960] [PATCH 00/10] Add 'guix shell' to subsume 'guix environment'

[Leo Famulari]

On Sat, Oct 09, 2021 at 03:38:55PM +0200, Ludovic Courtès wrote:
> What do you mean by “old behavior”?

I mean that all the packages listed after the option are affected by it.

> More importantly, I think having ‘-D’ affect only the immediately
> following package is more consistent with typical usage—I don’t think
> I’ve ever run ‘guix environment foo bar baz’ to get the merged
> development environments of all three packages.

I see what you mean: that is a weird behaviour. Let's disregard the
question I brought up.

[bug#50960] [EXT] [bug#50960] [PATCH 00/10] Add 'guix shell' to subsume 'guix environment'

[Thompson, David]

Hey Ludo,

First of all: Thanks for picking up where I left off all those years
ago.  You remember mailing list posts I wrote 4 years ago much better
than I do. ;)

On Sat, Oct 2, 2021 at 6:22 AM Ludovic Courtès <ludo@gnu.org> wrote:
>
> Hello Guix!
>
> Here comes ‘guix shell’, a proposed replacement for ‘guix environment’!
> ‘guix environment’ would stay around though, at least for some time,
> probably for a long time.
>
> The differences to ‘guix environment’ are:
>
>   1. ‘--ad-hoc’ is the default.
>
>        ‘guix shell hello’ ≍ ‘guix environment --ad-hoc hello’
>        ‘guix shell -D hello git’ ≍ ‘guix environment hello --ad-hoc git’
>
>   2. ‘guix shell’, without arguments, loads ‘guix.scm’ or ‘manifest.scm’
>      from the current directory or one of its ancestors.
>
>   3. ‘--load’/‘-l’ is not ‘-f’/‘--install-from-file’ for consistency with
>      ‘guix package’.
>
>   4. ‘guix shell’ without arguments maintains a cache, such that, the
>      second time you run it, it runs in ~0.1s (it does not even need to
>      connect to the daemon).
>
>      If you run ‘guix pull’ and run again ‘guix shell’, it recomputes
>      the environment, as is currently the case with ‘guix environment’.
>
> Here’s a summary of previous proposals:
>
>   - Dave Thompson: https://lists.gnu.org/archive/html/guix-devel/2017-08/msg00300.html
>     - [X] --ad-hoc is the default
>     - [X] caching
>     - [X] behavior with no arguments
>     - [ ] --load accepts <environment>
>     - [ ] Shepherd services
>     - [ ] 'guix environment --update' to explicitly update
>   - make --ad-hoc the default: https://issues.guix.gnu.org/38529
>     - [X] https://issues.guix.gnu.org/38529#17: proposal for a new subcommand
>           deprecation of ‘guix environment’
>
> I think <environment> records and Shepherd services could come later.
> As for ‘--update’, I prefer the behavior implemented here because it’s
> stateless and thus more predictable.
>
> Thoughts?  Are there other changes people would like to see?

A few thoughts:

1) I can't be the only one that thinks it's a pain to rebuild the
whole environment just because I updated the Guix client. It's
especially frustrating when there is a regression that breaks the
project, or the substitute servers are having a bad day.

2) One important use-case for tools like Bundler, npm, etc. is to
exactly (within their limited ability to do so, of course) reproduce
the environment for all developers working on the project.  With 'guix
environment', and now with this proposed 'guix shell', the resulting
development environment depends upon the version of Guix being used.
So, to make 'guix shell' a true universal replacement for these
language-specific tools, there should be a way to "lock" to a specific
version of Guix.  Bundler has Gemfile.lock, npm has package-lock.json,
etc.  Guix has good support for using older versions of Guix already,
so it seems feasible.  I spend a non-trivial amount of time sorting
out development environment issues for a couple dozen devs,
so a tool that took most of the variables out of the equation would be
really great.

3) I haven't looked at the patches to see if it has already been
implemented, but when running 'guix shell' with no arguments, I think
the search for the manifest file should follow the established
conventions of Git, Bundler, etc. and search not only the current
directory, but the parent directory and so on until it finds it or
gets to / and gives up. This will make the tool much more usable when
working inside a subdirectory within a project.

Just some things to think about.  I just don't want to see Guix commit
to an interface that slams the door on future improvement due to
compatibility reasons.  I mean, what would you even name the next
tool? guix... biome?

Anyway, great work!  Back to lurking.

- Dave

[bug#50960] [PATCH 00/10] Add 'guix shell' to subsume 'guix environment'

[Ludovic Courtès]

Hello!

"Thompson, David" <dthompson2@worcester.edu> skribis:

> 1) I can't be the only one that thinks it's a pain to rebuild the
> whole environment just because I updated the Guix client. It's
> especially frustrating when there is a regression that breaks the
> project, or the substitute servers are having a bad day.

The way I see it, I update Guix precisely to get a newer environment.
Otherwise, why would I update it for?

> 2) One important use-case for tools like Bundler, npm, etc. is to
> exactly (within their limited ability to do so, of course) reproduce
> the environment for all developers working on the project.  With 'guix
> environment', and now with this proposed 'guix shell', the resulting
> development environment depends upon the version of Guix being used.
> So, to make 'guix shell' a true universal replacement for these
> language-specific tools, there should be a way to "lock" to a specific
> version of Guix.  Bundler has Gemfile.lock, npm has package-lock.json,
> etc.  Guix has good support for using older versions of Guix already,
> so it seems feasible.  I spend a non-trivial amount of time sorting
> out development environment issues for a couple dozen devs,
> so a tool that took most of the variables out of the equation would be
> really great.

This part is currently addressed by ‘guix describe -f channels’ (to pin
the Guix revision) and ‘guix time-machine’ (to spin up a specific
revision).  The other tools are totally unaware of revisions and
channels.

As it stands, developers would use the pinned version by running:

  guix time-machine --commit=XYZ -- shell

or:

  guix time-machine -C channels.scm -- shell

Perhaps ‘guix time-machine’, too, could automatically detect
‘channels.scm’.

The temptation to add revision data to tools that don’t otherwise deal
with it is real; I think it’s better kept separate though, because
fundamentally, each revision lives in its own world.

> 3) I haven't looked at the patches to see if it has already been
> implemented, but when running 'guix shell' with no arguments, I think
> the search for the manifest file should follow the established
> conventions of Git, Bundler, etc. and search not only the current
> directory, but the parent directory and so on until it finds it or
> gets to / and gives up. This will make the tool much more usable when
> working inside a subdirectory within a project.

It’s implemented (and controversial).

> Just some things to think about.  I just don't want to see Guix commit
> to an interface that slams the door on future improvement due to
> compatibility reasons.  I mean, what would you even name the next
> tool? guix... biome?

Good point.  :-)

Thanks,
Ludo’.

[bug#50960] [PATCH 00/10] Add 'guix shell' to subsume 'guix environment'

[pelzflorian (Florian Pelz)]

I’m not a direnv user and frankly don’t really understand the gain
from loading by default the guix.scm file nor manifest.scm.

My fear is accidentally running code, possibly malicious, possibly
just a backup script one happened to call guix.scm.  In German we have
the word DAU for “dumbest assumable user”.  Guix without shell is
DAU-prove.

Both

- Konrad Hinsen’s suggestion of two different commands and

- Nicolò Balzarotti’s suggestion of having to explicitly allow a file

would take away my paranoia of accidentally running code.  (By just
not using Konrad Hinsen’s other command.)

- Hidden .-rc files in the current directory would not take away my
  paranoia of malicious code.

- I think printing a suggestion to use --file=../guix.scm is fine.

Bash is different from Guix Shell.  It does not load code from . or
arbitrary parent directories.

Regards,
Florian

[bug#50960] [EXT] Re: [bug#50960] [PATCH 00/10] Add 'guix shell' to subsume 'guix environment'

[Thompson, David]

On Thu, Oct 7, 2021 at 6:52 AM pelzflorian (Florian Pelz)
<pelzflorian@pelzflorian.de> wrote:
>
> I’m not a direnv user and frankly don’t really understand the gain
> from loading by default the guix.scm file nor manifest.scm.
>
> My fear is accidentally running code, possibly malicious, possibly
> just a backup script one happened to call guix.scm.  In German we have
> the word DAU for “dumbest assumable user”.  Guix without shell is
> DAU-prove.

I don't think your fear should outweigh the immense user friendliness
of having some established conventions to ease development.  Every
mainstream developer tool I use in the professional word has
conventions like this, and the result is that it is quick and easy to
jump into a new project and get the software you need to be
productive.  Guix needs to embrace this kind of usability if it hopes
to offer a real alternative to these other tools.  'guix shell' with
no other arguments should "just work" in the context of whatever
project I am in.

- Dave

[bug#50960] [EXT] Re: [bug#50960] [PATCH 00/10] Add 'guix shell' to subsume 'guix environment'

[pelzflorian (Florian Pelz)]

On Thu, Oct 07, 2021 at 07:17:43AM -0400, Thompson, David wrote:
> 'guix shell' with
> no other arguments should "just work" in the context of whatever
> project I am in.

If you only care about projects: Searching only guix.scm and
manifest.scm files when a .git subdirectory is present would indeed be
better than going through all parent directories.  I do not know git
well enough to judge if that could interfere with git worktrees and
git submodules.

The only remaining problem would be novice users
who know “guix shell program-a program-b” and who do not expect “guix
shell” without argument instead loads code from a file.

I’m also afraid of git hooks btw, which is why I clone old git
checkouts from the backup instead of copying, but at least they stay
local to the .git checkout.

I still don’t understand why not having to type --file is useful
though.

Regards,
Florian

[bug#50960] [EXT] Re: [bug#50960] [PATCH 00/10] Add 'guix shell' to subsume 'guix environment'

[Katherine Cox-Buday]

"Thompson, David" <dthompson2@worcester.edu> writes:

> I don't think your fear should outweigh the immense user friendliness
> of having some established conventions to ease development.  Every
> mainstream developer tool I use in the professional word has
> conventions like this, and the result is that it is quick and easy to
> jump into a new project and get the software you need to be
> productive.  Guix needs to embrace this kind of usability if it hopes
> to offer a real alternative to these other tools.  'guix shell' with
> no other arguments should "just work" in the context of whatever
> project I am in.
>
> - Dave

I agree with this.

-- 
Katherine

[bug#50960] [PATCH 00/10] Add 'guix shell' to subsume 'guix environment'

[zimoun]

Hi Ludo,

On Thu, 7 Oct 2021 at 11:27, Ludovic Courtès <ludo@gnu.org> wrote:
> "Thompson, David" <dthompson2@worcester.edu> skribis:
>
> > 1) I can't be the only one that thinks it's a pain to rebuild the
> > whole environment just because I updated the Guix client. It's
> > especially frustrating when there is a regression that breaks the
> > project, or the substitute servers are having a bad day.
>
> The way I see it, I update Guix precisely to get a newer environment.
> Otherwise, why would I update it for?

I think there is a kind of misunderstanding because one updates Guix
because of new project B but then rebuilds the whole environment for
project A.  And this can be unpleasant.

The solution is to use "guix time-machine -C channels.scm --
environment" but again this can be unpleasant because it builds a lot
of things; although before one runs "guix environment".

Well, you should tell me that the solution is to create a profile.
However, I often use "guix environment" with the options --pure or
--container, and a profile cannot, IIUC.


> Perhaps ‘guix time-machine’, too, could automatically detect
> ‘channels.scm’.

Well, I am not convince it helps for readibility.  At the end, it would read:

  guix time-machine -- shell

but then to know what this command does exactly, one should open the
manual check the "guix time-machine" part, check the presence of the
'channels.scm' file, then read the "guix shell" part, then check if if
is 'manifests.scm' or 'guix.scm'.  Maybe, it is just habits, but I
find much simpler:

  guix time-machine -C channels -- shell -m manifest.scm
  guix time-machine -C channels -- shell -f guix.scm

where I just have at the command-line type "guix time-machine --help"
or "guix shell --help" to remember the options and if I do not know
what manifest is, then I go to the manual.

With completion, it is not longer to type.  "Explicit is better than
implicit" as a good Zen says!


> The temptation to add revision data to tools that don’t otherwise deal
> with it is real; I think it’s better kept separate though, because
> fundamentally, each revision lives in its own world.

I agree.


All the best,
simon

[bug#50960] [PATCH 00/10] Add 'guix shell' to subsume 'guix environment'

[Konrad Hinsen]

Hi everyone,

I just discovered this discussion today... a lot to read...

First of all, I very much like this proposal. It provides many
improvements to today's "guix environment" for which I have immediate
use cases.

Looking at the proposal and then the discussion, I wonder if the main
issue with "guix shell" is that it's two very different things wrapped
into one:

  1) A convenience utility for interactive work, in particular software
     development, implemented as "guix shell" with no arguments.

  2) A modernized "guix environment" for both interactive and scripting
     use.

The behavior of "guix shell" with and without arguments is sufficiently
different, which will make it a challenge to document, and then a
challenge for users to understand. And part of the criticism related to
the tacit execution of files is about the convenience for 1) vs. the
risk in 2).

So... how about making this two different commands? They could of course
share most of the implementation.

I'd use "guix shell" for the scenario that actually starts a shell, and
something different, perhaps "guix process", for the infrastructure
command for use in scripts.

Cheers,
  Konrad

[bug#50960] [PATCH 00/10] Add 'guix shell' to subsume 'guix environment'

[Ludovic Courtès]

Hi Konrad,

Konrad Hinsen <konrad.hinsen@fastmail.net> skribis:

> Looking at the proposal and then the discussion, I wonder if the main
> issue with "guix shell" is that it's two very different things wrapped
> into one:
>
>   1) A convenience utility for interactive work, in particular software
>      development, implemented as "guix shell" with no arguments.
>
>   2) A modernized "guix environment" for both interactive and scripting
>      use.

To me it’s the same as ‘guix environment’, but with a slightly different
command-line interface; ‘guix environment’ was already serves these two
use cases.

> The behavior of "guix shell" with and without arguments is sufficiently
> different, which will make it a challenge to document, and then a
> challenge for users to understand. And part of the criticism related to
> the tacit execution of files is about the convenience for 1) vs. the
> risk in 2).

‘guix shell’ without arguments is equivalent to ‘guix environment -f
guix.scm’ or ‘guix environment -m manifest.scm’.

> So... how about making this two different commands? They could of course
> share most of the implementation.
>
> I'd use "guix shell" for the scenario that actually starts a shell, and
> something different, perhaps "guix process", for the infrastructure
> command for use in scripts.

I don’t think there’s enough to warrant two different commands (and
perhaps we could leave out the auto-detection of ‘guix.scm’ or
‘manifest.scm’ is that proves to be too controversial).

Maybe you’re hinting at the name and the fact that it suggests it’s
starting a shell.  I’m not fond of it, notably for that reason.  For the
record, it came out as the only proposal that seemed viable to me:

  https://issues.guix.gnu.org/38529#17

Thoughts?

Thanks for your feedback,
Ludo’.

[bug#50960] [PATCH 00/10] Add 'guix shell' to subsume 'guix environment'

[Liliana Marie Prikler]

Hi Ludo,

Am Donnerstag, den 07.10.2021, 10:34 +0200 schrieb Ludovic Courtès:
> > The behavior of "guix shell" with and without arguments is
> > sufficiently different, which will make it a challenge to document,
> > and then a challenge for users to understand. And part of the
> > criticism related to the tacit execution of files is about the
> > convenience for 1) vs. the risk in 2).
> 
> ‘guix shell’ without arguments is equivalent to ‘guix environment -f
> guix.scm’ or ‘guix environment -m manifest.scm’.
> 
> > So... how about making this two different commands? They could of
> > course share most of the implementation.
> > 
> > I'd use "guix shell" for the scenario that actually starts a shell,
> > and something different, perhaps "guix process", for the
> > infrastructure command for use in scripts.
> 
> I don’t think there’s enough to warrant two different commands (and
> perhaps we could leave out the auto-detection of ‘guix.scm’ or
> ‘manifest.scm’ is that proves to be too controversial).
Leaving out the auto-detection of ‘guix.scm’ and ‘manifest.scm’ for now
is a good idea, given that Guix itself would be an edge-case for that.

Perhaps to avoid this trouble, guix shell could read a file that simply
specifies which arguments to add one line at a time e.g.

  ;; .guix-shell-rc
  --manifest=manifest.scm

or

  ;; .guix-shell-rc
  --load=guix.scm

Similar behaviour is used in other (GNU) tools with lots of available
arguments, which operate on a directory basis, e.g. stow.

> Maybe you’re hinting at the name and the fact that it suggests it’s
> starting a shell.  I’m not fond of it, notably for that reason.  For
> the record, it came out as the only proposal that seemed viable to
> me:
> 
>   https://issues.guix.gnu.org/38529#17
> 
> Thoughts?
I think “guix shell” is fine as-is.  People typically don't complain
that “sh somefile.sh” doesn't launch a shell, so the same reasoning can
be applied to Guix tools in my opinion.  We would still have the always
non-intuitive naming choices of “do”, “start” and “run” at our disposal
if we ever need to discard the shell.

Regards,
Liliana

[bug#50960] [PATCH 00/10] Add 'guix shell' to subsume 'guix environment'

[Konrad Hinsen]

Hi Ludo and Liliana,

> To me it’s the same as ‘guix environment’, but with a slightly different
> command-line interface; ‘guix environment’ was already serves these two
> use cases.

Indeed, but the new proposal goes much further with supporting
interactive convenience, in particular via the controversial reading of
guix.scm and/or manifest.scm. That's a convenience I'd like to have
interactively, but not something I expect from an infrastructure tool
that ends up being used in lengthy shell script. For the latter, the
"Zen of Python wisdom" of "explicit is better than implicit" matters a
lot. On the command line, it makes life harder, in particular for bad
typists such as myself.

> Perhaps to avoid this trouble, guix shell could read a file that simply
> specifies which arguments to add one line at a time e.g.
>
>   ;; .guix-shell-rc
>   --manifest=manifest.scm

That's a nice idea, but again something I wouldn't want an
infrastructure tool to do. When I use "guix environment" or "guix shell"
in a script or workflow, I want whatever it does to be as independent as
possible from my machine or account, except when I explicitly say the
contrary.

Cheers,
  Konrad

[bug#50960] [PATCH 00/10] Add 'guix shell' to subsume 'guix environment'

[Liliana Marie Prikler]

Am Freitag, den 08.10.2021, 17:45 +0200 schrieb Konrad Hinsen:
> Hi Ludo and Liliana,
> 
> > To me it’s the same as ‘guix environment’, but with a slightly
> > different command-line interface; ‘guix environment’ was already
> > serves these two use cases.
> 
> Indeed, but the new proposal goes much further with supporting
> interactive convenience, in particular via the controversial reading
> of guix.scm and/or manifest.scm. That's a convenience I'd like to
> have interactively, but not something I expect from an infrastructure
> tool that ends up being used in lengthy shell script. For the latter,
> the "Zen of Python wisdom" of "explicit is better than implicit"
> matters a lot. On the command line, it makes life harder, in
> particular for bad typists such as myself.
> 
> > Perhaps to avoid this trouble, guix shell could read a file that
> > simply specifies which arguments to add one line at a time e.g.
> > 
> >   ;; .guix-shell-rc
> >   --manifest=manifest.scm
> 
> That's a nice idea, but again something I wouldn't want an
> infrastructure tool to do. When I use "guix environment" or "guix
> shell" in a script or workflow, I want whatever it does to be as
> independent as possible from my machine or account, except when I
> explicitly say the contrary.
People do use “emacs -Q” so there's no hard “one way or the other”
requirement.  All we would need to do is offer a similar flag to
prevent reading the rc file, no?

Cheers,
Liliana

[bug#50960] [PATCH 00/10] Add 'guix shell' to subsume 'guix environment'

[Ludovic Courtès]

Hi,

Konrad Hinsen <konrad.hinsen@fastmail.net> skribis:

>> To me it’s the same as ‘guix environment’, but with a slightly different
>> command-line interface; ‘guix environment’ was already serves these two
>> use cases.
>
> Indeed, but the new proposal goes much further with supporting
> interactive convenience, in particular via the controversial reading of
> guix.scm and/or manifest.scm. That's a convenience I'd like to have
> interactively, but not something I expect from an infrastructure tool
> that ends up being used in lengthy shell script. For the latter, the
> "Zen of Python wisdom" of "explicit is better than implicit" matters a
> lot. On the command line, it makes life harder, in particular for bad
> typists such as myself.

Agreed.  The automatic reading of guix.scm/manifest.scm, if we keep it,
should only happen in interactive use; I’ll double-check and make sure
this is the case.

Thanks,
Ludo’.

[bug#50960] [PATCH 00/10] Add 'guix shell' to subsume 'guix environment'

[Stefan]

Hi!

How about a directory argument to load some standard file? And using a special ... to search all parent directories?

guix shell .      ;# ./guix.scm
guix shell ..     ;# ../guix.scm
guix shell ...    ;# ./(../)*guix.scm


Bye

Stefan

[bug#50960] [PATCH v2 00/11] 'guix shell' strikes again

[Ludovic Courtès]

Hello Guix!

Here’s a v2 with the following changes:

  • ‘guix.scm’ and ‘manifest.scm’ only loaded in interactive mode,
    as suggested by Konrad.

  • Auto-loading happens if and only if the containing directory
    is listed in ~/.config/guix/shell-authorized-directories, as
    suggested by Florian.  When a file is found but not authorized,
    a hint is displayed explaining what to do.

  • ‘-q’ inhibits auto-loading.

  • Auto-detection of ‘guix.scm’ and ‘manifest.scm’ uses lexical
    “..” resolution and does not cross device boundaries.

  • Long version of ‘-f’ is now ‘--file’, like for ‘guix build’.

  • Caching also works with files explicitly specified with ‘-m’
    or ‘-f’.  That way, ‘guix shell -f guix.scm -- make’ benefits
    from it.

  • Cache key does not include file name, to avoid cache misses
    given that the file name is not canonicalized (see discussion
    with Maxime).

  • Documentation, including a tentative EOL date for ‘guix
    environment’ (“supported until at least May 1st, 2023”).  The
    intention is to give an incentive to switch while reassuring
    that it won’t disappear overnight.

How does that sound?  :-)

Ludo’.

Ludovic Courtès (11):
  packages: Add 'package-development-inputs'.
  profiles: Add 'package->development-manifest'.
  Add 'guix shell'.
  DRAFT shell: By default load the local 'guix.scm' or 'manifest.scm'
    file.
  DRAFT shell: Honor in ~/.config/guix/shell-authorized-directories.
  environment: Add tests for '--profile'.
  environment: Skip derivation computation when '--profile' is used.
  environment: Do not connect to the daemon when '--profile' is used.
  environment: Autoload some modules.
  cache: Gracefully handle non-existent cache.
  shell: Maintain a profile cache.

 Makefile.am                         |   2 +
 doc/contributing.texi               |   8 +-
 doc/guix.texi                       | 443 ++++++++++++++++++++++++++--
 guix/cache.scm                      |  10 +-
 guix/packages.scm                   |  10 +
 guix/profiles.scm                   |  19 ++
 guix/scripts/environment.scm        | 260 ++++++++--------
 guix/scripts/shell.scm              | 355 ++++++++++++++++++++++
 po/guix/POTFILES.in                 |   1 +
 tests/guix-environment-container.sh |   8 +
 tests/guix-environment.sh           |   7 +
 tests/guix-shell.sh                 | 100 +++++++
 tests/packages.scm                  |  14 +
 tests/profiles.scm                  |   7 +
 14 files changed, 1098 insertions(+), 146 deletions(-)
 create mode 100644 guix/scripts/shell.scm
 create mode 100644 tests/guix-shell.sh

-- 
2.33.0

[bug#50960] [PATCH v2 01/11] packages: Add 'package-development-inputs'.

[Ludovic Courtès]

* guix/packages.scm (package-development-inputs): New procedure.
* guix/scripts/environment.scm (package-environment-inputs): Use it.
* tests/packages.scm ("package-development-inputs")
("package-development-inputs, cross-compilation"): New tests.
* doc/guix.texi (package Reference): Document it.
---
 doc/guix.texi                | 41 ++++++++++++++++++++++++++++++++++++
 guix/packages.scm            | 10 +++++++++
 guix/scripts/environment.scm |  2 +-
 tests/packages.scm           | 14 ++++++++++++
 4 files changed, 66 insertions(+), 1 deletion(-)

diff --git a/doc/guix.texi b/doc/guix.texi
index a72a726b54..49399e792b 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -6847,6 +6847,47 @@ cross-compiling:
 It is an error to refer to @code{this-package} outside a package definition.
 @end deffn
 
+@cindex development inputs, of a package
+@cindex implicit inputs, of a package
+Sometimes you will want to obtain the list of inputs needed to
+@emph{develop} a package---all the inputs that are visible when the
+package is compiled.  This is what the @code{package-development-inputs}
+procedure returns.
+
+@deffn {Scheme Procedure} package-development-inputs @var{package} @
+   [@var{system}] [#:target #f]
+Return the list of inputs required by @var{package} for development
+purposes on @var{system}.  When @var{target} is true, return the inputs
+needed to cross-compile @var{package} from @var{system} to
+@var{triplet}, where @var{triplet} is a triplet such as
+@code{"aarch64-linux-gnu"}.
+
+Note that the result includes both explicit inputs and implicit
+inputs---inputs automatically added by the build system (@pxref{Build
+Systems}).  Let us take the @code{hello} package to illustrate that:
+
+@lisp
+(use-modules (gnu packages base) (guix packages))
+
+hello
+@result{} #<package hello@@2.10 gnu/packages/base.scm:79 7f585d4f6790>
+
+(package-direct-inputs hello)
+@result{} ()
+
+(package-development-inputs hello)
+@result{} (("source" @dots{}) ("tar" #<package tar@@1.32 @dots{}>) @dots{})
+@end lisp
+
+In this example, @code{package-direct-inputs} returns the empty list,
+because @code{hello} has zero explicit dependencies.  Conversely,
+@code{package-development-inputs} includes inputs implicitly added by
+@code{gnu-build-system} that are required to build @code{hello}: tar,
+gzip, GCC, libc, Bash, and more.  To visualize it, @command{guix graph
+hello} would show you explicit inputs, whereas @command{guix graph -t
+bag hello} would include implicit inputs (@pxref{Invoking guix graph}).
+@end deffn
+
 Because packages are regular Scheme objects that capture a complete
 dependency graph and associated build procedures, it is often useful to
 write procedures that take a package and return a modified version
diff --git a/guix/packages.scm b/guix/packages.scm
index 8c3a0b0b7b..43e0130793 100644
--- a/guix/packages.scm
+++ b/guix/packages.scm
@@ -153,6 +153,7 @@ (define-module (guix packages)
             bag-transitive-host-inputs
             bag-transitive-build-inputs
             bag-transitive-target-inputs
+            package-development-inputs
             package-closure
 
             default-guile
@@ -1070,6 +1071,15 @@ (define (bag-transitive-target-inputs bag)
                  (%current-system (bag-system bag)))
     (transitive-inputs (bag-target-inputs bag))))
 
+(define* (package-development-inputs package
+                                     #:optional (system (%current-system))
+                                     #:key target)
+  "Return the list of inputs required by PACKAGE for development purposes on
+SYSTEM.  When TARGET is true, return the inputs needed to cross-compile
+PACKAGE from SYSTEM to TRIPLET, where TRIPLET is a triplet such as
+\"aarch64-linux-gnu\"."
+  (bag-transitive-inputs (package->bag package system target)))
+
 (define* (package-closure packages #:key (system (%current-system)))
   "Return the closure of PACKAGES on SYSTEM--i.e., PACKAGES and the list of
 packages they depend on, recursively."
diff --git a/guix/scripts/environment.scm b/guix/scripts/environment.scm
index 6958bd6238..d555969b27 100644
--- a/guix/scripts/environment.scm
+++ b/guix/scripts/environment.scm
@@ -82,7 +82,7 @@ (define (package-environment-inputs package)
 packages for PACKAGE."
   ;; Remove non-package inputs such as origin records.
   (filter-map input->manifest-entry
-              (bag-transitive-inputs (package->bag package))))
+              (package-development-inputs package system)))
 
 (define (show-help)
   (display (G_ "Usage: guix environment [OPTION]... PACKAGE... [-- COMMAND...]
diff --git a/tests/packages.scm b/tests/packages.scm
index 3756877270..266b5aeb7a 100644
--- a/tests/packages.scm
+++ b/tests/packages.scm
@@ -353,6 +353,20 @@ (define read-at
           (package-transitive-supported-systems d)
           (package-transitive-supported-systems e))))
 
+(test-assert "package-development-inputs"
+  ;; Note: Due to propagated inputs, 'package-development-inputs' returns a
+  ;; couple more inputs, such as 'linux-libre-headers'.
+  (lset<= equal?
+          `(("source" ,(package-source hello)) ,@(standard-packages))
+          (package-development-inputs hello)))
+
+(test-assert "package-development-inputs, cross-compilation"
+  (lset<= equal?
+          `(("source" ,(package-source hello))
+            ,@(standard-cross-packages "mips64el-linux-gnu" 'host)
+            ,@(standard-cross-packages "mips64el-linux-gnu" 'target))
+          (package-development-inputs hello #:target "mips64el-linux-gnu")))
+
 (test-assert "package-closure"
   (let-syntax ((dummy-package/no-implicit
                 (syntax-rules ()
-- 
2.33.0

[bug#50960] [PATCH v2 01/11] packages: Add 'package-development-inputs'.

[zimoun]

Hi Ludo,

On Mon, 11 Oct 2021 at 23:37, Ludovic Courtès <ludo@gnu.org> wrote:
> * guix/packages.scm (package-development-inputs): New procedure.
> * guix/scripts/environment.scm (package-environment-inputs): Use it.
> * tests/packages.scm ("package-development-inputs")
> ("package-development-inputs, cross-compilation"): New tests.
> * doc/guix.texi (package Reference): Document it.

LGTM, except…

>  guix/packages.scm            | 10 +++++++++
>  guix/scripts/environment.scm |  2 +-

[...]

> +(define* (package-development-inputs package
> +                                     #:optional (system (%current-system))
> +                                     #:key target)
> +  "Return the list of inputs required by PACKAGE for development purposes on
> +SYSTEM.  When TARGET is true, return the inputs needed to cross-compile
> +PACKAGE from SYSTEM to TRIPLET, where TRIPLET is a triplet such as
> +\"aarch64-linux-gnu\"."
> +  (bag-transitive-inputs (package->bag package system target)))

[...]

>    ;; Remove non-package inputs such as origin records.
>    (filter-map input->manifest-entry
> -              (bag-transitive-inputs (package->bag package))))
> +              (package-development-inputs package system)))

Why ’system’ is used here?  Why is not simply?

 +              (package-development-inputs package)))

because ’system’ should be not defined, or I do not know where IIUC.


Cheers,
simon

[bug#50960] [PATCH v2 01/11] packages: Add 'package-development-inputs'.

[Ludovic Courtès]

zimoun <zimon.toutoune@gmail.com> skribis:

>> +(define* (package-development-inputs package
>> +                                     #:optional (system (%current-system))
>> +                                     #:key target)
>> +  "Return the list of inputs required by PACKAGE for development purposes on
>> +SYSTEM.  When TARGET is true, return the inputs needed to cross-compile
>> +PACKAGE from SYSTEM to TRIPLET, where TRIPLET is a triplet such as
>> +\"aarch64-linux-gnu\"."
>> +  (bag-transitive-inputs (package->bag package system target)))
>
> [...]
>
>>    ;; Remove non-package inputs such as origin records.
>>    (filter-map input->manifest-entry
>> -              (bag-transitive-inputs (package->bag package))))
>> +              (package-development-inputs package system)))
>
> Why ’system’ is used here?  Why is not simply?
>
>  +              (package-development-inputs package)))
>
> because ’system’ should be not defined, or I do not know where IIUC.

It’s an optional parameter above.

Ludo’.

[bug#50960] [PATCH v2 01/11] packages: Add 'package-development-inputs'.

[zimoun]

On Tue, 12 Oct 2021 at 11:54, Ludovic Courtès <ludo@gnu.org> wrote:
> zimoun <zimon.toutoune@gmail.com> skribis:
>
>>> +(define* (package-development-inputs package
>>> +                                     #:optional (system (%current-system))
>>> +                                     #:key target)
>>> +  "Return the list of inputs required by PACKAGE for development purposes on
>>> +SYSTEM.  When TARGET is true, return the inputs needed to cross-compile
>>> +PACKAGE from SYSTEM to TRIPLET, where TRIPLET is a triplet such as
>>> +\"aarch64-linux-gnu\"."
>>> +  (bag-transitive-inputs (package->bag package system target)))
>>
>> [...]
>>
>>>    ;; Remove non-package inputs such as origin records.
>>>    (filter-map input->manifest-entry
>>> -              (bag-transitive-inputs (package->bag package))))
>>> +              (package-development-inputs package system)))
>>
>> Why ’system’ is used here?  Why is not simply?
>>
>>  +              (package-development-inputs package)))
>>
>> because ’system’ should be not defined, or I do not know where IIUC.
>
> It’s an optional parameter above.

Sorry if I miss, to me, ’system’ is not bounded when calling:

--8<---------------cut here---------------start------------->8---
   ;; Remove non-package inputs such as origin records.
   (filter-map input->manifest-entry
-              (bag-transitive-inputs (package->bag package))))
+              (package-development-inputs package system)))
--8<---------------cut here---------------end--------------->8---

and it seems bounded to something:

    ;;; (#<procedure system (#:optional _)>)”

and I do not know from where this definition comes.

Maybe I misread what optional means.  From my understanding, it means:

    (package-development-inputs foo)

will use ’(%current-system)’ as default.  And it also means that:

    (package-development-inputs foo system)

where ’system’ is bounded to something and from my understanding this
something is not necessary ’(%current-system)’.


Again, sorry if I misread something.

Cheers,
simon

[bug#50960] [PATCH v2 02/11] profiles: Add 'package->development-manifest'.

[Ludovic Courtès]

From: Ludovic Courtès <ludovic.courtes@inria.fr>

* guix/profiles.scm (package->development-manifest): New procedure.
* guix/scripts/environment.scm (input->manifest-entry)
(package-environment-inputs): Remove.
* guix/scripts/environment.scm (options/resolve-packages): Use
'package->development-manifest' instead of 'package-environment-inputs'.
* tests/profiles.scm ("package->development-manifest"): New test.
---
 doc/guix.texi                | 11 +++++++++++
 guix/profiles.scm            | 19 +++++++++++++++++++
 guix/scripts/environment.scm | 27 +++++----------------------
 tests/profiles.scm           |  7 +++++++
 4 files changed, 42 insertions(+), 22 deletions(-)

diff --git a/doc/guix.texi b/doc/guix.texi
index 49399e792b..bc3f5a537b 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -3340,6 +3340,17 @@ objects, like this:
  '("emacs" "guile@@2.2" "guile@@2.2:debug"))
 @end lisp
 
+@findex package->development-manifest
+You might also want to create a manifest for all the dependencies of a
+package, rather than the package itself:
+
+@lisp
+(package->development-manifest (specification->package "emacs"))
+@end lisp
+
+The example above gives you all the software required to develop Emacs,
+similar to what @command{guix environment emacs} provides.
+
 @xref{export-manifest, @option{--export-manifest}}, to learn how to
 obtain a manifest file from an existing profile.
 
diff --git a/guix/profiles.scm b/guix/profiles.scm
index 2486f91d09..9f30349c69 100644
--- a/guix/profiles.scm
+++ b/guix/profiles.scm
@@ -124,6 +124,7 @@ (define-module (guix profiles)
 
             profile-manifest
             package->manifest-entry
+            package->development-manifest
             packages->manifest
             ca-certificate-bundle
             %default-profile-hooks
@@ -400,6 +401,24 @@ (define* (package->manifest-entry package #:optional (output "out")
                      (properties properties))))
     entry))
 
+(define* (package->development-manifest package
+                                        #:optional
+                                        (system (%current-system))
+                                        #:key target)
+  "Return a manifest for the \"development inputs\" of PACKAGE for SYSTEM,
+optionally when cross-compiling to TARGET.  Development inputs include both
+explicit and implicit inputs of PACKAGE."
+  (manifest
+   (filter-map (match-lambda
+                 ((label (? package? package))
+                  (package->manifest-entry package))
+                 ((label (? package? package) output)
+                  (package->manifest-entry package output))
+                 ;; TODO: Support <inferior-package>.
+                 (_
+                  #f))
+               (package-development-inputs package system #:target target))))
+
 (define (packages->manifest packages)
   "Return a list of manifest entries, one for each item listed in PACKAGES.
 Elements of PACKAGES can be either package objects or package/string tuples
diff --git a/guix/scripts/environment.scm b/guix/scripts/environment.scm
index d555969b27..54f48a7482 100644
--- a/guix/scripts/environment.scm
+++ b/guix/scripts/environment.scm
@@ -66,24 +66,6 @@ (define* (show-search-paths profile manifest #:key pure?)
                (newline)))
             (profile-search-paths profile manifest)))
 
-(define (input->manifest-entry input)
-  "Return a manifest entry for INPUT, or #f if INPUT does not correspond to a
-package."
-  (match input
-    ((_ (? package? package))
-     (package->manifest-entry package))
-    ((_ (? package? package) output)
-     (package->manifest-entry package output))
-    (_
-     #f)))
-
-(define (package-environment-inputs package)
-  "Return a list of manifest entries corresponding to the transitive input
-packages for PACKAGE."
-  ;; Remove non-package inputs such as origin records.
-  (filter-map input->manifest-entry
-              (package-development-inputs package system)))
-
 (define (show-help)
   (display (G_ "Usage: guix environment [OPTION]... PACKAGE... [-- COMMAND...]
 Build an environment that includes the dependencies of PACKAGE and execute
@@ -297,11 +279,11 @@ (define (packages->outputs packages mode)
       ((? package? package)
        (if (eq? mode 'ad-hoc-package)
            (list (package->manifest-entry* package))
-           (package-environment-inputs package)))
+           (manifest-entries (package->development-manifest package))))
       (((? package? package) (? string? output))
        (if (eq? mode 'ad-hoc-package)
            (list (package->manifest-entry* package output))
-           (package-environment-inputs package)))
+           (manifest-entries (package->development-manifest package))))
       ((lst ...)
        (append-map (cut packages->outputs <> mode) lst))))
 
@@ -313,8 +295,9 @@ (define (packages->outputs packages mode)
                                  (specification->package+output spec)))
                      (list (package->manifest-entry* package output))))
                   (('package 'package (? string? spec))
-                   (package-environment-inputs
-                    (transform (specification->package+output spec))))
+                   (manifest-entries
+                    (package->development-manifest
+                     (transform (specification->package+output spec)))))
                   (('expression mode str)
                    ;; Add all the outputs of the package STR evaluates to.
                    (packages->outputs (read/eval str) mode))
diff --git a/tests/profiles.scm b/tests/profiles.scm
index 06a0387221..cac5b73347 100644
--- a/tests/profiles.scm
+++ b/tests/profiles.scm
@@ -265,6 +265,13 @@ (define transform1
            (manifest-transaction-removal-candidate? guile-2.0.9 t)
            (null? install) (null? downgrade) (null? upgrade)))))
 
+(test-assert "package->development-manifest"
+  (let ((manifest (package->development-manifest packages:hello)))
+    (every (lambda (name)
+             (manifest-installed? manifest
+                                  (manifest-pattern (name name))))
+           '("gcc" "binutils" "glibc" "coreutils" "grep" "sed"))))
+
 (test-assertm "profile-derivation"
   (mlet* %store-monad
       ((entry ->   (package->manifest-entry %bootstrap-guile))
-- 
2.33.0

[bug#50960] [PATCH v2 02/11] profiles: Add 'package->development-manifest'.

[zimoun]

Hi Ludo,

On Mon, 11 Oct 2021 at 23:38, Ludovic Courtès <ludo@gnu.org> wrote:
> From: Ludovic Courtès <ludovic.courtes@inria.fr>
>
> * guix/profiles.scm (package->development-manifest): New procedure.
> * guix/scripts/environment.scm (input->manifest-entry)
> (package-environment-inputs): Remove.
> * guix/scripts/environment.scm (options/resolve-packages): Use
> 'package->development-manifest' instead of 'package-environment-inputs'.
> * tests/profiles.scm ("package->development-manifest"): New test.

LGTM!

(These patch 1 and 2 could be applied independently of the new “guix
shell” command. :-))


Cheers,
simon

[bug#50960] [PATCH v2 02/11] profiles: Add 'package->development-manifest'.

[Ludovic Courtès]

Hi!

zimoun <zimon.toutoune@gmail.com> skribis:

> (These patch 1 and 2 could be applied independently of the new “guix
> shell” command. :-))

Yes, definitely; actually most of the patches are independent of ‘guix
shell’.

Ludo’.

[bug#50960] [PATCH v2 03/11] Add 'guix shell'.

[Ludovic Courtès]

From: Ludovic Courtès <ludovic.courtes@inria.fr>

* guix/scripts/shell.scm, tests/guix-shell.sh: New files.
* Makefile.am (MODULES): Add 'shell.scm'.
(SH_TESTS): Add 'tests/guix-shell.sh'.
* guix/scripts/environment.scm (show-environment-options-help): New
procedure.
(show-help): Use it.
(guix-environment*): New procedure.
(guix-environment): Use it.
* po/guix/POTFILES.in: Add it.
* doc/guix.texi (Features): Refer to "guix shell"
(Invoking guix package): Likewise.
(Development): Likewise.
(Invoking guix shell): New node.
(Invoking guix environment): Add deprecation warning.
(Debugging Build Failures): Use 'guix shell' in examples.
(Invoking guix container): Refer to 'guix shell'.
(Invoking guix processes, Virtualization Services): Adjust examples to
use 'guix shell'.
* doc/contributing.texi (Building from Git): Refer to 'guix shell'.
---
 Makefile.am                  |   2 +
 doc/contributing.texi        |   8 +-
 doc/guix.texi                | 366 ++++++++++++++++++++++++++++++++---
 guix/scripts/environment.scm |  52 +++--
 guix/scripts/shell.scm       | 135 +++++++++++++
 po/guix/POTFILES.in          |   1 +
 tests/guix-shell.sh          |  54 ++++++
 7 files changed, 575 insertions(+), 43 deletions(-)
 create mode 100644 guix/scripts/shell.scm
 create mode 100644 tests/guix-shell.sh

diff --git a/Makefile.am b/Makefile.am
index b66789fa0b..c28c8799ec 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -315,6 +315,7 @@ MODULES =					\
   guix/scripts/import/stackage.scm		\
   guix/scripts/import/texlive.scm  		\
   guix/scripts/environment.scm			\
+  guix/scripts/shell.scm			\
   guix/scripts/publish.scm			\
   guix/scripts/edit.scm				\
   guix/scripts/size.scm				\
@@ -550,6 +551,7 @@ SH_TESTS =					\
   tests/guix-authenticate.sh			\
   tests/guix-environment.sh			\
   tests/guix-environment-container.sh		\
+  tests/guix-shell.sh				\
   tests/guix-graph.sh				\
   tests/guix-describe.sh			\
   tests/guix-repl.sh     			\
diff --git a/doc/contributing.texi b/doc/contributing.texi
index fbb3c47c78..363bbe9278 100644
--- a/doc/contributing.texi
+++ b/doc/contributing.texi
@@ -73,10 +73,10 @@ all the dependencies and appropriate environment variables are set up to
 hack on Guix:
 
 @example
-guix environment guix --pure
+guix shell -D guix --pure
 @end example
 
-@xref{Invoking guix environment}, for more information on that command.
+@xref{Invoking guix shell}, for more information on that command.
 
 If you are unable to use Guix when building Guix from a checkout, the
 following are the required packages in addition to those mentioned in the
@@ -92,10 +92,10 @@ installation instructions (@pxref{Requirements}).
 @end itemize
 
 On Guix, extra dependencies can be added by instead running @command{guix
-environment} with @option{--ad-hoc}:
+shell}:
 
 @example
-guix environment guix --pure --ad-hoc help2man git strace
+guix shell -D guix help2man git strace --pure
 @end example
 
 Run @command{./bootstrap} to generate the build system infrastructure
diff --git a/doc/guix.texi b/doc/guix.texi
index bc3f5a537b..b0d745b9e3 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -118,6 +118,7 @@ Documentation License''.
 
 @dircategory Software development
 @direntry
+* guix shell: (guix)Invoking guix shell.      Creating software environments.
 * guix environment: (guix)Invoking guix environment.  Building development environments with Guix.
 * guix build: (guix)Invoking guix build.      Building packages.
 * guix pack: (guix)Invoking guix pack.        Creating binary bundles.
@@ -261,6 +262,7 @@ Channels
 
 Development
 
+* Invoking guix shell::         Spawning one-off software environments.
 * Invoking guix environment::   Setting up development environments.
 * Invoking guix pack::          Creating software bundles.
 * The GCC toolchain::           Working with languages supported by GCC.
@@ -3066,10 +3068,10 @@ substitutes: they can force a local build and @emph{challenge} providers
 (@pxref{Invoking guix challenge}).
 
 Control over the build environment is a feature that is also useful for
-developers.  The @command{guix environment} command allows developers of
+developers.  The @command{guix shell} command allows developers of
 a package to quickly set up the right development environment for their
 package, without having to manually install the dependencies of the
-package into their profile (@pxref{Invoking guix environment}).
+package into their profile (@pxref{Invoking guix shell}).
 
 @cindex replication, of software environments
 @cindex provenance tracking, of software artifacts
@@ -3233,7 +3235,7 @@ As an example, @var{file} might contain a definition like this
 Developers may find it useful to include such a @file{guix.scm} file
 in the root of their project source tree that can be used to test
 development snapshots and create reproducible development environments
-(@pxref{Invoking guix environment}).
+(@pxref{Invoking guix shell}).
 
 The @var{file} may also contain a JSON representation of one or more
 package definitions.  Running @code{guix package -f} on
@@ -5558,31 +5560,352 @@ If you are a software developer, Guix provides tools that you should find
 helpful---independently of the language you're developing in.  This is what
 this chapter is about.
 
-The @command{guix environment} command provides a convenient way to set up
-@dfn{development environments} containing all the dependencies and tools
-necessary to work on the software package of your choice.  The @command{guix
+The @command{guix shell} command provides a convenient way to set up
+one-off software environments, be it for development purposes or to run
+a command without installing it in your profile.  The @command{guix
 pack} command allows you to create @dfn{application bundles} that can be
 easily distributed to users who do not run Guix.
 
 @menu
+* Invoking guix shell::         Spawning one-off software environments.
 * Invoking guix environment::   Setting up development environments.
 * Invoking guix pack::          Creating software bundles.
 * The GCC toolchain::           Working with languages supported by GCC.
 * Invoking guix git authenticate:: Authenticating Git repositories.
 @end menu
 
-@node Invoking guix environment
-@section Invoking @command{guix environment}
+@node Invoking guix shell
+@section Invoking @command{guix shell}
 
 @cindex reproducible build environments
 @cindex development environments
 @cindex @command{guix environment}
 @cindex environment, package build environment
-The purpose of @command{guix environment} is to assist hackers in
-creating reproducible development environments without polluting their
-package profile.  The @command{guix environment} tool takes one or more
-packages, builds all of their inputs, and creates a shell
-environment to use them.
+The purpose of @command{guix shell} is to make it easy to create one-off
+software environments, without changing one's profile.  It is typically
+used to create development environments; it is also a convenient way to
+run applications without ``polluting'' your profile.
+
+@quotation Note
+The @command{guix shell} command was recently introduced to supersede
+@command{guix environment} (@pxref{Invoking guix environment}).  If you
+are familiar with @command{guix environment}, you will notice that it is
+similar but also---we hope!---more convenient.
+@end quotation
+
+The general syntax is:
+
+@example
+guix shell [@var{options}] [@var{package}@dots{}]
+@end example
+
+The following example creates a environment containing Python and NumPy,
+building and downloading any missing package, and runs the
+@command{python3} command in that environment:
+
+@example
+guix shell python python-numpy -- python3
+@end example
+
+Development environments can be created as in the example below, which
+spawns an interactive shell containing all the dependencies and
+environment variables needed to work on Inkscape:
+
+@example
+guix shell --development inkscape
+@end example
+
+Exiting the shell places the user back in the original environment
+before @command{guix shell} was invoked.  The next garbage collection
+(@pxref{Invoking guix gc}) may clean up packages that were installed in
+the environment and that are no longer used outside of it.
+
+By default, the shell session or command runs in an @emph{augmented}
+environment, where the new packages are added to search path environment
+variables such as @code{PATH}.  You can, instead, choose to create an
+@emph{isolated} environment containing nothing but the packages you
+asked for.  Passing the @option{--pure} option clears environment
+variable definitions found in the parent environment@footnote{Users
+sometimes wrongfully augment environment variables such as @env{PATH} in
+their @file{~/.bashrc} file.  As a consequence, when @command{guix
+environment} launches it, Bash may read @file{~/.bashrc}, thereby
+introducing ``impurities'' in these environment variables.  It is an
+error to define such environment variables in @file{.bashrc}; instead,
+they should be defined in @file{.bash_profile}, which is sourced only by
+log-in shells.  @xref{Bash Startup Files,,, bash, The GNU Bash Reference
+Manual}, for details on Bash start-up files.}; passing
+@option{--container} goes one step further by spawning a @dfn{container}
+isolated from the rest of the system:
+
+@example
+guix shell --container emacs gcc-toolchain
+@end example
+
+The command above spawns an interactive shell in a container when
+nothing but @code{emacs}, @code{gcc-toolchain}, and their dependencies
+is available.  The container lacks network access and shares no files
+other than the current working directory with the surrounding
+environment.  This is useful to prevent access to system-wide resources
+such as @file{/usr/bin} on foreign distros.
+
+This @option{--container} option can also prove useful if you wish to
+run a security-sensitive application, such as a web browser, in an
+isolated environment.  For example, the command below launches
+Ungoogled-Chromium in an isolated environment, this time sharing network
+access with the host and preserving its @code{DISPLAY} environment
+variable, but without even sharing the current directory:
+
+@example
+guix shell --container --network --no-cwd ungoogled-chromium \
+  --preserve='^DISPLAY$' -- chromium
+@end example
+
+@vindex GUIX_ENVIRONMENT
+@command{guix shell} defines the @env{GUIX_ENVIRONMENT}
+variable in the shell it spawns; its value is the file name of the
+profile of this environment.  This allows users to, say, define a
+specific prompt for development environments in their @file{.bashrc}
+(@pxref{Bash Startup Files,,, bash, The GNU Bash Reference Manual}):
+
+@example
+if [ -n "$GUIX_ENVIRONMENT" ]
+then
+    export PS1="\u@@\h \w [dev]\$ "
+fi
+@end example
+
+@noindent
+...@: or to browse the profile:
+
+@example
+$ ls "$GUIX_ENVIRONMENT/bin"
+@end example
+
+The available options are summarized below.
+
+@table @code
+@item --development
+@itemx -D
+Cause @command{guix shell} to include in the environment the
+dependencies of the following package rather than the package itself.
+This can be combined with other packages.  For instance, the command
+below starts an interactive shell containing the build-time dependencies
+of GNU@tie{}Guile, plus Autoconf, Automake, and Libtool:
+
+@example
+guix shell -D guile autoconf automake libtool
+@end example
+
+@item --expression=@var{expr}
+@itemx -e @var{expr}
+Create an environment for the package or list of packages that
+@var{expr} evaluates to.
+
+For example, running:
+
+@example
+guix shell -D -e '(@@ (gnu packages maths) petsc-openmpi)'
+@end example
+
+starts a shell with the environment for this specific variant of the
+PETSc package.
+
+Running:
+
+@example
+guix shell -e '(@@ (gnu) %base-packages)'
+@end example
+
+starts a shell with all the base system packages available.
+
+The above commands only use the default output of the given packages.
+To select other outputs, two element tuples can be specified:
+
+@example
+guix shell -e '(list (@@ (gnu packages bash) bash) "include")'
+@end example
+
+@item --file=@var{file}
+@itemx -f @var{file}
+Create an environment containing the package or list of packages that
+the code within @var{file} evaluates to.
+
+As an example, @var{file} might contain a definition like this
+(@pxref{Defining Packages}):
+
+@lisp
+@verbatiminclude environment-gdb.scm
+@end lisp
+
+With the file above, you can enter a development environment for GDB by
+running:
+
+@example
+guix shell -D -f gdb-devel.scm
+@end example
+
+@item --manifest=@var{file}
+@itemx -m @var{file}
+Create an environment for the packages contained in the manifest object
+returned by the Scheme code in @var{file}.  This option can be repeated
+several times, in which case the manifests are concatenated.
+
+This is similar to the same-named option in @command{guix package}
+(@pxref{profile-manifest, @option{--manifest}}) and uses the same
+manifest files.
+
+@item --pure
+Unset existing environment variables when building the new environment, except
+those specified with @option{--preserve} (see below).  This has the effect of
+creating an environment in which search paths only contain package inputs.
+
+@item --preserve=@var{regexp}
+@itemx -E @var{regexp}
+When used alongside @option{--pure}, preserve the environment variables
+matching @var{regexp}---in other words, put them on a ``white list'' of
+environment variables that must be preserved.  This option can be repeated
+several times.
+
+@example
+guix shell --pure --preserve=^SLURM openmpi @dots{} \
+  -- mpirun @dots{}
+@end example
+
+This example runs @command{mpirun} in a context where the only environment
+variables defined are @env{PATH}, environment variables whose name starts
+with @samp{SLURM}, as well as the usual ``precious'' variables (@env{HOME},
+@env{USER}, etc.).
+
+@item --search-paths
+Display the environment variable definitions that make up the
+environment.
+
+@item --system=@var{system}
+@itemx -s @var{system}
+Attempt to build for @var{system}---e.g., @code{i686-linux}.
+
+@item --container
+@itemx -C
+@cindex container
+Run @var{command} within an isolated container.  The current working
+directory outside the container is mapped inside the container.
+Additionally, unless overridden with @option{--user}, a dummy home
+directory is created that matches the current user's home directory, and
+@file{/etc/passwd} is configured accordingly.
+
+The spawned process runs as the current user outside the container.  Inside
+the container, it has the same UID and GID as the current user, unless
+@option{--user} is passed (see below).
+
+@item --network
+@itemx -N
+For containers, share the network namespace with the host system.
+Containers created without this flag only have access to the loopback
+device.
+
+@item --link-profile
+@itemx -P
+For containers, link the environment profile to @file{~/.guix-profile}
+within the container and set @code{GUIX_ENVIRONMENT} to that.
+This is equivalent to making @file{~/.guix-profile} a symlink to the
+actual profile within the container.
+Linking will fail and abort the environment if the directory already
+exists, which will certainly be the case if @command{guix shell}
+was invoked in the user's home directory.
+
+Certain packages are configured to look in @file{~/.guix-profile} for
+configuration files and data;@footnote{For example, the
+@code{fontconfig} package inspects @file{~/.guix-profile/share/fonts}
+for additional fonts.}  @option{--link-profile} allows these programs to
+behave as expected within the environment.
+
+@item --user=@var{user}
+@itemx -u @var{user}
+For containers, use the username @var{user} in place of the current
+user.  The generated @file{/etc/passwd} entry within the container will
+contain the name @var{user}, the home directory will be
+@file{/home/@var{user}}, and no user GECOS data will be copied.  Furthermore,
+the UID and GID inside the container are 1000.  @var{user}
+need not exist on the system.
+
+Additionally, any shared or exposed path (see @option{--share} and
+@option{--expose} respectively) whose target is within the current user's
+home directory will be remapped relative to @file{/home/USER}; this
+includes the automatic mapping of the current working directory.
+
+@example
+# will expose paths as /home/foo/wd, /home/foo/test, and /home/foo/target
+cd $HOME/wd
+guix shell --container --user=foo \
+     --expose=$HOME/test \
+     --expose=/tmp/target=$HOME/target
+@end example
+
+While this will limit the leaking of user identity through home paths
+and each of the user fields, this is only one useful component of a
+broader privacy/anonymity solution---not one in and of itself.
+
+@item --no-cwd
+For containers, the default behavior is to share the current working
+directory with the isolated container and immediately change to that
+directory within the container.  If this is undesirable,
+@option{--no-cwd} will cause the current working directory to @emph{not}
+be automatically shared and will change to the user's home directory
+within the container instead.  See also @option{--user}.
+
+@item --expose=@var{source}[=@var{target}]
+@itemx --share=@var{source}[=@var{target}]
+For containers, @option{--expose} (resp. @option{--share}) exposes the
+file system @var{source} from the host system as the read-only
+(resp. writable) file system @var{target} within the container.  If
+@var{target} is not specified, @var{source} is used as the target mount
+point in the container.
+
+The example below spawns a Guile REPL in a container in which the user's
+home directory is accessible read-only via the @file{/exchange}
+directory:
+
+@example
+guix shell --container --expose=$HOME=/exchange guile -- guile
+@end example
+
+@item --root=@var{file}
+@itemx -r @var{file}
+@cindex persistent environment
+@cindex garbage collector root, for environments
+Make @var{file} a symlink to the profile for this environment, and
+register it as a garbage collector root.
+
+This is useful if you want to protect your environment from garbage
+collection, to make it ``persistent''.
+
+When this option is omitted, the environment is protected from garbage
+collection only for the duration of the @command{guix shell}
+session.  This means that next time you recreate the same environment,
+you could have to rebuild or re-download packages.  @xref{Invoking guix
+gc}, for more on GC roots.
+@end table
+
+@command{guix shell} also supports all of the common build options that
+@command{guix build} supports (@pxref{Common Build Options}) as well as
+package transformation options (@pxref{Package Transformation Options}).
+
+@node Invoking guix environment
+@section Invoking @command{guix environment}
+
+The purpose of @command{guix environment} is to assists in creating
+development environments.
+
+@quotation Deprecation warning
+The @command{guix environment} command is deprecated in favor of
+@command{guix shell}, which performs similar functions but is more
+convenient to use.  @xref{Invoking guix shell}.
+
+Being deprecated, @command{guix environment} is slated for eventual
+removal, but the Guix project is committed to keeping it until May 1st,
+2023.  Please get in touch with us at @email{guix-devel@@gnu.org} if you
+would like to discuss it.
+@end quotation
 
 The general syntax is:
 
@@ -11098,14 +11421,14 @@ a container similar to the one the build daemon creates:
 $ guix build -K foo
 @dots{}
 $ cd /tmp/guix-build-foo.drv-0
-$ guix environment --no-grafts -C foo --ad-hoc strace gdb
+$ guix shell --no-grafts -C foo strace gdb
 [env]# source ./environment-variables
 [env]# cd foo-1.2
 @end example
 
 Here, @command{guix environment -C} creates a container and spawns a new
-shell in it (@pxref{Invoking guix environment}).  The @command{--ad-hoc
-strace gdb} part adds the @command{strace} and @command{gdb} commands to
+shell in it (@pxref{Invoking guix shell}).  The @command{strace gdb}
+part adds the @command{strace} and @command{gdb} commands to
 the container, which you may find handy while debugging.  The
 @option{--no-grafts} option makes sure we get the exact same
 environment, with ungrafted packages (@pxref{Security Updates}, for more
@@ -11119,7 +11442,7 @@ remove @file{/bin/sh}:
 @end example
 
 (Don't worry, this is harmless: this is all happening in the throw-away
-container created by @command{guix environment}.)
+container created by @command{guix shell}.)
 
 The @command{strace} command is probably not in the search path, but we
 can run:
@@ -13315,8 +13638,8 @@ is subject to radical change in the future.
 
 The purpose of @command{guix container} is to manipulate processes
 running within an isolated environment, commonly known as a
-``container'', typically created by the @command{guix environment}
-(@pxref{Invoking guix environment}) and @command{guix system container}
+``container'', typically created by the @command{guix shell}
+(@pxref{Invoking guix shell}) and @command{guix system container}
 (@pxref{Invoking guix system}) commands.
 
 The general syntax is:
@@ -13502,7 +13825,7 @@ listed.}.  Here's an example of the information it returns:
 $ sudo guix processes
 SessionPID: 19002
 ClientPID: 19090
-ClientCommand: guix environment --ad-hoc python
+ClientCommand: guix shell python
 
 SessionPID: 19402
 ClientPID: 19367
@@ -29697,8 +30020,7 @@ When the service is running, you can view its console by connecting to
 it with a VNC client, for example with:
 
 @example
-guix environment --ad-hoc tigervnc-client -- \
-         vncviewer localhost:5900
+guix shell tigervnc-client -- vncviewer localhost:5900
 @end example
 
 The default configuration (see @code{hurd-vm-configuration} below)
diff --git a/guix/scripts/environment.scm b/guix/scripts/environment.scm
index 54f48a7482..77956fc018 100644
--- a/guix/scripts/environment.scm
+++ b/guix/scripts/environment.scm
@@ -50,7 +50,11 @@ (define-module (guix scripts environment)
   #:use-module (srfi srfi-37)
   #:use-module (srfi srfi-98)
   #:export (assert-container-features
-            guix-environment))
+            guix-environment
+            guix-environment*
+            show-environment-options-help
+            (%options . %environment-options)
+            (%default-options . %environment-default-options)))
 
 (define %default-shell
   (or (getenv "SHELL") "/bin/sh"))
@@ -66,23 +70,16 @@ (define* (show-search-paths profile manifest #:key pure?)
                (newline)))
             (profile-search-paths profile manifest)))
 
-(define (show-help)
-  (display (G_ "Usage: guix environment [OPTION]... PACKAGE... [-- COMMAND...]
-Build an environment that includes the dependencies of PACKAGE and execute
-COMMAND or an interactive shell in that environment.\n"))
+(define (show-environment-options-help)
+  "Print help about options shared between 'guix environment' and 'guix
+shell'."
   (display (G_ "
   -e, --expression=EXPR  create environment for the package that EXPR
                          evaluates to"))
   (display (G_ "
-  -l, --load=FILE        create environment for the package that the code within
-                         FILE evaluates to"))
-  (display (G_ "
   -m, --manifest=FILE    create environment with the manifest from FILE"))
   (display (G_ "
   -p, --profile=PATH     create environment from profile at PATH"))
-  (display (G_ "
-      --ad-hoc           include all specified packages in the environment instead
-                         of only their inputs"))
   (display (G_ "
       --pure             unset existing environment variables"))
   (display (G_ "
@@ -118,7 +115,24 @@ (define (show-help)
   (display (G_ "
   -v, --verbosity=LEVEL  use the given verbosity LEVEL"))
   (display (G_ "
-      --bootstrap        use bootstrap binaries to build the environment"))
+      --bootstrap        use bootstrap binaries to build the environment")))
+
+(define (show-help)
+  (display (G_ "Usage: guix environment [OPTION]... PACKAGE... [-- COMMAND...]
+Build an environment that includes the dependencies of PACKAGE and execute
+COMMAND or an interactive shell in that environment.\n"))
+  (warning (G_ "This command is deprecated in favor of 'guix shell'.\n"))
+  (newline)
+
+  ;; These two options are left out in 'guix shell'.
+  (display (G_ "
+  -l, --load=FILE        create environment for the package that the code within
+                         FILE evaluates to"))
+  (display (G_ "
+      --ad-hoc           include all specified packages in the environment instead
+                         of only their inputs"))
+
+  (show-environment-options-help)
   (newline)
   (show-build-options-help)
   (newline)
@@ -649,11 +663,15 @@ (define (register-gc-root target root)
 
 (define-command (guix-environment . args)
   (category development)
-  (synopsis "spawn one-off software environments")
+  (synopsis "spawn one-off software environments (deprecated)")
 
+  (guix-environment* (parse-args args)))
+
+(define (guix-environment* opts)
+  "Run the 'guix environment' command on OPTS, an alist resulting for
+command-line option processing with 'parse-command-line'."
   (with-error-handling
-    (let* ((opts       (parse-args args))
-           (pure?      (assoc-ref opts 'pure))
+    (let* ((pure?      (assoc-ref opts 'pure))
            (container? (assoc-ref opts 'container?))
            (link-prof? (assoc-ref opts 'link-profile?))
            (network?   (assoc-ref opts 'network?))
@@ -724,8 +742,8 @@ (define manifest
                                      (prof-drv   (manifest->derivation
                                                   manifest system bootstrap?))
                                      (profile -> (if profile
-                                                   (readlink* profile)
-                                                   (derivation->output-path prof-drv)))
+                                                     (readlink* profile)
+                                                     (derivation->output-path prof-drv)))
                                      (gc-root -> (assoc-ref opts 'gc-root)))
 
                   ;; First build the inputs.  This is necessary even for
diff --git a/guix/scripts/shell.scm b/guix/scripts/shell.scm
new file mode 100644
index 0000000000..190dd8837d
--- /dev/null
+++ b/guix/scripts/shell.scm
@@ -0,0 +1,135 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2021 Ludovic Courtès <ludo@gnu.org>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (guix scripts shell)
+  #:use-module (guix ui)
+  #:use-module (guix scripts environment)
+  #:autoload   (guix scripts build) (show-build-options-help)
+  #:autoload   (guix transformations) (show-transformation-options-help)
+  #:use-module (guix scripts)
+  #:use-module (srfi srfi-1)
+  #:use-module (srfi srfi-26)
+  #:use-module (srfi srfi-37)
+  #:use-module (srfi srfi-71)
+  #:use-module (ice-9 match)
+  #:export (guix-shell))
+
+(define (show-help)
+  (display (G_ "Usage: guix shell [OPTION] PACKAGES... [-- COMMAND...]
+Build an environment that includes PACKAGES and execute COMMAND or an
+interactive shell in that environment.\n"))
+  (newline)
+
+  ;; These two options differ from 'guix environment'.
+  (display (G_ "
+  -D, --development      include the development inputs of the next package"))
+  (display (G_ "
+  -f, --file=FILE        create environment for the package that the code within
+                         FILE evaluates to"))
+
+  (show-environment-options-help)
+  (newline)
+  (show-build-options-help)
+  (newline)
+  (show-transformation-options-help)
+  (newline)
+  (display (G_ "
+  -h, --help             display this help and exit"))
+  (display (G_ "
+  -V, --version          display version information and exit"))
+  (newline)
+  (show-bug-report-information))
+
+(define (tag-package-arg opts arg)
+  "Return a two-element list with the form (TAG ARG) that tags ARG with either
+'ad-hoc' in OPTS has the 'ad-hoc?' key set to #t, or 'inputs' otherwise."
+  (if (assoc-ref opts 'ad-hoc?)
+      `(ad-hoc-package ,arg)
+      `(package ,arg)))
+
+(define (ensure-ad-hoc alist)
+  (if (assq-ref alist 'ad-hoc?)
+      alist
+      `((ad-hoc? . #t) ,@alist)))
+
+(define (wrapped-option opt)
+  "Wrap OPT, a SRFI-37 option, such that its processor always adds the
+'ad-hoc?' flag to the resulting alist."
+  (option (option-names opt)
+          (option-required-arg? opt)
+          (option-optional-arg? opt)
+          (compose ensure-ad-hoc (option-processor opt))))
+
+(define %options
+  ;; Specification of the command-line options.
+  (let ((to-remove '("ad-hoc" "inherit" "load" "help" "version")))
+    (append
+        (list (option '(#\h "help") #f #f
+                      (lambda args
+                        (show-help)
+                        (exit 0)))
+              (option '(#\V "version") #f #f
+                      (lambda args
+                        (show-version-and-exit "guix shell")))
+
+              (option '(#\D "development") #f #f
+                      (lambda (opt name arg result)
+                        ;; Temporarily remove the 'ad-hoc?' flag from result.
+                        ;; The next option will put it back thanks to
+                        ;; 'wrapped-option'.
+                        (alist-delete 'ad-hoc? result)))
+
+              ;; For consistency with 'guix package', support '-f' rather than
+              ;; '-l' like 'guix environment' does.
+              (option '(#\f "file") #t #f
+                      (lambda (opt name arg result)
+                        (alist-cons 'load (tag-package-arg result arg)
+                                    result))))
+        (filter-map (lambda (opt)
+                      (and (not (any (lambda (name)
+                                       (member name to-remove))
+                                     (option-names opt)))
+                           (wrapped-option opt)))
+                    %environment-options))))
+
+(define %default-options
+  `((ad-hoc? . #t)                                ;always true
+    ,@%environment-default-options))
+
+(define (parse-args args)
+  "Parse the list of command line arguments ARGS."
+  (define (handle-argument arg result)
+    (alist-cons 'package (tag-package-arg result arg)
+                (ensure-ad-hoc result)))
+
+  ;; The '--' token is used to separate the command to run from the rest of
+  ;; the operands.
+  (let ((args command (break (cut string=? "--" <>) args)))
+    (let ((opts (parse-command-line args %options (list %default-options)
+                                    #:argument-handler handle-argument)))
+      (match command
+        (() opts)
+        (("--") opts)
+        (("--" command ...) (alist-cons 'exec command opts))))))
+
+\f
+(define-command (guix-shell . args)
+  (category development)
+  (synopsis "spawn one-off software environments")
+
+  (guix-environment* (parse-args args)))
diff --git a/po/guix/POTFILES.in b/po/guix/POTFILES.in
index f5b76bf582..f8abeb2d38 100644
--- a/po/guix/POTFILES.in
+++ b/po/guix/POTFILES.in
@@ -99,6 +99,7 @@ guix/derivations.scm
 guix/scripts/archive.scm
 guix/scripts/build.scm
 guix/scripts/environment.scm
+guix/scripts/shell.scm
 guix/scripts/time-machine.scm
 guix/scripts/import/cpan.scm
 guix/scripts/import/crate.scm
diff --git a/tests/guix-shell.sh b/tests/guix-shell.sh
new file mode 100644
index 0000000000..f08637f7ff
--- /dev/null
+++ b/tests/guix-shell.sh
@@ -0,0 +1,54 @@
+# GNU Guix --- Functional package management for GNU
+# Copyright © 2021 Ludovic Courtès <ludo@gnu.org>
+#
+# This file is part of GNU Guix.
+#
+# GNU Guix is free software; you can redistribute it and/or modify it
+# under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or (at
+# your option) any later version.
+#
+# GNU Guix is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
+
+#
+# Test the 'guix shell' alias.
+#
+
+guix shell --version
+
+tmpdir="t-guix-shell-$$"
+trap 'rm -r "$tmpdir"' EXIT
+mkdir "$tmpdir"
+
+guix shell --bootstrap --pure guile-bootstrap -- guile --version
+
+# '--ad-hoc' is a thing of the past.
+! guix shell --ad-hoc guile-bootstrap
+
+if guile -c '(getaddrinfo "www.gnu.org" "80" AI_NUMERICSERV)' 2> /dev/null
+then
+    # Compute the build environment for the initial GNU Make.
+    guix shell --bootstrap --no-substitutes --search-paths --pure \
+         -D -e '(@ (guix tests) gnu-make-for-tests)' > "$tmpdir/a"
+
+    # Make sure bootstrap binaries are in the profile.
+    profile=`grep "^export PATH" "$tmpdir/a" | sed -r 's|^.*="(.*)/bin"|\1|'`
+
+    # Make sure the bootstrap binaries are all listed where they belong.
+    grep -E "^export PATH=\"$profile/bin\""         "$tmpdir/a"
+    grep -E "^export CPATH=\"$profile/include\""    "$tmpdir/a"
+    grep -E "^export LIBRARY_PATH=\"$profile/lib\"" "$tmpdir/a"
+    for dep in bootstrap-binaries-0 gcc-bootstrap-0 glibc-bootstrap-0
+    do
+	guix gc --references "$profile" | grep "$dep"
+    done
+
+    # 'make-boot0' itself must not be listed.
+    ! guix gc --references "$profile" | grep make-boot0
+fi
-- 
2.33.0

[bug#50960] [PATCH v2 03/11] Add 'guix shell'.

[pelzflorian (Florian Pelz)]

Hello Ludo!

When I pull, I can’t use guix shell, perhaps because you did not add
it to guix/self.scm?

I like that the new documentation does not target only developers and
hackers.

On Mon, Oct 11, 2021 at 11:38:01PM +0200, Ludovic Courtès wrote:
> […]
> +The general syntax is:
> +
> +@example
> +guix shell [@var{options}] [@var{package}@dots{}]
> +@end example
> +
> +The following example creates a environment containing Python and NumPy,

*an* environment


> +building and downloading any missing package, and runs the

building *or* downloading


> […]
> +By default, the shell session or command runs in an @emph{augmented}
> +environment, where the new packages are added to search path environment
> +variables such as @code{PATH}.  You can, instead, choose to create an
> +@emph{isolated} environment containing nothing but the packages you

Hmmph.  I had seen the words “isolated environment” being used to
describe containers in "(guix)Managing Software the Guix Way".

Therefore I also had used the German term for “isolated” to explain
containers in other parts of the manual.  Probably this was not
necessary; if you apply guix shell with this term, I will change all
occurrences of “isoliert” in the German PO files.

But then again, maybe it would be better if you described pure
environments not as isolated.  I also see a mix-up further below:


> +This @option{--container} option can also prove useful if you wish to
> +run a security-sensitive application, such as a web browser, in an
> +isolated environment.  For example, the command below launches
> +Ungoogled-Chromium in an isolated environment, this time sharing network
> +access with the host and preserving its @code{DISPLAY} environment
> +variable, but without even sharing the current directory:
> +
> +@example
> +guix shell --container --network --no-cwd ungoogled-chromium \
> +  --preserve='^DISPLAY$' -- chromium
> +@end example
> […]

Back to typos:

> +@node Invoking guix environment
> +@section Invoking @command{guix environment}
> +
> +The purpose of @command{guix environment} is to assists in creating

is to *assist*

So much for the docs.  I have not looked further.

Regards,
Florian

[bug#50960] [PATCH v2 04/11] DRAFT shell: By default load the local 'guix.scm' or 'manifest.scm' file.

[Ludovic Courtès]

DRAFT: Add doc.

* guix/scripts/shell.scm (parse-args): Add call to 'auto-detect-manifest'.
(find-file-in-parent-directories, auto-detect-manifest): New procedures.
* tests/guix-shell.sh: Add test.
---
 guix/scripts/shell.scm | 67 ++++++++++++++++++++++++++++++++++++++----
 tests/guix-shell.sh    | 36 +++++++++++++++++++++++
 2 files changed, 98 insertions(+), 5 deletions(-)

diff --git a/guix/scripts/shell.scm b/guix/scripts/shell.scm
index 190dd8837d..39d843bde7 100644
--- a/guix/scripts/shell.scm
+++ b/guix/scripts/shell.scm
@@ -22,6 +22,8 @@ (define-module (guix scripts shell)
   #:autoload   (guix scripts build) (show-build-options-help)
   #:autoload   (guix transformations) (show-transformation-options-help)
   #:use-module (guix scripts)
+  #:use-module (guix packages)
+  #:use-module (guix profiles)
   #:use-module (srfi srfi-1)
   #:use-module (srfi srfi-26)
   #:use-module (srfi srfi-37)
@@ -41,6 +43,8 @@ (define (show-help)
   (display (G_ "
   -f, --file=FILE        create environment for the package that the code within
                          FILE evaluates to"))
+  (display (G_ "
+  -q                     inhibit loading of 'guix.scm' and 'manifest.scm'"))
 
   (show-environment-options-help)
   (newline)
@@ -99,7 +103,10 @@ (define %options
               (option '(#\f "file") #t #f
                       (lambda (opt name arg result)
                         (alist-cons 'load (tag-package-arg result arg)
-                                    result))))
+                                    result)))
+              (option '(#\q) #f #f
+                      (lambda (opt name arg result)
+                        (alist-cons 'explicit-loading? #t result))))
         (filter-map (lambda (opt)
                       (and (not (any (lambda (name)
                                        (member name to-remove))
@@ -122,10 +129,60 @@ (define (handle-argument arg result)
   (let ((args command (break (cut string=? "--" <>) args)))
     (let ((opts (parse-command-line args %options (list %default-options)
                                     #:argument-handler handle-argument)))
-      (match command
-        (() opts)
-        (("--") opts)
-        (("--" command ...) (alist-cons 'exec command opts))))))
+      (auto-detect-manifest
+       (match command
+         (() opts)
+         (("--") opts)
+         (("--" command ...) (alist-cons 'exec command opts)))))))
+
+(define (find-file-in-parent-directories candidates)
+  "Find one of CANDIDATES in the current directory or one of its ancestors."
+  (define start (getcwd))
+  (define device (stat:dev (stat start)))
+
+  (let loop ((directory start))
+    (let ((stat (stat directory)))
+      (and (= (stat:uid stat) (getuid))
+           (= (stat:dev stat) device)
+           (or (any (lambda (candidate)
+                      (let ((candidate (string-append directory "/" candidate)))
+                        (and (file-exists? candidate) candidate)))
+                    candidates)
+               (and (not (string=? directory "/"))
+                    (loop (dirname directory)))))))) ;lexical ".." resolution
+
+(define (auto-detect-manifest opts)
+  "If OPTS do not specify packages or a manifest, load a \"guix.scm\" or
+\"manifest.scm\" file from the current directory or one of its ancestors.
+Return the modified OPTS."
+  (define (options-contain-payload? opts)
+    (match opts
+      (() #f)
+      ((('package . _) . _) #t)
+      ((('load . _) . _) #t)
+      ((('manifest . _) . _) #t)
+      ((('expression . _) . _) #t)
+      ((_ . rest) (options-contain-payload? rest))))
+
+  (define interactive?
+    (not (assoc-ref opts 'exec)))
+
+  (define disallow-implicit-load?
+    (assoc-ref opts 'explicit-loading?))
+
+  (if (or (not interactive?)
+          disallow-implicit-load?
+          (options-contain-payload? opts))
+      opts
+      (match (find-file-in-parent-directories '("guix.scm" "manifest.scm"))
+        (#f
+         (warning (G_ "no packages specified; creating an empty environment~%"))
+         opts)
+        (file
+         (info (G_ "loading environment from '~a'...~%") file)
+         (match (basename file)
+           ("guix.scm" (alist-cons 'load `(package ,file) opts))
+           ("manifest.scm" (alist-cons 'manifest file opts)))))))
 
 \f
 (define-command (guix-shell . args)
diff --git a/tests/guix-shell.sh b/tests/guix-shell.sh
index f08637f7ff..0988ca0a75 100644
--- a/tests/guix-shell.sh
+++ b/tests/guix-shell.sh
@@ -31,6 +31,36 @@ guix shell --bootstrap --pure guile-bootstrap -- guile --version
 # '--ad-hoc' is a thing of the past.
 ! guix shell --ad-hoc guile-bootstrap
 
+# Ignoring 'manifest.scm' and 'guix.scm' in non-interactive use.
+cat > "$tmpdir/guix.scm" <<EOF
+This is a broken guix.scm file.
+EOF
+(cd "$tmpdir"; guix shell --bootstrap -- true)
+mv "$tmpdir/guix.scm" "$tmpdir/manifest.scm"
+(cd "$tmpdir"; guix shell --bootstrap -- true)
+rm "$tmpdir/manifest.scm"
+
+# Honoring the local 'manifest.scm' file.
+cat > "$tmpdir/manifest.scm" <<EOF
+(specifications->manifest '("guile-bootstrap"))
+EOF
+cat > "$tmpdir/fake-shell.sh" <<EOF
+#!$SHELL
+# This fake shell allows us to test interactive use.
+exec echo "\$GUIX_ENVIRONMENT"
+EOF
+chmod +x "$tmpdir/fake-shell.sh"
+profile1="$(cd "$tmpdir"; SHELL="$(realpath fake-shell.sh)" guix shell --bootstrap)"
+profile2="$(guix shell --bootstrap guile-bootstrap -- "$SHELL" -c 'echo $GUIX_ENVIRONMENT')"
+test -n "$profile1"
+test "$profile1" = "$profile2"
+rm "$tmpdir/manifest.scm"
+
+# Do not read manifest when passed '-q'.
+echo "Broken manifest." > "$tmpdir/manifest.scm"
+(cd "$tmpdir"; SHELL="$(realpath fake-shell.sh)" guix shell --bootstrap -q)
+rm "$tmpdir/manifest.scm"
+
 if guile -c '(getaddrinfo "www.gnu.org" "80" AI_NUMERICSERV)' 2> /dev/null
 then
     # Compute the build environment for the initial GNU Make.
@@ -51,4 +81,10 @@ then
 
     # 'make-boot0' itself must not be listed.
     ! guix gc --references "$profile" | grep make-boot0
+
+    # Honoring the local 'guix.scm' file.
+    echo '(@ (guix tests) gnu-make-for-tests)' > "$tmpdir/guix.scm"
+    (cd "$tmpdir"; guix shell --bootstrap --search-paths --pure > "b")
+    cmp "$tmpdir/a" "$tmpdir/b"
+    rm "$tmpdir/guix.scm"
 fi
-- 
2.33.0

[bug#50960] [PATCH v2 05/11] DRAFT shell: Honor in ~/.config/guix/shell-authorized-directories.

[Ludovic Courtès]

DRAFT: Squeeze with previous commit, or instead implement "guix shell ."
convention?

* guix/scripts/shell.scm (authorized-directory-file)
(authorized-shell-directory?): New procedure.
(auto-detect-manifest): Use it.
* doc/guix.texi (Invoking guix shell): Document it.
---
 doc/guix.texi          | 14 ++++++++++
 guix/scripts/shell.scm | 60 +++++++++++++++++++++++++++++++++++++++---
 tests/guix-shell.sh    | 16 ++++++++---
 3 files changed, 83 insertions(+), 7 deletions(-)

diff --git a/doc/guix.texi b/doc/guix.texi
index b0d745b9e3..b95025a39f 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -5620,6 +5620,20 @@ before @command{guix shell} was invoked.  The next garbage collection
 (@pxref{Invoking guix gc}) may clean up packages that were installed in
 the environment and that are no longer used outside of it.
 
+As an added convenience, when running from a directory that contains a
+@file{guix.scm} or a @file{manifest.scm} file, possibly in a parent
+directory, @command{guix shell} automatically loads the file---provided
+the directory is listed in
+@file{~/.config/guix/shell-authorized-directories}, and only for
+interactive use:
+
+@example
+guix shell
+@end example
+
+This provides an easy way to define, share, and enter development
+environments.
+
 By default, the shell session or command runs in an @emph{augmented}
 environment, where the new packages are added to search path environment
 variables such as @code{PATH}.  You can, instead, choose to create an
diff --git a/guix/scripts/shell.scm b/guix/scripts/shell.scm
index 39d843bde7..45fd536145 100644
--- a/guix/scripts/shell.scm
+++ b/guix/scripts/shell.scm
@@ -18,6 +18,7 @@
 
 (define-module (guix scripts shell)
   #:use-module (guix ui)
+  #:use-module ((guix diagnostics) #:select (location))
   #:use-module (guix scripts environment)
   #:autoload   (guix scripts build) (show-build-options-help)
   #:autoload   (guix transformations) (show-transformation-options-help)
@@ -29,6 +30,8 @@ (define-module (guix scripts shell)
   #:use-module (srfi srfi-37)
   #:use-module (srfi srfi-71)
   #:use-module (ice-9 match)
+  #:autoload   (ice-9 rdelim) (read-line)
+  #:autoload   (guix utils) (config-directory)
   #:export (guix-shell))
 
 (define (show-help)
@@ -151,6 +154,39 @@ (define device (stat:dev (stat start)))
                (and (not (string=? directory "/"))
                     (loop (dirname directory)))))))) ;lexical ".." resolution
 
+(define (authorized-directory-file)
+  "Return the name of the file listing directories for which 'guix shell' may
+automatically load 'guix.scm' or 'manifest.scm' files."
+  (string-append (config-directory) "/shell-authorized-directories"))
+
+(define (authorized-shell-directory? directory)
+  "Return true if DIRECTORY is among the authorized directories for automatic
+loading.  The list of authorized directories is read from
+'authorized-directory-file'; each line must be either: an absolute file name,
+a hash-prefixed comment, or a blank line."
+  (catch 'system-error
+    (lambda ()
+      (call-with-input-file (authorized-directory-file)
+        (lambda (port)
+          (let loop ()
+            (match (read-line port)
+              ((? eof-object?) #f)
+              ((= string-trim line)
+               (cond ((string-prefix? "#" line)   ;comment
+                      (loop))
+                     ((string-prefix? "/" line)   ;absolute file name
+                      (or (string=? line directory)
+                          (loop)))
+                     ((string-null? (string-trim-right line)) ;blank line
+                      (loop))
+                     (else                        ;bogus line
+                      (let ((loc (location (port-filename port)
+                                           (port-line port)
+                                           (port-column port))))
+                        (warning loc (G_ "ignoring invalid file name: '~a'~%")
+                                 line))))))))))
+    (const #f)))
+
 (define (auto-detect-manifest opts)
   "If OPTS do not specify packages or a manifest, load a \"guix.scm\" or
 \"manifest.scm\" file from the current directory or one of its ancestors.
@@ -179,10 +215,26 @@ (define disallow-implicit-load?
          (warning (G_ "no packages specified; creating an empty environment~%"))
          opts)
         (file
-         (info (G_ "loading environment from '~a'...~%") file)
-         (match (basename file)
-           ("guix.scm" (alist-cons 'load `(package ,file) opts))
-           ("manifest.scm" (alist-cons 'manifest file opts)))))))
+         (if (authorized-shell-directory? (dirname file))
+             (begin
+               (info (G_ "loading environment from '~a'...~%") file)
+               (match (basename file)
+                 ("guix.scm" (alist-cons 'load `(package ,file) opts))
+                 ("manifest.scm" (alist-cons 'manifest file opts))))
+             (begin
+               (warning (G_ "not loading '~a' because not authorized to do so~%")
+                        file)
+               (display-hint (format #f (G_ "To allow automatic loading of
+@file{~a} when running @command{guix shell}, you must explicitly authorize its
+directory, like so:
+
+@example
+echo ~a >> ~a
+@end example\n")
+                                     file
+                                     (dirname file)
+                                     (authorized-directory-file)))
+               opts))))))
 
 \f
 (define-command (guix-shell . args)
diff --git a/tests/guix-shell.sh b/tests/guix-shell.sh
index 0988ca0a75..95725cba2d 100644
--- a/tests/guix-shell.sh
+++ b/tests/guix-shell.sh
@@ -22,19 +22,29 @@
 
 guix shell --version
 
+configdir="t-guix-shell-config-$$"
 tmpdir="t-guix-shell-$$"
-trap 'rm -r "$tmpdir"' EXIT
-mkdir "$tmpdir"
+trap 'rm -r "$tmpdir" "$configdir"' EXIT
+mkdir "$tmpdir" "$configdir" "$configdir/guix"
+
+XDG_CONFIG_HOME="$(realpath $configdir)"
+export XDG_CONFIG_HOME
 
 guix shell --bootstrap --pure guile-bootstrap -- guile --version
 
 # '--ad-hoc' is a thing of the past.
 ! guix shell --ad-hoc guile-bootstrap
 
-# Ignoring 'manifest.scm' and 'guix.scm' in non-interactive use.
+# Ignoring unauthorized files.
 cat > "$tmpdir/guix.scm" <<EOF
 This is a broken guix.scm file.
 EOF
+(cd "$tmpdir"; SHELL="$(type -P true)" guix shell --bootstrap)
+
+# Authorize the directory.
+echo "$(realpath "$tmpdir")" > "$configdir/guix/shell-authorized-directories"
+
+# Ignoring 'manifest.scm' and 'guix.scm' in non-interactive use.
 (cd "$tmpdir"; guix shell --bootstrap -- true)
 mv "$tmpdir/guix.scm" "$tmpdir/manifest.scm"
 (cd "$tmpdir"; guix shell --bootstrap -- true)
-- 
2.33.0

[bug#50960] [PATCH v2 06/11] environment: Add tests for '--profile'.

[Ludovic Courtès]

This is a followup to a643deac2de81755a1843a3b41dd53857678bebc.

* tests/guix-environment-container.sh, tests/guix-environment.sh: Add
tests for '--profile'.
---
 tests/guix-environment-container.sh | 8 ++++++++
 tests/guix-environment.sh           | 7 +++++++
 2 files changed, 15 insertions(+)

diff --git a/tests/guix-environment-container.sh b/tests/guix-environment-container.sh
index f2d15c8d0c..2e238c501d 100644
--- a/tests/guix-environment-container.sh
+++ b/tests/guix-environment-container.sh
@@ -44,6 +44,14 @@ else
     test $? = 42
 fi
 
+# Try '--root' and '--profile'.
+root="$tmpdir/root"
+guix environment -C --ad-hoc --bootstrap guile-bootstrap -r "$root" -- guile --version
+guix environment -C -p "$root" --bootstrap -- guile --version
+path1=$(guix environment -C -p "$root" --bootstrap -- guile -c '(display (getenv "PATH"))')
+path2=$(guix environment -C --ad-hoc --bootstrap guile-bootstrap  -- guile -c '(display (getenv "PATH"))')
+test "$path1" = "$path2"
+
 # Make sure "localhost" resolves.
 guix environment --container --ad-hoc --bootstrap guile-bootstrap \
      -- guile -c '(exit (pair? (getaddrinfo "localhost" "80")))'
diff --git a/tests/guix-environment.sh b/tests/guix-environment.sh
index afadcbe195..f4fc2e39ed 100644
--- a/tests/guix-environment.sh
+++ b/tests/guix-environment.sh
@@ -119,6 +119,13 @@ test `readlink "$gcroot"` = "$expected"
 guix environment --bootstrap -r "$gcroot" --ad-hoc guile-bootstrap \
      -- guile -c 1
 test `readlink "$gcroot"` = "$expected"
+
+# Make sure '-p' works as expected.
+test $(guix environment -p "$gcroot" -- "$SHELL" -c 'echo $GUIX_ENVIRONMENT') = "$expected"
+paths1="$(guix environment -p "$gcroot" --search-paths)"
+paths2="$(guix environment --bootstrap --ad-hoc guile-bootstrap --search-paths)"
+test "$paths1" = "$paths2"
+
 rm "$gcroot"
 
 # Try '-r' with a relative file name.
-- 
2.33.0

[bug#50960] [PATCH v2 07/11] environment: Skip derivation computation when '--profile' is used.

[Ludovic Courtès]

* guix/scripts/environment.scm (guix-environment*): Bypass calls to
'package-derivation' and to 'manifest->derivation' when PROFILE is
true.
---
 guix/scripts/environment.scm | 23 +++++++++++++----------
 1 file changed, 13 insertions(+), 10 deletions(-)

diff --git a/guix/scripts/environment.scm b/guix/scripts/environment.scm
index 77956fc018..32f376fdd2 100644
--- a/guix/scripts/environment.scm
+++ b/guix/scripts/environment.scm
@@ -729,18 +729,21 @@ (define manifest
             ;; Use the bootstrap Guile when requested.
             (parameterize ((%graft? (assoc-ref opts 'graft?))
                            (%guile-for-build
-                            (package-derivation
-                             store
-                             (if bootstrap?
-                                 %bootstrap-guile
-                                 (default-guile)))))
+                            (and (or container? (not profile))
+                                 (package-derivation
+                                  store
+                                  (if bootstrap?
+                                      %bootstrap-guile
+                                      (default-guile))))))
               (run-with-store store
                 ;; Containers need a Bourne shell at /bin/sh.
                 (mlet* %store-monad ((bash       (environment-bash container?
                                                                    bootstrap?
                                                                    system))
-                                     (prof-drv   (manifest->derivation
-                                                  manifest system bootstrap?))
+                                     (prof-drv   (if profile
+                                                     (return #f)
+                                                     (manifest->derivation
+                                                      manifest system bootstrap?)))
                                      (profile -> (if profile
                                                      (readlink* profile)
                                                      (derivation->output-path prof-drv)))
@@ -750,9 +753,9 @@ (define manifest
                   ;; --search-paths.  Additionally, we might need to build bash for
                   ;; a container.
                   (mbegin %store-monad
-                    (built-derivations (if (derivation? bash)
-                                           (list prof-drv bash)
-                                           (list prof-drv)))
+                    (built-derivations (append
+                                           (if prof-drv (list prof-drv) '())
+                                           (if (derivation? bash) (list bash) '())))
                     (mwhen gc-root
                       (register-gc-root profile gc-root))
 
-- 
2.33.0

[bug#50960] [PATCH v2 08/11] environment: Do not connect to the daemon when '--profile' is used.

[Ludovic Courtès]

This further speeds up the 'guix environment -p PROFILE' case.

* guix/scripts/environment.scm (guix-environment*)[store-needed?]: New
variable.
[with-store/maybe]: New macro.
Use it instead of 'with-store', and remove 'with-build-handler' form.
---
 guix/scripts/environment.scm | 169 +++++++++++++++++++----------------
 1 file changed, 93 insertions(+), 76 deletions(-)

diff --git a/guix/scripts/environment.scm b/guix/scripts/environment.scm
index 32f376fdd2..e23d52df39 100644
--- a/guix/scripts/environment.scm
+++ b/guix/scripts/environment.scm
@@ -691,6 +691,26 @@ (define (guix-environment* opts)
            (mappings   (pick-all opts 'file-system-mapping))
            (white-list (pick-all opts 'inherit-regexp)))
 
+      (define store-needed?
+        ;; Whether connecting to the daemon is needed.
+        (or container? (not profile)))
+
+      (define-syntax-rule (with-store/maybe store exp ...)
+        ;; Evaluate EXP... with STORE bound to a connection, unless
+        ;; STORE-NEEDED? is false, in which case STORE is bound to #f.
+        (let ((proc (lambda (store) exp ...)))
+          (if store-needed?
+              (with-store s
+                (set-build-options-from-command-line s opts)
+                (with-build-handler (build-notifier #:use-substitutes?
+                                                    (assoc-ref opts 'substitutes?)
+                                                    #:verbosity
+                                                    (assoc-ref opts 'verbosity)
+                                                    #:dry-run?
+                                                    (assoc-ref opts 'dry-run?))
+                  (proc s)))
+              (proc #f))))
+
       (when container? (assert-container-features))
 
       (when (and (not container?) link-prof?)
@@ -701,88 +721,85 @@ (define (guix-environment* opts)
         (leave (G_ "--no-cwd cannot be used without --container~%")))
 
 
-      (with-store store
-        (with-build-handler (build-notifier #:use-substitutes?
-                                            (assoc-ref opts 'substitutes?)
-                                            #:verbosity
-                                            (assoc-ref opts 'verbosity)
-                                            #:dry-run?
-                                            (assoc-ref opts 'dry-run?))
-          (with-status-verbosity (assoc-ref opts 'verbosity)
-            (define manifest-from-opts
-              (options/resolve-packages store opts))
+      (with-store/maybe store
+        (with-status-verbosity (assoc-ref opts 'verbosity)
+          (define manifest-from-opts
+            (options/resolve-packages store opts))
 
-            (define manifest
-              (if profile
-                  (profile-manifest profile)
-                  manifest-from-opts))
+          (define manifest
+            (if profile
+                (profile-manifest profile)
+                manifest-from-opts))
 
-            (when (and profile
-                       (> (length (manifest-entries manifest-from-opts)) 0))
-              (leave (G_ "'--profile' cannot be used with package options~%")))
+          (when (and profile
+                     (> (length (manifest-entries manifest-from-opts)) 0))
+            (leave (G_ "'--profile' cannot be used with package options~%")))
 
-            (when (null? (manifest-entries manifest))
-              (warning (G_ "no packages specified; creating an empty environment~%")))
+          (when (null? (manifest-entries manifest))
+            (warning (G_ "no packages specified; creating an empty environment~%")))
 
-            (set-build-options-from-command-line store opts)
+          ;; Use the bootstrap Guile when requested.
+          (parameterize ((%graft? (assoc-ref opts 'graft?))
+                         (%guile-for-build
+                          (and store-needed?
+                               (package-derivation
+                                store
+                                (if bootstrap?
+                                    %bootstrap-guile
+                                    (default-guile))))))
+            (run-with-store store
+              ;; Containers need a Bourne shell at /bin/sh.
+              (mlet* %store-monad ((bash       (environment-bash container?
+                                                                 bootstrap?
+                                                                 system))
+                                   (prof-drv   (if profile
+                                                   (return #f)
+                                                   (manifest->derivation
+                                                    manifest system bootstrap?)))
+                                   (profile -> (if profile
+                                                   (readlink* profile)
+                                                   (derivation->output-path prof-drv)))
+                                   (gc-root -> (assoc-ref opts 'gc-root)))
 
-            ;; Use the bootstrap Guile when requested.
-            (parameterize ((%graft? (assoc-ref opts 'graft?))
-                           (%guile-for-build
-                            (and (or container? (not profile))
-                                 (package-derivation
-                                  store
-                                  (if bootstrap?
-                                      %bootstrap-guile
-                                      (default-guile))))))
-              (run-with-store store
-                ;; Containers need a Bourne shell at /bin/sh.
-                (mlet* %store-monad ((bash       (environment-bash container?
-                                                                   bootstrap?
-                                                                   system))
-                                     (prof-drv   (if profile
-                                                     (return #f)
-                                                     (manifest->derivation
-                                                      manifest system bootstrap?)))
-                                     (profile -> (if profile
-                                                     (readlink* profile)
-                                                     (derivation->output-path prof-drv)))
-                                     (gc-root -> (assoc-ref opts 'gc-root)))
-
-                  ;; First build the inputs.  This is necessary even for
-                  ;; --search-paths.  Additionally, we might need to build bash for
-                  ;; a container.
-                  (mbegin %store-monad
+                ;; First build the inputs.  This is necessary even for
+                ;; --search-paths.  Additionally, we might need to build bash for
+                ;; a container.
+                (mbegin %store-monad
+                  (mwhen store-needed?
                     (built-derivations (append
                                            (if prof-drv (list prof-drv) '())
-                                           (if (derivation? bash) (list bash) '())))
-                    (mwhen gc-root
-                      (register-gc-root profile gc-root))
+                                           (if (derivation? bash) (list bash) '()))))
+                  (mwhen gc-root
+                    (register-gc-root profile gc-root))
 
-                    (cond
-                     ((assoc-ref opts 'search-paths)
-                      (show-search-paths profile manifest #:pure? pure?)
-                      (return #t))
-                     (container?
-                      (let ((bash-binary
-                             (if bootstrap?
-                                 (derivation->output-path bash)
-                                 (string-append (derivation->output-path bash)
-                                                "/bin/sh"))))
-                        (launch-environment/container #:command command
-                                                      #:bash bash-binary
-                                                      #:user user
-                                                      #:user-mappings mappings
-                                                      #:profile profile
-                                                      #:manifest manifest
-                                                      #:white-list white-list
-                                                      #:link-profile? link-prof?
-                                                      #:network? network?
-                                                      #:map-cwd? (not no-cwd?))))
+                  (cond
+                   ((assoc-ref opts 'search-paths)
+                    (show-search-paths profile manifest #:pure? pure?)
+                    (return #t))
+                   (container?
+                    (let ((bash-binary
+                           (if bootstrap?
+                               (derivation->output-path bash)
+                               (string-append (derivation->output-path bash)
+                                              "/bin/sh"))))
+                      (launch-environment/container #:command command
+                                                    #:bash bash-binary
+                                                    #:user user
+                                                    #:user-mappings mappings
+                                                    #:profile profile
+                                                    #:manifest manifest
+                                                    #:white-list white-list
+                                                    #:link-profile? link-prof?
+                                                    #:network? network?
+                                                    #:map-cwd? (not no-cwd?))))
 
-                     (else
-                      (return
-                       (exit/status
-                        (launch-environment/fork command profile manifest
-                                                 #:white-list white-list
-                                                 #:pure? pure?)))))))))))))))
+                   (else
+                    (return
+                     (exit/status
+                      (launch-environment/fork command profile manifest
+                                               #:white-list white-list
+                                               #:pure? pure?))))))))))))))
+
+;;; Local Variables:
+;;; (put 'with-store/maybe 'scheme-indent-function 1)
+;;; End:
-- 
2.33.0

[bug#50960] [PATCH v2 09/11] environment: Autoload some modules.

[Ludovic Courtès]

This further speeds up the 'guix environment -p PROFILE' case.

* guix/scripts/environment.scm: Autoload a bunch of modules.
---
 guix/scripts/environment.scm | 19 +++++++++++--------
 1 file changed, 11 insertions(+), 8 deletions(-)

diff --git a/guix/scripts/environment.scm b/guix/scripts/environment.scm
index e23d52df39..05a43659da 100644
--- a/guix/scripts/environment.scm
+++ b/guix/scripts/environment.scm
@@ -34,15 +34,18 @@ (define-module (guix scripts environment)
   #:use-module (guix scripts)
   #:use-module (guix scripts build)
   #:use-module (guix transformations)
-  #:use-module (gnu build linux-container)
-  #:use-module (gnu build accounts)
-  #:use-module ((guix build syscalls) #:select (set-network-interface-up))
-  #:use-module (gnu system linux-container)
+  #:autoload   (gnu build linux-container) (call-with-container %namespaces
+                                            user-namespace-supported?
+                                            unprivileged-user-namespace-supported?
+                                            setgroups-supported?)
+  #:autoload   (gnu build accounts) (password-entry group-entry
+                                     password-entry-name password-entry-directory
+                                     write-passwd write-group)
+  #:autoload   (guix build syscalls) (set-network-interface-up)
   #:use-module (gnu system file-systems)
-  #:use-module (gnu packages)
-  #:use-module (gnu packages bash)
-  #:use-module ((gnu packages bootstrap)
-                #:select (bootstrap-executable %bootstrap-guile))
+  #:autoload   (gnu packages) (specification->package+output)
+  #:autoload   (gnu packages bash) (bash)
+  #:autoload   (gnu packages bootstrap) (bootstrap-executable %bootstrap-guile)
   #:use-module (ice-9 match)
   #:use-module (srfi srfi-1)
   #:use-module (srfi srfi-11)
-- 
2.33.0

[bug#50960] [PATCH v2 10/11] cache: Gracefully handle non-existent cache.

[Ludovic Courtès]

* guix/cache.scm (maybe-remove-expired-cache-entries): Ignore ENOENT
when writing EXPIRY-FILE.
---
 guix/cache.scm | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)

diff --git a/guix/cache.scm b/guix/cache.scm
index 0401a9d428..51009809bd 100644
--- a/guix/cache.scm
+++ b/guix/cache.scm
@@ -101,7 +101,13 @@ (define last-expiry-date
                                   #:now now
                                   #:entry-expiration entry-expiration
                                   #:delete-entry delete-entry)
-    (call-with-output-file expiry-file
-      (cute write (time-second now) <>))))
+    (catch 'system-error
+      (lambda ()
+        (call-with-output-file expiry-file
+          (cute write (time-second now) <>)))
+      (lambda args
+        ;; ENOENT means CACHE does not exist.
+        (unless (= ENOENT (system-error-errno args))
+          (apply throw args))))))
 
 ;;; cache.scm ends here
-- 
2.33.0

[bug#50960] [PATCH v2 11/11] shell: Maintain a profile cache.

[Ludovic Courtès]

shell: Maintain a profile cache.

With this change, running "guix shell" (no arguments) is equivalent to:

  guix environment -r ~/.cache/guix/profiles/some-root -l guix.scm

This is the cache miss.  On cache hit, it's equivalent to:

  guix environment -p ~/.cache/guix/profiles/some-root

... which can run in 0.1s.

* guix/scripts/shell.scm (options-with-caching): New procedure.
(parse-args): Use it.
(%profile-cache-directory): New variable.
(profile-cache-key, profile-cached-gc-root): New procedures.
(show-help, %options): Add '--rebuild-cache'.
(guix-shell)[cache-entries, entry-expiration]: New procedures.
Add call to 'maybe-remove-expired-cache-entries'.
* doc/guix.texi (Invoking guix shell): Document '--rebuild-cache'.
---
 doc/guix.texi          |  11 ++++
 guix/scripts/shell.scm | 127 ++++++++++++++++++++++++++++++++++++++---
 2 files changed, 130 insertions(+), 8 deletions(-)

diff --git a/doc/guix.texi b/doc/guix.texi
index b95025a39f..f3be6b5085 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -5768,6 +5768,17 @@ This is similar to the same-named option in @command{guix package}
 (@pxref{profile-manifest, @option{--manifest}}) and uses the same
 manifest files.
 
+@item --rebuild-cache
+When using @option{--manifest}, @option{--file}, or when invoked without
+arguments, @command{guix shell} caches the environment so that
+subsequent uses are instantaneous.  The cache is invalidated anytime the
+file is modified.
+
+The @option{--rebuild-cache} forces the cached environment to be
+refreshed even if the file has not changed.  This is useful if the
+@command{guix.scm} or @command{manifest.scm} has external dependencies,
+or if its behavior depends, say, on environment variables.
+
 @item --pure
 Unset existing environment variables when building the new environment, except
 those specified with @option{--preserve} (see below).  This has the effect of
diff --git a/guix/scripts/shell.scm b/guix/scripts/shell.scm
index 45fd536145..4062e8155d 100644
--- a/guix/scripts/shell.scm
+++ b/guix/scripts/shell.scm
@@ -31,7 +31,15 @@ (define-module (guix scripts shell)
   #:use-module (srfi srfi-71)
   #:use-module (ice-9 match)
   #:autoload   (ice-9 rdelim) (read-line)
-  #:autoload   (guix utils) (config-directory)
+  #:autoload   (guix base32) (bytevector->base32-string)
+  #:autoload   (rnrs bytevectors) (string->utf8)
+  #:autoload   (guix utils) (config-directory cache-directory)
+  #:autoload   (guix describe) (current-channels)
+  #:autoload   (guix channels) (channel-commit)
+  #:autoload   (gcrypt hash) (sha256)
+  #:use-module ((guix build utils) #:select (mkdir-p))
+  #:use-module (guix cache)
+  #:use-module ((ice-9 ftw) #:select (scandir))
   #:export (guix-shell))
 
 (define (show-help)
@@ -48,6 +56,8 @@ (define (show-help)
                          FILE evaluates to"))
   (display (G_ "
   -q                     inhibit loading of 'guix.scm' and 'manifest.scm'"))
+  (display (G_ "
+      --rebuild-cache    rebuild cached environment, if any"))
 
   (show-environment-options-help)
   (newline)
@@ -109,7 +119,10 @@ (define %options
                                     result)))
               (option '(#\q) #f #f
                       (lambda (opt name arg result)
-                        (alist-cons 'explicit-loading? #t result))))
+                        (alist-cons 'explicit-loading? #t result)))
+              (option '("rebuild-cache") #f #f
+                      (lambda (opt name arg result)
+                        (alist-cons 'rebuild-cache? #t result))))
         (filter-map (lambda (opt)
                       (and (not (any (lambda (name)
                                        (member name to-remove))
@@ -132,11 +145,12 @@ (define (handle-argument arg result)
   (let ((args command (break (cut string=? "--" <>) args)))
     (let ((opts (parse-command-line args %options (list %default-options)
                                     #:argument-handler handle-argument)))
-      (auto-detect-manifest
-       (match command
-         (() opts)
-         (("--") opts)
-         (("--" command ...) (alist-cons 'exec command opts)))))))
+      (options-with-caching
+       (auto-detect-manifest
+        (match command
+          (() opts)
+          (("--") opts)
+          (("--" command ...) (alist-cons 'exec command opts))))))))
 
 (define (find-file-in-parent-directories candidates)
   "Find one of CANDIDATES in the current directory or one of its ancestors."
@@ -187,6 +201,53 @@ (define (authorized-shell-directory? directory)
                                  line))))))))))
     (const #f)))
 
+(define (options-with-caching opts)
+  "If OPTS contains exactly one 'load' or one 'manifest' key, automatically
+add a 'profile' key (when a profile for that file is already in cache) or a
+'gc-root' key (to add the profile to cache)."
+  (define (single-file-for-caching opts)
+    (let loop ((opts opts)
+               (file #f))
+      (match opts
+        (() file)
+        ((('package . _) . _) #f)
+        ((('load . ('package candidate)) . rest)
+         (and (not file) (loop rest candidate)))
+        ((('manifest . candidate) . rest)
+         (and (not file) (loop rest candidate)))
+        ((('expression . _) . _) #f)
+        ((_ . rest) (loop rest file)))))
+
+  ;; Check whether there's a single 'load' or 'manifest' option.  When that is
+  ;; the case, arrange to automatically cache the resulting profile.
+  (match (single-file-for-caching opts)
+    (#f opts)
+    (file
+     (let* ((root (profile-cached-gc-root file))
+            (stat (and root (false-if-exception (lstat root)))))
+       (if (and (not (assoc-ref opts 'rebuild-cache?))
+                stat
+                (<= (stat:mtime ((@ (guile) stat) file))
+                    (stat:mtime stat)))
+           (let ((now (current-time)))
+             ;; Update the atime on ROOT to reflect usage.
+             (utime root
+                    now (stat:mtime stat) 0 (stat:mtimensec stat)
+                    AT_SYMLINK_NOFOLLOW)
+             (alist-cons 'profile root
+                         (remove (match-lambda
+                                   (('load . _) #t)
+                                   (('manifest . _) #t)
+                                   (_ #f))
+                                 opts)))          ;load right away
+           (if (and root (not (assq-ref opts 'gc-root)))
+               (begin
+                 (if stat
+                     (delete-file root)
+                     (mkdir-p (dirname root)))
+                 (alist-cons 'gc-root root opts))
+               opts))))))
+
 (define (auto-detect-manifest opts)
   "If OPTS do not specify packages or a manifest, load a \"guix.scm\" or
 \"manifest.scm\" file from the current directory or one of its ancestors.
@@ -236,9 +297,59 @@ (define disallow-implicit-load?
                                      (authorized-directory-file)))
                opts))))))
 
+\f
+;;;
+;;; Profile cache.
+;;;
+
+(define %profile-cache-directory
+  ;; Directory where profiles created by 'guix shell' alone (without extra
+  ;; options) are cached.
+  (make-parameter (string-append (cache-directory #:ensure? #f)
+                                 "/profiles")))
+
+(define (profile-cache-key file)
+  "Return the cache key for the profile corresponding to FILE, a 'guix.scm' or
+'manifest.scm' file, or #f if we lack channel information."
+  (match (current-channels)
+    (() #f)
+    (((= channel-commit commits) ...)
+     (let ((stat (stat file)))
+       (bytevector->base32-string
+        ;; Since FILE is not canonicalized, only include the device/inode
+        ;; numbers.  XXX: In some rare cases involving Btrfs and NFS, this can
+        ;; be insufficient: <https://lwn.net/Articles/866582/>.
+        (sha256 (string->utf8
+                 (string-append (string-join commits) ":"
+                                (number->string (stat:dev stat)) ":"
+                                (number->string (stat:ino stat))))))))))
+
+(define (profile-cached-gc-root file)
+  "Return the cached GC root for FILE, a 'guix.scm' or 'manifest.scm' file, or
+#f if we lack information to cache it."
+  (match (profile-cache-key file)
+    (#f  #f)
+    (key (string-append (%profile-cache-directory) "/" key))))
+
 \f
 (define-command (guix-shell . args)
   (category development)
   (synopsis "spawn one-off software environments")
 
-  (guix-environment* (parse-args args)))
+  (define (cache-entries directory)
+    (filter-map (match-lambda
+                  ((or "." "..") #f)
+                  (file (string-append directory "/" file)))
+                (or (scandir directory) '())))
+
+  (define* (entry-expiration file)
+    ;; Return the time at which FILE, a cached profile, is considered expired.
+    (match (false-if-exception (lstat file))
+      (#f 0)                       ;FILE may have been deleted in the meantime
+      (st (+ (stat:atime st) (* 60 60 24 7)))))
+
+  (let ((result (guix-environment* (parse-args args))))
+    (maybe-remove-expired-cache-entries (%profile-cache-directory)
+                                        cache-entries
+                                        #:entry-expiration entry-expiration)
+    result))
-- 
2.33.0

[bug#50960] [PATCH v2 00/11] 'guix shell' strikes again

[pelzflorian (Florian Pelz)]

On Mon, Oct 11, 2021 at 11:37:58PM +0200, Ludovic Courtès wrote:
> Hello Guix!
> 
> Here’s a v2 with the following changes:

Nice!

>   • Auto-loading happens if and only if the containing directory
>     is listed in ~/.config/guix/shell-authorized-directories, as
>     suggested by Florian.

Liliana Marie Prikler suggested it
<https://lists.gnu.org/r/guix-patches/2021-10/msg00380.html>.

>   […]
> How does that sound?  :-)

Much better, thank you!

Regards,
Florian

[bug#50960] [PATCH v2 00/11] 'guix shell' strikes again

[pelzflorian (Florian Pelz)]

On Tue, Oct 12, 2021 at 10:53:55AM +0200, pelzflorian (Florian Pelz) wrote:
> On Mon, Oct 11, 2021 at 11:37:58PM +0200, Ludovic Courtès wrote:
> >   • Auto-loading happens if and only if the containing directory
> >     is listed in ~/.config/guix/shell-authorized-directories, as
> >     suggested by Florian.
> 
> Liliana Marie Prikler suggested it
> <https://lists.gnu.org/r/guix-patches/2021-10/msg00380.html>.

Or was it Nicoló Balzarotti?

https://lists.gnu.org/r/guix-patches/2021-10/msg00196.html

Enough confusion.

Regards,
Florian

[bug#50960] [PATCH v2 00/11] 'guix shell' strikes again

[Ludovic Courtès]

"pelzflorian (Florian Pelz)" <pelzflorian@pelzflorian.de> skribis:

> On Tue, Oct 12, 2021 at 10:53:55AM +0200, pelzflorian (Florian Pelz) wrote:
>> On Mon, Oct 11, 2021 at 11:37:58PM +0200, Ludovic Courtès wrote:
>> >   • Auto-loading happens if and only if the containing directory
>> >     is listed in ~/.config/guix/shell-authorized-directories, as
>> >     suggested by Florian.
>> 
>> Liliana Marie Prikler suggested it
>> <https://lists.gnu.org/r/guix-patches/2021-10/msg00380.html>.
>
> Or was it Nicoló Balzarotti?
>
> https://lists.gnu.org/r/guix-patches/2021-10/msg00196.html

My bad, sorry for misattributing it.

Anyhow: big thanks to all of you for providing feedback!  :-)

Ludo’.

[bug#50960] [PATCH v3 00/10] Adding 'guix shell': last call!

[Ludovic Courtès]

Hi!

This is v3 of the ‘guix shell’ patch set.  Changes since v2:

  • Merged as a single commit the one that adds support for implicit
    ‘guix.scm’/‘manifest.scm’ and then one that adds authorization
    checks;

  • Fixed ‘package-development-inputs’ bug/typo reported by zimoun;

  • Fixed doc typos reported by Florian.

If there are no objections, I’ll go ahead with this patch series
in a few days.  I’ll also add a news entry, and I guess a blog post
introducing it would be welcome.

WDYT?

Ludo’.

Ludovic Courtès (10):
  packages: Add 'package-development-inputs'.
  profiles: Add 'package->development-manifest'.
  Add 'guix shell'.
  shell: By default load the local 'guix.scm' or 'manifest.scm' file.
  environment: Add tests for '--profile'.
  environment: Skip derivation computation when '--profile' is used.
  environment: Do not connect to the daemon when '--profile' is used.
  environment: Autoload some modules.
  cache: Gracefully handle non-existent cache.
  shell: Maintain a profile cache.

 Makefile.am                         |   2 +
 doc/contributing.texi               |   8 +-
 doc/guix.texi                       | 443 ++++++++++++++++++++++++++--
 guix/cache.scm                      |  10 +-
 guix/packages.scm                   |  10 +
 guix/profiles.scm                   |  19 ++
 guix/scripts/environment.scm        | 260 ++++++++--------
 guix/scripts/shell.scm              | 355 ++++++++++++++++++++++
 po/guix/POTFILES.in                 |   1 +
 tests/guix-environment-container.sh |   8 +
 tests/guix-environment.sh           |   7 +
 tests/guix-shell.sh                 | 100 +++++++
 tests/packages.scm                  |  14 +
 tests/profiles.scm                  |   7 +
 14 files changed, 1098 insertions(+), 146 deletions(-)
 create mode 100644 guix/scripts/shell.scm
 create mode 100644 tests/guix-shell.sh


base-commit: 9cda21cf20a5c9bdf97e3a6d6c8f901fc3e4307d
-- 
2.33.0

[bug#50960] [PATCH v3 01/10] packages: Add 'package-development-inputs'.

[Ludovic Courtès]

* guix/packages.scm (package-development-inputs): New procedure.
* guix/scripts/environment.scm (package-environment-inputs): Use it.
* tests/packages.scm ("package-development-inputs")
("package-development-inputs, cross-compilation"): New tests.
* doc/guix.texi (package Reference): Document it.
---
 doc/guix.texi                | 41 ++++++++++++++++++++++++++++++++++++
 guix/packages.scm            | 10 +++++++++
 guix/scripts/environment.scm |  2 +-
 tests/packages.scm           | 14 ++++++++++++
 4 files changed, 66 insertions(+), 1 deletion(-)

diff --git a/doc/guix.texi b/doc/guix.texi
index a49abc0554..deb4eac4ac 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -6848,6 +6848,47 @@ cross-compiling:
 It is an error to refer to @code{this-package} outside a package definition.
 @end deffn
 
+@cindex development inputs, of a package
+@cindex implicit inputs, of a package
+Sometimes you will want to obtain the list of inputs needed to
+@emph{develop} a package---all the inputs that are visible when the
+package is compiled.  This is what the @code{package-development-inputs}
+procedure returns.
+
+@deffn {Scheme Procedure} package-development-inputs @var{package} @
+   [@var{system}] [#:target #f]
+Return the list of inputs required by @var{package} for development
+purposes on @var{system}.  When @var{target} is true, return the inputs
+needed to cross-compile @var{package} from @var{system} to
+@var{triplet}, where @var{triplet} is a triplet such as
+@code{"aarch64-linux-gnu"}.
+
+Note that the result includes both explicit inputs and implicit
+inputs---inputs automatically added by the build system (@pxref{Build
+Systems}).  Let us take the @code{hello} package to illustrate that:
+
+@lisp
+(use-modules (gnu packages base) (guix packages))
+
+hello
+@result{} #<package hello@@2.10 gnu/packages/base.scm:79 7f585d4f6790>
+
+(package-direct-inputs hello)
+@result{} ()
+
+(package-development-inputs hello)
+@result{} (("source" @dots{}) ("tar" #<package tar@@1.32 @dots{}>) @dots{})
+@end lisp
+
+In this example, @code{package-direct-inputs} returns the empty list,
+because @code{hello} has zero explicit dependencies.  Conversely,
+@code{package-development-inputs} includes inputs implicitly added by
+@code{gnu-build-system} that are required to build @code{hello}: tar,
+gzip, GCC, libc, Bash, and more.  To visualize it, @command{guix graph
+hello} would show you explicit inputs, whereas @command{guix graph -t
+bag hello} would include implicit inputs (@pxref{Invoking guix graph}).
+@end deffn
+
 Because packages are regular Scheme objects that capture a complete
 dependency graph and associated build procedures, it is often useful to
 write procedures that take a package and return a modified version
diff --git a/guix/packages.scm b/guix/packages.scm
index 8c3a0b0b7b..43e0130793 100644
--- a/guix/packages.scm
+++ b/guix/packages.scm
@@ -153,6 +153,7 @@ (define-module (guix packages)
             bag-transitive-host-inputs
             bag-transitive-build-inputs
             bag-transitive-target-inputs
+            package-development-inputs
             package-closure
 
             default-guile
@@ -1070,6 +1071,15 @@ (define (bag-transitive-target-inputs bag)
                  (%current-system (bag-system bag)))
     (transitive-inputs (bag-target-inputs bag))))
 
+(define* (package-development-inputs package
+                                     #:optional (system (%current-system))
+                                     #:key target)
+  "Return the list of inputs required by PACKAGE for development purposes on
+SYSTEM.  When TARGET is true, return the inputs needed to cross-compile
+PACKAGE from SYSTEM to TRIPLET, where TRIPLET is a triplet such as
+\"aarch64-linux-gnu\"."
+  (bag-transitive-inputs (package->bag package system target)))
+
 (define* (package-closure packages #:key (system (%current-system)))
   "Return the closure of PACKAGES on SYSTEM--i.e., PACKAGES and the list of
 packages they depend on, recursively."
diff --git a/guix/scripts/environment.scm b/guix/scripts/environment.scm
index 6958bd6238..418f11c37e 100644
--- a/guix/scripts/environment.scm
+++ b/guix/scripts/environment.scm
@@ -82,7 +82,7 @@ (define (package-environment-inputs package)
 packages for PACKAGE."
   ;; Remove non-package inputs such as origin records.
   (filter-map input->manifest-entry
-              (bag-transitive-inputs (package->bag package))))
+              (package-development-inputs package)))
 
 (define (show-help)
   (display (G_ "Usage: guix environment [OPTION]... PACKAGE... [-- COMMAND...]
diff --git a/tests/packages.scm b/tests/packages.scm
index 3756877270..266b5aeb7a 100644
--- a/tests/packages.scm
+++ b/tests/packages.scm
@@ -353,6 +353,20 @@ (define read-at
           (package-transitive-supported-systems d)
           (package-transitive-supported-systems e))))
 
+(test-assert "package-development-inputs"
+  ;; Note: Due to propagated inputs, 'package-development-inputs' returns a
+  ;; couple more inputs, such as 'linux-libre-headers'.
+  (lset<= equal?
+          `(("source" ,(package-source hello)) ,@(standard-packages))
+          (package-development-inputs hello)))
+
+(test-assert "package-development-inputs, cross-compilation"
+  (lset<= equal?
+          `(("source" ,(package-source hello))
+            ,@(standard-cross-packages "mips64el-linux-gnu" 'host)
+            ,@(standard-cross-packages "mips64el-linux-gnu" 'target))
+          (package-development-inputs hello #:target "mips64el-linux-gnu")))
+
 (test-assert "package-closure"
   (let-syntax ((dummy-package/no-implicit
                 (syntax-rules ()
-- 
2.33.0

[bug#50960] [PATCH v3 02/10] profiles: Add 'package->development-manifest'.

[Ludovic Courtès]

From: Ludovic Courtès <ludovic.courtes@inria.fr>

* guix/profiles.scm (package->development-manifest): New procedure.
* guix/scripts/environment.scm (input->manifest-entry)
(package-environment-inputs): Remove.
* guix/scripts/environment.scm (options/resolve-packages): Use
'package->development-manifest' instead of 'package-environment-inputs'.
* tests/profiles.scm ("package->development-manifest"): New test.
---
 doc/guix.texi                | 11 +++++++++++
 guix/profiles.scm            | 19 +++++++++++++++++++
 guix/scripts/environment.scm | 27 +++++----------------------
 tests/profiles.scm           |  7 +++++++
 4 files changed, 42 insertions(+), 22 deletions(-)

diff --git a/doc/guix.texi b/doc/guix.texi
index deb4eac4ac..c1045b331b 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -3341,6 +3341,17 @@ objects, like this:
  '("emacs" "guile@@2.2" "guile@@2.2:debug"))
 @end lisp
 
+@findex package->development-manifest
+You might also want to create a manifest for all the dependencies of a
+package, rather than the package itself:
+
+@lisp
+(package->development-manifest (specification->package "emacs"))
+@end lisp
+
+The example above gives you all the software required to develop Emacs,
+similar to what @command{guix environment emacs} provides.
+
 @xref{export-manifest, @option{--export-manifest}}, to learn how to
 obtain a manifest file from an existing profile.
 
diff --git a/guix/profiles.scm b/guix/profiles.scm
index 2486f91d09..9f30349c69 100644
--- a/guix/profiles.scm
+++ b/guix/profiles.scm
@@ -124,6 +124,7 @@ (define-module (guix profiles)
 
             profile-manifest
             package->manifest-entry
+            package->development-manifest
             packages->manifest
             ca-certificate-bundle
             %default-profile-hooks
@@ -400,6 +401,24 @@ (define* (package->manifest-entry package #:optional (output "out")
                      (properties properties))))
     entry))
 
+(define* (package->development-manifest package
+                                        #:optional
+                                        (system (%current-system))
+                                        #:key target)
+  "Return a manifest for the \"development inputs\" of PACKAGE for SYSTEM,
+optionally when cross-compiling to TARGET.  Development inputs include both
+explicit and implicit inputs of PACKAGE."
+  (manifest
+   (filter-map (match-lambda
+                 ((label (? package? package))
+                  (package->manifest-entry package))
+                 ((label (? package? package) output)
+                  (package->manifest-entry package output))
+                 ;; TODO: Support <inferior-package>.
+                 (_
+                  #f))
+               (package-development-inputs package system #:target target))))
+
 (define (packages->manifest packages)
   "Return a list of manifest entries, one for each item listed in PACKAGES.
 Elements of PACKAGES can be either package objects or package/string tuples
diff --git a/guix/scripts/environment.scm b/guix/scripts/environment.scm
index 418f11c37e..54f48a7482 100644
--- a/guix/scripts/environment.scm
+++ b/guix/scripts/environment.scm
@@ -66,24 +66,6 @@ (define* (show-search-paths profile manifest #:key pure?)
                (newline)))
             (profile-search-paths profile manifest)))
 
-(define (input->manifest-entry input)
-  "Return a manifest entry for INPUT, or #f if INPUT does not correspond to a
-package."
-  (match input
-    ((_ (? package? package))
-     (package->manifest-entry package))
-    ((_ (? package? package) output)
-     (package->manifest-entry package output))
-    (_
-     #f)))
-
-(define (package-environment-inputs package)
-  "Return a list of manifest entries corresponding to the transitive input
-packages for PACKAGE."
-  ;; Remove non-package inputs such as origin records.
-  (filter-map input->manifest-entry
-              (package-development-inputs package)))
-
 (define (show-help)
   (display (G_ "Usage: guix environment [OPTION]... PACKAGE... [-- COMMAND...]
 Build an environment that includes the dependencies of PACKAGE and execute
@@ -297,11 +279,11 @@ (define (packages->outputs packages mode)
       ((? package? package)
        (if (eq? mode 'ad-hoc-package)
            (list (package->manifest-entry* package))
-           (package-environment-inputs package)))
+           (manifest-entries (package->development-manifest package))))
       (((? package? package) (? string? output))
        (if (eq? mode 'ad-hoc-package)
            (list (package->manifest-entry* package output))
-           (package-environment-inputs package)))
+           (manifest-entries (package->development-manifest package))))
       ((lst ...)
        (append-map (cut packages->outputs <> mode) lst))))
 
@@ -313,8 +295,9 @@ (define (packages->outputs packages mode)
                                  (specification->package+output spec)))
                      (list (package->manifest-entry* package output))))
                   (('package 'package (? string? spec))
-                   (package-environment-inputs
-                    (transform (specification->package+output spec))))
+                   (manifest-entries
+                    (package->development-manifest
+                     (transform (specification->package+output spec)))))
                   (('expression mode str)
                    ;; Add all the outputs of the package STR evaluates to.
                    (packages->outputs (read/eval str) mode))
diff --git a/tests/profiles.scm b/tests/profiles.scm
index 06a0387221..cac5b73347 100644
--- a/tests/profiles.scm
+++ b/tests/profiles.scm
@@ -265,6 +265,13 @@ (define transform1
            (manifest-transaction-removal-candidate? guile-2.0.9 t)
            (null? install) (null? downgrade) (null? upgrade)))))
 
+(test-assert "package->development-manifest"
+  (let ((manifest (package->development-manifest packages:hello)))
+    (every (lambda (name)
+             (manifest-installed? manifest
+                                  (manifest-pattern (name name))))
+           '("gcc" "binutils" "glibc" "coreutils" "grep" "sed"))))
+
 (test-assertm "profile-derivation"
   (mlet* %store-monad
       ((entry ->   (package->manifest-entry %bootstrap-guile))
-- 
2.33.0

[bug#50960] [PATCH v3 03/10] Add 'guix shell'.

[Ludovic Courtès]

From: Ludovic Courtès <ludovic.courtes@inria.fr>

* guix/scripts/shell.scm, tests/guix-shell.sh: New files.
* Makefile.am (MODULES): Add 'shell.scm'.
(SH_TESTS): Add 'tests/guix-shell.sh'.
* guix/scripts/environment.scm (show-environment-options-help): New
procedure.
(show-help): Use it.
(guix-environment*): New procedure.
(guix-environment): Use it.
* po/guix/POTFILES.in: Add it.
* doc/guix.texi (Features): Refer to "guix shell"
(Invoking guix package): Likewise.
(Development): Likewise.
(Invoking guix shell): New node.
(Invoking guix environment): Add deprecation warning.
(Debugging Build Failures): Use 'guix shell' in examples.
(Invoking guix container): Refer to 'guix shell'.
(Invoking guix processes, Virtualization Services): Adjust examples to
use 'guix shell'.
* doc/contributing.texi (Building from Git): Refer to 'guix shell'.
---
 Makefile.am                  |   2 +
 doc/contributing.texi        |   8 +-
 doc/guix.texi                | 366 ++++++++++++++++++++++++++++++++---
 guix/scripts/environment.scm |  52 +++--
 guix/scripts/shell.scm       | 135 +++++++++++++
 po/guix/POTFILES.in          |   1 +
 tests/guix-shell.sh          |  54 ++++++
 7 files changed, 575 insertions(+), 43 deletions(-)
 create mode 100644 guix/scripts/shell.scm
 create mode 100644 tests/guix-shell.sh

diff --git a/Makefile.am b/Makefile.am
index 41ec19eb89..239387c2f4 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -316,6 +316,7 @@ MODULES =					\
   guix/scripts/import/stackage.scm		\
   guix/scripts/import/texlive.scm  		\
   guix/scripts/environment.scm			\
+  guix/scripts/shell.scm			\
   guix/scripts/publish.scm			\
   guix/scripts/edit.scm				\
   guix/scripts/size.scm				\
@@ -552,6 +553,7 @@ SH_TESTS =					\
   tests/guix-authenticate.sh			\
   tests/guix-environment.sh			\
   tests/guix-environment-container.sh		\
+  tests/guix-shell.sh				\
   tests/guix-graph.sh				\
   tests/guix-describe.sh			\
   tests/guix-repl.sh     			\
diff --git a/doc/contributing.texi b/doc/contributing.texi
index 76ab913b0d..db0f836157 100644
--- a/doc/contributing.texi
+++ b/doc/contributing.texi
@@ -73,10 +73,10 @@ all the dependencies and appropriate environment variables are set up to
 hack on Guix:
 
 @example
-guix environment guix --pure
+guix shell -D guix --pure
 @end example
 
-@xref{Invoking guix environment}, for more information on that command.
+@xref{Invoking guix shell}, for more information on that command.
 
 If you are unable to use Guix when building Guix from a checkout, the
 following are the required packages in addition to those mentioned in the
@@ -92,10 +92,10 @@ installation instructions (@pxref{Requirements}).
 @end itemize
 
 On Guix, extra dependencies can be added by instead running @command{guix
-environment} with @option{--ad-hoc}:
+shell}:
 
 @example
-guix environment guix --pure --ad-hoc help2man git strace
+guix shell -D guix help2man git strace --pure
 @end example
 
 Run @command{./bootstrap} to generate the build system infrastructure
diff --git a/doc/guix.texi b/doc/guix.texi
index c1045b331b..63612728ed 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -119,6 +119,7 @@ Documentation License''.
 
 @dircategory Software development
 @direntry
+* guix shell: (guix)Invoking guix shell.      Creating software environments.
 * guix environment: (guix)Invoking guix environment.  Building development environments with Guix.
 * guix build: (guix)Invoking guix build.      Building packages.
 * guix pack: (guix)Invoking guix pack.        Creating binary bundles.
@@ -262,6 +263,7 @@ Channels
 
 Development
 
+* Invoking guix shell::         Spawning one-off software environments.
 * Invoking guix environment::   Setting up development environments.
 * Invoking guix pack::          Creating software bundles.
 * The GCC toolchain::           Working with languages supported by GCC.
@@ -3067,10 +3069,10 @@ substitutes: they can force a local build and @emph{challenge} providers
 (@pxref{Invoking guix challenge}).
 
 Control over the build environment is a feature that is also useful for
-developers.  The @command{guix environment} command allows developers of
+developers.  The @command{guix shell} command allows developers of
 a package to quickly set up the right development environment for their
 package, without having to manually install the dependencies of the
-package into their profile (@pxref{Invoking guix environment}).
+package into their profile (@pxref{Invoking guix shell}).
 
 @cindex replication, of software environments
 @cindex provenance tracking, of software artifacts
@@ -3234,7 +3236,7 @@ As an example, @var{file} might contain a definition like this
 Developers may find it useful to include such a @file{guix.scm} file
 in the root of their project source tree that can be used to test
 development snapshots and create reproducible development environments
-(@pxref{Invoking guix environment}).
+(@pxref{Invoking guix shell}).
 
 The @var{file} may also contain a JSON representation of one or more
 package definitions.  Running @code{guix package -f} on
@@ -5559,31 +5561,352 @@ If you are a software developer, Guix provides tools that you should find
 helpful---independently of the language you're developing in.  This is what
 this chapter is about.
 
-The @command{guix environment} command provides a convenient way to set up
-@dfn{development environments} containing all the dependencies and tools
-necessary to work on the software package of your choice.  The @command{guix
+The @command{guix shell} command provides a convenient way to set up
+one-off software environments, be it for development purposes or to run
+a command without installing it in your profile.  The @command{guix
 pack} command allows you to create @dfn{application bundles} that can be
 easily distributed to users who do not run Guix.
 
 @menu
+* Invoking guix shell::         Spawning one-off software environments.
 * Invoking guix environment::   Setting up development environments.
 * Invoking guix pack::          Creating software bundles.
 * The GCC toolchain::           Working with languages supported by GCC.
 * Invoking guix git authenticate:: Authenticating Git repositories.
 @end menu
 
-@node Invoking guix environment
-@section Invoking @command{guix environment}
+@node Invoking guix shell
+@section Invoking @command{guix shell}
 
 @cindex reproducible build environments
 @cindex development environments
 @cindex @command{guix environment}
 @cindex environment, package build environment
-The purpose of @command{guix environment} is to assist hackers in
-creating reproducible development environments without polluting their
-package profile.  The @command{guix environment} tool takes one or more
-packages, builds all of their inputs, and creates a shell
-environment to use them.
+The purpose of @command{guix shell} is to make it easy to create one-off
+software environments, without changing one's profile.  It is typically
+used to create development environments; it is also a convenient way to
+run applications without ``polluting'' your profile.
+
+@quotation Note
+The @command{guix shell} command was recently introduced to supersede
+@command{guix environment} (@pxref{Invoking guix environment}).  If you
+are familiar with @command{guix environment}, you will notice that it is
+similar but also---we hope!---more convenient.
+@end quotation
+
+The general syntax is:
+
+@example
+guix shell [@var{options}] [@var{package}@dots{}]
+@end example
+
+The following example creates an environment containing Python and NumPy,
+building or downloading any missing package, and runs the
+@command{python3} command in that environment:
+
+@example
+guix shell python python-numpy -- python3
+@end example
+
+Development environments can be created as in the example below, which
+spawns an interactive shell containing all the dependencies and
+environment variables needed to work on Inkscape:
+
+@example
+guix shell --development inkscape
+@end example
+
+Exiting the shell places the user back in the original environment
+before @command{guix shell} was invoked.  The next garbage collection
+(@pxref{Invoking guix gc}) may clean up packages that were installed in
+the environment and that are no longer used outside of it.
+
+By default, the shell session or command runs in an @emph{augmented}
+environment, where the new packages are added to search path environment
+variables such as @code{PATH}.  You can, instead, choose to create an
+@emph{isolated} environment containing nothing but the packages you
+asked for.  Passing the @option{--pure} option clears environment
+variable definitions found in the parent environment@footnote{Users
+sometimes wrongfully augment environment variables such as @env{PATH} in
+their @file{~/.bashrc} file.  As a consequence, when @command{guix
+environment} launches it, Bash may read @file{~/.bashrc}, thereby
+introducing ``impurities'' in these environment variables.  It is an
+error to define such environment variables in @file{.bashrc}; instead,
+they should be defined in @file{.bash_profile}, which is sourced only by
+log-in shells.  @xref{Bash Startup Files,,, bash, The GNU Bash Reference
+Manual}, for details on Bash start-up files.}; passing
+@option{--container} goes one step further by spawning a @dfn{container}
+isolated from the rest of the system:
+
+@example
+guix shell --container emacs gcc-toolchain
+@end example
+
+The command above spawns an interactive shell in a container when
+nothing but @code{emacs}, @code{gcc-toolchain}, and their dependencies
+is available.  The container lacks network access and shares no files
+other than the current working directory with the surrounding
+environment.  This is useful to prevent access to system-wide resources
+such as @file{/usr/bin} on foreign distros.
+
+This @option{--container} option can also prove useful if you wish to
+run a security-sensitive application, such as a web browser, in an
+isolated environment.  For example, the command below launches
+Ungoogled-Chromium in an isolated environment, this time sharing network
+access with the host and preserving its @code{DISPLAY} environment
+variable, but without even sharing the current directory:
+
+@example
+guix shell --container --network --no-cwd ungoogled-chromium \
+  --preserve='^DISPLAY$' -- chromium
+@end example
+
+@vindex GUIX_ENVIRONMENT
+@command{guix shell} defines the @env{GUIX_ENVIRONMENT}
+variable in the shell it spawns; its value is the file name of the
+profile of this environment.  This allows users to, say, define a
+specific prompt for development environments in their @file{.bashrc}
+(@pxref{Bash Startup Files,,, bash, The GNU Bash Reference Manual}):
+
+@example
+if [ -n "$GUIX_ENVIRONMENT" ]
+then
+    export PS1="\u@@\h \w [dev]\$ "
+fi
+@end example
+
+@noindent
+...@: or to browse the profile:
+
+@example
+$ ls "$GUIX_ENVIRONMENT/bin"
+@end example
+
+The available options are summarized below.
+
+@table @code
+@item --development
+@itemx -D
+Cause @command{guix shell} to include in the environment the
+dependencies of the following package rather than the package itself.
+This can be combined with other packages.  For instance, the command
+below starts an interactive shell containing the build-time dependencies
+of GNU@tie{}Guile, plus Autoconf, Automake, and Libtool:
+
+@example
+guix shell -D guile autoconf automake libtool
+@end example
+
+@item --expression=@var{expr}
+@itemx -e @var{expr}
+Create an environment for the package or list of packages that
+@var{expr} evaluates to.
+
+For example, running:
+
+@example
+guix shell -D -e '(@@ (gnu packages maths) petsc-openmpi)'
+@end example
+
+starts a shell with the environment for this specific variant of the
+PETSc package.
+
+Running:
+
+@example
+guix shell -e '(@@ (gnu) %base-packages)'
+@end example
+
+starts a shell with all the base system packages available.
+
+The above commands only use the default output of the given packages.
+To select other outputs, two element tuples can be specified:
+
+@example
+guix shell -e '(list (@@ (gnu packages bash) bash) "include")'
+@end example
+
+@item --file=@var{file}
+@itemx -f @var{file}
+Create an environment containing the package or list of packages that
+the code within @var{file} evaluates to.
+
+As an example, @var{file} might contain a definition like this
+(@pxref{Defining Packages}):
+
+@lisp
+@verbatiminclude environment-gdb.scm
+@end lisp
+
+With the file above, you can enter a development environment for GDB by
+running:
+
+@example
+guix shell -D -f gdb-devel.scm
+@end example
+
+@item --manifest=@var{file}
+@itemx -m @var{file}
+Create an environment for the packages contained in the manifest object
+returned by the Scheme code in @var{file}.  This option can be repeated
+several times, in which case the manifests are concatenated.
+
+This is similar to the same-named option in @command{guix package}
+(@pxref{profile-manifest, @option{--manifest}}) and uses the same
+manifest files.
+
+@item --pure
+Unset existing environment variables when building the new environment, except
+those specified with @option{--preserve} (see below).  This has the effect of
+creating an environment in which search paths only contain package inputs.
+
+@item --preserve=@var{regexp}
+@itemx -E @var{regexp}
+When used alongside @option{--pure}, preserve the environment variables
+matching @var{regexp}---in other words, put them on a ``white list'' of
+environment variables that must be preserved.  This option can be repeated
+several times.
+
+@example
+guix shell --pure --preserve=^SLURM openmpi @dots{} \
+  -- mpirun @dots{}
+@end example
+
+This example runs @command{mpirun} in a context where the only environment
+variables defined are @env{PATH}, environment variables whose name starts
+with @samp{SLURM}, as well as the usual ``precious'' variables (@env{HOME},
+@env{USER}, etc.).
+
+@item --search-paths
+Display the environment variable definitions that make up the
+environment.
+
+@item --system=@var{system}
+@itemx -s @var{system}
+Attempt to build for @var{system}---e.g., @code{i686-linux}.
+
+@item --container
+@itemx -C
+@cindex container
+Run @var{command} within an isolated container.  The current working
+directory outside the container is mapped inside the container.
+Additionally, unless overridden with @option{--user}, a dummy home
+directory is created that matches the current user's home directory, and
+@file{/etc/passwd} is configured accordingly.
+
+The spawned process runs as the current user outside the container.  Inside
+the container, it has the same UID and GID as the current user, unless
+@option{--user} is passed (see below).
+
+@item --network
+@itemx -N
+For containers, share the network namespace with the host system.
+Containers created without this flag only have access to the loopback
+device.
+
+@item --link-profile
+@itemx -P
+For containers, link the environment profile to @file{~/.guix-profile}
+within the container and set @code{GUIX_ENVIRONMENT} to that.
+This is equivalent to making @file{~/.guix-profile} a symlink to the
+actual profile within the container.
+Linking will fail and abort the environment if the directory already
+exists, which will certainly be the case if @command{guix shell}
+was invoked in the user's home directory.
+
+Certain packages are configured to look in @file{~/.guix-profile} for
+configuration files and data;@footnote{For example, the
+@code{fontconfig} package inspects @file{~/.guix-profile/share/fonts}
+for additional fonts.}  @option{--link-profile} allows these programs to
+behave as expected within the environment.
+
+@item --user=@var{user}
+@itemx -u @var{user}
+For containers, use the username @var{user} in place of the current
+user.  The generated @file{/etc/passwd} entry within the container will
+contain the name @var{user}, the home directory will be
+@file{/home/@var{user}}, and no user GECOS data will be copied.  Furthermore,
+the UID and GID inside the container are 1000.  @var{user}
+need not exist on the system.
+
+Additionally, any shared or exposed path (see @option{--share} and
+@option{--expose} respectively) whose target is within the current user's
+home directory will be remapped relative to @file{/home/USER}; this
+includes the automatic mapping of the current working directory.
+
+@example
+# will expose paths as /home/foo/wd, /home/foo/test, and /home/foo/target
+cd $HOME/wd
+guix shell --container --user=foo \
+     --expose=$HOME/test \
+     --expose=/tmp/target=$HOME/target
+@end example
+
+While this will limit the leaking of user identity through home paths
+and each of the user fields, this is only one useful component of a
+broader privacy/anonymity solution---not one in and of itself.
+
+@item --no-cwd
+For containers, the default behavior is to share the current working
+directory with the isolated container and immediately change to that
+directory within the container.  If this is undesirable,
+@option{--no-cwd} will cause the current working directory to @emph{not}
+be automatically shared and will change to the user's home directory
+within the container instead.  See also @option{--user}.
+
+@item --expose=@var{source}[=@var{target}]
+@itemx --share=@var{source}[=@var{target}]
+For containers, @option{--expose} (resp. @option{--share}) exposes the
+file system @var{source} from the host system as the read-only
+(resp. writable) file system @var{target} within the container.  If
+@var{target} is not specified, @var{source} is used as the target mount
+point in the container.
+
+The example below spawns a Guile REPL in a container in which the user's
+home directory is accessible read-only via the @file{/exchange}
+directory:
+
+@example
+guix shell --container --expose=$HOME=/exchange guile -- guile
+@end example
+
+@item --root=@var{file}
+@itemx -r @var{file}
+@cindex persistent environment
+@cindex garbage collector root, for environments
+Make @var{file} a symlink to the profile for this environment, and
+register it as a garbage collector root.
+
+This is useful if you want to protect your environment from garbage
+collection, to make it ``persistent''.
+
+When this option is omitted, the environment is protected from garbage
+collection only for the duration of the @command{guix shell}
+session.  This means that next time you recreate the same environment,
+you could have to rebuild or re-download packages.  @xref{Invoking guix
+gc}, for more on GC roots.
+@end table
+
+@command{guix shell} also supports all of the common build options that
+@command{guix build} supports (@pxref{Common Build Options}) as well as
+package transformation options (@pxref{Package Transformation Options}).
+
+@node Invoking guix environment
+@section Invoking @command{guix environment}
+
+The purpose of @command{guix environment} is to assist in creating
+development environments.
+
+@quotation Deprecation warning
+The @command{guix environment} command is deprecated in favor of
+@command{guix shell}, which performs similar functions but is more
+convenient to use.  @xref{Invoking guix shell}.
+
+Being deprecated, @command{guix environment} is slated for eventual
+removal, but the Guix project is committed to keeping it until May 1st,
+2023.  Please get in touch with us at @email{guix-devel@@gnu.org} if you
+would like to discuss it.
+@end quotation
 
 The general syntax is:
 
@@ -11099,14 +11422,14 @@ a container similar to the one the build daemon creates:
 $ guix build -K foo
 @dots{}
 $ cd /tmp/guix-build-foo.drv-0
-$ guix environment --no-grafts -C foo --ad-hoc strace gdb
+$ guix shell --no-grafts -C foo strace gdb
 [env]# source ./environment-variables
 [env]# cd foo-1.2
 @end example
 
 Here, @command{guix environment -C} creates a container and spawns a new
-shell in it (@pxref{Invoking guix environment}).  The @command{--ad-hoc
-strace gdb} part adds the @command{strace} and @command{gdb} commands to
+shell in it (@pxref{Invoking guix shell}).  The @command{strace gdb}
+part adds the @command{strace} and @command{gdb} commands to
 the container, which you may find handy while debugging.  The
 @option{--no-grafts} option makes sure we get the exact same
 environment, with ungrafted packages (@pxref{Security Updates}, for more
@@ -11120,7 +11443,7 @@ remove @file{/bin/sh}:
 @end example
 
 (Don't worry, this is harmless: this is all happening in the throw-away
-container created by @command{guix environment}.)
+container created by @command{guix shell}.)
 
 The @command{strace} command is probably not in the search path, but we
 can run:
@@ -13316,8 +13639,8 @@ is subject to radical change in the future.
 
 The purpose of @command{guix container} is to manipulate processes
 running within an isolated environment, commonly known as a
-``container'', typically created by the @command{guix environment}
-(@pxref{Invoking guix environment}) and @command{guix system container}
+``container'', typically created by the @command{guix shell}
+(@pxref{Invoking guix shell}) and @command{guix system container}
 (@pxref{Invoking guix system}) commands.
 
 The general syntax is:
@@ -13503,7 +13826,7 @@ listed.}.  Here's an example of the information it returns:
 $ sudo guix processes
 SessionPID: 19002
 ClientPID: 19090
-ClientCommand: guix environment --ad-hoc python
+ClientCommand: guix shell python
 
 SessionPID: 19402
 ClientPID: 19367
@@ -29695,8 +30018,7 @@ When the service is running, you can view its console by connecting to
 it with a VNC client, for example with:
 
 @example
-guix environment --ad-hoc tigervnc-client -- \
-         vncviewer localhost:5900
+guix shell tigervnc-client -- vncviewer localhost:5900
 @end example
 
 The default configuration (see @code{hurd-vm-configuration} below)
diff --git a/guix/scripts/environment.scm b/guix/scripts/environment.scm
index 54f48a7482..77956fc018 100644
--- a/guix/scripts/environment.scm
+++ b/guix/scripts/environment.scm
@@ -50,7 +50,11 @@ (define-module (guix scripts environment)
   #:use-module (srfi srfi-37)
   #:use-module (srfi srfi-98)
   #:export (assert-container-features
-            guix-environment))
+            guix-environment
+            guix-environment*
+            show-environment-options-help
+            (%options . %environment-options)
+            (%default-options . %environment-default-options)))
 
 (define %default-shell
   (or (getenv "SHELL") "/bin/sh"))
@@ -66,23 +70,16 @@ (define* (show-search-paths profile manifest #:key pure?)
                (newline)))
             (profile-search-paths profile manifest)))
 
-(define (show-help)
-  (display (G_ "Usage: guix environment [OPTION]... PACKAGE... [-- COMMAND...]
-Build an environment that includes the dependencies of PACKAGE and execute
-COMMAND or an interactive shell in that environment.\n"))
+(define (show-environment-options-help)
+  "Print help about options shared between 'guix environment' and 'guix
+shell'."
   (display (G_ "
   -e, --expression=EXPR  create environment for the package that EXPR
                          evaluates to"))
   (display (G_ "
-  -l, --load=FILE        create environment for the package that the code within
-                         FILE evaluates to"))
-  (display (G_ "
   -m, --manifest=FILE    create environment with the manifest from FILE"))
   (display (G_ "
   -p, --profile=PATH     create environment from profile at PATH"))
-  (display (G_ "
-      --ad-hoc           include all specified packages in the environment instead
-                         of only their inputs"))
   (display (G_ "
       --pure             unset existing environment variables"))
   (display (G_ "
@@ -118,7 +115,24 @@ (define (show-help)
   (display (G_ "
   -v, --verbosity=LEVEL  use the given verbosity LEVEL"))
   (display (G_ "
-      --bootstrap        use bootstrap binaries to build the environment"))
+      --bootstrap        use bootstrap binaries to build the environment")))
+
+(define (show-help)
+  (display (G_ "Usage: guix environment [OPTION]... PACKAGE... [-- COMMAND...]
+Build an environment that includes the dependencies of PACKAGE and execute
+COMMAND or an interactive shell in that environment.\n"))
+  (warning (G_ "This command is deprecated in favor of 'guix shell'.\n"))
+  (newline)
+
+  ;; These two options are left out in 'guix shell'.
+  (display (G_ "
+  -l, --load=FILE        create environment for the package that the code within
+                         FILE evaluates to"))
+  (display (G_ "
+      --ad-hoc           include all specified packages in the environment instead
+                         of only their inputs"))
+
+  (show-environment-options-help)
   (newline)
   (show-build-options-help)
   (newline)
@@ -649,11 +663,15 @@ (define (register-gc-root target root)
 
 (define-command (guix-environment . args)
   (category development)
-  (synopsis "spawn one-off software environments")
+  (synopsis "spawn one-off software environments (deprecated)")
 
+  (guix-environment* (parse-args args)))
+
+(define (guix-environment* opts)
+  "Run the 'guix environment' command on OPTS, an alist resulting for
+command-line option processing with 'parse-command-line'."
   (with-error-handling
-    (let* ((opts       (parse-args args))
-           (pure?      (assoc-ref opts 'pure))
+    (let* ((pure?      (assoc-ref opts 'pure))
            (container? (assoc-ref opts 'container?))
            (link-prof? (assoc-ref opts 'link-profile?))
            (network?   (assoc-ref opts 'network?))
@@ -724,8 +742,8 @@ (define manifest
                                      (prof-drv   (manifest->derivation
                                                   manifest system bootstrap?))
                                      (profile -> (if profile
-                                                   (readlink* profile)
-                                                   (derivation->output-path prof-drv)))
+                                                     (readlink* profile)
+                                                     (derivation->output-path prof-drv)))
                                      (gc-root -> (assoc-ref opts 'gc-root)))
 
                   ;; First build the inputs.  This is necessary even for
diff --git a/guix/scripts/shell.scm b/guix/scripts/shell.scm
new file mode 100644
index 0000000000..190dd8837d
--- /dev/null
+++ b/guix/scripts/shell.scm
@@ -0,0 +1,135 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2021 Ludovic Courtès <ludo@gnu.org>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (guix scripts shell)
+  #:use-module (guix ui)
+  #:use-module (guix scripts environment)
+  #:autoload   (guix scripts build) (show-build-options-help)
+  #:autoload   (guix transformations) (show-transformation-options-help)
+  #:use-module (guix scripts)
+  #:use-module (srfi srfi-1)
+  #:use-module (srfi srfi-26)
+  #:use-module (srfi srfi-37)
+  #:use-module (srfi srfi-71)
+  #:use-module (ice-9 match)
+  #:export (guix-shell))
+
+(define (show-help)
+  (display (G_ "Usage: guix shell [OPTION] PACKAGES... [-- COMMAND...]
+Build an environment that includes PACKAGES and execute COMMAND or an
+interactive shell in that environment.\n"))
+  (newline)
+
+  ;; These two options differ from 'guix environment'.
+  (display (G_ "
+  -D, --development      include the development inputs of the next package"))
+  (display (G_ "
+  -f, --file=FILE        create environment for the package that the code within
+                         FILE evaluates to"))
+
+  (show-environment-options-help)
+  (newline)
+  (show-build-options-help)
+  (newline)
+  (show-transformation-options-help)
+  (newline)
+  (display (G_ "
+  -h, --help             display this help and exit"))
+  (display (G_ "
+  -V, --version          display version information and exit"))
+  (newline)
+  (show-bug-report-information))
+
+(define (tag-package-arg opts arg)
+  "Return a two-element list with the form (TAG ARG) that tags ARG with either
+'ad-hoc' in OPTS has the 'ad-hoc?' key set to #t, or 'inputs' otherwise."
+  (if (assoc-ref opts 'ad-hoc?)
+      `(ad-hoc-package ,arg)
+      `(package ,arg)))
+
+(define (ensure-ad-hoc alist)
+  (if (assq-ref alist 'ad-hoc?)
+      alist
+      `((ad-hoc? . #t) ,@alist)))
+
+(define (wrapped-option opt)
+  "Wrap OPT, a SRFI-37 option, such that its processor always adds the
+'ad-hoc?' flag to the resulting alist."
+  (option (option-names opt)
+          (option-required-arg? opt)
+          (option-optional-arg? opt)
+          (compose ensure-ad-hoc (option-processor opt))))
+
+(define %options
+  ;; Specification of the command-line options.
+  (let ((to-remove '("ad-hoc" "inherit" "load" "help" "version")))
+    (append
+        (list (option '(#\h "help") #f #f
+                      (lambda args
+                        (show-help)
+                        (exit 0)))
+              (option '(#\V "version") #f #f
+                      (lambda args
+                        (show-version-and-exit "guix shell")))
+
+              (option '(#\D "development") #f #f
+                      (lambda (opt name arg result)
+                        ;; Temporarily remove the 'ad-hoc?' flag from result.
+                        ;; The next option will put it back thanks to
+                        ;; 'wrapped-option'.
+                        (alist-delete 'ad-hoc? result)))
+
+              ;; For consistency with 'guix package', support '-f' rather than
+              ;; '-l' like 'guix environment' does.
+              (option '(#\f "file") #t #f
+                      (lambda (opt name arg result)
+                        (alist-cons 'load (tag-package-arg result arg)
+                                    result))))
+        (filter-map (lambda (opt)
+                      (and (not (any (lambda (name)
+                                       (member name to-remove))
+                                     (option-names opt)))
+                           (wrapped-option opt)))
+                    %environment-options))))
+
+(define %default-options
+  `((ad-hoc? . #t)                                ;always true
+    ,@%environment-default-options))
+
+(define (parse-args args)
+  "Parse the list of command line arguments ARGS."
+  (define (handle-argument arg result)
+    (alist-cons 'package (tag-package-arg result arg)
+                (ensure-ad-hoc result)))
+
+  ;; The '--' token is used to separate the command to run from the rest of
+  ;; the operands.
+  (let ((args command (break (cut string=? "--" <>) args)))
+    (let ((opts (parse-command-line args %options (list %default-options)
+                                    #:argument-handler handle-argument)))
+      (match command
+        (() opts)
+        (("--") opts)
+        (("--" command ...) (alist-cons 'exec command opts))))))
+
+\f
+(define-command (guix-shell . args)
+  (category development)
+  (synopsis "spawn one-off software environments")
+
+  (guix-environment* (parse-args args)))
diff --git a/po/guix/POTFILES.in b/po/guix/POTFILES.in
index 5e77b3c230..ee0adbcf3c 100644
--- a/po/guix/POTFILES.in
+++ b/po/guix/POTFILES.in
@@ -140,5 +140,6 @@ guix/scripts/offload.scm
 guix/scripts/perform-download.scm
 guix/scripts/refresh.scm
 guix/scripts/repl.scm
+guix/scripts/shell.scm
 guix/scripts/system/reconfigure.scm
 nix/nix-daemon/guix-daemon.cc
diff --git a/tests/guix-shell.sh b/tests/guix-shell.sh
new file mode 100644
index 0000000000..f08637f7ff
--- /dev/null
+++ b/tests/guix-shell.sh
@@ -0,0 +1,54 @@
+# GNU Guix --- Functional package management for GNU
+# Copyright © 2021 Ludovic Courtès <ludo@gnu.org>
+#
+# This file is part of GNU Guix.
+#
+# GNU Guix is free software; you can redistribute it and/or modify it
+# under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or (at
+# your option) any later version.
+#
+# GNU Guix is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
+
+#
+# Test the 'guix shell' alias.
+#
+
+guix shell --version
+
+tmpdir="t-guix-shell-$$"
+trap 'rm -r "$tmpdir"' EXIT
+mkdir "$tmpdir"
+
+guix shell --bootstrap --pure guile-bootstrap -- guile --version
+
+# '--ad-hoc' is a thing of the past.
+! guix shell --ad-hoc guile-bootstrap
+
+if guile -c '(getaddrinfo "www.gnu.org" "80" AI_NUMERICSERV)' 2> /dev/null
+then
+    # Compute the build environment for the initial GNU Make.
+    guix shell --bootstrap --no-substitutes --search-paths --pure \
+         -D -e '(@ (guix tests) gnu-make-for-tests)' > "$tmpdir/a"
+
+    # Make sure bootstrap binaries are in the profile.
+    profile=`grep "^export PATH" "$tmpdir/a" | sed -r 's|^.*="(.*)/bin"|\1|'`
+
+    # Make sure the bootstrap binaries are all listed where they belong.
+    grep -E "^export PATH=\"$profile/bin\""         "$tmpdir/a"
+    grep -E "^export CPATH=\"$profile/include\""    "$tmpdir/a"
+    grep -E "^export LIBRARY_PATH=\"$profile/lib\"" "$tmpdir/a"
+    for dep in bootstrap-binaries-0 gcc-bootstrap-0 glibc-bootstrap-0
+    do
+	guix gc --references "$profile" | grep "$dep"
+    done
+
+    # 'make-boot0' itself must not be listed.
+    ! guix gc --references "$profile" | grep make-boot0
+fi
-- 
2.33.0

[bug#50960] [PATCH v3 04/10] shell: By default load the local 'guix.scm' or 'manifest.scm' file.

[Ludovic Courtès]

* guix/scripts/shell.scm (parse-args): Add call to 'auto-detect-manifest'.
(authorized-directory-file, authorized-shell-directory?)
(find-file-in-parent-directories, auto-detect-manifest): New procedures.
* tests/guix-shell.sh: Add test.
* doc/guix.texi (Invoking guix shell): Document it.
---
 doc/guix.texi          |  14 +++++
 guix/scripts/shell.scm | 119 +++++++++++++++++++++++++++++++++++++++--
 tests/guix-shell.sh    |  50 ++++++++++++++++-
 3 files changed, 176 insertions(+), 7 deletions(-)

diff --git a/doc/guix.texi b/doc/guix.texi
index 63612728ed..7c8f0c1f9b 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -5621,6 +5621,20 @@ before @command{guix shell} was invoked.  The next garbage collection
 (@pxref{Invoking guix gc}) may clean up packages that were installed in
 the environment and that are no longer used outside of it.
 
+As an added convenience, when running from a directory that contains a
+@file{guix.scm} or a @file{manifest.scm} file, possibly in a parent
+directory, @command{guix shell} automatically loads the file---provided
+the directory is listed in
+@file{~/.config/guix/shell-authorized-directories}, and only for
+interactive use:
+
+@example
+guix shell
+@end example
+
+This provides an easy way to define, share, and enter development
+environments.
+
 By default, the shell session or command runs in an @emph{augmented}
 environment, where the new packages are added to search path environment
 variables such as @code{PATH}.  You can, instead, choose to create an
diff --git a/guix/scripts/shell.scm b/guix/scripts/shell.scm
index 190dd8837d..45fd536145 100644
--- a/guix/scripts/shell.scm
+++ b/guix/scripts/shell.scm
@@ -18,15 +18,20 @@
 
 (define-module (guix scripts shell)
   #:use-module (guix ui)
+  #:use-module ((guix diagnostics) #:select (location))
   #:use-module (guix scripts environment)
   #:autoload   (guix scripts build) (show-build-options-help)
   #:autoload   (guix transformations) (show-transformation-options-help)
   #:use-module (guix scripts)
+  #:use-module (guix packages)
+  #:use-module (guix profiles)
   #:use-module (srfi srfi-1)
   #:use-module (srfi srfi-26)
   #:use-module (srfi srfi-37)
   #:use-module (srfi srfi-71)
   #:use-module (ice-9 match)
+  #:autoload   (ice-9 rdelim) (read-line)
+  #:autoload   (guix utils) (config-directory)
   #:export (guix-shell))
 
 (define (show-help)
@@ -41,6 +46,8 @@ (define (show-help)
   (display (G_ "
   -f, --file=FILE        create environment for the package that the code within
                          FILE evaluates to"))
+  (display (G_ "
+  -q                     inhibit loading of 'guix.scm' and 'manifest.scm'"))
 
   (show-environment-options-help)
   (newline)
@@ -99,7 +106,10 @@ (define %options
               (option '(#\f "file") #t #f
                       (lambda (opt name arg result)
                         (alist-cons 'load (tag-package-arg result arg)
-                                    result))))
+                                    result)))
+              (option '(#\q) #f #f
+                      (lambda (opt name arg result)
+                        (alist-cons 'explicit-loading? #t result))))
         (filter-map (lambda (opt)
                       (and (not (any (lambda (name)
                                        (member name to-remove))
@@ -122,10 +132,109 @@ (define (handle-argument arg result)
   (let ((args command (break (cut string=? "--" <>) args)))
     (let ((opts (parse-command-line args %options (list %default-options)
                                     #:argument-handler handle-argument)))
-      (match command
-        (() opts)
-        (("--") opts)
-        (("--" command ...) (alist-cons 'exec command opts))))))
+      (auto-detect-manifest
+       (match command
+         (() opts)
+         (("--") opts)
+         (("--" command ...) (alist-cons 'exec command opts)))))))
+
+(define (find-file-in-parent-directories candidates)
+  "Find one of CANDIDATES in the current directory or one of its ancestors."
+  (define start (getcwd))
+  (define device (stat:dev (stat start)))
+
+  (let loop ((directory start))
+    (let ((stat (stat directory)))
+      (and (= (stat:uid stat) (getuid))
+           (= (stat:dev stat) device)
+           (or (any (lambda (candidate)
+                      (let ((candidate (string-append directory "/" candidate)))
+                        (and (file-exists? candidate) candidate)))
+                    candidates)
+               (and (not (string=? directory "/"))
+                    (loop (dirname directory)))))))) ;lexical ".." resolution
+
+(define (authorized-directory-file)
+  "Return the name of the file listing directories for which 'guix shell' may
+automatically load 'guix.scm' or 'manifest.scm' files."
+  (string-append (config-directory) "/shell-authorized-directories"))
+
+(define (authorized-shell-directory? directory)
+  "Return true if DIRECTORY is among the authorized directories for automatic
+loading.  The list of authorized directories is read from
+'authorized-directory-file'; each line must be either: an absolute file name,
+a hash-prefixed comment, or a blank line."
+  (catch 'system-error
+    (lambda ()
+      (call-with-input-file (authorized-directory-file)
+        (lambda (port)
+          (let loop ()
+            (match (read-line port)
+              ((? eof-object?) #f)
+              ((= string-trim line)
+               (cond ((string-prefix? "#" line)   ;comment
+                      (loop))
+                     ((string-prefix? "/" line)   ;absolute file name
+                      (or (string=? line directory)
+                          (loop)))
+                     ((string-null? (string-trim-right line)) ;blank line
+                      (loop))
+                     (else                        ;bogus line
+                      (let ((loc (location (port-filename port)
+                                           (port-line port)
+                                           (port-column port))))
+                        (warning loc (G_ "ignoring invalid file name: '~a'~%")
+                                 line))))))))))
+    (const #f)))
+
+(define (auto-detect-manifest opts)
+  "If OPTS do not specify packages or a manifest, load a \"guix.scm\" or
+\"manifest.scm\" file from the current directory or one of its ancestors.
+Return the modified OPTS."
+  (define (options-contain-payload? opts)
+    (match opts
+      (() #f)
+      ((('package . _) . _) #t)
+      ((('load . _) . _) #t)
+      ((('manifest . _) . _) #t)
+      ((('expression . _) . _) #t)
+      ((_ . rest) (options-contain-payload? rest))))
+
+  (define interactive?
+    (not (assoc-ref opts 'exec)))
+
+  (define disallow-implicit-load?
+    (assoc-ref opts 'explicit-loading?))
+
+  (if (or (not interactive?)
+          disallow-implicit-load?
+          (options-contain-payload? opts))
+      opts
+      (match (find-file-in-parent-directories '("guix.scm" "manifest.scm"))
+        (#f
+         (warning (G_ "no packages specified; creating an empty environment~%"))
+         opts)
+        (file
+         (if (authorized-shell-directory? (dirname file))
+             (begin
+               (info (G_ "loading environment from '~a'...~%") file)
+               (match (basename file)
+                 ("guix.scm" (alist-cons 'load `(package ,file) opts))
+                 ("manifest.scm" (alist-cons 'manifest file opts))))
+             (begin
+               (warning (G_ "not loading '~a' because not authorized to do so~%")
+                        file)
+               (display-hint (format #f (G_ "To allow automatic loading of
+@file{~a} when running @command{guix shell}, you must explicitly authorize its
+directory, like so:
+
+@example
+echo ~a >> ~a
+@end example\n")
+                                     file
+                                     (dirname file)
+                                     (authorized-directory-file)))
+               opts))))))
 
 \f
 (define-command (guix-shell . args)
diff --git a/tests/guix-shell.sh b/tests/guix-shell.sh
index f08637f7ff..95725cba2d 100644
--- a/tests/guix-shell.sh
+++ b/tests/guix-shell.sh
@@ -22,15 +22,55 @@
 
 guix shell --version
 
+configdir="t-guix-shell-config-$$"
 tmpdir="t-guix-shell-$$"
-trap 'rm -r "$tmpdir"' EXIT
-mkdir "$tmpdir"
+trap 'rm -r "$tmpdir" "$configdir"' EXIT
+mkdir "$tmpdir" "$configdir" "$configdir/guix"
+
+XDG_CONFIG_HOME="$(realpath $configdir)"
+export XDG_CONFIG_HOME
 
 guix shell --bootstrap --pure guile-bootstrap -- guile --version
 
 # '--ad-hoc' is a thing of the past.
 ! guix shell --ad-hoc guile-bootstrap
 
+# Ignoring unauthorized files.
+cat > "$tmpdir/guix.scm" <<EOF
+This is a broken guix.scm file.
+EOF
+(cd "$tmpdir"; SHELL="$(type -P true)" guix shell --bootstrap)
+
+# Authorize the directory.
+echo "$(realpath "$tmpdir")" > "$configdir/guix/shell-authorized-directories"
+
+# Ignoring 'manifest.scm' and 'guix.scm' in non-interactive use.
+(cd "$tmpdir"; guix shell --bootstrap -- true)
+mv "$tmpdir/guix.scm" "$tmpdir/manifest.scm"
+(cd "$tmpdir"; guix shell --bootstrap -- true)
+rm "$tmpdir/manifest.scm"
+
+# Honoring the local 'manifest.scm' file.
+cat > "$tmpdir/manifest.scm" <<EOF
+(specifications->manifest '("guile-bootstrap"))
+EOF
+cat > "$tmpdir/fake-shell.sh" <<EOF
+#!$SHELL
+# This fake shell allows us to test interactive use.
+exec echo "\$GUIX_ENVIRONMENT"
+EOF
+chmod +x "$tmpdir/fake-shell.sh"
+profile1="$(cd "$tmpdir"; SHELL="$(realpath fake-shell.sh)" guix shell --bootstrap)"
+profile2="$(guix shell --bootstrap guile-bootstrap -- "$SHELL" -c 'echo $GUIX_ENVIRONMENT')"
+test -n "$profile1"
+test "$profile1" = "$profile2"
+rm "$tmpdir/manifest.scm"
+
+# Do not read manifest when passed '-q'.
+echo "Broken manifest." > "$tmpdir/manifest.scm"
+(cd "$tmpdir"; SHELL="$(realpath fake-shell.sh)" guix shell --bootstrap -q)
+rm "$tmpdir/manifest.scm"
+
 if guile -c '(getaddrinfo "www.gnu.org" "80" AI_NUMERICSERV)' 2> /dev/null
 then
     # Compute the build environment for the initial GNU Make.
@@ -51,4 +91,10 @@ then
 
     # 'make-boot0' itself must not be listed.
     ! guix gc --references "$profile" | grep make-boot0
+
+    # Honoring the local 'guix.scm' file.
+    echo '(@ (guix tests) gnu-make-for-tests)' > "$tmpdir/guix.scm"
+    (cd "$tmpdir"; guix shell --bootstrap --search-paths --pure > "b")
+    cmp "$tmpdir/a" "$tmpdir/b"
+    rm "$tmpdir/guix.scm"
 fi
-- 
2.33.0

[bug#50960] [PATCH v3 05/10] environment: Add tests for '--profile'.

[Ludovic Courtès]

This is a followup to a643deac2de81755a1843a3b41dd53857678bebc.

* tests/guix-environment-container.sh, tests/guix-environment.sh: Add
tests for '--profile'.
---
 tests/guix-environment-container.sh | 8 ++++++++
 tests/guix-environment.sh           | 7 +++++++
 2 files changed, 15 insertions(+)

diff --git a/tests/guix-environment-container.sh b/tests/guix-environment-container.sh
index f2d15c8d0c..2e238c501d 100644
--- a/tests/guix-environment-container.sh
+++ b/tests/guix-environment-container.sh
@@ -44,6 +44,14 @@ else
     test $? = 42
 fi
 
+# Try '--root' and '--profile'.
+root="$tmpdir/root"
+guix environment -C --ad-hoc --bootstrap guile-bootstrap -r "$root" -- guile --version
+guix environment -C -p "$root" --bootstrap -- guile --version
+path1=$(guix environment -C -p "$root" --bootstrap -- guile -c '(display (getenv "PATH"))')
+path2=$(guix environment -C --ad-hoc --bootstrap guile-bootstrap  -- guile -c '(display (getenv "PATH"))')
+test "$path1" = "$path2"
+
 # Make sure "localhost" resolves.
 guix environment --container --ad-hoc --bootstrap guile-bootstrap \
      -- guile -c '(exit (pair? (getaddrinfo "localhost" "80")))'
diff --git a/tests/guix-environment.sh b/tests/guix-environment.sh
index afadcbe195..f4fc2e39ed 100644
--- a/tests/guix-environment.sh
+++ b/tests/guix-environment.sh
@@ -119,6 +119,13 @@ test `readlink "$gcroot"` = "$expected"
 guix environment --bootstrap -r "$gcroot" --ad-hoc guile-bootstrap \
      -- guile -c 1
 test `readlink "$gcroot"` = "$expected"
+
+# Make sure '-p' works as expected.
+test $(guix environment -p "$gcroot" -- "$SHELL" -c 'echo $GUIX_ENVIRONMENT') = "$expected"
+paths1="$(guix environment -p "$gcroot" --search-paths)"
+paths2="$(guix environment --bootstrap --ad-hoc guile-bootstrap --search-paths)"
+test "$paths1" = "$paths2"
+
 rm "$gcroot"
 
 # Try '-r' with a relative file name.
-- 
2.33.0

[bug#50960] [PATCH v3 06/10] environment: Skip derivation computation when '--profile' is used.

[Ludovic Courtès]

* guix/scripts/environment.scm (guix-environment*): Bypass calls to
'package-derivation' and to 'manifest->derivation' when PROFILE is
true.
---
 guix/scripts/environment.scm | 23 +++++++++++++----------
 1 file changed, 13 insertions(+), 10 deletions(-)

diff --git a/guix/scripts/environment.scm b/guix/scripts/environment.scm
index 77956fc018..32f376fdd2 100644
--- a/guix/scripts/environment.scm
+++ b/guix/scripts/environment.scm
@@ -729,18 +729,21 @@ (define manifest
             ;; Use the bootstrap Guile when requested.
             (parameterize ((%graft? (assoc-ref opts 'graft?))
                            (%guile-for-build
-                            (package-derivation
-                             store
-                             (if bootstrap?
-                                 %bootstrap-guile
-                                 (default-guile)))))
+                            (and (or container? (not profile))
+                                 (package-derivation
+                                  store
+                                  (if bootstrap?
+                                      %bootstrap-guile
+                                      (default-guile))))))
               (run-with-store store
                 ;; Containers need a Bourne shell at /bin/sh.
                 (mlet* %store-monad ((bash       (environment-bash container?
                                                                    bootstrap?
                                                                    system))
-                                     (prof-drv   (manifest->derivation
-                                                  manifest system bootstrap?))
+                                     (prof-drv   (if profile
+                                                     (return #f)
+                                                     (manifest->derivation
+                                                      manifest system bootstrap?)))
                                      (profile -> (if profile
                                                      (readlink* profile)
                                                      (derivation->output-path prof-drv)))
@@ -750,9 +753,9 @@ (define manifest
                   ;; --search-paths.  Additionally, we might need to build bash for
                   ;; a container.
                   (mbegin %store-monad
-                    (built-derivations (if (derivation? bash)
-                                           (list prof-drv bash)
-                                           (list prof-drv)))
+                    (built-derivations (append
+                                           (if prof-drv (list prof-drv) '())
+                                           (if (derivation? bash) (list bash) '())))
                     (mwhen gc-root
                       (register-gc-root profile gc-root))
 
-- 
2.33.0

[bug#50960] [PATCH v3 07/10] environment: Do not connect to the daemon when '--profile' is used.

[Ludovic Courtès]

This further speeds up the 'guix environment -p PROFILE' case.

* guix/scripts/environment.scm (guix-environment*)[store-needed?]: New
variable.
[with-store/maybe]: New macro.
Use it instead of 'with-store', and remove 'with-build-handler' form.
---
 guix/scripts/environment.scm | 169 +++++++++++++++++++----------------
 1 file changed, 93 insertions(+), 76 deletions(-)

diff --git a/guix/scripts/environment.scm b/guix/scripts/environment.scm
index 32f376fdd2..e23d52df39 100644
--- a/guix/scripts/environment.scm
+++ b/guix/scripts/environment.scm
@@ -691,6 +691,26 @@ (define (guix-environment* opts)
            (mappings   (pick-all opts 'file-system-mapping))
            (white-list (pick-all opts 'inherit-regexp)))
 
+      (define store-needed?
+        ;; Whether connecting to the daemon is needed.
+        (or container? (not profile)))
+
+      (define-syntax-rule (with-store/maybe store exp ...)
+        ;; Evaluate EXP... with STORE bound to a connection, unless
+        ;; STORE-NEEDED? is false, in which case STORE is bound to #f.
+        (let ((proc (lambda (store) exp ...)))
+          (if store-needed?
+              (with-store s
+                (set-build-options-from-command-line s opts)
+                (with-build-handler (build-notifier #:use-substitutes?
+                                                    (assoc-ref opts 'substitutes?)
+                                                    #:verbosity
+                                                    (assoc-ref opts 'verbosity)
+                                                    #:dry-run?
+                                                    (assoc-ref opts 'dry-run?))
+                  (proc s)))
+              (proc #f))))
+
       (when container? (assert-container-features))
 
       (when (and (not container?) link-prof?)
@@ -701,88 +721,85 @@ (define (guix-environment* opts)
         (leave (G_ "--no-cwd cannot be used without --container~%")))
 
 
-      (with-store store
-        (with-build-handler (build-notifier #:use-substitutes?
-                                            (assoc-ref opts 'substitutes?)
-                                            #:verbosity
-                                            (assoc-ref opts 'verbosity)
-                                            #:dry-run?
-                                            (assoc-ref opts 'dry-run?))
-          (with-status-verbosity (assoc-ref opts 'verbosity)
-            (define manifest-from-opts
-              (options/resolve-packages store opts))
+      (with-store/maybe store
+        (with-status-verbosity (assoc-ref opts 'verbosity)
+          (define manifest-from-opts
+            (options/resolve-packages store opts))
 
-            (define manifest
-              (if profile
-                  (profile-manifest profile)
-                  manifest-from-opts))
+          (define manifest
+            (if profile
+                (profile-manifest profile)
+                manifest-from-opts))
 
-            (when (and profile
-                       (> (length (manifest-entries manifest-from-opts)) 0))
-              (leave (G_ "'--profile' cannot be used with package options~%")))
+          (when (and profile
+                     (> (length (manifest-entries manifest-from-opts)) 0))
+            (leave (G_ "'--profile' cannot be used with package options~%")))
 
-            (when (null? (manifest-entries manifest))
-              (warning (G_ "no packages specified; creating an empty environment~%")))
+          (when (null? (manifest-entries manifest))
+            (warning (G_ "no packages specified; creating an empty environment~%")))
 
-            (set-build-options-from-command-line store opts)
+          ;; Use the bootstrap Guile when requested.
+          (parameterize ((%graft? (assoc-ref opts 'graft?))
+                         (%guile-for-build
+                          (and store-needed?
+                               (package-derivation
+                                store
+                                (if bootstrap?
+                                    %bootstrap-guile
+                                    (default-guile))))))
+            (run-with-store store
+              ;; Containers need a Bourne shell at /bin/sh.
+              (mlet* %store-monad ((bash       (environment-bash container?
+                                                                 bootstrap?
+                                                                 system))
+                                   (prof-drv   (if profile
+                                                   (return #f)
+                                                   (manifest->derivation
+                                                    manifest system bootstrap?)))
+                                   (profile -> (if profile
+                                                   (readlink* profile)
+                                                   (derivation->output-path prof-drv)))
+                                   (gc-root -> (assoc-ref opts 'gc-root)))
 
-            ;; Use the bootstrap Guile when requested.
-            (parameterize ((%graft? (assoc-ref opts 'graft?))
-                           (%guile-for-build
-                            (and (or container? (not profile))
-                                 (package-derivation
-                                  store
-                                  (if bootstrap?
-                                      %bootstrap-guile
-                                      (default-guile))))))
-              (run-with-store store
-                ;; Containers need a Bourne shell at /bin/sh.
-                (mlet* %store-monad ((bash       (environment-bash container?
-                                                                   bootstrap?
-                                                                   system))
-                                     (prof-drv   (if profile
-                                                     (return #f)
-                                                     (manifest->derivation
-                                                      manifest system bootstrap?)))
-                                     (profile -> (if profile
-                                                     (readlink* profile)
-                                                     (derivation->output-path prof-drv)))
-                                     (gc-root -> (assoc-ref opts 'gc-root)))
-
-                  ;; First build the inputs.  This is necessary even for
-                  ;; --search-paths.  Additionally, we might need to build bash for
-                  ;; a container.
-                  (mbegin %store-monad
+                ;; First build the inputs.  This is necessary even for
+                ;; --search-paths.  Additionally, we might need to build bash for
+                ;; a container.
+                (mbegin %store-monad
+                  (mwhen store-needed?
                     (built-derivations (append
                                            (if prof-drv (list prof-drv) '())
-                                           (if (derivation? bash) (list bash) '())))
-                    (mwhen gc-root
-                      (register-gc-root profile gc-root))
+                                           (if (derivation? bash) (list bash) '()))))
+                  (mwhen gc-root
+                    (register-gc-root profile gc-root))
 
-                    (cond
-                     ((assoc-ref opts 'search-paths)
-                      (show-search-paths profile manifest #:pure? pure?)
-                      (return #t))
-                     (container?
-                      (let ((bash-binary
-                             (if bootstrap?
-                                 (derivation->output-path bash)
-                                 (string-append (derivation->output-path bash)
-                                                "/bin/sh"))))
-                        (launch-environment/container #:command command
-                                                      #:bash bash-binary
-                                                      #:user user
-                                                      #:user-mappings mappings
-                                                      #:profile profile
-                                                      #:manifest manifest
-                                                      #:white-list white-list
-                                                      #:link-profile? link-prof?
-                                                      #:network? network?
-                                                      #:map-cwd? (not no-cwd?))))
+                  (cond
+                   ((assoc-ref opts 'search-paths)
+                    (show-search-paths profile manifest #:pure? pure?)
+                    (return #t))
+                   (container?
+                    (let ((bash-binary
+                           (if bootstrap?
+                               (derivation->output-path bash)
+                               (string-append (derivation->output-path bash)
+                                              "/bin/sh"))))
+                      (launch-environment/container #:command command
+                                                    #:bash bash-binary
+                                                    #:user user
+                                                    #:user-mappings mappings
+                                                    #:profile profile
+                                                    #:manifest manifest
+                                                    #:white-list white-list
+                                                    #:link-profile? link-prof?
+                                                    #:network? network?
+                                                    #:map-cwd? (not no-cwd?))))
 
-                     (else
-                      (return
-                       (exit/status
-                        (launch-environment/fork command profile manifest
-                                                 #:white-list white-list
-                                                 #:pure? pure?)))))))))))))))
+                   (else
+                    (return
+                     (exit/status
+                      (launch-environment/fork command profile manifest
+                                               #:white-list white-list
+                                               #:pure? pure?))))))))))))))
+
+;;; Local Variables:
+;;; (put 'with-store/maybe 'scheme-indent-function 1)
+;;; End:
-- 
2.33.0

[bug#50960] [PATCH v3 08/10] environment: Autoload some modules.

[Ludovic Courtès]

This further speeds up the 'guix environment -p PROFILE' case.

* guix/scripts/environment.scm: Autoload a bunch of modules.
---
 guix/scripts/environment.scm | 19 +++++++++++--------
 1 file changed, 11 insertions(+), 8 deletions(-)

diff --git a/guix/scripts/environment.scm b/guix/scripts/environment.scm
index e23d52df39..05a43659da 100644
--- a/guix/scripts/environment.scm
+++ b/guix/scripts/environment.scm
@@ -34,15 +34,18 @@ (define-module (guix scripts environment)
   #:use-module (guix scripts)
   #:use-module (guix scripts build)
   #:use-module (guix transformations)
-  #:use-module (gnu build linux-container)
-  #:use-module (gnu build accounts)
-  #:use-module ((guix build syscalls) #:select (set-network-interface-up))
-  #:use-module (gnu system linux-container)
+  #:autoload   (gnu build linux-container) (call-with-container %namespaces
+                                            user-namespace-supported?
+                                            unprivileged-user-namespace-supported?
+                                            setgroups-supported?)
+  #:autoload   (gnu build accounts) (password-entry group-entry
+                                     password-entry-name password-entry-directory
+                                     write-passwd write-group)
+  #:autoload   (guix build syscalls) (set-network-interface-up)
   #:use-module (gnu system file-systems)
-  #:use-module (gnu packages)
-  #:use-module (gnu packages bash)
-  #:use-module ((gnu packages bootstrap)
-                #:select (bootstrap-executable %bootstrap-guile))
+  #:autoload   (gnu packages) (specification->package+output)
+  #:autoload   (gnu packages bash) (bash)
+  #:autoload   (gnu packages bootstrap) (bootstrap-executable %bootstrap-guile)
   #:use-module (ice-9 match)
   #:use-module (srfi srfi-1)
   #:use-module (srfi srfi-11)
-- 
2.33.0

[bug#50960] [PATCH v3 09/10] cache: Gracefully handle non-existent cache.

[Ludovic Courtès]

* guix/cache.scm (maybe-remove-expired-cache-entries): Ignore ENOENT
when writing EXPIRY-FILE.
---
 guix/cache.scm | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)

diff --git a/guix/cache.scm b/guix/cache.scm
index 0401a9d428..51009809bd 100644
--- a/guix/cache.scm
+++ b/guix/cache.scm
@@ -101,7 +101,13 @@ (define last-expiry-date
                                   #:now now
                                   #:entry-expiration entry-expiration
                                   #:delete-entry delete-entry)
-    (call-with-output-file expiry-file
-      (cute write (time-second now) <>))))
+    (catch 'system-error
+      (lambda ()
+        (call-with-output-file expiry-file
+          (cute write (time-second now) <>)))
+      (lambda args
+        ;; ENOENT means CACHE does not exist.
+        (unless (= ENOENT (system-error-errno args))
+          (apply throw args))))))
 
 ;;; cache.scm ends here
-- 
2.33.0

[bug#50960] [PATCH v3 10/10] shell: Maintain a profile cache.

[Ludovic Courtès]

shell: Maintain a profile cache.

With this change, running "guix shell" (no arguments) is equivalent to:

  guix environment -r ~/.cache/guix/profiles/some-root -l guix.scm

This is the cache miss.  On cache hit, it's equivalent to:

  guix environment -p ~/.cache/guix/profiles/some-root

... which can run in 0.1s.

* guix/scripts/shell.scm (options-with-caching): New procedure.
(parse-args): Use it.
(%profile-cache-directory): New variable.
(profile-cache-key, profile-cached-gc-root): New procedures.
(show-help, %options): Add '--rebuild-cache'.
(guix-shell)[cache-entries, entry-expiration]: New procedures.
Add call to 'maybe-remove-expired-cache-entries'.
* doc/guix.texi (Invoking guix shell): Document '--rebuild-cache'.
---
 doc/guix.texi          |  11 ++++
 guix/scripts/shell.scm | 127 ++++++++++++++++++++++++++++++++++++++---
 2 files changed, 130 insertions(+), 8 deletions(-)

diff --git a/doc/guix.texi b/doc/guix.texi
index 7c8f0c1f9b..dabd7fea1e 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -5769,6 +5769,17 @@ This is similar to the same-named option in @command{guix package}
 (@pxref{profile-manifest, @option{--manifest}}) and uses the same
 manifest files.
 
+@item --rebuild-cache
+When using @option{--manifest}, @option{--file}, or when invoked without
+arguments, @command{guix shell} caches the environment so that
+subsequent uses are instantaneous.  The cache is invalidated anytime the
+file is modified.
+
+The @option{--rebuild-cache} forces the cached environment to be
+refreshed even if the file has not changed.  This is useful if the
+@command{guix.scm} or @command{manifest.scm} has external dependencies,
+or if its behavior depends, say, on environment variables.
+
 @item --pure
 Unset existing environment variables when building the new environment, except
 those specified with @option{--preserve} (see below).  This has the effect of
diff --git a/guix/scripts/shell.scm b/guix/scripts/shell.scm
index 45fd536145..4062e8155d 100644
--- a/guix/scripts/shell.scm
+++ b/guix/scripts/shell.scm
@@ -31,7 +31,15 @@ (define-module (guix scripts shell)
   #:use-module (srfi srfi-71)
   #:use-module (ice-9 match)
   #:autoload   (ice-9 rdelim) (read-line)
-  #:autoload   (guix utils) (config-directory)
+  #:autoload   (guix base32) (bytevector->base32-string)
+  #:autoload   (rnrs bytevectors) (string->utf8)
+  #:autoload   (guix utils) (config-directory cache-directory)
+  #:autoload   (guix describe) (current-channels)
+  #:autoload   (guix channels) (channel-commit)
+  #:autoload   (gcrypt hash) (sha256)
+  #:use-module ((guix build utils) #:select (mkdir-p))
+  #:use-module (guix cache)
+  #:use-module ((ice-9 ftw) #:select (scandir))
   #:export (guix-shell))
 
 (define (show-help)
@@ -48,6 +56,8 @@ (define (show-help)
                          FILE evaluates to"))
   (display (G_ "
   -q                     inhibit loading of 'guix.scm' and 'manifest.scm'"))
+  (display (G_ "
+      --rebuild-cache    rebuild cached environment, if any"))
 
   (show-environment-options-help)
   (newline)
@@ -109,7 +119,10 @@ (define %options
                                     result)))
               (option '(#\q) #f #f
                       (lambda (opt name arg result)
-                        (alist-cons 'explicit-loading? #t result))))
+                        (alist-cons 'explicit-loading? #t result)))
+              (option '("rebuild-cache") #f #f
+                      (lambda (opt name arg result)
+                        (alist-cons 'rebuild-cache? #t result))))
         (filter-map (lambda (opt)
                       (and (not (any (lambda (name)
                                        (member name to-remove))
@@ -132,11 +145,12 @@ (define (handle-argument arg result)
   (let ((args command (break (cut string=? "--" <>) args)))
     (let ((opts (parse-command-line args %options (list %default-options)
                                     #:argument-handler handle-argument)))
-      (auto-detect-manifest
-       (match command
-         (() opts)
-         (("--") opts)
-         (("--" command ...) (alist-cons 'exec command opts)))))))
+      (options-with-caching
+       (auto-detect-manifest
+        (match command
+          (() opts)
+          (("--") opts)
+          (("--" command ...) (alist-cons 'exec command opts))))))))
 
 (define (find-file-in-parent-directories candidates)
   "Find one of CANDIDATES in the current directory or one of its ancestors."
@@ -187,6 +201,53 @@ (define (authorized-shell-directory? directory)
                                  line))))))))))
     (const #f)))
 
+(define (options-with-caching opts)
+  "If OPTS contains exactly one 'load' or one 'manifest' key, automatically
+add a 'profile' key (when a profile for that file is already in cache) or a
+'gc-root' key (to add the profile to cache)."
+  (define (single-file-for-caching opts)
+    (let loop ((opts opts)
+               (file #f))
+      (match opts
+        (() file)
+        ((('package . _) . _) #f)
+        ((('load . ('package candidate)) . rest)
+         (and (not file) (loop rest candidate)))
+        ((('manifest . candidate) . rest)
+         (and (not file) (loop rest candidate)))
+        ((('expression . _) . _) #f)
+        ((_ . rest) (loop rest file)))))
+
+  ;; Check whether there's a single 'load' or 'manifest' option.  When that is
+  ;; the case, arrange to automatically cache the resulting profile.
+  (match (single-file-for-caching opts)
+    (#f opts)
+    (file
+     (let* ((root (profile-cached-gc-root file))
+            (stat (and root (false-if-exception (lstat root)))))
+       (if (and (not (assoc-ref opts 'rebuild-cache?))
+                stat
+                (<= (stat:mtime ((@ (guile) stat) file))
+                    (stat:mtime stat)))
+           (let ((now (current-time)))
+             ;; Update the atime on ROOT to reflect usage.
+             (utime root
+                    now (stat:mtime stat) 0 (stat:mtimensec stat)
+                    AT_SYMLINK_NOFOLLOW)
+             (alist-cons 'profile root
+                         (remove (match-lambda
+                                   (('load . _) #t)
+                                   (('manifest . _) #t)
+                                   (_ #f))
+                                 opts)))          ;load right away
+           (if (and root (not (assq-ref opts 'gc-root)))
+               (begin
+                 (if stat
+                     (delete-file root)
+                     (mkdir-p (dirname root)))
+                 (alist-cons 'gc-root root opts))
+               opts))))))
+
 (define (auto-detect-manifest opts)
   "If OPTS do not specify packages or a manifest, load a \"guix.scm\" or
 \"manifest.scm\" file from the current directory or one of its ancestors.
@@ -236,9 +297,59 @@ (define disallow-implicit-load?
                                      (authorized-directory-file)))
                opts))))))
 
+\f
+;;;
+;;; Profile cache.
+;;;
+
+(define %profile-cache-directory
+  ;; Directory where profiles created by 'guix shell' alone (without extra
+  ;; options) are cached.
+  (make-parameter (string-append (cache-directory #:ensure? #f)
+                                 "/profiles")))
+
+(define (profile-cache-key file)
+  "Return the cache key for the profile corresponding to FILE, a 'guix.scm' or
+'manifest.scm' file, or #f if we lack channel information."
+  (match (current-channels)
+    (() #f)
+    (((= channel-commit commits) ...)
+     (let ((stat (stat file)))
+       (bytevector->base32-string
+        ;; Since FILE is not canonicalized, only include the device/inode
+        ;; numbers.  XXX: In some rare cases involving Btrfs and NFS, this can
+        ;; be insufficient: <https://lwn.net/Articles/866582/>.
+        (sha256 (string->utf8
+                 (string-append (string-join commits) ":"
+                                (number->string (stat:dev stat)) ":"
+                                (number->string (stat:ino stat))))))))))
+
+(define (profile-cached-gc-root file)
+  "Return the cached GC root for FILE, a 'guix.scm' or 'manifest.scm' file, or
+#f if we lack information to cache it."
+  (match (profile-cache-key file)
+    (#f  #f)
+    (key (string-append (%profile-cache-directory) "/" key))))
+
 \f
 (define-command (guix-shell . args)
   (category development)
   (synopsis "spawn one-off software environments")
 
-  (guix-environment* (parse-args args)))
+  (define (cache-entries directory)
+    (filter-map (match-lambda
+                  ((or "." "..") #f)
+                  (file (string-append directory "/" file)))
+                (or (scandir directory) '())))
+
+  (define* (entry-expiration file)
+    ;; Return the time at which FILE, a cached profile, is considered expired.
+    (match (false-if-exception (lstat file))
+      (#f 0)                       ;FILE may have been deleted in the meantime
+      (st (+ (stat:atime st) (* 60 60 24 7)))))
+
+  (let ((result (guix-environment* (parse-args args))))
+    (maybe-remove-expired-cache-entries (%profile-cache-directory)
+                                        cache-entries
+                                        #:entry-expiration entry-expiration)
+    result))
-- 
2.33.0

[bug#50960] [PATCH v3 00/10] Adding 'guix shell': last call!

[zimoun]

Hi,

On Mon, 18 Oct 2021 at 21:52, Ludovic Courtès <ludo@gnu.org> wrote:

> Ludovic Courtès (10):
>   packages: Add 'package-development-inputs'.
>   profiles: Add 'package->development-manifest'.

LGTM.  I have not checked the others. :-)

>   Add 'guix shell'.
>   shell: By default load the local 'guix.scm' or 'manifest.scm' file.
>   environment: Add tests for '--profile'.
>   environment: Skip derivation computation when '--profile' is used.
>   environment: Do not connect to the daemon when '--profile' is used.
>   environment: Autoload some modules.
>   cache: Gracefully handle non-existent cache.
>   shell: Maintain a profile cache.


Cheers,
simon

[bug#50960] [PATCH 00/10] Add 'guix shell' to subsume 'guix environment'

[zimoun]

Hi Ludo,

I have not reviewed all in detail, only applied the series and run some
checks based on my "workflow".  All looks good to me.


The environment variable named GUIX_ENVIRONMENT remains unchanged.  I
have no opinion if it is misleading or not for newcomers.


(I have not checked the behaviour about implicitly loading some files –
since I am not a big fan of implicit behaviour: «Explicit is better than
implicit.» and «In the face of ambiguity, refuse the temptation to
guess.», as a good Zen says – that’s not the consensus though.  The road
seems to go toward,

    guix time-machine -- shell

which appears to me handy in some cases but wrong on principles.  As I
explained here <http://issues.guix.gnu.org/50960#70>.)


About completion, “guix environment” is in zsh and fish, “guix shell” is
not yet.


Last, in the manual, this,

--8<---------------cut here---------------start------------->8---
Here, @command{guix environment -C} creates a container and spawns a new
shell in it (@pxref{Invoking guix shell}).  The @command{strace gdb}
part adds the @command{strace} and @command{gdb} commands to
the container, which you may find handy while debugging.  The
@option{--no-grafts} option makes sure we get the exact same
environment, with ungrafted packages (@pxref{Security Updates}, for more
info on grafts).
--8<---------------cut here---------------end--------------->8---

in the node «Debugging Build Failures» should be replaced.  Because the
example just above this snippet does not refer to “guix environment”.

Well, the section «Invoking guix environment» should be revamped.
Especially, all the examples.  Let do that by another commit (later).


Cheers,
simon

[bug#50960] [PATCH 00/10] Add 'guix shell' to subsume 'guix environment'

[Ludovic Courtès]

Hi!

zimoun <zimon.toutoune@gmail.com> skribis:

> The environment variable named GUIX_ENVIRONMENT remains unchanged.  I
> have no opinion if it is misleading or not for newcomers.

I think it’s fine.

[...]

> About completion, “guix environment” is in zsh and fish, “guix shell” is
> not yet.

Oops.  I updated “shell” support for Bash completion (see
80edb7df6586464aa40e84e103f0045452de95db).  Can someone proficient in
zsh or fish help with these two?

> Last, in the manual, this,
>
> Here, @command{guix environment -C} creates a container and spawns a new
> shell in it (@pxref{Invoking guix shell}).  The @command{strace gdb}
> part adds the @command{strace} and @command{gdb} commands to
> the container, which you may find handy while debugging.  The
> @option{--no-grafts} option makes sure we get the exact same
> environment, with ungrafted packages (@pxref{Security Updates}, for more
> info on grafts).
>
> in the node «Debugging Build Failures» should be replaced.  Because the
> example just above this snippet does not refer to “guix environment”.

Oops, fixed as well.

> Well, the section «Invoking guix environment» should be revamped.
> Especially, all the examples.  Let do that by another commit (later).

I don’t know.  I kept it as-is, in “frozen” state, on the grounds that
people will still be using it for some time and will want to see the
reference.  WDYT?

Thanks for these last-minute comments!

Ludo’.

[bug#50960] [PATCH 00/10] Add 'guix shell' to subsume 'guix environment'

[zimoun]

Hi Ludo,

On Mon, 25 Oct 2021 at 20:19, Ludovic Courtès <ludo@gnu.org> wrote:

>> Well, the section «Invoking guix environment» should be revamped.
>> Especially, all the examples.  Let do that by another commit (later).
>
> I don’t know.  I kept it as-is, in “frozen” state, on the grounds that
> people will still be using it for some time and will want to see the
> reference.  WDYT?

I agree with the messages I sent. :-)

<http://issues.guix.gnu.org/50960#33>
<http://issues.guix.gnu.org/50960#41>
<http://issues.guix.gnu.org/50960#43>

Other said, I would let the manual “frozen” until next release (or next
next release).  Then, I would remove the section and instead replace by
two points:

 1. the commit introducing “guix shell”, i.e.,
 4aa41a37f0819c9e1750bca0af47e632c2443d50 which is the last commit
 officially supporting “guix environment” somehow;

 2. if people want to run “guix environment” or read documentation, then
 they still can using “guix time-machine” with the mentioned commit.

Somehow, turn the current “guix environment” examples into “guix shell”,
remove “guix environment” section, and add an warning box to “guix
shell” section explaining the difference (--ad-hoc vs -D) and provide
the way to get the old manual.

Cheers,
simon

[bug#50960] [PATCH 00/10] Add 'guix shell' to subsume 'guix environment'

[Ludovic Courtès]

Hi!

Ludovic Courtès <ludo@gnu.org> skribis:

> This is v3 of the ‘guix shell’ patch set.  Changes since v2:
>
>   • Merged as a single commit the one that adds support for implicit
>     ‘guix.scm’/‘manifest.scm’ and then one that adds authorization
>     checks;
>
>   • Fixed ‘package-development-inputs’ bug/typo reported by zimoun;
>
>   • Fixed doc typos reported by Florian.
>
> If there are no objections, I’ll go ahead with this patch series
> in a few days.  I’ll also add a news entry, and I guess a blog post
> introducing it would be welcome.

I went ahead and pushed it, woohoo!

  4aa41a37f0 news: Add entry about "guix shell".
  9730692d9f shell: Maintain a profile cache.
  2cb0b3709a cache: Gracefully handle non-existent cache.
  3c96158438 environment: Autoload some modules.
  99499a2037 environment: Do not connect to the daemon when '--profile' is used.
  648a6eb03f environment: Skip derivation computation when '--profile' is used.
  10208952ea environment: Add tests for '--profile'.
  746584e0ca shell: By default load the local 'manifest.scm' or 'guix.scm' file.
  80edb7df65 Add 'guix shell'.
  23f99f1a29 profiles: Add 'package->development-manifest'.
  fb368f4e76 packages: Add 'package-development-inputs'.

I made minor changes:

  • addressing the issues zimoun reported (Bash completion and updating
    the “Debugging Failed Builds” node);

  • changing the order in which ‘guix shell’ without arguments looks for
    files, ‘manifest.scm’ first (because it’s the most expressive) and
    ‘guix.scm’ second, in case both are present (this was suggested by
    singpolyma on IRC¹).

Please report any issues you find!

There’s a draft blog post that I intend to publish tomorrow:

  https://git.savannah.gnu.org/cgit/guix/guix-artwork.git/tree/website/drafts/guix-environment-to-shell.md

Thanks everyone! 🎉

Ludo’.

¹ https://logs.guix.gnu.org/guix/2021-10-25.log#183552