* [bug#36469] [PATCH 0/2] 'guix pack' records environment variables @ 2019-07-02 8:37 Ludovic Courtès 2019-07-02 8:56 ` [bug#36469] [PATCH 1/2] pack: 'docker' backend records the profile's search paths Ludovic Courtès 2019-07-02 16:46 ` [bug#36469] [PATCH 0/2] 'guix pack' records environment variables zimoun 0 siblings, 2 replies; 9+ messages in thread From: Ludovic Courtès @ 2019-07-02 8:37 UTC (permalink / raw) To: 36469 Hello Guix! With these patches, images created by ‘guix pack -f docker’ or ‘guix pack -f singularity’ define the right environment variables for the profile. For instance, if you do: guix pack -f docker guile guile-json --entry-point=bin/guile Then: docker run THE-IMAGE will spawn Guile in an environment with $GUILE_LOAD_PATH pointing to the profile, such that the (json) module is automatically found. The change for Singularity is similar. I’ve tested it with an instance of Singularity 3.2.0. Unfortunately, we still have 2.x in Guix and that doesn’t quite support the environment file, so I’ve chosen to punt on this one. Thoughts? Ludo’. Ludovic Courtès (2): pack: 'docker' backend records the profile's search paths. pack: 'squashfs' backend records the profile's search paths. gnu/tests/docker.scm | 16 ++++++---- gnu/tests/singularity.scm | 18 ++++++++++- guix/docker.scm | 17 ++++++++--- guix/scripts/pack.scm | 64 ++++++++++++++++++++++++++++++++++++--- 4 files changed, 99 insertions(+), 16 deletions(-) -- 2.22.0 ^ permalink raw reply [flat|nested] 9+ messages in thread
* [bug#36469] [PATCH 1/2] pack: 'docker' backend records the profile's search paths. 2019-07-02 8:37 [bug#36469] [PATCH 0/2] 'guix pack' records environment variables Ludovic Courtès @ 2019-07-02 8:56 ` Ludovic Courtès 2019-07-02 8:56 ` [bug#36469] [PATCH 2/2] pack: 'squashfs' " Ludovic Courtès 2019-07-04 11:13 ` [bug#36469] [PATCH 1/2] pack: 'docker' backend records the profile's search paths Ricardo Wurmus 2019-07-02 16:46 ` [bug#36469] [PATCH 0/2] 'guix pack' records environment variables zimoun 1 sibling, 2 replies; 9+ messages in thread From: Ludovic Courtès @ 2019-07-02 8:56 UTC (permalink / raw) To: 36469; +Cc: Ludovic Courtès From: Ludovic Courtès <ludovic.courtes@inria.fr> * guix/docker.scm (config): Add #:environment parameter and honor it. (build-docker-image): Likewise, and pass it to 'config'. * guix/scripts/pack.scm (docker-image): Import (guix profiles) and (guix search-paths). Call 'profile-search-paths' and pass #:environment to 'build-docker-image'. * gnu/tests/docker.scm (run-docker-test)["Load docker image and run it"]: Add example that expects (json) to be available. * gnu/tests/docker.scm (build-tarball&run-docker-test): Replace %BOOTSTRAP-GUILE by GUILE-2.2 and GUILE-JSON in the environment. --- gnu/tests/docker.scm | 16 ++++++++++------ guix/docker.scm | 17 +++++++++++++---- guix/scripts/pack.scm | 23 +++++++++++++++++++---- 3 files changed, 42 insertions(+), 14 deletions(-) diff --git a/gnu/tests/docker.scm b/gnu/tests/docker.scm index f2674cdbe8..27fde49e75 100644 --- a/gnu/tests/docker.scm +++ b/gnu/tests/docker.scm @@ -27,7 +27,6 @@ #:use-module (gnu services networking) #:use-module (gnu services docker) #:use-module (gnu services desktop) - #:use-module (gnu packages bootstrap) ; %bootstrap-guile #:use-module (gnu packages docker) #:use-module (gnu packages guile) #:use-module (guix gexp) @@ -101,7 +100,7 @@ inside %DOCKER-OS." marionette)) (test-equal "Load docker image and run it" - '("hello world" "hi!") + '("hello world" "hi!" "JSON!") (marionette-eval `(begin (define slurp @@ -125,8 +124,13 @@ inside %DOCKER-OS." (response2 (slurp ;default entry point ,(string-append #$docker-cli "/bin/docker") "run" repository&tag - "-c" "(display \"hi!\")"))) - (list response1 response2))) + "-c" "(display \"hi!\")")) + (response3 (slurp ;default entry point + environment + ,(string-append #$docker-cli "/bin/docker") + "run" repository&tag + "-c" "(use-modules (json)) + (display (json-string->scm (scm->json-string \"JSON!\")))"))) + (list response1 response2 response3))) marionette)) (test-end) @@ -144,7 +148,7 @@ inside %DOCKER-OS." (version "0") (source #f) (build-system trivial-build-system) - (arguments `(#:guile ,%bootstrap-guile + (arguments `(#:guile ,guile-2.2 #:builder (let ((out (assoc-ref %outputs "out"))) (mkdir out) @@ -158,7 +162,7 @@ standard output device and then enters a new line.") (home-page #f) (license license:public-domain))) (profile (profile-derivation (packages->manifest - (list %bootstrap-guile + (list guile-2.2 guile-json guest-script-package)) #:hooks '() #:locales? #f)) diff --git a/guix/docker.scm b/guix/docker.scm index 7fe83d9797..b1bd226fa1 100644 --- a/guix/docker.scm +++ b/guix/docker.scm @@ -73,7 +73,7 @@ `((,(generate-tag path) . ((latest . ,id))))) ;; See https://github.com/opencontainers/image-spec/blob/master/config.md -(define* (config layer time arch #:key entry-point) +(define* (config layer time arch #:key entry-point (environment '())) "Generate a minimal image configuration for the given LAYER file." ;; "architecture" must be values matching "platform.arch" in the ;; runtime-spec at @@ -81,9 +81,13 @@ `((architecture . ,arch) (comment . "Generated by GNU Guix") (created . ,time) - (config . ,(if entry-point - `((entrypoint . ,entry-point)) - #nil)) + (config . ,`((env . ,(map (match-lambda + ((name . value) + (string-append name "=" value))) + environment)) + ,@(if entry-point + `((entrypoint . ,entry-point)) + '()))) (container_config . #nil) (os . "linux") (rootfs . ((type . "layers") @@ -113,6 +117,7 @@ return \"a\"." (system (utsname:machine (uname))) database entry-point + (environment '()) compressor (creation-time (current-time time-utc))) "Write to IMAGE a Docker image archive containing the given PATHS. PREFIX @@ -124,6 +129,9 @@ When DATABASE is true, copy it to /var/guix/db in the image and create When ENTRY-POINT is true, it must be a list of strings; it is stored as the entry point in the Docker image JSON structure. +ENVIRONMENT must be a list of name/value pairs. It specifies the environment +variables that must be defined in the resulting image. + SYMLINKS must be a list of (SOURCE -> TARGET) tuples describing symlinks to be created in the image, where each TARGET is relative to PREFIX. TRANSFORMATIONS must be a list of (OLD -> NEW) tuples describing how to @@ -234,6 +242,7 @@ SRFI-19 time-utc object, as the creation time in metadata." (lambda () (scm->json (config (string-append id "/layer.tar") time arch + #:environment environment #:entry-point entry-point)))) (with-output-to-file "manifest.json" (lambda () diff --git a/guix/scripts/pack.scm b/guix/scripts/pack.scm index c90b777222..bb6a8cda1a 100644 --- a/guix/scripts/pack.scm +++ b/guix/scripts/pack.scm @@ -27,6 +27,7 @@ #:use-module (guix utils) #:use-module (guix store) #:use-module ((guix status) #:select (with-status-verbosity)) + #:use-module ((guix self) #:select (make-config.scm)) #:use-module (guix grafts) #:autoload (guix inferior) (inferior-package?) #:use-module (guix monads) @@ -440,11 +441,24 @@ the image." (define build ;; Guile-JSON and Guile-Gcrypt are required by (guix docker). (with-extensions (list guile-json guile-gcrypt) - (with-imported-modules (source-module-closure '((guix docker) - (guix build store-copy)) - #:select? not-config?) + (with-imported-modules `(((guix config) => ,(make-config.scm)) + ,@(source-module-closure + `((guix docker) + (guix build store-copy) + (guix profiles) + (guix search-paths)) + #:select? not-config?)) #~(begin - (use-modules (guix docker) (srfi srfi-19) (guix build store-copy)) + (use-modules (guix docker) (guix build store-copy) + (guix profiles) (guix search-paths) + (srfi srfi-19) (ice-9 match)) + + (define environment + (map (match-lambda + ((spec . value) + (cons (search-path-specification-variable spec) + value))) + (profile-search-paths #$profile))) (setenv "PATH" (string-append #$archiver "/bin")) @@ -455,6 +469,7 @@ the image." #$profile #:database #+database #:system (or #$target (utsname:machine (uname))) + #:environment environment #:entry-point #$(and entry-point #~(string-append #$profile "/" #$entry-point)) -- 2.22.0 ^ permalink raw reply related [flat|nested] 9+ messages in thread
* [bug#36469] [PATCH 2/2] pack: 'squashfs' backend records the profile's search paths. 2019-07-02 8:56 ` [bug#36469] [PATCH 1/2] pack: 'docker' backend records the profile's search paths Ludovic Courtès @ 2019-07-02 8:56 ` Ludovic Courtès 2019-07-04 11:17 ` Ricardo Wurmus 2019-07-04 11:13 ` [bug#36469] [PATCH 1/2] pack: 'docker' backend records the profile's search paths Ricardo Wurmus 1 sibling, 1 reply; 9+ messages in thread From: Ludovic Courtès @ 2019-07-02 8:56 UTC (permalink / raw) To: 36469; +Cc: Ludovic Courtès From: Ludovic Courtès <ludovic.courtes@inria.fr> * guix/scripts/pack.scm (singularity-environment-file): New procedure. (squashfs-image): Use it, and create /.singularity/env/90-environment.sh. * gnu/tests/singularity.scm (run-singularity-test)["singularity run, with environment"]: New test, currently skipped. * gnu/tests/singularity.scm (build-tarball&run-singularity-test): Add GUILE-JSON to the profile. --- gnu/tests/singularity.scm | 18 ++++++++++++++++- guix/scripts/pack.scm | 41 ++++++++++++++++++++++++++++++++++++++- 2 files changed, 57 insertions(+), 2 deletions(-) diff --git a/gnu/tests/singularity.scm b/gnu/tests/singularity.scm index 668043a0bc..2f3a6f289d 100644 --- a/gnu/tests/singularity.scm +++ b/gnu/tests/singularity.scm @@ -111,6 +111,21 @@ "run" #$image "-c" "(exit 42)")) marionette)) + ;; FIXME: Singularity 2.x doesn't directly honor + ;; /.singularity.d/env/*.sh. Instead, you have to load those files + ;; manually, which we don't do. Remove 'test-skip' call once we've + ;; switch to Singularity 3.x. + (test-skip 1) + (test-equal "singularity run, with environment" + 0 + (marionette-eval + ;; Check whether GUILE_LOAD_PATH is properly set, allowing us to + ;; find the (json) module. + `(status:exit-val + (system* #$(file-append singularity "/bin/singularity") + "--debug" "run" #$image "-c" "(use-modules (json))")) + marionette)) + (test-end) (exit (= (test-runner-fail-count (test-runner-current)) 0))))) @@ -122,7 +137,8 @@ (guile (set-guile-for-build (default-guile))) ;; 'singularity exec' insists on having /bin/sh in the image. (profile (profile-derivation (packages->manifest - (list bash-minimal guile-2.2)) + (list bash-minimal + guile-2.2 guile-json)) #:hooks '() #:locales? #f)) (tarball (squashfs-image "singularity-pack" profile diff --git a/guix/scripts/pack.scm b/guix/scripts/pack.scm index bb6a8cda1a..4ac5dfc896 100644 --- a/guix/scripts/pack.scm +++ b/guix/scripts/pack.scm @@ -286,6 +286,32 @@ added to the pack." build #:references-graphs `(("profile" ,profile)))) +(define (singularity-environment-file profile) + "Return a shell script that defines the environment variables corresponding +to the search paths of PROFILE." + (define build + (with-extensions (list guile-gcrypt) + (with-imported-modules `(((guix config) => ,(make-config.scm)) + ,@(source-module-closure + `((guix profiles) + (guix search-paths)) + #:select? not-config?)) + #~(begin + (use-modules (guix profiles) (guix search-paths) + (ice-9 match)) + + (call-with-output-file #$output + (lambda (port) + (for-each (match-lambda + ((spec . value) + (format port "~a=~a~%export ~a~%" + (search-path-specification-variable spec) + value + (search-path-specification-variable spec)))) + (profile-search-paths #$profile)))))))) + + (computed-file "singularity-environment.sh" build)) + (define* (squashfs-image name profile #:key target (profile-name "guix-profile") @@ -305,6 +331,9 @@ added to the pack." (file-append (store-database (list profile)) "/db/db.sqlite"))) + (define environment + (singularity-environment-file profile)) + (define build (with-imported-modules (source-module-closure '((guix build utils) @@ -339,6 +368,7 @@ added to the pack." `(,@(map store-info-item (call-with-input-file "profile" read-reference-graph)) + #$environment ,#$output ;; Do not perform duplicate checking because we @@ -379,10 +409,19 @@ added to the pack." target))))))) '#$symlinks) + "-p" "/.singularity.d d 555 0 0" + + ;; Create the environment file. + "-p" "/.singularity.d/env d 555 0 0" + "-p" ,(string-append + "/.singularity.d/env/90-environment.sh s 777 0 0 " + (relative-file-name "/.singularity.d/env" + #$environment)) + ;; Create /.singularity.d/actions, and optionally the 'run' ;; script, used by 'singularity run'. - "-p" "/.singularity.d d 555 0 0" "-p" "/.singularity.d/actions d 555 0 0" + ,@(if entry-point `(;; This one if for Singularity 2.x. "-p" -- 2.22.0 ^ permalink raw reply related [flat|nested] 9+ messages in thread
* [bug#36469] [PATCH 2/2] pack: 'squashfs' backend records the profile's search paths. 2019-07-02 8:56 ` [bug#36469] [PATCH 2/2] pack: 'squashfs' " Ludovic Courtès @ 2019-07-04 11:17 ` Ricardo Wurmus 2019-07-04 16:12 ` [bug#36469] Name of the Singularity/squashfs backend? Ludovic Courtès 0 siblings, 1 reply; 9+ messages in thread From: Ricardo Wurmus @ 2019-07-04 11:17 UTC (permalink / raw) To: Ludovic Courtès; +Cc: 36469, Ludovic Courtès Ludovic Courtès <ludo@gnu.org> writes: > From: Ludovic Courtès <ludovic.courtes@inria.fr> > > * guix/scripts/pack.scm (singularity-environment-file): New procedure. > (squashfs-image): Use it, and create /.singularity/env/90-environment.sh. > * gnu/tests/singularity.scm (run-singularity-test)["singularity run, > with environment"]: New test, currently skipped. > * gnu/tests/singularity.scm (build-tarball&run-singularity-test): Add > GUILE-JSON to the profile. This looks good to me. I think it would be good to separate squashfs-image and singularity going forward as we discussed earlier, but for now mixing the two is fine. Thanks! -- Ricardo ^ permalink raw reply [flat|nested] 9+ messages in thread
* [bug#36469] Name of the Singularity/squashfs backend? 2019-07-04 11:17 ` Ricardo Wurmus @ 2019-07-04 16:12 ` Ludovic Courtès 0 siblings, 0 replies; 9+ messages in thread From: Ludovic Courtès @ 2019-07-04 16:12 UTC (permalink / raw) To: Ricardo Wurmus; +Cc: 36469 Hi! I’ve pushed these two patches, with an extra comment in the Docker test as you suggested. Ricardo Wurmus <rekado@elephly.net> skribis: > I think it would be good to separate squashfs-image and singularity > going forward as we discussed earlier, but for now mixing the two is > fine. So I was on the verge of renaming the ‘squashfs’ backend as a first step, but I wasn’t sure about the name—naming is hard! I thought about ‘singularity’, but that could be confused with SIF: https://github.com/sylabs/sif Then I thought about ‘singularity-squashfs’: it’s unambiguous but a bit long. Another option is the status quo since both ‘--help’ and the manual mention Singularity. WDYT? Ludo’. ^ permalink raw reply [flat|nested] 9+ messages in thread
* [bug#36469] [PATCH 1/2] pack: 'docker' backend records the profile's search paths. 2019-07-02 8:56 ` [bug#36469] [PATCH 1/2] pack: 'docker' backend records the profile's search paths Ludovic Courtès 2019-07-02 8:56 ` [bug#36469] [PATCH 2/2] pack: 'squashfs' " Ludovic Courtès @ 2019-07-04 11:13 ` Ricardo Wurmus 2019-07-04 15:06 ` Ludovic Courtès 1 sibling, 1 reply; 9+ messages in thread From: Ricardo Wurmus @ 2019-07-04 11:13 UTC (permalink / raw) To: Ludovic Courtès; +Cc: 36469 Ludovic Courtès <ludo@gnu.org> writes: > * gnu/tests/docker.scm (run-docker-test)["Load docker image and run it"]: > Add example that expects (json) to be available. This message here is clearer than the actual test. In the test all that JSON stuff is really only there to test that (json) is available, right? Maybe it’s worth adding a comment to the test itself. > diff --git a/guix/scripts/pack.scm b/guix/scripts/pack.scm > index c90b777222..bb6a8cda1a 100644 > --- a/guix/scripts/pack.scm > +++ b/guix/scripts/pack.scm > @@ -27,6 +27,7 @@ > #:use-module (guix utils) > #:use-module (guix store) > #:use-module ((guix status) #:select (with-status-verbosity)) > + #:use-module ((guix self) #:select (make-config.scm)) > #:use-module (guix grafts) > #:autoload (guix inferior) (inferior-package?) > #:use-module (guix monads) > @@ -440,11 +441,24 @@ the image." > (define build > ;; Guile-JSON and Guile-Gcrypt are required by (guix docker). > (with-extensions (list guile-json guile-gcrypt) > - (with-imported-modules (source-module-closure '((guix docker) > - (guix build store-copy)) > - #:select? not-config?) > + (with-imported-modules `(((guix config) => ,(make-config.scm)) > + ,@(source-module-closure > + `((guix docker) > + (guix build store-copy) > + (guix profiles) > + (guix search-paths)) > + #:select? not-config?)) Woah, that “=>” thing is to generate a module dynamically? I hadn’t seen this before. Nifty! The rest of this commit looks good to me. Thanks for working on it! -- Ricardo ^ permalink raw reply [flat|nested] 9+ messages in thread
* [bug#36469] [PATCH 1/2] pack: 'docker' backend records the profile's search paths. 2019-07-04 11:13 ` [bug#36469] [PATCH 1/2] pack: 'docker' backend records the profile's search paths Ricardo Wurmus @ 2019-07-04 15:06 ` Ludovic Courtès 0 siblings, 0 replies; 9+ messages in thread From: Ludovic Courtès @ 2019-07-04 15:06 UTC (permalink / raw) To: Ricardo Wurmus; +Cc: 36469 Ricardo Wurmus <rekado@elephly.net> skribis: > Ludovic Courtès <ludo@gnu.org> writes: > >> * gnu/tests/docker.scm (run-docker-test)["Load docker image and run it"]: >> Add example that expects (json) to be available. > > This message here is clearer than the actual test. In the test all that > JSON stuff is really only there to test that (json) is available, right? Yes, it ensures GUILE_LOAD_PATH is properly set. > Maybe it’s worth adding a comment to the test itself. Looks like it; I’ll add a comment. Thank you! Ludo’. ^ permalink raw reply [flat|nested] 9+ messages in thread
* [bug#36469] [PATCH 0/2] 'guix pack' records environment variables 2019-07-02 8:37 [bug#36469] [PATCH 0/2] 'guix pack' records environment variables Ludovic Courtès 2019-07-02 8:56 ` [bug#36469] [PATCH 1/2] pack: 'docker' backend records the profile's search paths Ludovic Courtès @ 2019-07-02 16:46 ` zimoun 2019-07-04 9:22 ` Ludovic Courtès 1 sibling, 1 reply; 9+ messages in thread From: zimoun @ 2019-07-02 16:46 UTC (permalink / raw) To: Ludovic Courtès; +Cc: 36469 Hi, That's awesome ! I have not tried the patch yet because I am not able to `git pull` (issue with gnu.org NS and the network of my University, whatever!) and my current version seems too old. > The change for Singularity is similar. I’ve tested it with an > instance of Singularity 3.2.0. Unfortunately, we still have 2.x > in Guix and that doesn’t quite support the environment file, so I’ve > chosen to punt on this one. In `docker.scm` you replace %bootstrap-guile by guile-2.2, that's you are talking about? And now both `docker.scm` and `singularity.scm` are consistent to guile-2.2 and there were not. :-) Thank you again for the patch simon ^ permalink raw reply [flat|nested] 9+ messages in thread
* [bug#36469] [PATCH 0/2] 'guix pack' records environment variables 2019-07-02 16:46 ` [bug#36469] [PATCH 0/2] 'guix pack' records environment variables zimoun @ 2019-07-04 9:22 ` Ludovic Courtès 0 siblings, 0 replies; 9+ messages in thread From: Ludovic Courtès @ 2019-07-04 9:22 UTC (permalink / raw) To: zimoun; +Cc: 36469 Hi, zimoun <zimon.toutoune@gmail.com> skribis: > That's awesome ! Glad you like it! >> The change for Singularity is similar. I’ve tested it with an >> instance of Singularity 3.2.0. Unfortunately, we still have 2.x >> in Guix and that doesn’t quite support the environment file, so I’ve >> chosen to punt on this one. > > In `docker.scm` you replace %bootstrap-guile by guile-2.2, that's you > are talking about? No no, I was talking about Singularity 2.x, not Guile 2.x. :-) Thanks for your feedback, Ludo’. ^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2019-07-04 16:14 UTC | newest] Thread overview: 9+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2019-07-02 8:37 [bug#36469] [PATCH 0/2] 'guix pack' records environment variables Ludovic Courtès 2019-07-02 8:56 ` [bug#36469] [PATCH 1/2] pack: 'docker' backend records the profile's search paths Ludovic Courtès 2019-07-02 8:56 ` [bug#36469] [PATCH 2/2] pack: 'squashfs' " Ludovic Courtès 2019-07-04 11:17 ` Ricardo Wurmus 2019-07-04 16:12 ` [bug#36469] Name of the Singularity/squashfs backend? Ludovic Courtès 2019-07-04 11:13 ` [bug#36469] [PATCH 1/2] pack: 'docker' backend records the profile's search paths Ricardo Wurmus 2019-07-04 15:06 ` Ludovic Courtès 2019-07-02 16:46 ` [bug#36469] [PATCH 0/2] 'guix pack' records environment variables zimoun 2019-07-04 9:22 ` Ludovic Courtès
Code repositories for project(s) associated with this public inbox https://git.savannah.gnu.org/cgit/guix.git This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).