Creating a docker image with Guix

Creating a docker image with Guix

[Ricardo Wurmus]

Hi Guix,

I just played around with Docker and built up a command to create a
Docker image for Emacs.

Can anyone find a more elegant way to do this?

--8<---------------cut here---------------start------------->8---
guix environment --ad-hoc \
    coreutils bash emacs-no-x-toolkit -- \
    sh -c 'tar -c $(guix gc --requisites $GUIX_ENVIRONMENT) | \
           docker import -c "ONBUILD RUN [\"$GUIX_ENVIRONMENT/bin/ln\", \"-s\", \"$GUIX_ENVIRONMENT/bin\", \"/bin\"]" - emacs-base' \
           && echo -e "FROM emacs-base\nCMD [\"/bin/emacs\"]" | \
           docker build -
--8<---------------cut here---------------end--------------->8---

What happens here is this:

* using “guix environment” build a temporary profile containing
  coreutils (for “ln”), bash (for “sh”), and emacs

* inside the environment $GUIX_ENVIRONMENT points to the profile; take
  all of the requisites for this profile (“guix gc --requisites”) and
  send them to “docker import” using a tar pipe

* modify the base image such that it links the profile’s “bin” directory
  to “/bin” when a derivative image is built

* name that base image “emacs-base”

* then build a derivative of this base image with “docker build” (this
  causes “ln -s” to be executed) and make it run the command
  “/bin/emacs” by default

The result is an unnamed image (“docker build” reports an image id such
as “ae8d6281b56f”) that can be run interactively like this:

    docker run -it --rm ae8d6281b56f

What do you think?  Gross?

~~ Ricardo

Re: Creating a docker image with Guix

[David Craven]

> What do you think?  Gross?

Cool!

Re: Creating a docker image with Guix

[Hartmut Goebel]

Hi Ricardo,

this is really cool! Creating a docker image without even using the
"scratch" image.

> Can anyone find a more elegant way to do this?

I'd suggest to write a small shell-script which dumps the part after "sh
-c" into a temporary file. This makes this part more readable. The
script would then be easy to extent to accept arguments, esp. the list
of packages to install. Maybe you could use some other shell-tricks to
avoid the temporary file.

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |

Re: Creating a docker image with Guix

[Pjotr Prins]

On Mon, Jan 02, 2017 at 01:54:30PM +0100, Ricardo Wurmus wrote:
> I just played around with Docker and built up a command to create a
> Docker image for Emacs.
> 
> Can anyone find a more elegant way to do this?

Not really - you always end up with something like this. We have run
the guix daemon, but that requires docker with privileges. 

Pj.

Re: Creating a docker image with Guix

[Ricardo Wurmus]

Pjotr Prins <pjotr.public12@thebird.nl> writes:

> On Mon, Jan 02, 2017 at 01:54:30PM +0100, Ricardo Wurmus wrote:
>> I just played around with Docker and built up a command to create a
>> Docker image for Emacs.
>> 
>> Can anyone find a more elegant way to do this?
>
> Not really - you always end up with something like this. We have run
> the guix daemon, but that requires docker with privileges. 

I see three options:

* build a base image like I did here
* use “FROM scratch” and ADD a closure.tar that contains everything
  “guix gc --requisites” reports (very slow)
* use guix-daemon inside the docker image, but that requires special
  privileges at container runtime

~~

Re: Creating a docker image with Guix

[John Darrington]

[-- Attachment #1: Type: text/plain, Size: 566 bytes --]

On Mon, Jan 02, 2017 at 01:54:30PM +0100, Ricardo Wurmus wrote:
     Hi Guix,
     
     I just played around with Docker and built up a command to create a
     Docker image for Emacs.
     
     Can anyone find a more elegant way to do this?
     
"Docker" and "elegant" are two concepts which are mutually exclusive.

J'

-- 
Avoid eavesdropping.  Send strong encrypted email.
PGP Public key ID: 1024D/2DE827B3 
fingerprint = 8797 A26D 0854 2EAB 0285  A290 8A67 719C 2DE8 27B3
See http://sks-keyservers.net or any PGP keyserver for public key.


[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: Creating a docker image with Guix

[Ludovic Courtès]

Hello!

Ricardo Wurmus <ricardo.wurmus@mdc-berlin.de> skribis:

> I just played around with Docker and built up a command to create a
> Docker image for Emacs.

Fun!

> Can anyone find a more elegant way to do this?
>
> guix environment --ad-hoc \
>     coreutils bash emacs-no-x-toolkit -- \
>     sh -c 'tar -c $(guix gc --requisites $GUIX_ENVIRONMENT) | \
>            docker import -c "ONBUILD RUN [\"$GUIX_ENVIRONMENT/bin/ln\", \"-s\", \"$GUIX_ENVIRONMENT/bin\", \"/bin\"]" - emacs-base' \
>            && echo -e "FROM emacs-base\nCMD [\"/bin/emacs\"]" | \
>            docker build -

What does the resulting image look like?

Would it be enough to generate an “Image JSON Description” in this
format: <https://github.com/docker/docker/blob/master/image/spec/v1.md>?

I’m not familiar enough with Docker but I’m under the impression that we
should be able to generate an image without even using Docker.  :-)

Thanks for sharing the neat hack!

Ludo’.

Re: Creating a docker image with Guix

[Ricardo Wurmus]

Ludovic Courtès <ludo@gnu.org> writes:

> Ricardo Wurmus <ricardo.wurmus@mdc-berlin.de> skribis:
>
>> I just played around with Docker and built up a command to create a
>> Docker image for Emacs.
>
> Fun!
>
>> Can anyone find a more elegant way to do this?
>>
>> guix environment --ad-hoc \
>>     coreutils bash emacs-no-x-toolkit -- \
>>     sh -c 'tar -c $(guix gc --requisites $GUIX_ENVIRONMENT) | \
>>            docker import -c "ONBUILD RUN [\"$GUIX_ENVIRONMENT/bin/ln\", \"-s\", \"$GUIX_ENVIRONMENT/bin\", \"/bin\"]" - emacs-base' \
>>            && echo -e "FROM emacs-base\nCMD [\"/bin/emacs\"]" | \
>>            docker build -
>
> What does the resulting image look like?

It’s very hard to tell because this command doesn’t spit out a file that
fully represents the image.  Instead it modifies some state in
/var/lib/docker.

> Would it be enough to generate an “Image JSON Description” in this
> format: <https://github.com/docker/docker/blob/master/image/spec/v1.md>?

No.  On Fedora (where I ran the above command) Docker uses the
devicemapper backend.  “docker build” appears to have created an XFS
file system in a file.  Different installations use different backends
(e.g. direct-lvm or aufs) — this probably means that we should not try
to build an image for a particular backend directly.

However, there is a “docker save” command to export an image as a tar
archive (which can be imported by a docker instance), and the format of
the resulting archive looks pretty simple:  It contains a couple of JSON
files and two “layer.tar” archives.  The first contains the files of the
base image (i.e. all the /gnu/store stuff), the second contains the
added symlink from “/bin” to “/gnu/store/.../bin” (and “/run/secrets”).

> I’m not familiar enough with Docker but I’m under the impression that we
> should be able to generate an image without even using Docker.  :-)

Absolutely!  Now that I know about “docker save” I’ll give it another
try.

~~

[PATCH] Creating a docker image with Guix

[Ricardo Wurmus]

[-- Attachment #1: Type: text/plain, Size: 1111 bytes --]


Ludovic Courtès <ludo@gnu.org> writes:

> I’m not familiar enough with Docker but I’m under the impression that we
> should be able to generate an image without even using Docker.  :-)

The attached patch adds a Docker export feature, so you can do this:

    docker load < \
      $(guix archive --export-docker-image=$(readlink -f ~/.guix-profile))

Then you can use “docker images” to show the available images.  For some
reason Docker won’t show the name and tag “guix archive” generates, so
just take the most recently added image.  Then run it, e.g. like this:

    docker run --rm -ti d11111472905 /bin/emacs

This starts the container and runs “/bin/emacs” interactively.  During
export “guix archive” also links the item’s “./bin” directory to “/bin”,
so users can run commands without having to know the long store path.

I used it successfully to build an Emacs Docker image like this:

    guix environment --ad-hoc coreutils bash emacs-no-x-toolkit
    docker load < $(guix archive --export-docker-image=$GUIX_ENVIRONMENT)

~~ Ricardo


[-- Attachment #2: 0001-guix-Add-Docker-image-export.patch --]
[-- Type: text/x-patch, Size: 8522 bytes --]

From d600db91078f28d82324671e3d43acaddc9b9608 Mon Sep 17 00:00:00 2001
From: Ricardo Wurmus <ricardo.wurmus@mdc-berlin.de>
Date: Tue, 3 Jan 2017 16:20:15 +0100
Subject: [PATCH] guix: Add Docker image export.

* guix/docker.scm: New file.
* Makefile.am (MODULES): Register it.
* guix/scripts/archive.scm (show-help, %options, guix-archive): Add
support for "--export-docker-image".
* doc/guix.texi (Invoking guix archive): Document it.
---
 Makefile.am              |   1 +
 doc/guix.texi            |   6 +++
 guix/docker.scm          | 120 +++++++++++++++++++++++++++++++++++++++++++++++
 guix/scripts/archive.scm |  11 +++++
 4 files changed, 138 insertions(+)
 create mode 100644 guix/docker.scm

diff --git a/Makefile.am b/Makefile.am
index fb08a004b..4317b83a2 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -158,6 +158,7 @@ MODULES =					\
 if HAVE_GUILE_JSON
 
 MODULES +=					\
+  guix/docker.scm	   			\
   guix/import/github.scm   			\
   guix/import/json.scm				\
   guix/import/crate.scm				\
diff --git a/doc/guix.texi b/doc/guix.texi
index 8c65f44da..1dd501261 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -2438,6 +2438,12 @@ Read a list of store file names from the standard input, one per line,
 and write on the standard output the subset of these files missing from
 the store.
 
+@item --export-docker-image=@var{directory}
+@cindex docker, export
+Recursively export the specified store directory as a Docker image in
+tar archive format.  The generated archive can be loaded by Docker using
+@command{docker load}.
+
 @item --generate-key[=@var{parameters}]
 @cindex signing, archives
 Generate a new key pair for the daemon.  This is a prerequisite before
diff --git a/guix/docker.scm b/guix/docker.scm
new file mode 100644
index 000000000..0f6c3bf90
--- /dev/null
+++ b/guix/docker.scm
@@ -0,0 +1,120 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2017 Ricardo Wurmus <rekado@elephly.net>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (guix docker)
+  #:use-module (guix hash)
+  #:use-module (guix store)
+  #:use-module ((guix build utils)
+                #:select (delete-file-recursively
+                          with-directory-excursion))
+  #:use-module (json)
+  #:use-module (rnrs bytevectors)
+  #:export (build-docker-image))
+
+(define (hexencode bv)
+  "Return the hexadecimal representation of the bytevector BV."
+  (format #f "~{~2,'0x~}" (bytevector->u8-list bv)))
+
+(define (docker-id path)
+  "Generate a 256-bit identifier in hexadecimal encoding for the Docker image
+containing the closure at PATH."
+  (hexencode (sha256 (string->utf8 path))))
+
+(define (layer-diff-id layer)
+  "Generate a layer DiffID for the given LAYER archive."
+  (string-append "sha256:" (hexencode (file-sha256 layer))))
+
+(define spec-version "1.0")
+
+(define (image-description id time)
+  "Generate a simple image description."
+  `((id . ,id)
+    (created . ,time)
+    (container_config . #nil)))
+
+(define (manifest id)
+  "Generate a simple image manifest."
+  `(((Config . "config.json")
+     (RepoTags . #nil)
+     (Layers . (,(string-append id "/layer.tar"))))))
+
+(define (repositories path id)
+  "Generate a repositories file referencing PATH and the image ID."
+  `((,(basename path) . ((latest . ,id)))))
+
+;; See https://github.com/opencontainers/image-spec/blob/master/config.md
+(define (config layer time)
+  "Generate a minimal image configuratio for the given LAYER file."
+  `((architecture . "amd64")
+    (comment . "Generated by GNU Guix")
+    (created . ,time)
+    (config . #nil)
+    (container_config . #nil)
+    (os . "linux")
+    (rootfs . ((type . "layers")
+               (diff_ids . (,(layer-diff-id layer)))))))
+
+;; TODO: heroically copied from guix/script/pull.scm
+(define (temporary-directory)
+  "Make a temporary directory and return its name."
+  (let ((name (tmpnam)))
+    (mkdir name)
+    (chmod name #o700)
+    name))
+
+(define (build-docker-image path)
+  "Generate a Docker image archive from the given store PATH.  The image
+contains the closure of the given store item."
+  (let ((id (docker-id path))
+        (directory (temporary-directory))
+        (time (strftime "%FT%TZ" (localtime (current-time)))))
+    (with-directory-excursion directory
+
+      ;; Add symlink from /bin to /gnu/store/.../bin
+      (symlink (string-append path "/bin") "bin")
+
+      (mkdir id)
+      (with-directory-excursion id
+        (with-output-to-file "VERSION"
+          (lambda () (display spec-version)))
+        (with-output-to-file "json"
+          (lambda () (scm->json (image-description id time))))
+
+        ;; Wrap it up
+        (let ((items (with-store store
+                       (requisites store (list path)))))
+          (and (zero? (apply system* "tar" "-cf" "layer.tar"
+                             (cons "../bin" items)))
+               (delete-file "../bin"))))
+
+      (with-output-to-file "config.json"
+        (lambda ()
+          (scm->json (config (string-append id "/layer.tar") time))))
+
+      (with-output-to-file "manifest.json"
+        (lambda ()
+          (scm->json (manifest id))))
+      (with-output-to-file "repositories"
+        (lambda ()
+          (scm->json (repositories path id)))))
+
+    (let ((name (string-append (getcwd)
+                               "/docker-image-" (basename path) ".tar")))
+      (and (zero? (system* "tar" "-C" directory "-cf" name "."))
+           (delete-file-recursively directory)
+           name))))
diff --git a/guix/scripts/archive.scm b/guix/scripts/archive.scm
index 7e432351e..ffa6e9f44 100644
--- a/guix/scripts/archive.scm
+++ b/guix/scripts/archive.scm
@@ -1,5 +1,6 @@
 ;;; GNU Guix --- Functional package management for GNU
 ;;; Copyright © 2013, 2014, 2015, 2016 Ludovic Courtès <ludo@gnu.org>
+;;; Copyright © 2017 Ricardo Wurmus <rekado@elephly.net>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -30,6 +31,7 @@
   #:use-module (guix ui)
   #:use-module (guix pki)
   #:use-module (guix pk-crypto)
+  #:use-module (guix docker)
   #:use-module (guix scripts)
   #:use-module (guix scripts build)
   #:use-module (gnu packages)
@@ -63,6 +65,9 @@ Export/import one or more packages from/to the store.\n"))
   (display (_ "
       --export           export the specified files/packages to stdout"))
   (display (_ "
+      --export-docker-image=DIR
+                         export the specified store item DIR as a Docker image"))
+  (display (_ "
   -r, --recursive        combined with '--export', include dependencies"))
   (display (_ "
       --import           import from the archive passed on stdin"))
@@ -117,6 +122,9 @@ Export/import one or more packages from/to the store.\n"))
          (option '("export") #f #f
                  (lambda (opt name arg result)
                    (alist-cons 'export #t result)))
+         (option '("export-docker-image") #t #f
+                 (lambda (opt name arg result . rest)
+                   (alist-cons 'export-docker-image arg result)))
          (option '(#\r "recursive") #f #f
                  (lambda (opt name arg result)
                    (alist-cons 'export-recursive? #t result)))
@@ -328,6 +336,9 @@ the input port."
                  generate-key-pair)
                 ((assoc-ref opts 'authorize)
                  (authorize-key))
+                ((assoc-ref opts 'export-docker-image)
+                 => (lambda (item)
+                      (format #t "~a\n" (build-docker-image item))))
                 (else
                  (with-store store
                    (cond ((assoc-ref opts 'export)
-- 
2.11.0

Re: [PATCH] Creating a docker image with Guix

[Chris Marusich]

[-- Attachment #1: Type: text/plain, Size: 1111 bytes --]

Ricardo Wurmus <ricardo.wurmus@mdc-berlin.de> writes:

> Ludovic Courtès <ludo@gnu.org> writes:
>
>> I’m not familiar enough with Docker but I’m under the impression that we
>> should be able to generate an image without even using Docker.  :-)
>
> The attached patch adds a Docker export feature, so you can do this:
>
>     docker load < \
>       $(guix archive --export-docker-image=$(readlink -f ~/.guix-profile))
>
> [...]
>
> I used it successfully to build an Emacs Docker image like this:
>
>     guix environment --ad-hoc coreutils bash emacs-no-x-toolkit
>     docker load < $(guix archive --export-docker-image=$GUIX_ENVIRONMENT)

Wow!  That's pretty cool.  If it's this easy to make a Docker image, I'm
sure it'll be a good incentive for Docker users to try out Guix!

What happens if I invoke this command while GC is trying to delete the
specified path?

> +;; See https://github.com/opencontainers/image-spec/blob/master/config.md
> +(define (config layer time)
> +  "Generate a minimal image configuratio for the given LAYER file."

Minor typo.

-- 
Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: [PATCH] Creating a docker image with Guix

[Ricardo Wurmus]

Chris Marusich <cmmarusich@gmail.com> writes:

> What happens if I invoke this command while GC is trying to delete the
> specified path?

It will do weird things, I guess.  It’s best to do this inside of “guix
environment” as I demonstrated because while you’re in the environment
the temporary profile is protected from GC.

>> +;; See https://github.com/opencontainers/image-spec/blob/master/config.md
>> +(define (config layer time)
>> +  "Generate a minimal image configuratio for the given LAYER file."
>
> Minor typo.

Oops!  Thanks.

-- 
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
http://elephly.net

Re: [PATCH] Creating a docker image with Guix

[Ludovic Courtès]

Ricardo Wurmus <rekado@elephly.net> skribis:

> Chris Marusich <cmmarusich@gmail.com> writes:
>
>> What happens if I invoke this command while GC is trying to delete the
>> specified path?
>
> It will do weird things, I guess.  It’s best to do this inside of “guix
> environment” as I demonstrated because while you’re in the environment
> the temporary profile is protected from GC.

But if you do “guix archive --export-docker-image=foo emacs”, then Emacs
is protected from GC for the dynamic extend of ‘with-store’ in
‘guix-archive’.

Also, in practice, the ‘list-runtime-roots’ helper will notice the file
name in the arguments to ‘guix archive’ and will make it a GC root.

So that does not introduce any difference compared to the existing
behavior, I think.

Ludo’.

Re: [PATCH] Creating a docker image with Guix

[Ludovic Courtès]

Howdy!

Ricardo Wurmus <ricardo.wurmus@mdc-berlin.de> skribis:

> Ludovic Courtès <ludo@gnu.org> writes:
>
>> I’m not familiar enough with Docker but I’m under the impression that we
>> should be able to generate an image without even using Docker.  :-)
>
> The attached patch adds a Docker export feature, so you can do this:
>
>     docker load < \
>       $(guix archive --export-docker-image=$(readlink -f ~/.guix-profile))

Woow, that was fast!

> Then you can use “docker images” to show the available images.  For some
> reason Docker won’t show the name and tag “guix archive” generates, so
> just take the most recently added image.  Then run it, e.g. like this:
>
>     docker run --rm -ti d11111472905 /bin/emacs
>
> This starts the container and runs “/bin/emacs” interactively.  During
> export “guix archive” also links the item’s “./bin” directory to “/bin”,
> so users can run commands without having to know the long store path.

Awesome!

> I used it successfully to build an Emacs Docker image like this:
>
>     guix environment --ad-hoc coreutils bash emacs-no-x-toolkit
>     docker load < $(guix archive --export-docker-image=$GUIX_ENVIRONMENT)

Be sure to let emacs-devel know.  ;-)

> From d600db91078f28d82324671e3d43acaddc9b9608 Mon Sep 17 00:00:00 2001
> From: Ricardo Wurmus <ricardo.wurmus@mdc-berlin.de>
> Date: Tue, 3 Jan 2017 16:20:15 +0100
> Subject: [PATCH] guix: Add Docker image export.
>
> * guix/docker.scm: New file.
> * Makefile.am (MODULES): Register it.
> * guix/scripts/archive.scm (show-help, %options, guix-archive): Add
> support for "--export-docker-image".
> * doc/guix.texi (Invoking guix archive): Document it.

Looks great!

> --- a/doc/guix.texi
> +++ b/doc/guix.texi
> @@ -2438,6 +2438,12 @@ Read a list of store file names from the standard input, one per line,
>  and write on the standard output the subset of these files missing from
>  the store.
>  
> +@item --export-docker-image=@var{directory}
> +@cindex docker, export
> +Recursively export the specified store directory as a Docker image in
> +tar archive format.  The generated archive can be loaded by Docker using
> +@command{docker load}.

Maybe “as a Docker image in tar archive format, as specified in
@uref{http://…, version 1.0 of the Foo Bar Spec}.”

I would be in favor of --format=FMT, where FMT can be one of “nar” or
“docker”.  Maybe there’ll be others in the future.  WDYT?

The paragraph that says “Archives are stored in the “normalized archive”
or “nar” format,“ should be updated.

Also, it seems that ‘-f docker’ would always imply ’-r’, right?  That’s
reasonable but would be worth mentioning.

> +(define (hexencode bv)
> +  "Return the hexadecimal representation of the bytevector BV."
> +  (format #f "~{~2,'0x~}" (bytevector->u8-list bv)))

Maybe use ‘bytevector->base16-string’ from (guix utils) instead.

> +(define spec-version "1.0")

Please add the URL to said spec as a comment.

> +;; TODO: heroically copied from guix/script/pull.scm
> +(define (temporary-directory)

Alternatively, there’s ‘call-with-temporary-directory’ in (guix utils).
:-)

> +          (and (zero? (apply system* "tar" "-cf" "layer.tar"
> +                             (cons "../bin" items)))
> +               (delete-file "../bin"))))

This reminds me we should steal this code of Mark’s sometime:

  https://github.com/spk121/guile100/blob/master/code/tar2.scm

Thank you!

Ludo’.

Re: [PATCH] Creating a docker image with Guix

[Ricardo Wurmus]

[-- Attachment #1: Type: text/plain, Size: 2260 bytes --]


Ludovic Courtès <ludo@gnu.org> writes:

>> --- a/doc/guix.texi
>> +++ b/doc/guix.texi
>> @@ -2438,6 +2438,12 @@ Read a list of store file names from the standard input, one per line,
>>  and write on the standard output the subset of these files missing from
>>  the store.
>>  
>> +@item --export-docker-image=@var{directory}
>> +@cindex docker, export
>> +Recursively export the specified store directory as a Docker image in
>> +tar archive format.  The generated archive can be loaded by Docker using
>> +@command{docker load}.
>
> Maybe “as a Docker image in tar archive format, as specified in
> @uref{http://…, version 1.0 of the Foo Bar Spec}.”

Okay.

> I would be in favor of --format=FMT, where FMT can be one of “nar” or
> “docker”.  Maybe there’ll be others in the future.  WDYT?

Sounds good.

> The paragraph that says “Archives are stored in the “normalized archive”
> or “nar” format,“ should be updated.
>
> Also, it seems that ‘-f docker’ would always imply ’-r’, right?  That’s
> reasonable but would be worth mentioning.

Okay.

>> +(define (hexencode bv)
>> +  "Return the hexadecimal representation of the bytevector BV."
>> +  (format #f "~{~2,'0x~}" (bytevector->u8-list bv)))
>
> Maybe use ‘bytevector->base16-string’ from (guix utils) instead.

Ah, didn’t know about this one.

>> +(define spec-version "1.0")
>
> Please add the URL to said spec as a comment.

I added a clarifying comment (because confusingly this is NOT the
version of the Docker image spec).

>> +;; TODO: heroically copied from guix/script/pull.scm
>> +(define (temporary-directory)
>
> Alternatively, there’s ‘call-with-temporary-directory’ in (guix utils).
> :-)

Neat!

>> +          (and (zero? (apply system* "tar" "-cf" "layer.tar"
>> +                             (cons "../bin" items)))
>> +               (delete-file "../bin"))))
>
> This reminds me we should steal this code of Mark’s sometime:
>
>   https://github.com/spk121/guile100/blob/master/code/tar2.scm

Yes, this would be nice.

Attached is a new patch with all requested changes and a couple of fixes
(generated images now have proper names and tags).

~~ Ricardo


[-- Attachment #2: 0001-guix-Add-Docker-image-export.patch --]
[-- Type: text/x-patch, Size: 10564 bytes --]

From fefd4f02d003dd35bd0ab459ec2ccc9f9ad62ffa Mon Sep 17 00:00:00 2001
From: Ricardo Wurmus <ricardo.wurmus@mdc-berlin.de>
Date: Tue, 3 Jan 2017 16:20:15 +0100
Subject: [PATCH] guix: Add Docker image export.

* guix/docker.scm: New file.
* Makefile.am (MODULES): Register it.
* guix/scripts/archive.scm (show-help, %options, guix-archive): Add
support for "--format".
* doc/guix.texi (Invoking guix archive): Document it.
---
 Makefile.am              |   1 +
 doc/guix.texi            |  18 +++++++-
 guix/docker.scm          | 117 +++++++++++++++++++++++++++++++++++++++++++++++
 guix/scripts/archive.scm |  14 +++++-
 4 files changed, 148 insertions(+), 2 deletions(-)
 create mode 100644 guix/docker.scm

diff --git a/Makefile.am b/Makefile.am
index fb08a004b..4317b83a2 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -158,6 +158,7 @@ MODULES =					\
 if HAVE_GUILE_JSON
 
 MODULES +=					\
+  guix/docker.scm	   			\
   guix/import/github.scm   			\
   guix/import/json.scm				\
   guix/import/crate.scm				\
diff --git a/doc/guix.texi b/doc/guix.texi
index 3a9ebe8a6..93a56a2b0 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -2394,7 +2394,7 @@ what you should use in this case (@pxref{Invoking guix copy}).
 
 @cindex nar, archive format
 @cindex normalized archive (nar)
-Archives are stored in the ``normalized archive'' or ``nar'' format, which is
+By default archives are stored in the ``normalized archive'' or ``nar'' format, which is
 comparable in spirit to `tar', but with differences
 that make it more appropriate for our purposes.  First, rather than
 recording all Unix metadata for each file, the nar format only mentions
@@ -2410,6 +2410,9 @@ verifies the signature and rejects the import in case of an invalid
 signature or if the signing key is not authorized.
 @c FIXME: Add xref to daemon doc about signatures.
 
+Optionally, archives can be exported as a Docker image in the tar
+archive format using @code{--format=docker}.
+
 The main options are:
 
 @table @code
@@ -2438,6 +2441,19 @@ Read a list of store file names from the standard input, one per line,
 and write on the standard output the subset of these files missing from
 the store.
 
+@item -f
+@item --format=@var{FMT}
+@cindex docker, export
+@cindex export format
+Specify the export format.  Acceptable arguments are @code{nar} and
+@code{docker}.  The default is the nar format.  When the format is
+@code{docker}, recursively export the specified store directory as a
+Docker image in tar archive format, as specified in
+@uref{https://github.com/docker/docker/blob/master/image/spec/v1.2.md,
+version 1.2.0 of the Docker Image Specification}.  Using
+@code{--format=docker} implies @code{--recursive}.  The generated
+archive can be loaded by Docker using @command{docker load}.
+
 @item --generate-key[=@var{parameters}]
 @cindex signing, archives
 Generate a new key pair for the daemon.  This is a prerequisite before
diff --git a/guix/docker.scm b/guix/docker.scm
new file mode 100644
index 000000000..0cc0f2af9
--- /dev/null
+++ b/guix/docker.scm
@@ -0,0 +1,117 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2017 Ricardo Wurmus <rekado@elephly.net>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (guix docker)
+  #:use-module (guix hash)
+  #:use-module (guix store)
+  #:use-module (guix utils)
+  #:use-module ((guix build utils)
+                #:select (delete-file-recursively
+                          with-directory-excursion))
+  #:use-module (json)
+  #:use-module (rnrs bytevectors)
+  #:use-module (ice-9 match)
+  #:export (build-docker-image))
+
+;; Generate a 256-bit identifier in hexadecimal encoding for the Docker image
+;; containing the closure at PATH.
+(define docker-id
+  (compose bytevector->base16-string sha256 string->utf8))
+
+(define (layer-diff-id layer)
+  "Generate a layer DiffID for the given LAYER archive."
+  (string-append "sha256:" (bytevector->base16-string (file-sha256 layer))))
+
+;; This is the semantic version of the JSON metadata schema according to
+;; https://github.com/docker/docker/blob/master/image/spec/v1.2.md
+;; It is NOT the version of the image specification.
+(define schema-version "1.0")
+
+(define (image-description id time)
+  "Generate a simple image description."
+  `((id . ,id)
+    (created . ,time)
+    (container_config . #nil)))
+
+(define (generate-tag path)
+  "Generate an image tag for the given PATH."
+  (match (string-split (basename path) #\-)
+    ((hash name . rest) (string-append name ":" hash))))
+
+(define (manifest path id)
+  "Generate a simple image manifest."
+  `(((Config . "config.json")
+     (RepoTags . (,(generate-tag path)))
+     (Layers . (,(string-append id "/layer.tar"))))))
+
+;; According to the specifications this is required for backwards
+;; compatibility.  It duplicates information provided by the manifest.
+(define (repositories path id)
+  "Generate a repositories file referencing PATH and the image ID."
+  `((,(generate-tag path) . ((latest . ,id)))))
+
+;; See https://github.com/opencontainers/image-spec/blob/master/config.md
+(define (config layer time)
+  "Generate a minimal image configuratio for the given LAYER file."
+  `((architecture . "amd64")
+    (comment . "Generated by GNU Guix")
+    (created . ,time)
+    (config . #nil)
+    (container_config . #nil)
+    (os . "linux")
+    (rootfs . ((type . "layers")
+               (diff_ids . (,(layer-diff-id layer)))))))
+
+(define (build-docker-image path)
+  "Generate a Docker image archive from the given store PATH.  The image
+contains the closure of the given store item."
+  (let ((id (docker-id path))
+        (time (strftime "%FT%TZ" (localtime (current-time))))
+        (name (string-append (getcwd)
+                             "/docker-image-" (basename path) ".tar")))
+    (and (call-with-temporary-directory
+          (lambda (directory)
+            (with-directory-excursion directory
+              ;; Add symlink from /bin to /gnu/store/.../bin
+              (symlink (string-append path "/bin") "bin")
+
+              (mkdir id)
+              (with-directory-excursion id
+                (with-output-to-file "VERSION"
+                  (lambda () (display schema-version)))
+                (with-output-to-file "json"
+                  (lambda () (scm->json (image-description id time))))
+
+                ;; Wrap it up
+                (let ((items (with-store store
+                               (requisites store (list path)))))
+                  (and (zero? (apply system* "tar" "-cf" "layer.tar"
+                                     (cons "../bin" items)))
+                       (delete-file "../bin"))))
+
+              (with-output-to-file "config.json"
+                (lambda ()
+                  (scm->json (config (string-append id "/layer.tar") time))))
+              (with-output-to-file "manifest.json"
+                (lambda ()
+                  (scm->json (manifest path id))))
+              (with-output-to-file "repositories"
+                (lambda ()
+                  (scm->json (repositories path id)))))
+            (zero? (system* "tar" "-C" directory "-cf" name "."))))
+         name)))
diff --git a/guix/scripts/archive.scm b/guix/scripts/archive.scm
index 7e432351e..8ae233cd1 100644
--- a/guix/scripts/archive.scm
+++ b/guix/scripts/archive.scm
@@ -1,5 +1,6 @@
 ;;; GNU Guix --- Functional package management for GNU
 ;;; Copyright © 2013, 2014, 2015, 2016 Ludovic Courtès <ludo@gnu.org>
+;;; Copyright © 2017 Ricardo Wurmus <rekado@elephly.net>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -30,6 +31,7 @@
   #:use-module (guix ui)
   #:use-module (guix pki)
   #:use-module (guix pk-crypto)
+  #:use-module (guix docker)
   #:use-module (guix scripts)
   #:use-module (guix scripts build)
   #:use-module (gnu packages)
@@ -63,6 +65,8 @@ Export/import one or more packages from/to the store.\n"))
   (display (_ "
       --export           export the specified files/packages to stdout"))
   (display (_ "
+      --format=FMT       export files/packages in the specified format FMT"))
+  (display (_ "
   -r, --recursive        combined with '--export', include dependencies"))
   (display (_ "
       --import           import from the archive passed on stdin"))
@@ -117,6 +121,9 @@ Export/import one or more packages from/to the store.\n"))
          (option '("export") #f #f
                  (lambda (opt name arg result)
                    (alist-cons 'export #t result)))
+         (option '(#\f "format") #t #f
+                 (lambda (opt name arg result . rest)
+                   (alist-cons 'format arg result)))
          (option '(#\r "recursive") #f #f
                  (lambda (opt name arg result)
                    (alist-cons 'export-recursive? #t result)))
@@ -331,7 +338,12 @@ the input port."
                 (else
                  (with-store store
                    (cond ((assoc-ref opts 'export)
-                          (export-from-store store opts))
+                          (cond ((equal? (assoc-ref opts 'format) "docker")
+                                 (match (car opts)
+                                   (('argument . (? store-path? item))
+                                    (format #t "~a\n" (build-docker-image item)))
+                                   (_ (leave (_ "argument must be a direct store path~%")))))
+                                (_ (export-from-store store opts))))
                          ((assoc-ref opts 'import)
                           (import-paths store (current-input-port)))
                          ((assoc-ref opts 'missing)
-- 
2.11.0

Re: [PATCH] Creating a docker image with Guix

[Thompson, David]

Hi Ricardo,

This is awesome! Just a couple things:

1) The amd64 architecture is hardcoded in the image.  Should this
instead be set to the current architecture?
2) s/configuratio/configuration/ in the docstring for 'config' in
guix/docker.scm

Ludovic, Ricardo, anyone else: It might be nice if in the future 'guix
environment' had a flag to output the resulting profile instead of
having to spawn the sub-shell and refer to $GUIX_ENVIRONMENT.  Food
for thought. :)

LGTM (Let's Get This Merged)!

- Dave

Re: [PATCH] Creating a docker image with Guix

[Ludovic Courtès]

Hey!

Ricardo Wurmus <ricardo.wurmus@mdc-berlin.de> skribis:

> From fefd4f02d003dd35bd0ab459ec2ccc9f9ad62ffa Mon Sep 17 00:00:00 2001
> From: Ricardo Wurmus <ricardo.wurmus@mdc-berlin.de>
> Date: Tue, 3 Jan 2017 16:20:15 +0100
> Subject: [PATCH] guix: Add Docker image export.
>
> * guix/docker.scm: New file.
> * Makefile.am (MODULES): Register it.
> * guix/scripts/archive.scm (show-help, %options, guix-archive): Add
> support for "--format".
> * doc/guix.texi (Invoking guix archive): Document it.

I agree with the changes David suggested.

Go for it!

As a bonus we could test it in a shell script when the ‘docker’ command
is available, and skip it otherwise.

As a second bonus, Someone could write a blog entry for the web site
explaining how Guix makes it super easy to create Docker images.  :-)

Thank you!

Ludo’.

[PATCH] guix archive: '-f docker' supports package names as arguments.

[Ludovic Courtès]

This allows users to type:

  guix archive -f docker emacs

as was already the case for the 'nar' format.

Reported by David Thompson.

* guix/scripts/archive.scm (%default-options): Add 'format'.
(export-from-store): Dispatch based on the 'format' key in OPTS.
(guix-archive): Call 'export-from-store' in all cases when the 'export'
key is in OPTS.
---
 guix/scripts/archive.scm | 30 ++++++++++++++++++------------
 1 file changed, 18 insertions(+), 12 deletions(-)

diff --git a/guix/scripts/archive.scm b/guix/scripts/archive.scm
index 6eba9e000..3e056fda9 100644
--- a/guix/scripts/archive.scm
+++ b/guix/scripts/archive.scm
@@ -53,7 +53,8 @@
 
 (define %default-options
   ;; Alist of default option values.
-  `((system . ,(%current-system))
+  `((format . "nar")
+    (system . ,(%current-system))
     (substitutes? . #t)
     (graft? . #t)
     (max-silent-time . 3600)
@@ -253,8 +254,21 @@ resulting archive to the standard output port."
 
     (if (or (assoc-ref opts 'dry-run?)
             (build-derivations store drv))
-        (export-paths store files (current-output-port)
-                      #:recursive? (assoc-ref opts 'export-recursive?))
+        (match (assoc-ref opts 'format)
+          ("nar"
+           (export-paths store files (current-output-port)
+                         #:recursive? (assoc-ref opts 'export-recursive?)))
+          ("docker"
+           (match files
+             ((file)
+              (let ((system (assoc-ref opts 'system)))
+                (format #t "~a\n"
+                        (build-docker-image file #:system system))))
+             (_
+              ;; TODO: Remove this restriction.
+              (leave (_ "only a single item can be exported to Docker~%")))))
+          (format
+           (leave (_ "~a: unknown archive format~%") format)))
         (leave (_ "unable to export the given packages~%")))))
 
 (define (generate-key-pair parameters)
@@ -338,15 +352,7 @@ the input port."
                 (else
                  (with-store store
                    (cond ((assoc-ref opts 'export)
-                          (cond ((equal? (assoc-ref opts 'format) "docker")
-                                 (match (car opts)
-                                   (('argument . (? store-path? item))
-                                    (format #t "~a\n"
-                                            (build-docker-image
-                                             item
-                                             #:system (assoc-ref opts 'system))))
-                                   (_ (leave (_ "argument must be a direct store path~%")))))
-                                (_ (export-from-store store opts))))
+                          (export-from-store store opts))
                          ((assoc-ref opts 'import)
                           (import-paths store (current-input-port)))
                          ((assoc-ref opts 'missing)
-- 
2.11.0

Re: [PATCH] guix archive: '-f docker' supports package names as arguments.

[Ricardo Wurmus]

Ludovic Courtès <ludo@gnu.org> writes:

> This allows users to type:
>
>   guix archive -f docker emacs
>
> as was already the case for the 'nar' format.
>
> Reported by David Thompson.
>
> * guix/scripts/archive.scm (%default-options): Add 'format'.
> (export-from-store): Dispatch based on the 'format' key in OPTS.
> (guix-archive): Call 'export-from-store' in all cases when the 'export'
> key is in OPTS.

Nice, looks good to me!  Thank you.

-- 
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
http://elephly.net

Re: [PATCH] guix archive: '-f docker' supports package names as arguments.

[Ludovic Courtès]

Hi!

Ricardo Wurmus <rekado@elephly.net> skribis:

> Ludovic Courtès <ludo@gnu.org> writes:
>
>> This allows users to type:
>>
>>   guix archive -f docker emacs
>>
>> as was already the case for the 'nar' format.
>>
>> Reported by David Thompson.
>>
>> * guix/scripts/archive.scm (%default-options): Add 'format'.
>> (export-from-store): Dispatch based on the 'format' key in OPTS.
>> (guix-archive): Call 'export-from-store' in all cases when the 'export'
>> key is in OPTS.
>
> Nice, looks good to me!  Thank you.

Thanks, applied.

Other things we could do:

  1. Write the archive to standard output, for consistency with what
     happens with the nar format.

  2. Allow several items to be passed on the command line, as is the
     case with nar exports.  We might need to create a profile
     transparently for this to make sense, but then it becomes more than
     just an archive.

Thoughts?

Ludo’.

Re: Creating a docker image with Guix

[Christopher Allan Webber]

Ricardo Wurmus writes:

> Hi Guix,
>
> I just played around with Docker and built up a command to create a
> Docker image for Emacs.
>
> Can anyone find a more elegant way to do this?
>
> --8<---------------cut here---------------start------------->8---
> guix environment --ad-hoc \
>     coreutils bash emacs-no-x-toolkit -- \
>     sh -c 'tar -c $(guix gc --requisites $GUIX_ENVIRONMENT) | \
>            docker import -c "ONBUILD RUN [\"$GUIX_ENVIRONMENT/bin/ln\", \"-s\", \"$GUIX_ENVIRONMENT/bin\", \"/bin\"]" - emacs-base' \
>            && echo -e "FROM emacs-base\nCMD [\"/bin/emacs\"]" | \
>            docker build -
> --8<---------------cut here---------------end--------------->8---
>
> What happens here is this:
>
> * using “guix environment” build a temporary profile containing
>   coreutils (for “ln”), bash (for “sh”), and emacs
>
> * inside the environment $GUIX_ENVIRONMENT points to the profile; take
>   all of the requisites for this profile (“guix gc --requisites”) and
>   send them to “docker import” using a tar pipe
>
> * modify the base image such that it links the profile’s “bin” directory
>   to “/bin” when a derivative image is built
>
> * name that base image “emacs-base”
>
> * then build a derivative of this base image with “docker build” (this
>   causes “ln -s” to be executed) and make it run the command
>   “/bin/emacs” by default
>
> The result is an unnamed image (“docker build” reports an image id such
> as “ae8d6281b56f”) that can be run interactively like this:
>
>     docker run -it --rm ae8d6281b56f
>
> What do you think?  Gross?
>
> ~~ Ricardo

I won't comment on the specific application, but I think it's a good
thing to have.  I was watching the conversation about providing an Emacs
Docker image on emacs-devel, and despite my caveats about Docker, I
think having Guix be a good way to get a "reproducible" image here is a
good idea.  If nothing else, maybe it could become a good reason for
Docker-using people to explore Guix...

Thanks for exploring this, Ricardo!
 - Chris

Re: Creating a docker image with Guix

[Jan Nieuwenhuizen]

Ricardo Wurmus writes:

> I just played around with Docker and built up a command to create a
> Docker image for Emacs.

Nice!

> Can anyone find a more elegant way to do this?
>
> guix environment --ad-hoc \
>     coreutils bash emacs-no-x-toolkit -- \
>     sh -c 'tar -c $(guix gc --requisites $GUIX_ENVIRONMENT) | \
>            docker import -c "ONBUILD RUN [\"$GUIX_ENVIRONMENT/bin/ln\", \"-s\", \"$GUIX_ENVIRONMENT/bin\", \"/bin\"]" - emacs-base' \
>            && echo -e "FROM emacs-base\nCMD [\"/bin/emacs\"]" | \
>            docker build -

Could this be used to package and deploy our cross built guile.exe?

I find myself kind of waiting for some inspiration to take the next
step in the MinGW cross builds...

Greetings,
Jan

-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar®  http://AvatarAcademy.nl  

Re: Creating a docker image with Guix

[Ludovic Courtès]

Jan Nieuwenhuizen <janneke@gnu.org> skribis:

> Ricardo Wurmus writes:
>
>> I just played around with Docker and built up a command to create a
>> Docker image for Emacs.
>
> Nice!
>
>> Can anyone find a more elegant way to do this?
>>
>> guix environment --ad-hoc \
>>     coreutils bash emacs-no-x-toolkit -- \
>>     sh -c 'tar -c $(guix gc --requisites $GUIX_ENVIRONMENT) | \
>>            docker import -c "ONBUILD RUN [\"$GUIX_ENVIRONMENT/bin/ln\", \"-s\", \"$GUIX_ENVIRONMENT/bin\", \"/bin\"]" - emacs-base' \
>>            && echo -e "FROM emacs-base\nCMD [\"/bin/emacs\"]" | \
>>            docker build -
>
> Could this be used to package and deploy our cross built guile.exe?
>
> I find myself kind of waiting for some inspiration to take the next
> step in the MinGW cross builds...

For MinGW, I think it’d be more convenient to just create a standalone
tarball like we do for the Guix binary tarball installation (does Docker
now run on Windows anyway?).

The ‘guix pack’ command I have in mind would generalize what “make
guix-binary.x86_64-linux.tar.xz” does.

Ludo’.