[bug#35281] [PATCH] gnu: docker: Add a couple go dependencies and enable docker-proxy.

[bug#35281] [PATCH] gnu: docker: Add a couple go dependencies and enable docker-proxy.

[Maxim Cournoyer]

[-- Attachment #1.1: Type: text/plain, Size: 970 bytes --]

Hello Guix!

Before this change, attempting to start a docker registry such as with
the following command[0]:

--8<---------------cut here---------------start------------->8---
docker run -d -p 5000:5000 --restart=always --name registry registry:2
--8<---------------cut here---------------end--------------->8---

Would give the following error:

--8<---------------cut here---------------start------------->8---
/gnu/store/dzaijl53fcd3jhkpd70vsf4cnvv10ywj-docker-cli-18.09.3/bin/docker:
Error response from daemon: driver failed programming external
connectivity on endpoint registry
(709754084a9e208c32075e47ea9584296a6f274deeef08283d0de9c9a5161112):
exec: "docker-proxy": executable file not found in $PATH.
--8<---------------cut here---------------end--------------->8---

This series of patches adds docker-libnetwork-cmd-proxy and a few new go
packages it required.

The docker service is modified to enable (or disable) using a 'docker-proxy'.

Thanks!

Maxim


[-- Attachment #1.2: 0001-gnu-Add-go-sctp.patch --]
[-- Type: text/x-patch, Size: 2509 bytes --]

From 2e46dd6c449679ecae6a13a7a922eaf6b6947164 Mon Sep 17 00:00:00 2001
From: Maxim Cournoyer <maxim.cournoyer@gmail.com>
Date: Fri, 5 Apr 2019 01:07:58 -0400
Subject: [PATCH 1/6] gnu: Add go-sctp.

* gnu/packages/networking.scm (go-sctp): New variable.
---
 gnu/packages/networking.scm | 27 +++++++++++++++++++++++++++
 1 file changed, 27 insertions(+)

diff --git a/gnu/packages/networking.scm b/gnu/packages/networking.scm
index 46aed1e549..5163726393 100644
--- a/gnu/packages/networking.scm
+++ b/gnu/packages/networking.scm
@@ -26,6 +26,7 @@
 ;;; Copyright © 2018 Marius Bakke <mbakke@fastmail.com>
 ;;; Copyright © 2018 Oleg Pykhalov <go.wigust@gmail.com>
 ;;; Copyright © 2018 Pierre Neidhardt <mail@ambrevar.xyz>
+;;; Copyright © 2019 Maxim Cournoyer <maxim.cournoyer@gmail.com>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -50,6 +51,7 @@
   #:use-module (guix build-system cmake)
   #:use-module (guix build-system glib-or-gtk)
   #:use-module (guix build-system gnu)
+  #:use-module (guix build-system go)
   #:use-module (guix build-system perl)
   #:use-module (guix build-system python)
   #:use-module (gnu packages)
@@ -730,6 +732,31 @@ manage, and delete Internet resources from Gandi.net such as domain names,
 virtual machines, and certificates.")
     (license license:gpl3+)))
 
+(define-public go-sctp
+  ;; docker-libnetwork-cmd-proxy requires this exact commit.
+  (let ((commit "07191f837fedd2f13d1ec7b5f885f0f3ec54b1cb")
+        (revision "1"))
+    (package
+      (name "go-sctp")
+      (version (git-version "0.0.0" revision commit))
+      (source (origin
+                (method git-fetch)
+                (uri (git-reference
+                      (url "https://github.com/ishidawataru/sctp.git")
+                      (commit commit)))
+                (file-name (git-file-name name version))
+                (sha256
+                 (base32
+                  "1mk9ncm10gwi5pn5wcw4skbyf4qg7n5qdf1mim4gf3mrckvi6g6h"))))
+      (build-system go-build-system)
+      (arguments
+       `(#:import-path "github.com/ishidawataru/sctp"))
+      (home-page "https://github.com/ishidawataru/sctp")
+      (synopsis "SCTP library for the Go programming language")
+      (description "This library provides methods for using the stream control
+transmission protocol (SCTP) in a Go application.")
+      (license license:asl2.0))))
+
 (define-public httping
   (package
     (name "httping")
-- 
2.20.1


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1.3: 0002-gnu-Add-go-netns.patch --]
[-- Type: text/x-patch, Size: 1954 bytes --]

From 0081c4231f9e25879c287fca54ec9db4929d1711 Mon Sep 17 00:00:00 2001
From: Maxim Cournoyer <maxim.cournoyer@gmail.com>
Date: Fri, 5 Apr 2019 01:53:00 -0400
Subject: [PATCH 2/6] gnu: Add go-netns.

* gnu/packages/networking.scm (go-netns): New variable.
---
 gnu/packages/networking.scm | 25 +++++++++++++++++++++++++
 1 file changed, 25 insertions(+)

diff --git a/gnu/packages/networking.scm b/gnu/packages/networking.scm
index 5163726393..1407f6208e 100644
--- a/gnu/packages/networking.scm
+++ b/gnu/packages/networking.scm
@@ -732,6 +732,31 @@ manage, and delete Internet resources from Gandi.net such as domain names,
 virtual machines, and certificates.")
     (license license:gpl3+)))
 
+(define-public go-netns
+  (let ((commit "13995c7128ccc8e51e9a6bd2b551020a27180abd")
+        (revision "1"))
+    (package
+      (name "go-netns")
+      (version (git-version "0.0.0" revision commit))
+      (source (origin
+                (method git-fetch)
+                (uri (git-reference
+                      (url "https://github.com/vishvananda/netns.git")
+                      (commit commit)))
+                (file-name (git-file-name name version))
+                (sha256
+                 (base32
+                  "1zk6w8158qi4niva5rijchbv9ixgmijsgqshh54wdaav4xrhjshn"))))
+      (build-system go-build-system)
+      (arguments
+       `(#:import-path "github.com/vishvananda/netns"
+         #:tests? #f))                  ;tests require root privileges
+      (home-page "https://github.com/vishvananda/netns")
+      (synopsis "Simple network namespace handling for Go")
+      (description "The netns package provides a simple interface for
+handling network namespaces in Go.")
+      (license license:asl2.0))))
+
 (define-public go-sctp
   ;; docker-libnetwork-cmd-proxy requires this exact commit.
   (let ((commit "07191f837fedd2f13d1ec7b5f885f0f3ec54b1cb")
-- 
2.20.1


[-- Attachment #1.4: 0003-gnu-Add-go-netlink.patch --]
[-- Type: text/x-patch, Size: 2839 bytes --]

From 2a8b23da6fe7ef09f0931231c67f25cd4c60f24f Mon Sep 17 00:00:00 2001
From: Maxim Cournoyer <maxim.cournoyer@gmail.com>
Date: Fri, 5 Apr 2019 01:57:44 -0400
Subject: [PATCH 3/6] gnu: Add go-netlink.

* gnu/packages/linux.scm (go-netlink): New variable.
---
 gnu/packages/linux.scm | 30 ++++++++++++++++++++++++++++++
 1 file changed, 30 insertions(+)

diff --git a/gnu/packages/linux.scm b/gnu/packages/linux.scm
index e8ee4df4f3..c8adf52ff8 100644
--- a/gnu/packages/linux.scm
+++ b/gnu/packages/linux.scm
@@ -33,6 +33,7 @@
 ;;; Copyright © 2018 Pierre Langlois <pierre.langlois@gmx.com>
 ;;; Copyright © 2018 Vasile Dumitrascu <va511e@yahoo.com>
 ;;; Copyright © 2019 Tim Gesthuizen <tim.gesthuizen@yahoo.de>
+;;; Copyright © 2019 Maxim Cournoyer <maxim.cournoyer@gmail.com>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -77,6 +78,7 @@
   #:use-module (gnu packages gcc)
   #:use-module (gnu packages gettext)
   #:use-module (gnu packages glib)
+  #:use-module (gnu packages golang)
   #:use-module (gnu packages gperf)
   #:use-module (gnu packages gtk)
   #:use-module (gnu packages libunwind)
@@ -116,6 +118,7 @@
   #:use-module (gnu packages swig)
   #:use-module (guix build-system cmake)
   #:use-module (guix build-system gnu)
+  #:use-module (guix build-system go)
   #:use-module (guix build-system python)
   #:use-module (guix build-system trivial)
   #:use-module (guix download)
@@ -5151,6 +5154,33 @@ nfnetlink_queue, nfnetlink_conntrack) and their respective users and/or
 management tools in userspace.")
     (license license:gpl2)))
 
+(define-public go-netlink
+  (package
+    (name "go-netlink")
+    (version "1.0.0")
+    (source (origin
+              (method git-fetch)
+              (uri (git-reference
+                    (url "https://github.com/vishvananda/netlink.git")
+                    (commit (string-append "v" version))))
+              (file-name (git-file-name name version))
+              (sha256
+               (base32
+                "0hpzghf1a4cwawzhkiwdzin80h6hd09fskl77d5ppgc084yvj8x0"))))
+    (build-system go-build-system)
+    (arguments
+     `(#:import-path "github.com/vishvananda/netlink"))
+    (native-inputs
+     `(("go-golang-org-x-sys-unix" ,go-golang-org-x-sys-unix)
+       ("go-netns" ,go-netns)))
+    (home-page "https://github.com/vishvananda/netlink")
+    (synopsis "Simple netlink library for Go")
+    (description "The netlink package provides a simple netlink library for
+Go.  Netlink is the interface a user-space program in Linux uses to
+communicate with the kernel.  It can be used to add and remove interfaces, set
+IP addresses and routes, and configure IPsec.")
+    (license license:asl2.0)))
+
 (define-public xfsprogs
   (package
     (name "xfsprogs")
-- 
2.20.1


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1.5: 0004-gnu-Add-docker-libnetwork.patch --]
[-- Type: text/x-patch, Size: 2830 bytes --]

From 721d1a93961bf653a02cccdbfaf92e0514a3ae59 Mon Sep 17 00:00:00 2001
From: Maxim Cournoyer <maxim.cournoyer@gmail.com>
Date: Fri, 5 Apr 2019 02:32:40 -0400
Subject: [PATCH 4/6] gnu: Add docker-libnetwork.

* gnu/packages/docker.scm (docker-libnetwork): New private variable.
---
 gnu/packages/docker.scm | 41 +++++++++++++++++++++++++++++++++++++++++
 1 file changed, 41 insertions(+)

diff --git a/gnu/packages/docker.scm b/gnu/packages/docker.scm
index a11ce266d2..df0bbca1bc 100644
--- a/gnu/packages/docker.scm
+++ b/gnu/packages/docker.scm
@@ -227,6 +227,47 @@ network attachments.")
     (home-page "http://containerd.io/")
     (license license:asl2.0)))
 
+;;; Private package that shouldn't be used directly; its purposes is to be used
+;;; as a template for the various packages it contains.
+(define docker-libnetwork
+  ;; There are no recent release for libnetwork, so choose the last commit of
+  ;; the branch that Docker uses, as can be seen in the Docker source file
+  ;; 'hack/dockerfile/install/proxy.installer'.
+  (let ((commit "4725f2163fb214a6312f3beae5991f838ec36326")
+        (version "18.09")
+        (revision "1"))
+    (package
+      (name "docker-libnetwork")
+      (version (git-version version "1" commit))
+      (source (origin
+                (method git-fetch)
+                (uri (git-reference
+                      (url "https://github.com/docker/libnetwork.git")
+                      (commit commit)))
+                (file-name (git-file-name name version))
+                (sha256
+                 (base32
+                  "1zpnxki8qfzha6ljahpwd3vkzmjhsvkmf73w6crm4ilxxw5vnpfb"))
+                ;; Delete bundled ("vendored") free software source code.
+                (modules '((guix build utils)))
+                (snippet '(begin
+                            (delete-file-recursively "vendor")
+                            #t))))
+      (build-system go-build-system)
+      (arguments
+       `(#:import-path "github.com/docker/libnetwork/"
+         ;; The tests fail with the error:
+         ;; src/github.com/docker/libnetwork/network.go:1057: Warnf format %q
+         ;; has arg n.configOnly of wrong type bool.
+         #:tests? #f))
+      (home-page "https://github.com/docker/libnetwork/")
+      (synopsis "Networking for containers")
+      (description "Libnetwork provides a native Go implementation for
+connecting containers.  The goal of @code{libnetwork} is to deliver a robust
+container network model that provides a consistent programming interface and
+the required network abstractions for applications.")
+      (license license:asl2.0))))
+
 ;; TODO: Patch out modprobes for ip_vs, nf_conntrack,
 ;; brige, nf_conntrack_netlink, aufs.
 (define-public docker
-- 
2.20.1


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1.6: 0005-gnu-Add-docker-libnetwork-cmd-proxy.patch --]
[-- Type: text/x-patch, Size: 2035 bytes --]

From 843ecd2ff5aa5f69ea8a83f2da8e0d783be4b36a Mon Sep 17 00:00:00 2001
From: Maxim Cournoyer <maxim.cournoyer@gmail.com>
Date: Fri, 5 Apr 2019 02:33:38 -0400
Subject: [PATCH 5/6] gnu: Add docker-libnetwork-cmd-proxy.

* gnu/packages/docker.scm (docker-libnetwork-cmd-proxy): New variable.
---
 gnu/packages/docker.scm | 23 +++++++++++++++++++++++
 1 file changed, 23 insertions(+)

diff --git a/gnu/packages/docker.scm b/gnu/packages/docker.scm
index df0bbca1bc..1067555296 100644
--- a/gnu/packages/docker.scm
+++ b/gnu/packages/docker.scm
@@ -36,6 +36,7 @@
   #:use-module (gnu packages glib)
   #:use-module (gnu packages golang)
   #:use-module (gnu packages linux)
+  #:use-module (gnu packages networking)
   #:use-module (gnu packages pkg-config)
   #:use-module (gnu packages python)
   #:use-module (gnu packages python-web)
@@ -268,6 +269,28 @@ container network model that provides a consistent programming interface and
 the required network abstractions for applications.")
       (license license:asl2.0))))
 
+(define-public docker-libnetwork-cmd-proxy
+  (package
+    (inherit docker-libnetwork)
+    (name "docker-libnetwork-cmd-proxy")
+    (arguments
+     `(#:import-path "github.com/docker/libnetwork/cmd/proxy"
+       #:unpack-path "github.com/docker/libnetwork"
+       #:install-source? #f))
+    (native-inputs
+     `(("go-sctp" ,go-sctp)
+       ;; For tests.
+       ("logrus" ,go-github-com-sirupsen-logrus)
+       ("go-netlink" ,go-netlink)
+       ("go-netns" ,go-netns)
+       ("go-golang-org-x-crypto-ssh-terminal"
+        ,go-golang-org-x-crypto-ssh-terminal)
+       ("go-golang-org-x-sys-unix" ,go-golang-org-x-sys-unix)))
+    (synopsis "Docker user-space proxy")
+    (description "A proxy running in the user space.  It is used by the
+built-in registry server of Docker.")
+    (license license:asl2.0)))
+
 ;; TODO: Patch out modprobes for ip_vs, nf_conntrack,
 ;; brige, nf_conntrack_netlink, aufs.
 (define-public docker
-- 
2.20.1


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1.7: 0006-services-docker-Add-new-fields-to-support-proxy.patch --]
[-- Type: text/x-patch, Size: 2798 bytes --]

From fd1003dc333ede95a8fa2813b7e8ab2f6cfe82d2 Mon Sep 17 00:00:00 2001
From: Maxim Cournoyer <maxim.cournoyer@gmail.com>
Date: Fri, 5 Apr 2019 02:34:16 -0400
Subject: [PATCH 6/6] services: docker: Add new fields to support proxy.

The Docker proxy enables inter-container and outside-to-container loopback,
and is required by the Docker registry server.

* gnu/services/docker.scm (docker-configuration)[proxy,
enable-proxy?]: Add fields.
(docker-shepherd-service): Use them.
(serialize-boolean): New function.
---
 gnu/services/docker.scm | 23 ++++++++++++++++++++---
 1 file changed, 20 insertions(+), 3 deletions(-)

diff --git a/gnu/services/docker.scm b/gnu/services/docker.scm
index 8b5edf5cb0..94a04c8996 100644
--- a/gnu/services/docker.scm
+++ b/gnu/services/docker.scm
@@ -31,13 +31,25 @@
   #:export (docker-configuration
             docker-service-type))
 
+;;; We're not using serialize-configuration, but we must define this because
+;;; the define-configuration macro validates it exists.
+(define (serialize-boolean field-name val)
+  "")
+
 (define-configuration docker-configuration
   (docker
    (package docker)
    "Docker daemon package.")
   (containerd
    (package containerd)
-   "containerd package."))
+   "containerd package.")
+  (proxy
+   (package docker-libnetwork-cmd-proxy)
+   "The proxy package to support inter-container and outside-container
+loop-back communications.")
+  (enable-proxy?
+   (boolean #t)
+   "Enable or disable the user-land proxy (enabled by default)."))
 
 (define %docker-accounts
   (list (user-group (name "docker") (system? #t))))
@@ -66,7 +78,9 @@
            (stop #~(make-kill-destructor)))))
 
 (define (docker-shepherd-service config)
-  (let* ((docker (docker-configuration-docker config)))
+  (let* ((docker (docker-configuration-docker config))
+         (enable-proxy? (docker-configuration-enable-proxy? config))
+         (proxy (docker-configuration-proxy config)))
     (shepherd-service
            (documentation "Docker daemon.")
            (provision '(dockerd))
@@ -83,7 +97,10 @@
                           udev))
            (start #~(make-forkexec-constructor
                      (list (string-append #$docker "/bin/dockerd")
-                           "-p" "/var/run/docker.pid")
+                           "-p" "/var/run/docker.pid"
+                           (if #$enable-proxy? "--userland-proxy" "")
+                           "--userland-proxy-path" (string-append #$proxy
+                                                                  "/bin/proxy"))
                      #:pid-file "/var/run/docker.pid"
                      #:log-file "/var/log/docker.log"))
            (stop #~(make-kill-destructor)))))
-- 
2.20.1


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

[bug#35281] [PATCH] gnu: docker: Add a couple go dependencies and enable docker-proxy.

[Danny Milosavljevic]

[-- Attachment #1: Type: text/plain, Size: 467 bytes --]

Hi,

On Sun, 14 Apr 2019 19:02:53 -0400
Maxim Cournoyer <maxim.cournoyer@gmail.com> wrote:

> +         ;; The tests fail with the error:
> +         ;; src/github.com/docker/libnetwork/network.go:1057: Warnf format %q
> +         ;; has arg n.configOnly of wrong type bool.
> +         #:tests? #f))
> +      (home-page "https://github.com/docker/libnetwork/")

It might make sense to report that bug upstream--they're helpful people.

Otherwise LGTM!

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

[bug#35281] [PATCH] gnu: docker: Add a couple go dependencies and enable docker-proxy.

[Maxim Cournoyer]

Hi Danny!

Danny Milosavljevic <dannym@scratchpost.org> writes:

> Hi,
>
> On Sun, 14 Apr 2019 19:02:53 -0400
> Maxim Cournoyer <maxim.cournoyer@gmail.com> wrote:
>
>> +         ;; The tests fail with the error:
>> +         ;; src/github.com/docker/libnetwork/network.go:1057: Warnf format %q
>> +         ;; has arg n.configOnly of wrong type bool.
>> +         #:tests? #f))
>> +      (home-page "https://github.com/docker/libnetwork/")
>
> It might make sense to report that bug upstream--they're helpful people.
>
> Otherwise LGTM!

Thanks for bringing my attention to that part; I've tried re-enabling
the tests but it seems to fail because of unmet dependencies now --
which makes sense since I'm not adding any inputs.  I've removed the
tests? #f argument and detailed in the top comment that this is simply
used as a template and doesn't even build.

Thanks for the review!

Maxim

bug#35281: [PATCH] gnu: docker: Add a couple go dependencies and enable docker-proxy.

[T460s laptop]

Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:

> Hi Danny!
>
> Danny Milosavljevic <dannym@scratchpost.org> writes:
>
>> Hi,
>>
>> On Sun, 14 Apr 2019 19:02:53 -0400
>> Maxim Cournoyer <maxim.cournoyer@gmail.com> wrote:
>>
>>> +         ;; The tests fail with the error:
>>> +         ;; src/github.com/docker/libnetwork/network.go:1057: Warnf format %q
>>> +         ;; has arg n.configOnly of wrong type bool.
>>> +         #:tests? #f))
>>> +      (home-page "https://github.com/docker/libnetwork/")
>>
>> It might make sense to report that bug upstream--they're helpful people.
>>
>> Otherwise LGTM!
>
> Thanks for bringing my attention to that part; I've tried re-enabling
> the tests but it seems to fail because of unmet dependencies now --
> which makes sense since I'm not adding any inputs.  I've removed the
> tests? #f argument and detailed in the top comment that this is simply
> used as a template and doesn't even build.
>
> Thanks for the review!
>
> Maxim

Pushed as 7a31d93a61. Thanks!

Maxim