bug#38432: dockerd is not started automatically

bug#38432: dockerd is not started automatically

[andreoss]

When defined in /etc/config.scm as the manual suggests[1]
>    (service docker-service-type)

dockerd is not started with other services.
It could be started manually by
$ herd start dockerd

[1] https://guix.gnu.org/manual/en/html_node/Miscellaneous-Services.html

bug#38432: dockerd is not started automatically

[Maxim Cournoyer]

Hello!

andreoss@SDF.ORG writes:

> When defined in /etc/config.scm as the manual suggests[1]
>>    (service docker-service-type)
>
> dockerd is not started with other services.
> It could be started manually by
> $ herd start dockerd
>
> [1] https://guix.gnu.org/manual/en/html_node/Miscellaneous-Services.html

FWIW, I started experimenting the same on my Guix System after a recent
guix pull & guix system reconfigure.

Maxim

bug#38432: dockerd is not started automatically

[Danny Milosavljevic]

[-- Attachment #1: Type: text/plain, Size: 542 bytes --]

Hi Maxim,

On Thu, 28 May 2020 09:24:58 -0400
Maxim Cournoyer <maxim.cournoyer@gmail.com> wrote:

> FWIW, I started experimenting the same on my Guix System after a recent
> guix pull & guix system reconfigure.

Really?  Back then I've fixed and replaced a lot of the modprobe stuff in
docker upstream--so now it should start up much more reliably on Guix.

I don't know what could be up with it now.  Could you please check logs
in /var/log/containerd.log and /var/log/docker.log (without
manually starting docker beforehand).

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

bug#38432: dockerd is not started automatically

[Maxim Cournoyer]

Hello Danny,

Sorry for the delay.

Danny Milosavljevic <dannym@scratchpost.org> writes:

> Hi Maxim,
>
> On Thu, 28 May 2020 09:24:58 -0400
> Maxim Cournoyer <maxim.cournoyer@gmail.com> wrote:
>
>> FWIW, I started experimenting the same on my Guix System after a recent
>> guix pull & guix system reconfigure.
>
> Really?  Back then I've fixed and replaced a lot of the modprobe stuff in
> docker upstream--so now it should start up much more reliably on Guix.

Really! :-/. I've reconfigured earlier and just rebooted now, and
dockerd was stopped:

--8<---------------cut here---------------start------------->8---
$ sudo herd status dockerd
Password: 
Status of dockerd:
  It is stopped.
  It is enabled.
  Provides (dockerd).
  Requires (containerd dbus-system elogind file-system-/sys/fs/cgroup/blkio file-system-/sys/fs/cgroup/cpu file-system-/sys/fs/cgroup/cpuset file-system-/sys/fs/cgroup/devices file-system-/sys/fs/cgroup/memory file-system-/sys/fs/cgroup/pids networking udev).
  Conflicts with ().
  Will be respawned.
--8<---------------cut here---------------end--------------->8---

> I don't know what could be up with it now.  Could you please check logs
> in /var/log/containerd.log and /var/log/docker.log (without
> manually starting docker beforehand).

Something interesting from containerd.log:

--8<---------------cut here---------------start------------->8---
time="2020-05-31T22:28:15.863091224-04:00" level=info msg="starting containerd" revision=.m version= 
time="2020-05-31T22:28:15.883644866-04:00" level=info msg="loading plugin "io.containerd.content.v1.content"..." type=io.containerd.content.v1 
time="2020-05-31T22:28:15.883714657-04:00" level=info msg="loading plugin "io.containerd.snapshotter.v1.btrfs"..." type=io.containerd.snapshotter.v1 
time="2020-05-31T22:28:15.883975218-04:00" level=info msg="loading plugin "io.containerd.snapshotter.v1.aufs"..." type=io.containerd.snapshotter.v1 
time="2020-05-31T22:28:15.885343166-04:00" level=warning msg="failed to load plugin io.containerd.snapshotter.v1.aufs" error="modprobe aufs failed: "modprobe: FATAL: Module aufs not found in directory /lib/modules/5.4.43-gnu\n": exit status 1" 
time="2020-05-31T22:28:15.885391522-04:00" level=info msg="loading plugin "io.containerd.snapshotter.v1.native"..." type=io.containerd.snapshotter.v1 
time="2020-05-31T22:28:15.885483513-04:00" level=info msg="loading plugin "io.containerd.snapshotter.v1.overlayfs"..." type=io.containerd.snapshotter.v1 
time="2020-05-31T22:28:15.885632392-04:00" level=info msg="loading plugin "io.containerd.snapshotter.v1.zfs"..." type=io.containerd.snapshotter.v1 
time="2020-05-31T22:28:15.885850392-04:00" level=warning msg="failed to load plugin io.containerd.snapshotter.v1.zfs" error="path /var/lib/containerd/io.containerd.snapshotter.v1.zfs must be a zfs filesystem to be used with the zfs snapshotter" 
time="2020-05-31T22:28:15.885871090-04:00" level=info msg="loading plugin "io.containerd.metadata.v1.bolt"..." type=io.containerd.metadata.v1 
time="2020-05-31T22:28:15.885918881-04:00" level=warning msg="could not use snapshotter aufs in metadata plugin" error="modprobe aufs failed: "modprobe: FATAL: Module aufs not found in directory /lib/modules/5.4.43-gnu\n": exit status 1" 
time="2020-05-31T22:28:15.885937520-04:00" level=warning msg="could not use snapshotter zfs in metadata plugin" error="path /var/lib/containerd/io.containerd.snapshotter.v1.zfs must be a zfs filesystem to be used with the zfs snapshotter" 
time="2020-05-31T22:28:15.911651119-04:00" level=info msg="loading plugin "io.containerd.differ.v1.walking"..." type=io.containerd.differ.v1 
time="2020-05-31T22:28:15.911693576-04:00" level=info msg="loading
plugin "io.containerd.gc.v1.scheduler"..." type=io.containerd.gc.v1
--8<---------------cut here---------------end--------------->8---

The only new lines to appear in docker.log following my last reboot are:

--8<---------------cut here---------------start------------->8---
time="2020-05-31T22:28:13.579626539-04:00" level=info msg="Starting up"
failed to start containerd: exec: "containerd": executable file not found in $PATH
--8<---------------cut here---------------end--------------->8---

So, it seems the failure is related to kernel modules not being found
where they are looked for?

I think there were some changes recently in this area, but I haven't
followed closely.

Maxim

bug#38432: dockerd is not started automatically

[Danny Milosavljevic]

[-- Attachment #1: Type: text/plain, Size: 1276 bytes --]

Hi Maxim,

On Sun, 31 May 2020 23:27:30 -0400
Maxim Cournoyer <maxim.cournoyer@gmail.com> wrote:

> time="2020-05-31T22:28:15.885343166-04:00" level=warning msg="failed to load plugin io.containerd.snapshotter.v1.aufs" error="modprobe aufs failed: "modprobe: FATAL: Module aufs not found in directory /lib/modules/5.4.43-gnu\n": exit status 1" 

We don't have aufs, but it's not mandatory anyway.

> The only new lines to appear in docker.log following my last reboot are:
> 
> --8<---------------cut here---------------start------------->8---
> time="2020-05-31T22:28:13.579626539-04:00" level=info msg="Starting up"
> failed to start containerd: exec: "containerd": executable file not found in $PATH
> --8<---------------cut here---------------end--------------->8---
> 
> So, it seems the failure is related to kernel modules not being found
> where they are looked for?

I don't think so this time, at least according to these logs.

Could it be that containerd takes too long to start up or something?  And then it
tries to start it on its own?

To find out, try adding sleep to gnu/services/docker.scm docker-shepherd-service
before make-forkexec-constructor, to make it read

 #~(begin
     (sleep 2)
     (make-forkexec-constructor ....

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

bug#38432: dockerd is not started automatically

[Maxim Cournoyer]

[-- Attachment #1: Type: text/plain, Size: 1983 bytes --]

Hi Danny!

Danny Milosavljevic <dannym@scratchpost.org> writes:

> Hi Maxim,
>
> On Sun, 31 May 2020 23:27:30 -0400
> Maxim Cournoyer <maxim.cournoyer@gmail.com> wrote:
>
>> time="2020-05-31T22:28:15.885343166-04:00" level=warning msg="failed
>> to load plugin io.containerd.snapshotter.v1.aufs" error="modprobe
>> aufs failed: "modprobe: FATAL: Module aufs not found in directory
>> /lib/modules/5.4.43-gnu\n": exit status 1"
>
> We don't have aufs, but it's not mandatory anyway.
>
>> The only new lines to appear in docker.log following my last reboot are:
>> 
>> --8<---------------cut here---------------start------------->8---
>> time="2020-05-31T22:28:13.579626539-04:00" level=info msg="Starting up"
>> failed to start containerd: exec: "containerd": executable file not found in $PATH
>> --8<---------------cut here---------------end--------------->8---
>> 
>> So, it seems the failure is related to kernel modules not being found
>> where they are looked for?
>
> I don't think so this time, at least according to these logs.
>
> Could it be that containerd takes too long to start up or something?  And then it
> tries to start it on its own?
>
> To find out, try adding sleep to gnu/services/docker.scm docker-shepherd-service
> before make-forkexec-constructor, to make it read
>
>  #~(begin
>      (sleep 2)
>      (make-forkexec-constructor ....

It looks like this! Which is weird, because containerd doesn't take much
time to start, even when networking is not yet functional (I suspected
something like this at first).

I wonder what we can do here... looks like we need to poll containerd to
know when it's ready in the start procedure of dockerd (unless I'm
missing something fancier that Shepherd would allow doing).  Else we
could let dockerd spawn its own containerd daemon, which it can do if we
tell it where it is.

Thoughts?

I've made the following changes while troubleshooting this.  It only
printed a couple more lines, but I guess it's nice to have:


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-gnu-services-docker-Add-a-debug-parameter.patch --]
[-- Type: text/x-patch, Size: 3666 bytes --]

From 80f3812e36cfb738b494343c5a6c20cb65588ad1 Mon Sep 17 00:00:00 2001
From: Maxim Cournoyer <maxim.cournoyer@gmail.com>
Date: Mon, 1 Jun 2020 20:54:40 -0400
Subject: [PATCH] gnu: services: docker: Add a debug? parameter.

* gnu/services/docker.scm (docker-configuration): Add a debug? field.
(containerd-shepherd-service): Pass the "--log-level=debug" argument when
DEBUG? is true.
(docker-shepherd-service): Pass the "--debug" and "--log-level=debug"
arguments when DEBUG? is true.
---
 doc/guix.texi           |  9 +++++++++
 gnu/services/docker.scm | 19 +++++++++++++++----
 2 files changed, 24 insertions(+), 4 deletions(-)

diff --git a/doc/guix.texi b/doc/guix.texi
index beeda34ea2..de958c9cf1 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -26287,6 +26287,15 @@ The Docker package to use.
 @item @code{containerd} (default: @var{containerd})
 The Containerd package to use.
 
+@item @code{proxy} (default @var{docker-libnetwork-cmd-proxy})
+The Docker user-land networking proxy package to use.
+
+@item @code{enable-proxy?} (default @code{#f})
+Enable or disable the use of the Docker user-land networking proxy.
+
+@item @code{debug?} (default @code{#f})
+Enable or disable debug output.
+
 @end table
 @end deftp
 
diff --git a/gnu/services/docker.scm b/gnu/services/docker.scm
index d6dc792821..ccdb67ed73 100644
--- a/gnu/services/docker.scm
+++ b/gnu/services/docker.scm
@@ -52,7 +52,10 @@
 loop-back communications.")
   (enable-proxy?
    (boolean #t)
-   "Enable or disable the user-land proxy (enabled by default)."))
+   "Enable or disable the user-land proxy (enabled by default).")
+  (debug?
+   (boolean #f)
+   "Enable or disable debug output."))
 
 (define %docker-accounts
   (list (user-group (name "docker") (system? #t))))
@@ -71,19 +74,24 @@ loop-back communications.")
         (mkdir-p #$state-dir))))
 
 (define (containerd-shepherd-service config)
-  (let* ((package (docker-configuration-containerd config)))
+  (let* ((package (docker-configuration-containerd config))
+         (debug? (docker-configuration-debug? config)))
     (shepherd-service
            (documentation "containerd daemon.")
            (provision '(containerd))
            (start #~(make-forkexec-constructor
-                     (list (string-append #$package "/bin/containerd"))
+                     (list (string-append #$package "/bin/containerd")
+                           #$@(if debug?
+                                  '("--log-level=debug")
+                                  '()))
                      #:log-file "/var/log/containerd.log"))
            (stop #~(make-kill-destructor)))))
 
 (define (docker-shepherd-service config)
   (let* ((docker (docker-configuration-docker config))
          (enable-proxy? (docker-configuration-enable-proxy? config))
-         (proxy (docker-configuration-proxy config)))
+         (proxy (docker-configuration-proxy config))
+         (debug? (docker-configuration-debug? config)))
     (shepherd-service
            (documentation "Docker daemon.")
            (provision '(dockerd))
@@ -101,6 +109,9 @@ loop-back communications.")
            (start #~(make-forkexec-constructor
                      (list (string-append #$docker "/bin/dockerd")
                            "-p" "/var/run/docker.pid"
+                           #$@(if debug?
+                                  '("--debug" "--log-level=debug")
+                                  '())
                            (if #$enable-proxy? "--userland-proxy" "")
                            "--userland-proxy-path" (string-append #$proxy
                                                                   "/bin/proxy"))
-- 
2.26.2


[-- Attachment #3: Type: text/plain, Size: 7 bytes --]


Maxim

bug#38432: dockerd is not started automatically

[Maxim Cournoyer]

Hello Danny!

The service definition here:
https://github.com/containerd/containerd/blob/master/containerd.service
uses the systemd notify mechanism to allow the application (containerd)
to message the init (systemd) that it is ready.

leoprikler on #guix had the idea to patch this away and replace it by
some code that'd write a PID file.  That's an interesting idea!  Even
nicer would be to have Shepherd listen on the socket that the sd_notify
mechanism relies on and natively support the systemd notify thing :-).

Food for thoughts...

Maxim