bug#55358: docker containers stopped when doing guix install or guix shell

bug#55358: docker containers stopped when doing guix install or guix shell

[Remco van 't Veer]

On a Guix system host, some running docker containers are stopped when
doing guix install or other guix operations like shell.  I noticed this
happing to mysql and postgres containers but an elasticsearch container
just keeps running.

Here's an example session:

  $ docker ps
  CONTAINER ID  IMAGE           COMMAND                  CREATED         STATUS        PORTS     NAMES
  $ docker run -d postgres:10.10
  ..
  2b52ee072b1f5584cae597afb033cdcc0e560bbe9145b17b41502c204034e60b
  $ docker ps
  CONTAINER ID  IMAGE           COMMAND                  CREATED         STATUS        PORTS     NAMES
  2b52ee072b1f  postgres:10.10  "docker-entrypoint.s…"  2 seconds ago   Up 1 seconds  5432/tcp  blah_blah
  $ guix shell xeyes -- xeyes
  substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
  0.0 MB will be downloaded
   xeyes-1.1.2  11KiB                                                                           613KiB/s 00:00 [##################] 100.0%
  The following derivation will be built:
    /gnu/store/xc002hxl4g8mskqmpm0grsk8s45m91gz-profile.drv

  applying 4 grafts for xeyes-1.1.2 ...
  building CA certificate bundle...
  listing Emacs sub-directories...
  building fonts directory...
  building directory of Info manuals...
  building profile with 1 package...
  $ docker ps
  CONTAINER ID  IMAGE           COMMAND                  CREATED         STATUS        PORTS     NAMES
  $ exit

First we see no docker containers are running, then we start postgres-10
from docker hub, we see its container is running, then we do something
using guix-shell on an application *not already available on this
system*, and now the container died.  This does not work the second time
when the "derivation" is already "built".

Cheers,
Remco

bug#55358: docker containers stopped when doing guix install or guix shell

[Maxim Cournoyer]

Hi,

Remco van 't Veer <remco@remworks.net> writes:

> On a Guix system host, some running docker containers are stopped when
> doing guix install or other guix operations like shell.  I noticed this
> happing to mysql and postgres containers but an elasticsearch container
> just keeps running.
>
> Here's an example session:
>
>   $ docker ps
>   CONTAINER ID  IMAGE           COMMAND                  CREATED         STATUS        PORTS     NAMES
>   $ docker run -d postgres:10.10
>   ..
>   2b52ee072b1f5584cae597afb033cdcc0e560bbe9145b17b41502c204034e60b
>   $ docker ps
>   CONTAINER ID  IMAGE           COMMAND                  CREATED         STATUS        PORTS     NAMES
>   2b52ee072b1f  postgres:10.10  "docker-entrypoint.s…"  2 seconds ago   Up 1 seconds  5432/tcp  blah_blah
>   $ guix shell xeyes -- xeyes
>   substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
>   0.0 MB will be downloaded
>    xeyes-1.1.2  11KiB                                                                           613KiB/s 00:00 [##################] 100.0%
>   The following derivation will be built:
>     /gnu/store/xc002hxl4g8mskqmpm0grsk8s45m91gz-profile.drv
>
>   applying 4 grafts for xeyes-1.1.2 ...
>   building CA certificate bundle...
>   listing Emacs sub-directories...
>   building fonts directory...
>   building directory of Info manuals...
>   building profile with 1 package...
>   $ docker ps
>   CONTAINER ID  IMAGE           COMMAND                  CREATED         STATUS        PORTS     NAMES
>   $ exit
>
> First we see no docker containers are running, then we start postgres-10
> from docker hub, we see its container is running, then we do something
> using guix-shell on an application *not already available on this
> system*, and now the container died.  This does not work the second time
> when the "derivation" is already "built".

Are you still able to reproduce this using the new version of docker
packaged in Guix?

Thanks,

Maxim

bug#55358: docker containers stopped when doing guix install or guix shell

[Remco van 't Veer]

2022/07/12 09:48, Maxim Cournoyer:

> Hi,
>
> Remco van 't Veer <remco@remworks.net> writes:
>
>> On a Guix system host, some running docker containers are stopped when
>> doing guix install or other guix operations like shell.  I noticed this
>> happing to mysql and postgres containers but an elasticsearch container
>> just keeps running.
>>
>> Here's an example session:
>>
>>   $ docker ps
>>   CONTAINER ID  IMAGE           COMMAND                  CREATED         STATUS        PORTS     NAMES
>>   $ docker run -d postgres:10.10
>>   ..
>>   2b52ee072b1f5584cae597afb033cdcc0e560bbe9145b17b41502c204034e60b
>>   $ docker ps
>>   CONTAINER ID  IMAGE           COMMAND                  CREATED         STATUS        PORTS     NAMES
>>   2b52ee072b1f  postgres:10.10  "docker-entrypoint.s…"  2 seconds ago   Up 1 seconds  5432/tcp  blah_blah
>>   $ guix shell xeyes -- xeyes
>>   substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
>>   0.0 MB will be downloaded
>>    xeyes-1.1.2  11KiB                                                                           613KiB/s 00:00 [##################] 100.0%
>>   The following derivation will be built:
>>     /gnu/store/xc002hxl4g8mskqmpm0grsk8s45m91gz-profile.drv
>>
>>   applying 4 grafts for xeyes-1.1.2 ...
>>   building CA certificate bundle...
>>   listing Emacs sub-directories...
>>   building fonts directory...
>>   building directory of Info manuals...
>>   building profile with 1 package...
>>   $ docker ps
>>   CONTAINER ID  IMAGE           COMMAND                  CREATED         STATUS        PORTS     NAMES
>>   $ exit
>>
>> First we see no docker containers are running, then we start postgres-10
>> from docker hub, we see its container is running, then we do something
>> using guix-shell on an application *not already available on this
>> system*, and now the container died.  This does not work the second time
>> when the "derivation" is already "built".
>
> Are you still able to reproduce this using the new version of docker
> packaged in Guix?

Yes, same problem after a guix pull and guix system reconfigure just now.

  $ guix describe
  Generation 72	Jul 12 2022 16:11:38	(current)
    guix 9173cb5
      repository URL: https://git.savannah.gnu.org/git/guix.git
      branch: master
      commit: 9173cb522ddc4f31f21948cee3fb214fd67ef616

Cheers,
Remco

bug#55358: docker containers stopped when doing guix install or guix shell

[Remco van 't Veer]

I think I know what is causing the issue.  Both the "standard" mysql and
postgres containers use user-id 999 to run the database service (this
seems like a common practice because the redis container is configured
similarly).  That user-id is also configured as guixbuilder01 so I guess
the guix daemon is killing those when processes when it finishes doing
builds.

Does that make sense?  If so can guix daemon be fixed to be a tad more
gentile to the processes not spawned on its behalf?


2022/07/12 16:37, Remco van 't Veer:

> 2022/07/12 09:48, Maxim Cournoyer:
>
>> Hi,
>>
>> Remco van 't Veer <remco@remworks.net> writes:
>>
>>> On a Guix system host, some running docker containers are stopped when
>>> doing guix install or other guix operations like shell.  I noticed this
>>> happing to mysql and postgres containers but an elasticsearch container
>>> just keeps running.
>>>
>>> Here's an example session:
>>>
>>>   $ docker ps
>>>   CONTAINER ID  IMAGE           COMMAND                  CREATED         STATUS        PORTS     NAMES
>>>   $ docker run -d postgres:10.10
>>>   ..
>>>   2b52ee072b1f5584cae597afb033cdcc0e560bbe9145b17b41502c204034e60b
>>>   $ docker ps
>>>   CONTAINER ID  IMAGE           COMMAND                  CREATED         STATUS        PORTS     NAMES
>>>   2b52ee072b1f  postgres:10.10  "docker-entrypoint.s…"  2 seconds ago   Up 1 seconds  5432/tcp  blah_blah
>>>   $ guix shell xeyes -- xeyes
>>>   substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
>>>   0.0 MB will be downloaded
>>>    xeyes-1.1.2  11KiB                                                                           613KiB/s 00:00 [##################] 100.0%
>>>   The following derivation will be built:
>>>     /gnu/store/xc002hxl4g8mskqmpm0grsk8s45m91gz-profile.drv
>>>
>>>   applying 4 grafts for xeyes-1.1.2 ...
>>>   building CA certificate bundle...
>>>   listing Emacs sub-directories...
>>>   building fonts directory...
>>>   building directory of Info manuals...
>>>   building profile with 1 package...
>>>   $ docker ps
>>>   CONTAINER ID  IMAGE           COMMAND                  CREATED         STATUS        PORTS     NAMES
>>>   $ exit
>>>
>>> First we see no docker containers are running, then we start postgres-10
>>> from docker hub, we see its container is running, then we do something
>>> using guix-shell on an application *not already available on this
>>> system*, and now the container died.  This does not work the second time
>>> when the "derivation" is already "built".
>>
>> Are you still able to reproduce this using the new version of docker
>> packaged in Guix?
>
> Yes, same problem after a guix pull and guix system reconfigure just now.
>
>   $ guix describe
>   Generation 72	Jul 12 2022 16:11:38	(current)
>     guix 9173cb5
>       repository URL: https://git.savannah.gnu.org/git/guix.git
>       branch: master
>       commit: 9173cb522ddc4f31f21948cee3fb214fd67ef616
>
> Cheers,
> Remco

Re: bug#55358: docker containers stopped when doing guix install or guix shell

[Remco van 't Veer]

Hi Maxim and Zimoun,

2023/02/09 13:26, Remco van 't Veer:

> I think I know what is causing the issue.  Both the "standard" mysql and
> postgres containers use user-id 999 to run the database service (this
> seems like a common practice because the redis container is configured
> similarly).  That user-id is also configured as guixbuilder01 so I guess
> the guix daemon is killing those when processes when it finishes doing
> builds.

I found a solution / workaround for this problem by using
"userns-remap".  This feature allows the remapping of uids and guids to
different ranges.  I tried it by hacking the required files into my
etc-directory and it works; guix no long kills my database containers.

I'd like to add this feature to docker-service-type having a new
configuration option named enable-userns-remap? which introduces a new
user and group (both named dockremap) to do the remapping by adding some
configurable number to the uids and guids of the running container.  In
/etc/subuid and /etc/subgid it would look like:

  dockremap:100000:65536

See https://docs.docker.com/engine/security/userns-remap/ for
documentation about this.

WDYT?

Cheers,
Remco


--
https://debbugs.gnu.org/cgi/bugreport.cgi?bug=55358

Re: bug#55358: docker containers stopped when doing guix install or guix shell

[Csepp]

Remco van 't Veer <remco@remworks.net> writes:

> Hi Maxim and Zimoun,
>
> 2023/02/09 13:26, Remco van 't Veer:
>
>> I think I know what is causing the issue.  Both the "standard" mysql and
>> postgres containers use user-id 999 to run the database service (this
>> seems like a common practice because the redis container is configured
>> similarly).  That user-id is also configured as guixbuilder01 so I guess
>> the guix daemon is killing those when processes when it finishes doing
>> builds.
>
> I found a solution / workaround for this problem by using
> "userns-remap".  This feature allows the remapping of uids and guids to
> different ranges.  I tried it by hacking the required files into my
> etc-directory and it works; guix no long kills my database containers.
>
> I'd like to add this feature to docker-service-type having a new
> configuration option named enable-userns-remap? which introduces a new
> user and group (both named dockremap) to do the remapping by adding some
> configurable number to the uids and guids of the running container.  In
> /etc/subuid and /etc/subgid it would look like:
>
>   dockremap:100000:65536
>
> See https://docs.docker.com/engine/security/userns-remap/ for
> documentation about this.
>
> WDYT?
>
> Cheers,
> Remco

The rootless podman example that was shared a few months ago could be
relevant to this, since that also adds a subuid/subgid mapping.

[PATCH] services: docker: Add 'enable-userns-remap?' argument.

[Remco van 't Veer]

* gnu/services/docker.scm (docker-configuration): Define the argument.
* gnu/services/docker.scm (docker-shepherd-service): Use it.
* doc/guix.texi (Docker Service): Document it.
---
 doc/guix.texi           | 27 ++++++++++++++++++++++++++-
 gnu/services/docker.scm | 28 +++++++++++++++++++++++++++-
 2 files changed, 53 insertions(+), 2 deletions(-)

diff --git a/doc/guix.texi b/doc/guix.texi
index f4cca66d76..ae185ced61 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -100,7 +100,7 @@
 Copyright @copyright{} 2021 muradm@*
 Copyright @copyright{} 2021, 2022 Andrew Tropin@*
 Copyright @copyright{} 2021 Sarah Morgensen@*
-Copyright @copyright{} 2022 Remco van 't Veer@*
+Copyright @copyright{} 2022, 2023 Remco van 't Veer@*
 Copyright @copyright{} 2022 Aleksandr Vityazev@*
 Copyright @copyright{} 2022 Philip M@sup{c}Grath@*
 Copyright @copyright{} 2022 Karl Hallsby@*
@@ -38533,6 +38533,31 @@ Miscellaneous Services
 @item @code{enable-iptables?} (default @code{#t})
 Enable or disable the addition of iptables rules.
 
+@item @code{enable-userns-remap?} (default @code{#f})
+Enable remapping and subordinate user and group IDs.
+
+A system user account named @code{dockremap} and user group named
+@code{dockremap} will be created.  They must be mapped using the
+@file{/etc/subuid} and @file{/etc/subguid} files otherwise docker fail
+to startup.
+
+Here's an example service to setup both files:
+
+@lisp
+(simple-service
+   'subuid-subgid etc-service-type
+   (list `("subuid"
+           ,(plain-file "subuid"
+                        "dockremap:65536:65536\n"))
+         `("subgid"
+           ,(plain-file "subgid"
+                        "dockremap:65536:65536\n"))))
+@end lisp
+
+The above will remap to UID 0 (root) to 65536, UID 1 to 65537 etc.  For
+more information regarding the format of these files, consult
+@command{man 5 subuid} and @command{man 5 subgid}.
+
 @item @code{environment-variables} (default: @code{()})
 List of environment variables to set for @command{dockerd}.
 
diff --git a/gnu/services/docker.scm b/gnu/services/docker.scm
index 741bab5a8c..e138a6be7e 100644
--- a/gnu/services/docker.scm
+++ b/gnu/services/docker.scm
@@ -5,6 +5,7 @@
 ;;; Copyright © 2020 Efraim Flashner <efraim@flashner.co.il>
 ;;; Copyright © 2020 Jesse Dowell <jessedowell@gmail.com>
 ;;; Copyright © 2021 Brice Waegeneire <brice@waegenei.re>
+;;; Copyright © 2023 Remco van 't Veer <remco@remworks.net>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -29,6 +30,7 @@ (define-module (gnu services docker)
   #:use-module (gnu services shepherd)
   #:use-module (gnu system setuid)
   #:use-module (gnu system shadow)
+  #:use-module (gnu packages admin)
   #:use-module (gnu packages docker)
   #:use-module (gnu packages linux)               ;singularity
   #:use-module (guix records)
@@ -62,6 +64,9 @@ (define-configuration docker-configuration
   (enable-iptables?
    (boolean #t)
    "Enable addition of iptables rules (enabled by default).")
+  (enable-userns-remap?
+   (boolean #f)
+   "Enable remapping and subordinate user and group IDs (disabled by default).")
   (environment-variables
    (list '())
    "Environment variables to set for dockerd")
@@ -107,6 +112,7 @@ (define (docker-shepherd-service config)
   (let* ((docker (docker-configuration-docker config))
          (enable-proxy? (docker-configuration-enable-proxy? config))
          (enable-iptables? (docker-configuration-enable-iptables? config))
+         (enable-userns-remap? (docker-configuration-enable-userns-remap? config))
          (environment-variables (docker-configuration-environment-variables config))
          (proxy (docker-configuration-proxy config))
          (debug? (docker-configuration-debug? config)))
@@ -135,6 +141,9 @@ (define (docker-shepherd-service config)
                                         #~(string-append
                                            "--userland-proxy-path=" #$proxy "/bin/proxy"))
                                   '("--userland-proxy=false"))
+                           #$@(if enable-userns-remap?
+                                  '("--userns-remap=dockremap")
+                                  '())
                            (if #$enable-iptables?
                                "--iptables"
                                "--iptables=false")
@@ -145,6 +154,18 @@ (define (docker-shepherd-service config)
                      #:log-file "/var/log/docker.log"))
            (stop #~(make-kill-destructor)))))
 
+(define %docker-remap-user-group
+  (user-group (name "dockremap")
+              (system? #t)))
+
+(define %docker-remap-user-account
+  (user-account (name "dockremap")
+                (group "dockremap")
+                (system? #t)
+                (comment "Docker user namespace remap user")
+                (home-directory "/var/empty")
+                (shell (file-append shadow "/sbin/nologin"))))
+
 (define docker-service-type
   (service-type (name 'docker)
                 (description "Provide capability to run Docker application
@@ -161,7 +182,12 @@ (define docker-service-type
                                        (list (containerd-shepherd-service config)
                                              (docker-shepherd-service config))))
                   (service-extension account-service-type
-                                     (const %docker-accounts))))
+                                     (lambda (config)
+                                       (if (docker-configuration-enable-userns-remap? config)
+                                           (cons* %docker-remap-user-group
+                                                  %docker-remap-user-account
+                                                  %docker-accounts)
+                                           %docker-accounts)))))
                 (default-value (docker-configuration))))
 
 \f

base-commit: 849286ba66c96534bddc04df1a47d5692cbc977e
-- 
2.40.1

Re: bug#55358: docker containers stopped when doing guix install or guix shell

[Remco van 't Veer]

Hi Csepp,

2023/05/20 00:29, Csepp:

> Remco van 't Veer <remco@remworks.net> writes:
>
>> Hi Maxim and Zimoun,
>>
>> 2023/02/09 13:26, Remco van 't Veer:
>>
>>> I think I know what is causing the issue.  Both the "standard" mysql and
>>> postgres containers use user-id 999 to run the database service (this
>>> seems like a common practice because the redis container is configured
>>> similarly).  That user-id is also configured as guixbuilder01 so I guess
>>> the guix daemon is killing those when processes when it finishes doing
>>> builds.
>>
>> I found a solution / workaround for this problem by using
>> "userns-remap".  This feature allows the remapping of uids and guids to
>> different ranges.  I tried it by hacking the required files into my
>> etc-directory and it works; guix no long kills my database containers.
>>
>> I'd like to add this feature to docker-service-type having a new
>> configuration option named enable-userns-remap? which introduces a new
>> user and group (both named dockremap) to do the remapping by adding some
>> configurable number to the uids and guids of the running container.  In
>> /etc/subuid and /etc/subgid it would look like:
>>
>>   dockremap:100000:65536
>>
>> See https://docs.docker.com/engine/security/userns-remap/ for
>> documentation about this.
>>
>> WDYT?
>>
>> Cheers,
>> Remco
>
> The rootless podman example that was shared a few months ago could be
> relevant to this, since that also adds a subuid/subgid mapping.

Thanks!  Borrowed that.

For future reference:

  https://lists.gnu.org/archive/html/guix-devel/2023-03/msg00176.html

Cheers,
Remco