Is anyone using `guix system container` in production?

Is anyone using `guix system container` in production?

[conjaroy]

I'm interested in deploying several system containers to a single cloud
VPS, and I had originally planned to build those via `guix system
docker-image`. Although Docker has some nice CLI tools for
starting/stopping/listing active containers, it occurs to me that an
alternative (`guix system container`) has at least one significant
advantage: containers come online in seconds, as opposed to the minutes it
takes to build and import a Docker image (or tens of minutes, if the build
host is a VM without /dev/kvm.) It might also be the case that using
/gnu/store for all containers is more disk-space-efficient than creating
self-contained Docker images for each one.

So I was wondering if anyone has experience running long-lived containers
built via `guix system container` in a production setting. Since I'm
running Guix on a foreign distro (Debian 10), it seems reasonable to build
a systemd service around the container script, but there may be pitfalls I
haven't considered:

# build container script and register it as a gc root with a well-known
name.
guix build --root=/home/guix/my-awesome-container $(guix system container
-d my-awesome-container.scm)

cat << EOF > /etc/systemd/system/my-awesome-container.service
[Unit]
Description=My Awesome Container

[Service]
ExecStart=/home/guix/my-awesome-container
TimeoutStopSec=30
StandardOutput=syslog
StandardError=syslog

[Install]
WantedBy=multi-user.target
EOF

Re: Is anyone using `guix system container` in production?

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 2456 bytes --]

We've switched from using systemd to manage guix containers and services
to using systemd user services to launch an instance of shepherd which
manages guix containers and services, with some custom sudo rules. As
far as using systemd and guix containers, here's one config that I still
have around¹

Our upgrade scheme was to run 'guix pull' about weekly and then restart
the container. Assuming it didn't break we'd let it ride. If it did
break then we'd have 'guix pull --roll-back' to roll-back and wait it
out or fix it.

On Wed, Jul 29, 2020 at 06:17:44PM -0400, conjaroy wrote:
> I'm interested in deploying several system containers to a single cloud
> VPS, and I had originally planned to build those via `guix system
> docker-image`. Although Docker has some nice CLI tools for
> starting/stopping/listing active containers, it occurs to me that an
> alternative (`guix system container`) has at least one significant
> advantage: containers come online in seconds, as opposed to the minutes it
> takes to build and import a Docker image (or tens of minutes, if the build
> host is a VM without /dev/kvm.) It might also be the case that using
> /gnu/store for all containers is more disk-space-efficient than creating
> self-contained Docker images for each one.
> 
> So I was wondering if anyone has experience running long-lived containers
> built via `guix system container` in a production setting. Since I'm
> running Guix on a foreign distro (Debian 10), it seems reasonable to build
> a systemd service around the container script, but there may be pitfalls I
> haven't considered:
> 
> # build container script and register it as a gc root with a well-known
> name.
> guix build --root=/home/guix/my-awesome-container $(guix system container
> -d my-awesome-container.scm)
> 
> cat << EOF > /etc/systemd/system/my-awesome-container.service
> [Unit]
> Description=My Awesome Container
> 
> [Service]
> ExecStart=/home/guix/my-awesome-container
> TimeoutStopSec=30
> StandardOutput=syslog
> StandardError=syslog
> 
> [Install]
> WantedBy=multi-user.target
> EOF

¹ http://git.genenetwork.org/guix-bioinformatics/guix-bioinformatics/src/branch/master/gn/services/bnw.service


-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Is anyone using `guix system container` in production?

[conjaroy]

Hi Efraim, thanks for sharing your experience. Was your change in order to
adopt more Guix-centric tools, or to address specific bugs/limitations of
systemd in the initial approach?

Jason


On Sun, Aug 2, 2020 at 4:35 AM Efraim Flashner <efraim@flashner.co.il>
wrote:

> We've switched from using systemd to manage guix containers and services
> to using systemd user services to launch an instance of shepherd which
> manages guix containers and services, with some custom sudo rules. As
> far as using systemd and guix containers, here's one config that I still
> have around¹
>
> Our upgrade scheme was to run 'guix pull' about weekly and then restart
> the container. Assuming it didn't break we'd let it ride. If it did
> break then we'd have 'guix pull --roll-back' to roll-back and wait it
> out or fix it.
>
> On Wed, Jul 29, 2020 at 06:17:44PM -0400, conjaroy wrote:
> > I'm interested in deploying several system containers to a single cloud
> > VPS, and I had originally planned to build those via `guix system
> > docker-image`. Although Docker has some nice CLI tools for
> > starting/stopping/listing active containers, it occurs to me that an
> > alternative (`guix system container`) has at least one significant
> > advantage: containers come online in seconds, as opposed to the minutes
> it
> > takes to build and import a Docker image (or tens of minutes, if the
> build
> > host is a VM without /dev/kvm.) It might also be the case that using
> > /gnu/store for all containers is more disk-space-efficient than creating
> > self-contained Docker images for each one.
> >
> > So I was wondering if anyone has experience running long-lived containers
> > built via `guix system container` in a production setting. Since I'm
> > running Guix on a foreign distro (Debian 10), it seems reasonable to
> build
> > a systemd service around the container script, but there may be pitfalls
> I
> > haven't considered:
> >
> > # build container script and register it as a gc root with a well-known
> > name.
> > guix build --root=/home/guix/my-awesome-container $(guix system container
> > -d my-awesome-container.scm)
> >
> > cat << EOF > /etc/systemd/system/my-awesome-container.service
> > [Unit]
> > Description=My Awesome Container
> >
> > [Service]
> > ExecStart=/home/guix/my-awesome-container
> > TimeoutStopSec=30
> > StandardOutput=syslog
> > StandardError=syslog
> >
> > [Install]
> > WantedBy=multi-user.target
> > EOF
>
> ¹
> http://git.genenetwork.org/guix-bioinformatics/guix-bioinformatics/src/branch/master/gn/services/bnw.service
>
>
> --
> Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
> GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
> Confidentiality cannot be guaranteed on emails sent or received unencrypted
>

Re: Is anyone using `guix system container` in production?

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 4416 bytes --]

I found the systemd approach actually worked fairly well. The downsides
were that the containers needed to be run as root and then have their
permissions dropped which wasn't always easy for me. I also didn't
really like using root systemd units to start user-specific services. We
tried to give each service or similar group of services a user which
started adding some overhead.

We're currently using one user named 'shepherd' who has as user systemd
service which starts GNU Shepherd as the shepherd user and runs all the
services, with the passwordless sudo help. The individual shepherd
services are a bit more complex to write than the simple systemd
services we had before, but when we upgrade to the next server we plan
on using Guix System so we wanted to make sure that it was all working
anyway. The repo for those services is here¹. The README is missing that
I had to enable linger for shepherd (something like systemctl
enable-linger shepherd) for the user systemd service to start. It's not
necessarily easier to setup but I've found it easier to manage.

¹ http://git.genenetwork.org/efraim/shepherd-services

On Sun, Aug 02, 2020 at 11:40:52AM -0400, conjaroy wrote:
> Hi Efraim, thanks for sharing your experience. Was your change in order to
> adopt more Guix-centric tools, or to address specific bugs/limitations of
> systemd in the initial approach?
> 
> Jason
> 
> 
> On Sun, Aug 2, 2020 at 4:35 AM Efraim Flashner <efraim@flashner.co.il>
> wrote:
> 
> > We've switched from using systemd to manage guix containers and services
> > to using systemd user services to launch an instance of shepherd which
> > manages guix containers and services, with some custom sudo rules. As
> > far as using systemd and guix containers, here's one config that I still
> > have around¹
> >
> > Our upgrade scheme was to run 'guix pull' about weekly and then restart
> > the container. Assuming it didn't break we'd let it ride. If it did
> > break then we'd have 'guix pull --roll-back' to roll-back and wait it
> > out or fix it.
> >
> > On Wed, Jul 29, 2020 at 06:17:44PM -0400, conjaroy wrote:
> > > I'm interested in deploying several system containers to a single cloud
> > > VPS, and I had originally planned to build those via `guix system
> > > docker-image`. Although Docker has some nice CLI tools for
> > > starting/stopping/listing active containers, it occurs to me that an
> > > alternative (`guix system container`) has at least one significant
> > > advantage: containers come online in seconds, as opposed to the minutes
> > it
> > > takes to build and import a Docker image (or tens of minutes, if the
> > build
> > > host is a VM without /dev/kvm.) It might also be the case that using
> > > /gnu/store for all containers is more disk-space-efficient than creating
> > > self-contained Docker images for each one.
> > >
> > > So I was wondering if anyone has experience running long-lived containers
> > > built via `guix system container` in a production setting. Since I'm
> > > running Guix on a foreign distro (Debian 10), it seems reasonable to
> > build
> > > a systemd service around the container script, but there may be pitfalls
> > I
> > > haven't considered:
> > >
> > > # build container script and register it as a gc root with a well-known
> > > name.
> > > guix build --root=/home/guix/my-awesome-container $(guix system container
> > > -d my-awesome-container.scm)
> > >
> > > cat << EOF > /etc/systemd/system/my-awesome-container.service
> > > [Unit]
> > > Description=My Awesome Container
> > >
> > > [Service]
> > > ExecStart=/home/guix/my-awesome-container
> > > TimeoutStopSec=30
> > > StandardOutput=syslog
> > > StandardError=syslog
> > >
> > > [Install]
> > > WantedBy=multi-user.target
> > > EOF
> >
> > ¹
> > http://git.genenetwork.org/guix-bioinformatics/guix-bioinformatics/src/branch/master/gn/services/bnw.service
> >
> >
> > --
> > Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
> > GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
> > Confidentiality cannot be guaranteed on emails sent or received unencrypted
> >

-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Is anyone using `guix system container` in production?

[conjaroy]

Thanks Efraim, that repo is a useful resource. As it happens, I've also
discovered your recent blog post on this topic, included here for posterity:

https://guix.gnu.org/blog/2020/gnu-shepherd-user-services/

I especially like the advice on modularizing the Shepherd init file.

Cheers,

Jason

On Mon, Aug 3, 2020 at 2:54 AM Efraim Flashner <efraim@flashner.co.il>
wrote:

> I found the systemd approach actually worked fairly well. The downsides
> were that the containers needed to be run as root and then have their
> permissions dropped which wasn't always easy for me. I also didn't
> really like using root systemd units to start user-specific services. We
> tried to give each service or similar group of services a user which
> started adding some overhead.
>
> We're currently using one user named 'shepherd' who has as user systemd
> service which starts GNU Shepherd as the shepherd user and runs all the
> services, with the passwordless sudo help. The individual shepherd
> services are a bit more complex to write than the simple systemd
> services we had before, but when we upgrade to the next server we plan
> on using Guix System so we wanted to make sure that it was all working
> anyway. The repo for those services is here¹. The README is missing that
> I had to enable linger for shepherd (something like systemctl
> enable-linger shepherd) for the user systemd service to start. It's not
> necessarily easier to setup but I've found it easier to manage.
>
> ¹ http://git.genenetwork.org/efraim/shepherd-services
>
> On Sun, Aug 02, 2020 at 11:40:52AM -0400, conjaroy wrote:
> > Hi Efraim, thanks for sharing your experience. Was your change in order
> to
> > adopt more Guix-centric tools, or to address specific bugs/limitations of
> > systemd in the initial approach?
> >
> > Jason
> >
> >
> > On Sun, Aug 2, 2020 at 4:35 AM Efraim Flashner <efraim@flashner.co.il>
> > wrote:
> >
> > > We've switched from using systemd to manage guix containers and
> services
> > > to using systemd user services to launch an instance of shepherd which
> > > manages guix containers and services, with some custom sudo rules. As
> > > far as using systemd and guix containers, here's one config that I
> still
> > > have around¹
> > >
> > > Our upgrade scheme was to run 'guix pull' about weekly and then restart
> > > the container. Assuming it didn't break we'd let it ride. If it did
> > > break then we'd have 'guix pull --roll-back' to roll-back and wait it
> > > out or fix it.
> > >
> > > On Wed, Jul 29, 2020 at 06:17:44PM -0400, conjaroy wrote:
> > > > I'm interested in deploying several system containers to a single
> cloud
> > > > VPS, and I had originally planned to build those via `guix system
> > > > docker-image`. Although Docker has some nice CLI tools for
> > > > starting/stopping/listing active containers, it occurs to me that an
> > > > alternative (`guix system container`) has at least one significant
> > > > advantage: containers come online in seconds, as opposed to the
> minutes
> > > it
> > > > takes to build and import a Docker image (or tens of minutes, if the
> > > build
> > > > host is a VM without /dev/kvm.) It might also be the case that using
> > > > /gnu/store for all containers is more disk-space-efficient than
> creating
> > > > self-contained Docker images for each one.
> > > >
> > > > So I was wondering if anyone has experience running long-lived
> containers
> > > > built via `guix system container` in a production setting. Since I'm
> > > > running Guix on a foreign distro (Debian 10), it seems reasonable to
> > > build
> > > > a systemd service around the container script, but there may be
> pitfalls
> > > I
> > > > haven't considered:
> > > >
> > > > # build container script and register it as a gc root with a
> well-known
> > > > name.
> > > > guix build --root=/home/guix/my-awesome-container $(guix system
> container
> > > > -d my-awesome-container.scm)
> > > >
> > > > cat << EOF > /etc/systemd/system/my-awesome-container.service
> > > > [Unit]
> > > > Description=My Awesome Container
> > > >
> > > > [Service]
> > > > ExecStart=/home/guix/my-awesome-container
> > > > TimeoutStopSec=30
> > > > StandardOutput=syslog
> > > > StandardError=syslog
> > > >
> > > > [Install]
> > > > WantedBy=multi-user.target
> > > > EOF
> > >
> > > ¹
> > >
> http://git.genenetwork.org/guix-bioinformatics/guix-bioinformatics/src/branch/master/gn/services/bnw.service
> > >
> > >
> > > --
> > > Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
> > > GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
> > > Confidentiality cannot be guaranteed on emails sent or received
> unencrypted
> > >
>
> --
> Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
> GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
> Confidentiality cannot be guaranteed on emails sent or received unencrypted
>