I found the systemd approach actually worked fairly well. The downsides
were that the containers needed to be run as root and then have their
permissions dropped which wasn't always easy for me. I also didn't
really like using root systemd units to start user-specific services. We
tried to give each service or similar group of services a user which
started adding some overhead.

We're currently using one user named 'shepherd' who has as user systemd
service which starts GNU Shepherd as the shepherd user and runs all the
services, with the passwordless sudo help. The individual shepherd
services are a bit more complex to write than the simple systemd
services we had before, but when we upgrade to the next server we plan
on using Guix System so we wanted to make sure that it was all working
anyway. The repo for those services is here¹. The README is missing that
I had to enable linger for shepherd (something like systemctl
enable-linger shepherd) for the user systemd service to start. It's not
necessarily easier to setup but I've found it easier to manage.

¹ http://git.genenetwork.org/efraim/shepherd-services

On Sun, Aug 02, 2020 at 11:40:52AM -0400, conjaroy wrote:
> Hi Efraim, thanks for sharing your experience. Was your change in order to
> adopt more Guix-centric tools, or to address specific bugs/limitations of
> systemd in the initial approach?
> 
> Jason
> 
> 
> On Sun, Aug 2, 2020 at 4:35 AM Efraim Flashner <efraim@flashner.co.il>
> wrote:
> 
> > We've switched from using systemd to manage guix containers and services
> > to using systemd user services to launch an instance of shepherd which
> > manages guix containers and services, with some custom sudo rules. As
> > far as using systemd and guix containers, here's one config that I still
> > have around¹
> >
> > Our upgrade scheme was to run 'guix pull' about weekly and then restart
> > the container. Assuming it didn't break we'd let it ride. If it did
> > break then we'd have 'guix pull --roll-back' to roll-back and wait it
> > out or fix it.
> >
> > On Wed, Jul 29, 2020 at 06:17:44PM -0400, conjaroy wrote:
> > > I'm interested in deploying several system containers to a single cloud
> > > VPS, and I had originally planned to build those via `guix system
> > > docker-image`. Although Docker has some nice CLI tools for
> > > starting/stopping/listing active containers, it occurs to me that an
> > > alternative (`guix system container`) has at least one significant
> > > advantage: containers come online in seconds, as opposed to the minutes
> > it
> > > takes to build and import a Docker image (or tens of minutes, if the
> > build
> > > host is a VM without /dev/kvm.) It might also be the case that using
> > > /gnu/store for all containers is more disk-space-efficient than creating
> > > self-contained Docker images for each one.
> > >
> > > So I was wondering if anyone has experience running long-lived containers
> > > built via `guix system container` in a production setting. Since I'm
> > > running Guix on a foreign distro (Debian 10), it seems reasonable to
> > build
> > > a systemd service around the container script, but there may be pitfalls
> > I
> > > haven't considered:
> > >
> > > # build container script and register it as a gc root with a well-known
> > > name.
> > > guix build --root=/home/guix/my-awesome-container $(guix system container
> > > -d my-awesome-container.scm)
> > >
> > > cat << EOF > /etc/systemd/system/my-awesome-container.service
> > > [Unit]
> > > Description=My Awesome Container
> > >
> > > [Service]
> > > ExecStart=/home/guix/my-awesome-container
> > > TimeoutStopSec=30
> > > StandardOutput=syslog
> > > StandardError=syslog
> > >
> > > [Install]
> > > WantedBy=multi-user.target
> > > EOF
> >
> > ¹
> > http://git.genenetwork.org/guix-bioinformatics/guix-bioinformatics/src/branch/master/gn/services/bnw.service
> >
> >
> > --
> > Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
> > GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
> > Confidentiality cannot be guaranteed on emails sent or received unencrypted
> >

-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted