Swineherd: Guix System container manager

Swineherd: Guix System container manager

[Ricardo Wurmus]

Hi there,

you know the Shepherd: it is an elegant service manager looking after a
herd of daemons.  Since it can be extended with Guile, I decided to do
just that to add an extra skill to the Shepherd, turning it into the
Swineherd.

The Swineherd is a manager of Guix System containers.  It is implemented
as a Shepherd service called “swineherd” that lets you manage Guix
System containers as Shepherd services.  These system container services
implement a few common features to bring up bridge networking, execute
commands inside the containers, peek at files, and get simple usage
stats.

Swineherd also provides an optional HTTP API server to launch, manage,
and destroy containers through an HTTP client.  This makes the Swineherd
useful for building web applications that launch Guix System containers
on demand.  It comes with an Info manual describing the assumptions, the
required setup, as well as the implemented commands and the HTTP API.

The Swineherd was designed to be used with Shepherd on foreign distros,
so it does not assume to be running on top of Guix System (for better or
worse).

You can find the code here:

    https://github.com/BIMSBbioinfo/swineherd

Of course the Swineherd is also available as a Guix package called
“swineherd”.

If this tickles your interests I’d be very happy to hear your comments
and incorporate patches to extend it beyond my original requirements.

-- 
Ricardo

Re: Swineherd: Guix System container manager

[Simon Tournier]

Hi Ricardo,

On Wed, 13 Sep 2023 at 11:06, Ricardo Wurmus <rekado@elephly.net> wrote:

[...]

>
> The Swineherd was designed to be used with Shepherd on foreign distros,
> so it does not assume to be running on top of Guix System (for better or
> worse).

Oh!  It is very promising.  Really cool!

> Of course the Swineherd is also available as a Guix package called
> “swineherd”.

Hum, I did (or adding guile):

    guix time-machine -q -- shell swineherd

Then I do not know what to do.  What are the basic steps for testing it?

Cheers,
simon

Re: Swineherd: Guix System container manager

[Katherine Cox-Buday]

On 9/13/23 3:06 AM, Ricardo Wurmus wrote:
> Hi there,
> 
> you know the Shepherd: it is an elegant service manager looking after a
> herd of daemons.  Since it can be extended with Guile, I decided to do
> just that to add an extra skill to the Shepherd, turning it into the
> Swineherd.
> 
> The Swineherd is a manager of Guix System containers.  It is implemented
> as a Shepherd service called “swineherd” that lets you manage Guix
> System containers as Shepherd services.  These system container services
> implement a few common features to bring up bridge networking, execute
> commands inside the containers, peek at files, and get simple usage
> stats.
> 
> Swineherd also provides an optional HTTP API server to launch, manage,
> and destroy containers through an HTTP client.  This makes the Swineherd
> useful for building web applications that launch Guix System containers
> on demand.  It comes with an Info manual describing the assumptions, the
> required setup, as well as the implemented commands and the HTTP API.
> 
> The Swineherd was designed to be used with Shepherd on foreign distros,
> so it does not assume to be running on top of Guix System (for better or
> worse).
> 
> You can find the code here:
> 
>      https://github.com/BIMSBbioinfo/swineherd
> 
> Of course the Swineherd is also available as a Guix package called
> “swineherd”.
> 
> If this tickles your interests I’d be very happy to hear your comments
> and incorporate patches to extend it beyond my original requirements.

This is really cool stuff! I'd love to see Guix continue to eat the 
dev-ops space, which, in my opinion, loves to stack complexity on top of 
complexity.

Can you share any ways you're currently using this?

Any plans/interest in hooking it into `guix deploy`?

Re: Swineherd: Guix System container manager

[Thompson, David]

Hey, nice job Ricardo! This looks awesome!

On Wed, Sep 13, 2023 at 11:29 AM Katherine Cox-Buday
<cox.katherine.e@gmail.com> wrote:
>
> On 9/13/23 3:06 AM, Ricardo Wurmus wrote:
> > Hi there,
> >
> > you know the Shepherd: it is an elegant service manager looking after a
> > herd of daemons.  Since it can be extended with Guile, I decided to do
> > just that to add an extra skill to the Shepherd, turning it into the
> > Swineherd.
> >
> > The Swineherd is a manager of Guix System containers.  It is implemented
> > as a Shepherd service called “swineherd” that lets you manage Guix
> > System containers as Shepherd services.  These system container services
> > implement a few common features to bring up bridge networking, execute
> > commands inside the containers, peek at files, and get simple usage
> > stats.
> >
> > Swineherd also provides an optional HTTP API server to launch, manage,
> > and destroy containers through an HTTP client.  This makes the Swineherd
> > useful for building web applications that launch Guix System containers
> > on demand.  It comes with an Info manual describing the assumptions, the
> > required setup, as well as the implemented commands and the HTTP API.
> >
> > The Swineherd was designed to be used with Shepherd on foreign distros,
> > so it does not assume to be running on top of Guix System (for better or
> > worse).
> >
> > You can find the code here:
> >
> >      https://github.com/BIMSBbioinfo/swineherd
> >
> > Of course the Swineherd is also available as a Guix package called
> > “swineherd”.
> >
> > If this tickles your interests I’d be very happy to hear your comments
> > and incorporate patches to extend it beyond my original requirements.
>
> This is really cool stuff! I'd love to see Guix continue to eat the
> dev-ops space, which, in my opinion, loves to stack complexity on top of
> complexity.

+1 to that. I did devops professionally for about 6 years and it
really is a mess.

> Can you share any ways you're currently using this?
>
> Any plans/interest in hooking it into `guix deploy`?

Hey now that's a cool idea!

Speaking of devops, I believe with swineherd it is now possible to
implement something I've wanted for many years: the Guix equivalent of
docker-compose.  It would be like an enhanced guix shell that also
starts up services needed for development like PostgreSQL or Redis or
whatever.

- Dave

Re: Swineherd: Guix System container manager

[Ricardo Wurmus]

> Can you share any ways you're currently using this?

We had been using this at work as part of a much bigger training
platform to launch customizable (but still reproducible) interactive
environments for workshops and courses.

An example is pre-configured RStudio installations with shared course
materials.  With guix.install (provided by r-guix-install) participants
could install any Guix packages from within their running R session.  We
also provided a web interface to customize the environment ahead of
time, and to save/restore the container root fs.

> Any plans/interest in hooking it into `guix deploy`?

Plans no, but interest yes!

-- 
Ricardo

Re: Swineherd: Guix System container manager

[Tanguy LE CARROUR]

Hi Simon,

Quoting Simon Tournier (2023-09-13 14:58:38)
> On Wed, 13 Sep 2023 at 11:06, Ricardo Wurmus <rekado@elephly.net> wrote:
> [...]
> >
> > The Swineherd was designed to be used with Shepherd on foreign distros,
> > so it does not assume to be running on top of Guix System (for better or
> > worse).
> 
> Oh!  It is very promising.  Really cool!
> 
> > Of course the Swineherd is also available as a Guix package called
> > “swineherd”.
> 
> Hum, I did (or adding guile):
> 
>     guix time-machine -q -- shell swineherd
> 
> Then I do not know what to do.  What are the basic steps for testing it?

I haven't tried it (yet), but I've read this: <https://github.com/BIMSBbioinfo/swineherd/blob/main/doc/swineherd.texi>.

Regards,

-- 
Tanguy

Re: Swineherd: Guix System container manager

[Ricardo Wurmus]

Simon Tournier <zimon.toutoune@gmail.com> writes:

>> Of course the Swineherd is also available as a Guix package called
>> “swineherd”.
>
> Hum, I did (or adding guile):
>
>     guix time-machine -q -- shell swineherd
>
> Then I do not know what to do.  What are the basic steps for testing it?

There are no executables.

The basic steps for setting things up are in the included manual.

You’ll first need to set up a root disk for all your disposable
container home directories.  See the “Getting Started” chapter in the
manual on instructions how to get this set up for testing.

You then launch a new Shepherd process with the included
shepherd-config.scm (it’s at $out/share/swineherd/shepherd-config.scm),
which registers the “swineherd” and “swineherd-http-api” services, and
starts the former.

You can then try the Shepherd actions documented in the “Command Line
Interface” section of the manual.

I haven’t tried loading the included shepherd-config.scm into my PID 1,
because I don’t need it and I don’t want to freeze my PID 1 due to bugs
:) But with a properly set GUILE_LOAD_PATH (including the Swineherd
modules) it should also work with PID 1.

-- 
Ricardo