Building around Guix, some observations

Building around Guix, some observations

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 6090 bytes --]

Hello Guix,

Over the last few months, I've been experimenting with setting up
packages, services and systems around the software used by GOV.UK.

This is not really related to Guix development, but I'd be very
interested if anyone has any feedback about the approach I've taken so
far, and if anyone knows of anything similar.

I've had two objectives with this, to learn more about Guix, and to make
my day to day work (which is working on the applications) easier, and
it feels like I've been doing quite well with both so far.

You can find the code here [1], which I'm using against a specific
revision of Guix with some patches that can be found here [2].

Specifically, the paragraphs below are about:
 - Bundler and gem packages
 - Using a specific revision of Guix
 - Shared networking in guix system container
 - Packaging Ruby on Rails applications
 - Configuration of service startup
 - Database connections and utilities

The (gds packages utils bundler) module includes utilities to extend a
package that contains a Gemfile and Gemfile.lock in the source, and
handles the downloading of the gem packages, and the installation in the
store. This is definitely not a great long term approach, but it
provided a quick way to get these services working.  I'd be interested
in hearing other approaches that people have taken when dealing with gem
packages and bundler?

As these modules are intended to be used with the specified version of
guix, there is a script guix-pre-inst-env that I believe does a good job
now of making it possible to do this regardless of the environment its
run in. To do this, guix environment is used twice, once to make the
specific version of guix available, and again to use that version of
guix to setup the desired environment. Then there are some changes to
environment variables. XDG_CONFIG_HOME is set to /dev/null, as this
prevents guix from loading package definitions, as this would lead to
nondeterministic behaviour. Also, rather than using the --pure argument
to guix environment, I ended up unsetting variables in bash as this made
it possible to pass through variable values by not unsetting them. Are
there any other approaches to using a specific revision of guix?

One of the earliest changes I made was to change the guix system
container command to share the host network in the same way as guix
environment does. I recently made an attempt of making this
configurable, like the guix environment --network option [3]. When
writing this, it felt quite brittle, but when I remember to add the
right option, it is very useful. Annoyingly, just adding and removing
the --network option is insufficient, as you can't have a
static-networking-service when you are sharing the host network, but,
something still needs to provide 'loopback, so I add a
pretend-loopback-service to the system by default. The differences
between building operating-systems to run in containers or not are
handled in some places, so the differences in networking services could
be handled similarly?

Many of the services are using the Ruby on Rails application
framework. I've had some difficulty trying to get these applications
running with the code still in the store, as they use the location of
the code to determine where to try to write data. At the moment, copying
everything out of the store, and then changing the paths in the files to
ensure that the store isn't used is a reliable way of making the
services work, but its not very elegant. Does anyone have examples of
packaging rails apps in a more elegant way?

Some of the systems that I have written so far, e.g. the
operating-system in (gds systems govuk publishing-e2e-tests) configure
the startup of some services by using a specific record
(service-startup-config). This is currently being used to hook in the
creation of databases and database users, as well as running scripts to
seed data before the service starts. Has anyone else had a similar need
to do this, as this might be a pattern that could be beneficial to apply
to services within Guix?

Within the (gds services utils databases) module hierarchy there are
modules for working with PostgreSQL, MongoDB, Redis and MYSQL/MariaDB
databases. For each, I created a record type to describe the details of
a connection to the database, and wrote a generic function
database-connection-config->environment-variables to get the environment
variables that usually correspond to that configuration. Also in these
modules are functions to create G-expressions for performing common
actions, e.g. ensure this user exists, load data from this file. I get
the feeling that both of these tools, the connection records, and the
functions for common operations might be generally applicable?


Looking forward, I'm interested in how I can utilise Guix and the store
for managing data, specifically dumps from databases. I've got some
crude packages working that can be used with the G-expressions for
loading data in to a database, but what is missing is a way to transform
the data in a modular way. For example, using Guix to orchestrate the
process of loading the data in to a database, then dumping it back out
in a different form. I'm not aware of any packages that use databases at
build time, but I'd be very interested in doing this. Package builds
using databases also has an additional benefit, as databases are often
used for running tests at build time. There are some utilities to make
this easier, I've used pg_virtualenv with Debian packages before, but as
Guix has a way to reason about services, some way to integrate that with
package build processes would be amazing.

So, I'd be very interested if anyone has anything to add on the points
I've mentioned, any feedback about the approach I've taken so far, and
if anyone knows of anything similar.

Thanks,

Chris

1:
https://github.com/alphagov/govuk-guix/tree/spike-systems-for-local-development
2:
https://github.com/alphagov/gnu-guix/tree/spike-local-development-system-container
3:
https://github.com/alphagov/gnu-guix/commit/ba5d6220a00143189411dc3fbf0c3e7eebc794ab

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 962 bytes --]

Re: Building around Guix, some observations

[Ludovic Courtès]

Hi Christopher,

Woow, that’s a lot of great work and a lot of questions!  :-)

Christopher Baines <mail@cbaines.net> skribis:

> Specifically, the paragraphs below are about:
>  - Bundler and gem packages
>  - Using a specific revision of Guix
>  - Shared networking in guix system container
>  - Packaging Ruby on Rails applications
>  - Configuration of service startup
>  - Database connections and utilities

It’s great to see that you’ve gone pretty far on these questions.  I’ll
skip the Ruby part since I’m ignorant (I think Dave Thompson uses Ruby
at work, so he might have ideas) and comment on the others.  Perhaps we
should start one thread per topic though.

> As these modules are intended to be used with the specified version of
> guix, there is a script guix-pre-inst-env that I believe does a good job
> now of making it possible to do this regardless of the environment its
> run in. To do this, guix environment is used twice, once to make the
> specific version of guix available, and again to use that version of
> guix to setup the desired environment. Then there are some changes to
> environment variables. XDG_CONFIG_HOME is set to /dev/null, as this
> prevents guix from loading package definitions, as this would lead to
> nondeterministic behaviour. Also, rather than using the --pure argument
> to guix environment, I ended up unsetting variables in bash as this made
> it possible to pass through variable values by not unsetting them. Are
> there any other approaches to using a specific revision of guix?

In commit 8a9cffb202414b20081910115ba76402924bdcdd Leo documented a
simple way of telling ‘guix pull’ how to get a specific revision.

The other possibility is of course to build Guix from Git, and do that
using ‘guix environment guix’, which seems to be essentially what you’re
doing.

The ultimate goal is to make it easier through the new ‘guix pull’
(discussed notably at <https://bugs.gnu.org/22629>.)  People should be
able to do:

  guix channel pull master COMMIT
  guix channel describe

Things like that.  Roughly, it should provide the same flexibility as
building from a Git checkout, but just as easy as running ‘guix pull’.

> One of the earliest changes I made was to change the guix system
> container command to share the host network in the same way as guix
> environment does. I recently made an attempt of making this
> configurable, like the guix environment --network option [3]. When
> writing this, it felt quite brittle, but when I remember to add the
> right option, it is very useful. Annoyingly, just adding and removing
> the --network option is insufficient, as you can't have a
> static-networking-service when you are sharing the host network, but,
> something still needs to provide 'loopback, so I add a
> pretend-loopback-service to the system by default. The differences
> between building operating-systems to run in containers or not are
> handled in some places, so the differences in networking services could
> be handled similarly?

Probably!  In (gnu system vm), ‘virtualized-operating-system’ returns an
OS config adjusted so that it can be used in a VM:

  http://git.savannah.gnu.org/cgit/guix.git/tree/gnu/system/vm.scm#n367

Perhaps we should provide a ‘container-operating-system’ that does
something similar, and have ‘guix system container’ automatically use
it?

> Some of the systems that I have written so far, e.g. the
> operating-system in (gds systems govuk publishing-e2e-tests) configure
> the startup of some services by using a specific record
> (service-startup-config). This is currently being used to hook in the
> creation of databases and database users, as well as running scripts to
> seed data before the service starts. Has anyone else had a similar need
> to do this, as this might be a pattern that could be beneficial to apply
> to services within Guix?

Usually services use “activation snippets” for pre-startup
initialization, like:

  http://git.savannah.gnu.org/cgit/guix.git/tree/gnu/services/databases.scm#n90

Would it be an option here?

Alternately, I imagine you could write your own initialization steps as
Shepherd services and introduce them as dependencies of the Shepherd
services that need to be started after this initialization step.  Does
that make sense?

> Within the (gds services utils databases) module hierarchy there are
> modules for working with PostgreSQL, MongoDB, Redis and MYSQL/MariaDB
> databases. For each, I created a record type to describe the details of
> a connection to the database, and wrote a generic function
> database-connection-config->environment-variables to get the environment
> variables that usually correspond to that configuration. Also in these
> modules are functions to create G-expressions for performing common
> actions, e.g. ensure this user exists, load data from this file. I get
> the feeling that both of these tools, the connection records, and the
> functions for common operations might be generally applicable?

The env. vars in
<https://github.com/alphagov/govuk-guix/blob/spike-systems-for-local-development/gds/services/utils/databases.scm>
could be added to /etc/environment by extending
‘session-environment-service-type’.

Things in
<https://github.com/alphagov/govuk-guix/blob/spike-systems-for-local-development/gds/services/utils/databases/postgresql.scm>
to initialize the PG database could be handled by
‘postgresql-service-type’, though perhaps it should be optional?

Databases are stateful by definition though, so there are limits as to
what can be done in a purely declarative way.

> Looking forward, I'm interested in how I can utilise Guix and the store
> for managing data, specifically dumps from databases. I've got some
> crude packages working that can be used with the G-expressions for
> loading data in to a database, but what is missing is a way to transform
> the data in a modular way. For example, using Guix to orchestrate the
> process of loading the data in to a database, then dumping it back out
> in a different form. I'm not aware of any packages that use databases at
> build time, but I'd be very interested in doing this. Package builds
> using databases also has an additional benefit, as databases are often
> used for running tests at build time. There are some utilities to make
> this easier, I've used pg_virtualenv with Debian packages before, but as
> Guix has a way to reason about services, some way to integrate that with
> package build processes would be amazing.

I guess I don’t know enough about these issues to provide meaningful
input.

Thanks for your feedback!

Ludo’.