Guix in Docker for automated CD/CI of complex software

Guix in Docker for automated CD/CI of complex software

[Vicente Eduardo]

Hello Guix Developers.

I am developing a project called MetaCall
(https://github.com/metacall/core), which is a library that allows
calling functions between programming languages. In short, it is a
polyglot that allows importing code from other languages and using it
as normal code.

Since one year ago (or more), I have been trying to package it with no
success. With Docker it has been relatively easily to do, but Docker
is not a good way of distributing software like MetaCall. Because as
it is a C/C++ project (and library), it should be able to be
distributed for link against it as a shared or static library and not
as a Docker image.

One of the last attempts I have done is trying Guix, and it seems to
be the definitive package system. With it I have achieved to package
MetaCall with Python and generate a tarball with all dependencies self
contained. This is a perfect solution to me, because I can distribute
the complete polyglot with a single zip, allowing it to be installed
in any distribution or any other package manager (npm, pip, nuget or
gem).

In order to achieve this I have developed a Docker image of Guix
(https://github.com/metacall/guix) based on Alpine Linux. I work a lot
with Docker because I am using MetaCall to build a FaaS (Function as a
Service) (https://metacall.io), and Docker is a great tool nowadays in
Cloud. So I my idea was to provide a Docker image of Guix to allow
building MetaCall inside a CI/CD environment like Travis or any other
well know managed CD/CI platform with free-tier.

Current Guix Docker image is based on one of the latest version of
Docker and it uses buildx, an extension of Docker that implements
extra functionality by means of Buildkit. This is needed because this
image needs to spawn new processes meanwhile it is being built, as
Guix uses a daemon that does exactly this when running commands, so it
required to provide that privilege.

If you want to try the Docker image, just run:
docker pull metacall/guix
docker run -it metacall/guix /bin/sh

And then you can run any command, or you can execute commands directly
like this:
docker run -it metacall/guix guix graph coreutils

The store and profile folder can be shared too if you want to make it
persistent in your PC and install programs there:
docker run -v /gnu:/gnu -v $HOME/.guix-profile/:/root/.guix-profile/
-it metacall/guix guix package -i guile guile-cairo

I hope it is useful for someone. I using it to build the complete
polyglot with support for Python, Ruby, NodeJS, JavaScript (V8) and C#
NetCore, which are the main supported languages and/or run-times
(https://github.com/metacall/distributable/blob/53caad582cd097c5e92798bf0ec16660bfc4030c/source/metacall.scm).
Although it is being difficult because MetaCall's build system is a
bit complex and relies a lot in CMake features like ExternalProject
(not allowed in Guix as it downloads during build time), I already
achieved to build it with Python.

You can take metacall/distributable as an example of how to use
metacall/guix image. This will be eventually run in a CI/CD too. For
building you only have to do is to run make. After building everything
you will see a tarball-pack.tar.gz in the out directory outside of the
container. Here is the build script:
https://github.com/metacall/distributable/blob/53caad582cd097c5e92798bf0ec16660bfc4030c/scripts/build.sh
. I will merge feature/guix-build into the master soon.

Thanks for reading.
Vicente.

Re: Guix in Docker for automated CD/CI of complex software

[Hartmut Goebel]

Hi,


Am 21.11.19 um 15:41 schrieb Vicente Eduardo:
> In order to achieve this I have developed a Docker image of Guix
> (https://github.com/metacall/guix) based on Alpine Linux. I work a lot

I'm curious: Why are you not just using `guix system |docker-image` to
create the docker image directly?
|

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |

Re: Guix in Docker for automated CD/CI of complex software

[zimoun]

Dear,

Thank you for pointing this interesting project.


On Fri, 22 Nov 2019 at 15:23, Hartmut Goebel
<h.goebel@crazy-compilers.com> wrote:

> Am 21.11.19 um 15:41 schrieb Vicente Eduardo:
> > In order to achieve this I have developed a Docker image of Guix
> > (https://github.com/metacall/guix) based on Alpine Linux. I work a lot
>
> I'm curious: Why are you not just using `guix system |docker-image` to
> create the docker image directly?

I have not dived into the details but I am also curious about:
 - how do you package? I mean where is the scheme definition of
"metacall" when you do "guix build metacall ..."?
 - why do not use "guix pack -f docker" to generate your Docker image?
I am sure that I missing a lot of details. :-)


All the best,
simon

Re: Guix in Docker for automated CD/CI of complex software

[Vicente Eduardo]

[-- Attachment #1: Type: text/plain, Size: 1657 bytes --]

I don't want to create a docker image of my software from guix, I want to
run guix inside docker in order to have a portable installation of guix for
common ci/cd enviroments.

Here's the metacall schema:
https://github.com/metacall/distributable/blob/53caad582cd097c5e92798bf0ec16660bfc4030c/source/metacall.scm

My idea is to generate a zip with all libraries self contained, rather than
a Docker image.


El vie., 22 nov. 2019 15:23, Hartmut Goebel <h.goebel@crazy-compilers.com>
escribió:

> Hi,
>
>
> Am 21.11.19 um 15:41 schrieb Vicente Eduardo:
> > In order to achieve this I have developed a Docker image of Guix
> > (https://github.com/metacall/guix) based on Alpine Linux. I work a lot
>
> I'm curious: Why are you not just using `guix system |docker-image` to
> create the docker image directly?
> |
>
> --
> Regards
> Hartmut Goebel
>
> | Hartmut Goebel          | h.goebel@crazy-compilers.com               |
> | www.crazy-compilers.com | compilers which you thought are impossible |
>
>
>
El vie., 22 nov. 2019 15:23, Hartmut Goebel <h.goebel@crazy-compilers.com>
escribió:

> Hi,
>
>
> Am 21.11.19 um 15:41 schrieb Vicente Eduardo:
> > In order to achieve this I have developed a Docker image of Guix
> > (https://github.com/metacall/guix) based on Alpine Linux. I work a lot
>
> I'm curious: Why are you not just using `guix system |docker-image` to
> create the docker image directly?
> |
>
> --
> Regards
> Hartmut Goebel
>
> | Hartmut Goebel          | h.goebel@crazy-compilers.com               |
> | www.crazy-compilers.com | compilers which you thought are impossible |
>
>
>

[-- Attachment #2: Type: text/html, Size: 3218 bytes --]

Re: Guix in Docker for automated CD/CI of complex software

[zimoun]

Hi,

On Fri, 22 Nov 2019 at 19:57, Vicente Eduardo <vic798@gmail.com> wrote:

> [...] I want to run guix inside docker in order to have a portable installation of guix for common ci/cd enviroments.

I was suggesting that maybe you can use Guix to create this image. :-)

--8<---------------cut here---------------start------------->8---
$ guix describe
Generation 57   Nov 25 2019 14:26:15    (current)
  guix b5d4d5b
    repository URL: https://git.savannah.gnu.org/git/guix.git
    branch: master
    commit: b5d4d5b9bcf267fddd02fcc14b88eac0bebf979f
--8<---------------cut here---------------end--------------->8---

--8<---------------cut here---------------start------------->8---
$ guix pack            \
     -f docker       \
     -C none         \
     -S /bin=bin     \
     -S /lib=lib     \
     -S /share=share \
     -S /etc=etc     \
     guix
$ docker load < /gnu/store/8vrdpas5s9alpczixc8bw83x9sxc8if5-docker-pack.tar
--8<---------------cut here---------------end--------------->8---

--8<---------------cut here---------------start------------->8---
$ docker run -ti guix guix --version
--8<---------------cut here---------------end--------------->8---


Obviously, it needs some love (dameon) to fit your use case. And I do
not know if it make sense but I would try to replace this Dockerfile
[1] by a manifest and "guix pack". Or maybe as a first step using
Dockerfile based on Guix instead of Alpine.

[1] https://github.com/metacall/guix/blob/master/Dockerfile

And even, instead of the raw instruction, I would use the installation
script [2].

[2] https://git.savannah.gnu.org/cgit/guix.git/plain/etc/guix-install.sh


> Here's the metacall schema: https://github.com/metacall/distributable/blob/53caad582cd097c5e92798bf0ec16660bfc4030c/source/metacall.scm
>
> My idea is to generate a zip with all libraries self contained, rather than a Docker image.

The "guix pack" should fit your need. But I am not sure because it is
a simple cmake-build-system with few (guix) dependencies, i.e., it
"relies a lot in CMake features like ExternalProject".



All the best,
simon

Re: Guix in Docker for automated CD/CI of complex software

[Ludovic Courtès]

Hi Vincente & all,

zimoun <zimon.toutoune@gmail.com> skribis:

> I was suggesting that maybe you can use Guix to create this image. :-)
>
> $ guix describe
> Generation 57   Nov 25 2019 14:26:15    (current)
>   guix b5d4d5b
>     repository URL: https://git.savannah.gnu.org/git/guix.git
>     branch: master
>     commit: b5d4d5b9bcf267fddd02fcc14b88eac0bebf979f
>
> $ guix pack            \
>      -f docker       \
>      -C none         \
>      -S /bin=bin     \
>      -S /lib=lib     \
>      -S /share=share \
>      -S /etc=etc     \
>      guix
> $ docker load < /gnu/store/8vrdpas5s9alpczixc8bw83x9sxc8if5-docker-pack.tar
>
> $ docker run -ti guix guix --version
>
>
> Obviously, it needs some love (dameon) to fit your use case.

The way I’d do it is by doing:

  guix system docker-image config.scm

where ‘config.scm’ is something minimalist like
<https://git.savannah.gnu.org/cgit/guix.git/tree/gnu/system/examples/bare-bones.tmpl>.

That gives you a Docker image in which ‘guix-daemon’ is running, and
thus you can do “docker run guix build” (roughly) to build things in
that image.  See
<https://guix.gnu.org/manual/devel/en/html_node/Invoking-guix-system.html>.

Would that work for you, Vincente?

Anyhow, thanks for sharing.  I’ve used Guix on top of a “foreign distro”
for CI, and I think it’s pretty useful in that context!

Ludo’.

Re: Guix in Docker for automated CD/CI of complex software

[zimoun]

Hi Ludo,

On Tue, 26 Nov 2019 at 10:59, Ludovic Courtès <ludo@gnu.org> wrote:

> > Obviously, it needs some love (dameon) to fit your use case.
>
> The way I’d do it is by doing:
>
>   guix system docker-image config.scm

[...]

> Anyhow, thanks for sharing.  I’ve used Guix on top of a “foreign distro”
> for CI, and I think it’s pretty useful in that context!

I did not know. I always thought that "guix system" was only related
to Guix System (or SD or OS) and not to "foreign distro".
Amazing!
This adds stuff to my TODO list... because this deserves an entry in
the Cookbook. :-)


Thank you!

Cheers,
simon