Building a Docker image for GitLab-CI

Building a Docker image for GitLab-CI

[Ludovic Courtès]

Hello Guix!

Has anyone succeeded in building a Docker image suitable for use in
GitLab-CI?  I haven’t.  Here’s what I tried.

Initially, I built an image with ‘guix system image -t docker …’ but
that doesn’t work because then the image’s “entry point” is shepherd,
but shepherd never returns.  Thus, GitLab-CI would spawn the image and
eventually time out.

So I tried this instead:

  guix pack guix bash-minimal coreutils-minimal grep net-base \
    --save-provenance -S /bin=bin -S /share=share -S /etc=etc \
    -f docker --max-layers=100

… with ‘.gitlab-ci.yml’ doing something like this:

--8<---------------cut here---------------start------------->8---
build:
  image: registry.gitlab.inria.fr/…
  tags: ["ci.inria.fr", "linux"]
  before_script:
    - echo "nameserver 10.0.2.3 # XXX" > /etc/resolv.conf
    - guix archive --authorize < /share/guix/ci.guix.gnu.org.pub
    - guix archive --authorize < /share/guix/bordeaux.guix.gnu.org.pub
    - guix-daemon --disable-chroot &
  script:
    - guix shell -m manifest.scm -- rubber --pdf article.tex
  artifacts:
    paths:
      - article.pdf
--8<---------------cut here---------------end--------------->8---

Problem is, name resolution appears to fail in the container image; the
‘resolv.conf’ trick was a crude attempt to work around it, but it
failed.  I guess the problem is that I don’t know how GitLab-CI or
Docker is supposed to set up networking inside those containers.

Thoughts?

Neat tip to upload your Guix-built image to a registry: use Skopeo.

  guix shell skopeo -- skopeo login registry.gitlab.inria.fr
  guix shell skopeo -- skopeo copy \
    docker-archive:///gnu/store/…-docker-image.tar.gz \
    docker://registry.gitlab.inria.fr/… \
    --insecure-policy

(“Insecure policy”, what could possibly go wrong?)

Ludo’.

Re: Building a Docker image for GitLab-CI

[Suhail]

Ludovic Courtès <ludovic.courtes@inria.fr> writes:

> Initially, I built an image with ‘guix system image -t docker …’ but
> that doesn’t work because then the image’s “entry point” is shepherd,
> but shepherd never returns.

Did you try resetting the entrypoint in .gitlab-ci.yml using the
image:entrypoint keyword? [1]

[1]: <https://docs.gitlab.com/ee/ci/yaml/#imageentrypoint>

-- 
Suhail

This email is not an offer capable of acceptance, does not evidence an
intention to enter into an agreement, has no operative effect until a
definitive agreement is signed in writing by both parties, and that no
party should act in reliance on the email or any representations of the
sender until a definitive agreement is signed in writing by both
parties.

Re: Building a Docker image for GitLab-CI

[Andreas Enge]

Am Tue, Feb 13, 2024 at 11:31:28AM +0100 schrieb Ludovic Courtès:
> Has anyone succeeded in building a Docker image suitable for use in
> GitLab-CI?  I haven’t.  Here’s what I tried.

A colleague of mine just found this:
   https://gitlab.com/daym/guix-on-docker/

:-)

Andreas

Re: Building a Docker image for GitLab-CI

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 638 bytes --]

On Tue, Feb 13, 2024 at 11:31:28AM +0100, Ludovic Courtès wrote:
> Hello Guix!
> 
> Has anyone succeeded in building a Docker image suitable for use in
> GitLab-CI?  I haven’t.  Here’s what I tried.

In the past I used a script to install guix using the shell script and
then ran guix pull before building my package.  I suppose you could use
a Debian image and run 'guix pull' first before building something.

-- 
Efraim Flashner   <efraim@flashner.co.il>   רנשלפ םירפא
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Building a Docker image for GitLab-CI

[Ludovic Courtès]

Hi,

Efraim Flashner <efraim@flashner.co.il> skribis:

> In the past I used a script to install guix using the shell script and
> then ran guix pull before building my package.  I suppose you could use
> a Debian image and run 'guix pull' first before building something.

I could… but that’d be cheating.  :-)

Ludo’.

Re: Building a Docker image for GitLab-CI

[Reza Housseini]

Ludovic Courtès <ludovic.courtes@inria.fr> writes:

sorry forgot to include the list...

Hi Ludo

> Has anyone succeeded in building a Docker image suitable for use in
> GitLab-CI?

I normally do the following and it seems to work fine with our gitlab
instance:

registry=registry.gitlab.ost.ch:45023/sciceg/teaching/eeu_mlds
archive=$(guix time-machine -C channels.scm -- pack -f docker -S /bin=bin -S /lib=lib -S /share=share -m manifest.scm)
tag=$(docker load -i $archive)
docker tag ${tag##*"Loaded image: "} $registry
docker push $registry

what seems to be crucial is to add the following packages to the
manifest file:

"bash" 
"coreutils"
"git"

than I can use the image in the gitlab-ci.yml file

generate-exercises:
  image: registry.gitlab.ost.ch:45023/sciceg/teaching/eeu_mlds:latest
  ...

Hope that helps!

Best,
Reza

PS: It would be really nice if one could provide a docker tag directly
to `guix pack -f docker`

Re: Building a Docker image for GitLab-CI

[Ludovic Courtès]

Hi Reza,

Reza Housseini <reza.housseini@gmail.com> skribis:

>> Has anyone succeeded in building a Docker image suitable for use in
>> GitLab-CI?
>
> I normally do the following and it seems to work fine with our gitlab
> instance:
>
> registry=registry.gitlab.ost.ch:45023/sciceg/teaching/eeu_mlds
> archive=$(guix time-machine -C channels.scm -- pack -f docker -S /bin=bin -S /lib=lib -S /share=share -m manifest.scm)
> tag=$(docker load -i $archive)
> docker tag ${tag##*"Loaded image: "} $registry
> docker push $registry
>
> what seems to be crucial is to add the following packages to the
> manifest file:
>
> "bash" 
> "coreutils"
> "git"

That’s nice, but unless I’m mistaken, Guix is missing from the image,
right?

My goal would be to be able to use Guix within the image, so I can have
GitLab-CI spawn ‘guix build’ commands (or similar).

> PS: It would be really nice if one could provide a docker tag directly
> to `guix pack -f docker`

I think it’s already possible, see ‘guix pack --help-docker-format’.

Thanks,
Ludo’.

Re: Building a Docker image for GitLab-CI

[Andreas Enge]

Hello,

Am Tue, Jun 04, 2024 at 01:29:22PM +0200 schrieb Ludovic Courtès:
> My goal would be to be able to use Guix within the image, so I can have
> GitLab-CI spawn ‘guix build’ commands (or similar).

with a colleague we have set up such a system. He has started from a Debian
image and written a docker script to install Guix "manually", which provides
our base image. Then Gitlab CI creates a new image from a channels file in
the git repository we want to monitor. The command we use is
"guix system image -t docker" and not "guix pack"; I am not really familiar
with the second command, and as far as I understand the first one has the
difference of running the shepherd as the docker command, which in our case
starts the guix-daemon and a guix-build-coordinator-agent.
(One addition: We use the patch in
   https://issues.guix.gnu.org/70933
to enable chroot in the container and then run the container in privileged
mode.)

Maybe we should write up a little blog post once everything is settled.

Andreas

Re: Building a Docker image for GitLab-CI

[Ludovic Courtès]

Hi,

Andreas Enge <andreas@enge.fr> skribis:

> Am Tue, Jun 04, 2024 at 01:29:22PM +0200 schrieb Ludovic Courtès:
>> My goal would be to be able to use Guix within the image, so I can have
>> GitLab-CI spawn ‘guix build’ commands (or similar).
>
> with a colleague we have set up such a system. He has started from a Debian
> image and written a docker script to install Guix "manually", which provides
> our base image. Then Gitlab CI creates a new image from a channels file in
> the git repository we want to monitor. The command we use is
> "guix system image -t docker" and not "guix pack";

To be able to do that, you need to have Guix already up and running in
the Docker image you pass to GitLab-CI: that’s the problem I’m trying to
solve.

Or did I misunderstand?

(It’s possible to sidestep that problem for instance by having GitLab-CI
offload to a runner in a machine that you control, where Guix is
installed.  But I’m looking for a solution that could work
out-of-the-box on any GitLab-CI or similar instance.)

Thanks,
Ludo’.

Re: Building a Docker image for GitLab-CI

[Reza Housseini]

Ludovic Courtès <ludovic.courtes@inria.fr> writes:

> That’s nice, but unless I’m mistaken, Guix is missing from the image,
> right?
>
> My goal would be to be able to use Guix within the image, so I can have
> GitLab-CI spawn ‘guix build’ commands (or similar).

Oh sorry this was a misunderstanding from my side. Have you tried to use
the guix system image command to create a docker container?

Best,
Reza

Re: Building a Docker image for GitLab-CI

[Ludovic Courtès]

Reza Housseini <reza.housseini@gmail.com> skribis:

> Ludovic Courtès <ludovic.courtes@inria.fr> writes:
>
>> That’s nice, but unless I’m mistaken, Guix is missing from the image,
>> right?
>>
>> My goal would be to be able to use Guix within the image, so I can have
>> GitLab-CI spawn ‘guix build’ commands (or similar).
>
> Oh sorry this was a misunderstanding from my side. Have you tried to use
> the guix system image command to create a docker container?

I did, but that’s not suitable:

  https://mail.gnu.org/archive/html/help-guix/2024-02/msg00066.html

Ludo’.

Re: Building a Docker image for GitLab-CI

[Andreas Enge]

Am Thu, Jun 06, 2024 at 11:23:20AM +0200 schrieb Ludovic Courtès:
> (It’s possible to sidestep that problem for instance by having GitLab-CI
> offload to a runner in a machine that you control, where Guix is
> installed.  But I’m looking for a solution that could work
> out-of-the-box on any GitLab-CI or similar instance.)

It turns out that our Gitlab instance enables docker containers in
privileged mode, so it just happened to work in my case...

Andreas

Re: Building a Docker image for GitLab-CI

[Cayetano Santos]

[-- Attachment #1: Type: text/plain, Size: 439 bytes --]


I just setup a toy project, here:

    https://gitlab.com/csantosb/gitlabci

Here, I reproduce the steps to build a docker image using guix system
image, and set te entrypoint to "", as you can see.

Still, a piece in the puzzle is missing, ideas ?

    https://gitlab.com/csantosb/gitlabci/-/jobs/8648358846

--
Cayetano Santos
GnuPG Key:   https://meta.sr.ht/~csantosb.pgp
FingerPrint: CCB8 1842 F9D7 058E CD67 377A BF5C DF4D F6BF 6682

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 259 bytes --]

Re: Building a Docker image for GitLab-CI

[Cayetano Santos]

[-- Attachment #1: Type: text/plain, Size: 351 bytes --]


In the test-entry branch, I’m testing the trick in the guix-on-docker
repository, where the entrypoint is given a guix script.

    https://gitlab.com/csantosb/gitlabci/-/jobs/8648429390

Getting closer ...

--
Cayetano Santos
GnuPG Key:   https://meta.sr.ht/~csantosb.pgp
FingerPrint: CCB8 1842 F9D7 058E CD67 377A BF5C DF4D F6BF 6682

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 259 bytes --]

Re: Building a Docker image for GitLab-CI

[Simon Josefsson via]

[-- Attachment #1: Type: text/plain, Size: 6187 bytes --]

All,

I am trying to get a Guix container usable in GitLab, and thought I'd
share my status.  I have established working networking in the resulting
Guix container, which seems like progress (whoohoo!).  tl;dr:

https://gitlab.com/debdistutils/guix/container/-/jobs/8652014833

The problem seems to be that GitLab (I suppose they use docker?) picks
the wrong container file system from the resulting image.  It doesn't
use the same file system as a local 'podman run', see below.  Could this
be a 'guix pack' OCI container bug/problem?  Maybe 'guix pack' produces
too many blobs and GitLab/docker stops looking after while?  Or the
container configuration is incorrect?  Some bad interaction with 'podman
load -i'?  My OCI-fu is too weak here, help appreciated.

Here is my work:

https://gitlab.com/debdistutils/guix/container/
https://gitlab.com/debdistutils/guix/container/-/blob/main/.gitlab-ci.yml

(The system-* jobs are inspired by Cayetano's work, they aren't
necessary unless there is something wrong with the 'guix pack' approach
below...)

The idea is:

1) First job 'debian-with-guix' creates a Debian trixie container with
Guix and podman, and a 'guix pull'.  This is for caching, it takes 25
minutes to run on a large compute node, using a GitLab-local mirror of
Guix because Savannah access makes it even slower.

https://gitlab.com/debdistutils/guix/container/-/blob/main/debian-with-guix/Containerfile?ref_type=heads

https://gitlab.com/debdistutils/guix/container/-/jobs/8649009536

Output container (quite useful on its own):

registry.gitlab.com/debdistutils/guix/container:debian-with-guix

2) Second job 'pack' uses the 'debian-with-guix' image, runs a 'guix
pack' and creates a container using 'podman' and uploads it into:

registry.gitlab.com/debdistutils/guix/container:pack

I suspect the problem is happening in the 'guix pack' or 'podman load'
commands here.  Output from last successful run:

https://gitlab.com/debdistutils/guix/container/-/jobs/8649183646

3) Third job 'pack-test' tries to use the pack image in a GitLab job, as
a normal GitLab CI/CD build would work.

Job output is here:

https://gitlab.com/debdistutils/guix/container/-/jobs/8652014833

It fails with networking errors just like Ludo's earlier e-mail:

fping: icmp: unknown protocol

What is really weird is this root directory:

Using docker image sha256:57160f1c13ce56799d6e3e83dd97da4c929993ac008404ac38c67317cded25d1 for registry.gitlab.com/debdistutils/guix/container:pack with digest registry.gitlab.com/debdistutils/guix/container@sha256:be1ad3a7af69669cf3d138c6ec2b1201a64294aad33320246212c6689a1e5c9d ...
...
$ ls -la /etc
total 20
drwxr-xr-x 2 0 0 4096 Dec 16 10:15 .
drwxr-xr-x 1 0 0 4096 Dec 16 10:15 ..
-rw-r--r-- 1 0 0   46 Dec 16 10:15 hostname
-rw-r--r-- 1 0 0  283 Dec 16 10:15 hosts
lrwxrwxrwx 1 0 0   12 Dec 16 10:15 mtab -> /proc/mounts
-rw-r--r-- 1 0 0  841 Dec 16 10:15 resolv.conf

There is no /etc/protocols!  No wonder things doesn't work.

However if I run the same container locally, it looks different:

jas@kaka:~/src/guix-container$ podman images|grep pack
registry.gitlab.com/debdistutils/guix/container                   pack                  57160f1c13ce  54 years ago   1.23 GB
jas@kaka:~/src/guix-container$ podman run --entrypoint /bin/bash -it --rm registry.gitlab.com/debdistutils/guix/container:pack
bash-5.1# ls -la /etc
lrwxrwxrwx 1 0 0 55 Jan  1  1970 /etc -> /gnu/store/kcpr09dgqnr8q29d167cjk3wsmgizzkb-profile/etc
bash-5.1# ls -la /etc/
total 92
...
lrwxrwxrwx 1 0 0   70 Jan  1  1970 protocols -> /gnu/store/bfp25w47fxn8z0fdwj45prx2609sx59j-net-base-5.3/etc/protocols
...
bash-5.1# 

What is going on here?  Seems like GitLab docker is finding another
container file system compared to my local podman?  The hashes are the
same, so I should be working on the same file.

Does anyone have docker installed?  What does the equivalent of this
command do with docker?  Does it work like podman or GitLab?

podman run registry.gitlab.com/debdistutils/guix/container:pack ls -l /etc/

Networking seems to be set up fine, except the missing /etc/protocols:

$ ip address
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
17: eth0@if18: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1460 qdisc noqueue state UP group default 
    link/ether 02:42:ac:11:00:03 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 172.17.0.3/16 brd 172.17.255.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fd76:5338:4685:1:0:242:ac11:3/64 scope global nodad 
       valid_lft forever preferred_lft forever
    inet6 fe80::42:acff:fe11:3/64 scope link tentative 
       valid_lft forever preferred_lft forever
$ ip -4 route
default via 172.17.0.1 dev eth0 
172.17.0.0/16 dev eth0 proto kernel scope link src 172.17.0.3 
$ ip -6 route
fd76:5338:4685:1::/64 dev eth0 proto kernel metric 256 pref medium
fe80::/64 dev eth0 proto kernel metric 256 pref medium
default via fd76:5338:4685:1::1 dev eth0 metric 1024 pref medium
...
$ fping -c3 -D 130.237.72.201
fping: icmp: unknown protocol

Adding a small /etc/protocols makes networking work:

$ printf 'icmp\t1\tICMP\nipv6-icmp\t58\tIPv6-ICMP\n' > /etc/protocols
$ fping -c3 -D 130.237.72.201
[1734345646.83678] 130.237.72.201 : [0], 64 bytes, 112 ms (112 avg, 0% loss)
[1734345647.83701] 130.237.72.201 : [1], 64 bytes, 111 ms (112 avg, 0% loss)
[1734345648.83694] 130.237.72.201 : [2], 64 bytes, 111 ms (112 avg, 0% loss)
130.237.72.201 : xmt/rcv/%loss = 3/3/0%, min/avg/max = 111/112/112
$ fping -c3 -D gnu.org
[1734345648.90485] gnu.org : [0], 64 bytes, 25.1 ms (25.1 avg, 0% loss)
[1734345649.86358] gnu.org : [1], 64 bytes, 24.2 ms (24.6 avg, 0% loss)
[1734345650.86371] gnu.org : [2], 64 bytes, 24.3 ms (24.5 avg, 0% loss)
gnu.org : xmt/rcv/%loss = 3/3/0%, min/avg/max = 24.2/24.5/25.1

/Simon

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 255 bytes --]

Re: Building a Docker image for GitLab-CI

[Andreas Enge]

Hello Simon,

Am Mon, Dec 16, 2024 at 11:42:34AM +0100 schrieb Simon Josefsson via:
> I am trying to get a Guix container usable in GitLab, and thought I'd
> share my status.  I have established working networking in the resulting
> Guix container, which seems like progress (whoohoo!).  tl;dr:

at work we are using gitlab CI to build guix docker containers and run a
node on openshift for the bordeaux build farm:
   https://gitlab.inria.fr/enge/plm-guix
The README.md is completely outdated and serves mainly as a reminder to
myself on how this docker thing works; every time I look at it after a
break of a few months I have forgotten how to use a docker container...

And of course I have already forgotten the details; probably we should
write a little blog post. I will talk about it with my colleague when I
meet him next year ;-)

We also start with a Debian image and use a Dockerfile to install Guix
in it, as described in the Guix manual. Then for CI, we use this fixed
docker image to create a new one every time our repository (with a
channels.scm file and the plmshift.scm OS configuration file) changes.
In our case, this second docker image is the artefact that we then deploy.
We use "docker in docker" to create the images, and if I understood
correctly, this requires some privileges; these may not be given on
gitlab.com, but are available in our self-hosted instance.

Andreas