Building a Docker image for GitLab-CI

Building a Docker image for GitLab-CI

[Ludovic Courtès]

Hello Guix!

Has anyone succeeded in building a Docker image suitable for use in
GitLab-CI?  I haven’t.  Here’s what I tried.

Initially, I built an image with ‘guix system image -t docker …’ but
that doesn’t work because then the image’s “entry point” is shepherd,
but shepherd never returns.  Thus, GitLab-CI would spawn the image and
eventually time out.

So I tried this instead:

  guix pack guix bash-minimal coreutils-minimal grep net-base \
    --save-provenance -S /bin=bin -S /share=share -S /etc=etc \
    -f docker --max-layers=100

… with ‘.gitlab-ci.yml’ doing something like this:

--8<---------------cut here---------------start------------->8---
build:
  image: registry.gitlab.inria.fr/…
  tags: ["ci.inria.fr", "linux"]
  before_script:
    - echo "nameserver 10.0.2.3 # XXX" > /etc/resolv.conf
    - guix archive --authorize < /share/guix/ci.guix.gnu.org.pub
    - guix archive --authorize < /share/guix/bordeaux.guix.gnu.org.pub
    - guix-daemon --disable-chroot &
  script:
    - guix shell -m manifest.scm -- rubber --pdf article.tex
  artifacts:
    paths:
      - article.pdf
--8<---------------cut here---------------end--------------->8---

Problem is, name resolution appears to fail in the container image; the
‘resolv.conf’ trick was a crude attempt to work around it, but it
failed.  I guess the problem is that I don’t know how GitLab-CI or
Docker is supposed to set up networking inside those containers.

Thoughts?

Neat tip to upload your Guix-built image to a registry: use Skopeo.

  guix shell skopeo -- skopeo login registry.gitlab.inria.fr
  guix shell skopeo -- skopeo copy \
    docker-archive:///gnu/store/…-docker-image.tar.gz \
    docker://registry.gitlab.inria.fr/… \
    --insecure-policy

(“Insecure policy”, what could possibly go wrong?)

Ludo’.

Re: Building a Docker image for GitLab-CI

[Suhail]

Ludovic Courtès <ludovic.courtes@inria.fr> writes:

> Initially, I built an image with ‘guix system image -t docker …’ but
> that doesn’t work because then the image’s “entry point” is shepherd,
> but shepherd never returns.

Did you try resetting the entrypoint in .gitlab-ci.yml using the
image:entrypoint keyword? [1]

[1]: <https://docs.gitlab.com/ee/ci/yaml/#imageentrypoint>

-- 
Suhail

This email is not an offer capable of acceptance, does not evidence an
intention to enter into an agreement, has no operative effect until a
definitive agreement is signed in writing by both parties, and that no
party should act in reliance on the email or any representations of the
sender until a definitive agreement is signed in writing by both
parties.

Re: Building a Docker image for GitLab-CI

[Andreas Enge]

Am Tue, Feb 13, 2024 at 11:31:28AM +0100 schrieb Ludovic Courtès:
> Has anyone succeeded in building a Docker image suitable for use in
> GitLab-CI?  I haven’t.  Here’s what I tried.

A colleague of mine just found this:
   https://gitlab.com/daym/guix-on-docker/

:-)

Andreas

Re: Building a Docker image for GitLab-CI

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 638 bytes --]

On Tue, Feb 13, 2024 at 11:31:28AM +0100, Ludovic Courtès wrote:
> Hello Guix!
> 
> Has anyone succeeded in building a Docker image suitable for use in
> GitLab-CI?  I haven’t.  Here’s what I tried.

In the past I used a script to install guix using the shell script and
then ran guix pull before building my package.  I suppose you could use
a Debian image and run 'guix pull' first before building something.

-- 
Efraim Flashner   <efraim@flashner.co.il>   רנשלפ םירפא
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Building a Docker image for GitLab-CI

[Ludovic Courtès]

Hi,

Efraim Flashner <efraim@flashner.co.il> skribis:

> In the past I used a script to install guix using the shell script and
> then ran guix pull before building my package.  I suppose you could use
> a Debian image and run 'guix pull' first before building something.

I could… but that’d be cheating.  :-)

Ludo’.

Re: Building a Docker image for GitLab-CI

[Reza Housseini]

Ludovic Courtès <ludovic.courtes@inria.fr> writes:

sorry forgot to include the list...

Hi Ludo

> Has anyone succeeded in building a Docker image suitable for use in
> GitLab-CI?

I normally do the following and it seems to work fine with our gitlab
instance:

registry=registry.gitlab.ost.ch:45023/sciceg/teaching/eeu_mlds
archive=$(guix time-machine -C channels.scm -- pack -f docker -S /bin=bin -S /lib=lib -S /share=share -m manifest.scm)
tag=$(docker load -i $archive)
docker tag ${tag##*"Loaded image: "} $registry
docker push $registry

what seems to be crucial is to add the following packages to the
manifest file:

"bash" 
"coreutils"
"git"

than I can use the image in the gitlab-ci.yml file

generate-exercises:
  image: registry.gitlab.ost.ch:45023/sciceg/teaching/eeu_mlds:latest
  ...

Hope that helps!

Best,
Reza

PS: It would be really nice if one could provide a docker tag directly
to `guix pack -f docker`

Re: Building a Docker image for GitLab-CI

[Ludovic Courtès]

Hi Reza,

Reza Housseini <reza.housseini@gmail.com> skribis:

>> Has anyone succeeded in building a Docker image suitable for use in
>> GitLab-CI?
>
> I normally do the following and it seems to work fine with our gitlab
> instance:
>
> registry=registry.gitlab.ost.ch:45023/sciceg/teaching/eeu_mlds
> archive=$(guix time-machine -C channels.scm -- pack -f docker -S /bin=bin -S /lib=lib -S /share=share -m manifest.scm)
> tag=$(docker load -i $archive)
> docker tag ${tag##*"Loaded image: "} $registry
> docker push $registry
>
> what seems to be crucial is to add the following packages to the
> manifest file:
>
> "bash" 
> "coreutils"
> "git"

That’s nice, but unless I’m mistaken, Guix is missing from the image,
right?

My goal would be to be able to use Guix within the image, so I can have
GitLab-CI spawn ‘guix build’ commands (or similar).

> PS: It would be really nice if one could provide a docker tag directly
> to `guix pack -f docker`

I think it’s already possible, see ‘guix pack --help-docker-format’.

Thanks,
Ludo’.

Re: Building a Docker image for GitLab-CI

[Andreas Enge]

Hello,

Am Tue, Jun 04, 2024 at 01:29:22PM +0200 schrieb Ludovic Courtès:
> My goal would be to be able to use Guix within the image, so I can have
> GitLab-CI spawn ‘guix build’ commands (or similar).

with a colleague we have set up such a system. He has started from a Debian
image and written a docker script to install Guix "manually", which provides
our base image. Then Gitlab CI creates a new image from a channels file in
the git repository we want to monitor. The command we use is
"guix system image -t docker" and not "guix pack"; I am not really familiar
with the second command, and as far as I understand the first one has the
difference of running the shepherd as the docker command, which in our case
starts the guix-daemon and a guix-build-coordinator-agent.
(One addition: We use the patch in
   https://issues.guix.gnu.org/70933
to enable chroot in the container and then run the container in privileged
mode.)

Maybe we should write up a little blog post once everything is settled.

Andreas

Re: Building a Docker image for GitLab-CI

[Ludovic Courtès]

Hi,

Andreas Enge <andreas@enge.fr> skribis:

> Am Tue, Jun 04, 2024 at 01:29:22PM +0200 schrieb Ludovic Courtès:
>> My goal would be to be able to use Guix within the image, so I can have
>> GitLab-CI spawn ‘guix build’ commands (or similar).
>
> with a colleague we have set up such a system. He has started from a Debian
> image and written a docker script to install Guix "manually", which provides
> our base image. Then Gitlab CI creates a new image from a channels file in
> the git repository we want to monitor. The command we use is
> "guix system image -t docker" and not "guix pack";

To be able to do that, you need to have Guix already up and running in
the Docker image you pass to GitLab-CI: that’s the problem I’m trying to
solve.

Or did I misunderstand?

(It’s possible to sidestep that problem for instance by having GitLab-CI
offload to a runner in a machine that you control, where Guix is
installed.  But I’m looking for a solution that could work
out-of-the-box on any GitLab-CI or similar instance.)

Thanks,
Ludo’.

Re: Building a Docker image for GitLab-CI

[Reza Housseini]

Ludovic Courtès <ludovic.courtes@inria.fr> writes:

> That’s nice, but unless I’m mistaken, Guix is missing from the image,
> right?
>
> My goal would be to be able to use Guix within the image, so I can have
> GitLab-CI spawn ‘guix build’ commands (or similar).

Oh sorry this was a misunderstanding from my side. Have you tried to use
the guix system image command to create a docker container?

Best,
Reza

Re: Building a Docker image for GitLab-CI

[Ludovic Courtès]

Reza Housseini <reza.housseini@gmail.com> skribis:

> Ludovic Courtès <ludovic.courtes@inria.fr> writes:
>
>> That’s nice, but unless I’m mistaken, Guix is missing from the image,
>> right?
>>
>> My goal would be to be able to use Guix within the image, so I can have
>> GitLab-CI spawn ‘guix build’ commands (or similar).
>
> Oh sorry this was a misunderstanding from my side. Have you tried to use
> the guix system image command to create a docker container?

I did, but that’s not suitable:

  https://mail.gnu.org/archive/html/help-guix/2024-02/msg00066.html

Ludo’.

Re: Building a Docker image for GitLab-CI

[Andreas Enge]

Am Thu, Jun 06, 2024 at 11:23:20AM +0200 schrieb Ludovic Courtès:
> (It’s possible to sidestep that problem for instance by having GitLab-CI
> offload to a runner in a machine that you control, where Guix is
> installed.  But I’m looking for a solution that could work
> out-of-the-box on any GitLab-CI or similar instance.)

It turns out that our Gitlab instance enables docker containers in
privileged mode, so it just happened to work in my case...

Andreas