Re: Building a Docker image for GitLab-CI

Re: Building a Docker image for GitLab-CI

[Suhail]

Ludovic Courtès <ludovic.courtes@inria.fr> writes:

> Initially, I built an image with ‘guix system image -t docker …’ but
> that doesn’t work because then the image’s “entry point” is shepherd,
> but shepherd never returns.

Did you try resetting the entrypoint in .gitlab-ci.yml using the
image:entrypoint keyword? [1]

[1]: <https://docs.gitlab.com/ee/ci/yaml/#imageentrypoint>

-- 
Suhail

This email is not an offer capable of acceptance, does not evidence an
intention to enter into an agreement, has no operative effect until a
definitive agreement is signed in writing by both parties, and that no
party should act in reliance on the email or any representations of the
sender until a definitive agreement is signed in writing by both
parties.

Re: Building a Docker image for GitLab-CI

[Cayetano Santos]

[-- Attachment #1: Type: text/plain, Size: 439 bytes --]


I just setup a toy project, here:

    https://gitlab.com/csantosb/gitlabci

Here, I reproduce the steps to build a docker image using guix system
image, and set te entrypoint to "", as you can see.

Still, a piece in the puzzle is missing, ideas ?

    https://gitlab.com/csantosb/gitlabci/-/jobs/8648358846

--
Cayetano Santos
GnuPG Key:   https://meta.sr.ht/~csantosb.pgp
FingerPrint: CCB8 1842 F9D7 058E CD67 377A BF5C DF4D F6BF 6682

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 259 bytes --]

Re: Building a Docker image for GitLab-CI

[Cayetano Santos]

[-- Attachment #1: Type: text/plain, Size: 351 bytes --]


In the test-entry branch, I’m testing the trick in the guix-on-docker
repository, where the entrypoint is given a guix script.

    https://gitlab.com/csantosb/gitlabci/-/jobs/8648429390

Getting closer ...

--
Cayetano Santos
GnuPG Key:   https://meta.sr.ht/~csantosb.pgp
FingerPrint: CCB8 1842 F9D7 058E CD67 377A BF5C DF4D F6BF 6682

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 259 bytes --]

Re: Building a Docker image for GitLab-CI

[Simon Josefsson via]

[-- Attachment #1: Type: text/plain, Size: 6187 bytes --]

All,

I am trying to get a Guix container usable in GitLab, and thought I'd
share my status.  I have established working networking in the resulting
Guix container, which seems like progress (whoohoo!).  tl;dr:

https://gitlab.com/debdistutils/guix/container/-/jobs/8652014833

The problem seems to be that GitLab (I suppose they use docker?) picks
the wrong container file system from the resulting image.  It doesn't
use the same file system as a local 'podman run', see below.  Could this
be a 'guix pack' OCI container bug/problem?  Maybe 'guix pack' produces
too many blobs and GitLab/docker stops looking after while?  Or the
container configuration is incorrect?  Some bad interaction with 'podman
load -i'?  My OCI-fu is too weak here, help appreciated.

Here is my work:

https://gitlab.com/debdistutils/guix/container/
https://gitlab.com/debdistutils/guix/container/-/blob/main/.gitlab-ci.yml

(The system-* jobs are inspired by Cayetano's work, they aren't
necessary unless there is something wrong with the 'guix pack' approach
below...)

The idea is:

1) First job 'debian-with-guix' creates a Debian trixie container with
Guix and podman, and a 'guix pull'.  This is for caching, it takes 25
minutes to run on a large compute node, using a GitLab-local mirror of
Guix because Savannah access makes it even slower.

https://gitlab.com/debdistutils/guix/container/-/blob/main/debian-with-guix/Containerfile?ref_type=heads

https://gitlab.com/debdistutils/guix/container/-/jobs/8649009536

Output container (quite useful on its own):

registry.gitlab.com/debdistutils/guix/container:debian-with-guix

2) Second job 'pack' uses the 'debian-with-guix' image, runs a 'guix
pack' and creates a container using 'podman' and uploads it into:

registry.gitlab.com/debdistutils/guix/container:pack

I suspect the problem is happening in the 'guix pack' or 'podman load'
commands here.  Output from last successful run:

https://gitlab.com/debdistutils/guix/container/-/jobs/8649183646

3) Third job 'pack-test' tries to use the pack image in a GitLab job, as
a normal GitLab CI/CD build would work.

Job output is here:

https://gitlab.com/debdistutils/guix/container/-/jobs/8652014833

It fails with networking errors just like Ludo's earlier e-mail:

fping: icmp: unknown protocol

What is really weird is this root directory:

Using docker image sha256:57160f1c13ce56799d6e3e83dd97da4c929993ac008404ac38c67317cded25d1 for registry.gitlab.com/debdistutils/guix/container:pack with digest registry.gitlab.com/debdistutils/guix/container@sha256:be1ad3a7af69669cf3d138c6ec2b1201a64294aad33320246212c6689a1e5c9d ...
...
$ ls -la /etc
total 20
drwxr-xr-x 2 0 0 4096 Dec 16 10:15 .
drwxr-xr-x 1 0 0 4096 Dec 16 10:15 ..
-rw-r--r-- 1 0 0   46 Dec 16 10:15 hostname
-rw-r--r-- 1 0 0  283 Dec 16 10:15 hosts
lrwxrwxrwx 1 0 0   12 Dec 16 10:15 mtab -> /proc/mounts
-rw-r--r-- 1 0 0  841 Dec 16 10:15 resolv.conf

There is no /etc/protocols!  No wonder things doesn't work.

However if I run the same container locally, it looks different:

jas@kaka:~/src/guix-container$ podman images|grep pack
registry.gitlab.com/debdistutils/guix/container                   pack                  57160f1c13ce  54 years ago   1.23 GB
jas@kaka:~/src/guix-container$ podman run --entrypoint /bin/bash -it --rm registry.gitlab.com/debdistutils/guix/container:pack
bash-5.1# ls -la /etc
lrwxrwxrwx 1 0 0 55 Jan  1  1970 /etc -> /gnu/store/kcpr09dgqnr8q29d167cjk3wsmgizzkb-profile/etc
bash-5.1# ls -la /etc/
total 92
...
lrwxrwxrwx 1 0 0   70 Jan  1  1970 protocols -> /gnu/store/bfp25w47fxn8z0fdwj45prx2609sx59j-net-base-5.3/etc/protocols
...
bash-5.1# 

What is going on here?  Seems like GitLab docker is finding another
container file system compared to my local podman?  The hashes are the
same, so I should be working on the same file.

Does anyone have docker installed?  What does the equivalent of this
command do with docker?  Does it work like podman or GitLab?

podman run registry.gitlab.com/debdistutils/guix/container:pack ls -l /etc/

Networking seems to be set up fine, except the missing /etc/protocols:

$ ip address
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
17: eth0@if18: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1460 qdisc noqueue state UP group default 
    link/ether 02:42:ac:11:00:03 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 172.17.0.3/16 brd 172.17.255.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fd76:5338:4685:1:0:242:ac11:3/64 scope global nodad 
       valid_lft forever preferred_lft forever
    inet6 fe80::42:acff:fe11:3/64 scope link tentative 
       valid_lft forever preferred_lft forever
$ ip -4 route
default via 172.17.0.1 dev eth0 
172.17.0.0/16 dev eth0 proto kernel scope link src 172.17.0.3 
$ ip -6 route
fd76:5338:4685:1::/64 dev eth0 proto kernel metric 256 pref medium
fe80::/64 dev eth0 proto kernel metric 256 pref medium
default via fd76:5338:4685:1::1 dev eth0 metric 1024 pref medium
...
$ fping -c3 -D 130.237.72.201
fping: icmp: unknown protocol

Adding a small /etc/protocols makes networking work:

$ printf 'icmp\t1\tICMP\nipv6-icmp\t58\tIPv6-ICMP\n' > /etc/protocols
$ fping -c3 -D 130.237.72.201
[1734345646.83678] 130.237.72.201 : [0], 64 bytes, 112 ms (112 avg, 0% loss)
[1734345647.83701] 130.237.72.201 : [1], 64 bytes, 111 ms (112 avg, 0% loss)
[1734345648.83694] 130.237.72.201 : [2], 64 bytes, 111 ms (112 avg, 0% loss)
130.237.72.201 : xmt/rcv/%loss = 3/3/0%, min/avg/max = 111/112/112
$ fping -c3 -D gnu.org
[1734345648.90485] gnu.org : [0], 64 bytes, 25.1 ms (25.1 avg, 0% loss)
[1734345649.86358] gnu.org : [1], 64 bytes, 24.2 ms (24.6 avg, 0% loss)
[1734345650.86371] gnu.org : [2], 64 bytes, 24.3 ms (24.5 avg, 0% loss)
gnu.org : xmt/rcv/%loss = 3/3/0%, min/avg/max = 24.2/24.5/25.1

/Simon

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 255 bytes --]

Re: Building a Docker image for GitLab-CI

[Andreas Enge]

Hello Simon,

Am Mon, Dec 16, 2024 at 11:42:34AM +0100 schrieb Simon Josefsson via:
> I am trying to get a Guix container usable in GitLab, and thought I'd
> share my status.  I have established working networking in the resulting
> Guix container, which seems like progress (whoohoo!).  tl;dr:

at work we are using gitlab CI to build guix docker containers and run a
node on openshift for the bordeaux build farm:
   https://gitlab.inria.fr/enge/plm-guix
The README.md is completely outdated and serves mainly as a reminder to
myself on how this docker thing works; every time I look at it after a
break of a few months I have forgotten how to use a docker container...

And of course I have already forgotten the details; probably we should
write a little blog post. I will talk about it with my colleague when I
meet him next year ;-)

We also start with a Debian image and use a Dockerfile to install Guix
in it, as described in the Guix manual. Then for CI, we use this fixed
docker image to create a new one every time our repository (with a
channels.scm file and the plmshift.scm OS configuration file) changes.
In our case, this second docker image is the artefact that we then deploy.
We use "docker in docker" to create the images, and if I understood
correctly, this requires some privileges; these may not be given on
gitlab.com, but are available in our self-hosted instance.

Andreas

Re: Building a Docker image for GitLab-CI

[Simon Josefsson via]

[-- Attachment #1: Type: text/plain, Size: 4359 bytes --]

Andreas Enge <andreas@enge.fr> writes:

> Hello Simon,
>
> Am Mon, Dec 16, 2024 at 11:42:34AM +0100 schrieb Simon Josefsson via:
>> I am trying to get a Guix container usable in GitLab, and thought I'd
>> share my status.  I have established working networking in the resulting
>> Guix container, which seems like progress (whoohoo!).  tl;dr:
>
> at work we are using gitlab CI to build guix docker containers and run a
> node on openshift for the bordeaux build farm:
>    https://gitlab.inria.fr/enge/plm-guix
> The README.md is completely outdated and serves mainly as a reminder to
> myself on how this docker thing works; every time I look at it after a
> break of a few months I have forgotten how to use a docker container...
>
> And of course I have already forgotten the details; probably we should
> write a little blog post. I will talk about it with my colleague when I
> meet him next year ;-)
>
> We also start with a Debian image and use a Dockerfile to install Guix
> in it, as described in the Guix manual. Then for CI, we use this fixed
> docker image to create a new one every time our repository (with a
> channels.scm file and the plmshift.scm OS configuration file) changes.
> In our case, this second docker image is the artefact that we then deploy.
> We use "docker in docker" to create the images, and if I understood
> correctly, this requires some privileges; these may not be given on
> gitlab.com, but are available in our self-hosted instance.

Hi Andreas!  This all sounds quite similar to what I'm doing, although
using a different software stack.  I'm reading through your work now,
after actually finishing my work which I've announced here:

https://blog.josefsson.org/2024/12/18/guix-container-images-for-gitlab-ci-cd/
https://gitlab.com/debdistutils/guix/container

Looking into details, it seems you run this command to create the image:

https://gitlab.inria.fr/enge/plm-guix/-/blob/bf87f970c316f20cea2cf80f2511a280b5a71ed8/.gitlab-ci.yml#L44

docker run --privileged -v ./config:/config $CI_REGISTRY_IMAGE/builder:latest sh -c 'cd /config && /guix-daemon.sh guix time-machine -C channels.scm -- system image -t docker plmshift.scm  >/dev/null 2>&1 && cat /gnu/store/*docker-image.tar.gz' > image/docker-image.tar.gz
docker load -i image/docker-image.tar.gz

Your docker file is here:

https://gitlab.inria.fr/enge/plm-guix/-/blob/master/docker/Dockerfile?ref_type=heads

The guix-daemon.sh script is here:

https://gitlab.inria.fr/enge/plm-guix/-/blob/master/docker/guix-daemon.sh?ref_type=heads

Your plmshift.scm file is here:

https://gitlab.inria.fr/enge/plm-guix/-/blob/master/config/plmshift.scm?ref_type=heads

For comparison, I'm creating the image like this:

https://gitlab.com/debdistutils/guix/container/-/blob/main/.gitlab-ci.yml?ref_type=heads#L61

GUIX_PACKS_SLIM: guix bash-minimal coreutils-minimal net-base lndir
GUIX_PACKS_LATEST: $GUIX_PACKS_SLIM git-minimal findutils diffutils gcc-toolchain make automake autoconf tar grep sed gawk m4 gzip xz bzip2 iproute2 inetutils libcap shadow wget nss-certs
...
pack=$(guix pack $GUIX_PACKS --save-provenance -S /bin=bin -S /share=share -f docker --image-tag=guix --max-layers=8 --verbosity=0)
podman load -i $pack

My containerfile is here:

https://gitlab.com/debdistutils/guix/container/-/blob/main/debian-with-guix/Containerfile?ref_type=heads

Some of the stuff you resolve by using guix-daemon.sh and guix system
image on the plmshift.scm I instead push onto the consumer of my work,
as in these instructions:

https://gitlab.com/debdistutils/guix/container#how-to-use

One difference is that you are using your previous image as a basis for
the next one, which means you are using native Guix to build the image,
whereas I'm using Debian+Guix to build the image.  There is no
fundamental reason for this in my approach, so I opened an issue about
it:

https://gitlab.com/debdistutils/guix/container/-/issues/1

One more fundamental issue is that you are using 'guix system image' and
I was inspired by Ludo's e-mail and used 'guix pack'.  My experiments
with system images were that they ended up being larger, but maybe that
can be resolved by removing stuff.  Are there any general thoughts on
which is better to use?  Guix system vs Guix pack?  I kind of like the
idea of adding on top off guix-pack rather than removing from
guix-system.

/Simon

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 255 bytes --]

Re: Building a Docker image for GitLab-CI

[Cayetano Santos via]

n<#secure method=pgpmime mode=sign>

Hi Andreas,

  Nice work ! I’ve been playing a bit with it, but failed miserably to
  use your image, as you can see:

    https://gitlab.com/csantosb/gitlabci/-/jobs/8681019372

  This is my dummy yaml file:

    https://gitlab.com/csantosb/gitlabci/-/blob/test-entry/.gitlab-ci.yml

  Do you have an example file on how to use the image ?

Thanks !

--
Cayetano Santos
GnuPG Key:   https://meta.sr.ht/~csantosb.pgp
FingerPrint: CCB8 1842 F9D7 058E CD67 377A BF5C DF4D F6BF 6682

Re: Building a Docker image for GitLab-CI

[Ludovic Courtès]

Hi Simon,

Simon Josefsson <simon@josefsson.org> skribis:

> https://gitlab.com/debdistutils/guix/container/-/blob/main/.gitlab-ci.yml

Yay, great that you went this far!

> It fails with networking errors just like Ludo's earlier e-mail:
>
> fping: icmp: unknown protocol
>
> What is really weird is this root directory:
>
> Using docker image sha256:57160f1c13ce56799d6e3e83dd97da4c929993ac008404ac38c67317cded25d1 for registry.gitlab.com/debdistutils/guix/container:pack with digest registry.gitlab.com/debdistutils/guix/container@sha256:be1ad3a7af69669cf3d138c6ec2b1201a64294aad33320246212c6689a1e5c9d ...
> ...
> $ ls -la /etc
> total 20
> drwxr-xr-x 2 0 0 4096 Dec 16 10:15 .
> drwxr-xr-x 1 0 0 4096 Dec 16 10:15 ..
> -rw-r--r-- 1 0 0   46 Dec 16 10:15 hostname
> -rw-r--r-- 1 0 0  283 Dec 16 10:15 hosts
> lrwxrwxrwx 1 0 0   12 Dec 16 10:15 mtab -> /proc/mounts
> -rw-r--r-- 1 0 0  841 Dec 16 10:15 resolv.conf
>
> There is no /etc/protocols!  No wonder things doesn't work.

And that’s in spite of you running ‘guix pack … net-base -S /etc=etc’.

Could it be that something in podman/Docker/GitLab-CI overrides /etc, or
overrides it specifically because it’s a symlink?  I’m not sure where to
look for that.

Ludo’.

Re: Building a Docker image for GitLab-CI

[Simon Josefsson via]

[-- Attachment #1: Type: text/plain, Size: 5858 bytes --]

Ludovic Courtès <ludovic.courtes@inria.fr> writes:

>> What is really weird is this root directory:
>>
>> Using docker image
>> sha256:57160f1c13ce56799d6e3e83dd97da4c929993ac008404ac38c67317cded25d1
>> for registry.gitlab.com/debdistutils/guix/container:pack with digest
>> registry.gitlab.com/debdistutils/guix/container@sha256:be1ad3a7af69669cf3d138c6ec2b1201a64294aad33320246212c6689a1e5c9d
>> ...
>> ...
>> $ ls -la /etc
>> total 20
>> drwxr-xr-x 2 0 0 4096 Dec 16 10:15 .
>> drwxr-xr-x 1 0 0 4096 Dec 16 10:15 ..
>> -rw-r--r-- 1 0 0   46 Dec 16 10:15 hostname
>> -rw-r--r-- 1 0 0  283 Dec 16 10:15 hosts
>> lrwxrwxrwx 1 0 0   12 Dec 16 10:15 mtab -> /proc/mounts
>> -rw-r--r-- 1 0 0  841 Dec 16 10:15 resolv.conf
>>
>> There is no /etc/protocols!  No wonder things doesn't work.
>
> And that’s in spite of you running ‘guix pack … net-base -S /etc=etc’.
>
> Could it be that something in podman/Docker/GitLab-CI overrides /etc, or
> overrides it specifically because it’s a symlink?  I’m not sure where to
> look for that.

Yes it seems like a GitLab/docker-specific problem since I don't get the
same /etc when running the generated image locally.

There is this interesting entry in /proc/mounts:

https://gitlab.com/debdistutils/guix/container/-/jobs/8652014833#L343

overlay / overlay rw,relatime,lowerdir=l/ZPVAK6UICUAWUUE4GUD6AYFCEM:l/HI6HL2SJWTZDQHPTA3SGR4PWNE:l/OAJJQ5NKJAJLIAOEWBNLI6WRNC:l/MRDSZ2V6PLTEQGMEDSOPX6FKEY:l/FG4SISAU6TZNHB6CQR5X5GNEJB:l/EZGDP6A5CMVPA5O6IKOOKPDMBE:l/DA5NZCY6NVIGU2X6U5XQQXV54M:l/P4MIVQ3I7VCYFTQ3AG6RCXZVW5:l/FGPCINKKCYRDHZI5BAAU7HEETW:l/MUYJFZRJLR4Z3BNBFRBRTGP4S5:l/UPGZHVDAILBRLLEH6T5RKWIFG7:l/YNTBNOTPU7QRK5K6RD63VSXG5Q:l/XUPHFGGN36OPHNU334M3V7HDXP:l/PVA2QRQE4D5MAKI6BMGVPETNZN:l/VTGADKUDL4KA4HPA2YCCQ2JQNI:l/WZHWY243PTUZTQIZJM26PVSYG4:l/7YLLGUSIRSUFPXIFI57F2UQSFQ:l/ZEOCYJR44JGRBRKGEM4AMWUHQE:l/YXFWBRXLFSBMYDDCTNODGJWWUV:l/NQK5YT5BWDWSTKFRB3KOGVGYLC:l/6CC2D5S3LSZOOKLULC2JJ5BUHG:l/OMQ7M7FSULZ2WHTQPIOIYP7HYQ:l/VRGMNYJNRPOBPP5IZCJR3YV7FP:l/5L2O6RAVRTGGTB7I2YKS6RMX64:l/57QINIJWW7ECMX3DKLCDL5UMUS:l/HTIS4VRXRVO24AWC6AYPQ4LPRG:l/DOTULUURRI6Z4XR2B4LPQTP33T:l/LZYLE6JUKQFJBIAEMBQMQIWZEE:l/2WEGBKQG6D3VAWLXJ5NCZPFGNP:l/QYVPGN6K2A3Y4VVVPKYLR5JVL7:l/IPS4YGXCRZ47O4AOKMZ4TYD2N3:l/2MZU46ZBHYS5IQI4NAVFD3PNYR:l/HW6B6GDN33YB7GBY3LEOW2XXB4:l/FRINUFWGYICMVPLOJULIHQ3XKV:l/HSHVRY3DQIT5LUS3EONQQCKNL3:l/CRUDGYNQRSSDKB4TYALBBVFIL3:l/QJRZZB6NOMXWO46YCAJ4U53VSH:l/BZPO5MYEYX3YFX5NXYR32E6VRM:l/FDWSYXJR7RNKG42AMXGSC6NUQE:l/Z6BTLASGE6ZXGQZUFIHG4QXQLU:l/DWTG7Y6N4DNP5AZLI6MIDZEBHQ:l/TY4DSPRKETTLFE5WB7KFBO2VSM:l/SAE7PNZP6FFK2NVDOQQTZMO2BV:l/ZCL7CRORSVNYVFKNW2WT7TMGFZ:l/KVH3MQTKJ6SH46B2FEHW7UCYGK:l/XMDKZU7KD755BFINQOBKPLKMZQ:l/MFKZPIVLWDKG5PIVI3UQUU3ILT:l/I24GPDX2SRV3Y4YSJDYVKEBDQE:l/W6OZHVZW2NCSQOJEGMP45P2D6W:l/ZD4RI6B4WQ65QO7EMDQZSCZFHS:l/ZS7D35LTVLE6NSGCCY4SQQANE5:l/2ZIJ6PAUHDPBOXR72A6HGU6L4B:l/ZL5XOQ6XMF6ZYQSXVAK7744SX7:l/A3WXIZD22NL62JYHBZVL5K36IK:l/KUDHAVWVBDXKHHQQVF33KDPHF4:l/6O5OF37ORZTEUV2JXNOPO3WNTQ:l/YVGTC6PVMDS2GVOF57TJWCN5DM:l/VYM3JKFOWAY7UIYGU6TPJTZVD5:l/2BT7MWUS2JMZMQXQ3MRLO5AH6I:l/CNZTAOCVMGIFCNP4IF77OQ5MU6:l/KQCUQ7H6EY7U423TWBX6N6QNU7:l/VNZNN2P4U26XHEDRSNIQ656GRE:l/7YG6BVIVDJYCCGLISSU4APAR4S:l/ELR3M2R3NLUI4U2YCKYKG6MWUV:l/PK34AGBR7JY4PUJIVEAO4J7UAK:l/ELIDWT3IMYDRR5L5VTA3REN3LH:l/DK7VEQCTNIWW4BOYCOVXZX2GQY:l/6BCYXFR5B6S3EYCMVCXKQPEP3M:l/GAF6PUKMPKABSXDZCMVE3NM2K5:l/ASVMZMXSKHAVGH5UFXXRFE7TUX:l/25QORLMIGZEEIBQGTV6UBNZEAH:l/FH5SA2MBXXRRAMDYI72FPK7RXU:l/77IRT3TXX3H7XH66YGR7O5AIYK:l/FIQQSP7XQLUH3IWBXF4DZXWTFN:l/NG6ZAASCTOH6SQVUBR2FR6YE4T:l/QAHEWNHBILTWWWJ4QMX4ISWIT4:l/VCSGZSH4SQRVHK4EWSLGFECG2S:l/7FAMLBB6DJW7VWYEOIN6FLBI2E:l/C4ZXSOLVY36PYMTRL2E2YIKTKY:l/NVH5IIZJ5SO422GGMFGNTFCAOA:l/E6EE3L3EB2B36E5A5PL5KF32FL:l/RPXH632CD3Y54UYAMMULLEZOE4:l/662KE3GHNLWZEUTFCRZXOKVA3P:l/3ZDSIX7ZFSLT2FIEYD6TB6GA7R:l/ZS7SJOXF7XF6OFTLVHYMRYZANK:l/F3CEOKDDPOK72QLUWGKHDIJTPG:l/65CFVU5ZFM4XVSTLDO2WGQE3GU:l/3C2OJ5ZVICH5QP733HTS2DHPJM:l/B37PUETDZKC3MUZZX6CT3M6ZEC:l/LTVIXAHIS7O45NFIVNPDVGYMR4:l/GGBS4BWCAD6UMV3755MJ3BQRI5:l/4XGXVIO5QQ2ARZTTG74M4UDK4V:l/X673P4Q5TGPWBLQIXOQ5JXKG2C:l/3K2Q6NGZ5EEIKCFBRMEWWZFWQK:l/ALTG2QI6YZVKXIP4UCRDMQWS5Q:l/P4AE6X6G5GK66GHHRBZ27BTBF3:l/2UEJWLKTB2GNGT2G6UWFZVDFRN:l/OCTYS4BHPFX4HC3OXJJVGABEPX:l/M766VP6Y5QL2OGI6EYO5OEB6H4:l/DNITYWJFFEVGWQPFQ37S57EIB5:l/PQE5T2ER2OKPNC2YNHZEJXVIY6,upperdir=98dac307f50ae5da4c8f2cc5fdf024465f63600e30fc3bb434fca31191e2efdc/diff,workdir=98dac307f50ae5da4c8f2cc5fdf024465f63600e30fc3bb434fca31191e2efdc/work 0 0

Could those entries correspond to blobs generated by 'guix pack'?

Compare

https://gitlab.com/debdistutils/guix/container/-/jobs/8649183646#L130

$ podman load -i /gnu/store/*-docker-pack.tar.gz
Getting image source signatures
Copying blob sha256:26c7e7107d11a712095a4bf12ff26c8f39fb86c347af15ad50bec4d9536a4144
Copying blob sha256:3c21466d0d8255e7a1dfcbc206c891fdf6cdd6241f461cdb038ca6ef7b508bce
...
Copying blob sha256:b6ead463213fb7ec39911848da3b34e404ed184ee48373737c8b2eb2abd0730a
Copying blob sha256:72ca44ded2f166add396cc6a890a5e8a19c182603e4528a9bfef3012ab59b6b5
Copying blob sha256:092becdd45260f6b3d07626b2a39a738cc0a2d5a1c9f4a000cd61e762da8fe1b
Copying blob sha256:5397467e8b6a3b911d0e61f722622b995850be0721121a8e36b26ca7037b2622
Copying blob sha256:e3646d587f1665642b0077dc27a60b3dfed78dc21aa99eca4391b56c754f4aa7
Copying blob sha256:bff8143cb75389795d58b8a9ddbb572496e2e2ea1369a7f025a6b6e15b3a8074
Copying config sha256:57160f1c13ce56799d6e3e83dd97da4c929993ac008404ac38c67317cded25d1
Writing manifest to image destination
Loaded image: localhost/guix-bash-minimal-coreutils-minimal-git:latest

Notice that there are many more 'Copying blob' lines than overlay mounts
above.  Are we just seeing overlayfs mount truncation here?

Is it possible to make 'guix pack' create a merged container instead of
all these layers?

I'll experiment a bit more...

/Simon

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 255 bytes --]

Re: Building a Docker image for GitLab-CI

[Ludovic Courtès]

Simon Josefsson <simon@josefsson.org> skribis:

> Yes it seems like a GitLab/docker-specific problem since I don't get the
> same /etc when running the generated image locally.
>
> There is this interesting entry in /proc/mounts:
>
> https://gitlab.com/debdistutils/guix/container/-/jobs/8652014833#L343
>
> overlay / overlay rw,relatime,lowerdir=l/ZPVAK6UICUAWUUE4GUD6AYFCEM:l/HI6HL2SJWTZDQHPTA3SGR4PWNE:l/OAJJQ5NKJAJLIAOEWBNLI6WRNC:l/MRDSZ2V6PLTEQGMEDSOPX6FKEY:l/FG4SISAU6TZNHB6CQR5X5GNEJB:l/EZGDP6A5CMVPA5O6IKOOKPDMBE:l/DA5NZCY6NVIGU2X6U5XQQXV54M:l/P4MIVQ3I7VCYFTQ3AG6RCXZVW5:l/FGPCINKKCYRDHZI5BAAU7HEETW:l/MUYJFZRJLR4Z3BNBFRBRTGP4S5:l/UPGZHVDAILBRLLEH6T5RKWIFG7:l/YNTBNOTPU7QRK5K6RD63VSXG5Q:l/XUPHFGGN36OPHNU334M3V7HDXP:l/PVA2QRQE4D5MAKI6BMGVPETNZN:l/VTGADKUDL4KA4HPA2YCCQ2JQNI:l/WZHWY243PTUZTQIZJM26PVSYG4:l/7YLLGUSIRSUFPXIFI57F2UQSFQ:l/ZEOCYJR44JGRBRKGEM4AMWUHQE:l/YXFWBRXLFSBMYDDCTNODGJWWUV:l/NQK5YT5BWDWSTKFRB3KOGVGYLC:l/6CC2D5S3LSZOOKLULC2JJ5BUHG:l/OMQ7M7FSULZ2WHTQPIOIYP7HYQ:l/VRGMNYJNRPOBPP5IZCJR3YV7FP:l/5L2O6RAVRTGGTB7I2YKS6RMX64:l/57QINIJWW7ECMX3DKLCDL5UMUS:l/HTIS4VRXRVO24AWC6AYPQ4LPRG:l/DOTULUURRI6Z4XR2B4LPQTP33T:l/LZYLE6JUKQFJBIAEMBQMQIWZEE:l/2WEGBKQG6D3VAWLXJ5NCZPFGNP:l/QYVPGN6K2A3Y4VVVPKYLR5JVL7:l/IPS4YGXCRZ47O4AOKMZ4TYD2N3:l/2MZU46ZBHYS5IQI4NAVFD3PNYR:l/HW6B6GDN33YB7GBY3LEOW2XXB4:l/FRINUFWGYICMVPLOJULIHQ3XKV:l/HSHVRY3DQIT5LUS3EONQQCKNL3:l/CRUDGYNQRSSDKB4TYALBBVFIL3:l/QJRZZB6NOMXWO46YCAJ4U53VSH:l/BZPO5MYEYX3YFX5NXYR32E6VRM:l/FDWSYXJR7RNKG42AMXGSC6NUQE:l/Z6BTLASGE6ZXGQZUFIHG4QXQLU:l/DWTG7Y6N4DNP5AZLI6MIDZEBHQ:l/TY4DSPRKETTLFE5WB7KFBO2VSM:l/SAE7PNZP6FFK2NVDOQQTZMO2BV:l/ZCL7CRORSVNYVFKNW2WT7TMGFZ:l/KVH3MQTKJ6SH46B2FEHW7UCYGK:l/XMDKZU7KD755BFINQOBKPLKMZQ:l/MFKZPIVLWDKG5PIVI3UQUU3ILT:l/I24GPDX2SRV3Y4YSJDYVKEBDQE:l/W6OZHVZW2NCSQOJEGMP45P2D6W:l/ZD4RI6B4WQ65QO7EMDQZSCZFHS:l/ZS7D35LTVLE6NSGCCY4SQQANE5:l/2ZIJ6PAUHDPBOXR72A6HGU6L4B:l/ZL5XOQ6XMF6ZYQSXVAK7744SX7:l/A3WXIZD22NL62JYHBZVL5K36IK:l/KUDHAVWVBDXKHHQQVF33KDPHF4:l/6O5OF37ORZTEUV2JXNOPO3WNTQ:l/YVGTC6PVMDS2GVOF57TJWCN5DM:l/VYM3JKFOWAY7UIYGU6TPJTZVD5:l/2BT7MWUS2JMZMQXQ3MRLO5AH6I:l/CNZTAOCVMGIFCNP4IF77OQ5MU6:l/KQCUQ7H6EY7U423TWBX6N6QNU7:l/VNZNN2P4U26XHEDRSNIQ656GRE:l/7YG6BVIVDJYCCGLISSU4APAR4S:l/ELR3M2R3NLUI4U2YCKYKG6MWUV:l/PK34AGBR7JY4PUJIVEAO4J7UAK:l/ELIDWT3IMYDRR5L5VTA3REN3LH:l/DK7VEQCTNIWW4BOYCOVXZX2GQY:l/6BCYXFR5B6S3EYCMVCXKQPEP3M:l/GAF6PUKMPKABSXDZCMVE3NM2K5:l/ASVMZMXSKHAVGH5UFXXRFE7TUX:l/25QORLMIGZEEIBQGTV6UBNZEAH:l/FH5SA2MBXXRRAMDYI72FPK7RXU:l/77IRT3TXX3H7XH66YGR7O5AIYK:l/FIQQSP7XQLUH3IWBXF4DZXWTFN:l/NG6ZAASCTOH6SQVUBR2FR6YE4T:l/QAHEWNHBILTWWWJ4QMX4ISWIT4:l/VCSGZSH4SQRVHK4EWSLGFECG2S:l/7FAMLBB6DJW7VWYEOIN6FLBI2E:l/C4ZXSOLVY36PYMTRL2E2YIKTKY:l/NVH5IIZJ5SO422GGMFGNTFCAOA:l/E6EE3L3EB2B36E5A5PL5KF32FL:l/RPXH632CD3Y54UYAMMULLEZOE4:l/662KE3GHNLWZEUTFCRZXOKVA3P:l/3ZDSIX7ZFSLT2FIEYD6TB6GA7R:l/ZS7SJOXF7XF6OFTLVHYMRYZANK:l/F3CEOKDDPOK72QLUWGKHDIJTPG:l/65CFVU5ZFM4XVSTLDO2WGQE3GU:l/3C2OJ5ZVICH5QP733HTS2DHPJM:l/B37PUETDZKC3MUZZX6CT3M6ZEC:l/LTVIXAHIS7O45NFIVNPDVGYMR4:l/GGBS4BWCAD6UMV3755MJ3BQRI5:l/4XGXVIO5QQ2ARZTTG74M4UDK4V:l/X673P4Q5TGPWBLQIXOQ5JXKG2C:l/3K2Q6NGZ5EEIKCFBRMEWWZFWQK:l/ALTG2QI6YZVKXIP4UCRDMQWS5Q:l/P4AE6X6G5GK66GHHRBZ27BTBF3:l/2UEJWLKTB2GNGT2G6UWFZVDFRN:l/OCTYS4BHPFX4HC3OXJJVGABEPX:l/M766VP6Y5QL2OGI6EYO5OEB6H4:l/DNITYWJFFEVGWQPFQ37S57EIB5:l/PQE5T2ER2OKPNC2YNHZEJXVIY6,upperdir=98dac307f50ae5da4c8f2cc5fdf024465f63600e30fc3bb434fca31191e2efdc/diff,workdir=98dac307f50ae5da4c8f2cc5fdf024465f63600e30fc3bb434fca31191e2efdc/work 0 0
>
> Could those entries correspond to blobs generated by 'guix pack'?

Not directly (the generated image doesn’t mount anything by itself), but
it could correspond to the image layers maybe?

> Is it possible to make 'guix pack' create a merged container instead of
> all these layers?

Maybe you could try different values of ‘guix pack --max-layers=…’?

When ‘--max-layers’ is omitted, the resulting image contains a single
layer AIUI (according to the default value of
‘%docker-image-max-layers’).

Ludo’.

Re: Building a Docker image for GitLab-CI

[Simon Josefsson via]

[-- Attachment #1: Type: text/plain, Size: 4067 bytes --]

I am happy to announce Guix container images:

https://gitlab.com/debdistutils/guix/container/

They are suitable for use in GitLab pipelines.

There are many things to continue discuss and resolve.  However it is
now possible to start a GitLab pipeline job that uses an 'image:'
pointing to a container file that were created by `guix pack` and it has
network access.  Andreas Enge posted a link on how to achieve this with
non-free 'docker' and some custom hosted gitlab runners with extra
permissions, but my variant uses only free software and the images can
be reproduced solely on the GitLab public runners (i.e., no image
uploading from a local laptop).  As far as I know this hasn't been done
before.

Successful example job that runs a Guix environment:

https://gitlab.com/debdistutils/guix/container/-/jobs/8670483694

Start the container like this:

sudo apt-get install podman || guix package -i podman
podman run -it registry.gitlab.com/debdistutils/guix/container:latest

The README mentions a few works that you will appreciate -- can you
suggest ways to resolve these matters?

- `export HOME=/`

- `export CWD=/`

- `guix-daemon --disable-chroot &`

- `GUIX_PROFILE=/root/.config/guix/current; . "$GUIX_PROFILE/etc/profile"`

- guix package -i fails: `guix perform-download: error: refusing to
  run with elevated privileges (UID 0)`

- GitLab pipeline job entrypoints: three possible entry-point usages
  behave somewhat different depending on how `guix pack` was invoked

- Adding `nss-certs` to the `guix pack` command breaks: `(symlink
  "NetLock_Arany_=Class_Gold=_F?tan?s?tv?ny.pem" #) Throw to key
  encoding-error' with args ("scm_to_stringn" "cannot convert wide
  string to output locale" 84 #f #f)'.`

Those things aside, the main problem right now is that I would expect
'guix package -i gcc automake autoconf' to work.  However as you can see
in the `test-amd64-package-install` job things fail to download
seamingly from an intentional privilege check:

guix perform-download: error: refusing to run with elevated privileges (UID 0)

https://gitlab.com/debdistutils/guix/container/-/jobs/8670514009

What is the best way to resolve this?  Ugly insecure workarounds are
acceptable too, I didn't find any --insecure or similar.

It would be nice to publish a couple of different images, one minimal
and one more complete with, say, gcc on it.  Thoughts on what packages
to include and how to name the images?

Doing arm64 images should be straight forward but I get an error running
'guix pull' in the Debian image: https://issues.guix.gnu.org/74925 --
Also I do not know how to push a combined amd64+arm64 image like Debian
do with the 'debian:trixie' image doing the right thing for both amd64
and arm64 under the same name.  Ideas?

Can someone try to use them in some other environments, maybe GitHub
actions or Codeberg Woodpecker?

Finally, you may wonder why things didn't work before.  Some of the
major reasons: 1) --max-layer=100 and 2) -S /etc=etc and 3) Missing
/etc/protocols etc.  GitLab's docker setup doesn't handle many layers,
and it happens to just mount a sub-set of layers (see mount output,
missing a lot of layers).  Which files are put at which layer seems to
vary between `guix pack` runs for some reason, making it really hard to
debug (sometimes things worked partially, sometimes not, depending on
which files ended up visible).  I use --max-layers=8 now.  Re /etc=etc
it seems GitLab's docker setup bind-mounts things below /etc/ and it
cannot handle the root /etc symlink.  A workaround is to use `lndir`
which I use in the `test-amd64-package-install` job.  This is limitation
of GitLab's docker setup: I tried running a `-S /etc=etc` image on my
own GitLab runner based on Trisquel [1] and it worked fine, it mounted
things below the symlinked tree properly.  Could `guix pack` be teached
how to do a lndir-approach for /etc instead of symlink, perhaps?

Happy hacking,
/Simon

[1] https://gitlab.com/debdistutils/debdistreproduce/-/blob/main/gitlab-runner-with-podman-on-trisquel-aramo.md?ref_type=heads

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 255 bytes --]

Re: Building a Docker image for GitLab-CI

[Ludovic Courtès]

Hi Simon!

Simon Josefsson <simon@josefsson.org> skribis:

> I am happy to announce Guix container images:
>
> https://gitlab.com/debdistutils/guix/container/
>
> They are suitable for use in GitLab pipelines.

Yay!

> - guix package -i fails: `guix perform-download: error: refusing to
>   run with elevated privileges (UID 0)`

Should be fixed by running guix-daemon with
“--build-users-group=whatever” so that downloads run as one of the build
users, not as root.

> - GitLab pipeline job entrypoints: three possible entry-point usages
>   behave somewhat different depending on how `guix pack` was invoked

This image uses Guix installed on Debian though, no?

> - Adding `nss-certs` to the `guix pack` command breaks: `(symlink
>   "NetLock_Arany_=Class_Gold=_F?tan?s?tv?ny.pem" #) Throw to key
>   encoding-error' with args ("scm_to_stringn" "cannot convert wide
>   string to output locale" 84 #f #f)'.`

You need to run guix-daemon in a UTF-8 locale like “C.UTF-8” (which is
supported out-of-the-box).

See also <https://issues.guix.gnu.org/75007>.

> Finally, you may wonder why things didn't work before.  Some of the
> major reasons: 1) --max-layer=100 and 2) -S /etc=etc and 3) Missing
> /etc/protocols etc.  GitLab's docker setup doesn't handle many layers,
> and it happens to just mount a sub-set of layers (see mount output,
> missing a lot of layers).  Which files are put at which layer seems to
> vary between `guix pack` runs for some reason, making it really hard to
> debug (sometimes things worked partially, sometimes not, depending on
> which files ended up visible).  I use --max-layers=8 now.

Interesting.  Did you find points in the code (Docker? GitLab?) about
this subset of layers being mounted?

> Re /etc=etc it seems GitLab's docker setup bind-mounts things below
> /etc/ and it cannot handle the root /etc symlink.  A workaround is to
> use `lndir` which I use in the `test-amd64-package-install` job.  This
> is limitation of GitLab's docker setup: I tried running a `-S
> /etc=etc` image on my own GitLab runner based on Trisquel [1] and it
> worked fine, it mounted things below the symlinked tree properly.
> Could `guix pack` be teached how to do a lndir-approach for /etc
> instead of symlink, perhaps?

It could symlink individual files and make /etc a directory.  (What’s
‘lndir’, if I may ask?)

Thanks for the investigation and all!

Ludo’.

Re: Building a Docker image for GitLab-CI

[Simon Josefsson via]

[-- Attachment #1: Type: text/plain, Size: 7133 bytes --]

Ludovic Courtès <ludo@gnu.org> writes:

>> - guix package -i fails: `guix perform-download: error: refusing to
>>   run with elevated privileges (UID 0)`
>
> Should be fixed by running guix-daemon with
> “--build-users-group=whatever” so that downloads run as one of the build
> users, not as root.

Yes, I discovered that too.

>> - GitLab pipeline job entrypoints: three possible entry-point usages
>>   behave somewhat different depending on how `guix pack` was invoked
>
> This image uses Guix installed on Debian though, no?

No - the resulting image is purely from 'guix pack', no Debian in the
final image at all.

I didn't test now but I think Debian images handle all three entrypoint
values, but the 'guix pack' image doesn't.

>> - Adding `nss-certs` to the `guix pack` command breaks: `(symlink
>>   "NetLock_Arany_=Class_Gold=_F?tan?s?tv?ny.pem" #) Throw to key
>>   encoding-error' with args ("scm_to_stringn" "cannot convert wide
>>   string to output locale" 84 #f #f)'.`
>
> You need to run guix-daemon in a UTF-8 locale like “C.UTF-8” (which is
> supported out-of-the-box).
>
> See also <https://issues.guix.gnu.org/75007>.

Thanks - I finally figured this one out in the same way.

>> Finally, you may wonder why things didn't work before.  Some of the
>> major reasons: 1) --max-layer=100 and 2) -S /etc=etc and 3) Missing
>> /etc/protocols etc.  GitLab's docker setup doesn't handle many layers,
>> and it happens to just mount a sub-set of layers (see mount output,
>> missing a lot of layers).  Which files are put at which layer seems to
>> vary between `guix pack` runs for some reason, making it really hard to
>> debug (sometimes things worked partially, sometimes not, depending on
>> which files ended up visible).  I use --max-layers=8 now.
>
> Interesting.  Did you find points in the code (Docker? GitLab?) about
> this subset of layers being mounted?

No, I didn't go down chasing the root cause.  I wonder if maybe it isn't
as simple as some Linux kernel restriction on mount point sizes?  If you
try to load a --max-layer=100 image on GitLab you end up with a
/proc/mounts containing lines like this:

rw,relatime,lowerdir=l/ZPVAK6UICUAWUUE4GUD6AYFCEM:l/HI6HL2SJWTZDQHPTA3SGR4PWNE:l/OAJJQ5NKJAJLIAOEWBNLI6WRNC:l/MRDSZ2V6PLTEQGMEDSOPX6FKEY:l/FG4SISAU6TZNHB6CQR5X5GNEJB:l/EZGDP6A5CMVPA5O6IKOOKPDMBE:l/DA5NZCY6NVIGU2X6U5XQQXV54M:l/P4MIVQ3I7VCYFTQ3AG6RCXZVW5:l/FGPCINKKCYRDHZI5BAAU7HEETW:l/MUYJFZRJLR4Z3BNBFRBRTGP4S5:l/UPGZHVDAILBRLLEH6T5RKWIFG7:l/YNTBNOTPU7QRK5K6RD63VSXG5Q:l/XUPHFGGN36OPHNU334M3V7HDXP:l/PVA2QRQE4D5MAKI6BMGVPETNZN:l/VTGADKUDL4KA4HPA2YCCQ2JQNI:l/WZHWY243PTUZTQIZJM26PVSYG4:l/7YLLGUSIRSUFPXIFI57F2UQSFQ:l/ZEOCYJR44JGRBRKGEM4AMWUHQE:l/YXFWBRXLFSBMYDDCTNODGJWWUV:l/NQK5YT5BWDWSTKFRB3KOGVGYLC:l/6CC2D5S3LSZOOKLULC2JJ5BUHG:l/OMQ7M7FSULZ2WHTQPIOIYP7HYQ:l/VRGMNYJNRPOBPP5IZCJR3YV7FP:l/5L2O6RAVRTGGTB7I2YKS6RMX64:l/57QINIJWW7ECMX3DKLCDL5UMUS:l/HTIS4VRXRVO24AWC6AYPQ4LPRG:l/DOTULUURRI6Z4XR2B4LPQTP33T:l/LZYLE6JUKQFJBIAEMBQMQIWZEE:l/2WEGBKQG6D3VAWLXJ5NCZPFGNP:l/QYVPGN6K2A3Y4VVVPKYLR5JVL7:l/IPS4YGXCRZ47O4AOKMZ4TYD2N3:l/2MZU46ZBHYS5IQI4NAVFD3PNYR:l/HW6B6GDN33YB7GBY3LEOW2XXB4:l/FRINUFWGYICMVPLOJULIHQ3XKV:l/HSHVRY3DQIT5LUS3EONQQCKNL3:l/CRUDGYNQRSSDKB4TYALBBVFIL3:l/QJRZZB6NOMXWO46YCAJ4U53VSH:l/BZPO5MYEYX3YFX5NXYR32E6VRM:l/FDWSYXJR7RNKG42AMXGSC6NUQE:l/Z6BTLASGE6ZXGQZUFIHG4QXQLU:l/DWTG7Y6N4DNP5AZLI6MIDZEBHQ:l/TY4DSPRKETTLFE5WB7KFBO2VSM:l/SAE7PNZP6FFK2NVDOQQTZMO2BV:l/ZCL7CRORSVNYVFKNW2WT7TMGFZ:l/KVH3MQTKJ6SH46B2FEHW7UCYGK:l/XMDKZU7KD755BFINQOBKPLKMZQ:l/MFKZPIVLWDKG5PIVI3UQUU3ILT:l/I24GPDX2SRV3Y4YSJDYVKEBDQE:l/W6OZHVZW2NCSQOJEGMP45P2D6W:l/ZD4RI6B4WQ65QO7EMDQZSCZFHS:l/ZS7D35LTVLE6NSGCCY4SQQANE5:l/2ZIJ6PAUHDPBOXR72A6HGU6L4B:l/ZL5XOQ6XMF6ZYQSXVAK7744SX7:l/A3WXIZD22NL62JYHBZVL5K36IK:l/KUDHAVWVBDXKHHQQVF33KDPHF4:l/6O5OF37ORZTEUV2JXNOPO3WNTQ:l/YVGTC6PVMDS2GVOF57TJWCN5DM:l/VYM3JKFOWAY7UIYGU6TPJTZVD5:l/2BT7MWUS2JMZMQXQ3MRLO5AH6I:l/CNZTAOCVMGIFCNP4IF77OQ5MU6:l/KQCUQ7H6EY7U423TWBX6N6QNU7:l/VNZNN2P4U26XHEDRSNIQ656GRE:l/7YG6BVIVDJYCCGLISSU4APAR4S:l/ELR3M2R3NLUI4U2YCKYKG6MWUV:l/PK34AGBR7JY4PUJIVEAO4J7UAK:l/ELIDWT3IMYDRR5L5VTA3REN3LH:l/DK7VEQCTNIWW4BOYCOVXZX2GQY:l/6BCYXFR5B6S3EYCMVCXKQPEP3M:l/GAF6PUKMPKABSXDZCMVE3NM2K5:l/ASVMZMXSKHAVGH5UFXXRFE7TUX:l/25QORLMIGZEEIBQGTV6UBNZEAH:l/FH5SA2MBXXRRAMDYI72FPK7RXU:l/77IRT3TXX3H7XH66YGR7O5AIYK:l/FIQQSP7XQLUH3IWBXF4DZXWTFN:l/NG6ZAASCTOH6SQVUBR2FR6YE4T:l/QAHEWNHBILTWWWJ4QMX4ISWIT4:l/VCSGZSH4SQRVHK4EWSLGFECG2S:l/7FAMLBB6DJW7VWYEOIN6FLBI2E:l/C4ZXSOLVY36PYMTRL2E2YIKTKY:l/NVH5IIZJ5SO422GGMFGNTFCAOA:l/E6EE3L3EB2B36E5A5PL5KF32FL:l/RPXH632CD3Y54UYAMMULLEZOE4:l/662KE3GHNLWZEUTFCRZXOKVA3P:l/3ZDSIX7ZFSLT2FIEYD6TB6GA7R:l/ZS7SJOXF7XF6OFTLVHYMRYZANK:l/F3CEOKDDPOK72QLUWGKHDIJTPG:l/65CFVU5ZFM4XVSTLDO2WGQE3GU:l/3C2OJ5ZVICH5QP733HTS2DHPJM:l/B37PUETDZKC3MUZZX6CT3M6ZEC:l/LTVIXAHIS7O45NFIVNPDVGYMR4:l/GGBS4BWCAD6UMV3755MJ3BQRI5:l/4XGXVIO5QQ2ARZTTG74M4UDK4V:l/X673P4Q5TGPWBLQIXOQ5JXKG2C:l/3K2Q6NGZ5EEIKCFBRMEWWZFWQK:l/ALTG2QI6YZVKXIP4UCRDMQWS5Q:l/P4AE6X6G5GK66GHHRBZ27BTBF3:l/2UEJWLKTB2GNGT2G6UWFZVDFRN:l/OCTYS4BHPFX4HC3OXJJVGABEPX:l/M766VP6Y5QL2OGI6EYO5OEB6H4:l/DNITYWJFFEVGWQPFQ37S57EIB5:l/PQE5T2ER2OKPNC2YNHZEJXVIY6,upperdir=98dac307f50ae5da4c8f2cc5fdf024465f63600e30fc3bb434fca31191e2efdc/diff,workdir=98dac307f50ae5da4c8f2cc5fdf024465f63600e30fc3bb434fca31191e2efdc/work

That doesn't contain all of the 100 layers (although I didn't double
count..).  Something is truncating those lines.

That would explain why it works for me locally with 'podman': it seems
podman uses much shorter names, like:

rw,relatime,lowerdir=0:1:2:3:4..

instead of those longer strings above.

>> Re /etc=etc it seems GitLab's docker setup bind-mounts things below
>> /etc/ and it cannot handle the root /etc symlink.  A workaround is to
>> use `lndir` which I use in the `test-amd64-package-install` job.  This
>> is limitation of GitLab's docker setup: I tried running a `-S
>> /etc=etc` image on my own GitLab runner based on Trisquel [1] and it
>> worked fine, it mounted things below the symlinked tree properly.
>> Could `guix pack` be teached how to do a lndir-approach for /etc
>> instead of symlink, perhaps?
>
> It could symlink individual files and make /etc a directory.

Is this possible now?  How?

The --symlink is defined like this now:

‘-S SPEC’
     Add the symlinks specified by SPEC to the pack.  This option can
     appear several times.
     SPEC has the form ‘SOURCE=TARGET’, where SOURCE is the symlink that
     will be created and TARGET is the symlink target.
     For instance, ‘-S /opt/gnu/bin=bin’ creates a ‘/opt/gnu/bin’
     symlink pointing to the ‘bin’ sub-directory of the profile.

Could we extend that somehow to support the model of recursively
creating directories, but symlink files within them?

Or a new --symlink-mkdir parameter or similar?

> (What’s ‘lndir’, if I may ask?)

Exactly that :)

One of those ancient tools that still is surprisingly useful:

https://manpages.debian.org/bookworm/xutils-dev/lndir.1.en.html
https://gitlab.freedesktop.org/xorg/util/lndir/-/blob/master/lndir.c

/Simon

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 255 bytes --]

Container image entrypoints on Gitlab (was: Re: Building a Docker image for GitLab-CI)

[Simon Josefsson via]

[-- Attachment #1: Type: text/plain, Size: 2722 bytes --]

Simon Josefsson via <help-guix@gnu.org> writes:

> I didn't test now but I think Debian images handle all three entrypoint
> values, but the 'guix pack' image doesn't.

That was not true!  Here the situation:

https://gitlab.com/debdistutils/guix/container/-/pipelines/1600726433

Debian fails on these two GitLab .gitlab-ci.yml variants:

  image:
    name: debian:stable
    entrypoint: ["/bin/sh"]

  image:
    name: debian:stable
    entrypoint: ["/bin/sh", "-c"]

And works on these two variants:

  image:
    name: debian:stable
    entrypoint: [""]

  image: debian:stable

The last one is what I suppose most people want to use.  One shouldn't
have to specify entrypoint's when using GitLab CI/CD images.

My Guix container:latest works on these three variants:

  image:
    name: debian:stable
    entrypoint: ["/bin/sh", "-c"]

  image:
    name: debian:stable
    entrypoint: [""]

  image: debian:stable

And fail on this variant:

  image:
    name: debian:stable
    entrypoint: ["/bin/sh"]

At least we are not worse than Debian here.

However there is another difference, local run of Debian images handle
both these variants:

jas@kaka:~/src/guix-container$ podman run -it --rm debian:trixie-slim
root@f9634c8d6b63:/# 
exit
jas@kaka:~/src/guix-container$ podman run --entrypoint /bin/sh -it --rm debian:trixie-slim
# 
jas@kaka:~/src/guix-container$ 

But my Guix images doesn't:

jas@kaka:~/src/guix-container$ podman run -it --rm registry.gitlab.com/debdistutils/guix/container:latest
Error: no command or entrypoint provided, and no CMD or ENTRYPOINT from image
jas@kaka:~/src/guix-container$ podman run --entrypoint /bin/sh -it --rm registry.gitlab.com/debdistutils/guix/container:latest
sh-5.1# exit
jas@kaka:~/src/guix-container$ 

Presumably because of a missing '--entry-point=/bin/sh' when building
the images using 'guix pack'.  So let's add it!  Yes this now works:

jas@kaka:~/src/guix-container$ podman run -it --rm registry.gitlab.com/debdistutils/guix/container:latest2
sh-5.1# exit
jas@kaka:~/src/guix-container$ podman run --entrypoint /bin/sh -it --rm registry.gitlab.com/debdistutils/guix/container:latest2
sh-5.1# exit
jas@kaka:~/src/guix-container$ 

However this breaks GitLab CI/CD usage:

  image: $CI_REGISTRY_IMAGE:latest2

https://gitlab.com/debdistutils/guix/container/-/jobs/8713593164

Ouch, that is the primary use-case we want to support!

So until someone figures this one out, I've opted that the default
images require --entrypoint when invoked local on laptops with podman
but work without any configuration from GitLab shared runners and normal
.gitlab-ci.yml usage.

/Simon

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 255 bytes --]

GitLab container /etc symlink problem (was: Re: Building a Docker image for GitLab-CI)

[Simon Josefsson via]

[-- Attachment #1: Type: text/plain, Size: 4001 bytes --]

Simon Josefsson via <help-guix@gnu.org> writes:

>>> Re /etc=etc it seems GitLab's docker setup bind-mounts things below
>>> /etc/ and it cannot handle the root /etc symlink.  A workaround is to
>>> use `lndir` which I use in the `test-amd64-package-install` job.  This
>>> is limitation of GitLab's docker setup: I tried running a `-S
>>> /etc=etc` image on my own GitLab runner based on Trisquel [1] and it
>>> worked fine, it mounted things below the symlinked tree properly.
>>> Could `guix pack` be teached how to do a lndir-approach for /etc
>>> instead of symlink, perhaps?
>>
>> It could symlink individual files and make /etc a directory.
>
> Is this possible now?  How?
>
> The --symlink is defined like this now:
>
> ‘-S SPEC’
>      Add the symlinks specified by SPEC to the pack.  This option can
>      appear several times.
>      SPEC has the form ‘SOURCE=TARGET’, where SOURCE is the symlink that
>      will be created and TARGET is the symlink target.
>      For instance, ‘-S /opt/gnu/bin=bin’ creates a ‘/opt/gnu/bin’
>      symlink pointing to the ‘bin’ sub-directory of the profile.
>
> Could we extend that somehow to support the model of recursively
> creating directories, but symlink files within them?
>
> Or a new --symlink-mkdir parameter or similar?
>
>> (What’s ‘lndir’, if I may ask?)
>
> Exactly that :)

I found out that the lndir approach doesn't work.  The /etc symlinks are
modified by 'guix package -i'.  It only occurs sometimes, probably
depending on if it happens to pull in a new version of 'net-base'.  It
seems 'guix package -i hello' works but 'guix package -i skopeo'
doesn't.

https://gitlab.com/debdistutils/guix/container/-/jobs/8713723100

...
substitute: updating substitutes from 'https://bordeaux.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
61.5 MB will be downloaded
retrying download of '/gnu/store/bfp25w47fxn8z0fdwj45prx2609sx59j-net-base-5.3' with other substitute URLs...
guix substitute: warning: ci.guix.gnu.org: host not found: Servname not supported for ai_socktype
guix substitute: error: failed to find alternative substitute for '/gnu/store/bfp25w47fxn8z0fdwj45prx2609sx59j-net-base-5.3'
substitution of /gnu/store/bfp25w47fxn8z0fdwj45prx2609sx59j-net-base-5.3 failed
guix package: error: some substitutes for the outputs of derivation `/gnu/store/cviappwxgh7ilhprl44x5ya2nypwlcj7-skopeo-1.17.0.drv' failed (usually happens due to networking issues); try `--fallback' to build derivation from source 

You can see that networking worked initially, but broke down later on.

lrwxrwxrwx 1 root 0   70 Dec 23 18:24 protocols -> /gnu/store/bfp25w47fxn8z0fdwj45prx2609sx59j-net-base-5.3/etc/protocols

After running 'guix package -i skopeo' that symlink is danling:

++ echo '$ ls -ld /gnu/store/*net-base*'
++ ls -ld /gnu/store/bfp25w47fxn8z0fdwj45prx2609sx59j-net-base-5.3.lock /gnu/store/i705faz4w4wnq1frrbbrk2jnhwhsr2k9-net-base-5.3-builder /gnu/store/kaqyk1rn9f4q36h8nc0919wrlam0xl0g-net-base-5.3.drv
-rw------- 1 root 0   0 Dec 23 18:26 /gnu/store/bfp25w47fxn8z0fdwj45prx2609sx59j-net-base-5.3.lock
-r--r--r-- 1 root 0 893 Jan  1  1970 /gnu/store/i705faz4w4wnq1frrbbrk2jnhwhsr2k9-net-base-5.3-builder
-r--r--r-- 1 root 0 990 Jan  1  1970 /gnu/store/kaqyk1rn9f4q36h8nc0919wrlam0xl0g-net-base-5.3.drv
++ echo '$ ls -ld /gnu/store/*profile*'

I've reverted back to this hack:

cp -rL /gnu/store/*profile/etc/* /etc/

Is there a clean solution to this?  Perhaps we could debug further why
GitLab/docker barfs on a /etc symlink.  It is probably related to them
having these mount points:

/dev/sda1 /etc/hostname ext4 rw,nosuid,nodev,relatime,commit=30 0 0
/dev/sda1 /etc/hosts ext4 rw,nosuid,nodev,relatime,commit=30 0 0
/dev/sda1 /etc/resolv.conf ext4 rw,nosuid,nodev,relatime,commit=30 0 0

Maybe -S /etc=etc works if we make sure to remove just those three
files..

/Simon

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 255 bytes --]