Hey Guixters ! I am experimenting with Gitlab CI for a Guile project I am working on. It is managed with Hall and I would like to benefit the Guix features. Currently, I struggle to run Guix inside the default Debian stable docker image (framagit.org). Here is my .gitlab-ci.yml : before_script: - apt -qq update - apt -qq install -y wget gpg xz-utils - wget https://git.savannah.gnu.org/cgit/guix.git/plain/etc/guix-install.sh - chmod +x guix-install.sh - wget https://sv.gnu.org/people/viewgpg.php?user_id=15145 -qO - | gpg --import - - printf '\n' | ./guix-install.sh - /root/.config/guix/current/bin/guix-daemon --build-users- group=guixbuild - guix pull stages: - bootstrap - configure - build - test job:bootstrap: stage: bootstrap script: autoreconf -vfi What I get : [1592641832.155]: [ FAIL ] unsupported init system; run the daemon manually: /root/.config/guix/current/bin/guix-daemon --build-users- group=guixbuild [1592641832.155]: [ INFO ] making the guix command available to other users Does anyone has done that before ? Is there a way to get a Guix docker image instead of the Debian one ? Have a nice weekend ! Jérémy
Jérémy Korwin-Zmijowski <jeremy@korwin-zmijowski.fr> writes:
> Hey Guixters !
>
> I am experimenting with Gitlab CI for a Guile project I am working on.
> It is managed with Hall and I would like to benefit the Guix features.
>
> Currently, I struggle to run Guix inside the default Debian stable
> docker image (framagit.org).
Can you specify a different Docker image? Then you could create one
containing Guix with “guix pack -f docker”.
--
Ricardo
Hello Jérémy, The problem is that guix-install.sh asks you whether you want to authorize the substitutes, but it can't because it is not used interactively, so it fails. There are other traps to avoid (do not forget to install ntp, call guix-daemon with --disable-chroot because you are in an unprivileged container, cache things to avoid pulling the world at each commit). I have a solution based on caching a guix pack containing all your packages: https://framagit.org/divoplade/guix-project variables: GUIX_REF: "--branch=master" PACKAGES: "guile-hall hello" before_script: - test -f cache.tar.gz || apt -qq update - test -f cache.tar.gz || apt -qq install -y wget gpg xz-utils ntp - test -f cache.tar.gz || wget https://sv.gnu.org/people/viewgpg.php?user_id=15145 -qO - | gpg -- import - - test -f cache.tar.gz || wget https://git.savannah.gnu.org/cgit/guix.git/plain/etc/guix-install.sh - test -f cache.tar.gz || chmod ugo+x guix-install.sh - test -f cache.tar.gz || (echo "" ; echo "y") | ./guix-install.sh || echo "Assume OK" - test -f cache.tar.gz || source /root/.config/guix/current/etc/profile - test -f cache.tar.gz || guix-daemon --build-users-group=guixbuild --disable-chroot & - test -f cache.tar.gz || guix pull $GUIX_REF - test -f cache.tar.gz || cp $(guix pack -C gzip $PACKAGES -S/opt/bin=bin) cache.tar.gz - cp cache.tar.gz / - (cd / && tar xf cache.tar.gz) - export PATH=/opt/bin:$PATH cache: key: "$GUIX_REF $PACKAGES" paths: - cache.tar.gz job: script: - hello - hall --help After a very long set up (1/2 hour with a somewhat fast runner), you will leverage the cached guix pack to run the jobs instantly, with the default gitlab runner (unprivileged, debian-based). If you want to update, clear the runner caches, or change the GUIX_REF variable. You can pin a specific commit (better for reproducibility, not so great for releases) with --commit=COMMIT. I could not get guix to run within the job, even with "guix" added to PACKAGES: it would need this in the script: - apt -qq update - apt -qq install -y ntp - (groupadd --system guixbuild && for i in $(seq 1 10) ; do useradd -g guixbuild -G guixbuild -d /var/empty -s $(which nologin) -c "Guix build user $i" --system guixbuilder$i; done) || true - guix-daemon --build-users-group=guixbuild --disable-chroot & - guix archive --authorize < $(dirname $(readlink /usr/local/bin/guix))/../share/guix/ci.guix.info.pub However, if you try it, you will notice a crash. I am looking forward to having guix shipped as a debian package, it will be way easier. divoplade Le samedi 20 juin 2020 à 10:42 +0200, Jérémy Korwin-Zmijowski a écrit : > Hey Guixters ! > > I am experimenting with Gitlab CI for a Guile project I am working > on. > It is managed with Hall and I would like to benefit the Guix > features. > > Currently, I struggle to run Guix inside the default Debian stable > docker image (framagit.org). > > Here is my .gitlab-ci.yml : > > before_script: > - apt -qq update > - apt -qq install -y wget gpg xz-utils > - wget > > https://git.savannah.gnu.org/cgit/guix.git/plain/etc/guix-install.sh > - chmod +x guix-install.sh > - wget https://sv.gnu.org/people/viewgpg.php?user_id=15145 -qO - | > gpg --import - > - printf '\n' | ./guix-install.sh > - /root/.config/guix/current/bin/guix-daemon --build-users- > group=guixbuild > - guix pull > > stages: > - bootstrap > - configure > - build > - test > > job:bootstrap: > stage: bootstrap > script: autoreconf -vfi > > What I get : > > [1592641832.155]: [ FAIL ] unsupported init system; run the daemon > manually: > /root/.config/guix/current/bin/guix-daemon --build-users- > group=guixbuild > [1592641832.155]: [ INFO ] making the guix command available to > other users > > Does anyone has done that before ? > Is there a way to get a Guix docker image instead of the Debian one ? > > Have a nice weekend ! > > Jérémy > >
Le samedi 20 juin 2020 à 15:40 +0200, Ricardo Wurmus a écrit :
>
> Can you specify a different Docker image? Then you could create one
> containing Guix with “guix pack -f docker”.
>
I am not the owmer of the Gitlab instance, so I think I can't... But I
can ask to the admin haha. Worth the shot !
Thank you Ricardo !
Hey divoplade ! Thank you very much for sharing your hack ! The caching section is very interesting... Does it save time between runs ? After a lot of attempts, I ended up with a green pipeline but it takes 43 minute to complete haha https://framagit.org/Jeko/guile-dummy/-/pipelines Jérémy
I tried to apply the caching as you did, but I error during project configuration. https://framagit.org/Jeko/guile-dummy/-/jobs/1029866 Jérémy Le mardi 23 juin 2020 à 12:15 +0200, Jérémy Korwin-Zmijowski a écrit : > Hey divoplade ! > > Thank you very much for sharing your hack ! The caching section is > very > interesting... Does it save time between runs ? > > After a lot of attempts, I ended up with a green pipeline but it > takes > 43 minute to complete haha > https://framagit.org/Jeko/guile-dummy/-/pipelines > > Jérémy > > > > > >
Hi Jérémy, On Sat, 20 Jun 2020 at 10:42, Jérémy Korwin-Zmijowski <jeremy@korwin-zmijowski.fr> wrote: > - printf '\n' | ./guix-install.sh [...] > [1592641832.155]: [ FAIL ] unsupported init system; run the daemon > manually: > /root/.config/guix/current/bin/guix-daemon --build-users- > group=guixbuild > [1592641832.155]: [ INFO ] making the guix command available to > other users > > Does anyone has done that before ? It is because the guix-install.sh script does not recognize the init system of the Docker image, see 'sys_enable_guix_daemon()'. Roughly speaking, it does not fall into upstart) neither systemd) neither sysv-init) so it falls into NA|*). I do not know how the Docker image you use is built. > Is there a way to get a Guix docker image instead of the Debian one ? The easiest is, IMHO, instead of "./guix-install.sh", you should run all the 8 steps described in [1] adapting the step #5. 1: https://guix.gnu.org/manual/devel/en/guix.html#Binary-Installation Cheers, simon
Hi Zimoun ! Thank you for taking time to help over here ! Thanks to divoplade, we succedded to make something up for relative simple projects (I don't know how it goes for more complicated ones). If anybody feel interested : https://framagit.org/Jeko/guile-dummy Cheers, Jérémy
[-- Attachment #1: Type: text/plain, Size: 1850 bytes --] Hello Jérémy, I have a new solution (it is quite different from the previous one) in which it is possible to run guix commands within a job, while still caching most of the files (see attached). I am sorry, my fork of your project is already a mess so I prefer to put the file in its entirety here. Anyways, everything will be way easier once we have a proper debian package for guix: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=850644 divoplade Le samedi 20 juin 2020 à 10:42 +0200, Jérémy Korwin-Zmijowski a écrit : > Hey Guixters ! > > I am experimenting with Gitlab CI for a Guile project I am working > on. > It is managed with Hall and I would like to benefit the Guix > features. > > Currently, I struggle to run Guix inside the default Debian stable > docker image (framagit.org). > > Here is my .gitlab-ci.yml : > > before_script: > - apt -qq update > - apt -qq install -y wget gpg xz-utils > - wget > > https://git.savannah.gnu.org/cgit/guix.git/plain/etc/guix-install.sh > - chmod +x guix-install.sh > - wget https://sv.gnu.org/people/viewgpg.php?user_id=15145 -qO - | > gpg --import - > - printf '\n' | ./guix-install.sh > - /root/.config/guix/current/bin/guix-daemon --build-users- > group=guixbuild > - guix pull > > stages: > - bootstrap > - configure > - build > - test > > job:bootstrap: > stage: bootstrap > script: autoreconf -vfi > > What I get : > > [1592641832.155]: [ FAIL ] unsupported init system; run the daemon > manually: > /root/.config/guix/current/bin/guix-daemon --build-users- > group=guixbuild > [1592641832.155]: [ INFO ] making the guix command available to > other users > > Does anyone has done that before ? > Is there a way to get a Guix docker image instead of the Debian one ? > > Have a nice weekend ! > > Jérémy > > [-- Attachment #2: .gitlab-ci.yml --] [-- Type: application/x-yaml, Size: 2088 bytes --]
Le jeudi 25 juin 2020 à 23:32 +0200, divoplade a écrit :
> Hello Jérémy,
>
> I have a new solution (it is quite different from the previous one)
> in
> which it is possible to run guix commands within a job, while still
> caching most of the files (see attached). I am sorry, my fork of your
> project is already a mess so I prefer to put the file in its entirety
> here.
>
> Anyways, everything will be way easier once we have a proper debian
> package for guix:
>
> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=850644
>
> divoplade
Hi divoplade !
Thank you for the new yaml file ! Unfortunately, I didn't manage to use
this one yet. I have to do more experiments I guess.
I the meantime I would like to try to put a Guix's docker image on
dockerhub and use it in Framagit/Gitlab if it is possible...
Cheers
Jérémy
[-- Attachment #1: Type: text/plain, Size: 318 bytes --] Jérémy Korwin-Zmijowski <jeremy@korwin-zmijowski.fr> writes: > I the meantime I would like to try to put a Guix's docker image on > dockerhub and use it in Framagit/Gitlab if it is possible... This would be fantastic, please let us know about your progress :) -- Pierre Neidhardt https://ambrevar.xyz/ [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 487 bytes --]
Jérémy Korwin-Zmijowski <jeremy@korwin-zmijowski.fr> writes: > Hey Guixters ! > > I am experimenting with Gitlab CI for a Guile project I am working on. > It is managed with Hall and I would like to benefit the Guix features. > > Does anyone has done that before ? > Is there a way to get a Guix docker image instead of the Debian one ? Hello Jérémy, I have been trying to accomplish a similar goal - using Guix in GitLab CI for a Guile project. To that end, I’ve built some Docker images you might find interesting: Guix in Alpine Linux https://hub.docker.com/r/singularsyntax/guix-bootstrap/tags https://gitlab.com/singularsyntax-docker-hub/guix-bootstrap (the Dockerfile and pipeline for deployment to Docker Hub) My goal is to enable typical Docker workflows with Guix, for example something like this Dockerfile: FROM gnu/guix:1.1.0 RUN guix package —install <some-build-and-test-package-dependencies> Then use the resulting Docker image in the build, test, etc. stages of a GitLab CI/CD pipeline. Unfortunately, Guix does not have a natural fit with Docker containerization since `guix-daemon` must be running for any Guix commands to work. I haven’t been able to get any Dockerfile workflows like above working, since `docker build` seems to skip execution of the regular image entry point command which starts `guix-daemon`. I think to make it work, you’d need to provide some kind of shell wrapper around the `guix` command which first starts `guix-daemon`, waits for it to be ready, then runs Guix commands. But you can use the Alpine image for Docker-in-Docker workflows where you first start a container with `docker run`, then issue successive `docker exec` commands to perform work inside the container, then extract the results from the container for use in other stages of a CI/CD pipeline. I also attempted to build a “pure” Guix container not requiring installation on top of another Linux distribution by using `guix system docker-image <…>`. I even had a GitLab pipeline making daily builds which you might find interesting (but I recently disabled it as I’ll explain below): https://hub.docker.com/r/singularsyntax/guix/tags https://gitlab.com/singularsyntax-docker-hub/guix (GitLab CI/CD pipeline for deployment to Docker Hub) It works fine for obtaining a sandbox for which to experiment interactively with Guix, but has the same problems as the Alpine image, and is even a little worse because there are no “anchor” binaries inside the container, not even `/bin/sh`, until the container init process establishes symlinks into the `/gnu/store` tree. If you try to use it in scripts or CI/CD pipelines, they often fail due to race conditions with the init script, and without `/bin/sh` it’s difficult to reliably assert any pre-conditions inside the container. So it’s a real pain to work with, and I think of little utility for scripting as it currently stands. Anyway, hopefully this provides some more insight, and good luck!
Hello! I’m late to the party but I’m interested in this discussion… :-) Jérémy Korwin-Zmijowski <jeremy@korwin-zmijowski.fr> skribis: > I am experimenting with Gitlab CI for a Guile project I am working on. > It is managed with Hall and I would like to benefit the Guix features. GitLab CI can run jobs directly in a Docker image: https://docs.gitlab.com/ce/ci/docker/using_docker_images.html So it would be interesting to have any easy way for people to create a Docker image for this purpose. Colleagues of mine tried ‘guix system docker-image’ for that but that turns out to not quite work (IIRC gitlab-runner tries to do ‘docker exec your build command’, which fails here because those images have shepherd as their entry point). So presumably a better option is to use an image created like this: guix pack -f docker -S /bin=bin guix --localstatedir and then, in the image, you first need to spawn guix-daemon, perhaps with ‘--disable-chroot’ because the image lacks build users. It would be useful also to set the user under which commands run in the image, similar to the ‘USER’ directive in Dockerfiles. Thoughts? It would be nice to make that as smooth as possible and to add a recipe in the cookbook! Ludo’.
Hello,
Le vendredi 24 juillet 2020 à 16:30 +0200, Ludovic Courtès a écrit :
>
> So presumably a better option is to use an image created like this:
>
> guix pack -f docker -S /bin=bin guix --localstatedir
>
As you may know, most gitlab server will not let you use the provided
docker registry, and so the only option will be to push to docker hub.
Moreover, running this command as part of a CI job just to push it to
docker hub will require a privileged container. So you will need to
either provide your own runner, or provide you own docker registry
(docker hub or else).
I prefer the solution in which the pack is a .tar.gz that is directly
untarred to /; this way it can be run on a shared unprivileged runner
and the storage can be achieved in the runner itself, via the gitlab CI
cache.
divoplade
Hi, divoplade <d@divoplade.fr> skribis: > Le vendredi 24 juillet 2020 à 16:30 +0200, Ludovic Courtès a écrit : >> >> So presumably a better option is to use an image created like this: >> >> guix pack -f docker -S /bin=bin guix --localstatedir >> > As you may know, most gitlab server will not let you use the provided > docker registry, and so the only option will be to push to docker hub. Yes. What I had in mind was that you would create that image, upload it to a registry GitLab-CI can pull from, and then refer to that image in your ‘.gitlab-ci.yml’. > I prefer the solution in which the pack is a .tar.gz that is directly > untarred to /; this way it can be run on a shared unprivileged runner > and the storage can be achieved in the runner itself, via the gitlab CI > cache. Ah yes, I guess that would also work, and it’s in fact less tedious. That amounts to running https://guix.gnu.org/install.sh every time though, no? Thanks, Ludo’.
Le lundi 27 juillet 2020 à 09:46 +0200, Ludovic Courtès a écrit :
> > I prefer the solution in which the pack is a .tar.gz that is
> > directly
> > untarred to /; this way it can be run on a shared unprivileged
> > runner
> > and the storage can be achieved in the runner itself, via the
> > gitlab CI
> > cache.
>
> Ah yes, I guess that would also work, and it’s in fact less tedious.
> That amounts to running https://guix.gnu.org/install.sh every time
> though, no?
Pretty much, except "every time" is really less often than expected,
because the gitlab runner can cache files between jobs. So with an
appropriate tar command, it is possible to pack the whole store plus
the /var/guix and untar it at the beginning of the job (or re-run the
guix install if the cache has been dropped). Or run "guix pack".
There is a very unfortunate problem that running "guix pull" or guix
installing something will require the "ntp" debian package, so each job
requires apt-get update and apt-get install -y ntp... I am
investigating how to cache the ntp package (and its dependencies) to
avoid running apt-get update, but that's quite disappointing.
divoplade
Ludovic Courtès writes: > Hello! I’m late to the party but I’m interested in this discussion… :-) > GitLab CI can run jobs directly in a Docker image: > https://docs.gitlab.com/ce/ci/docker/using_docker_images.html > So it would be interesting to have any easy way for people to create a Docker image for this purpose. I’ve updated my Alpine-based Docker image, and I think it should work now for many common use cases, but it would be great if it saw some more testing. Details here: https://hub.docker.com/r/singularsyntax/guix-bootstrap Cheers -ss