Releasing guix binary in Docker format too?

Releasing guix binary in Docker format too?

[Danny Milosavljevic]

[-- Attachment #1: Type: text/plain, Size: 1234 bytes --]

Hello,

I'm working with Heads on using Guix to build Heads' source code reproducibly.

Now, Heads has most of its stuff on CI servers that use Docker for a lot of
things.

Therefore, I used Guix pack in the usual way to create a recent Guix
binary release and then just loaded that into a docker container, using that to
run "guix build".

That got me thinking--this situation is pretty common.  Do you think it would
make sense to actually build and release a Docker container for Guix (not
guix system; just guix-the-package-manager)?

While I was researching gitlab, I came across a feature where they have their
own Docker container registry on there.  Apparently, the Docker CLI can fetch
from whatever server the user wants!

$ docker run [options] registry.gitlab.com/group/project/image [arguments]

That got me thinking, we could easily also release Guix on something like
that.  Is our HTTP webserver enough to have a Docker registry, without
installing extra stuff?

$ docker run localhost/foo
Unable to find image 'localhost/foo:latest' locally
docker: Error response from daemon: Get "http://localhost/v2/": dial tcp 127.0.0.1:80: connect: connection refused.

Aha!

Do we want to do it?

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Releasing guix binary in Docker format too?

[Pjotr Prins]

Hi Danny!

On Tue, Sep 22, 2020 at 11:51:05AM +0200, Danny Milosavljevic wrote:
> Now, Heads has most of its stuff on CI servers that use Docker for a lot of
> things.

It would be very interesting to have a Docker registry for a number of
(complicated) Guix packages. We have bioinformatics packages living in
channels such as
http://git.genenetwork.org/guix-bioinformatics/guix-bioinformatics.
Bonface built a GeneNetwork Docker image for CI. I would like to see
Docker images for vgtools, gemma, sambamba and other products we are
(co)developing.

A Guix Docker container could be the starting point for live installs
on highly divergent systems (including an HPC cluster we are
building).

> While I was researching gitlab, I came across a feature where they have their
> own Docker container registry on there.  Apparently, the Docker CLI can fetch
> from whatever server the user wants!
> 
> $ docker run [options] registry.gitlab.com/group/project/image [arguments]
> 
> That got me thinking, we could easily also release Guix on something like
> that.  Is our HTTP webserver enough to have a Docker registry, without
> installing extra stuff?
> 
> $ docker run localhost/foo
> Unable to find image 'localhost/foo:latest' locally
> docker: Error response from daemon: Get "http://localhost/v2/": dial tcp 127.0.0.1:80: connect: connection refused.
> 
> Aha!
> 
> Do we want to do it?

I say YES :). If we have a way to create these containers we can
distribute them through http IPFS. We'll help write a nice discovery
web interface which initially can be a simple web page. I don't think
we need Docker containers for every package. But if there is a demand
for a specific tool it can be very helpful to distribute software that
way.

In fact, one software I am writing depends on a host of programming
languages including Python, Racket and Rust. A Guix Docker container
is by far the simplest solution to distribute the software. And Guix
containers are tiny - which is really nice.

Pj.

Re: Releasing guix binary in Docker format too?

[zimoun]

Hi,

(Sorry Dany I have overlooked this email.)


> > While I was researching gitlab, I came across a feature where they have their
> > own Docker container registry on there.  Apparently, the Docker CLI can fetch
> > from whatever server the user wants!
> >
> > $ docker run [options] registry.gitlab.com/group/project/image [arguments]
> >
> > That got me thinking, we could easily also release Guix on something like
> > that.  Is our HTTP webserver enough to have a Docker registry, without
> > installing extra stuff?

Docker images can be quickly really big.  Can Berlin support such traffic?


> > $ docker run localhost/foo
> > Unable to find image 'localhost/foo:latest' locally
> > docker: Error response from daemon: Get "http://localhost/v2/": dial tcp 127.0.0.1:80: connect: connection refused.
> >
> > Aha!
> >
> > Do we want to do it?
>
> I say YES :). If we have a way to create these containers we can
> distribute them through http IPFS. We'll help write a nice discovery
> web interface which initially can be a simple web page. I don't think
> we need Docker containers for every package. But if there is a demand
> for a specific tool it can be very helpful to distribute software that
> way.

At FOSDEM19, in the session "world domination", we discussed a service
where the user provides a manifest.scm file or click to select various
packages (and optionally a channels.scm file) and then the service
returns the Docker image ready to be pulled (or docker run URL).
Well, this killer-feature "service" seems a far objective. :-)

Proposing a Docker image ready-to-use containing the package manager
could be nice and a first step in that direction, IMHO.

The same way <http://guix.gnu.org/en/download/latest/>, maybe adding a
Docker (or Singularity) image would be nice to have.


> In fact, one software I am writing depends on a host of programming
> languages including Python, Racket and Rust. A Guix Docker container
> is by far the simplest solution to distribute the software. And Guix
> containers are tiny - which is really nice.

Guix containers tiny?  It depends on what you pack. :-)
(For example, only Emacs is not tiny.)

All the best,
simon

Re: Releasing guix binary in Docker format too?

[Danny Milosavljevic]

[-- Attachment #1: Type: text/plain, Size: 464 bytes --]

Also, while doing that, using the guix binary 1.1.0 tarball from the website and
issuing guix pull (ONLY), a lot of weird stuff is updated, like libx11, fribidi,
graphviz, cairo, pixman, libjpeg-turbo, pango etc.  Is that really necessary?
I guess it's because of the profile hooks, but still... why.

I want to stress that the profile I'm building contains only guix, the package
manager.  Does that really need all those (presumably build-) dependencies?

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Releasing guix binary in Docker format too?

[zimoun]

Hi Danny,

Back to this.  Since it would be nice to try to release (somewhere?)
an experimental guix Docker image.

On Thu, 24 Sep 2020 at 10:00, Danny Milosavljevic
<dannym@scratchpost.org> wrote:

> Also, while doing that, using the guix binary 1.1.0 tarball from the website and
> issuing guix pull (ONLY), a lot of weird stuff is updated, like libx11, fribidi,
> graphviz, cairo, pixman, libjpeg-turbo, pango etc.  Is that really necessary?
> I guess it's because of the profile hooks, but still... why.

Well, all these come from 'graphviz'.

--8<---------------cut here---------------start------------->8---
for target in libx11 fribidi graphviz cairo pixman libjpeg-turbo pango;
do
    printf "guix->$target: ";
    guix graph --path guix $target \
           |head -n2|tail -n1;
done

guix->libx11: graphviz@2.42.3
guix->fribidi: graphviz@2.42.3
guix->graphviz: graphviz@2.42.3
guix->cairo: graphviz@2.42.3
guix->pixman: graphviz@2.42.3
guix->libjpeg-turbo: graphviz@2.42.3
guix->pango: graphviz@2.42.3
--8<---------------cut here---------------end--------------->8---

And AFAIU, this 'graphivz' package is only used to produce the images
in the manual.  Similarly, the package 'texinfo' which pulls *a lot*
could be removed.

Well, maybe it is worth to maintain a "guix-minimal" package and to
inherit from this one to have the current 'guix' package.  WDYT?


> I want to stress that the profile I'm building contains only guix, the package
> manager.  Does that really need all those (presumably build-) dependencies?

I agree.


All the best,
simon

Re: Releasing guix binary in Docker format too?

[Ludovic Courtès]

Hi,

zimoun <zimon.toutoune@gmail.com> skribis:

> Back to this.  Since it would be nice to try to release (somewhere?)
> an experimental guix Docker image.

Among the issues to be addressed is whether we can push to DockerHub
with free software only (there was a discussion recently on IRC about a
tool to do that, but I don’t remember the details.)

> guix->libx11: graphviz@2.42.3
> guix->fribidi: graphviz@2.42.3
> guix->graphviz: graphviz@2.42.3
> guix->cairo: graphviz@2.42.3
> guix->pixman: graphviz@2.42.3
> guix->libjpeg-turbo: graphviz@2.42.3
> guix->pango: graphviz@2.42.3
>
> And AFAIU, this 'graphivz' package is only used to produce the images
> in the manual.  Similarly, the package 'texinfo' which pulls *a lot*
> could be removed.

These are build-time dependencies so that should only be a problem when
one has to build ‘guix’ from source.

Thanks,
Ludo’.

Re: Releasing guix binary in Docker format too?

[zimoun]

Hi,

(adding Jelle because of 'skopeo' :-))

On Wed, 21 Oct 2020 at 11:12, Ludovic Courtès <ludo@gnu.org> wrote:

> > Back to this.  Since it would be nice to try to release (somewhere?)
> > an experimental guix Docker image.
>
> Among the issues to be addressed is whether we can push to DockerHub
> with free software only (there was a discussion recently on IRC about a
> tool to do that, but I don’t remember the details.)

The tool is 'skopeo' and packaged in Guix.

However, push to DockerHub requires an account (by Guix project) which
requires... probably non-free JS, at least once.  Then pulling does
not require an account so it is compliant with GNU standards---at as I
understand them.  Therefore, the question is: does the GNU Guix
project agree to use non-free tools to spread freedom?  (Note, GNU
Emacs should do something like that with Windows and MacOS
<https://www.gnu.org/software/emacs/download.html#nonfree>; I mean, it
is impossible to port Emacs and maintain it on Windows or MacOS
without owning at least one machine running Windows/MacOS.)

On the other hand, Guix could host its own registry
<https://docs.docker.com/registry/>.  Or maybe, an "institution" from
Guix HPC could help.

Last, 'skopeo' seems able to fetch "tarball images".  Therefore, on
the Guix side, it is nothing more than another item in the store.
However, it means that the user must have 'skopeo' to fetch the image.
It really decreases the audience.  Well, maybe it is the easiest first
experimental move:

 - generate by the CI the Docker image as tarball
 - add one experimental box on <http://guix.gnu.org/en/download/>
 - write the 'skopeo' command line

WDYT?


All the best,
simon

Re: Releasing guix binary in Docker format too?

[Ludovic Courtès]

Hi,

zimoun <zimon.toutoune@gmail.com> skribis:

> The tool is 'skopeo' and packaged in Guix.
>
> However, push to DockerHub requires an account (by Guix project) which
> requires... probably non-free JS, at least once.

Hmm OK.  Users of Docker wouldn’t have to create an account and run the
non-free JS, so this is probably acceptable.

The next problem is: what would this image contain?  ‘guix pack guix’ is
probably not enough because people expect the daemon to be running.
Adding ‘--entry-point=bin/guix-daemon’ probably isn’t good either
because ‘guix-daemon’ never exits, which I think is contrary to what
users expect.

For the same reason, an image produced by ‘guix system docker-image’ may
not be OK because ‘shepherd’ never returns.

Should we change ‘guix system docker-image’ such that ‘shepherd’ is
started in the background?

Thanks,
Ludo’.

Re: Releasing guix binary in Docker format too?

[Danny Milosavljevic]

[-- Attachment #1: Type: text/plain, Size: 1823 bytes --]

Hi Ludo,

On Wed, 21 Oct 2020 17:42:36 +0200
Ludovic Courtès <ludo@gnu.org> wrote:

> zimoun <zimon.toutoune@gmail.com> skribis:
> 
> > The tool is 'skopeo' and packaged in Guix.
> >
> > However, push to DockerHub requires an account (by Guix project) which
> > requires... probably non-free JS, at least once.  
> 
> Hmm OK.  Users of Docker wouldn’t have to create an account and run the
> non-free JS, so this is probably acceptable.
> 
> The next problem is: what would this image contain?  ‘guix pack guix’ is
> probably not enough because people expect the daemon to be running.
> Adding ‘--entry-point=bin/guix-daemon’ probably isn’t good either
> because ‘guix-daemon’ never exits, which I think is contrary to what
> users expect.

It really depends on how people use the images.

It's totally normal for docker containers to start services they need in
the foreground on container startup, and people then entering that
running container from the outside anyway (!) using docker tools.

The guix I packaged on https://gitlab.com/daym/guix-on-docker/ is
specifically used as a test runner, which is why I background
guix-daemon there.  Otherwise, the gitlab test runners would hang waiting
for guix-daemon to finish (which never happens).

If I wouldn't need that, I'd totally not background stuff.

> For the same reason, an image produced by ‘guix system docker-image’ may
> not be OK because ‘shepherd’ never returns.
> 
> Should we change ‘guix system docker-image’ such that ‘shepherd’ is
> started in the background?

Why?  Docker tools are perfectly capable of entering a docker container
where a foreground process is running.

Also, one has to be careful that the container doesn't get stopped since
the last process finished.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Releasing guix binary in Docker format too?

[Ludovic Courtès]

Hi!

Danny Milosavljevic <dannym@scratchpost.org> skribis:

> On Wed, 21 Oct 2020 17:42:36 +0200
> Ludovic Courtès <ludo@gnu.org> wrote:
>
>> zimoun <zimon.toutoune@gmail.com> skribis:
>> 
>> > The tool is 'skopeo' and packaged in Guix.
>> >
>> > However, push to DockerHub requires an account (by Guix project) which
>> > requires... probably non-free JS, at least once.  
>> 
>> Hmm OK.  Users of Docker wouldn’t have to create an account and run the
>> non-free JS, so this is probably acceptable.
>> 
>> The next problem is: what would this image contain?  ‘guix pack guix’ is
>> probably not enough because people expect the daemon to be running.
>> Adding ‘--entry-point=bin/guix-daemon’ probably isn’t good either
>> because ‘guix-daemon’ never exits, which I think is contrary to what
>> users expect.
>
> It really depends on how people use the images.
>
> It's totally normal for docker containers to start services they need in
> the foreground on container startup, and people then entering that
> running container from the outside anyway (!) using docker tools.

OK so this:

 guix pack guix -f docker -S /bin=bin \
   --entry-point="bin/guix-daemon --disable-chroot"

might work, right?

Someone needs to try because the devil is in the details.

To me there are still important unknowns regarding an official Docker
image (how to build it, how to upload it), so I’m not confident about
getting that done in time for 1.2, but progress on that front is welcome
anytime!

Thanks,
Ludo’.

Re: Releasing guix binary in Docker format too?

[zimoun]

Hi,

On Fri, 06 Nov 2020 at 10:59, Ludovic Courtès <ludo@gnu.org> wrote:

>  guix pack guix -f docker -S /bin=bin \
>    --entry-point="bin/guix-daemon --disable-chroot"
>
> might work, right?
>
> Someone needs to try because the devil is in the details.

I will try to give a try.  If someone does not beat me.


> To me there are still important unknowns regarding an official Docker
> image (how to build it, how to upload it), so I’m not confident about
> getting that done in time for 1.2, but progress on that front is welcome
> anytime!

I am proposing to add an experimental box to:

  https://guix.gnu.org/en/download/latest/

Docker image with maybe a link to Cookbook (or elsewhere).  It is the
best way to have feedback. :-)

Yeah, you are right, it is premature to communicate via the release v1.2
without more tests.


All the best,
simon

Re: Releasing guix binary in Docker format too?

[Danny Milosavljevic]

[-- Attachment #1: Type: text/plain, Size: 1522 bytes --]

Hi,

On Fri, 06 Nov 2020 13:47:02 +0100
zimoun <zimon.toutoune@gmail.com> wrote:

> On Fri, 06 Nov 2020 at 10:59, Ludovic Courtès <ludo@gnu.org> wrote:
> 
> >  guix pack guix -f docker -S /bin=bin \
> >    --entry-point="bin/guix-daemon --disable-chroot"

Why --disable-chroot here?  (I'm not objecting to it)

> > might work, right?
> >
> > Someone needs to try because the devil is in the details.  
> 
> I will try to give a try.  If someone does not beat me.

Please do.

Note: guix-daemon needs a usergroup in order to actually usefully build
things, with at least one member user.

If Docker actually has first-class composition operators (I don't know), then
that should not go into the same image--but that means the end user has to 
provide a /etc/group and /etc/passwd with at least the "guixbuild" group and
at least one member in that group as a composable image[maybe 1] in order
for guix-daemon to actually work.

Also, guix needs /etc/services for http and https and so on to resolve.

Or you could just add those to the main guix docker image--but I think that
would be kinda weird (even though I'm doing it for mine--but it has a kind
of narrow use case where this is fine for the time being).

Or provide the host /etc/passwd using the "-v" command line option (that
would be kinda weird, too).

Or expect the user to always create a dockerfile and use "FROM" to derive
from the offical image.

[1] https://docs.docker.com/compose/compose-file/compose-file-v2/

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Releasing guix binary in Docker format too?

[Ludovic Courtès]

Hi Danny,

Danny Milosavljevic <dannym@scratchpost.org> skribis:

> On Fri, 06 Nov 2020 13:47:02 +0100
> zimoun <zimon.toutoune@gmail.com> wrote:
>
>> On Fri, 06 Nov 2020 at 10:59, Ludovic Courtès <ludo@gnu.org> wrote:
>> 
>> >  guix pack guix -f docker -S /bin=bin \
>> >    --entry-point="bin/guix-daemon --disable-chroot"
>
> Why --disable-chroot here?  (I'm not objecting to it)

Because otherwise the container needs to be started with ‘--privileged’
if I’m not mistaken.  But maybe that’s OK?

> Note: guix-daemon needs a usergroup in order to actually usefully build
> things, with at least one member user.

So to me that means we need to augment ‘guix pack -f docker’ so it can
add the right Docker metadata regarding user accounts.

> If Docker actually has first-class composition operators (I don't know), then
> that should not go into the same image--but that means the end user has to 
> provide a /etc/group and /etc/passwd with at least the "guixbuild" group and
> at least one member in that group as a composable image[maybe 1] in order
> for guix-daemon to actually work.
>
> Also, guix needs /etc/services for http and https and so on to resolve.

Right, it’s just about adding ‘net-base’ to the image.

Thanks,
Ludo’.

Re: Releasing guix binary in Docker format too?

[zimoun]

Hi,

>> >  guix pack guix -f docker -S /bin=bin \
>> >    --entry-point="bin/guix-daemon --disable-chroot"

[...]

>> > Someone needs to try because the devil is in the details.  
>> 
>> I will try to give a try.  If someone does not beat me.
>
> Please do.

I did and probably am lost in Docker details (the mentioned devil? ;-))

--8<---------------cut here---------------start------------->8---
$ docker load < \
> $(guix time-machine --branch=version-1.2.0 \
> -- pack guix -f docker -S /bin=bin  --entry-point="bin/guix-daemon")
Updating channel 'guix' from Git repository at 'https://git.savannah.gnu.org/git/guix.git'...
Loaded image: guix:latest

$ docker images --format "{{.Repository}} {{.Tag}} {{.ID}} {{.Size}}"
guix latest 7a99e79f803c 502MB

$ docker run -d --name guix guix:latest
7ecfb3311619d1265f1e91a45f587b28e3017c39735c9527b91bac5b0d5d49e1

$ docker ps
7ecfb3311619 guix:latest "/gnu/store/62iy7phi…" 5 seconds ago Up 4 seconds guix

$ docker exec guix guix pack hello
user with UID 0 not found
substitute: guix substitute: warning: ACL for archive imports seems to be uninitialized, substitutes may be unavailable
substitute: guix substitute: warning: ci.guix.gnu.org: host not found: Servname not supported for ai_socktype
The following derivations will be built:
   /gnu/store/p8w0ickl7n4psa5rkybx0q856phzxmpp-glibc-utf8-locales-2.31.drv

[...]

   /gnu/store/ll89ynpa8xrk2q4fvyg2aj8ig98w4zls-mkfontdir-1.0.7.tar.bz2.drv

building /gnu/store/wj0dnfmik81160y3zf85qf5hvkp2x1hi-libX11-1.6.10.tar.bz2.drv...
builder for `/gnu/store/wj0dnfmik81160y3zf85qf5hvkp2x1hi-libX11-1.6.10.tar.bz2.drv' failed with exit code 1
build of /gnu/store/wj0dnfmik81160y3zf85qf5hvkp2x1hi-libX11-1.6.10.tar.bz2.drv failed
View build log at '/var/log/guix/drvs/wj/0dnfmik81160y3zf85qf5hvkp2x1hi-libX11-1.6.10.tar.bz2.drv.bz2'.
building /gnu/store/4d3cjvjrgs0q38fs3bgfajcw7apc0vml-Python-3.5.9.tar.xz.drv...
cannot build derivation `/gnu/store/mvf88n2v90jjxg9n8b315p22r6jrkbyb-libx11-1.6.A.drv': 1 dependencies couldn't be built
killing process 29
guix pack: error: build of `/gnu/store/mvf88n2v90jjxg9n8b315p22r6jrkbyb-libx11-1.6.A.drv' failed
--8<---------------cut here---------------end--------------->8---

I have tried various Docker option and I have failed to figure out the
correct one.  I should miss the obvious.  My eyes are a bit tired these
days.  Any hint?


BTW, the tool skopeo failed in the branch version-1.2.0 as ’weather’
confirms.  Therefore the Cookbook recipe will be for the next release.

Cheers,
simon

Re: Releasing guix binary in Docker format too?

[Danny Milosavljevic]

[-- Attachment #1: Type: text/plain, Size: 1515 bytes --]

Hi,

On Sun, 15 Nov 2020 19:30:44 +0100
zimoun <zimon.toutoune@gmail.com> wrote:

> $ docker exec guix guix pack hello
> user with UID 0 not found

Docker needs to generate a /etc/passwd with uid 0 and the guix build user accounts, and a /etc/group with the guixbuild group; and whatever other users the things that are composed together using docker compose[1] require.  How does this work in Docker-land ?

> substitute: guix substitute: warning: ACL for archive imports seems to be uninitialized, substitutes may be unavailable

We need to configure substitute server acls (/etc/guix/acl).  So this one is totally our fault.

> substitute: guix substitute: warning: ci.guix.gnu.org: host not found: Servname not supported for ai_socktype

Docker needs to generate /etc/services with a union of the required services of all the things that are composed together using docker compose.

All in all, these are all obvious things which guix-on-docker already works around (I don't know the official way that surely exists--hence the workarounds).

The question is: since Docker supports composition[1], how do they handle this standard case ?  How can we get Docker to generate /etc/services, /etc/passwd and /etc/group for the composed docker image ?

> BTW, the tool skopeo failed in the branch version-1.2.0 as ’weather’
> confirms.  Therefore the Cookbook recipe will be for the next release.

(I can't help with skopeo, never used it before)

[1] https://docs.docker.com/compose/

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Releasing guix binary in Docker format too?

[Bengt Richter]

Hi,

On +2020-11-17 17:38:09 +0100, Danny Milosavljevic wrote:
> Hi,
> 
> On Sun, 15 Nov 2020 19:30:44 +0100
> zimoun <zimon.toutoune@gmail.com> wrote:
> 
> > $ docker exec guix guix pack hello
> > user with UID 0 not found
> 
> Docker needs to generate a /etc/passwd with uid 0 and the guix build user accounts, and a /etc/group with the guixbuild group; and whatever other users the things that are composed together using docker compose[1] require.  How does this work in Docker-land ?
>
How much would change if the guix daemon were implemented a little differently?
E.g., (quoted from [1]), does the following mean that the guix daemon potentially could run "projects"
instead of guixbuilder* to create "Multiple isolated environments on a single host" ?

Is it suggestive to anyone else?

--8<---------------cut here---------------start------------->8---
The features of Compose that make it effective are:

    Multiple isolated environments on a single host
    Preserve volume data when containers are created
    Only recreate containers that have changed
    Variables and moving a composition between environments

Multiple isolated environments on a single host

Compose uses a project name to isolate environments from each other. You can make use of this project name in several different contexts:

    on a dev host, to create multiple copies of a single environment, such as when you want to run a stable copy for each feature branch of a project
    on a CI server, to keep builds from interfering with each other, you can set the project name to a unique build number
    on a shared host or dev host, to prevent different projects, which may use the same service names, from interfering with each other

The default project name is the basename of the project directory. You can set a custom project name by using the -p command line option or the COMPOSE_PROJECT_NAME environment variable.
--8<---------------cut here---------------end--------------->8---
[...]
> 
> The question is: since Docker supports composition[1], how do they handle this standard case ?  How can we get Docker to generate /etc/services, /etc/passwd and /etc/group for the composed docker image ?
>
I guess this question would morph if guixbuilder*  became "projects",
where "you can set the project name to a unique build number".
[...]
> 
> [1] https://docs.docker.com/compose/
-- 
Regards,
Bengt Richter

Re: Releasing guix binary in Docker format too?

[Ryan Prior]

[-- Attachment #1: Type: text/plain, Size: 2166 bytes --]

On November 18, 2020, Bengt Richter <bokr@bokr.com> wrote:
> E.g., (quoted from [1]), does the following mean that the guix daemon
> potentially could run "projects"
> instead of guixbuilder* to create "Multiple isolated environments on a
> single host" ?
> The features of Compose that make it effective are:
>
>  Multiple isolated environments on a single host
>  Preserve volume data when containers are created
>  Only recreate containers that have changed
>  Variables and moving a composition between environments

I love docker-compose and have used it every day. There are a bunch of
workflows I use compose for that I haven't figured out how to replace
with Guix yet, and I'm not in a huge hurry, but it would be great to see
features that are designed to appeal to Compose users.

For example, a common thing I do with docker-compose is to provide a
compose file with a service I'm working on that brings up the service
with all its dependencies (eg databases,  proxies, etc.) in a dev
configuration, with the source code from the repository mapped into the
container and live-reload enabled. That creates a "two-step" dev
environment for the service: you clone the repo then run "docker-compose
up."

Hashicorp has been working on Waypoint[1], a free software tool to
extend that concept even further by adding deployment and release
capabilities. Now you clone a repo and you have one tool to build,
deploy and release the software contained therein.

I'm interested to use Guix for similar purposes, but there are a few
primary obstacles in the way:
1) I don't have enough experience yet defining systems using Guix
similar to what I'd define using docker-compose, and I haven't found
documentation or examples yet that makes it seem within reach
2) The machines I'm deploying to run Kubernetes, not Guix System, so if
I do use Guix to build a bundle for deployment its utility ends when I
export to a Docker container image.
3) I collaborate with developers who use macOS and Windows and so can't
easily install and use Guix.

[1] https://www.waypointproject.io/

[-- Attachment #2: Type: text/html, Size: 4479 bytes --]

Re: Releasing guix binary in Docker format too?

[zimoun]

Hi,

Thank you for your feedback.

On Wed, 18 Nov 2020 at 14:53, Ryan Prior <ryanprior@hey.com> wrote:

>>  Multiple isolated environments on a single host
>>  Preserve volume data when containers are created
>>  Only recreate containers that have changed
>>  Variables and moving a composition between environments

[...]

> For example, a common thing I do with docker-compose is to provide a compose file with a service I'm working on that brings up the service with all its dependencies (eg databases,  proxies, etc.) in a dev configuration, with the source code from the repository mapped into the container and live-reload enabled. That creates a "two-step" dev environment for the service: you clone the repo then run "docker-compose up."

Do you have configuration files of minimal examples?  Just to
understand what you mean using concrete examples.


> Hashicorp has been working on Waypoint[1], a free software tool to extend that concept even further by adding deployment and release capabilities. Now you clone a repo and you have one tool to build, deploy and release the software contained therein.
>
> I'm interested to use Guix for similar purposes, but there are a few primary obstacles in the way:
> 1) I don't have enough experience yet defining systems using Guix similar to what I'd define using docker-compose, and I haven't found documentation or examples yet that makes it seem within reach
> 2) The machines I'm deploying to run Kubernetes, not Guix System, so if I do use Guix to build a bundle for deployment its utility ends when I export to a Docker container image.

Thanks for pointing this out.

> 3) I collaborate with developers who use macOS and Windows and so can't easily install and use Guix.

I speak for myself, the idea is to build Docker images using Guix for this case.

Otherwise, I do not know if it is worth but someone pointed this [1]
for Windows.

1: <https://github.com/giuliano108/guix-packages/blob/master/notes/Guix-on-WSL2.md>

All the best,
simon

Size of package 'guix'

[zimoun]

Hi,

On Wed, 21 Oct 2020 at 11:12, Ludovic Courtès <ludo@gnu.org> wrote:

> > guix->libx11: graphviz@2.42.3
> > guix->fribidi: graphviz@2.42.3
> > guix->graphviz: graphviz@2.42.3
> > guix->cairo: graphviz@2.42.3
> > guix->pixman: graphviz@2.42.3
> > guix->libjpeg-turbo: graphviz@2.42.3
> > guix->pango: graphviz@2.42.3
> >
> > And AFAIU, this 'graphivz' package is only used to produce the images
> > in the manual.  Similarly, the package 'texinfo' which pulls *a lot*
> > could be removed.
>
> These are build-time dependencies so that should only be a problem when
> one has to build ‘guix’ from source.

What do you mean by "build 'guix' from source"?  My point is
"guix-minimal" could be more minimal; not only remove "guile-ssh" but
also "texinfo", "grapviz", "po4a" at least and maybe more.

Cheers,
simon

Re: Size of package 'guix'

[Ludovic Courtès]

zimoun <zimon.toutoune@gmail.com> skribis:

> On Wed, 21 Oct 2020 at 11:12, Ludovic Courtès <ludo@gnu.org> wrote:
>
>> > guix->libx11: graphviz@2.42.3
>> > guix->fribidi: graphviz@2.42.3
>> > guix->graphviz: graphviz@2.42.3
>> > guix->cairo: graphviz@2.42.3
>> > guix->pixman: graphviz@2.42.3
>> > guix->libjpeg-turbo: graphviz@2.42.3
>> > guix->pango: graphviz@2.42.3
>> >
>> > And AFAIU, this 'graphivz' package is only used to produce the images
>> > in the manual.  Similarly, the package 'texinfo' which pulls *a lot*
>> > could be removed.
>>
>> These are build-time dependencies so that should only be a problem when
>> one has to build ‘guix’ from source.
>
> What do you mean by "build 'guix' from source"?

I mean that it doesn’t matter that ‘guix’ depends on ‘graphviz’ at build
time as long as you can get a substitute for it.  IOW, ‘graphviz’ is not
among the references of ‘guix’:

--8<---------------cut here---------------start------------->8---
$ guix size guix |grep graphviz
$ echo $?
1
--8<---------------cut here---------------end--------------->8---

Ludo’.

Re: Size of package 'guix'

[zimoun]

On Wed, 21 Oct 2020 at 17:44, Ludovic Courtès <ludo@gnu.org> wrote:

> >> These are build-time dependencies so that should only be a problem when
> >> one has to build ‘guix’ from source.
> >
> > What do you mean by "build 'guix' from source"?
>
> I mean that it doesn’t matter that ‘guix’ depends on ‘graphviz’ at build
> time as long as you can get a substitute for it.  IOW, ‘graphviz’ is not
> among the references of ‘guix’:

Maybe we are not talking about the same thing. :-)

Let's take the example of building Guix on GitLab CI.  So, it needs to
fetch a lot (as Danny said) on their infra, then build and publish the
produced "artifact".  The next time it will build, then it will fetch
a lot, again.   Well, it depends on the caching facilities and maybe
these dependencies are not required for this particular use case.


Cheers,
simon

Release: Docker Image? DockerHub? skopeo?

[zimoun]

Hi,

Some days ago, we discussed on #guix about releasing Docker images
ready-to-use.  And you pointed your project:

  https://gitlab.com/daym/guix-on-docker/


First question for Danny: is it ready to include it to the release?
Include meaning communicate about it – with the status experimental.

Second question for Ludo: does the Guix project officially push a Docker
image to DockerHub?

Third question for all: do we add an experimental box on the “Download”
section of the website [0]?

Fourth question for Jelle: do you have time to add examples to the
Cookbook using the package ’skopeo’?  If not, who could?


This is a follow-up to the thread, started here [1] and arguments about
DockerHub and co. there [2] and Ludo’s comments [3].


0: http://guix.gnu.org/en/download/
1: https://lists.gnu.org/archive/html/guix-devel/2020-09/msg00212.html
2: https://lists.gnu.org/archive/html/guix-devel/2020-10/msg00352.html
3: https://lists.gnu.org/archive/html/guix-devel/2020-10/msg00352.html


Feedback welcome.

All the best,
simon

Re: Release: Docker Image? DockerHub? skopeo?

[Jelle Licht]

Hello,

zimoun <zimon.toutoune@gmail.com> writes:

> Some days ago, we discussed on #guix about releasing Docker images
> ready-to-use.  And you pointed your project:
>
>   https://gitlab.com/daym/guix-on-docker/

[snip]

> Fourth question for Jelle: do you have time to add examples to the
> Cookbook using the package ’skopeo’?  If not, who could?

I was actually putting that on the back burner for now, as I felt I
first needed some time to understand Danny's `guix-on-docker' better in
order for the skopeo stuff to warrant a cookbook section. The entire
workflow as I use it now can be summarized by the following three-liner:

--8<---------------cut here---------------start------------->8---
REGISTRY_IMAGE=registry.gitlab.com/group/project 
IMG=$(guix pack --save-provenance -f docker --entry-point=bin/hello hello)
skopeo copy --dest-creds USERNAME[:PASSWORD] \
   "docker-archive://$IMG" \
   "docker://${REGISTRY_IMAGE}:latest"
--8<---------------cut here---------------end--------------->8---

> This is a follow-up to the thread, started here [1] and arguments about
> DockerHub and co. there [2] and Ludo’s comments [3].

Our timing is pretty bad here, as Dockerhub updated some of their
policies only two days ago regarding usage limits for both 'anonymous'
and free tier registered users [4]. 

> 1: https://lists.gnu.org/archive/html/guix-devel/2020-09/msg00212.html
> 2: https://lists.gnu.org/archive/html/guix-devel/2020-10/msg00352.html
> 3: https://lists.gnu.org/archive/html/guix-devel/2020-10/msg00352.html

[4]: https://www.docker.com/blog/what-you-need-to-know-about-upcoming-docker-hub-rate-limiting/

- Jelle

Re: Release: Docker Image? DockerHub? skopeo?

[Danny Milosavljevic]

[-- Attachment #1: Type: text/plain, Size: 911 bytes --]

Hi zimoun,

On Wed, 04 Nov 2020 12:05:28 +0100
zimoun <zimon.toutoune@gmail.com> wrote:

> Some days ago, we discussed on #guix about releasing Docker images
> ready-to-use.  And you pointed your project:
> 
>   https://gitlab.com/daym/guix-on-docker/
> 
> First question for Danny: is it ready to include it to the release?
> Include meaning communicate about it – with the status experimental.

Yes.  It's done.

For the use case I'm using it for, guix-on-docker is final now.
I doubt that I'll change anything in it, and certainly nothing major.

Right now, my gitlab project automatically tracks guix from git master on
savannah and rebuilds that and pushes the result to gitlab's docker
registry (every time someone commits to guix master).

If we wanted the gitlab project to track a release tag, we'd have to
configure it to do building from a tag, too.  If so, please tell me.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Release: Docker Image? DockerHub? skopeo?

[zimoun]

Hi Danny,

Sorry to be late.

On Wed, 04 Nov 2020 at 15:50, Danny Milosavljevic <dannym@scratchpost.org> wrote:

>> Some days ago, we discussed on #guix about releasing Docker images
>> ready-to-use.  And you pointed your project:
>> 
>>   https://gitlab.com/daym/guix-on-docker/

[...]

> For the use case I'm using it for, guix-on-docker is final now.
> I doubt that I'll change anything in it, and certainly nothing major.

I have simply tried:

   docker run -d --name guix registry.gitlab.com/daym/guix-on-docker
   docker exec guix guix pack hello

Wow!  The image is huge:

5.23GB registry.gitlab.com/daym/guix-on-docker latest 884b4a4ddf61 

Is it expected?   BTW, I am noting that there are couple of layers:

 "RootFS": {
     "Type": "layers",
     "Layers": [
         "sha256:c3deb477b7310360aaa445de77aa7c16f4cee76db5163c2c70573f49b2b26943",
         "sha256:8128b7241acd71f13fc7bdcc5cda139e0ab75a2a987db6268d8d0f74b0974fe9",
         "sha256:ba90a34fcea86f444a3ceb192ad2b6ad2d1ba86c1df4e9b4746114d86472f1a8",
         "sha256:c2f9dceb24a4df38f7e708db969792ac3061270f6322411b503c347ff98399b0",
         "sha256:39e027c52cb13e22b7b2a729b07e23f3574662ee482fb5c09b42c6cf51020cf6",
         "sha256:5a938600c9db6ce11a1e561b2c57448a999970ebeb82539276afead86bc44319",
         "sha256:541970bba9516720676f088c629ff12048ce478a2bd5e2b7da21c50feaa619f1",
         "sha256:3cc6fdb7c0cc6b5bca938df9a5fa03fe3438333f3c9f686f8a90b1fc785ff07f",
         "sha256:298ab852047104192cc6c5347154843f3669810c215377bcc280dbfb89277708",
         "sha256:a5a94f3837bafc49d54ef88f98eb3e162e78248c8b055ab17fd057b2d57d6d2e",
         "sha256:697ddc7e64d6cd06293d0840abc4876c3f27fccc70e35efd4f07d84d0e3e6d63" 
     ]
 },

I am not sure to understand why.  It reminds me this previous thread:

<https://yhetil.org/guix-user/CAKjnHz1X26QhPEWZfOmAbEmZHQA1qdU_Dq7U6kVXUqG7vCvuhw@mail.gmail.com>

Well, I do not know.


Then, the ’docker exec’ downloads \o/ and then fails with:

  guix pack: error: getting status of `/tmp': No such file or directory

Hum, as in the other message, I am sure I am missing the obvious with
Docker options.


> Right now, my gitlab project automatically tracks guix from git master on
> savannah and rebuilds that and pushes the result to gitlab's docker
> registry (every time someone commits to guix master).
>
> If we wanted the gitlab project to track a release tag, we'd have to
> configure it to do building from a tag, too.  If so, please tell me.

Currently, it builds master, right?  I have not looked but do you think
it could possible to track the both branches master and version-1.x.y?


Thanks,
simon

Re: Release: Docker Image? DockerHub? skopeo?

[Danny Milosavljevic]

[-- Attachment #1: Type: text/plain, Size: 1303 bytes --]

Hi,

On Sun, 15 Nov 2020 22:30:51 +0100
zimoun <zimon.toutoune@gmail.com> wrote:

> Wow!  The image is huge:
> 
> 5.23GB registry.gitlab.com/daym/guix-on-docker latest 884b4a4ddf61 
> 
> Is it expected?

Yes, because Docker is silly.  It will continue to grow.
It should be better when you use an external volume as /gnu/store and /var/guix.

> <https://yhetil.org/guix-user/CAKjnHz1X26QhPEWZfOmAbEmZHQA1qdU_Dq7U6kVXUqG7vCvuhw@mail.gmail.com>

> Then, the ’docker exec’ downloads \o/ and then fails with:

>   guix pack: error: getting status of `/tmp': No such file or directory
> 
> Hum, as in the other message, I am sure I am missing the obvious with
> Docker options.

Yeah, someone needs to create /tmp.  That someone is not Guix (it would be weird).

/tmp is a common directory, so it does not belong to any of the parts that
have been composed, or maybe there should be a standard "/tmp" part?

Is there such a standard "/tmp" part in Docker-land?

Similarly, but worse, with /etc/passwd, /etc/group, /etc/services and so on.

> Currently, it builds master, right?  I have not looked but do you think
> it could possible to track the both branches master and version-1.x.y?

Yes, right now it builds current master.

Sure, could easily make it do both.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Release: Docker Image? DockerHub? skopeo?

[Danny Milosavljevic]

[-- Attachment #1: Type: text/plain, Size: 333 bytes --]

Hi,

> Yeah, someone needs to create /tmp.  That someone is not Guix (it would be weird).

There is a workaround in /with-guix-daemon.scm in that guix-on-docker container,
which is how I'm using it.

The entrypoint I'm using (with the unmodified guix-on-docker image) is

   ["guix", "repl", "--", "/with-guix-daemon.scm"]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Release: Docker Image? DockerHub? skopeo?

[zimoun]

Hi Danny,

On Tue, 17 Nov 2020 at 17:31, Danny Milosavljevic <dannym@scratchpost.org> wrote:

>> Wow!  The image is huge:
>> 
>> 5.23GB registry.gitlab.com/daym/guix-on-docker latest 884b4a4ddf61 
>> 
>> Is it expected?
>
> Yes, because Docker is silly.  It will continue to grow.

Oh, bad news!  5.23GB is already too much.


> It should be better when you use an external volume as /gnu/store and /var/guix.
>
><https://yhetil.org/guix-user/CAKjnHz1X26QhPEWZfOmAbEmZHQA1qdU_Dq7U6kVXUqG7vCvuhw@mail.gmail.com>

Well, from the thread above, you be interested by the Chris’s message:

   <https://yhetil.org/guix-user/87h7vyxqrz.fsf@gmail.com/>


>> Hum, as in the other message, I am sure I am missing the obvious with
>> Docker options.
>
> Yeah, someone needs to create /tmp.  That someone is not Guix (it
> would be weird).

Created where?  Because /tmp is obviously on the host machine. :-)

If with the container:

--8<---------------cut here---------------start------------->8---
docker exec guix mkdir -h
OCI runtime exec failed: exec failed: container_linux.go:349: starting container process caused "exec: \"mkdir\": executable file not found in $PATH": unknown
--8<---------------cut here---------------end--------------->8---

Sorry to be so naive.


> /tmp is a common directory, so it does not belong to any of the parts that
> have been composed, or maybe there should be a standard "/tmp" part?
>
> Is there such a standard "/tmp" part in Docker-land?
>
> Similarly, but worse, with /etc/passwd, /etc/group, /etc/services and so on.

I do not know.


>> Currently, it builds master, right?  I have not looked but do you think
>> it could possible to track the both branches master and version-1.x.y?
>
> Yes, right now it builds current master.
>
> Sure, could easily make it do both.

Cool!  Thanks.


Cheers,
simon

Re: Release: Docker Image? DockerHub? skopeo?

[Danny Milosavljevic]

[-- Attachment #1: Type: text/plain, Size: 3158 bytes --]

Hi zimoun,

On Tue, 17 Nov 2020 18:30:57 +0100
zimoun <zimon.toutoune@gmail.com> wrote:

> > Yeah, someone needs to create /tmp.  That someone is not Guix (it
> > would be weird).  
> 
> Created where?

Created when creating the docker image--presumably by Docker when evaluating
Composefile.

AFAIK Docker images are supposed to be composed of multiple things using a
Composefile.[1]

So Docker has to have some mechanism to set up the shared space that all those
multiple things need.

Ideally, we/the user should use this mechanism to add /tmp--not create it
manually by some weird script in the running container.

> If with the container:
> 
> --8<---------------cut here---------------start------------->8---
> docker exec guix mkdir -h
> OCI runtime exec failed: exec failed: container_linux.go:349: starting container process caused "exec: \"mkdir\": executable file not found in $PATH": unknown

You have to use guix repl, then it will work.
There's no coreutils in there, just guix.

That's what /with-guix-daemon.scm in guix-on-docker does (it's already in the
image), among other weird things.

I paste it in full here:

(use-modules (ice-9 match))
(if (not (file-exists? "/tmp"))
    (mkdir "/tmp"))
(system "/root/.config/guix/current/bin/guix-daemon --build-users-group=guixbuild --disable-chroot &")
(sleep 2)
(match (command-line)
 ((me next-executable rest ...)
  (match (cons next-executable rest)
   ;; gitlab runner invokes "sh -c" for a shell existence check.
   ;; Replace it by bash.
   (("sh" "-c" rest ...)
    (apply execlp "guix" '("guix" "environment" "--ad-hoc" "bash" "coreutils" "--" "bash")))
   ((next-executable rest)
    (apply execlp next-executable (cons next-executable rest)))))
 ((me)
  ;; For gitlab runner, see <https://docs.gitlab.com/ee/ci/docker/using_docker_images.html#overriding-the-entrypoint-of-an-image>.
  (apply execlp "guix" '("guix" "repl"))))
(exit 1)

It's used as entrypoint by me manually (because it's full of weird workarounds
like this it's NOT the default entrypoint), overriding docker's entrypoint by
["guix", "repl", "--", "/with-guix-daemon.scm"].

> Sorry to be so naive.

No, I don't use docker that much--and when I do, it's to run simple images
others have created.  So I just really don't know how this is supposed to
be set up.  I mean there has to be a way to set this up--that is one of the
first things anyone would need--shared /tmp, /etc/passwd, /etc/group,
/etc/services and so on.  The parts that are composed together by Docker have
to negotiate a common version of those, right?

> > /tmp is a common directory, so it does not belong to any of the parts that
> > have been composed, or maybe there should be a standard "/tmp" part?
> >
> > Is there such a standard "/tmp" part in Docker-land?
> >
> > Similarly, but worse, with /etc/passwd, /etc/group, /etc/services and so on.  
> 
> I do not know.

I hope that others will chime in explaining what the standard way to do this
is.  The workaround above *does* work, though (and is the wrong thing to do).

[1] https://docs.docker.com/compose/

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Release: Docker Image? DockerHub? skopeo?

[Danny Milosavljevic]

[-- Attachment #1: Type: text/plain, Size: 269 bytes --]

Hmm, maybe I'm misunderstanding what Docker compose does entirely.

Reading the docs it seems to just manage multiple isolated Docker images and
deploy new ones?

What then is used to do the equivalent of a guix profile with multiple packages
in a Docker image?

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Release: Docker Image? DockerHub? skopeo?

[Ryan Prior]

[-- Attachment #1: Type: text/plain, Size: 1472 bytes --]

On November 17, 2020, Danny Milosavljevic <dannym@scratchpost.org>
wrote:
> Hmm, maybe I'm misunderstanding what Docker compose does entirely.

What docker-compose does is it creates a set of Docker containers for
you based on a configuration file. In that file you define services,
filesystem volumes and their mount points, virtual private networks,
environment variables, and so on. This saves you the trouble of running
a bunch of "docker run ..." commands to set up a cluster of containers.
But docker-compose doesn't do anything you couldn't already do with
Docker alone; it uses the Docker API to give you a more convenient
interface.

> Reading the docs it seems to just manage multiple isolated Docker
> images and
> deploy new ones?

That's exactly what it does.

> What then is used to do the equivalent of a guix profile with multiple
> packages
> in a Docker image?

Something a lot of Dockers users initially think you can do, but you
actually can't, is to merge two Docker images together. There's a "ruby"
image and there's a "python" image but you can't just say hey Docker I
want a "ruby+python" image. So there is no direct equivalent to a Guix
profile with multiple packages.

What you can do is start from one image, add what's needed from the
other to create a new base image, and then go from there. But you have
to manage that manually by running "apt" commands for Debian or "apk"
for Alpine or whatever, as part of your Dockerfile build process.

[-- Attachment #2: Type: text/html, Size: 3918 bytes --]

Re: Release: Docker Image? DockerHub? skopeo?

[Carlo Zancanaro]

Hey Danny,

On Wed, Nov 18 2020, Danny Milosavljevic wrote:
> Hmm, maybe I'm misunderstanding what Docker compose does 
> entirely.
>
> Reading the docs it seems to just manage multiple isolated 
> Docker images and deploy new ones?

Yep, that's all it does. It lets you link together ports and 
volumes, and expose ports to the host machine, but it doesn't do 
anything particularly magical. I think it only does things that 
you could have done by starting images with the right docker 
invocations.

> What then is used to do the equivalent of a guix profile with 
> multiple packages in a Docker image?

I don't think there is a way to compose multiple docker images 
into a single image.

Carlo

Re: Release: Docker Image? DockerHub? skopeo?

[zimoun]

Hi Danny,

Thank you for the explanations.  Make sense. :-)

On Tue, 17 Nov 2020 at 20:23, Danny Milosavljevic <dannym@scratchpost.org> wrote:

> No, I don't use docker that much--and when I do, it's to run simple images
> others have created.  So I just really don't know how this is supposed to
> be set up.  I mean there has to be a way to set this up--that is one of the
> first things anyone would need--shared /tmp, /etc/passwd, /etc/group,
> /etc/services and so on.  The parts that are composed together by Docker have
> to negotiate a common version of those, right?

I don’t use Docker that much neither–and when I do it’s to create
somehow Docker images via “guix pack -f docker” and run somehow this
simple image.  Well, I have never read cover-to-cover the Docker doc,
for example. :-)


> I hope that others will chime in explaining what the standard way to do this
> is.  The workaround above *does* work, though (and is the wrong thing to do).

Let’s postpone this Docker image work and start a new fresh thread once
v1.2.0 is published and the goal to have something for v1.3.0, well
that’s my point of view.


Cheers,
simon

Re: Release: Docker Image? DockerHub? skopeo?

[Danny Milosavljevic]

[-- Attachment #1: Type: text/plain, Size: 2075 bytes --]

Hi zimoun,

On Thu, 19 Nov 2020 10:21:16 +0100
zimoun <zimon.toutoune@gmail.com> wrote:

> Let’s postpone this Docker image work and start a new fresh thread once
> v1.2.0 is published and the goal to have something for v1.3.0, well
> that’s my point of view.

After Ryan Prior's E-Mail I'm pretty sure my workaround of creating /tmp,
/etc/passwd, /etc/group etc is what Docker actually expects one to do.

So we can just create those--either at runtime, or maybe even have

  guix system docker-image

do it at build time (if it doesn't already).

I have no opinion on when we should do that (at this release or the next
one), except to state that I am certain that it works (and pretty easily),
because guix-on-docker does that already and guix works just fine there.

That still leaves to explain how to prevent Docker from keeping older layers
when it doesn't need to.

In guix-on-docker I have a Dockerfile like this

FROM alpine:3.12 AS bootstrap-guix-on-alpine
[...]

FROM scratch AS guix-on-docker
COPY --from=bootstrap-guix-on-alpine /etc/guix /etc/guix
COPY --from=bootstrap-guix-on-alpine /var/guix /var/guix
COPY --from=bootstrap-guix-on-alpine /gnu /gnu
COPY --from=bootstrap-guix-on-alpine /usr/local /usr/local
COPY --from=bootstrap-guix-on-alpine /root/.config/guix /root/.config/guix
ADD set-mtimes.scm /
ADD etc/passwd /etc
ADD etc/group /etc
ADD etc/services /etc
ADD with-guix-daemon.scm /
RUN ["/usr/local/bin/guix", "repl", "/set-mtimes.scm"]

in order to prevent Docker from keeping older layers[1].

The "set-times.scm" invocation there is in order to fix the timestamps.

"COPY --from=" does not preserve timestamps.  Then guile is very annoyed
because it can't use any of the ".go" files--because they are older than
the respective ".scm" files.

Using set-times.scm means it will create yet another layer where the only
difference is the timestamps--so it doubles the size of the resulting image.

But then it works.

[1] https://docs.docker.com/develop/develop-images/multistage-build/

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

/etc/passwd & co. in Docker images

[Ludovic Courtès]

Hi,

Danny Milosavljevic <dannym@scratchpost.org> skribis:

> After Ryan Prior's E-Mail I'm pretty sure my workaround of creating /tmp,
> /etc/passwd, /etc/group etc is what Docker actually expects one to do.

I’ve been pondering whether to create those files in ‘guix pack -f
docker’ and…

> ADD etc/passwd /etc
> ADD etc/group /etc
> ADD etc/services /etc
> ADD with-guix-daemon.scm /
> RUN ["/usr/local/bin/guix", "repl", "/set-mtimes.scm"]

… what you’re doing here suggest that ‘guix pack’ should indeed create
those files.

Thoughts?

Ludo’.

Re: /etc/passwd & co. in Docker images

[Ryan Prior]

[-- Attachment #1: Type: text/plain, Size: 974 bytes --]

On November 20, 2020, "Ludovic Courtès" <ludo@gnu.org> wrote:
> … what you’re doing here suggest that ‘guix pack’ should indeed create
> those files.

Oh I think it should be an option. Many Docker containers do not need
those things, but it would be great to have the option to include them.

Another option that would be nice, for the sake of `guix deploy` and
`guix pack`, is to produce a static build. I've tried adjusting some
Guix packages to produce static builds and seen dramatic reduction in
bundle sizes. I've brought up those observations in IRC and from those
convos I know most people aren't interested in static builds, but for
deployments to a cluster, a tiny container image with a single static
binary in it can be just the thing you want. This maybe plays into the
"parametric packages" thread that's happening in parallel, because I
could see offering this as a "static?" parameter on packages that
support it.

[-- Attachment #2: Type: text/html, Size: 3297 bytes --]

Re: /etc/passwd & co. in Docker images

[Danny Milosavljevic]

[-- Attachment #1: Type: text/plain, Size: 2877 bytes --]

Hi Ludo,

On Fri, 20 Nov 2020 12:34:07 +0100
Ludovic Courtès <ludo@gnu.org> wrote:

> Danny Milosavljevic <dannym@scratchpost.org> skribis:
> 
> > After Ryan Prior's E-Mail I'm pretty sure my workaround of creating /tmp,
> > /etc/passwd, /etc/group etc is what Docker actually expects one to do.  
> 
> I’ve been pondering whether to create those files in ‘guix pack -f
> docker’ and…
> 
> > ADD etc/passwd /etc
> > ADD etc/group /etc
> > ADD etc/services /etc
> > ADD with-guix-daemon.scm /
> > RUN ["/usr/local/bin/guix", "repl", "/set-mtimes.scm"]  
> 
> … what you’re doing here suggest that ‘guix pack’ should indeed create
> those files.
> 
> Thoughts?

If guix pack can be used to put multiple packages into one pack,
are then packs like profiles, or would that be too much?

Because the question is what to do if you invoke

  guix pack -f docker guix postgresql

.

Both need user accounts--and thus the total required user accounts (and thus
the contents of /etc/passwd) are the union of the respective required user
accounts of both packages.

So someone needs to merge those, and for that the packages need to require
these user accounts in the first place.

But usually in Guix it's Guix *services* that require user accounts and not
Guix packages.  (which makes sense!)

So I would suggest that

  guix system docker-image ...

create /etc/passwd by merging the required user accounts like described
above, but

  guix pack -f docker a b c

really can't do that.  I mean where would it know the requirements from?

But creating /tmp with the right permissions should be easy enough (?).

That leaves how we want to do a Guix release to a Docker registry.

I think if you pack guix-the-package-manager, the question is whether
there are scenarios (100% offloading for example) that allow you to
use guix without the guix daemon also running inside the docker container.

If so, it doesn't make sense to add /etc/passwd and /etc/services and so
on for guix-daemon in the guix pack case, especially since it singles out
one Guix package, guix, for special consideration.

Then you'd use the guix pack'ed image for using guix-the-package-manager
with 100% offload.

We should release a minimal guix-the-operating-system to a docker registry
so the guix-daemon also works (i.e. built using guix system docker-image).

You'd use this Docker system image when you want to use both guix and
guix-daemon.
 
Are there any downsides to just using a trimmed-down operating-system
definition in order to have a docker image with a working guix-daemon ?

Or we can instead add a static /etc/passwd in Docker after the fact and then
release that to a Docker registry--that's what guix-on-docker does now
(though that doubles the size of the result because Docker is being silly).

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: /etc/passwd & co. in Docker images

[Ryan Prior]

[-- Attachment #1: Type: text/plain, Size: 1054 bytes --]

On November 22, 2020, Danny Milosavljevic <dannym@scratchpost.org>
wrote:
> Because the question is what to do if you invoke
>
>  guix pack -f docker guix postgresql
> [snip]
> So I would suggest that
>
>  guix system docker-image ...
>
> create /etc/passwd by merging the required user accounts like
> described
> above, but
>
>  guix pack -f docker a b c
>
> really can't do that.

This makes sense to me. If you pack just some packages, you get the
binaries/libs/whatever but no supporting structure. If you pack a
service or a system, you get the supporting structure to make it able to
run.
Then it's a matter of documenting this appropriately and making it
discoverable. For example, something I want to do is set up an action in
my CI that runs a Postgres db and a Rails service, then kicks off some
tests. So how do I figure out what Guix command to run so that I can get
Postgres and Rails into a Docker image, not just as bare minimal
binaries but as services that I can actually run?

[-- Attachment #2: Type: text/html, Size: 3337 bytes --]

Re: /etc/passwd & co. in Docker images

[Danny Milosavljevic]

[-- Attachment #1: Type: text/plain, Size: 474 bytes --]

On the other hand, we have this:

(define-public network-manager-openconnect
  (package
    (name "network-manager-openconnect")
[...]
    (properties `([...]
                  ;; The 'etc/dbus-1/system.d/nm-openconnect-service.conf'
                  ;; file refers to account "nm-openconnect".  Specify it here
                  ;; so that 'network-manager-service-type' creates it.
                  (user-accounts . ("nm-openconnect"))))))

So which is it?

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: /etc/passwd & co. in Docker images

[Ludovic Courtès]

Danny Milosavljevic <dannym@scratchpost.org> skribis:

> On the other hand, we have this:
>
> (define-public network-manager-openconnect
>   (package
>     (name "network-manager-openconnect")
> [...]
>     (properties `([...]
>                   ;; The 'etc/dbus-1/system.d/nm-openconnect-service.conf'
>                   ;; file refers to account "nm-openconnect".  Specify it here
>                   ;; so that 'network-manager-service-type' creates it.
>                   (user-accounts . ("nm-openconnect"))))))

Nice, I had forgotten about that.  That’s a very local solution though:
only for NM VPN plugins (see e52b953434b255f53e61125c7fd3653adcebdab8).

I don’t think we’d want to generalize that: that’d be redundant with the
service framework and not the right place to add that info since it’s
only partial information about the kind of resources the service needs.

Ludo’.