Building container images with nix2container

Building container images with nix2container

[Antoine Eiche]

Hello Guix,

Two years ago, i released nix2container [1], a Go library and binary to
build container images from Nix expressions.
However, this Go code is not tightly coupled with Nix and has been
designed to potentially work with Guix [2]!

nix2container offers the following main features:
- It uses a layering algorithm to group storepaths into layers [3]
- It avoids writing an image tarball in your Guix store (reduce IOs and storage)

Basically, to build a container image, nix2container relies on two steps:
1. The nix2container Go binary takes the reference graph [4]
   of the container image closure to generates a JSON file describing
   the image configuration and the layers.
2. This JSON file can then be consumed by a patched Skopeo version[5] to
   build or push an image [8].

In case you would like to try nix2container with Guix, in theory, you
would need to add the support of another input reference graph format
[6] and a write simple Guix derivations [7] calling the nix2container
binary.

Hoping it could be useful,
lewo.

[1] https://github.com/nlewo/nix2container
[2] https://github.com/nlewo/containers-image-nix/blob/e342762cf7274dd7449343f3488723898da63f00/nix/utils.go#L55
[3] https://grahamc.com/blog/nix-and-layered-docker-images/
[4] https://nixos.org/manual/nix/stable/language/advanced-attributes.html?highlight=exportReferencesGraph#adv-attr-exportReferencesGraph
[5] https://github.com/nlewo/containers-image-nix/blob/9d7f33ef0058f4df4c0912025f43c758a3289d76/default.nix#L31
[6] https://github.com/nlewo/containers-image-nix/blob/9d7f33ef0058f4df4c0912025f43c758a3289d76/data/closure-graph.json#L1
[7] https://github.com/nlewo/containers-image-nix/blob/9d7f33ef0058f4df4c0912025f43c758a3289d76/default.nix#L305
[8] https://github.com/nlewo/containers-image-nix/blob/9d7f33ef0058f4df4c0912025f43c758a3289d76/default.nix#L47

Re: Building container images with nix2container

[Ricardo Wurmus]

Antoine Eiche <lewo@abesis.fr> writes:

> In case you would like to try nix2container with Guix, in theory, you
> would need to add the support of another input reference graph format
> [6] and a write simple Guix derivations [7] calling the nix2container
> binary.

We have "guix pack" as part of Guix.  It builds Docker or squashfs
images as well as various other formats.  What does nix2container offer
beyond what we have?

-- 
Ricardo

Re: Building container images with nix2container

[Antoine Eiche]

Ricardo Wurmus <rekado@elephly.net> writes:

> We have "guix pack" as part of Guix.  It builds Docker or squashfs
> images as well as various other formats.  What does nix2container offer
> beyond what we have?

I acutally don't know how you currently build Docker images. But if
nix2container brings something, i think it would mainly be
optimizations (time and space).

Does your built images contains several layers?

nix2container uses an heuristic to group store paths into layers. The
goal is to share common layers between images and to avoid full image
rebuild when only a storepath differs.

Do you write the image tarball into your store when you build an image?

nix2container is able to build layers on the fly from the Nix store. The
goal is to reduce IOs and storage. Instead of writing an image tarball
into the store, it generates a script which stream layers from store
paths to the destination (a Docker registry, the Docker deamon, Podman
or a file).

nix2container also has more advanced features allowing to control the
layers that are rebuilt. For instance, if you work on a Python
application, nix2container would allow to isolate your application and
the Python libraries into dedicated layers. When you change something in
your application, the layers containing the Python libraries won't have
to be rebuilt and pushed to a registry.

lewo.

Re: Building container images with nix2container

[Ricardo Wurmus]

Antoine Eiche <lewo@abesis.fr> writes:

> Ricardo Wurmus <rekado@elephly.net> writes:
>
>> We have "guix pack" as part of Guix.  It builds Docker or squashfs
>> images as well as various other formats.  What does nix2container offer
>> beyond what we have?
>
> I acutally don't know how you currently build Docker images. But if
> nix2container brings something, i think it would mainly be
> optimizations (time and space).
>
> Does your built images contains several layers?

Yes.  See
https://guix.gnu.org/manual/devel/en/html_node/Invoking-guix-pack.html

-- 
Ricardo

Re: Building container images with nix2container

[Simon Tournier]

Hi lewo,

On lun., 26 févr. 2024 at 11:09, Antoine Eiche <lewo@abesis.fr> wrote:

> Does your built images contains several layers?

This had recently been introduced.

        0cf75c9b2f23869201144917cea7f6ad49683d3d
        AuthorDate: Tue Dec 26 03:54:12 2023 +0300
        CommitDate: Mon Jan 8 21:04:44 2024 +0300

> nix2container uses an heuristic to group store paths into layers. The
> goal is to share common layers between images and to avoid full image
> rebuild when only a storepath differs.

Well, I have not followed on which strategy Guix relies.  What is the
one of nix2container?  The one described here:

    https://grahamc.com/blog/nix-and-layered-docker-images/

> Do you write the image tarball into your store when you build an image?
>
> nix2container is able to build layers on the fly from the Nix store. The
> goal is to reduce IOs and storage. Instead of writing an image tarball
> into the store, it generates a script which stream layers from store
> paths to the destination (a Docker registry, the Docker deamon, Podman
> or a file).

To my knowledge, this is not implemented in Guix.  And indeed, it could
improve the dance.  Currently, it reads:

    docker load < $(guix pack -f docker …)


Cheers,
simon

Re: Building container images with nix2container

[Simon Tournier]

Hi Antoine,

Reading this blog post:

    https://lewo.abesis.fr/posts/nix-build-container-image/

and from my understanding, “guix pack” is currently something similar to
’dockerTools.buildImage’ [1]

On lun., 26 févr. 2024 at 18:33, Simon Tournier <zimon.toutoune@gmail.com> wrote:

> Well, I have not followed on which strategy Guix relies.  What is the
> one of nix2container?  The one described here:
>
>     https://grahamc.com/blog/nix-and-layered-docker-images/

To answer to my question, the way to build the container image is
different, hence it does not make much sense to speak about a
“strategy“. :-)

However, the blog post says:

        To address this issue, we could add a nonReproducible option in
        the containerTools.buildLayer function. Instead of only storing
        the digest, we would also store the tar. Note in practice, an
        important part of nixpkgs is bit reproducible and this would
        rarely be needed.

And so the question is how do you know beforehand if the flag
’nonReproducible’ must be applied or not?

Indeed, the approach of nix2container could be helpful in addition to
‘guix pack’.  Maybe an extension… :-)

Cheers,
simon

Re: Building container images with nix2container

[Antoine Eiche]

Simon Tournier <zimon.toutoune@gmail.com> writes:

>> Does your built images contains several layers?
>
> This had recently been introduced.
>
>         0cf75c9b2f23869201144917cea7f6ad49683d3d
>         AuthorDate: Tue Dec 26 03:54:12 2023 +0300
>         CommitDate: Mon Jan 8 21:04:44 2024 +0300

Thanks.

>> nix2container uses an heuristic to group store paths into layers. The
>> goal is to share common layers between images and to avoid full image
>> rebuild when only a storepath differs.
>
> Well, I have not followed on which strategy Guix relies.  What is the
> one of nix2container?  The one described here:
>
>     https://grahamc.com/blog/nix-and-layered-docker-images/

Yes.
But I think it could be improved by taking the nar size into account.

lewo.

Re: Building container images with nix2container

[Antoine Eiche]

Simon Tournier <zimon.toutoune@gmail.com> writes:

>> Well, I have not followed on which strategy Guix relies.  What is the
>> one of nix2container?  The one described here:
>>
>>     https://grahamc.com/blog/nix-and-layered-docker-images/
>
> To answer to my question, the way to build the container image is
> different, hence it does not make much sense to speak about a
> “strategy“. :-)

nix2container actually uses this strategy. To build an image,
nix2container builds several derivations. Each of these derivation
contains a set of layers. The layers of each set are determined by using
the algorithm mentionned above. At runtime, all of these derivations are
packed to build the final image.

> However, the blog post says:
>
>         To address this issue, we could add a nonReproducible option in
>         the containerTools.buildLayer function. Instead of only storing
>         the digest, we would also store the tar. Note in practice, an
>         important part of nixpkgs is bit reproducible and this would
>         rarely be needed.
>
> And so the question is how do you know beforehand if the flag
> ’nonReproducible’ must be applied or not?

Sorry, i have missed your mails:/ So, here is the answer I provided on
Mastodon few days ago.

Generally, i don't think it's possible to know when this flag is needed
before encountering an issue at image push time: Skopeo would then say
layer hashes mismatch.

However, in practice, it seems this flag is not often used.
To hit this issue, I think you need to substitute the JSON file from a
binary cache while building non bit reproducible store paths locally. In
this case, the hash specified in the JSON file could be different from
the hash of the derivation locally built. But, the locally built store
path is a dependency of the JSON file, they should also be in the binary
cache and would generally also be substituted.
I admit this is not rock-solid science... but it seems to make the job
in practice. And if it occurs, a strategy could be to isolate this
storepath into a layer with the non-reproducible flag.

There is also another option which is currently used by
nixpkgs.dockerTools.streamLayeredImage. To avoid the reproducibility
issue while still avoiding writing layer tarballs, we could introduce
another option to let nix2container generating the layer hashes on the
fly, just before pushing the image. This would then require more IOs
since nix2container would need to read layer store paths on each image
push.

> Indeed, the approach of nix2container could be helpful in addition to
> ‘guix pack’.  Maybe an extension… :-)

From my point of view, another advantage of the nix2container
approach is to delegate a important part of the build image mecanism
to Skopeo, a tool well maintained and used by a lot of people.

lewo.