Using Guix inside a Guix container

Using Guix inside a Guix container

[Konrad Hinsen]

Hi Guix,

I have been playing with nested Guix containers recently, with some
suprising findings, and I am wondering if what I am doing is considered
officially supported or not.

First: why? My use case is scientific workflows, for example using
snakemake. I want to run my workflows in Guix containers, for
reproducibility plus other reasons. But my workflows run other programs
in their tasks (basically just "shelling out"), and those tasks may use
their own Guix containers.

Superficially, this works fine if I add the "guix" package to my "outer"
container and expose the store plus the daemon's socket:

    guix shell -C guix \
      --expose=/var/guix/daemon-socket/socket \
      --expose=/gnu/store \
      -- \
      guix shell -C coreutils -- ls /

But now for the first surprise:

   $ guix describe
   Generation 35	janv. 19 2023 12:34:57	(current)
     guix 8221cb6
       repository URL: https://git.savannah.gnu.org/git/guix.git
       branch: master
       commit: 8221cb6d2ae5624829bf514d25ae234c073e35d5

   $ guix shell -C guix -- guix describe
     guix 9fe5b49
       repository URL: https://git.savannah.gnu.org/git/guix.git
       branch: master
       commit: 9fe5b490df83ff32e2e0a604bf636eca48b9e240

The Guix in my container is an older one, apparently the 1.4.0 release.
Why? Can I change this?

My first attempt was time-machine:

    guix shell -C -N guix nss-certs \
      --expose=/var/guix/daemon-socket/socket \
      --expose=/gnu/store \
      -- \
      guix time-machine -C channels.scm -- describe

   Updating channel 'guix' from Git repository at 'https://git.savannah.gnu.org/git/guix.git'...
   Authenticating channel 'guix', commits 9edb3f6 to 8221cb6 (331 new commits)...
   Computing Guix derivation for 'x86_64-linux'... |
     guix 8221cb6
       repository URL: https://git.savannah.gnu.org/git/guix.git
       branch: master
       commit: 8221cb6d2ae5624829bf514d25ae234c073e35d5

Great! Except that every time I run this command, it does the channel
update from scratch, so it's prohibitively slow. Sharing
${HOME}/.cache/guix seems to fix that. So... finally...

    guix shell -C -N guix nss-certs \
      --expose=/var/guix/daemon-socket/socket \
      --expose=/gnu/store \
      --share=${HOME}/.cache/guix \
      -- \
      guix time-machine -C channels.scm \
      -- \
      shell -C coreutils \
      -- \
      ls /

     guix shell: error: mount: mount "none" on
     "/tmp/guix-directory.vpOEDC/sys": Operation not permitted

Now I am lost. It doesn't matter which command I put on the last line,
it's creating a container via time-machine running in another container
that leads to the error.

Any ideas?

Cheers,
  Konrad.

Re: Using Guix inside a Guix container

[Wojtek Kosior via Development of GNU Guix and the GNU System distribution.]

[-- Attachment #1: Type: text/plain, Size: 3899 bytes --]

Hi Konrad,

I wanted to try specifying currently used version of Guix in some other
way to avoid computing Guix inside the container. I tested first
without a container

guix shell -e '((@ (gnu packages package-management) current-guix))' -- guix describe

and the result is also bad — it still tries to compute Guix from
scratch every time.

So it seems exposing of folders is unlikely to help here because the
problem lies elsewhere :/

Perhaps someone else will be able to give a solution?

Wojtek


-- (sig_start)
website: https://koszko.org/koszko.html
PGP: https://koszko.org/key.gpg
fingerprint: E972 7060 E3C5 637C 8A4F  4B42 4BC5 221C 5A79 FD1A

♥ R29kIGlzIHRoZXJlIGFuZCBsb3ZlcyBtZQ== | ÷ c2luIHNlcGFyYXRlZCBtZSBmcm9tIEhpbQ==
✝ YnV0IEplc3VzIGRpZWQgdG8gc2F2ZSBtZQ== | ? U2hhbGwgSSBiZWNvbWUgSGlzIGZyaWVuZD8=
-- (sig_end)


On Fri, 03 Feb 2023 17:52:22 +0100
Konrad Hinsen <konrad.hinsen@fastmail.net> wrote:

> Hi Guix,
> 
> I have been playing with nested Guix containers recently, with some
> suprising findings, and I am wondering if what I am doing is considered
> officially supported or not.
> 
> First: why? My use case is scientific workflows, for example using
> snakemake. I want to run my workflows in Guix containers, for
> reproducibility plus other reasons. But my workflows run other programs
> in their tasks (basically just "shelling out"), and those tasks may use
> their own Guix containers.
> 
> Superficially, this works fine if I add the "guix" package to my "outer"
> container and expose the store plus the daemon's socket:
> 
>     guix shell -C guix \
>       --expose=/var/guix/daemon-socket/socket \
>       --expose=/gnu/store \
>       -- \
>       guix shell -C coreutils -- ls /
> 
> But now for the first surprise:
> 
>    $ guix describe
>    Generation 35	janv. 19 2023 12:34:57	(current)
>      guix 8221cb6
>        repository URL: https://git.savannah.gnu.org/git/guix.git
>        branch: master
>        commit: 8221cb6d2ae5624829bf514d25ae234c073e35d5
> 
>    $ guix shell -C guix -- guix describe
>      guix 9fe5b49
>        repository URL: https://git.savannah.gnu.org/git/guix.git
>        branch: master
>        commit: 9fe5b490df83ff32e2e0a604bf636eca48b9e240
> 
> The Guix in my container is an older one, apparently the 1.4.0 release.
> Why? Can I change this?
> 
> My first attempt was time-machine:
> 
>     guix shell -C -N guix nss-certs \
>       --expose=/var/guix/daemon-socket/socket \
>       --expose=/gnu/store \
>       -- \
>       guix time-machine -C channels.scm -- describe
> 
>    Updating channel 'guix' from Git repository at 'https://git.savannah.gnu.org/git/guix.git'...
>    Authenticating channel 'guix', commits 9edb3f6 to 8221cb6 (331 new commits)...
>    Computing Guix derivation for 'x86_64-linux'... |
>      guix 8221cb6
>        repository URL: https://git.savannah.gnu.org/git/guix.git
>        branch: master
>        commit: 8221cb6d2ae5624829bf514d25ae234c073e35d5
> 
> Great! Except that every time I run this command, it does the channel
> update from scratch, so it's prohibitively slow. Sharing
> ${HOME}/.cache/guix seems to fix that. So... finally...
> 
>     guix shell -C -N guix nss-certs \
>       --expose=/var/guix/daemon-socket/socket \
>       --expose=/gnu/store \
>       --share=${HOME}/.cache/guix \
>       -- \
>       guix time-machine -C channels.scm \
>       -- \
>       shell -C coreutils \
>       -- \
>       ls /
> 
>      guix shell: error: mount: mount "none" on
>      "/tmp/guix-directory.vpOEDC/sys": Operation not permitted
> 
> Now I am lost. It doesn't matter which command I put on the last line,
> it's creating a container via time-machine running in another container
> that leads to the error.
> 
> Any ideas?
> 
> Cheers,
>   Konrad.
> 



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

Re: Using Guix inside a Guix container

[James Thomas]

Konrad Hinsen wrote:

> Hi Guix,
>
> I have been playing with nested Guix containers recently, with some
> suprising findings, and I am wondering if what I am doing is considered
> officially supported or not.
>
> First: why? My use case is scientific workflows, for example using
> snakemake. I want to run my workflows in Guix containers, for
> reproducibility plus other reasons. But my workflows run other programs
> in their tasks (basically just "shelling out"), and those tasks may use
> their own Guix containers.
>
> Superficially, this works fine if I add the "guix" package to my "outer"
> container and expose the store plus the daemon's socket:
>
>     guix shell -C guix \
>       --expose=/var/guix/daemon-socket/socket \
>       --expose=/gnu/store \
>       -- \
>       guix shell -C coreutils -- ls /
>
> But now for the first surprise:
>
>    $ guix describe
>    Generation 35	janv. 19 2023 12:34:57	(current)
>      guix 8221cb6
>        repository URL: https://git.savannah.gnu.org/git/guix.git
>        branch: master
>        commit: 8221cb6d2ae5624829bf514d25ae234c073e35d5
>
>    $ guix shell -C guix -- guix describe
>      guix 9fe5b49
>        repository URL: https://git.savannah.gnu.org/git/guix.git
>        branch: master
>        commit: 9fe5b490df83ff32e2e0a604bf636eca48b9e240
>
> The Guix in my container is an older one, apparently the 1.4.0 release.
> Why? Can I change this?

Makes sense to me, because the guix package definition was only updated
with the new release (see 'guix edit guix'). Maybe you can try:

  guix shell -C guix --with-version=guix=VERSION -- guix describe

Or --with-commit etc.

>
> My first attempt was time-machine:
>
>     guix shell -C -N guix nss-certs \
>       --expose=/var/guix/daemon-socket/socket \
>       --expose=/gnu/store \
>       -- \
>       guix time-machine -C channels.scm -- describe
>
>    Updating channel 'guix' from Git repository at 'https://git.savannah.gnu.org/git/guix.git'...
>    Authenticating channel 'guix', commits 9edb3f6 to 8221cb6 (331 new commits)...
>    Computing Guix derivation for 'x86_64-linux'... |
>      guix 8221cb6
>        repository URL: https://git.savannah.gnu.org/git/guix.git
>        branch: master
>        commit: 8221cb6d2ae5624829bf514d25ae234c073e35d5
>
> Great! Except that every time I run this command, it does the channel
> update from scratch, so it's prohibitively slow. Sharing
> ${HOME}/.cache/guix seems to fix that. So... finally...
>
>     guix shell -C -N guix nss-certs \
>       --expose=/var/guix/daemon-socket/socket \
>       --expose=/gnu/store \
>       --share=${HOME}/.cache/guix \
>       -- \
>       guix time-machine -C channels.scm \
>       -- \
>       shell -C coreutils \
>       -- \
>       ls /
>
>      guix shell: error: mount: mount "none" on
>      "/tmp/guix-directory.vpOEDC/sys": Operation not permitted

Maybe --expose tmp is needed in the outer container.

--

Re: Using Guix inside a Guix container

[Jim]

James Thomas wrote:

>   guix shell -C guix --with-version=guix=VERSION -- guix describe

Or even:

  guix shell -C -p /var/guix/profiles/per-user/.../current-guix -- guix
describe

--

Re: Using Guix inside a Guix container

[Konrad Hinsen]

James Thomas <jimjoe@gmx.net> writes:

> Makes sense to me, because the guix package definition was only updated
> with the new release (see 'guix edit guix'). Maybe you can try:

Indeed. That's at least an explanation, though I still don't see *why*
it is done this way. In fact, the role of the package "guix" isn't
really clear to me.\

>   guix shell -C guix --with-version=guix=VERSION -- guix describe
>
> Or --with-commit etc.

That's an interesting idea! I could also define my own package (in a
file) called "guix-current" or something like that, which inherits from
"guix" and updates the commit.


Jim <jimplyl@outlook.com> writes:

> Or even:
>
>   guix shell -C -p /var/guix/profiles/per-user/.../current-guix -- guix
> describe

That would create a container from the Guix profile. Not sure if this
works at all, but it's very probably not what I want because I have to
add other packages to my container, to get some real work done :-)

Thanks to both of you for your suggestions!

Cheers,
  Konrad

Re: Using Guix inside a Guix container

[Simon Tournier]

Hi Konrad,

On mer., 15 févr. 2023 at 10:49, Konrad Hinsen <konrad.hinsen@fastmail.net> wrote:

> That would create a container from the Guix profile. Not sure if this
> works at all, but it's very probably not what I want because I have to
> add other packages to my container, to get some real work done :-)

I am sure you have good reasons for doing this and I am lacking
imagination to find them. :-)

Which part of Guix do you need inside the containerized shell that you
cannot do outside?

Basically, you are authorizing the inside Guix to behave with the same
permissions as the outside Guix; therefore, why this dance?

Well, I understand the needs for running inside a containerized shell:
restricted permissions on filestystem, use of --emulate-fhs for allowing
untrusted binaries, etc.

But I miss what is the need to run Guix inside a containerized shell
where it has the permissions as the non-containerized shell.

Considering your use-case with Snakemake, what I am doing is to wrap
each rule with one containerized Guix shell which controls the
permissions, rule by rule; or a big containerized shell:

    guix shell -C -m manifest.scm --expose=…

where manifest.scm contains the tools of each rule and snakemake.

Could you provide some details about why you also need Guix?

Cheers,
simon

Re: Using Guix inside a Guix container

[Konrad Hinsen]

Hi Simon,

> Which part of Guix do you need inside the containerized shell that you
> cannot do outside?

That's not the right question. There's always a way to do what I want to
do outside. But that may be very inconvenient.

> Considering your use-case with Snakemake, what I am doing is to wrap
> each rule with one containerized Guix shell which controls the
> permissions, rule by rule; or a big containerized shell:
>
>     guix shell -C -m manifest.scm --expose=…

Nice example. I do the same: "guix shell" in every rule. Then I add
stuff to my Snakefile, which is a Python script after all. For example,
I import pandas to read a data frame from which I construct my workflow.
Now I am at the point where I'd like to run snakemake itself in a
container, to manage the dependencies of my Snakefile. In fact, given
that I have workflows that depend on specific Snakemake versions, I'd
really like to run Snakemake in a container all the time, even without
additional dependencies.

Without nested containers, I have to go through all the rules, collect
the packages from their manifest files (or command line), and add them
to the container in which I run the whole workflow. Possible, but not
convenient.

Another example: I run command-line programs from my Pharo image, and I
have developed the habit of doing this always through Guix. The
advantage is that my Pharo code becomes portable: it depends on Guix,
but not on my profile.

But if I want, one day, to move on to a full Guix system, I have to run
Pharo in a container with LFS simulation. And then all my command line
shell-outs will break.

Both examples are about composing tools freely, without worrying if they
use Guix internally or now.

Cheers,
  Konrad

Containerized workflow in containerized processes

[Simon Tournier]

Hi,

From thread in guix-devel:

        Using Guix inside a Guix container
        Sat, 18 Feb 2023 11:01:50 +0100
        id:m1mt5uk7y1.fsf@fastmail.net
        https://yhetil.org/guix/m1mt5uk7y1.fsf@fastmail.net/#r

The use-case “Containerized workflow in containerized processes” appears
to me interesting. :-)

It is almost done by design with GWL, no?

-------------------- Start of forwarded message --------------------
From: Konrad Hinsen <konrad.hinsen@fastmail.net>
To: Simon Tournier <zimon.toutoune@gmail.com>, Guix Devel <guix-devel@gnu.org>
Subject: Re: Using Guix inside a Guix container
Date: Sat, 18 Feb 2023 10:21:52 +0100

Hi Simon,

> Which part of Guix do you need inside the containerized shell that you
> cannot do outside?

That's not the right question. There's always a way to do what I want to
do outside. But that may be very inconvenient.

> Considering your use-case with Snakemake, what I am doing is to wrap
> each rule with one containerized Guix shell which controls the
> permissions, rule by rule; or a big containerized shell:
>
>     guix shell -C -m manifest.scm --expose=…

Nice example. I do the same: "guix shell" in every rule. Then I add
stuff to my Snakefile, which is a Python script after all. For example,
I import pandas to read a data frame from which I construct my workflow.
Now I am at the point where I'd like to run snakemake itself in a
container, to manage the dependencies of my Snakefile. In fact, given
that I have workflows that depend on specific Snakemake versions, I'd
really like to run Snakemake in a container all the time, even without
additional dependencies.

Without nested containers, I have to go through all the rules, collect
the packages from their manifest files (or command line), and add them
to the container in which I run the whole workflow. Possible, but not
convenient.

Another example: I run command-line programs from my Pharo image, and I
have developed the habit of doing this always through Guix. The
advantage is that my Pharo code becomes portable: it depends on Guix,
but not on my profile.

But if I want, one day, to move on to a full Guix system, I have to run
Pharo in a container with LFS simulation. And then all my command line
shell-outs will break.

Both examples are about composing tools freely, without worrying if they
use Guix internally or now.

Cheers,
  Konrad

-------------------- End of forwarded message --------------------

Cheers,
simon

Re: Containerized workflow in containerized processes

[Konrad Hinsen]

Hi Simon,

> The use-case “Containerized workflow in containerized processes” appears
> to me interesting. :-)
>
> It is almost done by design with GWL, no?

That's an interesting observation. But I don't see anything in the GWL
manual that explains how GWL manages processes. 

The chapter "Process engines" says:

  "The simplest way is to turn the workflow into a Guile script that
  sets up the desired environment and then executes the workflow
  processes on the current machine.

Fine, but is that script run in a container? If not, the programs and
code snippets from that process could run arbitrary binaries from the
file system.

In the examples shown, the workflow itself is launched from the command
line, so it is not running in a container either. In principle, the
Guile script defining the workflow could access arbitrary files, and
thus not be reproducible. I suspect that the risk is low in practice,
because I see no good reason for doing this.

Cheers,
  Konrad

>
> -------------------- Start of forwarded message --------------------
> From: Konrad Hinsen <konrad.hinsen@fastmail.net>
> To: Simon Tournier <zimon.toutoune@gmail.com>, Guix Devel <guix-devel@gnu.org>
> Subject: Re: Using Guix inside a Guix container
> Date: Sat, 18 Feb 2023 10:21:52 +0100
>
> Hi Simon,
>
>> Which part of Guix do you need inside the containerized shell that you
>> cannot do outside?
>
> That's not the right question. There's always a way to do what I want to
> do outside. But that may be very inconvenient.
>
>> Considering your use-case with Snakemake, what I am doing is to wrap
>> each rule with one containerized Guix shell which controls the
>> permissions, rule by rule; or a big containerized shell:
>>
>>     guix shell -C -m manifest.scm --expose=…
>
> Nice example. I do the same: "guix shell" in every rule. Then I add
> stuff to my Snakefile, which is a Python script after all. For example,
> I import pandas to read a data frame from which I construct my workflow.
> Now I am at the point where I'd like to run snakemake itself in a
> container, to manage the dependencies of my Snakefile. In fact, given
> that I have workflows that depend on specific Snakemake versions, I'd
> really like to run Snakemake in a container all the time, even without
> additional dependencies.
>
> Without nested containers, I have to go through all the rules, collect
> the packages from their manifest files (or command line), and add them
> to the container in which I run the whole workflow. Possible, but not
> convenient.
>
> Another example: I run command-line programs from my Pharo image, and I
> have developed the habit of doing this always through Guix. The
> advantage is that my Pharo code becomes portable: it depends on Guix,
> but not on my profile.
>
> But if I want, one day, to move on to a full Guix system, I have to run
> Pharo in a container with LFS simulation. And then all my command line
> shell-outs will break.
>
> Both examples are about composing tools freely, without worrying if they
> use Guix internally or now.
>
> Cheers,
>   Konrad
>
> -------------------- End of forwarded message --------------------
>
> Cheers,
> simon

Re: Using Guix inside a Guix container

[Simon Tournier]

Hi Konrad,

On sam., 18 févr. 2023 at 10:21, Konrad Hinsen <konrad.hinsen@fastmail.net> wrote:

> Both examples are about composing tools freely, without worrying if they
> use Guix internally or now.

Thanks for explaining and the use-cases.  Well, maybe a discussion for
gwl-devel… well, I have started one. :-)


Cheers,
simon

Re: Using Guix inside a Guix container

[Ludovic Courtès]

Hello!

Konrad Hinsen <konrad.hinsen@fastmail.net> skribis:

> I have been playing with nested Guix containers recently, with some
> suprising findings, and I am wondering if what I am doing is considered
> officially supported or not.
>
> First: why? My use case is scientific workflows, for example using
> snakemake. I want to run my workflows in Guix containers, for
> reproducibility plus other reasons. But my workflows run other programs
> in their tasks (basically just "shelling out"), and those tasks may use
> their own Guix containers.

That’s an interesting use case!  I guess we have a hard-enough time
getting the message through regarding the environment of tasks that we
didn’t really consider the environment of the “driver”.

(Well, in a way, GWL and Guix-Jupyter sidestep the issue by integrating
the mechanism to declare task environments.)

> Superficially, this works fine if I add the "guix" package to my "outer"
> container and expose the store plus the daemon's socket:
>
>     guix shell -C guix \
>       --expose=/var/guix/daemon-socket/socket \
>       --expose=/gnu/store \
>       -- \
>       guix shell -C coreutils -- ls /

I wasn’t sure ‘--expose=/gnu/store’ would even work… but it does!  Kinda
by chance though.

The thing is that ‘-C’ bind-mounts just the subset of the store that’s
needed.  To support nested containers, we need to bind-mount the whole
store because new store items may pop up in there over time.

I’d be inclined to add a new ‘-W’ (say) option to (1) share the whole
store, and (2) share the daemon socket.  That would be the documented
way to create a container with support for nested containers.


[...]

> Great! Except that every time I run this command, it does the channel
> update from scratch, so it's prohibitively slow. Sharing
> ${HOME}/.cache/guix seems to fix that. So... finally...
>
>     guix shell -C -N guix nss-certs \
>       --expose=/var/guix/daemon-socket/socket \
>       --expose=/gnu/store \
>       --share=${HOME}/.cache/guix \
>       -- \
>       guix time-machine -C channels.scm \
>       -- \
>       shell -C coreutils \
>       -- \
>       ls /
>
>      guix shell: error: mount: mount "none" on
>      "/tmp/guix-directory.vpOEDC/sys": Operation not permitted

That one’s interesting.  Reported here:

  https://issues.guix.gnu.org/61690

At least there’s a workaround: using ‘-CN’ in the nested container.

Ludo’.

Re: Using Guix inside a Guix container

[Konrad Hinsen]

Ludovic Courtès <ludovic.courtes@inria.fr> writes:

> That’s an interesting use case!  I guess we have a hard-enough time
> getting the message through regarding the environment of tasks that we
> didn’t really consider the environment of the “driver”.

It takes something messy such as Snakemake to illustrate the importance
of this.

> (Well, in a way, GWL and Guix-Jupyter sidestep the issue by integrating
> the mechanism to declare task environments.)

OK, let's pass a law that workflows must be written in Guile or in a
language implemented in Guile ;-)

> I’d be inclined to add a new ‘-W’ (say) option to (1) share the whole
> store, and (2) share the daemon socket.  That would be the documented
> way to create a container with support for nested containers.

Sounds good to me!

> That one’s interesting.  Reported here:
>
>   https://issues.guix.gnu.org/61690
>
> At least there’s a workaround: using ‘-CN’ in the nested container.

That's already good to know.

Thanks,
  Konrad.