Containerized workflow in containerized processes

Containerized workflow in containerized processes

[Simon Tournier]

Hi,

From thread in guix-devel:

        Using Guix inside a Guix container
        Sat, 18 Feb 2023 11:01:50 +0100
        id:m1mt5uk7y1.fsf@fastmail.net
        https://yhetil.org/guix/m1mt5uk7y1.fsf@fastmail.net/#r

The use-case “Containerized workflow in containerized processes” appears
to me interesting. :-)

It is almost done by design with GWL, no?

-------------------- Start of forwarded message --------------------
From: Konrad Hinsen <konrad.hinsen@fastmail.net>
To: Simon Tournier <zimon.toutoune@gmail.com>, Guix Devel <guix-devel@gnu.org>
Subject: Re: Using Guix inside a Guix container
Date: Sat, 18 Feb 2023 10:21:52 +0100

Hi Simon,

> Which part of Guix do you need inside the containerized shell that you
> cannot do outside?

That's not the right question. There's always a way to do what I want to
do outside. But that may be very inconvenient.

> Considering your use-case with Snakemake, what I am doing is to wrap
> each rule with one containerized Guix shell which controls the
> permissions, rule by rule; or a big containerized shell:
>
>     guix shell -C -m manifest.scm --expose=…

Nice example. I do the same: "guix shell" in every rule. Then I add
stuff to my Snakefile, which is a Python script after all. For example,
I import pandas to read a data frame from which I construct my workflow.
Now I am at the point where I'd like to run snakemake itself in a
container, to manage the dependencies of my Snakefile. In fact, given
that I have workflows that depend on specific Snakemake versions, I'd
really like to run Snakemake in a container all the time, even without
additional dependencies.

Without nested containers, I have to go through all the rules, collect
the packages from their manifest files (or command line), and add them
to the container in which I run the whole workflow. Possible, but not
convenient.

Another example: I run command-line programs from my Pharo image, and I
have developed the habit of doing this always through Guix. The
advantage is that my Pharo code becomes portable: it depends on Guix,
but not on my profile.

But if I want, one day, to move on to a full Guix system, I have to run
Pharo in a container with LFS simulation. And then all my command line
shell-outs will break.

Both examples are about composing tools freely, without worrying if they
use Guix internally or now.

Cheers,
  Konrad

-------------------- End of forwarded message --------------------

Cheers,
simon

Re: Containerized workflow in containerized processes

[Konrad Hinsen]

Hi Simon,

> The use-case “Containerized workflow in containerized processes” appears
> to me interesting. :-)
>
> It is almost done by design with GWL, no?

That's an interesting observation. But I don't see anything in the GWL
manual that explains how GWL manages processes. 

The chapter "Process engines" says:

  "The simplest way is to turn the workflow into a Guile script that
  sets up the desired environment and then executes the workflow
  processes on the current machine.

Fine, but is that script run in a container? If not, the programs and
code snippets from that process could run arbitrary binaries from the
file system.

In the examples shown, the workflow itself is launched from the command
line, so it is not running in a container either. In principle, the
Guile script defining the workflow could access arbitrary files, and
thus not be reproducible. I suspect that the risk is low in practice,
because I see no good reason for doing this.

Cheers,
  Konrad

>
> -------------------- Start of forwarded message --------------------
> From: Konrad Hinsen <konrad.hinsen@fastmail.net>
> To: Simon Tournier <zimon.toutoune@gmail.com>, Guix Devel <guix-devel@gnu.org>
> Subject: Re: Using Guix inside a Guix container
> Date: Sat, 18 Feb 2023 10:21:52 +0100
>
> Hi Simon,
>
>> Which part of Guix do you need inside the containerized shell that you
>> cannot do outside?
>
> That's not the right question. There's always a way to do what I want to
> do outside. But that may be very inconvenient.
>
>> Considering your use-case with Snakemake, what I am doing is to wrap
>> each rule with one containerized Guix shell which controls the
>> permissions, rule by rule; or a big containerized shell:
>>
>>     guix shell -C -m manifest.scm --expose=…
>
> Nice example. I do the same: "guix shell" in every rule. Then I add
> stuff to my Snakefile, which is a Python script after all. For example,
> I import pandas to read a data frame from which I construct my workflow.
> Now I am at the point where I'd like to run snakemake itself in a
> container, to manage the dependencies of my Snakefile. In fact, given
> that I have workflows that depend on specific Snakemake versions, I'd
> really like to run Snakemake in a container all the time, even without
> additional dependencies.
>
> Without nested containers, I have to go through all the rules, collect
> the packages from their manifest files (or command line), and add them
> to the container in which I run the whole workflow. Possible, but not
> convenient.
>
> Another example: I run command-line programs from my Pharo image, and I
> have developed the habit of doing this always through Guix. The
> advantage is that my Pharo code becomes portable: it depends on Guix,
> but not on my profile.
>
> But if I want, one day, to move on to a full Guix system, I have to run
> Pharo in a container with LFS simulation. And then all my command line
> shell-outs will break.
>
> Both examples are about composing tools freely, without worrying if they
> use Guix internally or now.
>
> Cheers,
>   Konrad
>
> -------------------- End of forwarded message --------------------
>
> Cheers,
> simon