Reproducible environments

Reproducible environments

[Konrad Hinsen]

Hi everyone,

I am exploring the use of Guix for reproducible environments in scientific computing. My goals are very similar to those described in the paper "Reproducible and User-Controlled Software Environments in HPC with Guix".

Section 4.1 of that paper says that a given environment, defined by a Guix profile, can be reproduced given (1) a manifest and (2) the Guix commit defining all the packages.

Problem #1: After reading the manual twice, I still don't see how I can tell "guix package" to use a given Guix commit.

Problem #2: What if my profile contains packages from several Guix commits (typically for getting specific older versions)? Or if it contains packages defined outside of the Guix distribution, in Guile modules on GUIX_PACKAGE_PATH?

If my understanding of Guix is correct, all the required information for rebuilding everything identically is available at a lower level, in the derivation files. Would it be feasible to collect all the derivation files underlying a profile in a tar archive, and then have them rebuilt and added to a fresh profile in an automatized way?

Konrad.

Re: Reproducible environments

[Pjotr Prins]

Hi Konrad,

On Tue, Oct 06, 2015 at 12:50:52PM +0200, Konrad Hinsen wrote:
> Hi everyone,
> 
> I am exploring the use of Guix for reproducible environments in scientific computing. My goals are very similar to those described in the paper "Reproducible and User-Controlled Software Environments in HPC with Guix".
> 
> Section 4.1 of that paper says that a given environment, defined by a Guix profile, can be reproduced given (1) a manifest and (2) the Guix commit defining all the packages.
> 
> Problem #1: After reading the manual twice, I still don't see how I can tell "guix package" to use a given Guix commit.

At this point (correct me if I am wrong), the route to take is to
checkout a commit of the Guix source tree (with packages). The SHA
value is captured on Hydra in the build itself. That is the only way
to fully reproduce GUIX as a point in time. And use guix package using
that tree.

See the revision of this package: http://hydra.gnu.org/build/694326#tabs-buildinputs

There is a similar discussion on the NIX mailing list about this right
now.

> Problem #2: What if my profile contains packages from several Guix commits (typically for getting specific older versions)? Or if it contains packages defined outside of the Guix distribution, in Guile modules on GUIX_PACKAGE_PATH?

You'd have to create a new profile that is tied to above build to be
fully reprocible. Managing multiple profiles is pretty
straightforward (using the -p switch). I use a compile profile for
guix, for example, containing gcc etc.

> If my understanding of Guix is correct, all the required information for rebuilding everything identically is available at a lower level, in the derivation files. Would it be feasible to collect all the derivation files underlying a profile in a tar archive, and then have them rebuilt and added to a fresh profile in an automatized way?

I don't think we need that functionality when you accept that a git
source tree of GUIX contains all the information you need. Maybe
others have ideas to make use a bit more convenient for end-users
without git - but the current setup works for me to recreate software
packages.

Note that you can also tar ball a binary package with all its
dependencies (using guix archive) and distribute that. That is 
reproducible at the binary level. That may be more useful for end-users.

Pj.
-- 

Re: Reproducible environments

[Ludovic Courtès]

Hi Konrad,

Just to complete what Pjotr already wrote:

Konrad Hinsen <konrad.hinsen@fastmail.net> skribis:

> Section 4.1 of that paper says that a given environment, defined by a Guix profile, can be reproduced given (1) a manifest and (2) the Guix commit defining all the packages.
>
> Problem #1: After reading the manual twice, I still don't see how I can tell "guix package" to use a given Guix commit.

Currently the ‘guix’ commands cannot be provided with a commit.  So in
practice, you have to git clone the Guix repo, jump to the commit of
interest, and run ‘./pre-inst-env guix package -i whatever’ from there.

There have been discussions to improve this, such as “versioning” the
~/.config/guix/latest symlink (which is created by ‘guix pull’), but
nothing concrete at this point.

Note that there are several levels of reproducibility that may be
interesting.  Having the Git commit allows you to rebuild the same
profile bit-for-bit (assuming deterministic package builds.)  Using
‘guix package --manifest’, possibly with a different commit, allows you
to reproduce the profile “symbolically”: same environment, but possibly
different package versions or builds.

> Problem #2: What if my profile contains packages from several Guix commits (typically for getting specific older versions)? Or if it contains packages defined outside of the Guix distribution, in Guile modules on GUIX_PACKAGE_PATH?

For that, you can always export the bits of the profile:

  guix archive -r --export $(readlink -f ~/.guix-profile)

and reimport them elsewhere.

This gives bitwise reproducibility, at the expense of composability,
similar to Docker or VM images.

> If my understanding of Guix is correct, all the required information for rebuilding everything identically is available at a lower level, in the derivation files. Would it be feasible to collect all the derivation files underlying a profile in a tar archive, and then have them rebuilt and added to a fresh profile in an automatized way?

Yes, that’s a good idea.  :-)

Assuming you know the profile’s derivation¹, you can always run:

  guix archive -r --export /gnu/store/…-profile.drv > foo.nar

The resulting archive will be relatively small and can be carried
around.  Then one can run:

  guix build /gnu/store/…-profile.drv

to rebuild it.

The pieces are in place, but the UIs could definitely be improved to
handle those use cases more easily.

Thanks,
Ludo’.

¹ ‘guix package’ prints the .drv file name, but otherwise this can be
   retrieved with hacks like this:

--8<---------------cut here---------------start------------->8---
scheme@(guile-user)> ,use(guix)
scheme@(guile-user)> (define s (open-connection))
scheme@(guile-user)> (path-info-deriver (query-path-info s (readlink (readlink (readlink "/home/ludo/.guix-profile")))))
$2 = "/gnu/store/8bgskqk38hr9qkbzn53g54iwxkh32y88-profile.drv"
--8<---------------cut here---------------end--------------->8---

Re: Reproducible environments

[Konrad Hinsen]

Hi Pjotr,

First of all, thanks for your explanations!

 > At this point (correct me if I am wrong), the route to take is to
 > checkout a commit of the Guix source tree (with packages). The SHA
 > value is captured on Hydra in the build itself. That is the only way
 > to fully reproduce GUIX as a point in time. And use guix package using
 > that tree.

OK, so the procedure is:

 1) Check out a specific commit of Guix.
 2) Make a manifest that defines a profile
 3) Use "guix package" with options
    -p  for defining the profile
    -L  for pointing to the directory containing the package descriptions
    -m  for choosing the manifest

I could then archive 1) and 2) for reproducing the environment later. That
sounds quite doable.

 > > Problem #2: What if my profile contains packages from several
 > > Guix commits (typically for getting specific older versions)? Or
 > > if it contains packages defined outside of the Guix distribution,
 > > in Guile modules on GUIX_PACKAGE_PATH?
 > 
 > You'd have to create a new profile that is tied to above build to be
 > fully reprocible. Managing multiple profiles is pretty
 > straightforward (using the -p switch). I use a compile profile for
 > guix, for example, containing gcc etc.

I already found out that creating profiles is easy, which is a nice feature.
But I may well need to mix different Guix commits in one and the same profile.

Here's an example (a simplified version of the real situation that motivated
me to check out Guix):

   - I need to use Program X that depends on libraries A and B.
   - The current versions are A-1.1 and B-42.0.1.
   - X requires "1.0 or later" for A but "41.*"  for B, because
     version 42.* of B is not fully backwards compatible.

If there's no Guix commit that has both A and B in the required
version range, then the easiest way to get what I need is to use an
older Guix commit for installing B than I use for A.

In fact, the only other alternative I see is to add a package definition
for the old version of B to a later Guix commit. That's likely to be
much more difficult.

 > Note that you can also tar ball a binary package with all its
 > dependencies (using guix archive) and distribute that. That is 
 > reproducible at the binary level. That may be more useful for end-users.

That's what I would use to share environments with collaborators. But
I wouldn't download binaries from a stranger, and I don't expect
others to have a different attitude.

Konrad.

Re: Reproducible environments

[Pjotr Prins]

On Tue, Oct 06, 2015 at 06:59:44PM +0200, Konrad Hinsen wrote:
> Here's an example (a simplified version of the real situation that motivated
> me to check out Guix):
> 
>    - I need to use Program X that depends on libraries A and B.
>    - The current versions are A-1.1 and B-42.0.1.
>    - X requires "1.0 or later" for A but "41.*"  for B, because
>      version 42.* of B is not fully backwards compatible.
> 
> If there's no Guix commit that has both A and B in the required
> version range, then the easiest way to get what I need is to use an
> older Guix commit for installing B than I use for A.
> 
> In fact, the only other alternative I see is to add a package definition
> for the old version of B to a later Guix commit. That's likely to be
> much more difficult.

Create multiple packages. Such as we do for Ruby 1.8, 2.1 and 2.2. You
can specify specific dependencies on each using one git checkout.

>  > Note that you can also tar ball a binary package with all its
>  > dependencies (using guix archive) and distribute that. That is 
>  > reproducible at the binary level. That may be more useful for end-users.
> 
> That's what I would use to share environments with collaborators. But
> I wouldn't download binaries from a stranger, and I don't expect
> others to have a different attitude.

Agree. But sometimes convenience wins. Also the binary tar balls may
help with rapid deployment.

Pj.

Re: Reproducible environments

[Ludovic Courtès]

Konrad Hinsen <konrad.hinsen@fastmail.net> skribis:

> Here's an example (a simplified version of the real situation that motivated
> me to check out Guix):
>
>    - I need to use Program X that depends on libraries A and B.
>    - The current versions are A-1.1 and B-42.0.1.
>    - X requires "1.0 or later" for A but "41.*"  for B, because
>      version 42.* of B is not fully backwards compatible.
>
> If there's no Guix commit that has both A and B in the required
> version range, then the easiest way to get what I need is to use an
> older Guix commit for installing B than I use for A.

That works if you use ‘guix package -i’ to incrementally build the
profile, using different Guix commits; but it prevents you from using
the declarative ‘--manifest’ approach.

> In fact, the only other alternative I see is to add a package definition
> for the old version of B to a later Guix commit. That's likely to be
> much more difficult.

It depends, but not necessarily.  It’s particularly easy to maintain
different versions of leaf packages, as shown in Figure 4 of the paper.

Ludo’.