Re: [rb-general] Paper preprint: Reproducible genomics analysis pipelines with GNU Guix

[Ricardo Wurmus <rekado@elephly.net>]

Catonano <catonano@gmail.com> writes:

> Ricardo, I don't understand the problem you're raising here (I didn't read
> the article yet, though)
>
> Would you mind to elaborate on that ?
>
> Why would you want to record the environment ?

I want to record the detected build environment so that I can restore it
at execution time.  Autoconf provides macros that probe the environment
and record the full path to detected tools.  For example, I’m looking
for Samtools, and the user may provide a particular variant of Samtools
at configure time.  I record the full path to the executable at
configure time and embed that path in a configuration file that is read
when the pipeline is run.

This works fine for tools, but doesn’t work very well at all for modules
in language environments.  Take R for example.  I can detect and record
the location of the R and Rscript executables, but I cannot easily
record the location of build-time R packages (such as r-deseq2) in a way
that allows me to rebuild the environment at runtime.

Instead of writing an Autoconf macro that records the exact location of
each of the detected R packages and their dependencies I chose to solve
the problem in Guix by wrapping the pipeline executables in R_SITE_LIBS,
because I figured that on systems without Guix you aren’t likely to
install R packages into separate unique locations anyway — on most
systems R packages end up being installed to one and the same directory.

I think the desire to restore the configured environment at runtime is
valid and we do this all the time when we run binaries that have
embedded absolute paths (to libraries or other tools).  It’s just that
it gets pretty awkward to do this for things like R packages or Python
modules (or Guile modules for that matter).

The Guix workflow language solves this problem by depending on Guix for
software deployment.  For PiGx we picked Snakemake early on and it does
not have a software deployment solution (it expects to either run inside
a suitable environment that the user provides or to have access to
pre-built Singularity application bundles).  I don’t like to treat
pipelines like some sort of collection of scripts that must be invoked
in a suitable environment.  I like to see pipelines as big software
packages that should know about the environment they need, that can be
configured like regular tools, and thus only require the packager to
assemble the environment, not the end-user.

--
Ricardo