Re: Conda environments and reproducibility

[Konrad Hinsen <konrad.hinsen@cnrs.fr>]

Hi Thibault,

> -----
> @luispedrocoelho
> Me, 6 months ago: I am going to save this conda
> environment with all the versions of all the packages so it can be
> recreated later; this is Reproducible Science!
>
> conda, today: these versions don't work together, lol.
> -----
>
> I simply can't explain how such a behavior can happen.

The error message is not exactly as cited. Conda doesn't claim that
these versions don't work together, it claims that it cannot find a
combination of package known to work together and available in the
archive.

One possible reason for this is an update in conda's build
infrastructure. That's what happened to the software environment for the
reproducible research MOOC on Fun (of which I am an author). We
published the environment file, but a few months later conda could not
reconstruct it any more. They had updated the compiler infrastructure,
which requires a rebuild of all packages. But they didn't rebuild all
the versions from the past, so most older environment files became
unusable.

The lesson is that packages are reproducible only if you can re-run the
construction of the entire environment, from source code. Which is what
Guix can do (though if you actually have to do this, it will be a very
long process).

There may be other causes for the conda problem cited, I don't claim to
be an authority of conda! After the MOOC experience, I have never used
conda again.

> Is it the case that someone at Anaconda would modify some package,
> keeping the same version tag and other identifiers used by conda, whilst
> at the same time marking this package as incompatible with packages it
> was previously compatible with?

That's in a way what happened in my scenario: rebuilding with a new
compilation infrastructure produces different packages that share
version numbers and tags with the prior ones.

Cheers,
  Konrad.
-- 
---------------------------------------------------------------------
Konrad Hinsen
Centre de Biophysique Moléculaire, CNRS Orléans
Synchrotron Soleil - Division Expériences
Saint Aubin - BP 48
91192 Gif sur Yvette Cedex, France
Tel. +33-1 69 35 97 15
E-Mail: konrad DOT hinsen AT cnrs DOT fr
http://dirac.cnrs-orleans.fr/~hinsen/
ORCID: https://orcid.org/0000-0003-0330-9428
Twitter: @khinsen
---------------------------------------------------------------------