Re: Conda environments and reproducibility

[Hugo Buddelmeijer <hugo@buddelmeijer.nl>]

[-- Attachment #1: Type: text/plain, Size: 4007 bytes --]

Hi Ludovic,

On Fri, 2 Dec 2022 at 12:05, Ludovic Courtès <ludo@gnu.org> wrote:

> Hi,
>
> I read this thread with interest—great to have first-hand feedback from
> Conda users and packagers who also understand Guix!
>
> Hugo Buddelmeijer <hugo@buddelmeijer.nl> skribis:
>
> > That is, "conda env export" should contain entries like
> > "scipy=1.8.0=py39hee8e79c_1", where the hee8e79c should uniquely define
> the
> > dependencies 'that matter', like which compiler is used. What goes into
> the
> > hash seems rather complicated, and grows over time.
>
> I think one source of many problems here is to think that there are
> dependencies that do not matter.


In the Python world, most dependencies are runtime dependencies. Those do
not actually affect the build, or the build result, and therefore arguably
'do not matter'. (I disagree, because what matters is whether the software
runs and creates the right results.)


> Another one, which those hashes appear
> to address, is to think that a name/version pair is enough to
> unambiguously designate a software artifact.
>
> This hash is a hash of the build result, not a hash of the input, is
> that correct?
>

No, this conda build hash is used to identify the build environment, not to
identify a particular package build.

The easiest way to explain is to show an example. Here is a small part of a
"conda env export" of one of my environments:
  - pybind11-abi=4=hd8ed1ab_3
  - pycodestyle=2.8.0=pyhd8ed1ab_0
  - pycosat=0.6.3=py39h3811e60_1009
  - pycparser=2.21=pyhd8ed1ab_0
  - pydocstyle=6.1.1=pyhd8ed1ab_0
  - pyerfa=2.0.0.1=py39hce5d2b2_1
  - pyflakes=2.4.0=pyhd8ed1ab_0
  - pygments=2.11.2=pyhd8ed1ab_0
  - pyopenssl=22.0.0=pyhd8ed1ab_0
  - pyqt=5.12.3=py39hf3d152e_8
  - pyqt-impl=5.12.3=py39hde8b62d_8
  - pyqt5-sip=4.19.18=py39he80948d_8
  - pyqtchart=5.12=py39h0fcd23e_8
  - pyqtwebengine=5.12.1=py39h0fcd23e_8

As you see, many packages share the "hd8ed1ab" build hash, two qt-related
packages have h0fcd23e, and some others have their own. The "hd8ed1ab" hash
is by far the most common in this environment. These "hd8ed1ab" packages
are mostly independent (with separate maintainers, etc), but are probably
all in conda-forge and probably all use the 'default' conda environment.

(The last digit/number is the build number. The "8" suggests that all
qt-packages are actually built together, even though their build hash
differs.)

I don't really understand what goes into the hash. It is described on
https://docs.conda.io/projects/conda-build/en/stable/resources/define-metadata.html#build-number-and-string

The goal of these hashes is to capture which package builds will work
together. So two package builds with the same build-hash should have been
made with the same environment and thus work together.

I'm not sure how it works if the hashes are different. Maybe they are
merkle trees? So it is possible to determine whether one hash is a
'superset' of another hash. Probably not.


>
> I think it would be great to have a blog post that walks through
> shortcomings and concrete issues one may encounter when trying to
> reproduce a software environment with Conda, contrasting it with how
> Guix does thing.  This would probably make more sense for people who use
> Conda everyday than a high-level overview of Guix.
>

A key difference might be how to handle different combinations of versions.

E.g. you might want to use numpy 3.0 and scipy 18.0, while I want to use
numpy 6.0 and scipy 15.0 (made up numbers, but on purpose with one lower
and one greater between us). Conda and Guix solve this in fundamentally
different ways.

Conda-forge (as a project) is kinda in between conda alone and Guix, and
can kinda be seen as a linux distribution itself (sans kernel). Conda forge
is moving closer to Guix every year, including more and more dependencies,
and more shared recreate-everything moments.

Greetings,,
Hugo

[-- Attachment #2: Type: text/html, Size: 5311 bytes --]