From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp11.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms5.migadu.com with LMTPS id qBHbOKoehWODPQAAbAwnHQ (envelope-from ) for ; Mon, 28 Nov 2022 21:48:42 +0100 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp11.migadu.com with LMTPS id 6G28OKoehWOeAwEA9RJhRA (envelope-from ) for ; Mon, 28 Nov 2022 21:48:42 +0100 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 6E8022A33 for ; Mon, 28 Nov 2022 21:48:42 +0100 (CET) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ozl3K-00075U-6k; Mon, 28 Nov 2022 15:48:34 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1ozl3I-000758-6J for guix-science@gnu.org; Mon, 28 Nov 2022 15:48:32 -0500 Received: from mail-wr1-x42d.google.com ([2a00:1450:4864:20::42d]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1ozl3F-0006D5-Hn for guix-science@gnu.org; Mon, 28 Nov 2022 15:48:31 -0500 Received: by mail-wr1-x42d.google.com with SMTP id z4so18952465wrr.3 for ; Mon, 28 Nov 2022 12:48:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:message-id:date:references :in-reply-to:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=Lxd9YjaTKjrAU6BUnQVLdnUFNQwzjQnhaLwU1mJKf1w=; b=WuoXsHaA1FeDaCwuLl0PdXtV3SIDk5QtfEKnq2HITCXbNfkj0pAdceRCAqMFeYRO6C nhBsHfdOU80DQd2+/LXQIX0f3FfMEstDeQkGyOME4uyPqxGUDuFUHpQbiFOGTOkvkm+Q xKJaoP3xG92XAQ/m1AIlgRakcwdrmmDr58I/0e9HwbmblnrtfsgGK7f1ZYueLmIin7ml OKX0zFGmB/z9OTU5nu0/gnRvRF9kBQA3SD0ngnrdx1/hoZDbpR95jWu6khYnw2RAeu8f z1ifeeHPr+4SpjbfokCwCL/7oXUSgZD32gzpJpVwrHUMm+GhEnGWjp0ATPo9fVapdBH0 XhaQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:message-id:date:references :in-reply-to:subject:to:from:x-gm-message-state:from:to:cc:subject :date:message-id:reply-to; bh=Lxd9YjaTKjrAU6BUnQVLdnUFNQwzjQnhaLwU1mJKf1w=; b=Vzf/dt3x0XNsBixjiLQKevkz6L9K9ZrGUGOXKvrKllhfkM72/QmDZniXa1EkPqIPBj YnhBchNEW0qPLh4h4ACOzYsLEfS7wlM29Oq2TbCYbB7dHjP9iWky4KZan3AeO3YpISOt bUvclItaBPjTsbUgGm0ttMIz94AmYdfZKT94F8lIebbej0Y5TLliMOFvk15fNpdpklcd UMxjQ/876u668U9IYzqWpAsMt9GNOA6YTmxVl1jtyu6vcwIwHNr2vgflGuTKmlNJgLMM /q0FWLfbBbhK+zWwInadnJGGnUI170JKuFEfKDB0RF/YE4NRoFZBEYClWMWIarF1QmCY by/g== X-Gm-Message-State: ANoB5pnGVr3DXdH5cqtCdD9fqlUVNk5EDxJ32f6gn5VBI1ckN3a/y1DF lIOgNae0G+TGLV3lW6yPU08yXm+Oulc= X-Google-Smtp-Source: AA0mqf5d+01oLZrcFUobWGdK+0/L4f01VsNNyhcz2Vjzx0//Wdf1zsKyBLG5xe8+Skty/VKrayoV/Q== X-Received: by 2002:adf:db4f:0:b0:242:1bbc:9f03 with SMTP id f15-20020adfdb4f000000b002421bbc9f03mr3175337wrj.377.1669668506884; Mon, 28 Nov 2022 12:48:26 -0800 (PST) Received: from lili ([2a01:e0a:59b:9120:65d2:2476:f637:db1e]) by smtp.gmail.com with ESMTPSA id co29-20020a0560000a1d00b00236e834f050sm12167449wrb.35.2022.11.28.12.48.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 28 Nov 2022 12:48:26 -0800 (PST) From: Simon Tournier To: Thibault Lestang , guix-science Subject: Re: Conda environments and reproducibility In-Reply-To: <87pmd7ar8k.fsf@imperial.ac.uk> References: <87pmd7ar8k.fsf@imperial.ac.uk> Date: Mon, 28 Nov 2022 21:46:05 +0100 Message-ID: <86v8my7qpe.fsf@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Received-SPF: pass client-ip=2a00:1450:4864:20::42d; envelope-from=zimon.toutoune@gmail.com; helo=mail-wr1-x42d.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: guix-science@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-science-bounces+larch=yhetil.org@gnu.org Sender: guix-science-bounces+larch=yhetil.org@gnu.org X-Migadu-Flow: FLOW_IN X-Migadu-Country: US ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1669668522; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post:dkim-signature; bh=Lxd9YjaTKjrAU6BUnQVLdnUFNQwzjQnhaLwU1mJKf1w=; b=VJflHEp1/VuN7monPo5tHfpxNCT+gvCRSCiE3T+jLjIwGumle1SLtHxarKpUTNgGfHMbPp 4JBtZxj1Fyica0jynjZ0rewlaKoErBaFwQs3W8AEIFaNh+n18CsOnU9eRI7F7Q9zh8qj6W 92pHiopk3CW0be+FHOBR3M9VHUmuyfn9j5TxXi3Yto9fQZQCergTGA5qDGsaZqP8UZBKUj aTnZMqyp/8zDwEYhDyL4jM/YJM3gSj2Bf4ec43N7KPyqRGoyulAq8w4QW9JFFZkUVzI+Cl spJ14mtYlBzvcQWnwQBZYon0eI8RprFw0Bbn5DgKLlcxU9+fQ3pyxuBPefhEjw== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1669668522; a=rsa-sha256; cv=none; b=VIrMEjfcfIAmH8/bwXbK8GnCM9+BEDT83ABfLvvlXS/4GI1bbrahF4HxpB/W4+eWSv8l4K THVNoRQLvEtouCel7xIGIAlQP8bcVeCBaoNS4xY0EoSy720crbJY4ftQgXkwr1UWzePX6H Z2kBllV0RgM80w9RQ+S6Po+Pq+iGX/HxiI92sc2Yu7WkZ3tztj4SrSv6bzJ+O6JJiffZ7H /UXYCnfesBPrRfRBIJdETX9RKhpgZCbbAo9vGSH/hcN2GubrEbf08x/iIN/0ovyPOxDyYx teMx30S6senFGiEfD/0omkLYYK8dl3BeV3IH2c/U8ZyJeYHHWMwIOdHFY/0uPg== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=WuoXsHaA; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (aspmx1.migadu.com: domain of "guix-science-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-science-bounces+larch=yhetil.org@gnu.org" X-Migadu-Spam-Score: -5.46 Authentication-Results: aspmx1.migadu.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=WuoXsHaA; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (aspmx1.migadu.com: domain of "guix-science-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-science-bounces+larch=yhetil.org@gnu.org" X-Migadu-Queue-Id: 6E8022A33 X-Spam-Score: -5.46 X-Migadu-Scanner: scn0.migadu.com X-TUID: IBkjdpGJIM4/ Hi, On Mon, 28 Nov 2022 at 17:28, Thibault Lestang w= rote: > ----- > @luispedrocoelho > Me, 6 months ago: I am going to save this conda > environment with all the versions of all the packages so it can be > recreated later; this is Reproducible Science! > > conda, today: these versions don't work together, lol. > ----- > > I simply can't explain how such a behavior can happen. One thing is the link rot. I do not know if it is currently estimated, but for sure, we always underestimate it. > I understand that conda ships pre-compiled binaries. I see how that's > bad for reproducibility and provenance tracking since it's not > straightforward to know how these binaries and dependencies were > compiled. I'm assuming that, when conda saves an environment, it records > version tags and "everything else required" to pull the same binaries > later. Okay - I see how binaries could /technically/ be modified at a > later stage whilst maintaning the same version tag (provenance tracking > issue). Aside, you are assuming the availability of such binaries. :-) Another thing, from the old time where I used Conda, and I may be wrong, is, I guess , the SAT solver [1]. Well, 6 months ago, you described your environment, for instance saying: 1.0 <=3D foo 2.0 <=3D bar <=3D 3.0 baz <=3D 4.0 then foo@1.1, foo@1.2 and foo@2.0 had been released in these past 6 months. But baz <=3D 4.0 only works with 0.9 <=3D foo <=3D 1.2 and the constraint on bar implies other constraints on foo and/or baz. The complexity about SAT solvers is exponential, IIRC, for sure really bad, and I do not know the state-of-the-art but I guess the problem to solve is going to be worse and worse as the time flies. >From my experience, you have only one solution to fight against the time: freeze. The question is then how or what to freeze. :-) One way for freezing is the binary container. Another way for freezing is to have a =E2=80=9Csummary=E2=80=9D capturing the whole (fixed) graph of dependencies. This is (usually named) the channels.scm file (guix describe). Then, the assumptions become: 1. solve the link rot; tackled by Software Heritage, 2. Linux kernel API backward compatibility, 3. hardware compatibility, to be able to rebuild. If I might, here some stuff: :-) https://www.nature.com/articles/s41597-022-01720-9 https://simon.tournier.info/posts/2022-11-08-bluehats.html https://simon.tournier.info/posts/2022-04-15-cafe-guix-long-term.html Cheers, simon 1: https://en.wikipedia.org/wiki/SAT_solver