From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp10.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms5.migadu.com with LMTPS id sID/C4gWhmN5xQAAbAwnHQ (envelope-from ) for ; Tue, 29 Nov 2022 15:26:16 +0100 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp10.migadu.com with LMTPS id AK4eC4gWhmOwHgAAG6o9tA (envelope-from ) for ; Tue, 29 Nov 2022 15:26:16 +0100 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id DCAD17602 for ; Tue, 29 Nov 2022 15:26:15 +0100 (CET) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1p01Yd-0001Iw-BJ; Tue, 29 Nov 2022 09:25:59 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1p01YX-0001HM-A3 for guix-science@gnu.org; Tue, 29 Nov 2022 09:25:53 -0500 Received: from mail-wm1-x32f.google.com ([2a00:1450:4864:20::32f]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1p01YV-0006om-8a for guix-science@gnu.org; Tue, 29 Nov 2022 09:25:53 -0500 Received: by mail-wm1-x32f.google.com with SMTP id ay14-20020a05600c1e0e00b003cf6ab34b61so14473479wmb.2 for ; Tue, 29 Nov 2022 06:25:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:message-id:date:references :in-reply-to:subject:cc:to:from:from:to:cc:subject:date:message-id :reply-to; bh=QIbLN34EVhzbILsRoc85podnYdPj62nkezBNLf5asOw=; b=WCv75pJGQs++1REX//0CZmCi44FlniAz2J6xPCQqvZd2TTRDMKmxwpn5wnGNzJrfeC dg9OZTvXCTepue35gIVq8h01zZFMLKVIf70ANJkImR1Gism/N84wUVvcfbrvOVssd564 cBmpLODcmzZ2GFOA9Kwk7Jn3ns+ErqgkLywT2OGG/zn21IwGMXo60kLSdOMwvaHK6MeI DsQozv1S4jdqku1doFFOcwOQfNOqcp+PdOaGxZPevHNcvIyn8MvlUFMassoHHVutIg3P jk7PMZzJnDTyJzxE80q+6NGteQtTbC7tjwSbT3s6XaACvLhjESbNEaPiAUzwIVKMoXw0 b4ZA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:message-id:date:references :in-reply-to:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=QIbLN34EVhzbILsRoc85podnYdPj62nkezBNLf5asOw=; b=xXld61IRpOjx3+wspLR76B98jTOjNfqGkxp+cfC/oKf5neU7A5bmIhQOEHwn8W/iX8 Rt0HMwvLR8VRbW4aUHiaK0ercLCX3vdyRQtpTq7ijEKa+NwXJyCCs5iTHSqfIuo+GP06 BqNWdNdXuD4YqNQc8e+0iEu7h7Cn3T9TwVKALsWUW4B1pT41YPTGnWidIlGDQGmn5wTP YJkAm6CP8ij3gNBXaOgswiiLSMvgGviCijOFtUl4vrK+TzQMoI14qmhVmDWMqWWbgGfA oAoKlLUSOZ3aS/DZwIUXdjd1fZhF/URfLi9NpAyxa3LwcKNgfocPpgZG4NpiwljiFdoJ 2myA== X-Gm-Message-State: ANoB5pnFS1QcAHlfrpH/BANrsxTOCaOdqVEdRPyO4GTiZgKl/IMpd+yP 93YsSfQJoP0+5dEwtavRPYnLZ6v0QGM= X-Google-Smtp-Source: AA0mqf7jYvl40JMKekwNkxu8oyj0bEid2JmEJjt2/bcJ3ub5dnNYo/FzxrCy3ULyefxQYHP3+xDpMA== X-Received: by 2002:a05:600c:34cd:b0:3cf:9877:c7e0 with SMTP id d13-20020a05600c34cd00b003cf9877c7e0mr35606240wmq.109.1669731943365; Tue, 29 Nov 2022 06:25:43 -0800 (PST) Received: from lili ([2a01:e0a:59b:9120:65d2:2476:f637:db1e]) by smtp.gmail.com with ESMTPSA id hn25-20020a05600ca39900b003b4935f04a4sm2574741wmb.5.2022.11.29.06.25.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 29 Nov 2022 06:25:42 -0800 (PST) From: Simon Tournier To: Thibault Lestang Cc: guix-science Subject: Re: Conda environments and reproducibility In-Reply-To: <87v8myypsg.fsf@imperial.ac.uk> References: <87pmd7ar8k.fsf@imperial.ac.uk> <86v8my7qpe.fsf@gmail.com> <87v8myypsg.fsf@imperial.ac.uk> Date: Tue, 29 Nov 2022 15:25:07 +0100 Message-ID: <86mt897s8s.fsf@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Received-SPF: pass client-ip=2a00:1450:4864:20::32f; envelope-from=zimon.toutoune@gmail.com; helo=mail-wm1-x32f.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: guix-science@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-science-bounces+larch=yhetil.org@gnu.org Sender: guix-science-bounces+larch=yhetil.org@gnu.org X-Migadu-Flow: FLOW_IN X-Migadu-Country: US ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1669731975; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post:dkim-signature; bh=QIbLN34EVhzbILsRoc85podnYdPj62nkezBNLf5asOw=; b=uqEI/MJJvNvbEhPkWATroP7Mi3eXEs2Kp+miCjq1I9+CRBQc0xVNPlM9UtluJam4HrUoCy FYKRv2MZ4KpYnvFibbKh7cjBzzkiLkW+3Mh3iUpmlLp8XHQTp6PYXxVugPhW0yH3sPEtCX AO3QynirSJOtgyFzOfneS3ucDh0sIeoFWK+vtlefM9Tb4nGU2snvnHfrk+3mH+o9I8hbcY Cag7C0VU+hp++JQ5XB+xiT2CKzWuxc+bmI0uMU3MPkc09Kazcl6nvtsPuvl5rB7NEGsN7g I75I1jCZk/hcsdy5K178ZuOYnJ8s2wrftf/W57qW2FYgiGWTiimn0ec6qjgIWQ== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1669731975; a=rsa-sha256; cv=none; b=bczL4OIDuuQbJE/ao3xFgdF/OyZmSyucixabr+a37WFyMvzoSmO4mRFbgGG8NqJh6Rdbtj dlrCG9lGEWa2hQi/mDRMamQDQ1IKfqUUHRKluphRF3OHCC9r+ZH7na6TCoKTif37i2GU4N 8IefexPz5MWAzM7W4Mf3DlD3+aUT4whsfGSSn+MwCxylq5JuYvctr+xASRO+X3NBefDTcP 5k63m9BQIEQtzEIHsqM1BeU72mlst3K+po/z5TBpIbRnsnukZ9+1AvlIz6EiOO9y++gKFy /RpHNG9hR/Mlit+E6c2h6mai1fty6rbhhNk7TiCsmp2owIFD5aCgx4F1uSutAg== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=WCv75pJG; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (aspmx1.migadu.com: domain of "guix-science-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-science-bounces+larch=yhetil.org@gnu.org" X-Migadu-Spam-Score: -4.46 Authentication-Results: aspmx1.migadu.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=WCv75pJG; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (aspmx1.migadu.com: domain of "guix-science-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-science-bounces+larch=yhetil.org@gnu.org" X-Migadu-Queue-Id: DCAD17602 X-Spam-Score: -4.46 X-Migadu-Scanner: scn0.migadu.com X-TUID: SRmPJzNJywWZ Hi Thibault, On Tue, 29 Nov 2022 at 10:41, Thibault Lestang w= rote: > I think the tweet above is about reproducing an enviroment after > effectively freezing constitutive packages and their dependenies as you > describe. They probably used something like > > conda env export > > Which outputs something similar to (trimmed) > > name: justnumpy > channels: > - defaults > dependencies: [...] > - ncurses=3D6.3=3Dh5eee18b_3 > - numpy=3D1.23.4=3Dpy310hd5efca6_0 > - numpy-base=3D1.23.4=3Dpy310h8e6c178_0 > - ... Do you list all the dependencies? Other said, dependencies of dependencies? Is it only run-time dependencies? Konrad pointed, (it =3D Conda) it claims that it cannot find a combination of package known to work together and available in the archive. and from my understanding, I think it is because the solver (SAT or else). Well, for instance, Theorem 1 Checking whether a single package P can be installed, giv= en a repository R, is NP-complete. https://www.mancoosi.org/edos/algorithmic/#toc15 Here (conda env export), you generated the Conda requirements using the repository in the state R. Then, later the repository becomes R=E2=80=99 (somehow it increases the number of combinations) and it does not matter if the constraints are foo <=3D 1.23 or are foo=3D1.2.3 or are foo=3D1.2.3=3Dabcd456. Maybe I am wrong, from my understanding, Conda builds the graph of dependencies by resolving a combinatorial problem. When you run, conda env create -f environment.yml then Conda relies on a =E2=80=9Cdependency=E2=80=9D solver documented here = [1]. And, IMHO, it is where it fails. Well, if instead of =E2=80=99conda env export= =E2=80=99 you run, conda list --explicit > spec-file.txt then later and elsewhere, conda create --name myenv --file spec-file.txt it should bypass the solver. But the documentation [1] reads, Since the solver is not involved, the dependencies of the explicit package(s) are not processed at all. This can leave the environment in an inconsistent state, which can be fixed by running conda update --all, for example. Done. :-) Conda environments are hard, if not impossible, to reproduce when time is flying. It is by design, IMHO. 1: 2: Cheers, simon