From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp10.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms5.migadu.com with LMTPS id GOTfBUb3iWPJPgAAbAwnHQ (envelope-from ) for ; Fri, 02 Dec 2022 14:01:58 +0100 Received: from aspmx1.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp10.migadu.com with LMTPS id KDIeBUb3iWOZHwAAG6o9tA (envelope-from ) for ; Fri, 02 Dec 2022 14:01:58 +0100 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id B81BFB4B2 for ; Fri, 2 Dec 2022 14:01:57 +0100 (CET) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1p15fk-0007D9-K0; Fri, 02 Dec 2022 08:01:44 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1p15fh-0007Aj-CA for guix-science@gnu.org; Fri, 02 Dec 2022 08:01:41 -0500 Received: from wout5-smtp.messagingengine.com ([64.147.123.21]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1p15fd-0005I4-Bh for guix-science@gnu.org; Fri, 02 Dec 2022 08:01:41 -0500 Received: from compute5.internal (compute5.nyi.internal [10.202.2.45]) by mailout.west.internal (Postfix) with ESMTP id 752813200908; Fri, 2 Dec 2022 08:01:29 -0500 (EST) Received: from mailfrontend1 ([10.202.2.162]) by compute5.internal (MEProxy); Fri, 02 Dec 2022 08:01:29 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:date:date:feedback-id:feedback-id:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:sender:subject:subject:to:to:x-me-proxy:x-me-proxy :x-me-sender:x-me-sender:x-sasl-enc; s=fm1; t=1669986088; x= 1670072488; bh=f2ip7ZM+oIdhoseqCquSPZfeq8GFspJ4eLo5ftZ+Fw0=; b=g 0ilPjfLvHT1e3Z4NMYz9X3f0zt10Kx8aDYQIokWDsq/B8vb9BcK/1fRVcAAOHhGQ dFb5G+FjSOTgnUitAHn6yYhFbRRH5g5lTHA+AB1iY2jz/DmnHeaz5W4hHahoKII0 f/v0WXyuO4J5wBkQcIs2KGuhFp9X494htB1i6ELJgqEn+rpvl/6lM7U673TtB4qI lL7ZPLheMGjf17pMTENweS4IS3hzvS+LDhP/CfUU5A00Z8boyLJKQJp3ECBjl8Hm Svk7e6qnDdOXgiGKak204+3KASSDPKBN6iDVc9PjD7HBfzs21uK/39r6rLJogXsD 0dzI+T8GUFGKiu0JSpJcw== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvhedrtdekgddvvdcutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc fjughrpefhvfevufgjfhffkfggtgfgsehtqhertddttdejnecuhfhrohhmpefmohhnrhgr ugcujfhinhhsvghnuceokhhonhhrrggurdhhihhnshgvnhestghnrhhsrdhfrheqnecugg ftrfgrthhtvghrnhepffefieelheduiedvtdelledvheehvddvudffgfffgeeujeejgeeh ueejgedtteejnecuffhomhgrihhnpegtnhhrshdqohhrlhgvrghnshdrfhhrpdhorhgtih gurdhorhhgnecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhho mhepkhhonhhrrggurdhhihhnshgvnhestghnrhhsrdhfrh X-ME-Proxy: Feedback-ID: if40146ab:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Fri, 2 Dec 2022 08:01:25 -0500 (EST) From: Konrad Hinsen To: Hugo Buddelmeijer Cc: guix-science Subject: Re: Conda environments and reproducibility In-Reply-To: References: <87pmd7ar8k.fsf@imperial.ac.uk> <87zgcayre2.fsf@imperial.ac.uk> Date: Fri, 02 Dec 2022 14:01:24 +0100 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Received-SPF: softfail client-ip=64.147.123.21; envelope-from=konrad.hinsen@cnrs.fr; helo=wout5-smtp.messagingengine.com X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, RCVD_IN_DNSWL_LOW=-0.7, SPF_HELO_PASS=-0.001, SPF_SOFTFAIL=0.665 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: guix-science@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-science-bounces+larch=yhetil.org@gnu.org Sender: guix-science-bounces+larch=yhetil.org@gnu.org X-Migadu-Flow: FLOW_IN X-Migadu-Country: US ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1669986117; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post:dkim-signature; bh=f2ip7ZM+oIdhoseqCquSPZfeq8GFspJ4eLo5ftZ+Fw0=; b=Glu4YfSOOXjGXsyd+wJ0gYJr89KvduSGm/Qn1SwGhoEBN8PA97m5/UH2f24gdT6R+ChqqS hIjXXzMI2X6y8yQerE0pA6M3qX8r3tnIUsDtiKS1BnJgimCjMRCxHZ4ibEVYX5DaifpBbk EKKw4W7jACaTd7qvpee03iCDZoLKZy5x363yjhzvntTixoHuHz1cjvXS1jIYACv+l4G3Z9 eJWRMV9EUio60x617rETY+s0dlbuQKeae1HdWjaWh0CKiN8q8h/XcB1NFltiaqauSvToUF CCQpNoOp9sIzMrE9kYDnKo56pZ6CbWPyl1nUe/b4ivttOB4qKvP9vaRJ2cRZsg== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1669986117; a=rsa-sha256; cv=none; b=TLp/8J68DZZznrnlo5Kbh1CtLiP/CpZlm33CeC6JR5BGODGaL1Roj/yZgBSsnSfTn3LZ2N osc26DVtckHZJGxBQcm6dDXfJoeoGEUxNsKVRkiZvTjK6LkRz76hhjOnTGD8xhKJWY8jta pvieOgR1HU1Kn66AwJJHzMY8gsk2cIdWjA4eqmSP4tkyQ99B4csPd9jH70cUzWfTAG4W2f Nl30Hcv07ofqcLsclhx0weuqy4NxceHe3g2FG9n06V0L037HEbiH0T3kgP/Uh3SlNTw5J7 DMc5pUje/omTcpchyJPiJAjntEQPLDo/KKo2zLOmuUKcHOCWDTXJBmL341b6Hw== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=messagingengine.com header.s=fm1 header.b="g 0ilPjf"; dmarc=none; spf=pass (aspmx1.migadu.com: domain of "guix-science-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-science-bounces+larch=yhetil.org@gnu.org" X-Migadu-Spam-Score: 2.63 Authentication-Results: aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=messagingengine.com header.s=fm1 header.b="g 0ilPjf"; dmarc=none; spf=pass (aspmx1.migadu.com: domain of "guix-science-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-science-bounces+larch=yhetil.org@gnu.org" X-Migadu-Queue-Id: B81BFB4B2 X-Spam-Score: 2.63 X-Migadu-Scanner: scn1.migadu.com X-TUID: JHNWdDOACtpM Hi Hugo, > Thanks. Those dependencies indeed do not contain the hashes, so it is > probably created with "conda env export --no-build". I can't say, I didn't set up this environment. > I think such a file without build hashes would probably be what you want > when you are giving a course, because it would allow students to install > these exact versions of the packages, but build for their specific > environment (e.g. Linux / macOS / Windows). It would provide limited That was exactly our objective. And we knew that in theory, reproducibility and multi-platform are incompatible. We just hoped that the conda approach would work long enough for the purposes of our course. It didn't. > reproducibility in the future, as you noticed. I guess you'd want three > sets of environment files for a conda environment for a course: Sounds good... in theory. In practice, we'd have to explain the reasons for these three environment files. Which we could do at best at the end of the course. And even then, it would have been a difficult task, as we couldn't go into all the details in a course aimed at junior researchers with little technical background in computing. > However, nowadays everyone can run linux, either directly, or through WSL > (windows subsystem for linux), or through containers. And everyone knows > how to do this, and it is integrated in IDE's and such. So conda isn't > really necessary anymore. Indeed. We are currently working on a follow-up for dealing with reproducibility at scale (big data, complex code, long computations). We decided to give up multi-platform, and concentrate on Linux (explaining why). The two approaches to reproducible environment we plan to cover are - Docker containers from reproducible Dockerfiles, based on Debian snapsho= ts - Guix The point is that, once you accept that Docker images are acceptable only when reproducible, Guix appears as a simplification. > I agree with you on a philosophical level; ultimately understanding > everything would be easier with guix. But we aren't there yet, I don't > understand most of the guix packages I've looked at. That is probably > because my guile/scheme skills are lacking. Maybe not. A big part of the complexity of Guix packaging is the need to patch most software, in order to make its build reproducible and in order to remove tacit dependencies in the build process on FHS conventions. Once Guix becomes the norm in the Linux world, the next step is to encourage software developers to develop with Guix in mind. Produce software that doesn't require any patches to compile under Guix. World dominance is in sight! ;-) Cheers, Konrad. --=20 --------------------------------------------------------------------- Konrad Hinsen Centre de Biophysique Mol=C3=A9culaire, CNRS Orl=C3=A9ans Synchrotron Soleil - Division Exp=C3=A9riences Saint Aubin - BP 48 91192 Gif sur Yvette Cedex, France Tel. +33-1 69 35 97 15 E-Mail: konrad DOT hinsen AT cnrs DOT fr http://dirac.cnrs-orleans.fr/~hinsen/ ORCID: https://orcid.org/0000-0003-0330-9428 Twitter: @khinsen ---------------------------------------------------------------------