From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp12.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id OEISGSh+DmIb3AAAgWs5BA (envelope-from ) for ; Thu, 17 Feb 2022 17:56:08 +0100 Received: from aspmx1.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp12.migadu.com with LMTPS id GKyyFSh+DmI9VwAAauVa8A (envelope-from ) for ; Thu, 17 Feb 2022 17:56:08 +0100 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 1689237D7C for ; Thu, 17 Feb 2022 17:56:08 +0100 (CET) Received: from localhost ([::1]:49220 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nKk4d-00048e-7g for larch@yhetil.org; Thu, 17 Feb 2022 11:56:07 -0500 Received: from eggs.gnu.org ([209.51.188.92]:52810) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nKk4S-00048R-K5 for guix-devel@gnu.org; Thu, 17 Feb 2022 11:55:56 -0500 Received: from out5-smtp.messagingengine.com ([66.111.4.29]:38223) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nKk4P-0007fm-SJ; Thu, 17 Feb 2022 11:55:56 -0500 Received: from compute5.internal (compute5.nyi.internal [10.202.2.45]) by mailout.nyi.internal (Postfix) with ESMTP id 916375C0134; Thu, 17 Feb 2022 11:55:48 -0500 (EST) Received: from mailfrontend2 ([10.202.2.163]) by compute5.internal (MEProxy); Thu, 17 Feb 2022 11:55:48 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fastmail.net; h= cc:cc:content-type:date:date:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:sender:subject :subject:to:to; s=fm2; bh=s9KryvHkta0A3ZvG/9GcBB9bLOnpi2vfFX8nCo C0p9o=; b=hx8nmBVM/+X3HjKHmIsa7XKvzT9pDrgBb1OkERb46Uxc6yOd58R9YS XHzW8K3Vy3pn8v+nn4HULfsmp+9CM7PAfMJNhnHFjg4UP0B40u4wRPeYdf1au2HT bSqsYY0VjfOy9dbOw+osmFh2rRR05eAhkkoUHOiTuHUirVTSvt1Kc8Sk53MM58XP Ny2DZy3RwVljt7VTm5MBRSRASlMAb3D+UcmsKRd3EcpYcjod0+pqkmfsZ7DBi0iq 0NMqRUMsnEtLjG0Y978MXTYdvd4qyTk3xK6MbxNC9b27SaM+NtP3rtvUVfW5HEpE joVbxV8zGdtNIBb8hii5Md6FtRNqtYsQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:date:date:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:sender:subject:subject:to:to:x-me-proxy:x-me-proxy :x-me-sender:x-me-sender:x-sasl-enc; s=fm2; bh=s9KryvHkta0A3ZvG/ 9GcBB9bLOnpi2vfFX8nCoC0p9o=; b=iqV8svV/vsOMrGPYGZAMwenEO3DSMW8gO jSelR0vgMwiDZf3uktZiNKk3uEzcTCRAUxdaiu8T/hhbjrKxgTEUe3yCfMVV7NLV OcCMmhwL0C7ZfOT4bwUOPUQkALa5rwX3o2o6lxHrR6307kiGdGQRK9UtEKDvS145 0WIkS+7c8KGUql/jv4NMDXRu8p8i2iQf26dTCiASQN3FcNUnTvUb+HNyYET6A8Z0 wOpBbgwn2JFHXjS4GAdFSkggjrnv/wSEFblGFmYmrYYwIijqbECLMrPFmO9vfhn9 w07pVXIQsJ46Dckj06ERhat5tjaDkVrOalZPqA8AZI6eFj+bFx2qQ== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvvddrjeekgdelvdcutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc fjughrpefhvffujghffffkgggtsehttdertddttddtnecuhfhrohhmpefmohhnrhgrugcu jfhinhhsvghnuceokhhonhhrrggurdhhihhnshgvnhesfhgrshhtmhgrihhlrdhnvghtqe enucggtffrrghtthgvrhhnpeeikeejjeevueeifeejhedtgeethfdutefgveffhfeuheej veeiieegvdfhtdeifeenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrih hlfhhrohhmpehkohhnrhgrugdrhhhinhhsvghnsehfrghsthhmrghilhdrnhgvth X-ME-Proxy: Received: by mail.messagingengine.com (Postfix) with ESMTPA; Thu, 17 Feb 2022 11:55:47 -0500 (EST) From: Konrad Hinsen To: zimoun , Bengt Richter , Ludovic =?utf-8?Q?Court=C3=A8s?= Subject: Re: Investigating a reproducibility failure In-Reply-To: <86pmnl4u28.fsf@gmail.com> References: <871r0l9fd1.fsf@gmail.com> <87tuddh06r.fsf@gnu.org> <20220215141031.GA13837@LionPure> <86bkz7587m.fsf@gmail.com> <86pmnl4u28.fsf@gmail.com> Date: Thu, 17 Feb 2022 17:55:45 +0100 Message-ID: MIME-Version: 1.0 Content-Type: text/plain Received-SPF: pass client-ip=66.111.4.29; envelope-from=konrad.hinsen@fastmail.net; helo=out5-smtp.messagingengine.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_LOW=-0.7, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Guix Devel Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: "Guix-devel" X-Migadu-Flow: FLOW_IN X-Migadu-Country: US ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1645116968; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post:dkim-signature; bh=s9KryvHkta0A3ZvG/9GcBB9bLOnpi2vfFX8nCoC0p9o=; b=XQ70J3fUdsCYJdfyo21BIRwokSOnlP/p5HGAnq+EX8EMVOYrJbzxfkZ3K2z3INed2rJcus qQ3pNoBfsVmlmrC5Eh6ziEHGDv7xLR+xA4HtiDoKCt97VYtYqTptWz8OsVVNzPkLk4cJr9 KA+q451m50eqL6iwS1EWBn7Ve9LPhEZHmWkqZQ0vRKWB2pVS0LsCKl8FKPBRmeLDT0B7Y1 LubVQYhPe0vy007RPdxJLuttFl0Dnfl17cvqKwJqVxKnwWxkfrt4pUfnSuoID0dJdf4ToO 98lLbLMCQHhOwk4TW+oXEoKWF/bhjioRU9UMqNpqCLcxCPIDyl8+lR2ovkhg2w== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1645116968; a=rsa-sha256; cv=none; b=ImWSepHZIGzAabFFbd19kCMdysMDBUy1KCEMdxBV5NWerv709gsfXPUGypYRpU3urd8ldY rCJ+445fpclHa8iPAeJKWiesdWngHGs3LYES4nk0ez52gYsEd8cj4XnllC/AqN7IMZJbl4 hmJj2lPzXeKK4XWEjnymUFFmqLnR0MYiFcM21vXYKGyJwUYQX1l+PTnYBeR0fXaqi9qivP IvPmJE38N/I8U19cgTzLpyW+fNDkGNSfSgDzMQfWsoYqVjcnHwEScRJMsgUY/aD1hna878 2fEPzWUbY4AYAed32onAwKsTH/mbpYG/KDHwIKZYNMh7dLSgyAr/r9OqFShVXQ== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=fastmail.net header.s=fm2 header.b=hx8nmBVM; dkim=fail ("headers rsa verify failed") header.d=messagingengine.com header.s=fm2 header.b="iqV8svV/"; dmarc=fail reason="SPF not aligned (relaxed)" header.from=fastmail.net (policy=none); spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org" X-Migadu-Spam-Score: -3.13 Authentication-Results: aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=fastmail.net header.s=fm2 header.b=hx8nmBVM; dkim=fail ("headers rsa verify failed") header.d=messagingengine.com header.s=fm2 header.b="iqV8svV/"; dmarc=fail reason="SPF not aligned (relaxed)" header.from=fastmail.net (policy=none); spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org" X-Migadu-Queue-Id: 1689237D7C X-Spam-Score: -3.13 X-Migadu-Scanner: scn0.migadu.com X-TUID: rZ1uqKGg6lOV Hi Simon, > We are far from OpenBLAS. :-) That's fine with me. The more distance between me and OpenBLAS, the happier I am ;-) > On Wed, 16 Feb 2022 at 14:04, Konrad Hinsen wrote: > >> Making scientific computations bit-for-bit reproducible is the moral >> equivalent of keeping a detailed lab notebook: doing your best to tell >> others exactly what you did. > > A detailed lab notebook implies transparency and full control of > variability, not bit-for-bit reproducibility. That's why I said "moral" equivalent. Computations are different from experiments. Typical mistakes are different, and technical possibilities are different. 1. You can't have the equivalent of bit-for-bit reproducibility with experiment. You can with computers, and with good tool support (Guix!) it can become a routine task that takes little effort. So... why *not* do it? 2. A computation involves many more details than any typical experiment. Just writing down what you did is *not* enough for documenting a computation, as experience has shown. So you need more than the lab notebook. If your computation is bit-for-bit reproducible, you know that you have documented every last detail. Inversely, if you cannot reproduce to the bit level, you know that *something* is out of your control. In the end, my argument is more pragmatic than philosophical. If bit-for-bit reproducibility is (1) useful for resolving issues in the future, and (2) cheap to get with good tool support, then we should go for it. The main reason why people argue against it is lack of tool support in their work environments. They conclude that it's a difficult goal to achieve, and then start to reason that it's not strictly necessary for the scientific method. Which is true. But... it's still very useful. >> And that's the role of bit-for-bit reproducibility. > > From my understanding, the validation of a reproduction depends on > trust: what is the confidence about this or that? Well, bit-for-bit > reproducibility is one criteria for establishing such trust. However, > IMHO, such criteria is not the unique one, and defeating it can be > compensated by other criteria used by many experimental sciences. Definitely. But in many cases, bit-for-bit reproducibility is the cheapest way to build trust, given good tool support. In other cases, e.g. HPC or exotic hardware, it's expensive, and then you look for something else. Cheers, Konrad.