From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp11.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id 0Kx7G4v2DGJASAAAgWs5BA (envelope-from ) for ; Wed, 16 Feb 2022 14:05:15 +0100 Received: from aspmx1.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp11.migadu.com with LMTPS id OFi7GIv2DGKkYQEA9RJhRA (envelope-from ) for ; Wed, 16 Feb 2022 14:05:15 +0100 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 202AE3B3DD for ; Wed, 16 Feb 2022 14:05:15 +0100 (CET) Received: from localhost ([::1]:55668 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nKJze-0001wk-9s for larch@yhetil.org; Wed, 16 Feb 2022 08:05:14 -0500 Received: from eggs.gnu.org ([209.51.188.92]:40052) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nKJz6-0001v5-RV for guix-devel@gnu.org; Wed, 16 Feb 2022 08:04:40 -0500 Received: from out4-smtp.messagingengine.com ([66.111.4.28]:37815) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nKJz4-0005tp-74; Wed, 16 Feb 2022 08:04:40 -0500 Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailout.nyi.internal (Postfix) with ESMTP id 69D065C01ED; Wed, 16 Feb 2022 08:04:35 -0500 (EST) Received: from mailfrontend1 ([10.202.2.162]) by compute1.internal (MEProxy); Wed, 16 Feb 2022 08:04:35 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fastmail.net; h= cc:cc:content-transfer-encoding:content-type:date:date:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:sender:subject:subject:to:to; s=fm2; bh=HVYjCX9o08Y/bD DueniIg0GyEIgyFTrFMpmqWQZDKt0=; b=NadCH+geQBUfSfNE9Dv/3KSHNmaL1Q 5aHbJIaDum4zBqk6iICSbr0baeZhT/tCvj0nARrFT7DtkXHk9tWvcKQFO0NEAXHY uL9WEomim+8lsJbrCnz62cOtuz0HUDSAIKUFhFs5qmgZgAQIu9/fOdlJ6PUgD+e4 PghNthlCSORQ5C9ewq2cBVVY1Y2Ekoytfim5YvW7TjoSctKRdchJGppbpywAxrpN 2p4y7AEAHnuRAJ3cBgm11exWnoZPLe+xz7zcuvE7gDeOsJm9G+QiJYMZKQJlM3yg y2haKgXMbyQoYWKflVGotIHiPr3K1MVnNPI3XKFdtUXVC34CS38LrlMg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:date:date:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:sender:subject :subject:to:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm2; bh=HVYjCX9o08Y/bDDueniIg0GyEIgyFTrFMpmqWQZDK t0=; b=j08YMzeisQbdIZtv1mn4uAriW8EAK/tx+U2MWTVYA+xjK/WQpWjlxJdva wFo4iHY70M3xVWHhdS00DFDBDkUX4+A65dSmpF1JiD1SXzwFClMoPgTsmQha6PmL mKlmCkngsJdK7WrD90fp6URqFl1DFQ9Fy9xECZKU/r/9H0pqmT8VmwHLrMrMOyMs CZFm6Idd+8Ah3R2McqAp9pn3hRP/Uo4SsVnhWLWEBsNYn5zrym71JVFB1Et9G+u1 y3PtIoiR1xGUK/etxnogdoyKFsLLXzGj3ElPCpwUa6HOaVKDaEgJoibk0ciN7shK G8OvmLcPq/6huDwvHfvEly/6nfBaw== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvvddrjeeigdegjecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc fjughrpefhvffujghffffkgggtgfesthhqredttddtjeenucfhrhhomhepmfhonhhrrggu ucfjihhnshgvnhcuoehkohhnrhgrugdrhhhinhhsvghnsehfrghsthhmrghilhdrnhgvth eqnecuggftrfgrthhtvghrnhepveehleevgeejuddvgfduieefveetffdvteegfeekgfdu ueejfeegvdehgeekhfejnecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrg hilhhfrhhomhepkhhonhhrrggurdhhihhnshgvnhesfhgrshhtmhgrihhlrdhnvght X-ME-Proxy: Received: by mail.messagingengine.com (Postfix) with ESMTPA; Wed, 16 Feb 2022 08:04:34 -0500 (EST) From: Konrad Hinsen To: zimoun , Bengt Richter , Ludovic =?utf-8?Q?Court=C3=A8s?= Subject: Re: Investigating a reproducibility failure In-Reply-To: <86bkz7587m.fsf@gmail.com> References: <871r0l9fd1.fsf@gmail.com> <87tuddh06r.fsf@gnu.org> <20220215141031.GA13837@LionPure> <86bkz7587m.fsf@gmail.com> Date: Wed, 16 Feb 2022 14:04:33 +0100 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Received-SPF: pass client-ip=66.111.4.28; envelope-from=konrad.hinsen@fastmail.net; helo=out4-smtp.messagingengine.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Guix Devel Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: "Guix-devel" X-Migadu-Flow: FLOW_IN X-Migadu-Country: US ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1645016715; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post:dkim-signature; bh=HVYjCX9o08Y/bDDueniIg0GyEIgyFTrFMpmqWQZDKt0=; b=tYK7dU/1AGbC79vETUc6IGLpghbYDAMMiAaz7lpRJUoElUo578flYQ9ble9oJNyNIoJnwq 3am0U7oPpiOWPk7ZrxhAY46XSVfbYnESkLN9zPLNu/2CAlmGKuZF323n3JTaPaJ+aFKtA8 HSkjDpzIXBr7BXUGbdR4Y7YKbb4qeNrTgALXwqZ8y3qX73B8DzYUX8sGTzpU1FEwtGv10J IAIv4QvibgCbVy5F7KpOs4rugs2gD/RAZ9S5Sp8RMFMsK3rvHbUZljy+/dOWFpoVojX16v qdXGU25SKUop8KdNyY3VcIqz/iIZ2pTlJh4fvNSVChGq27pSEW/++gHTml9X2w== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1645016715; a=rsa-sha256; cv=none; b=QKDssivwDqzOzXJrt9GBMmGp+XZD9N+QyyBwE7BP3k4jW6ZRW0dXh1r25Cc9jsJ4+6CHBw jdabmGqNzVUvjhHlPNeSnfPeQUldU/yAFI0yFrYv++hsJSd3SE8Pvw5KQeehYROY604jDj KE1lw5n/Y+dPkr83HnSNAzX9xElRX1E/03YWsJTJJhebLQbvDD6pSayP8UzuhH8UC4W3uK 447U4ZYMFlMRPrpRl/Xy0XbQ0BWpfrFUn1ERV4SM5MbYrWyTKwbNR21UoNar/9wMTahb/H rDtSFz5KayijwPM5kxTQVeFufJB24jc3PnYzvn+3zPFEPFnobkynTEWZIYevzw== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=fastmail.net header.s=fm2 header.b=NadCH+ge; dkim=fail ("headers rsa verify failed") header.d=messagingengine.com header.s=fm2 header.b=j08YMzei; dmarc=fail reason="SPF not aligned (relaxed)" header.from=fastmail.net (policy=none); spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org" X-Migadu-Spam-Score: -2.13 Authentication-Results: aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=fastmail.net header.s=fm2 header.b=NadCH+ge; dkim=fail ("headers rsa verify failed") header.d=messagingengine.com header.s=fm2 header.b=j08YMzei; dmarc=fail reason="SPF not aligned (relaxed)" header.from=fastmail.net (policy=none); spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org" X-Migadu-Queue-Id: 202AE3B3DD X-Spam-Score: -2.13 X-Migadu-Scanner: scn0.migadu.com X-TUID: +LoCr+lpKCCx Hi Bengt and Simon, zimoun writes: > Note that some people are calling for bit-to-bit scientific > reproduction. I am not. Because the meaning of =E2=80=9Csame=E2=80=9D o= r =E2=80=9Cequal=E2=80=9D I am. Not as a goal in itself, because in the larger scientific context it's robust replicability that matters, not bit-for-bit re-execution. And yet, the latter matters for two reasons : - It's verifiable automatically, making it cheap and fast to check. No need to bother an expert for a qualified opinion. - If you hit a case of non-replicability (scientifically relevant differences in two computations that everybody expects to yield equivalent results), then it is nearly impossible to investigate if the individual computations are not bit-for-bit reproducible. Making scientific computations bit-for-bit reproducible is the moral equivalent of keeping a detailed lab notebook: doing your best to tell others exactly what you did. > conclusions hold. Again, transparency and full control of the > variability are fundamental here. How to argue if they are not > satisfied? Exactly, that's very similar to my second point. Or, in Bengt's formulation: > The details of Fortran version or Julia/Clang or guile > pedigree only really come into play for forensics looking > for where the abstract was implemented differently. When the forensics are called in, then... > Thus far, "show me the code" is the usual way to ask someone > what they did, and guix makes is possible to answer in great > detail. ... "show me the code" is not sufficient. You must also be sure that the code you look at is really the code that was run. And that's the role of bit-for-bit reproducibility. Cheers, Konrad.