From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp11.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id gI/JOgc0DmLteAAAgWs5BA (envelope-from ) for ; Thu, 17 Feb 2022 12:39:51 +0100 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp11.migadu.com with LMTPS id wFA+OAc0DmJMWgEA9RJhRA (envelope-from ) for ; Thu, 17 Feb 2022 12:39:51 +0100 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 85AC8161BD for ; Thu, 17 Feb 2022 12:39:51 +0100 (CET) Received: from localhost ([::1]:37712 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nKf8X-0003GR-Rd for larch@yhetil.org; Thu, 17 Feb 2022 06:39:49 -0500 Received: from eggs.gnu.org ([209.51.188.92]:33792) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nKf0H-0005Hi-17 for guix-devel@gnu.org; Thu, 17 Feb 2022 06:31:17 -0500 Received: from [2a00:1450:4864:20::32a] (port=52830 helo=mail-wm1-x32a.google.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1nKf0F-0005C6-5T; Thu, 17 Feb 2022 06:31:16 -0500 Received: by mail-wm1-x32a.google.com with SMTP id w13so2484029wmi.2; Thu, 17 Feb 2022 03:31:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:in-reply-to:references:date:message-id :mime-version:content-transfer-encoding; bh=sxwh1eDq2LTlMiYzig+WMqXhQIjMsBRymk70zkDu0Mk=; b=jGRkO+gOv2b7pOxIBjzvYO7t0r3Lug3sSGwfljmdp10Tl44jhFoHjpHwpr52T3dkfi xcDG0Lfa3QS+5bfu5Sht9MI12O243ShpqOKDrt9R5Q82+vmZxla+dOjcKuPDZhOgElzc G8fh3evTHly5y8AQ6rTTrwEZcl7FjRge7m/NVlXDYlf8vr5/cwFYmehzVl+EY4tuf/QP ctEFYL9uogvFDzF1WQplvTVxbPwqm356vpAGDcAbL+8427ByCVFC7VS8jYY6lbp8p28E jyvCZGP/wabOqLPqjlnHtP0Z623702Z9PkUSBtO+eewcFaJL90vxw8E8P/J02nnRyht3 V9kw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:in-reply-to:references:date :message-id:mime-version:content-transfer-encoding; bh=sxwh1eDq2LTlMiYzig+WMqXhQIjMsBRymk70zkDu0Mk=; b=nu9DirTwoK787Kj+J/zTrZ1Do1/1aK/j3kWaMZ6kX+5w06a/IQVZmRPmU+1LGUCNvW N0pxKM7jQqEdm36Kat5LXKU7hYuSYSrTG+tBeyy0D2UteRVltwUmfICCf6waKnlv6L2n TW0WHKPPCZJivDbXZSuV+bZduglGLKbvJXBj2MIP1Kl+0cYLTReAwBrwa39CFVVy2xwZ VBzECy0oZ9oO7+V3pD3EIkAm7Jyj0EG82/p6wu1Amgp+PGF1lWSsq3yGkHYnhFBJ6/1F O49Ljb8OXP0HFeJw15grm+YRmMKE//YY17JoF5KJMpsPpxI3DmN4rN7E8aiLupD0JVyA 189A== X-Gm-Message-State: AOAM531TMFqlzFmDcfoepoJoOgReAeXzmniIKPnhFqUdKKEvht9+UaLz Qy9tDMLHnmNZVqjGKtavksW1JuKUvdI= X-Google-Smtp-Source: ABdhPJzZGImvoqKlc2pD44B5Z89qHhXTA61wbGHL6b4MmMNhkPRlM4Scx0njKfgWysfn87ANtvKKwA== X-Received: by 2002:a1c:7318:0:b0:37b:fa1b:1472 with SMTP id d24-20020a1c7318000000b0037bfa1b1472mr5704485wmb.57.1645097472904; Thu, 17 Feb 2022 03:31:12 -0800 (PST) Received: from lili ([2a01:e0a:59b:9120:65d2:2476:f637:db1e]) by smtp.gmail.com with ESMTPSA id n6sm20611751wru.114.2022.02.17.03.31.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 17 Feb 2022 03:31:12 -0800 (PST) From: zimoun To: Konrad Hinsen , Bengt Richter , Ludovic =?utf-8?Q?Court=C3=A8s?= Subject: Re: Investigating a reproducibility failure In-Reply-To: References: <871r0l9fd1.fsf@gmail.com> <87tuddh06r.fsf@gnu.org> <20220215141031.GA13837@LionPure> <86bkz7587m.fsf@gmail.com> Date: Thu, 17 Feb 2022 12:21:51 +0100 Message-ID: <86pmnl4u28.fsf@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Host-Lookup-Failed: Reverse DNS lookup failed for 2a00:1450:4864:20::32a (failed) Received-SPF: pass client-ip=2a00:1450:4864:20::32a; envelope-from=zimon.toutoune@gmail.com; helo=mail-wm1-x32a.google.com X-Spam_score_int: -2 X-Spam_score: -0.3 X-Spam_bar: / X-Spam_report: (-0.3 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, PDS_HP_HELO_NORDNS=0.978, RCVD_IN_DNSWL_NONE=-0.0001, RDNS_NONE=0.793, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Guix Devel Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: "Guix-devel" X-Migadu-Flow: FLOW_IN X-Migadu-Country: US ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1645097991; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post:dkim-signature; bh=sxwh1eDq2LTlMiYzig+WMqXhQIjMsBRymk70zkDu0Mk=; b=TGLX/KXRvBRXx81FydCqWKuqZk5Pv0UJ20SQu5joK84KMCqrc22H4uMV+Jo4ZdGxAHhtIg jWMG426OQsSRYkzsMksEL9VtG6BAJbyo33uzgT4l0ktVgfeCEknXdkVL2nWdUrfF59wosh hb8QcuISfkscgXJlMqVgA8i3881lUgcJOvTHNQWZHGAdpMbVHBra621N57OeODRAoKHmFl qHLOz/Y/4JkpVrH2ahB0Un/7+AD98JwdIQpB+JFnc2xqI6K6IRWFKBK15u66gmfX0R8XNV YYHChaeD4ydFzwesDDgtVsFnfMtt3KcHG6Ve/lSSZ6YdNvqLOL++1Qu+voo1Uw== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1645097991; a=rsa-sha256; cv=none; b=d/pmvZGMAvpT3tplkGrkqjxsueocTCFjNJ0mwFDWHfVJqt3mXVvpjla5hghq6eT+fLqLzb RWvMIQ8ad8neC8sBO9sUtbfwGUwam6ST/HqwtQ5uv7ipA2HCT2BQ3+AdKq9IWiN9nNVx5W pPSGdKMHxnJ1wgGAjX9YFIjiAQ6MA5hWB160+CevB0qwgqR9ipUdHVBu4VU2ZCrsjg1euo FxoAaDZY+ccKVR7FkIVRVXkf6SKgRRxBb5EjR6lONUOoC6i4mTmSD9hReXk/CYqWGUdoe/ +1ChGoYbpTkM3ABcQe97RLuw6gLZiXSUxNu56ab1Cb6zIvJyzMCi2JSWxbl+3w== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=jGRkO+gO; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org" X-Migadu-Spam-Score: -4.33 Authentication-Results: aspmx1.migadu.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=jGRkO+gO; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org" X-Migadu-Queue-Id: 85AC8161BD X-Spam-Score: -4.33 X-Migadu-Scanner: scn1.migadu.com X-TUID: dKNuwciq66QI Hi Konrad, We agree on the main points in the scope of Guix. :-) We probably disagree on some specific points about epistemology or epistemic justification; I am not sure to understand enough these terms to put them here. :-) We are far from OpenBLAS. :-) On Wed, 16 Feb 2022 at 14:04, Konrad Hinsen wr= ote: > Making scientific computations bit-for-bit reproducible is the moral > equivalent of keeping a detailed lab notebook: doing your best to tell > others exactly what you did. A detailed lab notebook implies transparency and full control of variability, not bit-for-bit reproducibility. If my detailed lab notebook tracks my experiment to test gravity and pendulum, as detailed and ideal (moral?) as it would be, i.e., providing the capacity to build and re-build two exact same benches, then two experiences would not provide the bit-for-bit numbers in a table measuring the oscillations. Because, for instance, it would depend on the two locations, on the touch of the experimenter, etc. In many fields, the experimental reproduction depends on the variability of the inputs or of the instruments and therefore the scientific community, field by field, somehow defines what =E2=80=9Csame=E2=80=9D mean= s, depending on their common variability from their field. For one, I do not see why it would be different for the computational processing part of the experiment. And two, asking bit-for-bit reproducibility for one part of the experiment is asking far more than for the others non-computational part of the same experiment. Because I use daily computers and am deeply interested in what a computation means, for sure, I advocate for bit-to-bit reproducibility. But then, I discuss with my colleagues biologist or MD and somehow my views are biased, i.e, I am trying to apply my own criteria defining =E2=80=9Csame=E2=80=9D from my =E2=80=9Cfield=E2=80=9D to their =E2=80=9Cfi= eld=E2=80=9D where the same =E2=80=9Csame=E2=80=9D must be applied to the all chain, computational processing included. Or at least they have to define what is acceptable for each part. Do not take me wrong, such computational part must be transparent where the variability must also be controlled, but no strictly more or totally less than the other parts. > When the forensics are called in, then... > >> Thus far, "show me the code" is the usual way to ask someone >> what they did, and guix makes is possible to answer in great >> detail. > > ... "show me the code" is not sufficient. You must also be sure that the > code you look at is really the code that was run. I agree. It is =E2=80=9Cshow me ALL the code=E2=80=9D and e.g., =E2=80=9Cg= uix graph python-scipy=E2=80=9D points it is a long read. :-) Therefore, being able to build, run, re-build and re-run are weak requirements to establish trust. > And that's the role of > bit-for-bit reproducibility. >From my understanding, the validation of a reproduction depends on trust: what is the confidence about this or that? Well, bit-for-bit reproducibility is one criteria for establishing such trust. However, IMHO, such criteria is not the unique one, and defeating it can be compensated by other criteria used by many experimental sciences. Bah for what my opinion is worth on this topic. :-) In any cases, thanks Konrad for the materials you provide about this topic. For the interested French reader: :-) - https://www.societe-informatique-de-france.fr/wp-content/uploads/2021/11= /1024_18_2021_11.html - https://webcast.in2p3.fr/video/les-enjeux-et-defis-de-la-recherche-repro= ductible - https://www.fun-mooc.fr/en/courses/reproducible-research-methodological-= principles-transparent-scie/ Cheers, simon