From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp11.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id OP88LCXpDGIucAEAgWs5BA (envelope-from ) for ; Wed, 16 Feb 2022 13:08:05 +0100 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp11.migadu.com with LMTPS id 2P+PKSXpDGIHCgEA9RJhRA (envelope-from ) for ; Wed, 16 Feb 2022 13:08:05 +0100 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 5FBDE2F094 for ; Wed, 16 Feb 2022 13:08:05 +0100 (CET) Received: from localhost ([::1]:40002 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nKJ6K-0003bE-DH for larch@yhetil.org; Wed, 16 Feb 2022 07:08:04 -0500 Received: from eggs.gnu.org ([209.51.188.92]:54708) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nKJ62-0003b3-6l for guix-devel@gnu.org; Wed, 16 Feb 2022 07:07:46 -0500 Received: from [2a00:1450:4864:20::42b] (port=46953 helo=mail-wr1-x42b.google.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1nKJ60-00038x-94; Wed, 16 Feb 2022 07:07:45 -0500 Received: by mail-wr1-x42b.google.com with SMTP id q7so3057807wrc.13; Wed, 16 Feb 2022 04:07:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:in-reply-to:references:date:message-id :mime-version:content-transfer-encoding; bh=j5M/ovMAqgZsaoD1WhLkiQgfngXY/12sEbE9j7U2nxI=; b=NmvmpHFV596Fm6k1ovzbPPC9RiAbmXRkxqii4Q9N2wuQ2gVBmiquxI/9wsD7dV3Wri tpEUKY1NJQKGJgzKA+fafBdbFnaltuzpVEWY864Y8751BGP67AnZPJhAEAqstwICevNx ueSBmWmKH+C8gSFaNbTGKh6RmLM/tDrj0HsKtWmlczeE6+rnSsGi/ZT60awaqWf41j5P uvkQvdnmN5Qu9gs/KC4bMoJGQiyZYKMz9bD5fM7K9ct5un60lkwu76YPBISRfgFi/Z1P 99xJuA3n7x3+hagP1pOMQP07/Bp8ddhDeT2bph5uMofS9v7UTGgWyNs3uhY/QyTmPhkN vMDA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:in-reply-to:references:date :message-id:mime-version:content-transfer-encoding; bh=j5M/ovMAqgZsaoD1WhLkiQgfngXY/12sEbE9j7U2nxI=; b=C6kOcRit1KDWII4QfAB9AVAXmzlZVSSz6onZPEpfvzi63NgJVJnrcZzormkMdckgS5 bjtoaEAMbUFG+hF9N1eZaRJLLPh8gBofNTsBNydfyRJAbrVG1WTrLtgu7Wt4qz+QxLzZ tCUtNIrZjBV+CXzz1z3d497vlsWG9vBHCAIcU1Bhzf09j62u4KzgnttnnOeEZwwQ/9kb g7yNUrYT5TlVy95LSCmzVKn8S+rjZCLN30mNbFLnrytCHVV+sjGE+6W7CNI05WBKQIh5 MQj0UglW63WJC47iNVHbBdq00PGaumb6cwrtg75vuY3cX0NiZZstk8fFm9xx4vOW//YA /6dQ== X-Gm-Message-State: AOAM533K0W0pTxxH5EtqXgl2PSNweVb1z3vQt+PRB281Y7MQcpWWx56k ZGJmmPQ26eDgf0TbfX3GeaumgUyoqbc= X-Google-Smtp-Source: ABdhPJyCZ8vNIftl3SUJttwNTbd3qKh4Yy89gj9wRpJINMNp0kjyEpfR8uVkKAzMzVloV1ojhEc0IA== X-Received: by 2002:a05:6000:1248:b0:1e4:a4bf:d92c with SMTP id j8-20020a056000124800b001e4a4bfd92cmr2097775wrx.260.1645013257607; Wed, 16 Feb 2022 04:07:37 -0800 (PST) Received: from lili ([2a01:e0a:59b:9120:65d2:2476:f637:db1e]) by smtp.gmail.com with ESMTPSA id x10sm17484086wmj.17.2022.02.16.04.07.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 16 Feb 2022 04:07:37 -0800 (PST) From: zimoun To: Bengt Richter , Ludovic =?utf-8?Q?Court=C3=A8s?= Subject: Re: Investigating a reproducibility failure In-Reply-To: <20220215141031.GA13837@LionPure> References: <871r0l9fd1.fsf@gmail.com> <87tuddh06r.fsf@gnu.org> <20220215141031.GA13837@LionPure> Date: Wed, 16 Feb 2022 13:03:57 +0100 Message-ID: <86bkz7587m.fsf@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Host-Lookup-Failed: Reverse DNS lookup failed for 2a00:1450:4864:20::42b (failed) Received-SPF: pass client-ip=2a00:1450:4864:20::42b; envelope-from=zimon.toutoune@gmail.com; helo=mail-wr1-x42b.google.com X-Spam_score_int: -3 X-Spam_score: -0.4 X-Spam_bar: / X-Spam_report: (-0.4 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, PDS_HP_HELO_NORDNS=0.904, RCVD_IN_DNSWL_NONE=-0.0001, RDNS_NONE=0.793, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Guix Devel Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: "Guix-devel" X-Migadu-Flow: FLOW_IN X-Migadu-Country: US ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1645013285; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post:dkim-signature; bh=j5M/ovMAqgZsaoD1WhLkiQgfngXY/12sEbE9j7U2nxI=; b=I/LoM64yrQ88HYrca4neep2koKLf2UKbPANjmDuxUXQhdoYC+4a2P7bKyK50fqg1sfFv2m zF6VkWCY3QcAYOVX5HvXex2HsMY3aVBPAtGDt9V9dz40XoC/2kYA0I0TA/Q9lySAQtyoz0 P6TVspjbdTmibFq5qCsVjv0SZKFjtz9yH+hNxKH2yRQ2OkqBR/+Ln0v6BfS35EpzJYlWsJ vgwNg7JeF+rFLZS05QJRJDfIoS3sdeebN0D7xtXwdQuyHforSQAGF/PpzxdAaGxdiYVRz8 slFOSoCofaCRLJG0Lws66RzIbebJl77kQkRFC7zjjYQd5Uny0SpmPP7nl4MhpA== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1645013285; a=rsa-sha256; cv=none; b=rv5mOZ/RgjWf8QyF7MFZMURz4JXm0FgwJqtLtHFTbncb0YyywG/eRWaPzALXgtYVtg9LZ8 5OaLOLJMKNGW1pl/V/eV4QEdXV75y3lARaafn44DcDULVdw6N0F7Q75UbDT6ZtBDdSgW+b j+0jYt9yJ3XQThLFWrAveQCqP9puDpSyIXGTTU2wpIE+IGsp0plk/G2+oNQQfAoAScMC0+ Psz6Ypzg9vdmdQcrtbpo5jsM7KI8ZcWTofwKjRoDMdeVm7ErtVAaCR3x3quQC0NZhZWUn/ IE12JcvruO2oLxKIvIO6p5f0SAnUcyFa/THUNfu0dzcGebV8GE860rp2sNlGiA== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=gmail.com header.s=20210112 header.b=NmvmpHFV; dmarc=fail reason="SPF not aligned (relaxed)" header.from=gmail.com (policy=none); spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org" X-Migadu-Spam-Score: -2.03 Authentication-Results: aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=gmail.com header.s=20210112 header.b=NmvmpHFV; dmarc=fail reason="SPF not aligned (relaxed)" header.from=gmail.com (policy=none); spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org" X-Migadu-Queue-Id: 5FBDE2F094 X-Spam-Score: -2.03 X-Migadu-Scanner: scn1.migadu.com X-TUID: ZcSIkPdr2FBs Hi, On Tue, 15 Feb 2022 at 15:10, Bengt Richter wrote: > I suspect what you really want to reproduce is not verbatim > code, but the abstract computation that it implements, > typically a digitally simulated experiment? [...] > Maybe the above pi computation could be a start on some kind > of abstract model validation test? It's simple, but it pulls > on a lot of simulation tool chains. WDYT? Well, it depends on the community which term they pick for which concept: - same team, same experimental setup - different team, same experimental setup - different team, different experimental setup and the terms are repeat, replicate, reproduce. For details, see [1]. Since Konrad is editor for the ReScience journal, I guess =E2=80=99reproduc= e=E2=80=99 means [2]: Reproduction of a computational study means running the same computation on the same input data, and then checking if the results are the same, or at least =E2=80=9Cclose enough=E2=80=9D wh= en it comes to numerical approximations. Reproduction can be considered as software testing at the level of a complete study. Where my understanding of your =E2=80=9Cabstract computation=E2=80=9D looks= more as [2]: Replication of a scientific study (computational or other) means repeating a published protocol, respecting its spirit and intentions but varying the technical details. For computational work, this would mean using different software, running a simulation from different initial conditions, etc. The idea is to change something that everyone believes shouldn=E2=80=99t matter= , and see if the scientific conclusions are affected or not. Therefore, again from my understanding, you are somehow proposing what science should be. :-) It is what the initiative GuixHPC [3] is trying to tackle. Transparency and full control of the variability=E2=80=93=E2=80=93the roots= of the scientific method=E2=80=93=E2=80=93allow to achieve, with more or less succ= ess, =E2=80=99reproduction=E2=80=99. Here and today, Guix plays a central role = for reproducing because Guix does not cheat with transparency and full control of variability. Note that some people are calling for bit-to-bit scientific reproduction. I am not. Because the meaning of =E2=80=9Csame=E2=80=9D or = =E2=80=9Cequal=E2=80=9D depends on the scientific fields. However, it is up to any scientific debate or controversy to draw the line for =E2=80=9Csame=E2=80=9D and argue= if the conclusions hold. Again, transparency and full control of the variability are fundamental here. How to argue if they are not satisfied? Then, and out of Guix scope, if the reproduced result matters enough, people can try to replicate, for confirmation, for performance improvements, or as a step targeting another results. This replication can use Guix to control the variability and also help the reproduction of the replication; but Guix does not take a central role here. Last, it is in this second and other steps that the =E2=80=9Cabstract model= =E2=80=9D could play role, and it is out of Guix scope, IMHO. 1: 2: 3: Cheers, simon