From mboxrd@z Thu Jan 1 00:00:00 1970 From: Konrad Hinsen Subject: Re: Feedback from JRES in Dijon Date: Thu, 05 Dec 2019 16:44:52 +0100 Message-ID: References: <8D474474-AF4C-4B03-9D38-3BB089BEE4EB@lepiller.eu> <87tv6ec048.fsf@ambrevar.xyz> <14A62244-3626-4146-B40E-BC5CED4B78D3@lepiller.eu> Mime-Version: 1.0 Content-Type: text/plain Return-path: Received: from eggs.gnu.org ([2001:470:142:3::10]:49865) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ictJR-0001R0-DC for guix-devel@gnu.org; Thu, 05 Dec 2019 10:45:07 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1ictJP-0006ft-9X for guix-devel@gnu.org; Thu, 05 Dec 2019 10:45:04 -0500 Received: from out5-smtp.messagingengine.com ([66.111.4.29]:49889) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1ictJN-0006c6-9s for guix-devel@gnu.org; Thu, 05 Dec 2019 10:45:01 -0500 In-Reply-To: <14A62244-3626-4146-B40E-BC5CED4B78D3@lepiller.eu> List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+gcggd-guix-devel=m.gmane.org@gnu.org Sender: "Guix-devel" To: Julien Lepiller , Pierre Neidhardt , guix-devel@gnu.org Hi Julien and Pierre, > So they are doing physical simulation (fluid dynamics), so they don't > (can't) get the same result when running the same experiment > twice. They wart replicability, that is, even if the results are > different, they are close enough to each other that you have to draw > the same scientific conclusion, independent of your compiler or other > package inputs. That's a common point of view in the numerical simulation community. What the people defending it don't realize is that both reproducibility and replicability matter, but in different situations and for different reasons. Reproducibility matters for verification ("was the computation done correctly?"), replicability matters for validation ("was the computation the right one for the scientific question?"). Moreover, there is a practical use for reproducibility when checking for replicability. Suppose you have a program and a result, then you run the program with a different compiler and get a different result, too different to be scientifically equivalent. No replicability. Then what? How do you figure out what went wrong? The very first thing you want to check is reproducibility: can you get the same result by using the same compiler? If yes, fine, you can then look at intermediate results in both versions of the computation to figure out where the differences come from. If not, there is no point in wasting time on that: there is something wrong with the code or the data you got, and you have to check your sources first. Giving up on reproducibility thus means giving up a valuable debugging technique. Cheers, Konrad