From: Konrad Hinsen <konrad.hinsen@fastmail.net>
To: zimoun <zimon.toutoune@gmail.com>,
"Bengt Richter" <bokr@bokr.com>, "Ludovic Courtès" <ludo@gnu.org>
Cc: Guix Devel <guix-devel@gnu.org>
Subject: Re: Investigating a reproducibility failure
Date: Thu, 17 Feb 2022 17:55:45 +0100 [thread overview]
Message-ID: <m1o835v3e6.fsf@fastmail.net> (raw)
In-Reply-To: <86pmnl4u28.fsf@gmail.com>
Hi Simon,
> We are far from OpenBLAS. :-)
That's fine with me. The more distance between me and OpenBLAS, the
happier I am ;-)
> On Wed, 16 Feb 2022 at 14:04, Konrad Hinsen <konrad.hinsen@fastmail.net> wrote:
>
>> Making scientific computations bit-for-bit reproducible is the moral
>> equivalent of keeping a detailed lab notebook: doing your best to tell
>> others exactly what you did.
>
> A detailed lab notebook implies transparency and full control of
> variability, not bit-for-bit reproducibility.
That's why I said "moral" equivalent. Computations are different from
experiments. Typical mistakes are different, and technical possibilities
are different.
1. You can't have the equivalent of bit-for-bit reproducibility with
experiment. You can with computers, and with good tool support
(Guix!) it can become a routine task that takes little
effort. So... why *not* do it?
2. A computation involves many more details than any typical experiment.
Just writing down what you did is *not* enough for documenting a
computation, as experience has shown. So you need more than the
lab notebook. If your computation is bit-for-bit reproducible, you
know that you have documented every last detail. Inversely, if you
cannot reproduce to the bit level, you know that *something* is out
of your control.
In the end, my argument is more pragmatic than philosophical. If
bit-for-bit reproducibility is (1) useful for resolving issues in the
future, and (2) cheap to get with good tool support, then we should
go for it.
The main reason why people argue against it is lack of tool support in
their work environments. They conclude that it's a difficult goal to
achieve, and then start to reason that it's not strictly necessary for
the scientific method. Which is true. But... it's still very useful.
>> And that's the role of bit-for-bit reproducibility.
>
> From my understanding, the validation of a reproduction depends on
> trust: what is the confidence about this or that? Well, bit-for-bit
> reproducibility is one criteria for establishing such trust. However,
> IMHO, such criteria is not the unique one, and defeating it can be
> compensated by other criteria used by many experimental sciences.
Definitely. But in many cases, bit-for-bit reproducibility is the
cheapest way to build trust, given good tool support. In other cases,
e.g. HPC or exotic hardware, it's expensive, and then you look for
something else.
Cheers,
Konrad.
next prev parent reply other threads:[~2022-02-17 16:56 UTC|newest]
Thread overview: 18+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-02-02 20:35 Investigating a reproducibility failure zimoun
2022-02-02 23:43 ` zimoun
2022-02-03 9:16 ` Konrad Hinsen
2022-02-03 11:41 ` Ricardo Wurmus
2022-02-03 17:05 ` Konrad Hinsen
2022-02-03 12:07 ` zimoun
2022-02-05 14:12 ` Ludovic Courtès
2022-02-15 14:10 ` Bengt Richter
2022-02-16 12:03 ` zimoun
2022-02-16 13:04 ` Konrad Hinsen
2022-02-17 11:21 ` zimoun
2022-02-17 16:55 ` Konrad Hinsen [this message]
-- strict thread matches above, loose matches on Subject: below --
2022-02-01 14:05 Konrad Hinsen
2022-02-01 14:30 ` Konrad Hinsen
2022-02-02 23:19 ` Ricardo Wurmus
2022-02-02 23:36 ` Ricardo Wurmus
2022-02-05 14:05 ` Ludovic Courtès
2022-02-08 5:57 ` Konrad Hinsen
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: https://guix.gnu.org/
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=m1o835v3e6.fsf@fastmail.net \
--to=konrad.hinsen@fastmail.net \
--cc=bokr@bokr.com \
--cc=guix-devel@gnu.org \
--cc=ludo@gnu.org \
--cc=zimon.toutoune@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://git.savannah.gnu.org/cgit/guix.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).