From: Konrad Hinsen <konrad.hinsen@fastmail.net>
To: zimoun <zimon.toutoune@gmail.com>,
"Bengt Richter" <bokr@bokr.com>, "Ludovic Courtès" <ludo@gnu.org>
Cc: Guix Devel <guix-devel@gnu.org>
Subject: Re: Investigating a reproducibility failure
Date: Wed, 16 Feb 2022 14:04:33 +0100 [thread overview]
Message-ID: <m1pmnnvu72.fsf@fastmail.net> (raw)
In-Reply-To: <86bkz7587m.fsf@gmail.com>
Hi Bengt and Simon,
zimoun <zimon.toutoune@gmail.com> writes:
> Note that some people are calling for bit-to-bit scientific
> reproduction. I am not. Because the meaning of “same” or “equal”
I am. Not as a goal in itself, because in the larger scientific context
it's robust replicability that matters, not bit-for-bit re-execution.
And yet, the latter matters for two reasons :
- It's verifiable automatically, making it cheap and fast to check.
No need to bother an expert for a qualified opinion.
- If you hit a case of non-replicability (scientifically relevant
differences in two computations that everybody expects to yield
equivalent results), then it is nearly impossible to investigate
if the individual computations are not bit-for-bit reproducible.
Making scientific computations bit-for-bit reproducible is the moral
equivalent of keeping a detailed lab notebook: doing your best to tell
others exactly what you did.
> conclusions hold. Again, transparency and full control of the
> variability are fundamental here. How to argue if they are not
> satisfied?
Exactly, that's very similar to my second point.
Or, in Bengt's formulation:
> The details of Fortran version or Julia/Clang or guile
> pedigree only really come into play for forensics looking
> for where the abstract was implemented differently.
When the forensics are called in, then...
> Thus far, "show me the code" is the usual way to ask someone
> what they did, and guix makes is possible to answer in great
> detail.
... "show me the code" is not sufficient. You must also be sure that the
code you look at is really the code that was run. And that's the role of
bit-for-bit reproducibility.
Cheers,
Konrad.
next prev parent reply other threads:[~2022-02-16 13:05 UTC|newest]
Thread overview: 18+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-02-02 20:35 Investigating a reproducibility failure zimoun
2022-02-02 23:43 ` zimoun
2022-02-03 9:16 ` Konrad Hinsen
2022-02-03 11:41 ` Ricardo Wurmus
2022-02-03 17:05 ` Konrad Hinsen
2022-02-03 12:07 ` zimoun
2022-02-05 14:12 ` Ludovic Courtès
2022-02-15 14:10 ` Bengt Richter
2022-02-16 12:03 ` zimoun
2022-02-16 13:04 ` Konrad Hinsen [this message]
2022-02-17 11:21 ` zimoun
2022-02-17 16:55 ` Konrad Hinsen
-- strict thread matches above, loose matches on Subject: below --
2022-02-01 14:05 Konrad Hinsen
2022-02-01 14:30 ` Konrad Hinsen
2022-02-02 23:19 ` Ricardo Wurmus
2022-02-02 23:36 ` Ricardo Wurmus
2022-02-05 14:05 ` Ludovic Courtès
2022-02-08 5:57 ` Konrad Hinsen
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: https://guix.gnu.org/
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=m1pmnnvu72.fsf@fastmail.net \
--to=konrad.hinsen@fastmail.net \
--cc=bokr@bokr.com \
--cc=guix-devel@gnu.org \
--cc=ludo@gnu.org \
--cc=zimon.toutoune@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://git.savannah.gnu.org/cgit/guix.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).