From: zimoun <zimon.toutoune@gmail.com>
To: "Konrad Hinsen" <konrad.hinsen@fastmail.net>,
"Bengt Richter" <bokr@bokr.com>, "Ludovic Courtès" <ludo@gnu.org>
Cc: Guix Devel <guix-devel@gnu.org>
Subject: Re: Investigating a reproducibility failure
Date: Thu, 17 Feb 2022 12:21:51 +0100 [thread overview]
Message-ID: <86pmnl4u28.fsf@gmail.com> (raw)
In-Reply-To: <m1pmnnvu72.fsf@fastmail.net>
Hi Konrad,
We agree on the main points in the scope of Guix. :-) We probably
disagree on some specific points about epistemology or epistemic
justification; I am not sure to understand enough these terms to put
them here. :-)
We are far from OpenBLAS. :-)
On Wed, 16 Feb 2022 at 14:04, Konrad Hinsen <konrad.hinsen@fastmail.net> wrote:
> Making scientific computations bit-for-bit reproducible is the moral
> equivalent of keeping a detailed lab notebook: doing your best to tell
> others exactly what you did.
A detailed lab notebook implies transparency and full control of
variability, not bit-for-bit reproducibility.
If my detailed lab notebook tracks my experiment to test gravity and
pendulum, as detailed and ideal (moral?) as it would be, i.e., providing
the capacity to build and re-build two exact same benches, then two
experiences would not provide the bit-for-bit numbers in a table
measuring the oscillations. Because, for instance, it would depend on
the two locations, on the touch of the experimenter, etc.
In many fields, the experimental reproduction depends on the variability
of the inputs or of the instruments and therefore the scientific
community, field by field, somehow defines what “same” means, depending
on their common variability from their field.
For one, I do not see why it would be different for the computational
processing part of the experiment. And two, asking bit-for-bit
reproducibility for one part of the experiment is asking far more than
for the others non-computational part of the same experiment.
Because I use daily computers and am deeply interested in what a
computation means, for sure, I advocate for bit-to-bit reproducibility.
But then, I discuss with my colleagues biologist or MD and somehow my
views are biased, i.e, I am trying to apply my own criteria defining
“same” from my “field” to their “field” where the same “same” must be
applied to the all chain, computational processing included. Or at
least they have to define what is acceptable for each part. Do not take
me wrong, such computational part must be transparent where the
variability must also be controlled, but no strictly more or totally
less than the other parts.
> When the forensics are called in, then...
>
>> Thus far, "show me the code" is the usual way to ask someone
>> what they did, and guix makes is possible to answer in great
>> detail.
>
> ... "show me the code" is not sufficient. You must also be sure that the
> code you look at is really the code that was run.
I agree. It is “show me ALL the code” and e.g., “guix graph
python-scipy” points it is a long read. :-) Therefore, being able to
build, run, re-build and re-run are weak requirements to establish
trust.
> And that's the role of
> bit-for-bit reproducibility.
From my understanding, the validation of a reproduction depends on
trust: what is the confidence about this or that? Well, bit-for-bit
reproducibility is one criteria for establishing such trust. However,
IMHO, such criteria is not the unique one, and defeating it can be
compensated by other criteria used by many experimental sciences.
Bah for what my opinion is worth on this topic. :-)
In any cases, thanks Konrad for the materials you provide about this
topic. For the interested French reader: :-)
- https://www.societe-informatique-de-france.fr/wp-content/uploads/2021/11/1024_18_2021_11.html
- https://webcast.in2p3.fr/video/les-enjeux-et-defis-de-la-recherche-reproductible
- https://www.fun-mooc.fr/en/courses/reproducible-research-methodological-principles-transparent-scie/
Cheers,
simon
next prev parent reply other threads:[~2022-02-17 11:39 UTC|newest]
Thread overview: 18+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-02-02 20:35 Investigating a reproducibility failure zimoun
2022-02-02 23:43 ` zimoun
2022-02-03 9:16 ` Konrad Hinsen
2022-02-03 11:41 ` Ricardo Wurmus
2022-02-03 17:05 ` Konrad Hinsen
2022-02-03 12:07 ` zimoun
2022-02-05 14:12 ` Ludovic Courtès
2022-02-15 14:10 ` Bengt Richter
2022-02-16 12:03 ` zimoun
2022-02-16 13:04 ` Konrad Hinsen
2022-02-17 11:21 ` zimoun [this message]
2022-02-17 16:55 ` Konrad Hinsen
-- strict thread matches above, loose matches on Subject: below --
2022-02-01 14:05 Konrad Hinsen
2022-02-01 14:30 ` Konrad Hinsen
2022-02-02 23:19 ` Ricardo Wurmus
2022-02-02 23:36 ` Ricardo Wurmus
2022-02-05 14:05 ` Ludovic Courtès
2022-02-08 5:57 ` Konrad Hinsen
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: https://guix.gnu.org/
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=86pmnl4u28.fsf@gmail.com \
--to=zimon.toutoune@gmail.com \
--cc=bokr@bokr.com \
--cc=guix-devel@gnu.org \
--cc=konrad.hinsen@fastmail.net \
--cc=ludo@gnu.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://git.savannah.gnu.org/cgit/guix.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).