From: zimoun <zimon.toutoune@gmail.com>
To: "Bengt Richter" <bokr@bokr.com>, "Ludovic Courtès" <ludo@gnu.org>
Cc: Guix Devel <guix-devel@gnu.org>
Subject: Re: Investigating a reproducibility failure
Date: Wed, 16 Feb 2022 13:03:57 +0100 [thread overview]
Message-ID: <86bkz7587m.fsf@gmail.com> (raw)
In-Reply-To: <20220215141031.GA13837@LionPure>
Hi,
On Tue, 15 Feb 2022 at 15:10, Bengt Richter <bokr@bokr.com> wrote:
> I suspect what you really want to reproduce is not verbatim
> code, but the abstract computation that it implements,
> typically a digitally simulated experiment?
[...]
> Maybe the above pi computation could be a start on some kind
> of abstract model validation test? It's simple, but it pulls
> on a lot of simulation tool chains. WDYT?
Well, it depends on the community which term they pick for which
concept:
- same team, same experimental setup
- different team, same experimental setup
- different team, different experimental setup
and the terms are repeat, replicate, reproduce. For details, see [1].
Since Konrad is editor for the ReScience journal, I guess ’reproduce’
means [2]:
Reproduction of a computational study means running the same
computation on the same input data, and then checking if the
results are the same, or at least “close enough” when it comes
to numerical approximations. Reproduction can be considered as
software testing at the level of a complete study.
Where my understanding of your “abstract computation” looks more as [2]:
Replication of a scientific study (computational or other) means
repeating a published protocol, respecting its spirit and
intentions but varying the technical details. For computational
work, this would mean using different software, running a
simulation from different initial conditions, etc. The idea is
to change something that everyone believes shouldn’t matter, and
see if the scientific conclusions are affected or not.
Therefore, again from my understanding, you are somehow proposing what
science should be. :-) It is what the initiative GuixHPC [3] is trying
to tackle.
Transparency and full control of the variability––the roots of the
scientific method––allow to achieve, with more or less success,
’reproduction’. Here and today, Guix plays a central role for
reproducing because Guix does not cheat with transparency and full
control of variability.
Note that some people are calling for bit-to-bit scientific
reproduction. I am not. Because the meaning of “same” or “equal”
depends on the scientific fields. However, it is up to any scientific
debate or controversy to draw the line for “same” and argue if the
conclusions hold. Again, transparency and full control of the
variability are fundamental here. How to argue if they are not
satisfied?
Then, and out of Guix scope, if the reproduced result matters enough,
people can try to replicate, for confirmation, for performance
improvements, or as a step targeting another results. This replication
can use Guix to control the variability and also help the reproduction
of the replication; but Guix does not take a central role here.
Last, it is in this second and other steps that the “abstract model”
could play role, and it is out of Guix scope, IMHO.
1: <https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5778115/>
2: <http://rescience.github.io/faq/>
3: <https://hpc.guix.info/>
Cheers,
simon
next prev parent reply other threads:[~2022-02-16 12:08 UTC|newest]
Thread overview: 18+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-02-02 20:35 Investigating a reproducibility failure zimoun
2022-02-02 23:43 ` zimoun
2022-02-03 9:16 ` Konrad Hinsen
2022-02-03 11:41 ` Ricardo Wurmus
2022-02-03 17:05 ` Konrad Hinsen
2022-02-03 12:07 ` zimoun
2022-02-05 14:12 ` Ludovic Courtès
2022-02-15 14:10 ` Bengt Richter
2022-02-16 12:03 ` zimoun [this message]
2022-02-16 13:04 ` Konrad Hinsen
2022-02-17 11:21 ` zimoun
2022-02-17 16:55 ` Konrad Hinsen
-- strict thread matches above, loose matches on Subject: below --
2022-02-01 14:05 Konrad Hinsen
2022-02-01 14:30 ` Konrad Hinsen
2022-02-02 23:19 ` Ricardo Wurmus
2022-02-02 23:36 ` Ricardo Wurmus
2022-02-05 14:05 ` Ludovic Courtès
2022-02-08 5:57 ` Konrad Hinsen
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: https://guix.gnu.org/
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=86bkz7587m.fsf@gmail.com \
--to=zimon.toutoune@gmail.com \
--cc=bokr@bokr.com \
--cc=guix-devel@gnu.org \
--cc=ludo@gnu.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://git.savannah.gnu.org/cgit/guix.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).