Re: Feedback from JRES in Dijon

[Konrad Hinsen <konrad.hinsen@fastmail.net>]

Hi Julien and Pierre,

> So they are doing physical simulation (fluid dynamics), so they don't
> (can't) get the same result when running the same experiment
> twice. They wart replicability, that is, even if the results are
> different, they are close enough to each other that you have to draw
> the same scientific conclusion, independent of your compiler or other
> package inputs.

That's a common point of view in the numerical simulation community.
What the people defending it don't realize is that both reproducibility
and replicability matter, but in different situations and for different
reasons. Reproducibility matters for verification ("was the computation
done correctly?"), replicability matters for validation ("was the
computation the right one for the scientific question?").

Moreover, there is a practical use for reproducibility when checking for
replicability. Suppose you have a program and a result, then you run the
program with a different compiler and get a different result, too
different to be scientifically equivalent. No replicability. Then what?
How do you figure out what went wrong? The very first thing you want to
check is reproducibility: can you get the same result by using the same
compiler? If yes, fine, you can then look at intermediate results in
both versions of the computation to figure out where the differences
come from. If not, there is no point in wasting time on that: there is
something wrong with the code or the data you got, and you have to check
your sources first.

Giving up on reproducibility thus means giving up a valuable debugging
technique.

Cheers,
  Konrad