Re: Feedback from JRES in Dijon

[Bengt Richter <bokr@bokr.com>]

Hi zimoun,

On +2019-12-05 16:52:47 +0100, zimoun wrote:
> Hi,
> 
> On Thu, 5 Dec 2019 at 16:45, Konrad Hinsen <konrad.hinsen@fastmail.net> wrote:
> 
> > > So they are doing physical simulation (fluid dynamics), so they don't
> > > (can't) get the same result when running the same experiment
> > > twice. They wart replicability, that is, even if the results are
> > > different, they are close enough to each other that you have to draw
> > > the same scientific conclusion, independent of your compiler or other
> > > package inputs.
> >
> > That's a common point of view in the numerical simulation community.
> > What the people defending it don't realize is that both reproducibility
> > and replicability matter, but in different situations and for different
> > reasons. Reproducibility matters for verification ("was the computation
> > done correctly?"), replicability matters for validation ("was the
> > computation the right one for the scientific question?").
> 
> If I might, one the best presentation [1] -- that I am aware of -- on
> this. Sorry in French.
> 
> [1] https://webcast.in2p3.fr/video/les-enjeux-et-defis-de-la-recherche-reproductible
> https://aramis.resinfo.org/wiki/lib/exe/fetch.php?media=pleniaires:aramis_keynote_enjeux-et-defis-recherche-reproductible_konrad_hinsen.pdf
>

Is [1] available as a libre video download?
My current youtube-dl could not find the video of [1], though I watched it with firefox.
I think it would be good for my French, such as it is, to watch that a few times ;-)

IMO if you can't reproduce bit-identical results, you should find out _exactly_ why.
And if you do get exactly the same result, you should also know exactly why ;-)

IME, in the process of figuring out the explanations you will always learn something.

You may be reproducing perfectly a biased result which could be improved as
a numerical result by counter-intuitively adding noise to the inputs
(to spread roundoff across the lsb roundoff boundary so that a mean will show
where in the interval the best estimate fell).

And if you are on the hairy edge of having a non-singular matrix to invert,
results can change because your thread got interrupted at a point where fp regs
got stored as 64 bits and thus restored with different precision than if the
interrupt didn't happen. Just because someone logged in remotely for unrelated reasons.

Such an infinitesimal value change can change the search order for pivot point in a matrix
and change the order of computation, or even make zero elements that otherwise differed
enough to make the inversion possible. And maybe you shouldn't actually store an inverse
from a library routine and then multiply, but rewrite your algorith to avoid truncations-
Of course your results may still be unusably sensitive ;-)

OTOH, you may notice that your algorithm would do much better if you pre-translated your
inputs to a better coordinate system orgin (e.g. to make lots of off-diagonal exact zeroes).

You may want to analyze how roundoff errors propagate through a Kalman filter vs a direct MLE version,
to see at what bit region in your numerical results input effects turn to computational artifacts,
to judge how to express your idea of the scientific significance of your results.

The video touches on IEEE 754 (and I do believe the lecturer understands these issues, but
he says (IIU.fr :) no programming language gives control over the FPU -- is that true?? I mean,
including on-the-metal x86_64 assembly language?? I thought you could set all those bits
that the firmware pays attention to, no?)

BTW, if you are interested in floating point, links from here should be fun:

--8<---------------cut here---------------start------------->8---
https://en.wikipedia.org/wiki/William_Kahan
--8<---------------cut here---------------end--------------->8---

Hey, in the '60s we wrote floating point in assembler, (after having written the assembler
for the custom system :) and were happy to get division timed under a millisecond ;-) Time flies :)

> Maybe we could convert it to an entry for the HPC blog. What do you think?
> 
> 
> Cheers,
> simon
> 

-- 
Regards,
Bengt Richter