Rproducibility for Python and beyond

Rproducibility for Python and beyond

[Simon TOURNIER]

Hi Konrad, all,

French speakers, here is an interesting presentation by Konrad about the state of Python for scientific computing and reproducibility.

https://reproducibility.gricad-pages.univ-grenoble-alpes.fr/web/presentation_110423.html#presentation_110423

Without watching the video, here the questions I would like to discuss. :-) 

1. Considering the Konrad's schema of some scientific computation (Model --technical choices--> Code --computational env--> Results), there are also technical choices about the computational environment, but they are implicit.  And often impossible to scrutinize because of the lack of transparency.  The key, IMHO, is not the determinism of the computation, instead the key is its transparency.  Determinism is one mean to obtain transparency and determinism is not the only mean.  For instance, this determinism is not affordable for very intensive computation, where is not doable to repeat.  How to think about determinism considering statistical training of machine learning models?  Other said, for some cases, the "compilation" (Code -> Results) of the scientific model is too costly.

2. The "redo" of computations is only possible when the citation is correct.  L'Inria is somehow proposing <https://hal.science/hal-02135891> with the BibLaTeX style <https://mirrors.ircam.fr/pub/CTAN/macros/latex/contrib/biblatex-contrib/biblatex-software/software-biblatex.pdf>.  However, this only captures, at best, some technical choices when implementing the model.  And this does not capture at all the complete computational environment.  What are your ideas for tackling this issue about the citation?

For instance, the file "guix describe -f channels" is one mean for capturing (and cite too!) one computational environment.  Do we need to make it more popular?  How to link this mean with the archiving part of source code (relying on SWH, say)?


Cheers,
simon

Re: Rproducibility for Python and beyond

[Konrad Hinsen]

Hi Simon,

> French speakers, here is an interesting presentation by Konrad about
> the state of Python for scientific computing and reproducibility.
>
> https://reproducibility.gricad-pages.univ-grenoble-alpes.fr/web/presentation_110423.html#presentation_110423
>
> Without watching the video, here the questions I would like to discuss. :-) 

Summary: Why you should use Guix rather than Conda to manage your Python
environments.

Now I'll jump to the end:

> For instance, the file "guix describe -f channels" is one mean for
> capturing (and cite too!) one computational environment.  Do we need
> to make it more popular?  How to link this mean with the archiving
> part of source code (relying on SWH, say)?

Yes, we should make this more popular. With Guix, a full description of
a computational environment is:

 - hardware architecture
 - channel file
 - manifest file

Leaving out the Linux kernel and file system, which should in principle
be listed but in practice never cause any problems.

It would be nice to have tools that automatically extract a list of
citations from such a description. That is not as easy as it seems
because the list should not really be exhaustive if it is meant to be
listed in a paper for human consumption.

> 1. Considering the Konrad's schema of some scientific computation
> (Model --technical choices--> Code --computational env--> Results),
> there are also technical choices about the computational environment,
> but they are implicit.  And often impossible to scrutinize because of

Indeed. Most people are happy to leave "the environment" as a black box,
which I think is fine as long as it is (1) archivable and (2)
transparent for those who are willing to open the box.

> the lack of transparency.  The key, IMHO, is not the determinism of
> the computation, instead the key is its transparency.  Determinism is
> one mean to obtain transparency and determinism is not the only mean.

Agreed as well. The reason I tend to speak about determinism is to
illustrate why we shouldn't consider irreproducibility normal but
surprising.

> For instance, this determinism is not affordable for very intensive
> computation, where is not doable to repeat.  How to think about

True, but a niche topic. Most computational science is not HPC, and yet
suffers from reproducibility issues.

> 2. The "redo" of computations is only possible when the citation is
> correct.  L'Inria is somehow proposing

Correct and complete.

Cheers,
  Konrad.
-- 
---------------------------------------------------------------------
Konrad Hinsen
Centre de Biophysique Moléculaire, CNRS Orléans
Synchrotron Soleil - Division Expériences
Saint Aubin - BP 48
91192 Gif sur Yvette Cedex, France
Tel. +33-1 69 35 97 15
E-Mail: konrad DOT hinsen AT cnrs DOT fr
http://dirac.cnrs-orleans.fr/~hinsen/
ORCID: https://orcid.org/0000-0003-0330-9428
Mastodon: @khinsen@scholar.social
---------------------------------------------------------------------