“Reproducible research articles, from source code to PDF”

“Reproducible research articles, from source code to PDF”

[Ludovic Courtès]

Hello!

This new post introduces the work I did to have a fully reproducible
replication (!) of a 13-year old article, using Guix to express the
whole pipeline:

  https://hpc.guix.info/blog/2020/06/reproducible-research-articles-from-source-code-to-pdf/

Feedback welcome!

Ludo’.

Re: “Reproducible research articles, from source code to PDF”

[zimoun]

Hi Ludo,

On Tue, 16 Jun 2020 at 14:25, Ludovic Courtès <ludovic.courtes@inria.fr> wrote:
> Hello!
>
> This new post introduces the work I did to have a fully reproducible
> replication (!) of a 13-year old article, using Guix to express the
> whole pipeline:
>
>   https://hpc.guix.info/blog/2020/06/reproducible-research-articles-from-source-code-to-pdf/

Really cool!
IMHO, it even deserves an entry to guix.gnu.org/blog. :-)

Aside, I have learnt the interesting project maneage.org.

As always, it gives me some food for thought.

For example, they are future bridges to think: connect the Guix archive
somehow with zenodo DOI and/or Software Heritage identifier.

When I read this comment in the review [1]:

        As  a final  note,  I  wonder if,  and  how  much, the  author's
        approach to reproducible computation/automated report generation
        is  feasible  for  the  average scientist,  in  particular  when
        compared to tools with a smoother learning curve, such as Docker
        containers,  Jupyter notebooks,  R  Markdown  documents and  the
        like. A brief  analysis of this topic with  a clear presentation
        of the  advantages of  the author's  approach in  the Discussion
        session would be worthwhile.

and then the Konrad's answer [2], I asked myself what pieces are
missing.  And what could be the articulation of "guix pack -f docker",
Guix-Jupyter or other notebooks (RMarkdown, Org)?  And what could be a
practical workflow? (Keeping in mind that the average scientist is not a
Linux guru but often run MacOS or Windows.)

1: https://github.com/ReScience/submissions/issues/32#issuecomment-633739558
2: https://github.com/ReScience/submissions/issues/32#issuecomment-634149030


Half-related to the blog post.  You mention elsewhere this baby channel
[3], maybe it could be worth to link it somewhere in the blog post.
Moreover, totally unrelated, I feel it lacks a list of "Scientific"
channels, as [4] or [5], maybe on hpc.guix.info

3: https://gitlab.inria.fr/guix-hpc/guix-past
4: https://github.com/BIMSBbioinfo/guix-bimsb
5: https://gitlab.inria.fr/guix-hpc/guix-hpc


Thanks,
simon

Re: “Reproducible research articles, from source code to PDF”

[Maxim Cournoyer]

Hi Ludovic!

Ludovic Courtès <ludovic.courtes@inria.fr> writes:

> Hello!
>
> This new post introduces the work I did to have a fully reproducible
> replication (!) of a 13-year old article, using Guix to express the
> whole pipeline:
>
>   https://hpc.guix.info/blog/2020/06/reproducible-research-articles-from-source-code-to-pdf/
>
> Feedback welcome!
>
> Ludo’.

Fun :-).  Thanks for sharing.

Maxim

Re: “Reproducible research articles, from source code to PDF”

[Ludovic Courtès]

Hi Simon!

zimoun <zimon.toutoune@gmail.com> skribis:

> For example, they are future bridges to think: connect the Guix archive
> somehow with zenodo DOI and/or Software Heritage identifier.
>
> When I read this comment in the review [1]:
>
>         As  a final  note,  I  wonder if,  and  how  much, the  author's
>         approach to reproducible computation/automated report generation
>         is  feasible  for  the  average scientist,  in  particular  when
>         compared to tools with a smoother learning curve, such as Docker
>         containers,  Jupyter notebooks,  R  Markdown  documents and  the
>         like. A brief  analysis of this topic with  a clear presentation
>         of the  advantages of  the author's  approach in  the Discussion
>         session would be worthwhile.
>
> and then the Konrad's answer [2], I asked myself what pieces are
> missing.  And what could be the articulation of "guix pack -f docker",
> Guix-Jupyter or other notebooks (RMarkdown, Org)?  And what could be a
> practical workflow? (Keeping in mind that the average scientist is not a
> Linux guru but often run MacOS or Windows.)
>
> 1: https://github.com/ReScience/submissions/issues/32#issuecomment-633739558
> 2: https://github.com/ReScience/submissions/issues/32#issuecomment-634149030

I don’t like the phrase “average scientist”, and we’re talking about
people with a PhD who definitely know how to learn.

Apart from that, I agree with the comments above: putting it in the
hands of scientists will be the real challenge.  I think providing
modules and ready-to-use “templates” for people who use R+RMarkdown, or
LaTeX, or Jupyter, etc. is a necessary step.

> Half-related to the blog post.  You mention elsewhere this baby channel
> [3], maybe it could be worth to link it somewhere in the blog post.
> Moreover, totally unrelated, I feel it lacks a list of "Scientific"
> channels, as [4] or [5], maybe on hpc.guix.info
>
> 3: https://gitlab.inria.fr/guix-hpc/guix-past
> 4: https://github.com/BIMSBbioinfo/guix-bimsb
> 5: https://gitlab.inria.fr/guix-hpc/guix-hpc

<https://hpc.guix.info/about/> has a list of channels.  I’ve added Guix
Past now.

Thanks for your feedback!

Ludo’.

Re: “Reproducible research articles, from source code to PDF”

[Konrad Hinsen]

Hi Ludo and Simon,

> I don’t like the phrase “average scientist”, and we’re talking about
> people with a PhD who definitely know how to learn.

I didn't take that phrase as a reference to ability, but to prior
knowledge. I am pretty sure that anyone who uses Python can also learn
to use Guile, but a computer scientist having experience with ten
languages will have less effort to do so than an archaeologist or a
wetlab biologist who has never used anything else but Python.

> Apart from that, I agree with the comments above: putting it in the
> hands of scientists will be the real challenge.  I think providing
> modules and ready-to-use “templates” for people who use R+RMarkdown, or
> LaTeX, or Jupyter, etc. is a necessary step.

I'd start somewhat differently: generate diverse use case examples.  The
contributions to the ReScience reproducibility challenge could be a nice
starting point: go through them, one by one, and try to re-implement the
authors' various approaches with Guix.  Then, in a second step, try to
identify additional tooling support in Guix that would make the recipes
simpler to implement. That might well lead to the development of ready
to use templates, but I prefer starting from a use case analysis to see
what is needed in real life.

Another obstacle to adoption is the difficulty of deployment. Right now,
if I use Guix to make my work reproducible, I require my readers to
install Guix on their computers, which is a lot of work for Linux users
and a major headache for Windows/macOS users. We really need to reduce
that barrier to deployment.

Some ideas: Simple deployment of a VM running Guix System on a major
cloud provider would be nice to have. Or a service like mybinder.org,
but based on Guix rather than Docker. Or, for local execution, a Docker
image containing Guix plus some tooling to do the equivalent of "guix
time-machine –commit=xxx – build -f guix.scm" plus copying the contents
of the generated package into the user's directory.

Cheers,
  Konrad
-- 
---------------------------------------------------------------------
Konrad Hinsen
Centre de Biophysique Moléculaire, CNRS Orléans
Synchrotron Soleil - Division Expériences
Saint Aubin - BP 48
91192 Gif sur Yvette Cedex, France
Tel. +33-1 69 35 97 15
E-Mail: konrad DOT hinsen AT cnrs DOT fr
http://dirac.cnrs-orleans.fr/~hinsen/
ORCID: https://orcid.org/0000-0003-0330-9428
Twitter: @khinsen
---------------------------------------------------------------------

Re: “Reproducible research articles, from source code to PDF”

[Ludovic Courtès]

Hi,

Konrad Hinsen <konrad.hinsen@cnrs.fr> skribis:

>> I don’t like the phrase “average scientist”, and we’re talking about
>> people with a PhD who definitely know how to learn.
>
> I didn't take that phrase as a reference to ability, but to prior
> knowledge. I am pretty sure that anyone who uses Python can also learn
> to use Guile, but a computer scientist having experience with ten
> languages will have less effort to do so than an archaeologist or a
> wetlab biologist who has never used anything else but Python.

Yes, I understand and agree with this assessment: making the tools
usable without being an expert will be crucial.

That said, these same people got used to Dockerfiles, CONDA, pip,
Jupyter, CWL, Python, Bash, and whatnot.  I think that stating that all
this, taken together, has a “smooth learning curve”, is inaccurate.

>> Apart from that, I agree with the comments above: putting it in the
>> hands of scientists will be the real challenge.  I think providing
>> modules and ready-to-use “templates” for people who use R+RMarkdown, or
>> LaTeX, or Jupyter, etc. is a necessary step.
>
> I'd start somewhat differently: generate diverse use case examples.  The
> contributions to the ReScience reproducibility challenge could be a nice
> starting point: go through them, one by one, and try to re-implement the
> authors' various approaches with Guix.  Then, in a second step, try to
> identify additional tooling support in Guix that would make the recipes
> simpler to implement. That might well lead to the development of ready
> to use templates, but I prefer starting from a use case analysis to see
> what is needed in real life.

Yes, sounds like a good plan!

> Another obstacle to adoption is the difficulty of deployment. Right now,
> if I use Guix to make my work reproducible, I require my readers to
> install Guix on their computers, which is a lot of work for Linux users
> and a major headache for Windows/macOS users. We really need to reduce
> that barrier to deployment.

On Debian and derivatives, it should be possible to run “apt-get install
guix” soonish, which should help.

For Windows and macOS, I don’t know (I’m personally less interested in
that but I agree it would be useful to improve the situation there.)

> Some ideas: Simple deployment of a VM running Guix System on a major
> cloud provider would be nice to have. Or a service like mybinder.org,
> but based on Guix rather than Docker. Or, for local execution, a Docker
> image containing Guix plus some tooling to do the equivalent of "guix
> time-machine –commit=xxx – build -f guix.scm" plus copying the contents
> of the generated package into the user's directory.

Yes, we should discuss individual solutions along these lines.

Thanks,
Ludo’.

Re: “Reproducible research articles, from source code to PDF”

[zimoun]

On Thu, 18 Jun 2020 at 13:42, Ludovic Courtès <ludovic.courtes@inria.fr> wrote:

>> I'd start somewhat differently: generate diverse use case examples.  The
>> contributions to the ReScience reproducibility challenge could be a nice
>> starting point: go through them, one by one, and try to re-implement the
>> authors' various approaches with Guix.  Then, in a second step, try to
>> identify additional tooling support in Guix that would make the recipes
>> simpler to implement. That might well lead to the development of ready
>> to use templates, but I prefer starting from a use case analysis to see
>> what is needed in real life.
>
> Yes, sounds like a good plan!

Maybe we could organize a Virtual Hackathon?  Over 1 day?  Or 2 days?
The power of collective motivation. :-)

Re: “Reproducible research articles, from source code to PDF”

[Ludovic Courtès]

zimoun <zimon.toutoune@gmail.com> skribis:

> On Thu, 18 Jun 2020 at 13:42, Ludovic Courtès <ludovic.courtes@inria.fr> wrote:
>
>>> I'd start somewhat differently: generate diverse use case examples.  The
>>> contributions to the ReScience reproducibility challenge could be a nice
>>> starting point: go through them, one by one, and try to re-implement the
>>> authors' various approaches with Guix.  Then, in a second step, try to
>>> identify additional tooling support in Guix that would make the recipes
>>> simpler to implement. That might well lead to the development of ready
>>> to use templates, but I prefer starting from a use case analysis to see
>>> what is needed in real life.
>>
>> Yes, sounds like a good plan!
>
> Maybe we could organize a Virtual Hackathon?  Over 1 day?  Or 2 days?
> The power of collective motivation. :-)

Sounds like a good idea!  Perhaps a one-day hackathon to begin with?
Early July maybe?

Ludo’.

Re: “Reproducible research articles, from source code to PDF”

[Konrad Hinsen]

Ludovic Courtès <ludovic.courtes@inria.fr> writes:

>> Maybe we could organize a Virtual Hackathon?  Over 1 day?  Or 2 days?
>> The power of collective motivation. :-)
>
> Sounds like a good idea!  Perhaps a one-day hackathon to begin with?
> Early July maybe?

Sounds fine. I am not much of a hackathon expert, so I don't propose
myself for organizing this, but I can make a preselection of suitable
submissions to the ReScience challenge (no proprietary software etc.)
with comments about the specific challenges.

Cheers,
  Konrad
-- 
---------------------------------------------------------------------
Konrad Hinsen
Centre de Biophysique Moléculaire, CNRS Orléans
Synchrotron Soleil - Division Expériences
Saint Aubin - BP 48
91192 Gif sur Yvette Cedex, France
Tel. +33-1 69 35 97 15
E-Mail: konrad DOT hinsen AT cnrs DOT fr
http://dirac.cnrs-orleans.fr/~hinsen/
ORCID: https://orcid.org/0000-0003-0330-9428
Twitter: @khinsen
---------------------------------------------------------------------

Re: “Reproducible research articles, from source code to PDF”

[Pjotr Prins]

On Thu, Jun 18, 2020 at 03:05:42PM +0200, Ludovic Courtès wrote:
> Sounds like a good idea!  Perhaps a one-day hackathon to begin with?
> Early July maybe?

If I have no other obligations I am game.

Pj.

Re: “Reproducible research articles, from source code to PDF”

[Ludovic Courtès]

Hi,

Konrad Hinsen <konrad.hinsen@cnrs.fr> skribis:

> Ludovic Courtès <ludovic.courtes@inria.fr> writes:
>
>>> Maybe we could organize a Virtual Hackathon?  Over 1 day?  Or 2 days?
>>> The power of collective motivation. :-)
>>
>> Sounds like a good idea!  Perhaps a one-day hackathon to begin with?
>> Early July maybe?
>
> Sounds fine. I am not much of a hackathon expert, so I don't propose
> myself for organizing this, but I can make a preselection of suitable
> submissions to the ReScience challenge (no proprietary software etc.)
> with comments about the specific challenges.

I’d happily defer to Simon; Simon, would you be able to organize this?

I guess we don’t need much actually: a date, an IRC channel, and
perhaps a web page on hpc.guix.info announcing the event so others can
join.

Ludo’.

Re: “Reproducible research articles, from source code to PDF”

[zimoun]

On Fri, 19 Jun 2020 at 14:04, Ludovic Courtès <ludovic.courtes@inria.fr> wrote:

> I’d happily defer to Simon; Simon, would you be able to organize this?

Yesterday evening, I have sent an email to guix-hpc with the subject:
"Let sync Hackathon reproducible and past" for this purpose. :-)
Is this email lost?  Or somehow blocked?  Let me know?

Well, I do not know if I would be able to organize but I can do my best. :-)

> I guess we don’t need much actually: a date, an IRC channel, and
> perhaps a web page on hpc.guix.info announcing the event so others can
> join.

The email mentioned above asks exactly that: first pick a date.  And
for the sake of archive, I started another thread.

Cheers,
simon

Re: “Reproducible research articles, from source code to PDF”

[Ludovic Courtès]

zimoun <zimon.toutoune@gmail.com> skribis:

> On Fri, 19 Jun 2020 at 14:04, Ludovic Courtès <ludovic.courtes@inria.fr> wrote:
>
>> I’d happily defer to Simon; Simon, would you be able to organize this?
>
> Yesterday evening, I have sent an email to guix-hpc with the subject:
> "Let sync Hackathon reproducible and past" for this purpose. :-)
> Is this email lost?  Or somehow blocked?  Let me know?

Oh great, let me check mail…

Ludo’.

Re: “Reproducible research articles, from source code to PDF”

[Konrad Hinsen]

Konrad Hinsen <konrad.hinsen@cnrs.fr> writes:

> Sounds fine. I am not much of a hackathon expert, so I don't propose
> myself for organizing this, but I can make a preselection of suitable
> submissions to the ReScience challenge (no proprietary software etc.)
> with comments about the specific challenges.

Here is my list of candidate projects. There are three general
categories:

1) Package old software that is of sufficiently wide interest
   (i.e. add to guix-past)
    - g77 (used in https://github.com/ReScience/submissions/issues/41)
    - SciPy ecosystem from 2007 (at least Python, NumPy, matplotlib)
      (used in https://github.com/ReScience/submissions/issues/14)

2) Package highly specialized research software

   These programs are too specialized for the Guix distribution, so
   "packaging" means writing a guix.scm. The long-term goal is to learn how
   to make this kind of packaging easier, to the point that scientists are
   willing to do it themselves. This means it must be doable with minimal
   Guile competence, ideally by modifying templates provided by experts.

   I have picked four cases, listed by increasing level of difficulty:

   a) https://github.com/ReScience/submissions/issues/42

   A rather standard Fortran code, with only the popular BLAS and LAPACK
   libraries as dependencies. Instructions are given for manual
   compilation.

   b) https://github.com/ReScience/submissions/issues/36

   A medium-sized Fortran program with a Makefile.

   c) https://github.com/ReScience/submissions/issues/41

   A mixed C-Fortran code from 2008, built with autotools. Looks simple,
   but the author did not succeed in compiling it on a modern machine
   because it requires the abandoned g77 compiler.

   d) https://github.com/ReScience/submissions/issues/20

   A medium-sized Fortran library with a Makefile. Tricky because it adds
   its own wrappers around the Fortran compiler.

3) Fully automated reproductions of results (typically figures)

   There is only one case (other than Ludo's which already uses Guix):

   - https://github.com/ReScience/submissions/issues/39

   A fully reproducible reproduction of two Open Source simulation software
   packages (C/C++), based on Debian and its debuerreotype system. The
   challenge is to demonstrate how Guix can do it better!

Cheers,
  Konrad
-- 
---------------------------------------------------------------------
Konrad Hinsen
Centre de Biophysique Moléculaire, CNRS Orléans
Synchrotron Soleil - Division Expériences
Saint Aubin - BP 48
91192 Gif sur Yvette Cedex, France
Tel. +33-1 69 35 97 15
E-Mail: konrad DOT hinsen AT cnrs DOT fr
http://dirac.cnrs-orleans.fr/~hinsen/
ORCID: https://orcid.org/0000-0003-0330-9428
Twitter: @khinsen
---------------------------------------------------------------------

Re: “Reproducible research articles, from source code to PDF”

[Ludovic Courtès]

Hi Konrad,

Konrad Hinsen <konrad.hinsen@cnrs.fr> skribis:

> Konrad Hinsen <konrad.hinsen@cnrs.fr> writes:
>
>> Sounds fine. I am not much of a hackathon expert, so I don't propose
>> myself for organizing this, but I can make a preselection of suitable
>> submissions to the ReScience challenge (no proprietary software etc.)
>> with comments about the specific challenges.
>
> Here is my list of candidate projects. There are three general
> categories:
>
> 1) Package old software that is of sufficiently wide interest
>    (i.e. add to guix-past)
>     - g77 (used in https://github.com/ReScience/submissions/issues/41)
>     - SciPy ecosystem from 2007 (at least Python, NumPy, matplotlib)
>       (used in https://github.com/ReScience/submissions/issues/14)

Makes sense.

> 2) Package highly specialized research software
>
>    These programs are too specialized for the Guix distribution, so
>    "packaging" means writing a guix.scm. The long-term goal is to learn how
>    to make this kind of packaging easier, to the point that scientists are
>    willing to do it themselves. This means it must be doable with minimal
>    Guile competence, ideally by modifying templates provided by experts.
>
>    I have picked four cases, listed by increasing level of difficulty:
>
>    a) https://github.com/ReScience/submissions/issues/42
>
>    A rather standard Fortran code, with only the popular BLAS and LAPACK
>    libraries as dependencies. Instructions are given for manual
>    compilation.
>
>    b) https://github.com/ReScience/submissions/issues/36
>
>    A medium-sized Fortran program with a Makefile.
>
>    c) https://github.com/ReScience/submissions/issues/41
>
>    A mixed C-Fortran code from 2008, built with autotools. Looks simple,
>    but the author did not succeed in compiling it on a modern machine
>    because it requires the abandoned g77 compiler.
>
>    d) https://github.com/ReScience/submissions/issues/20
>
>    A medium-sized Fortran library with a Makefile. Tricky because it adds
>    its own wrappers around the Fortran compiler.

That’s going to be less fun but I agree it’s important to do something
for this use case.

> 3) Fully automated reproductions of results (typically figures)
>
>    There is only one case (other than Ludo's which already uses Guix):
>
>    - https://github.com/ReScience/submissions/issues/39
>
>    A fully reproducible reproduction of two Open Source simulation software
>    packages (C/C++), based on Debian and its debuerreotype system. The
>    challenge is to demonstrate how Guix can do it better!

Heh, nice!  <https://github.com/debuerreotype/debuerreotype> is
interesting.

Thanks for sharing this overview, I guess we have enough on our plate now!
:-)

Ludo’.