persistent reproducibility ?

persistent reproducibility ?

[zimoun]

Hi Guixers,


Disclaimer: my background is about numerical analysis (PDE, Krylov
solver, etc.) and I am currently working in some Core Facility about
biology and bioinformatic (NGS pipelines, flow cytometry clustering,
etc.). Orthogonal communities but same issues. :-)


The typical research workflow is:

- Alice proposes new method and/or algorithm, publishes a paper and
illustrates that by the software `foo'. Let the best case: Alice
provides a Guix "recipe", and all the material is stored in Github
(let say). This software `foo' depends on both `bar' and `baz', one
also in Github and the other one included in the Guix package tree.

- It is easy for Bob to check out and experiment. Guix allows him to
straightforwardly build the bit identical `foo' (all dependencies
included). Nice!! Repeatability is there for free.

- New features are added to `foo', `bar' and `baz'. All the codes
evolve, especially the research ones.

- Now, Joe is implementing the Alice's method; science means
reproducible. And Joe would like to compare his implementation to the
Alice one provided by `foo'. However, how ? The `foo' "ecosystem" has
changed with the new features. Therefore, Joe has to navigate in the
Git tree of the Guix "recipe" of `foo', `bar', `baz' to be able to
produce the bit-identical `foo' used in the initial paper. I mean, it
is what I understand to do, and it does not seem reasonable.


My question is: does Guix provide any mechanism to build reproducible
software over the time ?


Last, `foo' and `bar' are stored in two Github repositories. And they
should disappear.
( I am not talking if it is good or not to use github, right now, it
just is used by many teams of researchers )

Could we used the Software Heritage initiative to maintain a kind of
persistency ?
https://www.softwareheritage.org


I do not know if my wishes make any sense.

All the best,
-simon

Re: persistent reproducibility ?

[Alex Sassmannshausen]

Hi Simon,

zimoun writes:

> Hi Guixers,
>
>
> Disclaimer: my background is about numerical analysis (PDE, Krylov
> solver, etc.) and I am currently working in some Core Facility about
> biology and bioinformatic (NGS pipelines, flow cytometry clustering,
> etc.). Orthogonal communities but same issues. :-)

Welcome! Sounds very exciting :-)

> The typical research workflow is:
>
> - Alice proposes new method and/or algorithm, publishes a paper and
> illustrates that by the software `foo'. Let the best case: Alice
> provides a Guix "recipe", and all the material is stored in Github
> (let say). This software `foo' depends on both `bar' and `baz', one
> also in Github and the other one included in the Guix package tree.
>
> - It is easy for Bob to check out and experiment. Guix allows him to
> straightforwardly build the bit identical `foo' (all dependencies
> included). Nice!! Repeatability is there for free.
>
> - New features are added to `foo', `bar' and `baz'. All the codes
> evolve, especially the research ones.
>
> - Now, Joe is implementing the Alice's method; science means
> reproducible. And Joe would like to compare his implementation to the
> Alice one provided by `foo'. However, how ? The `foo' "ecosystem" has
> changed with the new features. Therefore, Joe has to navigate in the
> Git tree of the Guix "recipe" of `foo', `bar', `baz' to be able to
> produce the bit-identical `foo' used in the initial paper. I mean, it
> is what I understand to do, and it does not seem reasonable.

There are others on here who may have more in-depth experience of
precisely this issue, but here's my impression of how this would work.

Alice's package recipe would specify the exact version of itself, and
also of `bar` and `baz` that that exactly version required.  As a
result, when software, and also package recipe versions evolve, all Joe
would have to do is locate the historic package recipe of `foo` that Joe
is trying to reproduce.  That package recipe would then point to the
historic versions of `bar` & `baz`, so Joe would not have to worry about
that.

> My question is: does Guix provide any mechanism to build reproducible
> software over the time ?

Guix just parses the recipe. The recipe details the exact versions of
software required.

If the exact recipe of a given time is not available then Guix cannot
help that (though if the recipe is added to Guix then it will be
available in Guix's repository).
If the repository hosting the exact version of the software that the
recipe specifies is no longer available then Guix cannot help.

> Last, `foo' and `bar' are stored in two Github repositories. And they
> should disappear.
> ( I am not talking if it is good or not to use github, right now, it
> just is used by many teams of researchers )
>
> Could we used the Software Heritage initiative to maintain a kind of
> persistency ?
> https://www.softwareheritage.org

There was some conversation about this at some point.  Again others may
know more about this. It certainly seems a nice means of ensuring
software remains available…

> I do not know if my wishes make any sense.

I think they definitely do :-)

HTH,

Alex

Re: persistent reproducibility ?

[Ludovic Courtès]

Hello!

zimoun <zimon.toutoune@gmail.com> skribis:

> The typical research workflow is:
>
> - Alice proposes new method and/or algorithm, publishes a paper and
> illustrates that by the software `foo'. Let the best case: Alice
> provides a Guix "recipe", and all the material is stored in Github
> (let say). This software `foo' depends on both `bar' and `baz', one
> also in Github and the other one included in the Guix package tree.
>
> - It is easy for Bob to check out and experiment. Guix allows him to
> straightforwardly build the bit identical `foo' (all dependencies
> included). Nice!! Repeatability is there for free.
>
> - New features are added to `foo', `bar' and `baz'. All the codes
> evolve, especially the research ones.
>
> - Now, Joe is implementing the Alice's method; science means
> reproducible. And Joe would like to compare his implementation to the
> Alice one provided by `foo'. However, how ? The `foo' "ecosystem" has
> changed with the new features. Therefore, Joe has to navigate in the
> Git tree of the Guix "recipe" of `foo', `bar', `baz' to be able to
> produce the bit-identical `foo' used in the initial paper. I mean, it
> is what I understand to do, and it does not seem reasonable.
>
>
> My question is: does Guix provide any mechanism to build reproducible
> software over the time ?

To add to what Alex wrote, yes it’s possible, though there are UI gaps
that we’ll be filling.  If you do a checkout of the Guix commit that
Alice mentioned in the paper, you can build the exact same software as
Alice.  ‘guix pull’ allows you to specify the Guix commit you’d like to
use, but it’s not that convenient that it’s something we’d like to
improve.

Ricardo and I wrote about the kind of workflows you describe in
<https://arxiv.org/abs/1506.02822>.  I hope you’ll find it interesting!

> Last, `foo' and `bar' are stored in two Github repositories. And they
> should disappear.
> ( I am not talking if it is good or not to use github, right now, it
> just is used by many teams of researchers )
>
> Could we used the Software Heritage initiative to maintain a kind of
> persistency ?
> https://www.softwareheritage.org

Definitely!  Software Heritage does not expose it yet, but when it does,
we can add it as a fallback mirror in our (guix git-download) module.
(I’ve discussed our use case with the Software Heritage folks a few
times, so they’re aware of it.  ;-))

I think the big picture is (where each arrow means “depends on”):

  repro science -> repro software environments -> stable archive

Things like ReScience can do the first step, Guix can do the second one,
and Software Heritage does the third one.

Thanks for sharing your use case!

Ludo’.

Re: persistent reproducibility ?

[zimoun]

Hi!

On 21 March 2017 at 17:19, Ludovic Courtès <ludo@gnu.org> wrote:
> Hello!
>
> zimoun <zimon.toutoune@gmail.com> skribis:
>
>> The typical research workflow is:
>>
>> - Alice proposes new method and/or algorithm, publishes a paper and
>> illustrates that by the software `foo'. Let the best case: Alice
>> provides a Guix "recipe", and all the material is stored in Github
>> (let say). This software `foo' depends on both `bar' and `baz', one
>> also in Github and the other one included in the Guix package tree.
>>
>> - It is easy for Bob to check out and experiment. Guix allows him to
>> straightforwardly build the bit identical `foo' (all dependencies
>> included). Nice!! Repeatability is there for free.
>>
>> - New features are added to `foo', `bar' and `baz'. All the codes
>> evolve, especially the research ones.
>>
>> - Now, Joe is implementing the Alice's method; science means
>> reproducible. And Joe would like to compare his implementation to the
>> Alice one provided by `foo'. However, how ? The `foo' "ecosystem" has
>> changed with the new features. Therefore, Joe has to navigate in the
>> Git tree of the Guix "recipe" of `foo', `bar', `baz' to be able to
>> produce the bit-identical `foo' used in the initial paper. I mean, it
>> is what I understand to do, and it does not seem reasonable.
>>
>>
>> My question is: does Guix provide any mechanism to build reproducible
>> software over the time ?
>
> To add to what Alex wrote, yes it’s possible, though there are UI gaps
> that we’ll be filling.  If you do a checkout of the Guix commit that
> Alice mentioned in the paper, you can build the exact same software as
> Alice.  ‘guix pull’ allows you to specify the Guix commit you’d like to
> use, but it’s not that convenient that it’s something we’d like to
> improve.

Thank you for your quick answer.

I was not aware of the `guix pull' commit specification. I am going to try.

If I understand well your both explanations, my question overlaps the
current discussion about Channels. I mean, in the same way that people
currently include in their project conda shell files or spack python
files or whatever in this flavour, they will replace by Guile ones
(even if lisp is hard to sell ;-), and there is questions about how to
glue these channels.

One of the issues is that the Guix packages tree will never include
some softwares, even if they are open source. Because the authors
apply weird licences or non-GNU compliant licences, or simply because
authors are not so motivated to push. Even if I totally agree with the
paragraph about Proprietary Softwares in your cited paper, it is just
a fact from my humble opinion.


Therefore, what should be the "standard" way to manipulate against
history version external and decentralised packages ? and guix repo
packages too ?


Well, if I understand your both answers, the correct process should
be: Alice publishes a paper containing the exact version (commit hash
or revision number or origin hash) of both the source tree and the
recipe tree, and their both uri location, and then, Joe "just" needs
to check out each (manually for now or possibly by nice UI glue).

From my current knowledge about Guix, considering two moments t1 and
t2, separated by let say several month, a way to build at t2 the
bit-identical `foo' of t1 should to provide at t1 a sort of manual
meta-package pointing to exacts version through `origin' and
`git-fetch' (imaging that `hg-fetch' and `svn-fetch' would be added,
or not!). If one piece of this information about the exact versions is
not pointed out at t1, it appears to me almost impossible to insure
the building of the bit-identical at t2.
If I understand well, this way is what you describe Alex, right ?

Another option should be: Guix add some meta-data in profile (kind of
manifest) that automatically tracks this information at t1. Then using
this profile at t2, it eases the building process, and it insures that
nothing had been forgotten.

Still from what I understand, Guix adds meta-data (roughly all the
propagated hashes) that allow to verify the bit-identity. However, I
am not sure if it does not lack meta-data included in profile or
elsewhere which describe the "state" at t (e.g., exact version by
commit specification) and then allow to revert from t+1 to t.

Maybe `archive' or `pack' already do the job ? Something able to pack
at the source level all the necessary materials to reproduce from
scratch a bit-identical environment.


>
> Ricardo and I wrote about the kind of workflows you describe in
> <https://arxiv.org/abs/1506.02822>.  I hope you’ll find it interesting!

Yes! I found it really interesting! :-)
I already read it long time ago and discovered spack and then how
HiePACS uses it. Thanks!
Anyway!
I am reading again with new glasses.


>
>> Last, `foo' and `bar' are stored in two Github repositories. And they
>> should disappear.
>> ( I am not talking if it is good or not to use github, right now, it
>> just is used by many teams of researchers )
>>
>> Could we used the Software Heritage initiative to maintain a kind of
>> persistency ?
>> https://www.softwareheritage.org
>
> Definitely!  Software Heritage does not expose it yet, but when it does,
> we can add it as a fallback mirror in our (guix git-download) module.
> (I’ve discussed our use case with the Software Heritage folks a few
> times, so they’re aware of it.  ;-))

Really nice !!

>
> I think the big picture is (where each arrow means “depends on”):
>
>   repro science -> repro software environments -> stable archive
>
> Things like ReScience can do the first step, Guix can do the second one,
> and Software Heritage does the third one.

"There is a long way to go, but the road is Free..." ;-)

>
> Thanks for sharing your use case!

Thanks for all this positive energy :-)
and thanks to give me a reason to learn scheme/guile because guix

--
simon

>
> Ludo’.

Re: persistent reproducibility ?

[Alex Sassmannshausen]

Hi Zimoun,

zimoun writes:

> Hi!
>
> On 21 March 2017 at 17:19, Ludovic Courtès <ludo@gnu.org> wrote:
>> Hello!
>>
>> zimoun <zimon.toutoune@gmail.com> skribis:
>>
>>> The typical research workflow is:
>>>
>>> - Alice proposes new method and/or algorithm, publishes a paper and
>>> illustrates that by the software `foo'. Let the best case: Alice
>>> provides a Guix "recipe", and all the material is stored in Github
>>> (let say). This software `foo' depends on both `bar' and `baz', one
>>> also in Github and the other one included in the Guix package tree.
>>>
>>> - It is easy for Bob to check out and experiment. Guix allows him to
>>> straightforwardly build the bit identical `foo' (all dependencies
>>> included). Nice!! Repeatability is there for free.
>>>
>>> - New features are added to `foo', `bar' and `baz'. All the codes
>>> evolve, especially the research ones.
>>>
>>> - Now, Joe is implementing the Alice's method; science means
>>> reproducible. And Joe would like to compare his implementation to the
>>> Alice one provided by `foo'. However, how ? The `foo' "ecosystem" has
>>> changed with the new features. Therefore, Joe has to navigate in the
>>> Git tree of the Guix "recipe" of `foo', `bar', `baz' to be able to
>>> produce the bit-identical `foo' used in the initial paper. I mean, it
>>> is what I understand to do, and it does not seem reasonable.
>>>
>>>
>>> My question is: does Guix provide any mechanism to build reproducible
>>> software over the time ?
>>
>> To add to what Alex wrote, yes it’s possible, though there are UI gaps
>> that we’ll be filling.  If you do a checkout of the Guix commit that
>> Alice mentioned in the paper, you can build the exact same software as
>> Alice.  ‘guix pull’ allows you to specify the Guix commit you’d like to
>> use, but it’s not that convenient that it’s something we’d like to
>> improve.
>
> Thank you for your quick answer.
>
> I was not aware of the `guix pull' commit specification. I am going to try.
>
> If I understand well your both explanations, my question overlaps the
> current discussion about Channels. I mean, in the same way that people
> currently include in their project conda shell files or spack python
> files or whatever in this flavour, they will replace by Guile ones
> (even if lisp is hard to sell ;-), and there is questions about how to
> glue these channels.
>
> One of the issues is that the Guix packages tree will never include
> some softwares, even if they are open source. Because the authors
> apply weird licences or non-GNU compliant licences, or simply because
> authors are not so motivated to push. Even if I totally agree with the
> paragraph about Proprietary Softwares in your cited paper, it is just
> a fact from my humble opinion.
>
>
> Therefore, what should be the "standard" way to manipulate against
> history version external and decentralised packages ? and guix repo
> packages too ?
>
>
> Well, if I understand your both answers, the correct process should
> be: Alice publishes a paper containing the exact version (commit hash
> or revision number or origin hash) of both the source tree and the
> recipe tree, and their both uri location, and then, Joe "just" needs
> to check out each (manually for now or possibly by nice UI glue).
>
> From my current knowledge about Guix, considering two moments t1 and
> t2, separated by let say several month, a way to build at t2 the
> bit-identical `foo' of t1 should to provide at t1 a sort of manual
> meta-package pointing to exacts version through `origin' and
> `git-fetch' (imaging that `hg-fetch' and `svn-fetch' would be added,
> or not!). If one piece of this information about the exact versions is
> not pointed out at t1, it appears to me almost impossible to insure
> the building of the bit-identical at t2.
> If I understand well, this way is what you describe Alex, right ?

Right — so I think I was a bit unclear in my first email.

A guix package recipe, as you point, points to an exact revision/version
of software (either through a version spec, or through a commit hash).
The package's dependencies are specified as pure scheme variables that
in turn resolve to package recipes.

These package recipes in turn specify the exact revision of the
dependencies.

So for `foo` to be fully reproducible at t1 you require the package tree
that corresponds to `foo`, `bar` & `baz` at t1.  This tree would carry
all the meta information related to `foo` and it's exact dependencies.

I guess you may also need the correct version of Guix.  I'm not sure how
that affects reproducibility, so I won't comment further on that.

In any case, if you have `foo`, `bar` & `baz` as part of a guix source
tree, then the revision of that source tree at t1 should be sufficient
to reproduce `foo` at t1.

I guess this is where the discussion of channels, and whether to package
the programs for Guix itself or somehow outside become relevant, but
again, I don't feel super qualified to comment on this.

All the meta-information required is carried by that revision.

I wasn't sure what you meant by manual-metapackage, so I hope my answer
provides you with the information you were looking for!

I won't comment on the latter parts of your email, as it sounds like
it's more related to fully reproducible environments rather than
individual packages.  Personally I use manifests, but that might not be
good enough for your requirements.

HTH,

Alex

Re: persistent reproducibility ?

[Ricardo Wurmus]

zimoun <zimon.toutoune@gmail.com> writes:

> One of the issues is that the Guix packages tree will never include
> some softwares, even if they are open source. Because the authors
> apply weird licences or non-GNU compliant licences, or simply because
> authors are not so motivated to push. Even if I totally agree with the
> paragraph about Proprietary Softwares in your cited paper, it is just
> a fact from my humble opinion.

If you mean “open source” in the sense of “using a license that is
certified by the Open Source Initiative” then that software is probably
Free Software.  There is no such thing as GNU compliance in licenses.

We do however follow the GNU FDSG (Free System Distribution Guidelines),
which may result in some software to be excluded or modified in rare
cases.  (One example is “Shogun”, which we modify to remove included
non-free software.)

> Therefore, what should be the "standard" way to manipulate against
> history version external and decentralised packages ? and guix repo
> packages too ?
>
>
> Well, if I understand your both answers, the correct process should
> be: Alice publishes a paper containing the exact version (commit hash
> or revision number or origin hash) of both the source tree and the
> recipe tree, and their both uri location, and then, Joe "just" needs
> to check out each (manually for now or possibly by nice UI glue).

It would be sufficient to provide two things: a manifest and the Guix
version (“git -C /path/to/guix.git describe”).  If additional package
repositories are used (such as guix-bimsb for my institute), their
versions have to be recorded as well.

Joe would then check out Guix and any additional package repositories at
the specified versions and then instantiate the manifest.  Provided that
all source tarballs are still obtainable (either directly or through a
mirror) and the builds are in fact bit-reproducible Joe would end up
with the same software environment that Alice fully specified in the
paper.

--
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net

Re: persistent reproducibility ?

[zimoun]

Hi!

Thank you for commenting and helping me to go forward.

>
> A guix package recipe, as you point, points to an exact revision/version
> of software (either through a version spec, or through a commit hash).
> The package's dependencies are specified as pure scheme variables that
> in turn resolve to package recipes.
>
> These package recipes in turn specify the exact revision of the
> dependencies.
>
> So for `foo` to be fully reproducible at t1 you require the package tree
> that corresponds to `foo`, `bar` & `baz` at t1.  This tree would carry
> all the meta information related to `foo` and it's exact dependencies.

I am not sure to understand, neither to be able to catch this exact
revision of the dependencies.

Let consider the example of the package Gmsh (I know a bit how it is developed):
https://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/maths.scm?id=v0.12.0-2125-g283f04b#n1097

Here, `gmsh' depends on `fltk', `gfortran', `lapack' etc.

If the commit hash of the Guix source tree is provided and known, then I agree.
It is possible to restore (build) the bit-identical `gmsh' at any time.
(assuming that the upstream is still available, which is another issue
and Ludo pointed out that some fallbacks by the Software Heritage are
in the pipes \o/)

At this point, I agree that on the paper, all the pieces are there and
it "just" lacks some UI glue.
And this point is a BIG step, compared to other solution on the
market. Thanks to Nix thesis and to you Guixers to implement it in
Guile. :-)


Now, as researcher, Alice forks Gmsh to explore a proof-of-concept of
her new algorithm.
Upstream refuses to include, for whatever reason, or Alice refuses to
push for whatever reason. Well this new algorithm will never be in the
Guix source tree.
Moreover, her new algorithm needs a specific patch of `lapack', and
this variant is too much specific to be included in the Guix source
tree.
At t1, Alice publishes all the materials, forked Gmsh commit hash and
its "reproducible hash", also the Guix source tree commit hash,
explained her `lapack' trick from the paper but she forgets to
precisely point the commit hash. Then, it is not possible to
bit-identically reproduce the forked Gmsh of the paper.

Ok, Guix cannot fix automagically what people does not do correctly. For sure!


But, should be somehow possible (and useful?) to add something to prevent that ?


I have in mind something in the flavor of `guix pack' at the source level.

When Alice releases her paper, she could release a binary pack of
everything useful to be able to run and in the same time a source pack
of everything useful to be able to build (over time). A kind of
derivation ? (it is what I called meta-package, hum?)


>
> I guess you may also need the correct version of Guix.  I'm not sure how
> that affects reproducibility, so I won't comment further on that.

I agree that experiments need to be performed.

>
> In any case, if you have `foo`, `bar` & `baz` as part of a guix source
> tree, then the revision of that source tree at t1 should be sufficient
> to reproduce `foo` at t1.

I agree.
And my question is: how to do if one of them is not part of a guix source tree ?

>
> I guess this is where the discussion of channels, and whether to package
> the programs for Guix itself or somehow outside become relevant, but
> again, I don't feel super qualified to comment on this.

I will browse the discussion about channels.

Thank you for your comment.
My feeling is that a feature lacks to Guix to deal with this
situation. Well, I will explore more details and see if I am able to
use your advices.

>
> All the meta-information required is carried by that revision.
>
> I wasn't sure what you meant by manual-metapackage, so I hope my answer
> provides you with the information you were looking for!

Writing by hand a package that provides all the necessary meta-data to
build bit-identical at any time.

If I install Guix, and for example I do "guix package -i foo -p pro".
Then, over months, I do operations as: create profiles, install,
remove, garbage collect, refresh, pull, etc.
Now, month after, I am not sure to be able to bit-identically build
the previous `foo' package.

Maybe I miss a point. Especially about `manifest'.

>
> I won't comment on the latter parts of your email, as it sounds like
> it's more related to fully reproducible environments rather than
> individual packages.  Personally I use manifests, but that might not be
> good enough for your requirements.

Thank you for your advices.
Time to play with keyboards :-)


All the best !

--
simon

Re: persistent reproducibility ?

[zimoun]

Hi!

Thank you the details.


>> One of the issues is that the Guix packages tree will never include
>> some softwares, even if they are open source. Because the authors
>> apply weird licences or non-GNU compliant licences, or simply because
>> authors are not so motivated to push. Even if I totally agree with the
>> paragraph about Proprietary Softwares in your cited paper, it is just
>> a fact from my humble opinion.
>
> If you mean “open source” in the sense of “using a license that is
> certified by the Open Source Initiative” then that software is probably
> Free Software.  There is no such thing as GNU compliance in licenses.

I mean "open source" any software publicly released with publicly
accessible source. It is large. ;-)

And I totally agree with the condition to be included or not in the
Guix infrastructure.

My point is that a lot of softwares released in scientific world will
never reach such condition. It is sad and I think all people here are
trying to change by convincing the authors. But, it is a pragmatic
fact.

Hum? I am not sure to get your point about GNU compliance. For example,
https://libreplanet.org/wiki/List_of_software_that_does_not_respect_the_Free_System_Distribution_Guidelines
`apt' and `debian-installer' illustrate the well-known and famous
debate between RMS and Debian. :-)

>
> We do however follow the GNU FDSG (Free System Distribution Guidelines),
> which may result in some software to be excluded or modified in rare
> cases.  (One example is “Shogun”, which we modify to remove included
> non-free software.)

Yes, the GNU FDSG defines "free" (as in speech). And there is "open
source" softwares which are not included in this definition (for the
good, for the bad, I am not arguing).
https://www.gnu.org/licenses/license-list.html#NonFreeSoftwareLicenses
For example, some versions of Scilab (clone of Matlab) with a "weird"
license (CeCILL-2).

For the record, let indicate an historical  point:
https://www.gnu.org/licenses/bsd.html :-)

I am not arguing about "bad" practices. I am just pointing that the
situation of scientific softwares is often really complicated. Because
they often glue a lot of different piece of softwares, e.g.,
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=617931

Note that, in the same way that non-free parts are removed from
`shogun', the guix package `gmsh' removes METIS non-free part (which
is "open source"). Hope that this clarifies my point and feeds
argument about channel. :-)

>
>> Therefore, what should be the "standard" way to manipulate against
>> history version external and decentralised packages ? and guix repo
>> packages too ?
>>
>>
>> Well, if I understand your both answers, the correct process should
>> be: Alice publishes a paper containing the exact version (commit hash
>> or revision number or origin hash) of both the source tree and the
>> recipe tree, and their both uri location, and then, Joe "just" needs
>> to check out each (manually for now or possibly by nice UI glue).
>
> It would be sufficient to provide two things: a manifest and the Guix
> version (“git -C /path/to/guix.git describe”).  If additional package
> repositories are used (such as guix-bimsb for my institute), their
> versions have to be recorded as well.

Thank you to clarify the way.

If I understand well, additional meta-data should be manually provided by Alice.

Extract a manifest is some UI glue, even, it is perhaps already possible.

However, the Guix version do not appear to me straightforward. Does
this information should be included when installing a package ?

>
> Joe would then check out Guix and any additional package repositories at
> the specified versions and then instantiate the manifest.  Provided that
> all source tarballs are still obtainable (either directly or through a
> mirror) and the builds are in fact bit-reproducible Joe would end up
> with the same software environment that Alice fully specified in the
> paper.


Should be possible to extract all the required information from a
profile with a kind of `guix pack' at the source level ?
This should be useful ? or not at all and I am wishing a twisted process ?


Thank you for your insights.
And for sharing the MDC's configuration!


All the best.

--
simon

>
> --
> Ricardo
>
> GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
> https://elephly.net
>

Re: persistent reproducibility ?

[Chris Marusich]

zimoun <zimon.toutoune@gmail.com> writes:

> Maybe `archive' or `pack' already do the job ? Something able to pack
> at the source level all the necessary materials to reproduce from
> scratch a bit-identical environment.

What about using 'guix archive' in combination with the 'guix build
--sources=transitive' command?  The manual explains ((guix) Additional
Build Options), the "--sources=transitive" option means:

"Build the source derivations of all packages, as well of all transitive
inputs to the packages.  This can be used e.g.  to prefetch package
source for later offline building."

Here's an example for the GNU Hello package:

--8<---------------cut here---------------start------------->8---
$ guix build --sources=transitive hello
/gnu/store/hbdalsf5lpf01x4dcknwx6xbn6n5km6k-hello-2.10.tar.gz
/gnu/store/cz1b1hn763zppafmlxsrlj7fnxa8zcna-tar-1.29.tar.xz
/gnu/store/f63c6mjqqai6xwypx5qslvqc2d302yn5-gzip-1.8.tar.xz
/gnu/store/bjpalrv9f49d2k40p7ga0y6dwdys9w5j-bzip2-1.0.6.tar.gz
/gnu/store/1pcksb758j9v7srain0k3fqp57h9gwmj-xz-5.2.2.tar.gz
/gnu/store/r56rrdk6h0zwb1d2y28q3bk71n73z84m-file-5.28.tar.gz
/gnu/store/rbaa5s5jpy91npas89q3p8yasrc6mr1h-diffutils-3.5.tar.xz
/gnu/store/iaigwb3k7fg0v8zrwlb4yhrdsgin6ir7-patch-2.7.5.tar.xz
/gnu/store/g0mffc9c0nxbwf0a6j0akh0dvkd2sn6w-sed-4.2.2.tar.xz
/gnu/store/76q5cqkqrx7paxhq27m3izag07n1vjs4-findutils-4.6.0.tar.xz
/gnu/store/vgs8flhpxbkgx0hx54hnihpai7lma72j-gawk-4.1.4.tar.xz
/gnu/store/5q4yicgzxrv13gq4rai6h07s7xp75ws6-grep-2.25.tar.xz
/gnu/store/8dqf59312llnmln89ccas1saq0z4qwi7-coreutils-8.25.tar.xz
/gnu/store/2g7nf5h6hnsz1sharnqxfjz9m6k7224z-make-4.2.1.tar.xz
/gnu/store/xwanyg9972rpznx89ja38v01c8483392-bash-4.4.tar.gz
/gnu/store/iv2nz8r001lpric4kr2xq3459zwl8051-binutils-2.27.tar.xz
/gnu/store/z2s847iq14psryslylm1x4k71b0vx4ng-gcc-4.9.4.tar.xz
/gnu/store/m9jym12mhczjrslpycwi6ipw57v849pa-gmp-6.0.0a.tar.xz
/gnu/store/xg7ir7g2yvxj9ixmgy2kfds690h3f6gg-mpfr-3.1.4.tar.xz
/gnu/store/y9akh452n3p4w2v631nj0injx7y0d68x-mpc-1.0.3.tar.gz
/gnu/store/z8mm3izh1hgllm6mlqzycsrx5nfkvhxi-glibc-2.24.tar.xz
/gnu/store/51f2f2ck80migsfr2hwy4ds30966lifi-linux-libre-4.4.18-gnu.tar.xz
--8<---------------cut here---------------end--------------->8---

You could archive those sources for later importation into the store on
another system.  For example:

  guix archive --export \
    $(guix build --sources=transitive hello) > hello-transitive-source.nar

If you import that archive (with 'guix archive --import') and build it
using the same version of the package definitions, theoretically it
should build even if the sources are no longer available for download
from their specified origin locations.

You could even do more.  By using 'guix gc', you could also include all
of the store items which are required to run GNU Hello:

--8<---------------cut here---------------start------------->8---
$ guix gc --requisites $(guix build hello)
/gnu/store/rvgmixpmsq5lqr9qflhkm70kg7a4rys2-bash-static-4.4.0
/gnu/store/iwgi9001dmmihrjg4rqhd6pa6788prjw-glibc-2.24
/gnu/store/cdi08kw7r6r684w8mk0xq0dkgpjhfpmd-gcc-4.9.4-lib
/gnu/store/rvs42awwwby7pq3j0znglmz3vyznvbh1-hello-2.10
--8<---------------cut here---------------end--------------->8---

Combine those two commands, and you should be able to export an archive
which contains all the files necessary to run GNU Hello, in addition to
all the sources necessary to build GNU Hello reproducibly (provided that
GNU Hello builds reproducibly, and provided that you build it using the
same version of the package definitions):

  guix archive --export \
    $(guix gc --requisites $(guix build hello)) \
    $(guix build --sources=transitive hello) > hello-requisites-and-sources.nar

You would also have to trust the key that was used to sign the archive
in order to import it (see 'guix archive --authorize').

By the way, in case you were wondering, one reason why we don't put the
sources for every package into the Guix source tree is because that
would make the Guix source tree prohibitively large.  The transitive
closure of sources for GNU Hello alone weighs in at 1750 MiB (wow!) when
uncompressed.  And that's only for those specific versions of the source
files.

-- 
Chris

Re: persistent reproducibility ?

[Quiliro]

El Thu, 23 Mar 2017 22:39:52 -0700
Chris Marusich <cmmarusich@gmail.com> escribió:

> By the way, in case you were wondering, one reason why we don't put
> the sources for every package into the Guix source tree is because
> that would make the Guix source tree prohibitively large.  The
> transitive closure of sources for GNU Hello alone weighs in at 1750
> MiB (wow!) when uncompressed.  And that's only for those specific
> versions of the source files.

How much space would the sources for every package in compressed
format take? 

-- 
Example of rude top posting:

A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
A: Top-posting.
Q: What is the most annoying thing in e-mail?

A: No.
Q: Should I leave quotations after my reply?

Saluton,
Quiliro
0987631031

Re: persistent reproducibility ?

[Ludovic Courtès]

Hi!

zimoun <zimon.toutoune@gmail.com> skribis:

>>> One of the issues is that the Guix packages tree will never include
>>> some softwares, even if they are open source. Because the authors
>>> apply weird licences or non-GNU compliant licences, or simply because
>>> authors are not so motivated to push. Even if I totally agree with the
>>> paragraph about Proprietary Softwares in your cited paper, it is just
>>> a fact from my humble opinion.
>>
>> If you mean “open source” in the sense of “using a license that is
>> certified by the Open Source Initiative” then that software is probably
>> Free Software.  There is no such thing as GNU compliance in licenses.
>
> I mean "open source" any software publicly released with publicly
> accessible source. It is large. ;-)

“Open source” as defined by the OSI means more that just “accessible
source”:

  https://opensource.org/definition

In effect it requires the 4 freedoms:

  https://www.gnu.org/philosophy/free-sw.html

Now, it is true that there’s software out there with “accessible source”
that is neither free software nor open source, especially on github.com
since GitHub makes it easy to publish code without specifying a license.

> My point is that a lot of softwares released in scientific world will
> never reach such condition. It is sad and I think all people here are
> trying to change by convincing the authors. But, it is a pragmatic
> fact.

I’m not sure.  Of course we’d have to be more specific than “a lot of”
;-), but I also see “a lot of” scientific software that is free; in
fact, I haven’t seen much non-free scientific software produced in the
CS research institutes here in France.

>> We do however follow the GNU FDSG (Free System Distribution Guidelines),
>> which may result in some software to be excluded or modified in rare
>> cases.  (One example is “Shogun”, which we modify to remove included
>> non-free software.)
>
> Yes, the GNU FDSG defines "free" (as in speech). And there is "open
> source" softwares which are not included in this definition (for the
> good, for the bad, I am not arguing).
> https://www.gnu.org/licenses/license-list.html#NonFreeSoftwareLicenses
> For example, some versions of Scilab (clone of Matlab) with a "weird"
> license (CeCILL-2).

The CeCILL licenses are all free software licenses, so CeCILL-licensed
software is welcome in Guix!

Thanks,
Ludo’.

Re: persistent reproducibility ?

[zimoun]

Hi !


> concerning my initial question

Thanks Chris!
It is exactly the `guix pack' at source level that I was looking for.
I am playing around.

I have still issues when redirecting the `export', e.g., `guix archive
--export hello' works, but not `guix archive --export hello >
hello.nar'
raising: `guix archive: error: corrupt input while restoring archive
from #<closed: file 0>'
Well, it is another topic.


> concerning license relative stuff

I am on the same wavelength and I almost agree.
My worries seems edge cases and I am maybe applying an overstatement
typical from south french people ;-)


Thanks to Guix community to share their positive energy.


All the best,
-simon


On 24 March 2017 at 16:45, Ludovic Courtès <ludo@gnu.org> wrote:
> Hi!
>
> zimoun <zimon.toutoune@gmail.com> skribis:
>
>>>> One of the issues is that the Guix packages tree will never include
>>>> some softwares, even if they are open source. Because the authors
>>>> apply weird licences or non-GNU compliant licences, or simply because
>>>> authors are not so motivated to push. Even if I totally agree with the
>>>> paragraph about Proprietary Softwares in your cited paper, it is just
>>>> a fact from my humble opinion.
>>>
>>> If you mean “open source” in the sense of “using a license that is
>>> certified by the Open Source Initiative” then that software is probably
>>> Free Software.  There is no such thing as GNU compliance in licenses.
>>
>> I mean "open source" any software publicly released with publicly
>> accessible source. It is large. ;-)
>
> “Open source” as defined by the OSI means more that just “accessible
> source”:
>
>   https://opensource.org/definition
>
> In effect it requires the 4 freedoms:
>
>   https://www.gnu.org/philosophy/free-sw.html
>
> Now, it is true that there’s software out there with “accessible source”
> that is neither free software nor open source, especially on github.com
> since GitHub makes it easy to publish code without specifying a license.
>
>> My point is that a lot of softwares released in scientific world will
>> never reach such condition. It is sad and I think all people here are
>> trying to change by convincing the authors. But, it is a pragmatic
>> fact.
>
> I’m not sure.  Of course we’d have to be more specific than “a lot of”
> ;-), but I also see “a lot of” scientific software that is free; in
> fact, I haven’t seen much non-free scientific software produced in the
> CS research institutes here in France.
>
>>> We do however follow the GNU FDSG (Free System Distribution Guidelines),
>>> which may result in some software to be excluded or modified in rare
>>> cases.  (One example is “Shogun”, which we modify to remove included
>>> non-free software.)
>>
>> Yes, the GNU FDSG defines "free" (as in speech). And there is "open
>> source" softwares which are not included in this definition (for the
>> good, for the bad, I am not arguing).
>> https://www.gnu.org/licenses/license-list.html#NonFreeSoftwareLicenses
>> For example, some versions of Scilab (clone of Matlab) with a "weird"
>> license (CeCILL-2).
>
> The CeCILL licenses are all free software licenses, so CeCILL-licensed
> software is welcome in Guix!
>
> Thanks,
> Ludo’.

Re: persistent reproducibility ?

[Chris Marusich]

[-- Attachment #1: Type: text/plain, Size: 3002 bytes --]

Quiliro <quiliro@riseup.net> writes:

> El Thu, 23 Mar 2017 22:39:52 -0700
> Chris Marusich <cmmarusich@gmail.com> escribió:
>
>> By the way, in case you were wondering, one reason why we don't put
>> the sources for every package into the Guix source tree is because
>> that would make the Guix source tree prohibitively large.  The
>> transitive closure of sources for GNU Hello alone weighs in at 1750
>> MiB (wow!) when uncompressed.  And that's only for those specific
>> versions of the source files.
>
> How much space would the sources for every package in compressed
> format take? 

According to 'guix size', the total size of the compressed sources for
hello is 213.3 MiB, which is still quite large:

--8<---------------cut here---------------start------------->8---
$ guix size $(guix build --sources=transitive hello)
store item                                                       total    self
/gnu/store/51f2f2ck80migsfr2hwy4ds30966lifi-linux-libre-4.4.18-gnu.tar.xz    81.1    81.1  38.0%
/gnu/store/z2s847iq14psryslylm1x4k71b0vx4ng-gcc-4.9.4.tar.xz        66.3    66.3  31.1%
/gnu/store/iv2nz8r001lpric4kr2xq3459zwl8051-binutils-2.27.tar.xz    18.9    18.9   8.9%
/gnu/store/z8mm3izh1hgllm6mlqzycsrx5nfkvhxi-glibc-2.24.tar.xz       13.7    13.7   6.4%
/gnu/store/xwanyg9972rpznx89ja38v01c8483392-bash-4.4.tar.gz          8.9     8.9   4.2%
/gnu/store/8dqf59312llnmln89ccas1saq0z4qwi7-coreutils-8.25.tar.xz     5.5     5.5   2.6%
/gnu/store/vgs8flhpxbkgx0hx54hnihpai7lma72j-gawk-4.1.4.tar.xz        2.3     2.3   1.1%
/gnu/store/cz1b1hn763zppafmlxsrlj7fnxa8zcna-tar-1.29.tar.xz          1.9     1.9   0.9%
/gnu/store/m9jym12mhczjrslpycwi6ipw57v849pa-gmp-6.0.0a.tar.xz        1.8     1.8   0.9%
/gnu/store/76q5cqkqrx7paxhq27m3izag07n1vjs4-findutils-4.6.0.tar.xz     1.7     1.7   0.8%
/gnu/store/1pcksb758j9v7srain0k3fqp57h9gwmj-xz-5.2.2.tar.gz          1.4     1.4   0.7%
/gnu/store/rbaa5s5jpy91npas89q3p8yasrc6mr1h-diffutils-3.5.tar.xz     1.3     1.3   0.6%
/gnu/store/5q4yicgzxrv13gq4rai6h07s7xp75ws6-grep-2.25.tar.xz         1.3     1.3   0.6%
/gnu/store/xg7ir7g2yvxj9ixmgy2kfds690h3f6gg-mpfr-3.1.4.tar.xz        1.1     1.1   0.5%
/gnu/store/2g7nf5h6hnsz1sharnqxfjz9m6k7224z-make-4.2.1.tar.xz        1.1     1.1   0.5%
/gnu/store/g0mffc9c0nxbwf0a6j0akh0dvkd2sn6w-sed-4.2.2.tar.xz         0.8     0.8   0.4%
/gnu/store/bjpalrv9f49d2k40p7ga0y6dwdys9w5j-bzip2-1.0.6.tar.gz       0.7     0.7   0.3%
/gnu/store/r56rrdk6h0zwb1d2y28q3bk71n73z84m-file-5.28.tar.gz         0.7     0.7   0.3%
/gnu/store/iaigwb3k7fg0v8zrwlb4yhrdsgin6ir7-patch-2.7.5.tar.xz       0.7     0.7   0.3%
/gnu/store/f63c6mjqqai6xwypx5qslvqc2d302yn5-gzip-1.8.tar.xz          0.7     0.7   0.3%
/gnu/store/hbdalsf5lpf01x4dcknwx6xbn6n5km6k-hello-2.10.tar.gz        0.7     0.7   0.3%
/gnu/store/y9akh452n3p4w2v631nj0injx7y0d68x-mpc-1.0.3.tar.gz         0.6     0.6   0.3%
total: 213.3 MiB
--8<---------------cut here---------------end--------------->8---

-- 
Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: persistent reproducibility ?

[Quiliro]

El Sun, 26 Mar 2017 02:39:19 -0700
Chris Marusich <cmmarusich@gmail.com> escribió:

> Quiliro <quiliro@riseup.net> writes:
> 
> > El Thu, 23 Mar 2017 22:39:52 -0700
> > Chris Marusich <cmmarusich@gmail.com> escribió:
> >  
> >> By the way, in case you were wondering, one reason why we don't put
> >> the sources for every package into the Guix source tree is because
> >> that would make the Guix source tree prohibitively large.  The
> >> transitive closure of sources for GNU Hello alone weighs in at 1750
> >> MiB (wow!) when uncompressed.  And that's only for those specific
> >> versions of the source files.  
> >
> > How much space would the sources for every package in compressed
> > format take?   
> 
> According to 'guix size', the total size of the compressed sources for
> hello is 213.3 MiB, which is still quite large:

Thank you very much for this example. I know hello requieres certain
dependencies that make it very heavy to download them. Perhaps these
will be part of the big weight of the whole. How much does the whole
GuixSD package repository weigh? How can it be calculated without
downloading everything?

-- 
Example of rude top posting:

A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
A: Top-posting.
Q: What is the most annoying thing in e-mail?

A: No.
Q: Should I leave quotations after my reply?

Saluton,
Quiliro
0987631031