* persistent reproducibility ? @ 2017-03-21 11:15 zimoun 2017-03-21 14:49 ` Alex Sassmannshausen 2017-03-21 16:19 ` Ludovic Courtès 0 siblings, 2 replies; 14+ messages in thread From: zimoun @ 2017-03-21 11:15 UTC (permalink / raw) To: help-guix Hi Guixers, Disclaimer: my background is about numerical analysis (PDE, Krylov solver, etc.) and I am currently working in some Core Facility about biology and bioinformatic (NGS pipelines, flow cytometry clustering, etc.). Orthogonal communities but same issues. :-) The typical research workflow is: - Alice proposes new method and/or algorithm, publishes a paper and illustrates that by the software `foo'. Let the best case: Alice provides a Guix "recipe", and all the material is stored in Github (let say). This software `foo' depends on both `bar' and `baz', one also in Github and the other one included in the Guix package tree. - It is easy for Bob to check out and experiment. Guix allows him to straightforwardly build the bit identical `foo' (all dependencies included). Nice!! Repeatability is there for free. - New features are added to `foo', `bar' and `baz'. All the codes evolve, especially the research ones. - Now, Joe is implementing the Alice's method; science means reproducible. And Joe would like to compare his implementation to the Alice one provided by `foo'. However, how ? The `foo' "ecosystem" has changed with the new features. Therefore, Joe has to navigate in the Git tree of the Guix "recipe" of `foo', `bar', `baz' to be able to produce the bit-identical `foo' used in the initial paper. I mean, it is what I understand to do, and it does not seem reasonable. My question is: does Guix provide any mechanism to build reproducible software over the time ? Last, `foo' and `bar' are stored in two Github repositories. And they should disappear. ( I am not talking if it is good or not to use github, right now, it just is used by many teams of researchers ) Could we used the Software Heritage initiative to maintain a kind of persistency ? https://www.softwareheritage.org I do not know if my wishes make any sense. All the best, -simon ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: persistent reproducibility ? 2017-03-21 11:15 persistent reproducibility ? zimoun @ 2017-03-21 14:49 ` Alex Sassmannshausen 2017-03-21 16:19 ` Ludovic Courtès 1 sibling, 0 replies; 14+ messages in thread From: Alex Sassmannshausen @ 2017-03-21 14:49 UTC (permalink / raw) To: zimoun; +Cc: help-guix Hi Simon, zimoun writes: > Hi Guixers, > > > Disclaimer: my background is about numerical analysis (PDE, Krylov > solver, etc.) and I am currently working in some Core Facility about > biology and bioinformatic (NGS pipelines, flow cytometry clustering, > etc.). Orthogonal communities but same issues. :-) Welcome! Sounds very exciting :-) > The typical research workflow is: > > - Alice proposes new method and/or algorithm, publishes a paper and > illustrates that by the software `foo'. Let the best case: Alice > provides a Guix "recipe", and all the material is stored in Github > (let say). This software `foo' depends on both `bar' and `baz', one > also in Github and the other one included in the Guix package tree. > > - It is easy for Bob to check out and experiment. Guix allows him to > straightforwardly build the bit identical `foo' (all dependencies > included). Nice!! Repeatability is there for free. > > - New features are added to `foo', `bar' and `baz'. All the codes > evolve, especially the research ones. > > - Now, Joe is implementing the Alice's method; science means > reproducible. And Joe would like to compare his implementation to the > Alice one provided by `foo'. However, how ? The `foo' "ecosystem" has > changed with the new features. Therefore, Joe has to navigate in the > Git tree of the Guix "recipe" of `foo', `bar', `baz' to be able to > produce the bit-identical `foo' used in the initial paper. I mean, it > is what I understand to do, and it does not seem reasonable. There are others on here who may have more in-depth experience of precisely this issue, but here's my impression of how this would work. Alice's package recipe would specify the exact version of itself, and also of `bar` and `baz` that that exactly version required. As a result, when software, and also package recipe versions evolve, all Joe would have to do is locate the historic package recipe of `foo` that Joe is trying to reproduce. That package recipe would then point to the historic versions of `bar` & `baz`, so Joe would not have to worry about that. > My question is: does Guix provide any mechanism to build reproducible > software over the time ? Guix just parses the recipe. The recipe details the exact versions of software required. If the exact recipe of a given time is not available then Guix cannot help that (though if the recipe is added to Guix then it will be available in Guix's repository). If the repository hosting the exact version of the software that the recipe specifies is no longer available then Guix cannot help. > Last, `foo' and `bar' are stored in two Github repositories. And they > should disappear. > ( I am not talking if it is good or not to use github, right now, it > just is used by many teams of researchers ) > > Could we used the Software Heritage initiative to maintain a kind of > persistency ? > https://www.softwareheritage.org There was some conversation about this at some point. Again others may know more about this. It certainly seems a nice means of ensuring software remains available… > I do not know if my wishes make any sense. I think they definitely do :-) HTH, Alex ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: persistent reproducibility ? 2017-03-21 11:15 persistent reproducibility ? zimoun 2017-03-21 14:49 ` Alex Sassmannshausen @ 2017-03-21 16:19 ` Ludovic Courtès 2017-03-22 17:39 ` zimoun 1 sibling, 1 reply; 14+ messages in thread From: Ludovic Courtès @ 2017-03-21 16:19 UTC (permalink / raw) To: zimoun; +Cc: help-guix Hello! zimoun <zimon.toutoune@gmail.com> skribis: > The typical research workflow is: > > - Alice proposes new method and/or algorithm, publishes a paper and > illustrates that by the software `foo'. Let the best case: Alice > provides a Guix "recipe", and all the material is stored in Github > (let say). This software `foo' depends on both `bar' and `baz', one > also in Github and the other one included in the Guix package tree. > > - It is easy for Bob to check out and experiment. Guix allows him to > straightforwardly build the bit identical `foo' (all dependencies > included). Nice!! Repeatability is there for free. > > - New features are added to `foo', `bar' and `baz'. All the codes > evolve, especially the research ones. > > - Now, Joe is implementing the Alice's method; science means > reproducible. And Joe would like to compare his implementation to the > Alice one provided by `foo'. However, how ? The `foo' "ecosystem" has > changed with the new features. Therefore, Joe has to navigate in the > Git tree of the Guix "recipe" of `foo', `bar', `baz' to be able to > produce the bit-identical `foo' used in the initial paper. I mean, it > is what I understand to do, and it does not seem reasonable. > > > My question is: does Guix provide any mechanism to build reproducible > software over the time ? To add to what Alex wrote, yes it’s possible, though there are UI gaps that we’ll be filling. If you do a checkout of the Guix commit that Alice mentioned in the paper, you can build the exact same software as Alice. ‘guix pull’ allows you to specify the Guix commit you’d like to use, but it’s not that convenient that it’s something we’d like to improve. Ricardo and I wrote about the kind of workflows you describe in <https://arxiv.org/abs/1506.02822>. I hope you’ll find it interesting! > Last, `foo' and `bar' are stored in two Github repositories. And they > should disappear. > ( I am not talking if it is good or not to use github, right now, it > just is used by many teams of researchers ) > > Could we used the Software Heritage initiative to maintain a kind of > persistency ? > https://www.softwareheritage.org Definitely! Software Heritage does not expose it yet, but when it does, we can add it as a fallback mirror in our (guix git-download) module. (I’ve discussed our use case with the Software Heritage folks a few times, so they’re aware of it. ;-)) I think the big picture is (where each arrow means “depends on”): repro science -> repro software environments -> stable archive Things like ReScience can do the first step, Guix can do the second one, and Software Heritage does the third one. Thanks for sharing your use case! Ludo’. ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: persistent reproducibility ? 2017-03-21 16:19 ` Ludovic Courtès @ 2017-03-22 17:39 ` zimoun 2017-03-23 8:44 ` Alex Sassmannshausen ` (2 more replies) 0 siblings, 3 replies; 14+ messages in thread From: zimoun @ 2017-03-22 17:39 UTC (permalink / raw) To: Ludovic Courtès; +Cc: help-guix Hi! On 21 March 2017 at 17:19, Ludovic Courtès <ludo@gnu.org> wrote: > Hello! > > zimoun <zimon.toutoune@gmail.com> skribis: > >> The typical research workflow is: >> >> - Alice proposes new method and/or algorithm, publishes a paper and >> illustrates that by the software `foo'. Let the best case: Alice >> provides a Guix "recipe", and all the material is stored in Github >> (let say). This software `foo' depends on both `bar' and `baz', one >> also in Github and the other one included in the Guix package tree. >> >> - It is easy for Bob to check out and experiment. Guix allows him to >> straightforwardly build the bit identical `foo' (all dependencies >> included). Nice!! Repeatability is there for free. >> >> - New features are added to `foo', `bar' and `baz'. All the codes >> evolve, especially the research ones. >> >> - Now, Joe is implementing the Alice's method; science means >> reproducible. And Joe would like to compare his implementation to the >> Alice one provided by `foo'. However, how ? The `foo' "ecosystem" has >> changed with the new features. Therefore, Joe has to navigate in the >> Git tree of the Guix "recipe" of `foo', `bar', `baz' to be able to >> produce the bit-identical `foo' used in the initial paper. I mean, it >> is what I understand to do, and it does not seem reasonable. >> >> >> My question is: does Guix provide any mechanism to build reproducible >> software over the time ? > > To add to what Alex wrote, yes it’s possible, though there are UI gaps > that we’ll be filling. If you do a checkout of the Guix commit that > Alice mentioned in the paper, you can build the exact same software as > Alice. ‘guix pull’ allows you to specify the Guix commit you’d like to > use, but it’s not that convenient that it’s something we’d like to > improve. Thank you for your quick answer. I was not aware of the `guix pull' commit specification. I am going to try. If I understand well your both explanations, my question overlaps the current discussion about Channels. I mean, in the same way that people currently include in their project conda shell files or spack python files or whatever in this flavour, they will replace by Guile ones (even if lisp is hard to sell ;-), and there is questions about how to glue these channels. One of the issues is that the Guix packages tree will never include some softwares, even if they are open source. Because the authors apply weird licences or non-GNU compliant licences, or simply because authors are not so motivated to push. Even if I totally agree with the paragraph about Proprietary Softwares in your cited paper, it is just a fact from my humble opinion. Therefore, what should be the "standard" way to manipulate against history version external and decentralised packages ? and guix repo packages too ? Well, if I understand your both answers, the correct process should be: Alice publishes a paper containing the exact version (commit hash or revision number or origin hash) of both the source tree and the recipe tree, and their both uri location, and then, Joe "just" needs to check out each (manually for now or possibly by nice UI glue). From my current knowledge about Guix, considering two moments t1 and t2, separated by let say several month, a way to build at t2 the bit-identical `foo' of t1 should to provide at t1 a sort of manual meta-package pointing to exacts version through `origin' and `git-fetch' (imaging that `hg-fetch' and `svn-fetch' would be added, or not!). If one piece of this information about the exact versions is not pointed out at t1, it appears to me almost impossible to insure the building of the bit-identical at t2. If I understand well, this way is what you describe Alex, right ? Another option should be: Guix add some meta-data in profile (kind of manifest) that automatically tracks this information at t1. Then using this profile at t2, it eases the building process, and it insures that nothing had been forgotten. Still from what I understand, Guix adds meta-data (roughly all the propagated hashes) that allow to verify the bit-identity. However, I am not sure if it does not lack meta-data included in profile or elsewhere which describe the "state" at t (e.g., exact version by commit specification) and then allow to revert from t+1 to t. Maybe `archive' or `pack' already do the job ? Something able to pack at the source level all the necessary materials to reproduce from scratch a bit-identical environment. > > Ricardo and I wrote about the kind of workflows you describe in > <https://arxiv.org/abs/1506.02822>. I hope you’ll find it interesting! Yes! I found it really interesting! :-) I already read it long time ago and discovered spack and then how HiePACS uses it. Thanks! Anyway! I am reading again with new glasses. > >> Last, `foo' and `bar' are stored in two Github repositories. And they >> should disappear. >> ( I am not talking if it is good or not to use github, right now, it >> just is used by many teams of researchers ) >> >> Could we used the Software Heritage initiative to maintain a kind of >> persistency ? >> https://www.softwareheritage.org > > Definitely! Software Heritage does not expose it yet, but when it does, > we can add it as a fallback mirror in our (guix git-download) module. > (I’ve discussed our use case with the Software Heritage folks a few > times, so they’re aware of it. ;-)) Really nice !! > > I think the big picture is (where each arrow means “depends on”): > > repro science -> repro software environments -> stable archive > > Things like ReScience can do the first step, Guix can do the second one, > and Software Heritage does the third one. "There is a long way to go, but the road is Free..." ;-) > > Thanks for sharing your use case! Thanks for all this positive energy :-) and thanks to give me a reason to learn scheme/guile because guix -- simon > > Ludo’. ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: persistent reproducibility ? 2017-03-22 17:39 ` zimoun @ 2017-03-23 8:44 ` Alex Sassmannshausen 2017-03-23 15:33 ` zimoun 2017-03-23 12:32 ` Ricardo Wurmus 2017-03-24 5:39 ` Chris Marusich 2 siblings, 1 reply; 14+ messages in thread From: Alex Sassmannshausen @ 2017-03-23 8:44 UTC (permalink / raw) To: zimoun; +Cc: help-guix Hi Zimoun, zimoun writes: > Hi! > > On 21 March 2017 at 17:19, Ludovic Courtès <ludo@gnu.org> wrote: >> Hello! >> >> zimoun <zimon.toutoune@gmail.com> skribis: >> >>> The typical research workflow is: >>> >>> - Alice proposes new method and/or algorithm, publishes a paper and >>> illustrates that by the software `foo'. Let the best case: Alice >>> provides a Guix "recipe", and all the material is stored in Github >>> (let say). This software `foo' depends on both `bar' and `baz', one >>> also in Github and the other one included in the Guix package tree. >>> >>> - It is easy for Bob to check out and experiment. Guix allows him to >>> straightforwardly build the bit identical `foo' (all dependencies >>> included). Nice!! Repeatability is there for free. >>> >>> - New features are added to `foo', `bar' and `baz'. All the codes >>> evolve, especially the research ones. >>> >>> - Now, Joe is implementing the Alice's method; science means >>> reproducible. And Joe would like to compare his implementation to the >>> Alice one provided by `foo'. However, how ? The `foo' "ecosystem" has >>> changed with the new features. Therefore, Joe has to navigate in the >>> Git tree of the Guix "recipe" of `foo', `bar', `baz' to be able to >>> produce the bit-identical `foo' used in the initial paper. I mean, it >>> is what I understand to do, and it does not seem reasonable. >>> >>> >>> My question is: does Guix provide any mechanism to build reproducible >>> software over the time ? >> >> To add to what Alex wrote, yes it’s possible, though there are UI gaps >> that we’ll be filling. If you do a checkout of the Guix commit that >> Alice mentioned in the paper, you can build the exact same software as >> Alice. ‘guix pull’ allows you to specify the Guix commit you’d like to >> use, but it’s not that convenient that it’s something we’d like to >> improve. > > Thank you for your quick answer. > > I was not aware of the `guix pull' commit specification. I am going to try. > > If I understand well your both explanations, my question overlaps the > current discussion about Channels. I mean, in the same way that people > currently include in their project conda shell files or spack python > files or whatever in this flavour, they will replace by Guile ones > (even if lisp is hard to sell ;-), and there is questions about how to > glue these channels. > > One of the issues is that the Guix packages tree will never include > some softwares, even if they are open source. Because the authors > apply weird licences or non-GNU compliant licences, or simply because > authors are not so motivated to push. Even if I totally agree with the > paragraph about Proprietary Softwares in your cited paper, it is just > a fact from my humble opinion. > > > Therefore, what should be the "standard" way to manipulate against > history version external and decentralised packages ? and guix repo > packages too ? > > > Well, if I understand your both answers, the correct process should > be: Alice publishes a paper containing the exact version (commit hash > or revision number or origin hash) of both the source tree and the > recipe tree, and their both uri location, and then, Joe "just" needs > to check out each (manually for now or possibly by nice UI glue). > > From my current knowledge about Guix, considering two moments t1 and > t2, separated by let say several month, a way to build at t2 the > bit-identical `foo' of t1 should to provide at t1 a sort of manual > meta-package pointing to exacts version through `origin' and > `git-fetch' (imaging that `hg-fetch' and `svn-fetch' would be added, > or not!). If one piece of this information about the exact versions is > not pointed out at t1, it appears to me almost impossible to insure > the building of the bit-identical at t2. > If I understand well, this way is what you describe Alex, right ? Right — so I think I was a bit unclear in my first email. A guix package recipe, as you point, points to an exact revision/version of software (either through a version spec, or through a commit hash). The package's dependencies are specified as pure scheme variables that in turn resolve to package recipes. These package recipes in turn specify the exact revision of the dependencies. So for `foo` to be fully reproducible at t1 you require the package tree that corresponds to `foo`, `bar` & `baz` at t1. This tree would carry all the meta information related to `foo` and it's exact dependencies. I guess you may also need the correct version of Guix. I'm not sure how that affects reproducibility, so I won't comment further on that. In any case, if you have `foo`, `bar` & `baz` as part of a guix source tree, then the revision of that source tree at t1 should be sufficient to reproduce `foo` at t1. I guess this is where the discussion of channels, and whether to package the programs for Guix itself or somehow outside become relevant, but again, I don't feel super qualified to comment on this. All the meta-information required is carried by that revision. I wasn't sure what you meant by manual-metapackage, so I hope my answer provides you with the information you were looking for! I won't comment on the latter parts of your email, as it sounds like it's more related to fully reproducible environments rather than individual packages. Personally I use manifests, but that might not be good enough for your requirements. HTH, Alex ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: persistent reproducibility ? 2017-03-23 8:44 ` Alex Sassmannshausen @ 2017-03-23 15:33 ` zimoun 0 siblings, 0 replies; 14+ messages in thread From: zimoun @ 2017-03-23 15:33 UTC (permalink / raw) To: alex.sassmannshausen; +Cc: help-guix Hi! Thank you for commenting and helping me to go forward. > > A guix package recipe, as you point, points to an exact revision/version > of software (either through a version spec, or through a commit hash). > The package's dependencies are specified as pure scheme variables that > in turn resolve to package recipes. > > These package recipes in turn specify the exact revision of the > dependencies. > > So for `foo` to be fully reproducible at t1 you require the package tree > that corresponds to `foo`, `bar` & `baz` at t1. This tree would carry > all the meta information related to `foo` and it's exact dependencies. I am not sure to understand, neither to be able to catch this exact revision of the dependencies. Let consider the example of the package Gmsh (I know a bit how it is developed): https://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/maths.scm?id=v0.12.0-2125-g283f04b#n1097 Here, `gmsh' depends on `fltk', `gfortran', `lapack' etc. If the commit hash of the Guix source tree is provided and known, then I agree. It is possible to restore (build) the bit-identical `gmsh' at any time. (assuming that the upstream is still available, which is another issue and Ludo pointed out that some fallbacks by the Software Heritage are in the pipes \o/) At this point, I agree that on the paper, all the pieces are there and it "just" lacks some UI glue. And this point is a BIG step, compared to other solution on the market. Thanks to Nix thesis and to you Guixers to implement it in Guile. :-) Now, as researcher, Alice forks Gmsh to explore a proof-of-concept of her new algorithm. Upstream refuses to include, for whatever reason, or Alice refuses to push for whatever reason. Well this new algorithm will never be in the Guix source tree. Moreover, her new algorithm needs a specific patch of `lapack', and this variant is too much specific to be included in the Guix source tree. At t1, Alice publishes all the materials, forked Gmsh commit hash and its "reproducible hash", also the Guix source tree commit hash, explained her `lapack' trick from the paper but she forgets to precisely point the commit hash. Then, it is not possible to bit-identically reproduce the forked Gmsh of the paper. Ok, Guix cannot fix automagically what people does not do correctly. For sure! But, should be somehow possible (and useful?) to add something to prevent that ? I have in mind something in the flavor of `guix pack' at the source level. When Alice releases her paper, she could release a binary pack of everything useful to be able to run and in the same time a source pack of everything useful to be able to build (over time). A kind of derivation ? (it is what I called meta-package, hum?) > > I guess you may also need the correct version of Guix. I'm not sure how > that affects reproducibility, so I won't comment further on that. I agree that experiments need to be performed. > > In any case, if you have `foo`, `bar` & `baz` as part of a guix source > tree, then the revision of that source tree at t1 should be sufficient > to reproduce `foo` at t1. I agree. And my question is: how to do if one of them is not part of a guix source tree ? > > I guess this is where the discussion of channels, and whether to package > the programs for Guix itself or somehow outside become relevant, but > again, I don't feel super qualified to comment on this. I will browse the discussion about channels. Thank you for your comment. My feeling is that a feature lacks to Guix to deal with this situation. Well, I will explore more details and see if I am able to use your advices. > > All the meta-information required is carried by that revision. > > I wasn't sure what you meant by manual-metapackage, so I hope my answer > provides you with the information you were looking for! Writing by hand a package that provides all the necessary meta-data to build bit-identical at any time. If I install Guix, and for example I do "guix package -i foo -p pro". Then, over months, I do operations as: create profiles, install, remove, garbage collect, refresh, pull, etc. Now, month after, I am not sure to be able to bit-identically build the previous `foo' package. Maybe I miss a point. Especially about `manifest'. > > I won't comment on the latter parts of your email, as it sounds like > it's more related to fully reproducible environments rather than > individual packages. Personally I use manifests, but that might not be > good enough for your requirements. Thank you for your advices. Time to play with keyboards :-) All the best ! -- simon ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: persistent reproducibility ? 2017-03-22 17:39 ` zimoun 2017-03-23 8:44 ` Alex Sassmannshausen @ 2017-03-23 12:32 ` Ricardo Wurmus 2017-03-23 16:46 ` zimoun 2017-03-24 5:39 ` Chris Marusich 2 siblings, 1 reply; 14+ messages in thread From: Ricardo Wurmus @ 2017-03-23 12:32 UTC (permalink / raw) To: zimoun; +Cc: help-guix zimoun <zimon.toutoune@gmail.com> writes: > One of the issues is that the Guix packages tree will never include > some softwares, even if they are open source. Because the authors > apply weird licences or non-GNU compliant licences, or simply because > authors are not so motivated to push. Even if I totally agree with the > paragraph about Proprietary Softwares in your cited paper, it is just > a fact from my humble opinion. If you mean “open source” in the sense of “using a license that is certified by the Open Source Initiative” then that software is probably Free Software. There is no such thing as GNU compliance in licenses. We do however follow the GNU FDSG (Free System Distribution Guidelines), which may result in some software to be excluded or modified in rare cases. (One example is “Shogun”, which we modify to remove included non-free software.) > Therefore, what should be the "standard" way to manipulate against > history version external and decentralised packages ? and guix repo > packages too ? > > > Well, if I understand your both answers, the correct process should > be: Alice publishes a paper containing the exact version (commit hash > or revision number or origin hash) of both the source tree and the > recipe tree, and their both uri location, and then, Joe "just" needs > to check out each (manually for now or possibly by nice UI glue). It would be sufficient to provide two things: a manifest and the Guix version (“git -C /path/to/guix.git describe”). If additional package repositories are used (such as guix-bimsb for my institute), their versions have to be recorded as well. Joe would then check out Guix and any additional package repositories at the specified versions and then instantiate the manifest. Provided that all source tarballs are still obtainable (either directly or through a mirror) and the builds are in fact bit-reproducible Joe would end up with the same software environment that Alice fully specified in the paper. -- Ricardo GPG: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAC https://elephly.net ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: persistent reproducibility ? 2017-03-23 12:32 ` Ricardo Wurmus @ 2017-03-23 16:46 ` zimoun 2017-03-24 15:45 ` Ludovic Courtès 0 siblings, 1 reply; 14+ messages in thread From: zimoun @ 2017-03-23 16:46 UTC (permalink / raw) To: Ricardo Wurmus; +Cc: help-guix Hi! Thank you the details. >> One of the issues is that the Guix packages tree will never include >> some softwares, even if they are open source. Because the authors >> apply weird licences or non-GNU compliant licences, or simply because >> authors are not so motivated to push. Even if I totally agree with the >> paragraph about Proprietary Softwares in your cited paper, it is just >> a fact from my humble opinion. > > If you mean “open source” in the sense of “using a license that is > certified by the Open Source Initiative” then that software is probably > Free Software. There is no such thing as GNU compliance in licenses. I mean "open source" any software publicly released with publicly accessible source. It is large. ;-) And I totally agree with the condition to be included or not in the Guix infrastructure. My point is that a lot of softwares released in scientific world will never reach such condition. It is sad and I think all people here are trying to change by convincing the authors. But, it is a pragmatic fact. Hum? I am not sure to get your point about GNU compliance. For example, https://libreplanet.org/wiki/List_of_software_that_does_not_respect_the_Free_System_Distribution_Guidelines `apt' and `debian-installer' illustrate the well-known and famous debate between RMS and Debian. :-) > > We do however follow the GNU FDSG (Free System Distribution Guidelines), > which may result in some software to be excluded or modified in rare > cases. (One example is “Shogun”, which we modify to remove included > non-free software.) Yes, the GNU FDSG defines "free" (as in speech). And there is "open source" softwares which are not included in this definition (for the good, for the bad, I am not arguing). https://www.gnu.org/licenses/license-list.html#NonFreeSoftwareLicenses For example, some versions of Scilab (clone of Matlab) with a "weird" license (CeCILL-2). For the record, let indicate an historical point: https://www.gnu.org/licenses/bsd.html :-) I am not arguing about "bad" practices. I am just pointing that the situation of scientific softwares is often really complicated. Because they often glue a lot of different piece of softwares, e.g., https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=617931 Note that, in the same way that non-free parts are removed from `shogun', the guix package `gmsh' removes METIS non-free part (which is "open source"). Hope that this clarifies my point and feeds argument about channel. :-) > >> Therefore, what should be the "standard" way to manipulate against >> history version external and decentralised packages ? and guix repo >> packages too ? >> >> >> Well, if I understand your both answers, the correct process should >> be: Alice publishes a paper containing the exact version (commit hash >> or revision number or origin hash) of both the source tree and the >> recipe tree, and their both uri location, and then, Joe "just" needs >> to check out each (manually for now or possibly by nice UI glue). > > It would be sufficient to provide two things: a manifest and the Guix > version (“git -C /path/to/guix.git describe”). If additional package > repositories are used (such as guix-bimsb for my institute), their > versions have to be recorded as well. Thank you to clarify the way. If I understand well, additional meta-data should be manually provided by Alice. Extract a manifest is some UI glue, even, it is perhaps already possible. However, the Guix version do not appear to me straightforward. Does this information should be included when installing a package ? > > Joe would then check out Guix and any additional package repositories at > the specified versions and then instantiate the manifest. Provided that > all source tarballs are still obtainable (either directly or through a > mirror) and the builds are in fact bit-reproducible Joe would end up > with the same software environment that Alice fully specified in the > paper. Should be possible to extract all the required information from a profile with a kind of `guix pack' at the source level ? This should be useful ? or not at all and I am wishing a twisted process ? Thank you for your insights. And for sharing the MDC's configuration! All the best. -- simon > > -- > Ricardo > > GPG: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAC > https://elephly.net > ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: persistent reproducibility ? 2017-03-23 16:46 ` zimoun @ 2017-03-24 15:45 ` Ludovic Courtès 2017-03-25 13:07 ` zimoun 0 siblings, 1 reply; 14+ messages in thread From: Ludovic Courtès @ 2017-03-24 15:45 UTC (permalink / raw) To: zimoun; +Cc: help-guix Hi! zimoun <zimon.toutoune@gmail.com> skribis: >>> One of the issues is that the Guix packages tree will never include >>> some softwares, even if they are open source. Because the authors >>> apply weird licences or non-GNU compliant licences, or simply because >>> authors are not so motivated to push. Even if I totally agree with the >>> paragraph about Proprietary Softwares in your cited paper, it is just >>> a fact from my humble opinion. >> >> If you mean “open source” in the sense of “using a license that is >> certified by the Open Source Initiative” then that software is probably >> Free Software. There is no such thing as GNU compliance in licenses. > > I mean "open source" any software publicly released with publicly > accessible source. It is large. ;-) “Open source” as defined by the OSI means more that just “accessible source”: https://opensource.org/definition In effect it requires the 4 freedoms: https://www.gnu.org/philosophy/free-sw.html Now, it is true that there’s software out there with “accessible source” that is neither free software nor open source, especially on github.com since GitHub makes it easy to publish code without specifying a license. > My point is that a lot of softwares released in scientific world will > never reach such condition. It is sad and I think all people here are > trying to change by convincing the authors. But, it is a pragmatic > fact. I’m not sure. Of course we’d have to be more specific than “a lot of” ;-), but I also see “a lot of” scientific software that is free; in fact, I haven’t seen much non-free scientific software produced in the CS research institutes here in France. >> We do however follow the GNU FDSG (Free System Distribution Guidelines), >> which may result in some software to be excluded or modified in rare >> cases. (One example is “Shogun”, which we modify to remove included >> non-free software.) > > Yes, the GNU FDSG defines "free" (as in speech). And there is "open > source" softwares which are not included in this definition (for the > good, for the bad, I am not arguing). > https://www.gnu.org/licenses/license-list.html#NonFreeSoftwareLicenses > For example, some versions of Scilab (clone of Matlab) with a "weird" > license (CeCILL-2). The CeCILL licenses are all free software licenses, so CeCILL-licensed software is welcome in Guix! Thanks, Ludo’. ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: persistent reproducibility ? 2017-03-24 15:45 ` Ludovic Courtès @ 2017-03-25 13:07 ` zimoun 0 siblings, 0 replies; 14+ messages in thread From: zimoun @ 2017-03-25 13:07 UTC (permalink / raw) To: Ludovic Courtès; +Cc: help-guix Hi ! > concerning my initial question Thanks Chris! It is exactly the `guix pack' at source level that I was looking for. I am playing around. I have still issues when redirecting the `export', e.g., `guix archive --export hello' works, but not `guix archive --export hello > hello.nar' raising: `guix archive: error: corrupt input while restoring archive from #<closed: file 0>' Well, it is another topic. > concerning license relative stuff I am on the same wavelength and I almost agree. My worries seems edge cases and I am maybe applying an overstatement typical from south french people ;-) Thanks to Guix community to share their positive energy. All the best, -simon On 24 March 2017 at 16:45, Ludovic Courtès <ludo@gnu.org> wrote: > Hi! > > zimoun <zimon.toutoune@gmail.com> skribis: > >>>> One of the issues is that the Guix packages tree will never include >>>> some softwares, even if they are open source. Because the authors >>>> apply weird licences or non-GNU compliant licences, or simply because >>>> authors are not so motivated to push. Even if I totally agree with the >>>> paragraph about Proprietary Softwares in your cited paper, it is just >>>> a fact from my humble opinion. >>> >>> If you mean “open source” in the sense of “using a license that is >>> certified by the Open Source Initiative” then that software is probably >>> Free Software. There is no such thing as GNU compliance in licenses. >> >> I mean "open source" any software publicly released with publicly >> accessible source. It is large. ;-) > > “Open source” as defined by the OSI means more that just “accessible > source”: > > https://opensource.org/definition > > In effect it requires the 4 freedoms: > > https://www.gnu.org/philosophy/free-sw.html > > Now, it is true that there’s software out there with “accessible source” > that is neither free software nor open source, especially on github.com > since GitHub makes it easy to publish code without specifying a license. > >> My point is that a lot of softwares released in scientific world will >> never reach such condition. It is sad and I think all people here are >> trying to change by convincing the authors. But, it is a pragmatic >> fact. > > I’m not sure. Of course we’d have to be more specific than “a lot of” > ;-), but I also see “a lot of” scientific software that is free; in > fact, I haven’t seen much non-free scientific software produced in the > CS research institutes here in France. > >>> We do however follow the GNU FDSG (Free System Distribution Guidelines), >>> which may result in some software to be excluded or modified in rare >>> cases. (One example is “Shogun”, which we modify to remove included >>> non-free software.) >> >> Yes, the GNU FDSG defines "free" (as in speech). And there is "open >> source" softwares which are not included in this definition (for the >> good, for the bad, I am not arguing). >> https://www.gnu.org/licenses/license-list.html#NonFreeSoftwareLicenses >> For example, some versions of Scilab (clone of Matlab) with a "weird" >> license (CeCILL-2). > > The CeCILL licenses are all free software licenses, so CeCILL-licensed > software is welcome in Guix! > > Thanks, > Ludo’. ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: persistent reproducibility ? 2017-03-22 17:39 ` zimoun 2017-03-23 8:44 ` Alex Sassmannshausen 2017-03-23 12:32 ` Ricardo Wurmus @ 2017-03-24 5:39 ` Chris Marusich 2017-03-24 12:05 ` Quiliro 2 siblings, 1 reply; 14+ messages in thread From: Chris Marusich @ 2017-03-24 5:39 UTC (permalink / raw) To: zimoun; +Cc: help-guix zimoun <zimon.toutoune@gmail.com> writes: > Maybe `archive' or `pack' already do the job ? Something able to pack > at the source level all the necessary materials to reproduce from > scratch a bit-identical environment. What about using 'guix archive' in combination with the 'guix build --sources=transitive' command? The manual explains ((guix) Additional Build Options), the "--sources=transitive" option means: "Build the source derivations of all packages, as well of all transitive inputs to the packages. This can be used e.g. to prefetch package source for later offline building." Here's an example for the GNU Hello package: --8<---------------cut here---------------start------------->8--- $ guix build --sources=transitive hello /gnu/store/hbdalsf5lpf01x4dcknwx6xbn6n5km6k-hello-2.10.tar.gz /gnu/store/cz1b1hn763zppafmlxsrlj7fnxa8zcna-tar-1.29.tar.xz /gnu/store/f63c6mjqqai6xwypx5qslvqc2d302yn5-gzip-1.8.tar.xz /gnu/store/bjpalrv9f49d2k40p7ga0y6dwdys9w5j-bzip2-1.0.6.tar.gz /gnu/store/1pcksb758j9v7srain0k3fqp57h9gwmj-xz-5.2.2.tar.gz /gnu/store/r56rrdk6h0zwb1d2y28q3bk71n73z84m-file-5.28.tar.gz /gnu/store/rbaa5s5jpy91npas89q3p8yasrc6mr1h-diffutils-3.5.tar.xz /gnu/store/iaigwb3k7fg0v8zrwlb4yhrdsgin6ir7-patch-2.7.5.tar.xz /gnu/store/g0mffc9c0nxbwf0a6j0akh0dvkd2sn6w-sed-4.2.2.tar.xz /gnu/store/76q5cqkqrx7paxhq27m3izag07n1vjs4-findutils-4.6.0.tar.xz /gnu/store/vgs8flhpxbkgx0hx54hnihpai7lma72j-gawk-4.1.4.tar.xz /gnu/store/5q4yicgzxrv13gq4rai6h07s7xp75ws6-grep-2.25.tar.xz /gnu/store/8dqf59312llnmln89ccas1saq0z4qwi7-coreutils-8.25.tar.xz /gnu/store/2g7nf5h6hnsz1sharnqxfjz9m6k7224z-make-4.2.1.tar.xz /gnu/store/xwanyg9972rpznx89ja38v01c8483392-bash-4.4.tar.gz /gnu/store/iv2nz8r001lpric4kr2xq3459zwl8051-binutils-2.27.tar.xz /gnu/store/z2s847iq14psryslylm1x4k71b0vx4ng-gcc-4.9.4.tar.xz /gnu/store/m9jym12mhczjrslpycwi6ipw57v849pa-gmp-6.0.0a.tar.xz /gnu/store/xg7ir7g2yvxj9ixmgy2kfds690h3f6gg-mpfr-3.1.4.tar.xz /gnu/store/y9akh452n3p4w2v631nj0injx7y0d68x-mpc-1.0.3.tar.gz /gnu/store/z8mm3izh1hgllm6mlqzycsrx5nfkvhxi-glibc-2.24.tar.xz /gnu/store/51f2f2ck80migsfr2hwy4ds30966lifi-linux-libre-4.4.18-gnu.tar.xz --8<---------------cut here---------------end--------------->8--- You could archive those sources for later importation into the store on another system. For example: guix archive --export \ $(guix build --sources=transitive hello) > hello-transitive-source.nar If you import that archive (with 'guix archive --import') and build it using the same version of the package definitions, theoretically it should build even if the sources are no longer available for download from their specified origin locations. You could even do more. By using 'guix gc', you could also include all of the store items which are required to run GNU Hello: --8<---------------cut here---------------start------------->8--- $ guix gc --requisites $(guix build hello) /gnu/store/rvgmixpmsq5lqr9qflhkm70kg7a4rys2-bash-static-4.4.0 /gnu/store/iwgi9001dmmihrjg4rqhd6pa6788prjw-glibc-2.24 /gnu/store/cdi08kw7r6r684w8mk0xq0dkgpjhfpmd-gcc-4.9.4-lib /gnu/store/rvs42awwwby7pq3j0znglmz3vyznvbh1-hello-2.10 --8<---------------cut here---------------end--------------->8--- Combine those two commands, and you should be able to export an archive which contains all the files necessary to run GNU Hello, in addition to all the sources necessary to build GNU Hello reproducibly (provided that GNU Hello builds reproducibly, and provided that you build it using the same version of the package definitions): guix archive --export \ $(guix gc --requisites $(guix build hello)) \ $(guix build --sources=transitive hello) > hello-requisites-and-sources.nar You would also have to trust the key that was used to sign the archive in order to import it (see 'guix archive --authorize'). By the way, in case you were wondering, one reason why we don't put the sources for every package into the Guix source tree is because that would make the Guix source tree prohibitively large. The transitive closure of sources for GNU Hello alone weighs in at 1750 MiB (wow!) when uncompressed. And that's only for those specific versions of the source files. -- Chris ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: persistent reproducibility ? 2017-03-24 5:39 ` Chris Marusich @ 2017-03-24 12:05 ` Quiliro 2017-03-26 9:39 ` Chris Marusich 0 siblings, 1 reply; 14+ messages in thread From: Quiliro @ 2017-03-24 12:05 UTC (permalink / raw) To: help-guix El Thu, 23 Mar 2017 22:39:52 -0700 Chris Marusich <cmmarusich@gmail.com> escribió: > By the way, in case you were wondering, one reason why we don't put > the sources for every package into the Guix source tree is because > that would make the Guix source tree prohibitively large. The > transitive closure of sources for GNU Hello alone weighs in at 1750 > MiB (wow!) when uncompressed. And that's only for those specific > versions of the source files. How much space would the sources for every package in compressed format take? -- Example of rude top posting: A: Because it messes up the order in which people normally read text. Q: Why is top-posting such a bad thing? A: Top-posting. Q: What is the most annoying thing in e-mail? A: No. Q: Should I leave quotations after my reply? Saluton, Quiliro 0987631031 ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: persistent reproducibility ? 2017-03-24 12:05 ` Quiliro @ 2017-03-26 9:39 ` Chris Marusich 2017-03-26 15:03 ` Quiliro 0 siblings, 1 reply; 14+ messages in thread From: Chris Marusich @ 2017-03-26 9:39 UTC (permalink / raw) To: Quiliro; +Cc: help-guix [-- Attachment #1: Type: text/plain, Size: 3002 bytes --] Quiliro <quiliro@riseup.net> writes: > El Thu, 23 Mar 2017 22:39:52 -0700 > Chris Marusich <cmmarusich@gmail.com> escribió: > >> By the way, in case you were wondering, one reason why we don't put >> the sources for every package into the Guix source tree is because >> that would make the Guix source tree prohibitively large. The >> transitive closure of sources for GNU Hello alone weighs in at 1750 >> MiB (wow!) when uncompressed. And that's only for those specific >> versions of the source files. > > How much space would the sources for every package in compressed > format take? According to 'guix size', the total size of the compressed sources for hello is 213.3 MiB, which is still quite large: --8<---------------cut here---------------start------------->8--- $ guix size $(guix build --sources=transitive hello) store item total self /gnu/store/51f2f2ck80migsfr2hwy4ds30966lifi-linux-libre-4.4.18-gnu.tar.xz 81.1 81.1 38.0% /gnu/store/z2s847iq14psryslylm1x4k71b0vx4ng-gcc-4.9.4.tar.xz 66.3 66.3 31.1% /gnu/store/iv2nz8r001lpric4kr2xq3459zwl8051-binutils-2.27.tar.xz 18.9 18.9 8.9% /gnu/store/z8mm3izh1hgllm6mlqzycsrx5nfkvhxi-glibc-2.24.tar.xz 13.7 13.7 6.4% /gnu/store/xwanyg9972rpznx89ja38v01c8483392-bash-4.4.tar.gz 8.9 8.9 4.2% /gnu/store/8dqf59312llnmln89ccas1saq0z4qwi7-coreutils-8.25.tar.xz 5.5 5.5 2.6% /gnu/store/vgs8flhpxbkgx0hx54hnihpai7lma72j-gawk-4.1.4.tar.xz 2.3 2.3 1.1% /gnu/store/cz1b1hn763zppafmlxsrlj7fnxa8zcna-tar-1.29.tar.xz 1.9 1.9 0.9% /gnu/store/m9jym12mhczjrslpycwi6ipw57v849pa-gmp-6.0.0a.tar.xz 1.8 1.8 0.9% /gnu/store/76q5cqkqrx7paxhq27m3izag07n1vjs4-findutils-4.6.0.tar.xz 1.7 1.7 0.8% /gnu/store/1pcksb758j9v7srain0k3fqp57h9gwmj-xz-5.2.2.tar.gz 1.4 1.4 0.7% /gnu/store/rbaa5s5jpy91npas89q3p8yasrc6mr1h-diffutils-3.5.tar.xz 1.3 1.3 0.6% /gnu/store/5q4yicgzxrv13gq4rai6h07s7xp75ws6-grep-2.25.tar.xz 1.3 1.3 0.6% /gnu/store/xg7ir7g2yvxj9ixmgy2kfds690h3f6gg-mpfr-3.1.4.tar.xz 1.1 1.1 0.5% /gnu/store/2g7nf5h6hnsz1sharnqxfjz9m6k7224z-make-4.2.1.tar.xz 1.1 1.1 0.5% /gnu/store/g0mffc9c0nxbwf0a6j0akh0dvkd2sn6w-sed-4.2.2.tar.xz 0.8 0.8 0.4% /gnu/store/bjpalrv9f49d2k40p7ga0y6dwdys9w5j-bzip2-1.0.6.tar.gz 0.7 0.7 0.3% /gnu/store/r56rrdk6h0zwb1d2y28q3bk71n73z84m-file-5.28.tar.gz 0.7 0.7 0.3% /gnu/store/iaigwb3k7fg0v8zrwlb4yhrdsgin6ir7-patch-2.7.5.tar.xz 0.7 0.7 0.3% /gnu/store/f63c6mjqqai6xwypx5qslvqc2d302yn5-gzip-1.8.tar.xz 0.7 0.7 0.3% /gnu/store/hbdalsf5lpf01x4dcknwx6xbn6n5km6k-hello-2.10.tar.gz 0.7 0.7 0.3% /gnu/store/y9akh452n3p4w2v631nj0injx7y0d68x-mpc-1.0.3.tar.gz 0.6 0.6 0.3% total: 213.3 MiB --8<---------------cut here---------------end--------------->8--- -- Chris [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 832 bytes --] ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: persistent reproducibility ? 2017-03-26 9:39 ` Chris Marusich @ 2017-03-26 15:03 ` Quiliro 0 siblings, 0 replies; 14+ messages in thread From: Quiliro @ 2017-03-26 15:03 UTC (permalink / raw) To: help-guix El Sun, 26 Mar 2017 02:39:19 -0700 Chris Marusich <cmmarusich@gmail.com> escribió: > Quiliro <quiliro@riseup.net> writes: > > > El Thu, 23 Mar 2017 22:39:52 -0700 > > Chris Marusich <cmmarusich@gmail.com> escribió: > > > >> By the way, in case you were wondering, one reason why we don't put > >> the sources for every package into the Guix source tree is because > >> that would make the Guix source tree prohibitively large. The > >> transitive closure of sources for GNU Hello alone weighs in at 1750 > >> MiB (wow!) when uncompressed. And that's only for those specific > >> versions of the source files. > > > > How much space would the sources for every package in compressed > > format take? > > According to 'guix size', the total size of the compressed sources for > hello is 213.3 MiB, which is still quite large: Thank you very much for this example. I know hello requieres certain dependencies that make it very heavy to download them. Perhaps these will be part of the big weight of the whole. How much does the whole GuixSD package repository weigh? How can it be calculated without downloading everything? -- Example of rude top posting: A: Because it messes up the order in which people normally read text. Q: Why is top-posting such a bad thing? A: Top-posting. Q: What is the most annoying thing in e-mail? A: No. Q: Should I leave quotations after my reply? Saluton, Quiliro 0987631031 ^ permalink raw reply [flat|nested] 14+ messages in thread
end of thread, other threads:[~2017-03-26 15:03 UTC | newest] Thread overview: 14+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2017-03-21 11:15 persistent reproducibility ? zimoun 2017-03-21 14:49 ` Alex Sassmannshausen 2017-03-21 16:19 ` Ludovic Courtès 2017-03-22 17:39 ` zimoun 2017-03-23 8:44 ` Alex Sassmannshausen 2017-03-23 15:33 ` zimoun 2017-03-23 12:32 ` Ricardo Wurmus 2017-03-23 16:46 ` zimoun 2017-03-24 15:45 ` Ludovic Courtès 2017-03-25 13:07 ` zimoun 2017-03-24 5:39 ` Chris Marusich 2017-03-24 12:05 ` Quiliro 2017-03-26 9:39 ` Chris Marusich 2017-03-26 15:03 ` Quiliro
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).