From mboxrd@z Thu Jan 1 00:00:00 1970 From: zimoun Subject: Re: persistent reproducibility ? Date: Thu, 23 Mar 2017 16:33:07 +0100 Message-ID: References: <87shm6wqju.fsf@gnu.org> <87wpbg5qn4.fsf@gmail.com> Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:43999) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1cr4jj-0006VK-DF for help-guix@gnu.org; Thu, 23 Mar 2017 11:33:16 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1cr4ji-0003YY-89 for help-guix@gnu.org; Thu, 23 Mar 2017 11:33:15 -0400 In-Reply-To: <87wpbg5qn4.fsf@gmail.com> List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: help-guix-bounces+gcggh-help-guix=m.gmane.org@gnu.org Sender: "Help-Guix" To: alex.sassmannshausen@gmail.com Cc: help-guix@gnu.org Hi! Thank you for commenting and helping me to go forward. > > A guix package recipe, as you point, points to an exact revision/version > of software (either through a version spec, or through a commit hash). > The package's dependencies are specified as pure scheme variables that > in turn resolve to package recipes. > > These package recipes in turn specify the exact revision of the > dependencies. > > So for `foo` to be fully reproducible at t1 you require the package tree > that corresponds to `foo`, `bar` & `baz` at t1. This tree would carry > all the meta information related to `foo` and it's exact dependencies. I am not sure to understand, neither to be able to catch this exact revision of the dependencies. Let consider the example of the package Gmsh (I know a bit how it is developed): https://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/maths.scm?id=v0.12.0-2125-g283f04b#n1097 Here, `gmsh' depends on `fltk', `gfortran', `lapack' etc. If the commit hash of the Guix source tree is provided and known, then I agree. It is possible to restore (build) the bit-identical `gmsh' at any time. (assuming that the upstream is still available, which is another issue and Ludo pointed out that some fallbacks by the Software Heritage are in the pipes \o/) At this point, I agree that on the paper, all the pieces are there and it "just" lacks some UI glue. And this point is a BIG step, compared to other solution on the market. Thanks to Nix thesis and to you Guixers to implement it in Guile. :-) Now, as researcher, Alice forks Gmsh to explore a proof-of-concept of her new algorithm. Upstream refuses to include, for whatever reason, or Alice refuses to push for whatever reason. Well this new algorithm will never be in the Guix source tree. Moreover, her new algorithm needs a specific patch of `lapack', and this variant is too much specific to be included in the Guix source tree. At t1, Alice publishes all the materials, forked Gmsh commit hash and its "reproducible hash", also the Guix source tree commit hash, explained her `lapack' trick from the paper but she forgets to precisely point the commit hash. Then, it is not possible to bit-identically reproduce the forked Gmsh of the paper. Ok, Guix cannot fix automagically what people does not do correctly. For sure! But, should be somehow possible (and useful?) to add something to prevent that ? I have in mind something in the flavor of `guix pack' at the source level. When Alice releases her paper, she could release a binary pack of everything useful to be able to run and in the same time a source pack of everything useful to be able to build (over time). A kind of derivation ? (it is what I called meta-package, hum?) > > I guess you may also need the correct version of Guix. I'm not sure how > that affects reproducibility, so I won't comment further on that. I agree that experiments need to be performed. > > In any case, if you have `foo`, `bar` & `baz` as part of a guix source > tree, then the revision of that source tree at t1 should be sufficient > to reproduce `foo` at t1. I agree. And my question is: how to do if one of them is not part of a guix source tree ? > > I guess this is where the discussion of channels, and whether to package > the programs for Guix itself or somehow outside become relevant, but > again, I don't feel super qualified to comment on this. I will browse the discussion about channels. Thank you for your comment. My feeling is that a feature lacks to Guix to deal with this situation. Well, I will explore more details and see if I am able to use your advices. > > All the meta-information required is carried by that revision. > > I wasn't sure what you meant by manual-metapackage, so I hope my answer > provides you with the information you were looking for! Writing by hand a package that provides all the necessary meta-data to build bit-identical at any time. If I install Guix, and for example I do "guix package -i foo -p pro". Then, over months, I do operations as: create profiles, install, remove, garbage collect, refresh, pull, etc. Now, month after, I am not sure to be able to bit-identically build the previous `foo' package. Maybe I miss a point. Especially about `manifest'. > > I won't comment on the latter parts of your email, as it sounds like > it's more related to fully reproducible environments rather than > individual packages. Personally I use manifests, but that might not be > good enough for your requirements. Thank you for your advices. Time to play with keyboards :-) All the best ! -- simon