From mboxrd@z Thu Jan 1 00:00:00 1970 From: Konrad Hinsen Subject: Re: Use guix to distribute data & reproducible (data) science Date: Mon, 12 Feb 2018 12:46:44 +0100 Message-ID: References: <365e13248634ac1e26cf6678611d550d@hypermove.net> <87mv0ixf07.fsf@gnu.org> <1cb709d0-b282-192c-ce1d-20fbff43430e@fastmail.net> Mime-Version: 1.0 Content-Type: text/plain Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:60618) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1elCZR-0001I6-G4 for guix-devel@gnu.org; Mon, 12 Feb 2018 06:46:54 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1elCZO-00080X-Bq for guix-devel@gnu.org; Mon, 12 Feb 2018 06:46:53 -0500 Received: from brouette.synchrotron.fr ([195.221.0.33]:34518 helo=brouette.synchrotron-soleil.fr) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1elCZN-0007vq-Vv for guix-devel@gnu.org; Mon, 12 Feb 2018 06:46:50 -0500 In-Reply-To: List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+gcggd-guix-devel=m.gmane.org@gnu.org Sender: "Guix-devel" To: Guix Devel Hi everyone, zimoun writes: > From my point of view, there is 2 kind of datasets: > a- the ones which are part of the software, e.g., used to pass the > tests. Therefore, they are usually small, not always; > b- the ones which are applied to the software and somehow they are > not in the source repository. They are big or not. I was thinking of the second kind only. > It does not appear to me a good idea to try to include in the store > datasets of case b-. > Is it not the job of data management tools ? e.g., database etc. It depends. The frontier between data and code is not as clear as it may seem. An example: the weights of a trained neural network can be seen as data (a bunch of numbers), but also as code for a special-purpose processor defined by the neural network. Starting from that example, consider that the weights of a neural network are not fundamentally different from fit parameters in other scientific models. For example the positions of the atoms in a protein structure. Using the same analogy as for the neural network, these positions are the code for a special-purpose processor that computes estimations for the Bragg reflections that are measured in protein crystallography. On the other hand, any sufficiently small dataset can be replaced by a piece of code that defines a literal data structure. Does that turn the data into code or not? My point is that trying to define a distinction between data and code is always arbitrary and rarely of practical interest. I prefer to take a pragmatic stance and ask the question: what are the advantages and problems associated with managing some piece of data in the store? And I suspect that exploring this question for a couple of applications will lead to new ways to profit from the store. However, distribution of data is an entirely different question from managing data on a single machine. I have no idea how well suited the store is for distributing data (Hydra does exactly that, right?), so I'll happily listen to the Guix experts. > It appear to me as a complement of these points ---and personnally, I > learn some points about the design of GWL--- with this thread: > https://lists.gnu.org/archive/html/guix-devel/2016-05/msg00380.html Thanks for the pointer! >> It would be nice if big datasets could conceptually be handled in the same >> way while being stored elsewhere - a bit like git-annex does for git. And >> for parallel computing, we could have special build daemons. > > Hum? the point is to add data management a la git-annex to GWL ? Is it ? At least consider it - I don't know where that will lead. Amirouche Boubekki writes: >> For big datasets, some other mechanism is required. > > Big as in bigger than ram? Bigger that one is willing to handle as files that are copied between machines for distribution. For me, personally and at this moment in time, that's somewhere between 1 GB and 10 GB. > What I was thinking about, is use guix to distribute data packages > just like we distribute softwares from pypi. The advantage of using > guix seems obvious, but apparantly it's not desirable or possible and > I don't understand why. I think there are two questions: 1. Can guix and its current infrastructure be used for data distribution? 2. Can a yet-to-be-designed data distribution infrastructure use the guix store as its storage backend? My understanding from Ludo's comment is that 1) may not be a good idea, but that still leaves 2) as a topic for exploration. > And for parallel computing, we could have special build daemons. > > That's where OWL comes in? Exactly. Konrad.