DAT

DAT

[Catonano]

[-- Attachment #1: Type: text/plain, Size: 2029 bytes --]

well, this is more about reproducible research and Guix than about DAT
specifically.

The current thread about pipelines is very interesting, but I feel like
there's a missing bit.

The data. The sets of files or the datasets

They are part of a pipeline and they should be versioned too. And sometimes
a pipeline produces a dataset. So there could be packages producing
packages.

There's this project, DAT, and it seems they are onto something, in this
domain.

http://dat-data.com/

Based on how they talk about the issue, I'd say they don't know about Guix.
But they do have the same concern about the exact same software running on
a dataset in order to produce a comparable result.

It seems to me that Guix has something to offer to the DAT community. They
want reproducible builds to use on their versioned bittorrent distributed
datasets, so they end up distributing code together with datasets (Gnunet
distributed substitues, anyone ?) AND because they don't know about Guix
they also end up into containerization.

They even touch the issue of the relationship between developers and users,
that I thing Guix and Guile are trying to blur.

But I didn't understand completely what she said about this because English
is not my native language.

As for distributing large amounts of data with DAG things and merkle hashes
and bittorrent like swarms, there's ipfs doing these things (ipfs aims to
distribute triples rather than raw files)  and now DAT too.

I was wondering whether GnuNet has something to say with regard to
reproducible research. These ideas about how to distribute research data in
a serverless fashion came from the "decentralize the web" arena, so maybe
GnuNet might have something to say

But this is not on topic, on the Guix list.

I just thought that you Guix people should be pointed to DAT and the issues
it raises and tries to solve.

It seems to me there's some overlapping and the respective communities
should be aware of each other.

I hope I didn't bother anyone of you with this message

[-- Attachment #2: Type: text/html, Size: 2482 bytes --]

Re: DAT

[Ludovic Courtès]

Hi,

Catonano <catonano@gmail.com> skribis:

> They are part of a pipeline and they should be versioned too. And sometimes
> a pipeline produces a dataset. So there could be packages producing
> packages.
>
> There's this project, DAT, and it seems they are onto something, in this
> domain.
>
> http://dat-data.com/

From a quick look it seems to me that DAT is primarily focusing on
efficient peer-to-peer data distribution, at least in its current form.

In that sense, I would say that DAT and Guix would be complementary
rather than overlapping in a reproducible science toolbox: Guix could be
used to described data sources, build processes, and pipelines, while
DAT would take care of retrieving data sets (DAT data sets could be
described using ‘origin’ in Guix.)

Thanks for sharing!

Ludo’.