Using IPFS to host mirrors of source packages

Using IPFS to host mirrors of source packages

[Pjotr Prins]

By hosting source packages on IPFS they become content addressable and
should scale for downloads. It would also become a safe way of
distributing data. 

https://github.com/ipfs/ipfs

Essentially the interplanetary file system allows for distributed
storage servers and a distributed DNS style resolution for finding
files. It allows people to share storage and by pooling storage we
can provide resilient access to all source packages. Especially the
ones that prove dodgy now.

This should be fairly trivial to implement as long as some of us can
commit some storage. We could use the build farm. 

Next we can do the same for binary Guix packages. It would make for
way faster downloads when the Guix-deamon supports the protocol.

Can we look into this? Who of us can host IPFS and expose port 4001? I
can put in some.

Pj.

Re: Using IPFS to host mirrors of source packages

[Alex Sassmannshausen]

Pjotr Prins writes:

> By hosting source packages on IPFS they become content addressable and
> should scale for downloads. It would also become a safe way of
> distributing data. 
>
> https://github.com/ipfs/ipfs
>
> Essentially the interplanetary file system allows for distributed
> storage servers and a distributed DNS style resolution for finding
> files. It allows people to share storage and by pooling storage we
> can provide resilient access to all source packages. Especially the
> ones that prove dodgy now.
>
> This should be fairly trivial to implement as long as some of us can
> commit some storage. We could use the build farm. 
>
> Next we can do the same for binary Guix packages. It would make for
> way faster downloads when the Guix-deamon supports the protocol.
>
> Can we look into this? Who of us can host IPFS and expose port 4001? I
> can put in some.

I would be willing to dedicate (part of the) resources on a UK based
(small) VPS for an initiative of this type (doesn't necessarily have to
be IPFS, but this sounds good so far :-) ).

Best wishes,

Alex

Re: Using IPFS to host mirrors of source packages

[Joshua Sierles]

At nextjournal.com, we're already looking at IPFS for long term storage.

We would be happy to provide any amount of storage necessary for this
project.

Joshua

Re: Using IPFS to host mirrors of source packages

[Christopher Allan Webber]

Pjotr Prins writes:

> By hosting source packages on IPFS they become content addressable and
> should scale for downloads. It would also become a safe way of
> distributing data.
>
> https://github.com/ipfs/ipfs
>
> Essentially the interplanetary file system allows for distributed
> storage servers and a distributed DNS style resolution for finding
> files. It allows people to share storage and by pooling storage we
> can provide resilient access to all source packages. Especially the
> ones that prove dodgy now.
>
> This should be fairly trivial to implement as long as some of us can
> commit some storage. We could use the build farm.
>
> Next we can do the same for binary Guix packages. It would make for
> way faster downloads when the Guix-deamon supports the protocol.
>
> Can we look into this? Who of us can host IPFS and expose port 4001? I
> can put in some.
>
> Pj.

I like the idea... a lot!

Though I think we'll have to address, how do we *get* the packages from
IPFS?  Do we use an http gateway (and whose)?  It would be even better
if by using Guix you automatically could enable contributing to the IPFS
pool, but I guess that would require a) having an IPFS client in Guile
and b) having the Nix daemon replaced with Guile to do efficiently.
(But maybe I'm overthinking it?)

 - Chris

Re: Using IPFS to host mirrors of source packages

[Pjotr Prins]

On Fri, Aug 11, 2017 at 02:17:36PM -0500, Christopher Allan Webber wrote:
> I like the idea... a lot!
> 
> Though I think we'll have to address, how do we *get* the packages from
> IPFS?  Do we use an http gateway (and whose)?  It would be even better
> if by using Guix you automatically could enable contributing to the IPFS
> pool, but I guess that would require a) having an IPFS client in Guile
> and b) having the Nix daemon replaced with Guile to do efficiently.
> (But maybe I'm overthinking it?)

We could start with an https gateway - probably the current guix
distribution server would be a good choice - to share source packages
hosted elsewhere. That would require no coding at all.

Later a Guile ipfs-client added to the daemon can mean that IPFS can
be queried directly.  The Python API does not look too complicated:

  https://github.com/ipfs/py-ipfs-api/tree/master/ipfsapi

Anyone interested in porting that?

Pj.
-- 

Re: Using IPFS to host mirrors of source packages

[Arun Isaac]

Pjotr Prins writes:

> By hosting source packages on IPFS they become content addressable and
> should scale for downloads. It would also become a safe way of
> distributing data.
>
> https://github.com/ipfs/ipfs
>
> Essentially the interplanetary file system allows for distributed
> storage servers and a distributed DNS style resolution for finding
> files. It allows people to share storage and by pooling storage we
> can provide resilient access to all source packages. Especially the
> ones that prove dodgy now.
>
> This should be fairly trivial to implement as long as some of us can
> commit some storage. We could use the build farm.
>
> Next we can do the same for binary Guix packages. It would make for
> way faster downloads when the Guix-deamon supports the protocol.
>
> Can we look into this? Who of us can host IPFS and expose port 4001? I
> can put in some.

Yes, I think we should look into this. I can host IPFS on port 4001 of
my home server. I have rather low bandwidth (around 1 Mbps of upspeed,
at best), but I can do my bit.

Re: Using IPFS to host mirrors of source packages

[Ludovic Courtès]

Hello,

Pjotr Prins <pjotr.public12@thebird.nl> skribis:

> By hosting source packages on IPFS they become content addressable and
> should scale for downloads. It would also become a safe way of
> distributing data. 
>
> https://github.com/ipfs/ipfs

I would love such a thing to happen (that’s also why we had a GSoC on
using GNUnet’s file sharing capabilities a while back.)

The simplest thing to get started, as you wrote, would be to have an
HTTPS gateway to IPFS (which ‘guix substitute’ could readily talk to
without any modification), though one would still need to populate IPFS
in the first place.

Ludo’.

Re: Using IPFS to host mirrors of source packages

[ng0]

[-- Attachment #1: Type: text/plain, Size: 1549 bytes --]

Ludovic Courtès transcribed 0.6K bytes:
> Hello,
> 
> Pjotr Prins <pjotr.public12@thebird.nl> skribis:
> 
> > By hosting source packages on IPFS they become content addressable and
> > should scale for downloads. It would also become a safe way of
> > distributing data. 
> >
> > https://github.com/ipfs/ipfs

I hope that you are aware that this isn't that safe.
As an intermediate solution it is okay. It is similar to providing
torrents. It works, but everyone wants the easy way, the
easy solution which works $now.

Maybe we have different goals, which would be okay aswell,
you can have different goals but still work on a common base
as teams.

On an update semi-related to this thread:
I don't have any illusions, gnunet-fs integration will take some
time going from the theory lynX, Grothoff and mainly myself thought of,
talked through and wrote down to having it in place. It will require
some additions to gnunet-fs which need to be discussed, etc.

> I would love such a thing to happen (that’s also why we had a GSoC on
> using GNUnet’s file sharing capabilities a while back.)
> 
> The simplest thing to get started, as you wrote, would be to have an
> HTTPS gateway to IPFS (which ‘guix substitute’ could readily talk to
> without any modification), though one would still need to populate IPFS
> in the first place.
> 
> Ludo’.
> 
> 

-- 
ng0
GnuPG: A88C8ADD129828D7EAC02E52E22F9BBFEE348588
GnuPG: https://n0is.noblogs.org/my-keys
https://www.infotropique.org https://krosos.org

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Using IPFS to host mirrors of source packages

[Pjotr Prins]

On Tue, Aug 22, 2017 at 08:38:32AM +0000, ng0 wrote:
> Ludovic Courtès transcribed 0.6K bytes:
> > Hello,
> > 
> > Pjotr Prins <pjotr.public12@thebird.nl> skribis:
> > 
> > > By hosting source packages on IPFS they become content addressable and
> > > should scale for downloads. It would also become a safe way of
> > > distributing data. 
> > >
> > > https://github.com/ipfs/ipfs
> 
> I hope that you are aware that this isn't that safe.
> As an intermediate solution it is okay. It is similar to providing
> torrents. It works, but everyone wants the easy way, the
> easy solution which works $now.

What do you mean? Gnunet has a focus on privacy (at a cost of speed).
IPFS is not really meant to be private (though you can host a private
network). For public data it is safe. You have to trust the original
party that provides the file, but that is similar to what we are doing
today fetching data from github, ftp etc. (a Guix SHA check is good
enough).

Today I was bitten again by R/bioconductor removing source files. It
would make good sense to me to mirror those files on IPFS. I think
that is a long term solution.

Pj.

Re: Using IPFS to host mirrors of source packages

[ng0]

[-- Attachment #1: Type: text/plain, Size: 2341 bytes --]

ng0 transcribed 2.6K bytes:
> Ludovic Courtès transcribed 0.6K bytes:
> > Hello,
> > 
> > Pjotr Prins <pjotr.public12@thebird.nl> skribis:
> > 
> > > By hosting source packages on IPFS they become content addressable and
> > > should scale for downloads. It would also become a safe way of
> > > distributing data. 
> > >
> > > https://github.com/ipfs/ipfs
> 
> I hope that you are aware that this isn't that safe.
> As an intermediate solution it is okay. It is similar to providing
> torrents. It works, but everyone wants the easy way, the
> easy solution which works $now.

Eh, let me revise that. It is safe, but IPFS solves different problems
than GNUnet. They are doing good work nevertheless.
I just think that putting work into something entirely different is
pointless, from where I stand. If everyone who complained about certain
things would work on them just a bit, we'd move faster. But hey, that's
life. Just a check on wether we'll have IPFS and GNUnet some day or
if I should just consider working on it completely on my own now as
you consider GNUnet doomed (or something).

> Maybe we have different goals, which would be okay aswell,
> you can have different goals but still work on a common base
> as teams.
> 
> On an update semi-related to this thread:
> I don't have any illusions, gnunet-fs integration will take some
> time going from the theory lynX, Grothoff and mainly myself thought of,
> talked through and wrote down to having it in place. It will require
> some additions to gnunet-fs which need to be discussed, etc.
> 
> > I would love such a thing to happen (that’s also why we had a GSoC on
> > using GNUnet’s file sharing capabilities a while back.)
> > 
> > The simplest thing to get started, as you wrote, would be to have an
> > HTTPS gateway to IPFS (which ‘guix substitute’ could readily talk to
> > without any modification), though one would still need to populate IPFS
> > in the first place.
> > 
> > Ludo’.
> > 
> > 
> 
> -- 
> ng0
> GnuPG: A88C8ADD129828D7EAC02E52E22F9BBFEE348588
> GnuPG: https://n0is.noblogs.org/my-keys
> https://www.infotropique.org https://krosos.org



-- 
ng0
GnuPG: A88C8ADD129828D7EAC02E52E22F9BBFEE348588
GnuPG: https://n0is.noblogs.org/my-keys
https://www.infotropique.org https://krosos.org

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Using IPFS to host mirrors of source packages

[Ludovic Courtès]

Pjotr Prins <pjotr.public12@thebird.nl> skribis:

> Today I was bitten again by R/bioconductor removing source files.

Were those files looked for at
https://mirror.hydra.gnu.org/file/…/sha256/…, and if so, were they
missing there?

Ludo’.

Re: Using IPFS to host mirrors of source packages

[Pjotr Prins]

On Tue, Aug 22, 2017 at 12:17:01PM +0200, Ludovic Courtès wrote:
> Pjotr Prins <pjotr.public12@thebird.nl> skribis:
> 
> > Today I was bitten again by R/bioconductor removing source files.
> 
> Were those files looked for at
> https://mirror.hydra.gnu.org/file/…/sha256/…, and if so, were they
> missing there?

*blush* I did not realise we are caching them. The packages are on my
GUIX_PACKAGE_PATH. Good reason to move them into Guix ;)

Btw I just redeployed an update of the massive genenetwork2 graph
against the latest Guix master. It worked pretty much in one go. Kudos
to all - I am very happy to be using Guix in professional deployments.

Pj.
-- 

Re: Using IPFS to host mirrors of source packages

[ng0]

[-- Attachment #1: Type: text/plain, Size: 1877 bytes --]

Pjotr Prins transcribed 1.1K bytes:
> On Tue, Aug 22, 2017 at 08:38:32AM +0000, ng0 wrote:
> > Ludovic Courtès transcribed 0.6K bytes:
> > > Hello,
> > > 
> > > Pjotr Prins <pjotr.public12@thebird.nl> skribis:
> > > 
> > > > By hosting source packages on IPFS they become content addressable and
> > > > should scale for downloads. It would also become a safe way of
> > > > distributing data. 
> > > >
> > > > https://github.com/ipfs/ipfs
> > 
> > I hope that you are aware that this isn't that safe.
> > As an intermediate solution it is okay. It is similar to providing
> > torrents. It works, but everyone wants the easy way, the
> > easy solution which works $now.
> 
> What do you mean? Gnunet has a focus on privacy (at a cost of speed).
> IPFS is not really meant to be private (though you can host a private
> network). For public data it is safe. You have to trust the original
> party that provides the file, but that is similar to what we are doing
> today fetching data from github, ftp etc. (a Guix SHA check is good
> enough).
> 
> Today I was bitten again by R/bioconductor removing source files. It
> would make good sense to me to mirror those files on IPFS. I think
> that is a long term solution.
> 
> Pj.

While I don't want to comment on everything, let me just comment this:

I think IPFS is the best for now. I'll continue working on GNUnet and
gnunet-fs on my side and will hope that Guix accepts it once gnunet-fs is
ready for this usage.

Offtopic question: What's with my request for inclusion on savannah?
At least a short comment in 4 weeks would be re-assuring. I'm not sure
if there are technical issues or if everyone is on vacation and otherwise
busy.
-- 
ng0
GnuPG: A88C8ADD129828D7EAC02E52E22F9BBFEE348588
GnuPG: https://n0is.noblogs.org/my-keys
https://www.infotropique.org https://krosos.org

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Using IPFS to host mirrors of source packages

[Ludovic Courtès]

Pjotr Prins <pjotr.public12@thebird.nl> skribis:

> On Tue, Aug 22, 2017 at 12:17:01PM +0200, Ludovic Courtès wrote:
>> Pjotr Prins <pjotr.public12@thebird.nl> skribis:
>> 
>> > Today I was bitten again by R/bioconductor removing source files.
>> 
>> Were those files looked for at
>> https://mirror.hydra.gnu.org/file/…/sha256/…, and if so, were they
>> missing there?
>
> *blush* I did not realise we are caching them. The packages are on my
> GUIX_PACKAGE_PATH. Good reason to move them into Guix ;)

Heh.  :-)

> Btw I just redeployed an update of the massive genenetwork2 graph
> against the latest Guix master. It worked pretty much in one go. Kudos
> to all - I am very happy to be using Guix in professional deployments.

Good to hear!

Ludo’.