Using GNUnet for binary package distribution

Using GNUnet for binary package distribution

[Ludovic Courtès]

[-- Attachment #1: Type: text/plain, Size: 1441 bytes --]

Hello GNUnet!

GNU Guix provides a transparent binary/source deployment model.  A
server can claim: “hey, I have the binary for
/nix/store/v9zic07iar8w90zcy398r745w78a7lqs-emacs-24.2!”, where the
base32 string uniquely identifies a build process.  If you trust that
server to provide genuine binaries, then you can grab them instead of
building Emacs locally.

The “traditional model” has been to have a build farm build and serve
binary packages.  In that model, users trust the build farm to provide
authentic binaries.

I’m interested in providing a /practical/ decentralized distribution
model.  It seems to me that GNUnet’s DHT would be the most appropriate
(as opposed to AFS).  WDYT?

One of the problems to be solved is authentication: users would have to
specify a list of GNUnet pseudonyms of trusted binary providers, or
something like that.  Managing this list would have to be as easy as
possible, to allow the system to scale.

Another issue is privacy: we want to give users an incentive to share
their binaries, but at the same time, they should have control over what
gets shared (for instance, Christian may want to hide the fact that he’s
installed Python and not Guile ;-)).

What do you think of the idea?  Would the DHT retain files long enough
for this to be practical?

I’m considering submitting this as a GSoC project (under the GNU
umbrella).

Thanks!

Ludo’.

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: Using GNUnet for binary package distribution

[Andreas Enge]

Am Donnerstag, 21. März 2013 schrieb Ludovic Courtès:
> I’m interested in providing a /practical/ decentralized distribution
> model.  It seems to me that GNUnet’s DHT would be the most appropriate
> (as opposed to AFS).  WDYT?

This sounds very interesting indeed! On the guix side, it would be useful 
to start by packaging gnunet. I intended to do so, but am stuck with 
failing tests in libextractor, which I did not find the time to debug.

Andreas

Re: Using GNUnet for binary package distribution

[Christian Grothoff]

On 03/21/2013 02:02 PM, Ludovic Courtès wrote:
> Hello GNUnet!
>
> GNU Guix provides a transparent binary/source deployment model.  A
> server can claim: “hey, I have the binary for
> /nix/store/v9zic07iar8w90zcy398r745w78a7lqs-emacs-24.2!”, where the
> base32 string uniquely identifies a build process.  If you trust that
> server to provide genuine binaries, then you can grab them instead of
> building Emacs locally.
>
> The “traditional model” has been to have a build farm build and serve
> binary packages.  In that model, users trust the build farm to provide
> authentic binaries.
>
> I’m interested in providing a /practical/ decentralized distribution
> model.  It seems to me that GNUnet’s DHT would be the most appropriate
> (as opposed to AFS).  WDYT?
>
> One of the problems to be solved is authentication: users would have to
> specify a list of GNUnet pseudonyms of trusted binary providers, or
> something like that.  Managing this list would have to be as easy as
> possible, to allow the system to scale.
>
> Another issue is privacy: we want to give users an incentive to share
> their binaries, but at the same time, they should have control over what
> gets shared (for instance, Christian may want to hide the fact that he’s
> installed Python and not Guile ;-)).
>
> What do you think of the idea?  Would the DHT retain files long enough
> for this to be practical?

Well, the GNUnet DHT expects that the data source periodically refreshes 
the values by re-issuing the PUT; without that, it cannot work. 
Furthermore, you need to consider that DHTs are typically only useful 
for small data pieces (think <= 64k), not for large files.  So what 
you'd store in the DHT is the meta data (where to find the large files), 
not the actual files.

gnunet-update (svn/gnunet-update/) is a little project where we started 
to work on a GNUnet installer that is supposed to include an update 
mechanism that downloads updates via GNUnet --- after all, if you are
using a recent version of GNUnet, sharing your installation binaries
costs you at least no disk space at all, and if censorship kicks in,
having a way to update in a decentralized fashion might become important.

So gnunet-update is planned to provide the means to locate files based 
on some package description (signatures, meta data) and download them
via the P2P network.  Fundamentally, there is nothing wrong with using
the basic ideas to distribute packages other than GNUnet itself.

Our current approach to package management is essentially to look at ldd 
and grab all dependencies (unless compatible versions are already
available on the target system, based on libtool versioning info); the
idea was to make it work with 'any' distribution as long as the 
architecture matches.  Naturally, that doesn't mean that in principle a 
different package manager could not be used/supported.

gnunet-update is not yet finished, we're currently planning to revise 
some internal part that gnunet-update will depend on (stream); still, 
help in moving this area along would be of course welcome.

Happy hacking!

Christian

Re: Using GNUnet for binary package distribution

[Sree Harsha Totakura]

On 03/21/2013 07:01 PM, Christian Grothoff wrote:
> On 03/21/2013 02:02 PM, Ludovic Courtès wrote:
>> Hello GNUnet!
>>
>> GNU Guix provides a transparent binary/source deployment model.  A
>> server can claim: “hey, I have the binary for
>> /nix/store/v9zic07iar8w90zcy398r745w78a7lqs-emacs-24.2!”, where the
>> base32 string uniquely identifies a build process.  If you trust that
>> server to provide genuine binaries, then you can grab them instead of
>> building Emacs locally.
>>
>> The “traditional model” has been to have a build farm build and serve
>> binary packages.  In that model, users trust the build farm to provide
>> authentic binaries.
>>
>> I’m interested in providing a /practical/ decentralized distribution
>> model.  It seems to me that GNUnet’s DHT would be the most appropriate
>> (as opposed to AFS).  WDYT?
>>
>> One of the problems to be solved is authentication: users would have to
>> specify a list of GNUnet pseudonyms of trusted binary providers, or
>> something like that.  Managing this list would have to be as easy as
>> possible, to allow the system to scale.
>>
>> Another issue is privacy: we want to give users an incentive to share
>> their binaries, but at the same time, they should have control over what
>> gets shared (for instance, Christian may want to hide the fact that he’s
>> installed Python and not Guile ;-)).
>>
>> What do you think of the idea?  Would the DHT retain files long enough
>> for this to be practical?
> 
> Well, the GNUnet DHT expects that the data source periodically refreshes
> the values by re-issuing the PUT; without that, it cannot work.
> Furthermore, you need to consider that DHTs are typically only useful
> for small data pieces (think <= 64k), not for large files.  So what
> you'd store in the DHT is the meta data (where to find the large files),
> not the actual files.
> 
> gnunet-update (svn/gnunet-update/) is a little project where we started
> to work on a GNUnet installer that is supposed to include an update
> mechanism that downloads updates via GNUnet --- after all, if you are
> using a recent version of GNUnet, sharing your installation binaries
> costs you at least no disk space at all, and if censorship kicks in,
> having a way to update in a decentralized fashion might become important.
> 
> So gnunet-update is planned to provide the means to locate files based
> on some package description (signatures, meta data) and download them
> via the P2P network.  Fundamentally, there is nothing wrong with using
> the basic ideas to distribute packages other than GNUnet itself.
> 
> Our current approach to package management is essentially to look at ldd
> and grab all dependencies (unless compatible versions are already
> available on the target system, based on libtool versioning info); the
> idea was to make it work with 'any' distribution as long as the
> architecture matches.  Naturally, that doesn't mean that in principle a
> different package manager could not be used/supported.
> 
> gnunet-update is not yet finished, we're currently planning to revise
> some internal part that gnunet-update will depend on (stream); still,
> help in moving this area along would be of course welcome.

For authentication, we intend to use GPG with gnunet-update.  The idea
is that the gnunet-updater would search for updates using GNUnet's File
Sharing service and downloads meta-data files.  It then verifies if the
meta-data files are signed by a trusted key (which is user-configurable)
and proceeds with the download of actual binaries.

Using this approach the meta-data files and the binaries pointed in
meta-data can be published by anyone and still be verified.  This could
improve the availability of both meta-data and the binaries.

--
Harsha

_______________________________________________
GNUnet-developers mailing list
GNUnet-developers@gnu.org
https://lists.gnu.org/mailman/listinfo/gnunet-developers

Re: [GNUnet-developers] Using GNUnet for binary package distribution

[Ludovic Courtès]

Hi,

Sree Harsha Totakura <totakura@in.tum.de> skribis:

> For authentication, we intend to use GPG with gnunet-update.  The idea
> is that the gnunet-updater would search for updates using GNUnet's File
> Sharing service and downloads meta-data files.  It then verifies if the
> meta-data files are signed by a trusted key (which is user-configurable)
> and proceeds with the download of actual binaries.

OK.  Wouldn’t using the AFS service be a bottleneck, in terms of
availability and bandwidth?  My impression is that it might be OK for
small updates like those of GNUnet itself, but not for a full distro.

For Guix, downloading binaries is an option that is only worthwhile if
it’s faster than building locally.

Ludo’.

Re: [GNUnet-developers] Using GNUnet for binary package distribution

[Ludovic Courtès]

Christian Grothoff <grothoff@in.tum.de> skribis:

> Well, the GNUnet DHT expects that the data source periodically
> refreshes the values by re-issuing the PUT; without that, it cannot
> work.

Sure.

> Furthermore, you need to consider that DHTs are typically only useful
> for small data pieces (think <= 64k), not for large files.  So what
> you'd store in the DHT is the meta data (where to find the large
> files), not the actual files.

Hmm, OK.  I’m concerned that AFS might be too slow and unavailable for
our purposes.

> gnunet-update (svn/gnunet-update/) is a little project where we
> started to work on a GNUnet installer that is supposed to include an
> update mechanism that downloads updates via GNUnet

Sounds nice.

[...]

> Our current approach to package management is essentially to look at
> ldd and grab all dependencies

We could imagine customizing that part for Guix since it already has
finer-grain info.

[...]

> gnunet-update is not yet finished, we're currently planning to revise
> some internal part that gnunet-update will depend on (stream); still,
> help in moving this area along would be of course welcome.

So what would you think of submitting this idea for GSoC?  It might be
more work on the GNUnet side than on the Guix side, AIUI.  Is it too
early?  Would it interfere with work being done by other people?

Thanks for the detailed reply!

Ludo’.

Re: [GNUnet-developers] Using GNUnet for binary package distribution

[Christian Grothoff]

On 03/22/2013 01:25 PM, Ludovic Courtès wrote:
> Hi,
>
> Sree Harsha Totakura<totakura@in.tum.de>  skribis:
>
>> For authentication, we intend to use GPG with gnunet-update.  The idea
>> is that the gnunet-updater would search for updates using GNUnet's File
>> Sharing service and downloads meta-data files.  It then verifies if the
>> meta-data files are signed by a trusted key (which is user-configurable)
>> and proceeds with the download of actual binaries.
>
> OK.  Wouldn’t using the AFS service be a bottleneck, in terms of
> availability and bandwidth?  My impression is that it might be OK for
> small updates like those of GNUnet itself, but not for a full distro.

We won't be using the *anonymous* file-sharing for this.

> For Guix, downloading binaries is an option that is only worthwhile if
> it’s faster than building locally.

Of course.

-Christian

Re: Using GNUnet for binary package distribution

[Ludovic Courtès]

Christian Grothoff <grothoff@in.tum.de> skribis:

> On 03/22/2013 01:25 PM, Ludovic Courtès wrote:

[...]

>> OK.  Wouldn’t using the AFS service be a bottleneck, in terms of
>> availability and bandwidth?  My impression is that it might be OK for
>> small updates like those of GNUnet itself, but not for a full distro.
>
> We won't be using the *anonymous* file-sharing for this.

Ah great, sorry for the misunderstanding.

Ludo’.

_______________________________________________
GNUnet-developers mailing list
GNUnet-developers@gnu.org
https://lists.gnu.org/mailman/listinfo/gnunet-developers

Re: [GNUnet-developers] Using GNUnet for binary package distribution

[Ludovic Courtès]

Christian Grothoff <grothoff@in.tum.de> skribis:

>>> gnunet-update is not yet finished, we're currently planning to revise
>>> some internal part that gnunet-update will depend on (stream); still,
>>> help in moving this area along would be of course welcome.
>>
>> So what would you think of submitting this idea for GSoC?  It might be
>> more work on the GNUnet side than on the Guix side, AIUI.  Is it too
>> early?  Would it interfere with work being done by other people?
>
> No, I don't think it would interfere. Matthias and Bart will generally
> be happy to mentor GSoC hacking this year, and I suspect Sree Harsha
> wouldn't mind giving advice either, so mentoring should not be an issue.

Great.

So how do we proceed?  On the GNU side, the submission as to be mailed
by Monday [0].

Since you’re already working on gnunet-update, perhaps it would make
more sense for one of you to be the official mentor.

Then I or anyone else on bug-guix can of course help for all things Guix.

WDYT?

Ludo’.

[0] http://www.gnu.org/software/soc-projects/ideas.html

Re: [GNUnet-developers] Using GNUnet for binary package distribution

[Sree Harsha Totakura]

On 03/22/2013 03:52 PM, Ludovic Courtès wrote:
>> > No, I don't think it would interfere. Matthias and Bart will generally
>> > be happy to mentor GSoC hacking this year, and I suspect Sree Harsha
>> > wouldn't mind giving advice either, so mentoring should not be an issue.
> Great.
> 
> So how do we proceed?  On the GNU side, the submission as to be mailed
> by Monday [0].
> 
> Since you’re already working on gnunet-update, perhaps it would make
> more sense for one of you to be the official mentor.
> 
> Then I or anyone else on bug-guix can of course help for all things Guix.
> 
> WDYT?

I prefer to register as a student for GSoC.  Once gnunet-update's gets
into good shape, I can then start hacking Guix to use gnunet.

--
Harsha

Re: [GNUnet-developers] Using GNUnet for binary package distribution

[Sree Harsha Totakura]

On 03/23/2013 09:51 PM, Sree Harsha Totakura wrote:
>> So how do we proceed?  On the GNU side, the submission as to be mailed
>> > by Monday [0].
>> > 
>> > Since you’re already working on gnunet-update, perhaps it would make
>> > more sense for one of you to be the official mentor.
>> > 
>> > Then I or anyone else on bug-guix can of course help for all things Guix.
>> > 
>> > WDYT?
> I prefer to register as a student for GSoC.  Once gnunet-update's gets
> into good shape, I can then start hacking Guix to use gnunet.

Hi Ludo,

I just discussed with Christian.  We are thinking to represent the idea
as follows since we see Guix and GNUnet as interdependent on delivering
updates.

* Under the GNU project, may be you can post the idea as "fetching
updates using a peer-to-peer network (gnunet)"
* Under the GNUnet project, it will be represented as "using Guix for
delivering updates to GNUnet"

Matthias and Bart shall be the mentors from GNUnet.

Shall we agree on this?

--
Harsha

Re: [GNUnet-developers] Using GNUnet for binary package distribution

[Christian Grothoff]

Just to clarify, the point is to for now list the project under both 
mentoring organizations, and then sort out details later.  Having
mentoring from both Guix and GNUnet would make sense anyway, and I
believe having Harsha do it will make a lot of sense and avoid any
possible conflict in terms of the direction of the development.

So for now, what we ask you (Ludo) to do is to make sure the Guix/GNUnet
integration is listed on the GNU ideas page with you (or some other
Guix hacker) listed as a mentor.

Thanks!

Happy hacking!

-Christian

On 03/25/2013 11:46 AM, Sree Harsha Totakura wrote:
> On 03/23/2013 09:51 PM, Sree Harsha Totakura wrote:
>>> So how do we proceed?  On the GNU side, the submission as to be mailed
>>>> by Monday [0].
>>>>
>>>> Since you’re already working on gnunet-update, perhaps it would make
>>>> more sense for one of you to be the official mentor.
>>>>
>>>> Then I or anyone else on bug-guix can of course help for all things Guix.
>>>>
>>>> WDYT?
>> I prefer to register as a student for GSoC.  Once gnunet-update's gets
>> into good shape, I can then start hacking Guix to use gnunet.
>
> Hi Ludo,
>
> I just discussed with Christian.  We are thinking to represent the idea
> as follows since we see Guix and GNUnet as interdependent on delivering
> updates.
>
> * Under the GNU project, may be you can post the idea as "fetching
> updates using a peer-to-peer network (gnunet)"
> * Under the GNUnet project, it will be represented as "using Guix for
> delivering updates to GNUnet"
>
> Matthias and Bart shall be the mentors from GNUnet.
>
> Shall we agree on this?
>
> --
> Harsha
>

Re: [GNUnet-developers] Using GNUnet for binary package distribution

[Ludovic Courtès]

Christian Grothoff <grothoff@in.tum.de> skribis:

> So for now, what we ask you (Ludo) to do is to make sure the Guix/GNUnet
> integration is listed on the GNU ideas page with you (or some other
> Guix hacker) listed as a mentor.

Great, I’ll post something later today.

Thanks!

Ludo’.