[GSoC] GNUnet binary distribution system

[GSoC] GNUnet binary distribution system

[Pierre-Antoine Rault]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi Guix,

I've been lurking and tweaking for some time now on #guix and have found
Guix to be of much interest. Recently hydra has come offline and i've
understood the need of a decentralized network (or at least some
redundancy) for a the binary distribution.
That is why I would like to apply as a GSoC student working on P2P
exchange of trusted signed binaries, hence the implementation of GNUnet
within Guix.

Having worked with libtorrent and kademila, i am quite familiar with dht
and P2P, and I think GNUnet shouldn't be much difficult to adapt to. I
am thus reading info about GNUnet, but I already have some features in
mind (not sure if redondant with other messages posted before):
* As said by Andreas Enge, users should all be able to share packages
themselves, provided they can cryptographically sign their binaries.
They would deploy a node (a personnal node) using a local deamon (could
be by passing a command to guix-daemon which would handle the task). The
local daemon would contact a list of known nodes (contained in a
mirrorlist) and send them hashes with their corresponding signature.
* Users should be able to trust a node. That would not mean trusting all
providers for all hashes recursively, but rather trust those trusted by
the node in cascade. Thus a distinction beetween trusted pairs and
trusted node pairs should be done.
* I think we should focus on ease of use, since it could become a good
alternative to setting up a full fledged server to share officially
supported packages and unofficial ones (like the AUR for Archlinux).

I still need to discuss my ideas about privacy, as I don't know what has
been said about it yet. Any comments appreciated :)

My Scheme level isn't high, but I'm learning Geiser/Paredit at the
moment and reading the docs thoroughly.

- - rigelk
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.22 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBAgAGBQJTHgd2AAoJEHfJ0QE7gLd6j+oH/3f5LVoRxrkq+fvPJBBYbDq/
CHkIS3YEEnSYqznfmzLKifbrugUchjxyQE7KeyT079HnF6eJyU3AwlQVEB4Kr2qt
TcUj1Ep4dps2h0C/tphvR8W4xcVc3irGKvTJ4eeK6TzA8nOhTbCyit1hMNNa7OWR
aGzmNwwg+01jJoyt5L0HVy6TVofmu5tAUc0UEwtTgz7JoGSvlAudzk+nFV9Ef4Hy
tq5zDPRpW0726Wo+JFJxKFCgl+SYvZL7gpYpGX0rL3lXaGy3mhh7WhABqusorkev
+PvlLoWFyQxS5jgrZn2g/HlKproOTYPnwphlLRDLGDHlg2RLqRkgpJKTzOu5FMk=
=qaQ/
-----END PGP SIGNATURE-----

Re: [GSoC] GNUnet binary distribution system

[Ludovic Courtès]

Hi!

Thanks for your proposal!

Pierre-Antoine Rault <par@rigelk.eu> skribis:

> Having worked with libtorrent and kademila, i am quite familiar with dht
> and P2P, and I think GNUnet shouldn't be much difficult to adapt to. I
> am thus reading info about GNUnet, but I already have some features in
> mind (not sure if redondant with other messages posted before):

Please do get in touch with the GNUnet folks as well.  I’ve Cc’d them so
you can say hi.  ;-)

> * As said by Andreas Enge, users should all be able to share packages
> themselves, provided they can cryptographically sign their binaries.
> They would deploy a node (a personnal node) using a local deamon (could
> be by passing a command to guix-daemon which would handle the task). The
> local daemon would contact a list of known nodes (contained in a
> mirrorlist) and send them hashes with their corresponding signature.

What exactly do is meant by “node” and “mirrorlist” here?

The initial discussion [0] left open the question of where binaries
themselves should be stored.  A possibility would be to use GNUnet’s DHT
simply as a discovery mechanism, and then to establish a connection
directly to the user’s machine, which would run, say, and HTTP server.

[0] http://lists.gnu.org/archive/html/bug-guix/2013-03/msg00085.html

> * Users should be able to trust a node. That would not mean trusting all
> providers for all hashes recursively, but rather trust those trusted by
> the node in cascade. Thus a distinction beetween trusted pairs and
> trusted node pairs should be done.

Please read the “Invoking guix archive” section of the manual (from
Git), to get a feel of the current approach.  See also Mark’s message at
<https://lists.gnu.org/archive/html/guix-devel/2014-02/msg00297.html>
for ideas of what we’d like.

> * I think we should focus on ease of use, since it could become a good
> alternative to setting up a full fledged server to share officially
> supported packages and unofficial ones (like the AUR for Archlinux).

That’s been discussed before (see
<http://lists.gnu.org/archive/html/guix-devel/2013-08/msg00127.html>),
but it’s orthogonal.

> My Scheme level isn't high, but I'm learning Geiser/Paredit at the
> moment and reading the docs thoroughly.

Depending on the approach taken, it could be an issue if you have little
experience with Scheme.

This proposal needs discussion with both Guix and GNUnet people (and
notably Sree Harsha, who is at the intersection of both projects :-)),
to work towards a concrete road map of things to hack on.

Thanks,
Ludo’.

_______________________________________________
GNUnet-developers mailing list
GNUnet-developers@gnu.org
https://lists.gnu.org/mailman/listinfo/gnunet-developers

Re: [GSoC] GNUnet binary distribution system

[Christian Grothoff]

[-- Attachment #1: Type: text/plain, Size: 740 bytes --]

On 03/10/2014 10:09 PM, Ludovic Courtès wrote:
> The initial discussion [0] left open the question of where binaries 
> themselves should be stored.  A possibility would be to use GNUnet’s
> DHT simply as a discovery mechanism, and then to establish a
> connection directly to the user’s machine, which would run, say, and
> HTTP server.

I'd ACK using the DHT, but the use of HTTP means that it (often) won't
work if the peer is behind NAT.  If you instead use GNUnet MESH to
establish a TCP-like connection to the file transfer, you might be
less efficient than a direct TCP/HTTP connection, but you should get
through NAT _and_ be less observable _and_ have "good" encryption
(not just TLS).

My 2 cents

Christian

[-- Attachment #2: 0x48426C7E.asc --]
[-- Type: application/pgp-keys, Size: 7211 bytes --]

[-- Attachment #3: Type: text/plain, Size: 162 bytes --]

_______________________________________________
GNUnet-developers mailing list
GNUnet-developers@gnu.org
https://lists.gnu.org/mailman/listinfo/gnunet-developers

Re: [GSoC] GNUnet binary distribution system

[Ludovic Courtès]

Christian Grothoff <christian@grothoff.org> skribis:

> On 03/10/2014 10:09 PM, Ludovic Courtès wrote:
>> The initial discussion [0] left open the question of where binaries 
>> themselves should be stored.  A possibility would be to use GNUnet’s
>> DHT simply as a discovery mechanism, and then to establish a
>> connection directly to the user’s machine, which would run, say, and
>> HTTP server.
>
> I'd ACK using the DHT, but the use of HTTP means that it (often) won't
> work if the peer is behind NAT.  If you instead use GNUnet MESH to
> establish a TCP-like connection to the file transfer, you might be
> less efficient than a direct TCP/HTTP connection, but you should get
> through NAT _and_ be less observable _and_ have "good" encryption
> (not just TLS).

Makes a lot of sense, indeed.

Thanks, that’s the kind of input I was expecting.  :-)

Ludo’.

_______________________________________________
GNUnet-developers mailing list
GNUnet-developers@gnu.org
https://lists.gnu.org/mailman/listinfo/gnunet-developers

Re: [GSoC] GNUnet binary distribution system

[Pierre-Antoine Rault]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi,

On 10/03/2014 22:09, Ludovic Courtès wrote:
> 
> Pierre-Antoine Rault <par@rigelk.eu> skribis:
> 
>> Having worked with libtorrent and kademila, i am quite familiar
>> with dht and P2P, and I think GNUnet shouldn't be much difficult
>> to adapt to. I am thus reading info about GNUnet, but I already
>> have some features in mind (not sure if redondant with other
>> messages posted before):
> 
> Please do get in touch with the GNUnet folks as well.  I’ve Cc’d
> them so you can say hi.  ;-)

Thanks ! :)

>> * As said by Andreas Enge, users should all be able to share
>> packages themselves, provided they can cryptographically sign
>> their binaries. They would deploy a node (a personnal node) using
>> a local deamon (could be by passing a command to guix-daemon
>> which would handle the task). The local daemon would contact a
>> list of known nodes (contained in a mirrorlist) and send them
>> hashes with their corresponding signature.
> 
> What exactly is meant by “node” and “mirrorlist” here?

A node [1] represents either a server like hydra or a user that has
decided to share binaries, in our dev case.

[1]
http://en.wikipedia.org/wiki/Node_(networking)#Distributed_system_nodes

A mirrorlist is a list of adresses to contact first (i.e: dht may be a
distributed system, it still needs a few adresses to initiate hash
search ; at least that's how most torrent systems handle dht). A user
should be able to manually add/remove dht nodes to be contacted first.

> The initial discussion [0] left open the question of where
> binaries themselves should be stored.  A possibility would be to
> use GNUnet’s DHT simply as a discovery mechanism, and then to
> establish a connection directly to the user’s machine, which would
> run, say, an HTTP server.

That's what I had in mind. Now, considered the post [2] by Christian
Grothoff, we might consider using either an HTTP server for
performance or GNUnet's MESH for anonymity (and security). We should
balance needs and ease of implementation.

[2] http://lists.gnu.org/archive/html/guix-devel/2014-03/msg00113.html

>> * Users should be able to trust a node. That would not mean
>> trusting all providers for all hashes recursively, but rather
>> trust those trusted by the node in cascade. Thus a distinction
>> beetween trusted pairs and trusted node pairs should be done.
> 
> Please read the “Invoking guix archive” section of the manual
> (from Git), to get a feel of the current approach.  See also Mark’s
> message at 
> <https://lists.gnu.org/archive/html/guix-devel/2014-02/msg00297.html>
>
> 
for ideas of what we’d like.
> 
>> * I think we should focus on ease of use, since it could become a
>> good alternative to setting up a full fledged server to share
>> officially supported packages and unofficial ones (like the AUR
>> for Archlinux).
> 
> That’s been discussed before (see 
> <http://lists.gnu.org/archive/html/guix-devel/2013-08/msg00127.html>),
>
> 
but it’s orthogonal.

Then we have to extend the guix tools to cover user modules (overlays,
that is).

> This proposal needs discussion with both Guix and GNUnet people
> (and notably Sree Harsha, who is at the intersection of both
> projects :-)), to work towards a concrete road map of things to
> hack on.

Sure ; It's the only way to have a clear and shared view of what parts
of Guix are involved in the project. I'm working on a roadmap draft
for now.

- - rigelk
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.22 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBAgAGBQJTHkiUAAoJEHfJ0QE7gLd6+pwH/0qKzeyF3S/78GEI8E999SAo
HF0tG3UdaDAmFlm69HVuzEwaHzJVzUsyo77rRDvRWv7urIrsuIakMTdfcQPeaFF5
X8s59AyE1KhVqbwoCPGwW21Cti5rGF1B0fUk9NTmTcbd14orYgeYsQJtGbOSATIU
up7OVnyDnyqVlCiHcMBjkyvED+d0IGG5CXS1DKhyHM1u/U2S2XatYsRem5nPZ+kE
iAvZAct1+rCJT1f3uQLGPAcCIxXs7BR2BWS9gslXOdZlCFIKSyiIsYlGar9j/WXi
ZwkmmLiylDmv2z9gqapH6PthhantZiEqGDh2dHarY0gqz0VnVgq7OobAjPm0bGs=
=W+vJ
-----END PGP SIGNATURE-----

Re: [GSoC] GNUnet binary distribution system

[Mark H Weaver]

Pierre-Antoine Rault <par@rigelk.eu> writes:

> On 10/03/2014 22:09, Ludovic Courtès wrote:
>> 
>> The initial discussion [0] left open the question of where
>> binaries themselves should be stored.  A possibility would be to
>> use GNUnet’s DHT simply as a discovery mechanism, and then to
>> establish a connection directly to the user’s machine, which would
>> run, say, an HTTP server.
>
> That's what I had in mind. Now, considered the post [2] by Christian
> Grothoff, we might consider using either an HTTP server for
> performance or GNUnet's MESH for anonymity (and security). We should
> balance needs and ease of implementation.
>
> [2] http://lists.gnu.org/archive/html/guix-devel/2014-03/msg00113.html

FWIW, I think it might be worthwhile to support BitTorrent magnet links
as well, as a middle ground between these two extremes.  Most users will
not be able to host binaries via HTTP; even if they have a server, the
bandwidth requirements are hard to predict and likely to be too high.

      Mark

Re: [GSoC] GNUnet binary distribution system

[Pierre-Antoine Rault]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 11/03/2014 04:35, Mark H Weaver wrote:
> Pierre-Antoine Rault <par@rigelk.eu> writes:
> 
>> On 10/03/2014 22:09, Ludovic Courtès wrote:
>>> 
>>> The initial discussion [0] left open the question of where 
>>> binaries themselves should be stored.  A possibility would be
>>> to use GNUnet’s DHT simply as a discovery mechanism, and then
>>> to establish a connection directly to the user’s machine, which
>>> would run, say, an HTTP server.
>> 
>> That's what I had in mind. Now, considered the post [2] by
>> Christian Grothoff, we might consider using either an HTTP server
>> for performance or GNUnet's MESH for anonymity (and security). We
>> should balance needs and ease of implementation.
>> 
>> [2]
>> http://lists.gnu.org/archive/html/guix-devel/2014-03/msg00113.html
>
>> 
> FWIW, I think it might be worthwhile to support BitTorrent magnet
> links as well, as a middle ground between these two extremes.  Most
> users will not be able to host binaries via HTTP; even if they have
> a server, the bandwidth requirements are hard to predict and likely
> to be too high.

I'm not sure to understand why you precise "BitTorrent" magnet links.
Magnet is a de-facto standard that is independant from BitTorrent [3]
[4], and using BitTorrent in our case wouldn't have better performance
than GNUnet's MESH. The only user case when BitTorrent might come in
handy is when one wants to download a binary (as one would download a
.deb) before manually installing it with guix ; i suppose this can
also be done through GNUnet. Then of course it's just a user case:
users might be more familiar with BitTorrent.

[3] http://magnet-uri.sourceforge.net/
[4] http://magnet-uri.sourceforge.net/magnet-draft-overview.txt

I hope this makes sense :)

- - rigelk
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.22 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBAgAGBQJTHrRnAAoJEHfJ0QE7gLd6mcMH/A3dlgu2UAJKo3JzR7RSdf8M
2V4xxw4d+vNNPqpIBRYsfJUVkRxePmAljntzsgTQ59AbibAjMt5uXI2htqYAGFia
nsXvSIv0YkkPoE15morUtL8XYohRNvkSmaSKR2EqX/+Vxxm7C3Ftni4tlIbLB13k
afK/ExDYV2+SoASS5myPqCaK3jusCmCeXlew2aDInbn7owGgtA4dxvti5TtfGaUi
IMp/xXb3jimzIyzWbX+rKCvouVRK+ncdBxGzw+kPMtE6h5hja70M0Mw6IaREvJqA
dj9mb5YU+OUoLrayVM/PcTML2gqYHhoXkSDQzqsWHLfbtxq0ymUWcnOe2d/WdGY=
=JIQ8
-----END PGP SIGNATURE-----

_______________________________________________
GNUnet-developers mailing list
GNUnet-developers@gnu.org
https://lists.gnu.org/mailman/listinfo/gnunet-developers

Re: [GSoC] GNUnet binary distribution system

[Christian Grothoff]

[-- Attachment #1.1.1: Type: text/plain, Size: 1015 bytes --]

On 03/11/2014 12:19 AM, Pierre-Antoine Rault wrote:
>> The initial discussion [0] left open the question of where
>> binaries themselves should be stored.  A possibility would be to
>> use GNUnet’s DHT simply as a discovery mechanism, and then to
>> establish a connection directly to the user’s machine, which would
>> run, say, an HTTP server.
> 
> That's what I had in mind. Now, considered the post [2] by Christian
> Grothoff, we might consider using either an HTTP server for
> performance or GNUnet's MESH for anonymity (and security). We should
> balance needs and ease of implementation.

Just to clarify here, I never said MESH would give you anonymity.
I only said it should give you better NAT traversal and better
authenticity and confidentiality than TLS/HTTPS (largely because you
avoid the CA mess).  Good anonymity will require more than MESH, and for
Guix I am not sure that having anonymity is necessary (it might of
course be fine to have that as an option in the future).

[-- Attachment #1.1.2: 0x48426C7E.asc --]
[-- Type: application/pgp-keys, Size: 14446 bytes --]

[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 242 bytes --]

[-- Attachment #2: Type: text/plain, Size: 162 bytes --]

_______________________________________________
GNUnet-developers mailing list
GNUnet-developers@gnu.org
https://lists.gnu.org/mailman/listinfo/gnunet-developers

Re: [GSoC] GNUnet binary distribution system

[Ludovic Courtès]

Pierre-Antoine Rault <par@rigelk.eu> skribis:

> On 10/03/2014 22:09, Ludovic Courtès wrote:

[...]

>> The initial discussion [0] left open the question of where
>> binaries themselves should be stored.  A possibility would be to
>> use GNUnet’s DHT simply as a discovery mechanism, and then to
>> establish a connection directly to the user’s machine, which would
>> run, say, an HTTP server.
>
> That's what I had in mind. Now, considered the post [2] by Christian
> Grothoff, we might consider using either an HTTP server for
> performance or GNUnet's MESH for anonymity (and security). We should
> balance needs and ease of implementation.
>
> [2] http://lists.gnu.org/archive/html/guix-devel/2014-03/msg00113.html

Yes.

Could you experiment with the MESH interface, and sketch how files would
be served concretely?  You could use the ‘gnunet-mesh’ command-line tool
for quick experimentation, although I haven’t found its documentation.

[...]

>>> * I think we should focus on ease of use, since it could become a
>>> good alternative to setting up a full fledged server to share
>>> officially supported packages and unofficial ones (like the AUR
>>> for Archlinux).
>>
>> That’s been discussed before (see
>> <http://lists.gnu.org/archive/html/guix-devel/2013-08/msg00127.html>),
>>
>>
> but it’s orthogonal.
>
> Then we have to extend the guix tools to cover user modules (overlays,
> that is).

Yes, but as I said, this is unrelated to the project at hand.

>> This proposal needs discussion with both Guix and GNUnet people
>> (and notably Sree Harsha, who is at the intersection of both
>> projects :-)), to work towards a concrete road map of things to
>> hack on.
>
> Sure ; It's the only way to have a clear and shared view of what parts
> of Guix are involved in the project. I'm working on a roadmap draft
> for now.

Great.  We can’t start this as a GSoC project unless we have a
reasonably clear view of what needs to be implemented.  We must not end
up re-discussing the general design once the coding period has started.
There are still a number of open issues, so it’s important to refine
that.

Thanks,
Ludo’.

_______________________________________________
GNUnet-developers mailing list
GNUnet-developers@gnu.org
https://lists.gnu.org/mailman/listinfo/gnunet-developers

Re: [GSoC] GNUnet binary distribution system

[Mark H Weaver]

Pierre-Antoine Rault <par@rigelk.eu> writes:

> On 11/03/2014 04:35, Mark H Weaver wrote:

>> FWIW, I think it might be worthwhile to support BitTorrent magnet
>> links as well, as a middle ground between these two extremes.  Most
>> users will not be able to host binaries via HTTP; even if they have
>> a server, the bandwidth requirements are hard to predict and likely
>> to be too high.
>
> I'm not sure to understand why you precise "BitTorrent" magnet links.
> Magnet is a de-facto standard that is independant from BitTorrent [3]
> [4],

I specified "BitTorrent" because it would presumably simplify our
implementation to support only one protocol.

> and using BitTorrent in our case wouldn't have better performance
> than GNUnet's MESH.

Okay.  My suggestion was based on a guess that GNUnet's more ambitious
goals (e.g. providing anonymity) would entail a significant performance
cost compared with BitTorrent.  If that's not the case, then I see no
reason to use BitTorrent.  I'd certainly prefer to support GNUnet.

    Thanks,
      Mark