[GSoC] Guix + GNUnet: what’s next

[GSoC] Guix + GNUnet: what’s next

[Rémi Birot-Delrue]

Hello Guix! Here’s a little update.

The current binary package distribution (through HTTP) is based on two
components: a substituer (that retrieves packages) and a publisher. The
current plan is to start with adapting the publisher for GNUnet before
attacking the substituer; this work will be started once I’ll have
successfully bound the download and publish operations.

Here’s an overview of how these components will work:

After a call to `guix package -i foo`, the GNUnet substituer will search
GNUnet for a specific build of the `foo` package and if it finds one, it
will download it and install it (as if it had been downloaded through
HTTP).

An interesting point is handling of metadata: in Guix (and Nix), each
binary package’s metadata (size, signature, compression method…) is
downloaded separatedly, before its archive. In the HTTP substituer, that
means you have to download two files: one for the metadata, one for the
archive. In the GNUnet substituer, we take advantage of the functionning
of GNUnet: when you search for a file, some metadata is automatically
retrieved (its size, signature, direct access URI…) with the search
result, and an archive’s metadata can be shipped along.

In order to make this work, the publisher will basically inline the
`narinfo' file of each archive into its metadata before indexing
(uploading) it on GNUnet.

As usual, feel free to contact me for any question, suggestion, advice,
idea, critic or joke.
-- 
Rémi Birot-Delrue

Re: [GSoC] Guix + GNUnet: what’s next

[Ludovic Courtès]

Hi Rémi,

Thanks for the update!

So I can see two milestones now, as we discussed before:

  1. Create a variant of ‘guix publish’ that publishes over GNUnet’s
     file sharing (FS) service, using the neat bindings that you wrote.

     For that you can literally copy guix/scripts/publish.scm as a
     starting point, and simply remove the HTTP-related code (which is a
     small fraction.)  The rest will be mostly identical: narinfo
     meta-data generation, signing.  I guess it will be easy to test it
     using gnunet-search and similar tools.

     As a 2nd step, we’ll see how we can refactor things to allow code
     sharing between the existing ‘guix publish’ and its GNUnet variant.

  2. Create a variant of ‘guix substitute’ for searches through GNUnet.
     Again, a large part of the code is about narinfo and signature
     checking, and it can be reused.

I’m not completely clear on how search for substitutes will work,
though.  Currently, when the user wants to build /gnu/store/xyz, ‘guix
substitute’ simply fetches http://hydra.gnu.org/xyz.narinfo.  How will
that work with GNUnet?  Are we going to look up their /gnu/store file
name?

Thanks,
Ludo’.

Re: [GSoC] Guix + GNUnet: what’s next

[Rémi Birot-Delrue]

Ludovic Courtès writes:

> I’m not completely clear on how search for substitutes will work,
> though.  Currently, when the user wants to build /gnu/store/xyz, ‘guix
> substitute’ simply fetches http://hydra.gnu.org/xyz.narinfo.  How will
> that work with GNUnet?  Are we going to look up their /gnu/store file
> name?

I’ve considered a solution for that: GNUnet allows one to create
specific namespace and publish files under this namespace. Unlike
publishing under the “global namespace” where keywords are used to
identify a file, when publishing under a specific namespace files are
identified with a choosen identifier. Moreover, as a namespace is
basically a cryptographic key pair, and publishing a file under your
namespace means signing, one’s assured nobody else will publish under
her or his namespace. By the way, the private key associated with a
namespace is named “ego” or “pseudonym”.

It’s easy to test this feature:

    # create a `test` ego/namespace
    $ gnunet-identity -C test

    # list the known egos in the form: `name - public key`
    $ gnunet-identity -d
    test - M2OC987U9LFJHQ8LC9SLCV4Q0ONHJV7FMTFQ2VRPE0M9R9MK5860
    …

    # index the file `foo.txt` under the `test` namespace
    $ gnunet-publish -P test -t foobarbaz foo.txt

    # find the file `foo.txt`
    $ gnunet-search gnunet://fs/sks/M2OC987U9LFJHQ8LC9SLCV4…/foobarbaz
    #0:
    gnunet-download -o "foo.txt" gnunet://fs/chk/PL217ODD8EDSMOIQ3UT0…

Now if Alice wants to publish her binaries, she creates an ego/namespace
and publishes everything under it; Bob adds her namespace’s public key
to his authorized substitutes list, and when installing `/gnu/store/xyz`
the substitute will search for `gnunet://fs/sks/<Alice’s key>/xyz`.

Instead of publishing an archive we might also directly publish/index
the build, but I don’t know if it’s viable.

Does it seem right to you?
-- 
Rémi Birot-Delrue

Re: [GSoC] Guix + GNUnet: what’s next

[Ludovic Courtès]

(Sorry, I had misspelled the GNUnet mailing list address.)

Rémi Birot-Delrue <asgeir@free.fr> skribis:

> Ludovic Courtès writes:
>
>> I’m not completely clear on how search for substitutes will work,
>> though.  Currently, when the user wants to build /gnu/store/xyz, ‘guix
>> substitute’ simply fetches http://hydra.gnu.org/xyz.narinfo.  How will
>> that work with GNUnet?  Are we going to look up their /gnu/store file
>> name?
>
> I’ve considered a solution for that: GNUnet allows one to create
> specific namespace and publish files under this namespace. Unlike
> publishing under the “global namespace” where keywords are used to
> identify a file, when publishing under a specific namespace files are
> identified with a choosen identifier. Moreover, as a namespace is
> basically a cryptographic key pair, and publishing a file under your
> namespace means signing, one’s assured nobody else will publish under
> her or his namespace. By the way, the private key associated with a
> namespace is named “ego” or “pseudonym”.
>
> It’s easy to test this feature:
>
>     # create a `test` ego/namespace
>     $ gnunet-identity -C test
>
>     # list the known egos in the form: `name - public key`
>     $ gnunet-identity -d
>     test - M2OC987U9LFJHQ8LC9SLCV4Q0ONHJV7FMTFQ2VRPE0M9R9MK5860
>     …
>
>     # index the file `foo.txt` under the `test` namespace
>     $ gnunet-publish -P test -t foobarbaz foo.txt
>
>     # find the file `foo.txt`
>     $ gnunet-search gnunet://fs/sks/M2OC987U9LFJHQ8LC9SLCV4…/foobarbaz
>     #0:
>     gnunet-download -o "foo.txt" gnunet://fs/chk/PL217ODD8EDSMOIQ3UT0…
>
> Now if Alice wants to publish her binaries, she creates an ego/namespace
> and publishes everything under it; Bob adds her namespace’s public key
> to his authorized substitutes list, and when installing `/gnu/store/xyz`
> the substitute will search for `gnunet://fs/sks/<Alice’s key>/xyz`.
>
> Instead of publishing an archive we might also directly publish/index
> the build, but I don’t know if it’s viable.
>
> Does it seem right to you?

Sounds good!  I wasn’t use whether the ID passed to ‘gnunet-publish -t’
could be used as a key to look up the data, but that seems to be the
case.

Thanks,
Ludo’.

_______________________________________________
GNUnet-developers mailing list
GNUnet-developers@gnu.org
https://lists.gnu.org/mailman/listinfo/gnunet-developers

Re: [OSM-dev] [GNUnet-developers] [GSoC] Guix + GNUnet: what’s next

[Daniel Golle]

Hi!

Following the discussion about using GNUnet for software package
distribution I started thinking about what kind of data *read access*
traces matter to people's privacy.
Map data bumped into my mind, because it's obvious that where one
zooms down and the places one searches for tell alot about ones
life.
Thus it might be desireable for users to access map data anonymously
e.g. using GNUnet.

On Thu, Jul 02, 2015 at 10:32:25PM +0200, Ludovic Courtès wrote:
> > It’s easy to test this feature:
> >
> >     # create a `test` ego/namespace
> >     $ gnunet-identity -C test
> >
> >     # list the known egos in the form: `name - public key`
> >     $ gnunet-identity -d
> >     test - M2OC987U9LFJHQ8LC9SLCV4Q0ONHJV7FMTFQ2VRPE0M9R9MK5860
> >     …
> >
> >     # index the file `foo.txt` under the `test` namespace
> >     $ gnunet-publish -P test -t foobarbaz foo.txt
> >
> >     # find the file `foo.txt`
> >     $ gnunet-search gnunet://fs/sks/M2OC987U9LFJHQ8LC9SLCV4…/foobarbaz
> >     #0:
> >     gnunet-download -o "foo.txt" gnunet://fs/chk/PL217ODD8EDSMOIQ3UT0…
> >
> > Now if Alice wants to publish her binaries, she creates an ego/namespace
> > and publishes everything under it; Bob adds her namespace’s public key
> > to his authorized substitutes list, and when installing `/gnu/store/xyz`
> > the substitute will search for `gnunet://fs/sks/<Alice’s key>/xyz`.
> >
> > Instead of publishing an archive we might also directly publish/index
> > the build, but I don’t know if it’s viable.
> >
> > Does it seem right to you?
> 
> Sounds good!  I wasn’t use whether the ID passed to ‘gnunet-publish -t’
> could be used as a key to look up the data, but that seems to be the
> case.

This sounds very useful eg. for Planet.osm distribution.
http://wiki.openstreetmap.org/wiki/Planet.osm
It's already available via BitTorrent
http://www.mantlik.cz/tracker/torrent_functions.php
but the regions are very large and thus the files are too...

Anonymous access to local OSM data as well as (cached) rendered map
tiles would greatly improve users privacy, obviously...

I'd greatly appreciaty a discussion on how Planet.osm can make it
into GNUnet and how that API could be promoted to the application
development community.

(I'm aware that one can just use TOR to access OSM, but that would
still allow a sufficiently resourceful 3rd party to associate the
timing of TCP connections on both ends and de-anonymize TOR users)


Cheers


Daniel

_______________________________________________
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] [GNUnet-developers] [GSoC] Guix + GNUnet: what’s next

[Paul Norman]

On 7/2/2015 2:58 PM, Daniel Golle wrote:
> This sounds very useful eg. for Planet.osm distribution.
> http://wiki.openstreetmap.org/wiki/Planet.osm
> It's already available via BitTorrent
> http://www.mantlik.cz/tracker/torrent_functions.php
> but the regions are very large and thus the files are too...
In practice using the torrent files results in most of the data from 
webseed, getting it from planet.openstreetmap.org. They can help a bit, 
but don't really form a parallel distribution system.

> Anonymous access to local OSM data as well as (cached) rendered map
> tiles would greatly improve users privacy, obviously...
With rendering on demand like tile.osm.org has, tiles might only be 
cached for 5 minutes. This is probably going to be a problem for any 
truly anonymous system. Also, to avoid the tile renderer knowing what 
you're accessing, everything has to be pre-rendered. Pre-rendered 
world-wide vector tiles are not a practical option.

This also leaves aside the fact that such a system probably isn't 
suitable for tile.osm.org, where the focus is on supporting map edits, 
and viewing the map is generally going to be associated with having 
edited the same area.

None of this stops someone from operating their own rendering server, 
even behind an air-gapped network. If someone downloads the planet dump, 
you don't know what part of it they're interested in, and even if 
consuming minutely updates, it still doesn't reveal where they are 
looking at.

_______________________________________________
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] [GSoC] Guix + GNUnet: what’s next

[Daniel Golle]

Hi Paul!

Thank you for your instant reply!


On Thu, Jul 02, 2015 at 03:56:57PM -0700, Paul Norman wrote:
> On 7/2/2015 2:58 PM, Daniel Golle wrote:
> >This sounds very useful eg. for Planet.osm distribution.
> >http://wiki.openstreetmap.org/wiki/Planet.osm
> >It's already available via BitTorrent
> >http://www.mantlik.cz/tracker/torrent_functions.php
> >but the regions are very large and thus the files are too...
> In practice using the torrent files results in most of the data from
> webseed, getting it from planet.openstreetmap.org. They can help a bit, but
> don't really form a parallel distribution system.

Imho that could be greatly improved by introducing a partitioning scheme
for Planet.osm splitting it up into a lot of small files (e.g. max 10 megs)
and providing an index. I'm aware that this is a non-trivial problem...

> >Anonymous access to local OSM data as well as (cached) rendered map
> >tiles would greatly improve users privacy, obviously...
> With rendering on demand like tile.osm.org has, tiles might only be cached
> for 5 minutes. This is probably going to be a problem for any truly
> anonymous system. Also, to avoid the tile renderer knowing what you're
> accessing, everything has to be pre-rendered. Pre-rendered world-wide vector
> tiles are not a practical option.

I reckon that renderring locally is the only feasible choice then.

> This also leaves aside the fact that such a system probably isn't suitable
> for tile.osm.org, where the focus is on supporting map edits, and viewing
> the map is generally going to be associated with having edited the same
> area.

Regular read-only snapshots being available for anonymous access
without having to download several gigabytes of data would still be
nice and much better than not having that option, leaving freshness
and privacy concerns when editing aside for now.


> None of this stops someone from operating their own rendering server, even
> behind an air-gapped network. If someone downloads the planet dump, you
> don't know what part of it they're interested in, and even if consuming
> minutely updates, it still doesn't reveal where they are looking at.

Sure, but this is not something I'd expect regular users to do, as most
people neither have the knowledge nor the resources to do that.


Cheers


Daniel