gnunet-fs integration

gnunet-fs integration

[ng0]

Hi,

While I'm still new to guile and guix, one of my main motivations
since I started contributing to guix was an item on the end of
the roadmap, using a decentralized system to distribute updates.

I have started to read through the scripts and underlying parts
of guix, and also our guix/gnunet.git repository where some
work already happened in GsoC 2015.

Because this is very likely short, not very explanatory in
detail, incomprehendable, you can view this email as a short
introduction of the task, longer emails will follow once I
have worked out texts and completed phase 0.

At the moment I assume that the following items and steps are
what's left to get closer to this system:

1. an up to date GNUnet
   a. 0.10.1 of gnunet-fs is not compatible to 0.9,
      gnunet-svn of gnunet-fs is not compatible to 0.10.1
      https://gnunet.org/bot/log/gnunet/2016-06-23#T1067897

2. GNUnet system service for shepherd
   a. it needs a basic service to start with, ideally we
      provide what gnunet-setup does through an extended
      system-service later on

3. stitching together the guile GNUnet bindings
   with additions to mechanisms which are
   at least guix {publish,pull,package}.

4. taking in consideration the current incompabilities of
   gnunet-fs versions.

5. -optional-
   construct `gnunetfs-download', similiar to git-download
   and others, for origins (of software) which are available
   in gnunet.

I have concrete ideas for solutions, interactions and
users consent and more, but those will end up in an
dedicated document which can be viewed online and/or
appended to an email later.


I will split the work into different phases:

Phase 0:
- learning about system services
- write the shepherd gnunet-service
- finish packaging gnunet-svn pinned to a functional svn position
- learn more guile (continously)
- a first theoretic design

Phase 1:
- learn more about the current guix publish, guix pull,
  guix package functionality
- solve design questions

Phase 2:
- writing a proto-type which can be used in a test setup

Phase 3:
- have a functional test setup which can (guix) publish, pull,
  and package from gnunet-fs in addition to the current
  methods


I will publish Phase 1 and later in a checkout of guix
with a branch `guix-gnunet', accessible once they are
created on both of these git urls:
git://git.far37qbrwiredyo5.onion:/guix.git
git://s.n0.is:/guix.git
optionally even a .gnu address when I had time to set it
up on the server


The current goal for Phase 3 is summer 2017, but I can't
predict how much time I can invest in the future or how
fast I will progress with learning.

--
♥Ⓐ  ng0
For non-prism friendly talk find me on
psyced.org / loupsycedyglgamf.onion

Re: gnunet-fs integration

[Ludovic Courtès]

Hello,

ng0@we.make.ritual.n0.is skribis:

> While I'm still new to guile and guix, one of my main motivations
> since I started contributing to guix was an item on the end of
> the roadmap, using a decentralized system to distribute updates.

I think many of us dream of such a thing.  :-)

> At the moment I assume that the following items and steps are
> what's left to get closer to this system:
>
> 1. an up to date GNUnet
>    a. 0.10.1 of gnunet-fs is not compatible to 0.9,
>       gnunet-svn of gnunet-fs is not compatible to 0.10.1
>       https://gnunet.org/bot/log/gnunet/2016-06-23#T1067897

Unfortunately GNUnet hasn’t seen any release since April 2014, which I
find worrying.

It may be worth considering other options such as IPFS and Bittorrent,
even though they obviously have different properties.

> 2. GNUnet system service for shepherd
>    a. it needs a basic service to start with, ideally we
>       provide what gnunet-setup does through an extended
>       system-service later on

This would be could.  I discussed it with tg(x) in November but never
took the time to actually do it.

> 3. stitching together the guile GNUnet bindings
>    with additions to mechanisms which are
>    at least guix {publish,pull,package}.

Note that Rémi, who worked on this for GSoC, posted prototypes for the
publish and substitute code (along with the bindings themselves, which
are available as the ‘guile-gnunet’ package in Guix.)  It’s probably a
good idea to revive them and see how far you can get.

Glad to see more work in this area!

Ludo’.

Re: gnunet-fs integration

[ng0]

Hi,

Ludovic Courtès writes:

> Hello,
>
> ng0@we.make.ritual.n0.is skribis:
>
>> While I'm still new to guile and guix, one of my main motivations
>> since I started contributing to guix was an item on the end of
>> the roadmap, using a decentralized system to distribute updates.
>
> I think many of us dream of such a thing.  :-)
>
>> At the moment I assume that the following items and steps are
>> what's left to get closer to this system:
>>
>> 1. an up to date GNUnet
>>    a. 0.10.1 of gnunet-fs is not compatible to 0.9,
>>       gnunet-svn of gnunet-fs is not compatible to 0.10.1
>>       https://gnunet.org/bot/log/gnunet/2016-06-23#T1067897
>
> Unfortunately GNUnet hasn’t seen any release since April 2014, which I
> find worrying.

Work is done towards 0.10.2, there are just some bugtickets left
but I can assure you that work is happening regulary and its
alive and well :)
I take the current incompabilities between 0.10.1 and svn in
consideration with the writing of the distributing system. from
what I know, I think it will be fixed with the release of 0.10.2

https://gnunet.org/bugs/roadmap_page.php
I'm fairly positive that 0.10.2 will be released soon, but I'll
ask Christian if there's a new ETA now.
Mantis says for gnunet 303 of 308 issue(s) resolved. Progress
(98%). for gnunet-gtk: 100% for 0.10.2

> It may be worth considering other options such as IPFS and Bittorrent,
> even though they obviously have different properties.

That's true, but I chose gnunet-fs because I have more insight
and participation in gnunet than in bittorrent or ipfs.
For bittorrent I'd even consider the trackerless torrent variant
which is popular now.
But my main motivation is gnunet, which I will base this upon.

>> 2. GNUnet system service for shepherd
>>    a. it needs a basic service to start with, ideally we
>>       provide what gnunet-setup does through an extended
>>       system-service later on
>
> This would be could.  I discussed it with tg(x) in November but never
> took the time to actually do it.
>
>> 3. stitching together the guile GNUnet bindings
>>    with additions to mechanisms which are
>>    at least guix {publish,pull,package}.
>
> Note that Rémi, who worked on this for GSoC, posted prototypes for the
> publish and substitute code (along with the bindings themselves, which
> are available as the ‘guile-gnunet’ package in Guix.)  It’s probably a
> good idea to revive them and see how far you can get.

Oh, I did not read everything of it all yet.
I added one more optional item to my list, which is add a method
for (source (origin)) to make use of the two relevant hash
methods which gnunet-fs uses.

If it was just for me, I'd not use 0.10.1 and go
straight for -svn pinned to a version and switch to 0.10.2 once
it is released, but I'd like to test and look for possibilities
to stay compatible. Maybe an 0.10.2 release happens while I write
this.

> Glad to see more work in this area!
>
> Ludo’.

--
♥Ⓐ  ng0
For non-prism friendly talk find me on
psyced.org / loupsycedyglgamf.onion

Re: gnunet-fs integration

[ng0]

Hi,

read an update inline commented.

ng0 writes:

> Hi,
>
> Ludovic Courtès writes:
>
>> Hello,
>>
>> ng0@we.make.ritual.n0.is skribis:
>>
>>> While I'm still new to guile and guix, one of my main motivations
>>> since I started contributing to guix was an item on the end of
>>> the roadmap, using a decentralized system to distribute updates.
>>
>> I think many of us dream of such a thing.  :-)
>>
>>> At the moment I assume that the following items and steps are
>>> what's left to get closer to this system:
>>>
>>> 1. an up to date GNUnet
>>>    a. 0.10.1 of gnunet-fs is not compatible to 0.9,
>>>       gnunet-svn of gnunet-fs is not compatible to 0.10.1
>>>       https://gnunet.org/bot/log/gnunet/2016-06-23#T1067897
>>
>> Unfortunately GNUnet hasn’t seen any release since April 2014, which I
>> find worrying.
>
> Work is done towards 0.10.2, there are just some bugtickets left
> but I can assure you that work is happening regulary and its
> alive and well :)
> I take the current incompabilities between 0.10.1 and svn in
> consideration with the writing of the distributing system. from
> what I know, I think it will be fixed with the release of 0.10.2
>
> https://gnunet.org/bugs/roadmap_page.php
> I'm fairly positive that 0.10.2 will be released soon, but I'll
> ask Christian if there's a new ETA now.
> Mantis says for gnunet 303 of 308 issue(s) resolved. Progress
> (98%). for gnunet-gtk: 100% for 0.10.2

Update: currently some refactoring is happening, grothf just
started refactoring the client api which requires every service
to be modified, which takes some time.
There's no guesstimated date, but we should not expect an release
before the end of summer.

>> It may be worth considering other options such as IPFS and Bittorrent,
>> even though they obviously have different properties.
>
> That's true, but I chose gnunet-fs because I have more insight
> and participation in gnunet than in bittorrent or ipfs.
> For bittorrent I'd even consider the trackerless torrent variant
> which is popular now.
> But my main motivation is gnunet, which I will base this upon.
>
>>> 2. GNUnet system service for shepherd
>>>    a. it needs a basic service to start with, ideally we
>>>       provide what gnunet-setup does through an extended
>>>       system-service later on
>>
>> This would be could.  I discussed it with tg(x) in November but never
>> took the time to actually do it.
>>
>>> 3. stitching together the guile GNUnet bindings
>>>    with additions to mechanisms which are
>>>    at least guix {publish,pull,package}.
>>
>> Note that Rémi, who worked on this for GSoC, posted prototypes for the
>> publish and substitute code (along with the bindings themselves, which
>> are available as the ‘guile-gnunet’ package in Guix.)  It’s probably a
>> good idea to revive them and see how far you can get.
>
> Oh, I did not read everything of it all yet.
> I added one more optional item to my list, which is add a method
> for (source (origin)) to make use of the two relevant hash
> methods which gnunet-fs uses.
>
> If it was just for me, I'd not use 0.10.1 and go
> straight for -svn pinned to a version and switch to 0.10.2 once
> it is released, but I'd like to test and look for possibilities
> to stay compatible. Maybe an 0.10.2 release happens while I write
> this.
>
>> Glad to see more work in this area!
>>
>> Ludo’.
>
> --
> ♥Ⓐ  ng0
> For non-prism friendly talk find me on
> psyced.org / loupsycedyglgamf.onion
>

-- 
♥Ⓐ  ng0
For non-prism friendly talk find me on
psyced.org / loupsycedyglgamf.onion

Re: gnunet-fs integration

[ng0]

ng0@we.make.ritual.n0.is writes:

> Hi,
>
> While I'm still new to guile and guix, one of my main motivations
> since I started contributing to guix was an item on the end of
> the roadmap, using a decentralized system to distribute updates.
>
> I have started to read through the scripts and underlying parts
> of guix, and also our guix/gnunet.git repository where some
> work already happened in GsoC 2015.
>
> Because this is very likely short, not very explanatory in
> detail, incomprehendable, you can view this email as a short
> introduction of the task, longer emails will follow once I
> have worked out texts and completed phase 0.
>
> At the moment I assume that the following items and steps are
> what's left to get closer to this system:
>
> 1. an up to date GNUnet
>    a. 0.10.1 of gnunet-fs is not compatible to 0.9,
>       gnunet-svn of gnunet-fs is not compatible to 0.10.1
>       https://gnunet.org/bot/log/gnunet/2016-06-23#T1067897

This one will be solved with 0.10.2 release of GNUnet...

> 2. GNUnet system service for shepherd
>    a. it needs a basic service to start with, ideally we
>       provide what gnunet-setup does through an extended
>       system-service later on
>
> 3. stitching together the guile GNUnet bindings
>    with additions to mechanisms which are
>    at least guix {publish,pull,package}.
>
> 4. taking in consideration the current incompabilities of
>    gnunet-fs versions.

...which is why I no longer need to beware incompabilities.

> 5. -optional-
>    construct `gnunetfs-download', similiar to git-download
>    and others, for origins (of software) which are available
>    in gnunet.
>
> I have concrete ideas for solutions, interactions and
> users consent and more, but those will end up in an
> dedicated document which can be viewed online and/or
> appended to an email later.
>
>
> I will split the work into different phases:
>
> Phase 0:
> - learning about system services
> - write the shepherd gnunet-service

no longer needed, as this is just a package to debug 0.10.2 until it is
released. Afterwards we'll see if -svn still makes sense to be in tree.
> - finish packaging gnunet-svn pinned to a functional svn position

> - learn more guile (continously)
> - a first theoretic design
>
> Phase 1:
> - learn more about the current guix publish, guix pull,
>   guix package functionality
> - solve design questions
>
> Phase 2:
> - writing a proto-type which can be used in a test setup
>
> Phase 3:
> - have a functional test setup which can (guix) publish, pull,
>   and package from gnunet-fs in addition to the current
>   methods
>
>
> I will publish Phase 1 and later in a checkout of guix
> with a branch `guix-gnunet', accessible once they are
> created on both of these git urls:
> git://git.far37qbrwiredyo5.onion:/guix.git
> git://s.n0.is:/guix.git
> optionally even a .gnu address when I had time to set it
> up on the server
>
>
> The current goal for Phase 3 is summer 2017, but I can't
> predict how much time I can invest in the future or how
> fast I will progress with learning.

-- 
♥Ⓐ  ng0
Current Keys: https://we.make.ritual.n0.is/ng0.txt
For non-prism friendly talk find me on http://www.psyced.org

Re: gnunet-fs integration

[ng0]

ng0@we.make.ritual.n0.is writes:

> Hi,
>
> Ludovic Courtès writes:
>
>> Hello,
>>
>> ng0@we.make.ritual.n0.is skribis:
>>
>>> While I'm still new to guile and guix, one of my main motivations
>>> since I started contributing to guix was an item on the end of
>>> the roadmap, using a decentralized system to distribute updates.
>>
>> I think many of us dream of such a thing.  :-)
>>
>>> At the moment I assume that the following items and steps are
>>> what's left to get closer to this system:
>>>
>>> 1. an up to date GNUnet
>>>    a. 0.10.1 of gnunet-fs is not compatible to 0.9,
>>>       gnunet-svn of gnunet-fs is not compatible to 0.10.1
>>>       https://gnunet.org/bot/log/gnunet/2016-06-23#T1067897
>>
>> Unfortunately GNUnet hasn’t seen any release since April 2014, which I
>> find worrying.
>
> Work is done towards 0.10.2, there are just some bugtickets left
> but I can assure you that work is happening regulary and its
> alive and well :)
> I take the current incompabilities between 0.10.1 and svn in
> consideration with the writing of the distributing system. from
> what I know, I think it will be fixed with the release of 0.10.2
>
> https://gnunet.org/bugs/roadmap_page.php
> I'm fairly positive that 0.10.2 will be released soon, but I'll
> ask Christian if there's a new ETA now.
> Mantis says for gnunet 303 of 308 issue(s) resolved. Progress
> (98%). for gnunet-gtk: 100% for 0.10.2
>
>> It may be worth considering other options such as IPFS and Bittorrent,
>> even though they obviously have different properties.

There's also a discussion/open bug about finding a system for more
regular tarball releases of gnunet builds. As you might or might not
know, the release is equal to a certain revision in the svn of
gnunet.org with no additional changes.

There was a discussion on this (ipfs) recently on psyced.org. This is
public and can be found after accessing psyced.org chatserver with
"/hist 500" or "+hist 500" depending on your client and the lines said
after I wrote this email.
The tl;dr is that ipfs lacks the anonymity gnunet-fs can offer, but
communication between development teams does happen from time to time or
is desired. For my use case it does not matter how well furbished the UI
of it is, Guix users will never be exposed to gnunet-download, gnunet-fs,
gnunet-gtk etc unless they choose so on their own.

I can't point to concrete papers at the moment (preoccupied with the
operation in 2 hours), but overall I know that choosing gnunet-fs, from
the papers I've read and the people I've discussed with, the publishing
and downloading of files will be secure and anonymous.

> That's true, but I chose gnunet-fs because I have more insight
> and participation in gnunet than in bittorrent or ipfs.
> For bittorrent I'd even consider the trackerless torrent variant
> which is popular now.
> But my main motivation is gnunet, which I will base this upon.
>
>>> 2. GNUnet system service for shepherd
>>>    a. it needs a basic service to start with, ideally we
>>>       provide what gnunet-setup does through an extended
>>>       system-service later on
>>
>> This would be could.  I discussed it with tg(x) in November but never
>> took the time to actually do it.
>>
>>> 3. stitching together the guile GNUnet bindings
>>>    with additions to mechanisms which are
>>>    at least guix {publish,pull,package}.
>>
>> Note that Rémi, who worked on this for GSoC, posted prototypes for the
>> publish and substitute code (along with the bindings themselves, which
>> are available as the ‘guile-gnunet’ package in Guix.)  It’s probably a
>> good idea to revive them and see how far you can get.
>
> Oh, I did not read everything of it all yet.
> I added one more optional item to my list, which is add a method
> for (source (origin)) to make use of the two relevant hash
> methods which gnunet-fs uses.
>
> If it was just for me, I'd not use 0.10.1 and go
> straight for -svn pinned to a version and switch to 0.10.2 once
> it is released, but I'd like to test and look for possibilities
> to stay compatible. Maybe an 0.10.2 release happens while I write
> this.
>
>> Glad to see more work in this area!
>>
>> Ludo’.
>

-- 
♥Ⓐ  ng0
Current Keys: https://we.make.ritual.n0.is/ng0.txt
For non-prism friendly talk find me on http://www.psyced.org