[GSoC] Draft of my proposition

[GSoC] Draft of my proposition

[vincent]

[-- Attachment #1.1: Type: text/plain, Size: 592 bytes --]

Hi everyone!

The admission for Google's Summer of Code have started and I want to spend 
the summer working on Guix. I have a proposition I hope you will find 
interesting! I have joined the draft to this e-mail.

I would really like if you could point out the part where I don't explain 
myself very well, if some French syntax has slipped through or if there is 
something that should be added to meet GNU's guideline.

Also I don't if any mentors are available for this project, and who I should 
talk to about that.

Don't hesitate if you have any question.
Thank you,

Vincent Cloutier

[-- Attachment #1.2: Type: text/html, Size: 775 bytes --]

[-- Attachment #2: GSoC guix.odt --]
[-- Type: application/vnd.oasis.opendocument.text, Size: 20243 bytes --]

Re: [GSoC] Draft of my proposition

[Ludovic Courtès]

Hello!

<vincent@cloutier.co> skribis:

> The admission for Google's Summer of Code have started and I want to spend 
> the summer working on Guix. I have a proposition I hope you will find 
> interesting! I have joined the draft to this e-mail.

Thanks for your proposal!

Preliminary question: could you resend it as plain text or Org-mode to
simply the life of people on this list?

Thanks in advance,  :-)
Ludo’.

Re: [GSoC] Draft of my proposition

[vincent]

[-- Attachment #1.1: Type: text/plain, Size: 496 bytes --]




21. Mar 2016 05:35 by ludo@gnu.org:


> Hello!
>
> <> vincent@cloutier.co> > skribis:
>
>> The admission for Google's Summer of Code have started and I want to spend
>> the summer working on Guix. I have a proposition I hope you will find
>> interesting! I have joined the draft to this e-mail.
> Preliminary question: could you resend it as plain text or Org-mode to
> simply the life of people on this list?
>
>

Of course! I added a plain text version to this email.





Vincent Cloutier


[-- Attachment #1.2: Type: text/html, Size: 1082 bytes --]

[-- Attachment #2: GSoC.txt --]
[-- Type: text/plain, Size: 4732 bytes --]

Guix – Adding peer-to-peer sharing
Vincent Cloutier
vincent@cloutier.co

Abstract

Large performance gains and servers savings could be made by fetching data from peers instead of a central set of servers. Since Guix users know in advance the hash of the data they want, downloading from peers has no security implications (and privacy can be done trough proxies).  

What I want to build

I have a fascination for peer-to-peer tech and I am constantly looking for the innovative new tech in this area (Bitcoin, Ethereum, etc). Less than a year ago I discovered IPFS, a project that takes the best ideas from BitTorrent and Git to create a simple and elegant protocol. 

IPFS allows one to find who has a piece of content and is ready to share it, when knowing only the content’s hash. Content is added in a reproducible manner and deduplication can be added via Merkle trees. IPFS is also content-agnostic, one could serve Guix’s programs without even running Guix. It would also be possible to share text or video documentation using IPFS. 

This could allow the community to chip in to Guix success more easily. Publishing the content of one’s store for anyone to download will be possible. Looking at GNU/Linux distributions’ torrents, everyone seeds to help the community. It is very likely people will want to do the same for Guix. 

This adds no restriction on how a chain of trusts of key contributor to the project can be built. When a way of building consensus on a package’s hash (maybe via an Ethereum smart contract) will be build, everything will be in place for a serverless and fully auditable OS. 

What I will do before the summer

There is currently a free implementation of the IPFS protocol written in go and another one in Javascript underway. I will package them for Guix and add them to the repository. This will be a great way to familiarize myself with the codebase. 

Lisp looks interesting having done some Haskell. I will make sure I am comfortable with it.

What I will do during the first half of the summer

I will add the option to fetch content via IPFS instead of via HTTP. Guix will spawn an IPFS daemon and run it in a container. I will build it in a way that it will not be a hard dependency, as it is preferable in some cases to use a client-server architecture, and the code I will write will handle without problems the absence of an IPFS client. I will also make sure this can run with any of IPFS free implementation.

What I will do during the second half of the summer 

I will made the publish command also spawn an IPFS daemon and publish everything available in the store. I will also be there if the Hydra build farm runs into any problem with this new code. There will be a option to turn it off at any time, and statitstics will be made available, ex: how much bandwidth has been used, what are the most requested packages, etc. 

I will look into making the data storage efficient by making the IPFS daemon not copy everything in its own data store. 

Stretch goals

If everything goes better than expected and everything works beautifully before the end of the summer, I have the following stretch goals in mind: 

A GPL’ed IPFS implementation in Lisp
Packaging more free software for Guix

Who I am 

My name is Vincent Cloutier and I am a french speaking developper in Québec. I am currently in cégep (https://en.wikipedia.org/wiki/CEGEP). 

I started programming when I was very young. My father started showing me some python when I was 8, and I have not stopped programming since. I worked in python for a few years - I was fascinated by infinite loops until I was 9 - then I switched to Perl for a while (where I tried to make chatbots), and then to PHP that I continue to use to this day. I also do haskell competitions for fun here (https://www.codingame.com/profile/8ee4bb16e866d398775fa5bc545e426e2789121), though I am not good at it yet.

I already contribute to free software. I made ipfs.pics with a friend. It is a picture sharing website that uses the P2P IPFS technology. I did a junior job on Owncloud two years ago (https://github.com/owncloud/core/pull/11029) to see what a big open source project looked like. I used Sagemath cloud on my chromebook for school and made two pull requests (https://github.com/sagemathinc/smc/pull/52 and https://github.com/sagemathinc/smc/pull/53) to improve the UI on small screens.

A couple of years ago I realized that every tool I had learn and everything that I tinkered with was free and open source software. Almost everything I achieved with computers was because of people who shared their knowledge and technologies and I want to contribute back. 

[-- Attachment #3: GSoC guix.odt --]
[-- Type: application/vnd.oasis.opendocument.text, Size: 20243 bytes --]

Re: [GSoC] Draft of my proposition

[Chris Marusich]

[-- Attachment #1: Type: text/plain, Size: 2866 bytes --]

<vincent@cloutier.co> writes:

> Since Guix users know in advance the hash of the data they want,
> downloading from peers has no security implications (and privacy can
> be done trough proxies).

How will trust work in the IPFS world?  I think maybe you touch on this
when you later mention "building consensus on a package’s hash", but it
wasn't entirely clear to me.

My understanding is that because Guix uses a cryptographic hash
function, it's true that if you have some data, you know the expected
hash value of that data, and the computed hash value of the data matches
the expected hash value, then you can be confident that the data hasn't
been corrupted or tampered with.  However, how do you know the expected
hash value was correct to begin with?  How can you trust it?

Currently, I believe that Guix handles trust by refusing to use
substitutes that are not signed by a trusted key.  The substitutes built
and vended by hydra.gnu.org are signed with Hydra's key, and users of
Guix must trust Hydra's key in order to use Hydra's substitutes.

> I have a fascination for peer-to-peer tech and I am constantly looking
> for the innovative new tech in this area (Bitcoin, Ethereum,
> etc). Less than a year ago I discovered IPFS, a project that takes the
> best ideas from BitTorrent and Git to create a simple and elegant
> protocol.
>
> IPFS allows one to find who has a piece of content and is ready to
> share it, when knowing only the content’s hash. Content is added in a
> reproducible manner and deduplication can be added via Merkle
> trees. IPFS is also content-agnostic, one could serve Guix’s programs
> without even running Guix. It would also be possible to share text or
> video documentation using IPFS.

This is a very compelling idea!  Thank you for sharing it; IPFS is new
to me, and it looks intriguing.  I understand that in the past, Rémi
Birot-Delrue did some work on a similar project to enable publication of
packages over GNUnet:

https://lists.gnu.org/archive/html/guix-devel/2015-05/msg00022.html

Although progress was made, I don't think the project to publish
packages over GNUnet was fully completed.  This seems to be the last
email thread from Rémi:

https://lists.gnu.org/archive/html/guix-devel/2015-08/msg00455.html

Have you considered picking up where Rémi left off?  Even if you choose
not to use GNUnet instead of IPFS, perhaps Rémi's prior work can help
you as you work on your project.

> A couple of years ago I realized that every tool I had learn and
> everything that I tinkered with was free and open source
> software. Almost everything I achieved with computers was because of
> people who shared their knowledge and technologies and I want to
> contribute back.

That's fantastic!  Thank you for stepping up and helping.

-- 
Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]

Re: [GSoC] Draft of my proposition

[Rémi Birot-Delrue]

Hello!

> Have you considered picking up where Rémi left off?  Even if you choose
> not to use GNUnet instead of IPFS, perhaps Rémi's prior work can help
> you as you work on your project.

Our work last summer around sharing packages through a P2P network was
tied to GNUnet’s architecture, but surely adaptable to any other system:
as described last year on this mailing list, one would publish packages
under a specific identity (= gpg key couple) and anyone, given the
identity’s public key, would be able to fetch the shared packages. It
goes well with Guix because everything is gpg signed, is it similar in
IPFS?

GNUnet and IPFS seem to share some concepts (immutability, peer
identity/namespace) but IPFS is less mature: for instance, there’s a
priori no way to search a file. You might hit the same difficulties I’ve
encountred: unstable API, bugs, inaccurate or inexisting
documentation. Anyway, IPFS’s hyperlinked nature could be great to
handle multiple versions of a package.

Work on this matter can only enrich our/my comprehension of the subject,
please feel free to contact me for any question :)
-- 
Rémi

Re: [GSoC] Draft of my proposition

[Jookia]

On Wed, Mar 23, 2016 at 09:33:25PM +0100, Rémi Birot-Delrue wrote:
> GNUnet and IPFS seem to share some concepts (immutability, peer
> identity/namespace) but IPFS is less mature: for instance, there’s a
> priori no way to search a file. You might hit the same difficulties I’ve
> encountred: unstable API, bugs, inaccurate or inexisting
> documentation. Anyway, IPFS’s hyperlinked nature could be great to
> handle multiple versions of a package.

Another issue with IPFS is that it's not anonymous. I've spent a few times
trying to get it to work with Tor but it absolutely hates proxies, and putting
strain on the Tor network like that is generally a bad idea. With such a system
like this, people, organizations or companies using Guix could have their Guix
version (thus all of the packages, patches, etc) found by analyzing a few
substitutes being fetched over the network, much worse than traditional distros.

> Work on this matter can only enrich our/my comprehension of the subject,
> please feel free to contact me for any question :)
> -- 
> Rémi

Jookia.

Re: [GSoC] Draft of my proposition

[vincent]

[-- Attachment #1: Type: text/plain, Size: 4487 bytes --]




23. Mar 2016 02:17 by cmmarusich@gmail.com:


> <> vincent@cloutier.co> > writes:
>
>> Since Guix users know in advance the hash of the data they want,
>> downloading from peers has no security implications (and privacy can
>> be done trough proxies).
>
> How will trust work in the IPFS world?  I think maybe you touch on this
> when you later mention "building consensus on a package’s hash", but it
> wasn't entirely clear to me.




IPFS checks the integrity of the content you download and that's it. You have 
to validate in other ways that the content you want has effectively a given 
hash.





I think it's a very big advantage, because it totally uncouples the p2p 
sharing from the trust system. Users do not have to agree to be on the same 
security scheme. One could configure his system in a way that it will only 
download content signed by, let's say the FSF and the EFF, and another user 
downloads only when signed by his employers. Both those users will be able to 
share the same data with one another, because the data is separated from the 
signature.





That's what I meant by "building consensus on a package’s hash", because the 
community will be able to innovate to build a trust system. :)


 


> My understanding is that because Guix uses a cryptographic hash
> function, it's true that if you have some data, you know the expected
> hash value of that data, and the computed hash value of the data matches
> the expected hash value, then you can be confident that the data hasn't
> been corrupted or tampered with.  However, how do you know the expected
> hash value was correct to begin with?  How can you trust it?

 

IPFS does not aim to solve the last part. 





> Currently, I believe that Guix handles trust by refusing to use
> substitutes that are not signed by a trusted key.  The substitutes built
> and vended by hydra.gnu.org are signed with Hydra's key, and users of
> Guix must trust Hydra's key in order to use Hydra's substitutes.




Users who trust Hydra will be able do download substitutes from one another, 
if the package is signed by Hydra. This should come as a relief to hydra's 
servers! :)




I could also add that a package must be signed by the all the user's trusted 
substitutes before downloading.





>> I have a fascination for peer-to-peer tech and I am constantly looking
>> for the innovative new tech in this area (Bitcoin, Ethereum,
>> etc). Less than a year ago I discovered IPFS, a project that takes the
>> best ideas from BitTorrent and Git to create a simple and elegant
>> protocol.
>>
>> IPFS allows one to find who has a piece of content and is ready to
>> share it, when knowing only the content’s hash. Content is added in a
>> reproducible manner and deduplication can be added via Merkle
>> trees. IPFS is also content-agnostic, one could serve Guix’s programs
>> without even running Guix. It would also be possible to share text or
>> video documentation using IPFS.
>
> This is a very compelling idea!  Thank you for sharing it; IPFS is new
> to me, and it looks intriguing.  I understand that in the past, Rémi
> Birot-Delrue did some work on a similar project to enable publication of
> packages over GNUnet:
>
> https://lists.gnu.org/archive/html/guix-devel/2015-05/msg00022.html
>
> Although progress was made, I don't think the project to publish
> packages over GNUnet was fully completed.  This seems to be the last
> email thread from Rémi:
>
> https://lists.gnu.org/archive/html/guix-devel/2015-08/msg00455.html
>
> Have you considered picking up where Rémi left off?  Even if you choose
> not to use GNUnet instead of IPFS, perhaps Rémi's prior work can help
> you as you work on your project.




I think IPFS simpler and more stable API is a must. But I will definitely be 
looking into reusing parts of his code, either for IPFS or making it usable 
for both. I could be an interesting stretch goal.





>> A couple of years ago I realized that every tool I had learn and
>> everything that I tinkered with was free and open source
>> software. Almost everything I achieved with computers was because of
>> people who shared their knowledge and technologies and I want to
>> contribute back.
>
> That's fantastic!  Thank you for stepping up and helping.




Thanks to you for making it possible :)








- Vincent


[-- Attachment #2: Type: text/html, Size: 5880 bytes --]

Re: [GSoC] Draft of my proposition

[vincent]

[-- Attachment #1: Type: text/plain, Size: 1352 bytes --]




23. Mar 2016 17:29 by 166291@gmail.com:


> On Wed, Mar 23, 2016 at 09:33:25PM +0100, Rémi Birot-Delrue wrote:
>> GNUnet and IPFS seem to share some concepts (immutability, peer
>> identity/namespace) but IPFS is less mature: for instance, there’s a
>> priori no way to search a file. You might hit the same difficulties I’ve
>> encountred: unstable API, bugs, inaccurate or inexisting
>> documentation. Anyway, IPFS’s hyperlinked nature could be great to
>> handle multiple versions of a package.
>
> Another issue with IPFS is that it's not anonymous. I've spent a few times
> trying to get it to work with Tor but it absolutely hates proxies, and 
> putting
> strain on the Tor network like that is generally a bad idea.




I want to add IPFS as an option, not as the only way to download stuff. It 
might be better for privacy-conscious people to use http over Tor. 





> With such a system
> like this, people, organizations or companies using Guix could have their 
> Guix
> version (thus all of the packages, patches, etc) found by analyzing a few
> substitutes being fetched over the network, much worse than traditional 
> distros.
>




A company or any large install could run IPFS/Guix in a LAN, with a build 
server, which would bring *huge* performance gains while being private.


[-- Attachment #2: Type: text/html, Size: 1869 bytes --]

Re: [GSoC] Draft of my proposition

[Jookia]

On Wed, Mar 23, 2016 at 10:45:05PM +0000, vincent@cloutier.co wrote:
> I want to add IPFS as an option, not as the only way to download stuff. It
> might be better for privacy-conscious people to use http over Tor. 

Perhaps, though I think it's worth thinking about if this issue is solvable for
now. If an IPFS implementation is widely used and solves a lot of load issues, I
can see it becoming the most commonly used option with tools relying on this
functionality, especially for people running their own build servers and only
using IPFS. Once in to the situation that a tool works for most people and it
becomes a defacto standard it's very hard to get people to use other tools,
especially if it would mean we'd end up running three networks: HTTP, IPFS and
whatever p2p option there is that both Tor users and non-Tor users can use.

> A company or any large install could run IPFS/Guix in a LAN, with a build
> server, which would bring *huge* performance gains while being private.

Why would they need IPFS in LAN if they have a build server?

Jookia.

Re: [GSoC] Draft of my proposition

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 1492 bytes --]

On Thu, Mar 24, 2016 at 10:11:53AM +1100, Jookia wrote:
> On Wed, Mar 23, 2016 at 10:45:05PM +0000, vincent@cloutier.co wrote:
> > I want to add IPFS as an option, not as the only way to download stuff. It
> > might be better for privacy-conscious people to use http over Tor. 
> 
> Perhaps, though I think it's worth thinking about if this issue is solvable for
> now. If an IPFS implementation is widely used and solves a lot of load issues, I
> can see it becoming the most commonly used option with tools relying on this
> functionality, especially for people running their own build servers and only
> using IPFS. Once in to the situation that a tool works for most people and it
> becomes a defacto standard it's very hard to get people to use other tools,
> especially if it would mean we'd end up running three networks: HTTP, IPFS and
> whatever p2p option there is that both Tor users and non-Tor users can use.
> 
> > A company or any large install could run IPFS/Guix in a LAN, with a build
> > server, which would bring *huge* performance gains while being private.
> 
> Why would they need IPFS in LAN if they have a build server?
> 
> Jookia.

They could have one machine in front as a build/cache server that the
other machines connect to

-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [GSoC] Draft of my proposition

[Ludovic Courtès]

Salut Vincent,

<vincent@cloutier.co> skribis:

> 23. Mar 2016 02:17 by cmmarusich@gmail.com:

[...]

>> This is a very compelling idea!  Thank you for sharing it; IPFS is new
>> to me, and it looks intriguing.  I understand that in the past, Rémi
>> Birot-Delrue did some work on a similar project to enable publication of
>> packages over GNUnet:
>>
>> https://lists.gnu.org/archive/html/guix-devel/2015-05/msg00022.html
>>
>> Although progress was made, I don't think the project to publish
>> packages over GNUnet was fully completed.  This seems to be the last
>> email thread from Rémi:
>>
>> https://lists.gnu.org/archive/html/guix-devel/2015-08/msg00455.html
>>
>> Have you considered picking up where Rémi left off?  Even if you choose
>> not to use GNUnet instead of IPFS, perhaps Rémi's prior work can help
>> you as you work on your project.
>
>
>
>
> I think IPFS simpler and more stable API is a must.

I’m not completely sure and, as Jookia pointed out, the two projects
share different goals, with GNUnet allowing for anonymous
communications.

> But I will definitely be looking into reusing parts of his code,
> either for IPFS or making it usable for both. I could be an
> interesting stretch goal.

I don’t think there’d be much code to reuse.

However, past design discussions should probably be taken into account,
because some of the questions we had about how to store things, or how
to reuse the code and protocols of the existing ‘guix publish’ and ‘guix
substitute’ tools is definitely needed.  WDYT?

Thanks,
Ludo’.

Re: [GSoC] Draft of my proposition

[Ludovic Courtès]

Hi Rémi,

Long time no see!  :-)

How do you view the project you worked on, in hindsight?  Would you pick
it up and complete it?

Cheers,
Ludo’.

Re: [GSoC] Draft of my proposition

[Frederick Muriithi]

I am currently looking into IPFS for Guix, and found this thread. I
would appreciate anyone pointing me to the work that was done, so that
I might figure out what is needed to continue ir.

For now, my main focus is to get an IPFS package defined for Guix,
that users could use to install IPFS on their machines.

-- 
Frederick M. Muriithi

Re: [GSoC] Draft of my proposition

[Ricardo Wurmus]

Hi Fred,

> I am currently looking into IPFS for Guix, and found this thread. I
> would appreciate anyone pointing me to the work that was done, so that
> I might figure out what is needed to continue ir.

previous work focused on Gnunet.  The project ended with initial Guile
bindings for Gnunet, IIRC.  There has been no work on IPFS.

> For now, my main focus is to get an IPFS package defined for Guix,
> that users could use to install IPFS on their machines.

This seems like a good first step.  A close second step could be to
write a service definition for GuixSD.

Thanks for working on this!

-- 
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net