gnunet-guile reboot & guix

gnunet-guile reboot & guix

[Amirouche Boubekki]

Héllo all,

I restarted from scratch the gnunet-guile bindings. It was made
much easier thanks to the work of ng0 on gnunet documentation and
guile-bytestructures to handle C structs and unions.

You need guix from today with latest guile-bytestructures 1.0.1.

You can get the code using the following command:

    git clone git://gnunet.org/gnunet-guile2.git

You can install a recent-ish gnunet using the following command:

   guix package -f guix.scm

Then, you can do usual cli dance:

   ./bootstrap && ./configure && make

Ahem, now it's time to run gnunet services. I put together
gnunet configuration that might not be very good even if it
works. For instance, it seems like gnunet manages to reach
the outside world. It's based on configuration files found
in gnunet distribution:

   gnunet-arm -c etc/p2.conf -s

At this point you will be able to test the bindings.

To publish a FILE, use the following command:

   ./pre-inst-env ./gnunet-guile publish etc/p2.conf FILE

To download the above file into OUT, you need to copy paste
the gnunet:// URI from the previous command output and execute
the following command:

   ./pre-inst-env ./gnunet-guile download etc/p2.conf URI OUT

That is all!

There is no support for identity and various stuff are missing.
There might be memory leaks and other issues (like proper disjoint
types for pointers). I just finished the code.

I think I need to know what's the plan/design for gnunet/guix
integration to continue.


TIA!

_______________________________________________
GNUnet-developers mailing list
GNUnet-developers@gnu.org
https://lists.gnu.org/mailman/listinfo/gnunet-developers

Re: gnunet-guile reboot & guix

[ng0]

[-- Attachment #1.1: Type: text/plain, Size: 3367 bytes --]

Amirouche Boubekki transcribed 1.5K bytes:
> Héllo all,
> 
> I restarted from scratch the gnunet-guile bindings. It was made
> much easier thanks to the work of ng0 on gnunet documentation and
> guile-bytestructures to handle C structs and unions.
> 
> You need guix from today with latest guile-bytestructures 1.0.1.
> 
> You can get the code using the following command:
> 
>    git clone git://gnunet.org/gnunet-guile2.git
> 
> You can install a recent-ish gnunet using the following command:
> 
>   guix package -f guix.scm
> 
> Then, you can do usual cli dance:
> 
>   ./bootstrap && ./configure && make
> 
> Ahem, now it's time to run gnunet services. I put together

Would gnunet-service (in GuixSD, and shepherd as user-process
on other systems) help here? I wanted to start debugging my
service again in february, maybe earlier.
I could delegate it to whoever wants to work on it though.

> gnunet configuration that might not be very good even if it
> works. For instance, it seems like gnunet manages to reach
> the outside world. It's based on configuration files found
> in gnunet distribution:
> 
>   gnunet-arm -c etc/p2.conf -s
> 
> At this point you will be able to test the bindings.
> 
> To publish a FILE, use the following command:
> 
>   ./pre-inst-env ./gnunet-guile publish etc/p2.conf FILE
> 
> To download the above file into OUT, you need to copy paste
> the gnunet:// URI from the previous command output and execute
> the following command:
> 
>   ./pre-inst-env ./gnunet-guile download etc/p2.conf URI OUT
> 
> That is all!

Hey! Pretty good progress for just a couple of days, very nice!

> There is no support for identity and various stuff are missing.
> There might be memory leaks and other issues (like proper disjoint
> types for pointers). I just finished the code.
> 
> I think I need to know what's the plan/design for gnunet/guix
> integration to continue.

If you want a (relative) unprocessed summary of its history, it's collected in here
(org-mode recommended):
https://gnunet.org/git/infotropique-artwork.git/tree/doc/guix-past-notes.txt

Now for the current goals I should manage to reply within a week (searching
my notes and past offlist emails). That is of course from my perspective
as a GNUnet developer.

On various occasions it was made clear to me that FS isn't ready for such
usage and we would need to extend it if we start working on it. I could be
wrong, but to some degree our implementation of our own designs has some
mistakes, if I remember Grothoff right.

Bits and pieces without what I came up with in winter 2016:

* anonymous levels aren't necessary for sharing code
** update on this: one (or more?) other OS' is looking into Guix+GNUnet
   as well and I'm not sure if they would rely on anonymity. Imo it makes
   no sense for the start. anonymity setting 0 would work for a start.

Before I share my part of the ideas, maybe someone who wasn't involved
in the on-and-off discussions could add their ideas? Fresh ideas could
bring new perspectives we haven't seen.


For the Guix part I just know that it should be an option, not a default.

> 
> TIA!
> 
> 

-- 
GnuPG: A88C8ADD129828D7EAC02E52E22F9BBFEE348588
GnuPG: https://c.n0.is/ng0_pubkeys/tree/keys
  WWW: https://n0.is/a/  ::  https://ea.n0.is

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[-- Attachment #2: Type: text/plain, Size: 162 bytes --]

_______________________________________________
GNUnet-developers mailing list
GNUnet-developers@gnu.org
https://lists.gnu.org/mailman/listinfo/gnunet-developers

Re: gnunet-guile reboot & guix

[Amirouche Boubekki]

On 2018-01-13 01:49, ng0 wrote:
> Amirouche Boubekki transcribed 1.5K bytes:
>> Héllo all,
>> 
>> I restarted from scratch the gnunet-guile bindings. It was made
>> much easier thanks to the work of ng0 on gnunet documentation and
>> guile-bytestructures to handle C structs and unions.

...

>> I think I need to know what's the plan/design for gnunet/guix
>> integration to continue.
> 
> If you want a (relative) unprocessed summary of its history, it's
> collected in here
> (org-mode recommended):
> https://gnunet.org/git/infotropique-artwork.git/tree/doc/guix-past-notes.txt

Here is an extract relevant to the current discussion:

* Design of guix/gnunet integration

>> So I can see two milestones now, as we discussed before:

>> 1. Create a variant of ‘guix publish’ that publishes over GNUnet’s
>> file sharing (FS) service, using the neat bindings that you wrote.

>> For that you can literally copy guix/scripts/publish.scm as a
>> starting point, and simply remove the HTTP-related code (which is a
>> small fraction.)  The rest will be mostly identical: narinfo
>> meta-data generation, signing.  I guess it will be easy to test it
>> using gnunet-search and similar tools.

>> As a 2nd step, we’ll see how we can refactor things to allow code
>> sharing between the existing ‘guix publish’ and its GNUnet variant.

>> 2. Create a variant of ‘guix substitute’ for searches through GNUnet.
>> Again, a large part of the code is about narinfo and signature
>> checking, and it can be reused.

>> I’m not completely clear on how search for substitutes will work,
>> though.  Currently, when the user wants to build /gnu/store/xyz, ‘guix
>> substitute’ simply fetches http://hydra.gnu.org/xyz.narinfo.  How will
>> that work with GNUnet?  Are we going to look up their /gnu/store file
>> name?


>> I’m not completely clear on how search for substitutes will work,
>> though.  Currently, when the user wants to build /gnu/store/xyz, ‘guix
>> substitute’ simply fetches http://hydra.gnu.org/xyz.narinfo.  How will
>> that work with GNUnet?  Are we going to look up their /gnu/store file
>> name?

> I’ve considered a solution for that: GNUnet allows one to create
> specific namespace and publish files under this namespace. Unlike
> publishing under the “global namespace” where keywords are used to
> identify a file, when publishing under a specific namespace files are
> identified with a choosen identifier. Moreover, as a namespace is
> basically a cryptographic key pair, and publishing a file under your
> namespace means signing, one’s assured nobody else will publish under
> her or his namespace. By the way, the private key associated with a
> namespace is named “ego” or “pseudonym”.

> It’s easy to test this feature:

> # create a `test` ego/namespace
> $ gnunet-identity -C test

> # list the known egos in the form: `name - public key`
> $ gnunet-identity -d
> test - M2OC987U9LFJHQ8LC9SLCV4Q0ONHJV7FMTFQ2VRPE0M9R9MK5860
> …

> # index the file `foo.txt` under the `test` namespace
> $ gnunet-publish -P test -t foobarbaz foo.txt

> # find the file `foo.txt`
> $ gnunet-search gnunet://fs/sks/M2OC987U9LFJHQ8LC9SLCV4…/foobarbaz
> #0:
> gnunet-download -o "foo.txt" gnunet://fs/chk/PL217ODD8EDSMOIQ3UT0…

> Now if Alice wants to publish her binaries, she creates an
> ego/namespace and publishes everything under it; Bob adds her
> namespace’s public key to his authorized substitutes list, and when
> installing `/gnu/store/xyz` the substitute will search for
> `gnunet://fs/sks/<Alice’s key>/xyz`.

> Instead of publishing an archive we might also directly publish/index
> the build, but I don’t know if it’s viable.

> Does it seem right to you?

* Another discusion about guix integration

>> Instead of using ‘file-system-tree’, this variant should probably use
>> ‘live-paths’ from (guix store), which returns the list of live store
>> items.

> Well, `file-system-tree` is only used to recursively index a random
> directory’s content (in our case, a single store item). It looked 
> viable
> for publishing a single store item, but won’t be good for indexing at
> once the entire set of live paths; I should ask the GNUnet team how to
> properly index such a huge amount of directories.

> On my machine, running `live-paths` takes ~2 seconds, but the
> publication of the entire store will probably take much longer anyway.

>> BTW, I noticed there’s quite a bunch of global variables that are
>> ‘set!’.  It would be better to avoid that, but I suppose the
>> continuation-passing style that the GNUnet libraries impose makes it
>> difficult.

> Hopefully, using the “closure” parameters of the GNUnet API in the
> bindings should reduce the need for global variables, and improve
> elegance of end-user programs.


>>> Nitpick: it’s a bit annoying that we have to specify a GNUnet
>>> configuration file.
>> 
>> Yes, GNUnet programs usually look for `~/.config/gnunet.conf`, and
>> `publish-gnunet` does the same. Now, maybe `publish-gnunet` could
>> somehow obtain the config file used by `gnunet-arm`?

> No, it would need the config file to figure out how to talk to
> gnunet-service-arm (or any other service). And we do support running
> many instances of peers on the same host, which really means the config
> is the only way to find out.

The rest of ng0 reply:

> 
> On various occasions it was made clear to me that FS isn't ready for 
> such
> usage and we would need to extend it if we start working on it. I could 
> be
> wrong, but to some degree our implementation of our own designs has 
> some
> mistakes, if I remember Grothoff right.
> 
> Bits and pieces without what I came up with in winter 2016:
> 
> * anonymous levels aren't necessary for sharing code
> ** update on this: one (or more?) other OS' is looking into Guix+GNUnet
>    as well and I'm not sure if they would rely on anonymity. Imo it 
> makes
>    no sense for the start. anonymity setting 0 would work for a start.

Anonymity is easily configurable.

> Before I share my part of the ideas, maybe someone who wasn't involved
> in the on-and-off discussions could add their ideas? Fresh ideas could
> bring new perspectives we haven't seen.
> 
> 
> For the Guix part I just know that it should be an option, not a 
> default.
> 

Oh really? I don't see that appear in the discussion with Remi.

_______________________________________________
GNUnet-developers mailing list
GNUnet-developers@gnu.org
https://lists.gnu.org/mailman/listinfo/gnunet-developers

Re: gnunet-guile reboot & guix

[ng0]

[-- Attachment #1.1: Type: text/plain, Size: 7778 bytes --]

Amirouche Boubekki transcribed 6.3K bytes:
> On 2018-01-13 01:49, ng0 wrote:
> > Amirouche Boubekki transcribed 1.5K bytes:
> > > Héllo all,
> > > 
> > > I restarted from scratch the gnunet-guile bindings. It was made
> > > much easier thanks to the work of ng0 on gnunet documentation and
> > > guile-bytestructures to handle C structs and unions.
> 
> ...
> 
> > > I think I need to know what's the plan/design for gnunet/guix
> > > integration to continue.
> > 
> > If you want a (relative) unprocessed summary of its history, it's
> > collected in here
> > (org-mode recommended):
> > https://gnunet.org/git/infotropique-artwork.git/tree/doc/guix-past-notes.txt
> 
> Here is an extract relevant to the current discussion:
> 
> * Design of guix/gnunet integration
> 
> > > So I can see two milestones now, as we discussed before:
> 
> > > 1. Create a variant of ‘guix publish’ that publishes over GNUnet’s
> > > file sharing (FS) service, using the neat bindings that you wrote.
> 
> > > For that you can literally copy guix/scripts/publish.scm as a
> > > starting point, and simply remove the HTTP-related code (which is a
> > > small fraction.)  The rest will be mostly identical: narinfo
> > > meta-data generation, signing.  I guess it will be easy to test it
> > > using gnunet-search and similar tools.
> 
> > > As a 2nd step, we’ll see how we can refactor things to allow code
> > > sharing between the existing ‘guix publish’ and its GNUnet variant.
> 
> > > 2. Create a variant of ‘guix substitute’ for searches through GNUnet.
> > > Again, a large part of the code is about narinfo and signature
> > > checking, and it can be reused.
> 
> > > I’m not completely clear on how search for substitutes will work,
> > > though.  Currently, when the user wants to build /gnu/store/xyz, ‘guix
> > > substitute’ simply fetches http://hydra.gnu.org/xyz.narinfo.  How will
> > > that work with GNUnet?  Are we going to look up their /gnu/store file
> > > name?
> 
> 
> > > I’m not completely clear on how search for substitutes will work,
> > > though.  Currently, when the user wants to build /gnu/store/xyz, ‘guix
> > > substitute’ simply fetches http://hydra.gnu.org/xyz.narinfo.  How will
> > > that work with GNUnet?  Are we going to look up their /gnu/store file
> > > name?
> 
> > I’ve considered a solution for that: GNUnet allows one to create
> > specific namespace and publish files under this namespace. Unlike
> > publishing under the “global namespace” where keywords are used to
> > identify a file, when publishing under a specific namespace files are
> > identified with a choosen identifier. Moreover, as a namespace is
> > basically a cryptographic key pair, and publishing a file under your
> > namespace means signing, one’s assured nobody else will publish under
> > her or his namespace. By the way, the private key associated with a
> > namespace is named “ego” or “pseudonym”.
> 
> > It’s easy to test this feature:
> 
> > # create a `test` ego/namespace
> > $ gnunet-identity -C test
> 
> > # list the known egos in the form: `name - public key`
> > $ gnunet-identity -d
> > test - M2OC987U9LFJHQ8LC9SLCV4Q0ONHJV7FMTFQ2VRPE0M9R9MK5860
> > …
> 
> > # index the file `foo.txt` under the `test` namespace
> > $ gnunet-publish -P test -t foobarbaz foo.txt
> 
> > # find the file `foo.txt`
> > $ gnunet-search gnunet://fs/sks/M2OC987U9LFJHQ8LC9SLCV4…/foobarbaz
> > #0:
> > gnunet-download -o "foo.txt" gnunet://fs/chk/PL217ODD8EDSMOIQ3UT0…
> 
> > Now if Alice wants to publish her binaries, she creates an
> > ego/namespace and publishes everything under it; Bob adds her
> > namespace’s public key to his authorized substitutes list, and when
> > installing `/gnu/store/xyz` the substitute will search for
> > `gnunet://fs/sks/<Alice’s key>/xyz`.
> 
> > Instead of publishing an archive we might also directly publish/index
> > the build, but I don’t know if it’s viable.
> 
> > Does it seem right to you?
> 
> * Another discusion about guix integration
> 
> > > Instead of using ‘file-system-tree’, this variant should probably use
> > > ‘live-paths’ from (guix store), which returns the list of live store
> > > items.
> 
> > Well, `file-system-tree` is only used to recursively index a random
> > directory’s content (in our case, a single store item). It looked viable
> > for publishing a single store item, but won’t be good for indexing at
> > once the entire set of live paths; I should ask the GNUnet team how to
> > properly index such a huge amount of directories.
> 
> > On my machine, running `live-paths` takes ~2 seconds, but the
> > publication of the entire store will probably take much longer anyway.
> 
> > > BTW, I noticed there’s quite a bunch of global variables that are
> > > ‘set!’.  It would be better to avoid that, but I suppose the
> > > continuation-passing style that the GNUnet libraries impose makes it
> > > difficult.
> 
> > Hopefully, using the “closure” parameters of the GNUnet API in the
> > bindings should reduce the need for global variables, and improve
> > elegance of end-user programs.
> 
> 
> > > > Nitpick: it’s a bit annoying that we have to specify a GNUnet
> > > > configuration file.
> > > 
> > > Yes, GNUnet programs usually look for `~/.config/gnunet.conf`, and
> > > `publish-gnunet` does the same. Now, maybe `publish-gnunet` could
> > > somehow obtain the config file used by `gnunet-arm`?
> 
> > No, it would need the config file to figure out how to talk to
> > gnunet-service-arm (or any other service). And we do support running
> > many instances of peers on the same host, which really means the config
> > is the only way to find out.
> 
> The rest of ng0 reply:
> 
> > 
> > On various occasions it was made clear to me that FS isn't ready for
> > such
> > usage and we would need to extend it if we start working on it. I could
> > be
> > wrong, but to some degree our implementation of our own designs has some
> > mistakes, if I remember Grothoff right.
> > 
> > Bits and pieces without what I came up with in winter 2016:
> > 
> > * anonymous levels aren't necessary for sharing code
> > ** update on this: one (or more?) other OS' is looking into Guix+GNUnet
> >    as well and I'm not sure if they would rely on anonymity. Imo it
> > makes
> >    no sense for the start. anonymity setting 0 would work for a start.
> 
> Anonymity is easily configurable.

Sure.

> > Before I share my part of the ideas, maybe someone who wasn't involved
> > in the on-and-off discussions could add their ideas? Fresh ideas could
> > bring new perspectives we haven't seen.
> > 
> > 
> > For the Guix part I just know that it should be an option, not a
> > default.
> > 
> 
> Oh really? I don't see that appear in the discussion with Remi.

The copied file was a collection of work not only of Remi. There's
2 or 3 threads at least.
The idea that GNUnet FS distribution should be an option, not a
default for Guix is my understanding of how more recent discussions
(on IRC and the mailinglist, possibly even AFK chats I had with some
of the other developers). My personal idea is to make it a standard
in my redistribution of GuixSD, but primarily we want to work on
what Guix wants, not what a distro framework in development close
to GNUnet wants, for now. Of course I wouldn't complain if Guix would
pick GNUnet as default, but we'd need a good amount of testing
before we'd make it so.


-- 
GnuPG: A88C8ADD129828D7EAC02E52E22F9BBFEE348588
GnuPG: https://c.n0.is/ng0_pubkeys/tree/keys
  WWW: https://n0.is/a/  ::  https://ea.n0.is

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[-- Attachment #2: Type: text/plain, Size: 162 bytes --]

_______________________________________________
GNUnet-developers mailing list
GNUnet-developers@gnu.org
https://lists.gnu.org/mailman/listinfo/gnunet-developers

Re: gnunet-guile reboot & guix

[Ludovic Courtès]

Hello,

Amirouche Boubekki <amirouche@hypermove.net> skribis:

> I restarted from scratch the gnunet-guile bindings. It was made
> much easier thanks to the work of ng0 on gnunet documentation and
> guile-bytestructures to handle C structs and unions.
>
> You need guix from today with latest guile-bytestructures 1.0.1.
>
> You can get the code using the following command:
>
>    git clone git://gnunet.org/gnunet-guile2.git

Thanks a lot for giving it some love!

Perhaps we should retire the repo at
<https://git.savannah.gnu.org/cgit/guix/gnunet.git/>?
(The ‘wip-monad’ branch might be of interest though.)

Ludo’.

_______________________________________________
GNUnet-developers mailing list
GNUnet-developers@gnu.org
https://lists.gnu.org/mailman/listinfo/gnunet-developers

Re: gnunet-guile reboot & guix

[ng0]

[-- Attachment #1.1: Type: text/plain, Size: 1731 bytes --]

Ludovic Courtès transcribed 1.0K bytes:
> Hello,
> 
> Amirouche Boubekki <amirouche@hypermove.net> skribis:
> 
> > I restarted from scratch the gnunet-guile bindings. It was made
> > much easier thanks to the work of ng0 on gnunet documentation and
> > guile-bytestructures to handle C structs and unions.
> >
> > You need guix from today with latest guile-bytestructures 1.0.1.
> >
> > You can get the code using the following command:
> >
> >    git clone git://gnunet.org/gnunet-guile2.git
> 
> Thanks a lot for giving it some love!
> 
> Perhaps we should retire the repo at
> <https://git.savannah.gnu.org/cgit/guix/gnunet.git/>?
> (The ‘wip-monad’ branch might be of interest though.)
> 
> Ludo’.
> 
> _______________________________________________
> GNUnet-developers mailing list
> GNUnet-developers@gnu.org
> https://lists.gnu.org/mailman/listinfo/gnunet-developers

I think both Amirouche as well as myself (and gnunet.org) have a
copy of the old code base. If I understand savannah hosting
correctly 'retiring' just means making it read-only, so that
would be alright.
Development happens on gnunet.org and future releases will be
placed on the GNU Ftp folder of gnunet.

Our mailinglist at gnunet-developers is open for patches
(no sign-up required) and whoever wants to help developing
it can get push access (of course Amirouche has to decide on
this project, etc).

Eventually we should remove the old repository https://gnunet.org/git/gnunet-guile,
or rename the current one (gnunet-guile2).

That is my position. I'm not working on this right now,
Amirouche does.
-- 
ng0 :: https://ea.n0.is
A88C8ADD129828D7EAC02E52E22F9BBFEE348588 :: https://ea.n0.is/keys/

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[-- Attachment #2: Type: text/plain, Size: 162 bytes --]

_______________________________________________
GNUnet-developers mailing list
GNUnet-developers@gnu.org
https://lists.gnu.org/mailman/listinfo/gnunet-developers