all messages for Guix-related lists mirrored at yhetil.org
 help / color / mirror / code / Atom feed
* [bug#52555] [RFC PATCH 0/3] Decentralized substitute distribution with ERIS
@ 2021-12-16 16:17 pukkamustard
  2021-12-16 16:20 ` [bug#52555] [RFC PATCH 1/3] publish: Add ERIS URN to narinfo pukkamustard
                   ` (4 more replies)
  0 siblings, 5 replies; 65+ messages in thread
From: pukkamustard @ 2021-12-16 16:17 UTC (permalink / raw)
  To: 52555; +Cc: pukkamustard, ~pukkamustard/eris

Hello Guix,

This is an initial patch and proposal towards decentralizing substitute
distribution with ERIS.

ERIS (Encoding for Robust Immutable Storage) [1] is an encoding of content into
uniformly sized, encryped and content-addressed blocks. The original content
can be reconstructed only with access to a read capability, which can be
encoded as an URN.

One key advantage of ERIS is that the encoding is protocol agnostic. Any
protocol that can transfer small (32KiB) sized blocks referenced by the hash of
their content will do. This can be done with things such as GNUNet, IPFS,
OpenDHT, HTTP or a USB stick on a bicycle.

The following patch allows substitutes to be published over IPFS using ERIS.
This is inspired and very similar to previous work on distributing substitutes
over IPFS [2].

The narinfos served by `guix publish` look like this:

--8<---------------cut here---------------start------------->8---
StorePath: /gnu/store/81bdcd5x4v50i28h98bfkvvkx9cky63w-hello-2.10
URL: nar/gzip/81bdcd5x4v50i28h98bfkvvkx9cky63w-hello-2.10
Compression: gzip
FileSize: 67363
ERIS: urn:erisx2:BIBC2LUTIQH43S2KRIAV7TBXNUUVPZTMV6KFA2M7AL5V6FNE77VNUDDVDAGJUEEAFATVO2QQT67SMOPTO3LGWCJFU7BZVCF5VXEQQW25BE
URL: nar/zstd/81bdcd5x4v50i28h98bfkvvkx9cky63w-hello-2.10
Compression: zstd
FileSize: 64917
ERIS: urn:erisx2:BIBO7KS7SAWHDNC43DVILOSQ3F3SRRHEV6YPLDCSZ7MMD6LZVCHQMEQ6FUBTJAPSNFF7XR5XPTP4OQ72OPABNEO7UYBUN42O46ARKHBTGM
NarHash: sha256:1sagsz1mnlqkr8r8s6gwkzvvhq619rlzhpbxl3h0b111n5hn2w9w
NarSize: 220704
References: 2fk1gz2s7ppdicynscra9b19byrrr866-glibc-2.33 81bdcd5x4v50i28h98bfkvvkx9cky63w-hello-2.10 90lbavffg0csrf208nw0ayj1bz5knl47-gcc-10.3.0-lib
Deriver: 260bk0ch4np4h2yz5yqhf8hjbsyhwpmr-hello-2.10.drv
Signature: 1;strawberry;KHNpZ25hdHVyZSAKIChkYXRhIAogIChmbGFncyByZmM2OTc5KQogIChoYXNoIHNoYTI1NiAjNDk4ODkwODZDNTY4MzQyRENFQzk3QzA3NDE4NEQ1RkRCOTNCNDA2MUNCRDM4MUExRjVBQzVDODI0MTgwMTU3OSMpCiAgKQogKHNpZy12YWwgCiAgKGVjZHNhIAogICAociAjMEU2NDlFODE4QzRFNjNGNEY2OUQ5QTAwRjUwNjRDMzQ3QjY3RDM0RTM0NTg2MkI4NTc3RTg5MUY5Q0Q3NDhBQiMpCiAgIChzICMwMTZGRjA1MDdCQjZGMzA2NUEzMjYzRDA2MTAyRDc5MTBEOEZGODc5RTdENjREODRFODBENDBGMTJFMTBBOTQ1IykKICAgKQogICkKIChwdWJsaWMta2V5IAogIChlY2MgCiAgIChjdXJ2ZSBFZDI1NTE5KQogICAocSAjMDRDMkY4ODk1QTU0NDNGNTlCODk2NDEwMEI1MDY0NzU4RjQ1N0YzMENEREE1MTQyQzE0MDc0NjExNTA1NTc5MCMpCiAgICkKICApCiApCg==
--8<---------------cut here---------------end--------------->8---

For every compressed nar the ERIS URN is computed and added.

If the `--ipfs` is used for `guix publish` then the encoded blocks are also
uploaded to the IPFS daemon. The nar could then be retrieved from anywhere like
this:

--8<---------------cut here---------------start------------->8---
(use-modules (eris)
	     (eris blocks ipfs))

(eris-decode->bytevector
 "urn:erisx2:BIBC2LUTIQH43S2KRIAV7TBXNUUVPZTMV6KFA2M7AL5V6FNE77VNUDDVDAGJUEEAFATVO2QQT67SMOPTO3LGWCJFU7BZVCF5VXEQQW25BE"
 eris-blocks-ipfs-ref)
--8<---------------cut here---------------end--------------->8---

These patches do not yet retrieve content from IPFS (TODO). But in principle,
anybody connected to IPFS can get the nar with the ERIS URN. This could be used
to reduce load on substitute server as they would only need to publish the ERIS
URN directly - substitutes could be delivered much more peer-to-peer.

Other transports that I have been looking in to and am pretty sure will work
include: HTTP (with RFC 2169 [3]), GNUNet, OpenDHT. This is, imho, the
advantage of ERIS over IPFS directly or GNUNet directly. The encoding and
identifiers (URN) are abstracted away from specific transports (and also
applications). ERIS is almost exactly the same encoding as used in GNUNet
(ECRS).

Blocks can be stored in any kind of databases (see for example the GDBM
bindings [4]).

A tricky things is figuring out how to multiplex all these different
transports and storages...

The ERIS specification is still considered "experimental". However we feel
confident to stabilize it and intend to do so around February/March 2022 with a
release 1.0.0 of the specification. This will ensure that the identifiers
remain stable for the forseeable future (until the crypto breaks). Before that
there is also a small external security audit of the specification planned
(thanks to NGI0/NLnet!).

This is just a little demo of the idea and some food for thought and
discussion. Give it a try and let me know what you think!

I've also pushed the patches to my personal Guix mirror if you want to check it
out from there:

https://inqlab.net/git/guix.git/log/?h=wip-eris

Also CCing ~pukkamustard/eris@lists.sr.ht where there is some general ERIS
related discussion.

Thanks,
-pukkamustard

[1] http://purl.org/eris
[2] https://issues.guix.gnu.org/33899
[3] https://www.ietf.org/rfc/rfc2169.txt
[4] https://inqlab.net/git/guile-eris.git/tree/eris/blocks/gdbm.scm


pukkamustard (3):
  publish: Add ERIS URN to narinfo
  WIP: gnu: guile-eris: Update to unreleased git version.
  publish: Add IPFS support.

 configure.ac                        |  5 ++
 gnu/packages/guile-xyz.scm          | 10 ++--
 gnu/packages/package-management.scm |  1 +
 guix/narinfo.scm                    | 10 ++--
 guix/scripts/publish.scm            | 79 ++++++++++++++++++++++-------
 5 files changed, 79 insertions(+), 26 deletions(-)

-- 
2.34.0





^ permalink raw reply	[flat|nested] 65+ messages in thread

end of thread, other threads:[~2023-12-28  9:41 UTC | newest]

Thread overview: 65+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2021-12-16 16:17 [bug#52555] [RFC PATCH 0/3] Decentralized substitute distribution with ERIS pukkamustard
2021-12-16 16:20 ` [bug#52555] [RFC PATCH 1/3] publish: Add ERIS URN to narinfo pukkamustard
2021-12-16 16:20   ` [bug#52555] [RFC PATCH 2/3] WIP: gnu: guile-eris: Update to unreleased git version pukkamustard
2021-12-16 16:20   ` [bug#52555] [RFC PATCH 3/3] publish: Add IPFS support pukkamustard
2021-12-20 16:25 ` [bug#52555] [RFC PATCH 0/3] Decentralized substitute distribution with ERIS Ludovic Courtès
2021-12-23 11:42   ` pukkamustard
2021-12-24 14:48     ` Ludovic Courtès
2022-01-25 19:21 ` [bug#52555] [RFC PATCH v2 0/5] " pukkamustard
2022-01-25 19:21   ` [bug#52555] [RFC PATCH v2 1/5] WIP: gnu: guile-eris: Update to unreleased git version pukkamustard
2022-01-25 19:21   ` [bug#52555] [RFC PATCH v2 2/5] publish: Add ERIS URN to narinfo pukkamustard
2022-01-29 21:09     ` Maxime Devos
2022-01-29 21:15     ` Maxime Devos
2022-02-02 10:16       ` pukkamustard
2022-01-25 19:21   ` [bug#52555] [RFC PATCH v2 3/5] Add (guix eris) pukkamustard
2022-01-29 21:23     ` Maxime Devos
2022-02-02 10:28       ` pukkamustard
2022-02-02 15:36         ` Maxime Devos
2022-01-29 21:24     ` Maxime Devos
2022-01-25 19:22   ` [bug#52555] [RFC PATCH v2 4/5] publish: Add support for storing ERIS encoded blocks to IPFS pukkamustard
2022-01-29 21:28     ` Maxime Devos
2022-02-02 10:24       ` pukkamustard
2022-01-25 19:22   ` [bug#52555] [RFC PATCH v2 5/5] substitute: Fetch substitutes using ERIS pukkamustard
2022-01-29 21:29     ` Maxime Devos
2022-02-02 10:11       ` pukkamustard
2022-01-29 21:33     ` Maxime Devos
2022-01-29 21:38     ` Maxime Devos
2022-01-29 21:40       ` Maxime Devos
2022-01-29 21:40     ` Maxime Devos
2022-02-02 10:38       ` pukkamustard
2022-01-29 21:00   ` [bug#52555] [RFC PATCH v2 0/5] Decentralized substitute distribution with ERIS Maxime Devos
2022-02-02  9:50     ` pukkamustard
2022-01-29 21:08   ` Maxime Devos
2022-02-02  9:56     ` pukkamustard
2022-02-02 11:09       ` Maxime Devos
2022-01-29 21:52   ` Maxime Devos
2022-02-02 11:10     ` pukkamustard
2022-02-03 20:36       ` Maxime Devos
2022-02-04 10:20         ` pukkamustard
2022-01-30 11:46   ` Maxime Devos
2022-02-02 10:51     ` pukkamustard
2022-02-02 11:27       ` Maxime Devos
2022-02-02 12:42         ` pukkamustard
2022-02-02 15:07           ` Maxime Devos
2022-02-02 15:27           ` Maxime Devos
2022-02-04 16:16       ` Maxime Devos
2022-12-29 18:13 ` [bug#52555] [PATCH v3 0/8] " pukkamustard
2022-12-29 18:13   ` [bug#52555] [PATCH v3 1/8] publish: Add ERIS URN to narinfo pukkamustard
2023-01-14 18:34     ` [bug#52555] [RFC PATCH 0/3] Decentralized substitute distribution with ERIS Ludovic Courtès
2022-12-29 18:13   ` [bug#52555] [PATCH v3 2/8] publish: Store ERIS encoded blocks to a local block store pukkamustard
2023-01-14 18:42     ` [bug#52555] [RFC PATCH 0/3] Decentralized substitute distribution with ERIS Ludovic Courtès
2022-12-29 18:13   ` [bug#52555] [PATCH v3 3/8] publish: Add HTTP endpoint for resolving ERIS blocks pukkamustard
2022-12-29 18:13   ` [bug#52555] [PATCH v3 4/8] WIP: substitute: Fetch substitutes using ERIS pukkamustard
2022-12-29 18:13   ` [bug#52555] [PATCH v3 5/8] eris/http: Add HTTP block de-referencer pukkamustard
2022-12-29 18:13   ` [bug#52555] [PATCH v3 6/8] WIP: eris: Use HTTP to get ERIS blocks pukkamustard
2022-12-29 18:13   ` [bug#52555] [PATCH v3 7/8] eris: Use parameterized %eris-peers when getting blocks pukkamustard
2022-12-29 18:13   ` [bug#52555] [PATCH v3 8/8] eris: Use IPFS to get ERIS blocks pukkamustard
2023-01-14 18:25   ` [bug#52555] [RFC PATCH 0/3] Decentralized substitute distribution with ERIS Ludovic Courtès
2023-12-28  9:40 ` [bug#52555] [PATCH v4 0/7] " pukkamustard
2023-12-28  9:40   ` [bug#52555] [PATCH v4 1/7] narinfo: Add ERIS field pukkamustard
2023-12-28  9:40   ` [bug#52555] [PATCH v4 2/7] gnu: Add guile-coap pukkamustard
2023-12-28  9:40   ` [bug#52555] [PATCH v4 3/7] gnu: guile-eris: Update to 1.2.0-dev pukkamustard
2023-12-28  9:40   ` [bug#52555] [PATCH v4 4/7] publish: Add ERIS URN to narinfo pukkamustard
2023-12-28  9:40   ` [bug#52555] [PATCH v4 5/7] eris: Connect with an ERIS Store over CoAP+Unix pukkamustard
2023-12-28  9:40   ` [bug#52555] [PATCH v4 6/7] substitute: Decode substitutes using ERIS pukkamustard
2023-12-28  9:40   ` [bug#52555] [PATCH v4 7/7] gnu: Add kapla pukkamustard

Code repositories for project(s) associated with this external index

	https://git.savannah.gnu.org/cgit/guix.git

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.