[bug#52555] [RFC PATCH 0/3] Decentralized substitute distribution with ERIS

["Ludovic Courtès" <ludo@gnu.org>]

Hi pukkamustard,

pukkamustard <pukkamustard@posteo.net> skribis:

> This is an initial patch and proposal towards decentralizing substitute
> distribution with ERIS.

Woohoo, sounds exciting!

> ERIS (Encoding for Robust Immutable Storage) [1] is an encoding of content into
> uniformly sized, encryped and content-addressed blocks. The original content
> can be reconstructed only with access to a read capability, which can be
> encoded as an URN.
>
> One key advantage of ERIS is that the encoding is protocol agnostic. Any
> protocol that can transfer small (32KiB) sized blocks referenced by the hash of
> their content will do. This can be done with things such as GNUNet, IPFS,
> OpenDHT, HTTP or a USB stick on a bicycle.

Yes, that’s nice.

> The following patch allows substitutes to be published over IPFS using ERIS.
> This is inspired and very similar to previous work on distributing substitutes
> over IPFS [2].
>
> The narinfos served by `guix publish` look like this:
>
> StorePath: /gnu/store/81bdcd5x4v50i28h98bfkvvkx9cky63w-hello-2.10
> URL: nar/gzip/81bdcd5x4v50i28h98bfkvvkx9cky63w-hello-2.10
> Compression: gzip
> FileSize: 67363
> ERIS: urn:erisx2:BIBC2LUTIQH43S2KRIAV7TBXNUUVPZTMV6KFA2M7AL5V6FNE77VNUDDVDAGJUEEAFATVO2QQT67SMOPTO3LGWCJFU7BZVCF5VXEQQW25BE
> URL: nar/zstd/81bdcd5x4v50i28h98bfkvvkx9cky63w-hello-2.10
> Compression: zstd
> FileSize: 64917
> ERIS: urn:erisx2:BIBO7KS7SAWHDNC43DVILOSQ3F3SRRHEV6YPLDCSZ7MMD6LZVCHQMEQ6FUBTJAPSNFF7XR5XPTP4OQ72OPABNEO7UYBUN42O46ARKHBTGM

Do we really need one URN per compression method?  Couldn’t we leave
compression (of individual chunks, possibly) as a “detail” handled by
the encoding or the transport layer?

> If the `--ipfs` is used for `guix publish` then the encoded blocks are also
> uploaded to the IPFS daemon. The nar could then be retrieved from anywhere like
> this:
>
> (use-modules (eris)
> 	     (eris blocks ipfs))
>
> (eris-decode->bytevector
>  "urn:erisx2:BIBC2LUTIQH43S2KRIAV7TBXNUUVPZTMV6KFA2M7AL5V6FNE77VNUDDVDAGJUEEAFATVO2QQT67SMOPTO3LGWCJFU7BZVCF5VXEQQW25BE"
>  eris-blocks-ipfs-ref)
>
> These patches do not yet retrieve content from IPFS (TODO). But in principle,
> anybody connected to IPFS can get the nar with the ERIS URN. This could be used
> to reduce load on substitute server as they would only need to publish the ERIS
> URN directly - substitutes could be delivered much more peer-to-peer.

Nice.  So adjusting ‘guix substitute’ should be relatively easy?

> Other transports that I have been looking in to and am pretty sure will work
> include: HTTP (with RFC 2169 [3]), GNUNet, OpenDHT. This is, imho, the
> advantage of ERIS over IPFS directly or GNUNet directly. The encoding and
> identifiers (URN) are abstracted away from specific transports (and also
> applications). ERIS is almost exactly the same encoding as used in GNUNet
> (ECRS).

As a first step, ‘guix publish’ could implement RFC 2169, too.

I gather implementing the HTTP and IPFS backends in ‘guix substitute’
should be relatively easy, right?

> Blocks can be stored in any kind of databases (see for example the GDBM
> bindings [4]).
>
> A tricky things is figuring out how to multiplex all these different
> transports and storages...

Yes.  We don’t know yet what performance and data availability will be
like on IPFS, for instance, so it’s important for users to be able to
set priorities.  It’s also important to gracefully fall back to direct
HTTP downloads when fancier p2p methods fail, regardless of how they
fail.

> The ERIS specification is still considered "experimental". However we feel
> confident to stabilize it and intend to do so around February/March 2022 with a
> release 1.0.0 of the specification. This will ensure that the identifiers
> remain stable for the forseeable future (until the crypto breaks). Before that
> there is also a small external security audit of the specification planned
> (thanks to NGI0/NLnet!).

Neat.

This is all very exciting.  I look forward to playing around with it!

Ludo’.