[bug#52555] [PATCH v4 0/7] Decentralized substitute distribution with ERIS

[pukkamustard <pukkamustard@posteo.net>]

Dear Guix,

This is the V4 of a proposal towards decentralized subsitute distirbution
using the ERIS encoding. The initial proposal was submitted in December 2021,
V2 and V3 a year later in 2022. All is still very much work-in-progress, but
I'm happy to submit it as is to keep up with good old traditions.

Thank you also for all the valuable comments and feedback via mail and live at
Guix Days last year. Sorry for being so slow to react.

The general idea of the proposal is to use ERIS (http://purl.org/eris) for
substitute distribution. This allows substitutes to be shared over various
protocols such as IPFS, GNUnet, NNCP, HTTP, CoAP, a USB stick or some
Spritely-esque fun.

ERIS itself defines an encoding of content into uniformly sized, encrypted and
content-addressed blocks. The original content can be decoded with a short
read capability that can be encoded as an URN and access to the blocks that
make up the content. Blocks can be shared over different protocols with low
security requirements on the transport layer itself. Read capabilities need to
be shared in a secure manner.

This series does following:

- Adds an `ERIS` field holding the ERIS read capability of a substitute to the
  Narinfo as published by `guix publish`
- Allows the daemon/substitute script to decode content using the ERIS URN
  instead of fetching a substitute via HTTP.
  
## Changes to previous version

### Fibers

When encoding and decoding it is important for performance to be able to fetch
multiple blocks concurrently (from potentially multiple peers). For this we
use Guile fibers.

Fiberization is currently very crude as there seem to be weird interactions
with the parallization strategies currently used by `guix publish` and `guix
substitute`. Help here is extremely welcome.

### ERIS-FS

In previous versions we encoded the Nar-file of the susbstitute. In this
version we use a specialized encoding of file-system trees for ERIS: ERIS-FS
(https://eris.codeberg.page/eer/eris-fs.xml).

ERIS-FS allows de-duplication of files and much easier parlallized decoding of
substitutes. The ideas are similar to the custom encoding defined for
substitutes over IPFS (https://issues.guix.gnu.org/33899).

One major consequence is that the SHA256 sum of the Nar file as published in
the Narinfo is no longer used to verify integrity of downloaded
susbtitute. Instead we must ensure that the `ERIS` field in the Narinfo is
trusted, `equivalent-narinfo?` and such have been changed towards this. Thanks
to Maxime for pointing out this subtlety!

### External Block Store Daemon

Ludovic Courtès <ludo@gnu.org> writes:

> Also, the store directory should be /var/cache/guix/publish/eris by
> default IMO.

This comment caused a lot of pondering and hacking! :)

The problem seems to be that two processes need to be able to access the block
store securely: `guix publish` when publishing substitutes and the Guix daemon
when fetching substitutes (blocks need to be stored when fetching for
de-duplication). `guix publish` is usually run as a non-privileged user,
whereas the Guix daemon runs with much higher privileges. The block store
needs to be secure in the sense that manipulating blocks should be prevented
by less privileged processes.

I'm afraid the only solution I came up with is that the block store is not a
Unix directory or database, but a daemon that listens on a Unix Socket.

Enter Kapla (https://codeberg.org/eris/kapla). Kapla is a Guile program that
stores and transports blocks. The Guix daemon (via substitute scripts) and the
`guix publish` server talk to Kapla over a Unix socket. Kapla stores blocks in
a database (SQlite) and connects with multiple peers and/or services such as
IPFS/GNUnet to get blocks.

I was fighting such an architecture for a long time. I would have preferred if
the Guix daemon and `guix publish` simply use a library for everything instead
of relying on another external service. However, there seem to be some advantages:

- Managing peers seem to be quite complex and stateful. Maybe better if this
  is externalized from Guix proper.
- The block storage and transport mechanisms can be used to share any files
  (try `kapla encode my-file` and `kapla decode ...`).
- We can experiment with more transports quicker in a service external to Guix
  and include dependencies that might not make sense to have in Guix proper.
  
The protocol that is used to talk to the external block store is CoAP over
Unix Sockets or TCP. Kapla is not the only software that speaks this
protocol. There is also a Golang block store that can be used
(https://codeberg.org/eris/eris-go).

### CoAP for block transport

Currently the protocol over which blocks are transported over networks is the
Constrained Application Protocol (CoAP)
(https://eris.codeberg.page/eer/coap.xml). CoAP is well suited as it allows
multiple asynchronous in-flight requests (unlike HTTP 1.1). This is important
for parallizing block retrieval. It also allows bi-directional requests,
making it attractive for more peer-to-peer systems where some peers might not
have a public IP.

As it is designed for constrained environment it is quite simple and the
implementation we use is in pure Guile.

More block transports can be added by implementing them in Kapla (e.g. IPFS or
GNUnet) or using something completely different than Kapla that speaks CoAP
over Unix Sockets (such as eris-go that also can use NNCP for block
transport).

### Comments to V3

I hope to have addressed all comments to the V3 (Thanks Ludo!). This included
things like organizing commits differently, using (ice-9 match) and how to
handle multiple return values.

## Testing

Testing is a bit complicated. We need to find a better and more systematic way
of doing so. Suggestions are very welcome!

Manually this is how to do it:

1. Authorize local substitutes

We will be running a local substitute server so we need to add the local
signing key to the list of authorized keys. In the system configurations:

```
  (modify-services %base-services
    (guix-service-type
     config =>
     (guix-configuration
      (inherit config)
      (authorized-keys
       (cons*
        ;; allow substitutes from ourselves for testing purposes
        (local-file "/etc/signing-key.pub")
        %default-authorized-guix-keys)))))
```

2. Configure the local Guix checkout

#+BEGIN_SRC shell
./bootstrap && ./configure --localstatedir=/var --sysconfdir=/etc && make
#+END_SRC

The ~--sysconfdir~ is required so that guix will use the ACL in ~/etc/guix/acl~.

3. Start a Kapla daemon:

```
./pre-inst-env guix build kapla -- kaplad -d
```

The `-d` option enables debug output. Kapla will listen for connection on a
Unix socket. The path will be output, e.g.:

```
2023-12-23 15:10:23 (INFO): Listening on Unix socket /run/user/1000/eris.sock
```

3. Build some package that we will 

```
$ ./pre-inst-env guix build libchop --no-grafts
/gnu/store/60m6qih391rq95ck64am8ir64z0sv0zr-libchop-0.5.2
```

4. Start a local publish server

```
sudo -E ./pre-inst-env guix publish --public-key=/etc/guix/signing-key.pub --private-key=/etc/guix/signing-key.sec --cache=/tmp/guix-publish-cache/ --eris=coap+unix:///run/user/1000/eris.sock
```

I need to run it with sudo in order to be able to use the proper signing keys.

The `--eris=STORE_URL` option defines where to store blocks. `--eris` can also
be provided without a URL which will cause the ERIS read capability to be
computed but blocks won't be stored anywhere. See "Proposed Deployment" on
when this might make sense.

5. Get the narinfo from the publish server

```
$ curl http://localhost:8080/60m6qih391rq95ck64am8ir64z0sv0zr.narinfo
StorePath: /gnu/store/60m6qih391rq95ck64am8ir64z0sv0zr-libchop-0.5.2
NarHash: sha256:1i2hhzw81qfsba0d1b09ax13694imgjrpay0122gqll85dx7k7ml
NarSize: 1021984
ERIS: urn:eris:BIARSURIJ3WYLEINE6W5ZF7LKTIHL42AE367TQ355ORW5UZVSTQGT5H5T2OLKF7XICML3VHLTLMDWXLUCQVKHRKNVREV3GMVX3J5RMT4GU
References: 1i0iz5rgixyva0zy4bmaasjil2683xrn-mit-krb5-1.20 2w976k6g70gkfih9wwhalqsni209vcqz-gdbm-1.23 4p1l5bdxxbyyqc3wh0d07jv9rp1pdcy7-guile-2.0.14 60m6qih391rq95ck64am8ir64z0sv0zr-libchop-0.5.2 620h3panf2lss42ns625rlay522g2hza-tdb-1.4.7 8y0pwifz8a3d7zbdfzsawa1amf4afx1s-libgcrypt-1.10.1 930nwsiysdvy2x5zv1sf6v7ym75z8ayk-gcc-11.3.0-lib gsjczqir1wbz8p770zndrpw4rnppmxi3-glibc-2.35 m9wi9hcrf7f9dm4ri32vw1jrbh1csywi-libgpg-error-1.45 pl09vk5g3cl8fxfln2hjk996pyahqk8m-bzip2-1.0.8 r5saysi65chivbv3y65nzsisx8rypp76-libtirpc-1.3.1 rib9g2ig1xf3kclyl076w28parmncg4k-bash-minimal-5.1.16 slzq3zqwj75lbrg4ly51hfhbv2vhryv5-zlib-1.2.13 z4likj1rnshd81hr0vwvwhdxpfwa1rz7-lzo-2.10
Deriver: zcly5fm6rsqis9csk33i7zfqjidjzy1a-libchop-0.5.2.drv
Signature: 1;strawberry;KHNpZ25hdHVyZSAKIChkYXRhIAogIChmbGFncyByZmM2OTc5KQogIChoYXNoIHNoYTI1NiAjRjFEQTI3RkJDQzA4RTI2MzZDQkY3NEE5NkMwNjQ5QzZBQ0U3QjEzNDRGNzJFNEM1OUMwMzIyRUIzMjFFMUY4RCMpCiAgKQogKHNpZy12YWwgCiAgKGVjZHNhIAogICAociAjMDQwQzYwNzBBOTlBQjI1NEFDNkMwOTdGREIxMUREMkZBNEExNTFGNzYyOTBBRUNENzdCQUMzQ0M0REVERkI0RCMpCiAgIChzICMwQzZGMEFGOTgzODg1NDE5NzAwNzI0NEE5NDU2RTYyMDAxNEE2NUI5NDgxODc4QTlFNjJDOTA2RDQ3NTVGM0YyIykKICAgKQogICkKIChwdWJsaWMta2V5IAogIChlY2MgCiAgIChjdXJ2ZSBFZDI1NTE5KQogICAocSAjMDRDMkY4ODk1QTU0NDNGNTlCODk2NDEwMEI1MDY0NzU4RjQ1N0YzMENEREE1MTQyQzE0MDc0NjExNTA1NTc5MCMpCiAgICkKICApCiApCg==
URL: nar/gzip/60m6qih391rq95ck64am8ir64z0sv0zr-libchop-0.5.2
Compression: gzip
FileSize: 345576
```

Note the new `ERIS` field. You might have to get the narinfo twice for all the
fields to appear.

The ERIS read capability URN (urn:eris:BIARSURIJ...) contains enough
information to decode the substitute. If you look at the kaplad log output you
will see that some blocks were stored. You could now run:

```
kapla decode urn:eris:BIARSURIJ3WYLEINE6W5ZF7LKTIHL42AE367TQ355ORW5UZVSTQGT5H5T2OLKF7XICML3VHLTLMDWXLUCQVKHRKNVREV3GMVX3J5RMT4GU out
```

to decode the substitute to the folder out. The Guix daemon can do the same
when getting substitutes.

6. Remove the libchop store item:

```
guix gc -D /gnu/store/60m6qih391rq95ck64am8ir64z0sv0zr-libchop-0.5.2
```

7. Start the guix daemon from the checkout:


```
sudo -E ./pre-inst-env guix-daemon --build-users-group=guixbuild --eris-store-url=coap+unix:///run/user/1001/eris.sock --substitute-urls=http://localhost:8080/
```

Note that the `--eris-store-url` argument points to the same store as what we
used for the publish server - the blocks come go to and come from the same
place. 

We need to use the local publish server for getting the narinfo with the
`ERIS` field.

8. Get the substitute:

```
./pre-inst-env guix build libchop --no-grafts
substitute: updating substitutes from 'http://localhost:8080/'.substitute: updating substitutes from 'http://localhost:8080/'.substitute: updating substitutes from 'http://localhost:8080/'.substitute: updating substitutes from 'http://localhost:8080/'.substitute: updating substitutes from 'http://localhost:8080/'.substitute: updating substitutes from 'http://localhost:8080/'.substitute: updating substitutes from 'http://localhost:8080/'.substitute: updating substitutes from 'http://localhost:8080/'.substitute: updating substitutes from 'http://localhost:8080/'... 100.0%
0.3 MB will be downloaded:
  /gnu/store/60m6qih391rq95ck64am8ir64z0sv0zr-libchop-0.5.2
substituting /gnu/store/60m6qih391rq95ck64am8ir64z0sv0zr-libchop-0.5.2...
Downloading urn:eris:BIARSURIJ3WYLEINE6W5ZF7LKTIHL42AE367TQ355ORW5UZVSTQGT5H5T2OLKF7XICML3VHLTLMDWXLUCQVKHRKNVREV3GMVX3J5RMT4GU...

/gnu/store/60m6qih391rq95ck64am8ir64z0sv0zr-libchop-0.5.2
```

Substitute was fetched using ERIS!

The daemon automatically tries to use ERIS when the `--eris-store-url` is
passed and the narinfo has a signed ERIS field. If not it will fetch
substitutes using HTTP. On failures while using ERIS it will also revert back
to getting substitutes via HTTP.

9. Restart your system guix daemon

```
sudo herd restart guix-daemon
```

10. Figure out a better way to test this.

## Proposed Deployment

As an initial deployment it would be nice if the official Guix substitute
servers include the ERIS read capability in the Narinfo withouth making blocks
available. This only requires a bit of CPU for computing the ERIS read
capability. No additional disk space or bandwidth is required by the official
substitute servers.

Users can get the signed ERIS read capability from the official Guix
susbtitute servers but can fetch blocks from anywhere while still being sure
to get the right substitutes.

This would allow community members to make blocks available independently and
experiment with various transports while not changing the trust-model when
fetching substitutes.

## TODOs

- [ ] Better fiberization of `guix publish` and `guix/scripts/substitute.scm`
- [ ] Debug issue where blocks of large substitutes are not properly stored
      (possibly related to hackey fiberization).
- [ ] Add Guix tests
- [ ] Add documentation.
- [ ] Write a RFC along the proposed Request-For-Comment process
      (https://issues.guix.gnu.org/66844)
- [ ] Allow encoding of blocks without running `guix publish` (maybe an `eris`
      format for `guix pack`).
- [ ] Dependencies (guile-eris and guile-coap) contain some fixes that need to
      be properly released.
- [ ] Implement CoAP peer connections and maybe IPFS transport in Kapla.
- [ ] Release initial version of Kapla.
- [ ] Update service definitions for `guix-publish`, `guix-daemon` with an
      option to enable decentralized substitute stuff and add service
      definition for `kapla`.

## Hic Sunt Dracones

Everything is still quite fragile. The fiberization of `guix substitute` and
`guix publish` is hackey. Kapla is still very limited and unreleased. But I
hope to have been able to capture the current state of things and paint a
picture of the plan.

Thanks for making it so far and for your comments, questions and hackings.

Greetings,
pukkamustard


pukkamustard (7):
  narinfo: Add ERIS field.
  gnu: Add guile-coap.
  gnu: guile-eris: Update to 1.2.0-dev.
  publish: Add ERIS URN to narinfo.
  eris: Connect with an ERIS Store over CoAP+Unix.
  substitute: Decode substitutes using ERIS.
  gnu: Add kapla.

 Makefile.am                         |   1 +
 configure.ac                        |   5 +
 gnu/packages/guile-xyz.scm          | 144 ++++++++++++++++++++++------
 gnu/packages/package-management.scm |   1 +
 guix/eris.scm                       | 137 ++++++++++++++++++++++++++
 guix/narinfo.scm                    |  17 +++-
 guix/scripts/publish.scm            |  58 ++++++++---
 guix/scripts/substitute.scm         |  76 +++++++++++----
 nix/nix-daemon/guix-daemon.cc       |   5 +
 9 files changed, 380 insertions(+), 64 deletions(-)
 create mode 100644 guix/eris.scm


base-commit: 4a1b3830a8ff6b05ad9a2b27c8a2cdbd00a45787
-- 
2.41.0