From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp2 ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id kPpgGY/UwGGQwwAAgWs5BA (envelope-from ) for ; Mon, 20 Dec 2021 20:07:59 +0100 Received: from aspmx1.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp2 with LMTPS id cNsIFY/UwGFKfQAAB5/wlQ (envelope-from ) for ; Mon, 20 Dec 2021 19:07:59 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 1BB5E15203 for ; Mon, 20 Dec 2021 20:07:55 +0100 (CET) Received: from localhost ([::1]:41080 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mzO0o-0008SH-8S for larch@yhetil.org; Mon, 20 Dec 2021 14:07:54 -0500 Received: from eggs.gnu.org ([209.51.188.92]:53734) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mzLUA-00043y-Gb for guix-patches@gnu.org; Mon, 20 Dec 2021 11:26:02 -0500 Received: from debbugs.gnu.org ([209.51.188.43]:39684) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1mzLUA-0001dO-8D for guix-patches@gnu.org; Mon, 20 Dec 2021 11:26:02 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1mzLUA-00059L-30 for guix-patches@gnu.org; Mon, 20 Dec 2021 11:26:02 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#52555] [RFC PATCH 0/3] Decentralized substitute distribution with ERIS Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Mon, 20 Dec 2021 16:26:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 52555 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: pukkamustard Cc: ~pukkamustard/eris@lists.sr.ht, 52555@debbugs.gnu.org Received: via spool by 52555-submit@debbugs.gnu.org id=B52555.164001751919703 (code B ref 52555); Mon, 20 Dec 2021 16:26:02 +0000 Received: (at 52555) by debbugs.gnu.org; 20 Dec 2021 16:25:19 +0000 Received: from localhost ([127.0.0.1]:51230 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mzLTT-00057j-1o for submit@debbugs.gnu.org; Mon, 20 Dec 2021 11:25:19 -0500 Received: from hera.aquilenet.fr ([185.233.100.1]:43422) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mzLTR-00057Q-De for 52555@debbugs.gnu.org; Mon, 20 Dec 2021 11:25:18 -0500 Received: from localhost (localhost [127.0.0.1]) by hera.aquilenet.fr (Postfix) with ESMTP id 927AC361; Mon, 20 Dec 2021 17:25:10 +0100 (CET) X-Virus-Scanned: Debian amavisd-new at aquilenet.fr Received: from hera.aquilenet.fr ([127.0.0.1]) by localhost (hera.aquilenet.fr [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 1Wfl5kQPKPVV; Mon, 20 Dec 2021 17:25:09 +0100 (CET) Received: from ribbon (unknown [193.50.110.211]) by hera.aquilenet.fr (Postfix) with ESMTPSA id D191B63; Mon, 20 Dec 2021 17:25:08 +0100 (CET) From: Ludovic =?UTF-8?Q?Court=C3=A8s?= References: <20211216161724.547-1-pukkamustard@posteo.net> Date: Mon, 20 Dec 2021 17:25:07 +0100 In-Reply-To: <20211216161724.547-1-pukkamustard@posteo.net> (pukkamustard@posteo.net's message of "Thu, 16 Dec 2021 16:17:24 +0000") Message-ID: <87h7b3gs64.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spamd-Bar: / X-Rspamd-Server: hera X-Rspamd-Queue-Id: 927AC361 X-Spamd-Result: default: False [-0.10 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; TO_DN_SOME(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; MID_RHS_MATCH_FROM(0.00)[] X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+larch=yhetil.org@gnu.org Sender: "Guix-patches" X-Migadu-Flow: FLOW_IN X-Migadu-Country: US ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1640027275; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:resent-cc: resent-from:resent-sender:resent-message-id:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post; bh=zrcbMxXPajLzlmbCD5DOtSuTtcYgfXnHvfkz54KDWYM=; b=uWNF0S8uR9wPnwjr/TidZ2zZe0vum/ALR/1BD79sEScWmgUmncLo/1SpaZm9s4OoCCa3ep 3bhJztvqbGXMyV3rLDtKHuIB1+Ppl0/lmmrMgkxEJGOYzusQ41xnLvLC9vNkVUI0qkv5qe TEtSOxergfu4CcW5Nn29pukb0xzhIatYL0I7S7uqJz4kRqsvB8wCZpAPa4CgW1fi6yHy2Q vkfh3LdfVZJHGiOhh0VIeyOlc4iEHjTyTzyaGBL+FRS/W2I2QkAUF9H/8ag/sgTUJ/I7N3 s60mQsxR0sToa49znsQmaQPSQ1fOe5r+wZTEIDFFTjqpcrrG2oLdUepV8RbOBw== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1640027275; a=rsa-sha256; cv=none; b=dNyvXKxR7bC+g4ZXc2ZdvzU99vVAo9crQ2PBVKKKm/hrGkHHPfGlT8VvGv/OpwcHHNbTED NS3r1Ji+KCnUPOn8dvJjX+muJB95cJvkBnKCvoBSytXgDP5RNs0ajge+tThDQdK6/MtQ85 ceu9JySE8hnBpDgVb8Nwu6c03MH4ba1uxjIRpKtR+2LhJZues/7ZdTcIHmAwFgVrGdZ2wT GgtqRaT3M3f6Dz3zZWsXp/XxG6kDuKIQokR0tHdLdQ7Uvf8GE+wH88oSwP7NRK+Nxqcqjz OyrbOZs7TC2IRudkjwSBSJ/z/3+Fhu+Elzs038f80a7d3VhEfnBlF39Moky9+Q== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=none; dmarc=pass (policy=none) header.from=gnu.org; spf=pass (aspmx1.migadu.com: domain of "guix-patches-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-patches-bounces+larch=yhetil.org@gnu.org" X-Migadu-Spam-Score: -3.52 Authentication-Results: aspmx1.migadu.com; dkim=none; dmarc=pass (policy=none) header.from=gnu.org; spf=pass (aspmx1.migadu.com: domain of "guix-patches-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-patches-bounces+larch=yhetil.org@gnu.org" X-Migadu-Queue-Id: 1BB5E15203 X-Spam-Score: -3.52 X-Migadu-Scanner: scn0.migadu.com X-TUID: aAVkQGV0aX6w Hi pukkamustard, pukkamustard skribis: > This is an initial patch and proposal towards decentralizing substitute > distribution with ERIS. Woohoo, sounds exciting! > ERIS (Encoding for Robust Immutable Storage) [1] is an encoding of conten= t into > uniformly sized, encryped and content-addressed blocks. The original cont= ent > can be reconstructed only with access to a read capability, which can be > encoded as an URN. > > One key advantage of ERIS is that the encoding is protocol agnostic. Any > protocol that can transfer small (32KiB) sized blocks referenced by the h= ash of > their content will do. This can be done with things such as GNUNet, IPFS, > OpenDHT, HTTP or a USB stick on a bicycle. Yes, that=E2=80=99s nice. > The following patch allows substitutes to be published over IPFS using ER= IS. > This is inspired and very similar to previous work on distributing substi= tutes > over IPFS [2]. > > The narinfos served by `guix publish` look like this: > > StorePath: /gnu/store/81bdcd5x4v50i28h98bfkvvkx9cky63w-hello-2.10 > URL: nar/gzip/81bdcd5x4v50i28h98bfkvvkx9cky63w-hello-2.10 > Compression: gzip > FileSize: 67363 > ERIS: urn:erisx2:BIBC2LUTIQH43S2KRIAV7TBXNUUVPZTMV6KFA2M7AL5V6FNE77VNUDDV= DAGJUEEAFATVO2QQT67SMOPTO3LGWCJFU7BZVCF5VXEQQW25BE > URL: nar/zstd/81bdcd5x4v50i28h98bfkvvkx9cky63w-hello-2.10 > Compression: zstd > FileSize: 64917 > ERIS: urn:erisx2:BIBO7KS7SAWHDNC43DVILOSQ3F3SRRHEV6YPLDCSZ7MMD6LZVCHQMEQ6= FUBTJAPSNFF7XR5XPTP4OQ72OPABNEO7UYBUN42O46ARKHBTGM Do we really need one URN per compression method? Couldn=E2=80=99t we leave compression (of individual chunks, possibly) as a =E2=80=9Cdetail=E2=80=9D = handled by the encoding or the transport layer? > If the `--ipfs` is used for `guix publish` then the encoded blocks are al= so > uploaded to the IPFS daemon. The nar could then be retrieved from anywher= e like > this: > > (use-modules (eris) > (eris blocks ipfs)) > > (eris-decode->bytevector > "urn:erisx2:BIBC2LUTIQH43S2KRIAV7TBXNUUVPZTMV6KFA2M7AL5V6FNE77VNUDDVDAGJ= UEEAFATVO2QQT67SMOPTO3LGWCJFU7BZVCF5VXEQQW25BE" > eris-blocks-ipfs-ref) > > These patches do not yet retrieve content from IPFS (TODO). But in princi= ple, > anybody connected to IPFS can get the nar with the ERIS URN. This could b= e used > to reduce load on substitute server as they would only need to publish th= e ERIS > URN directly - substitutes could be delivered much more peer-to-peer. Nice. So adjusting =E2=80=98guix substitute=E2=80=99 should be relatively = easy? > Other transports that I have been looking in to and am pretty sure will w= ork > include: HTTP (with RFC 2169 [3]), GNUNet, OpenDHT. This is, imho, the > advantage of ERIS over IPFS directly or GNUNet directly. The encoding and > identifiers (URN) are abstracted away from specific transports (and also > applications). ERIS is almost exactly the same encoding as used in GNUNet > (ECRS). As a first step, =E2=80=98guix publish=E2=80=99 could implement RFC 2169, t= oo. I gather implementing the HTTP and IPFS backends in =E2=80=98guix substitut= e=E2=80=99 should be relatively easy, right? > Blocks can be stored in any kind of databases (see for example the GDBM > bindings [4]). > > A tricky things is figuring out how to multiplex all these different > transports and storages... Yes. We don=E2=80=99t know yet what performance and data availability will= be like on IPFS, for instance, so it=E2=80=99s important for users to be able = to set priorities. It=E2=80=99s also important to gracefully fall back to dir= ect HTTP downloads when fancier p2p methods fail, regardless of how they fail. > The ERIS specification is still considered "experimental". However we feel > confident to stabilize it and intend to do so around February/March 2022 = with a > release 1.0.0 of the specification. This will ensure that the identifiers > remain stable for the forseeable future (until the crypto breaks). Before= that > there is also a small external security audit of the specification planned > (thanks to NGI0/NLnet!). Neat. This is all very exciting. I look forward to playing around with it! Ludo=E2=80=99.