From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp1 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id eF3cL8Vmu2G7UgEAgWs5BA (envelope-from ) for ; Thu, 16 Dec 2021 17:18:13 +0100 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp1 with LMTPS id 4Ch2K8Vmu2GaZQAAbx9fmQ (envelope-from ) for ; Thu, 16 Dec 2021 16:18:13 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 1E5C7C446 for ; Thu, 16 Dec 2021 17:18:13 +0100 (CET) Received: from localhost ([::1]:36622 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mxtSO-0008Oo-5q for larch@yhetil.org; Thu, 16 Dec 2021 11:18:12 -0500 Received: from eggs.gnu.org ([209.51.188.92]:55614) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mxtSF-0008Kp-14 for guix-patches@gnu.org; Thu, 16 Dec 2021 11:18:03 -0500 Received: from debbugs.gnu.org ([209.51.188.43]:53180) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1mxtSE-0002zX-Nt for guix-patches@gnu.org; Thu, 16 Dec 2021 11:18:02 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1mxtSE-0003s3-Jt for guix-patches@gnu.org; Thu, 16 Dec 2021 11:18:02 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#52555] [RFC PATCH 0/3] Decentralized substitute distribution with ERIS Resent-From: pukkamustard Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Thu, 16 Dec 2021 16:18:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: report 52555 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 52555@debbugs.gnu.org Cc: pukkamustard , ~pukkamustard/eris@lists.sr.ht X-Debbugs-Original-To: guix-patches@gnu.org Received: via spool by submit@debbugs.gnu.org id=B.163967147614866 (code B ref -1); Thu, 16 Dec 2021 16:18:02 +0000 Received: (at submit) by debbugs.gnu.org; 16 Dec 2021 16:17:56 +0000 Received: from localhost ([127.0.0.1]:36493 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mxtS1-0003rY-Se for submit@debbugs.gnu.org; Thu, 16 Dec 2021 11:17:56 -0500 Received: from lists.gnu.org ([209.51.188.17]:37690) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mxtRx-0003rN-Ab for submit@debbugs.gnu.org; Thu, 16 Dec 2021 11:17:48 -0500 Received: from eggs.gnu.org ([209.51.188.92]:55568) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mxtRu-0007zZ-Gl for guix-patches@gnu.org; Thu, 16 Dec 2021 11:17:43 -0500 Received: from mout01.posteo.de ([185.67.36.65]:54721) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mxtRr-0002xU-RE for guix-patches@gnu.org; Thu, 16 Dec 2021 11:17:42 -0500 Received: from submission (posteo.de [89.146.220.130]) by mout01.posteo.de (Postfix) with ESMTPS id A7664240027 for ; Thu, 16 Dec 2021 17:17:36 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=posteo.net; s=2017; t=1639671456; bh=LCOS0Ruzr1yXnOxpCv/ZshQ+w7IBoj9ooeByvyiV5d8=; h=From:To:Cc:Subject:Date:From; b=owd1T1Cl32kJHXMZf3WHFiPOsjnFVOytH4ghqM0GkhJppwP2dm/+FhcWST8y0jQaY xaz4l+BbbwPuyq2VuvI6a5T3AR5gSTVIcDp78szjoVq5/1egww/hG5Xn+xt538NMSO bkLL6RW2bEKcnQ40FniNb4CeLpO5dJnjkHF7clurTmifj4HuQ1YIRf6FRtud1oivee 3WSf/fz6ccTB+TzqjCLdVnYfCtK4iHo/Nscb5VOx/cB4qL2/zZUT+th9JHJ7O1SDyo oMWAft+8/jnyeQvogoCndtkL+Te/69ZSuMIB90XFQdk/MKs2NoXv7VXTSYp8MDL4so UMHM/LeItkubw== Received: from customer (localhost [127.0.0.1]) by submission (posteo.de) with ESMTPSA id 4JFHML4ndkz9rxQ; Thu, 16 Dec 2021 17:17:34 +0100 (CET) From: pukkamustard Date: Thu, 16 Dec 2021 16:17:24 +0000 Message-Id: <20211216161724.547-1-pukkamustard@posteo.net> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Received-SPF: pass client-ip=185.67.36.65; envelope-from=pukkamustard@posteo.net; helo=mout01.posteo.de X-Spam_score_int: -43 X-Spam_score: -4.4 X-Spam_bar: ---- X-Spam_report: (-4.4 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+larch=yhetil.org@gnu.org Sender: "Guix-patches" X-Migadu-Flow: FLOW_IN X-Migadu-Country: US ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1639671493; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding:resent-cc: resent-from:resent-sender:resent-message-id:list-id:list-help: list-unsubscribe:list-subscribe:list-post:dkim-signature; bh=PjnH3m+b13mc/pGgcPw8M2ZMqS9OnZE5TrwsZ61CNz0=; b=j2NxOXJSLWgAN/dhc0qUKXrqI+DfIofTd7Yiq8RNVOmBV9H703hrAq/tju3DBie/WxvCWi aBnosUsZGrD/LnZxaNmgCZ+UajpJMw0E0nUQ7MiQyMTLPKtD4pIbMWe5wP+GUzE2G55P+B 4xYl0KVKOXQzEx2eh4dm8ymSnD6eMNNfpWc1QDRpmHjLaw4QOeDHLkgtXYeP1YKHdDO7Ob 9iYmDZjTRiMzCTQaa81jAgQqdBYoMDL0yffSvBvjlnZWHDsb+fONOyomp5yp7k7EFEPXwQ z+HY3/9MNLkFNyrYNPi2kuQDH+r8RrBCYw1QcgTgi8VbOdOHKRvYo6YoRBq1HQ== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1639671493; a=rsa-sha256; cv=none; b=j01C6yCtJ5XRxiQCCtnjAL9sCQSKmqA4CGvVtSncPfrVqCxImGwvWhzuVEDhjnJQlhsddy b0diBlNf3CshsPABfyPtteftyJz8RhphPbg97TQjSofLAKOhyzb0v6ZNrqs2mYtbM2UUPm eIb0ER9w1GUkUL8zPmSomvX6Pfg0MEItxfITrIJec2QhyH1bDmimgkZA/ALjQVM31QcZfu QSTj5xYDq+U+LLLXuP3aV6v0/hw0Lt0v7y+M+skD8H4LIvDZh5qsAgNs3OuosrVprtsSAw 64Zewt7Y1UVApKm0Odo1TgGQNypb9I+iuSHQxD7dfphnTMb3m/NUgUH3Yl/bDA== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=posteo.net header.s=2017 header.b=owd1T1Cl; dmarc=fail reason="SPF not aligned (strict)" header.from=posteo.net (policy=none); spf=pass (aspmx1.migadu.com: domain of "guix-patches-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-patches-bounces+larch=yhetil.org@gnu.org" X-Migadu-Spam-Score: -2.19 Authentication-Results: aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=posteo.net header.s=2017 header.b=owd1T1Cl; dmarc=fail reason="SPF not aligned (strict)" header.from=posteo.net (policy=none); spf=pass (aspmx1.migadu.com: domain of "guix-patches-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-patches-bounces+larch=yhetil.org@gnu.org" X-Migadu-Queue-Id: 1E5C7C446 X-Spam-Score: -2.19 X-Migadu-Scanner: scn0.migadu.com X-TUID: z9RN6MQ7mOG9 Hello Guix, This is an initial patch and proposal towards decentralizing substitute distribution with ERIS. ERIS (Encoding for Robust Immutable Storage) [1] is an encoding of content into uniformly sized, encryped and content-addressed blocks. The original content can be reconstructed only with access to a read capability, which can be encoded as an URN. One key advantage of ERIS is that the encoding is protocol agnostic. Any protocol that can transfer small (32KiB) sized blocks referenced by the hash of their content will do. This can be done with things such as GNUNet, IPFS, OpenDHT, HTTP or a USB stick on a bicycle. The following patch allows substitutes to be published over IPFS using ERIS. This is inspired and very similar to previous work on distributing substitutes over IPFS [2]. The narinfos served by `guix publish` look like this: --8<---------------cut here---------------start------------->8--- StorePath: /gnu/store/81bdcd5x4v50i28h98bfkvvkx9cky63w-hello-2.10 URL: nar/gzip/81bdcd5x4v50i28h98bfkvvkx9cky63w-hello-2.10 Compression: gzip FileSize: 67363 ERIS: urn:erisx2:BIBC2LUTIQH43S2KRIAV7TBXNUUVPZTMV6KFA2M7AL5V6FNE77VNUDDVDAGJUEEAFATVO2QQT67SMOPTO3LGWCJFU7BZVCF5VXEQQW25BE URL: nar/zstd/81bdcd5x4v50i28h98bfkvvkx9cky63w-hello-2.10 Compression: zstd FileSize: 64917 ERIS: urn:erisx2:BIBO7KS7SAWHDNC43DVILOSQ3F3SRRHEV6YPLDCSZ7MMD6LZVCHQMEQ6FUBTJAPSNFF7XR5XPTP4OQ72OPABNEO7UYBUN42O46ARKHBTGM NarHash: sha256:1sagsz1mnlqkr8r8s6gwkzvvhq619rlzhpbxl3h0b111n5hn2w9w NarSize: 220704 References: 2fk1gz2s7ppdicynscra9b19byrrr866-glibc-2.33 81bdcd5x4v50i28h98bfkvvkx9cky63w-hello-2.10 90lbavffg0csrf208nw0ayj1bz5knl47-gcc-10.3.0-lib Deriver: 260bk0ch4np4h2yz5yqhf8hjbsyhwpmr-hello-2.10.drv Signature: 1;strawberry;KHNpZ25hdHVyZSAKIChkYXRhIAogIChmbGFncyByZmM2OTc5KQogIChoYXNoIHNoYTI1NiAjNDk4ODkwODZDNTY4MzQyRENFQzk3QzA3NDE4NEQ1RkRCOTNCNDA2MUNCRDM4MUExRjVBQzVDODI0MTgwMTU3OSMpCiAgKQogKHNpZy12YWwgCiAgKGVjZHNhIAogICAociAjMEU2NDlFODE4QzRFNjNGNEY2OUQ5QTAwRjUwNjRDMzQ3QjY3RDM0RTM0NTg2MkI4NTc3RTg5MUY5Q0Q3NDhBQiMpCiAgIChzICMwMTZGRjA1MDdCQjZGMzA2NUEzMjYzRDA2MTAyRDc5MTBEOEZGODc5RTdENjREODRFODBENDBGMTJFMTBBOTQ1IykKICAgKQogICkKIChwdWJsaWMta2V5IAogIChlY2MgCiAgIChjdXJ2ZSBFZDI1NTE5KQogICAocSAjMDRDMkY4ODk1QTU0NDNGNTlCODk2NDEwMEI1MDY0NzU4RjQ1N0YzMENEREE1MTQyQzE0MDc0NjExNTA1NTc5MCMpCiAgICkKICApCiApCg== --8<---------------cut here---------------end--------------->8--- For every compressed nar the ERIS URN is computed and added. If the `--ipfs` is used for `guix publish` then the encoded blocks are also uploaded to the IPFS daemon. The nar could then be retrieved from anywhere like this: --8<---------------cut here---------------start------------->8--- (use-modules (eris) (eris blocks ipfs)) (eris-decode->bytevector "urn:erisx2:BIBC2LUTIQH43S2KRIAV7TBXNUUVPZTMV6KFA2M7AL5V6FNE77VNUDDVDAGJUEEAFATVO2QQT67SMOPTO3LGWCJFU7BZVCF5VXEQQW25BE" eris-blocks-ipfs-ref) --8<---------------cut here---------------end--------------->8--- These patches do not yet retrieve content from IPFS (TODO). But in principle, anybody connected to IPFS can get the nar with the ERIS URN. This could be used to reduce load on substitute server as they would only need to publish the ERIS URN directly - substitutes could be delivered much more peer-to-peer. Other transports that I have been looking in to and am pretty sure will work include: HTTP (with RFC 2169 [3]), GNUNet, OpenDHT. This is, imho, the advantage of ERIS over IPFS directly or GNUNet directly. The encoding and identifiers (URN) are abstracted away from specific transports (and also applications). ERIS is almost exactly the same encoding as used in GNUNet (ECRS). Blocks can be stored in any kind of databases (see for example the GDBM bindings [4]). A tricky things is figuring out how to multiplex all these different transports and storages... The ERIS specification is still considered "experimental". However we feel confident to stabilize it and intend to do so around February/March 2022 with a release 1.0.0 of the specification. This will ensure that the identifiers remain stable for the forseeable future (until the crypto breaks). Before that there is also a small external security audit of the specification planned (thanks to NGI0/NLnet!). This is just a little demo of the idea and some food for thought and discussion. Give it a try and let me know what you think! I've also pushed the patches to my personal Guix mirror if you want to check it out from there: https://inqlab.net/git/guix.git/log/?h=wip-eris Also CCing ~pukkamustard/eris@lists.sr.ht where there is some general ERIS related discussion. Thanks, -pukkamustard [1] http://purl.org/eris [2] https://issues.guix.gnu.org/33899 [3] https://www.ietf.org/rfc/rfc2169.txt [4] https://inqlab.net/git/guile-eris.git/tree/eris/blocks/gdbm.scm pukkamustard (3): publish: Add ERIS URN to narinfo WIP: gnu: guile-eris: Update to unreleased git version. publish: Add IPFS support. configure.ac | 5 ++ gnu/packages/guile-xyz.scm | 10 ++-- gnu/packages/package-management.scm | 1 + guix/narinfo.scm | 10 ++-- guix/scripts/publish.scm | 79 ++++++++++++++++++++++------- 5 files changed, 79 insertions(+), 26 deletions(-) -- 2.34.0