From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp1 ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id eH08LZZyxGHT+QAAgWs5BA (envelope-from ) for ; Thu, 23 Dec 2021 13:59:02 +0100 Received: from aspmx1.migadu.com ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp1 with LMTPS id mNbqKJZyxGHARQAAbx9fmQ (envelope-from ) for ; Thu, 23 Dec 2021 12:59:02 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 55C9D265BC for ; Thu, 23 Dec 2021 13:59:02 +0100 (CET) Received: from localhost ([::1]:50544 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1n0NgT-0003Ul-9J for larch@yhetil.org; Thu, 23 Dec 2021 07:59:01 -0500 Received: from eggs.gnu.org ([209.51.188.92]:43154) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1n0N3q-0001PS-Kd for guix-patches@gnu.org; Thu, 23 Dec 2021 07:19:07 -0500 Received: from debbugs.gnu.org ([209.51.188.43]:49024) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1n0N3m-0007ST-I9 for guix-patches@gnu.org; Thu, 23 Dec 2021 07:19:06 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1n0N3m-0003F8-DK for guix-patches@gnu.org; Thu, 23 Dec 2021 07:19:02 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#52555] [RFC PATCH 0/3] Decentralized substitute distribution with ERIS Resent-From: pukkamustard Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Thu, 23 Dec 2021 12:19:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 52555 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Ludovic =?UTF-8?Q?Court=C3=A8s?= Cc: ~pukkamustard/eris@lists.sr.ht, 52555@debbugs.gnu.org Received: via spool by 52555-submit@debbugs.gnu.org id=B52555.164026190912428 (code B ref 52555); Thu, 23 Dec 2021 12:19:02 +0000 Received: (at 52555) by debbugs.gnu.org; 23 Dec 2021 12:18:29 +0000 Received: from localhost ([127.0.0.1]:60570 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1n0N3A-0003EI-Ic for submit@debbugs.gnu.org; Thu, 23 Dec 2021 07:18:29 -0500 Received: from mout02.posteo.de ([185.67.36.66]:52835) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1n0N38-0003E4-FE for 52555@debbugs.gnu.org; Thu, 23 Dec 2021 07:18:23 -0500 Received: from submission (posteo.de [89.146.220.130]) by mout02.posteo.de (Postfix) with ESMTPS id 2F987240105 for <52555@debbugs.gnu.org>; Thu, 23 Dec 2021 13:18:15 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=posteo.net; s=2017; t=1640261896; bh=JgqGKyEHLIdnmE6TeYENeP0wVNmtmDc4GKpcUJhHrYU=; h=From:To:Cc:Subject:Date:From; b=iW3bMzBy/BjgyPuHsO5+9LyOGNWnUQFwMfA/pKQ2N7sT2vDJlUho5P5oGdQg9U7cR 2Ezr5kuKqzBVCIoJlu30LWxEMyyQ1OfjizzCEz3NNk5WlTjq7HkqxajPzcaeQJwH+w 9XY7aXwKtfPqkEhYPq776u+6mnysrSdWphxAhC/ghdRNQgz9kT6a0a0bkFaRI1zLDi GiFDbSjV94XgkH54gVWFvomQOn+vccx+fur2ZEnOxTChnW6YYI19Y1k4svuCYlbOLW LPD1Is0csnx0JJkiZ8WexZlR2IFiMxvGLMwVVbs6T73h4WHmM72I8NHM8jXg4aWpOk +lQbMIAcjbkcA== Received: from customer (localhost [127.0.0.1]) by submission (posteo.de) with ESMTPSA id 4JKTjy5gg2z6tn0; Thu, 23 Dec 2021 13:18:14 +0100 (CET) References: <20211216161724.547-1-pukkamustard@posteo.net> <87h7b3gs64.fsf@gnu.org> From: pukkamustard Date: Thu, 23 Dec 2021 11:42:46 +0000 In-reply-to: <87h7b3gs64.fsf@gnu.org> Message-ID: <86bl17ms56.fsf@posteo.net> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+larch=yhetil.org@gnu.org Sender: "Guix-patches" X-Migadu-Flow: FLOW_IN X-Migadu-Country: US ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1640264342; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:resent-cc: resent-from:resent-sender:resent-message-id:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post:dkim-signature; bh=zU4yNoQ4afRH5P+IqWQwtji/6tFg0wM9H8IR63zUrJA=; b=JT2m5ucNPX5idiqF5BJW3umx9TjdGq82Qs1FfZiXoon8birgqt80s+k4q9gFm5YXdi1dhc nQyPYraLWYqZoSN+CpF2oaxrD3pJASsP7wi/gvc9fVU5WRkOOtlo5DhSQKOL5zpPKVJulm Ht1tntNfoP5tzEO4BloH/BnLguFEh4wTOcamnwBPOcQhxyDHFXsTvoD0fIsfGGiX44ogzj eozfWTqbxiFa3I3VGikHjE17M2mSQXWp/wwtRau8SGuAXn6UxXeqC50Z2fS5Q2vvuwFJ7S cmHBsZL3RPbjPrdNq+y4sv7T0RJqCdzJOGq1gNyiFOKXAnd5YpaBOfRnJtPjDg== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1640264342; a=rsa-sha256; cv=none; b=GN2O4/hMzLXmFM1D8TGielVy5FFhsRu23bdH7Sw7y8LHfyfvlagAgsJvbHkRsBYfho0iWD BAQ2IfAUjK3aIJTmRW5k6cjsDwWc29X+S+7O9ZqzW2gsUTFygG4/EZg44CeIujMB9h8XFk os/L6+LGuVjUoCDcOpj0TxiiglS/K2Ir4uzrCewixeKFNZsWcWGv3OCj3hKKhSNkap33UA 6gK9U5IScSLOKt3Ijm0q2/JDcr+lncSZese4Lz2QdQQItXydXzR96ArVqiiY4Z8/2iz5yY g1gHawR3uMP6dUQu0SSVIdQmVjnKWF59Q5ukvzjIk0xKCXKvGzjVk8ZCMXFDLA== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=posteo.net header.s=2017 header.b=iW3bMzBy; dmarc=fail reason="SPF not aligned (strict)" header.from=posteo.net (policy=none); spf=pass (aspmx1.migadu.com: domain of "guix-patches-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-patches-bounces+larch=yhetil.org@gnu.org" X-Migadu-Spam-Score: -2.95 Authentication-Results: aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=posteo.net header.s=2017 header.b=iW3bMzBy; dmarc=fail reason="SPF not aligned (strict)" header.from=posteo.net (policy=none); spf=pass (aspmx1.migadu.com: domain of "guix-patches-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-patches-bounces+larch=yhetil.org@gnu.org" X-Migadu-Queue-Id: 55C9D265BC X-Spam-Score: -2.95 X-Migadu-Scanner: scn0.migadu.com X-TUID: 4ucA8gjHgygO Hi Ludo, Thanks for your comments! Ludovic Court=C3=A8s writes: >> StorePath: /gnu/store/81bdcd5x4v50i28h98bfkvvkx9cky63w-hello-2.10 >> URL: nar/gzip/81bdcd5x4v50i28h98bfkvvkx9cky63w-hello-2.10 >> Compression: gzip >> FileSize: 67363 >> ERIS: urn:erisx2:BIBC2LUTIQH43S2KRIAV7TBXNUUVPZTMV6KFA2M7AL5V6FNE77VNUDD= VDAGJUEEAFATVO2QQT67SMOPTO3LGWCJFU7BZVCF5VXEQQW25BE >> URL: nar/zstd/81bdcd5x4v50i28h98bfkvvkx9cky63w-hello-2.10 >> Compression: zstd >> FileSize: 64917 >> ERIS: urn:erisx2:BIBO7KS7SAWHDNC43DVILOSQ3F3SRRHEV6YPLDCSZ7MMD6LZVCHQMEQ= 6FUBTJAPSNFF7XR5XPTP4OQ72OPABNEO7UYBUN42O46ARKHBTGM > > Do we really need one URN per compression method? Couldn=E2=80=99t we le= ave > compression (of individual chunks, possibly) as a =E2=80=9Cdetail=E2=80= =9D handled by > the encoding or the transport layer? > I agree that it would be nice to leave this to the encoding layer as that would allow certain optimizations (e.g. de-duplication). Unfortunately, we haven't figured out yet what the most suitable compression/format would be. Something like EROSFS seems good (as it aligns data to fixed block sizes) [1]. But this seems a bit "clunky" for just an archive format and there do not seem to be any libraries that we could use to neatly integrate. It seems possible to block-align a Tar archive, but that seems a bit hackey [2]. Other things to look into might be Tarlz [3] and ZPAQ [4]. To get started I suggest just using one of the compressions/formats already in Guix. zstd seems to be a reasonable choice (for the same reasons why it makes sense to use zstd with `--discover` [5]). Does that sound like a plan? [1] https://inqlab.net/git/guile-eris.git/tree/examples/dedup-fs/Readme.org [2] https://unix.stackexchange.com/questions/276908/make-tar-or-other-archi= ve-with-data-block-aligned-like-in-original-files-for/279384#279384 [3] http://lzip.nongnu.org/tarlz.html [4] http://mattmahoney.net/dc/zpaq.html [5] https://guix.gnu.org/en/blog/2021/getting-bytes-to-disk-more-quickly/ >> If the `--ipfs` is used for `guix publish` then the encoded blocks are a= lso >> uploaded to the IPFS daemon. The nar could then be retrieved from anywhe= re like >> this: >> >> (use-modules (eris) >> (eris blocks ipfs)) >> >> (eris-decode->bytevector >> "urn:erisx2:BIBC2LUTIQH43S2KRIAV7TBXNUUVPZTMV6KFA2M7AL5V6FNE77VNUDDVDAG= JUEEAFATVO2QQT67SMOPTO3LGWCJFU7BZVCF5VXEQQW25BE" >> eris-blocks-ipfs-ref) >> >> These patches do not yet retrieve content from IPFS (TODO). But in princ= iple, >> anybody connected to IPFS can get the nar with the ERIS URN. This could = be used >> to reduce load on substitute server as they would only need to publish t= he ERIS >> URN directly - substitutes could be delivered much more peer-to-peer. > > Nice. So adjusting =E2=80=98guix substitute=E2=80=99 should be relativel= y easy? Yes, relatively! :) I meant to send in a V2 that does this before going on holidays, but I'm afraid I won't make it. V2 will come in early January! >> Other transports that I have been looking in to and am pretty sure will = work >> include: HTTP (with RFC 2169 [3]), GNUNet, OpenDHT. This is, imho, the >> advantage of ERIS over IPFS directly or GNUNet directly. The encoding and >> identifiers (URN) are abstracted away from specific transports (and also >> applications). ERIS is almost exactly the same encoding as used in GNUNet >> (ECRS). > > As a first step, =E2=80=98guix publish=E2=80=99 could implement RFC 2169,= too. > > I gather implementing the HTTP and IPFS backends in =E2=80=98guix substit= ute=E2=80=99 > should be relatively easy, right? Yes, those seem to be the two easiest backends to implement. >> A tricky things is figuring out how to multiplex all these different >> transports and storages... > > Yes. We don=E2=80=99t know yet what performance and data availability wi= ll be > like on IPFS, for instance, so it=E2=80=99s important for users to be abl= e to > set priorities. It=E2=80=99s also important to gracefully fall back to d= irect > HTTP downloads when fancier p2p methods fail, regardless of how they > fail. Agree. Thanks, -pukkamustard