From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp12.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id EL6hHJZn+mF2iwAAgWs5BA (envelope-from ) for ; Wed, 02 Feb 2022 12:14:30 +0100 Received: from aspmx1.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp12.migadu.com with LMTPS id YB0eGZZn+mGAJwAAauVa8A (envelope-from ) for ; Wed, 02 Feb 2022 12:14:30 +0100 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id F3319373B6 for ; Wed, 2 Feb 2022 12:14:29 +0100 (CET) Received: from localhost ([::1]:45048 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nFDan-0008UB-27 for larch@yhetil.org; Wed, 02 Feb 2022 06:14:29 -0500 Received: from eggs.gnu.org ([209.51.188.92]:33044) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nFDXX-00083p-2b for guix-patches@gnu.org; Wed, 02 Feb 2022 06:11:07 -0500 Received: from debbugs.gnu.org ([209.51.188.43]:57727) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1nFDXT-00066E-41 for guix-patches@gnu.org; Wed, 02 Feb 2022 06:11:05 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1nFDXT-0001nt-08 for guix-patches@gnu.org; Wed, 02 Feb 2022 06:11:03 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#52555] [RFC PATCH v2 0/5] Decentralized substitute distribution with ERIS Resent-From: pukkamustard Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Wed, 02 Feb 2022 11:11:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 52555 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Maxime Devos Cc: ~pukkamustard/eris@lists.sr.ht, 52555@debbugs.gnu.org Received: via spool by 52555-submit@debbugs.gnu.org id=B52555.16438002366882 (code B ref 52555); Wed, 02 Feb 2022 11:11:02 +0000 Received: (at 52555) by debbugs.gnu.org; 2 Feb 2022 11:10:36 +0000 Received: from localhost ([127.0.0.1]:51622 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nFDX1-0001mw-VY for submit@debbugs.gnu.org; Wed, 02 Feb 2022 06:10:36 -0500 Received: from mout01.posteo.de ([185.67.36.65]:55259) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nFDX0-0001mf-DN for 52555@debbugs.gnu.org; Wed, 02 Feb 2022 06:10:35 -0500 Received: from submission (posteo.de [89.146.220.130]) by mout01.posteo.de (Postfix) with ESMTPS id 70918240026 for <52555@debbugs.gnu.org>; Wed, 2 Feb 2022 12:10:28 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=posteo.net; s=2017; t=1643800228; bh=fhRCmo5kXYeo7IZGuTKrgKCHCAviFP71C3PQ+lrMfbE=; h=From:To:Cc:Subject:Date:From; b=jMoMRoCb96oBZPOANWNyemN3BQqa4Kf3dAp62rRiqFixY72XJnqcsBxmoDt0QNBLh NPi2fjpO0bEYCVIjJqkjRlWJyKU1WOStnXd3gslJcoYrUuvTSgJWaOzL5cGRS3LbfQ YHyw4GHDnOwvUS+m3WKcVh1T3vPlMRy9Gppj/jQfD8oDALJahA4t84KoBOZLlavWXu ykW4Y4gSB0M/wMDc/hdnzcxAh04Pp5p9MF7ig9CfKKSwdky68OOTWOOoMxR+/f2whq yKdsO8yA8cPDPpm0XuJz5bVKgAJtoLlkC5GnVc8hQhePFcXsb4pedQfXxyJRXU+dzu YL4RRGDXPMw6Q== Received: from customer (localhost [127.0.0.1]) by submission (posteo.de) with ESMTPSA id 4JpfGp68ZNz9rxh; Wed, 2 Feb 2022 12:10:26 +0100 (CET) References: <20211216161724.547-1-pukkamustard@posteo.net> <20220125192201.7582-1-pukkamustard@posteo.net> From: pukkamustard Date: Wed, 02 Feb 2022 10:51:10 +0000 In-reply-to: Message-ID: <86fsp1h6ce.fsf@posteo.net> MIME-Version: 1.0 Content-Type: text/plain X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+larch=yhetil.org@gnu.org Sender: "Guix-patches" X-Migadu-Flow: FLOW_IN X-Migadu-Country: US ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1643800470; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:resent-cc:resent-from:resent-sender: resent-message-id:in-reply-to:in-reply-to:references:references: list-id:list-help:list-unsubscribe:list-subscribe:list-post: dkim-signature; bh=aBW8Jy68LxwbbPLNgWFV4w+tp2JCG/DYNKujmiks8HI=; b=tfqAwZvskvLJJ7VixqoDriNcZdSpkiqGyOmRNSTsDrVRXhi4OjTR/9pbA/8V/A7A+TMUw1 6i9CU+gmQxbVU11paRifl2FGACIweqmAdQ+vKapsOPP9pJEbWvnxMviAxa94yvg1/2Azm9 +HTZtSy4+2bzPPgiOXoCi5NNrzz0/T1+3h6zFweh0VlO9kHRkEd23yPlLZl2WwpVNAny11 8IiaR4RGuH/9r5RwJ7UOChwUwf6rDo69Cqc0QtmF1/DbXnU6Ex9Vyk+cuAwWwHNKX9R51F AQaTdQ67R/jG3i1NhJig6oMG+qUPGBJ8/JmdxBIA2mEqskb90+rEvKDpvcScdQ== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1643800470; a=rsa-sha256; cv=none; b=qgzw6wrbnJDNRRvVTYbIbVsv5QnPYr2dA61F5ux5TkWI3rTbTJ9muLvCjkO3DfRPoPaV5J d0cpHNrGkAssNlKaxTZfv0VWCY1auOmUK7csPhbI67xk9/2F0n+QabOByqAFzIiY0+xiG+ xujXP+AmoZNM2mks+wNCWQOETnqvyeDidWgpXJDn2Duf2NLwF8WiABEND2qyNRS/mdxzud pGL8rQlLPkhRQKIbKMyQzI8V4sGRWxbn5zevB4RBmyCAinDx4saQBOhr337FAZba2ipWK2 sy+MakJeoqbtY1/LXJ6jpkTd9rgelJy+XTXCLnfefJ5FTI0nVQsoULxDGtL1cw== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=posteo.net header.s=2017 header.b=jMoMRoCb; dmarc=fail reason="SPF not aligned (strict)" header.from=posteo.net (policy=none); spf=pass (aspmx1.migadu.com: domain of "guix-patches-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-patches-bounces+larch=yhetil.org@gnu.org" X-Migadu-Spam-Score: -2.23 Authentication-Results: aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=posteo.net header.s=2017 header.b=jMoMRoCb; dmarc=fail reason="SPF not aligned (strict)" header.from=posteo.net (policy=none); spf=pass (aspmx1.migadu.com: domain of "guix-patches-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-patches-bounces+larch=yhetil.org@gnu.org" X-Migadu-Queue-Id: F3319373B6 X-Spam-Score: -2.23 X-Migadu-Scanner: scn1.migadu.com X-TUID: J22pJnlbBxef Maxime Devos writes: > [[PGP Signed Part:Undecided]] > pukkamustard schreef op di 25-01-2022 om 19:21 [+0000]: >> I have only tested this for fairly small packages (up to a few MB). >> >> One issue with IPFS might be that we have to create a new HTTP connection to >> the IPFS daemon for every single block (32KiB). The IPFS daemon does not seem >> to support HTTP connection re-use > > According to , the IPFS > daemon supports connection reuse according to some people and doesn't > according to other people. Hm, from what I understand connection re-use is something introduced in HTTP/2 and go-ipfs does not do HTTP/2 (https://github.com/ipfs/go-ipfs/issues/5974). >> and neither does the Guile (web client). > > Guix supports connection reuse, see 'call-with-cached-connection' > in (guix scripts substitute). Ah ok. Cool! >> I fear this might become a performance issue. > > IIUC, the performance problem primarily lies in the round-tripping > between the client and the server. If the client and the server are on > the same machine, then this round trip time is presumably small > compared to, say, localhost contacting ci.guix.gnu.org. > > Still, connection reuse would be nice. Remains to be seen if this is a problem. It is considerably more pronounced than with regular usage of IPFS as we make a HTTP request to IPFS for every 32KiB block instead of for an entire file (what most people do when using the IPFS daemon). >> It seems possible to use IPFS more directly by exposing the Go code as a >> C library and then using that with the Guile FFI [1]. This is however a bit >> complicated and adds a lot of dependencies. In particular, this should not become >> a dependency of Guix itself. The performance of IPFS itself also needs to be >> evaluated, maybe the IPFS HTTP API will not be the bottle-neck. > > Security-wise, libipfs doesn't seem great: libipfs starts the IPFS > daemon inside the process and guix/scripts/substitute.scm is run > as root. I agree. >> As mentioned in previous mail a simple HTTP transport for blocks would be a >> good fallback. This would allow users to get missing blocks (things that >> somehow got dropped from IPFS) directly from a substitute server. [...] > > Seems a good idea to me -- DHTs can be unreliable. I presume this will > be implemented with some kind of timeout: if no block is received > within N seconds, fallback to HTTP? Yes, exactly. > Also, don't forget to insert this missing block back into > IPFS/GNUnet/BitTorrent/..., otherwise less and less blocks will be > available until nothing is available anymore. This might be a bit of a burden for users. As you mention the size of such a database might become considerable. >> In any case, it would be necessary for the substitute server to store encoded >> blocks of the NAR. For this I think it makes sense to use a small database. We >> have bindings to use ERIS with GDBM [2]. It might also make sense to use >> SQLite, especially if there are other use-cases for such a database. > > Wouldn't this be a huge database? IIRC, according to logs.guix.gnu.org > the size of the nars of the substitute servers are somewhere in the > 200G-2T range or something like that. > > To reduce the size of the database, perhaps you could let the database > be a mapping from block ids to the name of the nar + the position in > the nar, and encode the block on-demand? Yes! I've also been thinking of this - a "in-file" block store. I think this makes a lot of sense for Guix but also other things (e.g. sharing your music collection). Another problem with IPFS/GNUNet is that they have their own storage. So even if are clever about storing blocks in Guix, IPFS and GNUNet will have their own copy of the blocks on disk. I think it would be much nicer if DHTs/transport layers don't do block storage but are provided with a callback from where they can get stored blocks. I believe this is what OpenDHT does (https://github.com/savoirfairelinux/opendht/wiki/API-Overview). I think we should propose such a change to the GNUNet R5N specification (https://lsd.gnunet.org/lsd0004/). > The database doesn't seem necessary, the substitute server could have > some end-point > > /publish-this-nar-again-into-IPFS/name-of-the-nar > > which, when contacted, inserts the nar again into IPFS. Then when a > block was unavailable, the client contacts this end-point and retries. But for a HTTP block endpoint we would still need such a database/block storage. I think it is important that we do not rely on IPFS for block storage. The decentralized block distribution should work even if the IPFS daemon is not available. -pukkamustard