From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:470:142:3::10]:58374) by lists.gnu.org with esmtp (Exim 4.86_2) (envelope-from ) id 1hm1xb-0003rC-Fx for guix-patches@gnu.org; Fri, 12 Jul 2019 16:16:04 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1hm1xa-0006J7-GM for guix-patches@gnu.org; Fri, 12 Jul 2019 16:16:03 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:60187) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1hm1xa-0006Iw-Cw for guix-patches@gnu.org; Fri, 12 Jul 2019 16:16:02 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1hm1xa-00042S-6g for guix-patches@gnu.org; Fri, 12 Jul 2019 16:16:02 -0400 Subject: [bug#33899] [PATCH 0/5] Distributing substitutes over IPFS Resent-Message-ID: From: Ludovic =?UTF-8?Q?Court=C3=A8s?= References: <20181228231205.8068-1-ludo@gnu.org> <87r2dfv0nj.fsf@gnu.org> <8736pqthqm.fsf@gnu.org> <87zhlxe8t9.fsf@ambrevar.xyz> Date: Fri, 12 Jul 2019 22:15:02 +0200 In-Reply-To: <87zhlxe8t9.fsf@ambrevar.xyz> (Pierre Neidhardt's message of "Mon, 01 Jul 2019 23:36:34 +0200") Message-ID: <87ftnbf1rt.fsf@gnu.org> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+kyle=kyleam.com@gnu.org Sender: "Guix-patches" To: Pierre Neidhardt Cc: Hector Sanjuan , Antoine Eiche , "go-ipfs-wg@ipfs.io" , "33899@debbugs.gnu.org" <33899@debbugs.gnu.org> Hello! Pierre Neidhardt skribis: > A little update/recap after many months! :) Thank you, and apologies for the delay! > - Extra metadata: IPFS stores files on UnixFSv1 which does not > include the executable bit. > > - Right now we store a s-exp manifest with a list of files and a > list of executable bits. But maybe we don't have to roll out our o= wn. > > - UnixFSv1 has some metadata field, but H=C3=A9ctor and Alex did not > recommend using it (not sure why though). > > - We could use UnixFSv2 but it's not released yet and it's unclear wh= en > it's going to be released. So we can't really count on it right no= w. UnixFSv1 is not an option because it lacks the executable bit; UnixFSv2 would be appropriate, though it stores timestamps that we don=E2=80=99t need (not necessarily a problem). > - Flat storage vs. tree storage: Right now we are storing the files > separately, but this has some shortcomings, namely we need multiple > "get" requests instead of just one, and that IPFS does > not "know" that those files are related. (We lose the web view of > the tree, etc.) Storing them as tree could be better. > I don't understand if that would work with the "IPLD manifest" > suggested above. H=C3=A9ctor? I don=E2=80=99t consider the web view a strong argument :-) since we could = write tools to deal with whatever format we use. Regarding multiple GET requests: we could pipeline them, and it seems more like an implementation detail to me. The real question is whether making separate GET requests prevents some optimization in IPFS. > - Pinning: Pinning all files separately incurs an overhead. It's > enough to just pin the IPLD object since it propagates recursively. > When adding a tree, then it's no problem since pinning is only done o= nce. Where=E2=80=99s the overhead exactly? > - IPFS endpoint calls: instead of adding each file individually, it's > possible to add them all in one go. Can we add all files at once > while using a flat storage? (I.e. not adding them all under a common > root folder.) Again, is the concern that we=E2=80=99re making one GET and thus one round = trip per file, or is there some additional cost under the hood? Thanks for the summary and explanations! Ludo=E2=80=99.