From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:470:142:3::10]:51309) by lists.gnu.org with esmtp (Exim 4.86_2) (envelope-from ) id 1hmn3H-0005Q4-Fw for guix-patches@gnu.org; Sun, 14 Jul 2019 18:33:04 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1hmn3G-0004MQ-B4 for guix-patches@gnu.org; Sun, 14 Jul 2019 18:33:03 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:37353) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1hmn3G-0004Lt-7R for guix-patches@gnu.org; Sun, 14 Jul 2019 18:33:02 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1hmn3G-0008HT-3A for guix-patches@gnu.org; Sun, 14 Jul 2019 18:33:02 -0400 Subject: [bug#33899] [PATCH 0/5] Distributing substitutes over IPFS Resent-Message-ID: Date: Sun, 14 Jul 2019 22:31:43 +0000 From: Hector Sanjuan Message-ID: In-Reply-To: <87ftnbf1rt.fsf@gnu.org> References: <20181228231205.8068-1-ludo@gnu.org> <87r2dfv0nj.fsf@gnu.org> <8736pqthqm.fsf@gnu.org> <87zhlxe8t9.fsf@ambrevar.xyz> <87ftnbf1rt.fsf@gnu.org> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: Hector Sanjuan Errors-To: guix-patches-bounces+kyle=kyleam.com@gnu.org Sender: "Guix-patches" To: Ludovic =?UTF-8?Q?Court=C3=A8s?= Cc: Antoine Eiche , "go-ipfs-wg\\@ipfs.io" , Pierre Neidhardt , "33899\\@debbugs.gnu.org" <33899@debbugs.gnu.org> Hey! Thanks for reviving this discussion! =E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90 Original Me= ssage =E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90 On Friday, July 12, 2019 10:15 PM, Ludovic Court=C3=A8s wrot= e: > Hello! > > Pierre Neidhardt mail@ambrevar.xyz skribis: > > > A little update/recap after many months! :) > > Thank you, and apologies for the delay! > > > - Extra metadata: IPFS stores files on UnixFSv1 which does not > > include the executable bit. > > - Right now we store a s-exp manifest with a list of files and a > > list of executable bits. But maybe we don't have to roll out ou= r own. > > > > - UnixFSv1 has some metadata field, but H=C3=A9ctor and Alex did = not > > recommend using it (not sure why though). > > > > - We could use UnixFSv2 but it's not released yet and it's unclea= r when > > it's going to be released. So we can't really count on it right= now. > > > > UnixFSv1 is not an option because it lacks the executable bit; UnixFSv2 > would be appropriate, though it stores timestamps that we don=E2=80=99t n= eed > (not necessarily a problem). > > > - Flat storage vs. tree storage: Right now we are storing the files > > separately, but this has some shortcomings, namely we need multiple > > "get" requests instead of just one, and that IPFS does > > not "know" that those files are related. (We lose the web view of > > the tree, etc.) Storing them as tree could be better. > > I don't understand if that would work with the "IPLD manifest" > > suggested above. H=C3=A9ctor? > > > > I don=E2=80=99t consider the web view a strong argument :-) since we coul= d write > tools to deal with whatever format we use. > > Regarding multiple GET requests: we could pipeline them, and it seems > more like an implementation detail to me. The real question is whether > making separate GET requests prevents some optimization in IPFS. > > > - Pinning: Pinning all files separately incurs an overhead. It's > > enough to just pin the IPLD object since it propagates recursively. > > When adding a tree, then it's no problem since pinning is only done= once. > > > > Where=E2=80=99s the overhead exactly? There are reasons why we are proposing to create a single DAG with an IPLD object at the root. Pinning has a big overhead because it involves locking, reading, parsing, and writing an internal pin-DAG. This is specially relevant when the pinset is very large. Doing multiple GET requests also has overhead, like being unable to use a single bitswap session (which, when downloading something new means a big overhead since every request will have to find providers). And it's not just the web view, it's the ability to walk/traverse all the object related to a given root natively, which allows also to compare multiple trees and to be more efficient for some things ("pin update" for example). Your original idea is to create a manifest with references to different parts. I'm just asking you to create that manifest in a format where those references are understood not only by you, the file creator, but by IPFS and any tool that can read IPLD, by making this a IPLD object (which is just a json). The process of adding "something" to ipfs is as follows. ---- 1. Add to IPFS: multipart upload equivalent to "ipfs add -r": ~/ipfstest $ ipfs add -r -q . QmXVgwHR2c8KiPPxaoZAj4M4oNGW1qjZSsxMNE8sLWZWTP 2. Add manifest as IPLD object. dag/put a json file like: cat <