From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([209.51.188.92]:50347) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ggXyW-00052e-AH for guix-patches@gnu.org; Mon, 07 Jan 2019 11:42:05 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1ggXyU-0003Pl-W8 for guix-patches@gnu.org; Mon, 07 Jan 2019 11:42:04 -0500 Received: from debbugs.gnu.org ([209.51.188.43]:49618) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1ggXyU-0003Ow-L4 for guix-patches@gnu.org; Mon, 07 Jan 2019 11:42:02 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1ggXyU-0000XB-E8 for guix-patches@gnu.org; Mon, 07 Jan 2019 11:42:02 -0500 Subject: [bug#33899] [PATCH 0/5] Distributing substitutes over IPFS Resent-Message-ID: Received: from eggs.gnu.org ([209.51.188.92]:37113) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ggW8K-0007s6-TI for guix-patches@gnu.org; Mon, 07 Jan 2019 09:44:05 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1ggW8J-0005RN-PX for guix-patches@gnu.org; Mon, 07 Jan 2019 09:44:04 -0500 Received: from mail1.protonmail.ch ([185.70.40.18]:31631) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1ggW8J-0005QN-8L for guix-patches@gnu.org; Mon, 07 Jan 2019 09:44:03 -0500 Date: Mon, 07 Jan 2019 14:43:56 +0000 From: Hector Sanjuan Message-ID: In-Reply-To: <20181228231205.8068-1-ludo@gnu.org> References: <20181228231205.8068-1-ludo@gnu.org> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: Hector Sanjuan Errors-To: guix-patches-bounces+kyle=kyleam.com@gnu.org Sender: "Guix-patches" To: "33899@debbugs.gnu.org" <33899@debbugs.gnu.org> Cc: mail@ambrevar.xyz, go-ipfs-wg@ipfs.io =E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90 Original Me= ssage =E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90 On Saturday, December 29, 2018 12:12 AM, Ludovic Court=C3=A8s wrote: > Hello Guix! > > Here is a first draft adding support to distribute and retrieve substitut= es > over IPFS. This builds on discussions at the R-B Summit with H=C3=A9ctor = Sanjuan > of IPFS, lewo of Nix, Pierre Neidhardt, and also on the work Florian > Paul Schmidt posted on guix-devel last month. > > The IPFS daemon exposes an HTTP API and the (guix ipfs) module provides > bindings to a subset of that API. This module also implements a custom > =E2=80=9Cdirectory=E2=80=9D format to store directory trees in IPFS (IPFS= already provides > =E2=80=9CUnixFS=E2=80=9D and =E2=80=9Ctar=E2=80=9D but they store too man= y or too few file attributes.) > > =E2=80=98guix publish=E2=80=99 and =E2=80=98guix substitute=E2=80=99 use = (guix ipfs) to > store and retrieve store items. Complete directory trees are stored in > IPFS =E2=80=9Cas is=E2=80=9D, rather than as compressed archives (nars). = This allows for > deduplication in IPFS. =E2=80=98guix publish=E2=80=99 adds a new = =E2=80=9CIPFS=E2=80=9D field in > narinfos and =E2=80=98guix substitute=E2=80=99 can then query those objec= ts over IPFS. > So the idea is that you still get narinfos over HTTP(S), and then you > have the option of downloading substitutes over IPFS. > > I=E2=80=99ve pushed these patches in =E2=80=98wip-ipfs-substitutes= =E2=80=99. This is rough on the > edges and probably buggy, but the adventurous among us might want to give > it a spin. :-) > > Thanks, > Ludo=E2=80=99. Hey! Happy new year! This is great news. I'm very glad to see this. I haven't tried this yet but looking at the code there are a couple of things to point out. 1) The doc strings usually refer to the IPFS HTTP API as GATEWAY. go-ipfs has a read/write API (on :5001) and a read-only API that we call "gateway" and which runs on :8080. The gateway, apart from handling most of the read-only methods from the HTTP API, also handles paths like "/ipfs/" or "/ipns/" gracefully, and returns an autogenerated webpage for directory-type CIDs. The gateway does not allow to "publish". Therefore I t= hink the doc strings should say "IPFS daemon API" rather than "GATEWAY". 2) I'm not proficient enough in schema to grasp the details of the "directory" format. If I understand it right, you keep a separate manifest object listing the directory structure, the contents and the executable bit for each. Thus, when adding a store item you add all the files separately a= nd this manifest. And when retrieving a store item you fetch the manifest and reconstruct the tree by fetching the contents in it (and applying the executable flag). Is this correct? This works, but it can be improved: You can add all the files/folders in a single request. If I'm reading it right, now each files is added separately (and gets pinned separately). It would probably make sense to add it all in a single request= , letting IPFS to store the directory structure as "unixfs". You can additionally add the sexp file with the dir-structure and executable flags as an extra file to the root folder. This would allow to fetch the whole th= ing with a single request too /api/v0/get?arg=3D. And to pin a single has= h recursively (and not each separately). After getting the whole thing, you will need to chmod +x things accordingly. It will probably need some trial an error to get the multi-part right to upload all in a single request. The Go code HTTP Clients doing this can be found at: https://github.com/ipfs/go-ipfs-files/blob/master/multifilereader.go#L96 As you see, a directory part in the multipart will have the content-type He= ader set to "application/x-directory". The best way to see how "abspath" etc is = set is probably to sniff an `ipfs add -r ` operation (localhost:500= 1). Once UnixFSv2 lands, you will be in a position to just drop the sexp file altogether. Let me know if you have any doubts, I'll make my best to answer them. In th= e meantime I'll try to get more familiar with Guix. Cheers, Hector PS. There is a place where it says "ifps" instead of "ipfs". A very common = typo.