From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([209.51.188.92]:37749) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gj27w-0003Cz-Em for guix-patches@gnu.org; Mon, 14 Jan 2019 08:18:05 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gj27v-0003bL-9H for guix-patches@gnu.org; Mon, 14 Jan 2019 08:18:04 -0500 Received: from debbugs.gnu.org ([209.51.188.43]:59088) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1gj27v-0003b6-55 for guix-patches@gnu.org; Mon, 14 Jan 2019 08:18:03 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1gj27u-0003hY-GU for guix-patches@gnu.org; Mon, 14 Jan 2019 08:18:02 -0500 Subject: [bug#33899] [PATCH 0/5] Distributing substitutes over IPFS Resent-Message-ID: From: Ludovic =?UTF-8?Q?Court=C3=A8s?= References: <20181228231205.8068-1-ludo@gnu.org> Date: Mon, 14 Jan 2019 14:17:36 +0100 In-Reply-To: (Hector Sanjuan's message of "Mon, 07 Jan 2019 14:43:56 +0000") Message-ID: <87r2dfv0nj.fsf@gnu.org> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+kyle=kyleam.com@gnu.org Sender: "Guix-patches" To: Hector Sanjuan Cc: "go-ipfs-wg@ipfs.io" , Pierre Neidhardt , "33899@debbugs.gnu.org" <33899@debbugs.gnu.org> Hi Hector, Happy new year to you too! :-) Hector Sanjuan skribis: > 1) The doc strings usually refer to the IPFS HTTP API as GATEWAY. go-ipfs > has a read/write API (on :5001) and a read-only API that we call "gateway" > and which runs on :8080. The gateway, apart from handling most of the > read-only methods from the HTTP API, also handles paths like "/ipfs/" > or "/ipns/" gracefully, and returns an autogenerated webpage for > directory-type CIDs. The gateway does not allow to "publish". Therefore I= think > the doc strings should say "IPFS daemon API" rather than "GATEWAY". Indeed, I=E2=80=99ll change that. > 2) I'm not proficient enough in schema to grasp the details of the > "directory" format. If I understand it right, you keep a separate manifest > object listing the directory structure, the contents and the executable b= it > for each. Thus, when adding a store item you add all the files separately= and > this manifest. And when retrieving a store item you fetch the manifest and > reconstruct the tree by fetching the contents in it (and applying the > executable flag). Is this correct? This works, but it can be improved: That=E2=80=99s correct. > You can add all the files/folders in a single request. If I'm > reading it right, now each files is added separately (and gets pinned > separately). It would probably make sense to add it all in a single reque= st, > letting IPFS to store the directory structure as "unixfs". You can > additionally add the sexp file with the dir-structure and executable flags > as an extra file to the root folder. This would allow to fetch the whole = thing > with a single request too /api/v0/get?arg=3D. And to pin a single h= ash > recursively (and not each separately). After getting the whole thing, you > will need to chmod +x things accordingly. Yes, I=E2=80=99m well aware of =E2=80=9Cunixfs=E2=80=9D. The problems, as = I see it, is that it stores =E2=80=9Ctoo much=E2=80=9D in a way (we don=E2=80=99t need to store = the mtimes or permissions; we could ignore them upon reconstruction though), and =E2=80= =9Cnot enough=E2=80=9D in another way (the executable bit is lost, IIUC.) > It will probably need some trial an error to get the multi-part right > to upload all in a single request. The Go code HTTP Clients doing > this can be found at: > > https://github.com/ipfs/go-ipfs-files/blob/master/multifilereader.go#L96 > > As you see, a directory part in the multipart will have the content-type = Header > set to "application/x-directory". The best way to see how "abspath" etc i= s set > is probably to sniff an `ipfs add -r ` operation (localhost:5= 001). > > Once UnixFSv2 lands, you will be in a position to just drop the sexp file > altogether. Yes, that makes sense. In the meantime, I guess we have to keep using our own format. What are the performance implications of adding and retrieving files one by one like I did? I understand we=E2=80=99re doing N HTTP requests to the local IPFS daemon where =E2=80=9Cipfs add -r=E2=80=9D makes a single reques= t, but this alone can=E2=80=99t be much of a problem since communication is happening locally. Does pinning each file separately somehow incur additional overhead? Thanks for your feedback! Ludo=E2=80=99.