From: "Ludovic Courtès" <ludo@gnu.org>
To: Hector Sanjuan <code@hector.link>
Cc: "go-ipfs-wg@ipfs.io" <go-ipfs-wg@ipfs.io>,
Pierre Neidhardt <mail@ambrevar.xyz>,
"33899@debbugs.gnu.org" <33899@debbugs.gnu.org>
Subject: [bug#33899] [PATCH 0/5] Distributing substitutes over IPFS
Date: Mon, 14 Jan 2019 14:17:36 +0100 [thread overview]
Message-ID: <87r2dfv0nj.fsf@gnu.org> (raw)
In-Reply-To: <NIQ7NwCAv476krhArm683GoWozb7fXZrcRzXorY2s-X_QAZ71GhHzrFj7788bRBP3fpklDWfxy-we7928SLdmd-iairDLG_sNDRjbJwxYyo=@hector.link> (Hector Sanjuan's message of "Mon, 07 Jan 2019 14:43:56 +0000")
Hi Hector,
Happy new year to you too! :-)
Hector Sanjuan <code@hector.link> skribis:
> 1) The doc strings usually refer to the IPFS HTTP API as GATEWAY. go-ipfs
> has a read/write API (on :5001) and a read-only API that we call "gateway"
> and which runs on :8080. The gateway, apart from handling most of the
> read-only methods from the HTTP API, also handles paths like "/ipfs/<cid>"
> or "/ipns/<name>" gracefully, and returns an autogenerated webpage for
> directory-type CIDs. The gateway does not allow to "publish". Therefore I think
> the doc strings should say "IPFS daemon API" rather than "GATEWAY".
Indeed, I’ll change that.
> 2) I'm not proficient enough in schema to grasp the details of the
> "directory" format. If I understand it right, you keep a separate manifest
> object listing the directory structure, the contents and the executable bit
> for each. Thus, when adding a store item you add all the files separately and
> this manifest. And when retrieving a store item you fetch the manifest and
> reconstruct the tree by fetching the contents in it (and applying the
> executable flag). Is this correct? This works, but it can be improved:
That’s correct.
> You can add all the files/folders in a single request. If I'm
> reading it right, now each files is added separately (and gets pinned
> separately). It would probably make sense to add it all in a single request,
> letting IPFS to store the directory structure as "unixfs". You can
> additionally add the sexp file with the dir-structure and executable flags
> as an extra file to the root folder. This would allow to fetch the whole thing
> with a single request too /api/v0/get?arg=<hash>. And to pin a single hash
> recursively (and not each separately). After getting the whole thing, you
> will need to chmod +x things accordingly.
Yes, I’m well aware of “unixfs”. The problems, as I see it, is that it
stores “too much” in a way (we don’t need to store the mtimes or
permissions; we could ignore them upon reconstruction though), and “not
enough” in another way (the executable bit is lost, IIUC.)
> It will probably need some trial an error to get the multi-part right
> to upload all in a single request. The Go code HTTP Clients doing
> this can be found at:
>
> https://github.com/ipfs/go-ipfs-files/blob/master/multifilereader.go#L96
>
> As you see, a directory part in the multipart will have the content-type Header
> set to "application/x-directory". The best way to see how "abspath" etc is set
> is probably to sniff an `ipfs add -r <testfolder>` operation (localhost:5001).
>
> Once UnixFSv2 lands, you will be in a position to just drop the sexp file
> altogether.
Yes, that makes sense. In the meantime, I guess we have to keep using
our own format.
What are the performance implications of adding and retrieving files one
by one like I did? I understand we’re doing N HTTP requests to the
local IPFS daemon where “ipfs add -r” makes a single request, but this
alone can’t be much of a problem since communication is happening
locally. Does pinning each file separately somehow incur additional
overhead?
Thanks for your feedback!
Ludo’.
next prev parent reply other threads:[~2019-01-14 13:18 UTC|newest]
Thread overview: 23+ messages / expand[flat|nested] mbox.gz Atom feed top
2018-12-28 23:12 [bug#33899] [PATCH 0/5] Distributing substitutes over IPFS Ludovic Courtès
2018-12-28 23:15 ` [bug#33899] [PATCH 1/5] Add (guix json) Ludovic Courtès
2018-12-28 23:15 ` [bug#33899] [PATCH 2/5] tests: 'file=?' now recurses on directories Ludovic Courtès
2018-12-28 23:15 ` [bug#33899] [PATCH 3/5] Add (guix ipfs) Ludovic Courtès
2018-12-28 23:15 ` [bug#33899] [PATCH 4/5] publish: Add IPFS support Ludovic Courtès
2018-12-28 23:15 ` [bug#33899] [PATCH 5/5] DRAFT substitute: " Ludovic Courtès
2019-01-07 14:43 ` [bug#33899] [PATCH 0/5] Distributing substitutes over IPFS Hector Sanjuan
2019-01-14 13:17 ` Ludovic Courtès [this message]
2019-01-18 9:08 ` Hector Sanjuan
2019-01-18 9:52 ` Ludovic Courtès
2019-01-18 11:26 ` Hector Sanjuan
2019-07-01 21:36 ` Pierre Neidhardt
2019-07-06 8:44 ` Pierre Neidhardt
2019-07-12 20:02 ` Molly Mackinlay
2019-07-15 9:20 ` Alex Potsides
2019-07-12 20:15 ` Ludovic Courtès
2019-07-14 22:31 ` Hector Sanjuan
2019-07-15 9:24 ` Ludovic Courtès
2019-07-15 10:10 ` Pierre Neidhardt
2019-07-15 10:21 ` Hector Sanjuan
2019-05-13 18:51 ` Alex Griffin
2020-12-29 9:59 ` [bug#33899] Ludo's patch rebased on master Maxime Devos
2021-06-06 17:54 ` [bug#33899] [PATCH 0/5] Distributing substitutes over IPFS Tony Olagbaiye
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=87r2dfv0nj.fsf@gnu.org \
--to=ludo@gnu.org \
--cc=33899@debbugs.gnu.org \
--cc=code@hector.link \
--cc=go-ipfs-wg@ipfs.io \
--cc=mail@ambrevar.xyz \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this external index
https://git.savannah.gnu.org/cgit/guix.git
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.