[bug#33899] [PATCH 0/5] Distributing substitutes over IPFS

["Ludovic Courtès" <ludo@gnu.org>]

Hello!

Pierre Neidhardt <mail@ambrevar.xyz> skribis:

> A little update/recap after many months! :)

Thank you, and apologies for the delay!

>   - Extra metadata:  IPFS stores files on UnixFSv1 which does not
>     include the executable bit.
>
>     - Right now we store a s-exp manifest with a list of files and a
>       list of executable bits.  But maybe we don't have to roll out our own.
>
>     - UnixFSv1 has some metadata field, but Héctor and Alex did not
>       recommend using it (not sure why though).
>
>     - We could use UnixFSv2 but it's not released yet and it's unclear when
>       it's going to be released.  So we can't really count on it right now.

UnixFSv1 is not an option because it lacks the executable bit; UnixFSv2
would be appropriate, though it stores timestamps that we don’t need
(not necessarily a problem).

>   - Flat storage vs. tree storage: Right now we are storing the files
>     separately, but this has some shortcomings, namely we need multiple
>     "get" requests instead of just one, and that IPFS does
>     not "know" that those files are related.  (We lose the web view of
>     the tree, etc.)  Storing them as tree could be better.
>     I don't understand if that would work with the "IPLD manifest"
>     suggested above.  Héctor?

I don’t consider the web view a strong argument :-) since we could write
tools to deal with whatever format we use.

Regarding multiple GET requests: we could pipeline them, and it seems
more like an implementation detail to me.  The real question is whether
making separate GET requests prevents some optimization in IPFS.

>   - Pinning: Pinning all files separately incurs an overhead.  It's
>     enough to just pin the IPLD object since it propagates recursively.
>     When adding a tree, then it's no problem since pinning is only done once.

Where’s the overhead exactly?

>   - IPFS endpoint calls: instead of adding each file individually, it's
>     possible to add them all in one go.  Can we add all files at once
>     while using a flat storage? (I.e. not adding them all under a common
>     root folder.)

Again, is the concern that we’re making one GET and thus one round trip
per file, or is there some additional cost under the hood?

Thanks for the summary and explanations!

Ludo’.