From: Pierre Neidhardt <mail@ambrevar.xyz>
To: "Hector Sanjuan" <code@hector.link>,
"Ludovic Courtès" <ludo@gnu.org>,
"Antoine Eiche" <lewo@abesis.fr>
Cc: "go-ipfs-wg\\@ipfs.io" <go-ipfs-wg@ipfs.io>,
"33899\\@debbugs.gnu.org" <33899@debbugs.gnu.org>
Subject: [bug#33899] [PATCH 0/5] Distributing substitutes over IPFS
Date: Mon, 01 Jul 2019 23:36:34 +0200 [thread overview]
Message-ID: <87zhlxe8t9.fsf@ambrevar.xyz> (raw)
In-Reply-To: <pa258vxfuaaiuhOEdEpZPTqfDHe4hoICI7EPyJ284kk6qVWc_VakU3cwxbx46BVXobg0LMWyrjE_uX6nJ6XoJj78-mfjtsejqHlx3bmkNFw=@hector.link>
[-- Attachment #1: Type: text/plain, Size: 3324 bytes --]
Hi!
(Re-sending to debbugs, sorry for the double email :p)
A little update/recap after many months! :)
I talked with Héctor and some other people from IPFS + I reviewed Ludo's
patch so now I have a little better understanding of the current state
of affair.
- We could store the substitutes as tarballs on IPFS, but this has
some possible downsides:
- We would need to use IPFS' tar chunker to deduplicate the content of
the tarball. But the tar chunker is not well maintained currently,
and it's not clear whether it's reproducible at the moment, so it
would need some more work.
- Tarballs might induce some performance cost. Nix had attempted
something similar in the past and this may have incurred a significant
performance penalty, although this remains to be confirmed.
Lewo?
- Ludo's patch stores all files on IPFS individually. This way we don't
need to touch the tar chunker, so it's less work :)
This raises some other issues however:
- Extra metadata: IPFS stores files on UnixFSv1 which does not
include the executable bit.
- Right now we store a s-exp manifest with a list of files and a
list of executable bits. But maybe we don't have to roll out our own.
- UnixFSv1 has some metadata field, but Héctor and Alex did not
recommend using it (not sure why though).
- We could use UnixFSv2 but it's not released yet and it's unclear when
it's going to be released. So we can't really count on it right now.
- IPLD: As Héctor suggested in the previous email, we could leverage
IPLD and generate a JSON object that references the files with
their paths together with an "executable?" property.
A problem would arise if this IPLD object grows over the 2M
block-size limit because then we would have to shard it (something
that UnixFS would do automatically for us).
- Flat storage vs. tree storage: Right now we are storing the files
separately, but this has some shortcomings, namely we need multiple
"get" requests instead of just one, and that IPFS does
not "know" that those files are related. (We lose the web view of
the tree, etc.) Storing them as tree could be better.
I don't understand if that would work with the "IPLD manifest"
suggested above. Héctor?
- Pinning: Pinning all files separately incurs an overhead. It's
enough to just pin the IPLD object since it propagates recursively.
When adding a tree, then it's no problem since pinning is only done once.
- IPFS endpoint calls: instead of adding each file individually, it's
possible to add them all in one go. Can we add all files at once
while using a flat storage? (I.e. not adding them all under a common
root folder.)
To sum up, here is what remains to be done on the current patch:
- Add all files in one go without pinning them.
- Store as the file tree? Can we still us the IPLD object to reference
the files in the tree? Else use the "raw-leaves" option to avoid
wrapping small files in UnixFS blocks.
- Remove the Scheme manifest if IPLD can do.
- Generate the IPLD object and pin it.
Any corrections?
Thoughts?
Cheers!
--
Pierre Neidhardt
https://ambrevar.xyz/
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]
next prev parent reply other threads:[~2019-07-01 21:37 UTC|newest]
Thread overview: 23+ messages / expand[flat|nested] mbox.gz Atom feed top
2018-12-28 23:12 [bug#33899] [PATCH 0/5] Distributing substitutes over IPFS Ludovic Courtès
2018-12-28 23:15 ` [bug#33899] [PATCH 1/5] Add (guix json) Ludovic Courtès
2018-12-28 23:15 ` [bug#33899] [PATCH 2/5] tests: 'file=?' now recurses on directories Ludovic Courtès
2018-12-28 23:15 ` [bug#33899] [PATCH 3/5] Add (guix ipfs) Ludovic Courtès
2018-12-28 23:15 ` [bug#33899] [PATCH 4/5] publish: Add IPFS support Ludovic Courtès
2018-12-28 23:15 ` [bug#33899] [PATCH 5/5] DRAFT substitute: " Ludovic Courtès
2019-01-07 14:43 ` [bug#33899] [PATCH 0/5] Distributing substitutes over IPFS Hector Sanjuan
2019-01-14 13:17 ` Ludovic Courtès
2019-01-18 9:08 ` Hector Sanjuan
2019-01-18 9:52 ` Ludovic Courtès
2019-01-18 11:26 ` Hector Sanjuan
2019-07-01 21:36 ` Pierre Neidhardt [this message]
2019-07-06 8:44 ` Pierre Neidhardt
2019-07-12 20:02 ` Molly Mackinlay
2019-07-15 9:20 ` Alex Potsides
2019-07-12 20:15 ` Ludovic Courtès
2019-07-14 22:31 ` Hector Sanjuan
2019-07-15 9:24 ` Ludovic Courtès
2019-07-15 10:10 ` Pierre Neidhardt
2019-07-15 10:21 ` Hector Sanjuan
2019-05-13 18:51 ` Alex Griffin
2020-12-29 9:59 ` [bug#33899] Ludo's patch rebased on master Maxime Devos
2021-06-06 17:54 ` [bug#33899] [PATCH 0/5] Distributing substitutes over IPFS Tony Olagbaiye
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=87zhlxe8t9.fsf@ambrevar.xyz \
--to=mail@ambrevar.xyz \
--cc=33899@debbugs.gnu.org \
--cc=code@hector.link \
--cc=go-ipfs-wg@ipfs.io \
--cc=lewo@abesis.fr \
--cc=ludo@gnu.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this external index
https://git.savannah.gnu.org/cgit/guix.git
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.