[GSoC 23] distributed substitutes, cost of storage

[Attila Lendvai <attila@lendvai.name>]

welcome on board Anand!


> In case a user requests for a substitute and there is a missing
> block in the decoding process, a HTTP request for block would sent
> to the substitute server and the server will encode the
> corresponding block in real time and push it back into the
> network. The block will be searched again and retrieved.


something to consider here: whose responsibility should it be that a block, that is missing from a p2p network, is (re-)uploaded there? the clients? or the current substitute server?

my gut instinct says that it's better if the clients do the (re-)upload of the blocks.

in this architecture the substitute server is just another storage mechanism along the other storage backends (although with a different reliability characteristics), and it's the clients that are doing the mirroring/spreading/distribution of the blocks among the various backends. the clients of course will/should keep the current substitute servers at the bottom of their list of backends in their configuration.

this way the load is distributed, and we don't need to add (too much) extra complexity to the substitute server codebase, and the actors are less tightly coupled.

it's another question whether this mirroring should be enabled by default in the clients. probably it shouldn't, and the project infrastructure should be running clients where it is turned on. altruistic third parties could also enable this mirroring feature, and donate their bandwidth/resources.

there's an issue with this, though:

some p2p storage backends will require some form of payment/credentials to use their resources. arguably, all p2p storage networks that will survive into the future will have some mechanism to limit the infinite abuse of their resources. it is to be researched how these payment mechanisms work on the various p2p networks, and whether it is possible that the Guix project pays for the storage globally, and then the random clients will have the necessary credentials to (re-)upload the missing blocks.

this architecture shouldn't be impossible, because the content is authenticated by its hash, and if the payment/authorization mechanism is based on the hashes of the blocks (probably), then any client could (re-)upload a missing block that was already paid for.

i'll look into this, especially in the context of Swarm.

meta: i think such specific discussions should be kept off-list, but the financing of the storage fees is probably something that should be known about more widely.

--
• attila lendvai
• PGP: 963F 5D5F 45C7 DFCD 0A39
--
Every lie is a debt to the truth.