From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp12.migadu.com ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms9.migadu.com with LMTPS id +KN7OPmlIGQM4gAASxT56A (envelope-from ) for ; Sun, 26 Mar 2023 22:07:21 +0200 Received: from aspmx1.migadu.com ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp12.migadu.com with LMTPS id 2CJFOPmlIGT4cgEAauVa8A (envelope-from ) for ; Sun, 26 Mar 2023 22:07:21 +0200 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 9245D205A4 for ; Sun, 26 Mar 2023 22:07:21 +0200 (CEST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pgWdc-0007rz-Pv; Sun, 26 Mar 2023 16:06:48 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pgWdZ-0007pE-CG for guix-devel@gnu.org; Sun, 26 Mar 2023 16:06:45 -0400 Received: from mail-ed1-x52a.google.com ([2a00:1450:4864:20::52a]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1pgWdX-0005U2-5G for guix-devel@gnu.org; Sun, 26 Mar 2023 16:06:45 -0400 Received: by mail-ed1-x52a.google.com with SMTP id h8so27688659ede.8 for ; Sun, 26 Mar 2023 13:06:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; t=1679861200; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=vidyrSxSrKd1M8cI4G3T3xSJLbkUqg9uqSNL9/cbQ1w=; b=dSxaLrYfi5Fy2sA65kYpcZPFFSAw62kT0XT+Lrg4eh6W9WAmyZAmucnNSMZo61Q0oq rAnW+dk/nkRXGI97JCqB2jP3TFVn6Q4CTxVFc81XqMuELkNx+Tkd6UUQAVUQ3rrJmgrX L9W8+xCoX7snX01nT9RUbR7f1hIcTIh7MqQrL2l3l05xGMOKolomLZa3eo19DKzhy9Cj h/GIyDNj6DTtGptgZKRLdEniZEnngYRUFVSZTMx5Rh5/Wz7oQtE3OTIuaFRwvXBvHILt knvUCRC2HrSCvivC5QelgtMBVKJzPS9sctkqx6fCs6hs2r0ecpm4hYVwVbgdeAV3F1rv 8Vjw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1679861200; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=vidyrSxSrKd1M8cI4G3T3xSJLbkUqg9uqSNL9/cbQ1w=; b=S4x6HxjDtG3Vq9kyQSQv+Uv5tU1wV0Bkr9WUwp9jaeaRy+3BmMAYf33fdWXyHOir/b 1K8fTVYYtABXrxF+cQh4zAC2seR6GWPjygGbTJ/Gpnkt4tjSxi/w5sx0v28P3FOvrIts LwxneqG4LP+kH6I6AvE0fsPoKVN46iE6uqIxD/7EzJaq8dy1cl1yhKkbWp8yZHA4BFwc kyDf0J3O+A7QFWxlOKpQBPp5HTAkcVGAHr0bztd7UdRqogutCSHGz9Lr4IgHXb4HVMdr cS+L27CZg3+PFpsqzOBkuT/ou+kUT2zWJkRLZUdyl6Ny7Y7+y4bFV2WRXYxndlzSMxwv rLQg== X-Gm-Message-State: AAQBX9cIza8/NmVxNxJU2EfCWMWKwxyq7HUuz1YBPNNEdBA0+feC9CbD +bns3F9yhAgdAPNRO/AlMxcbwLv+sleAUU8Yshc= X-Google-Smtp-Source: AKy350Zd+vZoPQULAWD/+HWoZddoiI1jx0zzJrQGl5kKW3g5YqmdxXUWWUuQuFYmGRj9kA83pH0qNZU2XAA5dYybtEA= X-Received: by 2002:a17:906:fc1c:b0:8b2:8876:cdd4 with SMTP id ov28-20020a170906fc1c00b008b28876cdd4mr4706806ejb.7.1679861200435; Sun, 26 Mar 2023 13:06:40 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Vijaya Anand Date: Mon, 27 Mar 2023 01:36:29 +0530 Message-ID: Subject: Re: [GSoC 23] distributed substitutes, cost of storage To: Attila Lendvai Cc: pukkamustard , guix-devel@gnu.org Content-Type: multipart/alternative; boundary="0000000000001fbe8d05f7d32a2d" Received-SPF: pass client-ip=2a00:1450:4864:20::52a; envelope-from=sunrockers8@gmail.com; helo=mail-ed1-x52a.google.com X-Spam_score_int: -17 X-Spam_score: -1.8 X-Spam_bar: - X-Spam_report: (-1.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, FREEMAIL_FROM=0.001, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: guix-devel-bounces+larch=yhetil.org@gnu.org X-Migadu-Country: US X-Migadu-Flow: FLOW_IN ARC-Seal: i=1; s=key1; d=yhetil.org; t=1679861241; a=rsa-sha256; cv=none; b=O4IVDEb3trTLJY38bRkPcjpSuT22F79MFZivBAce4DkapzG98UZOnmbG1RBoefzFblX016 5xohTldCw2KyfjGq6YLs+oUaci2YaP9lB2qmLuldJ+rLvfETt4r3od7z4V4qkyyfI0XKb2 jT9cZmM9gb21WgxxlgPg+CAZhA7Y49k3YKPaEgJV1qw7zhtTMz6NptUg5rKfbL90dW5GLM ojttpzH6wCl+a2oGXyHwV8JjOIbzAjYGPHSilrqAFaol5x48UMtcTfrvOE/H0KxiGNi3fX Cpy7Td9QBNLs7TAngZK4o2iDsbhdayeU02wyqnttzcMvU+18MDvbpzIEhv1L8g== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=dSxaLrYf; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org" ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1679861241; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post:dkim-signature; bh=vidyrSxSrKd1M8cI4G3T3xSJLbkUqg9uqSNL9/cbQ1w=; b=FYZ68hq9hjRa4gb4l6won+atSlJh0Y/SGxZNADutwx9NJIjjXslsD3pC3YLZBAD+zRGzXy 2ifbI4nv+qznguI4UDQwNLUMl2L5fpFXVwP7aentXXfcYXvNO35gdeS+2XcR+ncLsxtHqP FS5UUKsAV8H3s6radUNBg6lChDais4Uy5HzzHarB/klHpUi18dpedBrf/iBne7npYMG+tY vMB4beixDZuQPpE4C+NeBCnfMRR3v9Ybyu/37MapKk1QDxaA4oHloNPKC9uiFMVinDevc6 wTpkwjgvIqgiyVEGQE5UFlETAdE/gEuy5r8i0XhXA03nkVu6FPEw2OTDgvmLaA== Authentication-Results: aspmx1.migadu.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=dSxaLrYf; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org" X-Migadu-Scanner: scn1.migadu.com X-Migadu-Spam-Score: -4.96 X-Spam-Score: -4.96 X-Migadu-Queue-Id: 9245D205A4 X-TUID: e+RMH4aB1u+I --0000000000001fbe8d05f7d32a2d Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hi Attila Thanks for the welcome! I agree that the responsibility of re-uploading the blocks back to the network should be with the clients rather than the substitute server. Also I didn't really think about the point about having to pay for the p2p services at some point of time. In this case we will have to pay for the storage of substitutes both on the p2p storage backend as well as for storage in the substitute server am I right? So ideally we will want to eliminate the usage of these substitute servers and shift totally to p2p services and in this case we will have to shift the responsibility of re-uploading the blocks to the clients itself. Also if we dont keep the re-uploading blocks option as default for the users, won't users usually not choose to enable it? Maybe we can keep it on as default and resource conscious users can choose to turn it off? Please let me know your thoughts on these points and I will change the implementation point of my proposal accordingly. Thank you Vijaya Anand On Sun, 26 Mar 2023 at 00:30, Attila Lendvai wrote: > welcome on board Anand! > > > > In case a user requests for a substitute and there is a missing > > block in the decoding process, a HTTP request for block would sent > > to the substitute server and the server will encode the > > corresponding block in real time and push it back into the > > network. The block will be searched again and retrieved. > > > something to consider here: whose responsibility should it be that a > block, that is missing from a p2p network, is (re-)uploaded there? the > clients? or the current substitute server? > > my gut instinct says that it's better if the clients do the (re-)upload o= f > the blocks. > > in this architecture the substitute server is just another storage > mechanism along the other storage backends (although with a different > reliability characteristics), and it's the clients that are doing the > mirroring/spreading/distribution of the blocks among the various backends= . > the clients of course will/should keep the current substitute servers at > the bottom of their list of backends in their configuration. > > this way the load is distributed, and we don't need to add (too much) > extra complexity to the substitute server codebase, and the actors are le= ss > tightly coupled. > > it's another question whether this mirroring should be enabled by default > in the clients. probably it shouldn't, and the project infrastructure > should be running clients where it is turned on. altruistic third parties > could also enable this mirroring feature, and donate their > bandwidth/resources. > > there's an issue with this, though: > > some p2p storage backends will require some form of payment/credentials t= o > use their resources. arguably, all p2p storage networks that will survive > into the future will have some mechanism to limit the infinite abuse of > their resources. it is to be researched how these payment mechanisms work > on the various p2p networks, and whether it is possible that the Guix > project pays for the storage globally, and then the random clients will > have the necessary credentials to (re-)upload the missing blocks. > > this architecture shouldn't be impossible, because the content is > authenticated by its hash, and if the payment/authorization mechanism is > based on the hashes of the blocks (probably), then any client could > (re-)upload a missing block that was already paid for. > > i'll look into this, especially in the context of Swarm. > > meta: i think such specific discussions should be kept off-list, but the > financing of the storage fees is probably something that should be known > about more widely. > > -- > =E2=80=A2 attila lendvai > =E2=80=A2 PGP: 963F 5D5F 45C7 DFCD 0A39 > -- > Every lie is a debt to the truth. > > --0000000000001fbe8d05f7d32a2d Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi Attila

Thanks for the welcome!
=
I agree that the responsibility of re-uploading the blocks back t= o the network=C2=A0should be with the clients rather than the substitute se= rver. Also I didn't really think about the point about having to pay fo= r the p2p services at some point of time. In this case we will have to pay = for the storage of substitutes both on the p2p storage backend as well as f= or storage in the substitute server am I right? So ideally we will want to = eliminate the usage of these substitute servers and shift totally to p2p se= rvices and in this case we will have to shift the responsibility of re-uplo= ading the blocks to the clients itself.
Also if we dont=C2= =A0keep the re-uploading blocks option as default for the users, won't = users usually not choose to enable it? Maybe we can keep it on as default= =C2=A0and resource conscious users can choose to turn it off? Please let me= know your thoughts on these points and I will change the implementation po= int of my proposal accordingly.=C2=A0

Thank you
Vijaya Anand

On Sun, 26 Mar 2023 at 00:30, Attila Lendvai <= attila@lendvai.name> wrote:
welcome on board Anand!


> In case a user requests for a substitute and there is a missing
> block in the decoding process, a HTTP request for block would sent
> to the substitute server and the server will encode the
> corresponding block in real time and push it back into the
> network. The block will be searched again and retrieved.


something to consider here: whose responsibility should it be that a block,= that is missing from a p2p network, is (re-)uploaded there? the clients? o= r the current substitute server?

my gut instinct says that it's better if the clients do the (re-)upload= of the blocks.

in this architecture the substitute server is just another storage mechanis= m along the other storage backends (although with a different reliability c= haracteristics), and it's the clients that are doing the mirroring/spre= ading/distribution of the blocks among the various backends. the clients of= course will/should keep the current substitute servers at the bottom of th= eir list of backends in their configuration.

this way the load is distributed, and we don't need to add (too much) e= xtra complexity to the substitute server codebase, and the actors are less = tightly coupled.

it's another question whether this mirroring should be enabled by defau= lt in the clients. probably it shouldn't, and the project infrastructur= e should be running clients where it is turned on. altruistic third parties= could also enable this mirroring feature, and donate their bandwidth/resou= rces.

there's an issue with this, though:

some p2p storage backends will require some form of payment/credentials to = use their resources. arguably, all p2p storage networks that will survive i= nto the future will have some mechanism to limit the infinite abuse of thei= r resources. it is to be researched how these payment mechanisms work on th= e various p2p networks, and whether it is possible that the Guix project pa= ys for the storage globally, and then the random clients will have the nece= ssary credentials to (re-)upload the missing blocks.

this architecture shouldn't be impossible, because the content is authe= nticated by its hash, and if the payment/authorization mechanism is based o= n the hashes of the blocks (probably), then any client could (re-)upload a = missing block that was already paid for.

i'll look into this, especially in the context of Swarm.

meta: i think such specific discussions should be kept off-list, but the fi= nancing of the storage fees is probably something that should be known abou= t more widely.

--
=E2=80=A2 attila lendvai
=E2=80=A2 PGP: 963F 5D5F 45C7 DFCD 0A39
--
Every lie is a debt to the truth.

--0000000000001fbe8d05f7d32a2d--