From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp12.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms9.migadu.com with LMTPS id UPIdJ/ZLI2SeEAEASxT56A (envelope-from ) for ; Tue, 28 Mar 2023 22:20:06 +0200 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp12.migadu.com with LMTPS id KJboJvZLI2SMPwAAauVa8A (envelope-from ) for ; Tue, 28 Mar 2023 22:20:06 +0200 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 24B00FCCA for ; Tue, 28 Mar 2023 22:20:06 +0200 (CEST) Authentication-Results: aspmx1.migadu.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=XiHGOo1P; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org"; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1680034806; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post:dkim-signature; bh=0aZRKXXIWYsuI0F45tHaNqs6bp74ravJTNJRLSOMUD8=; b=X13FCQeSm5cgHZBMJfMXs7EMTTmtiQqhABM+u78gluy/nuW87XYloT3ASZIgShj2t/kRIO qmnt04ORLT9ARS3kvGeJA735+dQB5vhuSrGb5NjX0HQV/sF+R2AEa1iXAUC8jFCJT7IWkP Z3kUUReSVLQ6/aKE6bY7+Qgnlv5HGQ6a70lLokmyT3LLQkAXg0t3v9dQdqDsKxPAXXgn+l cNGN1SLxIqOuZNdz83jlsi3n56jnPNTZxmoTsznzSLyy7EAdU26oXMbDPjh2HoBetefzJQ KALg4n2jFITERVU6Rgis8lDIH9GQF5T0hUmwBdyj1SNjxJe0v7//S0EX66nKOw== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1680034806; a=rsa-sha256; cv=none; b=j14QFEPt3ORdhbxm9ECX/pzJBC8HQed/kxbB8cRahaGdk2xe786Bu323D+9btSYEhMTmwS UTS2f4WmKLJtc5n0XpusrC10YATWPxPcxWCViokpcwfVXUrv3TSasjsQbF8cHaltz5yVhx BvBfVMaO7C9AYmOEoG+6AyHfhDJ2HXGwifPQGOd+N5uomcX8mxqpd1TX/DiNOumNjeZqRM eLF0NMHEgQsVnOV8fQBHF6ViJiVropYs30b4ILxtq3/F5orLiF+j59YyvKeS7Q2vDfouwQ vVnaZSGpe5ueTWdWApnD/Y+ypJ7oMzcgzk6Gd5qc1Da6rQqcqJu8nLvsuim8Gw== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=XiHGOo1P; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org"; dmarc=pass (policy=none) header.from=gmail.com Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1phFnB-0004v9-RQ; Tue, 28 Mar 2023 16:19:41 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1phFnA-0004v0-7D for guix-devel@gnu.org; Tue, 28 Mar 2023 16:19:40 -0400 Received: from mail-ed1-x52e.google.com ([2a00:1450:4864:20::52e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1phFn7-0004Us-HP for guix-devel@gnu.org; Tue, 28 Mar 2023 16:19:39 -0400 Received: by mail-ed1-x52e.google.com with SMTP id b20so54648141edd.1 for ; Tue, 28 Mar 2023 13:19:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; t=1680034775; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=0aZRKXXIWYsuI0F45tHaNqs6bp74ravJTNJRLSOMUD8=; b=XiHGOo1PrTxAoulHBcOWTSTHjJl72Dih1HyR/j4f50EEONecAg5ki7tMaOgTCvewhg Axu9t17fgNcekwqGtYuZv2Em8+3h5KhZeUtJdjazs6/RiaQ8+xWssvliYIH388BxD4UI 4xXNhCvu+Q/cYI4X0DzapwUk9lxWTfyuE1FwnDs15CbbvZtwHC6vFKKknISgSjq7vSGR rrJq43lvFvsiSNRt8yH5C4skMnATbAUGdstsCskpkEgcnvtPBkOXEPRH7lmFnnS66uR1 z+yrjV9yiEnp29c6p/g+unE5E4GOJbtnlIz4bzEhVg3wfb1wkOLXpdwws5G1VUo8LlCE nyYg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1680034775; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=0aZRKXXIWYsuI0F45tHaNqs6bp74ravJTNJRLSOMUD8=; b=pd3cvnOZgpvEhLmYF+wX3qr34wyZYwEhZmABNHYdo8Ww7ugEYbtofmC4tRhmKZICCP 0TnlF3whUY3+focP18L1NMUg7q+EiNaeMHl/eCOS20cTShryvMo6qeA1rV0vFMklPgQv Lk/ILrs4MH/pJyUaiWiXpDYHhMD3Z9HSsDjqhtjdkCe5QwH7SQny3Bs3z2wQBreZkEYA zlI0d/T2MCTsNYgWXgBvzmgI6N/BbNnxARLYj32faeIsRE9ougvSL2/ZYZ5Py/qzJuv9 d1eG/JJukhz9CcATBNJYB8GDiLgIZzcHBIwmlC9eGLNB8UUwrkZ901j5H5mvCI0a79nZ Tp5w== X-Gm-Message-State: AO0yUKV+gDceGG4soTjAB92Z/Jy3tt/teH7r73PxbE2+kIVOQLx67diT VzInOXilLQKX9AX1WTCoQtRNt3YGDoBlPA7Unw0= X-Google-Smtp-Source: AK7set+GteQK/dhfWOaUiQsXsnlNXysJZTkeoT1t9FmIcqsVXqQbYfl/GhjCQ9OVj5h+ZUxCmrB/IwNnO0H6DsY1P0Y= X-Received: by 2002:a17:907:d20e:b0:8d7:edbc:a7b6 with SMTP id vd14-20020a170907d20e00b008d7edbca7b6mr10409918ejc.2.1680034774995; Tue, 28 Mar 2023 13:19:34 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Vijaya Anand Date: Wed, 29 Mar 2023 01:49:23 +0530 Message-ID: Subject: Re: [GSoC 23] distributed substitutes, cost of storage To: Attila Lendvai Cc: guix-devel@gnu.org, pukkamustard Content-Type: multipart/alternative; boundary="000000000000f9571e05f7fb931e" Received-SPF: pass client-ip=2a00:1450:4864:20::52e; envelope-from=sunrockers8@gmail.com; helo=mail-ed1-x52e.google.com X-Spam_score_int: -17 X-Spam_score: -1.8 X-Spam_bar: - X-Spam_report: (-1.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, FREEMAIL_FROM=0.001, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: X-Migadu-Queue-Id: 24B00FCCA X-Spam-Score: -8.47 X-Migadu-Spam-Score: -8.47 X-Migadu-Scanner: scn0.migadu.com List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: guix-devel-bounces+larch=yhetil.org@gnu.org X-Migadu-Country: US X-Migadu-Flow: FLOW_IN X-TUID: u+uZvC1GsfVS --000000000000f9571e05f7fb931e Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hi, Sorry for the late reply. So in the case we are running swarm nodes that serves the network and hence help fund the substitute server, we can also use these to also upload eris encoded substitute blocks onto the network am I right? The total cost will thus be cost to run the swarm nodes + storage cost of substitute blocks in the network + cost to run substitute servers - money earned by running swarm nodes. But when we don't run swarm nodes which run to serve the network, the total cost is not really affect right as the cost to run swarm nodes will be lessened and no money will be earned. So in the context of fallback mechanism the user client can send request to the substitute server for the missing block and the substitute server will serve the eris encoded data block back to the user (using HTTP). The responsibility of uploading these missing blocks back to the network is of the third party nodes (which are running to serve the desired content to the network). But how do we send the message to the node to report the missing block on the network? Can it be done by the user itself? "i can dream about a future where there's a social network that is based on digital signatures and encryption, and my Guix client authorizes compiled binaries based on some weighted transitive closure of signatures of my trusted peers"....Interesting! In the case of accessing Guix substitutes from p2p network, we ensure authorization by Guix team by making sure the urn of the substitute is the urn mentioned in the narinfo (which we get from a trusted source like the substitute server). So in the case of accessing some random compiled binary from the network, we just need to verify the authority of the document providing the urn of the content? "i value consensual relationships, and that implies that the other party is well informed. and clogging someone's network bandwidth is not an expected behavior from installing a linux distribution." I agree with this point and also since there are already specialised nodes doing the work of uploading blocks onto the network I guess for now it is better to assign the task to them itself. Also I will give a read about how network's charge for uploading data onto them. As you had mentioned before if the payment is associated with a block id, then maybe we could have any random client upload data onto the network (if of course we are able to ship in with Guix itself, the configuration to upload data onto the said network). Vijaya Anand On Mon, 27 Mar 2023 at 2:49 AM Attila Lendvai wrote: > > Also I didn't really think about the point about having to pay for > > the p2p services at some point of time. > > > a quick note here: i forgot to mention that e.g. the Swarm Foundation has > programs for supporting opensource projects. so, chances are high that th= e > storage needs for Guix would be paid for by the foundation. > > > > In this case we will have to pay for the storage of substitutes both > > on the p2p storage backend as well as for storage in the substitute > > server am I right? > > > well, the substitute servers are currently already operated (and paid for= ) > by the Guix team. i don't think that p2p storage solutions have reached a > point of maturity that we could rely on them alone. there should definite= ly > be some time where both infrastructures are running in parallel. somewher= e > down the road a choice could be made to stop running the current substitu= te > servers, but we are far away from that. > > also, running swarm nodes that serve the network can earn money. so, the > cost of running enough swarm nodes to pay for the storage needs of Guix o= n > the swarm network should be in the same ballpark of the costs of running > the current substitute servers. the storage price will be market based > (this part is just being rolled out on the live network), so it's > reasonable to expect that people will fire up nodes if the storage price = go > well above the VPS costs. > > and not all p2p storage networks are made equal. e.g. IPFS is only a > registry of who is serving what. if you want to keep your data alive on > IPFS, then you need to run some nodes and make sure that they are serving > the content that you care about... and bear the costs of running these > nodes. i.e. the DoS attack surface of IPFS is much smaller. (IPFS stores > only the metadata in the DHT (i.e. where is what), while Swarm stores the= re > the data itself -- they are different architectures with different featur= es) > > (i need to learn more about GNUnet) > > > > So ideally we will want to eliminate the usage of these substitute > > servers and shift totally to p2p services and in this case we will > > have to shift the responsibility of re-uploading the blocks to the > > clients itself. > > > yep, that's my way of thinking, too. > > note that 'client' here has two meanings: > > 1) some part of the codebase > > 2) a program that is running on the computers of the Guix users > > i was using it in the first sense. > > without a functional Web of Trust solution, the Guix team will have to ru= n > nodes that compile packages, sign them with their PGP keys, and make them > available somewhere. currently it's published through a HTTP based servic= e > that we call 'substitute servers'. this GSoC project is about adding more > storage backends. > > but those backends don't solve the problem that the Guix users need to > trust someone with a private key who compiles and signs packages, regadle= ss > of the transport mechanism that gets the packages to the clients. > > i can dream about a future where there's a social network that is based o= n > digital signatures and encryption, and my Guix client authorizes compiled > binaries based on some weighted transitive closure of signatures of my > trusted peers... but we are not there yet. for now it's either trusting t= he > Guix team's signature, or setting up your own substitute servers and buil= d > workers (or trusting someone else, but i'm not aware of any third party > offering substitutes). > > > > Also if we dont keep the re-uploading blocks option as default for > > the users, won't users usually not choose to enable it? Maybe we can > > keep it on as default and resource conscious users can choose to > > turn it off? Please let me know your thoughts on these points and I > > will change the implementation point of my proposal accordingly. > > > first, it's a philosophical question: i value consensual relationships, > and that implies that the other party is well informed. and clogging > someone's network bandwidth is not an expected behavior from installing a > linux distribution. > > there's also a technical issue: these p2p backends will need to be > configured. i have my doubts that we could ship a default config with Gui= x > with which these p2p backends could just work out of the box, but... let'= s > hope that i'm wrong! > > and then there's the issue of payments: it's not obvious that a random > client can just upload binaries into a p2p storage network. on some p2p > networks someone needs to pay for that, and i don't yet understand well > enough how the data is authorized through payments (they are called posta= ge > stamps in swarm). > > a good, high level comparison of p2p storage solutions would be useful. > > -- > =E2=80=A2 attila lendvai > =E2=80=A2 PGP: 963F 5D5F 45C7 DFCD 0A39 > -- > =E2=80=9CThere is something in the human spirit that will survive and pre= vail, > there is a tiny and brilliant light burning in the heart of man that will > not go out no matter how dark the world becomes.=E2=80=9D > =E2=80=94 Leo Tolstoy (1828=E2=80=931910) > > --000000000000f9571e05f7fb931e Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi,

Sorry for the=C2=A0late reply.
So in the case we are running swarm nodes that serves the network and= hence help fund the substitute server, we can also use these to also uploa= d eris encoded substitute blocks onto the network am I right? The total cos= t will thus be cost to run the swarm nodes=C2=A0+ storage cost of substitut= e blocks in the network + cost to run substitute servers - money earned by = running swarm nodes. But when we don't run swarm nodes which run to ser= ve the network, the total cost is not really affect right as the cost to ru= n swarm nodes will be lessened and no money will be earned.=C2=A0
So in the context of fallback mechanism the user client can send request t= o the substitute server for the missing block and the substitute server wil= l serve the eris encoded data block back to the user (using HTTP). The resp= onsibility of uploading these missing blocks back to the network is of the = third party nodes (which are running to serve the desired content to the ne= twork). But how do we send the message to the node to report the missing bl= ock on the network? Can it be done by the user itself?

=
"i can dream about a future where there's a social network th= at is based on digital signatures and encryption, and my Guix client author= izes compiled binaries based on some weighted transitive closure of signatu= res of my trusted peers"....Interesting! In the case of accessing Guix= substitutes from p2p network, we ensure authorization by Guix team by maki= ng sure the urn of the substitute is the urn mentioned in the narinfo (whic= h we get from a trusted source like the substitute server). So in the case = of accessing some random compiled binary from the network, we just need to = verify the authority of the document providing the urn of the content?

"i value consensual relationships, and that impli= es that the other party is well informed. and clogging someone's networ= k bandwidth is not an expected behavior from installing a linux distributio= n."
I agree with this point and also since there are already= specialised nodes doing the work of uploading blocks onto the network I gu= ess for now it is better to assign the task to them itself. Also I will giv= e a read about how network's charge for uploading data onto them. As yo= u had mentioned before if the payment is associated with a block id, then m= aybe we could have any random client upload data onto the network (if of co= urse we are able to ship in with Guix itself, the configuration to upload d= ata onto the said network).

Vijaya Anand
=




On Mon, 27 Mar 2023 at 2:49 AM Attila Lendvai <attila@lendvai.name> wrote:
> Also I didn't really think about the point abou= t having to pay for
> the p2p services at some point of time.


a quick note here: i forgot to mention that e.g. the Swarm Foundation has p= rograms for supporting opensource projects. so, chances are high that the s= torage needs for Guix would be paid for by the foundation.


> In this case we will have to pay for the storage of substitutes both > on the p2p storage backend as well as for storage in the substitute > server am I right?


well, the substitute servers are currently already operated (and paid for) = by the Guix team. i don't think that p2p storage solutions have reached= a point of maturity that we could rely on them alone. there should definit= ely be some time where both infrastructures are running in parallel. somewh= ere down the road a choice could be made to stop running the current substi= tute servers, but we are far away from that.

also, running swarm nodes that serve the network can earn money. so, the co= st of running enough swarm nodes to pay for the storage needs of Guix on th= e swarm network should be in the same ballpark of the costs of running the = current substitute servers. the storage price will be market based (this pa= rt is just being rolled out on the live network), so it's reasonable to= expect that people will fire up nodes if the storage price go well above t= he VPS costs.

and not all p2p storage networks are made equal. e.g. IPFS is only a regist= ry of who is serving what. if you want to keep your data alive on IPFS, the= n you need to run some nodes and make sure that they are serving the conten= t that you care about... and bear the costs of running these nodes. i.e. th= e DoS attack surface of IPFS is much smaller. (IPFS stores only the metadat= a in the DHT (i.e. where is what), while Swarm stores there the data itself= -- they are different architectures with different features)

(i need to learn more about GNUnet)


> So ideally we will want to eliminate the usage of these substitute
> servers and shift totally to p2p services and in this case we will
> have to shift the responsibility of re-uploading the blocks to the
> clients itself.


yep, that's my way of thinking, too.

note that 'client' here has two meanings:

=C2=A01) some part of the codebase

=C2=A02) a program that is running on the computers of the Guix users

i was using it in the first sense.

without a functional Web of Trust solution, the Guix team will have to run = nodes that compile packages, sign them with their PGP keys, and make them a= vailable somewhere. currently it's published through a HTTP based servi= ce that we call 'substitute servers'. this GSoC project is about ad= ding more storage backends.

but those backends don't solve the problem that the Guix users need to = trust someone with a private key who compiles and signs packages, regadless= of the transport mechanism that gets the packages to the clients.

i can dream about a future where there's a social network that is based= on digital signatures and encryption, and my Guix client authorizes compil= ed binaries based on some weighted transitive closure of signatures of my t= rusted peers... but we are not there yet. for now it's either trusting = the Guix team's signature, or setting up your own substitute servers an= d build workers (or trusting someone else, but i'm not aware of any thi= rd party offering substitutes).


> Also if we dont keep the re-uploading blocks option as default for
> the users, won't users usually not choose to enable it? Maybe we c= an
> keep it on as default and resource conscious users can choose to
> turn it off? Please let me know your thoughts on these points and I > will change the implementation point of my proposal accordingly.


first, it's a philosophical question: i value consensual relationships,= and that implies that the other party is well informed. and clogging someo= ne's network bandwidth is not an expected behavior from installing a li= nux distribution.

there's also a technical issue: these p2p backends will need to be conf= igured. i have my doubts that we could ship a default config with Guix with= which these p2p backends could just work out of the box, but... let's = hope that i'm wrong!

and then there's the issue of payments: it's not obvious that a ran= dom client can just upload binaries into a p2p storage network. on some p2p= networks someone needs to pay for that, and i don't yet understand wel= l enough how the data is authorized through payments (they are called posta= ge stamps in swarm).

a good, high level comparison of p2p storage solutions would be useful.

--
=E2=80=A2 attila lendvai
=E2=80=A2 PGP: 963F 5D5F 45C7 DFCD 0A39
--
=E2=80=9CThere is something in the human spirit that will survive and preva= il, there is a tiny and brilliant light burning in the heart of man that wi= ll not go out no matter how dark the world becomes.=E2=80=9D
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =E2=80=94 Leo Tolstoy (1828=E2=80=931910)

--000000000000f9571e05f7fb931e--