Sustainable funding and maintenance for our infrastructure

Sustainable funding and maintenance for our infrastructure

[Ludovic Courtès]

Hello Guix!

We (Andreas, Chris, Ricardo, Romain, and myself) were having a
discussion about what it would take to set up a build farm similar to
what’s behind ci.guix: roughly 30 x86_64 servers, with 32-core/64-thread
CPUs and 128 GiB of RAM.  The reason for this discussion is that we were
thinking that we should not take our existing build farms for granted
and be prepared for the future.

The various options and back-of-the-envelope estimates we came up with
are as follows:

  1. Buying and hosting hardware:
      250k€ for hardware
      3k€/month (36k€/year)

  2. Renting machines (e.g., on Hetzner):
      6k€/month (72k€/year)

  3. Sponsored:
      get hardware and/or hosting sponsored (by academic institutions or
      companies).

Option #1 gives us “full control”, the downside being that it’s a lot of
work and a real burden (get crowdfunding for the initial funding, later
on to sustain funding to cover hosting, ensure Guix Foundation is up to
the task of managing the assets, and of course to take care of the
machines for their entire lifecycle).

Option #2 gives us less control (we don’t know exactly what hardware is
being used and have to trust the company hosting the machines).  The
upside is that it’s much less work over time (the company is responsible
for upgrading hardware) and less work initially (no need to raise as
much money to buy hardware).

Option #3 potentially gives less control (depending on the project’s
relation with the hosting organization) and makes the project dependent
on the sponsor and/or person(s) in touch with them.  On the upside, it
could significantly reduce costs (potentially to 0€).


This is an important topic for the project, one we should plan for:
socially, financially, technically.  This takes time, which is why
preparation is needed.

What do people think?

Ludo’ & co.

Re: Sustainable funding and maintenance for our infrastructure

[indieterminacy]

Hello,

Its worth pointing out the work of OpenBSD Amsterdam - which has raised 
over €40k for its respective foundation.

Its approach is to donate €10 per VM and €15 per VM renewal and has 850 
VMs.

Here are details on its hardware:
https://openbsd.amsterdam/hardware.html

It references this:
> Dell PowerEdge R630 w/ 2 x Intel(R) Xeon(R) CPU E5-2667 0 @ 3.20GHz
> 384G RAM
> Dell PERC H730 Mini

Hopefully such a long established initiative can provide some 
benchmarking approaches and ideas.
The lead behind it is very accomidating and knowlegable.

It also is used as a mechanism for highlighting projects they host:
https://openbsd.amsterdam/runs.html

Kind regards,


Jonathan

On 2024-07-02 10:24, Ludovic Courtès wrote:
> Hello Guix!
> 
> We (Andreas, Chris, Ricardo, Romain, and myself) were having a
> discussion about what it would take to set up a build farm similar to
> what’s behind ci.guix: roughly 30 x86_64 servers, with 
> 32-core/64-thread
> CPUs and 128 GiB of RAM.  The reason for this discussion is that we 
> were
> thinking that we should not take our existing build farms for granted
> and be prepared for the future.
> 
> The various options and back-of-the-envelope estimates we came up with
> are as follows:
> 
>   1. Buying and hosting hardware:
>       250k€ for hardware
>       3k€/month (36k€/year)
> 
>   2. Renting machines (e.g., on Hetzner):
>       6k€/month (72k€/year)
> 
>   3. Sponsored:
>       get hardware and/or hosting sponsored (by academic institutions 
> or
>       companies).
> 
> Option #1 gives us “full control”, the downside being that it’s a lot 
> of
> work and a real burden (get crowdfunding for the initial funding, later
> on to sustain funding to cover hosting, ensure Guix Foundation is up to
> the task of managing the assets, and of course to take care of the
> machines for their entire lifecycle).
> 
> Option #2 gives us less control (we don’t know exactly what hardware is
> being used and have to trust the company hosting the machines).  The
> upside is that it’s much less work over time (the company is 
> responsible
> for upgrading hardware) and less work initially (no need to raise as
> much money to buy hardware).
> 
> Option #3 potentially gives less control (depending on the project’s
> relation with the hosting organization) and makes the project dependent
> on the sponsor and/or person(s) in touch with them.  On the upside, it
> could significantly reduce costs (potentially to 0€).
> 
> 
> This is an important topic for the project, one we should plan for:
> socially, financially, technically.  This takes time, which is why
> preparation is needed.
> 
> What do people think?
> 
> Ludo’ & co.

Re: Sustainable funding and maintenance for our infrastructure

[Simon Tournier]

Hi,

On Tue, 02 Jul 2024 at 16:24, Ludovic Courtès <ludo@gnu.org> wrote:

>                           The reason for this discussion is that we were
> thinking that we should not take our existing build farms for granted
> and be prepared for the future.

Could you explain the rationale?  I understand and fully agree that
sustainable funding and maintenance for infrastructure are key topics
for the project.  Do we need to move ci.guix soon?  Related to Ricardo
announcement [1]?

Well, I am missing some context or steps.  Currently, the project is
mainly in Option #3 (sponsored).  The main sponsor is MDC located in
Berlin.  The second sponsor is personal funds coupled to hardware bought
by us or donated to us – I have in mind the build farm behind the name
Bordeaux; thanks Chris! And the third sponsor – at some extent – is
Inria located in Bordeaux.

We had discussions about reinforcing the second sponsor by replacing
personal funds by project-wide funds, say Guix Foundation, community,
etc.

Is this description correct?


> The various options and back-of-the-envelope estimates we came up with
> are as follows:
>
>   1. Buying and hosting hardware:
>       250k€ for hardware
>       3k€/month (36k€/year)
>
>   2. Renting machines (e.g., on Hetzner):
>       6k€/month (72k€/year)
>
>   3. Sponsored:
>       get hardware and/or hosting sponsored (by academic institutions or
>       companies).

Well, on the paper, option #1 appears to me appealing but how do we get
this 250k€?  Somehow, 250k€ would mean being able to secure 3k€/month
for over almost 7 years, right?

Except if we have a large donation that I am not aware, I do not see how
it would be possible to sign in being sure to secure 3k€/month for over
almost 7 years; considering the project has 12 years.

Other said, option #1 does not appear to me an option.

Option #2 could be a temporary option for a short time.  But again,
that’s something.


> Option #3 potentially gives less control (depending on the project’s
> relation with the hosting organization) and makes the project dependent
> on the sponsor and/or person(s) in touch with them.  On the upside, it
> could significantly reduce costs (potentially to 0€).

It remains option #3. :-)

For me, that’s the only viable option at scale.  The main costs should
be covered by sponsor as academical ones.  From my point of view, the
only sustainable option is to group people behind GuixHPC (I recall the
domain name hpc.guix.info is paid by Guix Foundation ;-)) and ask: a) if
their institutions are ready to donate and/or a) if we could run for
some grants altogether.

Somehow, Guix starts to be run in various scientific data centers and we
could take advantage of this opportunity.

Indeed, it locks in some relation with the hosting organizations and/or
the person in touch with them.  That’s said, Ricardo showed it works
well – or at least it can. :-) The key appears to me to not put all the
eggs in the same basket.

That’s my half-baked current opinion.  WDYT?

Cheers,
simon


1: I'm retiring (for a while); help needed
Ricardo Wurmus <rekado@elephly.net>
Fri, 31 May 2024 08:08:15 +0200
id:87o78mldio.fsf@elephly.net
https://lists.gnu.org/archive/html/guix-devel/2024-05
https://yhetil.org/guix/87o78mldio.fsf@elephly.net

Re: Sustainable funding and maintenance for our infrastructure

[Ricardo Wurmus]

Hi Simon,

> On Tue, 02 Jul 2024 at 16:24, Ludovic Courtès <ludo@gnu.org> wrote:
>
>>                           The reason for this discussion is that we were
>> thinking that we should not take our existing build farms for granted
>> and be prepared for the future.
>
> Could you explain the rationale?  I understand and fully agree that
> sustainable funding and maintenance for infrastructure are key topics
> for the project.  Do we need to move ci.guix soon?  Related to Ricardo
> announcement [1]?

There is no urgency.  The build farm at the MDC isn't going anywhere.

But it would be unwise for the project to assume that it will always
stay this way.  In the past we've also had some minor issues outside of
our immediate control that are attributable to hosting these servers at
a research institute, for example a trigger-happy firewall, or blanket
bans on large IP address ranges.

In the past we were given the opportunity to extend and upgrade the
build farm, but we cannot plan with good fortune like this.  As a
project it would be wise to continue our efforts to diversify our
distributed build farm.

>> The various options and back-of-the-envelope estimates we came up with
>> are as follows:
>>
>>   1. Buying and hosting hardware:
>>       250k€ for hardware
>>       3k€/month (36k€/year)
>>
>>   2. Renting machines (e.g., on Hetzner):
>>       6k€/month (72k€/year)
>>
>>   3. Sponsored:
>>       get hardware and/or hosting sponsored (by academic institutions or
>>       companies).
>
> Well, on the paper, option #1 appears to me appealing but how do we get
> this 250k€?  Somehow, 250k€ would mean being able to secure 3k€/month
> for over almost 7 years, right?
>
> Except if we have a large donation that I am not aware, I do not see how
> it would be possible to sign in being sure to secure 3k€/month for over
> almost 7 years; considering the project has 12 years.
>
> Other said, option #1 does not appear to me an option.

Correct.  I think it is a good reality check to see just how much value
there is (or was in 2019) in all these servers and what our realistic
options are to recreate this when eventually these machines are
decommissioned.  I don't see option #1 as realistic; not only is it a
prohibitively large up-front cost, it is also a serious continuous time
and money sink.  We'd also have to constantly play our cards well and
trade old hardware in for new hardware lest we are stuck with a metric
ton of e-waste.

> Option #2 could be a temporary option for a short time.  But again,
> that’s something.

I think option #2 is not actually terrible.  We like to say that the
cloud is just other people's machines, and our response to that is
aversion to a real or perceived loss of control.  But I'd like to put
this in perspective by asking how much control we *actually* have over
the build farm at the MDC right now.  In practice *I* have some
semblance of control over these machines because I have access to the
data centre.  For the most part, however, I treat these servers as warm
MDC furniture.

Yes, we'd lose a few more options when renting hardware via Hetzner (or
even the well-dressed monocled elephant over there: AWS), but I think we
should think carefully about how valuable our sacrifices are in exchange
for the practical advantages of not being stuck with a rack full of
industrial hardware.

Option #2 is rather quick to set up and quick to abandon should we run
out of money.  It does, however, depend on continuous donations, which
we are currently unable and possibly even unwilling to solicit.

-- 
Ricardo

Re: Sustainable funding and maintenance for our infrastructure

[Simon Tournier]

Hi Ricardo, all,

I agree with many words in the thread.

Reading all the messages in the thread, the solution seems to try a mix
of the three options.  We could imagine “buy and host” more exotic
hardware but not a complete data center neither, “rent” for specific
needs and ask for more “sponsor”.

However, the implicit question is a chicken-or-the-egg problem: in order
to know what we would like to invest in, we need to know how much we
would be able to invest; and to know how much, we need to determine for
what.


On Mon, 08 Jul 2024 at 14:02, Ricardo Wurmus <rekado@elephly.net> wrote:

>>> The various options and back-of-the-envelope estimates we came up with
>>> are as follows:
>>>
>>>   1. Buying and hosting hardware:
>>>       250k€ for hardware
>>>       3k€/month (36k€/year)
>>>
>>>   2. Renting machines (e.g., on Hetzner):
>>>       6k€/month (72k€/year)
>>>
>>>   3. Sponsored:
>>>       get hardware and/or hosting sponsored (by academic institutions or
>>>       companies).

[...]

> Option #2 is rather quick to set up and quick to abandon should we run
> out of money.  It does, however, depend on continuous donations, which
> we are currently unable and possibly even unwilling to solicit.

IMHO, the discussion of any potential option should be coupled with a
“investment plan”.  One without the other is a dead end, for what my
opinion is worth.

Well, because I am in the business of academics, I am able to imagine
some path forward for option #3 (sponsor).  For instance, [1] appears to
me a very good news.  Then, I am a bit clueless* about financing option
#1 or #2 – whatever the amount.  I mean, the first question is how to
secure several k€ or k$ – how much would we be able to secure?

Let assume Guix Foundation would be able to manage the money, the
discussion becomes: considering the current incomings, how do we scale?

Is the project ready to regularly look for incomings and run after them?
From my point of view, it’s easier to get hardware and/or hosting
sponsored by academic institutions or companies than to run by our own
for incomings in order to buy hardware and/or pay hosting.  But as said,
I am biased here. :-)

Then if we have a plan for buying hardware – depending on the amount –
it means we also need a plan to deal with such asset; on the financial
side and on the ecological side.

All in all, the first question in order to prepare the more or less next
years of our infrastructure seems about what appears financially
feasible.  Drawing what is financially doable right now, what would be
doable with work or what is just impossible in the near future, drawing
all this picture appears to me the next move in the discussion.

Other said, with my Guix Foundation hat, my proposal could be to set
this as an objective: polish Guix Foundation for preparing to scale up.


*clueless: Not fully. :-)  For sure, we could opt for crowd-founding or
 some grant here or there.  But still… how much?

Cheers,
simon

1: New North American based Guix Substitute Server, cuirass.genenetwork.org Now Available
"Collin J. Doering" <collin@rekahsoft.ca>
Sat, 06 Jul 2024 19:35:44 -0400
id:871q46ytyn.fsf@rekahsoft.ca
https://lists.gnu.org/archive/html/guix-devel/2024-07
https://yhetil.org/guix/871q46ytyn.fsf@rekahsoft.ca

Re: Sustainable funding and maintenance for our infrastructure

[Ludovic Courtès]

Hello,

Simon Tournier <zimon.toutoune@gmail.com> skribis:

> However, the implicit question is a chicken-or-the-egg problem: in order
> to know what we would like to invest in, we need to know how much we
> would be able to invest; and to know how much, we need to determine for
> what.

We currently have ~50k USD at the FSF’s Working Together fund, that’s a
start.  Once we’ve settled on a plan, we can call for contributions from
individuals and organizations and scale accordingly.

Thanks,
Ludo’.

Re: Sustainable funding and maintenance for our infrastructure

[Vagrant Cascadian]

[-- Attachment #1: Type: text/plain, Size: 1765 bytes --]

On 2024-07-02, Ludovic Courtès wrote:
> We (Andreas, Chris, Ricardo, Romain, and myself) were having a
> discussion about what it would take to set up a build farm similar to
> what’s behind ci.guix: roughly 30 x86_64 servers, with 32-core/64-thread
> CPUs and 128 GiB of RAM.  The reason for this discussion is that we were
> thinking that we should not take our existing build farms for granted
> and be prepared for the future.
>
> The various options and back-of-the-envelope estimates we came up with
> are as follows:
>
>   1. Buying and hosting hardware:
>       250k€ for hardware
>       3k€/month (36k€/year)
>
>   2. Renting machines (e.g., on Hetzner):
>       6k€/month (72k€/year)
>
>   3. Sponsored:
>       get hardware and/or hosting sponsored (by academic institutions or
>       companies).

This may be a little wild, but what are the downsides to doing some
combination of all of the above? Maybe higher bandwidth requirements
between the various pieces of infrastructure presumably being hosted in
different locations? Maybe also a little more complexity in the overall
setup?

A mixed strategy could reduce ... the upfront cost of buying and hosting
hardware (#1), the ongoing costs of renting (#2), and dependence on the
generosity of a third party for sponsored hardware & hosting (#3).

It seems like any strategy should have some redundancy (e.g. multiple
independent build farms) so that a failure in one datacenter does not
effectively take down the whole network...

In a sense, we already have some of that, with ci.guix.gnu.org and
bordeaux.guix.gnu.org, and also the new North American build farm
... though they are not full replacements for each other.

live well,
  vagrant

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

Re: Sustainable funding and maintenance for our infrastructure

[Vincent Legoll]

[-- Attachment #1: Type: text/plain, Size: 990 bytes --]

Hello,

On Mon, Jul 8, 2024 at 3:47 PM Vagrant Cascadian <vagrant@debian.org> wrote:

> This may be a little wild, but what are the downsides to doing some
> combination of all of the above?
>
> A mixed strategy could reduce ... the upfront cost of buying and hosting
> hardware (#1), the ongoing costs of renting (#2), and dependence on the
> generosity of a third party for sponsored hardware & hosting (#3).
>
> It seems like any strategy should have some redundancy (e.g. multiple
> independent build farms) so that a failure in one datacenter does not
> effectively take down the whole network...
>

That would be my opinion too.

But for the cloud renting I would first research if there are associated
network or other costs, because the computing is cheap only to lure
you into the (sometimes prohibitive) hidden costs.


> ... though they are not full replacements for each other.
>

Maybe that should be treated as a bug/issue.

-- 
Vincent Legoll

[-- Attachment #2: Type: text/html, Size: 1641 bytes --]

Re: Sustainable funding and maintenance for our infrastructure

[Tomas Volf]

[-- Attachment #1: Type: text/plain, Size: 3421 bytes --]

On 2024-07-08 18:28:23 +0000, Vincent Legoll wrote:
> Hello,
>
> On Mon, Jul 8, 2024 at 3:47 PM Vagrant Cascadian <vagrant@debian.org> wrote:
>
> > This may be a little wild, but what are the downsides to doing some
> > combination of all of the above?
> >
> > A mixed strategy could reduce ... the upfront cost of buying and hosting
> > hardware (#1), the ongoing costs of renting (#2), and dependence on the
> > generosity of a third party for sponsored hardware & hosting (#3).
> >
> > It seems like any strategy should have some redundancy (e.g. multiple
> > independent build farms) so that a failure in one datacenter does not
> > effectively take down the whole network...
> >
>
> That would be my opinion too.
>
> But for the cloud renting I would first research if there are associated
> network or other costs,

Yes, there are other costs.  You in general pay for the egress used.  I have no
idea how much traffic does our current farm use.  I will speak of AWS, because
that is the one I know, but other will be likely similar.  If we would have
setup just in one "Availability Zone" (think data center), traffic between VMs
is free.  Traffic to the internet however is not.  First 100 GB is free each
month, and after that (source [0]):

First 10 TB / Month		$0.09 per GB
Next 40 TB / Month		$0.09 per GB
Next 100 TB / Month		$0.07 per GB
Greater than 150 TB / Month	$0.05 per GB

The prices differ by region a bit, but Europe and US are the "cheap" ones.

Interesting point is that someone can just decide to download a lot from you,
and *you* would pay for it.  It is nice way to drive up bill of someone you do
not like.

You also pay for storage and various other things.  Doing cost estimates in the
cloud is hard, because everything is complex and there are lot of options to
spend on.

Than there also is the "moral" side of the clouds.

> because the computing is cheap only to lure you into the (sometimes
> prohibitive) hidden costs.

Even the computing is not cheap by itself, if you just want compute, Hetzner is
cheaper.  Clouds give the nice things on top, like storage snapshots, backups,
... (ignoring the cost).  But, at least in my experience, not cost saving,
unless you radically re-design your application/stack to match the cloud
providers infrastructure.

Naive "lift and shift" migrations to the cloud can have tangible benefits, but
cost saving is rarely one of them.

Now, this all describes just actual "clouds".  Hetzner, for example, does not
charge for traffic (with some exceptions[1]).  You pay 2 EUR for public IPv4,
but other than that, I did not notice any unexpected charges on my invoices.

I personally think that renting actual physical servers might be a reasonable
idea.  One issue to consider is that (Hetzner in particular) has a history of
MitM attacks against their customers (assumption is that it was due to court
order, but it was never confirmed AFAIK).  So I would expect physical compromise
(signing keys) to be possible as well.  I have no idea how that would compare to
the security in the current hosting.

Have a nice day,
Tomas

0: https://www.cloudflare.com/learning/cloud/what-is-aws-data-transfer-pricing/
1: https://docs.hetzner.com/robot/general/traffic/

--
There are only two hard things in Computer Science:
cache invalidation, naming things and off-by-one errors.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Sustainable funding and maintenance for our infrastructure

[Andreas Enge]

Hello Tomas,

Am Tue, Jul 09, 2024 at 11:47:38AM +0200 schrieb Tomas Volf:
> First 10 TB / Month		$0.09 per GB
> Next 40 TB / Month		$0.09 per GB
> Next 100 TB / Month		$0.07 per GB
> Greater than 150 TB / Month	$0.05 per GB

thanks for providing concrete figures!

I will continue without concrete figures and just back-of-the-envelope
computations. If I remember well (this is the very unconcrete point),
our throughput for bayfront at Aquilenet is about 100Mb/s. Some of it
is between bayfront and the bordeaux build machines (so would not count
if everything were in the same cloud), some of it is serving users (and
could stay on bayfront without being counted). After a complete
optimisation, every package would have to be transferred once out of the
cloud to the user facing server, while keeping all packages that are still
needed as inputs additionally in the cloud. I do not know how much this
would be; given that ideally every commit series should go through QA,
I think it could end up being quite a lot.

100Mb/s amounts to 32TB/month, which would cost about 3000€.

Compared to hosting, this is completely outrageous - notice that right
now at Aquilenet, we pay 75€ for one machine and not really limited
bandwidth.

After discussion with people at Aquilenet, I think it would be reasonable
to assume (no guarantee, also from a back-of-the-envelope computation)
we would need to pay about 100€/month for every additional 1U-machine we
would like to host in their rack. A small part of it is the rack space,
another small part the additional bandwidth, and the main part the
electricity cost. (Thus the estimate of 3000€/month for 30 machines.)

Andreas

Re: Sustainable funding and maintenance for our infrastructure

[Felix Lechner via Development of GNU Guix and the GNU System distribution.]

Hi Andreas,

On Thu, Jul 11 2024, Andreas Enge wrote:

> Compared to hosting, this is completely outrageous

This isn't my place to speak up, but I hoped to support the notion that
AWS can be expensive.  It's most useful when sites must adapt quickly.

I heard of someone getting a $40,000 bill when their blog became famous.

Kind regards
Felix

Re: Sustainable funding and maintenance for our infrastructure

[Ludovic Courtès]

Vagrant Cascadian <vagrant@debian.org> skribis:

> On 2024-07-02, Ludovic Courtès wrote:
>> We (Andreas, Chris, Ricardo, Romain, and myself) were having a
>> discussion about what it would take to set up a build farm similar to
>> what’s behind ci.guix: roughly 30 x86_64 servers, with 32-core/64-thread
>> CPUs and 128 GiB of RAM.  The reason for this discussion is that we were
>> thinking that we should not take our existing build farms for granted
>> and be prepared for the future.
>>
>> The various options and back-of-the-envelope estimates we came up with
>> are as follows:
>>
>>   1. Buying and hosting hardware:
>>       250k€ for hardware
>>       3k€/month (36k€/year)
>>
>>   2. Renting machines (e.g., on Hetzner):
>>       6k€/month (72k€/year)
>>
>>   3. Sponsored:
>>       get hardware and/or hosting sponsored (by academic institutions or
>>       companies).
>
> This may be a little wild, but what are the downsides to doing some
> combination of all of the above? Maybe higher bandwidth requirements
> between the various pieces of infrastructure presumably being hosted in
> different locations? Maybe also a little more complexity in the overall
> setup?

Good point.  In practice we’re already doing a combination of the above
and I agree that there are probably advantages to keep it that way.

My understanding is that Debian does #3 exclusively, with sponsorship
coming from a variety of organizations.

Ludo’.

Re: Sustainable funding and maintenance for our infrastructure

[Simon Tournier]

Hi,

On Thu, 11 Jul 2024 at 11:38, Ludovic Courtès <ludo@gnu.org> wrote:

> Good point.  In practice we’re already doing a combination of the above
> and I agree that there are probably advantages to keep it that way.

Well, from my perspective, the question is architecture per
architecture.  I mean, I think it’s doable to find sponsor (companies or
academic institutions) for x86_64 because it’s commonly used.  And
that’s not the case for less used others.

Other said, I think it would be a bad investment to buy x86_64 machines
when they could be donated and/or hosted.  However, it would be harder
for aarch64 or power9 or else, IMHO.

Cheers,
simon

Re: Sustainable funding and maintenance for our infrastructure

[Ludovic Courtès]

Simon Tournier <zimon.toutoune@gmail.com> skribis:

> Well, from my perspective, the question is architecture per
> architecture.  I mean, I think it’s doable to find sponsor (companies or
> academic institutions) for x86_64 because it’s commonly used.  And
> that’s not the case for less used others.

In the past we got donations from Arm:

  https://guix.gnu.org/en/blog/2018/aarch64-build-machines-donated/

I imagine the same could happen today for “alternative” architectures.

Ludo’.

Re: Sustainable funding and maintenance for our infrastructure

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 2861 bytes --]

On Tue, Jul 02, 2024 at 04:24:06PM +0200, Ludovic Courtès wrote:
> Hello Guix!
> 
> We (Andreas, Chris, Ricardo, Romain, and myself) were having a
> discussion about what it would take to set up a build farm similar to
> what’s behind ci.guix: roughly 30 x86_64 servers, with 32-core/64-thread
> CPUs and 128 GiB of RAM.  The reason for this discussion is that we were
> thinking that we should not take our existing build farms for granted
> and be prepared for the future.
> 
> The various options and back-of-the-envelope estimates we came up with
> are as follows:
> 
>   1. Buying and hosting hardware:
>       250k€ for hardware
>       3k€/month (36k€/year)
> 
>   2. Renting machines (e.g., on Hetzner):
>       6k€/month (72k€/year)
> 
>   3. Sponsored:
>       get hardware and/or hosting sponsored (by academic institutions or
>       companies).
> 
> Option #1 gives us “full control”, the downside being that it’s a lot of
> work and a real burden (get crowdfunding for the initial funding, later
> on to sustain funding to cover hosting, ensure Guix Foundation is up to
> the task of managing the assets, and of course to take care of the
> machines for their entire lifecycle).
> 
> Option #2 gives us less control (we don’t know exactly what hardware is
> being used and have to trust the company hosting the machines).  The
> upside is that it’s much less work over time (the company is responsible
> for upgrading hardware) and less work initially (no need to raise as
> much money to buy hardware).
> 
> Option #3 potentially gives less control (depending on the project’s
> relation with the hosting organization) and makes the project dependent
> on the sponsor and/or person(s) in touch with them.  On the upside, it
> could significantly reduce costs (potentially to 0€).
> 
> 
> This is an important topic for the project, one we should plan for:
> socially, financially, technically.  This takes time, which is why
> preparation is needed.
> 
> What do people think?
> 
> Ludo’ & co.

Looking at Hetzner, they have an option to rent a dedicated ARM server
with 80 cores/threads with 256GB of RAM and 2x3.84 TB NVMe drives for
under €300/month and a €94 setup charge. Correct me if I"m wrong, but
that one box is ~20x our current active aarch64/armv7 capacity.

Also looking at our current infrastructure at MDC, part of the reason we
have so many x86_64 machines is because that's what was bought with the
donated money, not because we actually needed quite that many, so some
of the numbers might be higher than we actually need.

-- 
Efraim Flashner   <efraim@flashner.co.il>   רנשלפ םירפא
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Sustainable funding and maintenance for our infrastructure

[Enrico Schwass]

Hi Guix

If you are looking for host services
I have good experience with netcup.de. They also have ARM machines.
My 3 machines have a great uptime

Bye
Enno


> Am 2024/07/08 um 18:28 schrieb Efraim Flashner <efraim@flashner.co.il>:
> 
> On Tue, Jul 02, 2024 at 04:24:06PM +0200, Ludovic Courtès wrote:
>> Hello Guix!
>> 
>> We (Andreas, Chris, Ricardo, Romain, and myself) were having a
>> discussion about what it would take to set up a build farm similar to
>> what’s behind ci.guix: roughly 30 x86_64 servers, with 32-core/64-thread
>> CPUs and 128 GiB of RAM.  The reason for this discussion is that we were
>> thinking that we should not take our existing build farms for granted
>> and be prepared for the future.
>> 
>> The various options and back-of-the-envelope estimates we came up with
>> are as follows:
>> 
>>  1. Buying and hosting hardware:
>>      250k€ for hardware
>>      3k€/month (36k€/year)
>> 
>>  2. Renting machines (e.g., on Hetzner):
>>      6k€/month (72k€/year)
>> 
>>  3. Sponsored:
>>      get hardware and/or hosting sponsored (by academic institutions or
>>      companies).
>> 
>> Option #1 gives us “full control”, the downside being that it’s a lot of
>> work and a real burden (get crowdfunding for the initial funding, later
>> on to sustain funding to cover hosting, ensure Guix Foundation is up to
>> the task of managing the assets, and of course to take care of the
>> machines for their entire lifecycle).
>> 
>> Option #2 gives us less control (we don’t know exactly what hardware is
>> being used and have to trust the company hosting the machines).  The
>> upside is that it’s much less work over time (the company is responsible
>> for upgrading hardware) and less work initially (no need to raise as
>> much money to buy hardware).
>> 
>> Option #3 potentially gives less control (depending on the project’s
>> relation with the hosting organization) and makes the project dependent
>> on the sponsor and/or person(s) in touch with them.  On the upside, it
>> could significantly reduce costs (potentially to 0€).
>> 
>> 
>> This is an important topic for the project, one we should plan for:
>> socially, financially, technically.  This takes time, which is why
>> preparation is needed.
>> 
>> What do people think?
>> 
>> Ludo’ & co.
> 
> Looking at Hetzner, they have an option to rent a dedicated ARM server
> with 80 cores/threads with 256GB of RAM and 2x3.84 TB NVMe drives for
> under €300/month and a €94 setup charge. Correct me if I"m wrong, but
> that one box is ~20x our current active aarch64/armv7 capacity.
> 
> Also looking at our current infrastructure at MDC, part of the reason we
> have so many x86_64 machines is because that's what was bought with the
> donated money, not because we actually needed quite that many, so some
> of the numbers might be higher than we actually need.
> 
> -- 
> Efraim Flashner   <efraim@flashner.co.il>   רנשלפ םירפא
> GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
> Confidentiality cannot be guaranteed on emails sent or received unencrypted
> <signature.asc>


> Am 2024/07/08 um 18:28 schrieb Efraim Flashner <efraim@flashner.co.il>:
> 
> On Tue, Jul 02, 2024 at 04:24:06PM +0200, Ludovic Courtès wrote:
>> Hello Guix!
>> 
>> We (Andreas, Chris, Ricardo, Romain, and myself) were having a
>> discussion about what it would take to set up a build farm similar to
>> what’s behind ci.guix: roughly 30 x86_64 servers, with 32-core/64-thread
>> CPUs and 128 GiB of RAM.  The reason for this discussion is that we were
>> thinking that we should not take our existing build farms for granted
>> and be prepared for the future.
>> 
>> The various options and back-of-the-envelope estimates we came up with
>> are as follows:
>> 
>>  1. Buying and hosting hardware:
>>      250k€ for hardware
>>      3k€/month (36k€/year)
>> 
>>  2. Renting machines (e.g., on Hetzner):
>>      6k€/month (72k€/year)
>> 
>>  3. Sponsored:
>>      get hardware and/or hosting sponsored (by academic institutions or
>>      companies).
>> 
>> Option #1 gives us “full control”, the downside being that it’s a lot of
>> work and a real burden (get crowdfunding for the initial funding, later
>> on to sustain funding to cover hosting, ensure Guix Foundation is up to
>> the task of managing the assets, and of course to take care of the
>> machines for their entire lifecycle).
>> 
>> Option #2 gives us less control (we don’t know exactly what hardware is
>> being used and have to trust the company hosting the machines).  The
>> upside is that it’s much less work over time (the company is responsible
>> for upgrading hardware) and less work initially (no need to raise as
>> much money to buy hardware).
>> 
>> Option #3 potentially gives less control (depending on the project’s
>> relation with the hosting organization) and makes the project dependent
>> on the sponsor and/or person(s) in touch with them.  On the upside, it
>> could significantly reduce costs (potentially to 0€).
>> 
>> 
>> This is an important topic for the project, one we should plan for:
>> socially, financially, technically.  This takes time, which is why
>> preparation is needed.
>> 
>> What do people think?
>> 
>> Ludo’ & co.
> 
> Looking at Hetzner, they have an option to rent a dedicated ARM server
> with 80 cores/threads with 256GB of RAM and 2x3.84 TB NVMe drives for
> under €300/month and a €94 setup charge. Correct me if I"m wrong, but
> that one box is ~20x our current active aarch64/armv7 capacity.
> 
> Also looking at our current infrastructure at MDC, part of the reason we
> have so many x86_64 machines is because that's what was bought with the
> donated money, not because we actually needed quite that many, so some
> of the numbers might be higher than we actually need.
> 
> --
> Efraim Flashner   <efraim@flashner.co.il>   רנשלפ םירפא
> GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
> Confidentiality cannot be guaranteed on emails sent or received unencrypted
> <signature.asc>

Re: Sustainable funding and maintenance for our infrastructure

[Andreas Enge]

Hello Enno,

thanks for the pointer!

Am Mon, Jul 08, 2024 at 07:21:38PM +0200 schrieb Enrico Schwass:
> If you are looking for host services
> I have good experience with netcup.de.

Looking at their virtual "root" servers:
   https://www.netcup.eu/vserver/
the most interesting price point would be the offer of 12 cores, 32GB of RAM
and 1TB of SSD for 32€/month.
The most expensive offer has twice the cores, 4 times the RAM and 4 times
the SSD for 108€/month.

In all cases, it would be shared, whereas Hetzner offers dedicated
servers at a higher price, which is a decision to make.

> They also have ARM machines.

There they only offer "vServers" with "vCores"; I do not quite understand
what that means. A maximum number of cores, and when there are too many
people on a machine, we get fewer? Well, the FAQ says that we would not be
root on this type of virtual machine, which does not look like what we want.

Andreas

Re: Sustainable funding and maintenance for our infrastructure

[Ludovic Courtès]

Hi,

Efraim Flashner <efraim@flashner.co.il> skribis:

> Looking at Hetzner, they have an option to rent a dedicated ARM server
> with 80 cores/threads with 256GB of RAM and 2x3.84 TB NVMe drives for
> under €300/month and a €94 setup charge. Correct me if I"m wrong, but
> that one box is ~20x our current active aarch64/armv7 capacity.

Interesting.

> Also looking at our current infrastructure at MDC, part of the reason we
> have so many x86_64 machines is because that's what was bought with the
> donated money, not because we actually needed quite that many, so some
> of the numbers might be higher than we actually need.

It was bought by the MDC, under the perception that this matched our
needs (we don’t need as much “normally” but there are peaks with
branches like ‘core-updates’ that can only be absorbed if there’s enough
computing power behind).

Ludo’.

Re: Sustainable funding and maintenance for our infrastructure

[Marek Paśnikowski]

Hello Guix and Administrators.

I know I am a bit late with this reply, but I would still like to share
the idea I had while reading this thread.

I believe there exists a way to relatively quickly and cheaply get access to more
computing power and possibly storage: DistCC + P2P distribution of the
data. It could be implemented as system services which individual
administrators could decide to enable on their systems.

While the immediate result of such implementation is not predictable, I
think it is necessary to realize strategies of decentralization and
antifragility. I wish I could express these ideas better and contribute
something more tangible, but I am currently struggling to organize my
personal life.

I hope this conceptual seed at least inspires some other out-of-the box
ideas.

I wish you all the best and thank you for this magical software of
Guix. It has already changed my life for the better and it will continue
improving it.

Re: Sustainable funding and maintenance for our infrastructure

[Jonathan Frederickson]

On Thu, Aug 1, 2024, at 6:11 PM, Marek Paśnikowski wrote:
> I believe there exists a way to relatively quickly and cheaply get access to more
> computing power and possibly storage: DistCC + P2P distribution of the
> data. It could be implemented as system services which individual
> administrators could decide to enable on their systems.

I have wanted something similar. The trust problem is difficult though; I have a Honeycomb LX2 that I'd gladly put to work contributing build power, and the thought of having distributed/community-run/P2P infrastructure for things like this is appealing, but build servers need to be trustworthy.

Guix accepting substitutes from servers without trusted signing keys if the same substitutes are available bit-for-bit on a trusted substitute server felt like it could be a hint at something. But your trusted build servers need to have built a package anyway for you to accept the same package from an untrusted one, so that doesn't avoid needing a lot of computing power in a trusted build farm.

Re: Sustainable funding and maintenance for our infrastructure

[Sergio Pastor Pérez]

"Jonathan Frederickson" <jonathan@terracrypt.net> writes:
> Guix accepting substitutes from servers without trusted signing keys if the same substitutes are available bit-for-bit on a trusted substitute server felt like it could be a hint at something. But your trusted build servers need to have built a package anyway for you to accept the same package from an untrusted one, so that doesn't avoid needing a lot of computing power in a trusted build farm.

Hello!

Wouldn't it be enough to have a few independent seeders that have the
same derivation output? We could have a field in the p2p service type
which allows the user to configure a "level of trust", where the user
specifies the minimum number of seeders with the same output for the
daemon to accept the substitute.

Regards,
Sergio.

Re: Sustainable funding and maintenance for our infrastructure

[Jonathan Frederickson]

On Tue, Aug 13, 2024, at 12:23 PM, Sergio Pastor Pérez wrote:
> Wouldn't it be enough to have a few independent seeders that have the
> same derivation output? We could have a field in the p2p service type
> which allows the user to configure a "level of trust", where the user
> specifies the minimum number of seeders with the same output for the
> daemon to accept the substitute.

This might be enough if you could do it, but the trouble is identifying "independent" seeders. If you get the same output from five different seeders, that could be five different people... or I could have set up five different nodes participating in the swarm serving my malicious substitutes. (This is known as a Sibyl attack.)

But maybe taking inspiration from this... perhaps you could do something more akin to some of the web-of-trust features of e.g. PGP. In other words, you might have the ability to partially trust a server's substitutes such that you'll only use a substitute if N other partially trusted servers (or at least one fully trusted server) serve up the same content. This would still not let you have a totally permissionless set of P2P substitutes, but it would allow the community to build a list of individuals who are at least trusted not to collude with one another, if not fully trusted.

Though there's a detail that might need addressing for this to work... you would want this to be an indication that multiple individuals were able to reproducibly build the same packages bit-for-bit. But my impression is that substitutes served by 'guix publish' are always signed with the substitute server's signing key, regardless of where they were built. That does mean that if 4 people were to pull substitutes of a package from one other person, those 5 people would end up serving substitutes originating from one person. You may want a way for someone running a substitute server to additionally attest that they had individually built the derivation in question.

Re: Sustainable funding and maintenance for our infrastructure

[Felix Lechner via Development of GNU Guix and the GNU System distribution.]

Hi Jonathan,

On Tue, Aug 13 2024, Jonathan Frederickson wrote:

> the trouble is identifying "independent" seeders.

Thank you for your valuable perspectives on this important topic.

The serving someone else's substitutes could also arise more innocently,
for example via a technical misconfiguration or because of an incentive
system that rewards the contribution of substitutes.

> you would want this to be an indication that multiple individuals were
> able to reproducibly build the same packages bit-for-bit.

> You may want a way for someone running a substitute server to
> additionally attest that they had individually built the derivation in
> question.

Is it possible for someone to reliably attest that they individually
built a reproducible work product?  I believe the needed variation in
inputs, like a hash, is incompatible with the goal of reproducability.

Kind regards
Felix

Re: Sustainable funding and maintenance for our infrastructure

[Jonathan Frederickson]

[-- Attachment #1: Type: text/plain, Size: 1111 bytes --]

On Wed, Aug 14, 2024, at 9:21 AM, Felix Lechner wrote:
> The serving someone else's substitutes could also arise more innocently,
> for example via a technical misconfiguration or because of an incentive
> system that rewards the contribution of substitutes.

Yes, indeed. And you may very well want such an incentive system, because having many people distribute substitutes in a P2P system is a natural way for people to contribute their own bandwidth.

> Is it possible for someone to reliably attest that they individually
> built a reproducible work product?  I believe the needed variation in
> inputs, like a hash, is incompatible with the goal of reproducability.

I think it's possible if the signature is detached from the reproducible work product to be signed. For example, it's like the difference between an embedded and detached signature of a file signed by GPG. Distributing a detached signature alongside a file doesn't change the hash of the file that's been signed.

Of course, you may not have built the build inputs yourself either - but those can be authenticated separately. (Recursion!)

[-- Attachment #2: Type: text/html, Size: 1545 bytes --]

P2P Guix package building and distribution

[Christine Lemmer-Webber]

"Jonathan Frederickson" <jonathan@terracrypt.net> writes:

> On Tue, Aug 13, 2024, at 12:23 PM, Sergio Pastor Pérez wrote:
>
>> Wouldn't it be enough to have a few independent seeders that have the
>> same derivation output? We could have a field in the p2p service type
>> which allows the user to configure a "level of trust", where the user
>> specifies the minimum number of seeders with the same output for the
>> daemon to accept the substitute.
>
> This might be enough if you could do it, but the trouble is
> identifying "independent" seeders. If you get the same output from
> five different seeders, that could be five different people... or I
> could have set up five different nodes participating in the swarm
> serving my malicious substitutes. (This is known as a Sibyl attack.)
>
> But maybe taking inspiration from this... perhaps you could do
> something more akin to some of the web-of-trust features of
> e.g. PGP. In other words, you might have the ability to partially
> trust a server's substitutes such that you'll only use a substitute if
> N other partially trusted servers (or at least one fully trusted
> server) serve up the same content. This would still not let you have a
> totally permissionless set of P2P substitutes, but it would allow the
> community to build a list of individuals who are at least trusted not
> to collude with one another, if not fully trusted.
>
> Though there's a detail that might need addressing for this to
> work... you would want this to be an indication that multiple
> individuals were able to reproducibly build the same packages
> bit-for-bit. But my impression is that substitutes served by 'guix
> publish' are always signed with the substitute server's signing key,
> regardless of where they were built. That does mean that if 4 people
> were to pull substitutes of a package from one other person, those 5
> people would end up serving substitutes originating from one
> person. You may want a way for someone running a substitute server to
> additionally attest that they had individually built the derivation in
> question.

I definitely think that this is a future we'd want with Guix.

Goals:
 - That our software be fully reproducible in the first place
 - P2P distribution mechanisms for inputs (this one is relatively easy!)
 - "Community participation" of building derivatives
 - P2P distribution of built artifacts (actually, if you have p2p
   distribution of inputs, you can have this one relatively for
   free/cheap)

There are challenges with all of this, but really we know enough what
p2p content addressed infrastructure looks like, this isn't the hard part.
Figuring out how to build a set of "semi-trusted sources" and the UX
around it is the hard part.

(In a weird way, compiling and verifying software is a "soft trap door
problem", I have been thinking.  Certainly not as much so as the
functions we require for cryptography to work, but it's still a trap
door, which is why build farms are expensive.)

I guess a worthwhile question is "where are the costs coming from"?
Ludovic said:

> The various options and back-of-the-envelope estimates we came up with
> are as follows:
>
>  1. Buying and hosting hardware:
>      250k€ for hardware
>      3k€/month (36k€/year)
>
>  2. Renting machines (e.g., on Hetzner):
>      6k€/month (72k€/year)
>
>  3. Sponsored:
>      get hardware and/or hosting sponsored (by academic institutions or
>      companies).

So I am guessing bandwith costs are significant but the 250k EUR for
hardware indicates this is especially a build farm issue rather than a
content distribution / bandwith issue.  (Do I have that right?)

Regardless, something I have thought about... #2 and #3 are cheaper but
"less preferable" than #1 because of security concerns, per my
understanding.

But... what if we managed to make #2 and #3 *more secure*?  Here is an
idea that is semi-p2p, and maybe a path towards a more full p2p option,
that we could possibly persue.

 - We have machines hosted that we trust a bit less, at some hosting
   facility, or possibly sponsored from someplace we trust even less.
   Let's imagine we went with the imaginary MegaCloud Inc.

 - We have a set of keys for semi-trusted "Guix Builders", who have
   machines that we run in our houses/lodgings/etc which sit around
   compiling Guix packages all day.  These could be eg people who aren't
   even committers but have gained the trust of committers and maybe
   have even come to something like Guix Days in person.

Now imagine for a moment that I wanted to download the latest version
of... let's go oldschool in our FOSS references and say some expensive
to compile browser named IceWeasel. ;)

I want to download the latest version of IceWeasel.  I could compile it
myself, or I could get a substitute.  #1 feels like the most
"trustworthy" option at first glance but actually it could be even a
single point of failure attack source.

Okay, but what if instead I had the option to download something signed
off by *all of* the MegaCloud build service and two "Guix Builders", and
they all came to the same hash?

This seems even better than #1 from a security/integrity perspective, I
think.

Just speculating...
 - Christine

Re: P2P Guix package building and distribution

[Andreas Enge]

Hello Christine,

Am Wed, Aug 21, 2024 at 06:07:58PM -0400 schrieb Christine Lemmer-Webber:
> >  1. Buying and hosting hardware:
> >      250k€ for hardware
> >      3k€/month (36k€/year)
> So I am guessing bandwith costs are significant but the 250k EUR for
> hardware indicates this is especially a build farm issue rather than a
> content distribution / bandwith issue.  (Do I have that right?)

to explain the figures, we counted 30 machines times 100€/month for
hosting, which is maybe what we could obtain if Aquilenet hosts them more
or less at cost for us. This includes bandwidth and electricity; from
what I understood, electricity is the major block, and it is directly
proportional to the number of machines. Bandwidth seems to be less of an
issue. First of all, it is computed somewhat strangely, like
"95% of the time you need to be below 1Gb/s, the remaining 5% you can be
above"; with bandwidth requirements for building coming in bursts
(sometimes inputs are downloaded, sometimes outputs are uploaded,
in-between there is computation without communication) adding a machine
might have the effect of smoothing out the distribution through the
law of large numbers without actually moving the 95-percentile.
And bandwidth is priced degressively.

This concerns bandwidth for building. Bandwidth for distribution is
proportional to the number of users, but so far has not been a bottleneck.
(Of course this will change when we get closer to world domination.)

> Okay, but what if instead I had the option to download something signed
> off by *all of* the MegaCloud build service and two "Guix Builders", and
> they all came to the same hash?

Would this not suppose that all these build instances are completely
disjoint from each other (like bordeaux or ci), and thus will have to
build everything from scratch? Since if a "Guix Builder" uses a MegaCloud
input, every build from then on is no more secure than a MegaCloud build.
Given the effort (in money and administrators' time) to run one build farm,
it does not look realistic that several people start their own build farm
at home.

Andreas

Re: P2P Guix package building and distribution

[Samuel Christie via Development of GNU Guix and the GNU System distribution.]

Andreas Enge <andreas@enge.fr> writes: 
> Am Wed, Aug 21, 2024 at 06:07:58PM -0400 schrieb Christine 
> Lemmer-Webber: 
>> Okay, but what if instead I had the option to download 
>> something signed off by *all of* the MegaCloud build service 
>> and two "Guix Builders", and they all came to the same hash?
>
> Would this not suppose that all these build instances are 
> completely disjoint from each other (like bordeaux or ci), and 
> thus will have to build everything from scratch? Since if a 
> "Guix Builder" uses a MegaCloud input, every build from then on 
> is no more secure than a MegaCloud build.

Yes; every step needs to be validated to ensure the final result 
is correct. That doesn't mean all builders need to validate the 
full tree, just one non-colluding party for each output.

Maybe one solution is to have the community perform the primary 
builds, with "official" builders arbitrating when there's a 
disagreement over the hash. As long as each package is built by at 
least one non-colluding peer, any deviations will be caught. This 
would be simpler than a full consensus protocol, but still avoid 
most conflicts and ensure correctness. Repeat offenders should 
eventually be ignored or banned somehow, but in the worst case it 
devolves to the system we have now of official servers building 
everything.

> Given the effort (in money and administrators' time) to run one 
> build farm, it does not look realistic that several people start 
> their own build farm at home.

Ideally, the software would be as simple as turning on a service 
to participate. And they shouldn't have to be "full" build farms, 
just share some of the load.

Since packages are already built locally if no substitutes are
available, it might be interesting to simply let the first few computers
that install the package build and share it. Then only ~2 people have to
build new packages (for minimal verification) instead of making everyone
do it until an official substitute exists.