Sustainable funding and maintenance for our infrastructure

Sustainable funding and maintenance for our infrastructure

[Ludovic Courtès]

Hello Guix!

We (Andreas, Chris, Ricardo, Romain, and myself) were having a
discussion about what it would take to set up a build farm similar to
what’s behind ci.guix: roughly 30 x86_64 servers, with 32-core/64-thread
CPUs and 128 GiB of RAM.  The reason for this discussion is that we were
thinking that we should not take our existing build farms for granted
and be prepared for the future.

The various options and back-of-the-envelope estimates we came up with
are as follows:

  1. Buying and hosting hardware:
      250k€ for hardware
      3k€/month (36k€/year)

  2. Renting machines (e.g., on Hetzner):
      6k€/month (72k€/year)

  3. Sponsored:
      get hardware and/or hosting sponsored (by academic institutions or
      companies).

Option #1 gives us “full control”, the downside being that it’s a lot of
work and a real burden (get crowdfunding for the initial funding, later
on to sustain funding to cover hosting, ensure Guix Foundation is up to
the task of managing the assets, and of course to take care of the
machines for their entire lifecycle).

Option #2 gives us less control (we don’t know exactly what hardware is
being used and have to trust the company hosting the machines).  The
upside is that it’s much less work over time (the company is responsible
for upgrading hardware) and less work initially (no need to raise as
much money to buy hardware).

Option #3 potentially gives less control (depending on the project’s
relation with the hosting organization) and makes the project dependent
on the sponsor and/or person(s) in touch with them.  On the upside, it
could significantly reduce costs (potentially to 0€).


This is an important topic for the project, one we should plan for:
socially, financially, technically.  This takes time, which is why
preparation is needed.

What do people think?

Ludo’ & co.

Sustainable funding and maintenance for our infrastructure

[Ludovic Courtès]

Hello Guix!

We (Andreas, Chris, Ricardo, Romain, and myself) were having a
discussion about what it would take to set up a build farm similar to
what’s behind ci.guix: roughly 30 x86_64 servers, with 32-core/64-thread
CPUs and 128 GiB of RAM.  The reason for this discussion is that we were
thinking that we should not take our existing build farms for granted
and be prepared for the future.

The various options and back-of-the-envelope estimates we came up with
are as follows:

  1. Buying and hosting hardware:
      250k€ for hardware
      3k€/month (36k€/year)

  2. Renting machines (e.g., on Hetzner):
      6k€/month (72k€/year)

  3. Sponsored:
      get hardware and/or hosting sponsored (by academic institutions or
      companies).

Option #1 gives us “full control”, the downside being that it’s a lot of
work and a real burden (get crowdfunding for the initial funding, later
on to sustain funding to cover hosting, ensure Guix Foundation is up to
the task of managing the assets, and of course to take care of the
machines for their entire lifecycle).

Option #2 gives us less control (we don’t know exactly what hardware is
being used and have to trust the company hosting the machines).  The
upside is that it’s much less work over time (the company is responsible
for upgrading hardware) and less work initially (no need to raise as
much money to buy hardware).

Option #3 potentially gives less control (depending on the project’s
relation with the hosting organization) and makes the project dependent
on the sponsor and/or person(s) in touch with them.  On the upside, it
could significantly reduce costs (potentially to 0€).  If you have any
potential contacts, please get in touch with us (via the private
guix-sysadmin@gnu.org mailing list).


This is an important topic for the project, one we should plan for:
socially, financially, technically.  This takes time, which is why
preparation is needed.

What do people think?

Ludo’ & co.

Re: Sustainable funding and maintenance for our infrastructure

[indieterminacy]

Hello,

Its worth pointing out the work of OpenBSD Amsterdam - which has raised 
over €40k for its respective foundation.

Its approach is to donate €10 per VM and €15 per VM renewal and has 850 
VMs.

Here are details on its hardware:
https://openbsd.amsterdam/hardware.html

It references this:
> Dell PowerEdge R630 w/ 2 x Intel(R) Xeon(R) CPU E5-2667 0 @ 3.20GHz
> 384G RAM
> Dell PERC H730 Mini

Hopefully such a long established initiative can provide some 
benchmarking approaches and ideas.
The lead behind it is very accomidating and knowlegable.

It also is used as a mechanism for highlighting projects they host:
https://openbsd.amsterdam/runs.html

Kind regards,


Jonathan

On 2024-07-02 10:24, Ludovic Courtès wrote:
> Hello Guix!
> 
> We (Andreas, Chris, Ricardo, Romain, and myself) were having a
> discussion about what it would take to set up a build farm similar to
> what’s behind ci.guix: roughly 30 x86_64 servers, with 
> 32-core/64-thread
> CPUs and 128 GiB of RAM.  The reason for this discussion is that we 
> were
> thinking that we should not take our existing build farms for granted
> and be prepared for the future.
> 
> The various options and back-of-the-envelope estimates we came up with
> are as follows:
> 
>   1. Buying and hosting hardware:
>       250k€ for hardware
>       3k€/month (36k€/year)
> 
>   2. Renting machines (e.g., on Hetzner):
>       6k€/month (72k€/year)
> 
>   3. Sponsored:
>       get hardware and/or hosting sponsored (by academic institutions 
> or
>       companies).
> 
> Option #1 gives us “full control”, the downside being that it’s a lot 
> of
> work and a real burden (get crowdfunding for the initial funding, later
> on to sustain funding to cover hosting, ensure Guix Foundation is up to
> the task of managing the assets, and of course to take care of the
> machines for their entire lifecycle).
> 
> Option #2 gives us less control (we don’t know exactly what hardware is
> being used and have to trust the company hosting the machines).  The
> upside is that it’s much less work over time (the company is 
> responsible
> for upgrading hardware) and less work initially (no need to raise as
> much money to buy hardware).
> 
> Option #3 potentially gives less control (depending on the project’s
> relation with the hosting organization) and makes the project dependent
> on the sponsor and/or person(s) in touch with them.  On the upside, it
> could significantly reduce costs (potentially to 0€).
> 
> 
> This is an important topic for the project, one we should plan for:
> socially, financially, technically.  This takes time, which is why
> preparation is needed.
> 
> What do people think?
> 
> Ludo’ & co.

Re: Sustainable funding and maintenance for our infrastructure

[Simon Tournier]

Hi,

On Tue, 02 Jul 2024 at 16:24, Ludovic Courtès <ludo@gnu.org> wrote:

>                           The reason for this discussion is that we were
> thinking that we should not take our existing build farms for granted
> and be prepared for the future.

Could you explain the rationale?  I understand and fully agree that
sustainable funding and maintenance for infrastructure are key topics
for the project.  Do we need to move ci.guix soon?  Related to Ricardo
announcement [1]?

Well, I am missing some context or steps.  Currently, the project is
mainly in Option #3 (sponsored).  The main sponsor is MDC located in
Berlin.  The second sponsor is personal funds coupled to hardware bought
by us or donated to us – I have in mind the build farm behind the name
Bordeaux; thanks Chris! And the third sponsor – at some extent – is
Inria located in Bordeaux.

We had discussions about reinforcing the second sponsor by replacing
personal funds by project-wide funds, say Guix Foundation, community,
etc.

Is this description correct?


> The various options and back-of-the-envelope estimates we came up with
> are as follows:
>
>   1. Buying and hosting hardware:
>       250k€ for hardware
>       3k€/month (36k€/year)
>
>   2. Renting machines (e.g., on Hetzner):
>       6k€/month (72k€/year)
>
>   3. Sponsored:
>       get hardware and/or hosting sponsored (by academic institutions or
>       companies).

Well, on the paper, option #1 appears to me appealing but how do we get
this 250k€?  Somehow, 250k€ would mean being able to secure 3k€/month
for over almost 7 years, right?

Except if we have a large donation that I am not aware, I do not see how
it would be possible to sign in being sure to secure 3k€/month for over
almost 7 years; considering the project has 12 years.

Other said, option #1 does not appear to me an option.

Option #2 could be a temporary option for a short time.  But again,
that’s something.


> Option #3 potentially gives less control (depending on the project’s
> relation with the hosting organization) and makes the project dependent
> on the sponsor and/or person(s) in touch with them.  On the upside, it
> could significantly reduce costs (potentially to 0€).

It remains option #3. :-)

For me, that’s the only viable option at scale.  The main costs should
be covered by sponsor as academical ones.  From my point of view, the
only sustainable option is to group people behind GuixHPC (I recall the
domain name hpc.guix.info is paid by Guix Foundation ;-)) and ask: a) if
their institutions are ready to donate and/or a) if we could run for
some grants altogether.

Somehow, Guix starts to be run in various scientific data centers and we
could take advantage of this opportunity.

Indeed, it locks in some relation with the hosting organizations and/or
the person in touch with them.  That’s said, Ricardo showed it works
well – or at least it can. :-) The key appears to me to not put all the
eggs in the same basket.

That’s my half-baked current opinion.  WDYT?

Cheers,
simon


1: I'm retiring (for a while); help needed
Ricardo Wurmus <rekado@elephly.net>
Fri, 31 May 2024 08:08:15 +0200
id:87o78mldio.fsf@elephly.net
https://lists.gnu.org/archive/html/guix-devel/2024-05
https://yhetil.org/guix/87o78mldio.fsf@elephly.net

Re: Sustainable funding and maintenance for our infrastructure

[Ricardo Wurmus]

Hi Simon,

> On Tue, 02 Jul 2024 at 16:24, Ludovic Courtès <ludo@gnu.org> wrote:
>
>>                           The reason for this discussion is that we were
>> thinking that we should not take our existing build farms for granted
>> and be prepared for the future.
>
> Could you explain the rationale?  I understand and fully agree that
> sustainable funding and maintenance for infrastructure are key topics
> for the project.  Do we need to move ci.guix soon?  Related to Ricardo
> announcement [1]?

There is no urgency.  The build farm at the MDC isn't going anywhere.

But it would be unwise for the project to assume that it will always
stay this way.  In the past we've also had some minor issues outside of
our immediate control that are attributable to hosting these servers at
a research institute, for example a trigger-happy firewall, or blanket
bans on large IP address ranges.

In the past we were given the opportunity to extend and upgrade the
build farm, but we cannot plan with good fortune like this.  As a
project it would be wise to continue our efforts to diversify our
distributed build farm.

>> The various options and back-of-the-envelope estimates we came up with
>> are as follows:
>>
>>   1. Buying and hosting hardware:
>>       250k€ for hardware
>>       3k€/month (36k€/year)
>>
>>   2. Renting machines (e.g., on Hetzner):
>>       6k€/month (72k€/year)
>>
>>   3. Sponsored:
>>       get hardware and/or hosting sponsored (by academic institutions or
>>       companies).
>
> Well, on the paper, option #1 appears to me appealing but how do we get
> this 250k€?  Somehow, 250k€ would mean being able to secure 3k€/month
> for over almost 7 years, right?
>
> Except if we have a large donation that I am not aware, I do not see how
> it would be possible to sign in being sure to secure 3k€/month for over
> almost 7 years; considering the project has 12 years.
>
> Other said, option #1 does not appear to me an option.

Correct.  I think it is a good reality check to see just how much value
there is (or was in 2019) in all these servers and what our realistic
options are to recreate this when eventually these machines are
decommissioned.  I don't see option #1 as realistic; not only is it a
prohibitively large up-front cost, it is also a serious continuous time
and money sink.  We'd also have to constantly play our cards well and
trade old hardware in for new hardware lest we are stuck with a metric
ton of e-waste.

> Option #2 could be a temporary option for a short time.  But again,
> that’s something.

I think option #2 is not actually terrible.  We like to say that the
cloud is just other people's machines, and our response to that is
aversion to a real or perceived loss of control.  But I'd like to put
this in perspective by asking how much control we *actually* have over
the build farm at the MDC right now.  In practice *I* have some
semblance of control over these machines because I have access to the
data centre.  For the most part, however, I treat these servers as warm
MDC furniture.

Yes, we'd lose a few more options when renting hardware via Hetzner (or
even the well-dressed monocled elephant over there: AWS), but I think we
should think carefully about how valuable our sacrifices are in exchange
for the practical advantages of not being stuck with a rack full of
industrial hardware.

Option #2 is rather quick to set up and quick to abandon should we run
out of money.  It does, however, depend on continuous donations, which
we are currently unable and possibly even unwilling to solicit.

-- 
Ricardo

Re: Sustainable funding and maintenance for our infrastructure

[Vagrant Cascadian]

[-- Attachment #1: Type: text/plain, Size: 1765 bytes --]

On 2024-07-02, Ludovic Courtès wrote:
> We (Andreas, Chris, Ricardo, Romain, and myself) were having a
> discussion about what it would take to set up a build farm similar to
> what’s behind ci.guix: roughly 30 x86_64 servers, with 32-core/64-thread
> CPUs and 128 GiB of RAM.  The reason for this discussion is that we were
> thinking that we should not take our existing build farms for granted
> and be prepared for the future.
>
> The various options and back-of-the-envelope estimates we came up with
> are as follows:
>
>   1. Buying and hosting hardware:
>       250k€ for hardware
>       3k€/month (36k€/year)
>
>   2. Renting machines (e.g., on Hetzner):
>       6k€/month (72k€/year)
>
>   3. Sponsored:
>       get hardware and/or hosting sponsored (by academic institutions or
>       companies).

This may be a little wild, but what are the downsides to doing some
combination of all of the above? Maybe higher bandwidth requirements
between the various pieces of infrastructure presumably being hosted in
different locations? Maybe also a little more complexity in the overall
setup?

A mixed strategy could reduce ... the upfront cost of buying and hosting
hardware (#1), the ongoing costs of renting (#2), and dependence on the
generosity of a third party for sponsored hardware & hosting (#3).

It seems like any strategy should have some redundancy (e.g. multiple
independent build farms) so that a failure in one datacenter does not
effectively take down the whole network...

In a sense, we already have some of that, with ci.guix.gnu.org and
bordeaux.guix.gnu.org, and also the new North American build farm
... though they are not full replacements for each other.

live well,
  vagrant

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

Re: Sustainable funding and maintenance for our infrastructure

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 2861 bytes --]

On Tue, Jul 02, 2024 at 04:24:06PM +0200, Ludovic Courtès wrote:
> Hello Guix!
> 
> We (Andreas, Chris, Ricardo, Romain, and myself) were having a
> discussion about what it would take to set up a build farm similar to
> what’s behind ci.guix: roughly 30 x86_64 servers, with 32-core/64-thread
> CPUs and 128 GiB of RAM.  The reason for this discussion is that we were
> thinking that we should not take our existing build farms for granted
> and be prepared for the future.
> 
> The various options and back-of-the-envelope estimates we came up with
> are as follows:
> 
>   1. Buying and hosting hardware:
>       250k€ for hardware
>       3k€/month (36k€/year)
> 
>   2. Renting machines (e.g., on Hetzner):
>       6k€/month (72k€/year)
> 
>   3. Sponsored:
>       get hardware and/or hosting sponsored (by academic institutions or
>       companies).
> 
> Option #1 gives us “full control”, the downside being that it’s a lot of
> work and a real burden (get crowdfunding for the initial funding, later
> on to sustain funding to cover hosting, ensure Guix Foundation is up to
> the task of managing the assets, and of course to take care of the
> machines for their entire lifecycle).
> 
> Option #2 gives us less control (we don’t know exactly what hardware is
> being used and have to trust the company hosting the machines).  The
> upside is that it’s much less work over time (the company is responsible
> for upgrading hardware) and less work initially (no need to raise as
> much money to buy hardware).
> 
> Option #3 potentially gives less control (depending on the project’s
> relation with the hosting organization) and makes the project dependent
> on the sponsor and/or person(s) in touch with them.  On the upside, it
> could significantly reduce costs (potentially to 0€).
> 
> 
> This is an important topic for the project, one we should plan for:
> socially, financially, technically.  This takes time, which is why
> preparation is needed.
> 
> What do people think?
> 
> Ludo’ & co.

Looking at Hetzner, they have an option to rent a dedicated ARM server
with 80 cores/threads with 256GB of RAM and 2x3.84 TB NVMe drives for
under €300/month and a €94 setup charge. Correct me if I"m wrong, but
that one box is ~20x our current active aarch64/armv7 capacity.

Also looking at our current infrastructure at MDC, part of the reason we
have so many x86_64 machines is because that's what was bought with the
donated money, not because we actually needed quite that many, so some
of the numbers might be higher than we actually need.

-- 
Efraim Flashner   <efraim@flashner.co.il>   רנשלפ םירפא
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Sustainable funding and maintenance for our infrastructure

[Enrico Schwass]

Hi Guix

If you are looking for host services
I have good experience with netcup.de. They also have ARM machines.
My 3 machines have a great uptime

Bye
Enno


> Am 2024/07/08 um 18:28 schrieb Efraim Flashner <efraim@flashner.co.il>:
> 
> On Tue, Jul 02, 2024 at 04:24:06PM +0200, Ludovic Courtès wrote:
>> Hello Guix!
>> 
>> We (Andreas, Chris, Ricardo, Romain, and myself) were having a
>> discussion about what it would take to set up a build farm similar to
>> what’s behind ci.guix: roughly 30 x86_64 servers, with 32-core/64-thread
>> CPUs and 128 GiB of RAM.  The reason for this discussion is that we were
>> thinking that we should not take our existing build farms for granted
>> and be prepared for the future.
>> 
>> The various options and back-of-the-envelope estimates we came up with
>> are as follows:
>> 
>>  1. Buying and hosting hardware:
>>      250k€ for hardware
>>      3k€/month (36k€/year)
>> 
>>  2. Renting machines (e.g., on Hetzner):
>>      6k€/month (72k€/year)
>> 
>>  3. Sponsored:
>>      get hardware and/or hosting sponsored (by academic institutions or
>>      companies).
>> 
>> Option #1 gives us “full control”, the downside being that it’s a lot of
>> work and a real burden (get crowdfunding for the initial funding, later
>> on to sustain funding to cover hosting, ensure Guix Foundation is up to
>> the task of managing the assets, and of course to take care of the
>> machines for their entire lifecycle).
>> 
>> Option #2 gives us less control (we don’t know exactly what hardware is
>> being used and have to trust the company hosting the machines).  The
>> upside is that it’s much less work over time (the company is responsible
>> for upgrading hardware) and less work initially (no need to raise as
>> much money to buy hardware).
>> 
>> Option #3 potentially gives less control (depending on the project’s
>> relation with the hosting organization) and makes the project dependent
>> on the sponsor and/or person(s) in touch with them.  On the upside, it
>> could significantly reduce costs (potentially to 0€).
>> 
>> 
>> This is an important topic for the project, one we should plan for:
>> socially, financially, technically.  This takes time, which is why
>> preparation is needed.
>> 
>> What do people think?
>> 
>> Ludo’ & co.
> 
> Looking at Hetzner, they have an option to rent a dedicated ARM server
> with 80 cores/threads with 256GB of RAM and 2x3.84 TB NVMe drives for
> under €300/month and a €94 setup charge. Correct me if I"m wrong, but
> that one box is ~20x our current active aarch64/armv7 capacity.
> 
> Also looking at our current infrastructure at MDC, part of the reason we
> have so many x86_64 machines is because that's what was bought with the
> donated money, not because we actually needed quite that many, so some
> of the numbers might be higher than we actually need.
> 
> -- 
> Efraim Flashner   <efraim@flashner.co.il>   רנשלפ םירפא
> GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
> Confidentiality cannot be guaranteed on emails sent or received unencrypted
> <signature.asc>


> Am 2024/07/08 um 18:28 schrieb Efraim Flashner <efraim@flashner.co.il>:
> 
> On Tue, Jul 02, 2024 at 04:24:06PM +0200, Ludovic Courtès wrote:
>> Hello Guix!
>> 
>> We (Andreas, Chris, Ricardo, Romain, and myself) were having a
>> discussion about what it would take to set up a build farm similar to
>> what’s behind ci.guix: roughly 30 x86_64 servers, with 32-core/64-thread
>> CPUs and 128 GiB of RAM.  The reason for this discussion is that we were
>> thinking that we should not take our existing build farms for granted
>> and be prepared for the future.
>> 
>> The various options and back-of-the-envelope estimates we came up with
>> are as follows:
>> 
>>  1. Buying and hosting hardware:
>>      250k€ for hardware
>>      3k€/month (36k€/year)
>> 
>>  2. Renting machines (e.g., on Hetzner):
>>      6k€/month (72k€/year)
>> 
>>  3. Sponsored:
>>      get hardware and/or hosting sponsored (by academic institutions or
>>      companies).
>> 
>> Option #1 gives us “full control”, the downside being that it’s a lot of
>> work and a real burden (get crowdfunding for the initial funding, later
>> on to sustain funding to cover hosting, ensure Guix Foundation is up to
>> the task of managing the assets, and of course to take care of the
>> machines for their entire lifecycle).
>> 
>> Option #2 gives us less control (we don’t know exactly what hardware is
>> being used and have to trust the company hosting the machines).  The
>> upside is that it’s much less work over time (the company is responsible
>> for upgrading hardware) and less work initially (no need to raise as
>> much money to buy hardware).
>> 
>> Option #3 potentially gives less control (depending on the project’s
>> relation with the hosting organization) and makes the project dependent
>> on the sponsor and/or person(s) in touch with them.  On the upside, it
>> could significantly reduce costs (potentially to 0€).
>> 
>> 
>> This is an important topic for the project, one we should plan for:
>> socially, financially, technically.  This takes time, which is why
>> preparation is needed.
>> 
>> What do people think?
>> 
>> Ludo’ & co.
> 
> Looking at Hetzner, they have an option to rent a dedicated ARM server
> with 80 cores/threads with 256GB of RAM and 2x3.84 TB NVMe drives for
> under €300/month and a €94 setup charge. Correct me if I"m wrong, but
> that one box is ~20x our current active aarch64/armv7 capacity.
> 
> Also looking at our current infrastructure at MDC, part of the reason we
> have so many x86_64 machines is because that's what was bought with the
> donated money, not because we actually needed quite that many, so some
> of the numbers might be higher than we actually need.
> 
> --
> Efraim Flashner   <efraim@flashner.co.il>   רנשלפ םירפא
> GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
> Confidentiality cannot be guaranteed on emails sent or received unencrypted
> <signature.asc>

Re: Sustainable funding and maintenance for our infrastructure

[Vincent Legoll]

[-- Attachment #1: Type: text/plain, Size: 990 bytes --]

Hello,

On Mon, Jul 8, 2024 at 3:47 PM Vagrant Cascadian <vagrant@debian.org> wrote:

> This may be a little wild, but what are the downsides to doing some
> combination of all of the above?
>
> A mixed strategy could reduce ... the upfront cost of buying and hosting
> hardware (#1), the ongoing costs of renting (#2), and dependence on the
> generosity of a third party for sponsored hardware & hosting (#3).
>
> It seems like any strategy should have some redundancy (e.g. multiple
> independent build farms) so that a failure in one datacenter does not
> effectively take down the whole network...
>

That would be my opinion too.

But for the cloud renting I would first research if there are associated
network or other costs, because the computing is cheap only to lure
you into the (sometimes prohibitive) hidden costs.


> ... though they are not full replacements for each other.
>

Maybe that should be treated as a bug/issue.

-- 
Vincent Legoll

[-- Attachment #2: Type: text/html, Size: 1641 bytes --]

Re: Sustainable funding and maintenance for our infrastructure

[Tomas Volf]

[-- Attachment #1: Type: text/plain, Size: 3421 bytes --]

On 2024-07-08 18:28:23 +0000, Vincent Legoll wrote:
> Hello,
>
> On Mon, Jul 8, 2024 at 3:47 PM Vagrant Cascadian <vagrant@debian.org> wrote:
>
> > This may be a little wild, but what are the downsides to doing some
> > combination of all of the above?
> >
> > A mixed strategy could reduce ... the upfront cost of buying and hosting
> > hardware (#1), the ongoing costs of renting (#2), and dependence on the
> > generosity of a third party for sponsored hardware & hosting (#3).
> >
> > It seems like any strategy should have some redundancy (e.g. multiple
> > independent build farms) so that a failure in one datacenter does not
> > effectively take down the whole network...
> >
>
> That would be my opinion too.
>
> But for the cloud renting I would first research if there are associated
> network or other costs,

Yes, there are other costs.  You in general pay for the egress used.  I have no
idea how much traffic does our current farm use.  I will speak of AWS, because
that is the one I know, but other will be likely similar.  If we would have
setup just in one "Availability Zone" (think data center), traffic between VMs
is free.  Traffic to the internet however is not.  First 100 GB is free each
month, and after that (source [0]):

First 10 TB / Month		$0.09 per GB
Next 40 TB / Month		$0.09 per GB
Next 100 TB / Month		$0.07 per GB
Greater than 150 TB / Month	$0.05 per GB

The prices differ by region a bit, but Europe and US are the "cheap" ones.

Interesting point is that someone can just decide to download a lot from you,
and *you* would pay for it.  It is nice way to drive up bill of someone you do
not like.

You also pay for storage and various other things.  Doing cost estimates in the
cloud is hard, because everything is complex and there are lot of options to
spend on.

Than there also is the "moral" side of the clouds.

> because the computing is cheap only to lure you into the (sometimes
> prohibitive) hidden costs.

Even the computing is not cheap by itself, if you just want compute, Hetzner is
cheaper.  Clouds give the nice things on top, like storage snapshots, backups,
... (ignoring the cost).  But, at least in my experience, not cost saving,
unless you radically re-design your application/stack to match the cloud
providers infrastructure.

Naive "lift and shift" migrations to the cloud can have tangible benefits, but
cost saving is rarely one of them.

Now, this all describes just actual "clouds".  Hetzner, for example, does not
charge for traffic (with some exceptions[1]).  You pay 2 EUR for public IPv4,
but other than that, I did not notice any unexpected charges on my invoices.

I personally think that renting actual physical servers might be a reasonable
idea.  One issue to consider is that (Hetzner in particular) has a history of
MitM attacks against their customers (assumption is that it was due to court
order, but it was never confirmed AFAIK).  So I would expect physical compromise
(signing keys) to be possible as well.  I have no idea how that would compare to
the security in the current hosting.

Have a nice day,
Tomas

0: https://www.cloudflare.com/learning/cloud/what-is-aws-data-transfer-pricing/
1: https://docs.hetzner.com/robot/general/traffic/

--
There are only two hard things in Computer Science:
cache invalidation, naming things and off-by-one errors.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Sustainable funding and maintenance for our infrastructure

[Simon Tournier]

Hi Ricardo, all,

I agree with many words in the thread.

Reading all the messages in the thread, the solution seems to try a mix
of the three options.  We could imagine “buy and host” more exotic
hardware but not a complete data center neither, “rent” for specific
needs and ask for more “sponsor”.

However, the implicit question is a chicken-or-the-egg problem: in order
to know what we would like to invest in, we need to know how much we
would be able to invest; and to know how much, we need to determine for
what.


On Mon, 08 Jul 2024 at 14:02, Ricardo Wurmus <rekado@elephly.net> wrote:

>>> The various options and back-of-the-envelope estimates we came up with
>>> are as follows:
>>>
>>>   1. Buying and hosting hardware:
>>>       250k€ for hardware
>>>       3k€/month (36k€/year)
>>>
>>>   2. Renting machines (e.g., on Hetzner):
>>>       6k€/month (72k€/year)
>>>
>>>   3. Sponsored:
>>>       get hardware and/or hosting sponsored (by academic institutions or
>>>       companies).

[...]

> Option #2 is rather quick to set up and quick to abandon should we run
> out of money.  It does, however, depend on continuous donations, which
> we are currently unable and possibly even unwilling to solicit.

IMHO, the discussion of any potential option should be coupled with a
“investment plan”.  One without the other is a dead end, for what my
opinion is worth.

Well, because I am in the business of academics, I am able to imagine
some path forward for option #3 (sponsor).  For instance, [1] appears to
me a very good news.  Then, I am a bit clueless* about financing option
#1 or #2 – whatever the amount.  I mean, the first question is how to
secure several k€ or k$ – how much would we be able to secure?

Let assume Guix Foundation would be able to manage the money, the
discussion becomes: considering the current incomings, how do we scale?

Is the project ready to regularly look for incomings and run after them?
From my point of view, it’s easier to get hardware and/or hosting
sponsored by academic institutions or companies than to run by our own
for incomings in order to buy hardware and/or pay hosting.  But as said,
I am biased here. :-)

Then if we have a plan for buying hardware – depending on the amount –
it means we also need a plan to deal with such asset; on the financial
side and on the ecological side.

All in all, the first question in order to prepare the more or less next
years of our infrastructure seems about what appears financially
feasible.  Drawing what is financially doable right now, what would be
doable with work or what is just impossible in the near future, drawing
all this picture appears to me the next move in the discussion.

Other said, with my Guix Foundation hat, my proposal could be to set
this as an objective: polish Guix Foundation for preparing to scale up.


*clueless: Not fully. :-)  For sure, we could opt for crowd-founding or
 some grant here or there.  But still… how much?

Cheers,
simon

1: New North American based Guix Substitute Server, cuirass.genenetwork.org Now Available
"Collin J. Doering" <collin@rekahsoft.ca>
Sat, 06 Jul 2024 19:35:44 -0400
id:871q46ytyn.fsf@rekahsoft.ca
https://lists.gnu.org/archive/html/guix-devel/2024-07
https://yhetil.org/guix/871q46ytyn.fsf@rekahsoft.ca

Re: Sustainable funding and maintenance for our infrastructure

[Ludovic Courtès]

Hello,

Simon Tournier <zimon.toutoune@gmail.com> skribis:

> However, the implicit question is a chicken-or-the-egg problem: in order
> to know what we would like to invest in, we need to know how much we
> would be able to invest; and to know how much, we need to determine for
> what.

We currently have ~50k USD at the FSF’s Working Together fund, that’s a
start.  Once we’ve settled on a plan, we can call for contributions from
individuals and organizations and scale accordingly.

Thanks,
Ludo’.

Re: Sustainable funding and maintenance for our infrastructure

[Ludovic Courtès]

Hi,

Efraim Flashner <efraim@flashner.co.il> skribis:

> Looking at Hetzner, they have an option to rent a dedicated ARM server
> with 80 cores/threads with 256GB of RAM and 2x3.84 TB NVMe drives for
> under €300/month and a €94 setup charge. Correct me if I"m wrong, but
> that one box is ~20x our current active aarch64/armv7 capacity.

Interesting.

> Also looking at our current infrastructure at MDC, part of the reason we
> have so many x86_64 machines is because that's what was bought with the
> donated money, not because we actually needed quite that many, so some
> of the numbers might be higher than we actually need.

It was bought by the MDC, under the perception that this matched our
needs (we don’t need as much “normally” but there are peaks with
branches like ‘core-updates’ that can only be absorbed if there’s enough
computing power behind).

Ludo’.

Re: Sustainable funding and maintenance for our infrastructure

[Ludovic Courtès]

Vagrant Cascadian <vagrant@debian.org> skribis:

> On 2024-07-02, Ludovic Courtès wrote:
>> We (Andreas, Chris, Ricardo, Romain, and myself) were having a
>> discussion about what it would take to set up a build farm similar to
>> what’s behind ci.guix: roughly 30 x86_64 servers, with 32-core/64-thread
>> CPUs and 128 GiB of RAM.  The reason for this discussion is that we were
>> thinking that we should not take our existing build farms for granted
>> and be prepared for the future.
>>
>> The various options and back-of-the-envelope estimates we came up with
>> are as follows:
>>
>>   1. Buying and hosting hardware:
>>       250k€ for hardware
>>       3k€/month (36k€/year)
>>
>>   2. Renting machines (e.g., on Hetzner):
>>       6k€/month (72k€/year)
>>
>>   3. Sponsored:
>>       get hardware and/or hosting sponsored (by academic institutions or
>>       companies).
>
> This may be a little wild, but what are the downsides to doing some
> combination of all of the above? Maybe higher bandwidth requirements
> between the various pieces of infrastructure presumably being hosted in
> different locations? Maybe also a little more complexity in the overall
> setup?

Good point.  In practice we’re already doing a combination of the above
and I agree that there are probably advantages to keep it that way.

My understanding is that Debian does #3 exclusively, with sponsorship
coming from a variety of organizations.

Ludo’.

Re: Sustainable funding and maintenance for our infrastructure

[Andreas Enge]

Hello Tomas,

Am Tue, Jul 09, 2024 at 11:47:38AM +0200 schrieb Tomas Volf:
> First 10 TB / Month		$0.09 per GB
> Next 40 TB / Month		$0.09 per GB
> Next 100 TB / Month		$0.07 per GB
> Greater than 150 TB / Month	$0.05 per GB

thanks for providing concrete figures!

I will continue without concrete figures and just back-of-the-envelope
computations. If I remember well (this is the very unconcrete point),
our throughput for bayfront at Aquilenet is about 100Mb/s. Some of it
is between bayfront and the bordeaux build machines (so would not count
if everything were in the same cloud), some of it is serving users (and
could stay on bayfront without being counted). After a complete
optimisation, every package would have to be transferred once out of the
cloud to the user facing server, while keeping all packages that are still
needed as inputs additionally in the cloud. I do not know how much this
would be; given that ideally every commit series should go through QA,
I think it could end up being quite a lot.

100Mb/s amounts to 32TB/month, which would cost about 3000€.

Compared to hosting, this is completely outrageous - notice that right
now at Aquilenet, we pay 75€ for one machine and not really limited
bandwidth.

After discussion with people at Aquilenet, I think it would be reasonable
to assume (no guarantee, also from a back-of-the-envelope computation)
we would need to pay about 100€/month for every additional 1U-machine we
would like to host in their rack. A small part of it is the rack space,
another small part the additional bandwidth, and the main part the
electricity cost. (Thus the estimate of 3000€/month for 30 machines.)

Andreas

Re: Sustainable funding and maintenance for our infrastructure

[Andreas Enge]

Hello Enno,

thanks for the pointer!

Am Mon, Jul 08, 2024 at 07:21:38PM +0200 schrieb Enrico Schwass:
> If you are looking for host services
> I have good experience with netcup.de.

Looking at their virtual "root" servers:
   https://www.netcup.eu/vserver/
the most interesting price point would be the offer of 12 cores, 32GB of RAM
and 1TB of SSD for 32€/month.
The most expensive offer has twice the cores, 4 times the RAM and 4 times
the SSD for 108€/month.

In all cases, it would be shared, whereas Hetzner offers dedicated
servers at a higher price, which is a decision to make.

> They also have ARM machines.

There they only offer "vServers" with "vCores"; I do not quite understand
what that means. A maximum number of cores, and when there are too many
people on a machine, we get fewer? Well, the FAQ says that we would not be
root on this type of virtual machine, which does not look like what we want.

Andreas

Re: Sustainable funding and maintenance for our infrastructure

[Felix Lechner via Development of GNU Guix and the GNU System distribution.]

Hi Andreas,

On Thu, Jul 11 2024, Andreas Enge wrote:

> Compared to hosting, this is completely outrageous

This isn't my place to speak up, but I hoped to support the notion that
AWS can be expensive.  It's most useful when sites must adapt quickly.

I heard of someone getting a $40,000 bill when their blog became famous.

Kind regards
Felix

Re: Sustainable funding and maintenance for our infrastructure

[Simon Tournier]

Hi,

On Thu, 11 Jul 2024 at 11:38, Ludovic Courtès <ludo@gnu.org> wrote:

> Good point.  In practice we’re already doing a combination of the above
> and I agree that there are probably advantages to keep it that way.

Well, from my perspective, the question is architecture per
architecture.  I mean, I think it’s doable to find sponsor (companies or
academic institutions) for x86_64 because it’s commonly used.  And
that’s not the case for less used others.

Other said, I think it would be a bad investment to buy x86_64 machines
when they could be donated and/or hosted.  However, it would be harder
for aarch64 or power9 or else, IMHO.

Cheers,
simon

Re: Sustainable funding and maintenance for our infrastructure

[Ludovic Courtès]

Simon Tournier <zimon.toutoune@gmail.com> skribis:

> Well, from my perspective, the question is architecture per
> architecture.  I mean, I think it’s doable to find sponsor (companies or
> academic institutions) for x86_64 because it’s commonly used.  And
> that’s not the case for less used others.

In the past we got donations from Arm:

  https://guix.gnu.org/en/blog/2018/aarch64-build-machines-donated/

I imagine the same could happen today for “alternative” architectures.

Ludo’.