Sustainable funding and maintenance for our infrastructure

Sustainable funding and maintenance for our infrastructure

[Ludovic Courtès]

Hello Guix!

We (Andreas, Chris, Ricardo, Romain, and myself) were having a
discussion about what it would take to set up a build farm similar to
what’s behind ci.guix: roughly 30 x86_64 servers, with 32-core/64-thread
CPUs and 128 GiB of RAM.  The reason for this discussion is that we were
thinking that we should not take our existing build farms for granted
and be prepared for the future.

The various options and back-of-the-envelope estimates we came up with
are as follows:

  1. Buying and hosting hardware:
      250k€ for hardware
      3k€/month (36k€/year)

  2. Renting machines (e.g., on Hetzner):
      6k€/month (72k€/year)

  3. Sponsored:
      get hardware and/or hosting sponsored (by academic institutions or
      companies).

Option #1 gives us “full control”, the downside being that it’s a lot of
work and a real burden (get crowdfunding for the initial funding, later
on to sustain funding to cover hosting, ensure Guix Foundation is up to
the task of managing the assets, and of course to take care of the
machines for their entire lifecycle).

Option #2 gives us less control (we don’t know exactly what hardware is
being used and have to trust the company hosting the machines).  The
upside is that it’s much less work over time (the company is responsible
for upgrading hardware) and less work initially (no need to raise as
much money to buy hardware).

Option #3 potentially gives less control (depending on the project’s
relation with the hosting organization) and makes the project dependent
on the sponsor and/or person(s) in touch with them.  On the upside, it
could significantly reduce costs (potentially to 0€).


This is an important topic for the project, one we should plan for:
socially, financially, technically.  This takes time, which is why
preparation is needed.

What do people think?

Ludo’ & co.

Re: Sustainable funding and maintenance for our infrastructure

[indieterminacy]

Hello,

Its worth pointing out the work of OpenBSD Amsterdam - which has raised 
over €40k for its respective foundation.

Its approach is to donate €10 per VM and €15 per VM renewal and has 850 
VMs.

Here are details on its hardware:
https://openbsd.amsterdam/hardware.html

It references this:
> Dell PowerEdge R630 w/ 2 x Intel(R) Xeon(R) CPU E5-2667 0 @ 3.20GHz
> 384G RAM
> Dell PERC H730 Mini

Hopefully such a long established initiative can provide some 
benchmarking approaches and ideas.
The lead behind it is very accomidating and knowlegable.

It also is used as a mechanism for highlighting projects they host:
https://openbsd.amsterdam/runs.html

Kind regards,


Jonathan

On 2024-07-02 10:24, Ludovic Courtès wrote:
> Hello Guix!
> 
> We (Andreas, Chris, Ricardo, Romain, and myself) were having a
> discussion about what it would take to set up a build farm similar to
> what’s behind ci.guix: roughly 30 x86_64 servers, with 
> 32-core/64-thread
> CPUs and 128 GiB of RAM.  The reason for this discussion is that we 
> were
> thinking that we should not take our existing build farms for granted
> and be prepared for the future.
> 
> The various options and back-of-the-envelope estimates we came up with
> are as follows:
> 
>   1. Buying and hosting hardware:
>       250k€ for hardware
>       3k€/month (36k€/year)
> 
>   2. Renting machines (e.g., on Hetzner):
>       6k€/month (72k€/year)
> 
>   3. Sponsored:
>       get hardware and/or hosting sponsored (by academic institutions 
> or
>       companies).
> 
> Option #1 gives us “full control”, the downside being that it’s a lot 
> of
> work and a real burden (get crowdfunding for the initial funding, later
> on to sustain funding to cover hosting, ensure Guix Foundation is up to
> the task of managing the assets, and of course to take care of the
> machines for their entire lifecycle).
> 
> Option #2 gives us less control (we don’t know exactly what hardware is
> being used and have to trust the company hosting the machines).  The
> upside is that it’s much less work over time (the company is 
> responsible
> for upgrading hardware) and less work initially (no need to raise as
> much money to buy hardware).
> 
> Option #3 potentially gives less control (depending on the project’s
> relation with the hosting organization) and makes the project dependent
> on the sponsor and/or person(s) in touch with them.  On the upside, it
> could significantly reduce costs (potentially to 0€).
> 
> 
> This is an important topic for the project, one we should plan for:
> socially, financially, technically.  This takes time, which is why
> preparation is needed.
> 
> What do people think?
> 
> Ludo’ & co.

Re: Sustainable funding and maintenance for our infrastructure

[Simon Tournier]

Hi,

On Tue, 02 Jul 2024 at 16:24, Ludovic Courtès <ludo@gnu.org> wrote:

>                           The reason for this discussion is that we were
> thinking that we should not take our existing build farms for granted
> and be prepared for the future.

Could you explain the rationale?  I understand and fully agree that
sustainable funding and maintenance for infrastructure are key topics
for the project.  Do we need to move ci.guix soon?  Related to Ricardo
announcement [1]?

Well, I am missing some context or steps.  Currently, the project is
mainly in Option #3 (sponsored).  The main sponsor is MDC located in
Berlin.  The second sponsor is personal funds coupled to hardware bought
by us or donated to us – I have in mind the build farm behind the name
Bordeaux; thanks Chris! And the third sponsor – at some extent – is
Inria located in Bordeaux.

We had discussions about reinforcing the second sponsor by replacing
personal funds by project-wide funds, say Guix Foundation, community,
etc.

Is this description correct?


> The various options and back-of-the-envelope estimates we came up with
> are as follows:
>
>   1. Buying and hosting hardware:
>       250k€ for hardware
>       3k€/month (36k€/year)
>
>   2. Renting machines (e.g., on Hetzner):
>       6k€/month (72k€/year)
>
>   3. Sponsored:
>       get hardware and/or hosting sponsored (by academic institutions or
>       companies).

Well, on the paper, option #1 appears to me appealing but how do we get
this 250k€?  Somehow, 250k€ would mean being able to secure 3k€/month
for over almost 7 years, right?

Except if we have a large donation that I am not aware, I do not see how
it would be possible to sign in being sure to secure 3k€/month for over
almost 7 years; considering the project has 12 years.

Other said, option #1 does not appear to me an option.

Option #2 could be a temporary option for a short time.  But again,
that’s something.


> Option #3 potentially gives less control (depending on the project’s
> relation with the hosting organization) and makes the project dependent
> on the sponsor and/or person(s) in touch with them.  On the upside, it
> could significantly reduce costs (potentially to 0€).

It remains option #3. :-)

For me, that’s the only viable option at scale.  The main costs should
be covered by sponsor as academical ones.  From my point of view, the
only sustainable option is to group people behind GuixHPC (I recall the
domain name hpc.guix.info is paid by Guix Foundation ;-)) and ask: a) if
their institutions are ready to donate and/or a) if we could run for
some grants altogether.

Somehow, Guix starts to be run in various scientific data centers and we
could take advantage of this opportunity.

Indeed, it locks in some relation with the hosting organizations and/or
the person in touch with them.  That’s said, Ricardo showed it works
well – or at least it can. :-) The key appears to me to not put all the
eggs in the same basket.

That’s my half-baked current opinion.  WDYT?

Cheers,
simon


1: I'm retiring (for a while); help needed
Ricardo Wurmus <rekado@elephly.net>
Fri, 31 May 2024 08:08:15 +0200
id:87o78mldio.fsf@elephly.net
https://lists.gnu.org/archive/html/guix-devel/2024-05
https://yhetil.org/guix/87o78mldio.fsf@elephly.net

Re: Sustainable funding and maintenance for our infrastructure

[Ricardo Wurmus]

Hi Simon,

> On Tue, 02 Jul 2024 at 16:24, Ludovic Courtès <ludo@gnu.org> wrote:
>
>>                           The reason for this discussion is that we were
>> thinking that we should not take our existing build farms for granted
>> and be prepared for the future.
>
> Could you explain the rationale?  I understand and fully agree that
> sustainable funding and maintenance for infrastructure are key topics
> for the project.  Do we need to move ci.guix soon?  Related to Ricardo
> announcement [1]?

There is no urgency.  The build farm at the MDC isn't going anywhere.

But it would be unwise for the project to assume that it will always
stay this way.  In the past we've also had some minor issues outside of
our immediate control that are attributable to hosting these servers at
a research institute, for example a trigger-happy firewall, or blanket
bans on large IP address ranges.

In the past we were given the opportunity to extend and upgrade the
build farm, but we cannot plan with good fortune like this.  As a
project it would be wise to continue our efforts to diversify our
distributed build farm.

>> The various options and back-of-the-envelope estimates we came up with
>> are as follows:
>>
>>   1. Buying and hosting hardware:
>>       250k€ for hardware
>>       3k€/month (36k€/year)
>>
>>   2. Renting machines (e.g., on Hetzner):
>>       6k€/month (72k€/year)
>>
>>   3. Sponsored:
>>       get hardware and/or hosting sponsored (by academic institutions or
>>       companies).
>
> Well, on the paper, option #1 appears to me appealing but how do we get
> this 250k€?  Somehow, 250k€ would mean being able to secure 3k€/month
> for over almost 7 years, right?
>
> Except if we have a large donation that I am not aware, I do not see how
> it would be possible to sign in being sure to secure 3k€/month for over
> almost 7 years; considering the project has 12 years.
>
> Other said, option #1 does not appear to me an option.

Correct.  I think it is a good reality check to see just how much value
there is (or was in 2019) in all these servers and what our realistic
options are to recreate this when eventually these machines are
decommissioned.  I don't see option #1 as realistic; not only is it a
prohibitively large up-front cost, it is also a serious continuous time
and money sink.  We'd also have to constantly play our cards well and
trade old hardware in for new hardware lest we are stuck with a metric
ton of e-waste.

> Option #2 could be a temporary option for a short time.  But again,
> that’s something.

I think option #2 is not actually terrible.  We like to say that the
cloud is just other people's machines, and our response to that is
aversion to a real or perceived loss of control.  But I'd like to put
this in perspective by asking how much control we *actually* have over
the build farm at the MDC right now.  In practice *I* have some
semblance of control over these machines because I have access to the
data centre.  For the most part, however, I treat these servers as warm
MDC furniture.

Yes, we'd lose a few more options when renting hardware via Hetzner (or
even the well-dressed monocled elephant over there: AWS), but I think we
should think carefully about how valuable our sacrifices are in exchange
for the practical advantages of not being stuck with a rack full of
industrial hardware.

Option #2 is rather quick to set up and quick to abandon should we run
out of money.  It does, however, depend on continuous donations, which
we are currently unable and possibly even unwilling to solicit.

-- 
Ricardo

Re: Sustainable funding and maintenance for our infrastructure

[Vagrant Cascadian]

[-- Attachment #1: Type: text/plain, Size: 1765 bytes --]

On 2024-07-02, Ludovic Courtès wrote:
> We (Andreas, Chris, Ricardo, Romain, and myself) were having a
> discussion about what it would take to set up a build farm similar to
> what’s behind ci.guix: roughly 30 x86_64 servers, with 32-core/64-thread
> CPUs and 128 GiB of RAM.  The reason for this discussion is that we were
> thinking that we should not take our existing build farms for granted
> and be prepared for the future.
>
> The various options and back-of-the-envelope estimates we came up with
> are as follows:
>
>   1. Buying and hosting hardware:
>       250k€ for hardware
>       3k€/month (36k€/year)
>
>   2. Renting machines (e.g., on Hetzner):
>       6k€/month (72k€/year)
>
>   3. Sponsored:
>       get hardware and/or hosting sponsored (by academic institutions or
>       companies).

This may be a little wild, but what are the downsides to doing some
combination of all of the above? Maybe higher bandwidth requirements
between the various pieces of infrastructure presumably being hosted in
different locations? Maybe also a little more complexity in the overall
setup?

A mixed strategy could reduce ... the upfront cost of buying and hosting
hardware (#1), the ongoing costs of renting (#2), and dependence on the
generosity of a third party for sponsored hardware & hosting (#3).

It seems like any strategy should have some redundancy (e.g. multiple
independent build farms) so that a failure in one datacenter does not
effectively take down the whole network...

In a sense, we already have some of that, with ci.guix.gnu.org and
bordeaux.guix.gnu.org, and also the new North American build farm
... though they are not full replacements for each other.

live well,
  vagrant

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

Re: Sustainable funding and maintenance for our infrastructure

[Vincent Legoll]

[-- Attachment #1: Type: text/plain, Size: 990 bytes --]

Hello,

On Mon, Jul 8, 2024 at 3:47 PM Vagrant Cascadian <vagrant@debian.org> wrote:

> This may be a little wild, but what are the downsides to doing some
> combination of all of the above?
>
> A mixed strategy could reduce ... the upfront cost of buying and hosting
> hardware (#1), the ongoing costs of renting (#2), and dependence on the
> generosity of a third party for sponsored hardware & hosting (#3).
>
> It seems like any strategy should have some redundancy (e.g. multiple
> independent build farms) so that a failure in one datacenter does not
> effectively take down the whole network...
>

That would be my opinion too.

But for the cloud renting I would first research if there are associated
network or other costs, because the computing is cheap only to lure
you into the (sometimes prohibitive) hidden costs.


> ... though they are not full replacements for each other.
>

Maybe that should be treated as a bug/issue.

-- 
Vincent Legoll

[-- Attachment #2: Type: text/html, Size: 1641 bytes --]

Re: Sustainable funding and maintenance for our infrastructure

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 2861 bytes --]

On Tue, Jul 02, 2024 at 04:24:06PM +0200, Ludovic Courtès wrote:
> Hello Guix!
> 
> We (Andreas, Chris, Ricardo, Romain, and myself) were having a
> discussion about what it would take to set up a build farm similar to
> what’s behind ci.guix: roughly 30 x86_64 servers, with 32-core/64-thread
> CPUs and 128 GiB of RAM.  The reason for this discussion is that we were
> thinking that we should not take our existing build farms for granted
> and be prepared for the future.
> 
> The various options and back-of-the-envelope estimates we came up with
> are as follows:
> 
>   1. Buying and hosting hardware:
>       250k€ for hardware
>       3k€/month (36k€/year)
> 
>   2. Renting machines (e.g., on Hetzner):
>       6k€/month (72k€/year)
> 
>   3. Sponsored:
>       get hardware and/or hosting sponsored (by academic institutions or
>       companies).
> 
> Option #1 gives us “full control”, the downside being that it’s a lot of
> work and a real burden (get crowdfunding for the initial funding, later
> on to sustain funding to cover hosting, ensure Guix Foundation is up to
> the task of managing the assets, and of course to take care of the
> machines for their entire lifecycle).
> 
> Option #2 gives us less control (we don’t know exactly what hardware is
> being used and have to trust the company hosting the machines).  The
> upside is that it’s much less work over time (the company is responsible
> for upgrading hardware) and less work initially (no need to raise as
> much money to buy hardware).
> 
> Option #3 potentially gives less control (depending on the project’s
> relation with the hosting organization) and makes the project dependent
> on the sponsor and/or person(s) in touch with them.  On the upside, it
> could significantly reduce costs (potentially to 0€).
> 
> 
> This is an important topic for the project, one we should plan for:
> socially, financially, technically.  This takes time, which is why
> preparation is needed.
> 
> What do people think?
> 
> Ludo’ & co.

Looking at Hetzner, they have an option to rent a dedicated ARM server
with 80 cores/threads with 256GB of RAM and 2x3.84 TB NVMe drives for
under €300/month and a €94 setup charge. Correct me if I"m wrong, but
that one box is ~20x our current active aarch64/armv7 capacity.

Also looking at our current infrastructure at MDC, part of the reason we
have so many x86_64 machines is because that's what was bought with the
donated money, not because we actually needed quite that many, so some
of the numbers might be higher than we actually need.

-- 
Efraim Flashner   <efraim@flashner.co.il>   רנשלפ םירפא
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Sustainable funding and maintenance for our infrastructure

[Enrico Schwass]

Hi Guix

If you are looking for host services
I have good experience with netcup.de. They also have ARM machines.
My 3 machines have a great uptime

Bye
Enno


> Am 2024/07/08 um 18:28 schrieb Efraim Flashner <efraim@flashner.co.il>:
> 
> On Tue, Jul 02, 2024 at 04:24:06PM +0200, Ludovic Courtès wrote:
>> Hello Guix!
>> 
>> We (Andreas, Chris, Ricardo, Romain, and myself) were having a
>> discussion about what it would take to set up a build farm similar to
>> what’s behind ci.guix: roughly 30 x86_64 servers, with 32-core/64-thread
>> CPUs and 128 GiB of RAM.  The reason for this discussion is that we were
>> thinking that we should not take our existing build farms for granted
>> and be prepared for the future.
>> 
>> The various options and back-of-the-envelope estimates we came up with
>> are as follows:
>> 
>>  1. Buying and hosting hardware:
>>      250k€ for hardware
>>      3k€/month (36k€/year)
>> 
>>  2. Renting machines (e.g., on Hetzner):
>>      6k€/month (72k€/year)
>> 
>>  3. Sponsored:
>>      get hardware and/or hosting sponsored (by academic institutions or
>>      companies).
>> 
>> Option #1 gives us “full control”, the downside being that it’s a lot of
>> work and a real burden (get crowdfunding for the initial funding, later
>> on to sustain funding to cover hosting, ensure Guix Foundation is up to
>> the task of managing the assets, and of course to take care of the
>> machines for their entire lifecycle).
>> 
>> Option #2 gives us less control (we don’t know exactly what hardware is
>> being used and have to trust the company hosting the machines).  The
>> upside is that it’s much less work over time (the company is responsible
>> for upgrading hardware) and less work initially (no need to raise as
>> much money to buy hardware).
>> 
>> Option #3 potentially gives less control (depending on the project’s
>> relation with the hosting organization) and makes the project dependent
>> on the sponsor and/or person(s) in touch with them.  On the upside, it
>> could significantly reduce costs (potentially to 0€).
>> 
>> 
>> This is an important topic for the project, one we should plan for:
>> socially, financially, technically.  This takes time, which is why
>> preparation is needed.
>> 
>> What do people think?
>> 
>> Ludo’ & co.
> 
> Looking at Hetzner, they have an option to rent a dedicated ARM server
> with 80 cores/threads with 256GB of RAM and 2x3.84 TB NVMe drives for
> under €300/month and a €94 setup charge. Correct me if I"m wrong, but
> that one box is ~20x our current active aarch64/armv7 capacity.
> 
> Also looking at our current infrastructure at MDC, part of the reason we
> have so many x86_64 machines is because that's what was bought with the
> donated money, not because we actually needed quite that many, so some
> of the numbers might be higher than we actually need.
> 
> -- 
> Efraim Flashner   <efraim@flashner.co.il>   רנשלפ םירפא
> GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
> Confidentiality cannot be guaranteed on emails sent or received unencrypted
> <signature.asc>


> Am 2024/07/08 um 18:28 schrieb Efraim Flashner <efraim@flashner.co.il>:
> 
> On Tue, Jul 02, 2024 at 04:24:06PM +0200, Ludovic Courtès wrote:
>> Hello Guix!
>> 
>> We (Andreas, Chris, Ricardo, Romain, and myself) were having a
>> discussion about what it would take to set up a build farm similar to
>> what’s behind ci.guix: roughly 30 x86_64 servers, with 32-core/64-thread
>> CPUs and 128 GiB of RAM.  The reason for this discussion is that we were
>> thinking that we should not take our existing build farms for granted
>> and be prepared for the future.
>> 
>> The various options and back-of-the-envelope estimates we came up with
>> are as follows:
>> 
>>  1. Buying and hosting hardware:
>>      250k€ for hardware
>>      3k€/month (36k€/year)
>> 
>>  2. Renting machines (e.g., on Hetzner):
>>      6k€/month (72k€/year)
>> 
>>  3. Sponsored:
>>      get hardware and/or hosting sponsored (by academic institutions or
>>      companies).
>> 
>> Option #1 gives us “full control”, the downside being that it’s a lot of
>> work and a real burden (get crowdfunding for the initial funding, later
>> on to sustain funding to cover hosting, ensure Guix Foundation is up to
>> the task of managing the assets, and of course to take care of the
>> machines for their entire lifecycle).
>> 
>> Option #2 gives us less control (we don’t know exactly what hardware is
>> being used and have to trust the company hosting the machines).  The
>> upside is that it’s much less work over time (the company is responsible
>> for upgrading hardware) and less work initially (no need to raise as
>> much money to buy hardware).
>> 
>> Option #3 potentially gives less control (depending on the project’s
>> relation with the hosting organization) and makes the project dependent
>> on the sponsor and/or person(s) in touch with them.  On the upside, it
>> could significantly reduce costs (potentially to 0€).
>> 
>> 
>> This is an important topic for the project, one we should plan for:
>> socially, financially, technically.  This takes time, which is why
>> preparation is needed.
>> 
>> What do people think?
>> 
>> Ludo’ & co.
> 
> Looking at Hetzner, they have an option to rent a dedicated ARM server
> with 80 cores/threads with 256GB of RAM and 2x3.84 TB NVMe drives for
> under €300/month and a €94 setup charge. Correct me if I"m wrong, but
> that one box is ~20x our current active aarch64/armv7 capacity.
> 
> Also looking at our current infrastructure at MDC, part of the reason we
> have so many x86_64 machines is because that's what was bought with the
> donated money, not because we actually needed quite that many, so some
> of the numbers might be higher than we actually need.
> 
> --
> Efraim Flashner   <efraim@flashner.co.il>   רנשלפ םירפא
> GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
> Confidentiality cannot be guaranteed on emails sent or received unencrypted
> <signature.asc>

Sustainable funding and maintenance for our infrastructure

[Ludovic Courtès]

Hello Guix!

We (Andreas, Chris, Ricardo, Romain, and myself) were having a
discussion about what it would take to set up a build farm similar to
what’s behind ci.guix: roughly 30 x86_64 servers, with 32-core/64-thread
CPUs and 128 GiB of RAM.  The reason for this discussion is that we were
thinking that we should not take our existing build farms for granted
and be prepared for the future.

The various options and back-of-the-envelope estimates we came up with
are as follows:

  1. Buying and hosting hardware:
      250k€ for hardware
      3k€/month (36k€/year)

  2. Renting machines (e.g., on Hetzner):
      6k€/month (72k€/year)

  3. Sponsored:
      get hardware and/or hosting sponsored (by academic institutions or
      companies).

Option #1 gives us “full control”, the downside being that it’s a lot of
work and a real burden (get crowdfunding for the initial funding, later
on to sustain funding to cover hosting, ensure Guix Foundation is up to
the task of managing the assets, and of course to take care of the
machines for their entire lifecycle).

Option #2 gives us less control (we don’t know exactly what hardware is
being used and have to trust the company hosting the machines).  The
upside is that it’s much less work over time (the company is responsible
for upgrading hardware) and less work initially (no need to raise as
much money to buy hardware).

Option #3 potentially gives less control (depending on the project’s
relation with the hosting organization) and makes the project dependent
on the sponsor and/or person(s) in touch with them.  On the upside, it
could significantly reduce costs (potentially to 0€).  If you have any
potential contacts, please get in touch with us (via the private
guix-sysadmin@gnu.org mailing list).


This is an important topic for the project, one we should plan for:
socially, financially, technically.  This takes time, which is why
preparation is needed.

What do people think?

Ludo’ & co.