Sustainable funding and maintenance for our infrastructure

Sustainable funding and maintenance for our infrastructure

[Ludovic Courtès]

Hello Guix!

We (Andreas, Chris, Ricardo, Romain, and myself) were having a
discussion about what it would take to set up a build farm similar to
what’s behind ci.guix: roughly 30 x86_64 servers, with 32-core/64-thread
CPUs and 128 GiB of RAM.  The reason for this discussion is that we were
thinking that we should not take our existing build farms for granted
and be prepared for the future.

The various options and back-of-the-envelope estimates we came up with
are as follows:

  1. Buying and hosting hardware:
      250k€ for hardware
      3k€/month (36k€/year)

  2. Renting machines (e.g., on Hetzner):
      6k€/month (72k€/year)

  3. Sponsored:
      get hardware and/or hosting sponsored (by academic institutions or
      companies).

Option #1 gives us “full control”, the downside being that it’s a lot of
work and a real burden (get crowdfunding for the initial funding, later
on to sustain funding to cover hosting, ensure Guix Foundation is up to
the task of managing the assets, and of course to take care of the
machines for their entire lifecycle).

Option #2 gives us less control (we don’t know exactly what hardware is
being used and have to trust the company hosting the machines).  The
upside is that it’s much less work over time (the company is responsible
for upgrading hardware) and less work initially (no need to raise as
much money to buy hardware).

Option #3 potentially gives less control (depending on the project’s
relation with the hosting organization) and makes the project dependent
on the sponsor and/or person(s) in touch with them.  On the upside, it
could significantly reduce costs (potentially to 0€).


This is an important topic for the project, one we should plan for:
socially, financially, technically.  This takes time, which is why
preparation is needed.

What do people think?

Ludo’ & co.

Sustainable funding and maintenance for our infrastructure

[Ludovic Courtès]

Hello Guix!

We (Andreas, Chris, Ricardo, Romain, and myself) were having a
discussion about what it would take to set up a build farm similar to
what’s behind ci.guix: roughly 30 x86_64 servers, with 32-core/64-thread
CPUs and 128 GiB of RAM.  The reason for this discussion is that we were
thinking that we should not take our existing build farms for granted
and be prepared for the future.

The various options and back-of-the-envelope estimates we came up with
are as follows:

  1. Buying and hosting hardware:
      250k€ for hardware
      3k€/month (36k€/year)

  2. Renting machines (e.g., on Hetzner):
      6k€/month (72k€/year)

  3. Sponsored:
      get hardware and/or hosting sponsored (by academic institutions or
      companies).

Option #1 gives us “full control”, the downside being that it’s a lot of
work and a real burden (get crowdfunding for the initial funding, later
on to sustain funding to cover hosting, ensure Guix Foundation is up to
the task of managing the assets, and of course to take care of the
machines for their entire lifecycle).

Option #2 gives us less control (we don’t know exactly what hardware is
being used and have to trust the company hosting the machines).  The
upside is that it’s much less work over time (the company is responsible
for upgrading hardware) and less work initially (no need to raise as
much money to buy hardware).

Option #3 potentially gives less control (depending on the project’s
relation with the hosting organization) and makes the project dependent
on the sponsor and/or person(s) in touch with them.  On the upside, it
could significantly reduce costs (potentially to 0€).  If you have any
potential contacts, please get in touch with us (via the private
guix-sysadmin@gnu.org mailing list).


This is an important topic for the project, one we should plan for:
socially, financially, technically.  This takes time, which is why
preparation is needed.

What do people think?

Ludo’ & co.

Re: Sustainable funding and maintenance for our infrastructure

[indieterminacy]

Hello,

Its worth pointing out the work of OpenBSD Amsterdam - which has raised 
over €40k for its respective foundation.

Its approach is to donate €10 per VM and €15 per VM renewal and has 850 
VMs.

Here are details on its hardware:
https://openbsd.amsterdam/hardware.html

It references this:
> Dell PowerEdge R630 w/ 2 x Intel(R) Xeon(R) CPU E5-2667 0 @ 3.20GHz
> 384G RAM
> Dell PERC H730 Mini

Hopefully such a long established initiative can provide some 
benchmarking approaches and ideas.
The lead behind it is very accomidating and knowlegable.

It also is used as a mechanism for highlighting projects they host:
https://openbsd.amsterdam/runs.html

Kind regards,


Jonathan

On 2024-07-02 10:24, Ludovic Courtès wrote:
> Hello Guix!
> 
> We (Andreas, Chris, Ricardo, Romain, and myself) were having a
> discussion about what it would take to set up a build farm similar to
> what’s behind ci.guix: roughly 30 x86_64 servers, with 
> 32-core/64-thread
> CPUs and 128 GiB of RAM.  The reason for this discussion is that we 
> were
> thinking that we should not take our existing build farms for granted
> and be prepared for the future.
> 
> The various options and back-of-the-envelope estimates we came up with
> are as follows:
> 
>   1. Buying and hosting hardware:
>       250k€ for hardware
>       3k€/month (36k€/year)
> 
>   2. Renting machines (e.g., on Hetzner):
>       6k€/month (72k€/year)
> 
>   3. Sponsored:
>       get hardware and/or hosting sponsored (by academic institutions 
> or
>       companies).
> 
> Option #1 gives us “full control”, the downside being that it’s a lot 
> of
> work and a real burden (get crowdfunding for the initial funding, later
> on to sustain funding to cover hosting, ensure Guix Foundation is up to
> the task of managing the assets, and of course to take care of the
> machines for their entire lifecycle).
> 
> Option #2 gives us less control (we don’t know exactly what hardware is
> being used and have to trust the company hosting the machines).  The
> upside is that it’s much less work over time (the company is 
> responsible
> for upgrading hardware) and less work initially (no need to raise as
> much money to buy hardware).
> 
> Option #3 potentially gives less control (depending on the project’s
> relation with the hosting organization) and makes the project dependent
> on the sponsor and/or person(s) in touch with them.  On the upside, it
> could significantly reduce costs (potentially to 0€).
> 
> 
> This is an important topic for the project, one we should plan for:
> socially, financially, technically.  This takes time, which is why
> preparation is needed.
> 
> What do people think?
> 
> Ludo’ & co.

Re: Sustainable funding and maintenance for our infrastructure

[Simon Tournier]

Hi,

On Tue, 02 Jul 2024 at 16:24, Ludovic Courtès <ludo@gnu.org> wrote:

>                           The reason for this discussion is that we were
> thinking that we should not take our existing build farms for granted
> and be prepared for the future.

Could you explain the rationale?  I understand and fully agree that
sustainable funding and maintenance for infrastructure are key topics
for the project.  Do we need to move ci.guix soon?  Related to Ricardo
announcement [1]?

Well, I am missing some context or steps.  Currently, the project is
mainly in Option #3 (sponsored).  The main sponsor is MDC located in
Berlin.  The second sponsor is personal funds coupled to hardware bought
by us or donated to us – I have in mind the build farm behind the name
Bordeaux; thanks Chris! And the third sponsor – at some extent – is
Inria located in Bordeaux.

We had discussions about reinforcing the second sponsor by replacing
personal funds by project-wide funds, say Guix Foundation, community,
etc.

Is this description correct?


> The various options and back-of-the-envelope estimates we came up with
> are as follows:
>
>   1. Buying and hosting hardware:
>       250k€ for hardware
>       3k€/month (36k€/year)
>
>   2. Renting machines (e.g., on Hetzner):
>       6k€/month (72k€/year)
>
>   3. Sponsored:
>       get hardware and/or hosting sponsored (by academic institutions or
>       companies).

Well, on the paper, option #1 appears to me appealing but how do we get
this 250k€?  Somehow, 250k€ would mean being able to secure 3k€/month
for over almost 7 years, right?

Except if we have a large donation that I am not aware, I do not see how
it would be possible to sign in being sure to secure 3k€/month for over
almost 7 years; considering the project has 12 years.

Other said, option #1 does not appear to me an option.

Option #2 could be a temporary option for a short time.  But again,
that’s something.


> Option #3 potentially gives less control (depending on the project’s
> relation with the hosting organization) and makes the project dependent
> on the sponsor and/or person(s) in touch with them.  On the upside, it
> could significantly reduce costs (potentially to 0€).

It remains option #3. :-)

For me, that’s the only viable option at scale.  The main costs should
be covered by sponsor as academical ones.  From my point of view, the
only sustainable option is to group people behind GuixHPC (I recall the
domain name hpc.guix.info is paid by Guix Foundation ;-)) and ask: a) if
their institutions are ready to donate and/or a) if we could run for
some grants altogether.

Somehow, Guix starts to be run in various scientific data centers and we
could take advantage of this opportunity.

Indeed, it locks in some relation with the hosting organizations and/or
the person in touch with them.  That’s said, Ricardo showed it works
well – or at least it can. :-) The key appears to me to not put all the
eggs in the same basket.

That’s my half-baked current opinion.  WDYT?

Cheers,
simon


1: I'm retiring (for a while); help needed
Ricardo Wurmus <rekado@elephly.net>
Fri, 31 May 2024 08:08:15 +0200
id:87o78mldio.fsf@elephly.net
https://lists.gnu.org/archive/html/guix-devel/2024-05
https://yhetil.org/guix/87o78mldio.fsf@elephly.net