November/December update on qa.guix.gnu.org and related things

November/December update on qa.guix.gnu.org and related things

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 1864 bytes --]

Hey!

Not much has changed since the last update. There's a new "waiting for
builds" status, I tweaked some code around build cancelation, I put in
place a mitigation for #67194 affecting the build coordinator, and did
some investigation of the hurd locale issue (#67507).

As previously set out, I'm planning to stop hosting the data service
instances this year. While I would like to stop hosting the server for
data.guix.gnu.org, I am going to keep that going at least for the time
being as that will allow me to test out pruning the bordeaux nars among
other things. I am still planning to shutdown data.qa.guix.gnu.org and
QA which depends on it within the next couple of weeks. I do hope it can
return some point though, and hopefully sooner rather than later.

On this like most decisions I'm indecisive, I could try and keep the
current server going, but it's not the most cost effective setup and
it's very low on disk space. I could replace the server with some
slightly better setup, but this would still mean I'm managing a key part
of the infrastructure, which is something I'm trying to move away
from. There was some discussion of the project taking over the hosting,
and maybe that will happen at some point, but it hasn't happened yet. So
while not having qa.guix.gnu.org for a time isn't ideal, I'm still going
with this approach.

Given that the first prototype setup came (and went I think) sometime
back in 2019 [1], this stuff has been going on for a while now. From
that prototyping over the last few years, I think the current software
setup is something that can work long term and be iterated on. To take
this technical setup forward though, it would ideally happen more
through the project and no longer have key parts be reliant on me for
hosting.

1: https://lists.gnu.org/archive/html/guix-devel/2019-02/msg00089.html

Thanks,

Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

Discontinuing data.guix.gnu.org?

[Ludovic Courtès]

Hello!

Christopher Baines <mail@cbaines.net> skribis:

> As previously set out, I'm planning to stop hosting the data service
> instances this year. While I would like to stop hosting the server for
> data.guix.gnu.org,

I forgot the outcome of previous discussions, but it seems to me that
the service itself and all the data it accumulated over the years are
super valuable.  I would be sad to see it go!

Is there something we can do to not lose it all?  It could be
distributing responsibility, reducing the scope, ensuring hosting is
managed collectively by the project, etc.  WDYT?

Ludo’.

Re: Discontinuing data.guix.gnu.org?

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 1594 bytes --]


Ludovic Courtès <ludo@gnu.org> writes:

> Hello!
>
> Christopher Baines <mail@cbaines.net> skribis:
>
>> As previously set out, I'm planning to stop hosting the data service
>> instances this year. While I would like to stop hosting the server for
>> data.guix.gnu.org,
>
> I forgot the outcome of previous discussions, but it seems to me that
> the service itself and all the data it accumulated over the years are
> super valuable.  I would be sad to see it go!

There was a discussion back in April, but no action came directly from
it.

Just having data.qa.guix.gnu.org was discussed, and at least
concentrating on getting to a sustainable hosting situation there seemed
like a sensible priority. The longer history provided by
data.guix.gnu.org does have value though in my view.

> Is there something we can do to not lose it all?  It could be
> distributing responsibility, reducing the scope, ensuring hosting is
> managed collectively by the project, etc.  WDYT?

Since that discussion, I have disabled the database dumps and backups,
which has reduced (to 62€ per month) the hosting costs (although
obviously not having backups isn't ideal). It's possible to further
reduce the hosting costs as well by switching away from a VM to a
physical machine at Hetzner.

But yeah, given that having at least one data service instance is a key
part of keeping the bordeaux build farm running, managing the hosting
through the project seems to be the way to go. I'm just not sure how we
can get there, or what I can do to move things in that direction.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

Re: Discontinuing data.guix.gnu.org?

[Ludovic Courtès]

Hello Christopher,

Christopher Baines <mail@cbaines.net> skribis:

> Ludovic Courtès <ludo@gnu.org> writes:

[...]

>> Christopher Baines <mail@cbaines.net> skribis:
>>
>>> As previously set out, I'm planning to stop hosting the data service
>>> instances this year. While I would like to stop hosting the server for
>>> data.guix.gnu.org,
>>
>> I forgot the outcome of previous discussions, but it seems to me that
>> the service itself and all the data it accumulated over the years are
>> super valuable.  I would be sad to see it go!
>
> There was a discussion back in April, but no action came directly from
> it.
>
> Just having data.qa.guix.gnu.org was discussed, and at least
> concentrating on getting to a sustainable hosting situation there seemed
> like a sensible priority. The longer history provided by
> data.guix.gnu.org does have value though in my view.
>
>> Is there something we can do to not lose it all?  It could be
>> distributing responsibility, reducing the scope, ensuring hosting is
>> managed collectively by the project, etc.  WDYT?
>
> Since that discussion, I have disabled the database dumps and backups,
> which has reduced (to 62€ per month) the hosting costs (although
> obviously not having backups isn't ideal). It's possible to further
> reduce the hosting costs as well by switching away from a VM to a
> physical machine at Hetzner.
>
> But yeah, given that having at least one data service instance is a key
> part of keeping the bordeaux build farm running, managing the hosting
> through the project seems to be the way to go. I'm just not sure how we
> can get there, or what I can do to move things in that direction.

Like you, I would have hoped for more reactions.  Unfortunately, I’m not
offering to help here because I have more than enough on my plate and
even a 1+ month backlog on guix-devel…

Maintainers, what’s your take on this?

Guix Foundation, is there any blocker to taking responsibility for
covering hosting expenses, possibly rediscussing the scope and cost?

Let 2024 be a thriving Guix year! :-)

Ludo’.

Re: Discontinuing data.guix.gnu.org?

[Julien Lepiller]

In terms of finance, our last blocker is that I still don't have access to the account, but Andreas does, so we should be able to take responsibility for the cost relatively easily.

Ideally, we would take ownership of the machine(s), so we don't overcomplicate our finances by having to reimburse someone regularly.

Le 9 janvier 2024 19:12:44 GMT+01:00, "Ludovic Courtès" <ludo@gnu.org> a écrit :
>Hello Christopher,
>
>Christopher Baines <mail@cbaines.net> skribis:
>
>> Ludovic Courtès <ludo@gnu.org> writes:
>
>[...]
>
>>> Christopher Baines <mail@cbaines.net> skribis:
>>>
>>>> As previously set out, I'm planning to stop hosting the data service
>>>> instances this year. While I would like to stop hosting the server for
>>>> data.guix.gnu.org,
>>>
>>> I forgot the outcome of previous discussions, but it seems to me that
>>> the service itself and all the data it accumulated over the years are
>>> super valuable.  I would be sad to see it go!
>>
>> There was a discussion back in April, but no action came directly from
>> it.
>>
>> Just having data.qa.guix.gnu.org was discussed, and at least
>> concentrating on getting to a sustainable hosting situation there seemed
>> like a sensible priority. The longer history provided by
>> data.guix.gnu.org does have value though in my view.
>>
>>> Is there something we can do to not lose it all?  It could be
>>> distributing responsibility, reducing the scope, ensuring hosting is
>>> managed collectively by the project, etc.  WDYT?
>>
>> Since that discussion, I have disabled the database dumps and backups,
>> which has reduced (to 62€ per month) the hosting costs (although
>> obviously not having backups isn't ideal). It's possible to further
>> reduce the hosting costs as well by switching away from a VM to a
>> physical machine at Hetzner.
>>
>> But yeah, given that having at least one data service instance is a key
>> part of keeping the bordeaux build farm running, managing the hosting
>> through the project seems to be the way to go. I'm just not sure how we
>> can get there, or what I can do to move things in that direction.
>
>Like you, I would have hoped for more reactions.  Unfortunately, I’m not
>offering to help here because I have more than enough on my plate and
>even a 1+ month backlog on guix-devel…
>
>Maintainers, what’s your take on this?
>
>Guix Foundation, is there any blocker to taking responsibility for
>covering hosting expenses, possibly rediscussing the scope and cost?
>
>Let 2024 be a thriving Guix year! :-)
>
>Ludo’.
>

Re: Discontinuing data.guix.gnu.org?

[Gábor Boskovits]

[-- Attachment #1: Type: text/plain, Size: 3206 bytes --]

Hello,

Julien Lepiller <julien@lepiller.eu> ezt írta (időpont: 2024. jan. 9., K
19:33):

> In terms of finance, our last blocker is that I still don't have access to
> the account, but Andreas does, so we should be able to take responsibility
> for the cost relatively easily.
>
> Ideally, we would take ownership of the machine(s), so we don't
> overcomplicate our finances by having to reimburse someone regularly.
>
> Le 9 janvier 2024 19:12:44 GMT+01:00, "Ludovic Courtès" <ludo@gnu.org> a
> écrit :
> >Hello Christopher,
> >
> >Christopher Baines <mail@cbaines.net> skribis:
> >
> >> Ludovic Courtès <ludo@gnu.org> writes:
> >
> >[...]
> >
> >>> Christopher Baines <mail@cbaines.net> skribis:
> >>>
> >>>> As previously set out, I'm planning to stop hosting the data service
> >>>> instances this year. While I would like to stop hosting the server for
> >>>> data.guix.gnu.org,
> >>>
> >>> I forgot the outcome of previous discussions, but it seems to me that
> >>> the service itself and all the data it accumulated over the years are
> >>> super valuable.  I would be sad to see it go!
> >>
> >> There was a discussion back in April, but no action came directly from
> >> it.
> >>
> >> Just having data.qa.guix.gnu.org was discussed, and at least
> >> concentrating on getting to a sustainable hosting situation there seemed
> >> like a sensible priority. The longer history provided by
> >> data.guix.gnu.org does have value though in my view.
> >>
> >>> Is there something we can do to not lose it all?  It could be
> >>> distributing responsibility, reducing the scope, ensuring hosting is
> >>> managed collectively by the project, etc.  WDYT?
> >>
> >> Since that discussion, I have disabled the database dumps and backups,
> >> which has reduced (to 62€ per month) the hosting costs (although
> >> obviously not having backups isn't ideal). It's possible to further
> >> reduce the hosting costs as well by switching away from a VM to a
> >> physical machine at Hetzner.
>

If we decide to go down this path what would be a reasonable hardware
specification for the physical machines (I guess we are talking about at
least two, because of backup). Also in this case it might make sense to see
what else we could run on them. I don't want to hijack the thread, feel
free to start a new one for this line of conversation.

Regards, g_bor

> >>
> >> But yeah, given that having at least one data service instance is a key
> >> part of keeping the bordeaux build farm running, managing the hosting
> >> through the project seems to be the way to go. I'm just not sure how we
> >> can get there, or what I can do to move things in that direction.
> >
> >Like you, I would have hoped for more reactions.  Unfortunately, I’m not
> >offering to help here because I have more than enough on my plate and
> >even a 1+ month backlog on guix-devel…
> >
> >Maintainers, what’s your take on this?
> >
> >Guix Foundation, is there any blocker to taking responsibility for
> >covering hosting expenses, possibly rediscussing the scope and cost?
> >
> >Let 2024 be a thriving Guix year! :-)
> >
> >Ludo’.
> >
>
>

[-- Attachment #2: Type: text/html, Size: 4732 bytes --]

Re: Discontinuing data.guix.gnu.org?

[Maxim Cournoyer]

Hi,

Ludovic Courtès <ludo@gnu.org> writes:

> Hello Christopher,
>
> Christopher Baines <mail@cbaines.net> skribis:
>
>> Ludovic Courtès <ludo@gnu.org> writes:
>
> [...]
>
>>> Christopher Baines <mail@cbaines.net> skribis:
>>>
>>>> As previously set out, I'm planning to stop hosting the data service
>>>> instances this year. While I would like to stop hosting the server for
>>>> data.guix.gnu.org,
>>>
>>> I forgot the outcome of previous discussions, but it seems to me that
>>> the service itself and all the data it accumulated over the years are
>>> super valuable.  I would be sad to see it go!
>>
>> There was a discussion back in April, but no action came directly from
>> it.
>>
>> Just having data.qa.guix.gnu.org was discussed, and at least
>> concentrating on getting to a sustainable hosting situation there seemed
>> like a sensible priority. The longer history provided by
>> data.guix.gnu.org does have value though in my view.
>>
>>> Is there something we can do to not lose it all?  It could be
>>> distributing responsibility, reducing the scope, ensuring hosting is
>>> managed collectively by the project, etc.  WDYT?
>>
>> Since that discussion, I have disabled the database dumps and backups,
>> which has reduced (to 62€ per month) the hosting costs (although
>> obviously not having backups isn't ideal). It's possible to further
>> reduce the hosting costs as well by switching away from a VM to a
>> physical machine at Hetzner.
>>
>> But yeah, given that having at least one data service instance is a key
>> part of keeping the bordeaux build farm running, managing the hosting
>> through the project seems to be the way to go. I'm just not sure how we
>> can get there, or what I can do to move things in that direction.
>
> Like you, I would have hoped for more reactions.  Unfortunately, I’m not
> offering to help here because I have more than enough on my plate and
> even a 1+ month backlog on guix-devel…
>
> Maintainers, what’s your take on this?

I don't have a particularly clear view of the matter, but observing
that:

1. People find the service provides value (can someone restate what that
value is exactly?  Is it needed e.g. to power
https://qa.guix.gnu.org/patches ?)
2. Are willing to help with paying for it.

Given the above, it seems to me that the ball is in the camp of the Guix
Foundation to sort how to transfer ownership of the Hetzner
accounts/payments from Christopher to them.

It could be interesting, as Gabor mentioned, to review if a machine
already part of our fleet (Milano, perhaps?) could be used for it, to
keep our operating costs low, but this can be done later.

My 2 cents,

-- 
Thanks,
Maxim

Shutting down qa.guix?

[Ludovic Courtès]

Hello,

Christopher Baines <mail@cbaines.net> skribis:

> I am still planning to shutdown data.qa.guix.gnu.org and
> QA which depends on it within the next couple of weeks. I do hope it can
> return some point though, and hopefully sooner rather than later.
>
> On this like most decisions I'm indecisive, I could try and keep the
> current server going, but it's not the most cost effective setup and
> it's very low on disk space. I could replace the server with some
> slightly better setup, but this would still mean I'm managing a key part
> of the infrastructure, which is something I'm trying to move away
> from. There was some discussion of the project taking over the hosting,
> and maybe that will happen at some point, but it hasn't happened yet. So
> while not having qa.guix.gnu.org for a time isn't ideal, I'm still going
> with this approach.

I think this underlines a collective failure to get our act together.

I believe there’s consensus that qa.guix is useful and has been a boost
for reviewers and contributors; we’d probably all want it to provide
quicker feedback, which is a sign of success: we’ve come to rely on it.

I know this has been discussed several times and it remains unclear to
me why as a project we never managed to move forward—maybe the comfort
of the status quo?


Anyway, would it be possible for you to transfer billing of the hardware
(Hetzner?) to Guix Foundation?  Does Guix Foundation know what it would
cost them?

The “spending committee” (Tobias, Ricardo, and myself), which oversees
expenditure from the funds held at the FSF, can also be in the loop to
provide additional financial support.

As for system administration, is there documentation that people willing
to help could look at?  Very concrete things like: what services are
running on which machines, what do I do if one of them is stuck or if I
get this error message, etc.

Thanks,
Ludo’.

PS: You’ve done amazing work over the years.  As a contributor, I find
    it super reassuring to know that you’re always available and
    focused, committed to improving patch workflows for many years.
    It’s been a long road already and I admire this level of commitment
    and hard work.

Re: Shutting down qa.guix?

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 3273 bytes --]


Ludovic Courtès <ludo@gnu.org> writes:

> Hello,
>
> Christopher Baines <mail@cbaines.net> skribis:
>
>> I am still planning to shutdown data.qa.guix.gnu.org and
>> QA which depends on it within the next couple of weeks. I do hope it can
>> return some point though, and hopefully sooner rather than later.
>>
>> On this like most decisions I'm indecisive, I could try and keep the
>> current server going, but it's not the most cost effective setup and
>> it's very low on disk space. I could replace the server with some
>> slightly better setup, but this would still mean I'm managing a key part
>> of the infrastructure, which is something I'm trying to move away
>> from. There was some discussion of the project taking over the hosting,
>> and maybe that will happen at some point, but it hasn't happened yet. So
>> while not having qa.guix.gnu.org for a time isn't ideal, I'm still going
>> with this approach.
>
> I think this underlines a collective failure to get our act together.
>
> I believe there’s consensus that qa.guix is useful and has been a boost
> for reviewers and contributors; we’d probably all want it to provide
> quicker feedback, which is a sign of success: we’ve come to rely on it.
>
> I know this has been discussed several times and it remains unclear to
> me why as a project we never managed to move forward—maybe the comfort
> of the status quo?

In addition, it's also unclear to me who should be making decisions on
things like this.

I also think that fundamentally I may think that processes and tooling
to make changes is more important than others regard it to be. While it
has no inherent value to users, personally I see it as so much more
important than actual Guix features or packages since the value to users
comes through Guix getting better faster, because of the increased pace
of changes and reduced number of regressions.

> Anyway, would it be possible for you to transfer billing of the hardware
> (Hetzner?) to Guix Foundation?  Does Guix Foundation know what it would
> cost them?

I believe so, at least I think that's possible. The costs have also been
discussed previously.

> The “spending committee” (Tobias, Ricardo, and myself), which oversees
> expenditure from the funds held at the FSF, can also be in the loop to
> provide additional financial support.
>
> As for system administration, is there documentation that people willing
> to help could look at?  Very concrete things like: what services are
> running on which machines, what do I do if one of them is stuck or if I
> get this error message, etc.

The configuration for beid, the machine that runs data.qa.guix.gnu.org
and Patchwork is in maintenance.git. It could probably use some more
comments to provide some context for the configuration.

There's also probably a benefit from making some high level architecture
diagrams for QA and the bordeaux build farm, and I can try and make a
start on these.

As for monitoring and responding to problems, that's a bit more
complicated, but in most cases a herd restart of the relevant bit will
temporarily resolve the issue. I'm still working on mitigating some of
the underlying problems that cause things to break.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

Re: Shutting down qa.guix?

[Tobias Geerinckx-Rice]

[-- Attachment #1: Type: text/plain, Size: 1302 bytes --]

Hi Chris,

I agree that Guix should step in to maintain the level of service 
that QA currently offers, by paying for hosting and sharing 
responsibility for system administration.

Whether the software's maintained or improved is something over 
which we've historically had very poor control.  Things go awry 
when we pretend otherwise.

Christopher Baines 写道：
> it's not the most cost effective setup

Has this been explained in more detail before?

> I also think that fundamentally I may think that processes and 
> tooling
> to make changes is more important than others regard it to be.

The disproportionate amount of effort you've put in, mostly 
unaided, implies as much.

(Proportionally disproportional thanks for that, by the way.)

Both

> more comments to provide some context for the configuration.

and

> making some high level architecture diagrams for QA and the 
> bordeaux
> build farm

would be very much appreciated!

> As for monitoring and responding to problems, that's a bit more
> complicated, but in most cases a herd restart of the relevant 
> bit will
> temporarily resolve the issue.

Chuffed that the Guix Data Service embodies the same debugging 
philosophy as the other Guix infrastructure.

Kind regards,

T G-R

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 247 bytes --]

Re: Shutting down qa.guix?

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 703 bytes --]


Tobias Geerinckx-Rice <me@tobias.gr> writes:

> Christopher Baines 写道：
>> it's not the most cost effective setup
>
> Has this been explained in more detail before?

Probably not, beid is currently a CPX51 Hetzner cloud server costing
€65.33 a month. This has been useful as it's enabled scaling the
resources dynamically, but it would be possible to reduce the costs and
still have sufficient RAM/disk space by using a Hetzner server auction
machine for example.

It's not all about cost though, given the data service is one of the
slow points of QA, if we want QA to get faster at giving feedback, it's
probably important to not try and cut costs on this part of the system.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

Re: Shutting down qa.guix?

[Maxim Cournoyer]

Hi,

Christopher Baines <mail@cbaines.net> writes:

> Tobias Geerinckx-Rice <me@tobias.gr> writes:
>
>> Christopher Baines 写道：
>>> it's not the most cost effective setup
>>
>> Has this been explained in more detail before?
>
> Probably not, beid is currently a CPX51 Hetzner cloud server costing
> €65.33 a month. This has been useful as it's enabled scaling the
> resources dynamically, but it would be possible to reduce the costs and
> still have sufficient RAM/disk space by using a Hetzner server auction
> machine for example.
>
> It's not all about cost though, given the data service is one of the
> slow points of QA, if we want QA to get faster at giving feedback, it's
> probably important to not try and cut costs on this part of the system.

Isn't QA mostly slow because of the lack of x86 build machines?  Does
the head node needs to be powerful itself?  What kind of resources does
it likes having the most?  CPU?  RAM?  Storage?

-- 
Thanks,
Maxim

Re: Shutting down qa.guix?

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 1983 bytes --]


Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:

> Hi,
>
> Christopher Baines <mail@cbaines.net> writes:
>
>> Tobias Geerinckx-Rice <me@tobias.gr> writes:
>>
>>> Christopher Baines 写道：
>>>> it's not the most cost effective setup
>>>
>>> Has this been explained in more detail before?
>>
>> Probably not, beid is currently a CPX51 Hetzner cloud server costing
>> €65.33 a month. This has been useful as it's enabled scaling the
>> resources dynamically, but it would be possible to reduce the costs and
>> still have sufficient RAM/disk space by using a Hetzner server auction
>> machine for example.
>>
>> It's not all about cost though, given the data service is one of the
>> slow points of QA, if we want QA to get faster at giving feedback, it's
>> probably important to not try and cut costs on this part of the system.
>
> Isn't QA mostly slow because of the lack of x86 build machines?  Does
> the head node needs to be powerful itself?  What kind of resources does
> it likes having the most?  CPU?  RAM?  Storage?

There are two key bottlenecks, processing the revisions in the data
service, then the build coordinator performing the builds.

For the data service, lots of RAM helps as computing and building the
derivations for Guix (similar to pull, time-machine, ...) is quite
expensive in CPU and RAM. Also computing all the derivations for each
revision takes a lot of RAM.

Storage is also an issue as beid currently is working with 340G of total
storage and that's almost full, and this doesn't leave any space for
maintenance. More storage means being able to store data about more
patch series at once.

For the build coordinator, the machine doesn't need to be powerful, it
has quite low requirements. While bayfront's storage isn't particularly
fast, it's more than sufficient in terms of hardware. More build
machines, including x86 ones would speed up the test results for patches
and branches though.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

Re: Shutting down qa.guix?

[Simon Tournier]

Hi,

On sam., 09 déc. 2023 at 11:54, Ludovic Courtès <ludo@gnu.org> wrote:

> I think this underlines a collective failure to get our act together.

I do not consider a collective failure considering the payment for the
service.  About the maintenance of such service, that’s another
question, IMHO. :-)


> Anyway, would it be possible for you to transfer billing of the hardware
> (Hetzner?) to Guix Foundation?  Does Guix Foundation know what it would
> cost them?

Just to put context about these questions for transferring billing:

 1. The initial discussion happened on the private mailing list
    guix-sysadmin.  Board of Guix Foundation had been CC on July 6th.

 2. Chris provided billing information on July 14th.  Then Andreas
    answered that it is sustainable for Guix Foundation, potentially
    with the help of Guix Spending Committee.

 3. On July 13th, the Solidary Administration Council of Guix Foundation
    had been solicited; as specified by Guix Foundation statutes.

 4. On September 12th, the SAC voted yes about supporting the financial
    cost of QA.  Moreover, since it is a recurrent cost, some means for
    collecting money had also been discussed.

 5. Then what has been unexpected is the energy that the Board of Guix
    Foundation invested in resolving an unrelated but blocking situation
    with our bank.  It probably slowed down all the operational part.

Well, from my understanding of the Guix Foundation side, all is ready
for transferring the current billing and, again from my understanding of
the situation, what is missing is the effective operational transfer.
Now, some relationships with our bank are smoothed… somehow. :-)


> The “spending committee” (Tobias, Ricardo, and myself), which oversees
> expenditure from the funds held at the FSF, can also be in the loop to
> provide additional financial support.

Since it is recurrent cost, yes it would nice to discuss between Guix
Foundation’s SAC and Spending Committee how to secure the financial
support of a recurrent cost.


> PS: You’ve done amazing work over the years.  As a contributor, I find
>     it super reassuring to know that you’re always available and
>     focused, committed to improving patch workflows for many years.
>     It’s been a long road already and I admire this level of commitment
>     and hard work.

I still remember the first time we met with Chris, it was back on
December 2018, right before Reproducible Build Summit in Paris.  Chris
already presented what could be done for improving the patch workflows.
It’s inspiring to see such commitment and hard work over the years.

Cheers,
simon

RFC (was Re: Shutting down qa.guix?)

[Simon Tournier]

Hi Ludo,

On sam., 09 déc. 2023 at 11:54, Ludovic Courtès <ludo@gnu.org> wrote:

> I know this has been discussed several times and it remains unclear to
> me why as a project we never managed to move forward—maybe the comfort
> of the status quo?

It would help if the project would have a process for such move.  Why
not some Request-For-Comment?

        Request-For-Comment process: concrete implementation
        Simon Tournier <zimon.toutoune@gmail.com>
        Tue, 31 Oct 2023 12:14:42 +0100
        id:87h6m7yrfh.fsf@gmail.com
        https://lists.gnu.org/archive/html/guix-devel/2023-10
        https://yhetil.org/guix/87h6m7yrfh.fsf@gmail.com

Cheers,
simon

Re: [Guix-europe-sac] Shutting down qa.guix?

[Andreas Enge]

Hello,

Am Sat, Dec 09, 2023 at 11:54:59AM +0100 schrieb Ludovic Courtès:
> I think this underlines a collective failure to get our act together.

indeed, and besides what Simon mentioned about the bank situation I think
there was a certain lack of consistency between deciding on the technical
and on the financial questions. And of course the lack of urgency, since
anyway things continued thanks to Chris... So thank you for all you have
done, Chris, and thank you for taking action now to force us to get our act
together! Indeed QA has become a central part of our project infrastructure.

I suggest the following, which resolves the lockstep between technology and
finance:
- Decide that the funds at the FSF pay for the hosting in its current form.
  Ideally move the billing to Guix Foundation, and then, as we use to do
  for bayfront, periodically ask the FSF to reimburse the hosting cost.
  I think we have an informal consensus for this, and just require a formal
  vote at the Guix spending committee and at the Guix Foundation SAC.
- Reimburse Chris for the costs incurred for hosting before this move.
  As Simon has said, we have a vote for this already, but could also
  start over with the exact amount once the first point is handled, so
  that Chris does not pay for future hosting any more.

Then in a separate step, other people can discuss about potential
technical changes to the hosting situation. It would probably be good
to have a small group of people, including Chris at least for a
transitory period, who take care of the sysadmin part.

Any thoughts on this proposal?

Andreas

Re: [Guix-europe-sac] Shutting down qa.guix?

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 1739 bytes --]


Andreas Enge <andreas@enge.fr> writes:

> Am Sat, Dec 09, 2023 at 11:54:59AM +0100 schrieb Ludovic Courtès:
>> I think this underlines a collective failure to get our act together.
>
> indeed, and besides what Simon mentioned about the bank situation I think
> there was a certain lack of consistency between deciding on the technical
> and on the financial questions. And of course the lack of urgency, since
> anyway things continued thanks to Chris... So thank you for all you have
> done, Chris, and thank you for taking action now to force us to get our act
> together! Indeed QA has become a central part of our project infrastructure.
>
> I suggest the following, which resolves the lockstep between technology and
> finance:
> - Decide that the funds at the FSF pay for the hosting in its current form.
>   Ideally move the billing to Guix Foundation, and then, as we use to do
>   for bayfront, periodically ask the FSF to reimburse the hosting cost.
>   I think we have an informal consensus for this, and just require a formal
>   vote at the Guix spending committee and at the Guix Foundation SAC.
> - Reimburse Chris for the costs incurred for hosting before this move.
>   As Simon has said, we have a vote for this already, but could also
>   start over with the exact amount once the first point is handled, so
>   that Chris does not pay for future hosting any more.
>
> Then in a separate step, other people can discuss about potential
> technical changes to the hosting situation. It would probably be good
> to have a small group of people, including Chris at least for a
> transitory period, who take care of the sysadmin part.
>
> Any thoughts on this proposal?

Sounds good to me.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

Re: Shutting down qa.guix?

[Jing]

[-- Attachment #1.1: Type: text/plain, Size: 1657 bytes --]

Hi everyone,

Forgive me for cutting in a conversation like this. I believe qa.guix 
can still be salvaged: I can host it per gratis, at least temporarily.

I am the admin of repo.jing.rocks, and I own the hardware. The servers 
are in my bedroom/living room. Adding qa.guix would cost nothing for me.

 > As for system administration, is there documentation that people 
willing to help could look at?  Very concrete things like: what services 
are running on which machines, what do I do if one of them is stuck or 
if I get this error message, etc.

Yes, docs are important, and I couldn't find one. What kind of spec does 
qa.guix require to keep operational? I see in the threads:

 > Storage is also an issue as beid currently is working with 340G of 
total storage and that's almost full [..]

I can spare a few terabytes of spinning rust storage. How about RAM and 
CPU? Does 64 vcpus (AMD EPYC 7773X) and 128GB RAM sound enough? If 
anyone wants to know to full spec, I can write it later.

Please note that the hardware I own requires non-free firmware, will 
this be a problem? Also, I'm not a good programmer, I can't maintain any 
package. I'm still learning Guile syntax and how to adapt to Guix. 
Someone else would have to work with me to keep the whole system going. 
Will this be an issue?


For the stability of repo.jing.rocks, please refer to [1] and [2]:

[1] https://mirror-master.debian.org/status/mirror-info/repo.jing.rocks.html
[2] https://archlinux.org/mirrors/jing.rocks/

-- 
Jing Luo
About me: https://jing.rocks/about/
PGP Fingerprint: 4E09 8D19 00AA 3F72 1899 2614 09B3 316E 13A1 1EFC

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 236 bytes --]