Re: Shutting down qa.guix?

Re: Shutting down qa.guix?

[Jing]

[-- Attachment #1.1: Type: text/plain, Size: 1657 bytes --]

Hi everyone,

Forgive me for cutting in a conversation like this. I believe qa.guix 
can still be salvaged: I can host it per gratis, at least temporarily.

I am the admin of repo.jing.rocks, and I own the hardware. The servers 
are in my bedroom/living room. Adding qa.guix would cost nothing for me.

 > As for system administration, is there documentation that people 
willing to help could look at?  Very concrete things like: what services 
are running on which machines, what do I do if one of them is stuck or 
if I get this error message, etc.

Yes, docs are important, and I couldn't find one. What kind of spec does 
qa.guix require to keep operational? I see in the threads:

 > Storage is also an issue as beid currently is working with 340G of 
total storage and that's almost full [..]

I can spare a few terabytes of spinning rust storage. How about RAM and 
CPU? Does 64 vcpus (AMD EPYC 7773X) and 128GB RAM sound enough? If 
anyone wants to know to full spec, I can write it later.

Please note that the hardware I own requires non-free firmware, will 
this be a problem? Also, I'm not a good programmer, I can't maintain any 
package. I'm still learning Guile syntax and how to adapt to Guix. 
Someone else would have to work with me to keep the whole system going. 
Will this be an issue?


For the stability of repo.jing.rocks, please refer to [1] and [2]:

[1] https://mirror-master.debian.org/status/mirror-info/repo.jing.rocks.html
[2] https://archlinux.org/mirrors/jing.rocks/

-- 
Jing Luo
About me: https://jing.rocks/about/
PGP Fingerprint: 4E09 8D19 00AA 3F72 1899 2614 09B3 316E 13A1 1EFC

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 236 bytes --]

November/December update on qa.guix.gnu.org and related things

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 1864 bytes --]

Hey!

Not much has changed since the last update. There's a new "waiting for
builds" status, I tweaked some code around build cancelation, I put in
place a mitigation for #67194 affecting the build coordinator, and did
some investigation of the hurd locale issue (#67507).

As previously set out, I'm planning to stop hosting the data service
instances this year. While I would like to stop hosting the server for
data.guix.gnu.org, I am going to keep that going at least for the time
being as that will allow me to test out pruning the bordeaux nars among
other things. I am still planning to shutdown data.qa.guix.gnu.org and
QA which depends on it within the next couple of weeks. I do hope it can
return some point though, and hopefully sooner rather than later.

On this like most decisions I'm indecisive, I could try and keep the
current server going, but it's not the most cost effective setup and
it's very low on disk space. I could replace the server with some
slightly better setup, but this would still mean I'm managing a key part
of the infrastructure, which is something I'm trying to move away
from. There was some discussion of the project taking over the hosting,
and maybe that will happen at some point, but it hasn't happened yet. So
while not having qa.guix.gnu.org for a time isn't ideal, I'm still going
with this approach.

Given that the first prototype setup came (and went I think) sometime
back in 2019 [1], this stuff has been going on for a while now. From
that prototyping over the last few years, I think the current software
setup is something that can work long term and be iterated on. To take
this technical setup forward though, it would ideally happen more
through the project and no longer have key parts be reliant on me for
hosting.

1: https://lists.gnu.org/archive/html/guix-devel/2019-02/msg00089.html

Thanks,

Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

Shutting down qa.guix?

[Ludovic Courtès]

Hello,

Christopher Baines <mail@cbaines.net> skribis:

> I am still planning to shutdown data.qa.guix.gnu.org and
> QA which depends on it within the next couple of weeks. I do hope it can
> return some point though, and hopefully sooner rather than later.
>
> On this like most decisions I'm indecisive, I could try and keep the
> current server going, but it's not the most cost effective setup and
> it's very low on disk space. I could replace the server with some
> slightly better setup, but this would still mean I'm managing a key part
> of the infrastructure, which is something I'm trying to move away
> from. There was some discussion of the project taking over the hosting,
> and maybe that will happen at some point, but it hasn't happened yet. So
> while not having qa.guix.gnu.org for a time isn't ideal, I'm still going
> with this approach.

I think this underlines a collective failure to get our act together.

I believe there’s consensus that qa.guix is useful and has been a boost
for reviewers and contributors; we’d probably all want it to provide
quicker feedback, which is a sign of success: we’ve come to rely on it.

I know this has been discussed several times and it remains unclear to
me why as a project we never managed to move forward—maybe the comfort
of the status quo?


Anyway, would it be possible for you to transfer billing of the hardware
(Hetzner?) to Guix Foundation?  Does Guix Foundation know what it would
cost them?

The “spending committee” (Tobias, Ricardo, and myself), which oversees
expenditure from the funds held at the FSF, can also be in the loop to
provide additional financial support.

As for system administration, is there documentation that people willing
to help could look at?  Very concrete things like: what services are
running on which machines, what do I do if one of them is stuck or if I
get this error message, etc.

Thanks,
Ludo’.

PS: You’ve done amazing work over the years.  As a contributor, I find
    it super reassuring to know that you’re always available and
    focused, committed to improving patch workflows for many years.
    It’s been a long road already and I admire this level of commitment
    and hard work.

Re: Shutting down qa.guix?

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 3273 bytes --]


Ludovic Courtès <ludo@gnu.org> writes:

> Hello,
>
> Christopher Baines <mail@cbaines.net> skribis:
>
>> I am still planning to shutdown data.qa.guix.gnu.org and
>> QA which depends on it within the next couple of weeks. I do hope it can
>> return some point though, and hopefully sooner rather than later.
>>
>> On this like most decisions I'm indecisive, I could try and keep the
>> current server going, but it's not the most cost effective setup and
>> it's very low on disk space. I could replace the server with some
>> slightly better setup, but this would still mean I'm managing a key part
>> of the infrastructure, which is something I'm trying to move away
>> from. There was some discussion of the project taking over the hosting,
>> and maybe that will happen at some point, but it hasn't happened yet. So
>> while not having qa.guix.gnu.org for a time isn't ideal, I'm still going
>> with this approach.
>
> I think this underlines a collective failure to get our act together.
>
> I believe there’s consensus that qa.guix is useful and has been a boost
> for reviewers and contributors; we’d probably all want it to provide
> quicker feedback, which is a sign of success: we’ve come to rely on it.
>
> I know this has been discussed several times and it remains unclear to
> me why as a project we never managed to move forward—maybe the comfort
> of the status quo?

In addition, it's also unclear to me who should be making decisions on
things like this.

I also think that fundamentally I may think that processes and tooling
to make changes is more important than others regard it to be. While it
has no inherent value to users, personally I see it as so much more
important than actual Guix features or packages since the value to users
comes through Guix getting better faster, because of the increased pace
of changes and reduced number of regressions.

> Anyway, would it be possible for you to transfer billing of the hardware
> (Hetzner?) to Guix Foundation?  Does Guix Foundation know what it would
> cost them?

I believe so, at least I think that's possible. The costs have also been
discussed previously.

> The “spending committee” (Tobias, Ricardo, and myself), which oversees
> expenditure from the funds held at the FSF, can also be in the loop to
> provide additional financial support.
>
> As for system administration, is there documentation that people willing
> to help could look at?  Very concrete things like: what services are
> running on which machines, what do I do if one of them is stuck or if I
> get this error message, etc.

The configuration for beid, the machine that runs data.qa.guix.gnu.org
and Patchwork is in maintenance.git. It could probably use some more
comments to provide some context for the configuration.

There's also probably a benefit from making some high level architecture
diagrams for QA and the bordeaux build farm, and I can try and make a
start on these.

As for monitoring and responding to problems, that's a bit more
complicated, but in most cases a herd restart of the relevant bit will
temporarily resolve the issue. I'm still working on mitigating some of
the underlying problems that cause things to break.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

Re: Shutting down qa.guix?

[Tobias Geerinckx-Rice]

[-- Attachment #1: Type: text/plain, Size: 1302 bytes --]

Hi Chris,

I agree that Guix should step in to maintain the level of service 
that QA currently offers, by paying for hosting and sharing 
responsibility for system administration.

Whether the software's maintained or improved is something over 
which we've historically had very poor control.  Things go awry 
when we pretend otherwise.

Christopher Baines 写道：
> it's not the most cost effective setup

Has this been explained in more detail before?

> I also think that fundamentally I may think that processes and 
> tooling
> to make changes is more important than others regard it to be.

The disproportionate amount of effort you've put in, mostly 
unaided, implies as much.

(Proportionally disproportional thanks for that, by the way.)

Both

> more comments to provide some context for the configuration.

and

> making some high level architecture diagrams for QA and the 
> bordeaux
> build farm

would be very much appreciated!

> As for monitoring and responding to problems, that's a bit more
> complicated, but in most cases a herd restart of the relevant 
> bit will
> temporarily resolve the issue.

Chuffed that the Guix Data Service embodies the same debugging 
philosophy as the other Guix infrastructure.

Kind regards,

T G-R

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 247 bytes --]

Re: Shutting down qa.guix?

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 703 bytes --]


Tobias Geerinckx-Rice <me@tobias.gr> writes:

> Christopher Baines 写道：
>> it's not the most cost effective setup
>
> Has this been explained in more detail before?

Probably not, beid is currently a CPX51 Hetzner cloud server costing
€65.33 a month. This has been useful as it's enabled scaling the
resources dynamically, but it would be possible to reduce the costs and
still have sufficient RAM/disk space by using a Hetzner server auction
machine for example.

It's not all about cost though, given the data service is one of the
slow points of QA, if we want QA to get faster at giving feedback, it's
probably important to not try and cut costs on this part of the system.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

Re: Shutting down qa.guix?

[Maxim Cournoyer]

Hi,

Christopher Baines <mail@cbaines.net> writes:

> Tobias Geerinckx-Rice <me@tobias.gr> writes:
>
>> Christopher Baines 写道：
>>> it's not the most cost effective setup
>>
>> Has this been explained in more detail before?
>
> Probably not, beid is currently a CPX51 Hetzner cloud server costing
> €65.33 a month. This has been useful as it's enabled scaling the
> resources dynamically, but it would be possible to reduce the costs and
> still have sufficient RAM/disk space by using a Hetzner server auction
> machine for example.
>
> It's not all about cost though, given the data service is one of the
> slow points of QA, if we want QA to get faster at giving feedback, it's
> probably important to not try and cut costs on this part of the system.

Isn't QA mostly slow because of the lack of x86 build machines?  Does
the head node needs to be powerful itself?  What kind of resources does
it likes having the most?  CPU?  RAM?  Storage?

-- 
Thanks,
Maxim

Re: Shutting down qa.guix?

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 1983 bytes --]


Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:

> Hi,
>
> Christopher Baines <mail@cbaines.net> writes:
>
>> Tobias Geerinckx-Rice <me@tobias.gr> writes:
>>
>>> Christopher Baines 写道：
>>>> it's not the most cost effective setup
>>>
>>> Has this been explained in more detail before?
>>
>> Probably not, beid is currently a CPX51 Hetzner cloud server costing
>> €65.33 a month. This has been useful as it's enabled scaling the
>> resources dynamically, but it would be possible to reduce the costs and
>> still have sufficient RAM/disk space by using a Hetzner server auction
>> machine for example.
>>
>> It's not all about cost though, given the data service is one of the
>> slow points of QA, if we want QA to get faster at giving feedback, it's
>> probably important to not try and cut costs on this part of the system.
>
> Isn't QA mostly slow because of the lack of x86 build machines?  Does
> the head node needs to be powerful itself?  What kind of resources does
> it likes having the most?  CPU?  RAM?  Storage?

There are two key bottlenecks, processing the revisions in the data
service, then the build coordinator performing the builds.

For the data service, lots of RAM helps as computing and building the
derivations for Guix (similar to pull, time-machine, ...) is quite
expensive in CPU and RAM. Also computing all the derivations for each
revision takes a lot of RAM.

Storage is also an issue as beid currently is working with 340G of total
storage and that's almost full, and this doesn't leave any space for
maintenance. More storage means being able to store data about more
patch series at once.

For the build coordinator, the machine doesn't need to be powerful, it
has quite low requirements. While bayfront's storage isn't particularly
fast, it's more than sufficient in terms of hardware. More build
machines, including x86 ones would speed up the test results for patches
and branches though.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

Re: Shutting down qa.guix?

[Simon Tournier]

Hi,

On sam., 09 déc. 2023 at 11:54, Ludovic Courtès <ludo@gnu.org> wrote:

> I think this underlines a collective failure to get our act together.

I do not consider a collective failure considering the payment for the
service.  About the maintenance of such service, that’s another
question, IMHO. :-)


> Anyway, would it be possible for you to transfer billing of the hardware
> (Hetzner?) to Guix Foundation?  Does Guix Foundation know what it would
> cost them?

Just to put context about these questions for transferring billing:

 1. The initial discussion happened on the private mailing list
    guix-sysadmin.  Board of Guix Foundation had been CC on July 6th.

 2. Chris provided billing information on July 14th.  Then Andreas
    answered that it is sustainable for Guix Foundation, potentially
    with the help of Guix Spending Committee.

 3. On July 13th, the Solidary Administration Council of Guix Foundation
    had been solicited; as specified by Guix Foundation statutes.

 4. On September 12th, the SAC voted yes about supporting the financial
    cost of QA.  Moreover, since it is a recurrent cost, some means for
    collecting money had also been discussed.

 5. Then what has been unexpected is the energy that the Board of Guix
    Foundation invested in resolving an unrelated but blocking situation
    with our bank.  It probably slowed down all the operational part.

Well, from my understanding of the Guix Foundation side, all is ready
for transferring the current billing and, again from my understanding of
the situation, what is missing is the effective operational transfer.
Now, some relationships with our bank are smoothed… somehow. :-)


> The “spending committee” (Tobias, Ricardo, and myself), which oversees
> expenditure from the funds held at the FSF, can also be in the loop to
> provide additional financial support.

Since it is recurrent cost, yes it would nice to discuss between Guix
Foundation’s SAC and Spending Committee how to secure the financial
support of a recurrent cost.


> PS: You’ve done amazing work over the years.  As a contributor, I find
>     it super reassuring to know that you’re always available and
>     focused, committed to improving patch workflows for many years.
>     It’s been a long road already and I admire this level of commitment
>     and hard work.

I still remember the first time we met with Chris, it was back on
December 2018, right before Reproducible Build Summit in Paris.  Chris
already presented what could be done for improving the patch workflows.
It’s inspiring to see such commitment and hard work over the years.

Cheers,
simon