Call for contribution to the Guix infrastructure

Call for contribution to the Guix infrastructure

[Ludovic Courtès]

[-- Attachment #1: Type: text/plain, Size: 6814 bytes --]

Since its inception, the Guix project has always valued its autonomy,
and that reflects in its infrastructure: our servers run Guix System and
exclusively free software, none of them is hosted by one of these
transnational companies, and they’re administered by volunteers.

Of course this comes at a cost and this is why we’re sending this call
for contributions.  Our hope is to make infrastructure-related activity
more legible so that maybe you can picture yourself helping in one of
these areas.

  • Coding

    We run many Guix-specific services; this is all lovely Scheme code
    but it tends to receive less attention than Guix itself:

      Build Farm Front-End: https://git.cbaines.net/guix/bffe
      Cuirass: https://guix.gnu.org/cuirass/
      Goggles (IRC logger): https://git.savannah.gnu.org/cgit/guix/maintenance.git/tree/hydra/goggles.scm
      Guix Build Coordinator: https://git.savannah.gnu.org/cgit/guix/build-coordinator.git/
      Guix Data Service: https://git.savannah.gnu.org/git/guix/data-service.git/
      Guix Packages Website: https://codeberg.org/luis-felipe/guix-packages-website.git
      mumi: https://git.savannah.gnu.org/cgit/guix/mumi.git/
      nar-herder: https://git.savannah.gnu.org/cgit/guix/nar-herder.git/
      QA Frontpage: https://git.savannah.gnu.org/git/guix/qa-frontpage.git

    There is no time constraint on this coding activity: any improvement
    is welcome, whenever it comes.  Most of these code bases are
    relatively small, which should make it easier to get started.

    Prerequisites: Familiarity with Guile, HTTP, and databases.

    If you wish to get started, check out the README of the project of
    your choice and get in touch with guix-devel and the primary
    developer(s) of the tool as per ‘git shortlog -s | sort -k1 -n’.

  • System administration

    Guix System configuration for all our systems is held in this
    repository:

       https://git.savannah.gnu.org/cgit/guix/maintenance.git/tree/hydra/

    The two front-ends are berlin.scm (the machine behind
    ci.guix.gnu.org) and bayfront.scm (the machine behind
    bordeaux.guix.gnu.org, guix.gnu.org, hpc.guix.info, qa.guix.gnu.org,
    and more).  Both connect to a number of build machines and helpers.

    Without even having SSH access to the machine, you can help by
    posting patches to improve the configuration (you can test it with
    ‘guix system vm’).  Here are ways you can help:

      - Improve infra monitoring: set up a dashboard to monitor all the
        infrastructure, and an out-of-band channel to communicate about
        downtime.

      - Implement web site redundancy: guix.gnu.org should be backed by
        several machines on different sites.  Get in touch with us
        and/or send a patch!

      - Implement substitute redundancy: likewise, bordeaux.guix.gnu.org
        and ci.guix.gnu.org should be backed by several head nodes.

      - Improve backup: there’s currently ad-hoc backup of selected
        pieces over rsync between the two head nodes; we can improve on
        that, for example with a dedicated backup site and proper
        testing of recoverability.

      - Support mirroring: We’d like to make it easy for others to
        mirror substitutes from ci.guix and bordeaux.guix, perhaps by
        offering public rsync access.

      - Optimize our web services: Monitor the performance of our
        services and tweak nginx config or whatever it takes to improve
        it.

    There is no time constraint on this activity: any improvement is
    welcome, whenever you can work on it.

    Prerequisite: Familiarity with Guix System administration and
    ideally with the infrastructure handbook:
    <https://git.savannah.gnu.org/cgit/guix/maintenance.git/tree/doc/infra-handbook.org>.

  • Day-to-day system administration

    We’re also looking for people who’d be willing to have SSH access to
    some of the infrastructure to help with day-to-day maintenance:
    restarting a build, restarting the occasional service that has gone
    wild (that can happen :-)), reconfiguring/upgrading a machine,
    rebooting, etc.

    This day-to-day activity requires you to be available some of the
    time (during office hours or not, during the week-end or not),
    whenever is convenient for you, so you can react to issues reported
    on IRC, on the mailing list, or elsewhere, and synchronize with
    other sysadmins.

    Prerequisite: Being a “known” member of the community, familiarity
    with Guix System administration, with some of the services/web sites
    being run, and with the infrastructure handbook:
    <https://git.savannah.gnu.org/cgit/guix/maintenance.git/tree/doc/infra-handbook.org>.

  • On-site intervention

    The first front-end is currently generously hosted by the Max
    Delbrück Center (MDC), a research institute in Berlin, Germany.
    Only authorized personnel can physically access it.

    The second one, bordeaux.guix.gnu.org, is hosted in Bordeaux,
    France, in a professional data center shared with non-profit ISP
    Aquilenet.  If you live in the region of Bordeaux and would like to
    help out when we need to go on-site, please make yourself known by
    emailing guix-sysadmin@gnu.org.

    On-site interventions are rare, but they’re usually in response to
    an emergency.

  • Hosting

    We’re looking for people who can host machines and help out whenever
    physical access is needed.  More specifically:

      - We need hosting of “small” machines such as single-board
        computers (AArch64, RISC-V) for use as build machines.

      - We need hosting for front-ends and x86_64 build machines in a
        data center where they can be racked and where, ideally, several
        local Guix sysadmins can physically access them.

    The machines should be accessible over Wireguard VPN most of the
    time, so longer power or network interruptions should be the
    exception.

    Prerequisites: Familiarity with installing and remotely
    administering Guix System.

  • Administrative tasks

    The infra remains up and running thanks to crucial administrative
    tasks, which includes:

      - Selecting and purchasing hardware, for example build machines.

      - Renewing domain names.

      - Securing funding, in particular via the Guix Foundation:
        <https://foundation.guix.info>.

    Prerequisites: Familiarity with hardware, and/or DNS registrars,
    and/or sponsorship, and/or crowdfunding.

The more we are, the less any single one of us has to do!

Ludo’, on behalf of the infrastructure team.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 853 bytes --]

Re: Call for contribution to the Guix infrastructure

[Jonathan Brielmaier]

Hello Ludo,

Am 24.05.24 um 15:51 schrieb Ludovic Courtès:
>    • Hosting
>
>      We’re looking for people who can host machines and help out whenever
>      physical access is needed.  More specifically:
>
>        - We need hosting of “small” machines such as single-board
>          computers (AArch64, RISC-V) for use as build machines.
>
>        - We need hosting for front-ends and x86_64 build machines in a
>          data center where they can be racked and where, ideally, several
>          local Guix sysadmins can physically access them.
>
>      The machines should be accessible over Wireguard VPN most of the
>      time, so longer power or network interruptions should be the
>      exception.
>
>      Prerequisites: Familiarity with installing and remotely
>      administering Guix System.

I offered my help in hosting an ARM build machine (Softiron Overdrive)
at home to guix-sysadmin@gnu.org in March 2024.
But apart from Maxim's reply I did not receive an answer from the Guix
Sysadmins...

~Jonathan

Re: Call for contribution to the Guix infrastructure

[Ludovic Courtès]

Hi Jonathan,

Jonathan Brielmaier <jonathan.brielmaier@web.de> skribis:

> Am 24.05.24 um 15:51 schrieb Ludovic Courtès:
>>    • Hosting
>>
>>      We’re looking for people who can host machines and help out whenever
>>      physical access is needed.  More specifically:
>>
>>        - We need hosting of “small” machines such as single-board
>>          computers (AArch64, RISC-V) for use as build machines.
>>
>>        - We need hosting for front-ends and x86_64 build machines in a
>>          data center where they can be racked and where, ideally, several
>>          local Guix sysadmins can physically access them.
>>
>>      The machines should be accessible over Wireguard VPN most of the
>>      time, so longer power or network interruptions should be the
>>      exception.
>>
>>      Prerequisites: Familiarity with installing and remotely
>>      administering Guix System.
>
> I offered my help in hosting an ARM build machine (Softiron Overdrive)
> at home to guix-sysadmin@gnu.org in March 2024.
> But apart from Maxim's reply I did not receive an answer from the Guix
> Sysadmins...

Yes, we have a chicken-and-egg problem: the sysadmin team is too weak,
sometimes poorly organized, to the point that even offers for help might
not get replies.  But we can fix this: that’s the idea of that call for
contribution.

On this particular topic, half of the OverDrives currently have a home
and are up and running, modulo occasional hiccups or downtime while
waiting for spare parts and such.

The other half may be off-line: monokuma (Chris), sergei and dmitri
(Tobias).  We need to check the status, preferably on guix-sysadmin
(which is private) since that might involve discussing personal
information.  Help is welcome also at this very basic level!

Information about available machines (apart from the x86 machines at the
MDC) is available actually in several places (and of course, some might
be stale):

  https://git.savannah.gnu.org/cgit/guix/maintenance.git/tree/hydra/machines.rec
  https://git.savannah.gnu.org/cgit/guix/maintenance.git/tree/hydra/berlin.scm#n633
  https://foundation.guix.info/assets/

HTH!

Ludo’.

Re: Call for contribution to the Guix infrastructure

[Carlo Zancanaro]

Hi Ludo,

On Fri, May 24 2024, Ludovic Courtès wrote:
>   • Day-to-day system administration
>
>     We’re also looking for people who’d be willing to have SSH access to
>     some of the infrastructure to help with day-to-day maintenance:
>     restarting a build, restarting the occasional service that has gone
>     wild (that can happen :-)), reconfiguring/upgrading a machine,
>     rebooting, etc.

I think I could help with this. I can't guarantee that I'll always be
available, but I'm able and willing to help where I can.

>     This day-to-day activity requires you to be available some of the
>     time (during office hours or not, during the week-end or not),
>     whenever is convenient for you, so you can react to issues reported
>     on IRC, on the mailing list, or elsewhere, and synchronize with
>     other sysadmins.

My timezone has generally made it hard for me to be actively involved in
real-time conversations about Guix (I'm in Australia), so hopefully that
wouldn't be too much of an impediment here. It might even be an asset,
if you don't already have people available in this timezone.

>     Prerequisite: Being a “known” member of the community, familiarity
>     with Guix System administration, with some of the services/web sites
>     being run, and with the infrastructure handbook:
>     <https://git.savannah.gnu.org/cgit/guix/maintenance.git/tree/doc/infra-handbook.org>.

I'm not sure if I count as a "known" member of the community, but I have
been around the periphery for a while. My earliest commit that made it
into Guix is from 2016, and I made it to the Guix day in 2018. I have
been running my own personal infrastructure primarily on Guix since at
least 2018.

Carlo

Re: Call for contribution to the Guix infrastructure

[Ludovic Courtès]

Hello Carlo,

Carlo Zancanaro <carlo@zancanaro.id.au> skribis:

> On Fri, May 24 2024, Ludovic Courtès wrote:
>>   • Day-to-day system administration
>>
>>     We’re also looking for people who’d be willing to have SSH access to
>>     some of the infrastructure to help with day-to-day maintenance:
>>     restarting a build, restarting the occasional service that has gone
>>     wild (that can happen :-)), reconfiguring/upgrading a machine,
>>     rebooting, etc.
>
> I think I could help with this. I can't guarantee that I'll always be
> available, but I'm able and willing to help where I can.

[...]

> My timezone has generally made it hard for me to be actively involved in
> real-time conversations about Guix (I'm in Australia), so hopefully that
> wouldn't be too much of an impediment here. It might even be an asset,
> if you don't already have people available in this timezone.

Excellent.  If you’re fine with this, I/we can add you to the
guix-sysadmin mailing list, which is private (we can discuss
non-security-sensitive issues such as bugs in infra software here, of
course).

A sample of the kind of tasks we’ve had in recent days: ci.guix needing
close monitoring to ensure it smoothly processes builds, redeployment
needed because i586-gnu builders of ci.guix went unavailable due to
<https://issues.guix.gnu.org/69401> (trying to do that currently),
qa.guix throwing exceptions for some issues.

Often reporting an issue of that sort on IRC is already one step towards
a solution.

> I'm not sure if I count as a "known" member of the community, but I have
> been around the periphery for a while. My earliest commit that made it
> into Guix is from 2016, and I made it to the Guix day in 2018. I have
> been running my own personal infrastructure primarily on Guix since at
> least 2018.

You definitely count as a “known member”.  :-)

Thanks for volunteering!

Ludo’.

Re: Call for contribution to the Guix infrastructure

[Wilko Meyer]

Hi Ludo,

Ludovic Courtès <ludo@gnu.org> writes:
>       - Improve infra monitoring: set up a dashboard to monitor all the
>         infrastructure, and an out-of-band channel to communicate about
>         downtime.
...
>       - Support mirroring: We’d like to make it easy for others to
>         mirror substitutes from ci.guix and bordeaux.guix, perhaps by
>         offering public rsync access.
...
>     We’re also looking for people who’d be willing to have SSH access to
>     some of the infrastructure to help with day-to-day maintenance:
>     restarting a build, restarting the occasional service that has gone
>     wild (that can happen :-)), reconfiguring/upgrading a machine,
>     rebooting, etc.

I think I could spare a couple of hours per week (approx. 4-5h
most of the time realistically with availability in the earlier morning
hours and around the evening, my current timezone is CEST) to assist
with these tasks, especially with maintenance/sysadmin things and with
uptime/infrastructure monitoring if help's still needed for these things.

Even though:

>     Prerequisite: Being a “known” member of the community, familiarity
>     with Guix System administration, with some of the services/web sites
>     being run, and with the infrastructure handbook:
>     <https://git.savannah.gnu.org/cgit/guix/maintenance.git/tree/doc/infra-handbook.org>.

I've been only around for a bit more than a year in this community, so
it's up to others to decide wether I already count as a "known" member
or not.

-- 
Kind regards,

Wilko Meyer
w@wmeyer.eu

Call for contribution to the Guix infrastructure

[Zach Oglesby]

I would be happy to give a hand, but I am not sure that I qualify as a
known guix community memeber. I have been around a while and am also
active in the system crafters community. I also had access to Fedora
Infrastructure for years to work on docs build systems, etc.

I have been doing System Administration for over 20 years and am more
than capiable of following a runbook (or writing new ones).

Thanks,
Zach Oglesby

Re: Call for contribution to the Guix infrastructure

[Ludovic Courtès]

Hi Wilko,

Wilko Meyer <w@wmeyer.eu> skribis:

> Ludovic Courtès <ludo@gnu.org> writes:
>>       - Improve infra monitoring: set up a dashboard to monitor all the
>>         infrastructure, and an out-of-band channel to communicate about
>>         downtime.
> ...
>>       - Support mirroring: We’d like to make it easy for others to
>>         mirror substitutes from ci.guix and bordeaux.guix, perhaps by
>>         offering public rsync access.
> ...
>>     We’re also looking for people who’d be willing to have SSH access to
>>     some of the infrastructure to help with day-to-day maintenance:
>>     restarting a build, restarting the occasional service that has gone
>>     wild (that can happen :-)), reconfiguring/upgrading a machine,
>>     rebooting, etc.
>
> I think I could spare a couple of hours per week (approx. 4-5h
> most of the time realistically with availability in the earlier morning
> hours and around the evening, my current timezone is CEST) to assist
> with these tasks, especially with maintenance/sysadmin things and with
> uptime/infrastructure monitoring if help's still needed for these things.

Excellent!

To be clear there are really two different kinds of tasks here:

  1. Setting up a dashboard showing the status of all the Guix services
     (ci.guix, bordeaux.guix, guix.gnu.org, etc.)  It’s really about
     setting up whatever service can be used for that and coming up with
     appropriate Guix System config and a plan to deploy it.

  2. Everyday monitoring, which might involve ssh’ing into the machines,
     checking logs, restarting failed services, diagnosing problems,
     etc.

In both cases, we need to subscribe you to the guix-sysadmin mailing
list (low traffic).

For #2, we’ll want to give you SSH access to parts of the infra you’d
like to get involved in.  Please check out maintenance.git and send a
patch to guix-sysadmin adding your SSH key to berlin (ci.guix,
guixwl.org, etc.) or bayfront (bordeaux.guix, guix.gnu.org, etc.).

I’m happy to mentor you and other new volunteers.  Please do ping me on
#guix on Libera.chat (I’m civodul) or ask questions here or on
guix-sysadmin.

>>     Prerequisite: Being a “known” member of the community, familiarity
>>     with Guix System administration, with some of the services/web sites
>>     being run, and with the infrastructure handbook:
>>     <https://git.savannah.gnu.org/cgit/guix/maintenance.git/tree/doc/infra-handbook.org>.
>
> I've been only around for a bit more than a year in this community, so
> it's up to others to decide wether I already count as a "known" member
> or not.

You’ve been packaging the kernels we run for a while now, so to me
you’re eligible for SSH access to (part of) the infrastructure.

Thanks!

Ludo’.

Re: Call for contribution to the Guix infrastructure

[Ludovic Courtès]

Hi Zach,

Zach Oglesby <zach@ghostcorp.net> skribis:

> I would be happy to give a hand, but I am not sure that I qualify as a
> known guix community memeber. I have been around a while and am also
> active in the system crafters community. I also had access to Fedora
> Infrastructure for years to work on docs build systems, etc.
>
> I have been doing System Administration for over 20 years and am more
> than capiable of following a runbook (or writing new ones).

Unless you have meant with other members of the sysadmin or core
development team, I’d be reluctant to grant you SSH access to the infra
for now.

However, as mentioned in my lengthy message, some of the sysadmin tasks
can be performed without having SSH access, by virtue of having Guix
System configs publicly available in
<https://git.savannah.gnu.org/cgit/guix/maintenance.git/tree/hydra>.

Maybe this is an area where you can start contributing?

If you want, we can subscribe you to the guix-sysadmin mailing list so
you can follow what problems we face on a daily basis and find a place
where you can help.  Let us know!

Thank you,
Ludo’.

Re: Call for contribution to the Guix infrastructure

[Zach Oglesby]

On June 17, 2024 8:36:10 AM EDT, "Ludovic Courtès" <ludo@gnu.org> wrote:
>Hi Zach,
>
>Unless you have meant with other members of the sysadmin or core
>development team, I’d be reluctant to grant you SSH access to the infra
>for now.
>
>However, as mentioned in my lengthy message, some of the sysadmin tasks
>can be performed without having SSH access, by virtue of having Guix
>System configs publicly available in
><https://git.savannah.gnu.org/cgit/guix/maintenance.git/tree/hydra>.
>
>Maybe this is an area where you can start contributing?
>
>If you want, we can subscribe you to the guix-sysadmin mailing list so
>you can follow what problems we face on a daily basis and find a place
>where you can help.  Let us know!
>
>Thank you,
>Ludo’.

Totally get it. Just looking to help any way I can. Add me to the mailing list and I will see how I can help. 

Thanks,
Zach Oglesby