Guix Day: Notes from the CI session

Guix Day: Notes from the CI session

[Ludovic Courtès]

[-- Attachment #1: Type: text/plain, Size: 131 bytes --]

Hello Guix!

Attached are the notes from the “continuous integration” session held a
few hours ago.

Thanks,
Ludo’.


[-- Attachment #2: notes --]
[-- Type: text/plain, Size: 2864 bytes --]


# Continous integration

## Latest changes in Cuirass

  - switch from sqlite to PostgreSQL
  - stop using the built-in offload mechanism
    - take inspiration from Hydra: offload straight from Cuirass to _workers_
    - head node talks to workers over ZMQ
    - head node feches builds from workers upon build completion, triggers "substitute baking" (for `guix publish`)
  - new [worker monitoring page](https://ci.guix.gnu.org/workers)
    - shows what each worker is doing
    - soon: will be able to view CPU/storage usage
  - build info now shows a _weather_ icon showing the build status evolution
    - per-specification RSS feeds allow users to monitor, say, `guix-master`
    - maybe soon: user accounts on the web interface
  - new _build priorities_, per spec (`guix-master`, `core-updates`, etc.)
  - `timeout` and `max-silent-time` properties of packages are now honored
  - integration with `guix pull`: `channel-with-available-substitutes`
  - many features in the works (see Mathieu's Org-mode file)
  
## Open issue: new machines

  - fast ARM servers available
  - criteria for hardware?
    - must run free system (stock Guix System)
  - hosting?
    - the MDC (in Berlin) wouldn't host Guix-specific non-x86 servers
    - could rent space (currently renting space for bayfront.guix in Bordeaux, France), but it's relatively costly
    - workstations could be in people's homes
      - https://store.avantek.co.uk/ampere-altra-64bit-arm-workstation.html
        - (benchmark info: https://www.phoronix.com/scan.php?page=article&item=ampere-altra-q80 - compete with AMD Epyc 7nm 2nd Gen)
      - 
    - small boards are unreliable or too slow
    - looking for volunteers to help
      - pick hardware we could buy
      - get in touch on `guix-devel` or `guix-sysadmin` (private) mailing lists

## Open issue: branching strategy

  - currently: building all of `master` + the "core" of `core-updates`
  - schedule
    - currently ad-hoc: volunteers get to choose when to freeze/merge
  - actions
    - pushes to `core-updates` should cancel all pending builds on that branch (see Guix Build Coordinator)
    - have a dashboard showing active branches, statuses (wild-west, frozen), and merge deadlines
      - could be implemented in Cuirass
      - specs could have extra properties (such as "merge deadline") that Cuirass would display
      - or it could use status info from the Data Service, as in https://data.guix-patches.cbaines.net/
    - naming convention: `core-updates-frozen` vs `core-updates`, etc.
    
 ## Open issue: missing CI features
  - be able to restart builds
  - detect build return codes: failed because of dependencies, failed because of timeout ...
  - short-term actions:
    - allow for more than two builds per worker on `berlin`
    - set a hard limit on the number of emulated builds by workers on `berlin`
  

Re: Guix Day: Notes from the CI session

[Leo Famulari]

On Mon, Feb 08, 2021 at 06:07:25PM +0100, Ludovic Courtès wrote:
> ## Open issue: new machines
> 
>   - fast ARM servers available
>   - criteria for hardware?
>     - must run free system (stock Guix System)
>   - hosting?
>     - the MDC (in Berlin) wouldn't host Guix-specific non-x86 servers
>     - could rent space (currently renting space for bayfront.guix in Bordeaux, France), but it's relatively costly
>     - workstations could be in people's homes
>       - https://store.avantek.co.uk/ampere-altra-64bit-arm-workstation.html
>         - (benchmark info: https://www.phoronix.com/scan.php?page=article&item=ampere-altra-q80 - compete with AMD Epyc 7nm 2nd Gen)
>       - 
>     - small boards are unreliable or too slow
>     - looking for volunteers to help
>       - pick hardware we could buy
>       - get in touch on `guix-devel` or `guix-sysadmin` (private) mailing lists

I think the status quo of 64-bit ARM for us is untenable. The emulated
builds cause mass failures that can't be reproduced on real hardware.
There is a growing demand for this platform among hobbyists and hackers
who we could convert to Guix contributors!

Regarding hosting, I heard objections to spending significant amounts of
money on it. But, there was also an objection to hosting the machines in
people's homes. Do we know anyone who could donate some rackspace? Or
office-space, if we acquire workstations instead of servers?

As for the hardware itself, yes, it's expensive, but not any more than
what we'd pay for our x86_64 build farm (~1500 CPU cores and terabytes
of RAM). Again, do we know anyone who would donate some? Should we try
negotiating a discount with vendors?

I don't think that using hobbyist SBCs is viable.  It doesn't work for
our armhf platform. Not to mention, the most powerful ones available are
merely dual-core old Cortex-A72 design. For purchasing build farm
hardware, we should look for Cortex-A76, at least, if not Neoverse N1.
Hobbyist SBCs are designed for low-power consumption, not performance;
they are basically re-using SOCs designed for TV boxes and embedded
applications.

Re: Guix Day: Notes from the CI session

[Leo Famulari]

On Mon, Feb 08, 2021 at 06:07:25PM +0100, Ludovic Courtès wrote:
> ## Open issue: branching strategy
> 
>   - currently: building all of `master` + the "core" of `core-updates`
>   - schedule
>     - currently ad-hoc: volunteers get to choose when to freeze/merge
>   - actions
>     - pushes to `core-updates` should cancel all pending builds on that branch (see Guix Build Coordinator)
>     - have a dashboard showing active branches, statuses (wild-west, frozen), and merge deadlines
>       - could be implemented in Cuirass
>       - specs could have extra properties (such as "merge deadline") that Cuirass would display
>       - or it could use status info from the Data Service, as in https://data.guix-patches.cbaines.net/
>     - naming convention: `core-updates-frozen` vs `core-updates`, etc.

So, does anyone want to change the naming conventions?

I like the idea of, in normal times, only pushing to "core-updates-next"
and "staging-next", and then freezing and building "core-updates" and
"staging".

branch naming conventions [was Re: Guix Day: Notes from the CI session]

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 1488 bytes --]

On Tue, Feb 09, 2021 at 04:46:36PM -0500, Leo Famulari wrote:
> On Mon, Feb 08, 2021 at 06:07:25PM +0100, Ludovic Courtès wrote:
> > ## Open issue: branching strategy
> > 
> >   - currently: building all of `master` + the "core" of `core-updates`
> >   - schedule
> >     - currently ad-hoc: volunteers get to choose when to freeze/merge
> >   - actions
> >     - pushes to `core-updates` should cancel all pending builds on that branch (see Guix Build Coordinator)
> >     - have a dashboard showing active branches, statuses (wild-west, frozen), and merge deadlines
> >       - could be implemented in Cuirass
> >       - specs could have extra properties (such as "merge deadline") that Cuirass would display
> >       - or it could use status info from the Data Service, as in https://data.guix-patches.cbaines.net/
> >     - naming convention: `core-updates-frozen` vs `core-updates`, etc.
> 
> So, does anyone want to change the naming conventions?
> 
> I like the idea of, in normal times, only pushing to "core-updates-next"
> and "staging-next", and then freezing and building "core-updates" and
> "staging".

My concern about that is that it basically swaps what we have now. Using
-frozen makes it a bit clearer that we're building it out.

-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: branch naming conventions [was Re: Guix Day: Notes from the CI session]

[Leo Famulari]

On Wed, Feb 10, 2021 at 04:09:15PM +0200, Efraim Flashner wrote:
> My concern about that is that it basically swaps what we have now. Using
> -frozen makes it a bit clearer that we're building it out.

Good idea, I like it.

Re: Guix Day: Notes from the CI session

[Mathieu Othacehe]

Hello,

> I think the status quo of 64-bit ARM for us is untenable. The emulated
> builds cause mass failures that can't be reproduced on real hardware.
> There is a growing demand for this platform among hobbyists and hackers
> who we could convert to Guix contributors!

Following discussions that happened on Monday, I limited the number of
Berlin workers emulating aarch64 builds by half (see
abfb17bfec0ba41399fb60e91aa0e98087fab469 on maintenance).

I also raised the number of Cuirass remote workers by machines to four
(see a5df328e5c833ca2d28105280484626bf4c1adb1 on maintenance).

Thanks to Ricardo support I was able to setup a Wireguard tunnel between
berlin and the overdrive1. It seems to work pretty well and
https://ci.guix.gnu.org/workers shows that it is building some packages.

I plan to connect the remaining overdrive machines, 2 at Tobias and 1 at
Andreas places. If this model of connecting build nodes through VPN
works sustainably, connecting more ARM workstations hosted at individual
could be a way to proceed.

WDYT?

Thanks,

Mathieu

Re: Guix Day: Notes from the CI session

[Leo Famulari]

On Wed, Feb 10, 2021 at 06:11:08PM +0100, Mathieu Othacehe wrote:
> Thanks to Ricardo support I was able to setup a Wireguard tunnel between
> berlin and the overdrive1. It seems to work pretty well and
> https://ci.guix.gnu.org/workers shows that it is building some packages.
> 
> I plan to connect the remaining overdrive machines, 2 at Tobias and 1 at
> Andreas places. If this model of connecting build nodes through VPN
> works sustainably, connecting more ARM workstations hosted at individual
> could be a way to proceed.
> 
> WDYT?

I think it's great! Let's get the other Overdrives running, and then
re-evaluate their capabilities.

Re: Guix Day: Notes from the CI session

[Ludovic Courtès]

Hi!

Mathieu Othacehe <othacehe@gnu.org> skribis:

>> I think the status quo of 64-bit ARM for us is untenable. The emulated
>> builds cause mass failures that can't be reproduced on real hardware.
>> There is a growing demand for this platform among hobbyists and hackers
>> who we could convert to Guix contributors!
>
> Following discussions that happened on Monday, I limited the number of
> Berlin workers emulating aarch64 builds by half (see
> abfb17bfec0ba41399fb60e91aa0e98087fab469 on maintenance).
>
> I also raised the number of Cuirass remote workers by machines to four
> (see a5df328e5c833ca2d28105280484626bf4c1adb1 on maintenance).
>
> Thanks to Ricardo support I was able to setup a Wireguard tunnel between
> berlin and the overdrive1. It seems to work pretty well and
> https://ci.guix.gnu.org/workers shows that it is building some packages.
>
> I plan to connect the remaining overdrive machines, 2 at Tobias and 1 at
> Andreas places. If this model of connecting build nodes through VPN
> works sustainably, connecting more ARM workstations hosted at individual
> could be a way to proceed.

Yay, thanks Ricardo & people at the MDC, and thank you!

In machines-for-berlin.scm, there’s also a couple of ARMv7 SBCs that,
while slow (no argument here), are helpful.  Do you think these could be
re-connected as well?

Thanks,
Ludo’.

Re: Guix Day: Notes from the CI session

[Ludovic Courtès]

Hi!

Leo Famulari <leo@famulari.name> skribis:

> As for the hardware itself, yes, it's expensive, but not any more than
> what we'd pay for our x86_64 build farm (~1500 CPU cores and terabytes
> of RAM). Again, do we know anyone who would donate some? Should we try
> negotiating a discount with vendors?

Yes, I think we should pick good-looking hardware like one of the
options listed in the notes of the CI session, and try to discuss
privately with the sales department of the company that makes it (with a
timeout, though).  They might be interested in providing hardware to
support free software development on the architecture, just like ARM
donated us two OverDrives three years ago.

Thanks,
Ludo’.

Re: branch naming conventions [was Re: Guix Day: Notes from the CI session]

[Ludovic Courtès]

Leo Famulari <leo@famulari.name> skribis:

> On Wed, Feb 10, 2021 at 04:09:15PM +0200, Efraim Flashner wrote:
>> My concern about that is that it basically swaps what we have now. Using
>> -frozen makes it a bit clearer that we're building it out.
>
> Good idea, I like it.

+1

Would someone be willing to add a couple of lines about this in the
manual, next to the place where branching is mentioned?

Ludo'.

Re: Guix Day: Notes from the CI session

[Andreas Enge]

Hello,

Am Tue, Feb 09, 2021 at 04:45:26PM -0500 schrieb Leo Famulari:
> Regarding hosting, I heard objections to spending significant amounts of
> money on it. But, there was also an objection to hosting the machines in
> people's homes. Do we know anyone who could donate some rackspace? Or
> office-space, if we acquire workstations instead of servers?

that is not quite what I objected :)  I think that if we buy many
not so expensive machines, we could host them at people's homes. I was
just a bit hesitant that individuals host a 10000€ machine. If there
is only one or two, I might be able to host them at my institute, however.

Andreas

Re: branch naming conventions [was Re: Guix Day: Notes from the CI session]

[Leo Famulari]

On Wed, Feb 10, 2021 at 10:49:04PM +0100, Ludovic Courtès wrote:
> Leo Famulari <leo@famulari.name> skribis:
> 
> > On Wed, Feb 10, 2021 at 04:09:15PM +0200, Efraim Flashner wrote:
> >> My concern about that is that it basically swaps what we have now. Using
> >> -frozen makes it a bit clearer that we're building it out.
> >
> > Good idea, I like it.
> 
> +1
> 
> Would someone be willing to add a couple of lines about this in the
> manual, next to the place where branching is mentioned?

Done in commit 175bea0ca42dba3e02109820efae0cac2804e73b

Re: Guix Day: Notes from the CI session

[Leo Famulari]

On Thu, Feb 11, 2021 at 10:04:43PM +0100, Andreas Enge wrote:
> that is not quite what I objected :)  I think that if we buy many
> not so expensive machines, we could host them at people's homes. I was
> just a bit hesitant that individuals host a 10000€ machine. If there
> is only one or two, I might be able to host them at my institute, however.

Oh, I understand more clearly now. I agree with your feelings on the
subject. Let's see what kind of hardware we are able to get, and then we
can start to find places to put it.

Re: branch naming conventions [was Re: Guix Day: Notes from the CI session]

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 991 bytes --]

On Thu, Feb 11, 2021 at 04:14:24PM -0500, Leo Famulari wrote:
> On Wed, Feb 10, 2021 at 10:49:04PM +0100, Ludovic Courtès wrote:
> > Leo Famulari <leo@famulari.name> skribis:
> > 
> > > On Wed, Feb 10, 2021 at 04:09:15PM +0200, Efraim Flashner wrote:
> > >> My concern about that is that it basically swaps what we have now. Using
> > >> -frozen makes it a bit clearer that we're building it out.
> > >
> > > Good idea, I like it.
> > 
> > +1
> > 
> > Would someone be willing to add a couple of lines about this in the
> > manual, next to the place where branching is mentioned?
> 
> Done in commit 175bea0ca42dba3e02109820efae0cac2804e73b

I don't think we still need to create the -next branch, staging and
staging-frozen seem like they should be enough.

-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Guix Day: Notes from the CI session

[Leo Famulari]

On Wed, Feb 10, 2021 at 06:11:08PM +0100, Mathieu Othacehe wrote:
> Thanks to Ricardo support I was able to setup a Wireguard tunnel between
> berlin and the overdrive1. It seems to work pretty well and
> https://ci.guix.gnu.org/workers shows that it is building some packages.

I noticed that the number of "slots" on overdrive1 was reduced to 1, at
least according to ci.guix.gnu.org/workers

I would have guessed that a single slot is appropriate for the machine,
but I'm curious what you saw that led to the change?

Re: Guix Day: Notes from the CI session

[Mathieu Othacehe]

Hey Leo,

> I would have guessed that a single slot is appropriate for the machine,
> but I'm curious what you saw that led to the change?

This is most likely due to a worker crash. Workers are removed from the
database when there are no signs from them since 120 seconds.

I'll try to investigate what caused it.

Mathieu

Re: Guix Day: Notes from the CI session

[Leo Famulari]

On Sun, Feb 14, 2021 at 09:42:46AM +0100, Mathieu Othacehe wrote:
> 
> Hey Leo,
> 
> > I would have guessed that a single slot is appropriate for the machine,
> > but I'm curious what you saw that led to the change?
> 
> This is most likely due to a worker crash. Workers are removed from the
> database when there are no signs from them since 120 seconds.
> 
> I'll try to investigate what caused it.

Ah, interesting. My first guess would be that memory was exhausted. It
has 8 GB RAM and 8 GB swap which should be enough for almost everything.