‘guix weather’ reports CI stats

‘guix weather’ reports CI stats

[Ludovic Courtès]

Hello Guix!

With commit 183445a6ed1cbac929ecb65303246945c8ccf39d, ‘guix weather’ can
now report continuous integration (CI) stats using the Hydra/Cuirass
HTTP interface:

--8<---------------cut here---------------start------------->8---
$ ./pre-inst-env guix weather --substitute-urls='https://berlin.guixsd.org https://hydra.gnu.org' -m ~/src/configuration/user-profile.scm
computing 352 package derivations for x86_64-linux...
looking for 421 store items on https://berlin.guixsd.org...
https://berlin.guixsd.org
  76.7% substitutes available (323 out of 421)
  753.4 MiB of nars (compressed)
  1,805.7 MiB on disk (uncompressed)
  0.000 seconds per request (0.2 seconds in total)
  2,740.3 requests per second

  0.0% (0 out of 98) of the missing items are queued
  965 queued builds
      aarch64-linux: 965 (100.0%)
  build rate: 23.41 builds per hour
      x86_64-linux: 11.16 builds per hour
      i686-linux: 6.03 builds per hour
      aarch64-linux: 6.41 builds per hour
looking for 421 store items on https://hydra.gnu.org...
https://hydra.gnu.org
  96.7% substitutes available (407 out of 421)
  1,296.8 MiB of nars (compressed)
  3,158.4 MiB on disk (uncompressed)
  0.001 seconds per request (0.5 seconds in total)
  808.5 requests per second

  867 queued builds
      x86_64-linux: 518 (59.7%)
      i686-linux: 221 (25.5%)
      armhf-linux: 128 (14.8%)
  build rate: 20.93 builds per hour
      i686-linux: 9.23 builds per hour
      armhf-linux: 8.75 builds per hour
      x86_64-linux: 2.95 builds per hour
--8<---------------cut here---------------end--------------->8---

In a way, this adds a forecasting capability to ‘guix weather’.

In this example we can also see that /api/queue?nr=10 returns
non-sensical data in the current Cuirass snapshot (the actual queue
contains lots of builds, and not only for aarch64!).

The build rates are also interesting and surprising.  On hydra, the low
x86_64 build rate is probably because x86_64 builds completed earlier,
so nothing happened lately.  On berlin, we have 20 Intel machines and
only 1 aarch64 machine, yet the Intel build rate is not 20 times that of
aarch64; I’m guessing we could do better.  :-)

Comments welcome!

Ludo’.

Re: ‘guix weather’ reports CI stats

[Danny Milosavljevic]

[-- Attachment #1: Type: text/plain, Size: 529 bytes --]

Hi Ludo,

very cool!

> In this example we can also see that /api/queue?nr=10 returns
> non-sensical data in the current Cuirass snapshot (the actual queue
> contains lots of builds, and not only for aarch64!).

Huh, well, the limit to 10 results is going to mess things up (you won't
see the majority of the entries, no?).

Otherwise you can try invoking

$ sqlite3 databasefile

and typing the select statement from database.scm:466 there.

Is it better there?  If so, maybe the grouping procedure has a bug.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: ‘guix weather’ reports CI stats

[Ludovic Courtès]

Heya Danny,

Danny Milosavljevic <dannym@scratchpost.org> skribis:

>> In this example we can also see that /api/queue?nr=10 returns
>> non-sensical data in the current Cuirass snapshot (the actual queue
>> contains lots of builds, and not only for aarch64!).
>
> Huh, well, the limit to 10 results is going to mess things up (you won't
> see the majority of the entries, no?).

The ordering was ignored for /api/queue.  This commit fixes it AFAICS:

  https://git.savannah.gnu.org/cgit/guix/guix-cuirass.git/commit/?id=8eefd24672d257e8bdfe7abe063da1d01d14d762

Ludo’.

Re: ‘guix weather’ reports CI stats

[Chris Marusich]

[-- Attachment #1: Type: text/plain, Size: 2444 bytes --]

ludo@gnu.org (Ludovic Courtès) writes:

> Hello Guix!
>
> With commit 183445a6ed1cbac929ecb65303246945c8ccf39d, ‘guix weather’ can
> now report continuous integration (CI) stats using the Hydra/Cuirass
> HTTP interface:
>
> $ ./pre-inst-env guix weather --substitute-urls='https://berlin.guixsd.org https://hydra.gnu.org' -m ~/src/configuration/user-profile.scm
> computing 352 package derivations for x86_64-linux...
> looking for 421 store items on https://berlin.guixsd.org...
> https://berlin.guixsd.org
>   76.7% substitutes available (323 out of 421)
>   753.4 MiB of nars (compressed)
>   1,805.7 MiB on disk (uncompressed)
>   0.000 seconds per request (0.2 seconds in total)
>   2,740.3 requests per second
>
>   0.0% (0 out of 98) of the missing items are queued
>   965 queued builds
>       aarch64-linux: 965 (100.0%)
>   build rate: 23.41 builds per hour
>       x86_64-linux: 11.16 builds per hour
>       i686-linux: 6.03 builds per hour
>       aarch64-linux: 6.41 builds per hour
> looking for 421 store items on https://hydra.gnu.org...
> https://hydra.gnu.org
>   96.7% substitutes available (407 out of 421)
>   1,296.8 MiB of nars (compressed)
>   3,158.4 MiB on disk (uncompressed)
>   0.001 seconds per request (0.5 seconds in total)
>   808.5 requests per second
>
>   867 queued builds
>       x86_64-linux: 518 (59.7%)
>       i686-linux: 221 (25.5%)
>       armhf-linux: 128 (14.8%)
>   build rate: 20.93 builds per hour
>       i686-linux: 9.23 builds per hour
>       armhf-linux: 8.75 builds per hour
>       x86_64-linux: 2.95 builds per hour
>
> In a way, this adds a forecasting capability to ‘guix weather’.

That is very cool!  It seems immediately useful for measuring the
performance of a build farm.

This shows us just a single snapshot in time.  It would be interesting
to have historical data, as well.  If Hydra or Cuirass logged metrics
(sometimes referred to as "telemetry" data) as it ran, it could be
useful for operating build farms and measuring improvements in their
performance.  There are a lot of great tools out there for analyzing
metrics / telemetry data, including fancy dashboarding software (e.g.,
Kibana and Grafana come to mind).  Does Guile have any existing
libraries for logging metrics in some kind of structured format that is
amenable to ad-hoc analysis or to ingestion into dashboarding software?

-- 
Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: ‘guix weather’ reports CI stats

[Ludovic Courtès]

Hello Chris,

Chris Marusich <cmmarusich@gmail.com> skribis:

> This shows us just a single snapshot in time.  It would be interesting
> to have historical data, as well.  If Hydra or Cuirass logged metrics
> (sometimes referred to as "telemetry" data) as it ran, it could be
> useful for operating build farms and measuring improvements in their
> performance.  There are a lot of great tools out there for analyzing
> metrics / telemetry data, including fancy dashboarding software (e.g.,
> Kibana and Grafana come to mind).  Does Guile have any existing
> libraries for logging metrics in some kind of structured format that is
> amenable to ad-hoc analysis or to ingestion into dashboarding software?

Not that I know of, but I agree it would be super useful!

Ludo’.