bug#55848: [cuirass] workers stalled

[Tom Fitzhenry <tom@tom-fitzhenry.me.uk>]

Mathieu Othacehe <othacehe@gnu.org> writes:

Substitutes for aarch64 are a lot healthier now. Thanks Ludovic!

* kreuzberg is now successfully building and has been for a while.
* ci.guix.gnu.has has 41% of substitutes (a low percentage, but likely a
  high percentage of toolchains). 0 jobs are queued, presumably because Curiass
  believes its up-to-date. This should increase over time, as packages
  are updated.
* bordeaux has 83.8% of substitutes.

A few issues remain for aarch64:

* grunewald and kreuzberg are not on <https://ci.guix.gnu.org/workers>.
  Perhaps they were taken down while the substitute ratio was low to
  avoid each worker independently recompiling expensive toolchains?
* rust@1.39.0 (and thus all of Rust) is missing from ci and bordeaux. I
  had expected this would have been working. I'll take a look and raise
  a separate issue.

--8<---------------cut here---------------start------------->8---
$ ./pre-inst-env guix weather -s aarch64-linux -c2000
computing 15514 package derivations for aarch64-linux...
looking for 16265 store items on https://ci.guix.gnu.org...
https://ci.guix.gnu.org
  41.0% substitutes available (6668 out of 16265)
  at least 34188.1 MiB of nars (compressed)
  45362.5 MiB on disk (uncompressed)
  0.015 seconds per request (144.9 seconds in total)
  66.2 requests per second

  0.0% (0 out of 9597) of the missing items are queued
  at least 1000 queued builds
      aarch64-linux: 110 (11.0%)
      powerpc64le-linux: 890 (89.0%)
  build rate: 36.81 builds per hour
      aarch64-linux: 17.23 builds per hour
      x86_64-linux: 14.25 builds per hour
      powerpc64le-linux: 1.01 builds per hour
      i686-linux: 4.83 builds per hour
1871 packages are missing from 'https://ci.guix.gnu.org' for 'aarch64-linux', among which:
  3479	rust@1.39.0	/gnu/store/xxlgndidxvhdd391k35vcmviixq5d9b0-rust-1.39.0-cargo /gnu/store/cfy1p8q4bwwy1i01cjfssfry21kpljz3-rust-1.39.0 
  2111	cairomm@1.14.2	/gnu/store/bxknxn3nbmmvavf537k0pggrynhrgsaf-cairomm-1.14.2-doc /gnu/store/3sn66mgr29v73zpp93c2v09a0rj87l3w-cairomm-1.14.2 
  2101	texlive-latex-pgf@59745	/gnu/store/l6jr7v8ygn3ybj4gxcwskf8ifsjcj6x1-texlive-latex-pgf-59745 
looking for 16265 store items on https://bordeaux.guix.gnu.org...
https://bordeaux.guix.gnu.org
  83.8% substitutes available (13624 out of 16265)
  35138.6 MiB of nars (compressed)
  109501.6 MiB on disk (uncompressed)
  0.060 seconds per request (699.4 seconds in total)
  16.7 requests per second
  (continuous integration information unavailable)
579 packages are missing from 'https://bordeaux.guix.gnu.org' for 'aarch64-linux', among which:
  3479	rust@1.39.0	/gnu/store/xxlgndidxvhdd391k35vcmviixq5d9b0-rust-1.39.0-cargo /gnu/store/cfy1p8q4bwwy1i01cjfssfry21kpljz3-rust-1.39.0
--8<---------------cut here---------------end--------------->8---



> Hello,
>
> The aarch64 workers were all idle whereas 70k builds were
> available. Once restarted, they started building again.
>
> The problem might be that when the server is unavailable for a while the
> worker connections expire and cannot be resumed once the server is
> available again.
>
> Thanks,
>
> Mathieu