Hi Vagrant,

Just wanted to follow up with some good news! cuirass.genenetwork.org is now building packages and serving substitutes again.

--8<---------------cut here---------------start------------->8---
% guix time-machine -- weather --substitute-urls=https://cuirass.genenetwork.org
<..>
computing 31,887 package derivations for x86_64-linux...
looking for 37,018 store items on https://cuirass.genenetwork.org...
https://cuirass.genenetwork.org ⛅
  74.6% substitutes available (27,605 out of 37,018)
  at least 37,141.3 MiB of nars (compressed)
  86,508.9 MiB on disk (uncompressed)
  0.009 seconds per request (329.8 seconds in total)
  112.2 requests per second

  3.5% (326 out of 9,413) of the missing items are queued
  at least 1,000 queued builds
      x86_64-linux: 998 (99.8%)
      i686-linux: 2 (.2%)
  build rate: 23.71 builds per hour
      x86_64-linux: 23.71 builds per hour
--8<---------------cut here---------------end--------------->8---

Please let me know if you have any further issues and I apologize for any inconveniences caused!

Further, I thought it would be valuable to provide a short post-mortem detailing the outage.

1.  Builds started failing on cuirass.genenetwork.org 2024-09-06, starting at evaluation
    155302[1] where the evaluation fails to proceed due to a failure to build coreutils.
2.  This same build succeeded on local systems, and on ci.guix.gnu.org, but continued to fail
    on cuirass.genenetwork.org. No changes were made to cuirass.genenetwork.org prior to this
    failure occurring.
3.  Some initial investigation showed that the tests/cp/reflink-auto.sh coreutils test was
    failing, but why was unknown (it was not reproducible on other guix systems, as well as
    other guix build farms).
4.  Guix bug #74203[2] was opened on 2024-10-04 detailing the failure to build coreutils, and
    suggesting a patch to disable the failing test (similar to what nix folks did for the same
    issue)
5.  Time to investigate the issue thoroughly didn't come until 2024-W45, where it was
    discovered that the coreutils test in question was flaky on btrfs systems in some situations.
    Though exact details need to be further investigated, using a tmpfs /tmp causes the test
    to no longer fail! Further detail can be seen in Guix bug #74203[2].
6.  The discovery made in 5 allowed cuirass.genenetwork.org to be updated, however the guix
    channel/s that were in use had a hash mismatch for the pango package. This was easily
    worked around by using a newer version of the guix channel.
7.  However, another odd failure was hit during updating; specifically, while building the
    guix package, its test suite would crash (not fail, but actually crash). An existing issue
    (#46413[3]) very closely captures the issue faced, however in this case the test-suite for
    guix was not being run manually. I will follow up on this issue, and if necessary, create
    a new one to track this specific failure.
8.  In order to work around the issue building guix in 7, I opted to fall back to using
    substitutes from ci.guix.gnu.org. This allowed me to update cuirass.genenetwork.org, and
    get it building again.
9.  Upon updating, the default version of postgres was updated upstream, so cuirass couldn't
    be started until its data was updated, which was quickly resolved.


PS: apologies for the double message - forgot to wide reply (mu4e changed the reply workflow and I haven't caught up yet!)

[1]: https://cuirass.genenetwork.org/eval/155302
[2]: https://issues.guix.gnu.org/74203
[3]: https://issues.guix.gnu.org/46413

On 03 Nov 2024 at 20:43, Vagrant Cascadian <vagrant@debian.org> wrote:

> On 2024-07-06, Collin J. Doering wrote:
>> I am excited to announce that Guix substitutes (for x86_64) are now
>> available in North America, thanks to the generous contribution of
>> server hardware and infrastructure from GeneNetwork.org.
>
> The last evaluation that actually seems to have succeeded was from late
> August:
>
>   https://cuirass.genenetwork.org/eval/154706
>
> All the evaluations since then have failed, up until around
> mid-september...
>
> I was pretty thrilled to use it while it was working... especially as it
> provided another reference point to check reproducible builds of guix!
>
> I imagine running a whole build farm would be pretty intensive resource.
> Soooo... I am curious what the current prospects are of getting it
> running again? :)
>
>
> live well,
>   vagrant

-- 
Collin J. Doering

http://rekahsoft.ca
http://blog.rekahsoft.ca
http://git.rekahsoft.ca