New North American based Guix Substitute Server, cuirass.genenetwork.org Now Available

New North American based Guix Substitute Server, cuirass.genenetwork.org Now Available

[Collin J. Doering]

[-- Attachment #1: Type: text/plain, Size: 2388 bytes --]

Dear Guix Community,

I am excited to announce that Guix substitutes (for x86_64) are now available in North America, thanks to the generous contribution of server hardware and infrastructure from GeneNetwork.org.

This new server enhances the availability and redundancy of Guix substitutes, ensuring a more robust and reliable experience for users in North America.

We will be posting a detailed blog article about the setup of the build farm in the coming weeks, but the Guix system configuration and notes about its setup can be found at https://git.rekahsoft.ca/rekahsoft/guix-north-america/. Stay tuned for more information!

# Using Substitutes from cuirass.genenetwork.org

If you're using Guix System, you can use the cuirass.genenetwork.org substitute server completing the following:

1. Add `https://cuirass.genenetwork.org` to the list of substitute servers (using the `substitute-urls` field of `guix-configuration` passed to the `guix-daemon` service).
2. Adjust guix-daemon ACLs to include the following public key (using the `authorized-keys` field of `guix-configuration` passed to the `guix-daemon` service).

```
(public-key 
 (ecc 
  (curve Ed25519)
  (q #9578AD6CDB23BA51F9C4185D5D5A32A7EEB47ACDD55F1CCB8CEE4E0570FBF961#)
  )
)
```

In the future, we hope to work with Guix maintainers to include this substitute server as one of the provided Guix System defaults.

When using Guix on a foreign distribution, you'll need to do the following to enable substitutes from cuirass.genenetwork.org:

1. Add the substitute url using the `--substitute-urls` option to guix-daemon. Assuming your foreign distribution uses systemd, this can be done using the following.

```
sudo systemctl edit --full guix-daemon
sudo systemctl restart guix-daemon.service
```

2. Add the public key for cuirass.genenetwork.org to the guix-daemon ACLs.

```
guix archive --authorize < /root/.config/guix/current/share/guix/cuirass.genenetwork.org.pub
```

If you want to just use ci.guix.gnu.org, or cuirass.genenetwork.org for that matter, you'll need to adjust the substitute URLs configuration for the guix-daemon to just refer to the substitute servers you want to use.

Best regards,

PS: thanks for the support Arun Isaac and Pjotr Prins!

-- 
Collin J. Doering

http://rekahsoft.ca
http://blog.rekahsoft.ca
http://git.rekahsoft.ca

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 255 bytes --]

Re: New North American based Guix Substitute Server, cuirass.genenetwork.org Now Available

[Ludovic Courtès]

Hi Collin,

"Collin J. Doering" <collin@rekahsoft.ca> skribis:

> I am excited to announce that Guix substitutes (for x86_64) are now available in North America, thanks to the generous contribution of server hardware and infrastructure from GeneNetwork.org.
>
> This new server enhances the availability and redundancy of Guix substitutes, ensuring a more robust and reliable experience for users in North America.
>
> We will be posting a detailed blog article about the setup of the build farm in the coming weeks, but the Guix system configuration and notes about its setup can be found at https://git.rekahsoft.ca/rekahsoft/guix-north-america/. Stay tuned for more information!

Congrats!  I’m confident many in the Americas will find it useful.

If I’m not mistaken, the machine is configured¹ to take substitutes from
ci.guix and bordeaux.guix.  It would be interesting (but obviously more
expensive CPU-wise…) to not do that as a way to get diverse builds that
can be checked with ‘guix challenge’.

Thanks,
Ludo’.

¹ https://git.rekahsoft.ca/rekahsoft/guix-north-america/src/branch/master/.guix/guix-na/config/balg02.scm

Re: New North American based Guix Substitute Server, cuirass.genenetwork.org Now Available

[Collin J. Doering]

[-- Attachment #1: Type: text/plain, Size: 1773 bytes --]

Hi Ludo,

Thanks! Following your advice, I have adjusted the system configuration to no longer use the default guix substitute servers[1] and deployed it to cuirass.genenetwork.org.

Kind regards,

PS: originally sent this only to Ludo due to an issue with my mail client.

Footnotes:
[1] https://git.rekahsoft.ca/rekahsoft/guix-north-america/commit/9e9816870a30d1345ae1120b5b8e8c77b17d34d4

On 11 Jul 2024 at 12:01, Ludovic Courtès <ludo@gnu.org> wrote:

> Hi Collin,
>
> "Collin J. Doering" <collin@rekahsoft.ca> skribis:
>
>> I am excited to announce that Guix substitutes (for x86_64) are now available in North America, thanks to the generous contribution of server hardware and infrastructure from GeneNetwork.org.
>>
>> This new server enhances the availability and redundancy of Guix substitutes, ensuring a more robust and reliable experience for users in North America.
>>
>> We will be posting a detailed blog article about the setup of the build farm in the coming
>> weeks, but the Guix system configuration and notes about its setup can be found at
>> https://git.rekahsoft.ca/rekahsoft/guix-north-america/. Stay tuned for more information!
>
> Congrats!  I’m confident many in the Americas will find it useful.
>
> If I’m not mistaken, the machine is configured¹ to take substitutes from
> ci.guix and bordeaux.guix.  It would be interesting (but obviously more
> expensive CPU-wise…) to not do that as a way to get diverse builds that
> can be checked with ‘guix challenge’.
>
> Thanks,
> Ludo’.
>
> ¹ https://git.rekahsoft.ca/rekahsoft/guix-north-america/src/branch/master/.guix/guix-na/config/balg02.scm

-- 
Collin J. Doering

http://rekahsoft.ca
http://blog.rekahsoft.ca
http://git.rekahsoft.ca

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 255 bytes --]

Re: New North American based Guix Substitute Server, cuirass.genenetwork.org Now Available

[Vagrant Cascadian]

[-- Attachment #1: Type: text/plain, Size: 771 bytes --]

On 2024-07-06, Collin J. Doering wrote:
> I am excited to announce that Guix substitutes (for x86_64) are now
> available in North America, thanks to the generous contribution of
> server hardware and infrastructure from GeneNetwork.org.

The last evaluation that actually seems to have succeeded was from late
August:

  https://cuirass.genenetwork.org/eval/154706

All the evaluations since then have failed, up until around
mid-september...

I was pretty thrilled to use it while it was working... especially as it
provided another reference point to check reproducible builds of guix!

I imagine running a whole build farm would be pretty intensive resource.
Soooo... I am curious what the current prospects are of getting it
running again? :)


live well,
  vagrant

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

Fwd: Re: New North American based Guix Substitute Server, cuirass.genenetwork.org Now Available

[Collin J. Doering]

[-- Attachment #1.1: Type: text/plain, Size: 377 bytes --]

Didn't send this to the mailing list by accident. Forwarding for reference.

-------------------- Start of forwarded message --------------------
From: "Collin J. Doering" <collin@rekahsoft.ca>
To: Vagrant Cascadian <vagrant@debian.org>
Subject: Re: New North American based Guix Substitute Server,
 cuirass.genenetwork.org Now Available
Date: Mon, 04 Nov 2024 10:55:09 -0500


[-- Attachment #1.2.1: Type: text/plain, Size: 2481 bytes --]

Hi Vagrant,

Firstly, thanks for reaching out!

> The last evaluation that actually seems to have succeeded was from late
> August:
>
>   https://cuirass.genenetwork.org/eval/154706
>
> All the evaluations since then have failed, up until around
> mid-september...

I am aware of the evaluation failures, but haven't got around to resolving it. However, your email was the push I needed (thank you!), and I have now opened https://issues.guix.gnu.org/74203. TLDR: builds started failing due to a coreutils test that occasionally can fail on systems using btrfs. I hope to get this resolved ASAP so newer evaluations and substitutes can be built.

> I was pretty thrilled to use it while it was working... especially as it
> provided another reference point to check reproducible builds of guix!

That's great to hear! We're excited about it too!

> I imagine running a whole build farm would be pretty intensive resource.
> Soooo... I am curious what the current prospects are of getting it
> running again? :)

Yes, definitely not a cheap endeavor. I'm very gracious for the folks at University of Tennessee for their support (and patience as I get this resolved).

Please keep an eye on the issue I opened up, as once resolved, I should be able to get builds running again on cuirass.genenetwork.org.

Thanks again for reaching out, and excited to hear you've been getting value out of cuirass.genenetwork.org (at least, while it was building)!

On 03 Nov 2024 at 20:43, Vagrant Cascadian <vagrant@debian.org> wrote:

> On 2024-07-06, Collin J. Doering wrote:
>> I am excited to announce that Guix substitutes (for x86_64) are now
>> available in North America, thanks to the generous contribution of
>> server hardware and infrastructure from GeneNetwork.org.
>
> The last evaluation that actually seems to have succeeded was from late
> August:
>
>   https://cuirass.genenetwork.org/eval/154706
>
> All the evaluations since then have failed, up until around
> mid-september...
>
> I was pretty thrilled to use it while it was working... especially as it
> provided another reference point to check reproducible builds of guix!
>
> I imagine running a whole build farm would be pretty intensive resource.
> Soooo... I am curious what the current prospects are of getting it
> running again? :)
>
>
> live well,
>   vagrant

-- 
Collin J. Doering

http://rekahsoft.ca
http://blog.rekahsoft.ca
http://git.rekahsoft.ca

[-- Attachment #1.2.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 255 bytes --]

[-- Attachment #1.3: Type: text/plain, Size: 168 bytes --]

-------------------- End of forwarded message --------------------

-- 
Collin J. Doering

http://rekahsoft.ca
http://blog.rekahsoft.ca
http://git.rekahsoft.ca

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 255 bytes --]

Re: New North American based Guix Substitute Server, cuirass.genenetwork.org Now Available

[Collin J. Doering]

[-- Attachment #1: Type: text/plain, Size: 4668 bytes --]

Hi Vagrant,

Just wanted to follow up with some good news! cuirass.genenetwork.org is now building packages and serving substitutes again.

--8<---------------cut here---------------start------------->8---
% guix time-machine -- weather --substitute-urls=https://cuirass.genenetwork.org
<..>
computing 31,887 package derivations for x86_64-linux...
looking for 37,018 store items on https://cuirass.genenetwork.org...
https://cuirass.genenetwork.org ⛅
  74.6% substitutes available (27,605 out of 37,018)
  at least 37,141.3 MiB of nars (compressed)
  86,508.9 MiB on disk (uncompressed)
  0.009 seconds per request (329.8 seconds in total)
  112.2 requests per second

  3.5% (326 out of 9,413) of the missing items are queued
  at least 1,000 queued builds
      x86_64-linux: 998 (99.8%)
      i686-linux: 2 (.2%)
  build rate: 23.71 builds per hour
      x86_64-linux: 23.71 builds per hour
--8<---------------cut here---------------end--------------->8---

Please let me know if you have any further issues and I apologize for any inconveniences caused!

Further, I thought it would be valuable to provide a short post-mortem detailing the outage.

1.  Builds started failing on cuirass.genenetwork.org 2024-09-06, starting at evaluation
    155302[1] where the evaluation fails to proceed due to a failure to build coreutils.
2.  This same build succeeded on local systems, and on ci.guix.gnu.org, but continued to fail
    on cuirass.genenetwork.org. No changes were made to cuirass.genenetwork.org prior to this
    failure occurring.
3.  Some initial investigation showed that the tests/cp/reflink-auto.sh coreutils test was
    failing, but why was unknown (it was not reproducible on other guix systems, as well as
    other guix build farms).
4.  Guix bug #74203[2] was opened on 2024-10-04 detailing the failure to build coreutils, and
    suggesting a patch to disable the failing test (similar to what nix folks did for the same
    issue)
5.  Time to investigate the issue thoroughly didn't come until 2024-W45, where it was
    discovered that the coreutils test in question was flaky on btrfs systems in some situations.
    Though exact details need to be further investigated, using a tmpfs /tmp causes the test
    to no longer fail! Further detail can be seen in Guix bug #74203[2].
6.  The discovery made in 5 allowed cuirass.genenetwork.org to be updated, however the guix
    channel/s that were in use had a hash mismatch for the pango package. This was easily
    worked around by using a newer version of the guix channel.
7.  However, another odd failure was hit during updating; specifically, while building the
    guix package, its test suite would crash (not fail, but actually crash). An existing issue
    (#46413[3]) very closely captures the issue faced, however in this case the test-suite for
    guix was not being run manually. I will follow up on this issue, and if necessary, create
    a new one to track this specific failure.
8.  In order to work around the issue building guix in 7, I opted to fall back to using
    substitutes from ci.guix.gnu.org. This allowed me to update cuirass.genenetwork.org, and
    get it building again.
9.  Upon updating, the default version of postgres was updated upstream, so cuirass couldn't
    be started until its data was updated, which was quickly resolved.


PS: apologies for the double message - forgot to wide reply (mu4e changed the reply workflow and I haven't caught up yet!)

[1]: https://cuirass.genenetwork.org/eval/155302
[2]: https://issues.guix.gnu.org/74203
[3]: https://issues.guix.gnu.org/46413

On 03 Nov 2024 at 20:43, Vagrant Cascadian <vagrant@debian.org> wrote:

> On 2024-07-06, Collin J. Doering wrote:
>> I am excited to announce that Guix substitutes (for x86_64) are now
>> available in North America, thanks to the generous contribution of
>> server hardware and infrastructure from GeneNetwork.org.
>
> The last evaluation that actually seems to have succeeded was from late
> August:
>
>   https://cuirass.genenetwork.org/eval/154706
>
> All the evaluations since then have failed, up until around
> mid-september...
>
> I was pretty thrilled to use it while it was working... especially as it
> provided another reference point to check reproducible builds of guix!
>
> I imagine running a whole build farm would be pretty intensive resource.
> Soooo... I am curious what the current prospects are of getting it
> running again? :)
>
>
> live well,
>   vagrant

-- 
Collin J. Doering

http://rekahsoft.ca
http://blog.rekahsoft.ca
http://git.rekahsoft.ca

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 255 bytes --]

Re: New North American based Guix Substitute Server, cuirass.genenetwork.org Now Available

[Ludovic Courtès]

Hi Collin,

"Collin J. Doering" <collin@rekahsoft.ca> skribis:

> Just wanted to follow up with some good news! cuirass.genenetwork.org is now building packages and serving substitutes again.
>
> % guix time-machine -- weather --substitute-urls=https://cuirass.genenetwork.org
> <..>
> computing 31,887 package derivations for x86_64-linux...
> looking for 37,018 store items on https://cuirass.genenetwork.org...
> https://cuirass.genenetwork.org ⛅
>   74.6% substitutes available (27,605 out of 37,018)

Thanks for the update and explanations!  It’s great to see it back up
and with good x86_64 coverage already.

Ludo’.