bug#71133: linux-libre-guix.tar.xz CI times out on aarch64-linux

bug#71133: linux-libre-guix.tar.xz CI times out on aarch64-linux

[Richard Sent]

Hi Guix!

On aarch64 platforms, the linux-libre-guix.tar.xz.drv build times out
[1].

This package is known to have a long build time [2].

Users can work around this by running the build locally with a (very,
very large) max-silent-time (~28800 seconds on my machine). Given the
impact of this issue (unable to build linux-libre on 64-bit Arm without
running a multi-hour build), I think it's worth revisiting if the
substitute server timeout should be increased again.

Alternatively, perhaps we could fetch the officially released
Linux-libre tarballs instead of computing them ourselves [3].

[1]: https://ci.guix.gnu.org/build/4711550/log/raw
[2]: https://mail.gnu.org/archive/html/guix-devel/2021-08/msg00077.html
[3]: http://linux-libre.fsfla.org/pub/linux-libre/releases/

CC'ing Christopher Baines since this deals with substitutes.

-- 
Take it easy,
Richard Sent
Making my computer weirder one commit at a time.

bug#71133: linux-libre-guix.tar.xz CI times out on aarch64-linux

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 1454 bytes --]

Richard Sent <richard@freakingpenguin.com> writes:

> On aarch64 platforms, the linux-libre-guix.tar.xz.drv build times out
> [1].
>
> This package is known to have a long build time [2].
>
> Users can work around this by running the build locally with a (very,
> very large) max-silent-time (~28800 seconds on my machine). Given the
> impact of this issue (unable to build linux-libre on 64-bit Arm without
> running a multi-hour build), I think it's worth revisiting if the
> substitute server timeout should be increased again.
>
> Alternatively, perhaps we could fetch the officially released
> Linux-libre tarballs instead of computing them ourselves [3].
>
> [1]: https://ci.guix.gnu.org/build/4711550/log/raw
> [2]: https://mail.gnu.org/archive/html/guix-devel/2021-08/msg00077.html
> [3]: http://linux-libre.fsfla.org/pub/linux-libre/releases/
>
> CC'ing Christopher Baines since this deals with substitutes.

This derivation seems to have been built fine by the bordeaux build
farm:

  https://data.guix.gnu.org/gnu/store/ny56fdcig9cd9bd3pssmlraz2c1q10q8-linux-libre-6.8.10-guix.tar.xz.drv

This is the timeout configuration for the bordeaux ARM build machines:

  (max-silent-time (* 12 3600))
  (timeout (* 72 3600))

e.g. https://git.savannah.gnu.org/cgit/guix/maintenance.git/tree/hydra/hatysa.scm#n262

The ci.guix.gnu.org Honeycomb timeouts are different, but don't look too
short. Maybe this build happened on an Overdrive system though.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

bug#71133: linux-libre-guix.tar.xz CI times out on aarch64-linux

[Richard Sent]

Christopher Baines <mail@cbaines.net> writes:

> This derivation seems to have been built fine by the bordeaux build
> farm:
>
>   https://data.guix.gnu.org/gnu/store/ny56fdcig9cd9bd3pssmlraz2c1q10q8-linux-libre-6.8.10-guix.tar.xz.drv

You're right. It looks like the derivation itself built fine based on
that, but something seems off when the nar is sent. Assuming I
understand the following correctly:

--8<---------------cut here---------------start------------->8---
~/code/cloned/guix/guix $ guix build linux-libre --system=aarch64-linux
substitute: updating substitutes from 'http://10.1.2.2:80'... 100.0%
substitute: updating substitutes from 'https://bordeaux.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
The following derivations will be built:
  /gnu/store/v471590wpsw1fcnqrrr9bwh52skbb5rn-linux-libre-6.8.10.drv
  /gnu/store/1wi10rg7236ck8k5vdrdfap5l7a9s9z0-linux-libre-6.8.10-guix.tar.xz.drv
143.1 MB will be downloaded:
  /gnu/store/y813phs2n9xnb7zbcr07g0j9509bzbsb-linux-libre-6.8.10-guix.tar.xz
substituting /gnu/store/y813phs2n9xnb7zbcr07g0j9509bzbsb-linux-libre-6.8.10-guix.tar.xz...
downloading from https://bordeaux.guix.gnu.org/nar/none/y813phs2n9xnb7zbcr07g0j9509bzbsb-linux-libre-6.8.10-guix.tar.xz ...
 linux-libre-6.8.10-guix.tar.xz  136.5MiB                                                                          23.9MiB/s 00:06 ▕█████████████████▋▏  98.0%guix substitute: error: corrupt input while restoring '/gnu/store/y813phs2n9xnb7zbcr07g0j9509bzbsb-linux-libre-6.8.10-guix.tar.xz' from #<input: string 7f9644c23b60>
substitution of /gnu/store/y813phs2n9xnb7zbcr07g0j9509bzbsb-linux-libre-6.8.10-guix.tar.xz failed
guix build: error: some substitutes for the outputs of derivation `/gnu/store/ny56fdcig9cd9bd3pssmlraz2c1q10q8-linux-libre-6.8.10-guix.tar.xz.drv' failed (usually happens due to networking issues); try `--fallback' to build derivation from source 
--8<---------------cut here---------------end--------------->8---

Even though the derivation was built, the substitution fails.

Just for fun, here's the output on a ARM64-native system:

--8<---------------cut here---------------start------------->8---
root@caustic ~# guix build linux-libre  
substitute: updating substitutes from 'http://10.1.2.2:80'... 100.0%
substitute: updating substitutes from 'https://bordeaux.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
The following derivations will be built:
  /gnu/store/v471590wpsw1fcnqrrr9bwh52skbb5rn-linux-libre-6.8.10.drv
  /gnu/store/1wi10rg7236ck8k5vdrdfap5l7a9s9z0-linux-libre-6.8.10-guix.tar.xz.drv
143.1 MB will be downloaded:
  /gnu/store/y813phs2n9xnb7zbcr07g0j9509bzbsb-linux-libre-6.8.10-guix.tar.xz
substituting /gnu/store/0jfsx4hljddyand45z7i77ynpvr0mhb5-module-import-compiled...
downloading from http://10.1.2.2/nar/zstd/0jfsx4hljddyand45z7i77ynpvr0mhb5-module-import-compiled ...
 module-import-compiled                                                                                                    4.2MiB/s 00:00 | 244KiB transferred

substituting /gnu/store/fwqy5d6xyar9x5yksny79r2d519s25cx-100gnu+freedo.patch...
downloading from http://10.1.2.2/nar/zstd/fwqy5d6xyar9x5yksny79r2d519s25cx-100gnu%2Bfreedo.patch ...
 100gnu%2Bfreedo.patch                                                                                                      1.8MiB/s 00:00 | 46KiB transferred

substituting /gnu/store/y813phs2n9xnb7zbcr07g0j9509bzbsb-linux-libre-6.8.10-guix.tar.xz...
downloading from https://bordeaux.guix.gnu.org/nar/none/y813phs2n9xnb7zbcr07g0j9509bzbsb-linux-libre-6.8.10-guix.tar.xz ...
 linux-libre-6.8.10-guix.tar.xz  136.5MiB                                                                           1.7MiB/s 01:21 ▕█████████████████▉▏  99.9%guix substitute: error: corrupt input while restoring '/gnu/store/y813phs2n9xnb7zbcr07g0j9509bzbsb-linux-libre-6.8.10-guix.tar.xz' from #<input: string ffff7daf3460>
substitution of /gnu/store/y813phs2n9xnb7zbcr07g0j9509bzbsb-linux-libre-6.8.10-guix.tar.xz failed
guix build: error: corrupt input while restoring archive from #<closed: file ffff977abb60>
--8<---------------cut here---------------end--------------->8---

-- 
Take it easy,
Richard Sent
Making my computer weirder one commit at a time.

bug#71133: linux-libre-guix.tar.xz CI times out on aarch64-linux

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 2627 bytes --]

Richard Sent <richard@freakingpenguin.com> writes:

> Christopher Baines <mail@cbaines.net> writes:
>
>> This derivation seems to have been built fine by the bordeaux build
>> farm:
>>
>>   https://data.guix.gnu.org/gnu/store/ny56fdcig9cd9bd3pssmlraz2c1q10q8-linux-libre-6.8.10-guix.tar.xz.drv
>
> You're right. It looks like the derivation itself built fine based on
> that, but something seems off when the nar is sent. Assuming I
> understand the following correctly:
>
> --8<---------------cut here---------------start------------->8---
> ~/code/cloned/guix/guix $ guix build linux-libre --system=aarch64-linux
> substitute: updating substitutes from 'http://10.1.2.2:80'... 100.0%
> substitute: updating substitutes from 'https://bordeaux.guix.gnu.org'... 100.0%
> substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
> The following derivations will be built:
>   /gnu/store/v471590wpsw1fcnqrrr9bwh52skbb5rn-linux-libre-6.8.10.drv
>   /gnu/store/1wi10rg7236ck8k5vdrdfap5l7a9s9z0-linux-libre-6.8.10-guix.tar.xz.drv
> 143.1 MB will be downloaded:
>   /gnu/store/y813phs2n9xnb7zbcr07g0j9509bzbsb-linux-libre-6.8.10-guix.tar.xz
> substituting /gnu/store/y813phs2n9xnb7zbcr07g0j9509bzbsb-linux-libre-6.8.10-guix.tar.xz...
> downloading from https://bordeaux.guix.gnu.org/nar/none/y813phs2n9xnb7zbcr07g0j9509bzbsb-linux-libre-6.8.10-guix.tar.xz ...
>  linux-libre-6.8.10-guix.tar.xz  136.5MiB                                                                          23.9MiB/s 00:06 ▕█████████████████▋▏  98.0%guix substitute: error: corrupt input while restoring '/gnu/store/y813phs2n9xnb7zbcr07g0j9509bzbsb-linux-libre-6.8.10-guix.tar.xz' from #<input: string 7f9644c23b60>
> substitution of /gnu/store/y813phs2n9xnb7zbcr07g0j9509bzbsb-linux-libre-6.8.10-guix.tar.xz failed
> guix build: error: some substitutes for the outputs of derivation `/gnu/store/ny56fdcig9cd9bd3pssmlraz2c1q10q8-linux-libre-6.8.10-guix.tar.xz.drv' failed (usually happens due to networking issues); try `--fallback' to build derivation from source 
> --8<---------------cut here---------------end--------------->8---
>
> Even though the derivation was built, the substitution fails.

Yeah, something's up here specifically with the bordeaux build farm,
feel free to open a new issue.

You can tell something is wrong just by looking at the narinfo:

  https://bordeaux.guix.gnu.org/y813phs2n9xnb7zbcr07g0j9509bzbsb.narinfo

Given there's no compression here, the file size should be the same as
the nar size, but it's not, it's missing a few bytes.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

bug#71133: linux-libre-guix.tar.xz CI times out on aarch64-linux

[Richard Sent]

New issue opened at https://issues.guix.gnu.org/71214. I'l close this
one. Thanks!

-- 
Take it easy,
Richard Sent
Making my computer weirder one commit at a time.