bug#67575: guix-daemon assertion failure upon substitution failure

bug#67575: guix-daemon assertion failure upon substitution failure

[Ludovic Courtès]

On a build machine running ‘cuirass remote-server’ with
/gnu/store/5g4c9kii8l50lw23s7yb2a4bvc7qzw5d-guix-1.4.0-10.4dfdd82/bin/guix-daemon
(from Guix commit 2ab5e449246f98b049888dde3c310f5b4a0a64a2), I see these
‘cuirass remote-worker’ errors:

--8<---------------cut here---------------start------------->8---
2023-11-27 06:49:54 CwQfwEE3: building derivation `/gnu/store/rivlxrqr10g3kj9knyz7i41aq1vzxn9z-r-ecoregime-0.1.3.drv' (system: x86_64-linux)
2023-11-27 06:50:11 error: CwQfwEE3: unexpected error while building '/gnu/store/rivlxrqr10g3kj9knyz7i41aq1vzxn9z-r-ecoregime-0.1.3.drv': #<&nar-error file: #f port: #<input-output: socket 52>>
[…]
2023-11-27 07:00:58 CwQfwEE3: building derivation `/gnu/store/b5qrpwpx5sbmlj5f7nx6wknnkczadjcn-r-cpp-0.1.0.drv' (system: x86_64-linux)
2023-11-27 07:01:20 error: CwQfwEE3: unexpected error while building '/gnu/store/b5qrpwpx5sbmlj5f7nx6wknnkczadjcn-r-cpp-0.1.0.drv': #<&nar-error file: #f port: #<input-output: socket 52>>
--8<---------------cut here---------------end--------------->8---

… corresponding to these guix-daemon messages:

--8<---------------cut here---------------start------------->8---
2023-11-27 06:49:54 accepted connection from pid 14007, user root
2023-11-27 06:50:11 guix-daemon: nix/libstore/build.cc:3322: void nix::Worker::childTerminated(pid_t, bool): Assertion `i != children.end()' failed.
[…]
2023-11-27 07:00:58 accepted connection from pid 14007, user root
2023-11-27 07:01:20 guix-daemon: nix/libstore/build.cc:3322: void nix::Worker::childTerminated(pid_t, bool): Assertion `i != children.end()' failed.
--8<---------------cut here---------------end--------------->8---

The corresponding build log (sent by ‘cuirass remote-worker’ to the
server):

--8<---------------cut here---------------start------------->8---
$ zcat /var/cache/cuirass/remote/rivlxrqr10g3kj9knyz7i41aq1vzxn9z.log.gz |tail
@ substituter-started /gnu/store/gymm7n27rfhyz83zx3lw0bwg4m6alpzw-ecoregime_0.1.3.tar.gz substitute
guix substitute: warning: while fetching https://ci.guix.gnu.org/nar/gymm7n27rfhyz83zx3lw0bwg4m6alpzw-ecoregime_0.1.3.tar.gz: server is somewhat slow
guix substitute: warning: try `--no-substitutes' if the problem persists
retrying download of '/gnu/store/gymm7n27rfhyz83zx3lw0bwg4m6alpzw-ecoregime_0.1.3.tar.gz' with other substitute URLs...
guix substitute: warning: while fetching https://ci.guix.gnu.org/nar/gymm7n27rfhyz83zx3lw0bwg4m6alpzw-ecoregime_0.1.3.tar.gz: server is somewhat slow
guix substitute: warning: try `--no-substitutes' if the problem persists
guix substitute: error: failed to find alternative substitute for '/gnu/store/gymm7n27rfhyz83zx3lw0bwg4m6alpzw-ecoregime_0.1.3.tar.gz'
fetching path `/gnu/store/gymm7n27rfhyz83zx3lw0bwg4m6alpzw-ecoregime_0.1.3.tar.gz' (empty status: '')
@ substituter-failed /gnu/store/gymm7n27rfhyz83zx3lw0bwg4m6alpzw-ecoregime_0.1.3.tar.gz  fetching path `/gnu/store/gymm7n27rfhyz83zx3lw0bwg4m6alpzw-ecoregime_0.1.3.tar.gz' (empty status: '')
fetching path `/gnu/store/yiqzanx7vb8s0sj9h15p2llxwrkv0xzh-r-descriptio-1.2'...

$ zcat /var/cache/cuirass/remote/b5qrpwpx5sbmlj5f7nx6wknnkczadjcn.log.gz |tail
substitute: updating substitutes from 'http://194.199.1.1:5557'... 100.0%
guix substitute: warning: while fetching https://ci.guix.gnu.org/nar/iqi6s452hqy3k3n335c5v9nb6wq8h9nn-CPP_0.1.0.tar.gz: server is somewhat slow
guix substitute: warning: try `--no-substitutes' if the problem persists
retrying download of '/gnu/store/iqi6s452hqy3k3n335c5v9nb6wq8h9nn-CPP_0.1.0.tar.gz' with other substitute URLs...
guix substitute: warning: while fetching https://ci.guix.gnu.org/nar/iqi6s452hqy3k3n335c5v9nb6wq8h9nn-CPP_0.1.0.tar.gz: server is somewhat slow
guix substitute: warning: try `--no-substitutes' if the problem persists
guix substitute: error: failed to find alternative substitute for '/gnu/store/iqi6s452hqy3k3n335c5v9nb6wq8h9nn-CPP_0.1.0.tar.gz'
fetching path `/gnu/store/iqi6s452hqy3k3n335c5v9nb6wq8h9nn-CPP_0.1.0.tar.gz' (empty status: '')
@ substituter-failed /gnu/store/iqi6s452hqy3k3n335c5v9nb6wq8h9nn-CPP_0.1.0.tar.gz  fetching path `/gnu/store/iqi6s452hqy3k3n335c5v9nb6wq8h9nn-CPP_0.1.0.tar.gz' (empty status: '')
fetching path `/gnu/store/b7rnd5y4zsc1995h0a8w294pzdgbmhr9-mc2d_0.2.0.tar.gz'...
--8<---------------cut here---------------end--------------->8---

In both cases, “failed to find alternative substitute” leads ‘guix
substitute’ to exit, which guix-daemon doesn’t really cope with.

Ludo’.

bug#67575: guix-daemon assertion failure upon substitution failure

[Ludovic Courtès]

Ludovic Courtès <ludo@gnu.org> skribis:

> 2023-11-27 07:00:58 accepted connection from pid 14007, user root
> 2023-11-27 07:01:20 guix-daemon: nix/libstore/build.cc:3322: void nix::Worker::childTerminated(pid_t, bool): Assertion `i != children.end()' failed.
>
>
> The corresponding build log (sent by ‘cuirass remote-worker’ to the
> server):
>
> $ zcat /var/cache/cuirass/remote/rivlxrqr10g3kj9knyz7i41aq1vzxn9z.log.gz |tail
> @ substituter-started /gnu/store/gymm7n27rfhyz83zx3lw0bwg4m6alpzw-ecoregime_0.1.3.tar.gz substitute
> guix substitute: warning: while fetching https://ci.guix.gnu.org/nar/gymm7n27rfhyz83zx3lw0bwg4m6alpzw-ecoregime_0.1.3.tar.gz: server is somewhat slow
> guix substitute: warning: try `--no-substitutes' if the problem persists
> retrying download of '/gnu/store/gymm7n27rfhyz83zx3lw0bwg4m6alpzw-ecoregime_0.1.3.tar.gz' with other substitute URLs...
> guix substitute: warning: while fetching https://ci.guix.gnu.org/nar/gymm7n27rfhyz83zx3lw0bwg4m6alpzw-ecoregime_0.1.3.tar.gz: server is somewhat slow
> guix substitute: warning: try `--no-substitutes' if the problem persists
> guix substitute: error: failed to find alternative substitute for '/gnu/store/gymm7n27rfhyz83zx3lw0bwg4m6alpzw-ecoregime_0.1.3.tar.gz'
> fetching path `/gnu/store/gymm7n27rfhyz83zx3lw0bwg4m6alpzw-ecoregime_0.1.3.tar.gz' (empty status: '')

This should be fixed by 06b9c1260c72935806957bf302c40c1db6101a63.

Ludo’.