bug#38062: Offloading + timeout + --keep-going leads to assertion failure

bug#38062: Offloading + timeout + --keep-going leads to assertion failure

[Ludovic Courtès]

Hello,

An offloaded build that times out in the presence of ‘--keep-going’
leads to an assertion failure.  To reproduce, set up offloading and run
something like this:

--8<---------------cut here---------------start------------->8---
$ guix build vim --no-substitutes --timeout=5 --keep-going
The following derivation will be built:
   /gnu/store/5mnnym9xwl145s1b88aqfmrka810w9ci-vim-8.1.0644.drv
process 4277 acquired build slot '/var/guix/offload/localhost:2222/0'
load on machine 'localhost' is 0.04 (normalized: 0.01)
building /gnu/store/5mnnym9xwl145s1b88aqfmrka810w9ci-vim-8.1.0644.drv...
sending 1 store item (48 MiB) to 'localhost'...
exporting path `/gnu/store/mlwyk5vcja0gqm20xxj8mwgf0fbqv8cz-vim-8.1.0644-checkout'
building of `/gnu/store/5mnnym9xwl145s1b88aqfmrka810w9ci-vim-8.1.0644.drv' timed out after 5 seconds
build of /gnu/store/5mnnym9xwl145s1b88aqfmrka810w9ci-vim-8.1.0644.drv failed
View build log at '/var/log/guix/drvs/5m/nnym9xwl145s1b88aqfmrka810w9ci-vim-8.1.0644.drv.bz2'.
guix build: error: corrupt input while restoring archive from #<closed: file 7ff2769cdc40>
--8<---------------cut here---------------end--------------->8---

The last error is due to premature EOF on the client socket: the child
‘guix-daemon’ process crashed, and thus ‘guix build’ gets EPIPE on its
client socket.  Here’s how the ‘guix-daemon’ process crashed:

--8<---------------cut here---------------start------------->8---
$ tail -3 /var/log/guix-daemon.log
accepted connection from pid 4270, user ludo
accepted connection from pid 4277, user root
guix-daemon: nix/libstore/build.cc:3448: void nix::Worker::run(const Goals&): Assertion `!settings.keepGoing || children.empty()' failed.
--8<---------------cut here---------------end--------------->8---

In Cuirass, this manifests itself with EPIPE while writing to the client
socket:

--8<---------------cut here---------------start------------->8---
2019-11-04T20:11:26 fatal: uncaught exception 'system-error' in 'restart-builds' fiber!
2019-11-04T20:11:26 exception arguments: ("fport_write" "~A" ("Broken pipe") (32))
In ice-9/boot-9.scm:
    829:9  5 (catch _ _ #<procedure 7f79d663c320 at cuirass/utils.s?> ?)
   751:25  4 (dispatch-exception 0 system-error ("fport_write" "~A" ?))
In cuirass/utils.scm:
    183:8  3 (_ _ "fport_write" "~A" ("Broken pipe") (32))
In ice-9/boot-9.scm:
    829:9  2 (catch #t #<procedure 7f79cc7e41c0 at cuirass/utils.sc?> ?)
In cuirass/utils.scm:
   184:22  1 (_)
In unknown file:
           0 (make-stack #t)
ERROR: In procedure make-stack:
In procedure fport_write: Broken pipe
--8<---------------cut here---------------end--------------->8---

This bug is almost certainly a consequence of commit
ada9a19a2dca74feafcf24df1152abd685d4142f.

To be continued…

Ludo’.

bug#38062: Offloading + timeout + --keep-going leads to assertion failure

[Ludovic Courtès]

Hi,

Ludovic Courtès <ludo@gnu.org> skribis:

> An offloaded build that times out in the presence of ‘--keep-going’
> leads to an assertion failure.  To reproduce, set up offloading and run
> something like this:
>
> $ guix build vim --no-substitutes --timeout=5 --keep-going
> The following derivation will be built:
>    /gnu/store/5mnnym9xwl145s1b88aqfmrka810w9ci-vim-8.1.0644.drv
> process 4277 acquired build slot '/var/guix/offload/localhost:2222/0'
> load on machine 'localhost' is 0.04 (normalized: 0.01)
> building /gnu/store/5mnnym9xwl145s1b88aqfmrka810w9ci-vim-8.1.0644.drv...
> sending 1 store item (48 MiB) to 'localhost'...
> exporting path `/gnu/store/mlwyk5vcja0gqm20xxj8mwgf0fbqv8cz-vim-8.1.0644-checkout'
> building of `/gnu/store/5mnnym9xwl145s1b88aqfmrka810w9ci-vim-8.1.0644.drv' timed out after 5 seconds
> build of /gnu/store/5mnnym9xwl145s1b88aqfmrka810w9ci-vim-8.1.0644.drv failed
> View build log at '/var/log/guix/drvs/5m/nnym9xwl145s1b88aqfmrka810w9ci-vim-8.1.0644.drv.bz2'.
> guix build: error: corrupt input while restoring archive from #<closed: file 7ff2769cdc40>
>
>
> The last error is due to premature EOF on the client socket: the child
> ‘guix-daemon’ process crashed, and thus ‘guix build’ gets EPIPE on its
> client socket.  Here’s how the ‘guix-daemon’ process crashed:
>
> $ tail -3 /var/log/guix-daemon.log
> accepted connection from pid 4270, user ludo
> accepted connection from pid 4277, user root
> guix-daemon: nix/libstore/build.cc:3448: void nix::Worker::run(const Goals&): Assertion `!settings.keepGoing || children.empty()' failed.

Fixed by af73beeba1fc9effab60b11aea1d7ed8c24e7367.

I’ll update the ‘guix’ package soonish.

Ludo’.