bug#50312: Recurring timeout eror while using offload

bug#50312: Recurring timeout eror while using offload

[Maxim Cournoyer]

Hello Guix,

I'm encountered timeout error that seem to be caused by a particular
payload rather than networking issues.   With a specific payload (a
bunch of rust package to build on the core-updates-frozen branch), it
seems to be triggered every time.

Here's the derivations to be built:

--8<---------------cut here---------------start------------->8---
$ ./pre-inst-env guix build rust -n
The following derivations would be built:
   /gnu/store/nv3l1lyx4x33gx80pxk8033g01b6bv0m-rust-1.54.0.drv
   /gnu/store/bmc82kpr11gpk1xy7d3l6hy37cs58i7d-rustc-1.54.0-src.tar.xz.drv
   /gnu/store/x1v25nclqps677j1g5dvyck2njaxy0ya-rust-1.53.0.drv
   /gnu/store/9v5yl0fg4dj1n82pkznksasmdjgi541w-rust-1.52.1.drv
   /gnu/store/72572pjayh2j5frr187i7axjzfd01kp6-rust-1.51.0.drv
   /gnu/store/0z13dkldzsxf87f4kgi3b2545bm39nnh-rustc-1.51.0-src.tar.xz.drv
   /gnu/store/q9mbicnmrpd1c98vmi5y0gyjnpr3gqbn-rust-1.50.0.drv
   /gnu/store/h5959ysczr862ldwh6f85h10895c4csp-rust-1.49.0.drv
   /gnu/store/izgl0l21r5lwd3d3glxqwmigl0315b26-rustc-1.49.0-src.tar.xz.drv
   /gnu/store/n0dpka76b9q0w4r2k912qq5p2a57bz6c-rust-1.48.0.drv
   /gnu/store/a9qwa9cl90xrm2dpa3hnd3ak54nqss8n-rust-1.47.0.drv
   /gnu/store/7ib14fx50651av5bq5g195cqf2azsvhc-rust-1.46.0.drv
   /gnu/store/h3mxv9zlx8q3x0m0s5gvin2n1ndliivw-rust-1.45.2.drv
   /gnu/store/4i5xm0wcasilh55f9gfcrzp2sjdkd507-rust-1.44.1.drv
   /gnu/store/m7z7jdd2my4p9p5mjwa0cna4fzhd1v8j-rust-1.43.0.drv
   /gnu/store/pw4ynvgbnk1j6fxlh37a9wd8fyi7bjnc-rustc-1.50.0-src.tar.xz.drv
   /gnu/store/g9izbhyhc6jcalxgbxc6aas3aa66h2aa-rustc-1.52.1-src.tar.xz.drv
   /gnu/store/jbr3f7360x3ww5rn6nnb803yxq7ppck3-rustc-1.53.0-src.tar.xz.drv
--8<---------------cut here---------------end--------------->8---

And here is what the error looks like:

--8<---------------cut here---------------start------------->8---
rustc-1.52.1-src/src/llvm-project/clang/test/Misc/message-length.c
rustc-1.52.1-src/src/llvm-project/clang/test/Driver/fmessage-length.c
rustc-1.52.1-src/src/llvm-project/compiler-rt/test/hwasan/TestCases/stack-history-length.c
rustc-1.52.1-src/src/llvm-project/clang/test/Sema/vector_swizzle_length.c
rustc-1.52.1-src/src/llvm-project/clang/test/Sema/array-bounds-ptr-arith.c
rustc-1.52.1-src/src/llvm-project/clang/test/Analysis/ptr-arith.c
rustc-1.52.1-src/vendor/openssl-src/openssl/crypto/engine/tb_asnmth.c
rustc-1.52.1-src/src/llvm-project/clang/test/CodeGen/aarch64-sve-intrinsics/acle_sve_cnth.c
rustc-1.52.1-src/vendor/openssl-src/openssl/crypto/pem/pem_oth.c
rustc-1.52.1-src/vendor/libgit2-sys/libgit2/src/transports/auth.c
guix offload: error: failed to connect to '127.0.0.1': Timeout connecting to 127.0.0.1
cannot build derivation `/gnu/store/4i5xm0wcasilh55f9gfcrzp2sjdkd507-rust-1.44.1.drv': 1 dependencies couldn't be built
cannot build derivation `/gnu/store/h3mxv9zlx8q3x0m0s5gvin2n1ndliivw-rust-1.45.2.drv': 1 dependencies couldn't be built
cannot build derivation `/gnu/store/7ib14fx50651av5bq5g195cqf2azsvhc-rust-1.46.0.drv': 1 dependencies couldn't be built
cannot build derivation `/gnu/store/a9qwa9cl90xrm2dpa3hnd3ak54nqss8n-rust-1.47.0.drv': 1 dependencies couldn't be built
cannot build derivation `/gnu/store/n0dpka76b9q0w4r2k912qq5p2a57bz6c-rust-1.48.0.drv': 1 dependencies couldn't be built
cannot build derivation `/gnu/store/h5959ysczr862ldwh6f85h10895c4csp-rust-1.49.0.drv': 1 dependencies couldn't be built
cannot build derivation `/gnu/store/q9mbicnmrpd1c98vmi5y0gyjnpr3gqbn-rust-1.50.0.drv': 1 dependencies couldn't be built
cannot build derivation `/gnu/store/72572pjayh2j5frr187i7axjzfd01kp6-rust-1.51.0.drv': 1 dependencies couldn't be built
cannot build derivation `/gnu/store/9v5yl0fg4dj1n82pkznksasmdjgi541w-rust-1.52.1.drv': 1 dependencies couldn't be built
cannot build derivation `/gnu/store/x1v25nclqps677j1g5dvyck2njaxy0ya-rust-1.53.0.drv': 1 dependencies couldn't be built
cannot build derivation `/gnu/store/nv3l1lyx4x33gx80pxk8033g01b6bv0m-rust-1.54.0.drv': 1 dependencies couldn't be built
guix build: error: build of `/gnu/store/nv3l1lyx4x33gx80pxk8033g01b6bv0m-rust-1.54.0.drv' failed

real    2m50.854s
user    0m11.302s
sys     0m0.661s
--8<---------------cut here---------------end--------------->8---

It occurs shortly after heavy scrolling starts.

I've verified that it's not my network link at least on my local
network; any ideas what I should look into for debugging?  Perhaps the
logs/strace of the remote SSH server?

Thanks,

Maxim

bug#50312: Recurring timeout eror while using offload

[Maxim Cournoyer]

Hello,

Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:

[...]

> And here is what the error looks like:
>
> rustc-1.52.1-src/src/llvm-project/clang/test/Misc/message-length.c
> rustc-1.52.1-src/src/llvm-project/clang/test/Driver/fmessage-length.c
> rustc-1.52.1-src/src/llvm-project/compiler-rt/test/hwasan/TestCases/stack-history-length.c
> rustc-1.52.1-src/src/llvm-project/clang/test/Sema/vector_swizzle_length.c
> rustc-1.52.1-src/src/llvm-project/clang/test/Sema/array-bounds-ptr-arith.c
> rustc-1.52.1-src/src/llvm-project/clang/test/Analysis/ptr-arith.c
> rustc-1.52.1-src/vendor/openssl-src/openssl/crypto/engine/tb_asnmth.c
> rustc-1.52.1-src/src/llvm-project/clang/test/CodeGen/aarch64-sve-intrinsics/acle_sve_cnth.c
> rustc-1.52.1-src/vendor/openssl-src/openssl/crypto/pem/pem_oth.c
> rustc-1.52.1-src/vendor/libgit2-sys/libgit2/src/transports/auth.c
> guix offload: error: failed to connect to '127.0.0.1': Timeout connecting to 127.0.0.1
> cannot build derivation `/gnu/store/4i5xm0wcasilh55f9gfcrzp2sjdkd507-rust-1.44.1.drv': 1 dependencies couldn't be built
> cannot build derivation `/gnu/store/h3mxv9zlx8q3x0m0s5gvin2n1ndliivw-rust-1.45.2.drv': 1 dependencies couldn't be built
> cannot build derivation `/gnu/store/7ib14fx50651av5bq5g195cqf2azsvhc-rust-1.46.0.drv': 1 dependencies couldn't be built
> cannot build derivation `/gnu/store/a9qwa9cl90xrm2dpa3hnd3ak54nqss8n-rust-1.47.0.drv': 1 dependencies couldn't be built
> cannot build derivation `/gnu/store/n0dpka76b9q0w4r2k912qq5p2a57bz6c-rust-1.48.0.drv': 1 dependencies couldn't be built
> cannot build derivation `/gnu/store/h5959ysczr862ldwh6f85h10895c4csp-rust-1.49.0.drv': 1 dependencies couldn't be built
> cannot build derivation `/gnu/store/q9mbicnmrpd1c98vmi5y0gyjnpr3gqbn-rust-1.50.0.drv': 1 dependencies couldn't be built
> cannot build derivation `/gnu/store/72572pjayh2j5frr187i7axjzfd01kp6-rust-1.51.0.drv': 1 dependencies couldn't be built
> cannot build derivation `/gnu/store/9v5yl0fg4dj1n82pkznksasmdjgi541w-rust-1.52.1.drv': 1 dependencies couldn't be built
> cannot build derivation `/gnu/store/x1v25nclqps677j1g5dvyck2njaxy0ya-rust-1.53.0.drv': 1 dependencies couldn't be built
> cannot build derivation `/gnu/store/nv3l1lyx4x33gx80pxk8033g01b6bv0m-rust-1.54.0.drv': 1 dependencies couldn't be built
> guix build: error: build of `/gnu/store/nv3l1lyx4x33gx80pxk8033g01b6bv0m-rust-1.54.0.drv' failed
>
> real    2m50.854s
> user    0m11.302s
> sys     0m0.661s
>
> It occurs shortly after heavy scrolling starts.
>
> I've verified that it's not my network link at least on my local
> network; any ideas what I should look into for debugging?  Perhaps the
> logs/strace of the remote SSH server?
>
> Thanks,
>
> Maxim

Now that I've manually built them by small batches of 4 (that would not
fail), I reached the last item rust-1.54, which failed like so:

--8<---------------cut here---------------start------------->8---
rustc-1.54.0-src/vendor/lzma-sys/xz-5.2/tests/files/good-0cat-empty.xz
source is at 'rustc-1.54.0-src'
Backtrace:
          12 (primitive-load "/gnu/store/j2hl7fmsbh7jmixpy8dy07finqp…")
In ice-9/eval.scm:
    619:8 11 (_ #(#(#<directory (guile-user) 7ffff5fdbc80> "rus…") #))
In ice-9/boot-9.scm:
    142:2 10 (dynamic-wind _ _ #<procedure 7ffff5fc8e00 at ice-9/eva…>)
In system/base/compile.scm:
   352:28  9 (compile _ #:from _ #:to _ #:env _ #:optimization-level …)
   265:44  8 (_ _ _)
   265:44  7 (_ _ _)
   265:44  6 (_ _ _)
   261:27  5 (_ _ _)
In ice-9/boot-9.scm:
   2835:4  4 (save-module-excursion _)
In language/bytecode/spec.scm:
    35:19  3 (_)
In ice-9/boot-9.scm:
   260:13  2 (for-each #<procedure delete-file-recursively (dir #:k…> …)
In guix/build/utils.scm:
   446:23  1 (delete-file-recursively "vendor/jemalloc-sys/jemalloc" …)
In unknown file:
           0 (lstat "vendor/jemalloc-sys/jemalloc")

ERROR: In procedure lstat:
In procedure lstat: No such file or directory: "vendor/jemalloc-sys/jemalloc"
note: keeping build directory `/tmp/guix-build-rustc-1.54.0-src.tar.xz.drv-0'
builder for `/gnu/store/bmc82kpr11gpk1xy7d3l6hy37cs58i7d-rustc-1.54.0-src.tar.xz.drv' failed with exit code 1
build of /gnu/store/bmc82kpr11gpk1xy7d3l6hy37cs58i7d-rustc-1.54.0-src.tar.xz.drv failed
View build log at '/var/log/guix/drvs/bm/c82kpr11gpk1xy7d3l6hy37cs58i7d-rustc-1.54.0-src.tar.xz.drv'.
cannot build derivation `/gnu/store/nv3l1lyx4x33gx80pxk8033g01b6bv0m-rust-1.54.0.drv': 1 dependencies couldn't be built
guix build: error: build of `/gnu/store/nv3l1lyx4x33gx80pxk8033g01b6bv0m-rust-1.54.0.drv' failed
--8<---------------cut here---------------end--------------->8---

Perhaps this failure was badly handled in the parallel build an reported
as a timeout?  I'm just guessing, but reporting it as it may be at
cause.

Thanks,

Maxim