foreign arch build taking too long `guix build -s armhf-linux help`

foreign arch build taking too long `guix build -s armhf-linux help`

[Giovanni Biscuolo]

[-- Attachment #1: Type: text/plain, Size: 2362 bytes --]

Hi Guix!

I've a x86_64 build machine I configured to build for arm and aarch
architetures this way:

--8<---------------cut here---------------start------------->8---

 (service qemu-binfmt-service-type
          (qemu-binfmt-configuration
           (platforms (lookup-qemu-platforms "arm" "aarch64"))
           (guix-support? #t)))

 (modify-services %base-services
   (guix-service-type config =>
     		 (guix-configuration
     		  (inherit config)
                       (use-substitutes? #f)
                       (authorized-keys
                        %authorized-guix-keys)))))))

--8<---------------cut here---------------end--------------->8---

This should enable transparent build for another architecture, using no
substitutes.

To test a build I did `guix build -s armhf-linux help`; after several
hours I'm still waiting the build to complete, it seems stuck at this
build stage:

--8<---------------cut here---------------start------------->8---
[...]
/gnu/store/zxgm9xn0dgqfp6dxynklqzr27957qkis-make-boot0-4.2.1/bin/make  subdir=stdlib -C stdlib ..=../ others
make[2]: Entering directory '/tmp/guix-build-glibc-intermediate-2.28.drv-0/glibc-2.28/stdlib'
--8<---------------cut here---------------end--------------->8---

and this process:

--8<---------------cut here---------------start------------->8---
 /gnu/store/sw2rrqmjij73wcy3ajd47ypvmzh12yz6-qemu-3.1.0/bin/qemu-arm /gnu/store/43ljk0rxwxl4c9msr4b9xwihhw6fak8v-bootstrap-binaries-0/bin/rm -f  /tmp/guix-build-glibc-intermediate-2.28.drv-0/build/stdlib/drand48.os.dt
--8<---------------cut here---------------end--------------->8---

The process run time taken from `top` now is 796:50.68 :-O

That file is still there and it seems it's not removed:

--8<---------------cut here---------------start------------->8---
~$ sudo ls -lah /tmp/guix-build-glibc-intermediate-2.28.drv-0/build/stdlib/drand48.os.dt
-rw-r--r-- 1 guixbuilder01 guixbuild 5.7K Jun 11 04:02 /tmp/guix-build-glibc-intermediate-2.28.drv-0/build/stdlib/drand48.os.dt
--8<---------------cut here---------------end--------------->8---

Looking at /var/log/guix-daemon.log does not help.

Any hint about what's going on here and/or understanding why that
process is stuck removing that file?

Thanks! Gio'.

-- 
Giovanni Biscuolo

Xelera IT Infrastructures

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: foreign arch build taking too long `guix build -s armhf-linux help`

[Giovanni Biscuolo]

[-- Attachment #1: Type: text/plain, Size: 8327 bytes --]

Hello Guix,

my first foreign arch build failed after two days :-)

I still do not understand why I had two stale processes during
build (if needed see details below)...

...looking at /var/log/messages, I see this (in reverse timing ordering)
some segfaults:

--8<---------------cut here---------------start------------->8---
Jun 12 09:45:39 localhost vmunix: [1958344.658696] Code: 0f 1f 40 00 66 0f ef c0 66 0f ef c9 66 0f ef d2 66 0f ef db 48 89 f8 48 89 f9 48 81 e1 ff 0f 00 00 48 81 f9 cf 0f 00 00 77 6a <f3> 0f 6f 20 66 0f 74 e0 66 0f d7 d4 85 d2 74 04 0f bc c2 c3 48 83
Jun 12 09:45:39 localhost vmunix: [1958344.616598] svn[35664]: segfault at 0 ip 00007ffff76be6b6 sp 00007fffffffbb18 error 4 in libc-2.28.so[7ffff764b000+13f000]
[...]
Jun 12 09:21:13 localhost vmunix: [1956879.360841] Code: c7 77 83 c2 7c 83 c6 5b 4d 8d 67 60 4c 89 74 24 40 89 94 24 8c 00 00 00 89 bc 24 88 00 00 00 89 4c 24 74 45 31 db 89 74 24 70 <48> 89 44 24 10 4d 89 ee 48 89 6c 24 08 48 8b 44 24 08 48 8b 2d 2c
Jun 12 09:21:13 localhost vmunix: [1956879.304954] git[43284]: segfault at 7ffffffdeff0 ip 00007ffff7e333a3 sp 00007ffffffdefe0 error 6 in libc-2.28.so[7ffff7dd6000+13f000]
[...]
Jun 12 08:55:02 localhost vmunix: [1955307.460504] Code: f8 04 0f 87 19 03 00 00 83 c0 77 89 44 24 28 e9 8c f9 ff ff 66 0f 1f 44 00 00 48 8d 3d e1 d9 0f 00 e8 54 da ff ff 0f 1f 40 00 <89> 54 24 28 e9 e0 fb ff ff 0f 1f 80 00 00 00 00 8d 4f 5b 8d bc 3f
Jun 12 08:55:02 localhost vmunix: [1955307.385398] git[11650]: segfault at 7ffffffdefd8 ip 00007ffff7e33960 sp 00007ffffffdefb0 error 6 in libc-2.28.so[7ffff7dd6000+13f000]
[...]
Jun 12 08:31:17 localhost vmunix: [1953883.012358] Code: e6 f9 ff ff 89 c2 31 c0 85 d2 74 0a 0f b6 7c 24 0f e8 f4 fa ff ff 48 83 c4 18 c3 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 <8b> 04 25 00 00 00 00 0f 0b 0f 1f 80 00 00 00 00 48 83 ec 18 48 89
Jun 12 08:31:17 localhost vmunix: [1953882.986533] python[22829]: segfault at 0 ip 00007ffff766cb10 sp 00007fffffffb238 error 4 in _testcapi.so[7ffff766c000+6000]
--8<---------------cut here---------------end--------------->8---

do I have to memtest my machine?!?

Thanks! Gio'.

Giovanni Biscuolo <g@xelera.eu> writes:

[...]

> and this process:
>
> --8<---------------cut here---------------start------------->8---
>  /gnu/store/sw2rrqmjij73wcy3ajd47ypvmzh12yz6-qemu-3.1.0/bin/qemu-arm /gnu/store/43ljk0rxwxl4c9msr4b9xwihhw6fak8v-bootstrap-binaries-0/bin/rm -f  /tmp/guix-build-glibc-intermediate-2.28.drv-0/build/stdlib/drand48.os.dt
> --8<---------------cut here---------------end--------------->8---
>
> The process run time taken from `top` now is 796:50.68 :-O
>
> That file is still there and it seems it's not removed:
>
> --8<---------------cut here---------------start------------->8---
> ~$ sudo ls -lah /tmp/guix-build-glibc-intermediate-2.28.drv-0/build/stdlib/drand48.os.dt
> -rw-r--r-- 1 guixbuilder01 guixbuild 5.7K Jun 11 04:02 /tmp/guix-build-glibc-intermediate-2.28.drv-0/build/stdlib/drand48.os.dt
> --8<---------------cut here---------------end--------------->8---

I did "manually" delete the above mentioned file and killed the stale
process, the build went on to the next steps...

Until today I realized that build was still frozen at this stage:

--8<---------------cut here---------------start------------->8---
make[2]: Leaving directory '/tmp/guix-build-libgc-7.6.6.drv-0/gc-7.6.6'
make  check-TESTS
make[2]: Entering directory '/tmp/guix-build-libgc-7.6.6.drv-0/gc-7.6.6'
make[3]: Entering directory '/tmp/guix-build-libgc-7.6.6.drv-0/gc-7.6.6'
PASS: leaktest
PASS: hugetest
PASS: initsecondarythread_test
PASS: threadleaktest
PASS: smashtest
PASS: cordtest
PASS: subthreadcreate_test
PASS: middletest
PASS: threadkey_test
PASS: staticrootstest
PASS: disclaim_test
PASS: test_cpp
PASS: realloc_test
PASS: disclaim_bench
--8<---------------cut here---------------end--------------->8---

with this stale process never ending:

--8<---------------cut here---------------start------------->8---
/gnu/store/sw2rrqmjij73wcy3ajd47ypvmzh12yz6-qemu-3.1.0/bin/qemu-arm /tmp/guix-build-libgc-7.6.6.drv-0/gc-7.6.6/.libs/gctest
--8<---------------cut here---------------end--------------->8---

so I killed the above process and the build stopped with a failure
(sorry for the verbosity):

--8<---------------cut here---------------start------------->8---
FAIL: gctest
============================================================================
Testsuite summary for gc 7.6.6
============================================================================
# TOTAL: 15
# PASS:  14
# SKIP:  0
# XFAIL: 0
# FAIL:  1
# XPASS: 0
# ERROR: 0
============================================================================
See ./test-suite.log
Please report to https://github.com/ivmai/bdwgc/issues
============================================================================
make[3]: *** [Makefile:1855: test-suite.log] Error 1
make[3]: Leaving directory '/tmp/guix-build-libgc-7.6.6.drv-0/gc-7.6.6'
make[2]: *** [Makefile:1963: check-TESTS] Error 2
make[2]: Leaving directory '/tmp/guix-build-libgc-7.6.6.drv-0/gc-7.6.6'
make[1]: *** [Makefile:2297: check-am] Error 2
make[1]: Leaving directory '/tmp/guix-build-libgc-7.6.6.drv-0/gc-7.6.6'
make: *** [Makefile:1740: check-recursive] Error 1

Test suite failed, dumping logs.

--- ./test-suite.log --------------------------------------------------------

================================
   gc 7.6.6: ./test-suite.log
================================

# TOTAL: 15
# PASS:  14
# SKIP:  0
# XFAIL: 0
# FAIL:  1
# XPASS: 0
# ERROR: 0

.. contents:: :depth: 2

FAIL: gctest
============                  

FAIL gctest (exit status: 143)


Backtrace:
In ice-9/boot-9.scm:
 157: 12 [catch #t #<catch-closure 410f20> ...]
In unknown file:
   ?: 11 [apply-smob/1 #<catch-closure 410f20>]
In ice-9/boot-9.scm:
  63: 10 [call-with-prompt prompt0 ...]
In ice-9/eval.scm:
 432: 9 [eval # #]
In ice-9/boot-9.scm:
2401: 8 [save-module-excursion #<procedure 419c40 at ice-9/boot-9.scm:4045:3 ()>]
4050: 7 [#<procedure 419c40 at ice-9/boot-9.scm:4045:3 ()>]
1724: 6 [%start-stack load-stack #<procedure 400fa0 at ice-9/boot-9.scm:4041:10 ()>]
1729: 5 [#<procedure 420ee8 ()>]
In unknown file:
   ?: 4 [primitive-load "/gnu/store/1hcanjqmfxqlx35cajxs72f0nl4gqk47-libgc-7.6.6-guile-builder"]
In ice-9/eval.scm:
 387: 3 [eval # ()]
In srfi/srfi-1.scm:
 830: 2 [every1 #<procedure 6e1d10 at /gnu/store/in8fwhkzg0arvzavp779wssvdmxy3mp9-module-import/guix/build/gnu-build-system.scm:795:9 (expr)> ...]
In /gnu/store/in8fwhkzg0arvzavp779wssvdmxy3mp9-module-import/guix/build/gnu-build-system.scm:
 799: 1 [#<procedure 6e1d10 at /gnu/store/in8fwhkzg0arvzavp779wssvdmxy3mp9-module-import/guix/build/gnu-build-system.scm:795:9 (expr)> #]
 369: 0 [check #:target #f #:make-flags () #:tests? #t #:test-target "check" ...]

/gnu/store/in8fwhkzg0arvzavp779wssvdmxy3mp9-module-import/guix/build/gnu-build-system.scm:369:6: In procedure check:
/gnu/store/in8fwhkzg0arvzavp779wssvdmxy3mp9-module-import/guix/build/gnu-build-system.scm:369:6: Throw to key `srfi-34' with args `(#<condition &invoke-error [program: "make" arguments: ("check" "-j" "32") exit-status: 2 term-signal: #f stop-signal: #f] 9c9ee0>)'.
builder for `/gnu/store/mqp6y1xmgg2nck8a1xvp3vh45y5qch7d-libgc-7.6.6.drv' failed with exit code 1
build of /gnu/store/mqp6y1xmgg2nck8a1xvp3vh45y5qch7d-libgc-7.6.6.drv failed

View build log at '/var/log/guix/drvs/mq/p6y1xmgg2nck8a1xvp3vh45y5qch7d-libgc-7.6.6.drv.bz2'.
cannot build derivation `/gnu/store/928y067397sjyh9sa42f3wv05zaxvyz7-guile-2.2.4.drv': 1 dependencies couldn't be built
cannot build derivation `/gnu/store/xg1np010mwj2skgny05i3rlwzns3vh3g-ld-wrapper-0.drv': 1 dependencies couldn't be built
cannot build derivation `/gnu/store/xb219dmn7n85pxl8a3dwm75qb5gizg17-make-4.2.1.drv': 1 dependencies couldn't be built
cannot build derivation `/gnu/store/yp46hszc04dx2zy6kscy1zmjfg9y8flq-hello-2.10.drv': 1 dependencies couldn't be built
guix build: error: build of `/gnu/store/yp46hszc04dx2zy6kscy1zmjfg9y8flq-hello-2.10.drv' failed
--8<---------------cut here---------------end--------------->8---


-- 
Giovanni Biscuolo

Xelera IT Infrastructures

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]