bug#41654: Test of guix-1.1.0-4.bdc801e is failing on aarch64

bug#41654: Test of guix-1.1.0-4.bdc801e is failing on aarch64

[Stefan]

[-- Attachment #1: Type: text/plain, Size: 2082 bytes --]

Hi!

Building guix-1.1.0-4.bdc801e is failing on aarch64 since about two weeks on ci.guix.gnu.org. 

http://ci.guix.gnu.org/search?query=guix-1.1.0-4.bdc801e+system%3Aaarch64-linux

There is always the same failure during the test. See for example the raw log from http://ci.guix.gnu.org/build/2794788/details:

test-name: network-interface-names
…
actual-error:
+ (wrong-type-arg
+   "list-tail"
+   "Wrong type argument in position ~A (expecting ~A): ~S"
+   (1 "pair" ())
+   (()))
result: FAIL


When building locally this particular test is passing (after about three days on an SBC with 1 GB RAM). 

But unfortunately building locally I get two different failures instead:

FAIL: tests/store.scm
FAIL: tests/guix-package.sh

Here are some more details:


test-name: verify-store + check-contents
location: /tmp/guix-build-guix-1.1.0-4.bdc801e.drv-0/source/tests/store.scm:972
…
actual-error:
+ (%exception
+   #<&store-protocol-error message: "path `dtmp/guix-tests/store/bpl866rgsjyzdczlngfw8ss2lkld6bim-mirrors' is not in the store" status: 1>)
result: FAIL

Note the “dtmp/” instead of “/tmp/”. I saw the same error some weeks ago already, but shortly after that was happening, there was a substitute available. But it seems this error is reproducible in my case.


guix install: error: profile t-guix-package-541/profile is locked by another process
+ true
+ kill 1387
+ rm -f t-profile-541 t-profile-541.lock t-profile-541-1-link t-guix-package-file-541
+ rm -rf t-guix-package-541 t-home-541
rm: cannot remove 't-guix-package-541': Directory not empty
FAIL tests/guix-package.sh (exit status: 1)


This may be due to a deleted but still opened file on an NFS share. There may be an intermediate hidden .nfs… file, which may get created in such a case (“delete on last close”, “silly rename”), However, the RFC-5661 for NFS demands even if OPEN4_RESULT_PRESERVE_UNLINKED is supported, that the directory entry of an open file must not be removed in this case, thus preventing a directory removal.


[-- Attachment #2: 54pswxwm35kq47zk2dswvsq3nm55kb-guix-1.1.0-4.bdc801e.drv.bz2 --]
[-- Type: application/applefile, Size: 119 bytes --]

bug#41654: Acknowledgement (Test of guix-1.1.0-4.bdc801e is failing on aarch64)

[Stefan]

[-- Attachment #1: Type: text/plain, Size: 256 bytes --]

Hi!

The two errors when testing guix are still reproducible, now with guix-1.2.0rc2-1.0d4b1af, see the log for details.

I’m really wondering about the “dtmp/” instead of “/tmp/” bug. May this be a bug in guile itself?


Bye

Stefan


[-- Attachment #2: fwpdy5zrhd8i1h8k7qh7a2kh3wawrh-guix-1.2.0rc2-1.0d4b1af.drv.bz2 --]
[-- Type: application/x-bzip2, Size: 730115 bytes --]

bug#41654: [PATCH] tests: Ignoring incomplete clean-up on NFS share.

[Stefan]

* tests/guix-package.sh: The 'rm -rf' for clean-up inside the trap may not
succeed on an NFS share, but this should not fail the test.
---
 tests/guix-package.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tests/guix-package.sh b/tests/guix-package.sh
index 3e5fa71d20..b1c6eeffb8 100644
--- a/tests/guix-package.sh
+++ b/tests/guix-package.sh
@@ -33,7 +33,7 @@ profile="t-profile-$$"
 tmpfile="t-guix-package-file-$$"
 rm -f "$profile" "$tmpfile"
 
-trap 'rm -f "$profile" "$profile.lock" "$profile-"[0-9]* "$tmpfile"; rm -rf "$module_dir" t-home-'"$$" EXIT
+trap 'rm -f "$profile" "$profile.lock" "$profile-"[0-9]* "$tmpfile"; rm -rf "$module_dir" t-home-'"$$"' || echo  "incomplete clean-up ignored"' EXIT
 
 # Use `-e' with a non-package expression.
 ! guix package --bootstrap -e +
-- 
2.29.2

bug#41654: test-name: verify-store + check-contents failing on guix-1.2.0rc2-1.0d4b1af

[Stefan]

Hi!

The “dtmp/” issue is not new. It appeared already in these bugs: 

https://issues.guix.gnu.org/27032
https://issues.guix.gnu.org/27034
https://issues.guix.gnu.org/29676

In all cases it happens in tests/store.scm:

test-name: verify-store + check-contents

This bug happened on x86_64 and aarch64.
 
The issues 27032 and 27034 where reported when the installer used a unionfs. Probably it was hidden when switching to an overlayfs.

This issue 41654 and 29676 show the “dtmp/” error when running this test on an NFS share.


Bye

Stefan

bug#41654: test-name: verify-store + check-contents failing on guix-1.2.0rc2-1.0d4b1af

[Stefan]

Hi Ludo’!

Could you please take a look at the bug report <https://issues.guix.gnu.org/41654>?

There are two reproducible problems when building the guix-1.2.0 package. 

There is a patch already for the first of the two problems (an incomplete cleanup) at <https://issues.guix.gnu.org/41654#2>.

For the other issue with a wrong path "dtmp/guix-tests/store/", there is a summary at <https://issues.guix.gnu.org/41654#3>. This problem is existing since years and seems to be depending on the underlying file-system. It seems to appear with a tmp-store on a unionfs-fuse or NFS. In the past it got hidden by switching to overlayfs or changes on the NFS server side. 

As the CI for aarch64 is always lagging behind these issues are kind of blockers to me. Today they block the build of guix-1.2.0-8.7624ebb.


Bye

Stefan

bug#41654: test-name: verify-store + check-contents failing on guix-1.2.0rc2-1.0d4b1af

[Stefan]

Hi!

Today I thought that it should be possible to work around this bug on my aarch64 system by mounting /tmp as a tmpfs, backed by swap on a small SD card, to prevent any issues due to my NFS root. This worked for the trap-cleanup-issue. But to my surprise this didn’t work out for the “dtmp/” error.

I was wondering, if using /tmp as a tmpfs on my virtual x86_64 system would make the test fail there as well. So I did 

sudo mount -t tmpfs -o size=768M tmpfs /tmp
guix build --check --no-grafts guix@1.2.0-8.7624ebb

But there it was passing.

This is the interesting part from the log of the aarch64 system:

test-name: verify-store + check-contents
location: /tmp/guix-build-guix-1.2.0-8.7624ebb.drv-0/source/tests/store.scm:1156
source:
+ (test-assert
+   "verify-store + check-contents"
+   (with-store
+     s
+     (let* ((text (random-text))
+            (drv (build-expression->derivation
+                   s
+                   "corrupt"
+                   `(let ((out (assoc-ref %outputs "out")))
+                      (call-with-output-file
+                        out
+                        (lambda (port) (display ,text port)))
+                      #t)
+                   #:guile-for-build
+                   (package-derivation
+                     s
+                     %bootstrap-guile
+                     (%current-system))))
+            (file (derivation->output-path drv)))
+       (with-derivation-substitute
+         drv
+         text
+         (and (build-derivations s (list drv))
+              (verify-store s #:check-contents? #t)
+              (begin
+                (chmod file 420)
+                (call-with-output-file
+                  file
+                  (lambda (port) (display "corrupt!" port)))
+                #t)
+              (not (verify-store s #:check-contents? #t))
+              (delete-paths s (list file)))))))
actual-value: #f
actual-error:
+ (%exception
+   #<&store-protocol-error message: "path `dtmp/guix-tests/store/rn6xq5kaipp8aanhcnz2hvyfmr3y2laa-mirrors' is not in the store" status: 1>)
result: FAIL


Bye

Stefan