bug#26936: grub_cmd_set_date test sometimes fail

bug#26936: grub_cmd_set_date test sometimes fail

[Maxim Cournoyer]

[-- Attachment #1: Type: text/plain, Size: 282 bytes --]

Hello,

I've been encountering this test failure a couple times (3 out of 4 last
build I've tried) recently on my GuixSD x86_64 system when building grub
version 2.02.

This test failure is not deterministic. The Guix automated build farm
has yet to stumble on that failure.

Maxim

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

bug#26936: grub_cmd_set_date test sometimes fail

[Mark H Weaver]

Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:

> I've been encountering this test failure a couple times (3 out of 4 last
> build I've tried) recently on my GuixSD x86_64 system when building grub
> version 2.02.

I've encountered the same problem on my x86_64 system.

      Mark

bug#26936: grub_cmd_set_date test sometimes fail

[Maxim Cournoyer]

Mark H Weaver <mhw@netris.org> writes:

> Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:
>
>> I've been encountering this test failure a couple times (3 out of 4 last
>> build I've tried) recently on my GuixSD x86_64 system when building grub
>> version 2.02.
>
> I've encountered the same problem on my x86_64 system.
>
>       Mark

Thanks for raising the flag as well. I was starting to wonder if there
was something about my system which kept creating race conditions that
only it would suffer about, since many failures I've been finding in the
passed weeks were never triggered on hydra.

Maxim

bug#26936: grub_cmd_set_date test sometimes fail

[Ludovic Courtès]

Hello,

Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:

> I've been encountering this test failure a couple times (3 out of 4 last
> build I've tried) recently on my GuixSD x86_64 system when building grub
> version 2.02.
>
> This test failure is not deterministic. The Guix automated build farm
> has yet to stumble on that failure.

I can reproduce it after a few tries.  Upon failure, the
‘test-suite.log’ file reads:

--8<---------------cut here---------------start------------->8---
FAIL: grub_cmd_set_date
=======================

qemu-system-i386: Trying to execute code outside RAM or ROM at 0xff62f000
This usually means one of the following happened:

(1) You told QEMU to execute a kernel for the wrong machine type, and it crashed on startup (eg trying to run a raspberry pi kernel on a versatilepb QEMU machine)
(2) You didn't give QEMU a kernel or BIOS filename at all, and QEMU executed a ROM full of no-op instructions until it fell off the end
(3) Your guest kernel has a bug and crashed by jumping off into nowhere

This is almost always one of the first two, so check your command line and that you are using the right type of kernel for this machine.
If you think option (3) is likely then you can try debugging your guest with the -d debug options; in particular -d guest_errors will cause the log to include a dump of the guest register state at this point.

Execution cannot continue; stopping here.

Test failed: 2017-06-02 19:03:48 Friday
--8<---------------cut here---------------end--------------->8---

If we add “set -x” in the ‘grub-shell’ script, which is used by the
test, we see that it invokes QEMU like this:

--8<---------------cut here---------------start------------->8---
+ ./grub-mkrescue --output=/tmp/guix-build-grub-2.02.drv-0/tmp.cwW8BQx8t8 --override-directory=./grub-core --rom-directory=/tmp/guix-build-grub-2.02.drv-0/tmp.d2FE5o8cWd --locale-directory=./po --themes-directory=./themes /boot/grub/grub.cfg=/tmp/guix-build-grub-2.02.drv-0/tmp.IxQUpfc7pT /boot/grub/testcase.cfg=/tmp/guix-build-grub-2.02.drv-0/tmp.v4GOg503yw
+ '[' xcd = xhd ']'
+ '[' xcd = xcd ']'
+ '[' i386-pc = arm64-efi ']'
+ '[' i386-pc = arm-efi ']'
+ '[' i386-pc = powerpc-ieee1275 ']'
+ device='cdrom '
+ bootdev='-boot d'
+ '[' xcd = xfd ']'
+ '[' xcd = xqemu ']'
+ '[' xcd = xmipsel_qemu ']'
+ '[' xcd = xmipsel_fulong2e ']'
+ '[' xcd = xmips_qemu ']'
+ '[' xcd = xcoreboot ']'
+ '[' i386-pc = mipsel-arc ']'
+ '[' xcd = xnet ']'
+ '[' xcd = xemu ']'
+ timeout -s KILL 60 qemu-system-i386 -nographic -monitor file:/dev/null -serial file:/dev/stdout -cdrom /tmp/guix-build-grub-2.02.drv-0/tmp.cwW8BQx8t8 -boot d
--8<---------------cut here---------------end--------------->8---

It would seem that the test uses unique files, which would rule out the
possibility of a race condition when running tests with -jN.

However, running “make check -j20” led to another failure, in
‘grub_cmd_sleep’:

--8<---------------cut here---------------start------------->8---
Interval not in range 1496434911-1496434898 != 10
--8<---------------cut here---------------end--------------->8---

All in all, I’m tempted to think this has to do with running the tests
in parallel.

As a stop-gap measure, I’ve committed a change to run tests
sequentially.  With this “guix build grub --rounds=5” passes for on my
x86_64 laptop.  Let’s reopen the bug if we find that’s not enough.

Ludo’.

bug#26936: grub_cmd_set_date test sometimes fail

[Leo Famulari]

[-- Attachment #1: Type: text/plain, Size: 612 bytes --]

On Sat, Jun 03, 2017 at 12:54:46AM +0200, Ludovic Courtès wrote:
> FAIL: grub_cmd_set_date
 
[...]

> All in all, I’m tempted to think this has to do with running the tests
> in parallel.
> 
> As a stop-gap measure, I’ve committed a change to run tests
> sequentially.  With this “guix build grub --rounds=5” passes for on my
> x86_64 laptop.  Let’s reopen the bug if we find that’s not enough.

This test just failed for me on x86_64, even with #:parallel-tests? #f.

So, perhaps it's not an inter-test race, or perhaps the tests are still
running in parallel, or some other thing.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

bug#26936: grub_cmd_set_date test sometimes fail

[Leo Famulari]

[-- Attachment #1.1: Type: text/plain, Size: 786 bytes --]

On Fri, Jun 23, 2017 at 03:30:10PM -0400, Leo Famulari wrote:
> On Sat, Jun 03, 2017 at 12:54:46AM +0200, Ludovic Courtès wrote:
> > FAIL: grub_cmd_set_date
> 
> > All in all, I’m tempted to think this has to do with running the tests
> > in parallel.
> > 
> > As a stop-gap measure, I’ve committed a change to run tests
> > sequentially.  With this “guix build grub --rounds=5” passes for on my
> > x86_64 laptop.  Let’s reopen the bug if we find that’s not enough.
> 
> This test just failed for me on x86_64, even with #:parallel-tests? #f.
> 
> So, perhaps it's not an inter-test race, or perhaps the tests are still
> running in parallel, or some other thing.

Spurred by another report of this test failing, here's a patch to
disable it completely.

[-- Attachment #1.2: 0001-gnu-grub-Disable-a-flaky-test.patch --]
[-- Type: text/plain, Size: 1551 bytes --]

From 71299a035f793c3e2ac723b326cd881e97df6274 Mon Sep 17 00:00:00 2001
From: Leo Famulari <leo@famulari.name>
Date: Sat, 24 Jun 2017 13:17:37 -0400
Subject: [PATCH] gnu: grub: Disable a flaky test.

* gnu/packages/bootloaders.scm (grub)[arguments]: Add 'disable-flaky-test' phase
and run the tests in parallel.
---
 gnu/packages/bootloaders.scm | 14 +++++++++-----
 1 file changed, 9 insertions(+), 5 deletions(-)

diff --git a/gnu/packages/bootloaders.scm b/gnu/packages/bootloaders.scm
index dec462b0c..9ae617528 100644
--- a/gnu/packages/bootloaders.scm
+++ b/gnu/packages/bootloaders.scm
@@ -91,11 +91,15 @@
                      ;; Make the font visible.
                      (copy-file (assoc-ref inputs "unifont") "unifont.bdf.gz")
                      (system* "gunzip" "unifont.bdf.gz")
-                     #t)))
-
-       ;; We suspect there are race conditions when running tests in parallel:
-       ;; <https://bugs.gnu.org/26936>.
-       #:parallel-tests? #f))
+                     #t))
+                  (add-before 'check 'disable-flaky-test
+                    (lambda _
+                      ;; This test is unreliable. For more information, see:
+                      ;; <https://bugs.gnu.org/26936>.
+                      (substitute* "Makefile.in"
+                        (("grub_cmd_date grub_cmd_set_date grub_cmd_sleep")
+                          "grub_cmd_date grub_cmd_sleep"))
+                      #t)))))
     (inputs
      `(("gettext" ,gettext-minimal)
 
-- 
2.13.1


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

bug#26936: grub_cmd_set_date test sometimes fail

[Marius Bakke]

[-- Attachment #1: Type: text/plain, Size: 1191 bytes --]

Leo Famulari <leo@famulari.name> writes:

> On Fri, Jun 23, 2017 at 03:30:10PM -0400, Leo Famulari wrote:
>> On Sat, Jun 03, 2017 at 12:54:46AM +0200, Ludovic Courtès wrote:
>> > FAIL: grub_cmd_set_date
>> 
>> > All in all, I’m tempted to think this has to do with running the tests
>> > in parallel.
>> > 
>> > As a stop-gap measure, I’ve committed a change to run tests
>> > sequentially.  With this “guix build grub --rounds=5” passes for on my
>> > x86_64 laptop.  Let’s reopen the bug if we find that’s not enough.
>> 
>> This test just failed for me on x86_64, even with #:parallel-tests? #f.
>> 
>> So, perhaps it's not an inter-test race, or perhaps the tests are still
>> running in parallel, or some other thing.
>
> Spurred by another report of this test failing, here's a patch to
> disable it completely.
> From 71299a035f793c3e2ac723b326cd881e97df6274 Mon Sep 17 00:00:00 2001
> From: Leo Famulari <leo@famulari.name>
> Date: Sat, 24 Jun 2017 13:17:37 -0400
> Subject: [PATCH] gnu: grub: Disable a flaky test.
>
> * gnu/packages/bootloaders.scm (grub)[arguments]: Add 'disable-flaky-test' phase
> and run the tests in parallel.

LGTM.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

bug#26936: grub_cmd_set_date test sometimes fail

[Leo Famulari]

[-- Attachment #1: Type: text/plain, Size: 482 bytes --]

On Sun, Jun 25, 2017 at 12:55:51PM +0200, Marius Bakke wrote:
> Leo Famulari <leo@famulari.name> writes:
> > Subject: [PATCH] gnu: grub: Disable a flaky test.
> >
> > * gnu/packages/bootloaders.scm (grub)[arguments]: Add 'disable-flaky-test' phase
> > and run the tests in parallel.
> 
> LGTM.

Thanks for the review! Pushed as 23a53a95f1f5f44df401c120276eebf196ba15d2.

Also, I reported it upstream:

https://lists.gnu.org/archive/html/bug-grub/2017-06/msg00012.html

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]