bug#23811: shepherd: tests/respawn-throttling.sh fails sporadically

bug#23811: shepherd: tests/respawn-throttling.sh fails sporadically

[Mark H Weaver]

On core-updates, both shepherd and dmd failed on i686 and armhf in the
same test:

tests/respawn-throttling.sh in shepherd:

  http://hydra.gnu.org/build/1241860 (shepherd-i686)
  http://hydra.gnu.org/build/1246005 (shepherd-armhf)

and tests/respawn.sh in dmd:

  http://hydra.gnu.org/build/1245596 (dmd-i686)
  http://hydra.gnu.org/build/1248237 (dmd-armhf)

This has been happening for a long time, and although it happens more
frequently on i686 and armhf, I've archived failure logs from hydra for
the same failures on mips64el in both shepherd and dmd, and it also
happened at least once with dmd on x86_64.  I can provide these logs if
requested.

      Mark

bug#30299: [core-updates] shepherd fails tests on all systems except x86_64

[Mark H Weaver]

On core-updates, Hydra has been unable to successfully build 'shepherd'
on any system except x86_64-linux.  I can also report that on my
mips64el-linux GuixSD system, which is running something close to
'core-updates', I had to disable tests on shepherd in order to build it.
I don't know about aarch64-linux.

These are the tests that commonly fail:

  FAIL: tests/respawn.sh
  FAIL: tests/respawn-throttling.sh
  FAIL: tests/basic.sh

      Mark

bug#30299: [core-updates] shepherd fails tests on all systems except x86_64

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 839 bytes --]

On Tue, Jan 30, 2018 at 10:07:35PM -0500, Mark H Weaver wrote:
> On core-updates, Hydra has been unable to successfully build 'shepherd'
> on any system except x86_64-linux.  I can also report that on my
> mips64el-linux GuixSD system, which is running something close to
> 'core-updates', I had to disable tests on shepherd in order to build it.
> I don't know about aarch64-linux.
> 
> These are the tests that commonly fail:
> 
>   FAIL: tests/respawn.sh
>   FAIL: tests/respawn-throttling.sh
>   FAIL: tests/basic.sh
> 
>       Mark

Shepherd built successfully on aarch64-linux, and again with '--check'.

-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

bug#30299: [core-updates] shepherd fails tests on all systems except x86_64

[Ludovic Courtès]

Hello,

Mark H Weaver <mhw@netris.org> skribis:

> On core-updates, Hydra has been unable to successfully build 'shepherd'
> on any system except x86_64-linux.  I can also report that on my
> mips64el-linux GuixSD system, which is running something close to
> 'core-updates', I had to disable tests on shepherd in order to build it.
> I don't know about aarch64-linux.
>
> These are the tests that commonly fail:
>
>   FAIL: tests/respawn.sh
>   FAIL: tests/respawn-throttling.sh
>   FAIL: tests/basic.sh

This is a non-deterministic failure.  I could reproduce the
tests/basic.sh one and it is fixed by this:

  https://git.savannah.gnu.org/cgit/shepherd.git/commit/?id=cc9564586729a5bb90dd5d2722b543fdde9ab821

I’ll roll a new Shepherd release soon and update the package.

The rest may be a duplicate of 23811, though I couldn’t reproduce it.

Thanks,
Ludo’.

bug#30299: [core-updates] shepherd fails tests on all systems except x86_64

[Mark H Weaver]

ludo@gnu.org (Ludovic Courtès) writes:

> Mark H Weaver <mhw@netris.org> skribis:
>
>> On core-updates, Hydra has been unable to successfully build 'shepherd'
>> on any system except x86_64-linux.  I can also report that on my
>> mips64el-linux GuixSD system, which is running something close to
>> 'core-updates', I had to disable tests on shepherd in order to build it.
>> I don't know about aarch64-linux.
>>
>> These are the tests that commonly fail:
>>
>>   FAIL: tests/respawn.sh
>>   FAIL: tests/respawn-throttling.sh
>>   FAIL: tests/basic.sh
>
> This is a non-deterministic failure.  I could reproduce the
> tests/basic.sh one and it is fixed by this:
>
>   https://git.savannah.gnu.org/cgit/shepherd.git/commit/?id=cc9564586729a5bb90dd5d2722b543fdde9ab821

Thank you!

> I’ll roll a new Shepherd release soon and update the package.

For now, I added your patch to the shepherd package in core-updates, in
commit f2d2ee42f168909f27c0c3b6532ef16febfd3b86.

      Mark

bug#30299: [core-updates] shepherd fails tests on all systems except x86_64

[Mark H Weaver]

Mark H Weaver <mhw@netris.org> writes:

> On core-updates, Hydra has been unable to successfully build 'shepherd'
> on any system except x86_64-linux.  I can also report that on my
> mips64el-linux GuixSD system, which is running something close to
> 'core-updates', I had to disable tests on shepherd in order to build it.
> I don't know about aarch64-linux.
>
> These are the tests that commonly fail:
>
>   FAIL: tests/respawn.sh
>   FAIL: tests/respawn-throttling.sh
>   FAIL: tests/basic.sh

FYI, after 18 failed attempts, Hydra finally built shepherd on i686
successfully on the 19th try:

  https://hydra.gnu.org/build/2425454#tabs-buildsteps

Now that I've added Ludovic's patch, hopefully it will require fewer
attempts this time :)

       Mark

bug#30299: [core-updates] shepherd fails tests on all systems except x86_64

[Ludovic Courtès]

Mark H Weaver <mhw@netris.org> skribis:

> ludo@gnu.org (Ludovic Courtès) writes:
>
>> Mark H Weaver <mhw@netris.org> skribis:
>>
>>> On core-updates, Hydra has been unable to successfully build 'shepherd'
>>> on any system except x86_64-linux.  I can also report that on my
>>> mips64el-linux GuixSD system, which is running something close to
>>> 'core-updates', I had to disable tests on shepherd in order to build it.
>>> I don't know about aarch64-linux.
>>>
>>> These are the tests that commonly fail:
>>>
>>>   FAIL: tests/respawn.sh
>>>   FAIL: tests/respawn-throttling.sh
>>>   FAIL: tests/basic.sh
>>
>> This is a non-deterministic failure.  I could reproduce the
>> tests/basic.sh one and it is fixed by this:
>>
>>   https://git.savannah.gnu.org/cgit/shepherd.git/commit/?id=cc9564586729a5bb90dd5d2722b543fdde9ab821
>
> Thank you!
>
>> I’ll roll a new Shepherd release soon and update the package.
>
> For now, I added your patch to the shepherd package in core-updates, in
> commit f2d2ee42f168909f27c0c3b6532ef16febfd3b86.

Thanks, and sorry for not acting earlier!  (I was waiting for a reply
from the Translation Project to make the new release, but that hasn’t
happened yet.)

Ludo’.

bug#30299: [core-updates] shepherd fails tests on all systems except x86_64

[Mark H Weaver]

Hi Ludovic,

Mark H Weaver <mhw@netris.org> writes:

> ludo@gnu.org (Ludovic Courtès) writes:
>
>> Mark H Weaver <mhw@netris.org> skribis:
>>
>>> On core-updates, Hydra has been unable to successfully build 'shepherd'
>>> on any system except x86_64-linux.  I can also report that on my
>>> mips64el-linux GuixSD system, which is running something close to
>>> 'core-updates', I had to disable tests on shepherd in order to build it.
>>> I don't know about aarch64-linux.
>>>
>>> These are the tests that commonly fail:
>>>
>>>   FAIL: tests/respawn.sh
>>>   FAIL: tests/respawn-throttling.sh
>>>   FAIL: tests/basic.sh
>>
>> This is a non-deterministic failure.  I could reproduce the
>> tests/basic.sh one and it is fixed by this:
>>
>>   https://git.savannah.gnu.org/cgit/shepherd.git/commit/?id=cc9564586729a5bb90dd5d2722b543fdde9ab821
>
> For now, I added your patch to the shepherd package in core-updates, in
> commit f2d2ee42f168909f27c0c3b6532ef16febfd3b86.

With your patch, Hydra built shepherd successfully on i686-linux on the
first try, which is much better than the 19th try :)

However, on armhf-linux, three tests failed: respawn.sh,
respawn-throttling.sh, and pid-file.sh.

  https://hydra.gnu.org/build/2499835

We should probably arrange for test-suite.log to be printed when "make
check" fails.  We could add this functionality to selected packages like
shepherd and guix the next time we update them, and maybe consider
adding something generic to gnu-build-system's check phase in the next
core-updates cycle.  What do you think?

      Mark

--8<---------------cut here---------------start------------->8---
starting phase `check'
make  check-am
make[1]: Entering directory '/tmp/guix-build-shepherd-0.3.2.drv-0/shepherd-0.3.2'
make  check-TESTS
make[2]: Entering directory '/tmp/guix-build-shepherd-0.3.2.drv-0/shepherd-0.3.2'
make[3]: Entering directory '/tmp/guix-build-shepherd-0.3.2.drv-0/shepherd-0.3.2'
PASS: tests/misbehaved-client.sh
FAIL: tests/respawn.sh
PASS: tests/no-home.sh
FAIL: tests/pid-file.sh
PASS: tests/status-sexp.sh
PASS: tests/sigint.sh
PASS: tests/basic.sh
FAIL: tests/respawn-throttling.sh
============================================================================
Testsuite summary for GNU Shepherd 0.3.2
============================================================================
# TOTAL: 8
# PASS:  5
# SKIP:  0
# XFAIL: 0
# FAIL:  3
# XPASS: 0
# ERROR: 0
============================================================================
See ./test-suite.log
Please report to bug-guix@gnu.org
============================================================================
make[3]: *** [Makefile:1220: test-suite.log] Error 1
make[3]: Leaving directory '/tmp/guix-build-shepherd-0.3.2.drv-0/shepherd-0.3.2'
make[2]: *** [Makefile:1328: check-TESTS] Error 2
make[2]: Leaving directory '/tmp/guix-build-shepherd-0.3.2.drv-0/shepherd-0.3.2'
make[1]: *** [Makefile:1527: check-am] Error 2
make[1]: Leaving directory '/tmp/guix-build-shepherd-0.3.2.drv-0/shepherd-0.3.2'
make: *** [Makefile:1529: check] Error 2
phase `check' failed after 29.6 seconds
builder for `/gnu/store/sk0l3ll4x1ddn9zwxmfqjipr578hiqx1-shepherd-0.3.2.drv' failed with exit code 1
@ build-failed /gnu/store/sk0l3ll4x1ddn9zwxmfqjipr578hiqx1-shepherd-0.3.2.drv - 1 builder for `/gnu/store/sk0l3ll4x1ddn9zwxmfqjipr578hiqx1-shepherd-0.3.2.drv' failed with exit code 1
--8<---------------cut here---------------end--------------->8---

bug#30299: [core-updates] shepherd fails tests on all systems except x86_64

[Ludovic Courtès]

Hi Mark,

Mark H Weaver <mhw@netris.org> skribis:

> Mark H Weaver <mhw@netris.org> writes:

[...]

>>> This is a non-deterministic failure.  I could reproduce the
>>> tests/basic.sh one and it is fixed by this:
>>>
>>>   https://git.savannah.gnu.org/cgit/shepherd.git/commit/?id=cc9564586729a5bb90dd5d2722b543fdde9ab821
>>
>> For now, I added your patch to the shepherd package in core-updates, in
>> commit f2d2ee42f168909f27c0c3b6532ef16febfd3b86.
>
> With your patch, Hydra built shepherd successfully on i686-linux on the
> first try, which is much better than the 19th try :)

We’re making progress.  :-)

> However, on armhf-linux, three tests failed: respawn.sh,
> respawn-throttling.sh, and pid-file.sh.
>
>   https://hydra.gnu.org/build/2499835

I’ll try to reproduce the failure here.

> We should probably arrange for test-suite.log to be printed when "make
> check" fails.  We could add this functionality to selected packages like
> shepherd and guix the next time we update them, and maybe consider
> adding something generic to gnu-build-system's check phase in the next
> core-updates cycle.  What do you think?

Definitely.  We discussed it before and I think it’s a good idea.  I
wanted to add it to ‘guix’ as a starting point but never got around to
doing it.

Thanks,
Ludo’.

bug#30299: [core-updates] shepherd fails tests on all systems except x86_64

[Ludovic Courtès]

Hello,

Mark H Weaver <mhw@netris.org> skribis:

> However, on armhf-linux, three tests failed: respawn.sh,
> respawn-throttling.sh, and pid-file.sh.
>
>   https://hydra.gnu.org/build/2499835

(Similar issue on aarch64:
<https://berlin.guixsd.org/build/2419870/log/raw>.  Though of course it
passed on the 2nd and 3rd attempts…)

I was able to reproduce a tests/respawn.sh failure on hardware (ARMv7).
The issue is that a service is not respawned, and the log shows:

--8<---------------cut here---------------start------------->8---
+ assert_killed_service_is_respawned t-service2-pid-695
++ cat t-service2-pid-695
+ old_pid=789
+ rm t-service2-pid-695
+ kill 789
+ wait_for_file t-service2-pid-695
+ i=0
+ test -f t-service2-pid-695
+ test 0 -lt 20
+ sleep 0.3
++ expr 0 + 1

[...]

2018-02-16 11:13:31 Service root has been started.
2018-02-16 11:13:32 Service test1 has been started.
2018-02-16 11:13:34 Service test2 has been started.
2018-02-16 11:13:35 Respawning test1.
2018-02-16 11:13:35 Service test1 has been started.
2018-02-16 11:13:36 Respawning test2.
2018-02-16 11:13:37 Service test2 has been started.
2018-02-16 11:13:37 Respawning test1.
2018-02-16 11:13:37 Service test1 has been started.
2018-02-16 11:13:38 Respawning test2.
2018-02-16 11:13:43 Service test2 could not be started.
--8<---------------cut here---------------end--------------->8---

So SIGCHLD was correctly delivered, but somehow restarting that service
didn’t work (its PID file didn’t show up again; the 5 seconds between
“Respawning” and “could not be started” correspond to the delay in
‘read-pid-file’ in (shepherd service)).  

These test failures seem to be more frequent when the machine is loaded.

Ludo’.

bug#30299: [core-updates] shepherd fails tests on all systems except x86_64

[Andreas Enge]

Hello,

I changed the severity to "serious", since this bug prevents installing
GuixSD on arm, or creating an installation image. Also, it is now present
on master instead of core-updates.

Andreas

bug#30299: [core-updates] shepherd fails tests on all systems except x86_64

[Brice Waegeneire]

Hello,

Does this bug is still relevant?
It was reported on core-updates 2 years ago with shepherd 0.3.2.
I can't see any CI failures[0] related to a test failing since cuirass 
was setup, the only time the build of shepherd failed[1] it wasn't due 
to a test.
I wasn't able to reproduce the failing test with some building rounds on 
armhf and i686.

> I changed the severity to "serious", since this bug prevents installing
> GuixSD on arm, or creating an installation image. Also, it is now 
> present
> on master instead of core-updates.

As far as I know this isn't the case anymore, so at least the priority 
should be lowered.

[0]: http://ci.guix.gnu.org/search?query=shepherd
[1]: http://ci.guix.gnu.org/build/1312246/details

Brice.

bug#30299: [core-updates] shepherd fails tests on all systems except x86_64

[Marius Bakke]

[-- Attachment #1: Type: text/plain, Size: 551 bytes --]

Brice Waegeneire <brice@waegenei.re> writes:

> Hello,
>
> Does this bug is still relevant?
> It was reported on core-updates 2 years ago with shepherd 0.3.2.
> I can't see any CI failures[0] related to a test failing since cuirass 
> was setup, the only time the build of shepherd failed[1] it wasn't due 
> to a test.
> I wasn't able to reproduce the failing test with some building rounds on 
> armhf and i686.

Thank you for doing bug triage Brice.  I'm closing the issue as we
haven't had problems building the shepherd in a while.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

bug#23811: shepherd: tests/respawn-throttling.sh fails sporadically

[Brice Waegeneire]

Hello,

#23811 (merged with #28588) are about the same tests failing as
the recently closed #30299 and were reported 4 et 3 years.
I suggest to close then too.

Brice.

bug#23811: shepherd: tests/respawn-throttling.sh fails sporadically

[Marius Bakke]

[-- Attachment #1: Type: text/plain, Size: 237 bytes --]

Brice Waegeneire <brice@waegenei.re> writes:

> Hello,
>
> #23811 (merged with #28588) are about the same tests failing as
> the recently closed #30299 and were reported 4 et 3 years.
> I suggest to close then too.

Good catch, closing!

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]