bug#24503: unstable test results (on hydra)

bug#24503: unstable test results (on hydra)

[Glenn Morris]

Package: emacs
Version: 25.1

On hydra (at least), some tests fail in one iteration and succeed in the
next, for no clear reason. This cause noise on the emacs-buildstatus
mailing list and makes it harder to spot real problems.

Someone (TM) should look into this and try to figure out if the tests
really have a problem, or if the problematic ones should just be
disabled on hydra.

eieio-test-37-obsolete-name-in-constructor is one I have noticed several
times.

Failed: http://hydra.nixos.org/build/40880158
Passed: http://hydra.nixos.org/build/40890883

There were no relevant changes between those builds.

bug#24503: unstable test results (on hydra)

[Glenn Morris]

Hi Stefan,

Do you have any idea why eieio-test-37-obsolete-name-in-constructor
sometimes (randomly?) fails with:

(eieio--testing "toto") -> [eieio-class-tag--eieio--testing]

rather than ("toto" 2) ?

Eg http://hydra.nixos.org/build/46213612/log/raw

bug#24503: unstable test results (on hydra)

[Stefan Monnier]

> Do you have any idea why eieio-test-37-obsolete-name-in-constructor
> sometimes (randomly?) fails with:
> (eieio--testing "toto") -> [eieio-class-tag--eieio--testing]
> rather than ("toto" 2) ?

Hmm... not off the top of my head no.  I'll have to dig deeper to give
you a better answer,


        Stefan

bug#24503: unstable test results (on hydra)

[Glenn Morris]

I've failed to reproduce this issue myself in 1000+ trials. So maybe
it is some hydra-specific weirdness, though I can't see how.
If all else fails, the test can just be skipped there.

bug#24503: unstable test results (on hydra)

[Stefan Monnier]

> I've failed to reproduce this issue myself in 1000+ trials. So maybe
> it is some hydra-specific weirdness, though I can't see how.
> If all else fails, the test can just be skipped there.

It could be due to a parallel-make issue where sometimes we load a .el
file and sometimes we load a .elc file depending on the relative order
of execution of the various parallel commands.


        Stefan "who still haven't looked at the actual code, so is just
                talking about very general ideas"

bug#24503: unstable test results (on hydra)

[Glenn Morris]

Stefan Monnier wrote:

> It could be due to a parallel-make issue where sometimes we load a .el
> file and sometimes we load a .elc file depending on the relative order
> of execution of the various parallel commands.

I think hydra builds are single-core.
But obviously we need to wait till you can have a proper look. :)

bug#24503: unstable test results (on hydra)

[Glenn Morris]

Another one I've seen intermittently fail:

Test eieio-test-method-order-list-6 condition:
    (error "eieio-test-methodinvoke.el: Test Failed:
    ((:STATIC C) (:STATIC C-base1) (:STATIC C-base2)) != ((:STATIC C))")

Eg http://hydra.nixos.org/build/51808781

bug#24503: unstable test results (on hydra)

[Glenn Morris]

Tests that "randomly" fail, like the ones mentioned in this report, are
a problem for downstream distributions that run "make check" and pay
attention to the results (I sometimes think they pay more attention than
Emacs does). They can waste their time investigating things that seem to
be platform-specific just because they happen to fail once on a
particular architecture.

Eg
https://bugs.debian.org/879020

So we do a disservice to the wider community with these tests that
sometimes fail. If they aren't good enough to be enabled on Emacs's own
CI system, they probably aren't good enough full-stop, and should be
disabled by default for everyone.

bug#24503: unstable test results (on hydra)

[Noam Postavsky]

Glenn Morris <rgm@gnu.org> writes:

> Tests that "randomly" fail, like the ones mentioned in this report, are
> a problem for downstream distributions that run "make check" and pay
> attention to the results (I sometimes think they pay more attention than
> Emacs does). They can waste their time investigating things that seem to
> be platform-specific just because they happen to fail once on a
> particular architecture.
>
> Eg
> https://bugs.debian.org/879020
>
> So we do a disservice to the wider community with these tests that
> sometimes fail. If they aren't good enough to be enabled on Emacs's own
> CI system, they probably aren't good enough full-stop, and should be
> disabled by default for everyone.

Hmm, I can't see any reason for these tests to fail in a
non-deterministic way.  This sounds like it could be a serious bug.  And
if happens on machines other than the Hydra ones, the bug is almost
certainly in Emacs.  I think we need to add some logging statements to
figure out what is going.  I will try to figure how to do that.

bug#24503: unstable test results (on hydra)

[Noam Postavsky]

Noam Postavsky <npostavs@users.sourceforge.net> writes:

> Hmm, I can't see any reason for these tests to fail in a
> non-deterministic way.  This sounds like it could be a serious bug.  And
> if happens on machines other than the Hydra ones, the bug is almost
> certainly in Emacs.  I think we need to add some logging statements to
> figure out what is going.  I will try to figure how to do that.

I didn't see any failures since I added tracing code [1: cfa50d30f2],
probably the tracing itself changed something which somehow prevents the
bug from triggering.

[1: cfa50d30f2]: 2017-12-03 15:39:02 -0500
  ; Tracing for eieio-test random failure (Bug#24503)
  https://git.savannah.gnu.org/cgit/emacs.git/commit/?id=cfa50d30f2876ca30158082e9a91d19e804a7e09

bug#24503: unstable test results (on hydra)

[Glenn Morris]

Version: 26.0.91

I've tagged the relevant tests as unstable and skipped them by default.
I will tag more tests as unstable if and when they come up.