Re: Flickering tests

[Andrea Corallo <acorallo@gnu.org>]

Michael Albinus <Michael.Albinus@gmx.de> writes:

> Andrea Corallo <acorallo@gnu.org> writes:
>
> Hi Andrea,
>
>> Mmhh, I've the impression this happen on my system only when it's under
>> heavy load.  I've 64 cores and if I "make check -j64" I've say 50%
>> probability to hit these, but if I run only:
>>
>> ./src/emacs -batch -l ert -l ./test/lisp/net/dbus-tests.el -f ert-run-tests-batch-and-exit
>>
>> they all pass.  Perhaps some timeout need to be adjusted?
>
> It's not the a I believe. D-Bus tests are sending events to the D-Buson
> OS level, and receiving events which will be interpreted. They are not
> written such a way that all tests can run in parallel.
>
> If you invoke "make check -j64", it could happen that some D-Bus tests
> run in parallel, and returning events do arrive in another Emacs
> instance but the one which has send the evnt. This might be even more
> problematic, because recently (on Sunday, commit
> 4e22ef870c4b650f29c4441ac51b6a2ac506ea57) a test which registers for
> wildcard events has been added. This test resides in my local Emacs git
> for 3 months w/o problems; I've simply waited until the FSF papers of
> the author, Robert Church, were on file.
>
> What could we do? Could we exclude dbus-tests.el from running as
> simultaneous job?

Mmmh not sure is possible but also desirable, we want to be able to run
as many tests as possible even with -jN.

> Or could we simply skip D-Bus tests, if we detect that the tests run via
> simultaneous make jobs? How to check, on Lisp level?

Perhaps using some lock file as mutex for the generation of D-Bus
messages from different test files?

> We always have the option to mark this test as :unstable, so we could
> stabilize the Emacs 30 release this way. But it would be a pity.
>
> Btw, could you check whether the D-Bus tests flickery goes away if we
> tag dbus-test05-register-signal-with-nils as :unstable?

From the bunch of test I did seems this is solving, I'll mark it as
:unstable for now till a better solution is found.

Thanks

  Andrea