Re: Flickering tests

[Andrea Corallo <acorallo@gnu.org>]

Andrea Corallo <acorallo@gnu.org> writes:

> Michael Albinus <Michael.Albinus@gmx.de> writes:
>
>> Andrea Corallo <acorallo@gnu.org> writes:
>>
>>> Hello,
>>
>> Hi Andrea,
>>
>>> lately I see some flickering tests in the testsuite.
>>>
>>> I see these tests failing irregularly on emacs-30 master and igc:
>>>
>>> dbus-test06-register-property-emits-signal
>>> dbus-test05-register-signal-with-nils
>>> dbus-test06-property-types
>>> dbus-test03-peer-interface
>>> secrets-test03-items
>>>
>>> Am I the only one?
>>
>> I don't see it here.
>>
>> Could you pls show the error messages of the failed tests?
>>
>>>   Andrea
>>
>> Best regards, Michael.
>
> Hi Michael,
>
> this what I saw on a recent master for
> dbus-test05-register-signal-with-nils and
> dbus-test05-register-signal-with-nils:
>
> Test dbus-test05-register-signal-with-nils backtrace:
>   signal(ert-test-failed (((should (equal dbus--test-signal-received '
>   ert-fail(((should (equal dbus--test-signal-received '("foo"))) :form
>   #f(compiled-function () #<bytecode 0xfe2dc592f6eb0e>)()
>   #f(compiled-function () #<bytecode 0x19b37652292aff8f>)()
>   handler-bind-1(#f(compiled-function () #<bytecode 0x19b37652292aff8f
>   ert--run-test-internal(#s(ert--test-execution-info :test #s(ert-test
>   ert-run-test(#s(ert-test :name dbus-test05-register-signal-with-nils
>   ert-run-or-rerun-test(#s(ert--stats :selector ... :tests ... :test-m
>   ert-run-tests((not (or (tag :expensive-test) (tag :unstable))) #f(co
>   ert-run-tests-batch((not (or (tag :expensive-test) (tag :unstable)))
>   ert-run-tests-batch-and-exit((not (or (tag :expensive-test) (tag :un
>   eval((ert-run-tests-batch-and-exit '(not (or (tag :expensive-test) (
>   command-line-1(("-L" ":." "-l" "ert" "--eval" "(setq treesit-extra-l
>   command-line()
>   normal-top-level()
> Test dbus-test05-register-signal-with-nils condition:
>     (ert-test-failed
>      ((should (equal dbus--test-signal-received '("foo"))) :form
>       (equal
>        ((":1.171874" "/org/gtk/vfs/mount/1" "foo.tar.gz"
>          "archive:host=file%253A%252F%252F%252Fhome%252Fandcor03%252Femacs-ci%252Femacs-ci-repo1%252Ftest%252Flisp%252Fnet%252Ftramp-archive-resources%252Ffoo.tar.gz"
>          ""
>          ". GThemedIcon drive-removable-media drive-removable drive drive-removable-media-symbolic drive-removable-symbolic drive-symbolic"
>          ". GThemedIcon drive-removable-media-symbolic drive-removable-symbolic drive-symbolic drive-removable-media drive-removable drive"
>          "" t ... ...))
>        ("foo"))
>       :value nil :explanation
>       (list-elt 0
>                 (different-types
>                  (":1.171874" "/org/gtk/vfs/mount/1" "foo.tar.gz"
>                   "archive:host=file%253A%252F%252F%252Fhome%252Fandcor03%252Femacs-ci%252Femacs-ci-repo1%252Ftest%252Flisp%252Fnet%252Ftramp-archive-resources%252Ffoo.tar.gz"
>                   ""
>                   ". GThemedIcon drive-removable-media drive-removable drive drive-removable-media-symbolic drive-removable-symbolic drive-symbolic"
>                   ". GThemedIcon drive-removable-media-symbolic drive-removable-symbolic drive-symbolic drive-removable-media drive-removable drive"
>                   "" t ... ...)
>                  "foo"))))
>    FAILED  12/17  dbus-test05-register-signal-with-nils (0.112019 sec) at lisp/net/dbus-tests.el:795
>
> Test dbus-test06-register-property-emits-signal backtrace:
>   signal(ert-test-failed (((should (equal dbus--test-signal-received `
>   ert-fail(((should (equal dbus--test-signal-received `(,dbus--test-in
>   #f(compiled-function () #<bytecode -0xdd8901871824899>)()
>   #f(compiled-function () #<bytecode 0x19b37652292aff8f>)()
>   handler-bind-1(#f(compiled-function () #<bytecode 0x19b37652292aff8f
>   ert--run-test-internal(#s(ert--test-execution-info :test #s(ert-test
>   ert-run-test(#s(ert-test :name dbus-test06-register-property-emits-s
>   ert-run-or-rerun-test(#s(ert--stats :selector ... :tests ... :test-m
>   ert-run-tests((not (or (tag :expensive-test) (tag :unstable))) #f(co
>   ert-run-tests-batch((not (or (tag :expensive-test) (tag :unstable)))
>   ert-run-tests-batch-and-exit((not (or (tag :expensive-test) (tag :un
>   eval((ert-run-tests-batch-and-exit '(not (or (tag :expensive-test) (
>   command-line-1(("-L" ":." "-l" "ert" "--eval" "(setq treesit-extra-l
>   command-line()
>   normal-top-level()
> Test dbus-test06-register-property-emits-signal condition:
>     (ert-test-failed
>      ((should (equal dbus--test-signal-received `...)) :form
>       (equal ("org.gnu.Emacs.TestDBus")
>              ("org.gnu.Emacs.TestDBus.Interface" (...) nil))
>       :value nil :explanation
>       (proper-lists-of-different-length 1 3 ("org.gnu.Emacs.TestDBus")
>                                         ("org.gnu.Emacs.TestDBus.Interface"
>                                          (...) nil)
>                                         first-mismatch-at 0)))
>    FAILED  15/17  dbus-test06-register-property-emits-signal (0.008444 sec) at lisp/net/dbus-tests.el:1163
>
>   Andrea

Mmhh, I've the impression this happen on my system only when it's under
heavy load.  I've 64 cores and if I "make check -j64" I've say 50%
probability to hit these, but if I run only:

./src/emacs -batch -l ert -l ./test/lisp/net/dbus-tests.el -f ert-run-tests-batch-and-exit

they all pass.  Perhaps some timeout need to be adjusted?

  Andrea