* Flickering tests @ 2024-07-02 9:47 Andrea Corallo 2024-07-02 12:43 ` Andrea Corallo 2024-07-02 14:28 ` Michael Albinus 0 siblings, 2 replies; 17+ messages in thread From: Andrea Corallo @ 2024-07-02 9:47 UTC (permalink / raw To: emacs-devel; +Cc: Michael Albinus Hello, lately I see some flickering tests in the testsuite. I see these tests failing irregularly on emacs-30 master and igc: dbus-test06-register-property-emits-signal dbus-test05-register-signal-with-nils dbus-test06-property-types dbus-test03-peer-interface secrets-test03-items Am I the only one? Andrea ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Flickering tests 2024-07-02 9:47 Flickering tests Andrea Corallo @ 2024-07-02 12:43 ` Andrea Corallo 2024-07-03 1:47 ` john muhl 2024-07-02 14:28 ` Michael Albinus 1 sibling, 1 reply; 17+ messages in thread From: Andrea Corallo @ 2024-07-02 12:43 UTC (permalink / raw To: emacs-devel; +Cc: Michael Albinus Andrea Corallo <acorallo@gnu.org> writes: > Hello, > > lately I see some flickering tests in the testsuite. > > I see these tests failing irregularly on emacs-30 master and igc: > > dbus-test06-register-property-emits-signal > dbus-test05-register-signal-with-nils > dbus-test06-property-types > dbus-test03-peer-interface > secrets-test03-items Forgot to add to the list these two: threads-condvar-wait track-changes-tests--random ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Flickering tests 2024-07-02 12:43 ` Andrea Corallo @ 2024-07-03 1:47 ` john muhl 2024-07-03 7:25 ` Andrea Corallo 0 siblings, 1 reply; 17+ messages in thread From: john muhl @ 2024-07-03 1:47 UTC (permalink / raw To: Andrea Corallo; +Cc: emacs-devel, Michael Albinus Andrea Corallo <acorallo@gnu.org> writes: > Andrea Corallo <acorallo@gnu.org> writes: > >> Hello, >> >> lately I see some flickering tests in the testsuite. >> >> I see these tests failing irregularly on emacs-30 master and igc: >> >> dbus-test06-register-property-emits-signal >> dbus-test05-register-signal-with-nils >> dbus-test06-property-types >> dbus-test03-peer-interface >> secrets-test03-items > > Forgot to add to the list these two: > > threads-condvar-wait I haven’t seen any of the above failing. > track-changes-tests--random This one has always been flaky on my test machines. I also see intermittent failures for: src/process-tests lisp/eshell/esh-cmd-tests lisp/eshell/esh-proc-tests lisp/eshell/esh-var-tests lisp/gnus/mml-sec-tests lisp/net/network-stream-tests lisp/progmodes/eglot-tests I assumed these were due to the machines the being underpowered but can dig up the logs if someone wants to have a look at them. ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Flickering tests 2024-07-03 1:47 ` john muhl @ 2024-07-03 7:25 ` Andrea Corallo 2024-07-03 20:14 ` Stefan Kangas 0 siblings, 1 reply; 17+ messages in thread From: Andrea Corallo @ 2024-07-03 7:25 UTC (permalink / raw To: john muhl; +Cc: emacs-devel, Michael Albinus john muhl <jm@pub.pink> writes: > Andrea Corallo <acorallo@gnu.org> writes: > >> Andrea Corallo <acorallo@gnu.org> writes: >> >>> Hello, >>> >>> lately I see some flickering tests in the testsuite. >>> >>> I see these tests failing irregularly on emacs-30 master and igc: >>> >>> dbus-test06-register-property-emits-signal >>> dbus-test05-register-signal-with-nils >>> dbus-test06-property-types >>> dbus-test03-peer-interface >>> secrets-test03-items >> >> Forgot to add to the list these two: >> >> threads-condvar-wait > > I haven’t seen any of the above failing. > >> track-changes-tests--random > > This one has always been flaky on my test machines. > > I also see intermittent failures for: > > src/process-tests > lisp/eshell/esh-cmd-tests > lisp/eshell/esh-proc-tests > lisp/eshell/esh-var-tests > lisp/gnus/mml-sec-tests > lisp/net/network-stream-tests > lisp/progmodes/eglot-tests > > I assumed these were due to the machines the being underpowered Mmmh, ideally tests should not be sensitive to the computational power of the machine. > but can dig up the logs if someone wants to have a look at them. I think we should put an effort in reaching a stable a clean testsuite, should we have separate bugs for each test where we do the investigation/fix? ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Flickering tests 2024-07-03 7:25 ` Andrea Corallo @ 2024-07-03 20:14 ` Stefan Kangas 2024-07-04 19:41 ` john muhl 0 siblings, 1 reply; 17+ messages in thread From: Stefan Kangas @ 2024-07-03 20:14 UTC (permalink / raw To: Andrea Corallo, john muhl; +Cc: emacs-devel, Michael Albinus Andrea Corallo <acorallo@gnu.org> writes: > I think we should put an effort in reaching a stable a clean testsuite, Agreed, that's an important goal. > should we have separate bugs for each test where we do the > investigation/fix? Yes please. ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Flickering tests 2024-07-03 20:14 ` Stefan Kangas @ 2024-07-04 19:41 ` john muhl 0 siblings, 0 replies; 17+ messages in thread From: john muhl @ 2024-07-04 19:41 UTC (permalink / raw To: Stefan Kangas; +Cc: Andrea Corallo, emacs-devel, Michael Albinus Stefan Kangas <stefankangas@gmail.com> writes: > Andrea Corallo <acorallo@gnu.org> writes: > >> I think we should put an effort in reaching a stable a clean testsuite, > > Agreed, that's an important goal. > >> should we have separate bugs for each test where we do the >> investigation/fix? > > Yes please. Will do. Let’s start with bug#71942. ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Flickering tests 2024-07-02 9:47 Flickering tests Andrea Corallo 2024-07-02 12:43 ` Andrea Corallo @ 2024-07-02 14:28 ` Michael Albinus 2024-07-02 15:25 ` Andrea Corallo 1 sibling, 1 reply; 17+ messages in thread From: Michael Albinus @ 2024-07-02 14:28 UTC (permalink / raw To: Andrea Corallo; +Cc: emacs-devel Andrea Corallo <acorallo@gnu.org> writes: > Hello, Hi Andrea, > lately I see some flickering tests in the testsuite. > > I see these tests failing irregularly on emacs-30 master and igc: > > dbus-test06-register-property-emits-signal > dbus-test05-register-signal-with-nils > dbus-test06-property-types > dbus-test03-peer-interface > secrets-test03-items > > Am I the only one? I don't see it here. Could you pls show the error messages of the failed tests? > Andrea Best regards, Michael. ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Flickering tests 2024-07-02 14:28 ` Michael Albinus @ 2024-07-02 15:25 ` Andrea Corallo 2024-07-03 14:52 ` Andrea Corallo 0 siblings, 1 reply; 17+ messages in thread From: Andrea Corallo @ 2024-07-02 15:25 UTC (permalink / raw To: Michael Albinus; +Cc: emacs-devel Michael Albinus <Michael.Albinus@gmx.de> writes: > Andrea Corallo <acorallo@gnu.org> writes: > >> Hello, > > Hi Andrea, > >> lately I see some flickering tests in the testsuite. >> >> I see these tests failing irregularly on emacs-30 master and igc: >> >> dbus-test06-register-property-emits-signal >> dbus-test05-register-signal-with-nils >> dbus-test06-property-types >> dbus-test03-peer-interface >> secrets-test03-items >> >> Am I the only one? > > I don't see it here. > > Could you pls show the error messages of the failed tests? > >> Andrea > > Best regards, Michael. Hi Michael, this what I saw on a recent master for dbus-test05-register-signal-with-nils and dbus-test05-register-signal-with-nils: Test dbus-test05-register-signal-with-nils backtrace: signal(ert-test-failed (((should (equal dbus--test-signal-received ' ert-fail(((should (equal dbus--test-signal-received '("foo"))) :form #f(compiled-function () #<bytecode 0xfe2dc592f6eb0e>)() #f(compiled-function () #<bytecode 0x19b37652292aff8f>)() handler-bind-1(#f(compiled-function () #<bytecode 0x19b37652292aff8f ert--run-test-internal(#s(ert--test-execution-info :test #s(ert-test ert-run-test(#s(ert-test :name dbus-test05-register-signal-with-nils ert-run-or-rerun-test(#s(ert--stats :selector ... :tests ... :test-m ert-run-tests((not (or (tag :expensive-test) (tag :unstable))) #f(co ert-run-tests-batch((not (or (tag :expensive-test) (tag :unstable))) ert-run-tests-batch-and-exit((not (or (tag :expensive-test) (tag :un eval((ert-run-tests-batch-and-exit '(not (or (tag :expensive-test) ( command-line-1(("-L" ":." "-l" "ert" "--eval" "(setq treesit-extra-l command-line() normal-top-level() Test dbus-test05-register-signal-with-nils condition: (ert-test-failed ((should (equal dbus--test-signal-received '("foo"))) :form (equal ((":1.171874" "/org/gtk/vfs/mount/1" "foo.tar.gz" "archive:host=file%253A%252F%252F%252Fhome%252Fandcor03%252Femacs-ci%252Femacs-ci-repo1%252Ftest%252Flisp%252Fnet%252Ftramp-archive-resources%252Ffoo.tar.gz" "" ". GThemedIcon drive-removable-media drive-removable drive drive-removable-media-symbolic drive-removable-symbolic drive-symbolic" ". GThemedIcon drive-removable-media-symbolic drive-removable-symbolic drive-symbolic drive-removable-media drive-removable drive" "" t ... ...)) ("foo")) :value nil :explanation (list-elt 0 (different-types (":1.171874" "/org/gtk/vfs/mount/1" "foo.tar.gz" "archive:host=file%253A%252F%252F%252Fhome%252Fandcor03%252Femacs-ci%252Femacs-ci-repo1%252Ftest%252Flisp%252Fnet%252Ftramp-archive-resources%252Ffoo.tar.gz" "" ". GThemedIcon drive-removable-media drive-removable drive drive-removable-media-symbolic drive-removable-symbolic drive-symbolic" ". GThemedIcon drive-removable-media-symbolic drive-removable-symbolic drive-symbolic drive-removable-media drive-removable drive" "" t ... ...) "foo")))) FAILED 12/17 dbus-test05-register-signal-with-nils (0.112019 sec) at lisp/net/dbus-tests.el:795 Test dbus-test06-register-property-emits-signal backtrace: signal(ert-test-failed (((should (equal dbus--test-signal-received ` ert-fail(((should (equal dbus--test-signal-received `(,dbus--test-in #f(compiled-function () #<bytecode -0xdd8901871824899>)() #f(compiled-function () #<bytecode 0x19b37652292aff8f>)() handler-bind-1(#f(compiled-function () #<bytecode 0x19b37652292aff8f ert--run-test-internal(#s(ert--test-execution-info :test #s(ert-test ert-run-test(#s(ert-test :name dbus-test06-register-property-emits-s ert-run-or-rerun-test(#s(ert--stats :selector ... :tests ... :test-m ert-run-tests((not (or (tag :expensive-test) (tag :unstable))) #f(co ert-run-tests-batch((not (or (tag :expensive-test) (tag :unstable))) ert-run-tests-batch-and-exit((not (or (tag :expensive-test) (tag :un eval((ert-run-tests-batch-and-exit '(not (or (tag :expensive-test) ( command-line-1(("-L" ":." "-l" "ert" "--eval" "(setq treesit-extra-l command-line() normal-top-level() Test dbus-test06-register-property-emits-signal condition: (ert-test-failed ((should (equal dbus--test-signal-received `...)) :form (equal ("org.gnu.Emacs.TestDBus") ("org.gnu.Emacs.TestDBus.Interface" (...) nil)) :value nil :explanation (proper-lists-of-different-length 1 3 ("org.gnu.Emacs.TestDBus") ("org.gnu.Emacs.TestDBus.Interface" (...) nil) first-mismatch-at 0))) FAILED 15/17 dbus-test06-register-property-emits-signal (0.008444 sec) at lisp/net/dbus-tests.el:1163 Andrea ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Flickering tests 2024-07-02 15:25 ` Andrea Corallo @ 2024-07-03 14:52 ` Andrea Corallo 2024-07-03 15:39 ` Michael Albinus 0 siblings, 1 reply; 17+ messages in thread From: Andrea Corallo @ 2024-07-03 14:52 UTC (permalink / raw To: Michael Albinus; +Cc: emacs-devel Andrea Corallo <acorallo@gnu.org> writes: > Michael Albinus <Michael.Albinus@gmx.de> writes: > >> Andrea Corallo <acorallo@gnu.org> writes: >> >>> Hello, >> >> Hi Andrea, >> >>> lately I see some flickering tests in the testsuite. >>> >>> I see these tests failing irregularly on emacs-30 master and igc: >>> >>> dbus-test06-register-property-emits-signal >>> dbus-test05-register-signal-with-nils >>> dbus-test06-property-types >>> dbus-test03-peer-interface >>> secrets-test03-items >>> >>> Am I the only one? >> >> I don't see it here. >> >> Could you pls show the error messages of the failed tests? >> >>> Andrea >> >> Best regards, Michael. > > Hi Michael, > > this what I saw on a recent master for > dbus-test05-register-signal-with-nils and > dbus-test05-register-signal-with-nils: > > Test dbus-test05-register-signal-with-nils backtrace: > signal(ert-test-failed (((should (equal dbus--test-signal-received ' > ert-fail(((should (equal dbus--test-signal-received '("foo"))) :form > #f(compiled-function () #<bytecode 0xfe2dc592f6eb0e>)() > #f(compiled-function () #<bytecode 0x19b37652292aff8f>)() > handler-bind-1(#f(compiled-function () #<bytecode 0x19b37652292aff8f > ert--run-test-internal(#s(ert--test-execution-info :test #s(ert-test > ert-run-test(#s(ert-test :name dbus-test05-register-signal-with-nils > ert-run-or-rerun-test(#s(ert--stats :selector ... :tests ... :test-m > ert-run-tests((not (or (tag :expensive-test) (tag :unstable))) #f(co > ert-run-tests-batch((not (or (tag :expensive-test) (tag :unstable))) > ert-run-tests-batch-and-exit((not (or (tag :expensive-test) (tag :un > eval((ert-run-tests-batch-and-exit '(not (or (tag :expensive-test) ( > command-line-1(("-L" ":." "-l" "ert" "--eval" "(setq treesit-extra-l > command-line() > normal-top-level() > Test dbus-test05-register-signal-with-nils condition: > (ert-test-failed > ((should (equal dbus--test-signal-received '("foo"))) :form > (equal > ((":1.171874" "/org/gtk/vfs/mount/1" "foo.tar.gz" > "archive:host=file%253A%252F%252F%252Fhome%252Fandcor03%252Femacs-ci%252Femacs-ci-repo1%252Ftest%252Flisp%252Fnet%252Ftramp-archive-resources%252Ffoo.tar.gz" > "" > ". GThemedIcon drive-removable-media drive-removable drive drive-removable-media-symbolic drive-removable-symbolic drive-symbolic" > ". GThemedIcon drive-removable-media-symbolic drive-removable-symbolic drive-symbolic drive-removable-media drive-removable drive" > "" t ... ...)) > ("foo")) > :value nil :explanation > (list-elt 0 > (different-types > (":1.171874" "/org/gtk/vfs/mount/1" "foo.tar.gz" > "archive:host=file%253A%252F%252F%252Fhome%252Fandcor03%252Femacs-ci%252Femacs-ci-repo1%252Ftest%252Flisp%252Fnet%252Ftramp-archive-resources%252Ffoo.tar.gz" > "" > ". GThemedIcon drive-removable-media drive-removable drive drive-removable-media-symbolic drive-removable-symbolic drive-symbolic" > ". GThemedIcon drive-removable-media-symbolic drive-removable-symbolic drive-symbolic drive-removable-media drive-removable drive" > "" t ... ...) > "foo")))) > FAILED 12/17 dbus-test05-register-signal-with-nils (0.112019 sec) at lisp/net/dbus-tests.el:795 > > Test dbus-test06-register-property-emits-signal backtrace: > signal(ert-test-failed (((should (equal dbus--test-signal-received ` > ert-fail(((should (equal dbus--test-signal-received `(,dbus--test-in > #f(compiled-function () #<bytecode -0xdd8901871824899>)() > #f(compiled-function () #<bytecode 0x19b37652292aff8f>)() > handler-bind-1(#f(compiled-function () #<bytecode 0x19b37652292aff8f > ert--run-test-internal(#s(ert--test-execution-info :test #s(ert-test > ert-run-test(#s(ert-test :name dbus-test06-register-property-emits-s > ert-run-or-rerun-test(#s(ert--stats :selector ... :tests ... :test-m > ert-run-tests((not (or (tag :expensive-test) (tag :unstable))) #f(co > ert-run-tests-batch((not (or (tag :expensive-test) (tag :unstable))) > ert-run-tests-batch-and-exit((not (or (tag :expensive-test) (tag :un > eval((ert-run-tests-batch-and-exit '(not (or (tag :expensive-test) ( > command-line-1(("-L" ":." "-l" "ert" "--eval" "(setq treesit-extra-l > command-line() > normal-top-level() > Test dbus-test06-register-property-emits-signal condition: > (ert-test-failed > ((should (equal dbus--test-signal-received `...)) :form > (equal ("org.gnu.Emacs.TestDBus") > ("org.gnu.Emacs.TestDBus.Interface" (...) nil)) > :value nil :explanation > (proper-lists-of-different-length 1 3 ("org.gnu.Emacs.TestDBus") > ("org.gnu.Emacs.TestDBus.Interface" > (...) nil) > first-mismatch-at 0))) > FAILED 15/17 dbus-test06-register-property-emits-signal (0.008444 sec) at lisp/net/dbus-tests.el:1163 > > Andrea Mmhh, I've the impression this happen on my system only when it's under heavy load. I've 64 cores and if I "make check -j64" I've say 50% probability to hit these, but if I run only: ./src/emacs -batch -l ert -l ./test/lisp/net/dbus-tests.el -f ert-run-tests-batch-and-exit they all pass. Perhaps some timeout need to be adjusted? Andrea ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Flickering tests 2024-07-03 14:52 ` Andrea Corallo @ 2024-07-03 15:39 ` Michael Albinus 2024-07-03 18:16 ` Andrea Corallo 0 siblings, 1 reply; 17+ messages in thread From: Michael Albinus @ 2024-07-03 15:39 UTC (permalink / raw To: Andrea Corallo; +Cc: emacs-devel Andrea Corallo <acorallo@gnu.org> writes: Hi Andrea, > Mmhh, I've the impression this happen on my system only when it's under > heavy load. I've 64 cores and if I "make check -j64" I've say 50% > probability to hit these, but if I run only: > > ./src/emacs -batch -l ert -l ./test/lisp/net/dbus-tests.el -f ert-run-tests-batch-and-exit > > they all pass. Perhaps some timeout need to be adjusted? It's not the a I believe. D-Bus tests are sending events to the D-Buson OS level, and receiving events which will be interpreted. They are not written such a way that all tests can run in parallel. If you invoke "make check -j64", it could happen that some D-Bus tests run in parallel, and returning events do arrive in another Emacs instance but the one which has send the evnt. This might be even more problematic, because recently (on Sunday, commit 4e22ef870c4b650f29c4441ac51b6a2ac506ea57) a test which registers for wildcard events has been added. This test resides in my local Emacs git for 3 months w/o problems; I've simply waited until the FSF papers of the author, Robert Church, were on file. What could we do? Could we exclude dbus-tests.el from running as simultaneous job? Or could we simply skip D-Bus tests, if we detect that the tests run via simultaneous make jobs? How to check, on Lisp level? We always have the option to mark this test as :unstable, so we could stabilize the Emacs 30 release this way. But it would be a pity. Btw, could you check whether the D-Bus tests flickery goes away if we tag dbus-test05-register-signal-with-nils as :unstable? > Andrea Best regards, Michael. ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Flickering tests 2024-07-03 15:39 ` Michael Albinus @ 2024-07-03 18:16 ` Andrea Corallo 2024-07-05 7:36 ` Michael Albinus 0 siblings, 1 reply; 17+ messages in thread From: Andrea Corallo @ 2024-07-03 18:16 UTC (permalink / raw To: Michael Albinus; +Cc: emacs-devel Michael Albinus <Michael.Albinus@gmx.de> writes: > Andrea Corallo <acorallo@gnu.org> writes: > > Hi Andrea, > >> Mmhh, I've the impression this happen on my system only when it's under >> heavy load. I've 64 cores and if I "make check -j64" I've say 50% >> probability to hit these, but if I run only: >> >> ./src/emacs -batch -l ert -l ./test/lisp/net/dbus-tests.el -f ert-run-tests-batch-and-exit >> >> they all pass. Perhaps some timeout need to be adjusted? > > It's not the a I believe. D-Bus tests are sending events to the D-Buson > OS level, and receiving events which will be interpreted. They are not > written such a way that all tests can run in parallel. > > If you invoke "make check -j64", it could happen that some D-Bus tests > run in parallel, and returning events do arrive in another Emacs > instance but the one which has send the evnt. This might be even more > problematic, because recently (on Sunday, commit > 4e22ef870c4b650f29c4441ac51b6a2ac506ea57) a test which registers for > wildcard events has been added. This test resides in my local Emacs git > for 3 months w/o problems; I've simply waited until the FSF papers of > the author, Robert Church, were on file. > > What could we do? Could we exclude dbus-tests.el from running as > simultaneous job? Mmmh not sure is possible but also desirable, we want to be able to run as many tests as possible even with -jN. > Or could we simply skip D-Bus tests, if we detect that the tests run via > simultaneous make jobs? How to check, on Lisp level? Perhaps using some lock file as mutex for the generation of D-Bus messages from different test files? > We always have the option to mark this test as :unstable, so we could > stabilize the Emacs 30 release this way. But it would be a pity. > > Btw, could you check whether the D-Bus tests flickery goes away if we > tag dbus-test05-register-signal-with-nils as :unstable? From the bunch of test I did seems this is solving, I'll mark it as :unstable for now till a better solution is found. Thanks Andrea ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Flickering tests 2024-07-03 18:16 ` Andrea Corallo @ 2024-07-05 7:36 ` Michael Albinus 2024-07-09 8:39 ` Andrea Corallo 0 siblings, 1 reply; 17+ messages in thread From: Michael Albinus @ 2024-07-05 7:36 UTC (permalink / raw To: Andrea Corallo; +Cc: emacs-devel [-- Attachment #1: Type: text/plain, Size: 1306 bytes --] Andrea Corallo <acorallo@gnu.org> writes: Hi Andrea, > Mmmh not sure is possible but also desirable, we want to be able to run > as many tests as possible even with -jN. Last days, I've digged further into this. I still couldn't provoke the error (my machine has only 8 processing units). However, ... >> Btw, could you check whether the D-Bus tests flickery goes away if we >> tag dbus-test05-register-signal-with-nils as :unstable? > > From the bunch of test I did seems this is solving, I'll mark it as > :unstable for now till a better solution is found. ... I've found suspicious code in this test. Usually, all tests work deterministic. A method call or a signal is raised, and the returned message from D-Bus is analyzed. But in this test, a signal handler with a wildcard filter is registered. That means, *any* signal from D-Bus is catched, also signals which aren't relevant for this test. The more pressure you give on the system ("make -j96"), the more likely it is. So I have modified the signal handler to handle only signals we're interested in. See the appended patch. Could you, pls, check whether it helps? I guess we need it only in the master branch, because the problem appears only in dbus-test.el. dbusbind.c and dbus.el are OK. > Thanks > > Andrea Best regards, Michael. [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #2: Type: text/x-patch, Size: 2851 bytes --] diff --git a/test/lisp/net/dbus-tests.el b/test/lisp/net/dbus-tests.el index b3337132fb6..0e0996b373e 100644 --- a/test/lisp/net/dbus-tests.el +++ b/test/lisp/net/dbus-tests.el @@ -25,7 +25,7 @@ (require 'ert-x) (require 'dbus) -(defvar dbus-debug nil) +(defvar dbus-debug) (declare-function dbus-get-unique-name "dbusbind.c" (bus)) (defconst dbus--test-enabled-session-bus @@ -732,12 +732,28 @@ dbus-test04-call-method-timeout ;; Cleanup. (dbus-unregister-service :session dbus--test-service))) +(defvar dbus--test-event-expected nil + "The expected event in `dbus--test-signal-handler'.") + (defvar dbus--test-signal-received nil "Received signal value in `dbus--test-signal-handler'.") (defun dbus--test-signal-handler (&rest args) "Signal handler for `dbus-test*-signal' and `dbus-test08-register-monitor'." - (setq dbus--test-signal-received args)) + (ignore-error dbus-error + (let ((last-input-event last-input-event)) + (when (or (null dbus--test-event-expected) + (and (equal (dbus-event-bus-name last-input-event) + (dbus-event-bus-name dbus--test-event-expected)) + (equal (dbus-event-message-type last-input-event) + (dbus-event-message-type dbus--test-event-expected)) + (equal (dbus-event-service-name last-input-event) + (dbus-event-service-name dbus--test-event-expected)) + (equal (dbus-event-path-name last-input-event) + (dbus-event-path-name dbus--test-event-expected)) + (equal (dbus-event-member-name last-input-event) + (dbus-event-member-name dbus--test-event-expected)))) + (setq dbus--test-signal-received args))))) (defun dbus--test-timeout-handler (&rest _ignore) "Timeout handler, reporting a failed test." @@ -804,6 +820,14 @@ dbus-test05-register-signal-with-nils (handler #'dbus--test-signal-handler) registered) + ;; Filter received signals in signal handler. + (setq dbus--test-event-expected + `(dbus-event :session ,dbus-message-type-signal + 0 ;; Serial number doesn't matter. + ,(dbus-get-unique-name :session) + nil ;; Destination doesn't matter. + ,dbus--test-path ,dbus--test-interface "Member" ,handler)) + ;; Register signal handler. (should (equal @@ -842,6 +866,7 @@ dbus-test05-register-signal-with-nils (should-not (dbus-unregister-object registered))) ;; Cleanup. + (setq dbus--test-event-expected nil) (dbus-unregister-service :session dbus--test-service))) (ert-deftest dbus-test06-register-property () ^ permalink raw reply related [flat|nested] 17+ messages in thread
* Re: Flickering tests 2024-07-05 7:36 ` Michael Albinus @ 2024-07-09 8:39 ` Andrea Corallo 2024-07-09 8:49 ` Michael Albinus 0 siblings, 1 reply; 17+ messages in thread From: Andrea Corallo @ 2024-07-09 8:39 UTC (permalink / raw To: Michael Albinus; +Cc: emacs-devel Michael Albinus <Michael.Albinus@gmx.de> writes: > Andrea Corallo <acorallo@gnu.org> writes: > > Hi Andrea, > >> Mmmh not sure is possible but also desirable, we want to be able to run >> as many tests as possible even with -jN. > > Last days, I've digged further into this. I still couldn't provoke the > error (my machine has only 8 processing units). However, ... > >>> Btw, could you check whether the D-Bus tests flickery goes away if we >>> tag dbus-test05-register-signal-with-nils as :unstable? >> >> From the bunch of test I did seems this is solving, I'll mark it as >> :unstable for now till a better solution is found. > > ... I've found suspicious code in this test. Usually, all tests work > deterministic. A method call or a signal is raised, and the returned > message from D-Bus is analyzed. But in this test, a signal handler with > a wildcard filter is registered. That means, *any* signal from D-Bus is > catched, also signals which aren't relevant for this test. The more > pressure you give on the system ("make -j96"), the more likely it is. > > So I have modified the signal handler to handle only signals we're > interested in. See the appended patch. Could you, pls, check whether it > helps? I guess we need it only in the master branch, because the problem > appears only in dbus-test.el. dbusbind.c and dbus.el are OK. Hi Michael, I tried your patch and seems to solve (for my few tries at least). But why do you think we need it only on master? Before being marked as unstable the test was flickering on emacs-30 as well. Thanks Andrea ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Flickering tests 2024-07-09 8:39 ` Andrea Corallo @ 2024-07-09 8:49 ` Michael Albinus 2024-07-09 9:47 ` Andrea Corallo 0 siblings, 1 reply; 17+ messages in thread From: Michael Albinus @ 2024-07-09 8:49 UTC (permalink / raw To: Andrea Corallo; +Cc: emacs-devel [-- Attachment #1: Type: text/plain, Size: 710 bytes --] Andrea Corallo <acorallo@gnu.org> writes: > Hi Michael, Hi Andrea, > I tried your patch and seems to solve (for my few tries at least). But > why do you think we need it only on master? Before being marked as > unstable the test was flickering on emacs-30 as well. The test isn't super-important, filtering it out on the emacs-30 branch by tagging it :unstable doesn't matter really. I've proposed this, because the emacs-30 branch should be kept stable. But if you like, I could install it there. Note, that I have improved the patch meanwhile. The test dbus-test08-register-monitor seems to be vulnerable as well. The patch looks meanwhile as appended. > Thanks > > Andrea Best regards, Michael. [-- Attachment #2: Type: text/x-patch, Size: 5611 bytes --] diff --git a/test/lisp/net/dbus-tests.el b/test/lisp/net/dbus-tests.el index a217c92b1f7..78ae79b8711 100644 --- a/test/lisp/net/dbus-tests.el +++ b/test/lisp/net/dbus-tests.el @@ -25,7 +25,7 @@ (require 'ert-x) (require 'dbus) -(defvar dbus-debug nil) +(defvar dbus-debug) (declare-function dbus-get-unique-name "dbusbind.c" (bus)) (defconst dbus--test-enabled-session-bus @@ -732,12 +732,29 @@ dbus-test04-call-method-timeout ;; Cleanup. (dbus-unregister-service :session dbus--test-service))) +(defvar dbus--test-event-expected nil + "The expected event in `dbus--test-signal-handler'.") + (defvar dbus--test-signal-received nil "Received signal value in `dbus--test-signal-handler'.") (defun dbus--test-signal-handler (&rest args) "Signal handler for `dbus-test*-signal' and `dbus-test08-register-monitor'." - (setq dbus--test-signal-received args)) + (ignore-error dbus-error + (message "%S" last-input-event) + (let ((last-input-event last-input-event)) + (when (or (null dbus--test-event-expected) + (and (equal (dbus-event-bus-name last-input-event) + (dbus-event-bus-name dbus--test-event-expected)) + (equal (dbus-event-message-type last-input-event) + (dbus-event-message-type dbus--test-event-expected)) + (equal (dbus-event-service-name last-input-event) + (dbus-event-service-name dbus--test-event-expected)) + (equal (dbus-event-path-name last-input-event) + (dbus-event-path-name dbus--test-event-expected)) + (equal (dbus-event-member-name last-input-event) + (dbus-event-member-name dbus--test-event-expected)))) + (setq dbus--test-signal-received args))))) (defun dbus--test-timeout-handler (&rest _ignore) "Timeout handler, reporting a failed test." @@ -796,7 +813,6 @@ dbus-test05-register-signal-with-nils "Check signal registration for an own service. SERVICE, PATH, INTERFACE and SIGNAL are ‘nil’. This is interpreted as a wildcard for the respective argument." - :tags '(:unstable) (skip-unless dbus--test-enabled-session-bus) (dbus-ignore-errors (dbus-unregister-service :session dbus--test-service)) @@ -805,6 +821,14 @@ dbus-test05-register-signal-with-nils (handler #'dbus--test-signal-handler) registered) + ;; Filter received signals in signal handler. + (setq dbus--test-event-expected + `(dbus-event :session ,dbus-message-type-signal + 0 ;; Serial number doesn't matter. + ,(dbus-get-unique-name :session) + nil ;; Destination doesn't matter. + ,dbus--test-path ,dbus--test-interface ,member ,handler)) + ;; Register signal handler. (should (equal @@ -843,6 +867,7 @@ dbus-test05-register-signal-with-nils (should-not (dbus-unregister-object registered))) ;; Cleanup. + (setq dbus--test-event-expected nil) (dbus-unregister-service :session dbus--test-service))) (ert-deftest dbus-test06-register-property () @@ -1934,19 +1959,32 @@ dbus-test08-register-monitor (skip-unless dbus--test-enabled-session-bus) (unwind-protect - (let (registered) + (let ((member "Member") + (handler #'dbus--test-signal-handler) + registered) + + ;; Filter received signals in signal handler. + (setq dbus--test-event-expected + `(dbus-event :session-private ,dbus-message-type-signal + 0 ;; Serial number doesn't matter. + ,(dbus-get-unique-name :session) + nil ;; Destination doesn't matter. + ,dbus--test-path ,dbus--test-interface ,member ,handler)) + + ;; Register monitor. (should (equal - (setq registered - (dbus-register-monitor :session #'dbus--test-signal-handler)) - '((:monitor :session-private) - (nil nil dbus--test-signal-handler)))) + (setq + registered + (dbus-register-monitor :session handler)) + `((:monitor :session-private) + (nil nil ,handler)))) ;; Send a signal, shall be traced. (setq dbus--test-signal-received nil) (dbus-send-signal :session dbus--test-service dbus--test-path - dbus--test-interface "Foo" "foo") + dbus--test-interface member "foo") (with-timeout (1 (dbus--test-timeout-handler)) (while (null dbus--test-signal-received) (read-event nil nil 0.1))) @@ -1959,13 +1997,18 @@ dbus-test08-register-monitor (setq dbus--test-signal-received nil) (dbus-send-signal :session dbus--test-service dbus--test-path - dbus--test-interface "Foo" "foo") + dbus--test-interface member "foo") (with-timeout (1 (ignore)) (while (null dbus--test-signal-received) (read-event nil nil 0.1))) - (should-not dbus--test-signal-received)) + (should-not dbus--test-signal-received) + + ;; Unregister monitor. + ;; TODO: This seems to be a noop. And it returns nil. + (dbus-unregister-object registered)) ;; Cleanup. + (setq dbus--test-event-expected nil) (dbus-unregister-service :session dbus--test-service))) (ert-deftest dbus-test09-get-managed-objects () ^ permalink raw reply related [flat|nested] 17+ messages in thread
* Re: Flickering tests 2024-07-09 8:49 ` Michael Albinus @ 2024-07-09 9:47 ` Andrea Corallo 2024-07-09 10:47 ` Michael Albinus 0 siblings, 1 reply; 17+ messages in thread From: Andrea Corallo @ 2024-07-09 9:47 UTC (permalink / raw To: Michael Albinus; +Cc: emacs-devel Michael Albinus <Michael.Albinus@gmx.de> writes: > Andrea Corallo <acorallo@gnu.org> writes: > >> Hi Michael, > > Hi Andrea, > >> I tried your patch and seems to solve (for my few tries at least). But >> why do you think we need it only on master? Before being marked as >> unstable the test was flickering on emacs-30 as well. > > The test isn't super-important, filtering it out on the emacs-30 branch > by tagging it :unstable doesn't matter really. I've proposed this, > because the emacs-30 branch should be kept stable. But if you like, I > could install it there. I see thanks, if you are happy with the coverage on emacs-30 with 'dbus-test05-register-signal-with-nils' waved out I suggest we install the current version on master and see how it goes removing the unstable tag from the test. If you feel would be important to cover emacs-30 as well I think we can install the patch there instead. Thanks Andrea ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Flickering tests 2024-07-09 9:47 ` Andrea Corallo @ 2024-07-09 10:47 ` Michael Albinus 2024-07-09 12:30 ` Andrea Corallo 0 siblings, 1 reply; 17+ messages in thread From: Michael Albinus @ 2024-07-09 10:47 UTC (permalink / raw To: Andrea Corallo; +Cc: emacs-devel Andrea Corallo <acorallo@gnu.org> writes: Hi Andrea, > I see thanks, if you are happy with the coverage on emacs-30 with > 'dbus-test05-register-signal-with-nils' waved out I suggest we install > the current version on master and see how it goes removing the unstable > tag from the test. If you feel would be important to cover emacs-30 as > well I think we can install the patch there instead. I've installed it in master. > Thanks > > Andrea Best regards, Michael. ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Flickering tests 2024-07-09 10:47 ` Michael Albinus @ 2024-07-09 12:30 ` Andrea Corallo 0 siblings, 0 replies; 17+ messages in thread From: Andrea Corallo @ 2024-07-09 12:30 UTC (permalink / raw To: Michael Albinus; +Cc: emacs-devel Michael Albinus <Michael.Albinus@gmx.de> writes: > Andrea Corallo <acorallo@gnu.org> writes: > > Hi Andrea, > >> I see thanks, if you are happy with the coverage on emacs-30 with >> 'dbus-test05-register-signal-with-nils' waved out I suggest we install >> the current version on master and see how it goes removing the unstable >> tag from the test. If you feel would be important to cover emacs-30 as >> well I think we can install the patch there instead. > > I've installed it in master. Thanks Andrea ^ permalink raw reply [flat|nested] 17+ messages in thread
end of thread, other threads:[~2024-07-09 12:30 UTC | newest] Thread overview: 17+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2024-07-02 9:47 Flickering tests Andrea Corallo 2024-07-02 12:43 ` Andrea Corallo 2024-07-03 1:47 ` john muhl 2024-07-03 7:25 ` Andrea Corallo 2024-07-03 20:14 ` Stefan Kangas 2024-07-04 19:41 ` john muhl 2024-07-02 14:28 ` Michael Albinus 2024-07-02 15:25 ` Andrea Corallo 2024-07-03 14:52 ` Andrea Corallo 2024-07-03 15:39 ` Michael Albinus 2024-07-03 18:16 ` Andrea Corallo 2024-07-05 7:36 ` Michael Albinus 2024-07-09 8:39 ` Andrea Corallo 2024-07-09 8:49 ` Michael Albinus 2024-07-09 9:47 ` Andrea Corallo 2024-07-09 10:47 ` Michael Albinus 2024-07-09 12:30 ` Andrea Corallo
Code repositories for project(s) associated with this external index https://git.savannah.gnu.org/cgit/emacs.git https://git.savannah.gnu.org/cgit/emacs/org-mode.git This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.