Re: Test suite timing issues?

[Michael J Gruber <git@grubix.eu>]

Michael J Gruber venit, vidit, dixit 2022-02-11 14:55:32:
> Hi there,
> 
> I'm trying to package notmuch for Redhat's enterprise linux and clones
> (EPEL, extra packages for enterprise linux). 
> 
> This looks mostly fine, including the tests, except for intermittent
> failures on epel-8-s390x. They look like the below, or in tests
> following those, and apparantly all have to do with "db not synced yet"
> or such, so that a message one subtest creates only shows up in the db
> for the next subtest.
> 
> I've also had completely fine test runs on epel-8-s390x, but I'm
> starting to wonder whether I've just been lucky so far on other
> platforms ... Could it be possible that generate_message(), even though
> adding the message, still returns false and therefore add_message() does
> not call "notmuch new"? One might want to drop the "&&" there in
> test-lib-common, I dunno. This shouldn't be "intermittent".
> 
> Michael
> 
> T210-raw: Testing notmuch show --format=raw
>  FAIL   Attempt to show multiple raw messages
>         --- T210-raw.1.expected 2022-02-11 13:22:28.011556841 +0000
>         +++ T210-raw.1.output   2022-02-11 13:22:28.011556841 +0000
>         @@ -1 +1,7 @@
>         -Error: search term did not match precisely one message (matched 2 messages).
>         +From: Notmuch Test Suite <test_suite@notmuchmail.org>
>         +To: Notmuch Test Suite <test_suite@notmuchmail.org>
>         +Message-Id: <msg-001@notmuch-test-suite>
>         +Subject: Test message #1
>         +Date: Fri, 05 Jan 2001 15:43:56 +0000
>         +
>         +This is just a test message (#1)
>  PASS   Show a raw message
>  FAIL   Show another raw message
>         --- T210-raw.3.expected 2022-02-11 13:22:28.031556841 +0000
>         +++ T210-raw.3.output   2022-02-11 13:22:28.031556841 +0000
>         @@ -1,7 +1 @@
>         -From: Notmuch Test Suite <test_suite@notmuchmail.org>
>         -To: Notmuch Test Suite <test_suite@notmuchmail.org>
>         -Message-Id: <msg-002@notmuch-test-suite>
>         -Subject: Test message #2
>         -Date: GENERATED_DATE
>          
>         -This is just a test message (#2)
> Error: search term did not match precisely one message (matched 0 messages).

Similar apparent timing issue on epel-8-aarch64:

 FAIL   Adding message with long ID
	--- T290-long-id.2.expected	2022-02-11 13:55:17.028415326 +0000
	+++ T290-long-id.2.output	2022-02-11 13:55:17.028415326 +0000
	@@ -1 +1 @@
	-Added 1 new message to the database.
	+No new mail.
 FAIL   Referencing long ID after adding
	--- T290-long-id.3.expected	2022-02-11 13:55:17.078415753 +0000
	+++ T290-long-id.3.output	2022-02-11 13:55:17.078415753 +0000
	@@ -1 +1 @@
	-Added 1 new message to the database.
	+Added 2 new messages to the database.

Note how that message from the 1st subtest shows up only in the second.