Re: Test suite timing issues?

[Michael J Gruber <git@grubix.eu>]

David Bremner venit, vidit, dixit 2022-02-12 01:03:00:
> Michael J Gruber <git@grubix.eu> writes:
> 
> > Hi there,
> >
> > I'm trying to package notmuch for Redhat's enterprise linux and clones
> > (EPEL, extra packages for enterprise linux). 
> >
> > This looks mostly fine, including the tests, except for intermittent
> > failures on epel-8-s390x. They look like the below, or in tests
> > following those, and apparantly all have to do with "db not synced yet"
> > or such, so that a message one subtest creates only shows up in the db
> > for the next subtest.
> >
> > I've also had completely fine test runs on epel-8-s390x, but I'm
> > starting to wonder whether I've just been lucky so far on other
> > platforms ... Could it be possible that generate_message(), even though
> > adding the message, still returns false and therefore add_message() does
> > not call "notmuch new"? One might want to drop the "&&" there in
> > test-lib-common, I dunno. This shouldn't be "intermittent".
> 
> It's hard to see what can go wrong (maybe perl doesn't work?), but
> failing to generate a message should be a fatal error. Maybe try
> something like
> 

Well, as you can see in both reports, the pattern is as follows:

The first subttest sees 1 less message than expected.
The second sees 1 more message than expected.

So in summary, they are generated but "notmuch new" is not run or does
not pick them up. Though it's a Heisenbug on a specific arch, I'll try
your suggestion and hammer our infra with it ...

> diff --git a/test/test-lib-common.sh b/test/test-lib-common.sh
> index ebbf4cdf..076777ca 100644
> --- a/test/test-lib-common.sh
> +++ b/test/test-lib-common.sh
> @@ -225,7 +225,7 @@ EOF
>  # All of the arguments and return values supported by generate_message
>  # are also supported here, so see that function for details.
>  add_message () {
> -    generate_message "$@" &&
> +    generate_message "$@" || error "failed to generate message"
>      notmuch new > /dev/null
>  }
> 
> 
> to see if you get more information