Re: test suite: FIXED messages are misordered with tests

[Tomi Ollila <tomi.ollila@iki.fi>]

On Tue, May 12 2020, Daniel Kahn Gillmor wrote:

> I'm debugging/diagnosing/trying to clean up some "FIXED" known-broken
> tests right now.
>
> Sometimes, depending on circumstances i can't predict (race
> conditions?), I see funny output like:
>
> ~~~
> Use "make V=1" to see the details for passing and known broken tests.
> INFO: using 2m timeout for tests
> INFO: running tests with moreutils parallel
>  FIXED  verify signed PKCS#7 subject (onepart-signed) signer User ID
>
> T356-protected-headers: Testing Message decryption with protected headers
>  BROKEN confirm signed and encrypted PKCS#7 subject (sign+enc)
>  BROKEN confirm signed and encrypted PKCS#7 subject (sign+enc) signer User ID
>  BROKEN confirm signed and encrypted PKCS#7 subject (sign+enc+legacy-disp)
>  BROKEN confirm signed and encrypted PKCS#7 subject (sign+enc+legacy-disp) signer User ID
>  BROKEN confirm encryption-protected PKCS#7 subject (enc+legacy-disp)
> ~~~
>
> Clearly, that FIXED should come *after* the "T356-protected-headers:"
> separator.


After your second mail I tried to reproduce by taking a test file,
marking one known_broken, even it is not to get FIXED, and then broke
next one.

And I got it reproduced.

the message that prints FIXED, does not execute print_test_description
but the ones that do BROKEN or FAIL does (did not fully check but yes).

And:

    print_test_description ()
    {
        test -z "$test_description_printed" || return 0
        echo
        echo $this_test: "Testing ${test_description}"
        test_description_printed=1
    }
    if [ -z "$NOTMUCH_TEST_QUIET" ]
    then
        print_test_description
    fi


Tomi


>
> This is a minor bug, i suppose, but i confess i don't understand the
> maze of shell functions in test-lib.sh well enough to see why this is
> happening, let alone to fix it.
>
> Anyone interested in fixing it should be able to do so by marking a good
> test "known broken" and then re-running the test suite.  The above
> output is taken from:
>
>     make -j4 check NOTMUCH_TESTS=T356-protected-headers.sh
>
> Sorry to send a bug report with no fixes!
>
>            --dkg
> _______________________________________________
> notmuch mailing list
> notmuch@notmuchmail.org
> https://notmuchmail.org/mailman/listinfo/notmuch