[PATCH v2] test: exit with nonzero value when not all tests completed successfully

[PATCH v2] test: exit with nonzero value when not all tests completed successfully

[Tomi Ollila]

If any of the tests in our test system is not passing the execution
of the test suite completes with nonzero exit value.

It is better to rely on the exit value of the test system instead
of some arbitrary strings in test output (or use both).
---

This is v2 of id:1378479935-26312-1-git-send-email-tomi.ollila@iki.fi
thanks to Mark's observation in id:87d2olg9bz.fsf@qmul.ac.uk
which widened my thought region :D

Tomi

 test/aggregate-results.sh | 9 +++++++++
 test/notmuch-test         | 3 +++
 2 files changed, 12 insertions(+)

diff --git a/test/aggregate-results.sh b/test/aggregate-results.sh
index 732d6ca..b016edb 100755
--- a/test/aggregate-results.sh
+++ b/test/aggregate-results.sh
@@ -1,5 +1,7 @@
 #!/usr/bin/env bash
 
+set -eu
+
 fixed=0
 success=0
 failed=0
@@ -79,3 +81,10 @@ if [ "$skipped" != "0" ]; then
     tests=$(pluralize "test" $skipped)
     echo "$skipped $tests skipped."
 fi
+
+if [ $success -gt 0 -a $fixed -eq 0 -a $failed -eq 0 -a $skipped -eq 0 ]
+then
+    exit 0
+else
+    exit 1
+fi
diff --git a/test/notmuch-test b/test/notmuch-test
index 6db7979..aa28bb0 100755
--- a/test/notmuch-test
+++ b/test/notmuch-test
@@ -97,6 +97,9 @@ trap - HUP INT TERM
 
 # Report results
 ./aggregate-results.sh test-results/*
+ev=$?
 
 # Clean up
 rm -rf test-results corpus.mail
+
+exit $ev
-- 
1.8.0

Re: [PATCH v2] test: exit with nonzero value when not all tests completed successfully

[Mark Walters]

LGTM +1

Mark

On Sun, 08 Sep 2013, Tomi Ollila <tomi.ollila@iki.fi> wrote:
> If any of the tests in our test system is not passing the execution
> of the test suite completes with nonzero exit value.
>
> It is better to rely on the exit value of the test system instead
> of some arbitrary strings in test output (or use both).
> ---
>
> This is v2 of id:1378479935-26312-1-git-send-email-tomi.ollila@iki.fi
> thanks to Mark's observation in id:87d2olg9bz.fsf@qmul.ac.uk
> which widened my thought region :D
>
> Tomi
>
>  test/aggregate-results.sh | 9 +++++++++
>  test/notmuch-test         | 3 +++
>  2 files changed, 12 insertions(+)
>
> diff --git a/test/aggregate-results.sh b/test/aggregate-results.sh
> index 732d6ca..b016edb 100755
> --- a/test/aggregate-results.sh
> +++ b/test/aggregate-results.sh
> @@ -1,5 +1,7 @@
>  #!/usr/bin/env bash
>  
> +set -eu
> +
>  fixed=0
>  success=0
>  failed=0
> @@ -79,3 +81,10 @@ if [ "$skipped" != "0" ]; then
>      tests=$(pluralize "test" $skipped)
>      echo "$skipped $tests skipped."
>  fi
> +
> +if [ $success -gt 0 -a $fixed -eq 0 -a $failed -eq 0 -a $skipped -eq 0 ]
> +then
> +    exit 0
> +else
> +    exit 1
> +fi
> diff --git a/test/notmuch-test b/test/notmuch-test
> index 6db7979..aa28bb0 100755
> --- a/test/notmuch-test
> +++ b/test/notmuch-test
> @@ -97,6 +97,9 @@ trap - HUP INT TERM
>  
>  # Report results
>  ./aggregate-results.sh test-results/*
> +ev=$?
>  
>  # Clean up
>  rm -rf test-results corpus.mail
> +
> +exit $ev
> -- 
> 1.8.0
>
> _______________________________________________
> notmuch mailing list
> notmuch@notmuchmail.org
> http://notmuchmail.org/mailman/listinfo/notmuch

Re: [PATCH v2] test: exit with nonzero value when not all tests completed successfully

[David Bremner]

Tomi Ollila <tomi.ollila@iki.fi> writes:

> If any of the tests in our test system is not passing the execution
> of the test suite completes with nonzero exit value.

pushed, 

d