[PATCH] test: Abort driver if a test script aborts

[PATCH] test: Abort driver if a test script aborts

[Austin Clements]

Previously, if a test script aborted (e.g., because it passed too few
arguments to a test function), the test driver loop would simply
continue on to the next test script and the final results would
declare that everything passed (except that the test count would look
suspiciously low, but maybe you just misremembered how many tests
there were).

Now, if a test script exits with a non-zero status and did not produce
a final results file, we propagate that failure out of the driver loop
immediately.

To keep this simple, this patch removes the PID from the test-results
file name.  This PID was inherited from the git test system and seems
unnecessary, since the file name already includes the name of the test
script and the test-results directory is created anew for each run.
---
 test/basic        |    2 +-
 test/notmuch-test |    6 ++++++
 test/test-lib.sh  |    2 +-
 3 files changed, 8 insertions(+), 2 deletions(-)

diff --git a/test/basic b/test/basic
index 1b842d2..b7feb07 100755
--- a/test/basic
+++ b/test/basic
@@ -79,7 +79,7 @@ test_begin_subtest "Ensure that -v does not suppress test output"
 output=$(cd $TEST_DIRECTORY; ./test-verbose -v 2>&1 | suppress_diff_date)
 expected=$(cat $EXPECTED/test-verbose-yes | suppress_diff_date)
 # Do not include the results of test-verbose in totals
-rm $TEST_DIRECTORY/test-results/test-verbose-*
+rm $TEST_DIRECTORY/test-results/test-verbose
 rm -r $TEST_DIRECTORY/tmp.test-verbose
 test_expect_equal "$output" "$expected"
 
diff --git a/test/notmuch-test b/test/notmuch-test
index 9a1b375..f275439 100755
--- a/test/notmuch-test
+++ b/test/notmuch-test
@@ -81,6 +81,12 @@ trap 'e=$?; kill $!; exit $e' HUP INT TERM
 for test in $TESTS; do
     $TEST_TIMEOUT_CMD ./$test "$@" &
     wait $!
+    # If the test failed without producing results, then it aborted,
+    # so we should abort, too.
+    RES=$?
+    if [[ $RES != 0 && ! -e "test-results/${test%.sh}" ]]; then
+        exit $RES
+    fi
 done
 trap - HUP INT TERM
 
diff --git a/test/test-lib.sh b/test/test-lib.sh
index e092231..77063a4 100644
--- a/test/test-lib.sh
+++ b/test/test-lib.sh
@@ -920,7 +920,7 @@ test_done () {
 	GIT_EXIT_OK=t
 	test_results_dir="$TEST_DIRECTORY/test-results"
 	mkdir -p "$test_results_dir"
-	test_results_path="$test_results_dir/${0%.sh}-$$"
+	test_results_path="$test_results_dir/${0%.sh}"
 
 	echo "total $test_count" >> $test_results_path
 	echo "success $test_success" >> $test_results_path
-- 
1.7.10.4

Re: [PATCH] test: Abort driver if a test script aborts

[Tomi Ollila]

On Wed, Nov 28 2012, Austin Clements <amdragon@MIT.EDU> wrote:

> Previously, if a test script aborted (e.g., because it passed too few
> arguments to a test function), the test driver loop would simply
> continue on to the next test script and the final results would
> declare that everything passed (except that the test count would look
> suspiciously low, but maybe you just misremembered how many tests
> there were).
>
> Now, if a test script exits with a non-zero status and did not produce
> a final results file, we propagate that failure out of the driver loop
> immediately.
>
> To keep this simple, this patch removes the PID from the test-results
> file name.  This PID was inherited from the git test system and seems
> unnecessary, since the file name already includes the name of the test
> script and the test-results directory is created anew for each run.
> ---

+1

Tomi


>  test/basic        |    2 +-
>  test/notmuch-test |    6 ++++++
>  test/test-lib.sh  |    2 +-
>  3 files changed, 8 insertions(+), 2 deletions(-)
>
> diff --git a/test/basic b/test/basic
> index 1b842d2..b7feb07 100755
> --- a/test/basic
> +++ b/test/basic
> @@ -79,7 +79,7 @@ test_begin_subtest "Ensure that -v does not suppress test output"
>  output=$(cd $TEST_DIRECTORY; ./test-verbose -v 2>&1 | suppress_diff_date)
>  expected=$(cat $EXPECTED/test-verbose-yes | suppress_diff_date)
>  # Do not include the results of test-verbose in totals
> -rm $TEST_DIRECTORY/test-results/test-verbose-*
> +rm $TEST_DIRECTORY/test-results/test-verbose
>  rm -r $TEST_DIRECTORY/tmp.test-verbose
>  test_expect_equal "$output" "$expected"
>  
> diff --git a/test/notmuch-test b/test/notmuch-test
> index 9a1b375..f275439 100755
> --- a/test/notmuch-test
> +++ b/test/notmuch-test
> @@ -81,6 +81,12 @@ trap 'e=$?; kill $!; exit $e' HUP INT TERM
>  for test in $TESTS; do
>      $TEST_TIMEOUT_CMD ./$test "$@" &
>      wait $!
> +    # If the test failed without producing results, then it aborted,
> +    # so we should abort, too.
> +    RES=$?
> +    if [[ $RES != 0 && ! -e "test-results/${test%.sh}" ]]; then
> +        exit $RES
> +    fi
>  done
>  trap - HUP INT TERM
>  
> diff --git a/test/test-lib.sh b/test/test-lib.sh
> index e092231..77063a4 100644
> --- a/test/test-lib.sh
> +++ b/test/test-lib.sh
> @@ -920,7 +920,7 @@ test_done () {
>  	GIT_EXIT_OK=t
>  	test_results_dir="$TEST_DIRECTORY/test-results"
>  	mkdir -p "$test_results_dir"
> -	test_results_path="$test_results_dir/${0%.sh}-$$"
> +	test_results_path="$test_results_dir/${0%.sh}"
>  
>  	echo "total $test_count" >> $test_results_path
>  	echo "success $test_success" >> $test_results_path
> -- 
> 1.7.10.4
>
> _______________________________________________
> notmuch mailing list
> notmuch@notmuchmail.org
> http://notmuchmail.org/mailman/listinfo/notmuch

Re: [PATCH] test: Abort driver if a test script aborts

[David Bremner]

Austin Clements <amdragon@MIT.EDU> writes:

> Previously, if a test script aborted (e.g., because it passed too few
> arguments to a test function), the test driver loop would simply
> continue on to the next test script and the final results would
> declare that everything passed (except that the test count would look
> suspiciously low, but maybe you just misremembered how many tests
> there were).

Pushed,

d