[PATCH 1/3] test: resolve `basename "$0" .sh` once for all in test-lib.sh

[PATCH 1/3] test: resolve `basename "$0" .sh` once for all in test-lib.sh

[Tomi Ollila]

test-lib.sh sometimes did equivalent of `basename "$0" .sh`, sometimes
skipping the basename part and sometimes .sh part. This worked as
we never had path components in $0 (more than ./) nor .sh ending.

Now the equivalent of `basename "$0" .sh` is done once and used
everywhere. In the future we may have .sh suffix in test names
-- removing those is a good idea.
---
 test/test-lib.sh | 12 +++++++-----
 1 file changed, 7 insertions(+), 5 deletions(-)

diff --git a/test/test-lib.sh b/test/test-lib.sh
index 2aa4dfc..808bb7f 100644
--- a/test/test-lib.sh
+++ b/test/test-lib.sh
@@ -25,6 +25,9 @@ fi
 # Make sure echo builtin does not expand backslash-escape sequences by default.
 shopt -u xpg_echo
 
+this_test=${0##*/}
+this_test=${this_test%.sh}
+
 # if --tee was passed, write the output not only to the terminal, but
 # additionally to the file test-results/$BASENAME.out, too.
 case "$GIT_TEST_TEE_STARTED, $* " in
@@ -33,7 +36,7 @@ done,*)
 	;;
 *' --tee '*|*' --va'*)
 	mkdir -p test-results
-	BASE=test-results/$(basename "$0" .sh)
+	BASE=test-results/$this_test
 	(GIT_TEST_TEE_STARTED=done ${SHELL-sh} "$0" "$@" 2>&1;
 	 echo $? > $BASE.exit) | tee $BASE.out
 	test "$(cat $BASE.exit)" = 0
@@ -187,7 +190,7 @@ then
 	exit 0
 fi
 
-echo $(basename "$0"): "Testing ${test_description}"
+echo $this_test: "Testing ${test_description}"
 
 exec 5>&1
 
@@ -956,7 +959,7 @@ test_done () {
 	GIT_EXIT_OK=t
 	test_results_dir="$TEST_DIRECTORY/test-results"
 	mkdir -p "$test_results_dir"
-	test_results_path="$test_results_dir/${0%.sh}"
+	test_results_path="$test_results_dir/$this_test"
 
 	echo "total $test_count" >> $test_results_path
 	echo "success $test_success" >> $test_results_path
@@ -1015,7 +1018,7 @@ test_emacs () {
 	test -z "$missing_dependencies" || return
 
 	if [ -z "$EMACS_SERVER" ]; then
-		emacs_tests="$(basename $0).el"
+		emacs_tests="${this_test}.el"
 		if [ -f "$TEST_DIRECTORY/$emacs_tests" ]; then
 			load_emacs_tests="--eval '(load \"$emacs_tests\")'"
 		else
@@ -1129,7 +1132,6 @@ else
 	exec 4>test.output 3>&4
 fi
 
-this_test=${0##*/}
 for skp in $NOTMUCH_SKIP_TESTS
 do
 	to_skip=
-- 
1.8.3.1

[PATCH 2/3] test: print empty line at the beginning of test script, not at end

[Tomi Ollila]

In preparation for quiet mode print empty line before writing the
test description. This is done now in function designed for it --
it will also be called when test fails.
---
 test/notmuch-test                          | 1 +
 test/test-lib.sh                           | 9 ++++++---
 test/test.expected-output/test-verbose-no  | 1 +
 test/test.expected-output/test-verbose-yes | 1 +
 4 files changed, 9 insertions(+), 3 deletions(-)

diff --git a/test/notmuch-test b/test/notmuch-test
index 18593f6..d6fdd3a 100755
--- a/test/notmuch-test
+++ b/test/notmuch-test
@@ -98,6 +98,7 @@ done
 trap - HUP INT TERM
 
 # Report results
+echo
 ./aggregate-results.sh test-results/*
 ev=$?
 
diff --git a/test/test-lib.sh b/test/test-lib.sh
index 808bb7f..e022e46 100644
--- a/test/test-lib.sh
+++ b/test/test-lib.sh
@@ -190,7 +190,12 @@ then
 	exit 0
 fi
 
-echo $this_test: "Testing ${test_description}"
+print_test_description ()
+{
+	echo
+	echo $this_test: "Testing ${test_description}"
+}
+print_test_description
 
 exec 5>&1
 
@@ -968,8 +973,6 @@ test_done () {
 	echo "failed $test_failure" >> $test_results_path
 	echo "" >> $test_results_path
 
-	echo
-
 	[ -n "$EMACS_SERVER" ] && test_emacs '(kill-emacs)'
 
 	if [ "$test_failure" = "0" ]; then
diff --git a/test/test.expected-output/test-verbose-no b/test/test.expected-output/test-verbose-no
index 0bca754..1a2ff61 100644
--- a/test/test.expected-output/test-verbose-no
+++ b/test/test.expected-output/test-verbose-no
@@ -1,3 +1,4 @@
+
 test-verbose: Testing the verbosity options of the test framework itself.
  PASS   print something in test_expect_success and pass
  FAIL   print something in test_expect_success and fail
diff --git a/test/test.expected-output/test-verbose-yes b/test/test.expected-output/test-verbose-yes
index ebe5187..d25466e 100644
--- a/test/test.expected-output/test-verbose-yes
+++ b/test/test.expected-output/test-verbose-yes
@@ -1,3 +1,4 @@
+
 test-verbose: Testing the verbosity options of the test framework itself.
 hello stdout
 hello stderr
-- 
1.8.3.1

[PATCH 3/3] test: implement and document NOTMUCH_TEST_QUIET variable usage

[Tomi Ollila]

When NOTMUCH_TEST_QUIET environment variable is set to non-null value
messages when new test script starts and when test PASSes are disabled.
This eases picking the cases when tests FAIL (as those are still printed).
---
 test/README                                      |  8 ++++++++
 test/basic                                       | 12 ++++++++++--
 test/test-lib.sh                                 | 11 ++++++++++-
 test/test.expected-output/test-quiet-verbose-no  | 20 ++++++++++++++++++++
 test/test.expected-output/test-quiet-verbose-yes | 24 ++++++++++++++++++++++++
 5 files changed, 72 insertions(+), 3 deletions(-)
 create mode 100644 test/test.expected-output/test-quiet-verbose-no
 create mode 100644 test/test.expected-output/test-quiet-verbose-yes

diff --git a/test/README b/test/README
index d12cff2..79a9b1b 100644
--- a/test/README
+++ b/test/README
@@ -76,6 +76,14 @@ the tests in one of the following ways.
 	TEST_EMACS=my-special-emacs TEST_EMACSCLIENT=my-emacsclient ./emacs
 	make test TEST_EMACS=my-special-emacs TEST_EMACSCLIENT=my-emacsclient
 
+Quiet Execution
+---------------
+
+Normally, when new script starts and when test PASSes you get a message
+printed on screen. This printing can be disabled by setting the
+NOTMUCH_TEST_QUIET variable to a non-null value. Message on test
+failures and skips are still printed.
+
 Skipping Tests
 --------------
 If, for any reason, you need to skip one or more tests, you can do so
diff --git a/test/basic b/test/basic
index 64eb7d7..3b7668b 100755
--- a/test/basic
+++ b/test/basic
@@ -73,14 +73,22 @@ suppress_diff_date() {
 	-e 's/\(.*\+\+\+ test-verbose\.4\.\output\).*/\1/'
 }
 
+if [ -z "$NOTMUCH_TEST_QUIET" ]
+then
+	test_verbose_no=$EXPECTED/test-verbose-no
+	test_verbose_yes=$EXPECTED/test-verbose-yes
+else
+	test_verbose_no=$EXPECTED/test-quiet-verbose-no
+	test_verbose_yes=$EXPECTED/test-quiet-verbose-yes
+fi
 test_begin_subtest "Ensure that test output is suppressed unless the test fails"
 output=$(cd $TEST_DIRECTORY; ./test-verbose 2>&1 | suppress_diff_date)
-expected=$(cat $EXPECTED/test-verbose-no | suppress_diff_date)
+expected=$(cat ${test_verbose_no} | suppress_diff_date)
 test_expect_equal "$output" "$expected"
 
 test_begin_subtest "Ensure that -v does not suppress test output"
 output=$(cd $TEST_DIRECTORY; ./test-verbose -v 2>&1 | suppress_diff_date)
-expected=$(cat $EXPECTED/test-verbose-yes | suppress_diff_date)
+expected=$(cat ${test_verbose_yes} | suppress_diff_date)
 # Do not include the results of test-verbose in totals
 rm $TEST_DIRECTORY/test-results/test-verbose
 rm -r $TEST_DIRECTORY/tmp.test-verbose
diff --git a/test/test-lib.sh b/test/test-lib.sh
index e022e46..4b342ac 100644
--- a/test/test-lib.sh
+++ b/test/test-lib.sh
@@ -195,7 +195,10 @@ print_test_description ()
 	echo
 	echo $this_test: "Testing ${test_description}"
 }
-print_test_description
+if [ -z "$NOTMUCH_TEST_QUIET" ]
+then
+	print_test_description
+fi
 
 exec 5>&1
 
@@ -703,6 +706,9 @@ test_ok_ () {
 		return
 	fi
 	test_success=$(($test_success + 1))
+	if test -n "$NOTMUCH_TEST_QUIET"; then
+		return 0
+	fi
 	say_color pass "%-6s" "PASS"
 	echo " $test_subtest_name"
 }
@@ -713,6 +719,9 @@ test_failure_ () {
 		return
 	fi
 	test_failure=$(($test_failure + 1))
+	if test -n "$NOTMUCH_TEST_QUIET"; then
+		print_test_description
+	fi
 	test_failure_message_ "FAIL" "$test_subtest_name" "$@"
 	test "$immediate" = "" || { GIT_EXIT_OK=t; exit 1; }
 	return 1
diff --git a/test/test.expected-output/test-quiet-verbose-no b/test/test.expected-output/test-quiet-verbose-no
new file mode 100644
index 0000000..74840b9
--- /dev/null
+++ b/test/test.expected-output/test-quiet-verbose-no
@@ -0,0 +1,20 @@
+
+test-verbose: Testing the verbosity options of the test framework itself.
+ FAIL   print something in test_expect_success and fail
+	
+	  echo "hello stdout" &&
+	  echo "hello stderr" >&2 &&
+	  false
+	
+hello stdout
+hello stderr
+
+test-verbose: Testing the verbosity options of the test framework itself.
+ FAIL   print something test_begin_subtest and test_expect_equal and fail
+	--- test-verbose.4.expected
+	+++ test-verbose.4.output
+	@@ -1 +1 @@
+	-b
+	+a
+hello stdout
+hello stderr
diff --git a/test/test.expected-output/test-quiet-verbose-yes b/test/test.expected-output/test-quiet-verbose-yes
new file mode 100644
index 0000000..51e759d
--- /dev/null
+++ b/test/test.expected-output/test-quiet-verbose-yes
@@ -0,0 +1,24 @@
+hello stdout
+hello stderr
+hello stdout
+hello stderr
+
+test-verbose: Testing the verbosity options of the test framework itself.
+ FAIL   print something in test_expect_success and fail
+	
+	  echo "hello stdout" &&
+	  echo "hello stderr" >&2 &&
+	  false
+	
+hello stdout
+hello stderr
+hello stdout
+hello stderr
+
+test-verbose: Testing the verbosity options of the test framework itself.
+ FAIL   print something test_begin_subtest and test_expect_equal and fail
+	--- test-verbose.4.expected
+	+++ test-verbose.4.output
+	@@ -1 +1 @@
+	-b
+	+a
-- 
1.8.3.1

Re: [PATCH 3/3] test: implement and document NOTMUCH_TEST_QUIET variable usage

[Austin Clements]

Quoth Tomi Ollila on Nov 12 at 10:41 pm:
> When NOTMUCH_TEST_QUIET environment variable is set to non-null value
> messages when new test script starts and when test PASSes are disabled.
> This eases picking the cases when tests FAIL (as those are still printed).
> ---
>  test/README                                      |  8 ++++++++
>  test/basic                                       | 12 ++++++++++--
>  test/test-lib.sh                                 | 11 ++++++++++-
>  test/test.expected-output/test-quiet-verbose-no  | 20 ++++++++++++++++++++
>  test/test.expected-output/test-quiet-verbose-yes | 24 ++++++++++++++++++++++++
>  5 files changed, 72 insertions(+), 3 deletions(-)
>  create mode 100644 test/test.expected-output/test-quiet-verbose-no
>  create mode 100644 test/test.expected-output/test-quiet-verbose-yes
> 
> diff --git a/test/README b/test/README
> index d12cff2..79a9b1b 100644
> --- a/test/README
> +++ b/test/README
> @@ -76,6 +76,14 @@ the tests in one of the following ways.
>  	TEST_EMACS=my-special-emacs TEST_EMACSCLIENT=my-emacsclient ./emacs
>  	make test TEST_EMACS=my-special-emacs TEST_EMACSCLIENT=my-emacsclient
>  
> +Quiet Execution
> +---------------
> +
> +Normally, when new script starts and when test PASSes you get a message
> +printed on screen. This printing can be disabled by setting the
> +NOTMUCH_TEST_QUIET variable to a non-null value. Message on test
> +failures and skips are still printed.
> +
>  Skipping Tests
>  --------------
>  If, for any reason, you need to skip one or more tests, you can do so
> diff --git a/test/basic b/test/basic
> index 64eb7d7..3b7668b 100755
> --- a/test/basic
> +++ b/test/basic
> @@ -73,14 +73,22 @@ suppress_diff_date() {
>  	-e 's/\(.*\+\+\+ test-verbose\.4\.\output\).*/\1/'
>  }
>  
> +if [ -z "$NOTMUCH_TEST_QUIET" ]
> +then
> +	test_verbose_no=$EXPECTED/test-verbose-no
> +	test_verbose_yes=$EXPECTED/test-verbose-yes
> +else
> +	test_verbose_no=$EXPECTED/test-quiet-verbose-no
> +	test_verbose_yes=$EXPECTED/test-quiet-verbose-yes
> +fi
>  test_begin_subtest "Ensure that test output is suppressed unless the test fails"
>  output=$(cd $TEST_DIRECTORY; ./test-verbose 2>&1 | suppress_diff_date)
> -expected=$(cat $EXPECTED/test-verbose-no | suppress_diff_date)
> +expected=$(cat ${test_verbose_no} | suppress_diff_date)
>  test_expect_equal "$output" "$expected"
>  
>  test_begin_subtest "Ensure that -v does not suppress test output"
>  output=$(cd $TEST_DIRECTORY; ./test-verbose -v 2>&1 | suppress_diff_date)
> -expected=$(cat $EXPECTED/test-verbose-yes | suppress_diff_date)
> +expected=$(cat ${test_verbose_yes} | suppress_diff_date)
>  # Do not include the results of test-verbose in totals
>  rm $TEST_DIRECTORY/test-results/test-verbose
>  rm -r $TEST_DIRECTORY/tmp.test-verbose
> diff --git a/test/test-lib.sh b/test/test-lib.sh
> index e022e46..4b342ac 100644
> --- a/test/test-lib.sh
> +++ b/test/test-lib.sh
> @@ -195,7 +195,10 @@ print_test_description ()
>  	echo
>  	echo $this_test: "Testing ${test_description}"
>  }
> -print_test_description
> +if [ -z "$NOTMUCH_TEST_QUIET" ]
> +then
> +	print_test_description
> +fi
>  
>  exec 5>&1
>  
> @@ -703,6 +706,9 @@ test_ok_ () {
>  		return
>  	fi
>  	test_success=$(($test_success + 1))
> +	if test -n "$NOTMUCH_TEST_QUIET"; then
> +		return 0
> +	fi
>  	say_color pass "%-6s" "PASS"
>  	echo " $test_subtest_name"
>  }
> @@ -713,6 +719,9 @@ test_failure_ () {
>  		return
>  	fi
>  	test_failure=$(($test_failure + 1))
> +	if test -n "$NOTMUCH_TEST_QUIET"; then
> +		print_test_description

This prints the test description for *every* failing test.  Was that
intentional?  I would think that, ideally, it would be only printed
before the first failing subtest in a test (maybe by setting a
variable in print_test_description on the first call and making it
return immediately if this variable is set?  Then you wouldn't even
need the condition here, just the call to print_test_description.)

> +	fi
>  	test_failure_message_ "FAIL" "$test_subtest_name" "$@"
>  	test "$immediate" = "" || { GIT_EXIT_OK=t; exit 1; }
>  	return 1

Stylistic nit: The three if's above use two different styles ([ vs
test and hanging 'then').  OTOH, maybe this is consistent with
test-lib's inconsistent style.

> diff --git a/test/test.expected-output/test-quiet-verbose-no b/test/test.expected-output/test-quiet-verbose-no
> new file mode 100644
> index 0000000..74840b9
> --- /dev/null
> +++ b/test/test.expected-output/test-quiet-verbose-no
> @@ -0,0 +1,20 @@
> +
> +test-verbose: Testing the verbosity options of the test framework itself.
> + FAIL   print something in test_expect_success and fail
> +	
> +	  echo "hello stdout" &&
> +	  echo "hello stderr" >&2 &&
> +	  false
> +	
> +hello stdout
> +hello stderr
> +
> +test-verbose: Testing the verbosity options of the test framework itself.
> + FAIL   print something test_begin_subtest and test_expect_equal and fail
> +	--- test-verbose.4.expected
> +	+++ test-verbose.4.output
> +	@@ -1 +1 @@
> +	-b
> +	+a
> +hello stdout
> +hello stderr
> diff --git a/test/test.expected-output/test-quiet-verbose-yes b/test/test.expected-output/test-quiet-verbose-yes
> new file mode 100644
> index 0000000..51e759d
> --- /dev/null
> +++ b/test/test.expected-output/test-quiet-verbose-yes
> @@ -0,0 +1,24 @@
> +hello stdout
> +hello stderr
> +hello stdout
> +hello stderr
> +
> +test-verbose: Testing the verbosity options of the test framework itself.
> + FAIL   print something in test_expect_success and fail
> +	
> +	  echo "hello stdout" &&
> +	  echo "hello stderr" >&2 &&
> +	  false
> +	
> +hello stdout
> +hello stderr
> +hello stdout
> +hello stderr
> +
> +test-verbose: Testing the verbosity options of the test framework itself.
> + FAIL   print something test_begin_subtest and test_expect_equal and fail
> +	--- test-verbose.4.expected
> +	+++ test-verbose.4.output
> +	@@ -1 +1 @@
> +	-b
> +	+a

Re: [PATCH 3/3] test: implement and document NOTMUCH_TEST_QUIET variable usage

[Tomi Ollila]

On Wed, Nov 13 2013, Austin Clements <amdragon@MIT.EDU> wrote:

> Quoth Tomi Ollila on Nov 12 at 10:41 pm:
>>  		return
>>  	fi
>>  	test_success=$(($test_success + 1))
>> +	if test -n "$NOTMUCH_TEST_QUIET"; then
>> +		return 0
>> +	fi
>>  	say_color pass "%-6s" "PASS"
>>  	echo " $test_subtest_name"
>>  }
>> @@ -713,6 +719,9 @@ test_failure_ () {
>>  		return
>>  	fi
>>  	test_failure=$(($test_failure + 1))
>> +	if test -n "$NOTMUCH_TEST_QUIET"; then
>> +		print_test_description
>
> This prints the test description for *every* failing test.  Was that
> intentional?  I would think that, ideally, it would be only printed
> before the first failing subtest in a test (maybe by setting a
> variable in print_test_description on the first call and making it
> return immediately if this variable is set?  Then you wouldn't even
> need the condition here, just the call to print_test_description.)

Your observation is correct.. I thought about it but dropped -- but
as it *increases* the output it should be addressed. My first solution
would be to do:

print_test_description ()
{
	echo
	echo $this_test: "Testing ${test_description}"
        print_test_description () { : already printed ; }
}

But I presume this receives some resistance from the audience ;/

I think this a bit -- this is post 0.17 release stuff anyway...

>
>> +	fi
>>  	test_failure_message_ "FAIL" "$test_subtest_name" "$@"
>>  	test "$immediate" = "" || { GIT_EXIT_OK=t; exit 1; }
>>  	return 1
>
> Stylistic nit: The three if's above use two different styles ([ vs
> test and hanging 'then').  OTOH, maybe this is consistent with
> test-lib's inconsistent style.

It is consistent with test-lib's inconsistent style: most often if []
is used but those places where I used if test just a line before
the diff context there were if test -format used.

Tomi

[PATCH 1/3] test: resolve `basename "$0" .sh` once for all in test-lib.sh

[Tomi Ollila]

test-lib.sh sometimes did equivalent of `basename "$0" .sh`, sometimes
skipping the basename part and sometimes .sh part. This worked as
we never had path components in $0 (more than ./) nor .sh ending.

Now the equivalent of `basename "$0" .sh` is done once and used
everywhere. In the future we may have .sh suffix in test names
-- removing those is a good idea.
---
 test/test-lib.sh | 12 +++++++-----
 1 file changed, 7 insertions(+), 5 deletions(-)

diff --git a/test/test-lib.sh b/test/test-lib.sh
index 8611ba5..6e47545 100644
--- a/test/test-lib.sh
+++ b/test/test-lib.sh
@@ -23,19 +23,22 @@ if [ ${BASH_VERSINFO[0]} -lt 4 ]; then
 fi
 
 # Make sure echo builtin does not expand backslash-escape sequences by default.
 shopt -u xpg_echo
 
+this_test=${0##*/}
+this_test=${this_test%.sh}
+
 # if --tee was passed, write the output not only to the terminal, but
 # additionally to the file test-results/$BASENAME.out, too.
 case "$GIT_TEST_TEE_STARTED, $* " in
 done,*)
 	# do not redirect again
 	;;
 *' --tee '*|*' --va'*)
 	mkdir -p test-results
-	BASE=test-results/$(basename "$0" .sh)
+	BASE=test-results/$this_test
 	(GIT_TEST_TEE_STARTED=done ${SHELL-sh} "$0" "$@" 2>&1;
 	 echo $? > $BASE.exit) | tee $BASE.out
 	test "$(cat $BASE.exit)" = 0
 	exit
 	;;
@@ -185,11 +188,11 @@ if test "$help" = "t"
 then
 	echo "Tests ${test_description}"
 	exit 0
 fi
 
-echo $(basename "$0"): "Testing ${test_description}"
+echo $this_test: "Testing ${test_description}"
 
 exec 5>&1
 
 test_failure=0
 test_count=0
@@ -965,11 +968,11 @@ test_when_finished () {
 
 test_done () {
 	GIT_EXIT_OK=t
 	test_results_dir="$TEST_DIRECTORY/test-results"
 	mkdir -p "$test_results_dir"
-	test_results_path="$test_results_dir/${0%.sh}"
+	test_results_path="$test_results_dir/$this_test"
 
 	echo "total $test_count" >> $test_results_path
 	echo "success $test_success" >> $test_results_path
 	echo "fixed $test_fixed" >> $test_results_path
 	echo "broken $test_broken" >> $test_results_path
@@ -1024,11 +1027,11 @@ test_emacs () {
 	test_require_external_prereq emacs || missing_dependencies=1
 	test_require_external_prereq ${TEST_EMACSCLIENT} || missing_dependencies=1
 	test -z "$missing_dependencies" || return
 
 	if [ -z "$EMACS_SERVER" ]; then
-		emacs_tests="$(basename $0).el"
+		emacs_tests="${this_test}.el"
 		if [ -f "$TEST_DIRECTORY/$emacs_tests" ]; then
 			load_emacs_tests="--eval '(load \"$emacs_tests\")'"
 		else
 			load_emacs_tests=
 		fi
@@ -1138,11 +1141,10 @@ then
 	exec 4>&2 3>&1
 else
 	exec 4>test.output 3>&4
 fi
 
-this_test=${0##*/}
 for skp in $NOTMUCH_SKIP_TESTS
 do
 	to_skip=
 	for skp in $NOTMUCH_SKIP_TESTS
 	do
-- 
1.8.4.2

[PATCH 2/3] test: print empty line at the beginning of test script, not at end

[Tomi Ollila]

In preparation for quiet mode print empty line before writing the
test description. This is done now in function designed for it --
it will also be called when test fails.
---
 test/notmuch-test                          |  1 +
 test/test-lib.sh                           | 12 +++++++++---
 test/test.expected-output/test-verbose-no  |  1 +
 test/test.expected-output/test-verbose-yes |  1 +
 4 files changed, 12 insertions(+), 3 deletions(-)

diff --git a/test/notmuch-test b/test/notmuch-test
index 18593f6..d6fdd3a 100755
--- a/test/notmuch-test
+++ b/test/notmuch-test
@@ -96,10 +96,11 @@ for test in $TESTS; do
     fi
 done
 trap - HUP INT TERM
 
 # Report results
+echo
 ./aggregate-results.sh test-results/*
 ev=$?
 
 # Clean up
 rm -rf test-results corpus.mail
diff --git a/test/test-lib.sh b/test/test-lib.sh
index 6e47545..34e0db6 100644
--- a/test/test-lib.sh
+++ b/test/test-lib.sh
@@ -188,11 +188,19 @@ if test "$help" = "t"
 then
 	echo "Tests ${test_description}"
 	exit 0
 fi
 
-echo $this_test: "Testing ${test_description}"
+test_description_printed=
+print_test_description ()
+{
+	test -z "$test_description_printed" || return 0
+	echo
+	echo $this_test: "Testing ${test_description}"
+	test_description_printed=1
+}
+print_test_description
 
 exec 5>&1
 
 test_failure=0
 test_count=0
@@ -977,12 +985,10 @@ test_done () {
 	echo "fixed $test_fixed" >> $test_results_path
 	echo "broken $test_broken" >> $test_results_path
 	echo "failed $test_failure" >> $test_results_path
 	echo "" >> $test_results_path
 
-	echo
-
 	[ -n "$EMACS_SERVER" ] && test_emacs '(kill-emacs)'
 
 	if [ "$test_failure" = "0" ]; then
 	    if [ "$test_broken" = "0" ]; then
 		rm -rf "$remove_tmp"
diff --git a/test/test.expected-output/test-verbose-no b/test/test.expected-output/test-verbose-no
index 0bca754..1a2ff61 100644
--- a/test/test.expected-output/test-verbose-no
+++ b/test/test.expected-output/test-verbose-no
@@ -1,5 +1,6 @@
+
 test-verbose: Testing the verbosity options of the test framework itself.
  PASS   print something in test_expect_success and pass
  FAIL   print something in test_expect_success and fail
 	
 	  echo "hello stdout" &&
diff --git a/test/test.expected-output/test-verbose-yes b/test/test.expected-output/test-verbose-yes
index ebe5187..d25466e 100644
--- a/test/test.expected-output/test-verbose-yes
+++ b/test/test.expected-output/test-verbose-yes
@@ -1,5 +1,6 @@
+
 test-verbose: Testing the verbosity options of the test framework itself.
 hello stdout
 hello stderr
  PASS   print something in test_expect_success and pass
 hello stdout
-- 
1.8.4.2

[PATCH 3/3] test: implement and document NOTMUCH_TEST_QUIET variable usage

[Tomi Ollila]

When NOTMUCH_TEST_QUIET environment variable is set to non-null value
messages when new test script starts and when test PASSes are disabled.
This eases picking the cases when tests FAIL (as those are still printed).
---
 test/README      |  8 ++++++++
 test/basic       |  4 ++--
 test/test-lib.sh | 11 ++++++++++-
 3 files changed, 20 insertions(+), 3 deletions(-)

diff --git a/test/README b/test/README
index d12cff2..79a9b1b 100644
--- a/test/README
+++ b/test/README
@@ -74,10 +74,18 @@ the tests in one of the following ways.
 
 	TEST_EMACS=my-special-emacs TEST_EMACSCLIENT=my-emacsclient make test
 	TEST_EMACS=my-special-emacs TEST_EMACSCLIENT=my-emacsclient ./emacs
 	make test TEST_EMACS=my-special-emacs TEST_EMACSCLIENT=my-emacsclient
 
+Quiet Execution
+---------------
+
+Normally, when new script starts and when test PASSes you get a message
+printed on screen. This printing can be disabled by setting the
+NOTMUCH_TEST_QUIET variable to a non-null value. Message on test
+failures and skips are still printed.
+
 Skipping Tests
 --------------
 If, for any reason, you need to skip one or more tests, you can do so
 by setting the NOTMUCH_SKIP_TESTS variable to the name of one or more
 sections of tests.
diff --git a/test/basic b/test/basic
index 64eb7d7..f7eed32 100755
--- a/test/basic
+++ b/test/basic
@@ -72,16 +72,16 @@ suppress_diff_date() {
     sed -e 's/\(.*\-\-\- test-verbose\.4\.\expected\).*/\1/' \
 	-e 's/\(.*\+\+\+ test-verbose\.4\.\output\).*/\1/'
 }
 
 test_begin_subtest "Ensure that test output is suppressed unless the test fails"
-output=$(cd $TEST_DIRECTORY; ./test-verbose 2>&1 | suppress_diff_date)
+output=$(cd $TEST_DIRECTORY; NOTMUCH_TEST_QUIET= ./test-verbose 2>&1 | suppress_diff_date)
 expected=$(cat $EXPECTED/test-verbose-no | suppress_diff_date)
 test_expect_equal "$output" "$expected"
 
 test_begin_subtest "Ensure that -v does not suppress test output"
-output=$(cd $TEST_DIRECTORY; ./test-verbose -v 2>&1 | suppress_diff_date)
+output=$(cd $TEST_DIRECTORY; NOTMUCH_TEST_QUIET= ./test-verbose -v 2>&1 | suppress_diff_date)
 expected=$(cat $EXPECTED/test-verbose-yes | suppress_diff_date)
 # Do not include the results of test-verbose in totals
 rm $TEST_DIRECTORY/test-results/test-verbose
 rm -r $TEST_DIRECTORY/tmp.test-verbose
 test_expect_equal "$output" "$expected"
diff --git a/test/test-lib.sh b/test/test-lib.sh
index 34e0db6..9d4a807 100644
--- a/test/test-lib.sh
+++ b/test/test-lib.sh
@@ -196,11 +196,14 @@ print_test_description ()
 	test -z "$test_description_printed" || return 0
 	echo
 	echo $this_test: "Testing ${test_description}"
 	test_description_printed=1
 }
-print_test_description
+if [ -z "$NOTMUCH_TEST_QUIET" ]
+then
+	print_test_description
+fi
 
 exec 5>&1
 
 test_failure=0
 test_count=0
@@ -715,20 +718,26 @@ test_ok_ () {
 	if test "$test_subtest_known_broken_" = "t"; then
 		test_known_broken_ok_
 		return
 	fi
 	test_success=$(($test_success + 1))
+	if test -n "$NOTMUCH_TEST_QUIET"; then
+		return 0
+	fi
 	say_color pass "%-6s" "PASS"
 	echo " $test_subtest_name"
 }
 
 test_failure_ () {
 	if test "$test_subtest_known_broken_" = "t"; then
 		test_known_broken_failure_ "$@"
 		return
 	fi
 	test_failure=$(($test_failure + 1))
+	if test -n "$NOTMUCH_TEST_QUIET"; then
+		print_test_description
+	fi
 	test_failure_message_ "FAIL" "$test_subtest_name" "$@"
 	test "$immediate" = "" || { GIT_EXIT_OK=t; exit 1; }
 	return 1
 }
 
-- 
1.8.4.2

Re: [PATCH 3/3] test: implement and document NOTMUCH_TEST_QUIET variable usage

[Austin Clements]

I just tried to use this and realized it hadn't been pushed yet.

This series LGTM except one minor nit below and the fact that it
introduces a lot of tab-indented code in sections of test-lib.sh that
appear to be space-indented.  Given that test-lib.sh is already a mess
of indentation styles, I don't know if we care, but it would be nice if
its entropy were at least non-increasing.

On Mon, 25 Nov 2013, Tomi Ollila <tomi.ollila@iki.fi> wrote:
> When NOTMUCH_TEST_QUIET environment variable is set to non-null value
> messages when new test script starts and when test PASSes are disabled.
> This eases picking the cases when tests FAIL (as those are still printed).
> ---
>  test/README      |  8 ++++++++
>  test/basic       |  4 ++--
>  test/test-lib.sh | 11 ++++++++++-
>  3 files changed, 20 insertions(+), 3 deletions(-)
>
> diff --git a/test/README b/test/README
> index d12cff2..79a9b1b 100644
> --- a/test/README
> +++ b/test/README
> @@ -74,10 +74,18 @@ the tests in one of the following ways.
>  
>  	TEST_EMACS=my-special-emacs TEST_EMACSCLIENT=my-emacsclient make test
>  	TEST_EMACS=my-special-emacs TEST_EMACSCLIENT=my-emacsclient ./emacs
>  	make test TEST_EMACS=my-special-emacs TEST_EMACSCLIENT=my-emacsclient
>  
> +Quiet Execution
> +---------------
> +
> +Normally, when new script starts and when test PASSes you get a message
> +printed on screen. This printing can be disabled by setting the
> +NOTMUCH_TEST_QUIET variable to a non-null value. Message on test
> +failures and skips are still printed.
> +
>  Skipping Tests
>  --------------
>  If, for any reason, you need to skip one or more tests, you can do so
>  by setting the NOTMUCH_SKIP_TESTS variable to the name of one or more
>  sections of tests.
> diff --git a/test/basic b/test/basic
> index 64eb7d7..f7eed32 100755
> --- a/test/basic
> +++ b/test/basic
> @@ -72,16 +72,16 @@ suppress_diff_date() {
>      sed -e 's/\(.*\-\-\- test-verbose\.4\.\expected\).*/\1/' \
>  	-e 's/\(.*\+\+\+ test-verbose\.4\.\output\).*/\1/'
>  }
>  
>  test_begin_subtest "Ensure that test output is suppressed unless the test fails"
> -output=$(cd $TEST_DIRECTORY; ./test-verbose 2>&1 | suppress_diff_date)
> +output=$(cd $TEST_DIRECTORY; NOTMUCH_TEST_QUIET= ./test-verbose 2>&1 | suppress_diff_date)
>  expected=$(cat $EXPECTED/test-verbose-no | suppress_diff_date)
>  test_expect_equal "$output" "$expected"
>  
>  test_begin_subtest "Ensure that -v does not suppress test output"
> -output=$(cd $TEST_DIRECTORY; ./test-verbose -v 2>&1 | suppress_diff_date)
> +output=$(cd $TEST_DIRECTORY; NOTMUCH_TEST_QUIET= ./test-verbose -v 2>&1 | suppress_diff_date)
>  expected=$(cat $EXPECTED/test-verbose-yes | suppress_diff_date)
>  # Do not include the results of test-verbose in totals
>  rm $TEST_DIRECTORY/test-results/test-verbose
>  rm -r $TEST_DIRECTORY/tmp.test-verbose
>  test_expect_equal "$output" "$expected"
> diff --git a/test/test-lib.sh b/test/test-lib.sh
> index 34e0db6..9d4a807 100644
> --- a/test/test-lib.sh
> +++ b/test/test-lib.sh
> @@ -196,11 +196,14 @@ print_test_description ()
>  	test -z "$test_description_printed" || return 0
>  	echo
>  	echo $this_test: "Testing ${test_description}"
>  	test_description_printed=1
>  }
> -print_test_description
> +if [ -z "$NOTMUCH_TEST_QUIET" ]
> +then
> +	print_test_description
> +fi
>  
>  exec 5>&1
>  
>  test_failure=0
>  test_count=0
> @@ -715,20 +718,26 @@ test_ok_ () {
>  	if test "$test_subtest_known_broken_" = "t"; then
>  		test_known_broken_ok_
>  		return
>  	fi
>  	test_success=$(($test_success + 1))
> +	if test -n "$NOTMUCH_TEST_QUIET"; then
> +		return 0
> +	fi
>  	say_color pass "%-6s" "PASS"
>  	echo " $test_subtest_name"
>  }
>  
>  test_failure_ () {
>  	if test "$test_subtest_known_broken_" = "t"; then
>  		test_known_broken_failure_ "$@"
>  		return
>  	fi
>  	test_failure=$(($test_failure + 1))
> +	if test -n "$NOTMUCH_TEST_QUIET"; then

Strictly speaking, this test isn't necessary, right?

> +		print_test_description
> +	fi
>  	test_failure_message_ "FAIL" "$test_subtest_name" "$@"
>  	test "$immediate" = "" || { GIT_EXIT_OK=t; exit 1; }
>  	return 1
>  }
>  
> -- 
> 1.8.4.2
>
> _______________________________________________
> notmuch mailing list
> notmuch@notmuchmail.org
> http://notmuchmail.org/mailman/listinfo/notmuch

Re: [PATCH 3/3] test: implement and document NOTMUCH_TEST_QUIET variable usage

[Austin Clements]

On Wed, 04 Dec 2013, Austin Clements <amdragon@MIT.EDU> wrote:
> I just tried to use this and realized it hadn't been pushed yet.
>
> This series LGTM except one minor nit below and the fact that it
> introduces a lot of tab-indented code in sections of test-lib.sh that
> appear to be space-indented.  Given that test-lib.sh is already a mess
> of indentation styles, I don't know if we care, but it would be nice if
> its entropy were at least non-increasing.

Ignore this comment.  Apparently I was confused by show-mode
transforming leading tabs into spaces.

> On Mon, 25 Nov 2013, Tomi Ollila <tomi.ollila@iki.fi> wrote:
>> When NOTMUCH_TEST_QUIET environment variable is set to non-null value
>> messages when new test script starts and when test PASSes are disabled.
>> This eases picking the cases when tests FAIL (as those are still printed).
>> ---
>>  test/README      |  8 ++++++++
>>  test/basic       |  4 ++--
>>  test/test-lib.sh | 11 ++++++++++-
>>  3 files changed, 20 insertions(+), 3 deletions(-)
>>
>> diff --git a/test/README b/test/README
>> index d12cff2..79a9b1b 100644
>> --- a/test/README
>> +++ b/test/README
>> @@ -74,10 +74,18 @@ the tests in one of the following ways.
>>  
>>  	TEST_EMACS=my-special-emacs TEST_EMACSCLIENT=my-emacsclient make test
>>  	TEST_EMACS=my-special-emacs TEST_EMACSCLIENT=my-emacsclient ./emacs
>>  	make test TEST_EMACS=my-special-emacs TEST_EMACSCLIENT=my-emacsclient
>>  
>> +Quiet Execution
>> +---------------
>> +
>> +Normally, when new script starts and when test PASSes you get a message
>> +printed on screen. This printing can be disabled by setting the
>> +NOTMUCH_TEST_QUIET variable to a non-null value. Message on test
>> +failures and skips are still printed.
>> +
>>  Skipping Tests
>>  --------------
>>  If, for any reason, you need to skip one or more tests, you can do so
>>  by setting the NOTMUCH_SKIP_TESTS variable to the name of one or more
>>  sections of tests.
>> diff --git a/test/basic b/test/basic
>> index 64eb7d7..f7eed32 100755
>> --- a/test/basic
>> +++ b/test/basic
>> @@ -72,16 +72,16 @@ suppress_diff_date() {
>>      sed -e 's/\(.*\-\-\- test-verbose\.4\.\expected\).*/\1/' \
>>  	-e 's/\(.*\+\+\+ test-verbose\.4\.\output\).*/\1/'
>>  }
>>  
>>  test_begin_subtest "Ensure that test output is suppressed unless the test fails"
>> -output=$(cd $TEST_DIRECTORY; ./test-verbose 2>&1 | suppress_diff_date)
>> +output=$(cd $TEST_DIRECTORY; NOTMUCH_TEST_QUIET= ./test-verbose 2>&1 | suppress_diff_date)
>>  expected=$(cat $EXPECTED/test-verbose-no | suppress_diff_date)
>>  test_expect_equal "$output" "$expected"
>>  
>>  test_begin_subtest "Ensure that -v does not suppress test output"
>> -output=$(cd $TEST_DIRECTORY; ./test-verbose -v 2>&1 | suppress_diff_date)
>> +output=$(cd $TEST_DIRECTORY; NOTMUCH_TEST_QUIET= ./test-verbose -v 2>&1 | suppress_diff_date)
>>  expected=$(cat $EXPECTED/test-verbose-yes | suppress_diff_date)
>>  # Do not include the results of test-verbose in totals
>>  rm $TEST_DIRECTORY/test-results/test-verbose
>>  rm -r $TEST_DIRECTORY/tmp.test-verbose
>>  test_expect_equal "$output" "$expected"
>> diff --git a/test/test-lib.sh b/test/test-lib.sh
>> index 34e0db6..9d4a807 100644
>> --- a/test/test-lib.sh
>> +++ b/test/test-lib.sh
>> @@ -196,11 +196,14 @@ print_test_description ()
>>  	test -z "$test_description_printed" || return 0
>>  	echo
>>  	echo $this_test: "Testing ${test_description}"
>>  	test_description_printed=1
>>  }
>> -print_test_description
>> +if [ -z "$NOTMUCH_TEST_QUIET" ]
>> +then
>> +	print_test_description
>> +fi
>>  
>>  exec 5>&1
>>  
>>  test_failure=0
>>  test_count=0
>> @@ -715,20 +718,26 @@ test_ok_ () {
>>  	if test "$test_subtest_known_broken_" = "t"; then
>>  		test_known_broken_ok_
>>  		return
>>  	fi
>>  	test_success=$(($test_success + 1))
>> +	if test -n "$NOTMUCH_TEST_QUIET"; then
>> +		return 0
>> +	fi
>>  	say_color pass "%-6s" "PASS"
>>  	echo " $test_subtest_name"
>>  }
>>  
>>  test_failure_ () {
>>  	if test "$test_subtest_known_broken_" = "t"; then
>>  		test_known_broken_failure_ "$@"
>>  		return
>>  	fi
>>  	test_failure=$(($test_failure + 1))
>> +	if test -n "$NOTMUCH_TEST_QUIET"; then
>
> Strictly speaking, this test isn't necessary, right?
>
>> +		print_test_description
>> +	fi
>>  	test_failure_message_ "FAIL" "$test_subtest_name" "$@"
>>  	test "$immediate" = "" || { GIT_EXIT_OK=t; exit 1; }
>>  	return 1
>>  }
>>  
>> -- 
>> 1.8.4.2

[PATCH v3 part 3/3] test: implement and document NOTMUCH_TEST_QUIET variable usage

[Tomi Ollila]

When NOTMUCH_TEST_QUIET environment variable is set to non-null value
messages when new test script starts and when test PASSes are disabled.
This eases picking the cases when tests FAIL (as those are still printed).
---

Diffdiff of part 3 to previous.

| diff --git a/test/test-lib.sh b/test/test-lib.sh
| index 9d4a807..d8e0d91 100644
| --- a/test/test-lib.sh
| +++ b/test/test-lib.sh
| @@ -733,9 +733,7 @@ test_failure_ () {
|  		return
|  	fi
|  	test_failure=$(($test_failure + 1))
| -	if test -n "$NOTMUCH_TEST_QUIET"; then
| -		print_test_description
| -	fi
| +	print_test_description
|  	test_failure_message_ "FAIL" "$test_subtest_name" "$@"
|  	test "$immediate" = "" || { GIT_EXIT_OK=t; exit 1; }
|  	return 1


 test/README      | 8 ++++++++
 test/basic       | 4 ++--
 test/test-lib.sh | 9 ++++++++-
 3 files changed, 18 insertions(+), 3 deletions(-)

diff --git a/test/README b/test/README
index d12cff2..79a9b1b 100644
--- a/test/README
+++ b/test/README
@@ -76,6 +76,14 @@ the tests in one of the following ways.
 	TEST_EMACS=my-special-emacs TEST_EMACSCLIENT=my-emacsclient ./emacs
 	make test TEST_EMACS=my-special-emacs TEST_EMACSCLIENT=my-emacsclient
 
+Quiet Execution
+---------------
+
+Normally, when new script starts and when test PASSes you get a message
+printed on screen. This printing can be disabled by setting the
+NOTMUCH_TEST_QUIET variable to a non-null value. Message on test
+failures and skips are still printed.
+
 Skipping Tests
 --------------
 If, for any reason, you need to skip one or more tests, you can do so
diff --git a/test/basic b/test/basic
index 64eb7d7..f7eed32 100755
--- a/test/basic
+++ b/test/basic
@@ -74,12 +74,12 @@ suppress_diff_date() {
 }
 
 test_begin_subtest "Ensure that test output is suppressed unless the test fails"
-output=$(cd $TEST_DIRECTORY; ./test-verbose 2>&1 | suppress_diff_date)
+output=$(cd $TEST_DIRECTORY; NOTMUCH_TEST_QUIET= ./test-verbose 2>&1 | suppress_diff_date)
 expected=$(cat $EXPECTED/test-verbose-no | suppress_diff_date)
 test_expect_equal "$output" "$expected"
 
 test_begin_subtest "Ensure that -v does not suppress test output"
-output=$(cd $TEST_DIRECTORY; ./test-verbose -v 2>&1 | suppress_diff_date)
+output=$(cd $TEST_DIRECTORY; NOTMUCH_TEST_QUIET= ./test-verbose -v 2>&1 | suppress_diff_date)
 expected=$(cat $EXPECTED/test-verbose-yes | suppress_diff_date)
 # Do not include the results of test-verbose in totals
 rm $TEST_DIRECTORY/test-results/test-verbose
diff --git a/test/test-lib.sh b/test/test-lib.sh
index 34e0db6..d8e0d91 100644
--- a/test/test-lib.sh
+++ b/test/test-lib.sh
@@ -198,7 +198,10 @@ print_test_description ()
 	echo $this_test: "Testing ${test_description}"
 	test_description_printed=1
 }
-print_test_description
+if [ -z "$NOTMUCH_TEST_QUIET" ]
+then
+	print_test_description
+fi
 
 exec 5>&1
 
@@ -717,6 +720,9 @@ test_ok_ () {
 		return
 	fi
 	test_success=$(($test_success + 1))
+	if test -n "$NOTMUCH_TEST_QUIET"; then
+		return 0
+	fi
 	say_color pass "%-6s" "PASS"
 	echo " $test_subtest_name"
 }
@@ -727,6 +733,7 @@ test_failure_ () {
 		return
 	fi
 	test_failure=$(($test_failure + 1))
+	print_test_description
 	test_failure_message_ "FAIL" "$test_subtest_name" "$@"
 	test "$immediate" = "" || { GIT_EXIT_OK=t; exit 1; }
 	return 1
-- 
1.8.0

Re: [PATCH v3 part 3/3] test: implement and document NOTMUCH_TEST_QUIET variable usage

[Austin Clements]

v3 LGTM.

Quoth Tomi Ollila on Dec 05 at  2:46 pm:
> When NOTMUCH_TEST_QUIET environment variable is set to non-null value
> messages when new test script starts and when test PASSes are disabled.
> This eases picking the cases when tests FAIL (as those are still printed).
> ---
> 
> Diffdiff of part 3 to previous.
> 
> | diff --git a/test/test-lib.sh b/test/test-lib.sh
> | index 9d4a807..d8e0d91 100644
> | --- a/test/test-lib.sh
> | +++ b/test/test-lib.sh
> | @@ -733,9 +733,7 @@ test_failure_ () {
> |  		return
> |  	fi
> |  	test_failure=$(($test_failure + 1))
> | -	if test -n "$NOTMUCH_TEST_QUIET"; then
> | -		print_test_description
> | -	fi
> | +	print_test_description
> |  	test_failure_message_ "FAIL" "$test_subtest_name" "$@"
> |  	test "$immediate" = "" || { GIT_EXIT_OK=t; exit 1; }
> |  	return 1
> 
> 
>  test/README      | 8 ++++++++
>  test/basic       | 4 ++--
>  test/test-lib.sh | 9 ++++++++-
>  3 files changed, 18 insertions(+), 3 deletions(-)

Re: [PATCH v3 part 3/3] test: implement and document NOTMUCH_TEST_QUIET variable usage

[David Bremner]

Tomi Ollila <tomi.ollila@iki.fi> writes:

> When NOTMUCH_TEST_QUIET environment variable is set to non-null value
> messages when new test script starts and when test PASSes are disabled.
> This eases picking the cases when tests FAIL (as those are still printed).
> ---

series pushed to master

d