[PATCH] test:Improve test behaviors when --root is used

[PATCH] test:Improve test behaviors when --root is used

[Mark Anderson]

Change add_email_corpus, emacs_deliver_message and tests to use
$TEST_DIRECTORY instead of '..'.

This improves the behavior of the usage of --root=<dir>, as the
assumption of '..' will be incorrect if the option is specified.

Document -root option in README and update valgrind to work with
-root.
---

I discovered the -root option when I wanted to run my test suite on
tmpfs, since my main drive is still spinning rust, and found test run
relocation is already implemented.

However, many tests were not well behaved when --root was specified,
which makes sense since it wasn't documented in the README, added in
this patch.

It seems like a good idea to try and run the test suite on a tmpfs
drive, but I don't know if there is a sufficiently generic standard
location or detection mechanism.  I use /dev/shm/notmuch_test, which
works for me.  I do wish I didn't have to keep specifying it, but I'm
not sure how well received putting this in notmuch's configure file
would be.


 test/README        |    9 +++++++++
 test/basic         |   10 +++++-----
 test/crypto        |    2 +-
 test/emacs         |    4 ++--
 test/symbol-hiding |    4 ++--
 test/test-lib.sh   |   18 +++++++++---------
 6 files changed, 28 insertions(+), 19 deletions(-)

diff --git a/test/README b/test/README
index be75e0e..8fbf78d 100644
--- a/test/README
+++ b/test/README
@@ -41,6 +41,15 @@ The following command-line options are available when running tests:
 	As the names depend on the tests' file names, it is safe to
 	run the tests with this option in parallel.
 
+--root=<dir>::
+	This runs the testsuites specified under a seperate directory.
+	However, caution is advised, as not all tests are maintained
+	with this relocation in mind, so some tests may behave
+	differently.
+
+	Pointing this argument at a tmpfs filesystem can improve the
+	speed of the test suite for some users.
+
 When invoking the test suite via "make test" any of the above options
 can be specified as follows:
 
diff --git a/test/basic b/test/basic
index d6e8c10..33bf711 100755
--- a/test/basic
+++ b/test/basic
@@ -51,9 +51,9 @@ test_expect_code 2 'failure to clean up causes the test to fail' '
 
 # Ensure that all tests are being run
 test_begin_subtest 'Ensure that all available tests will be run by notmuch-test'
-eval $(sed -n -e '/^TESTS="$/,/^"$/p' notmuch-test ../notmuch-test)
+eval $(sed -n -e '/^TESTS="$/,/^"$/p' notmuch-test $TEST_DIRECTORY/notmuch-test)
 tests_in_suite=$(for i in $TESTS; do echo $i; done | sort)
-available=$(ls -1 ../ | \
+available=$(ls -1 $TEST_DIRECTORY/ | \
     sed -r -e "/^(aggregate-results.sh|Makefile|Makefile.local|notmuch-test)/d" \
 	   -e "/^(README|test-lib.sh|test-lib.el|test-results|tmp.*|valgrind|corpus*)/d" \
 	   -e "/^(emacs.expected-output|smtp-dummy|smtp-dummy.c|test-verbose|symbol-test.cc)/d" \
@@ -63,19 +63,19 @@ available=$(ls -1 ../ | \
 	   | sort)
 test_expect_equal "$tests_in_suite" "$available"
 
-EXPECTED=../test.expected-output
+EXPECTED=$TEST_DIRECTORY/test.expected-output
 suppress_diff_date() {
     sed -e 's/\(.*\-\-\- test-verbose\.4\.\expected\).*/\1/' \
 	-e 's/\(.*\+\+\+ test-verbose\.4\.\output\).*/\1/'
 }
 
 test_begin_subtest "Ensure that test output is suppressed unless the test fails"
-output=$(cd ..; ./test-verbose 2>&1 | suppress_diff_date)
+output=$(cd $TEST_DIRECTORY; ./test-verbose 2>&1 | suppress_diff_date)
 expected=$(cat $EXPECTED/test-verbose-no | suppress_diff_date)
 test_expect_equal "$output" "$expected"
 
 test_begin_subtest "Ensure that -v does not suppress test output"
-output=$(cd ..; ./test-verbose -v 2>&1 | suppress_diff_date)
+output=$(cd $TEST_DIRECTORY; ./test-verbose -v 2>&1 | suppress_diff_date)
 expected=$(cat $EXPECTED/test-verbose-yes | suppress_diff_date)
 # Do not include the results of test-verbose in totals
 rm $TEST_DIRECTORY/test-results/test-verbose-*
diff --git a/test/crypto b/test/crypto
index 01daffe..7eb3559 100755
--- a/test/crypto
+++ b/test/crypto
@@ -12,7 +12,7 @@ add_gnupg_home ()
     local output
     [ -d ${GNUPGHOME} ] && return
     mkdir -m 0700 "$GNUPGHOME"
-    gpg --no-tty --import <../gnupg-secret-key.asc >"$GNUPGHOME"/import.log 2>&1
+    gpg --no-tty --import <$TEST_DIRECTORY/gnupg-secret-key.asc >"$GNUPGHOME"/import.log 2>&1
     test_debug "cat $GNUPGHOME/import.log"
     if (gpg --quick-random --version >/dev/null 2>&1) ; then
 	echo quick-random >> "$GNUPGHOME"/gpg.conf
diff --git a/test/emacs b/test/emacs
index 6f82b08..f91078e 100755
--- a/test/emacs
+++ b/test/emacs
@@ -2,7 +2,7 @@
 test_description="emacs interface"
 . test-lib.sh
 
-EXPECTED=../emacs.expected-output
+EXPECTED=$TEST_DIRECTORY/emacs.expected-output
 
 add_email_corpus
 
@@ -81,7 +81,7 @@ mkdir -p mail/sent/cur
 mkdir -p mail/sent/new
 mkdir -p mail/sent/tmp
 
-../smtp-dummy sent_message &
+$TEST_DIRECTORY/smtp-dummy sent_message &
 smtp_dummy_pid=$!
 test_emacs "(setq message-send-mail-function 'message-smtpmail-send-it) (setq smtpmail-smtp-server \"localhost\") (setq smtpmail-smtp-service \"25025\") (notmuch-hello) (notmuch-mua-mail) (message-goto-to) (insert \"user@example.com\nDate: Fri, 29 Mar 1974 10:00:00 -0000\") (message-goto-subject) (insert \"Testing message sent via SMTP\") (message-goto-body) (insert \"This is a test that messages are sent via SMTP\") (message-send-and-exit)" >/dev/null 2>&1
 wait ${smtp_dummy_pid}
diff --git a/test/symbol-hiding b/test/symbol-hiding
index bb55524..5d38fb7 100755
--- a/test/symbol-hiding
+++ b/test/symbol-hiding
@@ -12,13 +12,13 @@ test_description='exception symbol hiding'
 . ./test-lib.sh
 
 run_test(){
-    result=$(LD_LIBRARY_PATH=../../lib ./symbol-test 2>&1)
+    result=$(LD_LIBRARY_PATH=$TEST_DIRECTORY/../../lib ./symbol-test 2>&1)
 }
 
 output="A Xapian exception occurred opening database: Couldn't stat 'fakedb/.notmuch/xapian'
 caught No chert database found at path \`./nonexistant'"
 
-g++ -o symbol-test -I../../lib ../symbol-test.cc -L../../lib -lnotmuch -lxapian
+g++ -o symbol-test -I$TEST_DIRECTORY/../lib $TEST_DIRECTORY/symbol-test.cc -L$TEST_DIRECTORY/../lib -lnotmuch -lxapian
 mkdir -p fakedb/.notmuch
 test_expect_success 'running test' run_test
 test_begin_subtest 'checking output'
diff --git a/test/test-lib.sh b/test/test-lib.sh
index cc20f41..352be39 100755
--- a/test/test-lib.sh
+++ b/test/test-lib.sh
@@ -392,7 +392,7 @@ emacs_deliver_message ()
     shift 2
     # before we can send a message, we have to prepare the FCC maildir
     mkdir -p "$MAIL_DIR"/sent/{cur,new,tmp}
-    ../smtp-dummy sent_message &
+    $TEST_DIRECTORY/smtp-dummy sent_message &
     smtp_dummy_pid=$!
     test_emacs "(setq message-send-mail-function 'message-smtpmail-send-it) (setq smtpmail-smtp-server \"localhost\") (setq smtpmail-smtp-service \"25025\") (notmuch-hello) (notmuch-mua-mail) (message-goto-to) (insert \"test_suite@notmuchmail.org\nDate: 01 Jan 2000 12:00:00 -0000\") (message-goto-subject) (insert \"${subject}\") (message-goto-body) (insert \"${body}\") $@ (message-send-and-exit)" >/dev/null 2>&1
     wait ${smtp_dummy_pid}
@@ -408,12 +408,12 @@ emacs_deliver_message ()
 add_email_corpus ()
 {
     rm -rf ${MAIL_DIR}
-    if [ -d ../corpus.mail ]; then
-	cp -a ../corpus.mail ${MAIL_DIR}
+    if [ -d $TEST_DIRECTORY/corpus.mail ]; then
+	cp -a $TEST_DIRECTORY/corpus.mail ${MAIL_DIR}
     else
-	cp -a ../corpus ${MAIL_DIR}
+	cp -a $TEST_DIRECTORY/corpus ${MAIL_DIR}
 	notmuch new >/dev/null
-	cp -a ${MAIL_DIR} ../corpus.mail
+	cp -a ${MAIL_DIR} $TEST_DIRECTORY/corpus.mail
     fi
 }
 
@@ -860,8 +860,8 @@ fi
 # set-frame-width	80 columns (avoids crazy 10-column default of --batch)
 
 emacs \$BATCH --no-init-file --no-site-file \
-	--directory ../../emacs --load notmuch.el \
-	--directory .. --load test-lib.el \
+	--directory $TEST_DIRECTORY/../emacs --load notmuch.el \
+	--directory $TEST_DIRECTORY --load test-lib.el \
 	--eval "(defun notmuch-test-wait ()
 			(while (get-buffer-process (current-buffer))
 				(sleep-for 0.1)))" \
@@ -925,11 +925,11 @@ then
 		    test ! -d "$symlink_target" &&
 		    test "#!" != "$(head -c 2 < "$symlink_target")"
 		then
-			symlink_target=../valgrind.sh
+			symlink_target=$TEST_DIRECTORY/valgrind.sh
 		fi
 		case "$base" in
 		*.sh|*.perl)
-			symlink_target=../unprocessed-script
+			symlink_target=$TEST_DIRECTORY/unprocessed-script
 		esac
 		# create the link, or replace it if it is out of date
 		make_symlink "$symlink_target" "$GIT_VALGRIND/bin/$base" || exit
-- 
1.7.4.1

Re: [PATCH] test:Improve test behaviors when --root is used

[Austin Clements]

This looks great (modulo one bug, below).  I've wanted to run the
tests on tmpfs before, but was too lazy to actually fix the tests.

Given how easy it is to accidentally use "..", I wonder if there's a
way to force people to use $TEST_DIRECTORY?

On Mon, Jun 27, 2011 at 12:09 PM, Mark Anderson <ma.skies@gmail.com> wrote:
> --- a/test/symbol-hiding
> +++ b/test/symbol-hiding
> @@ -12,13 +12,13 @@ test_description='exception symbol hiding'
>  . ./test-lib.sh
>
>  run_test(){
> -    result=$(LD_LIBRARY_PATH=../../lib ./symbol-test 2>&1)
> +    result=$(LD_LIBRARY_PATH=$TEST_DIRECTORY/../../lib ./symbol-test 2>&1)

Did you mean $TEST_DIRECTORY/../lib?

Re: [PATCH] test:Improve test behaviors when --root is used

[Mark Anderson]

On Mon, 27 Jun 2011 15:50:47 -0500, Austin Clements <amdragon@mit.edu> wrote:
> This looks great (modulo one bug, below).  I've wanted to run the
> tests on tmpfs before, but was too lazy to actually fix the tests.
> 
> Given how easy it is to accidentally use "..", I wonder if there's a
> way to force people to use $TEST_DIRECTORY?

I was thinking of a test of the test-suite running in this mode, but I
didn't quite feel comfortable, since it feels a bit slow sometimes
already.

So an instruction to run 'make test', which would include a 'make test
OPTIONS="--root=/dev/shm/notmuch_test.$$"' or some such if that was
acceptable.

I didn't think about it very hard, but I was intrigued by the idea.

-Mark

> On Mon, Jun 27, 2011 at 12:09 PM, Mark Anderson <ma.skies@gmail.com> wrote:
> > --- a/test/symbol-hiding
> > +++ b/test/symbol-hiding
> > @@ -12,13 +12,13 @@ test_description='exception symbol hiding'
> >  . ./test-lib.sh
> >
> >  run_test(){
> > -    result=$(LD_LIBRARY_PATH=../../lib ./symbol-test 2>&1)
> > +    result=$(LD_LIBRARY_PATH=$TEST_DIRECTORY/../../lib ./symbol-test 2>&1)
> 
> Did you mean $TEST_DIRECTORY/../lib?
> _______________________________________________
> notmuch mailing list
> notmuch@notmuchmail.org
> http://notmuchmail.org/mailman/listinfo/notmuch
> 

Re: [PATCH] test:Improve test behaviors when --root is used

[Carl Worth]

[-- Attachment #1: Type: text/plain, Size: 850 bytes --]

On Mon, 27 Jun 2011 17:03:03 -0600, Mark Anderson <MarkR.Anderson@amd.com> wrote:
> On Mon, 27 Jun 2011 15:50:47 -0500, Austin Clements <amdragon@mit.edu> wrote:
> > This looks great (modulo one bug, below).  I've wanted to run the
> > tests on tmpfs before, but was too lazy to actually fix the tests.

Yes, it does look great. Thanks for the patch, Mark!

Did you miss Austin's note about a possible bug? (You quoted it but
didn't reply). See below:

> > >  run_test(){
> > > -    result=$(LD_LIBRARY_PATH=../../lib ./symbol-test 2>&1)
> > > +    result=$(LD_LIBRARY_PATH=$TEST_DIRECTORY/../../lib ./symbol-test 2>&1)
> > 
> > Did you mean $TEST_DIRECTORY/../lib?

If you could follow up with an updated patch, (or an argument that the
original patch is correct), that would be great.

-Carl

-- 
carl.d.worth@intel.com

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

[PATCH v2] test:Improve test behaviors when --root is used

[Mark Anderson]

Change add_email_corpus, emacs_deliver_message and tests to use
$TEST_DIRECTORY instead of '..'.

This improves the behavior of the usage of --root=<dir>, as the
assumption of what '..' means will usually be incorrect.

Document -root option in README and update valgrind to work with
-root.

Signed-off-by: Mark Anderson <ma.skies@gmail.com>
---
> If you could follow up with an updated patch, (or an argument that the
> original patch is correct), that would be great.

Updated the patch with Austin's suggestion.

I'm not personally ready to forbid the use of '..', although I certainly
appreciate the motivation to keep our tests working in --root mode.

 test/README        |    9 +++++++++
 test/basic         |   10 +++++-----
 test/crypto        |    2 +-
 test/emacs         |    4 ++--
 test/symbol-hiding |    4 ++--
 test/test-lib.sh   |   18 +++++++++---------
 6 files changed, 28 insertions(+), 19 deletions(-)

diff --git a/test/README b/test/README
index be75e0e..8fbf78d 100644
--- a/test/README
+++ b/test/README
@@ -41,6 +41,15 @@ The following command-line options are available when running tests:
 	As the names depend on the tests' file names, it is safe to
 	run the tests with this option in parallel.
 
+--root=<dir>::
+	This runs the testsuites specified under a seperate directory.
+	However, caution is advised, as not all tests are maintained
+	with this relocation in mind, so some tests may behave
+	differently.
+
+	Pointing this argument at a tmpfs filesystem can improve the
+	speed of the test suite for some users.
+
 When invoking the test suite via "make test" any of the above options
 can be specified as follows:
 
diff --git a/test/basic b/test/basic
index d6e8c10..33bf711 100755
--- a/test/basic
+++ b/test/basic
@@ -51,9 +51,9 @@ test_expect_code 2 'failure to clean up causes the test to fail' '
 
 # Ensure that all tests are being run
 test_begin_subtest 'Ensure that all available tests will be run by notmuch-test'
-eval $(sed -n -e '/^TESTS="$/,/^"$/p' notmuch-test ../notmuch-test)
+eval $(sed -n -e '/^TESTS="$/,/^"$/p' notmuch-test $TEST_DIRECTORY/notmuch-test)
 tests_in_suite=$(for i in $TESTS; do echo $i; done | sort)
-available=$(ls -1 ../ | \
+available=$(ls -1 $TEST_DIRECTORY/ | \
     sed -r -e "/^(aggregate-results.sh|Makefile|Makefile.local|notmuch-test)/d" \
 	   -e "/^(README|test-lib.sh|test-lib.el|test-results|tmp.*|valgrind|corpus*)/d" \
 	   -e "/^(emacs.expected-output|smtp-dummy|smtp-dummy.c|test-verbose|symbol-test.cc)/d" \
@@ -63,19 +63,19 @@ available=$(ls -1 ../ | \
 	   | sort)
 test_expect_equal "$tests_in_suite" "$available"
 
-EXPECTED=../test.expected-output
+EXPECTED=$TEST_DIRECTORY/test.expected-output
 suppress_diff_date() {
     sed -e 's/\(.*\-\-\- test-verbose\.4\.\expected\).*/\1/' \
 	-e 's/\(.*\+\+\+ test-verbose\.4\.\output\).*/\1/'
 }
 
 test_begin_subtest "Ensure that test output is suppressed unless the test fails"
-output=$(cd ..; ./test-verbose 2>&1 | suppress_diff_date)
+output=$(cd $TEST_DIRECTORY; ./test-verbose 2>&1 | suppress_diff_date)
 expected=$(cat $EXPECTED/test-verbose-no | suppress_diff_date)
 test_expect_equal "$output" "$expected"
 
 test_begin_subtest "Ensure that -v does not suppress test output"
-output=$(cd ..; ./test-verbose -v 2>&1 | suppress_diff_date)
+output=$(cd $TEST_DIRECTORY; ./test-verbose -v 2>&1 | suppress_diff_date)
 expected=$(cat $EXPECTED/test-verbose-yes | suppress_diff_date)
 # Do not include the results of test-verbose in totals
 rm $TEST_DIRECTORY/test-results/test-verbose-*
diff --git a/test/crypto b/test/crypto
index 01daffe..7eb3559 100755
--- a/test/crypto
+++ b/test/crypto
@@ -12,7 +12,7 @@ add_gnupg_home ()
     local output
     [ -d ${GNUPGHOME} ] && return
     mkdir -m 0700 "$GNUPGHOME"
-    gpg --no-tty --import <../gnupg-secret-key.asc >"$GNUPGHOME"/import.log 2>&1
+    gpg --no-tty --import <$TEST_DIRECTORY/gnupg-secret-key.asc >"$GNUPGHOME"/import.log 2>&1
     test_debug "cat $GNUPGHOME/import.log"
     if (gpg --quick-random --version >/dev/null 2>&1) ; then
 	echo quick-random >> "$GNUPGHOME"/gpg.conf
diff --git a/test/emacs b/test/emacs
index 6f82b08..f91078e 100755
--- a/test/emacs
+++ b/test/emacs
@@ -2,7 +2,7 @@
 test_description="emacs interface"
 . test-lib.sh
 
-EXPECTED=../emacs.expected-output
+EXPECTED=$TEST_DIRECTORY/emacs.expected-output
 
 add_email_corpus
 
@@ -81,7 +81,7 @@ mkdir -p mail/sent/cur
 mkdir -p mail/sent/new
 mkdir -p mail/sent/tmp
 
-../smtp-dummy sent_message &
+$TEST_DIRECTORY/smtp-dummy sent_message &
 smtp_dummy_pid=$!
 test_emacs "(setq message-send-mail-function 'message-smtpmail-send-it) (setq smtpmail-smtp-server \"localhost\") (setq smtpmail-smtp-service \"25025\") (notmuch-hello) (notmuch-mua-mail) (message-goto-to) (insert \"user@example.com\nDate: Fri, 29 Mar 1974 10:00:00 -0000\") (message-goto-subject) (insert \"Testing message sent via SMTP\") (message-goto-body) (insert \"This is a test that messages are sent via SMTP\") (message-send-and-exit)" >/dev/null 2>&1
 wait ${smtp_dummy_pid}
diff --git a/test/symbol-hiding b/test/symbol-hiding
index bb55524..5d38fb7 100755
--- a/test/symbol-hiding
+++ b/test/symbol-hiding
@@ -12,13 +12,13 @@ test_description='exception symbol hiding'
 . ./test-lib.sh
 
 run_test(){
-    result=$(LD_LIBRARY_PATH=../../lib ./symbol-test 2>&1)
+    result=$(LD_LIBRARY_PATH=$TEST_DIRECTORY/../../lib ./symbol-test 2>&1)
 }
 
 output="A Xapian exception occurred opening database: Couldn't stat 'fakedb/.notmuch/xapian'
 caught No chert database found at path \`./nonexistant'"
 
-g++ -o symbol-test -I../../lib ../symbol-test.cc -L../../lib -lnotmuch -lxapian
+g++ -o symbol-test -I$TEST_DIRECTORY/../lib $TEST_DIRECTORY/symbol-test.cc -L$TEST_DIRECTORY/../lib -lnotmuch -lxapian
 mkdir -p fakedb/.notmuch
 test_expect_success 'running test' run_test
 test_begin_subtest 'checking output'
diff --git a/test/test-lib.sh b/test/test-lib.sh
index cc20f41..352be39 100755
--- a/test/test-lib.sh
+++ b/test/test-lib.sh
@@ -392,7 +392,7 @@ emacs_deliver_message ()
     shift 2
     # before we can send a message, we have to prepare the FCC maildir
     mkdir -p "$MAIL_DIR"/sent/{cur,new,tmp}
-    ../smtp-dummy sent_message &
+    $TEST_DIRECTORY/smtp-dummy sent_message &
     smtp_dummy_pid=$!
     test_emacs "(setq message-send-mail-function 'message-smtpmail-send-it) (setq smtpmail-smtp-server \"localhost\") (setq smtpmail-smtp-service \"25025\") (notmuch-hello) (notmuch-mua-mail) (message-goto-to) (insert \"test_suite@notmuchmail.org\nDate: 01 Jan 2000 12:00:00 -0000\") (message-goto-subject) (insert \"${subject}\") (message-goto-body) (insert \"${body}\") $@ (message-send-and-exit)" >/dev/null 2>&1
     wait ${smtp_dummy_pid}
@@ -408,12 +408,12 @@ emacs_deliver_message ()
 add_email_corpus ()
 {
     rm -rf ${MAIL_DIR}
-    if [ -d ../corpus.mail ]; then
-	cp -a ../corpus.mail ${MAIL_DIR}
+    if [ -d $TEST_DIRECTORY/corpus.mail ]; then
+	cp -a $TEST_DIRECTORY/corpus.mail ${MAIL_DIR}
     else
-	cp -a ../corpus ${MAIL_DIR}
+	cp -a $TEST_DIRECTORY/corpus ${MAIL_DIR}
 	notmuch new >/dev/null
-	cp -a ${MAIL_DIR} ../corpus.mail
+	cp -a ${MAIL_DIR} $TEST_DIRECTORY/corpus.mail
     fi
 }
 
@@ -860,8 +860,8 @@ fi
 # set-frame-width	80 columns (avoids crazy 10-column default of --batch)
 
 emacs \$BATCH --no-init-file --no-site-file \
-	--directory ../../emacs --load notmuch.el \
-	--directory .. --load test-lib.el \
+	--directory $TEST_DIRECTORY/../emacs --load notmuch.el \
+	--directory $TEST_DIRECTORY --load test-lib.el \
 	--eval "(defun notmuch-test-wait ()
 			(while (get-buffer-process (current-buffer))
 				(sleep-for 0.1)))" \
@@ -925,11 +925,11 @@ then
 		    test ! -d "$symlink_target" &&
 		    test "#!" != "$(head -c 2 < "$symlink_target")"
 		then
-			symlink_target=../valgrind.sh
+			symlink_target=$TEST_DIRECTORY/valgrind.sh
 		fi
 		case "$base" in
 		*.sh|*.perl)
-			symlink_target=../unprocessed-script
+			symlink_target=$TEST_DIRECTORY/unprocessed-script
 		esac
 		# create the link, or replace it if it is out of date
 		make_symlink "$symlink_target" "$GIT_VALGRIND/bin/$base" || exit
-- 
1.7.4.1

[PATCH v3] test:Improve test behaviors when --root is used

[Mark Anderson]

Change add_email_corpus, emacs_deliver_message and tests to use
$TEST_DIRECTORY instead of '..'.

This improves the behavior of the usage of --root=<dir>, as the
assumption of what '..' means will usually be incorrect.

Document -root option in README and update valgrind to work with
-root.
---

This patch actually fixes what Austin pointed out.  

 test/README        |    9 +++++++++
 test/basic         |   10 +++++-----
 test/crypto        |    2 +-
 test/emacs         |    4 ++--
 test/symbol-hiding |    4 ++--
 test/test-lib.sh   |   18 +++++++++---------
 6 files changed, 28 insertions(+), 19 deletions(-)

diff --git a/test/README b/test/README
index be75e0e..8fbf78d 100644
--- a/test/README
+++ b/test/README
@@ -41,6 +41,15 @@ The following command-line options are available when running tests:
 	As the names depend on the tests' file names, it is safe to
 	run the tests with this option in parallel.
 
+--root=<dir>::
+	This runs the testsuites specified under a seperate directory.
+	However, caution is advised, as not all tests are maintained
+	with this relocation in mind, so some tests may behave
+	differently.
+
+	Pointing this argument at a tmpfs filesystem can improve the
+	speed of the test suite for some users.
+
 When invoking the test suite via "make test" any of the above options
 can be specified as follows:
 
diff --git a/test/basic b/test/basic
index d6e8c10..33bf711 100755
--- a/test/basic
+++ b/test/basic
@@ -51,9 +51,9 @@ test_expect_code 2 'failure to clean up causes the test to fail' '
 
 # Ensure that all tests are being run
 test_begin_subtest 'Ensure that all available tests will be run by notmuch-test'
-eval $(sed -n -e '/^TESTS="$/,/^"$/p' notmuch-test ../notmuch-test)
+eval $(sed -n -e '/^TESTS="$/,/^"$/p' notmuch-test $TEST_DIRECTORY/notmuch-test)
 tests_in_suite=$(for i in $TESTS; do echo $i; done | sort)
-available=$(ls -1 ../ | \
+available=$(ls -1 $TEST_DIRECTORY/ | \
     sed -r -e "/^(aggregate-results.sh|Makefile|Makefile.local|notmuch-test)/d" \
 	   -e "/^(README|test-lib.sh|test-lib.el|test-results|tmp.*|valgrind|corpus*)/d" \
 	   -e "/^(emacs.expected-output|smtp-dummy|smtp-dummy.c|test-verbose|symbol-test.cc)/d" \
@@ -63,19 +63,19 @@ available=$(ls -1 ../ | \
 	   | sort)
 test_expect_equal "$tests_in_suite" "$available"
 
-EXPECTED=../test.expected-output
+EXPECTED=$TEST_DIRECTORY/test.expected-output
 suppress_diff_date() {
     sed -e 's/\(.*\-\-\- test-verbose\.4\.\expected\).*/\1/' \
 	-e 's/\(.*\+\+\+ test-verbose\.4\.\output\).*/\1/'
 }
 
 test_begin_subtest "Ensure that test output is suppressed unless the test fails"
-output=$(cd ..; ./test-verbose 2>&1 | suppress_diff_date)
+output=$(cd $TEST_DIRECTORY; ./test-verbose 2>&1 | suppress_diff_date)
 expected=$(cat $EXPECTED/test-verbose-no | suppress_diff_date)
 test_expect_equal "$output" "$expected"
 
 test_begin_subtest "Ensure that -v does not suppress test output"
-output=$(cd ..; ./test-verbose -v 2>&1 | suppress_diff_date)
+output=$(cd $TEST_DIRECTORY; ./test-verbose -v 2>&1 | suppress_diff_date)
 expected=$(cat $EXPECTED/test-verbose-yes | suppress_diff_date)
 # Do not include the results of test-verbose in totals
 rm $TEST_DIRECTORY/test-results/test-verbose-*
diff --git a/test/crypto b/test/crypto
index 01daffe..7eb3559 100755
--- a/test/crypto
+++ b/test/crypto
@@ -12,7 +12,7 @@ add_gnupg_home ()
     local output
     [ -d ${GNUPGHOME} ] && return
     mkdir -m 0700 "$GNUPGHOME"
-    gpg --no-tty --import <../gnupg-secret-key.asc >"$GNUPGHOME"/import.log 2>&1
+    gpg --no-tty --import <$TEST_DIRECTORY/gnupg-secret-key.asc >"$GNUPGHOME"/import.log 2>&1
     test_debug "cat $GNUPGHOME/import.log"
     if (gpg --quick-random --version >/dev/null 2>&1) ; then
 	echo quick-random >> "$GNUPGHOME"/gpg.conf
diff --git a/test/emacs b/test/emacs
index 6f82b08..f91078e 100755
--- a/test/emacs
+++ b/test/emacs
@@ -2,7 +2,7 @@
 test_description="emacs interface"
 . test-lib.sh
 
-EXPECTED=../emacs.expected-output
+EXPECTED=$TEST_DIRECTORY/emacs.expected-output
 
 add_email_corpus
 
@@ -81,7 +81,7 @@ mkdir -p mail/sent/cur
 mkdir -p mail/sent/new
 mkdir -p mail/sent/tmp
 
-../smtp-dummy sent_message &
+$TEST_DIRECTORY/smtp-dummy sent_message &
 smtp_dummy_pid=$!
 test_emacs "(setq message-send-mail-function 'message-smtpmail-send-it) (setq smtpmail-smtp-server \"localhost\") (setq smtpmail-smtp-service \"25025\") (notmuch-hello) (notmuch-mua-mail) (message-goto-to) (insert \"user@example.com\nDate: Fri, 29 Mar 1974 10:00:00 -0000\") (message-goto-subject) (insert \"Testing message sent via SMTP\") (message-goto-body) (insert \"This is a test that messages are sent via SMTP\") (message-send-and-exit)" >/dev/null 2>&1
 wait ${smtp_dummy_pid}
diff --git a/test/symbol-hiding b/test/symbol-hiding
index bb55524..d0b31ae 100755
--- a/test/symbol-hiding
+++ b/test/symbol-hiding
@@ -12,13 +12,13 @@ test_description='exception symbol hiding'
 . ./test-lib.sh
 
 run_test(){
-    result=$(LD_LIBRARY_PATH=../../lib ./symbol-test 2>&1)
+    result=$(LD_LIBRARY_PATH=$TEST_DIRECTORY/../lib ./symbol-test 2>&1)
 }
 
 output="A Xapian exception occurred opening database: Couldn't stat 'fakedb/.notmuch/xapian'
 caught No chert database found at path \`./nonexistant'"
 
-g++ -o symbol-test -I../../lib ../symbol-test.cc -L../../lib -lnotmuch -lxapian
+g++ -o symbol-test -I$TEST_DIRECTORY/../lib $TEST_DIRECTORY/symbol-test.cc -L$TEST_DIRECTORY/../lib -lnotmuch -lxapian
 mkdir -p fakedb/.notmuch
 test_expect_success 'running test' run_test
 test_begin_subtest 'checking output'
diff --git a/test/test-lib.sh b/test/test-lib.sh
index cc20f41..352be39 100755
--- a/test/test-lib.sh
+++ b/test/test-lib.sh
@@ -392,7 +392,7 @@ emacs_deliver_message ()
     shift 2
     # before we can send a message, we have to prepare the FCC maildir
     mkdir -p "$MAIL_DIR"/sent/{cur,new,tmp}
-    ../smtp-dummy sent_message &
+    $TEST_DIRECTORY/smtp-dummy sent_message &
     smtp_dummy_pid=$!
     test_emacs "(setq message-send-mail-function 'message-smtpmail-send-it) (setq smtpmail-smtp-server \"localhost\") (setq smtpmail-smtp-service \"25025\") (notmuch-hello) (notmuch-mua-mail) (message-goto-to) (insert \"test_suite@notmuchmail.org\nDate: 01 Jan 2000 12:00:00 -0000\") (message-goto-subject) (insert \"${subject}\") (message-goto-body) (insert \"${body}\") $@ (message-send-and-exit)" >/dev/null 2>&1
     wait ${smtp_dummy_pid}
@@ -408,12 +408,12 @@ emacs_deliver_message ()
 add_email_corpus ()
 {
     rm -rf ${MAIL_DIR}
-    if [ -d ../corpus.mail ]; then
-	cp -a ../corpus.mail ${MAIL_DIR}
+    if [ -d $TEST_DIRECTORY/corpus.mail ]; then
+	cp -a $TEST_DIRECTORY/corpus.mail ${MAIL_DIR}
     else
-	cp -a ../corpus ${MAIL_DIR}
+	cp -a $TEST_DIRECTORY/corpus ${MAIL_DIR}
 	notmuch new >/dev/null
-	cp -a ${MAIL_DIR} ../corpus.mail
+	cp -a ${MAIL_DIR} $TEST_DIRECTORY/corpus.mail
     fi
 }
 
@@ -860,8 +860,8 @@ fi
 # set-frame-width	80 columns (avoids crazy 10-column default of --batch)
 
 emacs \$BATCH --no-init-file --no-site-file \
-	--directory ../../emacs --load notmuch.el \
-	--directory .. --load test-lib.el \
+	--directory $TEST_DIRECTORY/../emacs --load notmuch.el \
+	--directory $TEST_DIRECTORY --load test-lib.el \
 	--eval "(defun notmuch-test-wait ()
 			(while (get-buffer-process (current-buffer))
 				(sleep-for 0.1)))" \
@@ -925,11 +925,11 @@ then
 		    test ! -d "$symlink_target" &&
 		    test "#!" != "$(head -c 2 < "$symlink_target")"
 		then
-			symlink_target=../valgrind.sh
+			symlink_target=$TEST_DIRECTORY/valgrind.sh
 		fi
 		case "$base" in
 		*.sh|*.perl)
-			symlink_target=../unprocessed-script
+			symlink_target=$TEST_DIRECTORY/unprocessed-script
 		esac
 		# create the link, or replace it if it is out of date
 		make_symlink "$symlink_target" "$GIT_VALGRIND/bin/$base" || exit
-- 
1.7.4.1

Re: [PATCH v3] test:Improve test behaviors when --root is used

[Carl Worth]

[-- Attachment #1: Type: text/plain, Size: 478 bytes --]

On Tue, 28 Jun 2011 16:11:32 -0600, Mark Anderson <ma.skies@gmail.com> wrote:
> Change add_email_corpus, emacs_deliver_message and tests to use
> $TEST_DIRECTORY instead of '..'.
...
> Document -root option in README and update valgrind to work with
> -root.

Thanks for the features, Mark. These should all be quite handy.

> This patch actually fixes what Austin pointed out.  

And thanks for the eagle-eyed review, Austin!

This is pushed out now.

-Carl
<

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: [PATCH v3] test:Improve test behaviors when --root is used

[Pieter Praet]

On Tue, 28 Jun 2011 16:03:00 -0700, Carl Worth <cworth@cworth.org> wrote:
Non-text part: multipart/mixed
Non-text part: multipart/signed
> On Tue, 28 Jun 2011 16:11:32 -0600, Mark Anderson <ma.skies@gmail.com> wrote:
> > Change add_email_corpus, emacs_deliver_message and tests to use
> > $TEST_DIRECTORY instead of '..'.
> ...
> > Document -root option in README and update valgrind to work with
> > -root.
> 
> Thanks for the features, Mark. These should all be quite handy.

Very useful indeed, great improvement Mark!

> > This patch actually fixes what Austin pointed out.  
> 
> And thanks for the eagle-eyed review, Austin!
> 
> This is pushed out now.

Thanks Carl!

However, it appears that (while resolving the merge conflicts?) you've
accidentally introduced an [*] extra change in the Emacs tests, which
causes

 PASS   Message with .. in Message-Id:
 PASS   Sending a message via (fake) SMTP

to become

 PASS   Message with .. in Message-Id:
 PASS   Message with .. in Message-Id:

I'll be following up with a patch.


[*] Two extra changes actually, but the second one was merely the
insertion of a line break.

> -Carl
> <
Non-text part: application/pgp-signature
> _______________________________________________
> notmuch mailing list
> notmuch@notmuchmail.org
> http://notmuchmail.org/mailman/listinfo/notmuch

Peace

-- 
Pieter

[PATCH] test: revert non-intentional changes introduced in eb4cf465

[Pieter Praet]

eb4cf465 introduces changes which weren't part of the submitted
patch (id:"87liwlip2j.fsf@gmail.com"), presumably made during
resolving merge conflicts.

The first one causes the title of a test to be printed a second time,
in place of the correct title:

  PASS   Message with .. in Message-Id:
  PASS   Message with .. in Message-Id:

instead of:

  PASS   Message with .. in Message-Id:
  PASS   Sending a message via (fake) SMTP

The second one is simply the insertion of a line break, so no harm there.

This commit reverts both changes, as they were clearly accidental.

Signed-off-by: Pieter Praet <pieter@praet.org>
---
 test/emacs |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/test/emacs b/test/emacs
index e59de47..53f455a 100755
--- a/test/emacs
+++ b/test/emacs
@@ -123,13 +123,13 @@ test_emacs '(notmuch-search "id:\"123..456@example\"")
 output=$(notmuch search 'id:"123..456@example"' | notmuch_search_sanitize)
 test_expect_equal "$output" "thread:XXX   2001-01-05 [1/1] Notmuch Test Suite; Message with .. in Message-Id (inbox search-add show-add)"
 
+test_begin_subtest "Sending a message via (fake) SMTP"
 emacs_deliver_message \
     'Testing message sent via SMTP' \
     'This is a test that messages are sent via SMTP' \
     '(message-goto-to)
      (kill-whole-line)
      (insert "To: user@example.com\n")'
-
 sed \
     -e s',^User-Agent: Notmuch/.* Emacs/.*,User-Agent: Notmuch/XXX Emacs/XXX,' \
     -e s',^Message-ID: <.*>$,Message-ID: <XXX>,' < sent_message >OUTPUT
-- 
1.7.4.1

Re: [PATCH] test: revert non-intentional changes introduced in eb4cf465

[Carl Worth]

[-- Attachment #1: Type: text/plain, Size: 463 bytes --]

On Wed, 29 Jun 2011 22:23:47 +0200, Pieter Praet <pieter@praet.org> wrote:
> eb4cf465 introduces changes which weren't part of the submitted
> patch (id:"87liwlip2j.fsf@gmail.com"), presumably made during
> resolving merge conflicts.

Oops, you caught me.

> This commit reverts both changes, as they were clearly accidental.

Thanks. This is merged (automatically, so hopefully without errors!) and
pushed out.

-Carl

-- 
carl.d.worth@intel.com

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: [PATCH] test:Improve test behaviors when --root is used

[Pieter Praet]

On Mon, 27 Jun 2011 16:50:47 -0400, Austin Clements <amdragon@mit.edu> wrote:
> This looks great (modulo one bug, below).  I've wanted to run the
> tests on tmpfs before, but was too lazy to actually fix the tests.
> 
> Given how easy it is to accidentally use "..", I wonder if there's a
> way to force people to use $TEST_DIRECTORY?

Something like this? Mind you, my grep-fu is non-existent.

#+BEGIN_SRC sh
  dotdot=$(rgrep -nH -e "\.\.\/" . | grep -v -e "\$TEST_DIRECTORY")
  if [ -n "${dotdot}" ] ; then
    echo -e "Don't dot dot me! Use \$TEST_DIRECTORY as prefix instead.\n\nOffending lines:\n ${dotdot}"
  fi
#+END_SRC

> On Mon, Jun 27, 2011 at 12:09 PM, Mark Anderson <ma.skies@gmail.com> wrote:
> > --- a/test/symbol-hiding
> > +++ b/test/symbol-hiding
> > @@ -12,13 +12,13 @@ test_description='exception symbol hiding'
> >  . ./test-lib.sh
> >
> >  run_test(){
> > -    result=$(LD_LIBRARY_PATH=../../lib ./symbol-test 2>&1)
> > +    result=$(LD_LIBRARY_PATH=$TEST_DIRECTORY/../../lib ./symbol-test 2>&1)
> 
> Did you mean $TEST_DIRECTORY/../lib?
> _______________________________________________
> notmuch mailing list
> notmuch@notmuchmail.org
> http://notmuchmail.org/mailman/listinfo/notmuch

Peace

-- 
Pieter