unofficial mirror of notmuch@notmuchmail.org
 help / color / mirror / code / Atom feed
* Support testing notmuch as installed
@ 2023-01-04 12:41 David Bremner
  2023-01-04 12:41 ` [PATCH 1/4] test: mark some tests as broken when run as root David Bremner
                   ` (3 more replies)
  0 siblings, 4 replies; 7+ messages in thread
From: David Bremner @ 2023-01-04 12:41 UTC (permalink / raw)
  To: notmuch

This is mainly the same as
id:20211025145753.3031094-1-david@tethera.net, with some adjustments
for changes in notmuch in the last year.

The main place it could be improved is having less tests disabled in
installed mode. That seems like something that someone(TM) could do
incrementally. There are more than 1950 tests that run in installed
mode, so it's a big step up from the status quo.

The first two patches in the series are simple improvements to the
test suite, that could be applied independently.


^ permalink raw reply	[flat|nested] 7+ messages in thread

* [PATCH 1/4] test: mark some tests as broken when run as root.
  2023-01-04 12:41 Support testing notmuch as installed David Bremner
@ 2023-01-04 12:41 ` David Bremner
  2023-01-05 10:12   ` Tomi Ollila
  2023-01-04 12:41 ` [PATCH 2/4] test: check for empty/missing files in test_expect_equal_message_body David Bremner
                   ` (2 subsequent siblings)
  3 siblings, 1 reply; 7+ messages in thread
From: David Bremner @ 2023-01-04 12:41 UTC (permalink / raw)
  To: notmuch

File permission errors e.g., are hard to trigger as root.
---
 test/T050-new.sh     | 1 +
 test/T150-tagging.sh | 1 +
 test/test-lib.sh     | 6 ++++++
 3 files changed, 8 insertions(+)

diff --git a/test/T050-new.sh b/test/T050-new.sh
index cb67889c..240c1810 100755
--- a/test/T050-new.sh
+++ b/test/T050-new.sh
@@ -384,6 +384,7 @@ EOF
 test_expect_equal_file EXPECTED OUTPUT
 
 test_begin_subtest "Xapian exception: read only files"
+test_subtest_broken_for_root
 chmod u-w ${MAIL_DIR}/.notmuch/xapian/*.*
 output=$(NOTMUCH_NEW --debug 2>&1 | sed 's/: .*$//' )
 chmod u+w ${MAIL_DIR}/.notmuch/xapian/*.*
diff --git a/test/T150-tagging.sh b/test/T150-tagging.sh
index 1a2fd77e..ac3f2539 100755
--- a/test/T150-tagging.sh
+++ b/test/T150-tagging.sh
@@ -320,6 +320,7 @@ test_begin_subtest "Tag name beginning with -"
 test_expect_code 1 'notmuch tag +- One'
 
 test_begin_subtest "Xapian exception: read only files"
+test_subtest_broken_for_root
 chmod u-w ${MAIL_DIR}/.notmuch/xapian/*.*
 output=$(notmuch tag +something '*' 2>&1 | sed 's/: .*$//' )
 chmod u+w ${MAIL_DIR}/.notmuch/xapian/*.*
diff --git a/test/test-lib.sh b/test/test-lib.sh
index eec5c5b4..c2b8cd8d 100644
--- a/test/test-lib.sh
+++ b/test/test-lib.sh
@@ -740,6 +740,12 @@ test_subtest_known_broken () {
 	test_subtest_known_broken_=t
 }
 
+test_subtest_broken_for_root () {
+   if [ "$(id -u)" = "0" ]; then
+	test_subtest_known_broken_=t
+    fi
+}
+
 test_expect_success () {
 	exec 1>&6 2>&7		# Restore stdout and stderr
 	if [ -z "$inside_subtest" ]; then
-- 
2.39.0

^ permalink raw reply related	[flat|nested] 7+ messages in thread

* [PATCH 2/4] test: check for empty/missing files in test_expect_equal_message_body
  2023-01-04 12:41 Support testing notmuch as installed David Bremner
  2023-01-04 12:41 ` [PATCH 1/4] test: mark some tests as broken when run as root David Bremner
@ 2023-01-04 12:41 ` David Bremner
  2023-01-04 12:41 ` [PATCH 3/4] test: support testing notmuch as installed David Bremner
  2023-01-04 12:41 ` [PATCH 4/4] debian: add autopkgtests David Bremner
  3 siblings, 0 replies; 7+ messages in thread
From: David Bremner @ 2023-01-04 12:41 UTC (permalink / raw)
  To: notmuch

Messages can have empty bodies, but empty files are not messages.
---
 test/test-lib.sh | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/test/test-lib.sh b/test/test-lib.sh
index c2b8cd8d..a647790a 100644
--- a/test/test-lib.sh
+++ b/test/test-lib.sh
@@ -388,6 +388,14 @@ test_expect_equal_message_body () {
     test "$#" = 2 ||
 	error "bug in the test script: not 2 parameters to test_expect_equal_file"
 
+    for file in "$1" "$2"; do
+	if [ ! -s "$file" ]; then
+	    test_failure_ "Missing or zero length file: $file"
+	    inside_subtest=
+	    return $?
+	fi
+    done
+
     expected=$(sed '1,/^$/d' "$1")
     output=$(sed '1,/^$/d' "$2")
     test_expect_equal "$expected" "$output"
-- 
2.39.0

^ permalink raw reply related	[flat|nested] 7+ messages in thread

* [PATCH 3/4] test: support testing notmuch as installed
  2023-01-04 12:41 Support testing notmuch as installed David Bremner
  2023-01-04 12:41 ` [PATCH 1/4] test: mark some tests as broken when run as root David Bremner
  2023-01-04 12:41 ` [PATCH 2/4] test: check for empty/missing files in test_expect_equal_message_body David Bremner
@ 2023-01-04 12:41 ` David Bremner
  2023-01-04 12:41 ` [PATCH 4/4] debian: add autopkgtests David Bremner
  3 siblings, 0 replies; 7+ messages in thread
From: David Bremner @ 2023-01-04 12:41 UTC (permalink / raw)
  To: notmuch

We put some effort into testing the built copy rather than some
installed copy. On the other hand for people like packagers, testing
the installed copy is also of interest.

When NOTMUCH_TEST_INSTALLED is set to a nonempty value, tests do not
require a built notmuch tree or running configure.

Some of the tests marked as broken when running against installed
notmuch are probably fixable.

One questionable aspect is the file "defaults.sh" which is used to
fake a certain set of minimal responses to configure.
---
 test/T000-basic.sh             |  1 +
 test/T240-dump-restore.sh      |  2 ++
 test/T310-emacs.sh             |  2 ++
 test/T350-crypto.sh            |  1 +
 test/T360-symbol-hiding.sh     |  4 ++++
 test/T390-python.sh            |  4 ++++
 test/T391-python-cffi.sh       |  5 ++++-
 test/T395-ruby.sh              |  8 ++++++--
 test/T410-argument-parsing.sh  |  4 ++++
 test/T480-hex-escaping.sh      |  4 ++++
 test/T490-parse-time-string.sh |  4 ++++
 test/T550-db-features.sh       |  4 ++++
 test/T566-lib-message.sh       |  4 ++++
 test/T592-thread-breakage.sh   |  4 ++++
 test/T710-message-id.sh        |  4 ++++
 test/T800-asan.sh              |  1 +
 test/T850-git.sh               |  6 ++++++
 test/defaults.sh               |  5 +++++
 test/export-dirs.sh            |  2 +-
 test/notmuch-test              | 12 ++++++++++--
 test/test-lib-common.sh        | 16 +++++++++++++---
 test/test-lib-emacs.sh         | 14 +++++++++++---
 test/test-lib.sh               | 27 +++++++++++++++++++++------
 23 files changed, 120 insertions(+), 18 deletions(-)
 create mode 100644 test/defaults.sh

diff --git a/test/T000-basic.sh b/test/T000-basic.sh
index a2f4d93f..642f918d 100755
--- a/test/T000-basic.sh
+++ b/test/T000-basic.sh
@@ -66,6 +66,7 @@ test_begin_subtest 'NOTMUCH_CONFIG is set and points to an existing file'
 test_expect_success 'test -f "${NOTMUCH_CONFIG}"'
 
 test_begin_subtest 'PATH is set to build directory'
+test_subtest_broken_for_installed
 test_expect_equal \
     "$(dirname ${TEST_DIRECTORY})" \
     "$(echo $PATH|cut -f1 -d: | sed -e 's,/test/valgrind/bin$,,')"
diff --git a/test/T240-dump-restore.sh b/test/T240-dump-restore.sh
index a86f0fb7..8848e06c 100755
--- a/test/T240-dump-restore.sh
+++ b/test/T240-dump-restore.sh
@@ -139,6 +139,7 @@ notmuch dump --output=dump-outfile-dash-inbox.actual -- from:cworth
 test_expect_equal_file dump-cworth.expected dump-outfile-dash-inbox.actual
 
 test_begin_subtest "Check for a safe set of message-ids"
+test_subtest_broken_for_installed
 notmuch search --output=messages from:cworth | sed s/^id:// > EXPECTED
 notmuch search --output=messages from:cworth | sed s/^id:// |\
 	$TEST_DIRECTORY/hex-xcode --direction=encode > OUTPUT
@@ -246,6 +247,7 @@ notmuch dump --format=batch-tag > OUTPUT.$test_count
 test_expect_equal_file EXPECTED.$test_count OUTPUT.$test_count
 
 test_begin_subtest 'format=batch-tag, checking encoded output'
+test_subtest_broken_for_installed
 NOTMUCH_DUMP_TAGS --format=batch-tag -- from:cworth |\
 	 awk "{ print \"+$enc1 +$enc2 +$enc3 -- \" \$5 }" > EXPECTED.$test_count
 NOTMUCH_DUMP_TAGS --format=batch-tag -- from:cworth > OUTPUT.$test_count
diff --git a/test/T310-emacs.sh b/test/T310-emacs.sh
index 9d0df187..d3aa2e7d 100755
--- a/test/T310-emacs.sh
+++ b/test/T310-emacs.sh
@@ -258,6 +258,7 @@ EOF
 test_expect_equal_file EXPECTED OUTPUT
 
 test_begin_subtest "Verify that sent messages are saved/searchable (via FCC)"
+test_subtest_broken_for_installed
 notmuch new > /dev/null
 output=$(notmuch search 'subject:"testing message sent via SMTP"' | notmuch_search_sanitize)
 test_expect_equal "$output" "thread:XXX   2000-01-01 [1/1] Notmuch Test Suite; Testing message sent via SMTP (inbox)"
@@ -350,6 +351,7 @@ EOF
 test_expect_equal_file EXPECTED OUTPUT
 
 test_begin_subtest "Reply within emacs"
+test_subtest_broken_for_installed
 test_emacs '(let ((message-hidden-headers ''()))
 	    (notmuch-search "subject:\"testing message sent via SMTP\"")
 	    (notmuch-test-wait)
diff --git a/test/T350-crypto.sh b/test/T350-crypto.sh
index 721cbfdd..27c0e86d 100755
--- a/test/T350-crypto.sh
+++ b/test/T350-crypto.sh
@@ -30,6 +30,7 @@ msg_file=$(notmuch search --output=files subject:signed-message-sent-via-SMTP)
 test_expect_equal_message_body sent_message "$msg_file"
 
 test_begin_subtest "signed part content-type indexing"
+test_subtest_broken_for_installed
 notmuch search mimetype:multipart/signed and mimetype:application/pgp-signature | notmuch_search_sanitize > OUTPUT
 cat <<EOF >EXPECTED
 thread:XXX   2000-01-01 [1/1] Notmuch Test Suite; test signed message 001 (inbox signed)
diff --git a/test/T360-symbol-hiding.sh b/test/T360-symbol-hiding.sh
index 642457bf..5bf8da14 100755
--- a/test/T360-symbol-hiding.sh
+++ b/test/T360-symbol-hiding.sh
@@ -11,6 +11,10 @@ test_description='exception symbol hiding'
 
 . $(dirname "$0")/test-lib.sh || exit 1
 
+if [[ -n ${NOTMUCH_TEST_INSTALLED} ]]; then
+    test_done
+fi
+
 test_begin_subtest 'running test' run_test
 mkdir -p ${PWD}/fakedb/.notmuch
 $TEST_DIRECTORY/symbol-test ${PWD}/fakedb ${PWD}/nonexistent 2>&1 \
diff --git a/test/T390-python.sh b/test/T390-python.sh
index 9f71ce3c..710d2553 100755
--- a/test/T390-python.sh
+++ b/test/T390-python.sh
@@ -4,6 +4,10 @@ test_description="python bindings"
 
 test_require_external_prereq ${NOTMUCH_PYTHON}
 
+if [[ -n ${NOTMUCH_TEST_INSTALLED} ]]; then
+    test_done
+fi
+
 add_email_corpus
 add_gnupg_home
 
diff --git a/test/T391-python-cffi.sh b/test/T391-python-cffi.sh
index 30872af0..8f27148c 100755
--- a/test/T391-python-cffi.sh
+++ b/test/T391-python-cffi.sh
@@ -2,10 +2,13 @@
 test_description="python bindings (pytest)"
 . $(dirname "$0")/test-lib.sh || exit 1
 
-if [ $NOTMUCH_HAVE_PYTHON3_CFFI -eq 0 -o $NOTMUCH_HAVE_PYTHON3_PYTEST -eq 0 ]; then
+if [[ -n "${NOTMUCH_TEST_INSTALLED}" ]]; then
     test_done
 fi
 
+if [ $NOTMUCH_HAVE_PYTHON3_CFFI -eq 0 -o $NOTMUCH_HAVE_PYTHON3_PYTEST -eq 0 ]; then
+    test_done
+fi
 
 test_begin_subtest "python cffi tests (NOTMUCH_CONFIG set)"
 pytest_dir=$NOTMUCH_BUILDDIR/bindings/python-cffi/build/stage
diff --git a/test/T395-ruby.sh b/test/T395-ruby.sh
index e828efed..6a5cf615 100755
--- a/test/T395-ruby.sh
+++ b/test/T395-ruby.sh
@@ -2,7 +2,7 @@
 test_description="ruby bindings"
 . $(dirname "$0")/test-lib.sh || exit 1
 
-if [ "${NOTMUCH_HAVE_RUBY_DEV}" = "0" ]; then
+if [[ -z "${NOTMUCH_TEST_INSTALLED}" && "${NOTMUCH_HAVE_RUBY_DEV}" = "0" ]]; then
     test_subtest_missing_external_prereq_["ruby development files"]=t
 fi
 
@@ -15,7 +15,11 @@ test_ruby() {
 	db = Notmuch::Database.new('$MAIL_DIR')
 	EOF
 	cat
-    ) | $NOTMUCH_RUBY -I "$NOTMUCH_BUILDDIR/bindings/ruby"> OUTPUT
+    ) | if [[ -n "${NOTMUCH_TEST_INSTALLED}" ]]; then
+	ruby
+    else
+	$NOTMUCH_RUBY -I "$NOTMUCH_BUILDDIR/bindings/ruby"
+    fi> OUTPUT
     test_expect_equal_file EXPECTED OUTPUT
 }
 
diff --git a/test/T410-argument-parsing.sh b/test/T410-argument-parsing.sh
index d9aa8e2d..5af12916 100755
--- a/test/T410-argument-parsing.sh
+++ b/test/T410-argument-parsing.sh
@@ -2,6 +2,10 @@
 test_description="argument parsing"
 . $(dirname "$0")/test-lib.sh || exit 1
 
+if [[ -n ${NOTMUCH_TEST_INSTALLED} ]]; then
+    test_done
+fi
+
 test_begin_subtest "sanity check"
 $TEST_DIRECTORY/arg-test pos1 --keyword=one --boolean --string=foo pos2 --int=7 --flag=one --flag=three > OUTPUT
 cat <<EOF > EXPECTED
diff --git a/test/T480-hex-escaping.sh b/test/T480-hex-escaping.sh
index b2eb80b9..744d5042 100755
--- a/test/T480-hex-escaping.sh
+++ b/test/T480-hex-escaping.sh
@@ -2,6 +2,10 @@
 test_description="hex encoding and decoding"
 . $(dirname "$0")/test-lib.sh || exit 1
 
+if [[ -n ${NOTMUCH_TEST_INSTALLED} ]]; then
+    test_done
+fi
+
 test_begin_subtest "round trip"
 find $NOTMUCH_SRCDIR/test/corpora/default -type f -print | sort | xargs cat > EXPECTED
 $TEST_DIRECTORY/hex-xcode --direction=encode < EXPECTED | $TEST_DIRECTORY/hex-xcode --direction=decode > OUTPUT
diff --git a/test/T490-parse-time-string.sh b/test/T490-parse-time-string.sh
index f89755ed..fa7d547c 100755
--- a/test/T490-parse-time-string.sh
+++ b/test/T490-parse-time-string.sh
@@ -2,6 +2,10 @@
 test_description="date/time parser module"
 . $(dirname "$0")/test-lib.sh || exit 1
 
+if [[ -n ${NOTMUCH_TEST_INSTALLED} ]]; then
+    test_done
+fi
+
 # Sanity/smoke tests for the date/time parser independent of notmuch
 
 _date () {
diff --git a/test/T550-db-features.sh b/test/T550-db-features.sh
index 9d5a9e70..c3ece78b 100755
--- a/test/T550-db-features.sh
+++ b/test/T550-db-features.sh
@@ -3,6 +3,10 @@ test_description="database version and feature compatibility"
 
 . $(dirname "$0")/test-lib.sh || exit 1
 
+if [[ -n ${NOTMUCH_TEST_INSTALLED} ]]; then
+    test_done
+fi
+
 test_begin_subtest "future database versions abort open"
 ${TEST_DIRECTORY}/make-db-version ${MAIL_DIR} 9999 ""
 output=$(notmuch search x 2>&1 | sed 's/\(database at\) .*/\1 FILENAME/')
diff --git a/test/T566-lib-message.sh b/test/T566-lib-message.sh
index 7f0e8eb0..d8922fa4 100755
--- a/test/T566-lib-message.sh
+++ b/test/T566-lib-message.sh
@@ -3,6 +3,10 @@ test_description="API tests for notmuch_message_*"
 
 . $(dirname "$0")/test-lib.sh || exit 1
 
+if [[ -n ${NOTMUCH_TEST_INSTALLED} ]]; then
+    test_done
+fi
+
 add_email_corpus
 
 test_begin_subtest "building database"
diff --git a/test/T592-thread-breakage.sh b/test/T592-thread-breakage.sh
index 92a70e3e..fd9e628b 100755
--- a/test/T592-thread-breakage.sh
+++ b/test/T592-thread-breakage.sh
@@ -21,6 +21,10 @@ test_description='thread breakage during reindexing'
 
 . $(dirname "$0")/test-lib.sh || exit 1
 
+if [[ -n ${NOTMUCH_TEST_INSTALLED} ]]; then
+    test_done
+fi
+
 message_a () {
     mkdir -p ${MAIL_DIR}/cur
     cat > ${MAIL_DIR}/cur/a <<EOF
diff --git a/test/T710-message-id.sh b/test/T710-message-id.sh
index 5129d84c..340ec426 100755
--- a/test/T710-message-id.sh
+++ b/test/T710-message-id.sh
@@ -3,6 +3,10 @@ test_description="message id parsing"
 
 . $(dirname "$0")/test-lib.sh || exit 1
 
+if [[ -n "${NOTMUCH_TEST_INSTALLED}" ]]; then
+    test_done
+fi
+
 test_begin_subtest "good message ids"
 ${TEST_DIRECTORY}/message-id-parse <<EOF >OUTPUT
 <018b1a8f2d1df62e804ce88b65401304832dfbbf.1346614915.git.jani@nikula.org>
diff --git a/test/T800-asan.sh b/test/T800-asan.sh
index 8607732e..1ddff0fe 100755
--- a/test/T800-asan.sh
+++ b/test/T800-asan.sh
@@ -12,6 +12,7 @@ add_email_corpus
 TEST_CFLAGS="-fsanitize=address"
 
 test_begin_subtest "open and destroy"
+test_subtest_broken_for_installed
 test_C ${MAIL_DIR} ${NOTMUCH_CONFIG} <<EOF
 #include <notmuch.h>
 #include <stdio.h>
diff --git a/test/T850-git.sh b/test/T850-git.sh
index 55cec78a..cb7581e7 100755
--- a/test/T850-git.sh
+++ b/test/T850-git.sh
@@ -233,6 +233,7 @@ EOF
 test_expect_equal_file EXPECTED OUTPUT
 
 test_begin_subtest "invoke as nmbug sets defaults"
+test_subtest_broken_for_installed
 "$NOTMUCH_BUILDDIR"/nmbug -ldebug status |& grep '^\(prefix\|repository\)' | notmuch_dir_sanitize > OUTPUT
 cat <<EOF > EXPECTED
 prefix = notmuch::
@@ -241,6 +242,7 @@ EOF
 test_expect_equal_file EXPECTED OUTPUT
 
 test_begin_subtest "env variable NOTMUCH_GIT_DIR works when invoked as nmbug"
+test_subtest_broken_for_installed
 NOTMUCH_GIT_DIR=`pwd`/foo "$NOTMUCH_BUILDDIR"/nmbug -ldebug status |& grep '^repository' | notmuch_dir_sanitize > OUTPUT
 cat <<EOF > EXPECTED
 repository = CWD/foo
@@ -256,6 +258,7 @@ test_expect_equal_file EXPECTED OUTPUT
 
 
 test_begin_subtest "env variable NOTMUCH_GIT_DIR overrides config when invoked as 'nmbug'"
+test_subtest_broken_for_installed
 notmuch config set git.path `pwd`/bar
 NOTMUCH_GIT_DIR=`pwd`/remote.git  "$NOTMUCH_BUILDDIR"/nmbug -ldebug status |& grep '^repository' | notmuch_dir_sanitize > OUTPUT
 notmuch config set git.path
@@ -274,6 +277,7 @@ EOF
 test_expect_equal_file EXPECTED OUTPUT
 
 test_begin_subtest "env variable NOTMUCH_GIT_PREFIX works when invoked as 'nmbug'"
+test_subtest_broken_for_installed
 NOTMUCH_GIT_PREFIX=env:: "$NOTMUCH_BUILDDIR"/nmbug -ldebug status |& grep '^prefix' | notmuch_dir_sanitize > OUTPUT
 cat <<EOF > EXPECTED
 prefix = env::
@@ -281,6 +285,7 @@ EOF
 test_expect_equal_file EXPECTED OUTPUT
 
 test_begin_subtest "env variable NOTMUCH_GIT_PREFIX works when invoked as nmbug"
+test_subtest_broken_for_installed
 NOTMUCH_GIT_PREFIX=foo:: "$NOTMUCH_BUILDDIR"/nmbug -ldebug status |& grep '^prefix' | notmuch_dir_sanitize > OUTPUT
 cat <<EOF > EXPECTED
 prefix = foo::
@@ -288,6 +293,7 @@ EOF
 test_expect_equal_file EXPECTED OUTPUT
 
 test_begin_subtest "env variable NOTMUCH_GIT_PREFIX overrides config when invoked as 'nmbug'"
+test_subtest_broken_for_installed
 notmuch config set git.tag_prefix config::
 NOTMUCH_GIT_PREFIX=env:: "$NOTMUCH_BUILDDIR"/nmbug -ldebug status |& grep '^prefix' | notmuch_dir_sanitize > OUTPUT
 notmuch config set git.path
diff --git a/test/defaults.sh b/test/defaults.sh
new file mode 100644
index 00000000..6327e43b
--- /dev/null
+++ b/test/defaults.sh
@@ -0,0 +1,5 @@
+# Guesses at some defaults to avoid needing to run configure. Override as needed.
+NOTMUCH_PYTHON=${NOTMUCH_PYTHON-python3}
+NOTMUCH_HAVE_MAN=${NOTMUCH_HAVE_MAN-1}
+NOTMUCH_HAVE_SFSEXP=${NOTMUCH_HAVE_SFSEXP-1}
+NOTMUCH_HAVE_ASAN=${NOTMUCH_HAVE_ASAN-0}
diff --git a/test/export-dirs.sh b/test/export-dirs.sh
index 0a048e1f..3109da88 100644
--- a/test/export-dirs.sh
+++ b/test/export-dirs.sh
@@ -24,7 +24,7 @@ find_builddir () {
 if [[ -z "${NOTMUCH_BUILDDIR}" ]]; then
 	export NOTMUCH_BUILDDIR="$(find_builddir "$(pwd)")"
 
-	if [[ -z "${NOTMUCH_BUILDDIR}" ]]; then
+	if [[ -z "${NOTMUCH_BUILDDIR}"  &&  -z "${NOTMUCH_TEST_INSTALLED}" ]]; then
 		echo "Run tests in a subdir of built notmuch tree." >&2
 		exit 1
 	fi
diff --git a/test/notmuch-test b/test/notmuch-test
index 14747bdb..5d27e4d1 100755
--- a/test/notmuch-test
+++ b/test/notmuch-test
@@ -20,6 +20,14 @@ fi
 
 set -eu
 
+# Where to run the tests
+# XXX FIXME this code is duplicated with test-lib.sh
+if [[ -n "${NOTMUCH_BUILDDIR}" ]]; then
+    TEST_DIRECTORY=$NOTMUCH_BUILDDIR/test
+else
+    TEST_DIRECTORY=$NOTMUCH_SRCDIR/test
+fi
+
 TESTS=
 for test in ${NOTMUCH_TESTS-}; do
     TESTS="$TESTS $NOTMUCH_SRCDIR/test/$test"
@@ -80,7 +88,7 @@ for file in $TESTS
 do
     file=${file##*/} # drop leading path components
     file=${file%.sh} # drop trailing '.sh'
-    RESULT_FILES="$RESULT_FILES $NOTMUCH_BUILDDIR/test/test-results/$file"
+    RESULT_FILES="$RESULT_FILES $TEST_DIRECTORY/test-results/$file"
 done
 
 echo
@@ -94,6 +102,6 @@ if [ -n "$META_FAILURE" ]; then
 fi
 
 # Clean up
-rm -rf $NOTMUCH_BUILDDIR/test/test-results
+rm -rf $TEST_DIRECTORY/test-results
 
 exit $ev
diff --git a/test/test-lib-common.sh b/test/test-lib-common.sh
index 18fa29c0..5ec9719a 100644
--- a/test/test-lib-common.sh
+++ b/test/test-lib-common.sh
@@ -24,7 +24,7 @@
 #
 type die >/dev/null 2>&1 || die () { echo "$@" >&2; exit 1; }
 
-if [[ -z "$NOTMUCH_SRCDIR" ]] || [[ -z "$NOTMUCH_BUILDDIR" ]]; then
+if [[ -z "$NOTMUCH_SRCDIR" ]] || [[ -z "${NOTMUCH_TEST_INSTALLED}" && -z "$NOTMUCH_BUILDDIR" ]]; then
 	echo "internal: srcdir or builddir not set" >&2
 	exit 1
 fi
@@ -61,7 +61,12 @@ LD_LIBRARY_PATH=${TEST_DIRECTORY%/*}/lib${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}
 export LD_LIBRARY_PATH
 
 # configure output
-. "$NOTMUCH_BUILDDIR/sh.config" || exit 1
+if [[ -z "${NOTMUCH_TEST_INSTALLED}" ]]; then
+   . "$NOTMUCH_BUILDDIR/sh.config" || exit 1
+fi
+
+# fall back definitions for running without configuration
+. "${TEST_DIRECTORY}/defaults.sh"
 
 # load OS specifics
 if [[ -e "$NOTMUCH_SRCDIR/test/test-lib-$PLATFORM.sh" ]]; then
@@ -315,7 +320,12 @@ export PATH MANPATH
 
 # Test repository
 test="tmp.$(basename "$0" .sh)"
-TMP_DIRECTORY="$TEST_DIRECTORY/$test"
+if [[ -z "${NOTMUCH_TEST_INSTALLED}" ]]; then
+    TMP_DIRECTORY="$TEST_DIRECTORY/$test"
+else
+    TMP_DIRECTORY=$(mktemp -d "${TMPDIR:-/tmp}/notmuch-$test.XXXXXX")
+fi
+
 test ! -z "$debug" || remove_tmp=$TMP_DIRECTORY
 rm -rf "$TMP_DIRECTORY" || {
 	GIT_EXIT_OK=t
diff --git a/test/test-lib-emacs.sh b/test/test-lib-emacs.sh
index ad4c4aeb..6843790b 100644
--- a/test/test-lib-emacs.sh
+++ b/test/test-lib-emacs.sh
@@ -30,6 +30,7 @@ test_require_emacs () {
 # to the message and encrypting/signing.
 emacs_deliver_message () {
     local subject body smtp_dummy_pid smtp_dummy_port
+    test_subtest_broken_for_installed
     subject="$1"
     body="$2"
     shift 2
@@ -144,6 +145,13 @@ emacs_generate_script () {
 	# Construct a little test script here for the benefit of the user,
 	# (who can easily run "run_emacs" to get the same emacs environment
 	# for investigating any failures).
+    if [ -z "${NOTMUCH_TEST_INSTALLED}" ]; then
+	find_notmuch_el='--directory "$NOTMUCH_BUILDDIR/emacs"'
+    else
+	### XXX FIXME: this should really use the installed emacs lisp files
+	find_notmuch_el='--directory "$NOTMUCH_SRCDIR/emacs"'
+    fi
+
 	cat <<EOF >"$TMP_DIRECTORY/run_emacs"
 #!/bin/sh
 export PATH=$PATH
@@ -158,12 +166,12 @@ export NOTMUCH_CONFIG=$NOTMUCH_CONFIG
 #
 # --load		Force loading of notmuch.el and test-lib.el
 
-exec ${TEST_EMACS} --quick \
-	--directory "$NOTMUCH_BUILDDIR/emacs" --load notmuch.el \
+exec ${TEST_EMACS} ${find_notmuch_el} --quick \
+	${EXTRA_DIR} --load notmuch.el \
 	--directory "$NOTMUCH_SRCDIR/test" --load test-lib.el \
 	"\$@"
 EOF
-	chmod a+x "$TMP_DIRECTORY/run_emacs"
+    chmod a+x "$TMP_DIRECTORY/run_emacs"
 }
 
 test_emacs () {
diff --git a/test/test-lib.sh b/test/test-lib.sh
index a647790a..89e2ba55 100644
--- a/test/test-lib.sh
+++ b/test/test-lib.sh
@@ -29,8 +29,8 @@ shopt -u xpg_echo
 # Ensure NOTMUCH_SRCDIR and NOTMUCH_BUILDDIR are set.
 . $(dirname "$0")/export-dirs.sh || exit 1
 
-# It appears that people try to run tests without building...
-if [[ ! -x "$NOTMUCH_BUILDDIR/notmuch" ]]; then
+# We need either a built tree, or a promise of an installed notmuch
+if [[ -z "${NOTMUCH_TEST_INSTALLED}" && ! -x "$NOTMUCH_BUILDDIR/notmuch" ]]; then
 	echo >&2 'You do not seem to have built notmuch yet.'
 	exit 1
 fi
@@ -748,6 +748,12 @@ test_subtest_known_broken () {
 	test_subtest_known_broken_=t
 }
 
+test_subtest_broken_for_installed () {
+    if [[ -n "${NOTMUCH_TEST_INSTALLED}" ]]; then
+	test_subtest_known_broken_=t
+    fi
+}
+
 test_subtest_broken_for_root () {
    if [ "$(id -u)" = "0" ]; then
 	test_subtest_known_broken_=t
@@ -929,11 +935,16 @@ make_shim () {
 }
 
 notmuch_with_shim () {
-    local base_name shim_file
-    base_name="$1"
+    local base_name shim_file notmuch_cmd
+    if [[ -n "${NOTMUCH_TEST_INSTALLED}" ]]; then
+	notmuch_cmd="notmuch"
+    else
+	notmuch_cmd="notmuch-shared"
+    fi
+    base_name=$1
     shift
     shim_file="${base_name}.so"
-    LD_PRELOAD=${LD_PRELOAD:+:$LD_PRELOAD}:./${shim_file} notmuch-shared "$@"
+    LD_PRELOAD=${LD_PRELOAD:+:$LD_PRELOAD}:./${shim_file} $notmuch_cmd "$@"
 }
 
 # Creates a script that counts how much time it is executed and calls
@@ -985,7 +996,11 @@ test_init_ () {
 
 
 # Where to run the tests
-TEST_DIRECTORY=$NOTMUCH_BUILDDIR/test
+if [[ -n "${NOTMUCH_BUILDDIR}" ]]; then
+    TEST_DIRECTORY=$NOTMUCH_BUILDDIR/test
+else
+    TEST_DIRECTORY=$NOTMUCH_SRCDIR/test
+fi
 
 . "$NOTMUCH_SRCDIR/test/test-lib-common.sh" || exit 1
 
-- 
2.39.0

^ permalink raw reply related	[flat|nested] 7+ messages in thread

* [PATCH 4/4] debian: add autopkgtests
  2023-01-04 12:41 Support testing notmuch as installed David Bremner
                   ` (2 preceding siblings ...)
  2023-01-04 12:41 ` [PATCH 3/4] test: support testing notmuch as installed David Bremner
@ 2023-01-04 12:41 ` David Bremner
  3 siblings, 0 replies; 7+ messages in thread
From: David Bremner @ 2023-01-04 12:41 UTC (permalink / raw)
  To: notmuch

We generate output to stderr for BROKEN tests, which are not failures,
so tell the test runner not to fail because of output on stderr.
---
 debian/tests/control | 13 +++++++++++++
 test/T800-asan.sh    |  8 ++++++--
 test/defaults.sh     |  2 +-
 3 files changed, 20 insertions(+), 3 deletions(-)
 create mode 100644 debian/tests/control

diff --git a/debian/tests/control b/debian/tests/control
new file mode 100644
index 00000000..c8dc9ff7
--- /dev/null
+++ b/debian/tests/control
@@ -0,0 +1,13 @@
+Test-command: env NOTMUCH_TEST_INSTALLED=yes TERM=dumb ./test/notmuch-test
+Restrictions: allow-stderr
+Depends: @,
+ build-essential,
+ dtach,
+ emacs-nox,
+ gdb,
+ git,
+ gnupg,
+ gpgsm,
+ libtalloc-dev,
+ man,
+ xapian-tools
diff --git a/test/T800-asan.sh b/test/T800-asan.sh
index 1ddff0fe..658b88f3 100755
--- a/test/T800-asan.sh
+++ b/test/T800-asan.sh
@@ -2,17 +2,21 @@
 test_description='run code with ASAN enabled against the library'
 . $(dirname "$0")/test-lib.sh || exit 1
 
-if [ $NOTMUCH_HAVE_ASAN -ne 1 ]; then
+if [ "$NOTMUCH_HAVE_ASAN" -ne 1 ]; then
     printf "Skipping due to missing ASAN support\n"
     test_done
 fi
 
+if [ -n "$LD_PRELOAD" ]; then
+    printf "Skipping due to non-empty LD_PRELOAD\n"
+    test_done
+fi
+
 add_email_corpus
 
 TEST_CFLAGS="-fsanitize=address"
 
 test_begin_subtest "open and destroy"
-test_subtest_broken_for_installed
 test_C ${MAIL_DIR} ${NOTMUCH_CONFIG} <<EOF
 #include <notmuch.h>
 #include <stdio.h>
diff --git a/test/defaults.sh b/test/defaults.sh
index 6327e43b..ee4ded83 100644
--- a/test/defaults.sh
+++ b/test/defaults.sh
@@ -2,4 +2,4 @@
 NOTMUCH_PYTHON=${NOTMUCH_PYTHON-python3}
 NOTMUCH_HAVE_MAN=${NOTMUCH_HAVE_MAN-1}
 NOTMUCH_HAVE_SFSEXP=${NOTMUCH_HAVE_SFSEXP-1}
-NOTMUCH_HAVE_ASAN=${NOTMUCH_HAVE_ASAN-0}
+NOTMUCH_HAVE_ASAN=${NOTMUCH_HAVE_ASAN-1}
-- 
2.39.0

^ permalink raw reply related	[flat|nested] 7+ messages in thread

* Re: [PATCH 1/4] test: mark some tests as broken when run as root.
  2023-01-04 12:41 ` [PATCH 1/4] test: mark some tests as broken when run as root David Bremner
@ 2023-01-05 10:12   ` Tomi Ollila
  2023-01-06  0:38     ` David Bremner
  0 siblings, 1 reply; 7+ messages in thread
From: Tomi Ollila @ 2023-01-05 10:12 UTC (permalink / raw)
  To: David Bremner, notmuch

On Wed, Jan 04 2023, David Bremner wrote:

> File permission errors e.g., are hard to trigger as root.
> ---
>  test/T050-new.sh     | 1 +
>  test/T150-tagging.sh | 1 +
>  test/test-lib.sh     | 6 ++++++
>  3 files changed, 8 insertions(+)
>
> diff --git a/test/T050-new.sh b/test/T050-new.sh
> index cb67889c..240c1810 100755
> --- a/test/T050-new.sh
> +++ b/test/T050-new.sh
> @@ -384,6 +384,7 @@ EOF
>  test_expect_equal_file EXPECTED OUTPUT
>  
>  test_begin_subtest "Xapian exception: read only files"
> +test_subtest_broken_for_root
>  chmod u-w ${MAIL_DIR}/.notmuch/xapian/*.*
>  output=$(NOTMUCH_NEW --debug 2>&1 | sed 's/: .*$//' )
>  chmod u+w ${MAIL_DIR}/.notmuch/xapian/*.*
> diff --git a/test/T150-tagging.sh b/test/T150-tagging.sh
> index 1a2fd77e..ac3f2539 100755
> --- a/test/T150-tagging.sh
> +++ b/test/T150-tagging.sh
> @@ -320,6 +320,7 @@ test_begin_subtest "Tag name beginning with -"
>  test_expect_code 1 'notmuch tag +- One'
>  
>  test_begin_subtest "Xapian exception: read only files"
> +test_subtest_broken_for_root
>  chmod u-w ${MAIL_DIR}/.notmuch/xapian/*.*
>  output=$(notmuch tag +something '*' 2>&1 | sed 's/: .*$//' )
>  chmod u+w ${MAIL_DIR}/.notmuch/xapian/*.*
> diff --git a/test/test-lib.sh b/test/test-lib.sh
> index eec5c5b4..c2b8cd8d 100644
> --- a/test/test-lib.sh
> +++ b/test/test-lib.sh
> @@ -740,6 +740,12 @@ test_subtest_known_broken () {
>  	test_subtest_known_broken_=t
>  }
>  
> +test_subtest_broken_for_root () {
> +   if [ "$(id -u)" = "0" ]; then
> +	test_subtest_known_broken_=t

Good progress -- bash has:

$ bash -c 'set' | grep UID
EUID=1001
UID=1001

$ podman run --rm -it debian:10.9 bash -c 'set' | grep UID
EUID=0
UID=0

i.e. EUID as builtin variable ( man id says id -u prints 
"only the effective user ID" )


another question is whether test_subtest_broken_for_root
is good name. perhaps it is tolerable enough

Tomi

> +    fi
> +}
> +
>  test_expect_success () {
>  	exec 1>&6 2>&7		# Restore stdout and stderr
>  	if [ -z "$inside_subtest" ]; then
> -- 
> 2.39.0

^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: [PATCH 1/4] test: mark some tests as broken when run as root.
  2023-01-05 10:12   ` Tomi Ollila
@ 2023-01-06  0:38     ` David Bremner
  0 siblings, 0 replies; 7+ messages in thread
From: David Bremner @ 2023-01-06  0:38 UTC (permalink / raw)
  To: Tomi Ollila, notmuch

Tomi Ollila <tomi.ollila@iki.fi> writes:

>
> Good progress -- bash has:
>
> $ bash -c 'set' | grep UID
> EUID=1001
> UID=1001

I made that change.

> another question is whether test_subtest_broken_for_root
> is good name. perhaps it is tolerable enough

I couldn't think of anything better, but am happy to search and replace
if someome comes up with something.

Amended version applied to master.

d

^ permalink raw reply	[flat|nested] 7+ messages in thread

end of thread, other threads:[~2023-01-06  0:38 UTC | newest]

Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2023-01-04 12:41 Support testing notmuch as installed David Bremner
2023-01-04 12:41 ` [PATCH 1/4] test: mark some tests as broken when run as root David Bremner
2023-01-05 10:12   ` Tomi Ollila
2023-01-06  0:38     ` David Bremner
2023-01-04 12:41 ` [PATCH 2/4] test: check for empty/missing files in test_expect_equal_message_body David Bremner
2023-01-04 12:41 ` [PATCH 3/4] test: support testing notmuch as installed David Bremner
2023-01-04 12:41 ` [PATCH 4/4] debian: add autopkgtests David Bremner

Code repositories for project(s) associated with this public inbox

	https://yhetil.org/notmuch.git/

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).