* v4 test notmuch as installed
@ 2023-07-09 14:08 David Bremner
2023-07-09 14:08 ` [PATCH v4 1/5] test: treat undefined feature variables as 0 David Bremner
` (5 more replies)
0 siblings, 6 replies; 13+ messages in thread
From: David Bremner @ 2023-07-09 14:08 UTC (permalink / raw)
To: notmuch
This obsoletes the series at id:20230409142627.2216080-1-david@tethera.net.
I have updated the shell handling of uninitialized variables based on
feedback from Felipe and Tomi. No new bash-isms are introduced (at
least, not on purpose) by this series.
Rather than '"${NOTMUCH_TEST_INSTALLED-}" = 1', I ended up using '-n
"${NOTMUCH_TEST_INSTALLED-}"'. This is consistent with how we handle
NOTMUCH_TEST_SERIALIZE. For the variables normally set by configure, I
do use e.g. '"${NOTMUCH_HAVE_ASAN-0}" != "1"'. I recognize this is not
completely consistent but the tradeoff of minimizing logical changes
in the normal (build time tests) case seems worth it.
^ permalink raw reply [flat|nested] 13+ messages in thread
* [PATCH v4 1/5] test: treat undefined feature variables as 0
2023-07-09 14:08 v4 test notmuch as installed David Bremner
@ 2023-07-09 14:08 ` David Bremner
2023-07-11 15:11 ` Tomi Ollila
2023-07-09 14:08 ` [PATCH v4 2/5] test: check for empty/missing files in test_expect_equal_message_body David Bremner
` (4 subsequent siblings)
5 siblings, 1 reply; 13+ messages in thread
From: David Bremner @ 2023-07-09 14:08 UTC (permalink / raw)
To: notmuch
When running the test suite without building first, it is desirable to
have the tests consider these variables being undefined as equivalent
to the feature not being present, and in particular for the tests not
to generate errors.
---
test/T010-help-test.sh | 2 +-
test/T060-count.sh | 2 +-
test/T081-sexpr-search.sh | 2 +-
test/T095-address.sh | 2 +-
test/T150-tagging.sh | 2 +-
test/T160-json.sh | 2 +-
test/T220-reply.sh | 2 +-
test/T240-dump-restore.sh | 2 +-
test/T355-smime.sh | 2 +-
test/T391-python-cffi.sh | 3 +--
test/T392-python-cffi-notmuch.sh | 2 +-
test/T520-show.sh | 2 +-
test/T570-revision-tracking.sh | 2 +-
test/T700-reindex.sh | 2 +-
test/T800-asan.sh | 2 +-
test/T810-tsan.sh | 2 +-
test/T850-git.sh | 2 +-
17 files changed, 17 insertions(+), 18 deletions(-)
diff --git a/test/T010-help-test.sh b/test/T010-help-test.sh
index da45d3ae..1fbc46a2 100755
--- a/test/T010-help-test.sh
+++ b/test/T010-help-test.sh
@@ -12,7 +12,7 @@ test_expect_success 'notmuch help'
test_begin_subtest 'notmuch --version'
test_expect_success 'notmuch --version'
-if [ $NOTMUCH_HAVE_MAN -eq 1 ]; then
+if [ "${NOTMUCH_HAVE_MAN-0}" = "1" ]; then
test_begin_subtest 'notmuch --help tag'
test_expect_success 'notmuch --help tag'
diff --git a/test/T060-count.sh b/test/T060-count.sh
index 48146706..6e855b59 100755
--- a/test/T060-count.sh
+++ b/test/T060-count.sh
@@ -157,7 +157,7 @@ print("4: {} messages".format(query.count_messages()))
EOF
test_expect_equal_file EXPECTED OUTPUT
-if [ $NOTMUCH_HAVE_SFSEXP -eq 1 ]; then
+if [ "${NOTMUCH_HAVE_SFSEXP-0}" = "1" ]; then
test_begin_subtest "and of exact terms (query=sexp)"
output=$(notmuch count --query=sexp '(and "wonderful" "wizard")')
diff --git a/test/T081-sexpr-search.sh b/test/T081-sexpr-search.sh
index 0c7db9c2..8800b545 100755
--- a/test/T081-sexpr-search.sh
+++ b/test/T081-sexpr-search.sh
@@ -2,7 +2,7 @@
test_description='"notmuch search" in several variations'
. $(dirname "$0")/test-lib.sh || exit 1
-if [ $NOTMUCH_HAVE_SFSEXP -ne 1 ]; then
+if [ "${NOTMUCH_HAVE_SFSEXP-0}" != "1" ]; then
printf "Skipping due to missing sfsexp library\n"
test_done
fi
diff --git a/test/T095-address.sh b/test/T095-address.sh
index 8bb3627a..0cafbe20 100755
--- a/test/T095-address.sh
+++ b/test/T095-address.sh
@@ -325,7 +325,7 @@ cat <<EOF >EXPECTED
EOF
test_expect_equal_file EXPECTED OUTPUT
-if [[ NOTMUCH_HAVE_SFSEXP = 1 ]]; then
+if [ "${NOTMUCH_HAVE_SFSEXP-0}" = "1" ]; then
test_begin_subtest "sexpr query: all messages"
notmuch address '*' > EXPECTED
notmuch address --query=sexp '()' > OUTPUT
diff --git a/test/T150-tagging.sh b/test/T150-tagging.sh
index ac3f2539..273c0af3 100755
--- a/test/T150-tagging.sh
+++ b/test/T150-tagging.sh
@@ -328,7 +328,7 @@ test_expect_equal "$output" "A Xapian exception occurred opening database"
add_email_corpus
-if [ $NOTMUCH_HAVE_SFSEXP -eq 1 ]; then
+if [ "${NOTMUCH_HAVE_SFSEXP-0}" = "1" ]; then
test_query_syntax '(and "wonderful" "wizard")' 'wonderful and wizard'
test_query_syntax '(or "php" "wizard")' 'php or wizard'
diff --git a/test/T160-json.sh b/test/T160-json.sh
index 4a797f6a..f61c3f2d 100755
--- a/test/T160-json.sh
+++ b/test/T160-json.sh
@@ -66,7 +66,7 @@ test_expect_equal_json "$output" "[{\"thread\": \"XXX\",
\"unread\"]}]"
test_begin_subtest "Search message: json, 64-bit timestamp"
-if [ $NOTMUCH_HAVE_64BIT_TIME_T -ne 1 ]; then
+if [ "${NOTMUCH_HAVE_64BIT_TIME_T-0}" != "1" ]; then
test_subtest_known_broken
fi
add_message "[subject]=\"json-search-64bit-timestamp-subject\"" "[date]=\"Tue, 01 Jan 2999 12:00:00 -0000\"" "[body]=\"json-search-64bit-timestamp-message\""
diff --git a/test/T220-reply.sh b/test/T220-reply.sh
index 207f5788..120d7138 100755
--- a/test/T220-reply.sh
+++ b/test/T220-reply.sh
@@ -24,7 +24,7 @@ test_begin_subtest "Basic reply"
notmuch reply id:${gen_msg_id} >OUTPUT 2>&1 && echo OK >> OUTPUT
test_expect_equal_file basic.expected OUTPUT
-if [ $NOTMUCH_HAVE_SFSEXP -eq 1 ]; then
+if [ "${NOTMUCH_HAVE_SFSEXP-0}" = "1" ]; then
test_begin_subtest "Basic reply (query=sexp)"
notmuch reply --query=sexp "(id ${gen_msg_id})" >OUTPUT 2>&1 && echo OK >> OUTPUT
diff --git a/test/T240-dump-restore.sh b/test/T240-dump-restore.sh
index a86f0fb7..80939030 100755
--- a/test/T240-dump-restore.sh
+++ b/test/T240-dump-restore.sh
@@ -118,7 +118,7 @@ notmuch dump -- from:cworth > dump-dash-cworth.actual
test_expect_equal_file dump-cworth.expected dump-dash-cworth.actual
-if [ $NOTMUCH_HAVE_SFSEXP -eq 1 ]; then
+if [ "${NOTMUCH_HAVE_SFSEXP-0}" = "1" ]; then
test_begin_subtest "dump --query=sexp -- '(from cworth)'"
notmuch dump --query=sexp -- '(from cworth)' > dump-dash-cworth.actual2
diff --git a/test/T355-smime.sh b/test/T355-smime.sh
index 809274ce..3bd5a17e 100755
--- a/test/T355-smime.sh
+++ b/test/T355-smime.sh
@@ -184,7 +184,7 @@ output=$(notmuch show --format=json id:smime-onepart-signed@protected-headers.ex
test_valid_json "$output"
test_begin_subtest "Verify signature on PKCS#7 SignedData message"
-if [ $NOTMUCH_HAVE_64BIT_TIME_T -ne 1 ]; then
+if [ "${NOTMUCH_HAVE_64BIT_TIME_T-0}" != "1" ]; then
test_subtest_known_broken
fi
output=$(notmuch show --format=json id:smime-onepart-signed@protected-headers.example)
diff --git a/test/T391-python-cffi.sh b/test/T391-python-cffi.sh
index 30872af0..0ef9e0d3 100755
--- a/test/T391-python-cffi.sh
+++ b/test/T391-python-cffi.sh
@@ -2,11 +2,10 @@
test_description="python bindings (pytest)"
. $(dirname "$0")/test-lib.sh || exit 1
-if [ $NOTMUCH_HAVE_PYTHON3_CFFI -eq 0 -o $NOTMUCH_HAVE_PYTHON3_PYTEST -eq 0 ]; then
+if [ "${NOTMUCH_HAVE_PYTHON3_CFFI-0}" = "0" -o "${NOTMUCH_HAVE_PYTHON3_PYTEST-0}" = "0" ]; then
test_done
fi
-
test_begin_subtest "python cffi tests (NOTMUCH_CONFIG set)"
pytest_dir=$NOTMUCH_BUILDDIR/bindings/python-cffi/build/stage
printf "[pytest]\nminversion = 3.0\naddopts = -ra\n" > $pytest_dir/pytest.ini
diff --git a/test/T392-python-cffi-notmuch.sh b/test/T392-python-cffi-notmuch.sh
index 15c8fc6b..5a6dd1bf 100755
--- a/test/T392-python-cffi-notmuch.sh
+++ b/test/T392-python-cffi-notmuch.sh
@@ -2,7 +2,7 @@
test_description="python bindings (notmuch test suite)"
. $(dirname "$0")/test-lib.sh || exit 1
-if [ $NOTMUCH_HAVE_PYTHON3_CFFI -eq 0 -o $NOTMUCH_HAVE_PYTHON3_PYTEST -eq 0 ]; then
+if [ "${NOTMUCH_HAVE_PYTHON3_CFFI-0}" = "0" -o "${NOTMUCH_HAVE_PYTHON3_PYTEST}" = "0" ]; then
test_done
fi
diff --git a/test/T520-show.sh b/test/T520-show.sh
index c7b73a6d..6bcf109c 100755
--- a/test/T520-show.sh
+++ b/test/T520-show.sh
@@ -35,7 +35,7 @@ notmuch show --entire-thread=true --sort=oldest-first $QUERY > OUTPUT
test_expect_equal_file EXPECTED OUTPUT
-if [ $NOTMUCH_HAVE_SFSEXP -eq 1 ]; then
+if [ "${NOTMUCH_HAVE_SFSEXP-0}" = "1" ]; then
test_query_syntax '(and "wonderful" "wizard")' 'wonderful and wizard'
test_query_syntax '(or "php" "wizard")' 'php or wizard'
diff --git a/test/T570-revision-tracking.sh b/test/T570-revision-tracking.sh
index a7480050..bcc97dd9 100755
--- a/test/T570-revision-tracking.sh
+++ b/test/T570-revision-tracking.sh
@@ -95,7 +95,7 @@ subtotal=$(notmuch count lastmod:..$lastmod)
result=$(($subtotal == $total-1))
test_expect_equal 1 "$result"
-if [ $NOTMUCH_HAVE_SFSEXP -eq 1 ]; then
+if [ "${NOTMUCH_HAVE_SFSEXP-0}" = "1" ]; then
test_begin_subtest 'exclude one message using negative lastmod (sexp)'
total=$(notmuch count '*')
notmuch tag +${RANDOM} id:4EFC743A.3060609@april.org
diff --git a/test/T700-reindex.sh b/test/T700-reindex.sh
index 347f8483..af34ad7c 100755
--- a/test/T700-reindex.sh
+++ b/test/T700-reindex.sh
@@ -5,7 +5,7 @@ test_description='reindexing messages'
add_email_corpus
-if [ $NOTMUCH_HAVE_SFSEXP -eq 1 ]; then
+if [ "${NOTMUCH_HAVE_SFSEXP-0}" = "1" ]; then
count=$(notmuch count --lastmod '*' | cut -f 3)
for query in '()' '(not)' '(and)' '(or ())' '(or (not))' '(or (and))' \
diff --git a/test/T800-asan.sh b/test/T800-asan.sh
index 5055c93e..7c28dc7c 100755
--- a/test/T800-asan.sh
+++ b/test/T800-asan.sh
@@ -2,7 +2,7 @@
test_description='run code with ASAN enabled against the library'
. $(dirname "$0")/test-lib.sh || exit 1
-if [ $NOTMUCH_HAVE_ASAN -ne 1 ]; then
+if [ "${NOTMUCH_HAVE_ASAN-0}" != "1" ]; then
printf "Skipping due to missing ASAN support\n"
test_done
fi
diff --git a/test/T810-tsan.sh b/test/T810-tsan.sh
index 7e877b27..4071e296 100755
--- a/test/T810-tsan.sh
+++ b/test/T810-tsan.sh
@@ -8,7 +8,7 @@ test_description='run code with TSan enabled against the library'
. "$test_directory"/test-lib.sh || exit 1
-if [ $NOTMUCH_HAVE_TSAN -ne 1 ]; then
+if [ "${NOTMUCH_HAVE_TSAN-0}" != "1" ]; then
printf "Skipping due to missing TSan support\n"
test_done
fi
diff --git a/test/T850-git.sh b/test/T850-git.sh
index 55cec78a..a063aacf 100755
--- a/test/T850-git.sh
+++ b/test/T850-git.sh
@@ -2,7 +2,7 @@
test_description='"notmuch git" to save and restore tags'
. $(dirname "$0")/test-lib.sh || exit 1
-if [ $NOTMUCH_HAVE_SFSEXP -ne 1 ]; then
+if [ "${NOTMUCH_HAVE_SFSEXP}" != "1" ]; then
printf "Skipping due to missing sfsexp library\n"
test_done
fi
--
2.40.1
^ permalink raw reply related [flat|nested] 13+ messages in thread
* [PATCH v4 2/5] test: check for empty/missing files in test_expect_equal_message_body
2023-07-09 14:08 v4 test notmuch as installed David Bremner
2023-07-09 14:08 ` [PATCH v4 1/5] test: treat undefined feature variables as 0 David Bremner
@ 2023-07-09 14:08 ` David Bremner
2023-07-11 15:14 ` Tomi Ollila
2023-07-09 14:08 ` [PATCH v4 3/5] test: Guess a value for NOTMUCH_PYTHON David Bremner
` (3 subsequent siblings)
5 siblings, 1 reply; 13+ messages in thread
From: David Bremner @ 2023-07-09 14:08 UTC (permalink / raw)
To: notmuch
Messages can have empty bodies, but empty files are not messages.
---
test/test-lib.sh | 8 ++++++++
1 file changed, 8 insertions(+)
diff --git a/test/test-lib.sh b/test/test-lib.sh
index 1a6525df..b5aa94dd 100644
--- a/test/test-lib.sh
+++ b/test/test-lib.sh
@@ -388,6 +388,14 @@ test_expect_equal_message_body () {
test "$#" = 2 ||
error "bug in the test script: not 2 parameters to test_expect_equal_file"
+ for file in "$1" "$2"; do
+ if [ ! -s "$file" ]; then
+ test_failure_ "Missing or zero length file: $file"
+ inside_subtest=
+ return $?
+ fi
+ done
+
expected=$(sed '1,/^$/d' "$1")
output=$(sed '1,/^$/d' "$2")
test_expect_equal "$expected" "$output"
--
2.40.1
^ permalink raw reply related [flat|nested] 13+ messages in thread
* [PATCH v4 3/5] test: Guess a value for NOTMUCH_PYTHON
2023-07-09 14:08 v4 test notmuch as installed David Bremner
2023-07-09 14:08 ` [PATCH v4 1/5] test: treat undefined feature variables as 0 David Bremner
2023-07-09 14:08 ` [PATCH v4 2/5] test: check for empty/missing files in test_expect_equal_message_body David Bremner
@ 2023-07-09 14:08 ` David Bremner
2023-07-11 15:18 ` Tomi Ollila
2023-07-09 14:08 ` [PATCH v4 4/5] test: support testing notmuch as installed David Bremner
` (2 subsequent siblings)
5 siblings, 1 reply; 13+ messages in thread
From: David Bremner @ 2023-07-09 14:08 UTC (permalink / raw)
To: notmuch
python3 will work for many people, and reduce the friction to running
the tests without running configure first.
---
test/test-lib-common.sh | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/test/test-lib-common.sh b/test/test-lib-common.sh
index 18fa29c0..4d14e0b3 100644
--- a/test/test-lib-common.sh
+++ b/test/test-lib-common.sh
@@ -323,6 +323,10 @@ rm -rf "$TMP_DIRECTORY" || {
exit 1
}
+# Provide a guess at a usable Python, to support running tests without
+# running configure first.
+NOTMUCH_PYTHON=${NOTMUCH_PYTHON-python3}
+
# A temporary home directory is needed by at least:
# - emacs/"Sending a message via (fake) SMTP"
# - emacs/"Reply within emacs"
--
2.40.1
^ permalink raw reply related [flat|nested] 13+ messages in thread
* [PATCH v4 4/5] test: support testing notmuch as installed
2023-07-09 14:08 v4 test notmuch as installed David Bremner
` (2 preceding siblings ...)
2023-07-09 14:08 ` [PATCH v4 3/5] test: Guess a value for NOTMUCH_PYTHON David Bremner
@ 2023-07-09 14:08 ` David Bremner
2023-07-11 15:39 ` Tomi Ollila
2023-07-09 14:08 ` [PATCH v4 5/5] debian: add autopkgtests David Bremner
2023-07-21 11:01 ` v4 test notmuch as installed David Bremner
5 siblings, 1 reply; 13+ messages in thread
From: David Bremner @ 2023-07-09 14:08 UTC (permalink / raw)
To: notmuch
We put some effort into testing the built copy rather than some
installed copy. On the other hand for people like packagers, testing
the installed copy is also of interest.
When NOTMUCH_TEST_INSTALLED is set to a nonempty value, tests do not
require a built notmuch tree or running configure.
Some of the tests marked as broken when running against installed
notmuch are probably fixable.
---
test/README | 17 +++++++++++++++++
test/T000-basic.sh | 1 +
test/T010-help-test.sh | 3 +++
test/T160-json.sh | 2 ++
test/T240-dump-restore.sh | 2 ++
test/T310-emacs.sh | 2 ++
test/T350-crypto.sh | 1 +
test/T355-smime.sh | 2 ++
test/T360-symbol-hiding.sh | 4 ++++
test/T390-python.sh | 4 ++++
test/T391-python-cffi.sh | 4 ++++
test/T395-ruby.sh | 8 ++++++--
test/T410-argument-parsing.sh | 4 ++++
test/T480-hex-escaping.sh | 4 ++++
test/T490-parse-time-string.sh | 4 ++++
test/T550-db-features.sh | 4 ++++
test/T566-lib-message.sh | 4 ++++
| 4 ++++
test/T710-message-id.sh | 4 ++++
test/T800-asan.sh | 5 +++++
test/T850-git.sh | 6 ++++++
test/export-dirs.sh | 2 +-
test/notmuch-test | 12 ++++++++++--
test/test-lib-common.sh | 13 ++++++++++---
test/test-lib-emacs.sh | 12 ++++++++++--
test/test-lib.sh | 27 +++++++++++++++++++++------
26 files changed, 139 insertions(+), 16 deletions(-)
diff --git a/test/README b/test/README
index 10f127cb..a81808b1 100644
--- a/test/README
+++ b/test/README
@@ -137,6 +137,23 @@ detection of missing prerequisites. In the future we may treat tests
unable to run because of missing prerequisites, but not explicitly
skipped by the user, as failures.
+Testing installed notmuch
+-------------------------
+
+Systems integrators (e.g. Linux distros) may wish to test an installed
+version of notmuch. This can be done be running
+
+ $ NOTMUCH_TEST_INSTALLED=1 ./test/notmuch-test
+
+In this scenario the test suite does not assume a built tree, and in
+particular cannot rely on the output of 'configure'. You may want to
+set certain feature environment variables ('NOTMUCH_HAVE_*') directly
+if you know those apply to your installed notmuch). Consider also
+setting TERM=dumb if the value of TERM cannot be used (e.g. in a
+chroot with missing terminfo). Note that having a built tree may cause
+surprising/broken results for NOTMUCH_TEST_INSTALLED, so consider
+cleaning first.
+
Writing Tests
-------------
The test script is written as a shell script. It is to be named as
diff --git a/test/T000-basic.sh b/test/T000-basic.sh
index a2f4d93f..642f918d 100755
--- a/test/T000-basic.sh
+++ b/test/T000-basic.sh
@@ -66,6 +66,7 @@ test_begin_subtest 'NOTMUCH_CONFIG is set and points to an existing file'
test_expect_success 'test -f "${NOTMUCH_CONFIG}"'
test_begin_subtest 'PATH is set to build directory'
+test_subtest_broken_for_installed
test_expect_equal \
"$(dirname ${TEST_DIRECTORY})" \
"$(echo $PATH|cut -f1 -d: | sed -e 's,/test/valgrind/bin$,,')"
diff --git a/test/T010-help-test.sh b/test/T010-help-test.sh
index 1fbc46a2..827edc14 100755
--- a/test/T010-help-test.sh
+++ b/test/T010-help-test.sh
@@ -19,6 +19,9 @@ if [ "${NOTMUCH_HAVE_MAN-0}" = "1" ]; then
test_begin_subtest 'notmuch help tag'
test_expect_success 'notmuch help tag'
else
+ if [ -n "${NOTMUCH_TEST_INSTALLED-}" ]; then
+ test_done
+ fi
test_begin_subtest 'notmuch --help tag (man pages not available)'
test_expect_success 'test_must_fail notmuch --help tag >/dev/null'
diff --git a/test/T160-json.sh b/test/T160-json.sh
index f61c3f2d..318c9788 100755
--- a/test/T160-json.sh
+++ b/test/T160-json.sh
@@ -65,6 +65,7 @@ test_expect_equal_json "$output" "[{\"thread\": \"XXX\",
\"tags\": [\"inbox\",
\"unread\"]}]"
+if [ -z "${NOTMUCH_TEST_INSTALLED-}" ]; then
test_begin_subtest "Search message: json, 64-bit timestamp"
if [ "${NOTMUCH_HAVE_64BIT_TIME_T-0}" != "1" ]; then
test_subtest_known_broken
@@ -81,6 +82,7 @@ test_expect_equal_json "$output" "[{\"thread\": \"XXX\",
\"query\": [\"id:$gen_msg_id\", null],
\"tags\": [\"inbox\",
\"unread\"]}]"
+fi # NOTMUCH_TEST_INSTALLED undefined / empty
test_begin_subtest "Format version: too low"
test_expect_code 20 "notmuch search --format-version=0 \\*"
diff --git a/test/T240-dump-restore.sh b/test/T240-dump-restore.sh
index 80939030..c3f18839 100755
--- a/test/T240-dump-restore.sh
+++ b/test/T240-dump-restore.sh
@@ -139,6 +139,7 @@ notmuch dump --output=dump-outfile-dash-inbox.actual -- from:cworth
test_expect_equal_file dump-cworth.expected dump-outfile-dash-inbox.actual
test_begin_subtest "Check for a safe set of message-ids"
+test_subtest_broken_for_installed
notmuch search --output=messages from:cworth | sed s/^id:// > EXPECTED
notmuch search --output=messages from:cworth | sed s/^id:// |\
$TEST_DIRECTORY/hex-xcode --direction=encode > OUTPUT
@@ -246,6 +247,7 @@ notmuch dump --format=batch-tag > OUTPUT.$test_count
test_expect_equal_file EXPECTED.$test_count OUTPUT.$test_count
test_begin_subtest 'format=batch-tag, checking encoded output'
+test_subtest_broken_for_installed
NOTMUCH_DUMP_TAGS --format=batch-tag -- from:cworth |\
awk "{ print \"+$enc1 +$enc2 +$enc3 -- \" \$5 }" > EXPECTED.$test_count
NOTMUCH_DUMP_TAGS --format=batch-tag -- from:cworth > OUTPUT.$test_count
diff --git a/test/T310-emacs.sh b/test/T310-emacs.sh
index 9d0df187..d3aa2e7d 100755
--- a/test/T310-emacs.sh
+++ b/test/T310-emacs.sh
@@ -258,6 +258,7 @@ EOF
test_expect_equal_file EXPECTED OUTPUT
test_begin_subtest "Verify that sent messages are saved/searchable (via FCC)"
+test_subtest_broken_for_installed
notmuch new > /dev/null
output=$(notmuch search 'subject:"testing message sent via SMTP"' | notmuch_search_sanitize)
test_expect_equal "$output" "thread:XXX 2000-01-01 [1/1] Notmuch Test Suite; Testing message sent via SMTP (inbox)"
@@ -350,6 +351,7 @@ EOF
test_expect_equal_file EXPECTED OUTPUT
test_begin_subtest "Reply within emacs"
+test_subtest_broken_for_installed
test_emacs '(let ((message-hidden-headers ''()))
(notmuch-search "subject:\"testing message sent via SMTP\"")
(notmuch-test-wait)
diff --git a/test/T350-crypto.sh b/test/T350-crypto.sh
index 721cbfdd..27c0e86d 100755
--- a/test/T350-crypto.sh
+++ b/test/T350-crypto.sh
@@ -30,6 +30,7 @@ msg_file=$(notmuch search --output=files subject:signed-message-sent-via-SMTP)
test_expect_equal_message_body sent_message "$msg_file"
test_begin_subtest "signed part content-type indexing"
+test_subtest_broken_for_installed
notmuch search mimetype:multipart/signed and mimetype:application/pgp-signature | notmuch_search_sanitize > OUTPUT
cat <<EOF >EXPECTED
thread:XXX 2000-01-01 [1/1] Notmuch Test Suite; test signed message 001 (inbox signed)
diff --git a/test/T355-smime.sh b/test/T355-smime.sh
index 3bd5a17e..d2118c04 100755
--- a/test/T355-smime.sh
+++ b/test/T355-smime.sh
@@ -183,6 +183,7 @@ test_begin_subtest "show PKCS#7 SignedData outputs valid JSON"
output=$(notmuch show --format=json id:smime-onepart-signed@protected-headers.example)
test_valid_json "$output"
+if [ -z "${NOTMUCH_TEST_INSTALLED-}" ]; then
test_begin_subtest "Verify signature on PKCS#7 SignedData message"
if [ "${NOTMUCH_HAVE_64BIT_TIME_T-0}" != "1" ]; then
test_subtest_known_broken
@@ -194,6 +195,7 @@ test_json_nodes <<<"$output" \
'expires:[0][0][0]["crypto"]["signed"]["status"][0]["expires"]=2611032858' \
'fingerprint:[0][0][0]["crypto"]["signed"]["status"][0]["fingerprint"]="702BA4B157F1E2B7D16B0C6A5FFC8A7DE2057DEB"' \
'status:[0][0][0]["crypto"]["signed"]["status"][0]["status"]="good"'
+fi # NOTMUCH_TEST_INSTALLED undefined / empty
test_begin_subtest "Verify signature on PKCS#7 SignedData message signer User ID"
if [ $NOTMUCH_GMIME_X509_CERT_VALIDITY -ne 1 ]; then
diff --git a/test/T360-symbol-hiding.sh b/test/T360-symbol-hiding.sh
index 642457bf..ff06ff69 100755
--- a/test/T360-symbol-hiding.sh
+++ b/test/T360-symbol-hiding.sh
@@ -11,6 +11,10 @@ test_description='exception symbol hiding'
. $(dirname "$0")/test-lib.sh || exit 1
+if [ -n "${NOTMUCH_TEST_INSTALLED-}" ]; then
+ test_done
+fi
+
test_begin_subtest 'running test' run_test
mkdir -p ${PWD}/fakedb/.notmuch
$TEST_DIRECTORY/symbol-test ${PWD}/fakedb ${PWD}/nonexistent 2>&1 \
diff --git a/test/T390-python.sh b/test/T390-python.sh
index 9f71ce3c..21912431 100755
--- a/test/T390-python.sh
+++ b/test/T390-python.sh
@@ -4,6 +4,10 @@ test_description="python bindings"
test_require_external_prereq ${NOTMUCH_PYTHON}
+if [ -n "${NOTMUCH_TEST_INSTALLED-}" ]; then
+ test_done
+fi
+
add_email_corpus
add_gnupg_home
diff --git a/test/T391-python-cffi.sh b/test/T391-python-cffi.sh
index 0ef9e0d3..0059b050 100755
--- a/test/T391-python-cffi.sh
+++ b/test/T391-python-cffi.sh
@@ -2,6 +2,10 @@
test_description="python bindings (pytest)"
. $(dirname "$0")/test-lib.sh || exit 1
+if [ -n "${NOTMUCH_TEST_INSTALLED-}" ]; then
+ test_done
+fi
+
if [ "${NOTMUCH_HAVE_PYTHON3_CFFI-0}" = "0" -o "${NOTMUCH_HAVE_PYTHON3_PYTEST-0}" = "0" ]; then
test_done
fi
diff --git a/test/T395-ruby.sh b/test/T395-ruby.sh
index c066c842..ea9977f8 100755
--- a/test/T395-ruby.sh
+++ b/test/T395-ruby.sh
@@ -2,7 +2,7 @@
test_description="ruby bindings"
. $(dirname "$0")/test-lib.sh || exit 1
-if [ "${NOTMUCH_HAVE_RUBY_DEV}" = "0" ]; then
+if [ -z "${NOTMUCH_TEST_INSTALLED-}" -a "${NOTMUCH_HAVE_RUBY_DEV}" = "0" ]; then
test_subtest_missing_external_prereq_["ruby development files"]=t
fi
@@ -15,7 +15,11 @@ test_ruby() {
db = Notmuch::Database.new()
EOF
cat
- ) | $NOTMUCH_RUBY -I "$NOTMUCH_BUILDDIR/bindings/ruby"> OUTPUT
+ ) | if [ -n "${NOTMUCH_TEST_INSTALLED-}" ]; then
+ ruby
+ else
+ $NOTMUCH_RUBY -I "$NOTMUCH_BUILDDIR/bindings/ruby"
+ fi> OUTPUT
test_expect_equal_file EXPECTED OUTPUT
}
diff --git a/test/T410-argument-parsing.sh b/test/T410-argument-parsing.sh
index d9aa8e2d..40b625fe 100755
--- a/test/T410-argument-parsing.sh
+++ b/test/T410-argument-parsing.sh
@@ -2,6 +2,10 @@
test_description="argument parsing"
. $(dirname "$0")/test-lib.sh || exit 1
+if [ -n "${NOTMUCH_TEST_INSTALLED-}" ]; then
+ test_done
+fi
+
test_begin_subtest "sanity check"
$TEST_DIRECTORY/arg-test pos1 --keyword=one --boolean --string=foo pos2 --int=7 --flag=one --flag=three > OUTPUT
cat <<EOF > EXPECTED
diff --git a/test/T480-hex-escaping.sh b/test/T480-hex-escaping.sh
index b2eb80b9..8bddf3e7 100755
--- a/test/T480-hex-escaping.sh
+++ b/test/T480-hex-escaping.sh
@@ -2,6 +2,10 @@
test_description="hex encoding and decoding"
. $(dirname "$0")/test-lib.sh || exit 1
+if [ -n "${NOTMUCH_TEST_INSTALLED-}" ]; then
+ test_done
+fi
+
test_begin_subtest "round trip"
find $NOTMUCH_SRCDIR/test/corpora/default -type f -print | sort | xargs cat > EXPECTED
$TEST_DIRECTORY/hex-xcode --direction=encode < EXPECTED | $TEST_DIRECTORY/hex-xcode --direction=decode > OUTPUT
diff --git a/test/T490-parse-time-string.sh b/test/T490-parse-time-string.sh
index f89755ed..3b6e48c4 100755
--- a/test/T490-parse-time-string.sh
+++ b/test/T490-parse-time-string.sh
@@ -2,6 +2,10 @@
test_description="date/time parser module"
. $(dirname "$0")/test-lib.sh || exit 1
+if [ -n "${NOTMUCH_TEST_INSTALLED-}" ]; then
+ test_done
+fi
+
# Sanity/smoke tests for the date/time parser independent of notmuch
_date () {
diff --git a/test/T550-db-features.sh b/test/T550-db-features.sh
index 9d5a9e70..3048c7c4 100755
--- a/test/T550-db-features.sh
+++ b/test/T550-db-features.sh
@@ -3,6 +3,10 @@ test_description="database version and feature compatibility"
. $(dirname "$0")/test-lib.sh || exit 1
+if [ -n "${NOTMUCH_TEST_INSTALLED-}" ]; then
+ test_done
+fi
+
test_begin_subtest "future database versions abort open"
${TEST_DIRECTORY}/make-db-version ${MAIL_DIR} 9999 ""
output=$(notmuch search x 2>&1 | sed 's/\(database at\) .*/\1 FILENAME/')
diff --git a/test/T566-lib-message.sh b/test/T566-lib-message.sh
index 7f0e8eb0..69051937 100755
--- a/test/T566-lib-message.sh
+++ b/test/T566-lib-message.sh
@@ -3,6 +3,10 @@ test_description="API tests for notmuch_message_*"
. $(dirname "$0")/test-lib.sh || exit 1
+if [ -n "${NOTMUCH_TEST_INSTALLED}" ]; then
+ test_done
+fi
+
add_email_corpus
test_begin_subtest "building database"
--git a/test/T592-thread-breakage.sh b/test/T592-thread-breakage.sh
index 92a70e3e..2334fcaf 100755
--- a/test/T592-thread-breakage.sh
+++ b/test/T592-thread-breakage.sh
@@ -21,6 +21,10 @@ test_description='thread breakage during reindexing'
. $(dirname "$0")/test-lib.sh || exit 1
+if [ -n "${NOTMUCH_TEST_INSTALLED-}" ]; then
+ test_done
+fi
+
message_a () {
mkdir -p ${MAIL_DIR}/cur
cat > ${MAIL_DIR}/cur/a <<EOF
diff --git a/test/T710-message-id.sh b/test/T710-message-id.sh
index 5129d84c..a2d8ec71 100755
--- a/test/T710-message-id.sh
+++ b/test/T710-message-id.sh
@@ -3,6 +3,10 @@ test_description="message id parsing"
. $(dirname "$0")/test-lib.sh || exit 1
+if [ -n "${NOTMUCH_TEST_INSTALLED-}" ]; then
+ test_done
+fi
+
test_begin_subtest "good message ids"
${TEST_DIRECTORY}/message-id-parse <<EOF >OUTPUT
<018b1a8f2d1df62e804ce88b65401304832dfbbf.1346614915.git.jani@nikula.org>
diff --git a/test/T800-asan.sh b/test/T800-asan.sh
index 7c28dc7c..3a26b77e 100755
--- a/test/T800-asan.sh
+++ b/test/T800-asan.sh
@@ -7,6 +7,11 @@ if [ "${NOTMUCH_HAVE_ASAN-0}" != "1" ]; then
test_done
fi
+if [ -n "${LD_PRELOAD-}" ]; then
+ printf "Skipping due to non-empty LD_PRELOAD\n"
+ test_done
+fi
+
add_email_corpus
TEST_CFLAGS="${TEST_CFLAGS:-} -fsanitize=address"
diff --git a/test/T850-git.sh b/test/T850-git.sh
index a063aacf..47314a1b 100755
--- a/test/T850-git.sh
+++ b/test/T850-git.sh
@@ -233,6 +233,7 @@ EOF
test_expect_equal_file EXPECTED OUTPUT
test_begin_subtest "invoke as nmbug sets defaults"
+test_subtest_broken_for_installed
"$NOTMUCH_BUILDDIR"/nmbug -ldebug status |& grep '^\(prefix\|repository\)' | notmuch_dir_sanitize > OUTPUT
cat <<EOF > EXPECTED
prefix = notmuch::
@@ -241,6 +242,7 @@ EOF
test_expect_equal_file EXPECTED OUTPUT
test_begin_subtest "env variable NOTMUCH_GIT_DIR works when invoked as nmbug"
+test_subtest_broken_for_installed
NOTMUCH_GIT_DIR=`pwd`/foo "$NOTMUCH_BUILDDIR"/nmbug -ldebug status |& grep '^repository' | notmuch_dir_sanitize > OUTPUT
cat <<EOF > EXPECTED
repository = CWD/foo
@@ -256,6 +258,7 @@ test_expect_equal_file EXPECTED OUTPUT
test_begin_subtest "env variable NOTMUCH_GIT_DIR overrides config when invoked as 'nmbug'"
+test_subtest_broken_for_installed
notmuch config set git.path `pwd`/bar
NOTMUCH_GIT_DIR=`pwd`/remote.git "$NOTMUCH_BUILDDIR"/nmbug -ldebug status |& grep '^repository' | notmuch_dir_sanitize > OUTPUT
notmuch config set git.path
@@ -274,6 +277,7 @@ EOF
test_expect_equal_file EXPECTED OUTPUT
test_begin_subtest "env variable NOTMUCH_GIT_PREFIX works when invoked as 'nmbug'"
+test_subtest_broken_for_installed
NOTMUCH_GIT_PREFIX=env:: "$NOTMUCH_BUILDDIR"/nmbug -ldebug status |& grep '^prefix' | notmuch_dir_sanitize > OUTPUT
cat <<EOF > EXPECTED
prefix = env::
@@ -281,6 +285,7 @@ EOF
test_expect_equal_file EXPECTED OUTPUT
test_begin_subtest "env variable NOTMUCH_GIT_PREFIX works when invoked as nmbug"
+test_subtest_broken_for_installed
NOTMUCH_GIT_PREFIX=foo:: "$NOTMUCH_BUILDDIR"/nmbug -ldebug status |& grep '^prefix' | notmuch_dir_sanitize > OUTPUT
cat <<EOF > EXPECTED
prefix = foo::
@@ -288,6 +293,7 @@ EOF
test_expect_equal_file EXPECTED OUTPUT
test_begin_subtest "env variable NOTMUCH_GIT_PREFIX overrides config when invoked as 'nmbug'"
+test_subtest_broken_for_installed
notmuch config set git.tag_prefix config::
NOTMUCH_GIT_PREFIX=env:: "$NOTMUCH_BUILDDIR"/nmbug -ldebug status |& grep '^prefix' | notmuch_dir_sanitize > OUTPUT
notmuch config set git.path
diff --git a/test/export-dirs.sh b/test/export-dirs.sh
index 0a048e1f..f0374387 100644
--- a/test/export-dirs.sh
+++ b/test/export-dirs.sh
@@ -24,7 +24,7 @@ find_builddir () {
if [[ -z "${NOTMUCH_BUILDDIR}" ]]; then
export NOTMUCH_BUILDDIR="$(find_builddir "$(pwd)")"
- if [[ -z "${NOTMUCH_BUILDDIR}" ]]; then
+ if [ -z "${NOTMUCH_BUILDDIR}" -a "${NOTMUCH_TEST_INSTALLED-0}" = "0" ]; then
echo "Run tests in a subdir of built notmuch tree." >&2
exit 1
fi
diff --git a/test/notmuch-test b/test/notmuch-test
index 14747bdb..5d27e4d1 100755
--- a/test/notmuch-test
+++ b/test/notmuch-test
@@ -20,6 +20,14 @@ fi
set -eu
+# Where to run the tests
+# XXX FIXME this code is duplicated with test-lib.sh
+if [[ -n "${NOTMUCH_BUILDDIR}" ]]; then
+ TEST_DIRECTORY=$NOTMUCH_BUILDDIR/test
+else
+ TEST_DIRECTORY=$NOTMUCH_SRCDIR/test
+fi
+
TESTS=
for test in ${NOTMUCH_TESTS-}; do
TESTS="$TESTS $NOTMUCH_SRCDIR/test/$test"
@@ -80,7 +88,7 @@ for file in $TESTS
do
file=${file##*/} # drop leading path components
file=${file%.sh} # drop trailing '.sh'
- RESULT_FILES="$RESULT_FILES $NOTMUCH_BUILDDIR/test/test-results/$file"
+ RESULT_FILES="$RESULT_FILES $TEST_DIRECTORY/test-results/$file"
done
echo
@@ -94,6 +102,6 @@ if [ -n "$META_FAILURE" ]; then
fi
# Clean up
-rm -rf $NOTMUCH_BUILDDIR/test/test-results
+rm -rf $TEST_DIRECTORY/test-results
exit $ev
diff --git a/test/test-lib-common.sh b/test/test-lib-common.sh
index 4d14e0b3..f5d72e12 100644
--- a/test/test-lib-common.sh
+++ b/test/test-lib-common.sh
@@ -24,7 +24,7 @@
#
type die >/dev/null 2>&1 || die () { echo "$@" >&2; exit 1; }
-if [[ -z "$NOTMUCH_SRCDIR" ]] || [[ -z "$NOTMUCH_BUILDDIR" ]]; then
+if [[ -z "$NOTMUCH_SRCDIR" ]] || [ -z "${NOTMUCH_TEST_INSTALLED-}" -a -z "$NOTMUCH_BUILDDIR" ]; then
echo "internal: srcdir or builddir not set" >&2
exit 1
fi
@@ -61,7 +61,9 @@ LD_LIBRARY_PATH=${TEST_DIRECTORY%/*}/lib${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}
export LD_LIBRARY_PATH
# configure output
-. "$NOTMUCH_BUILDDIR/sh.config" || exit 1
+if [ -z "${NOTMUCH_TEST_INSTALLED-}" ]; then
+ . "$NOTMUCH_BUILDDIR/sh.config" || exit 1
+fi
# load OS specifics
if [[ -e "$NOTMUCH_SRCDIR/test/test-lib-$PLATFORM.sh" ]]; then
@@ -315,7 +317,12 @@ export PATH MANPATH
# Test repository
test="tmp.$(basename "$0" .sh)"
-TMP_DIRECTORY="$TEST_DIRECTORY/$test"
+if [ -z "${NOTMUCH_TEST_INSTALLED-}" ]; then
+ TMP_DIRECTORY="$TEST_DIRECTORY/$test"
+else
+ TMP_DIRECTORY=$(mktemp -d "${TMPDIR:-/tmp}/notmuch-$test.XXXXXX")
+fi
+
test ! -z "$debug" || remove_tmp=$TMP_DIRECTORY
rm -rf "$TMP_DIRECTORY" || {
GIT_EXIT_OK=t
diff --git a/test/test-lib-emacs.sh b/test/test-lib-emacs.sh
index ad4c4aeb..0ab58fc2 100644
--- a/test/test-lib-emacs.sh
+++ b/test/test-lib-emacs.sh
@@ -30,6 +30,7 @@ test_require_emacs () {
# to the message and encrypting/signing.
emacs_deliver_message () {
local subject body smtp_dummy_pid smtp_dummy_port
+ test_subtest_broken_for_installed
subject="$1"
body="$2"
shift 2
@@ -144,6 +145,13 @@ emacs_generate_script () {
# Construct a little test script here for the benefit of the user,
# (who can easily run "run_emacs" to get the same emacs environment
# for investigating any failures).
+ if [ -z "${NOTMUCH_TEST_INSTALLED-}" ]; then
+ find_notmuch_el='--directory "$NOTMUCH_BUILDDIR/emacs"'
+ else
+ ### XXX FIXME: this should really use the installed emacs lisp files
+ find_notmuch_el='--directory "$NOTMUCH_SRCDIR/emacs"'
+ fi
+
cat <<EOF >"$TMP_DIRECTORY/run_emacs"
#!/bin/sh
export PATH=$PATH
@@ -158,8 +166,8 @@ export NOTMUCH_CONFIG=$NOTMUCH_CONFIG
#
# --load Force loading of notmuch.el and test-lib.el
-exec ${TEST_EMACS} --quick \
- --directory "$NOTMUCH_BUILDDIR/emacs" --load notmuch.el \
+exec ${TEST_EMACS} ${find_notmuch_el} --quick \
+ ${EXTRA_DIR} --load notmuch.el \
--directory "$NOTMUCH_SRCDIR/test" --load test-lib.el \
"\$@"
EOF
diff --git a/test/test-lib.sh b/test/test-lib.sh
index b5aa94dd..8452fba2 100644
--- a/test/test-lib.sh
+++ b/test/test-lib.sh
@@ -29,8 +29,8 @@ shopt -u xpg_echo
# Ensure NOTMUCH_SRCDIR and NOTMUCH_BUILDDIR are set.
. $(dirname "$0")/export-dirs.sh || exit 1
-# It appears that people try to run tests without building...
-if [[ ! -x "$NOTMUCH_BUILDDIR/notmuch" ]]; then
+# We need either a built tree, or a promise of an installed notmuch
+if [ -z "${NOTMUCH_TEST_INSTALLED-}" -a ! -x "$NOTMUCH_BUILDDIR/notmuch" ]; then
echo >&2 'You do not seem to have built notmuch yet.'
exit 1
fi
@@ -748,6 +748,12 @@ test_subtest_known_broken () {
test_subtest_known_broken_=t
}
+test_subtest_broken_for_installed () {
+ if [ -n "${NOTMUCH_TEST_INSTALLED-}" ]; then
+ test_subtest_known_broken_=t
+ fi
+}
+
test_subtest_broken_for_root () {
if [ "$EUID" = "0" ]; then
test_subtest_known_broken_=t
@@ -929,11 +935,16 @@ make_shim () {
}
notmuch_with_shim () {
- local base_name shim_file
- base_name="$1"
+ local base_name shim_file notmuch_cmd
+ if [ -n "${NOTMUCH_TEST_INSTALLED-}" ]; then
+ notmuch_cmd="notmuch"
+ else
+ notmuch_cmd="notmuch-shared"
+ fi
+ base_name=$1
shift
shim_file="${base_name}.so"
- LD_PRELOAD=${LD_PRELOAD:+:$LD_PRELOAD}:./${shim_file} notmuch-shared "$@"
+ LD_PRELOAD=${LD_PRELOAD:+:$LD_PRELOAD}:./${shim_file} $notmuch_cmd "$@"
}
# Creates a script that counts how much time it is executed and calls
@@ -985,7 +996,11 @@ test_init_ () {
# Where to run the tests
-TEST_DIRECTORY=$NOTMUCH_BUILDDIR/test
+if [[ -n "${NOTMUCH_BUILDDIR}" ]]; then
+ TEST_DIRECTORY=$NOTMUCH_BUILDDIR/test
+else
+ TEST_DIRECTORY=$NOTMUCH_SRCDIR/test
+fi
. "$NOTMUCH_SRCDIR/test/test-lib-common.sh" || exit 1
--
2.40.1
^ permalink raw reply related [flat|nested] 13+ messages in thread
* [PATCH v4 5/5] debian: add autopkgtests
2023-07-09 14:08 v4 test notmuch as installed David Bremner
` (3 preceding siblings ...)
2023-07-09 14:08 ` [PATCH v4 4/5] test: support testing notmuch as installed David Bremner
@ 2023-07-09 14:08 ` David Bremner
2023-07-21 11:01 ` v4 test notmuch as installed David Bremner
5 siblings, 0 replies; 13+ messages in thread
From: David Bremner @ 2023-07-09 14:08 UTC (permalink / raw)
To: notmuch
We generate output to stderr for BROKEN tests, which are not failures,
so tell the test runner not to fail because of output on stderr.
---
debian/tests/control | 17 +++++++++++++++++
1 file changed, 17 insertions(+)
create mode 100644 debian/tests/control
diff --git a/debian/tests/control b/debian/tests/control
new file mode 100644
index 00000000..87e00dc8
--- /dev/null
+++ b/debian/tests/control
@@ -0,0 +1,17 @@
+Test-command: env NOTMUCH_TEST_INSTALLED=1 TERM=dumb
+ NOTMUCH_HAVE_MAN=1 NOTMUCH_HAVE_SFSEXP=1 NOTMUCH_HAVE_XAPIAN_DB_RETRY_LOCK=1
+ NOTMUCH_HAVE_PYTHON3_CFFI=1 NOTMUCH_HAVE_PYTHON3_PYTEST=1
+ NOTMUCH_HAVE_ASAN=1 NOTMUCH_HAVE_TSAN=1
+ ./test/notmuch-test
+Restrictions: allow-stderr
+Depends: @,
+ build-essential,
+ dtach,
+ emacs-nox,
+ gdb,
+ git,
+ gnupg,
+ gpgsm,
+ libtalloc-dev,
+ man,
+ xapian-tools
--
2.40.1
^ permalink raw reply related [flat|nested] 13+ messages in thread
* Re: [PATCH v4 1/5] test: treat undefined feature variables as 0
2023-07-09 14:08 ` [PATCH v4 1/5] test: treat undefined feature variables as 0 David Bremner
@ 2023-07-11 15:11 ` Tomi Ollila
0 siblings, 0 replies; 13+ messages in thread
From: Tomi Ollila @ 2023-07-11 15:11 UTC (permalink / raw)
To: David Bremner, notmuch
On Sun, Jul 09 2023, David Bremner wrote:
> When running the test suite without building first, it is desirable to
> have the tests consider these variables being undefined as equivalent
> to the feature not being present, and in particular for the tests not
> to generate errors.
> ---
...
> diff --git a/test/T392-python-cffi-notmuch.sh b/test/T392-python-cffi-notmuch.sh
> index 15c8fc6b..5a6dd1bf 100755
> --- a/test/T392-python-cffi-notmuch.sh
> +++ b/test/T392-python-cffi-notmuch.sh
> @@ -2,7 +2,7 @@
> test_description="python bindings (notmuch test suite)"
> . $(dirname "$0")/test-lib.sh || exit 1
>
> -if [ $NOTMUCH_HAVE_PYTHON3_CFFI -eq 0 -o $NOTMUCH_HAVE_PYTHON3_PYTEST -eq 0 ]; then
> +if [ "${NOTMUCH_HAVE_PYTHON3_CFFI-0}" = "0" -o "${NOTMUCH_HAVE_PYTHON3_PYTEST}" = "0" ]; then
here ensuring NOTMUCH_HAVE_PYTHON3_PYTEST is defined is missing...
> test_done
> fi
>
...
> diff --git a/test/T850-git.sh b/test/T850-git.sh
> index 55cec78a..a063aacf 100755
> --- a/test/T850-git.sh
> +++ b/test/T850-git.sh
> @@ -2,7 +2,7 @@
> test_description='"notmuch git" to save and restore tags'
> . $(dirname "$0")/test-lib.sh || exit 1
>
> -if [ $NOTMUCH_HAVE_SFSEXP -ne 1 ]; then
> +if [ "${NOTMUCH_HAVE_SFSEXP}" != "1" ]; then
here ensuring NOTMUCH_HAVE_SFSEXP is defined is missing
> printf "Skipping due to missing sfsexp library\n"
> test_done
> fi
> --
> 2.40.1
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [PATCH v4 2/5] test: check for empty/missing files in test_expect_equal_message_body
2023-07-09 14:08 ` [PATCH v4 2/5] test: check for empty/missing files in test_expect_equal_message_body David Bremner
@ 2023-07-11 15:14 ` Tomi Ollila
0 siblings, 0 replies; 13+ messages in thread
From: Tomi Ollila @ 2023-07-11 15:14 UTC (permalink / raw)
To: David Bremner, notmuch
On Sun, Jul 09 2023, David Bremner wrote:
> Messages can have empty bodies, but empty files are not messages.
> ---
> test/test-lib.sh | 8 ++++++++
> 1 file changed, 8 insertions(+)
>
> diff --git a/test/test-lib.sh b/test/test-lib.sh
> index 1a6525df..b5aa94dd 100644
> --- a/test/test-lib.sh
> +++ b/test/test-lib.sh
> @@ -388,6 +388,14 @@ test_expect_equal_message_body () {
> test "$#" = 2 ||
> error "bug in the test script: not 2 parameters to test_expect_equal_file"
>
> + for file in "$1" "$2"; do
> + if [ ! -s "$file" ]; then
> + test_failure_ "Missing or zero length file: $file"
> + inside_subtest=
the above line will set $? = 0
sh -c 'false; x=; echo $?' echoes 0
> + return $?
> + fi
> + done
> +
> expected=$(sed '1,/^$/d' "$1")
> output=$(sed '1,/^$/d' "$2")
> test_expect_equal "$expected" "$output"
> --
> 2.40.1
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [PATCH v4 3/5] test: Guess a value for NOTMUCH_PYTHON
2023-07-09 14:08 ` [PATCH v4 3/5] test: Guess a value for NOTMUCH_PYTHON David Bremner
@ 2023-07-11 15:18 ` Tomi Ollila
0 siblings, 0 replies; 13+ messages in thread
From: Tomi Ollila @ 2023-07-11 15:18 UTC (permalink / raw)
To: David Bremner, notmuch
On Sun, Jul 09 2023, David Bremner wrote:
> python3 will work for many people, and reduce the friction to running
> the tests without running configure first.
> ---
> test/test-lib-common.sh | 4 ++++
> 1 file changed, 4 insertions(+)
>
> diff --git a/test/test-lib-common.sh b/test/test-lib-common.sh
> index 18fa29c0..4d14e0b3 100644
> --- a/test/test-lib-common.sh
> +++ b/test/test-lib-common.sh
> @@ -323,6 +323,10 @@ rm -rf "$TMP_DIRECTORY" || {
> exit 1
> }
>
> +# Provide a guess at a usable Python, to support running tests without
> +# running configure first.
> +NOTMUCH_PYTHON=${NOTMUCH_PYTHON-python3}
one thing to consider -- in all cases (the above is probably consistent),
whether to use format
NOTMUCH_PYTHON=${NOTMUCH_PYTHON:-python3}
so that in case where the variable in question is defined, but empty,
to set the value to the given alternative (python3 in the case above)
it would be more robust in cases where empty value makes no sense.
(in any case this change LGTM.)
Tomi
> +
> # A temporary home directory is needed by at least:
> # - emacs/"Sending a message via (fake) SMTP"
> # - emacs/"Reply within emacs"
> --
> 2.40.1
>
> _______________________________________________
> notmuch mailing list -- notmuch@notmuchmail.org
> To unsubscribe send an email to notmuch-leave@notmuchmail.org
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [PATCH v4 4/5] test: support testing notmuch as installed
2023-07-09 14:08 ` [PATCH v4 4/5] test: support testing notmuch as installed David Bremner
@ 2023-07-11 15:39 ` Tomi Ollila
2023-07-21 10:38 ` David Bremner
2023-07-21 10:43 ` David Bremner
0 siblings, 2 replies; 13+ messages in thread
From: Tomi Ollila @ 2023-07-11 15:39 UTC (permalink / raw)
To: David Bremner, notmuch
On Sun, Jul 09 2023, David Bremner wrote:
Note: I almost commented cases where it was not ensured that variable
is defined, to notice later that those were actually defined for sure
-- so in some previous mails in this series I may have missed
same cases.
The comments for this message may be more accurate ;D
Tomi
> We put some effort into testing the built copy rather than some
> installed copy. On the other hand for people like packagers, testing
> the installed copy is also of interest.
>
> When NOTMUCH_TEST_INSTALLED is set to a nonempty value, tests do not
> require a built notmuch tree or running configure.
>
> Some of the tests marked as broken when running against installed
> notmuch are probably fixable.
> ---
> test/README | 17 +++++++++++++++++
> test/T000-basic.sh | 1 +
> test/T010-help-test.sh | 3 +++
> test/T160-json.sh | 2 ++
> test/T240-dump-restore.sh | 2 ++
> test/T310-emacs.sh | 2 ++
> test/T350-crypto.sh | 1 +
> test/T355-smime.sh | 2 ++
> test/T360-symbol-hiding.sh | 4 ++++
> test/T390-python.sh | 4 ++++
> test/T391-python-cffi.sh | 4 ++++
> test/T395-ruby.sh | 8 ++++++--
> test/T410-argument-parsing.sh | 4 ++++
> test/T480-hex-escaping.sh | 4 ++++
> test/T490-parse-time-string.sh | 4 ++++
> test/T550-db-features.sh | 4 ++++
> test/T566-lib-message.sh | 4 ++++
> test/T592-thread-breakage.sh | 4 ++++
> test/T710-message-id.sh | 4 ++++
> test/T800-asan.sh | 5 +++++
> test/T850-git.sh | 6 ++++++
> test/export-dirs.sh | 2 +-
> test/notmuch-test | 12 ++++++++++--
> test/test-lib-common.sh | 13 ++++++++++---
> test/test-lib-emacs.sh | 12 ++++++++++--
> test/test-lib.sh | 27 +++++++++++++++++++++------
> 26 files changed, 139 insertions(+), 16 deletions(-)
>
> diff --git a/test/README b/test/README
> index 10f127cb..a81808b1 100644
> --- a/test/README
> +++ b/test/README
> @@ -137,6 +137,23 @@ detection of missing prerequisites. In the future we may treat tests
> unable to run because of missing prerequisites, but not explicitly
> skipped by the user, as failures.
>
> +Testing installed notmuch
> +-------------------------
> +
> +Systems integrators (e.g. Linux distros) may wish to test an installed
> +version of notmuch. This can be done be running
> +
> + $ NOTMUCH_TEST_INSTALLED=1 ./test/notmuch-test
> +
> +In this scenario the test suite does not assume a built tree, and in
> +particular cannot rely on the output of 'configure'. You may want to
> +set certain feature environment variables ('NOTMUCH_HAVE_*') directly
> +if you know those apply to your installed notmuch). Consider also
> +setting TERM=dumb if the value of TERM cannot be used (e.g. in a
> +chroot with missing terminfo). Note that having a built tree may cause
> +surprising/broken results for NOTMUCH_TEST_INSTALLED, so consider
> +cleaning first.
> +
> Writing Tests
> -------------
> The test script is written as a shell script. It is to be named as
> diff --git a/test/T000-basic.sh b/test/T000-basic.sh
> index a2f4d93f..642f918d 100755
> --- a/test/T000-basic.sh
> +++ b/test/T000-basic.sh
> @@ -66,6 +66,7 @@ test_begin_subtest 'NOTMUCH_CONFIG is set and points to an existing file'
> test_expect_success 'test -f "${NOTMUCH_CONFIG}"'
>
> test_begin_subtest 'PATH is set to build directory'
> +test_subtest_broken_for_installed
> test_expect_equal \
> "$(dirname ${TEST_DIRECTORY})" \
> "$(echo $PATH|cut -f1 -d: | sed -e 's,/test/valgrind/bin$,,')"
> diff --git a/test/T010-help-test.sh b/test/T010-help-test.sh
> index 1fbc46a2..827edc14 100755
> --- a/test/T010-help-test.sh
> +++ b/test/T010-help-test.sh
> @@ -19,6 +19,9 @@ if [ "${NOTMUCH_HAVE_MAN-0}" = "1" ]; then
> test_begin_subtest 'notmuch help tag'
> test_expect_success 'notmuch help tag'
> else
> + if [ -n "${NOTMUCH_TEST_INSTALLED-}" ]; then
> + test_done
> + fi
> test_begin_subtest 'notmuch --help tag (man pages not available)'
> test_expect_success 'test_must_fail notmuch --help tag >/dev/null'
>
> diff --git a/test/T160-json.sh b/test/T160-json.sh
> index f61c3f2d..318c9788 100755
> --- a/test/T160-json.sh
> +++ b/test/T160-json.sh
> @@ -65,6 +65,7 @@ test_expect_equal_json "$output" "[{\"thread\": \"XXX\",
> \"tags\": [\"inbox\",
> \"unread\"]}]"
>
> +if [ -z "${NOTMUCH_TEST_INSTALLED-}" ]; then
> test_begin_subtest "Search message: json, 64-bit timestamp"
> if [ "${NOTMUCH_HAVE_64BIT_TIME_T-0}" != "1" ]; then
> test_subtest_known_broken
> @@ -81,6 +82,7 @@ test_expect_equal_json "$output" "[{\"thread\": \"XXX\",
> \"query\": [\"id:$gen_msg_id\", null],
> \"tags\": [\"inbox\",
> \"unread\"]}]"
> +fi # NOTMUCH_TEST_INSTALLED undefined / empty
>
> test_begin_subtest "Format version: too low"
> test_expect_code 20 "notmuch search --format-version=0 \\*"
> diff --git a/test/T240-dump-restore.sh b/test/T240-dump-restore.sh
> index 80939030..c3f18839 100755
> --- a/test/T240-dump-restore.sh
> +++ b/test/T240-dump-restore.sh
> @@ -139,6 +139,7 @@ notmuch dump --output=dump-outfile-dash-inbox.actual -- from:cworth
> test_expect_equal_file dump-cworth.expected dump-outfile-dash-inbox.actual
>
> test_begin_subtest "Check for a safe set of message-ids"
> +test_subtest_broken_for_installed
> notmuch search --output=messages from:cworth | sed s/^id:// > EXPECTED
> notmuch search --output=messages from:cworth | sed s/^id:// |\
> $TEST_DIRECTORY/hex-xcode --direction=encode > OUTPUT
> @@ -246,6 +247,7 @@ notmuch dump --format=batch-tag > OUTPUT.$test_count
> test_expect_equal_file EXPECTED.$test_count OUTPUT.$test_count
>
> test_begin_subtest 'format=batch-tag, checking encoded output'
> +test_subtest_broken_for_installed
> NOTMUCH_DUMP_TAGS --format=batch-tag -- from:cworth |\
> awk "{ print \"+$enc1 +$enc2 +$enc3 -- \" \$5 }" > EXPECTED.$test_count
> NOTMUCH_DUMP_TAGS --format=batch-tag -- from:cworth > OUTPUT.$test_count
> diff --git a/test/T310-emacs.sh b/test/T310-emacs.sh
> index 9d0df187..d3aa2e7d 100755
> --- a/test/T310-emacs.sh
> +++ b/test/T310-emacs.sh
> @@ -258,6 +258,7 @@ EOF
> test_expect_equal_file EXPECTED OUTPUT
>
> test_begin_subtest "Verify that sent messages are saved/searchable (via FCC)"
> +test_subtest_broken_for_installed
> notmuch new > /dev/null
> output=$(notmuch search 'subject:"testing message sent via SMTP"' | notmuch_search_sanitize)
> test_expect_equal "$output" "thread:XXX 2000-01-01 [1/1] Notmuch Test Suite; Testing message sent via SMTP (inbox)"
> @@ -350,6 +351,7 @@ EOF
> test_expect_equal_file EXPECTED OUTPUT
>
> test_begin_subtest "Reply within emacs"
> +test_subtest_broken_for_installed
> test_emacs '(let ((message-hidden-headers ''()))
> (notmuch-search "subject:\"testing message sent via SMTP\"")
> (notmuch-test-wait)
> diff --git a/test/T350-crypto.sh b/test/T350-crypto.sh
> index 721cbfdd..27c0e86d 100755
> --- a/test/T350-crypto.sh
> +++ b/test/T350-crypto.sh
> @@ -30,6 +30,7 @@ msg_file=$(notmuch search --output=files subject:signed-message-sent-via-SMTP)
> test_expect_equal_message_body sent_message "$msg_file"
>
> test_begin_subtest "signed part content-type indexing"
> +test_subtest_broken_for_installed
> notmuch search mimetype:multipart/signed and mimetype:application/pgp-signature | notmuch_search_sanitize > OUTPUT
> cat <<EOF >EXPECTED
> thread:XXX 2000-01-01 [1/1] Notmuch Test Suite; test signed message 001 (inbox signed)
> diff --git a/test/T355-smime.sh b/test/T355-smime.sh
> index 3bd5a17e..d2118c04 100755
> --- a/test/T355-smime.sh
> +++ b/test/T355-smime.sh
> @@ -183,6 +183,7 @@ test_begin_subtest "show PKCS#7 SignedData outputs valid JSON"
> output=$(notmuch show --format=json id:smime-onepart-signed@protected-headers.example)
> test_valid_json "$output"
>
> +if [ -z "${NOTMUCH_TEST_INSTALLED-}" ]; then
> test_begin_subtest "Verify signature on PKCS#7 SignedData message"
> if [ "${NOTMUCH_HAVE_64BIT_TIME_T-0}" != "1" ]; then
> test_subtest_known_broken
> @@ -194,6 +195,7 @@ test_json_nodes <<<"$output" \
> 'expires:[0][0][0]["crypto"]["signed"]["status"][0]["expires"]=2611032858' \
> 'fingerprint:[0][0][0]["crypto"]["signed"]["status"][0]["fingerprint"]="702BA4B157F1E2B7D16B0C6A5FFC8A7DE2057DEB"' \
> 'status:[0][0][0]["crypto"]["signed"]["status"][0]["status"]="good"'
> +fi # NOTMUCH_TEST_INSTALLED undefined / empty
>
> test_begin_subtest "Verify signature on PKCS#7 SignedData message signer User ID"
> if [ $NOTMUCH_GMIME_X509_CERT_VALIDITY -ne 1 ]; then
> diff --git a/test/T360-symbol-hiding.sh b/test/T360-symbol-hiding.sh
> index 642457bf..ff06ff69 100755
> --- a/test/T360-symbol-hiding.sh
> +++ b/test/T360-symbol-hiding.sh
> @@ -11,6 +11,10 @@ test_description='exception symbol hiding'
>
> . $(dirname "$0")/test-lib.sh || exit 1
>
> +if [ -n "${NOTMUCH_TEST_INSTALLED-}" ]; then
> + test_done
> +fi
> +
> test_begin_subtest 'running test' run_test
> mkdir -p ${PWD}/fakedb/.notmuch
> $TEST_DIRECTORY/symbol-test ${PWD}/fakedb ${PWD}/nonexistent 2>&1 \
> diff --git a/test/T390-python.sh b/test/T390-python.sh
> index 9f71ce3c..21912431 100755
> --- a/test/T390-python.sh
> +++ b/test/T390-python.sh
> @@ -4,6 +4,10 @@ test_description="python bindings"
>
> test_require_external_prereq ${NOTMUCH_PYTHON}
>
> +if [ -n "${NOTMUCH_TEST_INSTALLED-}" ]; then
> + test_done
> +fi
> +
> add_email_corpus
> add_gnupg_home
>
> diff --git a/test/T391-python-cffi.sh b/test/T391-python-cffi.sh
> index 0ef9e0d3..0059b050 100755
> --- a/test/T391-python-cffi.sh
> +++ b/test/T391-python-cffi.sh
> @@ -2,6 +2,10 @@
> test_description="python bindings (pytest)"
> . $(dirname "$0")/test-lib.sh || exit 1
>
> +if [ -n "${NOTMUCH_TEST_INSTALLED-}" ]; then
> + test_done
> +fi
> +
> if [ "${NOTMUCH_HAVE_PYTHON3_CFFI-0}" = "0" -o "${NOTMUCH_HAVE_PYTHON3_PYTEST-0}" = "0" ]; then
> test_done
> fi
> diff --git a/test/T395-ruby.sh b/test/T395-ruby.sh
> index c066c842..ea9977f8 100755
> --- a/test/T395-ruby.sh
> +++ b/test/T395-ruby.sh
> @@ -2,7 +2,7 @@
> test_description="ruby bindings"
> . $(dirname "$0")/test-lib.sh || exit 1
>
> -if [ "${NOTMUCH_HAVE_RUBY_DEV}" = "0" ]; then
> +if [ -z "${NOTMUCH_TEST_INSTALLED-}" -a "${NOTMUCH_HAVE_RUBY_DEV}" = "0" ]; then
the case where NOTMUCH_HAVE_RUBY_DEV not being defined does not cause a problem?
-- or is it always defined when getting this far (i.e. test-lib.sh defining it)
> test_subtest_missing_external_prereq_["ruby development files"]=t
> fi
>
> @@ -15,7 +15,11 @@ test_ruby() {
> db = Notmuch::Database.new()
> EOF
> cat
> - ) | $NOTMUCH_RUBY -I "$NOTMUCH_BUILDDIR/bindings/ruby"> OUTPUT
> + ) | if [ -n "${NOTMUCH_TEST_INSTALLED-}" ]; then
> + ruby
> + else
> + $NOTMUCH_RUBY -I "$NOTMUCH_BUILDDIR/bindings/ruby"
> + fi> OUTPUT
> test_expect_equal_file EXPECTED OUTPUT
> }
>
> diff --git a/test/T410-argument-parsing.sh b/test/T410-argument-parsing.sh
> index d9aa8e2d..40b625fe 100755
> --- a/test/T410-argument-parsing.sh
> +++ b/test/T410-argument-parsing.sh
> @@ -2,6 +2,10 @@
> test_description="argument parsing"
> . $(dirname "$0")/test-lib.sh || exit 1
>
> +if [ -n "${NOTMUCH_TEST_INSTALLED-}" ]; then
> + test_done
> +fi
> +
> test_begin_subtest "sanity check"
> $TEST_DIRECTORY/arg-test pos1 --keyword=one --boolean --string=foo pos2 --int=7 --flag=one --flag=three > OUTPUT
> cat <<EOF > EXPECTED
> diff --git a/test/T480-hex-escaping.sh b/test/T480-hex-escaping.sh
> index b2eb80b9..8bddf3e7 100755
> --- a/test/T480-hex-escaping.sh
> +++ b/test/T480-hex-escaping.sh
> @@ -2,6 +2,10 @@
> test_description="hex encoding and decoding"
> . $(dirname "$0")/test-lib.sh || exit 1
>
> +if [ -n "${NOTMUCH_TEST_INSTALLED-}" ]; then
> + test_done
> +fi
> +
> test_begin_subtest "round trip"
> find $NOTMUCH_SRCDIR/test/corpora/default -type f -print | sort | xargs cat > EXPECTED
> $TEST_DIRECTORY/hex-xcode --direction=encode < EXPECTED | $TEST_DIRECTORY/hex-xcode --direction=decode > OUTPUT
> diff --git a/test/T490-parse-time-string.sh b/test/T490-parse-time-string.sh
> index f89755ed..3b6e48c4 100755
> --- a/test/T490-parse-time-string.sh
> +++ b/test/T490-parse-time-string.sh
> @@ -2,6 +2,10 @@
> test_description="date/time parser module"
> . $(dirname "$0")/test-lib.sh || exit 1
>
> +if [ -n "${NOTMUCH_TEST_INSTALLED-}" ]; then
> + test_done
> +fi
> +
> # Sanity/smoke tests for the date/time parser independent of notmuch
>
> _date () {
> diff --git a/test/T550-db-features.sh b/test/T550-db-features.sh
> index 9d5a9e70..3048c7c4 100755
> --- a/test/T550-db-features.sh
> +++ b/test/T550-db-features.sh
> @@ -3,6 +3,10 @@ test_description="database version and feature compatibility"
>
> . $(dirname "$0")/test-lib.sh || exit 1
>
> +if [ -n "${NOTMUCH_TEST_INSTALLED-}" ]; then
> + test_done
> +fi
> +
> test_begin_subtest "future database versions abort open"
> ${TEST_DIRECTORY}/make-db-version ${MAIL_DIR} 9999 ""
> output=$(notmuch search x 2>&1 | sed 's/\(database at\) .*/\1 FILENAME/')
> diff --git a/test/T566-lib-message.sh b/test/T566-lib-message.sh
> index 7f0e8eb0..69051937 100755
> --- a/test/T566-lib-message.sh
> +++ b/test/T566-lib-message.sh
> @@ -3,6 +3,10 @@ test_description="API tests for notmuch_message_*"
>
> . $(dirname "$0")/test-lib.sh || exit 1
>
> +if [ -n "${NOTMUCH_TEST_INSTALLED}" ]; then
> + test_done
> +fi
> +
> add_email_corpus
>
> test_begin_subtest "building database"
> diff --git a/test/T592-thread-breakage.sh b/test/T592-thread-breakage.sh
> index 92a70e3e..2334fcaf 100755
> --- a/test/T592-thread-breakage.sh
> +++ b/test/T592-thread-breakage.sh
> @@ -21,6 +21,10 @@ test_description='thread breakage during reindexing'
>
> . $(dirname "$0")/test-lib.sh || exit 1
>
> +if [ -n "${NOTMUCH_TEST_INSTALLED-}" ]; then
> + test_done
> +fi
> +
> message_a () {
> mkdir -p ${MAIL_DIR}/cur
> cat > ${MAIL_DIR}/cur/a <<EOF
> diff --git a/test/T710-message-id.sh b/test/T710-message-id.sh
> index 5129d84c..a2d8ec71 100755
> --- a/test/T710-message-id.sh
> +++ b/test/T710-message-id.sh
> @@ -3,6 +3,10 @@ test_description="message id parsing"
>
> . $(dirname "$0")/test-lib.sh || exit 1
>
> +if [ -n "${NOTMUCH_TEST_INSTALLED-}" ]; then
> + test_done
> +fi
> +
> test_begin_subtest "good message ids"
> ${TEST_DIRECTORY}/message-id-parse <<EOF >OUTPUT
> <018b1a8f2d1df62e804ce88b65401304832dfbbf.1346614915.git.jani@nikula.org>
> diff --git a/test/T800-asan.sh b/test/T800-asan.sh
> index 7c28dc7c..3a26b77e 100755
> --- a/test/T800-asan.sh
> +++ b/test/T800-asan.sh
> @@ -7,6 +7,11 @@ if [ "${NOTMUCH_HAVE_ASAN-0}" != "1" ]; then
> test_done
> fi
>
> +if [ -n "${LD_PRELOAD-}" ]; then
Is it problem to chain LD_PRELOADs here ? I see later in this message
there is LD_PRELOAD's concatenated with :'s (in case non-empty) ?
> + printf "Skipping due to non-empty LD_PRELOAD\n"
> + test_done
> +fi
> +
> add_email_corpus
>
> TEST_CFLAGS="${TEST_CFLAGS:-} -fsanitize=address"
> diff --git a/test/T850-git.sh b/test/T850-git.sh
> index a063aacf..47314a1b 100755
> --- a/test/T850-git.sh
> +++ b/test/T850-git.sh
> @@ -233,6 +233,7 @@ EOF
> test_expect_equal_file EXPECTED OUTPUT
>
> test_begin_subtest "invoke as nmbug sets defaults"
> +test_subtest_broken_for_installed
> "$NOTMUCH_BUILDDIR"/nmbug -ldebug status |& grep '^\(prefix\|repository\)' | notmuch_dir_sanitize > OUTPUT
> cat <<EOF > EXPECTED
> prefix = notmuch::
> @@ -241,6 +242,7 @@ EOF
> test_expect_equal_file EXPECTED OUTPUT
>
> test_begin_subtest "env variable NOTMUCH_GIT_DIR works when invoked as nmbug"
> +test_subtest_broken_for_installed
> NOTMUCH_GIT_DIR=`pwd`/foo "$NOTMUCH_BUILDDIR"/nmbug -ldebug status |& grep '^repository' | notmuch_dir_sanitize > OUTPUT
> cat <<EOF > EXPECTED
> repository = CWD/foo
> @@ -256,6 +258,7 @@ test_expect_equal_file EXPECTED OUTPUT
>
>
> test_begin_subtest "env variable NOTMUCH_GIT_DIR overrides config when invoked as 'nmbug'"
> +test_subtest_broken_for_installed
> notmuch config set git.path `pwd`/bar
> NOTMUCH_GIT_DIR=`pwd`/remote.git "$NOTMUCH_BUILDDIR"/nmbug -ldebug status |& grep '^repository' | notmuch_dir_sanitize > OUTPUT
> notmuch config set git.path
> @@ -274,6 +277,7 @@ EOF
> test_expect_equal_file EXPECTED OUTPUT
>
> test_begin_subtest "env variable NOTMUCH_GIT_PREFIX works when invoked as 'nmbug'"
> +test_subtest_broken_for_installed
> NOTMUCH_GIT_PREFIX=env:: "$NOTMUCH_BUILDDIR"/nmbug -ldebug status |& grep '^prefix' | notmuch_dir_sanitize > OUTPUT
> cat <<EOF > EXPECTED
> prefix = env::
> @@ -281,6 +285,7 @@ EOF
> test_expect_equal_file EXPECTED OUTPUT
>
> test_begin_subtest "env variable NOTMUCH_GIT_PREFIX works when invoked as nmbug"
> +test_subtest_broken_for_installed
> NOTMUCH_GIT_PREFIX=foo:: "$NOTMUCH_BUILDDIR"/nmbug -ldebug status |& grep '^prefix' | notmuch_dir_sanitize > OUTPUT
> cat <<EOF > EXPECTED
> prefix = foo::
> @@ -288,6 +293,7 @@ EOF
> test_expect_equal_file EXPECTED OUTPUT
>
> test_begin_subtest "env variable NOTMUCH_GIT_PREFIX overrides config when invoked as 'nmbug'"
> +test_subtest_broken_for_installed
> notmuch config set git.tag_prefix config::
> NOTMUCH_GIT_PREFIX=env:: "$NOTMUCH_BUILDDIR"/nmbug -ldebug status |& grep '^prefix' | notmuch_dir_sanitize > OUTPUT
> notmuch config set git.path
> diff --git a/test/export-dirs.sh b/test/export-dirs.sh
> index 0a048e1f..f0374387 100644
> --- a/test/export-dirs.sh
> +++ b/test/export-dirs.sh
> @@ -24,7 +24,7 @@ find_builddir () {
> if [[ -z "${NOTMUCH_BUILDDIR}" ]]; then
> export NOTMUCH_BUILDDIR="$(find_builddir "$(pwd)")"
>
> - if [[ -z "${NOTMUCH_BUILDDIR}" ]]; then
> + if [ -z "${NOTMUCH_BUILDDIR}" -a "${NOTMUCH_TEST_INSTALLED-0}" = "0" ]; then
2 spaces before -a
> echo "Run tests in a subdir of built notmuch tree." >&2
> exit 1
> fi
> diff --git a/test/notmuch-test b/test/notmuch-test
> index 14747bdb..5d27e4d1 100755
> --- a/test/notmuch-test
> +++ b/test/notmuch-test
> @@ -20,6 +20,14 @@ fi
>
> set -eu
>
> +# Where to run the tests
> +# XXX FIXME this code is duplicated with test-lib.sh
> +if [[ -n "${NOTMUCH_BUILDDIR}" ]]; then
> + TEST_DIRECTORY=$NOTMUCH_BUILDDIR/test
> +else
> + TEST_DIRECTORY=$NOTMUCH_SRCDIR/test
> +fi
> +
> TESTS=
> for test in ${NOTMUCH_TESTS-}; do
> TESTS="$TESTS $NOTMUCH_SRCDIR/test/$test"
> @@ -80,7 +88,7 @@ for file in $TESTS
> do
> file=${file##*/} # drop leading path components
> file=${file%.sh} # drop trailing '.sh'
> - RESULT_FILES="$RESULT_FILES $NOTMUCH_BUILDDIR/test/test-results/$file"
> + RESULT_FILES="$RESULT_FILES $TEST_DIRECTORY/test-results/$file"
> done
>
> echo
> @@ -94,6 +102,6 @@ if [ -n "$META_FAILURE" ]; then
> fi
>
> # Clean up
> -rm -rf $NOTMUCH_BUILDDIR/test/test-results
> +rm -rf $TEST_DIRECTORY/test-results
>
> exit $ev
> diff --git a/test/test-lib-common.sh b/test/test-lib-common.sh
> index 4d14e0b3..f5d72e12 100644
> --- a/test/test-lib-common.sh
> +++ b/test/test-lib-common.sh
> @@ -24,7 +24,7 @@
> #
> type die >/dev/null 2>&1 || die () { echo "$@" >&2; exit 1; }
>
> -if [[ -z "$NOTMUCH_SRCDIR" ]] || [[ -z "$NOTMUCH_BUILDDIR" ]]; then
> +if [[ -z "$NOTMUCH_SRCDIR" ]] || [ -z "${NOTMUCH_TEST_INSTALLED-}" -a -z "$NOTMUCH_BUILDDIR" ]; then
> echo "internal: srcdir or builddir not set" >&2
> exit 1
> fi
> @@ -61,7 +61,9 @@ LD_LIBRARY_PATH=${TEST_DIRECTORY%/*}/lib${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}
> export LD_LIBRARY_PATH
>
> # configure output
> -. "$NOTMUCH_BUILDDIR/sh.config" || exit 1
> +if [ -z "${NOTMUCH_TEST_INSTALLED-}" ]; then
> + . "$NOTMUCH_BUILDDIR/sh.config" || exit 1
> +fi
>
> # load OS specifics
> if [[ -e "$NOTMUCH_SRCDIR/test/test-lib-$PLATFORM.sh" ]]; then
> @@ -315,7 +317,12 @@ export PATH MANPATH
>
> # Test repository
> test="tmp.$(basename "$0" .sh)"
> -TMP_DIRECTORY="$TEST_DIRECTORY/$test"
> +if [ -z "${NOTMUCH_TEST_INSTALLED-}" ]; then
> + TMP_DIRECTORY="$TEST_DIRECTORY/$test"
> +else
> + TMP_DIRECTORY=$(mktemp -d "${TMPDIR:-/tmp}/notmuch-$test.XXXXXX")
> +fi
> +
> test ! -z "$debug" || remove_tmp=$TMP_DIRECTORY
> rm -rf "$TMP_DIRECTORY" || {
> GIT_EXIT_OK=t
> diff --git a/test/test-lib-emacs.sh b/test/test-lib-emacs.sh
> index ad4c4aeb..0ab58fc2 100644
> --- a/test/test-lib-emacs.sh
> +++ b/test/test-lib-emacs.sh
> @@ -30,6 +30,7 @@ test_require_emacs () {
> # to the message and encrypting/signing.
> emacs_deliver_message () {
> local subject body smtp_dummy_pid smtp_dummy_port
> + test_subtest_broken_for_installed
> subject="$1"
> body="$2"
> shift 2
> @@ -144,6 +145,13 @@ emacs_generate_script () {
> # Construct a little test script here for the benefit of the user,
> # (who can easily run "run_emacs" to get the same emacs environment
> # for investigating any failures).
> + if [ -z "${NOTMUCH_TEST_INSTALLED-}" ]; then
> + find_notmuch_el='--directory "$NOTMUCH_BUILDDIR/emacs"'
> + else
> + ### XXX FIXME: this should really use the installed emacs lisp files
> + find_notmuch_el='--directory "$NOTMUCH_SRCDIR/emacs"'
> + fi
> +
> cat <<EOF >"$TMP_DIRECTORY/run_emacs"
> #!/bin/sh
> export PATH=$PATH
> @@ -158,8 +166,8 @@ export NOTMUCH_CONFIG=$NOTMUCH_CONFIG
> #
> # --load Force loading of notmuch.el and test-lib.el
>
> -exec ${TEST_EMACS} --quick \
> - --directory "$NOTMUCH_BUILDDIR/emacs" --load notmuch.el \
> +exec ${TEST_EMACS} ${find_notmuch_el} --quick \
> + ${EXTRA_DIR} --load notmuch.el \
> --directory "$NOTMUCH_SRCDIR/test" --load test-lib.el \
> "\$@"
> EOF
> diff --git a/test/test-lib.sh b/test/test-lib.sh
> index b5aa94dd..8452fba2 100644
> --- a/test/test-lib.sh
> +++ b/test/test-lib.sh
> @@ -29,8 +29,8 @@ shopt -u xpg_echo
> # Ensure NOTMUCH_SRCDIR and NOTMUCH_BUILDDIR are set.
> . $(dirname "$0")/export-dirs.sh || exit 1
>
> -# It appears that people try to run tests without building...
> -if [[ ! -x "$NOTMUCH_BUILDDIR/notmuch" ]]; then
> +# We need either a built tree, or a promise of an installed notmuch
> +if [ -z "${NOTMUCH_TEST_INSTALLED-}" -a ! -x "$NOTMUCH_BUILDDIR/notmuch" ]; then
2 spaces after -x
> echo >&2 'You do not seem to have built notmuch yet.'
> exit 1
> fi
> @@ -748,6 +748,12 @@ test_subtest_known_broken () {
> test_subtest_known_broken_=t
> }
>
> +test_subtest_broken_for_installed () {
> + if [ -n "${NOTMUCH_TEST_INSTALLED-}" ]; then
> + test_subtest_known_broken_=t
> + fi
> +}
> +
> test_subtest_broken_for_root () {
> if [ "$EUID" = "0" ]; then
> test_subtest_known_broken_=t
> @@ -929,11 +935,16 @@ make_shim () {
> }
>
> notmuch_with_shim () {
> - local base_name shim_file
> - base_name="$1"
> + local base_name shim_file notmuch_cmd
> + if [ -n "${NOTMUCH_TEST_INSTALLED-}" ]; then
> + notmuch_cmd="notmuch"
> + else
> + notmuch_cmd="notmuch-shared"
> + fi
> + base_name=$1
> shift
> shim_file="${base_name}.so"
> - LD_PRELOAD=${LD_PRELOAD:+:$LD_PRELOAD}:./${shim_file} notmuch-shared "$@"
> + LD_PRELOAD=${LD_PRELOAD:+:$LD_PRELOAD}:./${shim_file} $notmuch_cmd "$@"
hmm, should the "chaining" be done on other order (i.e. the shim_file
before anything if there... (that could make the case where there is
value in LD_PRELOAD to work (commented above))
> }
>
> # Creates a script that counts how much time it is executed and calls
> @@ -985,7 +996,11 @@ test_init_ () {
>
>
> # Where to run the tests
> -TEST_DIRECTORY=$NOTMUCH_BUILDDIR/test
> +if [[ -n "${NOTMUCH_BUILDDIR}" ]]; then
> + TEST_DIRECTORY=$NOTMUCH_BUILDDIR/test
> +else
> + TEST_DIRECTORY=$NOTMUCH_SRCDIR/test
> +fi
>
> . "$NOTMUCH_SRCDIR/test/test-lib-common.sh" || exit 1
>
> --
> 2.40.1
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [PATCH v4 4/5] test: support testing notmuch as installed
2023-07-11 15:39 ` Tomi Ollila
@ 2023-07-21 10:38 ` David Bremner
2023-07-21 10:43 ` David Bremner
1 sibling, 0 replies; 13+ messages in thread
From: David Bremner @ 2023-07-21 10:38 UTC (permalink / raw)
To: Tomi Ollila, notmuch
Tomi Ollila <tomi.ollila@iki.fi> writes:
>> fi
>>
>> +if [ -n "${LD_PRELOAD-}" ]; then
>
> Is it problem to chain LD_PRELOADs here ? I see later in this message
> there is LD_PRELOAD's concatenated with :'s (in case non-empty) ?
>
It is not a problem in general, but ASAN insists on being first in
LD_PRELOAD, so it's easier just to skip the tests for now. I updated
the printed message to
printf "Skipping due to ASAN LD_PRELOAD restrictions\n"
Since I also could not remember why that was there :P.
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [PATCH v4 4/5] test: support testing notmuch as installed
2023-07-11 15:39 ` Tomi Ollila
2023-07-21 10:38 ` David Bremner
@ 2023-07-21 10:43 ` David Bremner
1 sibling, 0 replies; 13+ messages in thread
From: David Bremner @ 2023-07-21 10:43 UTC (permalink / raw)
To: Tomi Ollila, notmuch
Tomi Ollila <tomi.ollila@iki.fi> writes:
>> - LD_PRELOAD=${LD_PRELOAD:+:$LD_PRELOAD}:./${shim_file} notmuch-shared "$@"
>> + LD_PRELOAD=${LD_PRELOAD:+:$LD_PRELOAD}:./${shim_file} $notmuch_cmd "$@"
>
> hmm, should the "chaining" be done on other order (i.e. the shim_file
> before anything if there... (that could make the case where there is
> value in LD_PRELOAD to work (commented above))
That seems like an orthogonal change, and as it turns out I don't think
it would directly help with ASAN.
d
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: v4 test notmuch as installed
2023-07-09 14:08 v4 test notmuch as installed David Bremner
` (4 preceding siblings ...)
2023-07-09 14:08 ` [PATCH v4 5/5] debian: add autopkgtests David Bremner
@ 2023-07-21 11:01 ` David Bremner
5 siblings, 0 replies; 13+ messages in thread
From: David Bremner @ 2023-07-21 11:01 UTC (permalink / raw)
To: notmuch
David Bremner <david@tethera.net> writes:
> This obsoletes the series at id:20230409142627.2216080-1-david@tethera.net.
>
I have applied this series (with some fixes from Tomi) to master.
d
^ permalink raw reply [flat|nested] 13+ messages in thread
end of thread, other threads:[~2023-07-22 11:41 UTC | newest]
Thread overview: 13+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2023-07-09 14:08 v4 test notmuch as installed David Bremner
2023-07-09 14:08 ` [PATCH v4 1/5] test: treat undefined feature variables as 0 David Bremner
2023-07-11 15:11 ` Tomi Ollila
2023-07-09 14:08 ` [PATCH v4 2/5] test: check for empty/missing files in test_expect_equal_message_body David Bremner
2023-07-11 15:14 ` Tomi Ollila
2023-07-09 14:08 ` [PATCH v4 3/5] test: Guess a value for NOTMUCH_PYTHON David Bremner
2023-07-11 15:18 ` Tomi Ollila
2023-07-09 14:08 ` [PATCH v4 4/5] test: support testing notmuch as installed David Bremner
2023-07-11 15:39 ` Tomi Ollila
2023-07-21 10:38 ` David Bremner
2023-07-21 10:43 ` David Bremner
2023-07-09 14:08 ` [PATCH v4 5/5] debian: add autopkgtests David Bremner
2023-07-21 11:01 ` v4 test notmuch as installed David Bremner
Code repositories for project(s) associated with this public inbox
https://yhetil.org/notmuch.git/
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).