be more honest about test suite failures due to old libraries

be more honest about test suite failures due to old libraries

[Daniel Kahn Gillmor]

We have several places where tests are skipped or marked as though
some test suite prereqs are missing, but in fact are due to building
against older versions of libraries that don't support certain
features.

This series tries to be more honest about some of those tests by
marking them as broken, rather than just skipping them.

The series still doesn't address T650-regexp-query.sh, which is
basically entirely skipped if Xapian doesn't have a field processor,
rather than marked as broken.  But it's still an improvement in
overall test suite honesty over the status quo.

[PATCH 1/3] test: searching by absolute date is broken without Xapian FieldProcessor

[Daniel Kahn Gillmor]

Previously, the test suite had simply silently skipped the absolute
date test if we're using an archaic version of Xapian.  For
correctness, we should instead mark the test as broken.

This also changes from string to numeric comparison when checking
NOMTUCH_HAVE_XAPIAN_FIELD_PROCESSOR for consistency with other tests.
---
 test/T500-search-date.sh | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)

diff --git a/test/T500-search-date.sh b/test/T500-search-date.sh
index 5c5b99a0..f84b0962 100755
--- a/test/T500-search-date.sh
+++ b/test/T500-search-date.sh
@@ -12,11 +12,12 @@ test_begin_subtest "Absolute date range with 'same' operator"
 output=$(notmuch search date:2010-12-16..! | notmuch_search_sanitize)
 test_expect_equal "$output" "thread:XXX   2010-12-16 [1/1] Olivier Berger; Essai accentué (inbox unread)"
 
-if [ "${NOTMUCH_HAVE_XAPIAN_FIELD_PROCESSOR}" = "1" ]; then
-    test_begin_subtest "Absolute date field"
-    output=$(notmuch search date:2010-12-16 | notmuch_search_sanitize)
-    test_expect_equal "$output" "thread:XXX   2010-12-16 [1/1] Olivier Berger; Essai accentué (inbox unread)"
+test_begin_subtest "Absolute date field"
+output=$(notmuch search date:2010-12-16 | notmuch_search_sanitize)
+if [ $NOTMUCH_HAVE_XAPIAN_FIELD_PROCESSOR -ne 1 ]; then
+    test_subtest_known_broken
 fi
+test_expect_equal "$output" "thread:XXX   2010-12-16 [1/1] Olivier Berger; Essai accentué (inbox unread)"
 
 test_begin_subtest "Absolute time range with TZ"
 notmuch search date:18-Nov-2009_02:19:26-0800..2009-11-18_04:49:52-06:00 | notmuch_search_sanitize > OUTPUT
-- 
2.15.0

[PATCH 2/3] test: named query tests are broken when missing Xapian FieldProcessor

[Daniel Kahn Gillmor]

Named queries don't work without Xapian FieldProcessor.  Rather than
silently skipping them, we should explictly mark them as broken when
building against an older version of Xapian.
---
 test/T600-named-queries.sh | 33 ++++++++++++++++++++-------------
 1 file changed, 20 insertions(+), 13 deletions(-)

diff --git a/test/T600-named-queries.sh b/test/T600-named-queries.sh
index 59496c3e..e38dc5bb 100755
--- a/test/T600-named-queries.sh
+++ b/test/T600-named-queries.sh
@@ -50,21 +50,28 @@ notmuch restore < BEFORE
 notmuch dump | grep '^#@' > OUTPUT
 test_expect_equal_file QUERIES.BEFORE OUTPUT
 
-if [ $NOTMUCH_HAVE_XAPIAN_FIELD_PROCESSOR -eq 1 ]; then
-    test_begin_subtest "search named query"
-    notmuch search query:test > OUTPUT
-    notmuch search $QUERYSTR > EXPECTED
-    test_expect_equal_file EXPECTED OUTPUT
+test_begin_subtest "search named query"
+notmuch search query:test > OUTPUT
+notmuch search $QUERYSTR > EXPECTED
+if [ $NOTMUCH_HAVE_XAPIAN_FIELD_PROCESSOR -ne 1 ]; then
+    test_subtest_known_broken
+fi
+test_expect_equal_file EXPECTED OUTPUT
 
-    test_begin_subtest "search named query with other terms"
-    notmuch search query:test and subject:Maildir > OUTPUT
-    notmuch search $QUERYSTR and subject:Maildir > EXPECTED
-    test_expect_equal_file EXPECTED OUTPUT
+test_begin_subtest "search named query with other terms"
+notmuch search query:test and subject:Maildir > OUTPUT
+notmuch search $QUERYSTR and subject:Maildir > EXPECTED
+if [ $NOTMUCH_HAVE_XAPIAN_FIELD_PROCESSOR -ne 1 ]; then
+    test_subtest_known_broken
+fi
+test_expect_equal_file EXPECTED OUTPUT
 
-    test_begin_subtest "search nested named query"
-    notmuch search query:test2 > OUTPUT
-    notmuch search $QUERYSTR2 > EXPECTED
-    test_expect_equal_file EXPECTED OUTPUT
+test_begin_subtest "search nested named query"
+notmuch search query:test2 > OUTPUT
+notmuch search $QUERYSTR2 > EXPECTED
+if [ $NOTMUCH_HAVE_XAPIAN_FIELD_PROCESSOR -ne 1 ]; then
+    test_subtest_known_broken
 fi
+test_expect_equal_file EXPECTED OUTPUT
 
 test_done
-- 
2.15.0

[PATCH 3/3] test: retrying lock tests are broken when built against older Xapian

[Daniel Kahn Gillmor]

If we're building against a version of Xapian that doesn't offer
retrying the lock, we should be honest and describe the tests as
broken, rather than marking them as missing a test prerequisite.

missing test prerequisites should be for specific components of the
test harness that are missing, not for the backend library notmuch
uses.
---
 test/T620-lock.sh | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)

diff --git a/test/T620-lock.sh b/test/T620-lock.sh
index 085ffe43..7aaaff2a 100755
--- a/test/T620-lock.sh
+++ b/test/T620-lock.sh
@@ -2,13 +2,12 @@
 test_description="locking"
 . $(dirname "$0")/test-lib.sh || exit 1
 
-if [ "${NOTMUCH_HAVE_XAPIAN_DB_RETRY_LOCK}" = "0" ]; then
-    test_subtest_missing_external_prereq_["lock retry support"]=t
-fi
-
 add_email_corpus
 
 test_begin_subtest "blocking open"
+if [ $NOTMUCH_HAVE_XAPIAN_DB_RETRY_LOCK -ne 1 ]; then
+    test_subtest_known_broken
+fi
 test_C ${MAIL_DIR} <<'EOF'
 #include <unistd.h>
 #include <stdlib.h>
@@ -70,6 +69,9 @@ inbox
 parent
 unread
 EOF
+if [ $NOTMUCH_HAVE_XAPIAN_DB_RETRY_LOCK -ne 1 ]; then
+    test_subtest_known_broken
+fi
 test_expect_equal_file EXPECTED OUTPUT
 
 test_done
-- 
2.15.0

Re: be more honest about test suite failures due to old libraries

[David Bremner]

Daniel Kahn Gillmor <dkg@fifthhorseman.net> writes:

> We have several places where tests are skipped or marked as though
> some test suite prereqs are missing, but in fact are due to building
> against older versions of libraries that don't support certain
> features.
>
> This series tries to be more honest about some of those tests by
> marking them as broken, rather than just skipping them.

pushed to master

d