unofficial mirror of notmuch@notmuchmail.org
 help / color / mirror / code / Atom feed
* Port to Xapian 1.5
@ 2020-06-17 11:04 David Bremner
  2020-06-17 11:04 ` [PATCH 1/3] lib: migrate to post Xapian 1.3.4 compact support David Bremner
                   ` (2 more replies)
  0 siblings, 3 replies; 5+ messages in thread
From: David Bremner @ 2020-06-17 11:04 UTC (permalink / raw)
  To: notmuch

"Port" is maybe overstating it a bit, but in order to build notmuch
with Xapian 1.5 (aka Xapian master), notmuch needs to migrate away
from some obsolete APIs, and not require use of chert databases in the
test suite.  With the three patches here, notmuch builds and the test
suite (minus the deleted upgrade tests) passes.

^ permalink raw reply	[flat|nested] 5+ messages in thread

* [PATCH 1/3] lib: migrate to post Xapian 1.3.4 compact support
  2020-06-17 11:04 Port to Xapian 1.5 David Bremner
@ 2020-06-17 11:04 ` David Bremner
  2020-06-17 11:04 ` [PATCH 2/3] lib: migrate from Xapian ValueRangeProcessor to RangeProcessor David Bremner
  2020-06-17 11:04 ` [PATCH 3/3] test: drop upgrade from v1 tests David Bremner
  2 siblings, 0 replies; 5+ messages in thread
From: David Bremner @ 2020-06-17 11:04 UTC (permalink / raw)
  To: notmuch

The old API was deprecated in Xapian 1.3.4 and (will be) removed in 1.5.0
---
 lib/database.cc | 6 +-----
 1 file changed, 1 insertion(+), 5 deletions(-)

diff --git a/lib/database.cc b/lib/database.cc
index 78b5fec9..43bfac4e 100644
--- a/lib/database.cc
+++ b/lib/database.cc
@@ -1281,11 +1281,7 @@ notmuch_database_compact (const char *path,
 
     try {
 	NotmuchCompactor compactor (status_cb, closure);
-
-	compactor.set_renumber (false);
-	compactor.add_source (xapian_path);
-	compactor.set_destdir (compact_xapian_path);
-	compactor.compact ();
+	notmuch->xapian_db->compact (compact_xapian_path, Xapian::DBCOMPACT_NO_RENUMBER, 0, compactor);
     } catch (const Xapian::Error &error) {
 	_notmuch_database_log (notmuch, "Error while compacting: %s\n", error.get_msg ().c_str ());
 	ret = NOTMUCH_STATUS_XAPIAN_EXCEPTION;
-- 
2.27.0

^ permalink raw reply related	[flat|nested] 5+ messages in thread

* [PATCH 2/3] lib: migrate from Xapian ValueRangeProcessor to RangeProcessor
  2020-06-17 11:04 Port to Xapian 1.5 David Bremner
  2020-06-17 11:04 ` [PATCH 1/3] lib: migrate to post Xapian 1.3.4 compact support David Bremner
@ 2020-06-17 11:04 ` David Bremner
  2020-06-17 11:04 ` [PATCH 3/3] test: drop upgrade from v1 tests David Bremner
  2 siblings, 0 replies; 5+ messages in thread
From: David Bremner @ 2020-06-17 11:04 UTC (permalink / raw)
  To: notmuch

This will be mandatory as of Xapian 1.5.  The API is also more
consistent with the FieldProcessor API, which helps code re-use a bit.

Note that this switches to using the built-in Xapian support for
prefixes on ranges (i.e. deleted code at beginning of
ParseTimeRangeProcessor::operator(), added prefix to constructor).

Another side effect of the migration is that we are generating smaller
queries, using one OP_VALUE_RANGE instead of an AND of two OP_VALUE_*
queries.
---
 lib/database-private.h |  6 ++--
 lib/database.cc        | 21 +++++++-------
 lib/parse-time-vrp.cc  | 62 +++++++++++++++++++++---------------------
 lib/parse-time-vrp.h   | 18 ++++++------
 4 files changed, 53 insertions(+), 54 deletions(-)

diff --git a/lib/database-private.h b/lib/database-private.h
index 87ae1bdf..76359007 100644
--- a/lib/database-private.h
+++ b/lib/database-private.h
@@ -218,9 +218,9 @@ struct _notmuch_database {
     unsigned long view;
     Xapian::QueryParser *query_parser;
     Xapian::TermGenerator *term_gen;
-    Xapian::ValueRangeProcessor *value_range_processor;
-    Xapian::ValueRangeProcessor *date_range_processor;
-    Xapian::ValueRangeProcessor *last_mod_range_processor;
+    Xapian::RangeProcessor *value_range_processor;
+    Xapian::RangeProcessor *date_range_processor;
+    Xapian::RangeProcessor *last_mod_range_processor;
 
     /* XXX it's slightly gross to use two parallel string->string maps
      * here, but at least they are small */
diff --git a/lib/database.cc b/lib/database.cc
index 43bfac4e..2f794164 100644
--- a/lib/database.cc
+++ b/lib/database.cc
@@ -385,8 +385,8 @@ _setup_query_field (const prefix_t *prefix, notmuch_database_t *notmuch)
 	Xapian::FieldProcessor *fp;
 
 	if (STRNCMP_LITERAL (prefix->name, "date") == 0)
-	    fp = (new DateFieldProcessor ())->release ();
-	else if (STRNCMP_LITERAL (prefix->name, "query") == 0)
+	    fp = (new DateFieldProcessor(NOTMUCH_VALUE_TIMESTAMP))->release ();
+	else if (STRNCMP_LITERAL(prefix->name, "query") == 0)
 	    fp = (new QueryFieldProcessor (*notmuch->query_parser, notmuch))->release ();
 	else if (STRNCMP_LITERAL (prefix->name, "thread") == 0)
 	    fp = (new ThreadFieldProcessor (*notmuch->query_parser, notmuch))->release ();
@@ -1036,17 +1036,16 @@ notmuch_database_open_verbose (const char *path,
 	notmuch->query_parser = new Xapian::QueryParser;
 	notmuch->term_gen = new Xapian::TermGenerator;
 	notmuch->term_gen->set_stemmer (Xapian::Stem ("english"));
-	notmuch->value_range_processor = new Xapian::NumberValueRangeProcessor (NOTMUCH_VALUE_TIMESTAMP);
-	notmuch->date_range_processor = new ParseTimeValueRangeProcessor (NOTMUCH_VALUE_TIMESTAMP);
-	notmuch->last_mod_range_processor = new Xapian::NumberValueRangeProcessor (NOTMUCH_VALUE_LAST_MOD, "lastmod:");
-
+	notmuch->value_range_processor = new Xapian::NumberRangeProcessor (NOTMUCH_VALUE_TIMESTAMP);
+	notmuch->date_range_processor = new ParseTimeRangeProcessor (NOTMUCH_VALUE_TIMESTAMP, "date:");
+	notmuch->last_mod_range_processor = new Xapian::NumberRangeProcessor (NOTMUCH_VALUE_LAST_MOD, "lastmod:");
 	notmuch->query_parser->set_default_op (Xapian::Query::OP_AND);
 	notmuch->query_parser->set_database (*notmuch->xapian_db);
 	notmuch->query_parser->set_stemmer (Xapian::Stem ("english"));
 	notmuch->query_parser->set_stemming_strategy (Xapian::QueryParser::STEM_SOME);
-	notmuch->query_parser->add_valuerangeprocessor (notmuch->value_range_processor);
-	notmuch->query_parser->add_valuerangeprocessor (notmuch->date_range_processor);
-	notmuch->query_parser->add_valuerangeprocessor (notmuch->last_mod_range_processor);
+	notmuch->query_parser->add_rangeprocessor (notmuch->value_range_processor);
+	notmuch->query_parser->add_rangeprocessor (notmuch->date_range_processor);
+	notmuch->query_parser->add_rangeprocessor (notmuch->last_mod_range_processor);
 
 	for (i = 0; i < ARRAY_SIZE (prefix_table); i++) {
 	    const prefix_t *prefix = &prefix_table[i];
@@ -1401,8 +1400,8 @@ handle_sigalrm (unused (int signal))
  */
 notmuch_status_t
 notmuch_database_upgrade (notmuch_database_t *notmuch,
-			  void (*progress_notify)(void *closure,
-						  double progress),
+			  void (*progress_notify) (void *closure,
+						   double progress),
 			  void *closure)
 {
     void *local = talloc_new (NULL);
diff --git a/lib/parse-time-vrp.cc b/lib/parse-time-vrp.cc
index 168d5810..d2495500 100644
--- a/lib/parse-time-vrp.cc
+++ b/lib/parse-time-vrp.cc
@@ -24,45 +24,47 @@
 #include "parse-time-vrp.h"
 #include "parse-time-string.h"
 
-#define PREFIX "date:"
+static Xapian::Query _make_query (Xapian::valueno slot, time_t from, time_t to) {
+    if (to < 0) {
+	return Xapian::Query (Xapian::Query::OP_VALUE_GE, slot,
+			      Xapian::sortable_serialise ((double) from));
+    } else if (from < 0) {
+	return Xapian::Query (Xapian::Query::OP_VALUE_LE, slot,
+			      Xapian::sortable_serialise ((double) to));
+    } else {
+	return Xapian::Query (Xapian::Query::OP_VALUE_RANGE, slot,
+			      Xapian::sortable_serialise ((double) from),
+			      Xapian::sortable_serialise ((double) to));
+    }
+}
 
-/* See *ValueRangeProcessor in xapian-core/api/valuerangeproc.cc */
-Xapian::valueno
-ParseTimeValueRangeProcessor::operator() (std::string &begin, std::string &end)
+Xapian::Query
+ParseTimeRangeProcessor::operator() (const std::string &begin, const std::string &end)
 {
-    time_t t, now;
-    std::string b;
-
-    /* Require date: prefix in start of the range... */
-    if (STRNCMP_LITERAL (begin.c_str (), PREFIX))
-	return Xapian::BAD_VALUENO;
-
-    /* ...and remove it. */
-    begin.erase (0, sizeof (PREFIX) - 1);
-    b = begin;
+    time_t from = -1, to = -1, now = -1;
+    std::string str;
 
     /* Use the same 'now' for begin and end. */
     if (time (&now) == (time_t) -1)
-	return Xapian::BAD_VALUENO;
-
-    if (! begin.empty ()) {
-	if (parse_time_string (begin.c_str (), &t, &now, PARSE_TIME_ROUND_DOWN))
-	    return Xapian::BAD_VALUENO;
+	throw Xapian::QueryParserError ("unable to get current time");
 
-	begin.assign (Xapian::sortable_serialise ((double) t));
-    }
+    if (!begin.empty ())
+	if (parse_time_string (begin.c_str (), &from, &now, PARSE_TIME_ROUND_DOWN))
+	    throw Xapian::QueryParserError ("Didn't understand date specification '" + begin + "'");
 
-    if (! end.empty ()) {
-	if (end == "!" && ! b.empty ())
-	    end = b;
 
-	if (parse_time_string (end.c_str (), &t, &now, PARSE_TIME_ROUND_UP_INCLUSIVE))
-	    return Xapian::BAD_VALUENO;
+    if (!end.empty ()) {
+	if (end == "!" && ! begin.empty ())
+	    str = begin;
+	else
+	    str = end;
 
-	end.assign (Xapian::sortable_serialise ((double) t));
+	if (parse_time_string (str.c_str (), &to, &now, PARSE_TIME_ROUND_UP_INCLUSIVE))
+	    if (parse_time_string (str.c_str (), &to, &now, PARSE_TIME_ROUND_UP_INCLUSIVE))
+		throw Xapian::QueryParserError ("Didn't understand date specification '" + str + "'");
     }
 
-    return valno;
+    return _make_query (slot, from, to);
 }
 
 /* XXX TODO: is throwing an exception the right thing to do here? */
@@ -81,7 +83,5 @@ DateFieldProcessor::operator() (const std::string & str)
     if (parse_time_string (str.c_str (), &to, &now, PARSE_TIME_ROUND_UP_INCLUSIVE))
 	throw Xapian::QueryParserError ("Didn't understand date specification '" + str + "'");
 
-    return Xapian::Query (Xapian::Query::OP_AND,
-			  Xapian::Query (Xapian::Query::OP_VALUE_GE, 0, Xapian::sortable_serialise ((double) from)),
-			  Xapian::Query (Xapian::Query::OP_VALUE_LE, 0, Xapian::sortable_serialise ((double) to)));
+    return _make_query (slot, from, to);
 }
diff --git a/lib/parse-time-vrp.h b/lib/parse-time-vrp.h
index 9fb1af60..f495e716 100644
--- a/lib/parse-time-vrp.h
+++ b/lib/parse-time-vrp.h
@@ -26,21 +26,21 @@
 #include <xapian.h>
 
 /* see *ValueRangeProcessor in xapian-core/include/xapian/queryparser.h */
-class ParseTimeValueRangeProcessor : public Xapian::ValueRangeProcessor {
-protected:
-    Xapian::valueno valno;
+class ParseTimeRangeProcessor : public Xapian::RangeProcessor {
 
 public:
-    ParseTimeValueRangeProcessor (Xapian::valueno slot_)
-	: valno (slot_)
-    {
-    }
+    ParseTimeRangeProcessor (Xapian::valueno slot_, const std::string prefix_)
+	:  Xapian::RangeProcessor(slot_, prefix_, 0) { }
 
-    Xapian::valueno operator() (std::string &begin, std::string &end);
+    Xapian::Query operator() (const std::string &begin, const std::string &end);
 };
 
 class DateFieldProcessor : public Xapian::FieldProcessor {
-    Xapian::Query operator() (const std::string & str);
+private:
+    Xapian::valueno slot;
+public:
+    DateFieldProcessor(Xapian::valueno slot_) : slot(slot_) { };
+    Xapian::Query operator()(const std::string & str);
 };
 
 #endif /* NOTMUCH_PARSE_TIME_VRP_H */
-- 
2.27.0

^ permalink raw reply related	[flat|nested] 5+ messages in thread

* [PATCH 3/3] test: drop upgrade from v1 tests
  2020-06-17 11:04 Port to Xapian 1.5 David Bremner
  2020-06-17 11:04 ` [PATCH 1/3] lib: migrate to post Xapian 1.3.4 compact support David Bremner
  2020-06-17 11:04 ` [PATCH 2/3] lib: migrate from Xapian ValueRangeProcessor to RangeProcessor David Bremner
@ 2020-06-17 11:04 ` David Bremner
  2 siblings, 0 replies; 5+ messages in thread
From: David Bremner @ 2020-06-17 11:04 UTC (permalink / raw)
  To: notmuch

These are less crucial since we stopped generating new database
versions and relied primarily on features. They also rely on a
pre-generated v1 database which happens to be chert format. This
backend is not supported by Xapian 1.5.

Also drop the tool gen-testdb.sh, which is currently broken, due to
changes in the testing infrastructure.
---
 .travis.yml                                   |   1 -
 configure                                     |   2 +-
 devel/gen-testdb.sh                           | 131 -----------------
 test/T530-upgrade.sh                          | 136 ------------------
 test/test-databases/.gitignore                |   1 -
 test/test-databases/Makefile                  |   7 -
 test/test-databases/Makefile.local            |  20 ---
 test/test-databases/database-v1.tar.xz.sha256 |   1 -
 8 files changed, 1 insertion(+), 298 deletions(-)
 delete mode 100755 devel/gen-testdb.sh
 delete mode 100755 test/T530-upgrade.sh
 delete mode 100644 test/test-databases/.gitignore
 delete mode 100644 test/test-databases/Makefile
 delete mode 100644 test/test-databases/Makefile.local
 delete mode 100644 test/test-databases/database-v1.tar.xz.sha256

diff --git a/.travis.yml b/.travis.yml
index 9dc03619..9dcec1ff 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -20,7 +20,6 @@ addons:
 
 script:
   - ./configure
-  - make download-test-databases
   - make test
 
 notifications:
diff --git a/configure b/configure
index 05ade05b..bdf11b70 100755
--- a/configure
+++ b/configure
@@ -48,7 +48,7 @@ case $PWD in ( *["$IFS"]* )
 esac
 
 subdirs="util compat lib parse-time-string completion doc emacs"
-subdirs="${subdirs} performance-test test test/test-databases"
+subdirs="${subdirs} performance-test test"
 subdirs="${subdirs} bindings"
 
 # For a non-srcdir configure invocation (such as ../configure), create
diff --git a/devel/gen-testdb.sh b/devel/gen-testdb.sh
deleted file mode 100755
index 61ae48a3..00000000
--- a/devel/gen-testdb.sh
+++ /dev/null
@@ -1,131 +0,0 @@
-#!/usr/bin/env bash
-#
-# NAME
-#	gen-testdb.sh - generate test databases
-#
-# SYNOPSIS
-#	gen-testdb.sh -v NOTMUCH-VERSION [-c CORPUS-PATH] [-s TAR-SUFFIX]
-#
-# DESCRIPTION
-#	Generate a tarball containing the specified test corpus and
-#	the corresponding notmuch database, indexed using a specific
-#	version of notmuch, resulting in a specific version of the
-#	database.
-#
-#	The specific version of notmuch will be built on the fly.
-#	Therefore the script must be run within a git repository to be
-#	able to build the old versions of notmuch.
-#
-#	This script reuses the test infrastructure, and the script
-#	must be run from within the test directory.
-#
-#	The output tarballs, named database-<TAR-SUFFIX>.tar.gz, are
-#	placed in the test/test-databases directory.
-#
-# OPTIONS
-#	-v NOTMUCH-VERSION
-#		Notmuch version in terms of a git tag or commit to use
-#		for generating the database. Required.
-#
-#	-c CORPUS-PATH
-#		Path to a corpus to use for generating the
-#		database. Due to CWD changes within the test
-#		infrastructure, use absolute paths. Defaults to the
-#		test corpus.
-#
-#	-s TAR-SUFFIX
-#		Suffix for the tarball basename. Empty by default.
-#
-# EXAMPLE
-#
-#	Generate a database indexed with notmuch 0.17. Use the default
-#	test corpus. Name the tarball database-v1.tar.gz to reflect
-#	the fact that notmuch 0.17 used database version 1.
-#
-#	$ cd test
-#	$ ../devel/gen-testdb.sh -v 0.17 -s v1
-#
-# CAVEATS
-#	Test infrastructure options won't work.
-#
-#	Any existing databases with the same name will be overwritten.
-#
-#	It may not be possible to build old versions of notmuch with
-#	the set of dependencies that satisfy building the current
-#	version of notmuch.
-#
-# AUTHOR
-#	Jani Nikula <jani@nikula.org>
-#
-# LICENSE
-#	Same as notmuch test infrastructure (GPLv2+).
-#
-
-test_description="database generation abusing test infrastructure"
-
-# immediate exit on subtest failure; see test_failure_ in test-lib.sh
-immediate=t
-
-VERSION=
-CORPUS=
-SUFFIX=
-
-while getopts v:c:s: opt; do
-    case "$opt" in
-	v) VERSION="$OPTARG";;
-	c) CORPUS="$OPTARG";;
-	s) SUFFIX="-$OPTARG";;
-    esac
-done
-shift `expr $OPTIND - 1`
-
-. ./test-lib.sh || exit 1
-
-SHORT_CORPUS=$(basename ${CORPUS:-database})
-DBNAME=${SHORT_CORPUS}${SUFFIX}
-TARBALLNAME=${DBNAME}.tar.xz
-
-CORPUS=${CORPUS:-${TEST_DIRECTORY}/corpus}
-
-test_expect_code 0 "notmuch version specified on the command line" \
-    "test -n ${VERSION}"
-
-test_expect_code 0 "the specified version ${VERSION} refers to a commit" \
-    "git show ${VERSION} >/dev/null 2>&1"
-
-BUILD_DIR="notmuch-${VERSION}"
-test_expect_code 0 "generate snapshot of notmuch version ${VERSION}" \
-    "git -C $TEST_DIRECTORY/.. archive --prefix=${BUILD_DIR}/ --format=tar ${VERSION} | tar x"
-
-# force version string
-git describe --match '[0-9.]*' ${VERSION} > ${BUILD_DIR}/version
-
-test_expect_code 0 "configure and build notmuch version ${VERSION}" \
-    "make -C ${BUILD_DIR}"
-
-# use the newly built notmuch
-export PATH=./${BUILD_DIR}:$PATH
-
-test_begin_subtest "verify the newly built notmuch version"
-test_expect_equal "`notmuch --version`" "notmuch `cat ${BUILD_DIR}/version`"
-
-# replace the existing mails, if any, with the specified corpus
-rm -rf ${MAIL_DIR}
-cp -a ${CORPUS} ${MAIL_DIR}
-
-test_expect_code 0 "index the corpus" \
-    "notmuch new"
-
-# wrap the resulting mail store and database in a tarball
-
-cp -a ${MAIL_DIR} ${TMP_DIRECTORY}/${DBNAME}
-tar Jcf ${TMP_DIRECTORY}/${TARBALLNAME} -C ${TMP_DIRECTORY} ${DBNAME}
-mkdir -p  ${TEST_DIRECTORY}/test-databases
-cp -a ${TMP_DIRECTORY}/${TARBALLNAME} ${TEST_DIRECTORY}/test-databases
-test_expect_code 0 "create the output tarball ${TARBALLNAME}" \
-    "test -f ${TEST_DIRECTORY}/test-databases/${TARBALLNAME}"
-
-# generate a checksum file
-test_expect_code 0 "compute checksum" \
-    "(cd ${TEST_DIRECTORY}/test-databases/ && sha256sum ${TARBALLNAME} > ${TARBALLNAME}.sha256)"
-test_done
diff --git a/test/T530-upgrade.sh b/test/T530-upgrade.sh
deleted file mode 100755
index 2124dde2..00000000
--- a/test/T530-upgrade.sh
+++ /dev/null
@@ -1,136 +0,0 @@
-#!/usr/bin/env bash
-test_description="database upgrade"
-
-. $(dirname "$0")/test-lib.sh || exit 1
-
-dbtarball=database-v1.tar.xz
-
-# XXX: Accomplish the same with test lib helpers
-if [ ! -e ${TEST_DIRECTORY}/test-databases/${dbtarball} ]; then
-    test_subtest_missing_external_prereq_["${dbtarball} - fetch with 'make download-test-databases'"]=t
-fi
-
-test_begin_subtest "database checksum"
-test_expect_success \
-    '( cd $TEST_DIRECTORY/test-databases &&
-       sha256sum --quiet --check --status ${dbtarball}.sha256 )'
-
-tar xf $TEST_DIRECTORY/test-databases/${dbtarball} -C ${MAIL_DIR} --strip-components=1
-
-test_begin_subtest "folder: search does not work with old database version"
-output=$(notmuch search folder:foo)
-test_expect_equal "$output" ""
-
-test_begin_subtest "path: search does not work with old database version"
-output=$(notmuch search path:foo)
-test_expect_equal "$output" ""
-
-test_begin_subtest "pre upgrade dump"
-test_expect_success 'notmuch dump | sort > pre-upgrade-dump'
-
-test_begin_subtest "database upgrade from format version 1"
-output=$(notmuch new | sed -e 's/^Backing up tags to .*$/Backing up tags to FILENAME/')
-test_expect_equal "$output" "\
-Welcome to a new version of notmuch! Your database will now be upgraded.
-This process is safe to interrupt.
-Backing up tags to FILENAME
-Your notmuch database has now been upgraded.
-No new mail."
-
-test_begin_subtest "tag backup matches pre-upgrade dump"
-gunzip -c ${MAIL_DIR}/.notmuch/dump-*.gz | sort > backup-dump
-test_expect_equal_file pre-upgrade-dump backup-dump
-
-test_begin_subtest "folder: no longer matches in the middle of path"
-output=$(notmuch search folder:baz)
-test_expect_equal "$output" ""
-
-test_begin_subtest "folder: search"
-output=$(notmuch search --output=files folder:foo | notmuch_search_files_sanitize | sort)
-test_expect_equal "$output" "MAIL_DIR/foo/06:2,
-MAIL_DIR/foo/cur/07:2,
-MAIL_DIR/foo/cur/08:2,
-MAIL_DIR/foo/new/03:2,
-MAIL_DIR/foo/new/09:2,
-MAIL_DIR/foo/new/10:2,"
-
-test_begin_subtest "top level folder: search"
-output=$(notmuch search --output=files folder:'""' | notmuch_search_files_sanitize | sort)
-# bar/18:2, is a duplicate of cur/51:2,
-test_expect_equal "$output" "MAIL_DIR/01:2,
-MAIL_DIR/02:2,
-MAIL_DIR/bar/18:2,
-MAIL_DIR/cur/29:2,
-MAIL_DIR/cur/30:2,
-MAIL_DIR/cur/31:2,
-MAIL_DIR/cur/32:2,
-MAIL_DIR/cur/33:2,
-MAIL_DIR/cur/34:2,
-MAIL_DIR/cur/35:2,
-MAIL_DIR/cur/36:2,
-MAIL_DIR/cur/37:2,
-MAIL_DIR/cur/38:2,
-MAIL_DIR/cur/39:2,
-MAIL_DIR/cur/40:2,
-MAIL_DIR/cur/41:2,
-MAIL_DIR/cur/42:2,
-MAIL_DIR/cur/43:2,
-MAIL_DIR/cur/44:2,
-MAIL_DIR/cur/45:2,
-MAIL_DIR/cur/46:2,
-MAIL_DIR/cur/47:2,
-MAIL_DIR/cur/48:2,
-MAIL_DIR/cur/49:2,
-MAIL_DIR/cur/50:2,
-MAIL_DIR/cur/51:2,
-MAIL_DIR/cur/52:2,
-MAIL_DIR/cur/53:2,
-MAIL_DIR/new/04:2,"
-
-test_begin_subtest "path: search"
-output=$(notmuch search --output=files path:"bar" | notmuch_search_files_sanitize | sort)
-# cur/51:2, is a duplicate of bar/18:2,
-test_expect_equal "$output" "MAIL_DIR/bar/17:2,
-MAIL_DIR/bar/18:2,
-MAIL_DIR/cur/51:2,"
-
-test_begin_subtest "top level path: search"
-output=$(notmuch search --output=files path:'""' | notmuch_search_files_sanitize | sort)
-test_expect_equal "$output" "MAIL_DIR/01:2,
-MAIL_DIR/02:2,"
-
-test_begin_subtest "recursive path: search"
-output=$(notmuch search --output=files path:"bar/**" | notmuch_search_files_sanitize | sort)
-# cur/51:2, is a duplicate of bar/18:2,
-test_expect_equal "$output" "MAIL_DIR/bar/17:2,
-MAIL_DIR/bar/18:2,
-MAIL_DIR/bar/baz/05:2,
-MAIL_DIR/bar/baz/23:2,
-MAIL_DIR/bar/baz/24:2,
-MAIL_DIR/bar/baz/cur/25:2,
-MAIL_DIR/bar/baz/cur/26:2,
-MAIL_DIR/bar/baz/new/27:2,
-MAIL_DIR/bar/baz/new/28:2,
-MAIL_DIR/bar/cur/19:2,
-MAIL_DIR/bar/cur/20:2,
-MAIL_DIR/bar/new/21:2,
-MAIL_DIR/bar/new/22:2,
-MAIL_DIR/cur/51:2,"
-
-test_begin_subtest "body: same as unprefixed before reindex"
-notmuch search --output=messages body:close > OUTPUT
-notmuch search --output=messages close  > EXPECTED
-test_expect_equal_file EXPECTED OUTPUT
-
-test_begin_subtest "body: subset of unprefixed after reindex"
-notmuch reindex '*'
-notmuch search --output=messages body:close | sort > BODY
-notmuch search --output=messages close | sort > UNPREFIXED
-diff -e UNPREFIXED BODY | cut -c2- > OUTPUT
-cat <<EOF > EXPECTED
-d
-d
-EOF
-test_expect_equal_file EXPECTED OUTPUT
-
-test_done
diff --git a/test/test-databases/.gitignore b/test/test-databases/.gitignore
deleted file mode 100644
index 9452199f..00000000
--- a/test/test-databases/.gitignore
+++ /dev/null
@@ -1 +0,0 @@
-/*.tar.xz
diff --git a/test/test-databases/Makefile b/test/test-databases/Makefile
deleted file mode 100644
index b250a8be..00000000
--- a/test/test-databases/Makefile
+++ /dev/null
@@ -1,7 +0,0 @@
-# See Makefile.local for the list of files to be compiled in this
-# directory.
-all:
-	$(MAKE) -C ../.. all
-
-.DEFAULT:
-	$(MAKE) -C ../.. $@
diff --git a/test/test-databases/Makefile.local b/test/test-databases/Makefile.local
deleted file mode 100644
index 7aedff70..00000000
--- a/test/test-databases/Makefile.local
+++ /dev/null
@@ -1,20 +0,0 @@
-# -*- makefile -*-
-
-TEST_DATABASE_MIRROR=https://notmuchmail.org/releases/test-databases
-
-dir := test/test-databases
-
-test_databases := $(dir)/database-v1.tar.xz
-
-%.tar.xz:
-	@exec 1>&2 ;\
-	if command -v wget >/dev/null ;\
-	then set -x; wget -nv -O $@ ${TEST_DATABASE_MIRROR}/$(notdir $@) ;\
-	elif command -v curl >/dev/null ;\
-	then set -x; curl -L -s -o $@ ${TEST_DATABASE_MIRROR}/$(notdir $@) ;\
-	else echo Cannot fetch databases, no wget nor curl available; exit 1 ;\
-	fi
-
-download-test-databases: ${test_databases}
-
-DATACLEAN := $(DATACLEAN) ${test_databases}
diff --git a/test/test-databases/database-v1.tar.xz.sha256 b/test/test-databases/database-v1.tar.xz.sha256
deleted file mode 100644
index 2cc4f965..00000000
--- a/test/test-databases/database-v1.tar.xz.sha256
+++ /dev/null
@@ -1 +0,0 @@
-4299e051b10e1fa7b33ea2862790a09ebfe96859681804e5251e130f800e69d2  database-v1.tar.xz
-- 
2.27.0

^ permalink raw reply related	[flat|nested] 5+ messages in thread

* [PATCH 2/3] lib: migrate from Xapian ValueRangeProcessor to RangeProcessor
  2020-07-07 10:56 v2 port to xapian 1.5 David Bremner
@ 2020-07-07 10:56 ` David Bremner
  0 siblings, 0 replies; 5+ messages in thread
From: David Bremner @ 2020-07-07 10:56 UTC (permalink / raw)
  To: notmuch; +Cc: David Bremner

This will be mandatory as of Xapian 1.5.  The API is also more
consistent with the FieldProcessor API, which helps code re-use a bit.

Note that this switches to using the built-in Xapian support for
prefixes on ranges (i.e. deleted code at beginning of
ParseTimeRangeProcessor::operator(), added prefix to constructor).

Another side effect of the migration is that we are generating smaller
queries, using one OP_VALUE_RANGE instead of an AND of two OP_VALUE_*
queries.
---
 lib/database-private.h |  6 ++--
 lib/database.cc        | 21 ++++++-------
 lib/parse-time-vrp.cc  | 69 +++++++++++++++++++++---------------------
 lib/parse-time-vrp.h   | 18 +++++------
 4 files changed, 56 insertions(+), 58 deletions(-)

diff --git a/lib/database-private.h b/lib/database-private.h
index 87ae1bdf..76359007 100644
--- a/lib/database-private.h
+++ b/lib/database-private.h
@@ -218,9 +218,9 @@ struct _notmuch_database {
     unsigned long view;
     Xapian::QueryParser *query_parser;
     Xapian::TermGenerator *term_gen;
-    Xapian::ValueRangeProcessor *value_range_processor;
-    Xapian::ValueRangeProcessor *date_range_processor;
-    Xapian::ValueRangeProcessor *last_mod_range_processor;
+    Xapian::RangeProcessor *value_range_processor;
+    Xapian::RangeProcessor *date_range_processor;
+    Xapian::RangeProcessor *last_mod_range_processor;
 
     /* XXX it's slightly gross to use two parallel string->string maps
      * here, but at least they are small */
diff --git a/lib/database.cc b/lib/database.cc
index 43bfac4e..2f794164 100644
--- a/lib/database.cc
+++ b/lib/database.cc
@@ -385,8 +385,8 @@ _setup_query_field (const prefix_t *prefix, notmuch_database_t *notmuch)
 	Xapian::FieldProcessor *fp;
 
 	if (STRNCMP_LITERAL (prefix->name, "date") == 0)
-	    fp = (new DateFieldProcessor ())->release ();
-	else if (STRNCMP_LITERAL (prefix->name, "query") == 0)
+	    fp = (new DateFieldProcessor(NOTMUCH_VALUE_TIMESTAMP))->release ();
+	else if (STRNCMP_LITERAL(prefix->name, "query") == 0)
 	    fp = (new QueryFieldProcessor (*notmuch->query_parser, notmuch))->release ();
 	else if (STRNCMP_LITERAL (prefix->name, "thread") == 0)
 	    fp = (new ThreadFieldProcessor (*notmuch->query_parser, notmuch))->release ();
@@ -1036,17 +1036,16 @@ notmuch_database_open_verbose (const char *path,
 	notmuch->query_parser = new Xapian::QueryParser;
 	notmuch->term_gen = new Xapian::TermGenerator;
 	notmuch->term_gen->set_stemmer (Xapian::Stem ("english"));
-	notmuch->value_range_processor = new Xapian::NumberValueRangeProcessor (NOTMUCH_VALUE_TIMESTAMP);
-	notmuch->date_range_processor = new ParseTimeValueRangeProcessor (NOTMUCH_VALUE_TIMESTAMP);
-	notmuch->last_mod_range_processor = new Xapian::NumberValueRangeProcessor (NOTMUCH_VALUE_LAST_MOD, "lastmod:");
-
+	notmuch->value_range_processor = new Xapian::NumberRangeProcessor (NOTMUCH_VALUE_TIMESTAMP);
+	notmuch->date_range_processor = new ParseTimeRangeProcessor (NOTMUCH_VALUE_TIMESTAMP, "date:");
+	notmuch->last_mod_range_processor = new Xapian::NumberRangeProcessor (NOTMUCH_VALUE_LAST_MOD, "lastmod:");
 	notmuch->query_parser->set_default_op (Xapian::Query::OP_AND);
 	notmuch->query_parser->set_database (*notmuch->xapian_db);
 	notmuch->query_parser->set_stemmer (Xapian::Stem ("english"));
 	notmuch->query_parser->set_stemming_strategy (Xapian::QueryParser::STEM_SOME);
-	notmuch->query_parser->add_valuerangeprocessor (notmuch->value_range_processor);
-	notmuch->query_parser->add_valuerangeprocessor (notmuch->date_range_processor);
-	notmuch->query_parser->add_valuerangeprocessor (notmuch->last_mod_range_processor);
+	notmuch->query_parser->add_rangeprocessor (notmuch->value_range_processor);
+	notmuch->query_parser->add_rangeprocessor (notmuch->date_range_processor);
+	notmuch->query_parser->add_rangeprocessor (notmuch->last_mod_range_processor);
 
 	for (i = 0; i < ARRAY_SIZE (prefix_table); i++) {
 	    const prefix_t *prefix = &prefix_table[i];
@@ -1401,8 +1400,8 @@ handle_sigalrm (unused (int signal))
  */
 notmuch_status_t
 notmuch_database_upgrade (notmuch_database_t *notmuch,
-			  void (*progress_notify)(void *closure,
-						  double progress),
+			  void (*progress_notify) (void *closure,
+						   double progress),
 			  void *closure)
 {
     void *local = talloc_new (NULL);
diff --git a/lib/parse-time-vrp.cc b/lib/parse-time-vrp.cc
index 168d5810..10809aa3 100644
--- a/lib/parse-time-vrp.cc
+++ b/lib/parse-time-vrp.cc
@@ -24,64 +24,63 @@
 #include "parse-time-vrp.h"
 #include "parse-time-string.h"
 
-#define PREFIX "date:"
-
-/* See *ValueRangeProcessor in xapian-core/api/valuerangeproc.cc */
-Xapian::valueno
-ParseTimeValueRangeProcessor::operator() (std::string &begin, std::string &end)
+Xapian::Query
+ParseTimeRangeProcessor::operator() (const std::string &begin, const std::string &end)
 {
-    time_t t, now;
-    std::string b;
-
-    /* Require date: prefix in start of the range... */
-    if (STRNCMP_LITERAL (begin.c_str (), PREFIX))
-	return Xapian::BAD_VALUENO;
-
-    /* ...and remove it. */
-    begin.erase (0, sizeof (PREFIX) - 1);
-    b = begin;
+    double from = DBL_MIN, to = DBL_MAX;
+    time_t parsed_time, now;
+    std::string str;
 
     /* Use the same 'now' for begin and end. */
     if (time (&now) == (time_t) -1)
-	return Xapian::BAD_VALUENO;
-
-    if (! begin.empty ()) {
-	if (parse_time_string (begin.c_str (), &t, &now, PARSE_TIME_ROUND_DOWN))
-	    return Xapian::BAD_VALUENO;
+	throw Xapian::QueryParserError ("unable to get current time");
 
-	begin.assign (Xapian::sortable_serialise ((double) t));
+    if (!begin.empty ()) {
+	if (parse_time_string (begin.c_str (), &parsed_time, &now, PARSE_TIME_ROUND_DOWN))
+	    throw Xapian::QueryParserError ("Didn't understand date specification '" + begin + "'");
+	else
+	    from = (double) parsed_time;
     }
 
-    if (! end.empty ()) {
-	if (end == "!" && ! b.empty ())
-	    end = b;
-
-	if (parse_time_string (end.c_str (), &t, &now, PARSE_TIME_ROUND_UP_INCLUSIVE))
-	    return Xapian::BAD_VALUENO;
+    if (!end.empty ()) {
+	if (end == "!" && ! begin.empty ())
+	    str = begin;
+	else
+	    str = end;
 
-	end.assign (Xapian::sortable_serialise ((double) t));
+	if (parse_time_string (str.c_str (), &parsed_time, &now, PARSE_TIME_ROUND_UP_INCLUSIVE))
+	    throw Xapian::QueryParserError ("Didn't understand date specification '" + str + "'");
+	else
+	    to = (double) parsed_time;
     }
 
-    return valno;
+    return Xapian::Query (Xapian::Query::OP_VALUE_RANGE, slot,
+			  Xapian::sortable_serialise (from),
+			  Xapian::sortable_serialise (to));
 }
 
 /* XXX TODO: is throwing an exception the right thing to do here? */
 Xapian::Query
 DateFieldProcessor::operator() (const std::string & str)
 {
-    time_t from, to, now;
+    double from = DBL_MIN, to = DBL_MAX;
+    time_t parsed_time, now;
 
     /* Use the same 'now' for begin and end. */
     if (time (&now) == (time_t) -1)
 	throw Xapian::QueryParserError ("Unable to get current time");
 
-    if (parse_time_string (str.c_str (), &from, &now, PARSE_TIME_ROUND_DOWN))
+    if (parse_time_string (str.c_str (), &parsed_time, &now, PARSE_TIME_ROUND_DOWN))
 	throw Xapian::QueryParserError ("Didn't understand date specification '" + str + "'");
+    else
+	from = (double) parsed_time;
 
-    if (parse_time_string (str.c_str (), &to, &now, PARSE_TIME_ROUND_UP_INCLUSIVE))
+    if (parse_time_string (str.c_str (), &parsed_time, &now, PARSE_TIME_ROUND_UP_INCLUSIVE))
 	throw Xapian::QueryParserError ("Didn't understand date specification '" + str + "'");
+    else
+	to = (double) parsed_time;
 
-    return Xapian::Query (Xapian::Query::OP_AND,
-			  Xapian::Query (Xapian::Query::OP_VALUE_GE, 0, Xapian::sortable_serialise ((double) from)),
-			  Xapian::Query (Xapian::Query::OP_VALUE_LE, 0, Xapian::sortable_serialise ((double) to)));
+    return Xapian::Query (Xapian::Query::OP_VALUE_RANGE, slot,
+			  Xapian::sortable_serialise (from),
+			  Xapian::sortable_serialise (to));
 }
diff --git a/lib/parse-time-vrp.h b/lib/parse-time-vrp.h
index 9fb1af60..f495e716 100644
--- a/lib/parse-time-vrp.h
+++ b/lib/parse-time-vrp.h
@@ -26,21 +26,21 @@
 #include <xapian.h>
 
 /* see *ValueRangeProcessor in xapian-core/include/xapian/queryparser.h */
-class ParseTimeValueRangeProcessor : public Xapian::ValueRangeProcessor {
-protected:
-    Xapian::valueno valno;
+class ParseTimeRangeProcessor : public Xapian::RangeProcessor {
 
 public:
-    ParseTimeValueRangeProcessor (Xapian::valueno slot_)
-	: valno (slot_)
-    {
-    }
+    ParseTimeRangeProcessor (Xapian::valueno slot_, const std::string prefix_)
+	:  Xapian::RangeProcessor(slot_, prefix_, 0) { }
 
-    Xapian::valueno operator() (std::string &begin, std::string &end);
+    Xapian::Query operator() (const std::string &begin, const std::string &end);
 };
 
 class DateFieldProcessor : public Xapian::FieldProcessor {
-    Xapian::Query operator() (const std::string & str);
+private:
+    Xapian::valueno slot;
+public:
+    DateFieldProcessor(Xapian::valueno slot_) : slot(slot_) { };
+    Xapian::Query operator()(const std::string & str);
 };
 
 #endif /* NOTMUCH_PARSE_TIME_VRP_H */
-- 
2.27.0

^ permalink raw reply related	[flat|nested] 5+ messages in thread

end of thread, other threads:[~2020-07-07 10:57 UTC | newest]

Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-06-17 11:04 Port to Xapian 1.5 David Bremner
2020-06-17 11:04 ` [PATCH 1/3] lib: migrate to post Xapian 1.3.4 compact support David Bremner
2020-06-17 11:04 ` [PATCH 2/3] lib: migrate from Xapian ValueRangeProcessor to RangeProcessor David Bremner
2020-06-17 11:04 ` [PATCH 3/3] test: drop upgrade from v1 tests David Bremner
  -- strict thread matches above, loose matches on Subject: below --
2020-07-07 10:56 v2 port to xapian 1.5 David Bremner
2020-07-07 10:56 ` [PATCH 2/3] lib: migrate from Xapian ValueRangeProcessor to RangeProcessor David Bremner

Code repositories for project(s) associated with this public inbox

	https://yhetil.org/notmuch.git/

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).