* [PATCH 3/3] test: drop upgrade from v1 tests
2020-06-17 11:04 Port to Xapian 1.5 David Bremner
@ 2020-06-17 11:04 ` David Bremner
0 siblings, 0 replies; 6+ messages in thread
From: David Bremner @ 2020-06-17 11:04 UTC (permalink / raw)
To: notmuch
These are less crucial since we stopped generating new database
versions and relied primarily on features. They also rely on a
pre-generated v1 database which happens to be chert format. This
backend is not supported by Xapian 1.5.
Also drop the tool gen-testdb.sh, which is currently broken, due to
changes in the testing infrastructure.
---
.travis.yml | 1 -
configure | 2 +-
devel/gen-testdb.sh | 131 -----------------
test/T530-upgrade.sh | 136 ------------------
test/test-databases/.gitignore | 1 -
test/test-databases/Makefile | 7 -
test/test-databases/Makefile.local | 20 ---
test/test-databases/database-v1.tar.xz.sha256 | 1 -
8 files changed, 1 insertion(+), 298 deletions(-)
delete mode 100755 devel/gen-testdb.sh
delete mode 100755 test/T530-upgrade.sh
delete mode 100644 test/test-databases/.gitignore
delete mode 100644 test/test-databases/Makefile
delete mode 100644 test/test-databases/Makefile.local
delete mode 100644 test/test-databases/database-v1.tar.xz.sha256
diff --git a/.travis.yml b/.travis.yml
index 9dc03619..9dcec1ff 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -20,7 +20,6 @@ addons:
script:
- ./configure
- - make download-test-databases
- make test
notifications:
diff --git a/configure b/configure
index 05ade05b..bdf11b70 100755
--- a/configure
+++ b/configure
@@ -48,7 +48,7 @@ case $PWD in ( *["$IFS"]* )
esac
subdirs="util compat lib parse-time-string completion doc emacs"
-subdirs="${subdirs} performance-test test test/test-databases"
+subdirs="${subdirs} performance-test test"
subdirs="${subdirs} bindings"
# For a non-srcdir configure invocation (such as ../configure), create
diff --git a/devel/gen-testdb.sh b/devel/gen-testdb.sh
deleted file mode 100755
index 61ae48a3..00000000
--- a/devel/gen-testdb.sh
+++ /dev/null
@@ -1,131 +0,0 @@
-#!/usr/bin/env bash
-#
-# NAME
-# gen-testdb.sh - generate test databases
-#
-# SYNOPSIS
-# gen-testdb.sh -v NOTMUCH-VERSION [-c CORPUS-PATH] [-s TAR-SUFFIX]
-#
-# DESCRIPTION
-# Generate a tarball containing the specified test corpus and
-# the corresponding notmuch database, indexed using a specific
-# version of notmuch, resulting in a specific version of the
-# database.
-#
-# The specific version of notmuch will be built on the fly.
-# Therefore the script must be run within a git repository to be
-# able to build the old versions of notmuch.
-#
-# This script reuses the test infrastructure, and the script
-# must be run from within the test directory.
-#
-# The output tarballs, named database-<TAR-SUFFIX>.tar.gz, are
-# placed in the test/test-databases directory.
-#
-# OPTIONS
-# -v NOTMUCH-VERSION
-# Notmuch version in terms of a git tag or commit to use
-# for generating the database. Required.
-#
-# -c CORPUS-PATH
-# Path to a corpus to use for generating the
-# database. Due to CWD changes within the test
-# infrastructure, use absolute paths. Defaults to the
-# test corpus.
-#
-# -s TAR-SUFFIX
-# Suffix for the tarball basename. Empty by default.
-#
-# EXAMPLE
-#
-# Generate a database indexed with notmuch 0.17. Use the default
-# test corpus. Name the tarball database-v1.tar.gz to reflect
-# the fact that notmuch 0.17 used database version 1.
-#
-# $ cd test
-# $ ../devel/gen-testdb.sh -v 0.17 -s v1
-#
-# CAVEATS
-# Test infrastructure options won't work.
-#
-# Any existing databases with the same name will be overwritten.
-#
-# It may not be possible to build old versions of notmuch with
-# the set of dependencies that satisfy building the current
-# version of notmuch.
-#
-# AUTHOR
-# Jani Nikula <jani@nikula.org>
-#
-# LICENSE
-# Same as notmuch test infrastructure (GPLv2+).
-#
-
-test_description="database generation abusing test infrastructure"
-
-# immediate exit on subtest failure; see test_failure_ in test-lib.sh
-immediate=t
-
-VERSION=
-CORPUS=
-SUFFIX=
-
-while getopts v:c:s: opt; do
- case "$opt" in
- v) VERSION="$OPTARG";;
- c) CORPUS="$OPTARG";;
- s) SUFFIX="-$OPTARG";;
- esac
-done
-shift `expr $OPTIND - 1`
-
-. ./test-lib.sh || exit 1
-
-SHORT_CORPUS=$(basename ${CORPUS:-database})
-DBNAME=${SHORT_CORPUS}${SUFFIX}
-TARBALLNAME=${DBNAME}.tar.xz
-
-CORPUS=${CORPUS:-${TEST_DIRECTORY}/corpus}
-
-test_expect_code 0 "notmuch version specified on the command line" \
- "test -n ${VERSION}"
-
-test_expect_code 0 "the specified version ${VERSION} refers to a commit" \
- "git show ${VERSION} >/dev/null 2>&1"
-
-BUILD_DIR="notmuch-${VERSION}"
-test_expect_code 0 "generate snapshot of notmuch version ${VERSION}" \
- "git -C $TEST_DIRECTORY/.. archive --prefix=${BUILD_DIR}/ --format=tar ${VERSION} | tar x"
-
-# force version string
-git describe --match '[0-9.]*' ${VERSION} > ${BUILD_DIR}/version
-
-test_expect_code 0 "configure and build notmuch version ${VERSION}" \
- "make -C ${BUILD_DIR}"
-
-# use the newly built notmuch
-export PATH=./${BUILD_DIR}:$PATH
-
-test_begin_subtest "verify the newly built notmuch version"
-test_expect_equal "`notmuch --version`" "notmuch `cat ${BUILD_DIR}/version`"
-
-# replace the existing mails, if any, with the specified corpus
-rm -rf ${MAIL_DIR}
-cp -a ${CORPUS} ${MAIL_DIR}
-
-test_expect_code 0 "index the corpus" \
- "notmuch new"
-
-# wrap the resulting mail store and database in a tarball
-
-cp -a ${MAIL_DIR} ${TMP_DIRECTORY}/${DBNAME}
-tar Jcf ${TMP_DIRECTORY}/${TARBALLNAME} -C ${TMP_DIRECTORY} ${DBNAME}
-mkdir -p ${TEST_DIRECTORY}/test-databases
-cp -a ${TMP_DIRECTORY}/${TARBALLNAME} ${TEST_DIRECTORY}/test-databases
-test_expect_code 0 "create the output tarball ${TARBALLNAME}" \
- "test -f ${TEST_DIRECTORY}/test-databases/${TARBALLNAME}"
-
-# generate a checksum file
-test_expect_code 0 "compute checksum" \
- "(cd ${TEST_DIRECTORY}/test-databases/ && sha256sum ${TARBALLNAME} > ${TARBALLNAME}.sha256)"
-test_done
diff --git a/test/T530-upgrade.sh b/test/T530-upgrade.sh
deleted file mode 100755
index 2124dde2..00000000
--- a/test/T530-upgrade.sh
+++ /dev/null
@@ -1,136 +0,0 @@
-#!/usr/bin/env bash
-test_description="database upgrade"
-
-. $(dirname "$0")/test-lib.sh || exit 1
-
-dbtarball=database-v1.tar.xz
-
-# XXX: Accomplish the same with test lib helpers
-if [ ! -e ${TEST_DIRECTORY}/test-databases/${dbtarball} ]; then
- test_subtest_missing_external_prereq_["${dbtarball} - fetch with 'make download-test-databases'"]=t
-fi
-
-test_begin_subtest "database checksum"
-test_expect_success \
- '( cd $TEST_DIRECTORY/test-databases &&
- sha256sum --quiet --check --status ${dbtarball}.sha256 )'
-
-tar xf $TEST_DIRECTORY/test-databases/${dbtarball} -C ${MAIL_DIR} --strip-components=1
-
-test_begin_subtest "folder: search does not work with old database version"
-output=$(notmuch search folder:foo)
-test_expect_equal "$output" ""
-
-test_begin_subtest "path: search does not work with old database version"
-output=$(notmuch search path:foo)
-test_expect_equal "$output" ""
-
-test_begin_subtest "pre upgrade dump"
-test_expect_success 'notmuch dump | sort > pre-upgrade-dump'
-
-test_begin_subtest "database upgrade from format version 1"
-output=$(notmuch new | sed -e 's/^Backing up tags to .*$/Backing up tags to FILENAME/')
-test_expect_equal "$output" "\
-Welcome to a new version of notmuch! Your database will now be upgraded.
-This process is safe to interrupt.
-Backing up tags to FILENAME
-Your notmuch database has now been upgraded.
-No new mail."
-
-test_begin_subtest "tag backup matches pre-upgrade dump"
-gunzip -c ${MAIL_DIR}/.notmuch/dump-*.gz | sort > backup-dump
-test_expect_equal_file pre-upgrade-dump backup-dump
-
-test_begin_subtest "folder: no longer matches in the middle of path"
-output=$(notmuch search folder:baz)
-test_expect_equal "$output" ""
-
-test_begin_subtest "folder: search"
-output=$(notmuch search --output=files folder:foo | notmuch_search_files_sanitize | sort)
-test_expect_equal "$output" "MAIL_DIR/foo/06:2,
-MAIL_DIR/foo/cur/07:2,
-MAIL_DIR/foo/cur/08:2,
-MAIL_DIR/foo/new/03:2,
-MAIL_DIR/foo/new/09:2,
-MAIL_DIR/foo/new/10:2,"
-
-test_begin_subtest "top level folder: search"
-output=$(notmuch search --output=files folder:'""' | notmuch_search_files_sanitize | sort)
-# bar/18:2, is a duplicate of cur/51:2,
-test_expect_equal "$output" "MAIL_DIR/01:2,
-MAIL_DIR/02:2,
-MAIL_DIR/bar/18:2,
-MAIL_DIR/cur/29:2,
-MAIL_DIR/cur/30:2,
-MAIL_DIR/cur/31:2,
-MAIL_DIR/cur/32:2,
-MAIL_DIR/cur/33:2,
-MAIL_DIR/cur/34:2,
-MAIL_DIR/cur/35:2,
-MAIL_DIR/cur/36:2,
-MAIL_DIR/cur/37:2,
-MAIL_DIR/cur/38:2,
-MAIL_DIR/cur/39:2,
-MAIL_DIR/cur/40:2,
-MAIL_DIR/cur/41:2,
-MAIL_DIR/cur/42:2,
-MAIL_DIR/cur/43:2,
-MAIL_DIR/cur/44:2,
-MAIL_DIR/cur/45:2,
-MAIL_DIR/cur/46:2,
-MAIL_DIR/cur/47:2,
-MAIL_DIR/cur/48:2,
-MAIL_DIR/cur/49:2,
-MAIL_DIR/cur/50:2,
-MAIL_DIR/cur/51:2,
-MAIL_DIR/cur/52:2,
-MAIL_DIR/cur/53:2,
-MAIL_DIR/new/04:2,"
-
-test_begin_subtest "path: search"
-output=$(notmuch search --output=files path:"bar" | notmuch_search_files_sanitize | sort)
-# cur/51:2, is a duplicate of bar/18:2,
-test_expect_equal "$output" "MAIL_DIR/bar/17:2,
-MAIL_DIR/bar/18:2,
-MAIL_DIR/cur/51:2,"
-
-test_begin_subtest "top level path: search"
-output=$(notmuch search --output=files path:'""' | notmuch_search_files_sanitize | sort)
-test_expect_equal "$output" "MAIL_DIR/01:2,
-MAIL_DIR/02:2,"
-
-test_begin_subtest "recursive path: search"
-output=$(notmuch search --output=files path:"bar/**" | notmuch_search_files_sanitize | sort)
-# cur/51:2, is a duplicate of bar/18:2,
-test_expect_equal "$output" "MAIL_DIR/bar/17:2,
-MAIL_DIR/bar/18:2,
-MAIL_DIR/bar/baz/05:2,
-MAIL_DIR/bar/baz/23:2,
-MAIL_DIR/bar/baz/24:2,
-MAIL_DIR/bar/baz/cur/25:2,
-MAIL_DIR/bar/baz/cur/26:2,
-MAIL_DIR/bar/baz/new/27:2,
-MAIL_DIR/bar/baz/new/28:2,
-MAIL_DIR/bar/cur/19:2,
-MAIL_DIR/bar/cur/20:2,
-MAIL_DIR/bar/new/21:2,
-MAIL_DIR/bar/new/22:2,
-MAIL_DIR/cur/51:2,"
-
-test_begin_subtest "body: same as unprefixed before reindex"
-notmuch search --output=messages body:close > OUTPUT
-notmuch search --output=messages close > EXPECTED
-test_expect_equal_file EXPECTED OUTPUT
-
-test_begin_subtest "body: subset of unprefixed after reindex"
-notmuch reindex '*'
-notmuch search --output=messages body:close | sort > BODY
-notmuch search --output=messages close | sort > UNPREFIXED
-diff -e UNPREFIXED BODY | cut -c2- > OUTPUT
-cat <<EOF > EXPECTED
-d
-d
-EOF
-test_expect_equal_file EXPECTED OUTPUT
-
-test_done
diff --git a/test/test-databases/.gitignore b/test/test-databases/.gitignore
deleted file mode 100644
index 9452199f..00000000
--- a/test/test-databases/.gitignore
+++ /dev/null
@@ -1 +0,0 @@
-/*.tar.xz
diff --git a/test/test-databases/Makefile b/test/test-databases/Makefile
deleted file mode 100644
index b250a8be..00000000
--- a/test/test-databases/Makefile
+++ /dev/null
@@ -1,7 +0,0 @@
-# See Makefile.local for the list of files to be compiled in this
-# directory.
-all:
- $(MAKE) -C ../.. all
-
-.DEFAULT:
- $(MAKE) -C ../.. $@
diff --git a/test/test-databases/Makefile.local b/test/test-databases/Makefile.local
deleted file mode 100644
index 7aedff70..00000000
--- a/test/test-databases/Makefile.local
+++ /dev/null
@@ -1,20 +0,0 @@
-# -*- makefile -*-
-
-TEST_DATABASE_MIRROR=https://notmuchmail.org/releases/test-databases
-
-dir := test/test-databases
-
-test_databases := $(dir)/database-v1.tar.xz
-
-%.tar.xz:
- @exec 1>&2 ;\
- if command -v wget >/dev/null ;\
- then set -x; wget -nv -O $@ ${TEST_DATABASE_MIRROR}/$(notdir $@) ;\
- elif command -v curl >/dev/null ;\
- then set -x; curl -L -s -o $@ ${TEST_DATABASE_MIRROR}/$(notdir $@) ;\
- else echo Cannot fetch databases, no wget nor curl available; exit 1 ;\
- fi
-
-download-test-databases: ${test_databases}
-
-DATACLEAN := $(DATACLEAN) ${test_databases}
diff --git a/test/test-databases/database-v1.tar.xz.sha256 b/test/test-databases/database-v1.tar.xz.sha256
deleted file mode 100644
index 2cc4f965..00000000
--- a/test/test-databases/database-v1.tar.xz.sha256
+++ /dev/null
@@ -1 +0,0 @@
-4299e051b10e1fa7b33ea2862790a09ebfe96859681804e5251e130f800e69d2 database-v1.tar.xz
--
2.27.0
^ permalink raw reply related [flat|nested] 6+ messages in thread
* v2 port to xapian 1.5
@ 2020-07-07 10:56 David Bremner
2020-07-07 10:56 ` [PATCH 1/3] lib: migrate to post Xapian 1.3.4 compact support David Bremner
` (3 more replies)
0 siblings, 4 replies; 6+ messages in thread
From: David Bremner @ 2020-07-07 10:56 UTC (permalink / raw)
To: notmuch
This obsoletes id:20200617110441.1262683-1-david@tethera.net
I have updated patch 2/3 based on discussions with Olly and Tomi.
Olly gave the previous series thumbs up on IRC, so I'll probably merge
this version if I don't here any corrections.
d
^ permalink raw reply [flat|nested] 6+ messages in thread
* [PATCH 1/3] lib: migrate to post Xapian 1.3.4 compact support
2020-07-07 10:56 v2 port to xapian 1.5 David Bremner
@ 2020-07-07 10:56 ` David Bremner
2020-07-07 10:56 ` [PATCH 2/3] lib: migrate from Xapian ValueRangeProcessor to RangeProcessor David Bremner
` (2 subsequent siblings)
3 siblings, 0 replies; 6+ messages in thread
From: David Bremner @ 2020-07-07 10:56 UTC (permalink / raw)
To: notmuch; +Cc: David Bremner
The old API was deprecated in Xapian 1.3.4 and (will be) removed in 1.5.0
---
lib/database.cc | 6 +-----
1 file changed, 1 insertion(+), 5 deletions(-)
diff --git a/lib/database.cc b/lib/database.cc
index 78b5fec9..43bfac4e 100644
--- a/lib/database.cc
+++ b/lib/database.cc
@@ -1281,11 +1281,7 @@ notmuch_database_compact (const char *path,
try {
NotmuchCompactor compactor (status_cb, closure);
-
- compactor.set_renumber (false);
- compactor.add_source (xapian_path);
- compactor.set_destdir (compact_xapian_path);
- compactor.compact ();
+ notmuch->xapian_db->compact (compact_xapian_path, Xapian::DBCOMPACT_NO_RENUMBER, 0, compactor);
} catch (const Xapian::Error &error) {
_notmuch_database_log (notmuch, "Error while compacting: %s\n", error.get_msg ().c_str ());
ret = NOTMUCH_STATUS_XAPIAN_EXCEPTION;
--
2.27.0
^ permalink raw reply related [flat|nested] 6+ messages in thread
* [PATCH 2/3] lib: migrate from Xapian ValueRangeProcessor to RangeProcessor
2020-07-07 10:56 v2 port to xapian 1.5 David Bremner
2020-07-07 10:56 ` [PATCH 1/3] lib: migrate to post Xapian 1.3.4 compact support David Bremner
@ 2020-07-07 10:56 ` David Bremner
2020-07-07 10:56 ` [PATCH 3/3] test: drop upgrade from v1 tests David Bremner
2020-07-11 20:29 ` v2 port to xapian 1.5 David Bremner
3 siblings, 0 replies; 6+ messages in thread
From: David Bremner @ 2020-07-07 10:56 UTC (permalink / raw)
To: notmuch; +Cc: David Bremner
This will be mandatory as of Xapian 1.5. The API is also more
consistent with the FieldProcessor API, which helps code re-use a bit.
Note that this switches to using the built-in Xapian support for
prefixes on ranges (i.e. deleted code at beginning of
ParseTimeRangeProcessor::operator(), added prefix to constructor).
Another side effect of the migration is that we are generating smaller
queries, using one OP_VALUE_RANGE instead of an AND of two OP_VALUE_*
queries.
---
lib/database-private.h | 6 ++--
lib/database.cc | 21 ++++++-------
lib/parse-time-vrp.cc | 69 +++++++++++++++++++++---------------------
lib/parse-time-vrp.h | 18 +++++------
4 files changed, 56 insertions(+), 58 deletions(-)
diff --git a/lib/database-private.h b/lib/database-private.h
index 87ae1bdf..76359007 100644
--- a/lib/database-private.h
+++ b/lib/database-private.h
@@ -218,9 +218,9 @@ struct _notmuch_database {
unsigned long view;
Xapian::QueryParser *query_parser;
Xapian::TermGenerator *term_gen;
- Xapian::ValueRangeProcessor *value_range_processor;
- Xapian::ValueRangeProcessor *date_range_processor;
- Xapian::ValueRangeProcessor *last_mod_range_processor;
+ Xapian::RangeProcessor *value_range_processor;
+ Xapian::RangeProcessor *date_range_processor;
+ Xapian::RangeProcessor *last_mod_range_processor;
/* XXX it's slightly gross to use two parallel string->string maps
* here, but at least they are small */
diff --git a/lib/database.cc b/lib/database.cc
index 43bfac4e..2f794164 100644
--- a/lib/database.cc
+++ b/lib/database.cc
@@ -385,8 +385,8 @@ _setup_query_field (const prefix_t *prefix, notmuch_database_t *notmuch)
Xapian::FieldProcessor *fp;
if (STRNCMP_LITERAL (prefix->name, "date") == 0)
- fp = (new DateFieldProcessor ())->release ();
- else if (STRNCMP_LITERAL (prefix->name, "query") == 0)
+ fp = (new DateFieldProcessor(NOTMUCH_VALUE_TIMESTAMP))->release ();
+ else if (STRNCMP_LITERAL(prefix->name, "query") == 0)
fp = (new QueryFieldProcessor (*notmuch->query_parser, notmuch))->release ();
else if (STRNCMP_LITERAL (prefix->name, "thread") == 0)
fp = (new ThreadFieldProcessor (*notmuch->query_parser, notmuch))->release ();
@@ -1036,17 +1036,16 @@ notmuch_database_open_verbose (const char *path,
notmuch->query_parser = new Xapian::QueryParser;
notmuch->term_gen = new Xapian::TermGenerator;
notmuch->term_gen->set_stemmer (Xapian::Stem ("english"));
- notmuch->value_range_processor = new Xapian::NumberValueRangeProcessor (NOTMUCH_VALUE_TIMESTAMP);
- notmuch->date_range_processor = new ParseTimeValueRangeProcessor (NOTMUCH_VALUE_TIMESTAMP);
- notmuch->last_mod_range_processor = new Xapian::NumberValueRangeProcessor (NOTMUCH_VALUE_LAST_MOD, "lastmod:");
-
+ notmuch->value_range_processor = new Xapian::NumberRangeProcessor (NOTMUCH_VALUE_TIMESTAMP);
+ notmuch->date_range_processor = new ParseTimeRangeProcessor (NOTMUCH_VALUE_TIMESTAMP, "date:");
+ notmuch->last_mod_range_processor = new Xapian::NumberRangeProcessor (NOTMUCH_VALUE_LAST_MOD, "lastmod:");
notmuch->query_parser->set_default_op (Xapian::Query::OP_AND);
notmuch->query_parser->set_database (*notmuch->xapian_db);
notmuch->query_parser->set_stemmer (Xapian::Stem ("english"));
notmuch->query_parser->set_stemming_strategy (Xapian::QueryParser::STEM_SOME);
- notmuch->query_parser->add_valuerangeprocessor (notmuch->value_range_processor);
- notmuch->query_parser->add_valuerangeprocessor (notmuch->date_range_processor);
- notmuch->query_parser->add_valuerangeprocessor (notmuch->last_mod_range_processor);
+ notmuch->query_parser->add_rangeprocessor (notmuch->value_range_processor);
+ notmuch->query_parser->add_rangeprocessor (notmuch->date_range_processor);
+ notmuch->query_parser->add_rangeprocessor (notmuch->last_mod_range_processor);
for (i = 0; i < ARRAY_SIZE (prefix_table); i++) {
const prefix_t *prefix = &prefix_table[i];
@@ -1401,8 +1400,8 @@ handle_sigalrm (unused (int signal))
*/
notmuch_status_t
notmuch_database_upgrade (notmuch_database_t *notmuch,
- void (*progress_notify)(void *closure,
- double progress),
+ void (*progress_notify) (void *closure,
+ double progress),
void *closure)
{
void *local = talloc_new (NULL);
diff --git a/lib/parse-time-vrp.cc b/lib/parse-time-vrp.cc
index 168d5810..10809aa3 100644
--- a/lib/parse-time-vrp.cc
+++ b/lib/parse-time-vrp.cc
@@ -24,64 +24,63 @@
#include "parse-time-vrp.h"
#include "parse-time-string.h"
-#define PREFIX "date:"
-
-/* See *ValueRangeProcessor in xapian-core/api/valuerangeproc.cc */
-Xapian::valueno
-ParseTimeValueRangeProcessor::operator() (std::string &begin, std::string &end)
+Xapian::Query
+ParseTimeRangeProcessor::operator() (const std::string &begin, const std::string &end)
{
- time_t t, now;
- std::string b;
-
- /* Require date: prefix in start of the range... */
- if (STRNCMP_LITERAL (begin.c_str (), PREFIX))
- return Xapian::BAD_VALUENO;
-
- /* ...and remove it. */
- begin.erase (0, sizeof (PREFIX) - 1);
- b = begin;
+ double from = DBL_MIN, to = DBL_MAX;
+ time_t parsed_time, now;
+ std::string str;
/* Use the same 'now' for begin and end. */
if (time (&now) == (time_t) -1)
- return Xapian::BAD_VALUENO;
-
- if (! begin.empty ()) {
- if (parse_time_string (begin.c_str (), &t, &now, PARSE_TIME_ROUND_DOWN))
- return Xapian::BAD_VALUENO;
+ throw Xapian::QueryParserError ("unable to get current time");
- begin.assign (Xapian::sortable_serialise ((double) t));
+ if (!begin.empty ()) {
+ if (parse_time_string (begin.c_str (), &parsed_time, &now, PARSE_TIME_ROUND_DOWN))
+ throw Xapian::QueryParserError ("Didn't understand date specification '" + begin + "'");
+ else
+ from = (double) parsed_time;
}
- if (! end.empty ()) {
- if (end == "!" && ! b.empty ())
- end = b;
-
- if (parse_time_string (end.c_str (), &t, &now, PARSE_TIME_ROUND_UP_INCLUSIVE))
- return Xapian::BAD_VALUENO;
+ if (!end.empty ()) {
+ if (end == "!" && ! begin.empty ())
+ str = begin;
+ else
+ str = end;
- end.assign (Xapian::sortable_serialise ((double) t));
+ if (parse_time_string (str.c_str (), &parsed_time, &now, PARSE_TIME_ROUND_UP_INCLUSIVE))
+ throw Xapian::QueryParserError ("Didn't understand date specification '" + str + "'");
+ else
+ to = (double) parsed_time;
}
- return valno;
+ return Xapian::Query (Xapian::Query::OP_VALUE_RANGE, slot,
+ Xapian::sortable_serialise (from),
+ Xapian::sortable_serialise (to));
}
/* XXX TODO: is throwing an exception the right thing to do here? */
Xapian::Query
DateFieldProcessor::operator() (const std::string & str)
{
- time_t from, to, now;
+ double from = DBL_MIN, to = DBL_MAX;
+ time_t parsed_time, now;
/* Use the same 'now' for begin and end. */
if (time (&now) == (time_t) -1)
throw Xapian::QueryParserError ("Unable to get current time");
- if (parse_time_string (str.c_str (), &from, &now, PARSE_TIME_ROUND_DOWN))
+ if (parse_time_string (str.c_str (), &parsed_time, &now, PARSE_TIME_ROUND_DOWN))
throw Xapian::QueryParserError ("Didn't understand date specification '" + str + "'");
+ else
+ from = (double) parsed_time;
- if (parse_time_string (str.c_str (), &to, &now, PARSE_TIME_ROUND_UP_INCLUSIVE))
+ if (parse_time_string (str.c_str (), &parsed_time, &now, PARSE_TIME_ROUND_UP_INCLUSIVE))
throw Xapian::QueryParserError ("Didn't understand date specification '" + str + "'");
+ else
+ to = (double) parsed_time;
- return Xapian::Query (Xapian::Query::OP_AND,
- Xapian::Query (Xapian::Query::OP_VALUE_GE, 0, Xapian::sortable_serialise ((double) from)),
- Xapian::Query (Xapian::Query::OP_VALUE_LE, 0, Xapian::sortable_serialise ((double) to)));
+ return Xapian::Query (Xapian::Query::OP_VALUE_RANGE, slot,
+ Xapian::sortable_serialise (from),
+ Xapian::sortable_serialise (to));
}
diff --git a/lib/parse-time-vrp.h b/lib/parse-time-vrp.h
index 9fb1af60..f495e716 100644
--- a/lib/parse-time-vrp.h
+++ b/lib/parse-time-vrp.h
@@ -26,21 +26,21 @@
#include <xapian.h>
/* see *ValueRangeProcessor in xapian-core/include/xapian/queryparser.h */
-class ParseTimeValueRangeProcessor : public Xapian::ValueRangeProcessor {
-protected:
- Xapian::valueno valno;
+class ParseTimeRangeProcessor : public Xapian::RangeProcessor {
public:
- ParseTimeValueRangeProcessor (Xapian::valueno slot_)
- : valno (slot_)
- {
- }
+ ParseTimeRangeProcessor (Xapian::valueno slot_, const std::string prefix_)
+ : Xapian::RangeProcessor(slot_, prefix_, 0) { }
- Xapian::valueno operator() (std::string &begin, std::string &end);
+ Xapian::Query operator() (const std::string &begin, const std::string &end);
};
class DateFieldProcessor : public Xapian::FieldProcessor {
- Xapian::Query operator() (const std::string & str);
+private:
+ Xapian::valueno slot;
+public:
+ DateFieldProcessor(Xapian::valueno slot_) : slot(slot_) { };
+ Xapian::Query operator()(const std::string & str);
};
#endif /* NOTMUCH_PARSE_TIME_VRP_H */
--
2.27.0
^ permalink raw reply related [flat|nested] 6+ messages in thread
* [PATCH 3/3] test: drop upgrade from v1 tests
2020-07-07 10:56 v2 port to xapian 1.5 David Bremner
2020-07-07 10:56 ` [PATCH 1/3] lib: migrate to post Xapian 1.3.4 compact support David Bremner
2020-07-07 10:56 ` [PATCH 2/3] lib: migrate from Xapian ValueRangeProcessor to RangeProcessor David Bremner
@ 2020-07-07 10:56 ` David Bremner
2020-07-11 20:29 ` v2 port to xapian 1.5 David Bremner
3 siblings, 0 replies; 6+ messages in thread
From: David Bremner @ 2020-07-07 10:56 UTC (permalink / raw)
To: notmuch; +Cc: David Bremner
These are less crucial since we stopped generating new database
versions and relied primarily on features. They also rely on a
pre-generated v1 database which happens to be chert format. This
backend is not supported by Xapian 1.5.
Also drop the tool gen-testdb.sh, which is currently broken, due to
changes in the testing infrastructure.
---
.travis.yml | 1 -
configure | 2 +-
devel/gen-testdb.sh | 131 -----------------
test/T530-upgrade.sh | 136 ------------------
test/test-databases/.gitignore | 1 -
test/test-databases/Makefile | 7 -
test/test-databases/Makefile.local | 20 ---
test/test-databases/database-v1.tar.xz.sha256 | 1 -
8 files changed, 1 insertion(+), 298 deletions(-)
delete mode 100755 devel/gen-testdb.sh
delete mode 100755 test/T530-upgrade.sh
delete mode 100644 test/test-databases/.gitignore
delete mode 100644 test/test-databases/Makefile
delete mode 100644 test/test-databases/Makefile.local
delete mode 100644 test/test-databases/database-v1.tar.xz.sha256
diff --git a/.travis.yml b/.travis.yml
index 9dc03619..9dcec1ff 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -20,7 +20,6 @@ addons:
script:
- ./configure
- - make download-test-databases
- make test
notifications:
diff --git a/configure b/configure
index 80cbac4f..4e16ff40 100755
--- a/configure
+++ b/configure
@@ -48,7 +48,7 @@ case $PWD in ( *["$IFS"]* )
esac
subdirs="util compat lib parse-time-string completion doc emacs"
-subdirs="${subdirs} performance-test test test/test-databases"
+subdirs="${subdirs} performance-test test"
subdirs="${subdirs} bindings"
# For a non-srcdir configure invocation (such as ../configure), create
diff --git a/devel/gen-testdb.sh b/devel/gen-testdb.sh
deleted file mode 100755
index 61ae48a3..00000000
--- a/devel/gen-testdb.sh
+++ /dev/null
@@ -1,131 +0,0 @@
-#!/usr/bin/env bash
-#
-# NAME
-# gen-testdb.sh - generate test databases
-#
-# SYNOPSIS
-# gen-testdb.sh -v NOTMUCH-VERSION [-c CORPUS-PATH] [-s TAR-SUFFIX]
-#
-# DESCRIPTION
-# Generate a tarball containing the specified test corpus and
-# the corresponding notmuch database, indexed using a specific
-# version of notmuch, resulting in a specific version of the
-# database.
-#
-# The specific version of notmuch will be built on the fly.
-# Therefore the script must be run within a git repository to be
-# able to build the old versions of notmuch.
-#
-# This script reuses the test infrastructure, and the script
-# must be run from within the test directory.
-#
-# The output tarballs, named database-<TAR-SUFFIX>.tar.gz, are
-# placed in the test/test-databases directory.
-#
-# OPTIONS
-# -v NOTMUCH-VERSION
-# Notmuch version in terms of a git tag or commit to use
-# for generating the database. Required.
-#
-# -c CORPUS-PATH
-# Path to a corpus to use for generating the
-# database. Due to CWD changes within the test
-# infrastructure, use absolute paths. Defaults to the
-# test corpus.
-#
-# -s TAR-SUFFIX
-# Suffix for the tarball basename. Empty by default.
-#
-# EXAMPLE
-#
-# Generate a database indexed with notmuch 0.17. Use the default
-# test corpus. Name the tarball database-v1.tar.gz to reflect
-# the fact that notmuch 0.17 used database version 1.
-#
-# $ cd test
-# $ ../devel/gen-testdb.sh -v 0.17 -s v1
-#
-# CAVEATS
-# Test infrastructure options won't work.
-#
-# Any existing databases with the same name will be overwritten.
-#
-# It may not be possible to build old versions of notmuch with
-# the set of dependencies that satisfy building the current
-# version of notmuch.
-#
-# AUTHOR
-# Jani Nikula <jani@nikula.org>
-#
-# LICENSE
-# Same as notmuch test infrastructure (GPLv2+).
-#
-
-test_description="database generation abusing test infrastructure"
-
-# immediate exit on subtest failure; see test_failure_ in test-lib.sh
-immediate=t
-
-VERSION=
-CORPUS=
-SUFFIX=
-
-while getopts v:c:s: opt; do
- case "$opt" in
- v) VERSION="$OPTARG";;
- c) CORPUS="$OPTARG";;
- s) SUFFIX="-$OPTARG";;
- esac
-done
-shift `expr $OPTIND - 1`
-
-. ./test-lib.sh || exit 1
-
-SHORT_CORPUS=$(basename ${CORPUS:-database})
-DBNAME=${SHORT_CORPUS}${SUFFIX}
-TARBALLNAME=${DBNAME}.tar.xz
-
-CORPUS=${CORPUS:-${TEST_DIRECTORY}/corpus}
-
-test_expect_code 0 "notmuch version specified on the command line" \
- "test -n ${VERSION}"
-
-test_expect_code 0 "the specified version ${VERSION} refers to a commit" \
- "git show ${VERSION} >/dev/null 2>&1"
-
-BUILD_DIR="notmuch-${VERSION}"
-test_expect_code 0 "generate snapshot of notmuch version ${VERSION}" \
- "git -C $TEST_DIRECTORY/.. archive --prefix=${BUILD_DIR}/ --format=tar ${VERSION} | tar x"
-
-# force version string
-git describe --match '[0-9.]*' ${VERSION} > ${BUILD_DIR}/version
-
-test_expect_code 0 "configure and build notmuch version ${VERSION}" \
- "make -C ${BUILD_DIR}"
-
-# use the newly built notmuch
-export PATH=./${BUILD_DIR}:$PATH
-
-test_begin_subtest "verify the newly built notmuch version"
-test_expect_equal "`notmuch --version`" "notmuch `cat ${BUILD_DIR}/version`"
-
-# replace the existing mails, if any, with the specified corpus
-rm -rf ${MAIL_DIR}
-cp -a ${CORPUS} ${MAIL_DIR}
-
-test_expect_code 0 "index the corpus" \
- "notmuch new"
-
-# wrap the resulting mail store and database in a tarball
-
-cp -a ${MAIL_DIR} ${TMP_DIRECTORY}/${DBNAME}
-tar Jcf ${TMP_DIRECTORY}/${TARBALLNAME} -C ${TMP_DIRECTORY} ${DBNAME}
-mkdir -p ${TEST_DIRECTORY}/test-databases
-cp -a ${TMP_DIRECTORY}/${TARBALLNAME} ${TEST_DIRECTORY}/test-databases
-test_expect_code 0 "create the output tarball ${TARBALLNAME}" \
- "test -f ${TEST_DIRECTORY}/test-databases/${TARBALLNAME}"
-
-# generate a checksum file
-test_expect_code 0 "compute checksum" \
- "(cd ${TEST_DIRECTORY}/test-databases/ && sha256sum ${TARBALLNAME} > ${TARBALLNAME}.sha256)"
-test_done
diff --git a/test/T530-upgrade.sh b/test/T530-upgrade.sh
deleted file mode 100755
index 2124dde2..00000000
--- a/test/T530-upgrade.sh
+++ /dev/null
@@ -1,136 +0,0 @@
-#!/usr/bin/env bash
-test_description="database upgrade"
-
-. $(dirname "$0")/test-lib.sh || exit 1
-
-dbtarball=database-v1.tar.xz
-
-# XXX: Accomplish the same with test lib helpers
-if [ ! -e ${TEST_DIRECTORY}/test-databases/${dbtarball} ]; then
- test_subtest_missing_external_prereq_["${dbtarball} - fetch with 'make download-test-databases'"]=t
-fi
-
-test_begin_subtest "database checksum"
-test_expect_success \
- '( cd $TEST_DIRECTORY/test-databases &&
- sha256sum --quiet --check --status ${dbtarball}.sha256 )'
-
-tar xf $TEST_DIRECTORY/test-databases/${dbtarball} -C ${MAIL_DIR} --strip-components=1
-
-test_begin_subtest "folder: search does not work with old database version"
-output=$(notmuch search folder:foo)
-test_expect_equal "$output" ""
-
-test_begin_subtest "path: search does not work with old database version"
-output=$(notmuch search path:foo)
-test_expect_equal "$output" ""
-
-test_begin_subtest "pre upgrade dump"
-test_expect_success 'notmuch dump | sort > pre-upgrade-dump'
-
-test_begin_subtest "database upgrade from format version 1"
-output=$(notmuch new | sed -e 's/^Backing up tags to .*$/Backing up tags to FILENAME/')
-test_expect_equal "$output" "\
-Welcome to a new version of notmuch! Your database will now be upgraded.
-This process is safe to interrupt.
-Backing up tags to FILENAME
-Your notmuch database has now been upgraded.
-No new mail."
-
-test_begin_subtest "tag backup matches pre-upgrade dump"
-gunzip -c ${MAIL_DIR}/.notmuch/dump-*.gz | sort > backup-dump
-test_expect_equal_file pre-upgrade-dump backup-dump
-
-test_begin_subtest "folder: no longer matches in the middle of path"
-output=$(notmuch search folder:baz)
-test_expect_equal "$output" ""
-
-test_begin_subtest "folder: search"
-output=$(notmuch search --output=files folder:foo | notmuch_search_files_sanitize | sort)
-test_expect_equal "$output" "MAIL_DIR/foo/06:2,
-MAIL_DIR/foo/cur/07:2,
-MAIL_DIR/foo/cur/08:2,
-MAIL_DIR/foo/new/03:2,
-MAIL_DIR/foo/new/09:2,
-MAIL_DIR/foo/new/10:2,"
-
-test_begin_subtest "top level folder: search"
-output=$(notmuch search --output=files folder:'""' | notmuch_search_files_sanitize | sort)
-# bar/18:2, is a duplicate of cur/51:2,
-test_expect_equal "$output" "MAIL_DIR/01:2,
-MAIL_DIR/02:2,
-MAIL_DIR/bar/18:2,
-MAIL_DIR/cur/29:2,
-MAIL_DIR/cur/30:2,
-MAIL_DIR/cur/31:2,
-MAIL_DIR/cur/32:2,
-MAIL_DIR/cur/33:2,
-MAIL_DIR/cur/34:2,
-MAIL_DIR/cur/35:2,
-MAIL_DIR/cur/36:2,
-MAIL_DIR/cur/37:2,
-MAIL_DIR/cur/38:2,
-MAIL_DIR/cur/39:2,
-MAIL_DIR/cur/40:2,
-MAIL_DIR/cur/41:2,
-MAIL_DIR/cur/42:2,
-MAIL_DIR/cur/43:2,
-MAIL_DIR/cur/44:2,
-MAIL_DIR/cur/45:2,
-MAIL_DIR/cur/46:2,
-MAIL_DIR/cur/47:2,
-MAIL_DIR/cur/48:2,
-MAIL_DIR/cur/49:2,
-MAIL_DIR/cur/50:2,
-MAIL_DIR/cur/51:2,
-MAIL_DIR/cur/52:2,
-MAIL_DIR/cur/53:2,
-MAIL_DIR/new/04:2,"
-
-test_begin_subtest "path: search"
-output=$(notmuch search --output=files path:"bar" | notmuch_search_files_sanitize | sort)
-# cur/51:2, is a duplicate of bar/18:2,
-test_expect_equal "$output" "MAIL_DIR/bar/17:2,
-MAIL_DIR/bar/18:2,
-MAIL_DIR/cur/51:2,"
-
-test_begin_subtest "top level path: search"
-output=$(notmuch search --output=files path:'""' | notmuch_search_files_sanitize | sort)
-test_expect_equal "$output" "MAIL_DIR/01:2,
-MAIL_DIR/02:2,"
-
-test_begin_subtest "recursive path: search"
-output=$(notmuch search --output=files path:"bar/**" | notmuch_search_files_sanitize | sort)
-# cur/51:2, is a duplicate of bar/18:2,
-test_expect_equal "$output" "MAIL_DIR/bar/17:2,
-MAIL_DIR/bar/18:2,
-MAIL_DIR/bar/baz/05:2,
-MAIL_DIR/bar/baz/23:2,
-MAIL_DIR/bar/baz/24:2,
-MAIL_DIR/bar/baz/cur/25:2,
-MAIL_DIR/bar/baz/cur/26:2,
-MAIL_DIR/bar/baz/new/27:2,
-MAIL_DIR/bar/baz/new/28:2,
-MAIL_DIR/bar/cur/19:2,
-MAIL_DIR/bar/cur/20:2,
-MAIL_DIR/bar/new/21:2,
-MAIL_DIR/bar/new/22:2,
-MAIL_DIR/cur/51:2,"
-
-test_begin_subtest "body: same as unprefixed before reindex"
-notmuch search --output=messages body:close > OUTPUT
-notmuch search --output=messages close > EXPECTED
-test_expect_equal_file EXPECTED OUTPUT
-
-test_begin_subtest "body: subset of unprefixed after reindex"
-notmuch reindex '*'
-notmuch search --output=messages body:close | sort > BODY
-notmuch search --output=messages close | sort > UNPREFIXED
-diff -e UNPREFIXED BODY | cut -c2- > OUTPUT
-cat <<EOF > EXPECTED
-d
-d
-EOF
-test_expect_equal_file EXPECTED OUTPUT
-
-test_done
diff --git a/test/test-databases/.gitignore b/test/test-databases/.gitignore
deleted file mode 100644
index 9452199f..00000000
--- a/test/test-databases/.gitignore
+++ /dev/null
@@ -1 +0,0 @@
-/*.tar.xz
diff --git a/test/test-databases/Makefile b/test/test-databases/Makefile
deleted file mode 100644
index b250a8be..00000000
--- a/test/test-databases/Makefile
+++ /dev/null
@@ -1,7 +0,0 @@
-# See Makefile.local for the list of files to be compiled in this
-# directory.
-all:
- $(MAKE) -C ../.. all
-
-.DEFAULT:
- $(MAKE) -C ../.. $@
diff --git a/test/test-databases/Makefile.local b/test/test-databases/Makefile.local
deleted file mode 100644
index 7aedff70..00000000
--- a/test/test-databases/Makefile.local
+++ /dev/null
@@ -1,20 +0,0 @@
-# -*- makefile -*-
-
-TEST_DATABASE_MIRROR=https://notmuchmail.org/releases/test-databases
-
-dir := test/test-databases
-
-test_databases := $(dir)/database-v1.tar.xz
-
-%.tar.xz:
- @exec 1>&2 ;\
- if command -v wget >/dev/null ;\
- then set -x; wget -nv -O $@ ${TEST_DATABASE_MIRROR}/$(notdir $@) ;\
- elif command -v curl >/dev/null ;\
- then set -x; curl -L -s -o $@ ${TEST_DATABASE_MIRROR}/$(notdir $@) ;\
- else echo Cannot fetch databases, no wget nor curl available; exit 1 ;\
- fi
-
-download-test-databases: ${test_databases}
-
-DATACLEAN := $(DATACLEAN) ${test_databases}
diff --git a/test/test-databases/database-v1.tar.xz.sha256 b/test/test-databases/database-v1.tar.xz.sha256
deleted file mode 100644
index 2cc4f965..00000000
--- a/test/test-databases/database-v1.tar.xz.sha256
+++ /dev/null
@@ -1 +0,0 @@
-4299e051b10e1fa7b33ea2862790a09ebfe96859681804e5251e130f800e69d2 database-v1.tar.xz
--
2.27.0
^ permalink raw reply related [flat|nested] 6+ messages in thread
* Re: v2 port to xapian 1.5
2020-07-07 10:56 v2 port to xapian 1.5 David Bremner
` (2 preceding siblings ...)
2020-07-07 10:56 ` [PATCH 3/3] test: drop upgrade from v1 tests David Bremner
@ 2020-07-11 20:29 ` David Bremner
3 siblings, 0 replies; 6+ messages in thread
From: David Bremner @ 2020-07-11 20:29 UTC (permalink / raw)
To: notmuch
David Bremner <david@tethera.net> writes:
> This obsoletes id:20200617110441.1262683-1-david@tethera.net
>
> I have updated patch 2/3 based on discussions with Olly and Tomi.
>
> Olly gave the previous series thumbs up on IRC, so I'll probably merge
> this version if I don't here any corrections.
Seeing none, merged.
d
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2020-07-11 20:30 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-07-07 10:56 v2 port to xapian 1.5 David Bremner
2020-07-07 10:56 ` [PATCH 1/3] lib: migrate to post Xapian 1.3.4 compact support David Bremner
2020-07-07 10:56 ` [PATCH 2/3] lib: migrate from Xapian ValueRangeProcessor to RangeProcessor David Bremner
2020-07-07 10:56 ` [PATCH 3/3] test: drop upgrade from v1 tests David Bremner
2020-07-11 20:29 ` v2 port to xapian 1.5 David Bremner
-- strict thread matches above, loose matches on Subject: below --
2020-06-17 11:04 Port to Xapian 1.5 David Bremner
2020-06-17 11:04 ` [PATCH 3/3] test: drop upgrade from v1 tests David Bremner
Code repositories for project(s) associated with this public inbox
https://yhetil.org/notmuch.git/
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).