field processor features, merged series

field processor features, merged series

[David Bremner]

This series merges and obsoletes

     id:1458437904-17677-1-git-send-email-david@tethera.net

and all except the last patch

    id:1458437904-17677-1-git-send-email-david@tethera.net

I represents an alternative initial motivation for the library config patches

The first 4 are independent, and provide date:foo queries without the trailing ..

 [PATCH 01/13] config: autodetect xapian-1.3
 [PATCH 02/13] configure: detect Xapian:FieldProcessor
 [PATCH 03/13] lib: optionally support single argument date: queries
 [PATCH 04/13] lib/cli: add library API / CLI for compile time options

This is just an optional cleanup, here to avoid conflicts.

 [PATCH 05/13] configure: check directly for xapian compaction API

This is the previous library config series, rebased. This could be
merged without the following 3, although the motivation is maybe a bit
unclear.

 [PATCH 06/13] lib: provide config API
 [PATCH 07/13] lib: config list iterators
 [PATCH 08/13] CLI: add print_status_database
 [PATCH 09/13] CLI: add optional config data to dump output.
 [PATCH 10/13] CLI: optionally restore config data.

Finally, add support for "named queries". I'm open to suggestions
about naming ("the hardest problem...") but I thought it was a bit
less confusing than "saved searches".

 [PATCH 11/13] CLI: add notmuch-config support for named queries
 [PATCH 12/13] lib: make a global constant for query parser flags
 [PATCH 13/13] lib: add support for named queries

Outstanding issues that I know about

 - missing docs for named queries. It might be nice to have optionally
   included sections in the docs. Or we could just write in something
   like "These search terms are only available if you have field
   processors. `notmuch config get option.field_processor` to find
   out.

- compiler warnings about visibility. I suspect I'm somehow failing at
  C++ here

- compiler warnings about deprecated compact API. This is only related
  in the sense that it is triggered by using xapian 1.3

- as I write this, I realise I probably need to test these patches
  again with xapian 1.2

[PATCH 01/13] config: autodetect xapian-1.3

[David Bremner]

Mimic the handling of python2 versus python3. In particular if both
xapian-config and xapian-config-1.3 are found, use xapian-config
---
 configure | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/configure b/configure
index a79f6bd..eb6dbac 100755
--- a/configure
+++ b/configure
@@ -51,7 +51,7 @@ CPPFLAGS=${CPPFLAGS:-}
 CXXFLAGS_for_sh=${CXXFLAGS:-${CFLAGS}}
 CXXFLAGS=${CXXFLAGS:-\$(CFLAGS)}
 LDFLAGS=${LDFLAGS:-}
-XAPIAN_CONFIG=${XAPIAN_CONFIG:-xapian-config}
+XAPIAN_CONFIG=${XAPIAN_CONFIG:-}
 PYTHON=${PYTHON:-}
 
 # We don't allow the EMACS or GZIP Makefile variables inherit values
@@ -341,7 +341,7 @@ fi
 
 printf "Checking for Xapian development files... "
 have_xapian=0
-for xapian_config in ${XAPIAN_CONFIG}; do
+for xapian_config in ${XAPIAN_CONFIG} xapian-config xapian-config-1.3; do
     if ${xapian_config} --version > /dev/null 2>&1; then
 	xapian_version=$(${xapian_config} --version | sed -e 's/.* //')
 	printf "Yes (%s).\n" ${xapian_version}
-- 
2.7.0

[PATCH 02/13] configure: detect Xapian:FieldProcessor

[David Bremner]

Rather than check versions, it seems more robust to do a test compile.
---
 configure | 26 +++++++++++++++++++++++++-
 1 file changed, 25 insertions(+), 1 deletion(-)

diff --git a/configure b/configure
index eb6dbac..c48f7ce 100755
--- a/configure
+++ b/configure
@@ -356,9 +356,10 @@ if [ ${have_xapian} = "0" ]; then
     errors=$((errors + 1))
 fi
 
-# Compaction is only supported on Xapian > 1.2.6
 have_xapian_compact=0
+have_xapian_field_processor=0
 if [ ${have_xapian} = "1" ]; then
+    # Compaction is only supported on Xapian > 1.2.6
     printf "Checking for Xapian compaction support... "
     case "${xapian_version}" in
 	0.*|1.[01].*|1.2.[0-5])
@@ -369,6 +370,21 @@ if [ ${have_xapian} = "1" ]; then
 	*)
 	    printf "Unknown version.\n" ;;
     esac
+
+    printf "Checking for Xapian FieldProcessor API... "
+    cat>_field_processor.cc<<EOF
+#include <xapian.h>
+class TitleFieldProcessor : public Xapian::FieldProcessor { };
+EOF
+    if ${CXX} ${CXXLAGS} ${xapian_cxxflags} -c _field_processor.cc -o _field_processor.o > /dev/null 2>&1
+    then
+	have_xapian_field_processor=1
+	printf "Yes.\n"
+    else
+	printf "No. (optional)\n"
+    fi
+
+    rm -f _field_processor.o _field_processor.cc
 fi
 
 
@@ -979,6 +995,9 @@ HAVE_D_TYPE = ${have_d_type}
 # Whether the Xapian version in use supports compaction
 HAVE_XAPIAN_COMPACT = ${have_xapian_compact}
 
+# Whether the Xapian version in use supports field processors
+HAVE_XAPIAN_FIELD_PROCESSOR = ${have_xapian_field_processor}
+
 # Whether the getpwuid_r function is standards-compliant
 # (if not, then notmuch will #define _POSIX_PTHREAD_SEMANTICS
 # to enable the standards-compliant version -- needed for Solaris)
@@ -1050,6 +1069,7 @@ CONFIGURE_CFLAGS = -DHAVE_GETLINE=\$(HAVE_GETLINE) \$(GMIME_CFLAGS)      \\
 		   -DSTD_GETPWUID=\$(STD_GETPWUID)                       \\
 		   -DSTD_ASCTIME=\$(STD_ASCTIME)                         \\
 		   -DHAVE_XAPIAN_COMPACT=\$(HAVE_XAPIAN_COMPACT)	 \\
+		   -DHAVE_XAPIAN_FIELD_PROCESSOR=\$(HAVE_XAPIAN_PROCESSOR) \\
 		   -DUTIL_BYTE_ORDER=\$(UTIL_BYTE_ORDER)
 
 CONFIGURE_CXXFLAGS = -DHAVE_GETLINE=\$(HAVE_GETLINE) \$(GMIME_CFLAGS)    \\
@@ -1064,6 +1084,7 @@ CONFIGURE_CXXFLAGS = -DHAVE_GETLINE=\$(HAVE_GETLINE) \$(GMIME_CFLAGS)    \\
 		     -DSTD_GETPWUID=\$(STD_GETPWUID)                     \\
 		     -DSTD_ASCTIME=\$(STD_ASCTIME)                       \\
 		     -DHAVE_XAPIAN_COMPACT=\$(HAVE_XAPIAN_COMPACT)       \\
+		     -DHAVE_XAPIAN_FIELD_PROCESSOR=\$(HAVE_XAPIAN_FIELD_PROCESSOR) \\
 		     -DUTIL_BYTE_ORDER=\$(UTIL_BYTE_ORDER)
 
 CONFIGURE_LDFLAGS =  \$(GMIME_LDFLAGS) \$(TALLOC_LDFLAGS) \$(ZLIB_LDFLAGS) \$(XAPIAN_LDFLAGS)
@@ -1077,6 +1098,9 @@ cat > sh.config <<EOF
 # Whether the Xapian version in use supports compaction
 NOTMUCH_HAVE_XAPIAN_COMPACT=${have_xapian_compact}
 
+# Whether the Xapian version in use supports field processors
+NOTMUCH_HAVE_XAPIAN_FIELD_PROCESSOR=${have_xapian_field_processor}
+
 # do we have man pages?
 NOTMUCH_HAVE_MAN=$((have_sphinx))
 
-- 
2.7.0

[PATCH 03/13] lib: optionally support single argument date: queries

[David Bremner]

This relies on the FieldProcessor API, which is only present in xapian
>= 1.3.
---
 lib/database-private.h   |  6 ++++++
 lib/database.cc          |  6 ++++++
 lib/parse-time-vrp.cc    | 21 +++++++++++++++++++++
 lib/parse-time-vrp.h     |  5 +++++
 test/T500-search-date.sh |  6 ++++++
 5 files changed, 44 insertions(+)

diff --git a/lib/database-private.h b/lib/database-private.h
index 3fb10f7..30a045e 100644
--- a/lib/database-private.h
+++ b/lib/database-private.h
@@ -144,6 +144,9 @@ operator&=(_notmuch_features &a, _notmuch_features b)
     return a;
 }
 
+#if HAVE_XAPIAN_FIELD_PROCESSOR
+class DateFieldProcessor;
+#endif
 struct _notmuch_database {
     notmuch_bool_t exception_reported;
 
@@ -176,6 +179,9 @@ struct _notmuch_database {
     Xapian::TermGenerator *term_gen;
     Xapian::ValueRangeProcessor *value_range_processor;
     Xapian::ValueRangeProcessor *date_range_processor;
+#if HAVE_XAPIAN_FIELD_PROCESSOR
+    DateFieldProcessor *date_field_processor;
+#endif
     Xapian::ValueRangeProcessor *last_mod_range_processor;
 };
 
diff --git a/lib/database.cc b/lib/database.cc
index 3b342f1..0e3c90c 100644
--- a/lib/database.cc
+++ b/lib/database.cc
@@ -1000,6 +1000,12 @@ notmuch_database_open_verbose (const char *path,
 	notmuch->term_gen->set_stemmer (Xapian::Stem ("english"));
 	notmuch->value_range_processor = new Xapian::NumberValueRangeProcessor (NOTMUCH_VALUE_TIMESTAMP);
 	notmuch->date_range_processor = new ParseTimeValueRangeProcessor (NOTMUCH_VALUE_TIMESTAMP);
+#if HAVE_XAPIAN_FIELD_PROCESSOR
+	/* This currently relies on the query parser to pass anything
+	 * with a .. to the range processor */
+	notmuch->date_field_processor = new DateFieldProcessor();
+	notmuch->query_parser->add_boolean_prefix("date", notmuch->date_field_processor);
+#endif
 	notmuch->last_mod_range_processor = new Xapian::NumberValueRangeProcessor (NOTMUCH_VALUE_LAST_MOD, "lastmod:");
 
 	notmuch->query_parser->set_default_op (Xapian::Query::OP_AND);
diff --git a/lib/parse-time-vrp.cc b/lib/parse-time-vrp.cc
index 03804cf..b15b77c 100644
--- a/lib/parse-time-vrp.cc
+++ b/lib/parse-time-vrp.cc
@@ -64,3 +64,24 @@ ParseTimeValueRangeProcessor::operator() (std::string &begin, std::string &end)
 
     return valno;
 }
+
+#if HAVE_XAPIAN_FIELD_PROCESSOR
+/* XXX TODO: is throwing an exception the right thing to do here? */
+Xapian::Query DateFieldProcessor::operator()(const std::string & str) {
+    time_t from, to, now;
+
+    /* Use the same 'now' for begin and end. */
+    if (time (&now) == (time_t) -1)
+	throw Xapian::QueryParserError("Unable to get current time");
+
+    if (parse_time_string (str.c_str (), &from, &now, PARSE_TIME_ROUND_DOWN))
+	throw Xapian::QueryParserError ("Didn't understand date specification '" + str + "'");
+
+    if (parse_time_string (str.c_str (), &to, &now, PARSE_TIME_ROUND_UP_INCLUSIVE))
+	throw Xapian::QueryParserError ("Didn't understand date specification '" + str + "'");
+
+    return Xapian::Query(Xapian::Query::OP_AND,
+			 Xapian::Query(Xapian::Query::OP_VALUE_GE, 0, Xapian::sortable_serialise ((double) from)),
+			 Xapian::Query(Xapian::Query::OP_VALUE_LE, 0, Xapian::sortable_serialise ((double) to)));
+}
+#endif
diff --git a/lib/parse-time-vrp.h b/lib/parse-time-vrp.h
index 094c4f8..3bd12bf 100644
--- a/lib/parse-time-vrp.h
+++ b/lib/parse-time-vrp.h
@@ -37,4 +37,9 @@ public:
     Xapian::valueno operator() (std::string &begin, std::string &end);
 };
 
+#if HAVE_XAPIAN_FIELD_PROCESSOR
+class DateFieldProcessor : public Xapian::FieldProcessor {
+    Xapian::Query operator()(const std::string & str);
+};
+#endif
 #endif /* NOTMUCH_PARSE_TIME_VRP_H */
diff --git a/test/T500-search-date.sh b/test/T500-search-date.sh
index f5cea42..198a2e6 100755
--- a/test/T500-search-date.sh
+++ b/test/T500-search-date.sh
@@ -12,6 +12,12 @@ test_begin_subtest "Absolute date range with 'same' operator"
 output=$(notmuch search date:2010-12-16..! | notmuch_search_sanitize)
 test_expect_equal "$output" "thread:XXX   2010-12-16 [1/1] Olivier Berger; Essai accentué (inbox unread)"
 
+if [ "${NOTMUCH_HAVE_XAPIAN_FIELD_PROCESSOR}" = "1" ]; then
+    test_begin_subtest "Absolute date field"
+    output=$(notmuch search date:2010-12-16 | notmuch_search_sanitize)
+    test_expect_equal "$output" "thread:XXX   2010-12-16 [1/1] Olivier Berger; Essai accentué (inbox unread)"
+fi
+
 test_begin_subtest "Absolute time range with TZ"
 notmuch search date:18-Nov-2009_02:19:26-0800..2009-11-18_04:49:52-06:00 | notmuch_search_sanitize > OUTPUT
 cat <<EOF >EXPECTED
-- 
2.7.0

[PATCH 04/13] lib/cli: add library API / CLI for compile time options

[David Bremner]

This is intentionally low tech; if we have more than two options it may
make sense to build up what infrastructure is provided.
---
 lib/Makefile.local  |  1 +
 lib/notmuch.h       | 10 ++++++++++
 lib/options.c       | 50 ++++++++++++++++++++++++++++++++++++++++++++++++++
 notmuch-config.c    | 20 ++++++++++++++++++++
 test/T030-config.sh |  6 ++++--
 test/T040-setup.sh  |  6 ++++--
 test/test-lib.sh    |  6 ++++++
 7 files changed, 95 insertions(+), 4 deletions(-)
 create mode 100644 lib/options.c

diff --git a/lib/Makefile.local b/lib/Makefile.local
index 3a07090..4ad0158 100644
--- a/lib/Makefile.local
+++ b/lib/Makefile.local
@@ -39,6 +39,7 @@ libnotmuch_c_srcs =		\
 	$(dir)/message-file.c	\
 	$(dir)/messages.c	\
 	$(dir)/sha1.c		\
+	$(dir)/options.c	\
 	$(dir)/tags.c
 
 libnotmuch_cxx_srcs =		\
diff --git a/lib/notmuch.h b/lib/notmuch.h
index cb46fc0..b29dd5f 100644
--- a/lib/notmuch.h
+++ b/lib/notmuch.h
@@ -1838,6 +1838,16 @@ notmuch_filenames_move_to_next (notmuch_filenames_t *filenames);
 void
 notmuch_filenames_destroy (notmuch_filenames_t *filenames);
 
+typedef enum {
+    NOTMUCH_OPTION_COMPACT = 1,
+    NOTMUCH_OPTION_FIELD_PROCESSOR = 2
+} notmuch_option_t;
+
+notmuch_bool_t
+notmuch_options_present (notmuch_option_t mask);
+
+notmuch_bool_t
+notmuch_options_get (const char *name);
 /* @} */
 
 NOTMUCH_END_DECLS
diff --git a/lib/options.c b/lib/options.c
new file mode 100644
index 0000000..4e15d92
--- /dev/null
+++ b/lib/options.c
@@ -0,0 +1,50 @@
+/* notmuch - Not much of an email program, (just index and search)
+ *
+ * Copyright © 2016 David Bremner
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation, either version 3 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see http://www.gnu.org/licenses/ .
+ *
+ * Author: David Bremner <david@tethera.net>
+ */
+
+#include "notmuch.h"
+#include "notmuch-private.h"
+
+notmuch_bool_t
+notmuch_options_present (notmuch_option_t mask)
+{
+    notmuch_option_t present = 0;
+
+#if HAVE_XAPIAN_COMPACT
+    present |= NOTMUCH_OPTION_COMPACT;
+#endif
+
+#if HAVE_XAPIAN_COMPACT
+    present |= NOTMUCH_OPTION_FIELD_PROCESSOR;
+#endif
+
+    return (mask & present) != 0;
+
+}
+
+notmuch_bool_t
+notmuch_options_get (const char *name) {
+    if (STRNCMP_LITERAL (name, "compact") == 0) {
+	return notmuch_options_present (NOTMUCH_OPTION_COMPACT);
+    } else if (STRNCMP_LITERAL (name, "field_processor") == 0) {
+	return notmuch_options_present (NOTMUCH_OPTION_FIELD_PROCESSOR);
+    } else {
+	return FALSE;
+    }
+}
diff --git a/notmuch-config.c b/notmuch-config.c
index d252bb2..cfc549d 100644
--- a/notmuch-config.c
+++ b/notmuch-config.c
@@ -750,6 +750,8 @@ _item_split (char *item, char **group, char **key)
     return 0;
 }
 
+#define OPTION_PREFIX "options."
+
 static int
 notmuch_config_command_get (notmuch_config_t *config, char *item)
 {
@@ -773,6 +775,9 @@ notmuch_config_command_get (notmuch_config_t *config, char *item)
 	tags = notmuch_config_get_new_tags (config, &length);
 	for (i = 0; i < length; i++)
 	    printf ("%s\n", tags[i]);
+    } else if (STRNCMP_LITERAL (item, OPTION_PREFIX) == 0) {
+	printf ("%s\n",
+	       notmuch_options_get (item + strlen (OPTION_PREFIX)) ? "true" : "false");
     } else {
 	char **value;
 	size_t i, length;
@@ -804,6 +809,11 @@ notmuch_config_command_set (notmuch_config_t *config, char *item, int argc, char
 {
     char *group, *key;
 
+    if (STRNCMP_LITERAL (item, OPTION_PREFIX) == 0) {
+	fprintf (stderr, "Error: read only option: %s\n", item);
+	return 1;
+    }
+
     if (_item_split (item, &group, &key))
 	return 1;
 
@@ -830,6 +840,15 @@ notmuch_config_command_set (notmuch_config_t *config, char *item, int argc, char
     return notmuch_config_save (config);
 }
 
+static
+void
+_notmuch_config_list_options () {
+    printf("options.compact=%s\n",
+	   notmuch_options_present(NOTMUCH_OPTION_COMPACT) ? "true" : "false");
+    printf("options.field_processor=%s\n",
+	   notmuch_options_present(NOTMUCH_OPTION_FIELD_PROCESSOR) ? "true" : "false");
+}
+
 static int
 notmuch_config_command_list (notmuch_config_t *config)
 {
@@ -865,6 +884,7 @@ notmuch_config_command_list (notmuch_config_t *config)
 
     g_strfreev (groups);
 
+    _notmuch_config_list_options ();
     return 0;
 }
 
diff --git a/test/T030-config.sh b/test/T030-config.sh
index f404908..c37ba21 100755
--- a/test/T030-config.sh
+++ b/test/T030-config.sh
@@ -44,7 +44,7 @@ test_expect_equal "$(notmuch config get foo.nonexistent)" ""
 
 test_begin_subtest "List all items"
 notmuch config set database.path "/canonical/path"
-output=$(notmuch config list)
+output=$(notmuch config list | notmuch_options_sanitize)
 test_expect_equal "$output" "\
 database.path=/canonical/path
 user.name=Notmuch Test Suite
@@ -56,7 +56,9 @@ search.exclude_tags=
 maildir.synchronize_flags=true
 crypto.gpg_path=gpg
 foo.string=this is another string value
-foo.list=this;is another;list value;"
+foo.list=this;is another;list value;
+options.compact=something
+options.field_processor=something"
 
 test_begin_subtest "Top level --config=FILE option"
 cp "${NOTMUCH_CONFIG}" alt-config
diff --git a/test/T040-setup.sh b/test/T040-setup.sh
index cf0c00b..5db03a6 100755
--- a/test/T040-setup.sh
+++ b/test/T040-setup.sh
@@ -19,7 +19,7 @@ another.suite@example.com
 foo bar
 baz
 EOF
-output=$(notmuch --config=new-notmuch-config config list)
+output=$(notmuch --config=new-notmuch-config config list | notmuch_options_sanitize)
 test_expect_equal "$output" "\
 database.path=/path/to/maildir
 user.name=Test Suite
@@ -29,6 +29,8 @@ new.tags=foo;bar;
 new.ignore=
 search.exclude_tags=baz;
 maildir.synchronize_flags=true
-crypto.gpg_path=gpg"
+crypto.gpg_path=gpg
+options.compact=something
+options.field_processor=something"
 
 test_done
diff --git a/test/test-lib.sh b/test/test-lib.sh
index cc08a98..49e6f40 100644
--- a/test/test-lib.sh
+++ b/test/test-lib.sh
@@ -733,6 +733,12 @@ notmuch_uuid_sanitize ()
 {
     sed 's/[0-9a-f]\{8\}-[0-9a-f]\{4\}-[0-9a-f]\{4\}-[0-9a-f]\{4\}-[0-9a-f]\{12\}/UUID/g'
 }
+
+notmuch_options_sanitize ()
+{
+    sed 's/^options[.]\(.*\)=.*$/options.\1=something/'
+}
+
 # End of notmuch helper functions
 
 # Use test_set_prereq to tell that a particular prerequisite is available.
-- 
2.7.0

[PATCH 05/13] configure: check directly for xapian compaction API

[David Bremner]

This is consistent with the check for FieldProcessor, and probably a bit
more robust.
---
 configure | 21 +++++++++++----------
 1 file changed, 11 insertions(+), 10 deletions(-)

diff --git a/configure b/configure
index c48f7ce..03f28d5 100755
--- a/configure
+++ b/configure
@@ -359,17 +359,18 @@ fi
 have_xapian_compact=0
 have_xapian_field_processor=0
 if [ ${have_xapian} = "1" ]; then
-    # Compaction is only supported on Xapian > 1.2.6
     printf "Checking for Xapian compaction support... "
-    case "${xapian_version}" in
-	0.*|1.[01].*|1.2.[0-5])
-	    printf "No (only available with Xapian > 1.2.6).\n" ;;
-	[1-9]*.[0-9]*.[0-9]*)
-	    have_xapian_compact=1
-	    printf "Yes.\n" ;;
-	*)
-	    printf "Unknown version.\n" ;;
-    esac
+    cat>_compact.cc<<EOF
+#include <xapian.h>
+class TestCompactor : public Xapian::Compactor { };
+EOF
+    if ${CXX} ${CXXLAGS} ${xapian_cxxflags} -c _compact.cc -o _compact.o > /dev/null 2>&1
+    then
+	have_xapian_compact=1
+	printf "Yes.\n"
+    else
+	printf "No.\n"
+    fi
 
     printf "Checking for Xapian FieldProcessor API... "
     cat>_field_processor.cc<<EOF
-- 
2.7.0

[PATCH 06/13] lib: provide config API

[David Bremner]

This is a thin wrapper around the Xapian metadata API. The job of this
layer is to keep the config key value pairs from colliding with other
metadata by transparently prefixing the keys, along with the usual glue
to provide a C interface.

The split of _get_config into two functions is to allow returning of the
return value with different memory ownership semantics.
---
 lib/Makefile.local     |  1 +
 lib/config.cc          | 90 ++++++++++++++++++++++++++++++++++++++++++++++++++
 lib/notmuch.h          | 20 +++++++++++
 test/T590-libconfig.sh | 58 ++++++++++++++++++++++++++++++++
 4 files changed, 169 insertions(+)
 create mode 100644 lib/config.cc
 create mode 100755 test/T590-libconfig.sh

diff --git a/lib/Makefile.local b/lib/Makefile.local
index 4ad0158..96899bf 100644
--- a/lib/Makefile.local
+++ b/lib/Makefile.local
@@ -49,6 +49,7 @@ libnotmuch_cxx_srcs =		\
 	$(dir)/index.cc		\
 	$(dir)/message.cc	\
 	$(dir)/query.cc		\
+	$(dir)/config.cc	\
 	$(dir)/thread.cc
 
 libnotmuch_modules := $(libnotmuch_c_srcs:.c=.o) $(libnotmuch_cxx_srcs:.cc=.o)
diff --git a/lib/config.cc b/lib/config.cc
new file mode 100644
index 0000000..af00d6f
--- /dev/null
+++ b/lib/config.cc
@@ -0,0 +1,90 @@
+/* metadata.cc - API for database metadata
+ *
+ * Copyright © 2015 David Bremner
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation, either version 3 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see http://www.gnu.org/licenses/ .
+ *
+ * Author: David Bremner <david@tethera.net>
+ */
+
+#include "notmuch.h"
+#include "notmuch-private.h"
+#include "database-private.h"
+
+static const std::string CONFIG_PREFIX="C";
+
+notmuch_status_t
+notmuch_database_set_config (notmuch_database_t *notmuch,
+			     const char *key,
+			     const char *value)
+{
+    notmuch_status_t status;
+    Xapian::WritableDatabase *db;
+
+    status = _notmuch_database_ensure_writable (notmuch);
+    if (status)
+	return status;
+
+    try {
+	db = static_cast <Xapian::WritableDatabase *> (notmuch->xapian_db);
+	db->set_metadata (CONFIG_PREFIX+key, value);
+    } catch (const Xapian::Error &error) {
+	status = NOTMUCH_STATUS_XAPIAN_EXCEPTION;
+	notmuch->exception_reported = TRUE;
+	if (! notmuch->exception_reported) {
+	    _notmuch_database_log (notmuch, "Error: A Xapian exception occurred setting metadata: %s\n",
+				   error.get_msg().c_str());
+	}
+    }
+    return NOTMUCH_STATUS_SUCCESS;
+}
+
+static notmuch_status_t
+_metadata_value (notmuch_database_t *notmuch,
+		 const char *key,
+		 std::string &value)
+{
+    notmuch_status_t status = NOTMUCH_STATUS_SUCCESS;
+
+    try {
+	value = notmuch->xapian_db->get_metadata (CONFIG_PREFIX+key);
+    } catch (const Xapian::Error &error) {
+	status = NOTMUCH_STATUS_XAPIAN_EXCEPTION;
+	notmuch->exception_reported = TRUE;
+	if (! notmuch->exception_reported) {
+	    _notmuch_database_log (notmuch, "Error: A Xapian exception occurred getting metadata: %s\n",
+				   error.get_msg().c_str());
+	}
+    }
+    return status;
+}
+
+notmuch_status_t
+notmuch_database_get_config (notmuch_database_t *notmuch,
+			     const char *key,
+			     char **value) {
+    std::string strval;
+    notmuch_status_t status;
+
+    if (!value)
+	return NOTMUCH_STATUS_NULL_POINTER;
+
+    status = _metadata_value (notmuch, key, strval);
+    if (status)
+	return status;
+
+    *value = strdup (strval.c_str ());
+
+    return NOTMUCH_STATUS_SUCCESS;
+}
diff --git a/lib/notmuch.h b/lib/notmuch.h
index b29dd5f..e416fb4 100644
--- a/lib/notmuch.h
+++ b/lib/notmuch.h
@@ -1838,6 +1838,26 @@ notmuch_filenames_move_to_next (notmuch_filenames_t *filenames);
 void
 notmuch_filenames_destroy (notmuch_filenames_t *filenames);
 
+
+/**
+ * set config 'key' to 'value'
+ *
+ */
+notmuch_status_t
+notmuch_database_set_config (notmuch_database_t *db, const char *key, const char *value);
+
+/**
+ * retrieve config item 'key', assign to  'value'
+ *
+ * keys which have not been previously set with n_d_set_config will
+ * return an empty string.
+ *
+ * return value is allocated by malloc and should be freed by the
+ * caller.
+ */
+notmuch_status_t
+notmuch_database_get_config (notmuch_database_t *db, const char *key, char **value);
+
 typedef enum {
     NOTMUCH_OPTION_COMPACT = 1,
     NOTMUCH_OPTION_FIELD_PROCESSOR = 2
diff --git a/test/T590-libconfig.sh b/test/T590-libconfig.sh
new file mode 100755
index 0000000..85e4497
--- /dev/null
+++ b/test/T590-libconfig.sh
@@ -0,0 +1,58 @@
+#!/usr/bin/env bash
+test_description="library config API"
+
+. ./test-lib.sh || exit 1
+
+add_email_corpus
+
+cat <<EOF > c_head
+#include <stdio.h>
+#include <string.h>
+#include <stdlib.h>
+#include <notmuch.h>
+
+void run(int line, notmuch_status_t ret)
+{
+   if (ret) {
+	fprintf (stderr, "line %d: %s\n", line, ret);
+	exit (1);
+   }
+}
+
+#define RUN(v)  run(__LINE__, v);
+
+int main (int argc, char** argv)
+{
+   notmuch_database_t *db;
+   char *val;
+   notmuch_status_t stat;
+
+   RUN(notmuch_database_open (argv[1], NOTMUCH_DATABASE_MODE_READ_WRITE, &db));
+
+EOF
+
+cat <<EOF > c_tail
+   RUN(notmuch_database_destroy(db));
+}
+EOF
+
+test_begin_subtest "notmuch_database_{set,get}_config"
+cat c_head - c_tail <<'EOF' | test_C ${MAIL_DIR}
+{
+   RUN(notmuch_database_set_config (db, "testkey1", "testvalue1"));
+   RUN(notmuch_database_set_config (db, "testkey2", "testvalue2"));
+   RUN(notmuch_database_get_config (db, "testkey1", &val));
+   printf("testkey1 = %s\n", val);
+   RUN(notmuch_database_get_config (db, "testkey2", &val));
+   printf("testkey2 = %s\n", val);
+}
+EOF
+cat <<'EOF' >EXPECTED
+== stdout ==
+testkey1 = testvalue1
+testkey2 = testvalue2
+== stderr ==
+EOF
+test_expect_equal_file EXPECTED OUTPUT
+
+test_done
-- 
2.7.0

[PATCH 07/13] lib: config list iterators

[David Bremner]

Since xapian provides the ability to restrict the iterator to a given
prefix, we expose this ability to the user. Otherwise we mimic the other
iterator interfances in notmuch (e.g. tags.c).
---
 lib/config.cc          | 104 +++++++++++++++++++++++++++++++++++++++++++++++++
 lib/notmuch.h          |  44 +++++++++++++++++++++
 test/T590-libconfig.sh |  60 ++++++++++++++++++++++++++++
 3 files changed, 208 insertions(+)

diff --git a/lib/config.cc b/lib/config.cc
index af00d6f..e581f32 100644
--- a/lib/config.cc
+++ b/lib/config.cc
@@ -24,6 +24,19 @@
 
 static const std::string CONFIG_PREFIX="C";
 
+struct _notmuch_config_list {
+    notmuch_database_t *notmuch;
+    Xapian::TermIterator *iterator;
+    char *current_key;
+    char *current_val;
+};
+
+static int
+_notmuch_config_list_destroy (notmuch_config_list_t *list) {
+    delete list->iterator;
+    return 0;
+}
+
 notmuch_status_t
 notmuch_database_set_config (notmuch_database_t *notmuch,
 			     const char *key,
@@ -88,3 +101,94 @@ notmuch_database_get_config (notmuch_database_t *notmuch,
 
     return NOTMUCH_STATUS_SUCCESS;
 }
+
+notmuch_status_t
+notmuch_database_get_config_list (notmuch_database_t *notmuch,
+				 const char *prefix,
+				 notmuch_config_list_t **out)
+{
+    notmuch_config_list_t *list = NULL;
+    notmuch_status_t status = NOTMUCH_STATUS_SUCCESS;
+
+    list = talloc (notmuch, notmuch_config_list_t);
+    if (!list) {
+	status = NOTMUCH_STATUS_OUT_OF_MEMORY;
+	goto DONE;
+    }
+
+    talloc_set_destructor (list, _notmuch_config_list_destroy);
+    list->iterator = new Xapian::TermIterator;
+    list->notmuch = notmuch;
+    list->current_key = NULL;
+    list->current_val = NULL;
+
+    try {
+
+	*list->iterator = notmuch->xapian_db->metadata_keys_begin (CONFIG_PREFIX + (prefix ? prefix : ""));
+
+    } catch (const Xapian::Error &error) {
+	_notmuch_database_log (notmuch, "A Xapian exception occurred getting metadata iterator: %s.\n",
+			       error.get_msg().c_str());
+	notmuch->exception_reported = TRUE;
+	status = NOTMUCH_STATUS_XAPIAN_EXCEPTION;
+    }
+
+    *out = list;
+
+ DONE:
+    if (status && list)
+	talloc_free (list);
+
+    return status;
+}
+
+notmuch_bool_t
+notmuch_config_list_valid (notmuch_config_list_t *metadata)
+{
+    if (*(metadata->iterator) == metadata->notmuch->xapian_db->metadata_keys_end())
+	return FALSE;
+
+    return TRUE;
+}
+
+const char *
+notmuch_config_list_key (notmuch_config_list_t *list)
+{
+    if (list->current_key)
+	talloc_free (list->current_key);
+
+    list->current_key = talloc_strdup (list, (**(list->iterator)).c_str () + CONFIG_PREFIX.length ());
+
+    return  list->current_key;
+}
+
+const char *
+notmuch_config_list_value (notmuch_config_list_t *list)
+{
+    std::string strval;
+    notmuch_status_t status;
+    const char *key = notmuch_config_list_key (list);
+
+    /* TODO: better error reporting?? */
+    status = _metadata_value (list->notmuch, key, strval);
+    if (status)
+	return NULL;
+
+    if (list->current_val)
+	talloc_free(list->current_val);
+
+    list->current_val = talloc_strdup(list, strval.c_str ());
+    return list->current_val;
+}
+
+void
+notmuch_config_list_move_to_next (notmuch_config_list_t *list)
+{
+    (*(list->iterator))++;
+}
+
+void
+notmuch_config_list_destroy (notmuch_config_list_t *list)
+{
+    talloc_free (list);
+}
diff --git a/lib/notmuch.h b/lib/notmuch.h
index e416fb4..2278822 100644
--- a/lib/notmuch.h
+++ b/lib/notmuch.h
@@ -206,6 +206,7 @@ typedef struct _notmuch_message notmuch_message_t;
 typedef struct _notmuch_tags notmuch_tags_t;
 typedef struct _notmuch_directory notmuch_directory_t;
 typedef struct _notmuch_filenames notmuch_filenames_t;
+typedef struct _notmuch_config_list notmuch_config_list_t;
 #endif /* __DOXYGEN__ */
 
 /**
@@ -1858,6 +1859,49 @@ notmuch_database_set_config (notmuch_database_t *db, const char *key, const char
 notmuch_status_t
 notmuch_database_get_config (notmuch_database_t *db, const char *key, char **value);
 
+/**
+ * Create an iterator for all config items with keys matching a given prefix
+ */
+notmuch_status_t
+notmuch_database_get_config_list (notmuch_database_t *db, const char *prefix, notmuch_config_list_t **out);
+
+/**
+ * Is 'config_list' iterator valid (i.e. _key, _value, _move_to_next can be called).
+ */
+notmuch_bool_t
+notmuch_config_list_valid (notmuch_config_list_t *config_list);
+
+/**
+ * return key for current config pair
+ *
+ * return value is owned by the iterator, and will be destroyed by the
+ * next call to notmuch_config_list_key or notmuch_config_list_destroy.
+ */
+const char *
+notmuch_config_list_key (notmuch_config_list_t *config_list);
+
+/**
+ * return 'value' for current config pair
+ *
+ * return value is owned by the iterator, and will be destroyed by the
+ * next call to notmuch_config_list_value or notmuch config_list_destroy
+ */
+const char *
+notmuch_config_list_value (notmuch_config_list_t *config_list);
+
+
+/**
+ * move 'config_list' iterator to the next pair
+ */
+void
+notmuch_config_list_move_to_next (notmuch_config_list_t *config_list);
+
+/**
+ * free any resources held by 'config_list'
+ */
+void
+notmuch_config_list_destroy (notmuch_config_list_t *config_list);
+
 typedef enum {
     NOTMUCH_OPTION_COMPACT = 1,
     NOTMUCH_OPTION_FIELD_PROCESSOR = 2
diff --git a/test/T590-libconfig.sh b/test/T590-libconfig.sh
index 85e4497..8ca6883 100755
--- a/test/T590-libconfig.sh
+++ b/test/T590-libconfig.sh
@@ -55,4 +55,64 @@ testkey2 = testvalue2
 EOF
 test_expect_equal_file EXPECTED OUTPUT
 
+
+test_begin_subtest "notmuch_database_get_config_list: empty list"
+cat c_head - c_tail <<'EOF' | test_C ${MAIL_DIR}
+{
+   notmuch_config_list_t *list;
+   RUN(notmuch_database_get_config_list (db, "nonexistent", &list));
+   printf("valid = %d\n", notmuch_config_list_valid (list));
+   notmuch_config_list_destroy (list);
+}
+EOF
+cat <<'EOF' >EXPECTED
+== stdout ==
+valid = 0
+== stderr ==
+EOF
+test_expect_equal_file EXPECTED OUTPUT
+
+
+test_begin_subtest "notmuch_database_get_config_list: all pairs"
+cat c_head - c_tail <<'EOF' | test_C ${MAIL_DIR}
+{
+   notmuch_config_list_t *list;
+   RUN(notmuch_database_set_config (db, "zzzafter", "afterval"));
+   RUN(notmuch_database_set_config (db, "aaabefore", "beforeval"));
+   RUN(notmuch_database_get_config_list (db, "", &list));
+   for (; notmuch_config_list_valid (list); notmuch_config_list_move_to_next (list)) {
+      printf("%s %s\n", notmuch_config_list_key (list), notmuch_config_list_value(list));
+   }
+   notmuch_config_list_destroy (list);
+}
+EOF
+cat <<'EOF' >EXPECTED
+== stdout ==
+aaabefore beforeval
+testkey1 testvalue1
+testkey2 testvalue2
+zzzafter afterval
+== stderr ==
+EOF
+test_expect_equal_file EXPECTED OUTPUT
+
+test_begin_subtest "notmuch_database_get_config_list: one prefix"
+cat c_head - c_tail <<'EOF' | test_C ${MAIL_DIR}
+{
+   notmuch_config_list_t *list;
+   RUN(notmuch_database_get_config_list (db, "testkey", &list));
+   for (; notmuch_config_list_valid (list); notmuch_config_list_move_to_next (list)) {
+      printf("%s %s\n", notmuch_config_list_key (list), notmuch_config_list_value(list));
+   }
+   notmuch_config_list_destroy (list);
+}
+EOF
+cat <<'EOF' >EXPECTED
+== stdout ==
+testkey1 testvalue1
+testkey2 testvalue2
+== stderr ==
+EOF
+test_expect_equal_file EXPECTED OUTPUT
+
 test_done
-- 
2.7.0

[PATCH 08/13] CLI: add print_status_database

[David Bremner]

This could probably be used at quite a few places in the existing code,
but in the immediate future I plan to use in some new code in
notmuch-dump
---
 notmuch-client.h |  5 +++++
 status.c         | 17 +++++++++++++++++
 2 files changed, 22 insertions(+)

diff --git a/notmuch-client.h b/notmuch-client.h
index 18e6c60..b3d0b66 100644
--- a/notmuch-client.h
+++ b/notmuch-client.h
@@ -462,6 +462,11 @@ print_status_query (const char *loc,
 		    const notmuch_query_t *query,
 		    notmuch_status_t status);
 
+notmuch_status_t
+print_status_database (const char *loc,
+		       const notmuch_database_t *database,
+		       notmuch_status_t status);
+
 #include "command-line-arguments.h"
 
 extern char *notmuch_requested_db_uuid;
diff --git a/status.c b/status.c
index 8fa81cb..45d3fb4 100644
--- a/status.c
+++ b/status.c
@@ -19,3 +19,20 @@ print_status_query (const char *loc,
     }
     return status;
 }
+
+notmuch_status_t
+print_status_database (const char *loc,
+		    const notmuch_database_t *notmuch,
+		    notmuch_status_t status)
+{
+    if (status) {
+	const char *msg;
+
+	fprintf (stderr, "%s: %s\n", loc,
+		 notmuch_status_to_string (status));
+	msg = notmuch_database_status_string (notmuch);
+	if (msg)
+	    fputs (msg, stderr);
+    }
+    return status;
+}
-- 
2.7.0

[PATCH 09/13] CLI: add optional config data to dump output.

[David Bremner]

This lacks at least documentation. Note that it changes the default dump
output format, but doesn't break existing notmuch-restore. It might
break user scripts though.
---
 notmuch-client.h          |  8 +++++
 notmuch-dump.c            | 80 +++++++++++++++++++++++++++++++++++++++++++++--
 notmuch-new.c             |  2 +-
 test/T150-tagging.sh      |  8 ++---
 test/T240-dump-restore.sh | 14 ++++-----
 test/T590-libconfig.sh    | 17 ++++++++++
 test/test-lib.sh          |  6 ++++
 7 files changed, 120 insertions(+), 15 deletions(-)

diff --git a/notmuch-client.h b/notmuch-client.h
index b3d0b66..ae6f124 100644
--- a/notmuch-client.h
+++ b/notmuch-client.h
@@ -446,11 +446,19 @@ typedef enum dump_formats {
     DUMP_FORMAT_SUP
 } dump_format_t;
 
+typedef enum dump_includes {
+    DUMP_INCLUDE_TAGS=1,
+    DUMP_INCLUDE_CONFIG=2,
+} dump_include_t;
+
+#define NOTMUCH_DUMP_VERSION 2
+
 int
 notmuch_database_dump (notmuch_database_t *notmuch,
 		       const char *output_file_name,
 		       const char *query_str,
 		       dump_format_t output_format,
+		       dump_include_t include,
 		       notmuch_bool_t gzip_output);
 
 /* If status is non-zero (i.e. error) print appropriate
diff --git a/notmuch-dump.c b/notmuch-dump.c
index 829781f..a6cf810 100644
--- a/notmuch-dump.c
+++ b/notmuch-dump.c
@@ -23,16 +23,82 @@
 #include "string-util.h"
 #include <zlib.h>
 
+static int
+database_dump_config (notmuch_database_t *notmuch, gzFile output)
+{
+    notmuch_config_list_t *list;
+    int ret = EXIT_FAILURE;
+    char *buffer = NULL;
+    size_t buffer_size = 0;
+
+    if (print_status_database ("notmuch dump", notmuch,
+			       notmuch_database_get_config_list (notmuch, NULL, &list)))
+	goto DONE;
+
+    for (; notmuch_config_list_valid (list); notmuch_config_list_move_to_next (list)) {
+	if (hex_encode (notmuch, notmuch_config_list_key (list),
+			&buffer, &buffer_size) != HEX_SUCCESS) {
+	    fprintf (stderr, "Error: failed to hex-encode config key %s\n",
+		     notmuch_config_list_key (list));
+	    goto DONE;
+	}
+	gzprintf (output, "#@ %s", buffer);
+
+	if (hex_encode (notmuch, notmuch_config_list_value (list),
+			&buffer, &buffer_size) != HEX_SUCCESS) {
+	    fprintf (stderr, "Error: failed to hex-encode config value %s\n",
+		     notmuch_config_list_value (list) );
+	    goto DONE;
+	}
+
+	gzprintf (output, " %s\n", buffer);
+    }
+
+    ret = EXIT_SUCCESS;
+
+ DONE:
+    if (list)
+	notmuch_config_list_destroy (list);
+
+    if (buffer)
+	talloc_free (buffer);
+
+    return ret;
+}
+
+static void
+print_dump_header (gzFile output, int output_format, int include)
+{
+    gzprintf (output, "#notmuch-dump %s:%d %s%s%s\n",
+	      (output_format == DUMP_FORMAT_SUP) ? "sup" : "batch-tag",
+	      NOTMUCH_DUMP_VERSION,
+	      (include & DUMP_INCLUDE_CONFIG) ? "config" : "",
+	      (include & DUMP_INCLUDE_TAGS) && (include & DUMP_INCLUDE_CONFIG) ? "," : "",
+	      (include & DUMP_INCLUDE_TAGS) ? "tags" : "");
+
+
+}
 
 static int
 database_dump_file (notmuch_database_t *notmuch, gzFile output,
-		    const char *query_str, int output_format)
+		    const char *query_str, int output_format, int include)
 {
     notmuch_query_t *query;
     notmuch_messages_t *messages;
     notmuch_message_t *message;
     notmuch_tags_t *tags;
 
+    print_dump_header (output, output_format, include);
+
+    if (include & DUMP_INCLUDE_CONFIG) {
+	if (print_status_database ("notmuch dump", notmuch,
+				   database_dump_config(notmuch,output)))
+	    return EXIT_FAILURE;
+    }
+
+    if (! (include & DUMP_INCLUDE_TAGS))
+	return EXIT_SUCCESS;
+
     if (! query_str)
 	query_str = "";
 
@@ -130,6 +196,7 @@ notmuch_database_dump (notmuch_database_t *notmuch,
 		       const char *output_file_name,
 		       const char *query_str,
 		       dump_format_t output_format,
+		       dump_include_t include,
 		       notmuch_bool_t gzip_output)
 {
     gzFile output = NULL;
@@ -164,7 +231,7 @@ notmuch_database_dump (notmuch_database_t *notmuch,
 	goto DONE;
     }
 
-    ret = database_dump_file (notmuch, output, query_str, output_format);
+    ret = database_dump_file (notmuch, output, query_str, output_format, include);
     if (ret) goto DONE;
 
     ret = gzflush (output, Z_FINISH);
@@ -226,6 +293,7 @@ notmuch_dump_command (notmuch_config_t *config, int argc, char *argv[])
     int opt_index;
 
     int output_format = DUMP_FORMAT_BATCH_TAG;
+    int include = 0;
     notmuch_bool_t gzip_output = 0;
 
     notmuch_opt_desc_t options[] = {
@@ -233,6 +301,9 @@ notmuch_dump_command (notmuch_config_t *config, int argc, char *argv[])
 	  (notmuch_keyword_t []){ { "sup", DUMP_FORMAT_SUP },
 				  { "batch-tag", DUMP_FORMAT_BATCH_TAG },
 				  { 0, 0 } } },
+	{ NOTMUCH_OPT_KEYWORD_FLAGS, &include, "include", 'I',
+	  (notmuch_keyword_t []){ { "config", DUMP_INCLUDE_CONFIG },
+				  { "tags", DUMP_INCLUDE_TAGS} } },
 	{ NOTMUCH_OPT_STRING, &output_file_name, "output", 'o', 0  },
 	{ NOTMUCH_OPT_BOOLEAN, &gzip_output, "gzip", 'z', 0 },
 	{ NOTMUCH_OPT_INHERIT, (void *) &notmuch_shared_options, NULL, 0, 0 },
@@ -245,6 +316,9 @@ notmuch_dump_command (notmuch_config_t *config, int argc, char *argv[])
 
     notmuch_process_shared_options (argv[0]);
 
+    if (include == 0)
+	include = DUMP_INCLUDE_CONFIG | DUMP_INCLUDE_TAGS;
+
     if (opt_index < argc) {
 	query_str = query_string_from_args (notmuch, argc - opt_index, argv + opt_index);
 	if (query_str == NULL) {
@@ -254,7 +328,7 @@ notmuch_dump_command (notmuch_config_t *config, int argc, char *argv[])
     }
 
     ret = notmuch_database_dump (notmuch, output_file_name, query_str,
-				 output_format, gzip_output);
+				 output_format, include, gzip_output);
 
     notmuch_database_destroy (notmuch);
 
diff --git a/notmuch-new.c b/notmuch-new.c
index e503776..fd2ff82 100644
--- a/notmuch-new.c
+++ b/notmuch-new.c
@@ -1041,7 +1041,7 @@ notmuch_new_command (notmuch_config_t *config, int argc, char *argv[])
 	    }
 
 	    if (notmuch_database_dump (notmuch, backup_name, "",
-				       DUMP_FORMAT_BATCH_TAG, TRUE)) {
+				       DUMP_FORMAT_BATCH_TAG, DUMP_INCLUDE_CONFIG | DUMP_INCLUDE_TAGS, TRUE)) {
 		fprintf (stderr, "Backup failed. Aborting upgrade.");
 		return EXIT_FAILURE;
 	    }
diff --git a/test/T150-tagging.sh b/test/T150-tagging.sh
index 8adcabc..f1a6df0 100755
--- a/test/T150-tagging.sh
+++ b/test/T150-tagging.sh
@@ -188,7 +188,7 @@ cat <<EOF > EXPECTED
 +%22%27%22%27%22%22%27%27 +inbox +tag4 +tag5 +unread -- id:msg-002@notmuch-test-suite
 EOF
 
-notmuch dump --format=batch-tag | sort > OUTPUT
+NOTMUCH_DUMP_TAGS > OUTPUT
 notmuch restore --format=batch-tag < BACKUP
 test_expect_equal_file EXPECTED OUTPUT
 
@@ -209,7 +209,7 @@ cat <<EOF > EXPECTED
 +%21@%23%20%24%25%5e%26%2a%29-_=+%5b%7b%5c%20%7c%3b%3a%27%20%22,.%3c%60%7e +inbox +tag5 +unread -- id:msg-001@notmuch-test-suite
 EOF
 
-notmuch dump --format=batch-tag | sort > OUTPUT
+NOTMUCH_DUMP_TAGS > OUTPUT
 notmuch restore --format=batch-tag < BACKUP
 test_expect_equal_file EXPECTED OUTPUT
 
@@ -235,7 +235,7 @@ cat <<EOF > EXPECTED
 +%2a@%7d%cf%b5%f4%85%80%adO3%da%a7 +=%e0%ac%95%c8%b3+%ef%aa%95%c8%a64w%c7%9d%c9%a2%cf%b3%d6%82%24B%c4%a9%c5%a1UX%ee%99%b0%27E7%ca%a4%d0%8b%5d +A%e1%a0%bc%de%8b%d5%b2V%d9%9b%f3%b5%a2%a3M%d8%a1u@%f0%a0%ac%948%7e%f0%ab%86%af%27 +L%df%85%ef%a1%a5m@%d3%96%c2%ab%d4%9f%ca%b8%f3%b3%a2%bf%c7%b1_u%d7%b4%c7%b1 +P%c4%98%2f +R +inbox +tag5 +unread +%7e%d1%8b%25%ec%a0%ae%d1%a0M%3b%e3%b6%b7%e9%a4%87%3c%db%9a%cc%a8%e1%96%9d +%c4%bf7%c7%ab9H%c4%99k%ea%91%bd%c3%8ck%e2%b3%8dk%c5%952V%e4%99%b2%d9%b3%e4%8b%bda%5b%24%c7%9b +%da%88=f%cc%b9I%ce%af%7b%c9%97%e3%b9%8bH%cb%92X%d2%8c6 +%dc%9crh%d2%86B%e5%97%a2%22t%ed%99%82d -- id:msg-001@notmuch-test-suite
 EOF
 
-notmuch dump --format=batch-tag | sort > OUTPUT
+NOTMUCH_DUMP_TAGS > OUTPUT
 notmuch restore --format=batch-tag < BACKUP
 test_expect_equal_file EXPECTED OUTPUT
 
@@ -260,7 +260,7 @@ cat <<EOF > EXPECTED
 +foo%3a%3abar%25 +found%3a%3ait +inbox +tag5 +unread +winner -- id:msg-001@notmuch-test-suite
 EOF
 
-notmuch dump --format=batch-tag | sort > OUTPUT
+NOTMUCH_DUMP_TAGS > OUTPUT
 notmuch restore --format=batch-tag < BACKUP
 test_expect_equal_file EXPECTED OUTPUT
 
diff --git a/test/T240-dump-restore.sh b/test/T240-dump-restore.sh
index e6976ff..758d355 100755
--- a/test/T240-dump-restore.sh
+++ b/test/T240-dump-restore.sh
@@ -97,7 +97,7 @@ test_expect_equal_file dump.expected dump.actual
 # Note, we assume all messages from cworth have a message-id
 # containing cworth.org
 
-grep 'cworth[.]org' dump.expected > dump-cworth.expected
+(head -1 dump.expected ; grep 'cworth[.]org' dump.expected) > dump-cworth.expected
 
 test_begin_subtest "dump -- from:cworth"
 notmuch dump -- from:cworth > dump-dash-cworth.actual
@@ -118,16 +118,16 @@ notmuch search --output=messages from:cworth | sed s/^id:// |\
 test_expect_equal_file OUTPUT EXPECTED
 
 test_begin_subtest "format=batch-tag, dump sanity check."
-notmuch dump --format=sup from:cworth | cut -f1 -d' ' | \
+NOTMUCH_DUMP_TAGS --format=sup from:cworth | cut -f1 -d' ' | \
     sort > EXPECTED.$test_count
-notmuch dump --format=batch-tag from:cworth | sed 's/^.*-- id://' | \
+NOTMUCH_DUMP_TAGS --format=batch-tag from:cworth | sed 's/^.*-- id://' | \
     sort > OUTPUT.$test_count
 test_expect_equal_file EXPECTED.$test_count OUTPUT.$test_count
 
 test_begin_subtest "format=batch-tag, missing newline"
 printf "+a_tag_without_newline -- id:20091117232137.GA7669@griffis1.net" > IN
 notmuch restore --accumulate < IN
-notmuch dump id:20091117232137.GA7669@griffis1.net > OUT
+NOTMUCH_DUMP_TAGS id:20091117232137.GA7669@griffis1.net > OUT
 cat <<EOF > EXPECTED
 +a_tag_without_newline +inbox +unread -- id:20091117232137.GA7669@griffis1.net
 EOF
@@ -155,7 +155,7 @@ cat <<EOF >EXPECTED.$test_count
 + -- id:20091117232137.GA7669@griffis1.net
 EOF
 notmuch restore --format=batch-tag < EXPECTED.$test_count
-notmuch dump --format=batch-tag id:20091117232137.GA7669@griffis1.net > OUTPUT.$test_count
+NOTMUCH_DUMP_TAGS --format=batch-tag id:20091117232137.GA7669@griffis1.net > OUTPUT.$test_count
 test_expect_equal_file EXPECTED.$test_count OUTPUT.$test_count
 
 tag1='comic_swear=$&^%$^%\\//-+$^%$'
@@ -217,9 +217,9 @@ notmuch dump --format=batch-tag > OUTPUT.$test_count
 test_expect_equal_file EXPECTED.$test_count OUTPUT.$test_count
 
 test_begin_subtest 'format=batch-tag, checking encoded output'
-notmuch dump --format=batch-tag -- from:cworth |\
+NOTMUCH_DUMP_TAGS --format=batch-tag -- from:cworth |\
 	 awk "{ print \"+$enc1 +$enc2 +$enc3 -- \" \$5 }" > EXPECTED.$test_count
-notmuch dump --format=batch-tag -- from:cworth  > OUTPUT.$test_count
+NOTMUCH_DUMP_TAGS --format=batch-tag -- from:cworth  > OUTPUT.$test_count
 test_expect_equal_file EXPECTED.$test_count OUTPUT.$test_count
 
 test_begin_subtest 'restoring sane tags'
diff --git a/test/T590-libconfig.sh b/test/T590-libconfig.sh
index 8ca6883..5ea5300 100755
--- a/test/T590-libconfig.sh
+++ b/test/T590-libconfig.sh
@@ -115,4 +115,21 @@ testkey2 testvalue2
 EOF
 test_expect_equal_file EXPECTED OUTPUT
 
+test_begin_subtest "dump config"
+cat c_head - c_tail <<'EOF' | test_C ${MAIL_DIR}
+{
+    RUN(notmuch_database_set_config (db, "key with spaces", "value, with, spaces!"));
+}
+EOF
+notmuch dump --include=config >OUTPUT
+cat <<'EOF' >EXPECTED
+#notmuch-dump batch-tag:2 config
+#@ aaabefore beforeval
+#@ key%20with%20spaces value,%20with,%20spaces%21
+#@ testkey1 testvalue1
+#@ testkey2 testvalue2
+#@ zzzafter afterval
+EOF
+test_expect_equal_file EXPECTED OUTPUT
+
 test_done
diff --git a/test/test-lib.sh b/test/test-lib.sh
index 49e6f40..98abbe2 100644
--- a/test/test-lib.sh
+++ b/test/test-lib.sh
@@ -673,6 +673,12 @@ NOTMUCH_NEW ()
     notmuch new "${@}" | grep -v -E -e '^Processed [0-9]*( total)? file|Found [0-9]* total file'
 }
 
+NOTMUCH_DUMP_TAGS ()
+{
+    # this relies on the default format being batch-tag, otherwise some tests will break
+    notmuch dump --include=tags "${@}" | sed '/^#/d' | sort
+}
+
 notmuch_search_sanitize ()
 {
     perl -pe 's/("?thread"?: ?)("?)................("?)/\1\2XXX\3/'
-- 
2.7.0

[PATCH 10/13] CLI: optionally restore config data.

[David Bremner]

Like the corresponding change to dump, this needs documentation
---
 notmuch-restore.c      | 53 ++++++++++++++++++++++++++++++++++++++++++++++++++
 test/T590-libconfig.sh | 11 +++++++++++
 2 files changed, 64 insertions(+)

diff --git a/notmuch-restore.c b/notmuch-restore.c
index 9abc64f..e06fbde 100644
--- a/notmuch-restore.c
+++ b/notmuch-restore.c
@@ -24,6 +24,38 @@
 #include "string-util.h"
 #include "zlib-extra.h"
 
+static int
+process_config_line(notmuch_database_t *notmuch, const char* line){
+    const char *key_p, *val_p;
+    char *key, *val;
+    size_t key_len,val_len;
+    const char *delim=" \t\n";
+    int ret = EXIT_FAILURE;
+
+    void *local = talloc_new(NULL);
+
+    key_p = strtok_len_c(line, delim, &key_len);
+    val_p = strtok_len_c(key_p+key_len, delim, &val_len);
+
+    key = talloc_strndup(local, key_p, key_len);
+    val = talloc_strndup(local, val_p, val_len);
+    if (hex_decode_inplace (key) != HEX_SUCCESS ||
+	hex_decode_inplace (val) != HEX_SUCCESS ) {
+	fprintf (stderr, "hex decoding failure on line %s\n", line);
+	goto DONE;
+    }
+
+    if (print_status_database ("notmuch restore", notmuch,
+			       notmuch_database_set_config (notmuch, key, val)))
+	goto DONE;
+
+    ret = EXIT_SUCCESS;
+
+ DONE:
+    talloc_free (local);
+    return ret;
+}
+
 static regex_t regex;
 
 /* Non-zero return indicates an error in retrieving the message,
@@ -137,6 +169,7 @@ notmuch_restore_command (notmuch_config_t *config, int argc, char *argv[])
 
     int ret = 0;
     int opt_index;
+    int include=0;
     int input_format = DUMP_FORMAT_AUTO;
 
     if (notmuch_database_open (notmuch_config_get_database_path (config),
@@ -152,6 +185,10 @@ notmuch_restore_command (notmuch_config_t *config, int argc, char *argv[])
 				  { "batch-tag", DUMP_FORMAT_BATCH_TAG },
 				  { "sup", DUMP_FORMAT_SUP },
 				  { 0, 0 } } },
+	{ NOTMUCH_OPT_KEYWORD_FLAGS, &include, "include", 'I',
+	  (notmuch_keyword_t []){ { "config", DUMP_INCLUDE_CONFIG },
+				  { "tags", DUMP_INCLUDE_TAGS} } },
+
 	{ NOTMUCH_OPT_STRING, &input_file_name, "input", 'i', 0 },
 	{ NOTMUCH_OPT_BOOLEAN,  &accumulate, "accumulate", 'a', 0 },
 	{ NOTMUCH_OPT_INHERIT, (void *) &notmuch_shared_options, NULL, 0, 0 },
@@ -167,6 +204,10 @@ notmuch_restore_command (notmuch_config_t *config, int argc, char *argv[])
     notmuch_process_shared_options (argv[0]);
     notmuch_exit_if_unmatched_db_uuid (notmuch);
 
+    if (include == 0) {
+	include = DUMP_INCLUDE_CONFIG | DUMP_INCLUDE_TAGS;
+    }
+
     name_for_error = input_file_name ? input_file_name : "stdin";
 
     if (! accumulate)
@@ -225,11 +266,23 @@ notmuch_restore_command (notmuch_config_t *config, int argc, char *argv[])
 	    ret = EXIT_FAILURE;
 	    goto DONE;
 	}
+
+	if ((include & DUMP_INCLUDE_CONFIG) && line_len >= 2 && line[0] == '#' && line[1] == '@') {
+	    ret = process_config_line(notmuch, line+2);
+	    if (ret)
+		goto DONE;
+	}
+
     } while ((line_len == 0) ||
 	     (line[0] == '#') ||
 	     /* the cast is safe because we checked about for line_len < 0 */
 	     (strspn (line, " \t\n") == (unsigned)line_len));
 
+    if (! (include & DUMP_INCLUDE_TAGS)) {
+	ret = EXIT_SUCCESS;
+	goto DONE;
+    }
+
     char *p;
     for (p = line; (input_format == DUMP_FORMAT_AUTO) && *p; p++) {
 	if (*p == '(')
diff --git a/test/T590-libconfig.sh b/test/T590-libconfig.sh
index 5ea5300..9c1e566 100755
--- a/test/T590-libconfig.sh
+++ b/test/T590-libconfig.sh
@@ -132,4 +132,15 @@ cat <<'EOF' >EXPECTED
 EOF
 test_expect_equal_file EXPECTED OUTPUT
 
+test_begin_subtest "restore config"
+notmuch dump --include=config >EXPECTED
+cat c_head - c_tail <<'EOF' | test_C ${MAIL_DIR}
+{
+    RUN(notmuch_database_set_config (db, "testkey1", "mutatedvalue"));
+}
+EOF
+notmuch restore --include=config <EXPECTED
+notmuch dump --include=config >OUTPUT
+test_expect_equal_file EXPECTED OUTPUT
+
 test_done
-- 
2.7.0

[PATCH 11/13] CLI: add notmuch-config support for named queries

[David Bremner]

Most of the infrastructure here is general, only the validation/dispatch
is hardcoded to a particular prefix.

A notable change in behaviour is that notmuch-config now opens the
database e.g. on every call to list, which fails with an error message
if the database doesn't exit yet.
---
 doc/man1/notmuch-config.rst |  5 +++
 notmuch-config.c            | 88 ++++++++++++++++++++++++++++++++++++++++++++-
 test/Makefile.local         |  2 +-
 test/T030-config.sh         | 12 ++++---
 test/T600-named-queries.sh  | 53 +++++++++++++++++++++++++++
 test/test-lib.sh            |  5 +++
 6 files changed, 158 insertions(+), 7 deletions(-)
 create mode 100755 test/T600-named-queries.sh

diff --git a/doc/man1/notmuch-config.rst b/doc/man1/notmuch-config.rst
index 40c1272..98b3c1a 100644
--- a/doc/man1/notmuch-config.rst
+++ b/doc/man1/notmuch-config.rst
@@ -132,6 +132,11 @@ The available configuration items are described below.
     
         Default: ``gpg``.
 
+    **query.<name>**
+
+        Expansion for named query called <name>. See
+        **notmuch-search-terms(7)** for more information about named
+        queries.
 
 ENVIRONMENT
 ===========
diff --git a/notmuch-config.c b/notmuch-config.c
index cfc549d..121fec6 100644
--- a/notmuch-config.c
+++ b/notmuch-config.c
@@ -751,6 +751,28 @@ _item_split (char *item, char **group, char **key)
 }
 
 #define OPTION_PREFIX "options."
+#define QUERY_PREFIX "query."
+
+static int
+_print_db_config(notmuch_config_t *config, const char *name)
+{
+    notmuch_database_t *notmuch;
+    char *val;
+
+    if (notmuch_database_open (notmuch_config_get_database_path (config),
+			       NOTMUCH_DATABASE_MODE_READ_ONLY, &notmuch))
+	return EXIT_FAILURE;
+
+    /* XXX Handle UUID mismatch? */
+
+    if (print_status_database ("notmuch config", notmuch,
+			       notmuch_database_get_config (notmuch, name, &val)))
+	return EXIT_FAILURE;
+
+     puts (val);
+
+    return EXIT_SUCCESS;
+}
 
 static int
 notmuch_config_command_get (notmuch_config_t *config, char *item)
@@ -778,6 +800,8 @@ notmuch_config_command_get (notmuch_config_t *config, char *item)
     } else if (STRNCMP_LITERAL (item, OPTION_PREFIX) == 0) {
 	printf ("%s\n",
 	       notmuch_options_get (item + strlen (OPTION_PREFIX)) ? "true" : "false");
+    } else if (STRNCMP_LITERAL (item, QUERY_PREFIX) == 0) {
+	return _print_db_config (config, item);
     } else {
 	char **value;
 	size_t i, length;
@@ -805,6 +829,39 @@ notmuch_config_command_get (notmuch_config_t *config, char *item)
 }
 
 static int
+_set_db_config(notmuch_config_t *config, const char *key, int argc, char **argv)
+{
+    notmuch_database_t *notmuch;
+    const char *val = "";
+
+    if (argc > 1) {
+	/* XXX handle lists? */
+	fprintf (stderr, "notmuch config set: at most one value expected for %s\n", key);
+	return EXIT_FAILURE;
+    }
+
+    if (argc > 0) {
+	val = argv[0];
+    }
+
+    if (notmuch_database_open (notmuch_config_get_database_path (config),
+			       NOTMUCH_DATABASE_MODE_READ_WRITE, &notmuch))
+	return EXIT_FAILURE;
+
+    /* XXX Handle UUID mismatch? */
+
+    if (print_status_database ("notmuch config", notmuch,
+			       notmuch_database_set_config (notmuch, key, val)))
+	return EXIT_FAILURE;
+
+    if (print_status_database ("notmuch config", notmuch,
+			       notmuch_database_close (notmuch)))
+	return EXIT_FAILURE;
+
+    return EXIT_SUCCESS;
+}
+
+static int
 notmuch_config_command_set (notmuch_config_t *config, char *item, int argc, char *argv[])
 {
     char *group, *key;
@@ -814,6 +871,10 @@ notmuch_config_command_set (notmuch_config_t *config, char *item, int argc, char
 	return 1;
     }
 
+    if (STRNCMP_LITERAL (item, QUERY_PREFIX) == 0) {
+	return _set_db_config (config, item, argc, argv);
+    }
+
     if (_item_split (item, &group, &key))
 	return 1;
 
@@ -850,6 +911,31 @@ _notmuch_config_list_options () {
 }
 
 static int
+_list_db_config (notmuch_config_t *config)
+{
+    notmuch_database_t *notmuch;
+    notmuch_config_list_t *list;
+
+    if (notmuch_database_open (notmuch_config_get_database_path (config),
+			       NOTMUCH_DATABASE_MODE_READ_ONLY, &notmuch))
+	return EXIT_FAILURE;
+
+    /* XXX Handle UUID mismatch? */
+
+
+    if (print_status_database ("notmuch config", notmuch,
+			       notmuch_database_get_config_list (notmuch, "", &list)))
+	return EXIT_FAILURE;
+
+    for (; notmuch_config_list_valid (list); notmuch_config_list_move_to_next (list)) {
+	printf("%s=%s\n", notmuch_config_list_key (list), notmuch_config_list_value(list));
+    }
+    notmuch_config_list_destroy (list);
+
+   return EXIT_SUCCESS;
+}
+
+static int
 notmuch_config_command_list (notmuch_config_t *config)
 {
     char **groups;
@@ -885,7 +971,7 @@ notmuch_config_command_list (notmuch_config_t *config)
     g_strfreev (groups);
 
     _notmuch_config_list_options ();
-    return 0;
+    return _list_db_config (config);
 }
 
 int
diff --git a/test/Makefile.local b/test/Makefile.local
index 30d420e..8c55441 100644
--- a/test/Makefile.local
+++ b/test/Makefile.local
@@ -19,7 +19,7 @@ $(dir)/hex-xcode: $(dir)/hex-xcode.o command-line-arguments.o util/libutil.a
 	$(call quiet,CC) $^ -o $@ $(LDFLAGS) $(TALLOC_LDFLAGS)
 
 random_corpus_deps =  $(dir)/random-corpus.o  $(dir)/database-test.o \
-			notmuch-config.o command-line-arguments.o \
+			notmuch-config.o status.o command-line-arguments.o \
 			lib/libnotmuch.a util/libutil.a \
 			parse-time-string/libparse-time-string.a
 
diff --git a/test/T030-config.sh b/test/T030-config.sh
index c37ba21..39ee885 100755
--- a/test/T030-config.sh
+++ b/test/T030-config.sh
@@ -43,10 +43,10 @@ notmuch config set foo.nonexistent
 test_expect_equal "$(notmuch config get foo.nonexistent)" ""
 
 test_begin_subtest "List all items"
-notmuch config set database.path "/canonical/path"
-output=$(notmuch config list | notmuch_options_sanitize)
-test_expect_equal "$output" "\
-database.path=/canonical/path
+notmuch config list 2>&1 | notmuch_config_sanitize > OUTPUT
+cat <<EOF > EXPECTED
+Error opening database at MAIL_DIR/.notmuch: No such file or directory
+database.path=MAIL_DIR
 user.name=Notmuch Test Suite
 user.primary_email=test_suite@notmuchmail.org
 user.other_email=test_suite_other@notmuchmail.org;test_suite@otherdomain.org
@@ -58,7 +58,9 @@ crypto.gpg_path=gpg
 foo.string=this is another string value
 foo.list=this;is another;list value;
 options.compact=something
-options.field_processor=something"
+options.field_processor=something
+EOF
+test_expect_equal_file EXPECTED OUTPUT
 
 test_begin_subtest "Top level --config=FILE option"
 cp "${NOTMUCH_CONFIG}" alt-config
diff --git a/test/T600-named-queries.sh b/test/T600-named-queries.sh
new file mode 100755
index 0000000..0922620
--- /dev/null
+++ b/test/T600-named-queries.sh
@@ -0,0 +1,53 @@
+#!/usr/bin/env bash
+test_description='named queries'
+. ./test-lib.sh || exit 1
+
+QUERYSTR="date:2009-11-18..2009-11-18 and tag:unread"
+
+test_expect_code 1 "error adding named query before initializing DB" \
+		 "notmuch config set query.test \"$QUERYSTR\""
+
+add_email_corpus
+
+test_expect_success "adding named query" \
+		    "notmuch config set query.test \"$QUERYSTR\""
+
+QUERYSTR2="query:test and subject:Maildir"
+test_expect_success "adding nested named query" \
+		    "notmuch config set query.test2 \"$QUERYSTR2\""
+
+test_begin_subtest "retrieve named query"
+output=$(notmuch config get query.test)
+test_expect_equal "$QUERYSTR" "$output"
+
+test_begin_subtest "List all queries"
+notmuch config list | grep ^query | notmuch_config_sanitize > OUTPUT
+cat <<EOF > EXPECTED
+query.test=date:2009-11-18..2009-11-18 and tag:unread
+query.test2=query:test and subject:Maildir
+EOF
+test_expect_equal_file EXPECTED OUTPUT
+
+test_begin_subtest "dump named queries"
+notmuch dump | grep '^#@' > OUTPUT
+cat<<EOF > QUERIES.BEFORE
+#@ query.test date%3a2009-11-18..2009-11-18%20and%20tag%3aunread
+#@ query.test2 query%3atest%20and%20subject%3aMaildir
+EOF
+test_expect_equal_file QUERIES.BEFORE OUTPUT
+
+test_begin_subtest "delete named queries"
+notmuch dump > BEFORE
+notmuch config set query.test
+notmuch dump | grep '^#@' > OUTPUT
+cat<<EOF > EXPECTED
+#@ query.test2 query%3atest%20and%20subject%3aMaildir
+EOF
+test_expect_equal_file EXPECTED OUTPUT
+
+test_begin_subtest "restore named queries"
+notmuch restore < BEFORE
+notmuch dump | grep '^#@' > OUTPUT
+test_expect_equal_file QUERIES.BEFORE OUTPUT
+
+test_done
diff --git a/test/test-lib.sh b/test/test-lib.sh
index 98abbe2..00df2bd 100644
--- a/test/test-lib.sh
+++ b/test/test-lib.sh
@@ -745,6 +745,11 @@ notmuch_options_sanitize ()
     sed 's/^options[.]\(.*\)=.*$/options.\1=something/'
 }
 
+notmuch_config_sanitize ()
+{
+    notmuch_dir_sanitize | notmuch_options_sanitize
+}
+
 # End of notmuch helper functions
 
 # Use test_set_prereq to tell that a particular prerequisite is available.
-- 
2.7.0

[PATCH 12/13] lib: make a global constant for query parser flags

[David Bremner]

It's already kindof gross that this is hardcoded in two different
places. We will also need these later in field processors calling back
into the query parser.
---
 lib/database-private.h |  4 ++++
 lib/query.cc           | 16 ++--------------
 2 files changed, 6 insertions(+), 14 deletions(-)

diff --git a/lib/database-private.h b/lib/database-private.h
index 30a045e..59708d0 100644
--- a/lib/database-private.h
+++ b/lib/database-private.h
@@ -144,6 +144,10 @@ operator&=(_notmuch_features &a, _notmuch_features b)
     return a;
 }
 
+#define NOTMUCH_QUERY_PARSER_FLAGS (Xapian::QueryParser::FLAG_BOOLEAN | Xapian::QueryParser::FLAG_PHRASE | \
+				    Xapian::QueryParser::FLAG_LOVEHATE | Xapian::QueryParser::FLAG_BOOLEAN_ANY_CASE | \
+				    Xapian::QueryParser::FLAG_WILDCARD | Xapian::QueryParser::FLAG_PURE_NOT)
+
 #if HAVE_XAPIAN_FIELD_PROCESSOR
 class DateFieldProcessor;
 #endif
diff --git a/lib/query.cc b/lib/query.cc
index e627bfc..972a389 100644
--- a/lib/query.cc
+++ b/lib/query.cc
@@ -212,12 +212,6 @@ notmuch_query_search_messages_st (notmuch_query_t *query,
 	Xapian::Query string_query, final_query, exclude_query;
 	Xapian::MSet mset;
 	Xapian::MSetIterator iterator;
-	unsigned int flags = (Xapian::QueryParser::FLAG_BOOLEAN |
-			      Xapian::QueryParser::FLAG_PHRASE |
-			      Xapian::QueryParser::FLAG_LOVEHATE |
-			      Xapian::QueryParser::FLAG_BOOLEAN_ANY_CASE |
-			      Xapian::QueryParser::FLAG_WILDCARD |
-			      Xapian::QueryParser::FLAG_PURE_NOT);
 
 	if (strcmp (query_string, "") == 0 ||
 	    strcmp (query_string, "*") == 0)
@@ -225,7 +219,7 @@ notmuch_query_search_messages_st (notmuch_query_t *query,
 	    final_query = mail_query;
 	} else {
 	    string_query = notmuch->query_parser->
-		parse_query (query_string, flags);
+		parse_query (query_string, NOTMUCH_QUERY_PARSER_FLAGS);
 	    final_query = Xapian::Query (Xapian::Query::OP_AND,
 					 mail_query, string_query);
 	}
@@ -565,12 +559,6 @@ notmuch_query_count_messages_st (notmuch_query_t *query, unsigned *count_out)
 						   "mail"));
 	Xapian::Query string_query, final_query, exclude_query;
 	Xapian::MSet mset;
-	unsigned int flags = (Xapian::QueryParser::FLAG_BOOLEAN |
-			      Xapian::QueryParser::FLAG_PHRASE |
-			      Xapian::QueryParser::FLAG_LOVEHATE |
-			      Xapian::QueryParser::FLAG_BOOLEAN_ANY_CASE |
-			      Xapian::QueryParser::FLAG_WILDCARD |
-			      Xapian::QueryParser::FLAG_PURE_NOT);
 
 	if (strcmp (query_string, "") == 0 ||
 	    strcmp (query_string, "*") == 0)
@@ -578,7 +566,7 @@ notmuch_query_count_messages_st (notmuch_query_t *query, unsigned *count_out)
 	    final_query = mail_query;
 	} else {
 	    string_query = notmuch->query_parser->
-		parse_query (query_string, flags);
+		parse_query (query_string, NOTMUCH_QUERY_PARSER_FLAGS);
 	    final_query = Xapian::Query (Xapian::Query::OP_AND,
 					 mail_query, string_query);
 	}
-- 
2.7.0

[PATCH 13/13] lib: add support for named queries

[David Bremner]

This relies on the optional presense of xapian field processors, and the
library config API.
---
 lib/Makefile.local         |  1 +
 lib/database-private.h     |  2 ++
 lib/database.cc            |  3 +++
 lib/query-fp.cc            | 42 ++++++++++++++++++++++++++++++++++++++++++
 lib/query-fp.h             | 42 ++++++++++++++++++++++++++++++++++++++++++
 test/T600-named-queries.sh | 17 +++++++++++++++++
 6 files changed, 107 insertions(+)
 create mode 100644 lib/query-fp.cc
 create mode 100644 lib/query-fp.h

diff --git a/lib/Makefile.local b/lib/Makefile.local
index 96899bf..fab1242 100644
--- a/lib/Makefile.local
+++ b/lib/Makefile.local
@@ -49,6 +49,7 @@ libnotmuch_cxx_srcs =		\
 	$(dir)/index.cc		\
 	$(dir)/message.cc	\
 	$(dir)/query.cc		\
+	$(dir)/query-fp.cc \
 	$(dir)/config.cc	\
 	$(dir)/thread.cc
 
diff --git a/lib/database-private.h b/lib/database-private.h
index 59708d0..f7c6f46 100644
--- a/lib/database-private.h
+++ b/lib/database-private.h
@@ -150,6 +150,7 @@ operator&=(_notmuch_features &a, _notmuch_features b)
 
 #if HAVE_XAPIAN_FIELD_PROCESSOR
 class DateFieldProcessor;
+class QueryFieldProcessor;
 #endif
 struct _notmuch_database {
     notmuch_bool_t exception_reported;
@@ -185,6 +186,7 @@ struct _notmuch_database {
     Xapian::ValueRangeProcessor *date_range_processor;
 #if HAVE_XAPIAN_FIELD_PROCESSOR
     DateFieldProcessor *date_field_processor;
+    QueryFieldProcessor *query_field_processor;
 #endif
     Xapian::ValueRangeProcessor *last_mod_range_processor;
 };
diff --git a/lib/database.cc b/lib/database.cc
index 0e3c90c..8df5555 100644
--- a/lib/database.cc
+++ b/lib/database.cc
@@ -20,6 +20,7 @@
 
 #include "database-private.h"
 #include "parse-time-vrp.h"
+#include "query-fp.h"
 #include "string-util.h"
 
 #include <iostream>
@@ -1005,6 +1006,8 @@ notmuch_database_open_verbose (const char *path,
 	 * with a .. to the range processor */
 	notmuch->date_field_processor = new DateFieldProcessor();
 	notmuch->query_parser->add_boolean_prefix("date", notmuch->date_field_processor);
+	notmuch->query_field_processor = new QueryFieldProcessor (*notmuch->query_parser, notmuch);
+	notmuch->query_parser->add_boolean_prefix("query", notmuch->query_field_processor);
 #endif
 	notmuch->last_mod_range_processor = new Xapian::NumberValueRangeProcessor (NOTMUCH_VALUE_LAST_MOD, "lastmod:");
 
diff --git a/lib/query-fp.cc b/lib/query-fp.cc
new file mode 100644
index 0000000..ee5a306
--- /dev/null
+++ b/lib/query-fp.cc
@@ -0,0 +1,42 @@
+/* query-fp.cc -  "query:" field processor glue glue
+ *
+ * This file is part of notmuch.
+ *
+ * Copyright © 2016 David Bremner
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation, either version 3 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see http://www.gnu.org/licenses/ .
+ *
+ * Author: David Bremner <david@tethera.net>
+ */
+
+#include "database-private.h"
+#include "query-fp.h"
+#include <iostream>
+
+#if HAVE_XAPIAN_FIELD_PROCESSOR
+
+Xapian::Query QueryFieldProcessor::operator()(const std::string & name) {
+    std::string key = "query." + name;
+    char *expansion;
+    notmuch_status_t status;
+
+    status = notmuch_database_get_config (notmuch, key.c_str(), &expansion);
+    if (status) {
+	throw Xapian::QueryParserError("error looking up key" + name);
+
+    }
+
+    return parser.parse_query (expansion, NOTMUCH_QUERY_PARSER_FLAGS);
+}
+#endif
diff --git a/lib/query-fp.h b/lib/query-fp.h
new file mode 100644
index 0000000..67f8705
--- /dev/null
+++ b/lib/query-fp.h
@@ -0,0 +1,42 @@
+/* query-fp.h - query field processor glue
+ *
+ * This file is part of notmuch.
+ *
+ * Copyright © 2016 David Bremner
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation, either version 3 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see http://www.gnu.org/licenses/ .
+ *
+ * Author: David Bremner <david@tethera.net>
+ */
+
+#ifndef NOTMUCH_QUERY_FP_H
+#define NOTMUCH_QUERY_FP_H
+
+#include <xapian.h>
+#include "notmuch.h"
+
+#if HAVE_XAPIAN_FIELD_PROCESSOR
+class QueryFieldProcessor : public Xapian::FieldProcessor {
+ protected:
+    Xapian::QueryParser &parser;
+    notmuch_database_t *notmuch;
+
+ public:
+    QueryFieldProcessor (Xapian::QueryParser &parser_, notmuch_database_t *notmuch_)
+	: parser(parser_), notmuch(notmuch_) { };
+
+    Xapian::Query operator()(const std::string & str);
+};
+#endif
+#endif /* NOTMUCH_QUERY_FP_H */
diff --git a/test/T600-named-queries.sh b/test/T600-named-queries.sh
index 0922620..f0ae24f 100755
--- a/test/T600-named-queries.sh
+++ b/test/T600-named-queries.sh
@@ -50,4 +50,21 @@ notmuch restore < BEFORE
 notmuch dump | grep '^#@' > OUTPUT
 test_expect_equal_file QUERIES.BEFORE OUTPUT
 
+if [ $NOTMUCH_HAVE_XAPIAN_FIELD_PROCESSOR -eq 1 ]; then
+    test_begin_subtest "search named query"
+    notmuch search query:test > OUTPUT
+    notmuch search $QUERYSTR > EXPECTED
+    test_expect_equal_file EXPECTED OUTPUT
+
+    test_begin_subtest "search named query with other terms"
+    notmuch search query:test and subject:Maildir > OUTPUT
+    notmuch search $QUERYSTR and subject:Maildir > EXPECTED
+    test_expect_equal_file EXPECTED OUTPUT
+
+    test_begin_subtest "search nested named query"
+    notmuch search query:test2 > OUTPUT
+    notmuch search $QUERYSTR2 > EXPECTED
+    test_expect_equal_file EXPECTED OUTPUT
+fi
+
 test_done
-- 
2.7.0