v2 sexpr parser

v2 sexpr parser

[David Bremner]

This is a substantially revised version of the series at [1]. As far
as I know, it now understands (a translation of) most of the queries
handled by the existing query parser. Some remaining limitations/issues

1) The new query parser is only hooked into the notmuch search
subcommand. It should be fairly rote to hook it into the other
relevant subcommands, but I want to wait until resolving (2) before
proceeding.

2) The command line option --query-syntax={sexp,xapian} is a bit
klunky. Also "xapian" should perhaps be renamed "infix" to match the
'infix' operator in the new parser.

3) There is no documentation. I think notmuch-search-terms(7) is too
long already, so there should probably be a separate manual page. I
don't want to write that until I'm sure we want the new parser.

4) There is still some uncertainty around utf8 handling in sfsexp.

5) I'm not too sure about the new API call
notmuch_query_create_sexpr. I guess a more idiomatic thing to do would
be to add a new function with an extra argument, and have the old
function call it.

6) The way that user defined headers are used in the new parser is a
bit different than the existing one. Instead of (List notmuch), you
currently have to write (header List notmuch). I don't know if that's
better or worse. It's a bit more typing, but it is maybe a bit clearer to read.
It would probably not be too hard to switch.

7) Trailing wildcards like "subject:foo*" are not implemented yet.

In [2] Hannu mentioned being unclear on the design goals of the
s-expression query parser, so let me try and articulate the main
design goals a bit better. I think the existing query parser is great
for making "easy things easy". But when things are not easy and/or the
user wants better diagnostics, it is nice to have an alternative. 

A) More consistent / predictable syntax.

The notmuch query parser adds several features to the Xapian query
parser. Mainly due for implementation reasons, this has resulted in a
somewhat quirky syntax, and often fairly painful escaping. Probably
the most egregious syntax quirk is that '*' (for all messages) cannot
be composed with other queries. In particular is should simplify and
make more reliable code like "notmuch-search-filter", which tries to
combine an existing query with some user specified filter.
With the new parser, this 15-20 lines can be replaced by

`(and (infix ,existing) (infix ,new))

B) Better error reporting.

Xapian's query parser is designed to be permissive and almost never
rejects a query string.  This is not always ideal, particularly with
debugging constructed queries.

C) Extensibility

The Xapian Query API has functionality that is not (yet) exposed via
the QueryParser. It turns out that some common feature requests are
easy to add [3]. For example, to match messages with a List-Id header,
you can use '(header List :any)'. 

[1]: id:20210714000239.804384-1-david@tethera.net
[2]: id:60f190f8.1c69fb81.7e7d2.40d1@mx.google.com
[3]: In fairness, they would probably be fairly easy to add to the
Xapian QueryParser as well. But then we'd need to depend on a
sufficiently recent version.

[PATCH 01/25] configure: optional library sfsexp

[David Bremner]

This is essentially the same as the other checks using pkg-config.
---
 configure | 23 ++++++++++++++++++++++-
 1 file changed, 22 insertions(+), 1 deletion(-)

diff --git a/configure b/configure
index cfa9c09b..15c1924f 100755
--- a/configure
+++ b/configure
@@ -820,6 +820,19 @@ else
     WITH_BASH=0
 fi
 
+printf "Checking for sfsexp... "
+if pkg-config --exists sfsexp; then
+    printf "Yes.\n"
+    have_sfsexp=1
+    sfsexp_cflags=$(pkg-config --cflags sfsexp)
+    sfsexp_ldflags=$(pkg-config --libs sfsexp)
+else
+    printf "No (will not enable s-expression queries).\n"
+    have_sfsexp=0
+    sfsexp_cflags=
+    sfsexp_ldflags=
+fi
+
 if [ -z "${EMACSLISPDIR-}" ]; then
     EMACSLISPDIR="\$(prefix)/share/emacs/site-lisp"
 fi
@@ -1443,6 +1456,13 @@ HAVE_VALGRIND = ${have_valgrind}
 # And if so, flags needed at compile time for valgrind macros
 VALGRIND_CFLAGS = ${valgrind_cflags}
 
+# Whether the sfsexp library is available
+HAVE_SFSEXP = ${have_sfsexp}
+
+# And if so, flags needed at compile/link time for sfsexp
+SFSEXP_CFLAGS = ${sfsexp_cflags}
+SFSEXP_LDFLAGS = ${sfsexp_ldflags}
+
 # Support for emacs
 WITH_EMACS = ${WITH_EMACS}
 
@@ -1459,6 +1479,7 @@ WITH_ZSH = ${WITH_ZSH}
 COMMON_CONFIGURE_CFLAGS = \\
 	\$(GMIME_CFLAGS) \$(TALLOC_CFLAGS) \$(ZLIB_CFLAGS)	\\
 	-DHAVE_VALGRIND=\$(HAVE_VALGRIND) \$(VALGRIND_CFLAGS)	\\
+	-DHAVE_SFSEXP=\$(HAVE_SFSEXP) \$(SFSEXP_CFLAGS)		\\
 	-DHAVE_GETLINE=\$(HAVE_GETLINE)				\\
 	-DWITH_EMACS=\$(WITH_EMACS)				\\
 	-DHAVE_CANONICALIZE_FILE_NAME=\$(HAVE_CANONICALIZE_FILE_NAME) \\
@@ -1475,7 +1496,7 @@ CONFIGURE_CFLAGS = \$(COMMON_CONFIGURE_CFLAGS)
 
 CONFIGURE_CXXFLAGS = \$(COMMON_CONFIGURE_CFLAGS) \$(XAPIAN_CXXFLAGS)
 
-CONFIGURE_LDFLAGS = \$(GMIME_LDFLAGS) \$(TALLOC_LDFLAGS) \$(ZLIB_LDFLAGS) \$(XAPIAN_LDFLAGS)
+CONFIGURE_LDFLAGS = \$(GMIME_LDFLAGS) \$(TALLOC_LDFLAGS) \$(ZLIB_LDFLAGS) \$(XAPIAN_LDFLAGS) \$(SFSEXP_LDFLAGS)
 EOF
 
 # construct the sh.config
-- 
2.30.2

[PATCH 02/25] lib: split notmuch_query_create

[David Bremner]

Most of the function will be re-usable when creating a query from an
s-expression.
---
 lib/query.cc | 19 ++++++++++++++++---
 1 file changed, 16 insertions(+), 3 deletions(-)

diff --git a/lib/query.cc b/lib/query.cc
index 792aba21..39b85e91 100644
--- a/lib/query.cc
+++ b/lib/query.cc
@@ -84,9 +84,9 @@ _notmuch_query_destructor (notmuch_query_t *query)
     return 0;
 }
 
-notmuch_query_t *
-notmuch_query_create (notmuch_database_t *notmuch,
-		      const char *query_string)
+static notmuch_query_t *
+_notmuch_query_constructor (notmuch_database_t *notmuch,
+			    const char *query_string)
 {
     notmuch_query_t *query;
 
@@ -116,6 +116,19 @@ notmuch_query_create (notmuch_database_t *notmuch,
     return query;
 }
 
+notmuch_query_t *
+notmuch_query_create (notmuch_database_t *notmuch,
+		      const char *query_string)
+{
+
+    notmuch_query_t *query = _notmuch_query_constructor (notmuch, query_string);
+
+    if (! query)
+	return NULL;
+
+    return query;
+}
+
 static notmuch_status_t
 _notmuch_query_ensure_parsed (notmuch_query_t *query)
 {
-- 
2.30.2

[PATCH 03/25] lib: define notmuch_query_create_sexpr

[David Bremner]

Set the parsing syntax when the (notmuch) query object is
created. Initially the library always returns a trivial query that
matches all messages.
---
 lib/notmuch.h |  4 ++++
 lib/query.cc  | 52 +++++++++++++++++++++++++++++++++++++++++++++++----
 2 files changed, 52 insertions(+), 4 deletions(-)

diff --git a/lib/notmuch.h b/lib/notmuch.h
index 3b28bea3..a9abdd18 100644
--- a/lib/notmuch.h
+++ b/lib/notmuch.h
@@ -961,6 +961,10 @@ notmuch_query_t *
 notmuch_query_create (notmuch_database_t *database,
 		      const char *query_string);
 
+notmuch_query_t *
+notmuch_query_create_sexpr (notmuch_database_t *database,
+			    const char *query_string);
+
 /**
  * Sort values for notmuch_query_set_sort.
  */
diff --git a/lib/query.cc b/lib/query.cc
index 39b85e91..d05bd193 100644
--- a/lib/query.cc
+++ b/lib/query.cc
@@ -23,6 +23,13 @@
 
 #include <glib.h> /* GHashTable, GPtrArray */
 
+#include "sexp.h"
+
+typedef enum {
+    NOTMUCH_QUERY_SYNTAX_XAPIAN,
+    NOTMUCH_QUERY_SYNTAX_SEXPR,
+} notmuch_query_syntax_t;
+
 struct _notmuch_query {
     notmuch_database_t *notmuch;
     const char *query_string;
@@ -30,6 +37,7 @@ struct _notmuch_query {
     notmuch_string_list_t *exclude_terms;
     notmuch_exclude_t omit_excluded;
     bool parsed;
+    notmuch_query_syntax_t syntax;
     Xapian::Query xapian_query;
     std::set<std::string> terms;
 };
@@ -126,15 +134,29 @@ notmuch_query_create (notmuch_database_t *notmuch,
     if (! query)
 	return NULL;
 
+    query->syntax = NOTMUCH_QUERY_SYNTAX_XAPIAN;
+
     return query;
 }
 
-static notmuch_status_t
-_notmuch_query_ensure_parsed (notmuch_query_t *query)
+notmuch_query_t *
+notmuch_query_create_sexpr (notmuch_database_t *notmuch,
+			    const char *query_string)
 {
-    if (query->parsed)
-	return NOTMUCH_STATUS_SUCCESS;
 
+    notmuch_query_t *query = _notmuch_query_constructor (notmuch, query_string);
+
+    if (! query)
+	return NULL;
+
+    query->syntax = NOTMUCH_QUERY_SYNTAX_SEXPR;
+
+    return query;
+}
+
+static notmuch_status_t
+_notmuch_query_ensure_parsed_xapian (notmuch_query_t *query)
+{
     try {
 	query->xapian_query =
 	    query->notmuch->query_parser->
@@ -167,6 +189,28 @@ _notmuch_query_ensure_parsed (notmuch_query_t *query)
     return NOTMUCH_STATUS_SUCCESS;
 }
 
+static notmuch_status_t
+_notmuch_query_ensure_parsed_sexpr (notmuch_query_t *query)
+{
+    if (query->parsed)
+	return NOTMUCH_STATUS_SUCCESS;
+
+    query->xapian_query = Xapian::Query::MatchAll;
+    return NOTMUCH_STATUS_SUCCESS;
+}
+
+static notmuch_status_t
+_notmuch_query_ensure_parsed (notmuch_query_t *query)
+{
+    if (query->parsed)
+	return NOTMUCH_STATUS_SUCCESS;
+
+    if (query->syntax == NOTMUCH_QUERY_SYNTAX_SEXPR)
+	return _notmuch_query_ensure_parsed_sexpr (query);
+
+    return _notmuch_query_ensure_parsed_xapian (query);
+}
+
 const char *
 notmuch_query_get_query_string (const notmuch_query_t *query)
 {
-- 
2.30.2

[PATCH 04/25] CLI/search+address: support sexpr queries

[David Bremner]

Initially support selection of query syntax in two subcommands to
enable testing.
---
 notmuch-search.c     | 16 +++++++++++++++-
 test/T080-search.sh  |  5 +++++
 test/T095-address.sh |  5 +++++
 3 files changed, 25 insertions(+), 1 deletion(-)

diff --git a/notmuch-search.c b/notmuch-search.c
index 244817a9..fd25a414 100644
--- a/notmuch-search.c
+++ b/notmuch-search.c
@@ -43,6 +43,11 @@ typedef enum {
     DEDUP_ADDRESS,
 } dedup_t;
 
+typedef enum {
+    QUERY_SYNTAX_XAPIAN,
+    QUERY_SYNTAX_SEXP
+} query_syntax_t;
+
 typedef enum {
     NOTMUCH_FORMAT_JSON,
     NOTMUCH_FORMAT_TEXT,
@@ -56,6 +61,7 @@ typedef struct {
     int format_sel;
     sprinter_t *format;
     int exclude;
+    int query_syntax;
     notmuch_query_t *query;
     int sort;
     int output;
@@ -721,7 +727,10 @@ _notmuch_search_prepare (search_context_t *ctx, int argc, char *argv[])
 	return EXIT_FAILURE;
     }
 
-    ctx->query = notmuch_query_create (ctx->notmuch, query_str);
+    if (ctx->query_syntax == QUERY_SYNTAX_SEXP)
+	ctx->query = notmuch_query_create_sexpr (ctx->notmuch, query_str);
+    else
+	ctx->query = notmuch_query_create (ctx->notmuch, query_str);
     if (ctx->query == NULL) {
 	fprintf (stderr, "Out of memory\n");
 	return EXIT_FAILURE;
@@ -771,6 +780,7 @@ static search_context_t search_context = {
     .format_sel = NOTMUCH_FORMAT_TEXT,
     .exclude = NOTMUCH_EXCLUDE_TRUE,
     .sort = NOTMUCH_SORT_NEWEST_FIRST,
+    .query_syntax = QUERY_SYNTAX_XAPIAN,
     .output = 0,
     .offset = 0,
     .limit = -1, /* unlimited */
@@ -789,6 +799,10 @@ static const notmuch_opt_desc_t common_options[] = {
 				  { "text", NOTMUCH_FORMAT_TEXT },
 				  { "text0", NOTMUCH_FORMAT_TEXT0 },
 				  { 0, 0 } } },
+    { .opt_keyword = &search_context.query_syntax, .name = "query-syntax", .keywords =
+	  (notmuch_keyword_t []){ { "xapian", QUERY_SYNTAX_XAPIAN },
+				  { "sexp", QUERY_SYNTAX_SEXP },
+				  { 0, 0 } } },
     { .opt_int = &notmuch_format_version, .name = "format-version" },
     { }
 };
diff --git a/test/T080-search.sh b/test/T080-search.sh
index a3f0dead..966e772a 100755
--- a/test/T080-search.sh
+++ b/test/T080-search.sh
@@ -189,4 +189,9 @@ test_begin_subtest "parts do not have adjacent term positions"
 output=$(notmuch search id:termpos and '"c x"')
 test_expect_equal "$output" ""
 
+test_begin_subtest "sexpr query: all messages"
+notmuch search '*' > EXPECTED
+notmuch search --query-syntax=sexp '()' > OUTPUT
+test_expect_equal_file EXPECTED OUTPUT
+
 test_done
diff --git a/test/T095-address.sh b/test/T095-address.sh
index 817be538..adf0b307 100755
--- a/test/T095-address.sh
+++ b/test/T095-address.sh
@@ -325,4 +325,9 @@ cat <<EOF >EXPECTED
 EOF
 test_expect_equal_file EXPECTED OUTPUT
 
+test_begin_subtest "sexpr query: all messages"
+notmuch address '*' > EXPECTED
+notmuch address --query-syntax=sexp '()' > OUTPUT
+test_expect_equal_file EXPECTED OUTPUT
+
 test_done
-- 
2.30.2

[PATCH 05/25] lib: add new status code for query syntax errors.

[David Bremner]

This will help provide more meaningful error messages without special
casing on the client side.
---
 bindings/python-cffi/notmuch2/_build.py  | 1 +
 bindings/python-cffi/notmuch2/_errors.py | 3 +++
 lib/database.cc                          | 2 ++
 lib/notmuch.h                            | 4 ++++
 4 files changed, 10 insertions(+)

diff --git a/bindings/python-cffi/notmuch2/_build.py b/bindings/python-cffi/notmuch2/_build.py
index f712b6c5..24df939e 100644
--- a/bindings/python-cffi/notmuch2/_build.py
+++ b/bindings/python-cffi/notmuch2/_build.py
@@ -53,6 +53,7 @@ ffibuilder.cdef(
         NOTMUCH_STATUS_NO_CONFIG,
         NOTMUCH_STATUS_NO_DATABASE,
         NOTMUCH_STATUS_DATABASE_EXISTS,
+        NOTMUCH_STATUS_BAD_QUERY_SYNTAX,
         NOTMUCH_STATUS_LAST_STATUS
     } notmuch_status_t;
     typedef enum {
diff --git a/bindings/python-cffi/notmuch2/_errors.py b/bindings/python-cffi/notmuch2/_errors.py
index 9301073e..f55cc96b 100644
--- a/bindings/python-cffi/notmuch2/_errors.py
+++ b/bindings/python-cffi/notmuch2/_errors.py
@@ -56,6 +56,8 @@ class NotmuchError(Exception):
                 NoDatabaseError,
             capi.lib.NOTMUCH_STATUS_DATABASE_EXISTS:
                 DatabaseExistsError,
+            capi.lib.NOTMUCH_STATUS_BAD_QUERY_SYNTAX:
+                QuerySyntaxError,
         }
         return types[status]
 
@@ -103,6 +105,7 @@ class IllegalArgumentError(NotmuchError): pass
 class NoConfigError(NotmuchError): pass
 class NoDatabaseError(NotmuchError): pass
 class DatabaseExistsError(NotmuchError): pass
+class QuerySyntaxError(NotmuchError): pass
 
 class ObjectDestroyedError(NotmuchError):
     """The object has already been destroyed and it's memory freed.
diff --git a/lib/database.cc b/lib/database.cc
index 31794900..7eb0de79 100644
--- a/lib/database.cc
+++ b/lib/database.cc
@@ -309,6 +309,8 @@ notmuch_status_to_string (notmuch_status_t status)
 	return "No database found";
     case NOTMUCH_STATUS_DATABASE_EXISTS:
 	return "Database exists, not recreated";
+    case NOTMUCH_STATUS_BAD_QUERY_SYNTAX:
+	return "Syntax error in query";
     default:
     case NOTMUCH_STATUS_LAST_STATUS:
 	return "Unknown error status value";
diff --git a/lib/notmuch.h b/lib/notmuch.h
index a9abdd18..e609f1c4 100644
--- a/lib/notmuch.h
+++ b/lib/notmuch.h
@@ -220,6 +220,10 @@ typedef enum _notmuch_status {
      * Database exists, so not (re)-created
      */
     NOTMUCH_STATUS_DATABASE_EXISTS,
+    /**
+     * Syntax error in query
+     */
+    NOTMUCH_STATUS_BAD_QUERY_SYNTAX,
     /**
      * Not an actual status value. Just a way to find out how many
      * valid status values there are.
-- 
2.30.2

[PATCH 06/25] lib/parse-sexp: parse 'and', 'not', 'or'

[David Bremner]

This is not too useful yet, but provides gluing together queries in
various ways, which is actually one of the main motivations for
the s-expression query format.
---
 lib/Makefile.local        |  3 +-
 lib/parse-sexp.cc         | 92 +++++++++++++++++++++++++++++++++++++++
 lib/parse-sexp.h          |  6 +++
 lib/query.cc              |  7 ++-
 test/T080-search.sh       |  5 ---
 test/T081-sexpr-search.sh | 34 +++++++++++++++
 6 files changed, 137 insertions(+), 10 deletions(-)
 create mode 100644 lib/parse-sexp.cc
 create mode 100644 lib/parse-sexp.h
 create mode 100755 test/T081-sexpr-search.sh

diff --git a/lib/Makefile.local b/lib/Makefile.local
index e2d4b91d..1378a74b 100644
--- a/lib/Makefile.local
+++ b/lib/Makefile.local
@@ -63,7 +63,8 @@ libnotmuch_cxx_srcs =		\
 	$(dir)/features.cc	\
 	$(dir)/prefix.cc	\
 	$(dir)/open.cc		\
-	$(dir)/init.cc
+	$(dir)/init.cc		\
+	$(dir)/parse-sexp.cc
 
 libnotmuch_modules := $(libnotmuch_c_srcs:.c=.o) $(libnotmuch_cxx_srcs:.cc=.o)
 
diff --git a/lib/parse-sexp.cc b/lib/parse-sexp.cc
new file mode 100644
index 00000000..dfbebf2b
--- /dev/null
+++ b/lib/parse-sexp.cc
@@ -0,0 +1,92 @@
+#include <xapian.h>
+#include "notmuch-private.h"
+#include "sexp.h"
+
+typedef struct  {
+    const char *name;
+    Xapian::Query::op xapian_op;
+    Xapian::Query initial;
+} _sexp_op_t;
+
+static _sexp_op_t operations[] =
+{
+    { "and",    Xapian::Query::OP_AND,          Xapian::Query::MatchAll },
+    { "not",    Xapian::Query::OP_AND_NOT,      Xapian::Query::MatchAll },
+    { "or",     Xapian::Query::OP_OR,           Xapian::Query::MatchNothing },
+    { }
+};
+
+static notmuch_status_t _sexp_to_xapian_query (notmuch_database_t *notmuch, const sexp_t *sx,
+					       Xapian::Query &output);
+
+static notmuch_status_t
+_sexp_combine_query (notmuch_database_t *notmuch,
+		     Xapian::Query::op operation,
+		     Xapian::Query left,
+		     const sexp_t *sx,
+		     Xapian::Query &output)
+{
+    Xapian::Query subquery;
+
+    notmuch_status_t status;
+
+    /* if we run out elements, return accumulator */
+
+    if (! sx) {
+	output = left;
+	return NOTMUCH_STATUS_SUCCESS;
+    }
+
+    status = _sexp_to_xapian_query (notmuch, sx, subquery);
+    if (status)
+	return status;
+
+    return _sexp_combine_query (notmuch,
+				operation,
+				Xapian::Query (operation, left, subquery),
+				sx->next, output);
+}
+
+notmuch_status_t
+_notmuch_sexp_string_to_xapian_query (notmuch_database_t *notmuch, const char *querystr,
+				      Xapian::Query &output)
+{
+    const sexp_t *sx = NULL;
+    char *buf = talloc_strdup (notmuch, querystr);
+
+    sx = parse_sexp (buf, strlen (querystr));
+    if (! sx)
+	return NOTMUCH_STATUS_BAD_QUERY_SYNTAX;
+
+    return _sexp_to_xapian_query (notmuch, sx, output);
+}
+
+/* Here we expect the s-expression to be a proper list, with first
+ * element defining and operation, or as a special case the empty
+ * list */
+
+static notmuch_status_t
+_sexp_to_xapian_query (notmuch_database_t *notmuch, const sexp_t *sx, Xapian::Query &output)
+{
+
+    const _sexp_op_t *op;
+
+    /* Currently we don't understand atoms */
+    assert (sx->ty == SEXP_LIST);
+
+    /* Empty list */
+    if (! sx->list) {
+	output = Xapian::Query::MatchAll;
+	return NOTMUCH_STATUS_SUCCESS;
+    }
+
+    for (op = operations; op && op->name; op++) {
+	if (strcasecmp (op->name, hd_sexp (sx)->val) == 0) {
+	    return _sexp_combine_query (notmuch, op->xapian_op, op->initial, sx->list->next, output);
+	}
+
+    }
+
+    _notmuch_database_log_append (notmuch, "unimplemented prefix %s\n", sx->list->val);
+    return NOTMUCH_STATUS_BAD_QUERY_SYNTAX;
+}
diff --git a/lib/parse-sexp.h b/lib/parse-sexp.h
new file mode 100644
index 00000000..a358bf26
--- /dev/null
+++ b/lib/parse-sexp.h
@@ -0,0 +1,6 @@
+#ifndef _PARSE_SEXP_H
+#define _PARSE_SEXP_H
+/* parse_sexp.cc */
+notmuch_status_t _notmuch_sexp_string_to_xapian_query (notmuch_database_t *notmuch, const
+						       char *querystr, Xapian::Query &output);
+#endif
diff --git a/lib/query.cc b/lib/query.cc
index d05bd193..a3cba662 100644
--- a/lib/query.cc
+++ b/lib/query.cc
@@ -20,11 +20,10 @@
 
 #include "notmuch-private.h"
 #include "database-private.h"
+#include "parse-sexp.h"
 
 #include <glib.h> /* GHashTable, GPtrArray */
 
-#include "sexp.h"
-
 typedef enum {
     NOTMUCH_QUERY_SYNTAX_XAPIAN,
     NOTMUCH_QUERY_SYNTAX_SEXPR,
@@ -195,8 +194,8 @@ _notmuch_query_ensure_parsed_sexpr (notmuch_query_t *query)
     if (query->parsed)
 	return NOTMUCH_STATUS_SUCCESS;
 
-    query->xapian_query = Xapian::Query::MatchAll;
-    return NOTMUCH_STATUS_SUCCESS;
+    return _notmuch_sexp_string_to_xapian_query (query->notmuch, query->query_string,
+						 query->xapian_query);
 }
 
 static notmuch_status_t
diff --git a/test/T080-search.sh b/test/T080-search.sh
index 966e772a..a3f0dead 100755
--- a/test/T080-search.sh
+++ b/test/T080-search.sh
@@ -189,9 +189,4 @@ test_begin_subtest "parts do not have adjacent term positions"
 output=$(notmuch search id:termpos and '"c x"')
 test_expect_equal "$output" ""
 
-test_begin_subtest "sexpr query: all messages"
-notmuch search '*' > EXPECTED
-notmuch search --query-syntax=sexp '()' > OUTPUT
-test_expect_equal_file EXPECTED OUTPUT
-
 test_done
diff --git a/test/T081-sexpr-search.sh b/test/T081-sexpr-search.sh
new file mode 100755
index 00000000..41a82886
--- /dev/null
+++ b/test/T081-sexpr-search.sh
@@ -0,0 +1,34 @@
+#!/usr/bin/env bash
+test_description='"notmuch search" in several variations'
+. $(dirname "$0")/test-lib.sh || exit 1
+
+add_email_corpus
+
+for query in '()' '(not)' '(and)' '(or ())' '(or (not))' '(or (and))' \
+	     '(or (and) (or) (not (and)))'; do
+    test_begin_subtest "all messages: $query"
+    notmuch search '*' > EXPECTED
+    notmuch search --query-syntax=sexp "$query" > OUTPUT
+    test_expect_equal_file EXPECTED OUTPUT
+done
+
+for query in '(or)' '(not ())' '(not (not))' '(not (and))' \
+		    '(not (or (and) (or) (not (and))))'; do
+    test_begin_subtest "no messages: $query"
+    notmuch search --query-syntax=sexp "$query" > OUTPUT
+    test_expect_equal_file /dev/null OUTPUT
+done
+
+test_begin_subtest "Unbalanced parens"
+# A code 1 indicates the error was handled (a crash will return e.g. 139).
+test_expect_code 1 "notmuch search --query-syntax=sexp '('"
+
+test_begin_subtest "unknown_prefix"
+notmuch search --query-syntax=sexp '(foo)' >OUTPUT 2>&1
+cat <<EOF > EXPECTED
+notmuch search: Syntax error in query
+unimplemented prefix foo
+EOF
+test_expect_equal_file EXPECTED OUTPUT
+
+test_done
-- 
2.30.2

[PATCH 07/25] lib/parse-sexp: parse 'subject'

[David Bremner]

Initially this only works for lists of individual terms, which is a
bit inconvenient. This will be improved in a following commit.
---
 lib/parse-sexp.cc         | 37 +++++++++++++++++++++++++++++++++++++
 test/T081-sexpr-search.sh | 15 +++++++++++++++
 2 files changed, 52 insertions(+)

diff --git a/lib/parse-sexp.cc b/lib/parse-sexp.cc
index dfbebf2b..898cfdd0 100644
--- a/lib/parse-sexp.cc
+++ b/lib/parse-sexp.cc
@@ -16,6 +16,17 @@ static _sexp_op_t operations[] =
     { }
 };
 
+typedef struct  {
+    const char *name;
+    Xapian::Query::op xapian_op;
+} _sexp_field_t;
+
+static _sexp_field_t fields[] =
+{
+    { "subject",        Xapian::Query::OP_PHRASE },
+    { }
+};
+
 static notmuch_status_t _sexp_to_xapian_query (notmuch_database_t *notmuch, const sexp_t *sx,
 					       Xapian::Query &output);
 
@@ -61,6 +72,27 @@ _notmuch_sexp_string_to_xapian_query (notmuch_database_t *notmuch, const char *q
     return _sexp_to_xapian_query (notmuch, sx, output);
 }
 
+static notmuch_status_t
+_sexp_combine_field (const char *prefix,
+		     Xapian::Query::op operation,
+		     const sexp_t *sx,
+		     Xapian::Query &output)
+{
+    std::vector<std::string> terms;
+
+    for (const sexp_t *cur = sx; cur; cur = cur->next) {
+	std::string pref_str = prefix;
+	std::string word = cur->val;
+
+	if (operation == Xapian::Query::OP_PHRASE)
+	    word = Xapian::Unicode::tolower (word);
+
+	terms.push_back (pref_str + word);
+    }
+    output = Xapian::Query (operation, terms.begin (), terms.end ());
+    return NOTMUCH_STATUS_SUCCESS;
+}
+
 /* Here we expect the s-expression to be a proper list, with first
  * element defining and operation, or as a special case the empty
  * list */
@@ -84,7 +116,12 @@ _sexp_to_xapian_query (notmuch_database_t *notmuch, const sexp_t *sx, Xapian::Qu
 	if (strcasecmp (op->name, hd_sexp (sx)->val) == 0) {
 	    return _sexp_combine_query (notmuch, op->xapian_op, op->initial, sx->list->next, output);
 	}
+    }
 
+    for (const _sexp_field_t *field = fields; field && field->name; field++) {
+	if (strcasecmp (field->name, hd_sexp (sx)->val) == 0)
+	    return _sexp_combine_field (_find_prefix (field->name), field->xapian_op, sx->list->next,
+					output);
     }
 
     _notmuch_database_log_append (notmuch, "unimplemented prefix %s\n", sx->list->val);
diff --git a/test/T081-sexpr-search.sh b/test/T081-sexpr-search.sh
index 41a82886..872f2603 100755
--- a/test/T081-sexpr-search.sh
+++ b/test/T081-sexpr-search.sh
@@ -19,6 +19,21 @@ for query in '(or)' '(not ())' '(not (not))' '(not (and))' \
     test_expect_equal_file /dev/null OUTPUT
 done
 
+test_begin_subtest "Search by subject"
+add_message [subject]=subjectsearchtest '[date]="Sat, 01 Jan 2000 12:00:00 -0000"'
+output=$(notmuch search --query-syntax=sexp '(subject subjectsearchtest)' | notmuch_search_sanitize)
+test_expect_equal "$output" "thread:XXX   2000-01-01 [1/1] Notmuch Test Suite; subjectsearchtest (inbox unread)"
+
+test_begin_subtest "Search by subject (case insensitive)"
+notmuch search tag:inbox and subject:maildir | notmuch_search_sanitize > EXPECTED
+notmuch search --query-syntax=sexp '(subject Maildir)' | notmuch_search_sanitize > OUTPUT
+test_expect_equal_file EXPECTED OUTPUT
+
+test_begin_subtest "Search by subject (utf-8):"
+add_message [subject]=utf8-sübjéct '[date]="Sat, 01 Jan 2000 12:00:00 -0000"'
+output=$(notmuch search --query-syntax=sexp '(subject utf8 sübjéct)' | notmuch_search_sanitize)
+test_expect_equal "$output" "thread:XXX   2000-01-01 [1/1] Notmuch Test Suite; utf8-sübjéct (inbox unread)"
+
 test_begin_subtest "Unbalanced parens"
 # A code 1 indicates the error was handled (a crash will return e.g. 139).
 test_expect_code 1 "notmuch search --query-syntax=sexp '('"
-- 
2.30.2\r

[PATCH 08/25] lib/parse-sexp: split terms in phrase mode

[David Bremner]

The goal is to have (subject foo-bar) match the same messages as
subject:foo-bar.
---
 lib/parse-sexp.cc         | 38 +++++++++++++++++++++++++++++++++-----
 test/T081-sexpr-search.sh |  8 ++++++++
 2 files changed, 41 insertions(+), 5 deletions(-)

diff --git a/lib/parse-sexp.cc b/lib/parse-sexp.cc
index 898cfdd0..fc6eb2d7 100644
--- a/lib/parse-sexp.cc
+++ b/lib/parse-sexp.cc
@@ -72,6 +72,34 @@ _notmuch_sexp_string_to_xapian_query (notmuch_database_t *notmuch, const char *q
     return _sexp_to_xapian_query (notmuch, sx, output);
 }
 
+static void
+_sexp_find_words (const char *str, std::string pref_str, std::vector<std::string> &terms)
+{
+    Xapian::Utf8Iterator p (str);
+    Xapian::Utf8Iterator end;
+
+    while (p != end) {
+	Xapian::Utf8Iterator start;
+	while (p != end && ! Xapian::Unicode::is_wordchar (*p))
+	    p++;
+
+	if (p == end)
+	    break;
+
+	start = p;
+
+	while (p != end && Xapian::Unicode::is_wordchar (*p))
+	    p++;
+
+	if (p != start) {
+	    std::string word (start, p);
+	    word = Xapian::Unicode::tolower (word);
+	    terms.push_back (pref_str + word);
+	}
+    }
+
+}
+
 static notmuch_status_t
 _sexp_combine_field (const char *prefix,
 		     Xapian::Query::op operation,
@@ -82,12 +110,12 @@ _sexp_combine_field (const char *prefix,
 
     for (const sexp_t *cur = sx; cur; cur = cur->next) {
 	std::string pref_str = prefix;
-	std::string word = cur->val;
 
-	if (operation == Xapian::Query::OP_PHRASE)
-	    word = Xapian::Unicode::tolower (word);
-
-	terms.push_back (pref_str + word);
+	if (operation == Xapian::Query::OP_PHRASE) {
+	    _sexp_find_words (cur->val, pref_str, terms);
+	} else {
+	    terms.push_back (pref_str + cur->val);
+	}
     }
     output = Xapian::Query (operation, terms.begin (), terms.end ());
     return NOTMUCH_STATUS_SUCCESS;
diff --git a/test/T081-sexpr-search.sh b/test/T081-sexpr-search.sh
index 872f2603..8e042f88 100755
--- a/test/T081-sexpr-search.sh
+++ b/test/T081-sexpr-search.sh
@@ -34,6 +34,14 @@ add_message [subject]=utf8-sübjéct '[date]="Sat, 01 Jan 2000 12:00:00 -0000"'
 output=$(notmuch search --query-syntax=sexp '(subject utf8 sübjéct)' | notmuch_search_sanitize)
 test_expect_equal "$output" "thread:XXX   2000-01-01 [1/1] Notmuch Test Suite; utf8-sübjéct (inbox unread)"
 
+test_begin_subtest "Search by 'subject' (utf-8, phrase-token):"
+output=$(notmuch search --query-syntax=sexp '(subject utf8-sübjéct)' | notmuch_search_sanitize)
+test_expect_equal "$output" "thread:XXX   2000-01-01 [1/1] Notmuch Test Suite; utf8-sübjéct (inbox unread)"
+
+test_begin_subtest "Search by 'subject' (utf-8, quoted string):"
+output=$(notmuch search --query-syntax=sexp '(subject "utf8 sübjéct")' | notmuch_search_sanitize)
+test_expect_equal "$output" "thread:XXX   2000-01-01 [1/1] Notmuch Test Suite; utf8-sübjéct (inbox unread)"
+
 test_begin_subtest "Unbalanced parens"
 # A code 1 indicates the error was handled (a crash will return e.g. 139).
 test_expect_code 1 "notmuch search --query-syntax=sexp '('"
-- 
2.30.2\r

[PATCH 09/25] lib/parse-sexp: handle most fields

[David Bremner]

date and query will need to be handled separately because they do not
fit the pattern of combining a bunch of terms with an operator.
---
 lib/parse-sexp.cc         |  13 ++++
 test/T081-sexpr-search.sh | 143 +++++++++++++++++++++++++++++++++++++-
 2 files changed, 153 insertions(+), 3 deletions(-)

diff --git a/lib/parse-sexp.cc b/lib/parse-sexp.cc
index fc6eb2d7..5865dc88 100644
--- a/lib/parse-sexp.cc
+++ b/lib/parse-sexp.cc
@@ -23,7 +23,20 @@ typedef struct  {
 
 static _sexp_field_t fields[] =
 {
+    { "attachment",     Xapian::Query::OP_PHRASE },
+    { "body",           Xapian::Query::OP_PHRASE },
+    { "from",           Xapian::Query::OP_PHRASE },
+    { "folder",         Xapian::Query::OP_OR },
+    { "id",             Xapian::Query::OP_OR },
+    { "is",             Xapian::Query::OP_AND },
+    { "mid",            Xapian::Query::OP_OR },
+    { "mimetype",       Xapian::Query::OP_PHRASE },
+    { "path",           Xapian::Query::OP_OR },
+    { "property",       Xapian::Query::OP_AND },
     { "subject",        Xapian::Query::OP_PHRASE },
+    { "tag",            Xapian::Query::OP_AND },
+    { "thread",         Xapian::Query::OP_OR },
+    { "to",             Xapian::Query::OP_PHRASE },
     { }
 };
 
diff --git a/test/T081-sexpr-search.sh b/test/T081-sexpr-search.sh
index 8e042f88..95837448 100755
--- a/test/T081-sexpr-search.sh
+++ b/test/T081-sexpr-search.sh
@@ -19,17 +19,110 @@ for query in '(or)' '(not ())' '(not (not))' '(not (and))' \
     test_expect_equal_file /dev/null OUTPUT
 done
 
-test_begin_subtest "Search by subject"
+test_begin_subtest "Search by 'attachment'"
+notmuch search attachment:notmuch-help.patch > EXPECTED
+notmuch search --query-syntax=sexp '(attachment notmuch-help.patch)' > OUTPUT
+test_expect_equal_file EXPECTED OUTPUT
+
+test_begin_subtest "Search by 'body'"
+add_message '[subject]="body search"' '[date]="Sat, 01 Jan 2000 12:00:00 -0000"' [body]=bodysearchtest
+output=$(notmuch search --query-syntax=sexp '(body bodysearchtest)' | notmuch_search_sanitize)
+test_expect_equal "$output" "thread:XXX   2000-01-01 [1/1] Notmuch Test Suite; body search (inbox unread)"
+
+test_begin_subtest "Search by 'body' (phrase)"
+add_message '[subject]="body search (phrase)"' '[date]="Sat, 01 Jan 2000 12:00:00 -0000"' '[body]="body search (phrase)"'
+add_message '[subject]="negative result"' '[date]="Sat, 01 Jan 2000 12:00:00 -0000"' '[body]="This phrase should not match the body search"'
+output=$(notmuch search --query-syntax=sexp '(body body search phrase)' | notmuch_search_sanitize)
+test_expect_equal "$output" "thread:XXX   2000-01-01 [1/1] Notmuch Test Suite; body search (phrase) (inbox unread)"
+
+test_begin_subtest "Search by 'body' (utf-8):"
+add_message '[subject]="utf8-message-body-subject"' '[date]="Sat, 01 Jan 2000 12:00:00 -0000"' '[body]="message body utf8: bödý"'
+output=$(notmuch search --query-syntax=sexp '(body bödý)' | notmuch_search_sanitize)
+test_expect_equal "$output" "thread:XXX   2000-01-01 [1/1] Notmuch Test Suite; utf8-message-body-subject (inbox unread)"
+
+test_begin_subtest "Search by 'from'"
+add_message '[subject]="search by from"' '[date]="Sat, 01 Jan 2000 12:00:00 -0000"' [from]=searchbyfrom
+output=$(notmuch search --query-syntax=sexp '(from searchbyfrom)' | notmuch_search_sanitize)
+test_expect_equal "$output" "thread:XXX   2000-01-01 [1/1] searchbyfrom; search by from (inbox unread)"
+
+test_begin_subtest "Search by 'from' (address)"
+add_message '[subject]="search by from (address)"' '[date]="Sat, 01 Jan 2000 12:00:00 -0000"' [from]=searchbyfrom@example.com
+output=$(notmuch search --query-syntax=sexp '(from searchbyfrom@example.com)' | notmuch_search_sanitize)
+test_expect_equal "$output" "thread:XXX   2000-01-01 [1/1] searchbyfrom@example.com; search by from (address) (inbox unread)"
+
+test_begin_subtest "Search by 'from' (name)"
+add_message '[subject]="search by from (name)"' '[date]="Sat, 01 Jan 2000 12:00:00 -0000"' '[from]="Search By From Name <test@example.com>"'
+output=$(notmuch search --query-syntax=sexp '(from "Search By From Name")' | notmuch_search_sanitize)
+test_expect_equal "$output" "thread:XXX   2000-01-01 [1/1] Search By From Name; search by from (name) (inbox unread)"
+
+test_begin_subtest "Search by 'from' (name and address)"
+output=$(notmuch search --query-syntax=sexp '(from "Search By From Name <test@example.com>")' | notmuch_search_sanitize)
+test_expect_equal "$output" "thread:XXX   2000-01-01 [1/1] Search By From Name; search by from (name) (inbox unread)"
+
+add_message '[dir]=bad' '[subject]="To the bone"'
+add_message '[dir]=.' '[subject]="Top level"'
+add_message '[dir]=bad/news' '[subject]="Bears"'
+mkdir -p "${MAIL_DIR}/duplicate/bad/news"
+cp "$gen_msg_filename" "${MAIL_DIR}/duplicate/bad/news"
+
+add_message '[dir]=things' '[subject]="These are a few"'
+add_message '[dir]=things/favorite' '[subject]="Raindrops, whiskers, kettles"'
+add_message '[dir]=things/bad' '[subject]="Bites, stings, sad feelings"'
+
+test_begin_subtest "Search by 'folder' (multiple)"
+output=$(notmuch search --query-syntax=sexp '(folder bad bad/news things/bad)' | notmuch_search_sanitize)
+test_expect_equal "$output" "thread:XXX   2001-01-05 [1/1] Notmuch Test Suite; To the bone (inbox unread)
+thread:XXX   2001-01-05 [1/1(2)] Notmuch Test Suite; Bears (inbox unread)
+thread:XXX   2001-01-05 [1/1] Notmuch Test Suite; Bites, stings, sad feelings (inbox unread)"
+
+test_begin_subtest "Search by 'folder': top level."
+notmuch search folder:'""' > EXPECTED
+notmuch search --query-syntax=sexp '(folder "")'  > OUTPUT
+test_expect_equal_file EXPECTED OUTPUT
+
+test_begin_subtest "Search by 'id'"
+add_message '[subject]="search by id"' '[date]="Sat, 01 Jan 2000 12:00:00 -0000"'
+output=$(notmuch search --query-syntax=sexp "(id ${gen_msg_id})" | notmuch_search_sanitize)
+test_expect_equal "$output" "thread:XXX   2000-01-01 [1/1] Notmuch Test Suite; search by id (inbox unread)"
+
+test_begin_subtest "Search by 'id' (or)"
+add_message '[subject]="search by id"' '[date]="Sat, 01 Jan 2000 12:00:00 -0000"'
+output=$(notmuch search --query-syntax=sexp "(id non-existent-mid ${gen_msg_id})" | notmuch_search_sanitize)
+test_expect_equal "$output" "thread:XXX   2000-01-01 [1/1] Notmuch Test Suite; search by id (inbox unread)"
+
+test_begin_subtest "Search by 'is' (multiple)"
+notmuch tag -inbox tag:searchbytag
+notmuch search is:inbox AND is:unread | notmuch_search_sanitize > EXPECTED
+notmuch search --query-syntax=sexp '(is inbox unread)' | notmuch_search_sanitize > OUTPUT
+notmuch tag +inbox tag:searchbytag
+test_expect_equal_file EXPECTED OUTPUT
+
+test_begin_subtest "Search by 'mid'"
+add_message '[subject]="search by mid"' '[date]="Sat, 01 Jan 2000 12:00:00 -0000"'
+output=$(notmuch search --query-syntax=sexp "(mid ${gen_msg_id})" | notmuch_search_sanitize)
+test_expect_equal "$output" "thread:XXX   2000-01-01 [1/1] Notmuch Test Suite; search by mid (inbox unread)"
+
+test_begin_subtest "Search by 'mid' (or)"
+add_message '[subject]="search by mid"' '[date]="Sat, 01 Jan 2000 12:00:00 -0000"'
+output=$(notmuch search --query-syntax=sexp "(mid non-existent-mid ${gen_msg_id})" | notmuch_search_sanitize)
+test_expect_equal "$output" "thread:XXX   2000-01-01 [1/1] Notmuch Test Suite; search by mid (inbox unread)"
+
+test_begin_subtest "Search by 'mimetype'"
+notmuch search mimetype:text/html > EXPECTED
+notmuch search --query-syntax=sexp '(mimetype text html)'  > OUTPUT
+test_expect_equal_file EXPECTED OUTPUT
+
+test_begin_subtest "Search by 'subject'"
 add_message [subject]=subjectsearchtest '[date]="Sat, 01 Jan 2000 12:00:00 -0000"'
 output=$(notmuch search --query-syntax=sexp '(subject subjectsearchtest)' | notmuch_search_sanitize)
 test_expect_equal "$output" "thread:XXX   2000-01-01 [1/1] Notmuch Test Suite; subjectsearchtest (inbox unread)"
 
-test_begin_subtest "Search by subject (case insensitive)"
+test_begin_subtest "Search by 'subject' (case insensitive)"
 notmuch search tag:inbox and subject:maildir | notmuch_search_sanitize > EXPECTED
 notmuch search --query-syntax=sexp '(subject Maildir)' | notmuch_search_sanitize > OUTPUT
 test_expect_equal_file EXPECTED OUTPUT
 
-test_begin_subtest "Search by subject (utf-8):"
+test_begin_subtest "Search by 'subject' (utf-8):"
 add_message [subject]=utf8-sübjéct '[date]="Sat, 01 Jan 2000 12:00:00 -0000"'
 output=$(notmuch search --query-syntax=sexp '(subject utf8 sübjéct)' | notmuch_search_sanitize)
 test_expect_equal "$output" "thread:XXX   2000-01-01 [1/1] Notmuch Test Suite; utf8-sübjéct (inbox unread)"
@@ -42,6 +135,50 @@ test_begin_subtest "Search by 'subject' (utf-8, quoted string):"
 output=$(notmuch search --query-syntax=sexp '(subject "utf8 sübjéct")' | notmuch_search_sanitize)
 test_expect_equal "$output" "thread:XXX   2000-01-01 [1/1] Notmuch Test Suite; utf8-sübjéct (inbox unread)"
 
+test_begin_subtest "Search by 'tag'"
+add_message '[subject]="search by tag"' '[date]="Sat, 01 Jan 2000 12:00:00 -0000"'
+notmuch tag +searchbytag id:${gen_msg_id}
+output=$(notmuch search --query-syntax=sexp '(tag searchbytag)' | notmuch_search_sanitize)
+test_expect_equal "$output" "thread:XXX   2000-01-01 [1/1] Notmuch Test Suite; search by tag (inbox searchbytag unread)"
+
+test_begin_subtest "Search by 'tag' (multiple)"
+notmuch tag -inbox tag:searchbytag
+notmuch search tag:inbox AND tag:unread | notmuch_search_sanitize > EXPECTED
+notmuch search --query-syntax=sexp '(tag inbox unread)' | notmuch_search_sanitize > OUTPUT
+notmuch tag +inbox tag:searchbytag
+test_expect_equal_file EXPECTED OUTPUT
+
+test_begin_subtest "Search by 'tag' and 'subject'"
+notmuch search tag:inbox and subject:maildir | notmuch_search_sanitize > EXPECTED
+notmuch search --query-syntax=sexp '(and (tag inbox) (subject maildir))' | notmuch_search_sanitize > OUTPUT
+test_expect_equal_file EXPECTED OUTPUT
+
+test_begin_subtest "Search by 'thread'"
+add_message '[subject]="search by thread"' '[date]="Sat, 01 Jan 2000 12:00:00 -0000"'
+thread_id=$(notmuch search id:${gen_msg_id} | sed -e "s/thread:\([a-f0-9]*\).*/\1/")
+output=$(notmuch search --query-syntax=sexp "(thread ${thread_id})" | notmuch_search_sanitize)
+test_expect_equal "$output" "thread:XXX   2000-01-01 [1/1] Notmuch Test Suite; search by thread (inbox unread)"
+
+test_begin_subtest "Search by 'to'"
+add_message '[subject]="search by to"' '[date]="Sat, 01 Jan 2000 12:00:00 -0000"' [to]=searchbyto
+output=$(notmuch search --query-syntax=sexp '(to searchbyto)' | notmuch_search_sanitize)
+test_expect_equal "$output" "thread:XXX   2000-01-01 [1/1] Notmuch Test Suite; search by to (inbox unread)"
+
+test_begin_subtest "Search by 'to' (address)"
+add_message '[subject]="search by to (address)"' '[date]="Sat, 01 Jan 2000 12:00:00 -0000"' [to]=searchbyto@example.com
+output=$(notmuch search --query-syntax=sexp '(to searchbyto@example.com)' | notmuch_search_sanitize)
+test_expect_equal "$output" "thread:XXX   2000-01-01 [1/1] Notmuch Test Suite; search by to (address) (inbox unread)"
+
+test_begin_subtest "Search by 'to' (name)"
+add_message '[subject]="search by to (name)"' '[date]="Sat, 01 Jan 2000 12:00:00 -0000"' '[to]="Search By To Name <test@example.com>"'
+output=$(notmuch search --query-syntax=sexp '(to "Search By To Name")' | notmuch_search_sanitize)
+test_expect_equal "$output" "thread:XXX   2000-01-01 [1/1] Notmuch Test Suite; search by to (name) (inbox unread)"
+
+test_begin_subtest "Search by 'to' (name and address)"
+output=$(notmuch search --query-syntax=sexp '(to "Search By To Name <test@example.com>")' | notmuch_search_sanitize)
+test_expect_equal "$output" "thread:XXX   2000-01-01 [1/1] Notmuch Test Suite; search by to (name) (inbox unread)"
+
+
 test_begin_subtest "Unbalanced parens"
 # A code 1 indicates the error was handled (a crash will return e.g. 139).
 test_expect_code 1 "notmuch search --query-syntax=sexp '('"
-- 
2.30.2\r

[PATCH 10/25] lib/parse-sexp: handle unprefixed terms.

[David Bremner]

This is equivalent to adding the same field name "" for multiple
prefixes in the Xapian query parser, but we have to explicitely
construct the resulting query.
---
 lib/parse-sexp.cc         | 15 ++++++++++++---
 test/T081-sexpr-search.sh | 27 +++++++++++++++++++++++++++
 2 files changed, 39 insertions(+), 3 deletions(-)

diff --git a/lib/parse-sexp.cc b/lib/parse-sexp.cc
index 5865dc88..c8bc3432 100644
--- a/lib/parse-sexp.cc
+++ b/lib/parse-sexp.cc
@@ -144,9 +144,18 @@ _sexp_to_xapian_query (notmuch_database_t *notmuch, const sexp_t *sx, Xapian::Qu
 
     const _sexp_op_t *op;
 
-    /* Currently we don't understand atoms */
-    assert (sx->ty == SEXP_LIST);
-
+    if (sx->ty == SEXP_VALUE) {
+	Xapian::Query accumulator;
+	for (const _sexp_field_t *field = fields; field && field->name; field++) {
+	    std::vector<std::string> terms;
+	    _sexp_find_words (sx->val, _find_prefix (field->name), terms);
+	    accumulator = Xapian::Query (Xapian::Query::OP_OR, accumulator,
+					 Xapian::Query (Xapian::Query::OP_PHRASE,
+							terms.begin (), terms.end ()));
+	}
+	output = accumulator;
+	return NOTMUCH_STATUS_SUCCESS;
+    }
     /* Empty list */
     if (! sx->list) {
 	output = Xapian::Query::MatchAll;
diff --git a/test/T081-sexpr-search.sh b/test/T081-sexpr-search.sh
index 95837448..80e3daf3 100755
--- a/test/T081-sexpr-search.sh
+++ b/test/T081-sexpr-search.sh
@@ -29,6 +29,10 @@ add_message '[subject]="body search"' '[date]="Sat, 01 Jan 2000 12:00:00 -0000"'
 output=$(notmuch search --query-syntax=sexp '(body bodysearchtest)' | notmuch_search_sanitize)
 test_expect_equal "$output" "thread:XXX   2000-01-01 [1/1] Notmuch Test Suite; body search (inbox unread)"
 
+test_begin_subtest "Search by body (unprefixed)"
+output=$(notmuch search --query-syntax=sexp '(and bodysearchtest)' | notmuch_search_sanitize)
+test_expect_equal "$output" "thread:XXX   2000-01-01 [1/1] Notmuch Test Suite; body search (inbox unread)"
+
 test_begin_subtest "Search by 'body' (phrase)"
 add_message '[subject]="body search (phrase)"' '[date]="Sat, 01 Jan 2000 12:00:00 -0000"' '[body]="body search (phrase)"'
 add_message '[subject]="negative result"' '[date]="Sat, 01 Jan 2000 12:00:00 -0000"' '[body]="This phrase should not match the body search"'
@@ -40,6 +44,29 @@ add_message '[subject]="utf8-message-body-subject"' '[date]="Sat, 01 Jan 2000 12
 output=$(notmuch search --query-syntax=sexp '(body bödý)' | notmuch_search_sanitize)
 test_expect_equal "$output" "thread:XXX   2000-01-01 [1/1] Notmuch Test Suite; utf8-message-body-subject (inbox unread)"
 
+add_message "[body]=thebody-1" "[subject]=kryptonite-1"
+add_message "[body]=nothing-to-see-here-1" "[subject]=thebody-1"
+
+test_begin_subtest 'search without body: prefix'
+notmuch search thebody > EXPECTED
+notmuch search --query-syntax=sexp '(and thebody)' > OUTPUT
+test_expect_equal_file EXPECTED OUTPUT
+
+test_begin_subtest 'negated body: prefix'
+notmuch search thebody and not body:thebody > EXPECTED
+notmuch search --query-syntax=sexp '(and (not (body thebody)) thebody)' > OUTPUT
+test_expect_equal_file EXPECTED OUTPUT
+
+test_begin_subtest 'search unprefixed for prefixed term'
+notmuch search kryptonite > EXPECTED
+notmuch search --query-syntax=sexp '(and kryptonite)' > OUTPUT
+test_expect_equal_file EXPECTED OUTPUT
+
+test_begin_subtest 'search with body: prefix for term only in subject'
+notmuch search body:kryptonite > EXPECTED
+notmuch search --query-syntax=sexp '(body kryptonite)' > OUTPUT
+test_expect_equal_file EXPECTED OUTPUT
+
 test_begin_subtest "Search by 'from'"
 add_message '[subject]="search by from"' '[date]="Sat, 01 Jan 2000 12:00:00 -0000"' [from]=searchbyfrom
 output=$(notmuch search --query-syntax=sexp '(from searchbyfrom)' | notmuch_search_sanitize)
-- 
2.30.2\r

[PATCH 11/25] lib: factor out date to query conversion

[David Bremner]

This is a bit messy, but throwing and catching
Xapian::QueryParserError exceptions outside of the Xapian query parser
seems worse.
---
 lib/parse-time-vrp.cc | 97 +++++++++++++++++++++++++++++++------------
 lib/parse-time-vrp.h  |  8 ++++
 2 files changed, 79 insertions(+), 26 deletions(-)

diff --git a/lib/parse-time-vrp.cc b/lib/parse-time-vrp.cc
index 22bf2ab5..75c67797 100644
--- a/lib/parse-time-vrp.cc
+++ b/lib/parse-time-vrp.cc
@@ -24,21 +24,26 @@
 #include "parse-time-vrp.h"
 #include "parse-time-string.h"
 
-Xapian::Query
-ParseTimeRangeProcessor::operator() (const std::string &begin, const std::string &end)
+notmuch_status_t
+_notmuch_time_range_to_query (Xapian::valueno slot, const std::string &begin, const std::string &end,
+			      std::string &msg, Xapian::Query &output)
 {
     double from = DBL_MIN, to = DBL_MAX;
     time_t parsed_time, now;
     std::string str;
 
     /* Use the same 'now' for begin and end. */
-    if (time (&now) == (time_t) -1)
-	throw Xapian::QueryParserError ("unable to get current time");
+    if (time (&now) == (time_t) -1) {
+	msg = "unable to get current time";
+	/* XXX Give a better status value */
+	return NOTMUCH_STATUS_ILLEGAL_ARGUMENT;
+    }
 
     if (! begin.empty ()) {
-	if (parse_time_string (begin.c_str (), &parsed_time, &now, PARSE_TIME_ROUND_DOWN))
-	    throw Xapian::QueryParserError ("Didn't understand date specification '" + begin + "'");
-	else
+	if (parse_time_string (begin.c_str (), &parsed_time, &now, PARSE_TIME_ROUND_DOWN)) {
+	    msg = "Didn't understand date specification '" + begin + "'";
+	    return NOTMUCH_STATUS_ILLEGAL_ARGUMENT;
+	} else
 	    from = (double) parsed_time;
     }
 
@@ -48,39 +53,79 @@ ParseTimeRangeProcessor::operator() (const std::string &begin, const std::string
 	else
 	    str = end;
 
-	if (parse_time_string (str.c_str (), &parsed_time, &now, PARSE_TIME_ROUND_UP_INCLUSIVE))
-	    throw Xapian::QueryParserError ("Didn't understand date specification '" + str + "'");
-	else
+	if (parse_time_string (str.c_str (), &parsed_time, &now, PARSE_TIME_ROUND_UP_INCLUSIVE)) {
+	    msg = "Didn't understand date specification '" + str + "'";
+	    return NOTMUCH_STATUS_ILLEGAL_ARGUMENT;
+	} else
 	    to = (double) parsed_time;
     }
 
-    return Xapian::Query (Xapian::Query::OP_VALUE_RANGE, slot,
-			  Xapian::sortable_serialise (from),
-			  Xapian::sortable_serialise (to));
+    output = Xapian::Query (Xapian::Query::OP_VALUE_RANGE, slot,
+			    Xapian::sortable_serialise (from),
+			    Xapian::sortable_serialise (to));
+
+    return NOTMUCH_STATUS_SUCCESS;
 }
 
-/* XXX TODO: is throwing an exception the right thing to do here? */
 Xapian::Query
-DateFieldProcessor::operator() (const std::string & str)
+ParseTimeRangeProcessor::operator() (const std::string &begin, const std::string &end)
+{
+    Xapian::Query output;
+    notmuch_status_t status;
+    std::string msg;
+
+    status = _notmuch_time_range_to_query (slot, begin, end, msg, output);
+    if (status)
+	throw Xapian::QueryParserError (msg);
+
+    return output;
+}
+
+notmuch_status_t
+_notmuch_time_string_to_query (Xapian::valueno slot, const std::string &str,
+			       std::string &msg, Xapian::Query &output)
 {
     double from = DBL_MIN, to = DBL_MAX;
     time_t parsed_time, now;
 
     /* Use the same 'now' for begin and end. */
-    if (time (&now) == (time_t) -1)
-	throw Xapian::QueryParserError ("Unable to get current time");
+    if (time (&now) == (time_t) -1) {
+	msg = "Unable to get current time";
+	return NOTMUCH_STATUS_ILLEGAL_ARGUMENT;
+    }
 
-    if (parse_time_string (str.c_str (), &parsed_time, &now, PARSE_TIME_ROUND_DOWN))
-	throw Xapian::QueryParserError ("Didn't understand date specification '" + str + "'");
-    else
+    if (parse_time_string (str.c_str (), &parsed_time, &now, PARSE_TIME_ROUND_DOWN)) {
+	msg = "Didn't understand date specification '" + str + "'";
+	return NOTMUCH_STATUS_ILLEGAL_ARGUMENT;
+    } else
 	from = (double) parsed_time;
 
-    if (parse_time_string (str.c_str (), &parsed_time, &now, PARSE_TIME_ROUND_UP_INCLUSIVE))
-	throw Xapian::QueryParserError ("Didn't understand date specification '" + str + "'");
-    else
+    if (parse_time_string (str.c_str (), &parsed_time, &now, PARSE_TIME_ROUND_UP_INCLUSIVE)) {
+	msg = "Didn't understand date specification '" + str + "'";
+	return NOTMUCH_STATUS_ILLEGAL_ARGUMENT;
+    } else
 	to = (double) parsed_time;
 
-    return Xapian::Query (Xapian::Query::OP_VALUE_RANGE, slot,
-			  Xapian::sortable_serialise (from),
-			  Xapian::sortable_serialise (to));
+    output = Xapian::Query (Xapian::Query::OP_VALUE_RANGE, slot,
+			    Xapian::sortable_serialise (from),
+			    Xapian::sortable_serialise (to));
+
+    return NOTMUCH_STATUS_SUCCESS;
+
+}
+
+/* XXX TODO: is throwing an exception the right thing to do here? */
+Xapian::Query
+DateFieldProcessor::operator() (const std::string & str)
+{
+    Xapian::Query output;
+    notmuch_status_t status;
+    std::string msg;
+
+    status = _notmuch_time_string_to_query (slot, str, msg, output);
+    if (status)
+	throw Xapian::QueryParserError (msg);
+
+    return output;
+
 }
diff --git a/lib/parse-time-vrp.h b/lib/parse-time-vrp.h
index f495e716..76d16eb2 100644
--- a/lib/parse-time-vrp.h
+++ b/lib/parse-time-vrp.h
@@ -25,6 +25,14 @@
 
 #include <xapian.h>
 
+/* for use outside the Xapian query parser */
+notmuch_status_t
+_notmuch_time_range_to_query (Xapian::valueno slot, const std::string &begin, const std::string &end,
+			      std::string &msg, Xapian::Query &output);
+notmuch_status_t
+_notmuch_time_string_to_query (Xapian::valueno slot, const std::string &str,
+			       std::string &msg, Xapian::Query &output);
+
 /* see *ValueRangeProcessor in xapian-core/include/xapian/queryparser.h */
 class ParseTimeRangeProcessor : public Xapian::RangeProcessor {
 
-- 
2.30.2

[PATCH 12/25] lib/parse-sexp: parse date fields

[David Bremner]

This is mainly just marshalling the arguments and calling the existing
query generator. It's not obvious why date is in the field table now,
but we will later use this to track what flags (options) are permitted
per field.
---
 lib/parse-sexp.cc         | 42 ++++++++++++++++++++++++++++++++++++++-
 test/T081-sexpr-search.sh | 15 +++++++++++++-
 2 files changed, 55 insertions(+), 2 deletions(-)

diff --git a/lib/parse-sexp.cc b/lib/parse-sexp.cc
index c8bc3432..29c5cd31 100644
--- a/lib/parse-sexp.cc
+++ b/lib/parse-sexp.cc
@@ -1,6 +1,7 @@
 #include <xapian.h>
 #include "notmuch-private.h"
 #include "sexp.h"
+#include "parse-time-vrp.h"
 
 typedef struct  {
     const char *name;
@@ -25,6 +26,7 @@ static _sexp_field_t fields[] =
 {
     { "attachment",     Xapian::Query::OP_PHRASE },
     { "body",           Xapian::Query::OP_PHRASE },
+    { "date",           Xapian::Query::OP_INVALID },
     { "from",           Xapian::Query::OP_PHRASE },
     { "folder",         Xapian::Query::OP_OR },
     { "id",             Xapian::Query::OP_OR },
@@ -134,6 +136,36 @@ _sexp_combine_field (const char *prefix,
     return NOTMUCH_STATUS_SUCCESS;
 }
 
+static notmuch_status_t
+_sexp_parse_date (notmuch_database_t *notmuch, const sexp_t *sx, Xapian::Query &output)
+{
+    std::string begin, end, msg;
+    notmuch_status_t status;
+    const sexp_t *cur = sx->list->next;
+
+    if (cur) {
+	begin = cur->val;
+	cur = cur->next;
+	if (cur) {
+	    end = cur->val;
+	    cur = cur->next;
+	    if (cur) {
+		_notmuch_database_log_append (notmuch, "extra argument(s) to date");
+		return NOTMUCH_STATUS_BAD_QUERY_SYNTAX;
+	    }
+	    status = _notmuch_time_range_to_query (NOTMUCH_VALUE_TIMESTAMP, begin, end, msg, output);
+	} else {
+	    status = _notmuch_time_string_to_query (NOTMUCH_VALUE_TIMESTAMP, begin, msg, output);
+	}
+	if (status)
+	    _notmuch_database_log_append (notmuch, "%s", msg.c_str ());
+	return status;
+    } else {
+	_notmuch_database_log_append (notmuch, "missing argument(s) to date");
+	return NOTMUCH_STATUS_BAD_QUERY_SYNTAX;
+    }
+}
+
 /* Here we expect the s-expression to be a proper list, with first
  * element defining and operation, or as a special case the empty
  * list */
@@ -148,6 +180,8 @@ _sexp_to_xapian_query (notmuch_database_t *notmuch, const sexp_t *sx, Xapian::Qu
 	Xapian::Query accumulator;
 	for (const _sexp_field_t *field = fields; field && field->name; field++) {
 	    std::vector<std::string> terms;
+	    if (field->xapian_op == Xapian::Query::OP_INVALID)
+		continue;
 	    _sexp_find_words (sx->val, _find_prefix (field->name), terms);
 	    accumulator = Xapian::Query (Xapian::Query::OP_OR, accumulator,
 					 Xapian::Query (Xapian::Query::OP_PHRASE,
@@ -169,9 +203,15 @@ _sexp_to_xapian_query (notmuch_database_t *notmuch, const sexp_t *sx, Xapian::Qu
     }
 
     for (const _sexp_field_t *field = fields; field && field->name; field++) {
-	if (strcasecmp (field->name, hd_sexp (sx)->val) == 0)
+	if (strcasecmp (field->name, hd_sexp (sx)->val) == 0) {
+	    /* some fields need to be handled specially */
+	    if (strcasecmp (field->name, "date") == 0) {
+		return _sexp_parse_date (notmuch, sx, output);
+	    }
+
 	    return _sexp_combine_field (_find_prefix (field->name), field->xapian_op, sx->list->next,
 					output);
+	}
     }
 
     _notmuch_database_log_append (notmuch, "unimplemented prefix %s\n", sx->list->val);
diff --git a/test/T081-sexpr-search.sh b/test/T081-sexpr-search.sh
index 80e3daf3..c9dd8f39 100755
--- a/test/T081-sexpr-search.sh
+++ b/test/T081-sexpr-search.sh
@@ -67,6 +67,20 @@ notmuch search body:kryptonite > EXPECTED
 notmuch search --query-syntax=sexp '(body kryptonite)' > OUTPUT
 test_expect_equal_file EXPECTED OUTPUT
 
+test_begin_subtest "date 0 arguments"
+test_expect_code 1 "notmuch search --query-syntax=sexp '(date)'"
+
+test_begin_subtest "date 1 argument"
+output=$(notmuch search --query-syntax=sexp '(date 2010-12-16)' | notmuch_search_sanitize)
+test_expect_equal "$output" "thread:XXX   2010-12-16 [1/1] Olivier Berger; Essai accentué (inbox unread)"
+
+test_begin_subtest "date 2 arguments"
+output=$(notmuch search --query-syntax=sexp '(date 2010-12-16 12/16/2010)' | notmuch_search_sanitize)
+test_expect_equal "$output" "thread:XXX   2010-12-16 [1/1] Olivier Berger; Essai accentué (inbox unread)"
+
+test_begin_subtest "date 3 arguments"
+test_expect_code 1 "notmuch search --query-syntax=sexp '(date 2010-12-16 12/16/2010 trailing-garbage)'"
+
 test_begin_subtest "Search by 'from'"
 add_message '[subject]="search by from"' '[date]="Sat, 01 Jan 2000 12:00:00 -0000"' [from]=searchbyfrom
 output=$(notmuch search --query-syntax=sexp '(from searchbyfrom)' | notmuch_search_sanitize)
@@ -205,7 +219,6 @@ test_begin_subtest "Search by 'to' (name and address)"
 output=$(notmuch search --query-syntax=sexp '(to "Search By To Name <test@example.com>")' | notmuch_search_sanitize)
 test_expect_equal "$output" "thread:XXX   2000-01-01 [1/1] Notmuch Test Suite; search by to (name) (inbox unread)"
 
-
 test_begin_subtest "Unbalanced parens"
 # A code 1 indicates the error was handled (a crash will return e.g. 139).
 test_expect_code 1 "notmuch search --query-syntax=sexp '('"
-- 
2.30.2\r

[PATCH 13/25] lib: factor out expansion of saved queries.

[David Bremner]

This is intended to allow use outside of the Xapian query parser.
---
 lib/query-fp.cc | 22 +++++++++++++++++++---
 lib/query-fp.h  |  4 ++++
 2 files changed, 23 insertions(+), 3 deletions(-)

diff --git a/lib/query-fp.cc b/lib/query-fp.cc
index b980b7f0..75b1d875 100644
--- a/lib/query-fp.cc
+++ b/lib/query-fp.cc
@@ -24,17 +24,33 @@
 #include "query-fp.h"
 #include <iostream>
 
-Xapian::Query
-QueryFieldProcessor::operator() (const std::string & name)
+notmuch_status_t
+_notmuch_query_name_to_query (notmuch_database_t *notmuch, const std::string name,
+			      Xapian::Query &output)
 {
     std::string key = "query." + name;
     char *expansion;
     notmuch_status_t status;
 
     status = notmuch_database_get_config (notmuch, key.c_str (), &expansion);
+    if (status)
+	return status;
+
+    output = notmuch->query_parser->parse_query (expansion, NOTMUCH_QUERY_PARSER_FLAGS);
+    return NOTMUCH_STATUS_SUCCESS;
+}
+
+Xapian::Query
+QueryFieldProcessor::operator() (const std::string & name)
+{
+    notmuch_status_t status;
+    Xapian::Query output;
+
+    status = _notmuch_query_name_to_query (notmuch, name, output);
     if (status) {
 	throw Xapian::QueryParserError ("error looking up key" + name);
     }
 
-    return parser.parse_query (expansion, NOTMUCH_QUERY_PARSER_FLAGS);
+    return output;
+
 }
diff --git a/lib/query-fp.h b/lib/query-fp.h
index beaaf405..a1b12bb9 100644
--- a/lib/query-fp.h
+++ b/lib/query-fp.h
@@ -26,6 +26,10 @@
 #include <xapian.h>
 #include "notmuch.h"
 
+notmuch_status_t
+_notmuch_query_name_to_query (notmuch_database_t *notmuch, const std::string name,
+			      Xapian::Query &output);
+
 class QueryFieldProcessor : public Xapian::FieldProcessor {
 protected:
     Xapian::QueryParser &parser;
-- 
2.30.2

[PATCH 14/25] lib/parse-sexp: handle saved queries

[David Bremner]

This provides functionality analogous to query: in the Xapian
QueryParser based parser. Perhaps counterintuitively, the saved
queries currently have to be in the original query syntax (i.e. not
s-expressions).
---
 lib/parse-sexp.cc         | 10 ++++++++++
 test/T081-sexpr-search.sh | 28 ++++++++++++++++++++++++++++
 2 files changed, 38 insertions(+)

diff --git a/lib/parse-sexp.cc b/lib/parse-sexp.cc
index 29c5cd31..1be2a4be 100644
--- a/lib/parse-sexp.cc
+++ b/lib/parse-sexp.cc
@@ -2,6 +2,7 @@
 #include "notmuch-private.h"
 #include "sexp.h"
 #include "parse-time-vrp.h"
+#include "query-fp.h"
 
 typedef struct  {
     const char *name;
@@ -35,6 +36,7 @@ static _sexp_field_t fields[] =
     { "mimetype",       Xapian::Query::OP_PHRASE },
     { "path",           Xapian::Query::OP_OR },
     { "property",       Xapian::Query::OP_AND },
+    { "query",          Xapian::Query::OP_INVALID },
     { "subject",        Xapian::Query::OP_PHRASE },
     { "tag",            Xapian::Query::OP_AND },
     { "thread",         Xapian::Query::OP_OR },
@@ -208,6 +210,14 @@ _sexp_to_xapian_query (notmuch_database_t *notmuch, const sexp_t *sx, Xapian::Qu
 	    if (strcasecmp (field->name, "date") == 0) {
 		return _sexp_parse_date (notmuch, sx, output);
 	    }
+	    if (strcasecmp (field->name, "query") == 0) {
+		if (! sx->list->next || ! sx->list->next->val) {
+		    _notmuch_database_log (notmuch, "missing query name\n");
+		    return NOTMUCH_STATUS_BAD_QUERY_SYNTAX;
+		} else {
+		    return _notmuch_query_name_to_query (notmuch, sx->list->next->val, output);
+		}
+	    }
 
 	    return _sexp_combine_field (_find_prefix (field->name), field->xapian_op, sx->list->next,
 					output);
diff --git a/test/T081-sexpr-search.sh b/test/T081-sexpr-search.sh
index c9dd8f39..89f1c36f 100755
--- a/test/T081-sexpr-search.sh
+++ b/test/T081-sexpr-search.sh
@@ -153,6 +153,34 @@ notmuch search mimetype:text/html > EXPECTED
 notmuch search --query-syntax=sexp '(mimetype text html)'  > OUTPUT
 test_expect_equal_file EXPECTED OUTPUT
 
+QUERYSTR="date:2009-11-18..2009-11-18 and tag:unread"
+QUERYSTR2="query:test and subject:Maildir"
+notmuch config set --database query.test "$QUERYSTR"
+notmuch config set query.test2 "$QUERYSTR2"
+
+test_begin_subtest "ill-formed named query search"
+notmuch search --query-syntax=sexp '(query)' > OUTPUT 2>&1
+cat <<EOF > EXPECTED
+notmuch search: Syntax error in query
+missing query name
+EOF
+test_expect_equal_file EXPECTED OUTPUT
+
+test_begin_subtest "search named query"
+notmuch search --query-syntax=sexp '(query test)' > OUTPUT
+notmuch search $QUERYSTR > EXPECTED
+test_expect_equal_file EXPECTED OUTPUT
+
+test_begin_subtest "search named query with other terms"
+notmuch search --query-syntax=sexp '(and (query test) (subject Maildir))' > OUTPUT
+notmuch search $QUERYSTR and subject:Maildir > EXPECTED
+test_expect_equal_file EXPECTED OUTPUT
+
+test_begin_subtest "search nested named query"
+notmuch search --query-syntax=sexp '(query test2)' > OUTPUT
+notmuch search $QUERYSTR2 > EXPECTED
+test_expect_equal_file EXPECTED OUTPUT
+
 test_begin_subtest "Search by 'subject'"
 add_message [subject]=subjectsearchtest '[date]="Sat, 01 Jan 2000 12:00:00 -0000"'
 output=$(notmuch search --query-syntax=sexp '(subject subjectsearchtest)' | notmuch_search_sanitize)
-- 
2.30.2

[PATCH 15/25] lib/parse-sexp: add keyword arguments for fields

[David Bremner]

Nothing is done with the flags here, only parsing and validation.
---
 lib/parse-sexp.cc         | 116 +++++++++++++++++++++++++++++++-------
 test/T081-sexpr-search.sh |  31 ++++++++++
 2 files changed, 127 insertions(+), 20 deletions(-)

diff --git a/lib/parse-sexp.cc b/lib/parse-sexp.cc
index 1be2a4be..14420c0e 100644
--- a/lib/parse-sexp.cc
+++ b/lib/parse-sexp.cc
@@ -10,6 +10,34 @@ typedef struct  {
     Xapian::Query initial;
 } _sexp_op_t;
 
+typedef enum {
+    SEXP_FLAG_NONE	= 0,
+    SEXP_FLAG_WILDCARD	= 1 << 0,
+} _sexp_flag_t;
+
+/*
+ * define bitwise operators to hide casts */
+
+inline _sexp_flag_t
+operator| (_sexp_flag_t a, _sexp_flag_t b)
+{
+    return static_cast<_sexp_flag_t>(
+	static_cast<unsigned>(a) | static_cast<unsigned>(b));
+}
+
+inline _sexp_flag_t
+operator& (_sexp_flag_t a, _sexp_flag_t b)
+{
+    return static_cast<_sexp_flag_t>(
+	static_cast<unsigned>(a) & static_cast<unsigned>(b));
+}
+
+typedef struct  {
+    const char *name;
+    Xapian::Query::op xapian_op;
+    _sexp_flag_t flags_allowed;
+} _sexp_field_t;
+
 static _sexp_op_t operations[] =
 {
     { "and",    Xapian::Query::OP_AND,          Xapian::Query::MatchAll },
@@ -18,29 +46,36 @@ static _sexp_op_t operations[] =
     { }
 };
 
-typedef struct  {
+static _sexp_field_t fields[] =
+{
+    { "attachment",   Xapian::Query::OP_PHRASE,       SEXP_FLAG_WILDCARD },
+    { "body",         Xapian::Query::OP_PHRASE,       SEXP_FLAG_WILDCARD },
+    { "date",         Xapian::Query::OP_INVALID,      SEXP_FLAG_NONE },
+    { "from",         Xapian::Query::OP_PHRASE,       SEXP_FLAG_NONE },
+    { "folder",       Xapian::Query::OP_OR,           SEXP_FLAG_NONE },
+    { "id",           Xapian::Query::OP_OR,           SEXP_FLAG_NONE },
+    { "is",           Xapian::Query::OP_AND,          SEXP_FLAG_WILDCARD },
+    { "mid",          Xapian::Query::OP_OR,           SEXP_FLAG_NONE },
+    { "mimetype",     Xapian::Query::OP_PHRASE,       SEXP_FLAG_NONE },
+    { "path",         Xapian::Query::OP_OR,           SEXP_FLAG_NONE },
+    { "property",     Xapian::Query::OP_AND,          SEXP_FLAG_WILDCARD },
+    { "query",        Xapian::Query::OP_INVALID,      SEXP_FLAG_NONE },
+    { "subject",      Xapian::Query::OP_PHRASE,       SEXP_FLAG_NONE },
+    { "tag",          Xapian::Query::OP_AND,          SEXP_FLAG_WILDCARD },
+    { "thread",       Xapian::Query::OP_OR,           SEXP_FLAG_NONE },
+    { "to",           Xapian::Query::OP_PHRASE,       SEXP_FLAG_NONE },
+    { }
+};
+
+typedef struct {
     const char *name;
-    Xapian::Query::op xapian_op;
-} _sexp_field_t;
+    _sexp_flag_t flag;
+} _sexp_keyword_t;
 
-static _sexp_field_t fields[] =
+static _sexp_keyword_t keywords[] =
 {
-    { "attachment",     Xapian::Query::OP_PHRASE },
-    { "body",           Xapian::Query::OP_PHRASE },
-    { "date",           Xapian::Query::OP_INVALID },
-    { "from",           Xapian::Query::OP_PHRASE },
-    { "folder",         Xapian::Query::OP_OR },
-    { "id",             Xapian::Query::OP_OR },
-    { "is",             Xapian::Query::OP_AND },
-    { "mid",            Xapian::Query::OP_OR },
-    { "mimetype",       Xapian::Query::OP_PHRASE },
-    { "path",           Xapian::Query::OP_OR },
-    { "property",       Xapian::Query::OP_AND },
-    { "query",          Xapian::Query::OP_INVALID },
-    { "subject",        Xapian::Query::OP_PHRASE },
-    { "tag",            Xapian::Query::OP_AND },
-    { "thread",         Xapian::Query::OP_OR },
-    { "to",             Xapian::Query::OP_PHRASE },
+    { "any", SEXP_FLAG_WILDCARD },
+    { "*", SEXP_FLAG_WILDCARD },
     { }
 };
 
@@ -168,6 +203,38 @@ _sexp_parse_date (notmuch_database_t *notmuch, const sexp_t *sx, Xapian::Query &
     }
 }
 
+static notmuch_status_t
+_sexp_parse_keywords (notmuch_database_t *notmuch, const char *prefix, const sexp_t *sx,
+		      _sexp_flag_t mask,
+		      _sexp_flag_t &flags, const sexp_t *&rest)
+{
+    flags = SEXP_FLAG_NONE;
+
+    for (; sx && sx->ty == SEXP_VALUE && sx->aty == SEXP_BASIC && sx->val[0] == ':'; sx = sx->next) {
+	_sexp_keyword_t *keyword;
+	for (keyword = keywords; keyword && keyword->name; keyword++) {
+	    if (strcasecmp (keyword->name, sx->val + 1) == 0) {
+		if ((mask & keyword->flag) == 0) {
+		    _notmuch_database_log_append (notmuch,
+						  "unsupported keyword '%s' for prefix '%s'\n",
+						  sx->val, prefix);
+		    return NOTMUCH_STATUS_BAD_QUERY_SYNTAX;
+		} else {
+		    flags = flags | keyword->flag;
+		    break;
+		}
+	    }
+	}
+	if (! keyword || ! keyword->name) {
+	    _notmuch_database_log (notmuch, "unknown keyword %s\n", sx->val);
+	    return NOTMUCH_STATUS_BAD_QUERY_SYNTAX;
+	}
+    }
+
+    rest = sx;
+    return NOTMUCH_STATUS_SUCCESS;
+}
+
 /* Here we expect the s-expression to be a proper list, with first
  * element defining and operation, or as a special case the empty
  * list */
@@ -206,6 +273,15 @@ _sexp_to_xapian_query (notmuch_database_t *notmuch, const sexp_t *sx, Xapian::Qu
 
     for (const _sexp_field_t *field = fields; field && field->name; field++) {
 	if (strcasecmp (field->name, hd_sexp (sx)->val) == 0) {
+	    _sexp_flag_t flags;
+	    const sexp_t *rest;
+
+	    notmuch_status_t status = _sexp_parse_keywords (notmuch, sx->list->val,
+							    sx->list->next, field->flags_allowed,
+							    flags, rest);
+	    if (status)
+		return status;
+
 	    /* some fields need to be handled specially */
 	    if (strcasecmp (field->name, "date") == 0) {
 		return _sexp_parse_date (notmuch, sx, output);
diff --git a/test/T081-sexpr-search.sh b/test/T081-sexpr-search.sh
index 89f1c36f..f4454ac2 100755
--- a/test/T081-sexpr-search.sh
+++ b/test/T081-sexpr-search.sh
@@ -259,4 +259,35 @@ unimplemented prefix foo
 EOF
 test_expect_equal_file EXPECTED OUTPUT
 
+test_begin_subtest "unknown keyword"
+notmuch search --query-syntax=sexp '(subject :foo)' >OUTPUT 2>&1
+cat <<EOF > EXPECTED
+notmuch search: Syntax error in query
+unknown keyword :foo
+EOF
+test_expect_equal_file EXPECTED OUTPUT
+
+test_begin_subtest "unsupported keyword for prefix"
+notmuch search --query-syntax=sexp '(subject :any)' >OUTPUT 2>&1
+cat <<EOF > EXPECTED
+notmuch search: Syntax error in query
+unsupported keyword ':any' for prefix 'subject'
+EOF
+test_expect_equal_file EXPECTED OUTPUT
+
+test_begin_subtest "known keyword :any"
+test_expect_success 'notmuch search --query-syntax=sexp "(tag :any)"'
+
+test_begin_subtest "known keyword :*"
+test_expect_success 'notmuch search --query-syntax=sexp "(tag :*)"'
+
+test_begin_subtest "multiple known keywords"
+test_expect_success 'notmuch search --query-syntax=sexp "(tag :any :*)"'
+
+test_begin_subtest "quoted unknown keyword"
+test_expect_success 'notmuch search --query-syntax=sexp "(subject \":foo\")"'
+
+test_begin_subtest "unknown keyword after non-keyword"
+test_expect_success 'notmuch search --query-syntax=sexp "(subject foo :foo)"'
+
 test_done
-- 
2.30.2

[PATCH 16/25] lib/parse-sexp: initial support for wildcard queries

[David Bremner]

Because of the implicit way body: queries are implemented, it is not
clear how to support (body :any) queries with current Xapian.
---
 lib/parse-sexp.cc         | 18 +++++++++++---
 test/T081-sexpr-search.sh | 50 +++++++++++++++++++++++++++++++++++++++
 2 files changed, 65 insertions(+), 3 deletions(-)

diff --git a/lib/parse-sexp.cc b/lib/parse-sexp.cc
index 14420c0e..95ee7c99 100644
--- a/lib/parse-sexp.cc
+++ b/lib/parse-sexp.cc
@@ -49,7 +49,7 @@ static _sexp_op_t operations[] =
 static _sexp_field_t fields[] =
 {
     { "attachment",   Xapian::Query::OP_PHRASE,       SEXP_FLAG_WILDCARD },
-    { "body",         Xapian::Query::OP_PHRASE,       SEXP_FLAG_WILDCARD },
+    { "body",         Xapian::Query::OP_PHRASE,       SEXP_FLAG_NONE },
     { "date",         Xapian::Query::OP_INVALID,      SEXP_FLAG_NONE },
     { "from",         Xapian::Query::OP_PHRASE,       SEXP_FLAG_NONE },
     { "folder",       Xapian::Query::OP_OR,           SEXP_FLAG_NONE },
@@ -274,6 +274,7 @@ _sexp_to_xapian_query (notmuch_database_t *notmuch, const sexp_t *sx, Xapian::Qu
     for (const _sexp_field_t *field = fields; field && field->name; field++) {
 	if (strcasecmp (field->name, hd_sexp (sx)->val) == 0) {
 	    _sexp_flag_t flags;
+	    const char *term_prefix;
 	    const sexp_t *rest;
 
 	    notmuch_status_t status = _sexp_parse_keywords (notmuch, sx->list->val,
@@ -295,8 +296,19 @@ _sexp_to_xapian_query (notmuch_database_t *notmuch, const sexp_t *sx, Xapian::Qu
 		}
 	    }
 
-	    return _sexp_combine_field (_find_prefix (field->name), field->xapian_op, sx->list->next,
-					output);
+	    term_prefix = _find_prefix (field->name);
+
+	    if (flags & SEXP_FLAG_WILDCARD) {
+		if (rest) {
+		    _notmuch_database_log (notmuch, "extra term(s) after wildcard\n", sx->val);
+		    return NOTMUCH_STATUS_BAD_QUERY_SYNTAX;
+		}
+		output = Xapian::Query (Xapian::Query::OP_WILDCARD, term_prefix);
+		return NOTMUCH_STATUS_SUCCESS;
+	    } else {
+		return _sexp_combine_field (term_prefix, field->xapian_op, rest,
+					    output);
+	    }
 	}
     }
 
diff --git a/test/T081-sexpr-search.sh b/test/T081-sexpr-search.sh
index f4454ac2..43781f44 100755
--- a/test/T081-sexpr-search.sh
+++ b/test/T081-sexpr-search.sh
@@ -290,4 +290,54 @@ test_expect_success 'notmuch search --query-syntax=sexp "(subject \":foo\")"'
 test_begin_subtest "unknown keyword after non-keyword"
 test_expect_success 'notmuch search --query-syntax=sexp "(subject foo :foo)"'
 
+test_begin_subtest "wildcard search for attachment"
+notmuch search tag:attachment > EXPECTED
+notmuch search --query-syntax=sexp '(attachment :*)' > OUTPUT
+test_expect_equal_file EXPECTED OUTPUT
+
+test_begin_subtest "wildcard search for attachment (extra terms)"
+notmuch search --query-syntax=sexp '(attachment :* trailing-garbage)' >& OUTPUT
+cat <<EOF > EXPECTED
+notmuch search: Syntax error in query
+extra term(s) after wildcard
+EOF
+test_expect_equal_file EXPECTED OUTPUT
+
+add_message '[subject]="empty body"' '[body]="."'
+notmuch tag +nobody id:${gen_msg_id}
+
+add_message '[subject]="no tags"'
+notag_mid=${gen_msg_id}
+notmuch tag -unread -inbox id:${notag_mid}
+
+test_begin_subtest "wildcard search for 'is'"
+notmuch search not id:${notag_mid} > EXPECTED
+notmuch search --query-syntax=sexp '(is :any)' > OUTPUT
+test_expect_equal_file EXPECTED OUTPUT
+
+test_begin_subtest "negated wildcard search for 'is'"
+notmuch search id:${notag_mid} > EXPECTED
+notmuch search --query-syntax=sexp '(not (is :any))' > OUTPUT
+test_expect_equal_file EXPECTED OUTPUT
+
+test_begin_subtest "wildcard search for 'tag'"
+notmuch search not id:${notag_mid} > EXPECTED
+notmuch search --query-syntax=sexp '(tag :any)' > OUTPUT
+test_expect_equal_file EXPECTED OUTPUT
+
+test_begin_subtest "negated wildcard search for 'tag'"
+notmuch search id:${notag_mid} > EXPECTED
+notmuch search --query-syntax=sexp '(not (tag :any))' > OUTPUT
+test_expect_equal_file EXPECTED OUTPUT
+
+add_message '[subject]="message with properties"'
+notmuch restore <<EOF
+#= ${gen_msg_id} foo=bar
+EOF
+
+test_begin_subtest "wildcard search for 'property'"
+notmuch search property:foo=bar > EXPECTED
+notmuch search --query-syntax=sexp '(property :any)' > OUTPUT
+test_expect_equal_file EXPECTED OUTPUT
+
 test_done
-- 
2.30.2

[PATCH 17/25] lib/query: generalize exclude handling to s-expression queries

[David Bremner]

In fact most of the code path is in common, only the caching of terms
in the query needs to be added for s-expression queries.
---
 lib/query.cc              | 34 +++++++++++++++++++++++-----------
 test/T081-sexpr-search.sh | 37 +++++++++++++++++++++++++++++++++++++
 2 files changed, 60 insertions(+), 11 deletions(-)

diff --git a/lib/query.cc b/lib/query.cc
index a3cba662..46bc1373 100644
--- a/lib/query.cc
+++ b/lib/query.cc
@@ -153,6 +153,19 @@ notmuch_query_create_sexpr (notmuch_database_t *notmuch,
     return query;
 }
 
+static void
+_notmuch_query_cache_terms (notmuch_query_t *query)
+{
+    /* Xapian doesn't support skip_to on terms from a query since
+     *  they are unordered, so cache a copy of all terms in
+     *  something searchable.
+     */
+
+    for (Xapian::TermIterator t = query->xapian_query.get_terms_begin ();
+	 t != query->xapian_query.get_terms_end (); ++t)
+	query->terms.insert (*t);
+}
+
 static notmuch_status_t
 _notmuch_query_ensure_parsed_xapian (notmuch_query_t *query)
 {
@@ -161,15 +174,7 @@ _notmuch_query_ensure_parsed_xapian (notmuch_query_t *query)
 	    query->notmuch->query_parser->
 	    parse_query (query->query_string, NOTMUCH_QUERY_PARSER_FLAGS);
 
-	/* Xapian doesn't support skip_to on terms from a query since
-	 *  they are unordered, so cache a copy of all terms in
-	 *  something searchable.
-	 */
-
-	for (Xapian::TermIterator t = query->xapian_query.get_terms_begin ();
-	     t != query->xapian_query.get_terms_end (); ++t)
-	    query->terms.insert (*t);
-
+	_notmuch_query_cache_terms (query);
 	query->parsed = true;
 
     } catch (const Xapian::Error &error) {
@@ -191,11 +196,18 @@ _notmuch_query_ensure_parsed_xapian (notmuch_query_t *query)
 static notmuch_status_t
 _notmuch_query_ensure_parsed_sexpr (notmuch_query_t *query)
 {
+    notmuch_status_t status;
+
     if (query->parsed)
 	return NOTMUCH_STATUS_SUCCESS;
 
-    return _notmuch_sexp_string_to_xapian_query (query->notmuch, query->query_string,
-						 query->xapian_query);
+    status = _notmuch_sexp_string_to_xapian_query (query->notmuch, query->query_string,
+						   query->xapian_query);
+    if (status)
+	return status;
+
+    _notmuch_query_cache_terms (query);
+    return NOTMUCH_STATUS_SUCCESS;
 }
 
 static notmuch_status_t
diff --git a/test/T081-sexpr-search.sh b/test/T081-sexpr-search.sh
index 43781f44..04eba2c0 100755
--- a/test/T081-sexpr-search.sh
+++ b/test/T081-sexpr-search.sh
@@ -340,4 +340,41 @@ notmuch search property:foo=bar > EXPECTED
 notmuch search --query-syntax=sexp '(property :any)' > OUTPUT
 test_expect_equal_file EXPECTED OUTPUT
 
+test_begin_subtest "Search, exclude \"deleted\" messages from search"
+notmuch config set search.exclude_tags deleted
+generate_message '[subject]="Not deleted"'
+not_deleted_id=$gen_msg_id
+generate_message '[subject]="Deleted"'
+notmuch new > /dev/null
+notmuch tag +deleted id:$gen_msg_id
+deleted_id=$gen_msg_id
+output=$(notmuch search --query-syntax=sexp '(subject deleted)' | notmuch_search_sanitize)
+test_expect_equal "$output" "thread:XXX   2001-01-05 [1/1] Notmuch Test Suite; Not deleted (inbox unread)"
+
+test_begin_subtest "Search, exclude \"deleted\" messages from message search --exclude=false"
+output=$(notmuch search --query-syntax=sexp --exclude=false --output=messages '(subject deleted)' | notmuch_search_sanitize)
+test_expect_equal "$output" "id:$not_deleted_id
+id:$deleted_id"
+
+test_begin_subtest "Search, exclude \"deleted\" messages from search, overridden"
+output=$(notmuch search --query-syntax=sexp '(and (subject deleted) (tag deleted))' | notmuch_search_sanitize)
+test_expect_equal "$output" "thread:XXX   2001-01-05 [1/1] Notmuch Test Suite; Deleted (deleted inbox unread)"
+
+test_begin_subtest "Search, exclude \"deleted\" messages from threads"
+add_message '[subject]="Not deleted reply"' '[in-reply-to]="<$gen_msg_id>"'
+output=$(notmuch search --query-syntax=sexp '(subject deleted)' | notmuch_search_sanitize)
+test_expect_equal "$output" "thread:XXX   2001-01-05 [1/1] Notmuch Test Suite; Not deleted (inbox unread)
+thread:XXX   2001-01-05 [1/2] Notmuch Test Suite; Not deleted reply (deleted inbox unread)"
+
+test_begin_subtest "Search, don't exclude \"deleted\" messages when --exclude=flag specified"
+output=$(notmuch search --query-syntax=sexp --exclude=flag '(subject deleted)' | notmuch_search_sanitize)
+test_expect_equal "$output" "thread:XXX   2001-01-05 [1/1] Notmuch Test Suite; Not deleted (inbox unread)
+thread:XXX   2001-01-05 [1/2] Notmuch Test Suite; Deleted (deleted inbox unread)"
+
+test_begin_subtest "Search, don't exclude \"deleted\" messages from search if not configured"
+notmuch config set search.exclude_tags
+output=$(notmuch search --query-syntax=sexp '(subject deleted)' | notmuch_search_sanitize)
+test_expect_equal "$output" "thread:XXX   2001-01-05 [1/1] Notmuch Test Suite; Not deleted (inbox unread)
+thread:XXX   2001-01-05 [2/2] Notmuch Test Suite; Deleted (deleted inbox unread)"
+
 test_done
-- 
2.30.2

[PATCH 18/25] lib: factor out query construction from regexp

[David Bremner]

This will allow re-use of this code outside of the Xapian query parser.
---
 lib/regexp-fields.cc | 81 +++++++++++++++++++++++++++++++-------------
 lib/regexp-fields.h  |  6 ++++
 2 files changed, 63 insertions(+), 24 deletions(-)

diff --git a/lib/regexp-fields.cc b/lib/regexp-fields.cc
index 0feb50e5..c6d9d94f 100644
--- a/lib/regexp-fields.cc
+++ b/lib/regexp-fields.cc
@@ -26,27 +26,32 @@
 #include "notmuch-private.h"
 #include "database-private.h"
 
-static void
-compile_regex (regex_t &regexp, const char *str)
+notmuch_status_t
+compile_regex (regex_t &regexp, const char *str, std::string &msg)
 {
     int err = regcomp (&regexp, str, REG_EXTENDED | REG_NOSUB);
 
     if (err != 0) {
 	size_t len = regerror (err, &regexp, NULL, 0);
 	char *buffer = new char[len];
-	std::string msg = "Regexp error: ";
+	msg = "Regexp error: ";
 	(void) regerror (err, &regexp, buffer, len);
 	msg.append (buffer, len);
 	delete[] buffer;
 
-	throw Xapian::QueryParserError (msg);
+	return NOTMUCH_STATUS_ILLEGAL_ARGUMENT;
     }
+    return NOTMUCH_STATUS_SUCCESS;
 }
 
 RegexpPostingSource::RegexpPostingSource (Xapian::valueno slot, const std::string &regexp)
     : slot_ (slot)
 {
-    compile_regex (regexp_, regexp.c_str ());
+    std::string msg;
+    notmuch_status_t status = compile_regex (regexp_, regexp.c_str (), msg);
+
+    if (status)
+	throw Xapian::QueryParserError (msg);
 }
 
 RegexpPostingSource::~RegexpPostingSource ()
@@ -141,18 +146,54 @@ _find_slot (std::string prefix)
 	return Xapian::BAD_VALUENO;
 }
 
-RegexpFieldProcessor::RegexpFieldProcessor (std::string prefix,
+RegexpFieldProcessor::RegexpFieldProcessor (std::string field_,
 					    notmuch_field_flag_t options_,
 					    Xapian::QueryParser &parser_,
 					    notmuch_database_t *notmuch_)
-    : slot (_find_slot (prefix)),
-    term_prefix (_find_prefix (prefix.c_str ())),
+    : slot (_find_slot (field_)),
+    field (field_),
+    term_prefix (_find_prefix (field_.c_str ())),
     options (options_),
     parser (parser_),
     notmuch (notmuch_)
 {
 };
 
+notmuch_status_t
+_notmuch_regexp_to_query (notmuch_database_t *notmuch, Xapian::valueno slot, std::string field,
+			  std::string regexp_str,
+			  Xapian::Query &output, std::string &msg)
+{
+    regex_t regexp;
+    notmuch_status_t status;
+
+    status = compile_regex (regexp, regexp_str.c_str (), msg);
+    if (status) {
+	_notmuch_database_log_append (notmuch, "error compiling regex %s", msg.c_str ());
+	return status;
+    }
+
+    if (slot == Xapian::BAD_VALUENO)
+	slot = _find_slot (field);
+
+    if (slot == Xapian::BAD_VALUENO) {
+	std::string term_prefix = _find_prefix (field.c_str ());
+	std::vector<std::string> terms;
+
+	for (Xapian::TermIterator it = notmuch->xapian_db->allterms_begin (term_prefix);
+	     it != notmuch->xapian_db->allterms_end (); ++it) {
+	    if (regexec (&regexp, (*it).c_str () + term_prefix.size (),
+			 0, NULL, 0) == 0)
+		terms.push_back (*it);
+	}
+	output = Xapian::Query (Xapian::Query::OP_OR, terms.begin (), terms.end ());
+    } else {
+	RegexpPostingSource *postings = new RegexpPostingSource (slot, regexp_str);
+	output = Xapian::Query (postings->release ());
+    }
+    return NOTMUCH_STATUS_SUCCESS;
+}
+
 Xapian::Query
 RegexpFieldProcessor::operator() (const std::string & str)
 {
@@ -168,23 +209,15 @@ RegexpFieldProcessor::operator() (const std::string & str)
 
     if (str.at (0) == '/') {
 	if (str.length () > 1 && str.at (str.size () - 1) == '/') {
+	    Xapian::Query query;
 	    std::string regexp_str = str.substr (1, str.size () - 2);
-	    if (slot != Xapian::BAD_VALUENO) {
-		RegexpPostingSource *postings = new RegexpPostingSource (slot, regexp_str);
-		return Xapian::Query (postings->release ());
-	    } else {
-		std::vector<std::string> terms;
-		regex_t regexp;
-
-		compile_regex (regexp, regexp_str.c_str ());
-		for (Xapian::TermIterator it = notmuch->xapian_db->allterms_begin (term_prefix);
-		     it != notmuch->xapian_db->allterms_end (); ++it) {
-		    if (regexec (&regexp, (*it).c_str () + term_prefix.size (),
-				 0, NULL, 0) == 0)
-			terms.push_back (*it);
-		}
-		return Xapian::Query (Xapian::Query::OP_OR, terms.begin (), terms.end ());
-	    }
+	    std::string msg;
+	    notmuch_status_t status;
+
+	    status = _notmuch_regexp_to_query (notmuch, slot, field, regexp_str, query, msg);
+	    if (status)
+		throw Xapian::QueryParserError (msg);
+	    return query;
 	} else {
 	    throw Xapian::QueryParserError ("unmatched regex delimiter in '" + str + "'");
 	}
diff --git a/lib/regexp-fields.h b/lib/regexp-fields.h
index a8cca243..9c871de7 100644
--- a/lib/regexp-fields.h
+++ b/lib/regexp-fields.h
@@ -30,6 +30,11 @@
 #include "database-private.h"
 #include "notmuch-private.h"
 
+notmuch_status_t
+_notmuch_regex_to_query (notmuch_database_t *notmuch, Xapian::valueno slot, std::string field,
+			 std::string regexp_str,
+			 Xapian::Query &output, std::string &msg);
+
 /* A posting source that returns documents where a value matches a
  * regexp.
  */
@@ -64,6 +69,7 @@ public:
 class RegexpFieldProcessor : public Xapian::FieldProcessor {
 protected:
     Xapian::valueno slot;
+    std::string field;
     std::string term_prefix;
     notmuch_field_flag_t options;
     Xapian::QueryParser &parser;
-- 
2.30.2

[PATCH 19/25] lib/parse-sexp: add support for regexp fields

[David Bremner]

This commit connects the previously added keyword / flag handling with
the previously refactored regexp to query refactoring.
---
 lib/parse-sexp.cc         | 35 +++++++++++++++--------
 lib/regexp-fields.h       |  8 +++---
 test/T081-sexpr-search.sh | 59 +++++++++++++++++++++++++++++++++++++++
 3 files changed, 86 insertions(+), 16 deletions(-)

diff --git a/lib/parse-sexp.cc b/lib/parse-sexp.cc
index 95ee7c99..c0d0b596 100644
--- a/lib/parse-sexp.cc
+++ b/lib/parse-sexp.cc
@@ -1,8 +1,8 @@
-#include <xapian.h>
-#include "notmuch-private.h"
+#include "database-private.h"
 #include "sexp.h"
 #include "parse-time-vrp.h"
 #include "query-fp.h"
+#include "regexp-fields.h"
 
 typedef struct  {
     const char *name;
@@ -13,6 +13,7 @@ typedef struct  {
 typedef enum {
     SEXP_FLAG_NONE	= 0,
     SEXP_FLAG_WILDCARD	= 1 << 0,
+    SEXP_FLAG_REGEXP	= 1 << 1,
 } _sexp_flag_t;
 
 /*
@@ -51,18 +52,18 @@ static _sexp_field_t fields[] =
     { "attachment",   Xapian::Query::OP_PHRASE,       SEXP_FLAG_WILDCARD },
     { "body",         Xapian::Query::OP_PHRASE,       SEXP_FLAG_NONE },
     { "date",         Xapian::Query::OP_INVALID,      SEXP_FLAG_NONE },
-    { "from",         Xapian::Query::OP_PHRASE,       SEXP_FLAG_NONE },
-    { "folder",       Xapian::Query::OP_OR,           SEXP_FLAG_NONE },
-    { "id",           Xapian::Query::OP_OR,           SEXP_FLAG_NONE },
-    { "is",           Xapian::Query::OP_AND,          SEXP_FLAG_WILDCARD },
-    { "mid",          Xapian::Query::OP_OR,           SEXP_FLAG_NONE },
+    { "from",         Xapian::Query::OP_PHRASE,       SEXP_FLAG_REGEXP },
+    { "folder",       Xapian::Query::OP_OR,           SEXP_FLAG_REGEXP },
+    { "id",           Xapian::Query::OP_OR,           SEXP_FLAG_REGEXP },
+    { "is",           Xapian::Query::OP_AND,          SEXP_FLAG_WILDCARD | SEXP_FLAG_REGEXP },
+    { "mid",          Xapian::Query::OP_OR,           SEXP_FLAG_REGEXP },
     { "mimetype",     Xapian::Query::OP_PHRASE,       SEXP_FLAG_NONE },
-    { "path",         Xapian::Query::OP_OR,           SEXP_FLAG_NONE },
-    { "property",     Xapian::Query::OP_AND,          SEXP_FLAG_WILDCARD },
+    { "path",         Xapian::Query::OP_OR,           SEXP_FLAG_REGEXP },
+    { "property",     Xapian::Query::OP_AND,          SEXP_FLAG_WILDCARD | SEXP_FLAG_REGEXP },
     { "query",        Xapian::Query::OP_INVALID,      SEXP_FLAG_NONE },
-    { "subject",      Xapian::Query::OP_PHRASE,       SEXP_FLAG_NONE },
-    { "tag",          Xapian::Query::OP_AND,          SEXP_FLAG_WILDCARD },
-    { "thread",       Xapian::Query::OP_OR,           SEXP_FLAG_NONE },
+    { "subject",      Xapian::Query::OP_PHRASE,       SEXP_FLAG_REGEXP },
+    { "tag",          Xapian::Query::OP_AND,          SEXP_FLAG_WILDCARD | SEXP_FLAG_REGEXP },
+    { "thread",       Xapian::Query::OP_OR,           SEXP_FLAG_REGEXP },
     { "to",           Xapian::Query::OP_PHRASE,       SEXP_FLAG_NONE },
     { }
 };
@@ -76,6 +77,7 @@ static _sexp_keyword_t keywords[] =
 {
     { "any", SEXP_FLAG_WILDCARD },
     { "*", SEXP_FLAG_WILDCARD },
+    { "rx", SEXP_FLAG_REGEXP },
     { }
 };
 
@@ -305,6 +307,15 @@ _sexp_to_xapian_query (notmuch_database_t *notmuch, const sexp_t *sx, Xapian::Qu
 		}
 		output = Xapian::Query (Xapian::Query::OP_WILDCARD, term_prefix);
 		return NOTMUCH_STATUS_SUCCESS;
+	    } else if (flags & SEXP_FLAG_REGEXP) {
+		if (! rest || ! rest->val) {
+		    _notmuch_database_log (notmuch, "missing regular expression\n");
+		    return NOTMUCH_STATUS_BAD_QUERY_SYNTAX;
+		} else {
+		    std::string msg; /* ignored */
+		    return _notmuch_regexp_to_query (notmuch, Xapian::BAD_VALUENO, field->name,
+						     rest->val, output, msg);
+		}
 	    } else {
 		return _sexp_combine_field (term_prefix, field->xapian_op, rest,
 					    output);
diff --git a/lib/regexp-fields.h b/lib/regexp-fields.h
index 9c871de7..aa8fd81c 100644
--- a/lib/regexp-fields.h
+++ b/lib/regexp-fields.h
@@ -27,13 +27,13 @@
 
 #include <sys/types.h>
 #include <regex.h>
-#include "database-private.h"
 #include "notmuch-private.h"
+#include "database-private.h"
 
 notmuch_status_t
-_notmuch_regex_to_query (notmuch_database_t *notmuch, Xapian::valueno slot, std::string field,
-			 std::string regexp_str,
-			 Xapian::Query &output, std::string &msg);
+_notmuch_regexp_to_query (notmuch_database_t *notmuch, Xapian::valueno slot, std::string field,
+			  std::string regexp_str,
+			  Xapian::Query &output, std::string &msg);
 
 /* A posting source that returns documents where a value matches a
  * regexp.
diff --git a/test/T081-sexpr-search.sh b/test/T081-sexpr-search.sh
index 04eba2c0..281a9bf7 100755
--- a/test/T081-sexpr-search.sh
+++ b/test/T081-sexpr-search.sh
@@ -377,4 +377,63 @@ output=$(notmuch search --query-syntax=sexp '(subject deleted)' | notmuch_search
 test_expect_equal "$output" "thread:XXX   2001-01-05 [1/1] Notmuch Test Suite; Not deleted (inbox unread)
 thread:XXX   2001-01-05 [2/2] Notmuch Test Suite; Deleted (deleted inbox unread)"
 
+test_begin_subtest "illegal regexp search"
+test_expect_code 1 "notmuch search --query-syntax=sexp '(body :rx foo)'"
+
+notmuch search --output=messages from:cworth > cworth.msg-ids
+
+test_begin_subtest "regexp 'folder' search"
+notmuch search 'folder:/^bar$/' | notmuch_search_sanitize > EXPECTED
+notmuch search --query-syntax=sexp '(folder :rx ^bar$)' | notmuch_search_sanitize > OUTPUT
+test_expect_equal_file EXPECTED OUTPUT
+
+test_begin_subtest "regexp from search"
+notmuch search --output=messages --query-syntax=sexp '(from :rx cworth)' > OUTPUT
+test_expect_equal_file cworth.msg-ids OUTPUT
+
+test_begin_subtest "regexp search for 'from' 2"
+notmuch search from:/cworth@cworth.org/ and subject:patch | notmuch_search_sanitize > EXPECTED
+notmuch search --query-syntax=sexp '(and (from :rx cworth@cworth.org) (subject patch))' \
+    | notmuch_search_sanitize > OUTPUT
+test_expect_equal_file EXPECTED OUTPUT
+
+test_begin_subtest "regexp 'id' search"
+notmuch search --output=messages --query-syntax=sexp '(id :rx yoom)' > OUTPUT
+test_expect_equal_file cworth.msg-ids OUTPUT
+
+test_begin_subtest "unanchored 'is' search"
+notmuch search tag:signed or tag:inbox > EXPECTED
+notmuch search --query-syntax=sexp '(is :rx i)' > OUTPUT
+test_expect_equal_file EXPECTED OUTPUT
+
+test_begin_subtest "anchored 'is' search"
+notmuch search tag:signed > EXPECTED
+notmuch search --query-syntax=sexp '(is :rx ^si)' > OUTPUT
+test_expect_equal_file EXPECTED OUTPUT
+
+test_begin_subtest "combine regexp mid and subject"
+notmuch search subject:/-C/ and mid:/y..m/ | notmuch_search_sanitize > EXPECTED
+notmuch search --query-syntax=sexp '(and (subject :rx -C) (mid :rx y..m))' | notmuch_search_sanitize > OUTPUT
+test_expect_equal_file EXPECTED OUTPUT
+
+test_begin_subtest "regexp 'path' search"
+notmuch search 'path:/^bar$/' | notmuch_search_sanitize > EXPECTED
+notmuch search --query-syntax=sexp '(path :rx ^bar$)' | notmuch_search_sanitize > OUTPUT
+test_expect_equal_file EXPECTED OUTPUT
+
+test_begin_subtest "regexp 'property' search"
+notmuch search property:foo=bar > EXPECTED
+notmuch search --query-syntax=sexp '(property :rx foo=.*)' > OUTPUT
+test_expect_equal_file EXPECTED OUTPUT
+
+test_begin_subtest "anchored 'tag' search"
+notmuch search tag:signed > EXPECTED
+notmuch search --query-syntax=sexp '(tag :rx ^si)' > OUTPUT
+test_expect_equal_file EXPECTED OUTPUT
+
+test_begin_subtest "regexp 'thread' search"
+notmuch search --output=threads '*' | grep '7$' > EXPECTED
+notmuch search --output=threads --query-syntax=sexp '(thread :rx 7$)' > OUTPUT
+test_expect_equal_file EXPECTED OUTPUT
+
 test_done
-- 
2.30.2

[PATCH 20/25] lib/thread-fp: factor out query expansion

[David Bremner]

The non-obvious choice of creating the query outside the factored-out
code is so the the new function can be used with different query
syntaxes.
---
 lib/database-private.h |  5 +++++
 lib/query.cc           | 26 ++++++++++++++++++++++++++
 lib/thread-fp.cc       | 25 +++++++++----------------
 3 files changed, 40 insertions(+), 16 deletions(-)

diff --git a/lib/database-private.h b/lib/database-private.h
index 9706c17e..72b74807 100644
--- a/lib/database-private.h
+++ b/lib/database-private.h
@@ -300,4 +300,9 @@ _notmuch_database_setup_standard_query_fields (notmuch_database_t *notmuch);
 notmuch_status_t
 _notmuch_database_setup_user_query_fields (notmuch_database_t *notmuch);
 
+/* query.cc */
+notmuch_status_t
+_notmuch_query_expand_to_threads (notmuch_query_t *subquery,
+				  Xapian::Query &output, std::string &msg);
+
 #endif
diff --git a/lib/query.cc b/lib/query.cc
index 46bc1373..0916860e 100644
--- a/lib/query.cc
+++ b/lib/query.cc
@@ -796,3 +796,29 @@ notmuch_query_get_database (const notmuch_query_t *query)
 {
     return query->notmuch;
 }
+
+notmuch_status_t
+_notmuch_query_expand_to_threads (notmuch_query_t *subquery, Xapian::Query &output, std::string &msg)
+{
+    notmuch_messages_t *messages;
+    std::set<std::string> terms;
+    notmuch_status_t status;
+    const std::string thread_prefix =  _find_prefix ("thread");
+
+    status = notmuch_query_search_messages (subquery, &messages);
+    if (status) {
+	msg = "failed to search messages for subquery";
+	return NOTMUCH_STATUS_ILLEGAL_ARGUMENT;
+    }
+
+    for (; notmuch_messages_valid (messages); notmuch_messages_move_to_next (messages)) {
+	std::string term = thread_prefix;
+	notmuch_message_t *message;
+	message = notmuch_messages_get (messages);
+	term += _notmuch_message_get_thread_id_only (message);
+	terms.insert (term);
+    }
+    output = Xapian::Query (Xapian::Query::OP_OR, terms.begin (), terms.end ());
+
+    return NOTMUCH_STATUS_SUCCESS;
+}
diff --git a/lib/thread-fp.cc b/lib/thread-fp.cc
index 06708ef2..d0680a22 100644
--- a/lib/thread-fp.cc
+++ b/lib/thread-fp.cc
@@ -34,28 +34,21 @@ ThreadFieldProcessor::operator() (const std::string & str)
 	if (str.size () <= 1 || str.at (str.size () - 1) != '}') {
 	    throw Xapian::QueryParserError ("missing } in '" + str + "'");
 	} else {
+	    notmuch_query_t *subquery;
+	    Xapian::Query query;
+	    std::string msg;
 	    std::string subquery_str = str.substr (1, str.size () - 2);
-	    notmuch_query_t *subquery = notmuch_query_create (notmuch, subquery_str.c_str ());
-	    notmuch_messages_t *messages;
-	    std::set<std::string> terms;
 
+	    subquery = notmuch_query_create (notmuch, subquery_str.c_str ());
 	    if (! subquery)
-		throw Xapian::QueryParserError ("failed to create subquery for '" + subquery_str +
-						"'");
+		throw Xapian::QueryParserError ("failed to create subquery for '" +
+						subquery_str + "'");
 
-	    status = notmuch_query_search_messages (subquery, &messages);
+	    status = _notmuch_query_expand_to_threads (subquery, query, msg);
 	    if (status)
-		throw Xapian::QueryParserError ("failed to search messages for '" + subquery_str +
-						"'");
+		throw Xapian::QueryParserError (msg);
 
-	    for (; notmuch_messages_valid (messages); notmuch_messages_move_to_next (messages)) {
-		std::string term = thread_prefix;
-		notmuch_message_t *message;
-		message = notmuch_messages_get (messages);
-		term += _notmuch_message_get_thread_id_only (message);
-		terms.insert (term);
-	    }
-	    return Xapian::Query (Xapian::Query::OP_OR, terms.begin (), terms.end ());
+	    return query;
 	}
     } else {
 	/* literal thread id */
-- 
2.30.2

[PATCH 21/25] lib: define _notmuch_query_from_sexp

[David Bremner]

This will be used in a following commit to provide the equivalant of
thread:{} syntax for s-expression queries.
---
 lib/parse-sexp.cc | 13 +++++--------
 lib/parse-sexp.h  | 14 ++++++++++++--
 lib/query.cc      | 23 ++++++++++++++++++++++-
 3 files changed, 39 insertions(+), 11 deletions(-)

diff --git a/lib/parse-sexp.cc b/lib/parse-sexp.cc
index c0d0b596..cfd503d9 100644
--- a/lib/parse-sexp.cc
+++ b/lib/parse-sexp.cc
@@ -1,5 +1,5 @@
 #include "database-private.h"
-#include "sexp.h"
+#include "parse-sexp.h"
 #include "parse-time-vrp.h"
 #include "query-fp.h"
 #include "regexp-fields.h"
@@ -81,9 +81,6 @@ static _sexp_keyword_t keywords[] =
     { }
 };
 
-static notmuch_status_t _sexp_to_xapian_query (notmuch_database_t *notmuch, const sexp_t *sx,
-					       Xapian::Query &output);
-
 static notmuch_status_t
 _sexp_combine_query (notmuch_database_t *notmuch,
 		     Xapian::Query::op operation,
@@ -102,7 +99,7 @@ _sexp_combine_query (notmuch_database_t *notmuch,
 	return NOTMUCH_STATUS_SUCCESS;
     }
 
-    status = _sexp_to_xapian_query (notmuch, sx, subquery);
+    status = _notmuch_sexp_to_xapian_query (notmuch, sx, subquery);
     if (status)
 	return status;
 
@@ -123,7 +120,7 @@ _notmuch_sexp_string_to_xapian_query (notmuch_database_t *notmuch, const char *q
     if (! sx)
 	return NOTMUCH_STATUS_BAD_QUERY_SYNTAX;
 
-    return _sexp_to_xapian_query (notmuch, sx, output);
+    return _notmuch_sexp_to_xapian_query (notmuch, sx, output);
 }
 
 static void
@@ -241,8 +238,8 @@ _sexp_parse_keywords (notmuch_database_t *notmuch, const char *prefix, const sex
  * element defining and operation, or as a special case the empty
  * list */
 
-static notmuch_status_t
-_sexp_to_xapian_query (notmuch_database_t *notmuch, const sexp_t *sx, Xapian::Query &output)
+notmuch_status_t
+_notmuch_sexp_to_xapian_query (notmuch_database_t *notmuch, const sexp_t *sx, Xapian::Query &output)
 {
 
     const _sexp_op_t *op;
diff --git a/lib/parse-sexp.h b/lib/parse-sexp.h
index a358bf26..eb78cbda 100644
--- a/lib/parse-sexp.h
+++ b/lib/parse-sexp.h
@@ -1,6 +1,16 @@
 #ifndef _PARSE_SEXP_H
 #define _PARSE_SEXP_H
 /* parse_sexp.cc */
-notmuch_status_t _notmuch_sexp_string_to_xapian_query (notmuch_database_t *notmuch, const
-						       char *querystr, Xapian::Query &output);
+#include <sexp.h>
+
+notmuch_status_t _notmuch_sexp_string_to_xapian_query (notmuch_database_t *notmuch,
+						       const char *querystr,
+						       Xapian::Query &output);
+
+notmuch_status_t _notmuch_sexp_to_xapian_query (notmuch_database_t *notmuch, const sexp_t *sx,
+						Xapian::Query &output);
+
+notmuch_status_t _notmuch_query_from_sexpr (notmuch_database_t *notmuch, const sexp_t *sexp,
+					    notmuch_query_t *&output);
+
 #endif
diff --git a/lib/query.cc b/lib/query.cc
index 0916860e..4ada3fe0 100644
--- a/lib/query.cc
+++ b/lib/query.cc
@@ -112,7 +112,10 @@ _notmuch_query_constructor (notmuch_database_t *notmuch,
 
     query->notmuch = notmuch;
 
-    query->query_string = talloc_strdup (query, query_string);
+    if (query_string)
+	query->query_string = talloc_strdup (query, query_string);
+    else
+	query->query_string = NULL;
 
     query->sort = NOTMUCH_SORT_NEWEST_FIRST;
 
@@ -193,6 +196,24 @@ _notmuch_query_ensure_parsed_xapian (notmuch_query_t *query)
     return NOTMUCH_STATUS_SUCCESS;
 }
 
+notmuch_status_t
+_notmuch_query_from_sexpr (notmuch_database_t *notmuch, const sexp_t *sexp, notmuch_query_t *&output)
+{
+    notmuch_status_t status;
+    notmuch_query_t *query = _notmuch_query_constructor (notmuch, NULL);
+
+    status = _notmuch_sexp_to_xapian_query (notmuch, sexp, query->xapian_query);
+    if (status)
+	return status;
+
+    query->syntax = NOTMUCH_QUERY_SYNTAX_SEXPR;
+    query->parsed = true;
+
+    output = query;
+
+    return NOTMUCH_STATUS_SUCCESS;
+}
+
 static notmuch_status_t
 _notmuch_query_ensure_parsed_sexpr (notmuch_query_t *query)
 {
-- 
2.30.2

[PATCH 22/25] lib: generate actual Xapian query for "*" and ""

[David Bremner]

The previous code had the somewhat bizarre effect that the (notmuch
specific) query string was "*" (interpreted as MatchAll) and the
allegedly parsed xapian_query was "MatchNothing".

This commit also reduces code duplication.
---
 lib/query.cc | 34 ++++++++++++++--------------------
 1 file changed, 14 insertions(+), 20 deletions(-)

diff --git a/lib/query.cc b/lib/query.cc
index 4ada3fe0..daaffb9f 100644
--- a/lib/query.cc
+++ b/lib/query.cc
@@ -173,11 +173,16 @@ static notmuch_status_t
 _notmuch_query_ensure_parsed_xapian (notmuch_query_t *query)
 {
     try {
-	query->xapian_query =
-	    query->notmuch->query_parser->
-	    parse_query (query->query_string, NOTMUCH_QUERY_PARSER_FLAGS);
+	if (strcmp (query->query_string, "") == 0 ||
+	    strcmp (query->query_string, "*") == 0) {
+	    query->xapian_query = Xapian::Query::MatchAll;
+	} else {
+	    query->xapian_query =
+		query->notmuch->query_parser->
+		parse_query (query->query_string, NOTMUCH_QUERY_PARSER_FLAGS);
 
-	_notmuch_query_cache_terms (query);
+	    _notmuch_query_cache_terms (query);
+	}
 	query->parsed = true;
 
     } catch (const Xapian::Error &error) {
@@ -338,7 +343,6 @@ _notmuch_query_search_documents (notmuch_query_t *query,
 				 notmuch_messages_t **out)
 {
     notmuch_database_t *notmuch = query->notmuch;
-    const char *query_string = query->query_string;
     notmuch_mset_messages_t *messages;
     notmuch_status_t status;
 
@@ -368,13 +372,9 @@ _notmuch_query_search_documents (notmuch_query_t *query,
 	Xapian::MSet mset;
 	Xapian::MSetIterator iterator;
 
-	if (strcmp (query_string, "") == 0 ||
-	    strcmp (query_string, "*") == 0) {
-	    final_query = mail_query;
-	} else {
-	    final_query = Xapian::Query (Xapian::Query::OP_AND,
-					 mail_query, query->xapian_query);
-	}
+	final_query = Xapian::Query (Xapian::Query::OP_AND,
+				     mail_query, query->xapian_query);
+
 	messages->base.excluded_doc_ids = NULL;
 
 	if ((query->omit_excluded != NOTMUCH_EXCLUDE_FALSE) && (query->exclude_terms)) {
@@ -695,7 +695,6 @@ notmuch_status_t
 _notmuch_query_count_documents (notmuch_query_t *query, const char *type, unsigned *count_out)
 {
     notmuch_database_t *notmuch = query->notmuch;
-    const char *query_string = query->query_string;
     Xapian::doccount count = 0;
     notmuch_status_t status;
 
@@ -711,13 +710,8 @@ _notmuch_query_count_documents (notmuch_query_t *query, const char *type, unsign
 	Xapian::Query final_query, exclude_query;
 	Xapian::MSet mset;
 
-	if (strcmp (query_string, "") == 0 ||
-	    strcmp (query_string, "*") == 0) {
-	    final_query = mail_query;
-	} else {
-	    final_query = Xapian::Query (Xapian::Query::OP_AND,
-					 mail_query, query->xapian_query);
-	}
+	final_query = Xapian::Query (Xapian::Query::OP_AND,
+				     mail_query, query->xapian_query);
 
 	exclude_query = _notmuch_exclude_tags (query);
 
-- 
2.30.2

[PATCH 23/25] lib/parse-sexp: support thread subqueries.

[David Bremner]

This essentially mimics the ThreadFieldProcessor code, but generating
the notmuch_query_t via the new notmuch_query_from_sexp using a
subtree of the already parsed s-expression.
---
 lib/parse-sexp.cc         | 27 +++++++++++++++++++++++----
 test/T081-sexpr-search.sh | 35 +++++++++++++++++++++++++++++++++++
 2 files changed, 58 insertions(+), 4 deletions(-)

diff --git a/lib/parse-sexp.cc b/lib/parse-sexp.cc
index cfd503d9..df3629af 100644
--- a/lib/parse-sexp.cc
+++ b/lib/parse-sexp.cc
@@ -11,9 +11,10 @@ typedef struct  {
 } _sexp_op_t;
 
 typedef enum {
-    SEXP_FLAG_NONE	= 0,
-    SEXP_FLAG_WILDCARD	= 1 << 0,
-    SEXP_FLAG_REGEXP	= 1 << 1,
+    SEXP_FLAG_NONE		= 0,
+    SEXP_FLAG_WILDCARD		= 1 << 0,
+    SEXP_FLAG_REGEXP		= 1 << 1,
+    SEXP_FLAG_THREAD_MATCH	= 1 << 2,
 } _sexp_flag_t;
 
 /*
@@ -63,7 +64,7 @@ static _sexp_field_t fields[] =
     { "query",        Xapian::Query::OP_INVALID,      SEXP_FLAG_NONE },
     { "subject",      Xapian::Query::OP_PHRASE,       SEXP_FLAG_REGEXP },
     { "tag",          Xapian::Query::OP_AND,          SEXP_FLAG_WILDCARD | SEXP_FLAG_REGEXP },
-    { "thread",       Xapian::Query::OP_OR,           SEXP_FLAG_REGEXP },
+    { "thread",       Xapian::Query::OP_OR,           SEXP_FLAG_REGEXP | SEXP_FLAG_THREAD_MATCH },
     { "to",           Xapian::Query::OP_PHRASE,       SEXP_FLAG_NONE },
     { }
 };
@@ -78,6 +79,7 @@ static _sexp_keyword_t keywords[] =
     { "any", SEXP_FLAG_WILDCARD },
     { "*", SEXP_FLAG_WILDCARD },
     { "rx", SEXP_FLAG_REGEXP },
+    { "match", SEXP_FLAG_THREAD_MATCH },
     { }
 };
 
@@ -313,6 +315,23 @@ _notmuch_sexp_to_xapian_query (notmuch_database_t *notmuch, const sexp_t *sx, Xa
 		    return _notmuch_regexp_to_query (notmuch, Xapian::BAD_VALUENO, field->name,
 						     rest->val, output, msg);
 		}
+	    } else if (flags & SEXP_FLAG_THREAD_MATCH) {
+		if (! rest) {
+		    _notmuch_database_log (notmuch, "missing subquery\n");
+		    return NOTMUCH_STATUS_BAD_QUERY_SYNTAX;
+		} else {
+		    notmuch_query_t *query;
+		    std::string msg;
+
+		    status = _notmuch_query_from_sexpr (notmuch, rest, query);
+		    if (status)
+			return status;
+		    status = _notmuch_query_expand_to_threads (query, output, msg);
+		    if (status) {
+			_notmuch_database_log (notmuch, "error expanding query %s\n", msg.c_str ());
+		    }
+		    return status;
+		}
 	    } else {
 		return _sexp_combine_field (term_prefix, field->xapian_op, rest,
 					    output);
diff --git a/test/T081-sexpr-search.sh b/test/T081-sexpr-search.sh
index 281a9bf7..f0a7efb0 100755
--- a/test/T081-sexpr-search.sh
+++ b/test/T081-sexpr-search.sh
@@ -436,4 +436,39 @@ notmuch search --output=threads '*' | grep '7$' > EXPECTED
 notmuch search --output=threads --query-syntax=sexp '(thread :rx 7$)' > OUTPUT
 test_expect_equal_file EXPECTED OUTPUT
 
+test_begin_subtest "Basic query that matches no messages"
+count=$(notmuch count from:keithp and to:keithp)
+test_expect_equal 0 "$count"
+
+test_begin_subtest "Same query against threads"
+notmuch search --query-syntax=sexp '(and (thread :match (from keithp)) (thread :match (to keithp)))' \
+    | notmuch_search_sanitize > OUTPUT
+cat<<EOF > EXPECTED
+thread:XXX   2009-11-18 [7/7] Lars Kellogg-Stedman, Mikhail Gusarov, Keith Packard, Carl Worth; [notmuch] Working with Maildir storage? (inbox signed unread)
+EOF
+test_expect_equal_file EXPECTED OUTPUT
+
+test_begin_subtest "Mix thread and non-threads query"
+notmuch search --query-syntax=sexp '(and (thread :match keithp) (to keithp))' | notmuch_search_sanitize > OUTPUT
+cat<<EOF > EXPECTED
+thread:XXX   2009-11-18 [1/7] Lars Kellogg-Stedman| Mikhail Gusarov, Keith Packard, Carl Worth; [notmuch] Working with Maildir storage? (inbox signed unread)
+EOF
+test_expect_equal_file EXPECTED OUTPUT
+
+test_begin_subtest "Compound subquery"
+notmuch search --query-syntax=sexp '(and (thread :match (and (from keithp) (date 2009))) (thread :match (to keithp)))' \
+    | notmuch_search_sanitize > OUTPUT
+cat<<EOF > EXPECTED
+thread:XXX   2009-11-18 [7/7] Lars Kellogg-Stedman, Mikhail Gusarov, Keith Packard, Carl Worth; [notmuch] Working with Maildir storage? (inbox signed unread)
+EOF
+test_expect_equal_file EXPECTED OUTPUT
+
+test_begin_subtest "missing subquery"
+notmuch search --query-syntax=sexp '(thread :match)' 1>OUTPUT 2>&1
+cat<<EOF > EXPECTED
+notmuch search: Syntax error in query
+missing subquery
+EOF
+test_expect_equal_file EXPECTED OUTPUT
+
 test_done
-- 
2.30.2

[PATCH 24/25] lib/parse-sexp: support infix subqueries

[David Bremner]

This is necessary so that programs can take infix syntax queries from
a user and use the sexp query syntax to construct e.g. a refinement of
that query.
---
 lib/parse-sexp.cc         | 32 ++++++++++++++++++++++++++++++++
 test/T081-sexpr-search.sh | 35 +++++++++++++++++++++++++++++++++++
 2 files changed, 67 insertions(+)

diff --git a/lib/parse-sexp.cc b/lib/parse-sexp.cc
index df3629af..502b1be0 100644
--- a/lib/parse-sexp.cc
+++ b/lib/parse-sexp.cc
@@ -56,6 +56,7 @@ static _sexp_field_t fields[] =
     { "from",         Xapian::Query::OP_PHRASE,       SEXP_FLAG_REGEXP },
     { "folder",       Xapian::Query::OP_OR,           SEXP_FLAG_REGEXP },
     { "id",           Xapian::Query::OP_OR,           SEXP_FLAG_REGEXP },
+    { "infix",        Xapian::Query::OP_INVALID,      SEXP_FLAG_NONE },
     { "is",           Xapian::Query::OP_AND,          SEXP_FLAG_WILDCARD | SEXP_FLAG_REGEXP },
     { "mid",          Xapian::Query::OP_OR,           SEXP_FLAG_REGEXP },
     { "mimetype",     Xapian::Query::OP_PHRASE,       SEXP_FLAG_NONE },
@@ -236,6 +237,34 @@ _sexp_parse_keywords (notmuch_database_t *notmuch, const char *prefix, const sex
     return NOTMUCH_STATUS_SUCCESS;
 }
 
+static notmuch_status_t
+_sexp_parse_infix (notmuch_database_t *notmuch, const sexp_t *sx, Xapian::Query &output)
+{
+
+    if (sx->ty != SEXP_VALUE) {
+	_notmuch_database_log (notmuch, "infix argument must be value, not list\n");
+	return NOTMUCH_STATUS_BAD_QUERY_SYNTAX;
+    }
+    try {
+	output = notmuch->query_parser->parse_query (sx->val, NOTMUCH_QUERY_PARSER_FLAGS);
+    } catch (const Xapian::QueryParserError &error) {
+	_notmuch_database_log (notmuch, "Syntax error in infix query: %s\n", sx->val);
+	return NOTMUCH_STATUS_BAD_QUERY_SYNTAX;
+    } catch (const Xapian::Error &error) {
+	if (! notmuch->exception_reported) {
+	    _notmuch_database_log (notmuch,
+				   "A Xapian exception occurred parsing query: %s\n",
+				   error.get_msg ().c_str ());
+	    _notmuch_database_log_append (notmuch,
+					  "Query string was: %s\n",
+					  sx->val);
+	    notmuch->exception_reported = true;
+	    return NOTMUCH_STATUS_XAPIAN_EXCEPTION;
+	}
+    }
+    return NOTMUCH_STATUS_SUCCESS;
+}
+
 /* Here we expect the s-expression to be a proper list, with first
  * element defining and operation, or as a special case the empty
  * list */
@@ -296,6 +325,9 @@ _notmuch_sexp_to_xapian_query (notmuch_database_t *notmuch, const sexp_t *sx, Xa
 		    return _notmuch_query_name_to_query (notmuch, sx->list->next->val, output);
 		}
 	    }
+	    if (strcasecmp (field->name, "infix") == 0) {
+		return _sexp_parse_infix (notmuch, rest, output);
+	    }
 
 	    term_prefix = _find_prefix (field->name);
 
diff --git a/test/T081-sexpr-search.sh b/test/T081-sexpr-search.sh
index f0a7efb0..45568ba5 100755
--- a/test/T081-sexpr-search.sh
+++ b/test/T081-sexpr-search.sh
@@ -471,4 +471,39 @@ missing subquery
 EOF
 test_expect_equal_file EXPECTED OUTPUT
 
+test_begin_subtest "infix query"
+notmuch search to:searchbyto | notmuch_search_sanitize > EXPECTED
+notmuch search --query-syntax=sexp '(infix "to:searchbyto")' |  notmuch_search_sanitize > OUTPUT
+test_expect_equal_file EXPECTED OUTPUT
+
+test_begin_subtest "bad infix query 1"
+notmuch search --query-syntax=sexp '(infix "from:/unbalanced")' 2>&1|  notmuch_search_sanitize > OUTPUT
+cat <<EOF > EXPECTED
+notmuch search: Syntax error in query
+Syntax error in infix query: from:/unbalanced
+EOF
+test_expect_equal_file EXPECTED OUTPUT
+
+test_begin_subtest "bad infix query 2"
+notmuch search --query-syntax=sexp '(infix "thread:{unbalanced")' 2>&1|  notmuch_search_sanitize > OUTPUT
+cat <<EOF > EXPECTED
+notmuch search: Syntax error in query
+Syntax error in infix query: thread:{unbalanced
+EOF
+test_expect_equal_file EXPECTED OUTPUT
+
+test_begin_subtest "infix query that matches no messages"
+notmuch search --query-syntax=sexp '(and (infix "from:keithp") (infix "to:keithp"))' > OUTPUT
+test_expect_equal_file /dev/null OUTPUT
+
+test_begin_subtest "compound infix query"
+notmuch search date:2009-11-18..2009-11-18 and tag:unread > EXPECTED
+notmuch search --query-syntax=sexp  '(infix "date:2009-11-18..2009-11-18 and tag:unread")' > OUTPUT
+test_expect_equal_file EXPECTED OUTPUT
+
+test_begin_subtest "compound infix query 2"
+notmuch search date:2009-11-18..2009-11-18 and tag:unread > EXPECTED
+notmuch search --query-syntax=sexp  '(and (infix "date:2009-11-18..2009-11-18") (infix "tag:unread"))' > OUTPUT
+test_expect_equal_file EXPECTED OUTPUT
+
 test_done
-- 
2.30.2

[PATCH 25/25] lib/parse-sexp: parse user headers

[David Bremner]

Rather than adding one prefix per user header, we create a single
'header' prefix that takes the configured name of the prefix as a parameter.
---
 lib/parse-sexp.cc         | 50 +++++++++++++++++++++++
 test/T081-sexpr-search.sh | 84 +++++++++++++++++++++++++++++++++++++++
 2 files changed, 134 insertions(+)

diff --git a/lib/parse-sexp.cc b/lib/parse-sexp.cc
index 502b1be0..207403a5 100644
--- a/lib/parse-sexp.cc
+++ b/lib/parse-sexp.cc
@@ -55,6 +55,7 @@ static _sexp_field_t fields[] =
     { "date",         Xapian::Query::OP_INVALID,      SEXP_FLAG_NONE },
     { "from",         Xapian::Query::OP_PHRASE,       SEXP_FLAG_REGEXP },
     { "folder",       Xapian::Query::OP_OR,           SEXP_FLAG_REGEXP },
+    { "header",       Xapian::Query::OP_INVALID,      SEXP_FLAG_WILDCARD },
     { "id",           Xapian::Query::OP_OR,           SEXP_FLAG_REGEXP },
     { "infix",        Xapian::Query::OP_INVALID,      SEXP_FLAG_NONE },
     { "is",           Xapian::Query::OP_AND,          SEXP_FLAG_WILDCARD | SEXP_FLAG_REGEXP },
@@ -265,6 +266,52 @@ _sexp_parse_infix (notmuch_database_t *notmuch, const sexp_t *sx, Xapian::Query
     return NOTMUCH_STATUS_SUCCESS;
 }
 
+static notmuch_status_t
+_sexp_parse_header (notmuch_database_t *notmuch, _sexp_flag_t flags, const sexp_t *sx,
+		    Xapian::Query &output)
+{
+    const char *term_prefix;
+    _sexp_flag_t new_flags;
+    const sexp_t *rest;
+    notmuch_status_t status;
+
+    if (! sx) {
+	_notmuch_database_log (notmuch, "missing header name\n");
+	return NOTMUCH_STATUS_BAD_QUERY_SYNTAX;
+    }
+
+    if (sx->ty != SEXP_VALUE) {
+	_notmuch_database_log (notmuch, "header name must be atom, not list\n");
+	return NOTMUCH_STATUS_BAD_QUERY_SYNTAX;
+    }
+
+    term_prefix = _notmuch_string_map_get (notmuch->user_prefix, sx->val);
+    if (! term_prefix) {
+	_notmuch_database_log (notmuch, "unknown header name %s\n", sx->val);
+	return NOTMUCH_STATUS_BAD_QUERY_SYNTAX;
+    }
+
+    status = _sexp_parse_keywords (notmuch, "header", sx->next, SEXP_FLAG_WILDCARD, new_flags, rest);
+    if (status)
+	return status;
+
+    if ((new_flags | flags) & SEXP_FLAG_WILDCARD) {
+	if (rest) {
+	    _notmuch_database_log (notmuch, "extra term(s) after wildcard\n", sx->val);
+	    return NOTMUCH_STATUS_BAD_QUERY_SYNTAX;
+	}
+	output = Xapian::Query (Xapian::Query::OP_WILDCARD, term_prefix);
+	return NOTMUCH_STATUS_SUCCESS;
+    }
+
+    if (! rest) {
+	_notmuch_database_log (notmuch, "missing header terms\n");
+	return NOTMUCH_STATUS_BAD_QUERY_SYNTAX;
+    }
+
+    return _sexp_combine_field (term_prefix, Xapian::Query::OP_PHRASE, rest, output);
+}
+
 /* Here we expect the s-expression to be a proper list, with first
  * element defining and operation, or as a special case the empty
  * list */
@@ -317,6 +364,9 @@ _notmuch_sexp_to_xapian_query (notmuch_database_t *notmuch, const sexp_t *sx, Xa
 	    if (strcasecmp (field->name, "date") == 0) {
 		return _sexp_parse_date (notmuch, sx, output);
 	    }
+	    if (strcasecmp (field->name, "header") == 0) {
+		return _sexp_parse_header (notmuch, flags, rest, output);
+	    }
 	    if (strcasecmp (field->name, "query") == 0) {
 		if (! sx->list->next || ! sx->list->next->val) {
 		    _notmuch_database_log (notmuch, "missing query name\n");
diff --git a/test/T081-sexpr-search.sh b/test/T081-sexpr-search.sh
index 45568ba5..ae76a6e6 100755
--- a/test/T081-sexpr-search.sh
+++ b/test/T081-sexpr-search.sh
@@ -506,4 +506,88 @@ notmuch search date:2009-11-18..2009-11-18 and tag:unread > EXPECTED
 notmuch search --query-syntax=sexp  '(and (infix "date:2009-11-18..2009-11-18") (infix "tag:unread"))' > OUTPUT
 test_expect_equal_file EXPECTED OUTPUT
 
+test_begin_subtest "user header (no name)"
+notmuch search --query-syntax=sexp '(header)' >& OUTPUT
+cat <<EOF > EXPECTED
+notmuch search: Syntax error in query
+missing header name
+EOF
+test_expect_equal_file EXPECTED OUTPUT
+
+test_begin_subtest "user header (illegal name)"
+notmuch search --query-syntax=sexp '(header (what?))' >& OUTPUT
+cat <<EOF > EXPECTED
+notmuch search: Syntax error in query
+header name must be atom, not list
+EOF
+test_expect_equal_file EXPECTED OUTPUT
+
+test_begin_subtest "user header (unknown header)"
+notmuch search --query-syntax=sexp '(header FooBar)' >& OUTPUT
+cat <<EOF > EXPECTED
+notmuch search: Syntax error in query
+unknown header name FooBar
+EOF
+test_expect_equal_file EXPECTED OUTPUT
+
+test_begin_subtest "adding user header"
+test_expect_code 0 "notmuch config set index.header.List \"List-Id\""
+
+test_begin_subtest "user header (no terms)"
+notmuch search --query-syntax=sexp '(header List)' >& OUTPUT
+cat <<EOF > EXPECTED
+notmuch search: Syntax error in query
+missing header terms
+EOF
+test_expect_equal_file EXPECTED OUTPUT
+
+test_begin_subtest "reindexing"
+test_expect_code 0 'notmuch reindex "*"'
+
+test_begin_subtest "wildcard search for user header"
+grep -Ril List-Id ${MAIL_DIR} | sort | notmuch_dir_sanitize > EXPECTED
+notmuch search --output=files --query-syntax=sexp '(header :any List)' | sort | notmuch_dir_sanitize > OUTPUT
+test_expect_equal_file EXPECTED OUTPUT
+
+test_begin_subtest "wildcard search for user header: extra tokens"
+notmuch search --query-syntax=sexp '(header :any List trailing-garbage)' >& OUTPUT
+cat <<EOF > EXPECTED
+notmuch search: Syntax error in query
+extra term(s) after wildcard
+EOF
+test_expect_equal_file EXPECTED OUTPUT
+
+test_begin_subtest "wildcard search for user header: extra tokens (2)"
+notmuch search --query-syntax=sexp '(header List :any trailing-garbage)' >& OUTPUT
+cat <<EOF > EXPECTED
+notmuch search: Syntax error in query
+extra term(s) after wildcard
+EOF
+test_expect_equal_file EXPECTED OUTPUT
+
+test_begin_subtest "wildcard search for user header (post header flag)"
+grep -Ril List-Id ${MAIL_DIR} | sort | notmuch_dir_sanitize > EXPECTED
+notmuch search --output=files --query-syntax=sexp '(header List :any)' | sort | notmuch_dir_sanitize > OUTPUT
+test_expect_equal_file EXPECTED OUTPUT
+
+test_begin_subtest "search for user header"
+notmuch search List:notmuch | notmuch_search_sanitize > EXPECTED
+notmuch search --query-syntax=sexp '(header List notmuch)' | notmuch_search_sanitize > OUTPUT
+test_expect_equal_file EXPECTED OUTPUT
+
+test_begin_subtest "search for user header (list token)"
+notmuch search List:notmuch | notmuch_search_sanitize > EXPECTED
+notmuch search --query-syntax=sexp '(header List notmuch.notmuchmail.org)' | notmuch_search_sanitize > OUTPUT
+test_expect_equal_file EXPECTED OUTPUT
+
+test_begin_subtest "search for user header (quoted string)"
+notmuch search 'List:"notmuch notmuchmail org"' | notmuch_search_sanitize > EXPECTED
+notmuch search --query-syntax=sexp '(header List "notmuch notmuchmail org")' | notmuch_search_sanitize > OUTPUT
+test_expect_equal_file EXPECTED OUTPUT
+
+test_begin_subtest "search for user header (atoms)"
+notmuch search 'List:"notmuch notmuchmail org"' | notmuch_search_sanitize > EXPECTED
+notmuch search --query-syntax=sexp '(header List notmuch notmuchmail org)' | notmuch_search_sanitize > OUTPUT
+test_expect_equal_file EXPECTED OUTPUT
+
 test_done
-- 
2.30.2