Early preview of s-expression based query parser

Early preview of s-expression based query parser

[David Bremner]

I have no plans to replace or deprecate the Xapian query parser based
syntax. On the other hand, having full control over the parser makes
it easier to support certain features. As an example, once the basic
parser is in place, the last patch in the series adds prefixed
wildcards with about 10 lines of code.

S-expressions are also hand for constructing queries from other
queries, which is not really well supported by the Xapian query
parser. So it should also make it possible to fix a few bugs resulting
from gluing strings together and hoping the result is a valid query.

There are quite a few missing features here, and a couple known bugs
(the treatment of body: queries is probably mostly broken in the new
parser). Finally, when I next re-roll the series I will probably push
the error handling earlier in the series.

Feedback of any kind is welcome, but particularly on UI / UX
issues. You can get a pretty good idea of the supported syntax by
looking at the tests.

It does add a dependency on sfsexp, a C s-expression parsing
library. It's not heavyweight, but it's also not available very many
places yet (other than on github).  Upstream has been great about
merging a few changes for me. One question mark is UTF8 support; it
seems to work OK, but upstream is in the process of documenting the
support status.

[PATCH 01/11] configure: optional library sfsexp

[David Bremner]

This is essentially the same as the other checks using pkg-config.
---
 configure | 23 ++++++++++++++++++++++-
 1 file changed, 22 insertions(+), 1 deletion(-)

diff --git a/configure b/configure
index cfa9c09b..15c1924f 100755
--- a/configure
+++ b/configure
@@ -820,6 +820,19 @@ else
     WITH_BASH=0
 fi
 
+printf "Checking for sfsexp... "
+if pkg-config --exists sfsexp; then
+    printf "Yes.\n"
+    have_sfsexp=1
+    sfsexp_cflags=$(pkg-config --cflags sfsexp)
+    sfsexp_ldflags=$(pkg-config --libs sfsexp)
+else
+    printf "No (will not enable s-expression queries).\n"
+    have_sfsexp=0
+    sfsexp_cflags=
+    sfsexp_ldflags=
+fi
+
 if [ -z "${EMACSLISPDIR-}" ]; then
     EMACSLISPDIR="\$(prefix)/share/emacs/site-lisp"
 fi
@@ -1443,6 +1456,13 @@ HAVE_VALGRIND = ${have_valgrind}
 # And if so, flags needed at compile time for valgrind macros
 VALGRIND_CFLAGS = ${valgrind_cflags}
 
+# Whether the sfsexp library is available
+HAVE_SFSEXP = ${have_sfsexp}
+
+# And if so, flags needed at compile/link time for sfsexp
+SFSEXP_CFLAGS = ${sfsexp_cflags}
+SFSEXP_LDFLAGS = ${sfsexp_ldflags}
+
 # Support for emacs
 WITH_EMACS = ${WITH_EMACS}
 
@@ -1459,6 +1479,7 @@ WITH_ZSH = ${WITH_ZSH}
 COMMON_CONFIGURE_CFLAGS = \\
 	\$(GMIME_CFLAGS) \$(TALLOC_CFLAGS) \$(ZLIB_CFLAGS)	\\
 	-DHAVE_VALGRIND=\$(HAVE_VALGRIND) \$(VALGRIND_CFLAGS)	\\
+	-DHAVE_SFSEXP=\$(HAVE_SFSEXP) \$(SFSEXP_CFLAGS)		\\
 	-DHAVE_GETLINE=\$(HAVE_GETLINE)				\\
 	-DWITH_EMACS=\$(WITH_EMACS)				\\
 	-DHAVE_CANONICALIZE_FILE_NAME=\$(HAVE_CANONICALIZE_FILE_NAME) \\
@@ -1475,7 +1496,7 @@ CONFIGURE_CFLAGS = \$(COMMON_CONFIGURE_CFLAGS)
 
 CONFIGURE_CXXFLAGS = \$(COMMON_CONFIGURE_CFLAGS) \$(XAPIAN_CXXFLAGS)
 
-CONFIGURE_LDFLAGS = \$(GMIME_LDFLAGS) \$(TALLOC_LDFLAGS) \$(ZLIB_LDFLAGS) \$(XAPIAN_LDFLAGS)
+CONFIGURE_LDFLAGS = \$(GMIME_LDFLAGS) \$(TALLOC_LDFLAGS) \$(ZLIB_LDFLAGS) \$(XAPIAN_LDFLAGS) \$(SFSEXP_LDFLAGS)
 EOF
 
 # construct the sh.config
-- 
2.30.2

[PATCH 02/11] lib: split notmuch_query_create

[David Bremner]

Most of the function will be re-usable when creating a query from an
s-expression.
---
 lib/query.cc | 19 ++++++++++++++++---
 1 file changed, 16 insertions(+), 3 deletions(-)

diff --git a/lib/query.cc b/lib/query.cc
index 792aba21..39b85e91 100644
--- a/lib/query.cc
+++ b/lib/query.cc
@@ -84,9 +84,9 @@ _notmuch_query_destructor (notmuch_query_t *query)
     return 0;
 }
 
-notmuch_query_t *
-notmuch_query_create (notmuch_database_t *notmuch,
-		      const char *query_string)
+static notmuch_query_t *
+_notmuch_query_constructor (notmuch_database_t *notmuch,
+			    const char *query_string)
 {
     notmuch_query_t *query;
 
@@ -116,6 +116,19 @@ notmuch_query_create (notmuch_database_t *notmuch,
     return query;
 }
 
+notmuch_query_t *
+notmuch_query_create (notmuch_database_t *notmuch,
+		      const char *query_string)
+{
+
+    notmuch_query_t *query = _notmuch_query_constructor (notmuch, query_string);
+
+    if (! query)
+	return NULL;
+
+    return query;
+}
+
 static notmuch_status_t
 _notmuch_query_ensure_parsed (notmuch_query_t *query)
 {
-- 
2.30.2

[PATCH 03/11] lib: define notmuch_query_create_sexpr

[David Bremner]

Set the parsing syntax when the (notmuch) query object is
created. Initially the library always returns a trivial query that
matches all messages.
---
 lib/notmuch.h |  4 ++++
 lib/query.cc  | 49 +++++++++++++++++++++++++++++++++++++++++++++----
 2 files changed, 49 insertions(+), 4 deletions(-)

diff --git a/lib/notmuch.h b/lib/notmuch.h
index 3b28bea3..a9abdd18 100644
--- a/lib/notmuch.h
+++ b/lib/notmuch.h
@@ -961,6 +961,10 @@ notmuch_query_t *
 notmuch_query_create (notmuch_database_t *database,
 		      const char *query_string);
 
+notmuch_query_t *
+notmuch_query_create_sexpr (notmuch_database_t *database,
+			    const char *query_string);
+
 /**
  * Sort values for notmuch_query_set_sort.
  */
diff --git a/lib/query.cc b/lib/query.cc
index 39b85e91..9d2b6e50 100644
--- a/lib/query.cc
+++ b/lib/query.cc
@@ -23,6 +23,13 @@
 
 #include <glib.h> /* GHashTable, GPtrArray */
 
+#include "sexp.h"
+
+typedef enum {
+    NOTMUCH_QUERY_SYNTAX_XAPIAN,
+    NOTMUCH_QUERY_SYNTAX_SEXPR,
+} notmuch_query_syntax_t;
+
 struct _notmuch_query {
     notmuch_database_t *notmuch;
     const char *query_string;
@@ -30,6 +37,7 @@ struct _notmuch_query {
     notmuch_string_list_t *exclude_terms;
     notmuch_exclude_t omit_excluded;
     bool parsed;
+    notmuch_query_syntax_t syntax;
     Xapian::Query xapian_query;
     std::set<std::string> terms;
 };
@@ -126,15 +134,29 @@ notmuch_query_create (notmuch_database_t *notmuch,
     if (! query)
 	return NULL;
 
+    query->syntax = NOTMUCH_QUERY_SYNTAX_XAPIAN;
+
     return query;
 }
 
-static notmuch_status_t
-_notmuch_query_ensure_parsed (notmuch_query_t *query)
+notmuch_query_t *
+notmuch_query_create_sexpr (notmuch_database_t *notmuch,
+			    const char *query_string)
 {
-    if (query->parsed)
-	return NOTMUCH_STATUS_SUCCESS;
 
+    notmuch_query_t *query = _notmuch_query_constructor (notmuch, query_string);
+
+    if (! query)
+	return NULL;
+
+    query->syntax = NOTMUCH_QUERY_SYNTAX_SEXPR;
+
+    return query;
+}
+
+static notmuch_status_t
+_notmuch_query_ensure_parsed_xapian (notmuch_query_t *query)
+{
     try {
 	query->xapian_query =
 	    query->notmuch->query_parser->
@@ -167,6 +189,25 @@ _notmuch_query_ensure_parsed (notmuch_query_t *query)
     return NOTMUCH_STATUS_SUCCESS;
 }
 
+static notmuch_status_t
+_notmuch_query_ensure_parsed_sexpr (notmuch_query_t *query)
+{
+    query->xapian_query = Xapian::Query::MatchAll;
+    return NOTMUCH_STATUS_SUCCESS;
+}
+
+static notmuch_status_t
+_notmuch_query_ensure_parsed (notmuch_query_t *query)
+{
+    if (query->parsed)
+	return NOTMUCH_STATUS_SUCCESS;
+
+    if (query->syntax == NOTMUCH_QUERY_SYNTAX_SEXPR)
+	return _notmuch_query_ensure_parsed_sexpr (query);
+
+    return _notmuch_query_ensure_parsed_xapian (query);
+}
+
 const char *
 notmuch_query_get_query_string (const notmuch_query_t *query)
 {
-- 
2.30.2

[PATCH 04/11] CLI/search+address: support sexpr queries

[David Bremner]

Initially support selection of query syntax in two subcommands to
enable testing.
---
 notmuch-search.c     | 16 +++++++++++++++-
 test/T080-search.sh  |  5 +++++
 test/T095-address.sh |  5 +++++
 3 files changed, 25 insertions(+), 1 deletion(-)

diff --git a/notmuch-search.c b/notmuch-search.c
index 244817a9..fd25a414 100644
--- a/notmuch-search.c
+++ b/notmuch-search.c
@@ -43,6 +43,11 @@ typedef enum {
     DEDUP_ADDRESS,
 } dedup_t;
 
+typedef enum {
+    QUERY_SYNTAX_XAPIAN,
+    QUERY_SYNTAX_SEXP
+} query_syntax_t;
+
 typedef enum {
     NOTMUCH_FORMAT_JSON,
     NOTMUCH_FORMAT_TEXT,
@@ -56,6 +61,7 @@ typedef struct {
     int format_sel;
     sprinter_t *format;
     int exclude;
+    int query_syntax;
     notmuch_query_t *query;
     int sort;
     int output;
@@ -721,7 +727,10 @@ _notmuch_search_prepare (search_context_t *ctx, int argc, char *argv[])
 	return EXIT_FAILURE;
     }
 
-    ctx->query = notmuch_query_create (ctx->notmuch, query_str);
+    if (ctx->query_syntax == QUERY_SYNTAX_SEXP)
+	ctx->query = notmuch_query_create_sexpr (ctx->notmuch, query_str);
+    else
+	ctx->query = notmuch_query_create (ctx->notmuch, query_str);
     if (ctx->query == NULL) {
 	fprintf (stderr, "Out of memory\n");
 	return EXIT_FAILURE;
@@ -771,6 +780,7 @@ static search_context_t search_context = {
     .format_sel = NOTMUCH_FORMAT_TEXT,
     .exclude = NOTMUCH_EXCLUDE_TRUE,
     .sort = NOTMUCH_SORT_NEWEST_FIRST,
+    .query_syntax = QUERY_SYNTAX_XAPIAN,
     .output = 0,
     .offset = 0,
     .limit = -1, /* unlimited */
@@ -789,6 +799,10 @@ static const notmuch_opt_desc_t common_options[] = {
 				  { "text", NOTMUCH_FORMAT_TEXT },
 				  { "text0", NOTMUCH_FORMAT_TEXT0 },
 				  { 0, 0 } } },
+    { .opt_keyword = &search_context.query_syntax, .name = "query-syntax", .keywords =
+	  (notmuch_keyword_t []){ { "xapian", QUERY_SYNTAX_XAPIAN },
+				  { "sexp", QUERY_SYNTAX_SEXP },
+				  { 0, 0 } } },
     { .opt_int = &notmuch_format_version, .name = "format-version" },
     { }
 };
diff --git a/test/T080-search.sh b/test/T080-search.sh
index a3f0dead..966e772a 100755
--- a/test/T080-search.sh
+++ b/test/T080-search.sh
@@ -189,4 +189,9 @@ test_begin_subtest "parts do not have adjacent term positions"
 output=$(notmuch search id:termpos and '"c x"')
 test_expect_equal "$output" ""
 
+test_begin_subtest "sexpr query: all messages"
+notmuch search '*' > EXPECTED
+notmuch search --query-syntax=sexp '()' > OUTPUT
+test_expect_equal_file EXPECTED OUTPUT
+
 test_done
diff --git a/test/T095-address.sh b/test/T095-address.sh
index 817be538..adf0b307 100755
--- a/test/T095-address.sh
+++ b/test/T095-address.sh
@@ -325,4 +325,9 @@ cat <<EOF >EXPECTED
 EOF
 test_expect_equal_file EXPECTED OUTPUT
 
+test_begin_subtest "sexpr query: all messages"
+notmuch address '*' > EXPECTED
+notmuch address --query-syntax=sexp '()' > OUTPUT
+test_expect_equal_file EXPECTED OUTPUT
+
 test_done
-- 
2.30.2

[PATCH 05/11] lib/parse-sexp: parse 'and', 'not', 'or'

[David Bremner]

This is not too useful yet, but provides gluing together queries in
various ways, which is actually one of the main motivations for
the s-expression query format.
---
 lib/Makefile.local        |  3 +-
 lib/parse-sexp.cc         | 72 +++++++++++++++++++++++++++++++++++++++
 lib/query.cc              |  7 ++--
 test/T080-search.sh       |  5 ---
 test/T081-sexpr-search.sh | 22 ++++++++++++
 5 files changed, 100 insertions(+), 9 deletions(-)
 create mode 100644 lib/parse-sexp.cc
 create mode 100755 test/T081-sexpr-search.sh

diff --git a/lib/Makefile.local b/lib/Makefile.local
index e2d4b91d..1378a74b 100644
--- a/lib/Makefile.local
+++ b/lib/Makefile.local
@@ -63,7 +63,8 @@ libnotmuch_cxx_srcs =		\
 	$(dir)/features.cc	\
 	$(dir)/prefix.cc	\
 	$(dir)/open.cc		\
-	$(dir)/init.cc
+	$(dir)/init.cc		\
+	$(dir)/parse-sexp.cc
 
 libnotmuch_modules := $(libnotmuch_c_srcs:.c=.o) $(libnotmuch_cxx_srcs:.cc=.o)
 
diff --git a/lib/parse-sexp.cc b/lib/parse-sexp.cc
new file mode 100644
index 00000000..eded98e7
--- /dev/null
+++ b/lib/parse-sexp.cc
@@ -0,0 +1,72 @@
+#include <xapian.h>
+#include "notmuch-private.h"
+#include "sexp.h"
+
+typedef struct  {
+    const char *name;
+    Xapian::Query::op xapian_op;
+    Xapian::Query initial;
+} _sexp_op_t;
+
+static _sexp_op_t operations[] =
+{
+    { "and",    Xapian::Query::OP_AND,          Xapian::Query::MatchAll },
+    { "not",    Xapian::Query::OP_AND_NOT,      Xapian::Query::MatchAll },
+    { "or",     Xapian::Query::OP_OR,           Xapian::Query::MatchNothing },
+    { }
+};
+
+static Xapian::Query _sexp_to_xapian_query (sexp_t *sx);
+
+static Xapian::Query
+_sexp_combine_query (Xapian::Query::op operation,
+		     Xapian::Query left,
+		     sexp_t *sx)
+{
+
+    /* if we run out elements, return accumulator */
+
+    if (! sx)
+	return left;
+
+    return _sexp_combine_query (operation,
+				Xapian::Query (operation,
+					       left,
+					       _sexp_to_xapian_query (sx)),
+				sx->next);
+}
+
+Xapian::Query
+_notmuch_sexp_string_to_xapian_query (notmuch_database_t *notmuch, const char *querystr)
+{
+    sexp_t *sx = NULL;
+    char *buf = talloc_strdup (notmuch, querystr);
+
+    sx = parse_sexp (buf, strlen (querystr));
+    return _sexp_to_xapian_query (sx);
+}
+
+/* Here we expect the s-expression to be a proper list, with first
+ * element defining and operation, or as a special case the empty
+ * list */
+
+static Xapian::Query
+_sexp_to_xapian_query (sexp_t *sx)
+{
+
+    const _sexp_op_t *op;
+
+    /* Currently we don't understand atoms */
+    assert (sx->ty == SEXP_LIST);
+
+    /* Empty list */
+    if (! sx->list)
+	return Xapian::Query::MatchAll;
+
+    for (op = operations; op && op->name; op++) {
+	if (strcasecmp (op->name, hd_sexp (sx)->val) == 0)
+	    return _sexp_combine_query (op->xapian_op, op->initial, sx->list->next);
+    }
+
+    INTERNAL_ERROR ("unimplemented prefix %s\n", sx->list->val);
+}
diff --git a/lib/query.cc b/lib/query.cc
index 9d2b6e50..6e89af60 100644
--- a/lib/query.cc
+++ b/lib/query.cc
@@ -20,11 +20,10 @@
 
 #include "notmuch-private.h"
 #include "database-private.h"
+#include "parse-sexp.h"
 
 #include <glib.h> /* GHashTable, GPtrArray */
 
-#include "sexp.h"
-
 typedef enum {
     NOTMUCH_QUERY_SYNTAX_XAPIAN,
     NOTMUCH_QUERY_SYNTAX_SEXPR,
@@ -192,7 +191,9 @@ _notmuch_query_ensure_parsed_xapian (notmuch_query_t *query)
 static notmuch_status_t
 _notmuch_query_ensure_parsed_sexpr (notmuch_query_t *query)
 {
-    query->xapian_query = Xapian::Query::MatchAll;
+
+    query->xapian_query = _notmuch_sexp_string_to_xapian_query (query->notmuch, query->query_string);
+
     return NOTMUCH_STATUS_SUCCESS;
 }
 
diff --git a/test/T080-search.sh b/test/T080-search.sh
index 966e772a..a3f0dead 100755
--- a/test/T080-search.sh
+++ b/test/T080-search.sh
@@ -189,9 +189,4 @@ test_begin_subtest "parts do not have adjacent term positions"
 output=$(notmuch search id:termpos and '"c x"')
 test_expect_equal "$output" ""
 
-test_begin_subtest "sexpr query: all messages"
-notmuch search '*' > EXPECTED
-notmuch search --query-syntax=sexp '()' > OUTPUT
-test_expect_equal_file EXPECTED OUTPUT
-
 test_done
diff --git a/test/T081-sexpr-search.sh b/test/T081-sexpr-search.sh
new file mode 100755
index 00000000..0b75334a
--- /dev/null
+++ b/test/T081-sexpr-search.sh
@@ -0,0 +1,22 @@
+#!/usr/bin/env bash
+test_description='"notmuch search" in several variations'
+. $(dirname "$0")/test-lib.sh || exit 1
+
+add_email_corpus
+
+for query in '()' '(not)' '(and)' '(or ())' '(or (not))' '(or (and))' \
+	     '(or (and) (or) (not (and)))'; do
+    test_begin_subtest "all messages: $query"
+    notmuch search '*' > EXPECTED
+    notmuch search --query-syntax=sexp "$query" > OUTPUT
+    test_expect_equal_file EXPECTED OUTPUT
+done
+
+for query in '(or)' '(not ())' '(not (not))' '(not (and))' \
+		    '(not (or (and) (or) (not (and))))'; do
+    test_begin_subtest "no messages: $query"
+    notmuch search --query-syntax=sexp "$query" > OUTPUT
+    test_expect_equal_file /dev/null OUTPUT
+done
+
+test_done
-- 
2.30.2

[PATCH 06/11] lib/parse-sexp: parse 'subject'

[David Bremner]

Initially this only works for lists of individual terms, which is a
bit inconvenient. This will be improved in a following commit.
---
 lib/parse-sexp.cc         | 36 ++++++++++++++++++++++++++++++++++++
 test/T081-sexpr-search.sh | 15 +++++++++++++++
 2 files changed, 51 insertions(+)

diff --git a/lib/parse-sexp.cc b/lib/parse-sexp.cc
index eded98e7..4a2fac8b 100644
--- a/lib/parse-sexp.cc
+++ b/lib/parse-sexp.cc
@@ -16,6 +16,17 @@ static _sexp_op_t operations[] =
     { }
 };
 
+typedef struct  {
+    const char *name;
+    Xapian::Query::op xapian_op;
+} _sexp_field_t;
+
+static _sexp_field_t fields[] =
+{
+    { "subject",        Xapian::Query::OP_PHRASE },
+    { }
+};
+
 static Xapian::Query _sexp_to_xapian_query (sexp_t *sx);
 
 static Xapian::Query
@@ -46,6 +57,26 @@ _notmuch_sexp_string_to_xapian_query (notmuch_database_t *notmuch, const char *q
     return _sexp_to_xapian_query (sx);
 }
 
+static Xapian::Query
+_sexp_combine_field (const char *prefix,
+		     Xapian::Query::op operation,
+		     sexp_t *sx)
+{
+    std::vector<std::string> terms;
+
+    for (sexp_t *cur = sx; cur; cur = cur->next) {
+	std::string pref_str = prefix;
+	std::string word = cur->val;
+
+	if (operation == Xapian::Query::OP_PHRASE)
+	    word = Xapian::Unicode::tolower (word);
+
+
+	terms.push_back (pref_str + word);
+    }
+    return Xapian::Query (operation, terms.begin (), terms.end ());
+}
+
 /* Here we expect the s-expression to be a proper list, with first
  * element defining and operation, or as a special case the empty
  * list */
@@ -68,5 +99,10 @@ _sexp_to_xapian_query (sexp_t *sx)
 	    return _sexp_combine_query (op->xapian_op, op->initial, sx->list->next);
     }
 
+    for (const _sexp_field_t *field = fields; field && field->name; field++) {
+	if (strcasecmp (field->name, hd_sexp (sx)->val) == 0)
+	    return _sexp_combine_field (_find_prefix (field->name), field->xapian_op, sx->list->next);
+    }
+
     INTERNAL_ERROR ("unimplemented prefix %s\n", sx->list->val);
 }
diff --git a/test/T081-sexpr-search.sh b/test/T081-sexpr-search.sh
index 0b75334a..1a80a133 100755
--- a/test/T081-sexpr-search.sh
+++ b/test/T081-sexpr-search.sh
@@ -19,4 +19,19 @@ for query in '(or)' '(not ())' '(not (not))' '(not (and))' \
     test_expect_equal_file /dev/null OUTPUT
 done
 
+test_begin_subtest "Search by subject"
+add_message [subject]=subjectsearchtest '[date]="Sat, 01 Jan 2000 12:00:00 -0000"'
+output=$(notmuch search --query-syntax=sexp '(subject subjectsearchtest)' | notmuch_search_sanitize)
+test_expect_equal "$output" "thread:XXX   2000-01-01 [1/1] Notmuch Test Suite; subjectsearchtest (inbox unread)"
+
+test_begin_subtest "Search by subject (case insensitive)"
+notmuch search tag:inbox and subject:maildir | notmuch_search_sanitize > EXPECTED
+notmuch search --query-syntax=sexp '(subject Maildir)' | notmuch_search_sanitize > OUTPUT
+test_expect_equal_file EXPECTED OUTPUT
+
+test_begin_subtest "Search by subject (utf-8):"
+add_message [subject]=utf8-sübjéct '[date]="Sat, 01 Jan 2000 12:00:00 -0000"'
+output=$(notmuch search --query-syntax=sexp '(subject utf8 sübjéct)' | notmuch_search_sanitize)
+test_expect_equal "$output" "thread:XXX   2000-01-01 [1/1] Notmuch Test Suite; utf8-sübjéct (inbox unread)"
+
 test_done
-- 
2.30.2\r

[PATCH 07/11] lib/parse-sexp: split terms in phrase mode

[David Bremner]

The goal is to have (subject foo-bar) match the same messages as
subject:foo-bar.
---
 lib/parse-sexp.cc         | 28 ++++++++++++++++++++++++----
 test/T081-sexpr-search.sh |  8 ++++++++
 2 files changed, 32 insertions(+), 4 deletions(-)

diff --git a/lib/parse-sexp.cc b/lib/parse-sexp.cc
index 4a2fac8b..26d4ee1f 100644
--- a/lib/parse-sexp.cc
+++ b/lib/parse-sexp.cc
@@ -66,13 +66,33 @@ _sexp_combine_field (const char *prefix,
 
     for (sexp_t *cur = sx; cur; cur = cur->next) {
 	std::string pref_str = prefix;
-	std::string word = cur->val;
 
-	if (operation == Xapian::Query::OP_PHRASE)
-	    word = Xapian::Unicode::tolower (word);
+	if (operation == Xapian::Query::OP_PHRASE) {
+	    Xapian::Utf8Iterator p (cur->val);
+	    Xapian::Utf8Iterator end;
 
+	    while (p != end) {
+		Xapian::Utf8Iterator start;
+		while (p != end && ! Xapian::Unicode::is_wordchar (*p))
+		    p++;
 
-	terms.push_back (pref_str + word);
+		if (p == end)
+		    break;
+
+		start = p;
+
+		while (p != end && Xapian::Unicode::is_wordchar (*p))
+		    p++;
+
+		if (p != start) {
+		    std::string word (start, p);
+		    word = Xapian::Unicode::tolower (word);
+		    terms.push_back (pref_str + word);
+		}
+	    }
+	} else {
+	    terms.push_back (pref_str + cur->val);
+	}
     }
     return Xapian::Query (operation, terms.begin (), terms.end ());
 }
diff --git a/test/T081-sexpr-search.sh b/test/T081-sexpr-search.sh
index 1a80a133..6369e483 100755
--- a/test/T081-sexpr-search.sh
+++ b/test/T081-sexpr-search.sh
@@ -34,4 +34,12 @@ add_message [subject]=utf8-sübjéct '[date]="Sat, 01 Jan 2000 12:00:00 -0000"'
 output=$(notmuch search --query-syntax=sexp '(subject utf8 sübjéct)' | notmuch_search_sanitize)
 test_expect_equal "$output" "thread:XXX   2000-01-01 [1/1] Notmuch Test Suite; utf8-sübjéct (inbox unread)"
 
+test_begin_subtest "Search by 'subject' (utf-8, phrase-token):"
+output=$(notmuch search --query-syntax=sexp '(subject utf8-sübjéct)' | notmuch_search_sanitize)
+test_expect_equal "$output" "thread:XXX   2000-01-01 [1/1] Notmuch Test Suite; utf8-sübjéct (inbox unread)"
+
+test_begin_subtest "Search by 'subject' (utf-8, quoted string):"
+output=$(notmuch search --query-syntax=sexp '(subject "utf8 sübjéct")' | notmuch_search_sanitize)
+test_expect_equal "$output" "thread:XXX   2000-01-01 [1/1] Notmuch Test Suite; utf8-sübjéct (inbox unread)"
+
 test_done
-- 
2.30.2\r

[PATCH 08/11] lib/parse-sexp: handle most fields

[David Bremner]

date and query will need to be handled seperately because they do not
fit the pattern of combining a bunch of terms with an operator.
---
 lib/parse-sexp.cc         |  13 ++++
 test/T081-sexpr-search.sh | 143 +++++++++++++++++++++++++++++++++++++-
 2 files changed, 153 insertions(+), 3 deletions(-)

diff --git a/lib/parse-sexp.cc b/lib/parse-sexp.cc
index 26d4ee1f..81bf5cee 100644
--- a/lib/parse-sexp.cc
+++ b/lib/parse-sexp.cc
@@ -23,7 +23,20 @@ typedef struct  {
 
 static _sexp_field_t fields[] =
 {
+    { "attachment",     Xapian::Query::OP_PHRASE },
+    { "body",           Xapian::Query::OP_PHRASE },
+    { "from",           Xapian::Query::OP_PHRASE },
+    { "folder",         Xapian::Query::OP_OR },
+    { "id",             Xapian::Query::OP_OR },
+    { "is",             Xapian::Query::OP_AND },
+    { "mid",            Xapian::Query::OP_OR },
+    { "mimetype",       Xapian::Query::OP_PHRASE },
+    { "path",           Xapian::Query::OP_OR },
+    { "property",       Xapian::Query::OP_AND },
     { "subject",        Xapian::Query::OP_PHRASE },
+    { "tag",            Xapian::Query::OP_AND },
+    { "thread",         Xapian::Query::OP_OR },
+    { "to",             Xapian::Query::OP_PHRASE },
     { }
 };
 
diff --git a/test/T081-sexpr-search.sh b/test/T081-sexpr-search.sh
index 6369e483..9ee9de7b 100755
--- a/test/T081-sexpr-search.sh
+++ b/test/T081-sexpr-search.sh
@@ -19,17 +19,110 @@ for query in '(or)' '(not ())' '(not (not))' '(not (and))' \
     test_expect_equal_file /dev/null OUTPUT
 done
 
-test_begin_subtest "Search by subject"
+test_begin_subtest "Search by 'attachment'"
+notmuch search attachment:notmuch-help.patch > EXPECTED
+notmuch search --query-syntax=sexp '(attachment notmuch-help.patch)' > OUTPUT
+test_expect_equal_file EXPECTED OUTPUT
+
+test_begin_subtest "Search by 'body'"
+add_message '[subject]="body search"' '[date]="Sat, 01 Jan 2000 12:00:00 -0000"' [body]=bodysearchtest
+output=$(notmuch search --query-syntax=sexp '(body bodysearchtest)' | notmuch_search_sanitize)
+test_expect_equal "$output" "thread:XXX   2000-01-01 [1/1] Notmuch Test Suite; body search (inbox unread)"
+
+test_begin_subtest "Search by 'body' (phrase)"
+add_message '[subject]="body search (phrase)"' '[date]="Sat, 01 Jan 2000 12:00:00 -0000"' '[body]="body search (phrase)"'
+add_message '[subject]="negative result"' '[date]="Sat, 01 Jan 2000 12:00:00 -0000"' '[body]="This phrase should not match the body search"'
+output=$(notmuch search --query-syntax=sexp '(body body search phrase)' | notmuch_search_sanitize)
+test_expect_equal "$output" "thread:XXX   2000-01-01 [1/1] Notmuch Test Suite; body search (phrase) (inbox unread)"
+
+test_begin_subtest "Search by 'body' (utf-8):"
+add_message '[subject]="utf8-message-body-subject"' '[date]="Sat, 01 Jan 2000 12:00:00 -0000"' '[body]="message body utf8: bödý"'
+output=$(notmuch search --query-syntax=sexp '(body bödý)' | notmuch_search_sanitize)
+test_expect_equal "$output" "thread:XXX   2000-01-01 [1/1] Notmuch Test Suite; utf8-message-body-subject (inbox unread)"
+
+test_begin_subtest "Search by 'from'"
+add_message '[subject]="search by from"' '[date]="Sat, 01 Jan 2000 12:00:00 -0000"' [from]=searchbyfrom
+output=$(notmuch search --query-syntax=sexp '(from searchbyfrom)' | notmuch_search_sanitize)
+test_expect_equal "$output" "thread:XXX   2000-01-01 [1/1] searchbyfrom; search by from (inbox unread)"
+
+test_begin_subtest "Search by 'from' (address)"
+add_message '[subject]="search by from (address)"' '[date]="Sat, 01 Jan 2000 12:00:00 -0000"' [from]=searchbyfrom@example.com
+output=$(notmuch search --query-syntax=sexp '(from searchbyfrom@example.com)' | notmuch_search_sanitize)
+test_expect_equal "$output" "thread:XXX   2000-01-01 [1/1] searchbyfrom@example.com; search by from (address) (inbox unread)"
+
+test_begin_subtest "Search by 'from' (name)"
+add_message '[subject]="search by from (name)"' '[date]="Sat, 01 Jan 2000 12:00:00 -0000"' '[from]="Search By From Name <test@example.com>"'
+output=$(notmuch search --query-syntax=sexp '(from "Search By From Name")' | notmuch_search_sanitize)
+test_expect_equal "$output" "thread:XXX   2000-01-01 [1/1] Search By From Name; search by from (name) (inbox unread)"
+
+test_begin_subtest "Search by 'from' (name and address)"
+output=$(notmuch search --query-syntax=sexp '(from "Search By From Name <test@example.com>")' | notmuch_search_sanitize)
+test_expect_equal "$output" "thread:XXX   2000-01-01 [1/1] Search By From Name; search by from (name) (inbox unread)"
+
+add_message '[dir]=bad' '[subject]="To the bone"'
+add_message '[dir]=.' '[subject]="Top level"'
+add_message '[dir]=bad/news' '[subject]="Bears"'
+mkdir -p "${MAIL_DIR}/duplicate/bad/news"
+cp "$gen_msg_filename" "${MAIL_DIR}/duplicate/bad/news"
+
+add_message '[dir]=things' '[subject]="These are a few"'
+add_message '[dir]=things/favorite' '[subject]="Raindrops, whiskers, kettles"'
+add_message '[dir]=things/bad' '[subject]="Bites, stings, sad feelings"'
+
+test_begin_subtest "Search by 'folder' (multiple)"
+output=$(notmuch search --query-syntax=sexp '(folder bad bad/news things/bad)' | notmuch_search_sanitize)
+test_expect_equal "$output" "thread:XXX   2001-01-05 [1/1] Notmuch Test Suite; To the bone (inbox unread)
+thread:XXX   2001-01-05 [1/1(2)] Notmuch Test Suite; Bears (inbox unread)
+thread:XXX   2001-01-05 [1/1] Notmuch Test Suite; Bites, stings, sad feelings (inbox unread)"
+
+test_begin_subtest "Search by 'folder': top level."
+notmuch search folder:'""' > EXPECTED
+notmuch search --query-syntax=sexp '(folder "")'  > OUTPUT
+test_expect_equal_file EXPECTED OUTPUT
+
+test_begin_subtest "Search by 'id'"
+add_message '[subject]="search by id"' '[date]="Sat, 01 Jan 2000 12:00:00 -0000"'
+output=$(notmuch search --query-syntax=sexp "(id ${gen_msg_id})" | notmuch_search_sanitize)
+test_expect_equal "$output" "thread:XXX   2000-01-01 [1/1] Notmuch Test Suite; search by id (inbox unread)"
+
+test_begin_subtest "Search by 'id' (or)"
+add_message '[subject]="search by id"' '[date]="Sat, 01 Jan 2000 12:00:00 -0000"'
+output=$(notmuch search --query-syntax=sexp "(id non-existent-mid ${gen_msg_id})" | notmuch_search_sanitize)
+test_expect_equal "$output" "thread:XXX   2000-01-01 [1/1] Notmuch Test Suite; search by id (inbox unread)"
+
+test_begin_subtest "Search by 'is' (multiple)"
+notmuch tag -inbox tag:searchbytag
+notmuch search is:inbox AND is:unread | notmuch_search_sanitize > EXPECTED
+notmuch search --query-syntax=sexp '(is inbox unread)' | notmuch_search_sanitize > OUTPUT
+notmuch tag +inbox tag:searchbytag
+test_expect_equal_file EXPECTED OUTPUT
+
+test_begin_subtest "Search by 'mid'"
+add_message '[subject]="search by mid"' '[date]="Sat, 01 Jan 2000 12:00:00 -0000"'
+output=$(notmuch search --query-syntax=sexp "(mid ${gen_msg_id})" | notmuch_search_sanitize)
+test_expect_equal "$output" "thread:XXX   2000-01-01 [1/1] Notmuch Test Suite; search by mid (inbox unread)"
+
+test_begin_subtest "Search by 'mid' (or)"
+add_message '[subject]="search by mid"' '[date]="Sat, 01 Jan 2000 12:00:00 -0000"'
+output=$(notmuch search --query-syntax=sexp "(mid non-existent-mid ${gen_msg_id})" | notmuch_search_sanitize)
+test_expect_equal "$output" "thread:XXX   2000-01-01 [1/1] Notmuch Test Suite; search by mid (inbox unread)"
+
+test_begin_subtest "Search by 'mimetype'"
+notmuch search mimetype:text/html > EXPECTED
+notmuch search --query-syntax=sexp '(mimetype text html)'  > OUTPUT
+test_expect_equal_file EXPECTED OUTPUT
+
+test_begin_subtest "Search by 'subject'"
 add_message [subject]=subjectsearchtest '[date]="Sat, 01 Jan 2000 12:00:00 -0000"'
 output=$(notmuch search --query-syntax=sexp '(subject subjectsearchtest)' | notmuch_search_sanitize)
 test_expect_equal "$output" "thread:XXX   2000-01-01 [1/1] Notmuch Test Suite; subjectsearchtest (inbox unread)"
 
-test_begin_subtest "Search by subject (case insensitive)"
+test_begin_subtest "Search by 'subject' (case insensitive)"
 notmuch search tag:inbox and subject:maildir | notmuch_search_sanitize > EXPECTED
 notmuch search --query-syntax=sexp '(subject Maildir)' | notmuch_search_sanitize > OUTPUT
 test_expect_equal_file EXPECTED OUTPUT
 
-test_begin_subtest "Search by subject (utf-8):"
+test_begin_subtest "Search by 'subject' (utf-8):"
 add_message [subject]=utf8-sübjéct '[date]="Sat, 01 Jan 2000 12:00:00 -0000"'
 output=$(notmuch search --query-syntax=sexp '(subject utf8 sübjéct)' | notmuch_search_sanitize)
 test_expect_equal "$output" "thread:XXX   2000-01-01 [1/1] Notmuch Test Suite; utf8-sübjéct (inbox unread)"
@@ -42,4 +135,48 @@ test_begin_subtest "Search by 'subject' (utf-8, quoted string):"
 output=$(notmuch search --query-syntax=sexp '(subject "utf8 sübjéct")' | notmuch_search_sanitize)
 test_expect_equal "$output" "thread:XXX   2000-01-01 [1/1] Notmuch Test Suite; utf8-sübjéct (inbox unread)"
 
+test_begin_subtest "Search by 'tag'"
+add_message '[subject]="search by tag"' '[date]="Sat, 01 Jan 2000 12:00:00 -0000"'
+notmuch tag +searchbytag id:${gen_msg_id}
+output=$(notmuch search --query-syntax=sexp '(tag searchbytag)' | notmuch_search_sanitize)
+test_expect_equal "$output" "thread:XXX   2000-01-01 [1/1] Notmuch Test Suite; search by tag (inbox searchbytag unread)"
+
+test_begin_subtest "Search by 'tag' (multiple)"
+notmuch tag -inbox tag:searchbytag
+notmuch search tag:inbox AND tag:unread | notmuch_search_sanitize > EXPECTED
+notmuch search --query-syntax=sexp '(tag inbox unread)' | notmuch_search_sanitize > OUTPUT
+notmuch tag +inbox tag:searchbytag
+test_expect_equal_file EXPECTED OUTPUT
+
+test_begin_subtest "Search by 'tag' and 'subject'"
+notmuch search tag:inbox and subject:maildir | notmuch_search_sanitize > EXPECTED
+notmuch search --query-syntax=sexp '(and (tag inbox) (subject maildir))' | notmuch_search_sanitize > OUTPUT
+test_expect_equal_file EXPECTED OUTPUT
+
+test_begin_subtest "Search by 'thread'"
+add_message '[subject]="search by thread"' '[date]="Sat, 01 Jan 2000 12:00:00 -0000"'
+thread_id=$(notmuch search id:${gen_msg_id} | sed -e "s/thread:\([a-f0-9]*\).*/\1/")
+output=$(notmuch search --query-syntax=sexp "(thread ${thread_id})" | notmuch_search_sanitize)
+test_expect_equal "$output" "thread:XXX   2000-01-01 [1/1] Notmuch Test Suite; search by thread (inbox unread)"
+
+test_begin_subtest "Search by 'to'"
+add_message '[subject]="search by to"' '[date]="Sat, 01 Jan 2000 12:00:00 -0000"' [to]=searchbyto
+output=$(notmuch search --query-syntax=sexp '(to searchbyto)' | notmuch_search_sanitize)
+test_expect_equal "$output" "thread:XXX   2000-01-01 [1/1] Notmuch Test Suite; search by to (inbox unread)"
+
+test_begin_subtest "Search by 'to' (address)"
+add_message '[subject]="search by to (address)"' '[date]="Sat, 01 Jan 2000 12:00:00 -0000"' [to]=searchbyto@example.com
+output=$(notmuch search --query-syntax=sexp '(to searchbyto@example.com)' | notmuch_search_sanitize)
+test_expect_equal "$output" "thread:XXX   2000-01-01 [1/1] Notmuch Test Suite; search by to (address) (inbox unread)"
+
+test_begin_subtest "Search by 'to' (name)"
+add_message '[subject]="search by to (name)"' '[date]="Sat, 01 Jan 2000 12:00:00 -0000"' '[to]="Search By To Name <test@example.com>"'
+output=$(notmuch search --query-syntax=sexp '(to "Search By To Name")' | notmuch_search_sanitize)
+test_expect_equal "$output" "thread:XXX   2000-01-01 [1/1] Notmuch Test Suite; search by to (name) (inbox unread)"
+
+test_begin_subtest "Search by 'to' (name and address)"
+output=$(notmuch search --query-syntax=sexp '(to "Search By To Name <test@example.com>")' | notmuch_search_sanitize)
+test_expect_equal "$output" "thread:XXX   2000-01-01 [1/1] Notmuch Test Suite; search by to (name) (inbox unread)"
+
+
 test_done
-- 
2.30.2\r

[PATCH 09/11] lib/parse-sexp: add error handling to internal API

[David Bremner]

The 'notmuch' argument is currently only used for logging in one
place, but this will most likely change in the future. Furthermore, it
will be needed for e.g. stored queries.
---
 lib/parse-sexp.cc         | 75 +++++++++++++++++++++++++--------------
 lib/query.cc              |  5 +--
 test/T081-sexpr-search.sh | 12 ++++++-
 3 files changed, 61 insertions(+), 31 deletions(-)

diff --git a/lib/parse-sexp.cc b/lib/parse-sexp.cc
index 81bf5cee..4a9e1aeb 100644
--- a/lib/parse-sexp.cc
+++ b/lib/parse-sexp.cc
@@ -40,40 +40,55 @@ static _sexp_field_t fields[] =
     { }
 };
 
-static Xapian::Query _sexp_to_xapian_query (sexp_t *sx);
+static notmuch_status_t _sexp_to_xapian_query (notmuch_database_t *notmuch, sexp_t *sx, Xapian::Query &output);
 
-static Xapian::Query
-_sexp_combine_query (Xapian::Query::op operation,
+static notmuch_status_t
+_sexp_combine_query (notmuch_database_t *notmuch,
+		     Xapian::Query::op operation,
 		     Xapian::Query left,
-		     sexp_t *sx)
+		     sexp_t *sx,
+		     Xapian::Query &output)
 {
+    Xapian::Query subquery;
+
+    notmuch_status_t status;
 
     /* if we run out elements, return accumulator */
 
-    if (! sx)
-	return left;
+    if (! sx) {
+	output = left;
+	return NOTMUCH_STATUS_SUCCESS;
+    }
 
-    return _sexp_combine_query (operation,
-				Xapian::Query (operation,
-					       left,
-					       _sexp_to_xapian_query (sx)),
-				sx->next);
+    status = _sexp_to_xapian_query (notmuch, sx, subquery);
+    if (status)
+	return status;
+
+    return _sexp_combine_query (notmuch,
+				operation,
+				Xapian::Query (operation, left, subquery),
+				sx->next, output);
 }
 
-Xapian::Query
-_notmuch_sexp_string_to_xapian_query (notmuch_database_t *notmuch, const char *querystr)
+notmuch_status_t
+_notmuch_sexp_string_to_xapian_query (notmuch_database_t *notmuch, const char *querystr,
+				      Xapian::Query &output)
 {
     sexp_t *sx = NULL;
     char *buf = talloc_strdup (notmuch, querystr);
 
     sx = parse_sexp (buf, strlen (querystr));
-    return _sexp_to_xapian_query (sx);
+    if (!sx)
+	return NOTMUCH_STATUS_ILLEGAL_ARGUMENT;
+
+    return _sexp_to_xapian_query (notmuch, sx, output);
 }
 
-static Xapian::Query
+static notmuch_status_t
 _sexp_combine_field (const char *prefix,
 		     Xapian::Query::op operation,
-		     sexp_t *sx)
+		     sexp_t *sx,
+		     Xapian::Query &output)
 {
     std::vector<std::string> terms;
 
@@ -107,15 +122,16 @@ _sexp_combine_field (const char *prefix,
 	    terms.push_back (pref_str + cur->val);
 	}
     }
-    return Xapian::Query (operation, terms.begin (), terms.end ());
+    output = Xapian::Query (operation, terms.begin (), terms.end ());
+    return NOTMUCH_STATUS_SUCCESS;
 }
 
 /* Here we expect the s-expression to be a proper list, with first
  * element defining and operation, or as a special case the empty
  * list */
 
-static Xapian::Query
-_sexp_to_xapian_query (sexp_t *sx)
+static notmuch_status_t
+_sexp_to_xapian_query (notmuch_database_t *notmuch, sexp_t *sx, Xapian::Query &output)
 {
 
     const _sexp_op_t *op;
@@ -124,18 +140,25 @@ _sexp_to_xapian_query (sexp_t *sx)
     assert (sx->ty == SEXP_LIST);
 
     /* Empty list */
-    if (! sx->list)
-	return Xapian::Query::MatchAll;
+    if (! sx->list) {
+	output = Xapian::Query::MatchAll;
+	return NOTMUCH_STATUS_SUCCESS;
+    }
 
     for (op = operations; op && op->name; op++) {
-	if (strcasecmp (op->name, hd_sexp (sx)->val) == 0)
-	    return _sexp_combine_query (op->xapian_op, op->initial, sx->list->next);
+	if (strcasecmp (op->name, hd_sexp (sx)->val) == 0) {
+	    return _sexp_combine_query (notmuch, op->xapian_op, op->initial, sx->list->next, output);
+	}
+
     }
 
     for (const _sexp_field_t *field = fields; field && field->name; field++) {
-	if (strcasecmp (field->name, hd_sexp (sx)->val) == 0)
-	    return _sexp_combine_field (_find_prefix (field->name), field->xapian_op, sx->list->next);
+	if (strcasecmp (field->name, hd_sexp (sx)->val) == 0) {
+	    return _sexp_combine_field (_find_prefix (field->name), field->xapian_op, sx->list->next,
+					output);
+	}
     }
 
-    INTERNAL_ERROR ("unimplemented prefix %s\n", sx->list->val);
+    _notmuch_database_log_append (notmuch, "unimplemented prefix %s\n", sx->list->val);
+    return NOTMUCH_STATUS_ILLEGAL_ARGUMENT;
 }
diff --git a/lib/query.cc b/lib/query.cc
index 6e89af60..4b42620e 100644
--- a/lib/query.cc
+++ b/lib/query.cc
@@ -191,10 +191,7 @@ _notmuch_query_ensure_parsed_xapian (notmuch_query_t *query)
 static notmuch_status_t
 _notmuch_query_ensure_parsed_sexpr (notmuch_query_t *query)
 {
-
-    query->xapian_query = _notmuch_sexp_string_to_xapian_query (query->notmuch, query->query_string);
-
-    return NOTMUCH_STATUS_SUCCESS;
+    return _notmuch_sexp_string_to_xapian_query (query->notmuch, query->query_string, query->xapian_query);
 }
 
 static notmuch_status_t
diff --git a/test/T081-sexpr-search.sh b/test/T081-sexpr-search.sh
index 9ee9de7b..b8229ebe 100755
--- a/test/T081-sexpr-search.sh
+++ b/test/T081-sexpr-search.sh
@@ -178,5 +178,15 @@ test_begin_subtest "Search by 'to' (name and address)"
 output=$(notmuch search --query-syntax=sexp '(to "Search By To Name <test@example.com>")' | notmuch_search_sanitize)
 test_expect_equal "$output" "thread:XXX   2000-01-01 [1/1] Notmuch Test Suite; search by to (name) (inbox unread)"
 
-
+test_begin_subtest "Unbalanced parens"
+# A code 1 indicates the error was handled (a crash will return e.g. 139).
+test_expect_code 1 "notmuch search --query-syntax=sexp '('"
+
+test_begin_subtest "unknown_prefix"
+notmuch search --query-syntax=sexp '(foo)' >OUTPUT 2>&1
+cat <<EOF > EXPECTED
+notmuch search: Illegal argument for function
+unimplemented prefix foo
+EOF
+test_expect_equal_file EXPECTED OUTPUT
 test_done
-- 
2.30.2

[PATCH 10/11] lib/parse-sexp: add keyword arguments for fields

[David Bremner]

Nothing is done with the flags here, only parsing and validation.
---
 lib/parse-sexp.cc         | 114 +++++++++++++++++++++++++++++++-------
 test/T081-sexpr-search.sh |  32 +++++++++++
 2 files changed, 126 insertions(+), 20 deletions(-)

diff --git a/lib/parse-sexp.cc b/lib/parse-sexp.cc
index 4a9e1aeb..bfa566b1 100644
--- a/lib/parse-sexp.cc
+++ b/lib/parse-sexp.cc
@@ -8,6 +8,34 @@ typedef struct  {
     Xapian::Query initial;
 } _sexp_op_t;
 
+typedef enum {
+    SEXP_FLAG_NONE	= 0,
+    SEXP_FLAG_WILDCARD	= 1 << 0,
+} _sexp_flag_t;
+
+/*
+ * define bitwise operators to hide casts */
+
+inline _sexp_flag_t
+operator| (_sexp_flag_t a, _sexp_flag_t b)
+{
+    return static_cast<_sexp_flag_t>(
+	static_cast<unsigned>(a) | static_cast<unsigned>(b));
+}
+
+inline _sexp_flag_t
+operator& (_sexp_flag_t a, _sexp_flag_t b)
+{
+    return static_cast<_sexp_flag_t>(
+	static_cast<unsigned>(a) & static_cast<unsigned>(b));
+}
+
+typedef struct  {
+    const char *name;
+    Xapian::Query::op xapian_op;
+    _sexp_flag_t flags_allowed;
+} _sexp_field_t;
+
 static _sexp_op_t operations[] =
 {
     { "and",    Xapian::Query::OP_AND,          Xapian::Query::MatchAll },
@@ -16,27 +44,34 @@ static _sexp_op_t operations[] =
     { }
 };
 
-typedef struct  {
+static _sexp_field_t fields[] =
+{
+    { "attachment",   Xapian::Query::OP_PHRASE,       SEXP_FLAG_WILDCARD },
+    { "body",         Xapian::Query::OP_PHRASE,       SEXP_FLAG_WILDCARD },
+    { "from",         Xapian::Query::OP_PHRASE,       SEXP_FLAG_NONE },
+    { "folder",       Xapian::Query::OP_OR,           SEXP_FLAG_NONE },
+    { "id",           Xapian::Query::OP_OR,           SEXP_FLAG_NONE },
+    { "is",           Xapian::Query::OP_AND,          SEXP_FLAG_WILDCARD },
+    { "mid",          Xapian::Query::OP_OR,           SEXP_FLAG_NONE },
+    { "mimetype",     Xapian::Query::OP_PHRASE,       SEXP_FLAG_NONE },
+    { "path",         Xapian::Query::OP_OR,           SEXP_FLAG_NONE },
+    { "property",     Xapian::Query::OP_AND,          SEXP_FLAG_WILDCARD },
+    { "subject",      Xapian::Query::OP_PHRASE,       SEXP_FLAG_NONE },
+    { "tag",          Xapian::Query::OP_AND,          SEXP_FLAG_WILDCARD },
+    { "thread",       Xapian::Query::OP_OR,           SEXP_FLAG_NONE },
+    { "to",           Xapian::Query::OP_PHRASE,       SEXP_FLAG_NONE },
+    { }
+};
+
+typedef struct {
     const char *name;
-    Xapian::Query::op xapian_op;
-} _sexp_field_t;
+    _sexp_flag_t flag;
+} _sexp_keyword_t;
 
-static _sexp_field_t fields[] =
+static _sexp_keyword_t keywords[] =
 {
-    { "attachment",     Xapian::Query::OP_PHRASE },
-    { "body",           Xapian::Query::OP_PHRASE },
-    { "from",           Xapian::Query::OP_PHRASE },
-    { "folder",         Xapian::Query::OP_OR },
-    { "id",             Xapian::Query::OP_OR },
-    { "is",             Xapian::Query::OP_AND },
-    { "mid",            Xapian::Query::OP_OR },
-    { "mimetype",       Xapian::Query::OP_PHRASE },
-    { "path",           Xapian::Query::OP_OR },
-    { "property",       Xapian::Query::OP_AND },
-    { "subject",        Xapian::Query::OP_PHRASE },
-    { "tag",            Xapian::Query::OP_AND },
-    { "thread",         Xapian::Query::OP_OR },
-    { "to",             Xapian::Query::OP_PHRASE },
+    { "any", SEXP_FLAG_WILDCARD },
+    { "*", SEXP_FLAG_WILDCARD },
     { }
 };
 
@@ -126,6 +161,37 @@ _sexp_combine_field (const char *prefix,
     return NOTMUCH_STATUS_SUCCESS;
 }
 
+
+static notmuch_status_t
+_sexp_parse_keywords (notmuch_database_t *notmuch, const char *prefix, sexp_t *sx, _sexp_flag_t mask,
+		      _sexp_flag_t &flags, sexp_t *&rest)
+{
+    flags = SEXP_FLAG_NONE;
+
+    for (; sx && sx->ty == SEXP_VALUE && sx->aty == SEXP_BASIC && sx->val[0] == ':'; sx = sx->next) {
+	_sexp_keyword_t *keyword;
+	for (keyword = keywords; keyword && keyword->name; keyword++) {
+	    if (strcasecmp (keyword->name, sx->val+1) == 0) {
+		if ((mask & keyword->flag) == 0) {
+		    _notmuch_database_log_append (notmuch, "unsupported keyword '%s' for prefix '%s'\n",
+						  sx->val, prefix);
+		    return NOTMUCH_STATUS_ILLEGAL_ARGUMENT;
+		} else {
+		    flags = flags | keyword->flag;
+		    break;
+		}
+	    }
+	}
+	if (! keyword || ! keyword->name) {
+	    _notmuch_database_log (notmuch, "unknown keyword %s\n", sx->val);
+	    return NOTMUCH_STATUS_ILLEGAL_ARGUMENT;
+	}
+    }
+
+    rest = sx;
+    return NOTMUCH_STATUS_SUCCESS;
+}
+
 /* Here we expect the s-expression to be a proper list, with first
  * element defining and operation, or as a special case the empty
  * list */
@@ -149,12 +215,20 @@ _sexp_to_xapian_query (notmuch_database_t *notmuch, sexp_t *sx, Xapian::Query &o
 	if (strcasecmp (op->name, hd_sexp (sx)->val) == 0) {
 	    return _sexp_combine_query (notmuch, op->xapian_op, op->initial, sx->list->next, output);
 	}
-
     }
 
     for (const _sexp_field_t *field = fields; field && field->name; field++) {
 	if (strcasecmp (field->name, hd_sexp (sx)->val) == 0) {
-	    return _sexp_combine_field (_find_prefix (field->name), field->xapian_op, sx->list->next,
+	    _sexp_flag_t flags;
+	    sexp_t *rest;
+
+	    notmuch_status_t status = _sexp_parse_keywords (notmuch, sx->list->val,
+							    sx->list->next, field->flags_allowed,
+							    flags, rest);
+	    if (status)
+		return status;
+
+	    return _sexp_combine_field (_find_prefix (field->name), field->xapian_op, rest,
 					output);
 	}
     }
diff --git a/test/T081-sexpr-search.sh b/test/T081-sexpr-search.sh
index b8229ebe..6e8687da 100755
--- a/test/T081-sexpr-search.sh
+++ b/test/T081-sexpr-search.sh
@@ -189,4 +189,36 @@ notmuch search: Illegal argument for function
 unimplemented prefix foo
 EOF
 test_expect_equal_file EXPECTED OUTPUT
+
+test_begin_subtest "unknown keyword"
+notmuch search --query-syntax=sexp '(subject :foo)' >OUTPUT 2>&1
+cat <<EOF > EXPECTED
+notmuch search: Illegal argument for function
+unknown keyword :foo
+EOF
+test_expect_equal_file EXPECTED OUTPUT
+
+test_begin_subtest "unsupported keyword for prefix"
+notmuch search --query-syntax=sexp '(subject :any)' >OUTPUT 2>&1
+cat <<EOF > EXPECTED
+notmuch search: Illegal argument for function
+unsupported keyword ':any' for prefix 'subject'
+EOF
+test_expect_equal_file EXPECTED OUTPUT
+
+test_begin_subtest "known keyword :any"
+test_expect_success 'notmuch search --query-syntax=sexp "(tag :any)"'
+
+test_begin_subtest "known keyword :*"
+test_expect_success 'notmuch search --query-syntax=sexp "(tag :*)"'
+
+test_begin_subtest "multiple known keywords"
+test_expect_success 'notmuch search --query-syntax=sexp "(tag :any :*)"'
+
+test_begin_subtest "quoted unknown keyword"
+test_expect_success 'notmuch search --query-syntax=sexp "(subject \":foo\")"'
+
+test_begin_subtest "unknown keyword after non-keyword"
+test_expect_success 'notmuch search --query-syntax=sexp "(subject foo :foo)"'
+
 test_done
-- 
2.30.2

[PATCH 11/11] lib/parse-sexp: initial support for wildcard queries

[David Bremner]

Because of the implicit way body: queries are implemented, additional
code will be needed for them.
---
 lib/parse-sexp.cc         | 12 +++++++--
 test/T081-sexpr-search.sh | 54 +++++++++++++++++++++++++++++++++++++++
 2 files changed, 64 insertions(+), 2 deletions(-)

diff --git a/lib/parse-sexp.cc b/lib/parse-sexp.cc
index bfa566b1..a0f5e5bf 100644
--- a/lib/parse-sexp.cc
+++ b/lib/parse-sexp.cc
@@ -221,6 +221,7 @@ _sexp_to_xapian_query (notmuch_database_t *notmuch, sexp_t *sx, Xapian::Query &o
 	if (strcasecmp (field->name, hd_sexp (sx)->val) == 0) {
 	    _sexp_flag_t flags;
 	    sexp_t *rest;
+	    const char *term_prefix;
 
 	    notmuch_status_t status = _sexp_parse_keywords (notmuch, sx->list->val,
 							    sx->list->next, field->flags_allowed,
@@ -228,8 +229,15 @@ _sexp_to_xapian_query (notmuch_database_t *notmuch, sexp_t *sx, Xapian::Query &o
 	    if (status)
 		return status;
 
-	    return _sexp_combine_field (_find_prefix (field->name), field->xapian_op, rest,
-					output);
+	    term_prefix = _find_prefix (field->name);
+
+	    if (flags & SEXP_FLAG_WILDCARD) {
+		output = Xapian::Query (Xapian::Query::OP_WILDCARD, term_prefix);
+		return NOTMUCH_STATUS_SUCCESS;
+	    } else {
+		return _sexp_combine_field (term_prefix, field->xapian_op, rest,
+					    output);
+	    }
 	}
     }
 
diff --git a/test/T081-sexpr-search.sh b/test/T081-sexpr-search.sh
index 6e8687da..d8a89c6f 100755
--- a/test/T081-sexpr-search.sh
+++ b/test/T081-sexpr-search.sh
@@ -221,4 +221,58 @@ test_expect_success 'notmuch search --query-syntax=sexp "(subject \":foo\")"'
 test_begin_subtest "unknown keyword after non-keyword"
 test_expect_success 'notmuch search --query-syntax=sexp "(subject foo :foo)"'
 
+test_begin_subtest "wildcard search for attachment"
+notmuch search tag:attachment > EXPECTED
+notmuch search --query-syntax=sexp '(attachment :*)' > OUTPUT
+test_expect_equal_file EXPECTED OUTPUT
+
+add_message '[subject]="empty body"' '[body]="."'
+notmuch tag +nobody id:${gen_msg_id}
+
+test_begin_subtest "wildcard search for body"
+test_subtest_known_broken
+notmuch search not tag:nobody > EXPECTED
+notmuch search --query-syntax=sexp '(body :any)' > OUTPUT
+test_expect_equal_file EXPECTED OUTPUT
+
+test_begin_subtest "negated wildcard search for body"
+test_subtest_known_broken
+notmuch search tag:nobody > EXPECTED
+notmuch search --query-syntax=sexp '(not (body :any))' > OUTPUT
+test_expect_equal_file EXPECTED OUTPUT
+
+add_message '[subject]="no tags"'
+notag_mid=${gen_msg_id}
+notmuch tag -unread -inbox id:${notag_mid}
+
+test_begin_subtest "wildcard search for 'is'"
+notmuch search not id:${notag_mid} > EXPECTED
+notmuch search --query-syntax=sexp '(is :any)' > OUTPUT
+test_expect_equal_file EXPECTED OUTPUT
+
+test_begin_subtest "negated wildcard search for 'is'"
+notmuch search id:${notag_mid} > EXPECTED
+notmuch search --query-syntax=sexp '(not (is :any))' > OUTPUT
+test_expect_equal_file EXPECTED OUTPUT
+
+test_begin_subtest "wildcard search for 'tag'"
+notmuch search not id:${notag_mid} > EXPECTED
+notmuch search --query-syntax=sexp '(tag :any)' > OUTPUT
+test_expect_equal_file EXPECTED OUTPUT
+
+test_begin_subtest "negated wildcard search for 'tag'"
+notmuch search id:${notag_mid} > EXPECTED
+notmuch search --query-syntax=sexp '(not (tag :any))' > OUTPUT
+test_expect_equal_file EXPECTED OUTPUT
+
+add_message '[subject]="message with properties"'
+notmuch restore <<EOF
+#= ${gen_msg_id} foo=bar
+EOF
+
+test_begin_subtest "wildcard search for 'property'"
+notmuch search property:foo=bar > EXPECTED
+notmuch search --query-syntax=sexp '(property :any)' > OUTPUT
+test_expect_equal_file EXPECTED OUTPUT
+
 test_done
-- 
2.30.2

Re: Early preview of s-expression based query parser

[Hannu Hartikainen]

On Tue, 13 Jul 2021 21:02:28 -0300, David Bremner <david@tethera.net> wrote:
> Feedback of any kind is welcome, but particularly on UI / UX
> issues. You can get a pretty good idea of the supported syntax by
> looking at the tests.

I read through the commits and it looks good to me implementation-wise.

I'm probably not the most experienced lisper around so take my ideas
with a grain of salt. I did work professionally on a Clojure codebase
for a couple of years and I've written a bunch of small programs in
Racket but that's pretty much it.

But looking at the sexp parser and the implementation of logical
connectors I can't help but think that isn't this patchset implementing
a tiny subset of a lisp? And wouldn't a full embedded lisp be much, much
more powerful?

I'd at least consider embedding something like s7 [0] or Janet [1],
writing bindings for enough Xapian functionality, and then writing the
rest in the lisp itself. That way you'd get a more powerful and
extensible sexp implementation, and you'd implement most of it in a much
more ergonomic language.

Of course I don't really know Xapian and I'm not sure of the design
goals of this sexp parser, but my experience with HoneySQL [2] tells me
that building queries with a lisp from lisp data structures can be
unbelievably powerful.

Hannu

[0]: https://ccrma.stanford.edu/software/snd/snd/s7.html
[1]: https://janet-lang.org/
[2]: https://github.com/seancorfield/honeysql

Re: Early preview of s-expression based query parser

[David Bremner]

Hannu Hartikainen <hannu.hartikainen@gmail.com> writes:

>
> But looking at the sexp parser and the implementation of logical
> connectors I can't help but think that isn't this patchset implementing
> a tiny subset of a lisp? And wouldn't a full embedded lisp be much, much
> more powerful?

I think the analogy here is between JSON, and JavaScript. Plenty of
programs find it useful to use JSON as an input format without embedding
JavaScript. That's not just because JavaScript is not everyone's
favourite programming language, but because a Turing complete input
language is a mixed blessing.

> I'd at least consider embedding something like s7 [0] or Janet [1],
> writing bindings for enough Xapian functionality, and then writing the
> rest in the lisp itself. That way you'd get a more powerful and
> extensible sexp implementation, and you'd implement most of it in a much
> more ergonomic language.

Since what I want to do is actually a fairly thin layer on top of the
Xapian API, it's not clear that writing the bindings would be less code
than the roughly 450 lines of C/C++ needed to parse the s-expressions.

We already face challenges maintaining bindings for Python and Ruby; I
suspect that finding people to work on bindings for fairly obscure
(defined by not being in Debian) lisp variants would not be easier.
Ditto for coding in said lisp in the notmuch core.

> Of course I don't really know Xapian and I'm not sure of the design
> goals of this sexp parser, but my experience with HoneySQL [2] tells me
> that building queries with a lisp from lisp data structures can be
> unbelievably powerful.

Sure, and one of the motivations is to be able to conveniently build
queries in emacs lisp (some other lisp would obviously work just as
well).

If someone wants to take on a more ambitious project involving a full
lisp, that probably makes the most sense as a client project that links
to notmuch. That leaves someone free to explore without waiting on our
somewhat ponderous development cycle.

d