v2 allow sexp queries in infix parser

v2 allow sexp queries in infix parser

[David Bremner]

This is a pretty useful feature, so I want to include it 0.36. That
means you have a bit less than a week to give me feedback.

Speaking of feedback, I have no idea why the "and of stemmed terms"
test was included in the first version of this patch. I have split it
out to it's own patch.

[PATCH v2 1/2] test/sexp: add test for and of stemmed terms.

[David Bremner]

Previously only singled stemmed terms were tested.
---
 test/T081-sexpr-search.sh | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/test/T081-sexpr-search.sh b/test/T081-sexpr-search.sh
index e2936cd7..07b12619 100755
--- a/test/T081-sexpr-search.sh
+++ b/test/T081-sexpr-search.sh
@@ -31,6 +31,13 @@ thread:XXX   2009-11-18 [1/3] Carl Worth| Jan Janak; [notmuch] What a great idea
 EOF
 test_expect_equal_file EXPECTED OUTPUT
 
+test_begin_subtest "and of stemmed terms"
+notmuch search --query=sexp '(and wonderful wizard)' | notmuch_search_sanitize > OUTPUT
+cat <<EOF > EXPECTED
+thread:XXX   2009-11-18 [1/3] Carl Worth| Jan Janak; [notmuch] What a great idea! (inbox unread)
+EOF
+test_expect_equal_file EXPECTED OUTPUT
+
 test_begin_subtest "or of exact terms"
 notmuch search --query=sexp '(or "php" "wizard")' | notmuch_search_sanitize > OUTPUT
 cat <<EOF > EXPECTED
-- 
2.35.1

[PATCH v2 2/2] lib: add sexp: prefix to Xapian (infix) query parser.

[David Bremner]

This is analogous to the "infix" prefix provided by the s-expression
based query parser.
---
 doc/man7/notmuch-search-terms.rst | 17 ++++++++++++-
 lib/Makefile.local                |  3 ++-
 lib/prefix.cc                     |  5 ++++
 lib/sexp-fp.cc                    | 40 ++++++++++++++++++++++++++++++
 lib/sexp-fp.h                     | 41 +++++++++++++++++++++++++++++++
 test/T081-sexpr-search.sh         | 25 +++++++++++++++++++
 6 files changed, 129 insertions(+), 2 deletions(-)
 create mode 100644 lib/sexp-fp.cc
 create mode 100644 lib/sexp-fp.h

diff --git a/doc/man7/notmuch-search-terms.rst b/doc/man7/notmuch-search-terms.rst
index f8ad1edb..4f616b7e 100644
--- a/doc/man7/notmuch-search-terms.rst
+++ b/doc/man7/notmuch-search-terms.rst
@@ -169,6 +169,12 @@ property:<key>=<value>
     can be present on a given message with several different values.
     See :any:`notmuch-properties(7)` for more details.
 
+sexp:<subquery>
+    The **sexp:** prefix allows subqueries in the format
+    documented in :any:`notmuch-sexp-queries(7)`. Note that subqueries containing
+    spaces must be quoted, and any embedded double quotes must be escaped
+    (see :any:`quoting`).
+
 User defined prefixes are also supported, see :any:`notmuch-config(1)` for
 details.
 
@@ -257,7 +263,7 @@ Boolean
 Probabilistic
   **body:**, **to:**, **attachment:**, **mimetype:**
 Special
-   **from:**, **query:**, **subject:**
+   **from:**, **query:**, **subject:**, **sexp:**
 
 Terms and phrases
 -----------------
@@ -297,6 +303,8 @@ Both of these will match a subject "Free Delicious Pizza" while
 
 will not.
 
+.. _quoting:
+
 Quoting
 -------
 
@@ -324,6 +332,13 @@ e.g.
    % notmuch search 'folder:"/^.*/(Junk|Spam)$/"'
    % notmuch search 'thread:"{from:mallory and date:2009}" and thread:{to:mallory}'
 
+Double quotes within query strings need to be doubled to escape them.
+
+::
+
+   % notmuch search 'tag:"""quoted tag"""'
+   % notmuch search 'sexp:"(or ""wizard"" ""php"")"'
+
 DATE AND TIME SEARCH
 ====================
 
diff --git a/lib/Makefile.local b/lib/Makefile.local
index 1378a74b..6d67a2a4 100644
--- a/lib/Makefile.local
+++ b/lib/Makefile.local
@@ -64,7 +64,8 @@ libnotmuch_cxx_srcs =		\
 	$(dir)/prefix.cc	\
 	$(dir)/open.cc		\
 	$(dir)/init.cc		\
-	$(dir)/parse-sexp.cc
+	$(dir)/parse-sexp.cc	\
+	$(dir)/sexp-fp.cc
 
 libnotmuch_modules := $(libnotmuch_c_srcs:.c=.o) $(libnotmuch_cxx_srcs:.cc=.o)
 
diff --git a/lib/prefix.cc b/lib/prefix.cc
index 857c05b9..06e2333a 100644
--- a/lib/prefix.cc
+++ b/lib/prefix.cc
@@ -3,6 +3,7 @@
 #include "thread-fp.h"
 #include "regexp-fields.h"
 #include "parse-time-vrp.h"
+#include "sexp-fp.h"
 
 typedef struct {
     const char *name;
@@ -60,6 +61,8 @@ prefix_t prefix_table[] = {
       NOTMUCH_FIELD_PROCESSOR },
     { "query",                  NULL,           NOTMUCH_FIELD_EXTERNAL |
       NOTMUCH_FIELD_PROCESSOR },
+    { "sexp",                  NULL,            NOTMUCH_FIELD_EXTERNAL |
+      NOTMUCH_FIELD_PROCESSOR },
     { "from",                   "XFROM",        NOTMUCH_FIELD_EXTERNAL |
       NOTMUCH_FIELD_PROBABILISTIC |
       NOTMUCH_FIELD_PROCESSOR },
@@ -138,6 +141,8 @@ _setup_query_field (const prefix_t *prefix, notmuch_database_t *notmuch)
 	    fp = (new QueryFieldProcessor (*notmuch->query_parser, notmuch))->release ();
 	else if (STRNCMP_LITERAL (prefix->name, "thread") == 0)
 	    fp = (new ThreadFieldProcessor (*notmuch->query_parser, notmuch))->release ();
+	else if (STRNCMP_LITERAL (prefix->name, "sexp") == 0)
+	    fp = (new SexpFieldProcessor (notmuch))->release ();
 	else
 	    fp = (new RegexpFieldProcessor (prefix->name, prefix->flags,
 					    *notmuch->query_parser, notmuch))->release ();
diff --git a/lib/sexp-fp.cc b/lib/sexp-fp.cc
new file mode 100644
index 00000000..ed26f6ec
--- /dev/null
+++ b/lib/sexp-fp.cc
@@ -0,0 +1,40 @@
+/* sexp-fp.cc - "sexp:" field processor glue
+ *
+ * This file is part of notmuch.
+ *
+ * Copyright © 2022 David Bremner
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation, either version 3 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see https://www.gnu.org/licenses/ .
+ *
+ * Author: David Bremner <david@tethera.net>
+ */
+
+#include "database-private.h"
+#include "sexp-fp.h"
+#include <iostream>
+
+Xapian::Query
+SexpFieldProcessor::operator() (const std::string & query_string)
+{
+    notmuch_status_t status;
+    Xapian::Query output;
+
+    status = _notmuch_sexp_string_to_xapian_query (notmuch, query_string.c_str (), output);
+    if (status) {
+	throw Xapian::QueryParserError ("error parsing " + query_string);
+    }
+
+    return output;
+
+}
diff --git a/lib/sexp-fp.h b/lib/sexp-fp.h
new file mode 100644
index 00000000..341dfa7e
--- /dev/null
+++ b/lib/sexp-fp.h
@@ -0,0 +1,41 @@
+/* sexp-fp.h - sexp field processor glue
+ *
+ * This file is part of notmuch.
+ *
+ * Copyright © 2022 David Bremner
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation, either version 3 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see https://www.gnu.org/licenses/ .
+ *
+ * Author: David Bremner <david@tethera.net>
+ */
+
+#ifndef NOTMUCH_SEXP_FP_H
+#define NOTMUCH_SEXP_FP_H
+
+#include <xapian.h>
+#include "notmuch.h"
+
+class SexpFieldProcessor : public Xapian::FieldProcessor {
+protected:
+    notmuch_database_t *notmuch;
+
+public:
+    SexpFieldProcessor (notmuch_database_t *notmuch_) : notmuch (notmuch_)
+    {
+    };
+
+    Xapian::Query operator() (const std::string & query_string);
+};
+
+#endif /* NOTMUCH_SEXP_FP_H */
diff --git a/test/T081-sexpr-search.sh b/test/T081-sexpr-search.sh
index 07b12619..da819190 100755
--- a/test/T081-sexpr-search.sh
+++ b/test/T081-sexpr-search.sh
@@ -46,6 +46,14 @@ thread:XXX   2009-11-18 [1/3] Carl Worth| Jan Janak; [notmuch] What a great idea
 EOF
 test_expect_equal_file EXPECTED OUTPUT
 
+test_begin_subtest "or of exact terms via field processor"
+notmuch search  'sexp:"(or ""php"" ""wizard"")"' | notmuch_search_sanitize > OUTPUT
+cat <<EOF > EXPECTED
+thread:XXX   2010-12-29 [1/1] François Boulogne; [aur-general] Guidelines: cp, mkdir vs install (inbox unread)
+thread:XXX   2009-11-18 [1/3] Carl Worth| Jan Janak; [notmuch] What a great idea! (inbox unread)
+EOF
+test_expect_equal_file EXPECTED OUTPUT
+
 test_begin_subtest "single term in body"
 notmuch search --query=sexp 'wizard' | notmuch_search_sanitize>OUTPUT
 cat <<EOF > EXPECTED
@@ -714,6 +722,11 @@ notmuch search property:foo=bar > EXPECTED
 notmuch search --query=sexp '(property (rx foo=.*))' > OUTPUT
 test_expect_equal_file EXPECTED OUTPUT
 
+test_begin_subtest "regexp 'property' search via field processor"
+notmuch search property:foo=bar > EXPECTED
+notmuch search  'sexp:"(property (rx foo=.*))"' > OUTPUT
+test_expect_equal_file EXPECTED OUTPUT
+
 test_begin_subtest "anchored 'tag' search"
 notmuch search tag:signed > EXPECTED
 notmuch search --query=sexp '(tag (rx ^si))' > OUTPUT
@@ -750,6 +763,13 @@ thread:XXX   2009-11-18 [7/7] Lars Kellogg-Stedman, Mikhail Gusarov, Keith Packa
 EOF
 test_expect_equal_file EXPECTED OUTPUT
 
+test_begin_subtest "Compound subquery via field processor"
+notmuch search 'sexp:"(thread (of (from keithp) (subject Maildir)))"' | notmuch_search_sanitize > OUTPUT
+cat<<EOF > EXPECTED
+thread:XXX   2009-11-18 [7/7] Lars Kellogg-Stedman, Mikhail Gusarov, Keith Packard, Carl Worth; [notmuch] Working with Maildir storage? (inbox signed unread)
+EOF
+test_expect_equal_file EXPECTED OUTPUT
+
 test_begin_subtest "empty subquery"
 notmuch search --query=sexp '(thread (of))' 1>OUTPUT 2>&1
 notmuch search '*' > EXPECTED
@@ -976,6 +996,11 @@ grep -Ril List-Id ${MAIL_DIR} | sort | notmuch_dir_sanitize > EXPECTED
 notmuch search --output=files --query=sexp '(List *)' | sort | notmuch_dir_sanitize > OUTPUT
 test_expect_equal_file EXPECTED OUTPUT
 
+test_begin_subtest "wildcard search for user header via field processor"
+grep -Ril List-Id ${MAIL_DIR} | sort | notmuch_dir_sanitize > EXPECTED
+notmuch search --output=files  'sexp:"(List *)"' | sort | notmuch_dir_sanitize > OUTPUT
+test_expect_equal_file EXPECTED OUTPUT
+
 test_begin_subtest "wildcard search for user header 2"
 grep -Ril List-Id ${MAIL_DIR} | sort | notmuch_dir_sanitize > EXPECTED
 notmuch search --output=files --query=sexp '(List (starts-with not))' | sort | notmuch_dir_sanitize > OUTPUT
-- 
2.35.1
\r

Re: [PATCH v2 2/2] lib: add sexp: prefix to Xapian (infix) query parser.

[David Bremner]

David Bremner <david@tethera.net> writes:

> This is analogous to the "infix" prefix provided by the s-expression
> based query parser.

series applied to master.

Re: [PATCH v2 2/2] lib: add sexp: prefix to Xapian (infix) query parser.

[Michael J Gruber]

[-- Attachment #1.1: Type: text/plain, Size: 904 bytes --]

This breaks builds if you don't have sfexp, unfortunately. (And since there
is no sfsexp release with the recent fixes, there is no sfsexp in Fedora,
for example.)

Not declaring _notmuch_sexp_string_to_xapian_query() twice
in lib/database-private.h (once unconditionally, once depending on
HAVE_SFSEXP) may be part of the fix, as well as guarding it in
lib/query.cc. But I'm not sure what SexpFieldProcessor::operator() should
do when sfsexp is not available - throw a Xapian error?

Am Fr., 15. Apr. 2022 um 13:32 Uhr schrieb David Bremner <david@tethera.net
>:

> David Bremner <david@tethera.net> writes:
>
> > This is analogous to the "infix" prefix provided by the s-expression
> > based query parser.
>
> series applied to master.
> _______________________________________________
> notmuch mailing list -- notmuch@notmuchmail.org
> To unsubscribe send an email to notmuch-leave@notmuchmail.org
>

[-- Attachment #1.2: Type: text/html, Size: 1454 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

[PATCH] fix build without sfsexp

[michaeljgruber+grubix+git]

From: Michael J Gruber <git@grubix.eu>

a1d139de ("lib: add sexp: prefix to Xapian (infix) query parser.",
2022-04-09) introduced sfsexp infix queries. This requires the infix
preprocessor to be built in in a way which does not require sfsexp when
notmuch is built without it.

Make the preprocessor throw a Xapian error in this case (and fix the
build).

Signed-off-by: Michael J Gruber <git@grubix.eu>
---
Maybe something like this - I have not test *with* sfsexp, though.

 lib/database-private.h | 4 ----
 lib/query.cc           | 2 ++
 lib/sexp-fp.cc         | 4 ++++
 3 files changed, 6 insertions(+), 4 deletions(-)

diff --git a/lib/database-private.h b/lib/database-private.h
index 657b1aa1..419b9fe6 100644
--- a/lib/database-private.h
+++ b/lib/database-private.h
@@ -354,10 +354,6 @@ _notmuch_query_string_to_xapian_query (notmuch_database_t *notmuch,
 				       std::string query_string,
 				       Xapian::Query &output,
 				       std::string &msg);
-/* parse-sexp.cc */
-notmuch_status_t
-_notmuch_sexp_string_to_xapian_query (notmuch_database_t *notmuch, const char *querystr,
-				      Xapian::Query &output);
 
 notmuch_status_t
 _notmuch_query_expand (notmuch_database_t *notmuch, const char *field, Xapian::Query subquery,
diff --git a/lib/query.cc b/lib/query.cc
index b0937fcc..707f6222 100644
--- a/lib/query.cc
+++ b/lib/query.cc
@@ -227,6 +227,7 @@ _notmuch_query_ensure_parsed_xapian (notmuch_query_t *query)
     return NOTMUCH_STATUS_SUCCESS;
 }
 
+#if HAVE_SFSEXP
 static notmuch_status_t
 _notmuch_query_ensure_parsed_sexpr (notmuch_query_t *query)
 {
@@ -243,6 +244,7 @@ _notmuch_query_ensure_parsed_sexpr (notmuch_query_t *query)
     _notmuch_query_cache_terms (query);
     return NOTMUCH_STATUS_SUCCESS;
 }
+#endif
 
 static notmuch_status_t
 _notmuch_query_ensure_parsed (notmuch_query_t *query)
diff --git a/lib/sexp-fp.cc b/lib/sexp-fp.cc
index ed26f6ec..1fdf5225 100644
--- a/lib/sexp-fp.cc
+++ b/lib/sexp-fp.cc
@@ -30,10 +30,14 @@ SexpFieldProcessor::operator() (const std::string & query_string)
     notmuch_status_t status;
     Xapian::Query output;
 
+#if HAVE_SFSEXP
     status = _notmuch_sexp_string_to_xapian_query (notmuch, query_string.c_str (), output);
     if (status) {
 	throw Xapian::QueryParserError ("error parsing " + query_string);
     }
+#else
+    throw Xapian::QueryParserError ("sexp query parser not available");
+#endif
 
     return output;
 
-- 
2.36.0.rc2.472.gf6a51f5f41

[PATCH] lib: don't compile sexp: prefix if no sfsexp library

[David Bremner]

As Michael Gruber pointed out, the build is otherwise broken on
systems without libsfsexp.
---
 lib/prefix.cc  | 4 ++++
 lib/sexp-fp.cc | 3 +++
 lib/sexp-fp.h  | 2 ++
 3 files changed, 9 insertions(+)

diff --git a/lib/prefix.cc b/lib/prefix.cc
index 06e2333a..538a2dd1 100644
--- a/lib/prefix.cc
+++ b/lib/prefix.cc
@@ -61,8 +61,10 @@ prefix_t prefix_table[] = {
       NOTMUCH_FIELD_PROCESSOR },
     { "query",                  NULL,           NOTMUCH_FIELD_EXTERNAL |
       NOTMUCH_FIELD_PROCESSOR },
+#if HAVE_SFSEXP
     { "sexp",                  NULL,            NOTMUCH_FIELD_EXTERNAL |
       NOTMUCH_FIELD_PROCESSOR },
+#endif
     { "from",                   "XFROM",        NOTMUCH_FIELD_EXTERNAL |
       NOTMUCH_FIELD_PROBABILISTIC |
       NOTMUCH_FIELD_PROCESSOR },
@@ -141,8 +143,10 @@ _setup_query_field (const prefix_t *prefix, notmuch_database_t *notmuch)
 	    fp = (new QueryFieldProcessor (*notmuch->query_parser, notmuch))->release ();
 	else if (STRNCMP_LITERAL (prefix->name, "thread") == 0)
 	    fp = (new ThreadFieldProcessor (*notmuch->query_parser, notmuch))->release ();
+#if HAVE_SFSEXP
 	else if (STRNCMP_LITERAL (prefix->name, "sexp") == 0)
 	    fp = (new SexpFieldProcessor (notmuch))->release ();
+#endif
 	else
 	    fp = (new RegexpFieldProcessor (prefix->name, prefix->flags,
 					    *notmuch->query_parser, notmuch))->release ();
diff --git a/lib/sexp-fp.cc b/lib/sexp-fp.cc
index ed26f6ec..eeb8be98 100644
--- a/lib/sexp-fp.cc
+++ b/lib/sexp-fp.cc
@@ -21,6 +21,8 @@
  */
 
 #include "database-private.h"
+
+#if HAVE_SFSEXP
 #include "sexp-fp.h"
 #include <iostream>
 
@@ -38,3 +40,4 @@ SexpFieldProcessor::operator() (const std::string & query_string)
     return output;
 
 }
+#endif /* HAVE_SFSEXP */
diff --git a/lib/sexp-fp.h b/lib/sexp-fp.h
index 341dfa7e..0e55b961 100644
--- a/lib/sexp-fp.h
+++ b/lib/sexp-fp.h
@@ -23,6 +23,7 @@
 #ifndef NOTMUCH_SEXP_FP_H
 #define NOTMUCH_SEXP_FP_H
 
+#if HAVE_SFSEXP
 #include <xapian.h>
 #include "notmuch.h"
 
@@ -38,4 +39,5 @@ public:
     Xapian::Query operator() (const std::string & query_string);
 };
 
+#endif /* HAVE_SFSEXP */
 #endif /* NOTMUCH_SEXP_FP_H */
-- 
2.35.1

Re: [PATCH] fix build without sfsexp

[David Bremner]

michaeljgruber+grubix+git@gmail.com writes:

> From: Michael J Gruber <git@grubix.eu>
>
> a1d139de ("lib: add sexp: prefix to Xapian (infix) query parser.",
> 2022-04-09) introduced sfsexp infix queries. This requires the infix
> preprocessor to be built in in a way which does not require sfsexp when
> notmuch is built without it.
>
> Make the preprocessor throw a Xapian error in this case (and fix the
> build).
>
> Signed-off-by: Michael J Gruber <git@grubix.eu>
> ---
> Maybe something like this - I have not test *with* sfsexp, though.

Either your patch or mine is OK with me. Yours might be slightly nicer
since it gives an actual error message, rather than falling back to the
corresponding phrase (generic Xapian query parser behaviour with unknown
prefixes).

I did test both with and without libsfsexp-dev installed.

d