v2 parameter expansion for regex and wildcard sexp queries

v2 parameter expansion for regex and wildcard sexp queries

[David Bremner]

This obsoletes the series at [1].

Compared to the previous series, this refactors environment search to
avoid code duplication (and remove slightly confusing 'break'). There
is a slight adjustment to error messages via this unification.

More importantly, this avoids a bug [2] with unknown parameters in the
previous version, and does error checking consistent with the other
places parameters are expanded.

[1]: id:20220614112217.1094083-1-david@tethera.net
[2]: infinite loop!

[PATCH v2 1/2] test/sexp: add known broken tests for macro param inside rx/wildcard

[David Bremner]

These tests replicate the problem reported by Eric Colson [1] (for the
regex case).

[1]: id:87o7yxqxy6.fsf@code.pm
---
 test/T081-sexpr-search.sh | 50 +++++++++++++++++++++++++++++++++++++++
 1 file changed, 50 insertions(+)

diff --git a/test/T081-sexpr-search.sh b/test/T081-sexpr-search.sh
index da819190..d28e5b76 100755
--- a/test/T081-sexpr-search.sh
+++ b/test/T081-sexpr-search.sh
@@ -1115,6 +1115,26 @@ too many arguments to macro
 EOF
 test_expect_equal_file EXPECTED OUTPUT
 
+test_begin_subtest "Saved Search: bad parameter syntax 5"
+test_subtest_known_broken
+notmuch config set squery.Bad5 '(macro (thing) (tag (rx ,thing)))'
+notmuch search --query=sexp '(Bad5 (1 2))' >OUTPUT 2>&1
+cat <<EOF > EXPECTED
+notmuch search: Syntax error in query
+'rx' expects single atom as argument
+EOF
+test_expect_equal_file EXPECTED OUTPUT
+
+test_begin_subtest "Saved Search: bad parameter syntax 6"
+test_subtest_known_broken
+notmuch config set squery.Bad6 '(macro (thing) (tag (starts-with ,thing)))'
+notmuch search --query=sexp '(Bad6 (1 2))' >OUTPUT 2>&1
+cat <<EOF > EXPECTED
+notmuch search: Syntax error in query
+'starts-with' expects single atom as argument
+EOF
+test_expect_equal_file EXPECTED OUTPUT
+
 test_begin_subtest "Saved Search: macro without body"
 notmuch config set squery.Bad3  '(macro (a b))'
 notmuch search --query=sexp '(Bad3)' >OUTPUT 2>&1
@@ -1166,6 +1186,20 @@ notmuch config set squery.TagSubject2  '(macro (tagname subj) (and (tag ,tagname
 notmuch search --query=sexp '(TagSubject2 inbox maildir)' | notmuch_search_sanitize > OUTPUT
 test_expect_equal_file EXPECTED OUTPUT
 
+test_begin_subtest "macro in regex"
+test_subtest_known_broken
+notmuch search tag:inbox and date:2009-11-17 | notmuch_search_sanitize > EXPECTED
+notmuch config set squery.D  '(macro (tagname)  (and (date 2009-11-17) (tag (rx ,tagname))))'
+notmuch search --query=sexp '(D inbo)' | notmuch_search_sanitize > OUTPUT
+test_expect_equal_file_nonempty EXPECTED OUTPUT
+
+test_begin_subtest "macro in wildcard"
+test_subtest_known_broken
+notmuch search tag:inbox and date:2009-11-17 | notmuch_search_sanitize > EXPECTED
+notmuch config set squery.W  '(macro (tagname)  (and (date 2009-11-17) (tag (starts-with ,tagname))))'
+notmuch search --query=sexp '(W inbo)' | notmuch_search_sanitize > OUTPUT
+test_expect_equal_file_nonempty EXPECTED OUTPUT
+
 test_begin_subtest "nested macros (shadowing)"
 notmuch search tag:inbox and subject:maildir | notmuch_search_sanitize > EXPECTED
 notmuch config set squery.Inner '(macro (x) (subject ,x))'
@@ -1183,6 +1217,22 @@ undefined parameter y
 EOF
 test_expect_equal_file EXPECTED OUTPUT
 
+test_begin_subtest "nested macros (shadowing, regex)"
+test_subtest_known_broken
+notmuch search tag:/inbo/ and subject:/Maildi/ | notmuch_search_sanitize > EXPECTED
+notmuch config set squery.Inner3 '(macro (x) (subject (rx ,x)))'
+notmuch config set squery.Outer3  '(macro (x y) (and (tag (rx ,x)) (Inner3 ,y)))'
+notmuch search --query=sexp '(Outer3 inbo Maildi)' | notmuch_search_sanitize > OUTPUT
+test_expect_equal_file_nonempty EXPECTED OUTPUT
+
+test_begin_subtest "nested macros (shadowing, wildcard)"
+test_subtest_known_broken
+notmuch search tag:inbox and subject:maildir | notmuch_search_sanitize > EXPECTED
+notmuch config set squery.Inner4 '(macro (x) (subject (starts-with ,x)))'
+notmuch config set squery.Outer4  '(macro (x y) (and (tag (starts-with ,x)) (Inner4 ,y)))'
+notmuch search --query=sexp '(Outer4 inbo maildi)' | notmuch_search_sanitize > OUTPUT
+test_expect_equal_file_nonempty EXPECTED OUTPUT
+
 test_begin_subtest "combine macro and user defined header"
 notmuch config set squery.About '(macro (name) (or (subject ,name) (List ,name)))'
 notmuch search subject:notmuch or List:notmuch | notmuch_search_sanitize > EXPECTED
-- 
2.35.2

[PATCH v2 2/2] lib/sexp: add parameter expansion for regex and wildcard

[David Bremner]

Fix the bug reported at [1].

The parameter expansion for regex and wildcard modifiers has to be
done a bit differently, because their arguments are not s-expressions
defining complete Xapian queries.

[1]: id:87o7yxqxy6.fsf@code.pm
---
 lib/parse-sexp.cc         | 91 +++++++++++++++++++++++++++++++++------
 test/T081-sexpr-search.sh | 18 ++++----
 2 files changed, 87 insertions(+), 22 deletions(-)

diff --git a/lib/parse-sexp.cc b/lib/parse-sexp.cc
index 08fd7037..ef55812d 100644
--- a/lib/parse-sexp.cc
+++ b/lib/parse-sexp.cc
@@ -187,6 +187,55 @@ _sexp_parse_phrase (std::string term_prefix, const char *phrase, Xapian::Query &
     return NOTMUCH_STATUS_SUCCESS;
 }
 
+static notmuch_status_t
+resolve_binding (notmuch_database_t *notmuch, const _sexp_binding_t *env, const char *name,
+		 const _sexp_binding_t **out)
+{
+    for (; env; env = env->next) {
+	if (strcmp (name, env->name) == 0) {
+	    *out = env;
+	    return NOTMUCH_STATUS_SUCCESS;
+	}
+    }
+
+    _notmuch_database_log (notmuch, "undefined parameter '%s'\n", name);
+    return NOTMUCH_STATUS_BAD_QUERY_SYNTAX;
+}
+
+static notmuch_status_t
+_sexp_expand_term (notmuch_database_t *notmuch,
+		   const _sexp_prefix_t *prefix,
+		   const _sexp_binding_t *env,
+		   const sexp_t *sx,
+		   const char **out)
+{
+    notmuch_status_t status;
+
+    if (! out)
+	return NOTMUCH_STATUS_NULL_POINTER;
+
+    while (sx->ty == SEXP_VALUE && sx->aty == SEXP_BASIC && sx->val[0] == ',') {
+	const char *name = sx->val + 1;
+	const _sexp_binding_t *binding;
+
+	status = resolve_binding (notmuch, env, name, &binding);
+	if (status)
+	    return status;
+
+	sx = binding->sx;
+	env = binding->context;
+    }
+
+    if (sx->ty != SEXP_VALUE) {
+	_notmuch_database_log (notmuch, "'%s' expects single atom as argument\n",
+			       prefix->name);
+	return NOTMUCH_STATUS_BAD_QUERY_SYNTAX;
+    }
+
+    *out = sx->val;
+    return NOTMUCH_STATUS_SUCCESS;
+}
+
 static notmuch_status_t
 _sexp_parse_wildcard (notmuch_database_t *notmuch,
 		      const _sexp_prefix_t *parent,
@@ -227,8 +276,8 @@ _sexp_parse_one_term (notmuch_database_t *notmuch, std::string term_prefix, cons
 notmuch_status_t
 _sexp_parse_regex (notmuch_database_t *notmuch,
 		   const _sexp_prefix_t *prefix, const _sexp_prefix_t *parent,
-		   unused(const _sexp_binding_t *env),
-		   std::string val, Xapian::Query &output)
+		   const _sexp_binding_t *env,
+		   const sexp_t *term, Xapian::Query &output)
 {
     if (! parent) {
 	_notmuch_database_log (notmuch, "illegal '%s' outside field\n",
@@ -243,9 +292,15 @@ _sexp_parse_regex (notmuch_database_t *notmuch,
     }
 
     std::string msg; /* ignored */
+    const char *str;
+    notmuch_status_t status;
+
+    status = _sexp_expand_term (notmuch, prefix, env, term, &str);
+    if (status)
+	return status;
 
     return _notmuch_regexp_to_query (notmuch, Xapian::BAD_VALUENO, parent->name,
-				     val, output, msg);
+				     str, output, msg);
 }
 
 
@@ -444,14 +499,16 @@ _sexp_expand_param (notmuch_database_t *notmuch, const _sexp_prefix_t *parent,
 		    const _sexp_binding_t *env, const char *name,
 		    Xapian::Query &output)
 {
-    for (; env; env = env->next) {
-	if (strcmp (name, env->name) == 0) {
-	    return _sexp_to_xapian_query (notmuch, parent, env->context, env->sx,
-					  output);
-	}
-    }
-    _notmuch_database_log (notmuch, "undefined parameter %s\n", name);
-    return NOTMUCH_STATUS_BAD_QUERY_SYNTAX;
+    notmuch_status_t status;
+
+    const _sexp_binding_t *binding;
+
+    status = resolve_binding (notmuch, env, name, &binding);
+    if (status)
+	return status;
+
+    return _sexp_to_xapian_query (notmuch, parent, binding->context, binding->sx,
+				  output);
 }
 
 static notmuch_status_t
@@ -638,11 +695,17 @@ _sexp_to_xapian_query (notmuch_database_t *notmuch, const _sexp_prefix_t *parent
 		return _notmuch_query_name_to_query (notmuch, sx->list->next->val, output);
 	    }
 
-	    if (prefix->xapian_op == Xapian::Query::OP_WILDCARD)
-		return _sexp_parse_wildcard (notmuch, parent, env, sx->list->next->val, output);
+	    if (prefix->xapian_op == Xapian::Query::OP_WILDCARD) {
+		const char *str;
+		status = _sexp_expand_term (notmuch, prefix, env, sx->list->next, &str);
+		if (status)
+		    return status;
+
+		return _sexp_parse_wildcard (notmuch, parent, env, str, output);
+	    }
 
 	    if (prefix->flags & SEXP_FLAG_DO_REGEX) {
-		return _sexp_parse_regex (notmuch, prefix, parent, env, sx->list->next->val, output);
+		return _sexp_parse_regex (notmuch, prefix, parent, env, sx->list->next, output);
 	    }
 
 	    if (prefix->flags & SEXP_FLAG_DO_EXPAND) {
diff --git a/test/T081-sexpr-search.sh b/test/T081-sexpr-search.sh
index d28e5b76..c089e382 100755
--- a/test/T081-sexpr-search.sh
+++ b/test/T081-sexpr-search.sh
@@ -1116,7 +1116,6 @@ EOF
 test_expect_equal_file EXPECTED OUTPUT
 
 test_begin_subtest "Saved Search: bad parameter syntax 5"
-test_subtest_known_broken
 notmuch config set squery.Bad5 '(macro (thing) (tag (rx ,thing)))'
 notmuch search --query=sexp '(Bad5 (1 2))' >OUTPUT 2>&1
 cat <<EOF > EXPECTED
@@ -1126,7 +1125,6 @@ EOF
 test_expect_equal_file EXPECTED OUTPUT
 
 test_begin_subtest "Saved Search: bad parameter syntax 6"
-test_subtest_known_broken
 notmuch config set squery.Bad6 '(macro (thing) (tag (starts-with ,thing)))'
 notmuch search --query=sexp '(Bad6 (1 2))' >OUTPUT 2>&1
 cat <<EOF > EXPECTED
@@ -1135,6 +1133,14 @@ notmuch search: Syntax error in query
 EOF
 test_expect_equal_file EXPECTED OUTPUT
 
+test_begin_subtest "Saved Search: bad parameter syntax 7"
+notmuch search --query=sexp '(subject (rx ,unknown))' >OUTPUT 2>&1
+cat <<EOF > EXPECTED
+notmuch search: Syntax error in query
+undefined parameter 'unknown'
+EOF
+test_expect_equal_file EXPECTED OUTPUT
+
 test_begin_subtest "Saved Search: macro without body"
 notmuch config set squery.Bad3  '(macro (a b))'
 notmuch search --query=sexp '(Bad3)' >OUTPUT 2>&1
@@ -1164,7 +1170,7 @@ notmuch config set squery.Bad6  '(macro (a) (and ,b (subject maildir)))'
 notmuch search --query=sexp '(Bad6 foo)' >OUTPUT 2>&1
 cat <<EOF > EXPECTED
 notmuch search: Syntax error in query
-undefined parameter b
+undefined parameter 'b'
 EOF
 test_expect_equal_file EXPECTED OUTPUT
 
@@ -1187,14 +1193,12 @@ notmuch search --query=sexp '(TagSubject2 inbox maildir)' | notmuch_search_sanit
 test_expect_equal_file EXPECTED OUTPUT
 
 test_begin_subtest "macro in regex"
-test_subtest_known_broken
 notmuch search tag:inbox and date:2009-11-17 | notmuch_search_sanitize > EXPECTED
 notmuch config set squery.D  '(macro (tagname)  (and (date 2009-11-17) (tag (rx ,tagname))))'
 notmuch search --query=sexp '(D inbo)' | notmuch_search_sanitize > OUTPUT
 test_expect_equal_file_nonempty EXPECTED OUTPUT
 
 test_begin_subtest "macro in wildcard"
-test_subtest_known_broken
 notmuch search tag:inbox and date:2009-11-17 | notmuch_search_sanitize > EXPECTED
 notmuch config set squery.W  '(macro (tagname)  (and (date 2009-11-17) (tag (starts-with ,tagname))))'
 notmuch search --query=sexp '(W inbo)' | notmuch_search_sanitize > OUTPUT
@@ -1213,12 +1217,11 @@ notmuch config set squery.Outer2  '(macro (x y) (and (tag ,x) (Inner2 ,y)))'
 notmuch search --query=sexp '(Outer2 inbox maildir)' > OUTPUT 2>&1
 cat <<EOF > EXPECTED
 notmuch search: Syntax error in query
-undefined parameter y
+undefined parameter 'y'
 EOF
 test_expect_equal_file EXPECTED OUTPUT
 
 test_begin_subtest "nested macros (shadowing, regex)"
-test_subtest_known_broken
 notmuch search tag:/inbo/ and subject:/Maildi/ | notmuch_search_sanitize > EXPECTED
 notmuch config set squery.Inner3 '(macro (x) (subject (rx ,x)))'
 notmuch config set squery.Outer3  '(macro (x y) (and (tag (rx ,x)) (Inner3 ,y)))'
@@ -1226,7 +1229,6 @@ notmuch search --query=sexp '(Outer3 inbo Maildi)' | notmuch_search_sanitize > O
 test_expect_equal_file_nonempty EXPECTED OUTPUT
 
 test_begin_subtest "nested macros (shadowing, wildcard)"
-test_subtest_known_broken
 notmuch search tag:inbox and subject:maildir | notmuch_search_sanitize > EXPECTED
 notmuch config set squery.Inner4 '(macro (x) (subject (starts-with ,x)))'
 notmuch config set squery.Outer4  '(macro (x y) (and (tag (starts-with ,x)) (Inner4 ,y)))'
-- 
2.35.2

Re: v2 parameter expansion for regex and wildcard sexp queries

[David Bremner]

David Bremner <david@tethera.net> writes:

> This obsoletes the series at [1].
>
> Compared to the previous series, this refactors environment search to
> avoid code duplication (and remove slightly confusing 'break'). There
> is a slight adjustment to error messages via this unification.
>
> More importantly, this avoids a bug [2] with unknown parameters in the
> previous version, and does error checking consistent with the other
> places parameters are expanded.
>
> [1]: id:20220614112217.1094083-1-david@tethera.net
> [2]: infinite loop!

Series applied to master

d