Parenthesised query breaks query it is embedded in

Parenthesised query breaks query it is embedded in

[Sean Whitton]

Hello,

I have this subquery:

    subject:("Cron <dak@" and "sudo -u dak-unpriv /srv/")

as one of many queries in a long one like this: (q1) or (q2) or (q3) ..

Including that particular subquery breaks the whole thing: results which
should be included by other disjuncts are not included.  I suspect but
have not confirmed that each disjunct after the problematic one stops
matching anything.

The query works fine on its own.  And I can work around the problem by
replacing it with

    subject:"Cron <dak@" and subject:"sudo -u dak-unpriv /srv/"

Seems like a parsing bug.  Thanks.

-- 
Sean Whitton

fix for parsing bracketed expression

[David Bremner]

This is not a complete fix, which is hard because of the way we
implement regular expressions. Sean's original examples still won't
work, but hopefully the tests in the second patch show how to make
something similar work. This is probably a good time to mention that
this kind of thing is easier in the sexp query parser.

d

[PATCH 1/2] test: known broken tests for bracketed terms in subject

[David Bremner]

The heuristics in the field processor currently incorrectly trigger
phrase parsing.
---
 test/T650-regexp-query.sh | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)

diff --git a/test/T650-regexp-query.sh b/test/T650-regexp-query.sh
index 55dc6c88..4ee6b171 100755
--- a/test/T650-regexp-query.sh
+++ b/test/T650-regexp-query.sh
@@ -65,6 +65,24 @@ thread:XXX   2001-01-05 [1/1] Notmuch Test Suite; - (inbox unread)
 EOF
 test_expect_equal_file EXPECTED OUTPUT
 
+test_begin_subtest "bracketed subject search (with dquotes)"
+test_subtest_known_broken
+notmuch search subject:notmuch and subject:show > EXPECTED
+notmuch search 'subject:"(show notmuch)"' > OUTPUT
+test_expect_equal_file_nonempty EXPECTED OUTPUT
+
+test_begin_subtest "bracketed subject search (with dquotes and operator 'or')"
+test_subtest_known_broken
+notmuch search subject:notmuch or subject:show > EXPECTED
+notmuch search 'subject:"(notmuch or show)"' > OUTPUT
+test_expect_equal_file_nonempty EXPECTED OUTPUT
+
+test_begin_subtest "bracketed subject search (with dquotes and operator 'and')"
+test_subtest_known_broken
+notmuch search subject:notmuch and subject:show > EXPECTED
+notmuch search 'subject:"(notmuch and show)"' > OUTPUT
+test_expect_equal_file_nonempty EXPECTED OUTPUT
+
 test_begin_subtest "xapian wildcard search for from:"
 notmuch search --output=messages 'from:cwo*' > OUTPUT
 test_expect_equal_file cworth.msg-ids OUTPUT
-- 
2.34.1

[PATCH 2/2] lib: do not phrase parse prefixed bracketed subexpressions

[David Bremner]

Since Xapian does not preserve quotes when passing the subquery to a
field processor, we have to make a guess as to what the user
intended. Here the added assumption is that a string surrounded by
parens is not intended to be a phrase.
---
 doc/man7/notmuch-search-terms.rst |  6 ++++--
 lib/regexp-fields.cc              |  3 ++-
 test/T650-regexp-query.sh         | 13 ++++++++++---
 3 files changed, 16 insertions(+), 6 deletions(-)

diff --git a/doc/man7/notmuch-search-terms.rst b/doc/man7/notmuch-search-terms.rst
index e80cc7d0..f8ad1edb 100644
--- a/doc/man7/notmuch-search-terms.rst
+++ b/doc/man7/notmuch-search-terms.rst
@@ -275,11 +275,13 @@ the same phrase.
 - a.list.of.words
 
 Both parenthesised lists of terms and quoted phrases are ok with
-probabilistic prefixes such as **to:**, **from:**, and **subject:**. In particular
+probabilistic prefixes such as **to:**, **from:**, and **subject:**.
+For prefixes supporting regex search, the parenthesised list should be
+quoted.  In particular
 
 ::
 
-   subject:(pizza free)
+   subject:"(pizza free)"
 
 is equivalent to
 
diff --git a/lib/regexp-fields.cc b/lib/regexp-fields.cc
index 7e9d959c..539915d8 100644
--- a/lib/regexp-fields.cc
+++ b/lib/regexp-fields.cc
@@ -227,7 +227,8 @@ RegexpFieldProcessor::operator() (const std::string & str)
 	     * phrase parsing, when possible */
 	    std::string query_str;
 
-	    if (*str.rbegin () != '*' || str.find (' ') != std::string::npos)
+	    if ((str.at (0) != '(' || *str.rbegin () != ')') &&
+		(*str.rbegin () != '*' || str.find (' ') != std::string::npos))
 		query_str = '"' + str + '"';
 	    else
 		query_str = str;
diff --git a/test/T650-regexp-query.sh b/test/T650-regexp-query.sh
index 4ee6b171..a9844501 100755
--- a/test/T650-regexp-query.sh
+++ b/test/T650-regexp-query.sh
@@ -66,23 +66,30 @@ EOF
 test_expect_equal_file EXPECTED OUTPUT
 
 test_begin_subtest "bracketed subject search (with dquotes)"
-test_subtest_known_broken
 notmuch search subject:notmuch and subject:show > EXPECTED
 notmuch search 'subject:"(show notmuch)"' > OUTPUT
 test_expect_equal_file_nonempty EXPECTED OUTPUT
 
 test_begin_subtest "bracketed subject search (with dquotes and operator 'or')"
-test_subtest_known_broken
 notmuch search subject:notmuch or subject:show > EXPECTED
 notmuch search 'subject:"(notmuch or show)"' > OUTPUT
 test_expect_equal_file_nonempty EXPECTED OUTPUT
 
 test_begin_subtest "bracketed subject search (with dquotes and operator 'and')"
-test_subtest_known_broken
 notmuch search subject:notmuch and subject:show > EXPECTED
 notmuch search 'subject:"(notmuch and show)"' > OUTPUT
 test_expect_equal_file_nonempty EXPECTED OUTPUT
 
+test_begin_subtest "bracketed subject search (with phrase, operator 'or')"
+notmuch search 'subject:"mailing list"' or subject:FreeBSD > EXPECTED
+notmuch search  'subject:"(""mailing list"" or FreeBSD)"' > OUTPUT
+test_expect_equal_file_nonempty EXPECTED OUTPUT
+
+test_begin_subtest "bracketed subject search (with phrase, operator 'and')"
+notmuch search  search 'subject:"notmuch show"' and subject:commands > EXPECTED
+notmuch search  'subject:"(""notmuch show"" and commands)"' > OUTPUT
+test_expect_equal_file_nonempty EXPECTED OUTPUT
+
 test_begin_subtest "xapian wildcard search for from:"
 notmuch search --output=messages 'from:cwo*' > OUTPUT
 test_expect_equal_file cworth.msg-ids OUTPUT
-- 
2.34.1

Re: [PATCH 1/2] test: known broken tests for bracketed terms in subject

[David Bremner]

David Bremner <david@tethera.net> writes:

> The heuristics in the field processor currently incorrectly trigger
> phrase parsing.

I have applied this series to master

d

Re: [PATCH 1/2] test: known broken tests for bracketed terms in subject

[Sean Whitton]

Hello,

On Sat 19 Mar 2022 at 07:38am -03, David Bremner wrote:

> David Bremner <david@tethera.net> writes:
>
>> The heuristics in the field processor currently incorrectly trigger
>> phrase parsing.
>
> I have applied this series to master

Nice, thanks!

-- 
Sean Whitton