revised foo:"" handling

revised foo:"" handling

[David Bremner]

This obsoletes the first two patches of

     id:20170318030303.17344-1-david@tethera.net
     
I think this is a more meaningful interpretation than matching all messages.

[PATCH 1/2] test: add known broken test for null from: and subject: query

[David Bremner]

These queries currently fail with field processors enabled because the
code expects a non-empty string.
---
 test/T650-regexp-query.sh | 20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)

diff --git a/test/T650-regexp-query.sh b/test/T650-regexp-query.sh
index 61739e87..f2ae1387 100755
--- a/test/T650-regexp-query.sh
+++ b/test/T650-regexp-query.sh
@@ -11,6 +11,26 @@ fi
 
 notmuch search --output=messages from:cworth > cworth.msg-ids
 
+# these headers will generate no document terms
+add_message '[from]="-" [subject]="empty from"'
+add_message '[subject]="-"'
+
+test_begin_subtest "null from: search"
+test_subtest_known_broken
+notmuch search 'from:""' | notmuch_search_sanitize > OUTPUT
+cat <<EOF > EXPECTED
+thread:XXX   2001-01-05 [1/1] -; empty from (inbox unread)
+EOF
+test_expect_equal_file EXPECTED OUTPUT
+
+test_begin_subtest "null subject: search"
+test_subtest_known_broken
+notmuch search 'subject:""' | notmuch_search_sanitize > OUTPUT
+cat <<EOF > EXPECTED
+thread:XXX   2001-01-05 [1/1] Notmuch Test Suite; - (inbox unread)
+EOF
+test_expect_equal_file EXPECTED OUTPUT
+
 test_begin_subtest "xapian wildcard search for from:"
 notmuch search --output=messages 'from:cwo*' > OUTPUT
 test_expect_equal_file cworth.msg-ids OUTPUT
-- 
2.11.0

[PATCH 2/2] lib: handle empty string in regexp field processors

[David Bremner]

The non-field processor behaviour is is convert the corresponding
queries into a search for the unprefixed terms. This yields pretty
surprising results so I decided to generate a query that would match
the terms (i.e. none with that prefix) generated for an empty header.
---
 lib/regexp-fields.cc      | 5 +++++
 test/T650-regexp-query.sh | 2 --
 2 files changed, 5 insertions(+), 2 deletions(-)

diff --git a/lib/regexp-fields.cc b/lib/regexp-fields.cc
index 9dcf9732..1651677c 100644
--- a/lib/regexp-fields.cc
+++ b/lib/regexp-fields.cc
@@ -148,6 +148,11 @@ RegexpFieldProcessor::RegexpFieldProcessor (std::string prefix, Xapian::QueryPar
 Xapian::Query
 RegexpFieldProcessor::operator() (const std::string & str)
 {
+    if (str.size () == 0)
+	return Xapian::Query(Xapian::Query::OP_AND_NOT,
+			     Xapian::Query::MatchAll,
+			     Xapian::Query (Xapian::Query::OP_WILDCARD, term_prefix));
+
     if (str.at (0) == '/') {
 	if (str.at (str.size () - 1) == '/'){
 	    RegexpPostingSource *postings = new RegexpPostingSource (slot, str.substr(1,str.size () - 2));
diff --git a/test/T650-regexp-query.sh b/test/T650-regexp-query.sh
index f2ae1387..9599c104 100755
--- a/test/T650-regexp-query.sh
+++ b/test/T650-regexp-query.sh
@@ -16,7 +16,6 @@ add_message '[from]="-" [subject]="empty from"'
 add_message '[subject]="-"'
 
 test_begin_subtest "null from: search"
-test_subtest_known_broken
 notmuch search 'from:""' | notmuch_search_sanitize > OUTPUT
 cat <<EOF > EXPECTED
 thread:XXX   2001-01-05 [1/1] -; empty from (inbox unread)
@@ -24,7 +23,6 @@ EOF
 test_expect_equal_file EXPECTED OUTPUT
 
 test_begin_subtest "null subject: search"
-test_subtest_known_broken
 notmuch search 'subject:""' | notmuch_search_sanitize > OUTPUT
 cat <<EOF > EXPECTED
 thread:XXX   2001-01-05 [1/1] Notmuch Test Suite; - (inbox unread)
-- 
2.11.0

Re: [PATCH 2/2] lib: handle empty string in regexp field processors

[David Bremner]

David Bremner <david@tethera.net> writes:

> +    if (str.size () == 0)
> +	return Xapian::Query(Xapian::Query::OP_AND_NOT,
> +			     Xapian::Query::MatchAll,
> +			     Xapian::Query (Xapian::Query::OP_WILDCARD, term_prefix));
> +

Full disclosure, this is a pretty expensive query. On an older i7, it
takes about 7.5s (elapsed) on my 466k messages to find 702 messages
without a subject.  I don't think it's a big deal, since I don't think

     notmuch search 'subject:""'

is likely to be typed by mistake.

For comparison, "grep -R '^Subject:$'" (which is not exactly the same
query,  since some messages completely lack a Subject: line).
takes about 390s (elapsed).

Re: revised foo:"" handling

[Tomi Ollila]

On Sat, Mar 25 2017, David Bremner <david@tethera.net> wrote:

> This obsoletes the first two patches of
>
>      id:20170318030303.17344-1-david@tethera.net
>      
> I think this is a more meaningful interpretation than matching all messages.

These changes look good (AFAIU). tests pass (debian unstable container on
fedora 25 host)

Tomi

Re: revised foo:"" handling

[David Bremner]

Tomi Ollila <tomi.ollila@iki.fi> writes:

> On Sat, Mar 25 2017, David Bremner <david@tethera.net> wrote:
>
>> This obsoletes the first two patches of
>>
>>      id:20170318030303.17344-1-david@tethera.net
>>      
>> I think this is a more meaningful interpretation than matching all messages.
>
> These changes look good (AFAIU). tests pass (debian unstable container on
> fedora 25 host)

I pushed those to release and master