Strange, incoherent query parsing

Strange, incoherent query parsing

[Jorge P. de Morais Neto]

Hi.  It seems I found another query parsing bug.  Look at the strange,
incoherent behavior below:

--8<---------------cut here---------------start------------->8---
$ NOTMUCH_DEBUG_QUERY=y notmuch count 'subject:("Saúde" "Geap") OR subject:xplitz'
Query string is:
subject:("Saúde" "Geap") OR subject:xplitz
Exclude query is:
Query(((Kdeleted OR Kspam) OR Ktrash))
Final query is:
Query(((Tmail AND (((geap@1 OR Ggeap@1 OR Kgeap@1 OR Kgeap@1 OR Qgeap@1 OR Qgeap@1 OR Pgeap@1 OR XPROPERTYgeap@1 OR XFOLDER:geap@1 OR XFROMgeap@1 OR XTOgeap@1 OR XATTACHMENTgeap@1 OR XMIMETYPEgeap@1 OR XSUBJECTgeap@1) AND (or@2 OR Gor@2 OR Kor@2 OR Kor@2 OR Qor@2 OR Qor@2 OR Por@2 OR XPROPERTYor@2 OR XFOLDER:or@2 OR XFROMor@2 OR XTOor@2 OR XATTACHMENTor@2 OR XMIMETYPEor@2 OR XSUBJECTor@2)) FILTER (XSUBJECTsaúde@1 OR XSUBJECTxplitz@1))) AND_NOT ((Kdeleted OR Kspam) OR Ktrash)))
0
$ NOTMUCH_DEBUG_QUERY=y notmuch count '(subject:"Saúde" AND subject:"Geap") OR subject:xplitz)'
Query string is:
(subject:"Saúde" AND subject:"Geap") OR subject:xplitz)
Exclude query is:
Query(((Kdeleted OR Kspam) OR Ktrash))
Final query is:
Query(((Tmail AND (((and@1 OR Gand@1 OR Kand@1 OR Kand@1 OR Qand@1 OR Qand@1 OR Pand@1 OR XPROPERTYand@1 OR XFOLDER:and@1 OR XFROMand@1 OR XTOand@1 OR XATTACHMENTand@1 OR XMIMETYPEand@1 OR XSUBJECTand@1) AND (or@2 OR Gor@2 OR Kor@2 OR Kor@2 OR Qor@2 OR Qor@2 OR Por@2 OR XPROPERTYor@2 OR XFOLDER:or@2 OR XFROMor@2 OR XTOor@2 OR XATTACHMENTor@2 OR XMIMETYPEor@2 OR XSUBJECTor@2)) FILTER (XSUBJECTsaúde@1 OR XSUBJECTgeap@1 OR XSUBJECTxplitz@1))) AND_NOT ((Kdeleted OR Kspam) OR Ktrash)))
0
$ NOTMUCH_DEBUG_QUERY=y notmuch count 'subject:"Saúde" AND subject:"Geap"'
Query string is:
subject:"Saúde" AND subject:"Geap"
Exclude query is:
Query(((Kdeleted OR Kspam) OR Ktrash))
Final query is:
Query(((Tmail AND (0 * XSUBJECTsaúde@1 AND 0 * XSUBJECTgeap@1)) AND_NOT ((Kdeleted OR Kspam) OR Ktrash)))
3
$ NOTMUCH_DEBUG_QUERY=y notmuch count 'subject:"Saúde" subject:"Geap"'
Query string is:
subject:"Saúde" subject:"Geap"
Exclude query is:
Query(((Kdeleted OR Kspam) OR Ktrash))
Final query is:
Query(((Tmail AND 0 * (XSUBJECTsaúde@1 OR XSUBJECTgeap@1)) AND_NOT ((Kdeleted OR Kspam) OR Ktrash)))
71
--8<---------------cut here---------------end--------------->8---
Note: "xplitz" is not a Portuguese word and does not occur on the Notmuch
database.

This happened here at my workplace.  The notmuch config is:

--8<---------------cut here---------------start------------->8---
# .notmuch-config - Configuration file for the notmuch mail system
#
# For more information about notmuch, see https://notmuchmail.org

# Database configuration
#
# The only value supported here is 'path' which should be the top-level
# directory where your mail currently exists and to where mail will be
# delivered in the future. Files should be individual email messages.
# Notmuch will store its database within a sub-directory of the path
# configured here named ".notmuch".
#

[database]
path=/home/jorge/offlineimap/Dtp

# User configuration
#
# Here is where you can let notmuch know how you would like to be
# addressed. Valid settings are
#
#	name		Your full name.
#	primary_email	Your primary email address.
#	other_email	A list (separated by ';') of other email addresses
#			at which you receive email.
#
# Notmuch will use the various email addresses configured here when
# formatting replies. It will avoid including your own addresses in the
# recipient list of replies, and will set the From address based on the
# address to which the original email was addressed.
#

[user]
name=Jorge P. de Morais Neto
primary_email=<REDACTED>
other_email=<REDACTED>;<REDACTED>;<REDACTED>;<REDACTED>;<REDACTED>;<REDACTED>;<REDACTED>;
# Configuration for "notmuch new"
#
# The following options are supported here:
#
#	tags	A list (separated by ';') of the tags that will be
#		added to all messages incorporated by "notmuch new".
#
#	ignore	A list (separated by ';') of file and directory names
#		that will not be searched for messages by "notmuch new".
#
#		NOTE: *Every* file/directory that goes by one of those
#		names will be ignored, independent of its depth/location
#		in the mail store.
#

[new]
# tags=unread;inbox;new;
# http://afew.readthedocs.io/en/latest/quickstart.html#initial-config
tags=new
ignore=

# Search configuration
#
# The following option is supported here:
#
#	exclude_tags
#		A ;-separated list of tags that will be excluded from
#		search results by default.  Using an excluded tag in a
#		query will override that exclusion.
#

[search]
exclude_tags=deleted;spam;trash;

# Maildir compatibility configuration
#
# The following option is supported here:
#
#	synchronize_flags      Valid values are true and false.
#
#	If true, then the following maildir flags (in message filenames)
#	will be synchronized with the corresponding notmuch tags:
#
#		Flag	Tag
#		----	-------
#		D	draft
#		F	flagged
#		P	passed
#		R	replied
#		S	unread (added when 'S' flag is not present)
#
#	The "notmuch new" command will notice flag changes in filenames
#	and update tags, while the "notmuch tag" and "notmuch restore"
#	commands will notice tag changes and update flags in filenames
#

[maildir]
synchronize_flags=true

# Cryptography related configuration
#
# The following *deprecated* option is currently supported:
#
#	gpg_path
#		binary name or full path to invoke gpg.
#		NOTE: In a future build, this option will be ignored.
#		Setting $PATH is a better approach.
#

[crypto]
gpg_path=gpg
--8<---------------cut here---------------end--------------->8---

Regards
-- 
- I am Brazilian.  I hope my English is correct and I welcome feedback
- Please adopt free formats like PDF, ODF, Org, LaTeX, Opus, WebM and 7z
- Free/libre software for Android: https://f-droid.org/
- [[https://www.gnu.org/philosophy/free-sw.html][What is free software?]]

Re: Strange, incoherent query parsing

[David Bremner]

jorge+list@disroot.org (Jorge P. de Morais Neto) writes:

> $ NOTMUCH_DEBUG_QUERY=y notmuch count '(subject:"Saúde" AND subject:"Geap") OR subject:xplitz)'
> Query string is:

I don't know about the other cases but here there is an  extra closing
')', which confuses the query parser.  I agree that an error report
would be nicer, but we don't currently have that option from Xapian.

Re: Strange, incoherent query parsing

[Jorge P. de Morais Neto]

Hello,

Em 2019-08-28T07:08:28-0300, David Bremner escreveu:

> I don't know about the other cases but here there is an  extra closing
> ')', which confuses the query parser.  I agree that an error report
> would be nicer, but we don't currently have that option from Xapian.

Oops.  Sorry about that.  But still there seem to be real errors.  The
first query[1] should match messages containing both "Saúde" and "Geap"
(sans the double quotes) in the subject, but does not.

1: 'subject:("Saúde" "Geap") OR subject:xplitz'

Also note that the fourth query[2] should have AND as the implicit
boolean operator, because 'subject:' is not a boolean prefix.  You
explained in the other thread the implicit OR operator only applies for
search terms with the same /boolean/ prefix.

2: 'subject:"Saúde" subject:"Geap"'

Regards
-- 
- I am Brazilian.  I hope my English is correct and I welcome feedback
- Please adopt free formats like PDF, ODF, Org, LaTeX, Opus, WebM and 7z
- Free/libre software for Android: https://f-droid.org/
- [[https://www.gnu.org/philosophy/free-sw.html][What is free software?]]

Re: Strange, incoherent query parsing

[David Bremner]

Jorge P. de Morais Neto <jorge+list@disroot.org> writes:

> Hello,
>
> Em 2019-08-28T07:08:28-0300, David Bremner escreveu:
>
>> I don't know about the other cases but here there is an  extra closing
>> ')', which confuses the query parser.  I agree that an error report
>> would be nicer, but we don't currently have that option from Xapian.
>
> Oops.  Sorry about that.  But still there seem to be real errors.  The
> first query[1] should match messages containing both "Saúde" and "Geap"
> (sans the double quotes) in the subject, but does not.
>
> 1: 'subject:("Saúde" "Geap") OR subject:xplitz'

There's two problems here. The first is because of the way regexp fields
are implimented subject: doesn't tolerate the unescaped/unquoted space.
So you'd have to write  'subject:"(Saúde Geap)"' to get anything sensible.

The other problem is that are treating the part within () as a quoted
phrase, which is not correct.

If you can build from source try the patch below [1]. With it, I think

   notmuch count 'subject:"(Saúde Geap)" OR subject:xplitz'

should behave more like what you want. Notice my use of quotes is bit
different that your original query.

>
> Also note that the fourth query[2] should have AND as the implicit
> boolean operator, because 'subject:' is not a boolean prefix.  You
> explained in the other thread the implicit OR operator only applies for
> search terms with the same /boolean/ prefix.

Yes, this is the same problem as in your other report. I posted some
patches that should fix it, but they need more testing.

[1]:

diff --git a/lib/regexp-fields.cc b/lib/regexp-fields.cc
index 198eb32f..858b1e24 100644
--- a/lib/regexp-fields.cc
+++ b/lib/regexp-fields.cc
@@ -189,6 +189,13 @@ RegexpFieldProcessor::operator() (const std::string & str)
        } else {
            throw Xapian::QueryParserError ("unmatched regex delimiter in '" + str + "'");
        }
+    }  else if (str.at (0) == '(') {
+       if (str.length () > 1 && str.at (str.size () - 1) == ')') {
+           std::string subexp_str = str.substr (1, str.size () - 2);
+           return parser.parse_query (subexp_str, NOTMUCH_QUERY_PARSER_FLAGS, term_prefix);
+       } else {
+           throw Xapian::QueryParserError ("unmatched '(' in '" + str + "'");
+       }
     } else {
        if (options & NOTMUCH_FIELD_PROBABILISTIC) {
            /* TODO replace this with a nicer API level triggering of