[PATCH] doc: add details about Xapian search syntax

[PATCH] doc: add details about Xapian search syntax

[David Bremner]

Questions related to the way that probabilistic prefixes and phrases
are handled come up quite often and it is nicer to have the documentation self contained.  Hopefully putting it in subsections prevents it from being overwhelming.
---
 doc/man7/notmuch-search-terms.rst | 46 +++++++++++++++++++++++++++++++++++++++
 1 file changed, 46 insertions(+)

diff --git a/doc/man7/notmuch-search-terms.rst b/doc/man7/notmuch-search-terms.rst
index 1acdaa0..5c65938 100644
--- a/doc/man7/notmuch-search-terms.rst
+++ b/doc/man7/notmuch-search-terms.rst
@@ -129,6 +129,52 @@ operators, but will have to be protected from interpretation by the
 shell, (such as by putting quotation marks around any parenthesized
 expression).
 
+Boolean and Probabilistic Prefixes
+----------------------------------
+
+Xapian (and hence notmuch) prefixes are either **boolean**, supporting
+exact matches like "tag:inbox"  or **probabilistic**, supporting a most flexible **term** based searching. The pre
+
++------------------+-----------------------+
+|Boolean           |Probabilistic          |
++------------------+-----------------------+
+| **tag:** **id:** | **from:** **to:**     |
+|**thread:**       |**subject:**           |
+|**folder:**       |**attachment**         |
+|**path:**         |                       |
+|                  |                       |
++------------------+-----------------------+
+
+Terms and phrases
+-----------------
+
+In general Xapian distinguishes between lists of terms and
+**phrases**. Phrases are indicated by double quotes (but beware you
+probably need to protect those from your shell) and insist that those
+unstemmed words occur in that order. One useful, but initially
+surprising feature is that the following are equivalant ways to write
+the same phrase.
+
+- "a list of words"
+- a-list-of-words
+- a/list/of/words
+- a.list.of.words
+
+Both parenthesised lists of terms and quoted phrases are ok with
+probabilisitic prefixes such as **to:**, **from:**, and **subject:**. In particular
+
+   subject:(pizza free)
+
+is equivalent to
+
+   subject:pizza and subject:free
+
+Both of these will match a subject "Free Delicious Pizza" while
+
+   subject:"pizza free"
+
+will not.
+
 DATE AND TIME SEARCH
 ====================
 
-- 
2.1.4

Re: [PATCH] doc: add details about Xapian search syntax

[Jani Nikula]

On Sun, 18 Jan 2015, David Bremner <david@tethera.net> wrote:
> Questions related to the way that probabilistic prefixes and phrases
> are handled come up quite often and it is nicer to have the documentation self contained.  Hopefully putting it in subsections prevents it from being overwhelming.

I think this is a good idea.

> ---
>  doc/man7/notmuch-search-terms.rst | 46 +++++++++++++++++++++++++++++++++++++++
>  1 file changed, 46 insertions(+)
>
> diff --git a/doc/man7/notmuch-search-terms.rst b/doc/man7/notmuch-search-terms.rst
> index 1acdaa0..5c65938 100644
> --- a/doc/man7/notmuch-search-terms.rst
> +++ b/doc/man7/notmuch-search-terms.rst
> @@ -129,6 +129,52 @@ operators, but will have to be protected from interpretation by the
>  shell, (such as by putting quotation marks around any parenthesized
>  expression).
>  
> +Boolean and Probabilistic Prefixes
> +----------------------------------
> +
> +Xapian (and hence notmuch) prefixes are either **boolean**, supporting
> +exact matches like "tag:inbox"  or **probabilistic**, supporting a most flexible **term** based searching. The pre

The pre?

> +
> ++------------------+-----------------------+
> +|Boolean           |Probabilistic          |
> ++------------------+-----------------------+
> +| **tag:** **id:** | **from:** **to:**     |
> +|**thread:**       |**subject:**           |
> +|**folder:**       |**attachment**         |
> +|**path:**         |                       |
> +|                  |                       |
> ++------------------+-----------------------+
> +
> +Terms and phrases
> +-----------------
> +
> +In general Xapian distinguishes between lists of terms and
> +**phrases**. Phrases are indicated by double quotes (but beware you
> +probably need to protect those from your shell) and insist that those
> +unstemmed words occur in that order. One useful, but initially
> +surprising feature is that the following are equivalant ways to write
> +the same phrase.
> +
> +- "a list of words"
> +- a-list-of-words
> +- a/list/of/words
> +- a.list.of.words
> +
> +Both parenthesised lists of terms and quoted phrases are ok with
> +probabilisitic prefixes such as **to:**, **from:**, and **subject:**. In particular
> +
> +   subject:(pizza free)
> +
> +is equivalent to
> +
> +   subject:pizza and subject:free
> +
> +Both of these will match a subject "Free Delicious Pizza" while
> +
> +   subject:"pizza free"
> +
> +will not.
> +

While at it, I think it would be useful to incorporate relevant parts of
http://notmuchmail.org/searching/ into the man page.

BR,
Jani.


>  DATE AND TIME SEARCH
>  ====================
>  
> -- 
> 2.1.4
>
> _______________________________________________
> notmuch mailing list
> notmuch@notmuchmail.org
> http://notmuchmail.org/mailman/listinfo/notmuch

[PATCH 1/3] doc: add details about Xapian search syntax

[David Bremner]

Questions related to the way that probabilistic prefixes and phrases
are handled come up quite often and it is nicer to have the documentation self contained.  Hopefully putting it in subsections prevents it from being overwhelming.
---
 doc/man7/notmuch-search-terms.rst | 52 +++++++++++++++++++++++++++++++++++++++
 1 file changed, 52 insertions(+)

diff --git a/doc/man7/notmuch-search-terms.rst b/doc/man7/notmuch-search-terms.rst
index 1acdaa0..52f6804 100644
--- a/doc/man7/notmuch-search-terms.rst
+++ b/doc/man7/notmuch-search-terms.rst
@@ -129,6 +129,58 @@ operators, but will have to be protected from interpretation by the
 shell, (such as by putting quotation marks around any parenthesized
 expression).
 
+Boolean and Probabilistic Prefixes
+----------------------------------
+
+Xapian (and hence notmuch) prefixes are either **boolean**, supporting
+exact matches like "tag:inbox"  or **probabilistic**, supporting a most flexible **term** based searching. The prefixes currently supported by notmuch are as follows.
+
++------------------+-----------------------+
+|Boolean           |Probabilistic          |
++------------------+-----------------------+
+| **tag:** **id:** | **from:** **to:**     |
+|**thread:**       |**subject:**           |
+|**folder:**       |**attachment**         |
+|**path:**         |                       |
+|                  |                       |
++------------------+-----------------------+
+
+Terms and phrases
+-----------------
+
+In general Xapian distinguishes between lists of terms and
+**phrases**. Phrases are indicated by double quotes (but beware you
+probably need to protect those from your shell) and insist that those
+unstemmed words occur in that order. One useful, but initially
+surprising feature is that the following are equivalant ways to write
+the same phrase.
+
+- "a list of words"
+- a-list-of-words
+- a/list/of/words
+- a.list.of.words
+
+Both parenthesised lists of terms and quoted phrases are ok with
+probabilisitic prefixes such as **to:**, **from:**, and **subject:**. In particular
+
+::
+
+   subject:(pizza free)
+
+is equivalent to
+
+::
+
+   subject:pizza and subject:free
+
+Both of these will match a subject "Free Delicious Pizza" while
+
+::
+
+   subject:"pizza free"
+
+will not.
+
 DATE AND TIME SEARCH
 ====================
 
-- 
2.1.4

[PATCH 2/3] doc: add material on stemming and wildcards

[David Bremner]

This is lightly massaged from the searching page on the wiki.
---
 doc/man7/notmuch-search-terms.rst | 28 ++++++++++++++++++++++++++++
 1 file changed, 28 insertions(+)

diff --git a/doc/man7/notmuch-search-terms.rst b/doc/man7/notmuch-search-terms.rst
index 52f6804..3b27053 100644
--- a/doc/man7/notmuch-search-terms.rst
+++ b/doc/man7/notmuch-search-terms.rst
@@ -129,6 +129,34 @@ operators, but will have to be protected from interpretation by the
 shell, (such as by putting quotation marks around any parenthesized
 expression).
 
+Stemming
+--------
+
+**Stemming** in notmuch means is that these searches
+
+::
+
+        notmuch search detailed
+        notmuch search details
+        notmuch search detail
+
+will all return identical results, because Xapian first "reduces" the
+term to the common stem (here 'detail') and then performs the search.
+
+There are two ways to turn this off: a search for a capitalized word
+will be performed unstemmed, so that one can search for "John" and not
+get results for "Johnson"; phrase searches are also unstemmed (see
+below for details).  Stemming is currently only supported for
+English. Words in other languages will be performed unstemmed unless
+somebody teaches Xapian how to perform stemming for that language.
+
+Wildcards
+---------
+
+It is possible to use a trailing '\*' as a wildcard. A search for
+'wildc\*' will match 'wildcard', 'wildcat', etc.
+
+
 Boolean and Probabilistic Prefixes
 ----------------------------------
 
-- 
2.1.4

[PATCH 3/3] doc: add more information on operators.

[David Bremner]

More material borrowed from the wiki page on "searching"
---
 doc/man7/notmuch-search-terms.rst | 43 ++++++++++++++++++++++++++++++++++-----
 1 file changed, 38 insertions(+), 5 deletions(-)

diff --git a/doc/man7/notmuch-search-terms.rst b/doc/man7/notmuch-search-terms.rst
index 3b27053..a315fbe 100644
--- a/doc/man7/notmuch-search-terms.rst
+++ b/doc/man7/notmuch-search-terms.rst
@@ -118,17 +118,50 @@ The time range can also be specified using timestamps with a syntax of:
 Each timestamp is a number representing the number of seconds since
 1970-01-01 00:00:00 UTC.
 
-In addition to individual terms, multiple terms can be combined with
-Boolean operators ( **and**, **or**, **not** , etc.). Each term in the
-query will be implicitly connected by a logical AND if no explicit
-operator is provided, (except that terms with a common prefix will be
-implicitly combined with OR until we get Xapian defect #402 fixed).
+Operators
+---------
 
+In addition to individual terms, multiple terms can be combined with
+Boolean operators ( **and**, **or**, **not**, and **xor**). Each term
+in the query will be implicitly connected by a logical AND if no
+explicit operator is provided, (except that terms with a common prefix
+will be implicitly combined with OR until we get Xapian defect #402
+fixed).  The shorthand '-<term>' can be used for 'not <term>' but
+unfortunately this does not work at the start of an expression.
 Parentheses can also be used to control the combination of the Boolean
 operators, but will have to be protected from interpretation by the
 shell, (such as by putting quotation marks around any parenthesized
 expression).
 
+In addition to the standard boolean operators, Xapian provides several
+operators specific to text searching.
+
+::
+
+        notmuch search term1 NEAR term2
+
+will return results where term1 is within 10 words of term2. The
+threshold can be set like this:
+
+::
+
+        notmuch search term1 NEAR/2 term2
+
+The search
+
+::
+
+        notmuch search term1 ADJ term2
+
+will return results where term1 is within 10 words of term2, but in the
+same order as in the query. The threshold can be set the same as with
+NEAR:
+
+::
+
+        notmuch search term1 ADJ/7 term2
+
+
 Stemming
 --------
 
-- 
2.1.4

Re: [PATCH 1/3] doc: add details about Xapian search syntax

[David Bremner]

David Bremner <david@tethera.net> writes:

> Questions related to the way that probabilistic prefixes and phrases
> are handled come up quite often and it is nicer to have the documentation self contained.  Hopefully putting it in subsections prevents it from being overwhelming.

Pushed the first patch to master.

d

Re: [PATCH 2/3] doc: add material on stemming and wildcards

[Jani Nikula]

On Sun, 18 Jan 2015, David Bremner <david@tethera.net> wrote:
> This is lightly massaged from the searching page on the wiki.
> ---
>  doc/man7/notmuch-search-terms.rst | 28 ++++++++++++++++++++++++++++
>  1 file changed, 28 insertions(+)
>
> diff --git a/doc/man7/notmuch-search-terms.rst b/doc/man7/notmuch-search-terms.rst
> index 52f6804..3b27053 100644
> --- a/doc/man7/notmuch-search-terms.rst
> +++ b/doc/man7/notmuch-search-terms.rst
> @@ -129,6 +129,34 @@ operators, but will have to be protected from interpretation by the
>  shell, (such as by putting quotation marks around any parenthesized
>  expression).
>  
> +Stemming
> +--------
> +
> +**Stemming** in notmuch means is that these searches

Unnecessary "is"?

> +
> +::
> +
> +        notmuch search detailed
> +        notmuch search details
> +        notmuch search detail
> +
> +will all return identical results, because Xapian first "reduces" the
> +term to the common stem (here 'detail') and then performs the search.
> +
> +There are two ways to turn this off: a search for a capitalized word
> +will be performed unstemmed, so that one can search for "John" and not
> +get results for "Johnson"; phrase searches are also unstemmed (see
> +below for details).  Stemming is currently only supported for
> +English. Words in other languages will be performed unstemmed unless

Maybe end the sentence here, and drop this line:

> +somebody teaches Xapian how to perform stemming for that language.
> +
> +Wildcards
> +---------
> +
> +It is possible to use a trailing '\*' as a wildcard. A search for
> +'wildc\*' will match 'wildcard', 'wildcat', etc.
> +
> +
>  Boolean and Probabilistic Prefixes
>  ----------------------------------
>  
> -- 
> 2.1.4

Re: [PATCH 3/3] doc: add more information on operators.

[Jani Nikula]

On Sun, 18 Jan 2015, David Bremner <david@tethera.net> wrote:
> More material borrowed from the wiki page on "searching"
> ---
>  doc/man7/notmuch-search-terms.rst | 43 ++++++++++++++++++++++++++++++++++-----
>  1 file changed, 38 insertions(+), 5 deletions(-)
>
> diff --git a/doc/man7/notmuch-search-terms.rst b/doc/man7/notmuch-search-terms.rst
> index 3b27053..a315fbe 100644
> --- a/doc/man7/notmuch-search-terms.rst
> +++ b/doc/man7/notmuch-search-terms.rst
> @@ -118,17 +118,50 @@ The time range can also be specified using timestamps with a syntax of:
>  Each timestamp is a number representing the number of seconds since
>  1970-01-01 00:00:00 UTC.
>  
> -In addition to individual terms, multiple terms can be combined with
> -Boolean operators ( **and**, **or**, **not** , etc.). Each term in the
> -query will be implicitly connected by a logical AND if no explicit
> -operator is provided, (except that terms with a common prefix will be
> -implicitly combined with OR until we get Xapian defect #402 fixed).
> +Operators
> +---------
>  
> +In addition to individual terms, multiple terms can be combined with
> +Boolean operators ( **and**, **or**, **not**, and **xor**). Each term

Extra space after (.

> +in the query will be implicitly connected by a logical AND if no
> +explicit operator is provided, (except that terms with a common prefix
> +will be implicitly combined with OR until we get Xapian defect #402
> +fixed).  The shorthand '-<term>' can be used for 'not <term>' but
> +unfortunately this does not work at the start of an expression.

I think people expect tag:inbox tag:unread to have implicit OR by now,
so we could just drop the reference to the xapian defect.

Otherwise these patches LGTM, and I like getting this info into the man
page rather than the wiki.

BR,
Jani.



>  Parentheses can also be used to control the combination of the Boolean
>  operators, but will have to be protected from interpretation by the
>  shell, (such as by putting quotation marks around any parenthesized
>  expression).
>  
> +In addition to the standard boolean operators, Xapian provides several
> +operators specific to text searching.
> +
> +::
> +
> +        notmuch search term1 NEAR term2
> +
> +will return results where term1 is within 10 words of term2. The
> +threshold can be set like this:
> +
> +::
> +
> +        notmuch search term1 NEAR/2 term2
> +
> +The search
> +
> +::
> +
> +        notmuch search term1 ADJ term2
> +
> +will return results where term1 is within 10 words of term2, but in the
> +same order as in the query. The threshold can be set the same as with
> +NEAR:
> +
> +::
> +
> +        notmuch search term1 ADJ/7 term2
> +
> +
>  Stemming
>  --------
>  
> -- 
> 2.1.4

(no subject)

[David Bremner]

This has Jani's suggestions fixed, along with a couple of other trivial patches.

I'll mark them ready at this point.

[Patch v2 1/4] doc: add material on stemming and wildcards

[David Bremner]

This is lightly massaged from the searching page on the wiki.
---
 doc/man7/notmuch-search-terms.rst | 27 +++++++++++++++++++++++++++
 1 file changed, 27 insertions(+)

diff --git a/doc/man7/notmuch-search-terms.rst b/doc/man7/notmuch-search-terms.rst
index 54138d2..7fb4a27 100644
--- a/doc/man7/notmuch-search-terms.rst
+++ b/doc/man7/notmuch-search-terms.rst
@@ -135,6 +135,33 @@ operators, but will have to be protected from interpretation by the
 shell, (such as by putting quotation marks around any parenthesized
 expression).
 
+Stemming
+--------
+
+**Stemming** in notmuch means that these searches
+
+::
+
+        notmuch search detailed
+        notmuch search details
+        notmuch search detail
+
+will all return identical results, because Xapian first "reduces" the
+term to the common stem (here 'detail') and then performs the search.
+
+There are two ways to turn this off: a search for a capitalized word
+will be performed unstemmed, so that one can search for "John" and not
+get results for "Johnson"; phrase searches are also unstemmed (see
+below for details).  Stemming is currently only supported for
+English. Searches for words in other languages will be performed unstemmed.
+
+Wildcards
+---------
+
+It is possible to use a trailing '\*' as a wildcard. A search for
+'wildc\*' will match 'wildcard', 'wildcat', etc.
+
+
 Boolean and Probabilistic Prefixes
 ----------------------------------
 
-- 
2.1.4

[Patch v2 2/4] doc: add more information on operators.

[David Bremner]

More material borrowed from the wiki page on "searching"
---
 doc/man7/notmuch-search-terms.rst | 42 ++++++++++++++++++++++++++++++++++-----
 1 file changed, 37 insertions(+), 5 deletions(-)

diff --git a/doc/man7/notmuch-search-terms.rst b/doc/man7/notmuch-search-terms.rst
index 7fb4a27..1bd3df5 100644
--- a/doc/man7/notmuch-search-terms.rst
+++ b/doc/man7/notmuch-search-terms.rst
@@ -124,17 +124,49 @@ The time range can also be specified using timestamps with a syntax of:
 Each timestamp is a number representing the number of seconds since
 1970-01-01 00:00:00 UTC.
 
-In addition to individual terms, multiple terms can be combined with
-Boolean operators ( **and**, **or**, **not** , etc.). Each term in the
-query will be implicitly connected by a logical AND if no explicit
-operator is provided, (except that terms with a common prefix will be
-implicitly combined with OR until we get Xapian defect #402 fixed).
+Operators
+---------
 
+In addition to individual terms, multiple terms can be combined with
+Boolean operators (**and**, **or**, **not**, and **xor**). Each term
+in the query will be implicitly connected by a logical AND if no
+explicit operator is provided, (except that terms with a common prefix
+will be implicitly combined with OR).  The shorthand '-<term>' can be used for 'not <term>' but
+unfortunately this does not work at the start of an expression.
 Parentheses can also be used to control the combination of the Boolean
 operators, but will have to be protected from interpretation by the
 shell, (such as by putting quotation marks around any parenthesized
 expression).
 
+In addition to the standard boolean operators, Xapian provides several
+operators specific to text searching.
+
+::
+
+        notmuch search term1 NEAR term2
+
+will return results where term1 is within 10 words of term2. The
+threshold can be set like this:
+
+::
+
+        notmuch search term1 NEAR/2 term2
+
+The search
+
+::
+
+        notmuch search term1 ADJ term2
+
+will return results where term1 is within 10 words of term2, but in the
+same order as in the query. The threshold can be set the same as with
+NEAR:
+
+::
+
+        notmuch search term1 ADJ/7 term2
+
+
 Stemming
 --------
 
-- 
2.1.4

[Patch v2 3/4] doc: typo fix for prefix discussion.

[David Bremner]

---
 doc/man7/notmuch-search-terms.rst | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/doc/man7/notmuch-search-terms.rst b/doc/man7/notmuch-search-terms.rst
index 1bd3df5..77abc70 100644
--- a/doc/man7/notmuch-search-terms.rst
+++ b/doc/man7/notmuch-search-terms.rst
@@ -198,7 +198,7 @@ Boolean and Probabilistic Prefixes
 ----------------------------------
 
 Xapian (and hence notmuch) prefixes are either **boolean**, supporting
-exact matches like "tag:inbox"  or **probabilistic**, supporting a most flexible **term** based searching. The prefixes currently supported by notmuch are as follows.
+exact matches like "tag:inbox"  or **probabilistic**, supporting a more flexible **term** based searching. The prefixes currently supported by notmuch are as follows.
 
 +------------------+-----------------------+
 |Boolean           |Probabilistic          |
-- 
2.1.4

[Patch v2 4/4] doc: update list of prefixes

[David Bremner]

'attachement' missed a colon, and 'mimetype' was not added to this table
at all.
---
 doc/man7/notmuch-search-terms.rst | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/doc/man7/notmuch-search-terms.rst b/doc/man7/notmuch-search-terms.rst
index 77abc70..ebb5e54 100644
--- a/doc/man7/notmuch-search-terms.rst
+++ b/doc/man7/notmuch-search-terms.rst
@@ -205,8 +205,8 @@ exact matches like "tag:inbox"  or **probabilistic**, supporting a more flexible
 +------------------+-----------------------+
 | **tag:** **id:** | **from:** **to:**     |
 |**thread:**       |**subject:**           |
-|**folder:**       |**attachment**         |
-|**path:**         |                       |
+|**folder:**       |**attachment:**        |
+|**path:**         |**mimetype:**          |
 |                  |                       |
 +------------------+-----------------------+
 
-- 
2.1.4

Re: [Patch v2 2/4] doc: add more information on operators.

[W. Trevor King]

[-- Attachment #1: Type: text/plain, Size: 1064 bytes --]

On Mon, Feb 23, 2015 at 09:05:35PM +0100, David Bremner wrote:
> +In addition to individual terms, multiple terms can be combined with
> +Boolean operators (**and**, **or**, **not**, and **xor**). Each term
> +in the query will be implicitly connected by a logical AND if no
> +explicit operator is provided, (except that terms with a common prefix
> +will be implicitly combined with OR).  The shorthand '-<term>' can be used for 'not <term>' but
> +unfortunately this does not work at the start of an expression.
>  Parentheses can also be used to control the combination of the Boolean
>  operators, but will have to be protected from interpretation by the
>  shell, (such as by putting quotation marks around any parenthesized
>  expression).

We should probably:

* Re-wrap the paragraph to avoid one long line.
* Drop the comma before the “(except …)” parens.

Cheers,
Trevor

-- 
This email may be signed or encrypted with GnuPG (http://www.gnupg.org).
For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re:

[David Bremner]

David Bremner <david@tethera.net> writes:

> This has Jani's suggestions fixed, along with a couple of other trivial patches.
>
> I'll mark them ready at this point.

I pushed these, with Trevor's suggested changes.

d