[RFC] Precedence of OR and AND

[RFC] Precedence of OR and AND

[Jed Brown]

Currently OR binds more weakly than AND, which is natural in most
contexts, but I think it is rarely desirably for this sort of search.
Suppose I am in looking at my inbox and decide to filter by

  term1 OR term2

Notmuch makes the query

  tag:inbox AND term1 OR term2

which is actually

  (tag:inbox AND term1) OR term2

and not at all what I wanted.  Adding the necessary parentheses to
notmuch-search-filter is trivial but it requires more parentheses for
the overwhelming majority of searches that I think are more common.

Are most searches indeed closer to conjunctive form?

Should OR bind tighter than AND?


Jed

Re: [RFC] Precedence of OR and AND

[Bart Trojanowski]

Wouldn't this problem be solved by each filter appending a bracketed
version of your filter?

You start with tag:inbox and you filter on "term1 or term2" you'd get:

        tag:inbox and (term1 or term2)

Doing it again would result in...

        tag:inbox and (term1 or term2) and (term3 or term4)

To me, it would seem the most intuitive solution.

Actually, I think I'll add that to notmuch.vim right now. :)

-Bart

* Jed Brown <jed@59A2.org> [091122 16:26]:
> Currently OR binds more weakly than AND, which is natural in most
> contexts, but I think it is rarely desirably for this sort of search.
> Suppose I am in looking at my inbox and decide to filter by
> 
>   term1 OR term2
> 
> Notmuch makes the query
> 
>   tag:inbox AND term1 OR term2
> 
> which is actually
> 
>   (tag:inbox AND term1) OR term2
> 
> and not at all what I wanted.  Adding the necessary parentheses to
> notmuch-search-filter is trivial but it requires more parentheses for
> the overwhelming majority of searches that I think are more common.
> 
> Are most searches indeed closer to conjunctive form?
> 
> Should OR bind tighter than AND?
> 
> 
> Jed
> _______________________________________________
> notmuch mailing list
> notmuch@notmuchmail.org
> http://notmuchmail.org/mailman/listinfo/notmuch

-- 
				WebSig: http://www.jukie.net/~bart/sig/

Re: [RFC] Precedence of OR and AND

[Jed Brown]

On Sun, 22 Nov 2009 16:36:49 -0500, Bart Trojanowski <bart@jukie.net> wrote:
> Wouldn't this problem be solved by each filter appending a bracketed
> version of your filter?
> 
> You start with tag:inbox and you filter on "term1 or term2" you'd get:
> 
>         tag:inbox and (term1 or term2)

Absolutely, and I have this applied locally to notmuch.el, but I didn't
fix notmuch-search-filter-by-tag because that would really need to parse
the expression.  I'm just asking if anyone else thinks binding OR
tighter than AND would be desirable.

Jed

Re: [RFC] Precedence of OR and AND

[Bart Trojanowski]

* Jed Brown <jed@59A2.org> [091122 16:43]:
> Absolutely, and I have this applied locally to notmuch.el, but I didn't
> fix notmuch-search-filter-by-tag because that would really need to parse
> the expression.  I'm just asking if anyone else thinks binding OR
> tighter than AND would be desirable.

Maybe some of the ambiguity would go away if we used && and || instead
of AND and OR.  Then no matter the default, we could distinguish if
someone meant 2 or 3 tags when they typed in 'foo and bar'.

-Bart

-- 
				WebSig: http://www.jukie.net/~bart/sig/

Re: [RFC] Precedence of OR and AND

[Carl Worth]

On Sun, 22 Nov 2009 22:43:30 +0100, Jed Brown <jed@59A2.org> wrote:
> On Sun, 22 Nov 2009 16:36:49 -0500, Bart Trojanowski <bart@jukie.net> wrote:
> Absolutely, and I have this applied locally to notmuch.el,

Patch please?

>                                                            but I didn't
> fix notmuch-search-filter-by-tag because that would really need to parse
> the expression.

I don't see the difference here. Any time we append to the search
string, we should be doing so with parentheses.

>                  I'm just asking if anyone else thinks binding OR
> tighter than AND would be desirable.

Right now, Xapian is doing all of our query parsing. So we'd have to
take things up there to get anything changed for now.

In the future we might be forced into writing our own query parser to
get all the functionality we want.

-Carl

[PATCH] Make search filters handle disjunctive queries.

[Jed Brown]

notmuch-search-filter accepts now accepts an arbitrary query and will
group if necessary so that we get

  tag:inbox AND (gravy OR biscuits)

notmuch-search-filter-tag now handles multiple terms.  All terms in the
query except AND and OR are interpreted as tags.

Signed-off-by: Jed Brown <jed@59A2.org>
---
 notmuch.el |   23 ++++++++++++++++++-----
 1 files changed, 18 insertions(+), 5 deletions(-)

diff --git a/notmuch.el b/notmuch.el
index 0cabbe2..43e0566 100644
--- a/notmuch.el
+++ b/notmuch.el
@@ -1057,15 +1057,28 @@ search."
 Runs a new search matching only messages that match both the
 current search results AND the additional query string provided."
   (interactive "sFilter search: ")
-  (notmuch-search (concat notmuch-search-query-string " and " query) notmuch-search-oldest-first))
+  (let ((grouped-query (if (string-match-p "\\<[oO][rR]\\>" query) (concat "( " query " )") query)))
+    (notmuch-search (concat notmuch-search-query-string " and " grouped-query) notmuch-search-oldest-first)))
 
-(defun notmuch-search-filter-by-tag (tag)
-  "Filter the current search results based on a single tag.
+(defun notmuch-search-filter-by-tag (query)
+  "Filter the current search results based on one or more tags.
 
 Runs a new search matching only messages that match both the
-current search results AND that are tagged with the given tag."
+current search results AND that are tagged with the given
+expression involving tags.  For example, the input
+
+  chicken and (gravy or biscuits)
+
+will filter the current search by
+
+  tag:chicken and ( tag:gravy or tag:biscuits )"
   (interactive "sFilter by tag: ")
-  (notmuch-search (concat notmuch-search-query-string " and tag:" tag) notmuch-search-oldest-first))
+  (let ((tagged-query (replace-regexp-in-string "\\([_\+\-]\\|\\w\\)+"
+						(lambda (match) ; Prepend `tag:' to all matches except AND and OR
+						  (if (string-match-p "\\([aA][nN][dD]\\)\\|\\([oO][rR]\\)" match)
+						      match (concat "tag:" match)))
+						query)))
+    (notmuch-search-filter tagged-query)))
 
 (defun notmuch ()
   "Run notmuch to display all mail with tag of 'inbox'"
-- 
1.6.5.3

[PATCH] Make search filters handle disjunctive queries.

[Jed Brown]

notmuch-search-filter accepts now accepts an arbitrary query and will
group if necessary so that we get

  tag:inbox AND (gravy OR biscuits)

notmuch-search-filter-tag now handles multiple terms.  All terms in the
query except AND and OR are interpreted as tags.

This version has nice regexes and handles NOT.

Signed-off-by: Jed Brown <jed@59A2.org>
---
 notmuch.el |   27 ++++++++++++++++++++++-----
 1 files changed, 22 insertions(+), 5 deletions(-)

diff --git a/notmuch.el b/notmuch.el
index 0cabbe2..fdd30ae 100644
--- a/notmuch.el
+++ b/notmuch.el
@@ -828,6 +828,10 @@ thread from that buffer can be show when done with this one)."
 (defvar notmuch-search-query-string)
 (defvar notmuch-search-oldest-first)
 
+(defvar notmuch-search-boolean-operator-regexp "\\([aA][nN][dD]\\|[oO][rR]\\|[nN][oO][tT]\\)")
+(defvar notmuch-search-valid-term-regexp       "\\([-+_.[:word:]]+\\)")
+(defvar notmuch-search-disjunctive-regexp      "\\<[oO][rR]\\>")
+
 (defun notmuch-search-scroll-up ()
   "Scroll up, moving point to last message in thread if at end."
   (interactive)
@@ -1057,15 +1061,28 @@ search."
 Runs a new search matching only messages that match both the
 current search results AND the additional query string provided."
   (interactive "sFilter search: ")
-  (notmuch-search (concat notmuch-search-query-string " and " query) notmuch-search-oldest-first))
+  (let ((grouped-query (if (string-match-p notmuch-search-disjunctive-regexp query) (concat "( " query " )") query)))
+    (notmuch-search (concat notmuch-search-query-string " and " grouped-query) notmuch-search-oldest-first)))
 
-(defun notmuch-search-filter-by-tag (tag)
-  "Filter the current search results based on a single tag.
+(defun notmuch-search-filter-by-tag (query)
+  "Filter the current search results based on one or more tags.
 
 Runs a new search matching only messages that match both the
-current search results AND that are tagged with the given tag."
+current search results AND that are tagged with the given
+expression involving tags.  For example, the input
+
+  chicken and (gravy or biscuits)
+
+will filter the current search by
+
+  tag:chicken and ( tag:gravy or tag:biscuits )"
   (interactive "sFilter by tag: ")
-  (notmuch-search (concat notmuch-search-query-string " and tag:" tag) notmuch-search-oldest-first))
+  (let ((tagged-query (replace-regexp-in-string notmuch-search-valid-term-regexp
+						(lambda (match) ; Prepend `tag:' to all except boolean operators
+						  (if (string-match-p notmuch-search-boolean-operator-regexp match)
+						      match (concat "tag:" match)))
+						query)))
+    (notmuch-search-filter tagged-query)))
 
 (defun notmuch ()
   "Run notmuch to display all mail with tag of 'inbox'"
-- 
1.6.5.3

Re: [PATCH] Make search filters handle disjunctive queries.

[Keith Packard]

[-- Attachment #1: Type: text/plain, Size: 1141 bytes --]

On Mon, 23 Nov 2009 19:07:23 +0100, Jed Brown <jed@59A2.org> wrote:

> notmuch-search-filter accepts now accepts an arbitrary query and will
> group if necessary so that we get
> 
>   tag:inbox AND (gravy OR biscuits)
> 
> notmuch-search-filter-tag now handles multiple terms.  All terms in the
> query except AND and OR are interpreted as tags.

Remember to split patches which do more than one thing into separate
commits.

> +  (let ((grouped-query (if (string-match-p notmuch-search-disjunctive-regexp query) (concat "( " query " )") query)))
> +    (notmuch-search (concat notmuch-search-query-string " and " grouped-query) notmuch-search-oldest-first)))

Is there some reason not to just always add the parens?

> +  (let ((tagged-query (replace-regexp-in-string notmuch-search-valid-term-regexp
> +						(lambda (match) ; Prepend `tag:' to all except boolean operators
> +						  (if (string-match-p notmuch-search-boolean-operator-regexp match)
> +						      match (concat "tag:" match)))
> +						query)))

This seems useful; how does it deal with the tag completion stuff?

-- 
keith.packard@intel.com

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [PATCH] Make search filters handle disjunctive queries.

[Jed Brown]

On Mon, 23 Nov 2009 10:26:47 -0800, Keith Packard <keithp@keithp.com> wrote:

> Remember to split patches which do more than one thing into separate
> commits.

These are variants of the same operation, but I'll split in the future.

> > +  (let ((grouped-query (if (string-match-p notmuch-search-disjunctive-regexp query) (concat "( " query " )") query)))
> > +    (notmuch-search (concat notmuch-search-query-string " and " grouped-query) notmuch-search-oldest-first)))
> 
> Is there some reason not to just always add the parens?

That's what I did initially, but it's messy to look at in the mode line
after applying successive filters.

> This seems useful; how does it deal with the tag completion stuff?

It doesn't do anything special, but I haven't been following that
carefully.  My thought was that eventually the regular interactive
search and filters, would have semantic completion, e.g. the user enters

  tag:ab<TAB>

and tags would be completed, but they could also do

  from:long.unwieldy.addr<TAB>

In filter-by-tag, the natural thing would be to only complete tags, the
user would be on their own to spell out AND, OR, and NOT.  Unfortunately
I'm not familiar enough with elisp to implement this quickly.

A closely related issue is managing the address book.  I guess the usual
advice is to use BBDB.  I don't know if it's better to hook into that or
for Notmuch to have it's own lightweight approach.  I could imagine
harvesting addresses of everyone I've sent mail to and giving me a
semi-automated way to merge addresses that are likely to point to the
same person.


Jed

Re: [PATCH] Make search filters handle disjunctive queries.

[Carl Worth]

[-- Attachment #1: Type: text/plain, Size: 1368 bytes --]

On Mon, 23 Nov 2009 19:07:23 +0100, Jed Brown <jed@59A2.org> wrote:
> notmuch-search-filter accepts now accepts an arbitrary query and will
> group if necessary so that we get
> 
>   tag:inbox AND (gravy OR biscuits)
> 
> notmuch-search-filter-tag now handles multiple terms.  All terms in the
> query except AND and OR are interpreted as tags.

Hi Jed,

Sorry this one has sat in my queue for a while. But I finally got around
to it.

The above commit message (and the patch below) really do implement two
separate features, so really should be separated into two separate
commits. I do like both of the ideas, and will be glad to commit both
when they arrive separate.

> This version has nice regexes and handles NOT.

And that kind of commentary, (referring to a previous revision of the
patch), is great in an email discussion like we're having here, but
won't make any sense within the context of the log of commit
messages. So such commentary is best placed *after* the "---" separator.

> Signed-off-by: Jed Brown <jed@59A2.org>
> ---

That is, the commentary would go here in the original message, and then
I won't have to "commit --amend" the commentary away after applying the
patch.

Thanks,

-Carl

PS. Hmm... "notmuch reply" needs to be more clever to avoid duplicate
addresses in the To: and Cc: lines. See above.

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [PATCH] Make search filters handle disjunctive queries.

[Jed Brown]

A patch for notmuch-search-filter follows, my change to
notmuch-search-filter-by-tag is not very useful since
notmuch-select-tag-with-completion does not allow a space to be
inserted.  I don't know how to get completion on multiple
space-separated terms.

On Tue, 01 Dec 2009 18:56:59 -0800, Carl Worth <cworth@cworth.org> wrote:
> PS. Hmm... "notmuch reply" needs to be more clever to avoid duplicate
> addresses in the To: and Cc: lines. See above.

Yes, uniquifying the headers actually requires some effort, unless I'm
missing something in the gmime API (which I've never looked at before).

Jed

[PATCH] Make search filters handle disjunctive queries.

[Jed Brown]

notmuch-search-filter now accepts an arbitrary query and will group if
necessary so that we get

  tag:inbox AND (gravy OR biscuits)

instead of the former

  tag:inbox AND gravy OR biscuits

Signed-off-by: Jed Brown <jed@59A2.org>
---
 notmuch.el |    5 ++++-
 1 files changed, 4 insertions(+), 1 deletions(-)

diff --git a/notmuch.el b/notmuch.el
index 2509651..0bf82f5 100644
--- a/notmuch.el
+++ b/notmuch.el
@@ -995,6 +995,8 @@ thread from that buffer can be show when done with this one)."
 (defvar notmuch-search-oldest-first t
   "Show the oldest mail first in the search-mode")
 
+(defvar notmuch-search-disjunctive-regexp      "\\<[oO][rR]\\>")
+
 (defun notmuch-search-scroll-up ()
   "Move forward through search results by one window's worth."
   (interactive)
@@ -1327,7 +1329,8 @@ search."
 Runs a new search matching only messages that match both the
 current search results AND the additional query string provided."
   (interactive "sFilter search: ")
-  (notmuch-search (concat notmuch-search-query-string " and " query) notmuch-search-oldest-first))
+  (let ((grouped-query (if (string-match-p notmuch-search-disjunctive-regexp query) (concat "( " query " )") query)))
+    (notmuch-search (concat notmuch-search-query-string " and " grouped-query) notmuch-search-oldest-first)))
 
 (defun notmuch-search-filter-by-tag (tag)
   "Filter the current search results based on a single tag.
-- 
1.6.5.3

Re: [PATCH] Make search filters handle disjunctive queries.

[Jan Janak]

On 02-12 11:59, Jed Brown wrote:
> A patch for notmuch-search-filter follows, my change to
> notmuch-search-filter-by-tag is not very useful since
> notmuch-select-tag-with-completion does not allow a space to be
> inserted.  I don't know how to get completion on multiple
> space-separated terms.

notmuch-select-tag-with-completion uses "\n" as tag separator. This is because
it gets the output of 'notmuch search-tags' as input and there is one tag on a
line.

I haven't been really following this thread in detail. What is that you need
from notmuch-select-tag-with-completion? To be able to process a list of tags
separated by spaces? Maybe I could help you with that.

-Jan

Re: [PATCH] Make search filters handle disjunctive queries.

[Jed Brown]

On Wed, 2 Dec 2009 14:18:08 +0100, Jan Janak <jan@ryngle.com> wrote:
> I haven't been really following this thread in detail. What is that you need
> from notmuch-select-tag-with-completion? To be able to process a list of tags
> separated by spaces? Maybe I could help you with that.

No, it would need to take input separated by spaces, now I just get `[No
match]' when I try to enter a space.  I think it's just not how
completing-read works, the idea would be for the user to be able to type

  shorttag and not long<TAB>

which would complete to

  shorttag and not longboringtagname

The point of my patch was to be able to filter using arbitrary
expressions where every non-operator, i.e. `and', `or', `not', `(', and
`)', is interpreted as a tag name.  Is there an easy way to accept this
input without sacrificing completion?

Jed

Re: [PATCH] Make search filters handle disjunctive queries.

[Jan Janak]

Jed,

On 02-12 14:46, Jed Brown wrote:
> On Wed, 2 Dec 2009 14:18:08 +0100, Jan Janak <jan@ryngle.com> wrote:
> > I haven't been really following this thread in detail. What is that you need
> > from notmuch-select-tag-with-completion? To be able to process a list of tags
> > separated by spaces? Maybe I could help you with that.
> 
> No, it would need to take input separated by spaces, now I just get `[No
> match]' when I try to enter a space.  I think it's just not how
> completing-read works, the idea would be for the user to be able to type
> 
>   shorttag and not long<TAB>
> 
> which would complete to
> 
>   shorttag and not longboringtagname

I see.

> The point of my patch was to be able to filter using arbitrary
> expressions where every non-operator, i.e. `and', `or', `not', `(', and
> `)', is interpreted as a tag name.  Is there an easy way to accept this
> input without sacrificing completion?

I am not completely sure, but there might be a way to do this with some of the
low-level completion functions. For example, the documentation on completion
styles:

http://www.gnu.org/software/emacs/elisp/html_node/Completion-Styles.html#Completion-Styles

mentions in the last paragraph that there is a style which completes each word
in the input separately. This can be enabled with (partial-completion-mode),
and indeed, if I enable this mode, try to visit a file and type ~/.n-c<TAB> then it correctly completes ~/.notmuch-config. 

This seems close enough to what you want and maybe we could somehow make it
work with notmuch, too. This would be interesting to have, because then we
could complete multiple tag names with a single TAB press not only in your
expressions, but everywhere.

Right now we can only add one tag at a time with notmuch-search-add-tag and
similar functions. If we could make partial completion work, we could add
several tags with one command and still have their names completed.

Let me know if you do not plan to work on this and I'll see if I can make it
work.

-Jan

Re: [PATCH] Make search filters handle disjunctive queries.

[Jed Brown]

Jan, thanks for looking into this.

On Wed, 2 Dec 2009 15:38:54 +0100, Jan Janak <jan@ryngle.com> wrote:
> This seems close enough to what you want and maybe we could somehow make it
> work with notmuch, too. This would be interesting to have, because then we
> could complete multiple tag names with a single TAB press not only in your
> expressions, but everywhere.

Yes, I think that is desirable.

> Let me know if you do not plan to work on this and I'll see if I can make it
> work.

I'm not using tags heavily at the moment, and am busy with other things
so I'm unlikely to get to it in the next couple weeks.  If you can make
completion work with this sort of input, I'll update my filter-by-tags
patch.

Jed

Re: [PATCH] Make search filters handle disjunctive queries.

[Carl Worth]

[-- Attachment #1: Type: text/plain, Size: 383 bytes --]

On Wed,  2 Dec 2009 12:00:35 +0100, Jed Brown <jed@59A2.org> wrote:
> notmuch-search-filter now accepts an arbitrary query and will group if
> necessary so that we get
> 
>   tag:inbox AND (gravy OR biscuits)
> 
> instead of the former
> 
>   tag:inbox AND gravy OR biscuits

Perfect. A nice clean patch for just the one feature. Thanks!

I've pushed this now.

-Carl

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]