Folder-based searching

Folder-based searching

[Carl Worth]

I'm as sad as anyone to see that I let more than a month go by without
committing to the notmuch repository.

But I've just now committed the much-desired folder-based search
feature.

This should be a handy feature for people bringing mail in from a system
that stores information in the directories in which mail is stored, (one
common case is users fetching mail from gmail via imap---in this case
all gmail tags will be represented as directories).

Someone bringing in mail from such a system might want to migrate from
information in directory names to instead be information in notmuch
tags. That might look something like this:

	notmuch tag +important folder:important

Or it might be useful to run on several directories:

	for dir in *; do notmuch tag +"$dir"; folder:"$dir"; done

etc.

Please note that with the code in git currently, only newly incorporated
mail will be searchable via the "folder:" specification. So if you're
wanting to search your entire email collection, you may want to rebuild
your entire notmuch database. As a reminder, that process is something
like the following:

    notmuch dump > notmuch.dump
    rm -rf ~/mail/.notmuch	# Careful with this! Have backups in place!
    notmuch new			# Of course, this can take a long time
    notmuch restore notmuch.dump

Before the next release, I plan to increment the database version so
that users will be notified of the need to do an upgrade like this. (But
before I do that, I plan to implement a few more database changes---and
some of them are likely to be much more invasive than this one).

Have fun out there!

-Carl

Re: Folder-based searching

[Xavier Maillard]

Hi,

On Sat, 15 Jan 2011 14:46:53 -0800, Carl Worth <cworth@cworth.org> wrote:
> I'm as sad as anyone to see that I let more than a month go by without
> committing to the notmuch repository.

Life goes like this...
 
> But I've just now committed the much-desired folder-based search
> feature.

Awesome ! 

> This should be a handy feature for people bringing mail in from a system
> that stores information in the directories in which mail is stored, (one
> common case is users fetching mail from gmail via imap---in this case
> all gmail tags will be represented as directories).

Is Imap with SIEVE scripts such a common case too ?

/Xavier

Re: Folder-based searching

[Sebastian Spaeth]

> Before the next release, I plan to increment the database version so
> that users will be notified of the need to do an upgrade like this. (But
> before I do that, I plan to implement a few more database changes---and
> some of them are likely to be much more invasive than this one).

Hurray, just tried it out and it works like a charm. Finally I can
automatically assign the "Junk" tag again. :-)

Sebastian

Re: Folder-based searching

[Sebastian Spaeth]

[-- Attachment #1: Type: text/plain, Size: 1178 bytes --]

On Mon, 17 Jan 2011 13:43:44 +0100, Sebastian Spaeth <Sebastian@SSpaeth.de> wrote:
> > Before the next release, I plan to increment the database version so
> > that users will be notified of the need to do an upgrade like this. (But
> > before I do that, I plan to implement a few more database changes---and
> > some of them are likely to be much more invasive than this one).
> 
> Hurray, just tried it out and it works like a charm. Finally I can
> automatically assign the "Junk" tag again. :-)

Well, it's never as easy as it looks at first glance. Initial run picks
up folder:junk and reports correctly (is it supposed to be
case-insensitive?).

However, when moving files into another folder and rerunning notmuch
new, the folder value is never updated, it seems. This is what I run to
move mails that had been wrongly classified as Junk and that I manually
untagged into my INBOX:

notmuch search --output=files not tag:junk and folder:junk | xargs mv -t
/home/spaetz/mail/INBOX/cur/

However, each subsequent run of:
notmuch search --output=files not tag:junk and folder:junk

will show the same files, even after a notmuch new...

Sebastian

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: Folder-based searching

[Rob Browning]

Carl Worth <cworth@cworth.org> writes:

> Someone bringing in mail from such a system might want to migrate from
> information in directory names to instead be information in notmuch
> tags. That might look something like this:
>
> 	notmuch tag +important folder:important

Nice.  So what are the path semantics?  For example, is the path case
sensitive or insensitive?  Can you say folder:foo/bar, and if so, is the
path rooted, or can it match path sub-segments?

i.e. which of these, if any, is "folder:foo/bar" like?

  (.*/)?foo/bar(/.*)?
  ^foo/bar$
  ^foo/bar(/.*)?

Thanks
-- 
Rob Browning
rlb @defaultvalue.org and @debian.org
GPG as of 2002-11-03 14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4

Re: Folder-based searching

[Carl Worth]

[-- Attachment #1: Type: text/plain, Size: 1099 bytes --]

On Mon, 17 Jan 2011 12:48:11 -0600, Rob Browning <rlb@defaultvalue.org> wrote:
> Nice.

Thanks.

> So what are the path semantics?  For example, is the path case
> sensitive or insensitive?  Can you say folder:foo/bar, and if so, is the
> path rooted, or can it match path sub-segments?
> 
> i.e. which of these, if any, is "folder:foo/bar" like?
> 
>   (.*/)?foo/bar(/.*)?

That's the right one.

It's a case-insensitive, non-rooted search specification.

This works in a similar way to 'subject:"some phrase"', (though more
people seem to be asking about these details for folder: than have ever
asked for subject:).

This is the easiest kind of searching to get out of Xapian. However, in
the future, I would like to enable users to specify rooted search
specifications, (whether with folder:, subject:, or any other prefix).

That will require a little care to get some additional terms indexed to
support the rooting, then the in-development custom query parser to
allow mapping symbols like '^' and '$' to these new symbols.

-Carl

-- 
carl.d.worth@intel.com

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: Folder-based searching

[Rob Browning]

Carl Worth <cworth@cworth.org> writes:

> This works in a similar way to 'subject:"some phrase"', (though more
> people seem to be asking about these details for folder: than have ever
> asked for subject:).

I'm not surprised.  Imagine:

  /misc
  /debian/misc

If I understand correctly, right now you'd have to say

  folder:misc and not folder:debian/misc

which is not nearly as handy as

  folder:^misc 

and at first, a lot of people may not even realize that they need the
"and not" clauses.

I'd be tempted to consider making folder: searches rooted by default.  I
wonder how often people really want "all folders named misc"?

> That will require a little care to get some additional terms indexed to
> support the rooting, then the in-development custom query parser to
> allow mapping symbols like '^' and '$' to these new symbols.

Sounds good.

Thanks
-- 
Rob Browning
rlb @defaultvalue.org and @debian.org
GPG as of 2002-11-03 14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4

Re: Folder-based searching

[Carl Worth]

[-- Attachment #1: Type: text/plain, Size: 364 bytes --]

On Tue, 18 Jan 2011 13:54:29 -0600, Rob Browning <rlb@defaultvalue.org> wrote:
> I'd be tempted to consider making folder: searches rooted by default.  I
> wonder how often people really want "all folders named misc"?

That's a possibility---*after* someone does the work to enable rooted
searches in the first place.

-Carl

-- 
carl.d.worth@intel.com

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: Folder-based searching

[Austin Clements]

[-- Attachment #1: Type: text/plain, Size: 1006 bytes --]

The folder search implementation in the custom query parser is always rooted
(both because that happened to be much easier to do in that design, and
because I agree with you that rooted searches seem preferable most of the
time).  What arguments do people have for or against rooted folder searches?

On a similar note, another difference is that cworth's folder search is
recursive (which, I suppose, follows from being non-rooted).  That is, a
search for "folder:foo" will find any message in any folder under foo.  The
folder search I posted is non-recursive.  Thus, "folder:foo" specifically
searches the foo maildir (that is, any mail under foo, foo/new, or foo/cur).
 I consider different folders to be, well, different, regardless of nesting,
but what are other people's thoughts?

On Tue, Jan 18, 2011 at 2:54 PM, Rob Browning <rlb@defaultvalue.org> wrote:
>
> I'd be tempted to consider making folder: searches rooted by default.  I
> wonder how often people really want "all folders named misc"?

[-- Attachment #2: Type: text/html, Size: 1319 bytes --]

[PATCH] test: "search-by-folder" single-word search terms should be updated correctly when directories are renamed.

[Pieter Praet]

On Mon, 17 Jan 2011 14:00:41 +0100, Sebastian Spaeth <Sebastian@SSpaeth.de> wrote:
> However, when moving files into another folder and rerunning notmuch
> new, the folder value is never updated, it seems.

This appears to occur only when searching with a single-word folder
path.

---
 test/search-by-folder |    6 +++++-
 1 files changed, 5 insertions(+), 1 deletions(-)

diff --git a/test/search-by-folder b/test/search-by-folder
index 162158c..63ed57b 100755
--- a/test/search-by-folder
+++ b/test/search-by-folder
@@ -28,13 +28,17 @@ notmuch new
 output=$(notmuch search folder:bad/news | notmuch_search_sanitize)
 test_expect_equal "$output" "thread:XXX   2001-01-05 [1/1] Notmuch Test Suite; Bears (inbox unread)"

-test_begin_subtest "After rename, old path returns nothing"
+test_begin_subtest "After rename, old multi-word path returns nothing"
 mv "${MAIL_DIR}/duplicate/bad/news" "${MAIL_DIR}/duplicate/bad/olds"
 increment_mtime "${MAIL_DIR}/duplicate/bad"
 notmuch new
 output=$(notmuch search folder:bad/news | notmuch_search_sanitize)
 test_expect_equal "$output" ""

+test_begin_subtest "After rename, old single-word path returns nothing"
+output=$(notmuch search folder:news | notmuch_search_sanitize)
+test_expect_equal "$output" ""
+
 test_begin_subtest "After rename, new path returns result"
 output=$(notmuch search folder:bad/olds | notmuch_search_sanitize)
 test_expect_equal "$output" "thread:XXX   2001-01-05 [1/1] Notmuch Test Suite; Bears (inbox unread)"
---

Re: [PATCH] test: "search-by-folder" single-word search terms should be updated correctly when directories are renamed.

[Pieter Praet]

On Sun, 13 Mar 2011 00:44:14 +0100, Pieter Praet <pieter@praet.org> wrote:
> On Mon, 17 Jan 2011 14:00:41 +0100, Sebastian Spaeth <Sebastian@SSpaeth.de> wrote:
> > However, when moving files into another folder and rerunning notmuch
> > new, the folder value is never updated, it seems.
> 
> This appears to occur only when searching with a single-word folder
> path.
> 
> ---
>  test/search-by-folder |    6 +++++-
>  1 files changed, 5 insertions(+), 1 deletions(-)
> 
> diff --git a/test/search-by-folder b/test/search-by-folder
> index 162158c..63ed57b 100755
> --- a/test/search-by-folder
> +++ b/test/search-by-folder
> @@ -28,13 +28,17 @@ notmuch new
>  output=$(notmuch search folder:bad/news | notmuch_search_sanitize)
>  test_expect_equal "$output" "thread:XXX   2001-01-05 [1/1] Notmuch Test Suite; Bears (inbox unread)"
> 
> -test_begin_subtest "After rename, old path returns nothing"
> +test_begin_subtest "After rename, old multi-word path returns nothing"
>  mv "${MAIL_DIR}/duplicate/bad/news" "${MAIL_DIR}/duplicate/bad/olds"
>  increment_mtime "${MAIL_DIR}/duplicate/bad"
>  notmuch new
>  output=$(notmuch search folder:bad/news | notmuch_search_sanitize)
>  test_expect_equal "$output" ""
> 
> +test_begin_subtest "After rename, old single-word path returns nothing"
> +output=$(notmuch search folder:news | notmuch_search_sanitize)
> +test_expect_equal "$output" ""
> +
>  test_begin_subtest "After rename, new path returns result"
>  output=$(notmuch search folder:bad/olds | notmuch_search_sanitize)
>  test_expect_equal "$output" "thread:XXX   2001-01-05 [1/1] Notmuch Test Suite; Bears (inbox unread)"
> ---

I've pushed a cleaner version of this patch to my github repo, for
inclusion in release-candidate/0.6.


Peace


[0] git://github.com/praet/notmuch.git
    for-review/test-folder-search
    da50c58545c2fbb77fed


-- 
Pieter

Re: [PATCH] test: "search-by-folder" single-word search terms should be updated correctly when directories are renamed.

[Jameson Graef Rollins]

[-- Attachment #1: Type: text/plain, Size: 875 bytes --]

On Thu, 12 May 2011 15:22:25 +0200, Pieter Praet <pieter@praet.org> wrote:
> I've pushed a cleaner version of this patch to my github repo, for
> inclusion in release-candidate/0.6.

Hey, Pieter.  I realize we've been passing around git urls a lot
recently, particularly in regards to the release-candidate/0.6 branch,
but I think we should try to retain the habit of sending all patches to
the list if we can.

In general, I think we should try to only use git branches when:

- compiling existing patch series for review.
- new patch series that are so long that sending them to the list would
  be cumbersome.

I've been bad about this myself, since I was pushing patches to the
r-c/0.6 branch, but I think in general we should try to avoid that.
It's nice to have all patches on the list, so everyone can easily review
them and publicly comment on them.

Thanks!

jamie.

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

[PATCH] test: do folder-based search with a single-term path after moving message

[Pieter Praet]

When moving mail to a different folder, there appear to be residual
incorrect "folder:" terms.

This test exposes an existing bug, so is currently failing.

Signed-off-by: Pieter Praet <pieter@praet.org>
---
 test/search-by-folder |    4 ++++
 1 files changed, 4 insertions(+), 0 deletions(-)

diff --git a/test/search-by-folder b/test/search-by-folder
index 162158c..62767fb 100755
--- a/test/search-by-folder
+++ b/test/search-by-folder
@@ -35,6 +35,10 @@ notmuch new
 output=$(notmuch search folder:bad/news | notmuch_search_sanitize)
 test_expect_equal "$output" ""
 
+test_begin_subtest "After rename, old single-term path returns nothing"
+output=$(notmuch search folder:news | notmuch_search_sanitize)
+test_expect_equal_failure "$output" ""
+
 test_begin_subtest "After rename, new path returns result"
 output=$(notmuch search folder:bad/olds | notmuch_search_sanitize)
 test_expect_equal "$output" "thread:XXX   2001-01-05 [1/1] Notmuch Test Suite; Bears (inbox unread)"
-- 
1.7.4.1