Folder-based searching

Folder-based searching

[Carl Worth]

I'm as sad as anyone to see that I let more than a month go by without
committing to the notmuch repository.

But I've just now committed the much-desired folder-based search
feature.

This should be a handy feature for people bringing mail in from a system
that stores information in the directories in which mail is stored, (one
common case is users fetching mail from gmail via imap---in this case
all gmail tags will be represented as directories).

Someone bringing in mail from such a system might want to migrate from
information in directory names to instead be information in notmuch
tags. That might look something like this:

	notmuch tag +important folder:important

Or it might be useful to run on several directories:

	for dir in *; do notmuch tag +"$dir"; folder:"$dir"; done

etc.

Please note that with the code in git currently, only newly incorporated
mail will be searchable via the "folder:" specification. So if you're
wanting to search your entire email collection, you may want to rebuild
your entire notmuch database. As a reminder, that process is something
like the following:

    notmuch dump > notmuch.dump
    rm -rf ~/mail/.notmuch	# Careful with this! Have backups in place!
    notmuch new			# Of course, this can take a long time
    notmuch restore notmuch.dump

Before the next release, I plan to increment the database version so
that users will be notified of the need to do an upgrade like this. (But
before I do that, I plan to implement a few more database changes---and
some of them are likely to be much more invasive than this one).

Have fun out there!

-Carl

Re: Folder-based searching

[Xavier Maillard]

Hi,

On Sat, 15 Jan 2011 14:46:53 -0800, Carl Worth <cworth@cworth.org> wrote:
> I'm as sad as anyone to see that I let more than a month go by without
> committing to the notmuch repository.

Life goes like this...
 
> But I've just now committed the much-desired folder-based search
> feature.

Awesome ! 

> This should be a handy feature for people bringing mail in from a system
> that stores information in the directories in which mail is stored, (one
> common case is users fetching mail from gmail via imap---in this case
> all gmail tags will be represented as directories).

Is Imap with SIEVE scripts such a common case too ?

/Xavier

Re: Folder-based searching

[Sebastian Spaeth]

> Before the next release, I plan to increment the database version so
> that users will be notified of the need to do an upgrade like this. (But
> before I do that, I plan to implement a few more database changes---and
> some of them are likely to be much more invasive than this one).

Hurray, just tried it out and it works like a charm. Finally I can
automatically assign the "Junk" tag again. :-)

Sebastian

Re: Folder-based searching

[Sebastian Spaeth]

[-- Attachment #1: Type: text/plain, Size: 1178 bytes --]

On Mon, 17 Jan 2011 13:43:44 +0100, Sebastian Spaeth <Sebastian@SSpaeth.de> wrote:
> > Before the next release, I plan to increment the database version so
> > that users will be notified of the need to do an upgrade like this. (But
> > before I do that, I plan to implement a few more database changes---and
> > some of them are likely to be much more invasive than this one).
> 
> Hurray, just tried it out and it works like a charm. Finally I can
> automatically assign the "Junk" tag again. :-)

Well, it's never as easy as it looks at first glance. Initial run picks
up folder:junk and reports correctly (is it supposed to be
case-insensitive?).

However, when moving files into another folder and rerunning notmuch
new, the folder value is never updated, it seems. This is what I run to
move mails that had been wrongly classified as Junk and that I manually
untagged into my INBOX:

notmuch search --output=files not tag:junk and folder:junk | xargs mv -t
/home/spaetz/mail/INBOX/cur/

However, each subsequent run of:
notmuch search --output=files not tag:junk and folder:junk

will show the same files, even after a notmuch new...

Sebastian

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: Folder-based searching

[Rob Browning]

Carl Worth <cworth@cworth.org> writes:

> Someone bringing in mail from such a system might want to migrate from
> information in directory names to instead be information in notmuch
> tags. That might look something like this:
>
> 	notmuch tag +important folder:important

Nice.  So what are the path semantics?  For example, is the path case
sensitive or insensitive?  Can you say folder:foo/bar, and if so, is the
path rooted, or can it match path sub-segments?

i.e. which of these, if any, is "folder:foo/bar" like?

  (.*/)?foo/bar(/.*)?
  ^foo/bar$
  ^foo/bar(/.*)?

Thanks
-- 
Rob Browning
rlb @defaultvalue.org and @debian.org
GPG as of 2002-11-03 14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4

Re: Folder-based searching

[Carl Worth]

[-- Attachment #1: Type: text/plain, Size: 1099 bytes --]

On Mon, 17 Jan 2011 12:48:11 -0600, Rob Browning <rlb@defaultvalue.org> wrote:
> Nice.

Thanks.

> So what are the path semantics?  For example, is the path case
> sensitive or insensitive?  Can you say folder:foo/bar, and if so, is the
> path rooted, or can it match path sub-segments?
> 
> i.e. which of these, if any, is "folder:foo/bar" like?
> 
>   (.*/)?foo/bar(/.*)?

That's the right one.

It's a case-insensitive, non-rooted search specification.

This works in a similar way to 'subject:"some phrase"', (though more
people seem to be asking about these details for folder: than have ever
asked for subject:).

This is the easiest kind of searching to get out of Xapian. However, in
the future, I would like to enable users to specify rooted search
specifications, (whether with folder:, subject:, or any other prefix).

That will require a little care to get some additional terms indexed to
support the rooting, then the in-development custom query parser to
allow mapping symbols like '^' and '$' to these new symbols.

-Carl

-- 
carl.d.worth@intel.com

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: Folder-based searching

[Rob Browning]

Carl Worth <cworth@cworth.org> writes:

> This works in a similar way to 'subject:"some phrase"', (though more
> people seem to be asking about these details for folder: than have ever
> asked for subject:).

I'm not surprised.  Imagine:

  /misc
  /debian/misc

If I understand correctly, right now you'd have to say

  folder:misc and not folder:debian/misc

which is not nearly as handy as

  folder:^misc 

and at first, a lot of people may not even realize that they need the
"and not" clauses.

I'd be tempted to consider making folder: searches rooted by default.  I
wonder how often people really want "all folders named misc"?

> That will require a little care to get some additional terms indexed to
> support the rooting, then the in-development custom query parser to
> allow mapping symbols like '^' and '$' to these new symbols.

Sounds good.

Thanks
-- 
Rob Browning
rlb @defaultvalue.org and @debian.org
GPG as of 2002-11-03 14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4

Re: Folder-based searching

[Carl Worth]

[-- Attachment #1: Type: text/plain, Size: 364 bytes --]

On Tue, 18 Jan 2011 13:54:29 -0600, Rob Browning <rlb@defaultvalue.org> wrote:
> I'd be tempted to consider making folder: searches rooted by default.  I
> wonder how often people really want "all folders named misc"?

That's a possibility---*after* someone does the work to enable rooted
searches in the first place.

-Carl

-- 
carl.d.worth@intel.com

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: Folder-based searching

[Austin Clements]

[-- Attachment #1: Type: text/plain, Size: 1006 bytes --]

The folder search implementation in the custom query parser is always rooted
(both because that happened to be much easier to do in that design, and
because I agree with you that rooted searches seem preferable most of the
time).  What arguments do people have for or against rooted folder searches?

On a similar note, another difference is that cworth's folder search is
recursive (which, I suppose, follows from being non-rooted).  That is, a
search for "folder:foo" will find any message in any folder under foo.  The
folder search I posted is non-recursive.  Thus, "folder:foo" specifically
searches the foo maildir (that is, any mail under foo, foo/new, or foo/cur).
 I consider different folders to be, well, different, regardless of nesting,
but what are other people's thoughts?

On Tue, Jan 18, 2011 at 2:54 PM, Rob Browning <rlb@defaultvalue.org> wrote:
>
> I'd be tempted to consider making folder: searches rooted by default.  I
> wonder how often people really want "all folders named misc"?

[-- Attachment #2: Type: text/html, Size: 1319 bytes --]

Re: Folder-based searching

[mazieres-6kd3dvy8f9kx6dc7vcsytfvnai]

This folder-based searching is very handy.  However, I have a scenario
in which I'd prefer to consolidate mail to multiple addresses into one
or a small number of physical mail folders.  Thus, what I really want
to search on is not the mail folder name, but the Delivered-To:
header.

Can someone tell me if there is any easy way to restrict a search to
the Delivered-To: header, or to add custom header support to notmuch?

Sorry if this is a very basic question.  I've just started checking
out notmuch and really like what I see so far.  My inability to figure
out how to search unambiguously for Delivered-To headers looks like it
could be the biggest obstacle to switching from Wanderlust (which lets
me customize search for arbitrary headers listed in
elmo-msgdb-extra-fields).

Thanks.

Re: Folder-based searching

[Carl Worth]

[-- Attachment #1: Type: text/plain, Size: 1171 bytes --]

On Fri, 21 Jan 2011 02:45:04 -0800, mazieres-6kd3dvy8f9kx6dc7vcsytfvnai@temporary-address.scs.stanford.edu wrote:
> This folder-based searching is very handy.

Hello, anonymous poster from Stanford. Welcome to notmuch!

I'm glad you're happy with the folder-based searching and I hope you
find notmuch useful.

> Can someone tell me if there is any easy way to restrict a search to
> the Delivered-To: header, or to add custom header support to notmuch?

Currently, this header is not being indexed by notmuch.

For sometime now I've had two features at the top of my personal todo
list for notmuch:

	1. Folder-based searching

	2. Indexing of arbitrary headers

The first is done now, and I'll be working on the second now.

These features are essential to so many people, (and have been obviously
missing for so long), that once these are done I might be inclined to
name the next release of notmuch 1.0.

I'm particularly inclined to increment the major version of notmuch
because these two features will also require users to rebuild their
entire mail database to enable the feature for all mail.

-Carl

-- 
carl.d.worth@intel.com

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]