Inspired but need help searching folder names containing spaces

Inspired but need help searching folder names containing spaces

[Matthew Lear]

Hello all,

As part of trying to cope with an extremely high amount of email at
work, after a couple years of using X1 Search on Windows in conjunction
with MS Outlook and many years getting incredibly frustrated at the lack
of, and then failures of, email search features built into linux desktop
environments such as KDE and Gnome, I craved a better tool for linux.
Recently, I stumbled on the combination of offlineimap and notmuch and
I'm convinced that this will make my world a better place. I'm still
finding my way in terms of optimum ways to tag and filter my email, but
this is a fun ongoing task that will continue over the coming weeks and
months and I'm seeing a light at the end of the tunnel. So firstly,
thank you.

So...One thing I'm struggling with is searching for emails in folders
which contain spaces in their name. The IMAP server at work is Exchange
based and I have Inbox, Deleted Items and Sent Items folders (among many
others) all at the same top level. I'm using the default offlineimap
configuration which uses '.' as the separator. After a sync I have...

mail/maildir/
├── INBOX
├── Deleted Items
...
└── Sent Items

....on the file system. These directories have the cur, new and tmp sub
directories associated with maildir. All ok so far.

I have several emails in Deleted Items and Sent Items. I can ls and cat
them just fine. What I cannot do, however, is search them with notmuch.
Reading the very helpful replies in this thread
(http://notmuchmail.org/pipermail/notmuch/2014/019053.html), I thought
I'd stumbled on a solution. However, searching with..

notmuch search 'folder:"Deleted Items"'

.. throws up zero results.

Confusingly, if I sync a different IMAP account with offlineimap and
index it with notmuch, I can successfully search folders which contain
spaces in their name, but in that account, these folders exist *under*
INBOX, eg

mail/maildir2/
├── INBOX
├── INBOX.Deleted Items
...
└── INBOX.Sent Items

So searching with..

notmuch search 'folder:"INBOX.Deleted Items"'

..works.

So why the difference? AFAICT the syntax I'm using is correct and should
allow the string to be searched in the Xapian database. I just don't
understand why one directory level seems to make a difference between a
successful and an unsuccessful search.

Any thoughts?

If it helps I'm running notmuch 0.19 and xapian 1.2.18 on Gentoo linux.

I suppose I don't need to sync Deleted Items, and I can tag -inbox
emails sent by me, but I'd still like to be able to search both these
folders if I need to.

Can anybody help suggest what could be going wrong?

Many thanks from a very happy new notmuch user.
--  Matt

Re: Inspired but need help searching folder names containing spaces

[David Bremner]

Matthew Lear <matt@bubblegen.co.uk> writes:

> mail/maildir/
> ├── INBOX
> ├── Deleted Items
> ...
> └── Sent Items
>
>
> notmuch search 'folder:"Deleted Items"'
>
> .. throws up zero results.
>

Hmm. I tried to duplicate your situation here, and it works for me. Two
things to try:

1) notmuch search --output=files $QUERY

where $QUERY is some query (using e.g. id:) that matches one of those
messages. Check that you have have the whole anchored path from the top
level.

2) NOTMUCH_DEBUG_QUERY=1 notmuch count -'folder:"Deleted Items"'

have a look for XFOLDER:Deleted Items

3) try both of the above with "--exclude=false" in case you have tagged
them with an excluded tag.  Note that by default "deleted" is such a tag.
You can change the excluded tags with "notmuch config"

d

Re: Inspired but need help searching folder names containing spaces

[Matthew Lear]

Hi David,

>> notmuch search 'folder:"Deleted Items"'
>>
>> .. throws up zero results.
>>
>
> Hmm. I tried to duplicate your situation here, and it works for me. Two
> things to try:
>
> 1) notmuch search --output=files $QUERY
>
> where $QUERY is some query (using e.g. id:) that matches one of those
> messages. Check that you have have the whole anchored path from the top
> level.
>
> 2) NOTMUCH_DEBUG_QUERY=1 notmuch count -'folder:"Deleted Items"'
>
> have a look for XFOLDER:Deleted Items
>
> 3) try both of the above with "--exclude=false" in case you have tagged
> them with an excluded tag.  Note that by default "deleted" is such a tag.
> You can change the excluded tags with "notmuch config"

As part of still playing around with notmuch and strategies for tagging
etc, I decided to start afresh with the database so I deleted it. After
running notmuch new, I could quite happily search with notmuch search
'folder:"Deleted Items"'. This surprised me greatly because I know that I
didn't apply any tags to mails in Deleted Items (couldn't find any in
there to tag) and I also already tried searching with --exclude=false. In
any case, it seems that searching in folders with spaces now works. Hurrah
:-)

A couple of others things have surprised me, though. Using the date search
option, such as date:"today" or date:"yesterday", hasn't quite thrown up
what I expected. Searching with date:"today" shows up no results (which I
know is wrong) and searching with date:"yesterday" shows up a handful of
emails all from the same person, in different threads which all arrived on
a day in May 2014... Got to be a bug there somewhere :-D

One last thing which I'd like to explore if I may is searching for emails
sent to me. By default, I think notmuch classes 'to me' as 'to or cc or
bcc me'. Is there a way to search/filter on emails sent only to me (ie
where my name / address is only in the to: field in the message header)?
This seems a reasonable requirement / use case so I'm assuming that
there's a good reason that notmuch views it differently.

An example is an automated bug tracking system which sends emails to the
current assignee of the bug (to) and also watchers (cc). I'd like to be
able to filter these emails differently (ie bugs I'm watching versus bugs
I'm responsible for fixing). Apart from using other fields in the message
body (which may be subject to change), there doesn't seem an easy way to
do this which is surprising given notmuch's flexibility and power.

Thoughts?

I'm still exploring strategies for tagging and searching etc. The links on
the wiki are helpful. The possibilities are almost infinite and while
everybody's mail is different, having a great deal of flexibility can
sometimes makes it tricky to decide on exactly what's best. Still, it's an
enjoyable task, despite being time consuming, and is well worth the
effort.

  Matt

Re: Inspired but need help searching folder names containing spaces

[Jani Nikula]

On Fri, 20 Feb 2015, Matthew Lear <matt@bubblegen.co.uk> wrote:
> A couple of others things have surprised me, though. Using the date search
> option, such as date:"today" or date:"yesterday", hasn't quite thrown up
> what I expected. Searching with date:"today" shows up no results (which I
> know is wrong) and searching with date:"yesterday" shows up a handful of
> emails all from the same person, in different threads which all arrived on
> a day in May 2014... Got to be a bug there somewhere :-D

This is a known limitation in the date: prefix queries. You have to
specify it as a range, for example date:today..today (from the beginning
of today until the end of today). If the underlying Xapian search engine
is not given a range expression (with the "..") it searches for the
words "date today" without punctuation, matching whatever it matches,
and notmuch does not even get an error for it.

HTH,
Jani.

Re: Inspired but need help searching folder names containing spaces

[Matthew Lear]

> On 20 Feb 2015, at 18:01, Jani Nikula <jani@nikula.org> wrote:
> 
>> On Fri, 20 Feb 2015, Matthew Lear <matt@bubblegen.co.uk> wrote:
>> A couple of others things have surprised me, though. Using the date search
>> option, such as date:"today" or date:"yesterday", hasn't quite thrown up
>> what I expected. Searching with date:"today" shows up no results (which I
>> know is wrong) and searching with date:"yesterday" shows up a handful of
>> emails all from the same person, in different threads which all arrived on
>> a day in May 2014... Got to be a bug there somewhere :-D
> 
> This is a known limitation in the date: prefix queries. You have to
> specify it as a range, for example date:today..today (from the beginning
> of today until the end of today). If the underlying Xapian search engine
> is not given a range expression (with the "..") it searches for the
> words "date today" without punctuation, matching whatever it matches,
> and notmuch does not even get an error for it.

Thanks. Explains everything. Any thoughts on searching on 'to' only? I'm not sure if this is also related to what Xapian needs to be provided with. If not, and if only searching in the to, cc and bcc fields of the header is possible, maybe we could consider adopting and integrating case sensitive syntax in notmuch such as TO:me rather than to:me..?
--  Matt

Re: Inspired but need help searching folder names containing spaces

[David Bremner]

Matthew Lear <matt@bubblegen.co.uk> writes:

>
> Thanks. Explains everything. Any thoughts on searching on 'to' only?
> I'm not sure if this is also related to what Xapian needs to be
> provided with. If not, and if only searching in the to, cc and bcc
> fields of the header is possible, maybe we could consider adopting and
> integrating case sensitive syntax in notmuch such as TO:me rather than
> to:me..?

I'm not sure about the exact syntax, but some separate prefix to mean
only in the To header would be possible. As far as I know we'd need to
add more terms to the database [1], so you'd need to convince people
that the increase in the size of everyone's database was worthwhile. And
of course someone needs to write the code, but it would not be that
hard, I think. 

d

[1]: this could also be solved with the semi-mythical custom query parser.

Re: Inspired but need help searching folder names containing spaces

[David Bremner]

David Bremner <david@tethera.net> writes:

>
> I'm not sure about the exact syntax, but some separate prefix to mean
> only in the To header would be possible. As far as I know we'd need to
> add more terms to the database [1], so you'd need to convince people
> that the increase in the size of everyone's database was worthwhile. And
> of course someone needs to write the code, but it would not be that
> hard, I think. 

Austin pointed out on IRC that if we had fields indexed seperately then
the current query parser then we could actually use the current Xapian
query parser to generate an OR of several database fields from a single
query prefix. So if someone wanted to take this on, that would probably
be the way to go.

d