Search using email headers does not work

Search using email headers does not work

[Michal Vyskocil]

[-- Attachment #1: Type: text/plain, Size: 1389 bytes --]

Hi,

I'd like to search for all emails with a defined email header. According
notmuch-search(1) [1] this should be a trivial. But I got a very limited
results

$ notmuch count --exclude=false "X-Mailinglist"
2

where simple
$ grep -R "^X-Mailinglist" .
./new/1360572283.M251897P7737Q200Ra1da4988633e0842.zelvantb:X-Mailinglist:
opensuse-factory
./new/1360315438.M35384P12994Q25807R1a5203f9da8b1b28.zelvantb:X-Mailinglist:
opensuse
./new/1360315441.M705450P13019Q25830R71a1deb928bb7cf3.zelvantb:X-Mailinglist:
opensuse
./new/1360315436.M459220P12984Q25797R13d6f3a1f39d2148.zelvantb:X-Mailinglist:
opensuse
./new/1360315381.M437845P12634Q25464R3bccd68f2b8fd54f.zelvantb:X-Mailinglist:
opensuse
./new/1360315556.M414918P13763Q26528R614e01d632d1c3d8.zelvantb:X-Mailinglist:
opensuse
^C

shows me, there are hundreds and hundreds of such emails in my local
maildir. I use very simple setup generated by notmuch-setup with a basic
set of inbox, undread signed and similar tags.

Can anyone point me on a correct query string for such search? I have
following versions of xapian and notmuch installed

$ rpm -q libxapian22 notmuch
libxapian22-1.2.8-2.1.2.x86_64
notmuch-0.14-20.3.x86_64

[1] http://notmuchmail.org/manpages/notmuch-search-1/

BTW: please CC me in a reply, I've just signed to the ML, but did not
get the you-were-subscribed email from mailman.

Thanks for a help
Michal Vyskocil

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Search using email headers does not work

[David Bremner]

Michal Vyskocil <mvyskocil@suse.cz> writes:

> Hi,
>
> I'd like to search for all emails with a defined email header. According
> notmuch-search(1) [1] this should be a trivial. But I got a very limited
> results
>
> $ notmuch count --exclude=false "X-Mailinglist"
> 2

Hi Michal;

Unfortunately currently headers other than those references in the
notmuch-search-terms man page are not indexed.  The only workaround I
can think of without patching notmuch is to use some tool to tag the
messages you care about.

d

Re: Search using email headers does not work

[Jameson Graef Rollins]

[-- Attachment #1: Type: text/plain, Size: 988 bytes --]

On Tue, Feb 12 2013, David Bremner <david@tethera.net> wrote:
> Unfortunately currently headers other than those references in the
> notmuch-search-terms man page are not indexed.  The only workaround I
> can think of without patching notmuch is to use some tool to tag the
> messages you care about.

I think even more importantly headers other than those that are parsed
are actually thrown to /dev/null.  They are not even indexed as plain
text, so you can search for terms in headers as you can for words in the
body.

I asked this on irc but it got lost in the shuffle: would it make sense
to index the entire header block as plain text?  If we did that then at
least the search that Michal was trying would return the expected
results.  It would also be a fairly simple change to the indexer.  I
don't have a good sense of how much it would pollute the index, though.

Given how frequently people are asking for alternate header indexing, it
might be something to consider.

jamie.

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

Re: Search using email headers does not work

[Michal Vyskocil]

[-- Attachment #1: Type: text/plain, Size: 1441 bytes --]

On Tue, Feb 12, 2013 at 09:03:01AM -0800, Jameson Graef Rollins wrote:
> On Tue, Feb 12 2013, David Bremner <david@tethera.net> wrote:
> > Unfortunately currently headers other than those references in the
> > notmuch-search-terms man page are not indexed.  The only workaround I
> > can think of without patching notmuch is to use some tool to tag the
> > messages you care about.
> 
> I think even more importantly headers other than those that are parsed
> are actually thrown to /dev/null.  They are not even indexed as plain
> text, so you can search for terms in headers as you can for words in the
> body.
> 
> I asked this on irc but it got lost in the shuffle: would it make sense
> to index the entire header block as plain text?  If we did that then at
> least the search that Michal was trying would return the expected
> results.  It would also be a fairly simple change to the indexer.  I
> don't have a good sense of how much it would pollute the index, though.

[skipping CC line and sending to list]

Hi,

I like such idea - but how hard would be to add a config options to
extend a list of default headers used for indexing? I am not sure there
is a strong need to index every bit from headers, just add few tips to
the default list.

Regards
Michal Vyskocil

> 
> Given how frequently people are asking for alternate header indexing, it
> might be something to consider.
> 
> jamie.



[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Search using email headers does not work

[Carl Worth]

[-- Attachment #1: Type: text/plain, Size: 1109 bytes --]

Michal Vyskocil <mvyskocil@suse.cz> writes:
> I like such idea - but how hard would be to add a config options to
> extend a list of default headers used for indexing?

Not hard.

Obviously, notmuch is already indexing some headers, (From, Date, To,
Subject), and it wouldn't be hard to index more. [*]

List-Id has been proposed often enough that it should probably be on by
default.

Then, there are other, slightly more special-case headers, (like headers
added by spam filtering software), that probably would be best added on
an opt-in basis based on a configuration option.

-Carl

[*] The existing set of indexed headers is really just historical
accident based on two things:

1. I was originally writing index code to match sup's index building, so
   I focused on the headers sup indexed.

2. I wrote code to index enough headers to satisfy my use case.

It's embarrassing to me that I added this note to the notmuch TODO file
in October 2009 (and it's still there):

    Add support for the user to specify custom headers to be indexed.

-- 
carl.d.worth@intel.com

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: Search using email headers does not work

[David Bremner]

Carl Worth <cworth@cworth.org> writes:
>
> It's embarrassing to me that I added this note to the notmuch TODO file
> in October 2009 (and it's still there):
>
>     Add support for the user to specify custom headers to be indexed.

One techicality is that we would presumeably want to support this both
for access via the CLI and via libnotmuch (e.g. for the python
bindings).  So there is the question of where to store the list of
indexed headers; perhaps in Xapian metadata. Or is there some simpler
solution?

d

Re: Search using email headers does not work

[Mark Walters]

I think I mentioned this on irc sometime ago: would indexing all the
headers as a separate free text entry (under headers: for example)
satisfy most of this. So then you could search for things like
headers:"List-Id: blah" or similar.

It is not as nice as individually indexed headers, but it does mean that
all notmuch instances would be the same and the same commands would
work, the library interface would be consistent etc.

Best wishes

Mark


 
On Tue, 12 Feb 2013, David Bremner <david@tethera.net> wrote:
> Carl Worth <cworth@cworth.org> writes:
>>
>> It's embarrassing to me that I added this note to the notmuch TODO file
>> in October 2009 (and it's still there):
>>
>>     Add support for the user to specify custom headers to be indexed.
>
> One techicality is that we would presumeably want to support this both
> for access via the CLI and via libnotmuch (e.g. for the python
> bindings).  So there is the question of where to store the list of
> indexed headers; perhaps in Xapian metadata. Or is there some simpler
> solution?
>
> d
> _______________________________________________
> notmuch mailing list
> notmuch@notmuchmail.org
> http://notmuchmail.org/mailman/listinfo/notmuch

Re: Search using email headers does not work

[Jameson Graef Rollins]

[-- Attachment #1: Type: text/plain, Size: 599 bytes --]

On Tue, Feb 12 2013, Mark Walters <markwalters1009@gmail.com> wrote:
> I think I mentioned this on irc sometime ago: would indexing all the
> headers as a separate free text entry (under headers: for example)
> satisfy most of this. So then you could search for things like
> headers:"List-Id: blah" or similar.
>
> It is not as nice as individually indexed headers, but it does mean that
> all notmuch instances would be the same and the same commands would
> work, the library interface would be consistent etc.

I like this idea.  Simple to implement, and covers all needs
automatically.

jamie.

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

Re: Search using email headers does not work

[Alexey Feldgendler]

On Wed, 13 Feb 2013 22:09:22 +0100, Jameson Graef Rollins  
<jrollins@finestructure.net> wrote:

>> I think I mentioned this on irc sometime ago: would indexing all the
>> headers as a separate free text entry (under headers: for example)
>> satisfy most of this. So then you could search for things like
>> headers:"List-Id: blah" or similar.
>>
>> It is not as nice as individually indexed headers, but it does mean that
>> all notmuch instances would be the same and the same commands would
>> work, the library interface would be consistent etc.

> I like this idea.  Simple to implement, and covers all needs
> automatically.

It will work for some headers, but not for others. Where a header's value  
is a list, there will be no precise way of specifying “the list contains  
X”. NEAR is only an approximation.


-- 
Alexey Feldgendler <alexey@feldgendler.ru>
[ICQ: 115226275] http://feldgendler.livejournal.com

Re: Search using email headers does not work

[Tomi Ollila]

On Tue, Feb 12 2013, David Bremner <david@tethera.net> wrote:

> Carl Worth <cworth@cworth.org> writes:
>>
>> It's embarrassing to me that I added this note to the notmuch TODO file
>> in October 2009 (and it's still there):
>>
>>     Add support for the user to specify custom headers to be indexed.
>
> One techicality is that we would presumeably want to support this both
> for access via the CLI and via libnotmuch (e.g. for the python
> bindings).  So there is the question of where to store the list of
> indexed headers; perhaps in Xapian metadata. Or is there some simpler
> solution?

BTW: how does clients using libnotmuch instead of CLI find the location
of mail store and Xapian database if those don't read the configuration
file used by CLI (and reference database.path there) ? :D

As the location of xapian database is known the location of mail store
and '.notmuch' directory is also known -- the '.notmuch' directory could
also contain some other dynamic information having accessor functions
in libnotmuch... (?)

Tomi