query on a subset of messages ?

query on a subset of messages ?

[Sebastien Binet]

[-- Attachment #1: Type: text/plain, Size: 1023 bytes --]


hi there,

I was trying to reduce the I/O stress during my usual email
fetching+tagging by writing a little program using the go bindings to
notmuch.

ie:
db, status := notmuch.OpenDatabase(db_path,
    		notmuch.DATABASE_MODE_READ_WRITE)
query := db.CreateQuery("(tag:new AND tag:inbox)")
msgs := query.SearchMessages()
for _,msg := range msgs {
  tag_msg(msg, tagqueries)
}


where tagqueries is a subquery of the form:
[
    {
        "Cmd": "+to-me",
        "Query": "(to:sebastien.binet@cern.ch and not tag:to-me)"
    },
    {
        "Cmd": "+sci-notmuch",
        "Query": "from:notmuch@notmuchmail.org or to:notmuch@notmuchmail.org or subject:notmuch"
    }
]


the idea being that I only need to crawl through the db only once and
then iteratively apply tags on those messages (instead of repeatedly
running "notmuch tag ..." for each and every of those many
'tag-queries')

I couldn't find any C-API to do such a thing using the notmuch library.
did I overlook something ?

Is it something useful to add ?

-s

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: query on a subset of messages ?

[Jameson Graef Rollins]

[-- Attachment #1: Type: text/plain, Size: 1691 bytes --]

On Mon, Jul 09 2012, Sebastien Binet <binet@cern.ch> wrote:
> I was trying to reduce the I/O stress during my usual email
> fetching+tagging by writing a little program using the go bindings to
> notmuch.
>
> ie:
> db, status := notmuch.OpenDatabase(db_path,
>     		notmuch.DATABASE_MODE_READ_WRITE)
> query := db.CreateQuery("(tag:new AND tag:inbox)")
> msgs := query.SearchMessages()
> for _,msg := range msgs {
>   tag_msg(msg, tagqueries)
> }
>
>
> where tagqueries is a subquery of the form:
> [
>     {
>         "Cmd": "+to-me",
>         "Query": "(to:sebastien.binet@cern.ch and not tag:to-me)"
>     },
>     {
>         "Cmd": "+sci-notmuch",
>         "Query": "from:notmuch@notmuchmail.org or to:notmuch@notmuchmail.org or subject:notmuch"
>     }
> ]


Hi, Sebastian.  It's really hard for me to believe that this is much
faster than simply making the two tagging calls in full:

notmuch tag +to-me -- tag:new and tag:inbox and (to:sebastien.binet@cern.ch and not tag:to-me)
notmuch tag +sci-notmuch -- tag:new and tag:inbox and from:notmuch@notmuchmail.org or to:notmuch@notmuchmail.org or subject:notmuch"

After the first call the cache will be fresh, so the overhead should be
minimal.  It looks to me you're looking in to this as a post-new hook.
I do pretty much the same thing, and with the above properly constructed
searches the tagging is super fast.

Have you tried profiling the two options?  Is it really high I/O stress
on your system?  If so, maybe there's another issue that can be
addressed.

As an aside I should point out that a lot of people want to see the
"to:me" search term.  But I think the right place to achieve that is in
the query parser.

jamie.

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

Re: query on a subset of messages ?

[Austin Clements]

Quoth Sebastien Binet on Jul 09 at 10:25 am:
> 
> hi there,
> 
> I was trying to reduce the I/O stress during my usual email
> fetching+tagging by writing a little program using the go bindings to
> notmuch.
> 
> ie:
> db, status := notmuch.OpenDatabase(db_path,
>     		notmuch.DATABASE_MODE_READ_WRITE)
> query := db.CreateQuery("(tag:new AND tag:inbox)")
> msgs := query.SearchMessages()
> for _,msg := range msgs {
>   tag_msg(msg, tagqueries)
> }
> 
> 
> where tagqueries is a subquery of the form:
> [
>     {
>         "Cmd": "+to-me",
>         "Query": "(to:sebastien.binet@cern.ch and not tag:to-me)"
>     },
>     {
>         "Cmd": "+sci-notmuch",
>         "Query": "from:notmuch@notmuchmail.org or to:notmuch@notmuchmail.org or subject:notmuch"
>     }
> ]
> 
> 
> the idea being that I only need to crawl through the db only once and
> then iteratively apply tags on those messages (instead of repeatedly
> running "notmuch tag ..." for each and every of those many
> 'tag-queries')
> 
> I couldn't find any C-API to do such a thing using the notmuch library.
> did I overlook something ?
> 
> Is it something useful to add ?
> 
> -s

Have you tried a more direct translation of the multiple notmuch tag
commands into Go, where you don't worry about subsetting the queries?
Unless you're tagging a huge number of messages, the cost of notmuch
tag is almost certainly the fsync that it does when it closes the
database (which every call to notmuch tag must do).  However, in Go,
you can keep the database open across all of the tagging operations
and then close and fsync it just once.

Note that there is an important optimization in notmuch tag that you
might have to replicate.  It manipulates the original query to exclude
messages that already have the desired tags, so that they get skipped
very efficiently at the earliest stage possible.

Re: query on a subset of messages ?

[Sebastien Binet]

[-- Attachment #1: Type: text/plain, Size: 1981 bytes --]

Jamie,

Jameson Graef Rollins <jrollins@finestructure.net> writes:

> On Mon, Jul 09 2012, Sebastien Binet <binet@cern.ch> wrote:
>> I was trying to reduce the I/O stress during my usual email
>> fetching+tagging by writing a little program using the go bindings to
>> notmuch.
>>
>> ie:
>> db, status := notmuch.OpenDatabase(db_path,
>>     		notmuch.DATABASE_MODE_READ_WRITE)
>> query := db.CreateQuery("(tag:new AND tag:inbox)")
>> msgs := query.SearchMessages()
>> for _,msg := range msgs {
>>   tag_msg(msg, tagqueries)
>> }
>>
>>
>> where tagqueries is a subquery of the form:
>> [
>>     {
>>         "Cmd": "+to-me",
>>         "Query": "(to:sebastien.binet@cern.ch and not tag:to-me)"
>>     },
>>     {
>>         "Cmd": "+sci-notmuch",
>>         "Query": "from:notmuch@notmuchmail.org or to:notmuch@notmuchmail.org or subject:notmuch"
>>     }
>> ]
>
>
> Hi, Sebastian.  It's really hard for me to believe that this is much
> faster than simply making the two tagging calls in full:
>
> notmuch tag +to-me -- tag:new and tag:inbox and (to:sebastien.binet@cern.ch and not tag:to-me)
> notmuch tag +sci-notmuch -- tag:new and tag:inbox and from:notmuch@notmuchmail.org or to:notmuch@notmuchmail.org or subject:notmuch"
>
> After the first call the cache will be fresh, so the overhead should be
> minimal.  It looks to me you're looking in to this as a post-new hook.
> I do pretty much the same thing, and with the above properly constructed
> searches the tagging is super fast.

well, of course I don't have just those two:

$ grep -He "^tag_new " emacs/notmuch-lib/notmuch-tag.sh | wc -l
54

also:
$ du -hs mail-notmuch/.notmuch 
2.9G	mail-notmuch/.notmuch

that said, most of the I/O wait seem to come from 'notmuch new' (after
having dropped all fs caches.)

ok, post-new hook it is then :)

(but there is probably something to do to speed-up a 'notmuch new' after
having dropped all fs caches... ?)

-s

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

post-new [was: Re: query on a subset of messages ?]

[Sebastien Binet]

[-- Attachment #1: Type: text/plain, Size: 2054 bytes --]

Sebastien Binet <binet@cern.ch> writes:

> Jamie,
>
> Jameson Graef Rollins <jrollins@finestructure.net> writes:
>
>> On Mon, Jul 09 2012, Sebastien Binet <binet@cern.ch> wrote:
>>> I was trying to reduce the I/O stress during my usual email
>>> fetching+tagging by writing a little program using the go bindings to
>>> notmuch.
>>>
>>> ie:
>>> db, status := notmuch.OpenDatabase(db_path,
>>>     		notmuch.DATABASE_MODE_READ_WRITE)
>>> query := db.CreateQuery("(tag:new AND tag:inbox)")
>>> msgs := query.SearchMessages()
>>> for _,msg := range msgs {
>>>   tag_msg(msg, tagqueries)
>>> }
>>>
>>>
>>> where tagqueries is a subquery of the form:
>>> [
>>>     {
>>>         "Cmd": "+to-me",
>>>         "Query": "(to:sebastien.binet@cern.ch and not tag:to-me)"
>>>     },
>>>     {
>>>         "Cmd": "+sci-notmuch",
>>>         "Query": "from:notmuch@notmuchmail.org or to:notmuch@notmuchmail.org or subject:notmuch"
>>>     }
>>> ]
>>
>>
>> Hi, Sebastian.  It's really hard for me to believe that this is much
>> faster than simply making the two tagging calls in full:
>>
>> notmuch tag +to-me -- tag:new and tag:inbox and (to:sebastien.binet@cern.ch and not tag:to-me)
>> notmuch tag +sci-notmuch -- tag:new and tag:inbox and from:notmuch@notmuchmail.org or to:notmuch@notmuchmail.org or subject:notmuch"
>>
>> After the first call the cache will be fresh, so the overhead should be
>> minimal.  It looks to me you're looking in to this as a post-new hook.
>> I do pretty much the same thing, and with the above properly constructed
>> searches the tagging is super fast.
>
> well, of course I don't have just those two:
>
> $ grep -He "^tag_new " emacs/notmuch-lib/notmuch-tag.sh | wc -l
> 54
>
> also:
> $ du -hs mail-notmuch/.notmuch 
> 2.9G	mail-notmuch/.notmuch
>
> that said, most of the I/O wait seem to come from 'notmuch new' (after
> having dropped all fs caches.)
>
> ok, post-new hook it is then :)

hum... is post-new supposed to be run even if there is no new message ?

-s

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: query on a subset of messages ?

[Sebastien Binet]

[-- Attachment #1: Type: text/plain, Size: 2184 bytes --]

Austin Clements <amdragon@MIT.EDU> writes:

> Quoth Sebastien Binet on Jul 09 at 10:25 am:
>> 
>> hi there,
>> 
>> I was trying to reduce the I/O stress during my usual email
>> fetching+tagging by writing a little program using the go bindings to
>> notmuch.
>> 
>> ie:
>> db, status := notmuch.OpenDatabase(db_path,
>>     		notmuch.DATABASE_MODE_READ_WRITE)
>> query := db.CreateQuery("(tag:new AND tag:inbox)")
>> msgs := query.SearchMessages()
>> for _,msg := range msgs {
>>   tag_msg(msg, tagqueries)
>> }
>> 
>> 
>> where tagqueries is a subquery of the form:
>> [
>>     {
>>         "Cmd": "+to-me",
>>         "Query": "(to:sebastien.binet@cern.ch and not tag:to-me)"
>>     },
>>     {
>>         "Cmd": "+sci-notmuch",
>>         "Query": "from:notmuch@notmuchmail.org or to:notmuch@notmuchmail.org or subject:notmuch"
>>     }
>> ]
>> 
>> 
>> the idea being that I only need to crawl through the db only once and
>> then iteratively apply tags on those messages (instead of repeatedly
>> running "notmuch tag ..." for each and every of those many
>> 'tag-queries')
>> 
>> I couldn't find any C-API to do such a thing using the notmuch library.
>> did I overlook something ?
>> 
>> Is it something useful to add ?
>> 
>> -s
>
> Have you tried a more direct translation of the multiple notmuch tag
> commands into Go, where you don't worry about subsetting the queries?
> Unless you're tagging a huge number of messages, the cost of notmuch
> tag is almost certainly the fsync that it does when it closes the
> database (which every call to notmuch tag must do).  However, in Go,
> you can keep the database open across all of the tagging operations
> and then close and fsync it just once.

nope, I haven't tried that, but will do.

>
> Note that there is an important optimization in notmuch tag that you
> might have to replicate.  It manipulates the original query to exclude
> messages that already have the desired tags, so that they get skipped
> very efficiently at the earliest stage possible.
I already have this in my original shell script.
(wouldn't be too hard to automatically do, though.)

-s

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: post-new [was: Re: query on a subset of messages ?]

[Jameson Graef Rollins]

[-- Attachment #1: Type: text/plain, Size: 229 bytes --]

On Mon, Jul 09 2012, Sebastien Binet <binet@cern.ch> wrote:
> hum... is post-new supposed to be run even if there is no new message ?

Hi, Sebastian.  Yes, I think it runs regardless if there are any new
messages or not.

jamie.

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

Re: post-new [was: Re: query on a subset of messages ?]

[Jani Nikula]

[-- Attachment #1: Type: text/plain, Size: 560 bytes --]

On Jul 9, 2012 8:12 PM, "Jameson Graef Rollins" <jrollins@finestructure.net>
wrote:
>
> On Mon, Jul 09 2012, Sebastien Binet <binet@cern.ch> wrote:
> > hum... is post-new supposed to be run even if there is no new message ?
>
> Hi, Sebastian.  Yes, I think it runs regardless if there are any new
> messages or not.

That's correct; only errors in notmuch new cause post-new hook to be
skipped.

>
> jamie.
>
> _______________________________________________
> notmuch mailing list
> notmuch@notmuchmail.org
> http://notmuchmail.org/mailman/listinfo/notmuch
>

[-- Attachment #2: Type: text/html, Size: 925 bytes --]

Re: post-new [was: Re: query on a subset of messages ?]

[Sebastien Binet]

[-- Attachment #1: Type: text/plain, Size: 805 bytes --]

Jani Nikula <jani@nikula.org> writes:

> On Jul 9, 2012 8:12 PM, "Jameson Graef Rollins" <jrollins@finestructure.net>
> wrote:
>>
>> On Mon, Jul 09 2012, Sebastien Binet <binet@cern.ch> wrote:
>> > hum... is post-new supposed to be run even if there is no new message ?
>>
>> Hi, Sebastian.  Yes, I think it runs regardless if there are any new
>> messages or not.
>
> That's correct; only errors in notmuch new cause post-new hook to be
> skipped.

ok. I thought using the post-new hook would have saved some i/o
resources over my current setup:
 offlineimap.postsynchook = ~/emacs/notmuch-lib/notmuch-tag.sh

where notmuch-tag.sh is (in pseudo-code):
##
/usr/bin/notmuch new

for tag,query in tag-queries:
  tag_new $tag $query

##

is there any advantage of using post-new compared to this setup ?

-s

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: post-new [was: Re: query on a subset of messages ?]

[Jani Nikula]

[-- Attachment #1: Type: text/plain, Size: 1344 bytes --]

On Jul 10, 2012 12:59 PM, "Sebastien Binet" <binet@cern.ch> wrote:
>
> Jani Nikula <jani@nikula.org> writes:
>
> > On Jul 9, 2012 8:12 PM, "Jameson Graef Rollins" <
jrollins@finestructure.net>
> > wrote:
> >>
> >> On Mon, Jul 09 2012, Sebastien Binet <binet@cern.ch> wrote:
> >> > hum... is post-new supposed to be run even if there is no new
message ?
> >>
> >> Hi, Sebastian.  Yes, I think it runs regardless if there are any new
> >> messages or not.
> >
> > That's correct; only errors in notmuch new cause post-new hook to be
> > skipped.
>
> ok. I thought using the post-new hook would have saved some i/o
> resources over my current setup:
>  offlineimap.postsynchook = ~/emacs/notmuch-lib/notmuch-tag.sh
>
> where notmuch-tag.sh is (in pseudo-code):
> ##
> /usr/bin/notmuch new
>
> for tag,query in tag-queries:
>   tag_new $tag $query
>
> ##
>
> is there any advantage of using post-new compared to this setup ?

There's no functional advantage. It does keep your initial tagging script
connected with notmuch new rather than offlinemap, if you ever need to run
notmuch new on its own.

If your tagging setup is really complicated, you could have some-tag in
new.tags config, and bail out early if notmuch count tag:some-tag outputs 0
(and obviously notmuch tag -some-tag tag:some-tag later in the script).
Just a thought.

J.

>
> -s

[-- Attachment #2: Type: text/html, Size: 1904 bytes --]

Re: post-new [was: Re: query on a subset of messages ?]

[Jameson Graef Rollins]

[-- Attachment #1: Type: text/plain, Size: 813 bytes --]

On Tue, Jul 10 2012, Sebastien Binet <binet@cern.ch> wrote:
> ok. I thought using the post-new hook would have saved some i/o
> resources over my current setup:
>  offlineimap.postsynchook = ~/emacs/notmuch-lib/notmuch-tag.sh
>
> where notmuch-tag.sh is (in pseudo-code):
> ##
> /usr/bin/notmuch new
>
> for tag,query in tag-queries:
>   tag_new $tag $query
>
> ##
>
> is there any advantage of using post-new compared to this setup ?

The hooks are just there for convenience, and as Jani says, they're
executed any time notmuch new is run, in any context.  In your setup it
looks like it would also save you having to maintain another script
(notmuch-tag.sh) since that script in it's entirety could just be the
post-new script, and the offlineimap.postsynchook could just execute
notmuch new directly.

jamie.

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

Re: query on a subset of messages ?

[Sebastien Binet]

[-- Attachment #1: Type: text/plain, Size: 2461 bytes --]

Sebastien Binet <binet@cern.ch> writes:

> Austin Clements <amdragon@MIT.EDU> writes:
>
>> Quoth Sebastien Binet on Jul 09 at 10:25 am:
>>> 
>>> hi there,
>>> 
>>> I was trying to reduce the I/O stress during my usual email
>>> fetching+tagging by writing a little program using the go bindings to
>>> notmuch.
>>> 
>>> ie:
>>> db, status := notmuch.OpenDatabase(db_path,
>>>     		notmuch.DATABASE_MODE_READ_WRITE)
>>> query := db.CreateQuery("(tag:new AND tag:inbox)")
>>> msgs := query.SearchMessages()
>>> for _,msg := range msgs {
>>>   tag_msg(msg, tagqueries)
>>> }
>>> 
>>> 
>>> where tagqueries is a subquery of the form:
>>> [
>>>     {
>>>         "Cmd": "+to-me",
>>>         "Query": "(to:sebastien.binet@cern.ch and not tag:to-me)"
>>>     },
>>>     {
>>>         "Cmd": "+sci-notmuch",
>>>         "Query": "from:notmuch@notmuchmail.org or to:notmuch@notmuchmail.org or subject:notmuch"
>>>     }
>>> ]
>>> 
>>> 
>>> the idea being that I only need to crawl through the db only once and
>>> then iteratively apply tags on those messages (instead of repeatedly
>>> running "notmuch tag ..." for each and every of those many
>>> 'tag-queries')
>>> 
>>> I couldn't find any C-API to do such a thing using the notmuch library.
>>> did I overlook something ?
>>> 
>>> Is it something useful to add ?
>>> 
>>> -s
>>
>> Have you tried a more direct translation of the multiple notmuch tag
>> commands into Go, where you don't worry about subsetting the queries?
>> Unless you're tagging a huge number of messages, the cost of notmuch
>> tag is almost certainly the fsync that it does when it closes the
>> database (which every call to notmuch tag must do).  However, in Go,
>> you can keep the database open across all of the tagging operations
>> and then close and fsync it just once.
>
> nope, I haven't tried that, but will do.
>
>>
>> Note that there is an important optimization in notmuch tag that you
>> might have to replicate.  It manipulates the original query to exclude
>> messages that already have the desired tags, so that they get skipped
>> very efficiently at the earliest stage possible.
> I already have this in my original shell script.
> (wouldn't be too hard to automatically do, though.)

FYI, I've put this into a new notmuch-mtag go-based binary over here:
https://github.com/sbinet/notmuch/blob/dev/go-bindings/bindings/go/src/notmuch-mtag/main.go


-s

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]