Possible addtions to notmuch new ?

Possible addtions to notmuch new ?

[Vladimir Marek]

Hi,

My mail setup is a directory containing several subdirectories each
subdirectory corresponds to one real mail account I am using. Each mail
account is synchronized differently - I am using offlineimap, fetchmeail
or even synthetically created emails (I am writing very simple jabber<->
mail gate).Every now and then I am running 'notmuch new' to discover new
emails and make them available in my MUA.

That works pretty well, but has some disadvantages too
 - notmuch new takes very long time (30s) during which the notmuch
   database seems to be locked for any other updates from my MUA
 - notmuch new takes long time because it always processes my archive
   dir containing many files. That's mostly un-necessary as typically
   there's no new mail delivered
 - I don't have the possibility of passing new mails through procmail.
   That would be useful for example for changing cron mail subjects,
   putting related automated mails into threads (bugzilla, etc.).


I was thinking that if we could split the new mail discovery from
it's processing, it would solve the disadvantages I'm facing. Something
like

notmuch new --verbose --dry-run [dir] | my_filter | notmuch insert -

It would work
 - --dry-run would not lock and change the database
 - --verbose would print the changes to stdout/stderr. Something like:

new mail/file.1
new mail/file.2
deleted mail/file.3
renamed mail/file.4 mail/file.5
...

[dir] would limit the scope of 'notmuch' new search to dir and it's
subdirectories. Alternatively we could have set of include or exclude
rules similarly to rsync (for example), but [dir] is easier to
implement.

'my_filter' would be my script which could change the contents of emails
before they are inserted into notmuch database.

Notmuch insert would be able to not only add new mails, but also remove
old ones or note that the file was renamed.

How would this sound?

I'm not saying I would implement this, I'm rather curious where would
you like to see notmuch in the future.

Cheers
-- 
	Vlad

Re: Possible addtions to notmuch new ?

[Austin Clements]

Quoth Vladimir Marek on Aug 12 at 11:34 am:
> Hi,
> 
> My mail setup is a directory containing several subdirectories each
> subdirectory corresponds to one real mail account I am using. Each mail
> account is synchronized differently - I am using offlineimap, fetchmeail
> or even synthetically created emails (I am writing very simple jabber<->
> mail gate).Every now and then I am running 'notmuch new' to discover new
> emails and make them available in my MUA.
> 
> That works pretty well, but has some disadvantages too
>  - notmuch new takes very long time (30s) during which the notmuch
>    database seems to be locked for any other updates from my MUA
>  - notmuch new takes long time because it always processes my archive
>    dir containing many files. That's mostly un-necessary as typically
>    there's no new mail delivered

Could you try this patch?  It's basically untested other than passing
the test suite, though in principle the worst harm it could do is make
notmuch new miss new messages or think renames are deletions.  If it
helps significantly with your performance problems, I'll clean it up
and add a test.

diff --git a/notmuch-new.c b/notmuch-new.c
index faa33f1..196c5cb 100644
--- a/notmuch-new.c
+++ b/notmuch-new.c
@@ -323,6 +323,9 @@ add_files (notmuch_database_t *notmuch,
     }
     db_mtime = directory ? notmuch_directory_get_mtime (directory) : 0;
 
+    if (directory && db_mtime == fs_mtime && st.st_nlink == 2)
+       goto DONE;
+
     /* If the database knows about this directory, then we sort based
      * on strcmp to match the database sorting. Otherwise, we can do
      * inode-based sorting for faster filesystem operation. */


>  - I don't have the possibility of passing new mails through procmail.
>    That would be useful for example for changing cron mail subjects,
>    putting related automated mails into threads (bugzilla, etc.).
> 
> 
> I was thinking that if we could split the new mail discovery from
> it's processing, it would solve the disadvantages I'm facing. Something
> like
> 
> notmuch new --verbose --dry-run [dir] | my_filter | notmuch insert -
> 
> It would work
>  - --dry-run would not lock and change the database
>  - --verbose would print the changes to stdout/stderr. Something like:
> 
> new mail/file.1
> new mail/file.2
> deleted mail/file.3
> renamed mail/file.4 mail/file.5
> ...
> 
> [dir] would limit the scope of 'notmuch' new search to dir and it's
> subdirectories. Alternatively we could have set of include or exclude
> rules similarly to rsync (for example), but [dir] is easier to
> implement.
> 
> 'my_filter' would be my script which could change the contents of emails
> before they are inserted into notmuch database.
> 
> Notmuch insert would be able to not only add new mails, but also remove
> old ones or note that the file was renamed.
> 
> How would this sound?
> 
> I'm not saying I would implement this, I'm rather curious where would
> you like to see notmuch in the future.
> 
> Cheers

Re: Possible addtions to notmuch new ?

[Adam Wolfe Gordon]

On Mon, Aug 12, 2013 at 3:34 AM, Vladimir Marek
<Vladimir.Marek@oracle.com> wrote:
> [dir] would limit the scope of 'notmuch' new search to dir and it's
> subdirectories. Alternatively we could have set of include or exclude
> rules similarly to rsync (for example), but [dir] is easier to
> implement.

I sent a series recently that added a command to essentially run
'notmuch new' on a single message file. One reviewer suggested that I
make it an option to notmuch new rather than its own command, which I
think is is a good idea.

When I have some time to get back to that patch series, I'll make the
change as suggested. I think it makes sense to let it take either a
subdirectory or a single file, which would make it work for this part
of your usecase.

-- Adam

Re: Re: Possible addtions to notmuch new ?

[Vladimir Marek]

> > My mail setup is a directory containing several subdirectories each
> > subdirectory corresponds to one real mail account I am using. Each mail
> > account is synchronized differently - I am using offlineimap, fetchmeail
> > or even synthetically created emails (I am writing very simple jabber<->
> > mail gate).Every now and then I am running 'notmuch new' to discover new
> > emails and make them available in my MUA.
> > 
> > That works pretty well, but has some disadvantages too
> >  - notmuch new takes very long time (30s) during which the notmuch
> >    database seems to be locked for any other updates from my MUA
> >  - notmuch new takes long time because it always processes my archive
> >    dir containing many files. That's mostly un-necessary as typically
> >    there's no new mail delivered
> 
> Could you try this patch?  It's basically untested other than passing
> the test suite, though in principle the worst harm it could do is make
> notmuch new miss new messages or think renames are deletions.  If it
> helps significantly with your performance problems, I'll clean it up
> and add a test.
> 
> diff --git a/notmuch-new.c b/notmuch-new.c
> index faa33f1..196c5cb 100644
> --- a/notmuch-new.c
> +++ b/notmuch-new.c
> @@ -323,6 +323,9 @@ add_files (notmuch_database_t *notmuch,
>      }
>      db_mtime = directory ? notmuch_directory_get_mtime (directory) : 0;
>  
> +    if (directory && db_mtime == fs_mtime && st.st_nlink == 2)
> +       goto DONE;
> +
>      /* If the database knows about this directory, then we sort based
>       * on strcmp to match the database sorting. Otherwise, we can do
>       * inode-based sorting for faster filesystem operation. */


I'm sorry for my late reply. It cuts the average time of 'notmuch new'
from 25s to 0.2s . Which is a bit scary :) But understandable. I have
the notmuch database on NFS mount which hopefully won't make any
difference here.

Thank you for the tip!
-- 
	Vlad

Re: Re: Re: Possible addtions to notmuch new ?

[Vladimir Marek]

Hi,

> > > My mail setup is a directory containing several subdirectories each
> > > subdirectory corresponds to one real mail account I am using. Each mail
> > > account is synchronized differently - I am using offlineimap, fetchmeail
> > > or even synthetically created emails (I am writing very simple jabber<->
> > > mail gate).Every now and then I am running 'notmuch new' to discover new
> > > emails and make them available in my MUA.
> > > 
> > > That works pretty well, but has some disadvantages too
> > >  - notmuch new takes very long time (30s) during which the notmuch
> > >    database seems to be locked for any other updates from my MUA
> > >  - notmuch new takes long time because it always processes my archive
> > >    dir containing many files. That's mostly un-necessary as typically
> > >    there's no new mail delivered
> > 
> > Could you try this patch?  It's basically untested other than passing
> > the test suite, though in principle the worst harm it could do is make
> > notmuch new miss new messages or think renames are deletions.  If it
> > helps significantly with your performance problems, I'll clean it up
> > and add a test.
> > 
> > diff --git a/notmuch-new.c b/notmuch-new.c
> > index faa33f1..196c5cb 100644
> > --- a/notmuch-new.c
> > +++ b/notmuch-new.c
> > @@ -323,6 +323,9 @@ add_files (notmuch_database_t *notmuch,
> >      }
> >      db_mtime = directory ? notmuch_directory_get_mtime (directory) : 0;
> >  
> > +    if (directory && db_mtime == fs_mtime && st.st_nlink == 2)
> > +       goto DONE;
> > +
> >      /* If the database knows about this directory, then we sort based
> >       * on strcmp to match the database sorting. Otherwise, we can do
> >       * inode-based sorting for faster filesystem operation. */
> 
> 
> I'm sorry for my late reply. It cuts the average time of 'notmuch new'
> from 25s to 0.2s . Which is a bit scary :) But understandable. I have
> the notmuch database on NFS mount which hopefully won't make any
> difference here.

Is there anything missing before including the patch into notmuch
sources? Today I rebuilt notmuch and forgot to include this patch and
"notmuch new" is dead slow again.

Cheers
-- 
	Vlad