Introducing notmuchsync

Introducing notmuchsync

[Sebastian Spaeth]

Dear list,

I really want to sync my maildir flags/folders with notmuch tags and as
I haven't seen a script to do that, I've written (the beginnings of?)
one.

It is probably pre-alpha but I am a fan of release early, release often
and if someone finds it useful to build upon, it was already worth it. I
put the code here: http://github.com/spaetz/notmuchsync

What does it do?
================
 - Synchronizes the "S" flag with the "unread" tag (1-way). The
 synchronization direction is decided by using either --sync (change
 maildir flags according to notmuch) or --revsync (change notmuch tags according to maildir). By default it always checks the mails from the previous 30
 days (but can also do --all mails if you have plenty of RAM and time).
 - Deletes all mail files that have the "delete" tag
 - Quiet/normal/verbose logging 

Workflow
========
1a) start out with notmuchsync -r in order to initialize the notmuch tag
database based on maildir 'S' flags. This is probably what everyone but
Carl Worth wants. :-) If the "--all" command works for you, even better,
but I doubt it as it would store all your mails in RAM.
1b) start out with not

What needs improvment
=====================
 - It's a python script in one file. The architecture might need some
 cleanup. Documentation needs work too.
 - It temporarily slurps in all your mails from the last 30 days into
 RAM. I am waiting for "notmuchs show blah --output filename --output
 tags" to improve that :). Generally the parsing of the output of
 "notmuch show" is a bit hackyish with regexps at the moment.
 - Support parsing "chunks" of "notmuch show" output or do several
   runs for -all, using monthly intervals to make --all work.
 - Support for the "T" (delete), "F" (flag), "D" (draft) tags/flags.
   Should be pretty easy to add.
 - Support syncing of the "inbox" tag if the mail is in a dir different
 than INBOX. No clue yet how to do. Perhaps if a maildir name
 corresponds to a notmuch tag name, we move the mail there?
 - It seems to work for me, but is untested in environments with spaces
 in paths and windows machines.
 - the --doeverything command that prunes and syncs in one go

What else
=========
- It is GPL v2.1+
- Please improve and/or fork this thing.
- If you don't like it, just ignore it. Thanks for your understanding.

-----------------------------------------------------------------------
Usage:
-p --prune	Prune deleted mails
-s --sync	Sync from notmuch tags to maildir flags.
		By default it will only look for mails from the last 30 days.
		Use --all to look at earlier mails.
		Beware, if timestamps are more than 30 days in the
                future, 
                we won't handle it.
-r --revsync	Sync tags from maildir to notmuch. See also -s.
Options:
-d		 Really verbose debug oputput.
-q		 Log only errors.
--dryrun	 Do not really modify notmuch db or mail files
                 Works together with -p -s -r

----------------------------------------------------------------------

Let me know what you like or don't. If you want github contributor
right, I am willing to hand out access quite liberal. And as always:
patches welcome. :-)

spaetz

Re: Introducing notmuchsync

[Sebastian Spaeth]

On Mon, 18 Jan 2010 16:12:28 +0100, "Sebastian Spaeth" <Sebastian@SSpaeth.de> wrote:
DOH, I should actually have finished the Workflow section before sending
it off :-):

Workflow
========
1a) start out with notmuchsync -r in order to initialize the notmuch tag
database based on maildir 'S' flags. This is probably what everyone but
Carl Worth wants. :-) If the "--all" command works for you, even better,
but I doubt it as it would store all your mails in RAM. I'll need to
work on some chunking for this.
1b) start out with nnotmuchsync -s if your notmuch tags are
authoritative and it will modify your Maildir flags accordingly.

After that a periodic:
2) "notmuchsync -s" syncs tags to flags (Currently 'S' only)
3) "notmuchsync -p" purges mail files with "delete" flags

Re: Introducing notmuchsync

[Michal Sojka]

On Monday 18 of January 2010 16:12:28 Sebastian Spaeth wrote:
> Dear list,
> 
> I really want to sync my maildir flags/folders with notmuch tags and as
> I haven't seen a script to do that, I've written (the beginnings of?)
> one.
> 
> It is probably pre-alpha but I am a fan of release early, release often
> and if someone finds it useful to build upon, it was already worth it. I
> put the code here: http://github.com/spaetz/notmuchsync

Hi,

with which Python version is this meant to work? I get the following error:
$ ./notmuchsync -r
Traceback (most recent call last):
  File "./notmuchsync", line 316, in <module>
    notmuch.syncTags(frommaildir=True, dryrun=dryrun)
  File "./notmuchsync", line 249, in syncTags
    searchterm = "{0}..{1}".format(now-2592000,now+2592000)
AttributeError: 'str' object has no attribute 'format'

Michal

Re: Introducing notmuchsync

[Marten Veldthuis]

On Mon, 18 Jan 2010 16:12:28 +0100, "Sebastian Spaeth" <Sebastian@SSpaeth.de> wrote:
> What does it do?
> ================
>  - Synchronizes the "S" flag with the "unread" tag (1-way). The
>  synchronization direction is decided by using either --sync (change
>  maildir flags according to notmuch) or --revsync (change notmuch tags according to maildir). By default it always checks the mails from the previous 30
>  days (but can also do --all mails if you have plenty of RAM and time).
>  - Deletes all mail files that have the "delete" tag
>  - Quiet/normal/verbose logging 

Excellent. Sounds like nearly exactly what I thought about doing last
weekend, except that I couldn't find a nice Ruby library quickly and
then my attention-span was over :)

Also, I'm curious as to Carl's opinion to this, but as far as I'm
concerned, not everything about notmuch needs to be coded in
C. Obviously, things interacting with the database directly need to, but
if it could be built upon the notmuch C-based CLI commands, and be fast
enough, would that be eligible to make it into the repository?

-- 
- Marten

Re: Introducing notmuchsync

[Sebastian Spaeth]

On Mon, 18 Jan 2010 16:41:25 +0100, Michal Sojka <sojkam1@fel.cvut.cz> wrote:
> with which Python version is this meant to work? I get the following error:
> $ ./notmuchsync -r

> AttributeError: 'str' object has no attribute 'format'

DOH, I just used 2.6-style formatting. But as there is little reason for
this, I just reverted to old-style formatting. Just pull from git again.

I have no clue about the needed version, but it should definitely be
<2.6 now.

spaetz

Re: Introducing notmuchsync

[Sebastian Spaeth]

> What needs improvment?
>  - Support for the "T" (delete), "F" (flagged), "D" (draft) tags/flags.

Having just finished this feature, I herewith declare notmuchsync to be version
0.0.1 :). It works for fine for me, I use it to sync the "FTDS" Maildir flags
(corresponding notmuch flags: 'unread','flagged','draft','delete'), as
well as for periodically pruning mail files that are marked as 'delete'.

Let me know if you have issues with it.

The 2 most urgent issues are: 
 - Move read files from 'new' to 'cur' folder. At what point is that
   moving typically done in Maildir? When the user has actually looked
   at the mail?
 - chunked output parsing (9k messages went through fine on my laptop,
   but I doubt that 200k msg would)

Sebastian

Re: Introducing notmuchsync

[Marten Veldthuis]

On Tue, 19 Jan 2010 14:37:03 +0100, "Sebastian Spaeth" <Sebastian@SSpaeth.de> wrote:
>  - Move read files from 'new' to 'cur' folder. At what point is that
>    moving typically done in Maildir? When the user has actually looked
>    at the mail?

Yes, exactly that.

-- 
- Marten

Re: Introducing notmuchsync

[Jameson Rollins]

[-- Attachment #1: Type: text/plain, Size: 1460 bytes --]

On Tue, Jan 19, 2010 at 04:00:06PM +0100, Marten Veldthuis wrote:
> On Tue, 19 Jan 2010 14:37:03 +0100, "Sebastian Spaeth" <Sebastian@SSpaeth.de> wrote:
> >  - Move read files from 'new' to 'cur' folder. At what point is that
> >    moving typically done in Maildir? When the user has actually looked
> >    at the mail?
> 
> Yes, exactly that.

Actually, I don't think this is exactly correct.  I believe that the
move from new to cur should be done by the MUA when the MUA actually
processes the message, but before the user views the message.  The S
flag should then be added by the MUA when the user actually views the
message.  The MTA should be delivering messages to new, and the MUA
should move it to cur.  That is at least how's it's described by DJB
for qmail (see THE MAILDIR STRUCTURE section in the qmail man page):

http://qmail.org/man/man5/maildir.html

I can't find an official "spec", though, and the cr.yp.to maildir page
is a little vague on the issue:

http://cr.yp.to/proto/maildir.html

Currently notmuch itself does not really conform to what I believe is
the maildir spec, which makes it a little difficult to use with other
MUAs.  Notmuch does not move messages from new to cur, or modify the
flags.  I've been arguing that notmuch should conform to this spec
directly, by moving messages from new to cur when it processes the
maildir, and by keeping tags synced with the maildir message flags.

jamie.

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: Introducing notmuchsync

[Sebastian Spaeth]

On Tue, 19 Jan 2010 16:00:06 +0100, Marten Veldthuis <marten@veldthuis.com> wrote:
> On Tue, 19 Jan 2010 14:37:03 +0100, "Sebastian Spaeth" <Sebastian@SSpaeth.de> wrote:
> >  - Move read files from 'new' to 'cur' folder. At what point is that
> >    moving typically done in Maildir? When the user has actually looked
> >    at the mail?
> 
> Yes, exactly that.

Thanks Marten. Implemented moving from "new" to "cur" now.

spaetz

Re: Introducing notmuchsync

[Sebastian Spaeth]

On Tue, 19 Jan 2010 10:24:27 -0500, Jameson Rollins <jrollins@finestructure.net> wrote:
> Actually, I don't think this is exactly correct.  I believe that the
> move from new to cur should be done by the MUA when the MUA actually
> processes the message, but before the user views the message.  The S
> flag should then be added by the MUA when the user actually views the
> message.  The MTA should be delivering messages to new, and the MUA
> should move it to cur.  That is at least how's it's described by DJB
> for qmail (see THE MAILDIR STRUCTURE section in the qmail man page):

I have done it now as suggested by Marten: move the mail to "cur" when the
user has actually seen it (as per lack of the 'unread' tag). While I
agree that it probably should move it earlier (when it is "processed"
whatever that means for notmuch), this was just the most convenient for
me. In addition, I have also not found a maildir specification, so the
behavior seems to be a bit undefined.

> Currently notmuch itself does not really conform to what I believe is
> the maildir spec, which makes it a little difficult to use with other
> MUAs.  Notmuch does not move messages from new to cur, or modify the
> flags.  

That is why I have coded my notmuchsync tool. It does all that. While I
agree that notmuch should probably (and faster) do all that itself, the
current design seems to be to keep notmuch flexible, small and to never touch
your mailstore. Until that changes, surrounding scripts will have to
perform these tasks.

My current synchronization script looks basically like this:

notmuch new       # make db consistent (earlier deleted mails etc)
notmuchsync -s -n # MailDir flags update and "cur" dir moving
offlineimap       # sync with IMAP server
notmuch new       # incorporate new mails in notmuch db

spaetz

Re: Introducing notmuchsync

[Jameson Rollins]

[-- Attachment #1: Type: text/plain, Size: 1387 bytes --]

On Tue, Jan 19, 2010 at 04:52:42PM +0100, Sebastian Spaeth wrote:
> That is why I have coded my notmuchsync tool. It does all that. While I
> agree that notmuch should probably (and faster) do all that itself, the
> current design seems to be to keep notmuch flexible, small and to never touch
> your mailstore. Until that changes, surrounding scripts will have to
> perform these tasks.

I should have mentioned in my previous mail that I think this tool is
a great idea, and I plan on using it.  I just hope that all of it's
functionality will be integrated directly into notmuch itself.

That said, I have vasilated just a bit on this, as to whether notmuch
should touch the mail at all, or just process it.  But having thought
about it a bit, I think that notmuch really *is* an MUA, or at least
the mail processing part of a MUA (MUA minus message reader), and
should therefore do the appropriate things with the maildir.

> My current synchronization script looks basically like this:
> 
> notmuch new       # make db consistent (earlier deleted mails etc)
> notmuchsync -s -n # MailDir flags update and "cur" dir moving
> offlineimap       # sync with IMAP server
> notmuch new       # incorporate new mails in notmuch db

Do you do another notmuchsync after the final notmuch new, to get any
new flags in the maildir synced with the database?

jamie.

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: Introducing notmuchsync

[Sebastian Spaeth]

On Tue, 19 Jan 2010 11:04:49 -0500, Jameson Rollins <jrollins@finestructure.net> wrote:
> I should have mentioned in my previous mail that I think this tool is
> a great idea, and I plan on using it.  I just hope that all of it's
> functionality will be integrated directly into notmuch itself.

I fully agree. 
[Snip more stuff I fully agree with]
[snip workflow]

> Do you do another notmuchsync after the final notmuch new, to get any
> new flags in the maildir synced with the database?

Ohh, I had never considered that, but true, if another mail client
modifies maildir flags, we probably want to have that reflected as notmuch
tags as well...

So, yes that should work even better (my real script is a bit fancier,
changing output options ans stuff):

notmuch new       # make db consistent (earlier deleted mails etc)
notmuchsync -s -n # MailDir flags update and "cur" dir moving
offlineimap       # sync with IMAP server
notmuch new       # incorporate new mails in notmuch db
notmuchsync -r -n # reflect changed MailDir flags in notmuch tags

Sorry for all the spam today about my toys. I'll try to keep a bit
more quiet in the future in order to not start annoying you. :-)

spaetz

Re: Introducing notmuchsync

[Servilio Afre Puentes]

2010/1/19 Jameson Rollins <jrollins@finestructure.net>:
[...]
> That said, I have vasilated just a bit on this, as to whether notmuch
> should touch the mail at all, or just process it.  But having thought
> about it a bit, I think that notmuch really *is* an MUA, or at least
> the mail processing part of a MUA (MUA minus message reader), and
> should therefore do the appropriate things with the maildir.

I on the other hand think that it is just the indexer, and that the
right way to integrate it into a MUA is by making this MUA call
notmuch to update the DB/index with the actions just performed on a
message/group of messages (moving, deleting, tagging, etc.).

For what I've read so far from messages of the authors in this mailing
list and the programmed behaviour, I think that is the intention
(though not fully implemented yet) and in my opinion the right one.
Maybe if the notmuch.el code would have been written as a set of hooks
into Gnus for customizing views this would have been more evident
(though I haven't neither used or looked at the vim front-end code),
or maybe I am missing the point entirely and the long-term goal of the
authors is to have a full-fledged MUA.

Regards,

Servilio

Re: Introducing notmuchsync

[Jameson Rollins]

[-- Attachment #1: Type: text/plain, Size: 937 bytes --]

On Tue, Jan 19, 2010 at 11:29:10AM -0500, Servilio Afre Puentes wrote:
> For what I've read so far from messages of the authors in this mailing
> list and the programmed behaviour, I think that is the intention
> (though not fully implemented yet) and in my opinion the right one.
> Maybe if the notmuch.el code would have been written as a set of hooks
> into Gnus for customizing views this would have been more evident
> (though I haven't neither used or looked at the vim front-end code),
> or maybe I am missing the point entirely and the long-term goal of the
> authors is to have a full-fledged MUA.

I don't think that the intention is to be a full-fledged MUA, but I'm
not sure it should just be an indexer either.  I think notmuch should
make itself basically a MUA minus mail reader, so that custom mail
readers can be built on top of it.  But like you, this is mostly just
my opinion of what notmuch should be doing.

jamie.

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: Introducing notmuchsync

[Tassilo Horn]

Jameson Rollins <jrollins@finestructure.net> writes:

Hi Jameson,

> That said, I have vasilated just a bit on this, as to whether notmuch
> should touch the mail at all, or just process it.  But having thought
> about it a bit, I think that notmuch really *is* an MUA, or at least
> the mail processing part of a MUA (MUA minus message reader), and
> should therefore do the appropriate things with the maildir.

I don't care if notmuch would somehow manipulate mail, but at least,
there has to be an option to avoid that.  I use it only as indexer and
search utility, and wrote some custom code to jump from notmuch buffers
to the right message in Gnus.  Notmuch indexes the mails directly in the
mail storage of my local IMAP server, which is not maildir, and I'm
pretty sure, my IMAP server wouldn't like someone modifying files in its
guts.

Bye,
Tassilo

Re: Introducing notmuchsync

[Carl Worth]

[-- Attachment #1: Type: text/plain, Size: 1836 bytes --]

On Mon, 18 Jan 2010 16:12:28 +0100, "Sebastian Spaeth" <Sebastian@SSpaeth.de> wrote:
> ================
>  - Synchronizes the "S" flag with the "unread" tag (1-way). The
>  synchronization direction is decided by using either --sync (change
>  maildir flags according to notmuch) or --revsync (change notmuch tags
>  according to maildir). By default it always checks the mails from the
>  previous 30 days (but can also do --all mails if you have plenty of
>  RAM and time).
>  - Deletes all mail files that have the "delete" tag
>  - Quiet/normal/verbose logging 

Thanks for contributing this, Sebastian.

Let me know if you'd like to host this within the contrib directory of
the notmuch repository.

>  - It temporarily slurps in all your mails from the last 30 days into
>  RAM. I am waiting for "notmuchs show blah --output filename --output
>  tags" to improve that :). Generally the parsing of the output of
>  "notmuch show" is a bit hackyish with regexps at the moment.

OK. So we'll be adding an --output option to give you just filenames
soon, and we've got JSON output now so you can avoid hacky regexps now.

Elsewhere in the thread Jameson Rollins wrote:
> I should have mentioned in my previous mail that I think this tool is
> a great idea, and I plan on using it.  I just hope that all of it's
> functionality will be integrated directly into notmuch itself.

I think that's the open question still. How much of this kind of
functionality do we integrate into notmuch itself. I don't know the
answer to that question yet, but I'm quite happy to see people
experimenting with doing scripts like this on top of notmuch already.

I do know that I want to do thing to make such scripting easier,
(independent of whether the current functionality gets folded into
notmuch).

-Carl


[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: Introducing notmuchsync

[Jameson Rollins]

[-- Attachment #1: Type: text/plain, Size: 1974 bytes --]

On Wed, 24 Feb 2010 10:19:06 -0800, Carl Worth <cworth@cworth.org> wrote:
> Elsewhere in the thread Jameson Rollins wrote:
> > I should have mentioned in my previous mail that I think this tool is
> > a great idea, and I plan on using it.  I just hope that all of it's
> > functionality will be integrated directly into notmuch itself.
> 
> I think that's the open question still. How much of this kind of
> functionality do we integrate into notmuch itself. I don't know the
> answer to that question yet, but I'm quite happy to see people
> experimenting with doing scripts like this on top of notmuch already.
> 
> I do know that I want to do thing to make such scripting easier,
> (independent of whether the current functionality gets folded into
> notmuch).

Hey, Carl.  Yeah, this was an old post and there's been lots of
discussion about it.  I'm certainly more flexible about my position now
than I think I was originally.

In fact, I actually gave up on syncing notmuch and maildir, and am
basically punting on maildir altogether.  This is slightly problematic
because notmuch is still missing some key features so I occasionally
have to use other mailers to achieve certain things (especially OpenPGP
stuff).  I worry about it down the line as well, if I ever have to use
another mailer for any reason.  All mail I've received since my switch
to notmuch will show up as "unread" in maildir readers.

I think notmuch wrapper scripts like notmuchsync are going to be crucial
for notmuch down the line, so I definitely agree that doing everything
possible to make it easier for them is a very good thing.  I am using
notmuchsync for deleting "delete" tagged messages, for which it's very
useful.  It's pretty slow, though, which I've been chalking up to the
fact that it has to parse the notmuch "show" output.  Once notmuch can
output just the paths to messages matching search results that should
get much much faster.

jamie.

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

Re: Introducing notmuchsync

[Sebastian Spaeth]

On Wed, 24 Feb 2010 10:19:06 -0800, Carl Worth <cworth@cworth.org> wrote:
> On Mon, 18 Jan 2010 16:12:28 +0100, "Sebastian Spaeth" <Sebastian@SSpaeth.de> wrote:
> > ================
> >  - Synchronizes the "S" flag with the "unread" tag (1-way). The
> >  synchronization direction is decided by using either --sync (change
> >  maildir flags according to notmuch) or --revsync (change notmuch tags
> >  according to maildir). By default it always checks the mails from the
> >  previous 30 days (but can also do --all mails if you have plenty of
> >  RAM and time).
> >  - Deletes all mail files that have the "delete" tag
> >  - Quiet/normal/verbose logging 
> 
> Thanks for contributing this, Sebastian.

No problem. It was just an itch I had :-). I have to say that I stopped
using the --revsync (change notmuch tags based on maildir flags) as I am
using the patch that does that from within "notmuch new" and which is
much faster than any external script could be.
 
> Let me know if you'd like to host this within the contrib directory of
> the notmuch repository.

I am fine with hosting it in contrib or on github whatever others
prefer.
 
> >  - It temporarily slurps in all your mails from the last 30 days into
> >  RAM. I am waiting for "notmuchs show blah --output filename --output
> >  tags" to improve that :). Generally the parsing of the output of
> >  "notmuch show" is a bit hackyish with regexps at the moment.
> 
> OK. So we'll be adding an --output option to give you just filenames
> soon, and we've got JSON output now so you can avoid hacky regexps now.

JSON will definitely help. I need to investigate that, and --output will
make help performance. So yes, these are good changes from a 3rd party
perspective. I was thinking of interfacing notmuch.so directly but as
long as the python bindings are still in development, I am not going to
look at this option.

> I think that's the open question still. How much of this kind of
> functionality do we integrate into notmuch itself. I don't know the
> answer to that question yet, but I'm quite happy to see people
> experimenting with doing scripts like this on top of notmuch already.

I'll comment on that in JRollins reply in a second :).
Sebastian

Re: Introducing notmuchsync

[Sebastian Spaeth]

On Wed, 24 Feb 2010 13:49:44 -0500, Jameson Rollins <jrollins@finestructure.net> wrote:
> On Wed, 24 Feb 2010 10:19:06 -0800, Carl Worth <cworth@cworth.org> wrote:
> > I think that's the open question still. How much of this kind of
> > functionality do we integrate into notmuch itself. I don't know the
> > answer to that question yet, but I'm quite happy to see people
> > experimenting with doing scripts like this on top of notmuch already.

It's a fine line and depends a lot on how people actually use
notmuch. One thing that I would strongly advocate is to take on the
"respect maildir flags when importing with notmuch new" patch. This
enables me to use webmail/thunderbird (my mom at my home place is curiously still
refusing to switch to notmuch) etc.

As for the rest, I'd agree that we can be conservative in accepting new
notmuch features (part of its appeal is that its so darn simple) while
trying to make things easier for an ecosystem of surrounding apps.

> In fact, I actually gave up on syncing notmuch and maildir, and am
> basically punting on maildir altogether.  This is slightly problematic
> because notmuch is still missing some key features so I occasionally
> have to use other mailers to achieve certain things (especially OpenPGP
> stuff).  I worry about it down the line as well, if I ever have to use
> another mailer for any reason.  All mail I've received since my switch
> to notmuch will show up as "unread" in maildir readers.


> I think notmuch wrapper scripts like notmuchsync are going to be crucial
> for notmuch down the line, so I definitely agree that doing everything
> possible to make it easier for them is a very good thing.  I am using
> notmuchsync for deleting "delete" tagged messages, for which it's very
> useful.  It's pretty slow, though, which I've been chalking up to the
> fact that it has to parse the notmuch "show" output.  Once notmuch can
> output just the paths to messages matching search results that should
> get much much faster.

It's slow for deleting messages? That would surprise me, as it only
needs to parse/look at messages with a 'delete' or 'maildir::trashed'
tag.
Syncing from notmuch to maildir flags *is* slow. Limiting the output
to filenames will help greatly there (I hope). However by design it
needs to compare tags from *all* your messages with the corresponding
file name as stored in notmuch. I have no solution as how to make this
easier, at least not without adding a "needsync" tag or so when "deleting, replying,
reading, etc".

Sebastian

Re: Introducing notmuchsync

[Michal Sojka]

On Wed, 24 Feb 2010 10:19:06 -0800, Carl Worth <cworth@cworth.org> wrote:
> Elsewhere in the thread Jameson Rollins wrote:
> > I should have mentioned in my previous mail that I think this tool is
> > a great idea, and I plan on using it.  I just hope that all of it's
> > functionality will be integrated directly into notmuch itself.
> 
> I think that's the open question still. How much of this kind of
> functionality do we integrate into notmuch itself. I don't know the
> answer to that question yet, but I'm quite happy to see people
> experimenting with doing scripts like this on top of notmuch already.

Hi, when you are on this topic, I'll put my two cents in. Currently
notmuch works only with mail-store comprised of files. People want to
work with their mails on multiple computers so there are several
ideas/solutions how to achieve that. Notmuchsync, which plays well with
offlineimap, is one of them. Another idea is Git based mail-store, which
I would really like to have.

If we want to make notmuch work with Git-based mail-store, it will be
necessary to make the interface between notmuch core and mail-store
handling code a bit more abstract so that it will be possible to
configure mail-store type to be used.

This abstract mail-store interface will contain methods for the
following operations:
1. read mail identified by a path from mail-store
   (current notmuch uses fread() for this, for git this will be
   something like system("git cat-file ..."))
2. get the list of new mails which need to be indexed
   (current notmuch does recursive file traversal, for git-based store
   it will be something like system("git diff-tree --name-status ...")
3. add/remove tags for a given message (this will be NULL for current
   notmuch functionality)

This way the mail-store will be able to store both mails and
corresponding tags and in case of git, it will be easy to synchronize
mail-stores on multiple computers.

Now, if we have this, it would be very easy to add support for
Maildir-based mail-store. The implementation of the first two methods
would be the same as what is currently in notmuch and the third method
would rename files in mailstore if certain tags (such as unread) are
added or removed. In case of rename, notmuch database would be
immediately updated to reflect the change.

So to summarize, I think there should be an abstract layer for
handling different types of mail-store. I can see at least three
possible implementations of this abstract interface: 1) immutable
mail-store (as currently implemented in notmuch) 2) Maildir-based
mail-store for limited synchronization with other Maildir tools and 
3) git-based mail-store for full synchronization.

I've already started working on this, but I'm still in the state where I
mainly study how notmuch works in order not to break its current
functionality.

I'd like to hear what others think about this idea.

Bye
Michal

Re: Introducing notmuchsync

[Ben Gamari]

Excerpts from Michal Sojka's message of Mon Mar 01 03:57:26 -0500 2010:
> Hi, when you are on this topic, I'll put my two cents in. Currently
> notmuch works only with mail-store comprised of files. People want to
> work with their mails on multiple computers so there are several
> ideas/solutions how to achieve that. Notmuchsync, which plays well with
> offlineimap, is one of them. Another idea is Git based mail-store, which
> I would really like to have.
> 
> If we want to make notmuch work with Git-based mail-store, it will be
> necessary to make the interface between notmuch core and mail-store
> handling code a bit more abstract so that it will be possible to
> configure mail-store type to be used.

I agree that we should make notmuch less tied to maildirs as the only
possible mailstore. In the interest of simplicity, however, it's
important to keep the surface area of the interface down.
> 
> This abstract mail-store interface will contain methods for the
> following operations:
> 1. read mail identified by a path from mail-store
>    (current notmuch uses fread() for this, for git this will be
>    something like system("git cat-file ..."))
>
Yep. Seems reasonable.

> 2. get the list of new mails which need to be indexed
>    (current notmuch does recursive file traversal, for git-based store
>    it will be something like system("git diff-tree --name-status ...")
>
Is this really necessary? Another option (which I believe has been
mentioned here in the past) is to simply pass notmuch new a list of
message "paths" to add (although I'm not sure if many mail delivery
programs give you that sort of information). How do you propose that the
backends keep track of what mail has been indexed?

> 3. add/remove tags for a given message (this will be NULL for current
>    notmuch functionality)
Yep, makes sense.

> 
> This way the mail-store will be able to store both mails and
> corresponding tags and in case of git, it will be easy to synchronize
> mail-stores on multiple computers.

Sounds great. Can't wait!
> 
> Now, if we have this, it would be very easy to add support for
> Maildir-based mail-store. The implementation of the first two methods
> would be the same as what is currently in notmuch and the third method
> would rename files in mailstore if certain tags (such as unread) are
> added or removed. In case of rename, notmuch database would be
> immediately updated to reflect the change.
> 
The interface here seems a little vague. How would the backend notify
notmuch that the filename has changed?

> So to summarize, I think there should be an abstract layer for
> handling different types of mail-store. I can see at least three
> possible implementations of this abstract interface: 1) immutable
> mail-store (as currently implemented in notmuch) 2) Maildir-based
> mail-store for limited synchronization with other Maildir tools and 
> 3) git-based mail-store for full synchronization.

Don't forget mbox. It seems like it would be pretty trivial to
implement (although I'm not sure what performance would look like).
> 
> I've already started working on this, but I'm still in the state where I
> mainly study how notmuch works in order not to break its current
> functionality.

With all of this infrastructure, it seems like it wouldn't be too
difficult to support messages from multiple backends in a single index.
Not sure if people would be interested enough to warrant the added
complexity though.

> 
> I'd like to hear what others think about this idea.
> 
That's my uninformed opinion. Take it with an appropriately sized grain
of salt. I do think that some sort of abstraction will be necessary
though. I'm glad you're taking a look at this. I've been wanting to
start hacking away at some sort of prototype git backend, but I haven't
wanted to start before we have the appropriate infrastructure in
notmuch.

Cheers,

- Ben

Re: Introducing notmuchsync

[Michal Sojka]

On Mon, 01 Mar 2010 11:27:07 -0500, Ben Gamari <bgamari@gmail.com> wrote:
> > 2. get the list of new mails which need to be indexed
> >    (current notmuch does recursive file traversal, for git-based store
> >    it will be something like system("git diff-tree --name-status ...")
> >
> Is this really necessary? Another option (which I believe has been
> mentioned here in the past) is to simply pass notmuch new a list of
> message "paths" to add (although I'm not sure if many mail delivery
> programs give you that sort of information). 

This could also be possible, but now, you say "notmuch new" and notmuch
itself figure out what to index. If passing notmuch a list on files to
index will be the only supported way, it might be hard for new users to
start with notmuch. A nice thing on notmuch is that it can be used
almost without any configuration.

> How do you propose that the backends keep track of what mail has been
> indexed?

For example by using Xapian metadata:
notmuch->xapian_db->set_metadata ("git-head", sha1);

> > Now, if we have this, it would be very easy to add support for
> > Maildir-based mail-store. The implementation of the first two methods
> > would be the same as what is currently in notmuch and the third method
> > would rename files in mailstore if certain tags (such as unread) are
> > added or removed. In case of rename, notmuch database would be
> > immediately updated to reflect the change.
> > 
> The interface here seems a little vague. How would the backend notify
> notmuch that the filename has changed?

notmuch new

The idea is that tags changed by notmuch are stored immediately (and
database is updated accordingly), whereas when the mail store is changed
by an external tool, you must explicitly ask notmuch to notice that.

> > So to summarize, I think there should be an abstract layer for
> > handling different types of mail-store. I can see at least three
> > possible implementations of this abstract interface: 1) immutable
> > mail-store (as currently implemented in notmuch) 2) Maildir-based
> > mail-store for limited synchronization with other Maildir tools and 
> > 3) git-based mail-store for full synchronization.
> 
> Don't forget mbox. It seems like it would be pretty trivial to
> implement (although I'm not sure what performance would look like).

I personally do not use mboxes, so I'm not interested in them.

> > I've already started working on this, but I'm still in the state where I
> > mainly study how notmuch works in order not to break its current
> > functionality.
> 
> With all of this infrastructure, it seems like it wouldn't be too
> difficult to support messages from multiple backends in a single index.
> Not sure if people would be interested enough to warrant the added
> complexity though.

I'm currently not interested in such a functionality, but we can add it
later if there is a need.


Bye
Michal

Re: Introducing notmuchsync

[Ben Gamari]

Excerpts from Michal Sojka's message of Mon Mar 01 12:18:55 -0500 2010:
> > Is this really necessary? Another option (which I believe has been
> > mentioned here in the past) is to simply pass notmuch new a list of
> > message "paths" to add (although I'm not sure if many mail delivery
> > programs give you that sort of information). 
> 
> This could also be possible, but now, you say "notmuch new" and notmuch
> itself figure out what to index. If passing notmuch a list on files to
> index will be the only supported way, it might be hard for new users to
> start with notmuch. A nice thing on notmuch is that it can be used
> almost without any configuration.

It seems like a script in contrib/ would suffice.

> 
> > How do you propose that the backends keep track of what mail has been
> > indexed?
> 
> For example by using Xapian metadata:
> notmuch->xapian_db->set_metadata ("git-head", sha1);

However, this means that the backend would need to know about the
indexing database, which seems to me like a implementation detail that
should be hidden from the backend (perhaps? Maybe not, I suppose). I guess this all
depends upon how much we want to abstract out the backends.

> 
> > > Now, if we have this, it would be very easy to add support for
> > > Maildir-based mail-store. The implementation of the first two methods
> > > would be the same as what is currently in notmuch and the third method
> > > would rename files in mailstore if certain tags (such as unread) are
> > > added or removed. In case of rename, notmuch database would be
> > > immediately updated to reflect the change.
> > > 
> > The interface here seems a little vague. How would the backend notify
> > notmuch that the filename has changed?
> 
> notmuch new
> 
> The idea is that tags changed by notmuch are stored immediately (and
> database is updated accordingly), whereas when the mail store is changed
> by an external tool, you must explicitly ask notmuch to notice that.
> 
Certainly, I understand that and believe that that is the only sane
approach. However, you currently have no mechanism in your interface to
allow the backend to notify notmuch that the file name has changed.
Considering this is the sole value identifying the message to notmuch,
you definitely need to tell notmuch about the change.

> > Don't forget mbox. It seems like it would be pretty trivial to
> > implement (although I'm not sure what performance would look like).
> 
> I personally do not use mboxes, so I'm not interested in them.

Fair enough. Just figured it wouldn't be too difficult (although I know
very little about the format).

> > With all of this infrastructure, it seems like it wouldn't be too
> > difficult to support messages from multiple backends in a single index.
> > Not sure if people would be interested enough to warrant the added
> > complexity though.
> 
> I'm currently not interested in such a functionality, but we can add it
> later if there is a need.

Certainly. Just throwing out ideas. 

- Ben

Re: Introducing notmuchsync

[Michal Sojka]

On Mon, 01 Mar 2010 13:43:24 -0500, Ben Gamari <bgamari@gmail.com> wrote:
> Excerpts from Michal Sojka's message of Mon Mar 01 12:18:55 -0500 2010:
> > > How do you propose that the backends keep track of what mail has been
> > > indexed?
> > 
> > For example by using Xapian metadata:
> > notmuch->xapian_db->set_metadata ("git-head", sha1);
> 
> However, this means that the backend would need to know about the
> indexing database, which seems to me like a implementation detail that
> should be hidden from the backend (perhaps? Maybe not, I suppose). I guess this all
> depends upon how much we want to abstract out the backends.

I do not see why the databse should be hidded. It will be easier to
implement if the backend knows about it. Also, I'm not sure whether this
should be called "backend". From certain point of view Xapian can be
considered as backend and mail store implemetation will be a frontend.

- Michal