Re: Introducing notmuchsync

[Ben Gamari <bgamari@gmail.com>]

Excerpts from Michal Sojka's message of Mon Mar 01 03:57:26 -0500 2010:
> Hi, when you are on this topic, I'll put my two cents in. Currently
> notmuch works only with mail-store comprised of files. People want to
> work with their mails on multiple computers so there are several
> ideas/solutions how to achieve that. Notmuchsync, which plays well with
> offlineimap, is one of them. Another idea is Git based mail-store, which
> I would really like to have.
> 
> If we want to make notmuch work with Git-based mail-store, it will be
> necessary to make the interface between notmuch core and mail-store
> handling code a bit more abstract so that it will be possible to
> configure mail-store type to be used.

I agree that we should make notmuch less tied to maildirs as the only
possible mailstore. In the interest of simplicity, however, it's
important to keep the surface area of the interface down.
> 
> This abstract mail-store interface will contain methods for the
> following operations:
> 1. read mail identified by a path from mail-store
>    (current notmuch uses fread() for this, for git this will be
>    something like system("git cat-file ..."))
>
Yep. Seems reasonable.

> 2. get the list of new mails which need to be indexed
>    (current notmuch does recursive file traversal, for git-based store
>    it will be something like system("git diff-tree --name-status ...")
>
Is this really necessary? Another option (which I believe has been
mentioned here in the past) is to simply pass notmuch new a list of
message "paths" to add (although I'm not sure if many mail delivery
programs give you that sort of information). How do you propose that the
backends keep track of what mail has been indexed?

> 3. add/remove tags for a given message (this will be NULL for current
>    notmuch functionality)
Yep, makes sense.

> 
> This way the mail-store will be able to store both mails and
> corresponding tags and in case of git, it will be easy to synchronize
> mail-stores on multiple computers.

Sounds great. Can't wait!
> 
> Now, if we have this, it would be very easy to add support for
> Maildir-based mail-store. The implementation of the first two methods
> would be the same as what is currently in notmuch and the third method
> would rename files in mailstore if certain tags (such as unread) are
> added or removed. In case of rename, notmuch database would be
> immediately updated to reflect the change.
> 
The interface here seems a little vague. How would the backend notify
notmuch that the filename has changed?

> So to summarize, I think there should be an abstract layer for
> handling different types of mail-store. I can see at least three
> possible implementations of this abstract interface: 1) immutable
> mail-store (as currently implemented in notmuch) 2) Maildir-based
> mail-store for limited synchronization with other Maildir tools and 
> 3) git-based mail-store for full synchronization.

Don't forget mbox. It seems like it would be pretty trivial to
implement (although I'm not sure what performance would look like).
> 
> I've already started working on this, but I'm still in the state where I
> mainly study how notmuch works in order not to break its current
> functionality.

With all of this infrastructure, it seems like it wouldn't be too
difficult to support messages from multiple backends in a single index.
Not sure if people would be interested enough to warrant the added
complexity though.

> 
> I'd like to hear what others think about this idea.
> 
That's my uninformed opinion. Take it with an appropriately sized grain
of salt. I do think that some sort of abstraction will be necessary
though. I'm glad you're taking a look at this. I've been wanting to
start hacking away at some sort of prototype git backend, but I haven't
wanted to start before we have the appropriate infrastructure in
notmuch.

Cheers,

- Ben