syncing mail by simply syncing the filesystem

syncing mail by simply syncing the filesystem

[Christoph Groth]

Dear notmuch hackers and users,

After 10 years of using Gnus I am considering to replace it with
notmuch.  On this occasion I would also like to make my e-mail-setup
distributed in a way which does not require any central infrastructure,
i.e. I would like to be able to read and write e-mail on several
computers without relying on a central IMAP server.

The ideal setup would be one which would allow to use any of my
computers independently (for example when I'm away with my laptop).
When necessary, the local mail stores would be synchronized by simply
syncing the home directory (I'm using unison for this).

Does anyone have a working setup like this?  I'm grateful for any tips
and ideas.

Alternatively, if syncing using solely the file system does not work, I
could setup dovecot on one of my machines and use it as a central IMAP
server.  On the clients I imagine a setup where mail would be fetched by
fetchmail, handed over to procmail which would put it into the local
maildir.  This maildir would be synchronized with the IMAP server by
offlineimap.  Any experiences with a setup like this?

Thanks,
Christoph

Re: syncing mail by simply syncing the filesystem

[David Bremner]

On Sat, 12 Jun 2010 09:13:07 +0200, Christoph Groth <cwg@falma.de> wrote:
> The ideal setup would be one which would allow to use any of my
> computers independently (for example when I'm away with my laptop).
> When necessary, the local mail stores would be synchronized by simply
> syncing the home directory (I'm using unison for this).

The issue with syncing notmuch is that you need to sync the tags
database as well. You can see one example way of doing that at 

         http://pivot.cs.unb.ca/git/?p=notmuch-scripts.git;a=blob;f=scripts/gitmuch;hb=HEAD

Another, potentially more elegant approach is to use "notmuch remote"
scripts written by Jesse Rosenthal. I haven't done this myself, but I
did think was feasible when I looked at them.

More information can be found in message

     m1k4rkkchy.fsf@watt.gilman.jhu.edu
or

 git clone http://jkr.acm.jhu.edu/git/nm-remote.git


I'm sure there are other approaches as well.

d

Re: syncing mail by simply syncing the filesystem

[Jesse Rosenthal]

> Another, potentially more elegant approach is to use "notmuch remote"
> scripts written by Jesse Rosenthal. I haven't done this myself, but I
> did think was feasible when I looked at them.

I'd certainly be happy to get some feedback on this idea, but I'm not
sure if it's the right tool for this problem, since it deals with
namespaces and he wants the universal namespace. In other words, I think
it mainly reduces to 'notmuch dump' and 'curl' there (though I could be
forgetting some cleverness on my part).

> On Sat, 12 Jun 2010 09:13:07 +0200, Christoph Groth <cwg@falma.de> wrote:
> > The ideal setup would be one which would allow to use any of my
> > computers independently (for example when I'm away with my laptop).
> > When necessary, the local mail stores would be synchronized by simply
> > syncing the home directory (I'm using unison for this).

Christoph -- some other ideas to think about:

1. If your other computers will have a consistent fast internet
connection (or a sufficient connection to use IMAP) consider remote
usage, keeping your database on one computer:

http://notmuchmail.org/remoteusage/

2. You could brute-force it. Include a nm_dump file as one of your
unison paths, along with your Maildir, then always run:

notmuch dump > nm_dump
unison
notmuch restore nm_dump

If you take a bit of care in how you tell unison nm_dump files, this
also has the benefit of not just assuming the most recent state is the
correct one. But the syntax is pretty easy, so that shouldn't be a
problem. I don't think unison lets you run hooks (I could be wrong), but
this could be easily scriptable.

2a. Just a thought -- would merging be easier if you used git/hg instead
of unison? Just make sure to gitignore your notmuch db.

3. Have an IMAP server on one computer, a notmuch-computer on
another. Have rsync passively pull down from your IMAP maildir. This
doesn't synchronize tags, but it might be good enough if you just want
to check on your phone occasionally.

FWIW, I have a combination of #3 and #1 above.

University --getmail/maildrop--> IMAP server --rsync--> home
                                                        / \
                                                       remote
                                                      /     \
                                                   work     laptop

Anyway, just some ideas. I'm sure there are plenty of others. (Sort of
curious to see them, since I'm sure there has been all sorts of
interesting hacks and workarounds). Everything above is sort of a hacks,
but it all works well enough for me. Hopefully there will be a canonical
solution to this sooner rather than later, though.

Best,
Jesse

Re: syncing mail by simply syncing the filesystem

[Arvid Picciani]

On Sat, 12 Jun 2010 09:13:07 +0200, Christoph Groth <cwg@falma.de> wrote:

> The ideal setup would be one which would allow to use any of my
> computers independently (for example when I'm away with my laptop).
> When necessary, the local mail stores would be synchronized by simply
> syncing the home directory (I'm using unison for this).

syncing maildir works.  notmuch's db resides inside your maildir, but its
probably not safe to do while its running. And you can only have one
place receive your mail, otherwise you loose tags.

Re: syncing mail by simply syncing the filesystem

[Jesse Rosenthal]

Sorry -- unclear on a couple of points:

On Sat, 12 Jun 2010 09:22:29 -0400, Jesse Rosenthal <jrosenthal@jhu.edu> wrote:
> 2. You could brute-force it. Include a nm_dump file as one of your
> unison paths, along with your Maildir, then always run:
> 
> notmuch dump > nm_dump
> unison

You have to run "notmuch new" here.

> notmuch restore nm_dump

> If you take a bit of care in how you tell unison nm_dump files...

I meant, of course, "how you tell unison to merge nm_dump files."

Re: syncing mail by simply syncing the filesystem

[David Bremner]

On Sat, 12 Jun 2010 15:23:45 +0200, Arvid Picciani <aep@exys.org> wrote:

> syncing maildir works.  notmuch's db resides inside your maildir, but its
> probably not safe to do while its running. And you can only have one
> place receive your mail, otherwise you loose tags.

Well, only one place where you mark things unread as well with this
approach. Otherwise updating the database in two places and attempting
to sync will lead to an unmergable conflict, won't it?

d

Re: syncing mail by simply syncing the filesystem

[Arvid Picciani]

On Sat, 12 Jun 2010 10:37:16 -0300, David Bremner <bremner@unb.ca> wrote:
> Well, only one place where you mark things unread as well with this
> approach. Otherwise updating the database in two places and attempting
> to sync will lead to an unmergable conflict, won't it?

Oh yeah, sorry. I'm assuming "unison" is similar to NFS or something.
I just mount stuff via sshfs, so i don't run into that problem at all.

Re: syncing mail by simply syncing the filesystem

[Christoph Groth]

Jesse,

thanks a lot for your ideas.

Jesse Rosenthal <jrosenthal@jhu.edu> writes:

> 1. If your other computers will have a consistent fast internet
> connection (or a sufficient connection to use IMAP) consider remote
> usage, keeping your database on one computer:

They do, most of the time.  But I really like to be able to stay
independent of any servers when I am on the road with occasional and
slow network access.

> 2. You could brute-force it. Include a nm_dump file as one of your
> unison paths, along with your Maildir, then always run:
>
> notmuch dump > nm_dump
> unison
> notmuch restore nm_dump

This might actually be the perfect solution, if notmuch is fast enough
to dump and restore the tags in a few seconds.

Actually, I am a bit confused by why many people seem to be using tags
for named searches.  IMHO tags only make sense when set by hand for some
non-searchable criteria.  For all the rest normal searches (perhaps with
saved search aliases) should be the better solution.

> 2a. Just a thought -- would merging be easier if you used git/hg instead
> of unison? Just make sure to gitignore your notmuch db.

You mean to store nm_dump?  This sounds too complicated without much
benefit.  I'm using git for things where I am interested in the history
of changes which is hardly the case for the tags.

Re: syncing mail by simply syncing the filesystem

[Jesse Rosenthal]

Hi Christoph,

On Sat, 12 Jun 2010 17:47:07 +0200, Christoph Groth <cwg@falma.de> wrote:
> > notmuch dump > nm_dump
> > unison
> > notmuch restore nm_dump
> 
> This might actually be the perfect solution, if notmuch is fast enough
> to dump and restore the tags in a few seconds.

Well, it's a bit slower than that. Restoring will be more in the
multiples of tens of seconds (though perhaps less if you're using
searches much more than tags). If you're only doing it once in a while,
it'll be okay (especially since it'll pale next to the speed of
unison-ing maildirs), but it could get to be a bit of a pain.

I was thinking about how to cut this down in a previous email exchange
on the list:

id:87aaru5yi2.fsf@jhu.edu

(http://mid.gmane.org/87aaru5yi2.fsf@jhu.edu)

The key part is that if I think restore (which is what takes more time)
leaves unmentioned messages as they are, so you can run it on a diff, so
something like:

nm dump desktop>desktop-dump; nm dump laptop>laptop-dump; 
diff laptop-dump desktop | grep "^>" | etc...

So when you restore, you'd only be restoring the changes, and it would
drop to a second or two. If you're scripting all the other stuff
together with unison, this could be part of the script as well.

Anyway, this is all rather raw -- my own use-cases haven't made it
necessary to do anything more than spout off on the mailing list about
it.

Best,
Jesse