notmuch and imap [musing, no code :)]

notmuch and imap [musing, no code :)]

[David Bremner]

Recent discussions on IRC (I forget with whom, sorry), brought me back
to thinking about syncing notmuch with imap.  In addition to the flags
\Seen, \Answered, \Draft, \Deleted, and \Flagged, imap servers can
optionally support user defined keywords (i.e. tags). At least courier
and dovecot do.  These keywords are imap "atoms", which, without tracing
though the BNF [1] completely look like they can can have (ascii)
alphanumeric, and punctuation other than brackets, quotes and "%", "*".

So the mapping is relatively nice between notmuch tags and imap
keywords.

One idea I had was to extend some imap syncing program to sync to
notmuch. mbsync (confusingly also named isync) is a C based one. It is
indeed relatively easy to add a new backend; I made a new "notmuch"
driver for mbsync in an hour or so that is actually just a maildir
driver.  I am a little discouraged by some of mbsync code (there are
lots of places with a buffer hard-coded to size 16, with a comment to
change that later when keyword support is added), but in principle, I
think this could work.  A more fundamental issue is that mbsync, like
most similar programs uses an index file to keep track of the sync
state, and it seems somehow wrong (or at least fragile) to keep a second
database with essentially the same information in it.

To write a custom sync program would require some imap code; this is
somewhere between trivial and not; mbsync's imap driver is about 1900
lines of C.  There is the uw c-client library; I remember some security
issues but that was more than a decade ago.  There are some newer
libraries like tinymail and libetpan, but they seem to have a whole
bunch of stuff not needed for notmuch.

d


[1]: http://www.faqs.org/rfcs/rfc3501.html

Re: notmuch and imap [musing, no code :)]

[Alec Berryman]

David Bremner on 2009-12-15 08:05:13 -0400:

> Recent discussions on IRC (I forget with whom, sorry), brought me back
> to thinking about syncing notmuch with imap.  In addition to the flags
> \Seen, \Answered, \Draft, \Deleted, and \Flagged, imap servers can
> optionally support user defined keywords (i.e. tags). At least courier
> and dovecot do.  These keywords are imap "atoms", which, without tracing
> though the BNF [1] completely look like they can can have (ascii)
> alphanumeric, and punctuation other than brackets, quotes and "%", "*".

I am also very interested in syncing tags between computers.  I started
implementing storage of tags in message headers.  It's has been done
before - several mutt extensions and systems use X-Label.  This approach
offers several advantages to notmuch dump/restore and IMAP flags:

  - compatability: most mail clients can search on headers, so even if
    you're not using notmuch full-time (squirrelmail? phone?), you can
    get some benefit from it

  - works with offlineimap without further effort: no new transport
    mechanism required

  - a migration path: text-based mail sorting tools like procmail can
    easily set headers

  - backups are easy: you can't miss backing up your tags because
    they're in the messages

There are security concerns (need to strip incoming messages of tags so
no one tags your mail for you), privacy concerns (if you forward the
entire message as an attachment, may want to strip tags), and space
concerns (how many flags?), but I think they can be worked around.

I haven't gotten very far with my implementation due to time
constraints.  It reads tags fine, but I haven't implemented writing,
which is the involved part.  I hope to get to this between Christmas and
New Years, but who knows.

Re: notmuch and imap [musing, no code :)]

[David Bremner]

On Tue, 15 Dec 2009 19:54:11 -0500, Alec Berryman <alec@thened.net> wrote:

> There are security concerns (need to strip incoming messages of tags so
> no one tags your mail for you), privacy concerns (if you forward the
> entire message as an attachment, may want to strip tags), and space
> concerns (how many flags?), but I think they can be worked around.

I agree that the labels-in-headers approach has some nice advantages.  I
haven't thought through merging of tag lists, but maybe that is no worse
than other approaches.  One thing that worries me a bit is that notmuch
updates tags often, and each of these updates would require rewriting
the message, at least in the obvious implementation. I'd hate to finally
have Xapian issue 350 fixed, only to take an equivalent hit by rewriting
the message. Maybe it is not that slow.

d

Re: notmuch and imap [musing, no code :)]

[Marten Veldthuis]

On Wed, 16 Dec 2009 09:18:16 -0400, David Bremner <david@tethera.net> wrote:
> I agree that the labels-in-headers approach has some nice advantages.  I
> haven't thought through merging of tag lists, but maybe that is no worse
> than other approaches.  One thing that worries me a bit is that notmuch
> updates tags often, and each of these updates would require rewriting
> the message, at least in the obvious implementation. I'd hate to finally
> have Xapian issue 350 fixed, only to take an equivalent hit by rewriting
> the message. Maybe it is not that slow.

If it is, maybe we could do the sync-back by renaming notmuch-new to
notmuch-sync, and doing it there.

-- 
- Marten

Re: notmuch and imap [musing, no code :)]

[martin f krafft]

[-- Attachment #1: Type: text/plain, Size: 926 bytes --]

also sprach David Bremner <david@tethera.net> [2009.12.17.0218 +1300]:
> I agree that the labels-in-headers approach has some nice
> advantages.  I haven't thought through merging of tag lists, but
> maybe that is no worse than other approaches.  One thing that
> worries me a bit is that notmuch updates tags often, and each of
> these updates would require rewriting the message, at least in the
> obvious implementation.

If we separated implicit and explicit tags, then that issue would
not exist, and from all I can tell, only the privacy issues, and the
risk of losing mail when files are rewritten persist.

-- 
martin | http://madduck.net/ | http://two.sentenc.es/
 
DISCLAIMER: this entire message is privileged communication, intended
for the sole use of its recipients only. If you read it even though
you know you aren't supposed to, you're a poopy-head.
 
spamtraps: madduck.bogus@madduck.net

[-- Attachment #2: Digital signature (see http://martin-krafft.net/gpg/) --]
[-- Type: application/pgp-signature, Size: 198 bytes --]