[PATCH] Mailstore abstraction v4 - part 2 (maildir synchronization)

[PATCH] Mailstore abstraction v4 - part 2 (maildir synchronization)

[Michal Sojka]

This is the second part of mailstore abstraction patches. I do not
want this to be merged yet, but there might be some people interested
in testing this.

This patch adds a mailstore, which bi-directionally synchronizes
certain tags with maildir flags.

I use it already four weeks and it works quite well. There are the
following know bugs:

1) Viewing/storing of attachments of unread messages doesn't work. The
   reason is that when you view the message its unread tag is removed
   by elisp code. This leads to rename of the file, but Emacs still
   uses the original name to access message with the attachment.

   Workaround: close the message and open it again.

   I'm working on the solution - if the mailstore cannot open the
   message with the name passed, it tries different names with
   different maildir flags.

2) If there several messages with the same ID (e.g. one in sent folder
   and one sent back by mailing list), the flags are synchronized to
   only one of these files.

   I plan to do this:

   - When a tag is added/removed in notmuch, flags of all
     files corresponding to the message will be updated.
   - If we detect (during notmuch new) that flags of one file were
     changed by somebody else, we also change the flags for the other
     files corresponding to the message.
   - If we detect (during notmuch new) that flags of two or more files
     were changed by somebody else, we have to solve the conflict
     somehow, but I didn't invent how, yet.

The full series is available at
http://rtime.felk.cvut.cz/gitweb/notmuch.git/shortlog/refs/heads/mailstore-abstraction-v4
and can be pulled by

   git pull git://rtime.felk.cvut.cz/notmuch.git mailstore-abstraction-v4

Besides the patch sent here, there are also tests for the maildir
mailstore and a not finished implementation of the solution for 1)
above.

--Michal
   

[PATCH] Add maildir-based mailstore

[Michal Sojka]

This mailstore allows bi-directional synchronization between maildir
flags and certain tags. The flag-to-tag mapping is defined by flag2tag
array.

The synchronization works this way:

1) Whenever notmuch new is executed, the following happens:
   o New messages are tagged with inbox.
   o New messages without maildir info in the file name (typically
     files in new/) are tagged with unread.
   o For new or renamed messages with maildir info present in the
     file name, the tags defined in flag2tag are either added or
     removed depending on the flags from the file name.

2) Whenever notmuch tag (or notmuch restore) is executed, a new set of
   flags based on the tags is constructed for every message and a new
   file name is prepared based on the old file name but with the new
   flags. If the old message file was in 'new' directory then this is
   replaced with 'cur' in the new file name. If the new and old file
   names differ, the file is renamed and notmuch database is updated
   accordingly.

   The rename happens before the database is updated. In case of crash
   between rename and database update, the next run of notmuch new
   brings the database in sync with the mail store again.

This mailstore is enabled by putting the following to your
.notmuch-config:

[mailstore]
type=maildir

Signed-off-by: Michal Sojka <sojkam1@fel.cvut.cz>
---
 lib/database.cc       |    7 ++
 lib/mailstore-files.c |  210 ++++++++++++++++++++++++++++++++++++++++++++++++-
 lib/mailstore.c       |    1 +
 lib/message.cc        |   41 +++++++++-
 lib/notmuch-private.h |    4 +
 lib/notmuch.h         |    1 +
 6 files changed, 260 insertions(+), 4 deletions(-)

diff --git a/lib/database.cc b/lib/database.cc
index bd64ed3..4f3ce88 100644
--- a/lib/database.cc
+++ b/lib/database.cc
@@ -1479,6 +1479,13 @@ notmuch_database_add_message (notmuch_database_t *notmuch,
 
 	_notmuch_message_add_filename (message, filename);
 
+	/* This is a new message or it has a new filename and as such,
+	 * its tags in database either do not exists or might be out
+	 * of date. Mailstore assigns the tags later in index_new(),
+	 * but until then we should not synchronize the tags back to
+	 * the mailstore. */
+	notmuch_message_set_flag(message, NOTMUCH_MESSAGE_FLAG_TAGS_INVALID, TRUE);
+
 	/* Is this a newly created message object? */
 	if (private_status == NOTMUCH_PRIVATE_STATUS_NO_DOCUMENT_FOUND) {
 	    _notmuch_message_add_term (message, "type", "mail");
diff --git a/lib/mailstore-files.c b/lib/mailstore-files.c
index f2cb8d7..e3e346f 100644
--- a/lib/mailstore-files.c
+++ b/lib/mailstore-files.c
@@ -24,6 +24,30 @@
 #include "notmuch.h"
 #include "mailstore-private.h"
 #include <dirent.h>
+#include <stdbool.h>
+
+#define ARRAY_SIZE(arr) (sizeof (arr) / sizeof (arr[0]))
+
+struct mailstore_priv {
+    void (*tag_new)(notmuch_message_t *message, const char *filename);
+    int  (*tag_renamed)(notmuch_message_t *message, const char *filename);
+};
+
+struct maildir_flag_tag {
+    char flag;
+    const char *tag;
+    bool inverse;
+};
+
+/* ASCII ordered table of Maildir flags and assiciated tags */
+struct maildir_flag_tag flag2tag[] = {
+    { 'D', "draft",   false},
+    { 'F', "flagged", false},
+    { 'P', "passed",  false},
+    { 'R', "replied", false},
+    { 'S', "unread",  true },
+    { 'T', "delete",  false},
+};
 
 typedef struct _filename_node {
     char *filename;
@@ -69,8 +93,9 @@ _filename_list_add (_filename_list_t *list,
 }
 
 static void
-tag_inbox_and_unread (notmuch_message_t *message)
+tag_inbox_and_unread (notmuch_message_t *message, const char *filename)
 {
+    (void)filename;
     notmuch_message_add_tag (message, "inbox");
     notmuch_message_add_tag (message, "unread");
 }
@@ -116,6 +141,163 @@ _entries_resemble_maildir (struct dirent **entries, int count)
     return 0;
 }
 
+static int
+tag_from_maildir_flags (notmuch_message_t *message, const char *filename)
+{
+    const char *flags, *p;
+    char f;
+    bool valid, unread;
+    unsigned i;
+
+    flags = strstr(filename, ":2,");
+    if (!flags)
+	return -1;
+    flags += 3;
+
+    /*  Check that the letters are valid Maildir flags */
+    f = 0;
+    valid = true;
+    for (p=flags; valid && *p; p++) {
+	switch (*p) {
+	case 'P':
+	case 'R':
+	case 'S':
+	case 'T':
+	case 'D':
+	case 'F':
+	    if (*p > f) f=*p;
+	    else valid = false;
+	break;
+	default:
+	    valid = false;
+	}
+    }
+    if (!valid) {
+	fprintf (stderr, "Warning: Invalid maildir flags in filename %s\n", filename);
+	return -1;
+    }
+
+    notmuch_message_freeze(message);
+    unread = true;
+    for (i = 0; i < ARRAY_SIZE(flag2tag); i++) {
+	if ((strchr(flags, flag2tag[i].flag) != NULL) ^ flag2tag[i].inverse) {
+	    notmuch_message_add_tag (message, flag2tag[i].tag);
+	} else {
+	    notmuch_message_remove_tag (message, flag2tag[i].tag);
+	}
+    }
+    notmuch_message_thaw(message);
+
+    /* From now on, we can synchronize the tags from the database to
+     * the mailstore. */
+    notmuch_message_set_flag(message, NOTMUCH_MESSAGE_FLAG_TAGS_INVALID, FALSE);
+    return 0;
+}
+
+static void
+get_new_flags(notmuch_message_t *message, char *flags)
+{
+    notmuch_tags_t *tags;
+    const char *tag;
+    unsigned i;
+    char *p;
+
+    for (i = 0; i < ARRAY_SIZE(flag2tag); i++)
+	flags[i] = flag2tag[i].inverse ? flag2tag[i].flag : '\0';
+
+    for (tags = notmuch_message_get_tags (message);
+	 notmuch_tags_valid (tags);
+	 notmuch_tags_move_to_next (tags))
+    {
+	tag = notmuch_tags_get (tags);
+	for (i = 0; i < ARRAY_SIZE(flag2tag); i++) {
+	    if (strcmp(tag, flag2tag[i].tag) == 0)
+		flags[i] = flag2tag[i].inverse ? '\0' : flag2tag[i].flag;
+	}
+    }
+
+    p = flags;
+    for (i = 0; i < ARRAY_SIZE(flag2tag); i++) {
+	if (flags[i])
+	    *p++ = flags[i];
+    }
+    *p = '\0';
+}
+
+static char *
+get_subdir (char *filename)
+{
+    char *p, *subdir = NULL;
+
+    p = filename + strlen (filename) - 1;
+    while (p > filename + 3 && *p != '/')
+	p--;
+    if (*p == '/') {
+	subdir = p - 3;
+	if (subdir > filename && *(subdir - 1) != '/')
+	    subdir = NULL;
+    }
+    return subdir;
+}
+
+/* Store maildir-related tags as maildir flags */
+static notmuch_private_status_t
+maildir_sync_tags (notmuch_mailstore_t *mailstore,
+		   notmuch_message_t *message)
+{
+    char flags[ARRAY_SIZE(flag2tag)+1];
+    const char *filename, *p, *db_path;
+    char *filename_new, *subdir = NULL;
+    int ret;
+    char *abs1, *abs2;
+
+    (void)mailstore;
+    get_new_flags (message, flags);
+
+    filename = notmuch_message_get_filename (message);
+    /* TODO: Iterate over all file names. */
+    p = strstr(filename, ":2,");
+    if (!p)
+	p = filename + strlen(filename);
+
+    filename_new = talloc_size(message, (p-filename) + 3 + sizeof(flags));
+    if (unlikely (filename_new == NULL))
+	return NOTMUCH_STATUS_OUT_OF_MEMORY;
+    memcpy(filename_new, filename, p-filename);
+    filename_new[p-filename] = '\0';
+
+    /* If message is in new/ move it under cur/. */
+    subdir = get_subdir (filename_new);
+    if (subdir && memcmp (subdir, "new/", 4) == 0)
+	memcpy (subdir, "cur/", 4);
+
+    strcpy (filename_new+(p-filename), ":2,");
+    strcpy (filename_new+(p-filename)+3, flags);
+
+    if (strcmp (filename, filename_new) != 0) {
+	db_path = notmuch_database_get_path (mailstore->notmuch);
+	asprintf(&abs1, "%s/%s", db_path, filename);
+	asprintf(&abs2, "%s/%s", db_path, filename_new);
+	ret = rename (abs1, abs2);
+	if (ret == -1) {
+	    perror (talloc_asprintf (message, "rename of %s to %s failed", abs1, abs2));
+	    exit (1);
+	}
+	free(abs1);
+	free(abs2);
+	return _notmuch_message_rename (message, filename_new);
+	/* _notmuch_message_sync is our caller. Do not call it here. */
+    }
+    return NOTMUCH_STATUS_SUCCESS;
+}
+
+static void
+tag_inbox_and_maildir_flags (notmuch_message_t *message, const char *filename)
+{
+    notmuch_message_add_tag (message, "inbox");
+    if (tag_from_maildir_flags(message, filename) != 0)
+	notmuch_message_add_tag (message, "unread");
+}
 
 /* Examine 'path' recursively as follows:
  *
@@ -171,6 +353,7 @@ add_files_recursive (notmuch_mailstore_t *mailstore,
     struct stat st;
     notmuch_bool_t is_maildir, new_directory;
     _indexing_context_priv_t *priv = state->priv;
+    struct mailstore_priv *mailstore_priv = mailstore->priv;
     notmuch_database_t *notmuch = mailstore->notmuch;
 
     if (stat (path, &st)) {
@@ -352,11 +535,12 @@ add_files_recursive (notmuch_mailstore_t *mailstore,
 	/* success */
 	case NOTMUCH_STATUS_SUCCESS:
 	    state->added_messages++;
-	    tag_inbox_and_unread (message);
+	    mailstore_priv->tag_new (message, next);
 	    break;
 	/* Non-fatal issues (go on to next file) */
 	case NOTMUCH_STATUS_DUPLICATE_MESSAGE_ID:
-	    /* Stay silent on this one. */
+	    if (mailstore_priv->tag_renamed)
+		mailstore_priv->tag_renamed (message, next);
 	    break;
 	case NOTMUCH_STATUS_FILE_NOT_EMAIL:
 	    fprintf (stderr, "Note: Ignoring non-mail file: %s\n",
@@ -618,6 +802,10 @@ open_file(notmuch_mailstore_t *mailstore, const char *filename)
     return file;
 }
 
+struct mailstore_priv files_priv = {
+    .tag_new = tag_inbox_and_unread,
+};
+
 /* Original notmuch mail store */
 struct _notmuch_mailstore mailstore_files = {
     .type = "files",
@@ -625,4 +813,20 @@ struct _notmuch_mailstore mailstore_files = {
     .index_new = index_new,
     .sync_tags = NULL,		/* We cannot store tags in this mailstore. */
     .open_file = open_file,
+    .priv = &files_priv,
+};
+
+struct mailstore_priv maildir_priv = {
+    .tag_new = tag_inbox_and_maildir_flags,
+    .tag_renamed = tag_from_maildir_flags,
+};
+
+/* Similar to mailstore_files, but does bi-directional synchronization between certain tags and maildir flags */
+struct _notmuch_mailstore mailstore_maildir = {
+    .type = "maildir",
+    .count_files = count_files,
+    .index_new = index_new,
+    .sync_tags = maildir_sync_tags,
+    .open_file = open_file,
+    .priv = &maildir_priv,
 };
diff --git a/lib/mailstore.c b/lib/mailstore.c
index 709db72..5dbb47c 100644
--- a/lib/mailstore.c
+++ b/lib/mailstore.c
@@ -25,6 +25,7 @@
 
 static notmuch_mailstore_t *available[] = {
     &mailstore_files,
+    &mailstore_maildir,
 };
 
 notmuch_mailstore_t *
diff --git a/lib/message.cc b/lib/message.cc
index c7eff7c..292e95e 100644
--- a/lib/message.cc
+++ b/lib/message.cc
@@ -578,7 +578,8 @@ _notmuch_message_sync (notmuch_message_t *message)
     if (message->notmuch->mode == NOTMUCH_DATABASE_MODE_READ_ONLY)
 	return;
 
-    if (message->notmuch->mailstore->sync_tags) {
+    if (message->notmuch->mailstore->sync_tags &&
+	!notmuch_message_get_flag(message, NOTMUCH_MESSAGE_FLAG_TAGS_INVALID)) {
 	status = message->notmuch->mailstore->sync_tags (message->notmuch->mailstore,
 							 message);
 	if (status != NOTMUCH_PRIVATE_STATUS_SUCCESS) {
@@ -700,6 +701,44 @@ _notmuch_message_remove_term (notmuch_message_t *message,
     return NOTMUCH_PRIVATE_STATUS_SUCCESS;
 }
 
+/* Change the message filename stored in the database.
+ *
+ * This change will not be reflected in the database until the next
+ * call to _notmuch_message_sync.
+ */
+notmuch_private_status_t
+_notmuch_message_rename (notmuch_message_t *message,
+			 const char *new_filename)
+{
+    void *local = talloc_new (message);
+    char *direntry;
+    Xapian::PostingIterator i, end;
+    Xapian::Document document;
+    notmuch_private_status_t pstatus;
+    notmuch_status_t status;
+    const char *old_filename;
+
+    old_filename = notmuch_message_get_filename(message);
+    old_filename = talloc_reference(local, old_filename);
+    if (unlikely(!old_filename))
+	return NOTMUCH_PRIVATE_STATUS_OUT_OF_MEMORY;
+
+    status = _notmuch_message_add_filename (message, new_filename);
+    if (status)
+	return (notmuch_private_status_t)status;
+
+    status = _notmuch_database_filename_to_direntry (local, message->notmuch,
+						     old_filename, &direntry);
+    if (status)
+	return (notmuch_private_status_t)status;
+
+    pstatus = _notmuch_message_remove_term (message, "file-direntry", direntry);
+
+    talloc_free (local);
+
+    return pstatus;
+}
+
 notmuch_status_t
 notmuch_message_add_tag (notmuch_message_t *message, const char *tag)
 {
diff --git a/lib/notmuch-private.h b/lib/notmuch-private.h
index bab2090..c257365 100644
--- a/lib/notmuch-private.h
+++ b/lib/notmuch-private.h
@@ -245,6 +245,10 @@ notmuch_status_t
 _notmuch_message_add_filename (notmuch_message_t *message,
 			       const char *filename);
 
+notmuch_private_status_t
+_notmuch_message_rename (notmuch_message_t *message,
+			 const char *new_filename);
+
 void
 _notmuch_message_ensure_thread_id (notmuch_message_t *message);
 
diff --git a/lib/notmuch.h b/lib/notmuch.h
index 54de0bd..942eef7 100644
--- a/lib/notmuch.h
+++ b/lib/notmuch.h
@@ -769,6 +769,7 @@ notmuch_message_get_mailstore (notmuch_message_t *message);
 /* Message flags */
 typedef enum _notmuch_message_flag {
     NOTMUCH_MESSAGE_FLAG_MATCH,
+    NOTMUCH_MESSAGE_FLAG_TAGS_INVALID,
 } notmuch_message_flag_t;
 
 /* Get a value of a flag for the email corresponding to 'message'. */
-- 
1.7.0.2

Re: [PATCH] Mailstore abstraction v4 - part 2 (maildir synchronization)

[martin f krafft]

[-- Attachment #1: Type: text/plain, Size: 1009 bytes --]

also sprach Michal Sojka <sojkam1@fel.cvut.cz> [2010.04.08.1713 +0200]:
>    I'm working on the solution - if the mailstore cannot open the
>    message with the name passed, it tries different names with
>    different maildir flags.

Wouldn't it be better to postpone synchronisation of the tags until
after emacs is done with the message?

I understand this might be hard to make work with mailstore
abstraction. Wouldn't it make more sense to have emacs call 'notmuch
cat', which returns the entire message, removes the unread tag,
changes the filename, and updates the database?

The message returned by cat would be stored in a temporary file for
use by the client (emacs). And if the message was needed again, you
could just search for it again.

I dislike the idea of heuristically probing a Maildir for files.

-- 
martin | http://madduck.net/ | http://two.sentenc.es/
 
"i don't think so," said rene descartes. just then, he vanished.
 
spamtraps: madduck.bogus@madduck.net

[-- Attachment #2: Digital signature (see http://martin-krafft.net/gpg/) --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [PATCH] Mailstore abstraction v4 - part 2 (maildir synchronization)

[Michal Sojka]

On Mon, 12 Apr 2010, martin f krafft wrote:
> also sprach Michal Sojka <sojkam1@fel.cvut.cz> [2010.04.08.1713 +0200]:
> >    I'm working on the solution - if the mailstore cannot open the
> >    message with the name passed, it tries different names with
> >    different maildir flags.
> 
> Wouldn't it be better to postpone synchronisation of the tags until
> after emacs is done with the message?

Theoretically, it would be possible, but if, for some reason, the
synchronization step would not happen, then the tags in the database and
in the mailstore will be inconsistent and next run of notmuch new would
reset the tags according to the state in mailstore.

The current implementation takes tags in mailstore as authoritative and
ensures that tags in mailstore are always updated before tags in the
database.

> I understand this might be hard to make work with mailstore
> abstraction. Wouldn't it make more sense to have emacs call 'notmuch
> cat', which returns the entire message, removes the unread tag,
> changes the filename, and updates the database?

I do not like the fact that cat would do two things - cat and tag. And
then, 'unread' tag is not the only one which can be changed.
 
> The message returned by cat would be stored in a temporary file for
> use by the client (emacs). And if the message was needed again, you
> could just search for it again.
> 
> I dislike the idea of heuristically probing a Maildir for files.

Well, I do not plan to use wired heuristics. At the end there will be
readdir() to traverse the cur/ directory to find the file with the same
part before flags. Since the S flag will probably be the most frequent
change, I may add one single test for added S flag before trying more
expensive readdir().

-Michal

Re: [PATCH] Mailstore abstraction v4 - part 2 (maildir synchronization)

[martin f krafft]

[-- Attachment #1: Type: text/plain, Size: 3146 bytes --]

also sprach Michal Sojka <sojkam1@fel.cvut.cz> [2010.04.12.1347 +0200]:
> > Wouldn't it be better to postpone synchronisation of the tags
> > until after emacs is done with the message?
> 
> Theoretically, it would be possible, but if, for some reason, the
> synchronization step would not happen, then the tags in the
> database and in the mailstore will be inconsistent and next run of
> notmuch new would reset the tags according to the state in
> mailstore.

Well, sure. But then again, notmuch (nor IMAP or Maildir) isn't
transactional anyway. There are many other ways in which the
database and store can get out of sync. And you are about to add
another redundancy.

> The current implementation takes tags in mailstore as
> authoritative and ensures that tags in mailstore are always
> updated before tags in the database.

So why store them in the database at all?

> > I understand this might be hard to make work with mailstore
> > abstraction. Wouldn't it make more sense to have emacs call
> > 'notmuch cat', which returns the entire message, removes the
> > unread tag, changes the filename, and updates the database?
> 
> I do not like the fact that cat would do two things - cat and tag.
> And then, 'unread' tag is not the only one which can be changed.

I wouldn't want an unread tag in the first place, especially not
with Maildir semantics. In this case, what should really happen is:

  1. cat feeds a message to client
  2. client instructs notmuch to update tags
     - some tags require changes in the database
     - others require filename changes, which must be completed in
       unison with a database update so the new filename is stored.
  3. user asks to see attachment, which the client can fulfill using
     either a cached copy from (1.) in a tempfile, or by simply
     asking for the message again, via notmuch search.

> > The message returned by cat would be stored in a temporary file
> > for use by the client (emacs). And if the message was needed
> > again, you could just search for it again.
> > 
> > I dislike the idea of heuristically probing a Maildir for files.
> 
> Well, I do not plan to use wired heuristics. At the end there will
> be readdir() to traverse the cur/ directory to find the file with
> the same part before flags. Since the S flag will probably be the
> most frequent change, I may add one single test for added S flag
> before trying more expensive readdir().

What is the point of storing these tags in the Maildir anyway? If
you want to make this information (e.g. new, seen, unread) available
to MUAs accessing Maildir directly, keep in mind that the database
and mailstore will very quickly grow inconsistent until the next
notmuch-new run, e.g. as messages are moved, or flags ('F') are
added in a way that the notmuch database is not updated.

-- 
martin | http://madduck.net/ | http://two.sentenc.es/
 
"friendships last when each friend thinks he has
 a slight superiority over the other."
                                                   -- honoré de balzac
 
spamtraps: madduck.bogus@madduck.net

[-- Attachment #2: Digital signature (see http://martin-krafft.net/gpg/) --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [PATCH] Mailstore abstraction v4 - part 2 (maildir synchronization)

[Michal Sojka]

On Mon, 12 Apr 2010, martin f krafft wrote:
> also sprach Michal Sojka <sojkam1@fel.cvut.cz> [2010.04.12.1347 +0200]:
> > > Wouldn't it be better to postpone synchronisation of the tags
> > > until after emacs is done with the message?
> > 
> > Theoretically, it would be possible, but if, for some reason, the
> > synchronization step would not happen, then the tags in the
> > database and in the mailstore will be inconsistent and next run of
> > notmuch new would reset the tags according to the state in
> > mailstore.
> 
> Well, sure. But then again, notmuch (nor IMAP or Maildir) isn't
> transactional anyway. There are many other ways in which the
> database and store can get out of sync. And you are about to add
> another redundancy.
> 
> > The current implementation takes tags in mailstore as
> > authoritative and ensures that tags in mailstore are always
> > updated before tags in the database.
> 
> So why store them in the database at all?

Because we want to use them in searches.

> > > I understand this might be hard to make work with mailstore
> > > abstraction. Wouldn't it make more sense to have emacs call
> > > 'notmuch cat', which returns the entire message, removes the
> > > unread tag, changes the filename, and updates the database?
> > 
> > I do not like the fact that cat would do two things - cat and tag.
> > And then, 'unread' tag is not the only one which can be changed.
> 
> I wouldn't want an unread tag in the first place, especially not
> with Maildir semantics. In this case, what should really happen is:

The bellow sounds reasonable. I only have a few notes.
> 
>   1. cat feeds a message to client

I'd not use cat here, I'd stay with show which transfers only text/*
parts. Recently I was surprised how fast notmuch is when client is
running locally and notmuch is invoked remotely over ssh (even over slow
connection).

>   2. client instructs notmuch to update tags
>      - some tags require changes in the database
>      - others require filename changes, which must be completed in
>        unison with a database update so the new filename is stored.
>   3. user asks to see attachment, which the client can fulfill using
>      either a cached copy from (1.) in a tempfile, or by simply
>      asking for the message again, via notmuch search.

The latter option sounds well for me, but my elisp skills are not
sufficient for implementing this. Maybe, dme will do it in his new
client. On the other side, I'm not sure whether it has sense search for
the message again (in step 3) if you are not using maildir mailstore.
One possibility would be, when using maildir mailstore, to cheat the
client and use message-id instead of filename. Then, the client would
use message-id in cat command and implementation for open_file() in
maildir mailstore will search for the real file name.

What do you think?

> > > The message returned by cat would be stored in a temporary file
> > > for use by the client (emacs). And if the message was needed
> > > again, you could just search for it again.
> > > 
> > > I dislike the idea of heuristically probing a Maildir for files.
> > 
> > Well, I do not plan to use wired heuristics. At the end there will
> > be readdir() to traverse the cur/ directory to find the file with
> > the same part before flags. Since the S flag will probably be the
> > most frequent change, I may add one single test for added S flag
> > before trying more expensive readdir().
> 
> What is the point of storing these tags in the Maildir anyway? 

My point is to synchronize the status of reading of my emails with IMAP
server (through offlineimap). I use notmuch as my primary emails client,
but my mobile phone checks for new emails on the IMAP server and from
time to time I need to use other IMAP clients (on other computers).

I have offlineimap running in background so that whenever I read a
message in notmuch, after some short time, this information is
propagated back to IMAP server and when I go home from work, my mobile
phone shows zero unread mails.

> If you want to make this information (e.g. new, seen, unread)
> available to MUAs accessing Maildir directly, keep in mind that the
> database and mailstore will very quickly grow inconsistent until the
> next notmuch-new run, e.g. as messages are moved, or flags ('F') are
> added in a way that the notmuch database is not updated.

It depends on how you define quickly and if you run notmuch new also
very quickly, there is almost no inconsistency :))

-Michal