[PATCH 1/4] lib: Return added message even if it already was in the database

[PATCH 1/4] lib: Return added message even if it already was in the database

[Michal Sojka]

---
 lib/database.cc |    3 ++-
 lib/notmuch.h   |    3 ++-
 2 files changed, 4 insertions(+), 2 deletions(-)

diff --git a/lib/database.cc b/lib/database.cc
index 6afc8d9..694b7ec 100644
--- a/lib/database.cc
+++ b/lib/database.cc
@@ -1613,7 +1613,8 @@ notmuch_database_add_message (notmuch_database_t *notmuch,
 
   DONE:
     if (message) {
-	if (ret == NOTMUCH_STATUS_SUCCESS && message_ret)
+	if ((ret == NOTMUCH_STATUS_SUCCESS ||
+	     ret == NOTMUCH_STATUS_DUPLICATE_MESSAGE_ID) && message_ret)
 	    *message_ret = message;
 	else
 	    notmuch_message_destroy (message);
diff --git a/lib/notmuch.h b/lib/notmuch.h
index 505ad19..0ba1416 100644
--- a/lib/notmuch.h
+++ b/lib/notmuch.h
@@ -238,7 +238,8 @@ notmuch_database_get_directory (notmuch_database_t *database,
  * notmuch database will reference the filename, and will not copy the
  * entire contents of the file.
  *
- * If 'message' is not NULL, then, on successful return '*message'
+ * If 'message' is not NULL, then, on successful return
+ * (NOTMUCH_STATUS_SUCCESS or NOTMUCH_STATUS_DUPLICATE_MESSAGE_ID) '*message'
  * will be initialized to a message object that can be used for things
  * such as adding tags to the just-added message. The user should call
  * notmuch_message_destroy when done with the message. On any failure
-- 
1.7.1

[PATCH 0/4] Maildir synchronization v2

[Michal Sojka]

Hi,

this is an updated version of patches sent in
id:1273580061-22580-1-git-send-email-sojkam1@fel.cvut.cz. Only the last
patch (tests) was actually updated to work with new test suite. These
patches has already been tested by several people and I think they are
ready to be merged. They can be pulled by:

git pull git://rtime.felk.cvut.cz/notmuch maildir-sync-v2

These patches implement synchronization between maildir flags and
notmuch tags. The synchronization can be configured to not happen at
all (default), to set/unset tags when importing new (or new and
renamed) messages and to happen in both directions - set/unset tags
during importing and change maildir flags during tagging.

THERE IS CURRENTLY ONE KNOWN ISSUE: Viewing/storing of attachments of
unread messages doesn't work. The reason is that when you view the
message its unread tag is removed which causes the file to be renamed,
but Emacs still uses the original name to access the attachment. You can
workaround this by closing the message and opening it again. This issue
will be fixed after we (I) implement "notmuch cat" command. With this
command, emacs would not access the messages by the file name, but by
running notmuch cat id:<message-id> which will always give the correct
content.

Michal Sojka (4):
  lib: Return added message even if it already was in the database
  Maildir synchronization
  Make maildir synchronization configurable
  Tests for maildir synchronization

 lib/database-private.h |    2 +-
 lib/database.cc        |   19 ++++-
 lib/message.cc         |  226 ++++++++++++++++++++++++++++++++++++++++++++++++
 lib/notmuch-private.h  |    4 +
 lib/notmuch.h          |   29 ++++++-
 notmuch-client.h       |    7 ++
 notmuch-config.c       |   48 ++++++++++
 notmuch-new.c          |    7 ++-
 notmuch-restore.c      |    2 +
 notmuch-setup.c        |   17 ++++
 notmuch-tag.c          |    2 +
 test/maildir-sync      |  216 +++++++++++++++++++++++++++++++++++++++++++++
 test/notmuch-test      |    2 +-
 test/test-lib.sh       |   14 +++-
 14 files changed, 588 insertions(+), 7 deletions(-)
 create mode 100755 test/maildir-sync

[PATCH 1/4] lib: Return added message even if it already was in the database

[Michal Sojka]

---
 lib/database.cc |    3 ++-
 lib/notmuch.h   |    3 ++-
 2 files changed, 4 insertions(+), 2 deletions(-)

diff --git a/lib/database.cc b/lib/database.cc
index e4ac970..9a4f715 100644
--- a/lib/database.cc
+++ b/lib/database.cc
@@ -1671,7 +1671,8 @@ notmuch_database_add_message (notmuch_database_t *notmuch,
 
   DONE:
     if (message) {
-	if (ret == NOTMUCH_STATUS_SUCCESS && message_ret)
+	if ((ret == NOTMUCH_STATUS_SUCCESS ||
+	     ret == NOTMUCH_STATUS_DUPLICATE_MESSAGE_ID) && message_ret)
 	    *message_ret = message;
 	else
 	    notmuch_message_destroy (message);
diff --git a/lib/notmuch.h b/lib/notmuch.h
index 505ad19..0ba1416 100644
--- a/lib/notmuch.h
+++ b/lib/notmuch.h
@@ -238,7 +238,8 @@ notmuch_database_get_directory (notmuch_database_t *database,
  * notmuch database will reference the filename, and will not copy the
  * entire contents of the file.
  *
- * If 'message' is not NULL, then, on successful return '*message'
+ * If 'message' is not NULL, then, on successful return
+ * (NOTMUCH_STATUS_SUCCESS or NOTMUCH_STATUS_DUPLICATE_MESSAGE_ID) '*message'
  * will be initialized to a message object that can be used for things
  * such as adding tags to the just-added message. The user should call
  * notmuch_message_destroy when done with the message. On any failure
-- 
1.7.1.3.g75e44

[PATCH 2/4] Maildir synchronization

[Michal Sojka]

This patch allows bi-directional synchronization between maildir
flags and certain tags. The flag-to-tag mapping is defined by flag2tag
array.

The synchronization works this way:

1) Whenever notmuch new is executed, the following happens:
   o New messages are tagged with configured new_tags.
   o For new or renamed messages with maildir info present in the file
     name, the tags defined in flag2tag are either added or removed
     depending on the flags from the file name.

2) Whenever notmuch tag (or notmuch restore) is executed, a new set of
   flags based on the tags is constructed for every message and a new
   file name is prepared based on the old file name but with the new
   flags. If the flags differs and the old message was in 'new'
   directory then this is replaced with 'cur' in the new file name. If
   the new and old file names differ, the file is renamed and notmuch
   database is updated accordingly.

   The rename happens before the database is updated. In case of crash
   between rename and database update, the next run of notmuch new
   brings the database in sync with the mail store again.

There is currenlty one known issue: Viewing/storing of attachments of
unread messages doesn't work. The reason is that when you view the
message its unread tag is removed which leads to rename of the file,
but Emacs still uses the original name to access the attachment.

Workaround: close the message and open it again.
---
 lib/database.cc       |    7 ++
 lib/message.cc        |  226 +++++++++++++++++++++++++++++++++++++++++++++++++
 lib/notmuch-private.h |    4 +
 lib/notmuch.h         |    7 ++
 notmuch-new.c         |    3 +-
 5 files changed, 246 insertions(+), 1 deletions(-)

diff --git a/lib/database.cc b/lib/database.cc
index 9a4f715..9652013 100644
--- a/lib/database.cc
+++ b/lib/database.cc
@@ -1643,6 +1643,13 @@ notmuch_database_add_message (notmuch_database_t *notmuch,
 
 	_notmuch_message_add_filename (message, filename);
 
+	/* This is a new message or it has a new filename and as such,
+	 * its tags in database either do not exists or might be out
+	 * of date. We assign the tags later in notmuch new, but until
+	 * then we should not synchronize the tags back to the maildir
+	 * flags (if notmuch is configured to do so). */
+	notmuch_message_set_flag(message, NOTMUCH_MESSAGE_FLAG_TAGS_INVALID, TRUE);
+
 	/* Is this a newly created message object? */
 	if (private_status == NOTMUCH_PRIVATE_STATUS_NO_DOCUMENT_FOUND) {
 	    _notmuch_message_add_term (message, "type", "mail");
diff --git a/lib/message.cc b/lib/message.cc
index 71f5619..96c402c 100644
--- a/lib/message.cc
+++ b/lib/message.cc
@@ -43,6 +43,24 @@ struct _notmuch_message {
     Xapian::Document doc;
 };
 
+#define ARRAY_SIZE(arr) (sizeof (arr) / sizeof (arr[0]))
+
+struct maildir_flag_tag {
+    char flag;
+    const char *tag;
+    bool inverse;
+};
+
+/* ASCII ordered table of Maildir flags and associated tags */
+struct maildir_flag_tag flag2tag[] = {
+    { 'D', "draft",   false},
+    { 'F', "flagged", false},
+    { 'P', "passed",  false},
+    { 'R', "replied", false},
+    { 'S', "unread",  true },
+    { 'T', "delete",  false},
+};
+
 /* We end up having to call the destructor explicitly because we had
  * to use "placement new" in order to initialize C++ objects within a
  * block that we allocated with talloc. So C++ is making talloc
@@ -595,15 +613,29 @@ _notmuch_message_set_date (notmuch_message_t *message,
 			    Xapian::sortable_serialise (time_value));
 }
 
+static notmuch_private_status_t
+_notmuch_message_tags_to_maildir (notmuch_message_t *message);
+
 /* Synchronize changes made to message->doc out into the database. */
 void
 _notmuch_message_sync (notmuch_message_t *message)
 {
     Xapian::WritableDatabase *db;
+    notmuch_private_status_t status;
 
     if (message->notmuch->mode == NOTMUCH_DATABASE_MODE_READ_ONLY)
 	return;
 
+    if (// todo_sync_enabled &&
+	!notmuch_message_get_flag(message, NOTMUCH_MESSAGE_FLAG_TAGS_INVALID)) {
+ 	status = _notmuch_message_tags_to_maildir (message);
+	if (status != NOTMUCH_PRIVATE_STATUS_SUCCESS) {
+	    fprintf (stderr, "Error: Cannot sync tags to maildir (%s)\n",
+		     notmuch_status_to_string ((notmuch_status_t)status));
+	    /* Exit to avoid unsynchronized mailstore. */
+	    exit(1);
+	}
+    }
     db = static_cast <Xapian::WritableDatabase *> (message->notmuch->xapian_db);
     db->replace_document (message->doc_id, message->doc);
 }
@@ -715,6 +747,44 @@ _notmuch_message_remove_term (notmuch_message_t *message,
     return NOTMUCH_PRIVATE_STATUS_SUCCESS;
 }
 
+/* Change the message filename stored in the database.
+ *
+ * This change will not be reflected in the database until the next
+ * call to _notmuch_message_sync.
+ */
+notmuch_private_status_t
+_notmuch_message_rename (notmuch_message_t *message,
+			 const char *new_filename)
+{
+    void *local = talloc_new (message);
+    char *direntry;
+    Xapian::PostingIterator i, end;
+    Xapian::Document document;
+    notmuch_private_status_t pstatus;
+    notmuch_status_t status;
+    const char *old_filename;
+
+    old_filename = notmuch_message_get_filename(message);
+    old_filename = talloc_reference(local, old_filename);
+    if (unlikely(!old_filename))
+	return NOTMUCH_PRIVATE_STATUS_OUT_OF_MEMORY;
+
+    status = _notmuch_message_add_filename (message, new_filename);
+    if (status)
+	return (notmuch_private_status_t)status;
+
+    status = _notmuch_database_filename_to_direntry (local, message->notmuch,
+						     old_filename, &direntry);
+    if (status)
+	return (notmuch_private_status_t)status;
+
+    pstatus = _notmuch_message_remove_term (message, "file-direntry", direntry);
+
+    talloc_free (local);
+
+    return pstatus;
+}
+
 notmuch_status_t
 notmuch_message_add_tag (notmuch_message_t *message, const char *tag)
 {
@@ -772,6 +842,162 @@ notmuch_message_remove_tag (notmuch_message_t *message, const char *tag)
 }
 
 notmuch_status_t
+notmuch_message_maildir_to_tags (notmuch_message_t *message, const char *filename)
+{
+    const char *flags, *p;
+    char f;
+    bool valid, unread;
+    unsigned i;
+    notmuch_status_t status;
+
+    flags = strstr (filename, ":2,");
+    if (!flags)
+	return NOTMUCH_STATUS_FILE_NOT_EMAIL;
+    flags += 3;
+
+    /*  Check that the letters are valid Maildir flags */
+    f = 0;
+    valid = true;
+    for (p=flags; valid && *p; p++) {
+	switch (*p) {
+	case 'P':
+	case 'R':
+	case 'S':
+	case 'T':
+	case 'D':
+	case 'F':
+	    if (*p > f) f=*p;
+	    else valid = false;
+	break;
+	default:
+	    valid = false;
+	}
+    }
+    if (!valid) {
+	fprintf (stderr, "Warning: Invalid maildir flags in filename %s\n", filename);
+	return NOTMUCH_STATUS_FILE_NOT_EMAIL;
+    }
+
+    status = notmuch_message_freeze (message);
+    if (status)
+	return status;
+    unread = true;
+    for (i = 0; i < ARRAY_SIZE(flag2tag); i++) {
+	if ((strchr (flags, flag2tag[i].flag) != NULL) ^ flag2tag[i].inverse) {
+	    status = notmuch_message_add_tag (message, flag2tag[i].tag);
+	} else {
+	    status = notmuch_message_remove_tag (message, flag2tag[i].tag);
+	}
+	if (status)
+	    return status;
+    }
+    status = notmuch_message_thaw (message);
+
+    /* From now on, we can synchronize the tags from the database to
+     * the mailstore. */
+    notmuch_message_set_flag (message, NOTMUCH_MESSAGE_FLAG_TAGS_INVALID, FALSE);
+    return status;
+}
+
+static void
+maildir_get_new_flags(notmuch_message_t *message, char *flags)
+{
+    notmuch_tags_t *tags;
+    const char *tag;
+    unsigned i;
+    char *p;
+
+    for (i = 0; i < ARRAY_SIZE(flag2tag); i++)
+	flags[i] = flag2tag[i].inverse ? flag2tag[i].flag : '\0';
+
+    for (tags = notmuch_message_get_tags (message);
+	 notmuch_tags_valid (tags);
+	 notmuch_tags_move_to_next (tags))
+    {
+	tag = notmuch_tags_get (tags);
+	for (i = 0; i < ARRAY_SIZE(flag2tag); i++) {
+	    if (strcmp(tag, flag2tag[i].tag) == 0)
+		flags[i] = flag2tag[i].inverse ? '\0' : flag2tag[i].flag;
+	}
+    }
+
+    p = flags;
+    for (i = 0; i < ARRAY_SIZE(flag2tag); i++) {
+	if (flags[i])
+	    *p++ = flags[i];
+    }
+    *p = '\0';
+}
+
+static char *
+maildir_get_subdir (char *filename)
+{
+    char *p, *subdir = NULL;
+
+    p = filename + strlen (filename) - 1;
+    while (p > filename + 3 && *p != '/')
+	p--;
+    if (*p == '/') {
+	subdir = p - 3;
+	if (subdir > filename && *(subdir - 1) != '/')
+	    subdir = NULL;
+    }
+    return subdir;
+}
+
+/* Rename the message file so that maildir flags corresponds to the
+ * tags and, if aplicable, move the message from new/ to cur/. */
+static notmuch_private_status_t
+_notmuch_message_tags_to_maildir (notmuch_message_t *message)
+{
+    char flags[ARRAY_SIZE(flag2tag)+1];
+    const char *filename, *p;
+    char *filename_new, *subdir = NULL;
+    int ret;
+
+    maildir_get_new_flags (message, flags);
+
+    filename = notmuch_message_get_filename (message);
+    /* TODO: Iterate over all file names. */
+    p = strstr(filename, ":2,");
+    if ((p && strcmp (p+3, flags) == 0) ||
+	(!p && flags[0] == '\0')) {
+	// Return if flags are not to be changed - this suppresses
+	// moving the message from new/ to cur/ during initial
+	// tagging.
+	return NOTMUCH_PRIVATE_STATUS_SUCCESS;
+    }
+    if (!p)
+	p = filename + strlen(filename);
+
+    filename_new = (char*)talloc_size(message, (p-filename) + 3 + sizeof(flags));
+    if (unlikely (filename_new == NULL))
+	return NOTMUCH_PRIVATE_STATUS_OUT_OF_MEMORY;
+    memcpy(filename_new, filename, p-filename);
+    filename_new[p-filename] = '\0';
+
+    /* If message is in new/ move it under cur/. */
+    subdir = maildir_get_subdir (filename_new);
+    if (subdir && memcmp (subdir, "new/", 4) == 0)
+	memcpy (subdir, "cur/", 4);
+
+    strcpy (filename_new+(p-filename), ":2,");
+    strcpy (filename_new+(p-filename)+3, flags);
+
+    if (strcmp (filename, filename_new) != 0) {
+	ret = rename (filename, filename_new);
+	if (ret == -1) {
+	    perror (talloc_asprintf (message, "rename of %s to %s failed",
+				     filename, filename_new));
+	    exit (1);
+	}
+	return _notmuch_message_rename (message, filename_new);
+	/* _notmuch_message_sync is our caller. Do not call it here. */
+    }
+    return NOTMUCH_PRIVATE_STATUS_SUCCESS;
+}
+
+notmuch_status_t
 notmuch_message_remove_all_tags (notmuch_message_t *message)
 {
     notmuch_private_status_t private_status;
diff --git a/lib/notmuch-private.h b/lib/notmuch-private.h
index 5a0cf92..1dc0a20 100644
--- a/lib/notmuch-private.h
+++ b/lib/notmuch-private.h
@@ -259,6 +259,10 @@ notmuch_status_t
 _notmuch_message_add_filename (notmuch_message_t *message,
 			       const char *filename);
 
+notmuch_private_status_t
+_notmuch_message_rename (notmuch_message_t *message,
+			 const char *new_filename);
+
 void
 _notmuch_message_ensure_thread_id (notmuch_message_t *message);
 
diff --git a/lib/notmuch.h b/lib/notmuch.h
index 0ba1416..408d633 100644
--- a/lib/notmuch.h
+++ b/lib/notmuch.h
@@ -770,6 +770,7 @@ notmuch_message_get_filename (notmuch_message_t *message);
 /* Message flags */
 typedef enum _notmuch_message_flag {
     NOTMUCH_MESSAGE_FLAG_MATCH,
+    NOTMUCH_MESSAGE_FLAG_TAGS_INVALID,
 } notmuch_message_flag_t;
 
 /* Get a value of a flag for the email corresponding to 'message'. */
@@ -886,6 +887,12 @@ notmuch_message_remove_tag (notmuch_message_t *message, const char *tag);
 notmuch_status_t
 notmuch_message_remove_all_tags (notmuch_message_t *message);
 
+/* Add or remove tags based on the maildir flags in the file name.
+ */
+notmuch_status_t
+notmuch_message_maildir_to_tags (notmuch_message_t *message,
+				 const char *filename);
+
 /* Freeze the current state of 'message' within the database.
  *
  * This means that changes to the message state, (via
diff --git a/notmuch-new.c b/notmuch-new.c
index 8818728..ed3f944 100644
--- a/notmuch-new.c
+++ b/notmuch-new.c
@@ -410,10 +410,11 @@ add_files_recursive (notmuch_database_t *notmuch,
 	    state->added_messages++;
 	    for (tag=state->new_tags; *tag != NULL; tag++)
 	        notmuch_message_add_tag (message, *tag);
+	    notmuch_message_maildir_to_tags (message, next);
 	    break;
 	/* Non-fatal issues (go on to next file) */
 	case NOTMUCH_STATUS_DUPLICATE_MESSAGE_ID:
-	    /* Stay silent on this one. */
+	    notmuch_message_maildir_to_tags (message, next);
 	    break;
 	case NOTMUCH_STATUS_FILE_NOT_EMAIL:
 	    fprintf (stderr, "Note: Ignoring non-mail file: %s\n",
-- 
1.7.1.3.g75e44

[PATCH 3/4] Make maildir synchronization configurable

[Michal Sojka]

This adds group [maildir] and key 'sync_level' to the configuration file.
The value of sync_level is used to control how the synchronization happens.
The default value is no synchronization.
---
 lib/database-private.h |    2 +-
 lib/database.cc        |    9 +++++++++
 lib/message.cc         |    2 +-
 lib/notmuch.h          |   19 +++++++++++++++++++
 notmuch-client.h       |    7 +++++++
 notmuch-config.c       |   48 ++++++++++++++++++++++++++++++++++++++++++++++++
 notmuch-new.c          |    8 ++++++--
 notmuch-restore.c      |    2 ++
 notmuch-setup.c        |   17 +++++++++++++++++
 notmuch-tag.c          |    2 ++
 10 files changed, 112 insertions(+), 4 deletions(-)

diff --git a/lib/database-private.h b/lib/database-private.h
index bd72f67..3c80f99 100644
--- a/lib/database-private.h
+++ b/lib/database-private.h
@@ -49,7 +49,7 @@ struct _notmuch_database {
     Xapian::QueryParser *query_parser;
     Xapian::TermGenerator *term_gen;
     Xapian::ValueRangeProcessor *value_range_processor;
-
+    enum notmuch_maildir_sync maildir_sync;
 };
 
 /* Convert tags from Xapian internal format to notmuch format.
diff --git a/lib/database.cc b/lib/database.cc
index 9652013..d6b6f47 100644
--- a/lib/database.cc
+++ b/lib/database.cc
@@ -689,6 +689,8 @@ notmuch_database_open (const char *path,
 	notmuch = NULL;
     }
 
+    notmuch_database_set_maildir_sync (notmuch, NOTMUCH_MAILDIR_SYNC_NONE);
+    
   DONE:
     if (notmuch_path)
 	free (notmuch_path);
@@ -718,6 +720,13 @@ notmuch_database_close (notmuch_database_t *notmuch)
     talloc_free (notmuch);
 }
 
+void
+notmuch_database_set_maildir_sync (notmuch_database_t *database,
+				   enum notmuch_maildir_sync maildir_sync)
+{
+    database->maildir_sync = maildir_sync;
+}
+
 const char *
 notmuch_database_get_path (notmuch_database_t *notmuch)
 {
diff --git a/lib/message.cc b/lib/message.cc
index 96c402c..1b945c6 100644
--- a/lib/message.cc
+++ b/lib/message.cc
@@ -626,7 +626,7 @@ _notmuch_message_sync (notmuch_message_t *message)
     if (message->notmuch->mode == NOTMUCH_DATABASE_MODE_READ_ONLY)
 	return;
 
-    if (// todo_sync_enabled &&
+    if (message->notmuch->maildir_sync == NOTMUCH_MAILDIR_SYNC_NEW_RENAMED_TAGGED &&
 	!notmuch_message_get_flag(message, NOTMUCH_MESSAGE_FLAG_TAGS_INVALID)) {
  	status = _notmuch_message_tags_to_maildir (message);
 	if (status != NOTMUCH_PRIVATE_STATUS_SUCCESS) {
diff --git a/lib/notmuch.h b/lib/notmuch.h
index 408d633..34b3bcf 100644
--- a/lib/notmuch.h
+++ b/lib/notmuch.h
@@ -108,6 +108,19 @@ typedef enum _notmuch_status {
 const char *
 notmuch_status_to_string (notmuch_status_t status);
 
+/* Level of synchronization between notmuch tags and maildir flags. */
+enum notmuch_maildir_sync {
+    NOTMUCH_MAILDIR_SYNC_INVALID = 0,
+    /* No synchronization */
+    NOTMUCH_MAILDIR_SYNC_NONE,
+    /* Tag new messages accoring to maildir flags */
+    NOTMUCH_MAILDIR_SYNC_NEW,
+    /* The above + update tags for renamed messages */
+    NOTMUCH_MAILDIR_SYNC_NEW_RENAMED,
+    /* The above + update flags when tags are added/removed. */
+    NOTMUCH_MAILDIR_SYNC_NEW_RENAMED_TAGGED,
+};
+
 /* Various opaque data types. For each notmuch_<foo>_t see the various
  * notmuch_<foo> functions below. */
 typedef struct _notmuch_database notmuch_database_t;
@@ -176,6 +189,12 @@ notmuch_database_open (const char *path,
 void
 notmuch_database_close (notmuch_database_t *database);
 
+/* Sets the level of synchronization between maildir flags and notmuch
+ * tags. */
+void
+notmuch_database_set_maildir_sync (notmuch_database_t *database,
+				   enum notmuch_maildir_sync maildir_sync);
+
 /* Return the database path of the given database.
  *
  * The return value is a string owned by notmuch so should not be
diff --git a/notmuch-client.h b/notmuch-client.h
index 20be43b..50be95c 100644
--- a/notmuch-client.h
+++ b/notmuch-client.h
@@ -191,6 +191,13 @@ notmuch_config_set_new_tags (notmuch_config_t *config,
 			     const char *new_tags[],
 			     size_t length);
 
+enum notmuch_maildir_sync
+notmuch_config_get_maildir_sync (notmuch_config_t *config);
+
+void
+notmuch_config_set_maildir_sync (notmuch_config_t *config,
+				 enum notmuch_maildir_sync maildir_sync);
+
 notmuch_bool_t
 debugger_is_active (void);
 
diff --git a/notmuch-config.c b/notmuch-config.c
index cf30603..60d5730 100644
--- a/notmuch-config.c
+++ b/notmuch-config.c
@@ -61,6 +61,21 @@ static const char user_config_comment[] =
     " recipient list of replies, and will set the From address based on the\n"
     " address to which the original email was addressed.\n";
 
+static const char maildir_config_comment[] =
+    " Maildir compatibility configuration\n"
+    "\n"
+    " Here you can configure whether and how will notmuch synchronize its\n"
+    " tags with maildir flags."
+    "\n"
+    "\tsync_level      Integer in the range 1 - 4 with the following meaning:\n"
+    "\t\t1 - No synchronization at all.\n"
+    "\t\t2 - 'notmuch new' tags the messages based on their maildir flags\n"
+    "\t\t    only when it sees them for the first time.\n"
+    "\t\t3 - Same as 2 plus 'notmuch new' updates tags when it detects the\n"
+    "\t\t    message was renamed.\n"
+    "\t\t4 - Same as 3 plus whenever message tags are changed, maildir\n"
+    "\t\t    flags are updated accordingly.\n";
+
 struct _notmuch_config {
     char *filename;
     GKeyFile *key_file;
@@ -72,6 +87,7 @@ struct _notmuch_config {
     size_t user_other_email_length;
     const char **new_tags;
     size_t new_tags_length;
+    enum notmuch_maildir_sync maildir_sync;
 };
 
 static int
@@ -191,6 +207,7 @@ notmuch_config_open (void *ctx,
     int file_had_database_group;
     int file_had_new_group;
     int file_had_user_group;
+    int file_had_maildir_group;
 
     if (is_new_ret)
 	*is_new_ret = 0;
@@ -221,6 +238,7 @@ notmuch_config_open (void *ctx,
     config->user_other_email_length = 0;
     config->new_tags = NULL;
     config->new_tags_length = 0;
+    config->maildir_sync = NOTMUCH_MAILDIR_SYNC_INVALID;
 
     if (! g_key_file_load_from_file (config->key_file,
 				     config->filename,
@@ -263,6 +281,7 @@ notmuch_config_open (void *ctx,
 						    "database");
     file_had_new_group = g_key_file_has_group (config->key_file, "new");
     file_had_user_group = g_key_file_has_group (config->key_file, "user");
+    file_had_maildir_group = g_key_file_has_group (config->key_file, "maildir");
 
 
     if (notmuch_config_get_database_path (config) == NULL) {
@@ -313,6 +332,10 @@ notmuch_config_open (void *ctx,
 	notmuch_config_set_new_tags (config, tags, 2);
     }
 
+    if (notmuch_config_get_maildir_sync (config) == NOTMUCH_MAILDIR_SYNC_INVALID) {
+	notmuch_config_set_maildir_sync (config, NOTMUCH_MAILDIR_SYNC_NONE);
+    }
+
     /* Whenever we know of configuration sections that don't appear in
      * the configuration file, we add some comments to help the user
      * understand what can be done. */
@@ -340,6 +363,12 @@ notmuch_config_open (void *ctx,
 				user_config_comment, NULL);
     }
 
+    if (! file_had_maildir_group)
+    {
+	g_key_file_set_comment (config->key_file, "maildir", NULL,
+				maildir_config_comment, NULL);
+    }
+
     if (is_new_ret)
 	*is_new_ret = is_new;
 
@@ -562,3 +591,22 @@ notmuch_config_set_new_tags (notmuch_config_t *config,
     config->new_tags = NULL;
 }
 
+enum notmuch_maildir_sync
+notmuch_config_get_maildir_sync (notmuch_config_t *config)
+{
+    if (config->maildir_sync == NOTMUCH_MAILDIR_SYNC_INVALID) {
+	config->maildir_sync = 
+	    g_key_file_get_integer (config->key_file,
+				    "maildir", "sync_level", NULL);
+    }
+    return config->maildir_sync;
+}
+
+void
+notmuch_config_set_maildir_sync (notmuch_config_t *config,
+				 enum notmuch_maildir_sync maildir_sync)
+{
+    g_key_file_set_integer (config->key_file,
+			    "maildir", "sync_level", maildir_sync);
+    config->maildir_sync = maildir_sync;
+}
diff --git a/notmuch-new.c b/notmuch-new.c
index ed3f944..40ea610 100644
--- a/notmuch-new.c
+++ b/notmuch-new.c
@@ -45,6 +45,7 @@ typedef struct {
 
     _filename_list_t *removed_files;
     _filename_list_t *removed_directories;
+    enum notmuch_maildir_sync maildir_sync;
 } add_files_state_t;
 
 static volatile sig_atomic_t do_add_files_print_progress = 0;
@@ -410,11 +411,13 @@ add_files_recursive (notmuch_database_t *notmuch,
 	    state->added_messages++;
 	    for (tag=state->new_tags; *tag != NULL; tag++)
 	        notmuch_message_add_tag (message, *tag);
-	    notmuch_message_maildir_to_tags (message, next);
+	    if (state->maildir_sync >= NOTMUCH_MAILDIR_SYNC_NEW)
+		notmuch_message_maildir_to_tags (message, next);
 	    break;
 	/* Non-fatal issues (go on to next file) */
 	case NOTMUCH_STATUS_DUPLICATE_MESSAGE_ID:
-	    notmuch_message_maildir_to_tags (message, next);
+	    if (state->maildir_sync >= NOTMUCH_MAILDIR_SYNC_NEW_RENAMED)
+		notmuch_message_maildir_to_tags (message, next);
 	    break;
 	case NOTMUCH_STATUS_FILE_NOT_EMAIL:
 	    fprintf (stderr, "Note: Ignoring non-mail file: %s\n",
@@ -738,6 +741,7 @@ notmuch_new_command (void *ctx, int argc, char *argv[])
 	return 1;
 
     add_files_state.new_tags = notmuch_config_get_new_tags (config, &add_files_state.new_tags_length);
+    add_files_state.maildir_sync = notmuch_config_get_maildir_sync (config);
     db_path = notmuch_config_get_database_path (config);
 
     dot_notmuch_path = talloc_asprintf (ctx, "%s/%s", db_path, ".notmuch");
diff --git a/notmuch-restore.c b/notmuch-restore.c
index b0a4e1c..b5c5c48 100644
--- a/notmuch-restore.c
+++ b/notmuch-restore.c
@@ -41,6 +41,8 @@ notmuch_restore_command (unused (void *ctx), int argc, char *argv[])
     if (notmuch == NULL)
 	return 1;
 
+    notmuch_database_set_maildir_sync (notmuch,
+				       notmuch_config_get_maildir_sync (config));
     if (argc) {
 	input = fopen (argv[0], "r");
 	if (input == NULL) {
diff --git a/notmuch-setup.c b/notmuch-setup.c
index 955deb7..8364854 100644
--- a/notmuch-setup.c
+++ b/notmuch-setup.c
@@ -195,6 +195,23 @@ notmuch_setup_command (unused (void *ctx),
 	g_ptr_array_free (tags, TRUE);
     }
 
+    printf ("\n"
+	    "Notmuch can synchronize certain tags with maildir flags. You can\n"
+	    "select between several levels of synchronization:\n"
+	    "1 - No synchronization at all.\n"
+	    "2 - 'notmuch new' tags the messages based on their maildir flags\n"
+	    "    only when it sees them for the first time.\n"
+	    "3 - Same as 2 plus 'notmuch new' updates tags when it detects the\n"
+	    "    message was renamed.\n"
+	    "4 - Same as 3 plus whenever message tags are changed, maildir\n"
+	    "    flags are updated accordingly.\n");
+
+    prompt ("Maildir synchronization level [%d]: ", notmuch_config_get_maildir_sync (config));
+    if (strlen (response) == 1 &&
+	response[0] >= '1' &&
+	response[0] <= '4')
+	notmuch_config_set_maildir_sync (config, atoi (response));
+
     if (! notmuch_config_save (config)) {
 	if (is_new)
 	  welcome_message_post_setup ();
diff --git a/notmuch-tag.c b/notmuch-tag.c
index fd54bc7..3a489a9 100644
--- a/notmuch-tag.c
+++ b/notmuch-tag.c
@@ -100,6 +100,8 @@ notmuch_tag_command (void *ctx, unused (int argc), unused (char *argv[]))
 				     NOTMUCH_DATABASE_MODE_READ_WRITE);
     if (notmuch == NULL)
 	return 1;
+    notmuch_database_set_maildir_sync (notmuch,
+				       notmuch_config_get_maildir_sync (config));
 
     query = notmuch_query_create (notmuch, query_string);
     if (query == NULL) {
-- 
1.7.1.3.g75e44

[PATCH 4/4] Tests for maildir synchronization

[Michal Sojka]

Signed-off-by: Michal Sojka <sojkam1@fel.cvut.cz>
---
 test/maildir-sync |  216 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 test/notmuch-test |    2 +-
 test/test-lib.sh  |   14 +++-
 3 files changed, 229 insertions(+), 3 deletions(-)
 create mode 100755 test/maildir-sync

diff --git a/test/maildir-sync b/test/maildir-sync
new file mode 100755
index 0000000..a5f8c36
--- /dev/null
+++ b/test/maildir-sync
@@ -0,0 +1,216 @@
+#!/bin/bash
+
+test_description="Test maildir synchronization"
+
+. ./test-lib.sh
+
+filter_show() {
+    sed -e 's/, /,\n/g'|sed -e "s|${MAIL_DIR}/||" -e '/^"tags"/d'
+    echo
+}
+
+cat >> "$NOTMUCH_CONFIG" <<EOF
+[maildir]
+sync_level=4
+EOF
+
+test_begin_subtest "No new messages"
+output=$(NOTMUCH_NEW)
+test_expect_equal "$output" "No new mail."
+
+cat > expected <<EOF
+Added 1 new message to the database.
+EOF
+test_expect_success "Add a message, no flags" '
+generate_message [subject]="\"test message\"" [date]="\"Sat, 01 Jan 2000 12:00:00 -0000\"" [filename]="\"msg-001:2,\"" &&
+NOTMUCH_NEW > actual &&
+test_cmp expected actual
+#emacs --eval "(gdb \"gdb --annotate=3 --args $(which notmuch) new\")"
+'
+cat > expected <<EOF
+thread:XXX   2000-01-01 [1/1] Notmuch Test Suite; test message (inbox unread)
+EOF
+test_expect_success 'Search for the message' '
+notmuch search tag:inbox and tag:unread | notmuch_search_sanitize > actual &&
+test_cmp expected actual
+'
+cat > expected <<EOF
+No new mail. Detected 1 file rename.
+EOF
+test_expect_success 'Add seen flag' '
+mv "${gen_msg_filename}" "${gen_msg_filename}S" &&
+increment_mtime "$(dirname "${gen_msg_filename}")" &&
+NOTMUCH_NEW > actual &&
+test_cmp expected actual
+'
+cat > expected <<EOF
+thread:XXX   2000-01-01 [1/1] Notmuch Test Suite; test message (inbox)
+EOF
+test_expect_success 'Check that tags were updated' '
+notmuch search tag:inbox and not tag:unread | notmuch_search_sanitize > actual &&
+test_cmp expected actual
+'
+cat > expected <<EOF
+Added 1 new message to the database.
+EOF
+test_expect_success "Add a seen message" '
+generate_message [subject]="\"test message 2\"" [date]="\"Sat, 01 Jan 2000 12:00:00 -0000\"" [filename]="\"msg-002:2,S\"" &&
+NOTMUCH_NEW > actual &&
+test_cmp expected actual
+'
+cat > expected <<EOF
+thread:XXX   2000-01-01 [1/1] Notmuch Test Suite; test message (inbox)
+thread:XXX   2000-01-01 [1/1] Notmuch Test Suite; test message 2 (inbox)
+EOF
+test_expect_success 'Check that the seen message is not tagged unread' '
+notmuch search tag:inbox and not tag:unread | notmuch_search_sanitize > actual &&
+test_cmp expected actual
+'
+test_expect_success 'Tag the seen messages as replied' '
+notmuch tag +replied -inbox tag:inbox and not tag:unread
+'
+
+cat > expected <<EOF
+msg-001:2,RS
+msg-002:2,RS
+EOF
+test_expect_success 'Check that R flag was added' '
+ls -1 "${MAIL_DIR}" > actual &&
+test_cmp expected actual
+'
+cat <<EOF > show-expected
+[[[{"id": "msg-001@notmuch-test-suite",
+"match": true,
+"filename": "msg-001:2,RS",
+"timestamp": 946728000,
+"date_relative": "2000-01-01",
+"headers": {"Subject": "test message",
+"From": "Notmuch Test Suite <test_suite@notmuchmail.org>",
+"To": "Notmuch Test Suite <test_suite@notmuchmail.org>",
+"Cc": "",
+"Bcc": "",
+"Date": "Sat,
+01 Jan 2000 12:00:00 -0000"},
+"body": [{"id": 1,
+"content-type": "text/plain",
+"content": "This is just a test message (#1)\n"}]},
+[]]]]
+EOF
+
+test_expect_success 'Renamed message can be shown without running notmuch new' '
+notmuch show --format=json id:msg-001@notmuch-test-suite | filter_show > show-actual &&
+test_cmp show-expected show-actual
+'
+
+test_expect_success 'Test that we can reply to the renamed message' '
+notmuch reply id:msg-001@notmuch-test-suite
+'
+
+echo "No new mail." > expected
+test_expect_success 'No rename should be detected by notmuch new' '
+increment_mtime "$(dirname "${gen_msg_filename}")" &&
+notmuch new > actual &&
+test_cmp expected actual
+'
+test_expect_success "Add a message to new/ without info" '
+generate_message [subject]="\"test message 3\"" [date]="\"Sat, 01 Jan 2000 12:00:00 -0000\"" [dir]=new &&
+NOTMUCH_NEW > actual &&
+test_cmp - actual <<EOF
+Added 1 new message to the database.
+EOF
+'
+test_expect_success "Check that the message has inbox and unread tags" '
+notmuch search tag:inbox and tag:unread | notmuch_search_sanitize > actual &&
+test_cmp - actual <<EOF
+thread:XXX   2000-01-01 [1/1] Notmuch Test Suite; test message 3 (inbox unread)
+EOF
+'
+test_expect_success "Tag the message with 'tmp' tag" '
+notmuch tag +tmp tag:inbox and tag:unread'
+
+test_expect_success "Check that the message was not moved from new/ to cur/" '
+echo filename:$gen_msg_filename > expected &&
+notmuch show id:$gen_msg_id|grep -o "filename:.*$" > actual &&
+test_cmp expected actual &&
+test -f "$gen_msg_filename"
+'
+test_expect_success "Check that the message was not renamed" '
+ls "${MAIL_DIR}/new" > actual &&
+test_cmp - actual <<EOF
+msg-003
+EOF
+'
+test_expect_success 'Removing of unread tag should fail without cur/' '
+test_must_fail notmuch tag -unread tag:inbox and tag:unread
+'
+test_expect_success "Check that the tags were not changed" '
+notmuch search tag:inbox and tag:unread | notmuch_search_sanitize > actual &&
+test_cmp - actual <<EOF
+thread:XXX   2000-01-01 [1/1] Notmuch Test Suite; test message 3 (inbox tmp unread)
+EOF
+'
+
+# notmuch new is not necessary here, but we run it in order to check
+# for 'no rename' later (*).
+test_expect_success 'Create cur/ and let notmuch know about it' '
+mkdir "$MAIL_DIR/cur" &&
+notmuch new
+'
+test_expect_success 'Removing of unread tag should pass with cur/' '
+notmuch tag -unread tag:inbox and tag:unread
+'
+test_expect_success 'Check that the message was moved to cur/' '\
+ls "$MAIL_DIR/cur" > actual &&
+test_cmp - actual <<EOF
+msg-003:2,S
+EOF
+'
+test_expect_success 'No rename should be detected by notmuch new' '
+increment_mtime "$MAIL_DIR/cur" &&
+notmuch new > actual &&
+test_cmp - actual <<EOF
+No new mail.
+EOF
+'
+# (*) If notmuch new was not run we've got "Processed 1 file in almost
+# no time" here. The reason is that removing unread tag in a previous
+# test created directory document in the database but this document
+# was not linked as subdirectory of $MAIL_DIR. Therefore notmuch new
+# could not reach the cur/ directory and its files in it during
+# recurive traversal.
+test_expect_success 'Remove info from file name' '
+mv "$MAIL_DIR/cur/msg-003:2,S" "$MAIL_DIR/cur/msg-003" &&
+increment_mtime "$MAIL_DIR/cur" &&
+NOTMUCH_NEW > actual
+test_cmp - actual <<EOF
+No new mail. Detected 1 file rename.
+EOF
+'
+test_expect_success "Check that removing info did not change tags" '
+notmuch search tag:inbox | notmuch_search_sanitize > actual &&
+test_cmp - actual <<EOF
+thread:XXX   2000-01-01 [1/1] Notmuch Test Suite; test message 3 (inbox tmp)
+EOF
+'
+test_expect_success "Add a message to fakenew/ without info" '
+generate_message [subject]="\"test message 4\"" [date]="\"Sat, 01 Jan 2000 12:00:00 -0000\"" [dir]=fakenew &&
+NOTMUCH_NEW > actual &&
+test_cmp - actual <<EOF
+Added 1 new message to the database.
+EOF
+'
+test_expect_success "Check that the message has inbox and unread tags" '
+notmuch search tag:inbox and tag:unread | notmuch_search_sanitize > actual &&
+test_cmp - actual <<EOF
+thread:XXX   2000-01-01 [1/1] Notmuch Test Suite; test message 4 (inbox unread)
+EOF
+'
+test_expect_success 'Removing of unread tag should leave the message in fakenew/' '
+notmuch tag -unread tag:inbox and tag:unread &&
+ls "$MAIL_DIR/fakenew" > actual &&
+test_cmp - actual <<EOF
+msg-004:2,S
+EOF
+'
+
+test_done
diff --git a/test/notmuch-test b/test/notmuch-test
index 13c5d80..39e0fea 100755
--- a/test/notmuch-test
+++ b/test/notmuch-test
@@ -8,7 +8,7 @@
 
 cd $(dirname "$0")
 
-TESTS="basic new search json thread-naming reply dump-restore uuencode thread-order author-order from-guessing long-id encoding"
+TESTS="basic new search json thread-naming reply dump-restore uuencode thread-order author-order from-guessing long-id encoding maildir-sync"
 
 # Clean up any results from a previous run
 rm -r test-results >/dev/null 2>/dev/null
diff --git a/test/test-lib.sh b/test/test-lib.sh
index 5f7fa14..2036669 100644
--- a/test/test-lib.sh
+++ b/test/test-lib.sh
@@ -237,6 +237,12 @@ increment_mtime ()
 #	Generate the message in directory 'directory/of/choice' within
 #	the mail store. The directory will be created if necessary.
 #
+#  [filename]=name
+#
+#	Store the message in file 'name'. The default is to store it
+#	in 'msg-<count>', where <count> is three-digit number of the
+#	message.
+#	
 #  [body]=text
 #
 #	Text to use as the body of the email message
@@ -273,10 +279,14 @@ generate_message ()
     local additional_headers
 
     gen_msg_cnt=$((gen_msg_cnt + 1))
-    gen_msg_name=msg-$(printf "%03d" $gen_msg_cnt)
+    if [ -z "${template[filename]}" ]; then
+	gen_msg_name="msg-$(printf "%03d" $gen_msg_cnt)"
+    else
+	gen_msg_name=${template[filename]}
+    fi
 
     if [ -z "${template[id]}" ]; then
-	gen_msg_id="${gen_msg_name}@notmuch-test-suite"
+	gen_msg_id="${gen_msg_name%:2,*}@notmuch-test-suite"
     else
 	gen_msg_id="${template[id]}"
     fi
-- 
1.7.1.3.g75e44

Re: [PATCH 0/4] Maildir synchronization v2

[Mike Kelly]

On Wed, 13 Oct 2010 14:13:54 +0200
Michal Sojka <sojkam1@fel.cvut.cz> wrote:

> Hi,
> 
> this is an updated version of patches sent in
> id:1273580061-22580-1-git-send-email-sojkam1@fel.cvut.cz. Only the
> last patch (tests) was actually updated to work with new test suite.
> These patches has already been tested by several people and I think
> they are ready to be merged. They can be pulled by:
> 
> git pull git://rtime.felk.cvut.cz/notmuch maildir-sync-v2

This sounds like just the sort of feature set I'd want from this sort
of patch, except unfortunately it won't compile for me on FreeBSD:

  gcc -c -DNOTMUCH_VERSION=0.3.1-96-g6e0dca9 -O2 -Wall -Wextra -Wwrite-strings -Wswitch-enum -Wmissing-declarations -DHAVE_GETLINE=1 -D_REENTRANT -I/usr/local/include/gmime-2.4 -I/usr/local/include/glib-2.0 -I/usr/local/lib/glib-2.0/include   -I/usr/local/include   -DHAVE_VALGRIND=0  -DHAVE_STRCASESTR=1  -Icompat -Ilib -fPIC notmuch-config.c -o notmuch-config.o
  notmuch-config.c:90: error: field 'maildir_sync' has incomplete type
  notmuch-config.c: In function 'notmuch_config_open':
  notmuch-config.c:241: error: 'NOTMUCH_MAILDIR_SYNC_INVALID' undeclared (first use in this function)
  notmuch-config.c:241: error: (Each undeclared identifier is reported only once
  notmuch-config.c:241: error: for each function it appears in.)
  notmuch-config.c:335: error: invalid use of undefined type 'enum notmuch_maildir_sync'
  notmuch-config.c:336: error: 'NOTMUCH_MAILDIR_SYNC_NONE' undeclared (first use in this function)
  notmuch-config.c:336: error: type of formal parameter 2 is incomplete
  notmuch-config.c: At top level:
  notmuch-config.c:596: error: return type is an incomplete type
  notmuch-config.c:596: error: conflicting types for 'notmuch_config_get_maildir_sync'
  notmuch-client.h:195: error: previous declaration of 'notmuch_config_get_maildir_sync' was here
  notmuch-config.c: In function 'notmuch_config_get_maildir_sync':
  notmuch-config.c:597: error: 'NOTMUCH_MAILDIR_SYNC_INVALID' undeclared (first use in this function)
  notmuch-config.c:602: warning: 'return' with a value, in function returning void
  notmuch-config.c: At top level:
  notmuch-config.c:607: error: parameter 2 ('maildir_sync') has incomplete type
  notmuch-config.c: In function 'notmuch_config_set_maildir_sync':
  notmuch-config.c:607: warning: unused parameter 'maildir_sync'
  gmake: *** [notmuch-config.o] Error 1

This is with "gcc (GCC) 4.2.1 20070719  [FreeBSD]"

Switching to gcc44 seems to allow it to compile correctly, however notmuch didn't used to have that dependency.

-- 
Mike Kelly

Re: [PATCH 2/4] Maildir synchronization

[Sebastian Spaeth]

[-- Attachment #1: Type: text/plain, Size: 810 bytes --]

> +/* ASCII ordered table of Maildir flags and associated tags */
> +struct maildir_flag_tag flag2tag[] = {
> +    { 'D', "draft",   false},
> +    { 'F', "flagged", false},
> +    { 'P', "passed",  false},
> +    { 'R', "replied", false},
> +    { 'S', "unread",  true },
> +    { 'T', "delete",  false},
> +};

for all of the 4 patches a:

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>

with one caveat: you use the tag "delete" to mark the state "trashed". I
think people have slowly settled on the convention "deleted" which is
also more consistent with a state, such as unread, replied, etc...

They work fine and obsolete my notmuchsync (except for the pruning of
deleted mails and the tagging based on folder names). +1 for merging in
notmuch-next or wherever we collect that stuff.

Sebastian

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: [PATCH 0/4] Maildir synchronization v2

[Mike Kelly]

On Wed, 13 Oct 2010 09:31:58 -0400
Mike Kelly <pioto@pioto.org> wrote:

> On Wed, 13 Oct 2010 14:13:54 +0200
> Michal Sojka <sojkam1@fel.cvut.cz> wrote:
> 
> > Hi,
> > 
> > this is an updated version of patches sent in
> > id:1273580061-22580-1-git-send-email-sojkam1@fel.cvut.cz. Only the
> > last patch (tests) was actually updated to work with new test suite.
> > These patches has already been tested by several people and I think
> > they are ready to be merged. They can be pulled by:
> > 
> > git pull git://rtime.felk.cvut.cz/notmuch maildir-sync-v2
> 
> This sounds like just the sort of feature set I'd want from this sort
> of patch, except unfortunately it won't compile for me on FreeBSD:

Looks like this may also require a newer xapian than i have now
(xapian-core 1.0.18), as notmuch new aborts with:

  terminate called after throwing an instance of 'Xapian::InvalidArgumentError'

I'll see if I can track down anything more concrete as a cause.

-- 
Mike Kelly

Re: [PATCH 0/4] Maildir synchronization v2

[Mike Kelly]

On Wed, 13 Oct 2010 10:24:25 -0400
Mike Kelly <pioto@pioto.org> wrote:

> Looks like this may also require a newer xapian than i have now
> (xapian-core 1.0.18), as notmuch new aborts with:
> 
>   terminate called after throwing an instance of
> 'Xapian::InvalidArgumentError'
> 
> I'll see if I can track down anything more concrete as a cause.

Well, the full backtrace looks like this:

  #0  __cxa_throw (obj=0x2883d560, tinfo=0x8070278, dest=0x806132e <~InvalidArgumentError>)
      at ../../.././../gcc-4.4-20100309/libstdc++-v3/libsupc++/eh_throw.cc:67
          header = (__cxxabiv1::__cxa_refcounted_exception *) 0x2883d500
  #1  0x28353b72 in Xapian::Document::Internal::remove_term () from /usr/local/lib/libxapian.so.21
  No symbol table info available.
  #2  0x28353c96 in Xapian::Document::remove_term () from /usr/local/lib/libxapian.so.21
  No symbol table info available.
  #3  0x080604bc in _notmuch_message_remove_term (message=0x28a85c10, prefix_name=0x806c887 "tag", value=0x806c564 "draft")
      at lib/message.cc:737
          term = 0x28806ab0 "Kdraft"
  #4  0x0806086a in notmuch_message_remove_tag (message=0x28a85c10, tag=0x806c564 "draft") at lib/message.cc:832
          private_status = 134662454
          status = NOTMUCH_STATUS_SUCCESS
  #5  0x08060a56 in notmuch_message_maildir_to_tags (message=0x28a85c10, 
      filename=0x28a860f0 "/usr/home/staff/mike/mail/staff-support/cur/1286944232_2.71920.pit84.pair.com,U=26762,FMD5=74eb4e66bae4700f6b79b81477ef9cfa:2,S") at lib/message.cc:889
          flags = 0x28a8616e "S"
          i = 0
          status = NOTMUCH_STATUS_SUCCESS
          p = 0x28a8616f ""
          f = 83 'S'
          valid = true
          unread = true
  #6  0x08050bbe in add_files_recursive (notmuch=0x2881e7f0, path=0x2881ec70 "/usr/home/staff/mike/mail/staff-support/cur", 
      state=0xbfbfe238) at notmuch-new.c:420
          err = 32
          dir = (DIR *) 0x0
          entry = (struct dirent *) 0x28a68640
          next = 0x28a860f0 "/usr/home/staff/mike/mail/staff-support/cur/1286944232_2.71920.pit84.pair.com,U=26762,FMD5=74eb4e66bae4700f6b79b81477ef9cfa:2,S"
          fs_mtime = 1286978382
          db_mtime = 1286944232
          status = NOTMUCH_STATUS_DUPLICATE_MESSAGE_ID
          ret = NOTMUCH_STATUS_SUCCESS
          message = (notmuch_message_t *) 0x28a85c10
          fs_entries = (struct dirent **) 0x28a6f000
          i = 4915
          num_fs_entries = 4922
          directory = (notmuch_directory_t *) 0x288069f0
          db_files = (notmuch_filenames_t *) 0x28806af0
          db_subdirs = (notmuch_filenames_t *) 0x28806b30
          st = {st_dev = 89, st_ino = 4172664, st_mode = 16832, st_nlink = 2, st_uid = 1118, st_gid = 0, st_rdev = 16689352, 
    st_atimespec = {tv_sec = 1286979943, tv_nsec = 0}, st_mtimespec = {tv_sec = 1286978382, tv_nsec = 0}, st_ctimespec = {
      tv_sec = 1286978382, tv_nsec = 0}, st_size = 1205760, st_blocks = 2400, st_blksize = 4096, st_flags = 0, st_gen = 0, st_lspare = 0, 
    st_birthtimespec = {tv_sec = 1264441077, tv_nsec = 0}}
          is_maildir = 0
          new_directory = 0
          tag = (const char **) 0x8069d62
  #7  0x08050780 in add_files_recursive (notmuch=0x2881e7f0, path=0x2881e790 "/usr/home/staff/mike/mail/staff-support", state=0xbfbfe238)
      at notmuch-new.c:302
          dir = (DIR *) 0x0
          entry = (struct dirent *) 0x288cc5b0
          next = 0x2881ec70 "/usr/home/staff/mike/mail/staff-support/cur"
          fs_mtime = 1264003223
          db_mtime = 1264003223
          status = NOTMUCH_STATUS_SUCCESS
          ret = NOTMUCH_STATUS_SUCCESS
          message = (notmuch_message_t *) 0x0
          fs_entries = (struct dirent **) 0x2883d480
          i = 2
          num_fs_entries = 5
          directory = (notmuch_directory_t *) 0x28806870
          db_files = (notmuch_filenames_t *) 0x28806a30
          db_subdirs = (notmuch_filenames_t *) 0x28806a70
          st = {st_dev = 89, st_ino = 3889110, st_mode = 16832, st_nlink = 5, st_uid = 1118, st_gid = 0, st_rdev = 15524535, 
    st_atimespec = {tv_sec = 1286979943, tv_nsec = 0}, st_mtimespec = {tv_sec = 1264003223, tv_nsec = 0}, st_ctimespec = {
      tv_sec = 1264473647, tv_nsec = 0}, st_size = 512, st_blocks = 4, st_blksize = 4096, st_flags = 0, st_gen = 0, st_lspare = 0, 
    st_birthtimespec = {tv_sec = 1264003223, tv_nsec = 0}}
          is_maildir = 1
          new_directory = 0
          tag = (const char **) 0x8069d62
  #8  0x08050780 in add_files_recursive (notmuch=0x2881e7f0, path=0x2881f680 "/usr/home/staff/mike/mail", state=0xbfbfe238)
      at notmuch-new.c:302
          dir = (DIR *) 0x0
          entry = (struct dirent *) 0x28808900
          next = 0x2881e790 "/usr/home/staff/mike/mail/staff-support"
          fs_mtime = 1284781182
          db_mtime = 1284781182
          status = NOTMUCH_STATUS_SUCCESS
          ret = NOTMUCH_STATUS_SUCCESS
          message = (notmuch_message_t *) 0x0
          fs_entries = (struct dirent **) 0x28825000
          i = 40
          num_fs_entries = 47
          directory = (notmuch_directory_t *) 0x28806730
          db_files = (notmuch_filenames_t *) 0x288068b0
          db_subdirs = (notmuch_filenames_t *) 0x28806970
          st = {st_dev = 89, st_ino = 3889107, st_mode = 16832, st_nlink = 47, st_uid = 1118, st_gid = 0, st_rdev = 15524532, 
    st_atimespec = {tv_sec = 1286979943, tv_nsec = 0}, st_mtimespec = {tv_sec = 1284781182, tv_nsec = 0}, st_ctimespec = {
      tv_sec = 1284781182, tv_nsec = 0}, st_size = 1024, st_blocks = 4, st_blksize = 4096, st_flags = 0, st_gen = 0, st_lspare = 0, 
    st_birthtimespec = {tv_sec = 1264181508, tv_nsec = 0}}
          is_maildir = 0
          new_directory = 0
          tag = (const char **) 0xbfbfe178
  #9  0x08050fb5 in add_files (notmuch=0x2881e7f0, path=0x2881f680 "/usr/home/staff/mike/mail", state=0xbfbfe238) at notmuch-new.c:553
          status = NOTMUCH_STATUS_SUCCESS
          action = {__sigaction_u = {__sa_handler = 0x8050166 <handle_sigalrm>, __sa_sigaction = 0x8050166 <handle_sigalrm>}, sa_flags = 2, 
    sa_mask = {__bits = {0, 0, 0, 0}}}
          timerval = {it_interval = {tv_sec = 1, tv_usec = 0}, it_value = {tv_sec = 1, tv_usec = 0}}
          timer_is_active = 1
          st = {st_dev = 89, st_ino = 3889107, st_mode = 16832, st_nlink = 47, st_uid = 1118, st_gid = 0, st_rdev = 15524532, 
    st_atimespec = {tv_sec = 1286979943, tv_nsec = 0}, st_mtimespec = {tv_sec = 1284781182, tv_nsec = 0}, st_ctimespec = {
      tv_sec = 1284781182, tv_nsec = 0}, st_size = 1024, st_blocks = 4, st_blksize = 4096, st_flags = 0, st_gen = 0, st_lspare = 0, 
    st_birthtimespec = {tv_sec = 1264181508, tv_nsec = 0}}
  #10 0x08051809 in notmuch_new_command (ctx=0x288090a0, argc=0, argv=0xbfbfe320) at notmuch-new.c:800
          config = (notmuch_config_t *) 0x2881e490
          notmuch = (notmuch_database_t *) 0x2881e7f0
          add_files_state = {output_is_a_tty = 1, verbose = 0, new_tags = 0x288064b0, new_tags_length = 0, total_files = 0, 
    processed_files = 1, added_messages = 0, tv_start = {tv_sec = 1286980142, tv_usec = 691823}, removed_files = 0x288066b0, 
    removed_directories = 0x288066f0, maildir_sync = NOTMUCH_MAILDIR_SYNC_NEW_RENAMED_TAGGED}
          elapsed = -0.12455515557656988
          tv_now = {tv_sec = 671753656, tv_usec = -1077943692}
          ret = 0
          st = {st_dev = 89, st_ino = 3886724, st_mode = 16877, st_nlink = 3, st_uid = 1118, st_gid = 0, st_rdev = 15527616, 
    st_atimespec = {tv_sec = 1286953297, tv_nsec = 0}, st_mtimespec = {tv_sec = 1273174498, tv_nsec = 0}, st_ctimespec = {
      tv_sec = 1273174498, tv_nsec = 0}, st_size = 512, st_blocks = 4, st_blksize = 4096, st_flags = 0, st_gen = 0, st_lspare = 0, 
    st_birthtimespec = {tv_sec = 1273174498, tv_nsec = 0}}
          db_path = 0x2881f680 "/usr/home/staff/mike/mail"
          dot_notmuch_path = 0x0
          action = {__sigaction_u = {__sa_handler = 0x8050181 <handle_sigint>, __sa_sigaction = 0x8050181 <handle_sigint>}, sa_flags = 2, 
    sa_mask = {__bits = {0, 0, 0, 0}}}
          f = (_filename_node_t *) 0x2d
          renamed_files = 672185856
          removed_files = 0
          status = 134639189
          i = 0
  #11 0x0804edd6 in main (argc=2, argv=0xbfbfe318) at notmuch.c:499
          local = (void *) 0x288090a0
          command = (command_t *) 0x80708f4
          i = 1

-- 
Mike Kelly

Re: [PATCH 0/4] Maildir synchronization v2

[Servilio Afre Puentes]

On 13 October 2010 08:13, Michal Sojka <sojkam1@fel.cvut.cz> wrote:
[...]
> THERE IS CURRENTLY ONE KNOWN ISSUE: Viewing/storing of attachments of
> unread messages doesn't work. The reason is that when you view the
> message its unread tag is removed which causes the file to be renamed,
> but Emacs still uses the original name to access the attachment. You can
> workaround this by closing the message and opening it again. This issue
> will be fixed after we (I) implement "notmuch cat" command. With this
> command, emacs would not access the messages by the file name, but by
> running notmuch cat id:<message-id> which will always give the correct
> content.

Wouldn't it be more efficient to query notmuch for the filename using
the message ID we store in the DB? When network usage is implemented,
tramp can give us transparent remote file access in emacs.

Regards,

Servilio

Re: [PATCH 0/4] Maildir synchronization v2

[David Bremner]

On Wed, 13 Oct 2010 10:50:36 -0400, Servilio Afre Puentes <servilio@gmail.com> wrote:
> 
> Wouldn't it be more efficient to query notmuch for the filename using
> the message ID we store in the DB? When network usage is implemented,
> tramp can give us transparent remote file access in emacs.
> 

We can of course do both. From an implementation point of view, they
would both fall under "selective output in plain text" which various
people have been meaning to do, and I think one is no harder than the
other. On the other hand I doubt that there is an efficiency advantage
to using tramp, it is is more fragile (for example it depends on the
remote shell prompt).

d

Re: [PATCH 0/4] Maildir synchronization v2

[Michal Sojka]

On Wed, 13 Oct 2010, Servilio Afre Puentes wrote:
> On 13 October 2010 08:13, Michal Sojka <sojkam1@fel.cvut.cz> wrote:
> [...]
> > THERE IS CURRENTLY ONE KNOWN ISSUE: Viewing/storing of attachments of
> > unread messages doesn't work. The reason is that when you view the
> > message its unread tag is removed which causes the file to be renamed,
> > but Emacs still uses the original name to access the attachment. You can
> > workaround this by closing the message and opening it again. This issue
> > will be fixed after we (I) implement "notmuch cat" command. With this
> > command, emacs would not access the messages by the file name, but by
> > running notmuch cat id:<message-id> which will always give the correct
> > content.
> 
> Wouldn't it be more efficient to query notmuch for the filename using
> the message ID we store in the DB? When network usage is implemented,
> tramp can give us transparent remote file access in emacs.

Tramp would not be available in non-emacs GUIs. Also, when notmuch cat
is implemented, emacs gui will be usable remotely by simply defining
notmuch-command variable to contain the name of the script containing:

     ssh user@host notmuch "$@"

-Michal

Re: [PATCH 0/4] Maildir synchronization v2

[Michal Sojka]

On Wed, 13 Oct 2010, Mike Kelly wrote:
> On Wed, 13 Oct 2010 10:24:25 -0400
> Mike Kelly <pioto@pioto.org> wrote:
> 
> > Looks like this may also require a newer xapian than i have now
> > (xapian-core 1.0.18), as notmuch new aborts with:
> > 
> >   terminate called after throwing an instance of
> > 'Xapian::InvalidArgumentError'
> > 
> > I'll see if I can track down anything more concrete as a cause.
> 
> Well, the full backtrace looks like this:

So you were able to compile it? What was the problem?

> 
>   #0  __cxa_throw (obj=0x2883d560, tinfo=0x8070278, dest=0x806132e <~InvalidArgumentError>)
>       at ../../.././../gcc-4.4-20100309/libstdc++-v3/libsupc++/eh_throw.cc:67
>           header = (__cxxabiv1::__cxa_refcounted_exception *) 0x2883d500
>   #1  0x28353b72 in Xapian::Document::Internal::remove_term () from /usr/local/lib/libxapian.so.21
>   No symbol table info available.
>   #2  0x28353c96 in Xapian::Document::remove_term () from /usr/local/lib/libxapian.so.21
>   No symbol table info available.
>   #3  0x080604bc in _notmuch_message_remove_term (message=0x28a85c10, prefix_name=0x806c887 "tag", value=0x806c564 "draft")
>       at lib/message.cc:737
>           term = 0x28806ab0 "Kdraft"
>   #4  0x0806086a in notmuch_message_remove_tag (message=0x28a85c10, tag=0x806c564 "draft") at lib/message.cc:832
>           private_status = 134662454
>           status = NOTMUCH_STATUS_SUCCESS

I guess the problem would be that it removes nonexistent tag. I do not
experience such a problem on Linux with Xapian 1.2.3.

-Michal

Re: [PATCH 0/4] Maildir synchronization v2

[Mike Kelly]

[-- Attachment #1: Type: text/plain, Size: 388 bytes --]

On Wed, 13 Oct 2010 22:59:49 +0200
Michal Sojka <sojkam1@fel.cvut.cz> wrote:

> So you were able to compile it? What was the problem?

I mentioned at the bottom of a previous message:

> This is with "gcc (GCC) 4.2.1 20070719  [FreeBSD]"
>
> Switching to gcc44 seems to allow it to compile correctly, however
> notmuch didn't used to have that dependency.

-- 
Mike Kelly


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [PATCH 0/4] Maildir synchronization v2

[Dirk Hohndel]

On Wed, 13 Oct 2010 22:34:34 +0200, Michal Sojka <sojkam1@fel.cvut.cz> wrote:
> On Wed, 13 Oct 2010, Servilio Afre Puentes wrote:
> > On 13 October 2010 08:13, Michal Sojka <sojkam1@fel.cvut.cz> wrote:
> > [...]
> > > THERE IS CURRENTLY ONE KNOWN ISSUE: Viewing/storing of attachments of
> > > unread messages doesn't work. The reason is that when you view the
> > > message its unread tag is removed which causes the file to be renamed,
> > > but Emacs still uses the original name to access the attachment. You can
> > > workaround this by closing the message and opening it again. This issue
> > > will be fixed after we (I) implement "notmuch cat" command. With this
> > > command, emacs would not access the messages by the file name, but by
> > > running notmuch cat id:<message-id> which will always give the correct
> > > content.
> > 
> > Wouldn't it be more efficient to query notmuch for the filename using
> > the message ID we store in the DB? When network usage is implemented,
> > tramp can give us transparent remote file access in emacs.
> 
> Tramp would not be available in non-emacs GUIs. Also, when notmuch cat
> is implemented, emacs gui will be usable remotely by simply defining
> notmuch-command variable to contain the name of the script containing:
> 
>      ssh user@host notmuch "$@"

That to me is certainly a very elegant solution... so what's stopping us
from implementing notmuch cat? No one taken the time to do it? Or is
there a design issue left to be resolved?

/D

Re: [PATCH 0/4] Maildir synchronization v2

[Michal Sojka]

On Thu, 14 Oct 2010, Dirk Hohndel wrote:
> On Wed, 13 Oct 2010 22:34:34 +0200, Michal Sojka <sojkam1@fel.cvut.cz> wrote:
> > 
> >      ssh user@host notmuch "$@"
> 
> That to me is certainly a very elegant solution... so what's stopping us
> from implementing notmuch cat? No one taken the time to do it?

Yes. I plan to work on this during the next days.

-Michal

> Or is there a design issue left to be resolved?
> 
> /D

Re: [PATCH 0/4] Maildir synchronization v2

[Mike Kelly]

On Wed, 13 Oct 2010 14:13:54 +0200
Michal Sojka <sojkam1@fel.cvut.cz> wrote:

> Hi,
> 
> this is an updated version of patches sent in
> id:1273580061-22580-1-git-send-email-sojkam1@fel.cvut.cz. Only the
> last patch (tests) was actually updated to work with new test suite.
> These patches has already been tested by several people and I think
> they are ready to be merged. They can be pulled by:
> 
> git pull git://rtime.felk.cvut.cz/notmuch maildir-sync-v2

I've tested these patches on Linux, and they seem to work as expected.
However, I'd like to suggest/request that the level-4 syncing of a
filename should happen in notmuch_(message|thread)_(add|remove)_tag(),
not in notmuch-tag.cc. Otherwise, anyone using libnotmuch directly
(e.g. alternative clients, things using python bindings, etc) will not
benefit from this.

-- 
Mike Kelly

Re: [PATCH 0/4] Maildir synchronization v2

[Michal Sojka]

On Mon, 18 Oct 2010, Mike Kelly wrote:
> Michal Sojka <sojkam1@fel.cvut.cz> wrote:
> > they are ready to be merged. They can be pulled by:
> > 
> > git pull git://rtime.felk.cvut.cz/notmuch maildir-sync-v2
> 
> I've tested these patches on Linux, and they seem to work as expected.
> However, I'd like to suggest/request that the level-4 syncing of a
> filename should happen in notmuch_(message|thread)_(add|remove)_tag(),
> not in notmuch-tag.cc. Otherwise, anyone using libnotmuch directly
> (e.g. alternative clients, things using python bindings, etc) will not
> benefit from this.

Hi, I'd agree but these pathes don't touch notmuch-tag.c in a
significant way. What's done there is to read the value from config and
to call a function (from libnotmuch) to setup an internal variable
accordingly. Other users of libnotmuch have probably different means of
configuration than notmuch itself and should call this function in a
similar way.

-Michal

Re: [PATCH 2/4] Maildir synchronization

[Michal Sojka]

On Wed, 13 Oct 2010, Sebastian Spaeth wrote:
> > +/* ASCII ordered table of Maildir flags and associated tags */
> > +struct maildir_flag_tag flag2tag[] = {
> > +    { 'D', "draft",   false},
> > +    { 'F', "flagged", false},
> > +    { 'P', "passed",  false},
> > +    { 'R', "replied", false},
> > +    { 'S', "unread",  true },
> > +    { 'T', "delete",  false},
> > +};
> 
> [...]
> 
> with one caveat: you use the tag "delete" to mark the state "trashed". I
> think people have slowly settled on the convention "deleted" which is
> also more consistent with a state, such as unread, replied, etc...

Hi,

I changed "delete" to "deleted" and the updated patch series is
reachable by:

git pull git://rtime.felk.cvut.cz/notmuch maildir-sync-v3

-Michal

Re: [PATCH 0/4] Maildir synchronization v2

[Mike Kelly]

On Wed, Oct 13, 2010 at 10:35:15AM -0400, Mike Kelly wrote:
> On Wed, 13 Oct 2010 10:24:25 -0400
> Mike Kelly <pioto@pioto.org> wrote:
> 
> > Looks like this may also require a newer xapian than i have now
> > (xapian-core 1.0.18), as notmuch new aborts with:
> > 
> >   terminate called after throwing an instance of
> > 'Xapian::InvalidArgumentError'
> > 
> > I'll see if I can track down anything more concrete as a cause.

As a bit of a followup... I wasn't able to reproduce this on a different
FreeBSD 8.0 system, so I think my workstation must be subtly broken,
which is lame. And, if I set synchronize_flags=false in my
~/.notmuch-config, I don't get this error on my workstation, either.

-- 
Mike Kelly