[PATCH] notmuch: Add Maildir directory name as tag name for messages

[PATCH] notmuch: Add Maildir directory name as tag name for messages

[Aneesh Kumar K.V]

From 24711481dfe770417aa0a13308a9bb842dfb03f4 Mon Sep 17 00:00:00 2001
From: Aneesh Kumar K.V <aneesh.kumar@linux.vnet.ibm.com>
Date: Wed, 18 Nov 2009 21:20:12 +0530
Subject: [PATCH] notmuch: Add Maildir directory name as tag name for messages

This patch adds maildir directory name as the tag name for
messages. This helps in adding tags using filtering already
provided by procmail.


Signed-off-by: Aneesh Kumar K.V <aneesh.kumar@linux.vnet.ibm.com>
---
 notmuch-new.c |   12 ++++++++++--
 1 files changed, 10 insertions(+), 2 deletions(-)

diff --git a/notmuch-new.c b/notmuch-new.c
index 5405a9f..50d0a5a 100644
--- a/notmuch-new.c
+++ b/notmuch-new.c
@@ -96,6 +96,7 @@ add_files_print_progress (add_files_state_t *state)
 static notmuch_status_t
 add_files_recursive (notmuch_database_t *notmuch,
 		     const char *path,
+		     const char *tag,
 		     struct stat *st,
 		     add_files_state_t *state)
 {
@@ -186,6 +187,7 @@ add_files_recursive (notmuch_database_t *notmuch,
 		    case NOTMUCH_STATUS_SUCCESS:
 			state->added_messages++;
 			tag_inbox_and_unread (message);
+			notmuch_message_add_tag (message, tag);
 			break;
 		    /* Non-fatal issues (go on to next file) */
 		    case NOTMUCH_STATUS_DUPLICATE_MESSAGE_ID:
@@ -223,7 +225,13 @@ add_files_recursive (notmuch_database_t *notmuch,
 		}
 	    }
 	} else if (S_ISDIR (st->st_mode)) {
-	    status = add_files_recursive (notmuch, next, st, state);
+		if ((strcmp (entry->d_name, "cur") == 0) ||
+			(strcmp (entry->d_name, "new") == 0) ||
+			(strcmp (entry->d_name, "tmp") == 0)) {
+			status = add_files_recursive (notmuch, next, tag, st, state);
+		} else {
+			status = add_files_recursive (notmuch, next, entry->d_name, st, state);
+		}
 	    if (status && ret == NOTMUCH_STATUS_SUCCESS)
 		ret = status;
 	}
@@ -285,7 +293,7 @@ add_files (notmuch_database_t *notmuch,
     timerval.it_value.tv_usec = 0;
     setitimer (ITIMER_REAL, &timerval, NULL);
 
-    status = add_files_recursive (notmuch, path, &st, state);
+    status = add_files_recursive (notmuch, path, basename(path), &st, state);
 
     /* Now stop the timer. */
     timerval.it_interval.tv_sec = 0;
-- 
1.6.5.2.74.g610f9

Re: [PATCH] notmuch: Add Maildir directory name as tag name for messages

[Carl Worth]

On Wed, 18 Nov 2009 21:25:28 +0530, aneesh.kumar@linux.vnet.ibm.com (Aneesh Kumar K.V) wrote:
> This patch adds maildir directory name as the tag name for
> messages. This helps in adding tags using filtering already
> provided by procmail.

A fantastic patch, Aneesh! I think this will be handy for a lot of
people importing mail from all kinds of systems. This is pushed now.

Thanks again,

-Carl

Re: [PATCH] notmuch: Add Maildir directory name as tag name for messages

[Carl Worth]

On Sat, 21 Nov 2009 19:39:22 +0100, Carl Worth <cworth@cworth.org> wrote:
> On Wed, 18 Nov 2009 21:25:28 +0530, aneesh.kumar@linux.vnet.ibm.com (Aneesh Kumar K.V) wrote:
> A fantastic patch, Aneesh! I think this will be handy for a lot of
> people importing mail from all kinds of systems. This is pushed now.

And sadly, I just pulled it out again.

I realized that I actually don't want my mail tagged based on the
maildir directories I'm using, (they are arbitrarily-named directories
used only to keep the per-directory number of files below about 10
thousand).

So we'll probably need to make this an opt-in feature from the
configuration file.

-Carl

Re: [PATCH] notmuch: Add Maildir directory name as tag name for messages

[Bart Trojanowski]

* Carl Worth <cworth@cworth.org> [091121 15:28]:
> And sadly, I just pulled it out again.
> 
> I realized that I actually don't want my mail tagged based on the
> maildir directories I'm using, (they are arbitrarily-named directories
> used only to keep the per-directory number of files below about 10
> thousand).
> 
> So we'll probably need to make this an opt-in feature from the
> configuration file.

I think notmuch needs something that will add tags based on the
attributes of a message (headers or body), as it imports data from a
maildir.

I am currently considering having procmail deliver to date based
(YYYY-MM) folders and have notmuch determine what tags they should get.

-Bart

-- 
				WebSig: http://www.jukie.net/~bart/sig/

Re: [PATCH] notmuch: Add Maildir directory name as tag name for messages

[Carl Worth]

On Sun, 22 Nov 2009 00:25:43 +0100, Michiel Buddingh' <michiel@michielbuddingh.net> wrote:
> (hi, new here, just subscribed today.  Wanted to reply to Carl's 
>  earlier message I read in the archives, but since I don't have that,
>  I'm replying to Bart's reply to that message)

Hi Michel, welcome to Notmuch!

> Any attempt to match tags up to directories will eventually have 
> to deal with with the fact that tags can't be neatly mapped onto 
> them.  If I remove a directory-tag from a message, does this 
> mean the message is removed from that directory?  What if a 
> message has two directory-tags, does it mean it's present in both
> directories?

Right. We definitely don't want a strong mapping here. I think one
answer could be that the initial import is different and can take the
directory names as a "hint" for initializing tag values. After that,
they are just tags in the database and the user can do whatever they
want.

This would mean that the user will need some other way to apply tags to
future messages that might be delivered to those same directories. And
we've got search-based tagging for that, (as well as manual tagging for
the case where a user was manually filing messages into folders before).

And while developing these search-based tags the user can use a
different tag name. Say there was a directory named "foo", then the user
might experiment with a search-based tag named "foo-search" and explore
the results of things like:

	notmuch search tag:foo and not tag:foo-search
	notmuch search tag:foo-search and not tag:foo

So even if in the end the user comes up with something that could
replace the original tag, it's probably still beneficial to have it
there when starting out.

> At the same time, this kind of interoperability would be highly
> desirable to those of us who access their mail using other  
> clients (webmail, mobile phones, etc.) that expect hierarchical
> ordering.

That kind of thing is going to be "harder".

So far we're trying to stick with the principle that notmuch itself
doesn't mess with the data store. But then, we also want notmuch to be
very scriptable, so someone might write a tool that uses notmuch search
to export a set of hierarchical maildirs based on the tag names. (These
could even just be populated with symlinks like mairix does.) So
something like that could be really useful for integrating.

I'm very much of the opinion that the user shouldn't care at all about
the storage of the actual mail files that notmuch indexes.

> In the mean time, I've made a smaller, hopefully more harmless 
> patch to let 'notmuch new' mark messages stored in a Maildir 'cur'
> folder as 'read' rather than 'unread'.

Can others who have more experience weigh in here? Will this do the
right thing for you? Do mail clients wait to move things into "cur"
after the user has actually read them, or is that just a place where
they are moved when the MUA _receives_ them?

-Carl

Re: [PATCH] notmuch: Add Maildir directory name as tag name for messages

[Michiel Buddingh']

On Sun, 22 Nov 2009 05:04:56 +0100, Carl Worth <cworth@cworth.org> wrote:
> Hi Michel, welcome to Notmuch!

Thanks, and apologies for the accidental base64 encoding.

First things first:

>> In the mean time, I've made a smaller, hopefully more harmless 
>> patch to let 'notmuch new' mark messages stored in a Maildir 'cur'
>> folder as 'read' rather than 'unread'.
> 
> Can others who have more experience weigh in here? Will this do the
> right thing for you? Do mail clients wait to move things into "cur"
> after the user has actually read them, or is that just a place where
> they are moved when the MUA _receives_ them?

You're absolutely right, and I'm a fool (because I _knew_ this, but
forgot).  Maildir stores flags (seen, replied, flagged, trashed,
passed) in file names.

On the positive side, this allows us to map these flags onto tags,
at least for indexing (the patch at the bottom implements this), and,
if I can pry you away from your principles, later for modification
as well.

>> Any attempt to match tags up to directories will eventually have 
>> to deal with with the fact that tags can't be neatly mapped onto 
>> them.  If I remove a directory-tag from a message, does this 
>> mean the message is removed from that directory?  What if a 
>> message has two directory-tags, does it mean it's present in both
>> directories?
> 
> Right. We definitely don't want a strong mapping here.

I propose that the maildir 'storage_type' could make an exception for
standard Maildir flags.  It'll take relatively little effort to
special-case the abovementioned flags, and it'd be a huge boon to
interoperability.

>> At the same time, this kind of interoperability would be highly
>> desirable to those of us who access their mail using other  
>> clients (webmail, mobile phones, etc.) that expect hierarchical
>> ordering.
> 
> That kind of thing is going to be "harder".
> 
> So far we're trying to stick with the principle that notmuch itself
> doesn't mess with the data store.

I respect your desire to stick to that principle.  But I also know 
that purity and simplicity, generally speaking, are unattainable
luxuries for most applications that handle mail.

> But then, we also want notmuch to be
> very scriptable, so someone might write a tool that uses notmuch search
> to export a set of hierarchical maildirs based on the tag names. (These
> could even just be populated with symlinks like mairix does.) So
> something like that could be really useful for integrating.

That is a very interesting idea.  On the other hand, interoperability
with Maildir mail stores is unlikely to be a corner case.  The MTA is
probably going to deliver new mail to a Maildir, procmail understands it,
etc.  I'd feel more comfortable relegating this integration to a 
scripted glue layer if I knew for certain such a glue layer would be
up to the task.
 
> I'm very much of the opinion that the user shouldn't care at all about
> the storage of the actual mail files that notmuch indexes.

The user certainly shouldn't, but I'm not sure that notmuch can remain
as agnostic about the actual storage of messages as planned.

Another thing; notmuch currently indexes by message-id (or SHA-1 hash
if that is not present).  Maildir, on the other hand, uses file names
as stable (when stripped of the parts trailing the colon) unique 
(knock on wood) identifiers.  A Maildir-aware notmuch could incorporate
this to be far more resistant to bulk mail moves done by other clients,
by using filename lookups to avoid accessing and parsing the mail 
files themselves.

I should re-iterate that I'm new to notmuch, and it's obvious that I'm
trying to beat it into becoming something it was never intended to be;
on the other hand, I'd like people to chime in on this.

again via webmail,
Michiel Buddingh'

---
 notmuch-client.h |   10 ++++++
 notmuch-config.c |   33 +++++++++++++++++--
 notmuch-new.c    |   95
+++++++++++++++++++++++++++++++++++++++++++++++------
 3 files changed, 124 insertions(+), 14 deletions(-)

diff --git a/notmuch-client.h b/notmuch-client.h
index ea77686..c39be06 100644
--- a/notmuch-client.h
+++ b/notmuch-client.h
@@ -69,12 +69,16 @@
 #define STRNCMP_LITERAL(var, literal) \
     strncmp ((var), (literal), sizeof (literal) - 1)
 
+enum storage_type { UNSET, NONE, MAILDIR };
+
 typedef void (*add_files_callback_t) (notmuch_message_t *message);
 
 typedef struct {
     int ignore_read_only_directories;
     int saw_read_only_directory;
 
+    enum storage_type storage_type;
+
     int total_files;
     int processed_files;
     int added_messages;
@@ -179,7 +183,13 @@ notmuch_config_set_user_other_email (notmuch_config_t
*config,
 				     const char *other_email[],
 				     size_t length);
 
+enum storage_type
+notmuch_config_get_storage_type (notmuch_config_t *config);
+
 notmuch_bool_t
 debugger_is_active (void);
 
+
+
+
 #endif
diff --git a/notmuch-config.c b/notmuch-config.c
index aaa0372..13bd341 100644
--- a/notmuch-config.c
+++ b/notmuch-config.c
@@ -31,11 +31,13 @@ static const char toplevel_config_comment[] =
 static const char database_config_comment[] =
     " Database configuration\n"
     "\n"
-    " The only value supported here is 'path' which should be the
top-level\n"
+    " The most important value here is 'path' which should be the
top-level\n"
     " directory where your mail currently exists and to where mail will
be\n"
     " delivered in the future. Files should be individual email
messages.\n"
     " Notmuch will store its database within a sub-directory of the
path\n"
-    " configured here named \".notmuch\".\n";
+    " configured here named \".notmuch\".\n"
+    " The other value is 'storage_type', which can currently be set to\n"
+    " 'maildir' or 'none'.\n";
 
 static const char user_config_comment[] =
     " User configuration\n"
@@ -62,6 +64,8 @@ struct _notmuch_config {
     char *user_primary_email;
     char **user_other_email;
     size_t user_other_email_length;
+
+    enum storage_type storage_type;
 };
 
 static int
@@ -193,6 +197,7 @@ notmuch_config_open (void *ctx,
     config->user_primary_email = NULL;
     config->user_other_email = NULL;
     config->user_other_email_length = 0;
+    config->storage_type = UNSET;
 
     if (! g_key_file_load_from_file (config->key_file,
 				     config->filename,
@@ -257,7 +262,7 @@ notmuch_config_open (void *ctx,
 	    talloc_free (email);
 	}
     }
-
+    
     /* When we create a new configuration file here, we  add some
      * comments to help the user understand what can be done. */
     if (is_new) {
@@ -334,6 +339,28 @@ notmuch_config_get_database_path (notmuch_config_t
*config)
     return config->database_path;
 }
 
+enum storage_type
+notmuch_config_get_storage_type (notmuch_config_t * config)
+{
+    char * type;
+
+    if (config->storage_type == UNSET) {
+	type = g_key_file_get_string (config->key_file,
+				      "database", "storage_type", NULL);
+	if (type) {
+	    if (strcasecmp(type, "maildir") == 0) {
+		config->storage_type = MAILDIR;
+	    } else if (strcasecmp(type, "none") == 0) {
+		config->storage_type = NONE;
+	    }
+	}
+    }
+
+    return config->storage_type;
+}
+
+
+
 void
 notmuch_config_set_database_path (notmuch_config_t *config,
 				  const char *database_path)
diff --git a/notmuch-new.c b/notmuch-new.c
index 0dd2784..0918774 100644
--- a/notmuch-new.c
+++ b/notmuch-new.c
@@ -41,12 +41,65 @@ handle_sigint (unused (int sig))
 }
 
 static void
-tag_inbox_and_unread (notmuch_message_t *message)
+tag_as_inbox (notmuch_message_t *message)
 {
     notmuch_message_add_tag (message, "inbox");
+}
+
+static void
+tag_as_unread (notmuch_message_t *message)
+{
     notmuch_message_add_tag (message, "unread");
 }
 
+
+static void
+derive_tags_from_maildir_flags (notmuch_message_t *message, const char *
path) 
+{
+    int seen = FALSE;
+    int end_of_flags = FALSE;
+    size_t l = strlen(path);
+
+    /* Non-experimental message flags start with this */
+    char * i = strstr(path, ":2,");
+    if (i != NULL) {
+
+	i += 3;
+	for (; i < (path + l) && !end_of_flags; i++) {
+	    switch (*i) {
+	    case 'F': /* flagged */
+		notmuch_message_add_tag (message, "flagged");
+		break;
+	    case 'R': /* the message has been replied to */
+		notmuch_message_add_tag (message, "replied");
+		break;
+	    case 'D': 
+		notmuch_message_add_tag (message, "draft");
+		break;
+	    case 'S': /* Indicate a message has been read */
+		notmuch_message_add_tag (message, "seen");
+		seen = TRUE;
+		break;
+	    case 'T': /* Indicates a message has been marked as trash */
+		notmuch_message_add_tag (message, "trashed");
+		break;
+	    case 'P': /* indicates a message has been forwarded */
+		notmuch_message_add_tag (message, "passed");
+		break;
+	    default:
+		end_of_flags = TRUE;
+		break;
+	    }
+	}
+    }
+    
+    if (i == NULL || !seen) { 
+	/* mark messages without any flags, or the seen flag as 'unseen' */
+	notmuch_message_add_tag (message, "unseen");
+    }
+}
+
+
 static void
 add_files_print_progress (add_files_state_t *state)
 {
@@ -104,6 +157,7 @@ static notmuch_status_t
 add_files_recursive (notmuch_database_t *notmuch,
 		     const char *path,
 		     struct stat *st,
+		     int scan_files,
 		     add_files_state_t *state)
 {
     DIR *dir = NULL;
@@ -119,11 +173,9 @@ add_files_recursive (notmuch_database_t *notmuch,
      * directory, (with this being a clear clue from the user to
      * Notmuch that new mail won't be arriving there and we need not
      * look. */
-    if (state->ignore_read_only_directories &&
-	(st->st_mode & S_IWUSR) == 0)
-    {
+    if (((st->st_mode & S_IWUSR) == 0) &&
(state->ignore_read_only_directories)) {
 	state->saw_read_only_directory = TRUE;
-	goto DONE;
+	goto DONE;	    
     }
 
     path_mtime = st->st_mtime;
@@ -173,7 +225,7 @@ add_files_recursive (notmuch_database_t *notmuch,
 	    continue;
 	}
 
-	if (S_ISREG (st->st_mode)) {
+	if (S_ISREG (st->st_mode) && scan_files) {
 	    /* If the file hasn't been modified since the last
 	     * add_files, then we need not look at it. */
 	    if (path_dbtime == 0 || st->st_mtime > path_dbtime) {
@@ -184,7 +236,11 @@ add_files_recursive (notmuch_database_t *notmuch,
 		    /* success */
 		    case NOTMUCH_STATUS_SUCCESS:
 			state->added_messages++;
-			tag_inbox_and_unread (message);
+			tag_as_inbox (message);
+			if (state->storage_type == MAILDIR)
+			    derive_tags_from_maildir_flags (message, next);
+			else 
+			    tag_as_unread (message);
 			break;
 		    /* Non-fatal issues (go on to next file) */
 		    case NOTMUCH_STATUS_DUPLICATE_MESSAGE_ID:
@@ -223,9 +279,25 @@ add_files_recursive (notmuch_database_t *notmuch,
 		}
 	    }
 	} else if (S_ISDIR (st->st_mode)) {
-	    status = add_files_recursive (notmuch, next, st, state);
-	    if (status && ret == NOTMUCH_STATUS_SUCCESS)
-		ret = status;
+	    if (state->storage_type == MAILDIR) {
+		char * leaf = basename(next);
+		
+		if (strcmp(leaf, "cur") == 0) {
+		    status = add_files_recursive (notmuch, next, st, TRUE, state);
+		} else if (strcmp(leaf, "tmp") == 0) {
+		    /* respect Maildir specification and don't touch files in tmp */
+		    continue;
+		} else if (strcmp(leaf, "new") == 0) {
+		    status = add_files_recursive (notmuch, next, st, TRUE, state);
+		} else {
+		    /* Perhaps add in tags? */
+		    status = add_files_recursive (notmuch, next, st, FALSE, state);
+		}
+	    } else {
+		status = add_files_recursive (notmuch, next, st, TRUE, state);
+		if (status && ret == NOTMUCH_STATUS_SUCCESS)
+		    ret = status;
+	    }
 	}
 
 	talloc_free (next);
@@ -292,7 +364,7 @@ add_files (notmuch_database_t *notmuch,
 	timer_is_active = TRUE;
     }
 
-    status = add_files_recursive (notmuch, path, &st, state);
+    status = add_files_recursive (notmuch, path, &st, TRUE, state);
 
     /* Now stop the timer. */
     if (timer_is_active) {
@@ -437,6 +509,7 @@ notmuch_new_command (void *ctx,
     add_files_state.saw_read_only_directory = FALSE;
     add_files_state.processed_files = 0;
     add_files_state.added_messages = 0;
+    add_files_state.storage_type = notmuch_config_get_storage_type
(config);
     gettimeofday (&add_files_state.tv_start, NULL);
 
     ret = add_files (notmuch, db_path, &add_files_state);
-- 
1.6.5.3

Re: [PATCH] notmuch: Add Maildir directory name as tag name for messages

[Dirk-Jan C. Binnema]

>>>>> "Carl" == Carl Worth <cworth@cworth.org> writes:

    Carl> On Sun, 22 Nov 2009 00:25:43 +0100, Michiel Buddingh'
    Carl> <michiel@michielbuddingh.net> wrote:

    Carl> Hi Michel, welcome to Notmuch!

    >> Any attempt to match tags up to directories will eventually have 
    >> to deal with with the fact that tags can't be neatly mapped onto 
    >> them.  If I remove a directory-tag from a message, does this 
    >> mean the message is removed from that directory?  What if a 
    >> message has two directory-tags, does it mean it's present in both
    >> directories?

    Carl> Right. We definitely don't want a strong mapping here. I think one
    Carl> answer could be that the initial import is different and can take the
    Carl> directory names as a "hint" for initializing tag values. After that,
    Carl> they are just tags in the database and the user can do whatever they
    Carl> want.

In 'mu' I took a slightly different approach, by simply allowing for searches
on the path to the message; ie., by adding the path as just another property
of the message, just like subject: or from:.

Now, the trouble with that the path of a message is not very stable -- people
move messages around. In 'mu' this was handled by simply considering the moved
message as a new one, and removing message from the database whose paths do
not point to message anymore. This makes things quite robust against people
moving things around or deleting them. Tags could be connected to the
hopefully unique Maildir file name then. 

    Carl> 	notmuch search tag:foo and not tag:foo-search
    Carl> 	notmuch search tag:foo-search and not tag:foo

    Carl> So even if in the end the user comes up with something that could
    Carl> replace the original tag, it's probably still beneficial to have it
    Carl> there when starting out.

So, an alternative would be something like:
             notmuch search path:inbox

But this requires notmuch to be able to update things when paths change.

Best wishes,
Dirk.

-- 
Dirk-Jan C. Binnema                  Helsinki, Finland
e:djcb@djcbsoftware.nl           w:www.djcbsoftware.nl
pgp: D09C E664 897D 7D39 5047 A178 E96A C7A1 017D DA3C

Re: [PATCH] notmuch: Add Maildir directory name as tag name for messages

[Dirk-Jan C. Binnema]

Hi Michiel,

>>>>> "Michiel" == Michiel Buddingh' <michiel@michielbuddingh.net> writes:

    Michiel> Another thing; notmuch currently indexes by message-id (or SHA-1
    Michiel> hash if that is not present).  Maildir, on the other hand, uses
    Michiel> file names as stable (when stripped of the parts trailing the
    Michiel> colon) unique (knock on wood) identifiers.  A Maildir-aware
    Michiel> notmuch could incorporate this to be far more resistant to bulk
    Michiel> mail moves done by other clients, by using filename lookups to
    Michiel> avoid accessing and parsing the mail files themselves.

I think that is a good idea.
    
    Michiel> I should re-iterate that I'm new to notmuch, and it's obvious
    Michiel> that I'm trying to beat it into becoming something it was never
    Michiel> intended to be; on the other hand, I'd like people to chime in on
    Michiel> this.

I'm new here too, so please be gentle as I learn how things work....
    
    Michiel> +
    Michiel> +static void
    Michiel> +derive_tags_from_maildir_flags (notmuch_message_t *message, const char *
    Michiel> path)

I see you don't handle the "N" -- is that deliberate? Also, a minor addition
may to also allow for '!' instead of ':' as a separator, as that's the
semi-official way to use Maildirs on (V)FAT filesystems (which don't allow for
colons in filenames).


Best wishes,
Dirk.


-- 
Dirk-Jan C. Binnema                  Helsinki, Finland
e:djcb@djcbsoftware.nl           w:www.djcbsoftware.nl
pgp: D09C E664 897D 7D39 5047 A178 E96A C7A1 017D DA3C

Re: [PATCH] notmuch: Add Maildir directory name as tag name for messages

[Carl Worth]

[quick reply to the text without looking at the patch yet]

On Sun, 22 Nov 2009 10:33:53 +0100, Michiel Buddingh' <michiel@michielbuddingh.net> wrote:
> Thanks, and apologies for the accidental base64 encoding.

Thanks for noticing. That would have been a tricky patch to reply to to
comment on.

> On the positive side, this allows us to map these flags onto tags,
> at least for indexing (the patch at the bottom implements this), and,
> if I can pry you away from your principles, later for modification
> as well.

Yes, we might end up doing this. We'll see.

> I respect your desire to stick to that principle.  But I also know 
> that purity and simplicity, generally speaking, are unattainable
> luxuries for most applications that handle mail.

Hehe. Thanks for the entertainment. You sound like someone who's already
got some of the battle scars from trying to write mail-handling
software. One is certainly forced to see an awful lot of awfully strange
things.

> Another thing; notmuch currently indexes by message-id (or SHA-1 hash
> if that is not present).  Maildir, on the other hand, uses file names
> as stable (when stripped of the parts trailing the colon) unique 
> (knock on wood) identifiers.  A Maildir-aware notmuch could incorporate
> this to be far more resistant to bulk mail moves done by other clients,
> by using filename lookups to avoid accessing and parsing the mail 
> files themselves.

I don't think opening a file to read out a message ID will ever be a
bottleneck. But yes, we could take advantage of the unique name if we
insisted that the storage have it.

Personally, I still regularly end up indexing stuff that's not in proper
maildir format, (like just manually copying a mail file "foo" down into
a directory that I know isn't being delivered to by any MDA but that I
want notmuch to index.) Maybe that's just me, because I'm always
bringing up little things for debugging, etc. But it is convenient at
least.

So I think I'd like to be careful to avoid notmuch ever *requiring*
maildir storage. (But yes, it should obviously support maildir well
since it's so common.)

> I should re-iterate that I'm new to notmuch, and it's obvious that I'm
> trying to beat it into becoming something it was never intended to be;

Actually, I don't think that's true at all. Notmuch is definitely
intended to become a lot more than it is right now. And if it's not
making it easy for you to deal with mail the way you'd like to, then
we definitely do want to look into expanding notmuch to be able to
address that.

> on the other hand, I'd like people to chime in on this.

I've got the patch tagged to give a closer look at it later.

Thanks again,

-Carl

Re: [PATCH] notmuch: Add Maildir directory name as tag name for messages

[Michiel Buddingh']

Carl Worth <cworth@cworth.org> wrote:

> > A Maildir-aware notmuch could incorporate this to be far more
> > resistant to bulk mail moves done by other clients, by using
> > filename lookups to avoid accessing and parsing the mail
> > files themselves.

> I don't think opening a file to read out a message ID will ever be
> a bottleneck. But yes, we could take advantage of the unique name
> if we insisted that the storage have it.

I'm not so sure.  On traditional unix-like filesystems, every file 
access is another potential disk seek.  

People use a lot of different strategies for keeping their mail 
accessible and performant; you yourself, I gather, use non-semantic
directories mainly intended to keep the number of files per directory
low; others store their mail per month, or per year.

I might choose to move all my mail from 2008 from my INBOX to the 
folder Archive.2008; such moves may change the paths of thousands 
of messages at a time.

> Personally, I still regularly end up indexing stuff that's not 
> in proper maildir format, (like just manually copying a mail
> file "foo" down into a directory that I know isn't being delivered 
> to by any MDA but that I want notmuch to index.) Maybe that's just
> me, because I'm always bringing up little things for debugging, 
> etc. But it is convenient at least.

Oh true.  And it occurs to me that notmuch is a quite sensible 
companion tool to MH users (if they're still around in any numbers)

> Actually, I don't think that's true at all. Notmuch is definitely
> intended to become a lot more than it is right now. And if it's not
> making it easy for you to deal with mail the way you'd like to, then
> we definitely do want to look into expanding notmuch to be able to
> address that.

Thanks for your consideration; on the other hand, I do think that it
is a good idea not to make matters more complex than they need to be,
so I can certainly sympathise with the principles you've set for this
project.

regards,
Michiel Buddingh'

Re: [PATCH] notmuch: Add Maildir directory name as tag name for messages

[Michiel Buddingh']

Dirk-Jan C. Binnema <djcb.bulk@gmail.com> wrote:
>     Michiel> +
>     Michiel> +static void
>     Michiel> +derive_tags_from_maildir_flags (notmuch_message_t 
>     Michiel> *message, const char *
>     Michiel> path)
>
> I see you don't handle the "N" -- is that deliberate? Also, a 
> minor addition may to also allow for '!' instead of ':' as a 
> separator, as that's the semi-official way to use Maildirs on 
> (V)FAT filesystems (which don't allow for colons in filenames).

Not deliberate.  Simply unaware of the "N" flag, nor aware of 
practices for storing Maildirs on (V)FAT.

I've used only this file as a reference.
http://cr.yp.to/proto/maildir.html

mvg,
Michiel Buddingh'

Re: [PATCH] notmuch: Add Maildir directory name as tag name for messages

[Dirk-Jan C. Binnema]

Hi Michiel,

>>>>> "MB" == Michiel Buddingh' <michiel@michielbuddingh.net> writes:

    MB> Dirk-Jan C. Binnema <djcb.bulk@gmail.com> wrote:
    Michiel> +
    Michiel> +static void
    Michiel> +derive_tags_from_maildir_flags (notmuch_message_t 
    Michiel> *message, const char *
    Michiel> path)
    >> 
    >> I see you don't handle the "N" -- is that deliberate? Also, a 
    >> minor addition may to also allow for '!' instead of ':' as a 
    >> separator, as that's the semi-official way to use Maildirs on 
    >> (V)FAT filesystems (which don't allow for colons in filenames).

    MB> Not deliberate.  Simply unaware of the "N" flag, nor aware of 
    MB> practices for storing Maildirs on (V)FAT.

Ah, you are right. In my code,
 http://gitorious.org/mu/mainline/blobs/master/msg/mu-msg-flags.c
I handled it because apparently some programs where using that. But I guess
it's better to stick to DJB's Maildir spec for now...

Regarding the VFAT-limitations, see eg. the Note in:
  http://docs.python.org/lib/mailbox-maildir.html
anyway, a minor point.


Best wishes,
Dirk.

-- 
Dirk-Jan C. Binnema                  Helsinki, Finland
e:djcb@djcbsoftware.nl           w:www.djcbsoftware.nl
pgp: D09C E664 897D 7D39 5047 A178 E96A C7A1 017D DA3C

Re: [PATCH] notmuch: Add Maildir directory name as tag name for messages

[Carl Worth]

On Sun, 22 Nov 2009 10:33:53 +0100, Michiel Buddingh' <michiel@michielbuddingh.net> wrote:
> On the positive side, this allows us to map these flags onto tags,
> at least for indexing (the patch at the bottom implements this), and,
> if I can pry you away from your principles, later for modification
> as well.

Hi Michiel,

I'm finally getting around to reviewing this patch. I think the
functionality of respecting these maildir flags for initial import is
going to be really important. So thanks for looking at this!

Some comments on the patches below.

> +enum storage_type
> +notmuch_config_get_storage_type (notmuch_config_t *config);
> +
>  notmuch_bool_t
>  debugger_is_active (void);
>  
> +
> +
> +
>  #endif

[nit] Try to look out for introducing excess whitespace like that.

> +    " The other value is 'storage_type', which can currently be set to\n"
> +    " 'maildir' or 'none'.\n";

This part of the patch I don't like. I've got a mail collection spanning
over a decade, and it's seen a lot of strange things. Most of my mail is
in maildir format, but not quite all of it. And I actually like the
ability to just shove random new messages into the mail store manually
without having to create a maildir name for it.

So I don't think a global configuration makes sense here. Meanwhile,
it's really easy to detect the presence of a maildir. Whenever we see
child directories of "cur", "new", and "tmp" then we should turn on the
processing of maildir flags for when processing mail in "cur" and "new".

> @@ -257,7 +262,7 @@ notmuch_config_open (void *ctx,
>  	    talloc_free (email);
>  	}
>      }
> -
> +    
>      /* When we create a new configuration file here, we  add some
>       * comments to help the user understand what can be done. */
>      if (is_new) {

[nit] Trailing whitespace inserted there as well.

Hmm... I was going to say that git ships with a pre-commit hook you can
turn on that checks for trailing whitespace and aborts the commit if
it's present. But it looks like the currently shipping pre-commit.sample
hook doesn't do this anymore. So I'll have to find out what the current
recommended way is to get this behavior. Does anybody here know?

> +	i += 3;
> +	for (; i < (path + l) && !end_of_flags; i++) {
> +	    switch (*i) {
> +	    case 'F': /* flagged */
> +		notmuch_message_add_tag (message, "flagged");
> +		break;
> +	    case 'R': /* the message has been replied to */
> +		notmuch_message_add_tag (message, "replied");
> +		break;
> +	    case 'D': 
> +		notmuch_message_add_tag (message, "draft");
> +		break;
> +	    case 'S': /* Indicate a message has been read */
> +		notmuch_message_add_tag (message, "seen");
> +		seen = TRUE;
> +		break;
> +	    case 'T': /* Indicates a message has been marked as trash */
> +		notmuch_message_add_tag (message, "trashed");
> +		break;
> +	    case 'P': /* indicates a message has been forwarded */
> +		notmuch_message_add_tag (message, "passed");
> +		break;
> +	    default:
> +		end_of_flags = TRUE;
> +		break;
> +	    }
> +	}
> +    }
> +    
> +    if (i == NULL || !seen) { 
> +	/* mark messages without any flags, or the seen flag as 'unseen' */
> +	notmuch_message_add_tag (message, "unseen");
> +    }

OK, now we're into the meat of things. Clearly, you're directly
supporting the documented flags of maildir. But we need to do a few
things differently here. Most importantly, notmuch is already using an
"unread" tag, so maildir's S flag should map that *that* rather than
adding new "unseen" and "seen" flags. So messages with the S flag would
not get the "unread" tag and messages without S would get the "unread"
tag.

The "flagged" and "replied" tags seem reasonable enough. But for
"trashed" and "passed" I think I'd rather see the tag names as "deleted"
and "forwarded". (Since I can imagine adding commands to notmuch for
"delete" and "forward" but not for "trash" nor "pass").

Finally, all of the new behavior here needs to be documented in both
notmuch.1 and notmuch.c where the current behavior with respect to
"inbox" and "unread" is already documented.

Oh, and setting the "inbox" tag correctly here based on the maildir tags
is the final and most important thing. It looks like that's missing from
the above. So, a missing "S" flag should map to adding both the "inbox"
and "unread" tags.

> (state->ignore_read_only_directories)) {
>  	state->saw_read_only_directory = TRUE;
> -	goto DONE;
> +	goto DONE;	    
>      }

Trailing whitespace strikes again! :-)


> +	    if (state->storage_type == MAILDIR) {
> +		char * leaf = basename(next);

You could save the basename call by examining the leaf name when it is
available as a standalone string up in the caller.

So this patch is close, but needs a few fixes.

Thanks again,

-Carl

Re: [PATCH] notmuch: Add Maildir directory name as tag name for messages

[Michiel Buddingh']

Carl Worth <cworth@cworth.org> wrote:
> > +    " The other value is 'storage_type', which can currently be set to\n"
> > +    " 'maildir' or 'none'.\n";
>
> This part of the patch I don't like. I've got a mail collection spanning
> over a decade, and it's seen a lot of strange things. Most of my mail is
> in maildir format, but not quite all of it. And I actually like the
> ability to just shove random new messages into the mail store manually
> without having to create a maildir name for it.
>
> So I don't think a global configuration makes sense here. Meanwhile,
> it's really easy to detect the presence of a maildir. Whenever we see
> child directories of "cur", "new", and "tmp" then we should turn on the
> processing of maildir flags for when processing mail in "cur" and "new".

  I considered that approach; ideally, we could test for the presence of
all three of cur, tmp and new--but this is rather messy to do in the
current treewalk structure.  Taking any one of them as proof positive of
a Maildir might lead to unpleasant surprises--it's not all that incon-
ceivable for someone to name a mail folder 'tmp'.

  There's another matter; Some mail stores will place (large) index files
in folder roots, i.e. one level above cur/, tmp/ and new/.  Looking
at the ones dovecot (an IMAP server) uses, I can make out a from header,
a subject header, and a message-id, as plaintext in the first 100k or
so.  It's not all that inconceivable that notmuch might register it as
a 'real' email, with unpleasant consequences for the index.

  I've seen some patches fly by that add support for multiple mail
stores.  Turning on Maildir support on a per-directory basis might
resolve that problem while still supporting heterogenous mail archives
to some degree.  I am not convinced we can do the right thing
automatically without causing some grief to a subset of users.

> > @@ -257,7 +262,7 @@ notmuch_config_open (void *ctx,
> >  	    talloc_free (email);
> >  	}
> >      }
> > -
> > +    
> >      /* When we create a new configuration file here, we  add some
> >       * comments to help the user understand what can be done. */
> >      if (is_new) {
>
> [nit] Trailing whitespace inserted there as well.

> Hmm... I was going to say that git ships with a pre-commit hook you can
> turn on that checks for trailing whitespace and aborts the commit if
> it's present. But it looks like the currently shipping pre-commit.sample
> hook doesn't do this anymore.

Haven't tested it, but it seems you can put

[core]
    whitespace = trailing-space,space-before-tab

into your ~/.gitconfig now.  I've also set emacs to mark trailing
whitespace with big red markers.

> OK, now we're into the meat of things. Clearly, you're directly
> supporting the documented flags of maildir. But we need to do a few
> things differently here. Most importantly, notmuch is already using an
> "unread" tag, so maildir's S flag should map that *that* rather than
> adding new "unseen" and "seen" flags. So messages with the S flag would
> not get the "unread" tag and messages without S would get the "unread"
> tag.

When writing the patch, I assumed there might be a minor (but important)
distinction between marking a mail 'seen' (i.e. the MUA storing the fact
that the file has been visited) and 'read' (i.e. the user marking the
contents of a mail as being read and understood).  As I found out later,
notmuch's interpretation of 'read' and 'unread' is the former, so there
is no distinction.

> The "flagged" and "replied" tags seem reasonable enough. But for
> "trashed" and "passed" I think I'd rather see the tag names as "deleted"
> and "forwarded". (Since I can imagine adding commands to notmuch for
> "delete" and "forward" but not for "trash" nor "pass").

Fair enough.

> Oh, and setting the "inbox" tag correctly here based on the maildir tags
> is the final and most important thing. It looks like that's missing from
> the above. So, a missing "S" flag should map to adding both the "inbox"
> and "unread" tags.

Makes sense, will do.

> > +	    if (state->storage_type == MAILDIR) {
> > +		char * leaf = basename(next);
>
> You could save the basename call by examining the leaf name when it is
> available as a standalone string up in the caller.

Which would require testing with S_ISDIR twice, which is uglier, but
essentially free, so I'll grant it's the better thing to do.

> So this patch is close, but needs a few fixes.

I'll be happy to implement them, although I'd like for others to chime
in on the configure-as-Maildir vs. autodetect-Maildir issue.  And thanks
for your patience in working through my patch.

-- 
Michiel Buddingh'

Re: [PATCH] notmuch: Add Maildir directory name as tag name for messages

[Ingmar Vanhassel]

Excerpts from Michiel Buddingh''s message of Thu Nov 26 22:12:02 +0100 2009:
> Haven't tested it, but it seems you can put
> 
> [core]
>     whitespace = trailing-space,space-before-tab
> 
> into your ~/.gitconfig now.  I've also set emacs to mark trailing
> whitespace with big red markers.

I think it should be in notmuch/.gitattributes, which doesn't relies on
new contributors to set that up. It also doesn't force this on for every
git repo one has.


-- 
Exherbo KDE, X.org maintainer

Re: [PATCH] notmuch: Add Maildir directory name as tag name for messages

[Jan Janak]

On 26-11 22:53, Ingmar Vanhassel wrote:
> Excerpts from Michiel Buddingh''s message of Thu Nov 26 22:12:02 +0100 2009:
> > Haven't tested it, but it seems you can put
> > 
> > [core]
> >     whitespace = trailing-space,space-before-tab
> > 
> > into your ~/.gitconfig now.  I've also set emacs to mark trailing
> > whitespace with big red markers.
> 
> I think it should be in notmuch/.gitattributes, which doesn't relies on
> new contributors to set that up. It also doesn't force this on for every
> git repo one has.

This enforces the restrictions also on people's local commits, even those they
may never want to submit for inclusion into the official repository.

Asking people to follow certain rules when submitting patches for inclusion is
sensible, but I don't think notmuch should enforce this on all local commits
with notmuch/.gitattributes.

  -- Jan

Re: [PATCH] notmuch: Add Maildir directory name as tag name for messages

[Carl Worth]

On Thu, 26 Nov 2009 22:12:02 +0100, Michiel Buddingh' <michiel@michielbuddingh.net> wrote:
> Carl Worth <cworth@cworth.org> wrote:
>   I considered that approach; ideally, we could test for the presence of
> all three of cur, tmp and new--but this is rather messy to do in the
> current treewalk structure.  Taking any one of them as proof positive of
> a Maildir might lead to unpleasant surprises--it's not all that incon-
> ceivable for someone to name a mail folder 'tmp'.

The auto-detection is just three additional stats (at most) for each
directory, right? That seems cheap enough to me.

>   There's another matter; Some mail stores will place (large) index files
> in folder roots, i.e. one level above cur/, tmp/ and new/.  Looking
> at the ones dovecot (an IMAP server) uses, I can make out a from header,
> a subject header, and a message-id, as plaintext in the first 100k or
> so.  It's not all that inconceivable that notmuch might register it as
> a 'real' email, with unpleasant consequences for the index.

That seems orthogonal to me. Would the dovecot index files be easy to
skip with a pattern-based blacklist?

>   I've seen some patches fly by that add support for multiple mail
> stores.  Turning on Maildir support on a per-directory basis might
> resolve that problem while still supporting heterogenous mail archives
> to some degree.  I am not convinced we can do the right thing
> automatically without causing some grief to a subset of users.

With sup, I had the opposite extreme compared to current notmuch. Every
maildir in the hierarchy had to be configured independently. That was a
lot of pain, and is precisely why notmuch started out by simply taking a
single top-level directory.

> Haven't tested it, but it seems you can put
> 
> [core]
>     whitespace = trailing-space,space-before-tab

Yes. According to the documentation of git-config, those two values are
the default. But the documentation also only says that these will make
"git diff" display the undesired whitespace in red, and "git apply
--whitespace=error" refuse to apply.

I can't find a builtin way to make "git commit" complain, or I would
recommend that.

> I'll be happy to implement them, although I'd like for others to chime
> in on the configure-as-Maildir vs. autodetect-Maildir issue.  And thanks
> for your patience in working through my patch.

No problem at all. I'll look forward to the next version of the
patch. Consider mine as a vote for autodetection of maildir rather than
configuration.

-Carl

Re: [PATCH] notmuch: Add Maildir directory name as tag name for messages

[Michiel Buddingh']

First of all, apologies for taking so long to get back to this.

On Fri, 27 Nov 2009, Carl Worth <cworth@cworth.org> wrote:
> The auto-detection is just three additional stats (at most) for each
> directory, right? That seems cheap enough to me.

If that's cheap enough, then I won't disagree with auto-detection.  
Jan Janak's patch seems to take most of the disk access cost out of it,
in any case.

> That seems orthogonal to me. Would the dovecot index files be easy to
> skip with a pattern-based blacklist?

Yes, and that's a much more elegant solution.

> > I'll be happy to implement them, although I'd like for others to
> > chime in on the configure-as-Maildir vs. autodetect-Maildir issue.
> > And thanks for your patience in working through my patch.

I didn't mean to call a vote--rather to solicit the opinions of others
with possibly even more exotic mail storage configurations.

A new patch is attached.  Apologies for the rather verbose Maildir
handling logic, but I couldn't find a way to minimize the calls to
is_maildir that was both neat and readable.

-- 
Michiel

---
 notmuch-client.h |    1 +
 notmuch-new.c    |   93 +++++++++++++++++++++++++++++++++++++++++++++++++++--
 2 files changed, 90 insertions(+), 4 deletions(-)

diff --git a/notmuch-client.h b/notmuch-client.h
index 50a30fe..7bc84a1 100644
--- a/notmuch-client.h
+++ b/notmuch-client.h
@@ -77,6 +77,7 @@ typedef struct {
     int saw_read_only_directory;
     int output_is_a_tty;
     int verbose;
+    int tag_maildir;
 
     int total_files;
     int processed_files;
diff --git a/notmuch-new.c b/notmuch-new.c
index 9d20616..8742ab4 100644
--- a/notmuch-new.c
+++ b/notmuch-new.c
@@ -109,6 +109,60 @@ is_maildir (struct dirent **entries, int count)
     return 0;
 }
 
+/* Tag new mail according to its Maildir attribute flags.
+ *
+ * Test if the mail file's filename contains any of the
+ * standard Maildir attributes, and translate these to
+ * the corresponding standard notmuch tags.
+ *
+ * If the message is not marked as 'seen', or if no
+ * flags are present, tag as 'inbox, unread'.
+ */
+static void
+derive_tags_from_maildir_flags (notmuch_message_t *message,
+				const char * path)
+{
+    int seen = FALSE;
+    int end_of_flags = FALSE;
+    size_t l = strlen(path);
+
+    /* Non-experimental message flags start with this */
+    char * i = strstr(path, ":2,");
+    i = (i) ? i : strstr(path, "!2,"); /* This format is used on VFAT */
+    if (i != NULL) {
+	i += 3;
+	for (; i < (path + l) && !end_of_flags; i++) {
+	    switch (*i) {
+	    case 'F' :
+		notmuch_message_add_tag (message, "flagged");
+		break;
+	    case 'R': /* replied */
+		notmuch_message_add_tag (message, "answered");
+		break;
+	    case 'D':
+		notmuch_message_add_tag (message, "draft");
+		break;
+	    case 'S': /* seen */
+		seen = TRUE;
+		break;
+	    case 'T': /* trashed */
+		notmuch_message_add_tag (message, "deleted");
+		break;
+	    case 'P': /* passed */
+		notmuch_message_add_tag (message, "forwarded");
+		break;
+	    default:
+		end_of_flags = TRUE;
+		break;
+	    }
+	}
+    }
+
+    if (i == NULL || !seen) {
+	tag_inbox_and_unread (message);
+    }
+}
+
 /* Examine 'path' recursively as follows:
  *
  *   o Ask the filesystem for the mtime of 'path' (path_mtime)
@@ -142,6 +196,7 @@ add_files_recursive (notmuch_database_t *notmuch,
     notmuch_status_t status, ret = NOTMUCH_STATUS_SUCCESS;
     notmuch_message_t *message = NULL;
     struct dirent **namelist = NULL;
+    int maildir_detected = -1; /* -1 = unset */
     int num_entries;
 
     /* If we're told to, we bail out on encountering a read-only
@@ -189,13 +244,37 @@ add_files_recursive (notmuch_database_t *notmuch,
 	if (strcmp (entry->d_name, ".") == 0 ||
 	    strcmp (entry->d_name, "..") == 0 ||
 	    (entry->d_type == DT_DIR &&
-	     (strcmp (entry->d_name, "tmp") == 0) &&
-	     is_maildir (namelist, num_entries)) ||
-	    strcmp (entry->d_name, ".notmuch") ==0)
+	     strcmp (entry->d_name, ".notmuch") == 0))
 	{
 	    continue;
 	}
 
+
+	/* If this directory is a Maildir folder, we need to
+	 * ignore any subdirectories marked tmp/, and scan for
+	 * Maildir attributes on messages contained in the sub-
+	 * directories 'new' and 'cur'. */
+	if (maildir_detected != 0 &&
+	    entry->d_type == DT_DIR &&
+	    ((strcmp (entry->d_name, "tmp") == 0) ||
+	     (strcmp (entry->d_name, "new") == 0) ||
+	     (strcmp (entry->d_name, "cur") == 0))) {
+
+	    /* is_maildir scans the entire directory.  No need to
+	       do this more than once, if at all */
+	    if (maildir_detected == -1) {
+		maildir_detected = is_maildir (namelist, num_entries);
+	    }
+
+	    if (maildir_detected == 1) {
+		if (strcmp (entry->d_name, "tmp") == 0) {
+		    continue;
+		} else {
+		    state->tag_maildir = TRUE;
+		}
+	    }
+	}
+
 	next = talloc_asprintf (notmuch, "%s/%s", path, entry->d_name);
 
 	if (stat (next, st)) {
@@ -240,7 +319,12 @@ add_files_recursive (notmuch_database_t *notmuch,
 		    /* success */
 		    case NOTMUCH_STATUS_SUCCESS:
 			state->added_messages++;
-			tag_inbox_and_unread (message);
+			if (state->tag_maildir) {
+			    derive_tags_from_maildir_flags (message,
+							    entry->d_name);
+			} else {
+			    tag_inbox_and_unread (message);
+			}
 			break;
 		    /* Non-fatal issues (go on to next file) */
 		    case NOTMUCH_STATUS_DUPLICATE_MESSAGE_ID:
@@ -282,6 +366,7 @@ add_files_recursive (notmuch_database_t *notmuch,
 	    status = add_files_recursive (notmuch, next, st, state);
 	    if (status && ret == NOTMUCH_STATUS_SUCCESS)
 		ret = status;
+	    state->tag_maildir = FALSE;
 	}
 
 	talloc_free (next);
-- 
1.6.5.4

Re: [PATCH] notmuch: Respect maildir message flags

[Tim Stoakes]

Michiel Buddingh'(michiel@michielbuddingh.net)@061209-20:55:
>
...
> A new patch is attached.  Apologies for the rather verbose Maildir
> handling logic, but I couldn't find a way to minimize the calls to
> is_maildir that was both neat and readable.


Hi notmuch-ers,

My apologies for dredging up an old thread. I don't want to restart the
religious war over whether notmuch should respect Maildir flags -
suffice to say that *I* want that, and the patch posted by Michiel
seemed to be the best way to make that happen.

Since it no longer applies cleanly, I've ported it forward to
79d3f9773c58d6fd7113871362687d8cfc0b1a59, to save someone else the
trouble. It works for me, but that's all the testing I've done.

Tim

-- 
Tim Stoakes



---
 notmuch-new.c |   86 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++-
 1 files changed, 85 insertions(+), 1 deletions(-)

diff --git a/notmuch-new.c b/notmuch-new.c
index f25c71f..3264653 100644
--- a/notmuch-new.c
+++ b/notmuch-new.c
@@ -39,6 +39,7 @@ typedef struct {
     int total_files;
     int processed_files;
     int added_messages;
+    int tag_maildir;
     struct timeval tv_start;
 
     _filename_list_t *removed_files;
@@ -169,6 +170,60 @@ _entries_resemble_maildir (struct dirent **entries, int count)
     return 0;
 }
 
+/* Tag new mail according to its Maildir attribute flags.
+ *
+ * Test if the mail file's filename contains any of the
+ * standard Maildir attributes, and translate these to
+ * the corresponding standard notmuch tags.
+ *
+ * If the message is not marked as 'seen', or if no
+ * flags are present, tag as 'inbox, unread'.
+ */
+static void
+derive_tags_from_maildir_flags (notmuch_message_t *message,
+				const char * path)
+{
+    int seen = FALSE;
+    int end_of_flags = FALSE;
+    size_t l = strlen(path);
+
+    /* Non-experimental message flags start with this */
+    char * i = strstr(path, ":2,");
+    i = (i) ? i : strstr(path, "!2,"); /* This format is used on VFAT */
+    if (i != NULL) {
+	i += 3;
+	for (; i < (path + l) && !end_of_flags; i++) {
+	    switch (*i) {
+	    case 'F' :
+		notmuch_message_add_tag (message, "flagged");
+		break;
+	    case 'R': /* replied */
+		notmuch_message_add_tag (message, "answered");
+		break;
+	    case 'D':
+		notmuch_message_add_tag (message, "draft");
+		break;
+	    case 'S': /* seen */
+		seen = TRUE;
+		break;
+	    case 'T': /* trashed */
+		notmuch_message_add_tag (message, "deleted");
+		break;
+	    case 'P': /* passed */
+		notmuch_message_add_tag (message, "forwarded");
+		break;
+	    default:
+		end_of_flags = TRUE;
+		break;
+	    }
+	}
+    }
+
+    if (i == NULL || !seen) {
+	tag_inbox_and_unread (message);
+    }
+}
+
 /* Examine 'path' recursively as follows:
  *
  *   o Ask the filesystem for the mtime of 'path' (fs_mtime)
@@ -222,6 +277,7 @@ add_files_recursive (notmuch_database_t *notmuch,
     notmuch_filenames_t *db_subdirs = NULL;
     struct stat st;
     notmuch_bool_t is_maildir, new_directory;
+    int maildir_detected = -1;
 
     if (stat (path, &st)) {
 	fprintf (stderr, "Error reading directory %s: %s\n",
@@ -301,6 +357,28 @@ add_files_recursive (notmuch_database_t *notmuch,
 	    continue;
 	}
 
+	/* If this directory is a Maildir folder, we need to
+	 * ignore any subdirectories marked tmp/, and scan for
+	 * Maildir attributes on messages contained in the sub-
+	 * directories 'new' and 'cur'. */
+	if (maildir_detected != 0 &&
+	    entry->d_type == DT_DIR &&
+	    ((strcmp (entry->d_name, "tmp") == 0) ||
+	     (strcmp (entry->d_name, "new") == 0) ||
+	     (strcmp (entry->d_name, "cur") == 0))) {
+
+    if (maildir_detected == -1) {
+      maildir_detected = _entries_resemble_maildir(fs_entries, num_fs_entries);
+    }
+    if (maildir_detected == 1) {
+      if (strcmp (entry->d_name, "tmp") == 0) {
+        continue;
+      } else {
+        state->tag_maildir = TRUE;
+      }
+    }
+  }
+
 	next = talloc_asprintf (notmuch, "%s/%s", path, entry->d_name);
 	status = add_files_recursive (notmuch, next, state);
 	if (status && ret == NOTMUCH_STATUS_SUCCESS)
@@ -412,7 +490,12 @@ add_files_recursive (notmuch_database_t *notmuch,
 	/* success */
 	case NOTMUCH_STATUS_SUCCESS:
 	    state->added_messages++;
-	    tag_inbox_and_unread (message);
+			if (state->tag_maildir) {
+			    derive_tags_from_maildir_flags (message,
+							    entry->d_name);
+			} else {
+			    tag_inbox_and_unread (message);
+			}
 	    break;
 	/* Non-fatal issues (go on to next file) */
 	case NOTMUCH_STATUS_DUPLICATE_MESSAGE_ID:
@@ -482,6 +565,7 @@ add_files_recursive (notmuch_database_t *notmuch,
 	status = notmuch_directory_set_mtime (directory, fs_mtime);
 	if (status && ret == NOTMUCH_STATUS_SUCCESS)
 	    ret = status;
+	    state->tag_maildir = FALSE;
     }
 
   DONE:
-- 
1.6.6.1

Re: [PATCH] notmuch: Respect maildir message flags

[Stewart Smith]

On Wed, Feb 10, 2010 at 01:43:39PM +1030, Tim Stoakes wrote:
> My apologies for dredging up an old thread. I don't want to restart the
> religious war over whether notmuch should respect Maildir flags -
> suffice to say that *I* want that, and the patch posted by Michiel
> seemed to be the best way to make that happen.

I want this too :)

I also found a bug....

> @@ -301,6 +357,28 @@ add_files_recursive (notmuch_database_t *notmuch,
>  	    continue;
>  	}
>  
> +	/* If this directory is a Maildir folder, we need to
> +	 * ignore any subdirectories marked tmp/, and scan for
> +	 * Maildir attributes on messages contained in the sub-
> +	 * directories 'new' and 'cur'. */
> +	if (maildir_detected != 0 &&
> +	    entry->d_type == DT_DIR &&
> +	    ((strcmp (entry->d_name, "tmp") == 0) ||
> +	     (strcmp (entry->d_name, "new") == 0) ||
> +	     (strcmp (entry->d_name, "cur") == 0))) {

should be
        (entry->d_type == DT_DIR || entry->d_type == DT_UNKNOWN) &&

as not everywhere is going to give you d_type (e.g. my machine).


(took me a while to find/figure that out :) 
-- 
Stewart Smith

[PATCH] notmuch: Respect maildir message flags

[Stewart Smith]

New patch that does it. Pretty much same as the old one, just with
that one bug I mentioned fixed. This is what I've currently used to
import my Maildir. I'm now happy :)

diff --git a/notmuch-new.c b/notmuch-new.c
index f25c71f..43371a3 100644
--- a/notmuch-new.c
+++ b/notmuch-new.c
@@ -39,6 +39,7 @@ typedef struct {
     int total_files;
     int processed_files;
     int added_messages;
+    int tag_maildir;
     struct timeval tv_start;
 
     _filename_list_t *removed_files;
@@ -169,6 +170,60 @@ _entries_resemble_maildir (struct dirent **entries, int count)
     return 0;
 }
 
+/* Tag new mail according to its Maildir attribute flags.
+ *
+ * Test if the mail file's filename contains any of the
+ * standard Maildir attributes, and translate these to
+ * the corresponding standard notmuch tags.
+ *
+ * If the message is not marked as 'seen', or if no
+ * flags are present, tag as 'inbox, unread'.
+ */
+static void
+derive_tags_from_maildir_flags (notmuch_message_t *message,
+				const char * path)
+{
+    int seen = FALSE;
+    int end_of_flags = FALSE;
+    size_t l = strlen(path);
+
+    /* Non-experimental message flags start with this */
+    char * i = strstr(path, ":2,");
+    i = (i) ? i : strstr(path, "!2,"); /* This format is used on VFAT */
+    if (i != NULL) {
+	i += 3;
+	for (; i < (path + l) && !end_of_flags; i++) {
+	    switch (*i) {
+	    case 'F' :
+		notmuch_message_add_tag (message, "flagged");
+		break;
+	    case 'R': /* replied */
+		notmuch_message_add_tag (message, "answered");
+		break;
+	    case 'D':
+		notmuch_message_add_tag (message, "draft");
+		break;
+	    case 'S': /* seen */
+		seen = TRUE;
+		break;
+	    case 'T': /* trashed */
+		notmuch_message_add_tag (message, "deleted");
+		break;
+	    case 'P': /* passed */
+		notmuch_message_add_tag (message, "forwarded");
+		break;
+	    default:
+		end_of_flags = TRUE;
+		break;
+	    }
+	}
+    }
+
+    if (i == NULL || !seen) {
+	tag_inbox_and_unread (message);
+    }
+}
+
 /* Examine 'path' recursively as follows:
  *
  *   o Ask the filesystem for the mtime of 'path' (fs_mtime)
@@ -222,6 +277,7 @@ add_files_recursive (notmuch_database_t *notmuch,
     notmuch_filenames_t *db_subdirs = NULL;
     struct stat st;
     notmuch_bool_t is_maildir, new_directory;
+    int maildir_detected = -1;
 
     if (stat (path, &st)) {
 	fprintf (stderr, "Error reading directory %s: %s\n",
@@ -301,6 +357,28 @@ add_files_recursive (notmuch_database_t *notmuch,
 	    continue;
 	}
 
+	/* If this directory is a Maildir folder, we need to
+	 * ignore any subdirectories marked tmp/, and scan for
+	 * Maildir attributes on messages contained in the sub-
+	 * directories 'new' and 'cur'. */
+	if (maildir_detected != 0 &&
+	    (entry->d_type == DT_DIR || entry->d_type == DT_UNKNOWN) &&
+	    ((strcmp (entry->d_name, "tmp") == 0) ||
+	     (strcmp (entry->d_name, "new") == 0) ||
+	     (strcmp (entry->d_name, "cur") == 0))) {
+
+    if (maildir_detected == -1) {
+      maildir_detected = _entries_resemble_maildir(fs_entries, num_fs_entries);
+    }
+    if (maildir_detected == 1) {
+      if (strcmp (entry->d_name, "tmp") == 0) {
+        continue;
+      } else {
+        state->tag_maildir = TRUE;
+      }
+    }
+  }
+
 	next = talloc_asprintf (notmuch, "%s/%s", path, entry->d_name);
 	status = add_files_recursive (notmuch, next, state);
 	if (status && ret == NOTMUCH_STATUS_SUCCESS)
@@ -412,7 +490,12 @@ add_files_recursive (notmuch_database_t *notmuch,
 	/* success */
 	case NOTMUCH_STATUS_SUCCESS:
 	    state->added_messages++;
-	    tag_inbox_and_unread (message);
+			if (state->tag_maildir) {
+			    derive_tags_from_maildir_flags (message,
+							    entry->d_name);
+			} else {
+			    tag_inbox_and_unread (message);
+			}
 	    break;
 	/* Non-fatal issues (go on to next file) */
 	case NOTMUCH_STATUS_DUPLICATE_MESSAGE_ID:


-- 
Stewart Smith

Re: [PATCH] notmuch: Respect maildir message flags

[martin f krafft]

[-- Attachment #1: Type: text/plain, Size: 888 bytes --]

also sprach Stewart Smith <stewart@flamingspork.com> [2010.02.16.1458 +1300]:
> +	    case 'R': /* replied */
> +		notmuch_message_add_tag (message, "answered");
> +		break;

'r' means replied, not 'answered'.

> +	    case 'T': /* trashed */
> +		notmuch_message_add_tag (message, "deleted");
> +		break;

Same. trashed and deleted are not the same thing.

I don't want to get into an argument over this, because I think this
already exposes a problem: you are putting into global namespace
something not everyone might want, or agree with.

Why not use 'maildirflags::replied' instead? People can always map
that to something in the global namespace.

-- 
martin | http://madduck.net/ | http://two.sentenc.es/
 
"geld ist das brecheisen der macht."
                                                 - friedrich nietzsche
 
spamtraps: madduck.bogus@madduck.net

[-- Attachment #2: Digital signature (see http://martin-krafft.net/gpg/) --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [PATCH] notmuch: Respect maildir message flags

[Stewart Smith]

[-- Attachment #1: Type: text/plain, Size: 5110 bytes --]

On Tue, Feb 16, 2010 at 03:12:50PM +1300, martin f krafft wrote:
> also sprach Stewart Smith <stewart@flamingspork.com> [2010.02.16.1458 +1300]:
> > +	    case 'R': /* replied */
> > +		notmuch_message_add_tag (message, "answered");
> > +		break;
> 
> 'r' means replied, not 'answered'.

fixed.

(i have to admit... i didn't look too closely at this... it just
worked enough for me)

> 
> > +	    case 'T': /* trashed */
> > +		notmuch_message_add_tag (message, "deleted");
> > +		break;
> 
> Same. trashed and deleted are not the same thing.

changed to 'trashed'.

> I don't want to get into an argument over this, because I think this
> already exposes a problem: you are putting into global namespace
> something not everyone might want, or agree with.
> 
> Why not use 'maildirflags::replied' instead? People can always map
> that to something in the global namespace.

What about putting them all in there except for the seen tag, with the
seen tag dictating if it gets marked 'unread' or not? I cannot imagine
where somebody would want this not to be the case... it was bad enough
discovering 100,000 unread messages :)

What about this patch (just with those few things fixed)?

diff --git a/notmuch-new.c b/notmuch-new.c
index f25c71f..8303047 100644
--- a/notmuch-new.c
+++ b/notmuch-new.c
@@ -39,6 +39,7 @@ typedef struct {
     int total_files;
     int processed_files;
     int added_messages;
+    int tag_maildir;
     struct timeval tv_start;
 
     _filename_list_t *removed_files;
@@ -169,6 +170,60 @@ _entries_resemble_maildir (struct dirent **entries, int count)
     return 0;
 }
 
+/* Tag new mail according to its Maildir attribute flags.
+ *
+ * Test if the mail file's filename contains any of the
+ * standard Maildir attributes, and translate these to
+ * the corresponding standard notmuch tags.
+ *
+ * If the message is not marked as 'seen', or if no
+ * flags are present, tag as 'inbox, unread'.
+ */
+static void
+derive_tags_from_maildir_flags (notmuch_message_t *message,
+				const char * path)
+{
+    int seen = FALSE;
+    int end_of_flags = FALSE;
+    size_t l = strlen(path);
+
+    /* Non-experimental message flags start with this */
+    char * i = strstr(path, ":2,");
+    i = (i) ? i : strstr(path, "!2,"); /* This format is used on VFAT */
+    if (i != NULL) {
+	i += 3;
+	for (; i < (path + l) && !end_of_flags; i++) {
+	    switch (*i) {
+	    case 'F' :
+		notmuch_message_add_tag (message, "maildir::flagged");
+		break;
+	    case 'R': /* replied */
+		notmuch_message_add_tag (message, "maildir::replied");
+		break;
+	    case 'D':
+		notmuch_message_add_tag (message, "maildir::draft");
+		break;
+	    case 'S': /* seen */
+		seen = TRUE;
+		break;
+	    case 'T': /* trashed */
+		notmuch_message_add_tag (message, "maildir::trashed");
+		break;
+	    case 'P': /* passed */
+		notmuch_message_add_tag (message, "maildir::forwarded");
+		break;
+	    default:
+		end_of_flags = TRUE;
+		break;
+	    }
+	}
+    }
+
+    if (i == NULL || !seen) {
+	tag_inbox_and_unread (message);
+    }
+}
+
 /* Examine 'path' recursively as follows:
  *
  *   o Ask the filesystem for the mtime of 'path' (fs_mtime)
@@ -222,6 +277,7 @@ add_files_recursive (notmuch_database_t *notmuch,
     notmuch_filenames_t *db_subdirs = NULL;
     struct stat st;
     notmuch_bool_t is_maildir, new_directory;
+    int maildir_detected = -1;
 
     if (stat (path, &st)) {
 	fprintf (stderr, "Error reading directory %s: %s\n",
@@ -301,6 +357,28 @@ add_files_recursive (notmuch_database_t *notmuch,
 	    continue;
 	}
 
+	/* If this directory is a Maildir folder, we need to
+	 * ignore any subdirectories marked tmp/, and scan for
+	 * Maildir attributes on messages contained in the sub-
+	 * directories 'new' and 'cur'. */
+	if (maildir_detected != 0 &&
+	    (entry->d_type == DT_DIR || entry->d_type == DT_UNKNOWN) &&
+	    ((strcmp (entry->d_name, "tmp") == 0) ||
+	     (strcmp (entry->d_name, "new") == 0) ||
+	     (strcmp (entry->d_name, "cur") == 0))) {
+
+    if (maildir_detected == -1) {
+      maildir_detected = _entries_resemble_maildir(fs_entries, num_fs_entries);
+    }
+    if (maildir_detected == 1) {
+      if (strcmp (entry->d_name, "tmp") == 0) {
+        continue;
+      } else {
+        state->tag_maildir = TRUE;
+      }
+    }
+  }
+
 	next = talloc_asprintf (notmuch, "%s/%s", path, entry->d_name);
 	status = add_files_recursive (notmuch, next, state);
 	if (status && ret == NOTMUCH_STATUS_SUCCESS)
@@ -412,7 +490,12 @@ add_files_recursive (notmuch_database_t *notmuch,
 	/* success */
 	case NOTMUCH_STATUS_SUCCESS:
 	    state->added_messages++;
-	    tag_inbox_and_unread (message);
+			if (state->tag_maildir) {
+			    derive_tags_from_maildir_flags (message,
+							    entry->d_name);
+			} else {
+			    tag_inbox_and_unread (message);
+			}
 	    break;
 	/* Non-fatal issues (go on to next file) */
 	case NOTMUCH_STATUS_DUPLICATE_MESSAGE_ID:

-- 
Stewart Smith

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [PATCH] notmuch: Respect maildir message flags

[martin f krafft]

[-- Attachment #1: Type: text/plain, Size: 772 bytes --]

also sprach Stewart Smith <stewart@flamingspork.com> [2010.02.16.1521 +1300]:
> What about putting them all in there except for the seen tag, with
> the seen tag dictating if it gets marked 'unread' or not? I cannot
> imagine where somebody would want this not to be the case... it
> was bad enough discovering 100,000 unread messages :)

Well, I do keep messages unread even after I saw them, until I can
actually read them...

> What about this patch (just with those few things fixed)?

Thanks!

-- 
martin | http://madduck.net/ | http://two.sentenc.es/
 
"i dislike arguments of any kind. they are always vulgar, and often
 convincing."
                                                        -- oscar wilde
 
spamtraps: madduck.bogus@madduck.net

[-- Attachment #2: Digital signature (see http://martin-krafft.net/gpg/) --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [PATCH] notmuch: Respect maildir message flags

[Michal Sojka]

On Tue, 16 Feb 2010 13:21:28 +1100, Stewart Smith <stewart@flamingspork.com> wrote:
> What about this patch (just with those few things fixed)?
> 
> diff --git a/notmuch-new.c b/notmuch-new.c
> index f25c71f..8303047 100644
> --- a/notmuch-new.c
> +++ b/notmuch-new.c
> @@ -39,6 +39,7 @@ typedef struct {
>      int total_files;
>      int processed_files;
>      int added_messages;
> +    int tag_maildir;

I think notmuch_bool_t will be better than int here.

BTW what is the reason for using notmuch_bool_t instead of bool from
stdbool.h?

> @@ -222,6 +277,7 @@ add_files_recursive (notmuch_database_t *notmuch,
>      notmuch_filenames_t *db_subdirs = NULL;
>      struct stat st;
>      notmuch_bool_t is_maildir, new_directory;
> +    int maildir_detected = -1;

Again, notmuch_bool_t is IMHO better. You seem only to use values -1 and
1 which is quite confusing.

>  
>      if (stat (path, &st)) {
>  	fprintf (stderr, "Error reading directory %s: %s\n",
> @@ -301,6 +357,28 @@ add_files_recursive (notmuch_database_t *notmuch,
>  	    continue;
>  	}
>  
> +	/* If this directory is a Maildir folder, we need to
> +	 * ignore any subdirectories marked tmp/, and scan for
> +	 * Maildir attributes on messages contained in the sub-
> +	 * directories 'new' and 'cur'. */
> +	if (maildir_detected != 0 &&
> +	    (entry->d_type == DT_DIR || entry->d_type == DT_UNKNOWN) &&
> +	    ((strcmp (entry->d_name, "tmp") == 0) ||
> +	     (strcmp (entry->d_name, "new") == 0) ||
> +	     (strcmp (entry->d_name, "cur") == 0))) {
> +
> +    if (maildir_detected == -1) {
> +      maildir_detected = _entries_resemble_maildir(fs_entries, num_fs_entries);
> +    }
> +    if (maildir_detected == 1) {
> +      if (strcmp (entry->d_name, "tmp") == 0) {
> +        continue;
> +      } else {
> +        state->tag_maildir = TRUE;

You might also want to set this to FALSE somewhere. It is very unlikely,
but somebody can create non-maildir under maildir.

> @@ -412,7 +490,12 @@ add_files_recursive (notmuch_database_t *notmuch,
>  	/* success */
>  	case NOTMUCH_STATUS_SUCCESS:
>  	    state->added_messages++;
> -	    tag_inbox_and_unread (message);
> +			if (state->tag_maildir) {
> +			    derive_tags_from_maildir_flags (message,
> +							    entry->d_name);
> +			} else {
> +			    tag_inbox_and_unread (message);
> +			}
>  	    break;
>  	/* Non-fatal issues (go on to next file) */
>  	case NOTMUCH_STATUS_DUPLICATE_MESSAGE_ID:

You add the tags only to newly discovered mails. If a file is renamed
(e.g. because another mail reader removed the S flag), the tags will not
be updated. It is a question, what is the proper behavior. I personally
use something like what is in your patch and then use notmuchsync to
handle renamed files. It has only one problem - notmuchsync is quite
slow, so if this is solved in notmuch, I'd be happy.

Cheers,
 Michal

[PATCH] notmuch-new: Respect maildir flags when importing a new message

[Sebastian Spaeth]

When importing a new mail do check for maildir tags and assign corresponding notmuch tags.

Based on a patch by Michiel Buddingh <michiel@michielbuddingh.net> and subsequently modified by Tim Stoakes, Stewart Smith, and Sebastian Spaeth (see mail thread around mail id:20100210031339.GH16686@mail.rocksoft.com)

Do note that this will only add tags when importing a really new message, and will not do anything when detecting a file rename (although someone should really make it honor file renames as well). Deleteing an existing message in another IMAP client will therefore not trigger tagging (as it counts as a file rename).

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
---
I cleaned up the patch by Stewart Smith a bit more, but it's basically his last version of the patch. The biggest caveat is really that file renames won't cause any change. So this approach is really only part of the solution to sync with e.g. thunderbird usage.
 notmuch-new.c |   64 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++-
 1 files changed, 63 insertions(+), 1 deletions(-)

diff --git a/notmuch-new.c b/notmuch-new.c
index f25c71f..5a75950 100644
--- a/notmuch-new.c
+++ b/notmuch-new.c
@@ -39,6 +39,7 @@ typedef struct {
     int total_files;
     int processed_files;
     int added_messages;
+    notmuch_bool_t tag_maildir;
     struct timeval tv_start;
 
     _filename_list_t *removed_files;
@@ -169,6 +170,60 @@ _entries_resemble_maildir (struct dirent **entries, int count)
     return 0;
 }
 
+/* Tag new mail according to its Maildir attribute flags.
+ *
+ * Test if the mail file's filename contains any of the
+ * standard Maildir attributes, and translate these to
+ * the corresponding standard notmuch tags.
+ *
+ * If the message is not marked as 'seen', or if no
+ * flags are present, tag as 'inbox, unread'.
+ */
+static void
+derive_tags_from_maildir_flags (notmuch_message_t *message,
+                           const char * path)
+{
+    int seen = FALSE;
+    int end_of_flags = FALSE;
+    size_t l = strlen(path);
+
+    /* Non-experimental message flags start with this */
+    char * i = strstr(path, ":2,");
+    i = (i) ? i : strstr(path, "!2,"); /* This format is used on VFAT */
+    if (i != NULL) {
+   i += 3;
+   for (; i < (path + l) && !end_of_flags; i++) {
+       switch (*i) {
+       case 'F' :
+           notmuch_message_add_tag (message, "maildir::flagged");
+           break;
+       case 'R': /* replied */
+           notmuch_message_add_tag (message, "maildir::replied");
+           break;
+       case 'D':
+           notmuch_message_add_tag (message, "maildir::draft");
+           break;
+       case 'S': /* seen */
+           seen = TRUE;
+           break;
+       case 'T': /* trashed */
+           notmuch_message_add_tag (message, "maildir::trashed");
+           break;
+       case 'P': /* passed */
+           notmuch_message_add_tag (message, "maildir::forwarded");
+           break;
+       default:
+           end_of_flags = TRUE;
+           break;
+       }
+   }
+    }
+
+    if (i == NULL || !seen) {
+   tag_inbox_and_unread (message);
+    }
+}
+
 /* Examine 'path' recursively as follows:
  *
  *   o Ask the filesystem for the mtime of 'path' (fs_mtime)
@@ -299,6 +354,8 @@ add_files_recursive (notmuch_database_t *notmuch,
 	    strcmp (entry->d_name, ".notmuch") ==0)
 	{
 	    continue;
+	} else {
+	    state->tag_maildir = TRUE;
 	}
 
 	next = talloc_asprintf (notmuch, "%s/%s", path, entry->d_name);
@@ -412,7 +469,12 @@ add_files_recursive (notmuch_database_t *notmuch,
 	/* success */
 	case NOTMUCH_STATUS_SUCCESS:
 	    state->added_messages++;
-	    tag_inbox_and_unread (message);
+	    if (state->tag_maildir) {
+	      derive_tags_from_maildir_flags (message,
+					      entry->d_name);
+	    } else {
+	        tag_inbox_and_unread (message);
+	    }
 	    break;
 	/* Non-fatal issues (go on to next file) */
 	case NOTMUCH_STATUS_DUPLICATE_MESSAGE_ID:
-- 
1.6.3.3

Re: [PATCH] notmuch-new: Respect maildir flags when importing a new message

[Carl Worth]

[-- Attachment #1: Type: text/plain, Size: 1491 bytes --]

On Mon,  1 Mar 2010 14:28:56 +0100, Sebastian Spaeth <Sebastian@SSpaeth.de> wrote:
> When importing a new mail do check for maildir tags and assign
> corresponding notmuch tags.

I think this is a useful thing to support, as obviously new users have
*some* state that's interesting to import (which messages have been
"seen" for example), and simply importing their entire email archive
with both the "inbox" and "unread" tags is not helpful. So I'd like to
figure out how to support this.

> Do note that this will only add tags when importing a really new
> message, and will not do anything when detecting a file rename
> (although someone should really make it honor file renames as
> well). Deleteing an existing message in another IMAP client will
> therefore not trigger tagging (as it counts as a file rename).

But I think we really need to fix that if we're going to claim that
notmuch "supports maildir flags" in any sense.

It's a fairly tricky issue though since we can have multiple files that
have the same message ID, but then have different maildir flags. And
it's the matter of arbitrary ordering which one of these files appears
as "new" and which one appears as a "rename".

It's not obvious to me what we can do here unless we make some
assumptions, (such as "mails always start without the seen flag, and
once it appears it can't be removed" or so). But I'd love some input
From someone who's thought harder about this than I have.

-Carl

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [PATCH] notmuch-new: Respect maildir flags when importing a new message

[Michal Sojka]

On Wed, 07 Apr 2010, Carl Worth wrote:
> On Mon,  1 Mar 2010 14:28:56 +0100, Sebastian Spaeth <Sebastian@SSpaeth.de> wrote:
> > When importing a new mail do check for maildir tags and assign
> > corresponding notmuch tags.
> 
> I think this is a useful thing to support, as obviously new users have
> *some* state that's interesting to import (which messages have been
> "seen" for example), and simply importing their entire email archive
> with both the "inbox" and "unread" tags is not helpful. So I'd like to
> figure out how to support this.

I'm solving this in my mailstore abstraction patches. I'll send the next
version in a while.

> > Do note that this will only add tags when importing a really new
> > message, and will not do anything when detecting a file rename
> > (although someone should really make it honor file renames as
> > well). Deleteing an existing message in another IMAP client will
> > therefore not trigger tagging (as it counts as a file rename).
> 
> But I think we really need to fix that if we're going to claim that
> notmuch "supports maildir flags" in any sense.
> 
> It's a fairly tricky issue though since we can have multiple files that
> have the same message ID, but then have different maildir flags. And
> it's the matter of arbitrary ordering which one of these files appears
> as "new" and which one appears as a "rename".

Yes, that's a problem. It is not currently solved in my pacthes, but I
have one solution in my mind. Let's discuss this with my patch.

> It's not obvious to me what we can do here unless we make some
> assumptions, (such as "mails always start without the seen flag, and
> once it appears it can't be removed" or so). But I'd love some input
> From someone who's thought harder about this than I have.

-Michal