[patch] store folder information

[patch] store folder information

[Andreas Klöckner]

[-- Attachment #1.1: Type: Text/Plain, Size: 223 bytes --]

Hi there,

I've patched notmuch to retain information on which folder emails are stored 
in. This makes the transition from a folders-and-procmail model somewhat 
easier. The resulting changes are attached.

Andreas

[-- Attachment #1.2: 0001-Preseve-folder-information-when-indexing.patch --]
[-- Type: text/x-patch, Size: 5052 bytes --]

From 179af7f436d8c21bad90e7eb88fc17c56f83c26c Mon Sep 17 00:00:00 2001
From: Andreas Kloeckner <inform@tiker.net>
Date: Mon, 14 Dec 2009 14:16:20 -0500
Subject: [PATCH] Preseve folder information when indexing.

---
 lib/database.cc |   14 +++++++++-----
 lib/index.cc    |    2 ++
 lib/notmuch.h   |    1 +
 notmuch-new.c   |   35 ++++++++++++++++++++++++++++++++++-
 4 files changed, 46 insertions(+), 6 deletions(-)

diff --git a/lib/database.cc b/lib/database.cc
index b6c4d07..e6b4272 100644
--- a/lib/database.cc
+++ b/lib/database.cc
@@ -70,10 +70,9 @@ typedef struct {
  *
  *	MESSAGE_ID:	The unique ID of the mail mess (see "id" above)
  *
- * In addition, terms from the content of the message are added with
- * "from", "to", "attachment", and "subject" prefixes for use by the
- * user in searching. But the database doesn't really care itself
- * about any of these.
+ * In addition, terms from the content of the message are added with "from",
+ * "to", "attachment", "subject", and "folder" prefixes for use by the user in
+ * searching. But the database doesn't really care itself about any of these.
  *
  * Timestamp document
  * ------------------
@@ -124,7 +123,8 @@ prefix_t PROBABILISTIC_PREFIX[]= {
     { "from", "XFROM" },
     { "to", "XTO" },
     { "attachment", "XATTACHMENT" },
-    { "subject", "XSUBJECT"}
+    { "subject", "XSUBJECT"},
+    { "folder", "XFOLDER"}
 };
 
 int
@@ -889,6 +889,7 @@ _notmuch_database_link_message (notmuch_database_t *notmuch,
 notmuch_status_t
 notmuch_database_add_message (notmuch_database_t *notmuch,
 			      const char *filename,
+			      const char *folder_name,
 			      notmuch_message_t **message_ret)
 {
     notmuch_message_file_t *message_file;
@@ -1006,6 +1007,9 @@ notmuch_database_add_message (notmuch_database_t *notmuch,
 
 	_notmuch_message_index_file (message, filename);
 
+        if (folder_name != NULL)
+            _notmuch_message_gen_terms (message, "folder", folder_name);
+
 	_notmuch_message_sync (message);
     } catch (const Xapian::Error &error) {
 	fprintf (stderr, "A Xapian exception occurred adding message: %s.\n",
diff --git a/lib/index.cc b/lib/index.cc
index 125fa6c..55f3fbc 100644
--- a/lib/index.cc
+++ b/lib/index.cc
@@ -116,6 +116,8 @@ skip_re_in_subject (const char *subject)
 	    s++;
 	if (strncasecmp (s, "re:", 3) == 0)
 	    s += 3;
+        else if (strncasecmp (s, "aw:", 3) == 0)
+	    s += 3;
 	else
 	    break;
     }
diff --git a/lib/notmuch.h b/lib/notmuch.h
index 60834fb..5d3b3c6 100644
--- a/lib/notmuch.h
+++ b/lib/notmuch.h
@@ -265,6 +265,7 @@ notmuch_database_get_timestamp (notmuch_database_t *database,
 notmuch_status_t
 notmuch_database_add_message (notmuch_database_t *database,
 			      const char *filename,
+			      const char *folder_name,
 			      notmuch_message_t **message);
 
 /* Find a message with the given message_id.
diff --git a/notmuch-new.c b/notmuch-new.c
index 9d20616..bc8adc8 100644
--- a/notmuch-new.c
+++ b/notmuch-new.c
@@ -21,6 +21,7 @@
 #include "notmuch-client.h"
 
 #include <unistd.h>
+#include <glib.h>
 
 static volatile sig_atomic_t do_add_files_print_progress = 0;
 
@@ -144,6 +145,34 @@ add_files_recursive (notmuch_database_t *notmuch,
     struct dirent **namelist = NULL;
     int num_entries;
 
+    gchar *full_folder_name = NULL;
+    gchar *folder_base_name = NULL;
+
+    /* Find name of "folder" containing the email. */
+    full_folder_name = g_strdup(path);
+    while (1)
+    {
+        folder_base_name = g_path_get_basename(full_folder_name);
+
+        if (strcmp(folder_base_name, "cur") == 0
+                || strcmp(folder_base_name, "new") == 0)
+        {
+            gchar *parent_name = g_path_get_dirname(full_folder_name);
+            g_free(full_folder_name);
+            full_folder_name = parent_name;
+        }
+        else
+            break;
+    }
+
+    g_free(full_folder_name);
+
+    if (strcmp(folder_base_name, ".") == 0)
+    {
+        g_free(folder_base_name);
+        folder_base_name = NULL;
+    }
+
     /* If we're told to, we bail out on encountering a read-only
      * directory, (with this being a clear clue from the user to
      * Notmuch that new mail won't be arriving there and we need not
@@ -235,7 +264,9 @@ add_files_recursive (notmuch_database_t *notmuch,
 		    fflush (stdout);
 		}
 
-		status = notmuch_database_add_message (notmuch, next, &message);
+		status = notmuch_database_add_message (notmuch, next, 
+                                                       folder_base_name, 
+                                                       &message);
 		switch (status) {
 		    /* success */
 		    case NOTMUCH_STATUS_SUCCESS:
@@ -301,6 +332,8 @@ add_files_recursive (notmuch_database_t *notmuch,
 	closedir (dir);
     if (namelist)
 	free (namelist);
+    if (folder_base_name)
+        g_free(folder_base_name);
 
     return ret;
 }
-- 
1.6.5


[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 190 bytes --]

Re: [patch] store folder information

[Ruben Pollan]

[-- Attachment #1: Type: text/plain, Size: 1187 bytes --]

Some errors applying the patch:

[meskio@blackspot:src/notmuch.orig]$ git apply ~/0001-Preseve-folder-information-when-indexing.patch
/home/meskio/0001-Preseve-folder-information-when-indexing.patch:136: trailing whitespace.
                status = notmuch_database_add_message (notmuch, next, 
/home/meskio/0001-Preseve-folder-information-when-indexing.patch:137: trailing whitespace.
                                                       folder_base_name, 
warning: 2 lines add whitespace errors.

It's just whitespaces at the end of the lines.


The patch works fine for me. I like it handles nicely the .foo.bar directories 
so I can do searches for "folder:foo" and for "folder:bar".

Reviewed-by: Ruben Pollan <meskio@sindominio.net>

On 14:21, Mon 14 Dec 09, Andreas Klöckner wrote:
> I've patched notmuch to retain information on which folder emails are stored 
> in. This makes the transition from a folders-and-procmail model somewhat 
> easier. The resulting changes are attached.

-- 
Rubén Pollán  | jabber:meskio@jabber.org
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Lo que pasa es que tienes envidia
por que las vocecitas me hablan a mi.

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [patch] store folder information

[Marten Veldthuis]

On Tue, 15 Dec 2009 13:16:35 +0100, Ruben Pollan <meskio@sindominio.net> wrote:
> The patch works fine for me. I like it handles nicely the .foo.bar directories 
> so I can do searches for "folder:foo" and for "folder:bar".
> 
> Reviewed-by: Ruben Pollan <meskio@sindominio.net>

Second that, and I actually think it's a nicer solution than
automatically adding tags based on folder.

Reviewed-by: Marten Veldthuis <marten@veldthuis.com>

-- 
- Marten

Re: [patch] store folder information

[Carl Worth]

[-- Attachment #1: Type: text/plain, Size: 1987 bytes --]

On Mon, 14 Dec 2009 14:21:50 -0500, Andreas Klöckner <lists@informa.tiker.net> wrote:
> I've patched notmuch to retain information on which folder emails are stored 
> in. This makes the transition from a folders-and-procmail model somewhat 
> easier. The resulting changes are attached.

Very nice. I like this idea very much.

(We recently had discussions about automatically adding tags based on
the directories in which mail files were found, and I said I'd prefer a
solution which allowed the user to search on the directory name
instead.)

> @@ -116,6 +116,8 @@ skip_re_in_subject (const char *subject)
>  	    s++;
>  	if (strncasecmp (s, "re:", 3) == 0)
>  	    s += 3;
> +        else if (strncasecmp (s, "aw:", 3) == 0)
> +	    s += 3;
>  	else
>  	    break;
>      }

This hunk looks unrelated to the rest. Could you submit that separately,
please?

> +    gchar *full_folder_name = NULL;
> +    gchar *folder_base_name = NULL;
> +
> +    /* Find name of "folder" containing the email. */
> +    full_folder_name = g_strdup(path);
> +    while (1)
> +    {
> +        folder_base_name = g_path_get_basename(full_folder_name);

The trailing directory name is available already during the
traversal. So you don't need to search it back out again. See the patch
in the following message:

	id:87fx8bygi7.fsf@linux.vnet.ibm.com

which simply passes the trailing directory name along, (but skipping a
name of "cur" or "new" while traversing).

That kind of approach looks a lot nice to me.

Beyond that, though, I imagine some people have hierarchical folders as
well, so it probably makes sense to store them as well.

To do that, it's probably just a matter of calling gen_terms on the
complete filename. I haven't tested, but doing that should allow
Xapian's phrase searching to do the right thing for something like:

	filename:portion/of/the/path/name

I think something like that is what I would like to see.

-Carl

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [patch] store folder information

[micah anderson]

[-- Attachment #1: Type: text/plain, Size: 3137 bytes --]


Hi Andreas,

I'm just writing because of the patch you sent to the notmuch list on
December 15th. It seems like many people are wanting this functionality,
I know I am myself and Carl has also indicated the same. However, there
were a couple of minor suggestions for improvements for your patch that
have not seen a reply from you yet. I'm particularly eager to see this
get accepted upstream, and it sounds like the changes necessary to do so
are relatively minor.

I'm wondering what your plans are for addressing these issues? I've come
to depend on this functionality, and would love to see it incorporated
upstream! 

Specifically these were:

1. Unrelated whitespace:

On December 16th,2009 Ruben Pollan <meskio@sindominio.net> wrote:

> [meskio@blackspot:src/notmuch.orig]$ git apply ~/0001-Preseve-folder-information-when-indexing.patch
> /home/meskio/0001-Preseve-folder-information-when-indexing.patch:136: trailing whitespace.
>		 status notmuch_database_add_message (notmuch, next,
> /home/meskio/0001-Preseve-folder-information-when-indexing.patch:137: trailing whitespace.
>							folder_base_name,
> warning: 2 lines add whitespace errors.
>
> It's just whitespaces at the end of the lines.

2. An unrelated hunk creeping in:

On Tue, 15 Dec 2009 13:22:19 -0800, Carl Worth <cworth@cworth.org> wrote:
> On Mon, 14 Dec 2009 14:21:50 -0500, Andreas Kl=C3=B6ckner <lists@informa.=
tiker.net> wrote:
> >
> > @@ -116,6 +116,8 @@ skip_re_in_subject (const char *subject)
> >  	    s++;
> >  	if (strncasecmp (s, "re:", 3) =3D=3D 0)
> >  	    s +=3D 3;
> > +        else if (strncasecmp (s, "aw:", 3) =3D=3D 0)
> > +	    s +=3D 3;
> >  	else
> >  	    break;
> >      }
>=20
> This hunk looks unrelated to the rest. Could you submit that separately,
> please?


3. Redundant trailing directory name traversal:

> > +    gchar *full_folder_name =3D NULL;
> > +    gchar *folder_base_name =3D NULL;
> > +
> > +    /* Find name of "folder" containing the email. */
> > +    full_folder_name =3D g_strdup(path);
> > +    while (1)
> > +    {
> > +        folder_base_name =3D g_path_get_basename(full_folder_name);
>
> The trailing directory name is available already during the
> traversal. So you don't need to search it back out again. See the patch
> in the following message:
>
> 	id:87fx8bygi7.fsf@linux.vnet.ibm.com
>
> which simply passes the trailing directory name along, (but skipping a
> name of "cur" or "new" while traversing).

4. supporting hierarchical folders (perhaps this is a later improvement
that does not need to be added before the original patch is accepted?):

> Beyond that, though, I imagine some people have hierarchical folders as
> well, so it probably makes sense to store them as well.
>
> To do that, it's probably just a matter of calling gen_terms on the
> complete filename. I haven't tested, but doing that should allow
> Xapian's phrase searching to do the right thing for something like:
>
> 	filename:portion/of/the/path/name

5. Probably the patch needs to be rebased off of master at this point.

Micah

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

Re: [patch] store folder information

[Michal Sojka]

On Wednesday 27 of January 2010 16:55:55 micah anderson wrote:
> have not seen a reply from you yet. I'm particularly eager to see this
> get accepted upstream, and it sounds like the changes necessary to do so
> are relatively minor.

Hi Micah and others,

I wanted to test this patch, so I rebased it to the current HEAD. I had to do 
it manually since notmuch evolved significantly since the original posting. 
I'll post in followup mails.

> I'm wondering what your plans are for addressing these issues? I've come
> to depend on this functionality, and would love to see it incorporated
> upstream!
> 
> Specifically these were:
> 
> 1. Unrelated whitespace:

Fixed.
 
> 2. An unrelated hunk creeping in:
> 
> On Tue, 15 Dec 2009 13:22:19 -0800, Carl Worth <cworth@cworth.org> wrote:
> > On Mon, 14 Dec 2009 14:21:50 -0500, Andreas Kl=C3=B6ckner
> > <lists@informa.=
> 
> tiker.net> wrote:
> > > @@ -116,6 +116,8 @@ skip_re_in_subject (const char *subject)
> > >  	    s++;
> > >  	if (strncasecmp (s, "re:", 3) =3D=3D 0)
> > >  	    s +=3D 3;
> > > +        else if (strncasecmp (s, "aw:", 3) =3D=3D 0)
> > > +	    s +=3D 3;
> > >  	else
> > >  	    break;
> > >      }

Fixed.

> 3. Redundant trailing directory name traversal:
> > > +    gchar *full_folder_name =3D NULL;
> > > +    gchar *folder_base_name =3D NULL;
> > > +
> > > +    /* Find name of "folder" containing the email. */
> > > +    full_folder_name =3D g_strdup(path);
> > > +    while (1)
> > > +    {
> > > +        folder_base_name =3D g_path_get_basename(full_folder_name);
> >
> > The trailing directory name is available already during the
> > traversal. So you don't need to search it back out again. See the patch
> > in the following message:
> >
> > 	id:87fx8bygi7.fsf@linux.vnet.ibm.com
> >
> > which simply passes the trailing directory name along, (but skipping a
> > name of "cur" or "new" while traversing).
> 
> 4. supporting hierarchical folders (perhaps this is a later improvement
> 
> that does not need to be added before the original patch is accepted?):
> > Beyond that, though, I imagine some people have hierarchical folders as
> > well, so it probably makes sense to store them as well.
> >
> > To do that, it's probably just a matter of calling gen_terms on the
> > complete filename. I haven't tested, but doing that should allow
> > Xapian's phrase searching to do the right thing for something like:
> >
> > 	filename:portion/of/the/path/name

I leave these two points for later since I do not have time for them now. If 
somebody want to do it, let me know.

Cheers
Michal

[PATCH 1/2] Skip German "aw:" prefix in subjects

[Michal Sojka]

This was originally present in Andreas Klöckner's patch.
---
 lib/index.cc |    2 ++
 1 files changed, 2 insertions(+), 0 deletions(-)

diff --git a/lib/index.cc b/lib/index.cc
index 7e2da08..eb97f91 100644
--- a/lib/index.cc
+++ b/lib/index.cc
@@ -116,6 +116,8 @@ skip_re_in_subject (const char *subject)
 	    s++;
 	if (strncasecmp (s, "re:", 3) == 0)
 	    s += 3;
+        else if (strncasecmp (s, "aw:", 3) == 0)
+	    s += 3;
 	else
 	    break;
     }
-- 
1.6.6

[PATCH 2/2] Preserve folder information when indexing

[Michal Sojka]

This patch was originally sent by Andreas Klöckner. This is a slightly
reworked version (only coding style changes) which was manually rebased
to the current HEAD.

There are still things to fix (see the previous Carl's email)
- Redundant trailing directory name traversal
- Supporting hierarchical folders
---
 lib/database.cc |   13 +++++++++----
 lib/notmuch.h   |    1 +
 notmuch-new.c   |   38 +++++++++++++++++++++++++++++++++++++-
 3 files changed, 47 insertions(+), 5 deletions(-)

diff --git a/lib/database.cc b/lib/database.cc
index cce7847..5f2f35d 100644
--- a/lib/database.cc
+++ b/lib/database.cc
@@ -84,9 +84,9 @@ typedef struct {
  *	MESSAGE_ID:	The unique ID of the mail mess (see "id" above)
  *
  * In addition, terms from the content of the message are added with
- * "from", "to", "attachment", and "subject" prefixes for use by the
- * user in searching. But the database doesn't really care itself
- * about any of these.
+ * "from", "to", "attachment", "subject" and "folder" prefixes for use
+ * by the user in searching. But the database doesn't really care
+ * itself about any of these.
  *
  * The data portion of a mail document is empty.
  *
@@ -154,7 +154,8 @@ prefix_t PROBABILISTIC_PREFIX[]= {
     { "from",			"XFROM" },
     { "to",			"XTO" },
     { "attachment",		"XATTACHMENT" },
-    { "subject",		"XSUBJECT"}
+    { "subject",		"XSUBJECT"},
+    { "folder", 		"XFOLDER"}
 };
 
 int
@@ -1317,6 +1318,7 @@ _notmuch_database_link_message (notmuch_database_t *notmuch,
 notmuch_status_t
 notmuch_database_add_message (notmuch_database_t *notmuch,
 			      const char *filename,
+			      const char *folder_name,
 			      notmuch_message_t **message_ret)
 {
     notmuch_message_file_t *message_file;
@@ -1432,6 +1434,9 @@ notmuch_database_add_message (notmuch_database_t *notmuch,
 	    date = notmuch_message_file_get_header (message_file, "date");
 	    _notmuch_message_set_date (message, date);
 
+	    if (folder_name != NULL)
+		_notmuch_message_gen_terms (message, "folder", folder_name);
+
 	    _notmuch_message_index_file (message, filename);
 	} else {
 	    ret = NOTMUCH_STATUS_DUPLICATE_MESSAGE_ID;
diff --git a/lib/notmuch.h b/lib/notmuch.h
index 15c9db4..3a5ab78 100644
--- a/lib/notmuch.h
+++ b/lib/notmuch.h
@@ -263,6 +263,7 @@ notmuch_database_get_directory (notmuch_database_t *database,
 notmuch_status_t
 notmuch_database_add_message (notmuch_database_t *database,
 			      const char *filename,
+			      const char *folder_name,
 			      notmuch_message_t **message);
 
 /* Remove a message from the given notmuch database.
diff --git a/notmuch-new.c b/notmuch-new.c
index f25c71f..b7c65db 100644
--- a/notmuch-new.c
+++ b/notmuch-new.c
@@ -21,6 +21,7 @@
 #include "notmuch-client.h"
 
 #include <unistd.h>
+#include <glib.h>
 
 typedef struct _filename_node {
     char *filename;
@@ -169,6 +170,35 @@ _entries_resemble_maildir (struct dirent **entries, int count)
     return 0;
 }
 
+static char*
+_get_folder_base_name(const char *path)
+{
+    gchar *full_folder_name = NULL;
+    gchar *folder_base_name = NULL;
+
+    /* Find name of "folder" containing the email. */
+    full_folder_name = g_strdup(path);
+    while (1) {
+	folder_base_name = g_path_get_basename(full_folder_name);
+
+	if (strcmp(folder_base_name, "cur") == 0
+	    || strcmp(folder_base_name, "new") == 0) {
+	    gchar *parent_name = g_path_get_dirname(full_folder_name);
+	    g_free(full_folder_name);
+	    full_folder_name = parent_name;
+	} else
+	    break;
+    }
+
+    g_free(full_folder_name);
+
+    if (strcmp(folder_base_name, ".") == 0) {
+	g_free(folder_base_name);
+	folder_base_name = NULL;
+    }
+    return folder_base_name;
+}
+
 /* Examine 'path' recursively as follows:
  *
  *   o Ask the filesystem for the mtime of 'path' (fs_mtime)
@@ -222,6 +252,7 @@ add_files_recursive (notmuch_database_t *notmuch,
     notmuch_filenames_t *db_subdirs = NULL;
     struct stat st;
     notmuch_bool_t is_maildir, new_directory;
+    char *folder_base_name = NULL;
 
     if (stat (path, &st)) {
 	fprintf (stderr, "Error reading directory %s: %s\n",
@@ -407,7 +438,10 @@ add_files_recursive (notmuch_database_t *notmuch,
 	    fflush (stdout);
 	}
 
-	status = notmuch_database_add_message (notmuch, next, &message);
+	folder_base_name = _get_folder_base_name(path);
+	status = notmuch_database_add_message (notmuch, next,
+					       folder_base_name,
+					       &message);
 	switch (status) {
 	/* success */
 	case NOTMUCH_STATUS_SUCCESS:
@@ -499,6 +533,8 @@ add_files_recursive (notmuch_database_t *notmuch,
 	notmuch_filenames_destroy (db_files);
     if (directory)
 	notmuch_directory_destroy (directory);
+    if (folder_base_name)
+	g_free(folder_base_name);
 
     return ret;
 }
-- 
1.6.6

Re: [PATCH 2/2] Preserve folder information when indexing

[micah anderson]

[-- Attachment #1: Type: text/plain, Size: 714 bytes --]

On Thu, 28 Jan 2010 16:25:17 +0100, Michal Sojka <sojkam1@fel.cvut.cz> wrote:
> This patch was originally sent by Andreas Klöckner. This is a slightly
> reworked version (only coding style changes) which was manually rebased
> to the current HEAD.

Just a quick note about this patch: I received an offline email back
From Andreas who said he is no longer going to improve the patch because
he has stopped using notmuch. I hope that this doesn't mean that this is
dropped, it provides a pretty useful functionality for transitioning
From a folder-based IMAP setup to notmuch! I wish I could volunteer to
pick this up and finish it myself, but I cannot commit to doing this at
the moment.

micah



[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

Re: [PATCH 2/2] Preserve folder information when indexing

[Michal Sojka]

On Thursday 28 of January 2010 18:13:06 micah anderson wrote:
> On Thu, 28 Jan 2010 16:25:17 +0100, Michal Sojka <sojkam1@fel.cvut.cz> 
wrote:
> > This patch was originally sent by Andreas Klöckner. This is a slightly
> > reworked version (only coding style changes) which was manually rebased
> > to the current HEAD.
> 
> Just a quick note about this patch: I received an offline email back
> From Andreas who said he is no longer going to improve the patch because
> he has stopped using notmuch. I hope that this doesn't mean that this is

I know. Andreas wrote it to the list as well. That was the reason why I tried 
to update the patch myself.

> dropped, it provides a pretty useful functionality for transitioning
> From a folder-based IMAP setup to notmuch! I wish I could volunteer to
> pick this up and finish it myself, but I cannot commit to doing this at
> the moment.

I have the same goal as you so I'll probably invest some time to this patch in 
order to be merged.

Michal

[PATCHv2] Preserve folder information when indexing

[Michal Sojka]

Stores the folder (directory name) of the message in the database as a
term with folder prefix.

This patch was originally sent by Andreas Klöckner. The differences
from the original patch are:
- Folder name is taken from strings generated during travesal. It no
  longer uses glib nor it allocates additional memory to determine the
  base name. The same approach as in
  id:87fx8bygi7.fsf@linux.vnet.ibm.com was used.
- Removed unrelated change which was submitted separately as
  id:1264691584-8290-2-git-send-email-sojkam1@fel.cvut.cz
- Changed the comment describing database schema.

TODO (see the previous Carl's email):
- Support hierarchical folders

---
 lib/database.cc |   17 ++++++++++++-----
 lib/notmuch.h   |    1 +
 notmuch-new.c   |    9 ++++++---
 3 files changed, 19 insertions(+), 8 deletions(-)

diff --git a/lib/database.cc b/lib/database.cc
index cce7847..ba2aa16 100644
--- a/lib/database.cc
+++ b/lib/database.cc
@@ -83,10 +83,10 @@ typedef struct {
  *
  *	MESSAGE_ID:	The unique ID of the mail mess (see "id" above)
  *
- * In addition, terms from the content of the message are added with
- * "from", "to", "attachment", and "subject" prefixes for use by the
- * user in searching. But the database doesn't really care itself
- * about any of these.
+ * In addition, terms from the content and path of the message are
+ * added with "from", "to", "attachment", "subject" and "folder"
+ * prefixes for use by the user in searching. But the database doesn't
+ * really care itself about any of these.
  *
  * The data portion of a mail document is empty.
  *
@@ -154,7 +154,10 @@ prefix_t PROBABILISTIC_PREFIX[]= {
     { "from",			"XFROM" },
     { "to",			"XTO" },
     { "attachment",		"XATTACHMENT" },
-    { "subject",		"XSUBJECT"}
+    { "subject",		"XSUBJECT"},
+    { "folder", 		"XFOLDER"}
+    // FIXME: Is folder realy a candidate for probabilistic prefix or
+    // it should be a boolean prefix instead?
 };
 
 int
@@ -1317,6 +1320,7 @@ _notmuch_database_link_message (notmuch_database_t *notmuch,
 notmuch_status_t
 notmuch_database_add_message (notmuch_database_t *notmuch,
 			      const char *filename,
+			      const char *folder_name,
 			      notmuch_message_t **message_ret)
 {
     notmuch_message_file_t *message_file;
@@ -1432,6 +1436,9 @@ notmuch_database_add_message (notmuch_database_t *notmuch,
 	    date = notmuch_message_file_get_header (message_file, "date");
 	    _notmuch_message_set_date (message, date);
 
+	    if (folder_name != NULL)
+		_notmuch_message_gen_terms (message, "folder", folder_name);
+
 	    _notmuch_message_index_file (message, filename);
 	} else {
 	    ret = NOTMUCH_STATUS_DUPLICATE_MESSAGE_ID;
diff --git a/lib/notmuch.h b/lib/notmuch.h
index 15c9db4..3a5ab78 100644
--- a/lib/notmuch.h
+++ b/lib/notmuch.h
@@ -263,6 +263,7 @@ notmuch_database_get_directory (notmuch_database_t *database,
 notmuch_status_t
 notmuch_database_add_message (notmuch_database_t *database,
 			      const char *filename,
+			      const char *folder_name,
 			      notmuch_message_t **message);
 
 /* Remove a message from the given notmuch database.
diff --git a/notmuch-new.c b/notmuch-new.c
index f25c71f..afd4e3d 100644
--- a/notmuch-new.c
+++ b/notmuch-new.c
@@ -207,6 +207,7 @@ _entries_resemble_maildir (struct dirent **entries, int count)
 static notmuch_status_t
 add_files_recursive (notmuch_database_t *notmuch,
 		     const char *path,
+		     const char *folder,
 		     add_files_state_t *state)
 {
     DIR *dir = NULL;
@@ -302,7 +303,9 @@ add_files_recursive (notmuch_database_t *notmuch,
 	}
 
 	next = talloc_asprintf (notmuch, "%s/%s", path, entry->d_name);
-	status = add_files_recursive (notmuch, next, state);
+	status = add_files_recursive (notmuch, next,
+				      is_maildir ? folder : entry->d_name,
+				      state);
 	if (status && ret == NOTMUCH_STATUS_SUCCESS)
 	    ret = status;
 	talloc_free (next);
@@ -407,7 +410,7 @@ add_files_recursive (notmuch_database_t *notmuch,
 	    fflush (stdout);
 	}
 
-	status = notmuch_database_add_message (notmuch, next, &message);
+	status = notmuch_database_add_message (notmuch, next, folder, &message);
 	switch (status) {
 	/* success */
 	case NOTMUCH_STATUS_SUCCESS:
@@ -546,7 +549,7 @@ add_files (notmuch_database_t *notmuch,
 	return NOTMUCH_STATUS_FILE_ERROR;
     }
 
-    status = add_files_recursive (notmuch, path, state);
+    status = add_files_recursive (notmuch, path, basename(path), state);
 
     if (timer_is_active) {
 	/* Now stop the timer. */
-- 
tg: (a2d919e..) t/Preserve-folder-information-when-indexing (depends on: master)

Re: [PATCHv2] Preserve folder information when indexing

[Jameson Rollins]

[-- Attachment #1: Type: text/plain, Size: 1449 bytes --]

On Tue,  2 Feb 2010 16:01:08 +0100, Michal Sojka <sojkam1@fel.cvut.cz> wrote:
> Stores the folder (directory name) of the message in the database as a
> term with folder prefix.
> 
> This patch was originally sent by Andreas Klöckner. The differences
> from the original patch are:
> - Folder name is taken from strings generated during travesal. It no
>   longer uses glib nor it allocates additional memory to determine the
>   base name. The same approach as in
>   id:87fx8bygi7.fsf@linux.vnet.ibm.com was used.
> - Removed unrelated change which was submitted separately as
>   id:1264691584-8290-2-git-send-email-sojkam1@fel.cvut.cz
> - Changed the comment describing database schema.
> 
> TODO (see the previous Carl's email):
> - Support hierarchical folders

Hey, Michal.  Thank you very much for this patch!  This is right on with
what I mentioned as high-priority improvements.

I just gave it a try, though, and it seemed to be behaving a little
strangely.  I currently have two subdirectories, "inbox" and "sent".
However, new mail to either directory was showing up under the search:

notmuch search folder:inbox

and nothing was showing up under the search:

notmuch search folder:sent

I wonder if there's a problem with the logic it uses to determine the
folder naming?  Do you think you could look into this?  I would really
like to push to have a patch like this applied upstream soon.

jamie.

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

Re: [PATCHv2] Preserve folder information when indexing

[Jameson Rollins]

[-- Attachment #1: Type: text/plain, Size: 1054 bytes --]

On Tue, 02 Feb 2010 11:20:02 -0500, Jameson Rollins <jrollins@finestructure.net> wrote:
> I just gave it a try, though, and it seemed to be behaving a little
> strangely.  I currently have two subdirectories, "inbox" and "sent".
> However, new mail to either directory was showing up under the search:
> 
> notmuch search folder:inbox
> 
> and nothing was showing up under the search:
> 
> notmuch search folder:sent
> 
> I wonder if there's a problem with the logic it uses to determine the
> folder naming?  Do you think you could look into this?  I would really
> like to push to have a patch like this applied upstream soon.

I think I was wrong about this.  It looks like I'm seeing a different
issue that may not be related.  As a test I was sending mail to myself,
which meant that it was showing up in both "inbox" *and* "sent", and it
looks like in the face of an email in duplicate directories, the "inbox"
was winning.  Mail sent to not me is showing up under folder:sent just
fine.  So I think this is working!

jamie.

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

Re: [PATCHv2] Preserve folder information when indexing

[Arvid Picciani]

On Tue,  2 Feb 2010 16:01:08 +0100, Michal Sojka<sojkam1@fel.cvut.cz> 
wrote:

> Stores the folder (directory name) of the message in the database as a
> term with folder prefix.

unfortunately it doesnt do anything here :/
i rebuilt the entire index, but no folder: field is added.
can anyone who got it working tell me their directory layout?
mine would be something like:

/media/mail/mail
├── cur
│   ├── 1265050537.H891745P1231.samir.ibcsolutions.de:2,S
│   ├── 1265050572.H419259P1443.samir.ibcsolutions.de:2,S
│   ├── 1265050598.H121639P1634.samir.ibcsolutions.de:2,S
│   ├── 1265050617.H309805P1774.samir.ibcsolutions.de:2,S
│   ├── 1265050625.H818906P1838.samir.ibcsolutions.de:2,S
│   ├── 1265050955.H593083P2020.samir.ibcsolutions.de:2,S

[...]
├── .list.bla
[...]
├── .list.foo
[...]
├── .list.unknown
│   ├── cur
│   │   └── 1265050581.H226772P1509.samir.ibcsolutions.de:2,
│   ├── dovecot.index.cache
│   ├── dovecot.index.log
│   ├── dovecot-uidlist
│   ├── new
│   └── tmp
├── new
├── .notmuch
│   └── xapian
│       ├── flintlock
│       ├── iamchert
│       ├── position.baseA
│       ├── position.baseB
│       ├── position.DB
│       ├── postlist.baseA
│       ├── postlist.baseB
│       ├── postlist.DB
│       ├── record.baseA
│       ├── record.baseB
│       ├── record.DB
│       ├── termlist.baseA
│       ├── termlist.baseB
│       └── termlist.DB
├── .Sent
│   ├── cur
│   │   ├── 1262650675.000000.mbox:2,S
│   │   ├── 1262650675.000001.mbox:2,S
│   │   ├── 1262650675.000002.mbox:2,S

[...]

Re: [PATCHv2] Preserve folder information when indexing

[Jameson Rollins]

[-- Attachment #1: Type: text/plain, Size: 1135 bytes --]

On Tue, 02 Feb 2010 18:48:25 +0100, Arvid Picciani <aep@exys.org> wrote:
> On Tue,  2 Feb 2010 16:01:08 +0100, Michal Sojka<sojkam1@fel.cvut.cz> 
> wrote:
> unfortunately it doesnt do anything here :/
> i rebuilt the entire index, but no folder: field is added.
> can anyone who got it working tell me their directory layout?
> mine would be something like:

What search terms are you using?  You shouldn't consider the cur/new/tmp
maildir subdirectories in the folder naming.  Those are fundamental
parts of maildirs.  You would want to look at the maildir name itself.

I'm starting to think that maybe the folder: field is not the right way
to do this, though.  What if a message moves?  Only new mails are having
this field modified, so if messages are moved that field is not
modified, and since it's being added as part of the message (like
"subject:") it's not modifiable.  It also can't be added down the line
to messages that don't already have it.  I'm going back to my original
suggestion, which was that the configuration should state tag that
should be added for messages in certain folders.

jamie.

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

Re: [PATCHv2] Preserve folder information when indexing

[Arvid Picciani]

It's all the dots fault.   it works just fine with folders not beginning
with .

On Tue, 02 Feb 2010 13:22:29 -0500, Jameson Rollins <jrollins@finestructure.net> wrote:
> I'm going back to my original
> suggestion, which was that the configuration should state tag that
> should be added for messages in certain folders.

well that would be almost what i want. My folders are autogenerated by a
generic list filter, and i'd be to lazy to check regulary for new
folders.
But tags are the proper solution here imo.

Re: [PATCHv2] Preserve folder information when indexing

[micah anderson]

[-- Attachment #1: Type: text/plain, Size: 2231 bytes --]

On Tue, 02 Feb 2010 13:22:29 -0500, Jameson Rollins <jrollins@finestructure.net> wrote:
> I'm starting to think that maybe the folder: field is not the right way
> to do this, though.  What if a message moves?  

The easiest way to answer this question is to try it. I did that, and
things didn't work as expected. I expected that once the message was
moved to a different folder, and 'notmuch new' was run, then the folder
search result would move with the message move. 

I tried to move a message from folder 'thinkpad' to folder 'Junk':

1. First I did a search that included the thinkpad folder, with a string
I know is in an email:

$ notmuch search folder:thinkpad 05K9519
thread:4ba02e2c665b09fb74dc6f1b6ea3aac2   2004-02-25 [1/1] J H. Maut; RE: [Thinkpad] thinkpad 600e LCD quesiton (thinkpad unread)

2. then I moved that email to my Junk folder and ran notmuch new:

$ mv INBOX.thinkpad/new1147465172.M760586P11592V0000000000000301I00141D45_7229.um\,S\=2421\:2\,S INBOX.Junk/new
$ notmuch new
Processed 1 file in almost no time.                    
No new mail. Detected 1 file rename.

3. then I tried to repeat the search I did in #1, searching for the
folder thinkpad and the string, and it still returns the result (this is
not expected! I would expect no result returned):

$ notmuch search folder:thinkpad 05K9519
thread:4ba02e2c665b09fb74dc6f1b6ea3aac2   2004-02-25 [1/1] J H. Maut; RE: [Thinkpad] thinkpad 600e LCD quesiton (thinkpad unread)

4. If i try and search for the same message in the Junk folder, I get no
results:

$ notmuch search folder:Junk 05K9519
$ 

5. If I do a show on the message, I see that notmuch knows that the
file is actually in the Junk folder:

$ notmuch show thread:4ba02e2c665b09fb74dc6f1b6ea3aac2

message{ id:BKEJKEHBAJJAFLAGIHJCGEEEECAA.J.H.Maut@somewhere.net depth:0 match:1 filename:/home/micah/Maildir/Personal/INBOX.Junk/new/1147465172.M760586P11592V0000000000000301I00141D45_7229.um,S=2421:2,S


> Only new mails are having this field modified, so if messages are
> moved that field is not modified, and since it's being added as part
> of the message (like "subject:") it's not modifiable.  

Exactly the problem.

micah

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

Re: [PATCHv2] Preserve folder information when indexing

[Michal Sojka]

On Tuesday 02 February 2010 22:31:38 micah anderson wrote:
> On Tue, 02 Feb 2010 13:22:29 -0500, Jameson Rollins 
<jrollins@finestructure.net> wrote:
> > I'm starting to think that maybe the folder: field is not the right way
> > to do this, though.  What if a message moves?
> 
> The easiest way to answer this question is to try it. I did that, and
> things didn't work as expected. I expected that once the message was
> moved to a different folder, and 'notmuch new' was run, then the folder
> search result would move with the message move.

I want it to work the same way as you expected. It seems it would be necessary 
to modify notmuch_database_remove_message() so that it changes folder term if 
it detects rename.

I guess it should be possible. If tags can be modified then folder terms can 
be modified as well.

Michal

Re: [PATCHv2] Preserve folder information when indexing

[Jameson Rollins]

[-- Attachment #1: Type: text/plain, Size: 914 bytes --]

On Tue, 2 Feb 2010 23:25:18 +0100, Michal Sojka <sojkam1@fel.cvut.cz> wrote:
> On Tuesday 02 February 2010 22:31:38 micah anderson wrote:
> > The easiest way to answer this question is to try it. I did that, and
> > things didn't work as expected. I expected that once the message was
> > moved to a different folder, and 'notmuch new' was run, then the folder
> > search result would move with the message move.
> 
> I want it to work the same way as you expected. It seems it would be necessary 
> to modify notmuch_database_remove_message() so that it changes folder term if 
> it detects rename.

Hey, Michal.  I agree that this is a necessary improvement.  It would
also be nice if messages that don't currently have the field have it
added, whether or not they're in the database to begin with.  This is
important for getting the field on messages that are already in the
database.

jamie.

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

Re: [PATCHv2] Preserve folder information when indexing

[Sebastian Spaeth]

On Tue, 2 Feb 2010 23:25:18 +0100, Michal Sojka <sojkam1@fel.cvut.cz> wrote:
> I want it to work the same way as you expected. It seems it would be necessary 
> to modify notmuch_database_remove_message() so that it changes folder term if 
> it detects rename.

On a tangetial issue: It would help if notmuch were able to set an
(optional) flag when detecting a rename. Similar on how it should set
"new" on new messages, a "moved" tag or whatever would make parsing with
3rd party apps much nicer.

E.g. my folder-to-tag syncer could then just look for moved mails and my
MailDirflag-to-notmuchtag syncer could then limit its search also to
"moved" mails, which would be much more efficient.

What do people think?
Basically what I want is:

notmuch.conf:
tag-for-new="inbox unread notspamchecked whatnothere"
tag-for-moved="moved"

Does this make sense?

Sebastian

Re: [PATCHv2] Preserve folder information when indexing

[Michal Sojka]

On Tuesday 02 February 2010 18:48:25 Arvid Picciani wrote:
> On Tue,  2 Feb 2010 16:01:08 +0100, Michal Sojka<sojkam1@fel.cvut.cz>
> 
> wrote:
> > Stores the folder (directory name) of the message in the database as a
> > term with folder prefix.

Hi,

I've just made several experiments. I tested both Maildir layout (your case)
and non-Maildir layout and both layouts work correctly.

Then I tried to create notmuch database without this patch and then add new mails
and run notmuch new with the patch. It seems that in that case no folder terms
are added to the database. 

Probably DATABASE_VERSION should be increased by this patch, but I do not want
to do it until it is clear that this patch will be merged. Otherwise, there 
might be collision with other patches.
> 
> unfortunately it doesnt do anything here :/
> i rebuilt the entire index, but no folder: field is added.

How did you rebuild it? I deleted the .notmuch directory and run notmuch new.

> can anyone who got it working tell me their directory layout?
> mine would be something like:

If the above didn't help, try to apply this patch. If you see your folder names
during notmuch new, it should work.

Michal

---- 8< ------
diff --git a/lib/database.cc b/lib/database.cc
index ba2aa16..a22e226 100644
--- a/lib/database.cc
+++ b/lib/database.cc
@@ -1436,6 +1436,7 @@ notmuch_database_add_message (notmuch_database_t *notmuch,
            date = notmuch_message_file_get_header (message_file, "date");
            _notmuch_message_set_date (message, date);
 
+           printf("folder=%s\n", folder_name);
            if (folder_name != NULL)
                _notmuch_message_gen_terms (message, "folder", folder_name);
 

Re: [PATCHv2] Preserve folder information when indexing

[Sebastian Spaeth]

> /media/mail/mail
> ├── cur
> │   ├── 1265050537.H891745P1231.samir.ibcsolutions.de:2,S
> │   ├── 1265050572.H419259P1443.samir.ibcsolutions.de:2,S
> │   ├── 1265050598.H121639P1634.samir.ibcsolutions.de:2,S
> │   ├── 1265050617.H309805P1774.samir.ibcsolutions.de:2,S
> │   ├── 1265050625.H818906P1838.samir.ibcsolutions.de:2,S
> │   ├── 1265050955.H593083P2020.samir.ibcsolutions.de:2,S

A related question, currently notmuch seems to store the full absolute
path to mails, but I remeber seeing some documentation saying that it's
either absolute or relative to the xapian db dir. Is that correct, or is
it reasonable to assume absolute directories (as e.g. notmuchsync
currently does).

Sebastian