[PATCH v2 00/13] literal folder: prefix, new path: prefix

[PATCH v2 00/13] literal folder: prefix, new path: prefix

[Jani Nikula]

Hi all, this is v2 of id:cover.1389304779.git.jani@nikula.org.

The new path: prefix is a literal boolean prefix matching the paths,
relative from the maildir root, of the message files. There's no
interpretation of the maildir special cur/new folders, but a recursive
match is provided with "/**" suffix. See the patch for details.

The folder: prefix becomes a literal boolean prefix, similar to path:,
except it matches the maildir cur/new folders in addition to the
specified path. There's no recursive version.

Patches 1-5 add the above.

Patches 6-8 change the test infrastructure to make it easier to add
multiple corpuses, and adds a new test for path: and folder:.

Patches 9-11 add support for testing the database upgrade.

Patches 12-13 update man pages.


I've dropped most of the content in patches 7 and 10 due to their
size. The patches (and the whole series) are available in the
boolean-folder-and-path-v2 branch at
git://gitorious.org/jani/notmuch.git. Web interface at
https://gitorious.org/jani/notmuch/commits/0b3dd2d1cc6c413ea07ea326883ac448499c0e79.


WARNING! The change requires a database format version bump, and a
database upgrade, which is automatically done on 'notmuch new'. The
upgrade is irreversible if you want to try this on your database! A
complete database rebuild is required for reverting the database format
version. Make sure your backups are in order!


BR,
Jani.


Jani Nikula (13):
  lib: refactor folder term update after filename removal
  lib: add support for path: prefix searches
  test: make insert test use the path: prefix
  lib: make folder: prefix literal
  test: fix test for literal folder: search
  test: make it possible to have several corpora
  test: add new corpus with folders
  test: add tests for the new boolean folder: and path: prefixes
  devel: add script to generate test databases
  test: add test database in format version 1
  test: add database upgrade test from format version 1 to 2
  man: update man pages for folder: and path: search terms
  man: try to clarify the folder: and path: vs. --output=files confusion

 devel/gen-testdb.sh                    | 124 ++++++++++++
 lib/database.cc                        |  45 ++++-
 lib/message.cc                         | 249 ++++++++++++++++-------
 lib/notmuch-private.h                  |   3 +
 man/man1/notmuch-search.1              |  10 +-
 man/man7/notmuch-search-terms.7        |  28 ++-
 test/.gitignore                        |   2 +-
 test/Makefile.local                    |   2 +-
 test/T070-insert.sh                    |  10 +-
 test/T100-search-by-folder.sh          |  24 ++-
 test/T101-search-by-folder-and-path.sh |  83 ++++++++
 test/T480-hex-escaping.sh              |   4 +-
 test/T530-upgrade.sh                   | 103 ++++++++++
 test/corpus/{ => default}/cur/01:2,    |   0
 test/corpus/{ => default}/cur/02:2,    |   0
 test/corpus/{ => default}/cur/03:2,    |   0
 test/corpus/{ => default}/cur/04:2,    |   0
 test/corpus/{ => default}/cur/05:2,    |   0
 test/corpus/{ => default}/cur/06:2,    |   0
 test/corpus/{ => default}/cur/07:2,    |   0
 test/corpus/{ => default}/cur/08:2,    |   0
 test/corpus/{ => default}/cur/09:2,    |   0
 test/corpus/{ => default}/cur/10:2,    |   0
 test/corpus/{ => default}/cur/11:2,    |   0
 test/corpus/{ => default}/cur/12:2,    |   0
 test/corpus/{ => default}/cur/13:2,    |   0
 test/corpus/{ => default}/cur/14:2,    |   0
 test/corpus/{ => default}/cur/15:2,    |   0
 test/corpus/{ => default}/cur/16:2,    |   0
 test/corpus/{ => default}/cur/17:2,    |   0
 test/corpus/{ => default}/cur/18:2,    |   0
 test/corpus/{ => default}/cur/19:2,    |   0
 test/corpus/{ => default}/cur/20:2,    |   0
 test/corpus/{ => default}/cur/21:2,    |   0
 test/corpus/{ => default}/cur/22:2,    |   0
 test/corpus/{ => default}/cur/23:2,    |   0
 test/corpus/{ => default}/cur/24:2,    |   0
 test/corpus/{ => default}/cur/25:2,    |   0
 test/corpus/{ => default}/cur/26:2,    |   0
 test/corpus/{ => default}/cur/27:2,    |   0
 test/corpus/{ => default}/cur/28:2,    |   0
 test/corpus/{ => default}/cur/29:2,    |   0
 test/corpus/{ => default}/cur/30:2,    |   0
 test/corpus/{ => default}/cur/31:2,    |   0
 test/corpus/{ => default}/cur/32:2,    |   0
 test/corpus/{ => default}/cur/33:2,    |   0
 test/corpus/{ => default}/cur/34:2,    |   0
 test/corpus/{ => default}/cur/35:2,    |   0
 test/corpus/{ => default}/cur/36:2,    |   0
 test/corpus/{ => default}/cur/37:2,    |   0
 test/corpus/{ => default}/cur/38:2,    |   0
 test/corpus/{ => default}/cur/39:2,    |   0
 test/corpus/{ => default}/cur/40:2,    |   0
 test/corpus/{ => default}/cur/41:2,    |   0
 test/corpus/{ => default}/cur/42:2,    |   0
 test/corpus/{ => default}/cur/43:2,    |   0
 test/corpus/{ => default}/cur/44:2,    |   0
 test/corpus/{ => default}/cur/45:2,    |   0
 test/corpus/{ => default}/cur/46:2,    |   0
 test/corpus/{ => default}/cur/47:2,    |   0
 test/corpus/{ => default}/cur/48:2,    |   0
 test/corpus/{ => default}/cur/49:2,    |   0
 test/corpus/{ => default}/cur/50:2,    |   0
 test/corpus/{ => default}/cur/51:2,    |   0
 test/corpus/{ => default}/cur/52:2,    |   0
 test/corpus/{ => default}/cur/53:2,    |   0
 test/corpus/folders/01:2,              |  34 ++++
 test/corpus/folders/02:2,              |  32 +++
 test/corpus/folders/bar/17:2,          |  23 +++
 test/corpus/folders/bar/18:2,          |  12 ++
 test/corpus/folders/bar/baz/05:2,      | 104 ++++++++++
 test/corpus/folders/bar/baz/23:2,      | 145 +++++++++++++
 test/corpus/folders/bar/baz/24:2,      | 204 +++++++++++++++++++
 test/corpus/folders/bar/baz/cur/25:2,  |  32 +++
 test/corpus/folders/bar/baz/cur/26:2,  | 121 +++++++++++
 test/corpus/folders/bar/baz/new/27:2,  |  21 ++
 test/corpus/folders/bar/baz/new/28:2,  |  38 ++++
 test/corpus/folders/bar/cur/19:2,      | 360 +++++++++++++++++++++++++++++++++
 test/corpus/folders/bar/cur/20:2,      | 101 +++++++++
 test/corpus/folders/bar/new/21:2,      | 102 ++++++++++
 test/corpus/folders/bar/new/22:2,      |  84 ++++++++
 test/corpus/folders/cur/29:2,          |  21 ++
 test/corpus/folders/cur/30:2,          |  75 +++++++
 test/corpus/folders/cur/31:2,          |  31 +++
 test/corpus/folders/cur/32:2,          | 165 +++++++++++++++
 test/corpus/folders/cur/33:2,          |  13 ++
 test/corpus/folders/cur/34:2,          |  46 +++++
 test/corpus/folders/cur/35:2,          |  24 +++
 test/corpus/folders/cur/36:2,          |  25 +++
 test/corpus/folders/cur/37:2,          |  22 ++
 test/corpus/folders/cur/38:2,          |  40 ++++
 test/corpus/folders/cur/39:2,          |  32 +++
 test/corpus/folders/cur/40:2,          |  31 +++
 test/corpus/folders/cur/41:2,          |  37 ++++
 test/corpus/folders/cur/42:2,          |  30 +++
 test/corpus/folders/cur/43:2,          |  26 +++
 test/corpus/folders/cur/44:2,          |  29 +++
 test/corpus/folders/cur/45:2,          |  41 ++++
 test/corpus/folders/cur/46:2,          |  57 ++++++
 test/corpus/folders/cur/47:2,          |  84 ++++++++
 test/corpus/folders/cur/48:2,          |  17 ++
 test/corpus/folders/cur/49:2,          |  33 +++
 test/corpus/folders/cur/50:2,          |  39 ++++
 test/corpus/folders/cur/52:2,          |  39 ++++
 test/corpus/folders/cur/53:2,          |  20 ++
 test/corpus/folders/foo/05:2,          | 104 ++++++++++
 test/corpus/folders/foo/06:2,          |  36 ++++
 test/corpus/folders/foo/baz/11:2,      |  27 +++
 test/corpus/folders/foo/baz/12:2,      |  27 +++
 test/corpus/folders/foo/baz/cur/13:2,  | 178 ++++++++++++++++
 test/corpus/folders/foo/baz/cur/14:2,  |  39 ++++
 test/corpus/folders/foo/baz/new/15:2,  |  22 ++
 test/corpus/folders/foo/baz/new/16:2,  |  27 +++
 test/corpus/folders/foo/cur/07:2,      |  57 ++++++
 test/corpus/folders/foo/cur/08:2,      |  87 ++++++++
 test/corpus/folders/foo/new/03:2,      |  93 +++++++++
 test/corpus/folders/foo/new/09:2,      |  33 +++
 test/corpus/folders/foo/new/10:2,      |  54 +++++
 test/corpus/folders/new/03:2,          |  93 +++++++++
 test/corpus/folders/new/04:2,          |  84 ++++++++
 test/notmuch-test                      |   2 +-
 test/test-databases/README             |   5 +
 test/test-databases/database-v1.tar.gz | Bin 0 -> 262063 bytes
 test/test-lib.sh                       |  21 +-
 124 files changed, 3954 insertions(+), 112 deletions(-)
 create mode 100755 devel/gen-testdb.sh
 create mode 100755 test/T101-search-by-folder-and-path.sh
 create mode 100755 test/T530-upgrade.sh
 rename test/corpus/{ => default}/cur/01:2, (100%)
 rename test/corpus/{ => default}/cur/02:2, (100%)
 rename test/corpus/{ => default}/cur/03:2, (100%)
 rename test/corpus/{ => default}/cur/04:2, (100%)
 rename test/corpus/{ => default}/cur/05:2, (100%)
 rename test/corpus/{ => default}/cur/06:2, (100%)
 rename test/corpus/{ => default}/cur/07:2, (100%)
 rename test/corpus/{ => default}/cur/08:2, (100%)
 rename test/corpus/{ => default}/cur/09:2, (100%)
 rename test/corpus/{ => default}/cur/10:2, (100%)
 rename test/corpus/{ => default}/cur/11:2, (100%)
 rename test/corpus/{ => default}/cur/12:2, (100%)
 rename test/corpus/{ => default}/cur/13:2, (100%)
 rename test/corpus/{ => default}/cur/14:2, (100%)
 rename test/corpus/{ => default}/cur/15:2, (100%)
 rename test/corpus/{ => default}/cur/16:2, (100%)
 rename test/corpus/{ => default}/cur/17:2, (100%)
 rename test/corpus/{ => default}/cur/18:2, (100%)
 rename test/corpus/{ => default}/cur/19:2, (100%)
 rename test/corpus/{ => default}/cur/20:2, (100%)
 rename test/corpus/{ => default}/cur/21:2, (100%)
 rename test/corpus/{ => default}/cur/22:2, (100%)
 rename test/corpus/{ => default}/cur/23:2, (100%)
 rename test/corpus/{ => default}/cur/24:2, (100%)
 rename test/corpus/{ => default}/cur/25:2, (100%)
 rename test/corpus/{ => default}/cur/26:2, (100%)
 rename test/corpus/{ => default}/cur/27:2, (100%)
 rename test/corpus/{ => default}/cur/28:2, (100%)
 rename test/corpus/{ => default}/cur/29:2, (100%)
 rename test/corpus/{ => default}/cur/30:2, (100%)
 rename test/corpus/{ => default}/cur/31:2, (100%)
 rename test/corpus/{ => default}/cur/32:2, (100%)
 rename test/corpus/{ => default}/cur/33:2, (100%)
 rename test/corpus/{ => default}/cur/34:2, (100%)
 rename test/corpus/{ => default}/cur/35:2, (100%)
 rename test/corpus/{ => default}/cur/36:2, (100%)
 rename test/corpus/{ => default}/cur/37:2, (100%)
 rename test/corpus/{ => default}/cur/38:2, (100%)
 rename test/corpus/{ => default}/cur/39:2, (100%)
 rename test/corpus/{ => default}/cur/40:2, (100%)
 rename test/corpus/{ => default}/cur/41:2, (100%)
 rename test/corpus/{ => default}/cur/42:2, (100%)
 rename test/corpus/{ => default}/cur/43:2, (100%)
 rename test/corpus/{ => default}/cur/44:2, (100%)
 rename test/corpus/{ => default}/cur/45:2, (100%)
 rename test/corpus/{ => default}/cur/46:2, (100%)
 rename test/corpus/{ => default}/cur/47:2, (100%)
 rename test/corpus/{ => default}/cur/48:2, (100%)
 rename test/corpus/{ => default}/cur/49:2, (100%)
 rename test/corpus/{ => default}/cur/50:2, (100%)
 rename test/corpus/{ => default}/cur/51:2, (100%)
 rename test/corpus/{ => default}/cur/52:2, (100%)
 rename test/corpus/{ => default}/cur/53:2, (100%)
 create mode 100644 test/corpus/folders/01:2,
 create mode 100644 test/corpus/folders/02:2,
 create mode 100644 test/corpus/folders/bar/17:2,
 create mode 100644 test/corpus/folders/bar/18:2,
 create mode 100644 test/corpus/folders/bar/baz/05:2,
 create mode 100644 test/corpus/folders/bar/baz/23:2,
 create mode 100644 test/corpus/folders/bar/baz/24:2,
 create mode 100644 test/corpus/folders/bar/baz/cur/25:2,
 create mode 100644 test/corpus/folders/bar/baz/cur/26:2,
 create mode 100644 test/corpus/folders/bar/baz/new/27:2,
 create mode 100644 test/corpus/folders/bar/baz/new/28:2,
 create mode 100644 test/corpus/folders/bar/cur/19:2,
 create mode 100644 test/corpus/folders/bar/cur/20:2,
 create mode 100644 test/corpus/folders/bar/new/21:2,
 create mode 100644 test/corpus/folders/bar/new/22:2,
 create mode 100644 test/corpus/folders/cur/29:2,
 create mode 100644 test/corpus/folders/cur/30:2,
 create mode 100644 test/corpus/folders/cur/31:2,
 create mode 100644 test/corpus/folders/cur/32:2,
 create mode 100644 test/corpus/folders/cur/33:2,
 create mode 100644 test/corpus/folders/cur/34:2,
 create mode 100644 test/corpus/folders/cur/35:2,
 create mode 100644 test/corpus/folders/cur/36:2,
 create mode 100644 test/corpus/folders/cur/37:2,
 create mode 100644 test/corpus/folders/cur/38:2,
 create mode 100644 test/corpus/folders/cur/39:2,
 create mode 100644 test/corpus/folders/cur/40:2,
 create mode 100644 test/corpus/folders/cur/41:2,
 create mode 100644 test/corpus/folders/cur/42:2,
 create mode 100644 test/corpus/folders/cur/43:2,
 create mode 100644 test/corpus/folders/cur/44:2,
 create mode 100644 test/corpus/folders/cur/45:2,
 create mode 100644 test/corpus/folders/cur/46:2,
 create mode 100644 test/corpus/folders/cur/47:2,
 create mode 100644 test/corpus/folders/cur/48:2,
 create mode 100644 test/corpus/folders/cur/49:2,
 create mode 100644 test/corpus/folders/cur/50:2,
 create mode 100644 test/corpus/folders/cur/52:2,
 create mode 100644 test/corpus/folders/cur/53:2,
 create mode 100644 test/corpus/folders/foo/05:2,
 create mode 100644 test/corpus/folders/foo/06:2,
 create mode 100644 test/corpus/folders/foo/baz/11:2,
 create mode 100644 test/corpus/folders/foo/baz/12:2,
 create mode 100644 test/corpus/folders/foo/baz/cur/13:2,
 create mode 100644 test/corpus/folders/foo/baz/cur/14:2,
 create mode 100644 test/corpus/folders/foo/baz/new/15:2,
 create mode 100644 test/corpus/folders/foo/baz/new/16:2,
 create mode 100644 test/corpus/folders/foo/cur/07:2,
 create mode 100644 test/corpus/folders/foo/cur/08:2,
 create mode 100644 test/corpus/folders/foo/new/03:2,
 create mode 100644 test/corpus/folders/foo/new/09:2,
 create mode 100644 test/corpus/folders/foo/new/10:2,
 create mode 100644 test/corpus/folders/new/03:2,
 create mode 100644 test/corpus/folders/new/04:2,
 create mode 100644 test/test-databases/README
 create mode 100644 test/test-databases/database-v1.tar.gz

-- 
1.8.5.3

[PATCH v2 01/13] lib: refactor folder term update after filename removal

[Jani Nikula]

Abstract some blocks of code for reuse. No functional changes.
---
 lib/message.cc | 135 ++++++++++++++++++++++++++++-----------------------------
 1 file changed, 66 insertions(+), 69 deletions(-)

diff --git a/lib/message.cc b/lib/message.cc
index c91f3a59836f..7aff4ae5111a 100644
--- a/lib/message.cc
+++ b/lib/message.cc
@@ -481,6 +481,68 @@ notmuch_message_get_replies (notmuch_message_t *message)
     return _notmuch_messages_create (message->replies);
 }
 
+static void
+_notmuch_message_remove_terms (notmuch_message_t *message, const char *prefix)
+{
+    Xapian::TermIterator i;
+    size_t prefix_len = strlen (prefix);
+
+    while (1) {
+	i = message->doc.termlist_begin ();
+	i.skip_to (prefix);
+
+	/* Terminate loop when no terms remain with desired prefix. */
+	if (i == message->doc.termlist_end () ||
+	    strncmp ((*i).c_str (), prefix, prefix_len))
+	    break;
+
+	try {
+	    message->doc.remove_term ((*i));
+	} catch (const Xapian::InvalidArgumentError) {
+	    /* Ignore failure to remove non-existent term. */
+	}
+    }
+}
+
+/* Add directory based terms for all filenames of the message. */
+static notmuch_status_t
+_notmuch_message_add_directory_terms (void *ctx, notmuch_message_t *message)
+{
+    const char *direntry_prefix = _find_prefix ("file-direntry");
+    int direntry_prefix_len = strlen (direntry_prefix);
+    Xapian::TermIterator i = message->doc.termlist_begin ();
+    notmuch_status_t status = NOTMUCH_STATUS_SUCCESS;
+
+    for (i.skip_to (direntry_prefix); i != message->doc.termlist_end (); i++) {
+	unsigned int directory_id;
+	const char *direntry, *directory;
+	char *colon;
+
+	/* Terminate loop at first term without desired prefix. */
+	if (strncmp ((*i).c_str (), direntry_prefix, direntry_prefix_len))
+	    break;
+
+	/* Indicate that there are filenames remaining. */
+	status = NOTMUCH_STATUS_DUPLICATE_MESSAGE_ID;
+
+	direntry = (*i).c_str ();
+	direntry += direntry_prefix_len;
+
+	directory_id = strtol (direntry, &colon, 10);
+
+	if (colon == NULL || *colon != ':')
+	    INTERNAL_ERROR ("malformed direntry");
+
+	directory = _notmuch_database_get_directory_path (ctx,
+							  message->notmuch,
+							  directory_id);
+	if (strlen (directory))
+	    _notmuch_message_gen_terms (message, "folder", directory);
+    }
+
+    return status;
+}
+
 /* Add an additional 'filename' for 'message'.
  *
  * This change will not be reflected in the database until the next
@@ -536,17 +598,12 @@ notmuch_status_t
 _notmuch_message_remove_filename (notmuch_message_t *message,
 				  const char *filename)
 {
-    const char *direntry_prefix = _find_prefix ("file-direntry");
-    int direntry_prefix_len = strlen (direntry_prefix);
-    const char *folder_prefix = _find_prefix ("folder");
-    int folder_prefix_len = strlen (folder_prefix);
     void *local = talloc_new (message);
+    const char *folder_prefix = _find_prefix ("folder");
     char *zfolder_prefix = talloc_asprintf(local, "Z%s", folder_prefix);
-    int zfolder_prefix_len = strlen (zfolder_prefix);
     char *direntry;
     notmuch_private_status_t private_status;
     notmuch_status_t status;
-    Xapian::TermIterator i, last;
 
     status = _notmuch_database_filename_to_direntry (
 	local, message->notmuch, filename, NOTMUCH_FIND_LOOKUP, &direntry);
@@ -567,73 +624,13 @@ _notmuch_message_remove_filename (notmuch_message_t *message,
      *  3. adding back terms for all remaining filenames of the message. */
 
     /* 1. removing all "folder:" terms */
-    while (1) {
-	i = message->doc.termlist_begin ();
-	i.skip_to (folder_prefix);
-
-	/* Terminate loop when no terms remain with desired prefix. */
-	if (i == message->doc.termlist_end () ||
-	    strncmp ((*i).c_str (), folder_prefix, folder_prefix_len))
-	{
-	    break;
-	}
-
-	try {
-	    message->doc.remove_term ((*i));
-	} catch (const Xapian::InvalidArgumentError) {
-	    /* Ignore failure to remove non-existent term. */
-	}
-    }
+    _notmuch_message_remove_terms (message, folder_prefix);
 
     /* 2. removing all "folder:" stemmed terms */
-    while (1) {
-	i = message->doc.termlist_begin ();
-	i.skip_to (zfolder_prefix);
-
-	/* Terminate loop when no terms remain with desired prefix. */
-	if (i == message->doc.termlist_end () ||
-	    strncmp ((*i).c_str (), zfolder_prefix, zfolder_prefix_len))
-	{
-	    break;
-	}
-
-	try {
-	    message->doc.remove_term ((*i));
-	} catch (const Xapian::InvalidArgumentError) {
-	    /* Ignore failure to remove non-existent term. */
-	}
-    }
+    _notmuch_message_remove_terms (message, zfolder_prefix);
 
     /* 3. adding back terms for all remaining filenames of the message. */
-    i = message->doc.termlist_begin ();
-    i.skip_to (direntry_prefix);
-
-    for (; i != message->doc.termlist_end (); i++) {
-	unsigned int directory_id;
-	const char *direntry, *directory;
-	char *colon;
-
-	/* Terminate loop at first term without desired prefix. */
-	if (strncmp ((*i).c_str (), direntry_prefix, direntry_prefix_len))
-	    break;
-
-	/* Indicate that there are filenames remaining. */
-	status = NOTMUCH_STATUS_DUPLICATE_MESSAGE_ID;
-
-	direntry = (*i).c_str ();
-	direntry += direntry_prefix_len;
-
-	directory_id = strtol (direntry, &colon, 10);
-
-	if (colon == NULL || *colon != ':')
-	    INTERNAL_ERROR ("malformed direntry");
-
-	directory = _notmuch_database_get_directory_path (local,
-							  message->notmuch,
-							  directory_id);
-	if (strlen (directory))
-	    _notmuch_message_gen_terms (message, "folder", directory);
-    }
+    status = _notmuch_message_add_directory_terms (local, message);
 
     talloc_free (local);
 
-- 
1.8.5.3

[PATCH v2 02/13] lib: add support for path: prefix searches

[Jani Nikula]

The path: prefix is a literal boolean prefix matching the paths,
relative from the maildir root, of the message files.

path:foo matches all message files in foo (but not in foo/new or
foo/cur).

path:foo/new matches all message files in foo/new.

path:"" matches all message files in the top level maildir.

path:foo/** matches all message files in foo and recursively in all
subdirectories of foo.

path:** matches all message files recursively, i.e. all messages.
---
 lib/database.cc |  7 ++++---
 lib/message.cc  | 52 +++++++++++++++++++++++++++++++++++++++++++++-------
 2 files changed, 49 insertions(+), 10 deletions(-)

diff --git a/lib/database.cc b/lib/database.cc
index f395061e3a73..93cc7f57e9db 100644
--- a/lib/database.cc
+++ b/lib/database.cc
@@ -100,8 +100,8 @@ typedef struct {
  * In addition, terms from the content of the message are added with
  * "from", "to", "attachment", and "subject" prefixes for use by the
  * user in searching. Similarly, terms from the path of the mail
- * message are added with a "folder" prefix. But the database doesn't
- * really care itself about any of these.
+ * message are added with "folder" and "path" prefixes. But the
+ * database doesn't really care itself about any of these.
  *
  * The data portion of a mail document is empty.
  *
@@ -208,7 +208,8 @@ static prefix_t BOOLEAN_PREFIX_EXTERNAL[] = {
     { "thread",			"G" },
     { "tag",			"K" },
     { "is",			"K" },
-    { "id",			"Q" }
+    { "id",			"Q" },
+    { "path",			"P" },
 };
 
 static prefix_t PROBABILISTIC_PREFIX[]= {
diff --git a/lib/message.cc b/lib/message.cc
index 7aff4ae5111a..21abe8e12b9d 100644
--- a/lib/message.cc
+++ b/lib/message.cc
@@ -504,6 +504,40 @@ _notmuch_message_remove_terms (notmuch_message_t *message, const char *prefix)
     }
 }
 
+#define RECURSIVE_SUFFIX "/**"
+
+/* Add "path:" terms for directory. */
+static notmuch_status_t
+_notmuch_message_add_path_terms (notmuch_message_t *message,
+				 const char *directory)
+{
+    /* Add exact "path:" term. */
+    _notmuch_message_add_term (message, "path", directory);
+
+    if (strlen (directory)) {
+	char *path, *p;
+
+	path = talloc_asprintf (NULL, "%s%s", directory, RECURSIVE_SUFFIX);
+	if (! path)
+	    return NOTMUCH_STATUS_OUT_OF_MEMORY;
+
+	/* Add recursive "path:" terms for directory and all parents. */
+	for (p = path + strlen (path) - 1; p > path; p--) {
+	    if (*p == '/') {
+		strcpy (p, RECURSIVE_SUFFIX);
+		_notmuch_message_add_term (message, "path", path);
+	    }
+	}
+
+	talloc_free (path);
+    }
+
+    /* Recursive all-matching path:** for consistency. */
+    _notmuch_message_add_term (message, "path", "**");
+
+    return NOTMUCH_STATUS_SUCCESS;
+}
+
 /* Add directory based terms for all filenames of the message. */
 static notmuch_status_t
 _notmuch_message_add_directory_terms (void *ctx, notmuch_message_t *message)
@@ -538,6 +572,8 @@ _notmuch_message_add_directory_terms (void *ctx, notmuch_message_t *message)
 							  directory_id);
 	if (strlen (directory))
 	    _notmuch_message_gen_terms (message, "folder", directory);
+
+	_notmuch_message_add_path_terms (message, directory);
     }
 
     return status;
@@ -577,6 +613,8 @@ _notmuch_message_add_filename (notmuch_message_t *message,
     /* New terms allow user to search with folder: specification. */
     _notmuch_message_gen_terms (message, "folder", directory);
 
+    _notmuch_message_add_path_terms (message, directory);
+
     talloc_free (local);
 
     return NOTMUCH_STATUS_SUCCESS;
@@ -618,18 +656,18 @@ _notmuch_message_remove_filename (notmuch_message_t *message,
     if (status)
 	return status;
 
-    /* Re-synchronize "folder:" terms for this message. This requires:
-     *  1. removing all "folder:" terms
-     *  2. removing all "folder:" stemmed terms
-     *  3. adding back terms for all remaining filenames of the message. */
+    /* Re-synchronize "folder:" and "path:" terms for this message. */
 
-    /* 1. removing all "folder:" terms */
+    /* Remove all "folder:" terms. */
     _notmuch_message_remove_terms (message, folder_prefix);
 
-    /* 2. removing all "folder:" stemmed terms */
+    /* Remove all "folder:" stemmed terms. */
     _notmuch_message_remove_terms (message, zfolder_prefix);
 
-    /* 3. adding back terms for all remaining filenames of the message. */
+    /* Remove all "path:" terms. */
+    _notmuch_message_remove_terms (message, _find_prefix ("path"));
+
+    /* Add back terms for all remaining filenames of the message. */
     status = _notmuch_message_add_directory_terms (local, message);
 
     talloc_free (local);
-- 
1.8.5.3

[PATCH v2 03/13] test: make insert test use the path: prefix

[Jani Nikula]

This is a more strict test for the insert test.
---
 test/T070-insert.sh | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/test/T070-insert.sh b/test/T070-insert.sh
index e8dc4c099ed1..5de5da43cebb 100755
--- a/test/T070-insert.sh
+++ b/test/T070-insert.sh
@@ -126,14 +126,14 @@ test_expect_equal "$dirname" "$MAIL_DIR/new"
 test_begin_subtest "Insert message into folder"
 gen_insert_msg
 notmuch insert --folder=Drafts < "$gen_msg_filename"
-output=$(notmuch search --output=files folder:Drafts)
+output=$(notmuch search --output=files path:Drafts/new)
 dirname=$(dirname "$output")
 test_expect_equal "$dirname" "$MAIL_DIR/Drafts/new"
 
 test_begin_subtest "Insert message into folder, add/remove tags"
 gen_insert_msg
 notmuch insert --folder=Drafts +draft -unread < "$gen_msg_filename"
-output=$(notmuch search --output=messages folder:Drafts tag:draft NOT tag:unread)
+output=$(notmuch search --output=messages path:Drafts/cur tag:draft NOT tag:unread)
 test_expect_equal "$output" "id:$gen_msg_id"
 
 gen_insert_msg
@@ -143,21 +143,21 @@ test_expect_code 1 "Insert message into non-existent folder" \
 test_begin_subtest "Insert message, create folder"
 gen_insert_msg
 notmuch insert --folder=F --create-folder +folder < "$gen_msg_filename"
-output=$(notmuch search --output=files folder:F tag:folder)
+output=$(notmuch search --output=files path:F/new tag:folder)
 basename=$(basename "$output")
 test_expect_equal_file "$gen_msg_filename" "$MAIL_DIR/F/new/${basename}"
 
 test_begin_subtest "Insert message, create subfolder"
 gen_insert_msg
 notmuch insert --folder=F/G/H/I/J --create-folder +folder < "$gen_msg_filename"
-output=$(notmuch search --output=files folder:F/G/H/I/J tag:folder)
+output=$(notmuch search --output=files path:F/G/H/I/J/new tag:folder)
 basename=$(basename "$output")
 test_expect_equal_file "$gen_msg_filename" "${MAIL_DIR}/F/G/H/I/J/new/${basename}"
 
 test_begin_subtest "Insert message, create existing subfolder"
 gen_insert_msg
 notmuch insert --folder=F/G/H/I/J --create-folder +folder < "$gen_msg_filename"
-output=$(notmuch count folder:F/G/H/I/J tag:folder)
+output=$(notmuch count path:F/G/H/I/J/new tag:folder)
 test_expect_equal "$output" "2"
 
 gen_insert_msg
-- 
1.8.5.3

[PATCH v2 04/13] lib: make folder: prefix literal

[Jani Nikula]

In xapian terms, convert folder: prefix from probabilistic to boolean
prefix, matching the paths, relative form the maildir root, of the
message files, ignoring the maildir new and cur leaf directories.

folder:foo matches all message files in foo, foo/new, and foo/cur.

folder:foo/new does *not* match message files in foo/new.

folder:"" matches all message files in the top level maildir and its
new and cur subdirectories.

This change constitutes a database change: bump the database version
and add database upgrade support for folder: terms. The upgrade also
adds path: terms.
---
 lib/database.cc       | 38 ++++++++++++++++++++++--
 lib/message.cc        | 80 ++++++++++++++++++++++++++++++++++++++++++++-------
 lib/notmuch-private.h |  3 ++
 3 files changed, 108 insertions(+), 13 deletions(-)

diff --git a/lib/database.cc b/lib/database.cc
index 93cc7f57e9db..186e3a7976fe 100644
--- a/lib/database.cc
+++ b/lib/database.cc
@@ -42,7 +42,7 @@ typedef struct {
     const char *prefix;
 } prefix_t;
 
-#define NOTMUCH_DATABASE_VERSION 1
+#define NOTMUCH_DATABASE_VERSION 2
 
 #define STRINGIFY(s) _SUB_STRINGIFY(s)
 #define _SUB_STRINGIFY(s) #s
@@ -210,6 +210,7 @@ static prefix_t BOOLEAN_PREFIX_EXTERNAL[] = {
     { "is",			"K" },
     { "id",			"Q" },
     { "path",			"P" },
+    { "folder",			"XFOLDER:" },
 };
 
 static prefix_t PROBABILISTIC_PREFIX[]= {
@@ -217,7 +218,6 @@ static prefix_t PROBABILISTIC_PREFIX[]= {
     { "to",			"XTO" },
     { "attachment",		"XATTACHMENT" },
     { "subject",		"XSUBJECT"},
-    { "folder",			"XFOLDER"}
 };
 
 const char *
@@ -1168,6 +1168,40 @@ notmuch_database_upgrade (notmuch_database_t *notmuch,
 	}
     }
 
+    /*
+     * Prior to version 2, the "folder:" prefix was probabilistic and
+     * stemmed. Change it to the current boolean prefix. Add "path:"
+     * prefixes while at it.
+     */
+    if (version < 2) {
+       notmuch_query_t *query = notmuch_query_create (notmuch, "");
+       notmuch_messages_t *messages;
+       notmuch_message_t *message;
+
+       count = 0;
+       total = notmuch_query_count_messages (query);
+
+       for (messages = notmuch_query_search_messages (query);
+            notmuch_messages_valid (messages);
+            notmuch_messages_move_to_next (messages)) {
+           if (do_progress_notify) {
+               progress_notify (closure, (double) count / total);
+               do_progress_notify = 0;
+           }
+
+           message = notmuch_messages_get (messages);
+
+           _notmuch_message_upgrade_folder (message);
+           _notmuch_message_sync (message);
+
+           notmuch_message_destroy (message);
+
+           count++;
+       }
+
+       notmuch_query_destroy (query);
+    }
+
     db->set_metadata ("version", STRINGIFY (NOTMUCH_DATABASE_VERSION));
     db->flush ();
 
diff --git a/lib/message.cc b/lib/message.cc
index 21abe8e12b9d..31cb9f107dd7 100644
--- a/lib/message.cc
+++ b/lib/message.cc
@@ -504,6 +504,56 @@ _notmuch_message_remove_terms (notmuch_message_t *message, const char *prefix)
     }
 }
 
+/* Return true if p points at "new" or "cur". */
+static bool is_maildir (const char *p)
+{
+    return strcmp (p, "cur") == 0 || strcmp (p, "new") == 0;
+}
+
+/* Add "folder:" term for directory. */
+static notmuch_status_t
+_notmuch_message_add_folder_terms (notmuch_message_t *message,
+				   const char *directory)
+{
+    char *folder, *last;
+
+    folder = talloc_strdup (NULL, directory);
+    if (! folder)
+       return NOTMUCH_STATUS_OUT_OF_MEMORY;
+
+    /*
+     * If the message file is in a leaf directory named "new" or
+     * "cur", presume maildir and index the parent directory. Thus a
+     * "folder:" prefix search matches messages in the specified
+     * maildir folder, i.e. in the specified directory and its "new"
+     * and "cur" subdirectories.
+     *
+     * Note that this means the "folder:" prefix can't be used for
+     * distinguishing between message files in "new" or "cur". The
+     * "path:" prefix needs to be used for that.
+     *
+     * Note the deliberate difference to _filename_is_in_maildir(). We
+     * don't want to index different things depending on the existence
+     * or non-existence of all maildir sibling directories "new",
+     * "cur", and "tmp". Doing so would be surprising, and difficult
+     * for the user to fix in case all subdirectories were not in
+     * place during indexing.
+     */
+    last = strrchr (folder, '/');
+    if (last) {
+       if (is_maildir (last + 1))
+           *last = '\0';
+    } else if (is_maildir (folder)) {
+       *folder = '\0';
+    }
+
+    _notmuch_message_add_term (message, "folder", folder);
+
+    talloc_free (folder);
+
+    return NOTMUCH_STATUS_SUCCESS;
+}
+
 #define RECURSIVE_SUFFIX "/**"
 
 /* Add "path:" terms for directory. */
@@ -570,9 +620,8 @@ _notmuch_message_add_directory_terms (void *ctx, notmuch_message_t *message)
 	directory = _notmuch_database_get_directory_path (ctx,
 							  message->notmuch,
 							  directory_id);
-	if (strlen (directory))
-	    _notmuch_message_gen_terms (message, "folder", directory);
 
+	_notmuch_message_add_folder_terms (message, directory);
 	_notmuch_message_add_path_terms (message, directory);
     }
 
@@ -610,9 +659,7 @@ _notmuch_message_add_filename (notmuch_message_t *message,
      * notmuch_directory_get_child_files() . */
     _notmuch_message_add_term (message, "file-direntry", direntry);
 
-    /* New terms allow user to search with folder: specification. */
-    _notmuch_message_gen_terms (message, "folder", directory);
-
+    _notmuch_message_add_folder_terms (message, directory);
     _notmuch_message_add_path_terms (message, directory);
 
     talloc_free (local);
@@ -637,8 +684,6 @@ _notmuch_message_remove_filename (notmuch_message_t *message,
 				  const char *filename)
 {
     void *local = talloc_new (message);
-    const char *folder_prefix = _find_prefix ("folder");
-    char *zfolder_prefix = talloc_asprintf(local, "Z%s", folder_prefix);
     char *direntry;
     notmuch_private_status_t private_status;
     notmuch_status_t status;
@@ -659,10 +704,7 @@ _notmuch_message_remove_filename (notmuch_message_t *message,
     /* Re-synchronize "folder:" and "path:" terms for this message. */
 
     /* Remove all "folder:" terms. */
-    _notmuch_message_remove_terms (message, folder_prefix);
-
-    /* Remove all "folder:" stemmed terms. */
-    _notmuch_message_remove_terms (message, zfolder_prefix);
+    _notmuch_message_remove_terms (message, _find_prefix ("folder"));
 
     /* Remove all "path:" terms. */
     _notmuch_message_remove_terms (message, _find_prefix ("path"));
@@ -675,6 +717,22 @@ _notmuch_message_remove_filename (notmuch_message_t *message,
     return status;
 }
 
+/* Upgrade the "folder:" prefix from V1 to V2. */
+#define FOLDER_PREFIX_V1       "XFOLDER"
+#define ZFOLDER_PREFIX_V1      "Z" FOLDER_PREFIX_V1
+void
+_notmuch_message_upgrade_folder (notmuch_message_t *message)
+{
+    /* Remove all old "folder:" terms. */
+    _notmuch_message_remove_terms (message, FOLDER_PREFIX_V1);
+
+    /* Remove all old "folder:" stemmed terms. */
+    _notmuch_message_remove_terms (message, ZFOLDER_PREFIX_V1);
+
+    /* Add new boolean "folder:" and "path:" terms. */
+    _notmuch_message_add_directory_terms (message, message);
+}
+
 char *
 _notmuch_message_talloc_copy_data (notmuch_message_t *message)
 {
diff --git a/lib/notmuch-private.h b/lib/notmuch-private.h
index af185c7c5ba8..59eb2bc285a5 100644
--- a/lib/notmuch-private.h
+++ b/lib/notmuch-private.h
@@ -263,6 +263,9 @@ _notmuch_message_gen_terms (notmuch_message_t *message,
 void
 _notmuch_message_upgrade_filename_storage (notmuch_message_t *message);
 
+void
+_notmuch_message_upgrade_folder (notmuch_message_t *message);
+
 notmuch_status_t
 _notmuch_message_add_filename (notmuch_message_t *message,
 			       const char *filename);
-- 
1.8.5.3

[PATCH v2 05/13] test: fix test for literal folder: search

[Jani Nikula]

Some of the folder: matching capabilities are lost in the
probabilistic to boolean prefix change. Fix them.
---
 test/T100-search-by-folder.sh | 24 +++++++++++++++++++++---
 1 file changed, 21 insertions(+), 3 deletions(-)

diff --git a/test/T100-search-by-folder.sh b/test/T100-search-by-folder.sh
index 5cc2ca8d388a..84ca43820031 100755
--- a/test/T100-search-by-folder.sh
+++ b/test/T100-search-by-folder.sh
@@ -3,6 +3,7 @@ test_description='"notmuch search" by folder: (with variations)'
 . ./test-lib.sh
 
 add_message '[dir]=bad' '[subject]="To the bone"'
+add_message '[dir]=.' '[subject]="Top level"'
 add_message '[dir]=bad/news' '[subject]="Bears"'
 mkdir -p "${MAIL_DIR}/duplicate/bad/news"
 cp "$gen_msg_filename" "${MAIL_DIR}/duplicate/bad/news"
@@ -12,29 +13,46 @@ add_message '[dir]=things/favorite' '[subject]="Raindrops, whiskers, kettles"'
 add_message '[dir]=things/bad' '[subject]="Bites, stings, sad feelings"'
 
 test_begin_subtest "Single-world folder: specification (multiple results)"
-output=$(notmuch search folder:bad | notmuch_search_sanitize)
+output=$(notmuch search folder:bad folder:bad/news folder:things/bad | notmuch_search_sanitize)
 test_expect_equal "$output" "thread:XXX   2001-01-05 [1/1] Notmuch Test Suite; To the bone (inbox unread)
 thread:XXX   2001-01-05 [1/1] Notmuch Test Suite; Bears (inbox unread)
 thread:XXX   2001-01-05 [1/1] Notmuch Test Suite; Bites, stings, sad feelings (inbox unread)"
 
+test_begin_subtest "Top level folder"
+output=$(notmuch search folder:'""' | notmuch_search_sanitize)
+test_expect_equal "$output" "thread:XXX   2001-01-05 [1/1] Notmuch Test Suite; Top level (inbox unread)"
+
 test_begin_subtest "Two-word path to narrow results to one"
 output=$(notmuch search folder:bad/news | notmuch_search_sanitize)
 test_expect_equal "$output" "thread:XXX   2001-01-05 [1/1] Notmuch Test Suite; Bears (inbox unread)"
 
+test_begin_subtest "Folder search with --output=files"
+output=$(notmuch search --output=files folder:bad/news | sed -e "s,$MAIL_DIR,MAIL_DIR,")
+test_expect_equal "$output" "MAIL_DIR/bad/news/msg-003
+MAIL_DIR/duplicate/bad/news/msg-003"
+
 test_begin_subtest "After removing duplicate instance of matching path"
 rm -r "${MAIL_DIR}/bad/news"
 notmuch new
 output=$(notmuch search folder:bad/news | notmuch_search_sanitize)
+test_expect_equal "$output" ""
+
+test_begin_subtest "Folder search with --output=files part #2"
+output=$(notmuch search --output=files folder:duplicate/bad/news | sed -e "s,$MAIL_DIR,MAIL_DIR,")
+test_expect_equal "$output" "MAIL_DIR/duplicate/bad/news/msg-003"
+
+test_begin_subtest "After removing duplicate instance of matching path part #2"
+output=$(notmuch search folder:duplicate/bad/news | notmuch_search_sanitize)
 test_expect_equal "$output" "thread:XXX   2001-01-05 [1/1] Notmuch Test Suite; Bears (inbox unread)"
 
 test_begin_subtest "After rename, old path returns nothing"
 mv "${MAIL_DIR}/duplicate/bad/news" "${MAIL_DIR}/duplicate/bad/olds"
 notmuch new
-output=$(notmuch search folder:bad/news | notmuch_search_sanitize)
+output=$(notmuch search folder:duplicate/bad/news | notmuch_search_sanitize)
 test_expect_equal "$output" ""
 
 test_begin_subtest "After rename, new path returns result"
-output=$(notmuch search folder:bad/olds | notmuch_search_sanitize)
+output=$(notmuch search folder:duplicate/bad/olds | notmuch_search_sanitize)
 test_expect_equal "$output" "thread:XXX   2001-01-05 [1/1] Notmuch Test Suite; Bears (inbox unread)"
 
 test_done
-- 
1.8.5.3

[PATCH v2 06/13] test: make it possible to have several corpora

[Jani Nikula]

Move the existing corpus under corpus/default, and make it possible to
have multiple corpora under the directory.
---
 test/.gitignore                     |  2 +-
 test/Makefile.local                 |  2 +-
 test/T480-hex-escaping.sh           |  4 ++--
 test/corpus/{ => default}/cur/01:2, |  0
 test/corpus/{ => default}/cur/02:2, |  0
 test/corpus/{ => default}/cur/03:2, |  0
 test/corpus/{ => default}/cur/04:2, |  0
 test/corpus/{ => default}/cur/05:2, |  0
 test/corpus/{ => default}/cur/06:2, |  0
 test/corpus/{ => default}/cur/07:2, |  0
 test/corpus/{ => default}/cur/08:2, |  0
 test/corpus/{ => default}/cur/09:2, |  0
 test/corpus/{ => default}/cur/10:2, |  0
 test/corpus/{ => default}/cur/11:2, |  0
 test/corpus/{ => default}/cur/12:2, |  0
 test/corpus/{ => default}/cur/13:2, |  0
 test/corpus/{ => default}/cur/14:2, |  0
 test/corpus/{ => default}/cur/15:2, |  0
 test/corpus/{ => default}/cur/16:2, |  0
 test/corpus/{ => default}/cur/17:2, |  0
 test/corpus/{ => default}/cur/18:2, |  0
 test/corpus/{ => default}/cur/19:2, |  0
 test/corpus/{ => default}/cur/20:2, |  0
 test/corpus/{ => default}/cur/21:2, |  0
 test/corpus/{ => default}/cur/22:2, |  0
 test/corpus/{ => default}/cur/23:2, |  0
 test/corpus/{ => default}/cur/24:2, |  0
 test/corpus/{ => default}/cur/25:2, |  0
 test/corpus/{ => default}/cur/26:2, |  0
 test/corpus/{ => default}/cur/27:2, |  0
 test/corpus/{ => default}/cur/28:2, |  0
 test/corpus/{ => default}/cur/29:2, |  0
 test/corpus/{ => default}/cur/30:2, |  0
 test/corpus/{ => default}/cur/31:2, |  0
 test/corpus/{ => default}/cur/32:2, |  0
 test/corpus/{ => default}/cur/33:2, |  0
 test/corpus/{ => default}/cur/34:2, |  0
 test/corpus/{ => default}/cur/35:2, |  0
 test/corpus/{ => default}/cur/36:2, |  0
 test/corpus/{ => default}/cur/37:2, |  0
 test/corpus/{ => default}/cur/38:2, |  0
 test/corpus/{ => default}/cur/39:2, |  0
 test/corpus/{ => default}/cur/40:2, |  0
 test/corpus/{ => default}/cur/41:2, |  0
 test/corpus/{ => default}/cur/42:2, |  0
 test/corpus/{ => default}/cur/43:2, |  0
 test/corpus/{ => default}/cur/44:2, |  0
 test/corpus/{ => default}/cur/45:2, |  0
 test/corpus/{ => default}/cur/46:2, |  0
 test/corpus/{ => default}/cur/47:2, |  0
 test/corpus/{ => default}/cur/48:2, |  0
 test/corpus/{ => default}/cur/49:2, |  0
 test/corpus/{ => default}/cur/50:2, |  0
 test/corpus/{ => default}/cur/51:2, |  0
 test/corpus/{ => default}/cur/52:2, |  0
 test/corpus/{ => default}/cur/53:2, |  0
 test/notmuch-test                   |  2 +-
 test/test-lib.sh                    | 21 +++++++++++++--------
 58 files changed, 18 insertions(+), 13 deletions(-)
 rename test/corpus/{ => default}/cur/01:2, (100%)
 rename test/corpus/{ => default}/cur/02:2, (100%)
 rename test/corpus/{ => default}/cur/03:2, (100%)
 rename test/corpus/{ => default}/cur/04:2, (100%)
 rename test/corpus/{ => default}/cur/05:2, (100%)
 rename test/corpus/{ => default}/cur/06:2, (100%)
 rename test/corpus/{ => default}/cur/07:2, (100%)
 rename test/corpus/{ => default}/cur/08:2, (100%)
 rename test/corpus/{ => default}/cur/09:2, (100%)
 rename test/corpus/{ => default}/cur/10:2, (100%)
 rename test/corpus/{ => default}/cur/11:2, (100%)
 rename test/corpus/{ => default}/cur/12:2, (100%)
 rename test/corpus/{ => default}/cur/13:2, (100%)
 rename test/corpus/{ => default}/cur/14:2, (100%)
 rename test/corpus/{ => default}/cur/15:2, (100%)
 rename test/corpus/{ => default}/cur/16:2, (100%)
 rename test/corpus/{ => default}/cur/17:2, (100%)
 rename test/corpus/{ => default}/cur/18:2, (100%)
 rename test/corpus/{ => default}/cur/19:2, (100%)
 rename test/corpus/{ => default}/cur/20:2, (100%)
 rename test/corpus/{ => default}/cur/21:2, (100%)
 rename test/corpus/{ => default}/cur/22:2, (100%)
 rename test/corpus/{ => default}/cur/23:2, (100%)
 rename test/corpus/{ => default}/cur/24:2, (100%)
 rename test/corpus/{ => default}/cur/25:2, (100%)
 rename test/corpus/{ => default}/cur/26:2, (100%)
 rename test/corpus/{ => default}/cur/27:2, (100%)
 rename test/corpus/{ => default}/cur/28:2, (100%)
 rename test/corpus/{ => default}/cur/29:2, (100%)
 rename test/corpus/{ => default}/cur/30:2, (100%)
 rename test/corpus/{ => default}/cur/31:2, (100%)
 rename test/corpus/{ => default}/cur/32:2, (100%)
 rename test/corpus/{ => default}/cur/33:2, (100%)
 rename test/corpus/{ => default}/cur/34:2, (100%)
 rename test/corpus/{ => default}/cur/35:2, (100%)
 rename test/corpus/{ => default}/cur/36:2, (100%)
 rename test/corpus/{ => default}/cur/37:2, (100%)
 rename test/corpus/{ => default}/cur/38:2, (100%)
 rename test/corpus/{ => default}/cur/39:2, (100%)
 rename test/corpus/{ => default}/cur/40:2, (100%)
 rename test/corpus/{ => default}/cur/41:2, (100%)
 rename test/corpus/{ => default}/cur/42:2, (100%)
 rename test/corpus/{ => default}/cur/43:2, (100%)
 rename test/corpus/{ => default}/cur/44:2, (100%)
 rename test/corpus/{ => default}/cur/45:2, (100%)
 rename test/corpus/{ => default}/cur/46:2, (100%)
 rename test/corpus/{ => default}/cur/47:2, (100%)
 rename test/corpus/{ => default}/cur/48:2, (100%)
 rename test/corpus/{ => default}/cur/49:2, (100%)
 rename test/corpus/{ => default}/cur/50:2, (100%)
 rename test/corpus/{ => default}/cur/51:2, (100%)
 rename test/corpus/{ => default}/cur/52:2, (100%)
 rename test/corpus/{ => default}/cur/53:2, (100%)

diff --git a/test/.gitignore b/test/.gitignore
index 97e024878728..ed20d1ac6e42 100644
--- a/test/.gitignore
+++ b/test/.gitignore
@@ -1,5 +1,5 @@
 test-results
-corpus.mail
+corpus.mail.*
 smtp-dummy
 symbol-test
 arg-test
diff --git a/test/Makefile.local b/test/Makefile.local
index 8870ca374848..20f17629dda3 100644
--- a/test/Makefile.local
+++ b/test/Makefile.local
@@ -57,4 +57,4 @@ CLEAN := $(CLEAN) $(dir)/smtp-dummy $(dir)/smtp-dummy.o \
 	 $(dir)/database-test.o \
 	 $(dir)/random-corpus $(dir)/random-corpus.o \
 	 $(dir)/parse-time $(dir)/parse-time.o \
-	 $(dir)/corpus.mail $(dir)/test-results $(dir)/tmp.*
+	 $(dir)/corpus.mail.* $(dir)/test-results $(dir)/tmp.*
diff --git a/test/T480-hex-escaping.sh b/test/T480-hex-escaping.sh
index ad50e1bcc071..15701650df83 100755
--- a/test/T480-hex-escaping.sh
+++ b/test/T480-hex-escaping.sh
@@ -3,7 +3,7 @@ test_description="hex encoding and decoding"
 . ./test-lib.sh
 
 test_begin_subtest "round trip"
-find $TEST_DIRECTORY/corpus -type f -print | sort | xargs cat > EXPECTED
+find $TEST_DIRECTORY/corpus/default -type f -print | sort | xargs cat > EXPECTED
 $TEST_DIRECTORY/hex-xcode --direction=encode < EXPECTED | $TEST_DIRECTORY/hex-xcode --direction=decode > OUTPUT
 test_expect_equal_file OUTPUT EXPECTED
 
@@ -25,7 +25,7 @@ $TEST_DIRECTORY/hex-xcode --direction=decode  < EXPECTED.$test_count |\
 test_expect_equal_file EXPECTED.$test_count OUTPUT.$test_count
 
 test_begin_subtest "round trip (in-place)"
-find $TEST_DIRECTORY/corpus -type f -print | sort | xargs cat > EXPECTED
+find $TEST_DIRECTORY/corpus/default -type f -print | sort | xargs cat > EXPECTED
 $TEST_DIRECTORY/hex-xcode --in-place --direction=encode < EXPECTED |\
      $TEST_DIRECTORY/hex-xcode --in-place --direction=decode > OUTPUT
 test_expect_equal_file OUTPUT EXPECTED
diff --git a/test/corpus/cur/01:2, b/test/corpus/default/cur/01:2,
similarity index 100%
rename from test/corpus/cur/01:2,
rename to test/corpus/default/cur/01:2,
diff --git a/test/corpus/cur/02:2, b/test/corpus/default/cur/02:2,
similarity index 100%
rename from test/corpus/cur/02:2,
rename to test/corpus/default/cur/02:2,
diff --git a/test/corpus/cur/03:2, b/test/corpus/default/cur/03:2,
similarity index 100%
rename from test/corpus/cur/03:2,
rename to test/corpus/default/cur/03:2,
diff --git a/test/corpus/cur/04:2, b/test/corpus/default/cur/04:2,
similarity index 100%
rename from test/corpus/cur/04:2,
rename to test/corpus/default/cur/04:2,
diff --git a/test/corpus/cur/05:2, b/test/corpus/default/cur/05:2,
similarity index 100%
rename from test/corpus/cur/05:2,
rename to test/corpus/default/cur/05:2,
diff --git a/test/corpus/cur/06:2, b/test/corpus/default/cur/06:2,
similarity index 100%
rename from test/corpus/cur/06:2,
rename to test/corpus/default/cur/06:2,
diff --git a/test/corpus/cur/07:2, b/test/corpus/default/cur/07:2,
similarity index 100%
rename from test/corpus/cur/07:2,
rename to test/corpus/default/cur/07:2,
diff --git a/test/corpus/cur/08:2, b/test/corpus/default/cur/08:2,
similarity index 100%
rename from test/corpus/cur/08:2,
rename to test/corpus/default/cur/08:2,
diff --git a/test/corpus/cur/09:2, b/test/corpus/default/cur/09:2,
similarity index 100%
rename from test/corpus/cur/09:2,
rename to test/corpus/default/cur/09:2,
diff --git a/test/corpus/cur/10:2, b/test/corpus/default/cur/10:2,
similarity index 100%
rename from test/corpus/cur/10:2,
rename to test/corpus/default/cur/10:2,
diff --git a/test/corpus/cur/11:2, b/test/corpus/default/cur/11:2,
similarity index 100%
rename from test/corpus/cur/11:2,
rename to test/corpus/default/cur/11:2,
diff --git a/test/corpus/cur/12:2, b/test/corpus/default/cur/12:2,
similarity index 100%
rename from test/corpus/cur/12:2,
rename to test/corpus/default/cur/12:2,
diff --git a/test/corpus/cur/13:2, b/test/corpus/default/cur/13:2,
similarity index 100%
rename from test/corpus/cur/13:2,
rename to test/corpus/default/cur/13:2,
diff --git a/test/corpus/cur/14:2, b/test/corpus/default/cur/14:2,
similarity index 100%
rename from test/corpus/cur/14:2,
rename to test/corpus/default/cur/14:2,
diff --git a/test/corpus/cur/15:2, b/test/corpus/default/cur/15:2,
similarity index 100%
rename from test/corpus/cur/15:2,
rename to test/corpus/default/cur/15:2,
diff --git a/test/corpus/cur/16:2, b/test/corpus/default/cur/16:2,
similarity index 100%
rename from test/corpus/cur/16:2,
rename to test/corpus/default/cur/16:2,
diff --git a/test/corpus/cur/17:2, b/test/corpus/default/cur/17:2,
similarity index 100%
rename from test/corpus/cur/17:2,
rename to test/corpus/default/cur/17:2,
diff --git a/test/corpus/cur/18:2, b/test/corpus/default/cur/18:2,
similarity index 100%
rename from test/corpus/cur/18:2,
rename to test/corpus/default/cur/18:2,
diff --git a/test/corpus/cur/19:2, b/test/corpus/default/cur/19:2,
similarity index 100%
rename from test/corpus/cur/19:2,
rename to test/corpus/default/cur/19:2,
diff --git a/test/corpus/cur/20:2, b/test/corpus/default/cur/20:2,
similarity index 100%
rename from test/corpus/cur/20:2,
rename to test/corpus/default/cur/20:2,
diff --git a/test/corpus/cur/21:2, b/test/corpus/default/cur/21:2,
similarity index 100%
rename from test/corpus/cur/21:2,
rename to test/corpus/default/cur/21:2,
diff --git a/test/corpus/cur/22:2, b/test/corpus/default/cur/22:2,
similarity index 100%
rename from test/corpus/cur/22:2,
rename to test/corpus/default/cur/22:2,
diff --git a/test/corpus/cur/23:2, b/test/corpus/default/cur/23:2,
similarity index 100%
rename from test/corpus/cur/23:2,
rename to test/corpus/default/cur/23:2,
diff --git a/test/corpus/cur/24:2, b/test/corpus/default/cur/24:2,
similarity index 100%
rename from test/corpus/cur/24:2,
rename to test/corpus/default/cur/24:2,
diff --git a/test/corpus/cur/25:2, b/test/corpus/default/cur/25:2,
similarity index 100%
rename from test/corpus/cur/25:2,
rename to test/corpus/default/cur/25:2,
diff --git a/test/corpus/cur/26:2, b/test/corpus/default/cur/26:2,
similarity index 100%
rename from test/corpus/cur/26:2,
rename to test/corpus/default/cur/26:2,
diff --git a/test/corpus/cur/27:2, b/test/corpus/default/cur/27:2,
similarity index 100%
rename from test/corpus/cur/27:2,
rename to test/corpus/default/cur/27:2,
diff --git a/test/corpus/cur/28:2, b/test/corpus/default/cur/28:2,
similarity index 100%
rename from test/corpus/cur/28:2,
rename to test/corpus/default/cur/28:2,
diff --git a/test/corpus/cur/29:2, b/test/corpus/default/cur/29:2,
similarity index 100%
rename from test/corpus/cur/29:2,
rename to test/corpus/default/cur/29:2,
diff --git a/test/corpus/cur/30:2, b/test/corpus/default/cur/30:2,
similarity index 100%
rename from test/corpus/cur/30:2,
rename to test/corpus/default/cur/30:2,
diff --git a/test/corpus/cur/31:2, b/test/corpus/default/cur/31:2,
similarity index 100%
rename from test/corpus/cur/31:2,
rename to test/corpus/default/cur/31:2,
diff --git a/test/corpus/cur/32:2, b/test/corpus/default/cur/32:2,
similarity index 100%
rename from test/corpus/cur/32:2,
rename to test/corpus/default/cur/32:2,
diff --git a/test/corpus/cur/33:2, b/test/corpus/default/cur/33:2,
similarity index 100%
rename from test/corpus/cur/33:2,
rename to test/corpus/default/cur/33:2,
diff --git a/test/corpus/cur/34:2, b/test/corpus/default/cur/34:2,
similarity index 100%
rename from test/corpus/cur/34:2,
rename to test/corpus/default/cur/34:2,
diff --git a/test/corpus/cur/35:2, b/test/corpus/default/cur/35:2,
similarity index 100%
rename from test/corpus/cur/35:2,
rename to test/corpus/default/cur/35:2,
diff --git a/test/corpus/cur/36:2, b/test/corpus/default/cur/36:2,
similarity index 100%
rename from test/corpus/cur/36:2,
rename to test/corpus/default/cur/36:2,
diff --git a/test/corpus/cur/37:2, b/test/corpus/default/cur/37:2,
similarity index 100%
rename from test/corpus/cur/37:2,
rename to test/corpus/default/cur/37:2,
diff --git a/test/corpus/cur/38:2, b/test/corpus/default/cur/38:2,
similarity index 100%
rename from test/corpus/cur/38:2,
rename to test/corpus/default/cur/38:2,
diff --git a/test/corpus/cur/39:2, b/test/corpus/default/cur/39:2,
similarity index 100%
rename from test/corpus/cur/39:2,
rename to test/corpus/default/cur/39:2,
diff --git a/test/corpus/cur/40:2, b/test/corpus/default/cur/40:2,
similarity index 100%
rename from test/corpus/cur/40:2,
rename to test/corpus/default/cur/40:2,
diff --git a/test/corpus/cur/41:2, b/test/corpus/default/cur/41:2,
similarity index 100%
rename from test/corpus/cur/41:2,
rename to test/corpus/default/cur/41:2,
diff --git a/test/corpus/cur/42:2, b/test/corpus/default/cur/42:2,
similarity index 100%
rename from test/corpus/cur/42:2,
rename to test/corpus/default/cur/42:2,
diff --git a/test/corpus/cur/43:2, b/test/corpus/default/cur/43:2,
similarity index 100%
rename from test/corpus/cur/43:2,
rename to test/corpus/default/cur/43:2,
diff --git a/test/corpus/cur/44:2, b/test/corpus/default/cur/44:2,
similarity index 100%
rename from test/corpus/cur/44:2,
rename to test/corpus/default/cur/44:2,
diff --git a/test/corpus/cur/45:2, b/test/corpus/default/cur/45:2,
similarity index 100%
rename from test/corpus/cur/45:2,
rename to test/corpus/default/cur/45:2,
diff --git a/test/corpus/cur/46:2, b/test/corpus/default/cur/46:2,
similarity index 100%
rename from test/corpus/cur/46:2,
rename to test/corpus/default/cur/46:2,
diff --git a/test/corpus/cur/47:2, b/test/corpus/default/cur/47:2,
similarity index 100%
rename from test/corpus/cur/47:2,
rename to test/corpus/default/cur/47:2,
diff --git a/test/corpus/cur/48:2, b/test/corpus/default/cur/48:2,
similarity index 100%
rename from test/corpus/cur/48:2,
rename to test/corpus/default/cur/48:2,
diff --git a/test/corpus/cur/49:2, b/test/corpus/default/cur/49:2,
similarity index 100%
rename from test/corpus/cur/49:2,
rename to test/corpus/default/cur/49:2,
diff --git a/test/corpus/cur/50:2, b/test/corpus/default/cur/50:2,
similarity index 100%
rename from test/corpus/cur/50:2,
rename to test/corpus/default/cur/50:2,
diff --git a/test/corpus/cur/51:2, b/test/corpus/default/cur/51:2,
similarity index 100%
rename from test/corpus/cur/51:2,
rename to test/corpus/default/cur/51:2,
diff --git a/test/corpus/cur/52:2, b/test/corpus/default/cur/52:2,
similarity index 100%
rename from test/corpus/cur/52:2,
rename to test/corpus/default/cur/52:2,
diff --git a/test/corpus/cur/53:2, b/test/corpus/default/cur/53:2,
similarity index 100%
rename from test/corpus/cur/53:2,
rename to test/corpus/default/cur/53:2,
diff --git a/test/notmuch-test b/test/notmuch-test
index b8437127c01f..6598f428d14f 100755
--- a/test/notmuch-test
+++ b/test/notmuch-test
@@ -49,6 +49,6 @@ echo
 ev=$?
 
 # Clean up
-rm -rf test-results corpus.mail
+rm -rf test-results corpus.mail.*
 
 exit $ev
diff --git a/test/test-lib.sh b/test/test-lib.sh
index 78af170fe739..69070f26e4fe 100644
--- a/test/test-lib.sh
+++ b/test/test-lib.sh
@@ -516,19 +516,24 @@ emacs_fcc_message ()
 
 # Generate a corpus of email and add it to the database.
 #
-# This corpus is fixed, (it happens to be 50 messages from early in
-# the history of the notmuch mailing list), which allows for reliably
-# testing commands that need to operate on a not-totally-trivial
-# number of messages.
+# The first parameter defines the corpus to pick under the corpus
+# directory, defaulting to "default".
+#
+# The default corpus is fixed, (it happens to be 50 messages from
+# early in the history of the notmuch mailing list), which allows for
+# reliably testing commands that need to operate on a
+# not-totally-trivial number of messages.
 add_email_corpus ()
 {
+    local corpus=${1:-default}
+
     rm -rf ${MAIL_DIR}
-    if [ -d $TEST_DIRECTORY/corpus.mail ]; then
-	cp -a $TEST_DIRECTORY/corpus.mail ${MAIL_DIR}
+    if [ -d $TEST_DIRECTORY/corpus.mail.${corpus} ]; then
+	cp -a $TEST_DIRECTORY/corpus.mail.${corpus} ${MAIL_DIR}
     else
-	cp -a $TEST_DIRECTORY/corpus ${MAIL_DIR}
+	cp -a $TEST_DIRECTORY/corpus/${corpus} ${MAIL_DIR}
 	notmuch new >/dev/null
-	cp -a ${MAIL_DIR} $TEST_DIRECTORY/corpus.mail
+	cp -a ${MAIL_DIR} $TEST_DIRECTORY/corpus.mail.${corpus}
     fi
 }
 
-- 
1.8.5.3

[PATCH v2 07/13] test: add new corpus with folders

[Jani Nikula]

---

corpus content dropped to save bandwidth, see

https://gitorious.org/jani/notmuch/commit/3aff18b55de46e3b27748376a59cda41b6cd7f6d

 test/corpus/folders/01:2,             |  34 ++++
 test/corpus/folders/02:2,             |  32 +++
 test/corpus/folders/bar/17:2,         |  23 +++
 test/corpus/folders/bar/18:2,         |  12 ++
 test/corpus/folders/bar/baz/05:2,     | 104 ++++++++++
 test/corpus/folders/bar/baz/23:2,     | 145 ++++++++++++++
 test/corpus/folders/bar/baz/24:2,     | 204 +++++++++++++++++++
 test/corpus/folders/bar/baz/cur/25:2, |  32 +++
 test/corpus/folders/bar/baz/cur/26:2, | 121 ++++++++++++
 test/corpus/folders/bar/baz/new/27:2, |  21 ++
 test/corpus/folders/bar/baz/new/28:2, |  38 ++++
 test/corpus/folders/bar/cur/19:2,     | 360 ++++++++++++++++++++++++++++++++++
 test/corpus/folders/bar/cur/20:2,     | 101 ++++++++++
 test/corpus/folders/bar/new/21:2,     | 102 ++++++++++
 test/corpus/folders/bar/new/22:2,     |  84 ++++++++
 test/corpus/folders/cur/29:2,         |  21 ++
 test/corpus/folders/cur/30:2,         |  75 +++++++
 test/corpus/folders/cur/31:2,         |  31 +++
 test/corpus/folders/cur/32:2,         | 165 ++++++++++++++++
 test/corpus/folders/cur/33:2,         |  13 ++
 test/corpus/folders/cur/34:2,         |  46 +++++
 test/corpus/folders/cur/35:2,         |  24 +++
 test/corpus/folders/cur/36:2,         |  25 +++
 test/corpus/folders/cur/37:2,         |  22 +++
 test/corpus/folders/cur/38:2,         |  40 ++++
 test/corpus/folders/cur/39:2,         |  32 +++
 test/corpus/folders/cur/40:2,         |  31 +++
 test/corpus/folders/cur/41:2,         |  37 ++++
 test/corpus/folders/cur/42:2,         |  30 +++
 test/corpus/folders/cur/43:2,         |  26 +++
 test/corpus/folders/cur/44:2,         |  29 +++
 test/corpus/folders/cur/45:2,         |  41 ++++
 test/corpus/folders/cur/46:2,         |  57 ++++++
 test/corpus/folders/cur/47:2,         |  84 ++++++++
 test/corpus/folders/cur/48:2,         |  17 ++
 test/corpus/folders/cur/49:2,         |  33 ++++
 test/corpus/folders/cur/50:2,         |  39 ++++
 test/corpus/folders/cur/52:2,         |  39 ++++
 test/corpus/folders/cur/53:2,         |  20 ++
 test/corpus/folders/foo/05:2,         | 104 ++++++++++
 test/corpus/folders/foo/06:2,         |  36 ++++
 test/corpus/folders/foo/baz/11:2,     |  27 +++
 test/corpus/folders/foo/baz/12:2,     |  27 +++
 test/corpus/folders/foo/baz/cur/13:2, | 178 +++++++++++++++++
 test/corpus/folders/foo/baz/cur/14:2, |  39 ++++
 test/corpus/folders/foo/baz/new/15:2, |  22 +++
 test/corpus/folders/foo/baz/new/16:2, |  27 +++
 test/corpus/folders/foo/cur/07:2,     |  57 ++++++
 test/corpus/folders/foo/cur/08:2,     |  87 ++++++++
 test/corpus/folders/foo/new/03:2,     |  93 +++++++++
 test/corpus/folders/foo/new/09:2,     |  33 ++++
 test/corpus/folders/foo/new/10:2,     |  54 +++++
 test/corpus/folders/new/03:2,         |  93 +++++++++
 test/corpus/folders/new/04:2,         |  84 ++++++++
 54 files changed, 3351 insertions(+)
 create mode 100644 test/corpus/folders/01:2,
 create mode 100644 test/corpus/folders/02:2,
 create mode 100644 test/corpus/folders/bar/17:2,
 create mode 100644 test/corpus/folders/bar/18:2,
 create mode 100644 test/corpus/folders/bar/baz/05:2,
 create mode 100644 test/corpus/folders/bar/baz/23:2,
 create mode 100644 test/corpus/folders/bar/baz/24:2,
 create mode 100644 test/corpus/folders/bar/baz/cur/25:2,
 create mode 100644 test/corpus/folders/bar/baz/cur/26:2,
 create mode 100644 test/corpus/folders/bar/baz/new/27:2,
 create mode 100644 test/corpus/folders/bar/baz/new/28:2,
 create mode 100644 test/corpus/folders/bar/cur/19:2,
 create mode 100644 test/corpus/folders/bar/cur/20:2,
 create mode 100644 test/corpus/folders/bar/new/21:2,
 create mode 100644 test/corpus/folders/bar/new/22:2,
 create mode 100644 test/corpus/folders/cur/29:2,
 create mode 100644 test/corpus/folders/cur/30:2,
 create mode 100644 test/corpus/folders/cur/31:2,
 create mode 100644 test/corpus/folders/cur/32:2,
 create mode 100644 test/corpus/folders/cur/33:2,
 create mode 100644 test/corpus/folders/cur/34:2,
 create mode 100644 test/corpus/folders/cur/35:2,
 create mode 100644 test/corpus/folders/cur/36:2,
 create mode 100644 test/corpus/folders/cur/37:2,
 create mode 100644 test/corpus/folders/cur/38:2,
 create mode 100644 test/corpus/folders/cur/39:2,
 create mode 100644 test/corpus/folders/cur/40:2,
 create mode 100644 test/corpus/folders/cur/41:2,
 create mode 100644 test/corpus/folders/cur/42:2,
 create mode 100644 test/corpus/folders/cur/43:2,
 create mode 100644 test/corpus/folders/cur/44:2,
 create mode 100644 test/corpus/folders/cur/45:2,
 create mode 100644 test/corpus/folders/cur/46:2,
 create mode 100644 test/corpus/folders/cur/47:2,
 create mode 100644 test/corpus/folders/cur/48:2,
 create mode 100644 test/corpus/folders/cur/49:2,
 create mode 100644 test/corpus/folders/cur/50:2,
 create mode 100644 test/corpus/folders/cur/52:2,
 create mode 100644 test/corpus/folders/cur/53:2,
 create mode 100644 test/corpus/folders/foo/05:2,
 create mode 100644 test/corpus/folders/foo/06:2,
 create mode 100644 test/corpus/folders/foo/baz/11:2,
 create mode 100644 test/corpus/folders/foo/baz/12:2,
 create mode 100644 test/corpus/folders/foo/baz/cur/13:2,
 create mode 100644 test/corpus/folders/foo/baz/cur/14:2,
 create mode 100644 test/corpus/folders/foo/baz/new/15:2,
 create mode 100644 test/corpus/folders/foo/baz/new/16:2,
 create mode 100644 test/corpus/folders/foo/cur/07:2,
 create mode 100644 test/corpus/folders/foo/cur/08:2,
 create mode 100644 test/corpus/folders/foo/new/03:2,
 create mode 100644 test/corpus/folders/foo/new/09:2,
 create mode 100644 test/corpus/folders/foo/new/10:2,
 create mode 100644 test/corpus/folders/new/03:2,
 create mode 100644 test/corpus/folders/new/04:2,

-- 
1.8.5.3

[PATCH v2 08/13] test: add tests for the new boolean folder: and path: prefixes

[Jani Nikula]

Additional tests for the boolean folder: and path: prefixes.
---
 test/T101-search-by-folder-and-path.sh | 83 ++++++++++++++++++++++++++++++++++
 1 file changed, 83 insertions(+)
 create mode 100755 test/T101-search-by-folder-and-path.sh

diff --git a/test/T101-search-by-folder-and-path.sh b/test/T101-search-by-folder-and-path.sh
new file mode 100755
index 000000000000..9f809e46e110
--- /dev/null
+++ b/test/T101-search-by-folder-and-path.sh
@@ -0,0 +1,83 @@
+#!/usr/bin/env bash
+test_description='"notmuch search" by folder: and path:'
+. ./test-lib.sh
+
+add_email_corpus folders
+
+test_begin_subtest "folder: search"
+output=$(notmuch search --output=files folder:foo | sed -e "s,$MAIL_DIR,MAIL_DIR," | sort)
+# bar/baz/05:2, and new/03:2, are duplicates of foo/05:2, and
+# foo/new/03:2, respectively
+test_expect_equal "$output" "MAIL_DIR/bar/baz/05:2,
+MAIL_DIR/foo/05:2,
+MAIL_DIR/foo/06:2,
+MAIL_DIR/foo/cur/07:2,
+MAIL_DIR/foo/cur/08:2,
+MAIL_DIR/foo/new/03:2,
+MAIL_DIR/foo/new/09:2,
+MAIL_DIR/foo/new/10:2,
+MAIL_DIR/new/03:2,"
+
+test_begin_subtest "top level folder: search"
+output=$(notmuch search --output=files folder:'""' | sed -e "s,$MAIL_DIR,MAIL_DIR," | sort)
+# foo/new/03:2, is a duplicate of new/03:2,
+test_expect_equal "$output" "MAIL_DIR/01:2,
+MAIL_DIR/02:2,
+MAIL_DIR/cur/29:2,
+MAIL_DIR/cur/30:2,
+MAIL_DIR/cur/31:2,
+MAIL_DIR/cur/32:2,
+MAIL_DIR/cur/33:2,
+MAIL_DIR/cur/34:2,
+MAIL_DIR/cur/35:2,
+MAIL_DIR/cur/36:2,
+MAIL_DIR/cur/37:2,
+MAIL_DIR/cur/38:2,
+MAIL_DIR/cur/39:2,
+MAIL_DIR/cur/40:2,
+MAIL_DIR/cur/41:2,
+MAIL_DIR/cur/42:2,
+MAIL_DIR/cur/43:2,
+MAIL_DIR/cur/44:2,
+MAIL_DIR/cur/45:2,
+MAIL_DIR/cur/46:2,
+MAIL_DIR/cur/47:2,
+MAIL_DIR/cur/48:2,
+MAIL_DIR/cur/49:2,
+MAIL_DIR/cur/50:2,
+MAIL_DIR/cur/52:2,
+MAIL_DIR/cur/53:2,
+MAIL_DIR/foo/new/03:2,
+MAIL_DIR/new/03:2,
+MAIL_DIR/new/04:2,"
+
+test_begin_subtest "path: search"
+output=$(notmuch search --output=files path:"bar" | sed -e "s,$MAIL_DIR,MAIL_DIR," | sort)
+# foo/05:2, is a duplicate of bar/baz/05:2,
+test_expect_equal "$output" "MAIL_DIR/bar/17:2,
+MAIL_DIR/bar/18:2,"
+
+test_begin_subtest "top level path: search"
+output=$(notmuch search --output=files path:'""' | sed -e "s,$MAIL_DIR,MAIL_DIR," | sort)
+test_expect_equal "$output" "MAIL_DIR/01:2,
+MAIL_DIR/02:2,"
+
+test_begin_subtest "recursive path: search"
+output=$(notmuch search --output=files path:"bar/**" | sed -e "s,$MAIL_DIR,MAIL_DIR," | sort)
+# foo/05:2, is a duplicate of bar/baz/05:2,
+test_expect_equal "$output" "MAIL_DIR/bar/17:2,
+MAIL_DIR/bar/18:2,
+MAIL_DIR/bar/baz/05:2,
+MAIL_DIR/bar/baz/23:2,
+MAIL_DIR/bar/baz/24:2,
+MAIL_DIR/bar/baz/cur/25:2,
+MAIL_DIR/bar/baz/cur/26:2,
+MAIL_DIR/bar/baz/new/27:2,
+MAIL_DIR/bar/baz/new/28:2,
+MAIL_DIR/bar/cur/19:2,
+MAIL_DIR/bar/cur/20:2,
+MAIL_DIR/bar/new/21:2,
+MAIL_DIR/bar/new/22:2,
+MAIL_DIR/foo/05:2,"
+
+test_done
-- 
1.8.5.3

[PATCH v2 09/13] devel: add script to generate test databases

[Jani Nikula]

Add script to generate notmuch test databases using specified versions
of notmuch. This is useful for generating material for database
upgrade tests.

This reuses the test infrastructure to have a sandbox environment for
notmuch new etc.
---
 devel/gen-testdb.sh | 124 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 124 insertions(+)
 create mode 100755 devel/gen-testdb.sh

diff --git a/devel/gen-testdb.sh b/devel/gen-testdb.sh
new file mode 100755
index 000000000000..c291dfff98c9
--- /dev/null
+++ b/devel/gen-testdb.sh
@@ -0,0 +1,124 @@
+#!/usr/bin/env bash
+#
+# NAME
+#	gen-testdb.sh - generate test databases
+#
+# SYNOPSIS
+#	gen-testdb.sh -v NOTMUCH-VERSION [-c CORPUS-PATH] [-s TAR-SUFFIX]
+#
+# DESCRIPTION
+#	Generate a tarball containing the specified test corpus and
+#	the corresponding notmuch database, indexed using a specific
+#	version of notmuch, resulting in a specific version of the
+#	database.
+#
+#	The specific version of notmuch will be built on the fly.
+#	Therefore the script must be run within a git repository to be
+#	able to build the old versions of notmuch.
+#
+#	This script reuses the test infrastructure, and the script
+#	must be run from within the test directory.
+#
+#	The output tarballs, named database-<TAR-SUFFIX>.tar.gz, are
+#	placed in the test/test-databases directory.
+#
+# OPTIONS
+#	-v NOTMUCH-VERSION
+#		Notmuch version in terms of a git tag or commit to use
+#		for generating the database. Required.
+#
+#	-c CORPUS-PATH
+#		Path to a corpus to use for generating the
+#		database. Due to CWD changes within the test
+#		infrastructure, use absolute paths. Defaults to the
+#		test corpus.
+#
+#	-s TAR-SUFFIX
+#		Suffix for the tarball basename. Empty by default.
+#
+# EXAMPLE
+#
+#	Generate a database indexed with notmuch 0.17. Use the default
+#	test corpus. Name the tarball database-v1.tar.gz to reflect
+#	the fact that notmuch 0.17 used database version 1.
+#
+#	$ cd test
+#	$ ../devel/gen-testdb.sh -v 0.17 -s v1
+#
+# CAVEATS
+#	Test infrastructure options won't work.
+#
+#	Any existing databases with the same name will be overwritten.
+#
+#	It may not be possible to build old versions of notmuch with
+#	the set of dependencies that satisfy building the current
+#	version of notmuch.
+#
+# AUTHOR
+#	Jani Nikula <jani@nikula.org>
+#
+# LICENSE
+#	Same as notmuch test infrastructure (GPLv2+).
+#
+
+test_description="database generation abusing test infrastructure"
+
+# immediate exit on subtest failure; see test_failure_ in test-lib.sh
+immediate=t
+
+VERSION=
+CORPUS=
+SUFFIX=
+
+while getopts v:c:s: opt; do
+    case "$opt" in
+	v) VERSION="$OPTARG";;
+	c) CORPUS="$OPTARG";;
+	s) SUFFIX="-$OPTARG";;
+    esac
+done
+shift `expr $OPTIND - 1`
+
+. ./test-lib.sh
+
+CORPUS=${CORPUS:-${TEST_DIRECTORY}/corpus}
+
+test_expect_code 0 "notmuch version specified on the command line" \
+    "test -n ${VERSION}"
+
+test_expect_code 0 "the specified version ${VERSION} refers to a commit" \
+    "git show ${VERSION} >/dev/null 2>&1"
+
+BUILD_DIR="notmuch-${VERSION}"
+test_expect_code 0 "generate snapshot of notmuch version ${VERSION}" \
+    "git -C $TEST_DIRECTORY/.. archive --prefix=${BUILD_DIR}/ --format=tar ${VERSION} | tar x"
+
+# force version string
+git describe --match '[0-9.]*' ${VERSION} > ${BUILD_DIR}/version
+
+test_expect_code 0 "configure and build notmuch version ${VERSION}" \
+    "make -C ${BUILD_DIR}"
+
+# use the newly built notmuch
+export PATH=./${BUILD_DIR}:$PATH
+
+test_begin_subtest "verify the newly built notmuch version"
+test_expect_equal "`notmuch --version`" "notmuch `cat ${BUILD_DIR}/version`"
+
+# replace the existing mails, if any, with the specified corpus
+rm -rf ${MAIL_DIR}
+cp -a ${CORPUS} ${MAIL_DIR}
+
+test_expect_code 0 "index the corpus" \
+    "notmuch new"
+
+# finally, wrap the resulting mail store and database in a tarball
+DBNAME=database${SUFFIX}
+cp -a ${MAIL_DIR} ${TMP_DIRECTORY}/${DBNAME}
+tar zcf ${TMP_DIRECTORY}/${DBNAME}.tar.gz -C ${TMP_DIRECTORY} ${DBNAME}
+mkdir -p  ${TEST_DIRECTORY}/test-databases
+cp -a ${TMP_DIRECTORY}/${DBNAME}.tar.gz ${TEST_DIRECTORY}/test-databases
+test_expect_code 0 "create the output tarball ${DBNAME}.tar.gz" \
+    "test -f ${TEST_DIRECTORY}/test-databases/${DBNAME}.tar.gz"
+
+test_done
-- 
1.8.5.3

[PATCH v2 10/13] test: add test database in format version 1

[Jani Nikula]

Generated using:

$ cd test
$ ../devel/gen-testdb.sh -v 0.17 -c $PWD/corpus/folders -s v1

This database contains old style probabilistic folder: terms and no
path: terms.
---

corpus/database content dropped to save bandwidth, see

https://gitorious.org/jani/notmuch/commit/88bcffe31441e62d19769a2a941a2965f8c91486

 test/test-databases/README             |   5 +++++
 test/test-databases/database-v1.tar.gz | Bin 0 -> 262063 bytes
 2 files changed, 5 insertions(+)
 create mode 100644 test/test-databases/README
 create mode 100644 test/test-databases/database-v1.tar.gz

diff --git a/test/test-databases/README b/test/test-databases/README
new file mode 100644
index 000000000000..af5defe80728
--- /dev/null
+++ b/test/test-databases/README
@@ -0,1 +1,5 @@
+Notmuch test databases
+======================
+
+This directory contains pre-generated databases with their source
+corpus, chiefly for the purpose of testing database upgrade.
-- 
1.8.5.3

[PATCH v2 11/13] test: add database upgrade test from format version 1 to 2

[Jani Nikula]

Test the upgrade from probabilistic to boolean folder: terms, and
addition of path: terms.
---
 test/T530-upgrade.sh | 103 +++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 103 insertions(+)
 create mode 100755 test/T530-upgrade.sh

diff --git a/test/T530-upgrade.sh b/test/T530-upgrade.sh
new file mode 100755
index 000000000000..cf9914e380be
--- /dev/null
+++ b/test/T530-upgrade.sh
@@ -0,0 +1,103 @@
+#!/usr/bin/env bash
+test_description="database upgrade"
+
+. ./test-lib.sh
+
+tar zxf $TEST_DIRECTORY/test-databases/database-v1.tar.gz -C ${MAIL_DIR} --strip-components=1
+
+test_begin_subtest "folder: search does not work with old database version"
+output=$(notmuch search folder:foo)
+test_expect_equal "$output" ""
+
+test_begin_subtest "path: search does not work with old database version"
+output=$(notmuch search path:foo)
+test_expect_equal "$output" ""
+
+test_begin_subtest "database upgrade from format version 1"
+output=$(notmuch new)
+test_expect_equal "$output" "\
+Welcome to a new version of notmuch! Your database will now be upgraded.
+Your notmuch database has now been upgraded to database format version 2.
+No new mail."
+
+test_begin_subtest "folder: no longer matches in the middle of path"
+output=$(notmuch search folder:baz)
+test_expect_equal "$output" ""
+
+test_begin_subtest "folder: search"
+output=$(notmuch search --output=files folder:foo | sed -e "s,$MAIL_DIR,MAIL_DIR," | sort)
+# bar/baz/05:2, and new/03:2, are duplicates of foo/05:2, and
+# foo/new/03:2, respectively
+test_expect_equal "$output" "MAIL_DIR/bar/baz/05:2,
+MAIL_DIR/foo/05:2,
+MAIL_DIR/foo/06:2,
+MAIL_DIR/foo/cur/07:2,
+MAIL_DIR/foo/cur/08:2,
+MAIL_DIR/foo/new/03:2,
+MAIL_DIR/foo/new/09:2,
+MAIL_DIR/foo/new/10:2,
+MAIL_DIR/new/03:2,"
+
+test_begin_subtest "top level folder: search"
+output=$(notmuch search --output=files folder:'""' | sed -e "s,$MAIL_DIR,MAIL_DIR," | sort)
+# foo/new/03:2, is a duplicate of new/03:2,
+test_expect_equal "$output" "MAIL_DIR/01:2,
+MAIL_DIR/02:2,
+MAIL_DIR/cur/29:2,
+MAIL_DIR/cur/30:2,
+MAIL_DIR/cur/31:2,
+MAIL_DIR/cur/32:2,
+MAIL_DIR/cur/33:2,
+MAIL_DIR/cur/34:2,
+MAIL_DIR/cur/35:2,
+MAIL_DIR/cur/36:2,
+MAIL_DIR/cur/37:2,
+MAIL_DIR/cur/38:2,
+MAIL_DIR/cur/39:2,
+MAIL_DIR/cur/40:2,
+MAIL_DIR/cur/41:2,
+MAIL_DIR/cur/42:2,
+MAIL_DIR/cur/43:2,
+MAIL_DIR/cur/44:2,
+MAIL_DIR/cur/45:2,
+MAIL_DIR/cur/46:2,
+MAIL_DIR/cur/47:2,
+MAIL_DIR/cur/48:2,
+MAIL_DIR/cur/49:2,
+MAIL_DIR/cur/50:2,
+MAIL_DIR/cur/52:2,
+MAIL_DIR/cur/53:2,
+MAIL_DIR/foo/new/03:2,
+MAIL_DIR/new/03:2,
+MAIL_DIR/new/04:2,"
+
+test_begin_subtest "path: search"
+output=$(notmuch search --output=files path:"bar" | sed -e "s,$MAIL_DIR,MAIL_DIR," | sort)
+# foo/05:2, is a duplicate of bar/baz/05:2,
+test_expect_equal "$output" "MAIL_DIR/bar/17:2,
+MAIL_DIR/bar/18:2,"
+
+test_begin_subtest "top level path: search"
+output=$(notmuch search --output=files path:'""' | sed -e "s,$MAIL_DIR,MAIL_DIR," | sort)
+test_expect_equal "$output" "MAIL_DIR/01:2,
+MAIL_DIR/02:2,"
+
+test_begin_subtest "recursive path: search"
+output=$(notmuch search --output=files path:"bar/**" | sed -e "s,$MAIL_DIR,MAIL_DIR," | sort)
+# foo/05:2, is a duplicate of bar/baz/05:2,
+test_expect_equal "$output" "MAIL_DIR/bar/17:2,
+MAIL_DIR/bar/18:2,
+MAIL_DIR/bar/baz/05:2,
+MAIL_DIR/bar/baz/23:2,
+MAIL_DIR/bar/baz/24:2,
+MAIL_DIR/bar/baz/cur/25:2,
+MAIL_DIR/bar/baz/cur/26:2,
+MAIL_DIR/bar/baz/new/27:2,
+MAIL_DIR/bar/baz/new/28:2,
+MAIL_DIR/bar/cur/19:2,
+MAIL_DIR/bar/cur/20:2,
+MAIL_DIR/bar/new/21:2,
+MAIL_DIR/bar/new/22:2,
+MAIL_DIR/foo/05:2,"
+
+test_done
-- 
1.8.5.3

[PATCH v2 12/13] man: update man pages for folder: and path: search terms

[Jani Nikula]

---
 man/man7/notmuch-search-terms.7 | 28 ++++++++++++++++++++++------
 1 file changed, 22 insertions(+), 6 deletions(-)

diff --git a/man/man7/notmuch-search-terms.7 b/man/man7/notmuch-search-terms.7
index a768b630a4d1..907403dd6f2e 100644
--- a/man/man7/notmuch-search-terms.7
+++ b/man/man7/notmuch-search-terms.7
@@ -54,6 +54,8 @@ terms to match against specific portions of an email, (where
 
 	folder:<directory-path>
 
+	path:<directory-path> or path:<directory-path>/**
+
 	date:<since>..<until>
 
 The
@@ -101,12 +103,26 @@ thread ID values can be seen in the first column of output from
 
 The
 .B folder:
-prefix can be used to search for email message files that are
-contained within particular directories within the mail store. If the
-same email message has multiple message files associated with it, it's
-sufficient for a match that at least one of the files is contained
-within a matching directory. Only the directory components below the
-top-level mail database path are available to be searched.
+and
+.B path:
+prefixes can be used to search for email message files that are
+contained within particular directories within the mail store. The
+directories are specified relative from the top-level mail database
+path, and thus only the directory components below that are available
+to be searched.
+
+The
+.B folder:
+prefix matches messages in the specified maildir folder, i.e. in the
+specified directory and its "new" and "cur" subdirectories. The
+.B path:
+prefix matches messages in the specified directory only, unless the
+"/**" suffix is used to denote the specified directory and all its
+subdirectories recursively. For both, the empty string "" matches the
+top level maildir folder or directory. If the same email message has
+multiple message files associated with it, it's sufficient for a match
+that at least one of the files is contained within a matching
+directory.
 
 The
 .B date:
-- 
1.8.5.3

[PATCH v2 13/13] man: try to clarify the folder: and path: vs. --output=files confusion

[Jani Nikula]

---
 man/man1/notmuch-search.1 | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)

diff --git a/man/man1/notmuch-search.1 b/man/man1/notmuch-search.1
index 55a81e79fce4..a2b1ae43f411 100644
--- a/man/man1/notmuch-search.1
+++ b/man/man1/notmuch-search.1
@@ -82,8 +82,14 @@ one per line (\-\-format=text), separated by null characters
 S-Expression list (\-\-format=sexp).
 
 Note that each message may have multiple filenames associated with it.
-All of them are included in the output, unless limited with the
-\-\-duplicate=N option.
+All of them are included in the output (unless limited with the
+\-\-duplicate=N option). This may be particularly confusing for
+.B folder:
+or
+.B path:
+searches in a specified directory, as the messages may have duplicates
+in other directories that are included in the output, although these
+files alone would not match the search.
 .RE
 .RS 4
 .TP 4
-- 
1.8.5.3

Re: [PATCH v2 00/13] literal folder: prefix, new path: prefix

[Mark Walters]

I have read most of this series, tested it and run the tests and LGTM +1.

I read the C code fairly carefully, the tests rather less so but they
looked sane, and I didn't really look at patch 9 for building old
databases.

Best wishes

Mark




On Sat, 22 Feb 2014, Jani Nikula <jani@nikula.org> wrote:
> Hi all, this is v2 of id:cover.1389304779.git.jani@nikula.org.
>
> The new path: prefix is a literal boolean prefix matching the paths,
> relative from the maildir root, of the message files. There's no
> interpretation of the maildir special cur/new folders, but a recursive
> match is provided with "/**" suffix. See the patch for details.
>
> The folder: prefix becomes a literal boolean prefix, similar to path:,
> except it matches the maildir cur/new folders in addition to the
> specified path. There's no recursive version.
>
> Patches 1-5 add the above.
>
> Patches 6-8 change the test infrastructure to make it easier to add
> multiple corpuses, and adds a new test for path: and folder:.
>
> Patches 9-11 add support for testing the database upgrade.
>
> Patches 12-13 update man pages.
>
>
> I've dropped most of the content in patches 7 and 10 due to their
> size. The patches (and the whole series) are available in the
> boolean-folder-and-path-v2 branch at
> git://gitorious.org/jani/notmuch.git. Web interface at
> https://gitorious.org/jani/notmuch/commits/0b3dd2d1cc6c413ea07ea326883ac448499c0e79.
>
>
> WARNING! The change requires a database format version bump, and a
> database upgrade, which is automatically done on 'notmuch new'. The
> upgrade is irreversible if you want to try this on your database! A
> complete database rebuild is required for reverting the database format
> version. Make sure your backups are in order!
>
>
> BR,
> Jani.
>
>
> Jani Nikula (13):
>   lib: refactor folder term update after filename removal
>   lib: add support for path: prefix searches
>   test: make insert test use the path: prefix
>   lib: make folder: prefix literal
>   test: fix test for literal folder: search
>   test: make it possible to have several corpora
>   test: add new corpus with folders
>   test: add tests for the new boolean folder: and path: prefixes
>   devel: add script to generate test databases
>   test: add test database in format version 1
>   test: add database upgrade test from format version 1 to 2
>   man: update man pages for folder: and path: search terms
>   man: try to clarify the folder: and path: vs. --output=files confusion
>
>  devel/gen-testdb.sh                    | 124 ++++++++++++
>  lib/database.cc                        |  45 ++++-
>  lib/message.cc                         | 249 ++++++++++++++++-------
>  lib/notmuch-private.h                  |   3 +
>  man/man1/notmuch-search.1              |  10 +-
>  man/man7/notmuch-search-terms.7        |  28 ++-
>  test/.gitignore                        |   2 +-
>  test/Makefile.local                    |   2 +-
>  test/T070-insert.sh                    |  10 +-
>  test/T100-search-by-folder.sh          |  24 ++-
>  test/T101-search-by-folder-and-path.sh |  83 ++++++++
>  test/T480-hex-escaping.sh              |   4 +-
>  test/T530-upgrade.sh                   | 103 ++++++++++
>  test/corpus/{ => default}/cur/01:2,    |   0
>  test/corpus/{ => default}/cur/02:2,    |   0
>  test/corpus/{ => default}/cur/03:2,    |   0
>  test/corpus/{ => default}/cur/04:2,    |   0
>  test/corpus/{ => default}/cur/05:2,    |   0
>  test/corpus/{ => default}/cur/06:2,    |   0
>  test/corpus/{ => default}/cur/07:2,    |   0
>  test/corpus/{ => default}/cur/08:2,    |   0
>  test/corpus/{ => default}/cur/09:2,    |   0
>  test/corpus/{ => default}/cur/10:2,    |   0
>  test/corpus/{ => default}/cur/11:2,    |   0
>  test/corpus/{ => default}/cur/12:2,    |   0
>  test/corpus/{ => default}/cur/13:2,    |   0
>  test/corpus/{ => default}/cur/14:2,    |   0
>  test/corpus/{ => default}/cur/15:2,    |   0
>  test/corpus/{ => default}/cur/16:2,    |   0
>  test/corpus/{ => default}/cur/17:2,    |   0
>  test/corpus/{ => default}/cur/18:2,    |   0
>  test/corpus/{ => default}/cur/19:2,    |   0
>  test/corpus/{ => default}/cur/20:2,    |   0
>  test/corpus/{ => default}/cur/21:2,    |   0
>  test/corpus/{ => default}/cur/22:2,    |   0
>  test/corpus/{ => default}/cur/23:2,    |   0
>  test/corpus/{ => default}/cur/24:2,    |   0
>  test/corpus/{ => default}/cur/25:2,    |   0
>  test/corpus/{ => default}/cur/26:2,    |   0
>  test/corpus/{ => default}/cur/27:2,    |   0
>  test/corpus/{ => default}/cur/28:2,    |   0
>  test/corpus/{ => default}/cur/29:2,    |   0
>  test/corpus/{ => default}/cur/30:2,    |   0
>  test/corpus/{ => default}/cur/31:2,    |   0
>  test/corpus/{ => default}/cur/32:2,    |   0
>  test/corpus/{ => default}/cur/33:2,    |   0
>  test/corpus/{ => default}/cur/34:2,    |   0
>  test/corpus/{ => default}/cur/35:2,    |   0
>  test/corpus/{ => default}/cur/36:2,    |   0
>  test/corpus/{ => default}/cur/37:2,    |   0
>  test/corpus/{ => default}/cur/38:2,    |   0
>  test/corpus/{ => default}/cur/39:2,    |   0
>  test/corpus/{ => default}/cur/40:2,    |   0
>  test/corpus/{ => default}/cur/41:2,    |   0
>  test/corpus/{ => default}/cur/42:2,    |   0
>  test/corpus/{ => default}/cur/43:2,    |   0
>  test/corpus/{ => default}/cur/44:2,    |   0
>  test/corpus/{ => default}/cur/45:2,    |   0
>  test/corpus/{ => default}/cur/46:2,    |   0
>  test/corpus/{ => default}/cur/47:2,    |   0
>  test/corpus/{ => default}/cur/48:2,    |   0
>  test/corpus/{ => default}/cur/49:2,    |   0
>  test/corpus/{ => default}/cur/50:2,    |   0
>  test/corpus/{ => default}/cur/51:2,    |   0
>  test/corpus/{ => default}/cur/52:2,    |   0
>  test/corpus/{ => default}/cur/53:2,    |   0
>  test/corpus/folders/01:2,              |  34 ++++
>  test/corpus/folders/02:2,              |  32 +++
>  test/corpus/folders/bar/17:2,          |  23 +++
>  test/corpus/folders/bar/18:2,          |  12 ++
>  test/corpus/folders/bar/baz/05:2,      | 104 ++++++++++
>  test/corpus/folders/bar/baz/23:2,      | 145 +++++++++++++
>  test/corpus/folders/bar/baz/24:2,      | 204 +++++++++++++++++++
>  test/corpus/folders/bar/baz/cur/25:2,  |  32 +++
>  test/corpus/folders/bar/baz/cur/26:2,  | 121 +++++++++++
>  test/corpus/folders/bar/baz/new/27:2,  |  21 ++
>  test/corpus/folders/bar/baz/new/28:2,  |  38 ++++
>  test/corpus/folders/bar/cur/19:2,      | 360 +++++++++++++++++++++++++++++++++
>  test/corpus/folders/bar/cur/20:2,      | 101 +++++++++
>  test/corpus/folders/bar/new/21:2,      | 102 ++++++++++
>  test/corpus/folders/bar/new/22:2,      |  84 ++++++++
>  test/corpus/folders/cur/29:2,          |  21 ++
>  test/corpus/folders/cur/30:2,          |  75 +++++++
>  test/corpus/folders/cur/31:2,          |  31 +++
>  test/corpus/folders/cur/32:2,          | 165 +++++++++++++++
>  test/corpus/folders/cur/33:2,          |  13 ++
>  test/corpus/folders/cur/34:2,          |  46 +++++
>  test/corpus/folders/cur/35:2,          |  24 +++
>  test/corpus/folders/cur/36:2,          |  25 +++
>  test/corpus/folders/cur/37:2,          |  22 ++
>  test/corpus/folders/cur/38:2,          |  40 ++++
>  test/corpus/folders/cur/39:2,          |  32 +++
>  test/corpus/folders/cur/40:2,          |  31 +++
>  test/corpus/folders/cur/41:2,          |  37 ++++
>  test/corpus/folders/cur/42:2,          |  30 +++
>  test/corpus/folders/cur/43:2,          |  26 +++
>  test/corpus/folders/cur/44:2,          |  29 +++
>  test/corpus/folders/cur/45:2,          |  41 ++++
>  test/corpus/folders/cur/46:2,          |  57 ++++++
>  test/corpus/folders/cur/47:2,          |  84 ++++++++
>  test/corpus/folders/cur/48:2,          |  17 ++
>  test/corpus/folders/cur/49:2,          |  33 +++
>  test/corpus/folders/cur/50:2,          |  39 ++++
>  test/corpus/folders/cur/52:2,          |  39 ++++
>  test/corpus/folders/cur/53:2,          |  20 ++
>  test/corpus/folders/foo/05:2,          | 104 ++++++++++
>  test/corpus/folders/foo/06:2,          |  36 ++++
>  test/corpus/folders/foo/baz/11:2,      |  27 +++
>  test/corpus/folders/foo/baz/12:2,      |  27 +++
>  test/corpus/folders/foo/baz/cur/13:2,  | 178 ++++++++++++++++
>  test/corpus/folders/foo/baz/cur/14:2,  |  39 ++++
>  test/corpus/folders/foo/baz/new/15:2,  |  22 ++
>  test/corpus/folders/foo/baz/new/16:2,  |  27 +++
>  test/corpus/folders/foo/cur/07:2,      |  57 ++++++
>  test/corpus/folders/foo/cur/08:2,      |  87 ++++++++
>  test/corpus/folders/foo/new/03:2,      |  93 +++++++++
>  test/corpus/folders/foo/new/09:2,      |  33 +++
>  test/corpus/folders/foo/new/10:2,      |  54 +++++
>  test/corpus/folders/new/03:2,          |  93 +++++++++
>  test/corpus/folders/new/04:2,          |  84 ++++++++
>  test/notmuch-test                      |   2 +-
>  test/test-databases/README             |   5 +
>  test/test-databases/database-v1.tar.gz | Bin 0 -> 262063 bytes
>  test/test-lib.sh                       |  21 +-
>  124 files changed, 3954 insertions(+), 112 deletions(-)
>  create mode 100755 devel/gen-testdb.sh
>  create mode 100755 test/T101-search-by-folder-and-path.sh
>  create mode 100755 test/T530-upgrade.sh
>  rename test/corpus/{ => default}/cur/01:2, (100%)
>  rename test/corpus/{ => default}/cur/02:2, (100%)
>  rename test/corpus/{ => default}/cur/03:2, (100%)
>  rename test/corpus/{ => default}/cur/04:2, (100%)
>  rename test/corpus/{ => default}/cur/05:2, (100%)
>  rename test/corpus/{ => default}/cur/06:2, (100%)
>  rename test/corpus/{ => default}/cur/07:2, (100%)
>  rename test/corpus/{ => default}/cur/08:2, (100%)
>  rename test/corpus/{ => default}/cur/09:2, (100%)
>  rename test/corpus/{ => default}/cur/10:2, (100%)
>  rename test/corpus/{ => default}/cur/11:2, (100%)
>  rename test/corpus/{ => default}/cur/12:2, (100%)
>  rename test/corpus/{ => default}/cur/13:2, (100%)
>  rename test/corpus/{ => default}/cur/14:2, (100%)
>  rename test/corpus/{ => default}/cur/15:2, (100%)
>  rename test/corpus/{ => default}/cur/16:2, (100%)
>  rename test/corpus/{ => default}/cur/17:2, (100%)
>  rename test/corpus/{ => default}/cur/18:2, (100%)
>  rename test/corpus/{ => default}/cur/19:2, (100%)
>  rename test/corpus/{ => default}/cur/20:2, (100%)
>  rename test/corpus/{ => default}/cur/21:2, (100%)
>  rename test/corpus/{ => default}/cur/22:2, (100%)
>  rename test/corpus/{ => default}/cur/23:2, (100%)
>  rename test/corpus/{ => default}/cur/24:2, (100%)
>  rename test/corpus/{ => default}/cur/25:2, (100%)
>  rename test/corpus/{ => default}/cur/26:2, (100%)
>  rename test/corpus/{ => default}/cur/27:2, (100%)
>  rename test/corpus/{ => default}/cur/28:2, (100%)
>  rename test/corpus/{ => default}/cur/29:2, (100%)
>  rename test/corpus/{ => default}/cur/30:2, (100%)
>  rename test/corpus/{ => default}/cur/31:2, (100%)
>  rename test/corpus/{ => default}/cur/32:2, (100%)
>  rename test/corpus/{ => default}/cur/33:2, (100%)
>  rename test/corpus/{ => default}/cur/34:2, (100%)
>  rename test/corpus/{ => default}/cur/35:2, (100%)
>  rename test/corpus/{ => default}/cur/36:2, (100%)
>  rename test/corpus/{ => default}/cur/37:2, (100%)
>  rename test/corpus/{ => default}/cur/38:2, (100%)
>  rename test/corpus/{ => default}/cur/39:2, (100%)
>  rename test/corpus/{ => default}/cur/40:2, (100%)
>  rename test/corpus/{ => default}/cur/41:2, (100%)
>  rename test/corpus/{ => default}/cur/42:2, (100%)
>  rename test/corpus/{ => default}/cur/43:2, (100%)
>  rename test/corpus/{ => default}/cur/44:2, (100%)
>  rename test/corpus/{ => default}/cur/45:2, (100%)
>  rename test/corpus/{ => default}/cur/46:2, (100%)
>  rename test/corpus/{ => default}/cur/47:2, (100%)
>  rename test/corpus/{ => default}/cur/48:2, (100%)
>  rename test/corpus/{ => default}/cur/49:2, (100%)
>  rename test/corpus/{ => default}/cur/50:2, (100%)
>  rename test/corpus/{ => default}/cur/51:2, (100%)
>  rename test/corpus/{ => default}/cur/52:2, (100%)
>  rename test/corpus/{ => default}/cur/53:2, (100%)
>  create mode 100644 test/corpus/folders/01:2,
>  create mode 100644 test/corpus/folders/02:2,
>  create mode 100644 test/corpus/folders/bar/17:2,
>  create mode 100644 test/corpus/folders/bar/18:2,
>  create mode 100644 test/corpus/folders/bar/baz/05:2,
>  create mode 100644 test/corpus/folders/bar/baz/23:2,
>  create mode 100644 test/corpus/folders/bar/baz/24:2,
>  create mode 100644 test/corpus/folders/bar/baz/cur/25:2,
>  create mode 100644 test/corpus/folders/bar/baz/cur/26:2,
>  create mode 100644 test/corpus/folders/bar/baz/new/27:2,
>  create mode 100644 test/corpus/folders/bar/baz/new/28:2,
>  create mode 100644 test/corpus/folders/bar/cur/19:2,
>  create mode 100644 test/corpus/folders/bar/cur/20:2,
>  create mode 100644 test/corpus/folders/bar/new/21:2,
>  create mode 100644 test/corpus/folders/bar/new/22:2,
>  create mode 100644 test/corpus/folders/cur/29:2,
>  create mode 100644 test/corpus/folders/cur/30:2,
>  create mode 100644 test/corpus/folders/cur/31:2,
>  create mode 100644 test/corpus/folders/cur/32:2,
>  create mode 100644 test/corpus/folders/cur/33:2,
>  create mode 100644 test/corpus/folders/cur/34:2,
>  create mode 100644 test/corpus/folders/cur/35:2,
>  create mode 100644 test/corpus/folders/cur/36:2,
>  create mode 100644 test/corpus/folders/cur/37:2,
>  create mode 100644 test/corpus/folders/cur/38:2,
>  create mode 100644 test/corpus/folders/cur/39:2,
>  create mode 100644 test/corpus/folders/cur/40:2,
>  create mode 100644 test/corpus/folders/cur/41:2,
>  create mode 100644 test/corpus/folders/cur/42:2,
>  create mode 100644 test/corpus/folders/cur/43:2,
>  create mode 100644 test/corpus/folders/cur/44:2,
>  create mode 100644 test/corpus/folders/cur/45:2,
>  create mode 100644 test/corpus/folders/cur/46:2,
>  create mode 100644 test/corpus/folders/cur/47:2,
>  create mode 100644 test/corpus/folders/cur/48:2,
>  create mode 100644 test/corpus/folders/cur/49:2,
>  create mode 100644 test/corpus/folders/cur/50:2,
>  create mode 100644 test/corpus/folders/cur/52:2,
>  create mode 100644 test/corpus/folders/cur/53:2,
>  create mode 100644 test/corpus/folders/foo/05:2,
>  create mode 100644 test/corpus/folders/foo/06:2,
>  create mode 100644 test/corpus/folders/foo/baz/11:2,
>  create mode 100644 test/corpus/folders/foo/baz/12:2,
>  create mode 100644 test/corpus/folders/foo/baz/cur/13:2,
>  create mode 100644 test/corpus/folders/foo/baz/cur/14:2,
>  create mode 100644 test/corpus/folders/foo/baz/new/15:2,
>  create mode 100644 test/corpus/folders/foo/baz/new/16:2,
>  create mode 100644 test/corpus/folders/foo/cur/07:2,
>  create mode 100644 test/corpus/folders/foo/cur/08:2,
>  create mode 100644 test/corpus/folders/foo/new/03:2,
>  create mode 100644 test/corpus/folders/foo/new/09:2,
>  create mode 100644 test/corpus/folders/foo/new/10:2,
>  create mode 100644 test/corpus/folders/new/03:2,
>  create mode 100644 test/corpus/folders/new/04:2,
>  create mode 100644 test/test-databases/README
>  create mode 100644 test/test-databases/database-v1.tar.gz
>
> -- 
> 1.8.5.3
>
> _______________________________________________
> notmuch mailing list
> notmuch@notmuchmail.org
> http://notmuchmail.org/mailman/listinfo/notmuch

Re: [PATCH v2 00/13] literal folder: prefix, new path: prefix

[Tomi Ollila]

On Sun, Feb 23 2014, Mark Walters <markwalters1009@gmail.com> wrote:

> I have read most of this series, tested it and run the tests and LGTM +1.
>
> I read the C code fairly carefully, the tests rather less so but they
> looked sane, and I didn't really look at patch 9 for building old
> databases.

Well, I can add (at this point) that patch 9 is tolerable...

>
> Best wishes
>
> Mark

Tomi


>
>
>
>
> On Sat, 22 Feb 2014, Jani Nikula <jani@nikula.org> wrote:
>> Hi all, this is v2 of id:cover.1389304779.git.jani@nikula.org.
>>
>> The new path: prefix is a literal boolean prefix matching the paths,
>> relative from the maildir root, of the message files. There's no
>> interpretation of the maildir special cur/new folders, but a recursive
>> match is provided with "/**" suffix. See the patch for details.
>>
>> The folder: prefix becomes a literal boolean prefix, similar to path:,
>> except it matches the maildir cur/new folders in addition to the
>> specified path. There's no recursive version.
>>
>> Patches 1-5 add the above.
>>
>> Patches 6-8 change the test infrastructure to make it easier to add
>> multiple corpuses, and adds a new test for path: and folder:.
>>
>> Patches 9-11 add support for testing the database upgrade.
>>
>> Patches 12-13 update man pages.
>>
>>
>> I've dropped most of the content in patches 7 and 10 due to their
>> size. The patches (and the whole series) are available in the
>> boolean-folder-and-path-v2 branch at
>> git://gitorious.org/jani/notmuch.git. Web interface at
>> https://gitorious.org/jani/notmuch/commits/0b3dd2d1cc6c413ea07ea326883ac448499c0e79.
>>
>>
>> WARNING! The change requires a database format version bump, and a
>> database upgrade, which is automatically done on 'notmuch new'. The
>> upgrade is irreversible if you want to try this on your database! A
>> complete database rebuild is required for reverting the database format
>> version. Make sure your backups are in order!
>>
>>
>> BR,
>> Jani.
>>
>>
>> Jani Nikula (13):
>>   lib: refactor folder term update after filename removal
>>   lib: add support for path: prefix searches
>>   test: make insert test use the path: prefix
>>   lib: make folder: prefix literal
>>   test: fix test for literal folder: search
>>   test: make it possible to have several corpora
>>   test: add new corpus with folders
>>   test: add tests for the new boolean folder: and path: prefixes
>>   devel: add script to generate test databases
>>   test: add test database in format version 1
>>   test: add database upgrade test from format version 1 to 2
>>   man: update man pages for folder: and path: search terms
>>   man: try to clarify the folder: and path: vs. --output=files confusion
>>
>>  devel/gen-testdb.sh                    | 124 ++++++++++++
>>  lib/database.cc                        |  45 ++++-
>>  lib/message.cc                         | 249 ++++++++++++++++-------
>>  lib/notmuch-private.h                  |   3 +
>>  man/man1/notmuch-search.1              |  10 +-
>>  man/man7/notmuch-search-terms.7        |  28 ++-
>>  test/.gitignore                        |   2 +-
>>  test/Makefile.local                    |   2 +-
>>  test/T070-insert.sh                    |  10 +-
>>  test/T100-search-by-folder.sh          |  24 ++-
>>  test/T101-search-by-folder-and-path.sh |  83 ++++++++
>>  test/T480-hex-escaping.sh              |   4 +-
>>  test/T530-upgrade.sh                   | 103 ++++++++++
>>  test/corpus/{ => default}/cur/01:2,    |   0
>>  test/corpus/{ => default}/cur/02:2,    |   0
>>  test/corpus/{ => default}/cur/03:2,    |   0
>>  test/corpus/{ => default}/cur/04:2,    |   0
>>  test/corpus/{ => default}/cur/05:2,    |   0
>>  test/corpus/{ => default}/cur/06:2,    |   0
>>  test/corpus/{ => default}/cur/07:2,    |   0
>>  test/corpus/{ => default}/cur/08:2,    |   0
>>  test/corpus/{ => default}/cur/09:2,    |   0
>>  test/corpus/{ => default}/cur/10:2,    |   0
>>  test/corpus/{ => default}/cur/11:2,    |   0
>>  test/corpus/{ => default}/cur/12:2,    |   0
>>  test/corpus/{ => default}/cur/13:2,    |   0
>>  test/corpus/{ => default}/cur/14:2,    |   0
>>  test/corpus/{ => default}/cur/15:2,    |   0
>>  test/corpus/{ => default}/cur/16:2,    |   0
>>  test/corpus/{ => default}/cur/17:2,    |   0
>>  test/corpus/{ => default}/cur/18:2,    |   0
>>  test/corpus/{ => default}/cur/19:2,    |   0
>>  test/corpus/{ => default}/cur/20:2,    |   0
>>  test/corpus/{ => default}/cur/21:2,    |   0
>>  test/corpus/{ => default}/cur/22:2,    |   0
>>  test/corpus/{ => default}/cur/23:2,    |   0
>>  test/corpus/{ => default}/cur/24:2,    |   0
>>  test/corpus/{ => default}/cur/25:2,    |   0
>>  test/corpus/{ => default}/cur/26:2,    |   0
>>  test/corpus/{ => default}/cur/27:2,    |   0
>>  test/corpus/{ => default}/cur/28:2,    |   0
>>  test/corpus/{ => default}/cur/29:2,    |   0
>>  test/corpus/{ => default}/cur/30:2,    |   0
>>  test/corpus/{ => default}/cur/31:2,    |   0
>>  test/corpus/{ => default}/cur/32:2,    |   0
>>  test/corpus/{ => default}/cur/33:2,    |   0
>>  test/corpus/{ => default}/cur/34:2,    |   0
>>  test/corpus/{ => default}/cur/35:2,    |   0
>>  test/corpus/{ => default}/cur/36:2,    |   0
>>  test/corpus/{ => default}/cur/37:2,    |   0
>>  test/corpus/{ => default}/cur/38:2,    |   0
>>  test/corpus/{ => default}/cur/39:2,    |   0
>>  test/corpus/{ => default}/cur/40:2,    |   0
>>  test/corpus/{ => default}/cur/41:2,    |   0
>>  test/corpus/{ => default}/cur/42:2,    |   0
>>  test/corpus/{ => default}/cur/43:2,    |   0
>>  test/corpus/{ => default}/cur/44:2,    |   0
>>  test/corpus/{ => default}/cur/45:2,    |   0
>>  test/corpus/{ => default}/cur/46:2,    |   0
>>  test/corpus/{ => default}/cur/47:2,    |   0
>>  test/corpus/{ => default}/cur/48:2,    |   0
>>  test/corpus/{ => default}/cur/49:2,    |   0
>>  test/corpus/{ => default}/cur/50:2,    |   0
>>  test/corpus/{ => default}/cur/51:2,    |   0
>>  test/corpus/{ => default}/cur/52:2,    |   0
>>  test/corpus/{ => default}/cur/53:2,    |   0
>>  test/corpus/folders/01:2,              |  34 ++++
>>  test/corpus/folders/02:2,              |  32 +++
>>  test/corpus/folders/bar/17:2,          |  23 +++
>>  test/corpus/folders/bar/18:2,          |  12 ++
>>  test/corpus/folders/bar/baz/05:2,      | 104 ++++++++++
>>  test/corpus/folders/bar/baz/23:2,      | 145 +++++++++++++
>>  test/corpus/folders/bar/baz/24:2,      | 204 +++++++++++++++++++
>>  test/corpus/folders/bar/baz/cur/25:2,  |  32 +++
>>  test/corpus/folders/bar/baz/cur/26:2,  | 121 +++++++++++
>>  test/corpus/folders/bar/baz/new/27:2,  |  21 ++
>>  test/corpus/folders/bar/baz/new/28:2,  |  38 ++++
>>  test/corpus/folders/bar/cur/19:2,      | 360 +++++++++++++++++++++++++++++++++
>>  test/corpus/folders/bar/cur/20:2,      | 101 +++++++++
>>  test/corpus/folders/bar/new/21:2,      | 102 ++++++++++
>>  test/corpus/folders/bar/new/22:2,      |  84 ++++++++
>>  test/corpus/folders/cur/29:2,          |  21 ++
>>  test/corpus/folders/cur/30:2,          |  75 +++++++
>>  test/corpus/folders/cur/31:2,          |  31 +++
>>  test/corpus/folders/cur/32:2,          | 165 +++++++++++++++
>>  test/corpus/folders/cur/33:2,          |  13 ++
>>  test/corpus/folders/cur/34:2,          |  46 +++++
>>  test/corpus/folders/cur/35:2,          |  24 +++
>>  test/corpus/folders/cur/36:2,          |  25 +++
>>  test/corpus/folders/cur/37:2,          |  22 ++
>>  test/corpus/folders/cur/38:2,          |  40 ++++
>>  test/corpus/folders/cur/39:2,          |  32 +++
>>  test/corpus/folders/cur/40:2,          |  31 +++
>>  test/corpus/folders/cur/41:2,          |  37 ++++
>>  test/corpus/folders/cur/42:2,          |  30 +++
>>  test/corpus/folders/cur/43:2,          |  26 +++
>>  test/corpus/folders/cur/44:2,          |  29 +++
>>  test/corpus/folders/cur/45:2,          |  41 ++++
>>  test/corpus/folders/cur/46:2,          |  57 ++++++
>>  test/corpus/folders/cur/47:2,          |  84 ++++++++
>>  test/corpus/folders/cur/48:2,          |  17 ++
>>  test/corpus/folders/cur/49:2,          |  33 +++
>>  test/corpus/folders/cur/50:2,          |  39 ++++
>>  test/corpus/folders/cur/52:2,          |  39 ++++
>>  test/corpus/folders/cur/53:2,          |  20 ++
>>  test/corpus/folders/foo/05:2,          | 104 ++++++++++
>>  test/corpus/folders/foo/06:2,          |  36 ++++
>>  test/corpus/folders/foo/baz/11:2,      |  27 +++
>>  test/corpus/folders/foo/baz/12:2,      |  27 +++
>>  test/corpus/folders/foo/baz/cur/13:2,  | 178 ++++++++++++++++
>>  test/corpus/folders/foo/baz/cur/14:2,  |  39 ++++
>>  test/corpus/folders/foo/baz/new/15:2,  |  22 ++
>>  test/corpus/folders/foo/baz/new/16:2,  |  27 +++
>>  test/corpus/folders/foo/cur/07:2,      |  57 ++++++
>>  test/corpus/folders/foo/cur/08:2,      |  87 ++++++++
>>  test/corpus/folders/foo/new/03:2,      |  93 +++++++++
>>  test/corpus/folders/foo/new/09:2,      |  33 +++
>>  test/corpus/folders/foo/new/10:2,      |  54 +++++
>>  test/corpus/folders/new/03:2,          |  93 +++++++++
>>  test/corpus/folders/new/04:2,          |  84 ++++++++
>>  test/notmuch-test                      |   2 +-
>>  test/test-databases/README             |   5 +
>>  test/test-databases/database-v1.tar.gz | Bin 0 -> 262063 bytes
>>  test/test-lib.sh                       |  21 +-
>>  124 files changed, 3954 insertions(+), 112 deletions(-)
>>  create mode 100755 devel/gen-testdb.sh
>>  create mode 100755 test/T101-search-by-folder-and-path.sh
>>  create mode 100755 test/T530-upgrade.sh
>>  rename test/corpus/{ => default}/cur/01:2, (100%)
>>  rename test/corpus/{ => default}/cur/02:2, (100%)
>>  rename test/corpus/{ => default}/cur/03:2, (100%)
>>  rename test/corpus/{ => default}/cur/04:2, (100%)
>>  rename test/corpus/{ => default}/cur/05:2, (100%)
>>  rename test/corpus/{ => default}/cur/06:2, (100%)
>>  rename test/corpus/{ => default}/cur/07:2, (100%)
>>  rename test/corpus/{ => default}/cur/08:2, (100%)
>>  rename test/corpus/{ => default}/cur/09:2, (100%)
>>  rename test/corpus/{ => default}/cur/10:2, (100%)
>>  rename test/corpus/{ => default}/cur/11:2, (100%)
>>  rename test/corpus/{ => default}/cur/12:2, (100%)
>>  rename test/corpus/{ => default}/cur/13:2, (100%)
>>  rename test/corpus/{ => default}/cur/14:2, (100%)
>>  rename test/corpus/{ => default}/cur/15:2, (100%)
>>  rename test/corpus/{ => default}/cur/16:2, (100%)
>>  rename test/corpus/{ => default}/cur/17:2, (100%)
>>  rename test/corpus/{ => default}/cur/18:2, (100%)
>>  rename test/corpus/{ => default}/cur/19:2, (100%)
>>  rename test/corpus/{ => default}/cur/20:2, (100%)
>>  rename test/corpus/{ => default}/cur/21:2, (100%)
>>  rename test/corpus/{ => default}/cur/22:2, (100%)
>>  rename test/corpus/{ => default}/cur/23:2, (100%)
>>  rename test/corpus/{ => default}/cur/24:2, (100%)
>>  rename test/corpus/{ => default}/cur/25:2, (100%)
>>  rename test/corpus/{ => default}/cur/26:2, (100%)
>>  rename test/corpus/{ => default}/cur/27:2, (100%)
>>  rename test/corpus/{ => default}/cur/28:2, (100%)
>>  rename test/corpus/{ => default}/cur/29:2, (100%)
>>  rename test/corpus/{ => default}/cur/30:2, (100%)
>>  rename test/corpus/{ => default}/cur/31:2, (100%)
>>  rename test/corpus/{ => default}/cur/32:2, (100%)
>>  rename test/corpus/{ => default}/cur/33:2, (100%)
>>  rename test/corpus/{ => default}/cur/34:2, (100%)
>>  rename test/corpus/{ => default}/cur/35:2, (100%)
>>  rename test/corpus/{ => default}/cur/36:2, (100%)
>>  rename test/corpus/{ => default}/cur/37:2, (100%)
>>  rename test/corpus/{ => default}/cur/38:2, (100%)
>>  rename test/corpus/{ => default}/cur/39:2, (100%)
>>  rename test/corpus/{ => default}/cur/40:2, (100%)
>>  rename test/corpus/{ => default}/cur/41:2, (100%)
>>  rename test/corpus/{ => default}/cur/42:2, (100%)
>>  rename test/corpus/{ => default}/cur/43:2, (100%)
>>  rename test/corpus/{ => default}/cur/44:2, (100%)
>>  rename test/corpus/{ => default}/cur/45:2, (100%)
>>  rename test/corpus/{ => default}/cur/46:2, (100%)
>>  rename test/corpus/{ => default}/cur/47:2, (100%)
>>  rename test/corpus/{ => default}/cur/48:2, (100%)
>>  rename test/corpus/{ => default}/cur/49:2, (100%)
>>  rename test/corpus/{ => default}/cur/50:2, (100%)
>>  rename test/corpus/{ => default}/cur/51:2, (100%)
>>  rename test/corpus/{ => default}/cur/52:2, (100%)
>>  rename test/corpus/{ => default}/cur/53:2, (100%)
>>  create mode 100644 test/corpus/folders/01:2,
>>  create mode 100644 test/corpus/folders/02:2,
>>  create mode 100644 test/corpus/folders/bar/17:2,
>>  create mode 100644 test/corpus/folders/bar/18:2,
>>  create mode 100644 test/corpus/folders/bar/baz/05:2,
>>  create mode 100644 test/corpus/folders/bar/baz/23:2,
>>  create mode 100644 test/corpus/folders/bar/baz/24:2,
>>  create mode 100644 test/corpus/folders/bar/baz/cur/25:2,
>>  create mode 100644 test/corpus/folders/bar/baz/cur/26:2,
>>  create mode 100644 test/corpus/folders/bar/baz/new/27:2,
>>  create mode 100644 test/corpus/folders/bar/baz/new/28:2,
>>  create mode 100644 test/corpus/folders/bar/cur/19:2,
>>  create mode 100644 test/corpus/folders/bar/cur/20:2,
>>  create mode 100644 test/corpus/folders/bar/new/21:2,
>>  create mode 100644 test/corpus/folders/bar/new/22:2,
>>  create mode 100644 test/corpus/folders/cur/29:2,
>>  create mode 100644 test/corpus/folders/cur/30:2,
>>  create mode 100644 test/corpus/folders/cur/31:2,
>>  create mode 100644 test/corpus/folders/cur/32:2,
>>  create mode 100644 test/corpus/folders/cur/33:2,
>>  create mode 100644 test/corpus/folders/cur/34:2,
>>  create mode 100644 test/corpus/folders/cur/35:2,
>>  create mode 100644 test/corpus/folders/cur/36:2,
>>  create mode 100644 test/corpus/folders/cur/37:2,
>>  create mode 100644 test/corpus/folders/cur/38:2,
>>  create mode 100644 test/corpus/folders/cur/39:2,
>>  create mode 100644 test/corpus/folders/cur/40:2,
>>  create mode 100644 test/corpus/folders/cur/41:2,
>>  create mode 100644 test/corpus/folders/cur/42:2,
>>  create mode 100644 test/corpus/folders/cur/43:2,
>>  create mode 100644 test/corpus/folders/cur/44:2,
>>  create mode 100644 test/corpus/folders/cur/45:2,
>>  create mode 100644 test/corpus/folders/cur/46:2,
>>  create mode 100644 test/corpus/folders/cur/47:2,
>>  create mode 100644 test/corpus/folders/cur/48:2,
>>  create mode 100644 test/corpus/folders/cur/49:2,
>>  create mode 100644 test/corpus/folders/cur/50:2,
>>  create mode 100644 test/corpus/folders/cur/52:2,
>>  create mode 100644 test/corpus/folders/cur/53:2,
>>  create mode 100644 test/corpus/folders/foo/05:2,
>>  create mode 100644 test/corpus/folders/foo/06:2,
>>  create mode 100644 test/corpus/folders/foo/baz/11:2,
>>  create mode 100644 test/corpus/folders/foo/baz/12:2,
>>  create mode 100644 test/corpus/folders/foo/baz/cur/13:2,
>>  create mode 100644 test/corpus/folders/foo/baz/cur/14:2,
>>  create mode 100644 test/corpus/folders/foo/baz/new/15:2,
>>  create mode 100644 test/corpus/folders/foo/baz/new/16:2,
>>  create mode 100644 test/corpus/folders/foo/cur/07:2,
>>  create mode 100644 test/corpus/folders/foo/cur/08:2,
>>  create mode 100644 test/corpus/folders/foo/new/03:2,
>>  create mode 100644 test/corpus/folders/foo/new/09:2,
>>  create mode 100644 test/corpus/folders/foo/new/10:2,
>>  create mode 100644 test/corpus/folders/new/03:2,
>>  create mode 100644 test/corpus/folders/new/04:2,
>>  create mode 100644 test/test-databases/README
>>  create mode 100644 test/test-databases/database-v1.tar.gz
>>
>> -- 
>> 1.8.5.3
>>
>> _______________________________________________
>> notmuch mailing list
>> notmuch@notmuchmail.org
>> http://notmuchmail.org/mailman/listinfo/notmuch
> _______________________________________________
> notmuch mailing list
> notmuch@notmuchmail.org
> http://notmuchmail.org/mailman/listinfo/notmuch

Re: [PATCH v2 05/13] test: fix test for literal folder: search

[David Bremner]

Jani Nikula <jani@nikula.org> writes:

> Some of the folder: matching capabilities are lost in the
> probabilistic to boolean prefix change. Fix them.

the first 5 look copacetic.

d

Re: [PATCH v2 00/13] literal folder: prefix, new path: prefix

[David Bremner]

Jani Nikula <jani@nikula.org> writes:
>
> I've dropped most of the content in patches 7 and 10 due to their
> size. The patches (and the whole series) are available in the
> boolean-folder-and-path-v2 branch at
> git://gitorious.org/jani/notmuch.git. Web interface at
> https://gitorious.org/jani/notmuch/commits/0b3dd2d1cc6c413ea07ea326883ac448499c0e79.
>

Hi Jani;

I put a some proposed changes at 

  http://pivot.cs.unb.ca/git?p=notmuch.git;a=shortlog;h=refs/heads/boolean-folder-and-path-v2

After playing with it a bit, I decided to 

      - leave the source for the corpuses in the main tree. As we
        discussed, we don't expect them to change much, and they are
        text files.

      - store the binary database blobs externally. We could make the
        same argument about infrequent changes here, except that even
        the tiniest change will in practice require a whole new copy be
        stored.

        - compute a checksum file
        - test: add machinery to fetch and publish test databases
        - test: commit database-v1.tar.xz checksum, ignore actual database

       - this is an independant bug fix that should probably be pushed anyway:

         - test: don't use $(dir) in recipes.

      - I also made some unrelated changes including switching to xz
        compression. I intentionally didn't squash my patches into yours
        so you see and decide

        - convert to XZ compression
        - convert test to XZ

        Originally I thought metadata in the database tarball is
        needed. It turns out not. I think it's harmless, but feel free
        to drop it.

        - add some metadata to the database tarball
        
      - I struggled to find a way that would allow people to manage the
        tarballs in a nice way while not requiring everyone to
        install/learn new tools. My conclusion is in
      
        - test: add support for git annex managing test databases

We still need to think about what to do for people installing from
tarballs. As it stands they will need to either have network access or
skip the relevant tests. Probably at least one followup patch is needed
here. The alternative would be to include the database tarballs in the
source; this will pretty quickly double the size of the source tarball.

Re: [PATCH v2 00/13] literal folder: prefix, new path: prefix

[Jani Nikula]

On Sun, 02 Mar 2014, David Bremner <david@tethera.net> wrote:
> Jani Nikula <jani@nikula.org> writes:
>>
>> I've dropped most of the content in patches 7 and 10 due to their
>> size. The patches (and the whole series) are available in the
>> boolean-folder-and-path-v2 branch at
>> git://gitorious.org/jani/notmuch.git. Web interface at
>> https://gitorious.org/jani/notmuch/commits/0b3dd2d1cc6c413ea07ea326883ac448499c0e79.
>>
>
> Hi Jani;
>
> I put a some proposed changes at 
>
>   http://pivot.cs.unb.ca/git?p=notmuch.git;a=shortlog;h=refs/heads/boolean-folder-and-path-v2
>
> After playing with it a bit, I decided to 
>
>       - leave the source for the corpuses in the main tree. As we
>         discussed, we don't expect them to change much, and they are
>         text files.
>
>       - store the binary database blobs externally. We could make the
>         same argument about infrequent changes here, except that even
>         the tiniest change will in practice require a whole new copy be
>         stored.
>
>         - compute a checksum file
>         - test: add machinery to fetch and publish test databases
>         - test: commit database-v1.tar.xz checksum, ignore actual database
>
>        - this is an independant bug fix that should probably be pushed anyway:
>
>          - test: don't use $(dir) in recipes.
>
>       - I also made some unrelated changes including switching to xz
>         compression. I intentionally didn't squash my patches into yours
>         so you see and decide
>
>         - convert to XZ compression
>         - convert test to XZ
>
>         Originally I thought metadata in the database tarball is
>         needed. It turns out not. I think it's harmless, but feel free
>         to drop it.
>
>         - add some metadata to the database tarball

I'm fine with all of the above.

>       - I struggled to find a way that would allow people to manage the
>         tarballs in a nice way while not requiring everyone to
>         install/learn new tools. My conclusion is in
>       
>         - test: add support for git annex managing test databases
>
> We still need to think about what to do for people installing from
> tarballs. As it stands they will need to either have network access or
> skip the relevant tests. Probably at least one followup patch is needed
> here. The alternative would be to include the database tarballs in the
> source; this will pretty quickly double the size of the source tarball.

I think we should use the prereq mechanism in the tests to check if a
required test database is in place, skipping the test if not. This
should be fairly straightforward to do, and does not care *how* the test
database is to be fetched. (I'm undecided whether the test should verify
the checksum, or the part that fetches the database. Minor detail.)

I know git annex in principle, but don't have any practical experience
with it. I don't really have any ideas for that.


BR,
Jani.

Re: [PATCH v2 00/13] literal folder: prefix, new path: prefix

[David Bremner]

Jani Nikula <jani@nikula.org> writes:

>
> I think we should use the prereq mechanism in the tests to check if a
> required test database is in place, skipping the test if not. This
> should be fairly straightforward to do, and does not care *how* the test
> database is to be fetched. (I'm undecided whether the test should verify
> the checksum, or the part that fetches the database. Minor detail.)

Yep, prereqs sounds fine.  SHA256 verification is pretty fast (on the
scale of test suite operations), so I'd lean to doing it late, but
I could be convinced otherwise.

>
> I know git annex in principle, but don't have any practical experience
> with it. I don't really have any ideas for that.

Well, if you don't mind the rest of the setup (SHA256 sums checked,
tarballs fetched if needed), we can forge ahead for now.

d

Re: [PATCH v2 00/13] literal folder: prefix, new path: prefix

[Tomi Ollila]

On Tue, Mar 04 2014, David Bremner <david@tethera.net> wrote:

> Jani Nikula <jani@nikula.org> writes:
>
>>
>> I think we should use the prereq mechanism in the tests to check if a
>> required test database is in place, skipping the test if not. This
>> should be fairly straightforward to do, and does not care *how* the test
>> database is to be fetched. (I'm undecided whether the test should verify
>> the checksum, or the part that fetches the database. Minor detail.)
>
> Yep, prereqs sounds fine.  SHA256 verification is pretty fast (on the
> scale of test suite operations), so I'd lean to doing it late, but
> I could be convinced otherwise.

I decline commenting on prereqs... ;/

>>
>> I know git annex in principle, but don't have any practical experience
>> with it. I don't really have any ideas for that.
>
> Well, if you don't mind the rest of the setup (SHA256 sums checked,
> tarballs fetched if needed), we can forge ahead for now.

I can contribute script with interface 

./fetch-and-check.sh [--verify-only] destdir url sha256sum 


>
> d


Tomi

Re: [PATCH v2 00/13] literal folder: prefix, new path: prefix

[David Bremner]

Tomi Ollila <tomi.ollila@iki.fi> writes:

>
> I can contribute script with interface 
>
> ./fetch-and-check.sh [--verify-only] destdir url sha256sum 
>

This script is basically embedded in 


http://pivot.cs.unb.ca/git?p=notmuch.git;a=commitdiff;h=f93e32b11e517e160c755355de55ab465206dc62

except that I rebased away  the verification in the actual test :(

Something like 

diff --git a/test/T530-upgrade.sh b/test/T530-upgrade.sh
index 3c0134f..768242d 100755
--- a/test/T530-upgrade.sh
+++ b/test/T530-upgrade.sh
@@ -5,6 +5,10 @@ test_description="database upgrade"
 
 dbtarball=database-v1.tar.xz
 
+cd $TEST_DIRECTORY/test-databases
+sha256sum --quiet --check ${dbtarball}.sha256 || error "checksum of database failed"
+cd - >& /dev/null
+
 tar Jxf $TEST_DIRECTORY/test-databases/${dbtarball} -C ${MAIL_DIR} --strip-component
 
 test_begin_subtest "folder: search does not work with old database version"

Re: [PATCH v2 00/13] literal folder: prefix, new path: prefix

[Tomi Ollila]

On Wed, Mar 05 2014, David Bremner <david@tethera.net> wrote:

> Tomi Ollila <tomi.ollila@iki.fi> writes:
>
>>
>> I can contribute script with interface 
>>
>> ./fetch-and-check.sh [--verify-only] destdir url sha256sum 
>>
>
> This script is basically embedded in 
>
>
> http://pivot.cs.unb.ca/git?p=notmuch.git;a=commitdiff;h=f93e32b11e517e160c755355de55ab465206dc62
>
> except that I rebased away  the verification in the actual test :(
>
> Something like 
>
> diff --git a/test/T530-upgrade.sh b/test/T530-upgrade.sh
> index 3c0134f..768242d 100755
> --- a/test/T530-upgrade.sh
> +++ b/test/T530-upgrade.sh
> @@ -5,6 +5,10 @@ test_description="database upgrade"
>  
>  dbtarball=database-v1.tar.xz
>  
> +cd $TEST_DIRECTORY/test-databases
> +sha256sum --quiet --check ${dbtarball}.sha256 || error "checksum of database failed"
> +cd - >& /dev/null

more portable alternative:

read sha256sum_exp < ${dbtarball}.sha256
#sha256sum_act=`openssl sha256 "${dbtarball}" | sed 's/ .*//'`
sha256sum_act=`openssl sha256 "${dbtarball}"`
test "$sha256sum_exp" = "$sha256sum_act" || error "checksum of database failed"

(IIRC openssl sha256 is available at least in Mac OS X, probably in other
BSD:s & Solaris too)


> +
>  tar Jxf $TEST_DIRECTORY/test-databases/${dbtarball} -C ${MAIL_DIR} --strip-component
>  
>  test_begin_subtest "folder: search does not work with old database version"

Re: [PATCH v2 00/13] literal folder: prefix, new path: prefix

[David Bremner]

Tomi Ollila <tomi.ollila@iki.fi> writes:
>
> (IIRC openssl sha256 is available at least in Mac OS X, probably in other
> BSD:s & Solaris too)
>

OTOH sha256sum is in coreutils, so everywhere with a gnu userland has
it. Some might not have openssl (it's priority optional on Debian,
e.g.).

d

Re: [PATCH v2 00/13] literal folder: prefix, new path: prefix

[Tomi Ollila]

On Wed, Mar 05 2014, David Bremner <david@tethera.net> wrote:

> Tomi Ollila <tomi.ollila@iki.fi> writes:
>>
>> (IIRC openssl sha256 is available at least in Mac OS X, probably in other
>> BSD:s & Solaris too)
>>
>
> OTOH sha256sum is in coreutils, so everywhere with a gnu userland has
> it. Some might not have openssl (it's priority optional on Debian,
> e.g.).

Aaah these response times are a pleasure... :D

cd $TEST_DIRECTORY/test-databases

if hash sha256sum 2>/dev/null
then
     sha256sum --quiet --check ${dbtarball}.sha256 || error "checksum of database failed"
else
     # opportunistically try to run openssl sha256 in case sha256sum is not available
     read sha256sum_exp filename < ${dbtarball}.sha256
     sha256sum_act=`openssl sha256 "${dbtarball}" | sed 's/ .*//'`
     test "$sha256sum_exp" = "$sha256sum_act" || error "checksum of database failed"
fi
cd - >& /dev/null


>
> d

Tomi