Revision tracking, round 2

Revision tracking, round 2

[David Bremner]

There is lots to tidy up here, but the series has balooned to 12 patches since since Austin posted

      id:1413181203-1676-1-git-send-email-aclements@csail.mit.edu

So I figure I better post it to get some feedback.

These are unmodified from Austin's post
 [WIP2 01/12] lib: Only sync modified message documents
 [WIP2 02/12] lib: Add per-message last modification tracking
 [WIP2 07/12] lib: Add "lastmod:" queries for filtering by last
 
This has two small bug fixes, a typo and an extra * removed.
 [WIP2 03/12] lib: API to retrieve database revision and UUID

Together, the next 5  patches implement output format 3
 [WIP2 04/12] cli: add type introspection to sprinter type
 [WIP2 05/12] cli: add two sprinter utility functions
 [WIP2 06/12] cli/show: add extra element to structured output for
 [WIP2 08/12] cli/show: add lastmod to structured output
 [WIP2 09/12] cli/search: add  metadata element to structured output

These two do a trivial convesion of parts of the emacs interface to use the new format:
 [WIP2 10/12] emacs: convert notmuch-search to format-version 3
 [WIP2 11/12] emacs: convert notmuch-tree to format-version 3

Finally, add db-revision argument. I'm not sure about the name, but I
went with what Austin had written.  This patch should probably be earlier in the sequence 

 [WIP2 12/12] cli: add global option "--db-revision"

[WIP2 01/12] lib: Only sync modified message documents

[David Bremner]

From: Austin Clements <amdragon@mit.edu>

Previously, we updated the database copy of a message on every call to
_notmuch_message_sync, even if nothing had changed.  In particular,
this always happens on a thaw, so a freeze/thaw pair with no
modifications between still caused a database update.

We only modify message documents in a handful of places, so keep track
of whether the document has been modified and only sync it when
necessary.  This will be particularly important when we add message
revision tracking.
---
 lib/message.cc | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/lib/message.cc b/lib/message.cc
index 5bc7aff..1ddce3c 100644
--- a/lib/message.cc
+++ b/lib/message.cc
@@ -43,6 +43,9 @@ struct visible _notmuch_message {
      * if each flag has been initialized. */
     unsigned long lazy_flags;
 
+    /* Message document modified since last sync */
+    notmuch_bool_t modified;
+
     Xapian::Document doc;
     Xapian::termcount termpos;
 };
@@ -539,6 +542,7 @@ _notmuch_message_remove_terms (notmuch_message_t *message, const char *prefix)
 
 	try {
 	    message->doc.remove_term ((*i));
+	    message->modified = TRUE;
 	} catch (const Xapian::InvalidArgumentError) {
 	    /* Ignore failure to remove non-existent term. */
 	}
@@ -793,6 +797,7 @@ void
 _notmuch_message_clear_data (notmuch_message_t *message)
 {
     message->doc.set_data ("");
+    message->modified = TRUE;
 }
 
 static void
@@ -990,6 +995,7 @@ _notmuch_message_set_header_values (notmuch_message_t *message,
 			    Xapian::sortable_serialise (time_value));
     message->doc.add_value (NOTMUCH_VALUE_FROM, from);
     message->doc.add_value (NOTMUCH_VALUE_SUBJECT, subject);
+    message->modified = TRUE;
 }
 
 /* Synchronize changes made to message->doc out into the database. */
@@ -1001,8 +1007,12 @@ _notmuch_message_sync (notmuch_message_t *message)
     if (message->notmuch->mode == NOTMUCH_DATABASE_MODE_READ_ONLY)
 	return;
 
+    if (! message->modified)
+	return;
+
     db = static_cast <Xapian::WritableDatabase *> (message->notmuch->xapian_db);
     db->replace_document (message->doc_id, message->doc);
+    message->modified = FALSE;
 }
 
 /* Delete a message document from the database. */
@@ -1077,6 +1087,7 @@ _notmuch_message_add_term (notmuch_message_t *message,
 	return NOTMUCH_PRIVATE_STATUS_TERM_TOO_LONG;
 
     message->doc.add_term (term, 0);
+    message->modified = TRUE;
 
     talloc_free (term);
 
@@ -1145,6 +1156,7 @@ _notmuch_message_remove_term (notmuch_message_t *message,
 
     try {
 	message->doc.remove_term (term);
+	message->modified = TRUE;
     } catch (const Xapian::InvalidArgumentError) {
 	/* We'll let the philosopher's try to wrestle with the
 	 * question of whether failing to remove that which was not
-- 
2.1.4

[WIP2 02/12] lib: Add per-message last modification tracking

[David Bremner]

From: Austin Clements <amdragon@mit.edu>

This adds a new document value that stores the revision of the last
modification to message metadata, where the revision number increases
monotonically with each database commit.

An alternative would be to store the wall-clock time of the last
modification of each message.  In principle this is simpler and has
the advantage that any process can determine the current timestamp
without support from libnotmuch.  However, even assuming a computer's
clock never goes backward and ignoring clock skew in networked
environments, this has a fatal flaw.  Xapian uses (optimistic)
snapshot isolation, which means reads can be concurrent with writes.
Given this, consider the following time line with a write and two read
transactions:

   write  |-X-A--------------|
   read 1       |---B---|
   read 2                      |---|

The write transaction modifies message X and records the wall-clock
time of the modification at A.  The writer hangs around for a while
and later commits its change.  Read 1 is concurrent with the write, so
it doesn't see the change to X.  It does some query and records the
wall-clock time of its results at B.  Transaction read 2 later starts
after the write commits and queries for changes since wall-clock time
B (say the reads are performing an incremental backup).  Even though
read 1 could not see the change to X, read 2 is told (correctly) that
X has not changed since B, the time of the last read.  In fact, X
changed before wall-clock time A, but the change was not visible until
*after* wall-clock time B, so read 2 misses the change to X.

This is tricky to solve in full-blown snapshot isolation, but because
Xapian serializes writes, we can use a simple, monotonically
increasing database revision number.  Furthermore, maintaining this
revision number requires no more IO than a wall-clock time solution
because Xapian already maintains statistics on the upper (and lower)
bound of each value stream.
---
 lib/database-private.h | 16 +++++++++++++++-
 lib/database.cc        | 49 +++++++++++++++++++++++++++++++++++++++++++++++--
 lib/message.cc         | 22 ++++++++++++++++++++++
 lib/notmuch-private.h  | 10 +++++++++-
 4 files changed, 93 insertions(+), 4 deletions(-)

diff --git a/lib/database-private.h b/lib/database-private.h
index 24243db..5c5a2bb 100644
--- a/lib/database-private.h
+++ b/lib/database-private.h
@@ -100,6 +100,12 @@ enum _notmuch_features {
      *
      * Introduced: version 3. */
     NOTMUCH_FEATURE_INDEXED_MIMETYPES = 1 << 5,
+
+    /* If set, messages store the revision number of the last
+     * modification in NOTMUCH_VALUE_LAST_MOD.
+     *
+     * Introduced: version 3. */
+    NOTMUCH_FEATURE_LAST_MOD = 1 << 6,
 };
 
 /* In C++, a named enum is its own type, so define bitwise operators
@@ -145,6 +151,8 @@ struct _notmuch_database {
 
     notmuch_database_mode_t mode;
     int atomic_nesting;
+    /* TRUE if changes have been made in this atomic section */
+    notmuch_bool_t atomic_dirty;
     Xapian::Database *xapian_db;
 
     /* Bit mask of features used by this database.  This is a
@@ -158,6 +166,11 @@ struct _notmuch_database {
      * next library call. May be NULL */
     char *status_string;
 
+    /* Highest committed revision number.  Modifications are recorded
+     * under a higher revision number, which can be generated with
+     * notmuch_database_new_revision. */
+    unsigned long revision;
+
     Xapian::QueryParser *query_parser;
     Xapian::TermGenerator *term_gen;
     Xapian::ValueRangeProcessor *value_range_processor;
@@ -179,7 +192,8 @@ struct _notmuch_database {
  * will have it). */
 #define NOTMUCH_FEATURES_CURRENT \
     (NOTMUCH_FEATURE_FILE_TERMS | NOTMUCH_FEATURE_DIRECTORY_DOCS | \
-     NOTMUCH_FEATURE_BOOL_FOLDER | NOTMUCH_FEATURE_GHOSTS)
+     NOTMUCH_FEATURE_BOOL_FOLDER | NOTMUCH_FEATURE_GHOSTS | \
+     NOTMUCH_FEATURE_LAST_MOD)
 
 /* Return the list of terms from the given iterator matching a prefix.
  * The prefix will be stripped from the strings in the returned list.
diff --git a/lib/database.cc b/lib/database.cc
index cffab62..4378979 100644
--- a/lib/database.cc
+++ b/lib/database.cc
@@ -101,6 +101,9 @@ typedef struct {
  *
  *	SUBJECT:	The value of the "Subject" header
  *
+ *	LAST_MOD:	The revision number as of the last tag or
+ *			filename change.
+ *
  * In addition, terms from the content of the message are added with
  * "from", "to", "attachment", and "subject" prefixes for use by the
  * user in searching. Similarly, terms from the path of the mail
@@ -310,6 +313,8 @@ static const struct {
      * them. */
     { NOTMUCH_FEATURE_INDEXED_MIMETYPES,
       "indexed MIME types", "w"},
+    { NOTMUCH_FEATURE_LAST_MOD,
+      "modification tracking", "w"},
 };
 
 const char *
@@ -729,6 +734,23 @@ _notmuch_database_ensure_writable (notmuch_database_t *notmuch)
     return NOTMUCH_STATUS_SUCCESS;
 }
 
+/* Allocate a revision number for the next change. */
+unsigned long
+_notmuch_database_new_revision (notmuch_database_t *notmuch)
+{
+    unsigned long new_revision = notmuch->revision + 1;
+
+    /* If we're in an atomic section, hold off on updating the
+     * committed revision number until we commit the atomic section.
+     */
+    if (notmuch->atomic_nesting)
+	notmuch->atomic_dirty = TRUE;
+    else
+	notmuch->revision = new_revision;
+
+    return new_revision;
+}
+
 /* Parse a database features string from the given database version.
  * Returns the feature bit set.
  *
@@ -890,6 +912,7 @@ notmuch_database_open_verbose (const char *path,
     notmuch->atomic_nesting = 0;
     try {
 	string last_thread_id;
+	string last_mod;
 
 	if (mode == NOTMUCH_DATABASE_MODE_READ_WRITE) {
 	    notmuch->xapian_db = new Xapian::WritableDatabase (xapian_path,
@@ -948,6 +971,14 @@ notmuch_database_open_verbose (const char *path,
 		INTERNAL_ERROR ("Malformed database last_thread_id: %s", str);
 	}
 
+	/* Get current highest revision number. */
+	last_mod = notmuch->xapian_db->get_value_upper_bound (
+	    NOTMUCH_VALUE_LAST_MOD);
+	if (last_mod.empty ())
+	    notmuch->revision = 0;
+	else
+	    notmuch->revision = Xapian::sortable_unserialise (last_mod);
+
 	notmuch->query_parser = new Xapian::QueryParser;
 	notmuch->term_gen = new Xapian::TermGenerator;
 	notmuch->term_gen->set_stemmer (Xapian::Stem ("english"));
@@ -1355,7 +1386,8 @@ notmuch_database_upgrade (notmuch_database_t *notmuch,
 
     /* Figure out how much total work we need to do. */
     if (new_features &
-	(NOTMUCH_FEATURE_FILE_TERMS | NOTMUCH_FEATURE_BOOL_FOLDER)) {
+	(NOTMUCH_FEATURE_FILE_TERMS | NOTMUCH_FEATURE_BOOL_FOLDER |
+	 NOTMUCH_FEATURE_LAST_MOD)) {
 	notmuch_query_t *query = notmuch_query_create (notmuch, "");
 	total += notmuch_query_count_messages (query);
 	notmuch_query_destroy (query);
@@ -1382,7 +1414,8 @@ notmuch_database_upgrade (notmuch_database_t *notmuch,
 
     /* Perform per-message upgrades. */
     if (new_features &
-	(NOTMUCH_FEATURE_FILE_TERMS | NOTMUCH_FEATURE_BOOL_FOLDER)) {
+	(NOTMUCH_FEATURE_FILE_TERMS | NOTMUCH_FEATURE_BOOL_FOLDER |
+	 NOTMUCH_FEATURE_LAST_MOD)) {
 	notmuch_query_t *query = notmuch_query_create (notmuch, "");
 	notmuch_messages_t *messages;
 	notmuch_message_t *message;
@@ -1419,6 +1452,13 @@ notmuch_database_upgrade (notmuch_database_t *notmuch,
 	    if (new_features & NOTMUCH_FEATURE_BOOL_FOLDER)
 		_notmuch_message_upgrade_folder (message);
 
+	    /* Prior to NOTMUCH_FEATURE_LAST_MOD, messages did not
+	     * track modification revisions.  Give all messages a
+	     * revision of 1.
+	     */
+	    if (new_features & NOTMUCH_FEATURE_LAST_MOD)
+		_notmuch_message_upgrade_last_mod (message);
+
 	    _notmuch_message_sync (message);
 
 	    notmuch_message_destroy (message);
@@ -1601,6 +1641,11 @@ notmuch_database_end_atomic (notmuch_database_t *notmuch)
 	return NOTMUCH_STATUS_XAPIAN_EXCEPTION;
     }
 
+    if (notmuch->atomic_dirty) {
+	++notmuch->revision;
+	notmuch->atomic_dirty = FALSE;
+    }
+
 DONE:
     notmuch->atomic_nesting--;
     return NOTMUCH_STATUS_SUCCESS;
diff --git a/lib/message.cc b/lib/message.cc
index 1ddce3c..26b5e76 100644
--- a/lib/message.cc
+++ b/lib/message.cc
@@ -998,6 +998,16 @@ _notmuch_message_set_header_values (notmuch_message_t *message,
     message->modified = TRUE;
 }
 
+/* Upgrade a message to support NOTMUCH_FEATURE_LAST_MOD.  The caller
+ * must call _notmuch_message_sync. */
+void
+_notmuch_message_upgrade_last_mod (notmuch_message_t *message)
+{
+    /* _notmuch_message_sync will update the last modification
+     * revision; we just have to ask it to. */
+    message->modified = TRUE;
+}
+
 /* Synchronize changes made to message->doc out into the database. */
 void
 _notmuch_message_sync (notmuch_message_t *message)
@@ -1010,6 +1020,18 @@ _notmuch_message_sync (notmuch_message_t *message)
     if (! message->modified)
 	return;
 
+    /* Update the last modification of this message. */
+    if (message->notmuch->features & NOTMUCH_FEATURE_LAST_MOD)
+	/* sortable_serialise gives a reasonably compact encoding,
+	 * which directly translates to reduced IO when scanning the
+	 * value stream.  Since it's built for doubles, we only get 53
+	 * effective bits, but that's still enough for the database to
+	 * last a few centuries at 1 million revisions per second. */
+	message->doc.add_value (NOTMUCH_VALUE_LAST_MOD,
+				Xapian::sortable_serialise (
+				    _notmuch_database_new_revision (
+					message->notmuch)));
+
     db = static_cast <Xapian::WritableDatabase *> (message->notmuch->xapian_db);
     db->replace_document (message->doc_id, message->doc);
     message->modified = FALSE;
diff --git a/lib/notmuch-private.h b/lib/notmuch-private.h
index cc9ce12..f52b4e4 100644
--- a/lib/notmuch-private.h
+++ b/lib/notmuch-private.h
@@ -107,7 +107,8 @@ typedef enum {
     NOTMUCH_VALUE_TIMESTAMP = 0,
     NOTMUCH_VALUE_MESSAGE_ID,
     NOTMUCH_VALUE_FROM,
-    NOTMUCH_VALUE_SUBJECT
+    NOTMUCH_VALUE_SUBJECT,
+    NOTMUCH_VALUE_LAST_MOD,
 } notmuch_value_t;
 
 /* Xapian (with flint backend) complains if we provide a term longer
@@ -194,6 +195,9 @@ void
 _notmuch_database_log (notmuch_database_t *notmuch,
 		       const char *format, ...);
 
+unsigned long
+_notmuch_database_new_revision (notmuch_database_t *notmuch);
+
 const char *
 _notmuch_database_relative_path (notmuch_database_t *notmuch,
 				 const char *path);
@@ -305,6 +309,10 @@ _notmuch_message_set_header_values (notmuch_message_t *message,
 				    const char *date,
 				    const char *from,
 				    const char *subject);
+
+void
+_notmuch_message_upgrade_last_mod (notmuch_message_t *message);
+
 void
 _notmuch_message_sync (notmuch_message_t *message);
 
-- 
2.1.4

[WIP2 03/12] lib: API to retrieve database revision and UUID

[David Bremner]

From: Austin Clements <aclements@csail.mit.edu>

This exposes the committed database revision to library users along
with a UUID that can be used to detect when revision numbers are no
longer comparable (e.g., because the database has been replaced).
---
 lib/database-private.h         |  1 +
 lib/database.cc                | 11 +++++++++++
 lib/notmuch.h                  | 18 ++++++++++++++++++
 test/T570-revision-tracking.sh | 37 +++++++++++++++++++++++++++++++++++++
 4 files changed, 67 insertions(+)
 create mode 100755 test/T570-revision-tracking.sh

diff --git a/lib/database-private.h b/lib/database-private.h
index 5c5a2bb..4e93257 100644
--- a/lib/database-private.h
+++ b/lib/database-private.h
@@ -170,6 +170,7 @@ struct _notmuch_database {
      * under a higher revision number, which can be generated with
      * notmuch_database_new_revision. */
     unsigned long revision;
+    const char *uuid;
 
     Xapian::QueryParser *query_parser;
     Xapian::TermGenerator *term_gen;
diff --git a/lib/database.cc b/lib/database.cc
index 4378979..f88bdb3 100644
--- a/lib/database.cc
+++ b/lib/database.cc
@@ -978,6 +978,8 @@ notmuch_database_open_verbose (const char *path,
 	    notmuch->revision = 0;
 	else
 	    notmuch->revision = Xapian::sortable_unserialise (last_mod);
+	notmuch->uuid = talloc_strdup (
+	    notmuch, notmuch->xapian_db->get_uuid ().c_str ());
 
 	notmuch->query_parser = new Xapian::QueryParser;
 	notmuch->term_gen = new Xapian::TermGenerator;
@@ -1651,6 +1653,15 @@ DONE:
     return NOTMUCH_STATUS_SUCCESS;
 }
 
+unsigned long
+notmuch_database_get_revision (notmuch_database_t *notmuch,
+				const char **uuid)
+{
+    if (uuid)
+	*uuid = notmuch->uuid;
+    return notmuch->revision;
+}
+
 /* We allow the user to use arbitrarily long paths for directories. But
  * we have a term-length limit. So if we exceed that, we'll use the
  * SHA-1 of the path for the database term.
diff --git a/lib/notmuch.h b/lib/notmuch.h
index 20c4e01..b6be727 100644
--- a/lib/notmuch.h
+++ b/lib/notmuch.h
@@ -461,6 +461,24 @@ notmuch_status_t
 notmuch_database_end_atomic (notmuch_database_t *notmuch);
 
 /**
+ * Return the committed database revision and UUID.
+ *
+ * The database revision number increases monotonically with each
+ * commit to the database.  Hence, all messages and message changes
+ * committed to the database (that is, visible to readers) have a last
+ * modification revision <= the committed database revision.  Any
+ * messages committed in the future will be assigned a modification
+ * revision > the committed database revision.
+ *
+ * The UUID is a NUL-terminated opaque string that uniquely identifies
+ * this database.  Two revision numbers are only comparable if they
+ * have the same database UUID.
+ */
+unsigned long
+notmuch_database_get_revision (notmuch_database_t *notmuch,
+				const char **uuid);
+
+/**
  * Retrieve a directory object from the database for 'path'.
  *
  * Here, 'path' should be a path relative to the path of 'database'
diff --git a/test/T570-revision-tracking.sh b/test/T570-revision-tracking.sh
new file mode 100755
index 0000000..8c12740
--- /dev/null
+++ b/test/T570-revision-tracking.sh
@@ -0,0 +1,37 @@
+#!/usr/bin/env bash
+test_description="database revision tracking"
+
+. ./test-lib.sh
+
+add_email_corpus
+
+test_begin_subtest "notmuch_database_get_revision"
+test_C ${MAIL_DIR} <<'EOF'
+#include <stdio.h>
+#include <string.h>
+#include <notmuch.h>
+int main (int argc, char** argv)
+{
+   notmuch_database_t *db;
+   notmuch_status_t stat;
+   long revision;
+   const char *uuid;
+
+   unsigned long rev;
+
+   stat = notmuch_database_open (argv[1], NOTMUCH_DATABASE_MODE_READ_ONLY, &db);
+   if (stat)
+       fputs ("open failed\n", stderr);
+   revision = notmuch_database_get_revision (db, &uuid);
+   printf("revision=%d\nuuid_len=%d\n", revision, strlen (uuid));
+}
+EOF
+cat <<'EOF' >EXPECTED
+== stdout ==
+revision=53
+uuid_len=36
+== stderr ==
+EOF
+test_expect_equal_file OUTPUT EXPECTED
+
+test_done
-- 
2.1.4

[WIP2 04/12] cli: add type introspection to sprinter type

[David Bremner]

This will make it easier to skip generating certain output for the
text format.
---
 sprinter-json.c |  1 +
 sprinter-sexp.c |  1 +
 sprinter-text.c |  1 +
 sprinter.h      | 10 ++++++++++
 4 files changed, 13 insertions(+)

diff --git a/sprinter-json.c b/sprinter-json.c
index 0a07790..8e5ffbe 100644
--- a/sprinter-json.c
+++ b/sprinter-json.c
@@ -175,6 +175,7 @@ sprinter_json_create (const void *ctx, FILE *stream)
 {
     static const struct sprinter_json template = {
 	.vtable = {
+	    .type = NOTMUCH_SPRINTER_JSON,
 	    .begin_map = json_begin_map,
 	    .begin_list = json_begin_list,
 	    .end = json_end,
diff --git a/sprinter-sexp.c b/sprinter-sexp.c
index 0aa51e8..e187c05 100644
--- a/sprinter-sexp.c
+++ b/sprinter-sexp.c
@@ -210,6 +210,7 @@ sprinter_sexp_create (const void *ctx, FILE *stream)
 {
     static const struct sprinter_sexp template = {
 	.vtable = {
+	    .type = NOTMUCH_SPRINTER_SEXP,
 	    .begin_map = sexp_begin_map,
 	    .begin_list = sexp_begin_list,
 	    .end = sexp_end,
diff --git a/sprinter-text.c b/sprinter-text.c
index 7779488..cd6cd56 100644
--- a/sprinter-text.c
+++ b/sprinter-text.c
@@ -117,6 +117,7 @@ sprinter_text_create (const void *ctx, FILE *stream)
 {
     static const struct sprinter_text template = {
 	.vtable = {
+	    .type = NOTMUCH_SPRINTER_TEXT,
 	    .begin_map = text_begin_map,
 	    .begin_list = text_begin_list,
 	    .end = text_end,
diff --git a/sprinter.h b/sprinter.h
index f859672..7e58f69 100644
--- a/sprinter.h
+++ b/sprinter.h
@@ -4,11 +4,21 @@
 /* Necessary for notmuch_bool_t */
 #include "notmuch-client.h"
 
+enum notmuch_sprinter_type {
+    NOTMUCH_SPRINTER_JSON,
+    NOTMUCH_SPRINTER_SEXP,
+    NOTMUCH_SPRINTER_TEXT
+};
+
 /* Structure printer interface. This is used to create output
  * structured as maps (with key/value pairs), lists and primitives
  * (strings, integers and booleans).
  */
 typedef struct sprinter {
+    /* Identify the type of sprinter
+     */
+    enum notmuch_sprinter_type type;
+
     /* Start a new map/dictionary structure. This should be followed by
      * a sequence of alternating calls to map_key and one of the
      * value-printing functions until the map is ended by end.
-- 
2.1.4

[WIP2 05/12] cli: add two sprinter utility functions

[David Bremner]

These are called directly, rather than via a vtable. They will
help add common metadata to search/show/reply
---
 devel/schemata   |  1 +
 sprinter-utils.c | 41 +++++++++++++++++++++++++++++++++++++++++
 sprinter.h       |  7 +++++++
 3 files changed, 49 insertions(+)
 create mode 100644 sprinter-utils.c

diff --git a/devel/schemata b/devel/schemata
index 41dc4a6..98a1b2b 100644
--- a/devel/schemata
+++ b/devel/schemata
@@ -147,6 +147,7 @@ search_files = [string*]
 search_tags = [string*]
 
 thread_summary = {
+    database:       database_metadata
     thread:         threadid,
     timestamp:      unix_time,
     date_relative:  string,   # user-friendly timestamp
diff --git a/sprinter-utils.c b/sprinter-utils.c
new file mode 100644
index 0000000..2ba588e
--- /dev/null
+++ b/sprinter-utils.c
@@ -0,0 +1,41 @@
+#include "notmuch-client.h"
+#include "sprinter.h"
+
+notmuch_status_t
+sprinter_start_output (sprinter_t *sp,
+		       const notmuch_query_t *query,
+		       const char *query_type)
+{
+    long int rev;
+    const char *uuid;
+    notmuch_database_t *notmuch;
+
+    sp->begin_list (sp);
+
+    if (sp->type != NOTMUCH_SPRINTER_TEXT && notmuch_format_version >= 3) {
+	notmuch = notmuch_query_get_database (query);
+
+	rev = notmuch_database_get_revision (notmuch, &uuid);
+
+	sp->begin_map (sp);
+	sp->map_key (sp, "query_type");
+	sp->string (sp, query_type);
+
+	sp->map_key (sp, "uuid");
+	sp->string (sp, uuid);
+
+	sp->map_key (sp, "lastmod");
+	sp->integer (sp, rev);
+
+	sp->end (sp);
+
+    }
+    return NOTMUCH_STATUS_SUCCESS;
+}
+
+notmuch_status_t
+sprinter_finish_output (sprinter_t *sp)
+{
+    sp->end (sp); /* map */
+    return NOTMUCH_STATUS_SUCCESS;
+}
diff --git a/sprinter.h b/sprinter.h
index 7e58f69..4873f69 100644
--- a/sprinter.h
+++ b/sprinter.h
@@ -91,4 +91,11 @@ sprinter_json_create (const void *ctx, FILE *stream);
 struct sprinter *
 sprinter_sexp_create (const void *ctx, FILE *stream);
 
+notmuch_status_t
+sprinter_start_output (struct sprinter *sp,
+		       const notmuch_query_t *query,
+		       const char *query_type);
+
+notmuch_status_t
+sprinter_finish_output (struct sprinter *sp);
 #endif // NOTMUCH_SPRINTER_H
-- 
2.1.4

[WIP2 06/12] cli/show: add extra element to structured output for metadata

[David Bremner]

Initially this will contain the database uuid and last revision,
and a query_type describing the remaining elements

The changes to emacs/notmuch-tree.el and test/* are to (temporarily)
force these to continue to use the old format.

One non-trivial thing already tested is that the text and mbox formats
do not change in format version 3.
---
 Makefile.local               |  1 +
 devel/schemata               | 14 ++++++++++++--
 emacs/notmuch-tree.el        |  2 +-
 lib/notmuch.h                |  6 ++++++
 lib/query.cc                 |  6 ++++++
 notmuch-client.h             |  4 ++--
 notmuch-show.c               |  4 ++--
 test/T070-insert.sh          |  2 +-
 test/T160-json.sh            | 10 +++++-----
 test/T170-sexp.sh            | 10 +++++-----
 test/T190-multipart.sh       |  4 ++--
 test/T340-maildir-sync.sh    |  2 +-
 test/T350-crypto.sh          | 14 +++++++-------
 test/T470-missing-headers.sh |  2 +-
 test/T510-thread-replies.sh  | 10 +++++-----
 test/test-lib.sh             |  4 ++++
 16 files changed, 61 insertions(+), 34 deletions(-)

diff --git a/Makefile.local b/Makefile.local
index 6d54742..8e17c13 100644
--- a/Makefile.local
+++ b/Makefile.local
@@ -290,6 +290,7 @@ notmuch_client_srcs =		\
 	sprinter-json.c		\
 	sprinter-sexp.c		\
 	sprinter-text.c		\
+	sprinter-utils.c	\
 	query-string.c		\
 	mime-node.c		\
 	crypto.c		\
diff --git a/devel/schemata b/devel/schemata
index 98a1b2b..76dad01 100644
--- a/devel/schemata
+++ b/devel/schemata
@@ -14,7 +14,7 @@ are interleaved. Keys are printed as keywords (symbols preceded by a
 colon), e.g. (:id "123" :time 54321 :from "foobar"). Null is printed as
 nil, true as t and false as nil.
 
-This is version 2 of the structured output format.
+This is version 3 of the structured output format.
 
 Version history
 ---------------
@@ -26,6 +26,9 @@ v1
 v2
 - Added the thread_summary.query field.
 
+v3
+- Initial dictionary of database metadata to notmuch show output
+
 Common non-terminals
 --------------------
 
@@ -38,12 +41,19 @@ threadid = string
 # Message ID, sans "id:"
 messageid = string
 
+database_metadata = {
+    query_type: string
+    uuid: string
+    lastmod: int
+}
+
 notmuch show schema
 -------------------
 
 # A top-level set of threads (do_show)
 # Returned by notmuch show without a --part argument
-thread_set = [thread*]
+
+thread_set = [database_metadata, thread*]
 
 # Top-level messages in a thread (show_messages)
 thread = [thread_node*]
diff --git a/emacs/notmuch-tree.el b/emacs/notmuch-tree.el
index 8b6cd51..ca5f9b6 100644
--- a/emacs/notmuch-tree.el
+++ b/emacs/notmuch-tree.el
@@ -871,7 +871,7 @@ the same as for the function notmuch-tree."
     (notmuch-tag-clear-cache)
     (let ((proc (notmuch-start-notmuch
 		 "notmuch-tree" (current-buffer) #'notmuch-tree-process-sentinel
-		 "show" "--body=false" "--format=sexp"
+		 "show" "--body=false" "--format=sexp" "--format-version=2"
 		 message-arg search-args))
 	  ;; Use a scratch buffer to accumulate partial output.
 	  ;; This buffer will be killed by the sentinel, which
diff --git a/lib/notmuch.h b/lib/notmuch.h
index b6be727..5c17d97 100644
--- a/lib/notmuch.h
+++ b/lib/notmuch.h
@@ -781,6 +781,12 @@ notmuch_sort_t
 notmuch_query_get_sort (notmuch_query_t *query);
 
 /**
+ * Return the associated notmuch database
+ */
+notmuch_database_t *
+notmuch_query_get_database (const notmuch_query_t *query);
+
+/**
  * Add a tag that will be excluded from the query results by default.
  * This exclusion will be overridden if this tag appears explicitly in
  * the query.
diff --git a/lib/query.cc b/lib/query.cc
index 9cedb6a..61f85c0 100644
--- a/lib/query.cc
+++ b/lib/query.cc
@@ -654,3 +654,9 @@ notmuch_query_count_threads (notmuch_query_t *query)
 
     return count;
 }
+
+notmuch_database_t *
+notmuch_query_get_database (const notmuch_query_t *query)
+{
+    return query->notmuch;
+}
diff --git a/notmuch-client.h b/notmuch-client.h
index ab0d188..76dbc38 100644
--- a/notmuch-client.h
+++ b/notmuch-client.h
@@ -139,7 +139,7 @@ chomp_newline (char *str)
  * this.  New (required) map fields can be added without increasing
  * this.
  */
-#define NOTMUCH_FORMAT_CUR 2
+#define NOTMUCH_FORMAT_CUR 3
 /* The minimum supported structured output format version.  Requests
  * for format versions below this will return an error. */
 #define NOTMUCH_FORMAT_MIN 1
@@ -148,7 +148,7 @@ chomp_newline (char *str)
  * Must be between NOTMUCH_FORMAT_MIN and NOTMUCH_FORMAT_CUR,
  * inclusive.
  */
-#define NOTMUCH_FORMAT_MIN_ACTIVE 1
+#define NOTMUCH_FORMAT_MIN_ACTIVE 2
 
 /* The output format version requested by the caller on the command
  * line.  If no format version is requested, this will be set to
diff --git a/notmuch-show.c b/notmuch-show.c
index b80933a..4489ea5 100644
--- a/notmuch-show.c
+++ b/notmuch-show.c
@@ -1019,7 +1019,7 @@ do_show (void *ctx,
     if (! threads)
 	return 1;
 
-    sp->begin_list (sp);
+    sprinter_start_output (sp, query, "threads");
 
     for ( ;
 	 notmuch_threads_valid (threads);
@@ -1041,7 +1041,7 @@ do_show (void *ctx,
 
     }
 
-    sp->end (sp);
+    sprinter_finish_output (sp);
 
     return res != NOTMUCH_STATUS_SUCCESS;
 }
diff --git a/test/T070-insert.sh b/test/T070-insert.sh
index 168345c..48d36af 100755
--- a/test/T070-insert.sh
+++ b/test/T070-insert.sh
@@ -38,7 +38,7 @@ cur_msg_filename=$(notmuch search --output=files "subject:insert-subject")
 test_expect_equal_file "$cur_msg_filename" "$gen_msg_filename"
 
 test_begin_subtest "Insert message adds default tags"
-output=$(notmuch show --format=json "subject:insert-subject")
+output=$(NOTMUCH_SHOW --format=json "subject:insert-subject")
 expected='[[[{
  "id": "'"${gen_msg_id}"'",
  "match": true,
diff --git a/test/T160-json.sh b/test/T160-json.sh
index c1cf649..46c1314 100755
--- a/test/T160-json.sh
+++ b/test/T160-json.sh
@@ -4,16 +4,16 @@ test_description="--format=json output"
 
 test_begin_subtest "Show message: json"
 add_message "[subject]=\"json-show-subject\"" "[date]=\"Sat, 01 Jan 2000 12:00:00 -0000\"" "[bcc]=\"test_suite+bcc@notmuchmail.org\"" "[reply-to]=\"test_suite+replyto@notmuchmail.org\"" "[body]=\"json-show-message\""
-output=$(notmuch show --format=json "json-show-message")
+output=$(NOTMUCH_SHOW --format=json "json-show-message")
 test_expect_equal_json "$output" "[[[{\"id\": \"${gen_msg_id}\", \"match\": true, \"excluded\": false, \"filename\": \"${gen_msg_filename}\", \"timestamp\": 946728000, \"date_relative\": \"2000-01-01\", \"tags\": [\"inbox\",\"unread\"], \"headers\": {\"Subject\": \"json-show-subject\", \"From\": \"Notmuch Test Suite <test_suite@notmuchmail.org>\", \"To\": \"Notmuch Test Suite <test_suite@notmuchmail.org>\", \"Bcc\": \"test_suite+bcc@notmuchmail.org\", \"Reply-To\": \"test_suite+replyto@notmuchmail.org\", \"Date\": \"Sat, 01 Jan 2000 12:00:00 +0000\"}, \"body\": [{\"id\": 1, \"content-type\": \"text/plain\", \"content\": \"json-show-message\n\"}]}, []]]]"
 
 # This should be the same output as above.
 test_begin_subtest "Show message: json --body=true"
-output=$(notmuch show --format=json --body=true "json-show-message")
+output=$(NOTMUCH_SHOW --format=json --body=true "json-show-message")
 test_expect_equal_json "$output" "[[[{\"id\": \"${gen_msg_id}\", \"match\": true, \"excluded\": false, \"filename\": \"${gen_msg_filename}\", \"timestamp\": 946728000, \"date_relative\": \"2000-01-01\", \"tags\": [\"inbox\",\"unread\"], \"headers\": {\"Subject\": \"json-show-subject\", \"From\": \"Notmuch Test Suite <test_suite@notmuchmail.org>\", \"To\": \"Notmuch Test Suite <test_suite@notmuchmail.org>\", \"Bcc\": \"test_suite+bcc@notmuchmail.org\", \"Reply-To\": \"test_suite+replyto@notmuchmail.org\", \"Date\": \"Sat, 01 Jan 2000 12:00:00 +0000\"}, \"body\": [{\"id\": 1, \"content-type\": \"text/plain\", \"content\": \"json-show-message\n\"}]}, []]]]"
 
 test_begin_subtest "Show message: json --body=false"
-output=$(notmuch show --format=json --body=false "json-show-message")
+output=$(NOTMUCH_SHOW --format=json --body=false "json-show-message")
 test_expect_equal_json "$output" "[[[{\"id\": \"${gen_msg_id}\", \"match\": true, \"excluded\": false, \"filename\": \"${gen_msg_filename}\", \"timestamp\": 946728000, \"date_relative\": \"2000-01-01\", \"tags\": [\"inbox\",\"unread\"], \"headers\": {\"Subject\": \"json-show-subject\", \"From\": \"Notmuch Test Suite <test_suite@notmuchmail.org>\", \"To\": \"Notmuch Test Suite <test_suite@notmuchmail.org>\", \"Bcc\": \"test_suite+bcc@notmuchmail.org\", \"Reply-To\": \"test_suite+replyto@notmuchmail.org\", \"Date\": \"Sat, 01 Jan 2000 12:00:00 +0000\"}}, []]]]"
 
 test_begin_subtest "Search message: json"
@@ -32,7 +32,7 @@ test_expect_equal_json "$output" "[{\"thread\": \"XXX\",
 
 test_begin_subtest "Show message: json, utf-8"
 add_message "[subject]=\"json-show-utf8-body-sübjéct\"" "[date]=\"Sat, 01 Jan 2000 12:00:00 -0000\"" "[body]=\"jsön-show-méssage\""
-output=$(notmuch show --format=json "jsön-show-méssage")
+output=$(NOTMUCH_SHOW --format=json "jsön-show-méssage")
 test_expect_equal_json "$output" "[[[{\"id\": \"${gen_msg_id}\", \"match\": true, \"excluded\": false, \"filename\": \"${gen_msg_filename}\", \"timestamp\": 946728000, \"date_relative\": \"2000-01-01\", \"tags\": [\"inbox\",\"unread\"], \"headers\": {\"Subject\": \"json-show-utf8-body-sübjéct\", \"From\": \"Notmuch Test Suite <test_suite@notmuchmail.org>\", \"To\": \"Notmuch Test Suite <test_suite@notmuchmail.org>\", \"Date\": \"Sat, 01 Jan 2000 12:00:00 +0000\"}, \"body\": [{\"id\": 1, \"content-type\": \"text/plain\", \"content\": \"jsön-show-méssage\n\"}]}, []]]]"
 
 test_begin_subtest "Show message: json, inline attachment filename"
@@ -44,7 +44,7 @@ emacs_fcc_message \
     "(mml-attach-file \"$TEST_DIRECTORY/README\" nil nil \"inline\")
      (message-goto-eoh)
      (insert \"Message-ID: <$id>\n\")"
-output=$(notmuch show --format=json "id:$id")
+output=$(NOTMUCH_SHOW --format=json "id:$id")
 filename=$(notmuch search --output=files "id:$id")
 # Get length of README after base64-encoding, minus additional newline.
 attachment_length=$(( $(base64 $TEST_DIRECTORY/README | wc -c) - 1 ))
diff --git a/test/T170-sexp.sh b/test/T170-sexp.sh
index 667e319..6c7402a 100755
--- a/test/T170-sexp.sh
+++ b/test/T170-sexp.sh
@@ -4,16 +4,16 @@ test_description="--format=sexp output"
 
 test_begin_subtest "Show message: sexp"
 add_message "[subject]=\"sexp-show-subject\"" "[date]=\"Sat, 01 Jan 2000 12:00:00 -0000\"" "[bcc]=\"test_suite+bcc@notmuchmail.org\"" "[reply-to]=\"test_suite+replyto@notmuchmail.org\"" "[body]=\"sexp-show-message\""
-output=$(notmuch show --format=sexp "sexp-show-message")
+output=$(NOTMUCH_SHOW --format=sexp "sexp-show-message")
 test_expect_equal "$output" "((((:id \"${gen_msg_id}\" :match t :excluded nil :filename \"${gen_msg_filename}\" :timestamp 946728000 :date_relative \"2000-01-01\" :tags (\"inbox\" \"unread\") :headers (:Subject \"sexp-show-subject\" :From \"Notmuch Test Suite <test_suite@notmuchmail.org>\" :To \"Notmuch Test Suite <test_suite@notmuchmail.org>\" :Bcc \"test_suite+bcc@notmuchmail.org\" :Reply-To \"test_suite+replyto@notmuchmail.org\" :Date \"Sat, 01 Jan 2000 12:00:00 +0000\") :body ((:id 1 :content-type \"text/plain\" :content \"sexp-show-message\n\"))) ())))"
 
 # This should be the same output as above.
 test_begin_subtest "Show message: sexp --body=true"
-output=$(notmuch show --format=sexp --body=true "sexp-show-message")
+output=$(NOTMUCH_SHOW --format=sexp --body=true "sexp-show-message")
 test_expect_equal "$output" "((((:id \"${gen_msg_id}\" :match t :excluded nil :filename \"${gen_msg_filename}\" :timestamp 946728000 :date_relative \"2000-01-01\" :tags (\"inbox\" \"unread\") :headers (:Subject \"sexp-show-subject\" :From \"Notmuch Test Suite <test_suite@notmuchmail.org>\" :To \"Notmuch Test Suite <test_suite@notmuchmail.org>\" :Bcc \"test_suite+bcc@notmuchmail.org\" :Reply-To \"test_suite+replyto@notmuchmail.org\" :Date \"Sat, 01 Jan 2000 12:00:00 +0000\") :body ((:id 1 :content-type \"text/plain\" :content \"sexp-show-message\n\"))) ())))"
 
 test_begin_subtest "Show message: sexp --body=false"
-output=$(notmuch show --format=sexp --body=false "sexp-show-message")
+output=$(NOTMUCH_SHOW --format=sexp --body=false "sexp-show-message")
 test_expect_equal "$output" "((((:id \"${gen_msg_id}\" :match t :excluded nil :filename \"${gen_msg_filename}\" :timestamp 946728000 :date_relative \"2000-01-01\" :tags (\"inbox\" \"unread\") :headers (:Subject \"sexp-show-subject\" :From \"Notmuch Test Suite <test_suite@notmuchmail.org>\" :To \"Notmuch Test Suite <test_suite@notmuchmail.org>\" :Bcc \"test_suite+bcc@notmuchmail.org\" :Reply-To \"test_suite+replyto@notmuchmail.org\" :Date \"Sat, 01 Jan 2000 12:00:00 +0000\")) ())))"
 
 test_begin_subtest "Search message: sexp"
@@ -23,7 +23,7 @@ test_expect_equal "$output" "((:thread \"0000000000000002\" :timestamp 946728000
 
 test_begin_subtest "Show message: sexp, utf-8"
 add_message "[subject]=\"sexp-show-utf8-body-sübjéct\"" "[date]=\"Sat, 01 Jan 2000 12:00:00 -0000\"" "[body]=\"jsön-show-méssage\""
-output=$(notmuch show --format=sexp "jsön-show-méssage")
+output=$(NOTMUCH_SHOW --format=sexp "jsön-show-méssage")
 test_expect_equal "$output" "((((:id \"${gen_msg_id}\" :match t :excluded nil :filename \"${gen_msg_filename}\" :timestamp 946728000 :date_relative \"2000-01-01\" :tags (\"inbox\" \"unread\") :headers (:Subject \"sexp-show-utf8-body-sübjéct\" :From \"Notmuch Test Suite <test_suite@notmuchmail.org>\" :To \"Notmuch Test Suite <test_suite@notmuchmail.org>\" :Date \"Sat, 01 Jan 2000 12:00:00 +0000\") :body ((:id 1 :content-type \"text/plain\" :content \"jsön-show-méssage\n\"))) ())))"
 
 test_begin_subtest "Show message: sexp, inline attachment filename"
@@ -35,7 +35,7 @@ emacs_fcc_message \
     "(mml-attach-file \"$TEST_DIRECTORY/README\" nil nil \"inline\")
      (message-goto-eoh)
      (insert \"Message-ID: <$id>\n\")"
-output=$(notmuch show --format=sexp "id:$id")
+output=$(NOTMUCH_SHOW --format=sexp "id:$id")
 filename=$(notmuch search --output=files "id:$id")
 # Get length of README after base64-encoding, minus additional newline.
 attachment_length=$(( $(base64 $TEST_DIRECTORY/README | wc -c) - 1 ))
diff --git a/test/T190-multipart.sh b/test/T190-multipart.sh
index ad8d29e..f8b805f 100755
--- a/test/T190-multipart.sh
+++ b/test/T190-multipart.sh
@@ -744,11 +744,11 @@ cat <<EOF >> EXPECTED.withhtml
 EOF
 
 test_begin_subtest "html parts excluded by default"
-notmuch show --format=json id:htmlmessage > OUTPUT
+NOTMUCH_SHOW --format=json id:htmlmessage > OUTPUT
 test_expect_equal_json "$(cat OUTPUT)" "$(cat EXPECTED.nohtml)"
 
 test_begin_subtest "html parts included"
-notmuch show --format=json --include-html id:htmlmessage > OUTPUT
+NOTMUCH_SHOW --format=json --include-html id:htmlmessage > OUTPUT
 test_expect_equal_json "$(cat OUTPUT)" "$(cat EXPECTED.withhtml)"
 
 test_begin_subtest "indexes mime-type #1"
diff --git a/test/T340-maildir-sync.sh b/test/T340-maildir-sync.sh
index 3186e70..50b1fb5 100755
--- a/test/T340-maildir-sync.sh
+++ b/test/T340-maildir-sync.sh
@@ -35,7 +35,7 @@ output=$(cd ${MAIL_DIR}/cur; ls -1 adding-replied*)
 test_expect_equal "$output" "adding-replied-tag:2,RS"
 
 test_begin_subtest "notmuch show works with renamed file (without notmuch new)"
-output=$(notmuch show --format=json id:${gen_msg_id} | notmuch_json_show_sanitize)
+output=$(NOTMUCH_SHOW --format=json id:${gen_msg_id} | notmuch_json_show_sanitize)
 test_expect_equal_json "$output" '[[[{"id": "XXXXX",
 "match": true,
 "excluded": false,
diff --git a/test/T350-crypto.sh b/test/T350-crypto.sh
index 477b397..d9e02d0 100755
--- a/test/T350-crypto.sh
+++ b/test/T350-crypto.sh
@@ -35,7 +35,7 @@ test_expect_success 'emacs delivery of signed message' \
     "(mml-secure-message-sign)"'
 
 test_begin_subtest "signature verification"
-output=$(notmuch show --format=json --verify subject:"test signed message 001" \
+output=$(NOTMUCH_SHOW --format=json --verify subject:"test signed message 001" \
     | notmuch_json_show_sanitize \
     | sed -e 's|"created": [1234567890]*|"created": 946728000|')
 expected='[[[{"id": "XXXXX",
@@ -69,7 +69,7 @@ test_begin_subtest "signature verification with full owner trust"
 # give the key full owner trust
 echo "${FINGERPRINT}:6:" | gpg --no-tty --import-ownertrust >>"$GNUPGHOME"/trust.log 2>&1
 gpg --no-tty --check-trustdb >>"$GNUPGHOME"/trust.log 2>&1
-output=$(notmuch show --format=json --verify subject:"test signed message 001" \
+output=$(NOTMUCH_SHOW --format=json --verify subject:"test signed message 001" \
     | notmuch_json_show_sanitize \
     | sed -e 's|"created": [1234567890]*|"created": 946728000|')
 expected='[[[{"id": "XXXXX",
@@ -103,7 +103,7 @@ test_expect_equal_json \
 test_begin_subtest "signature verification with signer key unavailable"
 # move the gnupghome temporarily out of the way
 mv "${GNUPGHOME}"{,.bak}
-output=$(notmuch show --format=json --verify subject:"test signed message 001" \
+output=$(NOTMUCH_SHOW --format=json --verify subject:"test signed message 001" \
     | notmuch_json_show_sanitize \
     | sed -e 's|"created": [1234567890]*|"created": 946728000|')
 expected='[[[{"id": "XXXXX",
@@ -177,7 +177,7 @@ test_expect_equal \
     "$expected"
 
 test_begin_subtest "decryption, --format=json"
-output=$(notmuch show --format=json --decrypt subject:"test encrypted message 001" \
+output=$(NOTMUCH_SHOW --format=json --decrypt subject:"test encrypted message 001" \
     | notmuch_json_show_sanitize \
     | sed -e 's|"created": [1234567890]*|"created": 946728000|')
 expected='[[[{"id": "XXXXX",
@@ -235,7 +235,7 @@ test_expect_equal_file OUTPUT TESTATTACHMENT
 test_begin_subtest "decryption failure with missing key"
 mv "${GNUPGHOME}"{,.bak}
 # The length of the encrypted attachment varies so must be normalized.
-output=$(notmuch show --format=json --decrypt subject:"test encrypted message 001" \
+output=$(NOTMUCH_SHOW --format=json --decrypt subject:"test encrypted message 001" \
     | notmuch_json_show_sanitize \
     | sed -e 's|"created": [1234567890]*|"created": 946728000|' \
     | sed -e 's|"content-length": 6[1234567890]*|"content-length": 652|')
@@ -272,7 +272,7 @@ test_expect_success 'emacs delivery of encrypted + signed message' \
     "(mml-secure-message-sign-encrypt)"'
 
 test_begin_subtest "decryption + signature verification"
-output=$(notmuch show --format=json --decrypt subject:"test encrypted message 002" \
+output=$(NOTMUCH_SHOW --format=json --decrypt subject:"test encrypted message 002" \
     | notmuch_json_show_sanitize \
     | sed -e 's|"created": [1234567890]*|"created": 946728000|')
 expected='[[[{"id": "XXXXX",
@@ -327,7 +327,7 @@ y
 " \
     | gpg --no-tty --quiet --command-fd 0 --armor --gen-revoke "0x${FINGERPRINT}!" 2>/dev/null \
     | gpg --no-tty --quiet --import
-output=$(notmuch show --format=json --verify subject:"test signed message 001" \
+output=$(NOTMUCH_SHOW --format=json --verify subject:"test signed message 001" \
     | notmuch_json_show_sanitize \
     | sed -e 's|"created": [1234567890]*|"created": 946728000|')
 expected='[[[{"id": "XXXXX",
diff --git a/test/T470-missing-headers.sh b/test/T470-missing-headers.sh
index cb38301..c3ee430 100755
--- a/test/T470-missing-headers.sh
+++ b/test/T470-missing-headers.sh
@@ -94,7 +94,7 @@ Body
 \fmessage}"
 
 test_begin_subtest "Show: json"
-output=$(notmuch show --format=json '*' | notmuch_json_show_sanitize)
+output=$(NOTMUCH_SHOW --format=json '*' | notmuch_json_show_sanitize)
 expected=$(notmuch_json_show_sanitize <<EOF
 [
     [
diff --git a/test/T510-thread-replies.sh b/test/T510-thread-replies.sh
index 1392fbe..30ac39b 100755
--- a/test/T510-thread-replies.sh
+++ b/test/T510-thread-replies.sh
@@ -17,7 +17,7 @@ add_message '[id]="foo@one.com"' \
 add_message '[in-reply-to]="mumble"' \
     '[references]="<foo@one.com>"' \
     '[subject]="Re: one"'
-output=$(notmuch show --format=json 'subject:one' | notmuch_json_show_sanitize)
+output=$(NOTMUCH_SHOW --format=json 'subject:one' | notmuch_json_show_sanitize)
 expected='[[[{"id": "foo@one.com",
  "match": true,
  "excluded": false,
@@ -51,7 +51,7 @@ add_message '[id]="foo@two.com"' \
 add_message '[in-reply-to]="<bar@baz.com>"' \
     '[references]="<foo@two.com>"' \
     '[subject]="Re: two"'
-output=$(notmuch show --format=json 'subject:two' | notmuch_json_show_sanitize)
+output=$(NOTMUCH_SHOW --format=json 'subject:two' | notmuch_json_show_sanitize)
 expected='[[[{"id": "foo@two.com",
  "match": true, "excluded": false,
  "filename": "YYYYY",
@@ -80,7 +80,7 @@ add_message '[id]="foo@three.com"' \
     '[subject]="three"'
 add_message '[in-reply-to]="<foo@three.com>"' \
     '[subject]="Re: three"'
-output=$(notmuch show --format=json 'subject:three' | notmuch_json_show_sanitize)
+output=$(NOTMUCH_SHOW --format=json 'subject:three' | notmuch_json_show_sanitize)
 expected='[[[{"id": "foo@three.com", "match": true, "excluded": false,
  "filename": "YYYYY",
  "timestamp": 978709437, "date_relative": "2001-01-05", "tags": ["inbox", "unread"],
@@ -109,7 +109,7 @@ add_message '[id]="bar@four.com"' \
 add_message '[in-reply-to]="<baz@four.com>"' \
     '[references]="<baz@four.com> <foo@four.com>"' \
     '[subject]="neither"'
-output=$(notmuch show --format=json 'subject:four' | notmuch_json_show_sanitize)
+output=$(NOTMUCH_SHOW --format=json 'subject:four' | notmuch_json_show_sanitize)
 expected='[[[{"id": "foo@four.com", "match": true, "excluded": false,
  "filename": "YYYYY",
  "timestamp": 978709437, "date_relative": "2001-01-05", "tags": ["inbox", "unread"],
@@ -143,7 +143,7 @@ add_message '[id]="foo@five.com"' \
 add_message '[id]="bar@five.com"' \
     '[references]="<foo@five.com> (garbage)"' \
     '[subject]="not-five"'
-output=$(notmuch show --format=json 'subject:five' | notmuch_json_show_sanitize)
+output=$(NOTMUCH_SHOW --format=json 'subject:five' | notmuch_json_show_sanitize)
 expected='[[[{"id": "XXXXX", "match": true, "excluded": false,
  "filename": "YYYYY", "timestamp": 42, "date_relative": "2001-01-05",
  "tags": ["inbox", "unread"], "headers": {"Subject": "five",
diff --git a/test/test-lib.sh b/test/test-lib.sh
index 486d1c4..d30608a 100644
--- a/test/test-lib.sh
+++ b/test/test-lib.sh
@@ -669,6 +669,10 @@ NOTMUCH_NEW ()
     notmuch new "${@}" | grep -v -E -e '^Processed [0-9]*( total)? file|Found [0-9]* total file'
 }
 
+NOTMUCH_SHOW () {
+    notmuch show --format-version=2 "${@}"
+}
+
 notmuch_search_sanitize ()
 {
     perl -pe 's/("?thread"?: ?)("?)................("?)/\1\2XXX\3/'
-- 
2.1.4

[WIP2 07/12] lib: Add "lastmod:" queries for filtering by last modification

[David Bremner]

From: Austin Clements <amdragon@mit.edu>

XXX Includes reference to notmuch search --db-revision, which doesn't
exist.
---
 doc/man7/notmuch-search-terms.rst | 8 ++++++++
 lib/database-private.h            | 1 +
 lib/database.cc                   | 4 ++++
 3 files changed, 13 insertions(+)

diff --git a/doc/man7/notmuch-search-terms.rst b/doc/man7/notmuch-search-terms.rst
index 1d27ac1..d9f213e 100644
--- a/doc/man7/notmuch-search-terms.rst
+++ b/doc/man7/notmuch-search-terms.rst
@@ -54,6 +54,8 @@ indicate user-supplied values):
 
 -  date:<since>..<until>
 
+-  lastmod:<since>..<until>
+
 The **from:** prefix is used to match the name or address of the sender
 of an email message.
 
@@ -124,6 +126,12 @@ The time range can also be specified using timestamps with a syntax of:
 Each timestamp is a number representing the number of seconds since
 1970-01-01 00:00:00 UTC.
 
+The **lastmod:** prefix can be used to restrict the result by the
+database revision number of when messages were last modified (tags
+were added/removed or filenames changed).  This is usually used in
+conjunction with the **--db-revision** argument to **notmuch search**
+to find messages that have changed since an earlier query.
+
 Operators
 ---------
 
diff --git a/lib/database-private.h b/lib/database-private.h
index 4e93257..3fb10f7 100644
--- a/lib/database-private.h
+++ b/lib/database-private.h
@@ -176,6 +176,7 @@ struct _notmuch_database {
     Xapian::TermGenerator *term_gen;
     Xapian::ValueRangeProcessor *value_range_processor;
     Xapian::ValueRangeProcessor *date_range_processor;
+    Xapian::ValueRangeProcessor *last_mod_range_processor;
 };
 
 /* Prior to database version 3, features were implied by the database
diff --git a/lib/database.cc b/lib/database.cc
index f88bdb3..afa34d6 100644
--- a/lib/database.cc
+++ b/lib/database.cc
@@ -986,6 +986,7 @@ notmuch_database_open_verbose (const char *path,
 	notmuch->term_gen->set_stemmer (Xapian::Stem ("english"));
 	notmuch->value_range_processor = new Xapian::NumberValueRangeProcessor (NOTMUCH_VALUE_TIMESTAMP);
 	notmuch->date_range_processor = new ParseTimeValueRangeProcessor (NOTMUCH_VALUE_TIMESTAMP);
+	notmuch->last_mod_range_processor = new Xapian::NumberValueRangeProcessor (NOTMUCH_VALUE_LAST_MOD, "lastmod:");
 
 	notmuch->query_parser->set_default_op (Xapian::Query::OP_AND);
 	notmuch->query_parser->set_database (*notmuch->xapian_db);
@@ -993,6 +994,7 @@ notmuch_database_open_verbose (const char *path,
 	notmuch->query_parser->set_stemming_strategy (Xapian::QueryParser::STEM_SOME);
 	notmuch->query_parser->add_valuerangeprocessor (notmuch->value_range_processor);
 	notmuch->query_parser->add_valuerangeprocessor (notmuch->date_range_processor);
+	notmuch->query_parser->add_valuerangeprocessor (notmuch->last_mod_range_processor);
 
 	for (i = 0; i < ARRAY_SIZE (BOOLEAN_PREFIX_EXTERNAL); i++) {
 	    prefix_t *prefix = &BOOLEAN_PREFIX_EXTERNAL[i];
@@ -1071,6 +1073,8 @@ notmuch_database_close (notmuch_database_t *notmuch)
     notmuch->value_range_processor = NULL;
     delete notmuch->date_range_processor;
     notmuch->date_range_processor = NULL;
+    delete notmuch->last_mod_range_processor;
+    notmuch->last_mod_range_processor = NULL;
 
     return status;
 }
-- 
2.1.4

[WIP2 08/12] cli/show: add lastmod to structured output

[David Bremner]

Here again we restrict a few tests to version 2, to keep them passing.
---
 devel/schemata         |  2 ++
 lib/message.cc         | 20 ++++++++++++++++++++
 lib/notmuch.h          |  8 ++++++++
 notmuch-show.c         |  7 +++++++
 test/T190-multipart.sh |  6 +++---
 test/T220-reply.sh     |  2 +-
 6 files changed, 41 insertions(+), 4 deletions(-)

diff --git a/devel/schemata b/devel/schemata
index 76dad01..02f7cc0 100644
--- a/devel/schemata
+++ b/devel/schemata
@@ -71,6 +71,8 @@ message = {
     match:          bool,
     filename:	    string,
     timestamp:      unix_time, # date header as unix time
+    lastmod:	    int,       # database revision when message
+			       # was update
     date_relative:  string,   # user-friendly timestamp
     tags:           [string*],
 
diff --git a/lib/message.cc b/lib/message.cc
index 26b5e76..9d04438 100644
--- a/lib/message.cc
+++ b/lib/message.cc
@@ -939,6 +939,26 @@ notmuch_message_get_date (notmuch_message_t *message)
     return Xapian::sortable_unserialise (value);
 }
 
+time_t
+notmuch_message_get_last_mod (notmuch_message_t *message)
+{
+    std::string value;
+
+    try {
+	value = message->doc.get_value (NOTMUCH_VALUE_LAST_MOD);
+    } catch (Xapian::Error &error) {
+	_notmuch_database_log(_notmuch_message_database (message), "A Xapian exception occurred when reading last modification: %s\n",
+		 error.get_msg().c_str());
+	message->notmuch->exception_reported = TRUE;
+	return 0;
+    }
+
+    if (value.empty ())
+	/* sortable_unserialise is undefined on empty string */
+	return -1;
+    return Xapian::sortable_unserialise (value);
+}
+
 notmuch_tags_t *
 notmuch_message_get_tags (notmuch_message_t *message)
 {
diff --git a/lib/notmuch.h b/lib/notmuch.h
index 5c17d97..b4897ab 100644
--- a/lib/notmuch.h
+++ b/lib/notmuch.h
@@ -1324,6 +1324,14 @@ time_t
 notmuch_message_get_date  (notmuch_message_t *message);
 
 /**
+ * Get the last database modifaction revision of 'message' as an
+ * integer.
+ *
+ */
+long int
+notmuch_message_get_last_mod  (notmuch_message_t *message);
+
+/**
  * Get the value of the specified header from 'message' as a UTF-8 string.
  *
  * Common headers are stored in the database when the message is
diff --git a/notmuch-show.c b/notmuch-show.c
index 4489ea5..3917b82 100644
--- a/notmuch-show.c
+++ b/notmuch-show.c
@@ -121,6 +121,7 @@ format_message_sprinter (sprinter_t *sp, notmuch_message_t *message)
     void *local = talloc_new (NULL);
     notmuch_tags_t *tags;
     time_t date;
+    long int revision;
     const char *relative_date;
 
     sp->map_key (sp, "id");
@@ -139,6 +140,12 @@ format_message_sprinter (sprinter_t *sp, notmuch_message_t *message)
     date = notmuch_message_get_date (message);
     sp->integer (sp, date);
 
+    if (notmuch_format_version >= 3) {
+	sp->map_key (sp, "lastmod");
+	revision = notmuch_message_get_last_mod (message);
+	sp->integer (sp, revision);
+    }
+
     sp->map_key (sp, "date_relative");
     relative_date = notmuch_time_relative_date (local, date);
     sp->string (sp, relative_date);
diff --git a/test/T190-multipart.sh b/test/T190-multipart.sh
index f8b805f..a6b4fca 100755
--- a/test/T190-multipart.sh
+++ b/test/T190-multipart.sh
@@ -343,7 +343,7 @@ test_expect_success \
     "notmuch show --format=text --part=8 'id:87liy5ap00.fsf@yoom.home.cworth.org'"
 
 test_begin_subtest "--format=json --part=0, full message"
-notmuch show --format=json --part=0 'id:87liy5ap00.fsf@yoom.home.cworth.org' >OUTPUT
+notmuch show --format=json --format-version=2 --part=0 'id:87liy5ap00.fsf@yoom.home.cworth.org' >OUTPUT
 cat <<EOF >EXPECTED
 {"id": "87liy5ap00.fsf@yoom.home.cworth.org", "match": true, "excluded": false, "filename": "${MAIL_DIR}/multipart", "timestamp": 978709437, "date_relative": "2001-01-05", "tags": ["attachment","inbox","signed","unread"], "headers": {"Subject": "Multipart message", "From": "Carl Worth <cworth@cworth.org>", "To": "cworth@cworth.org", "Date": "Fri, 05 Jan 2001 15:43:57 +0000"}, "body": [
 {"id": 1, "content-type": "multipart/signed", "content": [
@@ -449,7 +449,7 @@ notmuch show --format=raw 'id:87liy5ap00.fsf@yoom.home.cworth.org' >OUTPUT
 test_expect_equal_file OUTPUT "${MAIL_DIR}"/multipart
 
 test_begin_subtest "--format=raw --part=0, full message"
-notmuch show --format=raw --part=0 'id:87liy5ap00.fsf@yoom.home.cworth.org' >OUTPUT
+NOTMUCH_SHOW --format=raw --part=0 'id:87liy5ap00.fsf@yoom.home.cworth.org' >OUTPUT
 test_expect_equal_file OUTPUT "${MAIL_DIR}"/multipart
 
 test_begin_subtest "--format=raw --part=1, message body"
@@ -617,7 +617,7 @@ EOF
 test_expect_equal_file OUTPUT EXPECTED
 
 test_begin_subtest "'notmuch reply' to a multipart message with json format"
-notmuch reply --format=json 'id:87liy5ap00.fsf@yoom.home.cworth.org' | notmuch_json_show_sanitize >OUTPUT
+notmuch reply --format=json --format-version=2 'id:87liy5ap00.fsf@yoom.home.cworth.org' | notmuch_json_show_sanitize >OUTPUT
 notmuch_json_show_sanitize <<EOF >EXPECTED
 {"reply-headers": {"Subject": "Re: Multipart message",
  "From": "Notmuch Test Suite <test_suite@notmuchmail.org>",
diff --git a/test/T220-reply.sh b/test/T220-reply.sh
index b0d854a..e72f2e4 100755
--- a/test/T220-reply.sh
+++ b/test/T220-reply.sh
@@ -216,7 +216,7 @@ On Tue, 05 Jan 2010 15:43:56 -0000, ☃ <snowman@example.com> wrote:
 > Encoding"
 
 test_begin_subtest "Reply with RFC 2047-encoded headers (JSON)"
-output=$(notmuch reply --format=json id:${gen_msg_id})
+output=$(notmuch reply --format=json --format-version=2 id:${gen_msg_id})
 test_expect_equal_json "$output" '
 {
     "original": {
-- 
2.1.4

[WIP2 09/12] cli/search: add metadata element to structured output

[David Bremner]

As before, we force the test suite to use format version 2 in order
not to break. This is probably not the solution for the final version
of these patches.
---
 devel/schemata               | 10 +++++-----
 notmuch-search.c             | 18 +++++++++---------
 test/T070-insert.sh          |  2 +-
 test/T090-search-output.sh   | 20 ++++++++++----------
 test/T095-address.sh         |  4 ++--
 test/T160-json.sh            |  4 ++--
 test/T170-sexp.sh            |  4 ++--
 test/T470-missing-headers.sh |  2 +-
 test/test-lib.sh             |  8 ++++++++
 9 files changed, 40 insertions(+), 32 deletions(-)

diff --git a/devel/schemata b/devel/schemata
index 02f7cc0..e297adf 100644
--- a/devel/schemata
+++ b/devel/schemata
@@ -144,19 +144,19 @@ notmuch search schema
 ---------------------
 
 # --output=summary
-search_summary = [thread_summary*]
+search_summary = [database_metadata, thread_summary*]
 
 # --output=threads
-search_threads = [threadid*]
+search_threads = [database_metadata, threadid*]
 
 # --output=messages
-search_messages = [messageid*]
+search_messages = [database_metadata, messageid*]
 
 # --output=files
-search_files = [string*]
+search_files = [database_metadata, string*]
 
 # --output=tags
-search_tags = [string*]
+search_tags = [database_metadata, string*]
 
 thread_summary = {
     database:       database_metadata
diff --git a/notmuch-search.c b/notmuch-search.c
index 994ae58..5d17dac 100644
--- a/notmuch-search.c
+++ b/notmuch-search.c
@@ -122,7 +122,8 @@ do_search_threads (search_context_t *ctx)
     if (threads == NULL)
 	return 1;
 
-    format->begin_list (format);
+    sprinter_start_output (format, ctx->query,
+			   (ctx->output == OUTPUT_THREADS) ? "threads" : "summary");
 
     for (i = 0;
 	 notmuch_threads_valid (threads) && (ctx->limit < 0 || i < ctx->offset + ctx->limit);
@@ -238,8 +239,7 @@ do_search_threads (search_context_t *ctx)
 	notmuch_thread_destroy (thread);
     }
 
-    format->end (format);
-
+    sprinter_finish_output (format);
     return 0;
 }
 
@@ -423,7 +423,7 @@ do_search_messages (search_context_t *ctx)
     if (messages == NULL)
 	return 1;
 
-    format->begin_list (format);
+    sprinter_start_output (format, ctx->query, "messages");
 
     for (i = 0;
 	 notmuch_messages_valid (messages) && (ctx->limit < 0 || i < ctx->offset + ctx->limit);
@@ -486,8 +486,7 @@ do_search_messages (search_context_t *ctx)
 
     notmuch_messages_destroy (messages);
 
-    format->end (format);
-
+    sprinter_finish_output (format);
     return 0;
 }
 
@@ -505,6 +504,8 @@ do_search_tags (const search_context_t *ctx)
      * specified? */
 
     /* Special-case query of "*" for better performance. */
+
+
     if (strcmp (notmuch_query_get_query_string (query), "*") == 0) {
 	tags = notmuch_database_get_all_tags (notmuch);
     } else {
@@ -517,7 +518,7 @@ do_search_tags (const search_context_t *ctx)
     if (tags == NULL)
 	return 1;
 
-    format->begin_list (format);
+    sprinter_start_output (format, query, "tags");
 
     for (;
 	 notmuch_tags_valid (tags);
@@ -535,8 +536,7 @@ do_search_tags (const search_context_t *ctx)
     if (messages)
 	notmuch_messages_destroy (messages);
 
-    format->end (format);
-
+    sprinter_finish_output (format);
     return 0;
 }
 
diff --git a/test/T070-insert.sh b/test/T070-insert.sh
index 48d36af..67acbb0 100755
--- a/test/T070-insert.sh
+++ b/test/T070-insert.sh
@@ -64,7 +64,7 @@ output=$(notmuch search --output=files "subject:insert-subject" | wc -l)
 test_expect_equal "$output" 2
 
 test_begin_subtest "Duplicate message does not change tags"
-output=$(notmuch search --format=json --output=tags "subject:insert-subject")
+output=$(NOTMUCH_SEARCH --format=json --output=tags "subject:insert-subject")
 test_expect_equal_json "$output" '["inbox", "unread"]'
 
 test_begin_subtest "Insert message, add tag"
diff --git a/test/T090-search-output.sh b/test/T090-search-output.sh
index fe2ec9a..1dc74b6 100755
--- a/test/T090-search-output.sh
+++ b/test/T090-search-output.sh
@@ -35,7 +35,7 @@ EOF
 test_expect_equal_file OUTPUT EXPECTED
 
 test_begin_subtest "--output=threads --format=json"
-notmuch search --format=json --output=threads '*' | sed -e s/\".*\"/\"THREADID\"/ >OUTPUT
+NOTMUCH_SEARCH --format=json --output=threads '*' | sed -e s/\".*\"/\"THREADID\"/ >OUTPUT
 cat <<EOF >EXPECTED
 ["THREADID",
 "THREADID",
@@ -141,7 +141,7 @@ EOF
 test_expect_equal_file OUTPUT EXPECTED
 
 test_begin_subtest "--output=messages --format=json"
-notmuch search --format=json --output=messages '*' >OUTPUT
+NOTMUCH_SEARCH --format=json --output=messages '*' >OUTPUT
 cat <<EOF >EXPECTED
 ["4EFC743A.3060609@april.org",
 "877h1wv7mg.fsf@inf-8657.int-evry.fr",
@@ -199,19 +199,19 @@ EOF
 test_expect_equal_file OUTPUT EXPECTED
 
 test_begin_subtest "--output=messages --format=json --duplicate=1"
-notmuch search --output=messages --format=json --duplicate=1 '*' >OUTPUT
+NOTMUCH_SEARCH --output=messages --format=json --duplicate=1 '*' >OUTPUT
 # reuse same EXPECTED as above
 test_expect_equal_file OUTPUT EXPECTED
 
 test_begin_subtest "--output=messages --format=json --duplicate=2"
-notmuch search --output=messages --format=json --duplicate=2 '*' >OUTPUT
+NOTMUCH_SEARCH --output=messages --format=json --duplicate=2 '*' >OUTPUT
 cat <<EOF >EXPECTED
 ["20091117232137.GA7669@griffis1.net"]
 EOF
 test_expect_equal_file OUTPUT EXPECTED
 
 test_begin_subtest "--output=messages --format=json --duplicate=3"
-notmuch search --output=messages --format=json --duplicate=3 '*' >OUTPUT
+NOTMUCH_SEARCH --output=messages --format=json --duplicate=3 '*' >OUTPUT
 cat <<EOF >EXPECTED
 []
 EOF
@@ -338,7 +338,7 @@ EOF
 test_expect_equal_file OUTPUT EXPECTED
 
 test_begin_subtest "--output=files --format=json"
-notmuch search --format=json --output=files '*' | notmuch_search_files_sanitize \
+NOTMUCH_SEARCH --format=json --output=files '*' | notmuch_search_files_sanitize \
     | test_sort_json >OUTPUT
 cat <<EOF | test_sort_json >EXPECTED
 ["MAIL_DIR/cur/52:2,",
@@ -398,14 +398,14 @@ EOF
 test_expect_equal_file OUTPUT EXPECTED
 
 test_begin_subtest "--output=files --format=json --duplicate=2"
-notmuch search --format=json --output=files --duplicate=2 '*' | notmuch_search_files_sanitize >OUTPUT
+NOTMUCH_SEARCH --format=json --output=files --duplicate=2 '*' | notmuch_search_files_sanitize >OUTPUT
 cat <<EOF >EXPECTED
 ["$dup2"]
 EOF
 test_expect_equal_file OUTPUT EXPECTED
 
 test_begin_subtest "--output=tags"
-notmuch search --output=tags '*' >OUTPUT
+NOTMUCH_SEARCH --output=tags '*' >OUTPUT
 cat <<EOF >EXPECTED
 attachment
 inbox
@@ -415,7 +415,7 @@ EOF
 test_expect_equal_file OUTPUT EXPECTED
 
 test_begin_subtest "--output=tags --format=json"
-notmuch search --format=json --output=tags '*' >OUTPUT
+NOTMUCH_SEARCH --format=json --output=tags '*' >OUTPUT
 cat <<EOF >EXPECTED
 ["attachment",
 "inbox",
@@ -439,7 +439,7 @@ echo -n >EXPECTED
 test_expect_equal_file OUTPUT EXPECTED
 
 test_begin_subtest "search --format=json for non-existent message prints proper empty json"
-notmuch search --format=json "no-message-matches-this" > OUTPUT
+NOTMUCH_SEARCH --format=json "no-message-matches-this" > OUTPUT
 echo "[]" >EXPECTED
 test_expect_equal_file OUTPUT EXPECTED
 
diff --git a/test/T095-address.sh b/test/T095-address.sh
index ed0cac7..6eae30e 100755
--- a/test/T095-address.sh
+++ b/test/T095-address.sh
@@ -33,7 +33,7 @@ notmuch address '*' >OUTPUT
 test_expect_equal_file OUTPUT EXPECTED
 
 test_begin_subtest "--output=sender --format=json"
-notmuch address --output=sender --format=json '*' >OUTPUT
+NOTMUCH_ADDRESS --output=sender --format=json '*' >OUTPUT
 cat <<EOF >EXPECTED
 [{"name": "François Boulogne", "address": "boulogne.f@gmail.com", "name-addr": "François Boulogne <boulogne.f@gmail.com>"},
 {"name": "Olivier Berger", "address": "olivier.berger@it-sudparis.eu", "name-addr": "Olivier Berger <olivier.berger@it-sudparis.eu>"},
@@ -122,7 +122,7 @@ test_expect_equal_file OUTPUT EXPECTED
 test_begin_subtest "--output=count --format=json"
 # Since the iteration order of GHashTable is not specified, we
 # preprocess and sort the results to keep the order stable here.
-notmuch address --output=count --format=json '*' | \
+NOTMUCH_ADDRESS --output=count --format=json '*' | \
     sed -e 's/^\[//' -e 's/]$//' -e 's/,$//' | sort >OUTPUT
 cat <<EOF >EXPECTED
 {"name": "Adrian Perez de Castro", "address": "aperez@igalia.com", "name-addr": "Adrian Perez de Castro <aperez@igalia.com>", "count": 1}
diff --git a/test/T160-json.sh b/test/T160-json.sh
index 46c1314..10df42a 100755
--- a/test/T160-json.sh
+++ b/test/T160-json.sh
@@ -18,7 +18,7 @@ test_expect_equal_json "$output" "[[[{\"id\": \"${gen_msg_id}\", \"match\": true
 
 test_begin_subtest "Search message: json"
 add_message "[subject]=\"json-search-subject\"" "[date]=\"Sat, 01 Jan 2000 12:00:00 -0000\"" "[body]=\"json-search-message\""
-output=$(notmuch search --format=json "json-search-message" | notmuch_search_sanitize)
+output=$(NOTMUCH_SEARCH --format=json "json-search-message" | notmuch_search_sanitize)
 test_expect_equal_json "$output" "[{\"thread\": \"XXX\",
  \"timestamp\": 946728000,
  \"date_relative\": \"2000-01-01\",
@@ -52,7 +52,7 @@ test_expect_equal_json "$output" "[[[{\"id\": \"$id\", \"match\": true, \"exclud
 
 test_begin_subtest "Search message: json, utf-8"
 add_message "[subject]=\"json-search-utf8-body-sübjéct\"" "[date]=\"Sat, 01 Jan 2000 12:00:00 -0000\"" "[body]=\"jsön-search-méssage\""
-output=$(notmuch search --format=json "jsön-search-méssage" | notmuch_search_sanitize)
+output=$(NOTMUCH_SEARCH --format=json "jsön-search-méssage" | notmuch_search_sanitize)
 test_expect_equal_json "$output" "[{\"thread\": \"XXX\",
  \"timestamp\": 946728000,
  \"date_relative\": \"2000-01-01\",
diff --git a/test/T170-sexp.sh b/test/T170-sexp.sh
index 6c7402a..06a7f98 100755
--- a/test/T170-sexp.sh
+++ b/test/T170-sexp.sh
@@ -18,7 +18,7 @@ test_expect_equal "$output" "((((:id \"${gen_msg_id}\" :match t :excluded nil :f
 
 test_begin_subtest "Search message: sexp"
 add_message "[subject]=\"sexp-search-subject\"" "[date]=\"Sat, 01 Jan 2000 12:00:00 -0000\"" "[body]=\"sexp-search-message\""
-output=$(notmuch search --format=sexp "sexp-search-message" | notmuch_search_sanitize)
+output=$(NOTMUCH_SEARCH --format=sexp "sexp-search-message" | notmuch_search_sanitize)
 test_expect_equal "$output" "((:thread \"0000000000000002\" :timestamp 946728000 :date_relative \"2000-01-01\" :matched 1 :total 1 :authors \"Notmuch Test Suite\" :subject \"sexp-search-subject\" :query (\"id:$gen_msg_id\" nil) :tags (\"inbox\" \"unread\")))"
 
 test_begin_subtest "Show message: sexp, utf-8"
@@ -43,7 +43,7 @@ test_expect_equal "$output" "((((:id \"$id\" :match t :excluded nil :filename \"
 
 test_begin_subtest "Search message: sexp, utf-8"
 add_message "[subject]=\"sexp-search-utf8-body-sübjéct\"" "[date]=\"Sat, 01 Jan 2000 12:00:00 -0000\"" "[body]=\"jsön-search-méssage\""
-output=$(notmuch search --format=sexp "jsön-search-méssage" | notmuch_search_sanitize)
+output=$(NOTMUCH_SEARCH --format=sexp "jsön-search-méssage" | notmuch_search_sanitize)
 test_expect_equal "$output" "((:thread \"0000000000000005\" :timestamp 946728000 :date_relative \"2000-01-01\" :matched 1 :total 1 :authors \"Notmuch Test Suite\" :subject \"sexp-search-utf8-body-sübjéct\" :query (\"id:$gen_msg_id\" nil) :tags (\"inbox\" \"unread\")))"
 
 
diff --git a/test/T470-missing-headers.sh b/test/T470-missing-headers.sh
index c3ee430..2e24a74 100755
--- a/test/T470-missing-headers.sh
+++ b/test/T470-missing-headers.sh
@@ -29,7 +29,7 @@ thread:XXX   2001-01-05 [1/1] (null);  (inbox unread)
 thread:XXX   1970-01-01 [1/1] Notmuch Test Suite;  (inbox unread)"
 
 test_begin_subtest "Search: json"
-output=$(notmuch search --format=json '*' | notmuch_search_sanitize)
+output=$(NOTMUCH_SEARCH --format=json '*' | notmuch_search_sanitize)
 test_expect_equal_json "$output" '
 [
     {
diff --git a/test/test-lib.sh b/test/test-lib.sh
index d30608a..d9d8ea1 100644
--- a/test/test-lib.sh
+++ b/test/test-lib.sh
@@ -664,11 +664,19 @@ test_emacs_expect_t () {
 	fi
 }
 
+NOTMUCH_ADDRESS () {
+    notmuch address --format-version=2 "${@}"
+}
+
 NOTMUCH_NEW ()
 {
     notmuch new "${@}" | grep -v -E -e '^Processed [0-9]*( total)? file|Found [0-9]* total file'
 }
 
+NOTMUCH_SEARCH () {
+    notmuch search --format-version=2 "${@}"
+}
+
 NOTMUCH_SHOW () {
     notmuch show --format-version=2 "${@}"
 }
-- 
2.1.4

[WIP2 10/12] emacs: convert notmuch-search to format-version 3

[David Bremner]

XXX this checks every result to see if it is metadata. This might or
    might not be innefficient

XXX This ignores the actual metadata from the query. Should it do
    something useful?
---
 emacs/notmuch-query.el |  5 +++++
 emacs/notmuch.el       | 15 ++++++++-------
 test/T310-emacs.sh     |  2 +-
 3 files changed, 14 insertions(+), 8 deletions(-)

diff --git a/emacs/notmuch-query.el b/emacs/notmuch-query.el
index d1daffc..bfef22c 100644
--- a/emacs/notmuch-query.el
+++ b/emacs/notmuch-query.el
@@ -34,6 +34,11 @@ is a possibly empty forest of replies.
     (setq args (append args search-terms))
     (apply #'notmuch-call-notmuch-sexp args)))
 
+
+(defun notmuch-query-metadata-p (result)
+  "Determine if the result is a \"metadata\" object (format-version 3)"
+  (plist-get result :query_type))
+
 ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
 ;; Mapping functions across collections of messages.
 
diff --git a/emacs/notmuch.el b/emacs/notmuch.el
index ab00454..87caee4 100644
--- a/emacs/notmuch.el
+++ b/emacs/notmuch.el
@@ -793,12 +793,13 @@ non-authors is found, assume that all of the authors match."
 
 This is only called when a result is first inserted so it also
 sets the :orig-tag property."
-  (let ((new-result (plist-put result :orig-tags (plist-get result :tags)))
-	(pos (point-max)))
-    (notmuch-search-show-result new-result pos)
-    (when (string= (plist-get result :thread) notmuch-search-target-thread)
-      (setq notmuch-search-target-thread "found")
-      (goto-char pos))))
+  (unless (notmuch-query-metadata-p result)
+    (let ((new-result (plist-put result :orig-tags (plist-get result :tags)))
+	  (pos (point-max)))
+      (notmuch-search-show-result new-result pos)
+      (when (string= (plist-get result :thread) notmuch-search-target-thread)
+	(setq notmuch-search-target-thread "found")
+	(goto-char pos)))))
 
 (defun notmuch-search-process-filter (proc string)
   "Process and filter the output of \"notmuch search\""
@@ -935,7 +936,7 @@ the configured default sort order."
       (save-excursion
 	(let ((proc (notmuch-start-notmuch
 		     "notmuch-search" buffer #'notmuch-search-process-sentinel
-		     "search" "--format=sexp" "--format-version=2"
+		     "search" "--format=sexp" "--format-version=3"
 		     (if oldest-first
 			 "--sort=oldest-first"
 		       "--sort=newest-first")
diff --git a/test/T310-emacs.sh b/test/T310-emacs.sh
index d72799b..f460923 100755
--- a/test/T310-emacs.sh
+++ b/test/T310-emacs.sh
@@ -897,7 +897,7 @@ YYY/notmuch_fail exited with status 1 (see *Notmuch errors* for more details)
 === ERROR ===
 [XXX]
 YYY/notmuch_fail exited with status 1
-command: YYY/notmuch_fail search --format\=sexp --format-version\=2 --sort\=newest-first tag\:inbox
+command: YYY/notmuch_fail search --format\=sexp --format-version\=3 --sort\=newest-first tag\:inbox
 exit status: 1"
 
 test_begin_subtest "Search handles subprocess warnings"
-- 
2.1.4

[WIP2 11/12] emacs: convert notmuch-tree to format-version 3

[David Bremner]

This is similarly trivial conversion to that applied to
notmuch-search, and is subject to the same objections
---
 emacs/notmuch-tree.el | 11 ++++++-----
 1 file changed, 6 insertions(+), 5 deletions(-)

diff --git a/emacs/notmuch-tree.el b/emacs/notmuch-tree.el
index ca5f9b6..bae3de0 100644
--- a/emacs/notmuch-tree.el
+++ b/emacs/notmuch-tree.el
@@ -774,10 +774,11 @@ message together with all its descendents."
 
 (defun notmuch-tree-insert-forest-thread (forest-thread)
   "Insert a single complete thread."
-  (let (tree-status)
-    ;; Reset at the start of each main thread.
-    (setq notmuch-tree-previous-subject nil)
-    (notmuch-tree-insert-thread forest-thread 0 tree-status)))
+  (unless (notmuch-query-metadata-p forest-thread)
+    (let (tree-status)
+      ;; Reset at the start of each main thread.
+      (setq notmuch-tree-previous-subject nil)
+      (notmuch-tree-insert-thread forest-thread 0 tree-status))))
 
 (defun notmuch-tree-insert-forest (forest)
   "Insert a forest of threads.
@@ -871,7 +872,7 @@ the same as for the function notmuch-tree."
     (notmuch-tag-clear-cache)
     (let ((proc (notmuch-start-notmuch
 		 "notmuch-tree" (current-buffer) #'notmuch-tree-process-sentinel
-		 "show" "--body=false" "--format=sexp" "--format-version=2"
+		 "show" "--body=false" "--format=sexp" "--format-version=3"
 		 message-arg search-args))
 	  ;; Use a scratch buffer to accumulate partial output.
 	  ;; This buffer will be killed by the sentinel, which
-- 
2.1.4

[WIP2 12/12] cli: add global option "--db-revision"

[David Bremner]

The function notmuch_exit_if_unmatched_db_revision is split from
notmuch_process_shared_options because it needs an open notmuch
database.
---
 notmuch-client.h  |  5 +++++
 notmuch-compact.c |  4 ++++
 notmuch-config.c  |  4 ++++
 notmuch-count.c   |  2 ++
 notmuch-dump.c    |  2 ++
 notmuch-insert.c  |  2 ++
 notmuch-new.c     |  3 ++-
 notmuch-reply.c   |  2 ++
 notmuch-restore.c |  2 ++
 notmuch-search.c  |  2 ++
 notmuch-setup.c   |  4 ++++
 notmuch-show.c    |  2 ++
 notmuch-tag.c     |  2 ++
 notmuch.c         | 18 ++++++++++++++++++
 14 files changed, 53 insertions(+), 1 deletion(-)

diff --git a/notmuch-client.h b/notmuch-client.h
index 76dbc38..24aa78a 100644
--- a/notmuch-client.h
+++ b/notmuch-client.h
@@ -466,6 +466,11 @@ notmuch_database_dump (notmuch_database_t *notmuch,
 		       notmuch_bool_t gzip_output);
 
 #include "command-line-arguments.h"
+
+extern char *notmuch_requested_db_revision;
 extern const notmuch_opt_desc_t  notmuch_shared_options [];
 void notmuch_process_shared_options (const char* help_name);
+
+void notmuch_exit_if_unmatched_db_revision (notmuch_database_t *notmuch);
+
 #endif
diff --git a/notmuch-compact.c b/notmuch-compact.c
index 5be551d..3da4bfb 100644
--- a/notmuch-compact.c
+++ b/notmuch-compact.c
@@ -46,6 +46,10 @@ notmuch_compact_command (notmuch_config_t *config, int argc, char *argv[])
     if (opt_index < 0)
 	return EXIT_FAILURE;
 
+    if (notmuch_requested_db_revision)
+	fprintf (stderr, "Warning: ignoring --db-revision=%s\n",
+		 notmuch_requested_db_revision);
+
     notmuch_process_shared_options (argv[0]);
 
     if (! quiet)
diff --git a/notmuch-config.c b/notmuch-config.c
index 568b3dc..a9d4e73 100644
--- a/notmuch-config.c
+++ b/notmuch-config.c
@@ -883,6 +883,10 @@ notmuch_config_command (notmuch_config_t *config, int argc, char *argv[])
     if (opt_index < 0)
 	return EXIT_FAILURE;
 
+    if (notmuch_requested_db_revision)
+	fprintf (stderr, "Warning: ignoring --db-revision=%s\n",
+		 notmuch_requested_db_revision);
+
     notmuch_process_shared_options (argv[0]);
 
     /* skip subcommand argument */
diff --git a/notmuch-count.c b/notmuch-count.c
index 57a88a8..e29ac2b 100644
--- a/notmuch-count.c
+++ b/notmuch-count.c
@@ -175,6 +175,8 @@ notmuch_count_command (notmuch_config_t *config, int argc, char *argv[])
 			       NOTMUCH_DATABASE_MODE_READ_ONLY, &notmuch))
 	return EXIT_FAILURE;
 
+    notmuch_exit_if_unmatched_db_revision (notmuch);
+
     query_str = query_string_from_args (config, argc-opt_index, argv+opt_index);
     if (query_str == NULL) {
 	fprintf (stderr, "Out of memory.\n");
diff --git a/notmuch-dump.c b/notmuch-dump.c
index fab22bd..9c0b82d 100644
--- a/notmuch-dump.c
+++ b/notmuch-dump.c
@@ -215,6 +215,8 @@ notmuch_dump_command (notmuch_config_t *config, int argc, char *argv[])
 			       NOTMUCH_DATABASE_MODE_READ_WRITE, &notmuch))
 	return EXIT_FAILURE;
 
+    notmuch_exit_if_unmatched_db_revision (notmuch);
+
     char *output_file_name = NULL;
     int opt_index;
 
diff --git a/notmuch-insert.c b/notmuch-insert.c
index 697880f..63b7a48 100644
--- a/notmuch-insert.c
+++ b/notmuch-insert.c
@@ -536,6 +536,8 @@ notmuch_insert_command (notmuch_config_t *config, int argc, char *argv[])
 			       NOTMUCH_DATABASE_MODE_READ_WRITE, &notmuch))
 	return EXIT_FAILURE;
 
+    notmuch_exit_if_unmatched_db_revision (notmuch);
+
     /* Write the message to the Maildir new directory. */
     newpath = maildir_write_new (config, STDIN_FILENO, maildir);
     if (! newpath) {
diff --git a/notmuch-new.c b/notmuch-new.c
index 895f5d9..e494a3b 100644
--- a/notmuch-new.c
+++ b/notmuch-new.c
@@ -995,10 +995,11 @@ notmuch_new_command (notmuch_config_t *config, int argc, char *argv[])
 		fputs (status_string, stderr);
 		free (status_string);
 	    }
-
 	    return EXIT_FAILURE;
 	}
 
+	notmuch_exit_if_unmatched_db_revision (notmuch);
+
 	if (notmuch_database_needs_upgrade (notmuch)) {
 	    time_t now = time (NULL);
 	    struct tm *gm_time = gmtime (&now);
diff --git a/notmuch-reply.c b/notmuch-reply.c
index 4464741..52e0d9b 100644
--- a/notmuch-reply.c
+++ b/notmuch-reply.c
@@ -831,6 +831,8 @@ notmuch_reply_command (notmuch_config_t *config, int argc, char *argv[])
 			       NOTMUCH_DATABASE_MODE_READ_ONLY, &notmuch))
 	return EXIT_FAILURE;
 
+    notmuch_exit_if_unmatched_db_revision (notmuch);
+
     query = notmuch_query_create (notmuch, query_string);
     if (query == NULL) {
 	fprintf (stderr, "Out of memory\n");
diff --git a/notmuch-restore.c b/notmuch-restore.c
index 2a534dc..3e96aa8 100644
--- a/notmuch-restore.c
+++ b/notmuch-restore.c
@@ -165,6 +165,8 @@ notmuch_restore_command (notmuch_config_t *config, int argc, char *argv[])
     }
 
     notmuch_process_shared_options (argv[0]);
+    notmuch_exit_if_unmatched_db_revision (notmuch);
+
     name_for_error = input_file_name ? input_file_name : "stdin";
 
     if (! accumulate)
diff --git a/notmuch-search.c b/notmuch-search.c
index 5d17dac..453a041 100644
--- a/notmuch-search.c
+++ b/notmuch-search.c
@@ -583,6 +583,8 @@ _notmuch_search_prepare (search_context_t *ctx, notmuch_config_t *config, int ar
 	return EXIT_FAILURE;
     }
 
+    notmuch_exit_if_unmatched_db_revision (ctx->notmuch);
+
     query_str = query_string_from_args (ctx->notmuch, argc, argv);
     if (query_str == NULL) {
 	fprintf (stderr, "Out of memory.\n");
diff --git a/notmuch-setup.c b/notmuch-setup.c
index 5fc6e25..ddc6491 100644
--- a/notmuch-setup.c
+++ b/notmuch-setup.c
@@ -155,6 +155,10 @@ notmuch_setup_command (notmuch_config_t *config,
     if (opt_index < 0)
 	return EXIT_FAILURE;
 
+    if (notmuch_requested_db_revision)
+	fprintf (stderr, "Warning: ignoring --db-revision=%s\n",
+		 notmuch_requested_db_revision);
+
     notmuch_process_shared_options (argv[0]);
 
     if (notmuch_config_is_new (config))
diff --git a/notmuch-show.c b/notmuch-show.c
index 3917b82..de16db3 100644
--- a/notmuch-show.c
+++ b/notmuch-show.c
@@ -1220,6 +1220,8 @@ notmuch_show_command (notmuch_config_t *config, int argc, char *argv[])
 			       NOTMUCH_DATABASE_MODE_READ_ONLY, &notmuch))
 	return EXIT_FAILURE;
 
+    notmuch_exit_if_unmatched_db_revision (notmuch);
+
     query = notmuch_query_create (notmuch, query_string);
     if (query == NULL) {
 	fprintf (stderr, "Out of memory\n");
diff --git a/notmuch-tag.c b/notmuch-tag.c
index 35f971d..2c54eec 100644
--- a/notmuch-tag.c
+++ b/notmuch-tag.c
@@ -261,6 +261,8 @@ notmuch_tag_command (notmuch_config_t *config, int argc, char *argv[])
 			       NOTMUCH_DATABASE_MODE_READ_WRITE, &notmuch))
 	return EXIT_FAILURE;
 
+    notmuch_exit_if_unmatched_db_revision (notmuch);
+
     if (notmuch_config_get_maildir_synchronize_flags (config))
 	tag_flags |= TAG_FLAG_MAILDIR_SYNC;
 
diff --git a/notmuch.c b/notmuch.c
index c7f8c8f..90e6de3 100644
--- a/notmuch.c
+++ b/notmuch.c
@@ -47,10 +47,12 @@ static int
 _help_for (const char *topic);
 
 static notmuch_bool_t print_version = FALSE, print_help = FALSE;
+char *notmuch_requested_db_revision = NULL;
 
 const notmuch_opt_desc_t notmuch_shared_options [] = {
     { NOTMUCH_OPT_BOOLEAN, &print_version, "version", 'v', 0 },
     { NOTMUCH_OPT_BOOLEAN, &print_help, "help", 'h', 0 },
+    { NOTMUCH_OPT_STRING, &notmuch_requested_db_revision, "db-revision", 'd', 0 },
     {0, 0, 0, 0, 0}
 };
 
@@ -196,6 +198,22 @@ be supported in the future.\n", notmuch_format_version);
     }
 }
 
+void
+notmuch_exit_if_unmatched_db_revision (notmuch_database_t *notmuch)
+{
+    const char *uuid = NULL;
+
+    if (!notmuch_requested_db_revision)
+	return;
+    IGNORE_RESULT (notmuch_database_get_revision (notmuch, &uuid));
+
+    if (strcmp (notmuch_requested_db_revision, uuid) != 0){
+	fprintf (stderr, "Error: requested database revision %s does not match %s\n",
+		 notmuch_requested_db_revision, uuid);
+	exit (1);
+    }
+}
+
 static void
 exec_man (const char *page)
 {
-- 
2.1.4

Revision tracking, round 2

[Gaute Hope]

Excerpts from David Bremner's message of April 6, 2015 0:59:
> There is lots to tidy up here, but the series has balooned to 12 patches since since Austin posted
>
>       id:1413181203-1676-1-git-send-email-aclements@csail.mit.edu
>
> So I figure I better post it to get some feedback.
>
> These are unmodified from Austin's post
>  [WIP2 01/12] lib: Only sync modified message documents
>  [WIP2 02/12] lib: Add per-message last modification tracking
>  [WIP2 07/12] lib: Add "lastmod:" queries for filtering by last
>
> This has two small bug fixes, a typo and an extra * removed.
>  [WIP2 03/12] lib: API to retrieve database revision and UUID
>
> Together, the next 5  patches implement output format 3
>  [WIP2 04/12] cli: add type introspection to sprinter type
>  [WIP2 05/12] cli: add two sprinter utility functions
>  [WIP2 06/12] cli/show: add extra element to structured output for
>  [WIP2 08/12] cli/show: add lastmod to structured output
>  [WIP2 09/12] cli/search: add  metadata element to structured output
>
> These two do a trivial convesion of parts of the emacs interface to use the new format:
>  [WIP2 10/12] emacs: convert notmuch-search to format-version 3
>  [WIP2 11/12] emacs: convert notmuch-tree to format-version 3
>
> Finally, add db-revision argument. I'm not sure about the name, but I
> went with what Austin had written.  This patch should probably be earlier in the sequence
>
>  [WIP2 12/12] cli: add global option "--db-revision"

I have been running your 'revision-tracking' branch from
git://pivot.cs.unb.ca/notmuch for a week now, with heavy usage of the
revision features (I used the old series before that). This has all
worked without problems so far, and performance seems to be good.

- gaute

revision tracking patches, round 1

[David Bremner]

Since the previous WIP posting 

     id:1428274754-1698-10-git-send-email-david@tethera.net

I've reorganized a bit and added some new tests, and an option to
notmuch-count to output the global database revision.  I think the
first 6 patches are probably ready for serious review at this point.

The remaining patches in the series make the modification count more
useful (mainly by including it in structured output formats), but
these are already enough for library users (such as Gaute, who has
reported success running them).

[PATCH 1/6] lib: Only sync modified message documents

[David Bremner]

From: Austin Clements <amdragon@mit.edu>

Previously, we updated the database copy of a message on every call to
_notmuch_message_sync, even if nothing had changed.  In particular,
this always happens on a thaw, so a freeze/thaw pair with no
modifications between still caused a database update.

We only modify message documents in a handful of places, so keep track
of whether the document has been modified and only sync it when
necessary.  This will be particularly important when we add message
revision tracking.
---
 lib/message.cc | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/lib/message.cc b/lib/message.cc
index 5bc7aff..1ddce3c 100644
--- a/lib/message.cc
+++ b/lib/message.cc
@@ -43,6 +43,9 @@ struct visible _notmuch_message {
      * if each flag has been initialized. */
     unsigned long lazy_flags;
 
+    /* Message document modified since last sync */
+    notmuch_bool_t modified;
+
     Xapian::Document doc;
     Xapian::termcount termpos;
 };
@@ -539,6 +542,7 @@ _notmuch_message_remove_terms (notmuch_message_t *message, const char *prefix)
 
 	try {
 	    message->doc.remove_term ((*i));
+	    message->modified = TRUE;
 	} catch (const Xapian::InvalidArgumentError) {
 	    /* Ignore failure to remove non-existent term. */
 	}
@@ -793,6 +797,7 @@ void
 _notmuch_message_clear_data (notmuch_message_t *message)
 {
     message->doc.set_data ("");
+    message->modified = TRUE;
 }
 
 static void
@@ -990,6 +995,7 @@ _notmuch_message_set_header_values (notmuch_message_t *message,
 			    Xapian::sortable_serialise (time_value));
     message->doc.add_value (NOTMUCH_VALUE_FROM, from);
     message->doc.add_value (NOTMUCH_VALUE_SUBJECT, subject);
+    message->modified = TRUE;
 }
 
 /* Synchronize changes made to message->doc out into the database. */
@@ -1001,8 +1007,12 @@ _notmuch_message_sync (notmuch_message_t *message)
     if (message->notmuch->mode == NOTMUCH_DATABASE_MODE_READ_ONLY)
 	return;
 
+    if (! message->modified)
+	return;
+
     db = static_cast <Xapian::WritableDatabase *> (message->notmuch->xapian_db);
     db->replace_document (message->doc_id, message->doc);
+    message->modified = FALSE;
 }
 
 /* Delete a message document from the database. */
@@ -1077,6 +1087,7 @@ _notmuch_message_add_term (notmuch_message_t *message,
 	return NOTMUCH_PRIVATE_STATUS_TERM_TOO_LONG;
 
     message->doc.add_term (term, 0);
+    message->modified = TRUE;
 
     talloc_free (term);
 
@@ -1145,6 +1156,7 @@ _notmuch_message_remove_term (notmuch_message_t *message,
 
     try {
 	message->doc.remove_term (term);
+	message->modified = TRUE;
     } catch (const Xapian::InvalidArgumentError) {
 	/* We'll let the philosopher's try to wrestle with the
 	 * question of whether failing to remove that which was not
-- 
2.1.4

Re: [PATCH 1/6] lib: Only sync modified message documents

[David Bremner]

David Bremner <david@tethera.net> writes:

> From: Austin Clements <amdragon@mit.edu>
>
> Previously, we updated the database copy of a message on every call to
> _notmuch_message_sync, even if nothing had changed.  In particular,
> this always happens on a thaw, so a freeze/thaw pair with no
> modifications between still caused a database update.
>
> We only modify message documents in a handful of places, so keep track
> of whether the document has been modified and only sync it when
> necessary.  This will be particularly important when we add message
> revision tracking.

This patch makes some sense without the rest of lastmod series, so I
merged it early to get some testing.

d

[PATCH 2/6] lib: Add per-message last modification tracking

[David Bremner]

From: Austin Clements <amdragon@mit.edu>

This adds a new document value that stores the revision of the last
modification to message metadata, where the revision number increases
monotonically with each database commit.

An alternative would be to store the wall-clock time of the last
modification of each message.  In principle this is simpler and has
the advantage that any process can determine the current timestamp
without support from libnotmuch.  However, even assuming a computer's
clock never goes backward and ignoring clock skew in networked
environments, this has a fatal flaw.  Xapian uses (optimistic)
snapshot isolation, which means reads can be concurrent with writes.
Given this, consider the following time line with a write and two read
transactions:

   write  |-X-A--------------|
   read 1       |---B---|
   read 2                      |---|

The write transaction modifies message X and records the wall-clock
time of the modification at A.  The writer hangs around for a while
and later commits its change.  Read 1 is concurrent with the write, so
it doesn't see the change to X.  It does some query and records the
wall-clock time of its results at B.  Transaction read 2 later starts
after the write commits and queries for changes since wall-clock time
B (say the reads are performing an incremental backup).  Even though
read 1 could not see the change to X, read 2 is told (correctly) that
X has not changed since B, the time of the last read.  In fact, X
changed before wall-clock time A, but the change was not visible until
*after* wall-clock time B, so read 2 misses the change to X.

This is tricky to solve in full-blown snapshot isolation, but because
Xapian serializes writes, we can use a simple, monotonically
increasing database revision number.  Furthermore, maintaining this
revision number requires no more IO than a wall-clock time solution
because Xapian already maintains statistics on the upper (and lower)
bound of each value stream.
---
 lib/database-private.h | 16 +++++++++++++++-
 lib/database.cc        | 49 +++++++++++++++++++++++++++++++++++++++++++++++--
 lib/message.cc         | 22 ++++++++++++++++++++++
 lib/notmuch-private.h  | 10 +++++++++-
 4 files changed, 93 insertions(+), 4 deletions(-)

diff --git a/lib/database-private.h b/lib/database-private.h
index 24243db..5c5a2bb 100644
--- a/lib/database-private.h
+++ b/lib/database-private.h
@@ -100,6 +100,12 @@ enum _notmuch_features {
      *
      * Introduced: version 3. */
     NOTMUCH_FEATURE_INDEXED_MIMETYPES = 1 << 5,
+
+    /* If set, messages store the revision number of the last
+     * modification in NOTMUCH_VALUE_LAST_MOD.
+     *
+     * Introduced: version 3. */
+    NOTMUCH_FEATURE_LAST_MOD = 1 << 6,
 };
 
 /* In C++, a named enum is its own type, so define bitwise operators
@@ -145,6 +151,8 @@ struct _notmuch_database {
 
     notmuch_database_mode_t mode;
     int atomic_nesting;
+    /* TRUE if changes have been made in this atomic section */
+    notmuch_bool_t atomic_dirty;
     Xapian::Database *xapian_db;
 
     /* Bit mask of features used by this database.  This is a
@@ -158,6 +166,11 @@ struct _notmuch_database {
      * next library call. May be NULL */
     char *status_string;
 
+    /* Highest committed revision number.  Modifications are recorded
+     * under a higher revision number, which can be generated with
+     * notmuch_database_new_revision. */
+    unsigned long revision;
+
     Xapian::QueryParser *query_parser;
     Xapian::TermGenerator *term_gen;
     Xapian::ValueRangeProcessor *value_range_processor;
@@ -179,7 +192,8 @@ struct _notmuch_database {
  * will have it). */
 #define NOTMUCH_FEATURES_CURRENT \
     (NOTMUCH_FEATURE_FILE_TERMS | NOTMUCH_FEATURE_DIRECTORY_DOCS | \
-     NOTMUCH_FEATURE_BOOL_FOLDER | NOTMUCH_FEATURE_GHOSTS)
+     NOTMUCH_FEATURE_BOOL_FOLDER | NOTMUCH_FEATURE_GHOSTS | \
+     NOTMUCH_FEATURE_LAST_MOD)
 
 /* Return the list of terms from the given iterator matching a prefix.
  * The prefix will be stripped from the strings in the returned list.
diff --git a/lib/database.cc b/lib/database.cc
index 78a24f7..a68a487 100644
--- a/lib/database.cc
+++ b/lib/database.cc
@@ -101,6 +101,9 @@ typedef struct {
  *
  *	SUBJECT:	The value of the "Subject" header
  *
+ *	LAST_MOD:	The revision number as of the last tag or
+ *			filename change.
+ *
  * In addition, terms from the content of the message are added with
  * "from", "to", "attachment", and "subject" prefixes for use by the
  * user in searching. Similarly, terms from the path of the mail
@@ -310,6 +313,8 @@ static const struct {
      * them. */
     { NOTMUCH_FEATURE_INDEXED_MIMETYPES,
       "indexed MIME types", "w"},
+    { NOTMUCH_FEATURE_LAST_MOD,
+      "modification tracking", "w"},
 };
 
 const char *
@@ -729,6 +734,23 @@ _notmuch_database_ensure_writable (notmuch_database_t *notmuch)
     return NOTMUCH_STATUS_SUCCESS;
 }
 
+/* Allocate a revision number for the next change. */
+unsigned long
+_notmuch_database_new_revision (notmuch_database_t *notmuch)
+{
+    unsigned long new_revision = notmuch->revision + 1;
+
+    /* If we're in an atomic section, hold off on updating the
+     * committed revision number until we commit the atomic section.
+     */
+    if (notmuch->atomic_nesting)
+	notmuch->atomic_dirty = TRUE;
+    else
+	notmuch->revision = new_revision;
+
+    return new_revision;
+}
+
 /* Parse a database features string from the given database version.
  * Returns the feature bit set.
  *
@@ -890,6 +912,7 @@ notmuch_database_open_verbose (const char *path,
     notmuch->atomic_nesting = 0;
     try {
 	string last_thread_id;
+	string last_mod;
 
 	if (mode == NOTMUCH_DATABASE_MODE_READ_WRITE) {
 	    notmuch->xapian_db = new Xapian::WritableDatabase (xapian_path,
@@ -948,6 +971,14 @@ notmuch_database_open_verbose (const char *path,
 		INTERNAL_ERROR ("Malformed database last_thread_id: %s", str);
 	}
 
+	/* Get current highest revision number. */
+	last_mod = notmuch->xapian_db->get_value_upper_bound (
+	    NOTMUCH_VALUE_LAST_MOD);
+	if (last_mod.empty ())
+	    notmuch->revision = 0;
+	else
+	    notmuch->revision = Xapian::sortable_unserialise (last_mod);
+
 	notmuch->query_parser = new Xapian::QueryParser;
 	notmuch->term_gen = new Xapian::TermGenerator;
 	notmuch->term_gen->set_stemmer (Xapian::Stem ("english"));
@@ -1355,7 +1386,8 @@ notmuch_database_upgrade (notmuch_database_t *notmuch,
 
     /* Figure out how much total work we need to do. */
     if (new_features &
-	(NOTMUCH_FEATURE_FILE_TERMS | NOTMUCH_FEATURE_BOOL_FOLDER)) {
+	(NOTMUCH_FEATURE_FILE_TERMS | NOTMUCH_FEATURE_BOOL_FOLDER |
+	 NOTMUCH_FEATURE_LAST_MOD)) {
 	notmuch_query_t *query = notmuch_query_create (notmuch, "");
 	total += notmuch_query_count_messages (query);
 	notmuch_query_destroy (query);
@@ -1382,7 +1414,8 @@ notmuch_database_upgrade (notmuch_database_t *notmuch,
 
     /* Perform per-message upgrades. */
     if (new_features &
-	(NOTMUCH_FEATURE_FILE_TERMS | NOTMUCH_FEATURE_BOOL_FOLDER)) {
+	(NOTMUCH_FEATURE_FILE_TERMS | NOTMUCH_FEATURE_BOOL_FOLDER |
+	 NOTMUCH_FEATURE_LAST_MOD)) {
 	notmuch_query_t *query = notmuch_query_create (notmuch, "");
 	notmuch_messages_t *messages;
 	notmuch_message_t *message;
@@ -1419,6 +1452,13 @@ notmuch_database_upgrade (notmuch_database_t *notmuch,
 	    if (new_features & NOTMUCH_FEATURE_BOOL_FOLDER)
 		_notmuch_message_upgrade_folder (message);
 
+	    /* Prior to NOTMUCH_FEATURE_LAST_MOD, messages did not
+	     * track modification revisions.  Give all messages a
+	     * revision of 1.
+	     */
+	    if (new_features & NOTMUCH_FEATURE_LAST_MOD)
+		_notmuch_message_upgrade_last_mod (message);
+
 	    _notmuch_message_sync (message);
 
 	    notmuch_message_destroy (message);
@@ -1601,6 +1641,11 @@ notmuch_database_end_atomic (notmuch_database_t *notmuch)
 	return NOTMUCH_STATUS_XAPIAN_EXCEPTION;
     }
 
+    if (notmuch->atomic_dirty) {
+	++notmuch->revision;
+	notmuch->atomic_dirty = FALSE;
+    }
+
 DONE:
     notmuch->atomic_nesting--;
     return NOTMUCH_STATUS_SUCCESS;
diff --git a/lib/message.cc b/lib/message.cc
index 1ddce3c..26b5e76 100644
--- a/lib/message.cc
+++ b/lib/message.cc
@@ -998,6 +998,16 @@ _notmuch_message_set_header_values (notmuch_message_t *message,
     message->modified = TRUE;
 }
 
+/* Upgrade a message to support NOTMUCH_FEATURE_LAST_MOD.  The caller
+ * must call _notmuch_message_sync. */
+void
+_notmuch_message_upgrade_last_mod (notmuch_message_t *message)
+{
+    /* _notmuch_message_sync will update the last modification
+     * revision; we just have to ask it to. */
+    message->modified = TRUE;
+}
+
 /* Synchronize changes made to message->doc out into the database. */
 void
 _notmuch_message_sync (notmuch_message_t *message)
@@ -1010,6 +1020,18 @@ _notmuch_message_sync (notmuch_message_t *message)
     if (! message->modified)
 	return;
 
+    /* Update the last modification of this message. */
+    if (message->notmuch->features & NOTMUCH_FEATURE_LAST_MOD)
+	/* sortable_serialise gives a reasonably compact encoding,
+	 * which directly translates to reduced IO when scanning the
+	 * value stream.  Since it's built for doubles, we only get 53
+	 * effective bits, but that's still enough for the database to
+	 * last a few centuries at 1 million revisions per second. */
+	message->doc.add_value (NOTMUCH_VALUE_LAST_MOD,
+				Xapian::sortable_serialise (
+				    _notmuch_database_new_revision (
+					message->notmuch)));
+
     db = static_cast <Xapian::WritableDatabase *> (message->notmuch->xapian_db);
     db->replace_document (message->doc_id, message->doc);
     message->modified = FALSE;
diff --git a/lib/notmuch-private.h b/lib/notmuch-private.h
index cc9ce12..f52b4e4 100644
--- a/lib/notmuch-private.h
+++ b/lib/notmuch-private.h
@@ -107,7 +107,8 @@ typedef enum {
     NOTMUCH_VALUE_TIMESTAMP = 0,
     NOTMUCH_VALUE_MESSAGE_ID,
     NOTMUCH_VALUE_FROM,
-    NOTMUCH_VALUE_SUBJECT
+    NOTMUCH_VALUE_SUBJECT,
+    NOTMUCH_VALUE_LAST_MOD,
 } notmuch_value_t;
 
 /* Xapian (with flint backend) complains if we provide a term longer
@@ -194,6 +195,9 @@ void
 _notmuch_database_log (notmuch_database_t *notmuch,
 		       const char *format, ...);
 
+unsigned long
+_notmuch_database_new_revision (notmuch_database_t *notmuch);
+
 const char *
 _notmuch_database_relative_path (notmuch_database_t *notmuch,
 				 const char *path);
@@ -305,6 +309,10 @@ _notmuch_message_set_header_values (notmuch_message_t *message,
 				    const char *date,
 				    const char *from,
 				    const char *subject);
+
+void
+_notmuch_message_upgrade_last_mod (notmuch_message_t *message);
+
 void
 _notmuch_message_sync (notmuch_message_t *message);
 
-- 
2.1.4

Re: [PATCH 2/6] lib: Add per-message last modification tracking

[Daniel Schoepe]

Hi,

On Fri, 05 Jun 2015 19:28 +0200, David Bremner wrote:
> +	    /* Prior to NOTMUCH_FEATURE_LAST_MOD, messages did not
> +	     * track modification revisions.  Give all messages a
> +	     * revision of 1.
> +	     */
> +	    if (new_features & NOTMUCH_FEATURE_LAST_MOD)
> +		_notmuch_message_upgrade_last_mod (message);
> [..]
> +/* Upgrade a message to support NOTMUCH_FEATURE_LAST_MOD.  The caller
> + * must call _notmuch_message_sync. */
> +void
> +_notmuch_message_upgrade_last_mod (notmuch_message_t *message)
> +{
> +    /* _notmuch_message_sync will update the last modification
> +     * revision; we just have to ask it to. */
> +    message->modified = TRUE;
> +}
> +

The comment in the first part says that message without LAST_MOD get a
revision of 1, but as far as I can tell, _notmuch_message_sync will
actually put the next revision number on the message. If this is what's
happening, either the comment or the behavior should be changed,
depending on what's intended.

Best regards,
Daniel

Re: [PATCH 2/6] lib: Add per-message last modification tracking

[David Bremner]

Daniel Schoepe <daniel@schoepe.org> writes:


> On Fri, 05 Jun 2015 19:28 +0200, David Bremner wrote:
>> +	    /* Prior to NOTMUCH_FEATURE_LAST_MOD, messages did not
>> +	     * track modification revisions.  Give all messages a
>> +	     * revision of 1.
>> +	     */
>> +	    if (new_features & NOTMUCH_FEATURE_LAST_MOD)
>> +		_notmuch_message_upgrade_last_mod (message);
>> [..]
>> +/* Upgrade a message to support NOTMUCH_FEATURE_LAST_MOD.  The caller
>> + * must call _notmuch_message_sync. */
>> +void
>> +_notmuch_message_upgrade_last_mod (notmuch_message_t *message)
>> +{
>> +    /* _notmuch_message_sync will update the last modification
>> +     * revision; we just have to ask it to. */
>> +    message->modified = TRUE;
>> +}
>> +
>
> The comment in the first part says that message without LAST_MOD get a
> revision of 1, but as far as I can tell, _notmuch_message_sync will
> actually put the next revision number on the message. If this is what's
> happening, either the comment or the behavior should be changed,
> depending on what's intended.

I think the behaviour is OK, but you're correct the comment is
wrong. I'll see if Austin has any comment on that. I guess it would be
harmless to initialize upgraded messages to some low revision number,
but I don't see the benefit.

Re: [PATCH 2/6] lib: Add per-message last modification tracking

[Austin Clements]

[-- Attachment #1: Type: text/plain, Size: 1602 bytes --]

Hi David. I haven't had a chance to look back at the original code, but
your follow-up expanded comment agrees with how I remember this code
working.
On Aug 7, 2015 4:41 PM, "David Bremner" <david@tethera.net> wrote:

> Daniel Schoepe <daniel@schoepe.org> writes:
>
>
> > On Fri, 05 Jun 2015 19:28 +0200, David Bremner wrote:
> >> +        /* Prior to NOTMUCH_FEATURE_LAST_MOD, messages did not
> >> +         * track modification revisions.  Give all messages a
> >> +         * revision of 1.
> >> +         */
> >> +        if (new_features & NOTMUCH_FEATURE_LAST_MOD)
> >> +            _notmuch_message_upgrade_last_mod (message);
> >> [..]
> >> +/* Upgrade a message to support NOTMUCH_FEATURE_LAST_MOD.  The caller
> >> + * must call _notmuch_message_sync. */
> >> +void
> >> +_notmuch_message_upgrade_last_mod (notmuch_message_t *message)
> >> +{
> >> +    /* _notmuch_message_sync will update the last modification
> >> +     * revision; we just have to ask it to. */
> >> +    message->modified = TRUE;
> >> +}
> >> +
> >
> > The comment in the first part says that message without LAST_MOD get a
> > revision of 1, but as far as I can tell, _notmuch_message_sync will
> > actually put the next revision number on the message. If this is what's
> > happening, either the comment or the behavior should be changed,
> > depending on what's intended.
>
> I think the behaviour is OK, but you're correct the comment is
> wrong. I'll see if Austin has any comment on that. I guess it would be
> harmless to initialize upgraded messages to some low revision number,
> but I don't see the benefit.
>

[-- Attachment #2: Type: text/html, Size: 2179 bytes --]

[PATCH] fixup! lib: Add per-message last modification tracking

[David Bremner]

---

Now that I take a second look at the code, I think Austin's original
commit is correct, if a bit terse. Here is how I have expanded it in
the revised series.

 lib/database.cc | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/lib/database.cc b/lib/database.cc
index 0650c63..bab3334 100644
--- a/lib/database.cc
+++ b/lib/database.cc
@@ -1473,8 +1473,9 @@ notmuch_database_upgrade (notmuch_database_t *notmuch,
 		_notmuch_message_upgrade_folder (message);
 
 	    /* Prior to NOTMUCH_FEATURE_LAST_MOD, messages did not
-	     * track modification revisions.  Give all messages a
-	     * revision of 1.
+	     * track modification revisions.  Give all messages the
+	     * next available revision; since we just started tracking
+	     * revisions for this database, that will be 1.
 	     */
 	    if (new_features & NOTMUCH_FEATURE_LAST_MOD)
 		_notmuch_message_upgrade_last_mod (message);
-- 
2.1.4

[PATCH 3/6] lib: API to retrieve database revision and UUID

[David Bremner]

From: Austin Clements <aclements@csail.mit.edu>

This exposes the committed database revision to library users along
with a UUID that can be used to detect when revision numbers are no
longer comparable (e.g., because the database has been replaced).
---
 lib/database-private.h         |  1 +
 lib/database.cc                | 11 +++++++++++
 lib/notmuch.h                  | 18 ++++++++++++++++++
 test/T570-revision-tracking.sh | 37 +++++++++++++++++++++++++++++++++++++
 test/test-lib.sh               |  5 +++++
 5 files changed, 72 insertions(+)
 create mode 100755 test/T570-revision-tracking.sh

diff --git a/lib/database-private.h b/lib/database-private.h
index 5c5a2bb..4e93257 100644
--- a/lib/database-private.h
+++ b/lib/database-private.h
@@ -170,6 +170,7 @@ struct _notmuch_database {
      * under a higher revision number, which can be generated with
      * notmuch_database_new_revision. */
     unsigned long revision;
+    const char *uuid;
 
     Xapian::QueryParser *query_parser;
     Xapian::TermGenerator *term_gen;
diff --git a/lib/database.cc b/lib/database.cc
index a68a487..ba8b8d9 100644
--- a/lib/database.cc
+++ b/lib/database.cc
@@ -978,6 +978,8 @@ notmuch_database_open_verbose (const char *path,
 	    notmuch->revision = 0;
 	else
 	    notmuch->revision = Xapian::sortable_unserialise (last_mod);
+	notmuch->uuid = talloc_strdup (
+	    notmuch, notmuch->xapian_db->get_uuid ().c_str ());
 
 	notmuch->query_parser = new Xapian::QueryParser;
 	notmuch->term_gen = new Xapian::TermGenerator;
@@ -1651,6 +1653,15 @@ DONE:
     return NOTMUCH_STATUS_SUCCESS;
 }
 
+unsigned long
+notmuch_database_get_revision (notmuch_database_t *notmuch,
+				const char **uuid)
+{
+    if (uuid)
+	*uuid = notmuch->uuid;
+    return notmuch->revision;
+}
+
 /* We allow the user to use arbitrarily long paths for directories. But
  * we have a term-length limit. So if we exceed that, we'll use the
  * SHA-1 of the path for the database term.
diff --git a/lib/notmuch.h b/lib/notmuch.h
index 20c4e01..b6be727 100644
--- a/lib/notmuch.h
+++ b/lib/notmuch.h
@@ -461,6 +461,24 @@ notmuch_status_t
 notmuch_database_end_atomic (notmuch_database_t *notmuch);
 
 /**
+ * Return the committed database revision and UUID.
+ *
+ * The database revision number increases monotonically with each
+ * commit to the database.  Hence, all messages and message changes
+ * committed to the database (that is, visible to readers) have a last
+ * modification revision <= the committed database revision.  Any
+ * messages committed in the future will be assigned a modification
+ * revision > the committed database revision.
+ *
+ * The UUID is a NUL-terminated opaque string that uniquely identifies
+ * this database.  Two revision numbers are only comparable if they
+ * have the same database UUID.
+ */
+unsigned long
+notmuch_database_get_revision (notmuch_database_t *notmuch,
+				const char **uuid);
+
+/**
  * Retrieve a directory object from the database for 'path'.
  *
  * Here, 'path' should be a path relative to the path of 'database'
diff --git a/test/T570-revision-tracking.sh b/test/T570-revision-tracking.sh
new file mode 100755
index 0000000..74a7c49
--- /dev/null
+++ b/test/T570-revision-tracking.sh
@@ -0,0 +1,37 @@
+#!/usr/bin/env bash
+test_description="database revision tracking"
+
+. ./test-lib.sh
+
+add_email_corpus
+
+test_begin_subtest "notmuch_database_get_revision"
+test_C ${MAIL_DIR} <<'EOF'
+#include <stdio.h>
+#include <string.h>
+#include <notmuch.h>
+int main (int argc, char** argv)
+{
+   notmuch_database_t *db;
+   notmuch_status_t stat;
+   unsigned long revision;
+   const char *uuid;
+
+   unsigned long rev;
+
+   stat = notmuch_database_open (argv[1], NOTMUCH_DATABASE_MODE_READ_ONLY, &db);
+   if (stat)
+       fputs ("open failed\n", stderr);
+   revision = notmuch_database_get_revision (db, &uuid);
+   printf("%s\t%lu\n", uuid, revision);
+}
+EOF
+notmuch_uuid_sanitize < OUTPUT > CLEAN
+cat <<'EOF' >EXPECTED
+== stdout ==
+UUID	53
+== stderr ==
+EOF
+test_expect_equal_file EXPECTED CLEAN
+
+test_done
diff --git a/test/test-lib.sh b/test/test-lib.sh
index 23085e7..1ec6c5a 100644
--- a/test/test-lib.sh
+++ b/test/test-lib.sh
@@ -719,6 +719,11 @@ notmuch_date_sanitize ()
     sed \
 	-e 's/^Date: Fri, 05 Jan 2001 .*0000/Date: GENERATED_DATE/'
 }
+
+notmuch_uuid_sanitize ()
+{
+    sed  's/^[a-f0-9][a-f0-9-]*/UUID/'
+}
 # End of notmuch helper functions
 
 # Use test_set_prereq to tell that a particular prerequisite is available.
-- 
2.1.4

Re: [PATCH 3/6] lib: API to retrieve database revision and UUID

[Tomi Ollila]

On Fri, Jun 05 2015, David Bremner <david@tethera.net> wrote:

> From: Austin Clements <aclements@csail.mit.edu>
>
> This exposes the committed database revision to library users along
> with a UUID that can be used to detect when revision numbers are no
> longer comparable (e.g., because the database has been replaced).
> ---
>  lib/database-private.h         |  1 +
>  lib/database.cc                | 11 +++++++++++
>  lib/notmuch.h                  | 18 ++++++++++++++++++
>  test/T570-revision-tracking.sh | 37 +++++++++++++++++++++++++++++++++++++
>  test/test-lib.sh               |  5 +++++
>  5 files changed, 72 insertions(+)
>  create mode 100755 test/T570-revision-tracking.sh
>
> diff --git a/lib/database-private.h b/lib/database-private.h
> index 5c5a2bb..4e93257 100644
> --- a/lib/database-private.h
> +++ b/lib/database-private.h
> @@ -170,6 +170,7 @@ struct _notmuch_database {
>       * under a higher revision number, which can be generated with
>       * notmuch_database_new_revision. */
>      unsigned long revision;
> +    const char *uuid;
>  
>      Xapian::QueryParser *query_parser;
>      Xapian::TermGenerator *term_gen;
> diff --git a/lib/database.cc b/lib/database.cc
> index a68a487..ba8b8d9 100644
> --- a/lib/database.cc
> +++ b/lib/database.cc
> @@ -978,6 +978,8 @@ notmuch_database_open_verbose (const char *path,
>  	    notmuch->revision = 0;
>  	else
>  	    notmuch->revision = Xapian::sortable_unserialise (last_mod);
> +	notmuch->uuid = talloc_strdup (
> +	    notmuch, notmuch->xapian_db->get_uuid ().c_str ());
>  
>  	notmuch->query_parser = new Xapian::QueryParser;
>  	notmuch->term_gen = new Xapian::TermGenerator;
> @@ -1651,6 +1653,15 @@ DONE:
>      return NOTMUCH_STATUS_SUCCESS;
>  }
>  
> +unsigned long
> +notmuch_database_get_revision (notmuch_database_t *notmuch,
> +				const char **uuid)
> +{
> +    if (uuid)
> +	*uuid = notmuch->uuid;
> +    return notmuch->revision;
> +}
> +
>  /* We allow the user to use arbitrarily long paths for directories. But
>   * we have a term-length limit. So if we exceed that, we'll use the
>   * SHA-1 of the path for the database term.
> diff --git a/lib/notmuch.h b/lib/notmuch.h
> index 20c4e01..b6be727 100644
> --- a/lib/notmuch.h
> +++ b/lib/notmuch.h
> @@ -461,6 +461,24 @@ notmuch_status_t
>  notmuch_database_end_atomic (notmuch_database_t *notmuch);
>  
>  /**
> + * Return the committed database revision and UUID.
> + *
> + * The database revision number increases monotonically with each
> + * commit to the database.  Hence, all messages and message changes
> + * committed to the database (that is, visible to readers) have a last
> + * modification revision <= the committed database revision.  Any
> + * messages committed in the future will be assigned a modification
> + * revision > the committed database revision.
> + *
> + * The UUID is a NUL-terminated opaque string that uniquely identifies
> + * this database.  Two revision numbers are only comparable if they
> + * have the same database UUID.
> + */
> +unsigned long
> +notmuch_database_get_revision (notmuch_database_t *notmuch,
> +				const char **uuid);
> +
> +/**
>   * Retrieve a directory object from the database for 'path'.
>   *
>   * Here, 'path' should be a path relative to the path of 'database'
> diff --git a/test/T570-revision-tracking.sh b/test/T570-revision-tracking.sh
> new file mode 100755
> index 0000000..74a7c49
> --- /dev/null
> +++ b/test/T570-revision-tracking.sh
> @@ -0,0 +1,37 @@
> +#!/usr/bin/env bash
> +test_description="database revision tracking"
> +
> +. ./test-lib.sh
> +
> +add_email_corpus
> +
> +test_begin_subtest "notmuch_database_get_revision"
> +test_C ${MAIL_DIR} <<'EOF'
> +#include <stdio.h>
> +#include <string.h>
> +#include <notmuch.h>
> +int main (int argc, char** argv)
> +{
> +   notmuch_database_t *db;
> +   notmuch_status_t stat;
> +   unsigned long revision;
> +   const char *uuid;
> +
> +   unsigned long rev;
> +
> +   stat = notmuch_database_open (argv[1], NOTMUCH_DATABASE_MODE_READ_ONLY, &db);
> +   if (stat)
> +       fputs ("open failed\n", stderr);
> +   revision = notmuch_database_get_revision (db, &uuid);
> +   printf("%s\t%lu\n", uuid, revision);
> +}
> +EOF
> +notmuch_uuid_sanitize < OUTPUT > CLEAN
> +cat <<'EOF' >EXPECTED
> +== stdout ==
> +UUID	53
> +== stderr ==
> +EOF
> +test_expect_equal_file EXPECTED CLEAN
> +
> +test_done
> diff --git a/test/test-lib.sh b/test/test-lib.sh
> index 23085e7..1ec6c5a 100644
> --- a/test/test-lib.sh
> +++ b/test/test-lib.sh
> @@ -719,6 +719,11 @@ notmuch_date_sanitize ()
>      sed \
>  	-e 's/^Date: Fri, 05 Jan 2001 .*0000/Date: GENERATED_DATE/'
>  }
> +
> +notmuch_uuid_sanitize ()
> +{
> +    sed  's/^[a-f0-9][a-f0-9-]*/UUID/'
> +}

Pretty much good so far. Here the output in "%s\t%lu\n" is good but
I am not sure whether in --count output... more of that later.

notmuch_uuid_sanitize () could be more generic:

notmuch_uuid_sanitize ()
{
    sed 's/[0-9a-f]\{8\}-[0-9a-f]\{4\}-[0-9a-f]\{4\}-[0-9a-f]\{4\}-[0-9a-f]\{12\}/UUID/g'
}

Oh, and `. ./test-lib.sh || exit 1` :D


Tomi

>  # End of notmuch helper functions
>  
>  # Use test_set_prereq to tell that a particular prerequisite is available.
> -- 
> 2.1.4

[PATCH 4/6] cli/count: add --output=modifications

[David Bremner]

We need some way to extract the uuid/revision of the database, and
count seems like the least bad choice of current commands.
The (perhaps weak) argument for count over search is that count
already reports statistics about the entire database.
---
 notmuch-count.c                | 18 +++++++++++++++++-
 test/T570-revision-tracking.sh | 12 ++++++++++++
 2 files changed, 29 insertions(+), 1 deletion(-)

diff --git a/notmuch-count.c b/notmuch-count.c
index 57a88a8..7c61ccb 100644
--- a/notmuch-count.c
+++ b/notmuch-count.c
@@ -25,6 +25,7 @@ enum {
     OUTPUT_THREADS,
     OUTPUT_MESSAGES,
     OUTPUT_FILES,
+    OUTPUT_LASTMOD,
 };
 
 /* The following is to allow future options to be added more easily */
@@ -71,6 +72,9 @@ print_count (notmuch_database_t *notmuch, const char *query_str,
 {
     notmuch_query_t *query;
     size_t i;
+    unsigned long revision;
+    const char *uuid;
+    int ret = 0;
 
     query = notmuch_query_create (notmuch, query_str);
     if (query == NULL) {
@@ -91,11 +95,22 @@ print_count (notmuch_database_t *notmuch, const char *query_str,
     case OUTPUT_FILES:
 	printf ("%u\n", count_files (query));
 	break;
+    case OUTPUT_LASTMOD:
+	if (strcmp (notmuch_query_get_query_string (query), "*") != 0) {
+	    fprintf (stderr, "Error: Only '*' is currently supported "
+		     " with output=modifications\n");
+	    ret = 1;
+	    goto DONE;
+	}
+
+	revision = notmuch_database_get_revision (notmuch, &uuid);
+	printf ("%s\t%lu\n", uuid, revision);
     }
 
+ DONE:
     notmuch_query_destroy (query);
 
-    return 0;
+    return ret;
 }
 
 static int
@@ -139,6 +154,7 @@ notmuch_count_command (notmuch_config_t *config, int argc, char *argv[])
 	  (notmuch_keyword_t []){ { "threads", OUTPUT_THREADS },
 				  { "messages", OUTPUT_MESSAGES },
 				  { "files", OUTPUT_FILES },
+				  { "modifications", OUTPUT_LASTMOD },
 				  { 0, 0 } } },
 	{ NOTMUCH_OPT_KEYWORD, &exclude, "exclude", 'x',
 	  (notmuch_keyword_t []){ { "true", EXCLUDE_TRUE },
diff --git a/test/T570-revision-tracking.sh b/test/T570-revision-tracking.sh
index 74a7c49..062fd59 100755
--- a/test/T570-revision-tracking.sh
+++ b/test/T570-revision-tracking.sh
@@ -34,4 +34,16 @@ UUID	53
 EOF
 test_expect_equal_file EXPECTED CLEAN
 
+grep '^[0-9a-f]' OUTPUT > INITIAL_OUTPUT
+
+test_begin_subtest "output of count matches test code"
+notmuch count --output=modifications '*' > OUTPUT
+test_expect_equal_file INITIAL_OUTPUT OUTPUT
+
+test_begin_subtest "modification count increases"
+before=$(notmuch count --output=modifications '*' | cut -f2)
+notmuch tag +a-random-tag-8743632 '*'
+after=$(notmuch count --output=modifications '*' | cut -f2)
+result=$(($before < $after))
+test_expect_equal 1 ${result}
 test_done
-- 
2.1.4

[PATCH 5/6] cli: add global option "--db-revision"

[David Bremner]

The function notmuch_exit_if_unmatched_db_revision is split from
notmuch_process_shared_options because it needs an open notmuch
database.
---
 doc/man1/notmuch.rst           | 10 ++++++++--
 notmuch-client.h               |  4 ++++
 notmuch-compact.c              |  4 ++++
 notmuch-config.c               |  4 ++++
 notmuch-count.c                |  2 ++
 notmuch-dump.c                 |  2 ++
 notmuch-insert.c               |  2 ++
 notmuch-new.c                  |  3 ++-
 notmuch-reply.c                |  2 ++
 notmuch-restore.c              |  2 ++
 notmuch-search.c               |  2 ++
 notmuch-setup.c                |  4 ++++
 notmuch-show.c                 |  2 ++
 notmuch-tag.c                  |  2 ++
 notmuch.c                      | 18 ++++++++++++++++++
 test/T570-revision-tracking.sh | 26 ++++++++++++++++++++++++++
 test/random-corpus.c           |  2 ++
 17 files changed, 88 insertions(+), 3 deletions(-)

diff --git a/doc/man1/notmuch.rst b/doc/man1/notmuch.rst
index ae0461a..b3f75a4 100644
--- a/doc/man1/notmuch.rst
+++ b/doc/man1/notmuch.rst
@@ -49,9 +49,15 @@ Supported global options for ``notmuch`` include
         Specify the configuration file to use. This overrides any
         configuration file specified by ${NOTMUCH\_CONFIG}.
 
+    ``--db-revision=UUID``
+       Enforce that the database revision is UUID. This is useful to
+       detect rollover in modification counts on messages. You can
+       find this UUID in the first column of output from
+       ``notmuch count --output=modifications``
+
 All global options except ``--config`` can also be specified after the
-command. For example, ``notmuch subcommand --version`` is equivalent to
-``notmuch --version subcommand``.
+command. For example, ``notmuch subcommand --db-revision`` is
+equivalent to ``notmuch --db-revision subcommand``.
 
 COMMANDS
 ========
diff --git a/notmuch-client.h b/notmuch-client.h
index 78680aa..d3fab21 100644
--- a/notmuch-client.h
+++ b/notmuch-client.h
@@ -466,7 +466,11 @@ notmuch_database_dump (notmuch_database_t *notmuch,
 		       notmuch_bool_t gzip_output);
 
 #include "command-line-arguments.h"
+
+extern char *notmuch_requested_db_revision;
 extern const notmuch_opt_desc_t  notmuch_shared_options [];
+void notmuch_exit_if_unmatched_db_revision (notmuch_database_t *notmuch);
+
 void notmuch_process_shared_options (const char* subcommand_name);
 int notmuch_minimal_options (const char* subcommand_name,
 			     int argc, char **argv);
diff --git a/notmuch-compact.c b/notmuch-compact.c
index 5be551d..3da4bfb 100644
--- a/notmuch-compact.c
+++ b/notmuch-compact.c
@@ -46,6 +46,10 @@ notmuch_compact_command (notmuch_config_t *config, int argc, char *argv[])
     if (opt_index < 0)
 	return EXIT_FAILURE;
 
+    if (notmuch_requested_db_revision)
+	fprintf (stderr, "Warning: ignoring --db-revision=%s\n",
+		 notmuch_requested_db_revision);
+
     notmuch_process_shared_options (argv[0]);
 
     if (! quiet)
diff --git a/notmuch-config.c b/notmuch-config.c
index 9348278..f1826e4 100644
--- a/notmuch-config.c
+++ b/notmuch-config.c
@@ -878,6 +878,10 @@ notmuch_config_command (notmuch_config_t *config, int argc, char *argv[])
     if (opt_index < 0)
 	return EXIT_FAILURE;
 
+    if (notmuch_requested_db_revision)
+	fprintf (stderr, "Warning: ignoring --db-revision=%s\n",
+		 notmuch_requested_db_revision);
+
     /* skip at least subcommand argument */
     argc-= opt_index;
     argv+= opt_index;
diff --git a/notmuch-count.c b/notmuch-count.c
index 7c61ccb..dda85b5 100644
--- a/notmuch-count.c
+++ b/notmuch-count.c
@@ -191,6 +191,8 @@ notmuch_count_command (notmuch_config_t *config, int argc, char *argv[])
 			       NOTMUCH_DATABASE_MODE_READ_ONLY, &notmuch))
 	return EXIT_FAILURE;
 
+    notmuch_exit_if_unmatched_db_revision (notmuch);
+
     query_str = query_string_from_args (config, argc-opt_index, argv+opt_index);
     if (query_str == NULL) {
 	fprintf (stderr, "Out of memory.\n");
diff --git a/notmuch-dump.c b/notmuch-dump.c
index fab22bd..9c0b82d 100644
--- a/notmuch-dump.c
+++ b/notmuch-dump.c
@@ -215,6 +215,8 @@ notmuch_dump_command (notmuch_config_t *config, int argc, char *argv[])
 			       NOTMUCH_DATABASE_MODE_READ_WRITE, &notmuch))
 	return EXIT_FAILURE;
 
+    notmuch_exit_if_unmatched_db_revision (notmuch);
+
     char *output_file_name = NULL;
     int opt_index;
 
diff --git a/notmuch-insert.c b/notmuch-insert.c
index c277d62..42d346f 100644
--- a/notmuch-insert.c
+++ b/notmuch-insert.c
@@ -536,6 +536,8 @@ notmuch_insert_command (notmuch_config_t *config, int argc, char *argv[])
 			       NOTMUCH_DATABASE_MODE_READ_WRITE, &notmuch))
 	return EXIT_FAILURE;
 
+    notmuch_exit_if_unmatched_db_revision (notmuch);
+
     /* Write the message to the Maildir new directory. */
     newpath = maildir_write_new (config, STDIN_FILENO, maildir);
     if (! newpath) {
diff --git a/notmuch-new.c b/notmuch-new.c
index 8ff1ade..2064a86 100644
--- a/notmuch-new.c
+++ b/notmuch-new.c
@@ -995,10 +995,11 @@ notmuch_new_command (notmuch_config_t *config, int argc, char *argv[])
 		fputs (status_string, stderr);
 		free (status_string);
 	    }
-
 	    return EXIT_FAILURE;
 	}
 
+	notmuch_exit_if_unmatched_db_revision (notmuch);
+
 	if (notmuch_database_needs_upgrade (notmuch)) {
 	    time_t now = time (NULL);
 	    struct tm *gm_time = gmtime (&now);
diff --git a/notmuch-reply.c b/notmuch-reply.c
index 4464741..52e0d9b 100644
--- a/notmuch-reply.c
+++ b/notmuch-reply.c
@@ -831,6 +831,8 @@ notmuch_reply_command (notmuch_config_t *config, int argc, char *argv[])
 			       NOTMUCH_DATABASE_MODE_READ_ONLY, &notmuch))
 	return EXIT_FAILURE;
 
+    notmuch_exit_if_unmatched_db_revision (notmuch);
+
     query = notmuch_query_create (notmuch, query_string);
     if (query == NULL) {
 	fprintf (stderr, "Out of memory\n");
diff --git a/notmuch-restore.c b/notmuch-restore.c
index 2a534dc..3e96aa8 100644
--- a/notmuch-restore.c
+++ b/notmuch-restore.c
@@ -165,6 +165,8 @@ notmuch_restore_command (notmuch_config_t *config, int argc, char *argv[])
     }
 
     notmuch_process_shared_options (argv[0]);
+    notmuch_exit_if_unmatched_db_revision (notmuch);
+
     name_for_error = input_file_name ? input_file_name : "stdin";
 
     if (! accumulate)
diff --git a/notmuch-search.c b/notmuch-search.c
index b89a17e..919837c 100644
--- a/notmuch-search.c
+++ b/notmuch-search.c
@@ -583,6 +583,8 @@ _notmuch_search_prepare (search_context_t *ctx, notmuch_config_t *config, int ar
 	return EXIT_FAILURE;
     }
 
+    notmuch_exit_if_unmatched_db_revision (ctx->notmuch);
+
     query_str = query_string_from_args (ctx->notmuch, argc, argv);
     if (query_str == NULL) {
 	fprintf (stderr, "Out of memory.\n");
diff --git a/notmuch-setup.c b/notmuch-setup.c
index 7dd5822..dffda81 100644
--- a/notmuch-setup.c
+++ b/notmuch-setup.c
@@ -148,6 +148,10 @@ notmuch_setup_command (notmuch_config_t *config,
     if (notmuch_minimal_options ("setup", argc, argv) < 0)
 	return EXIT_FAILURE;
 
+    if (notmuch_requested_db_revision)
+	fprintf (stderr, "Warning: ignoring --db-revision=%s\n",
+		 notmuch_requested_db_revision);
+
     if (notmuch_config_is_new (config))
 	welcome_message_pre_setup ();
 
diff --git a/notmuch-show.c b/notmuch-show.c
index b80933a..2535c8c 100644
--- a/notmuch-show.c
+++ b/notmuch-show.c
@@ -1213,6 +1213,8 @@ notmuch_show_command (notmuch_config_t *config, int argc, char *argv[])
 			       NOTMUCH_DATABASE_MODE_READ_ONLY, &notmuch))
 	return EXIT_FAILURE;
 
+    notmuch_exit_if_unmatched_db_revision (notmuch);
+
     query = notmuch_query_create (notmuch, query_string);
     if (query == NULL) {
 	fprintf (stderr, "Out of memory\n");
diff --git a/notmuch-tag.c b/notmuch-tag.c
index 38d99aa..22446fe 100644
--- a/notmuch-tag.c
+++ b/notmuch-tag.c
@@ -261,6 +261,8 @@ notmuch_tag_command (notmuch_config_t *config, int argc, char *argv[])
 			       NOTMUCH_DATABASE_MODE_READ_WRITE, &notmuch))
 	return EXIT_FAILURE;
 
+    notmuch_exit_if_unmatched_db_revision (notmuch);
+
     if (notmuch_config_get_maildir_synchronize_flags (config))
 	tag_flags |= TAG_FLAG_MAILDIR_SYNC;
 
diff --git a/notmuch.c b/notmuch.c
index c528dce..fab737c 100644
--- a/notmuch.c
+++ b/notmuch.c
@@ -47,10 +47,12 @@ static int
 _help_for (const char *topic);
 
 static notmuch_bool_t print_version = FALSE, print_help = FALSE;
+char *notmuch_requested_db_revision = NULL;
 
 const notmuch_opt_desc_t notmuch_shared_options [] = {
     { NOTMUCH_OPT_BOOLEAN, &print_version, "version", 'v', 0 },
     { NOTMUCH_OPT_BOOLEAN, &print_help, "help", 'h', 0 },
+    { NOTMUCH_OPT_STRING, &notmuch_requested_db_revision, "db-revision", 'd', 0 },
     {0, 0, 0, 0, 0}
 };
 
@@ -218,6 +220,22 @@ be supported in the future.\n", notmuch_format_version);
     }
 }
 
+void
+notmuch_exit_if_unmatched_db_revision (notmuch_database_t *notmuch)
+{
+    const char *uuid = NULL;
+
+    if (!notmuch_requested_db_revision)
+	return;
+    IGNORE_RESULT (notmuch_database_get_revision (notmuch, &uuid));
+
+    if (strcmp (notmuch_requested_db_revision, uuid) != 0){
+	fprintf (stderr, "Error: requested database revision %s does not match %s\n",
+		 notmuch_requested_db_revision, uuid);
+	exit (1);
+    }
+}
+
 static void
 exec_man (const char *page)
 {
diff --git a/test/T570-revision-tracking.sh b/test/T570-revision-tracking.sh
index 062fd59..bb09bf6 100755
--- a/test/T570-revision-tracking.sh
+++ b/test/T570-revision-tracking.sh
@@ -46,4 +46,30 @@ notmuch tag +a-random-tag-8743632 '*'
 after=$(notmuch count --output=modifications '*' | cut -f2)
 result=$(($before < $after))
 test_expect_equal 1 ${result}
+
+notmuch count --output=modifications '*' | cut -f1 > UUID
+
+test_expect_success 'search succeeds with correct db revision' \
+		    "notmuch search --db-revision=$(cat UUID) '*'"
+
+test_expect_success 'db-revision works as global option ' \
+		    "notmuch --db-revision=$(cat UUID) search '*'"
+
+test_expect_code 1 'db-revision works as global option II' \
+		    "notmuch --db-revision=this-is-no-uuid search '*'"
+
+test_expect_code 1 'search fails with incorrect db revision' \
+		 "notmuch search --db-revision=this-is-no-uuid '*'"
+
+test_expect_success 'show succeeds with correct db revision' \
+		    "notmuch show --db-revision=$(cat UUID) '*'"
+
+test_expect_code 1 'show fails with incorrect db revision' \
+		 "notmuch show --db-revision=this-is-no-uuid '*'"
+
+test_expect_success 'tag succeeds with correct db revision' \
+		    "notmuch tag --db-revision=$(cat UUID) +test '*'"
+
+test_expect_code 1 'tag fails with incorrect db revision' \
+		 "notmuch show --db-revision=this-is-no-uuid '*' +test2"
 test_done
diff --git a/test/random-corpus.c b/test/random-corpus.c
index b377eb4..542b453 100644
--- a/test/random-corpus.c
+++ b/test/random-corpus.c
@@ -119,6 +119,8 @@ const notmuch_opt_desc_t notmuch_shared_options[] = {
 	{ 0, 0, 0, 0, 0 }
 };
 
+char *notmuch_requested_db_revision = NULL;
+
 void
 notmuch_process_shared_options (unused (const char *dummy))
 {
-- 
2.1.4

Re: [PATCH 5/6] cli: add global option "--db-revision"

[Daniel Schoepe]

On Fri, 05 Jun 2015 19:28 +0200, David Bremner wrote:
> +    ``--db-revision=UUID``
> +       Enforce that the database revision is UUID. This is useful to
> +       detect rollover in modification counts on messages. You can
> +       find this UUID in the first column of output from
> +       ``notmuch count --output=modifications``

I'm not sure if the wording here is entirely clear. When reading this
the first time, to me it sounded like it might give you a view of the
database from when the revision was UUID. Maybe we can clarify here that
this will result in notmuch exiting if the revision is not matched.

Best regards,
Daniel

Re: [PATCH 5/6] cli: add global option "--db-revision"

[David Bremner]

Daniel Schoepe <daniel@schoepe.org> writes:

> On Fri, 05 Jun 2015 19:28 +0200, David Bremner wrote:
>> +    ``--db-revision=UUID``
>> +       Enforce that the database revision is UUID. This is useful to
>> +       detect rollover in modification counts on messages. You can
>> +       find this UUID in the first column of output from
>> +       ``notmuch count --output=modifications``
>
> I'm not sure if the wording here is entirely clear. When reading this
> the first time, to me it sounded like it might give you a view of the
> database from when the revision was UUID. Maybe we can clarify here that
> this will result in notmuch exiting if the revision is not matched.

Sure, no problem with expanding what "Enforce" means. Some improvment of
the names as discussed in a previous message might help here also.

d

[PATCH 6/6] lib: Add "lastmod:" queries for filtering by last modification

[David Bremner]

From: Austin Clements <amdragon@mit.edu>

The implementation is essentially the same as the date range search
prior to Jani's fancy date parser.
---
 doc/man7/notmuch-search-terms.rst |  8 ++++++++
 lib/database-private.h            |  1 +
 lib/database.cc                   |  4 ++++
 test/T570-revision-tracking.sh    | 18 ++++++++++++++++++
 4 files changed, 31 insertions(+)

diff --git a/doc/man7/notmuch-search-terms.rst b/doc/man7/notmuch-search-terms.rst
index 1d27ac1..d9f213e 100644
--- a/doc/man7/notmuch-search-terms.rst
+++ b/doc/man7/notmuch-search-terms.rst
@@ -54,6 +54,8 @@ indicate user-supplied values):
 
 -  date:<since>..<until>
 
+-  lastmod:<since>..<until>
+
 The **from:** prefix is used to match the name or address of the sender
 of an email message.
 
@@ -124,6 +126,12 @@ The time range can also be specified using timestamps with a syntax of:
 Each timestamp is a number representing the number of seconds since
 1970-01-01 00:00:00 UTC.
 
+The **lastmod:** prefix can be used to restrict the result by the
+database revision number of when messages were last modified (tags
+were added/removed or filenames changed).  This is usually used in
+conjunction with the **--db-revision** argument to **notmuch search**
+to find messages that have changed since an earlier query.
+
 Operators
 ---------
 
diff --git a/lib/database-private.h b/lib/database-private.h
index 4e93257..3fb10f7 100644
--- a/lib/database-private.h
+++ b/lib/database-private.h
@@ -176,6 +176,7 @@ struct _notmuch_database {
     Xapian::TermGenerator *term_gen;
     Xapian::ValueRangeProcessor *value_range_processor;
     Xapian::ValueRangeProcessor *date_range_processor;
+    Xapian::ValueRangeProcessor *last_mod_range_processor;
 };
 
 /* Prior to database version 3, features were implied by the database
diff --git a/lib/database.cc b/lib/database.cc
index ba8b8d9..a7daa44 100644
--- a/lib/database.cc
+++ b/lib/database.cc
@@ -986,6 +986,7 @@ notmuch_database_open_verbose (const char *path,
 	notmuch->term_gen->set_stemmer (Xapian::Stem ("english"));
 	notmuch->value_range_processor = new Xapian::NumberValueRangeProcessor (NOTMUCH_VALUE_TIMESTAMP);
 	notmuch->date_range_processor = new ParseTimeValueRangeProcessor (NOTMUCH_VALUE_TIMESTAMP);
+	notmuch->last_mod_range_processor = new Xapian::NumberValueRangeProcessor (NOTMUCH_VALUE_LAST_MOD, "lastmod:");
 
 	notmuch->query_parser->set_default_op (Xapian::Query::OP_AND);
 	notmuch->query_parser->set_database (*notmuch->xapian_db);
@@ -993,6 +994,7 @@ notmuch_database_open_verbose (const char *path,
 	notmuch->query_parser->set_stemming_strategy (Xapian::QueryParser::STEM_SOME);
 	notmuch->query_parser->add_valuerangeprocessor (notmuch->value_range_processor);
 	notmuch->query_parser->add_valuerangeprocessor (notmuch->date_range_processor);
+	notmuch->query_parser->add_valuerangeprocessor (notmuch->last_mod_range_processor);
 
 	for (i = 0; i < ARRAY_SIZE (BOOLEAN_PREFIX_EXTERNAL); i++) {
 	    prefix_t *prefix = &BOOLEAN_PREFIX_EXTERNAL[i];
@@ -1071,6 +1073,8 @@ notmuch_database_close (notmuch_database_t *notmuch)
     notmuch->value_range_processor = NULL;
     delete notmuch->date_range_processor;
     notmuch->date_range_processor = NULL;
+    delete notmuch->last_mod_range_processor;
+    notmuch->last_mod_range_processor = NULL;
 
     return status;
 }
diff --git a/test/T570-revision-tracking.sh b/test/T570-revision-tracking.sh
index bb09bf6..7ad8b2a 100755
--- a/test/T570-revision-tracking.sh
+++ b/test/T570-revision-tracking.sh
@@ -72,4 +72,22 @@ test_expect_success 'tag succeeds with correct db revision' \
 
 test_expect_code 1 'tag fails with incorrect db revision' \
 		 "notmuch show --db-revision=this-is-no-uuid '*' +test2"
+
+test_begin_subtest 'lastmod:0.. matches everything'
+total=$(notmuch count '*')
+modtotal=$(notmuch count lastmod:0..)
+test_expect_equal "$total" "$modtotal"
+
+test_begin_subtest 'lastmod:1000000.. matches nothing'
+modtotal=$(notmuch count lastmod:1000000..)
+test_expect_equal 0 "$modtotal"
+
+test_begin_subtest 'exclude one message using lastmod'
+lastmod=$(notmuch count --output=modifications '*' | cut -f2)
+total=$(notmuch count '*')
+notmuch tag +4EFC743A.3060609@april.org id:4EFC743A.3060609@april.org
+subtotal=$(notmuch count lastmod:..$lastmod)
+result=$(($subtotal == $total-1))
+test_expect_equal 1 "$result"
+
 test_done
-- 
2.1.4

Re: [PATCH 6/6] lib: Add "lastmod:" queries for filtering by last modification

[Daniel Schoepe]

On Fri, 05 Jun 2015 19:28 +0200, David Bremner wrote:
> +The **lastmod:** prefix can be used to restrict the result by the
> +database revision number of when messages were last modified (tags
> +were added/removed or filenames changed).  This is usually used in
> +conjunction with the **--db-revision** argument to **notmuch search**
> +to find messages that have changed since an earlier query.

It might be helpful to describe why --db-revision is needed here. At
first glance, just using `notmuch search lastmod:UUID..' looks good
enough to get all changed messages since UUID. Indeed, it's not entirely
clear to me under what conditions just using this query causes problems.
Moreover, it would be quite nice and intuitive if that query did the job
without having to constrain the database revision manually as well.

Best regards,
Daniel

Re: [PATCH 6/6] lib: Add "lastmod:" queries for filtering by last modification

[David Bremner]

Daniel Schoepe <daniel@schoepe.org> writes:

>
> It might be helpful to describe why --db-revision is needed here. At
> first glance, just using `notmuch search lastmod:UUID..' looks good
> enough to get all changed messages since UUID. Indeed, it's not entirely
> clear to me under what conditions just using this query causes problems.
> Moreover, it would be quite nice and intuitive if that query did the job
> without having to constrain the database revision manually as well.

There are two notions of revision here, and I agree the naming could be
clearer. I blame Austin ;).

--db-revision refers to the major-revision/epoch ; this is only changed
  e.g. when the the database is re-created from scratch.

lastmod:a..b  refers to integers that increase with every database operation.

Maybe --db-revision should be called --db-uuid ; this is at least
consistent with the library API.

In answer to why it's needed, the database changing UUID signals any
externally stored modification information (e.g. for incremental
backups) is now invalid.

d