revisision tracking patches round 3

revisision tracking patches round 3

[David Bremner]

Ironically there is some uncertainty about what revision this is
due to confusing cover letters for previous rounds.

This obsoletes

     id:1433525318-23756-1-git-send-email-david@tethera.net

I think I've taken into account most (all?) of Tomi and Daniel's
comments. There was a fair amount of rebasing silliness so I thought
I'd get some more eyes on this to check for typos and bikeshed the
naming of the --uuid option. 

[PATCH 1/5] lib: Add per-message last modification tracking

[David Bremner]

From: Austin Clements <amdragon@mit.edu>

This adds a new document value that stores the revision of the last
modification to message metadata, where the revision number increases
monotonically with each database commit.

An alternative would be to store the wall-clock time of the last
modification of each message.  In principle this is simpler and has
the advantage that any process can determine the current timestamp
without support from libnotmuch.  However, even assuming a computer's
clock never goes backward and ignoring clock skew in networked
environments, this has a fatal flaw.  Xapian uses (optimistic)
snapshot isolation, which means reads can be concurrent with writes.
Given this, consider the following time line with a write and two read
transactions:

   write  |-X-A--------------|
   read 1       |---B---|
   read 2                      |---|

The write transaction modifies message X and records the wall-clock
time of the modification at A.  The writer hangs around for a while
and later commits its change.  Read 1 is concurrent with the write, so
it doesn't see the change to X.  It does some query and records the
wall-clock time of its results at B.  Transaction read 2 later starts
after the write commits and queries for changes since wall-clock time
B (say the reads are performing an incremental backup).  Even though
read 1 could not see the change to X, read 2 is told (correctly) that
X has not changed since B, the time of the last read.  In fact, X
changed before wall-clock time A, but the change was not visible until
*after* wall-clock time B, so read 2 misses the change to X.

This is tricky to solve in full-blown snapshot isolation, but because
Xapian serializes writes, we can use a simple, monotonically
increasing database revision number.  Furthermore, maintaining this
revision number requires no more IO than a wall-clock time solution
because Xapian already maintains statistics on the upper (and lower)
bound of each value stream.
---
 lib/database-private.h | 16 +++++++++++++++-
 lib/database.cc        | 50 ++++++++++++++++++++++++++++++++++++++++++++++++--
 lib/message.cc         | 22 ++++++++++++++++++++++
 lib/notmuch-private.h  | 10 +++++++++-
 4 files changed, 94 insertions(+), 4 deletions(-)

diff --git a/lib/database-private.h b/lib/database-private.h
index 24243db..5c5a2bb 100644
--- a/lib/database-private.h
+++ b/lib/database-private.h
@@ -100,6 +100,12 @@ enum _notmuch_features {
      *
      * Introduced: version 3. */
     NOTMUCH_FEATURE_INDEXED_MIMETYPES = 1 << 5,
+
+    /* If set, messages store the revision number of the last
+     * modification in NOTMUCH_VALUE_LAST_MOD.
+     *
+     * Introduced: version 3. */
+    NOTMUCH_FEATURE_LAST_MOD = 1 << 6,
 };
 
 /* In C++, a named enum is its own type, so define bitwise operators
@@ -145,6 +151,8 @@ struct _notmuch_database {
 
     notmuch_database_mode_t mode;
     int atomic_nesting;
+    /* TRUE if changes have been made in this atomic section */
+    notmuch_bool_t atomic_dirty;
     Xapian::Database *xapian_db;
 
     /* Bit mask of features used by this database.  This is a
@@ -158,6 +166,11 @@ struct _notmuch_database {
      * next library call. May be NULL */
     char *status_string;
 
+    /* Highest committed revision number.  Modifications are recorded
+     * under a higher revision number, which can be generated with
+     * notmuch_database_new_revision. */
+    unsigned long revision;
+
     Xapian::QueryParser *query_parser;
     Xapian::TermGenerator *term_gen;
     Xapian::ValueRangeProcessor *value_range_processor;
@@ -179,7 +192,8 @@ struct _notmuch_database {
  * will have it). */
 #define NOTMUCH_FEATURES_CURRENT \
     (NOTMUCH_FEATURE_FILE_TERMS | NOTMUCH_FEATURE_DIRECTORY_DOCS | \
-     NOTMUCH_FEATURE_BOOL_FOLDER | NOTMUCH_FEATURE_GHOSTS)
+     NOTMUCH_FEATURE_BOOL_FOLDER | NOTMUCH_FEATURE_GHOSTS | \
+     NOTMUCH_FEATURE_LAST_MOD)
 
 /* Return the list of terms from the given iterator matching a prefix.
  * The prefix will be stripped from the strings in the returned list.
diff --git a/lib/database.cc b/lib/database.cc
index 6a15174..52e2e8f 100644
--- a/lib/database.cc
+++ b/lib/database.cc
@@ -101,6 +101,9 @@ typedef struct {
  *
  *	SUBJECT:	The value of the "Subject" header
  *
+ *	LAST_MOD:	The revision number as of the last tag or
+ *			filename change.
+ *
  * In addition, terms from the content of the message are added with
  * "from", "to", "attachment", and "subject" prefixes for use by the
  * user in searching. Similarly, terms from the path of the mail
@@ -310,6 +313,8 @@ static const struct {
      * them. */
     { NOTMUCH_FEATURE_INDEXED_MIMETYPES,
       "indexed MIME types", "w"},
+    { NOTMUCH_FEATURE_LAST_MOD,
+      "modification tracking", "w"},
 };
 
 const char *
@@ -737,6 +742,23 @@ _notmuch_database_ensure_writable (notmuch_database_t *notmuch)
     return NOTMUCH_STATUS_SUCCESS;
 }
 
+/* Allocate a revision number for the next change. */
+unsigned long
+_notmuch_database_new_revision (notmuch_database_t *notmuch)
+{
+    unsigned long new_revision = notmuch->revision + 1;
+
+    /* If we're in an atomic section, hold off on updating the
+     * committed revision number until we commit the atomic section.
+     */
+    if (notmuch->atomic_nesting)
+	notmuch->atomic_dirty = TRUE;
+    else
+	notmuch->revision = new_revision;
+
+    return new_revision;
+}
+
 /* Parse a database features string from the given database version.
  * Returns the feature bit set.
  *
@@ -904,6 +926,7 @@ notmuch_database_open_verbose (const char *path,
     notmuch->atomic_nesting = 0;
     try {
 	string last_thread_id;
+	string last_mod;
 
 	if (mode == NOTMUCH_DATABASE_MODE_READ_WRITE) {
 	    notmuch->xapian_db = new Xapian::WritableDatabase (xapian_path,
@@ -962,6 +985,14 @@ notmuch_database_open_verbose (const char *path,
 		INTERNAL_ERROR ("Malformed database last_thread_id: %s", str);
 	}
 
+	/* Get current highest revision number. */
+	last_mod = notmuch->xapian_db->get_value_upper_bound (
+	    NOTMUCH_VALUE_LAST_MOD);
+	if (last_mod.empty ())
+	    notmuch->revision = 0;
+	else
+	    notmuch->revision = Xapian::sortable_unserialise (last_mod);
+
 	notmuch->query_parser = new Xapian::QueryParser;
 	notmuch->term_gen = new Xapian::TermGenerator;
 	notmuch->term_gen->set_stemmer (Xapian::Stem ("english"));
@@ -1369,7 +1400,8 @@ notmuch_database_upgrade (notmuch_database_t *notmuch,
 
     /* Figure out how much total work we need to do. */
     if (new_features &
-	(NOTMUCH_FEATURE_FILE_TERMS | NOTMUCH_FEATURE_BOOL_FOLDER)) {
+	(NOTMUCH_FEATURE_FILE_TERMS | NOTMUCH_FEATURE_BOOL_FOLDER |
+	 NOTMUCH_FEATURE_LAST_MOD)) {
 	notmuch_query_t *query = notmuch_query_create (notmuch, "");
 	total += notmuch_query_count_messages (query);
 	notmuch_query_destroy (query);
@@ -1396,7 +1428,8 @@ notmuch_database_upgrade (notmuch_database_t *notmuch,
 
     /* Perform per-message upgrades. */
     if (new_features &
-	(NOTMUCH_FEATURE_FILE_TERMS | NOTMUCH_FEATURE_BOOL_FOLDER)) {
+	(NOTMUCH_FEATURE_FILE_TERMS | NOTMUCH_FEATURE_BOOL_FOLDER |
+	 NOTMUCH_FEATURE_LAST_MOD)) {
 	notmuch_query_t *query = notmuch_query_create (notmuch, "");
 	notmuch_messages_t *messages;
 	notmuch_message_t *message;
@@ -1433,6 +1466,14 @@ notmuch_database_upgrade (notmuch_database_t *notmuch,
 	    if (new_features & NOTMUCH_FEATURE_BOOL_FOLDER)
 		_notmuch_message_upgrade_folder (message);
 
+	    /* Prior to NOTMUCH_FEATURE_LAST_MOD, messages did not
+	     * track modification revisions.  Give all messages the
+	     * next available revision; since we just started tracking
+	     * revisions for this database, that will be 1.
+	     */
+	    if (new_features & NOTMUCH_FEATURE_LAST_MOD)
+		_notmuch_message_upgrade_last_mod (message);
+
 	    _notmuch_message_sync (message);
 
 	    notmuch_message_destroy (message);
@@ -1615,6 +1656,11 @@ notmuch_database_end_atomic (notmuch_database_t *notmuch)
 	return NOTMUCH_STATUS_XAPIAN_EXCEPTION;
     }
 
+    if (notmuch->atomic_dirty) {
+	++notmuch->revision;
+	notmuch->atomic_dirty = FALSE;
+    }
+
 DONE:
     notmuch->atomic_nesting--;
     return NOTMUCH_STATUS_SUCCESS;
diff --git a/lib/message.cc b/lib/message.cc
index 1ddce3c..26b5e76 100644
--- a/lib/message.cc
+++ b/lib/message.cc
@@ -998,6 +998,16 @@ _notmuch_message_set_header_values (notmuch_message_t *message,
     message->modified = TRUE;
 }
 
+/* Upgrade a message to support NOTMUCH_FEATURE_LAST_MOD.  The caller
+ * must call _notmuch_message_sync. */
+void
+_notmuch_message_upgrade_last_mod (notmuch_message_t *message)
+{
+    /* _notmuch_message_sync will update the last modification
+     * revision; we just have to ask it to. */
+    message->modified = TRUE;
+}
+
 /* Synchronize changes made to message->doc out into the database. */
 void
 _notmuch_message_sync (notmuch_message_t *message)
@@ -1010,6 +1020,18 @@ _notmuch_message_sync (notmuch_message_t *message)
     if (! message->modified)
 	return;
 
+    /* Update the last modification of this message. */
+    if (message->notmuch->features & NOTMUCH_FEATURE_LAST_MOD)
+	/* sortable_serialise gives a reasonably compact encoding,
+	 * which directly translates to reduced IO when scanning the
+	 * value stream.  Since it's built for doubles, we only get 53
+	 * effective bits, but that's still enough for the database to
+	 * last a few centuries at 1 million revisions per second. */
+	message->doc.add_value (NOTMUCH_VALUE_LAST_MOD,
+				Xapian::sortable_serialise (
+				    _notmuch_database_new_revision (
+					message->notmuch)));
+
     db = static_cast <Xapian::WritableDatabase *> (message->notmuch->xapian_db);
     db->replace_document (message->doc_id, message->doc);
     message->modified = FALSE;
diff --git a/lib/notmuch-private.h b/lib/notmuch-private.h
index cc9ce12..f52b4e4 100644
--- a/lib/notmuch-private.h
+++ b/lib/notmuch-private.h
@@ -107,7 +107,8 @@ typedef enum {
     NOTMUCH_VALUE_TIMESTAMP = 0,
     NOTMUCH_VALUE_MESSAGE_ID,
     NOTMUCH_VALUE_FROM,
-    NOTMUCH_VALUE_SUBJECT
+    NOTMUCH_VALUE_SUBJECT,
+    NOTMUCH_VALUE_LAST_MOD,
 } notmuch_value_t;
 
 /* Xapian (with flint backend) complains if we provide a term longer
@@ -194,6 +195,9 @@ void
 _notmuch_database_log (notmuch_database_t *notmuch,
 		       const char *format, ...);
 
+unsigned long
+_notmuch_database_new_revision (notmuch_database_t *notmuch);
+
 const char *
 _notmuch_database_relative_path (notmuch_database_t *notmuch,
 				 const char *path);
@@ -305,6 +309,10 @@ _notmuch_message_set_header_values (notmuch_message_t *message,
 				    const char *date,
 				    const char *from,
 				    const char *subject);
+
+void
+_notmuch_message_upgrade_last_mod (notmuch_message_t *message);
+
 void
 _notmuch_message_sync (notmuch_message_t *message);
 
-- 
2.1.4

[PATCH 2/5] lib: API to retrieve database revision and UUID

[David Bremner]

From: Austin Clements <aclements@csail.mit.edu>

This exposes the committed database revision to library users along
with a UUID that can be used to detect when revision numbers are no
longer comparable (e.g., because the database has been replaced).
---
 lib/database-private.h         |  1 +
 lib/database.cc                | 11 +++++++++++
 lib/notmuch.h                  | 18 ++++++++++++++++++
 test/T570-revision-tracking.sh | 37 +++++++++++++++++++++++++++++++++++++
 test/test-lib.sh               |  5 +++++
 5 files changed, 72 insertions(+)
 create mode 100755 test/T570-revision-tracking.sh

diff --git a/lib/database-private.h b/lib/database-private.h
index 5c5a2bb..4e93257 100644
--- a/lib/database-private.h
+++ b/lib/database-private.h
@@ -170,6 +170,7 @@ struct _notmuch_database {
      * under a higher revision number, which can be generated with
      * notmuch_database_new_revision. */
     unsigned long revision;
+    const char *uuid;
 
     Xapian::QueryParser *query_parser;
     Xapian::TermGenerator *term_gen;
diff --git a/lib/database.cc b/lib/database.cc
index 52e2e8f..fc78769 100644
--- a/lib/database.cc
+++ b/lib/database.cc
@@ -992,6 +992,8 @@ notmuch_database_open_verbose (const char *path,
 	    notmuch->revision = 0;
 	else
 	    notmuch->revision = Xapian::sortable_unserialise (last_mod);
+	notmuch->uuid = talloc_strdup (
+	    notmuch, notmuch->xapian_db->get_uuid ().c_str ());
 
 	notmuch->query_parser = new Xapian::QueryParser;
 	notmuch->term_gen = new Xapian::TermGenerator;
@@ -1666,6 +1668,15 @@ DONE:
     return NOTMUCH_STATUS_SUCCESS;
 }
 
+unsigned long
+notmuch_database_get_revision (notmuch_database_t *notmuch,
+				const char **uuid)
+{
+    if (uuid)
+	*uuid = notmuch->uuid;
+    return notmuch->revision;
+}
+
 /* We allow the user to use arbitrarily long paths for directories. But
  * we have a term-length limit. So if we exceed that, we'll use the
  * SHA-1 of the path for the database term.
diff --git a/lib/notmuch.h b/lib/notmuch.h
index 421c19d..a7ac012 100644
--- a/lib/notmuch.h
+++ b/lib/notmuch.h
@@ -468,6 +468,24 @@ notmuch_status_t
 notmuch_database_end_atomic (notmuch_database_t *notmuch);
 
 /**
+ * Return the committed database revision and UUID.
+ *
+ * The database revision number increases monotonically with each
+ * commit to the database.  Hence, all messages and message changes
+ * committed to the database (that is, visible to readers) have a last
+ * modification revision <= the committed database revision.  Any
+ * messages committed in the future will be assigned a modification
+ * revision > the committed database revision.
+ *
+ * The UUID is a NUL-terminated opaque string that uniquely identifies
+ * this database.  Two revision numbers are only comparable if they
+ * have the same database UUID.
+ */
+unsigned long
+notmuch_database_get_revision (notmuch_database_t *notmuch,
+				const char **uuid);
+
+/**
  * Retrieve a directory object from the database for 'path'.
  *
  * Here, 'path' should be a path relative to the path of 'database'
diff --git a/test/T570-revision-tracking.sh b/test/T570-revision-tracking.sh
new file mode 100755
index 0000000..e0a5703
--- /dev/null
+++ b/test/T570-revision-tracking.sh
@@ -0,0 +1,37 @@
+#!/usr/bin/env bash
+test_description="database revision tracking"
+
+. ./test-lib.sh || exit 1
+
+add_email_corpus
+
+test_begin_subtest "notmuch_database_get_revision"
+test_C ${MAIL_DIR} <<'EOF'
+#include <stdio.h>
+#include <string.h>
+#include <notmuch.h>
+int main (int argc, char** argv)
+{
+   notmuch_database_t *db;
+   notmuch_status_t stat;
+   unsigned long revision;
+   const char *uuid;
+
+   unsigned long rev;
+
+   stat = notmuch_database_open (argv[1], NOTMUCH_DATABASE_MODE_READ_ONLY, &db);
+   if (stat)
+       fputs ("open failed\n", stderr);
+   revision = notmuch_database_get_revision (db, &uuid);
+   printf("%s\t%lu\n", uuid, revision);
+}
+EOF
+notmuch_uuid_sanitize < OUTPUT > CLEAN
+cat <<'EOF' >EXPECTED
+== stdout ==
+UUID	53
+== stderr ==
+EOF
+test_expect_equal_file EXPECTED CLEAN
+
+test_done
diff --git a/test/test-lib.sh b/test/test-lib.sh
index 0bf7163..126911f 100644
--- a/test/test-lib.sh
+++ b/test/test-lib.sh
@@ -720,6 +720,11 @@ notmuch_date_sanitize ()
     sed \
 	-e 's/^Date: Fri, 05 Jan 2001 .*0000/Date: GENERATED_DATE/'
 }
+
+notmuch_uuid_sanitize ()
+{
+    sed 's/[0-9a-f]\{8\}-[0-9a-f]\{4\}-[0-9a-f]\{4\}-[0-9a-f]\{4\}-[0-9a-f]\{12\}/UUID/g'
+}
 # End of notmuch helper functions
 
 # Use test_set_prereq to tell that a particular prerequisite is available.
-- 
2.1.4

[PATCH 3/5] cli/count: add --output=modifications

[David Bremner]

We need some way to extract the uuid/revision of the database, and
count seems like the least bad choice of current commands.
The (perhaps weak) argument for count over search is that count
already reports statistics about the entire database.
---
 notmuch-count.c                | 18 +++++++++++++++++-
 test/T570-revision-tracking.sh | 12 ++++++++++++
 2 files changed, 29 insertions(+), 1 deletion(-)

diff --git a/notmuch-count.c b/notmuch-count.c
index 57a88a8..7c61ccb 100644
--- a/notmuch-count.c
+++ b/notmuch-count.c
@@ -25,6 +25,7 @@ enum {
     OUTPUT_THREADS,
     OUTPUT_MESSAGES,
     OUTPUT_FILES,
+    OUTPUT_LASTMOD,
 };
 
 /* The following is to allow future options to be added more easily */
@@ -71,6 +72,9 @@ print_count (notmuch_database_t *notmuch, const char *query_str,
 {
     notmuch_query_t *query;
     size_t i;
+    unsigned long revision;
+    const char *uuid;
+    int ret = 0;
 
     query = notmuch_query_create (notmuch, query_str);
     if (query == NULL) {
@@ -91,11 +95,22 @@ print_count (notmuch_database_t *notmuch, const char *query_str,
     case OUTPUT_FILES:
 	printf ("%u\n", count_files (query));
 	break;
+    case OUTPUT_LASTMOD:
+	if (strcmp (notmuch_query_get_query_string (query), "*") != 0) {
+	    fprintf (stderr, "Error: Only '*' is currently supported "
+		     " with output=modifications\n");
+	    ret = 1;
+	    goto DONE;
+	}
+
+	revision = notmuch_database_get_revision (notmuch, &uuid);
+	printf ("%s\t%lu\n", uuid, revision);
     }
 
+ DONE:
     notmuch_query_destroy (query);
 
-    return 0;
+    return ret;
 }
 
 static int
@@ -139,6 +154,7 @@ notmuch_count_command (notmuch_config_t *config, int argc, char *argv[])
 	  (notmuch_keyword_t []){ { "threads", OUTPUT_THREADS },
 				  { "messages", OUTPUT_MESSAGES },
 				  { "files", OUTPUT_FILES },
+				  { "modifications", OUTPUT_LASTMOD },
 				  { 0, 0 } } },
 	{ NOTMUCH_OPT_KEYWORD, &exclude, "exclude", 'x',
 	  (notmuch_keyword_t []){ { "true", EXCLUDE_TRUE },
diff --git a/test/T570-revision-tracking.sh b/test/T570-revision-tracking.sh
index e0a5703..008c5d0 100755
--- a/test/T570-revision-tracking.sh
+++ b/test/T570-revision-tracking.sh
@@ -34,4 +34,16 @@ UUID	53
 EOF
 test_expect_equal_file EXPECTED CLEAN
 
+grep '^[0-9a-f]' OUTPUT > INITIAL_OUTPUT
+
+test_begin_subtest "output of count matches test code"
+notmuch count --output=modifications '*' > OUTPUT
+test_expect_equal_file INITIAL_OUTPUT OUTPUT
+
+test_begin_subtest "modification count increases"
+before=$(notmuch count --output=modifications '*' | cut -f2)
+notmuch tag +a-random-tag-8743632 '*'
+after=$(notmuch count --output=modifications '*' | cut -f2)
+result=$(($before < $after))
+test_expect_equal 1 ${result}
 test_done
-- 
2.1.4

[PATCH 4/5] cli: add global option "--uuid"

[David Bremner]

The function notmuch_exit_if_unmatched_db_uuid is split from
notmuch_process_shared_options because it needs an open notmuch
database.
---
 doc/man1/notmuch.rst           | 12 ++++++++++--
 notmuch-client.h               |  4 ++++
 notmuch-compact.c              |  4 ++++
 notmuch-config.c               |  4 ++++
 notmuch-count.c                |  2 ++
 notmuch-dump.c                 |  2 ++
 notmuch-insert.c               |  2 ++
 notmuch-new.c                  |  3 ++-
 notmuch-reply.c                |  2 ++
 notmuch-restore.c              |  2 ++
 notmuch-search.c               |  2 ++
 notmuch-setup.c                |  4 ++++
 notmuch-show.c                 |  2 ++
 notmuch-tag.c                  |  2 ++
 notmuch.c                      | 18 ++++++++++++++++++
 test/T570-revision-tracking.sh | 27 +++++++++++++++++++++++++++
 test/random-corpus.c           |  2 ++
 17 files changed, 91 insertions(+), 3 deletions(-)

diff --git a/doc/man1/notmuch.rst b/doc/man1/notmuch.rst
index 0401c91..0b89544 100644
--- a/doc/man1/notmuch.rst
+++ b/doc/man1/notmuch.rst
@@ -51,9 +51,17 @@ Supported global options for ``notmuch`` include
 	Specify the configuration file to use. This overrides any
 	configuration file specified by ${NOTMUCH\_CONFIG}.
 
+    ``--uuid=HEX``
+       Enforce that the database UUID (a unique identifier which
+       persists until e.g. the database is compacted)
+       is HEX; exit with an error if it is not. This is useful to
+       detect rollover in modification counts on messages. You can
+       find this UUID in the first column of output from
+       ``notmuch count --output=modifications``
+
 All global options except ``--config`` can also be specified after the
-command. For example, ``notmuch subcommand --version`` is equivalent to
-``notmuch --version subcommand``.
+command. For example, ``notmuch subcommand --uuid=HEX`` is
+equivalent to ``notmuch --uuid=HEX subcommand``.
 
 COMMANDS
 ========
diff --git a/notmuch-client.h b/notmuch-client.h
index 78680aa..4a4f86c 100644
--- a/notmuch-client.h
+++ b/notmuch-client.h
@@ -466,7 +466,11 @@ notmuch_database_dump (notmuch_database_t *notmuch,
 		       notmuch_bool_t gzip_output);
 
 #include "command-line-arguments.h"
+
+extern char *notmuch_requested_db_uuid;
 extern const notmuch_opt_desc_t  notmuch_shared_options [];
+void notmuch_exit_if_unmatched_db_uuid (notmuch_database_t *notmuch);
+
 void notmuch_process_shared_options (const char* subcommand_name);
 int notmuch_minimal_options (const char* subcommand_name,
 			     int argc, char **argv);
diff --git a/notmuch-compact.c b/notmuch-compact.c
index 5be551d..61fccd8 100644
--- a/notmuch-compact.c
+++ b/notmuch-compact.c
@@ -46,6 +46,10 @@ notmuch_compact_command (notmuch_config_t *config, int argc, char *argv[])
     if (opt_index < 0)
 	return EXIT_FAILURE;
 
+    if (notmuch_requested_db_uuid)
+	fprintf (stderr, "Warning: ignoring --uuid=%s\n",
+		 notmuch_requested_db_uuid);
+
     notmuch_process_shared_options (argv[0]);
 
     if (! quiet)
diff --git a/notmuch-config.c b/notmuch-config.c
index 9348278..d252bb2 100644
--- a/notmuch-config.c
+++ b/notmuch-config.c
@@ -878,6 +878,10 @@ notmuch_config_command (notmuch_config_t *config, int argc, char *argv[])
     if (opt_index < 0)
 	return EXIT_FAILURE;
 
+    if (notmuch_requested_db_uuid)
+	fprintf (stderr, "Warning: ignoring --uuid=%s\n",
+		 notmuch_requested_db_uuid);
+
     /* skip at least subcommand argument */
     argc-= opt_index;
     argv+= opt_index;
diff --git a/notmuch-count.c b/notmuch-count.c
index 7c61ccb..091e376 100644
--- a/notmuch-count.c
+++ b/notmuch-count.c
@@ -191,6 +191,8 @@ notmuch_count_command (notmuch_config_t *config, int argc, char *argv[])
 			       NOTMUCH_DATABASE_MODE_READ_ONLY, &notmuch))
 	return EXIT_FAILURE;
 
+    notmuch_exit_if_unmatched_db_uuid (notmuch);
+
     query_str = query_string_from_args (config, argc-opt_index, argv+opt_index);
     if (query_str == NULL) {
 	fprintf (stderr, "Out of memory.\n");
diff --git a/notmuch-dump.c b/notmuch-dump.c
index fab22bd..24fc2f2 100644
--- a/notmuch-dump.c
+++ b/notmuch-dump.c
@@ -215,6 +215,8 @@ notmuch_dump_command (notmuch_config_t *config, int argc, char *argv[])
 			       NOTMUCH_DATABASE_MODE_READ_WRITE, &notmuch))
 	return EXIT_FAILURE;
 
+    notmuch_exit_if_unmatched_db_uuid (notmuch);
+
     char *output_file_name = NULL;
     int opt_index;
 
diff --git a/notmuch-insert.c b/notmuch-insert.c
index c277d62..5205c17 100644
--- a/notmuch-insert.c
+++ b/notmuch-insert.c
@@ -536,6 +536,8 @@ notmuch_insert_command (notmuch_config_t *config, int argc, char *argv[])
 			       NOTMUCH_DATABASE_MODE_READ_WRITE, &notmuch))
 	return EXIT_FAILURE;
 
+    notmuch_exit_if_unmatched_db_uuid (notmuch);
+
     /* Write the message to the Maildir new directory. */
     newpath = maildir_write_new (config, STDIN_FILENO, maildir);
     if (! newpath) {
diff --git a/notmuch-new.c b/notmuch-new.c
index ee786a3..514e06a 100644
--- a/notmuch-new.c
+++ b/notmuch-new.c
@@ -1009,10 +1009,11 @@ notmuch_new_command (notmuch_config_t *config, int argc, char *argv[])
 		fputs (status_string, stderr);
 		free (status_string);
 	    }
-
 	    return EXIT_FAILURE;
 	}
 
+	notmuch_exit_if_unmatched_db_uuid (notmuch);
+
 	if (notmuch_database_needs_upgrade (notmuch)) {
 	    time_t now = time (NULL);
 	    struct tm *gm_time = gmtime (&now);
diff --git a/notmuch-reply.c b/notmuch-reply.c
index 4464741..7c5c28f 100644
--- a/notmuch-reply.c
+++ b/notmuch-reply.c
@@ -831,6 +831,8 @@ notmuch_reply_command (notmuch_config_t *config, int argc, char *argv[])
 			       NOTMUCH_DATABASE_MODE_READ_ONLY, &notmuch))
 	return EXIT_FAILURE;
 
+    notmuch_exit_if_unmatched_db_uuid (notmuch);
+
     query = notmuch_query_create (notmuch, query_string);
     if (query == NULL) {
 	fprintf (stderr, "Out of memory\n");
diff --git a/notmuch-restore.c b/notmuch-restore.c
index 2a534dc..9abc64f 100644
--- a/notmuch-restore.c
+++ b/notmuch-restore.c
@@ -165,6 +165,8 @@ notmuch_restore_command (notmuch_config_t *config, int argc, char *argv[])
     }
 
     notmuch_process_shared_options (argv[0]);
+    notmuch_exit_if_unmatched_db_uuid (notmuch);
+
     name_for_error = input_file_name ? input_file_name : "stdin";
 
     if (! accumulate)
diff --git a/notmuch-search.c b/notmuch-search.c
index b89a17e..3076c3f 100644
--- a/notmuch-search.c
+++ b/notmuch-search.c
@@ -583,6 +583,8 @@ _notmuch_search_prepare (search_context_t *ctx, notmuch_config_t *config, int ar
 	return EXIT_FAILURE;
     }
 
+    notmuch_exit_if_unmatched_db_uuid (ctx->notmuch);
+
     query_str = query_string_from_args (ctx->notmuch, argc, argv);
     if (query_str == NULL) {
 	fprintf (stderr, "Out of memory.\n");
diff --git a/notmuch-setup.c b/notmuch-setup.c
index 7dd5822..9aaf928 100644
--- a/notmuch-setup.c
+++ b/notmuch-setup.c
@@ -148,6 +148,10 @@ notmuch_setup_command (notmuch_config_t *config,
     if (notmuch_minimal_options ("setup", argc, argv) < 0)
 	return EXIT_FAILURE;
 
+    if (notmuch_requested_db_uuid)
+	fprintf (stderr, "Warning: ignoring --uuid=%s\n",
+		 notmuch_requested_db_uuid);
+
     if (notmuch_config_is_new (config))
 	welcome_message_pre_setup ();
 
diff --git a/notmuch-show.c b/notmuch-show.c
index b80933a..6ef3308 100644
--- a/notmuch-show.c
+++ b/notmuch-show.c
@@ -1213,6 +1213,8 @@ notmuch_show_command (notmuch_config_t *config, int argc, char *argv[])
 			       NOTMUCH_DATABASE_MODE_READ_ONLY, &notmuch))
 	return EXIT_FAILURE;
 
+    notmuch_exit_if_unmatched_db_uuid (notmuch);
+
     query = notmuch_query_create (notmuch, query_string);
     if (query == NULL) {
 	fprintf (stderr, "Out of memory\n");
diff --git a/notmuch-tag.c b/notmuch-tag.c
index 38d99aa..7ae98f6 100644
--- a/notmuch-tag.c
+++ b/notmuch-tag.c
@@ -261,6 +261,8 @@ notmuch_tag_command (notmuch_config_t *config, int argc, char *argv[])
 			       NOTMUCH_DATABASE_MODE_READ_WRITE, &notmuch))
 	return EXIT_FAILURE;
 
+    notmuch_exit_if_unmatched_db_uuid (notmuch);
+
     if (notmuch_config_get_maildir_synchronize_flags (config))
 	tag_flags |= TAG_FLAG_MAILDIR_SYNC;
 
diff --git a/notmuch.c b/notmuch.c
index 9580c3f..ce6c575 100644
--- a/notmuch.c
+++ b/notmuch.c
@@ -47,10 +47,12 @@ static int
 _help_for (const char *topic);
 
 static notmuch_bool_t print_version = FALSE, print_help = FALSE;
+char *notmuch_requested_db_uuid = NULL;
 
 const notmuch_opt_desc_t notmuch_shared_options [] = {
     { NOTMUCH_OPT_BOOLEAN, &print_version, "version", 'v', 0 },
     { NOTMUCH_OPT_BOOLEAN, &print_help, "help", 'h', 0 },
+    { NOTMUCH_OPT_STRING, &notmuch_requested_db_uuid, "uuid", 'u', 0 },
     {0, 0, 0, 0, 0}
 };
 
@@ -218,6 +220,22 @@ be supported in the future.\n", notmuch_format_version);
     }
 }
 
+void
+notmuch_exit_if_unmatched_db_uuid (notmuch_database_t *notmuch)
+{
+    const char *uuid = NULL;
+
+    if (!notmuch_requested_db_uuid)
+	return;
+    IGNORE_RESULT (notmuch_database_get_revision (notmuch, &uuid));
+
+    if (strcmp (notmuch_requested_db_uuid, uuid) != 0){
+	fprintf (stderr, "Error: requested database revision %s does not match %s\n",
+		 notmuch_requested_db_uuid, uuid);
+	exit (1);
+    }
+}
+
 static void
 exec_man (const char *page)
 {
diff --git a/test/T570-revision-tracking.sh b/test/T570-revision-tracking.sh
index 008c5d0..74e3ba9 100755
--- a/test/T570-revision-tracking.sh
+++ b/test/T570-revision-tracking.sh
@@ -46,4 +46,31 @@ notmuch tag +a-random-tag-8743632 '*'
 after=$(notmuch count --output=modifications '*' | cut -f2)
 result=$(($before < $after))
 test_expect_equal 1 ${result}
+
+notmuch count --output=modifications '*' | cut -f1 > UUID
+
+test_expect_success 'search succeeds with correct uuid' \
+		    "notmuch search --uuid=$(cat UUID) '*'"
+
+test_expect_success 'uuid works as global option ' \
+		    "notmuch --uuid=$(cat UUID) search '*'"
+
+test_expect_code 1 'uuid works as global option II' \
+		    "notmuch --uuid=this-is-no-uuid search '*'"
+
+test_expect_code 1 'search fails with incorrect uuid' \
+		 "notmuch search --uuid=this-is-no-uuid '*'"
+
+test_expect_success 'show succeeds with correct uuid' \
+		    "notmuch show --uuid=$(cat UUID) '*'"
+
+test_expect_code 1 'show fails with incorrect uuid' \
+		 "notmuch show --uuid=this-is-no-uuid '*'"
+
+test_expect_success 'tag succeeds with correct uuid' \
+		    "notmuch tag --uuid=$(cat UUID) +test '*'"
+
+test_expect_code 1 'tag fails with incorrect uuid' \
+		 "notmuch tag --uuid=this-is-no-uuid '*' +test2"
+
 test_done
diff --git a/test/random-corpus.c b/test/random-corpus.c
index b377eb4..d74271d 100644
--- a/test/random-corpus.c
+++ b/test/random-corpus.c
@@ -119,6 +119,8 @@ const notmuch_opt_desc_t notmuch_shared_options[] = {
 	{ 0, 0, 0, 0, 0 }
 };
 
+char *notmuch_requested_db_uuid = NULL;
+
 void
 notmuch_process_shared_options (unused (const char *dummy))
 {
-- 
2.1.4

[PATCH 5/5] lib: Add "lastmod:" queries for filtering by last modification

[David Bremner]

From: Austin Clements <amdragon@mit.edu>

The implementation is essentially the same as the date range search
prior to Jani's fancy date parser.
---
 doc/man7/notmuch-search-terms.rst |  8 ++++++++
 lib/database-private.h            |  1 +
 lib/database.cc                   |  4 ++++
 test/T570-revision-tracking.sh    | 17 +++++++++++++++++
 4 files changed, 30 insertions(+)

diff --git a/doc/man7/notmuch-search-terms.rst b/doc/man7/notmuch-search-terms.rst
index 1d27ac1..e71a525 100644
--- a/doc/man7/notmuch-search-terms.rst
+++ b/doc/man7/notmuch-search-terms.rst
@@ -54,6 +54,8 @@ indicate user-supplied values):
 
 -  date:<since>..<until>
 
+-  lastmod:<since>..<until>
+
 The **from:** prefix is used to match the name or address of the sender
 of an email message.
 
@@ -124,6 +126,12 @@ The time range can also be specified using timestamps with a syntax of:
 Each timestamp is a number representing the number of seconds since
 1970-01-01 00:00:00 UTC.
 
+The **lastmod:** prefix can be used to restrict the result by the
+database revision number of when messages were last modified (tags
+were added/removed or filenames changed).  This is usually used in
+conjunction with the **--uuid** argument to **notmuch search**
+to find messages that have changed since an earlier query.
+
 Operators
 ---------
 
diff --git a/lib/database-private.h b/lib/database-private.h
index 4e93257..3fb10f7 100644
--- a/lib/database-private.h
+++ b/lib/database-private.h
@@ -176,6 +176,7 @@ struct _notmuch_database {
     Xapian::TermGenerator *term_gen;
     Xapian::ValueRangeProcessor *value_range_processor;
     Xapian::ValueRangeProcessor *date_range_processor;
+    Xapian::ValueRangeProcessor *last_mod_range_processor;
 };
 
 /* Prior to database version 3, features were implied by the database
diff --git a/lib/database.cc b/lib/database.cc
index fc78769..bab3334 100644
--- a/lib/database.cc
+++ b/lib/database.cc
@@ -1000,6 +1000,7 @@ notmuch_database_open_verbose (const char *path,
 	notmuch->term_gen->set_stemmer (Xapian::Stem ("english"));
 	notmuch->value_range_processor = new Xapian::NumberValueRangeProcessor (NOTMUCH_VALUE_TIMESTAMP);
 	notmuch->date_range_processor = new ParseTimeValueRangeProcessor (NOTMUCH_VALUE_TIMESTAMP);
+	notmuch->last_mod_range_processor = new Xapian::NumberValueRangeProcessor (NOTMUCH_VALUE_LAST_MOD, "lastmod:");
 
 	notmuch->query_parser->set_default_op (Xapian::Query::OP_AND);
 	notmuch->query_parser->set_database (*notmuch->xapian_db);
@@ -1007,6 +1008,7 @@ notmuch_database_open_verbose (const char *path,
 	notmuch->query_parser->set_stemming_strategy (Xapian::QueryParser::STEM_SOME);
 	notmuch->query_parser->add_valuerangeprocessor (notmuch->value_range_processor);
 	notmuch->query_parser->add_valuerangeprocessor (notmuch->date_range_processor);
+	notmuch->query_parser->add_valuerangeprocessor (notmuch->last_mod_range_processor);
 
 	for (i = 0; i < ARRAY_SIZE (BOOLEAN_PREFIX_EXTERNAL); i++) {
 	    prefix_t *prefix = &BOOLEAN_PREFIX_EXTERNAL[i];
@@ -1085,6 +1087,8 @@ notmuch_database_close (notmuch_database_t *notmuch)
     notmuch->value_range_processor = NULL;
     delete notmuch->date_range_processor;
     notmuch->date_range_processor = NULL;
+    delete notmuch->last_mod_range_processor;
+    notmuch->last_mod_range_processor = NULL;
 
     return status;
 }
diff --git a/test/T570-revision-tracking.sh b/test/T570-revision-tracking.sh
index 74e3ba9..55337e6 100755
--- a/test/T570-revision-tracking.sh
+++ b/test/T570-revision-tracking.sh
@@ -73,4 +73,21 @@ test_expect_success 'tag succeeds with correct uuid' \
 test_expect_code 1 'tag fails with incorrect uuid' \
 		 "notmuch tag --uuid=this-is-no-uuid '*' +test2"
 
+test_begin_subtest 'lastmod:0.. matches everything'
+total=$(notmuch count '*')
+modtotal=$(notmuch count lastmod:0..)
+test_expect_equal "$total" "$modtotal"
+
+test_begin_subtest 'lastmod:1000000.. matches nothing'
+modtotal=$(notmuch count lastmod:1000000..)
+test_expect_equal 0 "$modtotal"
+
+test_begin_subtest 'exclude one message using lastmod'
+lastmod=$(notmuch count --output=modifications '*' | cut -f2)
+total=$(notmuch count '*')
+notmuch tag +4EFC743A.3060609@april.org id:4EFC743A.3060609@april.org
+subtotal=$(notmuch count lastmod:..$lastmod)
+result=$(($subtotal == $total-1))
+test_expect_equal 1 "$result"
+
 test_done
-- 
2.1.4

Re: [PATCH 4/5] cli: add global option "--uuid"

[Daniel Schoepe]

On Sun, 09 Aug 2015 11:24+0200, David Bremner wrote:
> +    ``--uuid=HEX``
> +       Enforce that the database UUID (a unique identifier which
> +       persists until e.g. the database is compacted)
> +       is HEX; exit with an error if it is not. This is useful to
> +       detect rollover in modification counts on messages. You can
> +       find this UUID in the first column of output from
> +       ``notmuch count --output=modifications``

I think it's not entirely clear what the connection between the UUID and
the revisions on messages is. For example, compacting a database
shouldn't make a difference "extensionally", so the naive assumption
might be that the UUID, or, maybe more importantly, the revision counts
don't change.

For example, if lastmod queries are used to incrementally back up or
synchronize tags, then it doesn't matter if the database got compacted,
all one cares about are the changes to messages' tags since an earlier
point.

Based on quick experiments, it seems that the revisions don't change
when compacting the database; is this something that the patches
guarantee or a coincidence?

Best regards,
Daniel

Re: [PATCH 4/5] cli: add global option "--uuid"

[David Bremner]

Daniel Schoepe <daniel@schoepe.org> writes:

> Based on quick experiments, it seems that the revisions don't change
> when compacting the database; is this something that the patches
> guarantee or a coincidence?

Now that you mention it, I can see that despite essentially quoting the
xapian docs, compacting might not be the best example.

According to the xapian docs:

   The UUID will persist for the lifetime of the database.

    Replicas (eg, made with the replication protocol, or by copying all
    the database files) will have the same UUID. However, copies (made
    with copydatabase, or xapian-compact) will have different UUIDs.

So compacting is in fact a bit of a strange case, it preserves the
lastmod counters (because those are just terms on documents), but not
the uuid. This means from notmuch point of view have no programmatic way
to know that we can trust the lastmod data, i.e. no way to distinguish a
previous notmuch-compact from "notmuch dump && rm the database &&
notmuch new && notmuch restore".  On the other hand, I _think_ we could
promise the user that lastmod data is preserved by compacting; they
would just have to tell whatever external tool to ignore uuid errors and
perhaps set the new uuid.

d

Re: [PATCH 4/5] cli: add global option "--uuid"

[Daniel Schoepe]

On Mon, 10 Aug 2015 13:32 +0200, David Bremner wrote:
> So compacting is in fact a bit of a strange case, it preserves the
> lastmod counters (because those are just terms on documents), but not
> the uuid. This means from notmuch point of view have no programmatic way
> to know that we can trust the lastmod data, i.e. no way to distinguish a
> previous notmuch-compact from "notmuch dump && rm the database &&
> notmuch new && notmuch restore".  On the other hand, I _think_ we could
> promise the user that lastmod data is preserved by compacting; they
> would just have to tell whatever external tool to ignore uuid errors and
> perhaps set the new uuid.

Yeah, I think it's reasonable to assume that if the user will delete and
recreate the database, they will also perform the necessary
reinitialization of whatever tool makes use of lastmod / UUID. I guess
this also means that tools often only care about lastmod values and not
the UUID, to allow for compacting without thinking the database is
completely different now.

Another option is to include lastmod values in the output of
notmuch-dump. Doing that in a clean way might be hard though and I think
this shouldn't be part of this series (which looks fine to me now).

Best regards,
Daniel

Re: [PATCH 3/5] cli/count: add --output=modifications

[Tomi Ollila]

On Sun, Aug 09 2015, David Bremner <david@tethera.net> wrote:

> We need some way to extract the uuid/revision of the database, and
> count seems like the least bad choice of current commands.
> The (perhaps weak) argument for count over search is that count
> already reports statistics about the entire database.

The '(perhaps weak)' part could be discussed outside of the commit message ;)
... but read below for (IMO) "real" issues...

> ---
>  notmuch-count.c                | 18 +++++++++++++++++-
>  test/T570-revision-tracking.sh | 12 ++++++++++++
>  2 files changed, 29 insertions(+), 1 deletion(-)
>
> diff --git a/notmuch-count.c b/notmuch-count.c
> index 57a88a8..7c61ccb 100644
> --- a/notmuch-count.c
> +++ b/notmuch-count.c
> @@ -25,6 +25,7 @@ enum {
>      OUTPUT_THREADS,
>      OUTPUT_MESSAGES,
>      OUTPUT_FILES,
> +    OUTPUT_LASTMOD,
>  };
>  
>  /* The following is to allow future options to be added more easily */
> @@ -71,6 +72,9 @@ print_count (notmuch_database_t *notmuch, const char *query_str,
>  {
>      notmuch_query_t *query;
>      size_t i;
> +    unsigned long revision;
> +    const char *uuid;
> +    int ret = 0;
>  
>      query = notmuch_query_create (notmuch, query_str);
>      if (query == NULL) {
> @@ -91,11 +95,22 @@ print_count (notmuch_database_t *notmuch, const char *query_str,
>      case OUTPUT_FILES:
>  	printf ("%u\n", count_files (query));
>  	break;
> +    case OUTPUT_LASTMOD:
> +	if (strcmp (notmuch_query_get_query_string (query), "*") != 0) {
> +	    fprintf (stderr, "Error: Only '*' is currently supported "
> +		     " with output=modifications\n");
> +	    ret = 1;
> +	    goto DONE;
> +	}

I have three comments on this series, the first one is hardest, second
trivial and one could be either way.

> +
> +	revision = notmuch_database_get_revision (notmuch, &uuid);
> +	printf ("%s\t%lu\n", uuid, revision);

Currently, `notmuch count` outputs lines that contain just one integer;
this changes this by introducing output with uuid ([0-9a-f-]) and integer
delimited by tab character.

To put it lightly, this looks "inconsistent" and don't please my aesthetic
eye.

One option (being it worse or better) could be that by default only
lastmod value is printed and with separate option it is prefixed with
database UUID (in every --output option).

BTW: I did not see notmuch-count.rst updated. good for now as it reduces
the amount of work during this bikeshed period ;D

continuing in next email...

Tomi



>      }
>  
> + DONE:
>      notmuch_query_destroy (query);
>  
> -    return 0;
> +    return ret;
>  }
>  
>  static int
> @@ -139,6 +154,7 @@ notmuch_count_command (notmuch_config_t *config, int argc, char *argv[])
>  	  (notmuch_keyword_t []){ { "threads", OUTPUT_THREADS },
>  				  { "messages", OUTPUT_MESSAGES },
>  				  { "files", OUTPUT_FILES },
> +				  { "modifications", OUTPUT_LASTMOD },
>  				  { 0, 0 } } },
>  	{ NOTMUCH_OPT_KEYWORD, &exclude, "exclude", 'x',
>  	  (notmuch_keyword_t []){ { "true", EXCLUDE_TRUE },
> diff --git a/test/T570-revision-tracking.sh b/test/T570-revision-tracking.sh
> index e0a5703..008c5d0 100755
> --- a/test/T570-revision-tracking.sh
> +++ b/test/T570-revision-tracking.sh
> @@ -34,4 +34,16 @@ UUID	53
>  EOF
>  test_expect_equal_file EXPECTED CLEAN
>  
> +grep '^[0-9a-f]' OUTPUT > INITIAL_OUTPUT
> +
> +test_begin_subtest "output of count matches test code"
> +notmuch count --output=modifications '*' > OUTPUT
> +test_expect_equal_file INITIAL_OUTPUT OUTPUT
> +
> +test_begin_subtest "modification count increases"
> +before=$(notmuch count --output=modifications '*' | cut -f2)
> +notmuch tag +a-random-tag-8743632 '*'
> +after=$(notmuch count --output=modifications '*' | cut -f2)
> +result=$(($before < $after))
> +test_expect_equal 1 ${result}
>  test_done
> -- 
> 2.1.4
>
> _______________________________________________
> notmuch mailing list
> notmuch@notmuchmail.org
> http://notmuchmail.org/mailman/listinfo/notmuch

Re: [PATCH 4/5] cli: add global option "--uuid"

[Tomi Ollila]

On Sun, Aug 09 2015, David Bremner <david@tethera.net> wrote:

> The function notmuch_exit_if_unmatched_db_uuid is split from
> notmuch_process_shared_options because it needs an open notmuch
> database.
> ---
>  doc/man1/notmuch.rst           | 12 ++++++++++--
>  notmuch-client.h               |  4 ++++
>  notmuch-compact.c              |  4 ++++
>  notmuch-config.c               |  4 ++++
>  notmuch-count.c                |  2 ++
>  notmuch-dump.c                 |  2 ++
>  notmuch-insert.c               |  2 ++
>  notmuch-new.c                  |  3 ++-
>  notmuch-reply.c                |  2 ++
>  notmuch-restore.c              |  2 ++
>  notmuch-search.c               |  2 ++
>  notmuch-setup.c                |  4 ++++
>  notmuch-show.c                 |  2 ++
>  notmuch-tag.c                  |  2 ++
>  notmuch.c                      | 18 ++++++++++++++++++
>  test/T570-revision-tracking.sh | 27 +++++++++++++++++++++++++++
>  test/random-corpus.c           |  2 ++
>  17 files changed, 91 insertions(+), 3 deletions(-)
>
> diff --git a/doc/man1/notmuch.rst b/doc/man1/notmuch.rst
> index 0401c91..0b89544 100644
> --- a/doc/man1/notmuch.rst
> +++ b/doc/man1/notmuch.rst
> @@ -51,9 +51,17 @@ Supported global options for ``notmuch`` include
>  	Specify the configuration file to use. This overrides any
>  	configuration file specified by ${NOTMUCH\_CONFIG}.
>  
> +    ``--uuid=HEX``
> +       Enforce that the database UUID (a unique identifier which
> +       persists until e.g. the database is compacted)
> +       is HEX; exit with an error if it is not. This is useful to
> +       detect rollover in modification counts on messages. You can
> +       find this UUID in the first column of output from
> +       ``notmuch count --output=modifications``
> +
>  All global options except ``--config`` can also be specified after the
> -command. For example, ``notmuch subcommand --version`` is equivalent to
> -``notmuch --version subcommand``.
> +command. For example, ``notmuch subcommand --uuid=HEX`` is
> +equivalent to ``notmuch --uuid=HEX subcommand``.
>  
>  COMMANDS
>  ========
> diff --git a/notmuch-client.h b/notmuch-client.h
> index 78680aa..4a4f86c 100644
> --- a/notmuch-client.h
> +++ b/notmuch-client.h
> @@ -466,7 +466,11 @@ notmuch_database_dump (notmuch_database_t *notmuch,
>  		       notmuch_bool_t gzip_output);
>  
>  #include "command-line-arguments.h"
> +
> +extern char *notmuch_requested_db_uuid;
>  extern const notmuch_opt_desc_t  notmuch_shared_options [];
> +void notmuch_exit_if_unmatched_db_uuid (notmuch_database_t *notmuch);
> +
>  void notmuch_process_shared_options (const char* subcommand_name);
>  int notmuch_minimal_options (const char* subcommand_name,
>  			     int argc, char **argv);
> diff --git a/notmuch-compact.c b/notmuch-compact.c
> index 5be551d..61fccd8 100644
> --- a/notmuch-compact.c
> +++ b/notmuch-compact.c
> @@ -46,6 +46,10 @@ notmuch_compact_command (notmuch_config_t *config, int argc, char *argv[])
>      if (opt_index < 0)
>  	return EXIT_FAILURE;
>  
> +    if (notmuch_requested_db_uuid)
> +	fprintf (stderr, "Warning: ignoring --uuid=%s\n",
> +		 notmuch_requested_db_uuid);
> +

I see in these commands this warning is issued, we don't open the database
(at this time, IIRC compact opens it later)...

... In this case I'm wondering whether giving --uuid should be considered
as error (and stopping) instead of printing warning and continuing.  In
some other cases we do warn and continue but here ... we're modifying
the database or configurations...
... later we could add database uuid check always if --uuid is given
and allow continuing if it matches in these cases...

One more (now the trivial thing) to go, do not stop here ;)

>      notmuch_process_shared_options (argv[0]);
>  
>      if (! quiet)
> diff --git a/notmuch-config.c b/notmuch-config.c
> index 9348278..d252bb2 100644
> --- a/notmuch-config.c
> +++ b/notmuch-config.c
> @@ -878,6 +878,10 @@ notmuch_config_command (notmuch_config_t *config, int argc, char *argv[])
>      if (opt_index < 0)
>  	return EXIT_FAILURE;
>  
> +    if (notmuch_requested_db_uuid)
> +	fprintf (stderr, "Warning: ignoring --uuid=%s\n",
> +		 notmuch_requested_db_uuid);
> +
>      /* skip at least subcommand argument */
>      argc-= opt_index;
>      argv+= opt_index;
> diff --git a/notmuch-count.c b/notmuch-count.c
> index 7c61ccb..091e376 100644
> --- a/notmuch-count.c
> +++ b/notmuch-count.c
> @@ -191,6 +191,8 @@ notmuch_count_command (notmuch_config_t *config, int argc, char *argv[])
>  			       NOTMUCH_DATABASE_MODE_READ_ONLY, &notmuch))
>  	return EXIT_FAILURE;
>  
> +    notmuch_exit_if_unmatched_db_uuid (notmuch);
> +
>      query_str = query_string_from_args (config, argc-opt_index, argv+opt_index);
>      if (query_str == NULL) {
>  	fprintf (stderr, "Out of memory.\n");
> diff --git a/notmuch-dump.c b/notmuch-dump.c
> index fab22bd..24fc2f2 100644
> --- a/notmuch-dump.c
> +++ b/notmuch-dump.c
> @@ -215,6 +215,8 @@ notmuch_dump_command (notmuch_config_t *config, int argc, char *argv[])
>  			       NOTMUCH_DATABASE_MODE_READ_WRITE, &notmuch))
>  	return EXIT_FAILURE;
>  
> +    notmuch_exit_if_unmatched_db_uuid (notmuch);
> +
>      char *output_file_name = NULL;
>      int opt_index;
>  
> diff --git a/notmuch-insert.c b/notmuch-insert.c
> index c277d62..5205c17 100644
> --- a/notmuch-insert.c
> +++ b/notmuch-insert.c
> @@ -536,6 +536,8 @@ notmuch_insert_command (notmuch_config_t *config, int argc, char *argv[])
>  			       NOTMUCH_DATABASE_MODE_READ_WRITE, &notmuch))
>  	return EXIT_FAILURE;
>  
> +    notmuch_exit_if_unmatched_db_uuid (notmuch);
> +
>      /* Write the message to the Maildir new directory. */
>      newpath = maildir_write_new (config, STDIN_FILENO, maildir);
>      if (! newpath) {
> diff --git a/notmuch-new.c b/notmuch-new.c
> index ee786a3..514e06a 100644
> --- a/notmuch-new.c
> +++ b/notmuch-new.c
> @@ -1009,10 +1009,11 @@ notmuch_new_command (notmuch_config_t *config, int argc, char *argv[])
>  		fputs (status_string, stderr);
>  		free (status_string);
>  	    }
> -
>  	    return EXIT_FAILURE;
>  	}
>  
> +	notmuch_exit_if_unmatched_db_uuid (notmuch);
> +
>  	if (notmuch_database_needs_upgrade (notmuch)) {
>  	    time_t now = time (NULL);
>  	    struct tm *gm_time = gmtime (&now);
> diff --git a/notmuch-reply.c b/notmuch-reply.c
> index 4464741..7c5c28f 100644
> --- a/notmuch-reply.c
> +++ b/notmuch-reply.c
> @@ -831,6 +831,8 @@ notmuch_reply_command (notmuch_config_t *config, int argc, char *argv[])
>  			       NOTMUCH_DATABASE_MODE_READ_ONLY, &notmuch))
>  	return EXIT_FAILURE;
>  
> +    notmuch_exit_if_unmatched_db_uuid (notmuch);
> +
>      query = notmuch_query_create (notmuch, query_string);
>      if (query == NULL) {
>  	fprintf (stderr, "Out of memory\n");
> diff --git a/notmuch-restore.c b/notmuch-restore.c
> index 2a534dc..9abc64f 100644
> --- a/notmuch-restore.c
> +++ b/notmuch-restore.c
> @@ -165,6 +165,8 @@ notmuch_restore_command (notmuch_config_t *config, int argc, char *argv[])
>      }
>  
>      notmuch_process_shared_options (argv[0]);
> +    notmuch_exit_if_unmatched_db_uuid (notmuch);
> +
>      name_for_error = input_file_name ? input_file_name : "stdin";
>  
>      if (! accumulate)
> diff --git a/notmuch-search.c b/notmuch-search.c
> index b89a17e..3076c3f 100644
> --- a/notmuch-search.c
> +++ b/notmuch-search.c
> @@ -583,6 +583,8 @@ _notmuch_search_prepare (search_context_t *ctx, notmuch_config_t *config, int ar
>  	return EXIT_FAILURE;
>      }
>  
> +    notmuch_exit_if_unmatched_db_uuid (ctx->notmuch);
> +
>      query_str = query_string_from_args (ctx->notmuch, argc, argv);
>      if (query_str == NULL) {
>  	fprintf (stderr, "Out of memory.\n");
> diff --git a/notmuch-setup.c b/notmuch-setup.c
> index 7dd5822..9aaf928 100644
> --- a/notmuch-setup.c
> +++ b/notmuch-setup.c
> @@ -148,6 +148,10 @@ notmuch_setup_command (notmuch_config_t *config,
>      if (notmuch_minimal_options ("setup", argc, argv) < 0)
>  	return EXIT_FAILURE;
>  
> +    if (notmuch_requested_db_uuid)
> +	fprintf (stderr, "Warning: ignoring --uuid=%s\n",
> +		 notmuch_requested_db_uuid);
> +
>      if (notmuch_config_is_new (config))
>  	welcome_message_pre_setup ();
>  
> diff --git a/notmuch-show.c b/notmuch-show.c
> index b80933a..6ef3308 100644
> --- a/notmuch-show.c
> +++ b/notmuch-show.c
> @@ -1213,6 +1213,8 @@ notmuch_show_command (notmuch_config_t *config, int argc, char *argv[])
>  			       NOTMUCH_DATABASE_MODE_READ_ONLY, &notmuch))
>  	return EXIT_FAILURE;
>  
> +    notmuch_exit_if_unmatched_db_uuid (notmuch);
> +
>      query = notmuch_query_create (notmuch, query_string);
>      if (query == NULL) {
>  	fprintf (stderr, "Out of memory\n");
> diff --git a/notmuch-tag.c b/notmuch-tag.c
> index 38d99aa..7ae98f6 100644
> --- a/notmuch-tag.c
> +++ b/notmuch-tag.c
> @@ -261,6 +261,8 @@ notmuch_tag_command (notmuch_config_t *config, int argc, char *argv[])
>  			       NOTMUCH_DATABASE_MODE_READ_WRITE, &notmuch))
>  	return EXIT_FAILURE;
>  
> +    notmuch_exit_if_unmatched_db_uuid (notmuch);
> +
>      if (notmuch_config_get_maildir_synchronize_flags (config))
>  	tag_flags |= TAG_FLAG_MAILDIR_SYNC;
>  
> diff --git a/notmuch.c b/notmuch.c
> index 9580c3f..ce6c575 100644
> --- a/notmuch.c
> +++ b/notmuch.c
> @@ -47,10 +47,12 @@ static int
>  _help_for (const char *topic);
>  
>  static notmuch_bool_t print_version = FALSE, print_help = FALSE;
> +char *notmuch_requested_db_uuid = NULL;
>  
>  const notmuch_opt_desc_t notmuch_shared_options [] = {
>      { NOTMUCH_OPT_BOOLEAN, &print_version, "version", 'v', 0 },
>      { NOTMUCH_OPT_BOOLEAN, &print_help, "help", 'h', 0 },
> +    { NOTMUCH_OPT_STRING, &notmuch_requested_db_uuid, "uuid", 'u', 0 },
>      {0, 0, 0, 0, 0}
>  };
>  
> @@ -218,6 +220,22 @@ be supported in the future.\n", notmuch_format_version);
>      }
>  }
>  
> +void
> +notmuch_exit_if_unmatched_db_uuid (notmuch_database_t *notmuch)
> +{
> +    const char *uuid = NULL;
> +
> +    if (!notmuch_requested_db_uuid)
> +	return;
> +    IGNORE_RESULT (notmuch_database_get_revision (notmuch, &uuid));
> +
> +    if (strcmp (notmuch_requested_db_uuid, uuid) != 0){
> +	fprintf (stderr, "Error: requested database revision %s does not match %s\n",
> +		 notmuch_requested_db_uuid, uuid);
> +	exit (1);
> +    }
> +}
> +
>  static void
>  exec_man (const char *page)
>  {
> diff --git a/test/T570-revision-tracking.sh b/test/T570-revision-tracking.sh
> index 008c5d0..74e3ba9 100755
> --- a/test/T570-revision-tracking.sh
> +++ b/test/T570-revision-tracking.sh
> @@ -46,4 +46,31 @@ notmuch tag +a-random-tag-8743632 '*'
>  after=$(notmuch count --output=modifications '*' | cut -f2)
>  result=$(($before < $after))
>  test_expect_equal 1 ${result}
> +
> +notmuch count --output=modifications '*' | cut -f1 > UUID

the above could be

UUID=$(notmuch count --output=modifications '*' | cut -f1)

> +
> +test_expect_success 'search succeeds with correct uuid' \
> +		    "notmuch search --uuid=$(cat UUID) '*'"

And then these just as 

test_expect_success 'search succeeds with correct uuid' \
		    "notmuch search --uuid=${UUID} '*'"


Tomi


> +
> +test_expect_success 'uuid works as global option ' \
> +		    "notmuch --uuid=$(cat UUID) search '*'"
> +
> +test_expect_code 1 'uuid works as global option II' \
> +		    "notmuch --uuid=this-is-no-uuid search '*'"
> +
> +test_expect_code 1 'search fails with incorrect uuid' \
> +		 "notmuch search --uuid=this-is-no-uuid '*'"
> +
> +test_expect_success 'show succeeds with correct uuid' \
> +		    "notmuch show --uuid=$(cat UUID) '*'"
> +
> +test_expect_code 1 'show fails with incorrect uuid' \
> +		 "notmuch show --uuid=this-is-no-uuid '*'"
> +
> +test_expect_success 'tag succeeds with correct uuid' \
> +		    "notmuch tag --uuid=$(cat UUID) +test '*'"
> +
> +test_expect_code 1 'tag fails with incorrect uuid' \
> +		 "notmuch tag --uuid=this-is-no-uuid '*' +test2"
> +
>  test_done
> diff --git a/test/random-corpus.c b/test/random-corpus.c
> index b377eb4..d74271d 100644
> --- a/test/random-corpus.c
> +++ b/test/random-corpus.c
> @@ -119,6 +119,8 @@ const notmuch_opt_desc_t notmuch_shared_options[] = {
>  	{ 0, 0, 0, 0, 0 }
>  };
>  
> +char *notmuch_requested_db_uuid = NULL;
> +
>  void
>  notmuch_process_shared_options (unused (const char *dummy))
>  {
> -- 
> 2.1.4

Re: [PATCH 3/5] cli/count: add --output=modifications

[David Bremner]

Tomi Ollila <tomi.ollila@iki.fi> writes:

> Currently, `notmuch count` outputs lines that contain just one integer;
> this changes this by introducing output with uuid ([0-9a-f-]) and integer
> delimited by tab character.
>
> To put it lightly, this looks "inconsistent" and don't please my aesthetic
> eye.
>
> One option (being it worse or better) could be that by default only
> lastmod value is printed and with separate option it is prefixed with
> database UUID (in every --output option).

Can you think of any use case for the uuid with the other count outputs?
It feels pretty artificial to me.

Another option is to make a "notmuch metadata" command. I'm not really
sure about the syntax, but perhaps a uuid option makes more sense
there, so e.g.

       notmuch metadata --with-uuid lastmod

I'm pretty convinced that we need report uuid and lastmod together (at
least optionally). I'm less sure we need a full get/set interface for
metadata, since people with that use case could use xapian-metadata.

Re: [PATCH 3/5] cli/count: add --output=modifications

[Tomi Ollila]

On Tue, Aug 11 2015, David Bremner <david@tethera.net> wrote:

> Tomi Ollila <tomi.ollila@iki.fi> writes:
>
>> Currently, `notmuch count` outputs lines that contain just one integer;
>> this changes this by introducing output with uuid ([0-9a-f-]) and integer
>> delimited by tab character.
>>
>> To put it lightly, this looks "inconsistent" and don't please my aesthetic
>> eye.
>>
>> One option (being it worse or better) could be that by default only
>> lastmod value is printed and with separate option it is prefixed with
>> database UUID (in every --output option).
>
> Can you think of any use case for the uuid with the other count outputs?
> It feels pretty artificial to me.

I don't... I just thought something consistent to be used w/ notmuch count...

> Another option is to make a "notmuch metadata" command. I'm not really
> sure about the syntax, but perhaps a uuid option makes more sense
> there, so e.g.
>
>        notmuch metadata --with-uuid lastmod

or notmuch metadata key [key [key]], then (currently) we could have:

$ notmuch metadata uuid
$ notmuch metadata lastmod
$ notmuch metadata uuid lastmod
$ notmuch metadata lastmod uuid 

Now we can bikeshed whether this is good idea -- and what to do (and when)
if unknown key is used in request...


Tomi

> I'm pretty convinced that we need report uuid and lastmod together (at
> least optionally). I'm less sure we need a full get/set interface for
> metadata, since people with that use case could use xapian-metadata.
>

Re: [PATCH 3/5] cli/count: add --output=modifications

[David Bremner]

Tomi Ollila <tomi.ollila@iki.fi> writes:


> $ notmuch metadata uuid
> $ notmuch metadata lastmod
> $ notmuch metadata uuid lastmod
> $ notmuch metadata lastmod uuid 
>
> Now we can bikeshed whether this is good idea --

it seems OK. We will have some set of 'fake' metadata keys that are not
actually stored in xapian metadata. Perversely this includes both
lastmod and uuid

> and what to do (and when)
> if unknown key is used in request...

Error out, I'd say, but if it's not one of our fake metadata items we
can in general just ask xapian for that value of that metadata, and
complain if it doesn't exist.

Re: [PATCH 3/5] cli/count: add --output=modifications

[Daniel Schoepe]

Maybe another consistent solution is to add a --format argument to
count, with JSON as one option. That way we could even add more
information to other count results (if that becomes needed at some
point) without breaking code using the JSON interface.

Best regards,
Daniel