[PATCH 0/7] Structed output versioning support

[PATCH 0/7] Structed output versioning support

[Austin Clements]

This series obsoletes [0] and must be applied on top of the Emacs CLI
error handling series [1].  This is much simpler than the original
series because it no longer includes the Emacs CLI error handling.
This also switches to a CLI-wide minimum version instead of a
per-command version, prints a warning if an old but still supported
version is requested to ease client maintenance, and renames
--use-schema to --format-version (which parallels --format better and
simplifies error text).

[0] id:1354416002-3557-1-git-send-email-amdragon@mit.edu
[1] id:1355601860-30121-1-git-send-email-amdragon@mit.edu

[PATCH 1/7] cli: Framework for structured output versioning

[Austin Clements]

Currently there is a period of pain whenever we make
backward-incompatible changes to the structured output format, which
discourages not only backward-incompatible improvements to the format,
but also backwards-compatible additions that may not be "perfect".  In
the end, these problems limit experimentation and innovation.

This series of patches introduces a way for CLI callers to request a
specific format version on the command line and to determine if the
CLI does not supported the requested version (and perhaps present a
useful diagnostic to the user).  Since the caller requests a format
version, it's also possible for the CLI to support multiple
incompatible versions simultaneously, unlike the alternate approach of
including version information in the output.

This patch lays the groundwork by introducing a versioning convention,
standard exit codes, and a utility function to check the requested
version and produce standardized diagnostic messages and exit
statuses.
---
 devel/schemata   |    2 ++
 notmuch-client.h |   45 +++++++++++++++++++++++++++++++++++++++++++++
 notmuch.c        |   32 ++++++++++++++++++++++++++++++++
 3 files changed, 79 insertions(+)

diff --git a/devel/schemata b/devel/schemata
index d1ab983..292f287 100644
--- a/devel/schemata
+++ b/devel/schemata
@@ -14,6 +14,8 @@ are interleaved. Keys are printed as keywords (symbols preceded by a
 colon), e.g. (:id "123" :time 54321 :from "foobar"). Null is printed as
 nil, true as t and false as nil.
 
+This is version 1 of the structured output format.
+
 Common non-terminals
 --------------------
 
diff --git a/notmuch-client.h b/notmuch-client.h
index 1c336dc..d7b352e 100644
--- a/notmuch-client.h
+++ b/notmuch-client.h
@@ -117,6 +117,51 @@ chomp_newline (char *str)
 	str[strlen(str)-1] = '\0';
 }
 
+/* Exit status code indicating the requested format version is too old
+ * (support for that version has been dropped).  CLI code should use
+ * notmuch_exit_if_unsupported_format rather than directly exiting
+ * with this code.
+ */
+#define NOTMUCH_EXIT_FORMAT_TOO_OLD 20
+/* Exit status code indicating the requested format version is newer
+ * than the version supported by the CLI.  CLI code should use
+ * notmuch_exit_if_unsupported_format rather than directly exiting
+ * with this code.
+ */
+#define NOTMUCH_EXIT_FORMAT_TOO_NEW 21
+
+/* The current structured output format version.  Requests for format
+ * versions above this will return an error.  Backwards-incompatible
+ * changes such as removing map fields, changing the meaning of map
+ * fields, or changing the meanings of list elements should increase
+ * this.  New (required) map fields can be added without increasing
+ * this.
+ */
+#define NOTMUCH_FORMAT_CUR 1
+/* The minimum supported structured output format version.  Requests
+ * for format versions below this will return an error. */
+#define NOTMUCH_FORMAT_MIN 1
+
+/* The output format version requested by the caller on the command
+ * line.  If no format version is requested, this will be set to
+ * NOTMUCH_FORMAT_CUR.  Even though the command-line option is
+ * per-command, this is global because commands can share structured
+ * output code.
+ */
+extern int notmuch_format_version;
+
+/* Commands that support structured output should support the
+ * following argument
+ *  { NOTMUCH_OPT_INT, &notmuch_format_version, "format-version", 0, 0 }
+ * and should invoke notmuch_exit_if_unsupported_format to check the
+ * requested version.  If notmuch_format_version is outside the
+ * supported range, this will print a detailed diagnostic message for
+ * the user and exit with NOTMUCH_EXIT_FORMAT_TOO_{OLD,NEW} to inform
+ * the invoking program of the problem.
+ */
+void
+notmuch_exit_if_unsupported_format (void);
+
 notmuch_crypto_context_t *
 notmuch_crypto_get_context (notmuch_crypto_t *crypto, const char *protocol);
 
diff --git a/notmuch.c b/notmuch.c
index 4ff66e3..9516dfb 100644
--- a/notmuch.c
+++ b/notmuch.c
@@ -82,6 +82,8 @@ static command_t commands[] = {
       "This message, or more detailed help for the named command." }
 };
 
+int notmuch_format_version;
+
 static void
 usage (FILE *out)
 {
@@ -109,6 +111,33 @@ usage (FILE *out)
     "and \"notmuch help search-terms\" for the common search-terms syntax.\n\n");
 }
 
+void
+notmuch_exit_if_unsupported_format (void)
+{
+    if (notmuch_format_version > NOTMUCH_FORMAT_CUR) {
+	fprintf (stderr, "\
+A caller requested output format version %d, but the installed notmuch\n\
+CLI only supports up to format version %d.  You may need to upgrade your\n\
+notmuch CLI.\n",
+		 notmuch_format_version, NOTMUCH_FORMAT_CUR);
+	exit (NOTMUCH_EXIT_FORMAT_TOO_NEW);
+    } else if (notmuch_format_version < NOTMUCH_FORMAT_MIN) {
+	fprintf (stderr, "\
+A caller requested output format version %d, which is no longer supported\n\
+by the notmuch CLI (it requires at least version %d).  You may need to\n\
+upgrade your notmuch front-end.\n",
+		 notmuch_format_version, NOTMUCH_FORMAT_MIN);
+	exit (NOTMUCH_EXIT_FORMAT_TOO_OLD);
+    } else if (notmuch_format_version != NOTMUCH_FORMAT_CUR) {
+	/* Warn about old version requests so compatibility issues are
+	 * less likely when we drop support for a deprecated format
+	 * versions. */
+	fprintf (stderr, "\
+A caller requested deprecated output format version %d, which may not\n\
+be supported in the future.\n", notmuch_format_version);
+    }
+}
+
 static void
 exec_man (const char *page)
 {
@@ -242,6 +271,9 @@ main (int argc, char *argv[])
     g_mime_init (0);
     g_type_init ();
 
+    /* Globally default to the current output format version. */
+    notmuch_format_version = NOTMUCH_FORMAT_CUR;
+
     if (argc == 1)
 	return notmuch (local);
 
-- 
1.7.10.4

[PATCH 2/7] search: Support --format-version

[Austin Clements]

---
 man/man1/notmuch-search.1 |   20 ++++++++++++++++++++
 notmuch-search.c          |    3 +++
 2 files changed, 23 insertions(+)

diff --git a/man/man1/notmuch-search.1 b/man/man1/notmuch-search.1
index 0aff348..5c771fa 100644
--- a/man/man1/notmuch-search.1
+++ b/man/man1/notmuch-search.1
@@ -32,6 +32,15 @@ Presents the results in either JSON, S-Expressions or plain-text (default).
 
 .RS 4
 .TP 4
+.BR \-\-format-version=N
+
+Use the specified structured output format version.  This is intended
+for programs that invoke \fBnotmuch\fR(1) internally.  If omitted, the
+latest supported version will be used.
+.RE
+
+.RS 4
+.TP 4
 .B \-\-output=(summary|threads|messages|files|tags)
 
 .RS 4
@@ -126,6 +135,17 @@ In this case all matching threads are returned but the "match count"
 is the number of matching non-excluded messages in the thread.
 .RE
 
+.SH EXIT STATUS
+
+This command supports the following special exit status codes
+
+.TP
+.B 20
+The requested format version is too old.
+.TP
+.B 21
+The requested format version is too new.
+
 .SH SEE ALSO
 
 \fBnotmuch\fR(1), \fBnotmuch-config\fR(1), \fBnotmuch-count\fR(1),
diff --git a/notmuch-search.c b/notmuch-search.c
index 6218622..7704915 100644
--- a/notmuch-search.c
+++ b/notmuch-search.c
@@ -318,6 +318,7 @@ notmuch_search_command (void *ctx, int argc, char *argv[])
 				  { "sexp", NOTMUCH_FORMAT_SEXP },
 				  { "text", NOTMUCH_FORMAT_TEXT },
 				  { 0, 0 } } },
+	{ NOTMUCH_OPT_INT, &notmuch_format_version, "format-version", 0, 0 },
 	{ NOTMUCH_OPT_KEYWORD, &output, "output", 'o',
 	  (notmuch_keyword_t []){ { "summary", OUTPUT_SUMMARY },
 				  { "threads", OUTPUT_THREADS },
@@ -356,6 +357,8 @@ notmuch_search_command (void *ctx, int argc, char *argv[])
 	INTERNAL_ERROR("no output format selected");
     }
 
+    notmuch_exit_if_unsupported_format ();
+
     config = notmuch_config_open (ctx, NULL, NULL);
     if (config == NULL)
 	return 1;
-- 
1.7.10.4

[PATCH 3/7] show: Support --format-version

[Austin Clements]

---
 man/man1/notmuch-show.1 |   20 ++++++++++++++++++++
 notmuch-show.c          |    3 +++
 2 files changed, 23 insertions(+)

diff --git a/man/man1/notmuch-show.1 b/man/man1/notmuch-show.1
index bd41c48..44cec46 100644
--- a/man/man1/notmuch-show.1
+++ b/man/man1/notmuch-show.1
@@ -128,6 +128,15 @@ message.
 
 .RS 4
 .TP 4
+.BR \-\-format-version=N
+
+Use the specified structured output format version.  This is intended
+for programs that invoke \fBnotmuch\fR(1) internally.  If omitted, the
+latest supported version will be used.
+.RE
+
+.RS 4
+.TP 4
 .B \-\-part=N
 
 Output the single decoded MIME part N of a single message.  The search
@@ -201,6 +210,17 @@ column of output from the
 .B notmuch search
 command.
 
+.SH EXIT STATUS
+
+This command supports the following special exit status codes
+
+.TP
+.B 20
+The requested format version is too old.
+.TP
+.B 21
+The requested format version is too new.
+
 .SH SEE ALSO
 
 \fBnotmuch\fR(1), \fBnotmuch-config\fR(1), \fBnotmuch-count\fR(1),
diff --git a/notmuch-show.c b/notmuch-show.c
index a83fef9..39faf21 100644
--- a/notmuch-show.c
+++ b/notmuch-show.c
@@ -1066,6 +1066,7 @@ notmuch_show_command (void *ctx, unused (int argc), unused (char *argv[]))
 				  { "mbox", NOTMUCH_FORMAT_MBOX },
 				  { "raw", NOTMUCH_FORMAT_RAW },
 				  { 0, 0 } } },
+	{ NOTMUCH_OPT_INT, &notmuch_format_version, "format-version", 0, 0 },
 	{ NOTMUCH_OPT_KEYWORD, &exclude, "exclude", 'x',
 	  (notmuch_keyword_t []){ { "true", EXCLUDE_TRUE },
 				  { "false", EXCLUDE_FALSE },
@@ -1128,6 +1129,8 @@ notmuch_show_command (void *ctx, unused (int argc), unused (char *argv[]))
 	break;
     }
 
+    notmuch_exit_if_unsupported_format ();
+
     /* Default is entire-thread = FALSE except for format=json and
      * format=sexp. */
     if (entire_thread == ENTIRE_THREAD_DEFAULT) {
-- 
1.7.10.4

[PATCH 4/7] reply: Support --format-version

[Austin Clements]

---
 man/man1/notmuch-reply.1 |   21 +++++++++++++++++++++
 notmuch-reply.c          |    3 +++
 2 files changed, 24 insertions(+)

diff --git a/man/man1/notmuch-reply.1 b/man/man1/notmuch-reply.1
index fa04c9e..9fa1956 100644
--- a/man/man1/notmuch-reply.1
+++ b/man/man1/notmuch-reply.1
@@ -57,6 +57,16 @@ to create a reply message intelligently.
 Only produces In\-Reply\-To, References, To, Cc, and Bcc headers.
 .RE
 .RE
+
+.RS
+.TP 4
+.BR \-\-format-version=N
+
+Use the specified structured output format version.  This is intended
+for programs that invoke \fBnotmuch\fR(1) internally.  If omitted, the
+latest supported version will be used.
+.RE
+
 .RS
 .TP 4
 .BR \-\-reply\-to= ( all | sender )
@@ -99,6 +109,17 @@ formats do not.
 .RE
 .RE
 
+.SH EXIT STATUS
+
+This command supports the following special exit status codes
+
+.TP
+.B 20
+The requested format version is too old.
+.TP
+.B 21
+The requested format version is too new.
+
 .SH SEE ALSO
 
 \fBnotmuch\fR(1), \fBnotmuch-config\fR(1), \fBnotmuch-count\fR(1),
diff --git a/notmuch-reply.c b/notmuch-reply.c
index 720749d..22c58ff 100644
--- a/notmuch-reply.c
+++ b/notmuch-reply.c
@@ -733,6 +733,7 @@ notmuch_reply_command (void *ctx, int argc, char *argv[])
 				  { "sexp", FORMAT_SEXP },
 				  { "headers-only", FORMAT_HEADERS_ONLY },
 				  { 0, 0 } } },
+	{ NOTMUCH_OPT_INT, &notmuch_format_version, "format-version", 0, 0 },
 	{ NOTMUCH_OPT_KEYWORD, &reply_all, "reply-to", 'r',
 	  (notmuch_keyword_t []){ { "all", TRUE },
 				  { "sender", FALSE },
@@ -759,6 +760,8 @@ notmuch_reply_command (void *ctx, int argc, char *argv[])
 	reply_format_func = notmuch_reply_format_default;
     }
 
+    notmuch_exit_if_unsupported_format ();
+
     config = notmuch_config_open (ctx, NULL, NULL);
     if (config == NULL)
 	return 1;
-- 
1.7.10.4

[PATCH 5/7] test: Sanity tests for the --format-version argument

[Austin Clements]

---
 test/json |    6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/test/json b/test/json
index bfafd55..8a01117 100755
--- a/test/json
+++ b/test/json
@@ -60,4 +60,10 @@ test_expect_equal_json "$output" "[{\"thread\": \"XXX\",
  \"tags\": [\"inbox\",
  \"unread\"]}]"
 
+test_expect_code 20 "Format version: too low" \
+    "notmuch search --format-version=0 \\*"
+
+test_expect_code 21 "Format version: too high" \
+    "notmuch search --format-version=999 \\*"
+
 test_done
-- 
1.7.10.4

[PATCH 6/7] emacs: Special handling for version mismatch errors

[Austin Clements]

Since Emacs has more semantic information, we suppress the generic
format version error from the CLI and give a more informative error.
---
 emacs/notmuch-lib.el |   11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)

diff --git a/emacs/notmuch-lib.el b/emacs/notmuch-lib.el
index 8f84087..9ec30ea 100644
--- a/emacs/notmuch-lib.el
+++ b/emacs/notmuch-lib.el
@@ -360,9 +360,18 @@ giving the output of command.  ERR-FILE, if provided, is the name
 of a file containing the error output of command.  OUTPUT and the
 contents of ERR-FILE will be included in the error message."
 
-  ;; This is implemented as a cond to make it easy to expand.
   (cond
    ((eq exit-status 0) t)
+   ((eq exit-status 20)
+    (notmuch-pop-up-error "Error: Version mismatch.
+Emacs requested an older output format than supported by the notmuch CLI.
+You may need to restart Emacs or upgrade your notmuch Emacs package.")
+    (error "notmuch CLI version mismatch"))
+   ((eq exit-status 21)
+    (notmuch-pop-up-error "Error: Version mismatch.
+Emacs requested a newer output format than supported by the notmuch CLI.
+You may need to restart Emacs or upgrade your notmuch package.")
+    (error "notmuch CLI version mismatch"))
    (t
     (notmuch-pop-up-error
      (concat
-- 
1.7.10.4

[PATCH 7/7] emacs: Use --format-version for search, show, and reply

[Austin Clements]

---
 emacs/notmuch-mua.el   |    2 +-
 emacs/notmuch-query.el |    2 +-
 emacs/notmuch.el       |    6 +++++-
 test/emacs             |    2 +-
 test/emacs-show        |    2 +-
 5 files changed, 9 insertions(+), 5 deletions(-)

diff --git a/emacs/notmuch-mua.el b/emacs/notmuch-mua.el
index ac2d29e..24eebff 100644
--- a/emacs/notmuch-mua.el
+++ b/emacs/notmuch-mua.el
@@ -146,7 +146,7 @@ list."
   (unless (bolp) (insert "\n")))
 
 (defun notmuch-mua-reply (query-string &optional sender reply-all)
-  (let ((args '("reply" "--format=json"))
+  (let ((args '("reply" "--format=json" "--format-version=1"))
 	reply
 	original)
     (when notmuch-show-process-crypto
diff --git a/emacs/notmuch-query.el b/emacs/notmuch-query.el
index e7e3520..6e9f406 100644
--- a/emacs/notmuch-query.el
+++ b/emacs/notmuch-query.el
@@ -29,7 +29,7 @@ A thread is a forest or list of trees. A tree is a two element
 list where the first element is a message, and the second element
 is a possibly empty forest of replies.
 "
-  (let ((args '("show" "--format=json")))
+  (let ((args '("show" "--format=json" "--format-version=1")))
     (if notmuch-show-process-crypto
 	(setq args (append args '("--decrypt"))))
     (setq args (append args search-terms))
diff --git a/emacs/notmuch.el b/emacs/notmuch.el
index b0fd387..63387a2 100644
--- a/emacs/notmuch.el
+++ b/emacs/notmuch.el
@@ -650,6 +650,10 @@ of the result."
 		      (insert "Incomplete search results (search process was killed).\n"))
 		  (when (eq status 'exit)
 		    (insert "End of search results.\n")
+		    ;; For version mismatch, there's no point in
+		    ;; showing the search buffer
+		    (when (or (= exit-status 20) (= exit-status 21))
+		      (kill-buffer))
 		    (condition-case nil
 			(notmuch-check-async-exit-status proc msg)
 		      ;; Suppress the error signal since strange
@@ -935,7 +939,7 @@ Other optional parameters are used as follows:
 	(let ((proc (start-process
 		     "notmuch-search" buffer
 		     notmuch-command "search"
-		     "--format=json"
+		     "--format=json" "--format-version=1"
 		     (if oldest-first
 			 "--sort=oldest-first"
 		       "--sort=newest-first")
diff --git a/test/emacs b/test/emacs
index 5067d67..6b18968 100755
--- a/test/emacs
+++ b/test/emacs
@@ -873,7 +873,7 @@ This is output
 Error: Unexpected output from notmuch search:
 This is an error
 End of search results.
-Error invoking notmuch.  $PWD/notmuch_fail search --format=json --sort=newest-first tag:inbox exited with status 1."
+Error invoking notmuch.  $PWD/notmuch_fail search --format=json --format-version=1 --sort=newest-first tag:inbox exited with status 1."
 
 
 test_done
diff --git a/test/emacs-show b/test/emacs-show
index 40fab61..ebf530b 100755
--- a/test/emacs-show
+++ b/test/emacs-show
@@ -178,7 +178,7 @@ test_emacs "(let ((notmuch-command \"$PWD/notmuch_fail\"))
 	       (with-current-buffer \"*Notmuch errors*\"
 		  (test-output \"ERROR\")))"
 test_expect_equal "$(cat OUTPUT ERROR)" "\
-Error invoking notmuch.  $PWD/notmuch_fail show --format=json --exclude=false ' * ' exited with status 1.
+Error invoking notmuch.  $PWD/notmuch_fail show --format=json --format-version=1 --exclude=false ' * ' exited with status 1.
 Error:
 This is an error
 Output:
-- 
1.7.10.4

Re: [PATCH 0/7] Structed output versioning support

[Tomi Ollila]

On Sun, Dec 16 2012, Austin Clements <amdragon@MIT.EDU> wrote:

> This series obsoletes [0] and must be applied on top of the Emacs CLI
> error handling series [1].  This is much simpler than the original
> series because it no longer includes the Emacs CLI error handling.
> This also switches to a CLI-wide minimum version instead of a
> per-command version, prints a warning if an old but still supported
> version is requested to ease client maintenance, and renames
> --use-schema to --format-version (which parallels --format better and
> simplifies error text).

LGTM.

> [0] id:1354416002-3557-1-git-send-email-amdragon@mit.edu
> [1] id:1355601860-30121-1-git-send-email-amdragon@mit.edu

Tomi

Re: [PATCH 0/7] Structed output versioning support

[Mark Walters]

LGTM +1

Best wishes

Mark

On Sun, 16 Dec 2012, Austin Clements <amdragon@MIT.EDU> wrote:
> This series obsoletes [0] and must be applied on top of the Emacs CLI
> error handling series [1].  This is much simpler than the original
> series because it no longer includes the Emacs CLI error handling.
> This also switches to a CLI-wide minimum version instead of a
> per-command version, prints a warning if an old but still supported
> version is requested to ease client maintenance, and renames
> --use-schema to --format-version (which parallels --format better and
> simplifies error text).
>
> [0] id:1354416002-3557-1-git-send-email-amdragon@mit.edu
> [1] id:1355601860-30121-1-git-send-email-amdragon@mit.edu

Re: [PATCH 1/7] cli: Framework for structured output versioning

[David Bremner]

Austin Clements <amdragon@MIT.EDU> writes:

>
> This series of patches introduces a way for CLI callers to request a
> specific format version on the command line and to determine if the
> CLI does not supported the requested version 

pushed, 

d