[PATCH v5 0/4] Reply enhancements (was: quoting HTML email ...)

[PATCH v5 0/4] Reply enhancements (was: quoting HTML email ...)

[Adam Wolfe Gordon]

Hi everyone,

There are relatively few changes from the last version [1], but the JSON
format has big changes again. A summary of all the changes:

* The JSON reply format now uses the new formatter from the show JSON
  format. This means that the MUA will not need to call notmuch show
  for text/* parts, except HTML, as the content of those parts will be
  included in the JSON output.

* The JSON reply format has changed in a few other minor ways, due to
  reusing the show code.

* Because the original message is now included in the reply format, the
  old show behavior of --format=json implies --entire-thread is reinstated.
  I still think this is a weird behavior, but there's no good reason to
  change it.

* The emacs code is simplified a bit by the changes to the JSON format, but
  its behavior is basically the same.

* Man pages and tests changed to reflect the above changes.

Note that, as implied above, this series relies on Austin's rewrite of the
show JSON formatter [2]. I should probably add the reply JSON format to
the devel/schemata file, but I'll leave that for another patch (perhaps
in a series with a news entry if this is deemed ready to push).

Thanks to everyone for the replies on previous versions, and please let
me know if you have any comments. Also, it would be great if others can
test this out on some interesting messages, as my email collection is
not very esoteric.

[1] id:"1328746916-25447-1-git-send-email-awg+notmuch@xvx.ca"
[2] id:"1329240823-7856-1-git-send-email-amdragon@mit.edu"

Adam Wolfe Gordon (4):
  test: Add broken test for the new JSON reply format.
  reply: Add a JSON reply format.
  man: Update notmuch-reply man page for JSON format.
  emacs: Use the new JSON reply format and message-cite-original

 emacs/notmuch-lib.el     |   39 +++++++++++
 emacs/notmuch-mua.el     |  123 +++++++++++++++++++++++++----------
 emacs/notmuch-show.el    |   24 +------
 man/man1/notmuch-reply.1 |    5 ++
 notmuch-client.h         |    3 +
 notmuch-reply.c          |  164 ++++++++++++++++++++++++++++++++++------------
 notmuch-show.c           |    2 +-
 test/emacs               |  101 ++++++++++++++++++++++++++++-
 test/multipart           |   51 +++++++++++++++++++++++++++++++++++++++++++++++++++
 9 files changed, 365 insertions(+), 103 deletions(-)

-- 
1.7.5.4

[PATCH v5 1/4] test: Add broken test for the new JSON reply format.

[Adam Wolfe Gordon]

---
 test/multipart |   51 +++++++++++++++++++++++++++++++++++++++++++++++++++
 1 files changed, 51 insertions(+), 0 deletions(-)

diff --git a/test/multipart b/test/multipart
index a3036b4..9651568 100755
--- a/test/multipart
+++ b/test/multipart
@@ -589,6 +589,57 @@ Non-text part: text/html
 EOF
 test_expect_equal_file OUTPUT EXPECTED
 
+test_begin_subtest "'notmuch reply' to a multipart message with json format"
+notmuch reply --format=json 'id:87liy5ap00.fsf@yoom.home.cworth.org' | notmuch_json_show_sanitize >OUTPUT
+cat <<EOF >EXPECTED
+{"reply-headers": {"from": "Notmuch Test Suite <test_suite@notmuchmail.org>",
+ "to": "Carl Worth <cworth@cworth.org>,
+ cworth@cworth.org",
+ "subject": "Re: Multipart message",
+ "in-reply-to": "<87liy5ap00.fsf@yoom.home.cworth.org>",
+ "references": " <87liy5ap00.fsf@yoom.home.cworth.org>"},
+ "original": {"id": "XXXXX",
+ "match": false,
+ "filename": "YYYYY",
+ "timestamp": 978709437,
+ "date_relative": "2001-01-05",
+ "tags": ["attachment","inbox","signed","unread"],
+ "headers": {"Subject": "Multipart message",
+ "From": "Carl Worth <cworth@cworth.org>",
+ "To": "cworth@cworth.org",
+ "Date": "Fri,
+ 05 Jan 2001 15:43:57 +0000"},
+ "body": [{"id": 1,
+ "content-type": "multipart/signed",
+ "content": [{"id": 2,
+ "content-type": "multipart/mixed",
+ "content": [{"id": 3,
+ "content-type": "message/rfc822",
+ "content": [{"headers": {"Subject": "html message",
+ "From": "Carl Worth <cworth@cworth.org>",
+ "To": "cworth@cworth.org",
+ "Date": "Fri,
+ 05 Jan 2001 15:42:57 +0000"},
+ "body": [{"id": 4,
+ "content-type": "multipart/alternative",
+ "content": [{"id": 5,
+ "content-type": "text/html"},
+ {"id": 6,
+ "content-type": "text/plain",
+ "content": "This is an embedded message,
+ with a multipart/alternative part.\n"}]}]}]},
+ {"id": 7,
+ "content-type": "text/plain",
+ "filename": "YYYYY",
+ "content": "This is a text attachment.\n"},
+ {"id": 8,
+ "content-type": "text/plain",
+ "content": "And this message is signed.\n\n-Carl\n"}]},
+ {"id": 9,
+ "content-type": "application/pgp-signature"}]}]}}
+EOF
+test_expect_equal_file OUTPUT EXPECTED
+
 test_begin_subtest "'notmuch show --part' does not corrupt a part with CRLF pair"
 notmuch show --format=raw --part=3 id:base64-part-with-crlf > crlf.out
 echo -n -e "\xEF\x0D\x0A" > crlf.expected
-- 
1.7.5.4

[PATCH v5 2/4] reply: Add a JSON reply format.

[Adam Wolfe Gordon]

This new JSON format for replies includes headers generated for a
reply message as well as the headers of the original message.  Using
this data, a client can intelligently create a reply. For example, the
emacs client will be able to create replies with quoted HTML parts by
parsing the HTML parts.

Reply now enforces that only one message is returned, as the semantics
of replying to multiple messages are not well-defined.
---
 notmuch-client.h |    3 +
 notmuch-reply.c  |  164 ++++++++++++++++++++++++++++++++++++++++--------------
 notmuch-show.c   |    2 +-
 3 files changed, 125 insertions(+), 44 deletions(-)

diff --git a/notmuch-client.h b/notmuch-client.h
index 60828aa..d28ea07 100644
--- a/notmuch-client.h
+++ b/notmuch-client.h
@@ -344,6 +344,9 @@ typedef struct mime_node {
     int next_part_num;
 } mime_node_t;
 
+void
+format_part_json (const void *ctx, mime_node_t *node, notmuch_bool_t first);
+
 /* Construct a new MIME node pointing to the root message part of
  * message.  If cryptoctx is non-NULL, it will be used to verify
  * signatures on any child parts.  If decrypt is true, then cryptoctx
diff --git a/notmuch-reply.c b/notmuch-reply.c
index 6b244e6..979ad86 100644
--- a/notmuch-reply.c
+++ b/notmuch-reply.c
@@ -505,6 +505,61 @@ guess_from_received_header (notmuch_config_t *config, notmuch_message_t *message
     return NULL;
 }
 
+static GMimeMessage *
+create_reply_message(void *ctx,
+		     notmuch_config_t *config,
+		     notmuch_message_t *message,
+		     notmuch_bool_t reply_all)
+{
+    const char *subject, *from_addr = NULL;
+    const char *in_reply_to, *orig_references, *references;
+
+    /* The 1 means we want headers in a "pretty" order. */
+    GMimeMessage *reply = g_mime_message_new (1);
+    if (reply == NULL) {
+	fprintf (stderr, "Out of memory\n");
+	return NULL;
+    }
+
+    subject = notmuch_message_get_header (message, "subject");
+    if (subject) {
+	if (strncasecmp (subject, "Re:", 3))
+	    subject = talloc_asprintf (ctx, "Re: %s", subject);
+	g_mime_message_set_subject (reply, subject);
+    }
+
+    from_addr = add_recipients_from_message (reply, config,
+					     message, reply_all);
+
+    if (from_addr == NULL)
+	from_addr = guess_from_received_header (config, message);
+
+    if (from_addr == NULL)
+	from_addr = notmuch_config_get_user_primary_email (config);
+
+    from_addr = talloc_asprintf (ctx, "%s <%s>",
+				 notmuch_config_get_user_name (config),
+				 from_addr);
+    g_mime_object_set_header (GMIME_OBJECT (reply),
+			      "From", from_addr);
+
+    in_reply_to = talloc_asprintf (ctx, "<%s>",
+				   notmuch_message_get_message_id (message));
+
+    g_mime_object_set_header (GMIME_OBJECT (reply),
+			      "In-Reply-To", in_reply_to);
+
+    orig_references = notmuch_message_get_header (message, "references");
+    references = talloc_asprintf (ctx, "%s%s%s",
+				  orig_references ? orig_references : "",
+				  orig_references ? " " : "",
+				  in_reply_to);
+    g_mime_object_set_header (GMIME_OBJECT (reply),
+			      "References", references);
+
+    return reply;
+}
+
 static int
 notmuch_reply_format_default(void *ctx,
 			     notmuch_config_t *config,
@@ -515,8 +570,6 @@ notmuch_reply_format_default(void *ctx,
     GMimeMessage *reply;
     notmuch_messages_t *messages;
     notmuch_message_t *message;
-    const char *subject, *from_addr = NULL;
-    const char *in_reply_to, *orig_references, *references;
     const notmuch_show_format_t *format = &format_reply;
 
     for (messages = notmuch_query_search_messages (query);
@@ -525,48 +578,10 @@ notmuch_reply_format_default(void *ctx,
     {
 	message = notmuch_messages_get (messages);
 
-	/* The 1 means we want headers in a "pretty" order. */
-	reply = g_mime_message_new (1);
-	if (reply == NULL) {
-	    fprintf (stderr, "Out of memory\n");
-	    return 1;
-	}
-
-	subject = notmuch_message_get_header (message, "subject");
-	if (subject) {
-	    if (strncasecmp (subject, "Re:", 3))
-		subject = talloc_asprintf (ctx, "Re: %s", subject);
-	    g_mime_message_set_subject (reply, subject);
-	}
-
-	from_addr = add_recipients_from_message (reply, config, message,
-						 reply_all);
-
-	if (from_addr == NULL)
-	    from_addr = guess_from_received_header (config, message);
-
-	if (from_addr == NULL)
-	    from_addr = notmuch_config_get_user_primary_email (config);
-
-	from_addr = talloc_asprintf (ctx, "%s <%s>",
-				     notmuch_config_get_user_name (config),
-				     from_addr);
-	g_mime_object_set_header (GMIME_OBJECT (reply),
-				  "From", from_addr);
-
-	in_reply_to = talloc_asprintf (ctx, "<%s>",
-			     notmuch_message_get_message_id (message));
-
-	g_mime_object_set_header (GMIME_OBJECT (reply),
-				  "In-Reply-To", in_reply_to);
+	reply = create_reply_message (ctx, config, message, reply_all);
 
-	orig_references = notmuch_message_get_header (message, "references");
-	references = talloc_asprintf (ctx, "%s%s%s",
-				      orig_references ? orig_references : "",
-				      orig_references ? " " : "",
-				      in_reply_to);
-	g_mime_object_set_header (GMIME_OBJECT (reply),
-				  "References", references);
+	if (!reply)
+	    continue;
 
 	show_reply_headers (reply);
 
@@ -584,6 +599,65 @@ notmuch_reply_format_default(void *ctx,
     return 0;
 }
 
+static int
+notmuch_reply_format_json(void *ctx,
+			  notmuch_config_t *config,
+			  notmuch_query_t *query,
+			  notmuch_show_params_t *params,
+			  notmuch_bool_t reply_all)
+{
+    GMimeMessage *reply;
+    notmuch_messages_t *messages;
+    notmuch_message_t *message;
+    mime_node_t *node;
+
+    const char *reply_headers[] = {"from", "to", "subject", "in-reply-to", "references"};
+    unsigned int hidx;
+
+    if (notmuch_query_count_messages (query) != 1) {
+	fprintf (stderr, "Error: search term did not match precisely one message.\n");
+	return 1;
+    }
+
+    messages = notmuch_query_search_messages (query);
+    message = notmuch_messages_get (messages);
+    if (mime_node_open (ctx, message, params->cryptoctx, params->decrypt,
+			&node) != NOTMUCH_STATUS_SUCCESS)
+	return 1;
+
+    reply = create_reply_message (ctx, config, message, reply_all);
+    if (!reply)
+	return 1;
+
+    /* Start a reply object */
+    printf ("{\"reply-headers\": {");
+
+    for (hidx = 0; hidx < ARRAY_SIZE (reply_headers); hidx++) {
+	if (hidx)
+	    printf (", ");
+
+	printf ("%s: %s", json_quote_str (ctx, reply_headers[hidx]),
+		json_quote_str (ctx, g_mime_object_get_header (GMIME_OBJECT (reply), reply_headers[hidx])));
+    }
+
+    g_object_unref (G_OBJECT (reply));
+    reply = NULL;
+
+    /* Done the headers for the reply, which has no body */
+    printf ("}");
+
+    /* Start the original */
+    printf (", \"original\": ");
+
+    format_part_json (ctx, node, TRUE);
+
+    /* End */
+    printf ("}\n");
+    notmuch_message_destroy (message);
+
+    return 0;
+}
+
 /* This format is currently tuned for a git send-email --notmuch hook */
 static int
 notmuch_reply_format_headers_only(void *ctx,
@@ -646,6 +720,7 @@ notmuch_reply_format_headers_only(void *ctx,
 
 enum {
     FORMAT_DEFAULT,
+    FORMAT_JSON,
     FORMAT_HEADERS_ONLY,
 };
 
@@ -665,6 +740,7 @@ notmuch_reply_command (void *ctx, int argc, char *argv[])
     notmuch_opt_desc_t options[] = {
 	{ NOTMUCH_OPT_KEYWORD, &format, "format", 'f',
 	  (notmuch_keyword_t []){ { "default", FORMAT_DEFAULT },
+				  { "json", FORMAT_JSON },
 				  { "headers-only", FORMAT_HEADERS_ONLY },
 				  { 0, 0 } } },
 	{ NOTMUCH_OPT_KEYWORD, &reply_all, "reply-to", 'r',
@@ -683,6 +759,8 @@ notmuch_reply_command (void *ctx, int argc, char *argv[])
 
     if (format == FORMAT_HEADERS_ONLY)
 	reply_format_func = notmuch_reply_format_headers_only;
+    else if (format == FORMAT_JSON)
+	reply_format_func = notmuch_reply_format_json;
     else
 	reply_format_func = notmuch_reply_format_default;
 
diff --git a/notmuch-show.c b/notmuch-show.c
index 6a171a4..c570a16 100644
--- a/notmuch-show.c
+++ b/notmuch-show.c
@@ -652,7 +652,7 @@ format_part_text (const void *ctx, mime_node_t *node,
     printf ("\f%s}\n", part_type);
 }
 
-static void
+void
 format_part_json (const void *ctx, mime_node_t *node, notmuch_bool_t first)
 {
     /* Any changes to the JSON format should be reflected in the file
-- 
1.7.5.4

[PATCH v5 3/4] man: Update notmuch-reply man page for JSON format.

[Adam Wolfe Gordon]

---
 man/man1/notmuch-reply.1 |    5 +++++
 1 files changed, 5 insertions(+), 0 deletions(-)

diff --git a/man/man1/notmuch-reply.1 b/man/man1/notmuch-reply.1
index 5160ece..307abee 100644
--- a/man/man1/notmuch-reply.1
+++ b/man/man1/notmuch-reply.1
@@ -43,6 +43,11 @@ include
 .BR default
 Includes subject and quoted message body.
 .TP
+.BR json
+Produces JSON output containing headers for a reply message and the
+contents of the original message. This output can be used by a client
+to create a reply message intelligently.
+.TP
 .BR headers\-only
 Only produces In\-Reply\-To, References, To, Cc, and Bcc headers.
 .RE
-- 
1.7.5.4

[PATCH v5 4/4] emacs: Use the new JSON reply format and message-cite-original

[Adam Wolfe Gordon]

Using the new JSON reply format allows emacs to quote HTML parts
nicely by using mm-display-part to turn them into displayable text,
then quoting them with message-cite-original. This is very useful for
users who regularly receive HTML-only email.

Use message-mode's message-cite-original function to create the
quoted body for reply messages. In order to make this act like the
existing notmuch defaults, you will need to set the following in
your emacs configuration:

message-citation-line-format "On %a, %d %b %Y, %f wrote:"
message-citation-line-function 'message-insert-formatted-citation-line

The test has been updated to reflect the (ugly) emacs default.
---
 emacs/notmuch-lib.el  |   39 +++++++++++++++
 emacs/notmuch-mua.el  |  123 +++++++++++++++++++++++++++++++++++--------------
 emacs/notmuch-show.el |   24 +---------
 test/emacs            |  101 +++++++++++++++++++++++++++++++++++++++-
 4 files changed, 228 insertions(+), 59 deletions(-)

diff --git a/emacs/notmuch-lib.el b/emacs/notmuch-lib.el
index d315f76..3fc7aff 100644
--- a/emacs/notmuch-lib.el
+++ b/emacs/notmuch-lib.el
@@ -21,6 +21,8 @@
 
 ;; This is an part of an emacs-based interface to the notmuch mail system.
 
+(eval-when-compile (require 'cl))
+
 (defvar notmuch-command "notmuch"
   "Command to run the notmuch binary.")
 
@@ -173,6 +175,43 @@ the user hasn't set this variable with the old or new value."
   (list 'when (< emacs-major-version 23)
 	form))
 
+(defun notmuch-split-content-type (content-type)
+  "Split content/type into 'content' and 'type'"
+  (split-string content-type "/"))
+
+(defun notmuch-match-content-type (t1 t2)
+  "Return t if t1 and t2 are matching content types, taking wildcards into account"
+  (let ((st1 (notmuch-split-content-type t1))
+	(st2 (notmuch-split-content-type t2)))
+    (if (or (string= (cadr st1) "*")
+	    (string= (cadr st2) "*"))
+	(string= (car st1) (car st2))
+      (string= t1 t2))))
+
+(defvar notmuch-multipart/alternative-discouraged
+  '(
+    ;; Avoid HTML parts.
+    "text/html"
+    ;; multipart/related usually contain a text/html part and some associated graphics.
+    "multipart/related"
+    ))
+
+(defun notmuch-multipart/alternative-choose (types)
+  "Return a list of preferred types from the given list of types"
+  ;; Based on `mm-preferred-alternative-precedence'.
+  (let ((seq types))
+    (dolist (pref (reverse notmuch-multipart/alternative-discouraged))
+      (dolist (elem (copy-sequence seq))
+	(when (string-match pref elem)
+	  (setq seq (nconc (delete elem seq) (list elem))))))
+    seq))
+
+(defun notmuch-parts-filter-by-type (parts type)
+  "Given a vector of message parts, return a vector containing the ones matching the given type."
+  (loop for part across parts
+	if (notmuch-match-content-type (cdr (assq 'content-type part)) type)
+	vconcat (list part)))
+
 ;; Compatibility functions for versions of emacs before emacs 23.
 ;;
 ;; Both functions here were copied from emacs 23 with the following copyright:
diff --git a/emacs/notmuch-mua.el b/emacs/notmuch-mua.el
index 4be7c13..371993f 100644
--- a/emacs/notmuch-mua.el
+++ b/emacs/notmuch-mua.el
@@ -19,11 +19,15 @@
 ;;
 ;; Authors: David Edmondson <dme@dme.org>
 
+(require 'json)
 (require 'message)
+(require 'format-spec)
 
 (require 'notmuch-lib)
 (require 'notmuch-address)
 
+(eval-when-compile (require 'cl))
+
 ;;
 
 (defcustom notmuch-mua-send-hook '(notmuch-mua-message-send-hook)
@@ -72,56 +76,105 @@ list."
 	    (push header message-hidden-headers)))
 	notmuch-mua-hidden-headers))
 
+(defun notmuch-mua-get-displayed-part (part query-string)
+  (with-temp-buffer
+    (if (assq 'content part)
+	(insert (cdr (assq 'content part)))
+      (call-process notmuch-command nil t nil "show" "--format=raw"
+		    (format "--part=%s" (cdr (assq 'id part)))
+		    query-string))
+
+    (let ((handle (mm-make-handle (current-buffer) (list (cdr (assq 'content-type part)))))
+	  (end-of-orig (point-max)))
+      (mm-display-part handle)
+      (delete-region (point-min) end-of-orig)
+      (buffer-substring (point-min) (point-max)))))
+
+(defun notmuch-mua-multipart/*-to-list (parts)
+  (loop for part across parts
+	collect (cdr (assq 'content-type part))))
+
+(defun notmuch-mua-get-quotable-parts (parts)
+  (loop for part across parts
+	if (notmuch-match-content-type (cdr (assq 'content-type part)) "multipart/alternative")
+	  append (let* ((subparts (cdr (assq 'content part)))
+			(types (notmuch-mua-multipart/*-to-list subparts))
+			(chosen-type (car (notmuch-multipart/alternative-choose types))))
+		   (notmuch-mua-get-quotable-parts (notmuch-parts-filter-by-type subparts chosen-type)))
+	else if (notmuch-match-content-type (cdr (assq 'content-type part)) "multipart/*")
+	  append (notmuch-mua-get-quotable-parts (cdr (assq 'content part)))
+	else if (notmuch-match-content-type (cdr (assq 'content-type part)) "text/*")
+	  collect part))
+
 (defun notmuch-mua-reply (query-string &optional sender reply-all)
-  (let (headers
-	body
-	(args '("reply")))
-    (if notmuch-show-process-crypto
-	(setq args (append args '("--decrypt"))))
+  (let ((args '("reply" "--format=json"))
+	reply
+	original)
+    (when notmuch-show-process-crypto
+      (setq args (append args '("--decrypt"))))
+
     (if reply-all
 	(setq args (append args '("--reply-to=all")))
       (setq args (append args '("--reply-to=sender"))))
     (setq args (append args (list query-string)))
-    ;; This make assumptions about the output of `notmuch reply', but
-    ;; really only that the headers come first followed by a blank
-    ;; line and then the body.
+
+    ;; Get the reply object as JSON, and parse it into an elisp object.
     (with-temp-buffer
       (apply 'call-process (append (list notmuch-command nil (list t t) nil) args))
       (goto-char (point-min))
-      (if (re-search-forward "^$" nil t)
-	  (save-excursion
-	    (save-restriction
-	      (narrow-to-region (point-min) (point))
-	      (goto-char (point-min))
-	      (setq headers (mail-header-extract)))))
-      (forward-line 1)
-      (setq body (buffer-substring (point) (point-max))))
-    ;; If sender is non-nil, set the From: header to its value.
-    (when sender
-      (mail-header-set 'from sender headers))
-    (let
-	;; Overlay the composition window on that being used to read
-	;; the original message.
-	((same-window-regexps '("\\*mail .*")))
-      (notmuch-mua-mail (mail-header 'to headers)
-			(mail-header 'subject headers)
-			(message-headers-to-generate headers t '(to subject))))
-    ;; insert the message body - but put it in front of the signature
-    ;; if one is present
-    (goto-char (point-max))
-    (if (re-search-backward message-signature-separator nil t)
+      (setq reply (json-read)))
+
+    ;; Extract the original message to simplify the following code.
+    (setq original (cdr (assq 'original reply)))
+
+    ;; Extract the headers of both the reply and the original message.
+    (let* ((original-headers (cdr (assq 'headers original)))
+	   (reply-headers (cdr (assq 'reply-headers reply))))
+
+      ;; If sender is non-nil, set the From: header to its value.
+      (when sender
+	(mail-header-set 'from sender reply-headers))
+      (let
+	  ;; Overlay the composition window on that being used to read
+	  ;; the original message.
+	  ((same-window-regexps '("\\*mail .*")))
+	(notmuch-mua-mail (mail-header 'to reply-headers)
+			  (mail-header 'subject reply-headers)
+			  (message-headers-to-generate reply-headers t '(to subject))))
+      ;; Insert the message body - but put it in front of the signature
+      ;; if one is present
+      (goto-char (point-max))
+      (if (re-search-backward message-signature-separator nil t)
 	  (forward-line -1)
-      (goto-char (point-max)))
-    (insert body)
-    (push-mark))
-  (set-buffer-modified-p nil)
+	(goto-char (point-max)))
+
+      (let ((from (cdr (assq 'From original-headers)))
+	    (date (cdr (assq 'Date original-headers)))
+	    (start (point)))
+
+	(insert "From: " from "\n")
+	(insert "Date: " date "\n\n")
+
+	;; Get the parts of the original message that should be quoted; this includes
+	;; all the text parts, except the non-preferred ones in a multipart/alternative.
+	(let ((quotable-parts (notmuch-mua-get-quotable-parts (cdr (assq 'body original)))))
+	  (mapc (lambda (part)
+		  (insert (notmuch-mua-get-displayed-part part query-string)))
+		quotable-parts))
+
+	(push-mark)
+	(goto-char start)
+	;; Quote the original message according to the user's configured style.
+	(message-cite-original))))
 
+  (push-mark)
   (message-goto-body)
   ;; Original message may contain (malicious) MML tags.  We must
   ;; properly quote them in the reply.  Note that using `point-max'
   ;; instead of `mark' here is wrong.  The buffer may include user's
   ;; signature which should not be MML-quoted.
-  (mml-quote-region (point) (mark)))
+  (mml-quote-region (point) (mark))
+  (set-buffer-modified-p nil))
 
 (defun notmuch-mua-forward-message ()
   (message-forward)
diff --git a/emacs/notmuch-show.el b/emacs/notmuch-show.el
index 43408d9..90cdd38 100644
--- a/emacs/notmuch-show.el
+++ b/emacs/notmuch-show.el
@@ -513,30 +513,13 @@ current buffer, if possible."
 	    (mm-display-part handle)
 	    t))))))
 
-(defvar notmuch-show-multipart/alternative-discouraged
-  '(
-    ;; Avoid HTML parts.
-    "text/html"
-    ;; multipart/related usually contain a text/html part and some associated graphics.
-    "multipart/related"
-    ))
-
 (defun notmuch-show-multipart/*-to-list (part)
   (mapcar (lambda (inner-part) (plist-get inner-part :content-type))
 	  (plist-get part :content)))
 
-(defun notmuch-show-multipart/alternative-choose (types)
-  ;; Based on `mm-preferred-alternative-precedence'.
-  (let ((seq types))
-    (dolist (pref (reverse notmuch-show-multipart/alternative-discouraged))
-      (dolist (elem (copy-sequence seq))
-	(when (string-match pref elem)
-	  (setq seq (nconc (delete elem seq) (list elem))))))
-    seq))
-
 (defun notmuch-show-insert-part-multipart/alternative (msg part content-type nth depth declared-type)
   (notmuch-show-insert-part-header nth declared-type content-type nil)
-  (let ((chosen-type (car (notmuch-show-multipart/alternative-choose (notmuch-show-multipart/*-to-list part))))
+  (let ((chosen-type (car (notmuch-multipart/alternative-choose (notmuch-show-multipart/*-to-list part))))
 	(inner-parts (plist-get part :content))
 	(start (point)))
     ;; This inserts all parts of the chosen type rather than just one,
@@ -775,9 +758,6 @@ current buffer, if possible."
 
 ;; Functions for determining how to handle MIME parts.
 
-(defun notmuch-show-split-content-type (content-type)
-  (split-string content-type "/"))
-
 (defun notmuch-show-handlers-for (content-type)
   "Return a list of content handlers for a part of type CONTENT-TYPE."
   (let (result)
@@ -788,7 +768,7 @@ current buffer, if possible."
 	  (list (intern (concat "notmuch-show-insert-part-*/*"))
 		(intern (concat
 			 "notmuch-show-insert-part-"
-			 (car (notmuch-show-split-content-type content-type))
+			 (car (notmuch-split-content-type content-type))
 			 "/*"))
 		(intern (concat "notmuch-show-insert-part-" content-type))))
     result))
diff --git a/test/emacs b/test/emacs
index d4a8d30..a6786d4 100755
--- a/test/emacs
+++ b/test/emacs
@@ -268,11 +268,107 @@ Subject: Re: Testing message sent via SMTP
 In-Reply-To: <XXX>
 Fcc: $(pwd)/mail/sent
 --text follows this line--
-On 01 Jan 2000 12:00:00 -0000, Notmuch Test Suite <test_suite@notmuchmail.org> wrote:
+Notmuch Test Suite <test_suite@notmuchmail.org> writes:
+
 > This is a test that messages are sent via SMTP
 EOF
 test_expect_equal_file OUTPUT EXPECTED
 
+test_begin_subtest "Reply within emacs to a multipart/mixed message"
+test_emacs '(notmuch-show "id:20091118002059.067214ed@hikari")
+		(notmuch-show-reply)
+		(test-output)'
+cat <<EOF >EXPECTED
+From: Notmuch Test Suite <test_suite@notmuchmail.org>
+To: Adrian Perez de Castro <aperez@igalia.com>, notmuch@notmuchmail.org
+Subject: Re: [notmuch] Introducing myself
+In-Reply-To: <20091118002059.067214ed@hikari>
+Fcc: ${MAIL_DIR}/sent
+--text follows this line--
+Adrian Perez de Castro <aperez@igalia.com> writes:
+
+> Hello to all,
+>
+> I have just heard about Not Much today in some random Linux-related news
+> site (LWN?), my name is Adrian Perez and I work as systems administrator
+> (although I can do some code as well :P). I have always thought that the
+> ideas behind Sup were great, but after some time using it, I got tired of
+> the oddities that it has. I also do not like doing things like having to
+> install Ruby just for reading and sorting mails. Some time ago I thought
+> about doing something like Not Much and in fact I played a bit with the
+> Python+Xapian and the Python+Whoosh combinations, because I find relaxing
+> to code things in Python when I am not working and also it is installed
+> by default on most distribution. I got to have some mailboxes indexed and
+> basic searching working a couple of months ago. Lately I have been very
+> busy and had no time for coding, and them... boom! Not Much appears -- and
+> it is almost exactly what I was trying to do, but faster. I have been
+> playing a bit with Not Much today, and I think it has potential.
+>
+> Also, I would like to share one idea I had in mind, that you might find
+> interesting: One thing I have found very annoying is having to re-tag my
+> mail when the indexes get b0rked (it happened a couple of times to me while
+> using Sup), so I was planning to mails as read/unread and adding the tags
+> not just to the index, but to the mail text itself, e.g. by adding a
+> "X-Tags" header field or by reusing the "Keywords" one. This way, the index
+> could be totally recreated by re-reading the mail directories, and this
+> would also allow to a tools like OfflineIMAP [1] to get the mails into a
+> local maildir, tagging and indexing the mails with the e-mail reader and
+> then syncing back the messages with the "X-Tags" header to the IMAP server.
+> This would allow to use the mail reader from a different computer and still
+> have everything tagged finely.
+>
+> Best regards,
+>
+>
+> ---
+> [1] http://software.complete.org/software/projects/show/offlineimap
+>
+> -- 
+> Adrian Perez de Castro <aperez@igalia.com>
+> Igalia - Free Software Engineering
+> _______________________________________________
+> notmuch mailing list
+> notmuch@notmuchmail.org
+> http://notmuchmail.org/mailman/listinfo/notmuch
+EOF
+test_expect_equal_file OUTPUT EXPECTED
+
+test_begin_subtest "Reply within emacs to a multipart/alternative message"
+test_emacs '(notmuch-show "id:cf0c4d610911171136h1713aa59w9cf9aa31f052ad0a@mail.gmail.com")
+		(notmuch-show-reply)
+		(test-output)'
+cat <<EOF >EXPECTED
+From: Notmuch Test Suite <test_suite@notmuchmail.org>
+To: Alex Botero-Lowry <alex.boterolowry@gmail.com>, notmuch@notmuchmail.org
+Subject: Re: [notmuch] preliminary FreeBSD support
+In-Reply-To: <cf0c4d610911171136h1713aa59w9cf9aa31f052ad0a@mail.gmail.com>
+Fcc: ${MAIL_DIR}/sent
+--text follows this line--
+Alex Botero-Lowry <alex.boterolowry@gmail.com> writes:
+
+> I saw the announcement this morning, and was very excited, as I had been
+> hoping sup would be turned into a library,
+> since I like the concept more than the UI (I'd rather an emacs interface).
+>
+> I did a preliminary compile which worked out fine, but
+> sysconf(_SC_SC_GETPW_R_SIZE_MAX) returns -1 on
+> FreeBSD, so notmuch_config_open segfaulted.
+>
+> Attached is a patch that supplies a default buffer size of 64 in cases where
+> -1 is returned.
+>
+> http://www.opengroup.org/austin/docs/austin_328.txt - seems to indicate this
+> is acceptable behavior,
+> and http://mail-index.netbsd.org/pkgsrc-bugs/2006/06/07/msg016808.htmlspecifically
+> uses 64 as the
+> buffer size.
+> _______________________________________________
+> notmuch mailing list
+> notmuch@notmuchmail.org
+> http://notmuchmail.org/mailman/listinfo/notmuch
+EOF
+test_expect_equal_file OUTPUT EXPECTED
+
 test_begin_subtest "Quote MML tags in reply"
 message_id='test-emacs-mml-quoting@message.id'
 add_message [id]="$message_id" \
@@ -288,7 +384,8 @@ Subject: Re: Quote MML tags in reply
 In-Reply-To: <test-emacs-mml-quoting@message.id>
 Fcc: ${MAIL_DIR}/sent
 --text follows this line--
-On Fri, 05 Jan 2001 15:43:57 +0000, Notmuch Test Suite <test_suite@notmuchmail.org> wrote:
+Notmuch Test Suite <test_suite@notmuchmail.org> writes:
+
 > <#!part disposition=inline>
 EOF
 test_expect_equal_file OUTPUT EXPECTED
-- 
1.7.5.4

[PATCH v5.1 1/4] test: Add broken test for the new JSON reply format.

[Adam Wolfe Gordon]

---

Adjusted the header display to be consistent with the other JSON formats.

 test/multipart |   51 +++++++++++++++++++++++++++++++++++++++++++++++++++
 1 files changed, 51 insertions(+), 0 deletions(-)

diff --git a/test/multipart b/test/multipart
index a3036b4..e7abcc2 100755
--- a/test/multipart
+++ b/test/multipart
@@ -589,6 +589,57 @@ Non-text part: text/html
 EOF
 test_expect_equal_file OUTPUT EXPECTED
 
+test_begin_subtest "'notmuch reply' to a multipart message with json format"
+notmuch reply --format=json 'id:87liy5ap00.fsf@yoom.home.cworth.org' | notmuch_json_show_sanitize >OUTPUT
+cat <<EOF >EXPECTED
+{"reply-headers": {"Subject": "Re: Multipart message",
+ "From": "Notmuch Test Suite <test_suite@notmuchmail.org>",
+ "To": "Carl Worth <cworth@cworth.org>,
+ cworth@cworth.org",
+ "In-reply-to": "<87liy5ap00.fsf@yoom.home.cworth.org>",
+ "References": " <87liy5ap00.fsf@yoom.home.cworth.org>"},
+ "original": {"id": "XXXXX",
+ "match": false,
+ "filename": "YYYYY",
+ "timestamp": 978709437,
+ "date_relative": "2001-01-05",
+ "tags": ["attachment","inbox","signed","unread"],
+ "headers": {"Subject": "Multipart message",
+ "From": "Carl Worth <cworth@cworth.org>",
+ "To": "cworth@cworth.org",
+ "Date": "Fri,
+ 05 Jan 2001 15:43:57 +0000"},
+ "body": [{"id": 1,
+ "content-type": "multipart/signed",
+ "content": [{"id": 2,
+ "content-type": "multipart/mixed",
+ "content": [{"id": 3,
+ "content-type": "message/rfc822",
+ "content": [{"headers": {"Subject": "html message",
+ "From": "Carl Worth <cworth@cworth.org>",
+ "To": "cworth@cworth.org",
+ "Date": "Fri,
+ 05 Jan 2001 15:42:57 +0000"},
+ "body": [{"id": 4,
+ "content-type": "multipart/alternative",
+ "content": [{"id": 5,
+ "content-type": "text/html"},
+ {"id": 6,
+ "content-type": "text/plain",
+ "content": "This is an embedded message,
+ with a multipart/alternative part.\n"}]}]}]},
+ {"id": 7,
+ "content-type": "text/plain",
+ "filename": "YYYYY",
+ "content": "This is a text attachment.\n"},
+ {"id": 8,
+ "content-type": "text/plain",
+ "content": "And this message is signed.\n\n-Carl\n"}]},
+ {"id": 9,
+ "content-type": "application/pgp-signature"}]}]}}
+EOF
+test_expect_equal_file OUTPUT EXPECTED
+
 test_begin_subtest "'notmuch show --part' does not corrupt a part with CRLF pair"
 notmuch show --format=raw --part=3 id:base64-part-with-crlf > crlf.out
 echo -n -e "\xEF\x0D\x0A" > crlf.expected
-- 
1.7.5.4

[PATCH v5.1 2/4] reply: Add a JSON reply format.

[Adam Wolfe Gordon]

This new JSON format for replies includes headers generated for a
reply message as well as the headers of the original message.  Using
this data, a client can intelligently create a reply. For example, the
emacs client will be able to create replies with quoted HTML parts by
parsing the HTML parts.

Reply now enforces that only one message is returned, as the semantics
of replying to multiple messages are not well-defined.
---

Adjusted the header display to be consistent with the other JSON formats.

 notmuch-client.h |    3 +
 notmuch-reply.c  |  188 +++++++++++++++++++++++++++++++++++++++++------------
 notmuch-show.c   |    2 +-
 3 files changed, 149 insertions(+), 44 deletions(-)

diff --git a/notmuch-client.h b/notmuch-client.h
index 60828aa..d28ea07 100644
--- a/notmuch-client.h
+++ b/notmuch-client.h
@@ -344,6 +344,9 @@ typedef struct mime_node {
     int next_part_num;
 } mime_node_t;
 
+void
+format_part_json (const void *ctx, mime_node_t *node, notmuch_bool_t first);
+
 /* Construct a new MIME node pointing to the root message part of
  * message.  If cryptoctx is non-NULL, it will be used to verify
  * signatures on any child parts.  If decrypt is true, then cryptoctx
diff --git a/notmuch-reply.c b/notmuch-reply.c
index 6b244e6..76995b1 100644
--- a/notmuch-reply.c
+++ b/notmuch-reply.c
@@ -505,6 +505,61 @@ guess_from_received_header (notmuch_config_t *config, notmuch_message_t *message
     return NULL;
 }
 
+static GMimeMessage *
+create_reply_message(void *ctx,
+		     notmuch_config_t *config,
+		     notmuch_message_t *message,
+		     notmuch_bool_t reply_all)
+{
+    const char *subject, *from_addr = NULL;
+    const char *in_reply_to, *orig_references, *references;
+
+    /* The 1 means we want headers in a "pretty" order. */
+    GMimeMessage *reply = g_mime_message_new (1);
+    if (reply == NULL) {
+	fprintf (stderr, "Out of memory\n");
+	return NULL;
+    }
+
+    subject = notmuch_message_get_header (message, "subject");
+    if (subject) {
+	if (strncasecmp (subject, "Re:", 3))
+	    subject = talloc_asprintf (ctx, "Re: %s", subject);
+	g_mime_message_set_subject (reply, subject);
+    }
+
+    from_addr = add_recipients_from_message (reply, config,
+					     message, reply_all);
+
+    if (from_addr == NULL)
+	from_addr = guess_from_received_header (config, message);
+
+    if (from_addr == NULL)
+	from_addr = notmuch_config_get_user_primary_email (config);
+
+    from_addr = talloc_asprintf (ctx, "%s <%s>",
+				 notmuch_config_get_user_name (config),
+				 from_addr);
+    g_mime_object_set_header (GMIME_OBJECT (reply),
+			      "From", from_addr);
+
+    in_reply_to = talloc_asprintf (ctx, "<%s>",
+				   notmuch_message_get_message_id (message));
+
+    g_mime_object_set_header (GMIME_OBJECT (reply),
+			      "In-Reply-To", in_reply_to);
+
+    orig_references = notmuch_message_get_header (message, "references");
+    references = talloc_asprintf (ctx, "%s%s%s",
+				  orig_references ? orig_references : "",
+				  orig_references ? " " : "",
+				  in_reply_to);
+    g_mime_object_set_header (GMIME_OBJECT (reply),
+			      "References", references);
+
+    return reply;
+}
+
 static int
 notmuch_reply_format_default(void *ctx,
 			     notmuch_config_t *config,
@@ -515,8 +570,6 @@ notmuch_reply_format_default(void *ctx,
     GMimeMessage *reply;
     notmuch_messages_t *messages;
     notmuch_message_t *message;
-    const char *subject, *from_addr = NULL;
-    const char *in_reply_to, *orig_references, *references;
     const notmuch_show_format_t *format = &format_reply;
 
     for (messages = notmuch_query_search_messages (query);
@@ -525,48 +578,10 @@ notmuch_reply_format_default(void *ctx,
     {
 	message = notmuch_messages_get (messages);
 
-	/* The 1 means we want headers in a "pretty" order. */
-	reply = g_mime_message_new (1);
-	if (reply == NULL) {
-	    fprintf (stderr, "Out of memory\n");
-	    return 1;
-	}
-
-	subject = notmuch_message_get_header (message, "subject");
-	if (subject) {
-	    if (strncasecmp (subject, "Re:", 3))
-		subject = talloc_asprintf (ctx, "Re: %s", subject);
-	    g_mime_message_set_subject (reply, subject);
-	}
-
-	from_addr = add_recipients_from_message (reply, config, message,
-						 reply_all);
-
-	if (from_addr == NULL)
-	    from_addr = guess_from_received_header (config, message);
-
-	if (from_addr == NULL)
-	    from_addr = notmuch_config_get_user_primary_email (config);
+	reply = create_reply_message (ctx, config, message, reply_all);
 
-	from_addr = talloc_asprintf (ctx, "%s <%s>",
-				     notmuch_config_get_user_name (config),
-				     from_addr);
-	g_mime_object_set_header (GMIME_OBJECT (reply),
-				  "From", from_addr);
-
-	in_reply_to = talloc_asprintf (ctx, "<%s>",
-			     notmuch_message_get_message_id (message));
-
-	g_mime_object_set_header (GMIME_OBJECT (reply),
-				  "In-Reply-To", in_reply_to);
-
-	orig_references = notmuch_message_get_header (message, "references");
-	references = talloc_asprintf (ctx, "%s%s%s",
-				      orig_references ? orig_references : "",
-				      orig_references ? " " : "",
-				      in_reply_to);
-	g_mime_object_set_header (GMIME_OBJECT (reply),
-				  "References", references);
+	if (!reply)
+	    continue;
 
 	show_reply_headers (reply);
 
@@ -584,6 +599,89 @@ notmuch_reply_format_default(void *ctx,
     return 0;
 }
 
+static void
+format_reply_headers_json (const void *ctx, GMimeMessage *message)
+{
+    void *local = talloc_new (ctx);
+    InternetAddressList *recipients;
+    const char *recipients_string;
+
+    printf ("{%s: %s",
+	    json_quote_str (local, "Subject"),
+	    json_quote_str (local, g_mime_message_get_subject (message)));
+    printf (", %s: %s",
+	    json_quote_str (local, "From"),
+	    json_quote_str (local, g_mime_message_get_sender (message)));
+    recipients = g_mime_message_get_recipients (message, GMIME_RECIPIENT_TYPE_TO);
+    recipients_string = internet_address_list_to_string (recipients, 0);
+    if (recipients_string)
+	printf (", %s: %s",
+		json_quote_str (local, "To"),
+		json_quote_str (local, recipients_string));
+    recipients = g_mime_message_get_recipients (message, GMIME_RECIPIENT_TYPE_CC);
+    recipients_string = internet_address_list_to_string (recipients, 0);
+    if (recipients_string)
+	printf (", %s: %s",
+		json_quote_str (local, "Cc"),
+		json_quote_str (local, recipients_string));
+
+    printf (", %s: %s",
+	    json_quote_str (local, "In-reply-to"),
+	    json_quote_str (local, g_mime_object_get_header (GMIME_OBJECT (message), "In-reply-to")));
+
+    printf (", %s: %s",
+	    json_quote_str (local, "References"),
+	    json_quote_str (local, g_mime_object_get_header (GMIME_OBJECT (message), "References")));
+
+    talloc_free (local);
+}
+
+static int
+notmuch_reply_format_json(void *ctx,
+			  notmuch_config_t *config,
+			  notmuch_query_t *query,
+			  notmuch_show_params_t *params,
+			  notmuch_bool_t reply_all)
+{
+    GMimeMessage *reply;
+    notmuch_messages_t *messages;
+    notmuch_message_t *message;
+    mime_node_t *node;
+
+    if (notmuch_query_count_messages (query) != 1) {
+	fprintf (stderr, "Error: search term did not match precisely one message.\n");
+	return 1;
+    }
+
+    messages = notmuch_query_search_messages (query);
+    message = notmuch_messages_get (messages);
+    if (mime_node_open (ctx, message, params->cryptoctx, params->decrypt,
+			&node) != NOTMUCH_STATUS_SUCCESS)
+	return 1;
+
+    reply = create_reply_message (ctx, config, message, reply_all);
+    if (!reply)
+	return 1;
+
+    /* The headers of the reply message we've created */
+    printf ("{\"reply-headers\": ");
+    format_reply_headers_json (ctx, reply);
+    printf ("}");
+    g_object_unref (G_OBJECT (reply));
+    reply = NULL;
+
+    /* Start the original */
+    printf (", \"original\": ");
+
+    format_part_json (ctx, node, TRUE);
+
+    /* End */
+    printf ("}\n");
+    notmuch_message_destroy (message);
+
+    return 0;
+}
+
 /* This format is currently tuned for a git send-email --notmuch hook */
 static int
 notmuch_reply_format_headers_only(void *ctx,
@@ -646,6 +744,7 @@ notmuch_reply_format_headers_only(void *ctx,
 
 enum {
     FORMAT_DEFAULT,
+    FORMAT_JSON,
     FORMAT_HEADERS_ONLY,
 };
 
@@ -665,6 +764,7 @@ notmuch_reply_command (void *ctx, int argc, char *argv[])
     notmuch_opt_desc_t options[] = {
 	{ NOTMUCH_OPT_KEYWORD, &format, "format", 'f',
 	  (notmuch_keyword_t []){ { "default", FORMAT_DEFAULT },
+				  { "json", FORMAT_JSON },
 				  { "headers-only", FORMAT_HEADERS_ONLY },
 				  { 0, 0 } } },
 	{ NOTMUCH_OPT_KEYWORD, &reply_all, "reply-to", 'r',
@@ -683,6 +783,8 @@ notmuch_reply_command (void *ctx, int argc, char *argv[])
 
     if (format == FORMAT_HEADERS_ONLY)
 	reply_format_func = notmuch_reply_format_headers_only;
+    else if (format == FORMAT_JSON)
+	reply_format_func = notmuch_reply_format_json;
     else
 	reply_format_func = notmuch_reply_format_default;
 
diff --git a/notmuch-show.c b/notmuch-show.c
index 6a171a4..c570a16 100644
--- a/notmuch-show.c
+++ b/notmuch-show.c
@@ -652,7 +652,7 @@ format_part_text (const void *ctx, mime_node_t *node,
     printf ("\f%s}\n", part_type);
 }
 
-static void
+void
 format_part_json (const void *ctx, mime_node_t *node, notmuch_bool_t first)
 {
     /* Any changes to the JSON format should be reflected in the file
-- 
1.7.5.4

[PATCH v5.1 4/4] emacs: Use the new JSON reply format and message-cite-original

[Adam Wolfe Gordon]

Using the new JSON reply format allows emacs to quote HTML parts
nicely by using mm-display-part to turn them into displayable text,
then quoting them with message-cite-original. This is very useful for
users who regularly receive HTML-only email.

Use message-mode's message-cite-original function to create the
quoted body for reply messages. In order to make this act like the
existing notmuch defaults, you will need to set the following in
your emacs configuration:

message-citation-line-format "On %a, %d %b %Y, %f wrote:"
message-citation-line-function 'message-insert-formatted-citation-line

The test has been updated to reflect the (ugly) emacs default.
---

Adjusted the header display to be consistent with the other JSON formats.

 emacs/notmuch-lib.el  |   39 +++++++++++++++
 emacs/notmuch-mua.el  |  123 +++++++++++++++++++++++++++++++++++--------------
 emacs/notmuch-show.el |   24 +---------
 test/emacs            |  101 +++++++++++++++++++++++++++++++++++++++-
 4 files changed, 228 insertions(+), 59 deletions(-)

diff --git a/emacs/notmuch-lib.el b/emacs/notmuch-lib.el
index d315f76..3fc7aff 100644
--- a/emacs/notmuch-lib.el
+++ b/emacs/notmuch-lib.el
@@ -21,6 +21,8 @@
 
 ;; This is an part of an emacs-based interface to the notmuch mail system.
 
+(eval-when-compile (require 'cl))
+
 (defvar notmuch-command "notmuch"
   "Command to run the notmuch binary.")
 
@@ -173,6 +175,43 @@ the user hasn't set this variable with the old or new value."
   (list 'when (< emacs-major-version 23)
 	form))
 
+(defun notmuch-split-content-type (content-type)
+  "Split content/type into 'content' and 'type'"
+  (split-string content-type "/"))
+
+(defun notmuch-match-content-type (t1 t2)
+  "Return t if t1 and t2 are matching content types, taking wildcards into account"
+  (let ((st1 (notmuch-split-content-type t1))
+	(st2 (notmuch-split-content-type t2)))
+    (if (or (string= (cadr st1) "*")
+	    (string= (cadr st2) "*"))
+	(string= (car st1) (car st2))
+      (string= t1 t2))))
+
+(defvar notmuch-multipart/alternative-discouraged
+  '(
+    ;; Avoid HTML parts.
+    "text/html"
+    ;; multipart/related usually contain a text/html part and some associated graphics.
+    "multipart/related"
+    ))
+
+(defun notmuch-multipart/alternative-choose (types)
+  "Return a list of preferred types from the given list of types"
+  ;; Based on `mm-preferred-alternative-precedence'.
+  (let ((seq types))
+    (dolist (pref (reverse notmuch-multipart/alternative-discouraged))
+      (dolist (elem (copy-sequence seq))
+	(when (string-match pref elem)
+	  (setq seq (nconc (delete elem seq) (list elem))))))
+    seq))
+
+(defun notmuch-parts-filter-by-type (parts type)
+  "Given a vector of message parts, return a vector containing the ones matching the given type."
+  (loop for part across parts
+	if (notmuch-match-content-type (cdr (assq 'content-type part)) type)
+	vconcat (list part)))
+
 ;; Compatibility functions for versions of emacs before emacs 23.
 ;;
 ;; Both functions here were copied from emacs 23 with the following copyright:
diff --git a/emacs/notmuch-mua.el b/emacs/notmuch-mua.el
index 4be7c13..c2fc8c5 100644
--- a/emacs/notmuch-mua.el
+++ b/emacs/notmuch-mua.el
@@ -19,11 +19,15 @@
 ;;
 ;; Authors: David Edmondson <dme@dme.org>
 
+(require 'json)
 (require 'message)
+(require 'format-spec)
 
 (require 'notmuch-lib)
 (require 'notmuch-address)
 
+(eval-when-compile (require 'cl))
+
 ;;
 
 (defcustom notmuch-mua-send-hook '(notmuch-mua-message-send-hook)
@@ -72,56 +76,105 @@ list."
 	    (push header message-hidden-headers)))
 	notmuch-mua-hidden-headers))
 
+(defun notmuch-mua-get-displayed-part (part query-string)
+  (with-temp-buffer
+    (if (assq 'content part)
+	(insert (cdr (assq 'content part)))
+      (call-process notmuch-command nil t nil "show" "--format=raw"
+		    (format "--part=%s" (cdr (assq 'id part)))
+		    query-string))
+
+    (let ((handle (mm-make-handle (current-buffer) (list (cdr (assq 'content-type part)))))
+	  (end-of-orig (point-max)))
+      (mm-display-part handle)
+      (delete-region (point-min) end-of-orig)
+      (buffer-substring (point-min) (point-max)))))
+
+(defun notmuch-mua-multipart/*-to-list (parts)
+  (loop for part across parts
+	collect (cdr (assq 'content-type part))))
+
+(defun notmuch-mua-get-quotable-parts (parts)
+  (loop for part across parts
+	if (notmuch-match-content-type (cdr (assq 'content-type part)) "multipart/alternative")
+	  append (let* ((subparts (cdr (assq 'content part)))
+			(types (notmuch-mua-multipart/*-to-list subparts))
+			(chosen-type (car (notmuch-multipart/alternative-choose types))))
+		   (notmuch-mua-get-quotable-parts (notmuch-parts-filter-by-type subparts chosen-type)))
+	else if (notmuch-match-content-type (cdr (assq 'content-type part)) "multipart/*")
+	  append (notmuch-mua-get-quotable-parts (cdr (assq 'content part)))
+	else if (notmuch-match-content-type (cdr (assq 'content-type part)) "text/*")
+	  collect part))
+
 (defun notmuch-mua-reply (query-string &optional sender reply-all)
-  (let (headers
-	body
-	(args '("reply")))
-    (if notmuch-show-process-crypto
-	(setq args (append args '("--decrypt"))))
+  (let ((args '("reply" "--format=json"))
+	reply
+	original)
+    (when notmuch-show-process-crypto
+      (setq args (append args '("--decrypt"))))
+
     (if reply-all
 	(setq args (append args '("--reply-to=all")))
       (setq args (append args '("--reply-to=sender"))))
     (setq args (append args (list query-string)))
-    ;; This make assumptions about the output of `notmuch reply', but
-    ;; really only that the headers come first followed by a blank
-    ;; line and then the body.
+
+    ;; Get the reply object as JSON, and parse it into an elisp object.
     (with-temp-buffer
       (apply 'call-process (append (list notmuch-command nil (list t t) nil) args))
       (goto-char (point-min))
-      (if (re-search-forward "^$" nil t)
-	  (save-excursion
-	    (save-restriction
-	      (narrow-to-region (point-min) (point))
-	      (goto-char (point-min))
-	      (setq headers (mail-header-extract)))))
-      (forward-line 1)
-      (setq body (buffer-substring (point) (point-max))))
-    ;; If sender is non-nil, set the From: header to its value.
-    (when sender
-      (mail-header-set 'from sender headers))
-    (let
-	;; Overlay the composition window on that being used to read
-	;; the original message.
-	((same-window-regexps '("\\*mail .*")))
-      (notmuch-mua-mail (mail-header 'to headers)
-			(mail-header 'subject headers)
-			(message-headers-to-generate headers t '(to subject))))
-    ;; insert the message body - but put it in front of the signature
-    ;; if one is present
-    (goto-char (point-max))
-    (if (re-search-backward message-signature-separator nil t)
+      (setq reply (json-read)))
+
+    ;; Extract the original message to simplify the following code.
+    (setq original (cdr (assq 'original reply)))
+
+    ;; Extract the headers of both the reply and the original message.
+    (let* ((original-headers (cdr (assq 'headers original)))
+	   (reply-headers (cdr (assq 'reply-headers reply))))
+
+      ;; If sender is non-nil, set the From: header to its value.
+      (when sender
+	(mail-header-set 'From sender reply-headers))
+      (let
+	  ;; Overlay the composition window on that being used to read
+	  ;; the original message.
+	  ((same-window-regexps '("\\*mail .*")))
+	(notmuch-mua-mail (mail-header 'To reply-headers)
+			  (mail-header 'Subject reply-headers)
+			  (message-headers-to-generate reply-headers t '(To Subject))))
+      ;; Insert the message body - but put it in front of the signature
+      ;; if one is present
+      (goto-char (point-max))
+      (if (re-search-backward message-signature-separator nil t)
 	  (forward-line -1)
-      (goto-char (point-max)))
-    (insert body)
-    (push-mark))
-  (set-buffer-modified-p nil)
+	(goto-char (point-max)))
+
+      (let ((from (cdr (assq 'From original-headers)))
+	    (date (cdr (assq 'Date original-headers)))
+	    (start (point)))
+
+	(insert "From: " from "\n")
+	(insert "Date: " date "\n\n")
+
+	;; Get the parts of the original message that should be quoted; this includes
+	;; all the text parts, except the non-preferred ones in a multipart/alternative.
+	(let ((quotable-parts (notmuch-mua-get-quotable-parts (cdr (assq 'body original)))))
+	  (mapc (lambda (part)
+		  (insert (notmuch-mua-get-displayed-part part query-string)))
+		quotable-parts))
+
+	(push-mark)
+	(goto-char start)
+	;; Quote the original message according to the user's configured style.
+	(message-cite-original))))
 
+  (push-mark)
   (message-goto-body)
   ;; Original message may contain (malicious) MML tags.  We must
   ;; properly quote them in the reply.  Note that using `point-max'
   ;; instead of `mark' here is wrong.  The buffer may include user's
   ;; signature which should not be MML-quoted.
-  (mml-quote-region (point) (mark)))
+  (mml-quote-region (point) (mark))
+  (set-buffer-modified-p nil))
 
 (defun notmuch-mua-forward-message ()
   (message-forward)
diff --git a/emacs/notmuch-show.el b/emacs/notmuch-show.el
index 43408d9..90cdd38 100644
--- a/emacs/notmuch-show.el
+++ b/emacs/notmuch-show.el
@@ -513,30 +513,13 @@ current buffer, if possible."
 	    (mm-display-part handle)
 	    t))))))
 
-(defvar notmuch-show-multipart/alternative-discouraged
-  '(
-    ;; Avoid HTML parts.
-    "text/html"
-    ;; multipart/related usually contain a text/html part and some associated graphics.
-    "multipart/related"
-    ))
-
 (defun notmuch-show-multipart/*-to-list (part)
   (mapcar (lambda (inner-part) (plist-get inner-part :content-type))
 	  (plist-get part :content)))
 
-(defun notmuch-show-multipart/alternative-choose (types)
-  ;; Based on `mm-preferred-alternative-precedence'.
-  (let ((seq types))
-    (dolist (pref (reverse notmuch-show-multipart/alternative-discouraged))
-      (dolist (elem (copy-sequence seq))
-	(when (string-match pref elem)
-	  (setq seq (nconc (delete elem seq) (list elem))))))
-    seq))
-
 (defun notmuch-show-insert-part-multipart/alternative (msg part content-type nth depth declared-type)
   (notmuch-show-insert-part-header nth declared-type content-type nil)
-  (let ((chosen-type (car (notmuch-show-multipart/alternative-choose (notmuch-show-multipart/*-to-list part))))
+  (let ((chosen-type (car (notmuch-multipart/alternative-choose (notmuch-show-multipart/*-to-list part))))
 	(inner-parts (plist-get part :content))
 	(start (point)))
     ;; This inserts all parts of the chosen type rather than just one,
@@ -775,9 +758,6 @@ current buffer, if possible."
 
 ;; Functions for determining how to handle MIME parts.
 
-(defun notmuch-show-split-content-type (content-type)
-  (split-string content-type "/"))
-
 (defun notmuch-show-handlers-for (content-type)
   "Return a list of content handlers for a part of type CONTENT-TYPE."
   (let (result)
@@ -788,7 +768,7 @@ current buffer, if possible."
 	  (list (intern (concat "notmuch-show-insert-part-*/*"))
 		(intern (concat
 			 "notmuch-show-insert-part-"
-			 (car (notmuch-show-split-content-type content-type))
+			 (car (notmuch-split-content-type content-type))
 			 "/*"))
 		(intern (concat "notmuch-show-insert-part-" content-type))))
     result))
diff --git a/test/emacs b/test/emacs
index d4a8d30..a6786d4 100755
--- a/test/emacs
+++ b/test/emacs
@@ -268,11 +268,107 @@ Subject: Re: Testing message sent via SMTP
 In-Reply-To: <XXX>
 Fcc: $(pwd)/mail/sent
 --text follows this line--
-On 01 Jan 2000 12:00:00 -0000, Notmuch Test Suite <test_suite@notmuchmail.org> wrote:
+Notmuch Test Suite <test_suite@notmuchmail.org> writes:
+
 > This is a test that messages are sent via SMTP
 EOF
 test_expect_equal_file OUTPUT EXPECTED
 
+test_begin_subtest "Reply within emacs to a multipart/mixed message"
+test_emacs '(notmuch-show "id:20091118002059.067214ed@hikari")
+		(notmuch-show-reply)
+		(test-output)'
+cat <<EOF >EXPECTED
+From: Notmuch Test Suite <test_suite@notmuchmail.org>
+To: Adrian Perez de Castro <aperez@igalia.com>, notmuch@notmuchmail.org
+Subject: Re: [notmuch] Introducing myself
+In-Reply-To: <20091118002059.067214ed@hikari>
+Fcc: ${MAIL_DIR}/sent
+--text follows this line--
+Adrian Perez de Castro <aperez@igalia.com> writes:
+
+> Hello to all,
+>
+> I have just heard about Not Much today in some random Linux-related news
+> site (LWN?), my name is Adrian Perez and I work as systems administrator
+> (although I can do some code as well :P). I have always thought that the
+> ideas behind Sup were great, but after some time using it, I got tired of
+> the oddities that it has. I also do not like doing things like having to
+> install Ruby just for reading and sorting mails. Some time ago I thought
+> about doing something like Not Much and in fact I played a bit with the
+> Python+Xapian and the Python+Whoosh combinations, because I find relaxing
+> to code things in Python when I am not working and also it is installed
+> by default on most distribution. I got to have some mailboxes indexed and
+> basic searching working a couple of months ago. Lately I have been very
+> busy and had no time for coding, and them... boom! Not Much appears -- and
+> it is almost exactly what I was trying to do, but faster. I have been
+> playing a bit with Not Much today, and I think it has potential.
+>
+> Also, I would like to share one idea I had in mind, that you might find
+> interesting: One thing I have found very annoying is having to re-tag my
+> mail when the indexes get b0rked (it happened a couple of times to me while
+> using Sup), so I was planning to mails as read/unread and adding the tags
+> not just to the index, but to the mail text itself, e.g. by adding a
+> "X-Tags" header field or by reusing the "Keywords" one. This way, the index
+> could be totally recreated by re-reading the mail directories, and this
+> would also allow to a tools like OfflineIMAP [1] to get the mails into a
+> local maildir, tagging and indexing the mails with the e-mail reader and
+> then syncing back the messages with the "X-Tags" header to the IMAP server.
+> This would allow to use the mail reader from a different computer and still
+> have everything tagged finely.
+>
+> Best regards,
+>
+>
+> ---
+> [1] http://software.complete.org/software/projects/show/offlineimap
+>
+> -- 
+> Adrian Perez de Castro <aperez@igalia.com>
+> Igalia - Free Software Engineering
+> _______________________________________________
+> notmuch mailing list
+> notmuch@notmuchmail.org
+> http://notmuchmail.org/mailman/listinfo/notmuch
+EOF
+test_expect_equal_file OUTPUT EXPECTED
+
+test_begin_subtest "Reply within emacs to a multipart/alternative message"
+test_emacs '(notmuch-show "id:cf0c4d610911171136h1713aa59w9cf9aa31f052ad0a@mail.gmail.com")
+		(notmuch-show-reply)
+		(test-output)'
+cat <<EOF >EXPECTED
+From: Notmuch Test Suite <test_suite@notmuchmail.org>
+To: Alex Botero-Lowry <alex.boterolowry@gmail.com>, notmuch@notmuchmail.org
+Subject: Re: [notmuch] preliminary FreeBSD support
+In-Reply-To: <cf0c4d610911171136h1713aa59w9cf9aa31f052ad0a@mail.gmail.com>
+Fcc: ${MAIL_DIR}/sent
+--text follows this line--
+Alex Botero-Lowry <alex.boterolowry@gmail.com> writes:
+
+> I saw the announcement this morning, and was very excited, as I had been
+> hoping sup would be turned into a library,
+> since I like the concept more than the UI (I'd rather an emacs interface).
+>
+> I did a preliminary compile which worked out fine, but
+> sysconf(_SC_SC_GETPW_R_SIZE_MAX) returns -1 on
+> FreeBSD, so notmuch_config_open segfaulted.
+>
+> Attached is a patch that supplies a default buffer size of 64 in cases where
+> -1 is returned.
+>
+> http://www.opengroup.org/austin/docs/austin_328.txt - seems to indicate this
+> is acceptable behavior,
+> and http://mail-index.netbsd.org/pkgsrc-bugs/2006/06/07/msg016808.htmlspecifically
+> uses 64 as the
+> buffer size.
+> _______________________________________________
+> notmuch mailing list
+> notmuch@notmuchmail.org
+> http://notmuchmail.org/mailman/listinfo/notmuch
+EOF
+test_expect_equal_file OUTPUT EXPECTED
+
 test_begin_subtest "Quote MML tags in reply"
 message_id='test-emacs-mml-quoting@message.id'
 add_message [id]="$message_id" \
@@ -288,7 +384,8 @@ Subject: Re: Quote MML tags in reply
 In-Reply-To: <test-emacs-mml-quoting@message.id>
 Fcc: ${MAIL_DIR}/sent
 --text follows this line--
-On Fri, 05 Jan 2001 15:43:57 +0000, Notmuch Test Suite <test_suite@notmuchmail.org> wrote:
+Notmuch Test Suite <test_suite@notmuchmail.org> writes:
+
 > <#!part disposition=inline>
 EOF
 test_expect_equal_file OUTPUT EXPECTED
-- 
1.7.5.4

Re: [PATCH v5 0/4] Reply enhancements (was: quoting HTML email ...)

[Adam Wolfe Gordon]

I've just sent new versions of parts 1, 2, and 4. The only change is
that I've changed the output of the reply headers to be consistent
with the other JSON formats (capitalized, in the right order). Of
course, that had a ripple effect on the tests and emacs.

Sorry for the double-mail - I just realized I should change these
things while reviewing Austin's patches.