[PATCH v7 00/10] Reply enhancements

[PATCH v7 00/10] Reply enhancements

[Adam Wolfe Gordon]

Hi everyone,

This is a new version of my reply series [1] that addresses Jani's and
Austin's reviews and fixes a couple of bugs I found in using the patches.

Summary of the changes:

* The patch that disallows replying to multiple messages with the default
  format has been dropped, as Jani and Austin agreed that there's no
  reason this should be changed now. The JSON format sill only allows
  replying to a single message.

* The emacs reply code now shares code to get message content with show.
  This should mean better handling of character sets and other things.

* I noticed that reply supports a --decrypt option, and the emacs interface
  actually does use it. I've documented the option in the reply man page.
  If someone can test that replying to encrypted messages actually works in
  emacs, that would be great - I have no way to test this.

[1] id:"1329893199-21630-1-git-send-email-awg+notmuch@xvx.ca"

Adam Wolfe Gordon (10):
  test: Add broken test for the new JSON reply format.
  reply: Factor out reply creation
  TODO: Add replying to multiple messages
  reply: Add a JSON reply format.
  schemata: Add documentation for JSON reply format.
  man: Update notmuch-reply man page for JSON format.
  man: Add --decrypt to reply flags
  emacs: Factor out useful functions into notmuch-lib
  test: Add broken tests for new emacs reply functionality
  emacs: Use the new JSON reply format and message-cite-original

 devel/TODO               |    8 +
 devel/schemata           |   27 +-
 emacs/notmuch-lib.el     |   44 ++
 emacs/notmuch-mua.el     |  136 ++++--
 emacs/notmuch-show.el    |   24 +-
 man/man1/notmuch-reply.1 |   18 +-
 notmuch-client.h         |   14 +-
 notmuch-reply.c          |  153 ++++--
 notmuch-show.c           |   29 +-
 test/emacs               |  101 ++++-
 test/multipart           |   52 ++
 test/test-lib            | 1242 ++++++++++++++++++++++++++++++++++++++++++++++
 12 files changed, 1726 insertions(+), 122 deletions(-)
 create mode 100755 test/test-lib

-- 
1.7.5.4

[PATCH v7 01/10] test: Add broken test for the new JSON reply format.

[Adam Wolfe Gordon]

---
 test/multipart |   53 +++
 test/test-lib  | 1242 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 1295 insertions(+), 0 deletions(-)
 create mode 100755 test/test-lib

diff --git a/test/multipart b/test/multipart
index 53782c6..80d6e88 100755
--- a/test/multipart
+++ b/test/multipart
@@ -589,6 +589,59 @@ Non-text part: text/html
 EOF
 test_expect_equal_file OUTPUT EXPECTED
 
+test_begin_subtest "'notmuch reply' to a multipart message with json format"
+test_subtest_known_broken
+notmuch reply --format=json 'id:87liy5ap00.fsf@yoom.home.cworth.org' | notmuch_json_show_sanitize >OUTPUT
+cat <<EOF >EXPECTED
+{"reply-headers": {"Subject": "Re: Multipart message",
+ "From": "Notmuch Test Suite <test_suite@notmuchmail.org>",
+ "To": "Carl Worth <cworth@cworth.org>,
+ cworth@cworth.org",
+ "In-reply-to": "<87liy5ap00.fsf@yoom.home.cworth.org>",
+ "References": " <87liy5ap00.fsf@yoom.home.cworth.org>"},
+ "original": {"id": "XXXXX",
+ "match": false,
+ "excluded": false,
+ "filename": "YYYYY",
+ "timestamp": 978709437,
+ "date_relative": "2001-01-05",
+ "tags": ["attachment","inbox","signed","unread"],
+ "headers": {"Subject": "Multipart message",
+ "From": "Carl Worth <cworth@cworth.org>",
+ "To": "cworth@cworth.org",
+ "Date": "Fri,
+ 05 Jan 2001 15:43:57 +0000"},
+ "body": [{"id": 1,
+ "content-type": "multipart/signed",
+ "content": [{"id": 2,
+ "content-type": "multipart/mixed",
+ "content": [{"id": 3,
+ "content-type": "message/rfc822",
+ "content": [{"headers": {"Subject": "html message",
+ "From": "Carl Worth <cworth@cworth.org>",
+ "To": "cworth@cworth.org",
+ "Date": "Fri,
+ 05 Jan 2001 15:42:57 +0000"},
+ "body": [{"id": 4,
+ "content-type": "multipart/alternative",
+ "content": [{"id": 5,
+ "content-type": "text/html"},
+ {"id": 6,
+ "content-type": "text/plain",
+ "content": "This is an embedded message,
+ with a multipart/alternative part.\n"}]}]}]},
+ {"id": 7,
+ "content-type": "text/plain",
+ "filename": "YYYYY",
+ "content": "This is a text attachment.\n"},
+ {"id": 8,
+ "content-type": "text/plain",
+ "content": "And this message is signed.\n\n-Carl\n"}]},
+ {"id": 9,
+ "content-type": "application/pgp-signature"}]}]}}
+EOF
+test_expect_equal_file OUTPUT EXPECTED
+
 test_begin_subtest "'notmuch show --part' does not corrupt a part with CRLF pair"
 notmuch show --format=raw --part=3 id:base64-part-with-crlf > crlf.out
 echo -n -e "\xEF\x0D\x0A" > crlf.expected
diff --git a/test/test-lib b/test/test-lib
new file mode 100755
index 0000000..8158328
--- /dev/null
+++ b/test/test-lib
@@ -0,0 +1,1242 @@
+#
+# Copyright (c) 2005 Junio C Hamano
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see http://www.gnu.org/licenses/ .
+
+if [ ${BASH_VERSINFO[0]} -lt 4 ]; then
+    echo "Error: The notmuch test suite requires a bash version >= 4.0"
+    echo "due to use of associative arrays within the test suite."
+    echo "Please try again with a newer bash (or help us fix the"
+    echo "test suite to be more portable). Thanks."
+    exit 1
+fi
+
+# if --tee was passed, write the output not only to the terminal, but
+# additionally to the file test-results/$BASENAME.out, too.
+case "$GIT_TEST_TEE_STARTED, $* " in
+done,*)
+	# do not redirect again
+	;;
+*' --tee '*|*' --va'*)
+	mkdir -p test-results
+	BASE=test-results/$(basename "$0" .sh)
+	(GIT_TEST_TEE_STARTED=done ${SHELL-sh} "$0" "$@" 2>&1;
+	 echo $? > $BASE.exit) | tee $BASE.out
+	test "$(cat $BASE.exit)" = 0
+	exit
+	;;
+esac
+
+# Keep the original TERM for say_color and test_emacs
+ORIGINAL_TERM=$TERM
+
+# For repeatability, reset the environment to known value.
+LANG=C
+LC_ALL=C
+PAGER=cat
+TZ=UTC
+TERM=dumb
+export LANG LC_ALL PAGER TERM TZ
+GIT_TEST_CMP=${GIT_TEST_CMP:-diff -u}
+TEST_EMACS=${TEST_EMACS:-${EMACS:-emacs}}
+
+# Protect ourselves from common misconfiguration to export
+# CDPATH into the environment
+unset CDPATH
+
+unset GREP_OPTIONS
+
+# Convenience
+#
+# A regexp to match 5 and 40 hexdigits
+_x05='[0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f]'
+_x40="$_x05$_x05$_x05$_x05$_x05$_x05$_x05$_x05"
+
+_x04='[0-9a-f][0-9a-f][0-9a-f][0-9a-f]'
+_x32="$_x04$_x04$_x04$_x04$_x04$_x04$_x04$_x04"
+
+# Each test should start with something like this, after copyright notices:
+#
+# test_description='Description of this test...
+# This test checks if command xyzzy does the right thing...
+# '
+# . ./test-lib.sh
+[ "x$ORIGINAL_TERM" != "xdumb" ] && (
+		TERM=$ORIGINAL_TERM &&
+		export TERM &&
+		[ -t 1 ] &&
+		tput bold >/dev/null 2>&1 &&
+		tput setaf 1 >/dev/null 2>&1 &&
+		tput sgr0 >/dev/null 2>&1
+	) &&
+	color=t
+
+while test "$#" -ne 0
+do
+	case "$1" in
+	-d|--d|--de|--deb|--debu|--debug)
+		debug=t; shift ;;
+	-i|--i|--im|--imm|--imme|--immed|--immedi|--immedia|--immediat|--immediate)
+		immediate=t; shift ;;
+	-l|--l|--lo|--lon|--long|--long-|--long-t|--long-te|--long-tes|--long-test|--long-tests)
+		GIT_TEST_LONG=t; export GIT_TEST_LONG; shift ;;
+	-h|--h|--he|--hel|--help)
+		help=t; shift ;;
+	-v|--v|--ve|--ver|--verb|--verbo|--verbos|--verbose)
+		verbose=t; shift ;;
+	-q|--q|--qu|--qui|--quie|--quiet)
+		quiet=t; shift ;;
+	--with-dashes)
+		with_dashes=t; shift ;;
+	--no-color)
+		color=; shift ;;
+	--no-python)
+		# noop now...
+		shift ;;
+	--va|--val|--valg|--valgr|--valgri|--valgrin|--valgrind)
+		valgrind=t; verbose=t; shift ;;
+	--tee)
+		shift ;; # was handled already
+	--root=*)
+		root=$(expr "z$1" : 'z[^=]*=\(.*\)')
+		shift ;;
+	*)
+		echo "error: unknown test option '$1'" >&2; exit 1 ;;
+	esac
+done
+
+if test -n "$debug"; then
+    print_subtest () {
+	printf " %-4s" "[$((test_count - 1))]"
+    }
+else
+    print_subtest () {
+	true
+    }
+fi
+
+if test -n "$color"; then
+	say_color () {
+		(
+		TERM=$ORIGINAL_TERM
+		export TERM
+		case "$1" in
+			error) tput bold; tput setaf 1;; # bold red
+			skip)  tput bold; tput setaf 2;; # bold green
+			pass)  tput setaf 2;;            # green
+			info)  tput setaf 3;;            # brown
+			*) test -n "$quiet" && return;;
+		esac
+		shift
+		printf " "
+		printf "$@"
+		tput sgr0
+		print_subtest
+		)
+	}
+else
+	say_color() {
+		test -z "$1" && test -n "$quiet" && return
+		shift
+		printf " "
+		printf "$@"
+		print_subtest
+	}
+fi
+
+error () {
+	say_color error "error: $*\n"
+	GIT_EXIT_OK=t
+	exit 1
+}
+
+say () {
+	say_color info "$*"
+}
+
+test "${test_description}" != "" ||
+error "Test script did not set test_description."
+
+if test "$help" = "t"
+then
+	echo "Tests ${test_description}"
+	exit 0
+fi
+
+echo $(basename "$0"): "Testing ${test_description}"
+
+exec 5>&1
+
+test_failure=0
+test_count=0
+test_fixed=0
+test_broken=0
+test_success=0
+
+die () {
+	code=$?
+	rm -rf "$TEST_TMPDIR"
+	if test -n "$GIT_EXIT_OK"
+	then
+		exit $code
+	else
+		echo >&5 "FATAL: Unexpected exit with code $code"
+		exit 1
+	fi
+}
+
+GIT_EXIT_OK=
+# Note: TEST_TMPDIR *NOT* exported!
+TEST_TMPDIR=$(mktemp -d "${TMPDIR:-/tmp}/notmuch-test-$$.XXXXXX")
+trap 'die' EXIT
+
+test_decode_color () {
+	sed	-e 's/.\[1m/<WHITE>/g' \
+		-e 's/.\[31m/<RED>/g' \
+		-e 's/.\[32m/<GREEN>/g' \
+		-e 's/.\[33m/<YELLOW>/g' \
+		-e 's/.\[34m/<BLUE>/g' \
+		-e 's/.\[35m/<MAGENTA>/g' \
+		-e 's/.\[36m/<CYAN>/g' \
+		-e 's/.\[m/<RESET>/g'
+}
+
+q_to_nul () {
+	perl -pe 'y/Q/\000/'
+}
+
+q_to_cr () {
+	tr Q '\015'
+}
+
+append_cr () {
+	sed -e 's/$/Q/' | tr Q '\015'
+}
+
+remove_cr () {
+	tr '\015' Q | sed -e 's/Q$//'
+}
+
+# Generate a new message in the mail directory, with a unique message
+# ID and subject. The message is not added to the index.
+#
+# After this function returns, the filename of the generated message
+# is available as $gen_msg_filename and the message ID is available as
+# $gen_msg_id .
+#
+# This function supports named parameters with the bash syntax for
+# assigning a value to an associative array ([name]=value). The
+# supported parameters are:
+#
+#  [dir]=directory/of/choice
+#
+#	Generate the message in directory 'directory/of/choice' within
+#	the mail store. The directory will be created if necessary.
+#
+#  [filename]=name
+#
+#	Store the message in file 'name'. The default is to store it
+#	in 'msg-<count>', where <count> is three-digit number of the
+#	message.
+#
+#  [body]=text
+#
+#	Text to use as the body of the email message
+#
+#  '[from]="Some User <user@example.com>"'
+#  '[to]="Some User <user@example.com>"'
+#  '[subject]="Subject of email message"'
+#  '[date]="RFC 822 Date"'
+#
+#	Values for email headers. If not provided, default values will
+#	be generated instead.
+#
+#  '[cc]="Some User <user@example.com>"'
+#  [reply-to]=some-address
+#  [in-reply-to]=<message-id>
+#  [references]=<message-id>
+#  [content-type]=content-type-specification
+#  '[header]=full header line, including keyword'
+#
+#	Additional values for email headers. If these are not provided
+#	then the relevant headers will simply not appear in the
+#	message.
+#
+#  '[id]=message-id'
+#
+#	Controls the message-id of the created message.
+gen_msg_cnt=0
+gen_msg_filename=""
+gen_msg_id=""
+generate_message ()
+{
+    # This is our (bash-specific) magic for doing named parameters
+    local -A template="($@)"
+    local additional_headers
+
+    gen_msg_cnt=$((gen_msg_cnt + 1))
+    if [ -z "${template[filename]}" ]; then
+	gen_msg_name="msg-$(printf "%03d" $gen_msg_cnt)"
+    else
+	gen_msg_name=${template[filename]}
+    fi
+
+    if [ -z "${template[id]}" ]; then
+	gen_msg_id="${gen_msg_name%:2,*}@notmuch-test-suite"
+    else
+	gen_msg_id="${template[id]}"
+    fi
+
+    if [ -z "${template[dir]}" ]; then
+	gen_msg_filename="${MAIL_DIR}/$gen_msg_name"
+    else
+	gen_msg_filename="${MAIL_DIR}/${template[dir]}/$gen_msg_name"
+	mkdir -p "$(dirname "$gen_msg_filename")"
+    fi
+
+    if [ -z "${template[body]}" ]; then
+	template[body]="This is just a test message (#${gen_msg_cnt})"
+    fi
+
+    if [ -z "${template[from]}" ]; then
+	template[from]="Notmuch Test Suite <test_suite@notmuchmail.org>"
+    fi
+
+    if [ -z "${template[to]}" ]; then
+	template[to]="Notmuch Test Suite <test_suite@notmuchmail.org>"
+    fi
+
+    if [ -z "${template[subject]}" ]; then
+	template[subject]="Test message #${gen_msg_cnt}"
+    fi
+
+    if [ -z "${template[date]}" ]; then
+	template[date]="Fri, 05 Jan 2001 15:43:57 +0000"
+    fi
+
+    additional_headers=""
+    if [ ! -z "${template[header]}" ]; then
+	additional_headers="${template[header]}
+${additional_headers}"
+    fi
+
+    if [ ! -z "${template[reply-to]}" ]; then
+	additional_headers="Reply-To: ${template[reply-to]}
+${additional_headers}"
+    fi
+
+    if [ ! -z "${template[in-reply-to]}" ]; then
+	additional_headers="In-Reply-To: ${template[in-reply-to]}
+${additional_headers}"
+    fi
+
+    if [ ! -z "${template[cc]}" ]; then
+	additional_headers="Cc: ${template[cc]}
+${additional_headers}"
+    fi
+
+    if [ ! -z "${template[references]}" ]; then
+	additional_headers="References: ${template[references]}
+${additional_headers}"
+    fi
+
+    if [ ! -z "${template[content-type]}" ]; then
+	additional_headers="Content-Type: ${template[content-type]}
+${additional_headers}"
+    fi
+
+    # Note that in the way we're setting it above and using it below,
+    # `additional_headers' will also serve as the header / body separator
+    # (empty line in between).
+
+    cat <<EOF >"$gen_msg_filename"
+From: ${template[from]}
+To: ${template[to]}
+Message-Id: <${gen_msg_id}>
+Subject: ${template[subject]}
+Date: ${template[date]}
+${additional_headers}
+${template[body]}
+EOF
+}
+
+# Generate a new message and add it to the database.
+#
+# All of the arguments and return values supported by generate_message
+# are also supported here, so see that function for details.
+add_message ()
+{
+    generate_message "$@" &&
+    notmuch new > /dev/null
+}
+
+# Deliver a message with emacs and add it to the database
+#
+# Uses emacs to generate and deliver a message to the mail store.
+# Accepts arbitrary extra emacs/elisp functions to modify the message
+# before sending, which is useful to doing things like attaching files
+# to the message and encrypting/signing.
+emacs_deliver_message ()
+{
+    local subject="$1"
+    local body="$2"
+    shift 2
+    # before we can send a message, we have to prepare the FCC maildir
+    mkdir -p "$MAIL_DIR"/sent/{cur,new,tmp}
+    $TEST_DIRECTORY/smtp-dummy sent_message &
+    smtp_dummy_pid=$!
+    test_emacs \
+	"(let ((message-send-mail-function 'message-smtpmail-send-it)
+	       (smtpmail-smtp-server \"localhost\")
+	       (smtpmail-smtp-service \"25025\"))
+	   (notmuch-hello)
+	   (notmuch-mua-mail)
+	   (message-goto-to)
+	   (insert \"test_suite@notmuchmail.org\nDate: 01 Jan 2000 12:00:00 -0000\")
+	   (message-goto-subject)
+	   (insert \"${subject}\")
+	   (message-goto-body)
+	   (insert \"${body}\")
+	   $@
+	   (message-send-and-exit))"
+    # opportunistically quit smtp-dummy in case above fails.
+    { echo QUIT > /dev/tcp/localhost/25025; } 2>/dev/null
+    wait ${smtp_dummy_pid}
+    notmuch new >/dev/null
+}
+
+# Generate a corpus of email and add it to the database.
+#
+# This corpus is fixed, (it happens to be 50 messages from early in
+# the history of the notmuch mailing list), which allows for reliably
+# testing commands that need to operate on a not-totally-trivial
+# number of messages.
+add_email_corpus ()
+{
+    rm -rf ${MAIL_DIR}
+    if [ -d $TEST_DIRECTORY/corpus.mail ]; then
+	cp -a $TEST_DIRECTORY/corpus.mail ${MAIL_DIR}
+    else
+	cp -a $TEST_DIRECTORY/corpus ${MAIL_DIR}
+	notmuch new >/dev/null
+	cp -a ${MAIL_DIR} $TEST_DIRECTORY/corpus.mail
+    fi
+}
+
+test_begin_subtest ()
+{
+    if [ -n "$inside_subtest" ]; then
+	exec 1>&6 2>&7		# Restore stdout and stderr
+	error "bug in test script: Missing test_expect_equal in ${BASH_SOURCE[1]}:${BASH_LINENO[0]}"
+    fi
+    test_subtest_name="$1"
+    test_reset_state_
+    # Remember stdout and stderr file descriptors and redirect test
+    # output to the previously prepared file descriptors 3 and 4 (see
+    # below)
+    if test "$verbose" != "t"; then exec 4>test.output 3>&4; fi
+    exec 6>&1 7>&2 >&3 2>&4
+    inside_subtest=t
+}
+
+# Pass test if two arguments match
+#
+# Note: Unlike all other test_expect_* functions, this function does
+# not accept a test name. Instead, the caller should call
+# test_begin_subtest before calling this function in order to set the
+# name.
+test_expect_equal ()
+{
+	exec 1>&6 2>&7		# Restore stdout and stderr
+	inside_subtest=
+	test "$#" = 3 && { prereq=$1; shift; } || prereq=
+	test "$#" = 2 ||
+	error "bug in the test script: not 2 or 3 parameters to test_expect_equal"
+
+	output="$1"
+	expected="$2"
+	if ! test_skip "$test_subtest_name"
+	then
+		if [ "$output" = "$expected" ]; then
+			test_ok_ "$test_subtest_name"
+		else
+			testname=$this_test.$test_count
+			echo "$expected" > $testname.expected
+			echo "$output" > $testname.output
+			test_failure_ "$test_subtest_name" "$(diff -u $testname.expected $testname.output)"
+		fi
+    fi
+}
+
+# Like test_expect_equal, but takes two filenames.
+test_expect_equal_file ()
+{
+	exec 1>&6 2>&7		# Restore stdout and stderr
+	inside_subtest=
+	test "$#" = 3 && { prereq=$1; shift; } || prereq=
+	test "$#" = 2 ||
+	error "bug in the test script: not 2 or 3 parameters to test_expect_equal"
+
+	output="$1"
+	expected="$2"
+	if ! test_skip "$test_subtest_name"
+	then
+		if diff -q "$expected" "$output" >/dev/null ; then
+			test_ok_ "$test_subtest_name"
+		else
+			testname=$this_test.$test_count
+			cp "$output" $testname.output
+			cp "$expected" $testname.expected
+			test_failure_ "$test_subtest_name" "$(diff -u $testname.expected $testname.output)"
+		fi
+    fi
+}
+
+test_emacs_expect_t () {
+	test "$#" = 2 && { prereq=$1; shift; } || prereq=
+	test "$#" = 1 ||
+	error "bug in the test script: not 1 or 2 parameters to test_emacs_expect_t"
+
+	# Run the test.
+	if ! test_skip "$test_subtest_name"
+	then
+		test_emacs "(notmuch-test-run $1)" >/dev/null
+
+		# Restore state after the test.
+		exec 1>&6 2>&7		# Restore stdout and stderr
+		inside_subtest=
+
+		# Report success/failure.
+		result=$(cat OUTPUT)
+		if [ "$result" = t ]
+		then
+			test_ok_ "$test_subtest_name"
+		else
+			test_failure_ "$test_subtest_name" "${result}"
+		fi
+	else
+		# Restore state after the (non) test.
+		exec 1>&6 2>&7		# Restore stdout and stderr
+		inside_subtest=
+	fi
+}
+
+NOTMUCH_NEW ()
+{
+    notmuch new | grep -v -E -e '^Processed [0-9]*( total)? file|Found [0-9]* total file'
+}
+
+notmuch_search_sanitize ()
+{
+    sed -r -e 's/("?thread"?: ?)("?)................("?)/\1\2XXX\3/'
+}
+
+NOTMUCH_SHOW_FILENAME_SQUELCH='s,filename:.*/mail,filename:/XXX/mail,'
+notmuch_show_sanitize ()
+{
+    sed -e "$NOTMUCH_SHOW_FILENAME_SQUELCH"
+}
+notmuch_show_sanitize_all ()
+{
+    sed \
+	-e 's| filename:.*| filename:XXXXX|' \
+	-e 's| id:[^ ]* | id:XXXXX |'
+}
+
+notmuch_json_show_sanitize ()
+{
+    sed -e 's|, |,\n |g' | \
+	sed \
+	-e 's|"id": "[^"]*",|"id": "XXXXX",|' \
+	-e 's|"filename": "[^"]*",|"filename": "YYYYY",|'
+}
+
+# End of notmuch helper functions
+
+# Use test_set_prereq to tell that a particular prerequisite is available.
+# The prerequisite can later be checked for in two ways:
+#
+# - Explicitly using test_have_prereq.
+#
+# - Implicitly by specifying the prerequisite tag in the calls to
+#   test_expect_{success,failure,code}.
+#
+# The single parameter is the prerequisite tag (a simple word, in all
+# capital letters by convention).
+
+test_set_prereq () {
+	satisfied="$satisfied$1 "
+}
+satisfied=" "
+
+test_have_prereq () {
+	case $satisfied in
+	*" $1 "*)
+		: yes, have it ;;
+	*)
+		! : nope ;;
+	esac
+}
+
+# declare prerequisite for the given external binary
+test_declare_external_prereq () {
+	binary="$1"
+	test "$#" = 2 && name=$2 || name="$binary(1)"
+
+	hash $binary 2>/dev/null || eval "
+	test_missing_external_prereq_${binary}_=t
+$binary () {
+	echo -n \"\$test_subtest_missing_external_prereqs_ \" | grep -qe \" $name \" ||
+	test_subtest_missing_external_prereqs_=\"\$test_subtest_missing_external_prereqs_ $name\"
+	false
+}"
+}
+
+# Explicitly require external prerequisite.  Useful when binary is
+# called indirectly (e.g. from emacs).
+# Returns success if dependency is available, failure otherwise.
+test_require_external_prereq () {
+	binary="$1"
+	if [ "$(eval echo -n \$test_missing_external_prereq_${binary}_)" = t ]; then
+		# dependency is missing, call the replacement function to note it
+		eval "$binary"
+	else
+		true
+	fi
+}
+
+# You are not expected to call test_ok_ and test_failure_ directly, use
+# the text_expect_* functions instead.
+
+test_ok_ () {
+	if test "$test_subtest_known_broken_" = "t"; then
+		test_known_broken_ok_ "$@"
+		return
+	fi
+	test_success=$(($test_success + 1))
+	say_color pass "%-6s" "PASS"
+	echo " $@"
+}
+
+test_failure_ () {
+	if test "$test_subtest_known_broken_" = "t"; then
+		test_known_broken_failure_ "$@"
+		return
+	fi
+	test_failure=$(($test_failure + 1))
+	test_failure_message_ "FAIL" "$@"
+	test "$immediate" = "" || { GIT_EXIT_OK=t; exit 1; }
+	return 1
+}
+
+test_failure_message_ () {
+	say_color error "%-6s" "$1"
+	echo " $2"
+	shift 2
+	echo "$@" | sed -e 's/^/	/'
+	if test "$verbose" != "t"; then cat test.output; fi
+}
+
+test_known_broken_ok_ () {
+	test_reset_state_
+	test_fixed=$(($test_fixed+1))
+	say_color pass "%-6s" "FIXED"
+	echo " $@"
+}
+
+test_known_broken_failure_ () {
+	test_reset_state_
+	test_broken=$(($test_broken+1))
+	test_failure_message_ "BROKEN" "$@"
+	return 1
+}
+
+test_debug () {
+	test "$debug" = "" || eval "$1"
+}
+
+test_run_ () {
+	test_cleanup=:
+	if test "$verbose" != "t"; then exec 4>test.output 3>&4; fi
+	eval >&3 2>&4 "$1"
+	eval_ret=$?
+	eval >&3 2>&4 "$test_cleanup"
+	return 0
+}
+
+test_skip () {
+	test_count=$(($test_count+1))
+	to_skip=
+	for skp in $NOTMUCH_SKIP_TESTS
+	do
+		case $this_test.$test_count in
+		$skp)
+			to_skip=t
+		esac
+	done
+	if test -z "$to_skip" && test -n "$prereq" &&
+	   ! test_have_prereq "$prereq"
+	then
+		to_skip=t
+	fi
+	case "$to_skip" in
+	t)
+		test_report_skip_ "$@"
+		;;
+	*)
+		test_check_missing_external_prereqs_ "$@"
+		;;
+	esac
+}
+
+test_check_missing_external_prereqs_ () {
+	if test -n "$test_subtest_missing_external_prereqs_"; then
+		say_color skip >&3 "missing prerequisites:"
+		echo "$test_subtest_missing_external_prereqs_" >&3
+		test_report_skip_ "$@"
+	else
+		false
+	fi
+}
+
+test_report_skip_ () {
+	test_reset_state_
+	say_color skip >&3 "skipping test:"
+	echo " $@" >&3
+	say_color skip "%-6s" "SKIP"
+	echo " $1"
+}
+
+test_subtest_known_broken () {
+	test_subtest_known_broken_=t
+}
+
+test_expect_success () {
+	test "$#" = 3 && { prereq=$1; shift; } || prereq=
+	test "$#" = 2 ||
+	error "bug in the test script: not 2 or 3 parameters to test-expect-success"
+	test_reset_state_
+	if ! test_skip "$@"
+	then
+		test_run_ "$2"
+		run_ret="$?"
+		# test_run_ may update missing external prerequisites
+		test_check_missing_external_prereqs_ "$@" ||
+		if [ "$run_ret" = 0 -a "$eval_ret" = 0 ]
+		then
+			test_ok_ "$1"
+		else
+			test_failure_ "$@"
+		fi
+	fi
+}
+
+test_expect_code () {
+	test "$#" = 4 && { prereq=$1; shift; } || prereq=
+	test "$#" = 3 ||
+	error "bug in the test script: not 3 or 4 parameters to test-expect-code"
+	test_reset_state_
+	if ! test_skip "$@"
+	then
+		test_run_ "$3"
+		run_ret="$?"
+		# test_run_ may update missing external prerequisites,
+		test_check_missing_external_prereqs_ "$@" ||
+		if [ "$run_ret" = 0 -a "$eval_ret" = "$1" ]
+		then
+			test_ok_ "$2"
+		else
+			test_failure_ "$@"
+		fi
+	fi
+}
+
+# test_external runs external test scripts that provide continuous
+# test output about their progress, and succeeds/fails on
+# zero/non-zero exit code.  It outputs the test output on stdout even
+# in non-verbose mode, and announces the external script with "* run
+# <n>: ..." before running it.  When providing relative paths, keep in
+# mind that all scripts run in "trash directory".
+# Usage: test_external description command arguments...
+# Example: test_external 'Perl API' perl ../path/to/test.pl
+test_external () {
+	test "$#" = 4 && { prereq=$1; shift; } || prereq=
+	test "$#" = 3 ||
+	error >&5 "bug in the test script: not 3 or 4 parameters to test_external"
+	descr="$1"
+	shift
+	test_reset_state_
+	if ! test_skip "$descr" "$@"
+	then
+		# Announce the script to reduce confusion about the
+		# test output that follows.
+		say_color "" " run $test_count: $descr ($*)"
+		# Run command; redirect its stderr to &4 as in
+		# test_run_, but keep its stdout on our stdout even in
+		# non-verbose mode.
+		"$@" 2>&4
+		if [ "$?" = 0 ]
+		then
+			test_ok_ "$descr"
+		else
+			test_failure_ "$descr" "$@"
+		fi
+	fi
+}
+
+# Like test_external, but in addition tests that the command generated
+# no output on stderr.
+test_external_without_stderr () {
+	# The temporary file has no (and must have no) security
+	# implications.
+	tmp="$TMPDIR"; if [ -z "$tmp" ]; then tmp=/tmp; fi
+	stderr="$tmp/git-external-stderr.$$.tmp"
+	test_external "$@" 4> "$stderr"
+	[ -f "$stderr" ] || error "Internal error: $stderr disappeared."
+	descr="no stderr: $1"
+	shift
+	if [ ! -s "$stderr" ]; then
+		rm "$stderr"
+		test_ok_ "$descr"
+	else
+		if [ "$verbose" = t ]; then
+			output=`echo; echo Stderr is:; cat "$stderr"`
+		else
+			output=
+		fi
+		# rm first in case test_failure exits.
+		rm "$stderr"
+		test_failure_ "$descr" "$@" "$output"
+	fi
+}
+
+# This is not among top-level (test_expect_success)
+# but is a prefix that can be used in the test script, like:
+#
+#	test_expect_success 'complain and die' '
+#           do something &&
+#           do something else &&
+#	    test_must_fail git checkout ../outerspace
+#	'
+#
+# Writing this as "! git checkout ../outerspace" is wrong, because
+# the failure could be due to a segv.  We want a controlled failure.
+
+test_must_fail () {
+	"$@"
+	test $? -gt 0 -a $? -le 129 -o $? -gt 192
+}
+
+# test_cmp is a helper function to compare actual and expected output.
+# You can use it like:
+#
+#	test_expect_success 'foo works' '
+#		echo expected >expected &&
+#		foo >actual &&
+#		test_cmp expected actual
+#	'
+#
+# This could be written as either "cmp" or "diff -u", but:
+# - cmp's output is not nearly as easy to read as diff -u
+# - not all diff versions understand "-u"
+
+test_cmp() {
+	$GIT_TEST_CMP "$@"
+}
+
+# This function can be used to schedule some commands to be run
+# unconditionally at the end of the test to restore sanity:
+#
+#	test_expect_success 'test core.capslock' '
+#		git config core.capslock true &&
+#		test_when_finished "git config --unset core.capslock" &&
+#		hello world
+#	'
+#
+# That would be roughly equivalent to
+#
+#	test_expect_success 'test core.capslock' '
+#		git config core.capslock true &&
+#		hello world
+#		git config --unset core.capslock
+#	'
+#
+# except that the greeting and config --unset must both succeed for
+# the test to pass.
+
+test_when_finished () {
+	test_cleanup="{ $*
+		} && (exit \"\$eval_ret\"); eval_ret=\$?; $test_cleanup"
+}
+
+test_done () {
+	GIT_EXIT_OK=t
+	test_results_dir="$TEST_DIRECTORY/test-results"
+	mkdir -p "$test_results_dir"
+	test_results_path="$test_results_dir/${0%.sh}-$$"
+
+	echo "total $test_count" >> $test_results_path
+	echo "success $test_success" >> $test_results_path
+	echo "fixed $test_fixed" >> $test_results_path
+	echo "broken $test_broken" >> $test_results_path
+	echo "failed $test_failure" >> $test_results_path
+	echo "" >> $test_results_path
+
+	echo
+
+	[ -n "$EMACS_SERVER" ] && test_emacs '(kill-emacs)'
+
+	if [ "$test_failure" = "0" ]; then
+	    if [ "$test_broken" = "0" ]; then
+		rm -rf "$remove_tmp"
+	    fi
+	    exit 0
+	else
+	    exit 1
+	fi
+}
+
+emacs_generate_script () {
+	# Construct a little test script here for the benefit of the user,
+	# (who can easily run "run_emacs" to get the same emacs environment
+	# for investigating any failures).
+	cat <<EOF >"$TMP_DIRECTORY/run_emacs"
+#!/bin/sh
+export PATH=$PATH
+export NOTMUCH_CONFIG=$NOTMUCH_CONFIG
+
+# Here's what we are using here:
+#
+# --no-init-file	Don't load users ~/.emacs
+#
+# --no-site-file	Don't load the site-wide startup stuff
+#
+# --directory		Ensure that the local elisp sources are found
+#
+# --load		Force loading of notmuch.el and test-lib.el
+
+exec ${TEST_EMACS} --no-init-file --no-site-file \
+	--directory "$TEST_DIRECTORY/../emacs" --load notmuch.el \
+	--directory "$TEST_DIRECTORY" --load test-lib.el \
+	"\$@"
+EOF
+	chmod a+x "$TMP_DIRECTORY/run_emacs"
+}
+
+test_emacs () {
+	# test dependencies beforehand to avoid the waiting loop below
+	missing_dependencies=
+	test_require_external_prereq dtach || missing_dependencies=1
+	test_require_external_prereq emacs || missing_dependencies=1
+	test_require_external_prereq emacsclient || missing_dependencies=1
+	test -z "$missing_dependencies" || return
+
+	if [ -z "$EMACS_SERVER" ]; then
+		server_name="notmuch-test-suite-$$"
+		# start a detached session with an emacs server
+		# user's TERM is given to dtach which assumes a minimally
+		# VT100-compatible terminal -- and emacs inherits that
+		TERM=$ORIGINAL_TERM dtach -n "$TEST_TMPDIR/emacs-dtach-socket.$$" \
+			sh -c "stty rows 24 cols 80; exec '$TMP_DIRECTORY/run_emacs' \
+				--no-window-system \
+				--eval '(setq server-name \"$server_name\")' \
+				--eval '(server-start)' \
+				--eval '(orphan-watchdog $$)'" || return
+		EMACS_SERVER="$server_name"
+		# wait until the emacs server is up
+		until test_emacs '()' >/dev/null 2>/dev/null; do
+			sleep 1
+		done
+	fi
+
+	emacsclient --socket-name="$EMACS_SERVER" --eval "(progn $@)"
+}
+
+test_python() {
+	export LD_LIBRARY_PATH=$TEST_DIRECTORY/../lib
+	export PYTHONPATH=$TEST_DIRECTORY/../bindings/python
+
+	# Some distros (e.g. Arch Linux) ship Python 2.* as /usr/bin/python2,
+	# most others as /usr/bin/python. So first try python2, and fallback to
+	# python if python2 doesn't exist.
+	cmd=python2
+	[[ "$test_missing_external_prereq_python2_" = t ]] && cmd=python
+
+	(echo "import sys; _orig_stdout=sys.stdout; sys.stdout=open('OUTPUT', 'w')"; cat) \
+		| $cmd -
+}
+
+# Creates a script that counts how much time it is executed and calls
+# notmuch.  $notmuch_counter_command is set to the path to the
+# generated script.  Use notmuch_counter_value() function to get the
+# current counter value.
+notmuch_counter_reset () {
+	notmuch_counter_command="$TMP_DIRECTORY/notmuch_counter"
+	if [ ! -x "$notmuch_counter_command" ]; then
+		notmuch_counter_state_path="$TMP_DIRECTORY/notmuch_counter.state"
+		cat >"$notmuch_counter_command" <<EOF || return
+#!/bin/sh
+
+read count < "$notmuch_counter_state_path"
+echo \$((count + 1)) > "$notmuch_counter_state_path"
+
+exec notmuch "\$@"
+EOF
+		chmod +x "$notmuch_counter_command" || return
+	fi
+
+	echo 0 > "$notmuch_counter_state_path"
+}
+
+# Returns the current notmuch counter value.
+notmuch_counter_value () {
+	if [ -r "$notmuch_counter_state_path" ]; then
+		read count < "$notmuch_counter_state_path"
+	else
+		count=0
+	fi
+	echo $count
+}
+
+test_reset_state_ () {
+	test -z "$test_init_done_" && test_init_
+
+	test_subtest_known_broken_=
+	test_subtest_missing_external_prereqs_=
+}
+
+# called once before the first subtest
+test_init_ () {
+	test_init_done_=t
+
+	# skip all tests if there were external prerequisites missing during init
+	test_check_missing_external_prereqs_ "all tests in $this_test" && test_done
+}
+
+
+find_notmuch_path ()
+{
+    dir="$1"
+
+    while [ -n "$dir" ]; do
+	bin="$dir/notmuch"
+	if [ -x "$bin" ]; then
+	    echo "$dir"
+	    return
+	fi
+	dir="$(dirname "$dir")"
+	if [ "$dir" = "/" ]; then
+	    break
+	fi
+    done
+}
+
+# Test the binaries we have just built.  The tests are kept in
+# test/ subdirectory and are run in 'trash directory' subdirectory.
+TEST_DIRECTORY=$(pwd)
+if test -n "$valgrind"
+then
+	make_symlink () {
+		test -h "$2" &&
+		test "$1" = "$(readlink "$2")" || {
+			# be super paranoid
+			if mkdir "$2".lock
+			then
+				rm -f "$2" &&
+				ln -s "$1" "$2" &&
+				rm -r "$2".lock
+			else
+				while test -d "$2".lock
+				do
+					say "Waiting for lock on $2."
+					sleep 1
+				done
+			fi
+		}
+	}
+
+	make_valgrind_symlink () {
+		# handle only executables
+		test -x "$1" || return
+
+		base=$(basename "$1")
+		symlink_target=$TEST_DIRECTORY/../$base
+		# do not override scripts
+		if test -x "$symlink_target" &&
+		    test ! -d "$symlink_target" &&
+		    test "#!" != "$(head -c 2 < "$symlink_target")"
+		then
+			symlink_target=$TEST_DIRECTORY/valgrind.sh
+		fi
+		case "$base" in
+		*.sh|*.perl)
+			symlink_target=$TEST_DIRECTORY/unprocessed-script
+		esac
+		# create the link, or replace it if it is out of date
+		make_symlink "$symlink_target" "$GIT_VALGRIND/bin/$base" || exit
+	}
+
+	# override notmuch executable in TEST_DIRECTORY/..
+	GIT_VALGRIND=$TEST_DIRECTORY/valgrind
+	mkdir -p "$GIT_VALGRIND"/bin
+	make_valgrind_symlink $TEST_DIRECTORY/../notmuch
+	OLDIFS=$IFS
+	IFS=:
+	for path in $PATH
+	do
+		ls "$path"/notmuch 2> /dev/null |
+		while read file
+		do
+			make_valgrind_symlink "$file"
+		done
+	done
+	IFS=$OLDIFS
+	PATH=$GIT_VALGRIND/bin:$PATH
+	GIT_EXEC_PATH=$GIT_VALGRIND/bin
+	export GIT_VALGRIND
+else # normal case
+	notmuch_path=`find_notmuch_path "$TEST_DIRECTORY"`
+	test -n "$notmuch_path" && PATH="$notmuch_path:$PATH"
+fi
+export PATH
+
+# Test repository
+test="tmp.$(basename "$0" .sh)"
+test -n "$root" && test="$root/$test"
+case "$test" in
+/*) TMP_DIRECTORY="$test" ;;
+ *) TMP_DIRECTORY="$TEST_DIRECTORY/$test" ;;
+esac
+test ! -z "$debug" || remove_tmp=$TMP_DIRECTORY
+rm -fr "$test" || {
+	GIT_EXIT_OK=t
+	echo >&5 "FATAL: Cannot prepare test area"
+	exit 1
+}
+
+# A temporary home directory is needed by at least:
+# - emacs/"Sending a message via (fake) SMTP"
+# - emacs/"Reply within emacs"
+# - crypto/emacs_deliver_message
+export HOME="${TMP_DIRECTORY}/home"
+mkdir -p "${HOME}"
+
+MAIL_DIR="${TMP_DIRECTORY}/mail"
+export GNUPGHOME="${TMP_DIRECTORY}/gnupg"
+export NOTMUCH_CONFIG="${TMP_DIRECTORY}/notmuch-config"
+
+mkdir -p "${test}"
+mkdir -p "${MAIL_DIR}"
+
+cat <<EOF >"${NOTMUCH_CONFIG}"
+[database]
+path=${MAIL_DIR}
+
+[user]
+name=Notmuch Test Suite
+primary_email=test_suite@notmuchmail.org
+other_email=test_suite_other@notmuchmail.org;test_suite@otherdomain.org
+EOF
+
+emacs_generate_script
+
+
+# Use -P to resolve symlinks in our working directory so that the cwd
+# in subprocesses like git equals our $PWD (for pathname comparisons).
+cd -P "$test" || error "Cannot setup test environment"
+
+if test "$verbose" = "t"
+then
+	exec 4>&2 3>&1
+else
+	exec 4>test.output 3>&4
+fi
+
+this_test=${0##*/}
+for skp in $NOTMUCH_SKIP_TESTS
+do
+	to_skip=
+	for skp in $NOTMUCH_SKIP_TESTS
+	do
+		case "$this_test" in
+		$skp)
+			to_skip=t
+		esac
+	done
+	case "$to_skip" in
+	t)
+		say_color skip >&3 "skipping test $this_test altogether"
+		say_color skip "skip all tests in $this_test"
+		test_done
+	esac
+done
+
+# Provide an implementation of the 'yes' utility
+yes () {
+	if test $# = 0
+	then
+		y=y
+	else
+		y="$*"
+	fi
+
+	while echo "$y"
+	do
+		:
+	done
+}
+
+# Fix some commands on Windows
+case $(uname -s) in
+*MINGW*)
+	# Windows has its own (incompatible) sort and find
+	sort () {
+		/usr/bin/sort "$@"
+	}
+	find () {
+		/usr/bin/find "$@"
+	}
+	sum () {
+		md5sum "$@"
+	}
+	# git sees Windows-style pwd
+	pwd () {
+		builtin pwd -W
+	}
+	# no POSIX permissions
+	# backslashes in pathspec are converted to '/'
+	# exec does not inherit the PID
+	;;
+*)
+	test_set_prereq POSIXPERM
+	test_set_prereq BSLASHPSPEC
+	test_set_prereq EXECKEEPSPID
+	;;
+esac
+
+test -z "$NO_PERL" && test_set_prereq PERL
+test -z "$NO_PYTHON" && test_set_prereq PYTHON
+
+# test whether the filesystem supports symbolic links
+ln -s x y 2>/dev/null && test -h y 2>/dev/null && test_set_prereq SYMLINKS
+rm -f y
+
+# declare prerequisites for external binaries used in tests
+test_declare_external_prereq dtach
+test_declare_external_prereq emacs
+test_declare_external_prereq emacsclient
+test_declare_external_prereq gdb
+test_declare_external_prereq gpg
+test_declare_external_prereq python
+test_declare_external_prereq python2
-- 
1.7.5.4

[PATCH v7 02/10] reply: Factor out reply creation

[Adam Wolfe Gordon]

Factor out the creation of a reply message based on an original
message so it can be shared by different reply formats.
---
 notmuch-reply.c |  104 ++++++++++++++++++++++++++++++++----------------------
 1 files changed, 62 insertions(+), 42 deletions(-)

diff --git a/notmuch-reply.c b/notmuch-reply.c
index 6b244e6..f1478cc 100644
--- a/notmuch-reply.c
+++ b/notmuch-reply.c
@@ -505,6 +505,61 @@ guess_from_received_header (notmuch_config_t *config, notmuch_message_t *message
     return NULL;
 }
 
+static GMimeMessage *
+create_reply_message(void *ctx,
+		     notmuch_config_t *config,
+		     notmuch_message_t *message,
+		     notmuch_bool_t reply_all)
+{
+    const char *subject, *from_addr = NULL;
+    const char *in_reply_to, *orig_references, *references;
+
+    /* The 1 means we want headers in a "pretty" order. */
+    GMimeMessage *reply = g_mime_message_new (1);
+    if (reply == NULL) {
+	fprintf (stderr, "Out of memory\n");
+	return NULL;
+    }
+
+    subject = notmuch_message_get_header (message, "subject");
+    if (subject) {
+	if (strncasecmp (subject, "Re:", 3))
+	    subject = talloc_asprintf (ctx, "Re: %s", subject);
+	g_mime_message_set_subject (reply, subject);
+    }
+
+    from_addr = add_recipients_from_message (reply, config,
+					     message, reply_all);
+
+    if (from_addr == NULL)
+	from_addr = guess_from_received_header (config, message);
+
+    if (from_addr == NULL)
+	from_addr = notmuch_config_get_user_primary_email (config);
+
+    from_addr = talloc_asprintf (ctx, "%s <%s>",
+				 notmuch_config_get_user_name (config),
+				 from_addr);
+    g_mime_object_set_header (GMIME_OBJECT (reply),
+			      "From", from_addr);
+
+    in_reply_to = talloc_asprintf (ctx, "<%s>",
+				   notmuch_message_get_message_id (message));
+
+    g_mime_object_set_header (GMIME_OBJECT (reply),
+			      "In-Reply-To", in_reply_to);
+
+    orig_references = notmuch_message_get_header (message, "references");
+    references = talloc_asprintf (ctx, "%s%s%s",
+				  orig_references ? orig_references : "",
+				  orig_references ? " " : "",
+				  in_reply_to);
+    g_mime_object_set_header (GMIME_OBJECT (reply),
+			      "References", references);
+
+    return reply;
+}
+
 static int
 notmuch_reply_format_default(void *ctx,
 			     notmuch_config_t *config,
@@ -515,8 +570,6 @@ notmuch_reply_format_default(void *ctx,
     GMimeMessage *reply;
     notmuch_messages_t *messages;
     notmuch_message_t *message;
-    const char *subject, *from_addr = NULL;
-    const char *in_reply_to, *orig_references, *references;
     const notmuch_show_format_t *format = &format_reply;
 
     for (messages = notmuch_query_search_messages (query);
@@ -525,49 +578,16 @@ notmuch_reply_format_default(void *ctx,
     {
 	message = notmuch_messages_get (messages);
 
-	/* The 1 means we want headers in a "pretty" order. */
-	reply = g_mime_message_new (1);
-	if (reply == NULL) {
-	    fprintf (stderr, "Out of memory\n");
-	    return 1;
-	}
+	reply = create_reply_message (ctx, config, message, reply_all);
 
-	subject = notmuch_message_get_header (message, "subject");
-	if (subject) {
-	    if (strncasecmp (subject, "Re:", 3))
-		subject = talloc_asprintf (ctx, "Re: %s", subject);
-	    g_mime_message_set_subject (reply, subject);
+	/* If reply creation failed, we're out of memory, so don't
+	 * bother trying any more messages.
+	 */
+	if (!reply) {
+	    notmuch_message_destroy (message);
+	    return 1;
 	}
 
-	from_addr = add_recipients_from_message (reply, config, message,
-						 reply_all);
-
-	if (from_addr == NULL)
-	    from_addr = guess_from_received_header (config, message);
-
-	if (from_addr == NULL)
-	    from_addr = notmuch_config_get_user_primary_email (config);
-
-	from_addr = talloc_asprintf (ctx, "%s <%s>",
-				     notmuch_config_get_user_name (config),
-				     from_addr);
-	g_mime_object_set_header (GMIME_OBJECT (reply),
-				  "From", from_addr);
-
-	in_reply_to = talloc_asprintf (ctx, "<%s>",
-			     notmuch_message_get_message_id (message));
-
-	g_mime_object_set_header (GMIME_OBJECT (reply),
-				  "In-Reply-To", in_reply_to);
-
-	orig_references = notmuch_message_get_header (message, "references");
-	references = talloc_asprintf (ctx, "%s%s%s",
-				      orig_references ? orig_references : "",
-				      orig_references ? " " : "",
-				      in_reply_to);
-	g_mime_object_set_header (GMIME_OBJECT (reply),
-				  "References", references);
-
 	show_reply_headers (reply);
 
 	g_object_unref (G_OBJECT (reply));
-- 
1.7.5.4

[PATCH v7 03/10] TODO: Add replying to multiple messages

[Adam Wolfe Gordon]

---
 devel/TODO |    8 ++++++++
 1 files changed, 8 insertions(+), 0 deletions(-)

diff --git a/devel/TODO b/devel/TODO
index 4dda6f4..7b750af 100644
--- a/devel/TODO
+++ b/devel/TODO
@@ -141,6 +141,14 @@ Simplify notmuch-reply to simply print the headers (we have the
 original values) rather than calling GMime (which encodes) and adding
 the confusing gmime-filter-headers.c code (which decodes).
 
+Properly handle replying to multiple messages. Currently, the JSON
+reply format only supports a single message, but the default reply
+format accepts searches returning multiple messages. The expected
+behavior of replying to multiple messages is not obvious, and there
+are multiple ideas that might make sense. Some consensus needs to be
+reached on this issue, and then both reply formats should be updated
+to be consistent.
+
 notmuch library
 ---------------
 Add support for custom flag<->tag mappings. In the notmuch
-- 
1.7.5.4

[PATCH v7 04/10] reply: Add a JSON reply format.

[Adam Wolfe Gordon]

This new JSON format for replies includes headers generated for a
reply message as well as the headers of the original message.  Using
this data, a client can intelligently create a reply. For example, the
emacs client will be able to create replies with quoted HTML parts by
parsing the HTML parts.
---
 notmuch-client.h |   14 ++++++++++----
 notmuch-reply.c  |   49 +++++++++++++++++++++++++++++++++++++++++++++++++
 notmuch-show.c   |   29 +++++++++++++++++++++--------
 test/multipart   |    1 -
 4 files changed, 80 insertions(+), 13 deletions(-)

diff --git a/notmuch-client.h b/notmuch-client.h
index f4a62cc..270daff 100644
--- a/notmuch-client.h
+++ b/notmuch-client.h
@@ -62,13 +62,13 @@
 #define STRINGIFY(s) STRINGIFY_(s)
 #define STRINGIFY_(s) #s
 
-struct mime_node;
+typedef struct mime_node mime_node_t;
 struct notmuch_show_params;
 
 typedef struct notmuch_show_format {
     const char *message_set_start;
     void (*part) (const void *ctx,
-		  struct mime_node *node, int indent,
+		  mime_node_t *node, int indent,
 		  const struct notmuch_show_params *params);
     const char *message_start;
     void (*message) (const void *ctx,
@@ -191,6 +191,12 @@ show_message_body (notmuch_message_t *message,
 notmuch_status_t
 show_one_part (const char *filename, int part);
 
+void
+format_part_json (const void *ctx, mime_node_t *node, notmuch_bool_t first);
+
+void
+format_headers_json (const void *ctx, GMimeMessage *message, notmuch_bool_t reply);
+
 char *
 json_quote_chararray (const void *ctx, const char *str, const size_t len);
 
@@ -288,7 +294,7 @@ debugger_is_active (void);
  * parts.  Message-type parts have one child, multipart-type parts
  * have multiple children, and leaf parts have zero children.
  */
-typedef struct mime_node {
+struct mime_node {
     /* The MIME object of this part.  This will be a GMimeMessage,
      * GMimePart, GMimeMultipart, or a subclass of one of these.
      *
@@ -351,7 +357,7 @@ typedef struct mime_node {
      * number to assign it (or -1 if unknown). */
     int next_child;
     int next_part_num;
-} mime_node_t;
+};
 
 /* Construct a new MIME node pointing to the root message part of
  * message.  If cryptoctx is non-NULL, it will be used to verify
diff --git a/notmuch-reply.c b/notmuch-reply.c
index f1478cc..e2b6c25 100644
--- a/notmuch-reply.c
+++ b/notmuch-reply.c
@@ -604,6 +604,51 @@ notmuch_reply_format_default(void *ctx,
     return 0;
 }
 
+static int
+notmuch_reply_format_json(void *ctx,
+			  notmuch_config_t *config,
+			  notmuch_query_t *query,
+			  notmuch_show_params_t *params,
+			  notmuch_bool_t reply_all)
+{
+    GMimeMessage *reply;
+    notmuch_messages_t *messages;
+    notmuch_message_t *message;
+    mime_node_t *node;
+
+    if (notmuch_query_count_messages (query) != 1) {
+	fprintf (stderr, "Error: search term did not match precisely one message.\n");
+	return 1;
+    }
+
+    messages = notmuch_query_search_messages (query);
+    message = notmuch_messages_get (messages);
+    if (mime_node_open (ctx, message, params->cryptoctx, params->decrypt,
+			&node) != NOTMUCH_STATUS_SUCCESS)
+	return 1;
+
+    reply = create_reply_message (ctx, config, message, reply_all);
+    if (!reply)
+	return 1;
+
+    /* The headers of the reply message we've created */
+    printf ("{\"reply-headers\": ");
+    format_headers_json (ctx, reply, TRUE);
+    g_object_unref (G_OBJECT (reply));
+    reply = NULL;
+
+    /* Start the original */
+    printf (", \"original\": ");
+
+    format_part_json (ctx, node, TRUE);
+
+    /* End */
+    printf ("}\n");
+    notmuch_message_destroy (message);
+
+    return 0;
+}
+
 /* This format is currently tuned for a git send-email --notmuch hook */
 static int
 notmuch_reply_format_headers_only(void *ctx,
@@ -666,6 +711,7 @@ notmuch_reply_format_headers_only(void *ctx,
 
 enum {
     FORMAT_DEFAULT,
+    FORMAT_JSON,
     FORMAT_HEADERS_ONLY,
 };
 
@@ -685,6 +731,7 @@ notmuch_reply_command (void *ctx, int argc, char *argv[])
     notmuch_opt_desc_t options[] = {
 	{ NOTMUCH_OPT_KEYWORD, &format, "format", 'f',
 	  (notmuch_keyword_t []){ { "default", FORMAT_DEFAULT },
+				  { "json", FORMAT_JSON },
 				  { "headers-only", FORMAT_HEADERS_ONLY },
 				  { 0, 0 } } },
 	{ NOTMUCH_OPT_KEYWORD, &reply_all, "reply-to", 'r',
@@ -703,6 +750,8 @@ notmuch_reply_command (void *ctx, int argc, char *argv[])
 
     if (format == FORMAT_HEADERS_ONLY)
 	reply_format_func = notmuch_reply_format_headers_only;
+    else if (format == FORMAT_JSON)
+	reply_format_func = notmuch_reply_format_json;
     else
 	reply_format_func = notmuch_reply_format_default;
 
diff --git a/notmuch-show.c b/notmuch-show.c
index 05d51b2..cb3a427 100644
--- a/notmuch-show.c
+++ b/notmuch-show.c
@@ -291,8 +291,8 @@ format_headers_message_part_text (GMimeMessage *message)
     printf ("Date: %s\n", g_mime_message_get_date_as_string (message));
 }
 
-static void
-format_headers_json (const void *ctx, GMimeMessage *message)
+void
+format_headers_json (const void *ctx, GMimeMessage *message, notmuch_bool_t reply)
 {
     void *local = talloc_new (ctx);
     InternetAddressList *recipients;
@@ -316,9 +316,22 @@ format_headers_json (const void *ctx, GMimeMessage *message)
 	printf (", %s: %s",
 		json_quote_str (local, "Cc"),
 		json_quote_str (local, recipients_string));
-    printf (", %s: %s}",
-	    json_quote_str (local, "Date"),
-	    json_quote_str (local, g_mime_message_get_date_as_string (message)));
+
+    if (reply) {
+	printf (", %s: %s",
+		json_quote_str (local, "In-reply-to"),
+		json_quote_str (local, g_mime_object_get_header (GMIME_OBJECT (message), "In-reply-to")));
+
+	printf (", %s: %s",
+		json_quote_str (local, "References"),
+		json_quote_str (local, g_mime_object_get_header (GMIME_OBJECT (message), "References")));
+    } else {
+	printf (", %s: %s",
+		json_quote_str (local, "Date"),
+		json_quote_str (local, g_mime_message_get_date_as_string (message)));
+    }
+
+    printf ("}");
 
     talloc_free (local);
 }
@@ -654,7 +667,7 @@ format_part_text (const void *ctx, mime_node_t *node,
     printf ("\f%s}\n", part_type);
 }
 
-static void
+void
 format_part_json (const void *ctx, mime_node_t *node, notmuch_bool_t first)
 {
     /* Any changes to the JSON format should be reflected in the file
@@ -665,7 +678,7 @@ format_part_json (const void *ctx, mime_node_t *node, notmuch_bool_t first)
 	format_message_json (ctx, node->envelope_file);
 
 	printf ("\"headers\": ");
-	format_headers_json (ctx, GMIME_MESSAGE (node->part));
+	format_headers_json (ctx, GMIME_MESSAGE (node->part), FALSE);
 
 	printf (", \"body\": [");
 	format_part_json (ctx, mime_node_child (node, 0), first);
@@ -739,7 +752,7 @@ format_part_json (const void *ctx, mime_node_t *node, notmuch_bool_t first)
     } else if (GMIME_IS_MESSAGE (node->part)) {
 	printf (", \"content\": [{");
 	printf ("\"headers\": ");
-	format_headers_json (local, GMIME_MESSAGE (node->part));
+	format_headers_json (local, GMIME_MESSAGE (node->part), FALSE);
 
 	printf (", \"body\": [");
 	terminator = "]}]";
diff --git a/test/multipart b/test/multipart
index 80d6e88..2383b9c 100755
--- a/test/multipart
+++ b/test/multipart
@@ -590,7 +590,6 @@ EOF
 test_expect_equal_file OUTPUT EXPECTED
 
 test_begin_subtest "'notmuch reply' to a multipart message with json format"
-test_subtest_known_broken
 notmuch reply --format=json 'id:87liy5ap00.fsf@yoom.home.cworth.org' | notmuch_json_show_sanitize >OUTPUT
 cat <<EOF >EXPECTED
 {"reply-headers": {"Subject": "Re: Multipart message",
-- 
1.7.5.4

[PATCH v7 05/10] schemata: Add documentation for JSON reply format.

[Adam Wolfe Gordon]

---
 devel/schemata |   27 +++++++++++++++++++++++++--
 1 files changed, 25 insertions(+), 2 deletions(-)

diff --git a/devel/schemata b/devel/schemata
index 24ad775..b1073a9 100644
--- a/devel/schemata
+++ b/devel/schemata
@@ -77,8 +77,9 @@ part = {
     content?:       string
 }
 
-# The headers of a message (format_headers_json with raw headers) or
-# a part (format_headers_message_part_json with pretty-printed headers)
+# The headers of a message (format_headers_json with raw headers
+# and reply = FALSE) or a part (format_headers_message_part_json
+# with pretty-printed headers)
 headers = {
     Subject:        string,
     From:           string,
@@ -136,3 +137,25 @@ thread = {
                               # matched and unmatched
     subject:        string
 }
+
+notmuch reply schema
+--------------------
+
+reply = {
+    # The headers of the constructed reply (format_headers_json with
+    # raw headers and reply = TRUE)
+    reply-headers: reply_headers,
+
+    # As in the show format (format_message_json)
+    original: message
+}
+
+reply_headers = {
+    Subject:        string,
+    From:           string,
+    To?:            string,
+    Cc?:            string,
+    Bcc?:           string,
+    In-reply-to:    string,
+    References:     string
+}
-- 
1.7.5.4

[PATCH v7 06/10] man: Update notmuch-reply man page for JSON format.

[Adam Wolfe Gordon]

---
 man/man1/notmuch-reply.1 |   10 ++++++++--
 1 files changed, 8 insertions(+), 2 deletions(-)

diff --git a/man/man1/notmuch-reply.1 b/man/man1/notmuch-reply.1
index 7ed7f0f..debc505 100644
--- a/man/man1/notmuch-reply.1
+++ b/man/man1/notmuch-reply.1
@@ -37,12 +37,17 @@ Supported options for
 include
 .RS
 .TP 4
-.BR \-\-format= ( default | headers\-only )
+.BR \-\-format= ( default | json | headers\-only )
 .RS
 .TP 4
 .BR default
 Includes subject and quoted message body.
 .TP
+.BR json
+Produces JSON output containing headers for a reply message and the
+contents of the original message. This output can be used by a client
+to create a reply message intelligently.
+.TP
 .BR headers\-only
 Only produces In\-Reply\-To, References, To, Cc, and Bcc headers.
 .RE
@@ -73,7 +78,8 @@ with a search string matching a single message, (such as
 id:<message-id>), but it can be useful to reply to several messages at
 once. For example, when a series of patches are sent in a single
 thread, replying to the entire thread allows for the reply to comment
-on issue found in multiple patches.
+on issues found in multiple patches. The default format supports
+replying to multiple messages at once, but the JSON format does not.
 .RE
 .RE
 
-- 
1.7.5.4

[PATCH v7 07/10] man: Add --decrypt to reply flags

[Adam Wolfe Gordon]

---
 man/man1/notmuch-reply.1 |   10 ++++++++++
 1 files changed, 10 insertions(+), 0 deletions(-)

diff --git a/man/man1/notmuch-reply.1 b/man/man1/notmuch-reply.1
index 36b60cb..e21559a 100644
--- a/man/man1/notmuch-reply.1
+++ b/man/man1/notmuch-reply.1
@@ -68,6 +68,16 @@ values from the first that contains something other than only the
 user's addresses.
 .RE
 .RE
+.RS
+.TP 4
+.B \-\-decrypt
+
+Decrypt any MIME encrypted parts found in the selected content
+(ie. "multipart/encrypted" parts). Status of the decryption will be
+reported (currently only supported with --format=json) and the
+multipart/encrypted part will be replaced by the decrypted
+content.
+.RE
 
 See \fBnotmuch-search-terms\fR(7)
 for details of the supported syntax for <search-terms>.
-- 
1.7.5.4

[PATCH v7 08/10] emacs: Factor out useful functions into notmuch-lib

[Adam Wolfe Gordon]

Move a few functions related to handling multipart/alternative parts
into notmuch-lib.el, so they can be used by future reply code.
---
 emacs/notmuch-lib.el  |   33 +++++++++++++++++++++++++++++++++
 emacs/notmuch-show.el |   24 ++----------------------
 2 files changed, 35 insertions(+), 22 deletions(-)

diff --git a/emacs/notmuch-lib.el b/emacs/notmuch-lib.el
index d315f76..7e3f110 100644
--- a/emacs/notmuch-lib.el
+++ b/emacs/notmuch-lib.el
@@ -21,6 +21,8 @@
 
 ;; This is an part of an emacs-based interface to the notmuch mail system.
 
+(eval-when-compile (require 'cl))
+
 (defvar notmuch-command "notmuch"
   "Command to run the notmuch binary.")
 
@@ -173,6 +175,37 @@ the user hasn't set this variable with the old or new value."
   (list 'when (< emacs-major-version 23)
 	form))
 
+(defun notmuch-split-content-type (content-type)
+  "Split content/type into 'content' and 'type'"
+  (split-string content-type "/"))
+
+(defun notmuch-match-content-type (t1 t2)
+  "Return t if t1 and t2 are matching content types, taking wildcards into account"
+  (let ((st1 (notmuch-split-content-type t1))
+	(st2 (notmuch-split-content-type t2)))
+    (if (or (string= (cadr st1) "*")
+	    (string= (cadr st2) "*"))
+	(string= (car st1) (car st2))
+      (string= t1 t2))))
+
+(defvar notmuch-multipart/alternative-discouraged
+  '(
+    ;; Avoid HTML parts.
+    "text/html"
+    ;; multipart/related usually contain a text/html part and some associated graphics.
+    "multipart/related"
+    ))
+
+(defun notmuch-multipart/alternative-choose (types)
+  "Return a list of preferred types from the given list of types"
+  ;; Based on `mm-preferred-alternative-precedence'.
+  (let ((seq types))
+    (dolist (pref (reverse notmuch-multipart/alternative-discouraged))
+      (dolist (elem (copy-sequence seq))
+	(when (string-match pref elem)
+	  (setq seq (nconc (delete elem seq) (list elem))))))
+    seq))
+
 ;; Compatibility functions for versions of emacs before emacs 23.
 ;;
 ;; Both functions here were copied from emacs 23 with the following copyright:
diff --git a/emacs/notmuch-show.el b/emacs/notmuch-show.el
index 4a60631..ed938bf 100644
--- a/emacs/notmuch-show.el
+++ b/emacs/notmuch-show.el
@@ -542,30 +542,13 @@ current buffer, if possible."
 	    (mm-display-part handle)
 	    t))))))
 
-(defvar notmuch-show-multipart/alternative-discouraged
-  '(
-    ;; Avoid HTML parts.
-    "text/html"
-    ;; multipart/related usually contain a text/html part and some associated graphics.
-    "multipart/related"
-    ))
-
 (defun notmuch-show-multipart/*-to-list (part)
   (mapcar (lambda (inner-part) (plist-get inner-part :content-type))
 	  (plist-get part :content)))
 
-(defun notmuch-show-multipart/alternative-choose (types)
-  ;; Based on `mm-preferred-alternative-precedence'.
-  (let ((seq types))
-    (dolist (pref (reverse notmuch-show-multipart/alternative-discouraged))
-      (dolist (elem (copy-sequence seq))
-	(when (string-match pref elem)
-	  (setq seq (nconc (delete elem seq) (list elem))))))
-    seq))
-
 (defun notmuch-show-insert-part-multipart/alternative (msg part content-type nth depth declared-type)
   (notmuch-show-insert-part-header nth declared-type content-type nil)
-  (let ((chosen-type (car (notmuch-show-multipart/alternative-choose (notmuch-show-multipart/*-to-list part))))
+  (let ((chosen-type (car (notmuch-multipart/alternative-choose (notmuch-show-multipart/*-to-list part))))
 	(inner-parts (plist-get part :content))
 	(start (point)))
     ;; This inserts all parts of the chosen type rather than just one,
@@ -808,9 +791,6 @@ current buffer, if possible."
 
 ;; Functions for determining how to handle MIME parts.
 
-(defun notmuch-show-split-content-type (content-type)
-  (split-string content-type "/"))
-
 (defun notmuch-show-handlers-for (content-type)
   "Return a list of content handlers for a part of type CONTENT-TYPE."
   (let (result)
@@ -821,7 +801,7 @@ current buffer, if possible."
 	  (list (intern (concat "notmuch-show-insert-part-*/*"))
 		(intern (concat
 			 "notmuch-show-insert-part-"
-			 (car (notmuch-show-split-content-type content-type))
+			 (car (notmuch-split-content-type content-type))
 			 "/*"))
 		(intern (concat "notmuch-show-insert-part-" content-type))))
     result))
-- 
1.7.5.4

[PATCH v7 09/10] test: Add broken tests for new emacs reply functionality

[Adam Wolfe Gordon]

Add tests for creating nice replies to multipart messages, including
those with HTML parts. These tests are expected to fail for now.
---
 test/emacs |   97 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 files changed, 97 insertions(+), 0 deletions(-)

diff --git a/test/emacs b/test/emacs
index 29a489c..01afdb6 100755
--- a/test/emacs
+++ b/test/emacs
@@ -273,6 +273,103 @@ On 01 Jan 2000 12:00:00 -0000, Notmuch Test Suite <test_suite@notmuchmail.org> w
 EOF
 test_expect_equal_file OUTPUT EXPECTED
 
+test_begin_subtest "Reply within emacs to a multipart/mixed message"
+test_subtest_known_broken
+test_emacs '(notmuch-show "id:20091118002059.067214ed@hikari")
+		(notmuch-show-reply)
+		(test-output)'
+cat <<EOF >EXPECTED
+From: Notmuch Test Suite <test_suite@notmuchmail.org>
+To: Adrian Perez de Castro <aperez@igalia.com>, notmuch@notmuchmail.org
+Subject: Re: [notmuch] Introducing myself
+In-Reply-To: <20091118002059.067214ed@hikari>
+Fcc: ${MAIL_DIR}/sent
+--text follows this line--
+Adrian Perez de Castro <aperez@igalia.com> writes:
+
+> Hello to all,
+>
+> I have just heard about Not Much today in some random Linux-related news
+> site (LWN?), my name is Adrian Perez and I work as systems administrator
+> (although I can do some code as well :P). I have always thought that the
+> ideas behind Sup were great, but after some time using it, I got tired of
+> the oddities that it has. I also do not like doing things like having to
+> install Ruby just for reading and sorting mails. Some time ago I thought
+> about doing something like Not Much and in fact I played a bit with the
+> Python+Xapian and the Python+Whoosh combinations, because I find relaxing
+> to code things in Python when I am not working and also it is installed
+> by default on most distribution. I got to have some mailboxes indexed and
+> basic searching working a couple of months ago. Lately I have been very
+> busy and had no time for coding, and them... boom! Not Much appears -- and
+> it is almost exactly what I was trying to do, but faster. I have been
+> playing a bit with Not Much today, and I think it has potential.
+>
+> Also, I would like to share one idea I had in mind, that you might find
+> interesting: One thing I have found very annoying is having to re-tag my
+> mail when the indexes get b0rked (it happened a couple of times to me while
+> using Sup), so I was planning to mails as read/unread and adding the tags
+> not just to the index, but to the mail text itself, e.g. by adding a
+> "X-Tags" header field or by reusing the "Keywords" one. This way, the index
+> could be totally recreated by re-reading the mail directories, and this
+> would also allow to a tools like OfflineIMAP [1] to get the mails into a
+> local maildir, tagging and indexing the mails with the e-mail reader and
+> then syncing back the messages with the "X-Tags" header to the IMAP server.
+> This would allow to use the mail reader from a different computer and still
+> have everything tagged finely.
+>
+> Best regards,
+>
+>
+> ---
+> [1] http://software.complete.org/software/projects/show/offlineimap
+>
+> -- 
+> Adrian Perez de Castro <aperez@igalia.com>
+> Igalia - Free Software Engineering
+> _______________________________________________
+> notmuch mailing list
+> notmuch@notmuchmail.org
+> http://notmuchmail.org/mailman/listinfo/notmuch
+EOF
+test_expect_equal_file OUTPUT EXPECTED
+
+test_begin_subtest "Reply within emacs to a multipart/alternative message"
+test_subtest_known_broken
+test_emacs '(notmuch-show "id:cf0c4d610911171136h1713aa59w9cf9aa31f052ad0a@mail.gmail.com")
+		(notmuch-show-reply)
+		(test-output)'
+cat <<EOF >EXPECTED
+From: Notmuch Test Suite <test_suite@notmuchmail.org>
+To: Alex Botero-Lowry <alex.boterolowry@gmail.com>, notmuch@notmuchmail.org
+Subject: Re: [notmuch] preliminary FreeBSD support
+In-Reply-To: <cf0c4d610911171136h1713aa59w9cf9aa31f052ad0a@mail.gmail.com>
+Fcc: ${MAIL_DIR}/sent
+--text follows this line--
+Alex Botero-Lowry <alex.boterolowry@gmail.com> writes:
+
+> I saw the announcement this morning, and was very excited, as I had been
+> hoping sup would be turned into a library,
+> since I like the concept more than the UI (I'd rather an emacs interface).
+>
+> I did a preliminary compile which worked out fine, but
+> sysconf(_SC_SC_GETPW_R_SIZE_MAX) returns -1 on
+> FreeBSD, so notmuch_config_open segfaulted.
+>
+> Attached is a patch that supplies a default buffer size of 64 in cases where
+> -1 is returned.
+>
+> http://www.opengroup.org/austin/docs/austin_328.txt - seems to indicate this
+> is acceptable behavior,
+> and http://mail-index.netbsd.org/pkgsrc-bugs/2006/06/07/msg016808.htmlspecifically
+> uses 64 as the
+> buffer size.
+> _______________________________________________
+> notmuch mailing list
+> notmuch@notmuchmail.org
+> http://notmuchmail.org/mailman/listinfo/notmuch
+EOF
+test_expect_equal_file OUTPUT EXPECTED
+
 test_begin_subtest "Quote MML tags in reply"
 message_id='test-emacs-mml-quoting@message.id'
 add_message [id]="$message_id" \
-- 
1.7.5.4

[PATCH v7 10/10] emacs: Use the new JSON reply format and message-cite-original

[Adam Wolfe Gordon]

Use the new JSON reply format to create replies in emacs. Quote HTML
parts nicely by using mm-display-part to turn them into displayable
text, then quoting them with message-cite-original. This is very
useful for users who regularly receive HTML-only email.

Use message-mode's message-cite-original function to create the
quoted body for reply messages. In order to make this act like the
existing notmuch defaults, you will need to set the following in
your emacs configuration:

message-citation-line-format "On %a, %d %b %Y, %f wrote:"
message-citation-line-function 'message-insert-formatted-citation-line

The tests have been updated to reflect the (ugly) emacs default.
---
 emacs/notmuch-lib.el |   11 ++++
 emacs/notmuch-mua.el |  136 +++++++++++++++++++++++++++++++++++---------------
 test/emacs           |    8 ++--
 3 files changed, 110 insertions(+), 45 deletions(-)

diff --git a/emacs/notmuch-lib.el b/emacs/notmuch-lib.el
index 7e3f110..8bac596 100644
--- a/emacs/notmuch-lib.el
+++ b/emacs/notmuch-lib.el
@@ -206,6 +206,17 @@ the user hasn't set this variable with the old or new value."
 	  (setq seq (nconc (delete elem seq) (list elem))))))
     seq))
 
+(defun notmuch-parts-filter-by-type (parts type)
+  "Given a list of message parts, return a list containing the ones matching
+the given type."
+  (remove-if-not
+   (lambda (part) (notmuch-match-content-type (plist-get part :content-type) type))
+   parts))
+
+(defun notmuch-plist-to-alist (plist)
+  (loop for (key value . rest) on plist by #'cddr
+	collect (cons (substring (symbol-name key) 1) value)))
+
 ;; Compatibility functions for versions of emacs before emacs 23.
 ;;
 ;; Both functions here were copied from emacs 23 with the following copyright:
diff --git a/emacs/notmuch-mua.el b/emacs/notmuch-mua.el
index 13244eb..5adf4d8 100644
--- a/emacs/notmuch-mua.el
+++ b/emacs/notmuch-mua.el
@@ -19,11 +19,15 @@
 ;;
 ;; Authors: David Edmondson <dme@dme.org>
 
+(require 'json)
 (require 'message)
+(require 'format-spec)
 
 (require 'notmuch-lib)
 (require 'notmuch-address)
 
+(eval-when-compile (require 'cl))
+
 ;;
 
 (defcustom notmuch-mua-send-hook '(notmuch-mua-message-send-hook)
@@ -72,54 +76,104 @@ list."
 	    (push header message-hidden-headers)))
 	notmuch-mua-hidden-headers))
 
+(defun notmuch-mua-get-displayed-part (part query-string)
+  (with-temp-buffer
+    (if (plist-get part :content)
+	(insert (plist-get part :content))
+      (call-process notmuch-command nil t nil "show" "--format=raw"
+		    (format "--part=%s" (plist-get part :id))
+		    query-string))
+
+    (let ((handle (mm-make-handle (current-buffer) (list (plist-get part :content-type))))
+	  (end-of-orig (point-max)))
+      (mm-display-part handle)
+      (delete-region (point-min) end-of-orig)
+      (buffer-substring (point-min) (point-max)))))
+
+(defun notmuch-mua-get-quotable-parts (parts)
+  (loop for part in parts
+	if (notmuch-match-content-type (plist-get part :content-type) "multipart/alternative")
+	  collect (let* ((subparts (plist-get part :content))
+			(types (mapcar (lambda (part) (plist-get part :content-type)) subparts))
+			(chosen-type (car (notmuch-multipart/alternative-choose types))))
+		   (loop for part in (reverse subparts)
+			 if (notmuch-match-content-type (plist-get part :content-type) chosen-type)
+			 return part))
+	else if (notmuch-match-content-type (plist-get part :content-type) "multipart/*")
+	  append (notmuch-mua-get-quotable-parts (plist-get part :content))
+	else if (notmuch-match-content-type (plist-get part :content-type) "text/*")
+	  collect part))
+
 (defun notmuch-mua-reply (query-string &optional sender reply-all)
-  (let (headers
-	body
-	(args '("reply")))
-    (if notmuch-show-process-crypto
-	(setq args (append args '("--decrypt"))))
+  (let ((args '("reply" "--format=json"))
+	(json-object-type 'plist)
+	(json-array-type 'list)
+	(json-false 'nil)
+	reply
+	original)
+    (when notmuch-show-process-crypto
+      (setq args (append args '("--decrypt"))))
+
     (if reply-all
 	(setq args (append args '("--reply-to=all")))
       (setq args (append args '("--reply-to=sender"))))
     (setq args (append args (list query-string)))
-    ;; This make assumptions about the output of `notmuch reply', but
-    ;; really only that the headers come first followed by a blank
-    ;; line and then the body.
+
+    ;; Get the reply object as JSON, and parse it into an elisp object.
     (with-temp-buffer
       (apply 'call-process (append (list notmuch-command nil (list t t) nil) args))
       (goto-char (point-min))
-      (if (re-search-forward "^$" nil t)
-	  (save-excursion
-	    (save-restriction
-	      (narrow-to-region (point-min) (point))
-	      (goto-char (point-min))
-	      (setq headers (mail-header-extract)))))
-      (forward-line 1)
-      ;; Original message may contain (malicious) MML tags. We must
-      ;; properly quote them in the reply.
-      (mml-quote-region (point) (point-max))
-      (setq body (buffer-substring (point) (point-max))))
-    ;; If sender is non-nil, set the From: header to its value.
-    (when sender
-      (mail-header-set 'from sender headers))
-    (let
-	;; Overlay the composition window on that being used to read
-	;; the original message.
-	((same-window-regexps '("\\*mail .*")))
-      (notmuch-mua-mail (mail-header 'to headers)
-			(mail-header 'subject headers)
-			(message-headers-to-generate headers t '(to subject))))
-    ;; insert the message body - but put it in front of the signature
-    ;; if one is present
-    (goto-char (point-max))
-    (if (re-search-backward message-signature-separator nil t)
+      (setq reply (json-read)))
+
+    ;; Extract the original message to simplify the following code.
+    (setq original (plist-get reply :original))
+
+    ;; Extract the headers of both the reply and the original message.
+    (let* ((original-headers (plist-get original :headers))
+	   (reply-headers (plist-get reply :reply-headers)))
+
+      ;; If sender is non-nil, set the From: header to its value.
+      (when sender
+	(plist-put reply-headers :From sender))
+      (let
+	  ;; Overlay the composition window on that being used to read
+	  ;; the original message.
+	  ((same-window-regexps '("\\*mail .*")))
+	(notmuch-mua-mail (plist-get reply-headers :To)
+			  (plist-get reply-headers :Subject)
+			  (notmuch-plist-to-alist reply-headers)))
+      ;; Insert the message body - but put it in front of the signature
+      ;; if one is present
+      (goto-char (point-max))
+      (if (re-search-backward message-signature-separator nil t)
 	  (forward-line -1)
-      (goto-char (point-max)))
-    (insert body)
-    (push-mark))
-  (set-buffer-modified-p nil)
-
-  (message-goto-body))
+	(goto-char (point-max)))
+
+      (let ((from (plist-get original-headers :From))
+	    (date (plist-get original-headers :Date))
+	    (start (point)))
+
+	;; message-cite-original constructs a citation line based on the From and Date
+	;; headers of the original message, which are assumed to be in the buffer.
+	(insert "From: " from "\n")
+	(insert "Date: " date "\n\n")
+
+	;; Get the parts of the original message that should be quoted; this includes
+	;; all the text parts, except the non-preferred ones in a multipart/alternative.
+	(let ((quotable-parts (notmuch-mua-get-quotable-parts (plist-get original :body))))
+	  (mapc (lambda (part)
+		  (insert (notmuch-mua-get-displayed-part part query-string)))
+		quotable-parts))
+
+	(set-mark (point))
+	(goto-char start)
+	;; Quote the original message according to the user's configured style.
+	(message-cite-original)
+	(goto-char (point-max)))))
+
+  (push-mark)
+  (message-goto-body)
+  (set-buffer-modified-p nil))
 
 (defun notmuch-mua-forward-message ()
   (message-forward)
@@ -145,7 +199,7 @@ OTHER-ARGS are passed through to `message-mail'."
       (when (not (string= "" user-agent))
 	(push (cons "User-Agent" user-agent) other-headers))))
 
-  (unless (mail-header 'from other-headers)
+  (unless (mail-header 'From other-headers)
     (push (cons "From" (concat
 			(notmuch-user-name) " <" (notmuch-user-primary-email) ">")) other-headers))
 
@@ -208,7 +262,7 @@ the From: address first."
   (interactive "P")
   (let ((other-headers
 	 (when (or prompt-for-sender notmuch-always-prompt-for-sender)
-	   (list (cons 'from (notmuch-mua-prompt-for-sender))))))
+	   (list (cons 'From (notmuch-mua-prompt-for-sender))))))
     (notmuch-mua-mail nil nil other-headers)))
 
 (defun notmuch-mua-new-forward-message (&optional prompt-for-sender)
diff --git a/test/emacs b/test/emacs
index 01afdb6..8a28705 100755
--- a/test/emacs
+++ b/test/emacs
@@ -268,13 +268,13 @@ Subject: Re: Testing message sent via SMTP
 In-Reply-To: <XXX>
 Fcc: ${MAIL_DIR}/sent
 --text follows this line--
-On 01 Jan 2000 12:00:00 -0000, Notmuch Test Suite <test_suite@notmuchmail.org> wrote:
+Notmuch Test Suite <test_suite@notmuchmail.org> writes:
+
 > This is a test that messages are sent via SMTP
 EOF
 test_expect_equal_file OUTPUT EXPECTED
 
 test_begin_subtest "Reply within emacs to a multipart/mixed message"
-test_subtest_known_broken
 test_emacs '(notmuch-show "id:20091118002059.067214ed@hikari")
 		(notmuch-show-reply)
 		(test-output)'
@@ -334,7 +334,6 @@ EOF
 test_expect_equal_file OUTPUT EXPECTED
 
 test_begin_subtest "Reply within emacs to a multipart/alternative message"
-test_subtest_known_broken
 test_emacs '(notmuch-show "id:cf0c4d610911171136h1713aa59w9cf9aa31f052ad0a@mail.gmail.com")
 		(notmuch-show-reply)
 		(test-output)'
@@ -385,7 +384,8 @@ Subject: Re: Quote MML tags in reply
 In-Reply-To: <test-emacs-mml-quoting@message.id>
 Fcc: ${MAIL_DIR}/sent
 --text follows this line--
-On Fri, 05 Jan 2001 15:43:57 +0000, Notmuch Test Suite <test_suite@notmuchmail.org> wrote:
+Notmuch Test Suite <test_suite@notmuchmail.org> writes:
+
 > <#!part disposition=inline>
 EOF
 test_expect_equal_file OUTPUT EXPECTED
-- 
1.7.5.4

Re: [PATCH v7 01/10] test: Add broken test for the new JSON reply format.

[Austin Clements]

Quoth Adam Wolfe Gordon on Mar 11 at 10:05 pm:
> ---
>  test/multipart |   53 +++
>  test/test-lib  | 1242 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>  2 files changed, 1295 insertions(+), 0 deletions(-)
>  create mode 100755 test/test-lib
> 
> diff --git a/test/multipart b/test/multipart
> index 53782c6..80d6e88 100755
> --- a/test/multipart
> +++ b/test/multipart
> @@ -589,6 +589,59 @@ Non-text part: text/html
>  EOF
>  test_expect_equal_file OUTPUT EXPECTED
>  
> +test_begin_subtest "'notmuch reply' to a multipart message with json format"
> +test_subtest_known_broken
> +notmuch reply --format=json 'id:87liy5ap00.fsf@yoom.home.cworth.org' | notmuch_json_show_sanitize >OUTPUT
> +cat <<EOF >EXPECTED
> +{"reply-headers": {"Subject": "Re: Multipart message",
> + "From": "Notmuch Test Suite <test_suite@notmuchmail.org>",
> + "To": "Carl Worth <cworth@cworth.org>,
> + cworth@cworth.org",
> + "In-reply-to": "<87liy5ap00.fsf@yoom.home.cworth.org>",
> + "References": " <87liy5ap00.fsf@yoom.home.cworth.org>"},
> + "original": {"id": "XXXXX",
> + "match": false,
> + "excluded": false,
> + "filename": "YYYYY",
> + "timestamp": 978709437,
> + "date_relative": "2001-01-05",
> + "tags": ["attachment","inbox","signed","unread"],
> + "headers": {"Subject": "Multipart message",
> + "From": "Carl Worth <cworth@cworth.org>",
> + "To": "cworth@cworth.org",
> + "Date": "Fri,
> + 05 Jan 2001 15:43:57 +0000"},
> + "body": [{"id": 1,
> + "content-type": "multipart/signed",
> + "content": [{"id": 2,
> + "content-type": "multipart/mixed",
> + "content": [{"id": 3,
> + "content-type": "message/rfc822",
> + "content": [{"headers": {"Subject": "html message",
> + "From": "Carl Worth <cworth@cworth.org>",
> + "To": "cworth@cworth.org",
> + "Date": "Fri,
> + 05 Jan 2001 15:42:57 +0000"},
> + "body": [{"id": 4,
> + "content-type": "multipart/alternative",
> + "content": [{"id": 5,
> + "content-type": "text/html"},
> + {"id": 6,
> + "content-type": "text/plain",
> + "content": "This is an embedded message,
> + with a multipart/alternative part.\n"}]}]}]},
> + {"id": 7,
> + "content-type": "text/plain",
> + "filename": "YYYYY",
> + "content": "This is a text attachment.\n"},
> + {"id": 8,
> + "content-type": "text/plain",
> + "content": "And this message is signed.\n\n-Carl\n"}]},
> + {"id": 9,
> + "content-type": "application/pgp-signature"}]}]}}
> +EOF
> +test_expect_equal_file OUTPUT EXPECTED
> +
>  test_begin_subtest "'notmuch show --part' does not corrupt a part with CRLF pair"
>  notmuch show --format=raw --part=3 id:base64-part-with-crlf > crlf.out
>  echo -n -e "\xEF\x0D\x0A" > crlf.expected
> diff --git a/test/test-lib b/test/test-lib
> new file mode 100755
> index 0000000..8158328
> --- /dev/null
> +++ b/test/test-lib

Spurious git add?  (IIRC, this is what test-lib.sh used to be called,
so I'd guess you have an old copy of this file laying around.)

> @@ -0,0 +1,1242 @@
> +#
> +# Copyright (c) 2005 Junio C Hamano
> +#
> +# This program is free software: you can redistribute it and/or modify
> +# it under the terms of the GNU General Public License as published by
> +# the Free Software Foundation, either version 2 of the License, or
> +# (at your option) any later version.
> +#
> +# This program is distributed in the hope that it will be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along with this program.  If not, see http://www.gnu.org/licenses/ .
> +
> +if [ ${BASH_VERSINFO[0]} -lt 4 ]; then
> +    echo "Error: The notmuch test suite requires a bash version >= 4.0"
> +    echo "due to use of associative arrays within the test suite."
> +    echo "Please try again with a newer bash (or help us fix the"
> +    echo "test suite to be more portable). Thanks."
> +    exit 1
> +fi
> +
> +# if --tee was passed, write the output not only to the terminal, but
> +# additionally to the file test-results/$BASENAME.out, too.
> +case "$GIT_TEST_TEE_STARTED, $* " in
> +done,*)
> +	# do not redirect again
> +	;;
> +*' --tee '*|*' --va'*)
> +	mkdir -p test-results
> +	BASE=test-results/$(basename "$0" .sh)
> +	(GIT_TEST_TEE_STARTED=done ${SHELL-sh} "$0" "$@" 2>&1;
> +	 echo $? > $BASE.exit) | tee $BASE.out
> +	test "$(cat $BASE.exit)" = 0
> +	exit
> +	;;
> +esac
> +
> +# Keep the original TERM for say_color and test_emacs
> +ORIGINAL_TERM=$TERM
> +
> +# For repeatability, reset the environment to known value.
> +LANG=C
> +LC_ALL=C
> +PAGER=cat
> +TZ=UTC
> +TERM=dumb
> +export LANG LC_ALL PAGER TERM TZ
> +GIT_TEST_CMP=${GIT_TEST_CMP:-diff -u}
> +TEST_EMACS=${TEST_EMACS:-${EMACS:-emacs}}
> +
> +# Protect ourselves from common misconfiguration to export
> +# CDPATH into the environment
> +unset CDPATH
> +
> +unset GREP_OPTIONS
> +
> +# Convenience
> +#
> +# A regexp to match 5 and 40 hexdigits
> +_x05='[0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f]'
> +_x40="$_x05$_x05$_x05$_x05$_x05$_x05$_x05$_x05"
> +
> +_x04='[0-9a-f][0-9a-f][0-9a-f][0-9a-f]'
> +_x32="$_x04$_x04$_x04$_x04$_x04$_x04$_x04$_x04"
> +
> +# Each test should start with something like this, after copyright notices:
> +#
> +# test_description='Description of this test...
> +# This test checks if command xyzzy does the right thing...
> +# '
> +# . ./test-lib.sh
> +[ "x$ORIGINAL_TERM" != "xdumb" ] && (
> +		TERM=$ORIGINAL_TERM &&
> +		export TERM &&
> +		[ -t 1 ] &&
> +		tput bold >/dev/null 2>&1 &&
> +		tput setaf 1 >/dev/null 2>&1 &&
> +		tput sgr0 >/dev/null 2>&1
> +	) &&
> +	color=t
> +
> +while test "$#" -ne 0
> +do
> +	case "$1" in
> +	-d|--d|--de|--deb|--debu|--debug)
> +		debug=t; shift ;;
> +	-i|--i|--im|--imm|--imme|--immed|--immedi|--immedia|--immediat|--immediate)
> +		immediate=t; shift ;;
> +	-l|--l|--lo|--lon|--long|--long-|--long-t|--long-te|--long-tes|--long-test|--long-tests)
> +		GIT_TEST_LONG=t; export GIT_TEST_LONG; shift ;;
> +	-h|--h|--he|--hel|--help)
> +		help=t; shift ;;
> +	-v|--v|--ve|--ver|--verb|--verbo|--verbos|--verbose)
> +		verbose=t; shift ;;
> +	-q|--q|--qu|--qui|--quie|--quiet)
> +		quiet=t; shift ;;
> +	--with-dashes)
> +		with_dashes=t; shift ;;
> +	--no-color)
> +		color=; shift ;;
> +	--no-python)
> +		# noop now...
> +		shift ;;
> +	--va|--val|--valg|--valgr|--valgri|--valgrin|--valgrind)
> +		valgrind=t; verbose=t; shift ;;
> +	--tee)
> +		shift ;; # was handled already
> +	--root=*)
> +		root=$(expr "z$1" : 'z[^=]*=\(.*\)')
> +		shift ;;
> +	*)
> +		echo "error: unknown test option '$1'" >&2; exit 1 ;;
> +	esac
> +done
> +
> +if test -n "$debug"; then
> +    print_subtest () {
> +	printf " %-4s" "[$((test_count - 1))]"
> +    }
> +else
> +    print_subtest () {
> +	true
> +    }
> +fi
> +
> +if test -n "$color"; then
> +	say_color () {
> +		(
> +		TERM=$ORIGINAL_TERM
> +		export TERM
> +		case "$1" in
> +			error) tput bold; tput setaf 1;; # bold red
> +			skip)  tput bold; tput setaf 2;; # bold green
> +			pass)  tput setaf 2;;            # green
> +			info)  tput setaf 3;;            # brown
> +			*) test -n "$quiet" && return;;
> +		esac
> +		shift
> +		printf " "
> +		printf "$@"
> +		tput sgr0
> +		print_subtest
> +		)
> +	}
> +else
> +	say_color() {
> +		test -z "$1" && test -n "$quiet" && return
> +		shift
> +		printf " "
> +		printf "$@"
> +		print_subtest
> +	}
> +fi
> +
> +error () {
> +	say_color error "error: $*\n"
> +	GIT_EXIT_OK=t
> +	exit 1
> +}
> +
> +say () {
> +	say_color info "$*"
> +}
> +
> +test "${test_description}" != "" ||
> +error "Test script did not set test_description."
> +
> +if test "$help" = "t"
> +then
> +	echo "Tests ${test_description}"
> +	exit 0
> +fi
> +
> +echo $(basename "$0"): "Testing ${test_description}"
> +
> +exec 5>&1
> +
> +test_failure=0
> +test_count=0
> +test_fixed=0
> +test_broken=0
> +test_success=0
> +
> +die () {
> +	code=$?
> +	rm -rf "$TEST_TMPDIR"
> +	if test -n "$GIT_EXIT_OK"
> +	then
> +		exit $code
> +	else
> +		echo >&5 "FATAL: Unexpected exit with code $code"
> +		exit 1
> +	fi
> +}
> +
> +GIT_EXIT_OK=
> +# Note: TEST_TMPDIR *NOT* exported!
> +TEST_TMPDIR=$(mktemp -d "${TMPDIR:-/tmp}/notmuch-test-$$.XXXXXX")
> +trap 'die' EXIT
> +
> +test_decode_color () {
> +	sed	-e 's/.\[1m/<WHITE>/g' \
> +		-e 's/.\[31m/<RED>/g' \
> +		-e 's/.\[32m/<GREEN>/g' \
> +		-e 's/.\[33m/<YELLOW>/g' \
> +		-e 's/.\[34m/<BLUE>/g' \
> +		-e 's/.\[35m/<MAGENTA>/g' \
> +		-e 's/.\[36m/<CYAN>/g' \
> +		-e 's/.\[m/<RESET>/g'
> +}
> +
> +q_to_nul () {
> +	perl -pe 'y/Q/\000/'
> +}
> +
> +q_to_cr () {
> +	tr Q '\015'
> +}
> +
> +append_cr () {
> +	sed -e 's/$/Q/' | tr Q '\015'
> +}
> +
> +remove_cr () {
> +	tr '\015' Q | sed -e 's/Q$//'
> +}
> +
> +# Generate a new message in the mail directory, with a unique message
> +# ID and subject. The message is not added to the index.
> +#
> +# After this function returns, the filename of the generated message
> +# is available as $gen_msg_filename and the message ID is available as
> +# $gen_msg_id .
> +#
> +# This function supports named parameters with the bash syntax for
> +# assigning a value to an associative array ([name]=value). The
> +# supported parameters are:
> +#
> +#  [dir]=directory/of/choice
> +#
> +#	Generate the message in directory 'directory/of/choice' within
> +#	the mail store. The directory will be created if necessary.
> +#
> +#  [filename]=name
> +#
> +#	Store the message in file 'name'. The default is to store it
> +#	in 'msg-<count>', where <count> is three-digit number of the
> +#	message.
> +#
> +#  [body]=text
> +#
> +#	Text to use as the body of the email message
> +#
> +#  '[from]="Some User <user@example.com>"'
> +#  '[to]="Some User <user@example.com>"'
> +#  '[subject]="Subject of email message"'
> +#  '[date]="RFC 822 Date"'
> +#
> +#	Values for email headers. If not provided, default values will
> +#	be generated instead.
> +#
> +#  '[cc]="Some User <user@example.com>"'
> +#  [reply-to]=some-address
> +#  [in-reply-to]=<message-id>
> +#  [references]=<message-id>
> +#  [content-type]=content-type-specification
> +#  '[header]=full header line, including keyword'
> +#
> +#	Additional values for email headers. If these are not provided
> +#	then the relevant headers will simply not appear in the
> +#	message.
> +#
> +#  '[id]=message-id'
> +#
> +#	Controls the message-id of the created message.
> +gen_msg_cnt=0
> +gen_msg_filename=""
> +gen_msg_id=""
> +generate_message ()
> +{
> +    # This is our (bash-specific) magic for doing named parameters
> +    local -A template="($@)"
> +    local additional_headers
> +
> +    gen_msg_cnt=$((gen_msg_cnt + 1))
> +    if [ -z "${template[filename]}" ]; then
> +	gen_msg_name="msg-$(printf "%03d" $gen_msg_cnt)"
> +    else
> +	gen_msg_name=${template[filename]}
> +    fi
> +
> +    if [ -z "${template[id]}" ]; then
> +	gen_msg_id="${gen_msg_name%:2,*}@notmuch-test-suite"
> +    else
> +	gen_msg_id="${template[id]}"
> +    fi
> +
> +    if [ -z "${template[dir]}" ]; then
> +	gen_msg_filename="${MAIL_DIR}/$gen_msg_name"
> +    else
> +	gen_msg_filename="${MAIL_DIR}/${template[dir]}/$gen_msg_name"
> +	mkdir -p "$(dirname "$gen_msg_filename")"
> +    fi
> +
> +    if [ -z "${template[body]}" ]; then
> +	template[body]="This is just a test message (#${gen_msg_cnt})"
> +    fi
> +
> +    if [ -z "${template[from]}" ]; then
> +	template[from]="Notmuch Test Suite <test_suite@notmuchmail.org>"
> +    fi
> +
> +    if [ -z "${template[to]}" ]; then
> +	template[to]="Notmuch Test Suite <test_suite@notmuchmail.org>"
> +    fi
> +
> +    if [ -z "${template[subject]}" ]; then
> +	template[subject]="Test message #${gen_msg_cnt}"
> +    fi
> +
> +    if [ -z "${template[date]}" ]; then
> +	template[date]="Fri, 05 Jan 2001 15:43:57 +0000"
> +    fi
> +
> +    additional_headers=""
> +    if [ ! -z "${template[header]}" ]; then
> +	additional_headers="${template[header]}
> +${additional_headers}"
> +    fi
> +
> +    if [ ! -z "${template[reply-to]}" ]; then
> +	additional_headers="Reply-To: ${template[reply-to]}
> +${additional_headers}"
> +    fi
> +
> +    if [ ! -z "${template[in-reply-to]}" ]; then
> +	additional_headers="In-Reply-To: ${template[in-reply-to]}
> +${additional_headers}"
> +    fi
> +
> +    if [ ! -z "${template[cc]}" ]; then
> +	additional_headers="Cc: ${template[cc]}
> +${additional_headers}"
> +    fi
> +
> +    if [ ! -z "${template[references]}" ]; then
> +	additional_headers="References: ${template[references]}
> +${additional_headers}"
> +    fi
> +
> +    if [ ! -z "${template[content-type]}" ]; then
> +	additional_headers="Content-Type: ${template[content-type]}
> +${additional_headers}"
> +    fi
> +
> +    # Note that in the way we're setting it above and using it below,
> +    # `additional_headers' will also serve as the header / body separator
> +    # (empty line in between).
> +
> +    cat <<EOF >"$gen_msg_filename"
> +From: ${template[from]}
> +To: ${template[to]}
> +Message-Id: <${gen_msg_id}>
> +Subject: ${template[subject]}
> +Date: ${template[date]}
> +${additional_headers}
> +${template[body]}
> +EOF
> +}
> +
> +# Generate a new message and add it to the database.
> +#
> +# All of the arguments and return values supported by generate_message
> +# are also supported here, so see that function for details.
> +add_message ()
> +{
> +    generate_message "$@" &&
> +    notmuch new > /dev/null
> +}
> +
> +# Deliver a message with emacs and add it to the database
> +#
> +# Uses emacs to generate and deliver a message to the mail store.
> +# Accepts arbitrary extra emacs/elisp functions to modify the message
> +# before sending, which is useful to doing things like attaching files
> +# to the message and encrypting/signing.
> +emacs_deliver_message ()
> +{
> +    local subject="$1"
> +    local body="$2"
> +    shift 2
> +    # before we can send a message, we have to prepare the FCC maildir
> +    mkdir -p "$MAIL_DIR"/sent/{cur,new,tmp}
> +    $TEST_DIRECTORY/smtp-dummy sent_message &
> +    smtp_dummy_pid=$!
> +    test_emacs \
> +	"(let ((message-send-mail-function 'message-smtpmail-send-it)
> +	       (smtpmail-smtp-server \"localhost\")
> +	       (smtpmail-smtp-service \"25025\"))
> +	   (notmuch-hello)
> +	   (notmuch-mua-mail)
> +	   (message-goto-to)
> +	   (insert \"test_suite@notmuchmail.org\nDate: 01 Jan 2000 12:00:00 -0000\")
> +	   (message-goto-subject)
> +	   (insert \"${subject}\")
> +	   (message-goto-body)
> +	   (insert \"${body}\")
> +	   $@
> +	   (message-send-and-exit))"
> +    # opportunistically quit smtp-dummy in case above fails.
> +    { echo QUIT > /dev/tcp/localhost/25025; } 2>/dev/null
> +    wait ${smtp_dummy_pid}
> +    notmuch new >/dev/null
> +}
> +
> +# Generate a corpus of email and add it to the database.
> +#
> +# This corpus is fixed, (it happens to be 50 messages from early in
> +# the history of the notmuch mailing list), which allows for reliably
> +# testing commands that need to operate on a not-totally-trivial
> +# number of messages.
> +add_email_corpus ()
> +{
> +    rm -rf ${MAIL_DIR}
> +    if [ -d $TEST_DIRECTORY/corpus.mail ]; then
> +	cp -a $TEST_DIRECTORY/corpus.mail ${MAIL_DIR}
> +    else
> +	cp -a $TEST_DIRECTORY/corpus ${MAIL_DIR}
> +	notmuch new >/dev/null
> +	cp -a ${MAIL_DIR} $TEST_DIRECTORY/corpus.mail
> +    fi
> +}
> +
> +test_begin_subtest ()
> +{
> +    if [ -n "$inside_subtest" ]; then
> +	exec 1>&6 2>&7		# Restore stdout and stderr
> +	error "bug in test script: Missing test_expect_equal in ${BASH_SOURCE[1]}:${BASH_LINENO[0]}"
> +    fi
> +    test_subtest_name="$1"
> +    test_reset_state_
> +    # Remember stdout and stderr file descriptors and redirect test
> +    # output to the previously prepared file descriptors 3 and 4 (see
> +    # below)
> +    if test "$verbose" != "t"; then exec 4>test.output 3>&4; fi
> +    exec 6>&1 7>&2 >&3 2>&4
> +    inside_subtest=t
> +}
> +
> +# Pass test if two arguments match
> +#
> +# Note: Unlike all other test_expect_* functions, this function does
> +# not accept a test name. Instead, the caller should call
> +# test_begin_subtest before calling this function in order to set the
> +# name.
> +test_expect_equal ()
> +{
> +	exec 1>&6 2>&7		# Restore stdout and stderr
> +	inside_subtest=
> +	test "$#" = 3 && { prereq=$1; shift; } || prereq=
> +	test "$#" = 2 ||
> +	error "bug in the test script: not 2 or 3 parameters to test_expect_equal"
> +
> +	output="$1"
> +	expected="$2"
> +	if ! test_skip "$test_subtest_name"
> +	then
> +		if [ "$output" = "$expected" ]; then
> +			test_ok_ "$test_subtest_name"
> +		else
> +			testname=$this_test.$test_count
> +			echo "$expected" > $testname.expected
> +			echo "$output" > $testname.output
> +			test_failure_ "$test_subtest_name" "$(diff -u $testname.expected $testname.output)"
> +		fi
> +    fi
> +}
> +
> +# Like test_expect_equal, but takes two filenames.
> +test_expect_equal_file ()
> +{
> +	exec 1>&6 2>&7		# Restore stdout and stderr
> +	inside_subtest=
> +	test "$#" = 3 && { prereq=$1; shift; } || prereq=
> +	test "$#" = 2 ||
> +	error "bug in the test script: not 2 or 3 parameters to test_expect_equal"
> +
> +	output="$1"
> +	expected="$2"
> +	if ! test_skip "$test_subtest_name"
> +	then
> +		if diff -q "$expected" "$output" >/dev/null ; then
> +			test_ok_ "$test_subtest_name"
> +		else
> +			testname=$this_test.$test_count
> +			cp "$output" $testname.output
> +			cp "$expected" $testname.expected
> +			test_failure_ "$test_subtest_name" "$(diff -u $testname.expected $testname.output)"
> +		fi
> +    fi
> +}
> +
> +test_emacs_expect_t () {
> +	test "$#" = 2 && { prereq=$1; shift; } || prereq=
> +	test "$#" = 1 ||
> +	error "bug in the test script: not 1 or 2 parameters to test_emacs_expect_t"
> +
> +	# Run the test.
> +	if ! test_skip "$test_subtest_name"
> +	then
> +		test_emacs "(notmuch-test-run $1)" >/dev/null
> +
> +		# Restore state after the test.
> +		exec 1>&6 2>&7		# Restore stdout and stderr
> +		inside_subtest=
> +
> +		# Report success/failure.
> +		result=$(cat OUTPUT)
> +		if [ "$result" = t ]
> +		then
> +			test_ok_ "$test_subtest_name"
> +		else
> +			test_failure_ "$test_subtest_name" "${result}"
> +		fi
> +	else
> +		# Restore state after the (non) test.
> +		exec 1>&6 2>&7		# Restore stdout and stderr
> +		inside_subtest=
> +	fi
> +}
> +
> +NOTMUCH_NEW ()
> +{
> +    notmuch new | grep -v -E -e '^Processed [0-9]*( total)? file|Found [0-9]* total file'
> +}
> +
> +notmuch_search_sanitize ()
> +{
> +    sed -r -e 's/("?thread"?: ?)("?)................("?)/\1\2XXX\3/'
> +}
> +
> +NOTMUCH_SHOW_FILENAME_SQUELCH='s,filename:.*/mail,filename:/XXX/mail,'
> +notmuch_show_sanitize ()
> +{
> +    sed -e "$NOTMUCH_SHOW_FILENAME_SQUELCH"
> +}
> +notmuch_show_sanitize_all ()
> +{
> +    sed \
> +	-e 's| filename:.*| filename:XXXXX|' \
> +	-e 's| id:[^ ]* | id:XXXXX |'
> +}
> +
> +notmuch_json_show_sanitize ()
> +{
> +    sed -e 's|, |,\n |g' | \
> +	sed \
> +	-e 's|"id": "[^"]*",|"id": "XXXXX",|' \
> +	-e 's|"filename": "[^"]*",|"filename": "YYYYY",|'
> +}
> +
> +# End of notmuch helper functions
> +
> +# Use test_set_prereq to tell that a particular prerequisite is available.
> +# The prerequisite can later be checked for in two ways:
> +#
> +# - Explicitly using test_have_prereq.
> +#
> +# - Implicitly by specifying the prerequisite tag in the calls to
> +#   test_expect_{success,failure,code}.
> +#
> +# The single parameter is the prerequisite tag (a simple word, in all
> +# capital letters by convention).
> +
> +test_set_prereq () {
> +	satisfied="$satisfied$1 "
> +}
> +satisfied=" "
> +
> +test_have_prereq () {
> +	case $satisfied in
> +	*" $1 "*)
> +		: yes, have it ;;
> +	*)
> +		! : nope ;;
> +	esac
> +}
> +
> +# declare prerequisite for the given external binary
> +test_declare_external_prereq () {
> +	binary="$1"
> +	test "$#" = 2 && name=$2 || name="$binary(1)"
> +
> +	hash $binary 2>/dev/null || eval "
> +	test_missing_external_prereq_${binary}_=t
> +$binary () {
> +	echo -n \"\$test_subtest_missing_external_prereqs_ \" | grep -qe \" $name \" ||
> +	test_subtest_missing_external_prereqs_=\"\$test_subtest_missing_external_prereqs_ $name\"
> +	false
> +}"
> +}
> +
> +# Explicitly require external prerequisite.  Useful when binary is
> +# called indirectly (e.g. from emacs).
> +# Returns success if dependency is available, failure otherwise.
> +test_require_external_prereq () {
> +	binary="$1"
> +	if [ "$(eval echo -n \$test_missing_external_prereq_${binary}_)" = t ]; then
> +		# dependency is missing, call the replacement function to note it
> +		eval "$binary"
> +	else
> +		true
> +	fi
> +}
> +
> +# You are not expected to call test_ok_ and test_failure_ directly, use
> +# the text_expect_* functions instead.
> +
> +test_ok_ () {
> +	if test "$test_subtest_known_broken_" = "t"; then
> +		test_known_broken_ok_ "$@"
> +		return
> +	fi
> +	test_success=$(($test_success + 1))
> +	say_color pass "%-6s" "PASS"
> +	echo " $@"
> +}
> +
> +test_failure_ () {
> +	if test "$test_subtest_known_broken_" = "t"; then
> +		test_known_broken_failure_ "$@"
> +		return
> +	fi
> +	test_failure=$(($test_failure + 1))
> +	test_failure_message_ "FAIL" "$@"
> +	test "$immediate" = "" || { GIT_EXIT_OK=t; exit 1; }
> +	return 1
> +}
> +
> +test_failure_message_ () {
> +	say_color error "%-6s" "$1"
> +	echo " $2"
> +	shift 2
> +	echo "$@" | sed -e 's/^/	/'
> +	if test "$verbose" != "t"; then cat test.output; fi
> +}
> +
> +test_known_broken_ok_ () {
> +	test_reset_state_
> +	test_fixed=$(($test_fixed+1))
> +	say_color pass "%-6s" "FIXED"
> +	echo " $@"
> +}
> +
> +test_known_broken_failure_ () {
> +	test_reset_state_
> +	test_broken=$(($test_broken+1))
> +	test_failure_message_ "BROKEN" "$@"
> +	return 1
> +}
> +
> +test_debug () {
> +	test "$debug" = "" || eval "$1"
> +}
> +
> +test_run_ () {
> +	test_cleanup=:
> +	if test "$verbose" != "t"; then exec 4>test.output 3>&4; fi
> +	eval >&3 2>&4 "$1"
> +	eval_ret=$?
> +	eval >&3 2>&4 "$test_cleanup"
> +	return 0
> +}
> +
> +test_skip () {
> +	test_count=$(($test_count+1))
> +	to_skip=
> +	for skp in $NOTMUCH_SKIP_TESTS
> +	do
> +		case $this_test.$test_count in
> +		$skp)
> +			to_skip=t
> +		esac
> +	done
> +	if test -z "$to_skip" && test -n "$prereq" &&
> +	   ! test_have_prereq "$prereq"
> +	then
> +		to_skip=t
> +	fi
> +	case "$to_skip" in
> +	t)
> +		test_report_skip_ "$@"
> +		;;
> +	*)
> +		test_check_missing_external_prereqs_ "$@"
> +		;;
> +	esac
> +}
> +
> +test_check_missing_external_prereqs_ () {
> +	if test -n "$test_subtest_missing_external_prereqs_"; then
> +		say_color skip >&3 "missing prerequisites:"
> +		echo "$test_subtest_missing_external_prereqs_" >&3
> +		test_report_skip_ "$@"
> +	else
> +		false
> +	fi
> +}
> +
> +test_report_skip_ () {
> +	test_reset_state_
> +	say_color skip >&3 "skipping test:"
> +	echo " $@" >&3
> +	say_color skip "%-6s" "SKIP"
> +	echo " $1"
> +}
> +
> +test_subtest_known_broken () {
> +	test_subtest_known_broken_=t
> +}
> +
> +test_expect_success () {
> +	test "$#" = 3 && { prereq=$1; shift; } || prereq=
> +	test "$#" = 2 ||
> +	error "bug in the test script: not 2 or 3 parameters to test-expect-success"
> +	test_reset_state_
> +	if ! test_skip "$@"
> +	then
> +		test_run_ "$2"
> +		run_ret="$?"
> +		# test_run_ may update missing external prerequisites
> +		test_check_missing_external_prereqs_ "$@" ||
> +		if [ "$run_ret" = 0 -a "$eval_ret" = 0 ]
> +		then
> +			test_ok_ "$1"
> +		else
> +			test_failure_ "$@"
> +		fi
> +	fi
> +}
> +
> +test_expect_code () {
> +	test "$#" = 4 && { prereq=$1; shift; } || prereq=
> +	test "$#" = 3 ||
> +	error "bug in the test script: not 3 or 4 parameters to test-expect-code"
> +	test_reset_state_
> +	if ! test_skip "$@"
> +	then
> +		test_run_ "$3"
> +		run_ret="$?"
> +		# test_run_ may update missing external prerequisites,
> +		test_check_missing_external_prereqs_ "$@" ||
> +		if [ "$run_ret" = 0 -a "$eval_ret" = "$1" ]
> +		then
> +			test_ok_ "$2"
> +		else
> +			test_failure_ "$@"
> +		fi
> +	fi
> +}
> +
> +# test_external runs external test scripts that provide continuous
> +# test output about their progress, and succeeds/fails on
> +# zero/non-zero exit code.  It outputs the test output on stdout even
> +# in non-verbose mode, and announces the external script with "* run
> +# <n>: ..." before running it.  When providing relative paths, keep in
> +# mind that all scripts run in "trash directory".
> +# Usage: test_external description command arguments...
> +# Example: test_external 'Perl API' perl ../path/to/test.pl
> +test_external () {
> +	test "$#" = 4 && { prereq=$1; shift; } || prereq=
> +	test "$#" = 3 ||
> +	error >&5 "bug in the test script: not 3 or 4 parameters to test_external"
> +	descr="$1"
> +	shift
> +	test_reset_state_
> +	if ! test_skip "$descr" "$@"
> +	then
> +		# Announce the script to reduce confusion about the
> +		# test output that follows.
> +		say_color "" " run $test_count: $descr ($*)"
> +		# Run command; redirect its stderr to &4 as in
> +		# test_run_, but keep its stdout on our stdout even in
> +		# non-verbose mode.
> +		"$@" 2>&4
> +		if [ "$?" = 0 ]
> +		then
> +			test_ok_ "$descr"
> +		else
> +			test_failure_ "$descr" "$@"
> +		fi
> +	fi
> +}
> +
> +# Like test_external, but in addition tests that the command generated
> +# no output on stderr.
> +test_external_without_stderr () {
> +	# The temporary file has no (and must have no) security
> +	# implications.
> +	tmp="$TMPDIR"; if [ -z "$tmp" ]; then tmp=/tmp; fi
> +	stderr="$tmp/git-external-stderr.$$.tmp"
> +	test_external "$@" 4> "$stderr"
> +	[ -f "$stderr" ] || error "Internal error: $stderr disappeared."
> +	descr="no stderr: $1"
> +	shift
> +	if [ ! -s "$stderr" ]; then
> +		rm "$stderr"
> +		test_ok_ "$descr"
> +	else
> +		if [ "$verbose" = t ]; then
> +			output=`echo; echo Stderr is:; cat "$stderr"`
> +		else
> +			output=
> +		fi
> +		# rm first in case test_failure exits.
> +		rm "$stderr"
> +		test_failure_ "$descr" "$@" "$output"
> +	fi
> +}
> +
> +# This is not among top-level (test_expect_success)
> +# but is a prefix that can be used in the test script, like:
> +#
> +#	test_expect_success 'complain and die' '
> +#           do something &&
> +#           do something else &&
> +#	    test_must_fail git checkout ../outerspace
> +#	'
> +#
> +# Writing this as "! git checkout ../outerspace" is wrong, because
> +# the failure could be due to a segv.  We want a controlled failure.
> +
> +test_must_fail () {
> +	"$@"
> +	test $? -gt 0 -a $? -le 129 -o $? -gt 192
> +}
> +
> +# test_cmp is a helper function to compare actual and expected output.
> +# You can use it like:
> +#
> +#	test_expect_success 'foo works' '
> +#		echo expected >expected &&
> +#		foo >actual &&
> +#		test_cmp expected actual
> +#	'
> +#
> +# This could be written as either "cmp" or "diff -u", but:
> +# - cmp's output is not nearly as easy to read as diff -u
> +# - not all diff versions understand "-u"
> +
> +test_cmp() {
> +	$GIT_TEST_CMP "$@"
> +}
> +
> +# This function can be used to schedule some commands to be run
> +# unconditionally at the end of the test to restore sanity:
> +#
> +#	test_expect_success 'test core.capslock' '
> +#		git config core.capslock true &&
> +#		test_when_finished "git config --unset core.capslock" &&
> +#		hello world
> +#	'
> +#
> +# That would be roughly equivalent to
> +#
> +#	test_expect_success 'test core.capslock' '
> +#		git config core.capslock true &&
> +#		hello world
> +#		git config --unset core.capslock
> +#	'
> +#
> +# except that the greeting and config --unset must both succeed for
> +# the test to pass.
> +
> +test_when_finished () {
> +	test_cleanup="{ $*
> +		} && (exit \"\$eval_ret\"); eval_ret=\$?; $test_cleanup"
> +}
> +
> +test_done () {
> +	GIT_EXIT_OK=t
> +	test_results_dir="$TEST_DIRECTORY/test-results"
> +	mkdir -p "$test_results_dir"
> +	test_results_path="$test_results_dir/${0%.sh}-$$"
> +
> +	echo "total $test_count" >> $test_results_path
> +	echo "success $test_success" >> $test_results_path
> +	echo "fixed $test_fixed" >> $test_results_path
> +	echo "broken $test_broken" >> $test_results_path
> +	echo "failed $test_failure" >> $test_results_path
> +	echo "" >> $test_results_path
> +
> +	echo
> +
> +	[ -n "$EMACS_SERVER" ] && test_emacs '(kill-emacs)'
> +
> +	if [ "$test_failure" = "0" ]; then
> +	    if [ "$test_broken" = "0" ]; then
> +		rm -rf "$remove_tmp"
> +	    fi
> +	    exit 0
> +	else
> +	    exit 1
> +	fi
> +}
> +
> +emacs_generate_script () {
> +	# Construct a little test script here for the benefit of the user,
> +	# (who can easily run "run_emacs" to get the same emacs environment
> +	# for investigating any failures).
> +	cat <<EOF >"$TMP_DIRECTORY/run_emacs"
> +#!/bin/sh
> +export PATH=$PATH
> +export NOTMUCH_CONFIG=$NOTMUCH_CONFIG
> +
> +# Here's what we are using here:
> +#
> +# --no-init-file	Don't load users ~/.emacs
> +#
> +# --no-site-file	Don't load the site-wide startup stuff
> +#
> +# --directory		Ensure that the local elisp sources are found
> +#
> +# --load		Force loading of notmuch.el and test-lib.el
> +
> +exec ${TEST_EMACS} --no-init-file --no-site-file \
> +	--directory "$TEST_DIRECTORY/../emacs" --load notmuch.el \
> +	--directory "$TEST_DIRECTORY" --load test-lib.el \
> +	"\$@"
> +EOF
> +	chmod a+x "$TMP_DIRECTORY/run_emacs"
> +}
> +
> +test_emacs () {
> +	# test dependencies beforehand to avoid the waiting loop below
> +	missing_dependencies=
> +	test_require_external_prereq dtach || missing_dependencies=1
> +	test_require_external_prereq emacs || missing_dependencies=1
> +	test_require_external_prereq emacsclient || missing_dependencies=1
> +	test -z "$missing_dependencies" || return
> +
> +	if [ -z "$EMACS_SERVER" ]; then
> +		server_name="notmuch-test-suite-$$"
> +		# start a detached session with an emacs server
> +		# user's TERM is given to dtach which assumes a minimally
> +		# VT100-compatible terminal -- and emacs inherits that
> +		TERM=$ORIGINAL_TERM dtach -n "$TEST_TMPDIR/emacs-dtach-socket.$$" \
> +			sh -c "stty rows 24 cols 80; exec '$TMP_DIRECTORY/run_emacs' \
> +				--no-window-system \
> +				--eval '(setq server-name \"$server_name\")' \
> +				--eval '(server-start)' \
> +				--eval '(orphan-watchdog $$)'" || return
> +		EMACS_SERVER="$server_name"
> +		# wait until the emacs server is up
> +		until test_emacs '()' >/dev/null 2>/dev/null; do
> +			sleep 1
> +		done
> +	fi
> +
> +	emacsclient --socket-name="$EMACS_SERVER" --eval "(progn $@)"
> +}
> +
> +test_python() {
> +	export LD_LIBRARY_PATH=$TEST_DIRECTORY/../lib
> +	export PYTHONPATH=$TEST_DIRECTORY/../bindings/python
> +
> +	# Some distros (e.g. Arch Linux) ship Python 2.* as /usr/bin/python2,
> +	# most others as /usr/bin/python. So first try python2, and fallback to
> +	# python if python2 doesn't exist.
> +	cmd=python2
> +	[[ "$test_missing_external_prereq_python2_" = t ]] && cmd=python
> +
> +	(echo "import sys; _orig_stdout=sys.stdout; sys.stdout=open('OUTPUT', 'w')"; cat) \
> +		| $cmd -
> +}
> +
> +# Creates a script that counts how much time it is executed and calls
> +# notmuch.  $notmuch_counter_command is set to the path to the
> +# generated script.  Use notmuch_counter_value() function to get the
> +# current counter value.
> +notmuch_counter_reset () {
> +	notmuch_counter_command="$TMP_DIRECTORY/notmuch_counter"
> +	if [ ! -x "$notmuch_counter_command" ]; then
> +		notmuch_counter_state_path="$TMP_DIRECTORY/notmuch_counter.state"
> +		cat >"$notmuch_counter_command" <<EOF || return
> +#!/bin/sh
> +
> +read count < "$notmuch_counter_state_path"
> +echo \$((count + 1)) > "$notmuch_counter_state_path"
> +
> +exec notmuch "\$@"
> +EOF
> +		chmod +x "$notmuch_counter_command" || return
> +	fi
> +
> +	echo 0 > "$notmuch_counter_state_path"
> +}
> +
> +# Returns the current notmuch counter value.
> +notmuch_counter_value () {
> +	if [ -r "$notmuch_counter_state_path" ]; then
> +		read count < "$notmuch_counter_state_path"
> +	else
> +		count=0
> +	fi
> +	echo $count
> +}
> +
> +test_reset_state_ () {
> +	test -z "$test_init_done_" && test_init_
> +
> +	test_subtest_known_broken_=
> +	test_subtest_missing_external_prereqs_=
> +}
> +
> +# called once before the first subtest
> +test_init_ () {
> +	test_init_done_=t
> +
> +	# skip all tests if there were external prerequisites missing during init
> +	test_check_missing_external_prereqs_ "all tests in $this_test" && test_done
> +}
> +
> +
> +find_notmuch_path ()
> +{
> +    dir="$1"
> +
> +    while [ -n "$dir" ]; do
> +	bin="$dir/notmuch"
> +	if [ -x "$bin" ]; then
> +	    echo "$dir"
> +	    return
> +	fi
> +	dir="$(dirname "$dir")"
> +	if [ "$dir" = "/" ]; then
> +	    break
> +	fi
> +    done
> +}
> +
> +# Test the binaries we have just built.  The tests are kept in
> +# test/ subdirectory and are run in 'trash directory' subdirectory.
> +TEST_DIRECTORY=$(pwd)
> +if test -n "$valgrind"
> +then
> +	make_symlink () {
> +		test -h "$2" &&
> +		test "$1" = "$(readlink "$2")" || {
> +			# be super paranoid
> +			if mkdir "$2".lock
> +			then
> +				rm -f "$2" &&
> +				ln -s "$1" "$2" &&
> +				rm -r "$2".lock
> +			else
> +				while test -d "$2".lock
> +				do
> +					say "Waiting for lock on $2."
> +					sleep 1
> +				done
> +			fi
> +		}
> +	}
> +
> +	make_valgrind_symlink () {
> +		# handle only executables
> +		test -x "$1" || return
> +
> +		base=$(basename "$1")
> +		symlink_target=$TEST_DIRECTORY/../$base
> +		# do not override scripts
> +		if test -x "$symlink_target" &&
> +		    test ! -d "$symlink_target" &&
> +		    test "#!" != "$(head -c 2 < "$symlink_target")"
> +		then
> +			symlink_target=$TEST_DIRECTORY/valgrind.sh
> +		fi
> +		case "$base" in
> +		*.sh|*.perl)
> +			symlink_target=$TEST_DIRECTORY/unprocessed-script
> +		esac
> +		# create the link, or replace it if it is out of date
> +		make_symlink "$symlink_target" "$GIT_VALGRIND/bin/$base" || exit
> +	}
> +
> +	# override notmuch executable in TEST_DIRECTORY/..
> +	GIT_VALGRIND=$TEST_DIRECTORY/valgrind
> +	mkdir -p "$GIT_VALGRIND"/bin
> +	make_valgrind_symlink $TEST_DIRECTORY/../notmuch
> +	OLDIFS=$IFS
> +	IFS=:
> +	for path in $PATH
> +	do
> +		ls "$path"/notmuch 2> /dev/null |
> +		while read file
> +		do
> +			make_valgrind_symlink "$file"
> +		done
> +	done
> +	IFS=$OLDIFS
> +	PATH=$GIT_VALGRIND/bin:$PATH
> +	GIT_EXEC_PATH=$GIT_VALGRIND/bin
> +	export GIT_VALGRIND
> +else # normal case
> +	notmuch_path=`find_notmuch_path "$TEST_DIRECTORY"`
> +	test -n "$notmuch_path" && PATH="$notmuch_path:$PATH"
> +fi
> +export PATH
> +
> +# Test repository
> +test="tmp.$(basename "$0" .sh)"
> +test -n "$root" && test="$root/$test"
> +case "$test" in
> +/*) TMP_DIRECTORY="$test" ;;
> + *) TMP_DIRECTORY="$TEST_DIRECTORY/$test" ;;
> +esac
> +test ! -z "$debug" || remove_tmp=$TMP_DIRECTORY
> +rm -fr "$test" || {
> +	GIT_EXIT_OK=t
> +	echo >&5 "FATAL: Cannot prepare test area"
> +	exit 1
> +}
> +
> +# A temporary home directory is needed by at least:
> +# - emacs/"Sending a message via (fake) SMTP"
> +# - emacs/"Reply within emacs"
> +# - crypto/emacs_deliver_message
> +export HOME="${TMP_DIRECTORY}/home"
> +mkdir -p "${HOME}"
> +
> +MAIL_DIR="${TMP_DIRECTORY}/mail"
> +export GNUPGHOME="${TMP_DIRECTORY}/gnupg"
> +export NOTMUCH_CONFIG="${TMP_DIRECTORY}/notmuch-config"
> +
> +mkdir -p "${test}"
> +mkdir -p "${MAIL_DIR}"
> +
> +cat <<EOF >"${NOTMUCH_CONFIG}"
> +[database]
> +path=${MAIL_DIR}
> +
> +[user]
> +name=Notmuch Test Suite
> +primary_email=test_suite@notmuchmail.org
> +other_email=test_suite_other@notmuchmail.org;test_suite@otherdomain.org
> +EOF
> +
> +emacs_generate_script
> +
> +
> +# Use -P to resolve symlinks in our working directory so that the cwd
> +# in subprocesses like git equals our $PWD (for pathname comparisons).
> +cd -P "$test" || error "Cannot setup test environment"
> +
> +if test "$verbose" = "t"
> +then
> +	exec 4>&2 3>&1
> +else
> +	exec 4>test.output 3>&4
> +fi
> +
> +this_test=${0##*/}
> +for skp in $NOTMUCH_SKIP_TESTS
> +do
> +	to_skip=
> +	for skp in $NOTMUCH_SKIP_TESTS
> +	do
> +		case "$this_test" in
> +		$skp)
> +			to_skip=t
> +		esac
> +	done
> +	case "$to_skip" in
> +	t)
> +		say_color skip >&3 "skipping test $this_test altogether"
> +		say_color skip "skip all tests in $this_test"
> +		test_done
> +	esac
> +done
> +
> +# Provide an implementation of the 'yes' utility
> +yes () {
> +	if test $# = 0
> +	then
> +		y=y
> +	else
> +		y="$*"
> +	fi
> +
> +	while echo "$y"
> +	do
> +		:
> +	done
> +}
> +
> +# Fix some commands on Windows
> +case $(uname -s) in
> +*MINGW*)
> +	# Windows has its own (incompatible) sort and find
> +	sort () {
> +		/usr/bin/sort "$@"
> +	}
> +	find () {
> +		/usr/bin/find "$@"
> +	}
> +	sum () {
> +		md5sum "$@"
> +	}
> +	# git sees Windows-style pwd
> +	pwd () {
> +		builtin pwd -W
> +	}
> +	# no POSIX permissions
> +	# backslashes in pathspec are converted to '/'
> +	# exec does not inherit the PID
> +	;;
> +*)
> +	test_set_prereq POSIXPERM
> +	test_set_prereq BSLASHPSPEC
> +	test_set_prereq EXECKEEPSPID
> +	;;
> +esac
> +
> +test -z "$NO_PERL" && test_set_prereq PERL
> +test -z "$NO_PYTHON" && test_set_prereq PYTHON
> +
> +# test whether the filesystem supports symbolic links
> +ln -s x y 2>/dev/null && test -h y 2>/dev/null && test_set_prereq SYMLINKS
> +rm -f y
> +
> +# declare prerequisites for external binaries used in tests
> +test_declare_external_prereq dtach
> +test_declare_external_prereq emacs
> +test_declare_external_prereq emacsclient
> +test_declare_external_prereq gdb
> +test_declare_external_prereq gpg
> +test_declare_external_prereq python
> +test_declare_external_prereq python2

Re: [PATCH v7 01/10] test: Add broken test for the new JSON reply format.

[Adam Wolfe Gordon]

On Tue, Mar 13, 2012 at 10:49, Austin Clements <amdragon@mit.edu> wrote:
>> diff --git a/test/test-lib b/test/test-lib
>> new file mode 100755
>> index 0000000..8158328
>> --- /dev/null
>> +++ b/test/test-lib
>
> Spurious git add?  (IIRC, this is what test-lib.sh used to be called,
> so I'd guess you have an old copy of this file laying around.)

Yep, looks like it. I wonder how I missed that.

Re: [PATCH v7 10/10] emacs: Use the new JSON reply format and message-cite-original

[Austin Clements]

Quoth Adam Wolfe Gordon on Mar 11 at 10:05 pm:
> Use the new JSON reply format to create replies in emacs. Quote HTML
> parts nicely by using mm-display-part to turn them into displayable
> text, then quoting them with message-cite-original. This is very
> useful for users who regularly receive HTML-only email.
> 
> Use message-mode's message-cite-original function to create the
> quoted body for reply messages. In order to make this act like the
> existing notmuch defaults, you will need to set the following in
> your emacs configuration:
> 
> message-citation-line-format "On %a, %d %b %Y, %f wrote:"
> message-citation-line-function 'message-insert-formatted-citation-line
> 
> The tests have been updated to reflect the (ugly) emacs default.

Hmm.  This patch looks the same as in v6, but your cover letter says
that you changed some things.

Re: [PATCH v7 10/10] emacs: Use the new JSON reply format and message-cite-original

[Adam Wolfe Gordon]

On Tue, Mar 13, 2012 at 11:02, Austin Clements <amdragon@mit.edu> wrote:
> Hmm.  This patch looks the same as in v6, but your cover letter says
> that you changed some things.

Damn, indeed. Looks like I rebased the changes into the wrong commit,
so they weren't included in this series.

Might be best to ignore this version of the series - I'll get this
figured out tonight and send a new version.

[PATCH v7.1 00/11] Reply enhancements, second attempt

[Adam Wolfe Gordon]

Hi everyone,

This is mostly a re-send of last night's series [1]. The only changes are:

* Fixed the emacs patch, where I had a rebase issue.
* Fixed the commit that re-added test/test-lib by accident.
* Fixed devel/schemata per Austin's review of the previous version.

Since this series seems to be getting close, I figured I'd add a NEWS entry
for good measure.

Thanks for all the reviews so far, and I hope this is almost ready!

[1] id:"1331525142-30539-1-git-send-email-awg+notmuch@xvx.ca"

Adam Wolfe Gordon (11):
  test: Add broken test for the new JSON reply format.
  reply: Factor out reply creation
  TODO: Add replying to multiple messages
  reply: Add a JSON reply format.
  schemata: Add documentation for JSON reply format.
  man: Update notmuch-reply man page for JSON format.
  man: Add --decrypt to reply flags
  emacs: Factor out useful functions into notmuch-lib
  test: Add broken tests for new emacs reply functionality
  emacs: Use the new JSON reply format and message-cite-original
  NEWS: news for reply enhancements

 NEWS                     |   20 ++++++
 devel/TODO               |    8 +++
 devel/schemata           |   27 ++++++++-
 emacs/notmuch-lib.el     |   63 +++++++++++++++++++
 emacs/notmuch-mua.el     |  124 +++++++++++++++++++++++++------------
 emacs/notmuch-show.el    |   55 +++--------------
 man/man1/notmuch-reply.1 |   20 ++++++-
 notmuch-client.h         |   14 +++-
 notmuch-reply.c          |  153 +++++++++++++++++++++++++++++++++-------------
 notmuch-show.c           |   29 ++++++---
 test/emacs               |  101 ++++++++++++++++++++++++++++++-
 test/multipart           |   52 ++++++++++++++++
 12 files changed, 518 insertions(+), 148 deletions(-)

-- 
1.7.5.4

[PATCH v7.1 01/11] test: Add broken test for the new JSON reply format.

[Adam Wolfe Gordon]

---
 test/multipart |   53 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 files changed, 53 insertions(+), 0 deletions(-)

diff --git a/test/multipart b/test/multipart
index 53782c6..80d6e88 100755
--- a/test/multipart
+++ b/test/multipart
@@ -589,6 +589,59 @@ Non-text part: text/html
 EOF
 test_expect_equal_file OUTPUT EXPECTED
 
+test_begin_subtest "'notmuch reply' to a multipart message with json format"
+test_subtest_known_broken
+notmuch reply --format=json 'id:87liy5ap00.fsf@yoom.home.cworth.org' | notmuch_json_show_sanitize >OUTPUT
+cat <<EOF >EXPECTED
+{"reply-headers": {"Subject": "Re: Multipart message",
+ "From": "Notmuch Test Suite <test_suite@notmuchmail.org>",
+ "To": "Carl Worth <cworth@cworth.org>,
+ cworth@cworth.org",
+ "In-reply-to": "<87liy5ap00.fsf@yoom.home.cworth.org>",
+ "References": " <87liy5ap00.fsf@yoom.home.cworth.org>"},
+ "original": {"id": "XXXXX",
+ "match": false,
+ "excluded": false,
+ "filename": "YYYYY",
+ "timestamp": 978709437,
+ "date_relative": "2001-01-05",
+ "tags": ["attachment","inbox","signed","unread"],
+ "headers": {"Subject": "Multipart message",
+ "From": "Carl Worth <cworth@cworth.org>",
+ "To": "cworth@cworth.org",
+ "Date": "Fri,
+ 05 Jan 2001 15:43:57 +0000"},
+ "body": [{"id": 1,
+ "content-type": "multipart/signed",
+ "content": [{"id": 2,
+ "content-type": "multipart/mixed",
+ "content": [{"id": 3,
+ "content-type": "message/rfc822",
+ "content": [{"headers": {"Subject": "html message",
+ "From": "Carl Worth <cworth@cworth.org>",
+ "To": "cworth@cworth.org",
+ "Date": "Fri,
+ 05 Jan 2001 15:42:57 +0000"},
+ "body": [{"id": 4,
+ "content-type": "multipart/alternative",
+ "content": [{"id": 5,
+ "content-type": "text/html"},
+ {"id": 6,
+ "content-type": "text/plain",
+ "content": "This is an embedded message,
+ with a multipart/alternative part.\n"}]}]}]},
+ {"id": 7,
+ "content-type": "text/plain",
+ "filename": "YYYYY",
+ "content": "This is a text attachment.\n"},
+ {"id": 8,
+ "content-type": "text/plain",
+ "content": "And this message is signed.\n\n-Carl\n"}]},
+ {"id": 9,
+ "content-type": "application/pgp-signature"}]}]}}
+EOF
+test_expect_equal_file OUTPUT EXPECTED
+
 test_begin_subtest "'notmuch show --part' does not corrupt a part with CRLF pair"
 notmuch show --format=raw --part=3 id:base64-part-with-crlf > crlf.out
 echo -n -e "\xEF\x0D\x0A" > crlf.expected
-- 
1.7.5.4

[PATCH v7.1 02/11] reply: Factor out reply creation

[Adam Wolfe Gordon]

Factor out the creation of a reply message based on an original
message so it can be shared by different reply formats.
---
 notmuch-reply.c |  104 ++++++++++++++++++++++++++++++++----------------------
 1 files changed, 62 insertions(+), 42 deletions(-)

diff --git a/notmuch-reply.c b/notmuch-reply.c
index 6b244e6..f1478cc 100644
--- a/notmuch-reply.c
+++ b/notmuch-reply.c
@@ -505,6 +505,61 @@ guess_from_received_header (notmuch_config_t *config, notmuch_message_t *message
     return NULL;
 }
 
+static GMimeMessage *
+create_reply_message(void *ctx,
+		     notmuch_config_t *config,
+		     notmuch_message_t *message,
+		     notmuch_bool_t reply_all)
+{
+    const char *subject, *from_addr = NULL;
+    const char *in_reply_to, *orig_references, *references;
+
+    /* The 1 means we want headers in a "pretty" order. */
+    GMimeMessage *reply = g_mime_message_new (1);
+    if (reply == NULL) {
+	fprintf (stderr, "Out of memory\n");
+	return NULL;
+    }
+
+    subject = notmuch_message_get_header (message, "subject");
+    if (subject) {
+	if (strncasecmp (subject, "Re:", 3))
+	    subject = talloc_asprintf (ctx, "Re: %s", subject);
+	g_mime_message_set_subject (reply, subject);
+    }
+
+    from_addr = add_recipients_from_message (reply, config,
+					     message, reply_all);
+
+    if (from_addr == NULL)
+	from_addr = guess_from_received_header (config, message);
+
+    if (from_addr == NULL)
+	from_addr = notmuch_config_get_user_primary_email (config);
+
+    from_addr = talloc_asprintf (ctx, "%s <%s>",
+				 notmuch_config_get_user_name (config),
+				 from_addr);
+    g_mime_object_set_header (GMIME_OBJECT (reply),
+			      "From", from_addr);
+
+    in_reply_to = talloc_asprintf (ctx, "<%s>",
+				   notmuch_message_get_message_id (message));
+
+    g_mime_object_set_header (GMIME_OBJECT (reply),
+			      "In-Reply-To", in_reply_to);
+
+    orig_references = notmuch_message_get_header (message, "references");
+    references = talloc_asprintf (ctx, "%s%s%s",
+				  orig_references ? orig_references : "",
+				  orig_references ? " " : "",
+				  in_reply_to);
+    g_mime_object_set_header (GMIME_OBJECT (reply),
+			      "References", references);
+
+    return reply;
+}
+
 static int
 notmuch_reply_format_default(void *ctx,
 			     notmuch_config_t *config,
@@ -515,8 +570,6 @@ notmuch_reply_format_default(void *ctx,
     GMimeMessage *reply;
     notmuch_messages_t *messages;
     notmuch_message_t *message;
-    const char *subject, *from_addr = NULL;
-    const char *in_reply_to, *orig_references, *references;
     const notmuch_show_format_t *format = &format_reply;
 
     for (messages = notmuch_query_search_messages (query);
@@ -525,49 +578,16 @@ notmuch_reply_format_default(void *ctx,
     {
 	message = notmuch_messages_get (messages);
 
-	/* The 1 means we want headers in a "pretty" order. */
-	reply = g_mime_message_new (1);
-	if (reply == NULL) {
-	    fprintf (stderr, "Out of memory\n");
-	    return 1;
-	}
+	reply = create_reply_message (ctx, config, message, reply_all);
 
-	subject = notmuch_message_get_header (message, "subject");
-	if (subject) {
-	    if (strncasecmp (subject, "Re:", 3))
-		subject = talloc_asprintf (ctx, "Re: %s", subject);
-	    g_mime_message_set_subject (reply, subject);
+	/* If reply creation failed, we're out of memory, so don't
+	 * bother trying any more messages.
+	 */
+	if (!reply) {
+	    notmuch_message_destroy (message);
+	    return 1;
 	}
 
-	from_addr = add_recipients_from_message (reply, config, message,
-						 reply_all);
-
-	if (from_addr == NULL)
-	    from_addr = guess_from_received_header (config, message);
-
-	if (from_addr == NULL)
-	    from_addr = notmuch_config_get_user_primary_email (config);
-
-	from_addr = talloc_asprintf (ctx, "%s <%s>",
-				     notmuch_config_get_user_name (config),
-				     from_addr);
-	g_mime_object_set_header (GMIME_OBJECT (reply),
-				  "From", from_addr);
-
-	in_reply_to = talloc_asprintf (ctx, "<%s>",
-			     notmuch_message_get_message_id (message));
-
-	g_mime_object_set_header (GMIME_OBJECT (reply),
-				  "In-Reply-To", in_reply_to);
-
-	orig_references = notmuch_message_get_header (message, "references");
-	references = talloc_asprintf (ctx, "%s%s%s",
-				      orig_references ? orig_references : "",
-				      orig_references ? " " : "",
-				      in_reply_to);
-	g_mime_object_set_header (GMIME_OBJECT (reply),
-				  "References", references);
-
 	show_reply_headers (reply);
 
 	g_object_unref (G_OBJECT (reply));
-- 
1.7.5.4

[PATCH v7.1 03/11] TODO: Add replying to multiple messages

[Adam Wolfe Gordon]

---
 devel/TODO |    8 ++++++++
 1 files changed, 8 insertions(+), 0 deletions(-)

diff --git a/devel/TODO b/devel/TODO
index 4dda6f4..7b750af 100644
--- a/devel/TODO
+++ b/devel/TODO
@@ -141,6 +141,14 @@ Simplify notmuch-reply to simply print the headers (we have the
 original values) rather than calling GMime (which encodes) and adding
 the confusing gmime-filter-headers.c code (which decodes).
 
+Properly handle replying to multiple messages. Currently, the JSON
+reply format only supports a single message, but the default reply
+format accepts searches returning multiple messages. The expected
+behavior of replying to multiple messages is not obvious, and there
+are multiple ideas that might make sense. Some consensus needs to be
+reached on this issue, and then both reply formats should be updated
+to be consistent.
+
 notmuch library
 ---------------
 Add support for custom flag<->tag mappings. In the notmuch
-- 
1.7.5.4

[PATCH v7.1 04/11] reply: Add a JSON reply format.

[Adam Wolfe Gordon]

This new JSON format for replies includes headers generated for a
reply message as well as the headers of the original message.  Using
this data, a client can intelligently create a reply. For example, the
emacs client will be able to create replies with quoted HTML parts by
parsing the HTML parts.
---
 notmuch-client.h |   14 ++++++++++----
 notmuch-reply.c  |   49 +++++++++++++++++++++++++++++++++++++++++++++++++
 notmuch-show.c   |   29 +++++++++++++++++++++--------
 test/multipart   |    1 -
 4 files changed, 80 insertions(+), 13 deletions(-)

diff --git a/notmuch-client.h b/notmuch-client.h
index f4a62cc..270daff 100644
--- a/notmuch-client.h
+++ b/notmuch-client.h
@@ -62,13 +62,13 @@
 #define STRINGIFY(s) STRINGIFY_(s)
 #define STRINGIFY_(s) #s
 
-struct mime_node;
+typedef struct mime_node mime_node_t;
 struct notmuch_show_params;
 
 typedef struct notmuch_show_format {
     const char *message_set_start;
     void (*part) (const void *ctx,
-		  struct mime_node *node, int indent,
+		  mime_node_t *node, int indent,
 		  const struct notmuch_show_params *params);
     const char *message_start;
     void (*message) (const void *ctx,
@@ -191,6 +191,12 @@ show_message_body (notmuch_message_t *message,
 notmuch_status_t
 show_one_part (const char *filename, int part);
 
+void
+format_part_json (const void *ctx, mime_node_t *node, notmuch_bool_t first);
+
+void
+format_headers_json (const void *ctx, GMimeMessage *message, notmuch_bool_t reply);
+
 char *
 json_quote_chararray (const void *ctx, const char *str, const size_t len);
 
@@ -288,7 +294,7 @@ debugger_is_active (void);
  * parts.  Message-type parts have one child, multipart-type parts
  * have multiple children, and leaf parts have zero children.
  */
-typedef struct mime_node {
+struct mime_node {
     /* The MIME object of this part.  This will be a GMimeMessage,
      * GMimePart, GMimeMultipart, or a subclass of one of these.
      *
@@ -351,7 +357,7 @@ typedef struct mime_node {
      * number to assign it (or -1 if unknown). */
     int next_child;
     int next_part_num;
-} mime_node_t;
+};
 
 /* Construct a new MIME node pointing to the root message part of
  * message.  If cryptoctx is non-NULL, it will be used to verify
diff --git a/notmuch-reply.c b/notmuch-reply.c
index f1478cc..e2b6c25 100644
--- a/notmuch-reply.c
+++ b/notmuch-reply.c
@@ -604,6 +604,51 @@ notmuch_reply_format_default(void *ctx,
     return 0;
 }
 
+static int
+notmuch_reply_format_json(void *ctx,
+			  notmuch_config_t *config,
+			  notmuch_query_t *query,
+			  notmuch_show_params_t *params,
+			  notmuch_bool_t reply_all)
+{
+    GMimeMessage *reply;
+    notmuch_messages_t *messages;
+    notmuch_message_t *message;
+    mime_node_t *node;
+
+    if (notmuch_query_count_messages (query) != 1) {
+	fprintf (stderr, "Error: search term did not match precisely one message.\n");
+	return 1;
+    }
+
+    messages = notmuch_query_search_messages (query);
+    message = notmuch_messages_get (messages);
+    if (mime_node_open (ctx, message, params->cryptoctx, params->decrypt,
+			&node) != NOTMUCH_STATUS_SUCCESS)
+	return 1;
+
+    reply = create_reply_message (ctx, config, message, reply_all);
+    if (!reply)
+	return 1;
+
+    /* The headers of the reply message we've created */
+    printf ("{\"reply-headers\": ");
+    format_headers_json (ctx, reply, TRUE);
+    g_object_unref (G_OBJECT (reply));
+    reply = NULL;
+
+    /* Start the original */
+    printf (", \"original\": ");
+
+    format_part_json (ctx, node, TRUE);
+
+    /* End */
+    printf ("}\n");
+    notmuch_message_destroy (message);
+
+    return 0;
+}
+
 /* This format is currently tuned for a git send-email --notmuch hook */
 static int
 notmuch_reply_format_headers_only(void *ctx,
@@ -666,6 +711,7 @@ notmuch_reply_format_headers_only(void *ctx,
 
 enum {
     FORMAT_DEFAULT,
+    FORMAT_JSON,
     FORMAT_HEADERS_ONLY,
 };
 
@@ -685,6 +731,7 @@ notmuch_reply_command (void *ctx, int argc, char *argv[])
     notmuch_opt_desc_t options[] = {
 	{ NOTMUCH_OPT_KEYWORD, &format, "format", 'f',
 	  (notmuch_keyword_t []){ { "default", FORMAT_DEFAULT },
+				  { "json", FORMAT_JSON },
 				  { "headers-only", FORMAT_HEADERS_ONLY },
 				  { 0, 0 } } },
 	{ NOTMUCH_OPT_KEYWORD, &reply_all, "reply-to", 'r',
@@ -703,6 +750,8 @@ notmuch_reply_command (void *ctx, int argc, char *argv[])
 
     if (format == FORMAT_HEADERS_ONLY)
 	reply_format_func = notmuch_reply_format_headers_only;
+    else if (format == FORMAT_JSON)
+	reply_format_func = notmuch_reply_format_json;
     else
 	reply_format_func = notmuch_reply_format_default;
 
diff --git a/notmuch-show.c b/notmuch-show.c
index 05d51b2..cb3a427 100644
--- a/notmuch-show.c
+++ b/notmuch-show.c
@@ -291,8 +291,8 @@ format_headers_message_part_text (GMimeMessage *message)
     printf ("Date: %s\n", g_mime_message_get_date_as_string (message));
 }
 
-static void
-format_headers_json (const void *ctx, GMimeMessage *message)
+void
+format_headers_json (const void *ctx, GMimeMessage *message, notmuch_bool_t reply)
 {
     void *local = talloc_new (ctx);
     InternetAddressList *recipients;
@@ -316,9 +316,22 @@ format_headers_json (const void *ctx, GMimeMessage *message)
 	printf (", %s: %s",
 		json_quote_str (local, "Cc"),
 		json_quote_str (local, recipients_string));
-    printf (", %s: %s}",
-	    json_quote_str (local, "Date"),
-	    json_quote_str (local, g_mime_message_get_date_as_string (message)));
+
+    if (reply) {
+	printf (", %s: %s",
+		json_quote_str (local, "In-reply-to"),
+		json_quote_str (local, g_mime_object_get_header (GMIME_OBJECT (message), "In-reply-to")));
+
+	printf (", %s: %s",
+		json_quote_str (local, "References"),
+		json_quote_str (local, g_mime_object_get_header (GMIME_OBJECT (message), "References")));
+    } else {
+	printf (", %s: %s",
+		json_quote_str (local, "Date"),
+		json_quote_str (local, g_mime_message_get_date_as_string (message)));
+    }
+
+    printf ("}");
 
     talloc_free (local);
 }
@@ -654,7 +667,7 @@ format_part_text (const void *ctx, mime_node_t *node,
     printf ("\f%s}\n", part_type);
 }
 
-static void
+void
 format_part_json (const void *ctx, mime_node_t *node, notmuch_bool_t first)
 {
     /* Any changes to the JSON format should be reflected in the file
@@ -665,7 +678,7 @@ format_part_json (const void *ctx, mime_node_t *node, notmuch_bool_t first)
 	format_message_json (ctx, node->envelope_file);
 
 	printf ("\"headers\": ");
-	format_headers_json (ctx, GMIME_MESSAGE (node->part));
+	format_headers_json (ctx, GMIME_MESSAGE (node->part), FALSE);
 
 	printf (", \"body\": [");
 	format_part_json (ctx, mime_node_child (node, 0), first);
@@ -739,7 +752,7 @@ format_part_json (const void *ctx, mime_node_t *node, notmuch_bool_t first)
     } else if (GMIME_IS_MESSAGE (node->part)) {
 	printf (", \"content\": [{");
 	printf ("\"headers\": ");
-	format_headers_json (local, GMIME_MESSAGE (node->part));
+	format_headers_json (local, GMIME_MESSAGE (node->part), FALSE);
 
 	printf (", \"body\": [");
 	terminator = "]}]";
diff --git a/test/multipart b/test/multipart
index 80d6e88..2383b9c 100755
--- a/test/multipart
+++ b/test/multipart
@@ -590,7 +590,6 @@ EOF
 test_expect_equal_file OUTPUT EXPECTED
 
 test_begin_subtest "'notmuch reply' to a multipart message with json format"
-test_subtest_known_broken
 notmuch reply --format=json 'id:87liy5ap00.fsf@yoom.home.cworth.org' | notmuch_json_show_sanitize >OUTPUT
 cat <<EOF >EXPECTED
 {"reply-headers": {"Subject": "Re: Multipart message",
-- 
1.7.5.4

[PATCH v7.1 05/11] schemata: Add documentation for JSON reply format.

[Adam Wolfe Gordon]

---
 devel/schemata |   27 +++++++++++++++++++++++++--
 1 files changed, 25 insertions(+), 2 deletions(-)

diff --git a/devel/schemata b/devel/schemata
index 24ad775..728a46f 100644
--- a/devel/schemata
+++ b/devel/schemata
@@ -77,8 +77,9 @@ part = {
     content?:       string
 }
 
-# The headers of a message (format_headers_json with raw headers) or
-# a part (format_headers_message_part_json with pretty-printed headers)
+# The headers of a message (format_headers_json with raw headers
+# and reply = FALSE) or a part (format_headers_message_part_json
+# with pretty-printed headers)
 headers = {
     Subject:        string,
     From:           string,
@@ -136,3 +137,25 @@ thread = {
                               # matched and unmatched
     subject:        string
 }
+
+notmuch reply schema
+--------------------
+
+reply = {
+    # The headers of the constructed reply (format_headers_json with
+    # raw headers and reply = TRUE)
+    reply-headers: reply_headers,
+
+    # As in the show format (format_part_json)
+    original: message
+}
+
+reply_headers = {
+    Subject:        string,
+    From:           string,
+    To?:            string,
+    Cc?:            string,
+    Bcc?:           string,
+    In-reply-to:    string,
+    References:     string
+}
-- 
1.7.5.4

[PATCH v7.1 06/11] man: Update notmuch-reply man page for JSON format.

[Adam Wolfe Gordon]

---
 man/man1/notmuch-reply.1 |   10 ++++++++--
 1 files changed, 8 insertions(+), 2 deletions(-)

diff --git a/man/man1/notmuch-reply.1 b/man/man1/notmuch-reply.1
index 7ed7f0f..debc505 100644
--- a/man/man1/notmuch-reply.1
+++ b/man/man1/notmuch-reply.1
@@ -37,12 +37,17 @@ Supported options for
 include
 .RS
 .TP 4
-.BR \-\-format= ( default | headers\-only )
+.BR \-\-format= ( default | json | headers\-only )
 .RS
 .TP 4
 .BR default
 Includes subject and quoted message body.
 .TP
+.BR json
+Produces JSON output containing headers for a reply message and the
+contents of the original message. This output can be used by a client
+to create a reply message intelligently.
+.TP
 .BR headers\-only
 Only produces In\-Reply\-To, References, To, Cc, and Bcc headers.
 .RE
@@ -73,7 +78,8 @@ with a search string matching a single message, (such as
 id:<message-id>), but it can be useful to reply to several messages at
 once. For example, when a series of patches are sent in a single
 thread, replying to the entire thread allows for the reply to comment
-on issue found in multiple patches.
+on issues found in multiple patches. The default format supports
+replying to multiple messages at once, but the JSON format does not.
 .RE
 .RE
 
-- 
1.7.5.4

[PATCH v7.1 07/11] man: Add --decrypt to reply flags

[Adam Wolfe Gordon]

---
 man/man1/notmuch-reply.1 |   10 ++++++++++
 1 files changed, 10 insertions(+), 0 deletions(-)

diff --git a/man/man1/notmuch-reply.1 b/man/man1/notmuch-reply.1
index debc505..d9912c4 100644
--- a/man/man1/notmuch-reply.1
+++ b/man/man1/notmuch-reply.1
@@ -68,6 +68,16 @@ values from the first that contains something other than only the
 user's addresses.
 .RE
 .RE
+.RS
+.TP 4
+.B \-\-decrypt
+
+Decrypt any MIME encrypted parts found in the selected content
+(ie. "multipart/encrypted" parts). Status of the decryption will be
+reported (currently only supported with --format=json) and the
+multipart/encrypted part will be replaced by the decrypted
+content.
+.RE
 
 See \fBnotmuch-search-terms\fR(7)
 for details of the supported syntax for <search-terms>.
-- 
1.7.5.4

[PATCH v7.1 08/11] emacs: Factor out useful functions into notmuch-lib

[Adam Wolfe Gordon]

Move a few functions related to handling multipart/alternative parts
into notmuch-lib.el, so they can be used by future reply code.
---
 emacs/notmuch-lib.el  |   33 +++++++++++++++++++++++++++++++++
 emacs/notmuch-show.el |   24 ++----------------------
 2 files changed, 35 insertions(+), 22 deletions(-)

diff --git a/emacs/notmuch-lib.el b/emacs/notmuch-lib.el
index d315f76..7e3f110 100644
--- a/emacs/notmuch-lib.el
+++ b/emacs/notmuch-lib.el
@@ -21,6 +21,8 @@
 
 ;; This is an part of an emacs-based interface to the notmuch mail system.
 
+(eval-when-compile (require 'cl))
+
 (defvar notmuch-command "notmuch"
   "Command to run the notmuch binary.")
 
@@ -173,6 +175,37 @@ the user hasn't set this variable with the old or new value."
   (list 'when (< emacs-major-version 23)
 	form))
 
+(defun notmuch-split-content-type (content-type)
+  "Split content/type into 'content' and 'type'"
+  (split-string content-type "/"))
+
+(defun notmuch-match-content-type (t1 t2)
+  "Return t if t1 and t2 are matching content types, taking wildcards into account"
+  (let ((st1 (notmuch-split-content-type t1))
+	(st2 (notmuch-split-content-type t2)))
+    (if (or (string= (cadr st1) "*")
+	    (string= (cadr st2) "*"))
+	(string= (car st1) (car st2))
+      (string= t1 t2))))
+
+(defvar notmuch-multipart/alternative-discouraged
+  '(
+    ;; Avoid HTML parts.
+    "text/html"
+    ;; multipart/related usually contain a text/html part and some associated graphics.
+    "multipart/related"
+    ))
+
+(defun notmuch-multipart/alternative-choose (types)
+  "Return a list of preferred types from the given list of types"
+  ;; Based on `mm-preferred-alternative-precedence'.
+  (let ((seq types))
+    (dolist (pref (reverse notmuch-multipart/alternative-discouraged))
+      (dolist (elem (copy-sequence seq))
+	(when (string-match pref elem)
+	  (setq seq (nconc (delete elem seq) (list elem))))))
+    seq))
+
 ;; Compatibility functions for versions of emacs before emacs 23.
 ;;
 ;; Both functions here were copied from emacs 23 with the following copyright:
diff --git a/emacs/notmuch-show.el b/emacs/notmuch-show.el
index 4a60631..ed938bf 100644
--- a/emacs/notmuch-show.el
+++ b/emacs/notmuch-show.el
@@ -542,30 +542,13 @@ current buffer, if possible."
 	    (mm-display-part handle)
 	    t))))))
 
-(defvar notmuch-show-multipart/alternative-discouraged
-  '(
-    ;; Avoid HTML parts.
-    "text/html"
-    ;; multipart/related usually contain a text/html part and some associated graphics.
-    "multipart/related"
-    ))
-
 (defun notmuch-show-multipart/*-to-list (part)
   (mapcar (lambda (inner-part) (plist-get inner-part :content-type))
 	  (plist-get part :content)))
 
-(defun notmuch-show-multipart/alternative-choose (types)
-  ;; Based on `mm-preferred-alternative-precedence'.
-  (let ((seq types))
-    (dolist (pref (reverse notmuch-show-multipart/alternative-discouraged))
-      (dolist (elem (copy-sequence seq))
-	(when (string-match pref elem)
-	  (setq seq (nconc (delete elem seq) (list elem))))))
-    seq))
-
 (defun notmuch-show-insert-part-multipart/alternative (msg part content-type nth depth declared-type)
   (notmuch-show-insert-part-header nth declared-type content-type nil)
-  (let ((chosen-type (car (notmuch-show-multipart/alternative-choose (notmuch-show-multipart/*-to-list part))))
+  (let ((chosen-type (car (notmuch-multipart/alternative-choose (notmuch-show-multipart/*-to-list part))))
 	(inner-parts (plist-get part :content))
 	(start (point)))
     ;; This inserts all parts of the chosen type rather than just one,
@@ -808,9 +791,6 @@ current buffer, if possible."
 
 ;; Functions for determining how to handle MIME parts.
 
-(defun notmuch-show-split-content-type (content-type)
-  (split-string content-type "/"))
-
 (defun notmuch-show-handlers-for (content-type)
   "Return a list of content handlers for a part of type CONTENT-TYPE."
   (let (result)
@@ -821,7 +801,7 @@ current buffer, if possible."
 	  (list (intern (concat "notmuch-show-insert-part-*/*"))
 		(intern (concat
 			 "notmuch-show-insert-part-"
-			 (car (notmuch-show-split-content-type content-type))
+			 (car (notmuch-split-content-type content-type))
 			 "/*"))
 		(intern (concat "notmuch-show-insert-part-" content-type))))
     result))
-- 
1.7.5.4

[PATCH v7.1 09/11] test: Add broken tests for new emacs reply functionality

[Adam Wolfe Gordon]

Add tests for creating nice replies to multipart messages, including
those with HTML parts. These tests are expected to fail for now.
---
 test/emacs |   97 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 files changed, 97 insertions(+), 0 deletions(-)

diff --git a/test/emacs b/test/emacs
index 29a489c..01afdb6 100755
--- a/test/emacs
+++ b/test/emacs
@@ -273,6 +273,103 @@ On 01 Jan 2000 12:00:00 -0000, Notmuch Test Suite <test_suite@notmuchmail.org> w
 EOF
 test_expect_equal_file OUTPUT EXPECTED
 
+test_begin_subtest "Reply within emacs to a multipart/mixed message"
+test_subtest_known_broken
+test_emacs '(notmuch-show "id:20091118002059.067214ed@hikari")
+		(notmuch-show-reply)
+		(test-output)'
+cat <<EOF >EXPECTED
+From: Notmuch Test Suite <test_suite@notmuchmail.org>
+To: Adrian Perez de Castro <aperez@igalia.com>, notmuch@notmuchmail.org
+Subject: Re: [notmuch] Introducing myself
+In-Reply-To: <20091118002059.067214ed@hikari>
+Fcc: ${MAIL_DIR}/sent
+--text follows this line--
+Adrian Perez de Castro <aperez@igalia.com> writes:
+
+> Hello to all,
+>
+> I have just heard about Not Much today in some random Linux-related news
+> site (LWN?), my name is Adrian Perez and I work as systems administrator
+> (although I can do some code as well :P). I have always thought that the
+> ideas behind Sup were great, but after some time using it, I got tired of
+> the oddities that it has. I also do not like doing things like having to
+> install Ruby just for reading and sorting mails. Some time ago I thought
+> about doing something like Not Much and in fact I played a bit with the
+> Python+Xapian and the Python+Whoosh combinations, because I find relaxing
+> to code things in Python when I am not working and also it is installed
+> by default on most distribution. I got to have some mailboxes indexed and
+> basic searching working a couple of months ago. Lately I have been very
+> busy and had no time for coding, and them... boom! Not Much appears -- and
+> it is almost exactly what I was trying to do, but faster. I have been
+> playing a bit with Not Much today, and I think it has potential.
+>
+> Also, I would like to share one idea I had in mind, that you might find
+> interesting: One thing I have found very annoying is having to re-tag my
+> mail when the indexes get b0rked (it happened a couple of times to me while
+> using Sup), so I was planning to mails as read/unread and adding the tags
+> not just to the index, but to the mail text itself, e.g. by adding a
+> "X-Tags" header field or by reusing the "Keywords" one. This way, the index
+> could be totally recreated by re-reading the mail directories, and this
+> would also allow to a tools like OfflineIMAP [1] to get the mails into a
+> local maildir, tagging and indexing the mails with the e-mail reader and
+> then syncing back the messages with the "X-Tags" header to the IMAP server.
+> This would allow to use the mail reader from a different computer and still
+> have everything tagged finely.
+>
+> Best regards,
+>
+>
+> ---
+> [1] http://software.complete.org/software/projects/show/offlineimap
+>
+> -- 
+> Adrian Perez de Castro <aperez@igalia.com>
+> Igalia - Free Software Engineering
+> _______________________________________________
+> notmuch mailing list
+> notmuch@notmuchmail.org
+> http://notmuchmail.org/mailman/listinfo/notmuch
+EOF
+test_expect_equal_file OUTPUT EXPECTED
+
+test_begin_subtest "Reply within emacs to a multipart/alternative message"
+test_subtest_known_broken
+test_emacs '(notmuch-show "id:cf0c4d610911171136h1713aa59w9cf9aa31f052ad0a@mail.gmail.com")
+		(notmuch-show-reply)
+		(test-output)'
+cat <<EOF >EXPECTED
+From: Notmuch Test Suite <test_suite@notmuchmail.org>
+To: Alex Botero-Lowry <alex.boterolowry@gmail.com>, notmuch@notmuchmail.org
+Subject: Re: [notmuch] preliminary FreeBSD support
+In-Reply-To: <cf0c4d610911171136h1713aa59w9cf9aa31f052ad0a@mail.gmail.com>
+Fcc: ${MAIL_DIR}/sent
+--text follows this line--
+Alex Botero-Lowry <alex.boterolowry@gmail.com> writes:
+
+> I saw the announcement this morning, and was very excited, as I had been
+> hoping sup would be turned into a library,
+> since I like the concept more than the UI (I'd rather an emacs interface).
+>
+> I did a preliminary compile which worked out fine, but
+> sysconf(_SC_SC_GETPW_R_SIZE_MAX) returns -1 on
+> FreeBSD, so notmuch_config_open segfaulted.
+>
+> Attached is a patch that supplies a default buffer size of 64 in cases where
+> -1 is returned.
+>
+> http://www.opengroup.org/austin/docs/austin_328.txt - seems to indicate this
+> is acceptable behavior,
+> and http://mail-index.netbsd.org/pkgsrc-bugs/2006/06/07/msg016808.htmlspecifically
+> uses 64 as the
+> buffer size.
+> _______________________________________________
+> notmuch mailing list
+> notmuch@notmuchmail.org
+> http://notmuchmail.org/mailman/listinfo/notmuch
+EOF
+test_expect_equal_file OUTPUT EXPECTED
+
 test_begin_subtest "Quote MML tags in reply"
 message_id='test-emacs-mml-quoting@message.id'
 add_message [id]="$message_id" \
-- 
1.7.5.4

[PATCH v7.1 10/11] emacs: Use the new JSON reply format and message-cite-original

[Adam Wolfe Gordon]

Use the new JSON reply format to create replies in emacs. Quote HTML
parts nicely by using mm-display-part to turn them into displayable
text, then quoting them with message-cite-original. This is very
useful for users who regularly receive HTML-only email.

Use message-mode's message-cite-original function to create the
quoted body for reply messages. In order to make this act like the
existing notmuch defaults, you will need to set the following in
your emacs configuration:

message-citation-line-format "On %a, %d %b %Y, %f wrote:"
message-citation-line-function 'message-insert-formatted-citation-line

The tests have been updated to reflect the (ugly) emacs default.
---
 emacs/notmuch-lib.el  |   30 ++++++++++++
 emacs/notmuch-mua.el  |  124 +++++++++++++++++++++++++++++++++----------------
 emacs/notmuch-show.el |   31 ++----------
 test/emacs            |    8 ++--
 4 files changed, 123 insertions(+), 70 deletions(-)

diff --git a/emacs/notmuch-lib.el b/emacs/notmuch-lib.el
index 7e3f110..c146748 100644
--- a/emacs/notmuch-lib.el
+++ b/emacs/notmuch-lib.el
@@ -206,6 +206,36 @@ the user hasn't set this variable with the old or new value."
 	  (setq seq (nconc (delete elem seq) (list elem))))))
     seq))
 
+(defun notmuch-parts-filter-by-type (parts type)
+  "Given a list of message parts, return a list containing the ones matching
+the given type."
+  (remove-if-not
+   (lambda (part) (notmuch-match-content-type (plist-get part :content-type) type))
+   parts))
+
+;; Helper for parts which are generally not included in the default
+;; JSON output.
+(defun notmuch-get-bodypart-internal (message-id part-number process-crypto)
+  (let ((args '("show" "--format=raw"))
+	(part-arg (format "--part=%s" part-number)))
+    (setq args (append args (list part-arg)))
+    (if process-crypto
+	(setq args (append args '("--decrypt"))))
+    (setq args (append args (list message-id)))
+    (with-temp-buffer
+      (let ((coding-system-for-read 'no-conversion))
+	(progn
+	  (apply 'call-process (append (list notmuch-command nil (list t nil) nil) args))
+	  (buffer-string))))))
+
+(defun notmuch-get-bodypart-content (msg part nth process-crypto)
+  (or (plist-get part :content)
+      (notmuch-get-bodypart-internal (concat "id:" (plist-get msg :id)) nth process-crypto)))
+
+(defun notmuch-plist-to-alist (plist)
+  (loop for (key value . rest) on plist by #'cddr
+	collect (cons (substring (symbol-name key) 1) value)))
+
 ;; Compatibility functions for versions of emacs before emacs 23.
 ;;
 ;; Both functions here were copied from emacs 23 with the following copyright:
diff --git a/emacs/notmuch-mua.el b/emacs/notmuch-mua.el
index 13244eb..6aae3a0 100644
--- a/emacs/notmuch-mua.el
+++ b/emacs/notmuch-mua.el
@@ -19,11 +19,15 @@
 ;;
 ;; Authors: David Edmondson <dme@dme.org>
 
+(require 'json)
 (require 'message)
+(require 'format-spec)
 
 (require 'notmuch-lib)
 (require 'notmuch-address)
 
+(eval-when-compile (require 'cl))
+
 ;;
 
 (defcustom notmuch-mua-send-hook '(notmuch-mua-message-send-hook)
@@ -72,54 +76,92 @@ list."
 	    (push header message-hidden-headers)))
 	notmuch-mua-hidden-headers))
 
+(defun notmuch-mua-get-quotable-parts (parts)
+  (loop for part in parts
+	if (notmuch-match-content-type (plist-get part :content-type) "multipart/alternative")
+	  collect (let* ((subparts (plist-get part :content))
+			(types (mapcar (lambda (part) (plist-get part :content-type)) subparts))
+			(chosen-type (car (notmuch-multipart/alternative-choose types))))
+		   (loop for part in (reverse subparts)
+			 if (notmuch-match-content-type (plist-get part :content-type) chosen-type)
+			 return part))
+	else if (notmuch-match-content-type (plist-get part :content-type) "multipart/*")
+	  append (notmuch-mua-get-quotable-parts (plist-get part :content))
+	else if (notmuch-match-content-type (plist-get part :content-type) "text/*")
+	  collect part))
+
 (defun notmuch-mua-reply (query-string &optional sender reply-all)
-  (let (headers
-	body
-	(args '("reply")))
-    (if notmuch-show-process-crypto
-	(setq args (append args '("--decrypt"))))
+  (let ((args '("reply" "--format=json"))
+	reply
+	original)
+    (when notmuch-show-process-crypto
+      (setq args (append args '("--decrypt"))))
+
     (if reply-all
 	(setq args (append args '("--reply-to=all")))
       (setq args (append args '("--reply-to=sender"))))
     (setq args (append args (list query-string)))
-    ;; This make assumptions about the output of `notmuch reply', but
-    ;; really only that the headers come first followed by a blank
-    ;; line and then the body.
+
+    ;; Get the reply object as JSON, and parse it into an elisp object.
     (with-temp-buffer
       (apply 'call-process (append (list notmuch-command nil (list t t) nil) args))
       (goto-char (point-min))
-      (if (re-search-forward "^$" nil t)
-	  (save-excursion
-	    (save-restriction
-	      (narrow-to-region (point-min) (point))
-	      (goto-char (point-min))
-	      (setq headers (mail-header-extract)))))
-      (forward-line 1)
-      ;; Original message may contain (malicious) MML tags. We must
-      ;; properly quote them in the reply.
-      (mml-quote-region (point) (point-max))
-      (setq body (buffer-substring (point) (point-max))))
-    ;; If sender is non-nil, set the From: header to its value.
-    (when sender
-      (mail-header-set 'from sender headers))
-    (let
-	;; Overlay the composition window on that being used to read
-	;; the original message.
-	((same-window-regexps '("\\*mail .*")))
-      (notmuch-mua-mail (mail-header 'to headers)
-			(mail-header 'subject headers)
-			(message-headers-to-generate headers t '(to subject))))
-    ;; insert the message body - but put it in front of the signature
-    ;; if one is present
-    (goto-char (point-max))
-    (if (re-search-backward message-signature-separator nil t)
+      (let ((json-object-type 'plist)
+	    (json-array-type 'list)
+	    (json-false 'nil))
+	(setq reply (json-read))))
+
+    ;; Extract the original message to simplify the following code.
+    (setq original (plist-get reply :original))
+
+    ;; Extract the headers of both the reply and the original message.
+    (let* ((original-headers (plist-get original :headers))
+	   (reply-headers (plist-get reply :reply-headers)))
+
+      ;; If sender is non-nil, set the From: header to its value.
+      (when sender
+	(plist-put reply-headers :From sender))
+      (let
+	  ;; Overlay the composition window on that being used to read
+	  ;; the original message.
+	  ((same-window-regexps '("\\*mail .*")))
+	(notmuch-mua-mail (plist-get reply-headers :To)
+			  (plist-get reply-headers :Subject)
+			  (notmuch-plist-to-alist reply-headers)))
+      ;; Insert the message body - but put it in front of the signature
+      ;; if one is present
+      (goto-char (point-max))
+      (if (re-search-backward message-signature-separator nil t)
 	  (forward-line -1)
-      (goto-char (point-max)))
-    (insert body)
-    (push-mark))
-  (set-buffer-modified-p nil)
-
-  (message-goto-body))
+	(goto-char (point-max)))
+
+      (let ((from (plist-get original-headers :From))
+	    (date (plist-get original-headers :Date))
+	    (start (point)))
+
+	;; message-cite-original constructs a citation line based on the From and Date
+	;; headers of the original message, which are assumed to be in the buffer.
+	(insert "From: " from "\n")
+	(insert "Date: " date "\n\n")
+
+	;; Get the parts of the original message that should be quoted; this includes
+	;; all the text parts, except the non-preferred ones in a multipart/alternative.
+	(let ((quotable-parts (notmuch-mua-get-quotable-parts (plist-get original :body))))
+	  (mapc (lambda (part)
+		  (insert (notmuch-get-bodypart-content original part
+							(plist-get part :id)
+							notmuch-show-process-crypto)))
+		quotable-parts))
+
+	(set-mark (point))
+	(goto-char start)
+	;; Quote the original message according to the user's configured style.
+	(message-cite-original))))
+
+  (goto-char (point-max))
+  (push-mark)
+  (message-goto-body)
+  (set-buffer-modified-p nil))
 
 (defun notmuch-mua-forward-message ()
   (message-forward)
@@ -145,7 +187,7 @@ OTHER-ARGS are passed through to `message-mail'."
       (when (not (string= "" user-agent))
 	(push (cons "User-Agent" user-agent) other-headers))))
 
-  (unless (mail-header 'from other-headers)
+  (unless (mail-header 'From other-headers)
     (push (cons "From" (concat
 			(notmuch-user-name) " <" (notmuch-user-primary-email) ">")) other-headers))
 
@@ -208,7 +250,7 @@ the From: address first."
   (interactive "P")
   (let ((other-headers
 	 (when (or prompt-for-sender notmuch-always-prompt-for-sender)
-	   (list (cons 'from (notmuch-mua-prompt-for-sender))))))
+	   (list (cons 'From (notmuch-mua-prompt-for-sender))))))
     (notmuch-mua-mail nil nil other-headers)))
 
 (defun notmuch-mua-new-forward-message (&optional prompt-for-sender)
diff --git a/emacs/notmuch-show.el b/emacs/notmuch-show.el
index ed938bf..0cd7d82 100644
--- a/emacs/notmuch-show.el
+++ b/emacs/notmuch-show.el
@@ -488,7 +488,7 @@ message at DEPTH in the current thread."
 	 (setq notmuch-show-process-crypto ,process-crypto)
 	 ;; Always acquires the part via `notmuch part', even if it is
 	 ;; available in the JSON output.
-	 (insert (notmuch-show-get-bodypart-internal ,message-id ,nth))
+	 (insert (notmuch-get-bodypart-internal ,message-id ,nth notmuch-show-process-crypto))
 	 ,@body))))
 
 (defun notmuch-show-save-part (message-id nth &optional filename content-type)
@@ -536,7 +536,7 @@ current buffer, if possible."
 	;; test whether we are able to inline it (which includes both
 	;; capability and suitability tests).
 	(when (mm-inlined-p handle)
-	  (insert (notmuch-show-get-bodypart-content msg part nth))
+	  (insert (notmuch-get-bodypart-content msg part nth notmuch-show-process-crypto))
 	  (when (mm-inlinable-p handle)
 	    (set-buffer display-buffer)
 	    (mm-display-part handle)
@@ -613,8 +613,8 @@ current buffer, if possible."
 	  ;; times (hundreds!), which results in many calls to
 	  ;; `notmuch part'.
 	  (unless content
-	    (setq content (notmuch-show-get-bodypart-internal (concat "id:" message-id)
-							      part-number))
+	    (setq content (notmuch-get-bodypart-internal (concat "id:" message-id)
+							      part-number notmuch-show-process-crypto))
 	    (with-current-buffer w3m-current-buffer
 	      (notmuch-show-w3m-cid-store-internal url
 						   message-id
@@ -734,7 +734,7 @@ current buffer, if possible."
     ;; insert a header to make this clear.
     (if (> nth 1)
 	(notmuch-show-insert-part-header nth declared-type content-type (plist-get part :filename)))
-    (insert (notmuch-show-get-bodypart-content msg part nth))
+    (insert (notmuch-get-bodypart-content msg part nth notmuch-show-process-crypto))
     (save-excursion
       (save-restriction
 	(narrow-to-region start (point-max))
@@ -744,7 +744,7 @@ current buffer, if possible."
 (defun notmuch-show-insert-part-text/calendar (msg part content-type nth depth declared-type)
   (notmuch-show-insert-part-header nth declared-type content-type (plist-get part :filename))
   (insert (with-temp-buffer
-	    (insert (notmuch-show-get-bodypart-content msg part nth))
+	    (insert (notmuch-get-bodypart-content msg part nth notmuch-show-process-crypto))
 	    (goto-char (point-min))
 	    (let ((file (make-temp-file "notmuch-ical"))
 		  result)
@@ -806,25 +806,6 @@ current buffer, if possible."
 		(intern (concat "notmuch-show-insert-part-" content-type))))
     result))
 
-;; Helper for parts which are generally not included in the default
-;; JSON output.
-(defun notmuch-show-get-bodypart-internal (message-id part-number)
-  (let ((args '("show" "--format=raw"))
-	(part-arg (format "--part=%s" part-number)))
-    (setq args (append args (list part-arg)))
-    (if notmuch-show-process-crypto
-	(setq args (append args '("--decrypt"))))
-    (setq args (append args (list message-id)))
-    (with-temp-buffer
-      (let ((coding-system-for-read 'no-conversion))
-	(progn
-	  (apply 'call-process (append (list notmuch-command nil (list t nil) nil) args))
-	  (buffer-string))))))
-
-(defun notmuch-show-get-bodypart-content (msg part nth)
-  (or (plist-get part :content)
-      (notmuch-show-get-bodypart-internal (concat "id:" (plist-get msg :id)) nth)))
-
 ;; \f
 
 (defun notmuch-show-insert-bodypart-internal (msg part content-type nth depth declared-type)
diff --git a/test/emacs b/test/emacs
index 01afdb6..8a28705 100755
--- a/test/emacs
+++ b/test/emacs
@@ -268,13 +268,13 @@ Subject: Re: Testing message sent via SMTP
 In-Reply-To: <XXX>
 Fcc: ${MAIL_DIR}/sent
 --text follows this line--
-On 01 Jan 2000 12:00:00 -0000, Notmuch Test Suite <test_suite@notmuchmail.org> wrote:
+Notmuch Test Suite <test_suite@notmuchmail.org> writes:
+
 > This is a test that messages are sent via SMTP
 EOF
 test_expect_equal_file OUTPUT EXPECTED
 
 test_begin_subtest "Reply within emacs to a multipart/mixed message"
-test_subtest_known_broken
 test_emacs '(notmuch-show "id:20091118002059.067214ed@hikari")
 		(notmuch-show-reply)
 		(test-output)'
@@ -334,7 +334,6 @@ EOF
 test_expect_equal_file OUTPUT EXPECTED
 
 test_begin_subtest "Reply within emacs to a multipart/alternative message"
-test_subtest_known_broken
 test_emacs '(notmuch-show "id:cf0c4d610911171136h1713aa59w9cf9aa31f052ad0a@mail.gmail.com")
 		(notmuch-show-reply)
 		(test-output)'
@@ -385,7 +384,8 @@ Subject: Re: Quote MML tags in reply
 In-Reply-To: <test-emacs-mml-quoting@message.id>
 Fcc: ${MAIL_DIR}/sent
 --text follows this line--
-On Fri, 05 Jan 2001 15:43:57 +0000, Notmuch Test Suite <test_suite@notmuchmail.org> wrote:
+Notmuch Test Suite <test_suite@notmuchmail.org> writes:
+
 > <#!part disposition=inline>
 EOF
 test_expect_equal_file OUTPUT EXPECTED
-- 
1.7.5.4

[PATCH v7.1 11/11] NEWS: news for reply enhancements

[Adam Wolfe Gordon]

---
 NEWS |   20 ++++++++++++++++++++
 1 files changed, 20 insertions(+), 0 deletions(-)

diff --git a/NEWS b/NEWS
index a739914..46fb191 100644
--- a/NEWS
+++ b/NEWS
@@ -11,6 +11,18 @@ Reply to sender
   to all. The feature is available through the new command line option
   --reply-to=(all|sender).
 
+JSON reply format
+
+  "notmuch reply" can now produce JSON output that contains the headers
+  for a reply message and full information about the original message
+  begin replied to. This allows MUAs to create replies intelligtently.
+  For example, an MUA that can parse HTML might quote HTML parts.
+
+  Calling notmuch reply with --format=json imposes the restriction that
+  only a single message is returned by the search, as replying to
+  multiple messages does not have a well-defined behavior. The default
+  retains its current behavior for multiple message replies.
+
 Tag exclusion
 
   Tags can be automatically excluded from search results by adding them
@@ -48,6 +60,14 @@ Reply to sender
   and search modes, 'r' has been bound to reply to sender, replacing
   reply to all, which now has key binding 'R'.
 
+Reply improvement using the JSON format
+
+  Emacs now uses the JSON reply format to create replies. It obeys
+  the customization variables message-citation-line-format and
+  message-citation-line-function when creating the first line of the
+  reply body, and it will quote HTML parts if no text/plain parts are
+  available.
+
 More flexible and consistent tagging operations
 
   All tagging operations ("+", "-", "*") now accept multiple tags with
-- 
1.7.5.4

Re: [PATCH v7.1 00/11] Reply enhancements, second attempt

[Austin Clements]

Quoth Adam Wolfe Gordon on Mar 13 at 10:30 pm:
> Hi everyone,
> 
> This is mostly a re-send of last night's series [1]. The only changes are:
> 
> * Fixed the emacs patch, where I had a rebase issue.
> * Fixed the commit that re-added test/test-lib by accident.
> * Fixed devel/schemata per Austin's review of the previous version.
> 
> Since this series seems to be getting close, I figured I'd add a NEWS entry
> for good measure.
> 
> Thanks for all the reviews so far, and I hope this is almost ready!
> 
> [1] id:"1331525142-30539-1-git-send-email-awg+notmuch@xvx.ca"

LGTM.

Re: [PATCH v7 00/10] Reply enhancements

[Jameson Graef Rollins]

[-- Attachment #1: Type: text/plain, Size: 728 bytes --]

On Sun, 11 Mar 2012 22:05:32 -0600, Adam Wolfe Gordon <awg+notmuch@xvx.ca> wrote:
> * I noticed that reply supports a --decrypt option, and the emacs interface
>   actually does use it. I've documented the option in the reply man page.
>   If someone can test that replying to encrypted messages actually works in
>   emacs, that would be great - I have no way to test this.

The test suite actually includes a test for replying to encrypted
messages (see crypto: "reply to encrypted message").  That test should
run as long as gnupg is available on the test system, and if it passes
you should be good to go.

But in any event I have tested this patch set, including the decryption
option, and it all looks good to me.

jamie.

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

Re: [PATCH v7 00/10] Reply enhancements

[Adam Wolfe Gordon]

On Sat, Mar 17, 2012 at 13:13, Jameson Graef Rollins
<jrollins@finestructure.net> wrote:
> The test suite actually includes a test for replying to encrypted
> messages (see crypto: "reply to encrypted message").  That test should
> run as long as gnupg is available on the test system, and if it passes
> you should be good to go.
>
> But in any event I have tested this patch set, including the decryption
> option, and it all looks good to me.

Thanks for testing.

The crypto test suite has never worked for me, even though I have
gnupg installed. I assume it's some sort of configuration issue on my
system, but I haven't been able to figure out what it is.

Re: [PATCH v7.1 00/11] Reply enhancements, second attempt

[David Bremner]

On Tue, 13 Mar 2012 22:30:05 -0600, Adam Wolfe Gordon <awg+notmuch@xvx.ca> wrote:
> Hi everyone,
> 
> This is mostly a re-send of last night's series [1]. The only changes are:
> 
> * Fixed the emacs patch, where I had a rebase issue.
> * Fixed the commit that re-added test/test-lib by accident.
> * Fixed devel/schemata per Austin's review of the previous version.

It looks like 04/11 doesn't apply anymore after the rest of the pending
patches were applied. It looked like the patches before that didn't
really make sense in isolation so I'm going to tag the whole series
stale.  When you rebase, no need for "needs-review" tags.

d

Re: [PATCH v7.1 11/11] NEWS: news for reply enhancements

[David Bremner]

On Tue, 13 Mar 2012 22:30:16 -0600, Adam Wolfe Gordon <awg+notmuch@xvx.ca> wrote:
> ---
>  NEWS |   20 ++++++++++++++++++++
>  1 files changed, 20 insertions(+), 0 deletions(-)

Thanks a lot for providing a NEWS item, but I just realized this should
be in the 0.13 section of NEWS. Which I created about 1 minute ago ;).

d

[PATCH v8 00/11] Reply enhancements (rebased)

[Adam Wolfe Gordon]

Hi everyone,

This is a resend of [1], rebased onto the current master. I've also moved the
NEWS items into the new 0.13 section, as David requested.

[1] id:"1331699416-30775-1-git-send-email-awg+notmuch@xvx.ca"

Adam Wolfe Gordon (11):
  test: Add broken test for the new JSON reply format.
  reply: Factor out reply creation
  TODO: Add replying to multiple messages
  reply: Add a JSON reply format.
  schemata: Add documentation for JSON reply format.
  man: Update notmuch-reply man page for JSON format.
  man: Add --decrypt to reply flags
  emacs: Factor out useful functions into notmuch-lib
  test: Add broken tests for new emacs reply functionality
  emacs: Use the new JSON reply format and message-cite-original
  NEWS: news for reply enhancements

 NEWS                     |   33 ++++++++++
 devel/TODO               |    8 +++
 devel/schemata           |   27 ++++++++-
 emacs/notmuch-lib.el     |   63 +++++++++++++++++++
 emacs/notmuch-mua.el     |  124 +++++++++++++++++++++++++------------
 emacs/notmuch-show.el    |   55 +++--------------
 man/man1/notmuch-reply.1 |   20 ++++++-
 notmuch-client.h         |   12 +++-
 notmuch-reply.c          |  153 +++++++++++++++++++++++++++++++++-------------
 notmuch-show.c           |   29 ++++++---
 test/emacs               |  101 ++++++++++++++++++++++++++++++-
 test/multipart           |   52 ++++++++++++++++
 12 files changed, 530 insertions(+), 147 deletions(-)

-- 
1.7.5.4

[PATCH v8 01/11] test: Add broken test for the new JSON reply format.

[Adam Wolfe Gordon]

---
 test/multipart |   53 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 files changed, 53 insertions(+), 0 deletions(-)

diff --git a/test/multipart b/test/multipart
index afc4fc8..e5de5d3 100755
--- a/test/multipart
+++ b/test/multipart
@@ -612,6 +612,59 @@ Non-text part: text/html
 EOF
 test_expect_equal_file OUTPUT EXPECTED
 
+test_begin_subtest "'notmuch reply' to a multipart message with json format"
+test_subtest_known_broken
+notmuch reply --format=json 'id:87liy5ap00.fsf@yoom.home.cworth.org' | notmuch_json_show_sanitize >OUTPUT
+cat <<EOF >EXPECTED
+{"reply-headers": {"Subject": "Re: Multipart message",
+ "From": "Notmuch Test Suite <test_suite@notmuchmail.org>",
+ "To": "Carl Worth <cworth@cworth.org>,
+ cworth@cworth.org",
+ "In-reply-to": "<87liy5ap00.fsf@yoom.home.cworth.org>",
+ "References": " <87liy5ap00.fsf@yoom.home.cworth.org>"},
+ "original": {"id": "XXXXX",
+ "match": false,
+ "excluded": false,
+ "filename": "YYYYY",
+ "timestamp": 978709437,
+ "date_relative": "2001-01-05",
+ "tags": ["attachment","inbox","signed","unread"],
+ "headers": {"Subject": "Multipart message",
+ "From": "Carl Worth <cworth@cworth.org>",
+ "To": "cworth@cworth.org",
+ "Date": "Fri,
+ 05 Jan 2001 15:43:57 +0000"},
+ "body": [{"id": 1,
+ "content-type": "multipart/signed",
+ "content": [{"id": 2,
+ "content-type": "multipart/mixed",
+ "content": [{"id": 3,
+ "content-type": "message/rfc822",
+ "content": [{"headers": {"Subject": "html message",
+ "From": "Carl Worth <cworth@cworth.org>",
+ "To": "cworth@cworth.org",
+ "Date": "Fri,
+ 05 Jan 2001 15:42:57 +0000"},
+ "body": [{"id": 4,
+ "content-type": "multipart/alternative",
+ "content": [{"id": 5,
+ "content-type": "text/html"},
+ {"id": 6,
+ "content-type": "text/plain",
+ "content": "This is an embedded message,
+ with a multipart/alternative part.\n"}]}]}]},
+ {"id": 7,
+ "content-type": "text/plain",
+ "filename": "YYYYY",
+ "content": "This is a text attachment.\n"},
+ {"id": 8,
+ "content-type": "text/plain",
+ "content": "And this message is signed.\n\n-Carl\n"}]},
+ {"id": 9,
+ "content-type": "application/pgp-signature"}]}]}}
+EOF
+test_expect_equal_file OUTPUT EXPECTED
+
 test_begin_subtest "'notmuch show --part' does not corrupt a part with CRLF pair"
 notmuch show --format=raw --part=3 id:base64-part-with-crlf > crlf.out
 echo -n -e "\xEF\x0D\x0A" > crlf.expected
-- 
1.7.5.4

[PATCH v8 02/11] reply: Factor out reply creation

[Adam Wolfe Gordon]

Factor out the creation of a reply message based on an original
message so it can be shared by different reply formats.
---
 notmuch-reply.c |  104 ++++++++++++++++++++++++++++++++----------------------
 1 files changed, 62 insertions(+), 42 deletions(-)

diff --git a/notmuch-reply.c b/notmuch-reply.c
index 6b244e6..f1478cc 100644
--- a/notmuch-reply.c
+++ b/notmuch-reply.c
@@ -505,6 +505,61 @@ guess_from_received_header (notmuch_config_t *config, notmuch_message_t *message
     return NULL;
 }
 
+static GMimeMessage *
+create_reply_message(void *ctx,
+		     notmuch_config_t *config,
+		     notmuch_message_t *message,
+		     notmuch_bool_t reply_all)
+{
+    const char *subject, *from_addr = NULL;
+    const char *in_reply_to, *orig_references, *references;
+
+    /* The 1 means we want headers in a "pretty" order. */
+    GMimeMessage *reply = g_mime_message_new (1);
+    if (reply == NULL) {
+	fprintf (stderr, "Out of memory\n");
+	return NULL;
+    }
+
+    subject = notmuch_message_get_header (message, "subject");
+    if (subject) {
+	if (strncasecmp (subject, "Re:", 3))
+	    subject = talloc_asprintf (ctx, "Re: %s", subject);
+	g_mime_message_set_subject (reply, subject);
+    }
+
+    from_addr = add_recipients_from_message (reply, config,
+					     message, reply_all);
+
+    if (from_addr == NULL)
+	from_addr = guess_from_received_header (config, message);
+
+    if (from_addr == NULL)
+	from_addr = notmuch_config_get_user_primary_email (config);
+
+    from_addr = talloc_asprintf (ctx, "%s <%s>",
+				 notmuch_config_get_user_name (config),
+				 from_addr);
+    g_mime_object_set_header (GMIME_OBJECT (reply),
+			      "From", from_addr);
+
+    in_reply_to = talloc_asprintf (ctx, "<%s>",
+				   notmuch_message_get_message_id (message));
+
+    g_mime_object_set_header (GMIME_OBJECT (reply),
+			      "In-Reply-To", in_reply_to);
+
+    orig_references = notmuch_message_get_header (message, "references");
+    references = talloc_asprintf (ctx, "%s%s%s",
+				  orig_references ? orig_references : "",
+				  orig_references ? " " : "",
+				  in_reply_to);
+    g_mime_object_set_header (GMIME_OBJECT (reply),
+			      "References", references);
+
+    return reply;
+}
+
 static int
 notmuch_reply_format_default(void *ctx,
 			     notmuch_config_t *config,
@@ -515,8 +570,6 @@ notmuch_reply_format_default(void *ctx,
     GMimeMessage *reply;
     notmuch_messages_t *messages;
     notmuch_message_t *message;
-    const char *subject, *from_addr = NULL;
-    const char *in_reply_to, *orig_references, *references;
     const notmuch_show_format_t *format = &format_reply;
 
     for (messages = notmuch_query_search_messages (query);
@@ -525,49 +578,16 @@ notmuch_reply_format_default(void *ctx,
     {
 	message = notmuch_messages_get (messages);
 
-	/* The 1 means we want headers in a "pretty" order. */
-	reply = g_mime_message_new (1);
-	if (reply == NULL) {
-	    fprintf (stderr, "Out of memory\n");
-	    return 1;
-	}
+	reply = create_reply_message (ctx, config, message, reply_all);
 
-	subject = notmuch_message_get_header (message, "subject");
-	if (subject) {
-	    if (strncasecmp (subject, "Re:", 3))
-		subject = talloc_asprintf (ctx, "Re: %s", subject);
-	    g_mime_message_set_subject (reply, subject);
+	/* If reply creation failed, we're out of memory, so don't
+	 * bother trying any more messages.
+	 */
+	if (!reply) {
+	    notmuch_message_destroy (message);
+	    return 1;
 	}
 
-	from_addr = add_recipients_from_message (reply, config, message,
-						 reply_all);
-
-	if (from_addr == NULL)
-	    from_addr = guess_from_received_header (config, message);
-
-	if (from_addr == NULL)
-	    from_addr = notmuch_config_get_user_primary_email (config);
-
-	from_addr = talloc_asprintf (ctx, "%s <%s>",
-				     notmuch_config_get_user_name (config),
-				     from_addr);
-	g_mime_object_set_header (GMIME_OBJECT (reply),
-				  "From", from_addr);
-
-	in_reply_to = talloc_asprintf (ctx, "<%s>",
-			     notmuch_message_get_message_id (message));
-
-	g_mime_object_set_header (GMIME_OBJECT (reply),
-				  "In-Reply-To", in_reply_to);
-
-	orig_references = notmuch_message_get_header (message, "references");
-	references = talloc_asprintf (ctx, "%s%s%s",
-				      orig_references ? orig_references : "",
-				      orig_references ? " " : "",
-				      in_reply_to);
-	g_mime_object_set_header (GMIME_OBJECT (reply),
-				  "References", references);
-
 	show_reply_headers (reply);
 
 	g_object_unref (G_OBJECT (reply));
-- 
1.7.5.4

[PATCH v8 03/11] TODO: Add replying to multiple messages

[Adam Wolfe Gordon]

---
 devel/TODO |    8 ++++++++
 1 files changed, 8 insertions(+), 0 deletions(-)

diff --git a/devel/TODO b/devel/TODO
index 4dda6f4..7b750af 100644
--- a/devel/TODO
+++ b/devel/TODO
@@ -141,6 +141,14 @@ Simplify notmuch-reply to simply print the headers (we have the
 original values) rather than calling GMime (which encodes) and adding
 the confusing gmime-filter-headers.c code (which decodes).
 
+Properly handle replying to multiple messages. Currently, the JSON
+reply format only supports a single message, but the default reply
+format accepts searches returning multiple messages. The expected
+behavior of replying to multiple messages is not obvious, and there
+are multiple ideas that might make sense. Some consensus needs to be
+reached on this issue, and then both reply formats should be updated
+to be consistent.
+
 notmuch library
 ---------------
 Add support for custom flag<->tag mappings. In the notmuch
-- 
1.7.5.4

[PATCH v8 04/11] reply: Add a JSON reply format.

[Adam Wolfe Gordon]

This new JSON format for replies includes headers generated for a
reply message as well as the headers of the original message.  Using
this data, a client can intelligently create a reply. For example, the
emacs client will be able to create replies with quoted HTML parts by
parsing the HTML parts.
---
 notmuch-client.h |   12 +++++++++---
 notmuch-reply.c  |   49 +++++++++++++++++++++++++++++++++++++++++++++++++
 notmuch-show.c   |   29 +++++++++++++++++++++--------
 test/multipart   |    1 -
 4 files changed, 79 insertions(+), 12 deletions(-)

diff --git a/notmuch-client.h b/notmuch-client.h
index a220fe4..fa04fa2 100644
--- a/notmuch-client.h
+++ b/notmuch-client.h
@@ -62,7 +62,7 @@
 #define STRINGIFY(s) STRINGIFY_(s)
 #define STRINGIFY_(s) #s
 
-struct mime_node;
+typedef struct mime_node mime_node_t;
 struct notmuch_show_params;
 
 typedef struct notmuch_show_format {
@@ -191,6 +191,12 @@ show_message_body (notmuch_message_t *message,
 notmuch_status_t
 show_one_part (const char *filename, int part);
 
+void
+format_part_json (const void *ctx, mime_node_t *node, notmuch_bool_t first);
+
+void
+format_headers_json (const void *ctx, GMimeMessage *message, notmuch_bool_t reply);
+
 char *
 json_quote_chararray (const void *ctx, const char *str, const size_t len);
 
@@ -288,7 +294,7 @@ debugger_is_active (void);
  * parts.  Message-type parts have one child, multipart-type parts
  * have multiple children, and leaf parts have zero children.
  */
-typedef struct mime_node {
+struct mime_node {
     /* The MIME object of this part.  This will be a GMimeMessage,
      * GMimePart, GMimeMultipart, or a subclass of one of these.
      *
@@ -351,7 +357,7 @@ typedef struct mime_node {
      * number to assign it (or -1 if unknown). */
     int next_child;
     int next_part_num;
-} mime_node_t;
+};
 
 /* Construct a new MIME node pointing to the root message part of
  * message.  If cryptoctx is non-NULL, it will be used to verify
diff --git a/notmuch-reply.c b/notmuch-reply.c
index f1478cc..e2b6c25 100644
--- a/notmuch-reply.c
+++ b/notmuch-reply.c
@@ -604,6 +604,51 @@ notmuch_reply_format_default(void *ctx,
     return 0;
 }
 
+static int
+notmuch_reply_format_json(void *ctx,
+			  notmuch_config_t *config,
+			  notmuch_query_t *query,
+			  notmuch_show_params_t *params,
+			  notmuch_bool_t reply_all)
+{
+    GMimeMessage *reply;
+    notmuch_messages_t *messages;
+    notmuch_message_t *message;
+    mime_node_t *node;
+
+    if (notmuch_query_count_messages (query) != 1) {
+	fprintf (stderr, "Error: search term did not match precisely one message.\n");
+	return 1;
+    }
+
+    messages = notmuch_query_search_messages (query);
+    message = notmuch_messages_get (messages);
+    if (mime_node_open (ctx, message, params->cryptoctx, params->decrypt,
+			&node) != NOTMUCH_STATUS_SUCCESS)
+	return 1;
+
+    reply = create_reply_message (ctx, config, message, reply_all);
+    if (!reply)
+	return 1;
+
+    /* The headers of the reply message we've created */
+    printf ("{\"reply-headers\": ");
+    format_headers_json (ctx, reply, TRUE);
+    g_object_unref (G_OBJECT (reply));
+    reply = NULL;
+
+    /* Start the original */
+    printf (", \"original\": ");
+
+    format_part_json (ctx, node, TRUE);
+
+    /* End */
+    printf ("}\n");
+    notmuch_message_destroy (message);
+
+    return 0;
+}
+
 /* This format is currently tuned for a git send-email --notmuch hook */
 static int
 notmuch_reply_format_headers_only(void *ctx,
@@ -666,6 +711,7 @@ notmuch_reply_format_headers_only(void *ctx,
 
 enum {
     FORMAT_DEFAULT,
+    FORMAT_JSON,
     FORMAT_HEADERS_ONLY,
 };
 
@@ -685,6 +731,7 @@ notmuch_reply_command (void *ctx, int argc, char *argv[])
     notmuch_opt_desc_t options[] = {
 	{ NOTMUCH_OPT_KEYWORD, &format, "format", 'f',
 	  (notmuch_keyword_t []){ { "default", FORMAT_DEFAULT },
+				  { "json", FORMAT_JSON },
 				  { "headers-only", FORMAT_HEADERS_ONLY },
 				  { 0, 0 } } },
 	{ NOTMUCH_OPT_KEYWORD, &reply_all, "reply-to", 'r',
@@ -703,6 +750,8 @@ notmuch_reply_command (void *ctx, int argc, char *argv[])
 
     if (format == FORMAT_HEADERS_ONLY)
 	reply_format_func = notmuch_reply_format_headers_only;
+    else if (format == FORMAT_JSON)
+	reply_format_func = notmuch_reply_format_json;
     else
 	reply_format_func = notmuch_reply_format_default;
 
diff --git a/notmuch-show.c b/notmuch-show.c
index a7463dc..ff9d427 100644
--- a/notmuch-show.c
+++ b/notmuch-show.c
@@ -200,8 +200,8 @@ _is_from_line (const char *line)
 	return 0;
 }
 
-static void
-format_headers_json (const void *ctx, GMimeMessage *message)
+void
+format_headers_json (const void *ctx, GMimeMessage *message, notmuch_bool_t reply)
 {
     void *local = talloc_new (ctx);
     InternetAddressList *recipients;
@@ -225,9 +225,22 @@ format_headers_json (const void *ctx, GMimeMessage *message)
 	printf (", %s: %s",
 		json_quote_str (local, "Cc"),
 		json_quote_str (local, recipients_string));
-    printf (", %s: %s}",
-	    json_quote_str (local, "Date"),
-	    json_quote_str (local, g_mime_message_get_date_as_string (message)));
+
+    if (reply) {
+	printf (", %s: %s",
+		json_quote_str (local, "In-reply-to"),
+		json_quote_str (local, g_mime_object_get_header (GMIME_OBJECT (message), "In-reply-to")));
+
+	printf (", %s: %s",
+		json_quote_str (local, "References"),
+		json_quote_str (local, g_mime_object_get_header (GMIME_OBJECT (message), "References")));
+    } else {
+	printf (", %s: %s",
+		json_quote_str (local, "Date"),
+		json_quote_str (local, g_mime_message_get_date_as_string (message)));
+    }
+
+    printf ("}");
 
     talloc_free (local);
 }
@@ -538,7 +551,7 @@ format_part_text (const void *ctx, mime_node_t *node,
     return NOTMUCH_STATUS_SUCCESS;
 }
 
-static void
+void
 format_part_json (const void *ctx, mime_node_t *node, notmuch_bool_t first)
 {
     /* Any changes to the JSON format should be reflected in the file
@@ -549,7 +562,7 @@ format_part_json (const void *ctx, mime_node_t *node, notmuch_bool_t first)
 	format_message_json (ctx, node->envelope_file);
 
 	printf ("\"headers\": ");
-	format_headers_json (ctx, GMIME_MESSAGE (node->part));
+	format_headers_json (ctx, GMIME_MESSAGE (node->part), FALSE);
 
 	printf (", \"body\": [");
 	format_part_json (ctx, mime_node_child (node, 0), first);
@@ -623,7 +636,7 @@ format_part_json (const void *ctx, mime_node_t *node, notmuch_bool_t first)
     } else if (GMIME_IS_MESSAGE (node->part)) {
 	printf (", \"content\": [{");
 	printf ("\"headers\": ");
-	format_headers_json (local, GMIME_MESSAGE (node->part));
+	format_headers_json (local, GMIME_MESSAGE (node->part), FALSE);
 
 	printf (", \"body\": [");
 	terminator = "]}]";
diff --git a/test/multipart b/test/multipart
index e5de5d3..72d3927 100755
--- a/test/multipart
+++ b/test/multipart
@@ -613,7 +613,6 @@ EOF
 test_expect_equal_file OUTPUT EXPECTED
 
 test_begin_subtest "'notmuch reply' to a multipart message with json format"
-test_subtest_known_broken
 notmuch reply --format=json 'id:87liy5ap00.fsf@yoom.home.cworth.org' | notmuch_json_show_sanitize >OUTPUT
 cat <<EOF >EXPECTED
 {"reply-headers": {"Subject": "Re: Multipart message",
-- 
1.7.5.4

[PATCH v8 05/11] schemata: Add documentation for JSON reply format.

[Adam Wolfe Gordon]

---
 devel/schemata |   27 +++++++++++++++++++++++++--
 1 files changed, 25 insertions(+), 2 deletions(-)

diff --git a/devel/schemata b/devel/schemata
index 24ad775..728a46f 100644
--- a/devel/schemata
+++ b/devel/schemata
@@ -77,8 +77,9 @@ part = {
     content?:       string
 }
 
-# The headers of a message (format_headers_json with raw headers) or
-# a part (format_headers_message_part_json with pretty-printed headers)
+# The headers of a message (format_headers_json with raw headers
+# and reply = FALSE) or a part (format_headers_message_part_json
+# with pretty-printed headers)
 headers = {
     Subject:        string,
     From:           string,
@@ -136,3 +137,25 @@ thread = {
                               # matched and unmatched
     subject:        string
 }
+
+notmuch reply schema
+--------------------
+
+reply = {
+    # The headers of the constructed reply (format_headers_json with
+    # raw headers and reply = TRUE)
+    reply-headers: reply_headers,
+
+    # As in the show format (format_part_json)
+    original: message
+}
+
+reply_headers = {
+    Subject:        string,
+    From:           string,
+    To?:            string,
+    Cc?:            string,
+    Bcc?:           string,
+    In-reply-to:    string,
+    References:     string
+}
-- 
1.7.5.4

[PATCH v8 06/11] man: Update notmuch-reply man page for JSON format.

[Adam Wolfe Gordon]

---
 man/man1/notmuch-reply.1 |   10 ++++++++--
 1 files changed, 8 insertions(+), 2 deletions(-)

diff --git a/man/man1/notmuch-reply.1 b/man/man1/notmuch-reply.1
index deb3ae1..ec8da1f 100644
--- a/man/man1/notmuch-reply.1
+++ b/man/man1/notmuch-reply.1
@@ -37,12 +37,17 @@ Supported options for
 include
 .RS
 .TP 4
-.BR \-\-format= ( default | headers\-only )
+.BR \-\-format= ( default | json | headers\-only )
 .RS
 .TP 4
 .BR default
 Includes subject and quoted message body.
 .TP
+.BR json
+Produces JSON output containing headers for a reply message and the
+contents of the original message. This output can be used by a client
+to create a reply message intelligently.
+.TP
 .BR headers\-only
 Only produces In\-Reply\-To, References, To, Cc, and Bcc headers.
 .RE
@@ -73,7 +78,8 @@ with a search string matching a single message, (such as
 id:<message-id>), but it can be useful to reply to several messages at
 once. For example, when a series of patches are sent in a single
 thread, replying to the entire thread allows for the reply to comment
-on issue found in multiple patches.
+on issues found in multiple patches. The default format supports
+replying to multiple messages at once, but the JSON format does not.
 .RE
 .RE
 
-- 
1.7.5.4

[PATCH v8 07/11] man: Add --decrypt to reply flags

[Adam Wolfe Gordon]

---
 man/man1/notmuch-reply.1 |   10 ++++++++++
 1 files changed, 10 insertions(+), 0 deletions(-)

diff --git a/man/man1/notmuch-reply.1 b/man/man1/notmuch-reply.1
index ec8da1f..7ee3ceb 100644
--- a/man/man1/notmuch-reply.1
+++ b/man/man1/notmuch-reply.1
@@ -68,6 +68,16 @@ values from the first that contains something other than only the
 user's addresses.
 .RE
 .RE
+.RS
+.TP 4
+.B \-\-decrypt
+
+Decrypt any MIME encrypted parts found in the selected content
+(ie. "multipart/encrypted" parts). Status of the decryption will be
+reported (currently only supported with --format=json) and the
+multipart/encrypted part will be replaced by the decrypted
+content.
+.RE
 
 See \fBnotmuch-search-terms\fR(7)
 for details of the supported syntax for <search-terms>.
-- 
1.7.5.4

[PATCH v8 08/11] emacs: Factor out useful functions into notmuch-lib

[Adam Wolfe Gordon]

Move a few functions related to handling multipart/alternative parts
into notmuch-lib.el, so they can be used by future reply code.
---
 emacs/notmuch-lib.el  |   33 +++++++++++++++++++++++++++++++++
 emacs/notmuch-show.el |   24 ++----------------------
 2 files changed, 35 insertions(+), 22 deletions(-)

diff --git a/emacs/notmuch-lib.el b/emacs/notmuch-lib.el
index d315f76..7e3f110 100644
--- a/emacs/notmuch-lib.el
+++ b/emacs/notmuch-lib.el
@@ -21,6 +21,8 @@
 
 ;; This is an part of an emacs-based interface to the notmuch mail system.
 
+(eval-when-compile (require 'cl))
+
 (defvar notmuch-command "notmuch"
   "Command to run the notmuch binary.")
 
@@ -173,6 +175,37 @@ the user hasn't set this variable with the old or new value."
   (list 'when (< emacs-major-version 23)
 	form))
 
+(defun notmuch-split-content-type (content-type)
+  "Split content/type into 'content' and 'type'"
+  (split-string content-type "/"))
+
+(defun notmuch-match-content-type (t1 t2)
+  "Return t if t1 and t2 are matching content types, taking wildcards into account"
+  (let ((st1 (notmuch-split-content-type t1))
+	(st2 (notmuch-split-content-type t2)))
+    (if (or (string= (cadr st1) "*")
+	    (string= (cadr st2) "*"))
+	(string= (car st1) (car st2))
+      (string= t1 t2))))
+
+(defvar notmuch-multipart/alternative-discouraged
+  '(
+    ;; Avoid HTML parts.
+    "text/html"
+    ;; multipart/related usually contain a text/html part and some associated graphics.
+    "multipart/related"
+    ))
+
+(defun notmuch-multipart/alternative-choose (types)
+  "Return a list of preferred types from the given list of types"
+  ;; Based on `mm-preferred-alternative-precedence'.
+  (let ((seq types))
+    (dolist (pref (reverse notmuch-multipart/alternative-discouraged))
+      (dolist (elem (copy-sequence seq))
+	(when (string-match pref elem)
+	  (setq seq (nconc (delete elem seq) (list elem))))))
+    seq))
+
 ;; Compatibility functions for versions of emacs before emacs 23.
 ;;
 ;; Both functions here were copied from emacs 23 with the following copyright:
diff --git a/emacs/notmuch-show.el b/emacs/notmuch-show.el
index 4a60631..ed938bf 100644
--- a/emacs/notmuch-show.el
+++ b/emacs/notmuch-show.el
@@ -542,30 +542,13 @@ current buffer, if possible."
 	    (mm-display-part handle)
 	    t))))))
 
-(defvar notmuch-show-multipart/alternative-discouraged
-  '(
-    ;; Avoid HTML parts.
-    "text/html"
-    ;; multipart/related usually contain a text/html part and some associated graphics.
-    "multipart/related"
-    ))
-
 (defun notmuch-show-multipart/*-to-list (part)
   (mapcar (lambda (inner-part) (plist-get inner-part :content-type))
 	  (plist-get part :content)))
 
-(defun notmuch-show-multipart/alternative-choose (types)
-  ;; Based on `mm-preferred-alternative-precedence'.
-  (let ((seq types))
-    (dolist (pref (reverse notmuch-show-multipart/alternative-discouraged))
-      (dolist (elem (copy-sequence seq))
-	(when (string-match pref elem)
-	  (setq seq (nconc (delete elem seq) (list elem))))))
-    seq))
-
 (defun notmuch-show-insert-part-multipart/alternative (msg part content-type nth depth declared-type)
   (notmuch-show-insert-part-header nth declared-type content-type nil)
-  (let ((chosen-type (car (notmuch-show-multipart/alternative-choose (notmuch-show-multipart/*-to-list part))))
+  (let ((chosen-type (car (notmuch-multipart/alternative-choose (notmuch-show-multipart/*-to-list part))))
 	(inner-parts (plist-get part :content))
 	(start (point)))
     ;; This inserts all parts of the chosen type rather than just one,
@@ -808,9 +791,6 @@ current buffer, if possible."
 
 ;; Functions for determining how to handle MIME parts.
 
-(defun notmuch-show-split-content-type (content-type)
-  (split-string content-type "/"))
-
 (defun notmuch-show-handlers-for (content-type)
   "Return a list of content handlers for a part of type CONTENT-TYPE."
   (let (result)
@@ -821,7 +801,7 @@ current buffer, if possible."
 	  (list (intern (concat "notmuch-show-insert-part-*/*"))
 		(intern (concat
 			 "notmuch-show-insert-part-"
-			 (car (notmuch-show-split-content-type content-type))
+			 (car (notmuch-split-content-type content-type))
 			 "/*"))
 		(intern (concat "notmuch-show-insert-part-" content-type))))
     result))
-- 
1.7.5.4

[PATCH v8 09/11] test: Add broken tests for new emacs reply functionality

[Adam Wolfe Gordon]

Add tests for creating nice replies to multipart messages, including
those with HTML parts. These tests are expected to fail for now.
---
 test/emacs |   97 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 files changed, 97 insertions(+), 0 deletions(-)

diff --git a/test/emacs b/test/emacs
index 29a489c..01afdb6 100755
--- a/test/emacs
+++ b/test/emacs
@@ -273,6 +273,103 @@ On 01 Jan 2000 12:00:00 -0000, Notmuch Test Suite <test_suite@notmuchmail.org> w
 EOF
 test_expect_equal_file OUTPUT EXPECTED
 
+test_begin_subtest "Reply within emacs to a multipart/mixed message"
+test_subtest_known_broken
+test_emacs '(notmuch-show "id:20091118002059.067214ed@hikari")
+		(notmuch-show-reply)
+		(test-output)'
+cat <<EOF >EXPECTED
+From: Notmuch Test Suite <test_suite@notmuchmail.org>
+To: Adrian Perez de Castro <aperez@igalia.com>, notmuch@notmuchmail.org
+Subject: Re: [notmuch] Introducing myself
+In-Reply-To: <20091118002059.067214ed@hikari>
+Fcc: ${MAIL_DIR}/sent
+--text follows this line--
+Adrian Perez de Castro <aperez@igalia.com> writes:
+
+> Hello to all,
+>
+> I have just heard about Not Much today in some random Linux-related news
+> site (LWN?), my name is Adrian Perez and I work as systems administrator
+> (although I can do some code as well :P). I have always thought that the
+> ideas behind Sup were great, but after some time using it, I got tired of
+> the oddities that it has. I also do not like doing things like having to
+> install Ruby just for reading and sorting mails. Some time ago I thought
+> about doing something like Not Much and in fact I played a bit with the
+> Python+Xapian and the Python+Whoosh combinations, because I find relaxing
+> to code things in Python when I am not working and also it is installed
+> by default on most distribution. I got to have some mailboxes indexed and
+> basic searching working a couple of months ago. Lately I have been very
+> busy and had no time for coding, and them... boom! Not Much appears -- and
+> it is almost exactly what I was trying to do, but faster. I have been
+> playing a bit with Not Much today, and I think it has potential.
+>
+> Also, I would like to share one idea I had in mind, that you might find
+> interesting: One thing I have found very annoying is having to re-tag my
+> mail when the indexes get b0rked (it happened a couple of times to me while
+> using Sup), so I was planning to mails as read/unread and adding the tags
+> not just to the index, but to the mail text itself, e.g. by adding a
+> "X-Tags" header field or by reusing the "Keywords" one. This way, the index
+> could be totally recreated by re-reading the mail directories, and this
+> would also allow to a tools like OfflineIMAP [1] to get the mails into a
+> local maildir, tagging and indexing the mails with the e-mail reader and
+> then syncing back the messages with the "X-Tags" header to the IMAP server.
+> This would allow to use the mail reader from a different computer and still
+> have everything tagged finely.
+>
+> Best regards,
+>
+>
+> ---
+> [1] http://software.complete.org/software/projects/show/offlineimap
+>
+> -- 
+> Adrian Perez de Castro <aperez@igalia.com>
+> Igalia - Free Software Engineering
+> _______________________________________________
+> notmuch mailing list
+> notmuch@notmuchmail.org
+> http://notmuchmail.org/mailman/listinfo/notmuch
+EOF
+test_expect_equal_file OUTPUT EXPECTED
+
+test_begin_subtest "Reply within emacs to a multipart/alternative message"
+test_subtest_known_broken
+test_emacs '(notmuch-show "id:cf0c4d610911171136h1713aa59w9cf9aa31f052ad0a@mail.gmail.com")
+		(notmuch-show-reply)
+		(test-output)'
+cat <<EOF >EXPECTED
+From: Notmuch Test Suite <test_suite@notmuchmail.org>
+To: Alex Botero-Lowry <alex.boterolowry@gmail.com>, notmuch@notmuchmail.org
+Subject: Re: [notmuch] preliminary FreeBSD support
+In-Reply-To: <cf0c4d610911171136h1713aa59w9cf9aa31f052ad0a@mail.gmail.com>
+Fcc: ${MAIL_DIR}/sent
+--text follows this line--
+Alex Botero-Lowry <alex.boterolowry@gmail.com> writes:
+
+> I saw the announcement this morning, and was very excited, as I had been
+> hoping sup would be turned into a library,
+> since I like the concept more than the UI (I'd rather an emacs interface).
+>
+> I did a preliminary compile which worked out fine, but
+> sysconf(_SC_SC_GETPW_R_SIZE_MAX) returns -1 on
+> FreeBSD, so notmuch_config_open segfaulted.
+>
+> Attached is a patch that supplies a default buffer size of 64 in cases where
+> -1 is returned.
+>
+> http://www.opengroup.org/austin/docs/austin_328.txt - seems to indicate this
+> is acceptable behavior,
+> and http://mail-index.netbsd.org/pkgsrc-bugs/2006/06/07/msg016808.htmlspecifically
+> uses 64 as the
+> buffer size.
+> _______________________________________________
+> notmuch mailing list
+> notmuch@notmuchmail.org
+> http://notmuchmail.org/mailman/listinfo/notmuch
+EOF
+test_expect_equal_file OUTPUT EXPECTED
+
 test_begin_subtest "Quote MML tags in reply"
 message_id='test-emacs-mml-quoting@message.id'
 add_message [id]="$message_id" \
-- 
1.7.5.4

[PATCH v8 10/11] emacs: Use the new JSON reply format and message-cite-original

[Adam Wolfe Gordon]

Use the new JSON reply format to create replies in emacs. Quote HTML
parts nicely by using mm-display-part to turn them into displayable
text, then quoting them with message-cite-original. This is very
useful for users who regularly receive HTML-only email.

Use message-mode's message-cite-original function to create the
quoted body for reply messages. In order to make this act like the
existing notmuch defaults, you will need to set the following in
your emacs configuration:

message-citation-line-format "On %a, %d %b %Y, %f wrote:"
message-citation-line-function 'message-insert-formatted-citation-line

The tests have been updated to reflect the (ugly) emacs default.
---
 emacs/notmuch-lib.el  |   30 ++++++++++++
 emacs/notmuch-mua.el  |  124 +++++++++++++++++++++++++++++++++----------------
 emacs/notmuch-show.el |   31 ++----------
 test/emacs            |    8 ++--
 4 files changed, 123 insertions(+), 70 deletions(-)

diff --git a/emacs/notmuch-lib.el b/emacs/notmuch-lib.el
index 7e3f110..c146748 100644
--- a/emacs/notmuch-lib.el
+++ b/emacs/notmuch-lib.el
@@ -206,6 +206,36 @@ the user hasn't set this variable with the old or new value."
 	  (setq seq (nconc (delete elem seq) (list elem))))))
     seq))
 
+(defun notmuch-parts-filter-by-type (parts type)
+  "Given a list of message parts, return a list containing the ones matching
+the given type."
+  (remove-if-not
+   (lambda (part) (notmuch-match-content-type (plist-get part :content-type) type))
+   parts))
+
+;; Helper for parts which are generally not included in the default
+;; JSON output.
+(defun notmuch-get-bodypart-internal (message-id part-number process-crypto)
+  (let ((args '("show" "--format=raw"))
+	(part-arg (format "--part=%s" part-number)))
+    (setq args (append args (list part-arg)))
+    (if process-crypto
+	(setq args (append args '("--decrypt"))))
+    (setq args (append args (list message-id)))
+    (with-temp-buffer
+      (let ((coding-system-for-read 'no-conversion))
+	(progn
+	  (apply 'call-process (append (list notmuch-command nil (list t nil) nil) args))
+	  (buffer-string))))))
+
+(defun notmuch-get-bodypart-content (msg part nth process-crypto)
+  (or (plist-get part :content)
+      (notmuch-get-bodypart-internal (concat "id:" (plist-get msg :id)) nth process-crypto)))
+
+(defun notmuch-plist-to-alist (plist)
+  (loop for (key value . rest) on plist by #'cddr
+	collect (cons (substring (symbol-name key) 1) value)))
+
 ;; Compatibility functions for versions of emacs before emacs 23.
 ;;
 ;; Both functions here were copied from emacs 23 with the following copyright:
diff --git a/emacs/notmuch-mua.el b/emacs/notmuch-mua.el
index 13244eb..6aae3a0 100644
--- a/emacs/notmuch-mua.el
+++ b/emacs/notmuch-mua.el
@@ -19,11 +19,15 @@
 ;;
 ;; Authors: David Edmondson <dme@dme.org>
 
+(require 'json)
 (require 'message)
+(require 'format-spec)
 
 (require 'notmuch-lib)
 (require 'notmuch-address)
 
+(eval-when-compile (require 'cl))
+
 ;;
 
 (defcustom notmuch-mua-send-hook '(notmuch-mua-message-send-hook)
@@ -72,54 +76,92 @@ list."
 	    (push header message-hidden-headers)))
 	notmuch-mua-hidden-headers))
 
+(defun notmuch-mua-get-quotable-parts (parts)
+  (loop for part in parts
+	if (notmuch-match-content-type (plist-get part :content-type) "multipart/alternative")
+	  collect (let* ((subparts (plist-get part :content))
+			(types (mapcar (lambda (part) (plist-get part :content-type)) subparts))
+			(chosen-type (car (notmuch-multipart/alternative-choose types))))
+		   (loop for part in (reverse subparts)
+			 if (notmuch-match-content-type (plist-get part :content-type) chosen-type)
+			 return part))
+	else if (notmuch-match-content-type (plist-get part :content-type) "multipart/*")
+	  append (notmuch-mua-get-quotable-parts (plist-get part :content))
+	else if (notmuch-match-content-type (plist-get part :content-type) "text/*")
+	  collect part))
+
 (defun notmuch-mua-reply (query-string &optional sender reply-all)
-  (let (headers
-	body
-	(args '("reply")))
-    (if notmuch-show-process-crypto
-	(setq args (append args '("--decrypt"))))
+  (let ((args '("reply" "--format=json"))
+	reply
+	original)
+    (when notmuch-show-process-crypto
+      (setq args (append args '("--decrypt"))))
+
     (if reply-all
 	(setq args (append args '("--reply-to=all")))
       (setq args (append args '("--reply-to=sender"))))
     (setq args (append args (list query-string)))
-    ;; This make assumptions about the output of `notmuch reply', but
-    ;; really only that the headers come first followed by a blank
-    ;; line and then the body.
+
+    ;; Get the reply object as JSON, and parse it into an elisp object.
     (with-temp-buffer
       (apply 'call-process (append (list notmuch-command nil (list t t) nil) args))
       (goto-char (point-min))
-      (if (re-search-forward "^$" nil t)
-	  (save-excursion
-	    (save-restriction
-	      (narrow-to-region (point-min) (point))
-	      (goto-char (point-min))
-	      (setq headers (mail-header-extract)))))
-      (forward-line 1)
-      ;; Original message may contain (malicious) MML tags. We must
-      ;; properly quote them in the reply.
-      (mml-quote-region (point) (point-max))
-      (setq body (buffer-substring (point) (point-max))))
-    ;; If sender is non-nil, set the From: header to its value.
-    (when sender
-      (mail-header-set 'from sender headers))
-    (let
-	;; Overlay the composition window on that being used to read
-	;; the original message.
-	((same-window-regexps '("\\*mail .*")))
-      (notmuch-mua-mail (mail-header 'to headers)
-			(mail-header 'subject headers)
-			(message-headers-to-generate headers t '(to subject))))
-    ;; insert the message body - but put it in front of the signature
-    ;; if one is present
-    (goto-char (point-max))
-    (if (re-search-backward message-signature-separator nil t)
+      (let ((json-object-type 'plist)
+	    (json-array-type 'list)
+	    (json-false 'nil))
+	(setq reply (json-read))))
+
+    ;; Extract the original message to simplify the following code.
+    (setq original (plist-get reply :original))
+
+    ;; Extract the headers of both the reply and the original message.
+    (let* ((original-headers (plist-get original :headers))
+	   (reply-headers (plist-get reply :reply-headers)))
+
+      ;; If sender is non-nil, set the From: header to its value.
+      (when sender
+	(plist-put reply-headers :From sender))
+      (let
+	  ;; Overlay the composition window on that being used to read
+	  ;; the original message.
+	  ((same-window-regexps '("\\*mail .*")))
+	(notmuch-mua-mail (plist-get reply-headers :To)
+			  (plist-get reply-headers :Subject)
+			  (notmuch-plist-to-alist reply-headers)))
+      ;; Insert the message body - but put it in front of the signature
+      ;; if one is present
+      (goto-char (point-max))
+      (if (re-search-backward message-signature-separator nil t)
 	  (forward-line -1)
-      (goto-char (point-max)))
-    (insert body)
-    (push-mark))
-  (set-buffer-modified-p nil)
-
-  (message-goto-body))
+	(goto-char (point-max)))
+
+      (let ((from (plist-get original-headers :From))
+	    (date (plist-get original-headers :Date))
+	    (start (point)))
+
+	;; message-cite-original constructs a citation line based on the From and Date
+	;; headers of the original message, which are assumed to be in the buffer.
+	(insert "From: " from "\n")
+	(insert "Date: " date "\n\n")
+
+	;; Get the parts of the original message that should be quoted; this includes
+	;; all the text parts, except the non-preferred ones in a multipart/alternative.
+	(let ((quotable-parts (notmuch-mua-get-quotable-parts (plist-get original :body))))
+	  (mapc (lambda (part)
+		  (insert (notmuch-get-bodypart-content original part
+							(plist-get part :id)
+							notmuch-show-process-crypto)))
+		quotable-parts))
+
+	(set-mark (point))
+	(goto-char start)
+	;; Quote the original message according to the user's configured style.
+	(message-cite-original))))
+
+  (goto-char (point-max))
+  (push-mark)
+  (message-goto-body)
+  (set-buffer-modified-p nil))
 
 (defun notmuch-mua-forward-message ()
   (message-forward)
@@ -145,7 +187,7 @@ OTHER-ARGS are passed through to `message-mail'."
       (when (not (string= "" user-agent))
 	(push (cons "User-Agent" user-agent) other-headers))))
 
-  (unless (mail-header 'from other-headers)
+  (unless (mail-header 'From other-headers)
     (push (cons "From" (concat
 			(notmuch-user-name) " <" (notmuch-user-primary-email) ">")) other-headers))
 
@@ -208,7 +250,7 @@ the From: address first."
   (interactive "P")
   (let ((other-headers
 	 (when (or prompt-for-sender notmuch-always-prompt-for-sender)
-	   (list (cons 'from (notmuch-mua-prompt-for-sender))))))
+	   (list (cons 'From (notmuch-mua-prompt-for-sender))))))
     (notmuch-mua-mail nil nil other-headers)))
 
 (defun notmuch-mua-new-forward-message (&optional prompt-for-sender)
diff --git a/emacs/notmuch-show.el b/emacs/notmuch-show.el
index ed938bf..0cd7d82 100644
--- a/emacs/notmuch-show.el
+++ b/emacs/notmuch-show.el
@@ -488,7 +488,7 @@ message at DEPTH in the current thread."
 	 (setq notmuch-show-process-crypto ,process-crypto)
 	 ;; Always acquires the part via `notmuch part', even if it is
 	 ;; available in the JSON output.
-	 (insert (notmuch-show-get-bodypart-internal ,message-id ,nth))
+	 (insert (notmuch-get-bodypart-internal ,message-id ,nth notmuch-show-process-crypto))
 	 ,@body))))
 
 (defun notmuch-show-save-part (message-id nth &optional filename content-type)
@@ -536,7 +536,7 @@ current buffer, if possible."
 	;; test whether we are able to inline it (which includes both
 	;; capability and suitability tests).
 	(when (mm-inlined-p handle)
-	  (insert (notmuch-show-get-bodypart-content msg part nth))
+	  (insert (notmuch-get-bodypart-content msg part nth notmuch-show-process-crypto))
 	  (when (mm-inlinable-p handle)
 	    (set-buffer display-buffer)
 	    (mm-display-part handle)
@@ -613,8 +613,8 @@ current buffer, if possible."
 	  ;; times (hundreds!), which results in many calls to
 	  ;; `notmuch part'.
 	  (unless content
-	    (setq content (notmuch-show-get-bodypart-internal (concat "id:" message-id)
-							      part-number))
+	    (setq content (notmuch-get-bodypart-internal (concat "id:" message-id)
+							      part-number notmuch-show-process-crypto))
 	    (with-current-buffer w3m-current-buffer
 	      (notmuch-show-w3m-cid-store-internal url
 						   message-id
@@ -734,7 +734,7 @@ current buffer, if possible."
     ;; insert a header to make this clear.
     (if (> nth 1)
 	(notmuch-show-insert-part-header nth declared-type content-type (plist-get part :filename)))
-    (insert (notmuch-show-get-bodypart-content msg part nth))
+    (insert (notmuch-get-bodypart-content msg part nth notmuch-show-process-crypto))
     (save-excursion
       (save-restriction
 	(narrow-to-region start (point-max))
@@ -744,7 +744,7 @@ current buffer, if possible."
 (defun notmuch-show-insert-part-text/calendar (msg part content-type nth depth declared-type)
   (notmuch-show-insert-part-header nth declared-type content-type (plist-get part :filename))
   (insert (with-temp-buffer
-	    (insert (notmuch-show-get-bodypart-content msg part nth))
+	    (insert (notmuch-get-bodypart-content msg part nth notmuch-show-process-crypto))
 	    (goto-char (point-min))
 	    (let ((file (make-temp-file "notmuch-ical"))
 		  result)
@@ -806,25 +806,6 @@ current buffer, if possible."
 		(intern (concat "notmuch-show-insert-part-" content-type))))
     result))
 
-;; Helper for parts which are generally not included in the default
-;; JSON output.
-(defun notmuch-show-get-bodypart-internal (message-id part-number)
-  (let ((args '("show" "--format=raw"))
-	(part-arg (format "--part=%s" part-number)))
-    (setq args (append args (list part-arg)))
-    (if notmuch-show-process-crypto
-	(setq args (append args '("--decrypt"))))
-    (setq args (append args (list message-id)))
-    (with-temp-buffer
-      (let ((coding-system-for-read 'no-conversion))
-	(progn
-	  (apply 'call-process (append (list notmuch-command nil (list t nil) nil) args))
-	  (buffer-string))))))
-
-(defun notmuch-show-get-bodypart-content (msg part nth)
-  (or (plist-get part :content)
-      (notmuch-show-get-bodypart-internal (concat "id:" (plist-get msg :id)) nth)))
-
 ;; \f
 
 (defun notmuch-show-insert-bodypart-internal (msg part content-type nth depth declared-type)
diff --git a/test/emacs b/test/emacs
index 01afdb6..8a28705 100755
--- a/test/emacs
+++ b/test/emacs
@@ -268,13 +268,13 @@ Subject: Re: Testing message sent via SMTP
 In-Reply-To: <XXX>
 Fcc: ${MAIL_DIR}/sent
 --text follows this line--
-On 01 Jan 2000 12:00:00 -0000, Notmuch Test Suite <test_suite@notmuchmail.org> wrote:
+Notmuch Test Suite <test_suite@notmuchmail.org> writes:
+
 > This is a test that messages are sent via SMTP
 EOF
 test_expect_equal_file OUTPUT EXPECTED
 
 test_begin_subtest "Reply within emacs to a multipart/mixed message"
-test_subtest_known_broken
 test_emacs '(notmuch-show "id:20091118002059.067214ed@hikari")
 		(notmuch-show-reply)
 		(test-output)'
@@ -334,7 +334,6 @@ EOF
 test_expect_equal_file OUTPUT EXPECTED
 
 test_begin_subtest "Reply within emacs to a multipart/alternative message"
-test_subtest_known_broken
 test_emacs '(notmuch-show "id:cf0c4d610911171136h1713aa59w9cf9aa31f052ad0a@mail.gmail.com")
 		(notmuch-show-reply)
 		(test-output)'
@@ -385,7 +384,8 @@ Subject: Re: Quote MML tags in reply
 In-Reply-To: <test-emacs-mml-quoting@message.id>
 Fcc: ${MAIL_DIR}/sent
 --text follows this line--
-On Fri, 05 Jan 2001 15:43:57 +0000, Notmuch Test Suite <test_suite@notmuchmail.org> wrote:
+Notmuch Test Suite <test_suite@notmuchmail.org> writes:
+
 > <#!part disposition=inline>
 EOF
 test_expect_equal_file OUTPUT EXPECTED
-- 
1.7.5.4

[PATCH v8 11/11] NEWS: news for reply enhancements

[Adam Wolfe Gordon]

---
 NEWS |   33 +++++++++++++++++++++++++++++++++
 1 files changed, 33 insertions(+), 0 deletions(-)

diff --git a/NEWS b/NEWS
index b49e252..5cfe222 100644
--- a/NEWS
+++ b/NEWS
@@ -1,6 +1,28 @@
 Notmuch 0.13 (2012-xx-xx)
 =========================
 
+Command-Line Interface
+----------------------
+
+Reply to sender
+
+  "notmuch reply" has gained the ability to create a reply template
+  for replying just to the sender of the message, in addition to reply
+  to all. The feature is available through the new command line option
+  --reply-to=(all|sender).
+
+JSON reply format
+
+  "notmuch reply" can now produce JSON output that contains the headers
+  for a reply message and full information about the original message
+  begin replied to. This allows MUAs to create replies intelligtently.
+  For example, an MUA that can parse HTML might quote HTML parts.
+
+  Calling notmuch reply with --format=json imposes the restriction that
+  only a single message is returned by the search, as replying to
+  multiple messages does not have a well-defined behavior. The default
+  retains its current behavior for multiple message replies.
+
 Tag exclusion
 
   Tags can be automatically excluded from search results by adding them
@@ -19,6 +41,17 @@ Tag exclusion
 
     notmuch config set search.exclude_tags deleted spam
 
+Emacs Interface
+---------------
+
+Reply improvement using the JSON format
+
+  Emacs now uses the JSON reply format to create replies. It obeys
+  the customization variables message-citation-line-format and
+  message-citation-line-function when creating the first line of the
+  reply body, and it will quote HTML parts if no text/plain parts are
+  available.
+
 Notmuch 0.12 (2012-xx-xx)
 =========================
 
-- 
1.7.5.4

Re: [PATCH v8 00/11] Reply enhancements (rebased)

[David Bremner]

On Sun, 18 Mar 2012 10:32:32 -0600, Adam Wolfe Gordon <awg+notmuch@xvx.ca> wrote:
> Hi everyone,
> 
> This is a resend of [1], rebased onto the current master. I've also moved the
> NEWS items into the new 0.13 section, as David requested.

pushed, 

d