revision 3: easing access to the cryptographic envelope

revision 3: easing access to the cryptographic envelope

[Daniel Kahn Gillmor]

This is the third revision of the series originally posted at
id:20190424183113.29242-1-dkg@fifthhorseman.net (revision 2 was at
id:20190520032228.27420-1-dkg@fifthhorseman.net)

This series addresses comments raised by David Bremner in his review.
Thanks, Bremner!

The most significant change here is that notmuch-show in --format=json
or --format=sexp now always emits a "crypto" member for every message,
regardless of whether there is any cryptographic envelope.  In the
case where there is no cryptographic envelope, the "crypto" member
will be empty.

------

E-mail structures are potentially arbitrarily complicated.
Cryptographic protection standards like S/MIME and OpenPGP or PGP/MIME
are often applicable to some elements of some messages.

Last year's "E-Fail" attacks made it clear that trying to provide
normal users with cryptographic protections on piecemeal parts of an
e-mail message is a recipe for disaster, both from an implementation
perspective and a user experience perspective.

I've argued in more detail at [0] about the need to treat
cryptographic protections at the message level, rather than at the
subpart level.

[0] https://dkg.fifthhorseman.net/blog/e-mail-cryptography.html

This series makes "notmuch show" track and emit message-wide
cryptographic state, providing an interface that simple clients that
use "notmuch show" can rely on for their UI and UX.

It doesn't yet apply this layer to the emacs interface, because at the
moment many users of the emacs interface are nerds who are as likely
to understand the intricacies of MIME structure as anyone, and for the
moment, just augmenting the notmuch show schemata in a sensible way is
enough of a chunk to bite off.

 (though i'd be happy to review and support the use of this
 per-message cryptographic state in notmuch-emacs if/when this lands!)

I'd appreciate any review and feedback!

Regards,

        --dkg

[PATCH v3 1/4] util/crypto: _notmuch_message_crypto: tracks message-wide crypto state

[Daniel Kahn Gillmor]

E-mail encryption and signatures reported by notmuch are at the MIME
part level.  This makes sense in the dirty details, but for users we
need to have a per-message conception of the cryptographic state of
the e-mail.  (see
https://dkg.fifthhorseman.net/blog/e-mail-cryptography.html for more
discussion of why this is important).

The object created in this patch is a useful for tracking the
cryptographic state of the underlying message as a whole, based on a
depth-first search of the message's MIME structure.

This object stores a signature list of the message, but we don't
handle it yet.  Further patches in this series will make use of the
signature list.
---
 util/crypto.c | 95 +++++++++++++++++++++++++++++++++++++++++++++++++++
 util/crypto.h | 65 +++++++++++++++++++++++++++++++++++
 2 files changed, 160 insertions(+)

diff --git a/util/crypto.c b/util/crypto.c
index 99104e78..d895932d 100644
--- a/util/crypto.c
+++ b/util/crypto.c
@@ -82,3 +82,98 @@ _notmuch_crypto_decrypt (bool *attempted,
 					     decrypt_result, err);
     return ret;
 }
+
+
+static int
+_notmuch_message_crypto_destructor (_notmuch_message_crypto_t *msg_crypto)
+{
+    if (!msg_crypto)
+	return 0;
+    if (msg_crypto->sig_list)
+	g_object_unref (msg_crypto->sig_list);
+    return 0;
+}
+
+_notmuch_message_crypto_t *
+_notmuch_message_crypto_new (void *ctx)
+{
+    _notmuch_message_crypto_t *ret = talloc_zero (ctx, _notmuch_message_crypto_t);
+    talloc_set_destructor (ret, _notmuch_message_crypto_destructor);
+    return ret;
+}
+
+
+notmuch_status_t
+_notmuch_message_crypto_potential_sig_list (_notmuch_message_crypto_t *msg_crypto, GMimeSignatureList *sigs)
+{
+    if (!msg_crypto)
+	return NOTMUCH_STATUS_NULL_POINTER;
+
+    /* Signatures that arrive after a payload part during DFS are not
+     * part of the cryptographic envelope: */
+    if (msg_crypto->payload_encountered)
+	return NOTMUCH_STATUS_SUCCESS;
+
+    if (msg_crypto->sig_list)
+	g_object_unref (msg_crypto->sig_list);
+
+    /* This signature list needs to persist as long as the _n_m_crypto
+     * object survives. Increasing its reference counter prevents
+     * garbage-collection until after _n_m_crypto_destroy is
+     * called. */
+    msg_crypto->sig_list = sigs;
+    if (sigs)
+	g_object_ref (sigs);
+
+    if (msg_crypto->decryption_status == NOTMUCH_MESSAGE_DECRYPTED_FULL)
+	msg_crypto->signature_encrypted = true;
+
+    return NOTMUCH_STATUS_SUCCESS;
+}
+
+
+notmuch_status_t
+_notmuch_message_crypto_potential_payload (_notmuch_message_crypto_t *msg_crypto, GMimeObject *payload, GMimeObject *parent, int childnum)
+{
+    if (!msg_crypto || !payload)
+	return NOTMUCH_STATUS_NULL_POINTER;
+
+    /* only fire on the first payload part encountered */
+    if (msg_crypto->payload_encountered)
+	return NOTMUCH_STATUS_SUCCESS;
+
+    /* the first child of multipart/encrypted that matches the
+     * encryption protocol should be "control information" metadata,
+     * not payload.  So we skip it. (see
+     * https://tools.ietf.org/html/rfc1847#page-8) */
+    if (parent && GMIME_IS_MULTIPART_ENCRYPTED (parent) && childnum == GMIME_MULTIPART_ENCRYPTED_VERSION) {
+	const char *enc_type = g_mime_object_get_content_type_parameter (parent, "protocol");
+	GMimeContentType *ct = g_mime_object_get_content_type (payload);
+	if (ct && enc_type) {
+	    const char *part_type = g_mime_content_type_get_mime_type (ct);
+	    if (part_type && strcmp (part_type, enc_type) == 0)
+		return NOTMUCH_STATUS_SUCCESS;
+	}
+    }
+
+    msg_crypto->payload_encountered = true;
+
+    return NOTMUCH_STATUS_SUCCESS;
+}
+
+
+notmuch_status_t
+_notmuch_message_crypto_successful_decryption (_notmuch_message_crypto_t *msg_crypto)
+{
+    if (!msg_crypto)
+	return NOTMUCH_STATUS_NULL_POINTER;
+
+    /* see the rationale for different values of
+     * _notmuch_message_decryption_status_t in util/crypto.h */
+    if (!msg_crypto->payload_encountered)
+	msg_crypto->decryption_status = NOTMUCH_MESSAGE_DECRYPTED_FULL;
+    else if (msg_crypto->decryption_status == NOTMUCH_MESSAGE_DECRYPTED_NONE)
+	msg_crypto->decryption_status = NOTMUCH_MESSAGE_DECRYPTED_PARTIAL;
+
+    return NOTMUCH_STATUS_SUCCESS;
+}
diff --git a/util/crypto.h b/util/crypto.h
index af3998e8..c6fa7f4b 100644
--- a/util/crypto.h
+++ b/util/crypto.h
@@ -25,6 +25,71 @@ _notmuch_crypto_decrypt (bool *attempted,
 void
 _notmuch_crypto_cleanup (_notmuch_crypto_t *crypto);
 
+/* The user probably wants to know if the entire message was in the
+ * clear.  When replying, the MUA probably wants to know whether there
+ * was any part decrypted in the message.  And when displaying to the
+ * user, we probably only want to display "encrypted message" if the
+ * entire message was covered by encryption. */
+typedef enum {
+    NOTMUCH_MESSAGE_DECRYPTED_NONE = 0,
+    NOTMUCH_MESSAGE_DECRYPTED_PARTIAL,
+    NOTMUCH_MESSAGE_DECRYPTED_FULL,
+} _notmuch_message_decryption_status_t;
+
+/* description of the cryptographic state of a given message overall;
+ * for use by simple user agents.
+ */
+typedef struct _notmuch_message_crypto {
+    /* encryption status: partial, full, none */
+    _notmuch_message_decryption_status_t decryption_status;
+    /* FIXME: can we show what key(s) a fully-encrypted message was
+     * encrypted to? This data is not necessarily cryptographically
+     * reliable; even when we decrypt, we might not know which public
+     * key was used (e.g. if we're using a session key). */
+
+    /* signature status of the whole message (either the whole message
+     * is signed, or it is not) -- this means that partially-signed
+     * messages will get no signature status. */
+    GMimeSignatureList * sig_list;
+    /* if part of the message was signed, and the MUA is clever, it
+     * can determine on its own exactly which part and try to make
+     * more sense of it. */
+
+    /* mark this flag once we encounter a payload (i.e. something that
+     * is not part of the cryptographic envelope) */
+    bool payload_encountered;
+
+    /* if both signed and encrypted, was the signature encrypted? */
+    bool signature_encrypted;
+} _notmuch_message_crypto_t;
+
+
+/* _notmuch_message_crypto_t objects should be released with
+ * talloc_free (), or they will be released along with their parent
+ * context.
+ */
+_notmuch_message_crypto_t *
+_notmuch_message_crypto_new (void *ctx);
+
+/* call potential_sig_list during a depth-first-search on a message to
+ * consider a particular signature as relevant for the message.
+ */
+notmuch_status_t
+_notmuch_message_crypto_potential_sig_list (_notmuch_message_crypto_t *msg_crypto, GMimeSignatureList *sigs);
+
+/* call successful_decryption during a depth-first-search on a message
+ * to indicate that a part was successfully decrypted.
+ */
+notmuch_status_t
+_notmuch_message_crypto_successful_decryption (_notmuch_message_crypto_t *msg_crypto);
+
+/* call potential_payload during a depth-first-search on a message
+ * when encountering a message part that is not part of the envelope.
+ */
+notmuch_status_t
+_notmuch_message_crypto_potential_payload (_notmuch_message_crypto_t *msg_crypto, GMimeObject *payload, GMimeObject *parent, int childnum);
+
+
 #ifdef __cplusplus
 }
 #endif
-- 
2.20.1

[PATCH v3 2/4] cli: expose message-wide crypto status from mime-node

[Daniel Kahn Gillmor]

The mime node context (a per-message context) gains a cryptographic
status object, and the mime_node_t object itself can return a view on
that status to an interested party.

The status is not yet populated, and for now we can keep that view
read-only, so that it can only be populated/modified during MIME tree
traversal.
---
 mime-node.c      | 7 +++++++
 notmuch-client.h | 3 +++
 2 files changed, 10 insertions(+)

diff --git a/mime-node.c b/mime-node.c
index e33336bb..66ff7446 100644
--- a/mime-node.c
+++ b/mime-node.c
@@ -34,6 +34,7 @@ typedef struct mime_node_context {
     GMimeStream *stream;
     GMimeParser *parser;
     GMimeMessage *mime_message;
+    _notmuch_message_crypto_t *msg_crypto;
 
     /* Context provided by the caller. */
     _notmuch_crypto_t *crypto;
@@ -54,6 +55,12 @@ _mime_node_context_free (mime_node_context_t *res)
     return 0;
 }
 
+const _notmuch_message_crypto_t*
+mime_node_get_message_crypto_status (mime_node_t *node)
+{
+    return node->ctx->msg_crypto;
+}
+
 notmuch_status_t
 mime_node_open (const void *ctx, notmuch_message_t *message,
 		_notmuch_crypto_t *crypto, mime_node_t **root_out)
diff --git a/notmuch-client.h b/notmuch-client.h
index d762d3cc..a82cb431 100644
--- a/notmuch-client.h
+++ b/notmuch-client.h
@@ -439,6 +439,9 @@ mime_node_child (mime_node_t *parent, int child);
 mime_node_t *
 mime_node_seek_dfs (mime_node_t *node, int n);
 
+const _notmuch_message_crypto_t*
+mime_node_get_message_crypto_status (mime_node_t *node);
+
 typedef enum dump_formats {
     DUMP_FORMAT_AUTO,
     DUMP_FORMAT_BATCH_TAG,
-- 
2.20.1

[PATCH v3 3/4] mime-node: track whole-message crypto state while walking the tree

[Daniel Kahn Gillmor]

Deliberately populate the message's cryptographic status while walking
the MIME tree from the CLI.

Note that the additional numchild argument added to _mime_node_create
is a passthrough needed to be able to adequately populate the crypto
state object.
---
 mime-node.c | 23 +++++++++++++++++++++--
 1 file changed, 21 insertions(+), 2 deletions(-)

diff --git a/mime-node.c b/mime-node.c
index 66ff7446..48b45247 100644
--- a/mime-node.c
+++ b/mime-node.c
@@ -135,6 +135,8 @@ mime_node_open (const void *ctx, notmuch_message_t *message,
 	goto DONE;
     }
 
+    mctx->msg_crypto = _notmuch_message_crypto_new (mctx);
+
     mctx->crypto = crypto;
 
     /* Create the root node */
@@ -180,6 +182,7 @@ static void
 node_verify (mime_node_t *node, GMimeObject *part)
 {
     GError *err = NULL;
+    notmuch_status_t status;
 
     node->verify_attempted = true;
     node->sig_list = g_mime_multipart_signed_verify
@@ -193,6 +196,10 @@ node_verify (mime_node_t *node, GMimeObject *part)
 
     if (err)
 	g_error_free (err);
+
+    status = _notmuch_message_crypto_potential_sig_list(node->ctx->msg_crypto, node->sig_list);
+    if (status) /* this is a warning, not an error */
+	fprintf (stderr, "Warning: failed to note signature status: %s.\n", notmuch_status_to_string (status));
 }
 
 /* Decrypt and optionally verify an encrypted mime node */
@@ -201,6 +208,7 @@ node_decrypt_and_verify (mime_node_t *node, GMimeObject *part)
 {
     GError *err = NULL;
     GMimeDecryptResult *decrypt_result = NULL;
+    notmuch_status_t status;
     GMimeMultipartEncrypted *encrypteddata = GMIME_MULTIPART_ENCRYPTED (part);
     notmuch_message_t *message = NULL;
 
@@ -223,6 +231,9 @@ node_decrypt_and_verify (mime_node_t *node, GMimeObject *part)
     }
 
     node->decrypt_success = true;
+    status = _notmuch_message_crypto_successful_decryption (node->ctx->msg_crypto);
+    if (status) /* this is a warning, not an error */
+	fprintf (stderr, "Warning: failed to note decryption status: %s.\n", notmuch_status_to_string (status));
 
     if (decrypt_result) {
 	/* This may be NULL if the part is not signed. */
@@ -231,6 +242,9 @@ node_decrypt_and_verify (mime_node_t *node, GMimeObject *part)
 	    node->verify_attempted = true;
 	    g_object_ref (node->sig_list);
 	    set_signature_list_destructor (node);
+	    status = _notmuch_message_crypto_potential_sig_list(node->ctx->msg_crypto, node->sig_list);
+	    if (status) /* this is a warning, not an error */
+		fprintf (stderr, "Warning: failed to note signature status: %s.\n", notmuch_status_to_string (status));
 	}
 
 	if (node->ctx->crypto->decrypt == NOTMUCH_DECRYPT_TRUE && message) {
@@ -251,9 +265,10 @@ node_decrypt_and_verify (mime_node_t *node, GMimeObject *part)
 }
 
 static mime_node_t *
-_mime_node_create (mime_node_t *parent, GMimeObject *part)
+_mime_node_create (mime_node_t *parent, GMimeObject *part, int numchild)
 {
     mime_node_t *node = talloc_zero (parent, mime_node_t);
+    notmuch_status_t status;
 
     /* Set basic node properties */
     node->part = part;
@@ -305,6 +320,10 @@ _mime_node_create (mime_node_t *parent, GMimeObject *part)
 	} else {
 	    node_verify (node, part);
 	}
+    } else {
+	status = _notmuch_message_crypto_potential_payload (node->ctx->msg_crypto, part, parent ? parent->part : NULL, numchild);
+	if (status)
+	    fprintf (stderr, "Warning: failed to record potential crypto payload (%s).\n", notmuch_status_to_string (status));
     }
 
     return node;
@@ -332,7 +351,7 @@ mime_node_child (mime_node_t *parent, int child)
 	INTERNAL_ERROR ("Unexpected GMimeObject type: %s",
 			g_type_name (G_OBJECT_TYPE (parent->part)));
     }
-    node = _mime_node_create (parent, sub);
+    node = _mime_node_create (parent, sub, child);
 
     if (child == parent->next_child && parent->next_part_num != -1) {
 	/* We're traversing in depth-first order.  Record the child's
-- 
2.20.1

[PATCH v3 4/4] cli/show: emit new whole-message crypto status output

[Daniel Kahn Gillmor]

This allows MUAs that don't want to think about per-mime-part
cryptographic status to have a simple high-level overview of the
message's cryptographic state.

Sensibly structured encrypted and/or signed messages will work fine
with this.  The only requirement for the simplest encryption + signing
is that the message have all of its encryption and signing protection
(the "cryptographic envelope") in a contiguous set of MIME layers at
the very outside of the message itself.

This is because messages with some subparts signed or encrypted, but
with other subparts with no cryptographic protection is very difficult
to reason about, and even harder for the user to make sense of or work
with.

For further characterization of the Cryptographic Envelope and some of
the usability tradeoffs, see here:

   https://dkg.fifthhorseman.net/blog/e-mail-cryptography.html#cryptographic-envelope
---
 devel/schemata               | 18 ++++++++++++++++++
 notmuch-show.c               | 29 +++++++++++++++++++++++++++++
 test/T070-insert.sh          |  1 +
 test/T160-json.sh            | 11 ++++++-----
 test/T170-sexp.sh            | 10 +++++-----
 test/T190-multipart.sh       |  4 +++-
 test/T220-reply.sh           |  1 +
 test/T340-maildir-sync.sh    |  1 +
 test/T350-crypto.sh          | 19 +++++++++++++++----
 test/T355-smime.sh           |  5 +++--
 test/T470-missing-headers.sh |  2 ++
 test/T510-thread-replies.sh  | 11 +++++++++++
 test/T670-duplicate-mid.sh   |  1 +
 13 files changed, 96 insertions(+), 17 deletions(-)

diff --git a/devel/schemata b/devel/schemata
index 42b1bcf3..72feb7b7 100644
--- a/devel/schemata
+++ b/devel/schemata
@@ -33,6 +33,8 @@ v3
 v4
 - replace signature error integer bitmask with a set of flags for
   individual errors.
+- (notmuch 0.29) added message.crypto to identify overall message
+  cryptographic state
 
 Common non-terminals
 --------------------
@@ -73,9 +75,25 @@ message = {
     tags:           [string*],
 
     headers:        headers,
+    crypto:         crypto,
     body?:          [part]    # omitted if --body=false
 }
 
+# when showing the message, was any or all of it decrypted?
+msgdecstatus: "full"|"partial"
+
+# The overall cryptographic state of the message as a whole:
+crypto = {
+    signed?:    {
+                  status:      sigstatus,
+                  # was the set of signatures described under encrypted cover?
+                  encrypted:   bool,
+                },
+    decrypted?: {
+                  status: msgdecstatus,
+                }
+}
+
 # A MIME part (format_part_sprinter)
 part = {
     id:             int|string, # part id (currently DFS part number)
diff --git a/notmuch-show.c b/notmuch-show.c
index b95fc389..c6a7a10a 100644
--- a/notmuch-show.c
+++ b/notmuch-show.c
@@ -628,6 +628,35 @@ format_part_sprinter (const void *ctx, sprinter_t *sp, mime_node_t *node,
 	    format_part_sprinter (ctx, sp, mime_node_child (node, 0), true, include_html);
 	    sp->end (sp);
 	}
+
+	if (notmuch_format_version >= 4) {
+	    const _notmuch_message_crypto_t *msg_crypto = mime_node_get_message_crypto_status (node);
+	    sp->map_key (sp, "crypto");
+	    sp->begin_map (sp);
+	    if (msg_crypto->sig_list ||
+		msg_crypto->decryption_status != NOTMUCH_MESSAGE_DECRYPTED_NONE) {
+		if (msg_crypto->sig_list) {
+		    sp->map_key (sp, "signed");
+		    sp->begin_map (sp);
+		    sp->map_key (sp, "status");
+		    format_part_sigstatus_sprinter (sp, msg_crypto->sig_list);
+		    if (msg_crypto->signature_encrypted) {
+			sp->map_key (sp, "encrypted");
+			sp->boolean (sp, msg_crypto->signature_encrypted);
+		    }
+		    sp->end (sp);
+		}
+		if (msg_crypto->decryption_status != NOTMUCH_MESSAGE_DECRYPTED_NONE) {
+		    sp->map_key (sp, "decrypted");
+		    sp->begin_map (sp);
+		    sp->map_key (sp, "status");
+		    sp->string (sp, msg_crypto->decryption_status == NOTMUCH_MESSAGE_DECRYPTED_FULL ? "full" : "partial");
+		    sp->end (sp);
+		}
+	    }
+	    sp->end (sp);
+	}
+
 	sp->end (sp);
 	return;
     }
diff --git a/test/T070-insert.sh b/test/T070-insert.sh
index 05be473a..48165caa 100755
--- a/test/T070-insert.sh
+++ b/test/T070-insert.sh
@@ -48,6 +48,7 @@ test_begin_subtest "Insert message adds default tags"
 output=$(notmuch show --format=json "subject:insert-subject")
 expected='[[[{
  "id": "'"${gen_msg_id}"'",
+ "crypto": {},
  "match": true,
  "excluded": false,
  "filename": ["'"${cur_msg_filename}"'"],
diff --git a/test/T160-json.sh b/test/T160-json.sh
index 91b98e5d..004adb4e 100755
--- a/test/T160-json.sh
+++ b/test/T160-json.sh
@@ -5,16 +5,16 @@ test_description="--format=json output"
 test_begin_subtest "Show message: json"
 add_message "[subject]=\"json-show-subject\"" "[date]=\"Sat, 01 Jan 2000 12:00:00 -0000\"" "[bcc]=\"test_suite+bcc@notmuchmail.org\"" "[reply-to]=\"test_suite+replyto@notmuchmail.org\"" "[body]=\"json-show-message\""
 output=$(notmuch show --format=json "json-show-message")
-test_expect_equal_json "$output" "[[[{\"id\": \"${gen_msg_id}\", \"match\": true, \"excluded\": false, \"filename\": [\"${gen_msg_filename}\"], \"timestamp\": 946728000, \"date_relative\": \"2000-01-01\", \"tags\": [\"inbox\",\"unread\"], \"headers\": {\"Subject\": \"json-show-subject\", \"From\": \"Notmuch Test Suite <test_suite@notmuchmail.org>\", \"To\": \"Notmuch Test Suite <test_suite@notmuchmail.org>\", \"Bcc\": \"test_suite+bcc@notmuchmail.org\", \"Reply-To\": \"test_suite+replyto@notmuchmail.org\", \"Date\": \"Sat, 01 Jan 2000 12:00:00 +0000\"}, \"body\": [{\"id\": 1, \"content-type\": \"text/plain\", \"content\": \"json-show-message\n\"}]}, []]]]"
+test_expect_equal_json "$output" "[[[{\"id\": \"${gen_msg_id}\", \"crypto\": {}, \"match\": true, \"excluded\": false, \"filename\": [\"${gen_msg_filename}\"], \"timestamp\": 946728000, \"date_relative\": \"2000-01-01\", \"tags\": [\"inbox\",\"unread\"], \"headers\": {\"Subject\": \"json-show-subject\", \"From\": \"Notmuch Test Suite <test_suite@notmuchmail.org>\", \"To\": \"Notmuch Test Suite <test_suite@notmuchmail.org>\", \"Bcc\": \"test_suite+bcc@notmuchmail.org\", \"Reply-To\": \"test_suite+replyto@notmuchmail.org\", \"Date\": \"Sat, 01 Jan 2000 12:00:00 +0000\"}, \"body\": [{\"id\": 1, \"content-type\": \"text/plain\", \"content\": \"json-show-message\n\"}]}, []]]]"
 
 # This should be the same output as above.
 test_begin_subtest "Show message: json --body=true"
 output=$(notmuch show --format=json --body=true "json-show-message")
-test_expect_equal_json "$output" "[[[{\"id\": \"${gen_msg_id}\", \"match\": true, \"excluded\": false, \"filename\": [\"${gen_msg_filename}\"], \"timestamp\": 946728000, \"date_relative\": \"2000-01-01\", \"tags\": [\"inbox\",\"unread\"], \"headers\": {\"Subject\": \"json-show-subject\", \"From\": \"Notmuch Test Suite <test_suite@notmuchmail.org>\", \"To\": \"Notmuch Test Suite <test_suite@notmuchmail.org>\", \"Bcc\": \"test_suite+bcc@notmuchmail.org\", \"Reply-To\": \"test_suite+replyto@notmuchmail.org\", \"Date\": \"Sat, 01 Jan 2000 12:00:00 +0000\"}, \"body\": [{\"id\": 1, \"content-type\": \"text/plain\", \"content\": \"json-show-message\n\"}]}, []]]]"
+test_expect_equal_json "$output" "[[[{\"id\": \"${gen_msg_id}\",  \"crypto\": {}, \"match\": true, \"excluded\": false, \"filename\": [\"${gen_msg_filename}\"], \"timestamp\": 946728000, \"date_relative\": \"2000-01-01\", \"tags\": [\"inbox\",\"unread\"], \"headers\": {\"Subject\": \"json-show-subject\", \"From\": \"Notmuch Test Suite <test_suite@notmuchmail.org>\", \"To\": \"Notmuch Test Suite <test_suite@notmuchmail.org>\", \"Bcc\": \"test_suite+bcc@notmuchmail.org\", \"Reply-To\": \"test_suite+replyto@notmuchmail.org\", \"Date\": \"Sat, 01 Jan 2000 12:00:00 +0000\"}, \"body\": [{\"id\": 1, \"content-type\": \"text/plain\", \"content\": \"json-show-message\n\"}]}, []]]]"
 
 test_begin_subtest "Show message: json --body=false"
 output=$(notmuch show --format=json --body=false "json-show-message")
-test_expect_equal_json "$output" "[[[{\"id\": \"${gen_msg_id}\", \"match\": true, \"excluded\": false, \"filename\": [\"${gen_msg_filename}\"], \"timestamp\": 946728000, \"date_relative\": \"2000-01-01\", \"tags\": [\"inbox\",\"unread\"], \"headers\": {\"Subject\": \"json-show-subject\", \"From\": \"Notmuch Test Suite <test_suite@notmuchmail.org>\", \"To\": \"Notmuch Test Suite <test_suite@notmuchmail.org>\", \"Bcc\": \"test_suite+bcc@notmuchmail.org\", \"Reply-To\": \"test_suite+replyto@notmuchmail.org\", \"Date\": \"Sat, 01 Jan 2000 12:00:00 +0000\"}}, []]]]"
+test_expect_equal_json "$output" "[[[{\"id\": \"${gen_msg_id}\",  \"crypto\": {}, \"match\": true, \"excluded\": false, \"filename\": [\"${gen_msg_filename}\"], \"timestamp\": 946728000, \"date_relative\": \"2000-01-01\", \"tags\": [\"inbox\",\"unread\"], \"headers\": {\"Subject\": \"json-show-subject\", \"From\": \"Notmuch Test Suite <test_suite@notmuchmail.org>\", \"To\": \"Notmuch Test Suite <test_suite@notmuchmail.org>\", \"Bcc\": \"test_suite+bcc@notmuchmail.org\", \"Reply-To\": \"test_suite+replyto@notmuchmail.org\", \"Date\": \"Sat, 01 Jan 2000 12:00:00 +0000\"}}, []]]]"
 
 test_begin_subtest "Search message: json"
 add_message "[subject]=\"json-search-subject\"" "[date]=\"Sat, 01 Jan 2000 12:00:00 -0000\"" "[body]=\"json-search-message\""
@@ -33,7 +33,7 @@ test_expect_equal_json "$output" "[{\"thread\": \"XXX\",
 test_begin_subtest "Show message: json, utf-8"
 add_message "[subject]=\"json-show-utf8-body-sübjéct\"" "[date]=\"Sat, 01 Jan 2000 12:00:00 -0000\"" "[body]=\"jsön-show-méssage\""
 output=$(notmuch show --format=json "jsön-show-méssage")
-test_expect_equal_json "$output" "[[[{\"id\": \"${gen_msg_id}\", \"match\": true, \"excluded\": false, \"filename\": [\"${gen_msg_filename}\"], \"timestamp\": 946728000, \"date_relative\": \"2000-01-01\", \"tags\": [\"inbox\",\"unread\"], \"headers\": {\"Subject\": \"json-show-utf8-body-sübjéct\", \"From\": \"Notmuch Test Suite <test_suite@notmuchmail.org>\", \"To\": \"Notmuch Test Suite <test_suite@notmuchmail.org>\", \"Date\": \"Sat, 01 Jan 2000 12:00:00 +0000\"}, \"body\": [{\"id\": 1, \"content-type\": \"text/plain\", \"content\": \"jsön-show-méssage\n\"}]}, []]]]"
+test_expect_equal_json "$output" "[[[{\"id\": \"${gen_msg_id}\",  \"crypto\": {}, \"match\": true, \"excluded\": false, \"filename\": [\"${gen_msg_filename}\"], \"timestamp\": 946728000, \"date_relative\": \"2000-01-01\", \"tags\": [\"inbox\",\"unread\"], \"headers\": {\"Subject\": \"json-show-utf8-body-sübjéct\", \"From\": \"Notmuch Test Suite <test_suite@notmuchmail.org>\", \"To\": \"Notmuch Test Suite <test_suite@notmuchmail.org>\", \"Date\": \"Sat, 01 Jan 2000 12:00:00 +0000\"}, \"body\": [{\"id\": 1, \"content-type\": \"text/plain\", \"content\": \"jsön-show-méssage\n\"}]}, []]]]"
 
 test_begin_subtest "Show message: json, inline attachment filename"
 subject='json-show-inline-attachment-filename'
@@ -48,7 +48,7 @@ output=$(notmuch show --format=json "id:$id")
 filename=$(notmuch search --output=files "id:$id")
 # Get length of README after base64-encoding, minus additional newline.
 attachment_length=$(( $(base64 $NOTMUCH_SRCDIR/test/README | wc -c) - 1 ))
-test_expect_equal_json "$output" "[[[{\"id\": \"$id\", \"match\": true, \"excluded\": false, \"filename\": [\"$filename\"], \"timestamp\": 946728000, \"date_relative\": \"2000-01-01\", \"tags\": [\"inbox\"], \"headers\": {\"Subject\": \"$subject\", \"From\": \"Notmuch Test Suite <test_suite@notmuchmail.org>\", \"To\": \"test_suite@notmuchmail.org\", \"Date\": \"Sat, 01 Jan 2000 12:00:00 +0000\"}, \"body\": [{\"id\": 1, \"content-type\": \"multipart/mixed\", \"content\": [{\"id\": 2, \"content-type\": \"text/plain\", \"content\": \"This is a test message with inline attachment with a filename\"}, {\"id\": 3, \"content-type\": \"application/octet-stream\", \"content-length\": $attachment_length, \"content-transfer-encoding\": \"base64\", \"content-disposition\": \"inline\", \"filename\": \"README\"}]}]}, []]]]"
+test_expect_equal_json "$output" "[[[{\"id\": \"$id\",  \"crypto\": {}, \"match\": true, \"excluded\": false, \"filename\": [\"$filename\"], \"timestamp\": 946728000, \"date_relative\": \"2000-01-01\", \"tags\": [\"inbox\"], \"headers\": {\"Subject\": \"$subject\", \"From\": \"Notmuch Test Suite <test_suite@notmuchmail.org>\", \"To\": \"test_suite@notmuchmail.org\", \"Date\": \"Sat, 01 Jan 2000 12:00:00 +0000\"}, \"body\": [{\"id\": 1, \"content-type\": \"multipart/mixed\", \"content\": [{\"id\": 2, \"content-type\": \"text/plain\", \"content\": \"This is a test message with inline attachment with a filename\"}, {\"id\": 3, \"content-type\": \"application/octet-stream\", \"content-length\": $attachment_length, \"content-transfer-encoding\": \"base64\", \"content-disposition\": \"inline\", \"filename\": \"README\"}]}]}, []]]]"
 
 test_begin_subtest "Search message: json, utf-8"
 add_message "[subject]=\"json-search-utf8-body-sübjéct\"" "[date]=\"Sat, 01 Jan 2000 12:00:00 -0000\"" "[body]=\"jsön-search-méssage\""
@@ -91,6 +91,7 @@ cat <<EOF > EXPECTED
                     "To": "Notmuch Test Suite <test_suite@notmuchmail.org>"
                 },
                 "id": "message-id@example.com",
+                "crypto": {},
                 "match": true,
                 "tags": [
                     "inbox",
diff --git a/test/T170-sexp.sh b/test/T170-sexp.sh
index c3dcf52a..fe7a9dff 100755
--- a/test/T170-sexp.sh
+++ b/test/T170-sexp.sh
@@ -5,16 +5,16 @@ test_description="--format=sexp output"
 test_begin_subtest "Show message: sexp"
 add_message "[subject]=\"sexp-show-subject\"" "[date]=\"Sat, 01 Jan 2000 12:00:00 -0000\"" "[bcc]=\"test_suite+bcc@notmuchmail.org\"" "[reply-to]=\"test_suite+replyto@notmuchmail.org\"" "[body]=\"sexp-show-message\""
 output=$(notmuch show --format=sexp "sexp-show-message")
-test_expect_equal "$output" "((((:id \"${gen_msg_id}\" :match t :excluded nil :filename (\"${gen_msg_filename}\") :timestamp 946728000 :date_relative \"2000-01-01\" :tags (\"inbox\" \"unread\") :headers (:Subject \"sexp-show-subject\" :From \"Notmuch Test Suite <test_suite@notmuchmail.org>\" :To \"Notmuch Test Suite <test_suite@notmuchmail.org>\" :Bcc \"test_suite+bcc@notmuchmail.org\" :Reply-To \"test_suite+replyto@notmuchmail.org\" :Date \"Sat, 01 Jan 2000 12:00:00 +0000\") :body ((:id 1 :content-type \"text/plain\" :content \"sexp-show-message\n\"))) ())))"
+test_expect_equal "$output" "((((:id \"${gen_msg_id}\" :match t :excluded nil :filename (\"${gen_msg_filename}\") :timestamp 946728000 :date_relative \"2000-01-01\" :tags (\"inbox\" \"unread\") :headers (:Subject \"sexp-show-subject\" :From \"Notmuch Test Suite <test_suite@notmuchmail.org>\" :To \"Notmuch Test Suite <test_suite@notmuchmail.org>\" :Bcc \"test_suite+bcc@notmuchmail.org\" :Reply-To \"test_suite+replyto@notmuchmail.org\" :Date \"Sat, 01 Jan 2000 12:00:00 +0000\") :body ((:id 1 :content-type \"text/plain\" :content \"sexp-show-message\n\")) :crypto ()) ())))"
 
 # This should be the same output as above.
 test_begin_subtest "Show message: sexp --body=true"
 output=$(notmuch show --format=sexp --body=true "sexp-show-message")
-test_expect_equal "$output" "((((:id \"${gen_msg_id}\" :match t :excluded nil :filename (\"${gen_msg_filename}\") :timestamp 946728000 :date_relative \"2000-01-01\" :tags (\"inbox\" \"unread\") :headers (:Subject \"sexp-show-subject\" :From \"Notmuch Test Suite <test_suite@notmuchmail.org>\" :To \"Notmuch Test Suite <test_suite@notmuchmail.org>\" :Bcc \"test_suite+bcc@notmuchmail.org\" :Reply-To \"test_suite+replyto@notmuchmail.org\" :Date \"Sat, 01 Jan 2000 12:00:00 +0000\") :body ((:id 1 :content-type \"text/plain\" :content \"sexp-show-message\n\"))) ())))"
+test_expect_equal "$output" "((((:id \"${gen_msg_id}\" :match t :excluded nil :filename (\"${gen_msg_filename}\") :timestamp 946728000 :date_relative \"2000-01-01\" :tags (\"inbox\" \"unread\") :headers (:Subject \"sexp-show-subject\" :From \"Notmuch Test Suite <test_suite@notmuchmail.org>\" :To \"Notmuch Test Suite <test_suite@notmuchmail.org>\" :Bcc \"test_suite+bcc@notmuchmail.org\" :Reply-To \"test_suite+replyto@notmuchmail.org\" :Date \"Sat, 01 Jan 2000 12:00:00 +0000\") :body ((:id 1 :content-type \"text/plain\" :content \"sexp-show-message\n\")) :crypto ()) ())))"
 
 test_begin_subtest "Show message: sexp --body=false"
 output=$(notmuch show --format=sexp --body=false "sexp-show-message")
-test_expect_equal "$output" "((((:id \"${gen_msg_id}\" :match t :excluded nil :filename (\"${gen_msg_filename}\") :timestamp 946728000 :date_relative \"2000-01-01\" :tags (\"inbox\" \"unread\") :headers (:Subject \"sexp-show-subject\" :From \"Notmuch Test Suite <test_suite@notmuchmail.org>\" :To \"Notmuch Test Suite <test_suite@notmuchmail.org>\" :Bcc \"test_suite+bcc@notmuchmail.org\" :Reply-To \"test_suite+replyto@notmuchmail.org\" :Date \"Sat, 01 Jan 2000 12:00:00 +0000\")) ())))"
+test_expect_equal "$output" "((((:id \"${gen_msg_id}\" :match t :excluded nil :filename (\"${gen_msg_filename}\") :timestamp 946728000 :date_relative \"2000-01-01\" :tags (\"inbox\" \"unread\") :headers (:Subject \"sexp-show-subject\" :From \"Notmuch Test Suite <test_suite@notmuchmail.org>\" :To \"Notmuch Test Suite <test_suite@notmuchmail.org>\" :Bcc \"test_suite+bcc@notmuchmail.org\" :Reply-To \"test_suite+replyto@notmuchmail.org\" :Date \"Sat, 01 Jan 2000 12:00:00 +0000\") :crypto ()) ())))"
 
 test_begin_subtest "Search message: sexp"
 add_message "[subject]=\"sexp-search-subject\"" "[date]=\"Sat, 01 Jan 2000 12:00:00 -0000\"" "[body]=\"sexp-search-message\""
@@ -24,7 +24,7 @@ test_expect_equal "$output" "((:thread \"0000000000000002\" :timestamp 946728000
 test_begin_subtest "Show message: sexp, utf-8"
 add_message "[subject]=\"sexp-show-utf8-body-sübjéct\"" "[date]=\"Sat, 01 Jan 2000 12:00:00 -0000\"" "[body]=\"jsön-show-méssage\""
 output=$(notmuch show --format=sexp "jsön-show-méssage")
-test_expect_equal "$output" "((((:id \"${gen_msg_id}\" :match t :excluded nil :filename (\"${gen_msg_filename}\") :timestamp 946728000 :date_relative \"2000-01-01\" :tags (\"inbox\" \"unread\") :headers (:Subject \"sexp-show-utf8-body-sübjéct\" :From \"Notmuch Test Suite <test_suite@notmuchmail.org>\" :To \"Notmuch Test Suite <test_suite@notmuchmail.org>\" :Date \"Sat, 01 Jan 2000 12:00:00 +0000\") :body ((:id 1 :content-type \"text/plain\" :content \"jsön-show-méssage\n\"))) ())))"
+test_expect_equal "$output" "((((:id \"${gen_msg_id}\" :match t :excluded nil :filename (\"${gen_msg_filename}\") :timestamp 946728000 :date_relative \"2000-01-01\" :tags (\"inbox\" \"unread\") :headers (:Subject \"sexp-show-utf8-body-sübjéct\" :From \"Notmuch Test Suite <test_suite@notmuchmail.org>\" :To \"Notmuch Test Suite <test_suite@notmuchmail.org>\" :Date \"Sat, 01 Jan 2000 12:00:00 +0000\") :body ((:id 1 :content-type \"text/plain\" :content \"jsön-show-méssage\n\")) :crypto ()) ())))"
 
 test_begin_subtest "Search message: sexp, utf-8"
 add_message "[subject]=\"sexp-search-utf8-body-sübjéct\"" "[date]=\"Sat, 01 Jan 2000 12:00:00 -0000\"" "[body]=\"jsön-search-méssage\""
@@ -44,6 +44,6 @@ output=$(notmuch show --format=sexp "id:$id")
 filename=$(notmuch search --output=files "id:$id")
 # Get length of README after base64-encoding, minus additional newline.
 attachment_length=$(( $(base64 $NOTMUCH_SRCDIR/test/README | wc -c) - 1 ))
-test_expect_equal "$output" "((((:id \"$id\" :match t :excluded nil :filename (\"$filename\") :timestamp 946728000 :date_relative \"2000-01-01\" :tags (\"inbox\") :headers (:Subject \"sexp-show-inline-attachment-filename\" :From \"Notmuch Test Suite <test_suite@notmuchmail.org>\" :To \"test_suite@notmuchmail.org\" :Date \"Sat, 01 Jan 2000 12:00:00 +0000\") :body ((:id 1 :content-type \"multipart/mixed\" :content ((:id 2 :content-type \"text/plain\" :content \"This is a test message with inline attachment with a filename\") (:id 3 :content-type \"application/octet-stream\" :content-disposition \"inline\" :filename \"README\" :content-transfer-encoding \"base64\" :content-length $attachment_length))))) ())))"
+test_expect_equal "$output" "((((:id \"$id\" :match t :excluded nil :filename (\"$filename\") :timestamp 946728000 :date_relative \"2000-01-01\" :tags (\"inbox\") :headers (:Subject \"sexp-show-inline-attachment-filename\" :From \"Notmuch Test Suite <test_suite@notmuchmail.org>\" :To \"test_suite@notmuchmail.org\" :Date \"Sat, 01 Jan 2000 12:00:00 +0000\") :body ((:id 1 :content-type \"multipart/mixed\" :content ((:id 2 :content-type \"text/plain\" :content \"This is a test message with inline attachment with a filename\") (:id 3 :content-type \"application/octet-stream\" :content-disposition \"inline\" :filename \"README\" :content-transfer-encoding \"base64\" :content-length $attachment_length)))) :crypto ()) ())))"
 
 test_done
diff --git a/test/T190-multipart.sh b/test/T190-multipart.sh
index 5cfa9d33..6f715ff9 100755
--- a/test/T190-multipart.sh
+++ b/test/T190-multipart.sh
@@ -378,7 +378,7 @@ test_expect_success "notmuch show --format=text --part=8 'id:87liy5ap00.fsf@yoom
 test_begin_subtest "--format=json --part=0, full message"
 notmuch show --format=json --part=0 'id:87liy5ap00.fsf@yoom.home.cworth.org' >OUTPUT
 cat <<EOF >EXPECTED
-{"id": "87liy5ap00.fsf@yoom.home.cworth.org", "match": true, "excluded": false, "filename": ["${MAIL_DIR}/multipart"], "timestamp": 978709437, "date_relative": "2001-01-05", "tags": ["attachment","inbox","signed","unread"], "headers": {"Subject": "Multipart message", "From": "Carl Worth <cworth@cworth.org>", "To": "cworth@cworth.org", "Date": "Fri, 05 Jan 2001 15:43:57 +0000"}, "body": [
+{"id": "87liy5ap00.fsf@yoom.home.cworth.org", "crypto": {}, "match": true, "excluded": false, "filename": ["${MAIL_DIR}/multipart"], "timestamp": 978709437, "date_relative": "2001-01-05", "tags": ["attachment","inbox","signed","unread"], "headers": {"Subject": "Multipart message", "From": "Carl Worth <cworth@cworth.org>", "To": "cworth@cworth.org", "Date": "Fri, 05 Jan 2001 15:43:57 +0000"}, "body": [
 {"id": 1, "content-type": "multipart/signed", "content": [
 {"id": 2, "content-type": "multipart/mixed", "content": [
 {"id": 3, "content-type": "message/rfc822", "content-disposition": "inline", "content": [{"headers": {"Subject": "html message", "From": "Carl Worth <cworth@cworth.org>", "To": "cworth@cworth.org", "Date": "Fri, 05 Jan 2001 15:42:57 +0000"}, "body": [
@@ -637,6 +637,7 @@ notmuch_json_show_sanitize <<EOF >EXPECTED
  "In-reply-to": "<87liy5ap00.fsf@yoom.home.cworth.org>",
  "References": "<87liy5ap00.fsf@yoom.home.cworth.org>"},
  "original": {"id": "XXXXX",
+ "crypto": {},
  "match": false,
  "excluded": false,
  "filename": ["YYYYY"],
@@ -728,6 +729,7 @@ cat_expected_head ()
 {
         cat <<EOF
 [[[{"id": "htmlmessage", "match":true, "excluded": false, "date_relative":"2000-01-01",
+   "crypto": {},
    "timestamp": 946684800,
    "filename": ["${MAIL_DIR}/include-html"],
    "tags": ["inbox", "unread"],
diff --git a/test/T220-reply.sh b/test/T220-reply.sh
index 4db3a958..b6d8f42a 100755
--- a/test/T220-reply.sh
+++ b/test/T220-reply.sh
@@ -278,6 +278,7 @@ test_expect_equal_json "$output" '
                 "id": 1
             }
         ],
+        "crypto": {},
         "date_relative": "2010-01-05",
         "excluded": false,
         "filename": ["'${MAIL_DIR}'/msg-014"],
diff --git a/test/T340-maildir-sync.sh b/test/T340-maildir-sync.sh
index 7fece5f2..7416dd61 100755
--- a/test/T340-maildir-sync.sh
+++ b/test/T340-maildir-sync.sh
@@ -45,6 +45,7 @@ test_expect_equal "$output" "adding-replied-tag:2,RS"
 test_begin_subtest "notmuch show works with renamed file (without notmuch new)"
 output=$(notmuch show --format=json id:${gen_msg_id} | notmuch_json_show_sanitize)
 test_expect_equal_json "$output" '[[[{"id": "XXXXX",
+"crypto": {},
 "match": true,
 "excluded": false,
 "filename": ["YYYYY"],
diff --git a/test/T350-crypto.sh b/test/T350-crypto.sh
index 3539bafe..0aada4df 100755
--- a/test/T350-crypto.sh
+++ b/test/T350-crypto.sh
@@ -25,7 +25,7 @@ test_expect_equal "$output" "thread:XXX   2000-01-01 [1/1] Notmuch Test Suite; t
 test_begin_subtest "signature verification"
 output=$(notmuch show --format=json --verify subject:"test signed message 001" \
     | notmuch_json_show_sanitize \
-    | sed -e 's|"created": [1234567890]*|"created": 946728000|')
+    | sed -e 's|"created": [1234567890]*|"created": 946728000|g')
 expected='[[[{"id": "XXXXX",
  "match": true,
  "excluded": false,
@@ -33,6 +33,7 @@ expected='[[[{"id": "XXXXX",
  "timestamp": 946728000,
  "date_relative": "2000-01-01",
  "tags": ["inbox","signed"],
+ "crypto": {"signed": {"status": [{ "status": "good", "created": 946728000, "fingerprint": "'$FINGERPRINT'", "userid": "'"$SELF_USERID"'"}]}},
  "headers": {"Subject": "test signed message 001",
  "From": "Notmuch Test Suite <test_suite@notmuchmail.org>",
  "To": "test_suite@notmuchmail.org",
@@ -74,6 +75,7 @@ expected='[[[{"id": "XXXXX",
  "timestamp": 946728000,
  "date_relative": "2000-01-01",
  "tags": ["inbox","signed"],
+ "crypto": {"signed": {"status": [{ "status": "bad", "keyid": "'$(echo $FINGERPRINT | cut -c 25-)'"}]}},
  "headers": {"Subject": "bad signed message 001",
  "From": "Notmuch Test Suite <test_suite@notmuchmail.org>",
  "To": "test_suite@notmuchmail.org",
@@ -112,6 +114,7 @@ output=$(notmuch show --format=json --verify subject:"bad signed message 002" \
     | notmuch_json_show_sanitize \
     | sed -e 's|"created": [1234567890]*|"created": 946728000|')
 expected='[[[{"id": "XXXXX",
+ "crypto": {},
  "match": true,
  "excluded": false,
  "filename": ["YYYYY"],
@@ -143,7 +146,7 @@ gpg --quiet --batch --no-tty --export-ownertrust > "$GNUPGHOME/ownertrust.bak"
 echo "${FINGERPRINT}:3:" | gpg --quiet --batch --no-tty --import-ownertrust
 output=$(notmuch show --format=json --verify subject:"test signed message 001" \
     | notmuch_json_show_sanitize \
-    | sed -e 's|"created": [1234567890]*|"created": 946728000|')
+    | sed -e 's|"created": [1234567890]*|"created": 946728000|g')
 expected='[[[{"id": "XXXXX",
  "match": true,
  "excluded": false,
@@ -151,6 +154,7 @@ expected='[[[{"id": "XXXXX",
  "timestamp": 946728000,
  "date_relative": "2000-01-01",
  "tags": ["inbox","signed"],
+ "crypto": {"signed": {"status": [{ "status": "good", "created": 946728000, "fingerprint": "'$FINGERPRINT'"}]}},
  "headers": {"Subject": "test signed message 001",
  "From": "Notmuch Test Suite <test_suite@notmuchmail.org>",
  "To": "test_suite@notmuchmail.org",
@@ -177,7 +181,7 @@ test_begin_subtest "signature verification with signer key unavailable"
 mv "${GNUPGHOME}"{,.bak}
 output=$(notmuch show --format=json --verify subject:"test signed message 001" \
     | notmuch_json_show_sanitize \
-    | sed -e 's|"created": [1234567890]*|"created": 946728000|')
+    | sed -e 's|"created": [1234567890]*|"created": 946728000|g')
 expected='[[[{"id": "XXXXX",
  "match": true,
  "excluded": false,
@@ -185,6 +189,7 @@ expected='[[[{"id": "XXXXX",
  "timestamp": 946728000,
  "date_relative": "2000-01-01",
  "tags": ["inbox","signed"],
+ "crypto": {"signed": {"status": [{"errors": {"key-missing": true}, "keyid": "'$(echo $FINGERPRINT | cut -c 25-)'", "status": "error"}]}},
  "headers": {"Subject": "test signed message 001",
  "From": "Notmuch Test Suite <test_suite@notmuchmail.org>",
  "To": "test_suite@notmuchmail.org",
@@ -264,6 +269,7 @@ expected='[[[{"id": "XXXXX",
  "timestamp": 946728000,
  "date_relative": "2000-01-01",
  "tags": ["encrypted","inbox"],
+ "crypto": {"decrypted": {"status": "full"}},
  "headers": {"Subject": "test encrypted message 001",
  "From": "Notmuch Test Suite <test_suite@notmuchmail.org>",
  "To": "test_suite@notmuchmail.org",
@@ -315,6 +321,7 @@ output=$(notmuch show --format=json --decrypt=true subject:"test encrypted messa
     | notmuch_json_show_sanitize \
     | sed -e 's|"created": [1234567890]*|"created": 946728000|')
 expected='[[[{"id": "XXXXX",
+ "crypto": {},
  "match": true,
  "excluded": false,
  "filename": ["YYYYY"],
@@ -350,7 +357,7 @@ test_expect_success \
 test_begin_subtest "decryption + signature verification"
 output=$(notmuch show --format=json --decrypt=true subject:"test encrypted message 002" \
     | notmuch_json_show_sanitize \
-    | sed -e 's|"created": [1234567890]*|"created": 946728000|')
+    | sed -e 's|"created": [1234567890]*|"created": 946728000|g')
 expected='[[[{"id": "XXXXX",
  "match": true,
  "excluded": false,
@@ -358,6 +365,9 @@ expected='[[[{"id": "XXXXX",
  "timestamp": 946728000,
  "date_relative": "2000-01-01",
  "tags": ["encrypted","inbox"],
+ "crypto": {"signed": {"status": [{ "status": "good", "created": 946728000, "fingerprint": "'$FINGERPRINT'", "userid": "'"$SELF_USERID"'"}],
+                       "encrypted": true },
+            "decrypted": {"status": "full"}},
  "headers": {"Subject": "test encrypted message 002",
  "From": "Notmuch Test Suite <test_suite@notmuchmail.org>",
  "To": "test_suite@notmuchmail.org",
@@ -433,6 +443,7 @@ expected='[[[{"id": "XXXXX",
  "timestamp": 946728000,
  "date_relative": "2000-01-01",
  "tags": ["inbox","signed"],
+ "crypto": {"signed": {"status": [{"errors": {"key-revoked": true}, "keyid": "'$(echo $FINGERPRINT | cut -c 25-)'", "status": "error"}]}},
  "headers": {"Subject": "test signed message 001",
  "From": "Notmuch Test Suite <test_suite@notmuchmail.org>",
  "To": "test_suite@notmuchmail.org",
diff --git a/test/T355-smime.sh b/test/T355-smime.sh
index e410286b..336da917 100755
--- a/test/T355-smime.sh
+++ b/test/T355-smime.sh
@@ -50,8 +50,8 @@ test_expect_equal_file EXPECTED OUTPUT
 test_begin_subtest "signature verification (notmuch CLI)"
 output=$(notmuch show --format=json --verify subject:"test signed message 001" \
     | notmuch_json_show_sanitize \
-    | sed -e 's|"created": [-1234567890]*|"created": 946728000|' \
-	  -e 's|"expires": [-1234567890]*|"expires": 424242424|' )
+    | sed -e 's|"created": [-1234567890]*|"created": 946728000|g' \
+	  -e 's|"expires": [-1234567890]*|"expires": 424242424|g' )
 expected='[[[{"id": "XXXXX",
  "match": true,
  "excluded": false,
@@ -59,6 +59,7 @@ expected='[[[{"id": "XXXXX",
  "timestamp": 946728000,
  "date_relative": "2000-01-01",
  "tags": ["inbox","signed"],
+ "crypto": {"signed": {"status": [{"fingerprint": "'$FINGERPRINT'", "status": "good","userid": "CN=Notmuch Test Suite","expires": 424242424, "created": 946728000}]}},
  "headers": {"Subject": "test signed message 001",
  "From": "Notmuch Test Suite <test_suite@notmuchmail.org>",
  "To": "test_suite@notmuchmail.org",
diff --git a/test/T470-missing-headers.sh b/test/T470-missing-headers.sh
index 555fd4e9..32b070ec 100755
--- a/test/T470-missing-headers.sh
+++ b/test/T470-missing-headers.sh
@@ -107,6 +107,7 @@ expected=$(notmuch_json_show_sanitize <<EOF
                         "id": 1
                     }
                 ],
+                "crypto": {},
                 "date_relative": "2001-01-05",
                 "excluded": false,
                 "filename": ["YYYYY"],
@@ -137,6 +138,7 @@ expected=$(notmuch_json_show_sanitize <<EOF
                         "id": 1
                     }
                 ],
+                "crypto": {},
                 "date_relative": "1970-01-01",
                 "excluded": false,
                 "filename": ["YYYYY"],
diff --git a/test/T510-thread-replies.sh b/test/T510-thread-replies.sh
index 5d6bea7e..8b96a1db 100755
--- a/test/T510-thread-replies.sh
+++ b/test/T510-thread-replies.sh
@@ -19,6 +19,7 @@ add_message '[in-reply-to]="mumble"' \
     '[subject]="Re: one"'
 output=$(notmuch show --format=json 'subject:one' | notmuch_json_show_sanitize)
 expected='[[[{"id": "foo@one.com",
+ "crypto": {},
  "match": true,
  "excluded": false,
  "filename": ["YYYYY"],
@@ -33,6 +34,7 @@ expected='[[[{"id": "foo@one.com",
  "content-type": "text/plain",
  "content": "This is just a test message (#1)\n"}]},
  [[{"id": "msg-002@notmuch-test-suite",
+ "crypto": {},
  "match": true, "excluded": false,
  "filename": ["YYYYY"],
  "timestamp": 978709437, "date_relative": "2001-01-05",
@@ -53,6 +55,7 @@ add_message '[in-reply-to]="Your message of December 31 1999 <bar@baz.com>"' \
     '[subject]="Re: two"'
 output=$(notmuch show --format=json 'subject:two' | notmuch_json_show_sanitize)
 expected='[[[{"id": "foo@two.com",
+ "crypto": {},
  "match": true, "excluded": false,
  "filename": ["YYYYY"],
  "timestamp": 978709437, "date_relative": "2001-01-05", "tags": ["inbox", "unread"],
@@ -63,6 +66,7 @@ expected='[[[{"id": "foo@two.com",
  "body": [{"id": 1, "content-type": "text/plain",
  "content": "This is just a test message (#3)\n"}]},
  [[{"id": "msg-004@notmuch-test-suite", "match": true, "excluded": false,
+ "crypto": {},
  "filename": ["YYYYY"],
  "timestamp": 978709437, "date_relative": "2001-01-05", "tags": ["inbox", "unread"],
  "headers": {"Subject": "Re: two",
@@ -82,6 +86,7 @@ add_message '[in-reply-to]="<foo@three.com>"' \
     '[subject]="Re: three"'
 output=$(notmuch show --format=json 'subject:three' | notmuch_json_show_sanitize)
 expected='[[[{"id": "foo@three.com", "match": true, "excluded": false,
+ "crypto": {},
  "filename": ["YYYYY"],
  "timestamp": 978709437, "date_relative": "2001-01-05", "tags": ["inbox", "unread"],
  "headers": {"Subject": "three",
@@ -90,6 +95,7 @@ expected='[[[{"id": "foo@three.com", "match": true, "excluded": false,
  "Date": "Fri, 05 Jan 2001 15:43:57 +0000"}, "body": [{"id": 1,
  "content-type": "text/plain", "content": "This is just a test message (#5)\n"}]},
  [[{"id": "msg-006@notmuch-test-suite", "match": true, "excluded": false,
+ "crypto": {},
  "filename": ["YYYYY"],
  "timestamp": 978709437, "date_relative": "2001-01-05", "tags": ["inbox", "unread"],
  "headers": {"Subject": "Re: three",
@@ -111,6 +117,7 @@ add_message '[in-reply-to]="<baz@four.com> (RFC822 4lyfe)"' \
     '[subject]="neither"'
 output=$(notmuch show --format=json 'subject:four' | notmuch_json_show_sanitize)
 expected='[[[{"id": "foo@four.com", "match": true, "excluded": false,
+ "crypto": {},
  "filename": ["YYYYY"],
  "timestamp": 978709437, "date_relative": "2001-01-05", "tags": ["inbox", "unread"],
  "headers": {"Subject": "four",
@@ -119,6 +126,7 @@ expected='[[[{"id": "foo@four.com", "match": true, "excluded": false,
  "Date": "Fri, 05 Jan 2001 15:43:57 +0000"}, "body": [{"id": 1,
  "content-type": "text/plain", "content": "This is just a test message (#7)\n"}]},
  [[{"id": "msg-009@notmuch-test-suite", "match": false, "excluded": false,
+ "crypto": {},
  "filename": ["YYYYY"],
  "timestamp": 978709437, "date_relative": "2001-01-05", "tags": ["inbox", "unread"],
  "headers": {"Subject": "neither",
@@ -127,6 +135,7 @@ expected='[[[{"id": "foo@four.com", "match": true, "excluded": false,
  "Date": "Fri, 05 Jan 2001 15:43:57 +0000"}, "body": [{"id": 1,
  "content-type": "text/plain", "content": "This is just a test message (#9)\n"}]},
  []]]]], [[{"id": "bar@four.com", "match": true, "excluded": false,
+ "crypto": {},
  "filename": ["YYYYY"],
  "timestamp": 978709437, "date_relative": "2001-01-05", "tags": ["inbox", "unread"],
  "headers": {"Subject": "not-four",
@@ -145,6 +154,7 @@ add_message '[id]="bar@five.com"' \
     '[subject]="not-five"'
 output=$(notmuch show --format=json 'subject:five' | notmuch_json_show_sanitize)
 expected='[[[{"id": "XXXXX", "match": true, "excluded": false,
+ "crypto": {},
  "filename": ["YYYYY"], "timestamp": 42, "date_relative": "2001-01-05",
  "tags": ["inbox", "unread"], "headers": {"Subject": "five",
  "From": "Notmuch Test Suite <test_suite@notmuchmail.org>",
@@ -153,6 +163,7 @@ expected='[[[{"id": "XXXXX", "match": true, "excluded": false,
  "content-type": "text/plain",
  "content": "This is just a test message (#10)\n"}]},
  [[{"id": "XXXXX", "match": true, "excluded": false,
+ "crypto": {},
  "filename": ["YYYYY"], "timestamp": 42, "date_relative": "2001-01-05",
  "tags": ["inbox", "unread"],
  "headers": {"Subject": "not-five",
diff --git a/test/T670-duplicate-mid.sh b/test/T670-duplicate-mid.sh
index fd7df057..c17ccb69 100755
--- a/test/T670-duplicate-mid.sh
+++ b/test/T670-duplicate-mid.sh
@@ -92,6 +92,7 @@ output=$(notmuch show --body=false --format=json id:duplicate |
 	     notmuch_json_show_sanitize | sed 's/message [0-9]/A_SUBJECT/')
 expected='[[[{
     "id": "XXXXX",
+    "crypto": {},
     "match": true,
     "excluded": false,
     "filename": [
-- 
2.20.1

Re: revision 3: easing access to the cryptographic envelope

[David Bremner]

Daniel Kahn Gillmor <dkg@fifthhorseman.net> writes:

> This is the third revision of the series originally posted at
> id:20190424183113.29242-1-dkg@fifthhorseman.net (revision 2 was at
> id:20190520032228.27420-1-dkg@fifthhorseman.net)
>
> This series addresses comments raised by David Bremner in his review.
> Thanks, Bremner!

I pushed this version, with 4 minor whitespace fixups (inserted spaces
and/or deleted blank lines) that I missed in my previous review.

d

Re: revision 3: easing access to the cryptographic envelope

[Daniel Kahn Gillmor]

[-- Attachment #1: Type: text/plain, Size: 848 bytes --]

On Sun 2019-05-26 09:01:46 -0300, David Bremner wrote:
> Daniel Kahn Gillmor <dkg@fifthhorseman.net> writes:
>
>> This is the third revision of the series originally posted at
>> id:20190424183113.29242-1-dkg@fifthhorseman.net (revision 2 was at
>> id:20190520032228.27420-1-dkg@fifthhorseman.net)
>>
>> This series addresses comments raised by David Bremner in his review.
>> Thanks, Bremner!
>
> I pushed this version, with 4 minor whitespace fixups (inserted spaces
> and/or deleted blank lines) that I missed in my previous review.

Thanks for these fixes, Bremner.  If you have a specific set of tooling
that you use that i can adopt to catch those kinds of coding convention
mishaps before submitting, i'd be happy to adopt it so things are
"clean".  Bonus points if it integrates into emacs via .dir-locals.el or
something :P

        --dkg

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

Re: revision 3: easing access to the cryptographic envelope

[David Bremner]

Daniel Kahn Gillmor <dkg@fifthhorseman.net> writes:

> On Sun 2019-05-26 09:01:46 -0300, David Bremner wrote:
>> Daniel Kahn Gillmor <dkg@fifthhorseman.net> writes:
>>
>>> This is the third revision of the series originally posted at
>>> id:20190424183113.29242-1-dkg@fifthhorseman.net (revision 2 was at
>>> id:20190520032228.27420-1-dkg@fifthhorseman.net)
>>>
>>> This series addresses comments raised by David Bremner in his review.
>>> Thanks, Bremner!
>>
>> I pushed this version, with 4 minor whitespace fixups (inserted spaces
>> and/or deleted blank lines) that I missed in my previous review.
>
> Thanks for these fixes, Bremner.  If you have a specific set of tooling
> that you use that i can adopt to catch those kinds of coding convention
> mishaps before submitting, i'd be happy to adopt it so things are
> "clean".  Bonus points if it integrates into emacs via .dir-locals.el or
> something :P

To be honest my main mechanism for catching those problems is Tomi
;). There is also a reasonable uncrustify configuration that I don't use
as often as I should, mainly because the baseline in various files is
not there. Perhaps if we did more some global whitespace cleanup this
would be more helpful. To try it out run

% uncrustify -c devel/uncrustify.cfg --replace $files

then

% git diff

to see what changed.

% find -name .git -prune -o -type f -a -regex '.*[.]\(c\|h\|cc\)' -print -exec uncrustify -c devel/uncrustify.cfg --replace {} \;

yields a whopping 1933 insertions and 1903 deletions. Perhaps there are
some places where the uncrustify config needs tuning, but none of the
output looked crazy to me. Two things I noticed causing lots of changes

1) "!foo" -> "! foo"
2) putting a leading * in front of multi-line comments

If we do decide to rip off the bandage, that will cause a certain amount
of rebasing pain for any patch series in flight; now (i.e after the
release) is actually a pretty good time from my point of view, but
others (e.g. you) might feel differently.

Re: revision 3: easing access to the cryptographic envelope

[Daniel Kahn Gillmor]

[-- Attachment #1: Type: text/plain, Size: 637 bytes --]

On Sun 2019-05-26 13:54:29 -0300, David Bremner wrote:
> % uncrustify -c devel/uncrustify.cfg --replace $files

thanks for this pointer, i will experiment with it.

> If we do decide to rip off the bandage, that will cause a certain amount
> of rebasing pain for any patch series in flight; now (i.e after the
> release) is actually a pretty good time from my point of view, but
> others (e.g. you) might feel differently.

I can live with this kind of pain, if it means less frustration later,
but what makes you think that "after the release" == "now" ? :P  we
still have ~5 days till the proposed feature freeze, right?

       --dkg

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

Re: revision 3: easing access to the cryptographic envelope

[Tomi Ollila]

On Sun, May 26 2019, David Bremner wrote:

> Daniel Kahn Gillmor <dkg@fifthhorseman.net> writes:
>
>> On Sun 2019-05-26 09:01:46 -0300, David Bremner wrote:
>>> Daniel Kahn Gillmor <dkg@fifthhorseman.net> writes:
>>>
>>>> This is the third revision of the series originally posted at
>>>> id:20190424183113.29242-1-dkg@fifthhorseman.net (revision 2 was at
>>>> id:20190520032228.27420-1-dkg@fifthhorseman.net)
>>>>
>>>> This series addresses comments raised by David Bremner in his review.
>>>> Thanks, Bremner!
>>>
>>> I pushed this version, with 4 minor whitespace fixups (inserted spaces
>>> and/or deleted blank lines) that I missed in my previous review.
>>
>> Thanks for these fixes, Bremner.  If you have a specific set of tooling
>> that you use that i can adopt to catch those kinds of coding convention
>> mishaps before submitting, i'd be happy to adopt it so things are
>> "clean".  Bonus points if it integrates into emacs via .dir-locals.el or
>> something :P
>
> To be honest my main mechanism for catching those problems is Tomi
> ;). 

I used to have simple trailing whitespace cleanup hook 
(with its problems). Since Now 2012 I've used ethan-wspace.el which
does cleanups only initially clean content (and higlights if not --
my favorite feature). Like w/ notmuch I also use a bit different version:

localhost$ diff -u vc/ext/ethan-wspace/lisp/ethan-wspace.el dotdir/elisp/ethan-wspace.el
--- vc/ext/ethan-wspace/lisp/ethan-wspace.el     2019-05-27 23:32:26.625456294 +0300
+++ dotdir/elisp/ethan-wspace.el      2019-05-27 23:38:34.150404797 +0300
@@ -800,8 +800,7 @@
           (setq require-final-newline nil))
 
         (when indent-tabs-mode
-          (setq ethan-wspace-errors
-                (remove 'tabs ethan-wspace-errors)))
+                  (ethan-wspace-highlight-tabs-mode))
 
         (run-hooks 'ethan-wspace-errors-in-buffer-hook)
         (ethan-wspace-update-buffer)
zsh: exit 1     diff -u ~/vc/ext/ethan-wspace/lisp/ethan-wspace.el 

(applied latest change from github just now, changed there 2019-05-21)

> There is also a reasonable uncrustify configuration that I don't use
> as often as I should, mainly because the baseline in various files is
> not there. Perhaps if we did more some global whitespace cleanup this
> would be more helpful. To try it out run
>
> % uncrustify -c devel/uncrustify.cfg --replace $files
>
> then
>
> % git diff
>
> to see what changed.
>
> % find -name .git -prune -o -type f -a -regex '.*[.]\(c\|h\|cc\)' -print -exec uncrustify -c devel/uncrustify.cfg --replace {} \;
>
> yields a whopping 1933 insertions and 1903 deletions. Perhaps there are
> some places where the uncrustify config needs tuning, but none of the
> output looked crazy to me. Two things I noticed causing lots of changes
>
> 1) "!foo" -> "! foo"
> 2) putting a leading * in front of multi-line comments

Last time I played w/ uncrustify it was like 2014 -- and I recall there
were things that uncrustify just could not do (more than above) -- like
indenting differently than emacs would do (and IMO something ugly way).

So I wonder if we ever can use uncrustify as tool to verify change
formatting (i.e. identical output or FAIL)...

... perhaps newer uncrustify (latest 0.69, in 2014 version was 0.60)
(with modified uncrustify.cfg) could help there (I also have done
some experiments with "uncrustify postprocessor"; I should take some
time to see how that works with latest tool...)

> If we do decide to rip off the bandage, that will cause a certain amount
> of rebasing pain for any patch series in flight; now (i.e after the
> release) is actually a pretty good time from my point of view, but
> others (e.g. you) might feel differently.

If uncrustifying can be done (even partially), IMO it could be done;
patch series authors just have to revisit their changes (keeping those
live is their duty anyway).


Tomi