v3: notmuch show --decrypt=stash

v3: notmuch show --decrypt=stash

[Daniel Kahn Gillmor]

This is an improvement on the series most recently sent in
id:20180110001228.2211-1-dkg@fifthhorseman.net (with the initial
version in id:20171212025225.11854-1-dkg@fifthhorseman.net).

The differences between this and v2 of this series are cleanup and
readability improvements suggested by David Bremner.

This series allows "notmuch show" to index the cleartext and stash the
session keys of an encrypted message while displaying it.

Background
----------

As of notmuch 0.26, cleartext indexing and session-keys make working
with encrypted e-mail significantly easier in notmuch.  However, the
underlying requirement is that at the time of message ingestion (and
"notmuch new" in particular), the user is likely to have access to
their long-term secret keys.

In practice, many people using GnuPG today have their secret keys
locked behind a passphrase, or on a smartcard, and also run "notmuch
new" in some sort of scheduled, backgrounded process.

The result is that for users with this workflow, GnuPG prompts for
their passphrase (or wants to make use of their smartcard) at
unpredictable times, depending on when their mail delivery happens,
and on how many encrypted messages they receive.  This is both
unfriendly and bad for security (we should not train users to approve
random prompts for secret key access when nothing they're doing
interactively seems to warrant it).

Outline
-------

For a friendlier experience, some users may prefer incoming encrypted
mail to stay in their inbox *without* being decrypted, until they
choose to look at it.  At the moment that they're looking at it, their
MUA is in the foreground and they're interacting with it, so being
prompted for their password or smartcard interactively makes sense at
that time.

This series makes it possible for this interaction to to actually
decrypt the message, index it, and stash any session keys the first
time the user interacts with the message through "notmuch show".

This is not a workflow that every MUA will choose to use (e.g. users
whose decryption-capable secret key is already cheaply available
without hassling the user at "notmuch new" shouldn't use it), but it
is a sensible workflow for some users that notmuch should support.

Furthermore, it is a more efficient use of secret key material -- a
user that wants to stash session keys of a message, but whose
long-term decryption secret key is on a smartcard should only be obliged
to trigger the smartcard once per message, ever.

Implementation details
----------------------

The most controversial part of this series is that it makes "notmuch
show" potentially not a read-only operation on the database.  This is
a tradeoff that the users of this workflow will need to consider,
since they are explicitly asking "notmuch show" to potentially modify
their index.

Note that i've made this R/O-to-R/W switch fairly coarse.  If the user
requests --decrypt=stash, then "notmuch show" will operate on a
read/write database, regardless of whether the message is actually
encrypted.  I used this coarse approach because i couldn't figure out
a safe way to reopen an existing read-only database in read-write
mode.  If someone more clever with Xapian than me wants to suggest a
way to do this in a more fine-grained fashion, i'd welcome patches or
pointers.  That said, if users don't like the idea of "notmuch show"
using the db read/write, they can also just not use --decrypt=stash :)


Let me know if you see anything else that could be improved here!

    --dkg

[PATCH v3 1/8] lib: expose notmuch_message_get_database()

[Daniel Kahn Gillmor]

We've had _notmuch_message_database() internally for a while, and it's
useful.  It turns out to be useful on the other side of the library
interface as well (i'll use it later in this series for "notmuch
show"), so we expose it publicly now.
---
 lib/index.cc            | 10 +++++-----
 lib/message-property.cc |  4 ++--
 lib/message.cc          | 14 +++++++-------
 lib/notmuch-private.h   |  2 --
 lib/notmuch.h           |  8 ++++++++
 5 files changed, 22 insertions(+), 16 deletions(-)

diff --git a/lib/index.cc b/lib/index.cc
index 0ad683fa..22ca9ec1 100644
--- a/lib/index.cc
+++ b/lib/index.cc
@@ -385,7 +385,7 @@ _index_mime_part (notmuch_message_t *message,
     const char *charset;
 
     if (! part) {
-	_notmuch_database_log (_notmuch_message_database (message),
+	_notmuch_database_log (notmuch_message_get_database (message),
 			      "Warning: Not indexing empty mime part.\n");
 	return;
     }
@@ -411,7 +411,7 @@ _index_mime_part (notmuch_message_t *message,
 					 g_mime_multipart_get_part (multipart, i));
 		    continue;
 		} else if (i != GMIME_MULTIPART_SIGNED_CONTENT) {
-		    _notmuch_database_log (_notmuch_message_database (message),
+		    _notmuch_database_log (notmuch_message_get_database (message),
 					   "Warning: Unexpected extra parts of multipart/signed. Indexing anyway.\n");
 		}
 	    }
@@ -424,7 +424,7 @@ _index_mime_part (notmuch_message_t *message,
 					       GMIME_MULTIPART_ENCRYPTED (part));
 		} else {
 		    if (i != GMIME_MULTIPART_ENCRYPTED_VERSION) {
-			_notmuch_database_log (_notmuch_message_database (message),
+			_notmuch_database_log (notmuch_message_get_database (message),
 					       "Warning: Unexpected extra parts of multipart/encrypted.\n");
 		    }
 		}
@@ -447,7 +447,7 @@ _index_mime_part (notmuch_message_t *message,
     }
 
     if (! (GMIME_IS_PART (part))) {
-	_notmuch_database_log (_notmuch_message_database (message),
+	_notmuch_database_log (notmuch_message_get_database (message),
 			      "Warning: Not indexing unknown mime part: %s.\n",
 			      g_type_name (G_OBJECT_TYPE (part)));
 	return;
@@ -528,7 +528,7 @@ _index_encrypted_mime_part (notmuch_message_t *message,
     if (!indexopts || (notmuch_indexopts_get_decrypt_policy (indexopts) == NOTMUCH_DECRYPT_FALSE))
 	return;
 
-    notmuch = _notmuch_message_database (message);
+    notmuch = notmuch_message_get_database (message);
 
     GMimeCryptoContext* crypto_ctx = NULL;
 #if (GMIME_MAJOR_VERSION < 3)
diff --git a/lib/message-property.cc b/lib/message-property.cc
index 6525fb24..210a15cc 100644
--- a/lib/message-property.cc
+++ b/lib/message-property.cc
@@ -44,7 +44,7 @@ _notmuch_message_modify_property (notmuch_message_t *message, const char *key, c
     notmuch_status_t status;
     char *term = NULL;
 
-    status = _notmuch_database_ensure_writable (_notmuch_message_database (message));
+    status = _notmuch_database_ensure_writable (notmuch_message_get_database (message));
     if (status)
 	return status;
 
@@ -92,7 +92,7 @@ _notmuch_message_remove_all_properties (notmuch_message_t *message, const char *
     notmuch_status_t status;
     const char * term_prefix;
 
-    status = _notmuch_database_ensure_writable (_notmuch_message_database (message));
+    status = _notmuch_database_ensure_writable (notmuch_message_get_database (message));
     if (status)
 	return status;
 
diff --git a/lib/message.cc b/lib/message.cc
index b2067076..a7e8c3ca 100644
--- a/lib/message.cc
+++ b/lib/message.cc
@@ -268,7 +268,7 @@ _notmuch_message_create_for_message_id (notmuch_database_t *notmuch,
 
 	doc_id = _notmuch_database_generate_doc_id (notmuch);
     } catch (const Xapian::Error &error) {
-	_notmuch_database_log(_notmuch_message_database (message), "A Xapian exception occurred creating message: %s\n",
+	_notmuch_database_log(notmuch_message_get_database (message), "A Xapian exception occurred creating message: %s\n",
 		 error.get_msg().c_str());
 	notmuch->exception_reported = true;
 	*status_ret = NOTMUCH_PRIVATE_STATUS_XAPIAN_EXCEPTION;
@@ -512,7 +512,7 @@ _notmuch_message_ensure_message_file (notmuch_message_t *message)
 	return;
 
     message->message_file = _notmuch_message_file_open_ctx (
-	_notmuch_message_database (message), message, filename);
+	notmuch_message_get_database (message), message, filename);
 }
 
 const char *
@@ -542,7 +542,7 @@ notmuch_message_get_header (notmuch_message_t *message, const char *header)
 		return talloc_strdup (message, value.c_str ());
 
 	} catch (Xapian::Error &error) {
-	    _notmuch_database_log(_notmuch_message_database (message), "A Xapian exception occurred when reading header: %s\n",
+	    _notmuch_database_log(notmuch_message_get_database (message), "A Xapian exception occurred when reading header: %s\n",
 		     error.get_msg().c_str());
 	    message->notmuch->exception_reported = true;
 	    return NULL;
@@ -663,7 +663,7 @@ _notmuch_message_remove_indexed_terms (notmuch_message_t *message)
 	    notmuch_database_t *notmuch = message->notmuch;
 
 	    if (!notmuch->exception_reported) {
-		_notmuch_database_log(_notmuch_message_database (message), "A Xapian exception occurred creating message: %s\n",
+		_notmuch_database_log(notmuch_message_get_database (message), "A Xapian exception occurred creating message: %s\n",
 				      error.get_msg().c_str());
 		notmuch->exception_reported = true;
 	    }
@@ -1059,7 +1059,7 @@ notmuch_message_get_date (notmuch_message_t *message)
     try {
 	value = message->doc.get_value (NOTMUCH_VALUE_TIMESTAMP);
     } catch (Xapian::Error &error) {
-	_notmuch_database_log(_notmuch_message_database (message), "A Xapian exception occurred when reading date: %s\n",
+	_notmuch_database_log(notmuch_message_get_database (message), "A Xapian exception occurred when reading date: %s\n",
 		 error.get_msg().c_str());
 	message->notmuch->exception_reported = true;
 	return 0;
@@ -1925,7 +1925,7 @@ notmuch_message_destroy (notmuch_message_t *message)
 }
 
 notmuch_database_t *
-_notmuch_message_database (notmuch_message_t *message)
+notmuch_message_get_database (notmuch_message_t *message)
 {
     return message->notmuch;
 }
@@ -2002,7 +2002,7 @@ notmuch_message_reindex (notmuch_message_t *message,
     /* strdup it because the metadata may be invalidated */
     orig_thread_id = talloc_strdup (message, orig_thread_id);
 
-    notmuch = _notmuch_message_database (message);
+    notmuch = notmuch_message_get_database (message);
 
     ret = _notmuch_database_ensure_writable (notmuch);
     if (ret)
diff --git a/lib/notmuch-private.h b/lib/notmuch-private.h
index 4598577f..3764a6a9 100644
--- a/lib/notmuch-private.h
+++ b/lib/notmuch-private.h
@@ -532,8 +532,6 @@ _notmuch_message_id_parse (void *ctx, const char *message_id, const char **next)
 void
 _notmuch_message_add_reply (notmuch_message_t *message,
 			    notmuch_message_t *reply);
-notmuch_database_t *
-_notmuch_message_database (notmuch_message_t *message);
 
 void
 _notmuch_message_remove_unprefixed_terms (notmuch_message_t *message);
diff --git a/lib/notmuch.h b/lib/notmuch.h
index 4db28a05..06842517 100644
--- a/lib/notmuch.h
+++ b/lib/notmuch.h
@@ -1345,6 +1345,14 @@ notmuch_messages_destroy (notmuch_messages_t *messages);
 notmuch_tags_t *
 notmuch_messages_collect_tags (notmuch_messages_t *messages);
 
+/**
+ * Get the database associated with this message.
+ *
+ * @since libnotmuch 5.2 (notmuch 0.27)
+ */
+notmuch_database_t *
+notmuch_message_get_database (notmuch_message_t *message);
+
 /**
  * Get the message ID of 'message'.
  *
-- 
2.17.0

[PATCH v3 2/8] properties: add notmuch_message_count_properties

[Daniel Kahn Gillmor]

The user can already do this manually, of course, but (a) it's nice to
have a convenience function, and (b) exposing this interface means
that someone more clever with a _notmuch_string_map_t than i am can
write a more efficient version if they like, and it will just
accelerate the users of the convenience function.
---
 lib/message-property.cc | 25 +++++++++++++++++++++++++
 lib/notmuch.h           | 16 ++++++++++++++++
 2 files changed, 41 insertions(+)

diff --git a/lib/message-property.cc b/lib/message-property.cc
index 210a15cc..710ba046 100644
--- a/lib/message-property.cc
+++ b/lib/message-property.cc
@@ -36,6 +36,31 @@ notmuch_message_get_property (notmuch_message_t *message, const char *key, const
     return NOTMUCH_STATUS_SUCCESS;
 }
 
+notmuch_status_t
+notmuch_message_count_properties (notmuch_message_t *message, const char *key, unsigned int *count)
+{
+    if (! count || ! key || ! message)
+	return NOTMUCH_STATUS_NULL_POINTER;
+
+    notmuch_string_map_t *map;
+    map = _notmuch_message_property_map (message);
+    if (! map)
+	return NOTMUCH_STATUS_NULL_POINTER;
+
+    notmuch_string_map_iterator_t *matcher = _notmuch_string_map_iterator_create (map, key, true);
+    if (! matcher)
+	return NOTMUCH_STATUS_OUT_OF_MEMORY;
+
+    *count = 0;
+    while (_notmuch_string_map_iterator_valid (matcher)) {
+	(*count)++;
+	_notmuch_string_map_iterator_move_to_next (matcher);
+    }
+
+    _notmuch_string_map_iterator_destroy (matcher);
+    return NOTMUCH_STATUS_SUCCESS;
+}
+
 static notmuch_status_t
 _notmuch_message_modify_property (notmuch_message_t *message, const char *key, const char *value,
 				  bool delete_it)
diff --git a/lib/notmuch.h b/lib/notmuch.h
index 06842517..df8f534a 100644
--- a/lib/notmuch.h
+++ b/lib/notmuch.h
@@ -1890,6 +1890,22 @@ typedef struct _notmuch_string_map_iterator notmuch_message_properties_t;
 notmuch_message_properties_t *
 notmuch_message_get_properties (notmuch_message_t *message, const char *key, notmuch_bool_t exact);
 
+/**
+ * Return the number of properties named "key" belonging to the specific message.
+ *
+ * @param[in] message  The message to examine
+ * @param[in] key      key to count
+ * @param[out] count   The number of matching properties associated with this message.
+ *
+ * @returns
+ *
+ * NOTMUCH_STATUS_SUCCESS: successful count, possibly some other error.
+ *
+ * @since libnotmuch 5.2 (notmuch 0.27)
+ */
+notmuch_status_t
+notmuch_message_count_properties (notmuch_message_t *message, const char *key, unsigned int *count);
+
 /**
  * Is the given *properties* iterator pointing at a valid (key,value)
  * pair.
-- 
2.17.0

[PATCH v3 3/8] lib: make notmuch_message_get_database() take a const notmuch_message_t*

[Daniel Kahn Gillmor]

This is technically an API change, but it is not an ABI change, and
it's merely a statement that limits what the library can do.

This is in parallel to notmuch_query_get_database(), which also takes
a const pointer.
---
 lib/message.cc | 2 +-
 lib/notmuch.h  | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/lib/message.cc b/lib/message.cc
index a7e8c3ca..153e4bed 100644
--- a/lib/message.cc
+++ b/lib/message.cc
@@ -1925,7 +1925,7 @@ notmuch_message_destroy (notmuch_message_t *message)
 }
 
 notmuch_database_t *
-notmuch_message_get_database (notmuch_message_t *message)
+notmuch_message_get_database (const notmuch_message_t *message)
 {
     return message->notmuch;
 }
diff --git a/lib/notmuch.h b/lib/notmuch.h
index df8f534a..c2a1912c 100644
--- a/lib/notmuch.h
+++ b/lib/notmuch.h
@@ -1351,7 +1351,7 @@ notmuch_messages_collect_tags (notmuch_messages_t *messages);
  * @since libnotmuch 5.2 (notmuch 0.27)
  */
 notmuch_database_t *
-notmuch_message_get_database (notmuch_message_t *message);
+notmuch_message_get_database (const notmuch_message_t *message);
 
 /**
  * Get the message ID of 'message'.
-- 
2.17.0

[PATCH v3 4/8] cli: add print_status_message()

[Daniel Kahn Gillmor]

This function is a parallel to print_status_query() or
print_status_database().  Thanks to David Bremner for the suggestion!
---
 notmuch-client.h |  5 +++++
 status.c         | 20 ++++++++++++++++++++
 2 files changed, 25 insertions(+)

diff --git a/notmuch-client.h b/notmuch-client.h
index 0985a7b0..6c84ecc0 100644
--- a/notmuch-client.h
+++ b/notmuch-client.h
@@ -481,6 +481,11 @@ print_status_query (const char *loc,
 		    const notmuch_query_t *query,
 		    notmuch_status_t status);
 
+notmuch_status_t
+print_status_message (const char *loc,
+		      const notmuch_message_t *message,
+		      notmuch_status_t status);
+
 notmuch_status_t
 print_status_database (const char *loc,
 		       const notmuch_database_t *database,
diff --git a/status.c b/status.c
index 8bc2fe4b..01fd8c99 100644
--- a/status.c
+++ b/status.c
@@ -20,6 +20,26 @@ print_status_query (const char *loc,
     return status;
 }
 
+notmuch_status_t
+print_status_message (const char *loc,
+		      const notmuch_message_t *message,
+		      notmuch_status_t status)
+{
+    if (status) {
+	const char *msg;
+	notmuch_database_t *notmuch;
+
+	fprintf (stderr, "%s: %s\n", loc,
+		 notmuch_status_to_string (status));
+
+	notmuch = notmuch_message_get_database (message);
+	msg = notmuch_database_status_string (notmuch);
+	if (msg)
+	    fputs (msg, stderr);
+    }
+    return status;
+}
+
 notmuch_status_t
 print_status_database (const char *loc,
 		    const notmuch_database_t *notmuch,
-- 
2.17.0

[PATCH v3 5/8] cli: write session keys to database, if asked to do so

[Daniel Kahn Gillmor]

If the decryption policy is NOTMUCH_DECRYPT_TRUE, that means we want
to stash session keys in the database.  Note that there is currently
no way from the command line to set it this way, though, so it is not
yet included in the test suite.
---
 mime-node.c | 22 ++++++++++++++++++----
 1 file changed, 18 insertions(+), 4 deletions(-)

diff --git a/mime-node.c b/mime-node.c
index 11df082b..2a24e537 100644
--- a/mime-node.c
+++ b/mime-node.c
@@ -197,16 +197,18 @@ node_decrypt_and_verify (mime_node_t *node, GMimeObject *part,
     GError *err = NULL;
     GMimeDecryptResult *decrypt_result = NULL;
     GMimeMultipartEncrypted *encrypteddata = GMIME_MULTIPART_ENCRYPTED (part);
+    notmuch_message_t *message = NULL;
 
     if (! node->decrypted_child) {
-	mime_node_t *parent;
-	for (parent = node; parent; parent = parent->parent)
-	    if (parent->envelope_file)
+	for (mime_node_t *parent = node; parent; parent = parent->parent)
+	    if (parent->envelope_file) {
+		message = parent->envelope_file;
 		break;
+	    }
 
 	node->decrypted_child = _notmuch_crypto_decrypt (&node->decrypt_attempted,
 							 node->ctx->crypto->decrypt,
-							 parent ? parent->envelope_file : NULL,
+							 message,
 							 cryptoctx, encrypteddata, &decrypt_result, &err);
     }
     if (! node->decrypted_child) {
@@ -225,6 +227,18 @@ node_decrypt_and_verify (mime_node_t *node, GMimeObject *part,
 	    g_object_ref (node->sig_list);
 	    set_signature_list_destructor (node);
 	}
+
+#if HAVE_GMIME_SESSION_KEYS
+	if (node->ctx->crypto->decrypt == NOTMUCH_DECRYPT_TRUE && message) {
+	    notmuch_database_t *db = notmuch_message_get_database (message);
+	    const char *session_key = g_mime_decrypt_result_get_session_key (decrypt_result);
+	    if (db && session_key)
+		print_status_message ("Failed to stash session key in the database",
+				      message,
+				      notmuch_message_add_property (message, "session-key",
+								    session_key));
+	}
+#endif
 	g_object_unref (decrypt_result);
     }
 
-- 
2.17.0

[PATCH v3 6/8] cli/show: reindex when we learned new session keys about a message

[Daniel Kahn Gillmor]

If the number of session keys for a given message increased after
running "notmuch show" then we just learned something new that might
let us do automatic decryption.  We should reindex this message using
our newfound knowledge.
---
 notmuch-show.c | 19 +++++++++++++++++++
 1 file changed, 19 insertions(+)

diff --git a/notmuch-show.c b/notmuch-show.c
index 9871159d..f34cf941 100644
--- a/notmuch-show.c
+++ b/notmuch-show.c
@@ -873,6 +873,11 @@ show_message (void *ctx,
     void *local = talloc_new (ctx);
     mime_node_t *root, *part;
     notmuch_status_t status;
+    unsigned int session_keys = 0;
+    notmuch_status_t session_key_count_error = NOTMUCH_STATUS_SUCCESS;
+
+    if (params->crypto.decrypt == NOTMUCH_DECRYPT_TRUE)
+	session_key_count_error = notmuch_message_count_properties (message, "session-key", &session_keys);
 
     status = mime_node_open (local, message, &(params->crypto), &root);
     if (status)
@@ -880,6 +885,20 @@ show_message (void *ctx,
     part = mime_node_seek_dfs (root, (params->part < 0 ? 0 : params->part));
     if (part)
 	status = format->part (local, sp, part, indent, params);
+
+    if (params->crypto.decrypt == NOTMUCH_DECRYPT_TRUE && session_key_count_error == NOTMUCH_STATUS_SUCCESS) {
+	unsigned int new_session_keys = 0;
+	if (notmuch_message_count_properties (message, "session-key", &new_session_keys) == NOTMUCH_STATUS_SUCCESS &&
+	    new_session_keys > session_keys) {
+	    /* try a quiet re-indexing */
+	    notmuch_indexopts_t *indexopts = notmuch_database_get_default_indexopts (notmuch_message_get_database (message));
+	    if (indexopts) {
+		notmuch_indexopts_set_decrypt_policy (indexopts, NOTMUCH_DECRYPT_AUTO);
+		print_status_message ("Error re-indexing message with --decrypt=stash",
+				      message, notmuch_message_reindex (message, indexopts));
+	    }
+	}
+    }
   DONE:
     talloc_free (local);
     return status;
-- 
2.17.0

[PATCH v3 7/8] test-lib: add notmuch_show_part for "notmuch show --format=text"

[Daniel Kahn Gillmor]

Thanks to David Bremner for this improved readability!
---
 test/T357-index-decryption.sh | 10 +++++-----
 test/test-lib.sh              |  5 +++++
 2 files changed, 10 insertions(+), 5 deletions(-)

diff --git a/test/T357-index-decryption.sh b/test/T357-index-decryption.sh
index 2b8e05b8..ad6c3616 100755
--- a/test/T357-index-decryption.sh
+++ b/test/T357-index-decryption.sh
@@ -50,7 +50,7 @@ test_expect_equal \
 
 test_begin_subtest "show the message body of the encrypted message"
 notmuch dump wumpus
-output=$(notmuch show wumpus | awk '/^\014part}/{ f=0 }; { if (f) { print $0 } } /^\014part{ ID: 3/{ f=1 }')
+output=$(notmuch show wumpus | notmuch_show_part 3)
 expected='This is a test encrypted message with a wumpus.'
 if [ $NOTMUCH_HAVE_GMIME_SESSION_KEYS -eq 0 ]; then
     test_subtest_known_broken
@@ -198,14 +198,14 @@ test_expect_equal \
     "$expected"
 
 test_begin_subtest "show one of the messages with --decrypt=true"
-output=$(notmuch show --decrypt=true thread:0000000000000001 | awk '/^\014part}/{ f=0 }; { if (f) { print $0 } } /^\014part{ ID: 3/{ f=1 }')
+output=$(notmuch show --decrypt=true thread:0000000000000001 | notmuch_show_part 3)
 expected='This is a test encrypted message with a wumpus.'
 test_expect_equal \
     "$output" \
     "$expected"
 
 test_begin_subtest "Ensure that we cannot show the message with --decrypt=auto"
-output=$(notmuch show thread:0000000000000001 | awk '/^\014part}/{ f=0 }; { if (f) { print $0 } } /^\014part{ ID: 3/{ f=1 }')
+output=$(notmuch show thread:0000000000000001 | notmuch_show_part 3)
 expected='Non-text part: application/octet-stream'
 test_expect_equal \
     "$output" \
@@ -256,7 +256,7 @@ test_expect_equal \
     "$expected"
 
 test_begin_subtest "notmuch show should show cleartext if session key is present"
-output=$(notmuch show id:simple-encrypted@crypto.notmuchmail.org | awk '/^\014part}/{ f=0 }; { if (f) { print $0 } } /^\014part{ ID: 3/{ f=1 }')
+output=$(notmuch show id:simple-encrypted@crypto.notmuchmail.org | notmuch_show_part 3)
 expected='This is a top sekrit message.'
 if [ $NOTMUCH_HAVE_GMIME_SESSION_KEYS -eq 0 ]; then
     test_subtest_known_broken
@@ -266,7 +266,7 @@ test_expect_equal \
     "$expected"
 
 test_begin_subtest "notmuch show should show nothing if decryption is explicitly disallowed"
-output=$(notmuch show --decrypt=false id:simple-encrypted@crypto.notmuchmail.org | awk '/^\014part}/{ f=0 }; { if (f) { print $0 } } /^\014part{ ID: 3/{ f=1 }')
+output=$(notmuch show --decrypt=false id:simple-encrypted@crypto.notmuchmail.org | notmuch_show_part 3)
 expected='Non-text part: application/octet-stream'
 test_expect_equal \
     "$output" \
diff --git a/test/test-lib.sh b/test/test-lib.sh
index 7e064021..fca5277d 100644
--- a/test/test-lib.sh
+++ b/test/test-lib.sh
@@ -621,6 +621,11 @@ notmuch_config_sanitize ()
     notmuch_dir_sanitize | notmuch_built_with_sanitize
 }
 
+notmuch_show_part ()
+{
+    awk '/^\014part}/{ f=0 }; { if (f) { print $0 } } /^\014part{ ID: '"$1"'/{ f=1 }'
+}
+
 # End of notmuch helper functions
 
 # Use test_set_prereq to tell that a particular prerequisite is available.
-- 
2.17.0

[PATCH v3 8/8] cli/show: enable --decrypt=stash

[Daniel Kahn Gillmor]

Add fancy new feature, which makes "notmuch show" capable of actually
indexing messages that it just decrypted.

This enables a workflow where messages can come in in the background
and be indexed using "--decrypt=auto".  But when showing an encrypted
message for the first time, it gets automatically indexed.

This is something of a departure for "notmuch show" -- in particular,
because it requires read/write access to the database.  However, this
might be a common use case -- people get mail delivered and indexed in
the background, but only want access to their secret key to happen
when they're directly interacting with notmuch itself.

In such a scenario, they couldn't search newly-delivered, encrypted
messages, but they could search for them once they've read them.

Documentation of this new feature also uses a table form, similar to
that found in the description of index.decrypt in notmuch-config(1).

A notmuch UI that wants to facilitate this workflow while also
offering an interactive search interface might instead make use of
these additional commands while the user is at the console:

Count received encrypted messages (if > 0, there are some things we
haven't yet tried to index, and therefore can't yet search):

     notmuch count tag:encrypted and \
         not property:index.decryption=success and \
         not property:index.decryption=failure

Reindex those messages:

     notmuch reindex --try-decrypt=true tag:encrypted and \
         not property:index.decryption=success and \
         not property:index.decryption=failure
---
 completion/notmuch-completion.bash |  2 +-
 doc/man1/notmuch-show.rst          | 40 +++++++++++++++++++++++++-----
 notmuch-show.c                     |  9 +++++--
 test/T357-index-decryption.sh      | 18 ++++++++++++++
 4 files changed, 60 insertions(+), 9 deletions(-)

diff --git a/completion/notmuch-completion.bash b/completion/notmuch-completion.bash
index 249b9664..15425697 100644
--- a/completion/notmuch-completion.bash
+++ b/completion/notmuch-completion.bash
@@ -522,7 +522,7 @@ _notmuch_show()
 	    return
 	    ;;
         --decrypt)
-	    COMPREPLY=( $( compgen -W "true auto false" -- "${cur}" ) )
+	    COMPREPLY=( $( compgen -W "true auto false stash" -- "${cur}" ) )
 	    return
 	    ;;
     esac
diff --git a/doc/man1/notmuch-show.rst b/doc/man1/notmuch-show.rst
index 2b825ccc..b2667537 100644
--- a/doc/man1/notmuch-show.rst
+++ b/doc/man1/notmuch-show.rst
@@ -110,7 +110,7 @@ Supported options for **show** include
     supported with --format=json and --format=sexp), and the
     multipart/signed part will be replaced by the signed data.
 
-``--decrypt=(false|auto|true)``
+``--decrypt=(false|auto|true|stash)``
     If ``true``, decrypt any MIME encrypted parts found in the
     selected content (i.e. "multipart/encrypted" parts). Status of
     the decryption will be reported (currently only supported
@@ -118,17 +118,45 @@ Supported options for **show** include
     decryption the multipart/encrypted part will be replaced by
     the decrypted content.
 
+    ``stash`` behaves like ``true``, but upon successful decryption it
+    will also stash the message's session key in the database, and
+    index the cleartext of the message, enabling automatic decryption
+    in the future.
+
     If ``auto``, and a session key is already known for the
     message, then it will be decrypted, but notmuch will not try
     to access the user's keys.
 
     Use ``false`` to avoid even automatic decryption.
 
-    Non-automatic decryption expects a functioning
-    **gpg-agent(1)** to provide any needed credentials. Without
-    one, the decryption will fail.
-
-    Note: ``true`` implies --verify.
+    Non-automatic decryption (``stash`` or ``true``, in the absence of
+    a stashed session key) expects a functioning **gpg-agent(1)** to
+    provide any needed credentials. Without one, the decryption will
+    fail.
+
+    Note: setting either ``true`` or ``stash`` here implies
+    ``--verify``.
+
+    Here is a table that summarizes each of these policies:
+
+    +------------------------+-------+------+------+-------+
+    |                        | false | auto | true | stash |
+    +========================+=======+======+======+=======+
+    | Show cleartext if      |       |  X   |  X   |   X   |
+    | session key is         |       |      |      |       |
+    | already known          |       |      |      |       |
+    +------------------------+-------+------+------+-------+
+    | Use secret keys to     |       |      |  X   |   X   |
+    | show cleartext         |       |      |      |       |
+    +------------------------+-------+------+------+-------+
+    | Stash any newly        |       |      |      |   X   |
+    | recovered session keys,|       |      |      |       |
+    | reindexing message if  |       |      |      |       |
+    | found                  |       |      |      |       |
+    +------------------------+-------+------+------+-------+
+
+    Note: ``--decrypt=stash`` requires write access to the database.
+    Otherwise, ``notmuch show`` operates entirely in read-only mode.
 
     Default: ``auto``
 
diff --git a/notmuch-show.c b/notmuch-show.c
index f34cf941..be3e4b50 100644
--- a/notmuch-show.c
+++ b/notmuch-show.c
@@ -1123,6 +1123,7 @@ notmuch_show_command (notmuch_config_t *config, int argc, char *argv[])
 	  (notmuch_keyword_t []){ { "false", NOTMUCH_DECRYPT_FALSE },
 				  { "auto", NOTMUCH_DECRYPT_AUTO },
 				  { "true", NOTMUCH_DECRYPT_NOSTASH },
+				  { "stash", NOTMUCH_DECRYPT_TRUE },
 				  { 0, 0 } } },
 	{ .opt_bool = &params.crypto.verify, .name = "verify" },
 	{ .opt_bool = &params.output_body, .name = "body" },
@@ -1138,7 +1139,8 @@ notmuch_show_command (notmuch_config_t *config, int argc, char *argv[])
     notmuch_process_shared_options (argv[0]);
 
     /* explicit decryption implies verification */
-    if (params.crypto.decrypt == NOTMUCH_DECRYPT_NOSTASH)
+    if (params.crypto.decrypt == NOTMUCH_DECRYPT_NOSTASH ||
+	params.crypto.decrypt == NOTMUCH_DECRYPT_TRUE)
 	params.crypto.verify = true;
 
     /* specifying a part implies single message display */
@@ -1201,8 +1203,11 @@ notmuch_show_command (notmuch_config_t *config, int argc, char *argv[])
     params.crypto.gpgpath = notmuch_config_get_crypto_gpg_path (config);
 #endif
 
+    notmuch_database_mode_t mode = NOTMUCH_DATABASE_MODE_READ_ONLY;
+    if (params.crypto.decrypt == NOTMUCH_DECRYPT_TRUE)
+	mode = NOTMUCH_DATABASE_MODE_READ_WRITE;
     if (notmuch_database_open (notmuch_config_get_database_path (config),
-			       NOTMUCH_DATABASE_MODE_READ_ONLY, &notmuch))
+			       mode, &notmuch))
 	return EXIT_FAILURE;
 
     notmuch_exit_if_unmatched_db_uuid (notmuch);
diff --git a/test/T357-index-decryption.sh b/test/T357-index-decryption.sh
index ad6c3616..c5435f4f 100755
--- a/test/T357-index-decryption.sh
+++ b/test/T357-index-decryption.sh
@@ -80,6 +80,24 @@ test_expect_equal \
     "$output" \
     "$expected"
 
+# show the message using stashing decryption
+test_begin_subtest "stash decryption during show"
+output=$(notmuch show --decrypt=stash tag:encrypted subject:002 | notmuch_show_part 3)
+expected='This is a test encrypted message with a wumpus.'
+test_expect_equal \
+    "$output" \
+    "$expected"
+
+test_begin_subtest "search should now find the contents"
+output=$(notmuch search wumpus)
+expected='thread:0000000000000003   2000-01-01 [1/1] Notmuch Test Suite; test encrypted message for cleartext index 002 (encrypted inbox unread)'
+if [ $NOTMUCH_HAVE_GMIME_SESSION_KEYS -eq 0 ]; then
+    test_subtest_known_broken
+fi
+test_expect_equal \
+    "$output" \
+    "$expected"
+
 # try reinserting it with decryption, should appear again, but now we
 # have two copies of the message:
 test_begin_subtest "message cleartext is present after reinserting with --decrypt=true"
-- 
2.17.0

Re: [PATCH v3 1/8] lib: expose notmuch_message_get_database()

[David Bremner]

Daniel Kahn Gillmor <dkg@fifthhorseman.net> writes:

> +/**
> + * Get the database associated with this message.
> + *
> + * @since libnotmuch 5.2 (notmuch 0.27)
> + */
> +notmuch_database_t *
> +notmuch_message_get_database (notmuch_message_t *message);
> +

Let's make sure we actually bump the library minor version this time.

d

Re: [PATCH v3 6/8] cli/show: reindex when we learned new session keys about a message

[David Bremner]

Daniel Kahn Gillmor <dkg@fifthhorseman.net> writes:

> +
> +    if (params->crypto.decrypt == NOTMUCH_DECRYPT_TRUE && session_key_count_error == NOTMUCH_STATUS_SUCCESS) {
> +	unsigned int new_session_keys = 0;
> +	if (notmuch_message_count_properties (message, "session-key", &new_session_keys) == NOTMUCH_STATUS_SUCCESS &&
> +	    new_session_keys > session_keys) {
> +	    /* try a quiet re-indexing */
> +	    notmuch_indexopts_t *indexopts = notmuch_database_get_default_indexopts (notmuch_message_get_database (message));
> +	    if (indexopts) {
> +		notmuch_indexopts_set_decrypt_policy (indexopts, NOTMUCH_DECRYPT_AUTO);
> +		print_status_message ("Error re-indexing message with --decrypt=stash",
> +				      message, notmuch_message_reindex (message, indexopts));
> +	    }
> +	}
> +    }

This message is just to remind me to #if guard this block when applying
the patch.

d

[PATCH] lib: bump minor version

[David Bremner]

This recognizes the addition of (at least)
notmuch_message_get_database to the API.
---
 lib/notmuch.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lib/notmuch.h b/lib/notmuch.h
index 4db28a05..b1bfcb0f 100644
--- a/lib/notmuch.h
+++ b/lib/notmuch.h
@@ -58,7 +58,7 @@ NOTMUCH_BEGIN_DECLS
  * version in Makefile.local.
  */
 #define LIBNOTMUCH_MAJOR_VERSION	5
-#define LIBNOTMUCH_MINOR_VERSION	1
+#define LIBNOTMUCH_MINOR_VERSION	2
 #define LIBNOTMUCH_MICRO_VERSION	0
 
 
-- 
2.17.0

Re: [PATCH] lib: bump minor version

[Daniel Kahn Gillmor]

On Tue 2018-05-22 18:47:49 -0700, David Bremner wrote:
> This recognizes the addition of (at least)
> notmuch_message_get_database to the API.
> ---
>  lib/notmuch.h | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/lib/notmuch.h b/lib/notmuch.h
> index 4db28a05..b1bfcb0f 100644
> --- a/lib/notmuch.h
> +++ b/lib/notmuch.h
> @@ -58,7 +58,7 @@ NOTMUCH_BEGIN_DECLS
>   * version in Makefile.local.
>   */
>  #define LIBNOTMUCH_MAJOR_VERSION	5
> -#define LIBNOTMUCH_MINOR_VERSION	1
> +#define LIBNOTMUCH_MINOR_VERSION	2
>  #define LIBNOTMUCH_MICRO_VERSION	0

yes, thank you for catching this!

     --dkg

Re: [PATCH v3 1/8] lib: expose notmuch_message_get_database()

[David Bremner]

Daniel Kahn Gillmor <dkg@fifthhorseman.net> writes:

> We've had _notmuch_message_database() internally for a while, and it's
> useful.  It turns out to be useful on the other side of the library
> interface as well (i'll use it later in this series for "notmuch
> show"), so we expose it publicly now.

series pushed to master, along with my addon patch