From: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
To: Notmuch Mail <notmuch@notmuchmail.org>
Subject: allow indexing cleartext of encrypted messages
Date: Wed, 9 Dec 2015 22:39:37 -0500 [thread overview]
Message-ID: <1449718786-28000-1-git-send-email-dkg@fifthhorseman.net> (raw)
Notmuch currently doesn't index the cleartext of encrypted mail. This
is the right choice by default, because the index is basically
cleartext-equivalent, and we wouldn't want every indexed mailstore to
leak the contents of its encrypted mails.
However, if a notmuch user has their index in a protected location,
they may prefer the convenience of being able to search the contents
of (at least some of) their encrypted mail.
This series of patches enables notmuch to index the cleartext of
specific encrypted messages when they're being added via "notmuch new"
or "notmuch insert", via a new --try-decrypt flag.
If --try-decrypt is used, and decryption is successful for part of a
message, the message gets an additional "index-decrypted" tag. If
decryption of part of a message fails, the message gets an additional
"index-decryption-failed" tag.
This tagging approach should allow people to figure out which messages
have been indexed in the clear (or not), and can be used to
selectively reindex them in batch with something like:
----------------
#!/usr/bin/env python3
'''notmuch-reindex.py -- a quick and dirty pythonic mechanism to
re-index specific messages in a notmuch database. This should
probably be properly implemented as a subcommand for /usr/bin/notmuch
itself'''
import notmuch
import sys
d = notmuch.Database(mode=notmuch.Database.MODE.READ_WRITE)
query = sys.argv[1]
q = d.create_query(query)
for m in q.search_messages():
mainfilename = m.get_filename()
origtags = m.get_tags()
tags = []
for t in origtags:
if t not in ['index-decrypted', 'index-decryption-failed']:
tags += [t]
d.begin_atomic()
for f in m.get_filenames():
d.remove_message(f)
(newm,stat) = d.add_message(mainfilename, try_decrypt=True)
for tag in tags:
newm.add_tag(tag)
d.end_atomic()
----------------
A couple key points:
* There is some code duplication between crypto.c (for the
notmuch-client) and lib/database.cc and lib/index.cc (for the
library) because both parts of the codebase use gmime to handle the
decryption. I don't want to contaminate the libnotmuch API with
gmime implementation details, so i don't quite see how to reuse the
code cleanly. I'd love suggestions on how to reduce the
duplications.
* the libnotmuch API is extended with
notmuch_database_add_message_try_decrypt(). This should probably
ultimately be more general, because there are a few additional
knobs that i can imagine fiddling at indexing time. For example:
* verifying cryptographic signatures and storing something about
those verifications in the notmuch db
* extracting OpenPGP session key information for a given message
and storing it in a lookaside table in the notmuch db, so that
it's possible to securely destroy old encryption-capable keys
and still have local access to the cleartext of the remaining
messages.
Some of these additional features might be orthogonal to one
another as well. I welcome suggestions for how to improve the API
so that we don't end up with a combinatorial explosion of
n_d_add_message_foo() functions.
* To properly complete this patch series, i think i want to make
notmuch-reindex.c and add a reindex subcommand, also with a
--try-decrypt option. It's not clear to me if the right approach
for that is to have a C implementation of the python script above
without modifying libnotmuch, or if i should start by creating a
notmuch_message_reindex function in libnotmuch, with a try_decrypt
flag. Again, suggestions welcome.
* Is the tagging approach the right thing to do to record success or
failure of decryption at index time? Is there a better approach?
next reply other threads:[~2015-12-10 3:40 UTC|newest]
Thread overview: 28+ messages / expand[flat|nested] mbox.gz Atom feed top
2015-12-10 3:39 Daniel Kahn Gillmor [this message]
2015-12-10 3:39 ` [PATCH 1/9] reorganize indexing of multipart/signed and multipart/encrypted Daniel Kahn Gillmor
2015-12-11 21:53 ` Tomi Ollila
2015-12-10 3:39 ` [PATCH 2/9] Add a lazily-initialized crypto context to notmuch_database_t Daniel Kahn Gillmor
2015-12-11 14:03 ` David Bremner
2015-12-11 14:36 ` Daniel Kahn Gillmor
2015-12-11 21:55 ` Tomi Ollila
2015-12-10 3:39 ` [PATCH 3/9] index encrypted parts when the message is flagged appropriately Daniel Kahn Gillmor
2015-12-10 3:39 ` [PATCH 4/9] Add new n_d_add_message_try_decrypt (analogous to to n_d_add_message) Daniel Kahn Gillmor
2015-12-10 3:39 ` [PATCH 5/9] Enable cleartext indexing in python bindings Daniel Kahn Gillmor
2015-12-10 3:39 ` [PATCH 6/9] search for a reasonable gpg implementation Daniel Kahn Gillmor
2015-12-11 21:56 ` Tomi Ollila
2015-12-11 22:18 ` J. Lewis Muir
2015-12-11 22:22 ` Daniel Kahn Gillmor
2015-12-10 3:39 ` [PATCH 7/9] add a gpg_path value for notmuch_database_t Daniel Kahn Gillmor
2015-12-11 22:02 ` Tomi Ollila
2015-12-11 22:25 ` Daniel Kahn Gillmor
2015-12-12 23:25 ` Tomi Ollila
2015-12-13 1:20 ` David Bremner
2015-12-13 11:00 ` Tomi Ollila
2015-12-13 11:17 ` Tomi Ollila
2016-01-15 19:11 ` Daniel Kahn Gillmor
2015-12-11 22:35 ` J. Lewis Muir
2015-12-12 4:10 ` Daniel Kahn Gillmor
2015-12-10 3:39 ` [PATCH 8/9] add --try-decrypt to notmuch insert Daniel Kahn Gillmor
2015-12-10 3:39 ` [PATCH 9/9] add --try-decrypt to notmuch new Daniel Kahn Gillmor
2015-12-11 15:34 ` allow indexing cleartext of encrypted messages Daniel Kahn Gillmor
2015-12-11 22:05 ` Tomi Ollila
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: https://notmuchmail.org/
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1449718786-28000-1-git-send-email-dkg@fifthhorseman.net \
--to=dkg@fifthhorseman.net \
--cc=notmuch@notmuchmail.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://yhetil.org/notmuch.git/
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).