Feature suggestion. Indexing encrypted mail?

Feature suggestion. Indexing encrypted mail?

[john.wyzer]

Hello!

Would it be possible to add the configurable option to also decrypt
encrypted messages on the fly while indexing to make them searchable,
too?

That would be really great for people that consider gnupg  mainly an
encryption for transport or have their complete hard drive encrypted...

Best regards!
John

Re: Feature suggestion. Indexing encrypted mail?

[David Bremner]

john.wyzer@gmx.de writes:

> Would it be possible to add the configurable option to also decrypt
> encrypted messages on the fly while indexing to make them searchable,
> too?
>
> That would be really great for people that consider gnupg  mainly an
> encryption for transport or have their complete hard drive encrypted...

As far I understand an attacker could reconstruct the message from the
index, so one question is whether the extra complexity in notmuch is
worth the minimal extra security over decrypting on delivery and storing
plaintext on the (presumably encrypted) disk. Of course decrypting on
delivery may be inconvenient (or impossible). I have CCed the two people
who have implemented most of the crypto related stuff in notmuch so they
can comment.

d

Re: Feature suggestion. Indexing encrypted mail?

[Jeremy Nickurak]

[-- Attachment #1: Type: text/plain, Size: 1181 bytes --]

Off the top of my head, you could have an encrypted index too, which you
can only search while able to decrypt. Certainly another level of
complexity.


On Sat, Apr 5, 2014 at 11:10 AM, David Bremner <david@tethera.net> wrote:

> john.wyzer@gmx.de writes:
>
> > Would it be possible to add the configurable option to also decrypt
> > encrypted messages on the fly while indexing to make them searchable,
> > too?
> >
> > That would be really great for people that consider gnupg  mainly an
> > encryption for transport or have their complete hard drive encrypted...
>
> As far I understand an attacker could reconstruct the message from the
> index, so one question is whether the extra complexity in notmuch is
> worth the minimal extra security over decrypting on delivery and storing
> plaintext on the (presumably encrypted) disk. Of course decrypting on
> delivery may be inconvenient (or impossible). I have CCed the two people
> who have implemented most of the crypto related stuff in notmuch so they
> can comment.
>
> d
> _______________________________________________
> notmuch mailing list
> notmuch@notmuchmail.org
> http://notmuchmail.org/mailman/listinfo/notmuch
>

[-- Attachment #2: Type: text/html, Size: 1796 bytes --]

Re: Feature suggestion. Indexing encrypted mail?

[john.wyzer]

Jeremy Nickurak <not-much@trk.nickurak.ca> writes:

> Off the top of my head, you could have an encrypted index too, which you
> can only search while able to decrypt. Certainly another level of
> complexity.
>

But why add so much complexity? 

If a user decides that either transport security is enough or
additionally the hard disk is encrypted (why store an encrypted index on
an encrypted hard disk?), said user could just switch on an option in
the notmuch configuration that causes notmuch to ask for the password
before or while indexing new messages and to add decrypted messages to the
normal index as well.


The level of security would be up to the user by means of said
configuration option and those that want the convenience of searching
encrypted messages could have it.

Personally I would argue that if an attacker has the means to access the
content of my hard disk either via the network or physically, there is
no difference between having whole disk encryption and storing an
encrypted index...

Re: Feature suggestion. Indexing encrypted mail?

[Jameson Graef Rollins]

[-- Attachment #1: Type: text/plain, Size: 2009 bytes --]

On Sat, Apr 05 2014, David Bremner <david@tethera.net> wrote:
> john.wyzer@gmx.de writes:
>
>> Would it be possible to add the configurable option to also decrypt
>> encrypted messages on the fly while indexing to make them searchable,
>> too?
>>
>> That would be really great for people that consider gnupg  mainly an
>> encryption for transport or have their complete hard drive encrypted...
>
> As far I understand an attacker could reconstruct the message from the
> index, so one question is whether the extra complexity in notmuch is
> worth the minimal extra security over decrypting on delivery and storing
> plaintext on the (presumably encrypted) disk. Of course decrypting on
> delivery may be inconvenient (or impossible). I have CCed the two people
> who have implemented most of the crypto related stuff in notmuch so they
> can comment.

Indexing encrypted email is a bit of a foot-gun, since, as David
mentions, it is apparently possible to reconstruct encrypted messages
From the index.  It therefore needs to be approached with care.

I think decrypting on "delivery" (or mail fetch or whatever) sounds
difficult and unwieldy.  In either event, it seems out of the scope of
notmuch.  If a user figured out how to have that done, no changes to
notmuch would be needed afaict.
 
I don't think it would be difficult modify notmuch new to decrypt at
indexing time.  Given that gnupg agent would be used for accessing the
users private key for decryption, the interface would be fairly
straightforward.

A couple of decryption options could be added to notmuch new:

* don't decrypt: don't attempt to decrypt and index any encrypted
  message (default)

* decrypt always: fail if any encrypted message could not be decrypted

* decrypt opportunistically: attempt to decrypt, but continue indexing
  if an encrypted message could not be decrypted

If something like this is enabled, we should make sure we make the
dangers clear to the users.

jamie.

[-- Attachment #2: Type: application/pgp-signature, Size: 818 bytes --]

Re: Feature suggestion. Indexing encrypted mail?

[Guyzmo]

Hi!

On Sat, Apr 05, 2014 at 12:09:32PM -0700, Jameson Graef Rollins wrote:
> On Sat, Apr 05 2014, David Bremner <david@tethera.net> wrote:
> > john.wyzer@gmx.de writes:
> >> Would it be possible to add the configurable option to also decrypt
> >> encrypted messages on the fly while indexing to make them searchable,
> >> too?
> > As far I understand an attacker could reconstruct the message from the
> > index, so one question is whether the extra complexity in notmuch is
> > worth the minimal extra security over decrypting on delivery and storing
> > plaintext on the (presumably encrypted) disk. Of course decrypting on
> > delivery may be inconvenient (or impossible). I have CCed the two people
> > who have implemented most of the crypto related stuff in notmuch so they
> > can comment.
> Indexing encrypted email is a bit of a foot-gun, since, as David
> mentions, it is apparently possible to reconstruct encrypted messages
> From the index.  It therefore needs to be approached with care.
> 
> I think decrypting on "delivery" (or mail fetch or whatever) sounds
> difficult and unwieldy.  In either event, it seems out of the scope of
> notmuch.  If a user figured out how to have that done, no changes to
> notmuch would be needed afaict.
[…]

    I indeed agree with this view, and I think the best process would be
to have the MUA decrypt and index an encrypted mail when the  user wants
it to be indexed. So the user do not get really  highly  secret messages
disclosable by the index, and for the others take that kind of risk.

    That way you wouldn't need to keep the secret in  the  gpg-agent for
too long, and/or need a password for an automated process.

my two cents,

-- 
Guyzmo

Re: Feature suggestion. Indexing encrypted mail?

[Daniel Kahn Gillmor]

[-- Attachment #1: Type: text/plain, Size: 2878 bytes --]

On 04/06/2014 05:15 AM, Guyzmo wrote:
>     I indeed agree with this view, and I think the best process would be
> to have the MUA decrypt and index an encrypted mail when the  user wants
> it to be indexed. So the user do not get really  highly  secret messages
> disclosable by the index, and for the others take that kind of risk.

At the moment, notmuch has a "no-modify" policy to the mail storage,
with the exception of changing a few well-known flags on maildir names.

I would be pretty sad to see that change, and i don't think that's a
good idea for notmuch in general.  let's keep access to the mail store
as read-only as possible.

additionally, stripping encryption in some cases would mean stripping
cryptographic signatures (e.g. most PGP/MIME encrypted messages are
encrypted+signed, but the signature is a separate PGP part and not a
MIME part) i think it would be bad to lose cryptographic signatures in
this case.

That said, i agree that there are some scenarios where having
well-indexed mail storage even for the cleartext of encrypted messages
would be useful and could even be done with some level of safety (e.g.
where the index is itself stored on an encrypted filesystem -- notmuch
has no explicit/builtin support for an encrypted index today).

I think the most sensible way to approach this goal for notmuch would be
a two-part series of generic notmuch enhancements, which could then be
leveraged by those who need them into a cleartext index for those
messages that they are willing to take a risk on.

here are the notmuch enhancements:

 * notmuch new id:$msgid

This capability would allow notmuch to reindex a given message, clearing
the entire index of any old references and adding new references to the
current filter.

 * notmuch new --filter=$foo

The --filter option for notmuch new (or something similar) would  pass
each message in question through a pipeline-style filter and operate on
it the stdout of the filter, rather than the raw message.

Given these two enhancements (some of which may be already underway, i
confess i haven't been following closely), it wouldn't be much extra
effort for someone to implement a filter that strips encryption from the
message.  (this might still have the problem mentioned above about also
stripping PGP/MIME signatures, but the signatures and the decrypted
message itself would remain intact so they could be shown directly by
notmuch show without trouble).

once such a filter was in place, my personal preference would be that
the messages would be imported as ciphertext initially, but then when an
end-user-facing MUA gets the user to decrypt a message that has not been
indexed with this filter, it could offer a button that says "index this
message in cleartext (will leak contents to anyone who can read the index)"

	--dkg


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 1010 bytes --]

Re: Feature suggestion. Indexing encrypted mail?

[john.wyzer]

Daniel Kahn Gillmor <dkg@fifthhorseman.net> writes:

> At the moment, notmuch has a "no-modify" policy to the mail storage,
> with the exception of changing a few well-known flags on maildir names.
>
> I would be pretty sad to see that change, and i don't think that's a
> good idea for notmuch in general.  let's keep access to the mail store
> as read-only as possible.
>
> additionally, stripping encryption in some cases would mean stripping
> cryptographic signatures (e.g. most PGP/MIME encrypted messages are
> encrypted+signed, but the signature is a separate PGP part and not a
> MIME part) i think it would be bad to lose cryptographic signatures in
> this case.

I would never have meant to suggest to change that. With decrypting
"on-the-fly" I tried to suggest the decryption for the sake of indexing
- but only during runtime and without changing the mail storage.

>
>  * notmuch new --filter=$foo
>
> The --filter option for notmuch new (or something similar) would  pass
> each message in question through a pipeline-style filter and operate on
> it the stdout of the filter, rather than the raw message.

That idea sounds very nice to me and would make reindexing with other
filters easy if needed.

> confess i haven't been following closely), it wouldn't be much extra
> effort for someone to implement a filter that strips encryption from the
> message.  (this might still have the problem mentioned above about also
> stripping PGP/MIME signatures, but the signatures and the decrypted
> message itself would remain intact so they could be shown directly by
> notmuch show without trouble).

I don't understand that. :-(
This sounds as if the view of the message is not generated from the
mail storage. Isn't the purpose of the index to find the appropriate
message file and everything else is generated from that file?

Re: Feature suggestion. Indexing encrypted mail?

[Jameson Graef Rollins]

[-- Attachment #1: Type: text/plain, Size: 808 bytes --]

On Mon, Apr 07 2014, john.wyzer@gmx.de wrote:
>> confess i haven't been following closely), it wouldn't be much extra
>> effort for someone to implement a filter that strips encryption from the
>> message.  (this might still have the problem mentioned above about also
>> stripping PGP/MIME signatures, but the signatures and the decrypted
>> message itself would remain intact so they could be shown directly by
>> notmuch show without trouble).
>
> I don't understand that. :-(
> This sounds as if the view of the message is not generated from the
> mail storage. Isn't the purpose of the index to find the appropriate
> message file and everything else is generated from that file?

I think that's exactly what Daniel is saying: what's viewed comes from
the message directly, and not from the db.

jamie.

[-- Attachment #2: Type: application/pgp-signature, Size: 818 bytes --]

Re: Feature suggestion. Indexing encrypted mail?

[Jeremy Nickurak]

[-- Attachment #1: Type: text/plain, Size: 1300 bytes --]

Nonetheess, if you can tell from the index that a given message contains
the words "hotel" "wine" "wife" "secret" and "rendezvous", you can infer a
*lot* about the contents of encrypted contents of the message.


On Mon, Apr 7, 2014 at 9:57 AM, Jameson Graef Rollins <
jrollins@finestructure.net> wrote:

> On Mon, Apr 07 2014, john.wyzer@gmx.de wrote:
> >> confess i haven't been following closely), it wouldn't be much extra
> >> effort for someone to implement a filter that strips encryption from the
> >> message.  (this might still have the problem mentioned above about also
> >> stripping PGP/MIME signatures, but the signatures and the decrypted
> >> message itself would remain intact so they could be shown directly by
> >> notmuch show without trouble).
> >
> > I don't understand that. :-(
> > This sounds as if the view of the message is not generated from the
> > mail storage. Isn't the purpose of the index to find the appropriate
> > message file and everything else is generated from that file?
>
> I think that's exactly what Daniel is saying: what's viewed comes from
> the message directly, and not from the db.
>
> jamie.
>
> _______________________________________________
> notmuch mailing list
> notmuch@notmuchmail.org
> http://notmuchmail.org/mailman/listinfo/notmuch
>
>

[-- Attachment #2: Type: text/html, Size: 1982 bytes --]

Re: Feature suggestion. Indexing encrypted mail?

[Jameson Graef Rollins]

[-- Attachment #1: Type: text/plain, Size: 565 bytes --]

On Mon, Apr 07 2014, Jeremy Nickurak <not-much@trk.nickurak.ca> wrote:
> Nonetheess, if you can tell from the index that a given message contains
> the words "hotel" "wine" "wife" "secret" and "rendezvous", you can infer a
> *lot* about the contents of encrypted contents of the message.

Of course.  Given that the content of the message will be stored
unencrypted in the db, indexing encrypted messages is potentially a foot
gun.  If we were to enable a feature like this, it would definitely be a
user-beware situation.  I don't see any way around that.

jamie.

[-- Attachment #2: Type: application/pgp-signature, Size: 818 bytes --]

Re: Feature suggestion. Indexing encrypted mail?

[Mark Walters]

On Mon, 07 Apr 2014, Jeremy Nickurak <not-much@trk.nickurak.ca> wrote:
> Nonetheess, if you can tell from the index that a given message contains
> the words "hotel" "wine" "wife" "secret" and "rendezvous", you can infer a
> *lot* about the contents of encrypted contents of the message.

I think it is worse that that: I think (from what people said on irc
some time ago) that the index contains the word and the position of that
word so essentially the whole message can be reconstructed from the
index.

Best wishes

Mark



>
>
> On Mon, Apr 7, 2014 at 9:57 AM, Jameson Graef Rollins <
> jrollins@finestructure.net> wrote:
>
>> On Mon, Apr 07 2014, john.wyzer@gmx.de wrote:
>> >> confess i haven't been following closely), it wouldn't be much extra
>> >> effort for someone to implement a filter that strips encryption from the
>> >> message.  (this might still have the problem mentioned above about also
>> >> stripping PGP/MIME signatures, but the signatures and the decrypted
>> >> message itself would remain intact so they could be shown directly by
>> >> notmuch show without trouble).
>> >
>> > I don't understand that. :-(
>> > This sounds as if the view of the message is not generated from the
>> > mail storage. Isn't the purpose of the index to find the appropriate
>> > message file and everything else is generated from that file?
>>
>> I think that's exactly what Daniel is saying: what's viewed comes from
>> the message directly, and not from the db.
>>
>> jamie.
>>
>> _______________________________________________
>> notmuch mailing list
>> notmuch@notmuchmail.org
>> http://notmuchmail.org/mailman/listinfo/notmuch
>>
>>
> _______________________________________________
> notmuch mailing list
> notmuch@notmuchmail.org
> http://notmuchmail.org/mailman/listinfo/notmuch

Re: Feature suggestion. Indexing encrypted mail?

[Daniel Kahn Gillmor]

[-- Attachment #1: Type: text/plain, Size: 1591 bytes --]

On 04/07/2014 05:06 PM, Mark Walters wrote:

> I think it is worse that that: I think (from what people said on irc
> some time ago) that the index contains the word and the position of that
> word so essentially the whole message can be reconstructed from the
> index.

Agree with Mark here, the warnings around such a feature should clearly
say "this stores a cleartext equivalent of your message in the notmuch
index."

Even if the index weren't structured in this way, modern natural
language processing techniques and a plausible training corpus should be
able to come very close to the original cleartext message, so it should
be treated as such.

fwiw, the workflow i outlined should make it so that users can receive
all messages encrypted; when they read each encrypted message, they get
a choice about whether to store a cleartext-equivalent in their notmuch
index. (note of course that it's possible to store your notmuch index on
an encrypted filesystem itself, for a different flavor of
confidentiality protection for the data once it's come to rest).

This per-message decision mechanism lets a thoughtful user make that
tradeoff on a piecemeal basis (it also allows for blanket
(mis)judgement, of course).  There are certainly some messages that one
might never want store in a cleartext index, while other messages might
be less sensitive to exposure while being more valuable to the user if
stored in a well-indexed, searchable local archive.

I think this is a feature worth having, despite the warning labels it
probably needs.

	--dkg


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 1010 bytes --]