[PATCH] NEWS: cleartext indexing

[PATCH] NEWS: cleartext indexing

[Daniel Kahn Gillmor]

give a short summary of cleartext indexing of encrypted e-mail
messages.
---
 NEWS | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)

diff --git a/NEWS b/NEWS
index c2ebb644..895b7627 100644
--- a/NEWS
+++ b/NEWS
@@ -11,6 +11,24 @@ Out-of-tree builds
   has been dropped. The same can now be achieved more reliably using
   out-of-tree builds.
 
+Encrypted Mail
+--------------
+
+Indexing cleartext of encrypted e-mails
+
+  It's now possible to include the cleartext of encrypted e-mails in
+  the notmuch index.  This makes it possible to search your encrypted
+  e-mails with the same ease as searching cleartext.  This can be done
+  on a per-message basis with the --try-decrypt argument to indexing
+  commands (new, insert, reindex), or by default by running "notmuch
+  config set index.try_decrypt true".
+
+  Note that the contents of the index are sufficient to roughly
+  reconstruct the cleartext of the message itself, so please ensure
+  that the notmuch index itself is adequately protected.  DO NOT USE
+  this feature without considering the security of your index.
+
+
 Notmuch 0.25.1 (2017-09-11)
 ===========================
 
-- 
2.14.2

Re: [PATCH] NEWS: cleartext indexing

[David Bremner]

Daniel Kahn Gillmor <dkg@fifthhorseman.net> writes:

> give a short summary of cleartext indexing of encrypted e-mail
> messages.

That's fine and I've pushed it, but as written reindex will need it's
own NEWS item.

We also need to decide if the instability of subjects under reindex (see
one of the currently broken tests) is worth delaying a release for.

d

[PATCH] NEWS: document notmuch reindex

[Daniel Kahn Gillmor]

Describe the introduction of the "reindex" subcommand.  This blurb
acknowledges Subject: instability under reindexing when multiple
copies exist (suggesting that this is something that needn't
necessarily hold up a release).
---
 NEWS | 13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/NEWS b/NEWS
index 895b7627..27d75e3f 100644
--- a/NEWS
+++ b/NEWS
@@ -11,6 +11,19 @@ Out-of-tree builds
   has been dropped. The same can now be achieved more reliably using
   out-of-tree builds.
 
+Command Line Interface
+----------------------
+
+Support for re-indexing existing messages
+
+  There is a new subcommand, `notmuch reindex`, which re-indexes all
+  messages matching supplied search terms.  This permits users to
+  change the way specific messages are indexed.
+
+  Note that for messages with multiple variants in the message
+  archive, the recorded Subject: of may change upon reindexing,
+  depending on the order in which the variants are indexed.
+
 Encrypted Mail
 --------------
 
-- 
2.14.2

Re: [PATCH] NEWS: document notmuch reindex

[David Bremner]

Daniel Kahn Gillmor <dkg@fifthhorseman.net> writes:

> Describe the introduction of the "reindex" subcommand.  This blurb
> acknowledges Subject: instability under reindexing when multiple
> copies exist (suggesting that this is something that needn't
> necessarily hold up a release).

I hemmed and hawed about giving a more precise statement, but eventually
I decided that some general warning is good enough for now.

pushed,

d

Re: [PATCH] NEWS: cleartext indexing

[Antoine Beaupré]

On 2017-10-22 11:36:34, Daniel Kahn Gillmor wrote:
> +  Note that the contents of the index are sufficient to roughly
> +  reconstruct the cleartext of the message itself, so please ensure
> +  that the notmuch index itself is adequately protected.  DO NOT USE
> +  this feature without considering the security of your index.

Could we expand on what those security options could be? Full disk
encryption? Or is there some way to PGP-encrypt the index and have it
decrypted on the fly?

Security, in this context, seems a little broad... I do have a antsy
feeling at decrypting all my private emails in a cleartext database
without additional measures. I'd sure love to see this notion expanded
here somehow.

By the way, I have similar concerns about the autocrypt approach, which
goes even further and says private key material should not be protected
by a password at all:

http://autocrypt.readthedocs.io/en/latest/level1.html#secret-key-protection-at-rest

It would be interesting to explain the rationale around those decisions
(which autocrypt does) and possible safeguards that mitigate those
issues (which autocrypt doesn't).

Thanks!

A.

-- 
À mesure que l'opression s'étend à tous les secteurs de la vie,
la révolte prend l'allure d'une guerre sociale. 
Les émeutes renaissent et annoncent la révolution à venir.
                        - Jean-François Brient, de la servitude moderne

Re: [PATCH] NEWS: cleartext indexing

[Daniel Kahn Gillmor]

[-- Attachment #1: Type: text/plain, Size: 4118 bytes --]

On Mon 2017-10-30 08:46:12 -0400, Antoine Beaupré wrote:
> On 2017-10-22 11:36:34, Daniel Kahn Gillmor wrote:
>> +  Note that the contents of the index are sufficient to roughly
>> +  reconstruct the cleartext of the message itself, so please ensure
>> +  that the notmuch index itself is adequately protected.  DO NOT USE
>> +  this feature without considering the security of your index.
>
> Could we expand on what those security options could be? Full disk
> encryption? Or is there some way to PGP-encrypt the index and have it
> decrypted on the fly?

This is deliberately out of scope of the series -- i don't want this
series to introduce notmuch-specific index encryption. Though if someone
wanted to propose that i'd be happy to review it; fwiw, i have looked at
a few deterministic mail index encryption schemes, and i'm not convinced
that they meet any realistic threat model.

But yes, my basic assumption is that people who care will keep their
index on an encrypted filesystem, or at least on a filesystem that sits
atop an encrypted block device layer.  I don't know whether it's
relevant if the encryption layer is "full-disk" or not.

> Security, in this context, seems a little broad... I do have a antsy
> feeling at decrypting all my private emails in a cleartext database
> without additional measures. I'd sure love to see this notion expanded
> here somehow.

expand away!  I'd be interested in reading your detailed thoughts on
this.

My basic rationale is:

 * i find myself not encrypting some e-mails because encrypted e-mails
   are too hard to use.  in particular, they're hard to search through,
   and they are expensive to render. ("where was the address of that
   restaurant again?  if only i had sent it in cleartext...")

 * the result is that i either send those mails in the clear (leaking
   the cleartext irrevocably), or i move my communications off of e-mail
   (typically to platforms on which i have even less control than
   e-mail).

 * This is actually a messaging security *anti-pattern*, and i've
   observed it in someone who takes communications confidentiality
   seriously (myself).

 * furthermore, it harms the ability for a conscientious sender to
   produce encrypted e-mail by default, because they're likely to be
   concerned about the same usability obstacles on the receiver's side.

 * making the use of encrypted e-mails more fluid and accessible should
   actually increase the total amount of encrypted mail, providing more
   protection in general.

 * the steps you need to take to secure your cleartext index are similar
   (though not congruent) to the steps you need to take to secure your
   MUA itself (endpoint-level security hardening).  If you aren't
   already taking basic steps to protect your endpoint, it's not clear
   to me that keeping encrypted messages out of your index protects them
   significantly against a motivated attacker.

I'd love to talk more at some point about dynamic or scoped access to a
subsets of an encrypted filesystem (i.e. limiting access to your
cleartext index so that it's *only* available to your MUA, and not to
other programs running as your user account).

Hopefully this series will trigger that as an additional discussion,
though, rather than block on it.  let's make encrypted e-mail as usable
as cleartext e-mail (or as close to it as possible) to protect more
communications overall.
   

> By the way, I have similar concerns about the autocrypt approach, which
> goes even further and says private key material should not be protected
> by a password at all:
>
> http://autocrypt.readthedocs.io/en/latest/level1.html#secret-key-protection-at-rest
>
> It would be interesting to explain the rationale around those decisions
> (which autocrypt does) and possible safeguards that mitigate those
> issues (which autocrypt doesn't).

Yes, we should have this discussion, but i don't think the notmuch
mailing list is the right place for it.  Feel free to come talk it over
on the autocrypt list!

       --dkg

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: [PATCH] NEWS: cleartext indexing

[Antoine Beaupré]

On 2017-10-30 16:47:49, Daniel Kahn Gillmor wrote:
> On Mon 2017-10-30 08:46:12 -0400, Antoine Beaupré wrote:
>> On 2017-10-22 11:36:34, Daniel Kahn Gillmor wrote:
>>> +  Note that the contents of the index are sufficient to roughly
>>> +  reconstruct the cleartext of the message itself, so please ensure
>>> +  that the notmuch index itself is adequately protected.  DO NOT USE
>>> +  this feature without considering the security of your index.
>>
>> Could we expand on what those security options could be? Full disk
>> encryption? Or is there some way to PGP-encrypt the index and have it
>> decrypted on the fly?
>
> This is deliberately out of scope of the series -- i don't want this
> series to introduce notmuch-specific index encryption. Though if someone
> wanted to propose that i'd be happy to review it; fwiw, i have looked at
> a few deterministic mail index encryption schemes, and i'm not convinced
> that they meet any realistic threat model.
>
> But yes, my basic assumption is that people who care will keep their
> index on an encrypted filesystem, or at least on a filesystem that sits
> atop an encrypted block device layer.  I don't know whether it's
> relevant if the encryption layer is "full-disk" or not.

I think that assumption should be made clear in the documentation,
because "security of your index" means nothing to me. Explicitly mention
FDE as an example may be a good start.

>> Security, in this context, seems a little broad... I do have a antsy
>> feeling at decrypting all my private emails in a cleartext database
>> without additional measures. I'd sure love to see this notion expanded
>> here somehow.
>
> expand away!  I'd be interested in reading your detailed thoughts on
> this.

Frankly, I don't have a good solution for this. I was thinking that
there may be a way to just encrypt the whole notmuch database with gpg
and decrypt it on the fly as needed, but that's probably a ludicrous
idea.

> My basic rationale is:
>
>  * i find myself not encrypting some e-mails because encrypted e-mails
>    are too hard to use.  in particular, they're hard to search through,
>    and they are expensive to render. ("where was the address of that
>    restaurant again?  if only i had sent it in cleartext...")
>
>  * the result is that i either send those mails in the clear (leaking
>    the cleartext irrevocably), or i move my communications off of e-mail
>    (typically to platforms on which i have even less control than
>    e-mail).
>
>  * This is actually a messaging security *anti-pattern*, and i've
>    observed it in someone who takes communications confidentiality
>    seriously (myself).
>
>  * furthermore, it harms the ability for a conscientious sender to
>    produce encrypted e-mail by default, because they're likely to be
>    concerned about the same usability obstacles on the receiver's side.
>
>  * making the use of encrypted e-mails more fluid and accessible should
>    actually increase the total amount of encrypted mail, providing more
>    protection in general.
>
>  * the steps you need to take to secure your cleartext index are similar
>    (though not congruent) to the steps you need to take to secure your
>    MUA itself (endpoint-level security hardening).  If you aren't
>    already taking basic steps to protect your endpoint, it's not clear
>    to me that keeping encrypted messages out of your index protects them
>    significantly against a motivated attacker.

So I hear all those arguments and mostly agree with them. That's the
"rationale about the decision" part, what I'm missing is the "mitigation
strategies". What I'm hearing is simply "use FDE", but I already do that
and I don't feel it brings much added security.

Having my emails encrypted adds another layer of security to that
content. FDE is good for data "at rest", e.g. when i travel with my
laptop. But when my laptop is opened and running, I like to think that a
part of it isn't accessible without the security layers behind PGP and
actual human confirmation. I don't feel that FDE brings that level of
security because "at rest" means "when the computer is shutdown" not
"when I'm not using the thing", unfortunately.

Now, I understand there may be no solution to this. But shifting the
burden of "secure this" to the user doesn't seem fair in this
context. We should clearly expose this as a compromise that the user
must be ready to take, not just be left as an exercise to the reader,
because there may be *no* solution.

In other words, what I think you are proposing with this patchset is to
consider PGP email encryption as a end to end encryption mechanism, but
*not* as a "at rest" encryption mechanism. I think that's a tradeoff I
may be ready to make, but at least it needs to be explicitly stated.

I am also not sure if it's the best way to implement such a
tradeoff. Why not simply decrypt the actual email on delivery and store
them in cleartext if you're going to have a cleartext copy on the side
anyways? That would seem like a much simpler solution to the problem
you're trying to solve here...

> I'd love to talk more at some point about dynamic or scoped access to a
> subsets of an encrypted filesystem (i.e. limiting access to your
> cleartext index so that it's *only* available to your MUA, and not to
> other programs running as your user account).

Yeah, that's definitely something that's missing from Linux
systems. Android also suffers from that problem, even though it really
tries hard to keep data from being shared between applications. This is
better explained by Matt Green here:

https://blog.cryptographyengineering.com/2016/11/24/android-n-encryption/

But basically, iOS encrypts file per app, not per disk, so that app A
doesn't have the crypto key material to decrypt data from app B at
all. This is a fundamentally different principle than the way we do
encryption now in Linux, and would require a fundamentally different
approach to a lot of things for this to work at all on our
workstations.

> Hopefully this series will trigger that as an additional discussion,
> though, rather than block on it.  let's make encrypted e-mail as usable
> as cleartext e-mail (or as close to it as possible) to protect more
> communications overall.

I definitely don't mean to block this. But I would like to see some
changes to the documentation to better explain those trade-offs, even if
it means just linking to this discussion. :)

>> By the way, I have similar concerns about the autocrypt approach, which
>> goes even further and says private key material should not be protected
>> by a password at all:
>>
>> http://autocrypt.readthedocs.io/en/latest/level1.html#secret-key-protection-at-rest
>>
>> It would be interesting to explain the rationale around those decisions
>> (which autocrypt does) and possible safeguards that mitigate those
>> issues (which autocrypt doesn't).
>
> Yes, we should have this discussion, but i don't think the notmuch
> mailing list is the right place for it.  Feel free to come talk it over
> on the autocrypt list!

Sure - I'll actually read the darn thing before commenting further
though. :p

Thanks for bringing this up, I hope this leads to a fruitful
discussion. :)

A.

-- 
À force de ne jamais réfléchir, on a un bonheur stupide
                        - Jean Cocteau

Re: [PATCH] NEWS: cleartext indexing

[Daniel Kahn Gillmor]

[-- Attachment #1: Type: text/plain, Size: 10322 bytes --]

On Mon 2017-10-30 12:16:25 -0400, Antoine Beaupré wrote:
> I think that assumption should be made clear in the documentation,
> because "security of your index" means nothing to me. Explicitly mention
> FDE as an example may be a good start.

again, i'm not convinced that "full disk" encryption is what's
warranted, although filesystem, directory, or per-file encryption might
be part of the solution.

I don't want the documentation produced here to prescribe a particular
solution, because i *want* people to experiment and investigate.

I also don't think that the notmuch documentation is the right place to
put a primer on filesystem encryption, or a treatise on the comparison
of data at rest to data in flight.  I wouldn't object to pointers to
more discussion here though, for people who want to read more.

> Frankly, I don't have a good solution for this. I was thinking that
> there may be a way to just encrypt the whole notmuch database with gpg
> and decrypt it on the fly as needed, but that's probably a ludicrous
> idea.

Given the size of the database on the corpora i'm used to running
notmuch over, i don't see an efficient way to do that workflow.  If the
database was smaller, that'd be no problem.

But even if it was doable efficiently, you're still left with a question
of when you plan to unlock or re-lock the database.  And while the
database is unlocked, how do you limit access to it to notmuch itself?

And if you do limit it to notmuch itself, what about the other tools
that might want to interface with notmuch?  are they allowed access to
the cleartext?

These are much more interesting questions to my mind than whether we use
gpg specifically or something else.

They would also apply to filesystem-level encryption.

> So I hear all those arguments and mostly agree with them. That's the
> "rationale about the decision" part, what I'm missing is the "mitigation
> strategies". What I'm hearing is simply "use FDE", but I already do that
> and I don't feel it brings much added security.

It's interesting that you heard "FDE" when i've explicitly said "not
FDE". Filesystem encryption ≠ "full disk" encryption ≠ per-directory
encryption.

For example, you could have a separate local filesystem that contains
only your message store and your notmuch index, mounted atop a distinct
crypto-mapped block device.

Then, when you're willing to allow access to your index (and the rest of
your "at-rest" mail) you simply map the block device and mount the
filesystem where the user account that uses notmuch can read it.

when you're done with mail, you umount and unmap.

This proposal also adds "encryption at rest" to the mail that wasn't
even encrypted in transit, as a bonus :)

You could even put the whole underlying block device on removable
storage, making it truly inaccessible when it is not plugged in.

Or you could try to use ext4's new-ish encryption features:

    https://lwn.net/Articles/639427/
    https://wiki.archlinux.org/index.php/Ext4#Using_file-based_encryption
    http://kernsec.org/files/lss2014/Halcrow_EXT4_Encryption.pdf

I'd be happy to experiment with that with you and report back to the
list if you like.  Maybe we could prototype a "notmuch lock" command?


Again, i don't think that notmuch documentation is a great place to
document these ideas, unless we're actually implementing them in a
simple and straightforward way so that the user can trigger the actions
easily.

And, sadly, note that my proposal above usually requires root access
(mount/umount/device-map) on most GNU/Linux systems (i haven't looked
into other systems in enough detail).  This is certainly an obstacle to
deployment.


> Having my emails encrypted adds another layer of security to that
> content. FDE is good for data "at rest", e.g. when i travel with my
> laptop. But when my laptop is opened and running, I like to think that a
> part of it isn't accessible without the security layers behind PGP and
> actual human confirmation.

It sounds to me like you think that all invocations of PGP are going to
be mapped behind human confirmation.  This is not the general GnuPG use
case these days (due to caching in gpg-agent), except for people with an
external crypto token that itself physically requires presence to
decrypt.

In the event of rendering a 10-message encrypted thread, the "human
confirmation" approach requires 10 touches of the cryptotoken.  IIRC,
even if the human is alert and ready to touch the token when needed,
we're talking about probably at least half a minute of delay between
"i'd like to read this thread" and "ah, there it is".  and the user
can't even do something else with that time because they're all tied up
watching for and performing the crypto-token-touch.

I don't think that's acceptable for an e-mail client that you expect
users to actually use, unless you're running some sort of skinnerian
behaviorist experiement.

So in practice, one authorization of your PGP key is likely to enable
arbitrary programs to access it for the duration of the gpg-agent
cache.  So your data is still actually accessible to any process that
can access both the agent and the message store.

That said, if there is a specific message which you think should not be
in the index, the cleartext-index series and the session-key series both
provide means for keeping a *specific* encrypted message in the
mailstore while ensuring that it is not indexed and no session-keys are
stashed.

An interesting proposal might be to add an additional per-message
property, which says explicitly "do not index cleartext or store session
keys for this message".  I don't believe there are many users who would
actually use this feature, but i would review a patch for it and provide
feedback.

> Now, I understand there may be no solution to this. But shifting the
> burden of "secure this" to the user doesn't seem fair in this
> context. We should clearly expose this as a compromise that the user
> must be ready to take, not just be left as an exercise to the reader,
> because there may be *no* solution.

I disagree.  This patch series doesn't shift any burden to the user.
Without the series, the user has no way to make this decision -- they're
stuck with "encryption in transit means complete encryption at rest",
with all the poor usability and crypto-discouragement that entails.  The
series provides users a way to *decide* to shift the burden to
themselves, to improve their prospects for actually using end-to-end
encryption in transit.

> In other words, what I think you are proposing with this patchset is to
> consider PGP email encryption as a end to end encryption mechanism, but
> *not* as a "at rest" encryption mechanism. I think that's a tradeoff I
> may be ready to make, but at least it needs to be explicitly stated.

That's exactly what the documentation says: do not enable this without
considering the security of your index.  And again, i don't want to
prescribe solutions unless we're going to offer an easy way to implement
them.

> I am also not sure if it's the best way to implement such a
> tradeoff. Why not simply decrypt the actual email on delivery and store
> them in cleartext if you're going to have a cleartext copy on the side
> anyways? That would seem like a much simpler solution to the problem
> you're trying to solve here...

I hear this proposal about every month, i think, usually from different
people.   I think it's a bad idea on several levels.

Personally, i expect my message store to be untouched, with the message
as-delivered.  This allows me to sync the message store itself to other
places (e.g. offline-imap, rsync, etc) without worrying that i'm
suddenly exposing new data.  It also allows me to inspect the delivered
messages for any metadata that might otherwise be destroyed during
decryption, if i ever want to analyze the traces that i'm leaving.  And
if i was to keep both copies of the message (as-delivered, and
as-decrypted), i'd double the size of my mailstore (well, i'd double the
size of my encrypted mailstore anyway).

No thanks!  These are bad tradeoffs, when i can get cleartext indexes
and fast rendering without bothering with all of these other downsides.

> Yeah, that's definitely something that's missing from Linux
> systems. Android also suffers from that problem, even though it really
> tries hard to keep data from being shared between applications. This is
> better explained by Matt Green here:
>
> https://blog.cryptographyengineering.com/2016/11/24/android-n-encryption/
>
> But basically, iOS encrypts file per app, not per disk, so that app A
> doesn't have the crypto key material to decrypt data from app B at
> all. This is a fundamentally different principle than the way we do
> encryption now in Linux, and would require a fundamentally different
> approach to a lot of things for this to work at all on our
> workstations.

I agree with you that per-app, per-user, and per-file encryption all
offer several great properties that are better than filesystem-level
encryption on its own.  and GNU/Linux is behind on that front (though
see my remarks about about ext4 encryption).

Note that a stored encrypted e-mail, with its session-key stored
separately in the index, is actually very similar to per-file
encryption.  The e-mail leaks more metadata than the file itself, but
the model's roughly equivalent.  The only thing that's missing is the
protection of the index itself, as you mention.

I'd be happy to explore what "per-app" encryption might mean on desktop
systems as well with you, but we should probably do that off-list, and
come back here when we've got something specific to propose.

> I definitely don't mean to block this. But I would like to see some
> changes to the documentation to better explain those trade-offs, even if
> it means just linking to this discussion. :)

Please propose a patch to the documentation that would satisfy you!  I
agree with you that having some more discussion would be useful, but the
full content of even this thread would be out of place in any of the
notmuch manpages.

        --dkg

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: [PATCH] NEWS: cleartext indexing

[Antoine Beaupré]

Hi,

Sorry for the long delay in my response, but it was a long email to
review - there's a lot of stuff in here - so I didn't quite know how to
respond. I'll just respond inline but will try to keep it brief.

On 2017-11-01 04:13:26, Daniel Kahn Gillmor wrote:
> On Mon 2017-10-30 12:16:25 -0400, Antoine Beaupré wrote:
>> I think that assumption should be made clear in the documentation,
>> because "security of your index" means nothing to me. Explicitly mention
>> FDE as an example may be a good start.
>
> again, i'm not convinced that "full disk" encryption is what's
> warranted, although filesystem, directory, or per-file encryption might
> be part of the solution.
>
> I don't want the documentation produced here to prescribe a particular
> solution, because i *want* people to experiment and investigate.
>
> I also don't think that the notmuch documentation is the right place to
> put a primer on filesystem encryption, or a treatise on the comparison
> of data at rest to data in flight.  I wouldn't object to pointers to
> more discussion here though, for people who want to read more.

The problem is we do not have such documentation to point to. The
original text I was critical of is this, to put things back in context:

> +  Note that the contents of the index are sufficient to roughly
> +  reconstruct the cleartext of the message itself, so please ensure
> +  that the notmuch index itself is adequately protected.  DO NOT USE
> +  this feature without considering the security of your index.

In other words, the documentation uses "adequately protected" and
"security of your index" as deliberately vague terms to urge the user to
consider practices. My concerns here is that those words may mean *very*
different things to different people. Just this discussion shows the
range of possible outcomes - I was thinking FDE, but then you suggested
that this was not adequate, with a plethora of possible solutions.

Which one do *you* use? I suspect most people that will adopt this
technology will simply *ignore* that warning and just store their index
in plaintext. In which case this warning is pointless and joins a long
series of fatigue-generating warnings that we constantly train our users
to ignore.

If we will warn, at least we should give guidelines of how to fix the
warning.

[... on-the-fly encryption of notmuch DB discarded ...]

[... possible actual solutions brainstorm for a notmuch lock / unlock
command based on ( ext4 encryption | notmuch-specific filesystem |
removable notmuch storage | etc ) ... ]

> Again, i don't think that notmuch documentation is a great place to
> document these ideas, unless we're actually implementing them in a
> simple and straightforward way so that the user can trigger the actions
> easily.

Sure. But this discussion shows that we do *not* have a clear and simple
recommendation for people. That's too bad, and bound to the limitation
of the system we're using of course (e.g. Linux instead of iOS) but we
shouldn't pretend there *are* solutions when there aren't, out of the
box, ways to secure the index.

[...]

>> Having my emails encrypted adds another layer of security to that
>> content. FDE is good for data "at rest", e.g. when i travel with my
>> laptop. But when my laptop is opened and running, I like to think that a
>> part of it isn't accessible without the security layers behind PGP and
>> actual human confirmation.

[...]

> So in practice, one authorization of your PGP key is likely to enable
> arbitrary programs to access it for the duration of the gpg-agent
> cache.  So your data is still actually accessible to any process that
> can access both the agent and the message store.

Sure. But the point is exactly that - there is *some* level of
control. I don't use a crypto token for encryption now, but when/if I
will there will be a satisfying, hardware, way of ensuring things are
encrypted at rest: just yank the key out. :) Short of that, yes, having
gpg-agent prompt me for stuff is a good compromise.

> That said, if there is a specific message which you think should not be
> in the index, the cleartext-index series and the session-key series both
> provide means for keeping a *specific* encrypted message in the
> mailstore while ensuring that it is not indexed and no session-keys are
> stashed.
>
> An interesting proposal might be to add an additional per-message
> property, which says explicitly "do not index cleartext or store session
> keys for this message".  I don't believe there are many users who would
> actually use this feature, but i would review a patch for it and provide
> feedback.

That would be an interesting feature, but I don't know how it could be
practically implemented. Notmuch feature discovery is limited, at best,
with the current feature set (e.g. there's no "Notmuch menu" where we
could show that feature) so I doubt we could easily add that to Notmuch
in any practical way.

[...]

> I disagree.  This patch series doesn't shift any burden to the user.

[...]

Granted.

[...]

>> I am also not sure if it's the best way to implement such a
>> tradeoff. Why not simply decrypt the actual email on delivery and store
>> them in cleartext if you're going to have a cleartext copy on the side
>> anyways? That would seem like a much simpler solution to the problem
>> you're trying to solve here...
>
> I hear this proposal about every month, i think, usually from different
> people.   I think it's a bad idea on several levels.

[...]

> No thanks!  These are bad tradeoffs, when i can get cleartext indexes
> and fast rendering without bothering with all of these other downsides.

Agreed, that was a straw man on my part. :)

[...]

> I'd be happy to explore what "per-app" encryption might mean on desktop
> systems as well with you, but we should probably do that off-list, and
> come back here when we've got something specific to propose.

Obviously, this will not happen in the short term, so we shouldn't block
this patch series based on the theorical idea of an eventual redesign of
the whole Linux filsystem encryption scheme. ;)

>> I definitely don't mean to block this. But I would like to see some
>> changes to the documentation to better explain those trade-offs, even if
>> it means just linking to this discussion. :)
>
> Please propose a patch to the documentation that would satisfy you!  I
> agree with you that having some more discussion would be useful, but the
> full content of even this thread would be out of place in any of the
> notmuch manpages.

Maybe not. But I believe a summary of alternatives like you did would be
way more honest and direct than alluding to ways of "securing the data"
when there aren't easy workable solutions.

Obviously, we have made this long thread about a fairly minor thing. I'd
love to see the session keys land, and I do not think the series should
block on this documentation item.

If anything, I'd love to hear more of your current thoughts about those
solutions in the documentation, if only have a pointer to this very
thread as a temporary fix. :)

Because, after all, this is history in writing: freezing our thoughts
into those emails may seem obvious to me and you, but will definitely be
useful and inspiring for others that come later. What better way to
discover those ideas than to embed them in the documentation? Otherwise
we expect people to rebuild those ideas from scratch and that seems like
a waste.

I, unfortunately, do not have an direct patch to fix the documentation,
but I hope I made you see the concerns I have about the current
wording. If I would be urged to make a patch or move on, I would add a
link to this thread and move on.

Thanks for your thoughts and hard work!

A.

-- 
The world is a dangerous place, not because of those who do evil,
but because of those who look on and do nothing.
                        - Albert Einstein

Re: [PATCH] NEWS: cleartext indexing

[Antoine Beaupré]

Another thing I forgot. You mentioned dedicated LUKS partitions as an
example solution. I wonder if you know about the `tomb` and `ctmg`
projects which more or less implement those features as commandline tool
wrappers.

Tomb is a simple shell-script wrapper around cryptsetup to easily create
and manage loop-mounted LUKS partitions:

http://tomb.dyne.org/

An example use of this for notmuch would be the `pass-tomb` extension to
the `pass` password manager, which uses tomb to hide password entries
when not in use:

https://github.com/roddhjav/pass-tomb

CTMG is basically the same thing but written by Donenfeld instead of
Jaromil:

https://git.zx2c4.com/ctmg/about/

Both require root to run. In both cases, my primary concern would be how
to manage the size of the LUKS partition just right: it shouldn't take
up space needlessly, but then it needs to expand when new space is
needed. As far as I know, none of those tools elegantly solve that
problem, except maybe the new ext4 encryption system...

Thanks again for this precious patchset, I hope it gets rolled in soon!
0.26 will be even more amazing it gets shipped with this.

A.

-- 
Gods don't like people not doing much work. People who aren't busy all
the time might start to think.
                        - Terry Pratchett, Small Gods