OpenPGP support

OpenPGP support

[Jameson Graef Rollins]

[-- Attachment #1: Type: text/plain, Size: 114 bytes --]

Hi, folks.  I know it's a hairy problem, but has anyone started
looking into OpenPGP support for notmuch?

jamie.

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: OpenPGP support

[Jameson Graef Rollins]

[-- Attachment #1: Type: text/plain, Size: 922 bytes --]

On Mon, Nov 23, 2009 at 08:00:09AM -0500, Jameson Graef Rollins wrote:
> Hi, folks.  I know it's a hairy problem, but has anyone started
> looking into OpenPGP support for notmuch?

Hey, folks.  I wanted to try to follow up on this question.

I would really like to start using notmuch with emacs beyond just
testing, but I really need to be able to handle/read/send mail with
PGP/MIME encoded attachments.  Do folks have any suggestions on how to
handle this?  Is there a separate emacs mode that people use for
signing/verifying/{de,en}crypting mail buffers, or is this something
that is going to have to be integrated into the notmuch mode?  I guess
the notmuch-show mode at least will need to do some verifying and
decrypting.

I've noticed that some people on this list sign their mails.  Are
those of you that do using notmuch?  If so, I would love to learn how
you handle it.  Thanks so much for the info.

jamie.i

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: OpenPGP support

[Aneesh Kumar K. V]

[-- Attachment #1: Type: text/plain, Size: 1168 bytes --]

On Thu, 26 Nov 2009 01:01:32 -0500, Jameson Graef Rollins <jrollins@finestructure.net> wrote:
> On Mon, Nov 23, 2009 at 08:00:09AM -0500, Jameson Graef Rollins wrote:
> > Hi, folks.  I know it's a hairy problem, but has anyone started
> > looking into OpenPGP support for notmuch?
> 
> Hey, folks.  I wanted to try to follow up on this question.
> 
> I would really like to start using notmuch with emacs beyond just
> testing, but I really need to be able to handle/read/send mail with
> PGP/MIME encoded attachments.  Do folks have any suggestions on how to
> handle this?  Is there a separate emacs mode that people use for
> signing/verifying/{de,en}crypting mail buffers, or is this something
> that is going to have to be integrated into the notmuch mode?  I guess
> the notmuch-show mode at least will need to do some verifying and
> decrypting.
> 
> I've noticed that some people on this list sign their mails.  Are
> those of you that do using notmuch?  If so, I would love to learn how
> you handle it.  Thanks so much for the info.
> 

message-mode can sign the message using C-c RET C-s
which is mml-secure-message-sign


-aneesh

[-- Attachment #2: Type: application/pgp-signature, Size: 489 bytes --]

Re: OpenPGP support

[Jameson Graef Rollins]

[-- Attachment #1: Type: text/plain, Size: 1595 bytes --]

On Thu, Nov 26, 2009 at 12:38:01PM +0530, Aneesh Kumar K. V wrote:
> message-mode can sign the message using C-c RET C-s
> which is mml-secure-message-sign

Thanks so much, Aneesh!  That's great.  I was not aware of those
abilities in mml mode.  I just noticed that epa-mail-{sign,encrypt}
can do this as well, but much less cleanly.  So the mml commands are
exactly what I was looking for.  Well, at least exactly *half* of what
I was looking for...

So the issue now is verifying/decrypting signed/encrypted received
mail.  Playing around with it a bit more, I've found a couple of
interesting things.  If I'm viewing an encrypted message with
notmuch-show, and if I type 'v' (notmuch-show-view-all-mime-parts), I
get the full message view, and then a prompt to:

Decrypt (PGP) part? (y or n)

Typing 'y', I see that there is some communication with my gpg agent
(I get a gpg agent password prompt), but then once the password is
entered, nothing happens, ie. I don't see the decrypted part.  I'm
trying to figure out where the problem is here.  Presumably something
is failing to capture the output of the gpg decryption.  I'm not clear
either about what mode is trying to do this decryption.  Is this epa?

Unfortunately nothing at all happens when I hit 'v' when viewing a
message that has OpenPGP signature attachment.

I also just discovered that there are some
epa-mail-{sign,verify,{de,en}crypt} commands that work similar to (but
not as cleanly as) the mml commands.  Unfortunately none of this is
very coherent.  And some sort of integration with notmuch-show would
be sweet.

jamie.

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: OpenPGP support

[Carl Worth]

On Mon, 23 Nov 2009 08:00:09 -0500, Jameson Graef Rollins <jrollins@finestructure.net> wrote:
> Hi, folks.  I know it's a hairy problem, but has anyone started
> looking into OpenPGP support for notmuch?

I definitely want this.

I've got message-mode setup to sign my messages automatically when I
start a new compose from within notmuch with 'm'. I did this with the
following in my .emacs:

; sign messages by default
(add-hook 'message-setup-hook
	  (lambda ()
	    (mml-secure-message-sign)))

I'm not sure how to make sure that I also get that when doing a reply
within notmuch. Keith, I think you worked this out, right? What did you
do?

I'm also interested in automating things like this as much as
possible. I'd much rather people flip one Boolean notmuch-sign-messages
variable then have to code up multiple lambda's within .emacs.

Meanwhile, the other pieces still needed are automatic signature
verification for multipart/signed messages, and then support for
decrypting GPG-encrypted messages.

I'm certain that the various mail-handling modes within emacs already
have support for these things, so it's really just a matter of finding
and hooking into that support.

Of course, if we code some of this up at the library or command-line
level, then the non-emacs interfaces could automatically benefit from it
as well.

And that same argument applies for viewing of MIME-encoded message
parts---I definitely want support at the command-line interface of
notmuch to at least extract a specific MIME part from a message.

In one sense, it's unfortunate that I figured out a way to let me view
these parts within emacs without that support in the notmuch
command-line. That definitely took the pressure off me to finish up
coding this piece, (which actually wouldn't be hard at all).

-Carl

Re: OpenPGP support

[Carl Worth]

On Thu, 26 Nov 2009 13:31:05 -0500, Jameson Graef Rollins <jrollins@finestructure.net> wrote:
> So the issue now is verifying/decrypting signed/encrypted received
> mail.  Playing around with it a bit more, I've found a couple of
> interesting things.  If I'm viewing an encrypted message with
> notmuch-show, and if I type 'v' (notmuch-show-view-all-mime-parts), I
> get the full message view, and then a prompt to:
> 
> Decrypt (PGP) part? (y or n)
> 
> Typing 'y', I see that there is some communication with my gpg agent
> (I get a gpg agent password prompt), but then once the password is
> entered, nothing happens, ie. I don't see the decrypted part.

Our integration with the MIME pieces of emacs definitely leaves
something to be desired. What currently happens when you press 'v' is
that notmuch creates a new buffer with the raw message content, then
runs mm-dissect-buffer and mm-display-parts on the content.

This works quite well for non-text parts, (either a mailcap-configured
viewer will be launched, or else you will be prompted to save the
file). But for text parts, the mm code appears to just write into the
temporary buffer that notmuch ends up killing before you can see
anything. So we just need to work out something proper here.

One thing I experimented with, that seems to work quite well, is to
construct a minimal buffer that the mm code will interpret as a
MIME-encoded message with just a single part.

So that's just a MIME-Version header, the original Content-type header,
a blank line, and then the original MIME content with its boundary
markers. So if we write a notmuch command to return that, then we'll be
able to easily support viewing of a single MIME part, (and I think it
will be easy to detect whether any text content ended up getting added
to the intermediate buffer that the user wants to see).

Any help experimenting more with this stuff would be very appreciated.

-Carl

Re: OpenPGP support

[Keith Packard]

[-- Attachment #1: Type: text/plain, Size: 790 bytes --]

On Thu, 26 Nov 2009 10:41:36 -0800, Carl Worth <cworth@cworth.org> wrote:

> I'm not sure how to make sure that I also get that when doing a reply
> within notmuch. Keith, I think you worked this out, right? What did you
> do?

; sign messages by default
(defun my-notmuch-message-mode-hook()
  (if (buffer-modified-p)
      (mml-secure-message-sign)))

(add-hook 'message-mode-hook 'my-notmuch-message-mode-hook)
(add-hook 'message-setup-hook 'my-notmuch-message-mode-hook)

The message-setup-hook is called for new messages, the message-mode-hook
is called for both; the hack is to check for the reply case by looking
to see if the buffer has been modified as the message mode hook gets
called on an empty buffer in the new message case.

-- 
keith.packard@intel.com

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: OpenPGP support

[Carl Worth]

[-- Attachment #1: Type: text/plain, Size: 642 bytes --]

On Thu, 26 Nov 2009 15:01:38 -0800, Keith Packard <keithp@keithp.com> wrote:
> On Thu, 26 Nov 2009 10:41:36 -0800, Carl Worth <cworth@cworth.org> wrote:
> 
> > I'm not sure how to make sure that I also get that when doing a reply
> > within notmuch. Keith, I think you worked this out, right? What did you
> > do?
> 
> ; sign messages by default
> (defun my-notmuch-message-mode-hook()
>   (if (buffer-modified-p)
>       (mml-secure-message-sign)))
> 
> (add-hook 'message-mode-hook 'my-notmuch-message-mode-hook)
> (add-hook 'message-setup-hook 'my-notmuch-message-mode-hook)

Thanks. And look! A signed reply now.

-Carl

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: OpenPGP support

[Jameson Graef Rollins]

[-- Attachment #1: Type: text/plain, Size: 1509 bytes --]

On Thu, Nov 26, 2009 at 10:41:36AM -0800, Carl Worth wrote:
> On Mon, 23 Nov 2009 08:00:09 -0500, Jameson Graef Rollins <jrollins@finestructure.net> wrote:
> > Hi, folks.  I know it's a hairy problem, but has anyone started
> > looking into OpenPGP support for notmuch?
> 
> I definitely want this.

Thanks so much for this response, Carl.  I'm glad to hear there's
other interest in the this.

> I've got message-mode setup to sign my messages automatically when I
> start a new compose from within notmuch with 'm'. I did this with the
> following in my .emacs:
> 
> ; sign messages by default
> (add-hook 'message-setup-hook
> 	  (lambda ()
> 	    (mml-secure-message-sign)))
>
> I'm not sure how to make sure that I also get that when doing a reply
> within notmuch. Keith, I think you worked this out, right? What did you
> do?

Ah, this is a great suggestion.  Thank you.  I saw Keith's reply as
well, which also works great with replies.

> Meanwhile, the other pieces still needed are automatic signature
> verification for multipart/signed messages, and then support for
> decrypting GPG-encrypted messages.

So I think this is maybe the most important thing missing from notmuch
right now.  I would really really like to see this working.
Unfortunately my emacs/lisp skills are poor to say the least.  I'll
try to look into how gnus or other emacs mail interfaces handle this
stuff, but if anyone else has any pointers for the list that would be
great.

jamie.

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

indexing encrypted messages (was: OpenPGP support)

[martin f krafft]

[-- Attachment #1: Type: text/plain, Size: 987 bytes --]

also sprach Jameson Graef Rollins <jrollins@finestructure.net> [2009.11.26.1901 +1300]:
> I would really like to start using notmuch with emacs beyond just
> testing, but I really need to be able to handle/read/send mail with
> PGP/MIME encoded attachments.  Do folks have any suggestions on how to
> handle this?  Is there a separate emacs mode that people use for
> signing/verifying/{de,en}crypting mail buffers, or is this something
> that is going to have to be integrated into the notmuch mode?  I guess
> the notmuch-show mode at least will need to do some verifying and
> decrypting.

How about indexing GPG-encrypted messages?

-- 
martin | http://madduck.net/ | http://two.sentenc.es/
 
"a scientist once wrote that all truth passes through three stages:
 first it is ridiculed, then violently opposed and eventually,
 accepted as self-evident."
                                                       -- schopenhauer
 
spamtraps: madduck.bogus@madduck.net

[-- Attachment #2: Digital signature (see http://martin-krafft.net/gpg/) --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: indexing encrypted messages (was: OpenPGP support)

[Mike Hommey]

On Fri, Jan 08, 2010 at 03:56:10PM +1300, martin f krafft wrote:
> also sprach Jameson Graef Rollins <jrollins@finestructure.net> [2009.11.26.1901 +1300]:
> > I would really like to start using notmuch with emacs beyond just
> > testing, but I really need to be able to handle/read/send mail with
> > PGP/MIME encoded attachments.  Do folks have any suggestions on how to
> > handle this?  Is there a separate emacs mode that people use for
> > signing/verifying/{de,en}crypting mail buffers, or is this something
> > that is going to have to be integrated into the notmuch mode?  I guess
> > the notmuch-show mode at least will need to do some verifying and
> > decrypting.
> 
> How about indexing GPG-encrypted messages?

That may leak decrypted form in the xapian index, though in a split
manner. But that'd still be a problem IMHO.

Mike

Re: indexing encrypted messages (was: OpenPGP support)

[martin f krafft]

[-- Attachment #1: Type: text/plain, Size: 586 bytes --]

also sprach Mike Hommey <mh+notmuch@glandium.org> [2010.01.08.2109 +1300]:
> That may leak decrypted form in the xapian index, though in
> a split manner. But that'd still be a problem IMHO.

Not for me, since the index is stored on encrypted media. Thus, this
should be off-by-default, but possible.

-- 
martin | http://madduck.net/ | http://two.sentenc.es/
 
"academia is really just a way to help those with high volumes of
 nothing to say to social status."
                             -- myself on #debian-devel, 01 Feb 2007
 
spamtraps: madduck.bogus@madduck.net

[-- Attachment #2: Digital signature (see http://martin-krafft.net/gpg/) --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: indexing encrypted messages (was: OpenPGP support)

[Ruben Pollan]

[-- Attachment #1: Type: text/plain, Size: 665 bytes --]

On 15:56, Fri 08 Jan 10, martin f krafft wrote:
> How about indexing GPG-encrypted messages?

I think that would be security hole. You should not store the encrypted messages
on a decrypted database. A solution whould be to encrypt as well the xapian DB,
but I think is too complex for the use.

You should be still able, with the actual notmuch, to search over the headers of 
your encrypted messages, or any other non-encrypted part of the message. Is not
like that?

-- 
Rubén Pollán  | jabber:meskio@jabber.org
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
 Cuando los que mandan pierden la vergüenza,
    los que obedecen pierden el respeto.

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: indexing encrypted messages (was: OpenPGP support)

[martin f krafft]

[-- Attachment #1: Type: text/plain, Size: 1007 bytes --]

also sprach Ruben Pollan <meskio@sindominio.net> [2010.01.08.2221 +1300]:
> I think that would be security hole. You should not store the
> encrypted messages on a decrypted database. A solution whould be
> to encrypt as well the xapian DB, but I think is too complex for
> the use.

As I said in <20100108091216.GC735@lapse.rw.madduck.net>, I think it
should be optionally possible for those that are encrypting the
xapian DB in other ways.

> You should be still able, with the actual notmuch, to search over
> the headers of your encrypted messages, or any other non-encrypted
> part of the message. Is not like that?

Most of the time, I search headers, but I do search bodies
regularly. So no, that would not be enough, at least not with the
ideal solution. And notmuch comes close to ideal already! ;)

-- 
martin | http://madduck.net/ | http://two.sentenc.es/
 
infinite loop: see 'loop, infinite'.
loop, infinite: see 'infinite loop'.
 
spamtraps: madduck.bogus@madduck.net

[-- Attachment #2: Digital signature (see http://martin-krafft.net/gpg/) --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: indexing encrypted messages (was: OpenPGP support)

[James Westby]

On Fri, 8 Jan 2010 15:56:10 +1300, martin f krafft <madduck@madduck.net> wrote:
> also sprach Jameson Graef Rollins <jrollins@finestructure.net> [2009.11.26.1901 +1300]:
> > I would really like to start using notmuch with emacs beyond just
> > testing, but I really need to be able to handle/read/send mail with
> > PGP/MIME encoded attachments.  Do folks have any suggestions on how to
> > handle this?  Is there a separate emacs mode that people use for
> > signing/verifying/{de,en}crypting mail buffers, or is this something
> > that is going to have to be integrated into the notmuch mode?  I guess
> > the notmuch-show mode at least will need to do some verifying and
> > decrypting.
> 
> How about indexing GPG-encrypted messages?

I think the difficulty will be interactivity. If notmuch-new can
potentially block watiting for a passphrase then it's not going to be
much use for non-interactive use, and whether someone can respond to a
GPG prompt is harder to determine that isatty().

Configuration may be a possible way around that, but looking at other
things such as opportunistic indexing could be good. For instance,
it could be the job of the UIs to decrypt content, and there could be a
nomuch function which takes a message id and decrypted content and
indexes it in to the DB. That means it's under the UI's control, where
the decryption UI should be, gets you indexing of encrypted content.

That would leave an open question over whether future notmuch show
invocations would return the plaintext or ciphertext. If it is the
latter then it requires decrypting every time you want to view it, but
it does mean that there is less information leakage (you could find out
whether an encrypted message contained a particular term, but not read
the whole message directly).

Thanks,

James

Re: indexing encrypted messages (was: OpenPGP support)

[micah anderson]

[-- Attachment #1: Type: text/plain, Size: 952 bytes --]

On Fri, 8 Jan 2010 10:21:21 +0100, Ruben Pollan <meskio@sindominio.net> wrote:
> On 15:56, Fri 08 Jan 10, martin f krafft wrote:
> > How about indexing GPG-encrypted messages?
> 
> I think that would be security hole. You should not store the
> encrypted messages on a decrypted database. A solution whould be to
> encrypt as well the xapian DB, but I think is too complex for the use.

Would you consider it a security hole if you stored your database on
encrypted media (such as on-disk block encryption)?

I know that sup does this, when it ran over my mail store, it would
trigger my gpg agent so that it could decrypt the encrypted
messages. This was annoying because this happened every time it ran,
which meant that unless I had used gpg recently, my agent would pop up
and ask me for my passphrase, which was often.

The way Mutt provides this functionality is by decrypting only when you
perform the search itself.

micah

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

Re: indexing encrypted messages (was: OpenPGP support)

[Ruben Pollan]

[-- Attachment #1: Type: text/plain, Size: 1883 bytes --]

On 14:41, Fri 08 Jan 10, micah anderson wrote:
> On Fri, 8 Jan 2010 10:21:21 +0100, Ruben Pollan <meskio@sindominio.net> wrote:
> > On 15:56, Fri 08 Jan 10, martin f krafft wrote:
> > > How about indexing GPG-encrypted messages?
> > 
> > I think that would be security hole. You should not store the
> > encrypted messages on a decrypted database. A solution whould be to
> > encrypt as well the xapian DB, but I think is too complex for the use.
> 
> Would you consider it a security hole if you stored your database on
> encrypted media (such as on-disk block encryption)?

No, in this case should be not a security hole. But anyway what is secure and
what not should be defined by the user. For some users may not be a security
hole to store the email decrypted.

But I think notmuch by default should not do so. This kind of things should be
something that the user activate by hand knowing what she is doing.

> I know that sup does this, when it ran over my mail store, it would
> trigger my gpg agent so that it could decrypt the encrypted
> messages. This was annoying because this happened every time it ran,
> which meant that unless I had used gpg recently, my agent would pop up
> and ask me for my passphrase, which was often.

I didn't use sup. Don't know how it works. But that feature is technically
possible. As I said before in my personal opinion that should not be the 
out-of-the-box behavior.

> The way Mutt provides this functionality is by decrypting only when you
> perform the search itself.

Yes, but notmuch can not do that. notmuch indexes the messages and mutt not.



-- 
Rubén Pollán  | jabber:meskio@jabber.org
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Lo hago para no volverme loco cuando noto
que solo me queda un demonio en un hombro
por que se ha cortado las venas
el ángel que había en el otro.

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: indexing encrypted messages (was: OpenPGP support)

[Olly Betts]

On 2010-01-08, James Westby wrote:
> That would leave an open question over whether future notmuch show
> invocations would return the plaintext or ciphertext. If it is the
> latter then it requires decrypting every time you want to view it, but
> it does mean that there is less information leakage (you could find out
> whether an encrypted message contained a particular term, but not read
> the whole message directly).

You can actually use the term position information to reconstruct the
original message text pretty well.  It misses capitalisation, punctuation,
and distinctions between whitespace, but is generally enough to allow
the message to be understood:

http://article.gmane.org/gmane.comp.search.xapian.general/2187

Cheers,
    Olly