a proposed change to JSON output to report verification of PGP/MIME signatures.

a proposed change to JSON output to report verification of PGP/MIME signatures.

[Daniel Kahn Gillmor]

[-- Attachment #1.1: Type: text/plain, Size: 8255 bytes --]

hi notmuch folks--

i've been trying to wrap my head around how to get notmuch to support
verifying cryptographically-signed mail.  i'm afraid my current
understanding of the problem space is that it is neither pretty nor
clean.  Sorry for the length of this message.

Scope:
------

I'm focusing initially here only on verifying PGP/MIME cleartext
signatures.  I'm proposing to do the verification in the backend, and to
report on the validity of the signatures to the frontend through
"notmuch show --format=json" (ignoring the other output formats for now).

This mail is only trying to explain how the JSON format might
communicate this information from the backend to the frontend.
(implementation will happen depending on the followup discussion, but i
don't mean for implementation questions to derail this first)


Proposal:
---------

No attempt to actually validate the signatures will be made unless the
new --verify flag is passed to "notmuch show".

A signed MIME part will contain a new element "signedby", which is a
list of part numbers identifying signatures that cover this part.

Signature parts (Content-Type: application/pgp-signature) will contain a
new element "signs", which points back to the list of parts this
signature covers.  It will also contain a "sigstatus" member, which is a
list of objects, each of which contain at least the following element:
 * "verified" -- one of the following values:
     "success" (the sig has been tested and is cryptographically valid)
     "failure" (the sig has been tested and does not match)
     "nokey"   (the sig could not be tested because pubkey is missing)
     "error"   (testing the sig failed for some other reason)
     "unknown" (testing was not tried)
 If "verified" is "success" in a "sigstatus" object, then the following
fields might also be present:
 * "signingkey" -- hexadecimal representation of 160-bit fingerprint of
                   the signing key
 * "digest" -- the hash over which the sig was made (e.g. "SHA1")
 * "timestamp" -- the time the signature claims to have been made
                  (let me know what format i should represent this in)
 * "pubkeyalgo" -- the signing key's asymmetric algorithm (e.g. "RSA")
 * "expires" -- if the signature has an expiration date, it goes here


Example:
--------

currently, the "body" element of a PGP/MIME signed message looks like
this with --format=json:

---------------------------
 "body": [
     {
         "content": "here is a test message i signed on 2010-11-11.\n\n
 --dkg\n\n",
         "content-type": "text/plain",
         "id": 1
     },
     {
         "content-type": "application/pgp-signature",
         "filename": "signature.asc",
         "id": 2
     }
 ],
---------------------------

It would end up like this (without the --verify flag):

---------------------------
 "body": [
     {
         "content": "here is a test message i signed on 2010-11-11.\n\n
 --dkg\n\n",
         "content-type": "text/plain",
         "id": 1,
         "signedby": [ 2 ]
     },
     {
         "content-type": "application/pgp-signature",
         "filename": "signature.asc",
         "id": 2,
         "signs": [ 1 ],
         "sigstatus": [ {
             "verified": "unknown"
         } ]
     }
 ],
---------------------------

and here it is with the --verify flag:

---------------------------
 "body": [
     {
         "content": "here is a test message i signed on 2010-11-11.\n\n
 --dkg\n\n",
         "content-type": "text/plain",
         "id": 1,
         "signedby": [ 2 ]
     },
     {
         "content-type": "application/pgp-signature",
         "filename": "signature.asc",
         "id": 2,
         "signs": [ 1 ],
         "sigstatus": [ {
             "verified": "success",
             "signingkey": "0EE5BE979282D80B9F7540F1CCD2ED94D21739E9",
             "digest": "SHA512",
             "timestamp": "2010-11-11 22:32:45 -0400",
             "pubkeyalgo": "RSA"
         } ]
     }
 ],
---------------------------


Observations:
-------------

i'm not covering key->userid bindings in this first pass -- it's already
complicated enough to say "the following key did actually sign this
message part".  I'm still not sure whether the front-end or the backend
should be responsible for resolution of key->userid bindings, but i'm OK
punting on that question for the moment.

Multipart messages can have some parts signed and other parts not
signed: think of mailing lists which tack on a footer to each relayed
mail; the footer isn't signed, though the rest of the message is.

One MIME signature can cover more than one MIME part: Think of a signed
e-mail with an attachment. In this case, the signature is actually over
the aggregate, not the individual parts.  For example, a signed two-part
message that says:
 [ (A) "this is the budget for 2011", and (B) an attached spreadsheet ]
is *not* the same as either (A) or (B) signed independently.

A multipart MIME message can contain more than one distinct signature on
different parts:  Think of a digest of a mailing list discussion between
several participants who each sign their own messages.  Each signature
needs to be bound to the relevant parts (and vice versa); and some
signatures within a message can fail while others succeed.

A single application/pgp-signature part could contain signing material
from multiple signers.  Think of a PGP/MIME-signed key transition document.

MIME is actually a tree structure, and any subtree can be signed.  But
currently, "notmuch show" hides the tree structure and produces what
appears to be a linear set of parts.

Even more perversely, the tree structure means that a single MIME part
could potentially be signed by multiple signatures, each of which
potentially has independent origin and independent validity.

I've attached a moderately nasty e-mail message to this one
demonstrating a confluence of a bunch of these observations.

The structure of the attached e-mail looks like this:

A└┬╴multipart/signed 10936 bytes
B ├┬╴multipart/mixed 7403 bytes
C │├╴text/plain 77 bytes
D │├╴image/jpeg attachment [dkg.jpg] 4753 bytes
E │└┬╴message/rfc822 2072 bytes
F │ └┬╴multipart/signed 1914 bytes
G │  ├╴text/plain 57 bytes
H │  └╴application/pgp-signature attachment [signature.asc] 900 bytes
I └╴application/pgp-signature attachment [signature.asc] 900 bytes

"notmuch show" emits it as 5 parts (omitting A, B, E, and F):

 1: C
 2: D
 3: G
 4: H
 5: I

Note that while C and D are both signed by I, G is actually signed by
both H and I.  yuck.  And since this example message is attached to the
e-mail i'm writing right now (which itself will be signed) it can
certainly get even yuckier.



Questions:
----------

Am i missing any data or relationships you think we might want?

Is anything broken, unexpected, or dangerous about the choice of JSON
modifications?

I realize i've gone down a bit of a rabbit hole in the corner cases here
(driven mainly by my observations section).  Are there any simplifying
assumptions we can safely make about what kinds of messages are worth
verifying?  That is, are there ways to make this more intelligible that
don't throw away our ability to accurately represent the verified state
of some non-trivial subset of messages?

If this method (or something similar to it) gets put into the notmuch
backend, is this something we can actually represent to a human with a
reasonable frontend?

Would it make more sense to do deeper structural modifications of the
json output (e.g. return the full MIME tree instead of a list of parts)
than to go with the current proposal?

It would be nice to also make this kind of reporting structure also work
for S/MIME and maybe other crypto-signature structures like DKIM.  Is
that doable within this framework?  are there other tweaks we might want
to consider to cover that possibility?



If you actually read this far, you are a champion!  I look forward to
any feedback you have.

OK, off to bed!

	--dkg

[-- Attachment #1.2: Fwd: a test signature.eml --]
[-- Type: message/rfc822, Size: 10932 bytes --]

[-- Attachment #1.2.1.1.1: Type: text/plain, Size: 80 bytes --]

This message wraps another one.  and includes a picture of me.  woo!

	--dkg

[-- Attachment #1.2.1.1.2: dkg.jpg --]
[-- Type: image/jpeg, Size: 3515 bytes --]

[-- Attachment #1.2.1.1.3: a test signature.eml --]
[-- Type: message/rfc822, Size: 1913 bytes --]

[-- Attachment #1.2.1.1.3.1.1: Type: text/plain, Size: 61 bytes --]

here is a test message i signed on 2010-11-11.

  --dkg


[-- Attachment #1.2.1.1.3.1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 900 bytes --]

[-- Attachment #1.2.1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 900 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 900 bytes --]

Re: a proposed change to JSON output to report verification of PGP/MIME signatures.

[David Bremner]

On Sat, 13 Nov 2010 02:55:50 -0500, Daniel Kahn Gillmor <dkg@fifthhorseman.net> wrote:

> A signed MIME part will contain a new element "signedby", which is a
> list of part numbers identifying signatures that cover this part.
> 
> Signature parts (Content-Type: application/pgp-signature) will contain a
> new element "signs", which points back to the list of parts this
> signature covers.

Are both the forward and backward pointers needed?

> 
> Would it make more sense to do deeper structural modifications of the
> json output (e.g. return the full MIME tree instead of a list of parts)
> than to go with the current proposal?
> 

Yeah, this occured to me too, especially since I think David Edmondson
has some changes in mind to support better handling of
multipart/alternative parts.

d

Re: a proposed change to JSON output to report verification of PGP/MIME signatures.

[Daniel Kahn Gillmor]

[-- Attachment #1: Type: text/plain, Size: 1790 bytes --]

Thanks for the reply, David!

On 11/13/2010 06:40 AM, David Bremner wrote:
> Are both the forward and backward pointers needed?

Technically, only the "signs" pointer is needed, i guess.  I had
included the "signedby" pointer so that frontends which process the list
linearly know that a signed part will be referred to later by one or
more signatures.

If you think that's not actually useful, i'd be happy to drop the
"signedby" pointer.  What do other people think?

>> Would it make more sense to do deeper structural modifications of the
>> json output (e.g. return the full MIME tree instead of a list of parts)
>> than to go with the current proposal?
> 
> Yeah, this occured to me too, especially since I think David Edmondson
> has some changes in mind to support better handling of
> multipart/alternative parts.

Another related concern that occurred to me is that parts E and F in my
example mail are technically part of the aggregate that is signed by I,
but they apparently the end-user won't even know they exist (unless the
frontend uses "notmuch show --format=mbox").

This seems problematic if the message headers of the included message
have relevance to the content.  e.g. an e-mail that says "here's the
phishing attempt i received" (Received: would be relevant) or "Look at
the nasty things this guy said!" (From:, Subject:, and Date: would be
relevant at least).

David Edmonson, i'd be interested in hearing your proposal for
restructuring the output to see how it might interact with my pieces here.

I also found it interesting to consider the range of possible non-leaf
MIME types:

 https://secure.wikimedia.org/wikipedia/en/wiki/MIME#Multipart_messages
 http://www.iana.org/assignments/media-types/multipart/

	--dkg


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 900 bytes --]

Re: a proposed change to JSON output to report verification of PGP/MIME signatures.

[David Edmondson]

On Sat, 13 Nov 2010 02:55:50 -0500, Daniel Kahn Gillmor <dkg@fifthhorseman.net> wrote:
> It would end up like this (without the --verify flag):
> 
> ---------------------------
>  "body": [
>      {
>          "content": "here is a test message i signed on 2010-11-11.\n\n
>  --dkg\n\n",
>          "content-type": "text/plain",
>          "id": 1,
>          "signedby": [ 2 ]
>      },
>      {
>          "content-type": "application/pgp-signature",
>          "filename": "signature.asc",
>          "id": 2,
>          "signs": [ 1 ],
>          "sigstatus": [ {
>              "verified": "unknown"
>          } ]
>      }
>  ],
> ---------------------------

I think that this should become:

"body": [
    {
        "content-type": "multipart/signed",
        "id": 1,
        "content": [
             {
                 "content": "here is a test message i signed on 2010-11-11.\n\n --dkg\n\n",
                 "content-type": "text/plain",
                 "id": 2,
                 "signedby": [ 3 ],
             },
             {
                 "content-type": "application/pgp-signature",
                 "filename": "signature.asc",
                 "id": 3,
                 "signs": [ 2 ],
                 "sigstatus": [ {
                     "verified": "unknown"
                     } ]
              }
         ]
    }
],

i.e. the existence of the multipart/signed wrapper should be
explicit. In general, all MIME parts should be visible. Your email would
end up output something like
	http://dme.org/emacs/notmuch.multipart/thread.json
(well, that's the thread, but you can see your message as the first in
the thread).

We'd render that as shown in:
	http://dme.org/emacs/notmuch.multipart/screenshot.png
(the indentation of the parts there is optional - in this case it helps
a lot to show the structure).

The JSON output and the rendering are from a previous prototype (branch
'mp3' of https://github.com/dme/notmuch).

> and here it is with the --verify flag:

This would change in a similar manner (only the
application/pgp-signature part would change).

Changing the JSON output in this way would not materially affect your
proposal, I believe. There'd be some implicit changes in the output (for
example, if a signature signs a multipart/mixed part your proposal would
list it as signing the sub-parts of the multipart/mixed, but with my
additional changes it should be listed as signing the multipart/mixed
itself).

dme.
-- 
David Edmondson, http://dme.org

Re: a proposed change to JSON output to report verification of PGP/MIME signatures.

[Daniel Kahn Gillmor]

[-- Attachment #1: Type: text/plain, Size: 955 bytes --]

On 11/15/2010 05:23 AM, David Edmondson wrote:
> i.e. the existence of the multipart/signed wrapper should be
> explicit. In general, all MIME parts should be visible. 

thanks, this makes sense to me.

> Changing the JSON output in this way would not materially affect your
> proposal, I believe. There'd be some implicit changes in the output (for
> example, if a signature signs a multipart/mixed part your proposal would
> list it as signing the sub-parts of the multipart/mixed, but with my
> additional changes it should be listed as signing the multipart/mixed
> itself).

this is interesting: under your proposed changes, the "signs" element
would not need to be a list any more. it could just be a single part ID.
 I think i like that.

"sigstatus" would still need to be a list, though, since you can have a
signature part that contains multiple signature packets.

Thanks for helping think this through, David.

	--dkg


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 900 bytes --]

Re: a proposed change to JSON output to report verification of PGP/MIME signatures.

[Carl Worth]

[-- Attachment #1: Type: text/plain, Size: 1491 bytes --]

On Sat, 13 Nov 2010 02:55:50 -0500, Daniel Kahn Gillmor <dkg@fifthhorseman.net> wrote:
> i've been trying to wrap my head around how to get notmuch to support
> verifying cryptographically-signed mail.  i'm afraid my current
> understanding of the problem space is that it is neither pretty nor
> clean.  Sorry for the length of this message.

No apology necessary! I really appreciate you putting a lot of thought
into this.

[snip many details of proposal]

> MIME is actually a tree structure, and any subtree can be signed.  But
> currently, "notmuch show" hides the tree structure and produces what
> appears to be a linear set of parts.

The current linearization of parts is a bug that should be fixed. And I
think several aspects of your proposal are effectively workarounds for
this bug. So I'd rather we fix the json output to emit the tree
structure first, and then see what parts of the proposal can be
eliminated.

[And I think David Edmondson's reply said the same as above, but with
more detail. Right?]

> If you actually read this far, you are a champion!  I look forward to
> any feedback you have.

The only other piece I think I'd like to see is actually making the
content of the signature pieces available in the json output. Then, a
client could do its own verification.

Then if we had that would we not want to add the --verify support into
notmuch? (My guess is that we still would want it.)

-Carl

-- 
carl.d.worth@intel.com

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: a proposed change to JSON output to report verification of PGP/MIME signatures.

[Daniel Kahn Gillmor]

[-- Attachment #1: Type: text/plain, Size: 2617 bytes --]

On 11/16/2010 02:47 PM, Carl Worth wrote:
> The current linearization of parts is a bug that should be fixed. And I
> think several aspects of your proposal are effectively workarounds for
> this bug. So I'd rather we fix the json output to emit the tree
> structure first, and then see what parts of the proposal can be
> eliminated.
> 
> [And I think David Edmondson's reply said the same as above, but with
> more detail. Right?]

ok, good to know.  that makes sense to me, and i'll plan my work around
the idea of future tree-structured output.  i didn't know whether the
linearized output was considered a feature or not.  tree-structured
output makes me happier.

> The only other piece I think I'd like to see is actually making the
> content of the signature pieces available in the json output. Then, a
> client could do its own verification.

once we have tree-structured output, we'd only need to be able to
request a literal byte-stream of any mime-part.  so if the mime parts
were structured this way:

1└┬╴multipart/signed 80029 bytes
2 ├┬╴multipart/mixed 78178 bytes
3 │├╴text/plain 699 bytes
4 │└╴text/plain attachment [a_dancing_monkey.png] 76978 bytes
5 └╴application/pgp-signature attachment [signature.asc] 892 bytes

then the client would need to be able to extract a clean (untampered,
internal-headers included) copy of part 2, to verify against the
signature found in part 5.

Note that "notmuch part" currently emits only the content of the emitted
parts.  it strips off the internal mime headers, and apparently does
some form of content mangling too (base64 decoding, etc).  this is
unacceptable for the purposes of signature verification.  Maybe notmuch
part needs a --unfiltered flag or something?

Note also that clients that are willing and able to parse mime
structures themselves can already do the signature verification
themselves by fetching the whole thing with --format=mbox.

> Then if we had that would we not want to add the --verify support into
> notmuch? (My guess is that we still would want it.)

i think it's a good idea to enable verification on the backend, because
we don't want to force each frontend author to reinvent the wheel.  I
also think it's a good idea to enable the possibility of frontend
verification for frontends who want to do that (e.g. if i ever use a
notmuch-based webmail app, i want my OpenPGP keyring stored on my local
machine, not on the web server.  so i'd want my verification to happen
on the locally-trusted hardware, too, not on the web server.

	--dkg


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 900 bytes --]

Re: a proposed change to JSON output to report verification of PGP/MIME signatures.

[Jameson Rollins]

[-- Attachment #1: Type: text/plain, Size: 840 bytes --]

On Tue, 16 Nov 2010 11:47:13 -0800, Carl Worth <cworth@cworth.org> wrote:
> The only other piece I think I'd like to see is actually making the
> content of the signature pieces available in the json output. Then, a
> client could do its own verification.
> 
> Then if we had that would we not want to add the --verify support into
> notmuch? (My guess is that we still would want it.)

Hey, Carl.  I think your suggestion to include the signatures in the
output is a reasonable.  However, (I could be misunderstanding your
suggestion but) I really think the Right thing is for notmuch to do the
verification itself.  I would almost say that --verify should be the
default, with a --no-verify option.  It will make things much easier for
all the UIs if notmuch handles the verification and just outputs the
result.

jamie.

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

Re: a proposed change to JSON output to report verification of PGP/MIME signatures.

[Daniel Kahn Gillmor]

[-- Attachment #1: Type: text/plain, Size: 587 bytes --]

On 11/16/2010 03:10 PM, Jameson Rollins wrote:
> I would almost say that --verify should be the
> default, with a --no-verify option.  It will make things much easier for
> all the UIs if notmuch handles the verification and just outputs the
> result.

For clients that do not know how to interpret/display the verification
results, i don't want the backend to incur the cost of verification that
will be thrown away.

For clients that do understand how to interpret/display the verification
results, well, they can understand how to use the --verify flag too :)

	--dkg


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 900 bytes --]

Re: a proposed change to JSON output to report verification of PGP/MIME signatures.

[Jameson Rollins]

[-- Attachment #1: Type: text/plain, Size: 535 bytes --]

On Tue, 16 Nov 2010 15:22:49 -0500, Daniel Kahn Gillmor <dkg@fifthhorseman.net> wrote:
> For clients that do not know how to interpret/display the verification
> results, i don't want the backend to incur the cost of verification that
> will be thrown away.

Aren't clients going to have to interpret/display the output regardless
of it's been verified or not?  It seems to me that understanding how to
display the verified output is really not that much more difficult than
understanding how to display the unverified output.

jamie.

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

Re: a proposed change to JSON output to report verification of PGP/MIME signatures.

[Daniel Kahn Gillmor]

[-- Attachment #1: Type: text/plain, Size: 1565 bytes --]

On 11/16/2010 03:44 PM, Jameson Rollins wrote:
> Aren't clients going to have to interpret/display the output regardless
> of it's been verified or not?  It seems to me that understanding how to
> display the verified output is really not that much more difficult than
> understanding how to display the unverified output.

With json (and similar formats), it's easy to write a parser that says
"i know what to do with data member $foo -- give me that one".  This
lets you remain in blissful ignorance of data members $bar and $baz.

and if your backend suddenly starts throwing $qux at you as well, you
can just ignore it too, until you find you want to make use of it.

I imagine that someone writing a frontend would want to start with
things like message display, and not bother with fancier bits (such as
signature verification) until later.

frontends that know that their backend is somehow resource constrained
might also want to indicate to the user that a message claims to be
signed but not verify the signature unless the user asks for it.  (i
wish my current MUA had that feature, actually -- some messages i get
are signed but i don't personally need to verify them on every reload
(or even ever), depending on their content, but my MUA always makes me
wait for the verification process to happen.

To be clear -- i think that signature verification should be the default
situation for MUAs that are capable of doing it, but i don't think that
translates into --verify being on by default in
/usr/bin/notmuch.

	--dkg


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 900 bytes --]

Re: a proposed change to JSON output to report verification of PGP/MIME signatures.

[Daniel Kahn Gillmor]

[-- Attachment #1: Type: text/plain, Size: 2380 bytes --]

On 11/16/2010 03:06 PM, Daniel Kahn Gillmor wrote:
> On 11/16/2010 02:47 PM, Carl Worth wrote:
>> The current linearization of parts is a bug that should be fixed. And I
>> think several aspects of your proposal are effectively workarounds for
>> this bug. So I'd rather we fix the json output to emit the tree
>> structure first, and then see what parts of the proposal can be
>> eliminated.
>>
>> [And I think David Edmondson's reply said the same as above, but with
>> more detail. Right?]
> 
> ok, good to know.  that makes sense to me, and i'll plan my work around
> the idea of future tree-structured output.  i didn't know whether the
> linearized output was considered a feature or not.  tree-structured
> output makes me happier.

After a bit of working on this today, i realized that implementing
signature verification on the current linearized MIME output involves a
lot of wasted data tracking. i found myself adding ways to pass around
extra metadata to each part, but that won't be necessary if we get
tree-based mime in JSON.

So instead of implementing signature verification, i've successfully
rebased David Edmondson's mp3 patchset to notmuch 0.5.

That branch is published with the name of mp3-on-0.5 at my notmuch git repo:

 git://lair.fifthhorseman.net/~dkg/notmuch

The current head is commit id de836c032c43d0b2bc06553433ed4271ab54f3f3

The rebase was relatively clean (only a few manual interventions
necessary), and the bulk of my work was normalizing the part ID numbers
between output formats and cleaning up the test suite.

I've got this version working now, and the emacs interface works fine
for me.

Unless i hear objections to it, i'll be basing my future signature
verification work off of this branch.

I plan to attach the previously-proposed sigstatus member of the json
output to the parent element instead of the child elements.

So a message structured like this:

 1 └┬╴multipart/signed 1909 bytes
 2  ├╴text/plain 53 bytes
 3  └╴application/pgp-signature attachment [signature.asc] 900 bytes

part 1's json object would have the new sigstatus member, and parts 2
and 3 would not have any change from the current json format.

It looks like this will dramatically simplify the signature verification.

Feedback would be very much appreciated!

Regards,

	--dkg


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 900 bytes --]