bug#42637: 27.0.91; mm-view-pkcs7 doesn't handle S/MIME signed andd encrypted messages

bug#42637: 27.0.91; mm-view-pkcs7 doesn't handle S/MIME signed andd encrypted messages

[Timothy J. Miller]

An S/MIME message that is both enveloped (outer) and signed 
(inner) is not processed correctly. The outer envelop is 
decrypted, but this inserts the signed message into the buffer as 
an opaque blob, like so:

```
Content-Type: application/x-pkcs7-mime; name=smime.p7m; 
smime-type=signed-data
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename=smime.p7m

MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAaCAJIAEggEOQ29u
[...]
```


-- T

bug#42637: 27.0.91; mm-view-pkcs7 doesn't handle S/MIME signed andd encrypted messages

[Lars Ingebrigtsen]

"Timothy J. Miller" <tmiller@mitre.org> writes:

> An S/MIME message that is both enveloped (outer) and signed (inner) is
> not processed correctly. The outer envelop is decrypted, but this
> inserts the signed message into the buffer as an opaque blob, like so:
>
> ```
> Content-Type: application/x-pkcs7-mime; name=smime.p7m;
> smime-type=signed-data
> Content-Transfer-Encoding: base64
> Content-Disposition: attachment; filename=smime.p7m
>
> MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAaCAJIAEggEOQ29u
> [...]
> ```

This was supposed to be fixed a couple of days ago in Emacs 27 -- are
you using an up-to-date checkout?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#42637: [EXT] Re: bug#42637: 27.0.91; mm-view-pkcs7 doesn't handle S/MIME signed andd encrypted messages

[Timothy J. Miller]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> This was supposed to be fixed a couple of days ago in Emacs 27 
> -- are
> you using an up-to-date checkout?

I hadn't seen this patch since I was looking for the wrong 
keywords. I
see that it landed in rc1 so I updated over the weekend.

I'm still getting problems with enveloped+signed messages, but it 
looks
like only from Outlook so far.  I haven't been able to reproduce 
it yet.
Send me a signed message so I can send you encrypted samples when 
I can
manage to do so.

-- T

bug#42637: [EXT] Re: bug#42637: 27.0.91; mm-view-pkcs7 doesn't handle S/MIME signed andd encrypted messages

[Lars Ingebrigtsen]

[-- Attachment #1: Type: text/plain, Size: 458 bytes --]


"Timothy J. Miller" <tmiller@mitre.org> writes:

> I'm still getting problems with enveloped+signed messages, but it
> looks like only from Outlook so far.  I haven't been able to reproduce
> it yet.  Send me a signed message so I can send you encrypted samples
> when I can manage to do so.

OK, this should be an S/MIME signed message...

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 3631 bytes --]

bug#42637: [EXT] Re: bug#42637: 27.0.91; mm-view-pkcs7 doesn't handle S/MIME signed andd encrypted messages

[Timothy J Miller]

AFAICT so far, the problem seems to be Outlook but I can't narrow it down further than that since it seems to span multiple versions and platforms. 

I'll send you a clear signed, opaque signed, encrypted, and signed+encrypted direct.  I *think* only the signed+encrypted will have an issue.

Alternatively, are you aware of this:

https://tools.ietf.org/html/rfc4134

This test suite would be useful at your end, I should think.  Passing this suite won't eliminate all the bugs b/c every MUA has ... issues regarding S/MIME standards compliance, but it's a good place to start.

-- T

bug#42637: [EXT] Re: bug#42637: 27.0.91; mm-view-pkcs7 doesn't handle S/MIME signed andd encrypted messages

[Lars Ingebrigtsen]

Timothy J Miller <tmiller@mitre.org> writes:

> AFAICT so far, the problem seems to be Outlook but I can't narrow it
> down further than that since it seems to span multiple versions and
> platforms.
>
> I'll send you a clear signed, opaque signed, encrypted, and
> signed+encrypted direct.  I *think* only the signed+encrypted will
> have an issue.

Yeah, it looks like it -- with the signed+encrypted I just got a blank
buffer after decrypting.  Is that what you're also seeing?

> Alternatively, are you aware of this:
>
> https://tools.ietf.org/html/rfc4134
>
> This test suite would be useful at your end, I should think.  Passing
> this suite won't eliminate all the bugs b/c every MUA has ... issues
> regarding S/MIME standards compliance, but it's a good place to start.

It would be, yes.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#42637: [EXT] Re: bug#42637: 27.0.91; mm-view-pkcs7 doesn't handle S/MIME signed andd encrypted messages

[Timothy J Miller]

Sometimes, but I usually get a block of Base64 text.  I'll capture some samples tonight.

-- T

On 8/4/20, 10:41, "Lars Ingebrigtsen" <larsi@gnus.org> wrote:

    Timothy J Miller <tmiller@mitre.org> writes:

    > AFAICT so far, the problem seems to be Outlook but I can't narrow it
    > down further than that since it seems to span multiple versions and
    > platforms.
    >
    > I'll send you a clear signed, opaque signed, encrypted, and
    > signed+encrypted direct.  I *think* only the signed+encrypted will
    > have an issue.

    Yeah, it looks like it -- with the signed+encrypted I just got a blank
    buffer after decrypting.  Is that what you're also seeing?

    > Alternatively, are you aware of this:
    >
    > https://tools.ietf.org/html/rfc4134
    >
    > This test suite would be useful at your end, I should think.  Passing
    > this suite won't eliminate all the bugs b/c every MUA has ... issues
    > regarding S/MIME standards compliance, but it's a good place to start.

    It would be, yes.

    -- 
    (domestic pets only, the antidote for overdose, milk.)
       bloggy blog: http://lars.ingebrigtsen.no

bug#42637: [EXT] Re: bug#42637: 27.0.91; mm-view-pkcs7 doesn't handle S/MIME signed andd encrypted messages

[Lars Ingebrigtsen]

Timothy J Miller <tmiller@mitre.org> writes:

> Sometimes, but I usually get a block of Base64 text.  I'll capture
> some samples tonight.

Hm.  I only got blank messages...

Anyway, I found the problem -- Outlook used CRLF line endings on the
encrypted message itself, which messed up the parsing of the parts.
This should now be fixed in Emacs 28 (I'm pretty sure the CRLF->NL
transform is safe there -- there can't be binary parts that aren't
content-transfer-encoded, I think?)

But since you're seeing something else, perhaps there's more than one
bug here.  Can you do a new "git pull" and test?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#42637: [EXT] Re: bug#42637: 27.0.91; mm-view-pkcs7 doesn't handle S/MIME signed andd encrypted messages

[Timothy J Miller]

Got some free time so I rebuilt.  The CR/LF patch looks like it's working.  

However, the S/MIME signature status/trust marking in the article body are missing for signed+encrypted.  I went through a slew of old encrypted messages and most look like non-MIME messages.  Occasionally one will present with a text/plain attachment, but most not.  Attachments also seem to be decoding properly but I didn't test all examples, just spot checked a couple where the attachment got saved out as corrupted base64.

If I could add to the feature backlog, it would be nice to have a status mark for encrypted emails similar to the signature trust status marking. 

-- T

On 8/4/20, 11:11, "Lars Ingebrigtsen" <larsi@gnus.org> wrote:

    Timothy J Miller <tmiller@mitre.org> writes:

    > Sometimes, but I usually get a block of Base64 text.  I'll capture
    > some samples tonight.

    Hm.  I only got blank messages...

    Anyway, I found the problem -- Outlook used CRLF line endings on the
    encrypted message itself, which messed up the parsing of the parts.
    This should now be fixed in Emacs 28 (I'm pretty sure the CRLF->NL
    transform is safe there -- there can't be binary parts that aren't
    content-transfer-encoded, I think?)

    But since you're seeing something else, perhaps there's more than one
    bug here.  Can you do a new "git pull" and test?

    -- 
    (domestic pets only, the antidote for overdose, milk.)
       bloggy blog: http://lars.ingebrigtsen.no

bug#42637: [EXT] Re: bug#42637: 27.0.91; mm-view-pkcs7 doesn't handle S/MIME signed andd encrypted messages

[Lars Ingebrigtsen]

Timothy J Miller <tmiller@mitre.org> writes:

> However, the S/MIME signature status/trust marking in the article body
> are missing for signed+encrypted.  I went through a slew of old
> encrypted messages and most look like non-MIME messages.  Occasionally
> one will present with a text/plain attachment, but most not.
> Attachments also seem to be decoding properly but I didn't test all
> examples, just spot checked a couple where the attachment got saved
> out as corrupted base64.

Corrupted base64?  That doesn't sound good?  Is this a bug?

> If I could add to the feature backlog, it would be nice to have a
> status mark for encrypted emails similar to the signature trust status
> marking.

Do you mean the

[[S/MIME Signed Part:Undecided]]

stuff?

Hm, yes, that's missing from the encrypted+signed messages...  *sigh*

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#42637: [EXT] Re: bug#42637: 27.0.91; mm-view-pkcs7 doesn't handle S/MIME signed andd encrypted messages

[Timothy J. Miller]

Lars Ingebrigtsen <larsi@gnus.org> writes:

>
> Corrupted base64?  That doesn't sound good?  Is this a bug?
>

If it happens again and I can nail it down I'll submit it 
separately.
Attachments were being written out as blocks of unmarked base64, 
but
with periodic insertion of invalid byte sequences.  I only saw 
this
happen with opaque signed S/MIME from Outlook; attachments on 
messages
with detached signatures were fine.  With the CR/LF patch I went
looking for recurrence and so far have come up empty.

> Do you mean the
>
> [[S/MIME Signed Part:Undecided]]
>
> stuff?
>
> Hm, yes, that's missing from the encrypted+signed messages... 
> *sigh*

:thumbsup:

-- T

bug#42637: [EXT] Re: bug#42637: 27.0.91; mm-view-pkcs7 doesn't handle S/MIME signed andd encrypted messages

[Lars Ingebrigtsen]

"Timothy J. Miller" <tmiller@mitre.org> writes:

>> Hm, yes, that's missing from the encrypted+signed messages... *sigh*
>
> :thumbsup:

Looking at the test mails from you (thanks, by the way), I'm not 100%
sure which one of the test cases I should be looking at when debugging
this.

Can you send me an email that's encrypted+signed for sure?  :-)  

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#42637: [EXT] Re: bug#42637: 27.0.91; mm-view-pkcs7 doesn't handle S/MIME signed andd encrypted messages

[Lars Ingebrigtsen]

Thanks; got the encrypted+signed message now.

Gnus prompts me for decrypting twice -- is that also a bug, or is it
really encrypted twice?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#42637: [EXT] Re: bug#42637: 27.0.91; mm-view-pkcs7 doesn't handle S/MIME signed andd encrypted messages

[Timothy J Miller]

Only once, but IDK the processing flow.  Is it possible the second prompt was for the attachment?

-- T 

On 8/6/20, 08:14, "Lars Ingebrigtsen" <larsi@gnus.org> wrote:

    Thanks; got the encrypted+signed message now.

    Gnus prompts me for decrypting twice -- is that also a bug, or is it
    really encrypted twice?

    -- 
    (domestic pets only, the antidote for overdose, milk.)
       bloggy blog: http://lars.ingebrigtsen.no

bug#42637: [EXT] Re: bug#42637: 27.0.91; mm-view-pkcs7 doesn't handle S/MIME signed andd encrypted messages

[Lars Ingebrigtsen]

Timothy J Miller <tmiller@mitre.org> writes:

> Only once, but IDK the processing flow.  Is it possible the second
> prompt was for the attachment?

It's possible...  can you send another without an attachment?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#42637: [EXT] Re: bug#42637: 27.0.91; mm-view-pkcs7 doesn't handle S/MIME signed andd encrypted messages

[Lars Ingebrigtsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Timothy J Miller <tmiller@mitre.org> writes:
>
>> Only once, but IDK the processing flow.  Is it possible the second
>> prompt was for the attachment?
>
> It's possible...  can you send another without an attachment?

It's still decrypting twice, so that's another bug, I guess.  :-/

Well, that gives me something to use for debugging.  Thanks.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#42637: [EXT] Re: bug#42637: 27.0.91; mm-view-pkcs7 doesn't handle S/MIME signed andd encrypted messages

[Lars Ingebrigtsen]

OK, I've finally started poking around in this stuff again.

To recap: If you have a signed + encrypted message, you won't get the
buttons where you can click to ensure a signature.

The "outer" message (i.e., the email itself) is on this form:

Content-Type: application/pkcs7-mime; smime-type=enveloped-data;
	name="smime.p7m"
Content-Disposition: attachment; filename="smime.p7m"
Content-Transfer-Encoding: base64

This data is encrypted, and Gnus will ask you "Decrypt (S/MIME) part? "?

If "yes", you'll end up with a new, complete MIME message of this type:

Content-Type: application/pkcs7-mime;
	smime-type=signed-data;
	name=smime.p7m
Content-disposition: attachment;
	filename="smime.p7m"
Content-transfer-encoding: base64

The data here is not encrypted, but it is a binary blob containing the
mail text itself, and also the signature.  (Gnus will then ask you, again
"Decrypt (S/MIME) part? ", which is a bug, and which I've now fixed.)

To get the mail text itself, mm-view-pkcs7-verify is then called -- it
will do more than verify; it also extracts the mail from the binary
blob.

So at this point, Gnus has the mail text, and can display it.

However, there's no buttons, because to get the buttons, the mail has to
be a */signed part (which is a multipart kinda thing), and the mail has
already been verified.

So...  Hm...  I guess we could create some structure that would allow
re-checking the non-detached signature...

Well, that's as far as I've gotten.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#42637: [EXT] Re: bug#42637: 27.0.91; mm-view-pkcs7 doesn't handle S/MIME signed andd encrypted messages

[Lars Ingebrigtsen]

[-- Attachment #1: Type: text/plain, Size: 411 bytes --]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Well, that's as far as I've gotten.

Getting there to fixing the problem wasn't that hard.  It turns out that
mm-view-pkcs7-verify doesn't really report back the status when we're
using gpgsm here, so I fixed that, and I also made the mm dissection
return a multipart for signed + encrypted data, so that we have a way to
show the status.

So this now looks like:


[-- Attachment #2: Type: image/png, Size: 11383 bytes --]

[-- Attachment #3: Type: text/plain, Size: 268 bytes --]


And...  with that I think all the stuff we discussed in this bug report
has been covered, and if there's more problems in this area, please open
a new bug report.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no