bug#45631: 27.1; regression: message-forward-ignored-headers no longer applies when forwarding as MIME

bug#45631: 27.1; regression: message-forward-ignored-headers no longer applies when forwarding as MIME

[Daniel Kahn Gillmor]

[-- Attachment #1: Type: text/plain, Size: 970 bytes --]

I've been using emacs as my MUA for years.

When I upgraded from emacs 26.3 to emacs 27.1, i noticed that
message-forward-as-mime now defaults to nil.  I prefer to forward as
MIME generally, so i set it back to t.

The only downside to forwarding as MIME is the inclusion of some headers
that the received message has accumulated in transit, which might have
privacy-sensitive implications. I've been using
message-forward-ignored-headers for a while now to trim out headers like
Received and Delivered-To when forwarding.

But as of 27.1, message-forward-ignored-headers doesn't work when
forwarding as MIME.  indeed, the help text for the variable now says

 >> This variable is only consulted when forwarding "normally", not when
 >> forwarding as MIME or the like.

But this is a regression from 26.3.  I'd expect it to keep working.

Please restore the functionality so that i can automatically strip
privacy-sensitive headers when forwarding.

Thanks,

     --dkg

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

bug#45631: 27.1; regression: message-forward-ignored-headers no longer applies when forwarding as MIME

[Robert Pluim]

Daniel Kahn Gillmor <dkg@fifthhorseman.net> writes:

> I've been using emacs as my MUA for years.
>
> When I upgraded from emacs 26.3 to emacs 27.1, i noticed that
> message-forward-as-mime now defaults to nil.  I prefer to forward as
> MIME generally, so i set it back to t.
>
> The only downside to forwarding as MIME is the inclusion of some headers
> that the received message has accumulated in transit, which might have
> privacy-sensitive implications. I've been using
> message-forward-ignored-headers for a while now to trim out headers like
> Received and Delivered-To when forwarding.
>
> But as of 27.1, message-forward-ignored-headers doesn't work when
> forwarding as MIME.  indeed, the help text for the variable now says
>
>  >> This variable is only consulted when forwarding "normally", not when
>  >> forwarding as MIME or the like.
>
> But this is a regression from 26.3.  I'd expect it to keep working.
>
> Please restore the functionality so that i can automatically strip
> privacy-sensitive headers when forwarding.

Iʼve compared emacs-26 and emacs-27, and the code is the same, which
leads me to suspect something different in your
configuration. 'message-forward-ignored-headers' is applied even when
forwarding as MIME (despite the docstring), except when
'message-forward-show-mml' is nil, or when itʼs 'best' and the
forwarded message is either signed or encrypted. Or maybe youʼre
forwarding from inside the *Article* buffer, I think Gnus behaves
differently then.

Robert

bug#45631: 27.1; regression: message-forward-ignored-headers no longer applies when forwarding as MIME

[Daniel Kahn Gillmor]

[-- Attachment #1: Type: text/plain, Size: 1736 bytes --]

Hi Robert--

Thanks for taking a look into this!

On Tue 2021-01-05 14:06:54 +0100, Robert Pluim wrote:
> Iʼve compared emacs-26 and emacs-27, and the code is the same, which
> leads me to suspect something different in your
> configuration. 'message-forward-ignored-headers' is applied even when
> forwarding as MIME (despite the docstring)

hm, the docstring change was recent, apparently in response to #27715 :

https://git.savannah.gnu.org/cgit/emacs.git/commit/?id=57fbf0cf7bd4a85f2ad6f14aa92494545106b887

that's why i assumed there had been a change.

> except when 'message-forward-show-mml' is nil, or when itʼs 'best' and
> the forwarded message is either signed or encrypted.

Hm, i'm using 'best' for message-forward-show-mml (as the default) and
yes, it looks like the issue is that i just noticed it happening when
i went to forward a signed message.  Maybe it wasn't an issue before
because i wasn't forwarding a signed message?  I no longer have emacs
26.3 installed so i can't check that handily right now.

The message headers (outside of the cryptographic envelope) do *not*
affect the digital signature, so they ought to be safe to trim out
without invalidating the digital signature.  These are the message
headers that i want to trim.

> Or maybe youʼre forwarding from inside the *Article* buffer, I think
> Gnus behaves differently then.

I'm using notmuch-emacs, not gnus, but it reuses a lot of the existing
emacs MUA codebase, which is why i'm reporting it here.

Do you have a suggestion for how i can apply
message-forward-ignored-headers to a signed message?  I only want it to
apply to headers that aren't covered by the digital signature anyway.

      --dkg

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

bug#45631: 27.1; regression: message-forward-ignored-headers no longer applies when forwarding as MIME

[Robert Pluim]

Daniel Kahn Gillmor <dkg@fifthhorseman.net> writes:

> Hi Robert--
>
> Thanks for taking a look into this!
>
> On Tue 2021-01-05 14:06:54 +0100, Robert Pluim wrote:
>> Iʼve compared emacs-26 and emacs-27, and the code is the same, which
>> leads me to suspect something different in your
>> configuration. 'message-forward-ignored-headers' is applied even when
>> forwarding as MIME (despite the docstring)
>
> hm, the docstring change was recent, apparently in response to #27715 :
>
> https://git.savannah.gnu.org/cgit/emacs.git/commit/?id=57fbf0cf7bd4a85f2ad6f14aa92494545106b887
>
> that's why i assumed there had been a change.
>

Documentation lags implementation, as always :-)

>> except when 'message-forward-show-mml' is nil, or when itʼs 'best' and
>> the forwarded message is either signed or encrypted.
>
> Hm, i'm using 'best' for message-forward-show-mml (as the default) and
> yes, it looks like the issue is that i just noticed it happening when
> i went to forward a signed message.  Maybe it wasn't an issue before
> because i wasn't forwarding a signed message?  I no longer have emacs
> 26.3 installed so i can't check that handily right now.
>
> The message headers (outside of the cryptographic envelope) do *not*
> affect the digital signature, so they ought to be safe to trim out
> without invalidating the digital signature.  These are the message
> headers that i want to trim.
>
>> Or maybe youʼre forwarding from inside the *Article* buffer, I think
>> Gnus behaves differently then.
>
> I'm using notmuch-emacs, not gnus, but it reuses a lot of the existing
> emacs MUA codebase, which is why i'm reporting it here.
>
> Do you have a suggestion for how i can apply
> message-forward-ignored-headers to a signed message?  I only want it to
> apply to headers that aren't covered by the digital signature anyway.

I think setting message-forward-show-mml to t will do what you want,
then message won't bother to check if the message is signed/encrypted,
and will thus apply message-forward-ignored-headers.

Robert

bug#45631: 27.1; regression: message-forward-ignored-headers no longer applies when forwarding as MIME

[Lars Ingebrigtsen]

Robert Pluim <rpluim@gmail.com> writes:

> I think setting message-forward-show-mml to t will do what you want,
> then message won't bother to check if the message is signed/encrypted,
> and will thus apply message-forward-ignored-headers.

Looking over that mess again, I think my analysis of when that variable
is used, and when it's supposed to be used, was wrong.  I've now make it
respect that variable even if message-forward-show-mml is nil.  The only
instance it won't be used is if message-forward-show-mml is `best', and
we're forwarding an encrypted/signed message.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#45631: 27.1; regression: message-forward-ignored-headers no longer applies when forwarding as MIME

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Date: Sun, 10 Jan 2021 16:18:23 +0100
> Cc: 45631@debbugs.gnu.org, Daniel Kahn Gillmor <dkg@fifthhorseman.net>
> 
> Robert Pluim <rpluim@gmail.com> writes:
> 
> > I think setting message-forward-show-mml to t will do what you want,
> > then message won't bother to check if the message is signed/encrypted,
> > and will thus apply message-forward-ignored-headers.
> 
> Looking over that mess again, I think my analysis of when that variable
> is used, and when it's supposed to be used, was wrong.  I've now make it
> respect that variable even if message-forward-show-mml is nil.

Should this be fixed on the emacs-27 branch?

bug#45631: 27.1; regression: message-forward-ignored-headers no longer applies when forwarding as MIME

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

>> Looking over that mess again, I think my analysis of when that variable
>> is used, and when it's supposed to be used, was wrong.  I've now make it
>> respect that variable even if message-forward-show-mml is nil.
>
> Should this be fixed on the emacs-27 branch?

If it doesn't introduce any regressions, but my confidence here isn't
exactly 100%...  I think we should wait a few weeks before backporting,
at least.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#45631: 27.1; regression: message-forward-ignored-headers no longer applies when forwarding as MIME

[Daniel Kahn Gillmor]

[-- Attachment #1: Type: text/plain, Size: 1320 bytes --]

Hi Lars--

On Sun 2021-01-10 16:18:23 +0100, Lars Ingebrigtsen wrote:
> Looking over that mess again, I think my analysis of when that variable
> is used, and when it's supposed to be used, was wrong.  I've now make it
> respect that variable even if message-forward-show-mml is nil.  The only
> instance it won't be used is if message-forward-show-mml is `best', and
> we're forwarding an encrypted/signed message.

Thanks for taking a look at this.  I'm trying to understand the
rationale for *not* trimming headers when message-forward-show-mml is
`best' and we're forwarding an encrypted/signed message.

If the headers being trimmed are strictly in the header section of the
forwarded message, then they aren't in the cryptographic envelope [0],
which means that they aren't implicated in either a standard PGP/MIME or
S/MIME signature or encryption.

Is the expectation that headers of internal parts of the message are
being trimmed (in which case, they might be implicated in the signature
or encryption)?  or, is there some situation i'm missing where they
might have an impact on the cryptographic structure?

      --dkg

[0]
    https://www.ietf.org/archive/id/draft-dkg-lamps-e2e-mail-guidance-00.html
    has definitions of "cryptographic envelope" and other
    hopefully-useful concepts and terminology.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

bug#45631: 27.1; regression: message-forward-ignored-headers no longer applies when forwarding as MIME

[Lars Ingebrigtsen]

Daniel Kahn Gillmor <dkg@fifthhorseman.net> writes:

> Is the expectation that headers of internal parts of the message are
> being trimmed (in which case, they might be implicated in the signature
> or encryption)?  or, is there some situation i'm missing where they
> might have an impact on the cryptographic structure?

Well, we'd have to include the relevant headers at a minimum:

  Content-Type: multipart/signed; boundary="=-=-=";
	micalg=pgp-sha256; protocol="application/pgp-signature"

But this code is almost a couple of decades old, and I have no idea what
the thought process behind this was at this date.  Anybody know?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#45631: 27.1; regression: message-forward-ignored-headers no longer applies when forwarding as MIME

[Daniel Kahn Gillmor]

[-- Attachment #1: Type: text/plain, Size: 1661 bytes --]

On Mon 2021-01-11 16:09:18 +0100, Lars Ingebrigtsen wrote:
> Well, we'd have to include the relevant headers at a minimum:
>
>   Content-Type: multipart/signed; boundary="=-=-=";
> 	micalg=pgp-sha256; protocol="application/pgp-signature"

I agree, but if the user is stripping the Content-Type header, i think
they're going to break a lot more than digital signatures or encryption
(imagine what that does to a multipart/alternative message).  I think
that stripping Content-Type is more of a case of "don't do that, then".
Maybe we even want to warn if the user tries to strip any of the
Content-* headers more generally.

> But this code is almost a couple of decades old, and I have no idea what
> the thought process behind this was at this date.  Anybody know?

As long as the code doesn't attempt to strip *internal* MIME headers
(that is, headers of subparts of the MIME structure) i think it should
be safe to apply it to the forwarded message.  

Note also that if we care about breakng cryptographic signatures more
generally, DKIM signatures are *more* likely to break if headers are
stripped than PGP/MIME or S/MIME, as DKIM is capable of covering headers
directly.  Even given that concern, i think the most we'd want the
"best" setting to do to constrain header stripping would be to compare
the stripped version of the file to the non-stripped version -- if the
non-stripped version passes DKIM validation, but the stripped does not,
then either produce a warning message about DKIM signature breakage or
(if in an interactive mode) prompt the user about whether they want to
apply the filter or not.

      --dkg

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

bug#45631: 27.1; regression: message-forward-ignored-headers no longer applies when forwarding as MIME

[Robert Pluim]

Daniel Kahn Gillmor <dkg@fifthhorseman.net> writes:

> generally, DKIM signatures are *more* likely to break if headers are
> stripped than PGP/MIME or S/MIME, as DKIM is capable of covering headers
> directly.  Even given that concern, i think the most we'd want the
> "best" setting to do to constrain header stripping would be to compare
> the stripped version of the file to the non-stripped version -- if the
> non-stripped version passes DKIM validation, but the stripped does not,
> then either produce a warning message about DKIM signature breakage or
> (if in an interactive mode) prompt the user about whether they want to
> apply the filter or not.

Adding unconditional checking of DKIM signatures to Emacs when
forwarding a message is a no-no. Think of people who work on email
when offline, or people who donʼt want to advertise the fact that
they've received email from a particular domain.

Robert

bug#45631: 27.1; regression: message-forward-ignored-headers no longer applies when forwarding as MIME

[Daniel Kahn Gillmor]

[-- Attachment #1: Type: text/plain, Size: 702 bytes --]

On Wed 2021-01-13 10:25:51 +0100, Robert Pluim wrote:
> Adding unconditional checking of DKIM signatures to Emacs when
> forwarding a message is a no-no. Think of people who work on email
> when offline, or people who donʼt want to advertise the fact that
> they've received email from a particular domain.

Agreed!  I'm not actually suggesting that we want to do DKIM signature
checking -- i'm just pointing out that if we're worried about header
removal damaging any sort of cryptographic signature, that's the most
likely scenario where that'll happen.  The other forms of cryptographic
signature (PGP/MIME and S/MIME) aren't likely to be affected by header
stripping.

        --dkg

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

bug#45631: 27.1; regression: message-forward-ignored-headers no longer applies when forwarding as MIME

[Lars Ingebrigtsen]

Daniel Kahn Gillmor <dkg@fifthhorseman.net> writes:

>>   Content-Type: multipart/signed; boundary="=-=-=";
>> 	micalg=pgp-sha256; protocol="application/pgp-signature"
>
> I agree, but if the user is stripping the Content-Type header, i think
> they're going to break a lot more than digital signatures or encryption
> (imagine what that does to a multipart/alternative message).  I think
> that stripping Content-Type is more of a case of "don't do that, then".
> Maybe we even want to warn if the user tries to strip any of the
> Content-* headers more generally.

This is bringing back memories...

When forwarding messages, we have three cases:

1) Forward as non-MIME

Heeding message-forward-included-headers is unproblematic here: The
recipient isn't going to be able to reconstruct the original message at
all, since none of the MIME headers (and stuff) is included in
`message-forward-included-headers'.  This basically means that the
recipient can just read the text itself.

2) MIME, but `message-forward-show-mml'

We can also remove all the headers here, because Message will
reconstruct all the MIME headers needed for the recipient to reconstruct
the entire original message, with multipart bits and all.

3) MIME, but not `message-forward-show-mml'

We don't remove any headers, because we don't know which ones are needed
for the recipient to actually read the resulting nested message.
Certainly not based on `message-forward-included-headers' -- this just
leaves From/Subject/etc.

So...  I think what message was doing before my last change was correct,
but insufficiently documented: If we have `message-forward-as-mime', but
not `message-forward-show-mml', then we can't remove headers based on
`message-forward-included-headers', because the resulting message will
be invalid.

If headers are to be removed in 3), then we have to come up with a
`message-forward-headers-not-removed-when-using-mime-but-not-mml' (phew)
variable, that contains the headers necessary for reconstructing the
resulting message.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#45631: 27.1; regression: message-forward-ignored-headers no longer applies when forwarding as MIME

[Daniel Kahn Gillmor]

[-- Attachment #1: Type: text/plain, Size: 869 bytes --]

I'm getting a little confused here, so i'm going to back up and try to
simplify.  There are two distinct (yet somehow overlapping) variables:

 - message-forward-included-headers (an "allowlist")
 - message-forward-ignored-headers (a "blocklist")

Can we agree on that?

Either or both (or neither) of these variables might be applied in any
of the three scenarios.  If both are applied, i have no idea which order
they are applied in, so applying both is probably not a great idea. ☹

The simplest approach would be the following:

On Mon 2021-01-18 16:50:10 +0100, Lars Ingebrigtsen wrote:
> 1) Forward as non-MIME

apply the allowlist.

> 2) MIME, but `message-forward-show-mml'

apply the blocklist.

> 3) MIME, but not `message-forward-show-mml'

apply the blocklist.

What are the downsides to approaching it this way?

     --dkg

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

bug#45631: 27.1; regression: message-forward-ignored-headers no longer applies when forwarding as MIME

[Lars Ingebrigtsen]

Daniel Kahn Gillmor <dkg@fifthhorseman.net> writes:

> Either or both (or neither) of these variables might be applied in any
> of the three scenarios.  If both are applied, i have no idea which order
> they are applied in, so applying both is probably not a great idea. ☹

Both are applied.

> The simplest approach would be the following:
>
> On Mon 2021-01-18 16:50:10 +0100, Lars Ingebrigtsen wrote:
>> 1) Forward as non-MIME
>
> apply the allowlist.
>
>> 2) MIME, but `message-forward-show-mml'
>
> apply the blocklist.
>
>> 3) MIME, but not `message-forward-show-mml'
>
> apply the blocklist.
>
> What are the downsides to approaching it this way?

We can't change when they're applied, because that will break people's
setups.  So we have to introduce a new variable for 3).

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#45631: 27.1; regression: message-forward-ignored-headers no longer applies when forwarding as MIME

[Daniel Kahn Gillmor]

On Tue 2021-01-19 04:12:02 +0100, Lars Ingebrigtsen wrote:
> Daniel Kahn Gillmor <dkg@fifthhorseman.net> writes:
>
>> Either or both (or neither) of these variables might be applied in any
>> of the three scenarios.  If both are applied, i have no idea which order
>> they are applied in, so applying both is probably not a great idea. ☹
>
> Both are applied.

In all cases?  I think neither are applied in case 3, right?

At any rate, if they're typically applied in tandem, it seems like their
help text should probably refer to each other -- this discussion is the
first place i've learned about message-forward-included-headers :/

> We can't change when they're applied, because that will break people's
> setups.  So we have to introduce a new variable for 3).

In case 3, neither are currently applied.

The only thing i'm asking for is to apply *only* the blocklist in case
3.  What setups will that break?

    --dkg

bug#45631: 27.1; regression: message-forward-ignored-headers no longer applies when forwarding as MIME

[Lars Ingebrigtsen]

Daniel Kahn Gillmor <dkg@fifthhorseman.net> writes:

>>> Either or both (or neither) of these variables might be applied in any
>>> of the three scenarios.  If both are applied, i have no idea which order
>>> they are applied in, so applying both is probably not a great idea. ☹
>>
>> Both are applied.
>
> In all cases?  I think neither are applied in case 3, right?
>
> At any rate, if they're typically applied in tandem, it seems like their
> help text should probably refer to each other -- this discussion is the
> first place i've learned about message-forward-included-headers :/

Indeed.

>> We can't change when they're applied, because that will break people's
>> setups.  So we have to introduce a new variable for 3).
>
> In case 3, neither are currently applied.
>
> The only thing i'm asking for is to apply *only* the blocklist in case
> 3.  What setups will that break?

It's just surprising behaviour -- that Message will strip a completely
different set of headers based on what forwarding method is used.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#45631: 27.1; regression: message-forward-ignored-headers no longer applies when forwarding as MIME

[Daniel Kahn Gillmor]

On Wed 2021-01-20 17:17:55 +0100, Lars Ingebrigtsen wrote:
> It's just surprising behaviour -- that Message will strip a completely
> different set of headers based on what forwarding method is used.

I agree that the current behavior is already surprising.

I think it reduces the amount of surprise to have the blocklist applied
when forwarding as MIME.

     --dkg

bug#45631: 27.1; regression: message-forward-ignored-headers no longer applies when forwarding as MIME

[Lars Ingebrigtsen]

Daniel Kahn Gillmor <dkg@fifthhorseman.net> writes:

> On Wed 2021-01-20 17:17:55 +0100, Lars Ingebrigtsen wrote:
>> It's just surprising behaviour -- that Message will strip a completely
>> different set of headers based on what forwarding method is used.
>
> I agree that the current behavior is already surprising.
>
> I think it reduces the amount of surprise to have the blocklist applied
> when forwarding as MIME.

The blocklist isn't very extensive -- the allowlist should also be
applied when forwarding as MIME, but that's not straightforward as
things are now.

I've now introduced the new variable, and headers seem to be
removed/kept in all the combinations of the variables now, I hope.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no