filtering headers from forwarded messages

filtering headers from forwarded messages

[Daniel Kahn Gillmor]

[-- Attachment #1: Type: text/plain, Size: 2281 bytes --]

hey folks--

i recently had cause to forward a set of messages to a colleague via
notmuch (thank you for "notmuch-show-forward-open-messages"!), and
noticed that forwarding messages that i've personally received leaks
quite a bit of metadata about message delivery paths that is (a)
generally not useful when i'm forwarding in order to transfer the
message content, and (b) potentially harmful to users whose message
routing path reveals something bad or awkward about their setup.

For example, maybe for some people, their incoming mail path shows that
they're actually reading their personal e-mail on their employer's
mailsystems, but they don't want to expose their place of employment to
someone just by forwarding a message.  (this path is exposed by
Received: headers)

Or, there are internal headers added by local antispam or antimalware
filters, and they don't want to expose the specifics of their filtering
defenses because it might enable attacks on those systems (or customized
bypass mechanisms).

So, it occurs to me that someone might want to forward a message (or
messages) while filtering the headers in some way.

Of course, for messages being forwarded for the purpose of debugging the
transit path, you *don't* want to filter out headers.

In notmuch-emacs, i can manually filter the headers by editing the reply
compose buffer, of course, but it's kind of a pain, and it'd be nice to
have it done automatically for me.

Some possible filters i can imagine (which might well have problems, i
would appreciate any review):

 - blocklist: remove all headers that are in a fixed set: (Received,
      Delivered-To, Received-SPF, X-Original-To, Return-Path,
      X-Virus-Check-By, X-Virus-Scanned, Authentication-Results, X-MS-*,
      X-Microsoft-*)
 
 - allowlist: remove all headers except for a fixed set (To, From, Cc,
      Subject, Date, Message-Id, References, In-Reply-To, MIME-Version,
      Content-*, List-*, Sender)

 - ordered removal: remove all headers up to and including the last
   Received line


Has anyone else considered this use case, or thought about how to make
it easy/simple to do the right thing when using Notmuch?  Are there
other factors that are worth considering?

    --dkg

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

Re: filtering headers from forwarded messages

[Teemu Likonen]

[-- Attachment #1: Type: text/plain, Size: 1009 bytes --]

Daniel Kahn Gillmor [2019-12-20T13:50:03-05] wrote:

> In notmuch-emacs, i can manually filter the headers by editing the
> reply compose buffer, of course, but it's kind of a pain, and it'd be
> nice to have it done automatically for me.

> Has anyone else considered this use case, or thought about how to make
> it easy/simple to do the right thing when using Notmuch? Are there
> other factors that are worth considering?

The underlying message-mode has these variables:

    message-forward-as-mime
    message-forward-before-signature
    message-forward-ignored-headers
    message-forward-included-headers
    message-forward-show-mml

I have not studied those very closely but at least I know that when
-as-mime is nil user can use -included-headers to set what headers user
wants to include.


-- 
///  OpenPGP key: 4E1055DC84E9DFF613D78557719D69D324539450
//  https://keys.openpgp.org/search?q=tlikonen@iki.fi
/  https://keybase.io/tlikonen  https://github.com/tlikonen

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 694 bytes --]

Re: filtering headers from forwarded messages

[Daniel Kahn Gillmor]

[-- Attachment #1: Type: text/plain, Size: 1036 bytes --]

On Mon 2019-12-30 15:16:59 +0200, Teemu Likonen wrote:
> Daniel Kahn Gillmor [2019-12-20T13:50:03-05] wrote:
>
>> In notmuch-emacs, i can manually filter the headers by editing the
>> reply compose buffer, of course, but it's kind of a pain, and it'd be
>> nice to have it done automatically for me.
>
>> Has anyone else considered this use case, or thought about how to make
>> it easy/simple to do the right thing when using Notmuch? Are there
>> other factors that are worth considering?
>
> The underlying message-mode has these variables:
>
>     message-forward-as-mime
>     message-forward-before-signature
>     message-forward-ignored-headers
>     message-forward-included-headers
>     message-forward-show-mml
>
> I have not studied those very closely but at least I know that when
> -as-mime is nil user can use -included-headers to set what headers user
> wants to include.

Thanks for the pointer!  it looks like
message-forward-{ignored,included}-headers should do (roughly) what i
want.  I'll try them out.

    --dkg

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

Re: filtering headers from forwarded messages

[Daniel Kahn Gillmor]

[-- Attachment #1: Type: text/plain, Size: 364 bytes --]

On Wed 2020-01-08 10:25:50 -0500, Daniel Kahn Gillmor wrote:
> Thanks for the pointer!  it looks like
> message-forward-{ignored,included}-headers should do (roughly) what i
> want.  I'll try them out.

Just reporting back that they do indeed suit my purposes.  Things are
much nicer now that i've customized them to suit my tastes.

Thanks, Teemu!

        --dkg

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

Re: filtering headers from forwarded messages

[Daniel Kahn Gillmor]

[-- Attachment #1.1: Type: text/plain, Size: 998 bytes --]

On Thu 2020-01-16 15:20:49 -0500, Daniel Kahn Gillmor wrote:
> On Wed 2020-01-08 10:25:50 -0500, Daniel Kahn Gillmor wrote:
>> Thanks for the pointer!  it looks like
>> message-forward-{ignored,included}-headers should do (roughly) what i
>> want.  I'll try them out.
>
> Just reporting back that they do indeed suit my purposes.  Things are
> much nicer now that i've customized them to suit my tastes.

Hm, it now looks to me like message-forward-ignored-headers isn't
working since i upgraded to emacs 27.1.  In particular, since:

   https://lists.defectivebydesign.org/archive/html/emacs-diffs/2018-04/msg00135.html

it now says:

>> This variable is only consulted when forwarding "normally", not when
>> forwarding as MIME or the like.

Since i have message-forward-as-mime set (this used to be the default,
but i guess it changed in emacs 27 as well -- i definitely prefer to
forward as MIME) i now no longer have this filter working.

Any suggestions on how to recover this?

       --dkg

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: filtering headers from forwarded messages

[Teemu Likonen]

[-- Attachment #1.1: Type: text/plain, Size: 2189 bytes --]

* 2020-12-24 12:56:09-0500, Daniel Kahn Gillmor wrote:

>> On Wed 2020-01-08 10:25:50 -0500, Daniel Kahn Gillmor wrote:
>>> Thanks for the pointer! it looks like
>>> message-forward-{ignored,included}-headers should do (roughly) what
>>> i want. I'll try them out.

> Hm, it now looks to me like message-forward-ignored-headers isn't
> working since i upgraded to emacs 27.1. In particular, since:
>
> https://lists.defectivebydesign.org/archive/html/emacs-diffs/2018-04/msg00135.html

I don't know any automatic way to remove headers from an email message
that is stored as a MIME part.

When a forwarded message is stored as inline text (setq
message-forward-as-mime nil) then the message is rendered at the time of
composing the message. Unnecessary headers are removed, character
encodings are decoded etc.

When the forwarded message is stored as MIME type "message/rfc822" (setq
message-forward-as-mime t) then the message is meant to be rendered by
the receiver's email program which will decode all necessary headers,
especially MIME headers, convert between character sets and pretty-print
the message's headers and some of the MIME "text/*" parts.

Filtering or editing headers of email MIME part (message/rfc822) can be
tricky: there are message's main headers which tell the "Content-Type"
of the body, and the body can contain different MIME parts with some of
their own headers. If we filter too much or convert between character
sets the message is not proper message/rfc822 part anymore and can't be
rendered correctly.

What about forwarding a message as MIME part which is just "text/plain"
(and not "message/rfc822")? At least this can be done by setting (setq
message-forward-as-mime nil) and manually inserting Emacs MML tags in
the (notmuch-)message-mode buffer:

    C-c RET p text/plain RET

or calling from Lisp code:

    (mml-insert-part "text/plain")

The inserted MML tags need to be put manually around the forwarded
message. With some hackery one could write a semi-automatic function for
that.

-- 
/// Teemu Likonen - .-.. https://www.iki.fi/tlikonen/
// OpenPGP: 4E1055DC84E9DFF613D78557719D69D324539450

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 251 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: filtering headers from forwarded messages

[Daniel Kahn Gillmor]

[-- Attachment #1.1: Type: text/plain, Size: 1483 bytes --]

Hi Teemu--

On Wed 2020-12-30 12:46:02 +0200, Teemu Likonen wrote:
> I don't know any automatic way to remove headers from an email message
> that is stored as a MIME part.

The straightforward way seems to be to parse/interpret the message with
our knowledge of rfc5322, identify the headers to be removed, and then
re-render the message.

iirc, it worked for me until i started using emacs 27.1, too ☹

Obviously, if "Content-Type" or other structural headers are removed,
you'll break the message.  But the answer there is "don't do that", i
think.

> What about forwarding a message as MIME part which is just "text/plain"
> (and not "message/rfc822")?

This might be useful for some people, but doesn't really satisfy my
goals.  When i want to forward a message, i want to forward the whole
message -- multipart, with attachments, etc.  My goal when forwarding is
to *not* mangle the message, but rather to supply it to the new
recipient in a parseable way.  I just don't think that most recipients
need to have access to (for example) the headers that are added by all
the mail transport agents along my receipt path.  I want them filtered
for privacy, which i think isn't unreasonable.  (i recognize that if the
use case is debugging the MTA path itself, i probably *don't* want those
headers stripped, but that's a different use case than the most common
message forwarding)

so i'm still stuck with this regression, sadly.

        --dkg

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: filtering headers from forwarded messages

[Teemu Likonen]

[-- Attachment #1.1: Type: text/plain, Size: 1567 bytes --]

* 2020-12-31 17:39:23-0500, Daniel Kahn Gillmor wrote:

> On Wed 2020-12-30 12:46:02 +0200, Teemu Likonen wrote:
>> What about forwarding a message as MIME part which is just "text/plain"
>> (and not "message/rfc822")?
>
> This might be useful for some people, but doesn't really satisfy my
> goals.  When i want to forward a message, i want to forward the whole
> message -- multipart, with attachments, etc.  My goal when forwarding is
> to *not* mangle the message, but rather to supply it to the new
> recipient in a parseable way.  I just don't think that most recipients
> need to have access to (for example) the headers that are added by all
> the mail transport agents along my receipt path.  I want them filtered
> for privacy, which i think isn't unreasonable.

OK, that is reasonable. Somehow I thought that you wanted a
pretty-printed message (old inline style) but still valid email form.
But yes, it would be nice to filter headers like "Received", maybe
"DKIM-Signature" and various unofficial headers starting with "X-".

For a quick first step I would probably try running "C-u M-|"
(shell-command-on-region) in Emacs and pipe the forwarded message
through command like this:

    formail -I Received -I DKIM-Signature -I X-Whatever

Probably after that I would integrate that shell command call in one
Emacs command and then implement the functionality in Emacs Lisp. But so
far I haven't cared enough.

-- 
/// Teemu Likonen - .-.. https://www.iki.fi/tlikonen/
// OpenPGP: 4E1055DC84E9DFF613D78557719D69D324539450

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 251 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: filtering headers from forwarded messages

[Daniel Kahn Gillmor]

[-- Attachment #1.1: Type: text/plain, Size: 1289 bytes --]

On Sun 2021-01-03 09:48:56 +0200, Teemu Likonen wrote:
> But yes, it would be nice to filter headers like "Received", maybe
> "DKIM-Signature" and various unofficial headers starting with "X-".

Right, my current value of message-forward-ignored-headers is:

    ("^Content-Transfer-Encoding:" "^X-Gnus:" "^Received:"
     "^Return-Path:" "^X-Original-To:" "^Delivered-To:"
     "^X-Virus-Check-By:" "^Received-SPF:")

(i don't care to strip out DKIM-Signature headers because i see that as
part of the initial message injection into the SMTP system)

> For a quick first step I would probably try running "C-u M-|"
> (shell-command-on-region) in Emacs and pipe the forwarded message
> through command like this:
>
>     formail -I Received -I DKIM-Signature -I X-Whatever
>
> Probably after that I would integrate that shell command call in one
> Emacs command and then implement the functionality in Emacs Lisp. But so
> far I haven't cared enough.

This all sounds like it would work, but it seems like a lot of trouble
compared to fixing a regression.  this filtering worked just fine for
me before i upgraded to emacs 27.1.

I've filed a report about the regression with emacs upstream:
https://debbugs.gnu.org/45631

Thanks for helping me think through the details, Teemu.

   --dkg

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]