What to do about message/rfc822 attachments?

What to do about message/rfc822 attachments?

[Andy Smith]

Hi,

In this age of DKIM and DMARC, some mailing lists have decided to
wrap the original message into a message/rfc822 attachment before
sending it out to subscribers.

Some do this unconditionally, for every message received. Some do it
conditionally, for example only when the poster's domain has a DMARC
policy of "reject".

Popular mailing list software such as Mailman (both v2 and v3) offer
both of those modes of operation. I am on multiple mailing lists
that use both methods, though it is still a minority.

What this means is that I see a message that looks like this:

    From: Joe Bloggs via somelist <somelist@lists.example.com>
    Message-ID: <whatever@lists.example.com>
    Subject: Joe's original subject
    References: (from Joe's post)
    In-Reply-To: (from Joe's post)

That then contains a message/rfc822 attachment that contains the
full original copy of Joe's post, with all its headers intact.

I am a relatively new notmuch user. Am I correct in thinking that
notmuch will look inside message/rfc822 containers and find the
text/plain (and/or text/html) parts within and index them for full
text search just as it would the message body in a conventional
message?

The threading should also work as intended too because the
References/In-Reply-To are retained.

So really I think the only wrinkle is that I need to remember that
searching by a person's email address is not going to find all
messages by them, because ones they send to such a list will not
actually appear to notmuch to be from them, but instead from the
lists's address.

That is already an issue with mailing lists that rewrite the From:
address (again for DKIM/DMARC reasons), but I was wondering if there
were anything that people do with notmuch to improve matters given
that all the info does actually exist in these wrapped messages. Is
it somehow possible, or would be a worthwhile idea to, look for
From: addresses and other useful headers also inside message/rfc822
containers?

Thanks,
Andy

Re: What to do about message/rfc822 attachments?

[David Bremner]

Andy Smith <andy@strugglers.net> writes:

>
> What this means is that I see a message that looks like this:
>
>     From: Joe Bloggs via somelist <somelist@lists.example.com>
>     Message-ID: <whatever@lists.example.com>
>     Subject: Joe's original subject
>     References: (from Joe's post)
>     In-Reply-To: (from Joe's post)
>
> That then contains a message/rfc822 attachment that contains the
> full original copy of Joe's post, with all its headers intact.
>
> I am a relatively new notmuch user. Am I correct in thinking that
> notmuch will look inside message/rfc822 containers and find the
> text/plain (and/or text/html) parts within and index them for full
> text search just as it would the message body in a conventional
> message?

Hopefully. I have been meaning to verify that. If not, it's a bug we
should try to fix.

> So really I think the only wrinkle is that I need to remember that
> searching by a person's email address is not going to find all
> messages by them, because ones they send to such a list will not
> actually appear to notmuch to be from them, but instead from the
> lists's address.
>
> That is already an issue with mailing lists that rewrite the From:
> address (again for DKIM/DMARC reasons), but I was wondering if there
> were anything that people do with notmuch to improve matters given
> that all the info does actually exist in these wrapped messages. Is
> it somehow possible, or would be a worthwhile idea to, look for
> From: addresses and other useful headers also inside message/rfc822
> containers?

How would we tell the difference between some list software forwarding a
message/rfc822 part, and a human just forwarding a message? I'd prefer
not to introduce (more) heuristics into the indexing process, but if
there is some written RFC/standard these lists are following then we
could try to follow it too.