[v2] introduction of content_id

[v2] introduction of content_id

[Eric Wong]

In addition to the git object_id (blob SHA-1) and Message-Id
header; it seems necessary to introduce an in-between identifier
for deduplicating which isn't as loose as Message-Id or as
strict as object_id: content_id

I think a hash of the following raw headers + raw body will
suffice:

	Subject, From, Date, Message-Id, References, To, Cc,
	In-Reply-To, MIME-Version, Content-Type,
	Content-Disposition, Content-Transfer-Encoding

Those should be relevant to what MUAs display to users.

Fwiw, ssoma only used Subject + raw body if Message-Id matched;
and public-inbox v1 only uses Message-Id.

List-Id, X-Mailing-List should be left out so different
readers/lists can share spam removals in cross posts.
Received: headers will definitely not be taken into account
as every recipient sees a different chain(*)

The code also won't be tied to any particular hash so it should
be possible to switch to another one as weaknesses are discovered;
and it could be possible to support multiple hashes in an
"inbox"

I also took a look at Razor and DCC anti-spam stuff, but both
seem to take fuzzy matches to avoid hash busting; but IMHO the
bar needs to be higher for handling removals from the archive.


(*) I noticed the first Received: header (last hop) is missing
    from the cregit sources; but the first remaining Received:
    header also includes the identity of the recipient in more
    recent mails...

Re: [v2] introduction of content_id

[Konstantin Ryabitsev]

[-- Attachment #1.1: Type: text/plain, Size: 1966 bytes --]

On 02/09/18 13:17, Eric Wong wrote:
> In addition to the git object_id (blob SHA-1) and Message-Id
> header; it seems necessary to introduce an in-between identifier
> for deduplicating which isn't as loose as Message-Id or as
> strict as object_id: content_id
> 
> I think a hash of the following raw headers + raw body will
> suffice:
> 
> 	Subject, From, Date, Message-Id, References, To, Cc,
> 	In-Reply-To, MIME-Version, Content-Type,
> 	Content-Disposition, Content-Transfer-Encoding

That's similar to what ARC/DKIM do. E.g. in my mailbox the message has
the following headers sealed:

    h=archived-at:list-post:list-owner:list-subscribe:list-unsubscribe
         :list-help:list-archive:precedence:list-id:content-disposition
         :mime-version:message-id:subject:to:from:date
         :arc-authentication-results;

If we trim the arc- and list-specific headers, that's:

precedence
content-disposition
mime-version
message-id
subject
to
from
date

I'm not sure we should care about content-transfer-encoding, because
that can be mangled by intermediate MTAs (at least that used to happen
all the time in the past -- not sure if it's still the case).

> List-Id, X-Mailing-List should be left out so different
> readers/lists can share spam removals in cross posts.

Note, that mailing lists that modify the Subject header (e.g. to add
[mailinglist] identifier) will also be impacted similarly.

> (*) I noticed the first Received: header (last hop) is missing
>     from the cregit sources; but the first remaining Received:
>     header also includes the identity of the recipient in more
>     recent mails...

I specifically sanitized all Received: headers that didn't say "by
vger.kernel.org" because these are donated by individual users and I
didn't want to expose their potentially private info.

Best,
-- 
Konstantin Ryabitsev
Director, IT Infrastructure Security
The Linux Foundation


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

Re: [v2] introduction of content_id

[Eric Wong]

Konstantin Ryabitsev <konstantin@linuxfoundation.org> wrote:
> On 02/09/18 13:17, Eric Wong wrote:
> > In addition to the git object_id (blob SHA-1) and Message-Id
> > header; it seems necessary to introduce an in-between identifier
> > for deduplicating which isn't as loose as Message-Id or as
> > strict as object_id: content_id
> > 
> > I think a hash of the following raw headers + raw body will
> > suffice:
> > 
> > 	Subject, From, Date, Message-Id, References, To, Cc,
> > 	In-Reply-To, MIME-Version, Content-Type,
> > 	Content-Disposition, Content-Transfer-Encoding
> 
> That's similar to what ARC/DKIM do. E.g. in my mailbox the message has
> the following headers sealed:
> 
>     h=archived-at:list-post:list-owner:list-subscribe:list-unsubscribe
>          :list-help:list-archive:precedence:list-id:content-disposition
>          :mime-version:message-id:subject:to:from:date
>          :arc-authentication-results;

Right, I checked typical DKIM signatures in case I missed some
things.

> If we trim the arc- and list-specific headers, that's:
> 
> precedence

OK, that's another missing header I noticed from the archives I got.

> content-disposition
> mime-version
> message-id
> subject
> to
> from
> date
> 
> I'm not sure we should care about content-transfer-encoding, because
> that can be mangled by intermediate MTAs (at least that used to happen
> all the time in the past -- not sure if it's still the case).

MIME-Version and Content-* can be no-ops in common cases;
so maybe they're not worth hashing for content_id...

> > List-Id, X-Mailing-List should be left out so different
> > readers/lists can share spam removals in cross posts.
> 
> Note, that mailing lists that modify the Subject header (e.g. to add
> [mailinglist] identifier) will also be impacted similarly.

Right; but I don't think Subject mangling is done for vger
lists.  I know fuse-devel has it, but that's hosted on SF.net

The content_id would also get tripped up by the vger unsubscribe
signature which AFAIK only existed before 2016.  I filtered that
sig out with PublicInbox::Filter::Vger for the git@vger import
and will likely do that for LKML.

There might be some crossposted messages from other lists which
have similar signatures which also get into the archives.  I
know it happened when getting messages from gmane.

> > (*) I noticed the first Received: header (last hop) is missing
> >     from the cregit sources; but the first remaining Received:
> >     header also includes the identity of the recipient in more
> >     recent mails...
> 
> I specifically sanitized all Received: headers that didn't say "by
> vger.kernel.org" because these are donated by individual users and I
> didn't want to expose their potentially private info.

Ah, but yes, it's still there in the remaining Received.  I only
noticed Willy's address and his address/involvement is hardly
secret; but there might be others I haven't noticed, yet...

I also noticed DKIM-Signature being missing, too.  I'm not sure
what else was removed (along with Precedence).

Re: [v2] introduction of content_id

[Eric Wong]

Eric Wong <e@80x24.org> wrote:
> In addition to the git object_id (blob SHA-1) and Message-Id
> header; it seems necessary to introduce an in-between identifier
> for deduplicating which isn't as loose as Message-Id or as
> strict as object_id: content_id

I think this will only be calculated-on-the-fly in cases the
Message-ID matches.  No need to cement it into the Xapian DB,
meaning we can tweak which headers we care about more freely.