Re: [v2] introduction of content_id

[Eric Wong <e@80x24.org>]

Konstantin Ryabitsev <konstantin@linuxfoundation.org> wrote:
> On 02/09/18 13:17, Eric Wong wrote:
> > In addition to the git object_id (blob SHA-1) and Message-Id
> > header; it seems necessary to introduce an in-between identifier
> > for deduplicating which isn't as loose as Message-Id or as
> > strict as object_id: content_id
> > 
> > I think a hash of the following raw headers + raw body will
> > suffice:
> > 
> > 	Subject, From, Date, Message-Id, References, To, Cc,
> > 	In-Reply-To, MIME-Version, Content-Type,
> > 	Content-Disposition, Content-Transfer-Encoding
> 
> That's similar to what ARC/DKIM do. E.g. in my mailbox the message has
> the following headers sealed:
> 
>     h=archived-at:list-post:list-owner:list-subscribe:list-unsubscribe
>          :list-help:list-archive:precedence:list-id:content-disposition
>          :mime-version:message-id:subject:to:from:date
>          :arc-authentication-results;

Right, I checked typical DKIM signatures in case I missed some
things.

> If we trim the arc- and list-specific headers, that's:
> 
> precedence

OK, that's another missing header I noticed from the archives I got.

> content-disposition
> mime-version
> message-id
> subject
> to
> from
> date
> 
> I'm not sure we should care about content-transfer-encoding, because
> that can be mangled by intermediate MTAs (at least that used to happen
> all the time in the past -- not sure if it's still the case).

MIME-Version and Content-* can be no-ops in common cases;
so maybe they're not worth hashing for content_id...

> > List-Id, X-Mailing-List should be left out so different
> > readers/lists can share spam removals in cross posts.
> 
> Note, that mailing lists that modify the Subject header (e.g. to add
> [mailinglist] identifier) will also be impacted similarly.

Right; but I don't think Subject mangling is done for vger
lists.  I know fuse-devel has it, but that's hosted on SF.net

The content_id would also get tripped up by the vger unsubscribe
signature which AFAIK only existed before 2016.  I filtered that
sig out with PublicInbox::Filter::Vger for the git@vger import
and will likely do that for LKML.

There might be some crossposted messages from other lists which
have similar signatures which also get into the archives.  I
know it happened when getting messages from gmane.

> > (*) I noticed the first Received: header (last hop) is missing
> >     from the cregit sources; but the first remaining Received:
> >     header also includes the identity of the recipient in more
> >     recent mails...
> 
> I specifically sanitized all Received: headers that didn't say "by
> vger.kernel.org" because these are donated by individual users and I
> didn't want to expose their potentially private info.

Ah, but yes, it's still there in the remaining Received.  I only
noticed Willy's address and his address/involvement is hardly
secret; but there might be others I haven't noticed, yet...

I also noticed DKIM-Signature being missing, too.  I'm not sure
what else was removed (along with Precedence).