From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on dcvr.yhbt.net X-Spam-Level: X-Spam-ASN: X-Spam-Status: No, score=-4.0 required=3.0 tests=ALL_TRUSTED,BAYES_00 shortcircuit=no autolearn=ham autolearn_force=no version=3.4.0 Received: from localhost (dcvr.yhbt.net [127.0.0.1]) by dcvr.yhbt.net (Postfix) with ESMTP id BEE8C1F404; Fri, 9 Feb 2018 20:11:59 +0000 (UTC) Date: Fri, 9 Feb 2018 20:11:59 +0000 From: Eric Wong To: Konstantin Ryabitsev Cc: meta@public-inbox.org Subject: Re: [v2] introduction of content_id Message-ID: <20180209201159.GA24388@dcvr> References: <20180209181718.GA8847@dcvr> <5d025ad2-9824-876d-09d1-3302886579a8@linuxfoundation.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <5d025ad2-9824-876d-09d1-3302886579a8@linuxfoundation.org> List-Id: Konstantin Ryabitsev wrote: > On 02/09/18 13:17, Eric Wong wrote: > > In addition to the git object_id (blob SHA-1) and Message-Id > > header; it seems necessary to introduce an in-between identifier > > for deduplicating which isn't as loose as Message-Id or as > > strict as object_id: content_id > > > > I think a hash of the following raw headers + raw body will > > suffice: > > > > Subject, From, Date, Message-Id, References, To, Cc, > > In-Reply-To, MIME-Version, Content-Type, > > Content-Disposition, Content-Transfer-Encoding > > That's similar to what ARC/DKIM do. E.g. in my mailbox the message has > the following headers sealed: > > h=archived-at:list-post:list-owner:list-subscribe:list-unsubscribe > :list-help:list-archive:precedence:list-id:content-disposition > :mime-version:message-id:subject:to:from:date > :arc-authentication-results; Right, I checked typical DKIM signatures in case I missed some things. > If we trim the arc- and list-specific headers, that's: > > precedence OK, that's another missing header I noticed from the archives I got. > content-disposition > mime-version > message-id > subject > to > from > date > > I'm not sure we should care about content-transfer-encoding, because > that can be mangled by intermediate MTAs (at least that used to happen > all the time in the past -- not sure if it's still the case). MIME-Version and Content-* can be no-ops in common cases; so maybe they're not worth hashing for content_id... > > List-Id, X-Mailing-List should be left out so different > > readers/lists can share spam removals in cross posts. > > Note, that mailing lists that modify the Subject header (e.g. to add > [mailinglist] identifier) will also be impacted similarly. Right; but I don't think Subject mangling is done for vger lists. I know fuse-devel has it, but that's hosted on SF.net The content_id would also get tripped up by the vger unsubscribe signature which AFAIK only existed before 2016. I filtered that sig out with PublicInbox::Filter::Vger for the git@vger import and will likely do that for LKML. There might be some crossposted messages from other lists which have similar signatures which also get into the archives. I know it happened when getting messages from gmane. > > (*) I noticed the first Received: header (last hop) is missing > > from the cregit sources; but the first remaining Received: > > header also includes the identity of the recipient in more > > recent mails... > > I specifically sanitized all Received: headers that didn't say "by > vger.kernel.org" because these are donated by individual users and I > didn't want to expose their potentially private info. Ah, but yes, it's still there in the remaining Received. I only noticed Willy's address and his address/involvement is hardly secret; but there might be others I haven't noticed, yet... I also noticed DKIM-Signature being missing, too. I'm not sure what else was removed (along with Precedence).