From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from localhost (localhost [127.0.0.1]) by olra.theworths.org (Postfix) with ESMTP id 5401A431FCF for ; Mon, 2 Jun 2014 06:55:01 -0700 (PDT) X-Virus-Scanned: Debian amavisd-new at olra.theworths.org X-Spam-Flag: NO X-Spam-Score: -2.299 X-Spam-Level: X-Spam-Status: No, score=-2.299 tagged_above=-999 required=5 tests=[RCVD_IN_DNSWL_MED=-2.3, UNPARSEABLE_RELAY=0.001] autolearn=disabled Received: from olra.theworths.org ([127.0.0.1]) by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id R36QTN2K6l8N for ; Mon, 2 Jun 2014 06:54:55 -0700 (PDT) Received: from aserp1040.oracle.com (aserp1040.oracle.com [141.146.126.69]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by olra.theworths.org (Postfix) with ESMTPS id 25BE7431FAF for ; Mon, 2 Jun 2014 06:54:55 -0700 (PDT) Received: from ucsinet22.oracle.com (ucsinet22.oracle.com [156.151.31.94]) by aserp1040.oracle.com (Sentrion-MTA-4.3.2/Sentrion-MTA-4.3.2) with ESMTP id s52Dsr6f006761 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK) for ; Mon, 2 Jun 2014 13:54:54 GMT Received: from aserz7022.oracle.com (aserz7022.oracle.com [141.146.126.231]) by ucsinet22.oracle.com (8.14.5+Sun/8.14.5) with ESMTP id s52DsqfD003482 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Mon, 2 Jun 2014 13:54:53 GMT Received: from abhmp0005.oracle.com (abhmp0005.oracle.com [141.146.116.11]) by aserz7022.oracle.com (8.14.4+Sun/8.14.4) with ESMTP id s52DspMM007132 for ; Mon, 2 Jun 2014 13:54:51 GMT Received: from virt.cz.oracle.com (/10.163.102.127) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Mon, 02 Jun 2014 06:54:51 -0700 Date: Mon, 2 Jun 2014 15:54:47 +0200 From: Vladimir Marek To: David Edmondson Subject: Re: Deduplication ? Message-ID: <20140602135438.GA8223@virt.cz.oracle.com> References: <20140602123212.GA12639@virt.cz.oracle.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.22.1-rc1 (2013-10-16) X-Source-IP: ucsinet22.oracle.com [156.151.31.94] Cc: notmuch@notmuchmail.org X-BeenThere: notmuch@notmuchmail.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: "Use and development of the notmuch mail system." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 Jun 2014 13:55:01 -0000 > > I want to import bigger chunk of archived messages into my notmuch > > database. It's about 100k messages. The problem is, that I most probably > > have quite a lot of those messages in the DB. Basically I would like to > > add only those I don't have already. > > > > There are two possibilities > > > > a) I will add all the 100k messages and then remove the duplicities. > > > > b) I will write a script which will parse the message ID's of the > > to-be-added messages and try to match them to the notmuch DB. Adding > > only files I can't find already. > > > > Ad b) might be better option, but I started to play with the idea of > > deduplication. I'm thinking about listing all the message IDs stored in > > DB, listing all files belonging to the IDs and deleting all but one. > > Also I'm thinking about implementing some simple algorithm telling me > > whether the messages are really very similar. Just to be sure I don't > > delete something I don't want to. > > > > Was anyone playing with the idea? > > notsync[1] used the (lack of) existence of a message id in the store to > decide whether to add something from an IMAP server, but it is old, > crufty, unused and unloved code. I see, that's close to my b) solution, thanks! -- Vlad