From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on dcvr.yhbt.net X-Spam-Level: X-Spam-ASN: AS6315 166.70.0.0/16 X-Spam-Status: No, score=-3.7 required=3.0 tests=AWL,BAYES_00, RCVD_IN_DNSWL_LOW,RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_PASS shortcircuit=no autolearn=ham autolearn_force=no version=3.4.1 Received: from out01.mta.xmission.com (out01.mta.xmission.com [166.70.13.231]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by dcvr.yhbt.net (Postfix) with ESMTPS id 231F31F597; Wed, 1 Aug 2018 16:42:08 +0000 (UTC) Received: from in01.mta.xmission.com ([166.70.13.51]) by out01.mta.xmission.com with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.87) (envelope-from ) id 1fkuCM-0001fI-WC; Wed, 01 Aug 2018 10:42:07 -0600 Received: from [97.119.167.31] (helo=x220.xmission.com) by in01.mta.xmission.com with esmtpsa (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.87) (envelope-from ) id 1fkuC7-0001KS-Gn; Wed, 01 Aug 2018 10:42:06 -0600 From: ebiederm@xmission.com (Eric W. Biederman) To: Eric Wong Cc: Date: Wed, 01 Aug 2018 11:41:47 -0500 Message-ID: <878t5qkpis.fsf@xmission.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/25.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-XM-SPF: eid=1fkuC7-0001KS-Gn;;;mid=<878t5qkpis.fsf@xmission.com>;;;hst=in01.mta.xmission.com;;;ip=97.119.167.31;;;frm=ebiederm@xmission.com;;;spf=neutral X-XM-AID: U2FsdGVkX185d0mYTLu4sG6FKdg2wEZT3qQzY7ysdog= X-SA-Exim-Connect-IP: 97.119.167.31 X-SA-Exim-Mail-From: ebiederm@xmission.com Subject: [PATCH 00/13]: Incremental index fixes X-SA-Exim-Version: 4.2.1 (built Thu, 05 May 2016 13:38:54 -0600) X-SA-Exim-Scanned: Yes (on in01.mta.xmission.com) List-Id: Eric, Recently I noticed some weird indexing on one of my mailboxes. So I dug in and made the index_sync tests more robust and found several corner cases where the current code can be problematic. What follows are the test cases and fixes for those cases. The big one is when a message with the highest message number is deleted. This can cause message numbers to be reassigned so I explicitly track the highest message number assigned. Very confusing when reading mail there is a case in SearchIdx that has been assigning message numbers in the wrong order. Depending on your time range that can be very confusing as old messages show up before newer ones. Finally in v2 deleted messages have not been being deleted from the msgmap. Which while great for keeping message numbers from going backwards it means things still show up that shouldn't. Eric W. Biederman (13): Import.pm: Don't assume {in} and {out} always exist t/v1reindex.t: Isolate the test cases t/v2reindex.t: Isolate the test cases more t/v[12]reindex.t: Place expected second in Xapian tests t/v[12]reindex.t: Test that the resulting msgmap is as expected t/v[12]reindex.t: Test incremental indexing works SearchIdx.pm: Always assign numbers backwards during incremental indexing Msgmap.pm: Track the largest value of num ever assigned t/v[12]reindex.t Verify num_highwater t/v[12]reindex.t: Verify the num highwater is as expected SearchIdx,V2Writeable: Update num_highwater on optimized deletes V2Writeable.pm: Ensure that a found message number is in the msgmap V2Writeable.pm: In unindex_oid delete the message from msgmap lib/PublicInbox/Import.pm | 1 + lib/PublicInbox/Msgmap.pm | 23 ++- lib/PublicInbox/SearchIdx.pm | 30 ++-- lib/PublicInbox/V2Writable.pm | 20 ++- t/v1reindex.t | 359 ++++++++++++++++++++++++++++++++++++------ t/v2reindex.t | 335 +++++++++++++++++++++++++++++++++++---- 6 files changed, 666 insertions(+), 102 deletions(-) Eric