From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on dcvr.yhbt.net X-Spam-Level: X-Spam-ASN: X-Spam-Status: No, score=-4.0 required=3.0 tests=ALL_TRUSTED,BAYES_00 shortcircuit=no autolearn=ham autolearn_force=no version=3.4.2 Received: from localhost (dcvr.yhbt.net [127.0.0.1]) by dcvr.yhbt.net (Postfix) with ESMTP id 8CA6F1F5AE for ; Thu, 22 Jul 2021 08:50:34 +0000 (UTC) From: Eric Wong To: meta@public-inbox.org Subject: [WIP] extindex: support --dedupe[=MSGID] Date: Thu, 22 Jul 2021 08:50:34 +0000 Message-Id: <20210722085034.31363-1-e@80x24.org> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit List-Id: This patch appears to do what its supposed to. But it also appears to be finding duplicates that were previously missed. That's a good thing, but I wish I understood what I seem to have fixed :x I'm not sure why the previous ExtSearchIdx.pm (blob 357312b8) was causing messages to be missed, even, and why this patch seems to fix it... And it's not infinite looping, either. Anyways, before this patch, "-extindex --dedupe" was taking ~5 min to no-op every message (after the initial full --dedupe run which took over a day to run). Now that it appears to be doing many more dedupes, so it could take 40-ish hours to finish the dedupe run on the machine running . Hopefully this will result in "BUG?" messages from View.pm going away after it finishes. -------8<------ Subject: [PATCH] extindex: support --dedupe[=MSGID] Sometimes I just want to dedupe a single Message-ID to test something, and this lets me do it. --- lib/PublicInbox/ExtSearchIdx.pm | 24 +++++++++++++++++------- script/public-inbox-extindex | 4 ++-- 2 files changed, 19 insertions(+), 9 deletions(-) diff --git a/lib/PublicInbox/ExtSearchIdx.pm b/lib/PublicInbox/ExtSearchIdx.pm index 357312b8..442ded46 100644 --- a/lib/PublicInbox/ExtSearchIdx.pm +++ b/lib/PublicInbox/ExtSearchIdx.pm @@ -887,23 +887,31 @@ sub dd_smsg { # git->cat_async callback } } -sub eidx_dedupe ($$) { - my ($self, $sync) = @_; +sub eidx_dedupe ($$$) { + my ($self, $sync, $msgids) = @_; $sync->{dedupe_cull} = 0; my $candidates = 0; my $nr_mid = 0; return unless eidxq_lock_acquire($self); - my $iter; + my ($iter, $cur_mid); my $min_id = 0; local $sync->{-regen_fmt} = "dedupe %u/".$self->{oidx}->max."\n"; # note: we could write this query more intelligently, # but that causes lock contention with read-only processes dedupe_restart: - $iter = $self->{oidx}->dbh->prepare(<{oidx}->dbh->prepare(< ? ORDER BY id ASC EOS - $iter->execute($min_id); + $iter->execute($min_id); + } else { + $iter = $self->{oidx}->dbh->prepare(< ? ORDER BY id ASC +EOS + $iter->execute($cur_mid, $min_id); + } while (my ($mid, $id) = $iter->fetchrow_array) { last if $sync->{quit}; $self->{current_info} = "dedupe $mid"; @@ -937,6 +945,8 @@ EOS goto dedupe_restart; } } + goto dedupe_restart if scalar(@$msgids); + my $n = delete $sync->{dedupe_cull}; if (my $pr = $sync->{-opt}->{-progress}) { $pr->("culled $n/$candidates candidates ($nr_mid msgids)\n"); @@ -974,9 +984,9 @@ sub eidx_sync { # main entry point for my $ibx (@{ibx_sorted($self)}) { $ibx->{-ibx_id} //= $self->{oidx}->ibx_id($ibx->eidx_key); } - if (delete($opt->{dedupe})) { + if (my $msgids = delete($opt->{dedupe})) { local $sync->{checkpoint_unlocks} = 1; - eidx_dedupe($self, $sync); + eidx_dedupe($self, $sync, $msgids); } if (delete($opt->{reindex})) { local $sync->{checkpoint_unlocks} = 1; diff --git a/script/public-inbox-extindex b/script/public-inbox-extindex index dcb12e5a..addd5ac6 100755 --- a/script/public-inbox-extindex +++ b/script/public-inbox-extindex @@ -17,7 +17,7 @@ usage: public-inbox-extindex [options] [EXTINDEX_DIR] [INBOX_DIR...] --batch-size=BYTES flush changes to OS after a given number of bytes --max-size=BYTES do not index messages larger than the given size --gc perform garbage collection instead of indexing - --dedupe fix prior deduplication errors + --dedupe[=MSGID] fix prior deduplication errors (may be repeated) --verbose | -v increase verbosity (may be repeated) --dry-run | -n dry-run on --dedupe @@ -29,7 +29,7 @@ GetOptions($opt, qw(verbose|v+ reindex rethread compact|c+ jobs|j=i fsync|sync! indexlevel|index-level|L=s max_size|max-size=s batch_size|batch-size=s - dedupe gc commit-interval=i watch scan! dry-run|n + dedupe:s@ gc commit-interval=i watch scan! dry-run|n all help|h)) or die $help; if ($opt->{help}) { print $help; exit 0 };