From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on dcvr.yhbt.net X-Spam-Level: X-Spam-ASN: X-Spam-Status: No, score=-4.0 required=3.0 tests=ALL_TRUSTED,BAYES_00 shortcircuit=no autolearn=ham autolearn_force=no version=3.4.2 Received: from localhost (dcvr.yhbt.net [127.0.0.1]) by dcvr.yhbt.net (Postfix) with ESMTP id EE8481F670 for ; Sat, 16 Oct 2021 01:48:36 +0000 (UTC) Date: Sat, 16 Oct 2021 01:48:36 +0000 From: Eric Wong To: meta@public-inbox.org Subject: [PATCH] extindex: avoid triggering a buggy unref Message-ID: <20211016014836.M253718@dcvr> References: <20211014054933.5104-1-e@80x24.org> <20211014060629.GA14615@dcvr> <20211014210953.M934295@dcvr> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: <20211014210953.M934295@dcvr> List-Id: Eric Wong wrote: > Eric Wong wrote: > > I noticed some unref messages which shouldn't have been > > happening, but they were. Which is troubling. So add > > a guard around an unref path until we can get to the bottom > > of this. > > And now I'm not seeing it... Heisenbugs :< OK, it worked as intended and help me find a bug \o/ -----------8<---------- Subject: [PATCH] extindex: avoid triggering a buggy unref We can't attempt to unref messages beyond the highwater mark of an inbox. This bugfix was found by commit c485036d0b1ce7ed (extindex: guard against buggy unrefs, 2021-10-14), which actually did its intended job and guarded against a buggy unref. The window for the bug is when one process is doing -index against an inbox, -extdindex --reindex is working on the same inbox. --- lib/PublicInbox/ExtSearchIdx.pm | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/lib/PublicInbox/ExtSearchIdx.pm b/lib/PublicInbox/ExtSearchIdx.pm index a08a9451..69d048fb 100644 --- a/lib/PublicInbox/ExtSearchIdx.pm +++ b/lib/PublicInbox/ExtSearchIdx.pm @@ -839,7 +839,7 @@ sub _unref_stale_range ($$$) { $r = $sync->{self}->{oidx}->dbh->selectall_arrayref( <{-ibx_id}); SELECT docid,xnum,oidbin FROM xref3 -WHERE ibx_id = ? AND xnum $lt_or_gt LIMIT $lim +WHERE ibx_id = ? AND $lt_or_gt LIMIT $lim EOS return if $sync->{quit}; for (@$r) { # hopefully rare, not worth optimizing: @@ -859,7 +859,7 @@ sub _reindex_check_ibx ($$$) { my $opt = { limit => $slice }; my ($beg, $end) = (1, $slice); my $err = sync_inbox($self, $sync, $ibx) and return; - my $max = $ibx->over->max; + my $max = $ibx->mm->num_highwater; $end = $max if $end > $max; # first, check if we missed any messages in target $ibx @@ -869,7 +869,7 @@ sub _reindex_check_ibx ($$$) { local $sync->{-regen_fmt} = "$ekey checking %u/$max\n"; ${$sync->{nr}} = 0; my $fast = $sync->{-opt}->{fast}; - my $dsu; # _unref_stale_range (< $lo) called + my $usr; # _unref_stale_range (< $lo) called my ($lo, $hi); while (scalar(@{$msgs = $ibx->over->query_xover($beg, $end, $opt)})) { ${$sync->{nr}} = $beg; @@ -880,7 +880,7 @@ sub _reindex_check_ibx ($$$) { reindex_checkpoint($self, $sync); # release lock } ($lo, $hi) = ($msgs->[0]->{num}, $msgs->[-1]->{num}); - $dsu //= _unref_stale_range($sync, $ibx, "< $lo"); + $usr //= _unref_stale_range($sync, $ibx, "xnum < $lo"); my $x3a = $self->{oidx}->dbh->selectall_arrayref( <<"", undef, $ibx_id, $lo, $hi); SELECT xnum,oidbin,docid FROM xref3 WHERE @@ -921,7 +921,8 @@ ibx_id = ? AND xnum >= ? AND xnum <= ? } } } - _unref_stale_range($sync, $ibx, "> $hi") if defined($hi); + defined($hi) and ($hi < $max) and + _unref_stale_range($sync, $ibx, "xnum > $hi AND xnum <= $max"); } sub _reindex_inbox ($$$) {