From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on dcvr.yhbt.net X-Spam-Level: X-Spam-ASN: X-Spam-Status: No, score=-4.0 required=3.0 tests=ALL_TRUSTED,BAYES_00 shortcircuit=no autolearn=ham autolearn_force=no version=3.4.1 Received: from localhost (dcvr.yhbt.net [127.0.0.1]) by dcvr.yhbt.net (Postfix) with ESMTP id B75A01F597; Thu, 2 Aug 2018 03:44:04 +0000 (UTC) Date: Thu, 2 Aug 2018 03:44:04 +0000 From: Eric Wong To: "Eric W. Biederman" Cc: meta@public-inbox.org Subject: [WIP] searchidx: support incremental indexing on indexlevel=basic Message-ID: <20180802034404.cnvfqlgvynamnc6n@whir> References: <878t5qkpis.fsf@xmission.com> <20180801164344.7911-8-ebiederm@xmission.com> <20180802030022.ly2tvbobs7tkfmn3@whir> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: <20180802030022.ly2tvbobs7tkfmn3@whir> List-Id: I wrote: > While testing this, it looks like I introduced a bug to > indexlevel=basic which broke incremental indexing when I made it > possible to upgrade to (medium|full). Patch coming for that in > a bit... Eep, I think there's deeper problems with indexlevel=basic and incremental updates, since it's still doing lookups against Xapian for deletes... This is my work-in-progress to stop full-reindexing, at least. ----8<----- Subject: [PATCH] searchidx: support incremental indexing on indexlevel=basic --- lib/PublicInbox/SearchIdx.pm | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/lib/PublicInbox/SearchIdx.pm b/lib/PublicInbox/SearchIdx.pm index 54f82aa..5cac08f 100644 --- a/lib/PublicInbox/SearchIdx.pm +++ b/lib/PublicInbox/SearchIdx.pm @@ -668,6 +668,15 @@ sub need_update ($$$) { ($n eq '' || $n > 0); } +sub _last_x_commit { + my ($self) = @_; + if ($self->{indexlevel} =~ $xapianlevels) { + $self->{xdb}->get_metadata('last_commit'); + } else { + $self->{mm}->last_commit || ''; + } +} + # indexes all unindexed messages (v1 only) sub _index_sync { my ($self, $opts) = @_; @@ -682,7 +691,7 @@ sub _index_sync { do { $xlog = undef; $mkey = 'last_commit'; - $last_commit = $xdb->get_metadata('last_commit'); + $last_commit = $self->_last_x_commit; $lx = $last_commit; if ($reindex) { $lx = ''; @@ -707,7 +716,7 @@ sub _index_sync { $xlog = _git_log($self, $range); $xdb = $self->begin_txn_lazy; - } while ($xdb->get_metadata('last_commit') ne $last_commit); + } while ($self->_last_x_commit ne $last_commit); my $dbh = $mm->{dbh} if $mm; my $cb = sub { -- EW