Re: [PATCH 08/13] v2writable: support reindexing Xapian

[Eric Wong <e@80x24.org>]

<e@80x24.org> wrote:
> --- a/lib/PublicInbox/SearchIdx.pm
> +++ b/lib/PublicInbox/SearchIdx.pm

> @@ -369,10 +369,12 @@ sub add_message {
>  			}
>  		}
>  
> +		$self->delete_article($num) if defined $num; # for reindexing
>  		if ($skel) {
>  			push @values, $mids, $xpath, $data;
>  			$skel->index_skeleton(\@values);
>  			$doc->add_boolean_term('Q' . $_) foreach @$mids;
> +			$doc->add_boolean_term('XNUM' . $num) if defined $num;
>  			$doc_id = $self->{xdb}->add_document($doc);
>  		} else {
>  			$doc_id = link_and_save($self, $doc, $mids, $refs,
> @@ -421,6 +423,16 @@ sub remove_message {
>  	}
>  }
>  
> +sub delete_article {
> +	my ($self, $num) = @_;
> +	my $ndel = 0;
> +	batch_do($self, 'XNUM' . $num, sub {
> +		my ($ids) = @_;
> +		$ndel += scalar @$ids;
> +		$self->{xdb}->delete_document($_) for @$ids;
> +	});
> +}

I will need to do some further investigation, but I must be
missing something and there's increases in Xapian DB size
which doesn't seem to get recovered on xapian-compact...

> diff --git a/lib/PublicInbox/SearchIdxSkeleton.pm b/lib/PublicInbox/SearchIdxSkeleton.pm
> index 78a1730..4f15816 100644
> --- a/lib/PublicInbox/SearchIdxSkeleton.pm
> +++ b/lib/PublicInbox/SearchIdxSkeleton.pm
> @@ -134,6 +134,7 @@ sub index_skeleton_real ($$) {
>  	$smsg->load_from_data($doc_data);
>  	my $num = $values->[PublicInbox::Search::NUM];
>  	my @refs = ($smsg->references =~ /<([^>]+)>/g);
> +	$self->delete_article($num) if defined $num; # for reindexing
>  	$self->link_and_save($doc, $mids, \@refs, $num, $xpath);
>  }