From: Eric Wong <e@80x24.org>
To: meta@public-inbox.org
Subject: [PATCH v2] codesearch: use retry_reopen for WWW
Date: Thu, 30 Nov 2023 21:40:47 +0000 [thread overview]
Message-ID: <20231130214048.M46988@dcvr> (raw)
In-Reply-To: <20231130114109.2577708-16-e@80x24.org>
As with mail search, a cindex may be updated while WWW is
serving requests. Thus we must reopen the Xapian DB when
the revision we're using becomes stale.
---
v2: avoid reintroducing load_ct as noted in
https://public-inbox.org/meta/20231130213641.M35664@dcvr/
lib/PublicInbox/CodeSearch.pm | 16 ++++++----------
1 file changed, 6 insertions(+), 10 deletions(-)
diff --git a/lib/PublicInbox/CodeSearch.pm b/lib/PublicInbox/CodeSearch.pm
index f4694686..3092718d 100644
--- a/lib/PublicInbox/CodeSearch.pm
+++ b/lib/PublicInbox/CodeSearch.pm
@@ -258,15 +258,11 @@ sub load_ct { # retry_reopen cb
sub load_commit_times { # each_cindex callback
my ($self, $todo) = @_; # todo = [ [ time, git ], [ time, git ] ...]
- my (@pending, $rec, $dir, @ids, $doc);
+ my (@pending, $rec, $ct);
while ($rec = shift @$todo) {
- @ids = docids_of_git_dir $self, $rec->[1]->{git_dir};
- if (@ids) {
- for (@ids) {
- $doc = $self->get_doc($_) // next;
- $rec->[0] = int_val($doc, CT);
- last;
- }
+ $ct = $self->retry_reopen(\&load_ct, $rec->[1]->{git_dir});
+ if (defined $ct) {
+ $rec->[0] = $ct;
} else { # may be in another cindex:
push @pending, $rec;
}
@@ -295,7 +291,7 @@ EOM
$git;
};
}
- my $jd = join_data($self) or return warn <<EOM;
+ my $jd = $self->retry_reopen(\&join_data, $self) or return warn <<EOM;
W: cindex.$name.topdir=$self->{topdir} has no usable join data for $cfg_f
EOM
my ($ekeys, $roots, $ibx2root) = @$jd{qw(ekeys roots ibx2root)};
@@ -366,7 +362,7 @@ sub repos_sorted {
my @recs = map { [ 0, $_ ] } @_; # PublicInbox::Git objects
my @todo = @recs;
$pi_cfg->each_cindex(\&load_commit_times, \@todo);
- @recs = sort { $b->[0] <=> $a->[0] } @recs;
+ @recs = sort { $b->[0] <=> $a->[0] } @recs; # sort by commit time
}
1;
prev parent reply other threads:[~2023-11-30 21:43 UTC|newest]
Thread overview: 20+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-11-30 11:40 [PATCH 00/15] various cindex fixes + speedups Eric Wong
2023-11-30 11:40 ` [PATCH 01/15] cindex: fix store_repo+repo_stored on no-op Eric Wong
2023-11-30 11:40 ` [PATCH 02/15] codesearch: allow inbox count to exceed matches Eric Wong
2023-11-30 11:40 ` [PATCH 03/15] config: reject newlines consistently in dir names Eric Wong
2023-11-30 11:40 ` [PATCH 04/15] cindex: only create {-cidx_err} field on failures Eric Wong
2023-11-30 11:40 ` [PATCH 05/15] cindex: keep batch pipe for pruning SHA-256 repos Eric Wong
2023-11-30 11:40 ` [PATCH 06/15] cindex: store extensions.objectFormat with repo data Eric Wong
2023-11-30 21:36 ` Eric Wong
2023-11-30 11:41 ` [PATCH 07/15] git: share unlinked pack checking code with gcf2 Eric Wong
2023-11-30 11:41 ` [PATCH 08/15] cindex: skip getpid guard for most OnDestroy use Eric Wong
2023-11-30 11:41 ` [PATCH 09/15] spawn: drop IO layer support from redirects Eric Wong
2023-11-30 11:41 ` [PATCH 10/15] cindex: speed up initial scan setup phase Eric Wong
2023-11-30 11:41 ` [PATCH 11/15] inbox: expire resources more aggressively Eric Wong
2023-11-30 11:41 ` [PATCH 12/15] git_async_cat: use git from "all" extindex if possible Eric Wong
2023-11-30 11:41 ` [PATCH 13/15] www_listing: support publicInbox.nameIsUrl Eric Wong
2023-12-01 1:29 ` Kyle Meyer
2023-12-01 2:01 ` [PATCH] doc: config: fix grammar for nameIsUrl Eric Wong
2023-11-30 11:41 ` [PATCH 14/15] inbox: shrink data structures for publicinbox.*.hide Eric Wong
2023-11-30 11:41 ` [PATCH 15/15] codesearch: use retry_reopen for WWW Eric Wong
2023-11-30 21:40 ` Eric Wong [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: https://public-inbox.org/README
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20231130214048.M46988@dcvr \
--to=e@80x24.org \
--cc=meta@public-inbox.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).