From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on dcvr.yhbt.net X-Spam-Level: X-Spam-ASN: X-Spam-Status: No, score=-4.2 required=3.0 tests=ALL_TRUSTED,AWL,BAYES_00, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF, T_SCC_BODY_TEXT_LINE shortcircuit=no autolearn=ham autolearn_force=no version=3.4.6 Received: from localhost (dcvr.yhbt.net [127.0.0.1]) by dcvr.yhbt.net (Postfix) with ESMTP id 3CDE41F406 for ; Mon, 20 Nov 2023 19:22:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=80x24.org; s=selector1; t=1700508153; bh=T5RtbpH8YE/Z6A+i/EKQ9DDeaBxpciJO1QfDlB+5RJ8=; h=From:To:Subject:Date:From; b=KXP6rNFeuLSEfQjIesmaPd6DPq4xT5sZGx0/4b1MjAmVRx1U8FDK+nPevh2cnmAzN uXR+E+j0e3YB041HKOURMu7moKmA4WBqojqizS7rmcmfSkOa35qgxY3GbAWbLrTFnL ToyShx6HFeuv7hVv0ya0oLuAHys2Gg4Ew0TvM3c8= From: Eric Wong To: meta@public-inbox.org Subject: [PATCH] searchidx: run `git patch-id' in parallel Date: Mon, 20 Nov 2023 19:22:32 +0000 Message-Id: <20231120192232.1701486-1-e@80x24.org> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit List-Id: Informal benchmarks show a rough 5% indexing improvement on an SMP system when there are idle cores due to Xapian shards being I/O bound (since `git patch-id' is mainly CPU bound). This is only parallelized on a per-patch basis. Further increasing parallelism would increase complexity and probably not be worth it since `git patch-id' is reasonably fast while our text indexing tends to be slow. --- lib/PublicInbox/CodeSearchIdx.pm | 2 +- lib/PublicInbox/SearchIdx.pm | 24 +++++++++++------------- 2 files changed, 12 insertions(+), 14 deletions(-) diff --git a/lib/PublicInbox/CodeSearchIdx.pm b/lib/PublicInbox/CodeSearchIdx.pm index 3a30b5e5..54a2ba22 100644 --- a/lib/PublicInbox/CodeSearchIdx.pm +++ b/lib/PublicInbox/CodeSearchIdx.pm @@ -675,7 +675,7 @@ sub get_roots ($$) { { 0 => $refs, 1 => $roots_fh }, \&index_repo, $self, $git) } -# for PublicInbox::SearchIdx::patch_id and with_umask +# for PublicInbox::SearchIdx `git patch-id' call and with_umask sub git { $_[0]->{git} } sub load_existing ($) { # for -u/--update diff --git a/lib/PublicInbox/SearchIdx.pm b/lib/PublicInbox/SearchIdx.pm index 9566b14d..32598b7c 100644 --- a/lib/PublicInbox/SearchIdx.pm +++ b/lib/PublicInbox/SearchIdx.pm @@ -22,7 +22,7 @@ use POSIX qw(strftime); use Fcntl qw(SEEK_SET); use Time::Local qw(timegm); use PublicInbox::OverIdx; -use PublicInbox::Spawn qw(run_wait run_qx); +use PublicInbox::Spawn qw(run_wait popen_rd); use PublicInbox::Git qw(git_unquote); use PublicInbox::MsgTime qw(msg_timestamp msg_datestamp); use PublicInbox::Address; @@ -350,20 +350,13 @@ sub index_diff ($$$) { index_text($self, join("\n", @$xnq), 1, 'XNQ'); } -sub patch_id { - my ($self, $sref) = @_; - my $git = ($self->{ibx} // $self->{eidx} // $self)->git; - my $opt = { 0 => [ ':utf8', $sref ], 2 => \(my $err) }; - my $id = run_qx($git->cmd(qw(patch-id --stable)), undef, $opt); - warn $err if $err; - $id =~ /\A([a-f0-9]{40,})/ ? $1 : undef; -} - sub index_body_text { my ($self, $doc, $sref) = @_; - if ($$sref =~ /^(?:diff|---|\+\+\+) /ms) { - my $id = patch_id($self, $sref); - $doc->add_term('XDFID'.$id) if defined($id); + my $rd; + if ($$sref =~ /^(?:diff|---|\+\+\+) /ms) { # start patch-id in parallel + my $git = ($self->{ibx} // $self->{eidx} // $self)->git; + $rd = popen_rd($git->cmd(qw(patch-id --stable)), undef, + { 0 => [ ':utf8', $sref ] }); } # split off quoted and unquoted blocks: @@ -387,6 +380,11 @@ sub index_body_text { } undef $txt; # free memory } + if (defined $rd) { # reap `git patch-id' + (readline($rd) // '') =~ /\A([a-f0-9]{40,})/ and + $doc->add_term('XDFID'.$1); + $rd->close or warn "W: git patch-id failed: \$?=$? (non-fatal)" + } } sub index_xapian { # msg_iter callback