From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on dcvr.yhbt.net X-Spam-Level: X-Spam-ASN: X-Spam-Status: No, score=-4.2 required=3.0 tests=ALL_TRUSTED,AWL,BAYES_00, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF shortcircuit=no autolearn=ham autolearn_force=no version=3.4.6 Received: from localhost (dcvr.yhbt.net [127.0.0.1]) by dcvr.yhbt.net (Postfix) with ESMTP id DD9F91F406; Thu, 24 Aug 2023 22:14:14 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=80x24.org; s=selector1; t=1692915254; bh=SeGjiGlgev5OyFKFajtJZllQhf/VeJbjBHZcOnC/5mk=; h=Date:From:To:Subject:References:In-Reply-To:From; b=iWOV11SESHjJGufNkNPUQgPsqg4CClK9lha67c8kR971/nJY7MsnPQXbvnQOmQZ0e IMz8a75Sms7UhndpGGtzdvf8xFU0hjR8jR/T7o5hJoVqDjpkG9mQL5GtmFNLMDBErO 1KNKWnEZPRLH3BAkmdoTC/GjBPQTBmXvQD8MM2xM= Date: Thu, 24 Aug 2023 22:14:14 +0000 From: Eric Wong To: meta@public-inbox.org Subject: [FAIL?] cindex: maximize SMP when dumping inboxes for --associate Message-ID: <20230824221414.M803225@dcvr> References: <20230824220746.13248-1-e@80x24.org> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: <20230824220746.13248-1-e@80x24.org> List-Id: Note: this appears to result in a slight performance regression, I think, but I'm on a busy system... In theory I expect it to work since smaller work units means better work distribution when we're dealing with a few gigantic inboxes (LKML) and many smaller ones; but perhaps the extra IPC overhead from the multitudes of small inboxes on my lore mirror ends up hurting. Instead of dumping all Xapian shards of an inbox with a single xap_helper command, issue one xap_helper command per-shard (as we already do with the cidx shards) to get smaller work units and improve work sharing. --- lib/PublicInbox/CodeSearchIdx.pm | 18 +++++++++++------- 1 file changed, 11 insertions(+), 7 deletions(-) diff --git a/lib/PublicInbox/CodeSearchIdx.pm b/lib/PublicInbox/CodeSearchIdx.pm index a9a7d313..d6348740 100644 --- a/lib/PublicInbox/CodeSearchIdx.pm +++ b/lib/PublicInbox/CodeSearchIdx.pm @@ -551,14 +551,18 @@ sub dump_roots_start { sub dump_ibx { # sends to xap_helper.h my ($self, $ibx_id) = @_; my $ibx = $IBX[$ibx_id] // die "BUG: no IBX[$ibx_id]"; - my @cmd = ('dump_ibx', $ibx->isrch->xh_args, - (map { ('-A', $_) } @ASSOC_PFX), - $ibx_id, $QRY_STR); - pipe(my ($r, $w)) or die "pipe: $!"; - $XHC->mkreq([$DUMP_IBX_WPIPE, $w], @cmd); + my @args = map { ('-A', $_) } @ASSOC_PFX; my $ekey = $ibx->eidx_key; - $self->{PENDING}->{$ekey} = $TODO{associate}; - PublicInbox::CidxXapHelperAux->new($r, $self, $ekey); + my $es = $ibx->isrch->{es}; + push(@args, '-O', $ekey) if defined $es; + push(@args, $ibx_id, $QRY_STR); + for my $d ($es ? $es->shard_dirs : $ibx->isrch->shard_dirs) { + pipe(my ($r, $w)) or die "pipe: $!"; + $XHC->mkreq([$DUMP_IBX_WPIPE, $w], qw(dump_ibx -d), $d, @args); + my $desc = "$ekey -d $d"; + $self->{PENDING}->{$desc} = $TODO{associate}; + PublicInbox::CidxXapHelperAux->new($r, $self, $desc); + } } sub dump_ibx_start {