From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on dcvr.yhbt.net X-Spam-Level: X-Spam-ASN: X-Spam-Status: No, score=-4.0 required=3.0 tests=ALL_TRUSTED,BAYES_00 shortcircuit=no autolearn=ham autolearn_force=no version=3.4.2 Received: from localhost (dcvr.yhbt.net [127.0.0.1]) by dcvr.yhbt.net (Postfix) with ESMTP id AD42C1F9FC for ; Sat, 30 Oct 2021 08:11:44 +0000 (UTC) From: Eric Wong To: meta@public-inbox.org Subject: [PATCH 2/5] lei_to_mail: limit workers for text, reply and v2 outputs Date: Sat, 30 Oct 2021 08:11:41 +0000 Message-Id: <20211030081144.1080-3-e@80x24.org> In-Reply-To: <20211030081144.1080-1-e@80x24.org> References: <20211030081144.1080-1-e@80x24.org> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit List-Id: "text" and "reply" outputs are intended for the pager, so parallelizing them is a waste of resources. v2 has shards, of course, so parallelizing writes to it is also a waste since the deduplication work is a bit more complex. --- lib/PublicInbox/LeiQuery.pm | 2 +- lib/PublicInbox/LeiToMail.pm | 2 ++ 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/lib/PublicInbox/LeiQuery.pm b/lib/PublicInbox/LeiQuery.pm index 3d0e5b144d3a..352ee60131aa 100644 --- a/lib/PublicInbox/LeiQuery.pm +++ b/lib/PublicInbox/LeiQuery.pm @@ -37,7 +37,7 @@ sub _start_query { # used by "lei q" and "lei up" $lms->lms_write_prepare->lms_pause; # just create } } - $l2m and $l2m->{-wq_nr_workers} = $mj // + $l2m and $l2m->{-wq_nr_workers} //= $mj // int($nproc * 0.75 + 0.5); # keep some CPU for git # descending docid order is cheapest, MUA controls sorting order diff --git a/lib/PublicInbox/LeiToMail.pm b/lib/PublicInbox/LeiToMail.pm index 83f58a29405b..90db30e53089 100644 --- a/lib/PublicInbox/LeiToMail.pm +++ b/lib/PublicInbox/LeiToMail.pm @@ -415,11 +415,13 @@ sub new { require PublicInbox::LeiViewText; $lei->{lvt} = PublicInbox::LeiViewText->new($lei, $fmt); $self->{base_type} = 'text'; + $self->{-wq_nr_workers} = 1; # for pager @conflict = qw(mua save); } elsif ($fmt eq 'v2') { die "--dedupe=oid and v2 are incompatible\n" if ($lei->{opt}->{dedupe}//'') eq 'oid'; $self->{base_type} = 'v2'; + $self->{-wq_nr_workers} = 1; # v2 has shards $lei->{opt}->{save} = \1; $dst = $lei->{ovv}->{dst} = $lei->abs_path($dst); @conflict = qw(mua sort);