From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on dcvr.yhbt.net X-Spam-Level: X-Spam-ASN: X-Spam-Status: No, score=-4.0 required=3.0 tests=ALL_TRUSTED,BAYES_00 shortcircuit=no autolearn=ham autolearn_force=no version=3.4.2 Received: from localhost (dcvr.yhbt.net [127.0.0.1]) by dcvr.yhbt.net (Postfix) with ESMTP id 37E3A205CE for ; Thu, 23 May 2019 09:37:07 +0000 (UTC) From: Eric Wong To: meta@public-inbox.org Subject: [PATCH 07/26] xapcmd: support spawn options Date: Thu, 23 May 2019 09:36:45 +0000 Message-Id: <20190523093704.18367-8-e@80x24.org> In-Reply-To: <20190523093704.18367-1-e@80x24.org> References: <20190523093704.18367-1-e@80x24.org> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit List-Id: copydatabase(1) is exceptionally noisy and it's output is confusing when run in parallel. Support redirects at least, and env while we're at it to give us future options. We can also stuff a -jobs parameter into the options to limit parallelism since it can be useful for low-priority upgrade jobs. --- lib/PublicInbox/Xapcmd.pm | 21 +++++++++++++++------ 1 file changed, 15 insertions(+), 6 deletions(-) diff --git a/lib/PublicInbox/Xapcmd.pm b/lib/PublicInbox/Xapcmd.pm index 586d7e6..999ddd1 100644 --- a/lib/PublicInbox/Xapcmd.pm +++ b/lib/PublicInbox/Xapcmd.pm @@ -25,7 +25,8 @@ sub commit_changes ($$$) { } sub run { - my ($ibx, $cmd) = @_; + my ($ibx, $cmd, $env, $opt) = @_; + $opt ||= {}; my $dir = $ibx->{mainrepo} or die "no mainrepo in inbox\n"; which($cmd->[0]) or die "$cmd->[0] not found in PATH\n"; $ibx->umask_prepare; @@ -50,13 +51,21 @@ sub run { die "No Xapian parts found in $old\n" unless @cmds; } my $im = $ibx->importer(0); + my $max = $opt->{jobs} || scalar(@cmds); $ibx->with_umask(sub { $im->lock_acquire; - my %pids = map {; spawn($_) => join(' ', @$_) } @cmds; - while (scalar keys %pids) { - my $pid = waitpid(-1, 0); - my $desc = delete $pids{$pid}; - die "$desc failed: $?\n" if $?; + my %pids; + while (@cmds) { + while (scalar(keys(%pids)) < $max && scalar(@cmds)) { + my $x = shift @cmds; + $pids{spawn($x, $env, $opt)} = $x; + } + + while (scalar keys %pids) { + my $pid = waitpid(-1, 0); + my $x = delete $pids{$pid}; + die join(' ', @$x)." failed: $?\n" if $?; + } } commit_changes($im, $old, $new); }); -- EW