From: Eric Wong <e@80x24.org>
To: meta@public-inbox.org
Subject: [PATCH 08/11] ds: avoid excessive queueing when reaping PIDs
Date: Mon, 31 Aug 2020 04:41:37 +0000 [thread overview]
Message-ID: <20200831044140.17027-9-e@80x24.org> (raw)
In-Reply-To: <20200831044140.17027-1-e@80x24.org>
We should not enqueue reap_pids() to run more than once per
EventLoop iteration. We'll start reformatting reap_pids
to tabs, too, since we're no longer Danga::Socket.
We should also be able to remove timer usage for reaping
down-the-line once we stop abusing dwaitpid() in -watch.
---
lib/PublicInbox/DS.pm | 43 ++++++++++++++++++++++++++-----------------
1 file changed, 26 insertions(+), 17 deletions(-)
diff --git a/lib/PublicInbox/DS.pm b/lib/PublicInbox/DS.pm
index a3f2e76c..b252ea3c 100644
--- a/lib/PublicInbox/DS.pm
+++ b/lib/PublicInbox/DS.pm
@@ -40,7 +40,7 @@ my $wait_pids; # list of [ pid, callback, callback_arg ]
my $later_queue; # list of callbacks to run at some later interval
my $EXPMAP; # fd -> idle_time
our $EXPTIME = 180; # 3 minutes
-my ($later_timer, $reap_timer, $exp_timer);
+my ($later_timer, $reap_armed, $reap_timer, $exp_timer);
my $ToClose; # sockets to close when event loop is done
our (
%DescriptorMap, # fd (num) -> PublicInbox::DS object
@@ -68,7 +68,7 @@ Reset all state
=cut
sub Reset {
%DescriptorMap = ();
- $in_loop = $wait_pids = $later_queue = undef;
+ $in_loop = $wait_pids = $later_queue = $reap_armed = undef;
$EXPMAP = {};
$nextq = $ToClose = $reap_timer = $later_timer = $exp_timer = undef;
$LoopTimeout = -1; # no timeout by default
@@ -225,24 +225,33 @@ sub RunTimers {
# and other things. So we scan the $wait_pids list, which is hopefully
# not too big. We keep $wait_pids small by not calling dwaitpid()
# until we've hit EOF when reading the stdout of the child.
+
sub reap_pids {
- my $tmp = $wait_pids or return;
- $wait_pids = $reap_timer = undef;
- foreach my $ary (@$tmp) {
- my ($pid, $cb, $arg) = @$ary;
- my $ret = waitpid($pid, WNOHANG);
- if ($ret == 0) {
- push @$wait_pids, $ary; # autovivifies @$wait_pids
- } elsif ($cb) {
- eval { $cb->($arg, $pid) };
- }
- }
- # we may not be done, yet, and could've missed/masked a SIGCHLD:
- $reap_timer = add_timer(1, \&reap_pids) if $wait_pids;
+ $reap_armed = undef;
+ my $tmp = $wait_pids or return;
+ $wait_pids = undef;
+ foreach my $ary (@$tmp) {
+ my ($pid, $cb, $arg) = @$ary;
+ my $ret = waitpid($pid, WNOHANG);
+ if ($ret == 0) {
+ push @$wait_pids, $ary; # autovivifies @$wait_pids
+ } elsif ($cb) {
+ eval { $cb->($arg, $pid) };
+ }
+ }
+ # we may not be done, yet, and could've missed/masked a SIGCHLD:
+ if ($wait_pids && !$reap_armed) {
+ $reap_timer //= add_timer(1, \&reap_pids_timed);
+ }
+}
+
+sub reap_pids_timed {
+ $reap_timer = undef;
+ goto \&reap_pids;
}
# reentrant SIGCHLD handler (since reap_pids is not reentrant)
-sub enqueue_reap ($) { push @$nextq, \&reap_pids }; # autovivifies
+sub enqueue_reap { $reap_armed //= requeue(\&reap_pids) }
sub in_loop () { $in_loop }
@@ -627,7 +636,7 @@ sub dwaitpid ($$$) {
push @$wait_pids, [ @_ ]; # [ $pid, $cb, $arg ]
# We could've just missed our SIGCHLD, cover it, here:
- requeue(\&reap_pids);
+ goto &enqueue_reap; # tail recursion
}
sub _run_later () {
next prev parent reply other threads:[~2020-08-31 4:41 UTC|newest]
Thread overview: 12+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-08-31 4:41 [PATCH 00/11] watch: fix contention w/ Maildir & NNTP Eric Wong
2020-08-31 4:41 ` [PATCH 01/11] watch: limit batch size of NNTP and IMAP workers, too Eric Wong
2020-08-31 4:41 ` [PATCH 02/11] watchmaildir: use v5.10.1, drop warnings Eric Wong
2020-08-31 4:41 ` [PATCH 03/11] rename WatchMaildir => Watch Eric Wong
2020-08-31 4:41 ` [PATCH 04/11] watch: log signal activities to STDERR Eric Wong
2020-08-31 4:41 ` [PATCH 05/11] watch: avoid unnecessary spawning on spam removals Eric Wong
2020-08-31 4:41 ` [PATCH 06/11] watch: block signals before fork on non-signalfd/kevent systems Eric Wong
2020-08-31 4:41 ` [PATCH 07/11] watch: comments and tiny cleanups Eric Wong
2020-08-31 4:41 ` Eric Wong [this message]
2020-08-31 4:41 ` [PATCH 09/11] watch: use EOFpipe to reduce dwaitpid wakeups Eric Wong
2020-08-31 4:41 ` [PATCH 10/11] ds: avoid unnecessary timer for waitpid Eric Wong
2020-08-31 4:41 ` [PATCH 11/11] replace ParentPipe with EOFpipe Eric Wong
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: https://public-inbox.org/README
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20200831044140.17027-9-e@80x24.org \
--to=e@80x24.org \
--cc=meta@public-inbox.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).