unofficial mirror of meta@public-inbox.org
 help / color / mirror / Atom feed
From: Eric Wong <e@80x24.org>
To: meta@public-inbox.org
Subject: [PATCH 3/2] qspawn: workaround Perl 5.16.3 leak, re-enable Deflater
Date: Fri, 27 Sep 2019 21:01:01 +0000	[thread overview]
Message-ID: <20190927210101.q22ilbp64x6uba7i@dcvr> (raw)
In-Reply-To: <20190926015038.29770-1-e@80x24.org>

The httpd-supplied write callback is the leak culprit under Perl
5.16.3.  undef-ing it immediately after use keeps a repeated
"git fetch" loop from monotonically increasing memory and FD use
on the Perl shipped with RHEL/CentOS 7.x.

Other endpoints tested showed no increase in memory use under
constant load with "ab -HAccept-Encoding:gzip -k", including the
async psgi_qx code path used by $INBOX_URL/$OBJECT_ID/s/ via
SolverGit module.
---
 Note: I initially tried this change, but thought it only slowed
 down the leaking because I had not yet discovered the
 workaround in commit cd71a869c7e9c811
 ("ds: workaround a memory leak in Perl 5.16.x").

 Now that both leaks are worked around, memory usage is completely
 flat when repeating a single request of any type with gzip-enabled.

 lib/PublicInbox/Qspawn.pm | 4 ++++
 script/public-inbox-httpd | 8 +-------
 2 files changed, 5 insertions(+), 7 deletions(-)

diff --git a/lib/PublicInbox/Qspawn.pm b/lib/PublicInbox/Qspawn.pm
index 5a30064..cb3dc51 100644
--- a/lib/PublicInbox/Qspawn.pm
+++ b/lib/PublicInbox/Qspawn.pm
@@ -281,6 +281,10 @@ sub psgi_return {
 								$buf, $filter);
 			$wcb->($r);
 		}
+
+		# Workaround a leak under Perl 5.16.3 when combined with
+		# Plack::Middleware::Deflater:
+		$wcb = undef;
 	};
 	$limiter ||= $def_limiter ||= PublicInbox::Qspawn::Limiter->new(32);
 	my $start_cb = sub { # may run later, much later...
diff --git a/script/public-inbox-httpd b/script/public-inbox-httpd
index 9b869f9..b2464f4 100755
--- a/script/public-inbox-httpd
+++ b/script/public-inbox-httpd
@@ -24,13 +24,7 @@ my $refresh = sub {
 		my $www = PublicInbox::WWW->new;
 		$www->preload;
 		$app = builder {
-			# Perl 5.16.3 leaks in our "push" response code path
-			# (e.g. Qspawn) due to something in
-			# Plack::Util::response_cb, regardless of whether the
-			# client is sending Accept-Encoding:gzip requests.
-			# perl5180delta documents many leak fixes, so assume
-			# 5.18+ is safe for now and bump the check as-need:
-			$] >= 5.018000 and eval {
+			eval {
 				enable 'Deflater',
 					content_type => [ qw(
 						text/html
-- 
EW


      parent reply	other threads:[~2019-09-27 21:01 UTC|newest]

Thread overview: 6+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-09-26  1:50 [PATCH 0/2] leak workarounds for Perl 5.16 on CentOS/RHEL 7 Eric Wong
2019-09-26  1:50 ` [PATCH 1/2] ds: workaround a memory leak in Perl 5.16.x Eric Wong
2019-09-26  1:50 ` [PATCH 2/2] httpd: disable Deflater middleware by default on Perl <5.18 Eric Wong
2019-09-26 12:36 ` [PATCH 0/2] leak workarounds for Perl 5.16 on CentOS/RHEL 7 Konstantin Ryabitsev
2019-09-26 21:16   ` Eric Wong
2019-09-27 21:01 ` Eric Wong [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: https://public-inbox.org/README

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20190927210101.q22ilbp64x6uba7i@dcvr \
    --to=e@80x24.org \
    --cc=meta@public-inbox.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).