From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <e@80x24.org>
X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on dcvr.yhbt.net
X-Spam-Level: 
X-Spam-ASN:  
X-Spam-Status: No, score=-4.0 required=3.0 tests=ALL_TRUSTED,BAYES_00
	shortcircuit=no autolearn=ham autolearn_force=no version=3.4.2
Received: from localhost (dcvr.yhbt.net [127.0.0.1])
	by dcvr.yhbt.net (Postfix) with ESMTP id BE3091F463;
	Thu, 26 Sep 2019 21:16:20 +0000 (UTC)
Date: Thu, 26 Sep 2019 21:16:20 +0000
From: Eric Wong <e@80x24.org>
To: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
Cc: meta@public-inbox.org
Subject: Re: [PATCH 0/2] leak workarounds for Perl 5.16 on CentOS/RHEL 7
Message-ID: <20190926211620.GB11146@dcvr>
References: <20190926015038.29770-1-e@80x24.org>
 <20190926123606.GA10467@chatter.i7.local>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
In-Reply-To: <20190926123606.GA10467@chatter.i7.local>
List-Id: <meta.public-inbox.org>

Konstantin Ryabitsev <konstantin@linuxfoundation.org> wrote:
> On Thu, Sep 26, 2019 at 01:50:36AM +0000, Eric Wong wrote:
> > After many hours of reviewing our code in PublicInbox::Qspawn,
> > PublicInbox::GitHTTPBackend, and PublicInbox::HTTP and finding
> > nothing but cleanups and documentation improvements; it seems
> > the leaks affecting lore is down to bugs in Perl 5.16.3.
> > 
> > After removing the warning for Deflater being missing in
> > d883d4a93b23be134038e28f421eafca70c3d838
> > ("httpd: get rid of Deflater warning"), I missed that my
> > own CentOS 7 VM was missing that module so was unable to
> > reproduce the FD leaks :x
> > 
> > The first patch is a straightforward workaround that I was
> > able to test without Plack::Middleware::Deflater being installed.
> > 
> > The second patch stops loading Deflater in httpd on Perls
> > earlier than 5.18.  (But I haven't built or tested 5.18 myself).
> > Enabling gzip in varnish will be needed for 5.16 users.
> 
> Yay, I can confirm that the latest master fixes all the FD leaks that have
> been plaguing lore.kernel.org for the past few weeks. The number of pipes is
> stable and there are no (deleted) tempfiles showing up. Thanks so much for
> this!

Good to know!  Also, how's memory use?

It ought to be stable if Perl is doing the right thing; but I
haven't tested all the endpoints thoroughly on 5.16.3
(especially the mbox.gz ones)

> Can you elaborate on gzip+varnish changes? I'm assuming it's something as
> simple as:
> 
> @@ -63,6 +63,10 @@
>        } else {
>                /* short TTL for up-to-dateness, our PSGI is not that slow */
>                set beresp.ttl = 10s;
> +               /* Compress text responses on CentOS7 */
> +               if (beresp.http.content-type ~ "text") {
> +                       set beresp.do_gzip = true;
> +               }

*shrug*  Whatever the varnish docs says :)

I still prefer to do gzip in the Perl process to minimize IPC
overhead (and to eventually make things easier-to-deploy w/o
needing Varnish).

I think I can add gzip support to WwwStream and WwwAtomStream
pretty easily, and we're already using a custom gzipper for
the mboxrd endpoints.