Re: [PATCH 0/2] leak workarounds for Perl 5.16 on CentOS/RHEL 7

[Eric Wong <e@80x24.org>]

Konstantin Ryabitsev <konstantin@linuxfoundation.org> wrote:
> On Thu, Sep 26, 2019 at 01:50:36AM +0000, Eric Wong wrote:
> > After many hours of reviewing our code in PublicInbox::Qspawn,
> > PublicInbox::GitHTTPBackend, and PublicInbox::HTTP and finding
> > nothing but cleanups and documentation improvements; it seems
> > the leaks affecting lore is down to bugs in Perl 5.16.3.
> > 
> > After removing the warning for Deflater being missing in
> > d883d4a93b23be134038e28f421eafca70c3d838
> > ("httpd: get rid of Deflater warning"), I missed that my
> > own CentOS 7 VM was missing that module so was unable to
> > reproduce the FD leaks :x
> > 
> > The first patch is a straightforward workaround that I was
> > able to test without Plack::Middleware::Deflater being installed.
> > 
> > The second patch stops loading Deflater in httpd on Perls
> > earlier than 5.18.  (But I haven't built or tested 5.18 myself).
> > Enabling gzip in varnish will be needed for 5.16 users.
> 
> Yay, I can confirm that the latest master fixes all the FD leaks that have
> been plaguing lore.kernel.org for the past few weeks. The number of pipes is
> stable and there are no (deleted) tempfiles showing up. Thanks so much for
> this!

Good to know!  Also, how's memory use?

It ought to be stable if Perl is doing the right thing; but I
haven't tested all the endpoints thoroughly on 5.16.3
(especially the mbox.gz ones)

> Can you elaborate on gzip+varnish changes? I'm assuming it's something as
> simple as:
> 
> @@ -63,6 +63,10 @@
>        } else {
>                /* short TTL for up-to-dateness, our PSGI is not that slow */
>                set beresp.ttl = 10s;
> +               /* Compress text responses on CentOS7 */
> +               if (beresp.http.content-type ~ "text") {
> +                       set beresp.do_gzip = true;
> +               }

*shrug*  Whatever the varnish docs says :)

I still prefer to do gzip in the Perl process to minimize IPC
overhead (and to eventually make things easier-to-deploy w/o
needing Varnish).

I think I can add gzip support to WwwStream and WwwAtomStream
pretty easily, and we're already using a custom gzipper for
the mboxrd endpoints.