From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on dcvr.yhbt.net X-Spam-Level: X-Spam-ASN: X-Spam-Status: No, score=-4.0 required=3.0 tests=ALL_TRUSTED,BAYES_00 shortcircuit=no autolearn=ham autolearn_force=no version=3.4.2 Received: from localhost (dcvr.yhbt.net [127.0.0.1]) by dcvr.yhbt.net (Postfix) with ESMTP id E5D961F8C1; Fri, 8 May 2020 01:59:01 +0000 (UTC) Date: Fri, 8 May 2020 01:59:01 +0000 From: Eric Wong To: meta@public-inbox.org Subject: [PATCH] www: preload: load all encodings at startup Message-ID: <20200508015901.GA27432@dcvr> References: <20200319083256.15593-1-e@yhbt.net> <20200421085208.GA28087@dcvr> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: <20200421085208.GA28087@dcvr> List-Id: Eric Wong wrote: > Eric Wong wrote: > > For long-lived daemons, perform immortal allocations as early as > > possible to reduce the likelyhood of heap fragmentation due to > > mixed-lifetime allocations happening once the process is fully > > loaded and serving requests, since per-request allocations > > should all be short-lived. > > Encode also loads lazily... -----------8<--------- Subject: [PATCH] www: preload: load all encodings at startup Encode lazy-loads encodings on an as-needed basis. This is great for short-lived programs, but leads to fragmentation in long-lived daemons where immortal allocations can get interleaved with short-lived, per-request allocations. Since we have no idea which encodings will be needed when there's a constant flow of incoming mail, just preload everything available at startup. --- lib/PublicInbox/WWW.pm | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/lib/PublicInbox/WWW.pm b/lib/PublicInbox/WWW.pm index 275e509f..3a428218 100644 --- a/lib/PublicInbox/WWW.pm +++ b/lib/PublicInbox/WWW.pm @@ -141,6 +141,12 @@ sub call { # fragmentation since common allocators favor a large contiguous heap. sub preload { my ($self) = @_; + + # populate caches used by Encode internally, since emails + # may show up with any encoding. + require Encode; + Encode::find_encoding($_) for Encode->encodings(':all'); + require PublicInbox::ExtMsg; require PublicInbox::Feed; require PublicInbox::View;