From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on dcvr.yhbt.net X-Spam-Level: X-Spam-ASN: X-Spam-Status: No, score=-4.1 required=3.0 tests=ALL_TRUSTED,AWL,BAYES_00, T_SCC_BODY_TEXT_LINE shortcircuit=no autolearn=ham autolearn_force=no version=3.4.6 Received: from localhost (dcvr.yhbt.net [127.0.0.1]) by dcvr.yhbt.net (Postfix) with ESMTP id 749741F44D for ; Sat, 24 Feb 2024 08:52:50 +0000 (UTC) From: Eric Wong To: meta@public-inbox.org Subject: [RFC] daemon: experimental PREALLOC_NR env knob Date: Sat, 24 Feb 2024 08:52:50 +0000 Message-ID: <20240224085250.3632004-1-e@80x24.org> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit List-Id: As I've observed using the mwrap-perl LD_PRELOAD wrapper, permanent 4080-byte arenas allocated by Perl late in the process will impede consolidation of freed adjacent blocks. In long-lived processes, this fragmentation from immortal arenas near the "wilderness"[1] area can force excessive memory to be requested from the kernel (via sbrk or mmap). Combining this with MALLOC_MMAP_THRESHOLD_=131072 appears to reduce memory use of long-lived, heavily-trafficked processes. [1] this is dlmalloc terminology: https://gee.cs.oswego.edu/dl/html/malloc.html --- Been testing various iterations and refining this off/on for a few weeks, now. Immortal allocations sprinkled throughout the heap is bad and Perl's arenas are a major remaining culprit... lib/PublicInbox/Daemon.pm | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/lib/PublicInbox/Daemon.pm b/lib/PublicInbox/Daemon.pm index e578f2e8..a80f2cbb 100644 --- a/lib/PublicInbox/Daemon.pm +++ b/lib/PublicInbox/Daemon.pm @@ -36,6 +36,22 @@ my %SCHEME2PORT = map { $KNOWN_TLS{$_} => $_ + 0 } keys %KNOWN_TLS; for (keys %KNOWN_STARTTLS) { $SCHEME2PORT{$KNOWN_STARTTLS{$_}} = $_ + 0 } $SCHEME2PORT{http} = 80; +# Preallocate SV and HE arenas early so they don't end up in/near +# the "wilderness" (in dlmalloc terminology). This should result +# in lower fragmentation and memory use. +# Not sure if other arenas can benefit from this. +# using PREALLOC_NR=500000 for the yhbt.net/lore mirror +BEGIN { + if (my $nr = $ENV{PREALLOC_NR}) { + my @tmp = map { $_ => $_ } (0..$nr); + # some extra arenas for HVs and AVs + $nr >>= 5; + $nr = 10000 if $nr < 10000; + @tmp = map { +{} } (0..$nr); + @tmp = map { [ $_ ] } (0..$nr); + } +} + our ($parent_pipe, %POST_ACCEPT, %XNETD); our %WORKER_SIG = ( INT => \&worker_quit,