From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <e@80x24.org>
X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on dcvr.yhbt.net
X-Spam-Level: 
X-Spam-ASN:  
X-Spam-Status: No, score=-4.0 required=3.0 tests=ALL_TRUSTED,BAYES_00
	shortcircuit=no autolearn=ham autolearn_force=no version=3.4.2
Received: from localhost (dcvr.yhbt.net [127.0.0.1])
	by dcvr.yhbt.net (Postfix) with ESMTP id 7933A1F8C8
	for <meta@public-inbox.org>; Mon, 27 Sep 2021 07:10:10 +0000 (UTC)
Date: Mon, 27 Sep 2021 07:10:10 +0000
From: Eric Wong <e@80x24.org>
To: meta@public-inbox.org
Subject: Re: httpd memory usage?
Message-ID: <20210927071010.GA27389@dcvr>
References: <20210904235305.GA22009@dcvr>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20210904235305.GA22009@dcvr>
List-Id: <meta.public-inbox.org>

Eric Wong <e@80x24.org> wrote:
> Is it high for anybody else?  I'm not seeing it for small
> instances (e.g. the instance that's powering
> public-inbox.org/{git,meta} is fine, but with lots of entries my
> lore mirror @ https://yhbt.net/lore/ the main sbrk heap seems to
> be growing slowly without bounds.

I suspect it's long-lived SQLite connections and associated
caches/etc (and having short-lived allocations mixed in with
long-term ones).

Anyways, I'm testing the below on yhbt.net/lore, but
there's other SQLite knobs worth investigating...
Preloading all DBs isn't realistic, either, due to the
potential number of inboxes.

diff --git a/lib/PublicInbox/Inbox.pm b/lib/PublicInbox/Inbox.pm
index 1d5fc708..8fcf6e81 100644
--- a/lib/PublicInbox/Inbox.pm
+++ b/lib/PublicInbox/Inbox.pm
@@ -43,7 +43,7 @@ sub cleanup_task () {
 		for my $git (@{$ibx->{-repo_objs}}) {
 			$again = 1 if $git->cleanup;
 		}
-		check_inodes($ibx);
+		delete @$ibx{qw(over mm)};
 		$next->{"$ibx"} = $ibx if $again;
 	}
 	$CLEANUP = $next;

> I think I'll need to finish porting mwrap over to Perl/XS
> (from Ruby/C)...

Yeah, still on my TODO list.  But right now lei IMAP(S) on
high-latency links is a terrible...