From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <e@80x24.org>
X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on dcvr.yhbt.net
X-Spam-Level: 
X-Spam-ASN:  
X-Spam-Status: No, score=-4.0 required=3.0 tests=ALL_TRUSTED,AWL,BAYES_00
	shortcircuit=no autolearn=ham autolearn_force=no version=3.4.2
Received: from localhost (dcvr.yhbt.net [127.0.0.1])
	by dcvr.yhbt.net (Postfix) with ESMTP id A2FE41F462;
	Wed, 12 Jun 2019 17:08:15 +0000 (UTC)
Date: Wed, 12 Jun 2019 17:08:15 +0000
From: Eric Wong <e@80x24.org>
To: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
Cc: meta@public-inbox.org
Subject: Re: how's memory usage on public-inbox-httpd?
Message-ID: <20190612170815.42trclr7kfmu6ewk@dcvr>
References: <20181201194429.d5aldesjkb56il5c@dcvr>
 <20190606190455.GA17362@chatter.i7.local>
 <20190606203752.7wpdla5ynemjlshs@dcvr>
 <20190606214509.GA4087@chatter.i7.local>
 <20190606221009.y4fe2e2rervvq3z4@dcvr>
 <20190609083918.gfr2kurah7f2hysx@dcvr>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
In-Reply-To: <20190609083918.gfr2kurah7f2hysx@dcvr>
List-Id: <meta.public-inbox.org>

Eric Wong <e@80x24.org> wrote:
> Maybe you have bigger messages outside of LKML.
> This prints all objects >1MB in a git dir:
> 
> 	git cat-file --buffer --batch-check --batch-all-objects \
> 		--unordered | awk '$3 > 1048576 { print }'
> 
> And I also remember you're supporting non-vger lists where HTML
> mail is allowed, so that can't be good for memory use at all :<

Btw, have you had a chance to run the above to scan your repos?

The 19GB RSS would interact poorly with more frequent fork+exec;
such as the change to do unconditional expiry of long-lived
cat-file --batch* processes:

	https://public-inbox.org/meta/20190601033706.18113-2-e@80x24.org/


If you run Inline::C + PERL_INLINE_DIRECTORY to use vfork
instead of fork, then RSS won't affect process spawning
speeds.

But yeah, 19GB RSS still sucks and I'd like to get to the bottom
of it.