netd/httpd memory usage

netd/httpd memory usage

[Eric Wong]

While memory use has mostly gotten under control with scratchpad
avoidance done around Sep (after v1.9.0), it's still higher than
I like :<

Suspiciously, PublicInbox::Smsg::load_from_data still seems to
be causing a big chunk of long-lived allocations on my system:

sub load_from_data ($$) {
	my ($self) = $_[0]; # data = $_[1]
	utf8::decode($_[1]);
	(
		$self->{subject},
		$self->{from},
		$self->{references},

		# To: and Cc: are stored to optimize HDR/XHDR in NNTP since
		# some NNTP clients will use that for message displays.
		# NNTP only, and only stored in Over(view), not Xapian
		$self->{to},
		$self->{cc},

		$self->{blob},
		$self->{mid},

		# NNTP only
		$self->{bytes},
		$self->{lines}
	) = split(/\n/, $_[1]);
}

I'm only on Perl 5.28.1, and looking at Perl 5.3x deltas I see
see some regexp-related leak fixes, perhaps an upgrade will fix
it...

(or work on https://80x24.org/mwrap-perl.git to expand into C
backtraces is required...)

Of course, even single-threaded git packing is using massive memory
use with the linux.git forkgroup :<

Re: netd/httpd memory usage

[Eric Wong]

Eric Wong <e@80x24.org> wrote:
> While memory use has mostly gotten under control with scratchpad
> avoidance done around Sep (after v1.9.0), it's still higher than
> I like :<
> 
> Suspiciously, PublicInbox::Smsg::load_from_data still seems to
> be causing a big chunk of long-lived allocations on my system:

OK, it doesn't seem to be a leak.  At least the 4080 byte
allocations are for long-lived arenas, and it's the main
allocator.

> (or work on https://80x24.org/mwrap-perl.git to expand into C
> backtraces is required...)

Done :>  (but stupid expensive :x)

> Of course, even single-threaded git packing is using massive memory
> use with the linux.git forkgroup :<

Ugh, yeah.