unofficial mirror of meta@public-inbox.org
 help / color / mirror / Atom feed
* [PATCH] pop3: reduce memory use while generating the mailbox cache
@ 2022-07-23  6:12 Eric Wong
  0 siblings, 0 replies; only message in thread
From: Eric Wong @ 2022-07-23  6:12 UTC (permalink / raw)
  To: meta

While the cache itself is relatively compact for 50K messages,
generating it was inefficient due to our schema and Over.pm APIs
being designed for NNTP.  While we won't change our schema for
now, we can choose better DBI APIs to use and limit our ephemeral
memory use.

This amounts to a 60% reduction in memory usage and a 5-10%
speedup against org.kernel.vger.git.0:

	{
		echo 'USER '$(uuidgen)'@org.kernel.vger.git.0'
		echo PASS anonymous
		echo STAT
		echo QUIT
	} | nc $HOST $PORT
---
 lib/PublicInbox/POP3.pm | 15 ++++++++++++---
 1 file changed, 12 insertions(+), 3 deletions(-)

diff --git a/lib/PublicInbox/POP3.pm b/lib/PublicInbox/POP3.pm
index 60eedea7..203c91a6 100644
--- a/lib/PublicInbox/POP3.pm
+++ b/lib/PublicInbox/POP3.pm
@@ -148,12 +148,21 @@ sub _stat_cache ($) {
 	my ($self) = @_;
 	my ($beg, $end) = (($self->{uid_dele} // -1) + 1, $self->{uid_max});
 	PublicInbox::IMAP::uid_clamp($self, \$beg, \$end);
-	my $opt = { limit => PublicInbox::IMAP::UID_SLICE };
-	my $m = $self->{ibx}->over(1)->do_get(<<'', $opt, $beg, $end);
+	my (@cache, $m);
+	my $sth = $self->{ibx}->over(1)->dbh->prepare_cached(<<'', undef, 1);
 SELECT num,ddd FROM over WHERE num >= ? AND num <= ?
 ORDER BY num ASC
 
-	[ map { ($_->{num}, $_->{bytes} + 0, $_->{blob}) } @$m ];
+	$sth->execute($beg, $end);
+	do {
+		$m = $sth->fetchall_arrayref({}, 1000);
+		for my $x (@$m) {
+			PublicInbox::Over::load_from_row($x);
+			push(@cache, $x->{num}, $x->{bytes} + 0, $x->{blob});
+			undef $x; # saves ~1.5M memory w/ 50k messages
+		}
+	} while (scalar(@$m) && ($beg = $cache[-3] + 1));
+	\@cache;
 }
 
 sub cmd_stat {

^ permalink raw reply	[flat|nested] only message in thread

only message in thread, other threads:[~2022-07-23  6:12 UTC | newest]

Thread overview: (only message) (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-07-23  6:12 [PATCH] pop3: reduce memory use while generating the mailbox cache Eric Wong

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).