unofficial mirror of meta@public-inbox.org
 help / color / mirror / Atom feed
From: Eric Wong <e@80x24.org>
To: meta@public-inbox.org
Subject: [PATCH 8/8] move ->ids_after from mm to over
Date: Thu, 26 Aug 2021 12:33:38 +0000	[thread overview]
Message-ID: <20210826123338.694-9-e@80x24.org> (raw)
In-Reply-To: <20210826123338.694-1-e@80x24.org>

Since we favor ->over in WWW and IMAP, move this method to
->over to reduce open files in common cases.

This fixes the /$EXTINDEX_NAME/all.mbox.gz endpoint for extindex
entries (which may get expensive...).
---
 lib/PublicInbox/Mbox.pm   | 10 ++++------
 lib/PublicInbox/Msgmap.pm | 11 -----------
 lib/PublicInbox/NNTP.pm   |  2 +-
 lib/PublicInbox/Over.pm   | 11 +++++++++++
 t/extindex-psgi.t         |  3 +++
 5 files changed, 19 insertions(+), 18 deletions(-)

diff --git a/lib/PublicInbox/Mbox.pm b/lib/PublicInbox/Mbox.pm
index 844099aa..f72af26b 100644
--- a/lib/PublicInbox/Mbox.pm
+++ b/lib/PublicInbox/Mbox.pm
@@ -161,19 +161,17 @@ sub all_ids_cb {
 			my $smsg = $ctx->{over}->get_art($num) or next;
 			return $smsg;
 		}
-		$ctx->{ids} = $ids = $ctx->{mm}->ids_after(\($ctx->{prev}));
+		$ctx->{ids} = $ids = $ctx->{over}->ids_after(\($ctx->{prev}));
 	} while (@$ids);
 }
 
 sub mbox_all_ids {
 	my ($ctx) = @_;
-	my $ibx = $ctx->{ibx};
 	my $prev = 0;
-	my $mm = $ctx->{mm} = $ibx->mm;
-	my $ids = $mm->ids_after(\$prev) or return
-		[404, [qw(Content-Type text/plain)], ["No results found\n"]];
-	$ctx->{over} = $ibx->over or
+	$ctx->{over} = $ctx->{ibx}->over or
 		return PublicInbox::WWW::need($ctx, 'Overview');
+	my $ids = $ctx->{over}->ids_after(\$prev) or return
+		[404, [qw(Content-Type text/plain)], ["No results found\n"]];
 	$ctx->{ids} = $ids;
 	$ctx->{prev} = $prev;
 	require PublicInbox::MboxGz;
diff --git a/lib/PublicInbox/Msgmap.pm b/lib/PublicInbox/Msgmap.pm
index 16a9a476..3887a9e6 100644
--- a/lib/PublicInbox/Msgmap.pm
+++ b/lib/PublicInbox/Msgmap.pm
@@ -189,17 +189,6 @@ CREATE TABLE IF NOT EXISTS meta (
 
 }
 
-# used by NNTP.pm
-sub ids_after {
-	my ($self, $num) = @_;
-	my $ids = $self->{dbh}->selectcol_arrayref(<<'', undef, $$num);
-SELECT num FROM msgmap WHERE num > ?
-ORDER BY num ASC LIMIT 1000
-
-	$$num = $ids->[-1] if @$ids;
-	$ids;
-}
-
 sub msg_range {
 	my ($self, $beg, $end, $cols) = @_;
 	$cols //= 'num,mid';
diff --git a/lib/PublicInbox/NNTP.pm b/lib/PublicInbox/NNTP.pm
index aea04c05..ea9ce183 100644
--- a/lib/PublicInbox/NNTP.pm
+++ b/lib/PublicInbox/NNTP.pm
@@ -210,7 +210,7 @@ sub listgroup_range_i {
 
 sub listgroup_all_i {
 	my ($self, $num) = @_;
-	my $ary = $self->{ibx}->mm(1)->ids_after($num);
+	my $ary = $self->{ibx}->over(1)->ids_after($num);
 	scalar(@$ary) or return;
 	more($self, join("\r\n", @$ary));
 	1;
diff --git a/lib/PublicInbox/Over.pm b/lib/PublicInbox/Over.pm
index 58fdea0e..19da056a 100644
--- a/lib/PublicInbox/Over.pm
+++ b/lib/PublicInbox/Over.pm
@@ -371,4 +371,15 @@ SELECT COUNT(*) FROM xref3 WHERE oidbin = ?
 
 sub blob_exists { oidbin_exists($_[0], pack('H*', $_[1])) }
 
+# used by NNTP.pm
+sub ids_after {
+	my ($self, $num) = @_;
+	my $ids = dbh($self)->selectcol_arrayref(<<'', undef, $$num);
+SELECT num FROM over WHERE num > ?
+ORDER BY num ASC LIMIT 1000
+
+	$$num = $ids->[-1] if @$ids;
+	$ids;
+}
+
 1;
diff --git a/t/extindex-psgi.t b/t/extindex-psgi.t
index b9acc979..d4761641 100644
--- a/t/extindex-psgi.t
+++ b/t/extindex-psgi.t
@@ -52,6 +52,9 @@ my $client = sub {
 	my $cfg = PublicInbox::Config->git_config_dump($f);
 	is($?, 0, 'no errors from git-config parsing');
 	ok($cfg->{'extindex.all.topdir'}, 'extindex.topdir defined');
+
+	$res = $cb->(GET('/all/all.mbox.gz'));
+	is($res->code, 200, 'all.mbox.gz');
 };
 test_psgi(sub { $www->call(@_) }, $client);
 %$env = (%$env, TMPDIR => $tmpdir, PI_CONFIG => $pi_config);

  parent reply	other threads:[~2021-08-26 12:33 UTC|newest]

Thread overview: 21+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-08-16 16:36 RFE: Long .onion URL breaks mobile view Konstantin Ryabitsev
2021-08-16 22:38 ` Eric Wong
2021-08-16 22:53   ` Eric Wong
2021-08-26 12:33 ` [PATCH 0/8] various WWW + extindex stuff Eric Wong
2021-08-26 12:33   ` [PATCH 1/8] get rid of unnecessary bytes::length usage Eric Wong
2021-08-26 12:33   ` [PATCH 2/8] ds: use bytes::substr and bytes::length module-wide for now Eric Wong
2021-08-26 12:33   ` [PATCH 3/8] www_stream: sh-friendly .onion URLs wrapping Eric Wong
2021-08-26 12:33   ` [PATCH 4/8] www: avoid incorrect instructions for extindex Eric Wong
2021-08-26 12:33   ` [PATCH 5/8] www_text: fix example config snippet " Eric Wong
2021-08-26 12:33   ` [PATCH 6/8] config: do not parse altid " Eric Wong
2021-08-26 12:33   ` [PATCH 7/8] www_text: add coderepo config support " Eric Wong
2021-08-26 12:33   ` Eric Wong [this message]
2021-08-26 13:27   ` [PATCH 0/8] various WWW + extindex stuff Konstantin Ryabitsev
2021-08-28 11:50     ` [PATCH 0/2] www: split out mirror to /text/ Eric Wong
2021-08-28 11:50       ` [PATCH 1/2] www: move mirror instructions " Eric Wong
2021-08-28 11:50       ` [PATCH 2/2] www_stream: description header links to top $INBOX_URL Eric Wong
2021-08-28 17:58       ` [PATCH 0/2] www: split out mirror to /text/ Konstantin Ryabitsev
2021-08-30 23:44         ` [PATCH 0/3] www: more footer and mirroring instructions tweaks Eric Wong
2021-08-30 23:44           ` [PATCH 1/3] www_stream: extra link to mirroring information in the footer Eric Wong
2021-08-30 23:44           ` [PATCH 2/3] www_text/mirror: spell out "external index" and "public inbox" Eric Wong
2021-08-30 23:44           ` [PATCH 3/3] www_listing: add note about mirroring information Eric Wong

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: https://public-inbox.org/README

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20210826123338.694-9-e@80x24.org \
    --to=e@80x24.org \
    --cc=meta@public-inbox.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).