From: "Eric Wong (Contractor, The Linux Foundation)" <e@80x24.org>
To: meta@public-inbox.org
Subject: [PATCH 0/7] optimize V2
Date: Tue, 3 Apr 2018 11:09:05 +0000 [thread overview]
Message-ID: <20180403110912.24231-1-e@80x24.org> (raw)
The switch to the SQLite overview DB introduced a regression in
the use of OFFSET. While Xapian largely unaffected by the
offset for pagination, SQLite basically gets the full result and
cuts it at the offset, so using large offsets gets slower as
time goes on.
There are some good side-effects of this, too:
* Using last-hop-Received timestamps for HTML pagination ought
to improve cacheability on the client-side and in search engines.
* Switching to using NNTP article numbers gives us race avoidance
for NEWNEWS so duplicates won't show up while iterating
through result batches (not that I've seen any logs this
command being used in the wild...)
I don't believe there are any more regressions from switching
to the Xapian skeleton to a SQLite over(view) DB. Homepage
performance seems consistent whether it's 2.8 million messages
or 2-3x that.
Now, NNTP performance should be improved across the board and
commands like XOVER/XHDR are over 20x faster than they were
in v1 with Xapian.
There's also a few cleanups and code simplifications
which should make future work easier.
Eric Wong (Contractor, The Linux Foundation) (7):
t/thread-all.t: modernize test to support modern inboxes
rename+rewrite test using Benchmark module
nntp: make XOVER, XHDR, OVER, HDR and NEWNEWS faster
view: avoid offset during pagination
mbox: remove remaining OFFSET usage in SQLite
msgmap: replace id_batch with ids_after
nntp: simplify the long_response API
MANIFEST | 3 +-
lib/PublicInbox/Feed.pm | 25 +--------
lib/PublicInbox/Inbox.pm | 4 +-
lib/PublicInbox/Mbox.pm | 30 ++++++++---
lib/PublicInbox/Msgmap.pm | 29 +++++++----
lib/PublicInbox/NNTP.pm | 114 ++++++++++++++++++----------------------
lib/PublicInbox/Over.pm | 50 ++++++++++++------
lib/PublicInbox/Search.pm | 8 +--
lib/PublicInbox/View.pm | 90 +++++++++++++++++++++++---------
t/nntpd.t | 1 +
t/perf-nntpd.t | 130 ++++++++++++++++++++++++++++++++++++++++++++++
t/perf-threading.t | 32 ++++++++++++
t/psgi_v2.t | 22 +++++++-
t/thread-all.t | 38 --------------
t/v2writable.t | 17 ++++++
15 files changed, 403 insertions(+), 190 deletions(-)
create mode 100644 t/perf-nntpd.t
create mode 100644 t/perf-threading.t
delete mode 100644 t/thread-all.t
--
EW
next reply other threads:[~2018-04-03 11:09 UTC|newest]
Thread overview: 8+ messages / expand[flat|nested] mbox.gz Atom feed top
2018-04-03 11:09 Eric Wong (Contractor, The Linux Foundation) [this message]
2018-04-03 11:09 ` [PATCH 1/7] t/thread-all.t: modernize test to support modern inboxes Eric Wong (Contractor, The Linux Foundation)
2018-04-03 11:09 ` [PATCH 2/7] rename+rewrite test using Benchmark module Eric Wong (Contractor, The Linux Foundation)
2018-04-03 11:09 ` [PATCH 3/7] nntp: make XOVER, XHDR, OVER, HDR and NEWNEWS faster Eric Wong (Contractor, The Linux Foundation)
2018-04-03 11:09 ` [PATCH 4/7] view: avoid offset during pagination Eric Wong (Contractor, The Linux Foundation)
2018-04-03 11:09 ` [PATCH 5/7] mbox: remove remaining OFFSET usage in SQLite Eric Wong (Contractor, The Linux Foundation)
2018-04-03 11:09 ` [PATCH 6/7] msgmap: replace id_batch with ids_after Eric Wong (Contractor, The Linux Foundation)
2018-04-03 11:09 ` [PATCH 7/7] nntp: simplify the long_response API Eric Wong (Contractor, The Linux Foundation)
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: https://public-inbox.org/README
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20180403110912.24231-1-e@80x24.org \
--to=e@80x24.org \
--cc=meta@public-inbox.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).