unofficial mirror of meta@public-inbox.org
 help / color / mirror / Atom feed
* [PATCH 0/7] optimize V2
@ 2018-04-03 11:09 Eric Wong (Contractor, The Linux Foundation)
  2018-04-03 11:09 ` [PATCH 1/7] t/thread-all.t: modernize test to support modern inboxes Eric Wong (Contractor, The Linux Foundation)
                   ` (6 more replies)
  0 siblings, 7 replies; 8+ messages in thread
From: Eric Wong (Contractor, The Linux Foundation) @ 2018-04-03 11:09 UTC (permalink / raw)
  To: meta

The switch to the SQLite overview DB introduced a regression in
the use of OFFSET.  While Xapian largely unaffected by the
offset for pagination, SQLite basically gets the full result and
cuts it at the offset, so using large offsets gets slower as
time goes on.

There are some good side-effects of this, too:

* Using last-hop-Received timestamps for HTML pagination ought
  to improve cacheability on the client-side and in search engines.

* Switching to using NNTP article numbers gives us race avoidance
  for NEWNEWS so duplicates won't show up while iterating
  through result batches (not that I've seen any logs this
  command being used in the wild...)

I don't believe there are any more regressions from switching
to the Xapian skeleton to a SQLite over(view) DB.  Homepage
performance seems consistent whether it's 2.8 million messages
or 2-3x that.

Now, NNTP performance should be improved across the board and
commands like XOVER/XHDR are over 20x faster than they were
in v1 with Xapian.

There's also a few cleanups and code simplifications
which should make future work easier.

Eric Wong (Contractor, The Linux Foundation) (7):
  t/thread-all.t: modernize test to support modern inboxes
  rename+rewrite test using Benchmark module
  nntp: make XOVER, XHDR, OVER, HDR and NEWNEWS faster
  view: avoid offset during pagination
  mbox: remove remaining OFFSET usage in SQLite
  msgmap: replace id_batch with ids_after
  nntp: simplify the long_response API

 MANIFEST                  |   3 +-
 lib/PublicInbox/Feed.pm   |  25 +--------
 lib/PublicInbox/Inbox.pm  |   4 +-
 lib/PublicInbox/Mbox.pm   |  30 ++++++++---
 lib/PublicInbox/Msgmap.pm |  29 +++++++----
 lib/PublicInbox/NNTP.pm   | 114 ++++++++++++++++++----------------------
 lib/PublicInbox/Over.pm   |  50 ++++++++++++------
 lib/PublicInbox/Search.pm |   8 +--
 lib/PublicInbox/View.pm   |  90 +++++++++++++++++++++++---------
 t/nntpd.t                 |   1 +
 t/perf-nntpd.t            | 130 ++++++++++++++++++++++++++++++++++++++++++++++
 t/perf-threading.t        |  32 ++++++++++++
 t/psgi_v2.t               |  22 +++++++-
 t/thread-all.t            |  38 --------------
 t/v2writable.t            |  17 ++++++
 15 files changed, 403 insertions(+), 190 deletions(-)
 create mode 100644 t/perf-nntpd.t
 create mode 100644 t/perf-threading.t
 delete mode 100644 t/thread-all.t

-- 
EW


^ permalink raw reply	[flat|nested] 8+ messages in thread

end of thread, other threads:[~2018-04-03 11:09 UTC | newest]

Thread overview: 8+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2018-04-03 11:09 [PATCH 0/7] optimize V2 Eric Wong (Contractor, The Linux Foundation)
2018-04-03 11:09 ` [PATCH 1/7] t/thread-all.t: modernize test to support modern inboxes Eric Wong (Contractor, The Linux Foundation)
2018-04-03 11:09 ` [PATCH 2/7] rename+rewrite test using Benchmark module Eric Wong (Contractor, The Linux Foundation)
2018-04-03 11:09 ` [PATCH 3/7] nntp: make XOVER, XHDR, OVER, HDR and NEWNEWS faster Eric Wong (Contractor, The Linux Foundation)
2018-04-03 11:09 ` [PATCH 4/7] view: avoid offset during pagination Eric Wong (Contractor, The Linux Foundation)
2018-04-03 11:09 ` [PATCH 5/7] mbox: remove remaining OFFSET usage in SQLite Eric Wong (Contractor, The Linux Foundation)
2018-04-03 11:09 ` [PATCH 6/7] msgmap: replace id_batch with ids_after Eric Wong (Contractor, The Linux Foundation)
2018-04-03 11:09 ` [PATCH 7/7] nntp: simplify the long_response API Eric Wong (Contractor, The Linux Foundation)

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).