From: Eric Wong <e@80x24.org>
To: meta@public-inbox.org
Subject: xref3 + NNTP problems...
Date: Mon, 30 Nov 2020 19:42:01 +0000 [thread overview]
Message-ID: <20201130194201.GA6687@dcvr> (raw)
In-Reply-To: <20201127095254.21624-13-e@80x24.org>
Eric Wong <e@80x24.org> wrote:
> Subject: Re: [PATCH 12/12] nntp: xref: use ->ALL extindex if available
>
> Getting Xref for cross-posted messages is an O(n) operation
> where `n' is the number of newsgroups on the server. This works
> acceptably when there are dozens of groups, but would be
> unnacceptable when there's tens of thousands of newsgroups.
>
> With ~140 newsgroups, a lore.kernel.org mirror already handles
> "XHDR Xref $MESSAGE_ID" requests around 30% faster after
> creating the xref3.idx_nntp index.
xref3-based on ContentHash is great for internal use to ensure
we don't archive redundant messages; but don't get fooled by
reused Message-IDs, either (because users can and do reuse
Message-IDs for completely different messages)
But there's also more cases where content-matching for messages
we don't match because of mailing list-added trailers
(e.g. unsubscribe and other list info info)
So every message posted to linux-mtd@lists.infradead.org
nntp://rskvuqcfnfizkjg6h5jvovwb3wkikzcwskf54lfpymus6mxrzw67b5ad.onion/org.infradead.lists.linux-mtd
fails to match cross posts to Xref:, which is bad, too...
I have an idea to make it less bad, maybe it'll work...
next prev parent reply other threads:[~2020-11-30 19:42 UTC|newest]
Thread overview: 15+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-11-27 9:52 [PATCH 00/12] some NNTP-related fixes + speedups Eric Wong
2020-11-27 9:52 ` [PATCH 01/12] nntp: use Inbox->uidvalidity instead of ->mm->created_at Eric Wong
2020-11-27 9:52 ` [PATCH 02/12] nntpd: share {groups} hash with {-by_newsgroup} in Config Eric Wong
2020-11-27 9:52 ` [PATCH 03/12] mm: min/max: return 0 instead of undef Eric Wong
2020-11-27 9:52 ` [PATCH 04/12] nntp: use grep operation for wildmat matching Eric Wong
2020-11-27 9:52 ` [PATCH 05/12] nntp: NEWNEWS: speed up filtering Eric Wong
2020-11-27 9:52 ` [PATCH 06/12] miscsearch: implement ->newsgroup_matches Eric Wong
2020-11-27 9:52 ` [PATCH 07/12] nntp: LIST ACTIVE.TIMES use angle brackets around address Eric Wong
2020-11-27 9:52 ` [PATCH 08/12] nntp: move LIST iterators to long_response Eric Wong
2020-11-27 9:52 ` [PATCH 09/12] t/extsearch: show a more realistic case Eric Wong
2020-11-27 9:52 ` [PATCH 10/12] nntp: some minor golfing Eric Wong
2020-11-27 9:52 ` [PATCH 11/12] nntp: xref: simplify sub signature Eric Wong
2020-11-27 9:52 ` [PATCH 12/12] nntp: xref: use ->ALL extindex if available Eric Wong
2020-11-30 19:42 ` Eric Wong [this message]
2020-11-30 23:37 ` [PATCH] nntp: make ->ALL Xref generation more fuzzy Eric Wong
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: https://public-inbox.org/README
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20201130194201.GA6687@dcvr \
--to=e@80x24.org \
--cc=meta@public-inbox.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).