From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on dcvr.yhbt.net X-Spam-Level: X-Spam-Status: No, score=-4.0 required=3.0 tests=ALL_TRUSTED,BAYES_00 shortcircuit=no autolearn=ham autolearn_force=no version=3.4.2 Received: from localhost (dcvr.yhbt.net [127.0.0.1]) by dcvr.yhbt.net (Postfix) with ESMTP id AC3AD1FA12 for ; Mon, 23 Nov 2020 07:06:02 +0000 (UTC) From: Eric Wong To: meta@public-inbox.org Subject: [PATCH 00/12] extindex: speed up manifest.js.gz generation Date: Mon, 23 Nov 2020 07:05:50 +0000 Message-Id: <20201123070602.9698-1-e@80x24.org> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit List-Id: manifest.js.gz generation gets faster with this series (~1000ms => ~40ms) on the current set of lore.kernel.org inboxes We may need to rely on varnish to handle things up to 30-100K inboxes, since manifest.js.gz generation won't monopolize the -httpd event loop. WwwListing (HTML) output still needs to be updated and searching for inboxes needs to be implemented along with pagination for 30-100K inboxes. Eric Wong (12): miscsearch: a new Xapian sub-DB for extindex move JSON module portability into PublicInbox::Config git: add manifest_entry method manifest: use ibx->git_epoch method for v2 inbox: git_epoch: remove ->version check miscidx: put grokmirror manifest entries in Xapian docdata extsearch: fix remaining "eindex" references miscidx: cleanup git processes after manifest indexing miscidx: store absolute git_dir of each epoch in docdata extsearchidx: do not short-circuit MiscIdx on no-op v2 prepare manifest: support faster generation via [extindex "all"] *search: simplify retry_reopen users MANIFEST | 3 + lib/PublicInbox/Config.pm | 15 ++++ lib/PublicInbox/ExtSearch.pm | 8 +- lib/PublicInbox/ExtSearchIdx.pm | 18 ++++- lib/PublicInbox/Git.pm | 53 +++++++++++++ lib/PublicInbox/Inbox.pm | 6 +- lib/PublicInbox/InboxWritable.pm | 2 - lib/PublicInbox/ManifestJsGz.pm | 108 +++++++++----------------- lib/PublicInbox/MiscIdx.pm | 125 +++++++++++++++++++++++++++++++ lib/PublicInbox/MiscSearch.pm | 98 ++++++++++++++++++++++++ lib/PublicInbox/Search.pm | 18 ++--- lib/PublicInbox/SearchIdx.pm | 7 +- lib/PublicInbox/V2Writable.pm | 5 ++ script/public-inbox-extindex | 1 + t/extsearch.t | 14 +++- t/miscsearch.t | 57 ++++++++++++++ t/www_listing.t | 5 +- 17 files changed, 446 insertions(+), 97 deletions(-) create mode 100644 lib/PublicInbox/MiscIdx.pm create mode 100644 lib/PublicInbox/MiscSearch.pm create mode 100644 t/miscsearch.t