From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on dcvr.yhbt.net X-Spam-Level: X-Spam-ASN: X-Spam-Status: No, score=-4.2 required=3.0 tests=ALL_TRUSTED,AWL,BAYES_00, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF shortcircuit=no autolearn=ham autolearn_force=no version=3.4.6 Received: from localhost (dcvr.yhbt.net [127.0.0.1]) by dcvr.yhbt.net (Postfix) with ESMTP id 2DB821F406 for ; Thu, 24 Aug 2023 01:22:36 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=80x24.org; s=selector1; t=1692840156; bh=i+++qFnEsKB/q64M7Vt9y/rElVmXJXQre63oak81V7g=; h=From:To:Subject:Date:From; b=ThYhi6RKgyKR5pW8qzET2oi/6KdqpVqo8yi30yn7xq/aQcXqctVvRjgABN3tgh8hi RDCdtRnfHcq/oPjkBPmB9cONSz3EoPf0jG2FsCqnLOt1G0KTXLUNfy8inzEU7njcS5 l4yhPB9MaZThucsFTPE85PcupL3riCoNkOSMJSJU= From: Eric Wong To: meta@public-inbox.org Subject: [PATCH 0/7] cindex: optional C++ Xapian helper Date: Thu, 24 Aug 2023 01:22:29 +0000 Message-Id: <20230824012236.3968030-1-e@80x24.org> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit List-Id: Associating inboxes with coderepos is an extremely expensive operation, especially for Perl (even with XS or SWIG) as Perl's method dispatch overhead to dump data out of Xapian becomes noticeable. The actual association is fast with POSIX sort(1) and join(1); but getting the necessary data out of Xapian to join on is expensive as neither quest(1) nor xapian-delve(1) are suitable for this task. The actual association data isn't stored or usable anywhere, yet, and some of them are too loose to be useful. More work is required on that point.... The association could probably be faster with rculfhash (from Userspace-RCU), but I don't think it's worth the maintenance and installation overhead for this (though I intend to use rculfhash for the FUSE shim). These performance problems weren't as noticeable in the past since our other Xapian uses spent significant amounts of time when retrieving document data from SQLite and blobs from git. Using the C++ implementation of xap_helper.h allows a full join (without limits or date ranges) of lore + git.kernel.org repos within one hour on my ancient system while the Perl+(XS|SWIG) implementation took roughly 8 hours. Eric Wong (7): search: hoist out shards_dir for future use cindex: read-only association dump cindex: add --show-roots switch introduce optional C++ xap_helper cindex: fix sorting and uniqueness cindex: implement dump_roots in C++ xap_helper: reopen+retry in MSetIterator loops MANIFEST | 7 + lib/PublicInbox/CidxRecvIbx.pm | 46 ++ lib/PublicInbox/CidxXapHelperAux.pm | 44 ++ lib/PublicInbox/CodeSearch.pm | 54 +- lib/PublicInbox/CodeSearchIdx.pm | 349 ++++++++-- lib/PublicInbox/Config.pm | 2 +- lib/PublicInbox/Isearch.pm | 5 + lib/PublicInbox/Search.pm | 92 ++- lib/PublicInbox/XapClient.pm | 50 ++ lib/PublicInbox/XapHelper.pm | 226 +++++++ lib/PublicInbox/XapHelperCxx.pm | 93 +++ lib/PublicInbox/xap_helper.h | 947 ++++++++++++++++++++++++++++ script/public-inbox-cindex | 4 +- t/xap_helper.t | 175 +++++ 14 files changed, 2021 insertions(+), 73 deletions(-) create mode 100644 lib/PublicInbox/CidxRecvIbx.pm create mode 100644 lib/PublicInbox/CidxXapHelperAux.pm create mode 100644 lib/PublicInbox/XapClient.pm create mode 100644 lib/PublicInbox/XapHelper.pm create mode 100644 lib/PublicInbox/XapHelperCxx.pm create mode 100644 lib/PublicInbox/xap_helper.h create mode 100644 t/xap_helper.t