From: Eric Wong <e@80x24.org>
To: meta@public-inbox.org
Subject: [PATCH 00/28] cindex coderepo commit indexer
Date: Tue, 21 Mar 2023 23:07:01 +0000 [thread overview]
Message-ID: <20230321230701.3019936-1-e@80x24.org> (raw)
Not wired up to WWW nor lei, yet; but indexing + pruning of
commits works.
I'm not sure if indexing (root) tree OIDs or committer
names+emails is worth it, since I don't think those are very
important terms to search for.
I first wanted to shoehorn this into extindex, but I think it
works better as a separate Xapian schema.
It allows both internal indexes ($GIT_DIR/public-inbox-cindex)
for unforked repos, as well as extindex-style external index
to encompass several projects.
The indexer is structured a bit more nicely than existing
indexers since I'm relying on OnDestroy and `local', more.
I would like to trickle some of these improvements back to
the mail indexers at some point.
--prune and --reindex currently block incremental updates, which
isn't great since both take a while for giant Xapian DBs.
Pruning is pretty important since it's much common for coderepos
(e.g. `seen' branch of git.git)
`lei cq' will probably be a new command which behaves
similarly to `lei q -f text', but takes `git log' options
for output...
Eric Wong (28):
ipc: move nproc_shards from v2writable
search: relocate all_terms from lei_search
admin: hoist out resolve_git_dir
admin: ensure resolved GIT_DIR is absolute
test_common: create_inbox: use `$!' properly on mkdir failure
codesearch: initial cut w/ -cindex tool
cindex: parallelize prep phases
cindex: use read-only shards during prep phases
searchidxshard: improve comment wording
cindex: use DS and workqueues for parallelism
ds: @post_loop_do replaces SetPostLoopCallback
cindex: implement --exclude= like -clone
cindex: show shard number in progress message
cindex: drop `unchanged' progress message
cindex: handle graceful shutdown by default
sigfd: pass signal name rather than number to callback
cindex: implement --max-size=SIZE
cindex: check for checkpoint before giant messages
cindex: truncate or drop body for over-sized commits
cindex: attempt to give oldest commits lowest docids
cindex: improve granularity of quit checks
spawn: show failing directory for chdir failures
cindex: filter out non-existent git directories
cindex: add support for --prune
cindex: implement reindex
cindex: squelch incompatible options
cindex: respect existing permissions
cindex: ignore SIGPIPE
MANIFEST | 4 +
lib/PublicInbox/Admin.pm | 18 +-
lib/PublicInbox/CodeSearch.pm | 121 +++++
lib/PublicInbox/CodeSearchIdx.pm | 835 ++++++++++++++++++++++++++++++
lib/PublicInbox/Config.pm | 2 +-
lib/PublicInbox/DS.pm | 30 +-
lib/PublicInbox/Daemon.pm | 4 +-
lib/PublicInbox/ExtSearchIdx.pm | 2 +-
lib/PublicInbox/IPC.pm | 33 +-
lib/PublicInbox/LEI.pm | 4 +-
lib/PublicInbox/LeiSearch.pm | 14 -
lib/PublicInbox/MiscIdx.pm | 2 +-
lib/PublicInbox/Search.pm | 77 ++-
lib/PublicInbox/SearchIdx.pm | 88 ++--
lib/PublicInbox/SearchIdxShard.pm | 7 +-
lib/PublicInbox/Sigfd.pm | 10 +-
lib/PublicInbox/Spawn.pm | 6 +-
lib/PublicInbox/SpawnPP.pm | 2 +-
lib/PublicInbox/TestCommon.pm | 47 +-
lib/PublicInbox/V2Writable.pm | 26 +-
lib/PublicInbox/Watch.pm | 2 +-
script/public-inbox-cindex | 86 +++
script/public-inbox-convert | 2 +-
t/cindex.t | 134 +++++
t/dir_idle.t | 6 +-
t/ds-leak.t | 8 +-
t/imapd.t | 6 +-
t/nntpd.t | 2 +-
t/sigfd.t | 7 +-
t/watch_maildir.t | 8 +-
xt/mem-imapd-tls.t | 7 +-
xt/mem-nntpd-tls.t | 8 +-
xt/net_writer-imap.t | 4 +-
33 files changed, 1424 insertions(+), 188 deletions(-)
create mode 100644 lib/PublicInbox/CodeSearch.pm
create mode 100644 lib/PublicInbox/CodeSearchIdx.pm
create mode 100755 script/public-inbox-cindex
create mode 100644 t/cindex.t
next reply other threads:[~2023-03-21 23:07 UTC|newest]
Thread overview: 30+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-03-21 23:07 Eric Wong [this message]
2023-03-21 23:07 ` [PATCH 01/28] ipc: move nproc_shards from v2writable Eric Wong
2023-03-21 23:07 ` [PATCH 02/28] search: relocate all_terms from lei_search Eric Wong
2023-03-21 23:07 ` [PATCH 03/28] admin: hoist out resolve_git_dir Eric Wong
2023-03-21 23:07 ` [PATCH 04/28] admin: ensure resolved GIT_DIR is absolute Eric Wong
2023-03-21 23:07 ` [PATCH 05/28] test_common: create_inbox: use `$!' properly on mkdir failure Eric Wong
2023-03-21 23:07 ` [PATCH 06/28] codesearch: initial cut w/ -cindex tool Eric Wong
2023-03-21 23:07 ` [PATCH 07/28] cindex: parallelize prep phases Eric Wong
2023-03-21 23:07 ` [PATCH 08/28] cindex: use read-only shards during " Eric Wong
2023-03-21 23:07 ` [PATCH 09/28] searchidxshard: improve comment wording Eric Wong
2023-03-21 23:07 ` [PATCH 10/28] cindex: use DS and workqueues for parallelism Eric Wong
2023-03-21 23:07 ` [PATCH 11/28] ds: @post_loop_do replaces SetPostLoopCallback Eric Wong
2023-03-21 23:07 ` [PATCH 12/28] cindex: implement --exclude= like -clone Eric Wong
2023-03-21 23:07 ` [PATCH 13/28] cindex: show shard number in progress message Eric Wong
2023-03-21 23:07 ` [PATCH 14/28] cindex: drop `unchanged' " Eric Wong
2023-03-21 23:07 ` [PATCH 15/28] cindex: handle graceful shutdown by default Eric Wong
2023-03-21 23:07 ` [PATCH 16/28] sigfd: pass signal name rather than number to callback Eric Wong
2023-03-21 23:07 ` [PATCH 17/28] cindex: implement --max-size=SIZE Eric Wong
2023-03-21 23:07 ` [PATCH 18/28] cindex: check for checkpoint before giant messages Eric Wong
2023-03-21 23:07 ` [PATCH 19/28] cindex: truncate or drop body for over-sized commits Eric Wong
2023-03-21 23:07 ` [PATCH 20/28] cindex: attempt to give oldest commits lowest docids Eric Wong
2023-03-21 23:07 ` [PATCH 21/28] cindex: improve granularity of quit checks Eric Wong
2023-03-21 23:07 ` [PATCH 22/28] spawn: show failing directory for chdir failures Eric Wong
2023-03-21 23:07 ` [PATCH 23/28] cindex: filter out non-existent git directories Eric Wong
2023-03-21 23:07 ` [PATCH 24/28] cindex: add support for --prune Eric Wong
2023-03-21 23:07 ` [PATCH 25/28] cindex: implement reindex Eric Wong
2023-03-21 23:07 ` [PATCH 26/28] cindex: squelch incompatible options Eric Wong
2023-03-21 23:07 ` [PATCH 27/28] cindex: respect existing permissions Eric Wong
2023-03-21 23:07 ` [PATCH 28/28] cindex: ignore SIGPIPE Eric Wong
2023-03-24 10:40 ` [PATCH 29/28] cindex: --prune checkpoints to avoid OOM Eric Wong
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: https://public-inbox.org/README
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20230321230701.3019936-1-e@80x24.org \
--to=e@80x24.org \
--cc=meta@public-inbox.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).