From: Eric Wong <e@80x24.org>
To: meta@public-inbox.org
Subject: [PATCH 00/22] lei query overview views
Date: Sun, 10 Jan 2021 12:14:57 +0000 [thread overview]
Message-ID: <20210110121519.17044-1-e@80x24.org> (raw)
Usage summary:
lei add-external /path/to/v1-or-v2-inbox
lei add-external /path/to/another-inbox-or-ext-index
# URLs aren't supported, yet :<
lei q SEARCH TERMS GO HERE... # pager should open with JSON output
For faster startup time than what Inline::C can give:
apt-get install libsocket-msghdr-perl # Socket::Msghdr
Having neither Inline::C nor Socket::Msghdr means parallel
queries won't work.
I went back-and-forth on a bunch of things but ultimately gave
up trying to support IO::FDPass since it got too fragile and
difficult to test with the work-queue distribution.
The pager runs from the client process (if using Socket::MsgHdr
or Inline::C), now. It took at fair amount of work from my slow
brain to get pager shutdown to be instantaneous, though queries
which haven't output anything aren't easily interruptible...
The wq_* IPC stuff will be reused in the normal read-only
WWW/IMAP search at some point, too.
Eric Wong (22):
lei query + pagination sorta working
lei q: deduplicate smsg
ds: block signals when reaping
ipc: add support for asynchronous callbacks
cmd_ipc: send FDs with buffer payload
ipc: avoid excessive evals
ipc: work queue support via SOCK_SEQPACKET
ipc: eliminate ipc_worker_stop method
ipc: wq: support dynamic worker count change
ipc: drop -ipc_parent_pid field
ipc: DESTROY and wq_workers methods
lei: rename $w to $wpager for warning message
lei: fix oneshot TTY detection by passing STD*{GLOB}
lei: query: ensure pager exit is instantaneous
ipc: start supporting sending/receiving more than 3 FDs
ipc: fix IO::FDPass use with a worker limit of 1
ipc: drop unused fields, default sighandlers for wq
lei: get rid of client {pid} field
lei: fork + FD cleanup
lei: run pager in client script
lei_xsearch: transfer 4 FDs internally, drop IO::FDPass
lei: query: restore JSON output overview
MANIFEST | 4 +
lib/PublicInbox/CmdIPC4.pm | 36 ++++
lib/PublicInbox/DS.pm | 16 +-
lib/PublicInbox/Daemon.pm | 10 +-
lib/PublicInbox/ExtSearchIdx.pm | 4 +-
lib/PublicInbox/IPC.pm | 280 ++++++++++++++++++++++++++++----
lib/PublicInbox/LEI.pm | 180 +++++++++++++-------
lib/PublicInbox/LeiDedupe.pm | 29 +++-
lib/PublicInbox/LeiExternal.pm | 33 ++--
lib/PublicInbox/LeiOverview.pm | 188 +++++++++++++++++++++
lib/PublicInbox/LeiQuery.pm | 92 +++++++++++
lib/PublicInbox/LeiStore.pm | 2 +-
lib/PublicInbox/LeiToMail.pm | 2 +
lib/PublicInbox/LeiXSearch.pm | 118 +++++++++++++-
lib/PublicInbox/Search.pm | 10 +-
lib/PublicInbox/SearchView.pm | 10 +-
lib/PublicInbox/Sigfd.pm | 12 +-
lib/PublicInbox/Spawn.pm | 85 ++++++----
lib/PublicInbox/Watch.pm | 8 +-
script/lei | 76 +++++----
script/public-inbox-watch | 4 +-
t/cmd_ipc.t | 82 ++++++++++
t/ipc.t | 115 ++++++++++++-
t/lei.t | 31 +++-
t/lei_dedupe.t | 14 ++
t/lei_xsearch.t | 5 +
t/spawn.t | 33 +---
27 files changed, 1233 insertions(+), 246 deletions(-)
create mode 100644 lib/PublicInbox/CmdIPC4.pm
create mode 100644 lib/PublicInbox/LeiOverview.pm
create mode 100644 lib/PublicInbox/LeiQuery.pm
create mode 100644 t/cmd_ipc.t
next reply other threads:[~2021-01-10 12:15 UTC|newest]
Thread overview: 23+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-01-10 12:14 Eric Wong [this message]
2021-01-10 12:14 ` [PATCH 01/22] lei query + pagination sorta working Eric Wong
2021-01-10 12:14 ` [PATCH 02/22] lei q: deduplicate smsg Eric Wong
2021-01-10 12:15 ` [PATCH 03/22] ds: block signals when reaping Eric Wong
2021-01-10 12:15 ` [PATCH 04/22] ipc: add support for asynchronous callbacks Eric Wong
2021-01-10 12:15 ` [PATCH 05/22] cmd_ipc: send FDs with buffer payload Eric Wong
2021-01-10 12:15 ` [PATCH 06/22] ipc: avoid excessive evals Eric Wong
2021-01-10 12:15 ` [PATCH 07/22] ipc: work queue support via SOCK_SEQPACKET Eric Wong
2021-01-10 12:15 ` [PATCH 08/22] ipc: eliminate ipc_worker_stop method Eric Wong
2021-01-10 12:15 ` [PATCH 09/22] ipc: wq: support dynamic worker count change Eric Wong
2021-01-10 12:15 ` [PATCH 10/22] ipc: drop -ipc_parent_pid field Eric Wong
2021-01-10 12:15 ` [PATCH 11/22] ipc: DESTROY and wq_workers methods Eric Wong
2021-01-10 12:15 ` [PATCH 12/22] lei: rename $w to $wpager for warning message Eric Wong
2021-01-10 12:15 ` [PATCH 13/22] lei: fix oneshot TTY detection by passing STD*{GLOB} Eric Wong
2021-01-10 12:15 ` [PATCH 14/22] lei: query: ensure pager exit is instantaneous Eric Wong
2021-01-10 12:15 ` [PATCH 15/22] ipc: start supporting sending/receiving more than 3 FDs Eric Wong
2021-01-10 12:15 ` [PATCH 16/22] ipc: fix IO::FDPass use with a worker limit of 1 Eric Wong
2021-01-10 12:15 ` [PATCH 17/22] ipc: drop unused fields, default sighandlers for wq Eric Wong
2021-01-10 12:15 ` [PATCH 18/22] lei: get rid of client {pid} field Eric Wong
2021-01-10 12:15 ` [PATCH 19/22] lei: fork + FD cleanup Eric Wong
2021-01-10 12:15 ` [PATCH 20/22] lei: run pager in client script Eric Wong
2021-01-10 12:15 ` [PATCH 21/22] lei_xsearch: transfer 4 FDs internally, drop IO::FDPass Eric Wong
2021-01-10 12:15 ` [PATCH 22/22] lei: query: restore JSON output overview Eric Wong
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: https://public-inbox.org/README
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20210110121519.17044-1-e@80x24.org \
--to=e@80x24.org \
--cc=meta@public-inbox.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).