Re: RFC: lei searches managed by users in git

[Eric Wong <e@80x24.org>]

Konstantin Ryabitsev <konstantin@linuxfoundation.org> wrote:
> Hello:
> 
> I am curious what is the best approach to have a centrally managed set of lei
> searches, for example via config files tracked in git. For example, the file
> could look like this:

I don't have one nor have I thought about it until now...

I wonder if `lei edit-search --all' could be a thing...
As would adding gitconfig output to ls-search
(`lei ls-search -l -f gitconfig').

Maybe `lei edit-search --all=lei.q' could cut down
on non-interesting fields when editing all searches.

I also wouldn't be opposed to supporting non-interactive use
for automation:

	lei edit-search --set lei.q=foo OUTPUT
	lei edit-search --add lei.q=foo OUTPUT

...but maybe it's not needed (see below on `lei q'):

> mricon.toml:

Any particular reason for toml?  I don't have a toml parser
installed and I suspect many don't, either.  git-config is
widely available and installed, of course.  I've been trying
hard to avoid data format and dependency proliferation.

>     [search.torvalds]
>         # All mail sent by torvalds
>         q = 'f:torvalds@linux-foundation.org'
>     [search.floppy]
>         # Any messages talking about floppies or touching floppy code
>         q = 'dfhh:floppy_* OR dfn:drivers/block/floppy.c OR s:floppy OR ((nq:bug OR nq:regression) AND nq:floppy)'
> 
> I could then have a small wrapper maintaining saved searches and making the
> mailboxes available via special newsgroups like:
> 
>     org.kernel.lei.mricon.torvalds
>     org.kernel.lei.mricon.floppy

Sidenote: I think `query' or similar is more appropriate than
using `lei' in a public newsgroup name since it's not `local' :>

> The goal is to make it possible for maintainers to define their own set of
> saved searches and have access to them at kernel.org via imap/pop3/nntp.
> 
> It's easy to write a simple wrapper that would invoke lei-edit-search and
> replace the search string when there are updates to the config files, but I'm
> curious if you already have thoughts on how to best implement something like
> this.

Rerunning `lei q' on existing destinations is fine for updating
existing queries.  This is especially nice with `-f v2', since
dedupe is faster on v2 than any other output format.

Thus there's no need to even bother with `lei edit-search' if
you're doing this non-interactively with your own files/formats
(e.g. via web UI or something).

> My biggest concern is someone committing an invalid query and not receiving
> any more email as a result -- so having a sane way to validate the query
> before sticking it into the saved search would be handy.

Perhaps `lei q --limit=1' could be appropriate for validating queries
with the below fix.  But validating searches can ahead of time is
a TOCTOU problem.  There can be added support for per-inbox prefixes
(e.g. altid with gmane: on public-inbox.org/git), and also when mixing
old/new versions via different HTTP(S) externals.

-------8<-------
Subject: [PATCH] lei q: set exit code for invalid Xapian queries

Xapian can't parse every query, so ensure we set the
exit code for the client.
---
 lib/PublicInbox/LeiXSearch.pm | 6 ++++--
 t/lei.t                       | 5 +++++
 2 files changed, 9 insertions(+), 2 deletions(-)

diff --git a/lib/PublicInbox/LeiXSearch.pm b/lib/PublicInbox/LeiXSearch.pm
index 5965274c..7f4911b3 100644
--- a/lib/PublicInbox/LeiXSearch.pm
+++ b/lib/PublicInbox/LeiXSearch.pm
@@ -186,7 +186,8 @@ sub query_one_mset { # for --threads and l2m w/o sort
 	}
 	my $first_ids;
 	do {
-		$mset = $srch->mset($mo->{qstr}, $mo);
+		$mset = eval { $srch->mset($mo->{qstr}, $mo) };
+		return $lei->child_error(22 << 8, "E: $@") if $@; # 22 from curl
 		mset_progress($lei, $dir, $mo->{offset} + $mset->size,
 				$mset->get_matches_estimated);
 		wait_startq($lei); # wait for keyword updates
@@ -249,7 +250,8 @@ sub query_combined_mset { # non-parallel for non-"--threads" users
 	}
 	my $each_smsg = $lei->{ovv}->ovv_each_smsg_cb($lei);
 	do {
-		$mset = $self->mset($mo->{qstr}, $mo);
+		$mset = eval { $self->mset($mo->{qstr}, $mo) };
+		return $lei->child_error(22 << 8, "E: $@") if $@; # 22 from curl
 		mset_progress($lei, 'xsearch', $mo->{offset} + $mset->size,
 				$mset->get_matches_estimated);
 		wait_startq($lei); # wait for keyword updates
diff --git a/t/lei.t b/t/lei.t
index d83bde69..1199ca75 100644
--- a/t/lei.t
+++ b/t/lei.t
@@ -147,6 +147,11 @@ my $test_fail = sub {
 	lei_ok('q', "foo\n");
 	like($lei_err, qr/trailing `\\n' removed/s, "noted `\\n' removal");
 
+	lei(qw(q from:infinity..));
+	is($? >> 8, 22, 'combined query fails on invalid range op');
+	lei(qw(q -t from:infinity..));
+	is($? >> 8, 22, 'single query fails on invalid range op');
+
 	for my $lk (qw(ei inbox)) {
 		my $d = "$home/newline\n$lk";
 		my $all = $lk eq 'ei' ? 'ALL' : 'all';