RFC: lei searches managed by users in git

RFC: lei searches managed by users in git

[Konstantin Ryabitsev]

Hello:

I am curious what is the best approach to have a centrally managed set of lei
searches, for example via config files tracked in git. For example, the file
could look like this:

mricon.toml:

    [search.torvalds]
        # All mail sent by torvalds
        q = 'f:torvalds@linux-foundation.org'
    [search.floppy]
        # Any messages talking about floppies or touching floppy code
        q = 'dfhh:floppy_* OR dfn:drivers/block/floppy.c OR s:floppy OR ((nq:bug OR nq:regression) AND nq:floppy)'

I could then have a small wrapper maintaining saved searches and making the
mailboxes available via special newsgroups like:

    org.kernel.lei.mricon.torvalds
    org.kernel.lei.mricon.floppy

The goal is to make it possible for maintainers to define their own set of
saved searches and have access to them at kernel.org via imap/pop3/nntp.

It's easy to write a simple wrapper that would invoke lei-edit-search and
replace the search string when there are updates to the config files, but I'm
curious if you already have thoughts on how to best implement something like
this.

My biggest concern is someone committing an invalid query and not receiving
any more email as a result -- so having a sane way to validate the query
before sticking it into the saved search would be handy.

-K

Re: RFC: lei searches managed by users in git

[Eric Wong]

Konstantin Ryabitsev <konstantin@linuxfoundation.org> wrote:
> Hello:
> 
> I am curious what is the best approach to have a centrally managed set of lei
> searches, for example via config files tracked in git. For example, the file
> could look like this:

I don't have one nor have I thought about it until now...

I wonder if `lei edit-search --all' could be a thing...
As would adding gitconfig output to ls-search
(`lei ls-search -l -f gitconfig').

Maybe `lei edit-search --all=lei.q' could cut down
on non-interesting fields when editing all searches.

I also wouldn't be opposed to supporting non-interactive use
for automation:

	lei edit-search --set lei.q=foo OUTPUT
	lei edit-search --add lei.q=foo OUTPUT

...but maybe it's not needed (see below on `lei q'):

> mricon.toml:

Any particular reason for toml?  I don't have a toml parser
installed and I suspect many don't, either.  git-config is
widely available and installed, of course.  I've been trying
hard to avoid data format and dependency proliferation.

>     [search.torvalds]
>         # All mail sent by torvalds
>         q = 'f:torvalds@linux-foundation.org'
>     [search.floppy]
>         # Any messages talking about floppies or touching floppy code
>         q = 'dfhh:floppy_* OR dfn:drivers/block/floppy.c OR s:floppy OR ((nq:bug OR nq:regression) AND nq:floppy)'
> 
> I could then have a small wrapper maintaining saved searches and making the
> mailboxes available via special newsgroups like:
> 
>     org.kernel.lei.mricon.torvalds
>     org.kernel.lei.mricon.floppy

Sidenote: I think `query' or similar is more appropriate than
using `lei' in a public newsgroup name since it's not `local' :>

> The goal is to make it possible for maintainers to define their own set of
> saved searches and have access to them at kernel.org via imap/pop3/nntp.
> 
> It's easy to write a simple wrapper that would invoke lei-edit-search and
> replace the search string when there are updates to the config files, but I'm
> curious if you already have thoughts on how to best implement something like
> this.

Rerunning `lei q' on existing destinations is fine for updating
existing queries.  This is especially nice with `-f v2', since
dedupe is faster on v2 than any other output format.

Thus there's no need to even bother with `lei edit-search' if
you're doing this non-interactively with your own files/formats
(e.g. via web UI or something).

> My biggest concern is someone committing an invalid query and not receiving
> any more email as a result -- so having a sane way to validate the query
> before sticking it into the saved search would be handy.

Perhaps `lei q --limit=1' could be appropriate for validating queries
with the below fix.  But validating searches can ahead of time is
a TOCTOU problem.  There can be added support for per-inbox prefixes
(e.g. altid with gmane: on public-inbox.org/git), and also when mixing
old/new versions via different HTTP(S) externals.

-------8<-------
Subject: [PATCH] lei q: set exit code for invalid Xapian queries

Xapian can't parse every query, so ensure we set the
exit code for the client.
---
 lib/PublicInbox/LeiXSearch.pm | 6 ++++--
 t/lei.t                       | 5 +++++
 2 files changed, 9 insertions(+), 2 deletions(-)

diff --git a/lib/PublicInbox/LeiXSearch.pm b/lib/PublicInbox/LeiXSearch.pm
index 5965274c..7f4911b3 100644
--- a/lib/PublicInbox/LeiXSearch.pm
+++ b/lib/PublicInbox/LeiXSearch.pm
@@ -186,7 +186,8 @@ sub query_one_mset { # for --threads and l2m w/o sort
 	}
 	my $first_ids;
 	do {
-		$mset = $srch->mset($mo->{qstr}, $mo);
+		$mset = eval { $srch->mset($mo->{qstr}, $mo) };
+		return $lei->child_error(22 << 8, "E: $@") if $@; # 22 from curl
 		mset_progress($lei, $dir, $mo->{offset} + $mset->size,
 				$mset->get_matches_estimated);
 		wait_startq($lei); # wait for keyword updates
@@ -249,7 +250,8 @@ sub query_combined_mset { # non-parallel for non-"--threads" users
 	}
 	my $each_smsg = $lei->{ovv}->ovv_each_smsg_cb($lei);
 	do {
-		$mset = $self->mset($mo->{qstr}, $mo);
+		$mset = eval { $self->mset($mo->{qstr}, $mo) };
+		return $lei->child_error(22 << 8, "E: $@") if $@; # 22 from curl
 		mset_progress($lei, 'xsearch', $mo->{offset} + $mset->size,
 				$mset->get_matches_estimated);
 		wait_startq($lei); # wait for keyword updates
diff --git a/t/lei.t b/t/lei.t
index d83bde69..1199ca75 100644
--- a/t/lei.t
+++ b/t/lei.t
@@ -147,6 +147,11 @@ my $test_fail = sub {
 	lei_ok('q', "foo\n");
 	like($lei_err, qr/trailing `\\n' removed/s, "noted `\\n' removal");
 
+	lei(qw(q from:infinity..));
+	is($? >> 8, 22, 'combined query fails on invalid range op');
+	lei(qw(q -t from:infinity..));
+	is($? >> 8, 22, 'single query fails on invalid range op');
+
 	for my $lk (qw(ei inbox)) {
 		my $d = "$home/newline\n$lk";
 		my $all = $lk eq 'ei' ? 'ALL' : 'all';