Re: [PATCH] lei q: do not collapse threads with `-tt'

[Maxim Mikityanskiy <maxtram95@gmail.com>]

On Tue, Feb 14, 2023 at 02:42:32AM +0000, Eric Wong wrote:
> Maxim Mikityanskiy <maxtram95@gmail.com> wrote:
> > lei q --no-save -a -o /tmp/lei-test -I 'https://lore.kernel.org/all' \
> >     -tt 'a:syzbot AND rt:2023-01-01..2023-01-07'
> 
> At first, I thought -a (--augment) was causing it...
> 
> Sidenote: you also don't need to quote the query (I forget the exact
> rules, but I tried to keep quotes easier for phrase searches).
> 
> > It looks as if the match works correctly, but the -tt option fails to
> > mark most of the matched emails as important, except a few that actually
> > got marked (I couldn't find a pattern here). It's also not consistent,
> > for example, after I removed /tmp/lei-test and restarted the lei q
> > command, I got many more important emails, almost in each thread, but
> > there were still threads without flagged emails.
> 
> Yes, now it seems it's the collapsing optimization.
> 
> > I'm checking the flags with mutt.
> > 
> > Does anyone know what could be the reason for such behavior?
> 
> I think the following patch fixes it.

Sorry for taking too long, I finally found a minute to test it, and
unfortunately I didn't see a difference. I queried for:

a:syzbot AND rt:2023-02-01..2023-02-07

and I still saw I lot of threads without a single flag.

I double-checked that the patch was actually applied, killed lei-daemon,
and removed the mailbox directory, but it didn't help.

> (I accidentally sent you a private copy with invalid blobs since
> I had other unpublished changes)
> 
> -----8<-------
> Subject: [PATCH] lei q: do not collapse threads with `-tt'
> 
> While having Xapian collapse threads is an easy way to reduce
> the amount of deduplication work we need to do when writing
> out threads; we can't rely on it when using `lei q -tt` since
> that needs to flag all hits.
> 
> Reported-by: Maxim Mikityanskiy <maxtram95@gmail.com>
> Link: https://public-inbox.org/git/Y+pgBmj0jxR+cVkD@mail.gmail.com/
> ---
>  lib/PublicInbox/Search.pm | 5 +++--
>  1 file changed, 3 insertions(+), 2 deletions(-)
> 
> diff --git a/lib/PublicInbox/Search.pm b/lib/PublicInbox/Search.pm
> index 2feb3e13..273cc57c 100644
> --- a/lib/PublicInbox/Search.pm
> +++ b/lib/PublicInbox/Search.pm
> @@ -460,8 +460,9 @@ sub _enquire_once { # retry_reopen callback
>  		$enquire->set_sort_by_relevance_then_value(TS, !$opts->{asc});
>  	}
>  
> -	# `mairix -t / --threads' or JMAP collapseThreads
> -	if ($opts->{threads} && has_threadid($self)) {
> +	# `lei q -t / --threads' or JMAP collapseThreads; but don't collapse
> +	# on `-tt' ({threads} > 1) which sets the Flagged|Important keyword
> +	if (($opts->{threads} // 0) == 1 && has_threadid($self)) {
>  		$enquire->set_collapse_key(THREADID);
>  	}
>  	$enquire->get_mset($opts->{offset} || 0, $opts->{limit} || 50);