Re: [Need Help] lei add quotes at the search

[Eric Wong <e@80x24.org>]

Hangbin Liu <liuhangbin@gmail.com> wrote:
> Hi Eric,
> 
> Thanks for the help.
> 
> On Sun, Oct 30, 2022 at 05:13:33AM +0000, Eric Wong wrote:
> > Hangbin Liu <liuhangbin@gmail.com> wrote:
> > > Hi,
> > > 
> > > I used to use a search like
> > > 
> > > lei q -I https://lore.kernel.org/all/ -o ~/Mail/liuhangbin --threads --dedupe=mid '((tc:liuhangbin AND rt:6.month.ago..) NOT (tc:stable@vger.kernel.org OR f:sfr@canb.auug.org.au)'
> > > 
> > > It works on fc35. But after I update to fc36 with lei-1.9.0-1.fc36. It start to
> > > add quotes in the search link and make the search never works. e.g.
> > 
> > Are you able to show the curl CLI from fc35?
> > Which public-inbox/lei version was it?
> > 
> > I'm actually curious fc35 worked at all, since the quoting would've
> > been broken, I think...
> 
> Sorry, I don't have the fc35 environment now.

No worries, I dont think fc35 is really a culprit.  Were you
running a pre-release version of public-inbox or lei before?

> > > $ lei q -I https://lore.kernel.org/all/ -o ~/Mail/liuhangbin --threads --dedupe=mid '((tc:liuhangbin AND rt:6.month.ago..) NOT (tc:stable@vger.kernel.org OR f:sfr@canb.auug.org.au)'
> > > # /home/Liu/.local/share/lei/store 0/0
> > > # /usr/bin/curl -Sf -s -d '' https://lore.kernel.org/all/?x=m&t=1&q=((tc%3A%22liuhangbin+AND+rt%3A6.month.ago..)+NOT+(tc%3Astable%40vger.kernel.org+OR+f%3Asfr%40canb.auug.org.au)%22
> > > # 0 written to /home/Liu/Mail/liuhangbin/ (0 matches)
> > > 
> > > Do you think if this is a bug, or I should update my search.
> > 
> > The %22 in fc36 is because your entire query is treated as one
> > element in argv and matches expected behavior.
> > 
> > Since '(' and ')' in the shell CLI is special, I suggest either:
> 
> I'm curious about why the quote(%22) is added after "tc", not after "("

It's because Xapian can only handle a phrase after the `tc:' prefix.
thus:	tc:"foo bar"	actually parses `tc:' as a prefix for To/Cc;
while:	"tc:foo bar"	looks for the phrase "tc:foo bar" anywhere
in the message, and won't limit to To/Cc headers.

This happens in the query_argv_to_string sub:

https://public-inbox.org/meta/2feb3e13b49d222bc7bd28430a9cf159692a933f/s/?b=lib/PublicInbox/Search.pm#n358

From the CLI:	lei q "tc:foo bar"	is indistinguishable
from	lei q tc:"foo bar"	, so it gets treated as the latter.

> > a) using --stdin to enter queries containing '(' and ')'
> > 
> > b) quoting (or escaping) only the '(' and ')':
> > 
> >     '('tc:liuhangbin AND rt:6.month.ago..')' NOT ...
> > 
> >                 or
> > 
> >     \(tc:liuhangbin AND rt:6.month.ago..\) NOT ...
> 
> with this way, the cmd line works. And in config file, it would looks like
> 
> [lei]
>         q = ((tc:liuhangbin
>         q = AND
>         q = rt:6.month.ago..)
>         q = NOT
>         q = (tc:stable@vger.kernel.org
>         q = OR
>         q = f:sfr@canb.auug.org.au)
> 
> But if I have a long search line. This will breaks too much and hard to edit.
> e.g. My real previous search is like
> 
> [lei]
>         q = (tc:liuhangbin OR \
>              (dfn:drivers/net/wireguard/ AND rt:6.month.ago..) OR \
>              (dfn:tools/testing/selftests/net/ AND rt:1.month.ago..) OR \
>              (dfn:drivers/net/team/ AND rt:6.month.ago..) OR \
>              (dfn:net/ipv4/igmp.c AND rt:6.month.ago..) OR \
>              (dfn:net/ipv6/mcast.c AND rt:6.month.ago..)) \
> 	     NOT (tc:stable@vger.kernel.org OR f:sfr@canb.auug.org.au)
> 
> If I add "\" on each "(", this will break to a very long config search.
> I tried to adjust it to

I think that can work if lei.internal.rawstr is set in the
config to indicate stdin was used (It's auto-set by --stdin).
I guess it also works if it's the only lei.q config entry
and the lei.q entry contains "\n"

cf. https://public-inbox.org/meta/20211110102837.41721-1-e@80x24.org/

> [lei]
>         q = (tc:liuhangbin OR \
>              (dfn:drivers/net/wireguard/ AND rt:6.month.ago..) OR \
>              (dfn:tools/testing/selftests/net/ AND rt:1.month.ago..) OR \
>              (dfn:drivers/net/team/ AND rt:6.month.ago..) OR \
>              (dfn:net/ipv4/igmp.c AND rt:6.month.ago..) OR \
>              (dfn:net/ipv6/mcast.c AND rt:6.month.ago..))
>         q = NOT
>         q = (tc:stable@vger.kernel.org
>         q = OR
>         q = f:sfr@canb.auug.org.au)
> 
> And now it works...

Sorta... at least for remotes it does:

> $ lei up /home/Liu/Mail/gmail/Linux_Kernel
> # https://lore.kernel.org/all/ limiting to 2022-09-30 17:00 +0800 and newer
> 60927 lei_xsearch 0 wq_worker: query_one_mset: Exception: Unknown range operation at /usr/share/perl5/vendor_perl/PublicInbox/IPC.pm line 254.

Note that Exception means it's not handling the first part of
the query when hitting the local Xapian DB.  It's not doing the
approxidate ($X.month.ago) substitution for the local Xapian DB,
thus you got the "Unknown range operation".

> # /usr/bin/curl -Sf -s -d '' https://lore.kernel.org/all/?x=m&t=1&q=((tc%3Aliuhangbin+OR+(dfn%3Adrivers%2Fnet%2Fwireguard%2F+AND+rt%3A6.month.ago..)+OR+(dfn%3Atools%2Ftesting%2Fselftests%2Fnet%2F+AND+rt%3A1.month.ago..)+OR+(dfn%3Adrivers%2Fnet%2Fteam%2F+AND+rt%3A6.month.ago..)+OR+(dfn%3Anet%2Fipv4%2Figmp.c+AND+rt%3A6.month.ago..)+OR+(dfn%3Anet%2Fipv6%2Fmcast.c+AND+rt%3A1651301673..))+NOT+(tc%3Astable%40vger.kernel.org+OR+f%3Asfr%40canb.auug.org.au))+AND+dt%3A20220930090001..
> # https://lore.kernel.org/all/ 43/?

Of course, the lack of approxidate parsing there inside lei is
fine, since the lore.kernel.org instance will do it remotely...

> So I want to know when/why *lei* add the quotes.

lei adds quotes since it can't distinguish if the shell user
used single or double quotes.  Xapian uses double quotes for
phrase search, and I wanted:	lei q "this is a phrase"
to work naturally, which means:	lei q 'this is a phrase'
(with single quotes) works the same way as with double quotes
because the difference is handled by the shell and lei never
sees it.