[Need Help] lei add quotes at the search

[Need Help] lei add quotes at the search

[Hangbin Liu]

Hi,

I used to use a search like

lei q -I https://lore.kernel.org/all/ -o ~/Mail/liuhangbin --threads --dedupe=mid '((tc:liuhangbin AND rt:6.month.ago..) NOT (tc:stable@vger.kernel.org OR f:sfr@canb.auug.org.au)'

It works on fc35. But after I update to fc36 with lei-1.9.0-1.fc36. It start to
add quotes in the search link and make the search never works. e.g.

$ lei q -I https://lore.kernel.org/all/ -o ~/Mail/liuhangbin --threads --dedupe=mid '((tc:liuhangbin AND rt:6.month.ago..) NOT (tc:stable@vger.kernel.org OR f:sfr@canb.auug.org.au)'
# /home/Liu/.local/share/lei/store 0/0
# /usr/bin/curl -Sf -s -d '' https://lore.kernel.org/all/?x=m&t=1&q=((tc%3A%22liuhangbin+AND+rt%3A6.month.ago..)+NOT+(tc%3Astable%40vger.kernel.org+OR+f%3Asfr%40canb.auug.org.au)%22
# 0 written to /home/Liu/Mail/liuhangbin/ (0 matches)

Do you think if this is a bug, or I should update my search.

Thanks
Hangbin

Re: [Need Help] lei add quotes at the search

[Eric Wong]

Hangbin Liu <liuhangbin@gmail.com> wrote:
> Hi,
> 
> I used to use a search like
> 
> lei q -I https://lore.kernel.org/all/ -o ~/Mail/liuhangbin --threads --dedupe=mid '((tc:liuhangbin AND rt:6.month.ago..) NOT (tc:stable@vger.kernel.org OR f:sfr@canb.auug.org.au)'
> 
> It works on fc35. But after I update to fc36 with lei-1.9.0-1.fc36. It start to
> add quotes in the search link and make the search never works. e.g.

Are you able to show the curl CLI from fc35?
Which public-inbox/lei version was it?

I'm actually curious fc35 worked at all, since the quoting would've
been broken, I think...

> $ lei q -I https://lore.kernel.org/all/ -o ~/Mail/liuhangbin --threads --dedupe=mid '((tc:liuhangbin AND rt:6.month.ago..) NOT (tc:stable@vger.kernel.org OR f:sfr@canb.auug.org.au)'
> # /home/Liu/.local/share/lei/store 0/0
> # /usr/bin/curl -Sf -s -d '' https://lore.kernel.org/all/?x=m&t=1&q=((tc%3A%22liuhangbin+AND+rt%3A6.month.ago..)+NOT+(tc%3Astable%40vger.kernel.org+OR+f%3Asfr%40canb.auug.org.au)%22
> # 0 written to /home/Liu/Mail/liuhangbin/ (0 matches)
> 
> Do you think if this is a bug, or I should update my search.

The %22 in fc36 is because your entire query is treated as one
element in argv and matches expected behavior.

Since '(' and ')' in the shell CLI is special, I suggest either:

a) using --stdin to enter queries containing '(' and ')'

b) quoting (or escaping) only the '(' and ')':

    '('tc:liuhangbin AND rt:6.month.ago..')' NOT ...

                or

    \(tc:liuhangbin AND rt:6.month.ago..\) NOT ...

  Which makes your argv something like:

    [ "(tc:liuhangbin", "AND", "rt:6.month.ago..)", "NOT", ... ]

Re: [Need Help] lei add quotes at the search

[Hangbin Liu]

Hi Eric,

Thanks for the help.

On Sun, Oct 30, 2022 at 05:13:33AM +0000, Eric Wong wrote:
> Hangbin Liu <liuhangbin@gmail.com> wrote:
> > Hi,
> > 
> > I used to use a search like
> > 
> > lei q -I https://lore.kernel.org/all/ -o ~/Mail/liuhangbin --threads --dedupe=mid '((tc:liuhangbin AND rt:6.month.ago..) NOT (tc:stable@vger.kernel.org OR f:sfr@canb.auug.org.au)'
> > 
> > It works on fc35. But after I update to fc36 with lei-1.9.0-1.fc36. It start to
> > add quotes in the search link and make the search never works. e.g.
> 
> Are you able to show the curl CLI from fc35?
> Which public-inbox/lei version was it?
> 
> I'm actually curious fc35 worked at all, since the quoting would've
> been broken, I think...

Sorry, I don't have the fc35 environment now.

> 
> > $ lei q -I https://lore.kernel.org/all/ -o ~/Mail/liuhangbin --threads --dedupe=mid '((tc:liuhangbin AND rt:6.month.ago..) NOT (tc:stable@vger.kernel.org OR f:sfr@canb.auug.org.au)'
> > # /home/Liu/.local/share/lei/store 0/0
> > # /usr/bin/curl -Sf -s -d '' https://lore.kernel.org/all/?x=m&t=1&q=((tc%3A%22liuhangbin+AND+rt%3A6.month.ago..)+NOT+(tc%3Astable%40vger.kernel.org+OR+f%3Asfr%40canb.auug.org.au)%22
> > # 0 written to /home/Liu/Mail/liuhangbin/ (0 matches)
> > 
> > Do you think if this is a bug, or I should update my search.
> 
> The %22 in fc36 is because your entire query is treated as one
> element in argv and matches expected behavior.
> 
> Since '(' and ')' in the shell CLI is special, I suggest either:

I'm curious about why the quote(%22) is added after "tc", not after "("

> 
> a) using --stdin to enter queries containing '(' and ')'
> 
> b) quoting (or escaping) only the '(' and ')':
> 
>     '('tc:liuhangbin AND rt:6.month.ago..')' NOT ...
> 
>                 or
> 
>     \(tc:liuhangbin AND rt:6.month.ago..\) NOT ...

with this way, the cmd line works. And in config file, it would looks like

[lei]
        q = ((tc:liuhangbin
        q = AND
        q = rt:6.month.ago..)
        q = NOT
        q = (tc:stable@vger.kernel.org
        q = OR
        q = f:sfr@canb.auug.org.au)

But if I have a long search line. This will breaks too much and hard to edit.
e.g. My real previous search is like

[lei]
        q = (tc:liuhangbin OR \
             (dfn:drivers/net/wireguard/ AND rt:6.month.ago..) OR \
             (dfn:tools/testing/selftests/net/ AND rt:1.month.ago..) OR \
             (dfn:drivers/net/team/ AND rt:6.month.ago..) OR \
             (dfn:net/ipv4/igmp.c AND rt:6.month.ago..) OR \
             (dfn:net/ipv6/mcast.c AND rt:6.month.ago..)) \
	     NOT (tc:stable@vger.kernel.org OR f:sfr@canb.auug.org.au)

If I add "\" on each "(", this will break to a very long config search.
I tried to adjust it to

[lei]
        q = (tc:liuhangbin OR \
             (dfn:drivers/net/wireguard/ AND rt:6.month.ago..) OR \
             (dfn:tools/testing/selftests/net/ AND rt:1.month.ago..) OR \
             (dfn:drivers/net/team/ AND rt:6.month.ago..) OR \
             (dfn:net/ipv4/igmp.c AND rt:6.month.ago..) OR \
             (dfn:net/ipv6/mcast.c AND rt:6.month.ago..))
        q = NOT
        q = (tc:stable@vger.kernel.org
        q = OR
        q = f:sfr@canb.auug.org.au)

And now it works...

$ lei up /home/Liu/Mail/gmail/Linux_Kernel
# https://lore.kernel.org/all/ limiting to 2022-09-30 17:00 +0800 and newer
60927 lei_xsearch 0 wq_worker: query_one_mset: Exception: Unknown range operation at /usr/share/perl5/vendor_perl/PublicInbox/IPC.pm line 254.
# /usr/bin/curl -Sf -s -d '' https://lore.kernel.org/all/?x=m&t=1&q=((tc%3Aliuhangbin+OR+(dfn%3Adrivers%2Fnet%2Fwireguard%2F+AND+rt%3A6.month.ago..)+OR+(dfn%3Atools%2Ftesting%2Fselftests%2Fnet%2F+AND+rt%3A1.month.ago..)+OR+(dfn%3Adrivers%2Fnet%2Fteam%2F+AND+rt%3A6.month.ago..)+OR+(dfn%3Anet%2Fipv4%2Figmp.c+AND+rt%3A6.month.ago..)+OR+(dfn%3Anet%2Fipv6%2Fmcast.c+AND+rt%3A1651301673..))+NOT+(tc%3Astable%40vger.kernel.org+OR+f%3Asfr%40canb.auug.org.au))+AND+dt%3A20220930090001..
# https://lore.kernel.org/all/ 43/?

So I want to know when/why *lei* add the quotes.

Thanks
Hangbin

Re: [Need Help] lei add quotes at the search

[Eric Wong]

Hangbin Liu <liuhangbin@gmail.com> wrote:
> Hi Eric,
> 
> Thanks for the help.
> 
> On Sun, Oct 30, 2022 at 05:13:33AM +0000, Eric Wong wrote:
> > Hangbin Liu <liuhangbin@gmail.com> wrote:
> > > Hi,
> > > 
> > > I used to use a search like
> > > 
> > > lei q -I https://lore.kernel.org/all/ -o ~/Mail/liuhangbin --threads --dedupe=mid '((tc:liuhangbin AND rt:6.month.ago..) NOT (tc:stable@vger.kernel.org OR f:sfr@canb.auug.org.au)'
> > > 
> > > It works on fc35. But after I update to fc36 with lei-1.9.0-1.fc36. It start to
> > > add quotes in the search link and make the search never works. e.g.
> > 
> > Are you able to show the curl CLI from fc35?
> > Which public-inbox/lei version was it?
> > 
> > I'm actually curious fc35 worked at all, since the quoting would've
> > been broken, I think...
> 
> Sorry, I don't have the fc35 environment now.

No worries, I dont think fc35 is really a culprit.  Were you
running a pre-release version of public-inbox or lei before?

> > > $ lei q -I https://lore.kernel.org/all/ -o ~/Mail/liuhangbin --threads --dedupe=mid '((tc:liuhangbin AND rt:6.month.ago..) NOT (tc:stable@vger.kernel.org OR f:sfr@canb.auug.org.au)'
> > > # /home/Liu/.local/share/lei/store 0/0
> > > # /usr/bin/curl -Sf -s -d '' https://lore.kernel.org/all/?x=m&t=1&q=((tc%3A%22liuhangbin+AND+rt%3A6.month.ago..)+NOT+(tc%3Astable%40vger.kernel.org+OR+f%3Asfr%40canb.auug.org.au)%22
> > > # 0 written to /home/Liu/Mail/liuhangbin/ (0 matches)
> > > 
> > > Do you think if this is a bug, or I should update my search.
> > 
> > The %22 in fc36 is because your entire query is treated as one
> > element in argv and matches expected behavior.
> > 
> > Since '(' and ')' in the shell CLI is special, I suggest either:
> 
> I'm curious about why the quote(%22) is added after "tc", not after "("

It's because Xapian can only handle a phrase after the `tc:' prefix.
thus:	tc:"foo bar"	actually parses `tc:' as a prefix for To/Cc;
while:	"tc:foo bar"	looks for the phrase "tc:foo bar" anywhere
in the message, and won't limit to To/Cc headers.

This happens in the query_argv_to_string sub:

https://public-inbox.org/meta/2feb3e13b49d222bc7bd28430a9cf159692a933f/s/?b=lib/PublicInbox/Search.pm#n358

From the CLI:	lei q "tc:foo bar"	is indistinguishable
from	lei q tc:"foo bar"	, so it gets treated as the latter.

> > a) using --stdin to enter queries containing '(' and ')'
> > 
> > b) quoting (or escaping) only the '(' and ')':
> > 
> >     '('tc:liuhangbin AND rt:6.month.ago..')' NOT ...
> > 
> >                 or
> > 
> >     \(tc:liuhangbin AND rt:6.month.ago..\) NOT ...
> 
> with this way, the cmd line works. And in config file, it would looks like
> 
> [lei]
>         q = ((tc:liuhangbin
>         q = AND
>         q = rt:6.month.ago..)
>         q = NOT
>         q = (tc:stable@vger.kernel.org
>         q = OR
>         q = f:sfr@canb.auug.org.au)
> 
> But if I have a long search line. This will breaks too much and hard to edit.
> e.g. My real previous search is like
> 
> [lei]
>         q = (tc:liuhangbin OR \
>              (dfn:drivers/net/wireguard/ AND rt:6.month.ago..) OR \
>              (dfn:tools/testing/selftests/net/ AND rt:1.month.ago..) OR \
>              (dfn:drivers/net/team/ AND rt:6.month.ago..) OR \
>              (dfn:net/ipv4/igmp.c AND rt:6.month.ago..) OR \
>              (dfn:net/ipv6/mcast.c AND rt:6.month.ago..)) \
> 	     NOT (tc:stable@vger.kernel.org OR f:sfr@canb.auug.org.au)
> 
> If I add "\" on each "(", this will break to a very long config search.
> I tried to adjust it to

I think that can work if lei.internal.rawstr is set in the
config to indicate stdin was used (It's auto-set by --stdin).
I guess it also works if it's the only lei.q config entry
and the lei.q entry contains "\n"

cf. https://public-inbox.org/meta/20211110102837.41721-1-e@80x24.org/

> [lei]
>         q = (tc:liuhangbin OR \
>              (dfn:drivers/net/wireguard/ AND rt:6.month.ago..) OR \
>              (dfn:tools/testing/selftests/net/ AND rt:1.month.ago..) OR \
>              (dfn:drivers/net/team/ AND rt:6.month.ago..) OR \
>              (dfn:net/ipv4/igmp.c AND rt:6.month.ago..) OR \
>              (dfn:net/ipv6/mcast.c AND rt:6.month.ago..))
>         q = NOT
>         q = (tc:stable@vger.kernel.org
>         q = OR
>         q = f:sfr@canb.auug.org.au)
> 
> And now it works...

Sorta... at least for remotes it does:

> $ lei up /home/Liu/Mail/gmail/Linux_Kernel
> # https://lore.kernel.org/all/ limiting to 2022-09-30 17:00 +0800 and newer
> 60927 lei_xsearch 0 wq_worker: query_one_mset: Exception: Unknown range operation at /usr/share/perl5/vendor_perl/PublicInbox/IPC.pm line 254.

Note that Exception means it's not handling the first part of
the query when hitting the local Xapian DB.  It's not doing the
approxidate ($X.month.ago) substitution for the local Xapian DB,
thus you got the "Unknown range operation".

> # /usr/bin/curl -Sf -s -d '' https://lore.kernel.org/all/?x=m&t=1&q=((tc%3Aliuhangbin+OR+(dfn%3Adrivers%2Fnet%2Fwireguard%2F+AND+rt%3A6.month.ago..)+OR+(dfn%3Atools%2Ftesting%2Fselftests%2Fnet%2F+AND+rt%3A1.month.ago..)+OR+(dfn%3Adrivers%2Fnet%2Fteam%2F+AND+rt%3A6.month.ago..)+OR+(dfn%3Anet%2Fipv4%2Figmp.c+AND+rt%3A6.month.ago..)+OR+(dfn%3Anet%2Fipv6%2Fmcast.c+AND+rt%3A1651301673..))+NOT+(tc%3Astable%40vger.kernel.org+OR+f%3Asfr%40canb.auug.org.au))+AND+dt%3A20220930090001..
> # https://lore.kernel.org/all/ 43/?

Of course, the lack of approxidate parsing there inside lei is
fine, since the lore.kernel.org instance will do it remotely...

> So I want to know when/why *lei* add the quotes.

lei adds quotes since it can't distinguish if the shell user
used single or double quotes.  Xapian uses double quotes for
phrase search, and I wanted:	lei q "this is a phrase"
to work naturally, which means:	lei q 'this is a phrase'
(with single quotes) works the same way as with double quotes
because the difference is handled by the shell and lei never
sees it.

Re: [Need Help] lei add quotes at the search

[Hangbin Liu]

On Sun, Oct 30, 2022 at 11:06:31PM +0000, Eric Wong wrote:
> Hangbin Liu <liuhangbin@gmail.com> wrote:
> > Sorry, I don't have the fc35 environment now.
> 
> No worries, I dont think fc35 is really a culprit.  Were you
> running a pre-release version of public-inbox or lei before?

Sorry, I forgot. Maybe I installed via `dnf copr enable icon/b4`
because I start using lei after reading blog
https://people.kernel.org/monsieuricon/lore-lei-part-1-getting-started

> > I'm curious about why the quote(%22) is added after "tc", not after "("
> 
> It's because Xapian can only handle a phrase after the `tc:' prefix.
> thus:	tc:"foo bar"	actually parses `tc:' as a prefix for To/Cc;
> while:	"tc:foo bar"	looks for the phrase "tc:foo bar" anywhere
> in the message, and won't limit to To/Cc headers.
> 
> This happens in the query_argv_to_string sub:
> 
> https://public-inbox.org/meta/2feb3e13b49d222bc7bd28430a9cf159692a933f/s/?b=lib/PublicInbox/Search.pm#n358
> 
> From the CLI:	lei q "tc:foo bar"	is indistinguishable
> from	lei q tc:"foo bar"	, so it gets treated as the latter.

Thanks for the explanation.

> > But if I have a long search line. This will breaks too much and hard to edit.
> > e.g. My real previous search is like
> > 
> > [lei]
> >         q = (tc:liuhangbin OR \
> >              (dfn:drivers/net/wireguard/ AND rt:6.month.ago..) OR \
> >              (dfn:tools/testing/selftests/net/ AND rt:1.month.ago..) OR \
> >              (dfn:drivers/net/team/ AND rt:6.month.ago..) OR \
> >              (dfn:net/ipv4/igmp.c AND rt:6.month.ago..) OR \
> >              (dfn:net/ipv6/mcast.c AND rt:6.month.ago..)) \
> > 	     NOT (tc:stable@vger.kernel.org OR f:sfr@canb.auug.org.au)
> > 
> > If I add "\" on each "(", this will break to a very long config search.
> > I tried to adjust it to
> 
> I think that can work if lei.internal.rawstr is set in the
> config to indicate stdin was used (It's auto-set by --stdin).
> I guess it also works if it's the only lei.q config entry
> and the lei.q entry contains "\n"
> 
> cf. https://public-inbox.org/meta/20211110102837.41721-1-e@80x24.org/
> 
> > [lei]
> >         q = (tc:liuhangbin OR \
> >              (dfn:drivers/net/wireguard/ AND rt:6.month.ago..) OR \
> >              (dfn:tools/testing/selftests/net/ AND rt:1.month.ago..) OR \
> >              (dfn:drivers/net/team/ AND rt:6.month.ago..) OR \
> >              (dfn:net/ipv4/igmp.c AND rt:6.month.ago..) OR \
> >              (dfn:net/ipv6/mcast.c AND rt:6.month.ago..))
> >         q = NOT
> >         q = (tc:stable@vger.kernel.org
> >         q = OR
> >         q = f:sfr@canb.auug.org.au)
> > 
> > And now it works...
> 
> Sorta... at least for remotes it does:
> 
> > $ lei up /home/Liu/Mail/gmail/Linux_Kernel
> > # https://lore.kernel.org/all/ limiting to 2022-09-30 17:00 +0800 and newer
> > 60927 lei_xsearch 0 wq_worker: query_one_mset: Exception: Unknown range operation at /usr/share/perl5/vendor_perl/PublicInbox/IPC.pm line 254.
> 
> Note that Exception means it's not handling the first part of
> the query when hitting the local Xapian DB.  It's not doing the
> approxidate ($X.month.ago) substitution for the local Xapian DB,
> thus you got the "Unknown range operation".
> 
> > # /usr/bin/curl -Sf -s -d '' https://lore.kernel.org/all/?x=m&t=1&q=((tc%3Aliuhangbin+OR+(dfn%3Adrivers%2Fnet%2Fwireguard%2F+AND+rt%3A6.month.ago..)+OR+(dfn%3Atools%2Ftesting%2Fselftests%2Fnet%2F+AND+rt%3A1.month.ago..)+OR+(dfn%3Adrivers%2Fnet%2Fteam%2F+AND+rt%3A6.month.ago..)+OR+(dfn%3Anet%2Fipv4%2Figmp.c+AND+rt%3A6.month.ago..)+OR+(dfn%3Anet%2Fipv6%2Fmcast.c+AND+rt%3A1651301673..))+NOT+(tc%3Astable%40vger.kernel.org+OR+f%3Asfr%40canb.auug.org.au))+AND+dt%3A20220930090001..
> > # https://lore.kernel.org/all/ 43/?
> 
> Of course, the lack of approxidate parsing there inside lei is
> fine, since the lore.kernel.org instance will do it remotely...
> 
> > So I want to know when/why *lei* add the quotes.
> 
> lei adds quotes since it can't distinguish if the shell user
> used single or double quotes.  Xapian uses double quotes for
> phrase search, and I wanted:	lei q "this is a phrase"
> to work naturally, which means:	lei q 'this is a phrase'
> (with single quotes) works the same way as with double quotes
> because the difference is handled by the shell and lei never
> sees it.

Thanks for the help.

Hangbin

Re: [Need Help] lei add quotes at the search

[Hangbin Liu]

On Sun, Oct 30, 2022 at 11:06:31PM +0000, Eric Wong wrote:
> > If I add "\" on each "(", this will break to a very long config search.
> > I tried to adjust it to
> 
> I think that can work if lei.internal.rawstr is set in the
> config to indicate stdin was used (It's auto-set by --stdin).

OH, BTW, I will get error
fatal: bad config line 4 in file [..snip..]/lei.saved-search
if adding '\' in the config file.

> I guess it also works if it's the only lei.q config entry
> and the lei.q entry contains "\n"

But with "\n" in the config file. My previous config will works fine.
> 
> cf. https://public-inbox.org/meta/20211110102837.41721-1-e@80x24.org/

Thanks
Hangbin