[Bug] lei: extra quotes inserted into query with AND/OR

[Bug] lei: extra quotes inserted into query with AND/OR

[Henrik Grimler]

Hi,

I recently found out about lei and installed it through archlinux's
package manager and am trying out queries. When using AND/OR extra
quotes are inserted in the curl command which messes it up, for
example:

$ lei q -I https://lore.kernel.org/all/ -o ~/mail/foo 'dfn:COPYING OR dfn:Makefile'
# /home/grimler/.local/share/lei/store 0/0
# /usr/bin/curl -Sf -s -d '' https://lore.kernel.org/all/?x=m&q=dfn%3A%22COPYING+OR+dfn%3AMakefile%22
# 0 written to /home/grimler/mail/foo/ (0 matches)

where it can be seen that it tries to search for 'dfn:"COPYING OR
dfn:Makefile"', and no hits are returned since there is no file named
"COPYING OR dfn:Makefile".

lei and perl-publicinbox are reported to be version 1.9.0-2. I tried
out lei from git repo (commit 270715407b0e ("doc: update
README.unsubscribe")) and had the same issue with that version as
well.

Is this a known issue, or am I doing something wrong?

Best regards,
Henrik Grimler

Re: [Bug] lei: extra quotes inserted into query with AND/OR

[Eric Wong]

Henrik Grimler <henrik@grimler.se> wrote:
> Hi,
> 
> I recently found out about lei and installed it through archlinux's
> package manager and am trying out queries. When using AND/OR extra
> quotes are inserted in the curl command which messes it up, for
> example:
> 
> $ lei q -I https://lore.kernel.org/all/ -o ~/mail/foo 'dfn:COPYING OR dfn:Makefile'
> # /home/grimler/.local/share/lei/store 0/0
> # /usr/bin/curl -Sf -s -d '' https://lore.kernel.org/all/?x=m&q=dfn%3A%22COPYING+OR+dfn%3AMakefile%22
> # 0 written to /home/grimler/mail/foo/ (0 matches)
> 
> where it can be seen that it tries to search for 'dfn:"COPYING OR
> dfn:Makefile"', and no hits are returned since there is no file named
> "COPYING OR dfn:Makefile".

Don't use quotes unless you want a phrase search.

Basically, I wanted the CLI and WWW search to feel the same.

> Is this a known issue, or am I doing something wrong?

I think you're the second user to add unnecessary quotes;
is it learned behavior from another search tool?

In my experience, generic web search engines don't use quotes
outside of phrase search, either...

My primary mail experience is from using mairix, so lei borrows
heavily from it.  But IIRC, mairix doesn't support phrase search.

Anyways, thanks for the note and any future comments you provide :>

Re: [Bug] lei: extra quotes inserted into query with AND/OR

[Henrik Grimler]

Hi Eric,

On Sun, Nov 12, 2023 at 12:10:50AM +0000, Eric Wong wrote:
> Henrik Grimler <henrik@grimler.se> wrote:
> > Hi,
> > 
> > I recently found out about lei and installed it through archlinux's
> > package manager and am trying out queries. When using AND/OR extra
> > quotes are inserted in the curl command which messes it up, for
> > example:
> > 
> > $ lei q -I https://lore.kernel.org/all/ -o ~/mail/foo 'dfn:COPYING OR dfn:Makefile'
> > # /home/grimler/.local/share/lei/store 0/0
> > # /usr/bin/curl -Sf -s -d '' https://lore.kernel.org/all/?x=m&q=dfn%3A%22COPYING+OR+dfn%3AMakefile%22
> > # 0 written to /home/grimler/mail/foo/ (0 matches)
> > 
> > where it can be seen that it tries to search for 'dfn:"COPYING OR
> > dfn:Makefile"', and no hits are returned since there is no file named
> > "COPYING OR dfn:Makefile".
> 
> Don't use quotes unless you want a phrase search.

The quotes are added by lei (or some dependency) when query contains
space. Happens also if I search for a single file:
  lei q -I https://lore.kernel.org/all/ -o ~/mail/foo ' dfn:COPYING'
which results in this curl cmd:
  /usr/bin/curl -Sf -s -d '' https://lore.kernel.org/all/?x=m&q=+dfn%3A%22COPYING%22
where %22 then is "

Without spaces in the query all is well:
  lei q -I https://lore.kernel.org/all/ -o ~/mail/foo 'dfn:COPYING'
which gives expected results
  /usr/bin/curl -Sf -s -d '' https://lore.kernel.org/all/?x=m&q=dfn%3ACOPYING

So, maybe there is an issue in some perl dependency on archlinux, any
suggestion where I should start digging?

Best regards,
Henrik Grimler

> Basically, I wanted the CLI and WWW search to feel the same.
> 
> > Is this a known issue, or am I doing something wrong?
> 
> I think you're the second user to add unnecessary quotes;
> is it learned behavior from another search tool?
> 
> In my experience, generic web search engines don't use quotes
> outside of phrase search, either...
> 
> My primary mail experience is from using mairix, so lei borrows
> heavily from it.  But IIRC, mairix doesn't support phrase search.
> 
> Anyways, thanks for the note and any future comments you provide :>

Re: [Bug] lei: extra quotes inserted into query with AND/OR

[Eric Wong]

Henrik Grimler <henrik@grimler.se> wrote:
> Hi Eric,
> 
> On Sun, Nov 12, 2023 at 12:10:50AM +0000, Eric Wong wrote:
> > Henrik Grimler <henrik@grimler.se> wrote:
> > > Hi,
> > > 
> > > I recently found out about lei and installed it through archlinux's
> > > package manager and am trying out queries. When using AND/OR extra
> > > quotes are inserted in the curl command which messes it up, for
> > > example:
> > > 
> > > $ lei q -I https://lore.kernel.org/all/ -o ~/mail/foo 'dfn:COPYING OR dfn:Makefile'
> > > # /home/grimler/.local/share/lei/store 0/0
> > > # /usr/bin/curl -Sf -s -d '' https://lore.kernel.org/all/?x=m&q=dfn%3A%22COPYING+OR+dfn%3AMakefile%22
> > > # 0 written to /home/grimler/mail/foo/ (0 matches)
> > > 
> > > where it can be seen that it tries to search for 'dfn:"COPYING OR
> > > dfn:Makefile"', and no hits are returned since there is no file named
> > > "COPYING OR dfn:Makefile".
> > 
> > Don't use quotes unless you want a phrase search.
> 
> The quotes are added by lei (or some dependency) when query contains
> space. Happens also if I search for a single file:
>   lei q -I https://lore.kernel.org/all/ -o ~/mail/foo ' dfn:COPYING'
> which results in this curl cmd:
>   /usr/bin/curl -Sf -s -d '' https://lore.kernel.org/all/?x=m&q=+dfn%3A%22COPYING%22
> where %22 then is "

Right, spaces require quotes in sh and lei inserts quotes when
it sees spaces assuming it's a phrase search.  Most queries
involving filenames don't have spaces, and your original query
shouldn't have spaces.  It's 3 separate args in @argv of
`lei_q': [ "dfn:COPYING", "OR", "dfn:Makefile" ]

In other words, no quotes or spaces are needed in your case at all:

$ lei q dfn:COPYING OR dfn:Makefile
(I've omitted the -I and -o args for brevity)

Your original query only passes 1 arg due to single or double quotes
handled in the shell (assuming POSIX-like sh or bash):

$ lei q 'dfn:COPYING OR dfn:Makefile' # don't do this
$ lei q "dfn:COPYING OR dfn:Makefile" # don't do this, either

In both cases the `lei_q' subroutine would only see
[ "dfn:COPYING OR dfn:Makefile" ] in its @argv.

If you have odd cases where you really need spaces in a single
token and maybe not phrase search, --stdin can probably get what
you want more reliably:

$ echo 'dfn:"some filename with spaces" AND something.else' | lei q --stdin

Hope that helps.

Re: [Bug] lei: extra quotes inserted into query with AND/OR

[Henrik Grimler]

Hi Eric,

On Sun, Nov 12, 2023 at 09:02:49AM +0000, Eric Wong wrote:
> Henrik Grimler <henrik@grimler.se> wrote:
> > Hi Eric,
> > 
> > On Sun, Nov 12, 2023 at 12:10:50AM +0000, Eric Wong wrote:
> > > Henrik Grimler <henrik@grimler.se> wrote:
> > > > Hi,
> > > > 
> > > > I recently found out about lei and installed it through archlinux's
> > > > package manager and am trying out queries. When using AND/OR extra
> > > > quotes are inserted in the curl command which messes it up, for
> > > > example:
> > > > 
> > > > $ lei q -I https://lore.kernel.org/all/ -o ~/mail/foo 'dfn:COPYING OR dfn:Makefile'
> > > > # /home/grimler/.local/share/lei/store 0/0
> > > > # /usr/bin/curl -Sf -s -d '' https://lore.kernel.org/all/?x=m&q=dfn%3A%22COPYING+OR+dfn%3AMakefile%22
> > > > # 0 written to /home/grimler/mail/foo/ (0 matches)
> > > > 
> > > > where it can be seen that it tries to search for 'dfn:"COPYING OR
> > > > dfn:Makefile"', and no hits are returned since there is no file named
> > > > "COPYING OR dfn:Makefile".
> > > 
> > > Don't use quotes unless you want a phrase search.
> > 
> > The quotes are added by lei (or some dependency) when query contains
> > space. Happens also if I search for a single file:
> >   lei q -I https://lore.kernel.org/all/ -o ~/mail/foo ' dfn:COPYING'
> > which results in this curl cmd:
> >   /usr/bin/curl -Sf -s -d '' https://lore.kernel.org/all/?x=m&q=+dfn%3A%22COPYING%22
> > where %22 then is "
> 
> Right, spaces require quotes in sh and lei inserts quotes when
> it sees spaces assuming it's a phrase search.  Most queries
> involving filenames don't have spaces, and your original query
> shouldn't have spaces.  It's 3 separate args in @argv of
> `lei_q': [ "dfn:COPYING", "OR", "dfn:Makefile" ]
> 
> In other words, no quotes or spaces are needed in your case at all:
> 
> $ lei q dfn:COPYING OR dfn:Makefile
> (I've omitted the -I and -o args for brevity)
> 
> Your original query only passes 1 arg due to single or double quotes
> handled in the shell (assuming POSIX-like sh or bash):
> 
> $ lei q 'dfn:COPYING OR dfn:Makefile' # don't do this
> $ lei q "dfn:COPYING OR dfn:Makefile" # don't do this, either
> 
> In both cases the `lei_q' subroutine would only see
> [ "dfn:COPYING OR dfn:Makefile" ] in its @argv.
> 
> If you have odd cases where you really need spaces in a single
> token and maybe not phrase search, --stdin can probably get what
> you want more reliably:
> 
> $ echo 'dfn:"some filename with spaces" AND something.else' | lei q --stdin
> 
> Hope that helps.

Aha, I see, thanks for the explanation! Without the single quotes, and
after escaping parantheses, lei works as expected.

For the record, I read some old posts where query was '' quoted, and
thought it was the way to do it (for example
https://people.kernel.org/monsieuricon/lore-lei-part-1-getting-started
and https://josefbacik.github.io/kernel/2021/10/18/lei-and-b4.html)

Best regards,
Henrik Grimler

Re: [Bug] lei: extra quotes inserted into query with AND/OR

[Eric Wong]

Henrik Grimler <henrik@grimler.se> wrote:
> Aha, I see, thanks for the explanation! Without the single quotes, and
> after escaping parantheses, lei works as expected.

Good to know.

> For the record, I read some old posts where query was '' quoted, and
> thought it was the way to do it (for example
> https://people.kernel.org/monsieuricon/lore-lei-part-1-getting-started
> and https://josefbacik.github.io/kernel/2021/10/18/lei-and-b4.html)

I'm not sure, I'm a little surprised that they do, but it may be
the preprocessing that the hacky approxidate wrapper (for
rt:1.month.ago..) somehow smooths things over a bit...

I'm sleepy now and head hurts from trying to figure out some
-cindex bugs :x, so maybe I missed some things.

I do recommend --stdin for more complex queries, and I just
posted a patch which should let it work better for terminals
(not just pipes/regular files):

https://public-inbox.org/meta/20231112131233.718614-1-e@80x24.org/