Re: [PATCH 0/5] lib: make folder: prefix literal

[Austin Clements <aclements@csail.mit.edu>]

Quoth Carl Worth on Jan 29 at 11:32 am:
> Jani Nikula <jani@nikula.org> writes:
> > Unfortunately, I haven't had the time to experiment with this. But it
> > bugs me that the probabilistic folder: prefix has stemming and it's case
> > insensitive. It's possible to work around the stemming with the anchors
> > you suggest or by quoting, but is there a way to have case sensitive
> > probabilistic prefixes?
> 
> The stemming and case insensitivity just has to do with which terms are
> shoved into the database, (you have to add extra terms to get these
> features). If we're getting those features for folder now, (and I agree
> that we don't want them), it's because we're calling some Xapian
> convenience function along the lines of "create a bunch of terms for
> this chunk of text".
> 
> The fix for that is to do the simple thing and simply break the path at
> each '/' and add a term for each component. Then these problems all go
> away.

I think you're assuming we have much more control over this than we
do.  It's true that we're using Xapian::TermGenerator for this, which
is what strips case and stems terms (and removes any punctuation like
$ or ^), but Xapian's current query parser only gives us two options
for a prefix: either don't parse them at all (boolean terms), or parse
them using TermGenerator (probabilistic terms).  We can index these
terms however we want, but there's simply no hook into the query
parser that would let us split the query at each '/' at search time.

> So fixes for this should not require switching from a probabilistic to a
> Boolean prefix.