Any package for boolean search?

Any package for boolean search?

[Jean Louis]

Hello to all,

Boolean Search: This is a method where search terms are combined using
boolean logic, as described by the operators AND, OR, and NOT.

I would like to implement this classic into the website search. And I
wish to keep it simple.

Here are 20 different queries using boolean search operators AND, OR, and NOT:

1. `apple AND fruit`
2. `car OR truck`
3. `not google`
4. `(apple OR banana) AND fruit`
5. `football NOT soccer`
6. `book AND author`
7. `(dog OR cat) NOT animal`
8. `house AND (furniture OR decor)`
9. `google AND (search OR engine)`
10. `not (car OR truck)`
11. `(phone OR laptop) AND technology`
12. `flower AND (garden OR plant)`
13. `(music OR video) NOT entertainment`
14. `amazon AND (shopping OR online)`
15. `not (phone OR computer)`
16. `(camera OR photo) AND equipment`
17. `travel AND (hotel OR resort)`
18. `(newspaper OR magazine) NOT media`
19. `(game OR puzzle) AND entertainment`
20. `(coffee OR tea) AND (breakfast OR morning)`

I think if there are no parenthesis I should even disregard the OR, is
it right?

I wish there would be some ready made package that handles those
issues.


Jean Louis

RE: [External] : Any package for boolean search?

[Drew Adams]

> Boolean Search: This is a method where search terms are combined using
> boolean logic, as described by the operators AND, OR, and NOT.
> 
> I would like to implement this classic into the website search. And I
> wish to keep it simple.

I can't help you wrt website searches etc.  Hopefully
someone else will.
___

But FWIW, Isearch+ provides such functionality, on
the fly, for incremental buffer/file searching.

https://www.emacswiki.org/emacs/DynamicIsearchFiltering

Re: [External] : Any package for boolean search?

[Jean Louis]

* Drew Adams <drew.adams@oracle.com> [2024-12-25 19:21]:
> > Boolean Search: This is a method where search terms are combined using
> > boolean logic, as described by the operators AND, OR, and NOT.
> > 
> > I would like to implement this classic into the website search. And I
> > wish to keep it simple.
> 
> I can't help you wrt website searches etc.  Hopefully
> someone else will.

I need more time, as I spend time in research. I think hashes are not
easiest way to go, so I will do this way:

1. Make a simple list of strings; 

2. Concatenate id, name, description, summary or full text without 
markup to single line;

3. Search lines;

4. Extract ID

5. Load nicely prepared name, description, image from hash

6. Display on the website.

We will see how fast it works.

> But FWIW, Isearch+ provides such functionality, on
> the fly, for incremental buffer/file searching.
> 
> https://www.emacswiki.org/emacs/DynamicIsearchFiltering

-- 
Jean Louis

Re: Any package for boolean search?

[Joel Reicher]

Jean Louis <bugs@gnu.support> writes:

> Boolean Search: This is a method where search terms are combined 
> using boolean logic, as described by the operators AND, OR, and 
> NOT.
>
> I would like to implement this classic into the website 
> search. And I wish to keep it simple.

Is there an Emacs or elisp aspect to this question I might be 
missing?

Are you after a package to submit a search to a particular search 
engine? Is that what you mean?

(There are a few such packages on melpa; I don't know how they 
compare.)

Regards,

        - Joel

Re: Any package for boolean search?

[Jean Louis]

* Joel Reicher <joel.reicher@gmail.com> [2024-12-26 09:03]:
> Jean Louis <bugs@gnu.support> writes:
> 
> > Boolean Search: This is a method where search terms are combined using
> > boolean logic, as described by the operators AND, OR, and NOT.
> > 
> > I would like to implement this classic into the website search. And I
> > wish to keep it simple.
> 
> Is there an Emacs or elisp aspect to this question I might be
> missing?

I don't know what you mean. I am looking for some ready package that
provides maybe string searching within other strings, by using boolean
operators.

I am starting the logic here below:

(defun wrs-search-query-process (query)
  (let ((query (wrs-search-clean-query query)))
    (cond
     ;; dispatch if there is no boolean
     ((not (string-match-p (rx word-start (or "OR" "NOT" "AND" word-end)) query))
      (wrs-search-query query))
     
     ;; dispatch when there is AND but not OR or NOT
     ((and (string-match-p (rx word-start "AND" word-end) query)
	   (not (string-match-p (rx word-start (or "OR" "NOT" word-end)) query)))
      (wrs-search-query-and query)))))

So there will be different ways how to process those parts of the
query separated by AND, OR, NOT and in various combinations.

> Are you after a package to submit a search to a particular search
> engine? Is that what you mean?

No. 

It is website search, when user enters some terms to get list of
website pages. I don't have many, but 1300+ in single website has some
key information for people.

-- 
Jean Louis

Re: Any package for boolean search?

[Jean Louis]

* Joel Reicher <joel.reicher@gmail.com> [2024-12-26 09:03]:
> Jean Louis <bugs@gnu.support> writes:
> 
> > Boolean Search: This is a method where search terms are combined using
> > boolean logic, as described by the operators AND, OR, and NOT.
> > 
> > I would like to implement this classic into the website search. And I
> > wish to keep it simple.
> 
> Is there an Emacs or elisp aspect to this question I might be missing?

Did you mean if it is in the context of Emacs Lisp? Of course, that is
why I am looking if there is any available package.

Maybe I should index all packages and summarize them to get database
of what is available.

package-list is not enough descriptive.

-- 
Jean Louis

Re: Any package for boolean search?

[Joel Reicher]

Jean Louis <bugs@gnu.support> writes:

> * Joel Reicher <joel.reicher@gmail.com> [2024-12-26 09:03]:

[...]

>> Is there an Emacs or elisp aspect to this question I might be 
>> missing?
>
> I don't know what you mean. I am looking for some ready package 
> that provides maybe string searching within other strings, by 
> using boolean operators.

Why are you expecting this to be a package? Strings are a 
native/primitive elisp datatype.

(info "(elisp) Text Comparison")

and also string-match at (info "(elisp) Regexp Search")

Regards,

        - Joel

Re: Any package for boolean search?

[Jean Louis]

* Joel Reicher <joel.reicher@gmail.com> [2024-12-27 02:16]:
> Jean Louis <bugs@gnu.support> writes:
> 
> > * Joel Reicher <joel.reicher@gmail.com> [2024-12-26 09:03]:
> 
> [...]
> 
> > > Is there an Emacs or elisp aspect to this question I might be
> > > missing?
> > 
> > I don't know what you mean. I am looking for some ready package that
> > provides maybe string searching within other strings, by using boolean
> > operators.
> 
> Why are you expecting this to be a package? Strings are a native/primitive
> elisp datatype.

Thanks, though, I do not expect, I am asking if someone knows about
it. Those AND, OR, NOT are pretty classic, not so?

> (info "(elisp) Text Comparison")

I can't find there anything related, sure as basic functions is very
useful, but doesn't parse the query.

- cat dog -- I would use this with AND automatically; but maybe add OR
  on the end of results;

- cat and dog -- this is ONLY AND, without OR results

- cat dog OR giraffe in Africa -- here I would parse it as:
                 
     - cat AND dog -- is it there?
     - but it can be also "giraffe AND Africa" 

There are too many combinations and I have to make it so that I can
first parse the query into smaller pieces.

Then this query is dispatched to more basic functions.

> and also string-match at (info "(elisp) Regexp Search")

Sure! Thanks much. I am using those functions and it is useful let's
say, if I need to search with "OR" some few words.

But what if there are complex queries:

cat plays with dog OR dog plays with dogs NOT giraffe

or maybe this way:

(cat plays with dog) OR (dog plays with dogs) NOT giraffe

Then I have to parse it, separate it:

- NOT giraffe
- is there any result for cat? But no giraffe
- or any result for dog? But no giraffe

I hope you understand the meaning by that example.

It is called Boolean query parser, but when I started this question I
did not know the title. Now I know it.

Here is example in PHP

skipperbent/pecee-boolean-query-parser: Convert a boolean search query into a query that is compatible with a fulltext search.
https://github.com/skipperbent/pecee-boolean-query-parser

As there is apparently no such ready package in Emacs, then I will go
slow and start with functional examples:

1. For any query, find all occurences matching exactly; for query "cat
   plays" it will find "Cat plays with ball"

2. Parse words and find all occurences matching same words but with
   AND: for query "cat plays" it will also find "Cat likes doggy which
   plays with the ball"

Then I can think from there how to develop OR and NOT.

-- 
Jean Louis

Re: Any package for boolean search?

[Joel Reicher]

Jean Louis <bugs@gnu.support> writes:

> * Joel Reicher <joel.reicher@gmail.com> [2024-12-27 02:16]:
>> Jean Louis <bugs@gnu.support> writes:
>>
>>> * Joel Reicher <joel.reicher@gmail.com> [2024-12-26 09:03]:
>>
>> [...]
>>
>>>> Is there an Emacs or elisp aspect to this question I might be 
>>>> missing?
>>>
>>> I don't know what you mean. I am looking for some ready 
>>> package that provides maybe string searching within other 
>>> strings, by using boolean operators.
>>
>> Why are you expecting this to be a package? Strings are a 
>> native/primitive elisp datatype.
>
> Thanks, though, I do not expect, I am asking if someone knows 
> about it. Those AND, OR, NOT are pretty classic, not so?

For algebra, yes. For searching, no.

But because searching is done with predicates (equality and 
perhaps an ordering, as I mentioned in another email) other 
predicates can be formed from the basic ones using algebraic 
operators.

>> (info "(elisp) Text Comparison")
>
> I can't find there anything related, sure as basic functions is 
> very useful, but doesn't parse the query.

Oh, you're asking for something to parse a query. Now I 
understand.

You can't expect there already to be a package for parsing a 
syntax you're designing now.

But if you're not designing the syntax and you want to find 
something that will do an existing syntax, possibly the simplest 
is Lisp itself since Lisp has all these operators and is 
homoiconic. If the query syntax is Lisp, the string can be passed 
to read and eval and you're done, I think.

Regards,

        - Joel

Re: Any package for boolean search?

[Jean Louis]

* Joel Reicher <joel.reicher@gmail.com> [2024-12-28 06:46]:
> But if you're not designing the syntax and you want to find something that
> will do an existing syntax, possibly the simplest is Lisp itself since Lisp
> has all these operators and is homoiconic. If the query syntax is Lisp, the
> string can be passed to read and eval and you're done, I think.

I would not like eval-ing on remote server for some reasons 🤔.

-- 
Jean Louis

Re: Any package for boolean search?

[Joel Reicher]

Jean Louis <bugs@gnu.support> writes:

> * Joel Reicher <joel.reicher@gmail.com> [2024-12-28 06:46]:
>> But if you're not designing the syntax and you want to find 
>> something that will do an existing syntax, possibly the 
>> simplest is Lisp itself since Lisp has all these operators and 
>> is homoiconic. If the query syntax is Lisp, the string can be 
>> passed to read and eval and you're done, I think.
>
> I would not like eval-ing on remote server for some reasons 🤔.

Solving a search query is an eval, but I think you are saying you 
want to avoid code injection which is fair enough.

Off the top of my head you can map over the sexpr returned by read 
and rewrite the symbols to ones you control before evaling.

In my opinion it is still worth avoiding a parse, which is what 
Lisp can give you.

Regards,

        - Joel

Re: Any package for boolean search?

[Jean Louis]

* Joel Reicher <joel.reicher@gmail.com> [2024-12-28 19:08]:
> Solving a search query is an eval, but I think you are saying you want to
> avoid code injection which is fair enough.

Search query is just a string I receive so parsing the string for OR
will be my first development.

> Off the top of my head you can map over the sexpr returned by read and
> rewrite the symbols to ones you control before evaling.

I understand that approach, and I understand there would be way to
allow only some symbols, though I assure you that reading the sexp is
not needed. Generally I do a lot of evals in my documents. I find the
feature fantastic. I am using it for template interpolation.

RCD Template Interpolation System for Emacs:
https://hyperscope.link/3/7/1/3/3/RCD-Template-Interpolation-System-for-Emacs.html

Though for website search it will be string parsing on chunks of
different types of queries.

> In my opinion it is still worth avoiding a parse, which is what Lisp can
> give you.

1. First stage of development

   - treat it as exact, collect IDs
   - allow 1-2 Levenshtein difference, 
      -- Function: string-distance string1 string2 &optional bytecompare
     This function returns the “Levenshtein distance” between the source
     string STRING1 and the target string STRING2.  The Levenshtein

   - get 5-10 pages where both or more words of the query, which are
     not stop words, appear apart from each other, unless user quoted
     the query

2. Second stage

   - recognize if there is any OR

-- 
Jean Louis