Relevance search in Emacs

Relevance search in Emacs

[Jean Louis]

Before weeks I was discussing and trying to find something that could
replace the `helm' in Emacs, so that I spare using external
libraries. Then discussion with Drew clarified that what I am looking
is rather type of a filter. But I would like to filter by
relevance. Some packages like `ivy' and `helm' do offer relevance
searches. Yet I wish to spare the basic package from using external
functions. This way me or other users can use any completion or Emacs
built-in completion function which is generally more user friendly in
my opinion.

Chapter 12. Full Text Search
https://www.postgresql.org/docs/current/textsearch-intro.html#TEXTSEARCH-MATCHING

What is good is that database PostgreSQL offers built-in relevance
searches and that spares coding and using other packages, so I am
transitioning to minimize usage of helm or take it completely out and
replace it with tabulated-list-mode that will offer options to act on.

I hope to replace the TAB in tabulated-list-mode so that it can offer
screen of various options or completion from minibuffer for actions
that may be invoked on a specific item.

In the manual there is little I can find related to "relevance". If
somebody knows if there is some "relevance" search in Emacs built-in
packages let me know, otherwise I will let database do the job.

RE: Relevance search in Emacs

[Drew Adams]

> Chapter 12. Full Text Search
> https://www.postgresql.org/docs/current/textsearch-intro.html*TEXTSEARCH-MATCHING

That's apparently what's often called full-text search,
and which is an indexed search.  The text is indexed
ahead of time, and the index is used for search "lookup".

It means that the text itself is not searched or searchable,
at least by those means, but it could of course be mixed
with a separate pass of actual search of the text (e.g.,
for regexp matching).

Typically words are indexed, along with their positions
(so nearness etc. can be among the search criteria).

> What is good is that database PostgreSQL offers built-in relevance
> searches..
> In the manual there is little I can find related to "relevance".

Which manual?

> If somebody knows if there is some "relevance" search in Emacs built-in
> packages let me know, otherwise I will let database do the job.

"Relevance" can mean anything.  Think of how much can go
into, say, a google search: "relevance" there can include
intimate detail about you as a person, your likes, etc.

Yes, some Emacs 3rd-party libraries do provide "scoring"
of some kinds of searches.  For example, for certain kinds
of fuzzy matching a score can indicate how "closely" a
given candidate is matched by your search pattern.

You'd do well, when looking for answers here or elsewhere,
to specify the kind of relevance you have in mind.

In general, it's up to you who define the search procedure
to define the relevant (!) relevance criteria, or those
that you make available to a user.

Re: Relevance search in Emacs

[Jean Louis]

* Drew Adams <drew.adams@oracle.com> [2020-12-06 00:07]:
> > Chapter 12. Full Text Search
> > https://www.postgresql.org/docs/current/textsearch-intro.html*TEXTSEARCH-MATCHING
> 
> That's apparently what's often called full-text search,
> and which is an indexed search.  The text is indexed
> ahead of time, and the index is used for search "lookup".

I am not using indexed method, just searching on the fly.

Relevance search could be simpler in Emacs just for strings.

This is example of an SQL query that gives results on the fly without
previous index and it searches only within hyperlink names and tags,
it is easy to add the body or text to it. It does not change visible
speed.

(format "SELECT DISTINCT hlinks_id, hlinks_name, hlinktypes_name,
actionstatuses_name FROM hlinks, hlinktypes, actionstatuses WHERE
(actionstatuses_id = hlinks_actionstatuses AND hlinks_hlinktypes =
hlinktypes_id) AND to_tsquery(%s) @@ to_tsvector(hlinks_name || ' ' ||
hlinks_tags) %s" query parent)

> > What is good is that database PostgreSQL offers built-in relevance
> > searches..  In the manual there is little I can find related to
> > "relevance".
> 
> Which manual?

Emacs manual. I have been expecting something similar in
completing-read or some other functions.

> > If somebody knows if there is some "relevance" search in Emacs built-in
> > packages let me know, otherwise I will let database do the job.
> 
> "Relevance" can mean anything.  Think of how much can go
> into, say, a google search: "relevance" there can include
> intimate detail about you as a person, your likes, etc.

What I mean is that words are closer to each other, for example that
term:

google intimate person

find your above quoted paragraph.

> Yes, some Emacs 3rd-party libraries do provide "scoring"
> of some kinds of searches.  For example, for certain kinds
> of fuzzy matching a score can indicate how "closely" a
> given candidate is matched by your search pattern.

Fuzzy matching could be good. Is there in Emacs similar? Not that I am
looking for outside libraries.

RE: Relevance search in Emacs

[Drew Adams]

> > https://www.postgresql.org/docs/current/textsearch-intro.html*TEXTSEARCH-MATCHING
> >
> > That's apparently what's often called full-text search,
> > and which is an indexed search.  The text is indexed
> > ahead of time, and the index is used for search "lookup".
> 
> I am not using indexed method, just searching on the fly.

I see.  I figured it was full-text search because PostGres is
a database.

> Relevance search could be simpler in Emacs just for strings.

It can be anything you want it to be.

> > "Relevance" can mean anything.  Think of how much can go
> > into, say, a google search: "relevance" there can include
> > intimate detail about you as a person, your likes, etc.
> 
> What I mean is that words are closer to each other, for example
> that term: google intimate person
> find your above quoted paragraph.

IOW, nearness measures.  That's typically used with indexed
text.  No, AFAIK, vanilla Emacs has no such thing.  And it
doesn't have any relevance sorting, except possibly for
recency of use.

> > Yes, some Emacs 3rd-party libraries do provide "scoring"
> > of some kinds of searches.  For example, for certain kinds
> > of fuzzy matching a score can indicate how "closely" a
> > given candidate is matched by your search pattern.
> 
> Fuzzy matching could be good. Is there in Emacs similar? Not that I am
> looking for outside libraries.

Not in vanilla Emacs.  Well, I guess maybe they've recently
added some pseudo-fuzzy matching as a completion style.
But you can easily add your own as a completion style, for
any Emacs version.

3rd-party libraries have fuzzy matching.

Isearch+ gives you the ability to use nearness matching,
using dynamic filtering:

https://www.emacswiki.org/emacs/DynamicIsearchFiltering

From that page:

  Search for something that is near something else – within
  a given number of characters, words, lists, or sentences
  (you can add to this list of distance units, using option
  ‘isearchp-movement-unit-alist’). You specify the nearness.
  You can also constrain the nearby pattern to be only before
  or only after the search hit.

  Search for stuff in the union of given contexts. Example:
  search for text that is near ‘cat’ or near ‘dog’.

  Search for stuff outside contexts: Use the complement of
  any set of contexts as the search space. Example: search
  for text that is not near ‘cat’ and not near ‘dog’.

You can do anything you want, from just vanilla Emacs, as you
know.  You can also look to the code or design of 3rd-party
features for help or inspiration.  Free software means you
have the source.  You can, but you need not, reinvent your
own wheels.

Re: Relevance search in Emacs

[TRS-80]

> On 2020-12-05 13:18, Jean Louis wrote:
> Before weeks I was discussing and trying to find something that could
> replace the `helm' in Emacs, so that I spare using external
> libraries. Then discussion with Drew clarified that what I am looking
> is rather type of a filter. But I would like to filter by
> relevance. Some packages like `ivy' and `helm' do offer relevance
> searches. Yet I wish to spare the basic package from using external
> functions. This way me or other users can use any completion or Emacs
> built-in completion function which is generally more user friendly in
> my opinion.

In fairness I never actually used Helm, because in my initial research
it looked to me to be much more obtrusuive and less in line with
established Emacs conventions.  I chose to go with Ivy at that time and
never looked back.  Apologies to anyone who prefer Helm, I am not trying
to start some flame war.  :)

Strictly speaking, you are correct, as under the hood my understanding
is that Ivy and Helm both implement a lot of their own things "on top
of" or "in place of" the "built-in" options.  However this is not
something I have ever really noticed in terms of being any sort of
"real" problem for me in the several years I have now been using Ivy.
But rather something I only even became aware of more recently as a
theoretical / structural criticism of Ivy/Helm coming from certain
circles[0] who instead seem to prefer to implement something... I dunno,
more "minimal" or...?

If I'm being honest those criticisms struck me as a bit self-serving,
especially coming from such a relative newcomer to the field.

Also, I seem to recall hearing that there was some discussion about
moving some part (or all?) of Ivy directly into Emacs itself?  I do know
that Ivy for a long time now have been assigning their copyrights to
GNU, they are very upfront about this to contributors at their GitHub in
fact.

Even those who would criticise Ivy/Helm[0] admit that there is a lot
more functionality that they offer than the "built-in" solutions.  I
realized this in developing some of my own (as yet unreleased) packages
where I depend on some of that functionality, and thus, in fact require
Ivy as a dependency.

If this was more about search itself than choice of completion
framework, I guess I totally missed the point, or maybe that's another
discussion to have.

Anyway, maybe give Ivy a try instead of Helm.  Or if you are swayed by
arguments at my footnote link, one of those other "minimal" completion
frameworks.

Cheers,
TRS-80

[0] https://github.com/raxod502/selectrum#why-use-selectrum

Re: Relevance search in Emacs

[Jean Louis]

* TRS-80 <lists.trs-80@isnotmyreal.name> [2020-12-06 01:24]:
> Also, I seem to recall hearing that there was some discussion about
> moving some part (or all?) of Ivy directly into Emacs itself?  I do know
> that Ivy for a long time now have been assigning their copyrights to
> GNU, they are very upfront about this to contributors at their GitHub in
> fact.

As ivy is in GNU ELPA those issues are already solved. It is up to
author to polish the package if necessary to include it. But is not
necessary as it is already in GNU ELPA and new packages can request it
and it will be easier installed than from repositories not being
straight in GNU Emacs.

> Even those who would criticise Ivy/Helm[0] admit that there is a lot
> more functionality that they offer than the "built-in" solutions.

What Emacs needs is dynamic menu that user can filter quickly and work
on it, it would be similar to dmenu X program but one can only choose
one item there, or fzf console program which is more similar to what I
mean. Helm and ivy and others try to be that. But we do not have very
nice menu system in Emacs. And I do not refer to menu bar on top as
such is limited, then not dynamic.

Helm in full screen is pretty much what I think but again it limits
itself to execution of it.

Image list of items such as products. One need maybe to mark such and
move from one category to other. Job is tedious by doing it with
helm. List of people to which one need to send email by selecting them
by hand. Helm offers C-SPC to select each item and then run TAB
actions to do something with them. But programming with helm is
wasting time pretty much.

Doing it with tabulated-list-mode is so much easier:

- filter items (provided function exist for this interface)
- mark items (those are ID numbers)
- do something with items

For now I am transitioning from helm to tabulated-list-mode as for its
simplicity and interface that remains on screen, it is does not
disappear like completion interfaces (helm, ivy).

It just lacks dynamic filtering.

> realized this in developing some of my own (as yet unreleased) packages
> where I depend on some of that functionality, and thus, in fact require
> Ivy as a dependency.

First I have started going helm dependency direction as that was first
that I have discovered, then I have tried to make it that users can
decide on completion. This is best approach and also does not bind
myself to specific completion. Run it with built-in completion or turn
any other completion package.

It is best to have packages liberated from specific completion
packages such as ivy, helm, etc.

For me ivy does well with relevance search for few words apart from
each other may be used to locate the item. But it is still
"completion", it does not allow me actually work on the filtered set
of items. It wants me to complete. I think helm too. 

> If this was more about search itself than choice of completion
> framework, I guess I totally missed the point, or maybe that's another
> discussion to have.
> 
> Anyway, maybe give Ivy a try instead of Helm.  Or if you are swayed by
> arguments at my footnote link, one of those other "minimal" completion
> frameworks.

I am testing them all. Selectrum is just core stuff, it is not
integrated well for end users and it is completion, not dynamic filter
after filter. I hope you get what I mean.