handling many matches [was: [ELPA] New package: transient]

[Drew Adams <drew.adams@oracle.com>]

> > But then you have the reverse problem: after typing `file-name` in
> > fido-mode you get too many false-positive for functions which aren't
> > "manipulating file names" but merely return a file-name.
> 
> That's a fair point, but still the discoverability is not too bad.
> Given weighted sorting, the relevant functions are mostly at the top,
> see the screenshot.
> 
> Although we might want to look into why backup-file-name-p is sorted
> before file-name-quoted-p.

A comment on the general situation here, hoping it helps -

The problem of getting many matches is a general
one, and one that's more prevalent when matching
methods get extended or combined.

Just extending from prefix to substring matching
can increase the number of matches greatly.  Fuzzy
and regexp matching can both add to the problem and
sometimes help reduce it.

(The "problem" is also a helpful feature, and an
opportunity, of course.)

Icicles has had to deal with this inherent problem
and opportunity longer than other completion
frameworks, and a few of its features can really
help to tame it.  I mention them here because,
regardless of how they might be implemented or
presented to users (the UI), I think similar
features could help Emacs.

1. Progressive matching, i.e., narrowing the set
   of candidates by matching another pattern.
   (Rinse and repeat.)

   Matching multiple simple patterns is both much
   easier and more flexible/powerful than trying
   to come up with a single complex pattern (or
   fewer but more complex patterns).

2. Pruning, by excluding matches.

   This builds on #1.  You add another pattern to
   match (#1), then hit a key to subtract the
   resulting set of matches from the previous set.

   In your screenshot, for example, if you know
   you're not interested in the `tramp' matches,
   you just use #1 to add matching `tramp' as an
   additional requirement, then hit a key (`C-~'
   in Icicles) to prune those matches.  Similarly
   for other, easy to recognize blocks of matches.

   It's very quick to filter matches this way, and
   it would be impossible, in the general case, to
   get the same result by any particular completion
   style/method.  Only you know what it makes sense
   to prune, in any given context.

   At each step of additional matching (i.e. #1)
   you can hit a key (Icicles uses `C-g') to annul
   that match requirement.  For example, after you
   match `tramp', if you hit `C-g' then the matches
   return to including `tramp'.  (If you see that
   you want to keep some but not all of the `tramp'
   matches then you can just use #1 again to narrow
   further.

3. On-demand help on individual candidates.

   Be able to hit a help key to get a short or
   long description for a given candidate.  This
   of course requires some way to indicate/pick
   a single candidate just for help.

   In Icicles there are several ways to do this,
   including (a) cycling among them and (b) #1
   above, followed by a help key, followed by
   `C-g' - i.e., narrow to a single candidate,
   show its description, then widen back.

   This feature is not a way to add/subtract
   multiple matches all at once.  It's more of a
   tell-me-more, to help you decide about further
   matching.

In Icicles there are many more ways to manipulate
a set of matches, to tame the plethora problem.
But I wanted to mention those 3 - particularly the
first 2.

___

Now a word about sorting in this context, since
you mention it.

Sorting is important for many reasons, especially,
but not only, in a context where you can (or must)
access candidates in order, e.g., cycle among them.

You mention orders based on candidate scoring:

  "Given weighted sorting, the relevant functions
   are mostly at the top"

That can be helpful, but I think it's often
overrated.  The view that it's important or very
helpful can just be the result of making a
virtue out of necessity: In contexts where the
main or only candidate access is by cycling, it
follows that order is very important - just for
ease/speed of access.

But when other powerful ways of accessing are
available (e.g. easy narrowing - #1, #2), the
sort order can serve various other, otherwise
useful and interesting purposes than just
accessing a candidate.

Beyond just contrasting ease of access: Scoring
tries to measure relevancy, but real candidate
relevancy in many contexts can go beyond any
what any name-matching regime can offer.  Real
relevancy can often be about matching things
about candidates that aren't contained in their
names.

The PREDICATE arg for `completing-read', etc.
makes it possible to match based on more than
just candidate names.  But unless some hoops
are jumped through, candidate sorting is based
only on names.  PREDICATE constrains matches,
but the result is a set of names, and those are
what get scored.

Which brings up a related point: the purpose of
completion - showing matching candidates.  If
the only purpose is to choose a candidate, that
colors what one thinks is important/helpful.
Other purposes come from the value of seeing
and manipulating a _set_ of things.

And sort order fits in here: It can be useful
to present a set of candidates in different
ways, including different orders, for different
purposes.  It's not just about sequential order
for accessing a candidate.

Consequently, it can be helpful to be able to
change the sort order on the fly (during
completion/matching), in order to present the
same set of matches in different ways.

These 2 Icicles pages give an idea of other
uses of sort orders.  They describe or show
orders that help you see relations among
candidates in a given set of matches.  (And
yes, such sorts do often involve candidate
info that goes beyond names.)

https://www.emacswiki.org/emacs/Icicles_-_Sorting_Candidates#DifferentSortsForDifferentUses

https://www.emacswiki.org/emacs/Icicles_-_Key_Completion#Sorting