Re: Emacs completion matches selection UI

[Toby Cubitt <tsc25@cantab.net>]

On Sat, Jan 04, 2014 at 06:39:11AM +0400, Dmitry Gutov wrote:
> On 03.01.2014 02:58, Toby Cubitt wrote:
> > It's a shame Company et al. didn't build on and contribute to
> > Completion-UI, and instead chose to (each) reinvent the wheel. But that's
> > the usual messy history of code (which maybe we're finally going to do
> > something to tidy up?).
> 
> Maybe the widget APIs weren't particularly fitting, though. See my 
> response to your proposal below.

Note that my proposal was a first stab at an API for generic Emacs
completion UI; the Completion-UI APIs are much richer than this.

The modern Completion-UI APIs are very likely either already sufficient
or easily extended to support everything Company needs. But that's
nowadays. The original 2006 Completion-UI APIs were much clunkier, and
almost certainly weren't up to the job.

I'm sure if you'd built Company on top of Completion-UI, we would have
rapidly improved the APIs to support everything you needed as Company
grew. But that's ancient history now, and no big deal.

> > Do you want fast predictive completion of plain text, or
> > text markup languages like LaTeX?
> 
> It could be a decent replacement for dabbrev (I mostly use it to quickly 
> complete to some symbol already present before point in the same 
> buffer), but the problem with "smart" algorithms that keep history is 
> that often enough you can't predict the first option it will offer you. 
> That can be annoying, so maybe I don't.

You can switch off the learning algorithms by toggling a customization
option, and freeze the ordering once and for all if you want
stability. If you're happy with the default frequency tables, you can
even switch it off from the start.

I originally thought I'd want to disable the learning algorithms once I'd
trained Predictive on a corpus of my writing, for precisely the reason
you describe. But in practice the order is stable enough that I never
found it necessary.

> >> Would you like to propose an API for it? Same as popup.el?
> >
> > As a first attempt, I'd propose something like
> >
> >    (complete-in-buffer COMPLETION-SOURCE &optional PREFIX-FUNCTION)
> >
> > where COMPLETION-SOURCE is a function to call to obtain completions, and
> > PREFIX-FUNCTION is an optional function to call to determine what to
> > complete.
> 
> But what if we have a list of completion sources (see the description 
> down below<*>)? Some of them can have a different notion of the prefix. 
> In Company, we try to remember which backend we're working with at the 
> moment, so that we only ask it for completions, candidate metadata,
> etc.

Sure, Completion-UI also remembers the backend ("completion source") that
was used to find the current set of completion candidates, so that you
can continue to interact with it sensibly.

Clearly any generic Emacs completion UI would have to do something
similar. I don't see a big problem supporting multiple different
completion sources. Indeed, I view it as a basic requirement of any
decent Emacs completion API if it's to be the generic UI.

Completion-UI stores state by marking the location of the in-progress
completion with an overlay, and storing data in overlay properties. You
could alternatively use text properties, as you mention later. But I find
overlays a better match for this kind of thing. (Presumably Company does
something similar.) This means you can, if you want, have multiple
pending completions using different sources in the same buffer at the
same time, and Completion-UI will cope just fine if you switch back and
forth between then. (Though the default setting of
`completion-how-to-resolve-old-completions' disables this, as it's at
least as confusing as `enable-recursive-minibuffers'.)


> Though maybe it could work if PREFIX-FUNCTION iterated though backends, 
> saved which one responded, and then we could use that value in 
> COMPLETION-SOURCE, as well as when implementing the commands providing 
> the additional info. In this case, we'll at least need to have a way to 
> know when the widget was dismissed or refreshed to erase the saved 
> backend value.

The above API was just a first stab at the most basic generic Elisp API
for displaying a list of completion candidates and letting the user
select one. It wasn't intended to be an API for defining a bunch of
different a completion sources (backends) and selecting between then.

As I wrote in my follow-up email, at this point we may be stuck with
`completion-at-point-functions' for the latter. (Though we can try
working on Stefan to see if he'll let us ditch it for something less
clunky ;-)

> In `company-capf', we iterate though the completion-at-point-functions
> each time. This works, and works well enough, since the "does it fit"
> code is usually fast, but it looks fairly bizarre to me, logically.

I agree. The `completion-at-point-functions' API is convoluted compared
to either the Company-native API or the Completion-UI API. On the other
hand, it's already part of Emacs and is sufficiently flexible (if a
little ugly). So we may be stuck with it now...

> Ideally, though, the widget interface would be a drawing API of sorts: 
> "display these candidates at this point with this current index", 
> "update index", "hide", since we have everything else implemented already.
> 
> > Calling this would invoke whatever completion widgets the user had
> > enabled in customize (with sensible defaults - maybe the popup.el
> > interface or similar, or perhaps the default should be the usual
> > *Completions* buffer).
> 
> I'd still like to a natural way to extend such widgets with new 
> commands. Passing a keymap as argument, or rebinding a dynamic variable 
> that points to keymap works, I guess, but it's not very nice.

Well, this is only meaningful for some of the widgets. Indeed, "widgets"
is mislead; the completion UIs in Completion-UI aren't necessarily
widgets at all. For example, there are no "commands" at all in the
dynamic completion UI, nor is it a "widget" in any normal sense of the
word.

But for those UIs for which it makes sense, Completion-UI already allows
them to be extended, if necessary in different ways for different
completion sources. In classic Emacs fashion, you supply
functions/keymaps/etc. that override the defaults used to create widget
contents. 

This API could rather easily be improved and extended once it becomes
clear what would be useful or more convenient. But the basic machinery is
already in place.

(The list of override functions, keymaps etc. is *not* hard-coded. The
API for adding a new completion user interface to Completion-UI allows
you to add additional configuration options recognised by the new UI. The
only limitation is that these won't be documented in the
`completion-ui-register-source' docstring.)

> >> A mouse-only menu won't fit Company because it won't work for idle
> >> completion, as I mentioned elsewhere in this thread, and we generally
> >> want to provide a consistent keyboard interface. There'll also be issues
> >> with implementation related to that.
> >
> > When you register a new UI with Completion-UI, you tell it whether the UI
> > is suitable for idle completion. If it is, it becomes one of the
> > customization options you can select for the idle completion UI.
> 
> Ok, it's reasonable approach, but that means that the UIs can't be 
> treated uniformly, which is sort of a bummer.

The UIs absolutely cannot be treated uniformly, because not all of them
are "display a list of completions"-style UIs. E.g. the "dynamic
completion" UI is a completely different beast.

Even amongst the "display a list of completions"-style UIs, the
mouse-driven toolkit menus cannot be used in the terminal, nor can pop-up
frames, nor can tooltips. That doesn't mean one should not provide these
interfaces at all as options. It just means they probably aren't good
default options.

The default idle-completion UI in Completion-UI uses popup.el, which
works everywhere. It's the best "display a list of completions"-style UI,
for this and many other reasons. But that doesn't mean one shouldn't
provide toolkit menus, or tooltips, or popup-frames *as well*, either so
they can be triggered manually or as alternative options for the idle UI.

Completion-UI has default keybindings to bring up the other UIs, even
though they're not displayed by default. This is particularly appropriate
for e.g. the mouse-driven toolkit menu and completion browser menu (which
steal the keyboard focus).

My philosophy with Completion-UI was to provide multiple UIs (and make it
easily and modularly extendable so people can add additional UIs if they
want), to have sensible default UI settings, but to let the user
enable/disable/reconfigure them as they wish via Customize.

Despite all this flexibility, and the very different types of UI it
supports (which are not all of the "display a list of completions"
variety), the UIs *are* treated uniformly in Completion-UI. If they
weren't, I couldn't support arbitrary third-party UIs. Take a look at
`completion-ui-register-interface' to see the API for all the different
UIs. All the bundled UIs are defined via exactly this same API, with no
special treatment.

Furthermore, the completion *sources* (backends) don't need to know
*anything* about how the UIs work. Indeed, they *never* invoke a specific
UI directly. They simply return a bunch of completions, and Completion-UI
presents them to the user via the various completion UIs, as customized
*by the user*.

> Ideally, we'd have a widget that's as useful as a menu when interacted
> with with a mouse, and works with keyboard, and doesn't interrupt the
> user's typing when displayed during idle completion.

Sure, it's called popup.el. If that's unavailable, Completion-UI has an
echo-area + hotkeys UI, which is a poor man's substitute for
popup.el-style menus. (You can also enable both at once if you like.)

> If I were to implement something like this for Company, it would mean 
> users who like this kind of menu would either have to forgo idle 
> completion, or see different kind of UIs whether completion is idle or 
> triggered manually. Which sounds weird, at least.

No, you'd provide sensible defaults that give a uniform UI, but let users
customize it to enable different kinds of UIs if they so desire.

The mouse-driven toolkit menu doesn't replace other (maybe better) UIs.
It complements it and/or is available as an alternative. I fail to see
the problem. If the user wants to use what you consider a weird
combination of UIs, I don't see any good reason to prevent them.

The Emacs way has always been to provide sensible default settings, but
then give users enough rope to shoot themselves in the foot if that's
what tickles their fancy(*).

(*) M-x reconfigure-rope-as-gun-and-shoot-foot


> > Sounds like adding it to `company-backends' is more analogous registering
> > it with `completion-ui-register-source'.
> 
> <*>
> 
> Not really. `company-backends' corresponds directly to 
> `auto-completion-source', 

Not quite. `company-backends' corresponds to the
`completion-ui-source-definitions' variable.

> only instead of one function we have a list of 
> them (which can also be nested one level deep, but that's an extra feature).

Sure, Completion-UI also allows a list of completion sources and selects
between them automagically. It even supports the same "extra feature" of
nesting sources one level deep (Combined source, in Completion-UI
terminology).

But the API for this isn't auto-completion-default-source(*). That merely
sets the default fall-back. The API for auto-selecting from a list of
sources is provided by `auto-completion-source-regexps',
`auto-completion-source-faces' and the `auto-completion-source-functions'
hook. (The former two are in fact variables used by functions in the
default `auto-completion-source-functions' definition.)

(*) I'm referring here to the git version of Completion-UI; the last
tarball release - which is woefully outdated - used a slightly different
API. (I've been too busy with real work for the last few years to get
around to rolling and testing a new tarball release; the git version
still gets updates periodically.)

> It's similar to `completion-at-point-functions' this way. First, each 
> backend knows whether it will be applicable at point (whether the 
> buffer's major mode fits, whether the syntax status at point fits, e.g. 
> some backends work in strings, others only work outside, etc). So 
> instead of calling an analog of `auto-completion-source', Company polls 
> all configured backends until it gets a response, then saves the name of 
> the backend thus found for the current completion invocation, and calls 
> it again for the list of completions. After that, it can call the 
> backend for calltips, candidate docs, et cetera.

Yup, Completion-UI does something very similar. Except that completion
source (backend) selection isn't directly tied to the backend. It's
configured independently. So if you need to, you can set up different
conditions under which a particular backend will be used in different
buffers, major-modes, minor-modes, or whatever.

Perhaps this is overkill. But as I keep coming back to, I designed
Completion-UI primarily to be a generic completion UI elisp library, not
as a user-level completion mode. And I do use some of this flexibility in
Predictive's sophisticated LaTeX support.


> > Customizing
> > `auto-completion-source' is something the user would do, not the Elisp
> > package coder.
> 
> Same with `company-backends', although we provide a reasonable default. 
> But the user can change the global list, of change its local values, for 
> example set it to a single-item value in some major mode hook.

Indeed, one of the best things about Company is all the backends that
come pre-configured. I'm sure that accounts for much of its
popularity. If I'd gone to the same effort of writing backends for
Completion-UI, probably more people would use it.

But the itch I was scratching wasn't to write a user-level completion
framework like Company. I just wanted a UI for Predictive, and figured I
might as well make it into a generic Elisp library so it could also be
used elsewhere. I'd kinda hoped people might pick it up and use it to
build something like Company or anything.el, so they could focus their
time on building a great user-level completion mode rather than
duplicating all the UI code, and instead help improve and contribute to
Completion-UI. But for various reasons that never happened.

Anyhow, I don't think company-mode is something that necessarily belongs
in a generic Emacs completion UI. At most, Emacs should include the
necessary definitions to hook the basic completion methods that come with
Emacs into the new generic UI (dabbrev, etags, minibuffer completion,
maybe a few others like elisp symbol completion...).

Indeed, I would have argued that we should *first* come up with a generic
Emacs completion UI and API *without* including any mechanism for
automatically selecting between completion sources. I would have proposed
to think later about when a source-selection mechanism belonged in Emacs,
or whether it's best left to ELPA packages like Company...

...except that `completion-at-point-functions' *already* defines an API
for selecting between completion sources. And it's already part of
Emacs. So I guess that decision has already been taken.

(I think there's still a discussion to be had about whether or not some
form of auto-completion-mode belongs in Emacs. I can see arguments both
ways.)


> > In fact, it sounds like the two APIs are rather similar:
> 
> > 2. Pass it to `completion-ui-register-source' (Completion-UI) or add it
> >     to `company-backends' (Company) to let them know about it.
> 
> Guess the main difference is that, since we definine new backends with 
> `defun', there's no way to update the list of available values, visible 
> to the `defcustom' interface.

You define new backends (completion functions) for Completion-UI with
`defun' in exactly the same way. You don't *have* to register a
completion function in order to use it with Completion-UI. If you want,
you can simply pass the completion function to `complete-in-buffer', much
like `completion-in-region'.

Registering a source just (1) makes life more convenient: you set all the
option values once when you define the completion source (backend), and
then you can refer to that source and all its associated options using a
single name (symbol). And (2) it makes the source available as an option
in the Customization interface, as you noted.

> It makes matters a bit worse for third-party backends, but not by much, 
> I think.

I agree, this is a minor difference, and Company doesn't lose much by it,
especially since it bundles almost all the backend definitions you're
ever likely to want.

On the other hand, it was more important for third-party completion
sources to be first-class citizens in Completion-UI, precisely *because*
I never put much effort into writing source definitions myself.

For a generic Emacs completion UI, it's obviously crucial that third-part
completion sources are first-class citizens, since almost all sources
will be third-party.


> > At some point in the distant past, you used to just set a variable to
> > tell Completion-UI about a new completion function. It became a macro
> > call when I made Completion-UI more generic, and there needed to be a way
> > of optionally supplying additional information about what the completion
> > function does (e.g. if it does something other than standard prefix
> > completion), and of defining optional call-back functions, etc.
> 
> We solve this problem by requiring backend functions to take the type of 
> question it's going to answer as the first argument. Function calls are 
> fast enough for our purpose, and this makes definitions quite succinct, 
> especially for simple backends.

Simple backends have trivial `completion-ui-register-source' calls in
Completion-UI, too. All you need is:

  (completion-ui-register-source my-completion-function)

The macro figures out everything it needs to know from the function
definition - the beauty of lisp macros!

It's shorter than typing:

  (add-hook 'completion-at-point-functions 'my-new-completion-function)

;-)

The optional macro arguments only come into play if you're doing
something more sophisticated (like extending the default completion
widgets, or overriding the default method of determining the text around
point that's to be completed, etc.)

> This pattern is used in many places in Emacs core, too.

In principle, this sounds like a perfectly reasonable API choice for
simple settings.

Though I don't completely get how it works here. What if a particular
completion function can't answer some types of question? E.g. maybe it
can expand prefixes, but it can't find regexp matches. What does the
completion function do if it's then asked for a regexp match? Or does the
backend answer "I can't answer this type of question" and Company moves
onto the next backend in the list?

However, this API is not so convenient if you want to pass more complex
settings, such as extending the UI for a particular source. You'd either
have to pass the same list of hook functions, keymaps etc. every time you
call the function that invokes the completion UI for the source. Or you'd
have to let-bind the same set of variables every time. Which would be a
pain.

Surely more convenient to have a way of telling Emacs: "use this set of
options, hook functions, keymaps, etc. every time I use this completion
source to complete something". Naturally, you can do this via
configuration variables that store
alists/plists/insert-your-favourite-associative-array-here. And guess
what? Most of what the `completion-ui-register-source' macro does is add
a bunch of elements to the appropriate plists for you. You can do this
manually if you prefer, but the macro call is so much more convenient.

This pattern is used in many places in Emacs core, too: `defcustom',
`defstruct', `defun*'... (Perhaps I should have called the macro
`defcompletion' instead of `completion-ui-register-source' :)

Also, the "pass the type of completion as the first argument" API means
you have a hard-coded set of "types" of completion. As usual, since it
was intended to be a generic completion Elisp library and not a
fully-fledged user-level completion framework, Completion-UI doesn't
impose any such restriction, but tries to be as flexible and general as
possible. I'm not saying this is necessarily a good thing; it could well
be that this flexibility is overkill.



Thanks for taking the time to explain and compare Company features. It's
very helpful in figuring out what a generic Emacs completion API should
look like, at least to me.

I increasingly have the impression that the Company and Completion-UI
feature sets and APIs are in fact functionally very similar, modulo API
details and a useful feature or two here and there on each side. (Leaving
aside the massive collection of Company backend definitions, which
Completion-UI completely(!) lacks.)

Which bodes well for coming up with a generic Emacs completion UI API and
code. If we've converged on a broadly similar set of features and APIs
for the UI parts of Completion-UI and Company, there's a reasonable
chance they're the features people will want from a generic Emacs API.

Best,
Toby
-- 
Dr T. S. Cubitt
Royal Society University Research Fellow
and Fellow of Churchill College, Cambridge
Centre for Quantum Information
DAMTP, University of Cambridge

email: tsc25@cantab.net
web:   www.dr-qubit.org