Re: Emacs completion matches selection UI

[Toby Cubitt <tsc25@cantab.net>]

On Mon, Jan 06, 2014 at 11:36:49AM +0400, Dmitry Gutov wrote:
> On 06.01.2014 06:39, Toby Cubitt wrote:
> > Completion-UI stores state by marking the location of the in-progress
> > completion with an overlay, and storing data in overlay properties. You
> > could alternatively use text properties, as you mention later. But I find
> > overlays a better match for this kind of thing.
> 
> I mentioned text properties in the context of merged backends, so the 
> properties would be on the candidate strings, not in the buffer. That's 
> not interchangeable with using an overlay.

OK, I understand what you meant now.

>  > (Presumably Company does something similar.)
> 
> It does store some data, as an implementation detail, but in no way that 
> data is sufficient for a front-end to handle a sequence of commands on 
> its own.

It's still Completion-UI that handles a sequence of commands, not the
backend. Completion-UI stores enough data about the currently active
completion process in the overlay properties to know what to do when the
user calls a follow-up command.

Sounds like Company stores that data in local variables, whereas I chose
to store it in overlay properties.

> Like I said, the front-ends use the buffer-local vars. Multiple pending
> completions have never been a goal so far.

It's a nice freebie you get from storing the data in overlays instead of
buffer-local variables, but I don't think it's all that important.

> >> I'd still like to a natural way to extend such widgets with new
> >> commands. Passing a keymap as argument, or rebinding a dynamic variable
> >> that points to keymap works, I guess, but it's not very nice.
> >
> > Well, this is only meaningful for some of the widgets. Indeed, "widgets"
> > is mislead; the completion UIs in Completion-UI aren't necessarily
> > widgets at all. For example, there are no "commands" at all in the
> > dynamic completion UI, nor is it a "widget" in any normal sense of the
> > word.
> 
> I'd call "widgets" any UIs that encapsulate data (store it in the 
> associated overlay, for example) and define interaction commands (e.g. 
> have an overlay keymap).

OK.

> > The UIs absolutely cannot be treated uniformly, because not all of them
> > are "display a list of completions"-style UIs. E.g. the "dynamic
> > completion" UI is a completely different beast.
> 
> That might be unfortunate from the API standpoint, making it more complex.

Have you looked at the Completion-UI API for defining a new widget? It's
uniform across all the widgets. Has to be, otherwise there would be no
way of allowing it to be extended with third-party widgets (see my
comment below).

What I meant was that the UIs themselves can act very differently. One
shouldn't make assumptions that they all behave in a similar way. The API
itself is uniform.

> > Even amongst the "display a list of completions"-style UIs, the
> > mouse-driven toolkit menus cannot be used in the terminal, nor can pop-up
> > frames, nor can tooltips. That doesn't mean one should not provide these
> > interfaces at all as options. It just means they probably aren't good
> > default options.
> 
> The problem of "cannot be used in the terminal", at least, can be solved 
> by doing a check at runtime once, and picking between a graphical and an 
> overlay-based interface. As long as they provide similar interaction, 
> I'd call that "treated uniformly".
> 
> >> Not really. `company-backends' corresponds directly to
> >> `auto-completion-source',
> >
> > Not quite. `company-backends' corresponds to the
> > `completion-ui-source-definitions' variable.
> 
> I'm inclined to stand by my previous assessment: nowhere in the code I 
> see you iterate over `completion-ui-source-definitions' at runtime.

You're insisting on trying to match the Completion-UI API one-to-one with
the Company API. But they don't match one-to-one, because they divide
things up slightly differently. Nonetheless, it's definitely wrong to
claim that company-backends corresponds to
auto-completion-default-source. The best comparison one can make is that
company-backends loosely corresponds to a mixture of
completion-ui-source-definitions and auto-completion-source-functions.

> In most places, it's used as a second argument to `assq', so it acts as
> a "source registry" of sorts, and we don't have a direct correspondence
> to that in Company.

That's because Company conflates the backend definitions with the
mechanism for auto-selecting a backend. Completion-UI keeps those
separate. It makes sense to me to separate these, for two reasons.
(1) the backend definitions will typically be written by elisp package
authors; the auto-selection configuration is something users will want to
customize. (2) I see no reason to limit each source of completions
(dabbrevs, etags...) to a single choice of when to use it.

By combining both the choice of whether to use a backend (nil or non-null
return value) and finding the list of completions (return value) into one
function, you force all the selection logic to be lumped in together with
the completion logic. If you want to use the same source of completions
(dabbrevs, etags...) in multiple different contexts, you either have to
code *all* the selection logic you'll ever need into the one backend, or
have multiple backends for the same source of completions.

I find it logically much cleaner to separate the function for completing
a string using a given source (dabbrevs, etags...), from the function for
selecting which source of completions to use.

> >> only instead of one function we have a list of them (which can also
> >> be nested one level deep, but that's an extra feature).
> >
> > Sure, Completion-UI also allows a list of completion sources and selects
> > between them automagically. It even supports the same "extra feature" of
> > nesting sources one level deep (Combined source, in Completion-UI
> > terminology).
> 
> I couldn't find the word "combined" in completion-ui.el. Could you point 
> at where it's used?

The `completion-ui-combining-complete' utility function, and the
pre-defined Combine completion source +
`completion-ui-combine-sources-alist' customization option.

Remember: Completion-UI, aimed primarily at elisp coders, not
users. `completion-ui-combining-complete' lets you write a completion
function that combines multiple sources together. The pre-defined Combine
completion source + `completion-ui-combine-sources-alist' customization
option mainly serves as a demonstration.

This is *definitely* not as simple or user-friendly as Company. Partly
because I think of combining multiple completion sources as something
that belongs in a user-level completion framework (like Company) built on
top of Completion-UI. Partly because my use case (Predictive) needed a
very flexible way of switching *between* different completion sources,
but for algorithmic reasons combining multiple sources needed to be done
within Predictive, not within Completion-UI.

I hate to say this ;-), but maybe `completion-at-point-functions' got
this right. By having one function (the hook function itself) that deals
with selecting a completion source, and a separate function (the hook
function's return value) for actually doing the completion. This both
keeps the two jobs cleanly separated (as in Completion-UI), *and* keeps
the two together in the same list.

Personally I'd find an alist with elements of the form
(collection . test) a slightly clearer API than a straight hook. But I
can see why using a hook appeals from a core Emacs perspective.


> > But the API for this isn't auto-completion-default-source(*). That merely
> > sets the default fall-back. The API for auto-selecting from a list of
> > sources is provided by `auto-completion-source-regexps',
> > `auto-completion-source-faces' and the `auto-completion-source-functions'
> > hook. (The former two are in fact variables used by functions in the
> > default `auto-completion-source-functions' definition.)
> 
> `auto-completion-source-functions' also could be thought of as similar 
> to `company-backends' because both contain logic of picking a suitable 
> source (the latter by the virtue of backends themselves knowing when 
> they're suitable), but the list of default detection functions (based on 
> text properties, overlays, faces at point) looks more limiting to me.

Why on earth is it limiting?

The detection functions (based on regexps, text properties, overlays,
faces at point) are provided for convenience, so you don't have to write
your own `auto-completion-source-functions' function and duplicate a
bunch of code if all you want to do is select a source based on regexp
matches, text properties, overlay properties, or faces. (Together with
the standard Emacs mechanisms for making this buffer-local, this already
covers many of the common cases.)

If you need a more complex selection logic, write a new selection
function which does whatever you want, and add it to
`auto-completion-source-functions'. (And remove the default regexp, face,
text-property etc. detection functions from the hook if you don't want
them.)

I really don't see how this is any different to writing a new Company
backend with the particular selection logic you require, and adding it to
company-backends. Except that you *only* need to code the new selection
function and add it to the auto-completion-source-functions hook; you
don't need to duplicate the code for finding the completions if all you
want to do is use an existing completion source in a new context.

> Granted, the first two source functions serve to support the explicit 
> overriding of the used source (feature not presend in Company, but I've 
> never seen a request for this so far),

I make heavy use of this in Predictive's sophisticated LaTeX and Texinfo
support. If I were to use Company, I would have to move all that logic
into Predictive, because Company doesn't provide it for me.

> but the second two offer only choices based on regexp or face at
> point. Which is not terrible, but more limiting than Company's
> free-form expectation that (backend-function 'prefix) returns nil when
> it's not applicable at point in the current buffer.

I don't get this at all. How is adding a free-form function to the
`auto-completion-source-functions' hook any less flexible than adding a
new backend to company-backends? Except that if you simply want to choose
the completion source based on a regexp match, face, text-property or
overlay-property, then you don't even have to write any new code (as you
would in Company) but can get away with simply configuring a variable.

I think you're trying to view Completion-UI through Company
spectacles. Some Company features don't exist of have limited support in
Completion-UI (e.g. combining sources), some Completion-UI features don't
exist or have limited support in Company (e.g. adding new custom UIs),
and some features exist in both but work differently. But Completion-UI's
programmatic selection of completion sources is just as powerful as
Company's.


> You could say that I can add another function or several to 
> `auto-completion-source-functions', but they won't be able to do 
> anything smart with third-party sources, I'll have to take care about 
> each source I might want to use, separately, in those functions.

Why on earth not?! They can do whatever they want - they're functions!
(OK, anything short of solving the Halting Problem ;-)

To belabour the point, Completion-UI separates the task of completion
source selection from the task of finding completions. To add the
equivalent of a new Company backend to Completion-UI, you would typically
add a new completion source, and add a new
`auto-completion-source-functions' hook function. (Unless one of the
pre-defined selection functions was sufficient, in which case you could
get away with just adding an entry to the appropriate customization
variable.)


> > Yup, Completion-UI does something very similar. Except that completion
> > source (backend) selection isn't directly tied to the backend. It's
> > configured independently. So if you need to, you can set up different
> > conditions under which a particular backend will be used in different
> > buffers, major-modes, minor-modes, or whatever.
> 
> Note that there's nothing stopping you from defining multiple conditions 
> for when a particular Company backend can be used. Most do that already: 
> the popular predicates are "does the major mode match this list", "is a 
> particular package loaded", "is the parse state at point in-string or 
> outside", "is this buffer visiting a file", "is a particular program 
> present in the system", "is there a file with a given name in a parent 
> directory" and "is point inside a symbol, after a symbol or outside of 
> any symbols".

Sure, but you have to duplicate the same logic in each backend function,
and you have to either lump *all* the selection logic you'll ever need
into one function, or have multiple backends for the same completion
source.

Keeping them separate is, to my mind, both cleaner and more
flexible. (It's also the route c-a-p-f went.)

> > Indeed, I would have argued that we should *first* come up with a generic
> > Emacs completion UI and API *without* including any mechanism for
> > automatically selecting between completion sources.
> 
> In other words, a programmatic widget API. This has come up already in 
> the discussion with Ted.

Yes, if what you mean is an API for defining completion UI widgets,
together with a bundle of standard widgets. As far as new widget authors
are concerned, it would be a completion widget API.

But let's be clear: as far as *completion* package authors are concerned,
it's not a widget API at all. A completion package won't call the tooltip
widget, or the popup.el widget, or the dynamic completion widget. It will
call something like `completion-at-point'.

At least, that's how I was imagining it.

> > Registering a source just (1) makes life more convenient: you set all the
> > option values once when you define the completion source (backend), and
> > then you can refer to that source and all its associated options using a
> > single name (symbol). And (2) it makes the source available as an option
> > in the Customization interface, as you noted.
> 
> I would think that, since Completion-UI is less user-facing and more 
> about writing Elisp to wire sources and UIs, the Customize interface is 
> considerably less important.

I strongly disagree. Almost all of the Completion-UI customization
options are about letting the user customize the UI. Those are both
important, and definitely belong together with an Emacs generic
completion UI.

The programmatic completion source (backend) selection interface would be
less important...except that it's already there in Emacs in the form of
`completion-at-point-functions'. We're probably stuck with c-a-p-f for
now, so any generic Emacs completion UI is going to have to integrate
with it somehow.

There are really two APIs here:

1. An API for defining new completion widgets, a customization interface
   to let the user configure them, and code to hook the new UIs into
   `completion-at-point', `completion-in-region', and eventually
   minibuffer completion and the rest.

2. An API for defining new completion sources and new source selection
   logic, and a customization interface to let the user configure
   them. This API already exists (c-a-p-f et al.), and the new UI code
   will have to integrate with it.


> >> It makes matters a bit worse for third-party backends, but not by much,
> >> I think.
> >
> > I agree, this is a minor difference, and Company doesn't lose much by it,
> > especially since it bundles almost all the backend definitions you're
> > ever likely to want.
> 
> We also document how to use `company-backends' and, for third-party 
> packages, how to add a specific backend to it. But it's a simple data 
> structure, so for users with some experience just knowing backend 
> function names is sufficient.

Sure, that's equally true of `completion-at-point-functions' and
`completion-ui-register-source'. (Except that new backends are
first-class citizens in Completion-UI, and show up automatically as
options in the completion UI customization interface where
appropriate. This would be harder to replicate with
`completion-at-point-functions', but I suspect with enough Customize
macro trickery it would still be possible with the existing Emacs
completion backend API.)

> > The optional macro arguments only come into play if you're doing
> > something more sophisticated (like extending the default completion
> > widgets, or overriding the default method of determining the text
> > around point that's to be completed, etc.)
> 
> Most Company backends at least have to check that the buffer is in 
> matching major mode.

Whereas in Completion-UI, you just use the standard Emacs mechanisms of
setting buffer-local variables from you major-mode hook, or using
`add-hook' with the LOCAL flag set to add a function to the
auto-completion-source-functions hook.

I prefer to use the standard, tried-and-tested Emacs mechanisms where
possible (major-mode hooks), rather than creating an entirely new
package-specific way of configuring things for different major-modes. (I
don't always stick to this myself, though I probably should ;-)


> >> This pattern is used in many places in Emacs core, too.
> >
> > In principle, this sounds like a perfectly reasonable API choice for
> > simple settings.
> >
> > Though I don't completely get how it works here. What if a particular
> > completion function can't answer some types of question?
> 
> Then we use the default setting, behavior, etc. It is somewhat limiting 
> (we can't discern whether a backend does not know about a question or it 
> consciously returns nil), but so far it's worked well nevertheless.

Another reason to separate source selection logic from finding
completions logic.

In any case, this discussion is somewhat moot. It's pretty clear we're
stuck with the c-a-p-f API for core Emacs, and that API already separates
source selection from completion.

>  > E.g. maybe it can expand prefixes, but it can't find regexp
> > matches. What does the completion function do if it's then asked for
> > a regexp match? Or does the backend answer "I can't answer this type
> > of question" and Company moves onto the next backend in the list?
> 
> We don't have support for regexp matches, but "can't answer" + "move on" 
> is a reasonable mechanism to support them in select backends.

So every backend has to look at what's being asked of it every time it's
invoked, and return nil if it's asked to do something it doesn't support?
And also check the major-mode and see if it's appropriate every time it's
invoked, even though the major-mode is very unlikely to change (and runs
a major-mode hook when it does)?

This sounds rather inefficient. I'm sure it's fast enough for Company,
but I'm skeptical it would be fast enough for predictive-mode's
auto-completion-mode (where tenths of a second matter).


> Or, to approach it another way, it may be a backend's choice to 
> interpret the prefix (or, more generally, the completion string, if it's 
> allowed to span after the point) as a regexp. For example, we have this 
> as a feature request now: 
> https://github.com/company-mode/company-mode/issues/45#issuecomment-31564029

>From experience with Completion-UI (which already supports
pattern-expansion-style completion), you have to slightly change the way
the UI works depending on whether you're doing prefix completion or
pattern expansion. The Completion-UI widgets already support this.

> Aside from handling the prefix->candidate replacement more carefully, 
> the only thing we'll have to worry about, I believe, is that the 
> front-ends won't be so sure that candidate starts with "prefix". For 
> `company-pseudo-tooltip', that means no highlighting of the "common" 
> part (or using a more complex algorithm), and it could mean different 
> things for `company-preview-if-just-one-frontend'. For now, I'm inclined 
> to just refrain from using it if the sole candidate does not start with 
> "prefix".

Non-prefix completion is already fully supported by all the Completion-UI
widgets and interfaces.

> > However, this API is not so convenient if you want to pass more complex
> > settings, such as extending the UI for a particular source. You'd either
> > have to pass the same list of hook functions, keymaps etc. every time you
> > call the function that invokes the completion UI for the source. Or you'd
> > have to let-bind the same set of variables every time. Which would be a
> > pain.
> 
> Sorry, I don't understand. Why would you extend a UI for some source? 
> Sources provide data, how can they have associated keymaps?

Maybe an example will help. When completing a LaTeX command name,
Predictive uses a custom menu and completion browser specifically
tailored to LaTeX commands. Similarly, it uses differently tailored
versions for completing LaTeX environment names, colour names, etc. etc.

It's not that the source returns a keymap or widget-building function
each time it's called. The keymaps or functions that tweak the UI for a
specific completion source are configured when the source is registered
with Completion-UI. The completion function itself just returns
completions (and associated data).

Hope that makes it clearer.


> > This pattern is used in many places in Emacs core, too: `defcustom',
> > `defstruct', `defun*'... (Perhaps I should have called the macro
> > `defcompletion' instead of `completion-ui-register-source' :)
> 
> Sure, Lisps have a long history of using and promoting macros, but 
> there's also a general recommendation to not write a new macro unless 
> you'll really really benefit from it. Maybe you do.

Yes, I really, really benefit from using a macro.

It would be virtually impossible to do everything
`completion-ui-register-source' does without using a macro. Inspecting
the completion function's definition to determine how to call it?
Automatically updating the Customization definitions? Defining a new
source-specific function? Automatically wrapping the completion function
to make it predictive (strictly speaking, this part lives in Predictive,
not Completion-UI)?

I don't even want to contemplate how to do those without lisp macros.


> > Also, the "pass the type of completion as the first argument" API means
> > you have a hard-coded set of "types" of completion. As usual, since it
> > was intended to be a generic completion Elisp library and not a
> > fully-fledged user-level completion framework, Completion-UI doesn't
> > impose any such restriction, but tries to be as flexible and general as
> > possible. I'm not saying this is necessarily a good thing; it could well
> > be that this flexibility is overkill.
> 
> See above. We only have prefix completion for now, so the first argument 
> definitely doesn't choose the type. See the list of possible values in 
> the `company-backends' docstring.

Thanks, makes sense now.

> > Which bodes well for coming up with a generic Emacs completion UI API and
> > code. If we've converged on a broadly similar set of features and APIs
> > for the UI parts of Completion-UI and Company, there's a reasonable
> > chance they're the features people will want from a generic Emacs API.
> 
> Yes, probably.

To summarise:

1. People want an Emacs completion UI widget API, mostly so they can have
   a more "modern" completion UI than the *Completions* buffer. As far as
   I know, Completion-UI is the only package that provides an API and for
   defining new completion widgets, and code to bind them all
   together. It also already provides a fairly large collection of UI
   widgets.

   Maybe we should look at the Completion-UI API for this
   (`completion-ui-register-interface'), and think about how it would
   need to be changed for core Emacs? I'm not wedded to the existing API
   if someone proposes something better or has good arguments for
   changing it.

2. We're probably stuck with c-a-p-f et al. for the completion source and
   source-selection API. How would this need to be extended to integrate
   it with a new Emacs completion UI? How could it be extended to better
   support Company and others?

3. The most popular and useful "list-the-available-completions" UI is
   popup.el. Is there any chance of getting copyright assignment for
   this? Or will we be forced to code something equivalent from scratch?


Best,
Toby
-- 
Dr T. S. Cubitt
Royal Society University Research Fellow
and Fellow of Churchill College, Cambridge
Centre for Quantum Information
DAMTP, University of Cambridge

email: tsc25@cantab.net
web:   www.dr-qubit.org