Re: Emacs completion matches selection UI

[Toby Cubitt <tsc25@cantab.net>]

On Tue, Jan 07, 2014 at 04:17:03AM +0400, Dmitry Gutov wrote:
> On 06.01.2014 19:51, Toby Cubitt wrote:
> > Have you looked at the Completion-UI API for defining a new widget? It's
> > uniform across all the widgets. Has to be, otherwise there would be no
> > way of allowing it to be extended with third-party widgets (see my
> > comment below).
> 
> Looking at it now. Seems reasonable, but the data is somewhat coupled to 
> the rest of Completion-UI.

Sure, the details will need consideration.

> Examples:
> 1. `prefix-replaced' and `common-substring' both seem to be tailored to 
> the dynamic interface.

Not at all. `prefix-replaced' tells you that the buffer substring that is
being completed has already been replaced with new text in the
buffer. This is important for non-prefix completion (e.g. pattern
matches).

Any UI that modifies the buffer text will almost certainly need to set
`prefix-replaced'; it just so happens that the only such UI that comes
with Completion-UI is the dynamic interface. Any third-party UI that
modifies the buffer text will almost certainly need to set/check this
property too. And the core Completion-UI code that ties all the UIs
together makes heavy use of the `prefix-replaced' property. So even in
Completion-UI it's not only used by the dynamic interface.

Similarly, `common-substring' demarcates the longest common substring if
completion text has been inserted in the buffer, which is information a
third-party UIs might very well want to make use of.

All the information passed to UIs is there for a UI widget to use or
ignore, as needed. I was reasonably careful to limit the documented
overlay properties to information that might be generically useful, not
stuff that's clearly only ever going to be used by one specific UI and no
other.

The properties documented in the `completion-ui-register-interface'
docstring are those that have a standard meaning in Completion-UI. Note
that nothing prevents a particular UI widget from storing its own data in
its own widget-specific overlay properties if so desired. Indeed, that's
a very good way of storing widget-specific data that persists until the
completion process is finished.

> 2. `completion-source' seems to have to refer to a Completion-UI source, 
> right?

> So how would one use an UI defined as such, in a different package? 
> Would Company define itself as a new source? Or add a source per each 
> backend?

Depends on what the generic Emacs API for specifying completion sources
ends up looking like. If we end up using c-a-p-f, `completion-source'
will need to store something that identifies the c-a-p-f entry that
returned the completion candidates for this completion process.

> > The best comparison one can make is that
> > company-backends loosely corresponds to a mixture of
> > completion-ui-source-definitions and auto-completion-source-functions.
> 
> The latter: yes, the former: no. It's never used the same way as you use 
> `completion-ui-source-definitions'.

I said "loosely corresponds", and I stand by that. One of the things a
`company-backends' entry does is to define when a given completion source
should be used. In Completion-UI, that part of company-backends' role is
performed by auto-completion-source-functions.


> >> In most places, it's used as a second argument to `assq', so it acts as
> >> a "source registry" of sorts, and we don't have a direct correspondence
> >> to that in Company.
> >
> > That's because Company conflates the backend definitions with the
> > mechanism for auto-selecting a backend. Completion-UI keeps those
> > separate. It makes sense to me to separate these, for two reasons.
> > (1) the backend definitions will typically be written by elisp package
> > authors; the auto-selection configuration is something users will want to
> > customize.
> 
> IME, it makes sense in practice because there usually is a set of 
> conditions when a backend is applicable, and there's no sense for the 
> user to write them out, when the package author can anticipate them.

Could well be. But you seem to be arguing both ways. You also argue that
the c-a-p-f API is bad because it's opaque and hard for users to
configure. If backend definitions and selection logic are always supplied
by package authors, then the fact that c-a-p-f is opaque isn't so
significant.

> > (2) I see no reason to limit each source of completions
> > (dabbrevs, etags...) to a single choice of when to use it.
> 
> The package author can always add one or several new of customization 
> variables when the users request it to make the logic more flexible. 
> Most of the time, it's usually just the list of major modes the backend 
> applies to, if it's major mode-sensitive.
> 
> Please take a look at `company-etags' and `company-dabbrev' and see if 
> you can point out the situations when the user might find the `prefix' 
> code of either too limiting.

dabbrev and etags are both sources that, if they're useful in a given
buffer, they're useful in the entire buffer.

Consider the LaTeX math mode source, and the LaTeX environment name
source, and the LaTeX preamble source, etc. Trying to code the selection
logic for all of these using the Company API looks very awkward to me,
compared to the simplicity of setting a few buffer-local variables in
Completion-UI.

Sure, I could move that logic into Predictive itself, and have a single
Predictive LaTeX backend. But that serves to demonstrate that the API
isn't flexible enough to let me do what I want easily. Other markup
languages and programming languages make similar demands on the API.

> > By combining both the choice of whether to use a backend (nil or non-null
> > return value) and finding the list of completions (return value) into one
> > function, you force all the selection logic to be lumped in together with
> > the completion logic. If you want to use the same source of completions
> > (dabbrevs, etags...) in multiple different contexts, you either have to
> > code *all* the selection logic you'll ever need into the one backend, or
> > have multiple backends for the same source of completions.
> 
> Or you can define new backends that would do some common checks in 
> `prefix' (maybe calling an extracted function with common code) and 
> simply delegate all other actions to the respective base backend. 
> Implementing this is trivial.

And once you've finished doing this, and factored out common selection
mechanisms like regexps, faces and text properties into utilities
functions...you'll have reimplemented something closer the Completion-UI
API or c-a-p-f :)

> > I find it logically much cleaner to separate the function for completing
> > a string using a given source (dabbrevs, etags...), from the function for
> > selecting which source of completions to use.
> 
> I understand the principle, really. But the more one "cleanly separates" 
> code, the harder it can be sometimes to read it, to get the full picture.

Indeed, which is why I listed grouping the completion and selection logic
into one place as one of the things I liked about Company's API.

Perhaps the cleanest and most flexible solution would be to have a list
with entries of the form (TEST-FUNCTION . COMPLETION-FUNCTION),
COMPLETION-FUNCTION is used if TEST-FUNCTION returns non-nil. And then
supply a bunch of standard utility functions for use as TEST-FUNCTION's,
for testing regexp matches, faces, text-properties, etc.

This keeps everything together, makes it trivial to avoid code
duplication, and cleanly separates the selection logic and completion
machinery.

> > I hate to say this ;-), but maybe `completion-at-point-functions' got
> > this right. By having one function (the hook function itself) that deals
> > with selecting a completion source, and a separate function (the hook
> > function's return value) for actually doing the completion. This both
> > keeps the two jobs cleanly separated (as in Completion-UI), *and* keeps
> > the two together in the same list.
> 
> It makes certain amount of sense, although it looks like it could make 
> creating a "merged" completion function more difficult.

I doubt it'll be insurmountable. Also, merged completion functions are a
rather advanced feature that may not belong in core Emacs anyway (though
it would be good if the API supported them without ugly hacks).

> We'll see.

Indeed, we'll see. Whilst I'd be happy to see the somewhat complicated
and opaque c-a-p-f API replaced with something cleaner and simpler, I
don't see us winning that argument.

> But again, this separation also makes things more opaque for the user.
> 
> >> `auto-completion-source-functions' also could be thought of as similar
> >> to `company-backends' because both contain logic of picking a suitable
> >> source (the latter by the virtue of backends themselves knowing when
> >> they're suitable), but the list of default detection functions (based on
> >> text properties, overlays, faces at point) looks more limiting to me.
> >
> > Why on earth is it limiting?
> 
> Note the word "default" above. This specific list is limiting.

It's just a reasonable default value, nothing more. It's was never
supposed to be exhaustive. That's why the hook exists: to have functions
added to it.

> But the general approach, while flexible on its surface, complicates 
> things if I intend to use any existing sources, written by third 
> parties. Because their authors are unlikely to have anticipated the 
> logic I'll add in my custom predicate function and to have written any 
> code in their packages I might use. Or, at least, that's considerably 
> less likely.

I don't get your argument here. You have to wrap the third-party
completion function in Company in order to code the appropriate backend
selection logic. In Completion-UI, you put that code in an
`auto-completion-source-function' instead, and probably don't need to
write a wrapper function around the third-party completion
function. Doesn't strike me that the Company way is any less effort; if
anything, it looks like more effort.

I tend to agree that configuring the source selection logic together with
the source completion function is easier to get your head around, though.

> Granted, if I only use Completion-UI for its interfaces and never for 
> any existing sources, it's not going to be a problem.

If you can convince Stefan to let us replace c-a-p-f with a simpler API
that does the same thing, I'm all for it. It would be straightforward to
modify Completion-UI to use whatever API we settle on for specifying
completion sources and selection logic.

I still think we'll lose this one, and be stuck with c-a-p-f, but I'd
love to be proven wrong.

> > I really don't see how this is any different to writing a new Company
> > backend with the particular selection logic you require, and adding it to
> > company-backends. Except that you *only* need to code the new selection
> > function and add it to the auto-completion-source-functions hook; you
> > don't need to duplicate the code for finding the completions if all you
> > want to do is use an existing completion source in a new context.
> 
> Like mentioned above, delegating the search for completions to an 
> existing backend is trivial. These are functions, and as such they are 
> stateless. Just call (other-backend 'candidates current-prefix).

But this results is convoluted and somewhat confusing code. If the
selection logic and completion function were separate, you wouldn't need
to use such hacks.

> > I don't get this at all. How is adding a free-form function to the
> > `auto-completion-source-functions' hook any less flexible than adding a
> > new backend to company-backends? Except that if you simply want to choose
> > the completion source based on a regexp match, face, text-property or
> > overlay-property, then you don't even have to write any new code (as you
> > would in Company) but can get away with simply configuring a variable.
> 
> Problem is, I can never get away with doing just one check. At the very 
> least, I need to check both for a regexp (or thing-at-point) match, 
> *and* the face (or, better yes, `syntax-ppss' return value). Not to 
> mention the major mode, but this check could be conceivably replaced by 
> a combination of `add-hook' and a buffer-local `company-backends' value.

You needed to code this selection logic yourself in any case. So you
could equally well have put that code into an
auto-completion-source-functions function instead of a company-backend
wrapper function.

I'll say this once more and then shut up because I've already said it
umpteen times now: `auto-completion-regexp-source' et al. are just simple
and convenient default entries in the `auto-completion-source-functions'
hook. For anything more complicated, add another function to that hook. I
don't understand why you're harping on about the limitations of the
default hook entries, when you can add whatever function you like to the
hook.

> > I think you're trying to view Completion-UI through Company
> > spectacles. Some Company features don't exist of have limited support in
> > Completion-UI (e.g. combining sources), some Completion-UI features don't
> > exist or have limited support in Company (e.g. adding new custom UIs),
> 
> Adding new UIs is simple, as long as they conform to the 
> `company-frontends' interface. I'm not completely sure yet if they're 
> decidedly "worse" in some sense than Completion-UI interfaces.

Sorry, I missed the `company-frontends' interface. I'll take a look.

> >> You could say that I can add another function or several to
> >> `auto-completion-source-functions', but they won't be able to do
> >> anything smart with third-party sources, I'll have to take care about
> >> each source I might want to use, separately, in those functions.
> >
> > Why on earth not?! They can do whatever they want - they're functions!
> > (OK, anything short of solving the Halting Problem ;-)
> 
> "Anything smart" in this context would be to use any metadata the 
> authors of third-party sources could have supplied if they were 
> encourages to do so.

They could equally well be encouraged to supply a c-a-p-f function (which
is purely selection logic) along with their source. Since c-a-p-f is
already in Emacs, this is likely to be more successful. But my bet is
you'll mostly be dealing with "dumb" backends for a long time yet, and
will have to continue coding the selection logic yourself.

> > Sure, but you have to duplicate the same logic in each backend function,
> > and you have to either lump *all* the selection logic you'll ever need
> > into one function, or have multiple backends for the same completion
> > source.
> 
> That depends on what one means by the "same logic". Any logically 
> different predicate can be extracted into a separate function. Take a 
> look at the existing Company backends, the `prefix' section are usually 
> quite succinct.

Indeed, though this is starting to look a lot like defining some standard
hook functions for auto-completion-source-functions. As you make the
company-backends API more flexible and convenient, you'll increasingly
find yourself reimplementing equivalent functionality to that of c-a-p-f
and the Completion-UI API. If were to start simplifying the Completion-UI
API or c-a-p-f, it'll increasingly look more like the Company API. Maybe
the sweet spot is in the middle?

> > Yes, if what you mean is an API for defining completion UI widgets,
> > together with a bundle of standard widgets. As far as new widget authors
> > are concerned, it would be a completion widget API.
> >
> > But let's be clear: as far as *completion* package authors are concerned,
> > it's not a widget API at all. A completion package won't call the tooltip
> > widget, or the popup.el widget, or the dynamic completion widget. It will
> > call something like `completion-at-point'.
> 
> Actually, I was thinking about the former option. Let's define widgets 
> with an API in the usual sense, so it can be used by both 
> `completion-at-point' and external packages. We'll need this kind of API 
> either way, in order to be able to write new widgets.

I'm confused. `completion-at-point' will never use an API for *defining*
new completion UI widgets (`completion-ui-register-interface'). It will
need to be modified to *invoke* the new UI widgets.

Copying and adapting some of the code from `complete-in-buffer' into
`completion-in-region' would suffice for that. Then
`completion-in-region' would remain the generic Emacs API for displaying
and selecting completion candidates (only now it would display them in a
nicer interface).


> > There are really two APIs here:
> >
> > 1. An API for defining new completion widgets, a customization interface
> >     to let the user configure them, and code to hook the new UIs into
> >     `completion-at-point', `completion-in-region', and eventually
> >     minibuffer completion and the rest.
> >
> > 2. An API for defining new completion sources and new source selection
> >     logic, and a customization interface to let the user configure
> >     them. This API already exists (c-a-p-f et al.), and the new UI code
> >     will have to integrate with it.
> 
> Yes, but see above. Using 1. from Company would be the current next step 
> toward integration, as I see it.

Really? Why should we use the Company UI widget API and interfaces in
Emacs, when the API is less flexible and less customizable than that the
Completion-UI widgetAPI, and it implements fewer widgets?

Did you really mean that you wanted to throw Completion-UI in the
garbage, rewrite Company yourself to be suitable as a generic Emacs
completion API, rewrite/port the missing UIs and features, and persuade
people to merge it into Emacs core? (If so, great! Let me know when
you're done, and then I can stop maintaining Completion-UI and switch
Predictive over to the new interface :-)

Probably I misunderstood what you wrote.

I'm happy to help when I have time, and to contribute Completion-UI code
and experience. But I'm equally happy for someone else to do all the
work. My interest is in seeing a completion UI get included in Emacs
which is powerful enough to replace Completion-UI in Predictive. Then I
can stop maintaining Completion-UI on my own, and instead contribute
patches to the Emacs UI.

I only really have one strong requirement: if some form of
auto-completion mode gets included in the Emacs completion UI, I will
argue hard for it to be as powerful as Completion-UI's
`auto-completion-mode'. (Take a look at `auto-completion-syntax-alist'
and `auto-completion-override-syntax-alist', and at the way all the UI
widgets are integrated.) Anything less powerful, and it will be useless
to me for Predictive.

> >> We also document how to use `company-backends' and, for third-party
> >> packages, how to add a specific backend to it. But it's a simple data
> >> structure, so for users with some experience just knowing backend
> >> function names is sufficient.
> >
> > Sure, that's equally true of `completion-at-point-functions' and
> > `completion-ui-register-source'. (Except that new backends are
> > first-class citizens in Completion-UI, and show up automatically as
> > options in the completion UI customization interface where
> > appropriate. This would be harder to replicate with
> > `completion-at-point-functions', but I suspect with enough Customize
> > macro trickery it would still be possible with the existing Emacs
> > completion backend API.)
> 
> Customizing hooks is a tricky business. I believe the opaqueness of 
> c-a-p-f to the user is the main problem with the current API.

But above you argued that backends and selection logic are supplied and
configured by package authors, not by users!? In which case package
authors can simply supply a c-a-p-f function, and users can add the
functions they want to c-a-p-f. (I believe Stefan made a similar point
earlier.)

Don't get me wrong. I'm no fan of the complexity of c-a-p-f. But you seem
to be arguing both ways.

> >> Most Company backends at least have to check that the buffer is in
> >> matching major mode.
> >
> > Whereas in Completion-UI, you just use the standard Emacs mechanisms of
> > setting buffer-local variables from you major-mode hook, or using
> > `add-hook' with the LOCAL flag set to add a function to the
> > auto-completion-source-functions hook.
> >
> > I prefer to use the standard, tried-and-tested Emacs mechanisms where
> > possible (major-mode hooks), rather than creating an entirely new
> > package-specific way of configuring things for different major-modes. (I
> > don't always stick to this myself, though I probably should ;-)
> 
> Emacs core also has custom variables that list major modes applicable 
> for a given function. Example: `font-lock-global-modes'.

Fair point. I use such alists in Predictive too
(`predictive-major-mode-alist').

> Imagine you have two sources, each applicable in a different major mode. 
> With Company, this means two backend functions, and a modified 
> `company-backends' value.
> 
> With Completion-UI, with hooks and buffer-local values, just count the 
> entities:
> 
> - foo completion function
> - bar completion function
> - foo predicate function
> - bar predicate function
> - function foo-set-source-functions, setting 
> `auto-completion-source-functions' to a relevant local value
> - function bar-set-source-functions, doing the same for bar
> - (add-hook 'foo-mode-hook 'foo-set-source-functions)
> - (add-hook 'bar-mode-hook 'bar-set-source-functions)
> 
> ...now scale this to 5-10 modes and completion functions and see the 
> amount of boilerplate explode.

Swings and roundabouts. I could always add a
`completion-ui-major-mode-source' function that checks an alist, and then
it would be:

- foo completion function
- bar completion function
- foo predicate function
- bar predicate function
- add two entries to the alist

versus Company's

- foo's bundled completion function and selection logic
- bar's bundled completion function and selection logic
- add two entries to company-backends

On the other hand, configuring the 8 different Predictive LateX sources
(not to mention the additional package-specific sources that get loaded
and unloaded dynamically as the user types text into the LaTeX document)
via Company backends would involve:

- code 1st bundled completion function and selection logic
- code 2nd bundled completion function and selection logic (which
  duplicates the logic in the 1st one, with different parameters)
  ...
- code 8th bundled completion function and selection logic (which
  duplicates the logic in the last 7, with different parameters)
- add 8 entries to company-backends
- arrange for package-specific sources to be added to company-backends
  dynamically

Contrast with Completion-UI's, which involves a *lot* less coding:

- register 1st source (no need to code a new completion function, just
  set one parameter when calling `completion-ui-register-source')
- register 2nd source (same)
- ...
- register 8th source
- add entries to a single alist
- push and pop entries from that alist dynamically as needed

So Completion-UI lets you do some things more simply, and Company lets
you do others more simply. I still think separating the selection logic
from the completion function is cleaner, more logical, and more flexible.

It's also what c-a-p-f does. (It's a shame the c-a-p-f API is marred by
the complex format; though I can understand why it ended up like that.)

Actually, now I think about it, Completion-UI's
`auto-completion-source-functions' hook is *very* similar to c-a-p-f: a
list of functions that are called in turn until one of them returns the
completion source to use. It just has a simpler format for the return
values (a symbol identifying a completion source, instead of the
multi-element list returned by c-a-p-f functions.) It would be almost
trivial to switch Completion-UI over to c-a-p-f. All I'd need to do to
replicate the existing functionality is add a few additional properties
to the PROPS return argument.


> > So every backend has to look at what's being asked of it every time it's
> > invoked, and return nil if it's asked to do something it doesn't support?
> > And also check the major-mode and see if it's appropriate every time it's
> > invoked, even though the major-mode is very unlikely to change (and runs
> > a major-mode hook when it does)?
> 
> That's only in the `prefix' call. So, once per user command or so.

Right.

> > This sounds rather inefficient. I'm sure it's fast enough for Company,
> > but I'm skeptical it would be fast enough for predictive-mode's
> > auto-completion-mode (where tenths of a second matter).
> 
> You just haven't measured it.
> 
>    (dotimes (_ 1000) (derived-mode-p 'emacs-lisp-mode))
> 
> takes less than 1 ms. IOW, the comparison itself takes less than 1e-6 s.

But that's without the complex selection logic needed e.g. in
Predictive's LaTeX support. I had to rewrite the Completion-UI selection
logic two or three times before it was efficient enough for this.

It's no big deal, though. Presumably I could always set a buffer-local
`company-backends' from predictive-mode that only contains relevant
backends if necessary.

> > To summarise:
>  >
> > 2. We're probably stuck with c-a-p-f et al. for the completion source and
> >     source-selection API. How would this need to be extended to integrate
> >     it with a new Emacs completion UI? How could it be extended to better
> >     support Company and others?
> 
> To repeat what I said earlier, I'd like to see it customizable by users, 

c-a-p-f is trivially customizable by users, as long as packages / Company
supplies suitable c-a-p-f functions for the backends. *Writing* c-a-p-f
is non-trivial, but as you said earlier those are/should be supplied by
package authors. Or have I misunderstood?

I'm not advocating c-a-p-f, I'm just running out of good arguments
against it.

> and I'd like to see a proper merging function.

Agreed.

> > 3. The most popular and useful "list-the-available-completions" UI is
> >     popup.el. Is there any chance of getting copyright assignment for
> >     this? Or will we be forced to code something equivalent from scratch?
> 
> https://github.com/auto-complete/popup-el/issues/50

Fingers crossed...!

Best,
Toby
-- 
Dr T. S. Cubitt
Royal Society University Research Fellow
and Fellow of Churchill College, Cambridge
Centre for Quantum Information
DAMTP, University of Cambridge

email: tsc25@cantab.net
web:   www.dr-qubit.org