Re: Emacs completion matches selection UI

[Dmitry Gutov <dgutov@yandex.ru>]

On 07.01.2014 07:32, Toby Cubitt wrote:
>> Examples:
>> 1. `prefix-replaced' and `common-substring' both seem to be tailored to
>> the dynamic interface.
>
> Not at all. `prefix-replaced' tells you that the buffer substring that is
> being completed has already been replaced with new text in the
> buffer. This is important for non-prefix completion (e.g. pattern
> matches).

I'm not sure if it's generally applicable. AFAIK, neither Company nor 
CAPF keep this kind of history: if the buffer was modified, what happens 
after is a new completion, or no completion at all (if circumstances 
told us we should abort).

> Any UI that modifies the buffer text will almost certainly need to set
> `prefix-replaced';

I don't really see why a non-prefix completion UI (or any other one) has 
to modify the buffer text. Company has an inline "preview" frontend, but 
it uses an overlay, and so the buffer text remains unmodified.

> And the core Completion-UI code that ties all the UIs
> together makes heavy use of the `prefix-replaced' property. So even in
> Completion-UI it's not only used by the dynamic interface.

Could you explain how?

> Similarly, `common-substring' demarcates the longest common substring if
> completion text has been inserted in the buffer, which is information a
> third-party UIs might very well want to make use of.

Why? How?

> The properties documented in the `completion-ui-register-interface'
> docstring are those that have a standard meaning in Completion-UI. Note
> that nothing prevents a particular UI widget from storing its own data in
> its own widget-specific overlay properties if so desired.

Sure.

>> So how would one use an UI defined as such, in a different package?
>> Would Company define itself as a new source? Or add a source per each
>> backend?
>
> Depends on what the generic Emacs API for specifying completion sources
> ends up looking like.

Hmm, I was expecting something easier, since IIRC you said Company 
should have been able to use it without major changes.

> If we end up using c-a-p-f, `completion-source'
> will need to store something that identifies the c-a-p-f entry that
> returned the completion candidates for this completion process.

Probably just the value returned by the successful completion function: 
start, end, collection (aka completion table, which is often a function).

>>> The best comparison one can make is that
>>> company-backends loosely corresponds to a mixture of
>>> completion-ui-source-definitions and auto-completion-source-functions.
>>
>> The latter: yes, the former: no. It's never used the same way as you use
>> `completion-ui-source-definitions'.
>
> I said "loosely corresponds", and I stand by that. One of the things a
> `company-backends' entry does is to define when a given completion source
> should be used. In Completion-UI, that part of company-backends' role is
> performed by auto-completion-source-functions.

Yes, that's why I agreed with this comparison (and also made it earlier, 
IIRC). I disagree with the other comparison.

> Could well be. But you seem to be arguing both ways. You also argue that
> the c-a-p-f API is bad because it's opaque and hard for users to
> configure. If backend definitions and selection logic are always supplied
> by package authors, then the fact that c-a-p-f is opaque isn't so
> significant.

The opaqueness in c-a-p-f is bad because the exact values of 
`company-backends' and `completion-at-point-functions' are significant. 
A third-party package author can only push a function in either of these 
lists, but they can be responsible for the eventual order of the elements.

And the order matters, because it influences which completion backend 
will get picked when several are suitable. When we get to the grouped 
backends, which I've mentioned several times, which Company supports, 
and CAPF will hopefully support in the future, being able to understand 
and maybe change the values of either list becomes even more important.

>> Please take a look at `company-etags' and `company-dabbrev' and see if
>> you can point out the situations when the user might find the `prefix'
>> code of either too limiting.
>
> dabbrev and etags are both sources that, if they're useful in a given
> buffer, they're useful in the entire buffer.
>
> Consider the LaTeX math mode source, and the LaTeX environment name

My question was, if you could point out problems with any of our bundled 
backends (or, failing that, third-party ones). If they look okay, maybe

> source, and the LaTeX preamble source, etc. Trying to code the selection
> logic for all of these using the Company API looks very awkward to me,
> compared to the simplicity of setting a few buffer-local variables in
> Completion-UI.

If performance is the problem, we could solve that by either:

1) Adding some pre-completion hook which would allow you to run some 
code once, set a buffer-local variable, which all backend functions for 
LaTeX can refer to later in `prefix' action.

2) Create a special "merged" backend that would collect the results from 
all LaTeX backend in a carefully defined fashion.

3) Indeed add some hook analogous to `auto-completion-source-functions'. 
But yeah, CAPF already does that.

> Sure, I could move that logic into Predictive itself, and have a single
> Predictive LaTeX backend. But that serves to demonstrate that the API
> isn't flexible enough to let me do what I want easily. Other markup
> languages and programming languages make similar demands on the API.

We bundle several backends, some of them for programming modes, and so 
far you haven't pointed out specific problems with any of them.

>> Or you can define new backends that would do some common checks in
>> `prefix' (maybe calling an extracted function with common code) and
>> simply delegate all other actions to the respective base backend.
>> Implementing this is trivial.
>
> And once you've finished doing this, and factored out common selection
> mechanisms like regexps, faces and text properties into utilities

Regexps we have already (company-grab-...), instead of faces one should 
be using `syntax-ppss', font-lock isn't always available, and text 
properties... hmm.

> functions...you'll have reimplemented something closer the Completion-UI
> API or c-a-p-f :)

Only if we add a similar hook, see 3) above.

>> I understand the principle, really. But the more one "cleanly separates"
>> code, the harder it can be sometimes to read it, to get the full picture.
>
> Indeed, which is why I listed grouping the completion and selection logic
> into one place as one of the things I liked about Company's API.
>
> Perhaps the cleanest and most flexible solution would be to have a list
> with entries of the form (TEST-FUNCTION . COMPLETION-FUNCTION),
> COMPLETION-FUNCTION is used if TEST-FUNCTION returns non-nil.

I believe this suffers from the LaTeX problem you've described above. If 
you have a dozen of completion functions for LaTeX, this scheme expects 
you to have a dozen of corresponding test functions, and when one fails, 
the next will be called, and it won't be able to use the results of the 
previous call (unless they set and use some common buffer-local 
variable, which Company backends could also do; but that's ugly).

 > And then
> supply a bunch of standard utility functions for use as TEST-FUNCTION's,
> for testing regexp matches, faces, text-properties, etc.

Like I described previously, such stand-alone tests probably won't be 
very useful as values of this alist. One usually has to call several of 
them to see if a completion-function is suitable.

>> It makes certain amount of sense, although it looks like it could make
>> creating a "merged" completion function more difficult.
>
> I doubt it'll be insurmountable. Also, merged completion functions are a
> rather advanced feature that may not belong in core Emacs anyway (though
> it would be good if the API supported them without ugly hacks).

It's a good feature enabling some kinds of backends that aren't usually 
useful on their own (like Yasnippet). In my book, that's a good argument 
to accept or reject an API.

>> We'll see.
>
> Indeed, we'll see. Whilst I'd be happy to see the somewhat complicated
> and opaque c-a-p-f API replaced with something cleaner and simpler, I
> don't see us winning that argument.

Personally, I'd probably be fine with c-a-p-f as long as it's powerful 
enough. Moving to a less featureful API is likely out of the question, 
but if it's demonstrated that c-a-p-f is fairly unsuitable for 
implementing some features, I believe it would be a good reason to rule 
it out.

>> But the general approach, while flexible on its surface, complicates
>> things if I intend to use any existing sources, written by third
>> parties. Because their authors are unlikely to have anticipated the
>> logic I'll add in my custom predicate function and to have written any
>> code in their packages I might use. Or, at least, that's considerably
>> less likely.
>
> I don't get your argument here. You have to wrap the third-party
> completion function in Company in order to code the appropriate backend
> selection logic. In Completion-UI, you put that code in an
> `auto-completion-source-function' instead, and probably don't need to

Which code? There's likely to be none.

Hence, more effort required on my part.

Here I'm describing an organizational problem caused by an API. Not a 
technical one.

>> Like mentioned above, delegating the search for completions to an
>> existing backend is trivial. These are functions, and as such they are
>> stateless. Just call (other-backend 'candidates current-prefix).
>
> But this results is convoluted and somewhat confusing code. If the
> selection logic and completion function were separate, you wouldn't need
> to use such hacks.

Looks straightforward to me. Function delegation is a rather simple concept.

We have examples of that in the Company frontends: take a look at 
`company-pseudo-tooltip-unless-just-one-frontend' and 
`company-preview-if-just-one-frontend'.

`company-dabbrev-code' also delegates to `company-dabbrev', but it just 
uses a public function from that backend's package, which is also a 
valid approach.

> It understand why you're harping on about the limitations of the
> default hook entries

I just can't see how you find them useful on their own. I'll shut up now. :)

> Indeed, though this is starting to look a lot like defining some standard
> hook functions for auto-completion-source-functions. As you make the
> company-backends API more flexible and convenient, you'll increasingly
> find yourself reimplementing equivalent functionality to that of c-a-p-f
> and the Completion-UI API. If were to start simplifying the Completion-UI
> API or c-a-p-f, it'll increasingly look more like the Company API. Maybe
> the sweet spot is in the middle?

Maybe. I can certainly see myself adding a 
auto-completion-source-functions analog in Company, as an advanced feature.

>> Actually, I was thinking about the former option. Let's define widgets
>> with an API in the usual sense, so it can be used by both
>> `completion-at-point' and external packages. We'll need this kind of API
>> either way, in order to be able to write new widgets.
>
> I'm confused. `completion-at-point' will never use an API for *defining*
> new completion UI widgets (`completion-ui-register-interface'). It will
> need to be modified to *invoke* the new UI widgets.

Here I'm thinking in terms of Company approach, I guess. The API for 
defining a backend or frontend and the way it is used are the same: a 
protocol consisting of messages and proper reactions or responses to them.

Thus, "widget has to respond to..." would be in the API for defining a 
widget, and "widgets respond to..." would be in the usage API. 
Essentially the same.

If you like to add convenience macros, extra infrastructure, etc, they 
may diverge, but we'll need an API a completion package can use easily 
use anyway, not just a bunch of private functions that can change will 
every release.

> Copying and adapting some of the code from `complete-in-buffer' into
> `completion-in-region' would suffice for that. Then
> `completion-in-region' would remain the generic Emacs API for displaying
> and selecting completion candidates (only now it would display them in a
> nicer interface).

Maybe. At a first glance, `completion-in-region' will need access to 
PROPS returned by completion functions, not just COLLECTIONS.

>> Yes, but see above. Using 1. from Company would be the current next step
>> toward integration, as I see it.
>
> Really? Why should we use the Company UI widget API and interfaces in
> Emacs, when the API is less flexible and less customizable than that the
> Completion-UI widgetAPI, and it implements fewer widgets?

Sorry for being unclear. I meant the reverse: Company would use the new 
widget API defined here, while still retaining its backends, at least 
initially.

Although while you don't own the popup widget, we do have one. Maybe 
porting that code won't be too hard.

> Did you really mean that you wanted to throw Completion-UI in the
> garbage, rewrite Company yourself to be suitable as a generic Emacs
> completion API, rewrite/port the missing UIs and features, and persuade
> people to merge it into Emacs core? (If so, great! Let me know when
> you're done, and then I can stop maintaining Completion-UI and switch
> Predictive over to the new interface :-)

I'll think about it. :) Not sure about the "persuade people to merge in 
into Emacs core", though.

But you could help kick this process off my filing an issue describing 
Company's backend API shortcomings. Is it just the lack of 
`auto-completion-source-functions'? Non-prefix completion?

> Probably I misunderstood what you wrote.

Yep. :)

> I only really have one strong requirement: if some form of
> auto-completion mode gets included in the Emacs completion UI, I will
> argue hard for it to be as powerful as Completion-UI's
> `auto-completion-mode'. (Take a look at `auto-completion-syntax-alist'
> and `auto-completion-override-syntax-alist', and at the way all the UI
> widgets are integrated.) Anything less powerful, and it will be useless
> to me for Predictive.

I'll take a look, thanks.

>> Customizing hooks is a tricky business. I believe the opaqueness of
>> c-a-p-f to the user is the main problem with the current API.
>
> But above you argued that backends and selection logic are supplied and
> configured by package authors, not by users!? In which case package
> authors can simply supply a c-a-p-f function, and users can add the
> functions they want to c-a-p-f. (I believe Stefan made a similar point
> earlier.)

Customizing a hook is tricky for user. Try `M-x customize-variable RET 
find-file-hook'. You'll only see the buffer-local value, and not the 
global one. But `run-hooks' uses both if the local value includes `t', 
which it usually does.

Users can modify the hooks programmatically, of course, but that's a 
step more difficult. And they'll also need to understand the values 
already there, to be able to remove or rearrange them.

We'll also should discourage lambdas there. At the moment, my 
find-file-hook contains this beauty, courtesy of autorevert.el:

#[nil "\302\301!\210\303\304\b!8\211\x11\207"
   [buffer-file-name auto-revert-tail-pos make-local-variable 7 
file-attributes]
   3]

We should have less of that.

> Don't get me wrong. I'm no fan of the complexity of c-a-p-f. But you seem
> to be arguing both ways.

Of course I'm arguing toward a middle ground comfortable to me. So: 
package authors deciding when their backend is suitable to use -- good; 
users deciding the order of trying backends and their groupings -- also 
good.

> Swings and roundabouts. I could always add a
> `completion-ui-major-mode-source' function that checks an alist, and then
> it would be:

Yes, well, here you are discarding the "standard, tried-and-tested Emacs 
mechanisms" of major mode hooks and buffer-local values. Which was 
exactly my point.

> - foo completion function
> - bar completion function
> - foo predicate function
> - bar predicate function
> - add two entries to the alist

You forgot adding `completion-ui-major-mode-source' to the relevant list.

> - code 1st bundled completion function and selection logic
> - code 2nd bundled completion function and selection logic (which
>    duplicates the logic in the 1st one, with different parameters)
>    ...
> - code 8th bundled completion function and selection logic (which
>    duplicates the logic in the last 7, with different parameters)

Could that be a one-line function call, in each case with different 
arguments? Aside from potential performance problems, it sounds rather neat.

> - add 8 entries to company-backends
> - arrange for package-specific sources to be added to company-backends
>    dynamically
>
> Contrast with Completion-UI's, which involves a *lot* less coding:
>
> - register 1st source (no need to code a new completion function, just
>    set one parameter when calling `completion-ui-register-source')

You can likewise delegate to the same completion function in Company 
backends. It's the same amount of code (1 line).

> It would be almost
> trivial to switch Completion-UI over to c-a-p-f. All I'd need to do to
> replicate the existing functionality is add a few additional properties
> to the PROPS return argument.

Great. I'd like to see the patch. :)

> Presumably I could always set a buffer-local
> `company-backends' from predictive-mode that only contains relevant
> backends if necessary.

Yes, of course.

> c-a-p-f is trivially customizable by users, as long as packages / Company
> supplies suitable c-a-p-f functions for the backends.

When you say "trivially customizable", do you mean via the Customize 
interface?

> *Writing* c-a-p-f
> is non-trivial, but as you said earlier those are/should be supplied by
> package authors. Or have I misunderstood?

I don't think I said that about c-a-p-f, but yes, they should be.

>>> 3. The most popular and useful "list-the-available-completions" UI is
>>>      popup.el. Is there any chance of getting copyright assignment for
>>>      this? Or will we be forced to code something equivalent from scratch?
>>
>> https://github.com/auto-complete/popup-el/issues/50
>
> Fingers crossed...!

I wouldn't hold my breath: IME, getting a response from that crowd on 
non-trivial issues is hard. Collecting copyright assignments could be 
harder still.

P.S. My email client displays each new message from you as a separate 
thread, possibly because your email address is timestamped, or maybe 
because they don't contain a "References" header. Could you do something 
about that?