Re: Emacs completion matches selection UI

[Toby Cubitt <tsc25@cantab.net>]

On Mon, Jan 06, 2014 at 06:38:37PM -0500, Ted Zlatanov wrote:
> On Sat, 4 Jan 2014 00:48:53 +0000 Toby Cubitt <tsc25@cantab.net> wrote: 
> TC> Since it's the completion UI that will have to interpret and make use of
> TC> the extras, one place to start is to think about what additional features
> TC> the UI might want to provide, beyond selecting from a list of possible
> TC> completions.
> 
> TC> Luckily, we have many existing completion frameworks to draw on for
> TC> inspiration. At the moment, location data and various forms of
> TC> documentation (e.g. type information in code completion, docstrings,
> TC> etc.) are the only ones that spring to mind.
> 
> Add: feedback from frontend to backend (Predictive use case we
> mentioned, needs API support through callbacks).  Pluggable completion
> frontends (not in the API)

Or rather, requires a separate API. Both Completion-UI and Company
already have APIs for this (though the discussion's mostly focused on the
completion UI API so far).

>  Dynamic limiting of completion candidates as characters are typed (not
> in the API).

All Completion-UI widgets already implement this(*), and the UI widget
API ensures that any third-part widgets will also automatically support
this too.

(*) The mouse-driven toolkit menus break it by stealing the keyboard
focus, but that's a technicality.

> The backend should be able to provide hints about the data.  For
> instance, "this is a list of ELisp symbols" could look different from
> "this is a list of installable packages."  Also icons are important IMO,
> based on the hints or the candidates.
> 
> TC> The keyword arguments to `completion-ui-register-source' are one starting
> TC> point for seeing what additional properties might be needed, as are the
> TC> corresponding APIs in Company, auto-complete, etc.
> 
> Can you or Dmitry compile a list?

Here are the important ones (copied from the docstring):

:activate FUNCTION
        The function called to activate the user-interface.

:deactivate FUNCTION
        The function called to deactivate the user-interface.

:update FUNCTION (optional)
        A function to call to update the user-interface after a
        change to the current completion. If it is not specified, the
        interface is updated by calling the :deactivate function,
        updating the completion, then calling the :activate function.

:auto-show FUNCTION (optional)
        A function to call to activate a this user-interface as
        an auto-show interface. The auto-show interface is chosen
        by setting the `completion-auto-show' customization
        option. Only one auto-show interface can be displayed at
        any one time.

:auto-show-helper FUNCTION (optional)
        A function to call whenever any auto-show interface is
        activated for a completion.


In Completion-UI, all of these functions are passed a single argument: an
overlay which demarcates where in the buffer the completion is taking
place, and whose properties contain relevant data about the completion
process. I rather like passing the data in overlay properties, because it
keeps the data together with the location in the buffer where things are
happening. But one could pass the data through buffer-local variables if
preferred.

The following overlay properties store the standard completion-related
data:

prefix
        The prefix or string being completed.

prefix-length
        Length of the prefix or string currently being completed. (For
        convenience, to avoid having to repeatedly call
        (length (overlay-get completion-overlay prefix))

completions
        A list of completion candidates.

completion-num
        Index of currently selected completion candidate, or nil
        if completions list is empty.

completion-overlay
        Always t for a completion overlay.

completion-source
        The completion source ("backend") used to obtain the completion
        candidates.

completion-prefix-function
        Function that was used to find the prefix or string to
        complete.

non-prefix-completion
        Nil if doing standard prefix completion, non-nil if doing
        something other than prefix completion (such as pattern
        matching).

prefix-replaced
        Non-nil if buffer string being completed (stored in prefix
        overlay property) has been replaced with new text in the
        buffer. (Only relevant when doing non-prefix completion.)

common-substring
        If set, an overlay marking the longest common substring of the
        completion text currently inserted in the buffer, if any.

keymap
        Keymap active within the completion overlay. Usually derived from
        `completion-overlay-map' or `auto-completion-overlay-map'.

auto-show
       Name of auto-show interface (symbol) currently being
       displayed, or nil if no auto-show interface is active.



> TC> Also, if we want to allow users to optionally customize the completion UI
> TC> per-backend (a la Completion-UI), there has to be some way of identifying
> TC> backends in Customize. This seems necessary to me if the new API is to
> TC> become the standard way of implementing a completion UI in Emacs. Think
> TC> minibuffer completion, versus filename completion, versus predictive
> TC> completion, versus elisp code completion etc, all of which might want to
> TC> use slightly different UI options. Again, it might be enough to support a
> TC> :name property (or some such) in the `completion-at-point-functions'
> TC> PROPS plist.
> 
> The frontend UI should depend (in my mind) on three things: 1) the
> invocation context as you showed; 2) the backend-provided hints about
> the data as a whole; and 3) the backend-provided hints about each
> completion candidate individually.  Does that make sense?

Yes, it makes sense. I suspect most of the time the UI will only need to
depend on which backend is being used (and often not even on that). I
can't think of any situations in which it will need to depend on the
completions returned by the backend. But maybe there are some cases.

> Some frontends may ignore some or all of those things, but I think
> those are the principal axis.  I don't know how those three things will
> be passed to the frontend though, and when.

Everything I've ever wanted to do with Completion-UI is covered by
allowing various configuration variables to take backend-specific
values. I've never needed the UI to depend on the return value of the
backend. Maybe Dmitry has examples from Company?

> >> - the frontend choices made by the user should be able to feed back into
> >> the data backend (the "predictive" use case)
> 
> TC> A few optional hooks/call-backs would very likely be sufficient. Again,
> TC> the `completion-at-point-functions' PROPS plist would be the obvious
> TC> place to specify them.
> 
> OK, this is your use case :)

I'm just bowing to the inevitable outcome that we'll end up extending
c-a-p-f rather than designing a new API for the completion sources.

> >> - we have enough interest and support in this work that it's worth
> >> undertaking in earnest after the code freeze.  The discussion should
> >> remain on emacs-devel (based on the wide interest so far).
> ...
> TC> (Note that I'm unlikely to have time to code anything myself until
> TC> March/April, although the feature freeze will likely delay things until
> TC> about then anyway. I'm happy to help in a more limited way before then,
> TC> as time allows.)
> 
> Cool.  We seem to be moving towards a specific API, refining the current
> Emacs internals, which is good.  I'd be OK with it looking like
> completion-ui as long as it was at least somewhat backwards compatible
> and could support the packages we mentioned.

By "looking like completion-UI", do you mean the API or what it actually
looks like?

I think the completion source API is likely to look exactly like
completion-at-point-functions, because that's already in Emacs and I'm
running out of compelling arguments for changing it, even though I don't
love it. But Dmitry may hold out longer than me on this ;-)

> To me, the clearest indication of a good approach would be if both the
> Emacs core and external packages were able to lose lines of code.

I'm hoping the Predictive package can lose 9 entire elisp files, for a
start (all the Completion-UI files ;-)

Toby
-- 
Dr T. S. Cubitt
Royal Society University Research Fellow
and Fellow of Churchill College, Cambridge
Centre for Quantum Information
DAMTP, University of Cambridge

email: tsc25@cantab.net
web:   www.dr-qubit.org