From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.org!not-for-mail From: Toby Cubitt Newsgroups: gmane.emacs.devel Subject: Re: Emacs completion matches selection UI Date: Mon, 6 Jan 2014 02:39:43 +0000 Message-ID: <20140106023943.GA31643@c3po> References: <52C7744F.3000906@yandex.ru> Reply-To: Toby Cubitt NNTP-Posting-Host: plane.gmane.org Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Trace: ger.gmane.org 1388976020 30154 80.91.229.3 (6 Jan 2014 02:40:20 GMT) X-Complaints-To: usenet@ger.gmane.org NNTP-Posting-Date: Mon, 6 Jan 2014 02:40:20 +0000 (UTC) Cc: emacs-devel@gnu.org To: Dmitry Gutov Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Mon Jan 06 03:40:27 2014 Return-path: Envelope-to: ged-emacs-devel@m.gmane.org Original-Received: from lists.gnu.org ([208.118.235.17]) by plane.gmane.org with esmtp (Exim 4.69) (envelope-from ) id 1W0078-0007yl-3z for ged-emacs-devel@m.gmane.org; Mon, 06 Jan 2014 03:40:26 +0100 Original-Received: from localhost ([::1]:60304 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1W0077-00017q-8o for ged-emacs-devel@m.gmane.org; Sun, 05 Jan 2014 21:40:25 -0500 Original-Received: from eggs.gnu.org ([2001:4830:134:3::10]:49016) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1W006v-00010F-AZ for emacs-devel@gnu.org; Sun, 05 Jan 2014 21:40:17 -0500 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1W006q-0002Kf-Tv for emacs-devel@gnu.org; Sun, 05 Jan 2014 21:40:13 -0500 Original-Received: from sanddollar.geekisp.com ([216.168.135.167]:46206) by eggs.gnu.org with smtp (Exim 4.71) (envelope-from ) id 1W006q-0002Jr-Pn for emacs-devel@gnu.org; Sun, 05 Jan 2014 21:40:08 -0500 Original-Received: (qmail 30302 invoked by uid 1003); 6 Jan 2014 02:40:07 -0000 Original-Received: from localhost (localhost.geekisp.com [127.0.0.1]) by localhost.geekisp.com (tmda-ofmipd) with ESMTP; Sun, 05 Jan 2014 21:40:00 -0500 Content-Disposition: inline In-Reply-To: <52C7744F.3000906@yandex.ru> X-PGP-Key: http://www.dr-qubit.org/gpg-toby-pub.asc User-Agent: Mutt/1.5.22 (2013-10-16) X-Delivery-Agent: TMDA/1.1.11 (Ladyburn) X-Primary-Address: toby@dr-qubit.org X-detected-operating-system: by eggs.gnu.org: OpenBSD 4.x-5.x [fuzzy] X-Received-From: 216.168.135.167 X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Original-Sender: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Xref: news.gmane.org gmane.emacs.devel:167429 Archived-At: On Sat, Jan 04, 2014 at 06:39:11AM +0400, Dmitry Gutov wrote: > On 03.01.2014 02:58, Toby Cubitt wrote: > > It's a shame Company et al. didn't build on and contribute to > > Completion-UI, and instead chose to (each) reinvent the wheel. But that's > > the usual messy history of code (which maybe we're finally going to do > > something to tidy up?). > > Maybe the widget APIs weren't particularly fitting, though. See my > response to your proposal below. Note that my proposal was a first stab at an API for generic Emacs completion UI; the Completion-UI APIs are much richer than this. The modern Completion-UI APIs are very likely either already sufficient or easily extended to support everything Company needs. But that's nowadays. The original 2006 Completion-UI APIs were much clunkier, and almost certainly weren't up to the job. I'm sure if you'd built Company on top of Completion-UI, we would have rapidly improved the APIs to support everything you needed as Company grew. But that's ancient history now, and no big deal. > > Do you want fast predictive completion of plain text, or > > text markup languages like LaTeX? > > It could be a decent replacement for dabbrev (I mostly use it to quickly > complete to some symbol already present before point in the same > buffer), but the problem with "smart" algorithms that keep history is > that often enough you can't predict the first option it will offer you. > That can be annoying, so maybe I don't. You can switch off the learning algorithms by toggling a customization option, and freeze the ordering once and for all if you want stability. If you're happy with the default frequency tables, you can even switch it off from the start. I originally thought I'd want to disable the learning algorithms once I'd trained Predictive on a corpus of my writing, for precisely the reason you describe. But in practice the order is stable enough that I never found it necessary. > >> Would you like to propose an API for it? Same as popup.el? > > > > As a first attempt, I'd propose something like > > > > (complete-in-buffer COMPLETION-SOURCE &optional PREFIX-FUNCTION) > > > > where COMPLETION-SOURCE is a function to call to obtain completions, and > > PREFIX-FUNCTION is an optional function to call to determine what to > > complete. > > But what if we have a list of completion sources (see the description > down below<*>)? Some of them can have a different notion of the prefix. > In Company, we try to remember which backend we're working with at the > moment, so that we only ask it for completions, candidate metadata, > etc. Sure, Completion-UI also remembers the backend ("completion source") that was used to find the current set of completion candidates, so that you can continue to interact with it sensibly. Clearly any generic Emacs completion UI would have to do something similar. I don't see a big problem supporting multiple different completion sources. Indeed, I view it as a basic requirement of any decent Emacs completion API if it's to be the generic UI. Completion-UI stores state by marking the location of the in-progress completion with an overlay, and storing data in overlay properties. You could alternatively use text properties, as you mention later. But I find overlays a better match for this kind of thing. (Presumably Company does something similar.) This means you can, if you want, have multiple pending completions using different sources in the same buffer at the same time, and Completion-UI will cope just fine if you switch back and forth between then. (Though the default setting of `completion-how-to-resolve-old-completions' disables this, as it's at least as confusing as `enable-recursive-minibuffers'.) > Though maybe it could work if PREFIX-FUNCTION iterated though backends, > saved which one responded, and then we could use that value in > COMPLETION-SOURCE, as well as when implementing the commands providing > the additional info. In this case, we'll at least need to have a way to > know when the widget was dismissed or refreshed to erase the saved > backend value. The above API was just a first stab at the most basic generic Elisp API for displaying a list of completion candidates and letting the user select one. It wasn't intended to be an API for defining a bunch of different a completion sources (backends) and selecting between then. As I wrote in my follow-up email, at this point we may be stuck with `completion-at-point-functions' for the latter. (Though we can try working on Stefan to see if he'll let us ditch it for something less clunky ;-) > In `company-capf', we iterate though the completion-at-point-functions > each time. This works, and works well enough, since the "does it fit" > code is usually fast, but it looks fairly bizarre to me, logically. I agree. The `completion-at-point-functions' API is convoluted compared to either the Company-native API or the Completion-UI API. On the other hand, it's already part of Emacs and is sufficiently flexible (if a little ugly). So we may be stuck with it now... > Ideally, though, the widget interface would be a drawing API of sorts: > "display these candidates at this point with this current index", > "update index", "hide", since we have everything else implemented already. > > > Calling this would invoke whatever completion widgets the user had > > enabled in customize (with sensible defaults - maybe the popup.el > > interface or similar, or perhaps the default should be the usual > > *Completions* buffer). > > I'd still like to a natural way to extend such widgets with new > commands. Passing a keymap as argument, or rebinding a dynamic variable > that points to keymap works, I guess, but it's not very nice. Well, this is only meaningful for some of the widgets. Indeed, "widgets" is mislead; the completion UIs in Completion-UI aren't necessarily widgets at all. For example, there are no "commands" at all in the dynamic completion UI, nor is it a "widget" in any normal sense of the word. But for those UIs for which it makes sense, Completion-UI already allows them to be extended, if necessary in different ways for different completion sources. In classic Emacs fashion, you supply functions/keymaps/etc. that override the defaults used to create widget contents. This API could rather easily be improved and extended once it becomes clear what would be useful or more convenient. But the basic machinery is already in place. (The list of override functions, keymaps etc. is *not* hard-coded. The API for adding a new completion user interface to Completion-UI allows you to add additional configuration options recognised by the new UI. The only limitation is that these won't be documented in the `completion-ui-register-source' docstring.) > >> A mouse-only menu won't fit Company because it won't work for idle > >> completion, as I mentioned elsewhere in this thread, and we generally > >> want to provide a consistent keyboard interface. There'll also be issues > >> with implementation related to that. > > > > When you register a new UI with Completion-UI, you tell it whether the UI > > is suitable for idle completion. If it is, it becomes one of the > > customization options you can select for the idle completion UI. > > Ok, it's reasonable approach, but that means that the UIs can't be > treated uniformly, which is sort of a bummer. The UIs absolutely cannot be treated uniformly, because not all of them are "display a list of completions"-style UIs. E.g. the "dynamic completion" UI is a completely different beast. Even amongst the "display a list of completions"-style UIs, the mouse-driven toolkit menus cannot be used in the terminal, nor can pop-up frames, nor can tooltips. That doesn't mean one should not provide these interfaces at all as options. It just means they probably aren't good default options. The default idle-completion UI in Completion-UI uses popup.el, which works everywhere. It's the best "display a list of completions"-style UI, for this and many other reasons. But that doesn't mean one shouldn't provide toolkit menus, or tooltips, or popup-frames *as well*, either so they can be triggered manually or as alternative options for the idle UI. Completion-UI has default keybindings to bring up the other UIs, even though they're not displayed by default. This is particularly appropriate for e.g. the mouse-driven toolkit menu and completion browser menu (which steal the keyboard focus). My philosophy with Completion-UI was to provide multiple UIs (and make it easily and modularly extendable so people can add additional UIs if they want), to have sensible default UI settings, but to let the user enable/disable/reconfigure them as they wish via Customize. Despite all this flexibility, and the very different types of UI it supports (which are not all of the "display a list of completions" variety), the UIs *are* treated uniformly in Completion-UI. If they weren't, I couldn't support arbitrary third-party UIs. Take a look at `completion-ui-register-interface' to see the API for all the different UIs. All the bundled UIs are defined via exactly this same API, with no special treatment. Furthermore, the completion *sources* (backends) don't need to know *anything* about how the UIs work. Indeed, they *never* invoke a specific UI directly. They simply return a bunch of completions, and Completion-UI presents them to the user via the various completion UIs, as customized *by the user*. > Ideally, we'd have a widget that's as useful as a menu when interacted > with with a mouse, and works with keyboard, and doesn't interrupt the > user's typing when displayed during idle completion. Sure, it's called popup.el. If that's unavailable, Completion-UI has an echo-area + hotkeys UI, which is a poor man's substitute for popup.el-style menus. (You can also enable both at once if you like.) > If I were to implement something like this for Company, it would mean > users who like this kind of menu would either have to forgo idle > completion, or see different kind of UIs whether completion is idle or > triggered manually. Which sounds weird, at least. No, you'd provide sensible defaults that give a uniform UI, but let users customize it to enable different kinds of UIs if they so desire. The mouse-driven toolkit menu doesn't replace other (maybe better) UIs. It complements it and/or is available as an alternative. I fail to see the problem. If the user wants to use what you consider a weird combination of UIs, I don't see any good reason to prevent them. The Emacs way has always been to provide sensible default settings, but then give users enough rope to shoot themselves in the foot if that's what tickles their fancy(*). (*) M-x reconfigure-rope-as-gun-and-shoot-foot > > Sounds like adding it to `company-backends' is more analogous registering > > it with `completion-ui-register-source'. > > <*> > > Not really. `company-backends' corresponds directly to > `auto-completion-source', Not quite. `company-backends' corresponds to the `completion-ui-source-definitions' variable. > only instead of one function we have a list of > them (which can also be nested one level deep, but that's an extra feature). Sure, Completion-UI also allows a list of completion sources and selects between them automagically. It even supports the same "extra feature" of nesting sources one level deep (Combined source, in Completion-UI terminology). But the API for this isn't auto-completion-default-source(*). That merely sets the default fall-back. The API for auto-selecting from a list of sources is provided by `auto-completion-source-regexps', `auto-completion-source-faces' and the `auto-completion-source-functions' hook. (The former two are in fact variables used by functions in the default `auto-completion-source-functions' definition.) (*) I'm referring here to the git version of Completion-UI; the last tarball release - which is woefully outdated - used a slightly different API. (I've been too busy with real work for the last few years to get around to rolling and testing a new tarball release; the git version still gets updates periodically.) > It's similar to `completion-at-point-functions' this way. First, each > backend knows whether it will be applicable at point (whether the > buffer's major mode fits, whether the syntax status at point fits, e.g. > some backends work in strings, others only work outside, etc). So > instead of calling an analog of `auto-completion-source', Company polls > all configured backends until it gets a response, then saves the name of > the backend thus found for the current completion invocation, and calls > it again for the list of completions. After that, it can call the > backend for calltips, candidate docs, et cetera. Yup, Completion-UI does something very similar. Except that completion source (backend) selection isn't directly tied to the backend. It's configured independently. So if you need to, you can set up different conditions under which a particular backend will be used in different buffers, major-modes, minor-modes, or whatever. Perhaps this is overkill. But as I keep coming back to, I designed Completion-UI primarily to be a generic completion UI elisp library, not as a user-level completion mode. And I do use some of this flexibility in Predictive's sophisticated LaTeX support. > > Customizing > > `auto-completion-source' is something the user would do, not the Elisp > > package coder. > > Same with `company-backends', although we provide a reasonable default. > But the user can change the global list, of change its local values, for > example set it to a single-item value in some major mode hook. Indeed, one of the best things about Company is all the backends that come pre-configured. I'm sure that accounts for much of its popularity. If I'd gone to the same effort of writing backends for Completion-UI, probably more people would use it. But the itch I was scratching wasn't to write a user-level completion framework like Company. I just wanted a UI for Predictive, and figured I might as well make it into a generic Elisp library so it could also be used elsewhere. I'd kinda hoped people might pick it up and use it to build something like Company or anything.el, so they could focus their time on building a great user-level completion mode rather than duplicating all the UI code, and instead help improve and contribute to Completion-UI. But for various reasons that never happened. Anyhow, I don't think company-mode is something that necessarily belongs in a generic Emacs completion UI. At most, Emacs should include the necessary definitions to hook the basic completion methods that come with Emacs into the new generic UI (dabbrev, etags, minibuffer completion, maybe a few others like elisp symbol completion...). Indeed, I would have argued that we should *first* come up with a generic Emacs completion UI and API *without* including any mechanism for automatically selecting between completion sources. I would have proposed to think later about when a source-selection mechanism belonged in Emacs, or whether it's best left to ELPA packages like Company... ...except that `completion-at-point-functions' *already* defines an API for selecting between completion sources. And it's already part of Emacs. So I guess that decision has already been taken. (I think there's still a discussion to be had about whether or not some form of auto-completion-mode belongs in Emacs. I can see arguments both ways.) > > In fact, it sounds like the two APIs are rather similar: > > > 2. Pass it to `completion-ui-register-source' (Completion-UI) or add it > > to `company-backends' (Company) to let them know about it. > > Guess the main difference is that, since we definine new backends with > `defun', there's no way to update the list of available values, visible > to the `defcustom' interface. You define new backends (completion functions) for Completion-UI with `defun' in exactly the same way. You don't *have* to register a completion function in order to use it with Completion-UI. If you want, you can simply pass the completion function to `complete-in-buffer', much like `completion-in-region'. Registering a source just (1) makes life more convenient: you set all the option values once when you define the completion source (backend), and then you can refer to that source and all its associated options using a single name (symbol). And (2) it makes the source available as an option in the Customization interface, as you noted. > It makes matters a bit worse for third-party backends, but not by much, > I think. I agree, this is a minor difference, and Company doesn't lose much by it, especially since it bundles almost all the backend definitions you're ever likely to want. On the other hand, it was more important for third-party completion sources to be first-class citizens in Completion-UI, precisely *because* I never put much effort into writing source definitions myself. For a generic Emacs completion UI, it's obviously crucial that third-part completion sources are first-class citizens, since almost all sources will be third-party. > > At some point in the distant past, you used to just set a variable to > > tell Completion-UI about a new completion function. It became a macro > > call when I made Completion-UI more generic, and there needed to be a way > > of optionally supplying additional information about what the completion > > function does (e.g. if it does something other than standard prefix > > completion), and of defining optional call-back functions, etc. > > We solve this problem by requiring backend functions to take the type of > question it's going to answer as the first argument. Function calls are > fast enough for our purpose, and this makes definitions quite succinct, > especially for simple backends. Simple backends have trivial `completion-ui-register-source' calls in Completion-UI, too. All you need is: (completion-ui-register-source my-completion-function) The macro figures out everything it needs to know from the function definition - the beauty of lisp macros! It's shorter than typing: (add-hook 'completion-at-point-functions 'my-new-completion-function) ;-) The optional macro arguments only come into play if you're doing something more sophisticated (like extending the default completion widgets, or overriding the default method of determining the text around point that's to be completed, etc.) > This pattern is used in many places in Emacs core, too. In principle, this sounds like a perfectly reasonable API choice for simple settings. Though I don't completely get how it works here. What if a particular completion function can't answer some types of question? E.g. maybe it can expand prefixes, but it can't find regexp matches. What does the completion function do if it's then asked for a regexp match? Or does the backend answer "I can't answer this type of question" and Company moves onto the next backend in the list? However, this API is not so convenient if you want to pass more complex settings, such as extending the UI for a particular source. You'd either have to pass the same list of hook functions, keymaps etc. every time you call the function that invokes the completion UI for the source. Or you'd have to let-bind the same set of variables every time. Which would be a pain. Surely more convenient to have a way of telling Emacs: "use this set of options, hook functions, keymaps, etc. every time I use this completion source to complete something". Naturally, you can do this via configuration variables that store alists/plists/insert-your-favourite-associative-array-here. And guess what? Most of what the `completion-ui-register-source' macro does is add a bunch of elements to the appropriate plists for you. You can do this manually if you prefer, but the macro call is so much more convenient. This pattern is used in many places in Emacs core, too: `defcustom', `defstruct', `defun*'... (Perhaps I should have called the macro `defcompletion' instead of `completion-ui-register-source' :) Also, the "pass the type of completion as the first argument" API means you have a hard-coded set of "types" of completion. As usual, since it was intended to be a generic completion Elisp library and not a fully-fledged user-level completion framework, Completion-UI doesn't impose any such restriction, but tries to be as flexible and general as possible. I'm not saying this is necessarily a good thing; it could well be that this flexibility is overkill. Thanks for taking the time to explain and compare Company features. It's very helpful in figuring out what a generic Emacs completion API should look like, at least to me. I increasingly have the impression that the Company and Completion-UI feature sets and APIs are in fact functionally very similar, modulo API details and a useful feature or two here and there on each side. (Leaving aside the massive collection of Company backend definitions, which Completion-UI completely(!) lacks.) Which bodes well for coming up with a generic Emacs completion UI API and code. If we've converged on a broadly similar set of features and APIs for the UI parts of Completion-UI and Company, there's a reasonable chance they're the features people will want from a generic Emacs API. Best, Toby -- Dr T. S. Cubitt Royal Society University Research Fellow and Fellow of Churchill College, Cambridge Centre for Quantum Information DAMTP, University of Cambridge email: tsc25@cantab.net web: www.dr-qubit.org