From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.org!not-for-mail From: Dmitry Gutov Newsgroups: gmane.emacs.devel Subject: Re: Emacs completion matches selection UI Date: Wed, 08 Jan 2014 03:23:21 +0400 Message-ID: <52CC8C69.2030705@yandex.ru> References: <20140107033209.GA30657@c3po> NNTP-Posting-Host: plane.gmane.org Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Trace: ger.gmane.org 1389137016 17682 80.91.229.3 (7 Jan 2014 23:23:36 GMT) X-Complaints-To: usenet@ger.gmane.org NNTP-Posting-Date: Tue, 7 Jan 2014 23:23:36 +0000 (UTC) Cc: emacs-devel@gnu.org To: Toby Cubitt Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Wed Jan 08 00:23:44 2014 Return-path: Envelope-to: ged-emacs-devel@m.gmane.org Original-Received: from lists.gnu.org ([208.118.235.17]) by plane.gmane.org with esmtp (Exim 4.69) (envelope-from ) id 1W0fzq-0007Dh-2z for ged-emacs-devel@m.gmane.org; Wed, 08 Jan 2014 00:23:42 +0100 Original-Received: from localhost ([::1]:44027 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1W0fzp-0000qk-JQ for ged-emacs-devel@m.gmane.org; Tue, 07 Jan 2014 18:23:41 -0500 Original-Received: from eggs.gnu.org ([2001:4830:134:3::10]:57894) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1W0fzg-0000mb-J2 for emacs-devel@gnu.org; Tue, 07 Jan 2014 18:23:38 -0500 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1W0fzb-00053j-51 for emacs-devel@gnu.org; Tue, 07 Jan 2014 18:23:32 -0500 Original-Received: from mail-la0-x229.google.com ([2a00:1450:4010:c03::229]:40759) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1W0fza-00053f-Ii for emacs-devel@gnu.org; Tue, 07 Jan 2014 18:23:27 -0500 Original-Received: by mail-la0-f41.google.com with SMTP id c6so626665lan.14 for ; Tue, 07 Jan 2014 15:23:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:content-type:content-transfer-encoding; bh=6/N4Th5Blld0coMLgOKiS/U1xXJQq6eXvZ14JmWEQpU=; b=i15WEFznYvCZvD3J2ryQO0b2iN4aPXjfos8lUoKQK0QUCMsUI52BGtvIFqLg21BhqD M/epZ+RBGEjyKXa4ma2CfMRVieZqbNQBO5vzOT8sbAht05Qsvhp5GAyY+kwBkLLE502U T64prtRrUaTQvnSatPhPui6b69ts5qLZR41spkOFXLtuQp4Q3PtfwOKiBI2cj6czeYkE lRJhArcKopHLfifyERUoZiGlSA9QqjUaBEnmTHU0+/S11jwHLruD2mfGH68g7QyfuY5U vOyM9xSarY/+BPXlOY8v3kp2f4G6v7qpd1h4ahaZ5Bf2nHXdKSMqHm1vWao5GQEz2POI fvtg== X-Received: by 10.152.19.65 with SMTP id c1mr4319475lae.49.1389137004871; Tue, 07 Jan 2014 15:23:24 -0800 (PST) Original-Received: from [192.168.1.3] ([178.252.98.87]) by mx.google.com with ESMTPSA id qx1sm45822918lbb.15.2014.01.07.15.23.22 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Tue, 07 Jan 2014 15:23:23 -0800 (PST) User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Thunderbird/24.2.0 In-Reply-To: <20140107033209.GA30657@c3po> X-detected-operating-system: by eggs.gnu.org: Error: Malformed IPv6 address (bad octet value). X-Received-From: 2a00:1450:4010:c03::229 X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Original-Sender: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Xref: news.gmane.org gmane.emacs.devel:167686 Archived-At: On 07.01.2014 07:32, Toby Cubitt wrote: >> Examples: >> 1. `prefix-replaced' and `common-substring' both seem to be tailored to >> the dynamic interface. > > Not at all. `prefix-replaced' tells you that the buffer substring that is > being completed has already been replaced with new text in the > buffer. This is important for non-prefix completion (e.g. pattern > matches). I'm not sure if it's generally applicable. AFAIK, neither Company nor CAPF keep this kind of history: if the buffer was modified, what happens after is a new completion, or no completion at all (if circumstances told us we should abort). > Any UI that modifies the buffer text will almost certainly need to set > `prefix-replaced'; I don't really see why a non-prefix completion UI (or any other one) has to modify the buffer text. Company has an inline "preview" frontend, but it uses an overlay, and so the buffer text remains unmodified. > And the core Completion-UI code that ties all the UIs > together makes heavy use of the `prefix-replaced' property. So even in > Completion-UI it's not only used by the dynamic interface. Could you explain how? > Similarly, `common-substring' demarcates the longest common substring if > completion text has been inserted in the buffer, which is information a > third-party UIs might very well want to make use of. Why? How? > The properties documented in the `completion-ui-register-interface' > docstring are those that have a standard meaning in Completion-UI. Note > that nothing prevents a particular UI widget from storing its own data in > its own widget-specific overlay properties if so desired. Sure. >> So how would one use an UI defined as such, in a different package? >> Would Company define itself as a new source? Or add a source per each >> backend? > > Depends on what the generic Emacs API for specifying completion sources > ends up looking like. Hmm, I was expecting something easier, since IIRC you said Company should have been able to use it without major changes. > If we end up using c-a-p-f, `completion-source' > will need to store something that identifies the c-a-p-f entry that > returned the completion candidates for this completion process. Probably just the value returned by the successful completion function: start, end, collection (aka completion table, which is often a function). >>> The best comparison one can make is that >>> company-backends loosely corresponds to a mixture of >>> completion-ui-source-definitions and auto-completion-source-functions. >> >> The latter: yes, the former: no. It's never used the same way as you use >> `completion-ui-source-definitions'. > > I said "loosely corresponds", and I stand by that. One of the things a > `company-backends' entry does is to define when a given completion source > should be used. In Completion-UI, that part of company-backends' role is > performed by auto-completion-source-functions. Yes, that's why I agreed with this comparison (and also made it earlier, IIRC). I disagree with the other comparison. > Could well be. But you seem to be arguing both ways. You also argue that > the c-a-p-f API is bad because it's opaque and hard for users to > configure. If backend definitions and selection logic are always supplied > by package authors, then the fact that c-a-p-f is opaque isn't so > significant. The opaqueness in c-a-p-f is bad because the exact values of `company-backends' and `completion-at-point-functions' are significant. A third-party package author can only push a function in either of these lists, but they can be responsible for the eventual order of the elements. And the order matters, because it influences which completion backend will get picked when several are suitable. When we get to the grouped backends, which I've mentioned several times, which Company supports, and CAPF will hopefully support in the future, being able to understand and maybe change the values of either list becomes even more important. >> Please take a look at `company-etags' and `company-dabbrev' and see if >> you can point out the situations when the user might find the `prefix' >> code of either too limiting. > > dabbrev and etags are both sources that, if they're useful in a given > buffer, they're useful in the entire buffer. > > Consider the LaTeX math mode source, and the LaTeX environment name My question was, if you could point out problems with any of our bundled backends (or, failing that, third-party ones). If they look okay, maybe > source, and the LaTeX preamble source, etc. Trying to code the selection > logic for all of these using the Company API looks very awkward to me, > compared to the simplicity of setting a few buffer-local variables in > Completion-UI. If performance is the problem, we could solve that by either: 1) Adding some pre-completion hook which would allow you to run some code once, set a buffer-local variable, which all backend functions for LaTeX can refer to later in `prefix' action. 2) Create a special "merged" backend that would collect the results from all LaTeX backend in a carefully defined fashion. 3) Indeed add some hook analogous to `auto-completion-source-functions'. But yeah, CAPF already does that. > Sure, I could move that logic into Predictive itself, and have a single > Predictive LaTeX backend. But that serves to demonstrate that the API > isn't flexible enough to let me do what I want easily. Other markup > languages and programming languages make similar demands on the API. We bundle several backends, some of them for programming modes, and so far you haven't pointed out specific problems with any of them. >> Or you can define new backends that would do some common checks in >> `prefix' (maybe calling an extracted function with common code) and >> simply delegate all other actions to the respective base backend. >> Implementing this is trivial. > > And once you've finished doing this, and factored out common selection > mechanisms like regexps, faces and text properties into utilities Regexps we have already (company-grab-...), instead of faces one should be using `syntax-ppss', font-lock isn't always available, and text properties... hmm. > functions...you'll have reimplemented something closer the Completion-UI > API or c-a-p-f :) Only if we add a similar hook, see 3) above. >> I understand the principle, really. But the more one "cleanly separates" >> code, the harder it can be sometimes to read it, to get the full picture. > > Indeed, which is why I listed grouping the completion and selection logic > into one place as one of the things I liked about Company's API. > > Perhaps the cleanest and most flexible solution would be to have a list > with entries of the form (TEST-FUNCTION . COMPLETION-FUNCTION), > COMPLETION-FUNCTION is used if TEST-FUNCTION returns non-nil. I believe this suffers from the LaTeX problem you've described above. If you have a dozen of completion functions for LaTeX, this scheme expects you to have a dozen of corresponding test functions, and when one fails, the next will be called, and it won't be able to use the results of the previous call (unless they set and use some common buffer-local variable, which Company backends could also do; but that's ugly). > And then > supply a bunch of standard utility functions for use as TEST-FUNCTION's, > for testing regexp matches, faces, text-properties, etc. Like I described previously, such stand-alone tests probably won't be very useful as values of this alist. One usually has to call several of them to see if a completion-function is suitable. >> It makes certain amount of sense, although it looks like it could make >> creating a "merged" completion function more difficult. > > I doubt it'll be insurmountable. Also, merged completion functions are a > rather advanced feature that may not belong in core Emacs anyway (though > it would be good if the API supported them without ugly hacks). It's a good feature enabling some kinds of backends that aren't usually useful on their own (like Yasnippet). In my book, that's a good argument to accept or reject an API. >> We'll see. > > Indeed, we'll see. Whilst I'd be happy to see the somewhat complicated > and opaque c-a-p-f API replaced with something cleaner and simpler, I > don't see us winning that argument. Personally, I'd probably be fine with c-a-p-f as long as it's powerful enough. Moving to a less featureful API is likely out of the question, but if it's demonstrated that c-a-p-f is fairly unsuitable for implementing some features, I believe it would be a good reason to rule it out. >> But the general approach, while flexible on its surface, complicates >> things if I intend to use any existing sources, written by third >> parties. Because their authors are unlikely to have anticipated the >> logic I'll add in my custom predicate function and to have written any >> code in their packages I might use. Or, at least, that's considerably >> less likely. > > I don't get your argument here. You have to wrap the third-party > completion function in Company in order to code the appropriate backend > selection logic. In Completion-UI, you put that code in an > `auto-completion-source-function' instead, and probably don't need to Which code? There's likely to be none. Hence, more effort required on my part. Here I'm describing an organizational problem caused by an API. Not a technical one. >> Like mentioned above, delegating the search for completions to an >> existing backend is trivial. These are functions, and as such they are >> stateless. Just call (other-backend 'candidates current-prefix). > > But this results is convoluted and somewhat confusing code. If the > selection logic and completion function were separate, you wouldn't need > to use such hacks. Looks straightforward to me. Function delegation is a rather simple concept. We have examples of that in the Company frontends: take a look at `company-pseudo-tooltip-unless-just-one-frontend' and `company-preview-if-just-one-frontend'. `company-dabbrev-code' also delegates to `company-dabbrev', but it just uses a public function from that backend's package, which is also a valid approach. > It understand why you're harping on about the limitations of the > default hook entries I just can't see how you find them useful on their own. I'll shut up now. :) > Indeed, though this is starting to look a lot like defining some standard > hook functions for auto-completion-source-functions. As you make the > company-backends API more flexible and convenient, you'll increasingly > find yourself reimplementing equivalent functionality to that of c-a-p-f > and the Completion-UI API. If were to start simplifying the Completion-UI > API or c-a-p-f, it'll increasingly look more like the Company API. Maybe > the sweet spot is in the middle? Maybe. I can certainly see myself adding a auto-completion-source-functions analog in Company, as an advanced feature. >> Actually, I was thinking about the former option. Let's define widgets >> with an API in the usual sense, so it can be used by both >> `completion-at-point' and external packages. We'll need this kind of API >> either way, in order to be able to write new widgets. > > I'm confused. `completion-at-point' will never use an API for *defining* > new completion UI widgets (`completion-ui-register-interface'). It will > need to be modified to *invoke* the new UI widgets. Here I'm thinking in terms of Company approach, I guess. The API for defining a backend or frontend and the way it is used are the same: a protocol consisting of messages and proper reactions or responses to them. Thus, "widget has to respond to..." would be in the API for defining a widget, and "widgets respond to..." would be in the usage API. Essentially the same. If you like to add convenience macros, extra infrastructure, etc, they may diverge, but we'll need an API a completion package can use easily use anyway, not just a bunch of private functions that can change will every release. > Copying and adapting some of the code from `complete-in-buffer' into > `completion-in-region' would suffice for that. Then > `completion-in-region' would remain the generic Emacs API for displaying > and selecting completion candidates (only now it would display them in a > nicer interface). Maybe. At a first glance, `completion-in-region' will need access to PROPS returned by completion functions, not just COLLECTIONS. >> Yes, but see above. Using 1. from Company would be the current next step >> toward integration, as I see it. > > Really? Why should we use the Company UI widget API and interfaces in > Emacs, when the API is less flexible and less customizable than that the > Completion-UI widgetAPI, and it implements fewer widgets? Sorry for being unclear. I meant the reverse: Company would use the new widget API defined here, while still retaining its backends, at least initially. Although while you don't own the popup widget, we do have one. Maybe porting that code won't be too hard. > Did you really mean that you wanted to throw Completion-UI in the > garbage, rewrite Company yourself to be suitable as a generic Emacs > completion API, rewrite/port the missing UIs and features, and persuade > people to merge it into Emacs core? (If so, great! Let me know when > you're done, and then I can stop maintaining Completion-UI and switch > Predictive over to the new interface :-) I'll think about it. :) Not sure about the "persuade people to merge in into Emacs core", though. But you could help kick this process off my filing an issue describing Company's backend API shortcomings. Is it just the lack of `auto-completion-source-functions'? Non-prefix completion? > Probably I misunderstood what you wrote. Yep. :) > I only really have one strong requirement: if some form of > auto-completion mode gets included in the Emacs completion UI, I will > argue hard for it to be as powerful as Completion-UI's > `auto-completion-mode'. (Take a look at `auto-completion-syntax-alist' > and `auto-completion-override-syntax-alist', and at the way all the UI > widgets are integrated.) Anything less powerful, and it will be useless > to me for Predictive. I'll take a look, thanks. >> Customizing hooks is a tricky business. I believe the opaqueness of >> c-a-p-f to the user is the main problem with the current API. > > But above you argued that backends and selection logic are supplied and > configured by package authors, not by users!? In which case package > authors can simply supply a c-a-p-f function, and users can add the > functions they want to c-a-p-f. (I believe Stefan made a similar point > earlier.) Customizing a hook is tricky for user. Try `M-x customize-variable RET find-file-hook'. You'll only see the buffer-local value, and not the global one. But `run-hooks' uses both if the local value includes `t', which it usually does. Users can modify the hooks programmatically, of course, but that's a step more difficult. And they'll also need to understand the values already there, to be able to remove or rearrange them. We'll also should discourage lambdas there. At the moment, my find-file-hook contains this beauty, courtesy of autorevert.el: #[nil "\302\301!\210\303\304!8\211\207" [buffer-file-name auto-revert-tail-pos make-local-variable 7 file-attributes] 3] We should have less of that. > Don't get me wrong. I'm no fan of the complexity of c-a-p-f. But you seem > to be arguing both ways. Of course I'm arguing toward a middle ground comfortable to me. So: package authors deciding when their backend is suitable to use -- good; users deciding the order of trying backends and their groupings -- also good. > Swings and roundabouts. I could always add a > `completion-ui-major-mode-source' function that checks an alist, and then > it would be: Yes, well, here you are discarding the "standard, tried-and-tested Emacs mechanisms" of major mode hooks and buffer-local values. Which was exactly my point. > - foo completion function > - bar completion function > - foo predicate function > - bar predicate function > - add two entries to the alist You forgot adding `completion-ui-major-mode-source' to the relevant list. > - code 1st bundled completion function and selection logic > - code 2nd bundled completion function and selection logic (which > duplicates the logic in the 1st one, with different parameters) > ... > - code 8th bundled completion function and selection logic (which > duplicates the logic in the last 7, with different parameters) Could that be a one-line function call, in each case with different arguments? Aside from potential performance problems, it sounds rather neat. > - add 8 entries to company-backends > - arrange for package-specific sources to be added to company-backends > dynamically > > Contrast with Completion-UI's, which involves a *lot* less coding: > > - register 1st source (no need to code a new completion function, just > set one parameter when calling `completion-ui-register-source') You can likewise delegate to the same completion function in Company backends. It's the same amount of code (1 line). > It would be almost > trivial to switch Completion-UI over to c-a-p-f. All I'd need to do to > replicate the existing functionality is add a few additional properties > to the PROPS return argument. Great. I'd like to see the patch. :) > Presumably I could always set a buffer-local > `company-backends' from predictive-mode that only contains relevant > backends if necessary. Yes, of course. > c-a-p-f is trivially customizable by users, as long as packages / Company > supplies suitable c-a-p-f functions for the backends. When you say "trivially customizable", do you mean via the Customize interface? > *Writing* c-a-p-f > is non-trivial, but as you said earlier those are/should be supplied by > package authors. Or have I misunderstood? I don't think I said that about c-a-p-f, but yes, they should be. >>> 3. The most popular and useful "list-the-available-completions" UI is >>> popup.el. Is there any chance of getting copyright assignment for >>> this? Or will we be forced to code something equivalent from scratch? >> >> https://github.com/auto-complete/popup-el/issues/50 > > Fingers crossed...! I wouldn't hold my breath: IME, getting a response from that crowd on non-trivial issues is hard. Collecting copyright assignments could be harder still. P.S. My email client displays each new message from you as a separate thread, possibly because your email address is timestamped, or maybe because they don't contain a "References" header. Could you do something about that?