Emacs completion matches selection UI

Emacs completion matches selection UI

[Ted Zlatanov]

In "(emacs.info) Completion Commands" there is help for using the
rudimentary buffer to select completion matches.  I was struck by how
hard to use this was, compared to more recent packages like these:

http://www.emacswiki.org/emacs/AutoComplete
http://www.emacswiki.org/emacs/CompanyMode

Is there any chance the default Emacs appearance for selecting
completion matches can be improved (not necessarily by default) for
24.4?  I would help with this but don't know what's "state of the art"
and if there's already work in this direction.

Thanks
Ted

Re: Emacs completion matches selection UI

[Bozhidar Batsov]

[-- Attachment #1: Type: text/plain, Size: 1162 bytes --]

Unlike auto-complete-mode, Emacs's default completion system is rock-solid.
:-) auto-complete uses some pretty messed up popup display library that
doesn't play well with many packages. company-mode is much better in this
regard.

I guess I'm so used to the default completion system that I don't see it as
something primitive and hard to work with. That said, it might not be a bad
idea to include a "more modern" completion system in Emacs itself, but I'd
opt for simplicity and reliability over fancy UIs.


On 18 November 2013 18:13, Ted Zlatanov <tzz@lifelogs.com> wrote:

> In "(emacs.info) Completion Commands" there is help for using the
> rudimentary buffer to select completion matches.  I was struck by how
> hard to use this was, compared to more recent packages like these:
>
> http://www.emacswiki.org/emacs/AutoComplete
> http://www.emacswiki.org/emacs/CompanyMode
>
> Is there any chance the default Emacs appearance for selecting
> completion matches can be improved (not necessarily by default) for
> 24.4?  I would help with this but don't know what's "state of the art"
> and if there's already work in this direction.
>
> Thanks
> Ted
>
>
>

[-- Attachment #2: Type: text/html, Size: 1855 bytes --]

Re: Emacs completion matches selection UI

[Andreas Röhler]

Am 18.11.2013 21:00, schrieb Bozhidar Batsov:
> Unlike auto-complete-mode, Emacs's default completion system is rock-solid.
> :-) auto-complete uses some pretty messed up popup display library that
> doesn't play well with many packages. company-mode is much better in this
> regard.
>
> I guess I'm so used to the default completion system that I don't see it as
> something primitive and hard to work with. That said, it might not be a bad
> idea to include a "more modern" completion system in Emacs itself, but I'd
> opt for simplicity and reliability over fancy UIs.
>

+1

Re: Emacs completion matches selection UI

[Ted Zlatanov]

On Mon, 18 Nov 2013 22:00:33 +0200 Bozhidar Batsov <bozhidar@batsov.com> wrote: 

BB> On 18 November 2013 18:13, Ted Zlatanov <tzz@lifelogs.com> wrote:

>> In "(emacs.info) Completion Commands" there is help for using the
>> rudimentary buffer to select completion matches.  I was struck by how
>> hard to use this was, compared to more recent packages like these:
>> 
>> http://www.emacswiki.org/emacs/AutoComplete
>> http://www.emacswiki.org/emacs/CompanyMode
>> 
>> Is there any chance the default Emacs appearance for selecting
>> completion matches can be improved (not necessarily by default) for
>> 24.4?  I would help with this but don't know what's "state of the art"
>> and if there's already work in this direction.

BB> Unlike auto-complete-mode, Emacs's default completion system is rock-solid.
BB> :-) auto-complete uses some pretty messed up popup display library that
BB> doesn't play well with many packages. company-mode is much better in this
BB> regard.

(argh, I shouldn't have said "appearance" above, I meant "UI")

I just gave examples, without intending to promote either package. 

BB> I guess I'm so used to the default completion system that I don't see it as
BB> something primitive and hard to work with. That said, it might not be a bad
BB> idea to include a "more modern" completion system in Emacs itself, but I'd
BB> opt for simplicity and reliability over fancy UIs.

The completion system is fine.  The selection of matches is the problem:

1) see a new buffer popup with minimal help text and no highlighting
2) left, right, up, down don't work
3) realize problem, switch to candidates buffer (mouse click or `C-x o')
4) select candidate you want, get popped in original buffer

That's not simple!  It's not intuitive either, forcing me to use the
mouse unless I've read the manual node referenced above.

An alternative UI doesn't have to be fancy or graphical, only allow me
to select from among the candidates immediately, without switching
buffers, using the intuitive keys.  I hope that explains my request better.

Ted

Re: Emacs completion matches selection UI

[Stefan Monnier]

>>> In "(emacs.info) Completion Commands" there is help for using the
>>> rudimentary buffer to select completion matches.  I was struck by how
>>> hard to use this was, compared to more recent packages like these:
>>> http://www.emacswiki.org/emacs/AutoComplete
>>> http://www.emacswiki.org/emacs/CompanyMode
>>> Is there any chance the default Emacs appearance for selecting
>>> completion matches can be improved (not necessarily by default) for
>>> 24.4?  I would help with this but don't know what's "state of the art"
>>> and if there's already work in this direction.
BB> Unlike auto-complete-mode, Emacs's default completion system is rock-solid.
BB> :-) auto-complete uses some pretty messed up popup display library that
BB> doesn't play well with many packages. company-mode is much better in this
BB> regard.

company-mode's popup is also far from rock-solid (I have no experience
with auto-complete's popup, so can't compare).  But FWIW, I'd like to
move company-mode into the core completion code.

I've implemented a bridge between company-mode and
completion-at-point-functions ("company-capf"), and moved company-mode's
completion tables for Elisp into lisp-mode.el.  But there needs
to be some further integration work to make sure company-mode works well
with existing completion-at-point-functions.

But it wouldn't be enabled by default, anyway.

> The completion system is fine.  The selection of matches is the problem:
> 1) see a new buffer popup with minimal help text and no highlighting
> 2) left, right, up, down don't work
> 3) realize problem, switch to candidates buffer (mouse click or `C-x o')
> 4) select candidate you want, get popped in original buffer
> That's not simple!  It's not intuitive either, forcing me to use the
> mouse unless I've read the manual node referenced above.
> An alternative UI doesn't have to be fancy or graphical, only allow me
> to select from among the candidates immediately, without switching
> buffers, using the intuitive keys.  I hope that explains my request better.

Oh, that should be reasonably easy.  Just add the corresponding
key-bindings in minibuffer-local-completion-map, mostly.  The main issue
is to try and avoid clashing with existing bindings (since there are
2 buffers at play: the minibuffer and the *Completions* buffer), and
without C-x o, you're still in the minibuffer where left/right should
still mean "cursor movement in the minibuffer".


        Stefan

Re: Emacs completion matches selection UI

[Juri Linkov]

> The completion system is fine.  The selection of matches is the problem:
>
> 1) see a new buffer popup with minimal help text and no highlighting
> 2) left, right, up, down don't work
> 3) realize problem, switch to candidates buffer (mouse click or `C-x o')
> 4) select candidate you want, get popped in original buffer
>
> That's not simple!  It's not intuitive either, forcing me to use the
> mouse unless I've read the manual node referenced above.
>
> An alternative UI doesn't have to be fancy or graphical, only allow me
> to select from among the candidates immediately, without switching
> buffers, using the intuitive keys.  I hope that explains my request better.

In modern UIs, auto-completion works such a way that after you type text
it pops up a drop-down menu with a list of candidates where you can
select one by using <down> and <up> arrow keys.

How this could map to Emacs?  The first difference is that in Emacs
the text entry field (the minibuffer) is at the bottom of the screen,
so the drop-down menu should pop up upwards using the display-window
property `near-minibuffer' (this will display completions nicely aligned
with the contents of the minibuffer).

Another difficulty is that in the minibuffer <up> and <down> arrow keys
are bound to previous-history-element and next-history-element,
so they can't be used to select an element from the list of candidates.

One solution is to automatically display the list of completions
when the user starts typing in the minibuffer and allow <up> and <down>
to navigate the list of completions if the minibuffer contents is not empty.
Otherwise, <up> and <down> will browse the history.

Since this is quite obtrusive change it might require special mode
much like ido-mode but closer to auto-completion provided by other
GUI applications.

Re: Emacs completion matches selection UI

[Dmitry Gutov]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> company-mode's popup is also far from rock-solid

Any bugs you'd like to report? ;)

Re: Emacs completion matches selection UI

[Tom]

Juri Linkov <juri <at> jurta.org> writes:

> In modern UIs, auto-completion works such a way that after you type text
> it pops up a drop-down menu with a list of candidates where you can
> select one by using <down> and <up> arrow keys.
> 

Helm works like this. You type and the candidates appear automatically.

It could also be considered as a possible alternative completion UI.

Re: Emacs completion matches selection UI

[Stefan Monnier]

>> company-mode's popup is also far from rock-solid
> Any bugs you'd like to report? ;)

proportional fonts?


        Stefan

Re: Emacs completion matches selection UI

[Ted Zlatanov]

On Mon, 18 Nov 2013 19:43:04 -0500 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 

SM> I've implemented a bridge between company-mode and
SM> completion-at-point-functions ("company-capf"), and moved company-mode's
SM> completion tables for Elisp into lisp-mode.el.  But there needs
SM> to be some further integration work to make sure company-mode works well
SM> with existing completion-at-point-functions.

SM> But it wouldn't be enabled by default, anyway.

The default match selection UI is not easy or intuitive, so that's one
problem.  I think it can be improved with better key bindings and better
formatting in the candidates buffer, but honestly any solution that pops
up a whole new buffer is unusable today in 2013.  Users are accustomed
to at least some kind of selection popup like what company-mode
provides.  Today's GUI toolkits on all our graphical platforms certainly
provide that functionality.

The second problem is that there are no alternatives to the default
match selection UI.  Your company-mode integration would help there.
Please let me know when it's ready for testing.

>> The completion system is fine.  The selection of matches is the problem:
>> 1) see a new buffer popup with minimal help text and no highlighting
>> 2) left, right, up, down don't work
>> 3) realize problem, switch to candidates buffer (mouse click or `C-x o')
>> 4) select candidate you want, get popped in original buffer
>> That's not simple!  It's not intuitive either, forcing me to use the
>> mouse unless I've read the manual node referenced above.
>> An alternative UI doesn't have to be fancy or graphical, only allow me
>> to select from among the candidates immediately, without switching
>> buffers, using the intuitive keys.  I hope that explains my request better.

SM> Oh, that should be reasonably easy.  Just add the corresponding
SM> key-bindings in minibuffer-local-completion-map, mostly.  The main issue
SM> is to try and avoid clashing with existing bindings (since there are
SM> 2 buffers at play: the minibuffer and the *Completions* buffer), and
SM> without C-x o, you're still in the minibuffer where left/right should
SM> still mean "cursor movement in the minibuffer".

I think these new keybindings should be the default for everyone so yes,
care is needed.  I don't know the right keys to use.

Remember you can also see the completion candidates buffer without the
minibuffer, if you e.g. do `complete-symbol' in a buffer.  So the
minibuffer is not the only context.

The most important thing IMO is to avoid making a new buffer that
requires `C-x o' and magical bindings.

On Tue, 19 Nov 2013 02:47:11 +0200 Juri Linkov <juri@jurta.org> wrote: 

JL> In modern UIs, auto-completion works such a way that after you type text
JL> it pops up a drop-down menu with a list of candidates where you can
JL> select one by using <down> and <up> arrow keys.

JL> How this could map to Emacs?  The first difference is that in Emacs
JL> the text entry field (the minibuffer) is at the bottom of the screen,
JL> so the drop-down menu should pop up upwards using the display-window
JL> property `near-minibuffer' (this will display completions nicely aligned
JL> with the contents of the minibuffer).

JL> Another difficulty is that in the minibuffer <up> and <down> arrow keys
JL> are bound to previous-history-element and next-history-element,
JL> so they can't be used to select an element from the list of candidates.

JL> One solution is to automatically display the list of completions
JL> when the user starts typing in the minibuffer and allow <up> and <down>
JL> to navigate the list of completions if the minibuffer contents is not empty.
JL> Otherwise, <up> and <down> will browse the history.

Note, as I mentioned above, that completion candidates can be presented
outside the minibuffer context as well.

JL> Since this is quite obtrusive change it might require special mode
JL> much like ido-mode but closer to auto-completion provided by other
JL> GUI applications.

I would argue that company-mode has solved the problem to some extent,
but I understand it's not a simple change.  I hope you and Stefan and
others have some idea of what needs to be done.  I'm happy to assist but
don't have the required knowledge of the internals to write code.

On Tue, 19 Nov 2013 11:18:09 +0000 (UTC) Tom <adatgyujto@gmail.com> wrote: 

T> [Helm] could also be considered as a possible alternative completion UI.

I use and love helm-mode, but it's not truly a completion UI, it's an
action selection UI.  It lacks context normally, meaning that I can call
`help-mini' any time in any buffer and it will be consistent.  That's
why I didn't propose it, even though it's very useful on its own.

The helm-mode code that narrows candidates and presents them in a nice
categorized buffer could certainly be considered together with
company-mode and the other completion UIs in existence.  I don't know if
it's usable in isolation from the rest of Helm.

Ted

Re: Emacs completion matches selection UI

[Stefan Monnier]

>> In modern UIs, auto-completion works such a way that after you type text
>> it pops up a drop-down menu with a list of candidates where you can
>> select one by using <down> and <up> arrow keys.
> Helm works like this. You type and the candidates appear automatically.

For the "appear" part you could start with the trivial minor-mode below.
But I think what Ted is after is the other part: down/up/...

While there is a risk of conflict in key-bindings, I do think it should
be possible to "keep the best of both worlds" such that it could be
enabled by default.

E.g.: keep the same behavior by default as we have now, except that we
add a new `select-completion-mode'.  In this new mode, cursor keys move
between elements of the completions list, RET selects the completion
element from there, and most other keys just exit the mode.

The main issue is then to figure out how/when to switch to this
new mode.  E.g. when the user hits `up' right after the *Completions*
buffer got displayed/updated?

> It could also be considered as a possible alternative completion UI.

Part of Helm is an alternative completion UI, indeed.  Another part is
a much fancier completion-list-mode.  And then yet another part relies
on this much fancier completion-list-mode to change the argument order
around, so that instead of first specifying the command and then the
object on which to apply it, you first specify the object and then
the command.


        Stefan


(defun verbose--icomplete-do ()
  (let ((beg (minibuffer-prompt-end))
        (end (point-max))
        (non-essential t))
    (when (and (not (eq beg end)) buffer-undo-list)
      (while-no-input
        (minibuffer-completion-help beg end)))))

(defun verbose--icomplete-setup ()
  (when minibuffer-completion-table
    (add-hook 'post-command-hook #'verbose--icomplete-do nil 'local)))

(define-minor-mode verbose-icomplete-mode
  ""
  :global t
  (if verbose-icomplete-mode
      (add-hook 'minibuffer-setup-hook 'verbose--icomplete-setup)
    (remove-hook 'minibuffer-setup-hook 'verbose--icomplete-setup)))

Re: Emacs completion matches selection UI

[Dmitry Gutov]

On 19.11.2013 15:41, Stefan Monnier wrote:
>>> company-mode's popup is also far from rock-solid
>> Any bugs you'd like to report? ;)
>
> proportional fonts?

If having fundamental incompatibilities with some modes of operation is 
the worst problem we have, I'd say that's pretty good.

In graphical mode, I think we can make a more reliable popup by adapting 
some code in `tooltip.el'. That would be incompatible with running in 
terminal, but there are no proportional fonts in terminal either (I 
think...?)

Re: Emacs completion matches selection UI

[Ted Zlatanov]

On Tue, 19 Nov 2013 16:00:49 +0200 Dmitry Gutov <dgutov@yandex.ru> wrote: 

DG> In graphical mode, I think we can make a more reliable popup by
DG> adapting some code in `tooltip.el'. That would be incompatible with
DG> running in terminal, but there are no proportional fonts in terminal
DG> either (I think...?)

Works for me, as long as it's standard (so we don't have the current
"reinvention of the wheel" everywhere).

On Tue, 19 Nov 2013 09:00:40 -0500 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 

SM> For the "appear" part you could start with the trivial minor-mode below.
SM> But I think what Ted is after is the other part: down/up/...

Yes.  Appearance is not as important as usability to me, except where it
interferes with usability (e.g. lack of highlighting of the currently
selected candidate, etc.).

SM> While there is a risk of conflict in key-bindings, I do think it should
SM> be possible to "keep the best of both worlds" such that it could be
SM> enabled by default.

SM> E.g.: keep the same behavior by default as we have now, except that we
SM> add a new `select-completion-mode'.  In this new mode, cursor keys move
SM> between elements of the completions list, RET selects the completion
SM> element from there, and most other keys just exit the mode.

Sounds good.  For usability, it may be better to lock users into the
mode until they press `RET' or `C-g' or `ESC' (as expected).

SM> The main issue is then to figure out how/when to switch to this
SM> new mode.  E.g. when the user hits `up' right after the *Completions*
SM> buffer got displayed/updated?

Maybe the trigger should be another `TAB'?  That's what I would press,
intuitively.

Do you think this can realistically happen for 24.4?  I would be excited
to help test and give feedback, and maybe even help with the code.

Ted

Re: Emacs completion matches selection UI

[Eli Zaretskii]

> Date: Tue, 19 Nov 2013 16:00:49 +0200
> From: Dmitry Gutov <dgutov@yandex.ru>
> Cc: emacs-devel@gnu.org
> 
> In graphical mode, I think we can make a more reliable popup by adapting 
> some code in `tooltip.el'.

I'd advise against that.  Tooltip frames are treated very specially by
Emacs display, so you will be up for a lot of work (in C) to force
tooltips serve as completion pop-ups.

Re: Emacs completion matches selection UI

[Sebastian Wiesner]

2013/11/18 Ted Zlatanov <tzz@lifelogs.com>:
> In "(emacs.info) Completion Commands" there is help for using the
> rudimentary buffer to select completion matches.  I was struck by how
> hard to use this was, compared to more recent packages like these:
>
> http://www.emacswiki.org/emacs/AutoComplete
> http://www.emacswiki.org/emacs/CompanyMode
>
> Is there any chance the default Emacs appearance for selecting
> completion matches can be improved (not necessarily by default) for
> 24.4?  I would help with this but don't know what's "state of the art"
> and if there's already work in this direction.

You may want to look at the ido-at-point package at
https://github.com/katspaugh/ido-at-point, which uses IDO to prompt
for completion candidates in the minibuffer.

Re: Emacs completion matches selection UI

[Stefan Monnier]

> > proportional fonts?
> If having fundamental incompatibilities with some modes of operation is the
> worst problem we have, I'd say that's pretty good.

I do think it's pretty good, indeed.  But these problems are serious
enough to be a reason not to enable it by default (tho, it's not the
only reason, obviously).

Note that the new rectangle highlighting suffers from the same
problem ;-)  But there I have the excuse that "the highlighting just
reflects what C-x r k does, so it's weird but it's not the
highlighting's fault".

DG> In graphical mode, I think we can make a more reliable popup by
DG> adapting some code in `tooltip.el'. That would be incompatible with
DG> running in terminal, but there are no proportional fonts in terminal
DG> either (I think...?)

Sounds good.

> Works for me, as long as it's standard (so we don't have the current
> "reinvention of the wheel" everywhere).

Good point.

> Sounds good.  For usability, it may be better to lock users into the
> mode until they press `RET' or `C-g' or `ESC' (as expected).

The popup I see in Firefox lets me "keep on typing" without having to
hit C-g/ESC or any such thing.

SM> The main issue is then to figure out how/when to switch to this
SM> new mode.  E.g. when the user hits `up' right after the *Completions*
SM> buffer got displayed/updated?
> Maybe the trigger should be another `TAB'?  That's what I would press,
> intuitively.

Another TAB currently means "scroll the *Completions* buffer".
I'm not sure if we want to get rid of that feature.

I personally never use "up" and prefer M-p for that.  So I tend to think
it's OK to hijack `up' and `down' to enter the *completions* when it is
displayed (tho I think the way this new mode should work, it should not
switch to the *Completions* window, but instead just change key-bindings
in the minibuffer to highlight/select an entry in *Completions*).

> Do you think this can realistically happen for 24.4?  I would be excited
> to help test and give feedback, and maybe even help with the code.

The freeze is planned for mid-december so there's no much time.  OTOH it
might not take that much time to write either.


        Stefan

Re: Emacs completion matches selection UI

[Stefan Monnier]

> The second problem is that there are no alternatives to the default
> match selection UI.  Your company-mode integration would help there.
> Please let me know when it's ready for testing.

There's nothing special there: just the company-capf in the latest
company package.

> Remember you can also see the completion candidates buffer without the
> minibuffer, if you e.g. do `complete-symbol' in a buffer.

I think you mean `completion-at-point'.  Indeed, that is a lot more
tricky, since in such a context, keys can have any number of
other bindings, and it's quite normal to hit `up' or `down' with the
intention of "move to the other line because I'm done writing this
completable element", so hijacking `up' or `down' in this context is
more delicate.

But I'm not completely sure whether the minibuffer completion UI has to be
identical to the in-buffer completion UI.  Of course, it's good for them
to be identical, all things being equal, but: things like company-mode don't
work well in the minibuffer, whereas the separate *Completions* buffer
actually works OK in that case (whereas it's much more problematic for
in-buffer completion where the *Completions* buffer may be fairly far
from point).

> The most important thing IMO is to avoid making a new buffer that
> requires `C-x o' and magical bindings.

My suggestion is to keep the separate buffer but let the user interact
with it without needing to switch to it via C-x o or some such action.

That seems doable without too much trouble in the current setup.  If we
want to avoid the separate buffer altogether, then we're in
auto-complete/company-mode territory, which won't happen for 24.4.


        Stefan

Re: Emacs completion matches selection UI

[Dmitry Gutov]

Juri Linkov <juri@jurta.org> writes:

> In modern UIs, auto-completion works such a way that after you type text
> it pops up a drop-down menu with a list of candidates where you can
> select one by using <down> and <up> arrow keys.

I would differentiate between completion of buffer contents (in this
case displaying a drop-down menu is fine) and completion of strings in
general. If the completion is unrelated to the buffer contents (or only
related on some higher level, like file names, buffer name or IMenu
elements), at the very least the completion popup will have to be
displayed near the minibuffer.

I'd like to see a popup in the middle of the frame for this case,
somewhat-but-not-entirely covering the buffers. Like in Sublime Text.

You can see it displayed in examples 3 and 4 on their front page:
http://www.sublimetext.com/

Re: Emacs completion matches selection UI

[Ted Zlatanov]

On Tue, 19 Nov 2013 12:50:38 -0500 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 

SM> The main issue is then to figure out how/when to switch to this
SM> new mode.  E.g. when the user hits `up' right after the *Completions*
SM> buffer got displayed/updated?
>> Maybe the trigger should be another `TAB'?  That's what I would press,
>> intuitively.

SM> Another TAB currently means "scroll the *Completions* buffer".
SM> I'm not sure if we want to get rid of that feature.
...
SM> (tho I think the way this new mode should work, it should not
SM> switch to the *Completions* window, but instead just change key-bindings
SM> in the minibuffer to highlight/select an entry in *Completions*).

Agreed, although then the *Completions* buffer should at least get
better highlighting.

>> The second problem is that there are no alternatives to the default
>> match selection UI.  Your company-mode integration would help there.
>> Please let me know when it's ready for testing.

SM> There's nothing special there: just the company-capf in the latest
SM> company package.

OK, thanks.

>> Remember you can also see the completion candidates buffer without the
>> minibuffer, if you e.g. do `complete-symbol' in a buffer.

SM> I think you mean `completion-at-point'.  Indeed, that is a lot more
SM> tricky, since in such a context, keys can have any number of
SM> other bindings, and it's quite normal to hit `up' or `down' with the
SM> intention of "move to the other line because I'm done writing this
SM> completable element", so hijacking `up' or `down' in this context is
SM> more delicate.

I agree, and my proposal is to "lock in" the user into the selection
mode until it's done.  If we do that, it becomes OK to hijack primary
motion commands until the selection is made.  In effect the buffer
should be unavailable during the selection.

SM> But I'm not completely sure whether the minibuffer completion UI has to be
SM> identical to the in-buffer completion UI.  Of course, it's good for them
SM> to be identical, all things being equal, but: things like company-mode don't
SM> work well in the minibuffer, whereas the separate *Completions* buffer
SM> actually works OK in that case (whereas it's much more problematic for
SM> in-buffer completion where the *Completions* buffer may be fairly far
SM> from point).

I understand.  I think an abstract popup facility would work in either
case, especially if it "locks in" the user as described above.  But if
something must be improved, it's the in-buffer completion UI IMO.

>> The most important thing IMO is to avoid making a new buffer that
>> requires `C-x o' and magical bindings.

SM> My suggestion is to keep the separate buffer but let the user interact
SM> with it without needing to switch to it via C-x o or some such action.

SM> That seems doable without too much trouble in the current setup.  If we
SM> want to avoid the separate buffer altogether, then we're in
SM> auto-complete/company-mode territory, which won't happen for 24.4.

OK, let's pursue that path for 24.4 and then see if a more radical
library or API can be used later.

I see the following 24.4 tasks so far:

- improve *Completions* buffer highlighting

- decide on "locking in" users during candidate selection (so more
  natural keybindings can be used) or stealing just a few keybindings
  temporarily

- improve minibuffer candidate selection UI keys (depends on "lock in" decision)

- improve in-buffer candidate selection UI keys (depends on "lock in" decision)

Does that sound reasonable?

Ted

Re: Emacs completion matches selection UI

[Gregor Zattler]

Hi Ted,
* Ted Zlatanov <tzz@lifelogs.com> [19. Nov. 2013]:

[... deleted 13 lines...]
> , but honestly any solution that pops
> up a whole new buffer is unusable today in 2013.

Buffers are great because the user is allowed to do everything
s/he can do with other text.[1]  Perhaps the problem lies in the
switching to/from and quitting of said buffers?

Ciao; Gregor
[1] see https://sites.google.com/site/steveyegge2/effective-emacs#TOC-Dialog-Boxes:-The-Root-of-All-Evil

Re: Emacs completion matches selection UI

[Stefan Monnier]

> Agreed, although then the *Completions* buffer should at least get
> better highlighting.

Not sure what you want here.  AFAIK the only thing missing is to
highlight the selected entry (which is already done when selecting with
the mouse, but not when selecting with the keyboard).

SM> I think you mean `completion-at-point'.  Indeed, that is a lot more
SM> tricky, since in such a context, keys can have any number of
SM> other bindings, and it's quite normal to hit `up' or `down' with the
SM> intention of "move to the other line because I'm done writing this
SM> completable element", so hijacking `up' or `down' in this context is
SM> more delicate.

> I agree, and my proposal is to "lock in" the user into the selection
> mode until it's done.

Once you're in that mode, we can use any number of key-bindings, there's
no problem there.  The issue is how to let the use *enter* that mode,
where `up' or `down' can be used in the minibuffer, but maybe not
for in-buffer completion.  Once we're in that mode, up, down, and pretty
much anything else can be hijacked just fine.

> - decide on "locking in" users during candidate selection (so more
>   natural keybindings can be used) or stealing just a few keybindings
>   temporarily

"just a few" can be many keybindings.  My preference is simply to let
the user exit that mode "seamlessly" without having to use a special
exit key-binding.  But that doesn't mean that we should feel restricted
in which keys can be used in that special mode.

> Does that sound reasonable?

Yes.


        Stefan

Re: Emacs completion matches selection UI

[Juri Linkov]

> The main issue is then to figure out how/when to switch to this
> new mode.  E.g. when the user hits `up' right after the *Completions*
> buffer got displayed/updated?

It seems `up' is the most intuitive key to switch to the
*Completions* buffer.  This is how auto-completion works
in web browsers - typing `down' switches from the text field
to the selection list where up/down moves between candidates.
Moving back from the first candidate switches back from the
selection list to the text entry field.  In the minibuffer this
would mean `up' switching to the *Completions* buffer whereas
`down' at the bottom of the *Completions* buffer switching back
to the minibuffer.

The bigger problem is in which order to sort completion candidates.
When navigating from the minibuffer to the *Completions* buffer
`up' should select the first candidate which is more natural to put
at the bottom of the *Completions* buffer sorted in reverse order.

OTOH, to navigate in the forward order from top to bottom
the *Completions* buffer could be displayed below the minibuffer
if this is possible.

Re: Emacs completion matches selection UI

[Ted Zlatanov]

On Wed, 20 Nov 2013 01:10:47 +0100 Gregor Zattler <telegraph@gmx.net> wrote: 

GZ> * Ted Zlatanov <tzz@lifelogs.com> [19. Nov. 2013]:

>> , but honestly any solution that pops
>> up a whole new buffer is unusable today in 2013.

GZ> Buffers are great because the user is allowed to do everything
GZ> s/he can do with other text.[1]  Perhaps the problem lies in the
GZ> switching to/from and quitting of said buffers?

GZ> [1] see https://sites.google.com/site/steveyegge2/effective-emacs#TOC-Dialog-Boxes:-The-Root-of-All-Evil

Selecting a completion candidate is a very specific use case; the
content is not text but a menu of options.  In any case, I'm not against
buffers but against the general UI flow, which requires either mouse
clicks or hitting `C-x o'.

Ted

Re: Emacs completion matches selection UI

[Ted Zlatanov]

On Tue, 19 Nov 2013 19:28:28 -0500 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 

>> Agreed, although then the *Completions* buffer should at least get
>> better highlighting.

SM> Not sure what you want here.  AFAIK the only thing missing is to
SM> highlight the selected entry (which is already done when selecting with
SM> the mouse, but not when selecting with the keyboard).

That, and to also highlight the portion of the selected entry that
matches if the match is partial.

>> I agree, and my proposal is to "lock in" the user into the selection
>> mode until it's done.

SM> Once you're in that mode, we can use any number of key-bindings, there's
SM> no problem there.  The issue is how to let the use *enter* that mode,
SM> where `up' or `down' can be used in the minibuffer, but maybe not
SM> for in-buffer completion.  Once we're in that mode, up, down, and pretty
SM> much anything else can be hijacked just fine.

I am OK with any simple key chord to enter it; `TAB' or `right' seem
most intuitive to me.  Juri suggests `up' which is also OK.

>> - decide on "locking in" users during candidate selection (so more
>> natural keybindings can be used) or stealing just a few keybindings
>> temporarily

SM> "just a few" can be many keybindings.  My preference is simply to let
SM> the user exit that mode "seamlessly" without having to use a special
SM> exit key-binding.  But that doesn't mean that we should feel restricted
SM> in which keys can be used in that special mode.

OK, I like that!

On Wed, 20 Nov 2013 02:45:50 +0200 Juri Linkov <juri@jurta.org> wrote: 

JL> The bigger problem is in which order to sort completion candidates.
JL> When navigating from the minibuffer to the *Completions* buffer
JL> `up' should select the first candidate which is more natural to put
JL> at the bottom of the *Completions* buffer sorted in reverse order.

JL> OTOH, to navigate in the forward order from top to bottom
JL> the *Completions* buffer could be displayed below the minibuffer
JL> if this is possible.

Good points, I don't know what's the best approach here.

Ted

Re: Emacs completion matches selection UI

[Stefan Monnier]

> That, and to also highlight the portion of the selected entry that
> matches if the match is partial.

Hmm... don't we do that already?

> Good points, I don't know what's the best approach here.

We can start with: `up' goes to the last displayed line of the
*Completions* buffer (on the assumption that it's displayed above the
minibuffer) and `down' goes to the first displayed line.


        Stefan

Re: Emacs completion matches selection UI

[martin rudalics]

 > The bigger problem is in which order to sort completion candidates.
 > When navigating from the minibuffer to the *Completions* buffer
 > `up' should select the first candidate which is more natural to put
 > at the bottom of the *Completions* buffer sorted in reverse order.

We could enter/leave the selection list at the top and do everything
else exactly like in browser but for the fact that the list is drawn
above the minibuffer.

martin

Re: Emacs completion matches selection UI

[Eric Abrahamsen]

Dmitry Gutov <dgutov@yandex.ru> writes:

> Juri Linkov <juri@jurta.org> writes:
>
>> In modern UIs, auto-completion works such a way that after you type text
>> it pops up a drop-down menu with a list of candidates where you can
>> select one by using <down> and <up> arrow keys.
>
> I would differentiate between completion of buffer contents (in this
> case displaying a drop-down menu is fine) and completion of strings in
> general. If the completion is unrelated to the buffer contents (or only
> related on some higher level, like file names, buffer name or IMenu
> elements), at the very least the completion popup will have to be
> displayed near the minibuffer.
>
> I'd like to see a popup in the middle of the frame for this case,
> somewhat-but-not-entirely covering the buffers. Like in Sublime Text.
>
> You can see it displayed in examples 3 and 4 on their front page:
> http://www.sublimetext.com/

I think this feature alone would put an end to many of the complaints
about emacs being "not modern". If you're constitutionally opposed to
shiny it could be objectionable, but practically-speaking it would be
very nice. The less emacs look like Dwarf Fortress the better.

Re: Emacs completion matches selection UI

[Tom]

martin rudalics <rudalics <at> gmx.at> writes:
> We could enter/leave the selection list at the top and do everything
> else exactly like in browser but for the fact that the list is drawn
> above the minibuffer.

Actually, this is exactly what the browser does if the completion list
cannot be show below the url field.

Try making your browser window very small by pulling its top to the
bottom, so that the url field is at the bottom of the screen. You can
also simply move the whole browser window down, so the url field is 
at the bottom of the screen and the rest of the browser window is
outside of the screen if your window manager allows it.

If you start typing an url with this setup then the completion popup is
shown above the url field, instead of below it. (Tested on Windows
with Firefox.)

Re: Emacs completion matches selection UI

[Josh]

On Tue, Nov 19, 2013 at 4:28 PM, Stefan Monnier
<monnier@iro.umontreal.ca> wrote:
>> - decide on "locking in" users during candidate selection (so more
>>   natural keybindings can be used) or stealing just a few keybindings
>>   temporarily
>
> "just a few" can be many keybindings.  My preference is simply to let
> the user exit that mode "seamlessly" without having to use a special
> exit key-binding.  But that doesn't mean that we should feel restricted
> in which keys can be used in that special mode.

Many people access `previous-history-element' in the minibuffer through
the up arrow, so I hope it is not commandeered for this purpose.  Instead,
what about entering the mode from the minibuffer via C-f and right arrow,
but only when the point is at the minibuffer's point-max (i.e. when
`forward-char' would currently complain "End of buffer")?  Besides being
less disruptive for users than changing the behavior of <Up>, using
C-f/<right> for this purpose has the mnemonic benefit of suggesting
"go beyond what I've typed".  WDYT?

Re: Emacs completion matches selection UI

[Ted Zlatanov]

On Tue, 19 Nov 2013 22:22:27 -0500 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 

>> That, and to also highlight the portion of the selected entry that
>> matches if the match is partial.

SM> Hmm... don't we do that already?

Not from in-buffer completion AFAICT.  Yes from minibuffer completion.

>> Good points, I don't know what's the best approach here.

SM> We can start with: `up' goes to the last displayed line of the
SM> *Completions* buffer (on the assumption that it's displayed above the
SM> minibuffer) and `down' goes to the first displayed line.

On Wed, 20 Nov 2013 07:57:35 -0800 Josh <josh@foxtail.org> wrote: 

J> Many people access `previous-history-element' in the minibuffer through
J> the up arrow, so I hope it is not commandeered for this purpose.  Instead,
J> what about entering the mode from the minibuffer via C-f and right arrow,
J> but only when the point is at the minibuffer's point-max (i.e. when
J> `forward-char' would currently complain "End of buffer")?  Besides being
J> less disruptive for users than changing the behavior of <Up>, using
J> C-f/<right> for this purpose has the mnemonic benefit of suggesting
J> "go beyond what I've typed".  WDYT?

After thinking about it, I agree with Josh.  Proposal:

1) in minibuffer completion:

`right' or `C-f' at minibuffer point-max enters completion candidates
buffer (where then `up' and `down' are remapped, we have the user captive)

`left' in the completion candidates buffer goes back to the minibuffer

2) in-buffer completion:

enter completion candidates buffer immediately (where then `up' and
`down' are remapped, we have the user captive).  Make it easy to get out
and back to the original buffer.

Ted

Re: Emacs completion matches selection UI

[John Yates]

[-- Attachment #1: Type: text/plain, Size: 651 bytes --]

On Tue, Nov 19, 2013 at 8:25 PM, Ted Zlatanov <tzz@lifelogs.com> wrote:

> I'm not against buffers but against the general UI flow, which requires
> either mouse clicks or hitting `C-x o'.
>

For me the crucial difference between mini-buffer completion and pop is
that one does not alter the layout of my windows while the other does.
 Have a helpful list of suggestions to the right while my cursor remains
undisturbed is very intuitive.  By contrast having a new window pop up
moves _my_ focus but fails to move emacs' focus.  I would be very happy if,
having configured emacs to pop a completion buffer emacs' focus moved there
immediately.

/john

[-- Attachment #2: Type: text/html, Size: 1044 bytes --]

Re: Emacs completion matches selection UI

[Stefan Monnier]

>>> That, and to also highlight the portion of the selected entry that
>>> matches if the match is partial.
SM> Hmm... don't we do that already?
> Not from in-buffer completion AFAICT.  Yes from minibuffer completion.

I see the same highlighting for in-buffer completion as for
minibuffer completion.  So please make a bug-report when you see
the problem.  It's probably an in-buffer completion which still uses
ad-hoc code rather than relying on the completion-at-point infrastructure.

> After thinking about it, I agree with Josh.  Proposal:
> 1) in minibuffer completion:
> `right' or `C-f' at minibuffer point-max enters completion candidates
> buffer (where then `up' and `down' are remapped, we have the user captive)
> `left' in the completion candidates buffer goes back to the minibuffer
> 2) in-buffer completion:
> enter completion candidates buffer immediately (where then `up' and
> `down' are remapped, we have the user captive).  Make it easy to get out
> and back to the original buffer.

Maybe "enter immediately" is also an option for the minibuffer case.
I agree that C-f at EOL in the minibuffer is a "safe" choice, but
I don't think that hijacking `up' and `down' would be problematic
either, because we'd only hijack them after displaying *Completions*,
and I expect users tend to rely on the history only *before* displaying
the *Completions*, but not so much afterwards.


        Stefan

Re: Emacs completion matches selection UI

[Ted Zlatanov]

On Wed, 20 Nov 2013 14:10:18 -0500 Stefan Monnier <monnier@IRO.UMontreal.CA> wrote: 

>>>> That, and to also highlight the portion of the selected entry that
>>>> matches if the match is partial.
SM> Hmm... don't we do that already?
>> Not from in-buffer completion AFAICT.  Yes from minibuffer completion.

SM> I see the same highlighting for in-buffer completion as for
SM> minibuffer completion.  So please make a bug-report when you see
SM> the problem.  It's probably an in-buffer completion which still uses
SM> ad-hoc code rather than relying on the completion-at-point
SM> infrastructure.

OK, I'll try to replicate it, thanks.

>> After thinking about it, I agree with Josh.  Proposal:
>> 1) in minibuffer completion:
>> `right' or `C-f' at minibuffer point-max enters completion candidates
>> buffer (where then `up' and `down' are remapped, we have the user captive)
>> `left' in the completion candidates buffer goes back to the minibuffer
>> 2) in-buffer completion:
>> enter completion candidates buffer immediately (where then `up' and
>> `down' are remapped, we have the user captive).  Make it easy to get out
>> and back to the original buffer.

SM> Maybe "enter immediately" is also an option for the minibuffer case.

It would work for me.  After `TAB' I expect either an immediate
completion if there's just one candidate, or an error if there are none,
or the completion candidates selection UI.

SM> I agree that C-f at EOL in the minibuffer is a "safe" choice, but
SM> I don't think that hijacking `up' and `down' would be problematic
SM> either, because we'd only hijack them after displaying *Completions*,
SM> and I expect users tend to rely on the history only *before* displaying
SM> the *Completions*, but not so much afterwards.

I understand now.  Yes, that would work fine.

Ted

Re: Emacs completion matches selection UI

[Juri Linkov]

>> The bigger problem is in which order to sort completion candidates.
>> When navigating from the minibuffer to the *Completions* buffer
>> `up' should select the first candidate which is more natural to put
>> at the bottom of the *Completions* buffer sorted in reverse order.
>
> We could enter/leave the selection list at the top and do everything
> else exactly like in browser but for the fact that the list is drawn
> above the minibuffer.

To display the minibuffer completions near the minibuffer I tried
this customization:

(add-to-list 'display-buffer-alist '("\\*Completions\\*" display-buffer-at-bottom (nil)))

but it works only in one configuration with two horizontally split windows.
I expected that this action was intended to display the buffer always
above the minibuffer.

OTOH, for in-buffer completions it seems a better action is

(add-to-list 'display-buffer-alist '("\\*Completions\\*" display-buffer-below-selected (nil)))

that could work like `dired-mark-pop-up' to display completions near point
as close to it as possible.  Maybe adding (window-height . fit-window-to-buffer)
will make it better formatted too.  This will imitate overlay-based auto-completions
by using the standard buffer.

Re: Emacs completion matches selection UI

[Juri Linkov]

> Many people access `previous-history-element' in the minibuffer through
> the up arrow, so I hope it is not commandeered for this purpose.  Instead,
> what about entering the mode from the minibuffer via C-f and right arrow,
> but only when the point is at the minibuffer's point-max (i.e. when
> `forward-char' would currently complain "End of buffer")?  Besides being
> less disruptive for users than changing the behavior of <Up>, using
> C-f/<right> for this purpose has the mnemonic benefit of suggesting
> "go beyond what I've typed".  WDYT?

The web browsers solve this problem by combining completions for
history and suggestions in the same list (separated by a horizontal line).
With history elements displayed in the *Completions* buffer you can use
the same up and down arrows to access them.

Re: Emacs completion matches selection UI

[Eli Zaretskii]

> From: Juri Linkov <juri@jurta.org>
> Date: Thu, 21 Nov 2013 02:52:30 +0200
> Cc: Stefan Monnier <monnier@iro.umontreal.ca>,
> 	emacs-devel <emacs-devel@gnu.org>
> 
> > Many people access `previous-history-element' in the minibuffer through
> > the up arrow, so I hope it is not commandeered for this purpose.  Instead,
> > what about entering the mode from the minibuffer via C-f and right arrow,
> > but only when the point is at the minibuffer's point-max (i.e. when
> > `forward-char' would currently complain "End of buffer")?  Besides being
> > less disruptive for users than changing the behavior of <Up>, using
> > C-f/<right> for this purpose has the mnemonic benefit of suggesting
> > "go beyond what I've typed".  WDYT?
> 
> The web browsers solve this problem by combining completions for
> history and suggestions in the same list (separated by a horizontal line).

The problem with that is that the resulting list is frustratingly
long.

Re: Emacs completion matches selection UI

[martin rudalics]

> To display the minibuffer completions near the minibuffer I tried
> this customization:
> 
> (add-to-list 'display-buffer-alist '("\\*Completions\\*" display-buffer-at-bottom (nil)))
> 
> but it works only in one configuration with two horizontally split windows.
> I expected that this action was intended to display the buffer always
> above the minibuffer.

Sorry, I forgot.  What is it that does not work here?

martin

Re: Emacs completion matches selection UI

[Dmitry Gutov]

On 19.11.2013 18:24, Eli Zaretskii wrote:
>> In graphical mode, I think we can make a more reliable popup by adapting
>> some code in `tooltip.el'.
>
> I'd advise against that.  Tooltip frames are treated very specially by
> Emacs display, so you will be up for a lot of work (in C) to force
> tooltips serve as completion pop-ups.

Or maybe "just" use a new, chromeless frame. Anyway, I'd expect some C 
code to have to be written for that feature.

Without new frame, we also won't be able to make Company tooltip work in 
minibuffer, Maybe we can adapt by rendering it in the nearest window, 
but that will leave a somewhat hackish impression, even if it works well.

Re: Emacs completion matches selection UI

[Eli Zaretskii]

> Date: Thu, 21 Nov 2013 12:17:17 +0200
> From: Dmitry Gutov <dgutov@yandex.ru>
> CC: monnier@iro.umontreal.ca, emacs-devel@gnu.org
> 
> On 19.11.2013 18:24, Eli Zaretskii wrote:
> >> In graphical mode, I think we can make a more reliable popup by adapting
> >> some code in `tooltip.el'.
> >
> > I'd advise against that.  Tooltip frames are treated very specially by
> > Emacs display, so you will be up for a lot of work (in C) to force
> > tooltips serve as completion pop-ups.
> 
> Or maybe "just" use a new, chromeless frame.

XEmacs did that many years ago (Stephen can tell if it still does).  I
didn't like it then: a new frame forces you to switch focus to it, at
least with some window managers.  That slows you down, and thus flies
in the face of the very reason to provide completion -- fast typing of
long strings.  A popup frame also obscures the window you are working
in, which is also a nuisance.

But that's me; YMMV.

> Anyway, I'd expect some C code to have to be written for that
> feature.

Not every possible design of that will require that, and we would like
to minimize that anyway.

Re: Emacs completion matches selection UI

[Juri Linkov]

>> > Many people access `previous-history-element' in the minibuffer through
>> > the up arrow, so I hope it is not commandeered for this purpose.
>>
>> The web browsers solve this problem by combining completions for
>> history and suggestions in the same list (separated by a horizontal line).
>
> The problem with that is that the resulting list is frustratingly
> long.

The browsers put the most frequently visited history items at the top.
As I see icomplete does the same when displaying completions in the
minibuffer.  So items could be sorted either by frequency or by recency
in the *Completions* buffer as well.

Re: Emacs completion matches selection UI

[Juri Linkov]

>> To display the minibuffer completions near the minibuffer I tried
>> this customization:
>>
>> (add-to-list 'display-buffer-alist '("\\*Completions\\*" display-buffer-at-bottom (nil)))
>>
>> but it works only in one configuration with two horizontally split windows.
>> I expected that this action was intended to display the buffer always
>> above the minibuffer.
>
> Sorry, I forgot.  What is it that does not work here?

It doesn't display completions above the minibuffer in many
window configurations.  For instance, with a single window on
the frame it splits the window horizontally instead of vertically.

Re: Emacs completion matches selection UI

[Stephen J. Turnbull]

Eli Zaretskii writes:

 > > Or maybe "just" use a new, chromeless frame.
 > 
 > XEmacs did that many years ago (Stephen can tell if it still does).

It still does that.

 > I didn't like it then: a new frame forces you to switch focus to
 > it, at least with some window managers.

Never heard of that.  However, our frame management code is rather
different from Emacs's.  Maybe somebody found a way to avoid that (or
maybe XEmacs users don't use those window managers).

 > A popup frame also obscures the window you are working in, which is
 > also a nuisance.

Yeah, we've never fixed that.  Of course the developer can position
the popup by hand, but that sucks, too.  But for something the size of
a toolip (typically with relatively small font size, too) and no frame
decorations, it doesn't bug me anywhere as much, YMMV.

If you've got transparency working, maybe that would be worth trying
(transparent background, alpha=.5 foreground?)  Modern browsers also
often fadeout the popup gradually (I think that's an admission that
the design -- using a popup -- is fundamentally broken, but again YMMV).

Re: Emacs completion matches selection UI

[Eli Zaretskii]

> From: Juri Linkov <juri@jurta.org>
> Cc: josh@foxtail.org,  monnier@iro.umontreal.ca,  emacs-devel@gnu.org
> Date: Fri, 22 Nov 2013 02:10:48 +0200
> 
> >> > Many people access `previous-history-element' in the minibuffer through
> >> > the up arrow, so I hope it is not commandeered for this purpose.
> >>
> >> The web browsers solve this problem by combining completions for
> >> history and suggestions in the same list (separated by a horizontal line).
> >
> > The problem with that is that the resulting list is frustratingly
> > long.
> 
> The browsers put the most frequently visited history items at the top.

And therein lies the problem: I frequently cannot locate the history
item I need in that list.

> As I see icomplete does the same when displaying completions in the
> minibuffer.  So items could be sorted either by frequency or by recency
> in the *Completions* buffer as well.

That only solves part of the use cases.  My typical history even for a
single day is very long, and will many times defeat these strategies.
We need to have a solution for such situations, which I believe is not
uncommon in Emacs uses.

Re: Emacs completion matches selection UI

[martin rudalics]

> It doesn't display completions above the minibuffer in many
> window configurations.  For instance, with a single window on
> the frame it splits the window horizontally instead of vertically.

Here with emacs -Q

(add-to-list 'display-buffer-alist '("\\*Completions\\*" display-buffer-at-bottom (nil)))

(display-buffer (get-buffer-create "*Completions*"))

gets me the *Completions* window below the *scratch* window.

martin

Re: Emacs completion matches selection UI

[Ted Zlatanov]

On Fri, 22 Nov 2013 09:36:20 +0200 Eli Zaretskii <eliz@gnu.org> wrote: 

>> From: Juri Linkov <juri@jurta.org>
>> Cc: josh@foxtail.org,  monnier@iro.umontreal.ca,  emacs-devel@gnu.org
>> Date: Fri, 22 Nov 2013 02:10:48 +0200
>> 
>> >> > Many people access `previous-history-element' in the minibuffer through
>> >> > the up arrow, so I hope it is not commandeered for this purpose.
>> >>
>> >> The web browsers solve this problem by combining completions for
>> >> history and suggestions in the same list (separated by a horizontal line).
>> >
>> > The problem with that is that the resulting list is frustratingly
>> > long.
>> 
>> The browsers put the most frequently visited history items at the top.

EZ> And therein lies the problem: I frequently cannot locate the history
EZ> item I need in that list.

I think it can be addressed: `up' goes into the popup, starting with the
history items in the order you expect.  `down' also goes into the popup
but starts with the completion candidates.  So if you want to select the
first completion candidate, you'd use `down RET'.  And for the first
history item, `up RET'.  That, plus searching (see below) would IMO be
a good UI.

>> As I see icomplete does the same when displaying completions in the
>> minibuffer.  So items could be sorted either by frequency or by recency
>> in the *Completions* buffer as well.

EZ> That only solves part of the use cases.  My typical history even for a
EZ> single day is very long, and will many times defeat these strategies.
EZ> We need to have a solution for such situations, which I believe is not
EZ> uncommon in Emacs uses.

Once you've entered the completion candidates selection UI, regular
letter keys can be used to filter the list by substring because you're
in a new keymap.  Like `Control-R' in a shell with libreadline history.

Ted

Re: Emacs completion matches selection UI

[Eli Zaretskii]

> From: "Stephen J. Turnbull" <stephen@xemacs.org>
> Cc: Dmitry Gutov <dgutov@yandex.ru>,
>     monnier@iro.umontreal.ca,
>     emacs-devel@gnu.org
> Date: Fri, 22 Nov 2013 14:38:12 +0900
> 
>  > A popup frame also obscures the window you are working in, which is
>  > also a nuisance.
> 
> Yeah, we've never fixed that.  Of course the developer can position
> the popup by hand, but that sucks, too.  But for something the size of
> a toolip (typically with relatively small font size, too) and no frame
> decorations, it doesn't bug me anywhere as much, YMMV.

Current tooltips are very small: 3-4 short lines at most.  If they are
used for completion, they will need to become much larger, and will
probably need scroll bars.  This begins to sound like a real frame.

> If you've got transparency working, maybe that would be worth trying
> (transparent background, alpha=.5 foreground?)

I find that I'm annoyed by transparency: it makes it hard for me to
catch the stuff at a glance.  But maybe it's just a thing that comes
with age.

Re: Emacs completion matches selection UI

[Stephen J. Turnbull]

Eli Zaretskii writes:

 > I find that I'm annoyed by transparency: it makes it hard for me to
 > catch the stuff at a glance.  But maybe it's just a thing that comes
 > with age.

Sounds to me like you just can't win. ;-) Seriously, I think your
feelings about the alternatives are a symptom indicating the popup
idea is poor design.  We should be able to do better.

Re: Emacs completion matches selection UI

[Ted Zlatanov]

On Fri, 22 Nov 2013 07:33:33 -0500 Ted Zlatanov <tzz@lifelogs.com> wrote: 

TZ> On Fri, 22 Nov 2013 09:36:20 +0200 Eli Zaretskii <eliz@gnu.org> wrote: 
>>> From: Juri Linkov <juri@jurta.org>
>>> Cc: josh@foxtail.org,  monnier@iro.umontreal.ca,  emacs-devel@gnu.org
>>> Date: Fri, 22 Nov 2013 02:10:48 +0200
>>> 
>>> >> > Many people access `previous-history-element' in the minibuffer through
>>> >> > the up arrow, so I hope it is not commandeered for this purpose.
>>> >>
>>> >> The web browsers solve this problem by combining completions for
>>> >> history and suggestions in the same list (separated by a horizontal line).
>>> >
>>> > The problem with that is that the resulting list is frustratingly
>>> > long.
>>> 
>>> The browsers put the most frequently visited history items at the top.

EZ> And therein lies the problem: I frequently cannot locate the history
EZ> item I need in that list.

TZ> I think it can be addressed: `up' goes into the popup, starting with the
TZ> history items in the order you expect.  `down' also goes into the popup
TZ> but starts with the completion candidates.  So if you want to select the
TZ> first completion candidate, you'd use `down RET'.  And for the first
TZ> history item, `up RET'.  That, plus searching (see below) would IMO be
TZ> a good UI.

>>> As I see icomplete does the same when displaying completions in the
>>> minibuffer.  So items could be sorted either by frequency or by recency
>>> in the *Completions* buffer as well.

EZ> That only solves part of the use cases.  My typical history even for a
EZ> single day is very long, and will many times defeat these strategies.
EZ> We need to have a solution for such situations, which I believe is not
EZ> uncommon in Emacs uses.

TZ> Once you've entered the completion candidates selection UI, regular
TZ> letter keys can be used to filter the list by substring because you're
TZ> in a new keymap.  Like `Control-R' in a shell with libreadline history.

From the lack of followups I think we've settled into some agreement on
this: trigger the completion candidates UI on `up' and/or `down' and
inside that mode, remap keys for the single purpose of filtering and
selecting match candidates.  We also said the input history should be
available in the same interface, so the UI will in effect be a general
"give me things of interest" interface.  The visuals (popup or better
*Completions* buffer) are not clear yet, but we have some votes for
either approach.

I don't know this area of the code well, but can probably give it a try
in a while.  Is anyone else able to attack this problem soon-ish?  I
would gladly assist with testing and documentation, debugging, etc.

Thanks
Ted

Re: Emacs completion matches selection UI

[Stefan Monnier]

> From the lack of followups I think we've settled into some agreement on
> this: trigger the completion candidates UI on `up' and/or `down' and
> inside that mode, remap keys for the single purpose of filtering and
> selecting match candidates.

That sounds fine, yes.

> We also said the input history should be available in the same
> interface, so the UI will in effect be a general "give me things of
> interest" interface.

I suggest you start by simply using the *Completions* buffer unchanged
(i.e. without input history).

> The visuals (popup or better *Completions* buffer) are not clear yet,
> but we have some votes for either approach.

I don't think it's an either/or choice.  But I don't think the
popup code will have time to mature enough for 24.4.


        Stefan

Re: Emacs completion matches selection UI

[Ted Zlatanov]

I looked carefully at the `completion-in-region-*' functions and
especially the `completion-in-region-mode' and its keymap.  This is for
the UI case of "I'm in a buffer and invoke `completion-at-point'".

It's a real mess.  We do a ton of special magic to avoid having a
special popup in favor of a *Completions* buffer.  The comment on
`completion-in-region--postch' says it pretty clearly:

;; It is difficult to know when to exit completion-in-region-mode (i.e. hide
;; the *Completions*).
;; - lisp-mode: never.
;; - comint: only do it if you hit SPC at the right time.
;; - pcomplete: pop it down on SPC or after some time-delay.
;; - semantic: use a post-command-hook check similar to this one.

If there was a special popup to show a list of items, like you see in
this snapshot of company-mode
http://company-mode.github.io/images/company-elisp.png , it would
abstract these difficulties and cut some nasty code out of
minibuffer.el.

We already have such popups natively for Customize with `widget-choose'
(supporting text in a nice buffer similar to *Completions* but with key
shortcuts and nice help test, and graphical UI with native popups) so I
hope it's not terribly hard to use that code more generally.  It seems
easier to improve `widget-choose' than to keep hacking on the special
*Completions* cases.

For the UI case of "I'm in the minibuffer and want to complete
something" I would rather wait for the general list popup to be
available with `widget-choose' than add the special down/up keybindings
that we discussed.  It will be a better UI in every way than the
*Completions* buffer IMO.

If you agree, I can pursue this after the upcoming release.  I think the
change is too risky for the short time we have this week.

Ted

Re: Emacs completion matches selection UI

[Stefan Monnier]

> ;; It is difficult to know when to exit completion-in-region-mode (i.e. hide
> ;; the *Completions*).
[...]
> If there was a special popup to show a list of items, [...], it would
> abstract these difficulties and cut some nasty code out of
> minibuffer.el.

No, it wouldn't.  The problem needs to be solved either way.
`company-mode' solves this problem not by popping up a special GUI
element, but by defining which operations can be performed while this
element is displayed, and which operations cause the element
to disappear.  And completion-in-region-mode does the same, tho using
the *Completions* buffer rather than a special GUI element, and making
different choices about which operations can be performed while the
completions are displayed and which operations make the
completions disappear.

> For the UI case of "I'm in the minibuffer and want to complete
> something" I would rather wait for the general list popup to be
> available with `widget-choose' than add the special down/up keybindings
> that we discussed.

Again, this has nothing to do with whether the list is displayed in
a *Completions* buffer or on a shoestring.  The questions are "when
should the list be displayed", and after that, "when should we enter the
special mode where the user can navigate in this list".
For company-mode, the second question is answered by "immediately" and
the first is answered by "after typing the first N chars of a completion
candidate" (IIRC).

> I think the change is too risky for the short time we have this week.

I'm not sure which change you're thinking of, but I probably agree.


        Stefan

Re: Emacs completion matches selection UI

[Ted Zlatanov]

On Mon, 16 Dec 2013 13:35:47 -0500 Stefan Monnier <monnier@IRO.UMontreal.CA> wrote: 

>> ;; It is difficult to know when to exit completion-in-region-mode (i.e. hide
>> ;; the *Completions*).
SM> [...]
>> If there was a special popup to show a list of items, [...], it would
>> abstract these difficulties and cut some nasty code out of
>> minibuffer.el.

SM> No, it wouldn't.  The problem needs to be solved either way.

SM> `company-mode' solves this problem not by popping up a special GUI
SM> element, but by defining which operations can be performed while this
SM> element is displayed, and which operations cause the element
SM> to disappear.  And completion-in-region-mode does the same, tho using
SM> the *Completions* buffer rather than a special GUI element, and making
SM> different choices about which operations can be performed while the
SM> completions are displayed and which operations make the
SM> completions disappear.

I am trying to say that Emacs itself, outside of the
window/frame/etc. logic, should have code to "show a list of choices and
pick one."  An API to list selection, from the minibuffer or from a
buffer, with consistent key bindings and no dependence on the context in
which it was invoked.  This seems to be what `widget-choose' could do
with some polish, and that's where I'd put effort.  Do you agree?  To
me, It seems crazy that there is no standard way to do this and that so
many packages, internal and external, have invented their own UI to do it.

The choices about available operations during selections and which
operations should cancel or complete the selection should be entirely
inside an API like `widget-choose' and *standard*.  I think the trigger
to bring the selection UI up is the only thing that should be up to the
client.  In the case of `completion-in-region-mode', do you see how it
would remove a large chunk of code that deals with post-change and key
customizations?  Or am I just not getting it?

>> For the UI case of "I'm in the minibuffer and want to complete
>> something" I would rather wait for the general list popup to be
>> available with `widget-choose' than add the special down/up keybindings
>> that we discussed.

SM> Again, this has nothing to do with whether the list is displayed in
SM> a *Completions* buffer or on a shoestring.  The questions are "when
SM> should the list be displayed", and after that, "when should we enter the
SM> special mode where the user can navigate in this list".

This was a separate UI case and I think we agreed (broadly) earlier on
these questions.  I am saying I'd rather wait to implement this until I
have a standard Emacs-wide API to display a list of candidates, so I am
not writing blocks of code to work around the currently implemented
*Completions* UI.  From `completion-in-region-mode' and reading the
other code around minibuffer.el I believe it will be hard to write
maintainable code to do this otherwise.

>> I think the change is too risky for the short time we have this week.

SM> I'm not sure which change you're thinking of, but I probably agree.

I meant the change to implement the second UI case, the one to make
up/down bring up the candidate selection UI as we discussed.

Ted

Re: Emacs completion matches selection UI

[Stefan Monnier]

> I am trying to say that Emacs itself, outside of the
> window/frame/etc. logic, should have code to "show a list of choices and
> pick one."  An API to list selection, from the minibuffer or from a
> buffer, with consistent key bindings and no dependence on the context in
> which it was invoked.  This seems to be what `widget-choose' could do
> with some polish, and that's where I'd put effort.  Do you agree?

I could agree, but do note that one of the main design goals of
completion-in-region-mode was that the mode is exited as a side-effect
of the user's normal editing.  IOW, the user is free to ignore the list
of completions and do whatever else she wants to do, rather than having
to somehow indicate explicitly "oh, never mind, I won't choose any of
these options".

But IIUC the "choice" API you suggest would allow something like
completion-in-region-mode as one possible UI.

> To me, It seems crazy that there is no standard way to do this and
> that so many packages, internal and external, have invented their own
> UI to do it.

There is one, which is completing-read.  But this one size doesn't
fit all.  Maybe there is some other one size which does, but maybe not.


        Stefan

Re: Emacs completion matches selection UI

[Ted Zlatanov]

On Mon, 16 Dec 2013 21:10:02 -0500 Stefan Monnier <monnier@IRO.UMontreal.CA> wrote: 

>> I am trying to say that Emacs itself, outside of the
>> window/frame/etc. logic, should have code to "show a list of choices and
>> pick one."  An API to list selection, from the minibuffer or from a
>> buffer, with consistent key bindings and no dependence on the context in
>> which it was invoked.  This seems to be what `widget-choose' could do
>> with some polish, and that's where I'd put effort.  Do you agree?

SM> I could agree, but do note that one of the main design goals of
SM> completion-in-region-mode was that the mode is exited as a side-effect
SM> of the user's normal editing.  IOW, the user is free to ignore the list
SM> of completions and do whatever else she wants to do, rather than having
SM> to somehow indicate explicitly "oh, never mind, I won't choose any of
SM> these options".

Yes, I understand.  But this is at odds with the way that
`completing-read' and `widget-choose' and most modern UIs work.  The
user has explicitly invoked a completion command, right?  Or is
`completion-in-region-mode' ever called implicitly?

SM> But IIUC the "choice" API you suggest would allow something like
SM> completion-in-region-mode as one possible UI.

Right, it wouldn't care about the invocation context, the "dismiss on
input when called in a buffer context" behavior would be an option.  For
those who like it, it can stay.  I don't think it should be the default.

>> To me, It seems crazy that there is no standard way to do this and
>> that so many packages, internal and external, have invented their own
>> UI to do it.

SM> There is one, which is completing-read.  But this one size doesn't
SM> fit all.  Maybe there is some other one size which does, but maybe not.

`completing-read' moves the focus of the user from point to the
minibuffer IIUC, and then makes the entire interaction
minibuffer-centric.  That seems to be the fundamental reason why we
invented `completion-in-region-mode', to avoid that context switch.
Otherwise yes, `completing-read' could be the list-selection API.

Could `completing-read' call `widget-choose' regardless of the
invocation point and show a graphical popup or a simple text buffer?
Even in the minibuffer I think that would look OK.

Or maybe `completing-read' could be called for in-buffer completion if
the minibuffer could somehow be moved closer to point?  Imagine if the
minibuffer could slide up to point during the completion (perhaps by
simply growing it upwards until it reaches point), then there wouldn't
be a disorienting focus switch.  Ideally the entire minibuffer would
slide up and look the same otherwise, but that's probably going to be
hard to implement.

Ted

Re: Emacs completion matches selection UI

[Stefan Monnier]

> Yes, I understand.  But this is at odds with the way that
> `completing-read' and `widget-choose' and most modern UIs work.

You mean it's feature that the other systems lack.  Yes, agreed ;-)

> The user has explicitly invoked a completion command, right?

There's no doubt that it can be made more modal, but it's more
convenient if the mode is exited seamlessly.  Historically, TAB
completion is non-mini buffers did not affect the subsequent behavior.
This was a convenient feature with the downside that the *Completions*
buffer was left around.  So I wanted to fix this problem while
preserving the property that the mode stays out of the way.

> Right, it wouldn't care about the invocation context, the "dismiss on
> input when called in a buffer context" behavior would be an option.  For
> those who like it, it can stay.  I don't think it should be the default.

I'm not sure why you see it as something the user wouldn't like.
More specifically, why would you want to force the user to use special
commands to leave the "completion choice mode"?
I must be missing something.

> `completing-read' moves the focus of the user from point to the
> minibuffer IIUC, and then makes the entire interaction
> minibuffer-centric.  That seems to be the fundamental reason why we
> invented `completion-in-region-mode', to avoid that context switch.

Indeed, using completing-read to complete text from a normal buffer
would be inconvenient (tho it's done occasionally).
`completion-in-region' is a lot more lightweight for the user, staying
out of the way as much as possible.

> Otherwise yes, `completing-read' could be the list-selection API.

I think "list-selection" is a restrictive way to think about it: hitting
TAB to complete doesn't necessarily mean "the user wants to select
a completion candidate".  It can also mean "the user wants to complete
"fo" to "fooba" and then complete this identifier by adding something
else that's not in any of the completion candidates.

> Could `completing-read' call `widget-choose' regardless of the
> invocation point and show a graphical popup or a simple text buffer?
> Even in the minibuffer I think that would look OK.

We wouldn't use completing-read, for API reasons.  But indeed, the
minibuffer.el code is slowly moving towards using the same completion
mechanism as in other buffers (completion-at-point).

> Or maybe `completing-read' could be called for in-buffer completion if
> the minibuffer could somehow be moved closer to point?  Imagine if the
> minibuffer could slide up to point during the completion (perhaps by
> simply growing it upwards until it reaches point), then there wouldn't
> be a disorienting focus switch.  Ideally the entire minibuffer would
> slide up and look the same otherwise, but that's probably going to be
> hard to implement.

Still very disruptive.


        Stefan

Re: Emacs completion matches selection UI

[Ted Zlatanov]

On Tue, 17 Dec 2013 13:29:30 -0500 Stefan Monnier <monnier@IRO.UMontreal.CA> wrote: 

>> Right, it wouldn't care about the invocation context, the "dismiss on
>> input when called in a buffer context" behavior would be an option.  For
>> those who like it, it can stay.  I don't think it should be the default.

SM> I'm not sure why you see it as something the user wouldn't like.

Because it's not familiar.

SM> More specifically, why would you want to force the user to use special
SM> commands to leave the "completion choice mode"?
SM> I must be missing something.

I am asking for more familiar behavior.  It's a standard "select from a
list of things" popup, sometimes with autocomplete.  Those have been
around for a long time and have well-established UI conventions, so we
don't have to invent our own.

In today's standard autocompleted lists, you finish the interaction by
either down{n}+RET to select, ESC to cancel, or with input that clears
the candidate list (so none match).  These are not special commands.
The user is not captive to the UI.  Do you need examples of how popular
and standard this behavior is today?  Do they matter to you?

Generally the candidates are displayed on a layer above the application
with transparency to indicate they are transient.

>> Otherwise yes, `completing-read' could be the list-selection API.

SM> I think "list-selection" is a restrictive way to think about it: hitting
SM> TAB to complete doesn't necessarily mean "the user wants to select
SM> a completion candidate".  It can also mean "the user wants to complete
SM> "fo" to "fooba" and then complete this identifier by adding something
SM> else that's not in any of the completion candidates.

Right, so perhaps it can do all of that.  But surely list selection is a
basic use case it should cover without trying to read minds?

>> Could `completing-read' call `widget-choose' regardless of the
>> invocation point and show a graphical popup or a simple text buffer?
>> Even in the minibuffer I think that would look OK.

SM> We wouldn't use completing-read, for API reasons.  But indeed, the
SM> minibuffer.el code is slowly moving towards using the same completion
SM> mechanism as in other buffers (completion-at-point).

OK, I understand.  It seems that you already have a direction in mind so
I only hope you make it closer to what's standard today, and that it's
exposed as an API.

Ted

Re: Emacs completion matches selection UI

[Stephen J. Turnbull]

Ted Zlatanov writes:
 > On Tue, 17 Dec 2013 13:29:30 -0500 Stefan Monnier <monnier@IRO.UMontreal.CA> wrote: 

 > SM> I'm not sure why you see it as something the user wouldn't like.
 > 
 > Because it's not familiar.

"Familiar" has never been a reason in itself for doing things in
Emacs, and obviously "unfamiliar" is a superset of "innovative", so
make sure you fish the baby out of the bathwater first.

 > In today's standard autocompleted lists, you finish the interaction
 > either down{n}+RET to select, ESC to cancel, or with input that
 > clears the candidate list (so none match).  These are not special
 > commands.

But they conflict with current Emacs interfaces (eg, ESC is sort of a
prefix key).

Why do you think it's worth doing that?  (See below for why I don't
think it's worth doing.)  Oh, and don't bother answering if in Emacs
you would substitute C-g for ESC (I don't feel that's a good idea
either, but I can't justify that intuitive feeling).

 > The user is not captive to the UI.

That's true in theory and in the long run, but in practice, in the
short run, of course users are captive.  UI gets into muscle memory.
RMS says traditional Emacs key sequences are *more* efficient, but I
don't agree.  It turns out to be very hard to prove that a particular
keyboard layout is overall more efficient UI (cf the many decades and
still continuing Dvorak vs. QWERTY controversy), and I'm pretty sure
the same is true of UI in general.[1]  What clearly does matter is
what you're used to, as anyone who must use multiple keyboards in the
course of a day can testify.

 > Do you need examples of how popular and standard this behavior is
 > today?

Ted, it's not about "popular" (especially not "popular with developers
who create applications that make Emacs developers feel sick") and
"standard".  Until you get that, you're going to waste a lot of bytes.

 > Do they matter to you?

Speaking for myself, "popular" and "standard" are idea sources, not
reasons for adoption.  In general, I'm very much in favor of
standards.  However, Emacs UI *is* different.  You can't buy much with
superficial familiarity if at some point Emacs is going to rip a hole
in your reality.  Using Emacs truly effectively is a continuous
learning process.  The unfamiliar UIs you talk about are less than 10%
of the barrier to using Emacs.

Of course you can use Emacs as a somewhat more powerful Notepad, but I
don't think Emacs-with-*Office-keybindings would be particularly
satisfying to folks who don't need more than that of a text editor.

 > SM> I think "list-selection" is a restrictive way to think about
 > SM> it: hitting TAB to complete doesn't necessarily mean "the user
 > SM> wants to select a completion candidate".  It can also mean "the
 > SM> user wants to complete "fo" to "fooba" and then complete this
 > SM> identifier by adding something else that's not in any of the
 > SM> completion candidates.
 > 
 > Right, so perhaps it can do all of that.  But surely list selection
 > is a basic use case it should cover without trying to read minds?

The standard widgets (Mozilla, Chrome, *Office, MS Office 2007) don't
do "all of that".  I find them only to be useful for recalling very
recent history (which in Emacs would be one or two kill-ring rotate
operations); otherwise I go to the more or less hierarchically
organized, *fully modal* "bookmarks" or "history" mechanisms.

*Office's inline completion behavior was the second thing I invoked
help for.[2]  Unable to figure out WTF[3], the third time I invoked help
was to find out how to disable it.  Since other people use those UIs
happily enough, I can only conclude that they violate my idiosyncratic
UI intuition, and (of course this is a guess, YMMV) I suspect they
inherently conflict with Emacs-style completion mechanims.


Footnotes: 
[1]  OTOH, muscle strain and the like can be measured, and the injury
implications of different keyboard form factors and shapes are well-
established.

[2]  The first was to learn how to disable spelling and grammar
auto-corruption.

[3]  OTOH, I quickly got used to iOS inline completion behavior, which
offers *one* candidate.  I have fat fingers, spelling corrections are
usually correct here and I appreciate the help, especially at iPhone
size.

Re: Emacs completion matches selection UI

[Ted Zlatanov]

On Wed, 18 Dec 2013 10:58:44 +0900 "Stephen J. Turnbull" <stephen@xemacs.org> wrote: 

SJT> Ted Zlatanov writes:
>> On Tue, 17 Dec 2013 13:29:30 -0500 Stefan Monnier <monnier@IRO.UMontreal.CA> wrote: 

SM> I'm not sure why you see it as something the user wouldn't like.
>> 
>> Because it's not familiar.

SJT> "Familiar" has never been a reason in itself for doing things in
SJT> Emacs

Sure it has.  We (the Emacs developers and users) simply do what's
familiar to us.  We're all used to "our" way of doing things and are
often puzzled when new users reject our "obvious intuitive" solutions.

SJT> and obviously "unfamiliar" is a superset of "innovative", so make
SJT> sure you fish the baby out of the bathwater first.

I think the "baby" is decades of HCI research we seem to ignore.

>> Do you need examples of how popular and standard this behavior is
>> today?

SJT> Ted, it's not about "popular" (especially not "popular with developers
SJT> who create applications that make Emacs developers feel sick") and
SJT> "standard".

Hey, I think you need an example!  libreadline (and most shells that use
it).  Attacking the Notepad/W32/crapGUI straw man won't help this
discussion.

Ted

Re: Emacs completion matches selection UI

[Stephen J. Turnbull]

Ted Zlatanov writes:
 > On Wed, 18 Dec 2013 10:58:44 +0900 "Stephen J. Turnbull" <stephen@xemacs.org> wrote: 
 > 
 > SJT> Ted Zlatanov writes:
 > >> On Tue, 17 Dec 2013 13:29:30 -0500 Stefan Monnier <monnier@IRO.UMontreal.CA> wrote: 
 > 
 > SM> I'm not sure why you see it as something the user wouldn't like.
 > >> 
 > >> Because it's not familiar.
 > 
 > SJT> "Familiar" has never been a reason in itself for doing things in
 > SJT> Emacs
 > 
 > Sure it has.

Oh, in *that* sense, yes -- but that's cheating.  *Your* argument
requires the meaning "familiar to people who aren't familiar with
Emacs", and I adopted your terminology.

 > SJT> and obviously "unfamiliar" is a superset of "innovative", so make
 > SJT> sure you fish the baby out of the bathwater first.
 > 
 > I think the "baby" is decades of HCI research we seem to ignore.

HCI research is mostly crap, AFAICT.  (The part that worked on Dvorak
vs. QWERTY has been thoroughly documented to be mostly crap.)  For
example, I don't really see any productivity differences among
C-c/C-x/C-v, M-w/C-w/C-y, and whatever it is that vi uses.  What's
important about CUA and Emacs is that they're *everywhere* in the
user's "app space".  No surprise there, and Emacs already has that,
since before CUA took over the world, too.  That's precisely why Emacs
users tend to "live" in Emacs: it provides a highly optimized UI that
they are familiar with, used to, and know how to extend.

Discoverability does matter, I admit.  Apple seems to have gotten that
right, but are we really going to revise all of Emacs to be compatible
with a one-button mouse?

 > >> Do you need examples of how popular and standard this behavior is
 > >> today?
 > 
 > SJT> Ted, it's not about "popular" (especially not "popular with developers
 > SJT> who create applications that make Emacs developers feel sick") and
 > SJT> "standard".
 > 
 > Hey, I think you need an example!  libreadline (and most shells that use
 > it).  Attacking the Notepad/W32/crapGUI straw man won't help this
 > discussion.

I wasn't talking about Notepad or the Win32 API at that point.  I was
talking about the fact that *Office isn't what Richard or I or the
org-mode devs :-) want in a word processor.

And it's not the sucky GUI that I was talking about when I mentioned
Notepad.  It's the reality discontinuity that occurs when you want to
do something more than libreadline or Notepad can do (except for the
height of the "text widget", they're arguably of similar power :-).
At that point Emacs and the rest of the world (including shells) part
company, and I don't see anything in your argument that deals with
that discontinuity.

Feel free to expand on the libreadline example, though.  I suspect it
falls apart in the sense that I know that I want *consistent*
completion throughout Emacs, where a minibuffer is just a buffer that
happens to be one line high.  Whereas libreadline is highly optimized
to work with the constraint that the buffer *is* exactly one line
high.  But that's mostly uninformed intuition, somewhat informed by
the idea of a "reality discontinuity".

Re: Emacs completion matches selection UI

[Ted Zlatanov]

On Wed, 18 Dec 2013 13:47:40 +0900 "Stephen J. Turnbull" <stephen@xemacs.org> wrote: 

SJT> Oh, in *that* sense, yes -- but that's cheating.  *Your* argument
SJT> requires the meaning "familiar to people who aren't familiar with
SJT> Emacs", and I adopted your terminology.

I am trying to explain that people like what's familiar to them and call
it "intuitive" and "easy to use."  The vast majority of applications in
existence have adopted some conventions for how to select items from a
list, so "users" in general are familiar with those conventions.

Emacs users and developers have developed their own UI conventions, many
predating modern user interfaces (both the conventions and the
users/developers!).  So any time you try to discuss improvements to the
Emacs UI, you have two versions of "familiar" and each side in the
debate is very attached to their version of it and somewhat blind to the
other side's version.  The best way forward seems to be to look at
specific solutions of the problem outside Emacs, preferably without
mentioning Apple or Microsoft products because they tend to polarize the
debate quickly.

Please also understand I am not talking about actual functionality here,
only about the user experience.  Emacs is by far more powerful and
useful than most products on the market, but without a good user
experience it's harder to discover that.

SJT> HCI research is mostly crap, AFAICT.

OK, I guess that settles it.

>> Hey, I think you need an example!  libreadline (and most shells that use
>> it).  Attacking the Notepad/W32/crapGUI straw man won't help this
>> discussion.

SJT> At that point Emacs and the rest of the world (including shells) part
SJT> company, and I don't see anything in your argument that deals with
SJT> that discontinuity.

I understand the concern, and think that the following suggestions are
not badly discontinuous for simple list selection, a very common use
case:

* selection candidates should be displayed near the point where the user
  requested the list.  In a buffer, this is (point).  In the minibuffer,
  this is the input position.

* they should be displayed without a dedicated *Completions* buffer,
  like `widget-choose' does it (special text buffer in text mode, nice
  popup in graphical mode)

* up/down should move between candidates until selection

* any other self-insert character should narrow the selection further
  and get inserted in the original context

* RET should complete the selection

* ESC or C-g should cancel the selection

* further TABs could have special effects, e.g. select current candidate
  and move on to next selection step (Stefan's example of doing more
  than basic list selection).  I don't feel strongly about this one,
  personally.

SJT> Feel free to expand on the libreadline example, though.  I suspect it
SJT> falls apart in the sense that I know that I want *consistent*
SJT> completion throughout Emacs, where a minibuffer is just a buffer that
SJT> happens to be one line high.  Whereas libreadline is highly optimized
SJT> to work with the constraint that the buffer *is* exactly one line
SJT> high.  But that's mostly uninformed intuition, somewhat informed by
SJT> the idea of a "reality discontinuity".

I was trying to give you examples of other solutions to the same problem
that are closer to the Emacs users' and developers' expectations,
because I feel talking about GUIs inevitably gets into the wrong kind of
arguments about Office and so on (none of which interests me, as I
mentioned above I only want to discuss the user experience, not the
functionality).  libreadline is very familiar to our group and is a good
API example to study, but it has significant limitations as you know.

zsh completion is another great example, where you have highly
context-sensitive modules to complete command options and values,
written in a fairly primitive language.  The user can choose to enable
or disable completion modules and change completion options.  zsh
completion supports multiline display of candidates and in many ways
behaves like the in-buffer completion Emacs does today, including
dismiss-on-input behavior.

Ted

Re: Emacs completion matches selection UI

[Eli Zaretskii]

> From: Ted Zlatanov <tzz@lifelogs.com>
> Date: Wed, 18 Dec 2013 09:43:46 -0500
> 
> * ESC or C-g should cancel the selection

The ESC part is un-Emacsy: ESC is needed for meta characters, so it
cannot be eaten.  I suggest ESC ESC ESC instead, which is the accepted
alternative to C-g in many other places.

Re: Emacs completion matches selection UI

[Ted Zlatanov]

On Wed, 18 Dec 2013 17:51:57 +0200 Eli Zaretskii <eliz@gnu.org> wrote: 

>> From: Ted Zlatanov <tzz@lifelogs.com>
>> Date: Wed, 18 Dec 2013 09:43:46 -0500
>> 
>> * ESC or C-g should cancel the selection

EZ> The ESC part is un-Emacsy: ESC is needed for meta characters, so it
EZ> cannot be eaten.  I suggest ESC ESC ESC instead, which is the accepted
EZ> alternative to C-g in many other places.

What meta characters make sense when you're selecting candidates from a
list?  I think in this specific case we can allow ESC to literally mean
"Escape."  But if that's not acceptable, then sure, `ESC ESC ESC' would
work.

Ted

Re: Emacs completion matches selection UI

[Eli Zaretskii]

> From: Ted Zlatanov <tzz@lifelogs.com>
> Date: Wed, 18 Dec 2013 11:38:09 -0500
> 
> >> * ESC or C-g should cancel the selection
> 
> EZ> The ESC part is un-Emacsy: ESC is needed for meta characters, so it
> EZ> cannot be eaten.  I suggest ESC ESC ESC instead, which is the accepted
> EZ> alternative to C-g in many other places.
> 
> What meta characters make sense when you're selecting candidates from a
> list?

E.g., many TTYs emit escape sequences from arrow keys.  M-p and M-n
also come to mind.  And of course, Emacs users can customize keys, so
any function one needs for browsing the candidates could potentially
be bound to some meta character.

Re: Emacs completion matches selection UI

[Ted Zlatanov]

On Wed, 18 Dec 2013 19:05:50 +0200 Eli Zaretskii <eliz@gnu.org> wrote: 

>> From: Ted Zlatanov <tzz@lifelogs.com>
>> Date: Wed, 18 Dec 2013 11:38:09 -0500
>> 
>> >> * ESC or C-g should cancel the selection
>> 
EZ> The ESC part is un-Emacsy: ESC is needed for meta characters, so it
EZ> cannot be eaten.  I suggest ESC ESC ESC instead, which is the accepted
EZ> alternative to C-g in many other places.
>> 
>> What meta characters make sense when you're selecting candidates from a
>> list?

EZ> E.g., many TTYs emit escape sequences from arrow keys.  M-p and M-n
EZ> also come to mind.  And of course, Emacs users can customize keys, so
EZ> any function one needs for browsing the candidates could potentially
EZ> be bound to some meta character.

Good points, I agree `ESC ESC ESC' is a better choice.

Ted

Re: Emacs completion matches selection UI

[Stephen J. Turnbull]

Ted Zlatanov writes:

 > I am trying to explain that people like what's familiar to them and
 > call it "intuitive" and "easy to use."

As far as I can tell, that's unecessary.  It's a pretty generally
accepted principle, and nobody has denied it.

 > The best way forward seems to be to look at specific solutions of
 > the problem outside Emacs,

"Solution" to what "problem"?  I don't understand what problem you're
trying to solve by emulating other application's UIs.

For example, it's easy to understand why Richard wants WYSIWYG.  I
have a good imagination, I find that restructured text looks a lot
like typeset printing (especially if I tilt my head to get italics).
But the attraction is that unlike TeX, which looks like code, ReST
looks like the target output (IMO which is the only one that matters
when I'm writing).  If Richard wants it to look more like the target
output, I can understand that even though I don't need it.  Thus
WYSIWYG is something that Emacs users want, in this case existing
Emacs users.

I don't understand what story you're trying to tell to show that the
completion style you're advocating will benefit Emacs users, either
existing ones or ones who plausibly might be encouraged to use Emacs
by the new style.

 > Please also understand I am not talking about actual functionality
 > here, only about the user experience.  Emacs is by far more
 > powerful and useful than most products on the market, but without a
 > good user experience it's harder to discover that.

My claim is that the kind of user who needs a "good user experience"
to get started won't discover Emacs's power anyway, because of the
discontinuity.  Which you are ignoring ....

 > preferably without mentioning Apple or Microsoft products because
 > they tend to polarize the debate quickly.

How can you avoid mentioning the two most "familiar" UIs in the
business (backed up by a pile of HCI research)?

 > * they should be displayed without a dedicated *Completions* buffer,
 >   like `widget-choose' does it (special text buffer in text mode, nice
 >   popup in graphical mode)

Huh?  *Completions* is a special text buffer, no?

Re: Emacs completion matches selection UI

[Ted Zlatanov]

On Thu, 19 Dec 2013 02:37:36 +0900 "Stephen J. Turnbull" <stephen@xemacs.org> wrote: 

SJT> Ted Zlatanov writes:
>> The best way forward seems to be to look at specific solutions of
>> the problem outside Emacs,

SJT> "Solution" to what "problem"?  I don't understand what problem you're
SJT> trying to solve by emulating other application's UIs.

I thought I explained it pretty clearly in this thread so I won't recap
it.  The topic is the current UI for selecting completion candidates
(and items from a list in general).  The question is whether it can be
improved; we have proposed some specific improvements and at least
having "down/up" go into the "select candidates" mode was reasonably
well received.

>> preferably without mentioning Apple or Microsoft products because
>> they tend to polarize the debate quickly.

SJT> How can you avoid mentioning the two most "familiar" UIs in the
SJT> business (backed up by a pile of HCI research)?

Watch me :)  I managed to give two relevant examples (libreadline and
zsh).  I can give more: Qt, GTK, Motif...

Here's a fairly standard autocomplete widget in today's Web, you can try
several use cases if your browser supports it:
http://jqueryui.com/autocomplete/

>> * they should be displayed without a dedicated *Completions* buffer,
>> like `widget-choose' does it (special text buffer in text mode, nice
>> popup in graphical mode)

SJT> Huh?  *Completions* is a special text buffer, no?

Not in the same way if I understand the code in minibuffer.el correctly.
But more importantly, I don't want to see a special text buffer in
graphical mode.  I'm willing to sacrifice text vs. graphical consistency
for better visuals.  If *Completions* is still used in text mode, that's
OK (although I think `widget-choose' does it better for small lists).

Ted

p.s. finally, some anti-tips for UI designers:
http://toastytech.com/guis/uirant.html (the whole thing is quite funny
and based on older operating systems)

"Never, ever, under any circumstance use the OS-native graphical
controls or widgets. Users get bored of the same old buttons, text
boxes, and stuff.

When possible, disable window management and use unusual, oddly placed
graphics for the windowing functions such as the window close option.

When writing your own controls or widgets, make absolutely sure they
look and feel nothing like the OS-native widgets or anything else the
user might expect. Otherwise you might accidentally make the user think
that your application is actually designed for their OS."

Re: Emacs completion matches selection UI

[chad]

Im not quite sure why anyone is arguing about whether or not users
want better completion (of various sorts) inside Emacs. Given the
large number of packages that build exactly this feature (autocomplete,
helm, completion, anything, company, etc), the demand is clearly
present. The issue Ted seems to be trying to solve is that these
packages are each reinventing wheels with different shapes, sizes,
gauges, and (especially) gearing. As a practical matter, users arent
just forced to use 3rd party code; theyre forced to pick and choose
from multiple incompatible features that they want, because the
wheels dont match.

The Emacs maintainers seems willing, in principle, to move towards
providing a better standardized wheel. Since wheels can have colors
(as well as sizes, shapes, etc) Im sure there will be some lively
debate about the specifics of that wheel. Do we really need to
debate the need for wheels at all?

If theres another point to this debate, please forgive me, because
I have missed it.

~Chad

Re: Emacs completion matches selection UI

[Stefan Monnier]

> If theres another point to this debate, please forgive me, because
> I have missed it.

Personally, I'm trying to understand what it is that Ted is suggesting.
Indeed, I do hope to bring something like company-mode into Emacs, maybe
even making it into the default in-buffer completion UI.
But the UI based on *Completions* is likely to stick around for a long
time, so it's important to integrate the two at some level.

For me the current "integration" is that they should all rely on
completion-at-point-functions, so that major modes can focus on
providing the best completion-at-point-function they can, so the
user can freely choose which UI they want to use.


        Stefan

Re: Emacs completion matches selection UI

[Stephen J. Turnbull]

chad writes:

 > Do we really need to debate the need for wheels at all?

There is no such debate.  The debate is about what constitutes a good
wheel.  Ted thinks a good wheel is one which non-Emacs-users will feel
comfortable with.  I think, and AIUI Stefan does too, that a good
wheel for Emacs is one which extends or consolidates Emacs traditions,
even if in CUAland or libreadline coordinates they look ellipsoidal.

Re: Emacs completion matches selection UI

[chad]

On 18 Dec 2013, at 23:53, Stephen J. Turnbull <stephen@xemacs.org> wrote:

> chad writes:
> 
>> Do we really need to debate the need for wheels at all?
> 
> There is no such debate.  The debate is about what constitutes a good
> wheel.  Ted thinks a good wheel is one which non-Emacs-users will feel
> comfortable with.  I think, and AIUI Stefan does too, that a good
> wheel for Emacs is one which extends or consolidates Emacs traditions,
> even if in CUAland or libreadline coordinates they look ellipsoidal.

Thank you both, Stefan and Stephen, for first-class responses to
what (upon re-reading) sounds like a fairly intemperate question.
It was nicer than I probably deserved, and I appreciate it.

To Stephens point, I think that theres a fairly large, demonstrated
desire for some sort of wheel that looks and works a lot like (for
example) auto-complete:

	http://www.emacswiki.org/emacs/AutoComplete

We had a big discussion about this (same or very similar) topic
several months ago when talking about Emacs as IDE, especially
comparing Emacs and Eclipse. Im not %100 sure that this is what Ted
is asking after, but I think so. Id also like to see such a wheel
in Emacs, ideally built-in. The recent iswitchb/icomplete/ido
discussion and conversations with several programmers (some current
Emacs users, some ex-Emacs users) suggests to me that this is a big
deal for potential Emacs users. Maybe its not desired or important
for most established Emacs users; I dunno.

Even for an external package, the iswitchb/icomplete/ido discussion
suggests that its pretty important to get an idea about the future
direction of the related pieces that are in Emacs, to avoid building
a big wheel that doesnt fit. Begging your indulgence, this suggests 
three big questions:

* Can we get a solid consensus on whether or not this sort of thing
  is desirable in Emacs?

* If we think it’s a good idea to enable, should it be inside Emacs or
  in some external package?

* If we think its a good idea to enable, what do we need in order to
  make it compatible with Emacs internal directions and reusable for
  multiple packages? 

I hope this helps. Thanks again.
~Chad

Re: Emacs completion matches selection UI

[Stephen J. Turnbull]

Ted Zlatanov writes:

 > I thought I explained it pretty clearly in this thread so I won't recap
 > it.  The topic is the current UI for selecting completion candidates
 > (and items from a list in general).  The question is whether it can be
 > improved;

No, it's not.  "Anything can be improved" is a general principle and
one of the fundamental raison d'etres of free software.  Stop pointing
to strawmen and maybe we can get somewhere.

 > we have proposed some specific improvements and at least having
 > "down/up" go into the "select candidates" mode was reasonably well
 > received.

"Modes" in the sense you are talking about here are an antipattern in
Emacs.

 > SJT> How can you avoid mentioning the two most "familiar" UIs in the
 > SJT> business (backed up by a pile of HCI research)?
 > 
 > Watch me :)  I managed to give two relevant examples (libreadline and
 > zsh).  I can give more: Qt, GTK, Motif...

Five patterns I'd rather not see Emacs emulate (though for different
reasons in the case of the shell libraries and in the case of the
GUIs).  IMO none of them are as good as Windows, let alone Mac OS/iOS
and Android, and the GUIs are obviously inferior emulations.  Was that
your point? ;-)

 > Here's a fairly standard autocomplete widget in today's Web,

Would you please stop assuming that people who disagree with you are
cavemen hunkered down over their teletypes with paper tape attachments,
ADM-3s and (wow, shiny!) Hazeltine 1510s who dream of buying a VT-220
like the one Mr. Sulu uses on Star Trek?

I know what such widgets look like, thank you, but I mostly ignore
them because their advice is way below the threshold of utility in 90%
of my workflows.  Ie, it's faster to type until the desired candidate
is at the top of the list, then select it -- very like the way Emacs
completion currently works, but suboptimal because of the extra
keystrokes.  The alternative of shifting gears, engaging the popup,
and select from a list is much slower in most cases, and in the
majority of cases where "keep typing" is inferior to selection from a
menu, minibuffer-style history rotation is faster than both.

You can argue that that's because I'm familiar with Emacs completion
and use Emacs in the majority of my workflows.  No doubt that's most
of the reason.  But that brings us back to the question of "why should
we adopt a goal of familiarity to *non*-Emacs-users?"  I don't see a
good reason *adapt Emacs to them*; I think it's preferable to help new
users to *adapt workflows to Emacs*.

 > >> * they should be displayed without a dedicated *Completions*
 > >> buffer, like `widget-choose' does it (special text buffer in
 > >> text mode, nice popup in graphical mode)
 > 
 > SJT> Huh?  *Completions* is a special text buffer, no?
 > 
 > Not in the same way if I understand the code in minibuffer.el
 > correctly.

Why are you referring to implementation technique here?  I thought you
were interested in look-and-feel?

 > But more importantly, I don't want to see a special text buffer in
 > graphical mode.

Maybe it's just me, but I can't interpret that, let alone agree with
it.  Do you mean you don't want the *Completions* buffer to be a
presented in an Emacs window with the usual decorations (modeline,
scrollbars, and whatnot)?  That I can agree with.

I imagine the basic underlying data structure being the usual
*Completions*[1] buffer, with three different presentations: (1) a
standard 2-D presentation of *Completions* in a multiline minibuffer
window (to reduce the decorations to a minimum), with the current
input at the bottom in a different face, (2) a (translucent) popup
overlay containing a 2-D presentation of the same buffer in GUI
(perhaps with different row x column dimensions), and (3) (not really
relevant to today's Emacsen, but just for creativity's sake) a popup
size-weighted "tag cloud"-style presentation with "higher-priority"
completions both larger and nearer the center (for touch-activated
devices, especially those with small form factors).  With "wide"
items, you could imagine the "current selection" "blowing up" as the
user scrolls through a 1-D list.

I haven't thought much about the selection interface (keymaps), but I
suppose for all three a click (or tap) would select, and for the first
two 2-D "arrow" navigation would be appropriate (if the items are
small enough), suggesting ordering the items by "Manhattan distance"
and warping the cursor to somewhere in the middle of the "completions
window", while for the tag cloud the arrow keys would move backward
and forward through a linear order.  N.B. For a 2-D display that
degenerates into a single column because of wide individual items, the
keymap would automatically reduce to forward = down and backward = up.


Footnotes: 
[1]  For the "tag cloud" presentation it might need to be a more
sophisticated data structure, since each item could have a cardinal
weight, not just an ordinal position.

Re: Emacs completion matches selection UI

[Stephen J. Turnbull]

chad writes:

 > To Stephens point, I think that theres a fairly large, demonstrated
 > desire for some sort of wheel that looks and works a lot like (for
 > example) auto-complete:

Conceded that there's demand.  I would argue that ac-mode is not good
enough for Emacs, because it's menu-based.  I assume from hints on the
page you linked that selection algorithm "learns" over time, but the
menu presented in the screenshot there is full of junk that almost
certainly is undesired.  By contrast, the trivial heuristic "it's in a
buffer somewhere in this Emacs instance" with in-place expansion
iterating over candidates with repeated invocations (dabbrev) is more
accurate, less obtrusive, and more Emacs-y IMHO (YMMV, of course).
And quite fast, even with several dozen buffers.

 > comparing Emacs and Eclipse. Im not %100 sure that this is what Ted
 > is asking after, but I think so. Id also like to see such a wheel
 > in Emacs, ideally built-in.

Unless auto-complete usually does a lot better than the menu in the
screenshot, I'd say no thanks to its algorithm in core.  The various
UIs should be expressed as standard APIs in core, with some sort of
mechanism for configuring them flexibly (the file handler system is
the mod4el I have in mind, although that's probably over-elaborate for
this task).

Traditional Emacs UIs as default would be my preference.

 > The recent iswitchb/icomplete/ido discussion and conversations with
 > several programmers (some current Emacs users, some ex-Emacs users)
 > suggests to me that this is a big deal for potential Emacs users.

It's still not obvious to me that we really need to cater to
"potential" Emacs users, for reasons I gave earlier.  The best way to
handle the issue (UI themes) isn't really available, except for
cua-mode.

 > * Can we get a solid consensus on whether or not this sort of thing
 >   is desirable in Emacs?

The necessary widgets (pretty popup menus, etc) should be in core for
the usual reasons that GUI toolkits provide them.  Even if core Emacs
developers don't particularly like them (including attempting to teach
new users to avoid them ;-), the demand is there, itches will be
scratched, and it's best if they're scratched consistently.

 > * If we think it’s a good idea to enable, should it be inside Emacs or
 >   in some external package?

Filtering and ordering algorithms don't seem to belong in core yet.
The screenshots of Emacs implementations and my experience with
non-Emacs implementations suggests there's a heck of a lot of
improvement needed.  Some API for plugging those into the completion
widgets would probably be a very good idea.

Re: Emacs completion matches selection UI

[Aaron Ecay]

2013ko abenudak 19an, "Stephen J. Turnbull"-ek idatzi zuen:

[...]

> 
>> * Can we get a solid consensus on whether or not this sort of thing
>> is desirable in Emacs?
> 
> The necessary widgets (pretty popup menus, etc) should be in core for
> the usual reasons that GUI toolkits provide them.  Even if core Emacs
> developers don't particularly like them (including attempting to teach
> new users to avoid them ;-), the demand is there, itches will be
> scratched, and it's best if they're scratched consistently.
> 
>> * If we think it’s a good idea to enable, should it be inside Emacs or
>> in some external package?
> 
> Filtering and ordering algorithms don't seem to belong in core yet.
> The screenshots of Emacs implementations and my experience with
> non-Emacs implementations suggests there's a heck of a lot of
> improvement needed.  Some API for plugging those into the completion
> widgets would probably be a very good idea.

In this vein, it may be useful to think of the CompletionUI package
<http://www.emacswiki.org/emacs/CompletionUI>.  Quoting from the
EmacsWiki description, presumably written by the author: “Typically, a
lot of code in packages providing some kind of text completion deals
with the user interface. The goal of CompletionUI is to be the
swiss-army knife of in-buffer completion user-interfaces, which any
source of completions can plug in to, thus freeing completion package
writers to concentrate on the task of finding the completions in the
first place.”

In addition to the ability to plug custom UIs into the underlying API,
the package claims to itself provide the following UI options (again
quoting the wiki):

- Dynamic completion: provisionally insert the best completion candidate
  in the buffer as you type. 
- Completion hotkeys: single-key selection of a completion candidate.
- Cycling: cycle through completion candidates.
- Tab-completion: “traditional” expansion to longest common substring.
- Help-echo: display a list of completion candidates in the echo-area.
- Tooltip: display and select from a list of completion candidates in a
  tool-tip located below the point. 
- Pop-up frame: display and select from a list of completion candidates
  in a tool-tip located below the point; toggle between showing some or
  all possible completions. 
- Completion menu: allow completion candidates to be selected from a
  drop-down menu located below the point. 
- Completion browser: browse through all possible completion candidates
  in a hierarchical menu located below the point. 
- auto-completion-mode: automatically complete words as you type.

Perhaps predictably <http://xkcd.com/927/>, the goal of other completion
package writers adopting this system seems to have gone unrealized.  But
maybe if a similar system were “blessed” by being included in emacs core
(tied to the existing completion-at-point-functions) things might go
better.

(I have no connection to the author; I simply learned about the package
as part of my never-entirely-successful quest for an Intellisense-style
in-buffer completion package that is actually usable without extensive
tweaking.)

-- 
Aaron Ecay

Re: Emacs completion matches selection UI

[Stephen J. Turnbull]

Aaron Ecay writes:

 > In this vein, it may be useful to think of the CompletionUI package
 > <http://www.emacswiki.org/emacs/CompletionUI>.  Quoting from the
 > EmacsWiki description, presumably written by the author: “Typically, a
 > lot of code in packages providing some kind of text completion deals
 > with the user interface. The goal of CompletionUI is to be the
 > swiss-army knife of in-buffer completion user-interfaces, which any
 > source of completions can plug in to, thus freeing completion package
 > writers to concentrate on the task of finding the completions in the
 > first place.”

+1 to that, but does CompletionUI achieve the goal?

 > Perhaps predictably <http://xkcd.com/927/>, the goal of other completion
 > package writers adopting this system seems to have gone unrealized.  But
 > maybe if a similar system were “blessed” by being included in emacs core
 > (tied to the existing completion-at-point-functions) things might go
 > better.

"Blessing" does help, a lot, but the big problem with systems of this
kind is that they *don't* achieve the goal, for several reasons:

1. The various gadgets on a Swiss Army knife are most useful if you
   don't have access to a workbench with fullsize versions.  So while
   the "Swiss Army knife" metaphor is standard in this kind of thing,
   a more accurate metaphor is "Snap-On toolbox on casters".  Are the
   gadgets in CompletionUI Snap-On-quality professional tools?

2. Are they sufficiently customizable by apps, while providing
   sufficient capability with very simple usage so that they make
   sense for use in RAD prototypes, then tweaked as the prototype is
   converted to a full-blown app?

3. Is the customization user-oriented, so that the programmer just
   plugs them in once, and users can easily choose the styles they
   prefer?  Bonus points if it's easy for programmers to provide a
   suite of defaults without hardcoding them in API calls.

4. Are both the APIs and UIs well-documented?  This goes a long way to
   alleviating the issue in #1, as initial adopters quickly fix bugs,
   improve interfaces, and adding features.

Re: Emacs completion matches selection UI

[Ted Zlatanov]

On Wed, 18 Dec 2013 22:40:39 -0500 Stefan Monnier <monnier@IRO.UMontreal.CA> wrote: 

>> If theres another point to this debate, please forgive me, because
>> I have missed it.

SM> Personally, I'm trying to understand what it is that Ted is suggesting.

I've explained it so many times I'm starting to wonder if my English is
the problem.  I started with noticing the inconsistency and mish-mash
code behind the completion candidate selection UI, then proposed a
better UI for completion candidates, eventually recommending a standard
API to select completion candidates (and list elements in general, if
possible) and gave specific suggestions for how it should act and look.

SM> Indeed, I do hope to bring something like company-mode into Emacs, maybe
SM> even making it into the default in-buffer completion UI.
SM> But the UI based on *Completions* is likely to stick around for a long
SM> time, so it's important to integrate the two at some level.

I see company-mode and others as hacks to get around the lack of a true
completion popup.  I really think `widget-choose' is the direction we
should be going in graphical mode, and if *Completions* stays it should
be only in text-mode.

SM> For me the current "integration" is that they should all rely on
SM> completion-at-point-functions, so that major modes can focus on
SM> providing the best completion-at-point-function they can, so the
SM> user can freely choose which UI they want to use.

Exactly, that's not a problem.  The data backend through
`completion-at-point-function' works great.

On Thu, 19 Dec 2013 00:45:47 -0500 chad <yandros@gmail.com> wrote: 

c> To Stephens point, I think that theres a fairly large, demonstrated
c> desire for some sort of wheel that looks and works a lot like (for
c> example) auto-complete:

c> 	http://www.emacswiki.org/emacs/AutoComplete

c> We had a big discussion about this (same or very similar) topic
c> several months ago when talking about Emacs as IDE, especially
c> comparing Emacs and Eclipse. Im not %100 sure that this is what Ted
c> is asking after, but I think so. Id also like to see such a wheel
c> in Emacs, ideally built-in. The recent iswitchb/icomplete/ido
c> discussion and conversations with several programmers (some current
c> Emacs users, some ex-Emacs users) suggests to me that this is a big
c> deal for potential Emacs users. Maybe its not desired or important
c> for most established Emacs users; I dunno.

Honestly, I think almost everyone has either hacked around it with the
add-ons you mentioned or has trained themselves to ignore the bad UI.

c> * Can we get a solid consensus on whether or not this sort of thing
c>   is desirable in Emacs?

c> * If we think it’s a good idea to enable, should it be inside Emacs or
c>   in some external package?

c> * If we think its a good idea to enable, what do we need in order to
c>   make it compatible with Emacs internal directions and reusable for
c>   multiple packages? 

Same questions here, and Stefan answered them to some degree, but
perhaps he should make a clear statement setting the direction after the
upcoming release.  I'm happy to work on improving the user experience,
but don't want to find out after spending a weekend on the code that
I've misunderstood where we're going.  See below as well.

On Thu, 19 Dec 2013 15:23:54 +0900 "Stephen J. Turnbull" <stephen@xemacs.org> wrote: 

>> Here's a fairly standard autocomplete widget in today's Web,

SJT> Would you please stop assuming that people who disagree with you are
SJT> cavemen hunkered down over their teletypes with paper tape attachments,
SJT> ADM-3s and (wow, shiny!) Hazeltine 1510s who dream of buying a VT-220
SJT> like the one Mr. Sulu uses on Star Trek?

Perhaps you can look at the URL and explore all the options and use
cases offered by the jQuery UI autocomplete widget instead of assuming
my goal is to traumatize you with shiny things.  The jQuery UI library
didn't become a de facto standard by accident, it really is a good API.

SJT> I know what such widgets look like, thank you, but I mostly ignore
SJT> them [...] You can argue that that's because I'm familiar with
SJT> Emacs completion and use Emacs in the majority of my workflows.

Yes.  Most of us are terrible at detecting bad UI in the tools we use
every day, and pretending otherwise is disingenuous.

SJT> But that brings us back to the question of "why should we adopt a
SJT> goal of familiarity to *non*-Emacs-users?"  I don't see a good
SJT> reason *adapt Emacs to them*; I think it's preferable to help new
SJT> users to *adapt workflows to Emacs*.

Understood, thank you for your opinion.

>> But more importantly, I don't want to see a special text buffer in
>> graphical mode.

SJT> Maybe it's just me, but I can't interpret that, let alone agree with
SJT> it.  Do you mean you don't want the *Completions* buffer to be a
SJT> presented in an Emacs window with the usual decorations (modeline,
SJT> scrollbars, and whatnot)?  That I can agree with.

Yes.  I think that looks and feels awkward, and the selection of
candidates is particularly awkward.

SJT> I imagine the basic underlying data structure being the usual
SJT> *Completions*[1] buffer, with three different presentations: (1) a
SJT> standard 2-D presentation of *Completions* in a multiline minibuffer
SJT> window (to reduce the decorations to a minimum), with the current
SJT> input at the bottom in a different face, (2) a (translucent) popup
SJT> overlay containing a 2-D presentation of the same buffer in GUI
SJT> (perhaps with different row x column dimensions), and (3) (not really
SJT> relevant to today's Emacsen, but just for creativity's sake) a popup
SJT> size-weighted "tag cloud"-style presentation with "higher-priority"
SJT> completions both larger and nearer the center

SJT> I haven't thought much about the selection interface (keymaps), but I
SJT> suppose for all three a click (or tap) would select, and for the first
SJT> two 2-D "arrow" navigation would be appropriate (if the items are
SJT> small enough), suggesting ordering the items by "Manhattan distance"
SJT> and warping the cursor to somewhere in the middle of the "completions
SJT> window", while for the tag cloud the arrow keys would move backward
SJT> and forward through a linear order.  N.B. For a 2-D display that
SJT> degenerates into a single column because of wide individual items, the
SJT> keymap would automatically reduce to forward = down and backward = up.

I think you've described (except for the tag cloud, which IMO is a
horrible user experience) more or less what I would like to see as well,
with some additions:

* hidden behind a standard built-in API call

* supporting plugins for look, feel, behavior

* the user can choose how the UI should look and behave in general

* works the same way in minibuffer and in-region invocations

* default look and keybindings are close to familiar/standard/etc. today
  (ideally using the native platform's widgets, which would DTRT
   depending on many things Emacs just doesn't know or control)

Thanks!
Ted

Re: Emacs completion matches selection UI

[Dmitry Gutov]

Ted Zlatanov <tzz@lifelogs.com> writes:

> SM> Indeed, I do hope to bring something like company-mode into Emacs, maybe
> SM> even making it into the default in-buffer completion UI.
> SM> But the UI based on *Completions* is likely to stick around for a long
> SM> time, so it's important to integrate the two at some level.
>
> I see company-mode and others as hacks to get around the lack of a true
> completion popup.  I really think `widget-choose' is the direction we
> should be going in graphical mode, and if *Completions* stays it should
> be only in text-mode.

You seem to be conflating company-mode with its overlay-based popup
feature, which is a small part of it.

Why don't you start with implementing a programmable graphical popup
that can display a list of options near any position in Emacs's window,
that completion engines can integrate with without losing the advantages
of the current behavior?

Re: Emacs completion matches selection UI

[Eli Zaretskii]

> From: "Stephen J. Turnbull" <stephen@xemacs.org>
> Date: Thu, 19 Dec 2013 16:03:39 +0900
> Cc: Stefan Monnier <monnier@iro.umontreal.ca>,
> 	EMACS development team <emacs-devel@gnu.org>
> 
> Conceded that there's demand.  I would argue that ac-mode is not good
> enough for Emacs, because it's menu-based.

It's not menu-based.  It _emulates_ a menu by using an overlay
(AFAIU).

> I assume from hints on the page you linked that selection algorithm
> "learns" over time, but the menu presented in the screenshot there
> is full of junk that almost certainly is undesired.  By contrast,
> the trivial heuristic "it's in a buffer somewhere in this Emacs
> instance" with in-place expansion iterating over candidates with
> repeated invocations (dabbrev) is more accurate, less obtrusive, and
> more Emacs-y IMHO (YMMV, of course).  And quite fast, even with
> several dozen buffers.

Is this discussion about how to compute the candidates, or how to
display them and let the user browse them and select one?  I though
the latter was the issue at hand, in which case the algorithms used
to compute the candidates are not part of that issue.

Apologies if I'm mistaken: I didn't read everything in this thread.

Re: Emacs completion matches selection UI

[Ted Zlatanov]

On Thu, 19 Dec 2013 19:30:17 +0200 Dmitry Gutov <dgutov@yandex.ru> wrote: 

DG> Ted Zlatanov <tzz@lifelogs.com> writes:
SM> Indeed, I do hope to bring something like company-mode into Emacs, maybe
SM> even making it into the default in-buffer completion UI.
SM> But the UI based on *Completions* is likely to stick around for a long
SM> time, so it's important to integrate the two at some level.
>> 
>> I see company-mode and others as hacks to get around the lack of a true
>> completion popup.  I really think `widget-choose' is the direction we
>> should be going in graphical mode, and if *Completions* stays it should
>> be only in text-mode.

DG> You seem to be conflating company-mode with its overlay-based popup
DG> feature, which is a small part of it.

Sorry, I don't mean to diminish the rest of the company-mode code.  I
meant specifically the part of it relevant to the discussion here.

DG> Why don't you start with implementing a programmable graphical popup
DG> that can display a list of options near any position in Emacs's window,
DG> that completion engines can integrate with without losing the advantages
DG> of the current behavior?

Because I don't want, after spending the time and effort, to be told it
wasn't the desired direction.  That truly sucks.  I'm choosing, instead,
to ask Stefan to state that direction explicitly.

I also don't have good toolkit knowledge like some of the other Emacs
contributors, but that's not fatal.  Finally, this is a big task and I'd
probably start by making a team of developers who are willing to assign
copyright, interested in improving the current situation, implementing
the desired solution, testing it, integrating it with helm,
company-mode, etc.  So ideally I'd get people like you and the other
popup-autocomplete-helm-company-like developers on board.

Ted

Re: Emacs completion matches selection UI

[Ted Zlatanov]

On Thu, 19 Dec 2013 19:41:32 +0200 Eli Zaretskii <eliz@gnu.org> wrote: 

EZ> Is this discussion about how to compute the candidates, or how to
EZ> display them and let the user browse them and select one?  I though
EZ> the latter was the issue at hand, in which case the algorithms used
EZ> to compute the candidates are not part of that issue.

The latter.  The algorithms and the data backend for completion
candidates and for list selection in general are not a problem IMO.

Ted

Re: Emacs completion matches selection UI

[Dmitry Gutov]

Ted Zlatanov <tzz@lifelogs.com> writes:

> DG> Why don't you start with implementing a programmable graphical popup
> DG> that can display a list of options near any position in Emacs's window,
> DG> that completion engines can integrate with without losing the advantages
> DG> of the current behavior?
>
> Because I don't want, after spending the time and effort, to be told it
> wasn't the desired direction.  That truly sucks.  I'm choosing, instead,
> to ask Stefan to state that direction explicitly.

Right now, you're already arguing minute details here, how it would
work, and insisting that it should be different from all existing
completion interfaces available for Emacs. Don't be surprised to see
rebuttals.

The hard technical part would be to draw the tooltip in the right place
above the frame, make it fast, and offer a good Elisp interface. Then
the code, of which there's a lot, can integrate it.

> Finally, this is a big task and I'd
> probably start by making a team of developers who are willing to assign
> copyright

What part of the work do you see yourself performing?

> company-mode, etc.

So yeah. No need to look for anyone else to do Company integration, I'm
quite willing myself.

Re: Emacs completion matches selection UI

[Stephen J. Turnbull]

Eli Zaretskii writes:

 > It's not menu-based.  It _emulates_ a menu by using an overlay
 > (AFAIU).

The selection may not be by menu (although Ted's push for up-down the
the primary selection method certainly colors the way I look at
everything in this thread).  My point is that Emacs should be able to
do better with something 2D in many circumstances, and therefore
traditional linear menu display is "not good enough".  A 2D candidate
display with horizontal arrow navigation as well as vertical
navigation may not be "familiar" but I suspect it's quite discoverable.

 > Is this discussion about how to compute the candidates, or how to
 > display them and let the user browse them and select one?

The theme of the thread is the latter, but any specific candidate may
suffer from flaws in the former.  I understood Aaron to be suggesting
auto-complete as a candidate implementation.

I think that an Emacs UI framework here should provide hooks for
filtering and learning, as well as producing lists of candidates.
Otherwise installing it is premature.  (Of course that doesn't mean
all "methods" need provide filters and learning algorithms, but
hopefully there would be some generic ones provided by the UI.)

Re: Emacs completion matches selection UI

[Ted Zlatanov]

On Thu, 19 Dec 2013 23:44:20 +0200 Dmitry Gutov <dgutov@yandex.ru> wrote: 

DG> Ted Zlatanov <tzz@lifelogs.com> writes:
DG> Why don't you start with implementing a programmable graphical popup
DG> that can display a list of options near any position in Emacs's window,
DG> that completion engines can integrate with without losing the advantages
DG> of the current behavior?
>> 
>> Because I don't want, after spending the time and effort, to be told it
>> wasn't the desired direction.  That truly sucks.  I'm choosing, instead,
>> to ask Stefan to state that direction explicitly.

DG> Right now, you're already arguing minute details here, how it would
DG> work, and insisting that it should be different from all existing
DG> completion interfaces available for Emacs. Don't be surprised to see
DG> rebuttals.

I don't think I ever said it should be different from all existing, just
that it should be a standard API we can all use and customize (the lack
of such a facility and the inconsistency of the existing code between
in-buffer and in-minibuffer completion is why I didn't charge straight
into the task).  I'm happy to have it look and behave more or less like
`widget-choose' as I said several times, or like a native widget.  My
suggestions for the behavior were based on what I consider reasonable
and familiar, but I don't think they are terribly different from
everyone else's ideas.

DG> The hard technical part would be to draw the tooltip in the right place
DG> above the frame, make it fast, and offer a good Elisp interface. Then
DG> the code, of which there's a lot, can integrate it.

`widget-choose' does this already to some degree, right?  Do you see any
shortcomings in it?

>> Finally, this is a big task and I'd probably start by making a team
>> of developers who are willing to assign copyright

DG> What part of the work do you see yourself performing?

Whatever others don't want to, in order to get things done.  I
appreciate your offer of help very much.

Ted

Re: Emacs completion matches selection UI

[Stephen J. Turnbull]

Ted Zlatanov writes:
 > On Wed, 18 Dec 2013 22:40:39 -0500 Stefan Monnier <monnier@IRO.UMontreal.CA> wrote: 
 > 
 > >> If theres another point to this debate, please forgive me, because
 > >> I have missed it.
 > 
 > SM> Personally, I'm trying to understand what it is that Ted is suggesting.
 > 
 > I've explained it so many times I'm starting to wonder if my English is
 > the problem.

It is.  Or more precisely, your abuse of Emacs technical terms.  Cf
the discussion of "special text buffer".

 > should be going in graphical mode, and if *Completions* stays it should
 > be only in text-mode.

There you go again.  *Completions* is a *buffer*, and does not imply
any particular display.

 > Perhaps you can look at the URL and explore all the options and use
 > cases offered by the jQuery UI autocomplete widget instead of assuming
 > my goal is to traumatize you with shiny things.

There you go again.

 > The jQuery UI library didn't become a de facto standard by
 > accident, it really is a good API.

For many people, especially those for whom a computer needs to be
white goods like a washing machine or toaster with one start button.
We *know* Emacs users are in general not members of that group.

That doesn't mean it's not good for Emacs users too, although I know
one Emacs user very well who ignores it when present because it
doesn't do a damn thing to float his boat in his workflows.  But
"became a standard in that world" is simply not appropriate logic for
bringing it (or similar) into Emacs.  At least, it never was in the
past.  "Try it, you'll like it" is a risky strategy, too.  Some people
won't, and if they're decision-makers you need to appeal to, well,
"sayonara, baby!"

 > Yes.  Most of us are terrible at detecting bad UI in the tools we use
 > every day, and pretending otherwise is disingenuous.

I believe that Emacs users are *much less likely* to be members of the
the "most of us" you describe here, and that they use Emacs as much
for the power implied by the ability to hack your UI as for the power
inherent in the ability to hack apps using powerful APIs.

I've watched Windows power users at work to see *how* they do it, and
I have to admit that their behavior is clearly more efficient than
mine in Mac OS X.  The difference is that they all paid $500 to $1000
for various amounts of training in Windows use, and I'm self-taught.
The point is that the Windows UI is *not* inefficient in itself, but
it's tuned for a certain type of user, it's not actually horrible for
newbies, and it's pretty bad on the discoverability feature.

What you haven't addressed is why the UIs you suggest are appropriate
to Emacs use.  One size does not fit all apps, and one size does not
fit all users.

 > I think you've described (except for the tag cloud, which IMO is a
 > horrible user experience)

I agree, in general.  But here the filters will be the same ones used
already, which all agree are pretty good, whereas I find tag cloud
filters to pretty much suck (I know how to type "US" and "China"
quickly, I don't need those two tags to take up 55% of the display,
thank you very much /Economist of London/).  Perhaps more important to
evaluating my suggestion, the "tag cloud"-style UI is a *proposed*
design targeted at *specific* consoles with *inaccurate* pointing
devices.

Re: Emacs completion matches selection UI

[Stephen J. Turnbull]

Ted Zlatanov writes:

 > Because I don't want, after spending the time and effort, to be told it
 > wasn't the desired direction.  That truly sucks.

"Think of it as evolution in action."  Some things really need to be
demo'd to get others to appreciate them.

Re: Emacs completion matches selection UI

[Stephen J. Turnbull]

Ted Zlatanov writes:

 > I don't think I ever said it should be different from all existing,
 > just that it should be a standard API we can all use and customize

Stefan has said this is a good idea.  Repeatedly.

 > I'm happy to have it look and behave more or less like
 > `widget-choose' as I said several times, or like a native widget.

And now you're wandering off into specifyinging behavior again, within
four lines of the same paragraph.  This is one important reason why
people are having trouble understanding just what you are proposing.

What I understand Stefan to want Now is a *single* API that can be
called for both at-point and in-minibuffer completion.  As I
understand it, he wants the first step to involve *decoupling* the
user-visible behavior from the completion API that the "rest of Emacs"
sees, allowing an array of *different* UIs (== "backend" behaviors) to
be attached to the *same* API ("frontend").  He seems to expect that
the *behavior* implemented initially will be based on (one of) the
existing at-point mechanisms available in core.

*I* believe that will make the contrast between old and new APIs for
developers of applications that use completion more clear, and
*perhaps* that is Stefan's rationale as well.  If such developers
prefer different behavior to the one (or few) initially implemented,
the first test of the value of the new API will be the ease with which
they can install such new behaviors.  The second test, of course, will
be the ease with which non-developer users can customize the UI (here
I mean "swap in different backends", not "GUI theming").

P.S. Note that I probably am using "frontend" and "backend" in a sense
reversed to what many people expect.  It's the same kind of thing as
the "client-server" reversal in the X Window System.  I'm taking that
point of view of the developer who is facing the completion API, and
therefore call the API "front".  After passing through the completion
implementation, he reaches the user-facing "back".

Re: Emacs completion matches selection UI

[Dmitry Gutov]

Ted Zlatanov <tzz@lifelogs.com> writes:

> DG> The hard technical part would be to draw the tooltip in the right place
> DG> above the frame, make it fast, and offer a good Elisp interface. Then
> DG> the code, of which there's a lot, can integrate it.

          ^^^ existing code

> `widget-choose' does this already to some degree, right?  Do you see any
> shortcomings in it?

1. It captures all user input.

That prohibits from using it for displaying completion candidates during
unobtrusive, idle completion. Which is a feature users value.

2. It has no support for incremental narrowing (consequence of item 1).

You call the completion command, but there are too many candidates!
What do you do? You type a few chars and see the list shortened on the fly.

`widget-choose' won't let you do that. You'll have to dismiss it, type a
couple of chars, display it again, see whether the list of short enough
now, and if it isn't, dismiss the widget and try again. Most people will
give up now and either type the whole word themselves, or start
scrolling the list with the mouse, hunting for the required candidate.

3. It's displayed near the mouse cursor.

Did I miss a way to change that?

4. It does not support long lists.

Re: Emacs completion matches selection UI

[Ted Zlatanov]

On Fri, 20 Dec 2013 05:08:22 +0200 Dmitry Gutov <dgutov@yandex.ru> wrote: 

DG> Ted Zlatanov <tzz@lifelogs.com> writes:
DG> The hard technical part would be to draw the tooltip in the right place
DG> above the frame, make it fast, and offer a good Elisp interface. Then
DG> the code, of which there's a lot, can integrate it.

DG>           ^^^ existing code

>> `widget-choose' does this already to some degree, right?  Do you see any
>> shortcomings in it?

DG> 1. It captures all user input.

DG> That prohibits from using it for displaying completion candidates during
DG> unobtrusive, idle completion. Which is a feature users value.

I agree that's a good API option (being able to specify asynchronous
vs. synchronous completion) and the widget should support it.

DG> 2. It has no support for incremental narrowing (consequence of item 1).

DG> You call the completion command, but there are too many candidates!
DG> What do you do? You type a few chars and see the list shortened on the fly.

DG> `widget-choose' won't let you do that. You'll have to dismiss it, type a
DG> couple of chars, display it again, see whether the list of short enough
DG> now, and if it isn't, dismiss the widget and try again. Most people will
DG> give up now and either type the whole word themselves, or start
DG> scrolling the list with the mouse, hunting for the required candidate.

Yup, absolutely.  The autocomplete web browser widgets (e.g. the jQuery
UI one) have that on-the-fly narrowing.  This specifically conflicts
with the way in-buffer completion works right now, and the in-minibuffer
completion kind of works like this if you turn `icomplete-mode' on and
hit TAB to see the candidates every time you type a character.  IMO this
should be an API option.

DG> 3. It's displayed near the mouse cursor.

DG> Did I miss a way to change that?

I'm sure that can be changed to be an API option.  Showing the
candidates near point is one of the big issues with the current
situation, where *Completions* can pop up at the other side of the
screen, so I think that's an important element as well.

DG> 4. It does not support long lists.

I'm not sure it's useful to show a very long list like we do currently.
The *Completions* buffer does it with multiple columns and TAB to move
to the next page and it's not great visually.  For instance showing all
the Unicode characters with `C-x 8 RET TAB' shows a huge list this way
(you can narrow it with "substring TAB") and would be a good use case.

Showing icons for every list element would be a nice option, especially
for the Unicode glyphs.

It may be better for the API call to allow multiple columns as an option
but default to showing just what fits on the screen in a single column,
indicating how many more candidates are not shown.  Something like this,
if only 9 lines fit on the screen (the indicators for "100 more" would
be appropriate to the native widget):

"[... 100 more ... ]
candidate1
candidate2
candidate3
candidate4
candidate5
candidate6
candidate7
[... 100 more ... ]"

And then you could handle a case like the Unicode characters by breaking
out into a table or something even denser.

Ted

Re: Emacs completion matches selection UI

[Dmitry Gutov]

Ted Zlatanov <tzz@lifelogs.com> writes:

> I agree that's a good API option (being able to specify asynchronous
> vs. synchronous completion) and the widget should support it.

I think synchronous vs. asynchronous is a concern of a backend, and it
has nothing to do with the widget. The latter only has to support the
mode of operation where it passes through most of the keystrokes (except
the ones specifically bound) through to Emacs. Or, better yet, captures
none of them and lets the main event loop trigger right commands in the
currently active maps, which in turn can modify the contents of the
widget.

Have you tried reading the company-mode source code?

> Yup, absolutely.  The autocomplete web browser widgets (e.g. the jQuery
> UI one) have that on-the-fly narrowing.  This specifically conflicts
> with the way in-buffer completion works right now

There's no real conflict, in-buffer completion could be made to work
this way. I believe it doesn't right now just because it could be too
performance-intensive when using some completion functions.

> IMO this should be an API option.

The API just has to allow updating the list on the fly.

> DG> 4. It does not support long lists.
>
> Showing icons for every list element would be a nice option, especially
> for the Unicode glyphs.

Will this require explicit support? Emacs supports inline images.

> It may be better for the API call to allow multiple columns as an
> option

I don't think multiple column display is a good option for a popup, at
all. Maybe as a special case, but that could be added later.

> but default to showing just what fits on the screen in a single column,
> indicating how many more candidates are not shown.  Something like this,
> if only 9 lines fit on the screen (the indicators for "100 more" would
> be appropriate to the native widget):

Yes. Or a scroll bar. Depending on how "native" the widget will be.

Another nice feature would be to allow "columns", but of a different
kind. The first column would be the completion options. The second one
would be their annotations (all lines up vertically). Maybe add the
option of having multiple annotations for the third, fourth, etc,
columns.

Re: Emacs completion matches selection UI

[Stefan Monnier]

>>> If theres another point to this debate, please forgive me, because
>>> I have missed it.
>> Personally, I'm trying to understand what it is that Ted is suggesting.
> I've explained it so many times I'm starting to wonder if my English is
> the problem.

It's never really control of language, but each word can mean many
different things, and we seem to have very different assumptions and
contexts, hence the difficulty.

The way I see it, the subject under discussion can be divided into various
mostly orthogonal elements:
- when does the completion code get invoked:
  - upon hitting TAB
  - after typing the first N chars of a completable element
  - when entering minibuffer
  - ...
- what completion operations take place:
  - insert a few chars at point
  - bring up a list of candidates
  - ...
- how to display the list of candidates:
  - in a separate buffer
  - at point, via clever overlay tricks
  - at point, via a special frame
  - at point, via a x-popup-menu
  - ...
- what operations can be performed while the list of candidates is displayed:
  - keep on editing as if the list was not displayed
  - select among candidates with the mouse
  - select with up/down
  - ...
- how/when to update the list of completions and to pop it down.
- what Lisp function to call to display the list of elements.
  We currently don't really have something clear here.  IIUC this is the
  API part you care about (the rest is mostly behavior rather than API).
- ...

Can you clarify which parts you care about?  As said, many are (or
should be) orthogonal.

> Yup, absolutely.  The autocomplete web browser widgets (e.g. the jQuery
> UI one) have that on-the-fly narrowing.  This specifically conflicts
> with the way in-buffer completion works right now,

It doesn't conflict at all.  The current UI doesn't update *Completions*
on-the-fly, but it very much could (and pretty easily too).

> and the in-minibuffer completion kind of works like this if you turn
> `icomplete-mode' on and hit TAB to see the candidates every time you
> type a character.  IMO this should be an API option.

I don't see why it should appear in the API.  It looks like a user
preference instead.  See below for a very crude approximation.


        Stefan


;; Verbose icomplete-mode

(defun verbose--icomplete-do ()
  (let ((beg (minibuffer-prompt-end))
        (end (point-max))
        (non-essential t))
    (when (and (not (eq beg end)) buffer-undo-list)
      (while-no-input
        (minibuffer-completion-help beg end)))))

(defun verbose-icomplete-setup ()
  (when minibuffer-completion-table
    (add-hook 'post-command-hook #'verbose--icomplete-do nil 'local)))

(define-minor-mode verbose-icomplete-mode
  ;; FIXME: find a better name
  ""
  :global t
  (if verbose-icomplete-mode
      (add-hook 'minibuffer-setup-hook 'verbose-icomplete-setup)
    (remove-hook 'minibuffer-setup-hook 'verbose-icomplete-setup)))

Re: Emacs completion matches selection UI

[Ted Zlatanov]

On Fri, 20 Dec 2013 15:18:15 +0200 Dmitry Gutov <dgutov@yandex.ru> wrote: 

DG> Ted Zlatanov <tzz@lifelogs.com> writes:
>> I agree that's a good API option (being able to specify asynchronous
>> vs. synchronous completion) and the widget should support it.

DG> I think synchronous vs. asynchronous is a concern of a backend, and it
DG> has nothing to do with the widget. The latter only has to support the
DG> mode of operation where it passes through most of the keystrokes (except
DG> the ones specifically bound) through to Emacs. Or, better yet, captures
DG> none of them and lets the main event loop trigger right commands in the
DG> currently active maps, which in turn can modify the contents of the
DG> widget.

DG> Have you tried reading the company-mode source code?

(that reminds me I should write a CFEngine-specific completion backend
for company-mode, like I did for general completion in cfengine.el :)

I've explored anything/Helm and company-mode and a few others.  They all
have pluses and minuses.  I do like how company-mode separates frontends
from backends and agree with you about where synchronous
vs. asynchronous as a choice belongs.  But I don't know if a standard
Emacs completion/selection API like what we're discussing would have
such a separation, or if it would be better to keep it simpler.

>> Yup, absolutely.  The autocomplete web browser widgets (e.g. the jQuery
>> UI one) have that on-the-fly narrowing.  This specifically conflicts
>> with the way in-buffer completion works right now

DG> There's no real conflict, in-buffer completion could be made to work
DG> this way. I believe it doesn't right now just because it could be too
DG> performance-intensive when using some completion functions.

Perhaps, I don't know the history and there may be some reasonable
concerns with performance and code complexity.  But I agree this is how
it *should* work in general, since it's already how the in-minibuffer
completion works as I mentioned (modulo the extra TABs to show the
narrowed selection).

>> IMO this should be an API option.

DG> The API just has to allow updating the list on the fly.

That's not a simple thing, since Emacs doesn't have data notifications,
does it?  It would have to be a callback?  But those have performance
penalties and may need to be queued...  For large lists we have to
consider all this.

DG> 4. It does not support long lists.
>> 
>> Showing icons for every list element would be a nice option, especially
>> for the Unicode glyphs.

DG> Will this require explicit support? Emacs supports inline images.

If the list display is purely done in Emacs, it could work (although
IIRC tooltips have special rules about graphics depending on the
platform).  If it's a native widget, they sometimes have a special slot
for icons.  I don't know for sure what's the right approach here but it
seems that displaying the list in a purely Emacsian popup, as opposed to
a native widget, would be the better choice for cross-platform
consistency and functionality.

>> It may be better for the API call to allow multiple columns as an
>> option

DG> I don't think multiple column display is a good option for a popup, at
DG> all. Maybe as a special case, but that could be added later.

OK, I don't have a strong opinion here.

>> but default to showing just what fits on the screen in a single column,
>> indicating how many more candidates are not shown.  Something like this,
>> if only 9 lines fit on the screen (the indicators for "100 more" would
>> be appropriate to the native widget):

DG> Yes. Or a scroll bar. Depending on how "native" the widget will be.

DG> Another nice feature would be to allow "columns", but of a different
DG> kind. The first column would be the completion options. The second one
DG> would be their annotations (all lines up vertically). Maybe add the
DG> option of having multiple annotations for the third, fourth, etc,
DG> columns.

Right, so for example the symbol completion could show the docstring and
current value (for defcustoms).  I agree this is a very useful extra.

Ted

Re: Emacs completion matches selection UI

[Ted Zlatanov]

On Fri, 20 Dec 2013 09:32:11 -0500 Stefan Monnier <monnier@IRO.UMontreal.CA> wrote: 

>>>> If theres another point to this debate, please forgive me, because
>>>> I have missed it.
>>> Personally, I'm trying to understand what it is that Ted is suggesting.
>> I've explained it so many times I'm starting to wonder if my English is
>> the problem.

SM> It's never really control of language, but each word can mean many
SM> different things, and we seem to have very different assumptions and
SM> contexts, hence the difficulty.

Yes, I see that.  Sorry for any confusion I've brought to the
discussion, and thanks for separating the various areas.

SM> The way I see it, the subject under discussion can be divided into various
SM> mostly orthogonal elements:
SM> - when does the completion code get invoked:
SM>   - upon hitting TAB
SM>   - after typing the first N chars of a completable element
SM>   - when entering minibuffer
SM>   - ...

I am not looking to improve this, it's quite good already.

SM> - what completion operations take place:
SM>   - insert a few chars at point
SM>   - bring up a list of candidates
SM>   - ...

I think this should be unified as much as possible into a single
operation that takes place in any context, outside of the user's
control.  IOW this should be the place where Emacs standardizes to only
one completion/selection API, possibly through the `completion-*'
functions.

SM> - how to display the list of candidates:
SM>   - in a separate buffer
SM>   - at point, via clever overlay tricks
SM>   - at point, via a special frame
SM>   - at point, via a x-popup-menu
SM>   - ...

This should probably be customizable, but the default IMO should be
different between graphical and text mode, the way that `widget-choose'
does it.  See the parallel discussion with Dmitry about the possible
approaches here, including icons and tables with annotations and so on.

SM> - what operations can be performed while the list of candidates is displayed:
SM>   - keep on editing as if the list was not displayed
SM>   - select among candidates with the mouse
SM>   - select with up/down
SM>   - ...

The current defaults for this are not consistent and were the original
reason for starting the thread.

The default here is tricky, since we have the in-buffer
vs. in-minibuffer split to consider.  I agree with Dmitry that
auto-narrowing the candidates is important by default; the up/down
selection is important too but must not interfere with buffer motion
when completing in a buffer of course.  If we can find something
consistent that works as a default in all the invocation contexts, I
think that would offer the biggest benefit to Emacs usability.

SM> - how/when to update the list of completions and to pop it down.

That should be entirely internal to the API and not exposed, I think.

SM> - what Lisp function to call to display the list of elements.
SM>   We currently don't really have something clear here.  IIUC this is the
SM>   API part you care about (the rest is mostly behavior rather than API).

Yes, this is the API that should get called by any package (internal or
external) that wants to ask the user to interactively select completion
candidates or just one item from a list.

>> Yup, absolutely.  The autocomplete web browser widgets (e.g. the jQuery
>> UI one) have that on-the-fly narrowing.  This specifically conflicts
>> with the way in-buffer completion works right now,

SM> It doesn't conflict at all.  The current UI doesn't update *Completions*
SM> on-the-fly, but it very much could (and pretty easily too).

OK, that's great.

>> and the in-minibuffer completion kind of works like this if you turn
>> `icomplete-mode' on and hit TAB to see the candidates every time you
>> type a character.  IMO this should be an API option.

SM> I don't see why it should appear in the API.  It looks like a user
SM> preference instead.  See below for a very crude approximation.

Yes, I think you're right, but I can't think of a case where I don't
want auto-narrowing, so I suggest always turning it on.

Ted

Re: Emacs completion matches selection UI

[Stephen J. Turnbull]

Ted Zlatanov writes:

 > SM> - what completion operations take place:
 > SM>   - insert a few chars at point
 > SM>   - bring up a list of candidates
 > SM>   - ...
 > 
 > I think this should be unified as much as possible into a single
 > operation that takes place in any context, outside of the user's
 > control.  IOW this should be the place where Emacs standardizes to only
 > one completion/selection API, possibly through the `completion-*'
 > functions.

Please explain what you mean by "unify".  If you mean
one-size-fits-all, I think that would be horrible.  There are
completion contexts where I don't want a list at all (dabbrev), and
others where I really, really do (browser completion of long URLs from
long histories).

If you simply mean there should be one function to invoke the
completion system, or a small number of such functions, with a "small"
number of parameters, I agree.  I think this points out the need for a
dispatching architecture, where the completion function exposed in the
API ends up dispatching to several handlers: construct the candidate
list, filter it, prioritize it, optionally present it to the user, and
optionally edit the target text ("auto-complete" suggests that edit
will always happen, but it's not always what is desired of a
completion mechanism -- eg, something I've occasionally wished for is
a "uniquifying completer", so that I don't accidentally duplicate
identifiers -- "you can check out but you can never leave" until the
item is unique in scope :-).

 > SM> - how to display the list of candidates:
 > SM>   - in a separate buffer
 > SM>   - at point, via clever overlay tricks
 > SM>   - at point, via a special frame
 > SM>   - at point, via a x-popup-menu
 > SM>   - ...
 > 
 > This should probably be customizable, but the default IMO should be
 > different between graphical and text mode, the way that `widget-choose'
 > does it.

If you architect the system as dispatching handlers to perform certain
tasks in sequence, you can postpone this decision (and, in fact, most
of the decisions you are discussing here, prematurely IMHO).  It seems
to me that the custom "selectors" for instancing faces (or whatever
the "keys" in a defface are called) would be a good choice for
specifying completion "list display" handlers.  Then you can start
with "one size fits all" with a selector of t and easily generalize.

The only nasty task in this architecture would be if it becomes
necessary to split a handler into two.  However, AFAICT from your
immediate agreement to the agenda Stefan set here, the three of us
(you, Stefan, me) agree 100% on what the options involved are, and I
suspect we could also agree quite quickly on what tasks deserve
separate handlers (some of the options probably need coordinated
decisions to avoid screwing each other up, so should not be in
separate handlers).  That's only 3 people, but I think it's good
evidence that agreement is going to be widespread, given how disparate
our opinions have been on everything else in this thread.

If somebody disagrees on the right setting for one of the options,
they just write their own handler; no need for discussion (unless they
want to propose it for default), and easy to experiment (as we've seen
there are a wide range of opinions on how to do this, experimentation
is an excellent idea).

The hard part is going to be designing selectors for each handler.
Some are going to be mostly dependent on display capability plus user
preference (the list display), others may be purely application-
dependent (selection of existing files is going to imply a list
generated by `directory-files'), so the selectors aren't going to be
one-size-fits-all.  But to start with the only handler selection
mechanism that needs to be implemented is t. :-)  Additional selection
criteria for a given handler can be added with only local surgery to
the dispatcher.

Steve

Re: Emacs completion matches selection UI

[Ted Zlatanov]

On Sat, 21 Dec 2013 12:32:19 +0900 "Stephen J. Turnbull" <stephen@xemacs.org> wrote: 

SJT> Ted Zlatanov writes:
SM> - what completion operations take place:
SM> - insert a few chars at point
SM> - bring up a list of candidates
SM> - ...
>> 
>> I think this should be unified as much as possible into a single
>> operation that takes place in any context, outside of the user's
>> control.  IOW this should be the place where Emacs standardizes to only
>> one completion/selection API, possibly through the `completion-*'
>> functions.

SJT> Please explain what you mean by "unify".  If you mean
SJT> one-size-fits-all, I think that would be horrible.  There are
SJT> completion contexts where I don't want a list at all (dabbrev), and
SJT> others where I really, really do (browser completion of long URLs from
SJT> long histories).

The context will be respected either by being a parameter or being
inferred from the invocation, or both.  The idea is to have a single API
call for all contexts, not to make the operation's effect the same in
all contexts.

SJT> If you simply mean there should be one function to invoke the
SJT> completion system, or a small number of such functions, with a "small"
SJT> number of parameters, I agree.

Right, exactly.

SJT> I think this points out the need for a dispatching architecture,
SJT> where the completion function exposed in the API ends up
SJT> dispatching to several handlers: construct the candidate list,
SJT> filter it, prioritize it, optionally present it to the user, and
SJT> optionally edit the target text ("auto-complete" suggests that edit
SJT> will always happen, but it's not always what is desired of a
SJT> completion mechanism -- eg, something I've occasionally wished for
SJT> is a "uniquifying completer", so that I don't accidentally
SJT> duplicate identifiers -- "you can check out but you can never
SJT> leave" until the item is unique in scope :-).

See the company-mode frontend/backend split, for instance.  This is a
common design.  I don't have a strong opinion about the whole data flow,
I just want to improve the frontend right now, but perhaps these
opportunities will appear if the project takes off.

SM> - how to display the list of candidates:
SM> - in a separate buffer
SM> - at point, via clever overlay tricks
SM> - at point, via a special frame
SM> - at point, via a x-popup-menu
SM> - ...
>> 
>> This should probably be customizable, but the default IMO should be
>> different between graphical and text mode, the way that `widget-choose'
>> does it.

SJT> If you architect the system as dispatching handlers to perform certain
SJT> tasks in sequence, you can postpone this decision (and, in fact, most
SJT> of the decisions you are discussing here, prematurely IMHO).  It seems
SJT> to me that the custom "selectors" for instancing faces (or whatever
SJT> the "keys" in a defface are called) would be a good choice for
SJT> specifying completion "list display" handlers.  Then you can start
SJT> with "one size fits all" with a selector of t and easily
SJT> generalize.

I think postponing these design decisions would not be optimal, but I
understand the temptation and will listen to advice when we get to that
point.

SJT> The only nasty task in this architecture would be if it becomes
SJT> necessary to split a handler into two.  However, AFAICT from your
SJT> immediate agreement to the agenda Stefan set here, the three of us
SJT> (you, Stefan, me) agree 100% on what the options involved are, and I
SJT> suspect we could also agree quite quickly on what tasks deserve
SJT> separate handlers (some of the options probably need coordinated
SJT> decisions to avoid screwing each other up, so should not be in
SJT> separate handlers).

I still need a clear agreement from Stefan before proceeding.  Then I
will try to get a team together and work outside of the normal Emacs
commit flow so we can iterate quickly.

SJT> That's only 3 people, but I think it's good evidence that agreement
SJT> is going to be widespread, given how disparate our opinions have
SJT> been on everything else in this thread.

Discussions of UI improvements are one of the rare occasions where
everyone has at least one strong opinion.  I think this one has been
quite smooth, I've only thrown one keyboard out the window and AFAIK
I'm not yet in Stefan's killfile ;)

Ted

Re: Emacs completion matches selection UI

[Stefan Monnier]

> (that reminds me I should write a CFEngine-specific completion backend
> for company-mode, like I did for general completion in cfengine.el :)

Please, make it a completion-at-point-function (company-mode can use it
just fine).  See lisp-completion-at-point for an example which provides
company-specific support.


        Stefan

Re: Emacs completion matches selection UI

[Stefan Monnier]

SM> - what completion operations take place:
SM>   - insert a few chars at point
SM>   - bring up a list of candidates
SM>   - ...
> I think this should be unified as much as possible into a single
> operation that takes place in any context, outside of the user's
> control.

Here I lost you.  Currently, the kind of operation can be:
 - list candidates (completion-help-at-point)
 - insert a few chars (completion-at-point)
 - cycle through candidates (completion-at-point with
   completion-cycle-threshold, or minibuffer-force-complete).
AFAIK the users often like to have the choice between the three.
Tho admittedly, TAB is the main entry point.
   
> IOW this should be the place where Emacs standardizes to only one
> completion/selection API, possibly through the
> `completion-*' functions.

And here I lost you as well: I thought this section was about commands
and behaviors, whereas "API" to me means "functions" (as in "called from
Elisp").

SM> - what operations can be performed while the list of candidates is displayed:
SM>   - keep on editing as if the list was not displayed
SM>   - select among candidates with the mouse
SM>   - select with up/down
SM>   - ...
> The current defaults for this are not consistent and were the original
> reason for starting the thread.

Hmm... Could you explain which parts are not consistent?

> The default here is tricky, since we have the in-buffer
> vs. in-minibuffer split to consider.  I agree with Dmitry that
> auto-narrowing the candidates is important by default;

That should be easy to implement (tho admittedly, currently this will
require 2 different implementations for in-buffer and in minibuffer).
For the in-buffer case, it might be as simple as adding a call to
completion-help-at-point at the right place.

> the up/down selection is important too but must not interfere with
> buffer motion when completing in a buffer of course.  If we can find
> something consistent that works as a default in all the invocation
> contexts, I think that would offer the biggest benefit to
> Emacs usability.

Usually, if it works for in-buffer, it works as well in the
minibuffer case.

SM> - how/when to update the list of completions and to pop it down.
> That should be entirely internal to the API and not exposed, I think.

"Internal to the API" is completely meaningless to me.  If it's not
exposed, then it's not in the API (i.e. it's only in the
chosen UI's implementation).

SM> - what Lisp function to call to display the list of elements.
SM> We currently don't really have something clear here.  IIUC this is the
SM> API part you care about (the rest is mostly behavior rather than API).
> Yes, this is the API that should get called by any package (internal or
> external) that wants to ask the user to interactively select completion
> candidates or just one item from a list.

I still don't quite understand what you want here.  Are you thinking of
a function like x-popup-menu or display-completion-list?


        Stefan

Re: Emacs completion matches selection UI

[Dmitry Gutov]

Stefan Monnier <monnier@IRO.UMontreal.CA> writes:

>> (that reminds me I should write a CFEngine-specific completion backend
>> for company-mode, like I did for general completion in cfengine.el :)
>
> Please, make it a completion-at-point-function (company-mode can use it
> just fine).  See lisp-completion-at-point for an example which provides
> company-specific support.

`company-capf' is not in `company-backends' by default, though.

So CFEngine will need to document adding it, or just setting
`company-backends' to `(company-capf)' locally in CFEngine buffers.

Re: Emacs completion matches selection UI

[Stefan Monnier]

> `company-capf' is not in `company-backends' by default, though.

Could you fix that, please?


        Stefan

Re: Emacs completion matches selection UI

[Dmitry Gutov]

Stefan Monnier <monnier@IRO.UMontreal.CA> writes:

>> `company-capf' is not in `company-backends' by default, though.
>
> Could you fix that, please?

Okay, it'll be there in the next release.

I had a few reservations about it, but the outward inefficiency of
calling all capf functions each time we issue a backend command
shouldn't be slow enough to notice in practice, and the Emacs 24
requirement can be taken care of by checking the version at load time.

By the way, do you think `lisp-completion-at-point' now has feature
parity with `company-elisp'? We could replace the latter with
`company-capf' depending on the Emacs version, but I'm not sure where to
make the cutoff. If it's just Emacs 24, the users of 24.1-24.3 will miss
out on features. If it's 24.4, CFEngine completion integration with
Company won't work with earlier versions.

The last point might be a problem for Ted anyway, though, since CFEngine
seems to support the earlier Emacs versions.

enabling company-capf support in cfengine.el (was: Emacs completion matches selection UI)

[Ted Zlatanov]

On Sun, 22 Dec 2013 18:07:01 +0200 Dmitry Gutov <dgutov@yandex.ru> wrote: 

DG> Stefan Monnier <monnier@IRO.UMontreal.CA> writes:
>>> `company-capf' is not in `company-backends' by default, though.
>> 
>> Could you fix that, please?

DG> Okay, it'll be there in the next release.

DG> I had a few reservations about it, but the outward inefficiency of
DG> calling all capf functions each time we issue a backend command
DG> shouldn't be slow enough to notice in practice, and the Emacs 24
DG> requirement can be taken care of by checking the version at load time.

DG> By the way, do you think `lisp-completion-at-point' now has feature
DG> parity with `company-elisp'? We could replace the latter with
DG> `company-capf' depending on the Emacs version, but I'm not sure where to
DG> make the cutoff. If it's just Emacs 24, the users of 24.1-24.3 will miss
DG> out on features. If it's 24.4, CFEngine completion integration with
DG> Company won't work with earlier versions.

DG> The last point might be a problem for Ted anyway, though, since CFEngine
DG> seems to support the earlier Emacs versions.

Not a problem.  I maintain two versions of cfengine.el: one in
https://github.com/cfengine/core/blob/master/contrib/cfengine.el which
maintains backwards compatibility and can miss new features, and one in
the Emacs trunk, which doesn't try to maintain compatibility.  So any
cool new things go into the Emacs trunk and then I will try to backport
them.

Anyhow, right now I do this in `cfengine3-mode':

#+begin_src lisp
  (add-hook 'completion-at-point-functions
            #'cfengine3-completion-function nil t)
#+end_src

Is that enough?  I can recommend to users that they should enable the
`company-capf' backend in the cfengine.el docs, if so.

Ted

Re: Emacs completion matches selection UI

[Ted Zlatanov]

On Sat, 21 Dec 2013 21:05:24 -0500 Stefan Monnier <monnier@IRO.UMontreal.CA> wrote: 

SM> - what completion operations take place:
SM> - insert a few chars at point
SM> - bring up a list of candidates
SM> - ...
>> I think this should be unified as much as possible into a single
>> operation that takes place in any context, outside of the user's
>> control.

SM> Here I lost you.  Currently, the kind of operation can be:
SM>  - list candidates (completion-help-at-point)
SM>  - insert a few chars (completion-at-point)
SM>  - cycle through candidates (completion-at-point with
SM>    completion-cycle-threshold, or minibuffer-force-complete).
SM> AFAIK the users often like to have the choice between the three.
SM> Tho admittedly, TAB is the main entry point.

I was thinking of these as the effect of the operation, and the
operation is "completion DWIM."  The semantics here are a minefield and
there is a huge range of customizations available.

>> IOW this should be the place where Emacs standardizes to only one
>> completion/selection API, possibly through the
>> `completion-*' functions.

SM> And here I lost you as well: I thought this section was about commands
SM> and behaviors, whereas "API" to me means "functions" (as in "called from
SM> Elisp").

I mean TAB (and its equivalent in-buffer keybinding) should be bound to
a single Elisp function, and the effect (list candidates, complete
immediately the single candidate, or cycle candidates depending on the
user preferences) should be consistent in any invocation context.

SM> - what operations can be performed while the list of candidates is displayed:
SM>   - keep on editing as if the list was not displayed
SM>   - select among candidates with the mouse
SM>   - select with up/down
SM>   - ...
>> The current defaults for this are not consistent and were the original
>> reason for starting the thread.

SM> Hmm... Could you explain which parts are not consistent?

"Select with up/down" is not consistent between in-minibuffer and
in-buffer completion.  You can always go into the *Completions* buffer
and then use up/down, but up/down immediately have different effects
otherwise.  I think they should do the same thing, enter a special popup
as Dmitry and I described.

"Keep on editing" acts differently as well.

Mouse selection works the same way, I think.

>> The default here is tricky, since we have the in-buffer
>> vs. in-minibuffer split to consider.  I agree with Dmitry that
>> auto-narrowing the candidates is important by default;

SM> That should be easy to implement (tho admittedly, currently this will
SM> require 2 different implementations for in-buffer and in minibuffer).
SM> For the in-buffer case, it might be as simple as adding a call to
SM> completion-help-at-point at the right place.

OK, so do we agree on this specific improvement?  "Whether in the
minibuffer or in a buffer, typing more characters should narrow the list
of candidates without having to press TAB or other special keys."

>> the up/down selection is important too but must not interfere with
>> buffer motion when completing in a buffer of course.  If we can find
>> something consistent that works as a default in all the invocation
>> contexts, I think that would offer the biggest benefit to
>> Emacs usability.

SM> Usually, if it works for in-buffer, it works as well in the
SM> minibuffer case.

Good point.  But why not simply enter an auto-narrowing popup in both
contexts?  The user has explicitly requested completion, right?  I keep
thinking that if in both contexts we locked the user into a selection
popup, the up/down and motion issues would be easily solved.  It would
also allow `C-1' through `C-9' (or some similar keys) to be used to
immediately select a candidate by position, like `widget-choose' does it.

SM> - how/when to update the list of completions and to pop it down.
>> That should be entirely internal to the API and not exposed, I think.

SM> "Internal to the API" is completely meaningless to me.  If it's not
SM> exposed, then it's not in the API (i.e. it's only in the
SM> chosen UI's implementation).

Right, sorry.  It should be internal to the implementation.

SM> - what Lisp function to call to display the list of elements.
SM> We currently don't really have something clear here.  IIUC this is the
SM> API part you care about (the rest is mostly behavior rather than API).
>> Yes, this is the API that should get called by any package (internal or
>> external) that wants to ask the user to interactively select completion
>> candidates or just one item from a list.

SM> I still don't quite understand what you want here.  Are you thinking of
SM> a function like x-popup-menu or display-completion-list?

Yes, `display-completion-list' but hacked to use `widget-choose' at
least in graphical mode and to auto-narrow.  This should work with
(interactive) calls, of course, as a common use case.

So let's say I'm inside gnutls.el and want to ask the user to select a
client certificate to present to the remote server.  I have a list of 3
such certificates.  I would say `(display-completion-list
list-of-certificates)' and expect the result to be nil or an element of
the list.  Does that usage make sense?

Ted

Re: enabling company-capf support in cfengine.el

[Dmitry Gutov]

Ted Zlatanov <tzz@lifelogs.com> writes:

> Anyhow, right now I do this in `cfengine3-mode':
>
> #+begin_src lisp
>   (add-hook 'completion-at-point-functions
>             #'cfengine3-completion-function nil t)
> #+end_src
>
> Is that enough?  I can recommend to users that they should enable the
> `company-capf' backend in the cfengine.el docs, if so.

Should be. As long as you can't (or don't indend to) support the more
advanced features, such as showing candidate signature (when it's a
function), its definition and documentation. No idea if any of that is
even applicable to CFEngine.

But you also might want to wait until the next Company version is
released, and then only ask users to install it, without having to
change the list of backends.

Re: enabling company-capf support in cfengine.el

[Ted Zlatanov]

On Mon, 23 Dec 2013 02:17:35 +0200 Dmitry Gutov <dgutov@yandex.ru> wrote: 

DG> Ted Zlatanov <tzz@lifelogs.com> writes:
>> Anyhow, right now I do this in `cfengine3-mode':
>> 
>> #+begin_src lisp
>> (add-hook 'completion-at-point-functions
>> #'cfengine3-completion-function nil t)
>> #+end_src
>> 
>> Is that enough?  I can recommend to users that they should enable the
>> `company-capf' backend in the cfengine.el docs, if so.

DG> Should be. As long as you can't (or don't indend to) support the more
DG> advanced features, such as showing candidate signature (when it's a
DG> function), its definition and documentation. No idea if any of that is
DG> even applicable to CFEngine.

It is.  I provide ElDoc support currently.  Is there a way to avoid
custom work for company-mode, and instead provide a commonly
understandable list of function definitions that works for ElDoc and for
company-mode?  ElDoc is pretty simple, it expects me to do all the work
to find the current function and write the mini-docstring for it.

CFEngine's syntax is actually very rigid and predictable.  There are
only specific things that can go at every point in the syntax, so it
could use completion not just for function calls.  It's a bit like XML
with an enforced schema, and autocompletion could work really well
there.  I haven't had the time to explore that direction, there's too
much on my TODO list :) but it seems like Semantic integration would
actually make sense instead of custom code for this.  Advice welcome.

DG> But you also might want to wait until the next Company version is
DG> released, and then only ask users to install it, without having to
DG> change the list of backends.

OK, sounds good.  Thanks.

Ted

Re: enabling company-capf support in cfengine.el

[Dmitry Gutov]

Ted Zlatanov <tzz@lifelogs.com> writes:

> It is.  I provide ElDoc support currently.  Is there a way to avoid
> custom work for company-mode, and instead provide a commonly
> understandable list of function definitions that works for ElDoc and for
> company-mode?  ElDoc is pretty simple, it expects me to do all the work
> to find the current function and write the mini-docstring for it.

Not at the moment, AFAIK. You're not the first person to notice this,
see the FIXME at lisp/emacs-lisp/lisp.el#755.

ElDoc works with the contents of the buffer, and until the user picks a
candidate, the buffer only contains the prefix.

But anyway, you're just duplicating a similar call from two places. The
bulk of signature-fetching logic should be reusable.

> it seems like Semantic integration would
> actually make sense instead of custom code for this.  Advice welcome.

Maybe. Sorry, I don't know too much about Semantic.

Re: Emacs completion matches selection UI

[Stefan Monnier]

> I had a few reservations about it, but the outward inefficiency of
> calling all capf functions each time we issue a backend command
> shouldn't be slow enough to notice in practice, and the Emacs 24
> requirement can be taken care of by checking the version at load time.

I'll trust your judgment on those issues.  But if there are performance
problems with having company-capf in company-backends, it would be very
useful to know it, so please add it by default, so we can fix those
problems (presumably in the respective completion-at-point-functions)
rather than avoid them.

> By the way, do you think `lisp-completion-at-point' now has feature
> parity with `company-elisp'?

In Emacs-24.4 it should, yes.

> We could replace the latter with `company-capf' depending on the Emacs
> version, but I'm not sure where to make the cutoff.  If it's just Emacs
> 24, the users of 24.1-24.3 will miss out on features. If it's 24.4,
> CFEngine completion integration with Company won't work with
> earlier versions.

AFAIK cfengine's completion support didn't exist before 24.4, so there's
no problem there.


        Stefan

Re: Emacs completion matches selection UI

[Stefan Monnier]

> I mean TAB (and its equivalent in-buffer keybinding) should be bound to
> a single Elisp function, and the effect (list candidates, complete
> immediately the single candidate, or cycle candidates depending on the
> user preferences) should be consistent in any invocation context.

Not sure what you feel is missing here.  `completion-at-point' is
basically it, AFAICT.  But it's OK if different UIs provide other
commands (or other ways, since completion can also happen as
a side-effect of normal commands, as is the case with auto-complete and
company-mode).

The important point is that the completion command to use shouldn't
depend on the "backend" (e.g. on the current major mode), and AFAIK
we're almost there (the new completion code in Emacs-23 and subsequent
bit-by-bit updates of major modes to make use of
completion-at-point-functions was dedicated to this task).
Can't remember what are the remaining exceptions, other than Eshell.

SM> Hmm... Could you explain which parts are not consistent?
> "Select with up/down" is not consistent between in-minibuffer and
> in-buffer completion.

AFAIK it is consistent: neither in minibuffer nor in in-buffer up/down
is bound to a completion-related command.  It's not what you want, but
it's consistent w.r.t the completion UI.

> You can always go into the *Completions* buffer and then use up/down,
> but up/down immediately have different effects otherwise.

Lots of other keys have different effects in minibuffers and normal
buffers, but these don't make the completion code inconsistent in
this respect.

> I think they should do the same thing, enter a special popup
> as Dmitry and I described.

If/when those keys are given a completion-related meaning I'd expect it
to be the same in both contexts, yes.

> "Keep on editing" acts differently as well.

Details?

> OK, so do we agree on this specific improvement?  "Whether in the
> minibuffer or in a buffer, typing more characters should narrow the list
> of candidates without having to press TAB or other special keys."

If you add the qualifier "when the completion list is displayed",
I think I agree, yes (I actually hesitated to add that feature when
wrote the new minibuffer.el code, but then decided to keep it for later,
sine there was plenty of other behavioral changes introduced at that
time already).  You might want to make it configurable, but I think
enabling it by default would be OK.

SM> Usually, if it works for in-buffer, it works as well in the
SM> minibuffer case.
> Good point.  But why not simply enter an auto-narrowing popup in both
> contexts?

Didn't I just agree in the previous paragraph?

> The user has explicitly requested completion, right?  I keep
> thinking that if in both contexts we locked the user into a selection
> popup, the up/down and motion issues would be easily solved.  It would
> also allow `C-1' through `C-9' (or some similar keys) to be used to
> immediately select a candidate by position, like `widget-choose' does it.

As I already said in earlier discussions, it might very well be
acceptable to hijack those keys.  But only experience can confirm it.

SM> I still don't quite understand what you want here.  Are you thinking of
SM> a function like x-popup-menu or display-completion-list?
> Yes, `display-completion-list' but hacked to use `widget-choose' at
> least in graphical mode and to auto-narrow.

Here again you're confusing API and implementation.  Be very careful,
please, because it's making this discussion difficult.

The API design should presumably make it possible to use company-mode's
overlay-based popups, or widget-choose's popup, or good ol'
*Completions* buffer.  It should maybe also make it possible for the
caller to provide some data that might influence which kind of popup to
use (in case the kind of popup to use might depend on the context: not
sure if that's really necessary), tho the ultimate choice should be with
the user's config, of course.

> This should work with (interactive) calls, of course, as a common
> use case.

I'm again lost here.  We're talking about a *function* which needs to be
told *by another Lisp function* what it is that needs to be displayed.
So it's meaningless to consider interactive calls.

> So let's say I'm inside gnutls.el and want to ask the user to select a
> client certificate to present to the remote server.  I have a list of 3
> such certificates.  I would say `(display-completion-list
> list-of-certificates)' and expect the result to be nil or an element of
> the list.  Does that usage make sense?

No.  That would seem to call for a completely different API meant to
generalize completing-read rather than display-completion-list.


        Stefan

Re: Emacs completion matches selection UI

[Dmitry Gutov]

On 23.12.2013 03:36, Stefan Monnier wrote:
> I'll trust your judgment on those issues.  But if there are performance
> problems with having company-capf in company-backends, it would be very
> useful to know it, so please add it by default, so we can fix those
> problems (presumably in the respective completion-at-point-functions)
> rather than avoid them.

Okay, I started on it, and there's a problem: `company-capf' doesn't 
work in IELM buffers, which will be a regression against 
https://github.com/company-mode/company-mode/issues/42.

Any suggestions for a quick fix? `ielm-complete-symbol' doesn't look 
like a proper completion function.

Re: Emacs completion matches selection UI

[Ted Zlatanov]

On Sun, 22 Dec 2013 20:59:49 -0500 Stefan Monnier <monnier@IRO.UMontreal.CA> wrote: 

>> I mean TAB (and its equivalent in-buffer keybinding) should be bound to
>> a single Elisp function, and the effect (list candidates, complete
>> immediately the single candidate, or cycle candidates depending on the
>> user preferences) should be consistent in any invocation context.

SM> Not sure what you feel is missing here.  `completion-at-point' is
SM> basically it, AFAICT.  But it's OK if different UIs provide other
SM> commands (or other ways, since completion can also happen as
SM> a side-effect of normal commands, as is the case with auto-complete and
SM> company-mode).

SM> The important point is that the completion command to use shouldn't
SM> depend on the "backend" (e.g. on the current major mode), and AFAIK
SM> we're almost there (the new completion code in Emacs-23 and subsequent
SM> bit-by-bit updates of major modes to make use of
SM> completion-at-point-functions was dedicated to this task).
SM> Can't remember what are the remaining exceptions, other than Eshell.

OK, is there a way to make a verification and testing plan for this?  If
there was some automation around completion support, it would certainly
speed things up, but we need at least a list of currently non-compliant
modes in Emacs.  How do you look for the exceptions, by searching for
old functions or testing manually?

SM> Hmm... Could you explain which parts are not consistent?
>> "Select with up/down" is not consistent between in-minibuffer and
>> in-buffer completion.

SM> AFAIK it is consistent: neither in minibuffer nor in in-buffer up/down
SM> is bound to a completion-related command.  It's not what you want, but
SM> it's consistent w.r.t the completion UI.

Using `up' in the minibuffer goes back in the history currently, which
invalidates the current completion candidates, so it does have some
effect.  But I agree it's not directly completion-related.

>> "Keep on editing" acts differently as well.

SM> Details?

Minor.

>> OK, so do we agree on this specific improvement?  "Whether in the
>> minibuffer or in a buffer, typing more characters should narrow the list
>> of candidates without having to press TAB or other special keys."

SM> If you add the qualifier "when the completion list is displayed",
SM> I think I agree, yes (I actually hesitated to add that feature when
SM> wrote the new minibuffer.el code, but then decided to keep it for later,
SM> sine there was plenty of other behavioral changes introduced at that
SM> time already).  You might want to make it configurable, but I think
SM> enabling it by default would be OK.

SM> Usually, if it works for in-buffer, it works as well in the
SM> minibuffer case.

>> Good point.  But why not simply enter an auto-narrowing popup in both
>> contexts?

SM> Didn't I just agree in the previous paragraph?

OK.

>> The user has explicitly requested completion, right?  I keep
>> thinking that if in both contexts we locked the user into a selection
>> popup, the up/down and motion issues would be easily solved.  It would
>> also allow `C-1' through `C-9' (or some similar keys) to be used to
>> immediately select a candidate by position, like `widget-choose' does it.

SM> As I already said in earlier discussions, it might very well be
SM> acceptable to hijack those keys.  But only experience can confirm it.

So auto-narrowing (customizable) and a graphical popup (customizable as
a frontend choice) are OK.  Hijacking keys is to be tested further and a
demo is needed.  All this will happen with `completion-at-point'.

SM> The API design should presumably make it possible to use company-mode's
SM> overlay-based popups, or widget-choose's popup, or good ol'
SM> *Completions* buffer.

As with company-mode, this can be a frontend choice (affecting
`display-completion-list', and see below for the suggested
`display-list' function).  How about a defcustom
`completion-ui-frontend' with some choices?

SM> It should maybe also make it possible for the caller to provide some
SM> data that might influence which kind of popup to use (in case the
SM> kind of popup to use might depend on the context: not sure if that's
SM> really necessary), tho the ultimate choice should be with the user's
SM> config, of course.

For the user, a new defcustom `completion-ui-preferences'?  And for the
caller, some kind of plist/alist passed to `completion-at-point' that
merges cleanly with `completion-ui-preferences'?

s/completion-ui/something-else/ as you see fit.

>> So let's say I'm inside gnutls.el and want to ask the user to select a
>> client certificate to present to the remote server.  I have a list of 3
>> such certificates.  I would say `(display-completion-list
>> list-of-certificates)' and expect the result to be nil or an element of
>> the list.  Does that usage make sense?

SM> No.  That would seem to call for a completely different API meant to
SM> generalize completing-read rather than display-completion-list.

I would like that, too, so choosing from a list of elements can use all
the improvements suggested here for `display-completion-list'.  Could
this be a new `display-list' function, used to display both completion
selection (through `display-completion-list') and list selection?

Ted

Re: Emacs completion matches selection UI

[Stefan Monnier]

>> I'll trust your judgment on those issues.  But if there are performance
>> problems with having company-capf in company-backends, it would be very
>> useful to know it, so please add it by default, so we can fix those
>> problems (presumably in the respective completion-at-point-functions)
>> rather than avoid them.
> Okay, I started on it, and there's a problem: `company-capf' doesn't work in
> IELM buffers,

Please make it a bug report.


        Stefan

Re: Emacs completion matches selection UI

[Stefan Monnier]

> OK, is there a way to make a verification and testing plan for this?  If
> there was some automation around completion support, it would certainly
> speed things up, but we need at least a list of currently non-compliant
> modes in Emacs.  How do you look for the exceptions, by searching for
> old functions or testing manually?

The code which does not use minibuffer.el for its completion generally
rolls its own.  Grepping for *Completions* and display-completion-list is
what I've done so far, but I probably missed some.

I don't think it's terribly important to make sure we got 100% of the cases.

> As with company-mode, this can be a frontend choice (affecting
> `display-completion-list', and see below for the suggested
> `display-list' function).  How about a defcustom
> `completion-ui-frontend' with some choices?

The general idea of completion-ui was great, yes.

> For the user, a new defcustom `completion-ui-preferences'?  And for the
> caller, some kind of plist/alist passed to `completion-at-point' that
> merges cleanly with `completion-ui-preferences'?

Could be.  Tho as mentioned before, I'm not sure that there's a need for
caller-provided preferences.

> I would like that, too, so choosing from a list of elements can use all
> the improvements suggested here for `display-completion-list'.

Not sure what you mean by that: minibuffer completion would presumably
use the same completion mechanism as in-buffer completion (as is
already the case, to a large extent), so completing-read would already
use the new mechanisms.

> Could this be a new `display-list' function, used to display both
> completion selection (through `display-completion-list') and
> list selection?

Could be.  I don't know enough in which contexts "list selection" could
be used to know if/how it would make sense.


        Stefan

Re: Emacs completion matches selection UI

[John Yates]

[-- Attachment #1: Type: text/plain, Size: 398 bytes --]

Been following this thread as I would love to see more uniformity in these
areas.

A related issue is how multiple completion / expansion mechanisms interact
at a given point.  I know trying to get yasnippet to play gracefully with
other completion mechanisms has generated much discussion.  So as you think
about APIs perhaps you could give some thought to higher level integration
issues.

/john

[-- Attachment #2: Type: text/html, Size: 581 bytes --]

Re: Emacs completion matches selection UI

[Ted Zlatanov]

On Mon, 23 Dec 2013 08:42:51 -0500 Stefan Monnier <monnier@IRO.UMontreal.CA> wrote: 

>> OK, is there a way to make a verification and testing plan for this?  If
>> there was some automation around completion support, it would certainly
>> speed things up, but we need at least a list of currently non-compliant
>> modes in Emacs.  How do you look for the exceptions, by searching for
>> old functions or testing manually?

SM> The code which does not use minibuffer.el for its completion generally
SM> rolls its own.  Grepping for *Completions* and display-completion-list is
SM> what I've done so far, but I probably missed some.

SM> I don't think it's terribly important to make sure we got 100% of the cases.

OK, I'll move on.  If the new UI proves compelling, users will tell us
they want it supported in their favorite modes :)

>> As with company-mode, this can be a frontend choice (affecting
>> `display-completion-list', and see below for the suggested
>> `display-list' function).  How about a defcustom
>> `completion-ui-frontend' with some choices?

SM> The general idea of completion-ui was great, yes.

>> For the user, a new defcustom `completion-ui-preferences'?  And for the
>> caller, some kind of plist/alist passed to `completion-at-point' that
>> merges cleanly with `completion-ui-preferences'?

SM> Could be.  Tho as mentioned before, I'm not sure that there's a need for
SM> caller-provided preferences.

OK, I'll work on this off-list to keep the noise down, I've generated
enough emacs-devel traffic so far!

Dmitry, Stephen, please let me know directly if you want to participate
in the design of `completion-ui-*' as described here.  I honestly can't
keep track of all the completion packages available, but I'm
xposting/CC-ing the Helm developers, and Toby Cubitt (because of
http://www.emacswiki.org/emacs/CompletionUI the prefix "completion-ui"
may need to be changed, but it seems to have similar ideas to the
frontends we're describing here).  Drew Adams (Icicles) and Kim Storm
(ido) may also be interested but I think they read emacs-devel.  For all
the autocomplete forks and add-ons, I have no idea who to CC.  Anyhow,
if you're interested, reply to me directly please.

Thanks
Ted

Re: Emacs completion matches selection UI

[Stefan Monnier]

> at a given point.  I know trying to get yasnippet to play gracefully with
> other completion mechanisms has generated much discussion.

I didn't know.  Can you describe some of the problems w.r.t interaction
between completion and yasnippet?


        Stefan

Re: Emacs completion matches selection UI

[John Yates]

[-- Attachment #1: Type: text/plain, Size: 796 bytes --]

Try a google search for "yasnippet autocomplete".  Many folk seem to want
to use them together.  Not sure why the attempt so often leads to a plea
for help.

Perhaps the wrinkle is yasnippet's completion model.  Based on context
preceding point yasnippet decides either that a match exists (leading to
template expansion and return) or that no match exists  causing
responsibility to be passed to a fallback completion handler.

/john


On Mon, Dec 23, 2013 at 11:02 AM, Stefan Monnier
<monnier@iro.umontreal.ca>wrote:

> > at a given point.  I know trying to get yasnippet to play gracefully with
> > other completion mechanisms has generated much discussion.
>
> I didn't know.  Can you describe some of the problems w.r.t interaction
> between completion and yasnippet?
>
>
>         Stefan
>

[-- Attachment #2: Type: text/html, Size: 1259 bytes --]

Re: Emacs completion matches selection UI

[Stefan Monnier]

> Perhaps the wrinkle is yasnippet's completion model.  Based on context
> preceding point yasnippet decides either that a match exists (leading to
> template expansion and return) or that no match exists  causing
> responsibility to be passed to a fallback completion handler.

Ah, right, the conflict is in the meaning of the TAB key.
There are two issues there:
1- the fact that there's no standard way to *extend* the meaning of
   TAB, so autocomplete and yasnippet may both *redefine* the key instead.
2- even if the two manage to redefine TAB "at the same time", the
   resulting behavior may prove too DWIMish.


        Stefan

Re: Emacs completion matches selection UI

[João Távora]

John Yates <john@yates-sheets.org> writes:

> Try a google search for "yasnippet autocomplete". Many folk seem to
> want to use them together. Not sure why the attempt so often leads to
> a plea for help.

Many of the search results will land you in yasnippet's issue tracker,
where I handled many such issues and admittedly never provided a
bullet-proof solution.

But I would say that 70% of the cases stem from a misunderstanding about
yasnippet. It is a abbrev expansion mechanism and not a completion
mechanism.

The fact that the key that attempts expansion is bound to TAB is a
detail. The fact that it sometimes prompts you when many expansions are
possible from a single abbreviation, and that that minibuffer prompting
*is* subject to usual emacs completion routines is also a different
issue, IMO.

So, those 70% are frequently caused by the fact that auto-complete
itself comes with a "yasnippet source", i.e. a plugin to auto-complete
that gathers possible snippets to expand at point and offers these
expansions as completion "targets". In this (completely different) use
case, auto-complete "drives" yasnippet (and yasnippet's TAB binding
should be disabled by the user). This use case is frequently broken by
reasons completely unrelated to this thread (i.e. API brea).

Anyway, the other 30% are probably legitimate cases where the user that
activated both yas-minor-mode and auto-complete-mode in a buffer
probably wants something like the following to happen:

   1. Press TAB

   2. If theres an active snippet key preceding point, expand it and
      leave

   3. Else attempt autocompletion at point, bring up a menu, whatever it
      does

   4. Else do whatever TAB is bound to

I wrote "probably" because I don't think I ever got a precise
description that clarified the desired interaction. That is, maybe some
users want 2 and 3 reversed.

Now, I'm not familiar with auto-complete to say if 3. can ever "fail"
and fall through to 4. To do that, it would probably need a fallback
mechanism similar to yasnippet's.

IIUC, it is this "attempt-command-but-maybe-fallback" mechanism that I
believe could be considered for inclusion in emacs, provided it's
designed sensibly, of course.

> Perhaps the wrinkle is yasnippet's completion model. Based on context
> preceding point yasnippet decides either that a match exists (leading
> to template expansion and return) or that no match exists causing
> responsibility to be passed to a fallback completion handler.

This is very close to the truth, but instead of "fallback completion
handler" yasnippet interactively calls whatever TAB was bound to before,
which normally is some indentation command, but can also be org-cycle as
in org-mode.

Not sure what it does if you enable yas-minor-mode in a completion-aware
minibuffer, for instance.

C-h k TAB in any buffer where yas-minor-mode is enabled should always
give you that info, btw. (I say "should" because it appears to currently
be buggy and always report "forward-button" as the fallback command).

Finally, note that autopair.el does something similar, but always runs
the fallback binding and installs its bindings in
`emulation-mode-map-alist' something that I found that is usually "on
top of everything else".

Yasnippet installs its TAB binding in a conventional minor mode map,
where IIRC it will conflict with other TAB bindings in other minor mode
maps. This used to be a common source of problems, but can be averted
with hacks like the ones described in yasnippet's faq (though these
might be out-of-date).

Hope I have helped and not unhelped with this input,
João

Re: Emacs completion matches selection UI

[John Yates]

[-- Attachment #1: Type: text/plain, Size: 838 bytes --]

On Sat, Dec 28, 2013 at 9:02 AM, Stefan Monnier <monnier@iro.umontreal.ca>wrote:

> Ah, right, the conflict is in the meaning of the TAB key.
> There are two issues there:
> 1- the fact that there's no standard way to *extend* the meaning of
>    TAB, so autocomplete and yasnippet may both *redefine* the key instead.
> 2- even if the two manage to redefine TAB "at the same time", the
>    resulting behavior may prove too DWIMish.
>

This seems to be an instance of a general pattern, namely a succession of
completion mechanisms desiring to be attempted based on a particular
triggering event.
 Does it not then follow that part of creating a more uniform completion
experience ought to include direct Emacs support for that pattern?  The
alternative is today's hodgepodge of per-package idiosyncratic redefinition
mechanisms.

/john

[-- Attachment #2: Type: text/html, Size: 1291 bytes --]

Re: Emacs completion matches selection UI

[John Yates]

[-- Attachment #1: Type: text/plain, Size: 1355 bytes --]

João, Thanks for the detailed reply.  Our replies to Stefan crossed each
other.

On Sat, Dec 28, 2013 at 10:59 AM, João Távora <joaotavora@gmail.com> wrote:
>
> Anyway, the other 30% are probably legitimate cases where the user that
> activated both yas-minor-mode and auto-complete-mode in a buffer
> probably wants something like the following to happen:
>
>    1. Press TAB
>
>    2. If theres an active snippet key preceding point, expand it and
>       leave
>
>    3. Else attempt autocompletion at point, bring up a menu, whatever it
>       does
>
>    4. Else do whatever TAB is bound to


> I wrote "probably" because I don't think I ever got a precise
> description that clarified the desired interaction. That is, maybe some
> users want 2 and 3 reversed.
>

Yes, that (yasnippet ahead of autocomplete) is what I have attempted to
configure for myself.  To ensure 3 remains accessible I try to give my
snippets names with low likelihoods for collision.

[..SNIP..]

Again, thanks for the long authoritative reply.  My take away is that
configuring multiple packages to trigger in some meaningful, predictable
manner from a single event is currently part of the "dark arts", not
something newbies should attempt.  It would be wonderful if some structure
and formalism could be brought to this area.

/john

[-- Attachment #2: Type: text/html, Size: 2051 bytes --]

Re: Emacs completion matches selection UI

[Josh]

On Sat, Dec 28, 2013 at 9:00 AM, John Yates <john@yates-sheets.org> wrote:
> My take away is that configuring multiple packages to trigger in some
> meaningful, predictable manner from a single event is currently part of the
> "dark arts", not something newbies should attempt.  It would be wonderful if
> some structure and formalism could be brought to this area.

Something built around `run-hook-with-args-until-success' might be a good
approach; see for example the docstring for `org-ctrl-c-ctrl-c-hook'.

Re: Emacs completion matches selection UI

[Stefan Monnier]

>  Does it not then follow that part of creating a more uniform completion
> experience ought to include direct Emacs support for that pattern?  The
> alternative is today's hodgepodge of per-package idiosyncratic redefinition
> mechanisms.

Yes.  We should make it easier to extend indent-for-tab-command.


        Stefan

Re: Emacs completion matches selection UI

[João Távora]

Stefan Monnier <monnier@IRO.UMontreal.CA> writes:

>>  Does it not then follow that part of creating a more uniform completion
>> experience ought to include direct Emacs support for that pattern?  The
>> alternative is today's hodgepodge of per-package idiosyncratic redefinition
>> mechanisms.
>
> Yes.  We should make it easier to extend indent-for-tab-command.

Do you think it's possible/desirable to generalize this mechanism (to
support the "fallback-maybe" pattern) so that we can make it easier to
extend any command, not just indent-for-tab-command?

In other words, I believe this could be useful for any binding. I've yet
to think of a nice interface, here's just a thought

(define-fallbackish-command maybe-foo (kbd "[somekey]") ()
  (interactive)
  (if (foop)
      (foo)
      (fallback)))

(define-fallbackish-command maybe-bar (kbd "[somekey]") (args...)
  (interactive ...)
  (if (barp)
      (bar)
      (fallback)))

Now, in a buffer where the active binding (set in whatever manner and
following the normal priority rules) for [somekey] is maybe-bar:

* if (barp) returns true, (bar) is run

* if (barp) fails and (foop) return true, (foo) is run

* if both (barp) and (foop) fail, we get the original keybinding
  [somekey], or nothing if it wasn't bound.

In a buffer where the active binding for [somekey] is maybe-foo, only
(foo) can ever be run, maybe-bar doesn't enter into play, because it was
defined later.

Also note that the fallback doesn't have to be mutually exclusive with
the actual (bar) and (foo) calls, I just thought of this example.

João

Re: Emacs completion matches selection UI

[Toby Cubitt]

Sorry to arrive late to the party :-)

Ted Zlatanov <tzz <at> lifelogs.com> writes:
> On Mon, 23 Dec 2013 08:42:51 -0500 Stefan Monnier <monnier <at>
IRO.UMontreal.CA> wrote: 
> OK, I'll move on.  If the new UI proves compelling, users will tell us
> they want it supported in their favorite modes :)
> 
> >> As with company-mode, this can be a frontend choice (affecting
> >> `display-completion-list', and see below for the suggested
> >> `display-list' function).  How about a defcustom
> >> `completion-ui-frontend' with some choices?
> 
> SM> The general idea of completion-ui was great, yes.
> 
> >> For the user, a new defcustom `completion-ui-preferences'?  And for the
> >> caller, some kind of plist/alist passed to `completion-at-point' that
> >> merges cleanly with `completion-ui-preferences'?
> 
> SM> Could be.  Tho as mentioned before, I'm not sure that there's a need for
> SM> caller-provided preferences.
> 
> OK, I'll work on this off-list to keep the noise down, I've generated
> enough emacs-devel traffic so far!
> 
> Dmitry, Stephen, please let me know directly if you want to participate
> in the design of `completion-ui-*' as described here.  I honestly can't
> keep track of all the completion packages available, but I'm
> xposting/CC-ing the Helm developers, and Toby Cubitt (because of
> http://www.emacswiki.org/emacs/CompletionUI the prefix "completion-ui"
> may need to be changed, but it seems to have similar ideas to the
> frontends we're describing here).

I'm wondering how much of what you're planning has already been coded
long ago in completion-UI, which already:

- has completely modularised completion user-interfaces, which can be
  used in any combination the user likes (within reason)

- comes with all the UIs people usually ask for: in-buffer text
  completion, tooltips (both real tooltips, and the "pop-up tip"
  overlay-based tooltips), drop-down menus, pop-up mini-frames, list of
  completions in echo area, hotkeys to select completions...

- lets you add a new completion UI with a single call to the
  `completion-ui-register-interface' macro

- has completely modularised completion sources (similar to company-mode,
  anything.el, auto-complete-mode, etc.)

- lets you register a new completion source with a single call to the
  `completion-ui-register-source' macro

- defines an `auto-completion-mode' (essentially, a more advanced version
  of auto-complete.el) that works with any completion source

- lets users customize their completion UIs at as fine-grained a level as
  they want: globally, separately for different completion sources, or
  (of course) per-major-mode etc. as usual

I haven't had time to read the whole thread, but from what I've skimmed
through so far it looks like much what you're discussing is already implemented.

Completion-UI isn't another company-mode or anything or auto-complete-mode.
It was always intended to be "plumbing": a generic completion user-interface
toolkit that other packages could use, to unify the interface for selecting
completions in Emacs. It would, I think, be rather easy to modify
company-mode, auto-complete-mode, anything.el, etc. to use completion-UI
instead of their built-in UI code.

Many years ago (pre-Stefan's tenure), I had an email discussion with
Richard Stallman about getting completion-UI into a state to be added to
Emacs, precisely to be a richer generic completion user interface for
Emacs. Richard pronounced himself "happy" at the end of the discussion, but
then he moved on to other things and I got busy with academic work, and it
never happened.

If you're interested in exploring how much is already there to be reused
or adapted, I'm happy to help (to the extent that my "real" work allows).

All the best,
Toby

PS: If it ever came to that point, I already have copyright papers on
file with the FSF, so donating most of the completion-UI code would be
straightforward. I use (a slightly modified version of) Tomohiro
Matsuyama's popup.el library for the overlay-based tooltips. I don't know
if Tomohiro has papers on file. But since the UIs are completely modular,
in the worst case that UI could be left out initially, and reimplemented
later.
-- 
Dr T. S. Cubitt
Royal Society University Research Fellow
and Fellow of Churchill College, Cambridge
Centre for Quantum Information
DAMTP, University of Cambridge

email: toby@dr-qubit.org
web:   www.dr-qubit.org

Re: Emacs completion matches selection UI

[Leo Liu]

On 2013-12-31 04:47 +0800, Toby Cubitt wrote:
> Many years ago (pre-Stefan's tenure), I had an email discussion with
> Richard Stallman about getting completion-UI into a state to be added to
> Emacs, precisely to be a richer generic completion user interface for
> Emacs. Richard pronounced himself "happy" at the end of the discussion, but
> then he moved on to other things and I got busy with academic work, and it
> never happened.

+1 for the work on completion-UI and undo-tree.

If this discussion has already happened with rms and he ok'd it, maybe
we should look for a way to integrate completion-UI into emacs now.

Leo

Re: Emacs completion matches selection UI

[joakim]

Leo Liu <sdl.web@gmail.com> writes:

> On 2013-12-31 04:47 +0800, Toby Cubitt wrote:
>> Many years ago (pre-Stefan's tenure), I had an email discussion with
>> Richard Stallman about getting completion-UI into a state to be added to
>> Emacs, precisely to be a richer generic completion user interface for
>> Emacs. Richard pronounced himself "happy" at the end of the discussion, but
>> then he moved on to other things and I got busy with academic work, and it
>> never happened.
>
> +1 for the work on completion-UI and undo-tree.
>
> If this discussion has already happened with rms and he ok'd it, maybe
> we should look for a way to integrate completion-UI into emacs now.

+1 to not reimplementing everything all the time.
Ive used completion ui, by way of predictive mode, and it is nice imho.

> Leo
>

-- 
Joakim Verona

Re: Emacs completion matches selection UI

[Dmitry Gutov]

Hello Toby,

When I looked into the different completion interfaces available for
Emacs, Completion-UI came out a lowest denominator in terms of features
that can be supported by completion backends.

Both Company and Auto-Complete support displaying candidate "summaries"
(usually calltips and/or first line of the candidate's docstring),
fuller documentation (in a new buffer, or in a popup, respectively), and
Company also allows backends to return the candidate's location (as a
position in a buffer or a line in a file), which has a respective "show
definition" command. These are quite useful for programming.

Completion-UI, from what I can tell, only considers candidates as
strings to be inserted, without any introspection facilities. Even the
Elisp source looks like this:

(completion-ui-register-source
 all-completions
 :completion-args 1
 :other-args (obarray)
 :name elisp
 :word-thing symbol)

...which is miles behind both `company-elisp' and
`lisp-completion-at-point'.

And because your source interface doesn't provide much, Completion-UI
user-interfaces don't handle the extra options either. So, as things
currently stand, if one was to write translation functions from Company
backends and Auto-Complete sources, a whole slice of their features
would be lost (see `company-backends' docstring for some details).

Conversely, Company also provides swappable front-ends (see the
docstring of `company-frontends'), so from where I stand, it should be
easier to adapt any widgets you have implemented that we don't have as
new Company front-ends.

Toby Cubitt <toby-predictive@dr-qubit.org> writes:
> - has completely modularised completion user-interfaces, which can be
>   used in any combination the user likes (within reason)

You can have some of that in Company by setting `company-frontends' to a
buffer-local value. Probably. I've never tried that, though, and I'm not
sure if I'll ever want to, personally.

> - comes with all the UIs people usually ask for: in-buffer text
>   completion, tooltips (both real tooltips, and the "pop-up tip"
>   overlay-based tooltips), drop-down menus, pop-up mini-frames, list of
>   completions in echo area, hotkeys to select completions...

Company doesn't have mini-frames and, I guess, drop-down menus. Is the
latter a graphical menu that only allows interaction with mouse and
arrow keys?

> - lets you add a new completion UI with a single call to the
>   `completion-ui-register-interface' macro

Company allows you to do that with a handy macro called `defun'.

> - lets you register a new completion source with a single call to the
>   `completion-ui-register-source' macro

Ditto.

> Completion-UI isn't another company-mode or anything or auto-complete-mode.
> It was always intended to be "plumbing": a generic completion user-interface
> toolkit that other packages could use, to unify the interface for selecting
> completions in Emacs. It would, I think, be rather easy to modify
> company-mode, auto-complete-mode, anything.el, etc. to use completion-UI
> instead of their built-in UI code.

See above. It would be a lossy conversion.

Also, I think `company-backends' provides a nicer API than
`completion-ui-register-source'.

> I use (a slightly modified version of) Tomohiro
> Matsuyama's popup.el library for the overlay-based tooltips. I don't know
> if Tomohiro has papers on file.

It would be nice to be able to use it, but from what I see Matsuyama is
not the sole significant contributor. I opened a related issue
(https://github.com/auto-complete/popup-el/issues/50), but there's been
no response so far.

By the way, why are you bundling a modified version of popup.el instead
of contributing the changes upstream?

Re: Emacs completion matches selection UI

[Toby Cubitt]

Hi Dmitry,

I really wasn't trying to promote completion-UI over Company. Indeed, I
completely agree (as I tried to say in my previous post) that
company-mode is much more sophisticated than completion-UI, because it
aims to do something much more ambitious.

Completion-UI was only ever intended as "plumbing": an elisp package that
provides functions and hooks to let you display and select completion
candidates in various UIs. Nothing more.

Company does *much* more than this. It's practically a way of life :)

I considered switching to Company for the predictive-mode UI, since I'm
really not that interested in coding and maintaining user-interface
code. (It's both tedious, and difficult to do well.) But Company was too
all-encompassing: the UI elements weren't cleanly separated out from the
rest of the mode, and I couldn't see how to make use of them without
buying into the whole company-mode shebang. Maybe that's changed now.
Though I still can't seem to see a generic UI package separate from the
rest of company-mode.

I've got nothing at all against company-mode. Indeed, if I remember
right, someone (you?) has added predictive-mode as a company-mode source,
which is great! But I didn't want to force people to use company-mode in
order to use my predictive package. I just wanted to provide simple UIs
for displaying and selecting completions.

I deliberately restricted completion-UI to do just that one simple task:
provide UIs for displaying and selecting completion candidates in a more
"modern" way than the *Completions* buffer. This seems to align with
what's being discussed here. Whereas I can't see Company being added to
Emacs as the generic Emacs completion UI.

I'd be very happy to see the UI parts of Company stripped out and made
into a simple, generic completion UI package and added to Emacs. Then I
could switch to using that, and ditch completion-UI and the concomitant
development and maintenance burden!

But it's inevitable each of the various completion frameworks will each
have a few very useful features that don't exist elsewhere. A better
approach is would be to, separate out and merge any useful parts of the
various packages that make sense as a generic Emacs completion UI
framework. But even if none of the code gets used, I still think you'll
end up with something much closer to completion-UI than Company.

[Further detailed comments inline, below]

Cheers,
Toby



On Tue, Dec 31, 2013 at 06:30:47PM +0400, Dmitry Gutov wrote:
> When I looked into the different completion interfaces available for
> Emacs, Completion-UI came out a lowest denominator in terms of features
> that can be supported by completion backends.

Good! Because it's precisely intended to be a "lowest common
denominator".

> Both Company and Auto-Complete support displaying candidate "summaries"
> (usually calltips and/or first line of the candidate's docstring),
> fuller documentation (in a new buffer, or in a popup, respectively), and
> Company also allows backends to return the candidate's location (as a
> position in a buffer or a line in a file), which has a respective "show
> definition" command. These are quite useful for programming.

Yes, I can see why that could be a useful feature for programming
modes. It doesn't sound particularly difficult to implement.

> Completion-UI, from what I can tell, only considers candidates as
> strings to be inserted, without any introspection facilities.

Yes, that's accurate.

Since completion-UI was originally written as the UI for predictive-mode
(which is most useful in text modes), it's strong on plain text features,
and weaker on programming-related features. For example, last time I
looked completion-UI's auto-completion-mode was much more sophisticated
than that in any other package (which lacked many of the features that
are crucial to implementing predictive completion).

That's why I think merging the best bits of the generic UI stuff from all
the various frameworks would be the best way to go.

> And because your source interface doesn't provide much, Completion-UI
> user-interfaces don't handle the extra options either. So, as things
> currently stand, if one was to write translation functions from Company
> backends and Auto-Complete sources, a whole slice of their features
> would be lost (see `company-backends' docstring for some details).
> 
> Conversely, Company also provides swappable front-ends (see the
> docstring of `company-frontends'), so from where I stand, it should be
> easier to adapt any widgets you have implemented that we don't have as
> new Company front-ends.

Yes, but then you have to buy into the whole company-mode world. Which is
fine if that's what you want. Not so useful for a generic Emacs
completion UI.

> Toby Cubitt <toby-predictive@dr-qubit.org> writes:
> > - has completely modularised completion user-interfaces, which can be
> >   used in any combination the user likes (within reason)
> 
> You can have some of that in Company by setting `company-frontends' to a
> buffer-local value. Probably. I've never tried that, though, and I'm not
> sure if I'll ever want to, personally.

Really? Some people like auto-displayed tooltips, some people hate
them. Some people like displaying completions in the echo area, some find
it a distraction. Makes sense to me (given that this is Emacs we're
talking about) to let people customize it the way *they* want via
customization options, with sensible defaults.

> > - comes with all the UIs people usually ask for: in-buffer text
> >   completion, tooltips (both real tooltips, and the "pop-up tip"
> >   overlay-based tooltips), drop-down menus, pop-up mini-frames, list of
> >   completions in echo area, hotkeys to select completions...
> 
> Company doesn't have mini-frames and, I guess, drop-down menus. Is the
> latter a graphical menu that only allows interaction with mouse and
> arrow keys?

Yes. And a "completion browser" that organises all the completions into a
hierarchical set of menus. (As with most things, this generic version can
be overridden by particular completion sources to provide a specific
version that's more useful. I use that e.g. in the predictive-mode LaTeX
support when completing LaTeX commands, environments, labels, etc.)

> > - lets you add a new completion UI with a single call to the
> >   `completion-ui-register-interface' macro
> 
> Company allows you to do that with a handy macro called `defun'.

Needlessly snarky. You still need some way to tell Company about the new
UI, so that it knows to invoke it.

> > - lets you register a new completion source with a single call to the
> >   `completion-ui-register-source' macro
> 
> Ditto.

Sure. And you'd hope all the other completion frameworks do too! (They
do.)

> > Completion-UI isn't another company-mode or anything or auto-complete-mode.
> > It was always intended to be "plumbing": a generic completion user-interface
> > toolkit that other packages could use, to unify the interface for selecting
> > completions in Emacs. It would, I think, be rather easy to modify
> > company-mode, auto-complete-mode, anything.el, etc. to use completion-UI
> > instead of their built-in UI code.
> 
> See above. It would be a lossy conversion.

It's not a conversion at all. The sophisticated parts of company aren't
about displaying menus, or popups, or tooltips, etc. I'm just saying that
if you had a generic Emacs completion UI, you could (and should) rebase
Company's UI on that. This is a small and boring part of
Company. Wouldn't you be happy to see that included in Emacs, so everyone
could benefit instead of reimplementing the wheel?

That's why I went to the effort many years ago now of separating the UI
code out of predictive-mode into something generic. Unfortunately,
everyone then went off and invented new wheels (the UI code in Company,
anything, auto-compelte, etc. etc.).

> Also, I think `company-backends' provides a nicer API than
> `completion-ui-register-source'.

Could well be. So help design a good API for a generic Emacs completion UI.

> > I use (a slightly modified version of) Tomohiro
> > Matsuyama's popup.el library for the overlay-based tooltips. I don't know
> > if Tomohiro has papers on file.
> 
> It would be nice to be able to use it, but from what I see Matsuyama is
> not the sole significant contributor. I opened a related issue
> (https://github.com/auto-complete/popup-el/issues/50), but there's been
> no response so far.

Shame.

> By the way, why are you bundling a modified version of popup.el instead
> of contributing the changes upstream?

I simply haven't found time. I haven't even had time to roll a new
predictive release for a while now (though the git version still gets
updates). Plus my popup.el change is a trivial one-liner.


-- 
Dr T. S. Cubitt
Royal Society University Research Fellow
and Fellow of Churchill College, Cambridge
Centre for Quantum Information
DAMTP, University of Cambridge

email: tsc25@cantab.net
web:   www.dr-qubit.org

Re: Emacs completion matches selection UI

[joakim]

Toby Cubitt <tsc25@cantab.net> writes:

> Hi Dmitry,
>
> I really wasn't trying to promote completion-UI over Company. Indeed, I
> completely agree (as I tried to say in my previous post) that
> company-mode is much more sophisticated than completion-UI, because it
> aims to do something much more ambitious.
>
> Completion-UI was only ever intended as "plumbing": an elisp package that
> provides functions and hooks to let you display and select completion
> candidates in various UIs. Nothing more.
>
> Company does *much* more than this. It's practically a way of life :)
>
> I considered switching to Company for the predictive-mode UI, since I'm
> really not that interested in coding and maintaining user-interface
> code. (It's both tedious, and difficult to do well.) But Company was too
> all-encompassing: the UI elements weren't cleanly separated out from the
> rest of the mode, and I couldn't see how to make use of them without
> buying into the whole company-mode shebang. Maybe that's changed now.
> Though I still can't seem to see a generic UI package separate from the
> rest of company-mode.
>
> I've got nothing at all against company-mode. Indeed, if I remember
> right, someone (you?) has added predictive-mode as a company-mode source,
> which is great! But I didn't want to force people to use company-mode in
> order to use my predictive package. I just wanted to provide simple UIs
> for displaying and selecting completions.

I did this at some point. Perhaps I can dig out the code if its deemed
interesting. I have vague recollection of mailing you the code.

atm  I  dont use it because the company popup didnt work well for me,
and then I got sidetracked into providing a gtk popup, which didnt pan out.



> I deliberately restricted completion-UI to do just that one simple task:
> provide UIs for displaying and selecting completion candidates in a more
> "modern" way than the *Completions* buffer. This seems to align with
> what's being discussed here. Whereas I can't see Company being added to
> Emacs as the generic Emacs completion UI.
>
> I'd be very happy to see the UI parts of Company stripped out and made
> into a simple, generic completion UI package and added to Emacs. Then I
> could switch to using that, and ditch completion-UI and the concomitant
> development and maintenance burden!
>
> But it's inevitable each of the various completion frameworks will each
> have a few very useful features that don't exist elsewhere. A better
> approach is would be to, separate out and merge any useful parts of the
> various packages that make sense as a generic Emacs completion UI
> framework. But even if none of the code gets used, I still think you'll
> end up with something much closer to completion-UI than Company.
>
> [Further detailed comments inline, below]
>
> Cheers,
> Toby
>
>
>
> On Tue, Dec 31, 2013 at 06:30:47PM +0400, Dmitry Gutov wrote:
>> When I looked into the different completion interfaces available for
>> Emacs, Completion-UI came out a lowest denominator in terms of features
>> that can be supported by completion backends.
>
> Good! Because it's precisely intended to be a "lowest common
> denominator".
>
>> Both Company and Auto-Complete support displaying candidate "summaries"
>> (usually calltips and/or first line of the candidate's docstring),
>> fuller documentation (in a new buffer, or in a popup, respectively), and
>> Company also allows backends to return the candidate's location (as a
>> position in a buffer or a line in a file), which has a respective "show
>> definition" command. These are quite useful for programming.
>
> Yes, I can see why that could be a useful feature for programming
> modes. It doesn't sound particularly difficult to implement.
>
>> Completion-UI, from what I can tell, only considers candidates as
>> strings to be inserted, without any introspection facilities.
>
> Yes, that's accurate.
>
> Since completion-UI was originally written as the UI for predictive-mode
> (which is most useful in text modes), it's strong on plain text features,
> and weaker on programming-related features. For example, last time I
> looked completion-UI's auto-completion-mode was much more sophisticated
> than that in any other package (which lacked many of the features that
> are crucial to implementing predictive completion).
>
> That's why I think merging the best bits of the generic UI stuff from all
> the various frameworks would be the best way to go.
>
>> And because your source interface doesn't provide much, Completion-UI
>> user-interfaces don't handle the extra options either. So, as things
>> currently stand, if one was to write translation functions from Company
>> backends and Auto-Complete sources, a whole slice of their features
>> would be lost (see `company-backends' docstring for some details).
>> 
>> Conversely, Company also provides swappable front-ends (see the
>> docstring of `company-frontends'), so from where I stand, it should be
>> easier to adapt any widgets you have implemented that we don't have as
>> new Company front-ends.
>
> Yes, but then you have to buy into the whole company-mode world. Which is
> fine if that's what you want. Not so useful for a generic Emacs
> completion UI.
>
>> Toby Cubitt <toby-predictive@dr-qubit.org> writes:
>> > - has completely modularised completion user-interfaces, which can be
>> >   used in any combination the user likes (within reason)
>> 
>> You can have some of that in Company by setting `company-frontends' to a
>> buffer-local value. Probably. I've never tried that, though, and I'm not
>> sure if I'll ever want to, personally.
>
> Really? Some people like auto-displayed tooltips, some people hate
> them. Some people like displaying completions in the echo area, some find
> it a distraction. Makes sense to me (given that this is Emacs we're
> talking about) to let people customize it the way *they* want via
> customization options, with sensible defaults.
>
>> > - comes with all the UIs people usually ask for: in-buffer text
>> >   completion, tooltips (both real tooltips, and the "pop-up tip"
>> >   overlay-based tooltips), drop-down menus, pop-up mini-frames, list of
>> >   completions in echo area, hotkeys to select completions...
>> 
>> Company doesn't have mini-frames and, I guess, drop-down menus. Is the
>> latter a graphical menu that only allows interaction with mouse and
>> arrow keys?
>
> Yes. And a "completion browser" that organises all the completions into a
> hierarchical set of menus. (As with most things, this generic version can
> be overridden by particular completion sources to provide a specific
> version that's more useful. I use that e.g. in the predictive-mode LaTeX
> support when completing LaTeX commands, environments, labels, etc.)
>
>> > - lets you add a new completion UI with a single call to the
>> >   `completion-ui-register-interface' macro
>> 
>> Company allows you to do that with a handy macro called `defun'.
>
> Needlessly snarky. You still need some way to tell Company about the new
> UI, so that it knows to invoke it.
>
>> > - lets you register a new completion source with a single call to the
>> >   `completion-ui-register-source' macro
>> 
>> Ditto.
>
> Sure. And you'd hope all the other completion frameworks do too! (They
> do.)
>
>> > Completion-UI isn't another company-mode or anything or auto-complete-mode.
>> > It was always intended to be "plumbing": a generic completion user-interface
>> > toolkit that other packages could use, to unify the interface for selecting
>> > completions in Emacs. It would, I think, be rather easy to modify
>> > company-mode, auto-complete-mode, anything.el, etc. to use completion-UI
>> > instead of their built-in UI code.
>> 
>> See above. It would be a lossy conversion.
>
> It's not a conversion at all. The sophisticated parts of company aren't
> about displaying menus, or popups, or tooltips, etc. I'm just saying that
> if you had a generic Emacs completion UI, you could (and should) rebase
> Company's UI on that. This is a small and boring part of
> Company. Wouldn't you be happy to see that included in Emacs, so everyone
> could benefit instead of reimplementing the wheel?
>
> That's why I went to the effort many years ago now of separating the UI
> code out of predictive-mode into something generic. Unfortunately,
> everyone then went off and invented new wheels (the UI code in Company,
> anything, auto-compelte, etc. etc.).
>
>> Also, I think `company-backends' provides a nicer API than
>> `completion-ui-register-source'.
>
> Could well be. So help design a good API for a generic Emacs completion UI.
>
>> > I use (a slightly modified version of) Tomohiro
>> > Matsuyama's popup.el library for the overlay-based tooltips. I don't know
>> > if Tomohiro has papers on file.
>> 
>> It would be nice to be able to use it, but from what I see Matsuyama is
>> not the sole significant contributor. I opened a related issue
>> (https://github.com/auto-complete/popup-el/issues/50), but there's been
>> no response so far.
>
> Shame.
>
>> By the way, why are you bundling a modified version of popup.el instead
>> of contributing the changes upstream?
>
> I simply haven't found time. I haven't even had time to roll a new
> predictive release for a while now (though the git version still gets
> updates). Plus my popup.el change is a trivial one-liner.

-- 
Joakim Verona

Re: Emacs completion matches selection UI

[Stefan Monnier]

I'd love to see company-mode and completion-ui consolidated into Emacs's
core code.  The "completion sources" would be provided via
completion-at-point-functions, so we already have a place for
that part.  We don't yet have an API for the "completion UI
frontend", OTOH, so if we can agree on a good API for that part, we
could restructure the minibuffer.el code around that and then add
more UIs.


        Stefan

Re: Emacs completion matches selection UI

[Dmitry Gutov]

On 31.12.2013 21:45, joakim@verona.se wrote:
> I did this at some point. Perhaps I can dig out the code if its deemed
> interesting. I have vague recollection of mailing you the code.

If you have the copyright papers signed, please feel free to submit it 
for inclusion in Company. Even if it's not working too well, I can try 
to fix it.

But Predictive would have to be available as an ELPA package.

> atm  I  dont use it because the company popup didnt work well for me,

It had some problems a while ago, but those were fixed, and I don't know 
of any serious complaints at the moment. popup.el, on the other hand, 
has quite a few unresolved issues filed.

It's, of course, more popular, which can justify the larger amount of 
known issues. But still.

Re: Emacs completion matches selection UI

[Dmitry Gutov]

On 31.12.2013 19:52, Toby Cubitt wrote:
> I really wasn't trying to promote completion-UI over Company. Indeed, I
> completely agree (as I tried to say in my previous post) that
> company-mode is much more sophisticated than completion-UI, because it
> aims to do something much more ambitious.
>
> Completion-UI was only ever intended as "plumbing": an elisp package that
> provides functions and hooks to let you display and select completion
> candidates in various UIs. Nothing more.

My impression was that you proposed it as a wrapper around Company 
and/or related packages. In that case, lowering the level of 
sophistication means lowering the amount of features, which is bad.

But if you'd like us to just use the widgets from Completion-UI (and 
maybe some code related to customizing them?), that would be okay, 
although I'd have to look into fitting any of them inside the Company's 
interaction with the command loop, or rewrite it somehow.

> Company does *much* more than this. It's practically a way of life :)

It's not hard to describe Company in the same terms: it's a framework 
designed to fetch completion candidates from the configured backends and 
allow selecting between them in the configured frontends. The "show 
documentation" and "show definition" actions are small bonus features, 
as far as complexity of implementation goes.

> I considered switching to Company for the predictive-mode UI, since I'm
> really not that interested in coding and maintaining user-interface
> code. (It's both tedious, and difficult to do well.) But Company was too
> all-encompassing: the UI elements weren't cleanly separated out from the
> rest of the mode, and I couldn't see how to make use of them without
> buying into the whole company-mode shebang.

Company front-ends are not independent widgets, indeed. They don't 
encapsulate data, instead using the same buffer-local vars Company 
operates with.

And the popup code is a relatively small part of the package, so 
"switching to Company" really means writing a backend for it, in this 
case. I wonder which part of the backend API you didn't like.

 > Maybe that's changed now.
> Though I still can't seem to see a generic UI package separate from the
> rest of company-mode.

No one's requested it, so far. There's the popular popup.el, and there's 
dropdown-list.el of unclear providence on EmacsWiki which Yasnippet uses 
optionally.

I've been thinking about splitting the Company popup code properly, but 
being able to reuse (and contribute to) the popular option which is 
popup.el would be better.

> I've got nothing at all against company-mode. Indeed, if I remember
> right, someone (you?) has added predictive-mode as a company-mode source,

Nope.

> which is great! But I didn't want to force people to use company-mode in
> order to use my predictive package.

The usual way to go about it is to have a core package (maybe using 
completion-at-point-functions) and a couple of integrations with the 
popular completion packages.

> I just wanted to provide simple UIs
> for displaying and selecting completions.

I'd wager Predictive would have a better adoption if it wasn't coupled 
to a yet-another completion package. I, for one, don't even consider 
using it for this reason. Although maybe I'm just not the target audience.

> I'd be very happy to see the UI parts of Company stripped out and made
> into a simple, generic completion UI package and added to Emacs.

Would you like to propose an API for it? Same as popup.el?

> Since completion-UI was originally written as the UI for predictive-mode
> (which is most useful in text modes), it's strong on plain text features,
> and weaker on programming-related features. For example, last time I
> looked completion-UI's auto-completion-mode was much more sophisticated
> than that in any other package (which lacked many of the features that
> are crucial to implementing predictive completion).
>
> That's why I think merging the best bits of the generic UI stuff from all
> the various frameworks would be the best way to go.

It sounds good as an abstract idea, but I'd like to see a more specific 
proposal.

>> Toby Cubitt <toby-predictive@dr-qubit.org> writes:
>>> - has completely modularised completion user-interfaces, which can be
>>>    used in any combination the user likes (within reason)
>>
>> You can have some of that in Company by setting `company-frontends' to a
>> buffer-local value. Probably. I've never tried that, though, and I'm not
>> sure if I'll ever want to, personally.
>
> Really? Some people like auto-displayed tooltips, some people hate
> them. Some people like displaying completions in the echo area, some find
> it a distraction. Makes sense to me (given that this is Emacs we're
> talking about) to let people customize it the way *they* want via
> customization options, with sensible defaults.

Sorry, I took it to mean that in Completion-UI you can configure 
user-interfaces per completion source, which is probably not the case.

The user can of course configure the used front-ends in Company, by 
customizing the global `company-frontends' value (as opposed to 
buffer-local ones I mentioned above).

> Yes. And a "completion browser" that organises all the completions into a
> hierarchical set of menus. (As with most things, this generic version can
> be overridden by particular completion sources to provide a specific
> version that's more useful. I use that e.g. in the predictive-mode LaTeX
> support when completing LaTeX commands, environments, labels, etc.)

The browser sounds useful, I'll have to look into it.

A mouse-only menu won't fit Company because it won't work for idle 
completion, as I mentioned elsewhere in this thread, and we generally 
want to provide a consistent keyboard interface. There'll also be issues 
with implementation related to that.

>>> - lets you add a new completion UI with a single call to the
>>>    `completion-ui-register-interface' macro
>>
>> Company allows you to do that with a handy macro called `defun'.
>
> Needlessly snarky.

Sorry. I was aiming for cute rather than aggressive.

 > You still need some way to tell Company about the new
> UI, so that it knows to invoke it.

Actually, no. After defining the backend function with `defun', the user 
will need to add it to `company-backends', but that's not much different 
from them having to customize `auto-completion-source' in Completion-UI.

> It's not a conversion at all. The sophisticated parts of company aren't
> about displaying menus, or popups, or tooltips, etc. I'm just saying that
> if you had a generic Emacs completion UI, you could (and should) rebase
> Company's UI on that. This is a small and boring part of
> Company. Wouldn't you be happy to see that included in Emacs, so everyone
> could benefit instead of reimplementing the wheel?

Of course I would. But a) you don't own the popup code, which we would 
like to use, b) it could be hard to augment Completion-UI widgets with 
new commands, such as, for example, which using Company you can 
currently press `f1' when the popup is active to pop up the 
documentation buffer, or `C-w' to see the function source. How hard 
would it be?

>
> That's why I went to the effort many years ago now of separating the UI
> code out of predictive-mode into something generic. Unfortunately,
> everyone then went off and invented new wheels (the UI code in Company,
> anything, auto-compelte, etc. etc.).
>
>> Also, I think `company-backends' provides a nicer API than
>> `completion-ui-register-source'.
>
> Could well be. So help design a good API for a generic Emacs completion UI.

Yes, sure. Would someone maybe submit an initial proposal?

Re: Emacs completion matches selection UI

[Toby Cubitt]

Hi Dmitry,

On Fri, Jan 03, 2014 at 12:20:38AM +0400, Dmitry Gutov wrote:
> On 31.12.2013 19:52, Toby Cubitt wrote:
> > I really wasn't trying to promote completion-UI over Company. Indeed, I
> > completely agree (as I tried to say in my previous post) that
> > company-mode is much more sophisticated than completion-UI, because it
> > aims to do something much more ambitious.
> >
> > Completion-UI was only ever intended as "plumbing": an elisp package that
> > provides functions and hooks to let you display and select completion
> > candidates in various UIs. Nothing more.
> 
> My impression was that you proposed it as a wrapper around Company 
> and/or related packages.

No, definitely not a wrapper. I was proposing using it as an Elisp
utility library. If anything, it's Company et al. that would build on top
of and wrap around a generic Emacs completion UI, rather than the other
way around.

> In that case, lowering the level of sophistication means lowering the
> amount of features, which is bad.

This thread was about including generic and more "modern" completion
user-interface(s) in Emacs. If this happened, I hope that Company and
other completion frameworks would make use of them, instead of each
rolling their own. Essentially, it would just move the generic parts of
the frameworks into Emacs. All the stuff on top of that that makes
Company more sophisticated would still need to be implemented in Company.

Providing a generic completion user-interface seems fairly close to what
I was trying to do when I separated out completion-UI from predictive
into a separate package (about 10 years ago now).

> But if you'd like us to just use the widgets from Completion-UI (and 
> maybe some code related to customizing them?), that would be okay, 

Yup, that's basically all I meant.

I imagine implementing Company on top of a generic Emacs completion UI
library would save you some maintenance effort. And a generic Emacs
completion UI would benefit from patches and improvements from all the
actively-developed Emacs completion packages.

Completion-UI *is* just a library providing completion user-interfaces,
plus a generic and somewhat sophisticated auto-completion-mode. It was
never intended to be a "completion framework", just a library of
integrated completion UI widgets and code. From where I stand, this seems
to be closer to what people were asking for in this thread.

I'm not pushing hard for Completion-UI inclusion. Just pointing out that
there's a lot of existing code out there, and it would make sense to
reuse it instead of reinventing the wheel for the umpteenth time. For
obvious reasons I know Completion-UI best, whereas I can't sensibly
comment much on the others. So I posted about what I know. I really don't
mind if people take that code from Completion-UI, or from Company,
auto-complete, anything, or any of the myriad others I've missed.

I'd hope that if and when Emacs included a generic completion UI, I'd be
able to declare Completion-UI obsolete and switch to using the standard
Emacs implementation in Predictive.

> although I'd have to look into fitting any of them inside the Company's 
> interaction with the command loop,

...which BTW is why I suspect the full Company framework isn't suitable
as a generic Emacs completion UI. 

Completion-UI doesn't need to touch the command loop. You pass a
completion function to `complete-in-buffer', and it displays the
completion candidates and lets the user select one, via the UI they've
configured through customize.

The only purpose of "registering" a completion function as a named
completion source, is to let users customize the UI separately for that
source (if they want to), and make it available as an choice in some
customization options.

The generic Completion-UI `auto-completion-mode' minor-mode is slightly
more invasive. It's not entirely clear to me at this point whether a
generic Emacs completion UI should include an auto-completion minor-mode
or not. I'm sure others will have opinions on this.

> > Company does *much* more than this. It's practically a way of life :)
> 
> It's not hard to describe Company in the same terms: it's a framework 
> designed to fetch completion candidates from the configured backends and 
> allow selecting between them in the configured frontends.

Sure, but as far as I understand (I could well be wrong - I've never used
it in anger) you use Company by enabling company-mode in your buffer as a
user, and this minor-mode changes how you interact with Emacs. Whereas
you use Completion-UI by calling its functions from Elisp when coding a
new Emacs package. It's really aimed at a "lower" level.

> The "show documentation" and "show definition" actions are small bonus
> features, as far as complexity of implementation goes.

Indeed. I only commented on them because you brought them up.

> > I considered switching to Company for the predictive-mode UI, since I'm
> > really not that interested in coding and maintaining user-interface
> > code. (It's both tedious, and difficult to do well.) But Company was too
> > all-encompassing: the UI elements weren't cleanly separated out from the
> > rest of the mode, and I couldn't see how to make use of them without
> > buying into the whole company-mode shebang.
> 
> Company front-ends are not independent widgets, indeed. They don't 
> encapsulate data, instead using the same buffer-local vars Company 
> operates with.
> 
> And the popup code is a relatively small part of the package, so 
> "switching to Company" really means writing a backend for it, in this 
> case. I wonder which part of the backend API you didn't like.
> 
>  > Maybe that's changed now.
> > Though I still can't seem to see a generic UI package separate from the
> > rest of company-mode.
> 
> No one's requested it, so far. There's the popular popup.el, and there's 
> dropdown-list.el of unclear providence on EmacsWiki which Yasnippet uses 
> optionally.
> 
> I've been thinking about splitting the Company popup code properly, but 
> being able to reuse (and contribute to) the popular option which is 
> popup.el would be better.

popup.el is great, but it's a way of simulating tooltips using overlays,
not a generic Emacs completion UI. (Though it would be nice if it was
part of it.)

> > I've got nothing at all against company-mode. Indeed, if I remember
> > right, someone (you?) has added predictive-mode as a company-mode source,
> 
> Nope.

Seems it was Joakim. My impression was Company backend definitions
belong in Company, so perhaps his code should be added there?

> > which is great! But I didn't want to force people to use company-mode in
> > order to use my predictive package.
> 
> The usual way to go about it is to have a core package (maybe using 
> completion-at-point-functions) and a couple of integrations with the 
> popular completion packages.

There is a core package: predictive. It's completely separate from
Completion-UI (even if the latter comes bundled with it in the
tarball). No doubt that's why Joakim could fairly easily write a
predictive backend for Company.

If Company backend definitions should live in the completion package
rather than Company, I could add it to Predictive instead.

> > I just wanted to provide simple UIs
> > for displaying and selecting completions.
> 
> I'd wager Predictive would have a better adoption if it wasn't coupled 
> to a yet-another completion package.

Well, I believe Completion-UI predates at least Company, anything, and
auto-complete. So "yet-another-completion-package" is misleading. Had
those packages existed when I coded Predictive, I doubt I'd have bothered
coding Completion-UI.

It's a shame Company et al. didn't build on and contribute to
Completion-UI, and instead chose to (each) reinvent the wheel. But that's
the usual messy history of code (which maybe we're finally going to do
something to tidy up?).

I'm really not bothered about pushing for widespread adoption. I write
the code because it's useful to me, I make it available in case it's
useful to others, and I respond to bug reports and feature requests since
dumping code on the web without maintaining it seems irresponsible. If
people want to use it as is, great. If people want to use it via Company,
great. It's not difficult to do the latter, as Joakim demonstrated. I
seem to recall someone integrated Predictive into anything, too.

> I, for one, don't even consider using it for this reason. Although
> maybe I'm just not the target audience.

I have no idea. Do you want fast predictive completion of plain text, or
text markup languages like LaTeX? If so, you're the target audience. If
not, then probably not. (Adding predictive code-completion features to
some IDE-style Emacs completion package like CEDET might be useful, but I
don't do enough hard-core coding to implement that myself.)

> > I'd be very happy to see the UI parts of Company stripped out and made
> > into a simple, generic completion UI package and added to Emacs.
> 
> Would you like to propose an API for it? Same as popup.el?

As a first attempt, I'd propose something like 

  (complete-in-buffer COMPLETION-SOURCE &optional PREFIX-FUNCTION)

where COMPLETION-SOURCE is a function to call to obtain completions, and
PREFIX-FUNCTION is an optional function to call to determine what to
complete.

Calling this would invoke whatever completion widgets the user had
enabled in customize (with sensible defaults - maybe the popup.el
interface or similar, or perhaps the default should be the usual
*Completions* buffer).

> > Since completion-UI was originally written as the UI for predictive-mode
> > (which is most useful in text modes), it's strong on plain text features,
> > and weaker on programming-related features. For example, last time I
> > looked completion-UI's auto-completion-mode was much more sophisticated
> > than that in any other package (which lacked many of the features that
> > are crucial to implementing predictive completion).
> >
> > That's why I think merging the best bits of the generic UI stuff from all
> > the various frameworks would be the best way to go.
> 
> It sounds good as an abstract idea, but I'd like to see a more specific 
> proposal.
> 
> >> Toby Cubitt <toby-predictive@dr-qubit.org> writes:
> >>> - has completely modularised completion user-interfaces, which can be
> >>>    used in any combination the user likes (within reason)
> >>
> >> You can have some of that in Company by setting `company-frontends' to a
> >> buffer-local value. Probably. I've never tried that, though, and I'm not
> >> sure if I'll ever want to, personally.
> >
> > Really? Some people like auto-displayed tooltips, some people hate
> > them. Some people like displaying completions in the echo area, some find
> > it a distraction. Makes sense to me (given that this is Emacs we're
> > talking about) to let people customize it the way *they* want via
> > customization options, with sensible defaults.
> 
> Sorry, I took it to mean that in Completion-UI you can configure 
> user-interfaces per completion source, which is probably not the case.

You can if you want to; you don't have to.

Customizing the UI per completion source very often makes more sense than
customizing it per buffer. If you're completing filenames, you probably
want the same interface in every buffer. Whereas you probably don't want
the same interface for both filename completion and for plain-text
predictive completion.

> The user can of course configure the used front-ends in Company, by 
> customizing the global `company-frontends' value (as opposed to 
> buffer-local ones I mentioned above).
> 
> > Yes. And a "completion browser" that organises all the completions into a
> > hierarchical set of menus. (As with most things, this generic version can
> > be overridden by particular completion sources to provide a specific
> > version that's more useful. I use that e.g. in the predictive-mode LaTeX
> > support when completing LaTeX commands, environments, labels, etc.)
> 
> The browser sounds useful, I'll have to look into it.

It seems to be popular, especially for completing from a large standard
set of keywords which you can't quite remember (as in LaTeX). At least, I
get complaints by email whenever I break it ;-)

> A mouse-only menu won't fit Company because it won't work for idle 
> completion, as I mentioned elsewhere in this thread, and we generally 
> want to provide a consistent keyboard interface. There'll also be issues 
> with implementation related to that.

When you register a new UI with Completion-UI, you tell it whether the UI
is suitable for idle completion. If it is, it becomes one of the
customization options you can select for the idle completion UI.

> >>> - lets you add a new completion UI with a single call to the
> >>>    `completion-ui-register-interface' macro
> >>
> >> Company allows you to do that with a handy macro called `defun'.
> >
> > Needlessly snarky.
> 
> Sorry. I was aiming for cute rather than aggressive.

OK, no worries!

>  > You still need some way to tell Company about the new
> > UI, so that it knows to invoke it.
> 
> Actually, no. After defining the backend function with `defun', the user 
> will need to add it to `company-backends', but that's not much different 
> from them having to customize `auto-completion-source' in Completion-UI.

Sounds like adding it to `company-backends' is more analogous registering
it with `completion-ui-register-source'. Customizing
`auto-completion-source' is something the user would do, not the Elisp
package coder.

In fact, it sounds like the two APIs are rather similar:

1. Define a completion function with defun (Completion-UI and Company)

2. Pass it to `completion-ui-register-source' (Completion-UI) or add it
   to `company-backends' (Company) to let them know about it.

At some point in the distant past, you used to just set a variable to
tell Completion-UI about a new completion function. It became a macro
call when I made Completion-UI more generic, and there needed to be a way
of optionally supplying additional information about what the completion
function does (e.g. if it does something other than standard prefix
completion), and of defining optional call-back functions, etc.

> > It's not a conversion at all. The sophisticated parts of company aren't
> > about displaying menus, or popups, or tooltips, etc. I'm just saying that
> > if you had a generic Emacs completion UI, you could (and should) rebase
> > Company's UI on that. This is a small and boring part of
> > Company. Wouldn't you be happy to see that included in Emacs, so everyone
> > could benefit instead of reimplementing the wheel?
> 
> Of course I would. But a) you don't own the popup code, which we would 
> like to use,

Indeed. Popup is one particularly nice UI. If it can't be donated, we'd
probably have to be reimplement it from scratch, which would be a
shame. Unfortunately, that's an issue however we decide to implement a
generic Emacs completion UI.

> b) it could be hard to augment Completion-UI widgets with new commands,
> such as, for example, which using Company you can currently press `f1'
> when the popup is active to pop up the documentation buffer, or `C-w'
> to see the function source. How hard would it be?

Trivial. In fact, it's already possible by overriding default functions
in the source definition. But it seems generically useful enough to be
worth creating a simpler API for.

> > That's why I went to the effort many years ago now of separating the
> > UI code out of predictive-mode into something generic. Unfortunately,
> > everyone then went off and invented new wheels (the UI code in
> > Company, anything, auto-compelte, etc. etc.).
> >
> >> Also, I think `company-backends' provides a nicer API than
> >> `completion-ui-register-source'.
> >
> > Could well be. So help design a good API for a generic Emacs
> > completion UI.
> 
> Yes, sure. Would someone maybe submit an initial proposal?

I second that. I'd like to see what people really want to see merged
before I'm likely to devote time to it. (Whether I *have* time to devote
to it is a separate question...)

Best,
Toby
-- 
Dr T. S. Cubitt
Royal Society University Research Fellow
and Fellow of Churchill College, Cambridge
Centre for Quantum Information
DAMTP, University of Cambridge

email: tsc25@cantab.net
web:   www.dr-qubit.org

Re: Emacs completion matches selection UI

[Stefan Monnier]

> Seems it was Joakim.  My impression was Company backend definitions
> belong in Company, so perhaps his code should be added there?

As long as company is "non-standard", that's probably true.  But the
completion-at-point-functions for Predictive should be in Predictive.
And that should be sufficient to make it available in Company as well
since Company can use completion-at-point-functions.


        Stefan

Re: Emacs completion matches selection UI

[Toby Cubitt]

On Thu, Jan 02, 2014 at 09:40:51PM -0500, Stefan Monnier wrote:
> > Seems it was Joakim.  My impression was Company backend definitions
> > belong in Company, so perhaps his code should be added there?
> 
> As long as company is "non-standard", that's probably true.  But the
> completion-at-point-functions for Predictive should be in Predictive.
> And that should be sufficient to make it available in Company as well
> since Company can use completion-at-point-functions.

Thanks Stefan, that's useful to know. I admit I haven't followed the
`completion-at-point' et al. developments closely. I'll look into adding
definitions next time I hack on Predictive.

(Though I don't think the `completion-at-point' mechanism is really rich
enough to let Predictive do its stuff properly; in particular updating
the frequency data after completing.)

Best,
Toby

-- 
Dr T. S. Cubitt
Royal Society University Research Fellow
and Fellow of Churchill College, Cambridge
Centre for Quantum Information
DAMTP, University of Cambridge

email: tsc25@cantab.net
web:   www.dr-qubit.org

Re: Emacs completion matches selection UI

[Dmitry Gutov]

On 03.01.2014 18:30, Toby Cubitt wrote:
> (Though I don't think the `completion-at-point' mechanism is really rich
> enough to let Predictive do its stuff properly; in particular updating
> the frequency data after completing.)

Seems that :exit-function can be used for this. See 
`completion-extra-properties'.

Re: Emacs completion matches selection UI

[Toby Cubitt]

On Thu, Jan 02, 2014 at 10:58:31PM +0000, Toby Cubitt wrote:
> > Would you like to propose an API for it? Same as popup.el?
> 
> As a first attempt, I'd propose something like 
> 
>   (complete-in-buffer COMPLETION-SOURCE &optional PREFIX-FUNCTION)
> 
> where COMPLETION-SOURCE is a function to call to obtain completions, and
> PREFIX-FUNCTION is an optional function to call to determine what to
> complete.
> 
> Calling this would invoke whatever completion widgets the user had
> enabled in customize (with sensible defaults - maybe the popup.el
> interface or similar, or perhaps the default should be the usual
> *Completions* buffer).

Or one could simply enhance `completion-in-region' to display completions
in the "more modern" UI, which the user can configure via customize.

Would that be usable in Company, or is the `completion-in-region' API
missing things you'd need? (It's currently missing things I'd need for
Predictive, but not too much. I think they wouldn't be too hard to add.)

Toby
-- 
Dr T. S. Cubitt
Royal Society University Research Fellow
and Fellow of Churchill College, Cambridge
Centre for Quantum Information
DAMTP, University of Cambridge

email: tsc25@cantab.net
web:   www.dr-qubit.org

Re: Emacs completion matches selection UI

[Toby Cubitt]

On Fri, Jan 03, 2014 at 08:23:06PM +0400, Dmitry Gutov wrote:
> On 03.01.2014 18:30, Toby Cubitt wrote:
> > (Though I don't think the `completion-at-point' mechanism is really rich
> > enough to let Predictive do its stuff properly; in particular updating
> > the frequency data after completing.)
> 
> Seems that :exit-function can be used for this. See 
> `completion-extra-properties'.

Thanks for the pointer. Unfortunately this isn't enough. In a typical
predictive-mode scenario, there are many possible completion candidates,
and Predictive needs to display them, and be told which of the candidates
was selected (or rejected).

Whereas, as I understand it, `completion-at-point' does "tab-completion",
i.e. expansion to the longest common prefix. :exit-function tells you
whether there was a unique completion, or if it was partially expanded
but not complete, or if it is complete can could be expanded further.

Predictive does allow standard "tab-completion", but it isn't so useful
for predictive completion. It isn't the main completion mechanism in
Predictive, nor is it the primary mechanism by which predictive trains
its frequency tables.

Toby
-- 
Dr T. S. Cubitt
Royal Society University Research Fellow
and Fellow of Churchill College, Cambridge
Centre for Quantum Information
DAMTP, University of Cambridge

email: tsc25@cantab.net
web:   www.dr-qubit.org

Re: Emacs completion matches selection UI

[Thierry Volpiatto]

Toby Cubitt <tsc25@cantab.net> writes:

> Or one could simply enhance `completion-in-region' to display completions
> in the "more modern" UI, which the user can configure via customize.

You have now `completion-in-region-function' (without "s") which is by
default bound to `completion--in-region'.
You can create your own function and bind
`completion-in-region-function' to it.
This will allow `completion-at-point', `completing-read-multiple' to use
your function.
It is what helm does, it is really nice feature.

-- 
Thierry
Get my Gnupg key:
gpg --keyserver pgp.mit.edu --recv-keys 59F29997 

Re: Emacs completion matches selection UI

[Ted Zlatanov]

On Thu, 02 Jan 2014 13:10:11 -0500 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 

SM> I'd love to see company-mode and completion-ui consolidated into Emacs's
SM> core code.  The "completion sources" would be provided via
SM> completion-at-point-functions, so we already have a place for
SM> that part.  We don't yet have an API for the "completion UI
SM> frontend", OTOH, so if we can agree on a good API for that part, we
SM> could restructure the minibuffer.el code around that and then add
SM> more UIs.

Based on the discussion so far, it seems that:

- there are good use cases for both completion-ui and company-mode, we
  can't claim either is a superset of the other.  The two are not
  fundamentally incompatible.

- I personally used company-mode heavily in the last few weeks and it
  works great, but I don't like that after a timeout, the up/down keys
  get taken over.  I think displaying the completion candidates could
  happen after a timeout, but selecting from them should happen only
  when the user explicitly requests it with M-TAB in a buffer or TAB in
  the minibuffer.

- it should be possible to implement a generic frontend for completion
  that both can use, and other tools like yasnippet can use IIUC.  The
  eventual goal is to have "something" standard in Emacs that all such
  packages (except perhaps helm, which is completely apart in its UI)
  would use.  We have to ask Dmitry and Toby, among others, to guide us
  with their experience of writing it once already, and we have to make
  sure we don't end up with a solution no one wants to use :)

- the backend data should be able to carry more than plain strings,
  things like docstrings and such, and IMO we should try to standardize
  the possible extras at least somewhat.  We may need a deeper backend
  abstraction than `completion-at-point-functions' because of this.

- the frontend choices made by the user should be able to feed back into
  the data backend (the "predictive" use case)

- we have enough interest and support in this work that it's worth
  undertaking in earnest after the code freeze.  The discussion should
  remain on emacs-devel (based on the wide interest so far).

I hope everyone agrees.
Ted

Re: Emacs completion matches selection UI

[Stefan Monnier]

>> > (Though I don't think the `completion-at-point' mechanism is really rich
>> > enough to let Predictive do its stuff properly; in particular updating
>> > the frequency data after completing.)
>> Seems that :exit-function can be used for this. See 
>> `completion-extra-properties'.
> Thanks for the pointer. Unfortunately this isn't enough. In a typical
> predictive-mode scenario, there are many possible completion candidates,
> and Predictive needs to display them, and be told which of the candidates
> was selected (or rejected).

I suspect that :exit-function is "enough", tho it might require some
ugly unreliable hacks to get at the needed data.  Please try it out and
M-x report-emacs-bug requesting the extra features/data to be able to
"do it right".


        Stefan

Re: Emacs completion matches selection UI

[Stefan Monnier]

> - the backend data should be able to carry more than plain strings,
>   things like docstrings and such, and IMO we should try to standardize
>   the possible extras at least somewhat.  We may need a deeper backend
>   abstraction than `completion-at-point-functions' because of this.

completion-at-point-functions can already provide docstrings (which are
then used by Company), see lisp-completion-at-point for an example.


        Stefan

Re: Emacs completion matches selection UI

[Toby Cubitt]

On Fri, Jan 03, 2014 at 02:32:40PM -0500, Stefan Monnier wrote:
> >> > (Though I don't think the `completion-at-point' mechanism is really rich
> >> > enough to let Predictive do its stuff properly; in particular updating
> >> > the frequency data after completing.)
> >> Seems that :exit-function can be used for this. See 
> >> `completion-extra-properties'.
> > Thanks for the pointer. Unfortunately this isn't enough. In a typical
> > predictive-mode scenario, there are many possible completion candidates,
> > and Predictive needs to display them, and be told which of the candidates
> > was selected (or rejected).
> 
> I suspect that :exit-function is "enough", tho it might require some
> ugly unreliable hacks to get at the needed data.  Please try it out and
> M-x report-emacs-bug requesting the extra features/data to be able to
> "do it right".

I still don't see how, if by "enough" you mean enough to emulate the
current Predictive functionality without effectively bypassing the
`completion-at-point' mechanism and running Predictive/Completion-UI code
instead. `completion-at-point' is designed around "tab-completion", and
that just isn't a good match for predictive completion (or a popup.el
style UI, for that matter).

Anyway, I think we basically agree: at the moment
`completion-in-region-functions' and `completion-extra-properties' aren't
enough to implement something like Predictive without ugly hacks. I also
think if we can design a good generic Emacs completion UI and API, these
issues will go away.

Toby
-- 
Dr T. S. Cubitt
Royal Society University Research Fellow
and Fellow of Churchill College, Cambridge
Centre for Quantum Information
DAMTP, University of Cambridge

email: tsc25@cantab.net
web:   www.dr-qubit.org

Re: Emacs completion matches selection UI

[Toby Cubitt]

Thanks for the great summary, Ted!

On Fri, Jan 03, 2014 at 12:49:35PM -0500, Ted Zlatanov wrote:
> On Thu, 02 Jan 2014 13:10:11 -0500 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 
> 
> SM> I'd love to see company-mode and completion-ui consolidated into Emacs's
> SM> core code.  The "completion sources" would be provided via
> SM> completion-at-point-functions, so we already have a place for
> SM> that part.  We don't yet have an API for the "completion UI
> SM> frontend", OTOH, so if we can agree on a good API for that part, we
> SM> could restructure the minibuffer.el code around that and then add
> SM> more UIs.
> 
> Based on the discussion so far, it seems that:
> 
> - there are good use cases for both completion-ui and company-mode, we
>   can't claim either is a superset of the other.  The two are not
>   fundamentally incompatible.

Agreed. My feeling is once people decide what it is they want included in
Emacs, and what the API should look like, re-purposing (parts of) the
Company and/or Completion-UI code bases (maybe others too?) will be a big
head start (assuming no copyright assignment roadblocks).

> - I personally used company-mode heavily in the last few weeks and it
>   works great, but I don't like that after a timeout, the up/down keys
>   get taken over.  I think displaying the completion candidates could
>   happen after a timeout, but selecting from them should happen only
>   when the user explicitly requests it with M-TAB in a buffer or TAB in
>   the minibuffer.

This is a minor detail, which I'm sure would be trivial to change in
Company if so desired. (Wouldn't surprise me if it can already be
disabled by rebinding keys in company-mode keymaps.)

If you enable a `completion-auto-show' interface and
`completion-auto-show-delay' (both enabled by default), Completion-UI
behaves in exactly the same way as Company. People seem to expect the
up/down keys to cycling through completions when they're displayed in a
tooltip - I got bug reports before I implemented it.

> - it should be possible to implement a generic frontend for completion
>   that both can use, and other tools like yasnippet can use IIUC.  The
>   eventual goal is to have "something" standard in Emacs that all such
>   packages (except perhaps helm, which is completely apart in its UI)
>   would use.  We have to ask Dmitry and Toby, among others, to guide us
>   with their experience of writing it once already, and we have to make
>   sure we don't end up with a solution no one wants to use :)

Indeed. It would be nice to get input from the other completion package
authors too (anything.el, auto-complete, icicles...)

> - the backend data should be able to carry more than plain strings,
>   things like docstrings and such, and IMO we should try to standardize
>   the possible extras at least somewhat.

Agreed. 

Since it's the completion UI that will have to interpret and make use of
the extras, one place to start is to think about what additional features
the UI might want to provide, beyond selecting from a list of possible
completions.

Luckily, we have many existing completion frameworks to draw on for
inspiration. At the moment, location data and various forms of
documentation (e.g. type information in code completion, docstrings,
etc.) are the only ones that spring to mind.

> We may need a deeper backend abstraction than
> `completion-at-point-functions' because of this.

It might be sufficient to extend the list of supported properties in the
`completion-at-point-functions' PROPS plist.

The keyword arguments to `completion-ui-register-source' are one starting
point for seeing what additional properties might be needed, as are the
corresponding APIs in Company, auto-complete, etc.

Also, if we want to allow users to optionally customize the completion UI
per-backend (a la Completion-UI), there has to be some way of identifying
backends in Customize. This seems necessary to me if the new API is to
become the standard way of implementing a completion UI in Emacs. Think
minibuffer completion, versus filename completion, versus predictive
completion, versus elisp code completion etc, all of which might want to
use slightly different UI options. Again, it might be enough to support a
:name property (or some such) in the `completion-at-point-functions'
PROPS plist.

> - the frontend choices made by the user should be able to feed back into
>   the data backend (the "predictive" use case)

A few optional hooks/call-backs would very likely be sufficient. Again,
the `completion-at-point-functions' PROPS plist would be the obvious
place to specify them.

> - we have enough interest and support in this work that it's worth
>   undertaking in earnest after the code freeze.  The discussion should
>   remain on emacs-devel (based on the wide interest so far).

At the moment, it sounds to me like the existing
`completion-at-point-functions' and `completion-at-point' /
`completion-in-region' APIs could be extended to provide a more "modern",
richer, and customizable generic Emacs completion UI. Then minibuffer
completion, Company, Predictive, anything, etc. could all eventually be
converted to use this.

> I hope everyone agrees.

Sounds good to me.

(Note that I'm unlikely to have time to code anything myself until
March/April, although the feature freeze will likely delay things until
about then anyway. I'm happy to help in a more limited way before then,
as time allows.)

Toby
-- 
Dr T. S. Cubitt
Royal Society University Research Fellow
and Fellow of Churchill College, Cambridge
Centre for Quantum Information
DAMTP, University of Cambridge

email: tsc25@cantab.net
web:   www.dr-qubit.org

Re: Emacs completion matches selection UI

[Dmitry Gutov]

On 03.01.2014 02:58, Toby Cubitt wrote:
> No, definitely not a wrapper. I was proposing using it as an Elisp
> utility library. If anything, it's Company et al. that would build on top
> of and wrap around a generic Emacs completion UI, rather than the other
> way around.

Ah, that sounds good.

>> although I'd have to look into fitting any of them inside the Company's
>> interaction with the command loop,
>
> ...which BTW is why I suspect the full Company framework isn't suitable
> as a generic Emacs completion UI.

Yes, you could say that. Company is flexible, but could be not "general" 
enough.

> Sure, but as far as I understand (I could well be wrong - I've never used
> it in anger) you use Company by enabling company-mode in your buffer as a
> user, and this minor-mode changes how you interact with Emacs. Whereas
> you use Completion-UI by calling its functions from Elisp when coding a
> new Emacs package. It's really aimed at a "lower" level.

That's normally how it works, yes.

> If Company backend definitions should live in the completion package
> rather than Company, I could add it to Predictive instead.

They can, and they often do. A backend definition is just a function 
that can respond to a variable number of arguments with somewhat 
predetermined values.

> Well, I believe Completion-UI predates at least Company, anything, and
> auto-complete. So "yet-another-completion-package" is misleading. Had
> those packages existed when I coded Predictive, I doubt I'd have bothered
> coding Completion-UI.

I didn't know that. Indeed, it seems to predate Company by a year or so.

> It's a shame Company et al. didn't build on and contribute to
> Completion-UI, and instead chose to (each) reinvent the wheel. But that's
> the usual messy history of code (which maybe we're finally going to do
> something to tidy up?).

Maybe the widget APIs weren't particularly fitting, though. See my 
response to your proposal below.

> Do you want fast predictive completion of plain text, or
> text markup languages like LaTeX?

It could be a decent replacement for dabbrev (I mostly use it to quickly 
complete to some symbol already present before point in the same 
buffer), but the problem with "smart" algorithms that keep history is 
that often enough you can't predict the first option it will offer you. 
That can be annoying, so maybe I don't.

>>> I'd be very happy to see the UI parts of Company stripped out and made
>>> into a simple, generic completion UI package and added to Emacs.
>>
>> Would you like to propose an API for it? Same as popup.el?
>
> As a first attempt, I'd propose something like
>
>    (complete-in-buffer COMPLETION-SOURCE &optional PREFIX-FUNCTION)
>
> where COMPLETION-SOURCE is a function to call to obtain completions, and
> PREFIX-FUNCTION is an optional function to call to determine what to
> complete.

But what if we have a list of completion sources (see the description 
down below<*>)? Some of them can have a different notion of the prefix. 
In Company, we try to remember which backend we're working with at the 
moment, so that we only ask it for completions, candidate metadata, etc.

Though maybe it could work if PREFIX-FUNCTION iterated though backends, 
saved which one responded, and then we could use that value in 
COMPLETION-SOURCE, as well as when implementing the commands providing 
the additional info. In this case, we'll at least need to have a way to 
know when the widget was dismissed or refreshed to erase the saved 
backend value.

In `company-capf', we iterate though the completion-at-point-functions 
each time. This works, and works well enough, since the "does it fit" 
code is usually fast, but it looks fairly bizarre to me, logically.

Ideally, though, the widget interface would be a drawing API of sorts: 
"display these candidates at this point with this current index", 
"update index", "hide", since we have everything else implemented already.

> Calling this would invoke whatever completion widgets the user had
> enabled in customize (with sensible defaults - maybe the popup.el
> interface or similar, or perhaps the default should be the usual
> *Completions* buffer).

I'd still like to a natural way to extend such widgets with new 
commands. Passing a keymap as argument, or rebinding a dynamic variable 
that points to keymap works, I guess, but it's not very nice.

>> A mouse-only menu won't fit Company because it won't work for idle
>> completion, as I mentioned elsewhere in this thread, and we generally
>> want to provide a consistent keyboard interface. There'll also be issues
>> with implementation related to that.
>
> When you register a new UI with Completion-UI, you tell it whether the UI
> is suitable for idle completion. If it is, it becomes one of the
> customization options you can select for the idle completion UI.

Ok, it's reasonable approach, but that means that the UIs can't be 
treated uniformly, which is sort of a bummer. Ideally, we'd have a 
widget that's as useful as a menu when interacted with with a mouse, and 
works with keyboard, and doesn't interrupt the user's typing when 
displayed during idle completion.

If I were to implement something like this for Company, it would mean 
users who like this kind of menu would either have to forgo idle 
completion, or see different kind of UIs whether completion is idle or 
triggered manually. Which sounds weird, at least.

> Sounds like adding it to `company-backends' is more analogous registering
> it with `completion-ui-register-source'.

<*>

Not really. `company-backends' corresponds directly to 
`auto-completion-source', only instead of one function we have a list of 
them (which can also be nested one level deep, but that's an extra feature).

It's similar to `completion-at-point-functions' this way. First, each 
backend knows whether it will be applicable at point (whether the 
buffer's major mode fits, whether the syntax status at point fits, e.g. 
some backends work in strings, others only work outside, etc). So 
instead of calling an analog of `auto-completion-source', Company polls 
all configured backends until it gets a response, then saves the name of 
the backend thus found for the current completion invocation, and calls 
it again for the list of completions. After that, it can call the 
backend for calltips, candidate docs, et cetera.

> Customizing
> `auto-completion-source' is something the user would do, not the Elisp
> package coder.

Same with `company-backends', although we provide a reasonable default. 
But the user can change the global list, of change its local values, for 
example set it to a single-item value in some major mode hook.

> In fact, it sounds like the two APIs are rather similar:

> 2. Pass it to `completion-ui-register-source' (Completion-UI) or add it
>     to `company-backends' (Company) to let them know about it.

Guess the main difference is that, since we definine new backends with 
`defun', there's no way to update the list of available values, visible 
to the `defcustom' interface.

It makes matters a bit worse for third-party backends, but not by much, 
I think.

> At some point in the distant past, you used to just set a variable to
> tell Completion-UI about a new completion function. It became a macro
> call when I made Completion-UI more generic, and there needed to be a way
> of optionally supplying additional information about what the completion
> function does (e.g. if it does something other than standard prefix
> completion), and of defining optional call-back functions, etc.

We solve this problem by requiring backend functions to take the type of 
question it's going to answer as the first argument. Function calls are 
fast enough for our purpose, and this makes definitions quite succinct, 
especially for simple backends.

This pattern is used in many places in Emacs core, too.

Re: Emacs completion matches selection UI

[Stefan Monnier]

>> I suspect that :exit-function is "enough", tho it might require some
>> ugly unreliable hacks to get at the needed data.  Please try it out and
>> M-x report-emacs-bug requesting the extra features/data to be able to
>> "do it right".
> I still don't see how, if by "enough" you mean enough to emulate the
> current Predictive functionality without effectively bypassing the
> `completion-at-point' mechanism and running Predictive/Completion-UI code
> instead.

I don't know enough about Predictive to be able to
answer.  :exit-function is called when the completion ends and gives
access to the choice made by the user, so it is sufficient to provide
some kind of feedback about which was the right choice.  But there might
indeed be issues about exactly when the :exit-function is called, for
example, or how/when the choices are presented.

> `completion-at-point' is designed around "tab-completion", and
> that just isn't a good match for predictive completion (or a popup.el
> style UI, for that matter).

completion-at-point-functions is largely designed for the purpose of
completion-at-point, but it is not restricted to that; so do not confuse
the two.


        Stefan

Re: Emacs completion matches selection UI

[Stefan Monnier]

> Emacs, and what the API should look like, re-purposing (parts of) the
> Company and/or Completion-UI code bases (maybe others too?) will be a big
> head start (assuming no copyright assignment roadblocks).

Company is in GNU ELPA, so its copyright status is already clear.
I'd be happy to see completion-ui in GNU ELPA as well.

> Think minibuffer completion, versus filename completion, versus
> predictive completion, versus elisp code completion etc, all of which
> might want to use slightly different UI options.

Not sure why.  Maybe I just don't understand what you mean by "UI
options".  Do you have concrete examples in mind?


        Stefan

Re: Emacs completion matches selection UI

[Stefan Monnier]

> In `company-capf', we iterate though the completion-at-point-functions each
> time. This works, and works well enough, since the "does it fit" code is
> usually fast, but it looks fairly bizarre to me, logically.

Agreed.  It was just "the easy" way to fit it in.
"The right thing" would be to replace company-backends with
completion-at-point-functions.


        Stefan

Re: Emacs completion matches selection UI

[João Távora]

Ted Zlatanov <tzz@lifelogs.com> writes:
> Based on the discussion so far, it seems that:
>
> [...]
> - it should be possible to implement a generic frontend for completion
>   that both can use, and other tools like yasnippet can use IIUC.  The

Thanks for the summary of the discussion so far, but this last statement
might bring some confusion... Yasnippet can curiously fit into many
parts of the problem:

* It can be a source to completion frameworks, auto-complete and company
  have such sources.

* It prompts the user when it finds more than one snippet for an abbrev,
  and uses its own `yas-prompt-functions' abstraction. Some of the
  functions offer completion.

* It installs a keybinding for TAB which is problematic because it is
  very sought-after binding for completion-related stuff. The parts of
  this discussion that concerned yasnippet dealed with this, AFAIK.

>   with their experience of writing it once already, and we have to make
>   sure we don't end up with a solution no one wants to use :)

Yep. For me personally it would make sense to evaluate alternatives
based a matrix like the one proposed by Stefan earlier:

http://article.gmane.org/gmane.emacs.devel/166661/match=completion+ui

João

Re: Emacs completion matches selection UI

[Stefan Monnier]

>> Yes.  We should make it easier to extend indent-for-tab-command.
> Do you think it's possible/desirable to generalize this mechanism (to
> support the "fallback-maybe" pattern) so that we can make it easier to
> extend any command, not just indent-for-tab-command?

Everything's possible and yes, I think there's interest in supporting
this cleanly.  There are different approaches, tho.

One is advice-add.  This makes it possible to extend a command with new
behavior, with a clean and simple fallback mechanism.

If OTOH you want to "extend a key-binding", we don't have a good story.
You can do things like

   (define-key map [?\C-i]
     `(menu-item "" mycommand
       :filter ,(lambda (cmd) (if <pred> cmd))))

which will use `mycommand' if <pred> is true and fallback on the next
binding otherwise.  But this is not very satisfactory:
- no way to do something after the fallback command is run.
- C-h k will only tell you about one of the possible commands.
- Can't have more than one such "conditional binding" for the same key
  in the same keymap.

> (define-fallbackish-command maybe-foo (kbd "[somekey]") ()

I don't understand what's the intended relation between [somekey] and
`maybe-foo'.


        Stefan

Re: Emacs completion matches selection UI

[João Távora]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> (define-fallbackish-command maybe-foo (kbd "[somekey]") ()
>
> I don't understand what's the intended relation between [somekey] and
> `maybe-foo'.

Sorry, I should have provided some implementation, or made this more
clear. I

The generated part of the `maybe-foo' would do two things:

* register the association between maybe-foo and [somekey] in a some
  global variable, pushing in front of any existing ones also defined
  with the same macro. This keeps track of the "fallbackish commands"
  (horrible name, sorry) for [somekey]

* bind maybe-foo to [somekey] in the (global) keymap, maybe replacing
  someone else's binding.

When (if ever) `fallback' is called from a command's body, it looks up
the first variable. Its a list of commands that are still fallbacks to
the current. It removes the current command from the list, let-binds the
list and calls the next command.

Now, I haven't even attempted this, but it sounded good (in my head :))
and think it could resolve most problems. I'll try to implement it and
get back to you, unless someone immediately detects some big error.

One problem is that you more or less now have two bindings to
synchronize, and this may be quite bad. The map's binding and the other
global variable. I have no idea how it would work with multiple keymaps.

> binding otherwise.  But this is not very satisfactory:
> - no way to do something after the fallback command is run.

`fallback' could be called anywhere from `maybe-foo's body.

> - C-h k will only tell you about one of the possible commands.

yasnippet does this with

   (put 'yas-expand  'function-documentation
        '(yas--expand-from-trigger-key-doc t))

in which the latter function calls a `yas--keybinding-beyond-yasnippet'
function. I believe this could also be done with the above idea.

> - Can't have more than one such "conditional binding" for the same key
>   in the same keymap.

This is the main use case i'm trying to cover.

João

Re: Emacs completion matches selection UI

[Dmitry Gutov]

On 04.01.2014 07:54, Stefan Monnier wrote:
>> In `company-capf', we iterate though the completion-at-point-functions each
>> time. This works, and works well enough, since the "does it fit" code is
>> usually fast, but it looks fairly bizarre to me, logically.
>
> Agreed.  It was just "the easy" way to fit it in.

Can't we save the last `company--capf-data' value in the `prefix' case? 
Company only calls the rest of the cases after it, and only when the 
returned value is non-nil. Since prefix value has to stay fresh, the 
cached data will also be.

> "The right thing" would be to replace company-backends with
> completion-at-point-functions.

I'm not convinced yet that it's the right thing to do.

a) `c-a-p-f' looks less accessible to the end user than 
`company-backends', and the former isn't even a defcustom. The structure 
of the latter, on the other hand, is easier to read and modify in users' 
init files and major mode hooks.

The Company backends are also easier to spot among other functions in 
the packages that contain them. The names of the c-a-p-functions used by 
e.g. IELM follow no discernible convention.

b) There's no direct analog to "merged" Company backends in 
`completion-at-point-functions'. `company-backends' can contain list 
values, and `company--multi-backend-adapter' handles the merging of the 
returned data.

I'm also tinkering with the idea to make individual candidates remember 
the backends they came from (using text properties, most likely), so 
that we will know which backend's function to use with this candidate. 
Calling `post-completion' action from the wrong backend would be the 
best example of how it's currently broken.

Re: Emacs completion matches selection UI

[Stefan Monnier]

> a) `c-a-p-f' looks less accessible to the end user than `company-backends',

That's because it's mostly not meant for the users to change, and that's
because users should not need to touch it.

`company-backends' in contrast contains a mishmash of things, 90% of
which is irrelevant to any given situation.

> The names of the c-a-p-functions used by e.g. IELM follow no
> discernible convention.

A good naming convention for them would be welcome, indeed.

> b) There's no direct analog to "merged" Company backends in
> completion-at-point-functions'. `company-backends' can contain list values,
> and `company--multi-backend-adapter' handles the merging of the
> returned data.

Indeed, there's no such thing yet.  Nothing prevents us from providing
a "completion-at-point-merge-backends" function which takes a list of
completion-at-point-functions and returns a new completion-at-point-function.

company-backends and completion-at-point-functions don't work 100%
identically, but the differences are present only because of history.
If Company had started with completion-at-point-functions it would live
very happily with it.


        Stefan

Re: Emacs completion matches selection UI

[Stefan Monnier]

>>> (define-fallbackish-command maybe-foo (kbd "[somekey]") ()
>> I don't understand what's the intended relation between [somekey] and
>> `maybe-foo'.
> Sorry, I should have provided some implementation, or made this more
> clear. I
> The generated part of the `maybe-foo' would do two things:
> * register the association between maybe-foo and [somekey] in a some
>   global variable, pushing in front of any existing ones also defined
>   with the same macro. This keeps track of the "fallbackish commands"
>   (horrible name, sorry) for [somekey]
> * bind maybe-foo to [somekey] in the (global) keymap, maybe replacing
>   someone else's binding.

We could do the above fairly easily with

  (add-function :around (lookup-key global-map [somekey])
                (lambda (orig &rest args)
                  (if (foop) (foo) (apply orig args))))

> I have no idea how it would work with multiple keymaps.

Right, that's where it gets more interesting.  E.g. some major mode
might want to extend the meaning of TAB, but it should do it in the
major mode's map, not in the global map.


        Stefan

Re: Emacs completion matches selection UI

[Toby Cubitt]

On Sat, Jan 04, 2014 at 06:39:11AM +0400, Dmitry Gutov wrote:
> On 03.01.2014 02:58, Toby Cubitt wrote:
> > It's a shame Company et al. didn't build on and contribute to
> > Completion-UI, and instead chose to (each) reinvent the wheel. But that's
> > the usual messy history of code (which maybe we're finally going to do
> > something to tidy up?).
> 
> Maybe the widget APIs weren't particularly fitting, though. See my 
> response to your proposal below.

Note that my proposal was a first stab at an API for generic Emacs
completion UI; the Completion-UI APIs are much richer than this.

The modern Completion-UI APIs are very likely either already sufficient
or easily extended to support everything Company needs. But that's
nowadays. The original 2006 Completion-UI APIs were much clunkier, and
almost certainly weren't up to the job.

I'm sure if you'd built Company on top of Completion-UI, we would have
rapidly improved the APIs to support everything you needed as Company
grew. But that's ancient history now, and no big deal.

> > Do you want fast predictive completion of plain text, or
> > text markup languages like LaTeX?
> 
> It could be a decent replacement for dabbrev (I mostly use it to quickly 
> complete to some symbol already present before point in the same 
> buffer), but the problem with "smart" algorithms that keep history is 
> that often enough you can't predict the first option it will offer you. 
> That can be annoying, so maybe I don't.

You can switch off the learning algorithms by toggling a customization
option, and freeze the ordering once and for all if you want
stability. If you're happy with the default frequency tables, you can
even switch it off from the start.

I originally thought I'd want to disable the learning algorithms once I'd
trained Predictive on a corpus of my writing, for precisely the reason
you describe. But in practice the order is stable enough that I never
found it necessary.

> >> Would you like to propose an API for it? Same as popup.el?
> >
> > As a first attempt, I'd propose something like
> >
> >    (complete-in-buffer COMPLETION-SOURCE &optional PREFIX-FUNCTION)
> >
> > where COMPLETION-SOURCE is a function to call to obtain completions, and
> > PREFIX-FUNCTION is an optional function to call to determine what to
> > complete.
> 
> But what if we have a list of completion sources (see the description 
> down below<*>)? Some of them can have a different notion of the prefix. 
> In Company, we try to remember which backend we're working with at the 
> moment, so that we only ask it for completions, candidate metadata,
> etc.

Sure, Completion-UI also remembers the backend ("completion source") that
was used to find the current set of completion candidates, so that you
can continue to interact with it sensibly.

Clearly any generic Emacs completion UI would have to do something
similar. I don't see a big problem supporting multiple different
completion sources. Indeed, I view it as a basic requirement of any
decent Emacs completion API if it's to be the generic UI.

Completion-UI stores state by marking the location of the in-progress
completion with an overlay, and storing data in overlay properties. You
could alternatively use text properties, as you mention later. But I find
overlays a better match for this kind of thing. (Presumably Company does
something similar.) This means you can, if you want, have multiple
pending completions using different sources in the same buffer at the
same time, and Completion-UI will cope just fine if you switch back and
forth between then. (Though the default setting of
`completion-how-to-resolve-old-completions' disables this, as it's at
least as confusing as `enable-recursive-minibuffers'.)


> Though maybe it could work if PREFIX-FUNCTION iterated though backends, 
> saved which one responded, and then we could use that value in 
> COMPLETION-SOURCE, as well as when implementing the commands providing 
> the additional info. In this case, we'll at least need to have a way to 
> know when the widget was dismissed or refreshed to erase the saved 
> backend value.

The above API was just a first stab at the most basic generic Elisp API
for displaying a list of completion candidates and letting the user
select one. It wasn't intended to be an API for defining a bunch of
different a completion sources (backends) and selecting between then.

As I wrote in my follow-up email, at this point we may be stuck with
`completion-at-point-functions' for the latter. (Though we can try
working on Stefan to see if he'll let us ditch it for something less
clunky ;-)

> In `company-capf', we iterate though the completion-at-point-functions
> each time. This works, and works well enough, since the "does it fit"
> code is usually fast, but it looks fairly bizarre to me, logically.

I agree. The `completion-at-point-functions' API is convoluted compared
to either the Company-native API or the Completion-UI API. On the other
hand, it's already part of Emacs and is sufficiently flexible (if a
little ugly). So we may be stuck with it now...

> Ideally, though, the widget interface would be a drawing API of sorts: 
> "display these candidates at this point with this current index", 
> "update index", "hide", since we have everything else implemented already.
> 
> > Calling this would invoke whatever completion widgets the user had
> > enabled in customize (with sensible defaults - maybe the popup.el
> > interface or similar, or perhaps the default should be the usual
> > *Completions* buffer).
> 
> I'd still like to a natural way to extend such widgets with new 
> commands. Passing a keymap as argument, or rebinding a dynamic variable 
> that points to keymap works, I guess, but it's not very nice.

Well, this is only meaningful for some of the widgets. Indeed, "widgets"
is mislead; the completion UIs in Completion-UI aren't necessarily
widgets at all. For example, there are no "commands" at all in the
dynamic completion UI, nor is it a "widget" in any normal sense of the
word.

But for those UIs for which it makes sense, Completion-UI already allows
them to be extended, if necessary in different ways for different
completion sources. In classic Emacs fashion, you supply
functions/keymaps/etc. that override the defaults used to create widget
contents. 

This API could rather easily be improved and extended once it becomes
clear what would be useful or more convenient. But the basic machinery is
already in place.

(The list of override functions, keymaps etc. is *not* hard-coded. The
API for adding a new completion user interface to Completion-UI allows
you to add additional configuration options recognised by the new UI. The
only limitation is that these won't be documented in the
`completion-ui-register-source' docstring.)

> >> A mouse-only menu won't fit Company because it won't work for idle
> >> completion, as I mentioned elsewhere in this thread, and we generally
> >> want to provide a consistent keyboard interface. There'll also be issues
> >> with implementation related to that.
> >
> > When you register a new UI with Completion-UI, you tell it whether the UI
> > is suitable for idle completion. If it is, it becomes one of the
> > customization options you can select for the idle completion UI.
> 
> Ok, it's reasonable approach, but that means that the UIs can't be 
> treated uniformly, which is sort of a bummer.

The UIs absolutely cannot be treated uniformly, because not all of them
are "display a list of completions"-style UIs. E.g. the "dynamic
completion" UI is a completely different beast.

Even amongst the "display a list of completions"-style UIs, the
mouse-driven toolkit menus cannot be used in the terminal, nor can pop-up
frames, nor can tooltips. That doesn't mean one should not provide these
interfaces at all as options. It just means they probably aren't good
default options.

The default idle-completion UI in Completion-UI uses popup.el, which
works everywhere. It's the best "display a list of completions"-style UI,
for this and many other reasons. But that doesn't mean one shouldn't
provide toolkit menus, or tooltips, or popup-frames *as well*, either so
they can be triggered manually or as alternative options for the idle UI.

Completion-UI has default keybindings to bring up the other UIs, even
though they're not displayed by default. This is particularly appropriate
for e.g. the mouse-driven toolkit menu and completion browser menu (which
steal the keyboard focus).

My philosophy with Completion-UI was to provide multiple UIs (and make it
easily and modularly extendable so people can add additional UIs if they
want), to have sensible default UI settings, but to let the user
enable/disable/reconfigure them as they wish via Customize.

Despite all this flexibility, and the very different types of UI it
supports (which are not all of the "display a list of completions"
variety), the UIs *are* treated uniformly in Completion-UI. If they
weren't, I couldn't support arbitrary third-party UIs. Take a look at
`completion-ui-register-interface' to see the API for all the different
UIs. All the bundled UIs are defined via exactly this same API, with no
special treatment.

Furthermore, the completion *sources* (backends) don't need to know
*anything* about how the UIs work. Indeed, they *never* invoke a specific
UI directly. They simply return a bunch of completions, and Completion-UI
presents them to the user via the various completion UIs, as customized
*by the user*.

> Ideally, we'd have a widget that's as useful as a menu when interacted
> with with a mouse, and works with keyboard, and doesn't interrupt the
> user's typing when displayed during idle completion.

Sure, it's called popup.el. If that's unavailable, Completion-UI has an
echo-area + hotkeys UI, which is a poor man's substitute for
popup.el-style menus. (You can also enable both at once if you like.)

> If I were to implement something like this for Company, it would mean 
> users who like this kind of menu would either have to forgo idle 
> completion, or see different kind of UIs whether completion is idle or 
> triggered manually. Which sounds weird, at least.

No, you'd provide sensible defaults that give a uniform UI, but let users
customize it to enable different kinds of UIs if they so desire.

The mouse-driven toolkit menu doesn't replace other (maybe better) UIs.
It complements it and/or is available as an alternative. I fail to see
the problem. If the user wants to use what you consider a weird
combination of UIs, I don't see any good reason to prevent them.

The Emacs way has always been to provide sensible default settings, but
then give users enough rope to shoot themselves in the foot if that's
what tickles their fancy(*).

(*) M-x reconfigure-rope-as-gun-and-shoot-foot


> > Sounds like adding it to `company-backends' is more analogous registering
> > it with `completion-ui-register-source'.
> 
> <*>
> 
> Not really. `company-backends' corresponds directly to 
> `auto-completion-source', 

Not quite. `company-backends' corresponds to the
`completion-ui-source-definitions' variable.

> only instead of one function we have a list of 
> them (which can also be nested one level deep, but that's an extra feature).

Sure, Completion-UI also allows a list of completion sources and selects
between them automagically. It even supports the same "extra feature" of
nesting sources one level deep (Combined source, in Completion-UI
terminology).

But the API for this isn't auto-completion-default-source(*). That merely
sets the default fall-back. The API for auto-selecting from a list of
sources is provided by `auto-completion-source-regexps',
`auto-completion-source-faces' and the `auto-completion-source-functions'
hook. (The former two are in fact variables used by functions in the
default `auto-completion-source-functions' definition.)

(*) I'm referring here to the git version of Completion-UI; the last
tarball release - which is woefully outdated - used a slightly different
API. (I've been too busy with real work for the last few years to get
around to rolling and testing a new tarball release; the git version
still gets updates periodically.)

> It's similar to `completion-at-point-functions' this way. First, each 
> backend knows whether it will be applicable at point (whether the 
> buffer's major mode fits, whether the syntax status at point fits, e.g. 
> some backends work in strings, others only work outside, etc). So 
> instead of calling an analog of `auto-completion-source', Company polls 
> all configured backends until it gets a response, then saves the name of 
> the backend thus found for the current completion invocation, and calls 
> it again for the list of completions. After that, it can call the 
> backend for calltips, candidate docs, et cetera.

Yup, Completion-UI does something very similar. Except that completion
source (backend) selection isn't directly tied to the backend. It's
configured independently. So if you need to, you can set up different
conditions under which a particular backend will be used in different
buffers, major-modes, minor-modes, or whatever.

Perhaps this is overkill. But as I keep coming back to, I designed
Completion-UI primarily to be a generic completion UI elisp library, not
as a user-level completion mode. And I do use some of this flexibility in
Predictive's sophisticated LaTeX support.


> > Customizing
> > `auto-completion-source' is something the user would do, not the Elisp
> > package coder.
> 
> Same with `company-backends', although we provide a reasonable default. 
> But the user can change the global list, of change its local values, for 
> example set it to a single-item value in some major mode hook.

Indeed, one of the best things about Company is all the backends that
come pre-configured. I'm sure that accounts for much of its
popularity. If I'd gone to the same effort of writing backends for
Completion-UI, probably more people would use it.

But the itch I was scratching wasn't to write a user-level completion
framework like Company. I just wanted a UI for Predictive, and figured I
might as well make it into a generic Elisp library so it could also be
used elsewhere. I'd kinda hoped people might pick it up and use it to
build something like Company or anything.el, so they could focus their
time on building a great user-level completion mode rather than
duplicating all the UI code, and instead help improve and contribute to
Completion-UI. But for various reasons that never happened.

Anyhow, I don't think company-mode is something that necessarily belongs
in a generic Emacs completion UI. At most, Emacs should include the
necessary definitions to hook the basic completion methods that come with
Emacs into the new generic UI (dabbrev, etags, minibuffer completion,
maybe a few others like elisp symbol completion...).

Indeed, I would have argued that we should *first* come up with a generic
Emacs completion UI and API *without* including any mechanism for
automatically selecting between completion sources. I would have proposed
to think later about when a source-selection mechanism belonged in Emacs,
or whether it's best left to ELPA packages like Company...

...except that `completion-at-point-functions' *already* defines an API
for selecting between completion sources. And it's already part of
Emacs. So I guess that decision has already been taken.

(I think there's still a discussion to be had about whether or not some
form of auto-completion-mode belongs in Emacs. I can see arguments both
ways.)


> > In fact, it sounds like the two APIs are rather similar:
> 
> > 2. Pass it to `completion-ui-register-source' (Completion-UI) or add it
> >     to `company-backends' (Company) to let them know about it.
> 
> Guess the main difference is that, since we definine new backends with 
> `defun', there's no way to update the list of available values, visible 
> to the `defcustom' interface.

You define new backends (completion functions) for Completion-UI with
`defun' in exactly the same way. You don't *have* to register a
completion function in order to use it with Completion-UI. If you want,
you can simply pass the completion function to `complete-in-buffer', much
like `completion-in-region'.

Registering a source just (1) makes life more convenient: you set all the
option values once when you define the completion source (backend), and
then you can refer to that source and all its associated options using a
single name (symbol). And (2) it makes the source available as an option
in the Customization interface, as you noted.

> It makes matters a bit worse for third-party backends, but not by much, 
> I think.

I agree, this is a minor difference, and Company doesn't lose much by it,
especially since it bundles almost all the backend definitions you're
ever likely to want.

On the other hand, it was more important for third-party completion
sources to be first-class citizens in Completion-UI, precisely *because*
I never put much effort into writing source definitions myself.

For a generic Emacs completion UI, it's obviously crucial that third-part
completion sources are first-class citizens, since almost all sources
will be third-party.


> > At some point in the distant past, you used to just set a variable to
> > tell Completion-UI about a new completion function. It became a macro
> > call when I made Completion-UI more generic, and there needed to be a way
> > of optionally supplying additional information about what the completion
> > function does (e.g. if it does something other than standard prefix
> > completion), and of defining optional call-back functions, etc.
> 
> We solve this problem by requiring backend functions to take the type of 
> question it's going to answer as the first argument. Function calls are 
> fast enough for our purpose, and this makes definitions quite succinct, 
> especially for simple backends.

Simple backends have trivial `completion-ui-register-source' calls in
Completion-UI, too. All you need is:

  (completion-ui-register-source my-completion-function)

The macro figures out everything it needs to know from the function
definition - the beauty of lisp macros!

It's shorter than typing:

  (add-hook 'completion-at-point-functions 'my-new-completion-function)

;-)

The optional macro arguments only come into play if you're doing
something more sophisticated (like extending the default completion
widgets, or overriding the default method of determining the text around
point that's to be completed, etc.)

> This pattern is used in many places in Emacs core, too.

In principle, this sounds like a perfectly reasonable API choice for
simple settings.

Though I don't completely get how it works here. What if a particular
completion function can't answer some types of question? E.g. maybe it
can expand prefixes, but it can't find regexp matches. What does the
completion function do if it's then asked for a regexp match? Or does the
backend answer "I can't answer this type of question" and Company moves
onto the next backend in the list?

However, this API is not so convenient if you want to pass more complex
settings, such as extending the UI for a particular source. You'd either
have to pass the same list of hook functions, keymaps etc. every time you
call the function that invokes the completion UI for the source. Or you'd
have to let-bind the same set of variables every time. Which would be a
pain.

Surely more convenient to have a way of telling Emacs: "use this set of
options, hook functions, keymaps, etc. every time I use this completion
source to complete something". Naturally, you can do this via
configuration variables that store
alists/plists/insert-your-favourite-associative-array-here. And guess
what? Most of what the `completion-ui-register-source' macro does is add
a bunch of elements to the appropriate plists for you. You can do this
manually if you prefer, but the macro call is so much more convenient.

This pattern is used in many places in Emacs core, too: `defcustom',
`defstruct', `defun*'... (Perhaps I should have called the macro
`defcompletion' instead of `completion-ui-register-source' :)

Also, the "pass the type of completion as the first argument" API means
you have a hard-coded set of "types" of completion. As usual, since it
was intended to be a generic completion Elisp library and not a
fully-fledged user-level completion framework, Completion-UI doesn't
impose any such restriction, but tries to be as flexible and general as
possible. I'm not saying this is necessarily a good thing; it could well
be that this flexibility is overkill.



Thanks for taking the time to explain and compare Company features. It's
very helpful in figuring out what a generic Emacs completion API should
look like, at least to me.

I increasingly have the impression that the Company and Completion-UI
feature sets and APIs are in fact functionally very similar, modulo API
details and a useful feature or two here and there on each side. (Leaving
aside the massive collection of Company backend definitions, which
Completion-UI completely(!) lacks.)

Which bodes well for coming up with a generic Emacs completion UI API and
code. If we've converged on a broadly similar set of features and APIs
for the UI parts of Completion-UI and Company, there's a reasonable
chance they're the features people will want from a generic Emacs API.

Best,
Toby
-- 
Dr T. S. Cubitt
Royal Society University Research Fellow
and Fellow of Churchill College, Cambridge
Centre for Quantum Information
DAMTP, University of Cambridge

email: tsc25@cantab.net
web:   www.dr-qubit.org

Re: Emacs completion matches selection UI

[Toby Cubitt]

On Fri, Jan 03, 2014 at 10:39:00PM -0500, Stefan Monnier wrote:
> >> I suspect that :exit-function is "enough", tho it might require some
> >> ugly unreliable hacks to get at the needed data.  Please try it out and
> >> M-x report-emacs-bug requesting the extra features/data to be able to
> >> "do it right".
> > I still don't see how, if by "enough" you mean enough to emulate the
> > current Predictive functionality without effectively bypassing the
> > `completion-at-point' mechanism and running Predictive/Completion-UI code
> > instead.
> 
> I don't know enough about Predictive to be able to answer.

Predictive needs to know what was the final text that got inserted in the
buffer after the whole completion process finished. Expanding a string to
the longest common prefix doesn't count as finished as far as Predictive
is concerned.

> :exit-function is called when the completion ends and gives access to
> the choice made by the user, so it is sufficient to provide some kind
> of feedback about which was the right choice.

If :exit-function is called with STATUS = `finished` if and only if the
STRING that was inserted in the buffer was one of the candidates returned
by the `completion-at-point-functions' hook, I think that would be
sufficient for Predictive.

I couldn't easily understand from the docstring or from skimming the code
whether that's what it does. I'll try playing around with it more fully
to see what happens as soon as I have a spare moment. But for that I need
to hack together a completion function and :exit-function to test with.

> But there might indeed be issues about exactly when the :exit-function
> is called, for example, or how/when the choices are presented.

Certainly a richer completion UI would probably require some changes and
additions to `completion-extra-properties'.


> > `completion-at-point' is designed around "tab-completion", and
> > that just isn't a good match for predictive completion (or a popup.el
> > style UI, for that matter).
> 
> completion-at-point-functions is largely designed for the purpose of
> completion-at-point, but it is not restricted to that; so do not confuse
> the two.

I don't think I was confused about the relation between
completion-at-point and completion-at-point-functions. I was confused
about what exactly completion-at-point does, but I think I understand
now. I was wrong about it not being a good match for the popup.el UI,
which could quite easily replace the *Completions* buffer.

Just to be clear, by "tab-completion" I meant the style of completion
that expands a string to the longest common prefix if possible and stops,
or displays a list of possible completions if there's no common
prefix. Which is exactly what `completion-at-point' does.

This style isn't a good fit to predictive completion, since arguably the
whole point of predictive completion is to guess the most likely
completions from all the possible candidates, and make it as easy as
possible to quickly select one of those. Expanding to a longest common
prefix and then stopping has no place in this.

Anyway, this is a digression. It would already be possible to define
Predictive as a `completion-at-point' function. The current
`completion-at-point' *user-interface* isn't a particularly good fit to
Predictive, so the result would be crippled compared to
predictive-mode. But this thread is all about improving that UI. I see no
reason why, with a richer Emacs completion UI, Predictive couldn't work
just fine with `completion-at-point'.

Best,
Toby
-- 
Dr T. S. Cubitt
Royal Society University Research Fellow
and Fellow of Churchill College, Cambridge
Centre for Quantum Information
DAMTP, University of Cambridge

email: tsc25@cantab.net
web:   www.dr-qubit.org

Re: Emacs completion matches selection UI

[Stefan Monnier]

> I agree.  The `completion-at-point-functions' API is convoluted compared
> to either the Company-native API or the Completion-UI API.

Partly for historical reasons, but partly because it handles more
complex kinds of completions, such as file-name completion with
partial-completion.

> On the other hand, it's already part of Emacs and is sufficiently
> flexible (if a little ugly).  So we may be stuck with it now...

Extending the company API or the completion-ui API to handle things like
filename completion (including the issues linked to handling
quoting/unquoting) is far from obvious, so whether we like it or not,
I think that completion-at-point-functions is not really negotiable.


        Stefan

Re: Emacs completion matches selection UI

[Dmitry Gutov]

On 05.01.2014 20:04, Stefan Monnier wrote:
>> a) `c-a-p-f' looks less accessible to the end user than `company-backends',
>
> That's because it's mostly not meant for the users to change, and that's
> because users should not need to touch it.

I disagree. Two examples:

1. I'm planning to write a new backend that would integrate with 
Yasnippet. But it's not particularly useful as a stand-alone completion 
function, because when the user types "if", they might want to either 
expand it to "if (...) { }", or, for example, call a function whose name 
starts with "if", and they should see both options. In Company terms, 
that means that those backends should be merged: the first item in 
`company-backends' will be a list with the names of these two backends 
(or maybe more). Doing this programmatically for the user could be 
fairly surprising, and, in the case of `c-a-p-f', opaque.

2. We have two backends that are applicable in a similar set of major 
modes: company-semantic and company-clang. We can check if we can use 
either (by seeing if Semantic is enabled and by looking for the Clang 
program), but if both are usable, user might want to prefer to use the 
one or the other. How to let them pick? Providing a couple of minor 
modes, where each would only enable/disable a specific backend/capf is 
possible, but very clunky, as far as I'm concerned, and requires special 
treatment. (Do I provide a minor mode for this backend function? How 
about this other one?)

> `company-backends' in contrast contains a mishmash of things, 90% of
> which is irrelevant to any given situation.

That's just the default value.

Users can set it to smaller lists in specific major modes to make it 
more "relevant" if they want, but the relevancy check is fast, so by 
default the list is global and is indeed a mishmash, to make getting 
started easier.

> Nothing prevents us from providing
> a "completion-at-point-merge-backends" function which takes a list of
> completion-at-point-functions and returns a new completion-at-point-function.

Indeed. Although it seems to me that remembering where each candidate 
came from (feature not yet present in Company either) would be harder to 
implement, because completion functions are pretty much nameless as far 
as the code using them is concerned. So propertizing the candidates with 
(backend . company-elisp) won't work like it would in Company.

> company-backends and completion-at-point-functions don't work 100%
> identically, but the differences are present only because of history.

I believe the main difference is "users customize it" vs. "users 
generally don't know how to change it".

> If Company had started with completion-at-point-functions it would live
> very happily with it.

That's possible.

Re: Emacs completion matches selection UI

[Toby Cubitt]

On Fri, Jan 03, 2014 at 10:45:37PM -0500, Stefan Monnier wrote:
> > Emacs, and what the API should look like, re-purposing (parts of) the
> > Company and/or Completion-UI code bases (maybe others too?) will be a big
> > head start (assuming no copyright assignment roadblocks).
> 
> Company is in GNU ELPA, so its copyright status is already clear.
> I'd be happy to see completion-ui in GNU ELPA as well.
> 
> > Think minibuffer completion, versus filename completion, versus
> > predictive completion, versus elisp code completion etc, all of which
> > might want to use slightly different UI options.
> 
> Not sure why.  Maybe I just don't understand what you mean by "UI
> options".  Do you have concrete examples in mind?

Sure.

If you're doing predictive plain-text completion, you probably don't want
to display all possible completions all the time. You probably want to
limit the list to the 10 or so most likely completions. You might also
want to set a minimum prefix length below which Emacs refuses to search
for completions (finding all completions of the letter "t" in an English
dictionary can take a while...unless you're using Predictive of course,
in which case it takes fractions of a second ;-)

Whereas for filename completion, you most likely always want to list all
the possible completions, and you probably want to show completions even
for single-letter prefixes.


Another example: the "dynamic completion" UI in Completion-UI is
something like `hippie-expand', mimicking the kind of thing you see in
IDEs, or LibreOffice, or mobile phones: the best completion candidate is
displayed highlighted in the buffer, and typing a space or punctuation
character inserts that completion. This a very useful and widely-used UI
for predictive plain-text completion.

For code completion, this UI is infuriating. You'll almost certainly
either want to disable it completely, or at least configure it so that
the highlighted completion only gets inserted in the buffer when you
explicitly tell it to with a specific key binding.


Yet another: For plain-test predictive completion, single-character
"hotkey" bindings to quickly select one of the first n completions are
quite useful, e.g. you might use the characters 0-9. But these
single-character bindings mean you can't type any of those characters
whilst you're completing. For filename completion, you'll almost
certainly want to disable these bindings.


I could go on. But maybe that's enough to convince you that it's useful
to be able to configure the UI slightly differently for different kinds
of completion. Even if most of the time you'll want to configure the UI
globally.

In Completion-UI, I have a `defcustom-per-source' macro (a drop-in
replacement for `defcustom') that automatically turns a standard
defcustom definition into one that can optionally take different values
for particular completion sources. So making customization options
configurable per-source is zero effort.

Hope that helps,
Toby
-- 
Dr T. S. Cubitt
Royal Society University Research Fellow
and Fellow of Churchill College, Cambridge
Centre for Quantum Information
DAMTP, University of Cambridge

email: tsc25@cantab.net
web:   www.dr-qubit.org

Re: Emacs completion matches selection UI

[Toby Cubitt]

On Mon, Jan 06, 2014 at 08:25:34AM +0400, Dmitry Gutov wrote:
> On 05.01.2014 20:04, Stefan Monnier wrote:
> >> a) `c-a-p-f' looks less accessible to the end user than `company-backends',
> >
> > That's because it's mostly not meant for the users to change, and that's
> > because users should not need to touch it.
> 
> I disagree. Two examples:
> 
> 1. I'm planning to write a new backend that would integrate with 
> Yasnippet. But it's not particularly useful as a stand-alone completion 
> function, because when the user types "if", they might want to either 
> expand it to "if (...) { }", or, for example, call a function whose name 
> starts with "if", and they should see both options. In Company terms, 
> that means that those backends should be merged: the first item in 
> `company-backends' will be a list with the names of these two backends 
> (or maybe more). Doing this programmatically for the user could be 
> fairly surprising, and, in the case of `c-a-p-f', opaque.

We could allow the COLLECTION argument to be a list of collections, that
then get merged by completion-at-point. I agree it's opaque.

> 2. We have two backends that are applicable in a similar set of major 
> modes: company-semantic and company-clang. We can check if we can use 
> either (by seeing if Semantic is enabled and by looking for the Clang 
> program), but if both are usable, user might want to prefer to use the 
> one or the other. How to let them pick? Providing a couple of minor 
> modes, where each would only enable/disable a specific backend/capf is 
> possible, but very clunky, as far as I'm concerned, and requires special 
> treatment. (Do I provide a minor mode for this backend function? How 
> about this other one?)

Presumably the user's supposed to write a function that returns whichever
capf they want to use, and add it to the c-a-p-f hook.

Or, brainstorming here, you could have a Company customization option
that lets them specify a backend preference order (including lists of
merged backends), and add a function to the c-a-p-f hook that chooses the
best applicable backend. (Though I guess this is essentially bypassing
c-a-p-f and substituting it with company-backends.)

My feeling is `completion-at-point-functions' is in principle general
enough to allow everything we want. But it's seriously creaking at the
seams. And it's only likely to get worse as we extend the PROPS argument
to cope with a richer set of UIs than the *Completions* buffer.

> > Nothing prevents us from providing
> > a "completion-at-point-merge-backends" function which takes a list of
> > completion-at-point-functions and returns a new completion-at-point-function.
> 
> Indeed. Although it seems to me that remembering where each candidate 
> came from (feature not yet present in Company either) would be harder to 
> implement, because completion functions are pretty much nameless as far 
> as the code using them is concerned. So propertizing the candidates with 
> (backend . company-elisp) won't work like it would in Company.
> 
> > company-backends and completion-at-point-functions don't work 100%
> > identically, but the differences are present only because of history.
> 
> I believe the main difference is "users customize it" vs. "users 
> generally don't know how to change it".

In Completion-UI, the "users don't change it" and "users customize it"
parts of the definitions are cleanly separated: the
`completion-ui-source-definitions' variable, which is modified by calling
the `completion-ui-register-source' macro from Elisp code to add a new
backend definition; and the `auto-completion-source-regexps',
`auto-completion-source-faces' and `auto-completion-source-functions'
variables, which have simple Customization interfaces.

c-a-p-f doesn't cleanly separate the things users might want to
customize, from the things only Elisp completion package authors should
have to deal with.

> > If Company had started with completion-at-point-functions it would
> > live very happily with it.
> 
> That's possible.

Or it would have been straining the limits of
completion-at-point-functions without company-backends to fall back on,
and c-a-p-f would have been extended until it looked more like
company-backends ;-)

T.
-- 
Dr T. S. Cubitt
Royal Society University Research Fellow
and Fellow of Churchill College, Cambridge
Centre for Quantum Information
DAMTP, University of Cambridge

email: tsc25@cantab.net
web:   www.dr-qubit.org

Re: Emacs completion matches selection UI

[Toby Cubitt]

On Sun, Jan 05, 2014 at 11:03:45PM -0500, Stefan Monnier wrote:
> > I agree.  The `completion-at-point-functions' API is convoluted compared
> > to either the Company-native API or the Completion-UI API.
> 
> Partly for historical reasons, but partly because it handles more
> complex kinds of completions, such as file-name completion with
> partial-completion.

Unless I've misunderstood what file-name completion with
partial-completion is, I'm pretty sure Completion-UI can either handle
this already with its non-prefix-completion sources, or these could
readily be extended to do so.

Completion-UI can handle completion sources that select an arbitrary
substring of the buffer to be 'completed', and substitute arbitrary text
for that string, whilst keeping track of what the original string was so
you can continue completing it.

> > On the other hand, it's already part of Emacs and is sufficiently
> > flexible (if a little ugly).  So we may be stuck with it now...
> 
> Extending the company API or the completion-ui API to handle things
> like filename completion (including the issues linked to handling
> quoting/unquoting) is far from obvious,

What's the roadblock here? I don't see it.

> so whether we like it or not, I think that
> completion-at-point-functions is not really negotiable.

I figured you'd say that :-/

T.
-- 
Dr T. S. Cubitt
Royal Society University Research Fellow
and Fellow of Churchill College, Cambridge
Centre for Quantum Information
DAMTP, University of Cambridge

email: tsc25@cantab.net
web:   www.dr-qubit.org

Re: Emacs completion matches selection UI

[Dmitry Gutov]

On 06.01.2014 09:35, Toby Cubitt wrote:
>> Extending the company API or the completion-ui API to handle things
>> like filename completion (including the issues linked to handling
>> quoting/unquoting) is far from obvious,
>
> What's the roadblock here? I don't see it.

I'd also like to read some specifics.

Re: Emacs completion matches selection UI

[Dmitry Gutov]

On 06.01.2014 06:39, Toby Cubitt wrote:
> I'm sure if you'd built Company on top of Completion-UI, we would have
> rapidly improved the APIs to support everything you needed as Company
> grew.

I'm just the maintainer. :) Company's author is Nikolaj Schumacher, 
though he's stepped away from its development.

> I originally thought I'd want to disable the learning algorithms once I'd
> trained Predictive on a corpus of my writing, for precisely the reason
> you describe. But in practice the order is stable enough that I never
> found it necessary.

I'll have to look into it.

> Completion-UI stores state by marking the location of the in-progress
> completion with an overlay, and storing data in overlay properties. You
> could alternatively use text properties, as you mention later. But I find
> overlays a better match for this kind of thing.

I mentioned text properties in the context of merged backends, so the 
properties would be on the candidate strings, not in the buffer. That's 
not interchangeable with using an overlay.

 > (Presumably Company does something similar.)

It does store some data, as an implementation detail, but in no way that 
data is sufficient for a front-end to handle a sequence of commands on 
its own. Like I said, the front-ends use the buffer-local vars. Multiple 
pending completions have never been a goal so far.

>> I'd still like to a natural way to extend such widgets with new
>> commands. Passing a keymap as argument, or rebinding a dynamic variable
>> that points to keymap works, I guess, but it's not very nice.
>
> Well, this is only meaningful for some of the widgets. Indeed, "widgets"
> is mislead; the completion UIs in Completion-UI aren't necessarily
> widgets at all. For example, there are no "commands" at all in the
> dynamic completion UI, nor is it a "widget" in any normal sense of the
> word.

I'd call "widgets" any UIs that encapsulate data (store it in the 
associated overlay, for example) and define interaction commands (e.g. 
have an overlay keymap).

> The UIs absolutely cannot be treated uniformly, because not all of them
> are "display a list of completions"-style UIs. E.g. the "dynamic
> completion" UI is a completely different beast.

That might be unfortunate from the API standpoint, making it more complex.

> Even amongst the "display a list of completions"-style UIs, the
> mouse-driven toolkit menus cannot be used in the terminal, nor can pop-up
> frames, nor can tooltips. That doesn't mean one should not provide these
> interfaces at all as options. It just means they probably aren't good
> default options.

The problem of "cannot be used in the terminal", at least, can be solved 
by doing a check at runtime once, and picking between a graphical and an 
overlay-based interface. As long as they provide similar interaction, 
I'd call that "treated uniformly".

>> Not really. `company-backends' corresponds directly to
>> `auto-completion-source',
>
> Not quite. `company-backends' corresponds to the
> `completion-ui-source-definitions' variable.

I'm inclined to stand by my previous assessment: nowhere in the code I 
see you iterate over `completion-ui-source-definitions' at runtime. In 
most places, it's used as a second argument to `assq', so it acts as a 
"source registry" of sorts, and we don't have a direct correspondence to 
that in Company.

The only place it's iterated over is in 
`completion-ui-customize-list-sources', which provides the list of 
possible values to customize `auto-completion-default-source' (in the 
Git version; I've been looking at the contents of the tarball previously).

>> only instead of one function we have a list of
>> them (which can also be nested one level deep, but that's an extra feature).
>
> Sure, Completion-UI also allows a list of completion sources and selects
> between them automagically. It even supports the same "extra feature" of
> nesting sources one level deep (Combined source, in Completion-UI
> terminology).

I couldn't find the word "combined" in completion-ui.el. Could you point 
at where it's used?

> But the API for this isn't auto-completion-default-source(*). That merely
> sets the default fall-back. The API for auto-selecting from a list of
> sources is provided by `auto-completion-source-regexps',
> `auto-completion-source-faces' and the `auto-completion-source-functions'
> hook. (The former two are in fact variables used by functions in the
> default `auto-completion-source-functions' definition.)

`auto-completion-source-functions' also could be thought of as similar 
to `company-backends' because both contain logic of picking a suitable 
source (the latter by the virtue of backends themselves knowing when 
they're suitable), but the list of default detection functions (based on 
text properties, overlays, faces at point) looks more limiting to me.

Granted, the first two source functions serve to support the explicit 
overriding of the used source (feature not presend in Company, but I've 
never seen a request for this so far), but the second two offer only 
choices based on regexp or face at point. Which is not terrible, but 
more limiting than Company's free-form expectation that 
(backend-function 'prefix) returns nil when it's not applicable at point 
in the current buffer.

You could say that I can add another function or several to 
`auto-completion-source-functions', but they won't be able to do 
anything smart with third-party sources, I'll have to take care about 
each source I might want to use, separately, in those functions.

> (*) I'm referring here to the git version of Completion-UI; the last
> tarball release - which is woefully outdated - used a slightly different
> API.

Thanks for mentioning this.

> Yup, Completion-UI does something very similar. Except that completion
> source (backend) selection isn't directly tied to the backend. It's
> configured independently. So if you need to, you can set up different
> conditions under which a particular backend will be used in different
> buffers, major-modes, minor-modes, or whatever.

Note that there's nothing stopping you from defining multiple conditions 
for when a particular Company backend can be used. Most do that already: 
the popular predicates are "does the major mode match this list", "is a 
particular package loaded", "is the parse state at point in-string or 
outside", "is this buffer visiting a file", "is a particular program 
present in the system", "is there a file with a given name in a parent 
directory" and "is point inside a symbol, after a symbol or outside of 
any symbols".

> Anyhow, I don't think company-mode is something that necessarily belongs
> in a generic Emacs completion UI. At most, Emacs should include the
> necessary definitions to hook the basic completion methods that come with
> Emacs into the new generic UI (dabbrev, etags, minibuffer completion,
> maybe a few others like elisp symbol completion...).

Maybe so.

> Indeed, I would have argued that we should *first* come up with a generic
> Emacs completion UI and API *without* including any mechanism for
> automatically selecting between completion sources.

In other words, a programmatic widget API. This has come up already in 
the discussion with Ted.

> Registering a source just (1) makes life more convenient: you set all the
> option values once when you define the completion source (backend), and
> then you can refer to that source and all its associated options using a
> single name (symbol). And (2) it makes the source available as an option
> in the Customization interface, as you noted.

I would think that, since Completion-UI is less user-facing and more 
about writing Elisp to wire sources and UIs, the Customize interface is 
considerably less important.

>> It makes matters a bit worse for third-party backends, but not by much,
>> I think.
>
> I agree, this is a minor difference, and Company doesn't lose much by it,
> especially since it bundles almost all the backend definitions you're
> ever likely to want.

We also document how to use `company-backends' and, for third-party 
packages, how to add a specific backend to it. But it's a simple data 
structure, so for users with some experience just knowing backend 
function names is sufficient.

> The optional macro arguments only come into play if you're doing
> something more sophisticated (like extending the default completion
> widgets, or overriding the default method of determining the text around
> point that's to be completed, etc.)

Most Company backends at least have to check that the buffer is in 
matching major mode.

>> This pattern is used in many places in Emacs core, too.
>
> In principle, this sounds like a perfectly reasonable API choice for
> simple settings.
>
> Though I don't completely get how it works here. What if a particular
> completion function can't answer some types of question?

Then we use the default setting, behavior, etc. It is somewhat limiting 
(we can't discern whether a backend does not know about a question or it 
consciously returns nil), but so far it's worked well nevertheless.

 > E.g. maybe it
> can expand prefixes, but it can't find regexp matches. What does the
> completion function do if it's then asked for a regexp match? Or does the
> backend answer "I can't answer this type of question" and Company moves
> onto the next backend in the list?

We don't have support for regexp matches, but "can't answer" + "move on" 
is a reasonable mechanism to support them in select backends.

Or, to approach it another way, it may be a backend's choice to 
interpret the prefix (or, more generally, the completion string, if it's 
allowed to span after the point) as a regexp. For example, we have this 
as a feature request now: 
https://github.com/company-mode/company-mode/issues/45#issuecomment-31564029

Aside from handling the prefix->candidate replacement more carefully, 
the only thing we'll have to worry about, I believe, is that the 
front-ends won't be so sure that candidate starts with "prefix". For 
`company-pseudo-tooltip', that means no highlighting of the "common" 
part (or using a more complex algorithm), and it could mean different 
things for `company-preview-if-just-one-frontend'. For now, I'm inclined 
to just refrain from using it if the sole candidate does not start with 
"prefix".

> However, this API is not so convenient if you want to pass more complex
> settings, such as extending the UI for a particular source. You'd either
> have to pass the same list of hook functions, keymaps etc. every time you
> call the function that invokes the completion UI for the source. Or you'd
> have to let-bind the same set of variables every time. Which would be a
> pain.

Sorry, I don't understand. Why would you extend a UI for some source? 
Sources provide data, how can they have associated keymaps?

> This pattern is used in many places in Emacs core, too: `defcustom',
> `defstruct', `defun*'... (Perhaps I should have called the macro
> `defcompletion' instead of `completion-ui-register-source' :)

Sure, Lisps have a long history of using and promoting macros, but 
there's also a general recommendation to not write a new macro unless 
you'll really really benefit from it. Maybe you do.

> Also, the "pass the type of completion as the first argument" API means
> you have a hard-coded set of "types" of completion. As usual, since it
> was intended to be a generic completion Elisp library and not a
> fully-fledged user-level completion framework, Completion-UI doesn't
> impose any such restriction, but tries to be as flexible and general as
> possible. I'm not saying this is necessarily a good thing; it could well
> be that this flexibility is overkill.

See above. We only have prefix completion for now, so the first argument 
definitely doesn't choose the type. See the list of possible values in 
the `company-backends' docstring.

> Which bodes well for coming up with a generic Emacs completion UI API and
> code. If we've converged on a broadly similar set of features and APIs
> for the UI parts of Completion-UI and Company, there's a reasonable
> chance they're the features people will want from a generic Emacs API.

Yes, probably.

Re: Emacs completion matches selection UI

[Stefan Monnier]

>> Extending the company API or the completion-ui API to handle things
>> like filename completion (including the issues linked to handling
>> quoting/unquoting) is far from obvious,
> What's the roadblock here? I don't see it.

I know.  That's why it's non-obvious.


        Stefan

Re: Emacs completion matches selection UI

[Toby Cubitt]

On Mon, Jan 06, 2014 at 11:36:49AM +0400, Dmitry Gutov wrote:
> On 06.01.2014 06:39, Toby Cubitt wrote:
> > Completion-UI stores state by marking the location of the in-progress
> > completion with an overlay, and storing data in overlay properties. You
> > could alternatively use text properties, as you mention later. But I find
> > overlays a better match for this kind of thing.
> 
> I mentioned text properties in the context of merged backends, so the 
> properties would be on the candidate strings, not in the buffer. That's 
> not interchangeable with using an overlay.

OK, I understand what you meant now.

>  > (Presumably Company does something similar.)
> 
> It does store some data, as an implementation detail, but in no way that 
> data is sufficient for a front-end to handle a sequence of commands on 
> its own.

It's still Completion-UI that handles a sequence of commands, not the
backend. Completion-UI stores enough data about the currently active
completion process in the overlay properties to know what to do when the
user calls a follow-up command.

Sounds like Company stores that data in local variables, whereas I chose
to store it in overlay properties.

> Like I said, the front-ends use the buffer-local vars. Multiple pending
> completions have never been a goal so far.

It's a nice freebie you get from storing the data in overlays instead of
buffer-local variables, but I don't think it's all that important.

> >> I'd still like to a natural way to extend such widgets with new
> >> commands. Passing a keymap as argument, or rebinding a dynamic variable
> >> that points to keymap works, I guess, but it's not very nice.
> >
> > Well, this is only meaningful for some of the widgets. Indeed, "widgets"
> > is mislead; the completion UIs in Completion-UI aren't necessarily
> > widgets at all. For example, there are no "commands" at all in the
> > dynamic completion UI, nor is it a "widget" in any normal sense of the
> > word.
> 
> I'd call "widgets" any UIs that encapsulate data (store it in the 
> associated overlay, for example) and define interaction commands (e.g. 
> have an overlay keymap).

OK.

> > The UIs absolutely cannot be treated uniformly, because not all of them
> > are "display a list of completions"-style UIs. E.g. the "dynamic
> > completion" UI is a completely different beast.
> 
> That might be unfortunate from the API standpoint, making it more complex.

Have you looked at the Completion-UI API for defining a new widget? It's
uniform across all the widgets. Has to be, otherwise there would be no
way of allowing it to be extended with third-party widgets (see my
comment below).

What I meant was that the UIs themselves can act very differently. One
shouldn't make assumptions that they all behave in a similar way. The API
itself is uniform.

> > Even amongst the "display a list of completions"-style UIs, the
> > mouse-driven toolkit menus cannot be used in the terminal, nor can pop-up
> > frames, nor can tooltips. That doesn't mean one should not provide these
> > interfaces at all as options. It just means they probably aren't good
> > default options.
> 
> The problem of "cannot be used in the terminal", at least, can be solved 
> by doing a check at runtime once, and picking between a graphical and an 
> overlay-based interface. As long as they provide similar interaction, 
> I'd call that "treated uniformly".
> 
> >> Not really. `company-backends' corresponds directly to
> >> `auto-completion-source',
> >
> > Not quite. `company-backends' corresponds to the
> > `completion-ui-source-definitions' variable.
> 
> I'm inclined to stand by my previous assessment: nowhere in the code I 
> see you iterate over `completion-ui-source-definitions' at runtime.

You're insisting on trying to match the Completion-UI API one-to-one with
the Company API. But they don't match one-to-one, because they divide
things up slightly differently. Nonetheless, it's definitely wrong to
claim that company-backends corresponds to
auto-completion-default-source. The best comparison one can make is that
company-backends loosely corresponds to a mixture of
completion-ui-source-definitions and auto-completion-source-functions.

> In most places, it's used as a second argument to `assq', so it acts as
> a "source registry" of sorts, and we don't have a direct correspondence
> to that in Company.

That's because Company conflates the backend definitions with the
mechanism for auto-selecting a backend. Completion-UI keeps those
separate. It makes sense to me to separate these, for two reasons.
(1) the backend definitions will typically be written by elisp package
authors; the auto-selection configuration is something users will want to
customize. (2) I see no reason to limit each source of completions
(dabbrevs, etags...) to a single choice of when to use it.

By combining both the choice of whether to use a backend (nil or non-null
return value) and finding the list of completions (return value) into one
function, you force all the selection logic to be lumped in together with
the completion logic. If you want to use the same source of completions
(dabbrevs, etags...) in multiple different contexts, you either have to
code *all* the selection logic you'll ever need into the one backend, or
have multiple backends for the same source of completions.

I find it logically much cleaner to separate the function for completing
a string using a given source (dabbrevs, etags...), from the function for
selecting which source of completions to use.

> >> only instead of one function we have a list of them (which can also
> >> be nested one level deep, but that's an extra feature).
> >
> > Sure, Completion-UI also allows a list of completion sources and selects
> > between them automagically. It even supports the same "extra feature" of
> > nesting sources one level deep (Combined source, in Completion-UI
> > terminology).
> 
> I couldn't find the word "combined" in completion-ui.el. Could you point 
> at where it's used?

The `completion-ui-combining-complete' utility function, and the
pre-defined Combine completion source +
`completion-ui-combine-sources-alist' customization option.

Remember: Completion-UI, aimed primarily at elisp coders, not
users. `completion-ui-combining-complete' lets you write a completion
function that combines multiple sources together. The pre-defined Combine
completion source + `completion-ui-combine-sources-alist' customization
option mainly serves as a demonstration.

This is *definitely* not as simple or user-friendly as Company. Partly
because I think of combining multiple completion sources as something
that belongs in a user-level completion framework (like Company) built on
top of Completion-UI. Partly because my use case (Predictive) needed a
very flexible way of switching *between* different completion sources,
but for algorithmic reasons combining multiple sources needed to be done
within Predictive, not within Completion-UI.

I hate to say this ;-), but maybe `completion-at-point-functions' got
this right. By having one function (the hook function itself) that deals
with selecting a completion source, and a separate function (the hook
function's return value) for actually doing the completion. This both
keeps the two jobs cleanly separated (as in Completion-UI), *and* keeps
the two together in the same list.

Personally I'd find an alist with elements of the form
(collection . test) a slightly clearer API than a straight hook. But I
can see why using a hook appeals from a core Emacs perspective.


> > But the API for this isn't auto-completion-default-source(*). That merely
> > sets the default fall-back. The API for auto-selecting from a list of
> > sources is provided by `auto-completion-source-regexps',
> > `auto-completion-source-faces' and the `auto-completion-source-functions'
> > hook. (The former two are in fact variables used by functions in the
> > default `auto-completion-source-functions' definition.)
> 
> `auto-completion-source-functions' also could be thought of as similar 
> to `company-backends' because both contain logic of picking a suitable 
> source (the latter by the virtue of backends themselves knowing when 
> they're suitable), but the list of default detection functions (based on 
> text properties, overlays, faces at point) looks more limiting to me.

Why on earth is it limiting?

The detection functions (based on regexps, text properties, overlays,
faces at point) are provided for convenience, so you don't have to write
your own `auto-completion-source-functions' function and duplicate a
bunch of code if all you want to do is select a source based on regexp
matches, text properties, overlay properties, or faces. (Together with
the standard Emacs mechanisms for making this buffer-local, this already
covers many of the common cases.)

If you need a more complex selection logic, write a new selection
function which does whatever you want, and add it to
`auto-completion-source-functions'. (And remove the default regexp, face,
text-property etc. detection functions from the hook if you don't want
them.)

I really don't see how this is any different to writing a new Company
backend with the particular selection logic you require, and adding it to
company-backends. Except that you *only* need to code the new selection
function and add it to the auto-completion-source-functions hook; you
don't need to duplicate the code for finding the completions if all you
want to do is use an existing completion source in a new context.

> Granted, the first two source functions serve to support the explicit 
> overriding of the used source (feature not presend in Company, but I've 
> never seen a request for this so far),

I make heavy use of this in Predictive's sophisticated LaTeX and Texinfo
support. If I were to use Company, I would have to move all that logic
into Predictive, because Company doesn't provide it for me.

> but the second two offer only choices based on regexp or face at
> point. Which is not terrible, but more limiting than Company's
> free-form expectation that (backend-function 'prefix) returns nil when
> it's not applicable at point in the current buffer.

I don't get this at all. How is adding a free-form function to the
`auto-completion-source-functions' hook any less flexible than adding a
new backend to company-backends? Except that if you simply want to choose
the completion source based on a regexp match, face, text-property or
overlay-property, then you don't even have to write any new code (as you
would in Company) but can get away with simply configuring a variable.

I think you're trying to view Completion-UI through Company
spectacles. Some Company features don't exist of have limited support in
Completion-UI (e.g. combining sources), some Completion-UI features don't
exist or have limited support in Company (e.g. adding new custom UIs),
and some features exist in both but work differently. But Completion-UI's
programmatic selection of completion sources is just as powerful as
Company's.


> You could say that I can add another function or several to 
> `auto-completion-source-functions', but they won't be able to do 
> anything smart with third-party sources, I'll have to take care about 
> each source I might want to use, separately, in those functions.

Why on earth not?! They can do whatever they want - they're functions!
(OK, anything short of solving the Halting Problem ;-)

To belabour the point, Completion-UI separates the task of completion
source selection from the task of finding completions. To add the
equivalent of a new Company backend to Completion-UI, you would typically
add a new completion source, and add a new
`auto-completion-source-functions' hook function. (Unless one of the
pre-defined selection functions was sufficient, in which case you could
get away with just adding an entry to the appropriate customization
variable.)


> > Yup, Completion-UI does something very similar. Except that completion
> > source (backend) selection isn't directly tied to the backend. It's
> > configured independently. So if you need to, you can set up different
> > conditions under which a particular backend will be used in different
> > buffers, major-modes, minor-modes, or whatever.
> 
> Note that there's nothing stopping you from defining multiple conditions 
> for when a particular Company backend can be used. Most do that already: 
> the popular predicates are "does the major mode match this list", "is a 
> particular package loaded", "is the parse state at point in-string or 
> outside", "is this buffer visiting a file", "is a particular program 
> present in the system", "is there a file with a given name in a parent 
> directory" and "is point inside a symbol, after a symbol or outside of 
> any symbols".

Sure, but you have to duplicate the same logic in each backend function,
and you have to either lump *all* the selection logic you'll ever need
into one function, or have multiple backends for the same completion
source.

Keeping them separate is, to my mind, both cleaner and more
flexible. (It's also the route c-a-p-f went.)

> > Indeed, I would have argued that we should *first* come up with a generic
> > Emacs completion UI and API *without* including any mechanism for
> > automatically selecting between completion sources.
> 
> In other words, a programmatic widget API. This has come up already in 
> the discussion with Ted.

Yes, if what you mean is an API for defining completion UI widgets,
together with a bundle of standard widgets. As far as new widget authors
are concerned, it would be a completion widget API.

But let's be clear: as far as *completion* package authors are concerned,
it's not a widget API at all. A completion package won't call the tooltip
widget, or the popup.el widget, or the dynamic completion widget. It will
call something like `completion-at-point'.

At least, that's how I was imagining it.

> > Registering a source just (1) makes life more convenient: you set all the
> > option values once when you define the completion source (backend), and
> > then you can refer to that source and all its associated options using a
> > single name (symbol). And (2) it makes the source available as an option
> > in the Customization interface, as you noted.
> 
> I would think that, since Completion-UI is less user-facing and more 
> about writing Elisp to wire sources and UIs, the Customize interface is 
> considerably less important.

I strongly disagree. Almost all of the Completion-UI customization
options are about letting the user customize the UI. Those are both
important, and definitely belong together with an Emacs generic
completion UI.

The programmatic completion source (backend) selection interface would be
less important...except that it's already there in Emacs in the form of
`completion-at-point-functions'. We're probably stuck with c-a-p-f for
now, so any generic Emacs completion UI is going to have to integrate
with it somehow.

There are really two APIs here:

1. An API for defining new completion widgets, a customization interface
   to let the user configure them, and code to hook the new UIs into
   `completion-at-point', `completion-in-region', and eventually
   minibuffer completion and the rest.

2. An API for defining new completion sources and new source selection
   logic, and a customization interface to let the user configure
   them. This API already exists (c-a-p-f et al.), and the new UI code
   will have to integrate with it.


> >> It makes matters a bit worse for third-party backends, but not by much,
> >> I think.
> >
> > I agree, this is a minor difference, and Company doesn't lose much by it,
> > especially since it bundles almost all the backend definitions you're
> > ever likely to want.
> 
> We also document how to use `company-backends' and, for third-party 
> packages, how to add a specific backend to it. But it's a simple data 
> structure, so for users with some experience just knowing backend 
> function names is sufficient.

Sure, that's equally true of `completion-at-point-functions' and
`completion-ui-register-source'. (Except that new backends are
first-class citizens in Completion-UI, and show up automatically as
options in the completion UI customization interface where
appropriate. This would be harder to replicate with
`completion-at-point-functions', but I suspect with enough Customize
macro trickery it would still be possible with the existing Emacs
completion backend API.)

> > The optional macro arguments only come into play if you're doing
> > something more sophisticated (like extending the default completion
> > widgets, or overriding the default method of determining the text
> > around point that's to be completed, etc.)
> 
> Most Company backends at least have to check that the buffer is in 
> matching major mode.

Whereas in Completion-UI, you just use the standard Emacs mechanisms of
setting buffer-local variables from you major-mode hook, or using
`add-hook' with the LOCAL flag set to add a function to the
auto-completion-source-functions hook.

I prefer to use the standard, tried-and-tested Emacs mechanisms where
possible (major-mode hooks), rather than creating an entirely new
package-specific way of configuring things for different major-modes. (I
don't always stick to this myself, though I probably should ;-)


> >> This pattern is used in many places in Emacs core, too.
> >
> > In principle, this sounds like a perfectly reasonable API choice for
> > simple settings.
> >
> > Though I don't completely get how it works here. What if a particular
> > completion function can't answer some types of question?
> 
> Then we use the default setting, behavior, etc. It is somewhat limiting 
> (we can't discern whether a backend does not know about a question or it 
> consciously returns nil), but so far it's worked well nevertheless.

Another reason to separate source selection logic from finding
completions logic.

In any case, this discussion is somewhat moot. It's pretty clear we're
stuck with the c-a-p-f API for core Emacs, and that API already separates
source selection from completion.

>  > E.g. maybe it can expand prefixes, but it can't find regexp
> > matches. What does the completion function do if it's then asked for
> > a regexp match? Or does the backend answer "I can't answer this type
> > of question" and Company moves onto the next backend in the list?
> 
> We don't have support for regexp matches, but "can't answer" + "move on" 
> is a reasonable mechanism to support them in select backends.

So every backend has to look at what's being asked of it every time it's
invoked, and return nil if it's asked to do something it doesn't support?
And also check the major-mode and see if it's appropriate every time it's
invoked, even though the major-mode is very unlikely to change (and runs
a major-mode hook when it does)?

This sounds rather inefficient. I'm sure it's fast enough for Company,
but I'm skeptical it would be fast enough for predictive-mode's
auto-completion-mode (where tenths of a second matter).


> Or, to approach it another way, it may be a backend's choice to 
> interpret the prefix (or, more generally, the completion string, if it's 
> allowed to span after the point) as a regexp. For example, we have this 
> as a feature request now: 
> https://github.com/company-mode/company-mode/issues/45#issuecomment-31564029

>From experience with Completion-UI (which already supports
pattern-expansion-style completion), you have to slightly change the way
the UI works depending on whether you're doing prefix completion or
pattern expansion. The Completion-UI widgets already support this.

> Aside from handling the prefix->candidate replacement more carefully, 
> the only thing we'll have to worry about, I believe, is that the 
> front-ends won't be so sure that candidate starts with "prefix". For 
> `company-pseudo-tooltip', that means no highlighting of the "common" 
> part (or using a more complex algorithm), and it could mean different 
> things for `company-preview-if-just-one-frontend'. For now, I'm inclined 
> to just refrain from using it if the sole candidate does not start with 
> "prefix".

Non-prefix completion is already fully supported by all the Completion-UI
widgets and interfaces.

> > However, this API is not so convenient if you want to pass more complex
> > settings, such as extending the UI for a particular source. You'd either
> > have to pass the same list of hook functions, keymaps etc. every time you
> > call the function that invokes the completion UI for the source. Or you'd
> > have to let-bind the same set of variables every time. Which would be a
> > pain.
> 
> Sorry, I don't understand. Why would you extend a UI for some source? 
> Sources provide data, how can they have associated keymaps?

Maybe an example will help. When completing a LaTeX command name,
Predictive uses a custom menu and completion browser specifically
tailored to LaTeX commands. Similarly, it uses differently tailored
versions for completing LaTeX environment names, colour names, etc. etc.

It's not that the source returns a keymap or widget-building function
each time it's called. The keymaps or functions that tweak the UI for a
specific completion source are configured when the source is registered
with Completion-UI. The completion function itself just returns
completions (and associated data).

Hope that makes it clearer.


> > This pattern is used in many places in Emacs core, too: `defcustom',
> > `defstruct', `defun*'... (Perhaps I should have called the macro
> > `defcompletion' instead of `completion-ui-register-source' :)
> 
> Sure, Lisps have a long history of using and promoting macros, but 
> there's also a general recommendation to not write a new macro unless 
> you'll really really benefit from it. Maybe you do.

Yes, I really, really benefit from using a macro.

It would be virtually impossible to do everything
`completion-ui-register-source' does without using a macro. Inspecting
the completion function's definition to determine how to call it?
Automatically updating the Customization definitions? Defining a new
source-specific function? Automatically wrapping the completion function
to make it predictive (strictly speaking, this part lives in Predictive,
not Completion-UI)?

I don't even want to contemplate how to do those without lisp macros.


> > Also, the "pass the type of completion as the first argument" API means
> > you have a hard-coded set of "types" of completion. As usual, since it
> > was intended to be a generic completion Elisp library and not a
> > fully-fledged user-level completion framework, Completion-UI doesn't
> > impose any such restriction, but tries to be as flexible and general as
> > possible. I'm not saying this is necessarily a good thing; it could well
> > be that this flexibility is overkill.
> 
> See above. We only have prefix completion for now, so the first argument 
> definitely doesn't choose the type. See the list of possible values in 
> the `company-backends' docstring.

Thanks, makes sense now.

> > Which bodes well for coming up with a generic Emacs completion UI API and
> > code. If we've converged on a broadly similar set of features and APIs
> > for the UI parts of Completion-UI and Company, there's a reasonable
> > chance they're the features people will want from a generic Emacs API.
> 
> Yes, probably.

To summarise:

1. People want an Emacs completion UI widget API, mostly so they can have
   a more "modern" completion UI than the *Completions* buffer. As far as
   I know, Completion-UI is the only package that provides an API and for
   defining new completion widgets, and code to bind them all
   together. It also already provides a fairly large collection of UI
   widgets.

   Maybe we should look at the Completion-UI API for this
   (`completion-ui-register-interface'), and think about how it would
   need to be changed for core Emacs? I'm not wedded to the existing API
   if someone proposes something better or has good arguments for
   changing it.

2. We're probably stuck with c-a-p-f et al. for the completion source and
   source-selection API. How would this need to be extended to integrate
   it with a new Emacs completion UI? How could it be extended to better
   support Company and others?

3. The most popular and useful "list-the-available-completions" UI is
   popup.el. Is there any chance of getting copyright assignment for
   this? Or will we be forced to code something equivalent from scratch?


Best,
Toby
-- 
Dr T. S. Cubitt
Royal Society University Research Fellow
and Fellow of Churchill College, Cambridge
Centre for Quantum Information
DAMTP, University of Cambridge

email: tsc25@cantab.net
web:   www.dr-qubit.org

Re: Emacs completion matches selection UI

[Toby Cubitt]

On Mon, Jan 06, 2014 at 09:47:49AM -0500, Stefan Monnier wrote:
> >> Extending the company API or the completion-ui API to handle things
> >> like filename completion (including the issues linked to handling
> >> quoting/unquoting) is far from obvious,
> > What's the roadblock here? I don't see it.
> 
> I know.  That's why it's non-obvious.

Witty, but not very illuminating!

If you say it would be tough to support filename completion (I assume
we're talking minibuffer filename completion here?), I of course trust
your judgement. But I'd be interested to understand why.

Any pointers?

Toby
-- 
Dr T. S. Cubitt
Royal Society University Research Fellow
and Fellow of Churchill College, Cambridge
Centre for Quantum Information
DAMTP, University of Cambridge

email: tsc25@cantab.net
web:   www.dr-qubit.org

Re: Emacs completion matches selection UI

[Stefan Monnier]

>> >> Extending the company API or the completion-ui API to handle things
>> >> like filename completion (including the issues linked to handling
>> >> quoting/unquoting) is far from obvious,
>> > What's the roadblock here? I don't see it.
>> I know.  That's why it's non-obvious.
> Witty, but not very illuminating!

I don't really know how to illuminate here, sadly.
I did go through the trouble of getting these things to work in
minibuffer.el, but I don't have a good insight to provide as to why it
was tricky.


        Stefan

Re: Emacs completion matches selection UI

[Ted Zlatanov]

On Sat, 04 Jan 2014 09:21:33 +0000 joaotavora@gmail.com (João Távora) wrote: 

JT> Ted Zlatanov <tzz@lifelogs.com> writes:
>> Based on the discussion so far, it seems that:
>> 
>> [...]
>> - it should be possible to implement a generic frontend for completion
>> that both can use, and other tools like yasnippet can use IIUC.  The

JT> Thanks for the summary of the discussion so far, but this last statement
JT> might bring some confusion... Yasnippet can curiously fit into many
JT> parts of the problem:

JT> * It can be a source to completion frameworks, auto-complete and company
JT>   have such sources.

Right.  A "completion backend" I think is the current term.

JT> * It prompts the user when it finds more than one snippet for an abbrev,
JT>   and uses its own `yas-prompt-functions' abstraction. Some of the
JT>   functions offer completion.

This is a "completion frontend" IIUC.  yasnippet.el can have its own
prompt functions if they work with the larger completion API (as
followup queries), otherwise it's probably best to avoid integrating
Yasnippet as I described.

JT> * It installs a keybinding for TAB which is problematic because it is
JT>   very sought-after binding for completion-related stuff. The parts of
JT>   this discussion that concerned yasnippet dealed with this, AFAIK.

Yes, this is a fun debate.  I'd like to leave it up to the user and make
the default conservative, but it should be context-dependent and easy to
augment by a package like Yasnippet, IMHO.

>> with their experience of writing it once already, and we have to make
>> sure we don't end up with a solution no one wants to use :)

JT> Yep. For me personally it would make sense to evaluate alternatives
JT> based a matrix like the one proposed by Stefan earlier:

JT> http://article.gmane.org/gmane.emacs.devel/166661/match=completion+ui

OK, but I don't think a matrix can cover all the subtleties.  I think
fitting Emacs' current completion cases, company-mode, and
completion-ui+Predictive into a single framework would be a good start
and then you can see if Yasnippet can fit there.  Maybe it can't without
a complete refit, and in that case we leave it alone.  (I think Helm
will also be in this category, its UI is too integrated and is not
really a completion framework in the traditional sense.)

Ted

Re: Emacs completion matches selection UI

[Ted Zlatanov]

On Sat, 4 Jan 2014 00:48:53 +0000 Toby Cubitt <tsc25@cantab.net> wrote: 

TC> My feeling is once people decide what it is they want included in
TC> Emacs, and what the API should look like, re-purposing (parts of)
TC> the Company and/or Completion-UI code bases (maybe others too?) will
TC> be a big head start (assuming no copyright assignment roadblocks).

Yes, exactly.  Those are, in my mind, two cases that should work with
the API in order to prove its viability, and then we can try to fit
others.

>> - it should be possible to implement a generic frontend for completion
>> that both can use, and other tools like yasnippet can use IIUC.  The
>> eventual goal is to have "something" standard in Emacs that all such
>> packages (except perhaps helm, which is completely apart in its UI)
>> would use.  We have to ask Dmitry and Toby, among others, to guide us
>> with their experience of writing it once already, and we have to make
>> sure we don't end up with a solution no one wants to use :)

TC> Indeed. It would be nice to get input from the other completion package
TC> authors too (anything.el, auto-complete, icicles...)

anything.el became Helm and Thierry has been doing a great job
maintaining it.  I think he follows emacs-devel but as I mentioned I
have my doubts that Helm will fit any completion API.  It's very
integrated.  Maybe Thierry wants to say?

TC> Since it's the completion UI that will have to interpret and make use of
TC> the extras, one place to start is to think about what additional features
TC> the UI might want to provide, beyond selecting from a list of possible
TC> completions.

TC> Luckily, we have many existing completion frameworks to draw on for
TC> inspiration. At the moment, location data and various forms of
TC> documentation (e.g. type information in code completion, docstrings,
TC> etc.) are the only ones that spring to mind.

Add: feedback from frontend to backend (Predictive use case we
mentioned, needs API support through callbacks).  Pluggable completion
frontends (not in the API).  Dynamic limiting of completion candidates
as characters are typed (not in the API).

The backend should be able to provide hints about the data.  For
instance, "this is a list of ELisp symbols" could look different from
"this is a list of installable packages."  Also icons are important IMO,
based on the hints or the candidates.

TC> The keyword arguments to `completion-ui-register-source' are one starting
TC> point for seeing what additional properties might be needed, as are the
TC> corresponding APIs in Company, auto-complete, etc.

Can you or Dmitry compile a list?

TC> Also, if we want to allow users to optionally customize the completion UI
TC> per-backend (a la Completion-UI), there has to be some way of identifying
TC> backends in Customize. This seems necessary to me if the new API is to
TC> become the standard way of implementing a completion UI in Emacs. Think
TC> minibuffer completion, versus filename completion, versus predictive
TC> completion, versus elisp code completion etc, all of which might want to
TC> use slightly different UI options. Again, it might be enough to support a
TC> :name property (or some such) in the `completion-at-point-functions'
TC> PROPS plist.

The frontend UI should depend (in my mind) on three things: 1) the
invocation context as you showed; 2) the backend-provided hints about
the data as a whole; and 3) the backend-provided hints about each
completion candidate individually.  Does that make sense?  Some
frontends may ignore some or all of those things, but I think those are
the principal axis.  I don't know how those three things will be passed
to the frontend though, and when.

>> - the frontend choices made by the user should be able to feed back into
>> the data backend (the "predictive" use case)

TC> A few optional hooks/call-backs would very likely be sufficient. Again,
TC> the `completion-at-point-functions' PROPS plist would be the obvious
TC> place to specify them.

OK, this is your use case :)

>> - we have enough interest and support in this work that it's worth
>> undertaking in earnest after the code freeze.  The discussion should
>> remain on emacs-devel (based on the wide interest so far).
...
TC> (Note that I'm unlikely to have time to code anything myself until
TC> March/April, although the feature freeze will likely delay things until
TC> about then anyway. I'm happy to help in a more limited way before then,
TC> as time allows.)

Cool.  We seem to be moving towards a specific API, refining the current
Emacs internals, which is good.  I'd be OK with it looking like
completion-ui as long as it was at least somewhat backwards compatible
and could support the packages we mentioned.  To me, the clearest
indication of a good approach would be if both the Emacs core and
external packages were able to lose lines of code.

Ted

Re: Emacs completion matches selection UI

[Ted Zlatanov]

On Sun, 05 Jan 2014 23:03:45 -0500 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 

SM> Extending the company API or the completion-ui API to handle things like
SM> filename completion (including the issues linked to handling
SM> quoting/unquoting) is far from obvious, so whether we like it or not,
SM> I think that completion-at-point-functions is not really negotiable.

In a hypothetical new "completion data backend" world, this would
essentially be the default data backend for all Emacs modes, right?
Then the user or packages or modes can enable more or alternate
completion backends as needed, augmenting or overriding that default.

Ted

Re: Emacs completion matches selection UI

[Ted Zlatanov]

On Sun, 05 Jan 2014 11:11:00 -0500 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 

>>>> (define-fallbackish-command maybe-foo (kbd "[somekey]") ()
>>> I don't understand what's the intended relation between [somekey] and
>>> `maybe-foo'.
>> Sorry, I should have provided some implementation, or made this more
>> clear. I
>> The generated part of the `maybe-foo' would do two things:
>> * register the association between maybe-foo and [somekey] in a some
>> global variable, pushing in front of any existing ones also defined
>> with the same macro. This keeps track of the "fallbackish commands"
>> (horrible name, sorry) for [somekey]
>> * bind maybe-foo to [somekey] in the (global) keymap, maybe replacing
>> someone else's binding.

SM> We could do the above fairly easily with

SM>   (add-function :around (lookup-key global-map [somekey])
SM>                 (lambda (orig &rest args)
SM>                   (if (foop) (foo) (apply orig args))))

Please, no.  It's a real pain to debug this with a user.

>> I have no idea how it would work with multiple keymaps.

SM> Right, that's where it gets more interesting.  E.g. some major mode
SM> might want to extend the meaning of TAB, but it should do it in the
SM> major mode's map, not in the global map.

I think the mykie.el approach has potential here ;)

Ted

Re: Emacs completion matches selection UI

[Dmitry Gutov]

On 06.01.2014 19:51, Toby Cubitt wrote:
> Have you looked at the Completion-UI API for defining a new widget? It's
> uniform across all the widgets. Has to be, otherwise there would be no
> way of allowing it to be extended with third-party widgets (see my
> comment below).

Looking at it now. Seems reasonable, but the data is somewhat coupled to 
the rest of Completion-UI. Examples:

1. `prefix-replaced' and `common-substring' both seem to be tailored to 
the dynamic interface.
2. `completion-source' seems to have to refer to a Completion-UI source, 
right?

So how would one use an UI defined as such, in a different package? 
Would Company define itself as a new source? Or add a source per each 
backend?

> What I meant was that the UIs themselves can act very differently. One
> shouldn't make assumptions that they all behave in a similar way. The API
> itself is uniform.

It's the same in Company. All four base stock frontends are different. 
One displays completions in a dropdown, another shows the (sole) 
completion inline in the buffer, and the other two use the echo area.

> The best comparison one can make is that
> company-backends loosely corresponds to a mixture of
> completion-ui-source-definitions and auto-completion-source-functions.

The latter: yes, the former: no. It's never used the same way as you use 
`completion-ui-source-definitions'.

>> In most places, it's used as a second argument to `assq', so it acts as
>> a "source registry" of sorts, and we don't have a direct correspondence
>> to that in Company.
>
> That's because Company conflates the backend definitions with the
> mechanism for auto-selecting a backend. Completion-UI keeps those
> separate. It makes sense to me to separate these, for two reasons.
> (1) the backend definitions will typically be written by elisp package
> authors; the auto-selection configuration is something users will want to
> customize.

IME, it makes sense in practice because there usually is a set of 
conditions when a backend is applicable, and there's no sense for the 
user to write them out, when the package author can anticipate them.

> (2) I see no reason to limit each source of completions
> (dabbrevs, etags...) to a single choice of when to use it.

The package author can always add one or several new of customization 
variables when the users request it to make the logic more flexible. 
Most of the time, it's usually just the list of major modes the backend 
applies to, if it's major mode-sensitive.

Please take a look at `company-etags' and `company-dabbrev' and see if 
you can point out the situations when the user might find the `prefix' 
code of either too limiting.

> By combining both the choice of whether to use a backend (nil or non-null
> return value) and finding the list of completions (return value) into one
> function, you force all the selection logic to be lumped in together with
> the completion logic. If you want to use the same source of completions
> (dabbrevs, etags...) in multiple different contexts, you either have to
> code *all* the selection logic you'll ever need into the one backend, or
> have multiple backends for the same source of completions.

Or you can define new backends that would do some common checks in 
`prefix' (maybe calling an extracted function with common code) and 
simply delegate all other actions to the respective base backend. 
Implementing this is trivial.

> I find it logically much cleaner to separate the function for completing
> a string using a given source (dabbrevs, etags...), from the function for
> selecting which source of completions to use.

I understand the principle, really. But the more one "cleanly separates" 
code, the harder it can be sometimes to read it, to get the full picture.

> I hate to say this ;-), but maybe `completion-at-point-functions' got
> this right. By having one function (the hook function itself) that deals
> with selecting a completion source, and a separate function (the hook
> function's return value) for actually doing the completion. This both
> keeps the two jobs cleanly separated (as in Completion-UI), *and* keeps
> the two together in the same list.

It makes certain amount of sense, although it looks like it could make 
creating a "merged" completion function more difficult. We'll see.

But again, this separation also makes things more opaque for the user.

>> `auto-completion-source-functions' also could be thought of as similar
>> to `company-backends' because both contain logic of picking a suitable
>> source (the latter by the virtue of backends themselves knowing when
>> they're suitable), but the list of default detection functions (based on
>> text properties, overlays, faces at point) looks more limiting to me.
>
> Why on earth is it limiting?

Note the word "default" above. This specific list is limiting.

But the general approach, while flexible on its surface, complicates 
things if I intend to use any existing sources, written by third 
parties. Because their authors are unlikely to have anticipated the 
logic I'll add in my custom predicate function and to have written any 
code in their packages I might use. Or, at least, that's considerably 
less likely.

Granted, if I only use Completion-UI for its interfaces and never for 
any existing sources, it's not going to be a problem.

> I really don't see how this is any different to writing a new Company
> backend with the particular selection logic you require, and adding it to
> company-backends. Except that you *only* need to code the new selection
> function and add it to the auto-completion-source-functions hook; you
> don't need to duplicate the code for finding the completions if all you
> want to do is use an existing completion source in a new context.

Like mentioned above, delegating the search for completions to an 
existing backend is trivial. These are functions, and as such they are 
stateless. Just call (other-backend 'candidates current-prefix).

> I don't get this at all. How is adding a free-form function to the
> `auto-completion-source-functions' hook any less flexible than adding a
> new backend to company-backends? Except that if you simply want to choose
> the completion source based on a regexp match, face, text-property or
> overlay-property, then you don't even have to write any new code (as you
> would in Company) but can get away with simply configuring a variable.

Problem is, I can never get away with doing just one check. At the very 
least, I need to check both for a regexp (or thing-at-point) match, 
*and* the face (or, better yes, `syntax-ppss' return value). Not to 
mention the major mode, but this check could be conceivably replaced by 
a combination of `add-hook' and a buffer-local `company-backends' value.

I think the only backend we have that does only one check is 
`company-dabbrev'.

> I think you're trying to view Completion-UI through Company
> spectacles. Some Company features don't exist of have limited support in
> Completion-UI (e.g. combining sources), some Completion-UI features don't
> exist or have limited support in Company (e.g. adding new custom UIs),

Adding new UIs is simple, as long as they conform to the 
`company-frontends' interface. I'm not completely sure yet if they're 
decidedly "worse" in some sense than Completion-UI interfaces.

>> You could say that I can add another function or several to
>> `auto-completion-source-functions', but they won't be able to do
>> anything smart with third-party sources, I'll have to take care about
>> each source I might want to use, separately, in those functions.
>
> Why on earth not?! They can do whatever they want - they're functions!
> (OK, anything short of solving the Halting Problem ;-)

"Anything smart" in this context would be to use any metadata the 
authors of third-party sources could have supplied if they were 
encourages to do so.

> Sure, but you have to duplicate the same logic in each backend function,
> and you have to either lump *all* the selection logic you'll ever need
> into one function, or have multiple backends for the same completion
> source.

That depends on what one means by the "same logic". Any logically 
different predicate can be extracted into a separate function. Take a 
look at the existing Company backends, the `prefix' section are usually 
quite succinct.

> Yes, if what you mean is an API for defining completion UI widgets,
> together with a bundle of standard widgets. As far as new widget authors
> are concerned, it would be a completion widget API.
>
> But let's be clear: as far as *completion* package authors are concerned,
> it's not a widget API at all. A completion package won't call the tooltip
> widget, or the popup.el widget, or the dynamic completion widget. It will
> call something like `completion-at-point'.

Actually, I was thinking about the former option. Let's define widgets 
with an API in the usual sense, so it can be used by both 
`completion-at-point' and external packages. We'll need this kind of API 
either way, in order to be able to write new widgets.

> There are really two APIs here:
>
> 1. An API for defining new completion widgets, a customization interface
>     to let the user configure them, and code to hook the new UIs into
>     `completion-at-point', `completion-in-region', and eventually
>     minibuffer completion and the rest.
>
> 2. An API for defining new completion sources and new source selection
>     logic, and a customization interface to let the user configure
>     them. This API already exists (c-a-p-f et al.), and the new UI code
>     will have to integrate with it.

Yes, but see above. Using 1. from Company would be the current next step 
toward integration, as I see it.

>> We also document how to use `company-backends' and, for third-party
>> packages, how to add a specific backend to it. But it's a simple data
>> structure, so for users with some experience just knowing backend
>> function names is sufficient.
>
> Sure, that's equally true of `completion-at-point-functions' and
> `completion-ui-register-source'. (Except that new backends are
> first-class citizens in Completion-UI, and show up automatically as
> options in the completion UI customization interface where
> appropriate. This would be harder to replicate with
> `completion-at-point-functions', but I suspect with enough Customize
> macro trickery it would still be possible with the existing Emacs
> completion backend API.)

Customizing hooks is a tricky business. I believe the opaqueness of 
c-a-p-f to the user is the main problem with the current API.

>> Most Company backends at least have to check that the buffer is in
>> matching major mode.
>
> Whereas in Completion-UI, you just use the standard Emacs mechanisms of
> setting buffer-local variables from you major-mode hook, or using
> `add-hook' with the LOCAL flag set to add a function to the
> auto-completion-source-functions hook.
>
> I prefer to use the standard, tried-and-tested Emacs mechanisms where
> possible (major-mode hooks), rather than creating an entirely new
> package-specific way of configuring things for different major-modes. (I
> don't always stick to this myself, though I probably should ;-)

Emacs core also has custom variables that list major modes applicable 
for a given function. Example: `font-lock-global-modes'.

Imagine you have two sources, each applicable in a different major mode. 
With Company, this means two backend functions, and a modified 
`company-backends' value.

With Completion-UI, with hooks and buffer-local values, just count the 
entities:

- foo completion function
- bar completion function
- foo predicate function
- bar predicate function
- function foo-set-source-functions, setting 
`auto-completion-source-functions' to a relevant local value
- function bar-set-source-functions, doing the same for bar
- (add-hook 'foo-mode-hook 'foo-set-source-functions)
- (add-hook 'bar-mode-hook 'bar-set-source-functions)

...now scale this to 5-10 modes and completion functions and see the 
amount of boilerplate explode.

> So every backend has to look at what's being asked of it every time it's
> invoked, and return nil if it's asked to do something it doesn't support?
> And also check the major-mode and see if it's appropriate every time it's
> invoked, even though the major-mode is very unlikely to change (and runs
> a major-mode hook when it does)?

That's only in the `prefix' call. So, once per user command or so.

> This sounds rather inefficient. I'm sure it's fast enough for Company,
> but I'm skeptical it would be fast enough for predictive-mode's
> auto-completion-mode (where tenths of a second matter).

You just haven't measured it.

   (dotimes (_ 1000) (derived-mode-p 'emacs-lisp-mode))

takes less than 1 ms. IOW, the comparison itself takes less than 1e-6 s.

> It's not that the source returns a keymap or widget-building function
> each time it's called. The keymaps or functions that tweak the UI for a
> specific completion source are configured when the source is registered
> with Completion-UI. The completion function itself just returns
> completions (and associated data).
>
> Hope that makes it clearer.

It does, thanks.

> To summarise:
 >
> 2. We're probably stuck with c-a-p-f et al. for the completion source and
>     source-selection API. How would this need to be extended to integrate
>     it with a new Emacs completion UI? How could it be extended to better
>     support Company and others?

To repeat what I said earlier, I'd like to see it customizable by users, 
and I'd like to see a proper merging function.

> 3. The most popular and useful "list-the-available-completions" UI is
>     popup.el. Is there any chance of getting copyright assignment for
>     this? Or will we be forced to code something equivalent from scratch?

https://github.com/auto-complete/popup-el/issues/50

Re: Emacs completion matches selection UI

[Stefan Monnier]

>>> I have no idea how it would work with multiple keymaps.
SM> Right, that's where it gets more interesting.  E.g. some major mode
SM> might want to extend the meaning of TAB, but it should do it in the
SM> major mode's map, not in the global map.
> I think the mykie.el approach has potential here ;)

I don't think how it helps.  It just builds a single function, so
doesn't really do you any good w.r.t making a combined command out of
several different bindings in different keymaps.


        Stefan

Re: Emacs completion matches selection UI

[Stefan Monnier]

SM> We could do the above fairly easily with

SM> (add-function :around (lookup-key global-map [somekey])
SM>               (lambda (orig &rest args)
SM>                 (if (foop) (foo) (apply orig args))))

> Please, no.  It's a real pain to debug this with a user.

Not sure what you see as a debugging problem in the above.


        Stefan

Re: Emacs completion matches selection UI

[Toby Cubitt]

On Tue, Jan 07, 2014 at 04:17:03AM +0400, Dmitry Gutov wrote:
> On 06.01.2014 19:51, Toby Cubitt wrote:
> > Have you looked at the Completion-UI API for defining a new widget? It's
> > uniform across all the widgets. Has to be, otherwise there would be no
> > way of allowing it to be extended with third-party widgets (see my
> > comment below).
> 
> Looking at it now. Seems reasonable, but the data is somewhat coupled to 
> the rest of Completion-UI.

Sure, the details will need consideration.

> Examples:
> 1. `prefix-replaced' and `common-substring' both seem to be tailored to 
> the dynamic interface.

Not at all. `prefix-replaced' tells you that the buffer substring that is
being completed has already been replaced with new text in the
buffer. This is important for non-prefix completion (e.g. pattern
matches).

Any UI that modifies the buffer text will almost certainly need to set
`prefix-replaced'; it just so happens that the only such UI that comes
with Completion-UI is the dynamic interface. Any third-party UI that
modifies the buffer text will almost certainly need to set/check this
property too. And the core Completion-UI code that ties all the UIs
together makes heavy use of the `prefix-replaced' property. So even in
Completion-UI it's not only used by the dynamic interface.

Similarly, `common-substring' demarcates the longest common substring if
completion text has been inserted in the buffer, which is information a
third-party UIs might very well want to make use of.

All the information passed to UIs is there for a UI widget to use or
ignore, as needed. I was reasonably careful to limit the documented
overlay properties to information that might be generically useful, not
stuff that's clearly only ever going to be used by one specific UI and no
other.

The properties documented in the `completion-ui-register-interface'
docstring are those that have a standard meaning in Completion-UI. Note
that nothing prevents a particular UI widget from storing its own data in
its own widget-specific overlay properties if so desired. Indeed, that's
a very good way of storing widget-specific data that persists until the
completion process is finished.

> 2. `completion-source' seems to have to refer to a Completion-UI source, 
> right?

> So how would one use an UI defined as such, in a different package? 
> Would Company define itself as a new source? Or add a source per each 
> backend?

Depends on what the generic Emacs API for specifying completion sources
ends up looking like. If we end up using c-a-p-f, `completion-source'
will need to store something that identifies the c-a-p-f entry that
returned the completion candidates for this completion process.

> > The best comparison one can make is that
> > company-backends loosely corresponds to a mixture of
> > completion-ui-source-definitions and auto-completion-source-functions.
> 
> The latter: yes, the former: no. It's never used the same way as you use 
> `completion-ui-source-definitions'.

I said "loosely corresponds", and I stand by that. One of the things a
`company-backends' entry does is to define when a given completion source
should be used. In Completion-UI, that part of company-backends' role is
performed by auto-completion-source-functions.


> >> In most places, it's used as a second argument to `assq', so it acts as
> >> a "source registry" of sorts, and we don't have a direct correspondence
> >> to that in Company.
> >
> > That's because Company conflates the backend definitions with the
> > mechanism for auto-selecting a backend. Completion-UI keeps those
> > separate. It makes sense to me to separate these, for two reasons.
> > (1) the backend definitions will typically be written by elisp package
> > authors; the auto-selection configuration is something users will want to
> > customize.
> 
> IME, it makes sense in practice because there usually is a set of 
> conditions when a backend is applicable, and there's no sense for the 
> user to write them out, when the package author can anticipate them.

Could well be. But you seem to be arguing both ways. You also argue that
the c-a-p-f API is bad because it's opaque and hard for users to
configure. If backend definitions and selection logic are always supplied
by package authors, then the fact that c-a-p-f is opaque isn't so
significant.

> > (2) I see no reason to limit each source of completions
> > (dabbrevs, etags...) to a single choice of when to use it.
> 
> The package author can always add one or several new of customization 
> variables when the users request it to make the logic more flexible. 
> Most of the time, it's usually just the list of major modes the backend 
> applies to, if it's major mode-sensitive.
> 
> Please take a look at `company-etags' and `company-dabbrev' and see if 
> you can point out the situations when the user might find the `prefix' 
> code of either too limiting.

dabbrev and etags are both sources that, if they're useful in a given
buffer, they're useful in the entire buffer.

Consider the LaTeX math mode source, and the LaTeX environment name
source, and the LaTeX preamble source, etc. Trying to code the selection
logic for all of these using the Company API looks very awkward to me,
compared to the simplicity of setting a few buffer-local variables in
Completion-UI.

Sure, I could move that logic into Predictive itself, and have a single
Predictive LaTeX backend. But that serves to demonstrate that the API
isn't flexible enough to let me do what I want easily. Other markup
languages and programming languages make similar demands on the API.

> > By combining both the choice of whether to use a backend (nil or non-null
> > return value) and finding the list of completions (return value) into one
> > function, you force all the selection logic to be lumped in together with
> > the completion logic. If you want to use the same source of completions
> > (dabbrevs, etags...) in multiple different contexts, you either have to
> > code *all* the selection logic you'll ever need into the one backend, or
> > have multiple backends for the same source of completions.
> 
> Or you can define new backends that would do some common checks in 
> `prefix' (maybe calling an extracted function with common code) and 
> simply delegate all other actions to the respective base backend. 
> Implementing this is trivial.

And once you've finished doing this, and factored out common selection
mechanisms like regexps, faces and text properties into utilities
functions...you'll have reimplemented something closer the Completion-UI
API or c-a-p-f :)

> > I find it logically much cleaner to separate the function for completing
> > a string using a given source (dabbrevs, etags...), from the function for
> > selecting which source of completions to use.
> 
> I understand the principle, really. But the more one "cleanly separates" 
> code, the harder it can be sometimes to read it, to get the full picture.

Indeed, which is why I listed grouping the completion and selection logic
into one place as one of the things I liked about Company's API.

Perhaps the cleanest and most flexible solution would be to have a list
with entries of the form (TEST-FUNCTION . COMPLETION-FUNCTION),
COMPLETION-FUNCTION is used if TEST-FUNCTION returns non-nil. And then
supply a bunch of standard utility functions for use as TEST-FUNCTION's,
for testing regexp matches, faces, text-properties, etc.

This keeps everything together, makes it trivial to avoid code
duplication, and cleanly separates the selection logic and completion
machinery.

> > I hate to say this ;-), but maybe `completion-at-point-functions' got
> > this right. By having one function (the hook function itself) that deals
> > with selecting a completion source, and a separate function (the hook
> > function's return value) for actually doing the completion. This both
> > keeps the two jobs cleanly separated (as in Completion-UI), *and* keeps
> > the two together in the same list.
> 
> It makes certain amount of sense, although it looks like it could make 
> creating a "merged" completion function more difficult.

I doubt it'll be insurmountable. Also, merged completion functions are a
rather advanced feature that may not belong in core Emacs anyway (though
it would be good if the API supported them without ugly hacks).

> We'll see.

Indeed, we'll see. Whilst I'd be happy to see the somewhat complicated
and opaque c-a-p-f API replaced with something cleaner and simpler, I
don't see us winning that argument.

> But again, this separation also makes things more opaque for the user.
> 
> >> `auto-completion-source-functions' also could be thought of as similar
> >> to `company-backends' because both contain logic of picking a suitable
> >> source (the latter by the virtue of backends themselves knowing when
> >> they're suitable), but the list of default detection functions (based on
> >> text properties, overlays, faces at point) looks more limiting to me.
> >
> > Why on earth is it limiting?
> 
> Note the word "default" above. This specific list is limiting.

It's just a reasonable default value, nothing more. It's was never
supposed to be exhaustive. That's why the hook exists: to have functions
added to it.

> But the general approach, while flexible on its surface, complicates 
> things if I intend to use any existing sources, written by third 
> parties. Because their authors are unlikely to have anticipated the 
> logic I'll add in my custom predicate function and to have written any 
> code in their packages I might use. Or, at least, that's considerably 
> less likely.

I don't get your argument here. You have to wrap the third-party
completion function in Company in order to code the appropriate backend
selection logic. In Completion-UI, you put that code in an
`auto-completion-source-function' instead, and probably don't need to
write a wrapper function around the third-party completion
function. Doesn't strike me that the Company way is any less effort; if
anything, it looks like more effort.

I tend to agree that configuring the source selection logic together with
the source completion function is easier to get your head around, though.

> Granted, if I only use Completion-UI for its interfaces and never for 
> any existing sources, it's not going to be a problem.

If you can convince Stefan to let us replace c-a-p-f with a simpler API
that does the same thing, I'm all for it. It would be straightforward to
modify Completion-UI to use whatever API we settle on for specifying
completion sources and selection logic.

I still think we'll lose this one, and be stuck with c-a-p-f, but I'd
love to be proven wrong.

> > I really don't see how this is any different to writing a new Company
> > backend with the particular selection logic you require, and adding it to
> > company-backends. Except that you *only* need to code the new selection
> > function and add it to the auto-completion-source-functions hook; you
> > don't need to duplicate the code for finding the completions if all you
> > want to do is use an existing completion source in a new context.
> 
> Like mentioned above, delegating the search for completions to an 
> existing backend is trivial. These are functions, and as such they are 
> stateless. Just call (other-backend 'candidates current-prefix).

But this results is convoluted and somewhat confusing code. If the
selection logic and completion function were separate, you wouldn't need
to use such hacks.

> > I don't get this at all. How is adding a free-form function to the
> > `auto-completion-source-functions' hook any less flexible than adding a
> > new backend to company-backends? Except that if you simply want to choose
> > the completion source based on a regexp match, face, text-property or
> > overlay-property, then you don't even have to write any new code (as you
> > would in Company) but can get away with simply configuring a variable.
> 
> Problem is, I can never get away with doing just one check. At the very 
> least, I need to check both for a regexp (or thing-at-point) match, 
> *and* the face (or, better yes, `syntax-ppss' return value). Not to 
> mention the major mode, but this check could be conceivably replaced by 
> a combination of `add-hook' and a buffer-local `company-backends' value.

You needed to code this selection logic yourself in any case. So you
could equally well have put that code into an
auto-completion-source-functions function instead of a company-backend
wrapper function.

I'll say this once more and then shut up because I've already said it
umpteen times now: `auto-completion-regexp-source' et al. are just simple
and convenient default entries in the `auto-completion-source-functions'
hook. For anything more complicated, add another function to that hook. I
don't understand why you're harping on about the limitations of the
default hook entries, when you can add whatever function you like to the
hook.

> > I think you're trying to view Completion-UI through Company
> > spectacles. Some Company features don't exist of have limited support in
> > Completion-UI (e.g. combining sources), some Completion-UI features don't
> > exist or have limited support in Company (e.g. adding new custom UIs),
> 
> Adding new UIs is simple, as long as they conform to the 
> `company-frontends' interface. I'm not completely sure yet if they're 
> decidedly "worse" in some sense than Completion-UI interfaces.

Sorry, I missed the `company-frontends' interface. I'll take a look.

> >> You could say that I can add another function or several to
> >> `auto-completion-source-functions', but they won't be able to do
> >> anything smart with third-party sources, I'll have to take care about
> >> each source I might want to use, separately, in those functions.
> >
> > Why on earth not?! They can do whatever they want - they're functions!
> > (OK, anything short of solving the Halting Problem ;-)
> 
> "Anything smart" in this context would be to use any metadata the 
> authors of third-party sources could have supplied if they were 
> encourages to do so.

They could equally well be encouraged to supply a c-a-p-f function (which
is purely selection logic) along with their source. Since c-a-p-f is
already in Emacs, this is likely to be more successful. But my bet is
you'll mostly be dealing with "dumb" backends for a long time yet, and
will have to continue coding the selection logic yourself.

> > Sure, but you have to duplicate the same logic in each backend function,
> > and you have to either lump *all* the selection logic you'll ever need
> > into one function, or have multiple backends for the same completion
> > source.
> 
> That depends on what one means by the "same logic". Any logically 
> different predicate can be extracted into a separate function. Take a 
> look at the existing Company backends, the `prefix' section are usually 
> quite succinct.

Indeed, though this is starting to look a lot like defining some standard
hook functions for auto-completion-source-functions. As you make the
company-backends API more flexible and convenient, you'll increasingly
find yourself reimplementing equivalent functionality to that of c-a-p-f
and the Completion-UI API. If were to start simplifying the Completion-UI
API or c-a-p-f, it'll increasingly look more like the Company API. Maybe
the sweet spot is in the middle?

> > Yes, if what you mean is an API for defining completion UI widgets,
> > together with a bundle of standard widgets. As far as new widget authors
> > are concerned, it would be a completion widget API.
> >
> > But let's be clear: as far as *completion* package authors are concerned,
> > it's not a widget API at all. A completion package won't call the tooltip
> > widget, or the popup.el widget, or the dynamic completion widget. It will
> > call something like `completion-at-point'.
> 
> Actually, I was thinking about the former option. Let's define widgets 
> with an API in the usual sense, so it can be used by both 
> `completion-at-point' and external packages. We'll need this kind of API 
> either way, in order to be able to write new widgets.

I'm confused. `completion-at-point' will never use an API for *defining*
new completion UI widgets (`completion-ui-register-interface'). It will
need to be modified to *invoke* the new UI widgets.

Copying and adapting some of the code from `complete-in-buffer' into
`completion-in-region' would suffice for that. Then
`completion-in-region' would remain the generic Emacs API for displaying
and selecting completion candidates (only now it would display them in a
nicer interface).


> > There are really two APIs here:
> >
> > 1. An API for defining new completion widgets, a customization interface
> >     to let the user configure them, and code to hook the new UIs into
> >     `completion-at-point', `completion-in-region', and eventually
> >     minibuffer completion and the rest.
> >
> > 2. An API for defining new completion sources and new source selection
> >     logic, and a customization interface to let the user configure
> >     them. This API already exists (c-a-p-f et al.), and the new UI code
> >     will have to integrate with it.
> 
> Yes, but see above. Using 1. from Company would be the current next step 
> toward integration, as I see it.

Really? Why should we use the Company UI widget API and interfaces in
Emacs, when the API is less flexible and less customizable than that the
Completion-UI widgetAPI, and it implements fewer widgets?

Did you really mean that you wanted to throw Completion-UI in the
garbage, rewrite Company yourself to be suitable as a generic Emacs
completion API, rewrite/port the missing UIs and features, and persuade
people to merge it into Emacs core? (If so, great! Let me know when
you're done, and then I can stop maintaining Completion-UI and switch
Predictive over to the new interface :-)

Probably I misunderstood what you wrote.

I'm happy to help when I have time, and to contribute Completion-UI code
and experience. But I'm equally happy for someone else to do all the
work. My interest is in seeing a completion UI get included in Emacs
which is powerful enough to replace Completion-UI in Predictive. Then I
can stop maintaining Completion-UI on my own, and instead contribute
patches to the Emacs UI.

I only really have one strong requirement: if some form of
auto-completion mode gets included in the Emacs completion UI, I will
argue hard for it to be as powerful as Completion-UI's
`auto-completion-mode'. (Take a look at `auto-completion-syntax-alist'
and `auto-completion-override-syntax-alist', and at the way all the UI
widgets are integrated.) Anything less powerful, and it will be useless
to me for Predictive.

> >> We also document how to use `company-backends' and, for third-party
> >> packages, how to add a specific backend to it. But it's a simple data
> >> structure, so for users with some experience just knowing backend
> >> function names is sufficient.
> >
> > Sure, that's equally true of `completion-at-point-functions' and
> > `completion-ui-register-source'. (Except that new backends are
> > first-class citizens in Completion-UI, and show up automatically as
> > options in the completion UI customization interface where
> > appropriate. This would be harder to replicate with
> > `completion-at-point-functions', but I suspect with enough Customize
> > macro trickery it would still be possible with the existing Emacs
> > completion backend API.)
> 
> Customizing hooks is a tricky business. I believe the opaqueness of 
> c-a-p-f to the user is the main problem with the current API.

But above you argued that backends and selection logic are supplied and
configured by package authors, not by users!? In which case package
authors can simply supply a c-a-p-f function, and users can add the
functions they want to c-a-p-f. (I believe Stefan made a similar point
earlier.)

Don't get me wrong. I'm no fan of the complexity of c-a-p-f. But you seem
to be arguing both ways.

> >> Most Company backends at least have to check that the buffer is in
> >> matching major mode.
> >
> > Whereas in Completion-UI, you just use the standard Emacs mechanisms of
> > setting buffer-local variables from you major-mode hook, or using
> > `add-hook' with the LOCAL flag set to add a function to the
> > auto-completion-source-functions hook.
> >
> > I prefer to use the standard, tried-and-tested Emacs mechanisms where
> > possible (major-mode hooks), rather than creating an entirely new
> > package-specific way of configuring things for different major-modes. (I
> > don't always stick to this myself, though I probably should ;-)
> 
> Emacs core also has custom variables that list major modes applicable 
> for a given function. Example: `font-lock-global-modes'.

Fair point. I use such alists in Predictive too
(`predictive-major-mode-alist').

> Imagine you have two sources, each applicable in a different major mode. 
> With Company, this means two backend functions, and a modified 
> `company-backends' value.
> 
> With Completion-UI, with hooks and buffer-local values, just count the 
> entities:
> 
> - foo completion function
> - bar completion function
> - foo predicate function
> - bar predicate function
> - function foo-set-source-functions, setting 
> `auto-completion-source-functions' to a relevant local value
> - function bar-set-source-functions, doing the same for bar
> - (add-hook 'foo-mode-hook 'foo-set-source-functions)
> - (add-hook 'bar-mode-hook 'bar-set-source-functions)
> 
> ...now scale this to 5-10 modes and completion functions and see the 
> amount of boilerplate explode.

Swings and roundabouts. I could always add a
`completion-ui-major-mode-source' function that checks an alist, and then
it would be:

- foo completion function
- bar completion function
- foo predicate function
- bar predicate function
- add two entries to the alist

versus Company's

- foo's bundled completion function and selection logic
- bar's bundled completion function and selection logic
- add two entries to company-backends

On the other hand, configuring the 8 different Predictive LateX sources
(not to mention the additional package-specific sources that get loaded
and unloaded dynamically as the user types text into the LaTeX document)
via Company backends would involve:

- code 1st bundled completion function and selection logic
- code 2nd bundled completion function and selection logic (which
  duplicates the logic in the 1st one, with different parameters)
  ...
- code 8th bundled completion function and selection logic (which
  duplicates the logic in the last 7, with different parameters)
- add 8 entries to company-backends
- arrange for package-specific sources to be added to company-backends
  dynamically

Contrast with Completion-UI's, which involves a *lot* less coding:

- register 1st source (no need to code a new completion function, just
  set one parameter when calling `completion-ui-register-source')
- register 2nd source (same)
- ...
- register 8th source
- add entries to a single alist
- push and pop entries from that alist dynamically as needed

So Completion-UI lets you do some things more simply, and Company lets
you do others more simply. I still think separating the selection logic
from the completion function is cleaner, more logical, and more flexible.

It's also what c-a-p-f does. (It's a shame the c-a-p-f API is marred by
the complex format; though I can understand why it ended up like that.)

Actually, now I think about it, Completion-UI's
`auto-completion-source-functions' hook is *very* similar to c-a-p-f: a
list of functions that are called in turn until one of them returns the
completion source to use. It just has a simpler format for the return
values (a symbol identifying a completion source, instead of the
multi-element list returned by c-a-p-f functions.) It would be almost
trivial to switch Completion-UI over to c-a-p-f. All I'd need to do to
replicate the existing functionality is add a few additional properties
to the PROPS return argument.


> > So every backend has to look at what's being asked of it every time it's
> > invoked, and return nil if it's asked to do something it doesn't support?
> > And also check the major-mode and see if it's appropriate every time it's
> > invoked, even though the major-mode is very unlikely to change (and runs
> > a major-mode hook when it does)?
> 
> That's only in the `prefix' call. So, once per user command or so.

Right.

> > This sounds rather inefficient. I'm sure it's fast enough for Company,
> > but I'm skeptical it would be fast enough for predictive-mode's
> > auto-completion-mode (where tenths of a second matter).
> 
> You just haven't measured it.
> 
>    (dotimes (_ 1000) (derived-mode-p 'emacs-lisp-mode))
> 
> takes less than 1 ms. IOW, the comparison itself takes less than 1e-6 s.

But that's without the complex selection logic needed e.g. in
Predictive's LaTeX support. I had to rewrite the Completion-UI selection
logic two or three times before it was efficient enough for this.

It's no big deal, though. Presumably I could always set a buffer-local
`company-backends' from predictive-mode that only contains relevant
backends if necessary.

> > To summarise:
>  >
> > 2. We're probably stuck with c-a-p-f et al. for the completion source and
> >     source-selection API. How would this need to be extended to integrate
> >     it with a new Emacs completion UI? How could it be extended to better
> >     support Company and others?
> 
> To repeat what I said earlier, I'd like to see it customizable by users, 

c-a-p-f is trivially customizable by users, as long as packages / Company
supplies suitable c-a-p-f functions for the backends. *Writing* c-a-p-f
is non-trivial, but as you said earlier those are/should be supplied by
package authors. Or have I misunderstood?

I'm not advocating c-a-p-f, I'm just running out of good arguments
against it.

> and I'd like to see a proper merging function.

Agreed.

> > 3. The most popular and useful "list-the-available-completions" UI is
> >     popup.el. Is there any chance of getting copyright assignment for
> >     this? Or will we be forced to code something equivalent from scratch?
> 
> https://github.com/auto-complete/popup-el/issues/50

Fingers crossed...!

Best,
Toby
-- 
Dr T. S. Cubitt
Royal Society University Research Fellow
and Fellow of Churchill College, Cambridge
Centre for Quantum Information
DAMTP, University of Cambridge

email: tsc25@cantab.net
web:   www.dr-qubit.org

Re: Emacs completion matches selection UI

[Toby Cubitt]

On Mon, Jan 06, 2014 at 06:38:37PM -0500, Ted Zlatanov wrote:
> On Sat, 4 Jan 2014 00:48:53 +0000 Toby Cubitt <tsc25@cantab.net> wrote: 
> TC> Since it's the completion UI that will have to interpret and make use of
> TC> the extras, one place to start is to think about what additional features
> TC> the UI might want to provide, beyond selecting from a list of possible
> TC> completions.
> 
> TC> Luckily, we have many existing completion frameworks to draw on for
> TC> inspiration. At the moment, location data and various forms of
> TC> documentation (e.g. type information in code completion, docstrings,
> TC> etc.) are the only ones that spring to mind.
> 
> Add: feedback from frontend to backend (Predictive use case we
> mentioned, needs API support through callbacks).  Pluggable completion
> frontends (not in the API)

Or rather, requires a separate API. Both Completion-UI and Company
already have APIs for this (though the discussion's mostly focused on the
completion UI API so far).

>  Dynamic limiting of completion candidates as characters are typed (not
> in the API).

All Completion-UI widgets already implement this(*), and the UI widget
API ensures that any third-part widgets will also automatically support
this too.

(*) The mouse-driven toolkit menus break it by stealing the keyboard
focus, but that's a technicality.

> The backend should be able to provide hints about the data.  For
> instance, "this is a list of ELisp symbols" could look different from
> "this is a list of installable packages."  Also icons are important IMO,
> based on the hints or the candidates.
> 
> TC> The keyword arguments to `completion-ui-register-source' are one starting
> TC> point for seeing what additional properties might be needed, as are the
> TC> corresponding APIs in Company, auto-complete, etc.
> 
> Can you or Dmitry compile a list?

Here are the important ones (copied from the docstring):

:activate FUNCTION
        The function called to activate the user-interface.

:deactivate FUNCTION
        The function called to deactivate the user-interface.

:update FUNCTION (optional)
        A function to call to update the user-interface after a
        change to the current completion. If it is not specified, the
        interface is updated by calling the :deactivate function,
        updating the completion, then calling the :activate function.

:auto-show FUNCTION (optional)
        A function to call to activate a this user-interface as
        an auto-show interface. The auto-show interface is chosen
        by setting the `completion-auto-show' customization
        option. Only one auto-show interface can be displayed at
        any one time.

:auto-show-helper FUNCTION (optional)
        A function to call whenever any auto-show interface is
        activated for a completion.


In Completion-UI, all of these functions are passed a single argument: an
overlay which demarcates where in the buffer the completion is taking
place, and whose properties contain relevant data about the completion
process. I rather like passing the data in overlay properties, because it
keeps the data together with the location in the buffer where things are
happening. But one could pass the data through buffer-local variables if
preferred.

The following overlay properties store the standard completion-related
data:

prefix
        The prefix or string being completed.

prefix-length
        Length of the prefix or string currently being completed. (For
        convenience, to avoid having to repeatedly call
        (length (overlay-get completion-overlay prefix))

completions
        A list of completion candidates.

completion-num
        Index of currently selected completion candidate, or nil
        if completions list is empty.

completion-overlay
        Always t for a completion overlay.

completion-source
        The completion source ("backend") used to obtain the completion
        candidates.

completion-prefix-function
        Function that was used to find the prefix or string to
        complete.

non-prefix-completion
        Nil if doing standard prefix completion, non-nil if doing
        something other than prefix completion (such as pattern
        matching).

prefix-replaced
        Non-nil if buffer string being completed (stored in prefix
        overlay property) has been replaced with new text in the
        buffer. (Only relevant when doing non-prefix completion.)

common-substring
        If set, an overlay marking the longest common substring of the
        completion text currently inserted in the buffer, if any.

keymap
        Keymap active within the completion overlay. Usually derived from
        `completion-overlay-map' or `auto-completion-overlay-map'.

auto-show
       Name of auto-show interface (symbol) currently being
       displayed, or nil if no auto-show interface is active.



> TC> Also, if we want to allow users to optionally customize the completion UI
> TC> per-backend (a la Completion-UI), there has to be some way of identifying
> TC> backends in Customize. This seems necessary to me if the new API is to
> TC> become the standard way of implementing a completion UI in Emacs. Think
> TC> minibuffer completion, versus filename completion, versus predictive
> TC> completion, versus elisp code completion etc, all of which might want to
> TC> use slightly different UI options. Again, it might be enough to support a
> TC> :name property (or some such) in the `completion-at-point-functions'
> TC> PROPS plist.
> 
> The frontend UI should depend (in my mind) on three things: 1) the
> invocation context as you showed; 2) the backend-provided hints about
> the data as a whole; and 3) the backend-provided hints about each
> completion candidate individually.  Does that make sense?

Yes, it makes sense. I suspect most of the time the UI will only need to
depend on which backend is being used (and often not even on that). I
can't think of any situations in which it will need to depend on the
completions returned by the backend. But maybe there are some cases.

> Some frontends may ignore some or all of those things, but I think
> those are the principal axis.  I don't know how those three things will
> be passed to the frontend though, and when.

Everything I've ever wanted to do with Completion-UI is covered by
allowing various configuration variables to take backend-specific
values. I've never needed the UI to depend on the return value of the
backend. Maybe Dmitry has examples from Company?

> >> - the frontend choices made by the user should be able to feed back into
> >> the data backend (the "predictive" use case)
> 
> TC> A few optional hooks/call-backs would very likely be sufficient. Again,
> TC> the `completion-at-point-functions' PROPS plist would be the obvious
> TC> place to specify them.
> 
> OK, this is your use case :)

I'm just bowing to the inevitable outcome that we'll end up extending
c-a-p-f rather than designing a new API for the completion sources.

> >> - we have enough interest and support in this work that it's worth
> >> undertaking in earnest after the code freeze.  The discussion should
> >> remain on emacs-devel (based on the wide interest so far).
> ...
> TC> (Note that I'm unlikely to have time to code anything myself until
> TC> March/April, although the feature freeze will likely delay things until
> TC> about then anyway. I'm happy to help in a more limited way before then,
> TC> as time allows.)
> 
> Cool.  We seem to be moving towards a specific API, refining the current
> Emacs internals, which is good.  I'd be OK with it looking like
> completion-ui as long as it was at least somewhat backwards compatible
> and could support the packages we mentioned.

By "looking like completion-UI", do you mean the API or what it actually
looks like?

I think the completion source API is likely to look exactly like
completion-at-point-functions, because that's already in Emacs and I'm
running out of compelling arguments for changing it, even though I don't
love it. But Dmitry may hold out longer than me on this ;-)

> To me, the clearest indication of a good approach would be if both the
> Emacs core and external packages were able to lose lines of code.

I'm hoping the Predictive package can lose 9 entire elisp files, for a
start (all the Completion-UI files ;-)

Toby
-- 
Dr T. S. Cubitt
Royal Society University Research Fellow
and Fellow of Churchill College, Cambridge
Centre for Quantum Information
DAMTP, University of Cambridge

email: tsc25@cantab.net
web:   www.dr-qubit.org

Re: Emacs completion matches selection UI

[Ted Zlatanov]

On Mon, 06 Jan 2014 22:16:15 -0500 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 

SM> We could do the above fairly easily with
SM> (add-function :around (lookup-key global-map [somekey])
SM> (lambda (orig &rest args)
SM> (if (foop) (foo) (apply orig args))))

>> Please, no.  It's a real pain to debug this with a user.

SM> Not sure what you see as a debugging problem in the above.

I find advised functions harder to debug, personally.

Ted

Re: Emacs completion matches selection UI

[Ted Zlatanov]

On Mon, 06 Jan 2014 22:15:28 -0500 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 

>>>> I have no idea how it would work with multiple keymaps.
SM> Right, that's where it gets more interesting.  E.g. some major mode
SM> might want to extend the meaning of TAB, but it should do it in the
SM> major mode's map, not in the global map.
>> I think the mykie.el approach has potential here ;)

SM> I don't think how it helps.  It just builds a single function, so
SM> doesn't really do you any good w.r.t making a combined command out of
SM> several different bindings in different keymaps.

I don't mean the current mykie.el interface but extending the call
syntax to support this case better, like mykie.el does for its use
cases.

Ted

Re: Emacs completion matches selection UI

[João Távora]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> Right, that's where it gets more interesting.	 E.g. some major mode
> might want to extend the meaning of TAB, but it should do it in the
> major mode's map, not in the global map.

Yep, I tried a quick prototype and hit this wall.

> I don't think how it helps.  It just builds a single function, so
> doesn't really do you any good w.r.t making a combined command out of
> several different bindings in different keymaps.

Are any functions available that return the keymap lookup sequence used
normally ? I wish there was something like this, but it seems hidden
behind functions like `key-binding'.

João

Re: Emacs completion matches selection UI

[Josh]

On Tue, Jan 7, 2014 at 9:18 AM, João Távora <joaot@siscog.pt> wrote:
> Are any functions available that return the keymap lookup sequence used
> normally ? I wish there was something like this, but it seems hidden
> behind functions like `key-binding'.

Related: http://debbugs.gnu.org/cgi/bugreport.cgi?bug=13948

Re: Emacs completion matches selection UI

[David Kastrup]

Josh <josh@foxtail.org> writes:

> On Tue, Jan 7, 2014 at 9:18 AM, João Távora <joaot@siscog.pt> wrote:
>> Are any functions available that return the keymap lookup sequence used
>> normally ? I wish there was something like this, but it seems hidden
>> behind functions like `key-binding'.
>
> Related: http://debbugs.gnu.org/cgi/bugreport.cgi?bug=13948

current-active-maps is a built-in function in `C source code'.

(current-active-maps &optional OLP POSITION)

Return a list of the currently active keymaps.
OLP if non-nil indicates that we should obey `overriding-local-map' and
`overriding-terminal-local-map'.  POSITION can specify a click position
like in the respective argument of `key-binding'.

[back]


Actually, I think this was written by me once upon a time.

-- 
David Kastrup

Re: Emacs completion matches selection UI

[Dmitry Gutov]

On 07.01.2014 07:32, Toby Cubitt wrote:
>> Examples:
>> 1. `prefix-replaced' and `common-substring' both seem to be tailored to
>> the dynamic interface.
>
> Not at all. `prefix-replaced' tells you that the buffer substring that is
> being completed has already been replaced with new text in the
> buffer. This is important for non-prefix completion (e.g. pattern
> matches).

I'm not sure if it's generally applicable. AFAIK, neither Company nor 
CAPF keep this kind of history: if the buffer was modified, what happens 
after is a new completion, or no completion at all (if circumstances 
told us we should abort).

> Any UI that modifies the buffer text will almost certainly need to set
> `prefix-replaced';

I don't really see why a non-prefix completion UI (or any other one) has 
to modify the buffer text. Company has an inline "preview" frontend, but 
it uses an overlay, and so the buffer text remains unmodified.

> And the core Completion-UI code that ties all the UIs
> together makes heavy use of the `prefix-replaced' property. So even in
> Completion-UI it's not only used by the dynamic interface.

Could you explain how?

> Similarly, `common-substring' demarcates the longest common substring if
> completion text has been inserted in the buffer, which is information a
> third-party UIs might very well want to make use of.

Why? How?

> The properties documented in the `completion-ui-register-interface'
> docstring are those that have a standard meaning in Completion-UI. Note
> that nothing prevents a particular UI widget from storing its own data in
> its own widget-specific overlay properties if so desired.

Sure.

>> So how would one use an UI defined as such, in a different package?
>> Would Company define itself as a new source? Or add a source per each
>> backend?
>
> Depends on what the generic Emacs API for specifying completion sources
> ends up looking like.

Hmm, I was expecting something easier, since IIRC you said Company 
should have been able to use it without major changes.

> If we end up using c-a-p-f, `completion-source'
> will need to store something that identifies the c-a-p-f entry that
> returned the completion candidates for this completion process.

Probably just the value returned by the successful completion function: 
start, end, collection (aka completion table, which is often a function).

>>> The best comparison one can make is that
>>> company-backends loosely corresponds to a mixture of
>>> completion-ui-source-definitions and auto-completion-source-functions.
>>
>> The latter: yes, the former: no. It's never used the same way as you use
>> `completion-ui-source-definitions'.
>
> I said "loosely corresponds", and I stand by that. One of the things a
> `company-backends' entry does is to define when a given completion source
> should be used. In Completion-UI, that part of company-backends' role is
> performed by auto-completion-source-functions.

Yes, that's why I agreed with this comparison (and also made it earlier, 
IIRC). I disagree with the other comparison.

> Could well be. But you seem to be arguing both ways. You also argue that
> the c-a-p-f API is bad because it's opaque and hard for users to
> configure. If backend definitions and selection logic are always supplied
> by package authors, then the fact that c-a-p-f is opaque isn't so
> significant.

The opaqueness in c-a-p-f is bad because the exact values of 
`company-backends' and `completion-at-point-functions' are significant. 
A third-party package author can only push a function in either of these 
lists, but they can be responsible for the eventual order of the elements.

And the order matters, because it influences which completion backend 
will get picked when several are suitable. When we get to the grouped 
backends, which I've mentioned several times, which Company supports, 
and CAPF will hopefully support in the future, being able to understand 
and maybe change the values of either list becomes even more important.

>> Please take a look at `company-etags' and `company-dabbrev' and see if
>> you can point out the situations when the user might find the `prefix'
>> code of either too limiting.
>
> dabbrev and etags are both sources that, if they're useful in a given
> buffer, they're useful in the entire buffer.
>
> Consider the LaTeX math mode source, and the LaTeX environment name

My question was, if you could point out problems with any of our bundled 
backends (or, failing that, third-party ones). If they look okay, maybe

> source, and the LaTeX preamble source, etc. Trying to code the selection
> logic for all of these using the Company API looks very awkward to me,
> compared to the simplicity of setting a few buffer-local variables in
> Completion-UI.

If performance is the problem, we could solve that by either:

1) Adding some pre-completion hook which would allow you to run some 
code once, set a buffer-local variable, which all backend functions for 
LaTeX can refer to later in `prefix' action.

2) Create a special "merged" backend that would collect the results from 
all LaTeX backend in a carefully defined fashion.

3) Indeed add some hook analogous to `auto-completion-source-functions'. 
But yeah, CAPF already does that.

> Sure, I could move that logic into Predictive itself, and have a single
> Predictive LaTeX backend. But that serves to demonstrate that the API
> isn't flexible enough to let me do what I want easily. Other markup
> languages and programming languages make similar demands on the API.

We bundle several backends, some of them for programming modes, and so 
far you haven't pointed out specific problems with any of them.

>> Or you can define new backends that would do some common checks in
>> `prefix' (maybe calling an extracted function with common code) and
>> simply delegate all other actions to the respective base backend.
>> Implementing this is trivial.
>
> And once you've finished doing this, and factored out common selection
> mechanisms like regexps, faces and text properties into utilities

Regexps we have already (company-grab-...), instead of faces one should 
be using `syntax-ppss', font-lock isn't always available, and text 
properties... hmm.

> functions...you'll have reimplemented something closer the Completion-UI
> API or c-a-p-f :)

Only if we add a similar hook, see 3) above.

>> I understand the principle, really. But the more one "cleanly separates"
>> code, the harder it can be sometimes to read it, to get the full picture.
>
> Indeed, which is why I listed grouping the completion and selection logic
> into one place as one of the things I liked about Company's API.
>
> Perhaps the cleanest and most flexible solution would be to have a list
> with entries of the form (TEST-FUNCTION . COMPLETION-FUNCTION),
> COMPLETION-FUNCTION is used if TEST-FUNCTION returns non-nil.

I believe this suffers from the LaTeX problem you've described above. If 
you have a dozen of completion functions for LaTeX, this scheme expects 
you to have a dozen of corresponding test functions, and when one fails, 
the next will be called, and it won't be able to use the results of the 
previous call (unless they set and use some common buffer-local 
variable, which Company backends could also do; but that's ugly).

 > And then
> supply a bunch of standard utility functions for use as TEST-FUNCTION's,
> for testing regexp matches, faces, text-properties, etc.

Like I described previously, such stand-alone tests probably won't be 
very useful as values of this alist. One usually has to call several of 
them to see if a completion-function is suitable.

>> It makes certain amount of sense, although it looks like it could make
>> creating a "merged" completion function more difficult.
>
> I doubt it'll be insurmountable. Also, merged completion functions are a
> rather advanced feature that may not belong in core Emacs anyway (though
> it would be good if the API supported them without ugly hacks).

It's a good feature enabling some kinds of backends that aren't usually 
useful on their own (like Yasnippet). In my book, that's a good argument 
to accept or reject an API.

>> We'll see.
>
> Indeed, we'll see. Whilst I'd be happy to see the somewhat complicated
> and opaque c-a-p-f API replaced with something cleaner and simpler, I
> don't see us winning that argument.

Personally, I'd probably be fine with c-a-p-f as long as it's powerful 
enough. Moving to a less featureful API is likely out of the question, 
but if it's demonstrated that c-a-p-f is fairly unsuitable for 
implementing some features, I believe it would be a good reason to rule 
it out.

>> But the general approach, while flexible on its surface, complicates
>> things if I intend to use any existing sources, written by third
>> parties. Because their authors are unlikely to have anticipated the
>> logic I'll add in my custom predicate function and to have written any
>> code in their packages I might use. Or, at least, that's considerably
>> less likely.
>
> I don't get your argument here. You have to wrap the third-party
> completion function in Company in order to code the appropriate backend
> selection logic. In Completion-UI, you put that code in an
> `auto-completion-source-function' instead, and probably don't need to

Which code? There's likely to be none.

Hence, more effort required on my part.

Here I'm describing an organizational problem caused by an API. Not a 
technical one.

>> Like mentioned above, delegating the search for completions to an
>> existing backend is trivial. These are functions, and as such they are
>> stateless. Just call (other-backend 'candidates current-prefix).
>
> But this results is convoluted and somewhat confusing code. If the
> selection logic and completion function were separate, you wouldn't need
> to use such hacks.

Looks straightforward to me. Function delegation is a rather simple concept.

We have examples of that in the Company frontends: take a look at 
`company-pseudo-tooltip-unless-just-one-frontend' and 
`company-preview-if-just-one-frontend'.

`company-dabbrev-code' also delegates to `company-dabbrev', but it just 
uses a public function from that backend's package, which is also a 
valid approach.

> It understand why you're harping on about the limitations of the
> default hook entries

I just can't see how you find them useful on their own. I'll shut up now. :)

> Indeed, though this is starting to look a lot like defining some standard
> hook functions for auto-completion-source-functions. As you make the
> company-backends API more flexible and convenient, you'll increasingly
> find yourself reimplementing equivalent functionality to that of c-a-p-f
> and the Completion-UI API. If were to start simplifying the Completion-UI
> API or c-a-p-f, it'll increasingly look more like the Company API. Maybe
> the sweet spot is in the middle?

Maybe. I can certainly see myself adding a 
auto-completion-source-functions analog in Company, as an advanced feature.

>> Actually, I was thinking about the former option. Let's define widgets
>> with an API in the usual sense, so it can be used by both
>> `completion-at-point' and external packages. We'll need this kind of API
>> either way, in order to be able to write new widgets.
>
> I'm confused. `completion-at-point' will never use an API for *defining*
> new completion UI widgets (`completion-ui-register-interface'). It will
> need to be modified to *invoke* the new UI widgets.

Here I'm thinking in terms of Company approach, I guess. The API for 
defining a backend or frontend and the way it is used are the same: a 
protocol consisting of messages and proper reactions or responses to them.

Thus, "widget has to respond to..." would be in the API for defining a 
widget, and "widgets respond to..." would be in the usage API. 
Essentially the same.

If you like to add convenience macros, extra infrastructure, etc, they 
may diverge, but we'll need an API a completion package can use easily 
use anyway, not just a bunch of private functions that can change will 
every release.

> Copying and adapting some of the code from `complete-in-buffer' into
> `completion-in-region' would suffice for that. Then
> `completion-in-region' would remain the generic Emacs API for displaying
> and selecting completion candidates (only now it would display them in a
> nicer interface).

Maybe. At a first glance, `completion-in-region' will need access to 
PROPS returned by completion functions, not just COLLECTIONS.

>> Yes, but see above. Using 1. from Company would be the current next step
>> toward integration, as I see it.
>
> Really? Why should we use the Company UI widget API and interfaces in
> Emacs, when the API is less flexible and less customizable than that the
> Completion-UI widgetAPI, and it implements fewer widgets?

Sorry for being unclear. I meant the reverse: Company would use the new 
widget API defined here, while still retaining its backends, at least 
initially.

Although while you don't own the popup widget, we do have one. Maybe 
porting that code won't be too hard.

> Did you really mean that you wanted to throw Completion-UI in the
> garbage, rewrite Company yourself to be suitable as a generic Emacs
> completion API, rewrite/port the missing UIs and features, and persuade
> people to merge it into Emacs core? (If so, great! Let me know when
> you're done, and then I can stop maintaining Completion-UI and switch
> Predictive over to the new interface :-)

I'll think about it. :) Not sure about the "persuade people to merge in 
into Emacs core", though.

But you could help kick this process off my filing an issue describing 
Company's backend API shortcomings. Is it just the lack of 
`auto-completion-source-functions'? Non-prefix completion?

> Probably I misunderstood what you wrote.

Yep. :)

> I only really have one strong requirement: if some form of
> auto-completion mode gets included in the Emacs completion UI, I will
> argue hard for it to be as powerful as Completion-UI's
> `auto-completion-mode'. (Take a look at `auto-completion-syntax-alist'
> and `auto-completion-override-syntax-alist', and at the way all the UI
> widgets are integrated.) Anything less powerful, and it will be useless
> to me for Predictive.

I'll take a look, thanks.

>> Customizing hooks is a tricky business. I believe the opaqueness of
>> c-a-p-f to the user is the main problem with the current API.
>
> But above you argued that backends and selection logic are supplied and
> configured by package authors, not by users!? In which case package
> authors can simply supply a c-a-p-f function, and users can add the
> functions they want to c-a-p-f. (I believe Stefan made a similar point
> earlier.)

Customizing a hook is tricky for user. Try `M-x customize-variable RET 
find-file-hook'. You'll only see the buffer-local value, and not the 
global one. But `run-hooks' uses both if the local value includes `t', 
which it usually does.

Users can modify the hooks programmatically, of course, but that's a 
step more difficult. And they'll also need to understand the values 
already there, to be able to remove or rearrange them.

We'll also should discourage lambdas there. At the moment, my 
find-file-hook contains this beauty, courtesy of autorevert.el:

#[nil "\302\301!\210\303\304\b!8\211\x11\207"
   [buffer-file-name auto-revert-tail-pos make-local-variable 7 
file-attributes]
   3]

We should have less of that.

> Don't get me wrong. I'm no fan of the complexity of c-a-p-f. But you seem
> to be arguing both ways.

Of course I'm arguing toward a middle ground comfortable to me. So: 
package authors deciding when their backend is suitable to use -- good; 
users deciding the order of trying backends and their groupings -- also 
good.

> Swings and roundabouts. I could always add a
> `completion-ui-major-mode-source' function that checks an alist, and then
> it would be:

Yes, well, here you are discarding the "standard, tried-and-tested Emacs 
mechanisms" of major mode hooks and buffer-local values. Which was 
exactly my point.

> - foo completion function
> - bar completion function
> - foo predicate function
> - bar predicate function
> - add two entries to the alist

You forgot adding `completion-ui-major-mode-source' to the relevant list.

> - code 1st bundled completion function and selection logic
> - code 2nd bundled completion function and selection logic (which
>    duplicates the logic in the 1st one, with different parameters)
>    ...
> - code 8th bundled completion function and selection logic (which
>    duplicates the logic in the last 7, with different parameters)

Could that be a one-line function call, in each case with different 
arguments? Aside from potential performance problems, it sounds rather neat.

> - add 8 entries to company-backends
> - arrange for package-specific sources to be added to company-backends
>    dynamically
>
> Contrast with Completion-UI's, which involves a *lot* less coding:
>
> - register 1st source (no need to code a new completion function, just
>    set one parameter when calling `completion-ui-register-source')

You can likewise delegate to the same completion function in Company 
backends. It's the same amount of code (1 line).

> It would be almost
> trivial to switch Completion-UI over to c-a-p-f. All I'd need to do to
> replicate the existing functionality is add a few additional properties
> to the PROPS return argument.

Great. I'd like to see the patch. :)

> Presumably I could always set a buffer-local
> `company-backends' from predictive-mode that only contains relevant
> backends if necessary.

Yes, of course.

> c-a-p-f is trivially customizable by users, as long as packages / Company
> supplies suitable c-a-p-f functions for the backends.

When you say "trivially customizable", do you mean via the Customize 
interface?

> *Writing* c-a-p-f
> is non-trivial, but as you said earlier those are/should be supplied by
> package authors. Or have I misunderstood?

I don't think I said that about c-a-p-f, but yes, they should be.

>>> 3. The most popular and useful "list-the-available-completions" UI is
>>>      popup.el. Is there any chance of getting copyright assignment for
>>>      this? Or will we be forced to code something equivalent from scratch?
>>
>> https://github.com/auto-complete/popup-el/issues/50
>
> Fingers crossed...!

I wouldn't hold my breath: IME, getting a response from that crowd on 
non-trivial issues is hard. Collecting copyright assignments could be 
harder still.

P.S. My email client displays each new message from you as a separate 
thread, possibly because your email address is timestamped, or maybe 
because they don't contain a "References" header. Could you do something 
about that?

Re: Emacs completion matches selection UI

[Toby Cubitt]

On Wed, Jan 08, 2014 at 03:23:21AM +0400, Dmitry Gutov wrote:
> On 07.01.2014 07:32, Toby Cubitt wrote:
> >> Examples:
> >> 1. `prefix-replaced' and `common-substring' both seem to be tailored to
> >> the dynamic interface.
> >
> > Not at all. `prefix-replaced' tells you that the buffer substring
> > that is being completed has already been replaced with new text in
> > the buffer. This is important for non-prefix completion (e.g. pattern
> > matches).
> 
> > I'm not sure if it's generally applicable.
[...]
> > Any UI that modifies the buffer text will almost certainly need to set
> > `prefix-replaced';
> 
> I don't really see why a non-prefix completion UI (or any other one) has 
> to modify the buffer text. Company has an inline "preview" frontend, but 
> it uses an overlay, and so the buffer text remains unmodified.

Ah, you mean using the `before-string' or `after-string' overlay
property? That does sound like a better implementation.


> AFAIK, neither Company nor CAPF keep this kind of history: if the
> buffer was modified, what happens after is a new completion, or no
> completion at all (if circumstances told us we should abort).

Neither Company nor CAPF support refining the completion list by adding
more characters to the prefix/pattern (somewhat similar to
isearch). Completion-UI does, and it's very useful. Indeed, this is one
of the desired features Ted explicitly asked for in his recent post.

In general, this may require storing a bit more history about the
completion process, so you can continue from where you left off
correctly. At the very least, you need to store which completion backend
was used (which I think Company does too; CAPF makes this trickier, since
it doesn't have a simple way of identifying completion sources).

You've convinced me that prefix-replaced isn't needed, as long as UIs
aren't allowed to manipulate the buffer text. I need to think about it
more, but perhaps that could be a good restriction to impose as it
prevents UI bugs from mangling the buffer text.


> > And the core Completion-UI code that ties all the UIs together makes
> > heavy use of the `prefix-replaced' property. So even in Completion-UI
> > it's not only used by the dynamic interface.
> 
> Could you explain how?

I didn't impose any restriction on the UIs in what they're allowed to do
with the buffer text, so the core uses this e.g. when figuring out what
text it should leave behind in the buffer when a completion is accepted.

But I think imposing a restriction that UIs don't modify the buffer text
during the completion process is probably a good one, in which case it
may be unnecessary.

> > Similarly, `common-substring' demarcates the longest common substring if
> > completion text has been inserted in the buffer, which is information a
> > third-party UI might very well want to make use of.
> 
> Why? How?

Who knows how someone might want to make use of it in their UI? They
might want to highlight the common substring differently, say. But this
one's just for convenience anyway. A UI can recalculate it from the
completion list in any case.

> > The properties documented in the `completion-ui-register-interface'
> > docstring are those that have a standard meaning in Completion-UI. Note
> > that nothing prevents a particular UI widget from storing its own data in
> > its own widget-specific overlay properties if so desired.
> 
> Sure.
> 
> >> 2. `completion-source' seems to have to refer to a Completion-UI
> >> source, right?
> >>
> >> So how would one use an UI defined as such, in a different package?
> >> Would Company define itself as a new source? Or add a source per each
> >> backend?
> >
> > Depends on what the generic Emacs API for specifying completion
> > sources ends up looking like.  If we end up using c-a-p-f,
> > `completion-source' will need to store something that identifies the
> > c-a-p-f entry that returned the completion candidates for this
> > completion process.
> 
> Hmm, I was expecting something easier, since IIRC you said Company 
> should have been able to use it without major changes.

Maybe I misunderstood what you were asking. What I meant was, until we've
settled on the API for completion sources, we can't decide on the format
of the data that records which completion source ended up being used.

If you meant what would Company have done in the *current* Completion-UI
implementation, then yes: Company would have simply defined its backends
via the Completion-UI API, instead of via CAPF or company-backends.

But we seem to heading towards a different API for defining and selecting
completion sources. Probably CAPF, but perhaps something more similar to
company-backends.

If the former, we'll very likely want to record which CAPF source
returned the completions somehow. If the latter, then the equivalent of
the backend name is fine. (In the current Completion-UI implementation,
this property stores the completion source name, which modulo
implementation differences is directly equivalent to your backend names.)

I think you're reading way too much into the details of the current
Completion-UI implementation. The details will inevitably change if we
integrate it into part of a generic Emacs completion API. But whatever
the specifics of the data stored, and whatever we call the
variable/property that stores that data, we'll still need some way of
recording which completion source returned the completions we're dealing
with. Without this information, a number of features become tricky or
impossible to implement in the UI. As you surely know, since both Company
already records and uses information about which backend was used, just
like Completion-UI does.

> > If we end up using c-a-p-f, `completion-source'
> > will need to store something that identifies the c-a-p-f entry that
> > returned the completion candidates for this completion process.
> 
> Probably just the value returned by the successful completion function: 
> start, end, collection (aka completion table, which is often a function).

But how would we use that in the Customization interface, to allow users
to customize the UI in different ways for different sources?

It seems useful to have this feature, so users can make use it if they
want to. (See the use-case examples I gave in my reply to Stefan.) I'd be
sorry to lose the per-source customization feature. It's one I make light
but frequent use of myself. And users are bound to ask for something like
this later!

It's a bummer that CAPF doesn't associate unique names with completion
sources, in the way that Company and Completion-UI do.

One possibility might be to have CAPF functions optionally set a :name
property in the optional PROPS return value if they want to support
per-source UI customization for that source. What I don't see at the
moment is how Customize would discover what all the possible CAPF source
names are, in order to conveniently list them as options in a menu. (In
Completion-UI, this is one of the things handled by the macro used to
define new sources.) But that's a detail.


> >>> The best comparison one can make is that
> >>> company-backends loosely corresponds to a mixture of
> >>> completion-ui-source-definitions and auto-completion-source-functions.
> >>
> >> The latter: yes, the former: no. It's never used the same way as you use
> >> `completion-ui-source-definitions'.
> >
> > I said "loosely corresponds", and I stand by that. One of the things a
> > `company-backends' entry does is to define when a given completion source
> > should be used. In Completion-UI, that part of company-backends' role is
> > performed by auto-completion-source-functions.
> 
> Yes, that's why I agreed with this comparison (and also made it earlier, 
> IIRC). I disagree with the other comparison.

Sorry, my bad. I read what you wrote the wrong way around.

> > Could well be. But you seem to be arguing both ways. You also argue that
> > the c-a-p-f API is bad because it's opaque and hard for users to
> > configure. If backend definitions and selection logic are always supplied
> > by package authors, then the fact that c-a-p-f is opaque isn't so
> > significant.
> 
> The opaqueness in c-a-p-f is bad because the exact values of 
> `company-backends' and `completion-at-point-functions' are significant. 
> A third-party package author can only push a function in either of these 
> lists, but they can be responsible for the eventual order of the elements.
>
> And the order matters, because it influences which completion backend 
> will get picked when several are suitable. When we get to the grouped 
> backends, which I've mentioned several times, which Company supports, 
> and CAPF will hopefully support in the future, being able to understand 
> and maybe change the values of either list becomes even more important.

I completely agree.

How do you suggest we could improve it, without replacing CAPF or
breaking backwards compatibility?

This is something you care about more and have more experience of in
Company than I do with Predictive. I'm pretty confident that I can adapt
the Completion-UI code to whatever API we settle on for defining
completion sources and selection logic, without too much pain.

> >> Please take a look at `company-etags' and `company-dabbrev' and see if
> >> you can point out the situations when the user might find the `prefix'
> >> code of either too limiting.
> >
> > dabbrev and etags are both sources that, if they're useful in a given
> > buffer, they're useful in the entire buffer.
> >
> > Consider the LaTeX math mode source, and the LaTeX environment name
> 
> My question was, if you could point out problems with any of our bundled 
> backends (or, failing that, third-party ones). If they look okay, maybe

Did some text get cut off here in your reply?

> > source, and the LaTeX preamble source, etc. Trying to code the selection
> > logic for all of these using the Company API looks very awkward to me,
> > compared to the simplicity of setting a few buffer-local variables in
> > Completion-UI.
> 
> If performance is the problem, we could solve that by either:
> 
> 1) Adding some pre-completion hook which would allow you to run some 
> code once, set a buffer-local variable, which all backend functions for 
> LaTeX can refer to later in `prefix' action.

I don't think this can work, because the choice of source depends on the
location of point and the text in the buffer, which changes from
completion to completion.

For the Predictive LaTeX support, I use a mish mash of piggy-backing on
jit-lock where possible, some regexp match tests, and a limited form of
incremental parsing of the LaTeX code when the rest aren't enough. (I
used to exclusively use the latter, implemented using my auto-overlays
package. Surprisingly this was more than efficient enough to cause no
noticeable delay when typing and auto-completing, even on very old
hardware. But writing and debugging the incremental parser definitions
were a nightmare, largely because TeX doesn't have a well-defined
grammar. So I replaced as much as possible of the parsing with simpler
methods.)

This kind of thing is simpler to do in proper languages (assuming you
already have an incremental parser like CEDET, or a framework like
nxml-mode).


> 2) Create a special "merged" backend that would collect the results from 
> all LaTeX backend in a carefully defined fashion.

This effectively means moving the selection logic out of Company and into
Predictive. Which isn't necessarily a bad solution. It's how I used to do
it in Predictive, until I generalised the selection logic and moved it
into Completion-UI so other sources could make use of it.

If we stick with the CAPF API, I suspect I'll end up moving my source
selection logic back into Predictive, and making it Predictive-specific
again.

> 3) Indeed add some hook analogous to `auto-completion-source-functions'. 
> But yeah, CAPF already does that.
> 
> > Sure, I could move that logic into Predictive itself, and have a single
> > Predictive LaTeX backend. But that serves to demonstrate that the API
> > isn't flexible enough to let me do what I want easily. Other markup
> > languages and programming languages make similar demands on the API.
> 
> We bundle several backends, some of them for programming modes, and so 
> far you haven't pointed out specific problems with any of them.

Well, so far none of the existing Company backends are trying to do
auto-completion from a dictionary of 100,000 words, in a language with no
regular grammar, and where even tenths of a second lag are sufficient to
make typing unusable.

It would be an interesting experiment to stress-test the selection
mechanism by reimplementing Predictive's LaTeX support in Company. But it
would take some effort to code, and I don't have time (nor really the
interest) to code this up, especially when the Emacs API for this is
likely to look quite different.


> >> Or you can define new backends that would do some common checks in
> >> `prefix' (maybe calling an extracted function with common code) and
> >> simply delegate all other actions to the respective base backend.
> >> Implementing this is trivial.
> >
> > And once you've finished doing this, and factored out common selection
> > mechanisms like regexps, faces and text properties into utilities
> 
> Regexps we have already (company-grab-...),

How fast is this if you have to go through, say, 100 moderately complex
regexps checking for a match? (See the predictive-latex.el in the old
Predictive tarball release for examples.)

> instead of faces one should be using `syntax-ppss', font-lock isn't
> always available, and text properties... hmm.

Font-lock faces are *very* useful when you care about efficiency, because
of all the effort that's gone into optimising jit-look. (Font-lock has
very similar speed constraints to Predictive: it must not get in the way
of typing.)

> > functions...you'll have reimplemented something closer the Completion-UI
> > API or c-a-p-f :)
> 
> Only if we add a similar hook, see 3) above.
> 
> >> I understand the principle, really. But the more one "cleanly separates"
> >> code, the harder it can be sometimes to read it, to get the full picture.
> >
> > Indeed, which is why I listed grouping the completion and selection logic
> > into one place as one of the things I liked about Company's API.
> >
> > Perhaps the cleanest and most flexible solution would be to have a list
> > with entries of the form (TEST-FUNCTION . COMPLETION-FUNCTION),
> > COMPLETION-FUNCTION is used if TEST-FUNCTION returns non-nil.
> 
> I believe this suffers from the LaTeX problem you've described above. If 
> you have a dozen of completion functions for LaTeX, this scheme expects 
> you to have a dozen of corresponding test functions,

Indeed.

> and when one fails, the next will be called, and it won't be able to
> use the results of the previous call (unless they set and use some
> common buffer-local variable, which Company backends could also do; but
> that's ugly).

In my use-case, there's no useful information from previous tests anyway.

But I don't think this is such an important use-case to consider for a
generic API. Predictive has unusual requirements (always-on
auto-completion as the main use-case imposes extreme efficiency demands).
And (La)TeX and also Texinfo are odd cases, because their lack of a
regular grammar makes background incremental parsing more difficult
compared to programming languages.

Probably the best solution is to move the fast-selection logic I
implemented for use in LaTeX and Texinfo can back into Predictive.


> > And then supply a bunch of standard utility functions for use as
> > TEST-FUNCTION's, for testing regexp matches, faces, text-properties,
> > etc.
> 
> Like I described previously, such stand-alone tests probably won't be 
> very useful as values of this alist. One usually has to call several of 
> them to see if a completion-function is suitable.

I meant to write "for use *in* test functions". Simple utility functions
for testing regexps (which you already have), faces, etc. which can be
combined to build a test function.


> >> It makes certain amount of sense, although it looks like it could make
> >> creating a "merged" completion function more difficult.
> >
> > I doubt it'll be insurmountable. Also, merged completion functions are a
> > rather advanced feature that may not belong in core Emacs anyway (though
> > it would be good if the API supported them without ugly hacks).
> 
> It's a good feature enabling some kinds of backends that aren't usually 
> useful on their own (like Yasnippet). In my book, that's a good argument 
> to accept or reject an API.

I'd be very happy to see a solution in Emacs if you can come up with one
and convince people to merge it into core. As I said, adapting the
Completion-UI code to a different completion source API shouldn't be
difficult.

> >> We'll see.
> >
> > Indeed, we'll see. Whilst I'd be happy to see the somewhat complicated
> > and opaque c-a-p-f API replaced with something cleaner and simpler, I
> > don't see us winning that argument.
> 
> Personally, I'd probably be fine with c-a-p-f as long as it's powerful 
> enough. Moving to a less featureful API is likely out of the question, 
> but if it's demonstrated that c-a-p-f is fairly unsuitable for 
> implementing some features, I believe it would be a good reason to rule 
> it out.

In Completion-UI, I implemented combined completion sources through a
`completion-ui-combining-complete' wrapper function. (They're not
functionally equivalent to your merged sources, but the requirements are
somewhat similar.)

Would using something like that to build merged CAPF functions be a
solution for Company? I know this isn't particularly user-friendly if you
want to allow users to easily define their own merged sources. I don't
know if that's something people regularly do in Company.


As I said, it's not me you need to persuade if you want to replace
CAPF. I would be happy to see a nicer API than CAPF make its way into
Emacs core. But I also understand why there will be resistance from Emacs
devs to replacing an existing API unless there's a clear and pressing need.


> >> But the general approach, while flexible on its surface, complicates
> >> things if I intend to use any existing sources, written by third
> >> parties. Because their authors are unlikely to have anticipated the
> >> logic I'll add in my custom predicate function and to have written any
> >> code in their packages I might use. Or, at least, that's considerably
> >> less likely.
> >
> > I don't get your argument here. You have to wrap the third-party
> > completion function in Company in order to code the appropriate backend
> > selection logic. In Completion-UI, you put that code in an
> > `auto-completion-source-function' instead, and probably don't need to
> 
> Which code? There's likely to be none.
> 
> Hence, more effort required on my part.
> 
> Here I'm describing an organizational problem caused by an API. Not a 
> technical one.

I think we probably agree. Not worth pursuing this particular discussion
further really, since I'm not suggesting we use the Completion-UI API for
source selection.

There's one thing we should perhaps think a bit more about.

Is it right to say that the majority of the Company backends are selected
based on global properties of a buffer (e.g. major-modes)? This seems to
be the case for the default `company-backends' list. In Completion-UI I
was almost entirely focused on selecting backends based on local
properties at different locations within a buffer (regexps, faces,
syntax, etc.).

How do you envisage supporting local source selection in the new (or
enhanced CAPF) API? Would this kind of local completion source selection
always be implemented within a single backend, like the Company CEDET
backend? Or should there be a convenient way of supporting it in the
generic source selection API?

This kind of relates to your point about the ordering being important...


> >> Like mentioned above, delegating the search for completions to an
> >> existing backend is trivial. These are functions, and as such they are
> >> stateless. Just call (other-backend 'candidates current-prefix).
> >
> > But this results is convoluted and somewhat confusing code. If the
> > selection logic and completion function were separate, you wouldn't need
> > to use such hacks.
> 
> Looks straightforward to me. Function delegation is a rather simple concept.
> 
> We have examples of that in the Company frontends: take a look at 
> `company-pseudo-tooltip-unless-just-one-frontend' and 
> `company-preview-if-just-one-frontend'.
> 
> `company-dabbrev-code' also delegates to `company-dabbrev', but it just 
> uses a public function from that backend's package, which is also a 
> valid approach.

Following the code path is unnecessarily convoluted. I can't tell that
company-dabbrev-code internally delegates to the company-dabbrev backend
just from looking at `company-backends'. If it looked more like:

((use-dabbrev-code-p . dabbrev-completion)
 (use-dabbrev-p . dabbrev-completion))

it would be immediately obvious that these are two different ways of
using the same source of completions.

> > It understand why you're harping on about the limitations of the
> > default hook entries
> 
> I just can't see how you find them useful on their own. I'll shut up
> now. :)

Look at predictive-latex.el, which uses them on their own :)

But I'll shut up too. It seems clear to me from our discussion that we
should either use CAPF or something closer to Company for the completion
source selection API.

From a Completion-UI perspective, I don't much care what we
use. Switching between completion sources was never the main focus of
Completion-UI (the UI was the main focus). I should be able to adapt the
code to whatever we settle on. Most of it's abstracted into accessor
functions/macros internally, anyway.

> > Indeed, though this is starting to look a lot like defining some standard
> > hook functions for auto-completion-source-functions. As you make the
> > company-backends API more flexible and convenient, you'll increasingly
> > find yourself reimplementing equivalent functionality to that of c-a-p-f
> > and the Completion-UI API. If were to start simplifying the Completion-UI
> > API or c-a-p-f, it'll increasingly look more like the Company API. Maybe
> > the sweet spot is in the middle?
> 
> Maybe. I can certainly see myself adding a
> auto-completion-source-functions analog in Company, as an advanced
> feature.

The question for now is, what should go in the generic Emacs API? Vanilla
CAPF, perhaps with some additional standard PROPS properties? If the
generic Emacs API can't replace company-backends in Company, then in my
view the API is no good.

You're the expert here :)


> >> Actually, I was thinking about the former option. Let's define widgets
> >> with an API in the usual sense, so it can be used by both
> >> `completion-at-point' and external packages. We'll need this kind of API
> >> either way, in order to be able to write new widgets.
> >
> > I'm confused. `completion-at-point' will never use an API for *defining*
> > new completion UI widgets (`completion-ui-register-interface'). It will
> > need to be modified to *invoke* the new UI widgets.
> 
> Here I'm thinking in terms of Company approach, I guess. The API for 
> defining a backend or frontend and the way it is used are the same: a 
> protocol consisting of messages and proper reactions or responses to them.
> 
> Thus, "widget has to respond to..." would be in the API for defining a 
> widget, and "widgets respond to..." would be in the usage API. 
> Essentially the same.

Sounds pretty close to the Completion-UI widget API. A widget has to
provide "function that responds to x", "function that responds to y".

> If you like to add convenience macros, extra infrastructure, etc, they 
> may diverge, but we'll need an API a completion package can use easily 
> use anyway, not just a bunch of private functions that can change will 
> every release.

Absolutely.

The way I imagine it, completion packages like Company would rarely need
to use the API for defining new widgets. (Unless they're defining custom
widgets that are so package-specific they don't belong in core Emacs. But
that seems like a rare case.)

What they *will* all use is the API for *invoking* the completion
interface, which we began to discuss below.

> > Copying and adapting some of the code from `complete-in-buffer' into
> > `completion-in-region' would suffice for that. Then
> > `completion-in-region' would remain the generic Emacs API for displaying
> > and selecting completion candidates (only now it would display them in a
> > nicer interface).
> 
> Maybe. At a first glance, `completion-in-region' will need access to 
> PROPS returned by completion functions, not just COLLECTIONS.

Agreed. I figured we could add an additional PROPS optional argument, as
this wouldn't break backwards compatibility...

> >> Yes, but see above. Using 1. from Company would be the current next step
> >> toward integration, as I see it.
> >
> > Really? Why should we use the Company UI widget API and interfaces in
> > Emacs, when the API is less flexible and less customizable than that the
> > Completion-UI widgetAPI, and it implements fewer widgets?
> 
> Sorry for being unclear. I meant the reverse: Company would use the new 
> widget API defined here, while still retaining its backends, at least 
> initially.

Thanks, makes sense now.

> Although while you don't own the popup widget, we do have one. Maybe 
> porting that code won't be too hard.

Great! It would be a good test of the API too, to see how easy it is to
port.

Sounds like your equivalent of the "dynamic" interface is better, too
(you avoid modifying the buffer text). I'd like to either port the
Company version, or adapt the Completion-UI one to use the Company
mechanism (whichever is simpler).


> > Did you really mean that you wanted to throw Completion-UI in the
> > garbage, rewrite Company yourself to be suitable as a generic Emacs
> > completion API, rewrite/port the missing UIs and features, and persuade
> > people to merge it into Emacs core? (If so, great! Let me know when
> > you're done, and then I can stop maintaining Completion-UI and switch
> > Predictive over to the new interface :-)
> 
> I'll think about it. :) Not sure about the "persuade people to merge in 
> into Emacs core", though.

Ah, but that's the hard part - the rest is easy ;-)

> But you could help kick this process off my filing an issue describing 
> Company's backend API shortcomings. Is it just the lack of 
> `auto-completion-source-functions'? Non-prefix completion?

Erm, I don't see any major shortcomings in the API. Probably that hasn't
come across in the discussion :-)

Minor ones that have come here were:

- No support for non-prefix completion (but it looks easy enough to
  extend the API).

- Difficult (impossible?) to distinguish between "no completions
  available" and "backend not applicable".


> > Probably I misunderstood what you wrote.
> 
> Yep. :)

Shame! I was hoping you were volunteering to do all the work ;-)

> > I only really have one strong requirement: if some form of
> > auto-completion mode gets included in the Emacs completion UI, I will
> > argue hard for it to be as powerful as Completion-UI's
> > `auto-completion-mode'. (Take a look at `auto-completion-syntax-alist'
> > and `auto-completion-override-syntax-alist', and at the way all the UI
> > widgets are integrated.) Anything less powerful, and it will be useless
> > to me for Predictive.
> 
> I'll take a look, thanks.
> 
> >> Customizing hooks is a tricky business. I believe the opaqueness of
> >> c-a-p-f to the user is the main problem with the current API.
> >
> > But above you argued that backends and selection logic are supplied and
> > configured by package authors, not by users!? In which case package
> > authors can simply supply a c-a-p-f function, and users can add the
> > functions they want to c-a-p-f. (I believe Stefan made a similar point
> > earlier.)
> 
> Customizing a hook is tricky for user. Try `M-x customize-variable RET 
> find-file-hook'. You'll only see the buffer-local value, and not the 
> global one. But `run-hooks' uses both if the local value includes `t', 
> which it usually does.

I see the global value in Customize, even when I deliberately add a local
hook function. Showing the global value in Customize seems the correct
thing to do. Is that not what Emacs does? (I wouldn't know - I've never
Customized a hook. I always use `add-hook' in my .emacs for that.)

> Users can modify the hooks programmatically, of course, but that's a 
> step more difficult. And they'll also need to understand the values 
> already there, to be able to remove or rearrange them.
> 
> We'll also should discourage lambdas there. At the moment, my 
> find-file-hook contains this beauty, courtesy of autorevert.el:
> 
> #[nil "\302\301!\210\303\304\b!8\211\x11\207"
>    [buffer-file-name auto-revert-tail-pos make-local-variable 7 
> file-attributes]
>    3]
> 
> We should have less of that.

:-)

> > Don't get me wrong. I'm no fan of the complexity of c-a-p-f. But you
> > seem to be arguing both ways.
> 
> Of course I'm arguing toward a middle ground comfortable to me. So: 
> package authors deciding when their backend is suitable to use -- good; 
> users deciding the order of trying backends and their groupings -- also 
> good.

OK, makes sense. If I understand correctly, CAPF is fine for letting
users decide the order of trying backends. It's the groupings part that's
less clear, right?


> > Swings and roundabouts. I could always add a
> > `completion-ui-major-mode-source' function that checks an alist, and then
> > it would be:
> 
> Yes, well, here you are discarding the "standard, tried-and-tested Emacs 
> mechanisms" of major mode hooks and buffer-local values. Which was 
> exactly my point.

Now I'm confused. I thought that was *my* point ;-)


> > - foo completion function
> > - bar completion function
> > - foo predicate function
> > - bar predicate function
> > - add two entries to the alist
> 
> You forgot adding `completion-ui-major-mode-source' to the relevant list.
> 
> > - code 1st bundled completion function and selection logic
> > - code 2nd bundled completion function and selection logic (which
> >    duplicates the logic in the 1st one, with different parameters)
> >    ...
> > - code 8th bundled completion function and selection logic (which
> >    duplicates the logic in the last 7, with different parameters)
> 
> Could that be a one-line function call, in each case with different 
> arguments? Aside from potential performance problems, it sounds rather neat.

Not really. The selection logic for each source involves a list of
regexps, faces, etc. to test, and these are different for each
source. Passing lists of hard-coded regexps and faces around would be
ugly.

I guess you could store the regexps etc. in configuration variables (one
variable per source, or a single alist), and just pass an identifier as
an argument.

> > - add 8 entries to company-backends
> > - arrange for package-specific sources to be added to company-backends
> >    dynamically
> >
> > Contrast with Completion-UI's, which involves a *lot* less coding:
> >
> > - register 1st source (no need to code a new completion function, just
> >    set one parameter when calling `completion-ui-register-source')
> 
> You can likewise delegate to the same completion function in Company 
> backends. It's the same amount of code (1 line).

But in Completion-UI, you don't even have to write the code - the macro
does it for you :)

> > It would be almost trivial to switch Completion-UI over to
> > c-a-p-f. All I'd need to do to replicate the existing functionality
> > is add a few additional properties to the PROPS return argument.
> 
> Great. I'd like to see the patch. :)

I'll start working on it as soon as we agree on the completion source
API, and as soon as I have some spare time. (Which unfortunately isn't
likely to be before the end of term here in Cambridge, as I have a
lecture course to prepare from scratch. But the feature-freeze makes it
less pressing for now.)


> > Presumably I could always set a buffer-local `company-backends' from
> > predictive-mode that only contains relevant backends if necessary.
> 
> Yes, of course.
> 
> > c-a-p-f is trivially customizable by users, as long as packages / Company
> > supplies suitable c-a-p-f functions for the backends.
> 
> When you say "trivially customizable", do you mean via the Customize 
> interface?

I just meant that it's a straight list of functions, so it's easy for
users to understand how to configure it.

I take your point about configuring hooks via the customization interface
being tricky. But at least configuring hooks is standard Emacs foo. I
suspect most users who want to change the order of functions in CAPF will
know how to set hooks.

It would be nice to define a Customization interface that lists the
available CAPF functions in a drop-down menu (whilst also allowing
free-form entries, of course). If I find a way of porting the
Completion-UI auto-updating customization interface magic to CAPF, this
would be feasible...


> > *Writing* c-a-p-f is non-trivial, but as you said earlier those
> > are/should be supplied by package authors. Or have I misunderstood?
> 
> I don't think I said that about c-a-p-f, but yes, they should be.

Good, we're on the same page.

> >>> 3. The most popular and useful "list-the-available-completions" UI is
> >>>      popup.el. Is there any chance of getting copyright assignment for
> >>>      this? Or will we be forced to code something equivalent from scratch?
> >>
> >> https://github.com/auto-complete/popup-el/issues/50
> >
> > Fingers crossed...!
> 
> I wouldn't hold my breath: IME, getting a response from that crowd on 
> non-trivial issues is hard. Collecting copyright assignments could be 
> harder still.

Shame. I guess we'll be recoding it then. At least it won't be from
scratch, since you've already got something similar in Company.

> P.S. My email client displays each new message from you as a separate 
> thread, possibly because your email address is timestamped, or maybe 
> because they don't contain a "References" header. Could you do something 
> about that?

Strange. Mutt threads them fine for me. The ones I've checked all contain
an "In-Reply-To" header. I haven't changed the mutt reply header
settings, so presumably mutt doesn't add "References" headers by default?

Perhaps your email client is ignoring the In-Reply-To header? (I somehow
doubt the date stamps in the tagged addresses are the issue. Threads
always contain messages from different senders, so it threading should
work even if your mail client considers the changing dated addresses to
be different senders.)

All the best,
Toby
-- 
Dr T. S. Cubitt
Royal Society University Research Fellow
and Fellow of Churchill College, Cambridge
Centre for Quantum Information
DAMTP, University of Cambridge

email: tsc25@cantab.net
web:   www.dr-qubit.org

Re: Emacs completion matches selection UI

[Bob Bobeck]

This has got to be the longest stream of crap I've ever seen pushed
out on this mailing list. Are you all a bunch of novelists or are you
hackers? Pinch this turd off and start coding or shut the hell up.

On 1/7/14, Toby Cubitt <tsc25@cantab.net> wrote:
> On Wed, Jan 08, 2014 at 03:23:21AM +0400, Dmitry Gutov wrote:
>> On 07.01.2014 07:32, Toby Cubitt wrote:
>> >> Examples:
>> >> 1. `prefix-replaced' and `common-substring' both seem to be tailored
>> >> to
>> >> the dynamic interface.
>> >
>> > Not at all. `prefix-replaced' tells you that the buffer substring
>> > that is being completed has already been replaced with new text in
>> > the buffer. This is important for non-prefix completion (e.g. pattern
>> > matches).
>>
>> > I'm not sure if it's generally applicable.
> [...]
>> > Any UI that modifies the buffer text will almost certainly need to set
>> > `prefix-replaced';
>>
>> I don't really see why a non-prefix completion UI (or any other one) has
>> to modify the buffer text. Company has an inline "preview" frontend, but
>> it uses an overlay, and so the buffer text remains unmodified.
>
> Ah, you mean using the `before-string' or `after-string' overlay
> property? That does sound like a better implementation.
>
>
>> AFAIK, neither Company nor CAPF keep this kind of history: if the
>> buffer was modified, what happens after is a new completion, or no
>> completion at all (if circumstances told us we should abort).
>
> Neither Company nor CAPF support refining the completion list by adding
> more characters to the prefix/pattern (somewhat similar to
> isearch). Completion-UI does, and it's very useful. Indeed, this is one
> of the desired features Ted explicitly asked for in his recent post.
>
> In general, this may require storing a bit more history about the
> completion process, so you can continue from where you left off
> correctly. At the very least, you need to store which completion backend
> was used (which I think Company does too; CAPF makes this trickier, since
> it doesn't have a simple way of identifying completion sources).
>
> You've convinced me that prefix-replaced isn't needed, as long as UIs
> aren't allowed to manipulate the buffer text. I need to think about it
> more, but perhaps that could be a good restriction to impose as it
> prevents UI bugs from mangling the buffer text.
>
>
>> > And the core Completion-UI code that ties all the UIs together makes
>> > heavy use of the `prefix-replaced' property. So even in Completion-UI
>> > it's not only used by the dynamic interface.
>>
>> Could you explain how?
>
> I didn't impose any restriction on the UIs in what they're allowed to do
> with the buffer text, so the core uses this e.g. when figuring out what
> text it should leave behind in the buffer when a completion is accepted.
>
> But I think imposing a restriction that UIs don't modify the buffer text
> during the completion process is probably a good one, in which case it
> may be unnecessary.
>
>> > Similarly, `common-substring' demarcates the longest common substring
>> > if
>> > completion text has been inserted in the buffer, which is information a
>> > third-party UI might very well want to make use of.
>>
>> Why? How?
>
> Who knows how someone might want to make use of it in their UI? They
> might want to highlight the common substring differently, say. But this
> one's just for convenience anyway. A UI can recalculate it from the
> completion list in any case.
>
>> > The properties documented in the `completion-ui-register-interface'
>> > docstring are those that have a standard meaning in Completion-UI. Note
>> > that nothing prevents a particular UI widget from storing its own data
>> > in
>> > its own widget-specific overlay properties if so desired.
>>
>> Sure.
>>
>> >> 2. `completion-source' seems to have to refer to a Completion-UI
>> >> source, right?
>> >>
>> >> So how would one use an UI defined as such, in a different package?
>> >> Would Company define itself as a new source? Or add a source per each
>> >> backend?
>> >
>> > Depends on what the generic Emacs API for specifying completion
>> > sources ends up looking like.  If we end up using c-a-p-f,
>> > `completion-source' will need to store something that identifies the
>> > c-a-p-f entry that returned the completion candidates for this
>> > completion process.
>>
>> Hmm, I was expecting something easier, since IIRC you said Company
>> should have been able to use it without major changes.
>
> Maybe I misunderstood what you were asking. What I meant was, until we've
> settled on the API for completion sources, we can't decide on the format
> of the data that records which completion source ended up being used.
>
> If you meant what would Company have done in the *current* Completion-UI
> implementation, then yes: Company would have simply defined its backends
> via the Completion-UI API, instead of via CAPF or company-backends.
>
> But we seem to heading towards a different API for defining and selecting
> completion sources. Probably CAPF, but perhaps something more similar to
> company-backends.
>
> If the former, we'll very likely want to record which CAPF source
> returned the completions somehow. If the latter, then the equivalent of
> the backend name is fine. (In the current Completion-UI implementation,
> this property stores the completion source name, which modulo
> implementation differences is directly equivalent to your backend names.)
>
> I think you're reading way too much into the details of the current
> Completion-UI implementation. The details will inevitably change if we
> integrate it into part of a generic Emacs completion API. But whatever
> the specifics of the data stored, and whatever we call the
> variable/property that stores that data, we'll still need some way of
> recording which completion source returned the completions we're dealing
> with. Without this information, a number of features become tricky or
> impossible to implement in the UI. As you surely know, since both Company
> already records and uses information about which backend was used, just
> like Completion-UI does.
>
>> > If we end up using c-a-p-f, `completion-source'
>> > will need to store something that identifies the c-a-p-f entry that
>> > returned the completion candidates for this completion process.
>>
>> Probably just the value returned by the successful completion function:
>> start, end, collection (aka completion table, which is often a function).
>
> But how would we use that in the Customization interface, to allow users
> to customize the UI in different ways for different sources?
>
> It seems useful to have this feature, so users can make use it if they
> want to. (See the use-case examples I gave in my reply to Stefan.) I'd be
> sorry to lose the per-source customization feature. It's one I make light
> but frequent use of myself. And users are bound to ask for something like
> this later!
>
> It's a bummer that CAPF doesn't associate unique names with completion
> sources, in the way that Company and Completion-UI do.
>
> One possibility might be to have CAPF functions optionally set a :name
> property in the optional PROPS return value if they want to support
> per-source UI customization for that source. What I don't see at the
> moment is how Customize would discover what all the possible CAPF source
> names are, in order to conveniently list them as options in a menu. (In
> Completion-UI, this is one of the things handled by the macro used to
> define new sources.) But that's a detail.
>
>
>> >>> The best comparison one can make is that
>> >>> company-backends loosely corresponds to a mixture of
>> >>> completion-ui-source-definitions and
>> >>> auto-completion-source-functions.
>> >>
>> >> The latter: yes, the former: no. It's never used the same way as you
>> >> use
>> >> `completion-ui-source-definitions'.
>> >
>> > I said "loosely corresponds", and I stand by that. One of the things a
>> > `company-backends' entry does is to define when a given completion
>> > source
>> > should be used. In Completion-UI, that part of company-backends' role
>> > is
>> > performed by auto-completion-source-functions.
>>
>> Yes, that's why I agreed with this comparison (and also made it earlier,
>> IIRC). I disagree with the other comparison.
>
> Sorry, my bad. I read what you wrote the wrong way around.
>
>> > Could well be. But you seem to be arguing both ways. You also argue
>> > that
>> > the c-a-p-f API is bad because it's opaque and hard for users to
>> > configure. If backend definitions and selection logic are always
>> > supplied
>> > by package authors, then the fact that c-a-p-f is opaque isn't so
>> > significant.
>>
>> The opaqueness in c-a-p-f is bad because the exact values of
>> `company-backends' and `completion-at-point-functions' are significant.
>> A third-party package author can only push a function in either of these
>> lists, but they can be responsible for the eventual order of the
>> elements.
>>
>> And the order matters, because it influences which completion backend
>> will get picked when several are suitable. When we get to the grouped
>> backends, which I've mentioned several times, which Company supports,
>> and CAPF will hopefully support in the future, being able to understand
>> and maybe change the values of either list becomes even more important.
>
> I completely agree.
>
> How do you suggest we could improve it, without replacing CAPF or
> breaking backwards compatibility?
>
> This is something you care about more and have more experience of in
> Company than I do with Predictive. I'm pretty confident that I can adapt
> the Completion-UI code to whatever API we settle on for defining
> completion sources and selection logic, without too much pain.
>
>> >> Please take a look at `company-etags' and `company-dabbrev' and see if
>> >> you can point out the situations when the user might find the `prefix'
>> >> code of either too limiting.
>> >
>> > dabbrev and etags are both sources that, if they're useful in a given
>> > buffer, they're useful in the entire buffer.
>> >
>> > Consider the LaTeX math mode source, and the LaTeX environment name
>>
>> My question was, if you could point out problems with any of our bundled
>> backends (or, failing that, third-party ones). If they look okay, maybe
>
> Did some text get cut off here in your reply?
>
>> > source, and the LaTeX preamble source, etc. Trying to code the
>> > selection
>> > logic for all of these using the Company API looks very awkward to me,
>> > compared to the simplicity of setting a few buffer-local variables in
>> > Completion-UI.
>>
>> If performance is the problem, we could solve that by either:
>>
>> 1) Adding some pre-completion hook which would allow you to run some
>> code once, set a buffer-local variable, which all backend functions for
>> LaTeX can refer to later in `prefix' action.
>
> I don't think this can work, because the choice of source depends on the
> location of point and the text in the buffer, which changes from
> completion to completion.
>
> For the Predictive LaTeX support, I use a mish mash of piggy-backing on
> jit-lock where possible, some regexp match tests, and a limited form of
> incremental parsing of the LaTeX code when the rest aren't enough. (I
> used to exclusively use the latter, implemented using my auto-overlays
> package. Surprisingly this was more than efficient enough to cause no
> noticeable delay when typing and auto-completing, even on very old
> hardware. But writing and debugging the incremental parser definitions
> were a nightmare, largely because TeX doesn't have a well-defined
> grammar. So I replaced as much as possible of the parsing with simpler
> methods.)
>
> This kind of thing is simpler to do in proper languages (assuming you
> already have an incremental parser like CEDET, or a framework like
> nxml-mode).
>
>
>> 2) Create a special "merged" backend that would collect the results from
>> all LaTeX backend in a carefully defined fashion.
>
> This effectively means moving the selection logic out of Company and into
> Predictive. Which isn't necessarily a bad solution. It's how I used to do
> it in Predictive, until I generalised the selection logic and moved it
> into Completion-UI so other sources could make use of it.
>
> If we stick with the CAPF API, I suspect I'll end up moving my source
> selection logic back into Predictive, and making it Predictive-specific
> again.
>
>> 3) Indeed add some hook analogous to `auto-completion-source-functions'.
>> But yeah, CAPF already does that.
>>
>> > Sure, I could move that logic into Predictive itself, and have a single
>> > Predictive LaTeX backend. But that serves to demonstrate that the API
>> > isn't flexible enough to let me do what I want easily. Other markup
>> > languages and programming languages make similar demands on the API.
>>
>> We bundle several backends, some of them for programming modes, and so
>> far you haven't pointed out specific problems with any of them.
>
> Well, so far none of the existing Company backends are trying to do
> auto-completion from a dictionary of 100,000 words, in a language with no
> regular grammar, and where even tenths of a second lag are sufficient to
> make typing unusable.
>
> It would be an interesting experiment to stress-test the selection
> mechanism by reimplementing Predictive's LaTeX support in Company. But it
> would take some effort to code, and I don't have time (nor really the
> interest) to code this up, especially when the Emacs API for this is
> likely to look quite different.
>
>
>> >> Or you can define new backends that would do some common checks in
>> >> `prefix' (maybe calling an extracted function with common code) and
>> >> simply delegate all other actions to the respective base backend.
>> >> Implementing this is trivial.
>> >
>> > And once you've finished doing this, and factored out common selection
>> > mechanisms like regexps, faces and text properties into utilities
>>
>> Regexps we have already (company-grab-...),
>
> How fast is this if you have to go through, say, 100 moderately complex
> regexps checking for a match? (See the predictive-latex.el in the old
> Predictive tarball release for examples.)
>
>> instead of faces one should be using `syntax-ppss', font-lock isn't
>> always available, and text properties... hmm.
>
> Font-lock faces are *very* useful when you care about efficiency, because
> of all the effort that's gone into optimising jit-look. (Font-lock has
> very similar speed constraints to Predictive: it must not get in the way
> of typing.)
>
>> > functions...you'll have reimplemented something closer the
>> > Completion-UI
>> > API or c-a-p-f :)
>>
>> Only if we add a similar hook, see 3) above.
>>
>> >> I understand the principle, really. But the more one "cleanly
>> >> separates"
>> >> code, the harder it can be sometimes to read it, to get the full
>> >> picture.
>> >
>> > Indeed, which is why I listed grouping the completion and selection
>> > logic
>> > into one place as one of the things I liked about Company's API.
>> >
>> > Perhaps the cleanest and most flexible solution would be to have a list
>> > with entries of the form (TEST-FUNCTION . COMPLETION-FUNCTION),
>> > COMPLETION-FUNCTION is used if TEST-FUNCTION returns non-nil.
>>
>> I believe this suffers from the LaTeX problem you've described above. If
>> you have a dozen of completion functions for LaTeX, this scheme expects
>> you to have a dozen of corresponding test functions,
>
> Indeed.
>
>> and when one fails, the next will be called, and it won't be able to
>> use the results of the previous call (unless they set and use some
>> common buffer-local variable, which Company backends could also do; but
>> that's ugly).
>
> In my use-case, there's no useful information from previous tests anyway.
>
> But I don't think this is such an important use-case to consider for a
> generic API. Predictive has unusual requirements (always-on
> auto-completion as the main use-case imposes extreme efficiency demands).
> And (La)TeX and also Texinfo are odd cases, because their lack of a
> regular grammar makes background incremental parsing more difficult
> compared to programming languages.
>
> Probably the best solution is to move the fast-selection logic I
> implemented for use in LaTeX and Texinfo can back into Predictive.
>
>
>> > And then supply a bunch of standard utility functions for use as
>> > TEST-FUNCTION's, for testing regexp matches, faces, text-properties,
>> > etc.
>>
>> Like I described previously, such stand-alone tests probably won't be
>> very useful as values of this alist. One usually has to call several of
>> them to see if a completion-function is suitable.
>
> I meant to write "for use *in* test functions". Simple utility functions
> for testing regexps (which you already have), faces, etc. which can be
> combined to build a test function.
>
>
>> >> It makes certain amount of sense, although it looks like it could make
>> >> creating a "merged" completion function more difficult.
>> >
>> > I doubt it'll be insurmountable. Also, merged completion functions are
>> > a
>> > rather advanced feature that may not belong in core Emacs anyway
>> > (though
>> > it would be good if the API supported them without ugly hacks).
>>
>> It's a good feature enabling some kinds of backends that aren't usually
>> useful on their own (like Yasnippet). In my book, that's a good argument
>> to accept or reject an API.
>
> I'd be very happy to see a solution in Emacs if you can come up with one
> and convince people to merge it into core. As I said, adapting the
> Completion-UI code to a different completion source API shouldn't be
> difficult.
>
>> >> We'll see.
>> >
>> > Indeed, we'll see. Whilst I'd be happy to see the somewhat complicated
>> > and opaque c-a-p-f API replaced with something cleaner and simpler, I
>> > don't see us winning that argument.
>>
>> Personally, I'd probably be fine with c-a-p-f as long as it's powerful
>> enough. Moving to a less featureful API is likely out of the question,
>> but if it's demonstrated that c-a-p-f is fairly unsuitable for
>> implementing some features, I believe it would be a good reason to rule
>> it out.
>
> In Completion-UI, I implemented combined completion sources through a
> `completion-ui-combining-complete' wrapper function. (They're not
> functionally equivalent to your merged sources, but the requirements are
> somewhat similar.)
>
> Would using something like that to build merged CAPF functions be a
> solution for Company? I know this isn't particularly user-friendly if you
> want to allow users to easily define their own merged sources. I don't
> know if that's something people regularly do in Company.
>
>
> As I said, it's not me you need to persuade if you want to replace
> CAPF. I would be happy to see a nicer API than CAPF make its way into
> Emacs core. But I also understand why there will be resistance from Emacs
> devs to replacing an existing API unless there's a clear and pressing need.
>
>
>> >> But the general approach, while flexible on its surface, complicates
>> >> things if I intend to use any existing sources, written by third
>> >> parties. Because their authors are unlikely to have anticipated the
>> >> logic I'll add in my custom predicate function and to have written any
>> >> code in their packages I might use. Or, at least, that's considerably
>> >> less likely.
>> >
>> > I don't get your argument here. You have to wrap the third-party
>> > completion function in Company in order to code the appropriate backend
>> > selection logic. In Completion-UI, you put that code in an
>> > `auto-completion-source-function' instead, and probably don't need to
>>
>> Which code? There's likely to be none.
>>
>> Hence, more effort required on my part.
>>
>> Here I'm describing an organizational problem caused by an API. Not a
>> technical one.
>
> I think we probably agree. Not worth pursuing this particular discussion
> further really, since I'm not suggesting we use the Completion-UI API for
> source selection.
>
> There's one thing we should perhaps think a bit more about.
>
> Is it right to say that the majority of the Company backends are selected
> based on global properties of a buffer (e.g. major-modes)? This seems to
> be the case for the default `company-backends' list. In Completion-UI I
> was almost entirely focused on selecting backends based on local
> properties at different locations within a buffer (regexps, faces,
> syntax, etc.).
>
> How do you envisage supporting local source selection in the new (or
> enhanced CAPF) API? Would this kind of local completion source selection
> always be implemented within a single backend, like the Company CEDET
> backend? Or should there be a convenient way of supporting it in the
> generic source selection API?
>
> This kind of relates to your point about the ordering being important...
>
>
>> >> Like mentioned above, delegating the search for completions to an
>> >> existing backend is trivial. These are functions, and as such they are
>> >> stateless. Just call (other-backend 'candidates current-prefix).
>> >
>> > But this results is convoluted and somewhat confusing code. If the
>> > selection logic and completion function were separate, you wouldn't
>> > need
>> > to use such hacks.
>>
>> Looks straightforward to me. Function delegation is a rather simple
>> concept.
>>
>> We have examples of that in the Company frontends: take a look at
>> `company-pseudo-tooltip-unless-just-one-frontend' and
>> `company-preview-if-just-one-frontend'.
>>
>> `company-dabbrev-code' also delegates to `company-dabbrev', but it just
>> uses a public function from that backend's package, which is also a
>> valid approach.
>
> Following the code path is unnecessarily convoluted. I can't tell that
> company-dabbrev-code internally delegates to the company-dabbrev backend
> just from looking at `company-backends'. If it looked more like:
>
> ((use-dabbrev-code-p . dabbrev-completion)
>  (use-dabbrev-p . dabbrev-completion))
>
> it would be immediately obvious that these are two different ways of
> using the same source of completions.
>
>> > It understand why you're harping on about the limitations of the
>> > default hook entries
>>
>> I just can't see how you find them useful on their own. I'll shut up
>> now. :)
>
> Look at predictive-latex.el, which uses them on their own :)
>
> But I'll shut up too. It seems clear to me from our discussion that we
> should either use CAPF or something closer to Company for the completion
> source selection API.
>
> From a Completion-UI perspective, I don't much care what we
> use. Switching between completion sources was never the main focus of
> Completion-UI (the UI was the main focus). I should be able to adapt the
> code to whatever we settle on. Most of it's abstracted into accessor
> functions/macros internally, anyway.
>
>> > Indeed, though this is starting to look a lot like defining some
>> > standard
>> > hook functions for auto-completion-source-functions. As you make the
>> > company-backends API more flexible and convenient, you'll increasingly
>> > find yourself reimplementing equivalent functionality to that of
>> > c-a-p-f
>> > and the Completion-UI API. If were to start simplifying the
>> > Completion-UI
>> > API or c-a-p-f, it'll increasingly look more like the Company API.
>> > Maybe
>> > the sweet spot is in the middle?
>>
>> Maybe. I can certainly see myself adding a
>> auto-completion-source-functions analog in Company, as an advanced
>> feature.
>
> The question for now is, what should go in the generic Emacs API? Vanilla
> CAPF, perhaps with some additional standard PROPS properties? If the
> generic Emacs API can't replace company-backends in Company, then in my
> view the API is no good.
>
> You're the expert here :)
>
>
>> >> Actually, I was thinking about the former option. Let's define widgets
>> >> with an API in the usual sense, so it can be used by both
>> >> `completion-at-point' and external packages. We'll need this kind of
>> >> API
>> >> either way, in order to be able to write new widgets.
>> >
>> > I'm confused. `completion-at-point' will never use an API for
>> > *defining*
>> > new completion UI widgets (`completion-ui-register-interface'). It will
>> > need to be modified to *invoke* the new UI widgets.
>>
>> Here I'm thinking in terms of Company approach, I guess. The API for
>> defining a backend or frontend and the way it is used are the same: a
>> protocol consisting of messages and proper reactions or responses to
>> them.
>>
>> Thus, "widget has to respond to..." would be in the API for defining a
>> widget, and "widgets respond to..." would be in the usage API.
>> Essentially the same.
>
> Sounds pretty close to the Completion-UI widget API. A widget has to
> provide "function that responds to x", "function that responds to y".
>
>> If you like to add convenience macros, extra infrastructure, etc, they
>> may diverge, but we'll need an API a completion package can use easily
>> use anyway, not just a bunch of private functions that can change will
>> every release.
>
> Absolutely.
>
> The way I imagine it, completion packages like Company would rarely need
> to use the API for defining new widgets. (Unless they're defining custom
> widgets that are so package-specific they don't belong in core Emacs. But
> that seems like a rare case.)
>
> What they *will* all use is the API for *invoking* the completion
> interface, which we began to discuss below.
>
>> > Copying and adapting some of the code from `complete-in-buffer' into
>> > `completion-in-region' would suffice for that. Then
>> > `completion-in-region' would remain the generic Emacs API for
>> > displaying
>> > and selecting completion candidates (only now it would display them in
>> > a
>> > nicer interface).
>>
>> Maybe. At a first glance, `completion-in-region' will need access to
>> PROPS returned by completion functions, not just COLLECTIONS.
>
> Agreed. I figured we could add an additional PROPS optional argument, as
> this wouldn't break backwards compatibility...
>
>> >> Yes, but see above. Using 1. from Company would be the current next
>> >> step
>> >> toward integration, as I see it.
>> >
>> > Really? Why should we use the Company UI widget API and interfaces in
>> > Emacs, when the API is less flexible and less customizable than that
>> > the
>> > Completion-UI widgetAPI, and it implements fewer widgets?
>>
>> Sorry for being unclear. I meant the reverse: Company would use the new
>> widget API defined here, while still retaining its backends, at least
>> initially.
>
> Thanks, makes sense now.
>
>> Although while you don't own the popup widget, we do have one. Maybe
>> porting that code won't be too hard.
>
> Great! It would be a good test of the API too, to see how easy it is to
> port.
>
> Sounds like your equivalent of the "dynamic" interface is better, too
> (you avoid modifying the buffer text). I'd like to either port the
> Company version, or adapt the Completion-UI one to use the Company
> mechanism (whichever is simpler).
>
>
>> > Did you really mean that you wanted to throw Completion-UI in the
>> > garbage, rewrite Company yourself to be suitable as a generic Emacs
>> > completion API, rewrite/port the missing UIs and features, and persuade
>> > people to merge it into Emacs core? (If so, great! Let me know when
>> > you're done, and then I can stop maintaining Completion-UI and switch
>> > Predictive over to the new interface :-)
>>
>> I'll think about it. :) Not sure about the "persuade people to merge in
>> into Emacs core", though.
>
> Ah, but that's the hard part - the rest is easy ;-)
>
>> But you could help kick this process off my filing an issue describing
>> Company's backend API shortcomings. Is it just the lack of
>> `auto-completion-source-functions'? Non-prefix completion?
>
> Erm, I don't see any major shortcomings in the API. Probably that hasn't
> come across in the discussion :-)
>
> Minor ones that have come here were:
>
> - No support for non-prefix completion (but it looks easy enough to
>   extend the API).
>
> - Difficult (impossible?) to distinguish between "no completions
>   available" and "backend not applicable".
>
>
>> > Probably I misunderstood what you wrote.
>>
>> Yep. :)
>
> Shame! I was hoping you were volunteering to do all the work ;-)
>
>> > I only really have one strong requirement: if some form of
>> > auto-completion mode gets included in the Emacs completion UI, I will
>> > argue hard for it to be as powerful as Completion-UI's
>> > `auto-completion-mode'. (Take a look at `auto-completion-syntax-alist'
>> > and `auto-completion-override-syntax-alist', and at the way all the UI
>> > widgets are integrated.) Anything less powerful, and it will be useless
>> > to me for Predictive.
>>
>> I'll take a look, thanks.
>>
>> >> Customizing hooks is a tricky business. I believe the opaqueness of
>> >> c-a-p-f to the user is the main problem with the current API.
>> >
>> > But above you argued that backends and selection logic are supplied and
>> > configured by package authors, not by users!? In which case package
>> > authors can simply supply a c-a-p-f function, and users can add the
>> > functions they want to c-a-p-f. (I believe Stefan made a similar point
>> > earlier.)
>>
>> Customizing a hook is tricky for user. Try `M-x customize-variable RET
>> find-file-hook'. You'll only see the buffer-local value, and not the
>> global one. But `run-hooks' uses both if the local value includes `t',
>> which it usually does.
>
> I see the global value in Customize, even when I deliberately add a local
> hook function. Showing the global value in Customize seems the correct
> thing to do. Is that not what Emacs does? (I wouldn't know - I've never
> Customized a hook. I always use `add-hook' in my .emacs for that.)
>
>> Users can modify the hooks programmatically, of course, but that's a
>> step more difficult. And they'll also need to understand the values
>> already there, to be able to remove or rearrange them.
>>
>> We'll also should discourage lambdas there. At the moment, my
>> find-file-hook contains this beauty, courtesy of autorevert.el:
>>
>> #[nil "\302\301!\210\303\304 !8\211 \207"
>>    [buffer-file-name auto-revert-tail-pos make-local-variable 7
>> file-attributes]
>>    3]
>>
>> We should have less of that.
>
> :-)
>
>> > Don't get me wrong. I'm no fan of the complexity of c-a-p-f. But you
>> > seem to be arguing both ways.
>>
>> Of course I'm arguing toward a middle ground comfortable to me. So:
>> package authors deciding when their backend is suitable to use -- good;
>> users deciding the order of trying backends and their groupings -- also
>> good.
>
> OK, makes sense. If I understand correctly, CAPF is fine for letting
> users decide the order of trying backends. It's the groupings part that's
> less clear, right?
>
>
>> > Swings and roundabouts. I could always add a
>> > `completion-ui-major-mode-source' function that checks an alist, and
>> > then
>> > it would be:
>>
>> Yes, well, here you are discarding the "standard, tried-and-tested Emacs
>> mechanisms" of major mode hooks and buffer-local values. Which was
>> exactly my point.
>
> Now I'm confused. I thought that was *my* point ;-)
>
>
>> > - foo completion function
>> > - bar completion function
>> > - foo predicate function
>> > - bar predicate function
>> > - add two entries to the alist
>>
>> You forgot adding `completion-ui-major-mode-source' to the relevant list.
>>
>> > - code 1st bundled completion function and selection logic
>> > - code 2nd bundled completion function and selection logic (which
>> >    duplicates the logic in the 1st one, with different parameters)
>> >    ...
>> > - code 8th bundled completion function and selection logic (which
>> >    duplicates the logic in the last 7, with different parameters)
>>
>> Could that be a one-line function call, in each case with different
>> arguments? Aside from potential performance problems, it sounds rather
>> neat.
>
> Not really. The selection logic for each source involves a list of
> regexps, faces, etc. to test, and these are different for each
> source. Passing lists of hard-coded regexps and faces around would be
> ugly.
>
> I guess you could store the regexps etc. in configuration variables (one
> variable per source, or a single alist), and just pass an identifier as
> an argument.
>
>> > - add 8 entries to company-backends
>> > - arrange for package-specific sources to be added to company-backends
>> >    dynamically
>> >
>> > Contrast with Completion-UI's, which involves a *lot* less coding:
>> >
>> > - register 1st source (no need to code a new completion function, just
>> >    set one parameter when calling `completion-ui-register-source')
>>
>> You can likewise delegate to the same completion function in Company
>> backends. It's the same amount of code (1 line).
>
> But in Completion-UI, you don't even have to write the code - the macro
> does it for you :)
>
>> > It would be almost trivial to switch Completion-UI over to
>> > c-a-p-f. All I'd need to do to replicate the existing functionality
>> > is add a few additional properties to the PROPS return argument.
>>
>> Great. I'd like to see the patch. :)
>
> I'll start working on it as soon as we agree on the completion source
> API, and as soon as I have some spare time. (Which unfortunately isn't
> likely to be before the end of term here in Cambridge, as I have a
> lecture course to prepare from scratch. But the feature-freeze makes it
> less pressing for now.)
>
>
>> > Presumably I could always set a buffer-local `company-backends' from
>> > predictive-mode that only contains relevant backends if necessary.
>>
>> Yes, of course.
>>
>> > c-a-p-f is trivially customizable by users, as long as packages /
>> > Company
>> > supplies suitable c-a-p-f functions for the backends.
>>
>> When you say "trivially customizable", do you mean via the Customize
>> interface?
>
> I just meant that it's a straight list of functions, so it's easy for
> users to understand how to configure it.
>
> I take your point about configuring hooks via the customization interface
> being tricky. But at least configuring hooks is standard Emacs foo. I
> suspect most users who want to change the order of functions in CAPF will
> know how to set hooks.
>
> It would be nice to define a Customization interface that lists the
> available CAPF functions in a drop-down menu (whilst also allowing
> free-form entries, of course). If I find a way of porting the
> Completion-UI auto-updating customization interface magic to CAPF, this
> would be feasible...
>
>
>> > *Writing* c-a-p-f is non-trivial, but as you said earlier those
>> > are/should be supplied by package authors. Or have I misunderstood?
>>
>> I don't think I said that about c-a-p-f, but yes, they should be.
>
> Good, we're on the same page.
>
>> >>> 3. The most popular and useful "list-the-available-completions" UI is
>> >>>      popup.el. Is there any chance of getting copyright assignment
>> >>> for
>> >>>      this? Or will we be forced to code something equivalent from
>> >>> scratch?
>> >>
>> >> https://github.com/auto-complete/popup-el/issues/50
>> >
>> > Fingers crossed...!
>>
>> I wouldn't hold my breath: IME, getting a response from that crowd on
>> non-trivial issues is hard. Collecting copyright assignments could be
>> harder still.
>
> Shame. I guess we'll be recoding it then. At least it won't be from
> scratch, since you've already got something similar in Company.
>
>> P.S. My email client displays each new message from you as a separate
>> thread, possibly because your email address is timestamped, or maybe
>> because they don't contain a "References" header. Could you do something
>> about that?
>
> Strange. Mutt threads them fine for me. The ones I've checked all contain
> an "In-Reply-To" header. I haven't changed the mutt reply header
> settings, so presumably mutt doesn't add "References" headers by default?
>
> Perhaps your email client is ignoring the In-Reply-To header? (I somehow
> doubt the date stamps in the tagged addresses are the issue. Threads
> always contain messages from different senders, so it threading should
> work even if your mail client considers the changing dated addresses to
> be different senders.)
>
> All the best,
> Toby
> --
> Dr T. S. Cubitt
> Royal Society University Research Fellow
> and Fellow of Churchill College, Cambridge
> Centre for Quantum Information
> DAMTP, University of Cambridge
>
> email: tsc25@cantab.net
> web:   www.dr-qubit.org
>
>

Re: Emacs completion matches selection UI

[Toby Cubitt]

On Tue, Jan 07, 2014 at 10:38:40PM -0500, Bob Bobeck wrote:
> This has got to be the longest stream of crap I've ever seen pushed
> out on this mailing list.

On Fri, Jan 03, 2014 at 12:49:35PM -0500, Ted Zlatanov wrote:
TZ>  The discussion should remain on emacs-devel (based on the wide
TZ>  interest so far).

> Are you all a bunch of novelists or are you hackers? Pinch this turd
> off and start coding or shut the hell up.

Congratulations. You've made in into my kill-file.

Goodbye.

-- 
Dr T. S. Cubitt
Royal Society University Research Fellow
and Fellow of Churchill College, Cambridge
Centre for Quantum Information
DAMTP, University of Cambridge

email: tsc25@cantab.net
web:   www.dr-qubit.org

Re: Emacs completion matches selection UI

[Josh]

On Tue, Jan 7, 2014 at 11:42 AM, David Kastrup <dak@gnu.org> wrote:
> Josh <josh@foxtail.org> writes:
>
>> On Tue, Jan 7, 2014 at 9:18 AM, João Távora <joaot@siscog.pt> wrote:
>>> Are any functions available that return the keymap lookup sequence used
>>> normally ? I wish there was something like this, but it seems hidden
>>> behind functions like `key-binding'.
>>
>> Related: http://debbugs.gnu.org/cgi/bugreport.cgi?bug=13948
>
> current-active-maps is a built-in function in `C source code'.
>
> (current-active-maps &optional OLP POSITION)
>
> Return a list of the currently active keymaps.
> OLP if non-nil indicates that we should obey `overriding-local-map' and
> `overriding-terminal-local-map'.  POSITION can specify a click position
> like in the respective argument of `key-binding'.

I can easily imagine current-active-maps playing a role in the
implementation of a solution to the bug I linked, but I hope we agree
that it is not a solution in itself.  From what I've seen, the confusion
and frustration of less experienced Emacsers in this area is most often
related to key binding precedence/shadowing, and this motivated my
suggested change to `describe-key' output[0].

[0] http://debbugs.gnu.org/cgi/bugreport.cgi?bug=13948#8

Re: Emacs completion matches selection UI

[Stefan Monnier]

> Neither Company nor CAPF support refining the completion list by adding
> more characters to the prefix/pattern (somewhat similar to
> isearch).

FWIW, this feature is indeed currently not provided by the
completion-at-point UI, but that is not because of
completion-at-point-functions (CAPF) which only provides the backends.


        Stefan

Re: Emacs completion matches selection UI

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

    This has got to be the longest stream of crap I've ever seen pushed
    out on this mailing list. Are you all a bunch of novelists or are you
    hackers? Pinch this turd off and start coding or shut the hell up.

They are talking about a substantive issue, and without rancor.  If
you're not interested in that particular issue, please don't intervene
with hostility.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Emacs completion matches selection UI

[Ted Zlatanov]

On Tue, 07 Jan 2014 23:49:27 -0500 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 

>> Neither Company nor CAPF support refining the completion list by adding
>> more characters to the prefix/pattern (somewhat similar to
>> isearch).

SM> FWIW, this feature is indeed currently not provided by the
SM> completion-at-point UI, but that is not because of
SM> completion-at-point-functions (CAPF) which only provides the backends.

Yes.  Narrowing the list of candidates should be a frontend feature.

Ted

Re: Emacs completion matches selection UI

[Ted Zlatanov]

On Tue, 7 Jan 2014 22:38:40 -0500 Bob Bobeck <bob.bobeck@gmail.com> wrote: 

BB> This has got to be the longest stream of crap I've ever seen pushed
BB> out on this mailing list. Are you all a bunch of novelists or are you
BB> hackers? Pinch this turd off and start coding or shut the hell up.

I strain to find a point in your loaded message.

Number two, we're in feature freeze, so we're circling around new ideas
before we drop features into the trunk.

Ted "probably should have sent this anonymously"

Re: Emacs completion matches selection UI

[Toby Cubitt]

On Wed, Jan 08, 2014 at 10:33:02AM -0500, Ted Zlatanov wrote:
> On Tue, 07 Jan 2014 23:49:27 -0500 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 
> 
> >> Neither Company nor CAPF support refining the completion list by adding
> >> more characters to the prefix/pattern (somewhat similar to
> >> isearch).
> 
> SM> FWIW, this feature is indeed currently not provided by the
> SM> completion-at-point UI, but that is not because of
> SM> completion-at-point-functions (CAPF) which only provides the backends.
> 
> Yes.  Narrowing the list of candidates should be a frontend feature.

Yes, that's what I meant. In Completion-UI it is implemented as a
frontend feature.

Toby
-- 
Dr T. S. Cubitt
Royal Society University Research Fellow
and Fellow of Churchill College, Cambridge
Centre for Quantum Information
DAMTP, University of Cambridge

email: tsc25@cantab.net
web:   www.dr-qubit.org

Re: Emacs completion matches selection UI

[Dmitry Gutov]

On 08.01.2014 06:58, Toby Cubitt wrote:
>> I don't really see why a non-prefix completion UI (or any other one) has
>> to modify the buffer text. Company has an inline "preview" frontend, but
>> it uses an overlay, and so the buffer text remains unmodified.
>
> Ah, you mean using the `before-string' or `after-string' overlay
> property? That does sound like a better implementation.

Yep. Company's popup and popup.el are also implemented using this property.

> Neither Company nor CAPF support refining the completion list by adding
> more characters to the prefix/pattern (somewhat similar to
> isearch). Completion-UI does, and it's very useful. Indeed, this is one
> of the desired features Ted explicitly asked for in his recent post.

Company has as isearch-like function (press `C-s' when you see the 
completions popup). It doesn't *add* to the pattern as much as just 
searches within the candidates list. It could refine it, though.
Again, there's no buffer modification going on during that.

> At the very least, you need to store which completion backend
> was used (which I think Company does too; CAPF makes this trickier, since
> it doesn't have a simple way of identifying completion sources).

You may be able to get away with just storing the value returned by the 
completion function. Instead of refreshing the completion boundaries 
after each command, you could just assume that they stay in place, and 
the closing boundary just moves according to the amount of text 
inserted/removed during completion.

> If the former, we'll very likely want to record which CAPF source
> returned the completions somehow. If the latter, then the equivalent of
> the backend name is fine. (In the current Completion-UI implementation,
> this property stores the completion source name, which modulo
> implementation differences is directly equivalent to your backend names.)

Come to think of it, that should be easy to do in 
`completion--capf-wrapper', as long as FUN is a symbol, not an anonymous 
function.

> But how would we use that in the Customization interface, to allow users
> to customize the UI in different ways for different sources?

Hm, maybe the function name too, see above. We'll need to retain all 
PROPS anyway.

> What I don't see at the
> moment is how Customize would discover what all the possible CAPF source
> names are, in order to conveniently list them as options in a menu.

I don't really see that happening at all.

>> The opaqueness in c-a-p-f is bad because the exact values of
>> `company-backends' and `completion-at-point-functions' are significant.
>> A third-party package author can only push a function in either of these
>> lists, but they can be responsible for the eventual order of the elements.
>>
>> And the order matters, because it influences which completion backend
>> will get picked when several are suitable. When we get to the grouped
>> backends, which I've mentioned several times, which Company supports,
>> and CAPF will hopefully support in the future, being able to understand
>> and maybe change the values of either list becomes even more important.
>
> I completely agree.
>
> How do you suggest we could improve it, without replacing CAPF or
> breaking backwards compatibility?
>
> This is something you care about more and have more experience of in
> Company than I do with Predictive. I'm pretty confident that I can adapt
> the Completion-UI code to whatever API we settle on for defining
> completion sources and selection logic, without too much pain.

Sorry, no idea. `company-backends' is not a hook, and this simple nature 
makes it more transparent, while somewhat less powerful. Making c-a-p-f 
less powerful is probably a non-starter.

>> My question was, if you could point out problems with any of our bundled
>> backends (or, failing that, third-party ones). If they look okay, maybe
>
> Did some text get cut off here in your reply?

Sorry, forgot to finish it. Roughly the idea is this: if the backends we 
bundle look okay and flexible enough, maybe the API is sounds for most 
cases, barring extreme examples.

Anyway, this doesn't matter much, because we'll be refining CAPF.

>> 1) Adding some pre-completion hook which would allow you to run some
>> code once, set a buffer-local variable, which all backend functions for
>> LaTeX can refer to later in `prefix' action.
>
> I don't think this can work, because the choice of source depends on the
> location of point and the text in the buffer, which changes from
> completion to completion.

One *per completion*, of course. As opposed to once per backend per 
completion.

> For the Predictive LaTeX support, I use a mish mash of piggy-backing on
> jit-lock where possible, some regexp match tests, and a limited form of
> incremental parsing of the LaTeX code when the rest aren't enough. (I
> used to exclusively use the latter, implemented using my auto-overlays
> package. Surprisingly this was more than efficient enough to cause no
> noticeable delay when typing and auto-completing, even on very old
> hardware. But writing and debugging the incremental parser definitions
> were a nightmare, largely because TeX doesn't have a well-defined
> grammar. So I replaced as much as possible of the parsing with simpler
> methods.)

Sounds scary.

> This effectively means moving the selection logic out of Company and into
> Predictive. Which isn't necessarily a bad solution. It's how I used to do
> it in Predictive, until I generalised the selection logic and moved it
> into Completion-UI so other sources could make use of it.
>
> If we stick with the CAPF API, I suspect I'll end up moving my source
> selection logic back into Predictive, and making it Predictive-specific
> again.

I don't think you have to. A completion function can similarly return 
different completion tables based on the context.

>> Regexps we have already (company-grab-...),
>
> How fast is this if you have to go through, say, 100 moderately complex
> regexps checking for a match? (See the predictive-latex.el in the old
> Predictive tarball release for examples.)

I'm not really sure what to look at. `predictive-latex-load-regexps'? I 
probably won't be able to measure it without rewriting that code.

> In Completion-UI, I implemented combined completion sources through a
> `completion-ui-combining-complete' wrapper function. (They're not
> functionally equivalent to your merged sources, but the requirements are
> somewhat similar.)
>
> Would using something like that to build merged CAPF functions be a
> solution for Company? I know this isn't particularly user-friendly if you
> want to allow users to easily define their own merged sources. I don't
> know if that's something people regularly do in Company.

In Company, you just add a list value to `company-backends'. There is 
one in the default value already. Its handling is not perfect, but does 
the job currently, as long as the backends are compatible enough.

> There's one thing we should perhaps think a bit more about.
>
> Is it right to say that the majority of the Company backends are selected
> based on global properties of a buffer (e.g. major-modes)? This seems to
> be the case for the default `company-backends' list. In Completion-UI I
> was almost entirely focused on selecting backends based on local
> properties at different locations within a buffer (regexps, faces,
> syntax, etc.).
>
> How do you envisage supporting local source selection in the new (or
> enhanced CAPF) API? Would this kind of local completion source selection
> always be implemented within a single backend, like the Company CEDET
> backend? Or should there be a convenient way of supporting it in the
> generic source selection API?

Generally, you'd just replace several existing entries in 
`completion-at-point-functions' with a new one that would first check 
local properties and then call an appropriate function among the 
replaced ones. Or, if the new function does not handle existing 
completion functions, scratch out he "replace[d]" bit.

> Following the code path is unnecessarily convoluted. I can't tell that
> company-dabbrev-code internally delegates to the company-dabbrev backend
> just from looking at `company-backends'. If it looked more like:
>
> ((use-dabbrev-code-p . dabbrev-completion)
>   (use-dabbrev-p . dabbrev-completion))
>
> it would be immediately obvious that these are two different ways of
> using the same source of completions.

This wouldn't work because company-dabbrev-code uses case-sensitive 
search and doesn't downcase the results, as opposed to company-dabbrev. 
But they share code anyway.

> The question for now is, what should go in the generic Emacs API? Vanilla
> CAPF, perhaps with some additional standard PROPS properties? If the
> generic Emacs API can't replace company-backends in Company, then in my
> view the API is no good.
>
> You're the expert here :)

So far I don't see obvious missing parts in CAPF API. But there are some 
that could be harder to support in frontends, like partial completion, 
and it also seems to be suboptimal for implementing backends with 
external services, see the tangential discussion at 
http://debbugs.gnu.org/cgi/bugreport.cgi?bug=16334.

> The way I imagine it, completion packages like Company would rarely need
> to use the API for defining new widgets. (Unless they're defining custom
> widgets that are so package-specific they don't belong in core Emacs. But
> that seems like a rare case.)
>
> What they *will* all use is the API for *invoking* the completion
> interface, which we began to discuss below.

Yep.

> Erm, I don't see any major shortcomings in the API. Probably that hasn't
> come across in the discussion :-)
>
> Minor ones that have come here were:
>
> - No support for non-prefix completion (but it looks easy enough to
>    extend the API).

So far I don't see an easy way to visualize non-prefix completion in 
frontends. Specifically, the way candidates relate to an input string. 
With prefix completion, it's easy: just highlight the prefix, but 
partial completion, for example, requires re-parsing the returned 
candidates on the fly, to try to match the separate letters.

> - Difficult (impossible?) to distinguish between "no completions
>    available" and "backend not applicable".

That's easy. "Backend not applicable" means `prefix' returned nil. "No 
completion available" means `prefix' returned non-nil, but `candidates' 
returned nil.

> OK, makes sense. If I understand correctly, CAPF is fine for letting
> users decide the order of trying backends.

I'm not sure. When someone's using `add-hook', you don't really look at 
the order of all elements, you just add a new one at the beginning or 
the end of the list. Should be all right most of the time, though.

> It would be nice to define a Customization interface that lists the
> available CAPF functions in a drop-down menu (whilst also allowing
> free-form entries, of course). If I find a way of porting the
> Completion-UI auto-updating customization interface magic to CAPF, this
> would be feasible...

This looks impossible to me. CAPF functions are defined in many places, 
including third-party packages, using plain `defun' interface. Can there 
really be a way to track them?

Re: enabling company-capf support in cfengine.el

[Dmitry Gutov]

On 23.12.2013 02:17, Dmitry Gutov wrote:
> But you also might want to wait until the next Company version is
> released

It is now, and company-capf is included in company-backends, as long as 
Emacs version is > 24.3.50.

Re: enabling company-capf support in cfengine.el

[Stefan Monnier]

>> But you also might want to wait until the next Company version is
>> released
> It is now, and company-capf is included in company-backends, as long as
> Emacs version is > 24.3.50.

I don't see why you wouldn't want to include it for Emacs-24.1 already.
I do understand why you would prefer keeping company-elisp for <=24.3,
but AFAIK company-capf works well in 24.1.

BTW, the next step would be to supplant other company backends, like
we've done with company-elisp: company-css, company-nxml, and
company-semantic look like prime candidates.


        Stefan

Re: enabling company-capf support in cfengine.el

[Dmitry Gutov]

On 16.01.2014 18:06, Stefan Monnier wrote:
> I don't see why you wouldn't want to include it for Emacs-24.1 already.
> I do understand why you would prefer keeping company-elisp for <=24.3,
> but AFAIK company-capf works well in 24.1.

Mostly simplicity, one less version check. And I don't really consider 
completion functions without :company- properties as viable substitutes 
for Company backends. And many of them still behave inappropriately, 
like `semantic-ia-complete-symbol' and `nxml-complete'.

> BTW, the next step would be to supplant other company backends, like
> we've done with company-elisp: company-css, company-nxml, and
> company-semantic look like prime candidates.

Indeed. Although I'm not sure what would be an appropriate counterpart 
in CAPF for `company-semantic--pre-prefix-length'.

With Semantic, we also have the question of alternatives. At the moment, 
`company-semantic' goes before `company-clang', so that if semantic-mode 
is enabled, the former is used, and otherwise, the latter.

If we remove `company-semantic' from the list (and keep `company-capf' 
after the more specific backends), `company-clang' will have precedence 
and will get enabled by default whenever `clang' is present in the system.

Re: enabling company-capf support in cfengine.el

[Stefan Monnier]

> And many of them still behave inappropriately, like
> semantic-ia-complete-symbol' and `nxml-complete'.

Whenever you bump into such problems, do report them.  I'll take a look
at the above two.

>> BTW, the next step would be to supplant other company backends, like
>> we've done with company-elisp: company-css, company-nxml, and
>> company-semantic look like prime candidates.
> Indeed.  Although I'm not sure what would be an appropriate counterpart in
> CAPF for `company-semantic--pre-prefix-length'.

You mean, what should be the counterpart to:

   cannot \(e.g. if it is in the middle of a string\).  Instead of a string,
   the back-end may return a cons where car is the prefix and cdr is used in
   `company-minimum-prefix-length' test.  It's either number or t, in which
   case the test automatically succeeds.

A straightforward option might be to have CAPF return
a new :company-prefix-length-offset property which is then added to the
prefix length by company-capf.

From a more CAPF-centric point of view, in the case of Semantic, another
option is to return as prefix not "ch" but "fr->ch", and then specify
a boundary between "fr->" and "ch".

> With Semantic, we also have the question of alternatives.  At the moment,
> company-semantic' goes before `company-clang', so that if semantic-mode is
> enabled, the former is used, and otherwise, the latter.

That's easy to solve: turn company-clang into its CAPF equivalent, then
place it within completion-at-point-functions after the Semantic one.


        Stefan

Re: enabling company-capf support in cfengine.el

[Dmitry Gutov]

On 17.01.2014 15:04, Stefan Monnier wrote:
> Whenever you bump into such problems, do report them.  I'll take a look
> at the above two.

Thank you. Another one that I've noticed is `message-completion-function'.

>>> BTW, the next step would be to supplant other company backends, like
>>> we've done with company-elisp: company-css, company-nxml, and
>>> company-semantic look like prime candidates.
>> Indeed.  Although I'm not sure what would be an appropriate counterpart in
>> CAPF for `company-semantic--pre-prefix-length'.
>
> You mean, what should be the counterpart to:
>
>     cannot \(e.g. if it is in the middle of a string\).  Instead of a string,
>     the back-end may return a cons where car is the prefix and cdr is used in
>     `company-minimum-prefix-length' test.  It's either number or t, in which
>     case the test automatically succeeds.
>
> A straightforward option might be to have CAPF return
> a new :company-prefix-length-offset property which is then added to the
> prefix length by company-capf.

Note the option of returning `t', it doesn't fit the proposed name 
(-prefix-length).

And actually, I'm not aware of any users of `company-semantic' (though 
there probably are some), whereas `company-clang' uses that `t' return 
value after a user request, which I think makes more sense. Maybe we 
should deprecate the numeric cdr altogether.

And when the choice is between returning `t' or nothing, the choice of 
whether idle completion is appropriate at point is made only by the 
backend. For example, in C-based languages, if the current word goes 
after "." or "->", it's quite likely the choice of completions is known 
and limited, and many users, used to the popular IDE behavior, even like 
when the idle completion popup is displayed right after typing "." or 
"->" (and waiting a certain fraction of a second).

So yes, a new property might be appropriate, but with different semantics.

>  From a more CAPF-centric point of view, in the case of Semantic, another
> option is to return as prefix not "ch" but "fr->ch", and then specify
> a boundary between "fr->" and "ch".

Maybe. But the notion of completion boundaries is unrelated to idle 
completion, and the latter is the sole purpose of that return value.

>> With Semantic, we also have the question of alternatives.  At the moment,
>> company-semantic' goes before `company-clang', so that if semantic-mode is
>> enabled, the former is used, and otherwise, the latter.
>
> That's easy to solve: turn company-clang into its CAPF equivalent, then
> place it within completion-at-point-functions after the Semantic one.

Which body of code would contain that clang-completion-function, and 
perform the adding? Would that hook addition be global, forcing 
clang-completion-function to include a major-mode check (hitherto unseen 
behavior in CAPF functions, AFAIK), or would it iterate over applicable 
major mode hooks?

Anyway, as I see it, none of the options would provide a smooth 
transition from company-clang being included in company-backends. The 
users will have to install a package, enable a minor mode, or do 
something equivalent.

Guess a simpler solution would be to keep company-clang as-is, but move 
it behind company-capf.

Re: enabling company-capf support in cfengine.el

[Stefan Monnier]

>> Whenever you bump into such problems, do report them.  I'll take a look
>> at the above two.
> Thank you. Another one that I've noticed is `message-completion-function'.

OK.  Could you make a bug report listing these (3 so far) problematic
functions, and explaining (as much as you can remember) what kind of
problem they cause?
[ Don't waste too much time trying to remember what problem they cause:
  if you remember, great, but if not, it might be obvious when I look at
  it anyway.  ]
That will help me not forget.

> Note the option of returning `t', it doesn't fit the proposed name
> (-prefix-length).

Ah, I see in the rest of your answer than this is all about "not
waiting".  So, yes, the name should rather be :company-immediate or
something like that.  It could also be "integer or t".  And indeed, the
integer case is probably not needed.

> So yes, a new property might be appropriate, but with different semantics.

Right.

>> From a more CAPF-centric point of view, in the case of Semantic, another
>> option is to return as prefix not "ch" but "fr->ch", and then specify
>> a boundary between "fr->" and "ch".
> Maybe. But the notion of completion boundaries is unrelated to idle
> completion, and the latter is the sole purpose of that return value.

Indeed, sorry.  I had not understood the purpose.  Apparently the
docstring's reference to `company-minimum-prefix-length' wasn't
sufficient for me to figure it out ;-)

>> That's easy to solve: turn company-clang into its CAPF equivalent, then
>> place it within completion-at-point-functions after the Semantic one.
> Which body of code would contain that clang-completion-function, and perform
> the adding?

How 'bout company-clang.el at first?
Could later be renamed to capf-clang.el, or cc-clang.el, or ... ?

> Would that hook addition be global,

Since company-clang is added globally to company-backends, yes, I'd
expect company-capf-clang to be added to the global part of
completion-at-point-functions as well.

> forcing clang-completion-function to include a major-mode check
> (hitherto unseen behavior in CAPF functions, AFAIK),

Don't know if it's already seen or not, but I don't see why it'd be
a problem.

> Anyway, as I see it, none of the options would provide a smooth transition
> from company-clang being included in company-backends. The users will have
> to install a package, enable a minor mode, or do something equivalent.

I really don't see why it's hard:
- change company-clang.el so that it supports the "CAPF protocol" rather
  than the "company-backend protocol".
- actually make it support both protocols (with the help of
  company-capf to translate from the CAPF protocol to the other).
- in company.el (add-hook 'completion-at-point-function 'company-capf-clang)
  and add company-clang to company-backends if Emacs is too old to use
  company-capf.

> Guess a simpler solution would be to keep company-clang as-is, but move it
> behind company-capf.

But moving it to CAPF means that it becomes useful/usable not only for
Company for also for good ol' competion-at-point.


        Stefan

Re: enabling company-capf support in cfengine.el

[Dmitry Gutov]

On 19.01.2014 04:37, Stefan Monnier wrote:
> Could you make a bug report listing these (3 so far) problematic
> functions, and explaining (as much as you can remember) what kind of
> problem they cause?

Done: http://debbugs.gnu.org/16496

> Ah, I see in the rest of your answer than this is all about "not
> waiting".  So, yes, the name should rather be :company-immediate or
> something like that.  It could also be "integer or t".  And indeed, the
> integer case is probably not needed.

I'll have to think about the name more. Maybe Completion-UI gives some 
inspiration.

>>> That's easy to solve: turn company-clang into its CAPF equivalent, then
>>> place it within completion-at-point-functions after the Semantic one.
>> Which body of code would contain that clang-completion-function, and perform
>> the adding?
>
> How 'bout company-clang.el at first?
> Could later be renamed to capf-clang.el, or cc-clang.el, or ... ?

Sometime ago I've been told that RMS dislikes Clang strongly enough to 
oppose inclusion of any code using it in Emacs.

Unless it has changed (or is no longer a major factor), separating the 
code from Company won't be particularly valuable. Ours isn't a very 
sophisticated integration anyway, there are other packages doing more, 
such as spinning up a persistent server to eliminate the latency of 
launching new process and scanning all project files anew each time. 
Alas, they're all using auto-complete ATM.

>> forcing clang-completion-function to include a major-mode check
>> (hitherto unseen behavior in CAPF functions, AFAIK),
>
> Don't know if it's already seen or not, but I don't see why it'd be
> a problem.

Yeah, OK.

>> Anyway, as I see it, none of the options would provide a smooth transition
>> from company-clang being included in company-backends. The users will have
>> to install a package, enable a minor mode, or do something equivalent.
>
> I really don't see why it's hard:
> - change company-clang.el so that it supports the "CAPF protocol" rather
>    than the "company-backend protocol".
> - actually make it support both protocols (with the help of
>    company-capf to translate from the CAPF protocol to the other).

This is doable, yes.

> - in company.el (add-hook 'completion-at-point-function 'company-capf-clang)
>    and add company-clang to company-backends if Emacs is too old to use
>    company-capf.

Would it be at the top-level? I.e. whenever you have company.el loaded, 
the global value of completion-at-point-functions will include 
company-capf-clag. Sounds invasive.

Doing it in global-company-mode is not an option, I believe 
(define-globalized-minor-mode has no BODY argument), and even if it 
were, the users don't have to call this function, they can use 
company-mode directly.

If we're doing it inside company-mode function body, we'll have to 
change the local value instead.

Two last options would look weird either way: why would company-mode 
function concern itself with the value of completion-at-point-functions, 
and Clang specifically? That's why I suggested another minor mode.

>> Guess a simpler solution would be to keep company-clang as-is, but move it
>> behind company-capf.
>
> But moving it to CAPF means that it becomes useful/usable not only for
> Company for also for good ol' competion-at-point.

Would a Company user benefit from this, really? I'd like to hear from 
one person that would first.

As long as (add-hook 'completion-at-point-function 'company-capf-clang) 
is only done when company-mode is enabled, there's really not much benefit.

Re: enabling company-capf support in cfengine.el

[Dmitry Gutov]

On 19.01.2014 04:37, Stefan Monnier wrote:
> Indeed, sorry.  I had not understood the purpose.  Apparently the
> docstring's reference to `company-minimum-prefix-length' wasn't
> sufficient for me to figure it out ;-)

Maybe it could've been explained better, but 
`company-minimum-prefix-length' is mentioned in several docstrings, and 
repeating its purpose every time would be redundant. The docstring of 
this var itself mentions "automatic completion".

Re: enabling company-capf support in cfengine.el

[Stefan Monnier]

> Sometime ago I've been told that RMS dislikes Clang strongly enough to
> oppose inclusion of any code using it in Emacs.

AFAIK Clang is Free Software.  So, I don't see a valid reason to reject
inclusion of company-clang or equivalent into Emacs.  If it's in GNU
ELPA it's (virtually) in Emacs already anyway (we use the same rules for
the two, specifically so we can easily move code from one to the other).

> Unless it has changed (or is no longer a major factor), separating the code
> from Company won't be particularly valuable.

My interest is in making Company into nothing more than an alternative
UI.  All the backends would be separate and usable as much by Company as
by completion-at-point.

Most of it (UI and backends) should also get integrated into Emacs.

> Clang specifically? That's why I suggested another minor mode.

Yes, ultimately clang-completion should be a separate package enabled
separately.  In any case I don't think any of those issues are serious
enough to be a reason not to make the clang backend work via CAPF.

>> But moving it to CAPF means that it becomes useful/usable not only for
>> Company for also for good ol' competion-at-point.
> Would a Company user benefit from this, really?

Wrong question: the benefit would be for non-Company users.

> As long as (add-hook 'completion-at-point-function 'company-capf-clang) is
> only done when company-mode is enabled, there's really not much benefit.

Indeed, that's why it should also be done/doable when Company is not
being used.


        Stefan

Re: enabling company-capf support in cfengine.el

[Stefan Monnier]

> would be redundant. The docstring of this var itself mentions "automatic
> completion".

I think I failed to make the connection based on "automatic completion"
(which, to be honest, doesn't mean anything to me.  At best, I'd expect
it to complete the word for me without my typing anything, more like
what lightning completion does).


        Stefan

Re: enabling company-capf support in cfengine.el

[David Engster]

Stefan Monnier writes:
>> Sometime ago I've been told that RMS dislikes Clang strongly enough to
>> oppose inclusion of any code using it in Emacs.
>
> AFAIK Clang is Free Software.  So, I don't see a valid reason to reject
> inclusion of company-clang or equivalent into Emacs.

We had this discussion here:

http://thread.gmane.org/gmane.emacs.devel/155124

While RMS does not state it explicitly there, he has asked me to not
merge our semantic-clang file in CEDET to Emacs, which is also why I
have given up on Clang for improving our C/C++ parser.

-David

Re: enabling company-capf support in cfengine.el

[Dmitry Gutov]

On 19.01.2014 22:21, Stefan Monnier wrote:
> I think I failed to make the connection based on "automatic completion"
> (which, to be honest, doesn't mean anything to me.  At best, I'd expect
> it to complete the word for me without my typing anything, more like
> what lightning completion does).

Yes, there's also some term conflation going on: "automatic completion" 
in company-minimum-prefix-length docstring is not the same thing as 
"auto-complete" in company-auto-complete docstring.

I'll try to make it more clear. Unfortunately though, "idle completion" 
is a less familiar term than "auto-complete", to an average user.

Re: enabling company-capf support in cfengine.el

[Dmitry Gutov]

On 19.01.2014 22:19, Stefan Monnier wrote:
>> Sometime ago I've been told that RMS dislikes Clang strongly enough to
>> oppose inclusion of any code using it in Emacs.
>
> AFAIK Clang is Free Software.  So, I don't see a valid reason to reject
> inclusion of company-clang or equivalent into Emacs.  If it's in GNU
> ELPA it's (virtually) in Emacs already anyway (we use the same rules for
> the two, specifically so we can easily move code from one to the other).

Richard's argument in the thread linked by David was not based on 
Clang's legal status, other than it being distributed under a 
non-copyleft license.

Since that argument doesn't make sense to me, I won't claim to fully 
understand it, or whether there is any difference between Emacs and GNU 
ELPA, as far as that argument is concerned.

>> Unless it has changed (or is no longer a major factor), separating the code
>> from Company won't be particularly valuable.
>
> My interest is in making Company into nothing more than an alternative
> UI.  All the backends would be separate and usable as much by Company as
> by completion-at-point.

I'm interested in this general direction as well, as long as the 
migration causes no regression in features available to Company users.

But I think moving backends that have no existing corresponding 
completion functions in Emacs core is low priority.

>> Clang specifically? That's why I suggested another minor mode.
>
> Yes, ultimately clang-completion should be a separate package enabled
> separately.  In any case I don't think any of those issues are serious
> enough to be a reason not to make the clang backend work via CAPF.

Ok. Then my main point is, there's no reason to do the CAPF conversion 
until it's in a separate package/minor mode. The conversion itself 
should be easy.

And ideally, the splitting would be performed by someone who works with 
C/C++ and Clang with any regularity. I.e. not myself. :)

Re: enabling company-capf support in cfengine.el

[Stephen J. Turnbull]

 > On 19.01.2014 22:19, Stefan Monnier wrote:

 >> AFAIK Clang is Free Software.  So, I don't see a valid reason to
 >> reject inclusion of company-clang or equivalent into Emacs.  If
 >> it's in GNU ELPA it's (virtually) in Emacs already anyway (we use
 >> the same rules for the two, specifically so we can easily move
 >> code from one to the other).

I wonder if Richard would agree that the rules are the same.

Dmitry Gutov writes:

 > Richard's argument in the thread linked by David was not based on 
 > Clang's legal status, other than it being distributed under a 
 > non-copyleft license.

Richard never argues from legal status.  He always argues from
freedom, although legal status is often convenient evidence of the
effect on freedom.

He has no objection to clang (or LLVM) itself, because it *is* free
software.  However, the GNU Project sets higher standards, and Emacs
(and GCC) try to conform to them.  Specifically, *defending* freedom,
including *shutting the door* on cooperation with non-free software:

    Do you mean, [gcc-xml outputs] the entire parse tree in full
    detail?
    Would it be conceivable to feed this into a nonfree back-end?
    Would this mean that nonfree backends could take advantage
    of our free front-ends?

    If so, it is very dangerous -- it would open the door to a
    terrible setback for our defense of users' freedom.  Namely, the
    use of free software as part of compilers that are partly nonfree.
    I don't remember, but I would guess that is why we have refused to
    merge it into GCC.

    LLVM and Clang open the door to the same terrible setback.  Since
    they are not copylefted, their front-ends can be used with nonfree
    back-ends and vice versa.  [from the cited thread]

So his objection is to emission of information that could be
conveniently used by non-free software to integrate free software into
a non-free toolchain.  AIUI, this is basically the same configuration
that led to the confrontation with Steve Jobs over Objective-C, except
that if the output of the compiler front-end is part of the spec, you
would have almost no leverage in court to claim that it's a single
Work which is a derivative of the copyleft front-end.

I don't understand the use of clang here, so I can't help you with
whether company-clang treads on the same territory.  Since the thread
is on "completion", I would guess not.

Disclaimer:  I am not a lawyer, I do not speak for Richard (any
sentence that refers to his opinions, objections, etc should be
considered to be prefaced by "it seems to me"), and I do not agree
with the logic.  So my interpretation may be inaccurate.

Re: enabling company-capf support in cfengine.el

[Stefan Monnier]

> But I think moving backends that have no existing corresponding completion
> functions in Emacs core is low priority.

Agreed.


        Stefan

Re: enabling company-capf support in cfengine.el

[Stefan Monnier]

> Unfortunately though, "idle completion" is a less familiar term than
> "auto-complete", to an average user.

Right, it makes just as little sense as "automatic completion" to me.
How 'bout:

(defcustom company-minimum-prefix-length 3
  "The minimum prefix length before Company pops up completion options."


-- Stefan

Re: enabling company-capf support in cfengine.el

[Dmitry Gutov]

On 20.01.2014 04:20, Stefan Monnier wrote:
>> Unfortunately though, "idle completion" is a less familiar term than
>> "auto-complete", to an average user.
>
> Right, it makes just as little sense as "automatic completion" to me.

"idle completion" is completion initiated automatically after Emacs has 
been idle for some time, also see "idle timer". How would you call it 
better?

> How 'bout:
>
> (defcustom company-minimum-prefix-length 3
>    "The minimum prefix length before Company pops up completion options."
                                                ^^^^ ^^
Also not ideal: although Company uses the "pseudo-tooltip" by default, 
depending on the value of `company-frontends' it may display completion 
candidates inline (when there's just one candidate, it does that 
anyway), or in the echo area. Or using some other way, provided someone 
writes a new frontend.

And this description mixes up notions of space and time. The "minimum 
prefix length" is a necessary condition, but the variable responsible 
for "before" is `company-idle-delay'.

Re: enabling company-capf support in cfengine.el

[John Yates]

[-- Attachment #1: Type: text/plain, Size: 2099 bytes --]

On Sun, Jan 19, 2014 at 9:14 PM, Stephen J. Turnbull <stephen@xemacs.org>wrote:

>
> He has no objection to clang (or LLVM) itself, because it *is* free
> software.  However, the GNU Project sets higher standards, and Emacs
> (and GCC) try to conform to them.  Specifically, *defending* freedom,
> including *shutting the door* on cooperation with non-free software:
>
>     Do you mean, [gcc-xml outputs] the entire parse tree in full
>     detail?
>     Would it be conceivable to feed this into a nonfree back-end?
>     Would this mean that nonfree backends could take advantage
>     of our free front-ends?
>

That cat already seems to be out of the bag: http://dragonegg.llvm.org/

    If so, it is very dangerous -- it would open the door to a
>     terrible setback for our defense of users' freedom.  Namely, the
>     use of free software as part of compilers that are partly nonfree.
>     I don't remember, but I would guess that is why we have refused to
>     merge it into GCC.
>
>     LLVM and Clang open the door to the same terrible setback.  Since
>     they are not copylefted, their front-ends can be used with nonfree
>     back-ends and vice versa.  [from the cited thread]
>
> So his objection is to emission of information that could be
> conveniently used by non-free software to integrate free software into
> a non-free toolchain.  AIUI, this is basically the same configuration
> that led to the confrontation with Steve Jobs over Objective-C, except
> that if the output of the compiler front-end is part of the spec, you
> would have almost no leverage in court to claim that it's a single
> Work which is a derivative of the copyleft front-end.
>

RMS may have an inflated sense of the extent to which the greater compiler
community (those developing and those using compilers) value gcc over clang
/ llvm.  For many reason the latter is winning the day.  Speed, memory
footprint, modularity, ease of entry, size of development community all
favor clang / llvm.  Anecdotal evidence: my startup (Gnu / Linux based
product) has just switched from gcc to clang.

/john

[-- Attachment #2: Type: text/html, Size: 2868 bytes --]

Re: enabling company-capf support in cfengine.el

[Stephen J. Turnbull]

John Yates writes:

 > RMS may have an inflated sense of the extent to which the greater
 > compiler community (those developing and those using compilers)
 > value gcc over clang / llvm.

RMS doesn't care about "greater communities", AFAICT.  He cares about
the free software community (more or less, the GPL-using community),
and about preserving a haven for freedom for all users.

Re: enabling company-capf support in cfengine.el

[Óscar Fuentes]

John Yates <john@yates-sheets.org> writes:

> That cat already seems to be out of the bag: http://dragonegg.llvm.org/

dragonegg is GPLed.

> RMS may have an inflated sense of the extent to which the greater compiler
> community (those developing and those using compilers) value gcc over clang
> / llvm.  For many reason the latter is winning the day.  Speed, memory
> footprint, modularity, ease of entry, size of development community all
> favor clang / llvm.
[snip]

This is very true.

The reason is easy to understand: why should I, as a software developer,
should use GCC instead of Clang when *both are Free Software* and
Clang's development is focused on creating the best tool for the user
while GCC is hindered by someone else's views about how to protect me
from hypothetical perils? Why should I use the crippled tool?

Re: enabling company-capf support in cfengine.el

[David Kastrup]

"Stephen J. Turnbull" <stephen@xemacs.org> writes:

> John Yates writes:
>
>  > RMS may have an inflated sense of the extent to which the greater
>  > compiler community (those developing and those using compilers)
>  > value gcc over clang / llvm.
>
> RMS doesn't care about "greater communities", AFAICT.  He cares about
> the free software community (more or less, the GPL-using community),
> and about preserving a haven for freedom for all users.

Yes.  Free Software was never about winning popularity contests.  It was
about being available to those who care.

-- 
David Kastrup

Re: enabling company-capf support in cfengine.el

[Stefan Monnier]

> And this description mixes up notions of space and time. The "minimum prefix
> length" is a necessary condition, but the variable responsible for "before"
> is `company-idle-delay'.

Right.  I guess "automatic completion" is OK for the first line.
But subsequent lines should explain in more detail what the variable
really does.


        Stefan

Re: enabling company-capf support in cfengine.el

[John Yates]

[-- Attachment #1: Type: text/plain, Size: 565 bytes --]

On Sun, Jan 19, 2014 at 10:32 PM, Óscar Fuentes <ofv@wanadoo.es> wrote:

> dragonegg is GPLed.


Meaning it is above reproach.

The crucial fact is that RMS delayed gcc implementation of useful,
attractive functionality at a time when such was (or was becoming)
available from clang / llvm.  Yet despite his stance he was unable to
prevent gnu frontends from being integrated (in a perfectly legal way) with
the BSD-licensed llvm backend.  The net effect was to further reduce gcc's
attractiveness within the "open source compiler community".

/john

[-- Attachment #2: Type: text/html, Size: 980 bytes --]

Re: enabling company-capf support in cfengine.el

[John Yates]

[-- Attachment #1: Type: text/plain, Size: 699 bytes --]

On Sun, Jan 19, 2014 at 10:22 PM, Stephen J. Turnbull <stephen@xemacs.org>wrote:

> John Yates writes:
>
>  > RMS may have an inflated sense of the extent to which the greater
>  > compiler community (those developing and those using compilers)
>  > value gcc over clang / llvm.
>

It might have been clearer if I had said the "open source compiler
community".

RMS doesn't care about "greater communities", AFAICT.  He cares about
> the free software community (more or less, the GPL-using community),
> and about preserving a haven for freedom for all users.
>

At RMS's behest emacs used GPL3 bzr instead of GPL2 git.  Now he has
relented.  Would he still have relented were git were BSD?

/john

[-- Attachment #2: Type: text/html, Size: 1267 bytes --]

Re: enabling company-capf support in cfengine.el

[David Kastrup]

John Yates <john@yates-sheets.org> writes:

> On Sun, Jan 19, 2014 at 10:32 PM, Óscar Fuentes <ofv@wanadoo.es> wrote:
>
>> dragonegg is GPLed.
>
>
> Meaning it is above reproach.

Not really.  "reproach" is something concerning intent rather than
execution.  The GPL as a license is pretty good in being incompatible
mostly with intents interfering with the well-being of Free Software.
But it has no will of its own and it is a legal, not a moral tool.  The
world is always changing, and that also concerns the relation of the GPL
to the goals of Free Software which are of moral, not legal nature.
Realigning the effectiveness of the GPL regarding Free Software goals
under the restraints of developments in the real world already required
the large and cumbersome update from GPLv2 to GPLv3.

GPLed software is hard to use as a weapon against Free Software by
design, but even a blunt weapon can do harm.

> The crucial fact is that RMS delayed gcc implementation of useful,
> attractive functionality at a time when such was (or was becoming)
> available from clang / llvm.  Yet despite his stance he was unable to
> prevent gnu frontends from being integrated (in a perfectly legal way)
> with the BSD-licensed llvm backend.  The net effect was to further
> reduce gcc's attractiveness within the "open source compiler
> community".

The priorities of the GNU project are not focused about "market share".
The priorities are to make the best possible software available to those
who care about Free Software, and to not help with locking them into
offerings not providing the Software Freedoms the GNU project has set
out to preserve.

It's not surprising that watered-down versions of our goals and
principles enjoy a lot of success: as you can see with the kind of
unfreedoms democratically elected governments can get away with, there
is a lot of leeway before the general public gets annoyed enough to even
care.

Does that mean that we should stop caring, too?  Nobody can force us to
give up the little (and not so little) we can call our own.  Hindsight
is always a great thing to have.  And to be honest, I wish I'd be living
in a world where hindsight managed to prove Richard and his precautions
wrong more often.

-- 
David Kastrup

Re: enabling company-capf support in cfengine.el

[David Kastrup]

John Yates <john@yates-sheets.org> writes:

> On Sun, Jan 19, 2014 at 10:22 PM, Stephen J. Turnbull <stephen@xemacs.org>wrote:
>
>> John Yates writes:
>>
>>  > RMS may have an inflated sense of the extent to which the greater
>>  > compiler community (those developing and those using compilers)
>>  > value gcc over clang / llvm.
>>
>
> It might have been clearer if I had said the "open source compiler
> community".
>
> RMS doesn't care about "greater communities", AFAICT.  He cares about
>> the free software community (more or less, the GPL-using community),
>> and about preserving a haven for freedom for all users.
>>
>
> At RMS's behest emacs used GPL3 bzr instead of GPL2 git.  Now he has
> relented.  Would he still have relented were git were BSD?

The licenses itself are not as important as the perspectives and goals
behind their choice.

The GPL has been designed to make it hard to _work_ against our goals.
It's powerless against people _being_ against our goals, even though it
makes software placed under it hard to use for undermining our goals.

So it makes little sense stipulating hypotheticals about licensing when
a large consideration is the frame of mind of the underlying community.
While the GPL provides mechanical resistance against a community turning
bad and thus some long-term perspective, it's not a cure-all.

-- 
David Kastrup

clang vs free software (was: enabling company-capf support in cfengine.el)

[Stefan Monnier]

Please stop discussing this on emacs-devel where it does not belong.


        Stefan

Re: enabling company-capf support in cfengine.el

[Stephen J. Turnbull]

John Yates writes:

 >> John Yates writes:
 >>> RMS may have an inflated sense of the extent to which the greater
 >>> compiler community (those developing and those using compilers)
 >>> value gcc over clang / llvm.

 > It might have been clearer if I had said the "open source compiler
 > community".

I knew what you meant.

 > At RMS's behest emacs used GPL3 bzr instead of GPL2 git.  Now he
 > has relented.  Would he still have relented were git were BSD?

Dunno, and it's not relevant to the issue he has with clang and
gcc-xml AIUI.

Re: enabling company-capf support in cfengine.el

[John Yates]

[-- Attachment #1: Type: text/plain, Size: 622 bytes --]

On Mon, Jan 20, 2014 at 9:30 AM, David Kastrup <dak@gnu.org> wrote:

> John Yates <john@yates-sheets.org> writes:
> > On Sun, Jan 19, 2014 at 10:32 PM, Óscar Fuentes <ofv@wanadoo.es> wrote:
> >> dragonegg is GPLed.
> > Meaning it is above reproach.
>
> Not really.  "reproach" is something concerning intent rather than
> execution.


Point well taken.  I should have said "entirely legal".  And just for the
record, given awareness of the gcc project's goals and philosophy
(something I am sure the dragonegg developers possess) I acknowledge that
dragonegg is clearly a questionable undertaking.

/john

[-- Attachment #2: Type: text/html, Size: 1109 bytes --]

Re: clang vs free software

[David Engster]

Stefan Monnier writes:
> Please stop discussing this on emacs-devel where it does not belong.

Fine, but what's the policy now regarding features that require an
installed clang to work; can they be merged to Emacs proper or not?

-David

Re: clang vs free software

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

    Fine, but what's the policy now regarding features that require an
    installed clang to work; can they be merged to Emacs proper or not?

Emacs (and all GNU packages involving C code) should be designed to
work best with GCC.  There should be nothing to encourage users not to
use our compiler.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: clang vs free software

[David Kastrup]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>     Fine, but what's the policy now regarding features that require an
>     installed clang to work; can they be merged to Emacs proper or not?
>
> Emacs (and all GNU packages involving C code) should be designed to
> work best with GCC.  There should be nothing to encourage users not to
> use our compiler.

Well, if we are talking about a generally desirable feature, the
question then is what needs to be done in order to have GCC fit the
problem space.  Even with proprietary systems, the Emacs stance has been
to not support any Windows-only or MacOSX-only features, but to provide
general functionality when working under them.  I think that this may
have been some incentive in the past for some Emacs developers to port
some system-specifically implemented features over in order to have them
generally available and thus accepted.

So the question is what the roadmap would be to have the software in
question work with GCC as the underlying technology.  And if it turns
out (I have no idea whether it actually would) that the answer to that
would be "we did not want interfaces to do that in GCC since that would
have allowed using them with proprietary software as well" then we might
need to reevaluate our chosen set of compromises and policies and see
whether they need rebalancing in order to better help our cause.

-- 
David Kastrup

Re: clang vs free software

[Dmitry Gutov]

Richard Stallman <rms@gnu.org> writes:

>     Fine, but what's the policy now regarding features that require an
>     installed clang to work; can they be merged to Emacs proper or not?
>
> Emacs (and all GNU packages involving C code) should be designed to
> work best with GCC.  There should be nothing to encourage users not to
> use our compiler.

The only way the features in question can "work best" with GCC is if
they are not implemented at all, because GCC still doesn't offer the
necessary interface that would allow us to use it in the capacity that
we can use Clang.

Note that the user will still be able to use GCC as the compiler, but
would have to have Clang installed for code assistance.

Re: clang vs free software

[John Yates]

[-- Attachment #1: Type: text/plain, Size: 1305 bytes --]

On Tue, Jan 21, 2014 at 9:42 AM, Richard Stallman <rms@gnu.org> wrote:

>     Fine, but what's the policy now regarding features that require an
>     installed clang to work; can they be merged to Emacs proper or not?
>
> Emacs (and all GNU packages involving C code) should be designed to
> work best with GCC.  There should be nothing to encourage users not to
> use our compiler.


Richard,

That really does not answer David's question.  If you read it carefully he
called out "features that require an installed clang to work".  That is the
crux of the tension here.  While the most obvious manifestation of clang is
as a C/C++ compiler very similar to gcc it is also much more.  That is
because it is architected as suite of libraries intended to be integrated
into more environments than just a compiler:

http://clang.llvm.org/features.html#libraryarch
http://clang.llvm.org/features.html#ideintegration

Tooling for analyzing and manipulating C/C++ is simply a space that gcc is
not addressing.  Based on its architecture it likely never will.

There exist now various packages integrating emacs will elements of clang.
 These packages are not supporting clang as alternative compiler.  Rather
by exploiting clang interfaces that have no gcc analog they offer exciting
IDE-like features.

/john

[-- Attachment #2: Type: text/html, Size: 1981 bytes --]

Re: clang vs free software

[Rüdiger Sonderfeld]

On Tuesday 21 January 2014 10:34:01 John Yates wrote:
> That really does not answer David's question.  If you read it carefully he
> called out "features that require an installed clang to work".  That is the
> crux of the tension here.  While the most obvious manifestation of clang is
> as a C/C++ compiler very similar to gcc it is also much more.  That is
> because it is architected as suite of libraries intended to be integrated
> into more environments than just a compiler:
> 
> http://clang.llvm.org/features.html#libraryarch
> http://clang.llvm.org/features.html#ideintegration
> 
> Tooling for analyzing and manipulating C/C++ is simply a space that gcc is
> not addressing.  Based on its architecture it likely never will.
> 
> There exist now various packages integrating emacs will elements of clang.
>  These packages are not supporting clang as alternative compiler.  Rather
> by exploiting clang interfaces that have no gcc analog they offer exciting
> IDE-like features.

GCC provides a plugin interface now which can be used to extract information 
about the source.  I started writing a plugin and Emacs interface based on 
that.  It can show the callgraph, jump to definition, and show some 
information about symbols.  You simply add it to your normal compiler call.  
But I didn't get very far and it requires a patched version of the gcc-python 
plugin.

I hope I'll find enough time to continue working on it.  Maybe Clang provides 
better integration.  But without an FFI it would have to be linked into Emacs.  
And GCC is the main compiler I use anyway.

(Well I actually started by writing a GCC plugin to generate ETAG files

https://github.com/ruediger/gcc-etags

But it got disappointing when I had to throw away a lot of the nice 
information that GCC provides and generate the simplified ETAG format.)

Regards,
Rüdiger

Re: clang vs free software

[joakim]

Rüdiger Sonderfeld <ruediger@c-plusplus.de> writes:

> On Tuesday 21 January 2014 10:34:01 John Yates wrote:
>> That really does not answer David's question.  If you read it carefully he
>> called out "features that require an installed clang to work".  That is the
>> crux of the tension here.  While the most obvious manifestation of clang is
>> as a C/C++ compiler very similar to gcc it is also much more.  That is
>> because it is architected as suite of libraries intended to be integrated
>> into more environments than just a compiler:
>> 
>> http://clang.llvm.org/features.html#libraryarch> http://clang.llvm.org/features.html#ideintegration> 
>> Tooling for analyzing and manipulating C/C++ is simply a space that gcc is
>> not addressing.  Based on its architecture it likely never will.
>> 
>> There exist now various packages integrating emacs will elements of clang.
>>  These packages are not supporting clang as alternative compiler.  Rather
>> by exploiting clang interfaces that have no gcc analog they offer exciting
>> IDE-like features.
>
> GCC provides a plugin interface now which can be used to extract information 
> about the source.  I started writing a plugin and Emacs interface based on 
> that.  It can show the callgraph, jump to definition, and show some 
> information about symbols.  You simply add it to your normal compiler call.  
> But I didn't get very far and it requires a patched version of the gcc-python 
> plugin.
>
> I hope I'll find enough time to continue working on it.  Maybe Clang provides 
> better integration.  But without an FFI it would have to be linked into Emacs.  
> And GCC is the main compiler I use anyway.

Interesting. Do you know if the interfaces you use are also provided via
gnome object introspection? Then I would love to try it out using the
Emacs Xwidget branch, which provides an attempt at a GIR bridge for emacs.

>
> (Well I actually started by writing a GCC plugin to generate ETAG files
>
> https://github.com/ruediger/gcc-etags
>
> But it got disappointing when I had to throw away a lot of the nice 
> information that GCC provides and generate the simplified ETAG format.)
>
> Regards,
> Rüdiger
>

-- 
Joakim Verona

Re: clang vs free software

[Rüdiger Sonderfeld]

On Tuesday 21 January 2014 17:25:59 joakim@verona.se wrote:
> > GCC provides a plugin interface now which can be used to extract
> > information about the source.  I started writing a plugin and Emacs
> > interface based on that.  It can show the callgraph, jump to definition,
> > and show some information about symbols.  You simply add it to your
> > normal compiler call. But I didn't get very far and it requires a patched
> > version of the gcc-python plugin.
> > 
> > I hope I'll find enough time to continue working on it.  Maybe Clang
> > provides better integration.  But without an FFI it would have to be
> > linked into Emacs. And GCC is the main compiler I use anyway.
> 
> Interesting. Do you know if the interfaces you use are also provided via
> gnome object introspection? Then I would love to try it out using the
> Emacs Xwidget branch, which provides an attempt at a GIR bridge for emacs.

I'm not loading any library in Emacs itself.  I have written plugins for GCC.  
They are loaded during the compile process (add -fplugin=./gccetags.so to the 
CXXFLAGS) and write data to a file.  For gcc-etags to a file in etag format 
and for the other experiment a sexp.

I haven't really looked at libclang.  Which could be loaded through an FFI.  I 
doubt that they are using Glib though or what would be needed for gnome object 
introspection.

Regards,
Rüdiger

Re: clang vs free software

[joakim]

Rüdiger Sonderfeld <ruediger@c-plusplus.de> writes:

> On Tuesday 21 January 2014 17:25:59 joakim@verona.se wrote:
>> > GCC provides a plugin interface now which can be used to extract
>> > information about the source.  I started writing a plugin and Emacs
>> > interface based on that.  It can show the callgraph, jump to definition,
>> > and show some information about symbols.  You simply add it to your
>> > normal compiler call. But I didn't get very far and it requires a patched
>> > version of the gcc-python plugin.
>> > 
>> > I hope I'll find enough time to continue working on it.  Maybe Clang
>> > provides better integration.  But without an FFI it would have to be
>> > linked into Emacs. And GCC is the main compiler I use anyway.
>> 
>> Interesting. Do you know if the interfaces you use are also provided via
>> gnome object introspection? Then I would love to try it out using the
>> Emacs Xwidget branch, which provides an attempt at a GIR bridge for emacs.
>
> I'm not loading any library in Emacs itself.  I have written plugins for GCC.  
> They are loaded during the compile process (add -fplugin=./gccetags.so to the 
> CXXFLAGS) and write data to a file.  For gcc-etags to a file in etag format 
> and for the other experiment a sexp.
>
> I haven't really looked at libclang.  Which could be loaded through an FFI.  I 
> doubt that they are using Glib though or what would be needed for gnome object 
> introspection.

I meant if gcc could provide the feature you use via gir.


> Regards,
> Rüdiger
>

-- 
Joakim Verona

Re: clang vs free software

[Thien-Thi Nguyen]

[-- Attachment #1: Type: text/plain, Size: 839 bytes --]

() Rüdiger Sonderfeld <ruediger@c-plusplus.de>
() Tue, 21 Jan 2014 17:34:10 +0100

   For gcc-etags to a file in etag format 
   and for the other experiment a sexp.

Why not write a server to which this plugin can send the hard-won
information?  The server's job is to persist all those (incoming)
goodies in a database somewhere (say, $projroot/.static-bits.db).
Other clients can request the subset that is interesting to them,
e.g., for etags (re)format(ting).

This architecture is HURD-friendly, too (IIUC the HURD mindset).
GNU Serveez + Guile-PG + SMOP ... 149 lines of Scheme, max.  :-D

-- 
Thien-Thi Nguyen
   GPG key: 4C807502
   (if you're human and you know it)
      read my lisp: (responsep (questions 'technical)
                               (not (via 'mailing-list)))
                     => nil

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: clang vs free software

[Rüdiger Sonderfeld]

[-- Attachment #1: Type: text/plain, Size: 965 bytes --]

On Tuesday 21 January 2014 19:00:26 Thien-Thi Nguyen wrote:
> () Rüdiger Sonderfeld <ruediger@c-plusplus.de>
> () Tue, 21 Jan 2014 17:34:10 +0100
> 
>    For gcc-etags to a file in etag format
>    and for the other experiment a sexp.
> 
> Why not write a server to which this plugin can send the hard-won
> information?  The server's job is to persist all those (incoming)
> goodies in a database somewhere (say, $projroot/.static-bits.db).
> Other clients can request the subset that is interesting to them,
> e.g., for etags (re)format(ting).
> 
> This architecture is HURD-friendly, too (IIUC the HURD mindset).
> GNU Serveez + Guile-PG + SMOP ... 149 lines of Scheme, max.  :-D

I think at the moment it writes a file for each code file to a subdirectory in 
the $projroot.  This could of course be changed.  But right now the project is 
in a very early stage and I didn't have the time to work on it in over a year.

Regards,
Rüdiger

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 230 bytes --]

Re: clang vs free software

[Rüdiger Sonderfeld]

On Tuesday 21 January 2014 17:38:13 joakim@verona.se wrote:
> Rüdiger Sonderfeld <ruediger@c-plusplus.de> writes:
> > On Tuesday 21 January 2014 17:25:59 joakim@verona.se wrote:
> >> > GCC provides a plugin interface now which can be used to extract
> >> > information about the source.  I started writing a plugin and Emacs
> >> > interface based on that.  It can show the callgraph, jump to
> >> > definition,
> >> > and show some information about symbols.  You simply add it to your
> >> > normal compiler call. But I didn't get very far and it requires a
> >> > patched
> >> > version of the gcc-python plugin.
> >> > 
> >> > I hope I'll find enough time to continue working on it.  Maybe Clang
> >> > provides better integration.  But without an FFI it would have to be
> >> > linked into Emacs. And GCC is the main compiler I use anyway.
> >> 
> >> Interesting. Do you know if the interfaces you use are also provided via
> >> gnome object introspection? Then I would love to try it out using the
> >> Emacs Xwidget branch, which provides an attempt at a GIR bridge for
> >> emacs.
> > 
> > I'm not loading any library in Emacs itself.  I have written plugins for
> > GCC. They are loaded during the compile process (add
> > -fplugin=./gccetags.so to the CXXFLAGS) and write data to a file.  For
> > gcc-etags to a file in etag format and for the other experiment a sexp.
> > 
> > I haven't really looked at libclang.  Which could be loaded through an
> > FFI.  I doubt that they are using Glib though or what would be needed for
> > gnome object introspection.
> 
> I meant if gcc could provide the feature you use via gir.

GCC doesn't provide a real API.  You basically get access to all of GCCs 
internal structures and functions.  You write your module and compile it as a 
shared library which is then loaded by GCC.

See (info "(gccint) Plugins")

I don't think gir could be used here and I don't think it would help.

Regards,
Rüdiger

Re: clang vs free software

[unic0rn]

On Tue, Jan 21, 2014 at 3:42 PM, Richard Stallman <rms@gnu.org> wrote:
> Emacs (and all GNU packages involving C code) should be designed to
> work best with GCC.  There should be nothing to encourage users not to
> use our compiler.

IMHO, users should be encouraged to explore their freedom.
wasn't it Apple that decided that they know best what's better for
their userbase?

Re: clang vs free software

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

GNU Emacs does not exist in isolation.  It is a part of the GNU
Project.  "Success" for GNU Emacs means success for GNU.

Our policy is not to install features in GNU package that would work
with Clang and not with GCC.  Emacs will not encourage people to
switch away from GCC.  Indeed, when we have the chance, we should
encourage GCC over Clang.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: clang vs free software

[Dmitry Gutov]

On 22.01.2014 17:31, Richard Stallman wrote:
> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
> GNU Emacs does not exist in isolation.  It is a part of the GNU
> Project.  "Success" for GNU Emacs means success for GNU.
>
> Our policy is not to install features in GNU package that would work
> with Clang and not with GCC.  Emacs will not encourage people to
> switch away from GCC.  Indeed, when we have the chance, we should
> encourage GCC over Clang.

I see. Thank you for the clarification.

Re: clang vs free software

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

    There exist now various packages integrating emacs will elements of clang.
     These packages are not supporting clang as alternative compiler.  Rather
    by exploiting clang interfaces that have no gcc analog they offer exciting
    IDE-like features.

They nonetheless encourage people to switch from GCC to Clang.  The
features may be useful but this way of providing them hurts our cause.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: clang vs free software

[David Kastrup]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>     There exist now various packages integrating emacs will elements of clang.
>      These packages are not supporting clang as alternative compiler.  Rather
>     by exploiting clang interfaces that have no gcc analog they offer exciting
>     IDE-like features.
>
> They nonetheless encourage people to switch from GCC to Clang.  The
> features may be useful but this way of providing them hurts our cause.

Which brings us back to the question: what would be required to provide
them via GCC or other GNU software?  If nobody bothers with even
considering the question, it would appear that it is not all that
important...

-- 
David Kastrup

Re: clang vs free software

[Helmut Eller]

On Thu, Jan 23 2014, David Kastrup wrote:

> Which brings us back to the question: what would be required to provide
> them via GCC or other GNU software?

GCC should have some analog to libclang:
  http://clang.llvm.org/doxygen/group__CINDEX.html
i.e. a somewhat stable API that makes the AST available without exposing
too many internal details.

Maybe that is possible with a GCC plugin, but since GCC could have
offered such functionality since decades, it's likely that it was
prevented for some political reason (like "you could then use GCC's
front-end for a proprietary IDE").

> If nobody bothers with even
> considering the question, it would appear that it is not all that
> important...

Maybe nobody bothers because using clang is easier than to fight with
FSF policies.

Helmut

Re: clang vs free software

[Rüdiger Sonderfeld]

On Thursday 23 January 2014 12:01:39 David Kastrup wrote:
> Richard Stallman <rms@gnu.org> writes:
> > [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> > [[[ whether defending the US Constitution against all enemies,     ]]]
> > [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
> > 
> >     There exist now various packages integrating emacs will elements of
> >     clang.
> >     
> >      These packages are not supporting clang as alternative compiler. 
> >      Rather
> >     
> >     by exploiting clang interfaces that have no gcc analog they offer
> >     exciting
> >     IDE-like features.
> > 
> > They nonetheless encourage people to switch from GCC to Clang.  The
> > features may be useful but this way of providing them hurts our cause.
> 
> Which brings us back to the question: what would be required to provide
> them via GCC or other GNU software?  If nobody bothers with even
> considering the question, it would appear that it is not all that
> important...

AFAIK there have been several attempts to make the AST or GCC frontend more 
easily available (e.g., gcc-xml) and there have been several discussions on 
the GCC mailing lists about this.  But all attempts were blocked by the fear 
that this might be abused by non-free software.

I personally think it was the wrong decision because it did hurt free software 
IDEs and tools, which would have benefited a lot from having a C++ frontend 
easily available.  Non-free software developers usually could afford non-free 
frontends such as EDG.

Now many free software IDEs and tools seem to start using or are developed 
around using libclang.  Which will promote clang over GCC in the long run.  I 
hope that the FSF and GCC developers change their position on this issue.  The 
risk that non-free software might abuse such a move seems to have been 
significantly reduced due to libclang's license being friendly to non-free 
software.

Regards,
Rüdiger

Re: clang vs free software

[David Engster]

Rüdiger Sonderfeld writes:
> AFAIK there have been several attempts to make the AST or GCC frontend more 
> easily available (e.g., gcc-xml) and there have been several discussions on 
> the GCC mailing lists about this.  But all attempts were blocked by the fear 
> that this might be abused by non-free software.

This is not true anymore. Simply compile a file with -fdump-tree-all and
you'll get more information than you'll ever need.

-David

Re: clang vs free software

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

We don't want to make a program's entire AST available for parsing
because that would make it easy to extend GCC with proprietary
programs.  If we had allowed this, we would have brought on the same
problem that LLVM is causing: GCC would then change from a free
compiler into a platform for nonfree compilers.

However, making available the symbol table (identifiers and their types)
would not cause this problem.  It just needs someone to write the code.

    Maybe nobody bothers because using clang is easier than to fight with
    FSF policies.

If you mean the policy that we don't let GCC become a platform for
proprietary compilers, what does it mean to think of this as something
to "fight"?

It means that one does not value defending freedom for the users.  It
means considering technical progress more important.

Those are the sort of values that make freedom vulnerable -- that gave
us LLVM, for instance.  The consequences will be seen in the non-free
compilers based on LLVM which, had they been based on GCC instead,
would have been free.


-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: clang vs free software

[Daniel Colascione]

On 01/25/2014 03:02 PM, Richard Stallman wrote:
> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
> We don't want to make a program's entire AST available for parsing
> because that would make it easy to extend GCC with proprietary
> programs.

It would also have made it easy to add modern features to Emacs and 
other free editors. Symbol table information is flatly inadequate given 
that in modern languages, typing information is highly contextual.

Users will adopt tools that provide these features when FSF programs 
support these features or not. If you keep these features out of GCC, 
users will go to Clang. If you keep Clang integration out of Emacs, 
users will either maintain out-of-tree integration or (eventually) just 
fork Emacs, as the various starter-kit packages have already essentially 
done.

Free software is great, but if nobody uses it, the entire enterprise is 
futile, sad, and ultimately irrelevant. How will the world be a better 
place when almost every every free operating system and free development 
environment is based on Clang and explicitly non-free derivatives are 
rampant?

Re: clang vs free software

[Lennart Borgman]

[-- Attachment #1: Type: text/plain, Size: 1465 bytes --]

On Sun, Jan 26, 2014 at 12:28 AM, Daniel Colascione <dancol@dancol.org>wrote:

> On 01/25/2014 03:02 PM, Richard Stallman wrote:
>
>> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
>> [[[ whether defending the US Constitution against all enemies,     ]]]
>> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>>
>> We don't want to make a program's entire AST available for parsing
>> because that would make it easy to extend GCC with proprietary
>> programs.
>>
>
> It would also have made it easy to add modern features to Emacs and other
> free editors. Symbol table information is flatly inadequate given that in
> modern languages, typing information is highly contextual.
>
> Users will adopt tools that provide these features when FSF programs
> support these features or not. If you keep these features out of GCC, users
> will go to Clang. If you keep Clang integration out of Emacs, users will
> either maintain out-of-tree integration or (eventually) just fork Emacs, as
> the various starter-kit packages have already essentially done.
>
> Free software is great, but if nobody uses it, the entire enterprise is
> futile, sad, and ultimately irrelevant. How will the world be a better
> place when almost every every free operating system and free development
> environment is based on Clang and explicitly non-free derivatives are
> rampant?
>
>
> +1

It even make it look like free software can't do this.

[-- Attachment #2: Type: text/html, Size: 2186 bytes --]

Re: clang vs free software

[David Kastrup]

Daniel Colascione <dancol@dancol.org> writes:

> On 01/25/2014 03:02 PM, Richard Stallman wrote:
>> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
>> [[[ whether defending the US Constitution against all enemies,     ]]]
>> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>>
>> We don't want to make a program's entire AST available for parsing
>> because that would make it easy to extend GCC with proprietary
>> programs.
>
> It would also have made it easy to add modern features to Emacs and
> other free editors. Symbol table information is flatly inadequate
> given that in modern languages, typing information is highly
> contextual.
>
> Users will adopt tools that provide these features when FSF programs
> support these features or not. If you keep these features out of GCC,
> users will go to Clang. If you keep Clang integration out of Emacs,
> users will either maintain out-of-tree integration or (eventually)
> just fork Emacs, as the various starter-kit packages have already
> essentially done.

It has been explained to you what the rationale behind making GCC
unsuitable as a black box component of parsing solutions is: there is no
point in licensing GCC under the GPL if we pull the basic teeth the GPL
has by allowing integration into a larger whole without having to obey
the GPL.

An adequate reaction to that would be to see what technical
possibilities there are for making GCC support the use cases you have in
mind without throwing it wide open, rendering its licensing choice
useless.

Instead, you ignore the reasons you have been given and choose to throw
a tantrum.  This will not serve to achieve anything but leave everybody
more annoyed.

> Free software is great, but if nobody uses it, the entire enterprise
> is futile, sad, and ultimately irrelevant.

That's exactly the situation GCC has started with.  So why would it have
become relevant?

> How will the world be a better place when almost every every free
> operating system and free development environment is based on Clang
> and explicitly non-free derivatives are rampant?

Because there will still be a free software solution available without
the need to hope that everybody will be playing nice when extending it.

-- 
David Kastrup

Re: clang vs free software

[Helmut Eller]

On Sun, Jan 26 2014, Richard Stallman wrote:

>     Maybe nobody bothers because using clang is easier than to fight with
>     FSF policies.
>
> If you mean the policy that we don't let GCC become a platform for
> proprietary compilers, what does it mean to think of this as something
> to "fight"?

With "fight" I mean explaining for the hundredth time that the FSF
policy of introducing artificial technical hurdles to prevent some
nonfree programs does

 a) cause more "collateral damage" than it prevents real damage.  If
    nonfree programs don't have access to the AST then so don't have
    free programs.

 b) does not actually work.  LLVM used GCC as front-end (known as
    llvm-gcc) before Clang existed and before GCC had a plugin
    mechanism.  Today the DragonEgg plugin makes it possible to use GCC
    front-ends, e.g. for Fortran or Go, with LLVM as backend.  Which
    sounds a lot like the thing that the FSF wants to avoid.

IMO, we would be better served with legal hurdles than with technical
hurdles.  E.g. the license could say that using GCC as platform for
proprietary compilers (DragonEgg) are not allowed, while using GCC as
platform for free compilers (or editors like Emacs) is allowed and
welcome.

(Clang/LLVM is free software, as far as I can tell. So discouraging
integration of Clang with Emacs has probably not so much to do with a
free/nonfree distinction but more with a gnu/nongnu distinction.)

Helmut

Re: clang vs free software

[David Kastrup]

Helmut Eller <eller.helmut@gmail.com> writes:

> On Sun, Jan 26 2014, Richard Stallman wrote:
>
>>     Maybe nobody bothers because using clang is easier than to fight with
>>     FSF policies.
>>
>> If you mean the policy that we don't let GCC become a platform for
>> proprietary compilers, what does it mean to think of this as something
>> to "fight"?
>
> With "fight" I mean explaining for the hundredth time that the FSF
> policy of introducing artificial technical hurdles to prevent some
> nonfree programs does

Since the whole point of the GPL is to introduce an "artificial hurdle"
preventing turning code into proprietary programs, and since it works,
according to the copyright laws it relies on, by covering works "as a
whole", any technical measures intended to provide an interface that
separates components into separate identifiable wholes have an effect on
the range of the GPL.

The GPL introduces restrictions, and making those restrictions be part
of an overall strategy requires making decisions that take into account
the reach of those restrictions.

That's not artificial.  It's an inherent consequence of the approach
taken by the GPL that we'll be explaining and weighing consequences for
the hundredth and the thousandth time since the whole purpose of the GPL
is to execute a measure of control over the consequences.

>  a) cause more "collateral damage" than it prevents real damage.

This does not get any more true by repeating it the thousandth time.  If
you take a look at something like MacOSX, it is a largely closed-down
system used for restricting user freedom, and its operating system basis
is BSD UNIX.  This is a real and lasting damage to the freedom of
software users, and Apple has been handed the power to keep doing this
damage by the undiscriminating licensing of BSD derivatives.

Most compilers for special processors like GPUs are kept locked down,
denying the freedom of users to work with their hardware according to
their own problems.  They also deny the freedom of other hardware
vendors to study and learn from the code and improve on the design of
both hard- and software, thus advancing the field in a manner where the
advances are available to everyone.

Yes, the mechanisms of the GPL works through restrictions, and
restrictions apply to everyone.  That is why we have to vet our
technical decisions against the purpose of the GPL and the respective
effects of the restrictions all the time.

[...]

> IMO, we would be better served with legal hurdles than with technical
> hurdles.

It is wishful thinking that one can be separated from the other.
Copyright covers copyrightable entities, and entities are determined
mostly by technical designs and decision.

> E.g. the license could say that using GCC as platform for proprietary
> compilers (DragonEgg) are not allowed, while using GCC as platform for
> free compilers (or editors like Emacs) is allowed and welcome.

No, that is absolutely one thing that the license could not say.
Whatever is not prohibited by default by copyright is _nothing_ that we
have any power to enforce.  The GPL is not a contract, it is a license.
It spells out the conditions under which the full restrictions granted
by copyright will get waived.  It is not in our power to add additional
restrictions.  Running a program that you legally acquired, for any
purpose whatsoever, is your default right.

It may be easy to forget given all the fairy tale restrictions big
vendors place into their "licenses" that are actually contracts with
various mechanism for purported agreement, but that's not how licenses
work.  Licenses are permissions.

> (Clang/LLVM is free software, as far as I can tell. So discouraging
> integration of Clang with Emacs has probably not so much to do with a
> free/nonfree distinction but more with a gnu/nongnu distinction.)

It has nothing to do with free/nonfree with regard to the use itself.
What it has to do with is with encouraging solutions that are clearly
not trying to be a _sustainable_ source of freedom.  It's worse in that
respect than FreeBSD/NetBSD and similar since _those_ are at least
currently driven by communities that are, for better or worse, fighting
for _their_ kind of freedom.  In contrast, a lot of the substantial
support for Clang comes from parties who are rather interested in having
an upstream available that can be convenient for their kind of
unfreedom.

That's an area that we don't want GCC to compete in.  So we want to make
our technical decisions in a manner where we don't open GCC up for
integration into a market of proprietary tools, and that means that it's
not possible to take the technical measures where GCC can be used as an
entirely separate and independently licensed component for free
solutions, either.

That's a balancing act, and the question is not whether to do that
balancing act (we would not have the GPL if we had not answered that
question with "yes") but how.

And it's rather exasperating if people keep pretending that nobody has
thought about this before and that they have the better answers
rendering decades of painstaking legal and technical work redundant.

-- 
David Kastrup

Re: clang vs free software

[Daniel Colascione]

On 01/25/2014 10:45 PM, David Kastrup wrote:
> Daniel Colascione <dancol@dancol.org> writes:
>
>> On 01/25/2014 03:02 PM, Richard Stallman wrote:
>>> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
>>> [[[ whether defending the US Constitution against all enemies,     ]]]
>>> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>>>
>>> We don't want to make a program's entire AST available for parsing
>>> because that would make it easy to extend GCC with proprietary
>>> programs.
>>
>> It would also have made it easy to add modern features to Emacs and
>> other free editors. Symbol table information is flatly inadequate
>> given that in modern languages, typing information is highly
>> contextual.
>>
>> Users will adopt tools that provide these features when FSF programs
>> support these features or not. If you keep these features out of GCC,
>> users will go to Clang. If you keep Clang integration out of Emacs,
>> users will either maintain out-of-tree integration or (eventually)
>> just fork Emacs, as the various starter-kit packages have already
>> essentially done.
>
> It has been explained to you what the rationale behind making GCC
> unsuitable as a black box component of parsing solutions is: there is no
> point in licensing GCC under the GPL if we pull the basic teeth the GPL
> has by allowing integration into a larger whole without having to obey
> the GPL.
>
> An adequate reaction to that would be to see what technical
> possibilities there are for making GCC support the use cases you have in
> mind without throwing it wide open, rendering its licensing choice
> useless.
>
> Instead, you ignore the reasons you have been given and choose to throw
> a tantrum.  This will not serve to achieve anything but leave everybody
> more annoyed.

The previous three paragraphs demonstrate succinctly why effort and 
attention have shifted away from GCC and toward LLVM. The latter system 
provides utility and none of the sanctimony. It allows users to 
accomplish their goals instead of hearing complaints that their goals 
are politically incompatible with the software.

Imagine applying your stance to web browsers: you might argue that a 
free web browser would be pointless if it allows users to run non-free 
JavaScript and subvert the GPL --- now strip the JavaScript interpreter 
from Firefox and see how much good that effort does users, software 
freedom, and the broader world.

Componentization is simply table stakes for a modern compiler suite. 
That this architecture permits interaction with non-free tools is 
unavoidable in the same way that free operating systems must be able to 
run non-free programs. Without this functionality, GCC's userbase will 
evaporate, and without users, GCC will become a very ineffective vehicle 
for advancing software freedom.

I hope that the GCC leadership comes to its senses here, because you're 
right about the component model encouraging the development of a 
non-free ecosystem. But users are still better off with a componentized 
GCC than they are with a componentized Clang: at least in the former 
case, the individual components retain their copyleft character, and 
free solutions can compete on their merits in niches were non-free 
components exist. In an equilibrium in which a permissively licensed 
compiler dominates, non-free software can simply appropriate from all 
components at will.

At the present rate, though, you might as well just add, on top of the 
GCC README, "look at my influence, ye mighty, and despair!".

>> Free software is great, but if nobody uses it, the entire enterprise
>> is futile, sad, and ultimately irrelevant.
>
> That's exactly the situation GCC has started with.  So why would it have
> become relevant?

Do you really need the difference spelled out for you? Previously, there 
was no free compiler of acceptable quality aside from GCC. Previously, 
integration with external tools was less important. Now Clang and LLVM 
exist. The world is much different than it was during GCC's earlier 
existence.  GCC will never again pull off another coup like the release 
of the Objective-C compiler.

>> How will the world be a better place when almost every every free
>> operating system and free development environment is based on Clang
>> and explicitly non-free derivatives are rampant?
>
> Because there will still be a free software solution available without
> the need to hope that everybody will be playing nice when extending it.

Will there be? If your "solution" doesn't actually meet user needs, it's 
not a solution. Users won't put up with significantly impaired 
functionality for the sake of using copyleft software when libre 
software (likely with non-free but gratis extensions) exists that 
fulfills their needs.

Re: clang vs free software

[Lennart Borgman]

[-- Attachment #1: Type: text/plain, Size: 5027 bytes --]

On Sun, Jan 26, 2014 at 1:13 PM, Daniel Colascione <dancol@dancol.org>wrote:

> On 01/25/2014 10:45 PM, David Kastrup wrote:
>
>> Daniel Colascione <dancol@dancol.org> writes:
>>
>>  On 01/25/2014 03:02 PM, Richard Stallman wrote:
>>>
>>>> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
>>>> [[[ whether defending the US Constitution against all enemies,     ]]]
>>>> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>>>>
>>>> We don't want to make a program's entire AST available for parsing
>>>> because that would make it easy to extend GCC with proprietary
>>>> programs.
>>>>
>>>
>>> It would also have made it easy to add modern features to Emacs and
>>> other free editors. Symbol table information is flatly inadequate
>>> given that in modern languages, typing information is highly
>>> contextual.
>>>
>>> Users will adopt tools that provide these features when FSF programs
>>> support these features or not. If you keep these features out of GCC,
>>> users will go to Clang. If you keep Clang integration out of Emacs,
>>> users will either maintain out-of-tree integration or (eventually)
>>> just fork Emacs, as the various starter-kit packages have already
>>> essentially done.
>>>
>>
>> It has been explained to you what the rationale behind making GCC
>> unsuitable as a black box component of parsing solutions is: there is no
>> point in licensing GCC under the GPL if we pull the basic teeth the GPL
>> has by allowing integration into a larger whole without having to obey
>> the GPL.
>>
>> An adequate reaction to that would be to see what technical
>> possibilities there are for making GCC support the use cases you have in
>> mind without throwing it wide open, rendering its licensing choice
>> useless.
>>
>> Instead, you ignore the reasons you have been given and choose to throw
>> a tantrum.  This will not serve to achieve anything but leave everybody
>> more annoyed.
>>
>
> The previous three paragraphs demonstrate succinctly why effort and
> attention have shifted away from GCC and toward LLVM. The latter system
> provides utility and none of the sanctimony. It allows users to accomplish
> their goals instead of hearing complaints that their goals are politically
> incompatible with the software.
>
> Imagine applying your stance to web browsers: you might argue that a free
> web browser would be pointless if it allows users to run non-free
> JavaScript and subvert the GPL --- now strip the JavaScript interpreter
> from Firefox and see how much good that effort does users, software
> freedom, and the broader world.
>
> Componentization is simply table stakes for a modern compiler suite. That
> this architecture permits interaction with non-free tools is unavoidable in
> the same way that free operating systems must be able to run non-free
> programs. Without this functionality, GCC's userbase will evaporate, and
> without users, GCC will become a very ineffective vehicle for advancing
> software freedom.
>
> I hope that the GCC leadership comes to its senses here, because you're
> right about the component model encouraging the development of a non-free
> ecosystem. But users are still better off with a componentized GCC than
> they are with a componentized Clang: at least in the former case, the
> individual components retain their copyleft character, and free solutions
> can compete on their merits in niches were non-free components exist. In an
> equilibrium in which a permissively licensed compiler dominates, non-free
> software can simply appropriate from all components at will.
>
> At the present rate, though, you might as well just add, on top of the GCC
> README, "look at my influence, ye mighty, and despair!".
>
>
>  Free software is great, but if nobody uses it, the entire enterprise
>>> is futile, sad, and ultimately irrelevant.
>>>
>>
>> That's exactly the situation GCC has started with.  So why would it have
>> become relevant?
>>
>
> Do you really need the difference spelled out for you? Previously, there
> was no free compiler of acceptable quality aside from GCC. Previously,
> integration with external tools was less important. Now Clang and LLVM
> exist. The world is much different than it was during GCC's earlier
> existence.  GCC will never again pull off another coup like the release of
> the Objective-C compiler.
>
>
>  How will the world be a better place when almost every every free
>>> operating system and free development environment is based on Clang
>>> and explicitly non-free derivatives are rampant?
>>>
>>
>> Because there will still be a free software solution available without
>> the need to hope that everybody will be playing nice when extending it.
>>
>
> Will there be? If your "solution" doesn't actually meet user needs, it's
> not a solution. Users won't put up with significantly impaired
> functionality for the sake of using copyleft software when libre software
> (likely with non-free but gratis extensions) exists that fulfills their
> needs.
>
> +1

[-- Attachment #2: Type: text/html, Size: 6917 bytes --]

Re: clang vs free software

[Helmut Eller]

On Sun, Jan 26 2014, David Kastrup wrote:

[...]
> And it's rather exasperating if people keep pretending that nobody has
> thought about this before and that they have the better answers
> rendering decades of painstaking legal and technical work redundant.

Now you have me convinced: working on GCC/Emacs integration would be a
waste of my time.

Helmut

Re: clang vs free software

[David Kastrup]

Daniel Colascione <dancol@dancol.org> writes:

> On 01/25/2014 10:45 PM, David Kastrup wrote:
>
>> It has been explained to you what the rationale behind making GCC
>> unsuitable as a black box component of parsing solutions is: there is no
>> point in licensing GCC under the GPL if we pull the basic teeth the GPL
>> has by allowing integration into a larger whole without having to obey
>> the GPL.
>>
>> An adequate reaction to that would be to see what technical
>> possibilities there are for making GCC support the use cases you have in
>> mind without throwing it wide open, rendering its licensing choice
>> useless.
>>
>> Instead, you ignore the reasons you have been given and choose to throw
>> a tantrum.  This will not serve to achieve anything but leave everybody
>> more annoyed.
>
> The previous three paragraphs demonstrate succinctly why effort and
> attention have shifted away from GCC and toward LLVM.  The latter
> system provides utility and none of the sanctimony.

Quite like an Apple iPhone provides utility and none of the sanctimony.
The purpose of the GNU project is not to serve as many people as
possible at a given point of time but to ensure the continued
availability of Free Software to its users, those that care about it.

There will be people who don't care about it.  They may or may not use
Free Software for other reasons, and they are free to do so.  But the
principal target of Free Software are those who care about software
being and remaining free, not those who care foremost about convenience.

> It allows users to accomplish their goals instead of hearing
> complaints that their goals are politically incompatible with the
> software.

If you choose not to listen, you can at least refrain from
misrepresentation.  What happens here is that users are told that their
requirements are not easily met while ensuring at the same time that
work built based on this software will remain freely available for
everyone.

If you refuse participating with the effort of reconciling your wishlist
with the goals of Free Software and think by just ignoring everything
said and stomping your feet and yelling "but I want it!", then you will
not be part of the effort of finding better solutions.

> In an equilibrium in which a permissively licensed compiler dominates,
> non-free software can simply appropriate from all components at will.

Which is exactly why the GNU project is not interested in supporting
LLVM-based solutions over that using GCC.

> At the present rate, though, you might as well just add, on top of the
> GCC README, "look at my influence, ye mighty, and despair!".

Take a hike, seriously.  I have no influence at all.  I do not represent
the GNU project in any form or capacity.  All that I am doing is trying
to save you some of the work you'd need to invest to understand the
issues involved so that you can meaningfully participate in a mature
solution finding process.

If you are not interested in doing that, the least you can do is refrain
from poisoning the atmosphere in mailing lists that are at best
marginally topical for your rants.

> Do you really need the difference spelled out for you? Previously,
> there was no free compiler of acceptable quality aside from
> GCC.

pcc has been available for decades, and all the commercial Unix variants
relied on it.  Get your history right.

> Previously, integration with external tools was less important. Now
> Clang and LLVM exist. The world is much different than it was during
> GCC's earlier existence.  GCC will never again pull off another coup
> like the release of the Objective-C compiler.

>>> How will the world be a better place when almost every every free
>>> operating system and free development environment is based on Clang
>>> and explicitly non-free derivatives are rampant?
>>
>> Because there will still be a free software solution available without
>> the need to hope that everybody will be playing nice when extending it.
>
> Will there be? If your "solution" doesn't actually meet user needs,
> it's not a solution. Users won't put up with significantly impaired
> functionality for the sake of using copyleft software when libre
> software (likely with non-free but gratis extensions) exists that
> fulfills their needs.

Those users you are talking about are not, fortunately, dictating the
choices of the GNU project.  If they were, there would be no GCC, no
GNU, and not even the GPLed Linux kernel.

It is a visibility problem in a way that GNU and Free Software have
become popular for reasons other than what they have been created for
and that a large majority of users could care less about its original
mission and act up a storm whenever they find some of the basic choices
underlying GNU to affect their convenience.

As long as you are not interested in working on solutions, you are not
doing anybody a favor by venting.

-- 
David Kastrup

Re: clang vs free software

[David Kastrup]

Helmut Eller <eller.helmut@gmail.com> writes:

> On Sun, Jan 26 2014, David Kastrup wrote:
>
> [...]
>> And it's rather exasperating if people keep pretending that nobody has
>> thought about this before and that they have the better answers
>> rendering decades of painstaking legal and technical work redundant.
>
> Now you have me convinced: working on GCC/Emacs integration would be a
> waste of my time.

If you are not willing to communicate with those who have to make and
support decisions for maintaining the role GCC plays for promoting Free
Software, it is indeed quite likely that you would invest time and
effort that will ultimately not be integrated into GCC upstream (of
course, you can still distribute your own GPLed version of GCC).

Of course, this "I'll take over your project now and do everything right
since you are incapable of doing so" attitude is not likely to lead to
convincing results in a lot of other endeavors not necessarily related
to Free Software.

-- 
David Kastrup

Re: clang vs free software

[Stefan Monnier]

> We don't want to make a program's entire AST available for parsing
> because that would make it easy to extend GCC with proprietary
> programs.

Let me just say that even tho I understand and agree with the goal,
I disagree.


        Stefan

Re: clang vs free software

[Lennart Borgman]

[-- Attachment #1: Type: text/plain, Size: 348 bytes --]

On Sun, Jan 26, 2014 at 4:12 PM, Stefan Monnier <monnier@iro.umontreal.ca>wrote:

> > We don't want to make a program's entire AST available for parsing
> > because that would make it easy to extend GCC with proprietary
> > programs.
>
> Let me just say that even tho I understand and agree with the goal,
> I disagree.
>
>
>         Stefan
>
> +1

[-- Attachment #2: Type: text/html, Size: 1025 bytes --]

Re: clang vs free software

[David Kastrup]

Lennart Borgman <lennart.borgman@gmail.com> writes:

> On Sun, Jan 26, 2014 at 4:12 PM, Stefan Monnier <monnier@iro.umontreal.ca>wrote:
>
>> > We don't want to make a program's entire AST available for parsing
>> > because that would make it easy to extend GCC with proprietary
>> > programs.
>>
>> Let me just say that even tho I understand and agree with the goal,
>> I disagree.
>>
>> +1

Too bad the GNU project is not a democracy where you carry around
protest signs and then somebody is supposed to do the work for making
things so that you like them.

Finding solutions takes more work than saying "I disagree".

Seriously.

I am glad I don't have Richard's totally ungrateful job, but then he
picked it himself.

-- 
David Kastrup

Re: clang vs free software

[Lennart Borgman]

[-- Attachment #1: Type: text/plain, Size: 682 bytes --]

On Sun, Jan 26, 2014 at 4:35 PM, David Kastrup <dak@gnu.org> wrote:

> Lennart Borgman <lennart.borgman@gmail.com> writes:
>
> > On Sun, Jan 26, 2014 at 4:12 PM, Stefan Monnier <
> monnier@iro.umontreal.ca>wrote:
> >
> >> > We don't want to make a program's entire AST available for parsing
> >> > because that would make it easy to extend GCC with proprietary
> >> > programs.
> >>
> >> Let me just say that even tho I understand and agree with the goal,
> >> I disagree.
> >>
> >> +1
>
> I am glad I don't have Richard's totally ungrateful job, but then he
> picked it himself.
>
> I think you are misinterpreting goal vs policies. The comment is in line
with the long term goal.

[-- Attachment #2: Type: text/html, Size: 1994 bytes --]

Re: clang vs free software

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

    > We don't want to make a program's entire AST available for parsing
    > because that would make it easy to extend GCC with proprietary
    > programs.

    It would also have made it easy to add modern features to Emacs and 
    other free editors. Symbol table information is flatly inadequate given 
    that in modern languages, typing information is highly contextual.

You seem to have interpreted "symbol table information" in the
narrowest possible way.  Try a broader interpretation, please.

    Free software is great, but if nobody uses it, the entire enterprise is 
    futile, sad, and ultimately irrelevant.

Free software is equally futile, sad, and irrelevant if everyone uses
it...as a base for nonfree software.

You mention the threat that GCC might be superseded.  That is real.
But the threat that GCC might become a base for nonfree software is
also real -- and has been real for 25 years.  This sad irrelevance
could have happened decades ago if we had not taken care to prevent
it.  I saw it close up when I met with Jobs at Next, and he asked me
if they could distribute their Objective C front end as binaries and
let the user link it with the .o files of GCC.

For freedom's sake I had to find a path between those two threats.  I
think I have done a pretty good job; but ultimately Jobs dedicated the
resources of a powerful company that hates freedom to fighting us.

Perhaps there was a better way to do it.  I am not perfect as a
strategist, after all.  But you can't find a better way to avoid both
threats unless you recognize them both.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: clang vs free software

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

    The previous three paragraphs demonstrate succinctly why effort and 
    attention have shifted away from GCC and toward LLVM. The latter system 
    provides utility and none of the sanctimony.

If you deride our ethical goals as "sanctimony", you should not be
surprised when the GNU Project ignores your advice: it's based on
rejection of our goals.

Furthermore, insulting us is not going to win you anything.

We don't cater to people whose goals differ from ours, and certainly
not to people who badger us with demands.  You've stated your views,
now please hold your peace.

    Imagine applying your stance to web browsers: you might argue that a 
    free web browser would be pointless if it allows users to run non-free 
    JavaScript and subvert the GPL

Nonfree JavaScript code is a big threat to web users' freedom.  Many
sites are simply unusable for the free world.  See
http://www.gnu.org/philosophy/javascript-trap.html.

I wish we had been able to start our campaign against nonfree
Javascript 5 years ago or more, before the web became so pervasively
polluted with it.  We are starting from very far behind, but it
has to be done.

The Javascript issue is different in its details from the GCC issue,
both technically and legally.  It is similar only in the most general
issue at stake.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: clang vs free software

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

    IMO, we would be better served with legal hurdles than with technical
    hurdles.  E.g. the license could say that using GCC as platform for
    proprietary compilers (DragonEgg) are not allowed, while using GCC as
    platform for free compilers (or editors like Emacs) is allowed and
    welcome.

We have made an effort in this direction already.  I'm in favor of
doing more.  Designing such schemes is not easy, but we can try.
They need to be based on specific real scenarios, so we can find the
place to draw the line between the good scenarios and the bad ones.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.