Completions in Semantic

Completions in Semantic

[Chong Yidong]

Hi Eric,

The completion code implemented by the semantic/complete package appears
to be quirky.  For instance:

 emacs
 M-x semantic-mode RET
 C-x C-f ~/emacs/src/xdisp.c   [visit xdisp.c in in the Emacs tree]
 C-c , j
 window[TAB]

Semantic now displays a *Completions* window with the following
contents:

 Possible completions are:
 window

But this message is misleading, because that's definitely not the only
completion.  Type:

 _[TAB]

and the *Completions* window is updated with

 Possible completions are:
 window_text_bottom_y 	window_box_width 	window_box_height

On a more general note, this is one of the drawbacks to the approach
taken in semantic/complete.el, i.e. trying to reimplement completion.
We've painstakingly ironed out a lot of bugs in the standard Emacs
completion code, and it makes no sense to redo all that for Semantic
unless there's a good reason.  (Not to mention inconsistency issues).

In the commentary to semantic/complete.el, you write that using
`all-completions' and `try-completion' isn't practical because they are
too slow when there are large numbers of tags.  How sure are you that
that's true?  Nowadays, the completions code is very flexible; for
instance, you can pass `all-completions' and `try-completion' a
collector function instead of a flat completion table.  Can't
semantic/complete.el make use of this functionality?

Re: Completions in Semantic

[Miles Bader]

Chong Yidong <cyd@stupidchicken.com> writes:
> On a more general note, this is one of the drawbacks to the approach
> taken in semantic/complete.el, i.e. trying to reimplement completion.
> We've painstakingly ironed out a lot of bugs in the standard Emacs
> completion code, and it makes no sense to redo all that for Semantic
> unless there's a good reason.  (Not to mention inconsistency issues).

Yes, and it won't properly respect user customizations of the completion
stuff.

[I've often felt that semantic was acting "weird" during completion, but
it's hard to pin down exactly what the problem is (by the time I realize
that it is, I often can't reproduce the offending key sequence).]

-Miles

-- 
Vote, v. The instrument and symbol of a freeman's power to make a fool of
himself and a wreck of his country.

Re: Completions in Semantic

[Eric M. Ludlam]

Hi,

You are right, it is a bit quirky.  The comment in my code about using
try-completion being slow isn't about try-completion itself.  It is
about Semantic converting its table set into something a more normal
completion prompt can use.  For a large code base I work on, it used to
take about 8 minutes.  (I profiled it to make sure it wasn't hung, and
that also slows things down a bunch, so maybe 1 minute unprofiled?)  The
semantic/completion code takes a shortest path approach with what you
type, keeping track of old completion lists and tables and refining it
as you go.  I don't doubt that there's some funny stuff there that
provides the quirkyness.  In your example, I think it's just because
"window" is an exact match.

If you look at semantic-read-symbol, it does what you suggest, and
builds a table from the current buffer, and passes the whole thing into
completing-read.  That's ok for a single buffer, but not when you want
some symbol from anywhere in your project.  Of course, there is a basic
assumption that the user will do something like your example below,
typing in some prefix, before pressing TAB.  If the prompt comes up and
the user presses TAB right away, then there is no win.

Anyway, semantic-read-symbol works fine unless there are multiple
symbols with the same name, thus, a second thing that made me write my
own completion code is that tags are a little special, since they have
both names, and spacial data.  Sometimes, two tags are only different
based on their spacial data.  For example, when jumping to a tag, if
there are many hits with the same name (such as "window" in your example
below), you can press TAB several times, and it will flash where you
would jump to if you hit return, allowing you to differentiate.

For your particular example in xdisp.c, it seems the contents of xdisp.c
fails to parse properly.  It seems to be related to the P_ macro, and
one of the uses of it involving a line-end in the middle.  I've narrowed
it down and will see if I can fix it.

The long and short of all that is, if someone wants to take on a better
(consistent) completion prompt, that'd be great, and I'd be happy to do
the back-end side of things as needed.  (When writing the semantic
completion system, I discovered that I don't really like writing
completion prompts.)  The API you describe sounds (in brief) sufficient
to do some of what is needed.  In the meantime, the current version
works for me going back to Emacs 21, and XEmacs, so I'll stick with it
till I can get my current release out.  After that I'd want to focus on
more recent Emacsen and simplifying that stuff sounds like a fine idea.

Eric

On Sun, 2009-10-18 at 19:00 -0400, Chong Yidong wrote:
> Hi Eric,
> 
> The completion code implemented by the semantic/complete package appears
> to be quirky.  For instance:
> 
>  emacs
>  M-x semantic-mode RET
>  C-x C-f ~/emacs/src/xdisp.c   [visit xdisp.c in in the Emacs tree]
>  C-c , j
>  window[TAB]
> 
> Semantic now displays a *Completions* window with the following
> contents:
> 
>  Possible completions are:
>  window
> 
> But this message is misleading, because that's definitely not the only
> completion.  Type:
> 
>  _[TAB]
> 
> and the *Completions* window is updated with
> 
>  Possible completions are:
>  window_text_bottom_y 	window_box_width 	window_box_height
> 
> On a more general note, this is one of the drawbacks to the approach
> taken in semantic/complete.el, i.e. trying to reimplement completion.
> We've painstakingly ironed out a lot of bugs in the standard Emacs
> completion code, and it makes no sense to redo all that for Semantic
> unless there's a good reason.  (Not to mention inconsistency issues).
> 
> In the commentary to semantic/complete.el, you write that using
> `all-completions' and `try-completion' isn't practical because they are
> too slow when there are large numbers of tags.  How sure are you that
> that's true?  Nowadays, the completions code is very flexible; for
> instance, you can pass `all-completions' and `try-completion' a
> collector function instead of a flat completion table.  Can't
> semantic/complete.el make use of this functionality?

Re: Completions in Semantic

[Stefan Monnier]

> If you look at semantic-read-symbol, it does what you suggest, and
> builds a table from the current buffer, and passes the whole thing into
> completing-read.  That's ok for a single buffer, but not when you want
> some symbol from anywhere in your project.  Of course, there is a basic
> assumption that the user will do something like your example below,
> typing in some prefix, before pressing TAB.  If the prompt comes up and
> the user presses TAB right away, then there is no win.

I'm not sure I understand what you're saying, but it seems what you
describe can be done as well with the "standard" completion mechanism,
since the completion-table can be a function.

E.g. Info-complete-menu-item builds the table dynamically, using the
prefix to try and build a smaller table, and reusing the previously
built table if the completion is in the same context as the last one and
the earlier prefix is a prefix of the new one.

> Anyway, semantic-read-symbol works fine unless there are multiple
> symbols with the same name, thus, a second thing that made me write my
> own completion code is that tags are a little special, since they have
> both names, and spacial data.  Sometimes, two tags are only different
> based on their spacial data.  For example, when jumping to a tag, if
> there are many hits with the same name (such as "window" in your example
> below), you can press TAB several times, and it will flash where you
> would jump to if you hit return, allowing you to differentiate.

That would require some custom completion code, indeed, since the
current completion does not offer any hook for such things.  Of course,
we could add such a hook, but we'd need to figure out where/how to
do it.


        Stefan

Re: Completions in Semantic

[Eric M. Ludlam]

On Mon, 2009-10-19 at 09:48 -0400, Stefan Monnier wrote:
> > If you look at semantic-read-symbol, it does what you suggest, and
> > builds a table from the current buffer, and passes the whole thing into
> > completing-read.  That's ok for a single buffer, but not when you want
> > some symbol from anywhere in your project.  Of course, there is a basic
> > assumption that the user will do something like your example below,
> > typing in some prefix, before pressing TAB.  If the prompt comes up and
> > the user presses TAB right away, then there is no win.
> 
> I'm not sure I understand what you're saying, but it seems what you
> describe can be done as well with the "standard" completion mechanism,
> since the completion-table can be a function.
> 
> E.g. Info-complete-menu-item builds the table dynamically, using the
> prefix to try and build a smaller table, and reusing the previously
> built table if the completion is in the same context as the last one and
> the earlier prefix is a prefix of the new one.

I agree.  It just wasn't available when I wrote that stuff against Emacs
21.  (2003?)

> > Anyway, semantic-read-symbol works fine unless there are multiple
> > symbols with the same name, thus, a second thing that made me write my
> > own completion code is that tags are a little special, since they have
> > both names, and spacial data.  Sometimes, two tags are only different
> > based on their spacial data.  For example, when jumping to a tag, if
> > there are many hits with the same name (such as "window" in your example
> > below), you can press TAB several times, and it will flash where you
> > would jump to if you hit return, allowing you to differentiate.
> 
> That would require some custom completion code, indeed, since the
> current completion does not offer any hook for such things.  Of course,
> we could add such a hook, but we'd need to figure out where/how to
> do it.

There are a lot of interesting completing completion engines out there
on Emacswiki.  I would expect that a discussion on this topic would
invite a lot of opinions from different Emacs hackers.

Here are some of the things that could be discussed as worthwhile in the
built-in completion system:

* Differentiate between similarly named entities (as above).
  I called it a 'focus', meaning the user was focusing on 1 of many 
  options, but has not chosen it.
* Completion list with different text from what shows up on the prompt
  (another way to differentiate.)
* Inline completion in a buffer.  (ie - keymap, recursive-edit, 
  minibuffer like, but in the buffer text with the intention of 
  inserting that text..)
* Option for using a CLOS object as an input to these tools.
  (ie - completion classes inherit from some interface.)
  as a way of managing persistent data between completion events.
  Just a favorite of mine, I'm sure.

Thanks
Eric

Re: Completions in Semantic

[Stefan Monnier]

>> E.g. Info-complete-menu-item builds the table dynamically, using the
>> prefix to try and build a smaller table, and reusing the previously
>> built table if the completion is in the same context as the last one and
>> the earlier prefix is a prefix of the new one.

> I agree.  It just wasn't available when I wrote that stuff against Emacs
> 21.  (2003?)

Not that it matters, but the version of minibuf.c from "Sun 1993-05-23
03:32:05 +0000" says:

   DEFUN ("try-completion", Ftry_completion, Stry_completion, 2, 3, 0,
   [...]
   ALIST can also be a function to do the completion itself.\n\
   It receives three arguments: the values STRING, PREDICATE and nil.\n\
   Whatever it returns becomes the value of `try-completion'.\n\

> * Inline completion in a buffer.  (ie - keymap, recursive-edit,
>   minibuffer like, but in the buffer text with the intention of
>   inserting that text..)

You mean like lisp-complete-symbol but modal?

> * Option for using a CLOS object as an input to these tools.
>   (ie - completion classes inherit from some interface.)
>   as a way of managing persistent data between completion events.
>   Just a favorite of mine, I'm sure.

The functions that you can pass to try-completion and friends are
basically objcts: the 3rd argument is the method to call.  So it should
be trivial to write a wrapper that takes a CLOS object obeying
a completion-interface and returns a lambda expression suitable for
try-completion and friends.


        Stefan

Re: Completions in Semantic

[Eric M. Ludlam]

On Mon, 2009-10-19 at 14:29 -0400, Stefan Monnier wrote:

> > * Inline completion in a buffer.  (ie - keymap, recursive-edit,
> >   minibuffer like, but in the buffer text with the intention of
> >   inserting that text..)
> 
> You mean like lisp-complete-symbol but modal?

Yes, that's what I meant, but in light of your simplified description, I
suppose it doesn't matter much.  If there was a single function like
'completing-read' that took arguments to control what proposed text
after the cursor was, I suppose it wouldn't matter what happened
underneath.  I used overlays and overlay keymaps.

You could try semantic-complete-analyze-inline on something that needs
completing to see the effect I was going for.  It should work for Lisp
or C code.

I would expect designing such a thing would be challenging since the
rules for what is before the cursor is a bit vague and language
specific, so it might belong in a tool like Semantic.

> > * Option for using a CLOS object as an input to these tools.
> >   (ie - completion classes inherit from some interface.)
> >   as a way of managing persistent data between completion events.
> >   Just a favorite of mine, I'm sure.
> 
> The functions that you can pass to try-completion and friends are
> basically objcts: the 3rd argument is the method to call.  So it should
> be trivial to write a wrapper that takes a CLOS object obeying
> a completion-interface and returns a lambda expression suitable for
> try-completion and friends.

I hadn't thought of that.  That sounds like a cool hack idea I'll have
to fiddle with.

Not as nice as an official interface though. ;)

Eric

Re: Completions in Semantic

[Stefan Monnier]

>> You mean like lisp-complete-symbol but modal?

> Yes, that's what I meant, but in light of your simplified description, I
> suppose it doesn't matter much.  If there was a single function like
> 'completing-read' that took arguments to control what proposed text
> after the cursor was, I suppose it wouldn't matter what happened
> underneath.  I used overlays and overlay keymaps.

> You could try semantic-complete-analyze-inline on something that needs
> completing to see the effect I was going for.  It should work for Lisp
> or C code.

> I would expect designing such a thing would be challenging since the
> rules for what is before the cursor is a bit vague and language
> specific, so it might belong in a tool like Semantic.

We have `symbol-complete' although I'm not very happy about its API.
But something along thses lines would be good.  When I rewrite the
minibuffer completion code I tried to make it possible to use it in
non-minibuffer contexts, so for example lisp-complete-symbol nowadays
uses minibuffer-complete internally.

I also intend to consolidate the comint completion this way, tho it's
more difficult because its current UI is slightly different.  Part of
the difficulty is how to determine when the "completion is done" so as
to know when to pop-down the *Completions* buffer (if any).  Maybe the
best option is to pop-down *Completions* eagerly ("all the time").


        Stefan

Re: Completions in Semantic

[Eric M. Ludlam]

On Mon, 2009-10-19 at 16:06 -0400, Stefan Monnier wrote:
> >> You mean like lisp-complete-symbol but modal?
> 
> > Yes, that's what I meant, but in light of your simplified description, I
> > suppose it doesn't matter much.  If there was a single function like
> > 'completing-read' that took arguments to control what proposed text
> > after the cursor was, I suppose it wouldn't matter what happened
> > underneath.  I used overlays and overlay keymaps.
> 
> > You could try semantic-complete-analyze-inline on something that needs
> > completing to see the effect I was going for.  It should work for Lisp
> > or C code.
> 
> > I would expect designing such a thing would be challenging since the
> > rules for what is before the cursor is a bit vague and language
> > specific, so it might belong in a tool like Semantic.
> 
> We have `symbol-complete' although I'm not very happy about its API.
> But something along thses lines would be good.  When I rewrite the
> minibuffer completion code I tried to make it possible to use it in
> non-minibuffer contexts, so for example lisp-complete-symbol nowadays
> uses minibuffer-complete internally.

I was unaware of that, but it sounds useful.

> I also intend to consolidate the comint completion this way, tho it's
> more difficult because its current UI is slightly different.  Part of
> the difficulty is how to determine when the "completion is done" so as
> to know when to pop-down the *Completions* buffer (if any).  Maybe the
> best option is to pop-down *Completions* eagerly ("all the time").

For my completion engine, I used an overlay to wrap the text being
completed.  As soon as the cursor leaves the overlay, or if the user
types a character that doesn't belong in the symbol such as SPC, ., or
other punctuation, then it exists.  The post-command-hook is a bit
complex during the semantic inline completion.  Perhaps that style of
interface could be used, if not directly.

I was going for something where the completion part could activate at
idle time without interfering with regular editing commands.  ie, if the
user was typing away or using point motion, nothing would change, but if
they saw the decoration indicating active completion, they would know
that TAB was special.

A side effect is that a *completions* buffer is not always desired, so
there are some experiments on completion showing that might not need a
buffer, such as the tooltip, or ghost-text.  These options have varying
success in this task.

The semantic complete code for inline stuff uses the same completion and
display engines as the minibuffer or inline prompts, so you can mix and
match the pieces.  Hopefully an official API can do something similar,
so you can complete in the minibuffer or inline with the same functions.

Eric

Re: Completions in Semantic

[Eric M. Ludlam]

On Sun, 2009-10-18 at 23:50 -0400, Eric M. Ludlam wrote:
> For your particular example in xdisp.c, it seems the contents of
> xdisp.c
> fails to parse properly.  It seems to be related to the P_ macro, and
> one of the uses of it involving a line-end in the middle.  I've
> narrowed
> it down and will see if I can fix it.

To reply to myself, I checked changes into the CEDET repository for the
C preprocessor handling to fix the bug found here in xdisp.

By way of side effect, the completion problem originally posed is also
fixed, as there is no "window" symbol in the file anymore.

Eric

Re: Completions in Semantic

[Lennart Borgman]

On Tue, Oct 20, 2009 at 12:17 AM, Eric M. Ludlam <eric@siege-engine.com> wrote:
>
> The semantic complete code for inline stuff uses the same completion and
> display engines as the minibuffer or inline prompts, so you can mix and
> match the pieces.  Hopefully an official API can do something similar,
> so you can complete in the minibuffer or inline with the same functions.

I think the authors of company-mode and completion-UI (Nikolaj and
Toby) are interested in this too. And Drew, of course.

Re: Completions in Semantic

[Stefan Monnier]

> For my completion engine, I used an overlay to wrap the text being
> completed.

That's also what lisp-complete-symbol uses to tell minibuffer-completion
what is to be completed (more specifically minibuffer-completion
operates on a `field', so I create an overlay with a field property).

> As soon as the cursor leaves the overlay, or if the user
> types a character that doesn't belong in the symbol such as SPC, ., or
> other punctuation, then it exists.

That's the kind of UI I was thinking of, yes.  Can you point me to the
relevant code you use for that (especially things like
post-command-hook)?

> A side effect is that a *completions* buffer is not always desired, so
> there are some experiments on completion showing that might not need a
> buffer, such as the tooltip, or ghost-text.  These options have varying
> success in this task.

Yes, rather than a *completions* buffer, we could have tooltip-like
frames show up.  That will need to be chosen by the caller.


        Stefan

Re: Completions in Semantic

[Eric M. Ludlam]

On Mon, 2009-10-19 at 20:14 -0400, Stefan Monnier wrote:
> > For my completion engine, I used an overlay to wrap the text being
> > completed.
> 
> That's also what lisp-complete-symbol uses to tell minibuffer-completion
> what is to be completed (more specifically minibuffer-completion
> operates on a `field', so I create an overlay with a field property).
> 
> > As soon as the cursor leaves the overlay, or if the user
> > types a character that doesn't belong in the symbol such as SPC, ., or
> > other punctuation, then it exists.
> 
> That's the kind of UI I was thinking of, yes.  Can you point me to the
> relevant code you use for that (especially things like
> post-command-hook)?

It is in the cedet/semantic/complete.el, and the function is
semantic-complete-post-command-hook for the inline completion exit
conditions.

Enjoy
Eric

Re: Completions in Semantic

[Lluis]

Just as a side-note, now that you've raised the topic of a completion interface.

I don't know about the features of current code, but I miss some features on
some (if not all) the completion interfaces I've tested. So here's a list of
(my) desired features (all of which should be configurable by the user):

- symbol name
  Of course, this is what already provides every completion UI
- extra information
  This can be filled ub with symbol "metadata". Which metadata appears on the
  completion UI should be configurable by the user:
    - return type
    - arguments (type and/or name)
    - definition location (aka file)
    - short documentation
    - long documentation
  All this metadata should be located anywhere around the symbol name and/or the
  minibuffer (e.g., I think company-mode shows short documentation on
  minibuffer, until user presses F1, when full documentation is shown.
  Some metadata might be shown in the minibuffer after completion selection
  (e.g., prototype).
- result narrowing
  A-la company-mode.
- argument placeholders
  So that argument type and/or name is shown as placeholders, such that the user
  simply TABs (or whatever) to fill-in the blanks.

In any case, I've recently jumped into Emacs, so my elisp coding abilities are
still quite low...

Read you,
    Lluis

-- 
 "And it's much the same thing with knowledge, for whenever you learn
 something new, the whole world becomes that much richer."
 -- The Princess of Pure Reason, as told by Norton Juster in The Phantom
 Tollbooth

Re: Completions in Semantic

[Eric M. Ludlam]

On Wed, 2009-10-21 at 12:58 +0200, Lluis wrote:
> Just as a side-note, now that you've raised the topic of a completion interface.
> 
> I don't know about the features of current code, but I miss some features on
> some (if not all) the completion interfaces I've tested. So here's a list of
> (my) desired features (all of which should be configurable by the user):
> 
> - symbol name
>   Of course, this is what already provides every completion UI
> - extra information
>   This can be filled ub with symbol "metadata". Which metadata appears on the
>   completion UI should be configurable by the user:
>     - return type
>     - arguments (type and/or name)
>     - definition location (aka file)
>     - short documentation
>     - long documentation
>   All this metadata should be located anywhere around the symbol name and/or the
>   minibuffer (e.g., I think company-mode shows short documentation on
>   minibuffer, until user presses F1, when full documentation is shown.
>   Some metadata might be shown in the minibuffer after completion selection
>   (e.g., prototype).
> - result narrowing
>   A-la company-mode.
> - argument placeholders
>   So that argument type and/or name is shown as placeholders, such that the user
>   simply TABs (or whatever) to fill-in the blanks.

Hi Lluis,

Just as an FYI about the semantic tag info (which you seem to be
alluding to), is that it provides all that info, though not necessarily
from the completion engine via a UI.

The internal tag table format of semantic includes data type info,
argument lists, positional info, and the doc is available.  You can also
pass a semantic native tag table directly into 'try-completions' or any
other completion prompt in Emacs.

The completion engine in Semantic also takes advantage of this and
allows ways to focus on a tag based on it's positional information, or
distinguish based on arg list or some such.  Since it uses a code
analyzer, you tend not to need too much in the way of narrowing of scope
since it knows what symbols don't fit in a particular context, so it
doesn't provide them as completions.

If core emacs completion prompts were to select some metadata based
approach to completion tables, the Semantic tag format is pretty simple,
and could provide what is needed.  A tag table is a list, with each
element formated like this:

("NAME" CLASS ATTRIBUTES PROPERTIES POSITION)

Where attributes and properties are plists, and POSITION is an optional
element which could be either an overlay, or a vector of two numbers.
Attributes are features of whatever the tag is, such as arg lists, or
datatype.  Properties are features of the tag itself, like the filename
it is in, or other data needed to maintain the tag, like parser
information.  Of course there may not be a use case for this outside of
tag data, so perhaps it is a moot point.

Eric

Re: Completions in Semantic

[Lluis]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=utf-8, Size: 2970 bytes --]

El Wed, Oct 21, 2009 at 08:35:29AM -0400, Eric M. Ludlam ens deleità amb les següents paraules:
> Just as an FYI about the semantic tag info (which you seem to be
> alluding to), is that it provides all that info, though not necessarily
> from the completion engine via a UI.

Well, it's been a long time since I played with the completion interfaces
provided in CEDET, but I already knew all that information was available
somewhere in semantic :)

I just didn't know documentation was also available as one of the tag's
attributes, which reminds me that:
    1) One-liner doxygen description shows up with the `*/' at the end
    2) Descriptions of the form: /*< description */ 
       are not assigned to the correct variable
    3) One more nice feature would be to have the "show long documentation"
       parse doxygen-specific commands such that parameters descriptions are
       shown more clearly

I also tried with `completion-ui', `auto-complete' and `company-mode', which are
the ones I've found that provide completion UIs through overlays (hope I'm using
the correct term), which is the form i feel more comfortable with.

But as far as I've been able to see, none of these three provide the features
I've mentioned, although they do provide semantic-based backends for completion.


[...]
> Since it uses a code analyzer, you tend not to need too much in the way of
> narrowing of scope since it knows what symbols don't fit in a particular
> context, so it doesn't provide them as completions.

Right, that's just a feature I found in company-mode, and thought it was nice
when working with a large amount of possible completions; which is not usually
my case, now that I work mostly in C++.


> If core emacs completion prompts were to select some metadata based
> approach to completion tables, the Semantic tag format is pretty simple,
> and could provide what is needed.

See, my Emacs/elisp knowledge is too scarce to have an opinion on this :)

I once tried diving into semantic to accomplish what I wanted, together with
completion-ui, but back then it was just my first or second week with emacs and
lisp, so I think accomplishing that really was out of my possibilities :)

In any case, I don't know if there is general interest on the features I have
described. If not, I might eventually try to implement that, but would need some
advice on which are the "standard" mechanisms from which to build up so that the
solution is near to the least convoluted outcome.

But this will be "some day"... my first contact is still to be finished with a
small improvement to VC to allow filling the commit report with "C-x 4 a"
without having to fill up a changelog file; which I've been postponing for too
long.

Thanks,
    Lluis

-- 
 "And it's much the same thing with knowledge, for whenever you learn
 something new, the whole world becomes that much richer."
 -- The Princess of Pure Reason, as told by Norton Juster in The Phantom
 Tollbooth

Re: Completions in Semantic

[Chong Yidong]

"Eric M. Ludlam" <eric@siege-engine.com> writes:

> On Sun, 2009-10-18 at 23:50 -0400, Eric M. Ludlam wrote:
>> For your particular example in xdisp.c, it seems the contents of
>> xdisp.c fails to parse properly.  It seems to be related to the P_
>> macro, and one of the uses of it involving a line-end in the middle.
>> I've narrowed it down and will see if I can fix it.
>
> To reply to myself, I checked changes into the CEDET repository for the
> C preprocessor handling to fix the bug found here in xdisp.
>
> By way of side effect, the completion problem originally posed is also
> fixed, as there is no "window" symbol in the file anymore.

I've merged your changes into the Emacs repository:

  * cedet/semantic/bovine/c.el (semantic-c-debug-mode-init)
  (semantic-c-debug-mode-init-pch): New functions.
  (semantic-c-debug-mode-init-last-mode): New var.
  (semantic-c-parse-lexical-token): Use them.

  * cedet/semantic/lex-spp.el (semantic-lex-spp-anlyzer-do-replace):
  When extracting the argument list, limit only by point-max.

I assume it's the lex-spp change you're referring to.  However, the bug
has not gone away for me: `C-c , j window [TAB]' in xdisp.c still brings
up "window" as the only completion.

Re: Completions in Semantic

[Eric M. Ludlam]

On Wed, 2009-10-21 at 10:07 -0400, Chong Yidong wrote:
> "Eric M. Ludlam" <eric@siege-engine.com> writes:
> 
> > On Sun, 2009-10-18 at 23:50 -0400, Eric M. Ludlam wrote:
> >> For your particular example in xdisp.c, it seems the contents of
> >> xdisp.c fails to parse properly.  It seems to be related to the P_
> >> macro, and one of the uses of it involving a line-end in the middle.
> >> I've narrowed it down and will see if I can fix it.
> >
> > To reply to myself, I checked changes into the CEDET repository for the
> > C preprocessor handling to fix the bug found here in xdisp.
> >
> > By way of side effect, the completion problem originally posed is also
> > fixed, as there is no "window" symbol in the file anymore.
> 
> I've merged your changes into the Emacs repository:
> 
>   * cedet/semantic/bovine/c.el (semantic-c-debug-mode-init)
>   (semantic-c-debug-mode-init-pch): New functions.
>   (semantic-c-debug-mode-init-last-mode): New var.
>   (semantic-c-parse-lexical-token): Use them.
> 
>   * cedet/semantic/lex-spp.el (semantic-lex-spp-anlyzer-do-replace):
>   When extracting the argument list, limit only by point-max.
> 
> I assume it's the lex-spp change you're referring to.  However, the bug

Correct.  The other is related to a different bug with a users
c-mode-hook that might throw an error, and how  to detect and tell them
about it.

> has not gone away for me: `C-c , j window [TAB]' in xdisp.c still brings
> up "window" as the only completion.

That is probably because the old tagging information is still being
used.  If you run

M-x semantic-clear-toplevel-cache RET

in that buffer, it will flush old parsing data, or you can go into your
~/.semanticdb/ directory, and delete the cache file for emacs/src before
starting emacs, which might be better since multiple files  there could
be affected.

Eric

Re: Completions in Semantic

[Eric M. Ludlam]

On Wed, 2009-10-21 at 15:28 +0200, Lluis wrote:
> > Just as an FYI about the semantic tag info (which you seem to be
> > alluding to), is that it provides all that info, though not
> necessarily
> > from the completion engine via a UI.
> 
> Well, it's been a long time since I played with the completion
> interfaces
> provided in CEDET, but I already knew all that information was
> available
> somewhere in semantic :)
> 
> I just didn't know documentation was also available as one of the
> tag's
> attributes, which reminds me that:
>     1) One-liner doxygen description shows up with the `*/' at the end
>     2) Descriptions of the form: /*< description */ 
>        are not assigned to the correct variable
>     3) One more nice feature would be to have the "show long
> documentation"
>        parse doxygen-specific commands such that parameters
> descriptions are
>        shown more clearly

The doc in Semantic tags is only for languages that have explicit doc,
like Emacs Lisp.  For other languages, when doc is requested of a tag,
there is some code that goes and looks for comments.  It doesn't follow
doxygen style rules though.

On the flip side, the srecode tool has a documentation feature that can
do doxygen style comment reading/modification, going so far as to take
boring comments and convert them to be doxygen compatible.  You can use
it with srecode-document-insert-comment, or C-c / C if srecode minor
mode is on.  I think I only created templates for C/C++, and Java.

I suppose it would make sense to merge the comment reader pieces back
together.

For folks hacking Emacs C code, if there is a comment style/behavior
that is repetitive/special in some way, it should be possible to create
srecode template specialization for it.  I've been interested in having
EDE provided project template specializations, but haven't had a use
case since I only use the GNUish or doxygen ones which are language
specific.

> I also tried with `completion-ui', `auto-complete' and `company-mode',
> which are
> the ones I've found that provide completion UIs through overlays (hope
> I'm using
> the correct term), which is the form i feel more comfortable with.
> 
> But as far as I've been able to see, none of these three provide the
> features
> I've mentioned, although they do provide semantic-based backends for
> completion.

Those tools are pretty generic completion engines, and thus don't have
any special features that use the extra tagging info.  That I know of
anyway.

Eric

Re: Completions in Semantic

[Stefan Monnier]

> It is in the cedet/semantic/complete.el, and the function is
> semantic-complete-post-command-hook for the inline completion exit
> conditions.

:-(  It's not much prettier than the hacks I've come up with so far.


        Stefan

Re: Completions in Semantic

[Stefan Monnier]

> I don't know about the features of current code, but I miss some
> features on some (if not all) the completion interfaces I've tested.
> So here's a list of (my) desired features (all of which should be
> configurable by the user):

Sounds interesting, but I must be missing some context because there's
a lot of references I don't understand.  Could you expand on them?

> - symbol name
>   Of course, this is what already provides every completion UI

No idea what you're talking about here.

> - extra information
>   This can be filled ub with symbol "metadata".  Which metadata
>   appears on the completion UI should be configurable by the user:
>     - return type
>     - arguments (type and/or name)
>     - definition location (aka file)
>     - short documentation
>     - long documentation
>   All this metadata should be located anywhere around the symbol name
>   and/or the minibuffer (e.g., I think company-mode shows short
>   documentation on minibuffer, until user presses F1, when full
>   documentation is shown.  Some metadata might be shown in the
>   minibuffer after completion selection (e.g., prototype).

Oh... wait, are you talking specifically about completion in code
buffers, so "symbol name" above referred to the ability to complete an
identifier?
As for this "extra info", I see what you mean, but usually completion
involves several potential candidates, so listing them all plus all
their info would take way too much space in general (if not, then
something like completion-annotate-function should work).  So usually
this extra info is provided outside of the completion functionality
(e.g. via eldoc-mode or something similar).  But, yes, I'd like to
extend the *Completions* buffer so that you could ask for more info
(either on all entries, or just on one at a time).

> - result narrowing
>   A-la company-mode.

No idea to what this is referring.

> - argument placeholders So that argument type and/or name is shown as
> placeholders, such that the user simply TABs (or whatever) to fill-in
> the blanks.

Idem.  Unless you mean something like skeletons/templates, but then
I fail to see the connection with completion.


        Stefan

Re: Completions in Semantic

[Chong Yidong]

"Eric M. Ludlam" <eric@siege-engine.com> writes:

>> has not gone away for me: `C-c , j window [TAB]' in xdisp.c still brings
>> up "window" as the only completion.
>
> That is probably because the old tagging information is still being
> used.  If you run
>
> M-x semantic-clear-toplevel-cache RET
>
> in that buffer, it will flush old parsing data, or you can go into your
> ~/.semanticdb/ directory, and delete the cache file for emacs/src before
> starting emacs, which might be better since multiple files  there could
> be affected.

No, I have no semanticdb directory.  C-c,j[TAB] brings up "window" as
the only completion.  Repeating [TAB] twice brings up the other
completions.  But the expected behavior should be to show the other
completions on the first [TAB].

Re: Completions in Semantic

[Eric M. Ludlam]

On Thu, 2009-10-22 at 21:01 -0400, Chong Yidong wrote:
> "Eric M. Ludlam" <eric@siege-engine.com> writes:
> 
> >> has not gone away for me: `C-c , j window [TAB]' in xdisp.c still brings
> >> up "window" as the only completion.
> >
> > That is probably because the old tagging information is still being
> > used.  If you run
> >
> > M-x semantic-clear-toplevel-cache RET
> >
> > in that buffer, it will flush old parsing data, or you can go into your
> > ~/.semanticdb/ directory, and delete the cache file for emacs/src before
> > starting emacs, which might be better since multiple files  there could
> > be affected.
> 
> No, I have no semanticdb directory.  C-c,j[TAB] brings up "window" as
> the only completion.  Repeating [TAB] twice brings up the other
> completions.  But the expected behavior should be to show the other
> completions on the first [TAB].

Well, I know the behavior you are talking about, but I don't get that
specific example anymore after the lexical analyzer change you merged
recently, so my assumption is some sort of cached old parser data.

The below patch removes that specific behavior from that prompt.  I'm
assuming it is in there for some reason for one of the various
completion or display types.  I'll try to figure out what it is, and
perhaps it will be unique to some kind of completion display.

The patch is against my repository.  It's in
semantic-complete-do-completion, 3rd conditional.

Eric


*** semantic-complete.el.~1.63.~	2009-10-22 21:21:12.000000000 -0400
--- semantic-complete.el	2009-10-22 21:21:21.000000000 -0400
***************
*** 528,535 ****
  	(semantic-displayor-set-completions
  	 displayor
  	 (or
! 	  (and (not (eq na 'displayend))
! 	       (semantic-collector-current-exact-match collector))
  	  (semantic-collector-all-completions collector contents))
  	 contents)
  	;; Ask the displayor to display them.
--- 528,535 ----
  	(semantic-displayor-set-completions
  	 displayor
  	 (or
! 	  ;(and (not (eq na 'displayend))
! 	  ;     (semantic-collector-current-exact-match collector))
  	  (semantic-collector-all-completions collector contents))
  	 contents)
  	;; Ask the displayor to display them.

Re: Completions in Semantic

[Lluis]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=utf-8, Size: 3781 bytes --]

Sorry for the delay and the scarce response... too much work.


El Thu, Oct 22, 2009 at 04:12:09PM -0400, Stefan Monnier ens deleità amb les següents paraules:
> > I don't know about the features of current code, but I miss some
> > features on some (if not all) the completion interfaces I've tested.
> > So here's a list of (my) desired features (all of which should be
> > configurable by the user):
> 
> Sounds interesting, but I must be missing some context because there's
> a lot of references I don't understand.  Could you expand on them?

Sure.


> > - symbol name
> >   Of course, this is what already provides every completion UI
> 
> No idea what you're talking about here.
> 
> > - extra information
> >   This can be filled ub with symbol "metadata".  Which metadata
> >   appears on the completion UI should be configurable by the user:
> >     - return type
> >     - arguments (type and/or name)
> >     - definition location (aka file)
> >     - short documentation
> >     - long documentation
> >   All this metadata should be located anywhere around the symbol name
> >   and/or the minibuffer (e.g., I think company-mode shows short
> >   documentation on minibuffer, until user presses F1, when full
> >   documentation is shown.  Some metadata might be shown in the
> >   minibuffer after completion selection (e.g., prototype).
> 
> Oh... wait, are you talking specifically about completion in code
> buffers, so "symbol name" above referred to the ability to complete an
> identifier?

That's right.


> As for this "extra info", I see what you mean, but usually completion
> involves several potential candidates, so listing them all plus all
> their info would take way too much space in general (if not, then
> something like completion-annotate-function should work).

What I described is similar to `completion-annotate-function', but more
flexible (prefixed in the symbol/identifier, postfixed; some in overlay, some in
minibuffer, etc).


> So usually this extra info is provided outside of the completion functionality
> (e.g. via eldoc-mode or something similar).  But, yes, I'd like to extend the
> *Completions* buffer so that you could ask for more info (either on all
> entries, or just on one at a time).

The key here is that completions do not necessarily have to go (only) into the
*Completions* buffer.


> > - result narrowing
> >   A-la company-mode.
> 
> No idea to what this is referring.

Company-mode provides a binding that, given the current table of possible
completions, the user can narrow it through various mechanisms (e.g., regexps).


> > - argument placeholders So that argument type and/or name is shown as
> > placeholders, such that the user simply TABs (or whatever) to fill-in
> > the blanks.
> 
> Idem.  Unless you mean something like skeletons/templates, but then
> I fail to see the connection with completion.

Completing a symbol triggers an arbitrary function, which could, for example:
    - show symbol definition
    - show short and/or extended documentation for symbol
    - insert skeleton/template-like symbol definition (such that user simply
      tabs to sellect next argument placeholder, which initially contains
      argument name and/or type).

This could be achieved if completions where "complex objects", so each could
describe what/where to show on the specific current completion UI, a "hover"
callback (e.g. show short doc or symbol signature on minibuffer), and a
selection callback (e.g., same possibilities as hover, plus the skeleton thing).

Read you,
    Lluis

-- 
 "And it's much the same thing with knowledge, for whenever you learn
 something new, the whole world becomes that much richer."
 -- The Princess of Pure Reason, as told by Norton Juster in The Phantom
 Tollbooth

Re: Completions in Semantic

[Stefan Monnier]

>> As for this "extra info", I see what you mean, but usually completion
>> involves several potential candidates, so listing them all plus all
>> their info would take way too much space in general (if not, then
>> something like completion-annotate-function should work).

> What I described is similar to `completion-annotate-function', but
> more flexible (prefixed in the symbol/identifier, postfixed; some in
> overlay, some in minibuffer, etc).

Right, so the question is: how should the generic code make such
flexibility available (and conversely) how shojuld the client of the
completion code be able to exploit/provide such flexibility.

I'm very interested in adding such hooks into the completion code, but
I haven't come up with any convincing idea of what they should look like
(the closest I got is completion-annotate-function which is very
limited).  So suggestions are welcome (patches as well, of course, tho
it's not crucial).

>> So usually this extra info is provided outside of the completion
>> functionality (e.g. via eldoc-mode or something similar).  But, yes,
>> I'd like to extend the *Completions* buffer so that you could ask for
>> more info (either on all entries, or just on one at a time).
> The key here is that completions do not necessarily have to go (only)
> into the *Completions* buffer.

Of course.  company-mode is an obvious example.

>> > - result narrowing
>> >   A-la company-mode.
>> No idea to what this is referring.
> Company-mode provides a binding that, given the current table of
> possible completions, the user can narrow it through various
> mechanisms (e.g., regexps).

Would that be like IDO's C-SPC (aka ido-restrict-to-matches)?
It shouldn't be too hard to add that to the default completion.
Maybe even without having to change minibuffer.el (i.e. there should
already be enough hooks for that).

>> > - argument placeholders So that argument type and/or name is shown as
>> > placeholders, such that the user simply TABs (or whatever) to fill-in
>> > the blanks.
>> 
>> Idem.  Unless you mean something like skeletons/templates, but then
>> I fail to see the connection with completion.

> Completing a symbol triggers an arbitrary function, which could, for example:
>     - show symbol definition
>     - show short and/or extended documentation for symbol
>     - insert skeleton/template-like symbol definition (such that user simply
>       tabs to sellect next argument placeholder, which initially contains
>       argument name and/or type).

That would be like the minibuffer-exit-hook, but for completion inside
a normal buffer, right?  That sounds like a good idea.  Currently,
figuring out when such "inline completion" ends is actually pretty
difficult, but it's something very much needed.


        Stefan

Re: Completions in Semantic

[Eric M. Ludlam]

On Tue, 2009-10-27 at 20:56 -0400, Stefan Monnier wrote:
> >> As for this "extra info", I see what you mean, but usually completion
> >> involves several potential candidates, so listing them all plus all
> >> their info would take way too much space in general (if not, then
> >> something like completion-annotate-function should work).
> 
> > What I described is similar to `completion-annotate-function', but
> > more flexible (prefixed in the symbol/identifier, postfixed; some in
> > overlay, some in minibuffer, etc).
> 
> Right, so the question is: how should the generic code make such
> flexibility available (and conversely) how shojuld the client of the
> completion code be able to exploit/provide such flexibility.
> 
> I'm very interested in adding such hooks into the completion code, but
> I haven't come up with any convincing idea of what they should look like
> (the closest I got is completion-annotate-function which is very
> limited).  So suggestions are welcome (patches as well, of course, tho
> it's not crucial).

Are you looking for examples of hooks that would be needed, or examples
of behaviors that need hooks to be designed?

Here are some thoughts:

* Annotate completions buffer output (as you describe)
- I used to add () to a fcn, or other little things but clicking on a
  completion name would then insert too much, so I removed the feature
  from my version.

* Alternate completion list display mechanisms
- Inline completions like using a popup window.  I used a tooltip, but 
  it doesn't doesn't look as nice and isn't interactive

* Focus concept
- Once a completions list is short enough, TABs that don't complete can
  instead 'focus' on a completion.  Each new TAB moves the focus.

  The focus could do almost anything depending on what is being
completed.  For tags, it will flash the location of the tag in a buffer.
It could show doc, or any other differentiating data that might be
available.  ie - in Info nodes it could show the first paragraph.  For
files it could show the first 500 bytes or some file status, for a
buffer it could flip to it temporarily.  I expect most completable
things could do something special.

  For inline completion, the focus can be used to ghost in the text that
would be inserted if one selected that focus.  Pressing RET when
something is focused makes that item become the select entry w/out
having to type it.

Eric

Re: Completions in Semantic

[Stefan Monnier]

> Are you looking for examples of hooks that would be needed, or examples
> of behaviors that need hooks to be designed?

Examples of hooks that could make such things possible.

> * Annotate completions buffer output (as you describe)
> - I used to add () to a fcn, or other little things but clicking on a
>   completion name would then insert too much, so I removed the feature
>   from my version.

completion-annotate-function should cover these needs

> * Focus concept
> - Once a completions list is short enough, TABs that don't complete can
>   instead 'focus' on a completion.  Each new TAB moves the focus.

That sounds like the "cycling" behavior of some completion packages (or
like minibuffer-force-complete).  `pcomplete' for example has such
a feature, where if there are less than N completion candidates, TAB
cycles among them rather than completing the common prefix.

>   The focus could do almost anything depending on what is being
> completed.  For tags, it will flash the location of the tag in a buffer.
> It could show doc, or any other differentiating data that might be
> available.  ie - in Info nodes it could show the first paragraph.  For
> files it could show the first 500 bytes or some file status, for a
> buffer it could flip to it temporarily.  I expect most completable
> things could do something special.

I guess that could be done by adding a hook called when inserting
a "complete" completion.


        Stefan

Re: Completions in Semantic

[Lluis]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=utf-8, Size: 4185 bytes --]

El Tue, Oct 27, 2009 at 11:23:44PM -0400, Stefan Monnier ens deleità amb les següents paraules:
> > Are you looking for examples of hooks that would be needed, or examples
> > of behaviors that need hooks to be designed?
> 
> Examples of hooks that could make such things possible.

Ok, so here's a high-level shot on a possible completion infrastructure.

Note that much of what I describe is oriented towards programming.

* Completion candidates
** Information
   Contains whatever information the system could gather about the symbol. When
   programming with semantic activated, this could be the semantic tag structure.
** Actions
   Various actions that can be applied on a candidate.
   This is basically the interface to view the candidate information.
*** get-*
    Recollect the interesting bits from information, so that the user can easily
    access it independently of the "information backend" (e.g., semantic tags).
**** get-name
     Candidate name
**** get-location
     Where the candidate has been defined
**** get-return
     Return type of the candidate (only meaningful on programming candidates)
**** get-arguments
     Arguments of the candidate (idem)
**** get-short-help
     Short version of candidate help
**** get-long-help
     Long version of candidate help
*** show
    Shows an arbitrary composition of get-*.
    E.g., (concat (get-signature candidate) "\n" (get-long-help candidate))
    Where that sould be shown, must be user selectable (see below
*** insert-*
    Insert "something" at point (cannot think of use-cases where insertion takes
    place in other places other than point).
**** insert-name
     Insert the good old completion candidate name.
**** insert-template
     Insert a "template" based on the candidate information.
     E.g.,
        open(<<const char *pathname>>, <<int flags>>)
     The <<...>> parts are yasnippet-like placeholders for the user to fill.
     This could be complicated even more, like:
       * insert a placeholder for the result if compeltion is at first non-blank
         column.
       * insert ";" at end if completion is at first non-blank column.
       * show parameter/return help (e.g., when help follows doxygen style) when
         moving between placeholders.
* UI
** Types
   The completion-ui package has implementations for some (if not all) of them.
   The UI 
*** buffer
    Is the current *Completions* buffer.
*** minibuffer
    The completion system used by, e.g., predictive-mode.
*** overlay
    That's the one I prefer, see company-mode as an example.
*** others
    There are many others, like menus, etc.
** Operations
   All action triggered by operations should be possibly be configured on a
   per-information-backend basis (e.g., completions from yasnippet do not have
   any meaningful `get-return').
*** start
    Start the completion process. E.g.:
        (show current-ui get-return get-name get-arguments)
    would show the full signature of a function.
*** focus
    Let's the user "highlight" an entry of the possible candidates. It's the
    good old next/prev in an overlay UI.
    Focusing an element can trigger other actions. This will mainly be
        (show <where> ...)
    The <where> could be any of minibuffer, another buffer, tooltip, etc.
*** select
    Called when the user selects a candidate. This can trigger insert and/or
    show actions.
    In fact, triggering only show actions could be used to reuse this very same
    infrastructure for other purposes other than completion, like a
    user-configurable tool to show help on symbol at point.
*** completion-map
    Extra actions the user can trigger through keyboard when the completion UI
    is "active".
    E.g., i can focus to an interesting symbol and trigger
        (show some-buffer get-long-help)

This probably means a whole lot of coding and I'm neither knowledgeable of
current state of functionalities nor proficient in elisp.

Just a thought,
    Lluis

-- 
 "And it's much the same thing with knowledge, for whenever you learn
 something new, the whole world becomes that much richer."
 -- The Princess of Pure Reason, as told by Norton Juster in The Phantom
 Tollbooth

Re: Completions in Semantic

[Toby Cubitt]

On Tue, Oct 20, 2009 at 02:07:07AM +0200, Lennart Borgman wrote:
> On Tue, Oct 20, 2009 at 12:17 AM, Eric M. Ludlam <eric@siege-engine.com> wrote:
> >
> > The semantic complete code for inline stuff uses the same completion and
> > display engines as the minibuffer or inline prompts, so you can mix and
> > match the pieces.  Hopefully an official API can do something similar,
> > so you can complete in the minibuffer or inline with the same functions.
>
> I think the authors of company-mode and completion-UI (Nikolaj and
> Toby) are interested in this too. And Drew, of course.

I'm a bit late to the party, but I think completion-UI already provides
hooks for much of what's being discussed (e.g. annotating completions
with additional information, a generic framework for defining new
completion UIs and completion sources...).

I had a discussion a couple of years ago (more?) with RMS about
submitting completion-UI for inclusion in Emacs, which led me to rework
parts of it based on his suggestions. By the end of the discussion, it
was in a state that RMS was happy with.

However, it was in the middle of the version 22 feature freeze, and I
promised to submit it for possible inclusion after the end of the
freeze. There were three reasons I never got around to it: 1) I forgot,
2) I still wasn't quite happy with the code, and 3) I hadn't documented
it (other than docstrings). Since then, I've at least fixed 2) so that
the code is much cleaner and clearer. I still haven't fixed 3).

But maybe it's time I put completion-UI up for inclusion, and if it's
favourably received it might give me the incentive to document it
properly...

If you're interested, the code can be found at
www.dr-qubit.org/emacs.php, and the direct download link is:

http://dr-qubit.org/predictive/completion-ui-0.11.7.tar.gz

Toby

--
Dr T. S. Cubitt
Quantum Information Theory group
Department of Mathematics
University of Bristol
United Kingdom

email: tsc25@cantab.net
web: www.dr-qubit.org

Re: Completions in Semantic

[Lennart Borgman]

On Fri, Oct 30, 2009 at 10:17 PM, Toby Cubitt
<toby-predictive-dated-1257369490.0ef45f@dr-qubit.org> wrote:
>
> I had a discussion a couple of years ago (more?) with RMS about
> submitting completion-UI for inclusion in Emacs, which led me to rework
> parts of it based on his suggestions. By the end of the discussion, it
> was in a state that RMS was happy with.
>
> However, it was in the middle of the version 22 feature freeze, and I
> promised to submit it for possible inclusion after the end of the
> freeze. There were three reasons I never got around to it: 1) I forgot,
> 2) I still wasn't quite happy with the code, and 3) I hadn't documented
> it (other than docstrings). Since then, I've at least fixed 2) so that
> the code is much cleaner and clearer. I still haven't fixed 3).
>
> But maybe it's time I put completion-UI up for inclusion, and if it's
> favourably received it might give me the incentive to document it
> properly...


As you know I think a framework like completion-UI och company-mode
should be included in Emacs. However I suggest that you and Nikolaj
try to merge your frameworks first if it is possible.

Re: Completions in Semantic

[Stefan Monnier]

>> > Are you looking for examples of hooks that would be needed, or examples
>> > of behaviors that need hooks to be designed?
>> Examples of hooks that could make such things possible.

> Ok, so here's a high-level shot on a possible completion infrastructure.
> Note that much of what I describe is oriented towards programming.

Yes, this discussion is difficult because my main interest is in reusing
the current standard completion code (used for most minibuffer
completions) for most/all other "in-buffer" completions.  OTOH you're
only interested in completion of variables and method/function
identifiers, which is itself just a special case of
"in-buffer" completion.

> * Completion candidates
> ** Information Contains whatever information the system could gather
>    about the symbol. When programming with semantic activated, this could
>    be the semantic tag structure.

If you don't know what that info is going to be, then it is useless in
the generic API since anybody using the generic API will expect to work
for any kind of candidate and so can't use this value since it has no
idea what it might be.

> ** Actions
>    Various actions that can be applied on a candidate.
>    This is basically the interface to view the candidate information.

In the generic completion code, we currently have 4 actions:
test-completion, try-completion, all-completions, and the new
completion-boundaries (which in the case of function-name completion is
always trivial).

For reasons of backward compatibility, it's difficult to add new actions
in the generic code (the new completion-boundaries is the first time we
add an action since ... before 1993), but one could easily imagine
adding actions for specific kinds of completions, which could then only
be used for completion tables that support those extra actions.

> **** get-return
>      Return type of the candidate (only meaningful on programming candidates)
> **** get-arguments
>      Arguments of the candidate (idem)

This is very specialized.  It doesn't even apply to all programming
languages (e.g. Prolog).  Also even in functional programming languages,
we usually keep the two together, i.e. we'd rather see that + has type
"int * int -> int" than that it has args of types "int, int" and return
type "int", and there's a good reason for that: sometimes it's not so
clear where's the boundary between args and return values: the function
"map" (aka "mapcar" in Elisp) has typically type "(a -> b) -> List a ->
List b" (with an implicit "∀ a,b" around it), but that can be read as
"map take a function argument of type (a -> b) and returns a new
function of type (List a -> List b)", or it can be read equivalently as
"map takes a function argument of type (a -> b) and a list of type List
a and returns a list of type List b".

All this to say: if you conflate "get-return" and "get-arguments"
together, you'll get a system that will fit many more languages.

> **** get-short-help
>      Short version of candidate help
> **** get-long-help
>      Long version of candidate help
> *** show
>     Shows an arbitrary composition of get-*.
>     E.g., (concat (get-signature candidate) "\n" (get-long-help candidate))
>     Where that sould be shown, must be user selectable (see below

Isn't this "show" redundant with get-short-help or get-long-help?

> *** insert-*
>     Insert "something" at point (cannot think of use-cases where insertion takes
>     place in other places other than point).

I think side-effects have no business being here.  Let the client code
do the insertion, and limit the candidate object to returning the
info necessary to figure out what to insert.

> * UI
> ** Types
>    The completion-ui package has implementations for some (if not all) of them.
>    The UI
> *** buffer
>     Is the current *Completions* buffer.
> *** minibuffer
>     The completion system used by, e.g., predictive-mode.
> *** overlay
>     That's the one I prefer, see company-mode as an example.
> *** others
>     There are many others, like menus, etc.

Or icomplete-mode, yes.  These seem to only affect
minibuffer-completion-help (the standard function that displays the list
of possible completions, usually in *Completions*).  When I use
completion, I generally try to avoid getting to that point, tho.

This points to a more important aspect: when is this list displayed,
when is it updated, and when is it brought down.  For each kind of
completion-list, those choices are not always the same
(e.g. icomplete-mode brings is up right away and updates it after each
keystroke, whereas what you call "buffer" only brings it up on a few
specific operations and only updates it on those same operations).
This behavior-variation depending on the display-style is something that
makes it trickier to write the generic code.

> *** start
>     Start the completion process. E.g.:
>         (show current-ui get-return get-name get-arguments)
>     would show the full signature of a function.

I don't understand it: when I start completion, I don't have any
function chosen yet (that's the whole reason for me to try to complete
it, usually), so there's no signature to show yet.

> *** focus
>     Let's the user "highlight" an entry of the possible candidates. It's the
>     good old next/prev in an overlay UI.
>     Focusing an element can trigger other actions. This will mainly be
>         (show <where> ...)
>     The <where> could be any of minibuffer, another buffer, tooltip, etc.

Oh, so "show" is not a method that returns a string, it's a method that
has some side-effect that hopefully "shows" the candidate?  That makes
more sense now.

> *** completion-map
>     Extra actions the user can trigger through keyboard when the completion UI
>     is "active".
>     E.g., i can focus to an interesting symbol and trigger
>         (show some-buffer get-long-help)

That might be tricky to support in a generic way, since depending on the
UI, it's not necessarily clear where the completion-map should be active
and with which other keybindings it might collide.


        Stefan

Re: Completions in Semantic

[Lluís]

On Sat, 31 Oct 2009 16:18:17 -0400, Stefan Monnier wrote:
[...]
> > * Completion candidates
> > ** Information Contains whatever information the system could gather
> >    about the symbol. When programming with semantic activated, this could
> >    be the semantic tag structure.
> 
> If you don't know what that info is going to be, then it is useless in
> the generic API since anybody using the generic API will expect to work
> for any kind of candidate and so can't use this value since it has no
> idea what it might be.

I was thinking on a scenario where each completion source provides its own
implementation of "Actions", and "Information" is thus not publicly visible
but for the "Actions".


> > ** Actions
> >    Various actions that can be applied on a candidate.
> >    This is basically the interface to view the candidate information.
> 
> In the generic completion code, we currently have 4 actions:
> test-completion, try-completion, all-completions, and the new
> completion-boundaries (which in the case of function-name completion is
> always trivial).
> 
> For reasons of backward compatibility, it's difficult to add new actions
> in the generic code (the new completion-boundaries is the first time we
> add an action since ... before 1993), but one could easily imagine
> adding actions for specific kinds of completions, which could then only
> be used for completion tables that support those extra actions.

So, any higher-level code could attach new actions as it sees fits. That's
just what I was thinking about.


> > **** get-return
> >      Return type of the candidate (only meaningful on programming candidates)
> > **** get-arguments
> >      Arguments of the candidate (idem)
> 
[...]
> All this to say: if you conflate "get-return" and "get-arguments"
> together, you'll get a system that will fit many more languages.

You're right, this might be reduced to something like:
       - get-signature
       - get-definition

The difference is that the first provides just a type signature, while the
latter provides the whole definition (including the symbol name). But still
it should be selectable whether you want argument names or just types.

On the other hand, if the implementation of all the actions is provided by
each completion source, the "get-*" actions simply are internal
source-specific helpers, and thus out of the question.


[...]
> > *** insert-*
> >     Insert "something" at point (cannot think of use-cases where insertion takes
> >     place in other places other than point).
> 
> I think side-effects have no business being here.  Let the client code
> do the insertion, and limit the candidate object to returning the
> info necessary to figure out what to insert.

I completely agree, but like show, this was more on the conceptual level
where you can show and/or insert a specific candidate.


> > * UI
> > ** Types
> >    The completion-ui package has implementations for some (if not all) of them.
> >    The UI
> > *** buffer
> >     Is the current *Completions* buffer.
> > *** minibuffer
> >     The completion system used by, e.g., predictive-mode.
> > *** overlay
> >     That's the one I prefer, see company-mode as an example.
> > *** others
> >     There are many others, like menus, etc.
> 
> Or icomplete-mode, yes.  These seem to only affect
> minibuffer-completion-help (the standard function that displays the list
> of possible completions, usually in *Completions*).  When I use
> completion, I generally try to avoid getting to that point, tho.

Sorry, I don't understand what you mean here.


> This points to a more important aspect: when is this list displayed,
> when is it updated, and when is it brought down.  For each kind of
> completion-list, those choices are not always the same
> (e.g. icomplete-mode brings is up right away and updates it after each
> keystroke, whereas what you call "buffer" only brings it up on a few
> specific operations and only updates it on those same operations).
> This behavior-variation depending on the display-style is something that
> makes it trickier to write the generic code.

Ok, I might say something totally stupid, but you just need to map the UI
actions to specific keys, which can be selected by the user. And on the
case of "continuous completion" the specific UI catches all keystrokes to
issue the refresh action (the obvious option is to remap all keys, but I
don't know how is this really implemented).


> > *** start
> >     Start the completion process. E.g.:
> >         (show current-ui get-return get-name get-arguments)
> >     would show the full signature of a function.
> 
> I don't understand it: when I start completion, I don't have any
> function chosen yet (that's the whole reason for me to try to complete
> it, usually), so there's no signature to show yet.

Sorry, this was not detailed enough. What I meant is that you somehow tell
the framework that each candidate must be shown as:
    return name arguments


> > *** focus
> >     Let's the user "highlight" an entry of the possible candidates. It's the
> >     good old next/prev in an overlay UI.
> >     Focusing an element can trigger other actions. This will mainly be
> >         (show <where> ...)
> >     The <where> could be any of minibuffer, another buffer, tooltip, etc.
> 
> Oh, so "show" is not a method that returns a string, it's a method that
> has some side-effect that hopefully "shows" the candidate?  That makes
> more sense now.

Yes, that's the idea I tried to express.


> > *** completion-map
> >     Extra actions the user can trigger through keyboard when the completion UI
> >     is "active".
> >     E.g., i can focus to an interesting symbol and trigger
> >         (show some-buffer get-long-help)
> 
> That might be tricky to support in a generic way, since depending on the
> UI, it's not necessarily clear where the completion-map should be active
> and with which other keybindings it might collide.

As I see it, there are two types of interfaces:
- in-buffer: this includes overlays, menus, etc.
- in another buffer: the minibuffer is just a specific case of using
  another buffer for the completion UI.

In the first case, once started, the user is interacting directly with the
UI: moving focus, triggering "extra actions", etc. This UI can, in turn,
modify the underlying buffer (partial completions, the user who keeps
writing, etc.).

On the second case, the user can indirectly interact with the UI through
its current buffer (e.g., accept completion, stop completion, partial
completion, etc.), or either switch to the completion buffer and trigger
the rest of actions (e.g., move to a specific candidate and trigger the
"show extended help" action).

What I see more problematic is that some actions simply have no meaning in
some specific candidate sources. E.g., yasnippet has no use for a "extended
help" action. But at least, we can always ignore the action if the
source/backend of a specific candidate provides no implementation for it.


Read you,
     Lluis

--
 "And it's much the same thing with knowledge, for whenever you learn
 something new, the whole world becomes that much richer."
 -- The Princess of Pure Reason, as told by Norton Juster in The Phantom
 Tollbooth

Re: Completions in Semantic

[Stefan Monnier]

>> Or icomplete-mode, yes.  These seem to only affect
>> minibuffer-completion-help (the standard function that displays the list
>> of possible completions, usually in *Completions*).  When I use
>> completion, I generally try to avoid getting to that point, tho.
> Sorry, I don't understand what you mean here.

Just that you focus on "list the completion options", whereas usually
I'm more interested in "insert the common part of all the possible
completions that you can get from the text I already typed".

But to some extent it's true that the completion listing is the trickier
point left over now (Emacs-23's new completion code was specifically
designed to better address the "do the completion" part, and it's one of
the reason why I want to get most other completion code to use that new
code, so as to benefit from things like partial-completion).

> Ok, I might say something totally stupid, but you just need to map the UI
> actions to specific keys, which can be selected by the user. And on the

It's not stupid, but the question is how/where to put the hooks that
allow the particular UI backend to setup such bindings.  For minibuffer
completion, we have minibuffer-local-completion-map, but for in-buffer
completion we have no such thing (standard in-buffer completion like
lisp-complete-symbol doesn't use a keymap).


        Stefan

Re: Completions in Semantic

[Eric M. Ludlam]

On Mon, 2009-11-02 at 01:12 -0500, Stefan Monnier wrote:

> > Ok, I might say something totally stupid, but you just need to map the UI
> > actions to specific keys, which can be selected by the user. And on the
> 
> It's not stupid, but the question is how/where to put the hooks that
> allow the particular UI backend to setup such bindings.  For minibuffer
> completion, we have minibuffer-local-completion-map, but for in-buffer
> completion we have no such thing (standard in-buffer completion like
> lisp-complete-symbol doesn't use a keymap).

This is where an OO like system (built on something like EIEIO with data
and methods) would be ideal.  If someone needs to specialize completion,
they just subclass, and override the bit they need.

Instead of calling a built-in core completion method, users would
subclass the core completion object, and then override the bits they
need to make a new completion engine.

This would be easier to design now than divining all future hook names
and finding a way to make a function call with an extensible number of
hook/action types.

Eric