Is intellisense features integration in Emacs technically possible?

Is intellisense features integration in Emacs technically possible?

[Jorge Araya Navarro]

[-- Attachment #1: Type: text/plain, Size: 1100 bytes --]

Hello!

I'm a Python and C++ programmer using Emacs 24.3! :D
I wasn't complete happy with Emacs incapacity to have intellisense for C
++ (and Python have some sort of it through Jedi), so I decide to use
CEDET but for me was a painful job to make it work on Emacs, because
those elisp functions from CEDET that actually were void-functions, no
intellisense after trying to fix those errors, etc. (I cannot make it
works actually). It is amazing for me that something so useful as CEDET
is proportionally hard to make it work for the majority (I guess) of
Emasc users.

Anyway, I was wondering if Emacs will grow on functionality not through
elisp libraries but native code (I remember that Gimp have this issue
too), because intellisense for a wide range of programming languages
(with simple hooks and configuration to use as back-end for other
libraries like auto-complete) is a very good idea to integrate into
Emacs code. If it were technically possible, of course!
-- 
Pax et bonum.
Jorge Araya Navarro.
Diseñador publicitario, programador Python/C++ y colaborador en Parabola
GNU/Linux-libre.

[-- Attachment #2: Type: text/html, Size: 1458 bytes --]

Re: Is intellisense features integration in Emacs technically possible?

[Tom]

Jorge Araya Navarro <elcorreo <at> deshackra.com> writes:

> Anyway, I was wondering if Emacs will grow on functionality not through 
elisp libraries but native code (I remember that Gimp have this issue too), 
because intellisense for a wide range of programming languages (with simple 
hooks and configuration to use as back-end for other libraries like auto-
complete) is a very good idea to integrate into Emacs code. If it were 
technically possible, of course!

Creating mature and extensive intellisense features is hard work,
it requires lots of man hours and while it's technically possible to
implement it in emacs it is unlikely to happen, because emacs lacks
the developing resources.

A better way is to build on the hard work of other and interface
emacs with an external tool.

Here are solutions for C++ and you can find more with Google:

https://github.com/Sarcasm/irony-mode
http://root42.blogspot.hu/2012/07/nice-c-autocomplete-configuration-for.html


Similarly, for Java Eclim is a good solution, because it can
instantly provide features which are not available from other
emacs completion packages (e.g. code refactoring, suggestions
for code fixes, etc.)

It makes more sense to improve these interfacing solutions, because
implementing code sense which provide the same level of support
like Eclipse will never happen with an emacs native solutions
(lack of resources, Eclipse has many thousand developer hours
in its code).

Re: Is intellisense features integration in Emacs technically possible?

[Eli Zaretskii]

> From: Tom <adatgyujto@gmail.com>
> Date: Tue, 21 Jan 2014 18:59:10 +0000 (UTC)
> 
> Creating mature and extensive intellisense features is hard work,
> it requires lots of man hours and while it's technically possible to
> implement it in emacs it is unlikely to happen, because emacs lacks
> the developing resources.

According to this logic something like the rewrite of the display
engine that happened between Emacs 20 and Emacs 21, or bidirectional
editing support for Emacs 24 would never have happened.  But it did.
Each one of these took many man-months of work.

Look at the amount of changes that get committed every day to the
Emacs repository, and try to estimate the effort that goes into that.
Sometimes I wish I had such resources at my disposal on my daytime
job.

I think the shortage is not in development resources, but in motivated
individuals who'd sit down and do the job, and lobby others to come on
board and help.  Volunteers are welcome.

> A better way is to build on the hard work of other and interface
> emacs with an external tool.

Personally, I think implementing such features via external programs
is a terrible design.  It will never be smooth and responsive enough,
and on top of that you'd need to track development of those other
tools.  And what if they become abandoned some day?

Re: Is intellisense features integration in Emacs technically possible?

[Tom]

Eli Zaretskii <eliz <at> gnu.org> writes:

> 
> According to this logic something like the rewrite of the display
> engine that happened between Emacs 20 and Emacs 21, or bidirectional
> editing support for Emacs 24 would never have happened.  But it did.
> Each one of these took many man-months of work.

Yes, but I don't think their complexity is comparable to, for example,
the Java support of Eclipse which has been continously developed
for many years by a bunch of people. Repeating this effort is
no small feat. 

> Look at the amount of changes that get committed every day to the
> Emacs repository, and try to estimate the effort that goes into that.
> Sometimes I wish I had such resources at my disposal on my daytime
> job.

Yet, Emacs cannot provide the same level of language support like
other tools for Java and C++, so it is apparent the language support
part has not enough resources.

> I think the shortage is not in development resources, but in motivated
> individuals who'd sit down and do the job, and lobby others to come on
> board and help.  Volunteers are welcome.

Motivation can have multiple forms. For example, my idea was financial
motivation:

http://lists.gnu.org/archive/html/emacs-devel/2012-04/msg00476.html

The idea is to make it possible for people to sponsor specific 
features and there is enough bounty then someone will come and
do it. If people don't want to work on them in their free time,
then users can create a money pool (everyone giving a small
amount) and if enough users donates money then someone can work
on it full time until the feature is implemented.

> Personally, I think implementing such features via external programs
> is a terrible design.  It will never be smooth and responsive enough,
> and on top of that you'd need to track development of those other
> tools.  

I agree a native solution would be better, but for Java Eclim provides 
these features right now, while a native solution (if it happens at all)
will provide them next year, or a year after that?

http://www.skybert.net/emacs/java/

> And what if they become abandoned some day?

These interface packages should not be complicated. They
just talk to the server and then present the data to an emacs 
frontend (like autocomplete). So if a tool is abandoned then
there is an other tool instead, and only this interface
layer needs to be reimplemented which shouldn't be a lot 
of work.

Re: Is intellisense features integration in Emacs technically possible?

[Eli Zaretskii]

> From: Tom <adatgyujto@gmail.com>
> Date: Tue, 21 Jan 2014 19:58:21 +0000 (UTC)
> 
> Eli Zaretskii <eliz <at> gnu.org> writes:
> 
> > 
> > According to this logic something like the rewrite of the display
> > engine that happened between Emacs 20 and Emacs 21, or bidirectional
> > editing support for Emacs 24 would never have happened.  But it did.
> > Each one of these took many man-months of work.
> 
> Yes, but I don't think their complexity is comparable to, for example,
> the Java support of Eclipse which has been continously developed
> for many years by a bunch of people. Repeating this effort is
> no small feat. 

The Emacs display engine is many tens of thousands of C code plus many
thousands of Lisp.  I'm quite sure rewriting all that is more than
would be needed for intellisense support.

> > Look at the amount of changes that get committed every day to the
> > Emacs repository, and try to estimate the effort that goes into that.
> > Sometimes I wish I had such resources at my disposal on my daytime
> > job.
> 
> Yet, Emacs cannot provide the same level of language support like
> other tools for Java and C++, so it is apparent the language support
> part has not enough resources.

Like I said: lack of motivation, not lack of resources.

Re: Is intellisense features integration in Emacs technically possible?

[Óscar Fuentes]

Eli Zaretskii <eliz@gnu.org> writes:

> The Emacs display engine is many tens of thousands of C code plus many
> thousands of Lisp.

AFAIU this is largely a consequence of the development baggage of Emacs.
That is, the complexity of the current code base is far greater than the
complexity of its purpose.

> I'm quite sure rewriting all that is more than would be needed for
> intellisense support.

Writing a C++ parser and semantic analyzer from scratch requires several
man-years of work for world-class compiler writers. And with every new
C++ standard that comes out the effort increases significantly.

[snip]

Re: Is intellisense features integration in Emacs technically possible?

[David Kastrup]

Óscar Fuentes <ofv@wanadoo.es> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>> The Emacs display engine is many tens of thousands of C code plus many
>> thousands of Lisp.
>
> AFAIU this is largely a consequence of the development baggage of Emacs.
> That is, the complexity of the current code base is far greater than the
> complexity of its purpose.

If I understand correctly, It has been more or less rewritten from
scratch for Emacs 21.  That's not all that long in terms of the total
development history.

-- 
David Kastrup

Re: Is intellisense features integration in Emacs technically possible?

[Stephen J. Turnbull]

I apologize for responding to DAK's post but I lost Óscar's.

 > Óscar Fuentes <ofv@wanadoo.es> writes:
 > 
 > > Eli Zaretskii <eliz@gnu.org> writes:
 > >
 > >> The Emacs display engine is many tens of thousands of C code plus many
 > >> thousands of Lisp.
 > >
 > > AFAIU this is largely a consequence of the development baggage of Emacs.
 > > That is, the complexity of the current code base is far greater than the
 > > complexity of its purpose.

It could surely be a lot smaller if it were restricted to a single
multiplatform toolkit such as wxWindows.  However, users seem to
prefer platform-specific code, leading to a lot of duplication.  It's
not clear to me that's bad, given that both user preference and
developer skills are often very platform-specific.

By comparison, it's hard to say exactly (depends on what you mean by
"display"), but XEmacs's display engine is about 3.5KLOC of C code, of
which less than 1.5KLOC are in the platform-independent parts.

Re: Is intellisense features integration in Emacs technically possible?

[David Kastrup]

"Stephen J. Turnbull" <stephen@xemacs.org> writes:

> I apologize for responding to DAK's post but I lost Óscar's.
>
>  > Óscar Fuentes <ofv@wanadoo.es> writes:
>  > 
>  > > Eli Zaretskii <eliz@gnu.org> writes:
>  > >
>  > >> The Emacs display engine is many tens of thousands of C code plus many
>  > >> thousands of Lisp.
>  > >
>  > > AFAIU this is largely a consequence of the development baggage of Emacs.
>  > > That is, the complexity of the current code base is far greater than the
>  > > complexity of its purpose.
>
> It could surely be a lot smaller if it were restricted to a single
> multiplatform toolkit such as wxWindows.  However, users seem to
> prefer platform-specific code, leading to a lot of duplication.  It's
> not clear to me that's bad, given that both user preference and
> developer skills are often very platform-specific.
>
> By comparison, it's hard to say exactly (depends on what you mean by
> "display"), but XEmacs's display engine is about 3.5KLOC of C code, of
> which less than 1.5KLOC are in the platform-independent parts.

IIUC correctly, XEmacs' display engine factors quite more code and
behavior into a platform independent part of the code base.  From an
engineering and maintenance perspective that seems like a sane thing to
do and certainly factored in significantly in the headstart XEmacs had
over Emacs regarding the support of graphical interfaces.

In the long run, it may have made it harder for XEmacs to keep up with
the changing evolution of desktop environment looks, never mind how
little this may have actually to do with engineering or usability.
XEmacs looks a lot like XEmacs on every platform, or at least it did so
when I looked at it the last time, admittedly considerable time ago.
And that look makes it quite more apparent that XEmacs was competing
with the likes of the Athena widget set and Motif than one can see with
Emacs.

I am actually grateful that I can compile my Emacs with
--without-toolkit-scroll-bars (why is that only a compile-time option?)
and get the Lucid scrollbars for my otherwise GTK+ Emacs since, ugly
looks aside, the semantics of the old Athena scrollbar design are vastly
superior over that of the prettier GTK+ scrollbars, and the GTK+
maintainers are usually quite opposed to provide configurability.

But as the user interface wars we had in Emacs alone over the issue of
whether the scrollbar should be to the left or the right side of the
window show, for a lot of people blending into the respective
environment feels very important.

-- 
David Kastrup

Re: Is intellisense features integration in Emacs technically possible?

[Stephen J. Turnbull]

David Kastrup writes:

 > In the long run, [XEmacs-style factoring of (re-)display
 > functionality] may have made it harder for XEmacs to keep up with
 > the changing evolution of desktop environment looks,

Unlikely.  XEmacs developers just emphasized application functionality
over desktop integration (except with X, where ICCCM conformance was
an early goal -- of course platforms like Windows and Mac OS X make it
easy to conform at that level, whereas in practice ICCCM conformance
is impossible).

True, the native widget support was poorly done, which made everything
except Xt/Athena look like crap when compared with implementing
directly as done in Emacs.  The widgets look like a 14-year-old boy at
a sock hop, never quite able to fit in with the cool kids.  But that's
because they guy who implemented native widget support thought like a
Windows programmer, and didn't really like the idea of delegating to
widgets.  So his code tries to control everything from the Emacs
window level down, in C.  Yuck.  Not to mention hard to modify at all,
let alone beautify.

OTOH, Bill Perry's original work on GTK+ support had all the cool stuff
-- tear off menus, Glade and GNOME integration, and an FFI for anything
that came along later.  That all bitrotted because BeOpen went under
so financial support for GTK+ did too, and the people who ported to GTK+
v2 again just wanted the toplevel effect of the GTK look, and so
didn't do a great job of bringing the cool stuff along with them.

N.B. While obviously I have strong opinions on how these things
*should* have been done in retrospect, it's not clear to me what was
the right thing to do at the time.

The main point here is that I think what we would need to improve
platform integration is *more* refactoring, not less.

 > XEmacs looks a lot like XEmacs on every platform, or at least it
 > did so when I looked at it the last time, admittedly considerable
 > time ago.

So does Emacs.  On the Mac as supplied by MacPorts it looks a lot like
a late '90s X application. :-)  And spews ugly GTK QA warnings like
crazy -- uh, excuse me, like all the other GTK apps.

 > I am actually grateful that I can compile my Emacs with
 > --without-toolkit-scroll-bars (why is that only a compile-time
 > option?)

Because nobody really took the Lucid Widget promise of true toolkit
independence seriously, and that's probably because they never really
delivered on it.  It's not really possible to plug and play different
toolkits at runtime, although it *is* possible to mix and match at
compile time.  So you'd have to rewrite lwlib for that kind of feature.

 > But as the user interface wars we had in Emacs alone over the issue
 > of whether the scrollbar should be to the left or the right side of
 > the window show, for a lot of people blending into the respective
 > environment feels very important.

Apparently so.  But few of them were XEmacs developers, so that has
never been emphasized in XEmacs.

Re: Is intellisense features integration in Emacs technically possible?

[David Kastrup]

"Stephen J. Turnbull" <stephen@xemacs.org> writes:

> N.B. While obviously I have strong opinions on how these things
> *should* have been done in retrospect, it's not clear to me what was
> the right thing to do at the time.

Oh, hindsight is clearly 100/100.  Whatever else the great Emacs schism
might have caused, it has given me two histories to extrapolate from and
pontificate about.  I may be overgeneralizing a lot, but then two data
points are a dream constellation for curve fitting.

While two data points do not seem like exactly much in the vast space of
the Emacsen universe, this adds a whole new dimension over one data
point.

And things like Joe, Microemacs, Hemlock are just too distant viewpoints
for facilitating depth perception and for being considered part of a
quackworthy continuity.

-- 
David Kastrup

Re: Is intellisense features integration in Emacs technically possible?

[Stefan Monnier]

> --without-toolkit-scroll-bars (why is that only a compile-time option?)

Because nobody went through the trouble to make it a run-time option.
There's no technical difficulty, AFAIR.  And I'd welcome a patch for that.


        Stefan

Re: Is intellisense features integration in Emacs technically possible?

[Eli Zaretskii]

> From: "Stephen J. Turnbull" <stephen@xemacs.org>
> Date: Wed, 22 Jan 2014 16:26:49 +0900
> 
> By comparison, it's hard to say exactly (depends on what you mean by
> "display")

I meant the core device-independent portions of it, which in Emacs are
xdisp.c, dispnew.c, dispextern.h, composite.c, and bidi.c --
everything required for basic layout of text and redisplay.  Together
they weigh in at about 45KLOC.  This still leaves out other parts,
like fringe.c, image.c, xfaces.c, the font stuff, the menu support,
and the device dependent code (*term.c, *fns.c, etc.).  If I add
everything together, I get about 135KLOC in the current development
code (up from about 93KLOC in Emacs 21).

> but XEmacs's display engine is about 3.5KLOC of C code, of
> which less than 1.5KLOC are in the platform-independent parts.

Not sure how you get these numbers.  Just redisplay.c, redisplay.h,
and redisplay-out.c are about 13KLOC.  Maybe our concept of what
constitutes the display engine are very different, or maybe I don't
know how to count lines.

As a reference point, the total number of LOC in both projects is
almost identical: around 370K.  Emacs has display features that XEmacs
doesn't (like bidi), so I would expect the code to be smaller, but
certainly not by an order of magnitude.

Again, this doesn't seem to be relevant at all to the issue at hand,
which is whether introducing intellisense for select languages is or
isn't practically possible in Emacs development.  I just brought the 2
examples of features that required a comparable, if not greater,
effort, and were implemented juts recently, because there was someone
who picked up the gauntlet.

Re: Is intellisense features integration in Emacs technically possible?

[Stephen J. Turnbull]

Eli Zaretskii writes:

 > > From: "Stephen J. Turnbull" <stephen@xemacs.org>
 > > Date: Wed, 22 Jan 2014 16:26:49 +0900
 > > 
 > > By comparison, it's hard to say exactly (depends on what you mean by
 > > "display")
 > 
 > I meant the core device-independent portions of it, which in Emacs are
 > xdisp.c, dispnew.c, dispextern.h, composite.c, and bidi.c --
 > everything required for basic layout of text and redisplay.  Together
 > they weigh in at about 45KLOC.  This still leaves out other parts,
 > like fringe.c, image.c, xfaces.c, the font stuff, the menu support,
 > and the device dependent code (*term.c, *fns.c, etc.).  If I add
 > everything together, I get about 135KLOC in the current development
 > code (up from about 93KLOC in Emacs 21).
 > 
 > > but XEmacs's display engine is about 3.5KLOC of C code, of
 > > which less than 1.5KLOC are in the platform-independent parts.
 > 
 > Not sure how you get these numbers.  Just redisplay.c, redisplay.h,
 > and redisplay-out.c are about 13KLOC.  Maybe our concept of what
 > constitutes the display engine are very different, or maybe I don't
 > know how to count lines.

No, you're right.  It's me that has lived in Japan too long and
started counting by 10,000s instead of 1,000s.

So the core is about 13KLOC (redisplay.c, redisplay.h, and
redisplay-output.c).  By contrast, if you take out composite.c and
bidi.c (features XEmacs doesn't have), xdisp.c, dispnew.c, and
dispextern.h are about 40KLOC, 3X as much.

I'm sure by now Emacs has additional features that XEmacs doesn't, but
even so that doesn't account for factor of 3.  The point is that
XEmacs code is very differently factored.  Whether overall that's good
or bad is in the eye of the beholder, but for two beholders (Bill
Perry who did the GTK+ port and helped produce a prototype Qt port)
and Andrew Choi (who did "Carbon" ports for both Emacs and XEmacs)
praised XEmacs for the ease with which they were able to support new
GUI platforms.

 > Again, this doesn't seem to be relevant at all to the issue at
 > hand

Not as you put it.  However, David made a claim that highly factored
design could make development slower, and I wanted to put that claim
to rest because it's *easy* to factor intellisense given Emacs's
current architecture, and I want to argue that people shouldn't argue
about how hard it is to do comprehensively.  They should just jump in
and do the language and features *they* need and let others do what
they want to do.  (Which I believe is your point, too!)

 > which is whether introducing intellisense for select languages is or
 > isn't practically possible in Emacs development.

Re: Is intellisense features integration in Emacs technically possible?

[David Kastrup]

"Stephen J. Turnbull" <stephen@xemacs.org> writes:

> Not as you put it.  However, David made a claim that highly factored
> design could make development slower, and I wanted to put that claim
> to rest

I think the usual way to get rid of strawmen is to burn them.

What I was talking about is that basically factoring the whole
look-and-feel into a unified display engine in XEmacs might have sped up
its initial spread across platforms but has had drawbacks in its
long-term popularity.  With one result being that it has not rallied
enough interest behind it to cover all basic desktop environments well
in _spite_ of the porting possibly being easier.

It may well be easier to provide the display engine support for
"Intellisense" on four different major platforms on XEmacs than on
Emacs.

> because it's *easy* to factor intellisense given Emacs's current
> architecture, and I want to argue that people shouldn't argue about
> how hard it is to do comprehensively.  They should just jump in and do
> the language and features *they* need and let others do what they want
> to do.  (Which I believe is your point, too!)

Emacs 21 started with a number of features only being available
under X11.  That's a viable starting point, and with its current
developers I consider it likely that the support for an interesting
feature would not remain stuck in that state for as long the overall
graphic display support did in the past.

At any rate, I doubt that the main stumbling block is the display
support.

>  > which is whether introducing intellisense for select languages is
>  > or isn't practically possible in Emacs development.

I lost context here, but in typical programming styles, R2L is mostly
relevant for comments.

-- 
David Kastrup

Re: Is intellisense features integration in Emacs technically possible?

[Eli Zaretskii]

> From: "Stephen J. Turnbull" <stephen@xemacs.org>
> Cc: emacs-devel@gnu.org
> Date: Thu, 23 Jan 2014 17:13:50 +0900
> 
> So the core is about 13KLOC (redisplay.c, redisplay.h, and
> redisplay-output.c).  By contrast, if you take out composite.c and
> bidi.c (features XEmacs doesn't have), xdisp.c, dispnew.c, and
> dispextern.h are about 40KLOC, 3X as much.

Not sure it's worth to continue this line counting, but just for the
record: AFAIU, you should add more XEmacs files to the "core", because
xdisp.c covers some areas which in XEmacs are on separate files.  For
example, display of the tool bar, echo-area messages, frame titles,
horizontal scrolling, glyphs, and mouse highlight are all implemented
on xdisp.c.  Also, about 1500 lines in xdisp.c are due to bidi
support, so they should be subtracted from the Emacs numbers.

I actually believe that the only meaningful comparison is if we add
everything that is somehow related to display, because different
architectures and modularization, as well as non-overlapping features,
make it very hard to compare some arbitrary parts of the display
engine.

Re: Is intellisense features integration in Emacs technically possible?

[Stephen J. Turnbull]

Eli Zaretskii writes:
 > > From: "Stephen J. Turnbull" <stephen@xemacs.org>
 > > Cc: emacs-devel@gnu.org
 > > Date: Thu, 23 Jan 2014 17:13:50 +0900
 > > 
 > > So the core is about 13KLOC (redisplay.c, redisplay.h, and
 > > redisplay-output.c).  By contrast, if you take out composite.c and
 > > bidi.c (features XEmacs doesn't have), xdisp.c, dispnew.c, and
 > > dispextern.h are about 40KLOC, 3X as much.
 > 
 > Not sure it's worth to continue this line counting, but just for the
 > record: AFAIU, you should add more XEmacs files to the "core", because
 > xdisp.c covers some areas which in XEmacs are on separate files.

No, that's backwards to my point.  I don't care if XEmacs has less
code to support redisplay than Emacs does (modulo the "less
functionality" aspect, I'd be disappointed if the comparison isn't
approximately proportional to functionality).  My point is precisely
that Emacs's "core" is much bigger than XEmacs's because it includes
additional functionality.

Of course in C (and most languages) division into files is rather
arbitrary.  Perhaps the size of xdisp.c *doesn't* reflect
substantially greater coupling among the functions it defines.  But
division into more files does help the developer to *reduce* coupling
(by use of "static" to remind her that this variable shouldn't be used
outside of this file, for example).  And reduced coupling often leads
to easier porting.

Re: Is intellisense features integration in Emacs technically possible?

[Eli Zaretskii]

> From: "Stephen J. Turnbull" <stephen@xemacs.org>
> Date: Fri, 24 Jan 2014 11:57:42 +0900
> Cc: emacs-devel@gnu.org
> 
>  > Not sure it's worth to continue this line counting, but just for the
>  > record: AFAIU, you should add more XEmacs files to the "core", because
>  > xdisp.c covers some areas which in XEmacs are on separate files.
> 
> No, that's backwards to my point.  I don't care if XEmacs has less
> code to support redisplay than Emacs does (modulo the "less
> functionality" aspect, I'd be disappointed if the comparison isn't
> approximately proportional to functionality).  My point is precisely
> that Emacs's "core" is much bigger than XEmacs's because it includes
> additional functionality.

Perhaps you could explain why this matters in the context of this
discussion.  As long as the file is clearly separated into sections
that deal with different groups of functionality, the size of the file
should matter only to the compiler, no?

> division into more files does help the developer to *reduce* coupling
> (by use of "static" to remind her that this variable shouldn't be used
> outside of this file, for example).

It's actually the other way around: the larger the file, the more
functions can be static, even if they are interfaces between
conceptually separate modules in the program's architectural design,
and will have to have external linkage if a large file is subdivided
into several smaller ones.

Re: Is intellisense features integration in Emacs technically possible?

[Rüdiger Sonderfeld]

On Wednesday 22 January 2014 05:36:40 Óscar Fuentes wrote:
> > I'm quite sure rewriting all that is more than would be needed for
> > intellisense support.
> 
> Writing a C++ parser and semantic analyzer from scratch requires several
> man-years of work for world-class compiler writers. And with every new
> C++ standard that comes out the effort increases significantly.

Why would we have to write a C++ parser and semantic analyzer?  There is one 
in CEDET, there is one in GCC, there is one in libclang, and there is one in 
the KDevelop libraries.

Regards,
Rüdiger

Re: Is intellisense features integration in Emacs technically possible?

[Óscar Fuentes]

Rüdiger Sonderfeld <ruediger@c-plusplus.de> writes:

> Why would we have to write a C++ parser and semantic analyzer?

If I understood Eli correctly he is advocating the independence of Emacs
for that purposes.

> There is one in CEDET, there is one in GCC, there is one in libclang,
> and there is one in the KDevelop libraries.

CEDET is in line with Eli's views, while the other options we should
avoid (*) (Except, perhaps, GCC.)

OTOH CEDET's internal capabilities for C++ are very limited last time I
checked. It is more about "C with classes" with some crude hacks for
templates. It does not real C++ semantic analysis.

* Again, unless I misunderstood him.

Re: Is intellisense features integration in Emacs technically possible?

[Eli Zaretskii]

> From: Óscar Fuentes <ofv@wanadoo.es>
> Date: Wed, 22 Jan 2014 12:53:08 +0100
> 
> Rüdiger Sonderfeld <ruediger@c-plusplus.de> writes:
> 
> > Why would we have to write a C++ parser and semantic analyzer?
> 
> If I understood Eli correctly he is advocating the independence of Emacs
> for that purposes.

Indeed; but so does Rüdiger.

> > There is one in CEDET, there is one in GCC, there is one in libclang,
> > and there is one in the KDevelop libraries.
> 
> CEDET is in line with Eli's views, while the other options we should
> avoid (*) (Except, perhaps, GCC.)
> 
> OTOH CEDET's internal capabilities for C++ are very limited last time I
> checked. It is more about "C with classes" with some crude hacks for
> templates. It does not real C++ semantic analysis.
> 
> * Again, unless I misunderstood him.

You didn't.

My point is that we already have a lot of infrastructure in CEDET that
can (and IMO should) be leveraged towards these goals.

Re: Is intellisense features integration in Emacs technically possible?

[Stefan Monnier]

>> > Why would we have to write a C++ parser and semantic analyzer?
>> If I understood Eli correctly he is advocating the independence of Emacs
>> for that purposes.

FWIW, for the languages I care about in my work (things like SML,
Haskell, OCaml, Coq, Agda, Twelf), parsing the syntax is not sufficient
to provide good support: we also want type information; and inferring
the type is very much non-trivial (it's a significant part of the whole
implementation in some cases).

So it is important in the longer run to provide a good way to use
external tools to get that info, since re-implementing it in Elisp
is illusory.


        Stefan

Re: Is intellisense features integration in Emacs technically possible?

[David Engster]

Stefan Monnier writes:
>>> > Why would we have to write a C++ parser and semantic analyzer?
>>> If I understood Eli correctly he is advocating the independence of Emacs
>>> for that purposes.
>
> FWIW, for the languages I care about in my work (things like SML,
> Haskell, OCaml, Coq, Agda, Twelf), parsing the syntax is not sufficient
> to provide good support: we also want type information; and inferring
> the type is very much non-trivial (it's a significant part of the whole
> implementation in some cases).

Yes. Another example are highly dynamic languages like Javascript.

> So it is important in the longer run to provide a good way to use
> external tools to get that info, since re-implementing it in Elisp
> is illusory.

In CEDET, the idea is to convert the output of external tools to the
tag structures Semantic uses. This is usually not difficult to do, and
there are many packages like this already, like for cscope, ectags,
clang, and even Firefox through MozRepl (for completing Javascript).

-David

Re: Is intellisense features integration in Emacs technically possible?

[David Engster]

Óscar Fuentes writes:
> OTOH CEDET's internal capabilities for C++ are very limited last time I
> checked. It is more about "C with classes" with some crude hacks for
> templates. It does not real C++ semantic analysis.

The template support is very limited, yes. I have a big patch in line
which will improve that. Otherwise, I'm not sure what you mean with
"real semantic analysis". Our parser is a tagging parser, meaning it
does not parse every line of code, but of course it is aware of much
more than, say, ctags.

-David

Re: Is intellisense features integration in Emacs technically possible?

[Eli Zaretskii]

> From: Óscar Fuentes <ofv@wanadoo.es>
> Date: Wed, 22 Jan 2014 05:36:40 +0100
> 
> Eli Zaretskii <address@hidden> writes:
> 
> > The Emacs display engine is many tens of thousands of C code plus many
> > thousands of Lisp.
> 
> AFAIU this is largely a consequence of the development baggage of Emacs.
> That is, the complexity of the current code base is far greater than the
> complexity of its purpose.

For this argument to be taken seriously, you'd have to point to
concrete chunks of code or design traits that constitute this
"baggage", and perhaps propose an alternative design that would be
free of that.

(I happen to disagree with your assessment, if it isn't clear.)

In any case, this has nothing to do with the issue at hand.  Even if
the entire effort invested in the rewrite of the display engine was a
waste, the point is that it could be, and have been, done.

Re: Is intellisense features integration in Emacs technically possible?

[David Engster]

Óscar Fuentes writes:
> Eli Zaretskii <eliz@gnu.org> writes:
>> I'm quite sure rewriting all that is more than would be needed for
>> intellisense support.
>
> Writing a C++ parser and semantic analyzer from scratch requires several
> man-years of work for world-class compiler writers.

You don't need a full parser. For providing completions, it is
sufficient to parse only a small subset of the code (declarations, most
importantly).

-David

Re: Is intellisense features integration in Emacs technically possible?

[Dmitry Gutov]

David Engster <deng@randomsample.de> writes:

>> Writing a C++ parser and semantic analyzer from scratch requires several
>> man-years of work for world-class compiler writers.
>
> You don't need a full parser. For providing completions, it is
> sufficient to parse only a small subset of the code (declarations, most
> importantly).

You may not have to parse every line of code, but you should be able to, no?

Otherwise, how would you know which type the variable at point has, or
the return type of the method it calls, etc? I believe you should have
to parse at least the body of the current function, from the beginning
to the current line (maybe farther).

Re: Is intellisense features integration in Emacs technically possible?

[David Engster]

Dmitry Gutov writes:
> David Engster <deng@randomsample.de> writes:
>
>>> Writing a C++ parser and semantic analyzer from scratch requires several
>>> man-years of work for world-class compiler writers.
>>
>> You don't need a full parser. For providing completions, it is
>> sufficient to parse only a small subset of the code (declarations, most
>> importantly).
>
> You may not have to parse every line of code, but you should be able to, no?
>
> Otherwise, how would you know which type the variable at point has,

By scanning the function's body for local variable declarations, and
simply ignore everything else. The C++ grammar simply has a bunch of
pretty generic expression rules without any actions, so we can skip over
the uninteresting stuff.

So yes, I wasn't specific enough: technically, we do parse function
bodies. But practically, we ignore most of it.

Of course there's a drawback that we don't fully parse the body - we
might get the local context wrong, for instance. But at least from my
experience it's working very well.

> or the return type of the method it calls

By parsing the declaration of the method.

-David

Re: Is intellisense features integration in Emacs technically possible?

[Óscar Fuentes]

David Engster <deng@randomsample.de> writes:

>> Writing a C++ parser and semantic analyzer from scratch requires several
>> man-years of work for world-class compiler writers.
>
> You don't need a full parser. For providing completions, it is
> sufficient to parse only a small subset of the code (declarations, most
> importantly).

Even for providing completions you need semantic analysis. The parser
turns to be insufficient on very simple cases:

auto foo = bar();
foo. ????

The need for semantic analysis is unavoidable even on the "old" C++ 98:

template <typename T> struct Foo {
  typedef T Type;
};

struct Bar {
  int something() {
    return 42;
  }
};

void some_func() {
  typename Foo<Bar>::Type t;
  t. ???
}

Re: Is intellisense features integration in Emacs technically possible?

[David Engster]

Óscar Fuentes writes:
> David Engster <deng@randomsample.de> writes:
>
>>> Writing a C++ parser and semantic analyzer from scratch requires several
>>> man-years of work for world-class compiler writers.
>>
>> You don't need a full parser. For providing completions, it is
>> sufficient to parse only a small subset of the code (declarations, most
>> importantly).
>
> Even for providing completions you need semantic analysis.

Of course you need that. Where did I say otherwise? I'm saying that when
you get parsing of declarations right, and have some good heuristics to
parse the local context, then this will suffice for most cases.

Parsing code like this also has some advantages, mostly that it is more
fault-tolerant. This is actually very important, since while you're
coding, your code will often be syntactically wrong.

> The parser turns to be insufficient on very simple cases:
>
> auto foo = bar();
> foo. ????

Well, type inference is hard. Currently, C++11 isn't on my agenda. But
I'm confident simple cases like the above will be manageable.

> The need for semantic analysis is unavoidable even on the "old" C++ 98:
>
> template <typename T> struct Foo {
>   typedef T Type;
> };
>
> struct Bar {
>   int something() {
>     return 42;
>   }
> };
>
> void some_func() {
>   typename Foo<Bar>::Type t;
>   t. ???
> }

This will work after I've applied my template patch.

-David

Re: Is intellisense features integration in Emacs technically possible?

[Andreas Röhler]

Am 21.01.2014 20:29, schrieb Eli Zaretskii:
>> From: Tom <adatgyujto@gmail.com>
>> Date: Tue, 21 Jan 2014 18:59:10 +0000 (UTC)
>>
>> Creating mature and extensive intellisense features is hard work,
>> it requires lots of man hours and while it's technically possible to
>> implement it in emacs it is unlikely to happen, because emacs lacks
>> the developing resources.
>
> According to this logic something like the rewrite of the display
> engine that happened between Emacs 20 and Emacs 21, or bidirectional
> editing support for Emacs 24 would never have happened.  But it did.
> Each one of these took many man-months of work.
>
> Look at the amount of changes that get committed every day to the
> Emacs repository, and try to estimate the effort that goes into that.
> Sometimes I wish I had such resources at my disposal on my daytime
> job.
>
> I think the shortage is not in development resources, but in motivated
> individuals who'd sit down and do the job, and lobby others to come on
> board and help.  Volunteers are welcome.
>

You are missing the point. Bidirection is at the core of any editor - thanks a lot BTW.
Intellisense features must come from accessing the programming languages itself.

Re: Is intellisense features integration in Emacs technically possible?

[Eli Zaretskii]

> Date: Tue, 21 Jan 2014 21:03:29 +0100
> From: Andreas Röhler <andreas.roehler@online.de>
> 
> Bidirection is at the core of any editor - thanks a lot BTW.
> Intellisense features must come from accessing the programming languages itself.

Both are extremely important for a modern programmer's editor.  So I
don't see the difference, really.

Re: Is intellisense features integration in Emacs technically possible?

[Stephen J. Turnbull]

Eli Zaretskii writes:
 > > Date: Tue, 21 Jan 2014 21:03:29 +0100
 > > From: Andreas Röhler <andreas.roehler@online.de>
 > > 
 > > Bidirection is at the core of any editor - thanks a lot BTW.
 > > Intellisense features must come from accessing the programming
 > > languages itself.
 > 
 > Both are extremely important for a modern programmer's editor.  So I
 > don't see the difference, really.

There's a big difference.  Specifically, intellisense in the abstract
is basically completion, the change in UI is a SMOP, and people have
shown a lot of interest in the UI.  Something will happen, although
I'm not sure how close it will be to Visual Studio.

What's left to compare is necessary skills.  Adding bidirection
requires intimate knowledge of both UAX#9 and the Emacs display
engine, and greatly benefits from a native knowledge of what "properly
formatted Hebrew" (or Arabic) looks like.  Not many folks with those
qualifications, or who want to acquire them.  It also may as well be
done all-at-once because UAX#9 exists -- we don't need to fool around
and figure out what is a good algorithm.  We already have a good one,
the only question is whether our implementation is correct.  So it's a
big job few can tackle.

Adding a new language to intellisense, OTOH, is something anybody who
uses the language and knows enough Elisp to write defuns in their
.emacs can help with.  Nor does it need to be done all-at-once, as
long as the basic interface makes it easy to say "shut up and let me
type" case by case (this is what annoys the hell out of me about
intellisense in Japan input methods -- they're pretty bad about
guessing what I'm trying to say, and they don't get out of the way
smoothly -- it's like when you meet somebody in a narrow passage and
you both move to the east, then to the west, then ...).

Adding a new algorithm (eg, based on a full-blown C++ parser) is
harder, but unnecessary to make progress.

Re: Is intellisense features integration in Emacs technically possible?

[Eli Zaretskii]

> From: "Stephen J. Turnbull" <stephen@xemacs.org>
> Cc: Andreas Röhler <andreas.roehler@online.de>,
>     emacs-devel@gnu.org
> Date: Wed, 22 Jan 2014 15:28:10 +0900
> 
>  > > Bidirection is at the core of any editor - thanks a lot BTW.
>  > > Intellisense features must come from accessing the programming
>  > > languages itself.
>  > 
>  > Both are extremely important for a modern programmer's editor.  So I
>  > don't see the difference, really.
> 
> There's a big difference.  Specifically, intellisense in the abstract
> is basically completion, the change in UI is a SMOP, and people have
> shown a lot of interest in the UI.  Something will happen, although
> I'm not sure how close it will be to Visual Studio.
> 
> What's left to compare is necessary skills.  Adding bidirection
> requires intimate knowledge of both UAX#9 and the Emacs display
> engine, and greatly benefits from a native knowledge of what "properly
> formatted Hebrew" (or Arabic) looks like.  Not many folks with those
> qualifications, or who want to acquire them.

With around 550 million people speaking R2L languages world wide, I
think "not many" is somewhat accurate.  According to this:

  http://java.dzone.com/articles/how-many-java-developers-are

there are between 6.8 and 10.7 million Java developers in the world,
out of 5 billion people living in developed regions, which means tens
of thousands of Java developers in R2L cultures.  All of them are
potential candidates to want this in Emacs (and that's for Java
alone).  I won't be surprised if the numbers for Python or Ruby or C++
are higher.

> It also may as well be done all-at-once because UAX#9 exists -- we
> don't need to fool around and figure out what is a good algorithm.
> We already have a good one, the only question is whether our
> implementation is correct.

Not true.  All the implementations of the UBA I know of are not good
for Emacs, because they are batch implementations: they reorder chunks
of text in one go.  By contrast, the Emacs display engine examines
buffer text one character at a time, and so needed a _sequential_
implementation of the UBA.  Moreover, the UBA itself is described in
the Unicode Standard assuming batch-style reordering, evidently
because it was written by people who either don't understand the
difference between requirements and implementation, or didn't bother
to formulate requirements, assuming that only batch-style
implementations will be needed anyway.

So the job actually constituted mentally reverse-engineering the UBA
to formulate the missing requirements, then implementing that.

And, to add insult to injury, Unicode 6.3 made 2 significant changes
in the UBA, which means Someone™ will now have to go back and extend
all that to support the new features.  Not really a once-and-for-all
job, I'd say.

> So it's a big job few can tackle.

Not sure how you took this leap of logic: if the algorithm is clear,
why shouldn't it be possible for more than "a few" to come up with a
suitable implementation?  And in fact, at the time the bidi support in
Emacs was actively discussed (I'm talking around 2001), there were at
least 3 candidate implementations proposed, and 2 of them were
actually prototyped in Emacs.  More than enough to start with.

And still, nothing happened for 10 more years.  So clearly, other
factors are at work that determine if and when some major feature is
implemented in Emacs.

> Adding a new language to intellisense, OTOH, is something anybody who
> uses the language and knows enough Elisp to write defuns in their
> .emacs can help with.  Nor does it need to be done all-at-once, as
> long as the basic interface makes it easy to say "shut up and let me
> type" case by case

I think you are arguing here that adding Intellisense is a smaller job
than the display rewrite or bidi -- in which case I'm in violent
agreement.

Re: Is intellisense features integration in Emacs technically possible?

[Stephen J. Turnbull]

Eli Zaretskii writes:

 > With around 550 million people speaking R2L languages world wide, I
 > think "not many" is somewhat accurate.  According to this:
 > 
 >   http://java.dzone.com/articles/how-many-java-developers-are
 > 
 > there are between 6.8 and 10.7 million Java developers in the world,
 > out of 5 billion people living in developed regions, which means tens
 > of thousands of Java developers in R2L cultures.  All of them are
 > potential candidates to want this in Emacs (and that's for Java
 > alone).  I won't be surprised if the numbers for Python or Ruby or C++
 > are higher.

Your math is correct, your statistics suck.  Given that developer
residence is *highly* biased toward living in the US, you can't just
multiply fraction of Java developers by fraction of R2L region
residents (and what does Java skill have to do with developing for
Emacs, may I ask?)

 > > It also may as well be done all-at-once because UAX#9 exists -- we
 > > don't need to fool around and figure out what is a good algorithm.
 > > We already have a good one, the only question is whether our
 > > implementation is correct.
 > 
 > Not true.  All the implementations of the UBA I know of are not good
 > for Emacs, because they are batch implementations:

True, but I didn't say there was a good *implementation*, I said there
was a good *algorithm*, which you yourself chose to implement.
Subject to this caveat:

 > So the job actually constituted mentally reverse-engineering the UBA
 > to formulate the missing requirements, then implementing that.

Sure, it ain't easy, but that was *my* point, no?  That's *why* you
need not only the *desire* to have it in Emacs and the *knowledge* of
bidi to recognize a bug when you see it, but you *also* need a fairly
high level of understanding of Emacs redisplay.  Nor was there really
a choice of algorithm was there?  You still promise to get the same
results, right?  And people would complain if you didn't, right?

 > And, to add insult to injury, Unicode 6.3 made 2 significant changes
 > in the UBA, which means Someone™ will now have to go back and extend
 > all that to support the new features.  Not really a once-and-for-all
 > job, I'd say.

I didn't say "once-and-for-all", and there's no way I would: I'm a guy
who can cite chapter and verse (ok, with the help of the IETF website)
of the differences among the past versions and future candidates for
STD 11 (ie, RFCs 632, 733, 822, 2822, and 5322) -- I know there's no
"once-and-for-all" in computing.

BTW, they slipped that one past me.  My condolences.  But it certainly
shows you're a promise-keeper that you even think about trying to keep
up with that moving target.

 > > So it's a big job few can tackle.
 > 
 > Not sure how you took this leap of logic: if the algorithm is clear,
 > why shouldn't it be possible for more than "a few" to come up with a
 > suitable implementation?

How many people are there with the energy, knowledge, and stubbornness
to reengineer a batch algorithm for use in Emacs redisplay?  I stand
by my statement.  This was not a job to be dismissed with "the rest is
a SMOP.  Why do I (who only dream of being able to do it) have to tell
you?

 > And still, nothing happened for 10 more years.  So clearly, other
 > factors are at work that determine if and when some major feature is
 > implemented in Emacs.

Not my point at all.

 > > Adding a new language to intellisense, OTOH, is something anybody who
 > > uses the language and knows enough Elisp to write defuns in their
 > > .emacs can help with.  Nor does it need to be done all-at-once, as
 > > long as the basic interface makes it easy to say "shut up and let me
 > > type" case by case
 > 
 > I think you are arguing here that adding Intellisense is a smaller job
 > than the display rewrite or bidi -- in which case I'm in violent
 > agreement.

With the emphasis on "violent".  Love you too, Eli!  :-)  Happy (and
productive!) New Year to you!  And don't spend it all on UBA updates!

Steve

Re: Is intellisense features integration in Emacs technically possible?

[Phillip Lord]

Eli Zaretskii <eliz@gnu.org> writes:
>> A better way is to build on the hard work of other and interface
>> emacs with an external tool.
>
> Personally, I think implementing such features via external programs
> is a terrible design.  It will never be smooth and responsive enough,
> and on top of that you'd need to track development of those other
> tools.  And what if they become abandoned some day?


I think that it depends on the language. Introspecting over, for
example, Java would require an awful of elisp, which would be difficult
to write. Getting Java to do this work is quite a lot less effort.
Hence, the JDEEs use of Java for this (via bsh). Likewise, Clojure and
Scala both of which use their own language to do much of the work. Or
for that matter, common lisp with slime/swank. Or even, for that matter,
English with aspell. I didn't have a problem with responsiveness with
any of these.



Phil

Re: Is intellisense features integration in Emacs technically possible?

[Jorgen Schaefer]

On Wed, 22 Jan 2014 17:29:15 +0000
phillip.lord@newcastle.ac.uk (Phillip Lord) wrote:

> Eli Zaretskii <eliz@gnu.org> writes:
> >> A better way is to build on the hard work of other and interface
> >> emacs with an external tool.
> >
> > Personally, I think implementing such features via external programs
> > is a terrible design.  It will never be smooth and responsive
> > enough, and on top of that you'd need to track development of those
> > other tools.  And what if they become abandoned some day?
> 
> I think that it depends on the language. Introspecting over, for
> example, Java would require an awful of elisp, which would be
> difficult to write. Getting Java to do this work is quite a lot less
> effort. Hence, the JDEEs use of Java for this (via bsh). Likewise,
> Clojure and Scala both of which use their own language to do much of
> the work. Or for that matter, common lisp with slime/swank. Or even,
> for that matter, English with aspell. I didn't have a problem with
> responsiveness with any of these.

Yeah, roughly that.

"External program" instead of "integrated" means very little these
days. I wrote one of the various Emacs modes that provide semantic
completion for Python, interfacing with two different Python libraries
which do the actual semantic analysis. In this case, Emacs communicates
with the external program via a specified protocol, which, from the
Emacs side, looks exactly like calling some kind of Emacs Lisp
function. (Abstraction is great.)

The only problem I have is that keeping state shared between the
processes is difficult. Right now, what I'm doing is to re-send the
current buffer to the other process, including the position of point,
for every completion request. This is highly inefficient, and makes it
difficult to create a tighter coupling for better completion or
analysis from the Emacs side. To improve on that, there would need to
be a way of sharing the current contents of a buffer with a subprocess
without writing it to a file.

But this is an optimization problem, not a capability problem. The
current approach I use is "fast enough", so fast actually that I
haven't even implemented a speed-up idea of using temporary files
instead of sending the buffer contents as encoded strings to the other
process.

Considering there already are so many different modes that provide
semantic completion for Emacs, the main obstacle for IntelliSense (as
far as I understand it) *on the Emacs side* is actually not that big.
It's mainly a common interface for such external programs so that we
can add more languages more easily. The current effort of unifying the
completion interface as well as supporting company mode as a front-end
is going a great deal forward in that.

The biggest problems are outside of Emacs. Good libraries that provide
intelligent completion and code introspection are rare. I know of three
for Python, one of which is not maintained anymore, one was mostly-dead
for a few years, and all of which have different deficiencies. They
also have trouble keeping up with the development of the language. I
expect similar problems for other languages. Reimplementing these
libraries in Emacs Lisp will just cause more problems keeping them
up-to-date.

Though if someone wants to do that, do not let that stop you. It's most
certainly not "impossible".

Regards,
Jorgen

Re: Is intellisense features integration in Emacs technically possible?

[Andreas Röhler]

Am 22.01.2014 19:49, schrieb Jorgen Schaefer:
> On Wed, 22 Jan 2014 17:29:15 +0000
> phillip.lord@newcastle.ac.uk (Phillip Lord) wrote:
>
>> Eli Zaretskii <eliz@gnu.org> writes:
>>>> A better way is to build on the hard work of other and interface
>>>> emacs with an external tool.
>>>
>>> Personally, I think implementing such features via external programs
>>> is a terrible design.  It will never be smooth and responsive
>>> enough, and on top of that you'd need to track development of those
>>> other tools.  And what if they become abandoned some day?
>>
>> I think that it depends on the language. Introspecting over, for
>> example, Java would require an awful of elisp, which would be
>> difficult to write. Getting Java to do this work is quite a lot less
>> effort. Hence, the JDEEs use of Java for this (via bsh). Likewise,
>> Clojure and Scala both of which use their own language to do much of
>> the work. Or for that matter, common lisp with slime/swank. Or even,
>> for that matter, English with aspell. I didn't have a problem with
>> responsiveness with any of these.
>

[ ... ]
>
> The biggest problems are outside of Emacs. Good libraries that provide
> intelligent completion and code introspection are rare. I know of three
> for Python, one of which is not maintained anymore, one was mostly-dead
> for a few years, and all of which have different deficiencies. They
> also have trouble keeping up with the development of the language. I
> expect similar problems for other languages. Reimplementing these
> libraries in Emacs Lisp will just cause more problems keeping them
> up-to-date.
>

Good point(s).
An aspect not seeing mentioned so far: Python for example will cancel all service when encountering a syntax-error.
A case where a decent support from Emacs Lips side will be helpful.
Seems wise to implement some basic stuff in Emacs Lisp, while accessing the languages resources if available.

Re: Is intellisense features integration in Emacs technically possible?

[Jorgen Schaefer]

On Thu, 23 Jan 2014 10:00:53 +0100
Andreas Röhler <andreas.roehler@online.de> wrote:

> An aspect not seeing mentioned so far: Python for example will cancel
> all service when encountering a syntax-error. A case where a decent
> support from Emacs Lips side will be helpful. Seems wise to implement
> some basic stuff in Emacs Lisp, while accessing the languages
> resources if available.

I'm not sure I understand what you mean. Both Rope as well as Jedi (the
two Python libraries for completion and code introspection I mentioned)
handle syntactically incorrect input quite fine, without any special
treatment from the Emacs side. What do you mean with "will cancel all
service"?

Regards,
Jorgen

Re: Is intellisense features integration in Emacs technically possible?

[Phillip Lord]

Jorgen Schaefer <forcer@forcix.cx> writes:
>> I think that it depends on the language. Introspecting over, for
>> example, Java would require an awful of elisp, which would be
>> difficult to write. Getting Java to do this work is quite a lot less
>> effort. Hence, the JDEEs use of Java for this (via bsh). Likewise,
>> Clojure and Scala both of which use their own language to do much of
>> the work. Or for that matter, common lisp with slime/swank. Or even,
>> for that matter, English with aspell. I didn't have a problem with
>> responsiveness with any of these.
>
> "External program" instead of "integrated" means very little these
> days. I wrote one of the various Emacs modes that provide semantic
> completion for Python, interfacing with two different Python libraries
> which do the actual semantic analysis. In this case, Emacs communicates
> with the external program via a specified protocol, which, from the
> Emacs side, looks exactly like calling some kind of Emacs Lisp
> function. (Abstraction is great.)

This also has the considerable advantage that it makes it possible to
physically separate the two; consider running ESS on your desktop and
the R process somewhere else.


> The only problem I have is that keeping state shared between the
> processes is difficult. Right now, what I'm doing is to re-send the
> current buffer to the other process, including the position of point,
> for every completion request. This is highly inefficient, and makes it
> difficult to create a tighter coupling for better completion or
> analysis from the Emacs side. To improve on that, there would need to
> be a way of sharing the current contents of a buffer with a subprocess
> without writing it to a file.

JDEE for java had that problem. Some operations required that files be
saved first or, even worse, compiled first.

Languages like clojure tend not to cause this problem, since their unit
of compilation is smaller (give or take, it's a function), so you just
send a string across the socket.

> It's mainly a common interface for such external programs so that we
> can add more languages more easily. The current effort of unifying the
> completion interface as well as supporting company mode as a front-end
> is going a great deal forward in that.

I'm not convinced that a single interface would work; again, using
clojure as example, this has moved away from a single interface in Emacs
(i.e. slime/swank) and toward a single interface for Clojure (so that
the Clojure side offers a single server, for different editors).

Having said that, there are definately utility functions that could
help. One problem with this strategy wrt Emacs is that it's single
threaded, but utilities for dealing with this such as generating
callbacks would help. No doubt closures will help here.


> The biggest problems are outside of Emacs. Good libraries that provide
> intelligent completion and code introspection are rare. 
> I know of three for Python, one of which is not maintained anymore,
> one was mostly-dead for a few years, and all of which have different
> deficiencies. They also have trouble keeping up with the development
> of the language. I expect similar problems for other languages.
> Reimplementing these libraries in Emacs Lisp will just cause more
> problems keeping them up-to-date.


I'm sure this is true. I don't know of any for Java.

Phil

Re: Is intellisense features integration in Emacs technically possible?

[Stefan Monnier]

>> It's mainly a common interface for such external programs so that we
>> can add more languages more easily. The current effort of unifying the
>> completion interface as well as supporting company mode as a front-end
>> is going a great deal forward in that.
> I'm not convinced that a single interface would work; again, using
> clojure as example, this has moved away from a single interface in Emacs
> (i.e. slime/swank) and toward a single interface for Clojure (so that
> the Clojure side offers a single server, for different editors).

He's not advocating a common interface for all languages at the level
where Emacs interacts with the external process.  He's only advocating
a common interface at the Elisp level so that the major mode only needs
to adapt this interface to the underlying external process's own
interface (or to the Elisp code that does the parsing if it's
implemented in Elisp).


        Stefan

Re: Is intellisense features integration in Emacs technically possible?

[Jorgen Schaefer]

On Thu, 23 Jan 2014 10:12:44 -0500
Stefan Monnier <monnier@iro.umontreal.ca> wrote:

> >> It's mainly a common interface for such external programs so that
> >> we can add more languages more easily. The current effort of
> >> unifying the completion interface as well as supporting company
> >> mode as a front-end is going a great deal forward in that.
> > I'm not convinced that a single interface would work; again, using
> > clojure as example, this has moved away from a single interface in
> > Emacs (i.e. slime/swank) and toward a single interface for Clojure
> > (so that the Clojure side offers a single server, for different
> > editors).
> 
> He's not advocating a common interface for all languages at the level
> where Emacs interacts with the external process.  He's only advocating
> a common interface at the Elisp level so that the major mode only
> needs to adapt this interface to the underlying external process's own
> interface (or to the Elisp code that does the parsing if it's
> implemented in Elisp).

This, exactly. Let me see if I can elaborate.

There are currently four main RPC calls that I use Python for. They
have more or less good support on the Emacs side.


Eldoc
-----

Eldoc is a great example. In its basic form, it's exactly what I
want: Your major mode sets a variable - `eldoc-documentation-function'
- to a function that simply returns the documentation to show. Simple
programming interface to implement, and all modes using it provide a
unified user interface.

There are only two things this is lacking for me right now.

First, it could be improved with more output options. For example,
emacs-jedi.el provides popup-style display for signatures:

http://farm3.staticflickr.com/2845/8793986161_e1c58607f0_o.png

Allowing the user to customize eldoc to use one or the other,
and then having that user's preference take effect in all programming
modes, would be excellent.

Second, and this is a recurring thing for Emacs APIs, it does not
handle asynchronous calls too well. One of the nice things about using
an external process to do the heavy munging of source code is that you
can ignore Emacs single-threadedness and simply keep accepting user
input until the response is there. I currently work around this by
returning the old eldoc string and calling `eldoc-message' in the
asynchronous callback, but I have no idea if eldoc actually expects
this. In the case of auto-completion, this is a lot more complicated.


Completion
-----------

The main topic being argued here. I'm currently using auto-complete.el,
but I hope to eventually use `completion-at-point-functions' once
company-mode is integrated. I'm not sure it will have all the features
I'd like, but those likely could be implemented.

Important features I haven't seen for c-a-p-f yet: Provide an overlay
of the most likely completion candidate while you type for quick
completion with TAB; add annotations to completion candidates, for
example to indicate symbol type; ability to provide documentation for a
completion candidate so that can be shown while browsing candidates.

Examples (using auto-complete and elpy):

Default completion overlay:
http://i.imgur.com/0Fml2dd.png

Some annotations and documentation:
http://i.imgur.com/Vs7WKRa.png

More annotations (using emacs-jedi):
http://farm9.staticflickr.com/8261/8804536872_8d266b88ed_o.png

Oh, and support for getting completion candidates asynchronously. This
is quite tricky, as the user might have moved point in the time the
candidates were returned, and it's not always necessary to re-request
the candidates then. auto-complete.el handles this "mostly ok" using
an init function and caching the response, but has some hard to trace
problems.


Find Definition
---------------

This is what is bound to M-. and M-* in some modes. Providing a single
variable to put a function in that goes to definition at point which is
then called by M-., instead of hard-coding it to `find-tag', would go a
long way. Especially when the definition also keeps track of movement
and adds that to the movement ring for M-* by itself.

Hell, we could even make emacs-lisp-mode provide that and not force the
user to use C-h f all the time to go to the definition! :o)


Show Documentation
------------------

C-h f <symbol> does this for Emacs Lisp, and I don't think there's a
standard key binding for it. Again, a simple variable that can be
changed to return the docstring for the symbol at point which then
shows a help buffer like C-h f does would be nice as a standard
interface.



On the topic of a "unified RPC interface", it does grate me a bit that
every mode implements its own RPC with a major language (elpy
implements a simple JSON-RPC one, emacs-jedi uses the elaborate EPC
library, ropemacs uses Pymacs which uses a very idiosyncratic protocol,
slime does the swank stuff, clojure apparently has its own API now,
etc. etc.), but I'm not sure if that's a solvable problem. Choice of the
RPC mechanism depends as much on Emacs as it does on the capabilities of
the language being talked to. The JSON-RPC code in elpy is a total of
450 lines, both the Emacs Lisp as well as the Python side, including
docstrings and comments, so not having a standard one is not really
a huge problem.

Having a simple interface on the Emacs side that can be used by major
modes so that Emacs presents a unified interface to the user with
minimal effort of reimplementing the wheel every time would be very
useful, though, and quite doable.

Once we have that, it'd be possible to provide one or more "standard
RPC mechanisms" that simply plug in to that API and talk to the
compliant subprocess, but that's not nearly as important right now.


I hope the above explains my remark about the "single interface"
in Emacs. :-)

Regards,
Jorgen


PS. Please do not think that the above is meant as a request that
"someone should implement this"; it's just my thoughts on the topic of
whether intellisense/IDE features integration in Emacs is possible, and
how it could be done. If it happens, great. If not, ok. If it annoys me
too much, I'll work on that eventually, if and when I find the time.

Re: Is intellisense features integration in Emacs technically possible?

[Stefan Monnier]

> First, it could be improved with more output options. For example,
> emacs-jedi.el provides popup-style display for signatures:
> http://farm3.staticflickr.com/2845/8793986161_e1c58607f0_o.png
> Allowing the user to customize eldoc to use one or the other,
> and then having that user's preference take effect in all programming
> modes, would be excellent.

eldoc-mode is fairly old and hasn't seen much development, so it's
fairly primitive.  I'd like to enable it globally by default, and I'd
welcome some improvements in the UI as well.
I personally really like the fact that it uses the echo area, but
I agree that it would be good to provide the user with some
alternative UIs.

> Second, and this is a recurring thing for Emacs APIs, it does not
> handle asynchronous calls too well.

Indeed, this is a serious problem for eldoc.  But it should be fairly
easy to address.  E.g. we could decide that instead of returning
a string, eldoc-documentation-function can return a function which is
then called with a "continuation".  Or we could introduce a new
eldoc-async-documentation-function.

> Important features I haven't seen for c-a-p-f yet: Provide an overlay
> of the most likely completion candidate while you type for quick
> completion with TAB; add annotations to completion candidates, for
> example to indicate symbol type; ability to provide documentation for a
> completion candidate so that can be shown while browsing candidates.

c-a-p-f AFAIK refers to "completion-at-point-functions", which is where
the *backends* live.  AFAIK none of what you cite would be affected by
or need changes in completion-at-point-functions.  Instead those issues
affect the completion UI used on top of completion-at-point-functions,
which could be completion-at-point, company, icomplete, or anything else.

> this. In the case of auto-completion, this is a lot more complicated.

Yes, asynchrony and c-a-p-f can be more problematic.

For completion-at-point, it's not really a problem because by the time
we use c-a-p-f we know we want the answer "right now".  But for Company
we'd instead want to support asynchrony so the external process can take
a little while to return the "current" completion candidates while the
user keeps on typing.  If the candidates arrive too late (the user has
moved on to greener pastures), then just drop the result, and otherwise
display it.

But returning completion candidates asynchronously is not compatible
with the current all-completions/try-completion API, so we'd need
a fairly serious rework of minibuffer.el.


        Stefan

Re: Is intellisense features integration in Emacs technically possible?

[Jorgen Schaefer]

On Thu, 23 Jan 2014 17:13:47 -0500
Stefan Monnier <monnier@iro.umontreal.ca> wrote:

> > Important features I haven't seen for c-a-p-f yet: Provide an
> > overlay of the most likely completion candidate while you type for
> > quick completion with TAB; add annotations to completion
> > candidates, for example to indicate symbol type; ability to provide
> > documentation for a completion candidate so that can be shown while
> > browsing candidates.
> 
> c-a-p-f AFAIK refers to "completion-at-point-functions", which is
> where the *backends* live.  AFAIK none of what you cite would be
> affected by or need changes in completion-at-point-functions.
> Instead those issues affect the completion UI used on top of
> completion-at-point-functions, which could be completion-at-point,
> company, icomplete, or anything else.

I think there is currently no provision for the backend to return
annotation information or documentation that complements the actual
completions? That I meant is that the completion table returned by a
member in `completion-at-point-functions' would need some way of
returning not only the possible completions at point, but also
additional information in some standard way so that the frontends can
display them in addition to the completions.

> But returning completion candidates asynchronously is not compatible
> with the current all-completions/try-completion API, so we'd need
> a fairly serious rework of minibuffer.el.

Do you think reworking minibuffer.el to support both types of calls
with a unified interface (for example with the possibility to block
until the asynchronous call returns if we need the completions "right
now") would be the right thing? Alternatively, a separate in-buffer
completion behavior akin to or based on auto-complete.el might make
more sense?

Jorgen

Re: Is intellisense features integration in Emacs technically possible?

[Stefan Monnier]

> I think there is currently no provision for the backend to return
> annotation information or documentation that complements the actual
> completions?

Yes and no.  The completion-at-point-functions can return any number of
extra properties, and Company uses that to let the backend provide
various extra info (see lisp-completion-at-point in a recent lisp.el for
an example).  IOW you can provide as much extra info as is currently
supported by Company.

>> But returning completion candidates asynchronously is not compatible
>> with the current all-completions/try-completion API, so we'd need
>> a fairly serious rework of minibuffer.el.
> Do you think reworking minibuffer.el to support both types of calls
> with a unified interface (for example with the possibility to block
> until the asynchronous call returns if we need the completions "right
> now") would be the right thing?

Could be.  I haven't thought enough about it to know.  The problematic
part that immediately springs to mind is things like partial-completion
which make various calls to the backend to construct the "list of
candidates".  This would need to be rewritten in a "continuation passing
style" (or event-driven style, if you prefer), I guess, but that'd be
rather inconvenient.

> Alternatively, a separate in-buffer completion behavior akin to or
> based on auto-complete.el might make more sense?

Not sure what that would look like.

`completion-at-point-functions' has 2 "call levels":
- first level is: we call the functions on that hook to know if there's
  a completion and (if there is) what kind of completion it is
  (boundaries, completion-table, properties, ...).
- second level is: we call the completion-table to get the list
  of candidates.

Doing the second level asynchronously means to rewrite
partial-completion and friends in CPS.

But maybe we can get by with only doing the first asynchronously.
IOW the first level could return an :async property which is a function
which you call with a continuation.  That function will contact some
external process and when it's ready it will call the continuation,
passing it the real completion-table.  And of course, we'd need to make
sure that non-async uses can also just wait for the process to return
the completion data.


        Stefan

Re: Is intellisense features integration in Emacs technically possible?

[Jorgen Schaefer]

On Thu, 23 Jan 2014 20:40:36 -0500
Stefan Monnier <monnier@iro.umontreal.ca> wrote:

> > I think there is currently no provision for the backend to return
> > annotation information or documentation that complements the actual
> > completions?
> 
> Yes and no.  The completion-at-point-functions can return any number
> of extra properties, and Company uses that to let the backend provide
> various extra info (see lisp-completion-at-point in a recent lisp.el
> for an example).  IOW you can provide as much extra info as is
> currently supported by Company.

Ok, that was what I was missing. Thank you. :-)

> `completion-at-point-functions' has 2 "call levels":
> - first level is: we call the functions on that hook to know if
> there's a completion and (if there is) what kind of completion it is
>   (boundaries, completion-table, properties, ...).
> - second level is: we call the completion-table to get the list
>   of candidates.
> 
> Doing the second level asynchronously means to rewrite
> partial-completion and friends in CPS.
> 
> But maybe we can get by with only doing the first asynchronously.
> IOW the first level could return an :async property which is a
> function which you call with a continuation.  That function will
> contact some external process and when it's ready it will call the
> continuation, passing it the real completion-table.  And of course,
> we'd need to make sure that non-async uses can also just wait for the
> process to return the completion data.

That sounds sensible. The asynchronous calls I use all return a list of
possible completion candidates and where they start, which is more
robust than auto-complete currently trying to identify the prefix with
Emacs Lisp code. This response can then be used to construct the
completion table without the need for any further asynchronous calls.

Even documentation right now is returned by the initial asynchronous
call and then cached for later use by auto-complete.el.

I'm not sure about "returns an :async property which is called with a
continuation". There does not seem to be a need to return a function
which is called to run another function? Just letting the Emacs code
know that we might add completions later and then calling some
well-known function with additional completions once they are available
would be sufficient I think?


Jorgen

Re: Is intellisense features integration in Emacs technically possible?

[Thien-Thi Nguyen]

[-- Attachment #1: Type: text/plain, Size: 834 bytes --]

() Jorgen Schaefer <forcer@forcix.cx>
() Fri, 24 Jan 2014 11:25:20 +0100

   There does not seem to be a need to return a function which is called
   to run another function? Just letting the Emacs code know that we
   might add completions later and then calling some well-known function
   with additional completions once they are available would be
   sufficient I think?

That is sufficient for 1:1 communication.  To support multiple channels
(which is likely for Emacs -- think multiple buffers), some kind of tag
(identifying cookie) is required.  A tag w/ direction is a continuation.

-- 
Thien-Thi Nguyen
   GPG key: 4C807502
   (if you're human and you know it)
      read my lisp: (responsep (questions 'technical)
                               (not (via 'mailing-list)))
                     => nil

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: Is intellisense features integration in Emacs technically possible?

[Stefan Monnier]

> continuation". There does not seem to be a need to return a function
> which is called to run another function? Just letting the Emacs code
> know that we might add completions later and then calling some
> well-known function with additional completions once they are available
> would be sufficient I think?

It's the same, except that the "well-known" won't be a function name but
the function you receive as argument, which makes it possible for it to
be a closure that holds various side-info which can change over time.


        Stefan

Re: Is intellisense features integration in Emacs technically possible?

[Dmitry Gutov]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> But maybe we can get by with only doing the first asynchronously.
> IOW the first level could return an :async property which is a function
> which you call with a continuation.  That function will contact some
> external process and when it's ready it will call the continuation,
> passing it the real completion-table.  And of course, we'd need to make
> sure that non-async uses can also just wait for the process to return
> the completion data.

Note that a backend that works asynchronously will probably fetch every
kind of data asynchronously, i.e. not only the list of completions, but
also their annotations, location, etc. (though completion-at-point only
supports :annotation, so I mostly mean :company- properties here).

This kind of API would easier to implement in the company-backends,
because all actions are treated equally there. So we could support
something like ('async . (lambda (cont) ...)) as return value for any
action. Except for `prefix', I guess, because that would be weird. And
most of the actions, except `candidates' would still work synchronously
to the user's eye, because they are triggered by explicit user actions.

Still, this would save backend implementers from writing code like
`(while (eq value 'trash) (sit-for 0.01))' each time.

Re: Is intellisense features integration in Emacs technically possible?

[Phillip Lord]

Jorgen Schaefer <forcer@forcix.cx> writes:

> On Thu, 23 Jan 2014 10:12:44 -0500
> Stefan Monnier <monnier@iro.umontreal.ca> wrote:
>
>> >> It's mainly a common interface for such external programs so that
>> >> we can add more languages more easily. The current effort of
>> >> unifying the completion interface as well as supporting company
>> >> mode as a front-end is going a great deal forward in that.
>> > I'm not convinced that a single interface would work; again, using
>> > clojure as example, this has moved away from a single interface in
>> > Emacs (i.e. slime/swank) and toward a single interface for Clojure
>> > (so that the Clojure side offers a single server, for different
>> > editors).
>> 
>
> On the topic of a "unified RPC interface", it does grate me a bit that
> every mode implements its own RPC with a major language (elpy
> implements a simple JSON-RPC one, emacs-jedi uses the elaborate EPC
> library, ropemacs uses Pymacs which uses a very idiosyncratic protocol,
> slime does the swank stuff, clojure apparently has its own API now,
> etc. etc.), but I'm not sure if that's a solvable problem.

This is the sort of thing I was talking about when I said "I'm not
convinced a single interface would work".


> Choice of the RPC mechanism depends as much on Emacs as it does on the
> capabilities of the language being talked to. The JSON-RPC code in
> elpy is a total of 450 lines, both the Emacs Lisp as well as the
> Python side, including docstrings and comments, so not having a
> standard one is not really a huge problem.

But that the mechanism is an RPC, I think, is more or less a given.

Having a set of standard call-back functions for instance would help. If
they all used the same interface, then you could chain them together and
compose them in a sane way. So, for instance, you might be able to pick
up an "parse a JSON-RPC into lisp datastructures" callback, an "adapt
the lisp data structures from my library", and a "do a completion
callback", then compose them and use them. The first could be generic
for a given syntax family of RPC calls, the last generic for the
functionality. So you'd only need to write the middle bit for a given
language.

Just a thought.

Phil

Re: Is intellisense features integration in Emacs technically possible?

[Dmitry Gutov]

Hi Jorgen,

Jorgen Schaefer <forcer@forcix.cx> writes:

> Completion
> -----------
>
> The main topic being argued here. I'm currently using auto-complete.el,
> but I hope to eventually use `completion-at-point-functions' once
> company-mode is integrated.

Like Stefan mentioned, you can already support Company through
`completion-at-point-functions'. Company will need to be installed
manually by each user, though.

> Important features I haven't seen for c-a-p-f yet: Provide an overlay
> of the most likely completion candidate while you type for quick
> completion with TAB; add annotations to completion candidates, for
> example to indicate symbol type; ability to provide documentation for a
> completion candidate so that can be shown while browsing candidates.

See above.

But what's a "most likely completion"? When there's just one suggested
completion, then yes, we show an inline overlay. Otherwise, the full
list.

> Oh, and support for getting completion candidates asynchronously. This
> is quite tricky, as the user might have moved point in the time the
> candidates were returned, and it's not always necessary to re-request
> the candidates then. auto-complete.el handles this "mostly ok" using
> an init function and caching the response, but has some hard to trace
> problems.

I remember you creating a Company issue, me writing you an example
snippet, and you going away seemingly (?) satisfied.

Have you had any progress using it? As long as we don't have
asynchronous users, there's really not much material for me to work with
to improve the API, as well as not much motivation.

Re: Is intellisense features integration in Emacs technically possible?

[Jorgen Schaefer]

On Sun, 26 Jan 2014 01:53:09 +0200
Dmitry Gutov <dgutov@yandex.ru> wrote:

> > Important features I haven't seen for c-a-p-f yet: Provide an
> > overlay of the most likely completion candidate while you type for
> > quick completion with TAB; add annotations to completion
> > candidates, for example to indicate symbol type; ability to provide
> > documentation for a completion candidate so that can be shown while
> > browsing candidates.
> 
> [...]
> 
> But what's a "most likely completion"? When there's just one suggested
> completion, then yes, we show an inline overlay. Otherwise, the full
> list.

The "first" one. ;-) Basically, what you get when you hit RET in
Company, without typing anything. The delay for this can be a lot
shorter, up to "almost instantaneous", than the delay for popping up the
completion pop-up.

(Come to think of it, isn't there a way of sorting completion
results? It's one of the things I want to get auto-complete to do,
because completions from Python backends are most of the time of higher
quality than, say, dabbrev completions.)

> > Oh, and support for getting completion candidates asynchronously.
> > This is quite tricky, as the user might have moved point in the
> > time the candidates were returned, and it's not always necessary to
> > re-request the candidates then. auto-complete.el handles this
> > "mostly ok" using an init function and caching the response, but
> > has some hard to trace problems.
> 
> I remember you creating a Company issue, me writing you an example
> snippet, and you going away seemingly (?) satisfied.
> 
> Have you had any progress using it? As long as we don't have
> asynchronous users, there's really not much material for me to work
> with to improve the API, as well as not much motivation.

I did not look much closer. The solution you provided is more or less
what auto-complete does with its init call and caching, which would
likely work slightly better than auto-complete because I have more
control over it, but well, a-c already has it implemented. Hence, I was
quite satisfied - it solves my problem, and I can not and do not expect
any more from you, you were very helpful already - but it's not what I
was hoping for.

Originally, I created the issue because I was hoping Company would be
able to fully replace auto-complete just requiring fewer hacks and
work on my part. Sadly, while working with it, I realized Company as is
does not really replace auto-complete for me. It's close, but not quite
there yet.

I want to look at Company again at some point in the future and see if
I can make a list of things I miss and maybe start working on them, but
at the moment I'm very low on time, so I pushed that down my list quite
a bit I'm afraid.


As I said in my original post, that was not intended as a request to
get all of this implemented or even as a complaint, just as a note on
what the technical hurdles for Emacs implementing IntelliSense features
actually are: Very few. The features are mostly there, and what is
missing is mainly a common API and some polishing here and there.

Regards,
Jorgen

Re: Is intellisense features integration in Emacs technically possible?

[Dmitry Gutov]

On 26.01.2014 12:15, Jorgen Schaefer wrote:
> The "first" one. ;-) Basically, what you get when you hit RET in
> Company, without typing anything. The delay for this can be a lot
> shorter, up to "almost instantaneous", than the delay for popping up the
> completion pop-up.

I don't really see how that's useful: you can indeed type RET and get 
the same result.

The delay is non-zero by default not only so that the user is not 
constantly interrupted by the popup, but also because the candidates 
retrieval is not instantaneous (and usually blocks Emacs), and to 
retrieve the "first" candidate we'll have to retrieve all candidates anyway.

On the subject of interruptions, personally I'd be almost as annoyed by 
the inline hint overlay appearing too quickly, as by the full popup 
doing the same.

> (Come to think of it, isn't there a way of sorting completion
> results? It's one of the things I want to get auto-complete to do,
> because completions from Python backends are most of the time of higher
> quality than, say, dabbrev completions.)

In Company, completions from merged backends are sorted automatically 
(see `company--multi-backend-adapter', it returns nil for `sorted'). But 
you could write a similar merging adapter that would only sort 
indivitual candidate lists returned from given backends, and then simply 
concatenate them.

> I did not look much closer. The solution you provided is more or less
> what auto-complete does with its init call and caching, which would
> likely work slightly better than auto-complete because I have more
> control over it, but well, a-c already has it implemented. Hence, I was
> quite satisfied - it solves my problem, and I can not and do not expect
> any more from you, you were very helpful already - but it's not what I
> was hoping for.

Well, as long as you're satisfied with the current situation, it's fine, 
I guess.

> I want to look at Company again at some point in the future and see if
> I can make a list of things I miss and maybe start working on them, but
> at the moment I'm very low on time, so I pushed that down my list quite
> a bit I'm afraid.

Ok.

Re: Is intellisense features integration in Emacs technically possible?

[Eric M. Ludlam]

On 01/22/2014 12:29 PM, Phillip Lord wrote:
> Eli Zaretskii<eliz@gnu.org>  writes:
>>> A better way is to build on the hard work of other and interface
>>> emacs with an external tool.
>>
>> Personally, I think implementing such features via external programs
>> is a terrible design.  It will never be smooth and responsive enough,
>> and on top of that you'd need to track development of those other
>> tools.  And what if they become abandoned some day?
>
>
> I think that it depends on the language. Introspecting over, for
> example, Java would require an awful of elisp, which would be difficult
> to write. Getting Java to do this work is quite a lot less effort.
> Hence, the JDEEs use of Java for this (via bsh). Likewise, Clojure and
> Scala both of which use their own language to do much of the work. Or
> for that matter, common lisp with slime/swank. Or even, for that matter,
> English with aspell. I didn't have a problem with responsiveness with
> any of these.

I don't think it is a good idea to consider the problem of doing smart 
completion in Emacs as "Emacs Lisp VS External library".  It is much 
more advantageous to figure out what each is best at and do that instead.

There are certainly examples on both extremes.  CEDET and Semantic by 
themselves can parse surprisingly large C++ code bases and provide smart 
completion quite accurately, but as the code base grows, it gets slower. 
  It also took a really long time to implement in my spare time in spite 
of the vast amount of help I've gotten from its users.

There are also completion you can implement by passing entire buffers 
into an external parser as explained by Jorgen for Python, and I think 
clang could do something similar.  These were coded more quickly, and 
can provide rapid results.

I think it would be better to have a strong mix between the two.  I am 
of course a bit biased since I've put support for this in CEDET quite a 
while ago.  The premise for those who haven't studied how the smart 
completion engine in CEDET works is pretty simple.  Emacs is in charge 
of basic buffer parsing.  As David explained, it is a simple tagging 
parser, but unlike etags/ctags, it collects much more detailed 
information.   It also has a local context parser, allowing it to find 
local variables, and identify the kind of command the cursor is sitting 
on, such as an assignment, struct dereferences, etc.

Once that basic part is done, the smart completion engine starts looking 
up symbols.  It does this via a series of 'semanticdb' EIEIO objects 
that are associated with the buffer.  With no external program support, 
it uses an all Emacs solution to lookup header files or whatever that 
language uses.  Alternately, there are external programs wrapped up in 
this EIEIO class that can provide the same services.   Thus for Java 
there is a 'javap' service that can find symbols in .jar files to 
provide the answer.

There are many external programs such as GNU Global, idutils, cscope, 
and javap that are already supported, and each provides different levels 
of support based on what they can do.  They all make CEDET faster or 
more accurate when in use.

It is tempting to go with an entirely external solution.  They can 
usually be wrapped up by Emacs in pretty quickly.  There are additional 
advantages aside from smart completion to adding parsing support for a 
language directly to Emacs however.  What CEDET does is place overlays 
in the buffer identifying the tags and their nesting.  Simple queries 
can tell you exactly where you are structurally in a program with no 
regexp calls, searching, or other time-consuming activities.  You can 
use this for breadcrumbs, decorations, named bookmarks, navigation, and 
any kind of function that needs to know where you are by name. 
Semantic also can provide access to the current buffer tag-list at next 
to 0 cost when asked (assuming the idle timers have had a chance to run) 
making think like imenu, which-func, speedbar, and ECB very fast.  It 
seems unlikely an external tool could do all that without quite a bit of 
effort.

Lastly, most parts of CEDET/Semantic's parsing and completion engines 
can be replaced per-mode at almost any level, not just at the database 
level.  As such, if someone really wants to use an external tool to 
parse a buffer or provide completion (2 different operations), it will 
let you do it, and all the existing CEDET tools will then work just fine 
on top of the result.

Eric

Re: Is intellisense features integration in Emacs technically possible?

[Phillip Lord]

"Eric M. Ludlam" <eric@siege-engine.com> writes:
>> I think that it depends on the language. Introspecting over, for
>> example, Java would require an awful of elisp, which would be difficult
>> to write. Getting Java to do this work is quite a lot less effort.
>> Hence, the JDEEs use of Java for this (via bsh). Likewise, Clojure and
>> Scala both of which use their own language to do much of the work. Or
>> for that matter, common lisp with slime/swank. Or even, for that matter,
>> English with aspell. I didn't have a problem with responsiveness with
>> any of these.
>
> I don't think it is a good idea to consider the problem of doing smart
> completion in Emacs as "Emacs Lisp VS External library".  It is much more
> advantageous to figure out what each is best at and do that instead.


> I think it would be better to have a strong mix between the two.  I am of
> course a bit biased since I've put support for this in CEDET quite a while
> ago.  The premise for those who haven't studied how the smart completion
> engine in CEDET works is pretty simple.  Emacs is in charge of basic buffer
> parsing.  As David explained, it is a simple tagging parser, but unlike
> etags/ctags, it collects much more detailed information.   It also has a local
> context parser, allowing it to find local variables, and identify the kind of
> command the cursor is sitting on, such as an assignment, struct dereferences,
> etc.


I would agree with this, I think. JDEEs use of CEDET and introspection
worked quite nicely together. Combined with (yet another) analysis step
for fontification.

My main point was to argue that an all elisp solution isn't necessarily
sensible. To use a (strained) example in an English buffer
`forward-word' is always going to be best implemented in elisp, while
having "spell-check" in aspell seems reasonable.

Phil

Re: Is intellisense features integration in Emacs technically possible?

[David Engster]

Tom writes:
> Creating mature and extensive intellisense features is hard work,
> it requires lots of man hours and while it's technically possible to
> implement it in emacs it is unlikely to happen, because emacs lacks
> the developing resources.

Most of the hard work for implementing smart completions was already
done (and yes, it took many years). We have the necessary infrastructure
and fairly well working parsers for C/C++, Java, Python, Fortran90, and
others. Yes, there is still a lot of work to do, and more contributors
would be nice, for sure, but it is absolutely doable.

-David

Re: Is intellisense features integration in Emacs technically possible?

[Tom]

David Engster <deng <at> randomsample.de> writes:
> 
> Most of the hard work for implementing smart completions was already
> done (and yes, it took many years). We have the necessary infrastructure
> and fairly well working parsers for C/C++, Java, Python, Fortran90, and
> others. Yes, there is still a lot of work to do, and more contributors
> would be nice, for sure, but it is absolutely doable.

In your assessment when this support can get to the point when
it's comparable for example to the support Eclipse provides for 
Java (refactoring, on the fly syntax checking, offering automatic
code solutions for problems, etc.)?

Re: Is intellisense features integration in Emacs technically possible?

[David Engster]

Tom writes:
> David Engster <deng <at> randomsample.de> writes:
>> 
>> Most of the hard work for implementing smart completions was already
>> done (and yes, it took many years). We have the necessary infrastructure
>> and fairly well working parsers for C/C++, Java, Python, Fortran90, and
>> others. Yes, there is still a lot of work to do, and more contributors
>> would be nice, for sure, but it is absolutely doable.
>
> In your assessment when this support can get to the point when
> it's comparable for example to the support Eclipse provides for 
> Java (refactoring, on the fly syntax checking, offering automatic
> code solutions for problems, etc.)?

For Java? Never, unless someone seriously starts working on it. However,
your statement that it is futile in the first place will surely not
attract anybody, so please don't make such claims.

-David

Re: Is intellisense features integration in Emacs technically possible?

[Tom]

David Engster <deng <at> randomsample.de> writes:
 
> For Java? Never, unless someone seriously starts working on it. However,
> your statement that it is futile in the first place will surely not
> attract anybody, so please don't make such claims.

My point was that much more can be gained with much less work by
working on interface packages than natively reimplementing these
features.

For example, a polished Eclim package besides providing an
excellent Java support would also provide an excellent way for
developers working with Eclipse to try Emacs, because they could
use the exact same working environment. They could instantly use
Emacs with their projects without any extra setup, so it would be
very easy for them to give Emacs a try.

Re: Is intellisense features integration in Emacs technically possible?

[David Engster]

Tom writes:
> David Engster <deng <at> randomsample.de> writes:
>  
>> For Java? Never, unless someone seriously starts working on it. However,
>> your statement that it is futile in the first place will surely not
>> attract anybody, so please don't make such claims.
>
> My point was that much more can be gained with much less work by
> working on interface packages than natively reimplementing these
> features.

Maybe. I've did my fair share in writing interfaces to external
binaries, and found it to be mostly frustrating. But anyway: people will
work on whatever they want to work on. I for one have no interest in
leaving all the actually rewarding and interesting work to an external
binary. Where's the fun in that?

-David

Re: Is intellisense features integration in Emacs technically possible?

[Eli Zaretskii]

> From: Tom <adatgyujto@gmail.com>
> Date: Tue, 21 Jan 2014 20:24:34 +0000 (UTC)
> 
> David Engster <deng <at> randomsample.de> writes:
>  
> > For Java? Never, unless someone seriously starts working on it. However,
> > your statement that it is futile in the first place will surely not
> > attract anybody, so please don't make such claims.
> 
> My point was that much more can be gained with much less work by
> working on interface packages than natively reimplementing these
> features.

And even more will be gained by using Visual Studio instead of Emacs,
but so what?

Like David points out: a lot of required infrastructure already
exists, so this is doable, given enough motivation.  It's not an
unthinkable effort.

Re: Is intellisense features integration in Emacs technically possible?

[Andreas Röhler]

Am 22.01.2014 04:55, schrieb Eli Zaretskii:
>> From: Tom <adatgyujto@gmail.com>
>> Date: Tue, 21 Jan 2014 20:24:34 +0000 (UTC)
>>
>> David Engster <deng <at> randomsample.de> writes:
>>
>>> For Java? Never, unless someone seriously starts working on it. However,
>>> your statement that it is futile in the first place will surely not
>>> attract anybody, so please don't make such claims.
>>
>> My point was that much more can be gained with much less work by
>> working on interface packages than natively reimplementing these
>> features.
>
> And even more will be gained by using Visual Studio instead of Emacs,

Really?

But seriously: Why not make Emacs user profit the maximum from the environment?

Re: Is intellisense features integration in Emacs technically possible?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

    But seriously: Why not make Emacs user profit the maximum from the environment?

Implicitly that presumes Emacs is an isolated project, assuming that
the principal goal is the usefulness and success of Emacs in
isolation.

The reason we don't make decisions in the way you suggest is that
Emacs is part of a larger project (the GNU Project) with a larger goal
(freedom for software users).

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Is intellisense features integration in Emacs technically possible?

[Jorge Araya Navarro]

[-- Attachment #1: Type: text/plain, Size: 1153 bytes --]

El mar, 21-01-2014 a las 23:02 -0500, emacs-devel-request@gnu.org
escribió:

> From: Eli Zaretskii <eliz@gnu.org>
> > From: Tom <adatgyujto@gmail.com>
> > Date: Tue, 21 Jan 2014 20:24:34 +0000 (UTC)
> > 
> > David Engster <deng <at> randomsample.de> writes:
> >  
> > > For Java? Never, unless someone seriously starts working on it. However,
> > > your statement that it is futile in the first place will surely not
> > > attract anybody, so please don't make such claims.
> > 
> > My point was that much more can be gained with much less work by
> > working on interface packages than natively reimplementing these
> > features.
> 
> And even more will be gained by using Visual Studio instead of Emacs,
> but so what?
> 
> Like David points out: a lot of required infrastructure already
> exists, so this is doable, given enough motivation.  It's not an
> unthinkable effort.


Well, I started to learn elisp today! :D. Ok, that may not be so useful
for the size of the task, but I have to start in some place, right?

-- 
Pax et bonum.
Jorge Araya Navarro.
Diseñador publicitario, programador Python/C++ y colaborador en Parabola
GNU/Linux-libre.

[-- Attachment #2: Type: text/html, Size: 2005 bytes --]

Re: Is intellisense features integration in Emacs technically possible?

[Eli Zaretskii]

> From: Jorge Araya Navarro <elcorreo@deshackra.com>
> Date: Wed, 22 Jan 2014 01:39:02 -0600
> 
> Well, I started to learn elisp today! :D.

Way to go!

> Ok, that may not be so useful for the size of the task, but I have
> to start in some place, right?

Right, and don't be intimidated by a supposedly colossal task, while
you are at it.