IDE

IDE

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

Emacs with GUD is an IDE.  It has a big advantage compared with other IDEs:
when you see a source file, you're editing it with Emacs.

If it falls short compared with other IDEs, I think this would be a
great area for improvement of Emacs.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: IDE

[Tom]

> Emacs with GUD is an IDE.  It has a big advantage compared with
> other IDEs: when you see a source file, you're editing it with Emacs.

This is an advantage for Emacs users who want to use Emacs. Other
IDE users do not care about this that much.

> If it falls short compared with other IDEs, I think this would be a
> great area for improvement of Emacs.

The number one requirement by IDE users today is perfect code completion
and powerful refactoring support for the language they develop in
(Java, C++, etc.).

Any IDE which wants to compete with the popular IDEs must have these,
because users find these features much more helpful in development,
than keyboard macro support and such.

So if Emacs wants to compete with these tools then it has to have seamless,
context aware code completion and refactoring support, and GNU tools
has to provide Emacs the necessary information to implement these
features.

Re: IDE

[Eli Zaretskii]

> From: Tom <adatgyujto@gmail.com>
> Date: Sat, 10 Oct 2015 04:33:59 +0000 (UTC)
> 
> > If it falls short compared with other IDEs, I think this would be a
> > great area for improvement of Emacs.
> 
> The number one requirement by IDE users today is perfect code completion
> and powerful refactoring support for the language they develop in
> (Java, C++, etc.).
> 
> Any IDE which wants to compete with the popular IDEs must have these,
> because users find these features much more helpful in development,
> than keyboard macro support and such.
> 
> So if Emacs wants to compete with these tools then it has to have seamless,
> context aware code completion and refactoring support, and GNU tools
> has to provide Emacs the necessary information to implement these
> features.

I agree.  But to have that, the only way is to have motivated
volunteers step forward and work on these features.  Otherwise we will
never have them.

Right now, no one is working on that, though everyone is talking.  the
same as with weather.

Re: IDE

[Dmitry Gutov]

On 10/10/2015 10:56 AM, Eli Zaretskii wrote:

> Right now, no one is working on that, though everyone is talking.  the
> same as with weather.

No one?

There are quite a few packages that interface with external programs or 
daemons to provide code completion already. Such as Tern for JavaScript, 
Racer for Rust, Compliment for Clojure, gocode for Go, Eclim or Malabar 
for Java, ENSIME for Scala (there has been some movement lately in 
adding Java support, too), Distel for Erlang, Jedi for Python, ocp-index 
for OCaml, ESS for R, maybe some others.

For C/C++, the community has Irony and Rtags, both based on libclang. If 
libclang is unacceptable for you, you probably know a more appropriate 
mailing list to bring that up at.

Would you expect the programs mentioned above to become a part of Emacs? 
For most of them, it's technically unfeasible (not to mention 
organizationally), e.g. because they target several different editors 
(or aim to, in the future).

Re: IDE

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Sat, 10 Oct 2015 11:15:38 +0300
> 
> On 10/10/2015 10:56 AM, Eli Zaretskii wrote:
> 
> > Right now, no one is working on that, though everyone is talking.  the
> > same as with weather.
> 
> No one?

No one.

> There are quite a few packages that interface with external programs or 
> daemons to provide code completion already. Such as Tern for JavaScript, 
> Racer for Rust, Compliment for Clojure, gocode for Go, Eclim or Malabar 
> for Java, ENSIME for Scala (there has been some movement lately in 
> adding Java support, too), Distel for Erlang, Jedi for Python, ocp-index 
> for OCaml, ESS for R, maybe some others.

I was talking about working on IDE, not on completion.  And for the
most popular languages in the industry, not just for some a few niche
languages.

> For C/C++, the community has Irony and Rtags, both based on libclang. If 
> libclang is unacceptable for you, you probably know a more appropriate 
> mailing list to bring that up at.

Let's not reiterate past discussions: you forget CEDET.

And if anyone _really_ cares about supporting C/C++, they should be
working with and on GCC's libcc1, which is available for quite some
time already.

Instead, all we have is heated discussions and hurt feelings.  That
will never get us anywhere.

> Would you expect the programs mentioned above to become a part of Emacs? 

I expect to see a coherent, orchestrated effort towards having an IDE
mode in Emacs.  I don't see it, certainly not in discussions on this
list.  I also don't see any commits that would provide evidence of
such an effort.

If such activities happen somewhere else, I would suggest their
participants to come here and work with and within the core.  For
starters, I don't imagine they would succeed without some significant
changes and additions in the core infrastructure.  The place to
discuss that is here.

> For most of them, it's technically unfeasible (not to mention 
> organizationally), e.g. because they target several different editors 
> (or aim to, in the future).

Then what exactly is the nature of your objections to my observations?
It seems we agree on the bottom line: no one works on this.  The
reasons are immaterial.

Re: IDE

[Dmitry Gutov]

On 10/10/2015 11:30 AM, Eli Zaretskii wrote:

> I was talking about working on IDE, not on completion.  And for the
> most popular languages in the industry, not just for some a few niche
> languages.

You quoted the message that said "accurate code completion and powerful 
refactoring support". I can agree that the latter is barely touched (*), 
but it looked like you ignored the former.

> Let's not reiterate past discussions: you forget CEDET.

I was enumerating external programs. But sure, CEDET is a yet another 
option for completion. Though not too "accurate" one, if we're talking 
anything but C.

> And if anyone _really_ cares about supporting C/C++, they should be
> working with and on GCC's libcc1, which is available for quite some
> time already.

I wasn't aware of it. Is its API stable? Is it a good-enough replacement 
for libclang for the purposes of completion?

If you like, I can pass along the request to use it as an alternative to 
the Irony and Rtags issue trackers. But some more details wouldn't hurt, 
it's hard for me to advocate libcc1 myself.

> Instead, all we have is heated discussions and hurt feelings.  That
> will never get us anywhere.

My feelings aren't hurt, I just meant to add more information to the 
discussion.

> I expect to see a coherent, orchestrated effort towards having an IDE
> mode in Emacs.  I don't see it, certainly not in discussions on this
> list.  I also don't see any commits that would provide evidence of
> such an effort.

We definitely could have more in this department, yes. But what would 
you even call an "IDE mode"? A fixed multi-window setup a la ECB?

I prefer to approach this problem from the bottom - like adding new 
commands that perform certain advanced functions.

> If such activities happen somewhere else, I would suggest their
> participants to come here and work with and within the core.

That's... unlikely. At least, for most of them. My guess is that the 
mailing list interface is off-putting, but maybe we just need a better 
"community outreach" program, or something like that.

That would be something for the new maintainer(s) to consider.

> For
> starters, I don't imagine they would succeed without some significant
> changes and additions in the core infrastructure.  The place to
> discuss that is here.

For refactoring, yes. But just "accurate code completion", without 
extras like expanding the arguments or displaying the method source, can 
be (and is, in certain packages) implemented though the 
completion-at-point-function interface, present in Emacs since 24.1.

And you can have the extras using Company, which should be bundled with 
Emacs in the upcoming version. Or if the ELPA bundling isn't ready by 
then, in the version after that.

> Then what exactly is the nature of your objections to my observations?
> It seems we agree on the bottom line: no one works on this.  The
> reasons are immaterial.

If anything, my view of the situation is a lot brighter than yours.

I also should have more time at the end of this month to put into 
improving xref, which is a step, as you know, in adding a common 
framework for some of the IDE-ish features.

(*) There are some third-party packages providing refactoring commands 
(for dynamic languages, mostly), but they would definitely benefit from 
a nice common UI.

Re: IDE

[David Kastrup]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Tom <adatgyujto@gmail.com>
>> Date: Sat, 10 Oct 2015 04:33:59 +0000 (UTC)
>> 
>> > If it falls short compared with other IDEs, I think this would be a
>> > great area for improvement of Emacs.
>> 
>> The number one requirement by IDE users today is perfect code completion
>> and powerful refactoring support for the language they develop in
>> (Java, C++, etc.).
>> 
>> Any IDE which wants to compete with the popular IDEs must have these,
>> because users find these features much more helpful in development,
>> than keyboard macro support and such.
>> 
>> So if Emacs wants to compete with these tools then it has to have seamless,
>> context aware code completion and refactoring support, and GNU tools
>> has to provide Emacs the necessary information to implement these
>> features.
>
> I agree.  But to have that, the only way is to have motivated
> volunteers step forward and work on these features.  Otherwise we will
> never have them.
>
> Right now, no one is working on that, though everyone is talking.  the
> same as with weather.

Not quite.  IIRC, company-mode integrated (integrates?) the parsing
facilities of Clang with Emacs.  This contribution was rejected (though
I don't have an overview over the actual execution of the rejection)
because of promoting non-GCC compilers.  However, attempts of letting
GCC similarly output AST data were rejected as making it too easy to
support non-free applications (obviously, if Emacs can use GCC for
purposes of syntax analysis from the command line, so can anybody else).
Using the recently admitted GCC plugin interface (previously, plugins
were rejected because they would make supporting non-free applications
too easy) for outputting a full AST was rejected since it would...

You get the drift.  A number of would-be contributors, partly with
solutions or the impetus and skills for creating them, finally went
elsewhere in disgust rather than trying to figure out the maze of which
interoperations between GNU applications were acceptable and which not.

So "no one is working on that, though everyone is talking" is sort of an
unfair characterization because it implies that no one was willing to
put his money and time where his mouth is.

The main problem here is that the default behavior of GNU code is to
refrain from serious interoperability since the GPL (as well as pretty
much anything relying on copyright alone) does not reach across
interoperable interfaces, until such a time that a concrete,
strategically important application requires such interoperability, and
then ways are searched to restrict the interoperability to the
particular combination/application.

So people's hands are bound until a complete plan has been worked out
and/or blessed by Richard and shouting "no one is working on that" is
disingenuous.

I don't think this kind of micromanagement of interoperation can scale.
Lots of worthwhile things come into being by plugging together existing
components in surprising ways and that is an essential component of
academic work and also of software freedom.  Copping out whenever it
gets serious is not going to help free software.

At the same time, the FSF does not turn on a dime.  Which is one of its
strengths.  The future of Emacs is one of the things which has an
impact, and so is the future of GCC.  The GPLv3 is a large "poison pill"
that actually uses copyright in order to disarm the use of patents as a
weapon.  In my opinion, we should rely on its leverage for keeping the
worst of the proprietary misuses in check and give mostly free rein on
interoperability with regard to technical measures.  It's been good
enough for making Apple go away and leave GCC alone for building its
future developer prisons.

For that reason I am glad that John has volunteered to take a look Emacs
maintainership and that he brings the motivation and skills to work on
Emacs being a good IDE and hopefully the motivation to work out with GCC
and RMS about what would be required and how to get there.

Because we've made use of the "it's all talk" excuse far too long for
postponing tackling the increasing issue of interoperability as a design
criterion for Free Software under the GNU umbrella of the FSF.

It's not "all talk" with John on board and I'm glad that he is willing
to talk his application through with Richard.  I severely doubt that it
will be the only time he'll need to talk over things with Richard.

And they are coming at Emacs from different sides and views that will
never be the same and don't need to be.  But the current incompatibility
seems gratuitously bad to me and I think progress on that is quite
necessary.

-- 
David Kastrup

Re: IDE

[Dmitry Gutov]

On 10/10/2015 12:00 PM, David Kastrup wrote:

> Not quite.  IIRC, company-mode integrated (integrates?) the parsing
> facilities of Clang with Emacs.  This contribution was rejected (though
> I don't have an overview over the actual execution of the rejection)
> because of promoting non-GCC compilers.

company-clang is still in GNU ELPA (but shh). It could be considered a 
"toy" completion backend, though, because it only uses the clang 
executable, not libclang, so it doesn't, for instance, do any caching of 
the parsing results, which is necessary for speedy completion in real 
C++ projects.

If there's a program in GCC suite with similar features to 'clang 
-code-completion-at', I'd be happy to integrate it likewise.

Re: IDE

[Eli Zaretskii]

> Cc: adatgyujto@gmail.com, emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Sat, 10 Oct 2015 11:59:34 +0300
> 
> On 10/10/2015 11:30 AM, Eli Zaretskii wrote:
> 
> > I was talking about working on IDE, not on completion.  And for the
> > most popular languages in the industry, not just for some a few niche
> > languages.
> 
> You quoted the message that said "accurate code completion and powerful 
> refactoring support".

No, I responded to a response.  The origin was this:

> If it falls short compared with other IDEs, I think this would be a
> great area for improvement of Emacs.

IOW, the issue is making Emacs a good IDE in general, including
features for browsing code, refactoring, debugging, and all the other
features users expect from a modern IDE.

Come to think of that, even coming up with a list of such features
would be real progress, as I don't think we have that, or ever had.

> I can agree that the latter is barely touched (*), 
> but it looked like you ignored the former.

I didn't ignore that.  I just don't see an effort to make Emacs a
modern IDE.  Working on separate parts of that in separate
uncoordinated activities is not the way we should pursue this, IMO.
It's inefficient at best, and at worst will end up with those
uncoordinated parts falling into oblivion when their driving forces
move on.

> > Let's not reiterate past discussions: you forget CEDET.
> 
> I was enumerating external programs. But sure, CEDET is a yet another 
> option for completion. Though not too "accurate" one, if we're talking 
> anything but C.

It needs to be actively developed.  Much more actively than it is now.

> > And if anyone _really_ cares about supporting C/C++, they should be
> > working with and on GCC's libcc1, which is available for quite some
> > time already.
> 
> I wasn't aware of it. Is its API stable?

I don't know.  It's for someone who will work on this to find out.  I
know that a motivated individual in the GDB development team already
based a useful set of commands on it -- you can compile and inject
code into your program while debugging it.

> Is it a good-enough replacement for libclang for the purposes of
> completion?

I don't know.  If it isn't, then the team who will work on the Emacs
IDE will have to take care of extending libcc1 as well, or find
someone with the GCC team to do that.  Exactly like that GDB developer
did when he needed features that were missing: he implemented them
himself, with guidance from GCC developers.

> > I expect to see a coherent, orchestrated effort towards having an IDE
> > mode in Emacs.  I don't see it, certainly not in discussions on this
> > list.  I also don't see any commits that would provide evidence of
> > such an effort.
> 
> We definitely could have more in this department, yes. But what would 
> you even call an "IDE mode"? A fixed multi-window setup a la ECB?

I don't know, and neither do we as a project.  A useful step would be
to produce a detailed answer to that question.  That answer could both
serve as base for useful discussions, and might provide some anchor
for all those external packages you mentioned to target some coherent
vision.

> I prefer to approach this problem from the bottom - like adding new 
> commands that perform certain advanced functions.

I don't believe comprehensive features such as IDE can be developed
exclusively bottom up.  There should be some basic set of assumptions
and design rules/decisions that everyone should target and abide by.
There should also be some unified leadership.

> > If such activities happen somewhere else, I would suggest their
> > participants to come here and work with and within the core.
> 
> That's... unlikely. At least, for most of them. My guess is that the 
> mailing list interface is off-putting, but maybe we just need a better 
> "community outreach" program, or something like that.

When the work begins in earnest, discussions are rarely needed, except
for discussing some very important design decisions.  Most of the time
you just develop your corner.

Lots of discussions about some feature is IME the first sign that no
one is actually working on it in any serious way.

I remember the beginning of the bidi development: it started by a few
years of discussions (on a separate mailing list) that led nowhere,
and most of them didn't even contribute any useful ideas for what
eventually became the implementation we now have in Emacs.  Once I
started to actually work on this, there was a small number of threads
(maybe 5) here, when I felt I needed to share my main design decisions
and get some feedback.

> That would be something for the new maintainer(s) to consider.

Indeed.

> > For
> > starters, I don't imagine they would succeed without some significant
> > changes and additions in the core infrastructure.  The place to
> > discuss that is here.
> 
> For refactoring, yes. But just "accurate code completion", without 
> extras like expanding the arguments or displaying the method source, can 
> be (and is, in certain packages) implemented though the 
> completion-at-point-function interface, present in Emacs since 24.1.

We didn't even decide that we want that as our mechanism.  Did anyone
consider how this compares with what the other modern IDEs offer?
What if we build our completion on a UI that today's developers will
dislike?  Unlike with many traditional Emacs features, which were
developed when there was no prior art, the IDE features have lots of
prior art.  No need to invent the wheel, just implement similar look
and feel.

> > Then what exactly is the nature of your objections to my observations?
> > It seems we agree on the bottom line: no one works on this.  The
> > reasons are immaterial.
> 
> If anything, my view of the situation is a lot brighter than yours.

I'll be happy to stand corrected.  Unfortunately, I don't yet see any
significant changes in the right direction, so my pessimism is for now
at least as justified as your optimism.

Re: IDE

[David Engster]

Eli Zaretskii writes:
> And if anyone _really_ cares about supporting C/C++, they should be
> working with and on GCC's libcc1,

Getting the AST is not a technical problem; libcc1 does not change
anything.

-David

Re: IDE

[Eli Zaretskii]

> From: David Kastrup <dak@gnu.org>
> Cc: Tom <adatgyujto@gmail.com>,  emacs-devel@gnu.org
> Date: Sat, 10 Oct 2015 11:00:46 +0200
> 
> attempts of letting GCC similarly output AST data were rejected as
> making it too easy to support non-free applications (obviously, if
> Emacs can use GCC for purposes of syntax analysis from the command
> line, so can anybody else).

That's not how that discussion ended.  It ended by Richard saying he
wanted to study the issue in more depth, before making his decision.

In any case, libcc1 is a fait accompli, for several months at least,
and still no one (AFAIK) investigated whether it can serve as basis
for any of the IDE-related features.

> You get the drift.  A number of would-be contributors, partly with
> solutions or the impetus and skills for creating them, finally went
> elsewhere in disgust rather than trying to figure out the maze of which
> interoperations between GNU applications were acceptable and which not.

Yes, that's the "hurt feelings" part of my previous message.

I agree that you need to have some serious will power, perseverance,
and sometimes just stubbornness to get stuff like that done.  I still
hope motivated individuals with enough of that will emerge.

> So "no one is working on that, though everyone is talking" is sort of an
> unfair characterization because it implies that no one was willing to
> put his money and time where his mouth is.

Well, they are not willing badly enough.

Sorry for being blunt, but that's my opinion, being the one who did
something similar, for 10 years, with a task that, given its size and
complexity, was clearly beyond my humble talents.  To this day, I
still don't understand how I succeeded.

> So people's hands are bound until a complete plan has been worked out
> and/or blessed by Richard

No one's hands are bound.  Whoever is motivated enough will find a way
to bypass the restrictions and limitations.  It certainly means more
work, but that's life.

> and shouting "no one is working on that" is disingenuous.

It's a simple fact.  There's nothing disingenuous in reiterating
facts.  If people are unhappy about such a blunt representation of the
facts, then that's fine by me: I actually want people to become
unhappy, because that just might cause someone to stop complaining and
start working.

Re: IDE

[David Engster]

Eli Zaretskii writes:
> I agree that you need to have some serious will power, perseverance,
> and sometimes just stubbornness to get stuff like that done.

Yes, like that guy who ported gccxml from version to version for
*years*. It's now called CastXML.

>  people are unhappy about such a blunt representation of the facts,
> then that's fine by me: I actually want people to become unhappy,
> because that just might cause someone to stop complaining and start
> working.

Oh, I do.

-David

Re: IDE

[Eli Zaretskii]

> From: David Engster <deng@randomsample.de>
> Cc: Dmitry Gutov <dgutov@yandex.ru>,  adatgyujto@gmail.com,  emacs-devel@gnu.org
> Date: Sat, 10 Oct 2015 11:51:14 +0200
> 
> Eli Zaretskii writes:
> > And if anyone _really_ cares about supporting C/C++, they should be
> > working with and on GCC's libcc1,
> 
> Getting the AST is not a technical problem; libcc1 does not change
> anything.

We don't even know whether libcc1 provides any features useful for an
IDE, including information that could be used fro completion.

We also don't know what is the final verdict on having GCC emit tree
information.

I'd expect a person who is motivated to work on these features, if we
have such a person among us, to be on top of those two issues, as well
as many others.

But I agree: it's not a technical problem.  It's a problem of being
motivated enough to overcome such obstacles.

Re: IDE

[Dmitry Gutov]

On 10/10/2015 12:40 PM, Eli Zaretskii wrote:

> I didn't ignore that.  I just don't see an effort to make Emacs a
> modern IDE.  Working on separate parts of that in separate
> uncoordinated activities is not the way we should pursue this, IMO.

At least we "have volunteers" for that.

> It's inefficient at best, and at worst will end up with those
> uncoordinated parts falling into oblivion when their driving forces
> move on.

That would be accurate to say about projects at early stages of 
development, but the respective third-party packages already work now. 
If anything, we could keep them working, even Emacs undergoes large 
internal changes.

> It needs to be actively developed.  Much more actively than it is now.

I suppose.

> I don't know.  It's for someone who will work on this to find out.  I
> know that a motivated individual in the GDB development team already
> based a useful set of commands on it -- you can compile and inject
> code into your program while debugging it.

That seems orthogonal to code completion capabilities, for where I'm 
standing. But I'm not a good person to judge.

This does make a good material for a feature request for Irony, 
unfortunately. Someone more knowledgeable should do that instead.

>> We definitely could have more in this department, yes. But what would
>> you even call an "IDE mode"? A fixed multi-window setup a la ECB?
>
> I don't know, and neither do we as a project.  A useful step would be
> to produce a detailed answer to that question.  That answer could both
> serve as base for useful discussions, and might provide some anchor
> for all those external packages you mentioned to target some coherent
> vision.

"We need a common interface for refactoring tools" sounds like a good 
problem statement. I hope that capability can grow from the xref 
interface, but that needs more work and thought.

> I don't believe comprehensive features such as IDE can be developed
> exclusively bottom up.  There should be some basic set of assumptions
> and design rules/decisions that everyone should target and abide by.
> There should also be some unified leadership.

A comprehensive set of IDE features might be too lofty a goal for us, in 
the foreseeable future.

> We didn't even decide that we want that as our mechanism.  Did anyone
> consider how this compares with what the other modern IDEs offer?

completion-at-point-functions is the API for backends to implement. We 
have a few frontends for it already. Company can use it, for one.

> What if we build our completion on a UI that today's developers will
> dislike?  Unlike with many traditional Emacs features, which were
> developed when there was no prior art, the IDE features have lots of
> prior art.  No need to invent the wheel, just implement similar look
> and feel.

Hence we're bundling Company.

Re: IDE

[Eli Zaretskii]

> From: David Engster <deng@randomsample.de>
> Date: Sat, 10 Oct 2015 12:02:11 +0200
> Cc: David Kastrup <dak@gnu.org>, adatgyujto@gmail.com, emacs-devel@gnu.org
> 
> Eli Zaretskii writes:
> > I agree that you need to have some serious will power, perseverance,
> > and sometimes just stubbornness to get stuff like that done.
> 
> Yes, like that guy who ported gccxml from version to version for
> *years*. It's now called CastXML.

There were 2 bidi implementations for Emacs when I started to work on
what we have now.  Both of them were not good enough, according to the
then Emacs head maintainer.  Imagine what would we have now if I were
less determined to get it right, even though I didn't yet understand
then why those existing implementations couldn't be used.  (I do now,
10 years and many more gray hair later.)

Yes, the situation with the IDE is different, but the morale still
stands: given enough will power, nothing can stand in one's way that
cannot be solved/overcome/worked around.

Re: IDE

[David Kastrup]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: David Kastrup <dak@gnu.org>
>> Cc: Tom <adatgyujto@gmail.com>,  emacs-devel@gnu.org
>> Date: Sat, 10 Oct 2015 11:00:46 +0200
>> 
>> attempts of letting GCC similarly output AST data were rejected as
>> making it too easy to support non-free applications (obviously, if
>> Emacs can use GCC for purposes of syntax analysis from the command
>> line, so can anybody else).
>
> That's not how that discussion ended.  It ended by Richard saying he
> wanted to study the issue in more depth, before making his decision.

It's been more than half a year.  The time frames in which people
involve themselves in personal projects are smaller than that.  So the
issue becomes a theoretic one and does no longer warrant taking risks or
making difficult decisions.

Rinse and repeat.

That's why I think we need a general course change regarding
interoperability without an immediate tangible purpose.  Because
otherwise we kill incubation.  The most important tool of the programmer
is the garbage bin.  If we take that away, if we demand a proof of
success before opening any possibility, we are taking one of the most
important assets of freedom: the possibility that someone may use that
freedom for purposes beyond our imagination.  Yes, that may cut both
ways.  But we do have the GPLv3 and we should trust it to keep the
damage in check because the alternative is limiting what we can do with
free software for advancing free software.

>> So people's hands are bound until a complete plan has been worked out
>> and/or blessed by Richard
>
> No one's hands are bound.  Whoever is motivated enough will find a way
> to bypass the restrictions and limitations.

That's not particularly inspiring.

-- 
David Kastrup

Re: IDE

[David Engster]

Eli Zaretskii writes:
> Yes, the situation with the IDE is different, but the morale still
> stands: given enough will power, nothing can stand in one's way that
> cannot be solved/overcome/worked around.

You still seem to think I abandoned my work. I did not. I merely
switched libraries.

-David

Re: IDE

[Eli Zaretskii]

> Cc: adatgyujto@gmail.com, emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Sat, 10 Oct 2015 13:14:53 +0300
> 
> On 10/10/2015 12:40 PM, Eli Zaretskii wrote:
> 
> > I don't know.  It's for someone who will work on this to find out.  I
> > know that a motivated individual in the GDB development team already
> > based a useful set of commands on it -- you can compile and inject
> > code into your program while debugging it.
> 
> That seems orthogonal to code completion capabilities, for where I'm 
> standing.

Of course, it is.  I didn't mean to say that injecting code and
completion/refactoring need the same capabilities.  But libcc1 doesn't
provide only what GDB uses, and it can be extended.

> >> We definitely could have more in this department, yes. But what would
> >> you even call an "IDE mode"? A fixed multi-window setup a la ECB?
> >
> > I don't know, and neither do we as a project.  A useful step would be
> > to produce a detailed answer to that question.  That answer could both
> > serve as base for useful discussions, and might provide some anchor
> > for all those external packages you mentioned to target some coherent
> > vision.
> 
> "We need a common interface for refactoring tools" sounds like a good 
> problem statement.

Is IDE just about refactoring?  I thought it meant much more.

> > I don't believe comprehensive features such as IDE can be developed
> > exclusively bottom up.  There should be some basic set of assumptions
> > and design rules/decisions that everyone should target and abide by.
> > There should also be some unified leadership.
> 
> A comprehensive set of IDE features might be too lofty a goal for us, in 
> the foreseeable future.

Depends on how many people will work on it.  In any case, having some
high-level design that is targeted by all the components will ensure
more or less seamless integration when each component becomes
available.

> > What if we build our completion on a UI that today's developers will
> > dislike?  Unlike with many traditional Emacs features, which were
> > developed when there was no prior art, the IDE features have lots of
> > prior art.  No need to invent the wheel, just implement similar look
> > and feel.
> 
> Hence we're bundling Company.

Last time I looked the IDEs I sometimes look at (Visual Studio and
Eclipse) present a much more pleasant UI for completion.  Why can't we
present something similar?

Re: IDE

[Eli Zaretskii]

> From: David Kastrup <dak@gnu.org>
> Cc: adatgyujto@gmail.com,  emacs-devel@gnu.org
> Date: Sat, 10 Oct 2015 12:23:11 +0200
> 
> >> So people's hands are bound until a complete plan has been worked out
> >> and/or blessed by Richard
> >
> > No one's hands are bound.  Whoever is motivated enough will find a way
> > to bypass the restrictions and limitations.
> 
> That's not particularly inspiring.

I hope it is.  All I have is my personal example; if that's no
inspiring, then I don't know what will or could be.

Re: IDE

[Eli Zaretskii]

> From: David Engster <deng@randomsample.de>
> Cc: adatgyujto@gmail.com,  emacs-devel@gnu.org
> Date: Sat, 10 Oct 2015 12:29:52 +0200
> 
> Eli Zaretskii writes:
> > Yes, the situation with the IDE is different, but the morale still
> > stands: given enough will power, nothing can stand in one's way that
> > cannot be solved/overcome/worked around.
> 
> You still seem to think I abandoned my work. I did not. I merely
> switched libraries.

If that means your work will some day be accepted into Emacs, I'm glad
to hear that.

TIA

Re: IDE

[David Engster]

Eli Zaretskii writes:
>> From: David Engster <deng@randomsample.de>
>> Cc: adatgyujto@gmail.com,  emacs-devel@gnu.org
>> Date: Sat, 10 Oct 2015 12:29:52 +0200
>
>> 
>> Eli Zaretskii writes:
>> > Yes, the situation with the IDE is different, but the morale still
>> > stands: given enough will power, nothing can stand in one's way that
>> > cannot be solved/overcome/worked around.
>> 
>> You still seem to think I abandoned my work. I did not. I merely
>> switched libraries.
>
> If that means your work will some day be accepted into Emacs, I'm glad
> to hear that.

Not for me to decide.

-David

Re: IDE

[David Kastrup]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: David Kastrup <dak@gnu.org>
>> Cc: adatgyujto@gmail.com,  emacs-devel@gnu.org
>> Date: Sat, 10 Oct 2015 12:23:11 +0200
>> 
>> >> So people's hands are bound until a complete plan has been worked out
>> >> and/or blessed by Richard
>> >
>> > No one's hands are bound.  Whoever is motivated enough will find a
>> > way to bypass the restrictions and limitations.
>> 
>> That's not particularly inspiring.
>
> I hope it is.  All I have is my personal example;

Your personal example was not about a policy and decision outside of
your influence ruling your work unwelcome.

Your example was about overcoming technical difficulties and surpassing
a threshold of code/performance quality.

-- 
David Kastrup

Re: IDE

[Dmitry Gutov]

On 10/10/2015 01:34 PM, Eli Zaretskii wrote:

>> "We need a common interface for refactoring tools" sounds like a good
>> problem statement.
>
> Is IDE just about refactoring?  I thought it meant much more.

The above more focused and, as such, more useful. "Comprehensive IDE 
features" is not as useful.

>> A comprehensive set of IDE features might be too lofty a goal for us, in
>> the foreseeable future.
>
> Depends on how many people will work on it.

How many people would you expect to work on it in the near future, 
realistically?

> In any case, having some
> high-level design that is targeted by all the components will ensure
> more or less seamless integration when each component becomes
> available.

 From where I'm standing, this sentence is not useful. Not only you're 
asking for a big design, you don't present a justification for it, e.g. 
how it would be reflected in all components.

> Last time I looked the IDEs I sometimes look at (Visual Studio and
> Eclipse) present a much more pleasant UI for completion.  Why can't we
> present something similar?

Well, that hurts (a bit). If Company's tooltip is not pleasant, what 
would be pleasant for you?

Re: IDE

[Eli Zaretskii]

> From: David Kastrup <dak@gnu.org>
> Cc: adatgyujto@gmail.com,  emacs-devel@gnu.org
> Date: Sat, 10 Oct 2015 12:47:44 +0200
> 
> > I hope it is.  All I have is my personal example;
> 
> Your personal example was not about a policy and decision outside of
> your influence

You don't know all the details.

I also don't see how the exact cause for rejection of existing work
can matter here.

> ruling your work unwelcome.

Ah, so we are again talking about hurt feelings?

> Your example was about overcoming technical difficulties and surpassing
> a threshold of code/performance quality.

Policy decisions can be easily translated into "technical
difficulties", by writing more code.

Re: IDE

[Eli Zaretskii]

> Cc: adatgyujto@gmail.com, emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Sat, 10 Oct 2015 13:50:59 +0300
> 
> On 10/10/2015 01:34 PM, Eli Zaretskii wrote:
> 
> >> "We need a common interface for refactoring tools" sounds like a good
> >> problem statement.
> >
> > Is IDE just about refactoring?  I thought it meant much more.
> 
> The above more focused and, as such, more useful. "Comprehensive IDE 
> features" is not as useful.

But it narrows the field too much, IMO.

> >> A comprehensive set of IDE features might be too lofty a goal for us, in
> >> the foreseeable future.
> >
> > Depends on how many people will work on it.
> 
> How many people would you expect to work on it in the near future, 
> realistically?

I don't know.

If we cannot find enough, then it simply means it will take more time
to implement the features sequentially rather than in parallel.

> > In any case, having some
> > high-level design that is targeted by all the components will ensure
> > more or less seamless integration when each component becomes
> > available.
> 
>  From where I'm standing, this sentence is not useful. Not only you're 
> asking for a big design, you don't present a justification for it, e.g. 
> how it would be reflected in all components.

Are you saying that high-level design is generally not useful, and
should be avoided, unless "justified"?  That goes against the
engineering principles that the whole industry abides by.

> > Last time I looked the IDEs I sometimes look at (Visual Studio and
> > Eclipse) present a much more pleasant UI for completion.  Why can't we
> > present something similar?
> 
> Well, that hurts (a bit).

Sorry.

> If Company's tooltip is not pleasant, what would be pleasant for
> you?

I just gave you 2 examples.

And it's not really about me, it's about the expectations of users out
there.  Don't they expect something they see elsewhere?

Re: IDE

[David Kastrup]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: David Kastrup <dak@gnu.org>
>> Cc: adatgyujto@gmail.com,  emacs-devel@gnu.org
>> Date: Sat, 10 Oct 2015 12:47:44 +0200
>> 
>> > I hope it is.  All I have is my personal example;
>> 
>> Your personal example was not about a policy and decision outside of
>> your influence
>
> You don't know all the details.
>
> I also don't see how the exact cause for rejection of existing work
> can matter here.
>
>> ruling your work unwelcome.
>
> Ah, so we are again talking about hurt feelings?

I had no work invested in that area, so trying to turn this into an ad
hominem attack is a bit pointless.  Anyway, as long as we rely on most
software being written by humans, "hurt feelings" is not a category we
can consider entirely irrelevant.  Even though in this case it is
entirely a straw man you are trying to create from my choice of words.

>> Your example was about overcoming technical difficulties and surpassing
>> a threshold of code/performance quality.
>
> Policy decisions can be easily translated into "technical
> difficulties", by writing more code.

Shrug.  One can rewrite GCC in Elisp in order to avoid the interdict
against generic interfaces and data exchange.  Sure.  But that's the
kind of obstacle one expects proprietary software to pose, not free
software entirely under copyright and control of the FSF.

Rewriting something from scratch for the sake of escaping the shackles
of proprietary software is a motivation we can expect some contributors
to Emacs and GCC to have.  Rewriting something from scratch for the sake
of escaping the shackles of applied FSF policies isn't.  That's just not
the typical target clientele of GCC and Emacs and it shouldn't need to
be.

-- 
David Kastrup

Re: IDE

[Eli Zaretskii]

> From: David Kastrup <dak@gnu.org>
> Cc: adatgyujto@gmail.com,  emacs-devel@gnu.org
> Date: Sat, 10 Oct 2015 13:20:48 +0200
> 
> >> ruling your work unwelcome.
> >
> > Ah, so we are again talking about hurt feelings?
> 
> I had no work invested in that area, so trying to turn this into an ad
> hominem attack is a bit pointless.

No attack here, David, certainly not on you.  I just wanted to know
whether we are talking about technical difficulties or psychological
ones.

The latter is exactly where will power should come into play.

> > Policy decisions can be easily translated into "technical
> > difficulties", by writing more code.
> 
> Shrug.  One can rewrite GCC in Elisp in order to avoid the interdict
> against generic interfaces and data exchange.

I don't think one needs to rewrite GCC in any language to get the
information we need.  Some coding is indeed in order.

> Rewriting something from scratch for the sake of escaping the shackles
> of proprietary software is a motivation we can expect some contributors
> to Emacs and GCC to have.  Rewriting something from scratch for the sake
> of escaping the shackles of applied FSF policies isn't.  That's just not
> the typical target clientele of GCC and Emacs and it shouldn't need to
> be.

Now you are creating a straw man.

Re: IDE

[Óscar Fuentes]

Eli Zaretskii <eliz@gnu.org> writes:

> In any case, libcc1 is a fait accompli, for several months at least,
> and still no one (AFAIK) investigated whether it can serve as basis
> for any of the IDE-related features.

Here's the answer, from the horse's mouth:

https://lwn.net/Articles/629259/

Read the comments written by mlopezibanez.

Re: IDE

[Dmitry Gutov]

On 10/10/2015 02:03 PM, Eli Zaretskii wrote:

> Are you saying that high-level design is generally not useful, and
> should be avoided, unless "justified"?

A justification would further the discussion. Simply asking for a big 
design, doesn't.

>> If Company's tooltip is not pleasant, what would be pleasant for
>> you?
>
> I just gave you 2 examples.

You didn't explain how they are better.

> And it's not really about me, it's about the expectations of users out
> there.  Don't they expect something they see elsewhere?

Sometimes. A common question is for a popup that doesn't conflict with 
other packages that use overlay rendering for other things.

Did you also mean this aspect?

Re: IDE

[Dmitry Gutov]

On 10/10/2015 02:03 PM, Eli Zaretskii wrote:

>> The above more focused and, as such, more useful. "Comprehensive IDE
>> features" is not as useful.
>
> But it narrows the field too much, IMO.

I wonder.

 From what I've seen, Emacs facilities that try to do too much, end up 
over-specializing. That limits the number of users and, consequently, 
volunteers that would want to support it further. In my view, CEDET is 
an example of that.

Another example is ECB that controls how all windows are displayed, and 
trying to do that in the fashion that many IDE users are accustomed to.

My story (and many other users' too, I'm sure) is that I've tried using 
it for a while, and in the end decided that mimicking the IDE workflow 
in that respect is not very valuable. It hasn't seen any new development 
for years.

Re: IDE

[Eli Zaretskii]

> Cc: adatgyujto@gmail.com, emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Sat, 10 Oct 2015 17:15:07 +0300
> 
> On 10/10/2015 02:03 PM, Eli Zaretskii wrote:
> 
> > Are you saying that high-level design is generally not useful, and
> > should be avoided, unless "justified"?
> 
> A justification would further the discussion. Simply asking for a big 
> design, doesn't.

It's standard software engineering practice, why should you ask for
its justification?

> >> If Company's tooltip is not pleasant, what would be pleasant for
> >> you?
> >
> > I just gave you 2 examples.
> 
> You didn't explain how they are better.

They look nicer.  I don't know how to explain better.

> > And it's not really about me, it's about the expectations of users out
> > there.  Don't they expect something they see elsewhere?
> 
> Sometimes. A common question is for a popup that doesn't conflict with 
> other packages that use overlay rendering for other things.

The other IDEs use something similar to a tooltip, or a drop-down menu
with different fonts and colors.

Re: IDE

[Eli Zaretskii]

> Cc: adatgyujto@gmail.com, emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Sat, 10 Oct 2015 17:20:46 +0300
> 
> On 10/10/2015 02:03 PM, Eli Zaretskii wrote:
> 
> >> The above more focused and, as such, more useful. "Comprehensive IDE
> >> features" is not as useful.
> >
> > But it narrows the field too much, IMO.
> 
> I wonder.
> 
>  From what I've seen, Emacs facilities that try to do too much, end up 
> over-specializing. That limits the number of users and, consequently, 
> volunteers that would want to support it further. In my view, CEDET is 
> an example of that.

I didn't suggest to use CEDET as the starting point for this purpose.
I suggested to look at the popular IDEs out there, and use their
features as such a starting point.

Once again, we have prior art at our fingertips.  I believe the
features provided by the existing IDEs are a good approximation for
what people will generally expect from an IDE.  I think making a list
of the features we would like to see in the Emacs IDE, based on the
existing prior art, would be a good step forward.

But I repeat myself.  If you still don't agree, let's agree to
disagree on this.

> Another example is ECB that controls how all windows are displayed, and 
> trying to do that in the fashion that many IDE users are accustomed to.

Don't we lack features to support that?  Emacs generally doesn't let
you "dedicate" windows quite like IDEs do, even though we try.

Re: IDE

[David Engster]

Eli Zaretskii writes:
>> Cc: adatgyujto@gmail.com, emacs-devel@gnu.org
>> From: Dmitry Gutov <dgutov@yandex.ru>
>> Date: Sat, 10 Oct 2015 17:20:46 +0300
>> Another example is ECB that controls how all windows are displayed, and 
>> trying to do that in the fashion that many IDE users are accustomed to.
>
> Don't we lack features to support that?

Yes. ECB uses lots of advices to achieve what it does. I think somebody
(Martin?) worked on a 'window group' feature which would make this
easier.

-David

Re: IDE

[Eli Zaretskii]

> From: David Engster <deng@randomsample.de>
> Cc: Dmitry Gutov <dgutov@yandex.ru>,  adatgyujto@gmail.com,  emacs-devel@gnu.org
> Date: Sat, 10 Oct 2015 17:02:39 +0200
> 
> Eli Zaretskii writes:
> >> Cc: adatgyujto@gmail.com, emacs-devel@gnu.org
> >> From: Dmitry Gutov <dgutov@yandex.ru>
> >> Date: Sat, 10 Oct 2015 17:20:46 +0300
> >> Another example is ECB that controls how all windows are displayed, and 
> >> trying to do that in the fashion that many IDE users are accustomed to.
> >
> > Don't we lack features to support that?
> 
> Yes. ECB uses lots of advices to achieve what it does. I think somebody
> (Martin?) worked on a 'window group' feature which would make this
> easier.

That's what I thought.

So I guess we should ask Martin to please make that happen, as part of
work on Emacs IDE.

Re: IDE

[Eric Ludlam]

On 10/10/2015 04:59 AM, Dmitry Gutov wrote:
> On 10/10/2015 11:30 AM, Eli Zaretskii wrote:
>
>> I was talking about working on IDE, not on completion.  And for the
>> most popular languages in the industry, not just for some a few niche
>> languages.
>
> You quoted the message that said "accurate code completion and powerful
> refactoring support". I can agree that the latter is barely touched (*),
> but it looked like you ignored the former.
>
>> Let's not reiterate past discussions: you forget CEDET.
>
> I was enumerating external programs. But sure, CEDET is a yet another
> option for completion. Though not too "accurate" one, if we're talking
> anything but C.

I had always intended CEDET to be a baseline for IDE like features. 
Looking just at tagging files, those familiar with it's internals 
recognize that it can use external tools as weak as ctags or GLOBAL, and 
can use a more powerful external tool for parsing as well, such as using 
JAVAC for decompiling .jar files into tags.  As a backup, it also has 
in-buffer parsing for when you've only half-written your code, or when 
no external tool is available.

The same philosophy is throughout CEDET.  Unfortunately, those who are 
interested in tooling who brush casually past CEDET only see that people 
complain that it doesn't always complete right, or that the C++ part is 
hard to setup, and don't see that the cool piece they want integrated 
into Emacs could use CEDET as a framework.

A side effect is that every completion engine out there has a custom way 
of integrating random tools in.  In theory, completion engines could 
call into CEDET, and let CEDET dispatch to the other tools.  In that 
world, bootstrapping new tools is simpler with only one backend to worry 
about, and there would be more language independent creativity for IDE 
like features.

If we look at the two starting key features listed for IDEs, 
"completion" and "refactoring", the basics are all there.  CEDET has a 
completion engine, but it is only as good as the input data.  If better 
external tools (like GCC) or easier to configure tools could feed it, it 
would be more accurate.  CEDET also has 'srecode' which is about 
generating code.  This bit is trickier, as the intention is you could 
write one highlevel tool that might extract tag data from your file, 
permute the language-independent tags, and then write them back out with 
srecode.  There are tests showing this working, but they are simple and 
proof of concept and not stuff someone would depend on day-to-day.

What it needs is folks who care about cross language tooling, and not 
just the language specific part.  While I continue to maintain CEDET, I 
don't code for a living anymore, so do not run into CEDET's rough 
corners enough to know to fix them.

Eric

Re: IDE

[martin rudalics]

 >> Yes. ECB uses lots of advices to achieve what it does. I think somebody
 >> (Martin?) worked on a 'window group' feature which would make this
 >> easier.
 >
 > That's what I thought.
 >
 > So I guess we should ask Martin to please make that happen, as part of
 > work on Emacs IDE.

Several years ago I've been discussing this issue with Klaus Berndl.
The basic problem at that time was to get rid of the advices.  In the
sequel I installed the side windows code in window.el and later adapted
it to the then new buffer display functions.  There are two functions
‘display-buffer-in-side-window’, ‘display-buffer-in-major-side-window’
and a number of window parameters (‘window-side’, ‘window-slot’,
‘delete-window’ and ‘other-window’) which should suffice to emulate
practically everything I found in ECB.

I never use side windows so I can't tell whether they still work.  I
have written a frame-tabs.el package based on side windows but I don't
use that either.  At the time I installed the side windows functions I
also added a texinfo section but Stefan later asked me to remove it.
That info does not reflect later changes to the code so it might be
outdated.  You have to live with the doc-strings which should be fairly
accurate.

martin

Re: IDE

[martin rudalics]

 > The other IDEs use something similar to a tooltip, or a drop-down menu
 > with different fonts and colors.

Cedet does have ‘semantic-displayor-tooltip-mode’.  And I'm using
tooltips all day for eldoc.  The only drawback is that Emacs tooltips
are too voracious.

martin

Re: IDE

[John Wiegley]

>>>>> martin rudalics <rudalics@gmx.at> writes:

> I never use side windows so I can't tell whether they still work. I have
> written a frame-tabs.el package based on side windows but I don't use that
> either. At the time I installed the side windows functions I also added a
> texinfo section but Stefan later asked me to remove it. That info does not
> reflect later changes to the code so it might be outdated. You have to live
> with the doc-strings which should be fairly accurate.

We should define what we want from a "more IDE" Emacs. For example, I do not
want window-layout functionality. That's a presentation aspect of IDEs that's
entirely separate from what I want from them.

Right now we have a pretty nice infrastructure for things like indenting code.
That is, there are standard keybindings (TAB, C-M-\), a standard per-buffer
override variable (indent-line-function), a standard command
(indent-according-to-mode), and ways for packages like Yasnippet to override
the meaning of TAB without ruining functionality.

I think that what an "IDE is" has little to do with what it looks like. Emacs
being a better IDE, to me, means making IDE-like functionality a first-class
citizen, as we do with indenting. This would provide a core for fancy display
modules to build on top of.

I also don't think core Emacs should *provide* this functionality, since
that's impossible, given how many different languages and use cases there are.
It's more about giving developers a common API to base their work on, so that
we maximize collaboration between them.

Here is a list of functionality I think should be first-class, structurally
speaking (that is, an API will exist, but it may not do anything until a
contributor implements the functionality for their language). The ones with
a * mean areas where we're already succeeding to some degree:

  * indentation (see above)
    reformatting
  * syntax highlighting (font-lock)
    help at point
    documentation lookup (sadly, fewer projects use Info these days)
  ? completion
    refactoring
    semantic editing (for example, paredit)
  * compilation (M-x compile)
    live compilation (flymake/flycheck)
  * REPL (comint)
    running tests
  * debugging (GUD)
  * version control (VC)
    profiling
    code coverage
    app interaction

The reason I don't have a * next to completion, despite having so many things
capable of doing it (company, auto-complete, ido, hippie-expand, helm, ivy,
etc., etc.), is that there are too MANY ways to do it. This is where I think
proper IDE support could help.

If we have a single paradigm for "determining interesting details about the
buffer, and near point", with the ability to refine the query based on what is
need, optionally cache results, etc., then the competing libraries we have
today could share functionality. The present day `all-completions` function is
too spare to fit this bill, so it's less of an IDE feature in my book and more
just a Lisp library function. For example, I've written nearly the same
backend code for at least 4 different completion/lookup packages, and each
time I wonder how we could do better here. The differences are so minimal.

To reiterate: I think Emacs becomes more of an IDE when it provides the
backbone of an IDE, and not when it looks like one. We have some pieces of
that backbone already, which have avoided fragmentation in those areas, but we
need more. A standardized way to do tooltips, popups, visual completion
selection (or at least the structure), REPLs that interact with buffer
contents, etc., would give us a foundation to move to the next step.

John

Re: IDE

[David Kastrup]

"John Wiegley" <johnw@newartisans.com> writes:

> Right now we have a pretty nice infrastructure for things like
> indenting code.  That is, there are standard keybindings (TAB, C-M-\),
> a standard per-buffer override variable (indent-line-function), a
> standard command (indent-according-to-mode), and ways for packages
> like Yasnippet to override the meaning of TAB without ruining
> functionality.

Emacs sucks at multiple-language buffers out of the box.  They are
inherent for any form of literate programming, they are rather prevalent
for embedding languages in HTML, they are useful for things like Texinfo
with embedded examples, I'd need them for LilyPond (which integrates
Scheme code).  Many IDEs are single-language to start with, and if they
aren't, they are either generic or single-language per file.

There is mmm-mode but it's not part of ELPA (and probably not
copyright-assignable, at least that's my guess of why it might be out).
And I strongly suspect that some stronger core features inside of Emacs
would be desirable, like keeping indentation and font-locking engines
and syntax constrained to regions identified with overlays or text
properties.

-- 
David Kastrup

Re: IDE

[Eli Zaretskii]

> From: "John Wiegley" <johnw@newartisans.com>
> Date: Sat, 10 Oct 2015 11:31:13 -0700
> 
> To reiterate: I think Emacs becomes more of an IDE when it provides the
> backbone of an IDE, and not when it looks like one.

I don't think this will fly.  In effect, you are telling Emacs users
they will have to code their own IDE by using the infrastructure
(a.k.a. "backbone") that we provide.  If I were that user, I'd
immediately turn away and look elsewhere.

An analogy to that would be if, instead of Gnus, we'd provide an
infrastructure for a news- and mail-reader.

To me, an IDE is not a set of functionalities.  It's a coherent
application that provides an IDE-like look-and-feel, and all the
related functions already integrated and ready for me to be used.
That includes window-layout, btw, because configuring Emacs windows
for IDE-like behavior is an exceedingly complex task, one that's
impossible without good command of ELisp.  Not something I'd offer a
user whose only wish is to build a project in some language we
support.

We cannot provide a toolbox and tell users make your own IDE from
these tools.  Integrating those tools together is a non-trivial task a
user should not be expected to perform.

Re: IDE

[Eli Zaretskii]

> From: David Kastrup <dak@gnu.org>
> Date: Sat, 10 Oct 2015 20:46:13 +0200
> 
> Emacs sucks at multiple-language buffers out of the box.  They are
> inherent for any form of literate programming, they are rather prevalent
> for embedding languages in HTML, they are useful for things like Texinfo
> with embedded examples, I'd need them for LilyPond (which integrates
> Scheme code).  Many IDEs are single-language to start with, and if they
> aren't, they are either generic or single-language per file.
> 
> There is mmm-mode but it's not part of ELPA (and probably not
> copyright-assignable, at least that's my guess of why it might be out).
> And I strongly suspect that some stronger core features inside of Emacs
> would be desirable, like keeping indentation and font-locking engines
> and syntax constrained to regions identified with overlays or text
> properties.

I indeed think that we should have infrastructure to turn on a major
mode in a region of a buffer.

I'm not sure we should use text properties or overlays for that,
though.  The region could be part of the command that turns on the
mode with region limits stored in markers.

Re: IDE

[David Kastrup]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: "John Wiegley" <johnw@newartisans.com>
>> Date: Sat, 10 Oct 2015 11:31:13 -0700
>> 
>> To reiterate: I think Emacs becomes more of an IDE when it provides
>> the backbone of an IDE, and not when it looks like one.
>
> I don't think this will fly.  In effect, you are telling Emacs users
> they will have to code their own IDE by using the infrastructure
> (a.k.a. "backbone") that we provide.

I don't see that.  He is telling Emacs developers they'll be able to
rely on common infrastructure for creating an IDE catered to some
particular language/system.

If we are talking about smart completion and debugging support for
languages compiled with GCC, this could mean a lot of things working out
of the box (source-level debugging with expression
evaluation/inspection, syntactic indentation, jumping by statements or
other logical units, semantic information and completion etc etc) before
you even know the language we are talking about.

> To me, an IDE is not a set of functionalities.  It's a coherent
> application that provides an IDE-like look-and-feel, and all the
> related functions already integrated and ready for me to be used.

Without a uniform framework to create such IDEs, no uniformly usable IDE
for different languages will fall out.

> We cannot provide a toolbox and tell users make your own IDE from
> these tools.

It wouldn't be the task of the users to create those.

> Integrating those tools together is a non-trivial task a user should
> not be expected to perform.

Forcing every developer to redo it from scratch will lead to very spotty
and inconsistent support of languages.

-- 
David Kastrup

Re: IDE

[David Kastrup]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: David Kastrup <dak@gnu.org>
>> Date: Sat, 10 Oct 2015 20:46:13 +0200
>> 
>> Emacs sucks at multiple-language buffers out of the box.  They are
>> inherent for any form of literate programming, they are rather prevalent
>> for embedding languages in HTML, they are useful for things like Texinfo
>> with embedded examples, I'd need them for LilyPond (which integrates
>> Scheme code).  Many IDEs are single-language to start with, and if they
>> aren't, they are either generic or single-language per file.
>> 
>> There is mmm-mode but it's not part of ELPA (and probably not
>> copyright-assignable, at least that's my guess of why it might be out).
>> And I strongly suspect that some stronger core features inside of Emacs
>> would be desirable, like keeping indentation and font-locking engines
>> and syntax constrained to regions identified with overlays or text
>> properties.
>
> I indeed think that we should have infrastructure to turn on a major
> mode in a region of a buffer.
>
> I'm not sure we should use text properties or overlays for that,
> though.  The region could be part of the command that turns on the
> mode with region limits stored in markers.

A typical LilyPond score file switches into Scheme thousands of times
(most of the time just for a single scalar Scheme constant where an
actual mode switch would not necessarily be required, but also for all
user-defined functions and any non-trivial non-music expression).

-- 
David Kastrup

Re: IDE

[Eli Zaretskii]

> From: David Kastrup <dak@gnu.org>
> Cc: John Wiegley <johnw@newartisans.com>,  emacs-devel@gnu.org
> Date: Sat, 10 Oct 2015 21:07:50 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> From: "John Wiegley" <johnw@newartisans.com>
> >> Date: Sat, 10 Oct 2015 11:31:13 -0700
> >> 
> >> To reiterate: I think Emacs becomes more of an IDE when it provides
> >> the backbone of an IDE, and not when it looks like one.
> >
> > I don't think this will fly.  In effect, you are telling Emacs users
> > they will have to code their own IDE by using the infrastructure
> > (a.k.a. "backbone") that we provide.
> 
> I don't see that.  He is telling Emacs developers they'll be able to
> rely on common infrastructure for creating an IDE catered to some
> particular language/system.

OK, but IMO we should aim for more than that.  What John wrote sounded
like providing the infrastructure _is_ the goal.  Apologies if I
misunderstood.

Re: IDE

[Eli Zaretskii]

> From: David Kastrup <dak@gnu.org>
> Cc: emacs-devel@gnu.org
> Date: Sat, 10 Oct 2015 21:11:57 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > I indeed think that we should have infrastructure to turn on a major
> > mode in a region of a buffer.
> >
> > I'm not sure we should use text properties or overlays for that,
> > though.  The region could be part of the command that turns on the
> > mode with region limits stored in markers.
> 
> A typical LilyPond score file switches into Scheme thousands of times
> (most of the time just for a single scalar Scheme constant where an
> actual mode switch would not necessarily be required, but also for all
> user-defined functions and any non-trivial non-music expression).

Not sure what you are saying.  Is it that a region with two ends is
not enough, and we need a list of regions?  If so, I'm okay with that,
and I don't see any serious obstacles to implement that.

Re: IDE

[John Wiegley]

>>>>> Eli Zaretskii <eliz@gnu.org> writes:

>> From: David Kastrup <dak@gnu.org>
>> 
>> Emacs sucks at multiple-language buffers out of the box.

> I indeed think that we should have infrastructure to turn on a major
> mode in a region of a buffer.

I strongly agree as well.  TextMate does an awesome job at this.

> I'm not sure we should use text properties or overlays for that,
> though.  The region could be part of the command that turns on the
> mode with region limits stored in markers.

I'm not sure what the right technical answer is either, but I'll add this to
the list.

John

Re: IDE

[John Wiegley]

>>>>> Eli Zaretskii <eliz@gnu.org> writes:

>> >> To reiterate: I think Emacs becomes more of an IDE when it provides
>> >> the backbone of an IDE, and not when it looks like one.
>> >
>> > I don't think this will fly.  In effect, you are telling Emacs users
>> > they will have to code their own IDE by using the infrastructure
>> > (a.k.a. "backbone") that we provide.
>> 
>> I don't see that.  He is telling Emacs developers they'll be able to
>> rely on common infrastructure for creating an IDE catered to some
>> particular language/system.

> OK, but IMO we should aim for more than that.  What John wrote sounded
> like providing the infrastructure _is_ the goal.  Apologies if I
> misunderstood.

Ah, I should have been clearer, Eli. What I wrote was addressed to Emacs
developers. I think we have succeeded *towards* having an IDE when we have the
right backbone in place.

Next comes the package contributors, who will build out functionality on top
of this backbone. If we do it right, it will mostly mean porting the backends
of existing libraries that already do a superb job, like company and helm.

Finally comes knitting it all together into a good user experience. This takes
some art, but we have a few examples where this has been done right already.

At the end of that road, we'd have an excellent IDE environment that a
non-Emacs user could intuitively appreciate, the way so many do with Org or
Spacemacs (each of which has won new Emacs users on its own, before those
users understood what Emacs was really about).

John

RE: IDE

[Drew Adams]

> Not sure what you are saying.  Is it that a region with two ends is
> not enough, and we need a list of regions?  If so, I'm okay with
> that, and I don't see any serious obstacles to implement that.

FWIW, library `zones.el' provides that, along with various
commands for defining and using such multiple-zone "regions".

http://www.emacswiki.org/emacs/Zones

You can sort, intersect, or unite zones; narrow the buffer to
a zone or select one as the region, (un)highlight zones, make
a list of zones persistent (re-created with a bookmark), or
use incremental search across it (overlapping zones in the list
are united for this).  You can name lists of zones and switch
among them for different multi-zone operations.  You can
associate arbitrary information with a zone, which is otherwise
just a pair of buffer positions (e.g. markers).

This does _not_ provide the feature of having zones that are
in different major modes.  I agree that that particular
feature is sorely missing.  There have been various attempts
at providing it, but I'm not aware of any that has been really
satisfactory in a general way.

Re: IDE

[David Engster]

Eli Zaretskii writes:
>> From: David Engster <deng@randomsample.de>
>> Cc: Dmitry Gutov <dgutov@yandex.ru>,  adatgyujto@gmail.com,  emacs-devel@gnu.org
>> Date: Sat, 10 Oct 2015 11:51:14 +0200
>> 
>> Eli Zaretskii writes:
>> > And if anyone _really_ cares about supporting C/C++, they should be
>> > working with and on GCC's libcc1,
>> 
>> Getting the AST is not a technical problem; libcc1 does not change
>> anything.
>
> We don't even know whether libcc1 provides any features useful for an
> IDE, including information that could be used fro completion.

AFAICS, libcc1 does not provide anything new. It seems to be some kind
of library wrapper for a plugin to access gcc's frontend. It's not even
mentioned in gcc's release notes.

> We also don't know what is the final verdict on having GCC emit tree
> information.

Any year now.

> I'd expect a person who is motivated to work on these features, if we
> have such a person among us, to be on top of those two issues, as well
> as many others.
>
> But I agree: it's not a technical problem.  It's a problem of being
> motivated enough to overcome such obstacles.

Yes, I should've stuck with clang in the first place.

-David

Re: IDE

[David Kastrup]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: David Kastrup <dak@gnu.org>
>> Cc: emacs-devel@gnu.org
>> Date: Sat, 10 Oct 2015 21:11:57 +0200
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> > I indeed think that we should have infrastructure to turn on a major
>> > mode in a region of a buffer.
>> >
>> > I'm not sure we should use text properties or overlays for that,
>> > though.  The region could be part of the command that turns on the
>> > mode with region limits stored in markers.
>> 
>> A typical LilyPond score file switches into Scheme thousands of times
>> (most of the time just for a single scalar Scheme constant where an
>> actual mode switch would not necessarily be required, but also for all
>> user-defined functions and any non-trivial non-music expression).
>
> Not sure what you are saying.  Is it that a region with two ends is
> not enough, and we need a list of regions?  If so, I'm okay with that,
> and I don't see any serious obstacles to implement that.

It's rather a host of regions determined by syntax.

This can look like:

compressMMRests =
#(define-music-function (music) (ly:music?)
  (_i "Remove the empty bars created by multi-measure rests,
leaving just the first bar containing the MM rest itself.")
   (music-map
    (lambda (m)
      (if (eq? 'MultiMeasureRestMusic (ly:music-property m 'name))
          #{ \once \set Score.skipBars = ##t #m #}
          #{ #m #} ))
    music))

The first # switches into Scheme mode until the end, but then every #{
escape backs into LilyPond mode, and every # inside until #} goes back
to Scheme in order to evaluate expressions #t, m, and m in Scheme again.

Basically, inside of LilyPond, # escapes into Scheme for a single sexp,
and inside of Scheme, #{ escapes into LilyPond until the matching #} is
encountered, of course with #sexp again escaping into Scheme.  Either
can easily form multi-line expressions (or not) but definitely wants to
change the mode of font-locking.

-- 
David Kastrup

Re: IDE

[Dmitry Gutov]

On 10/10/2015 09:58 PM, Eli Zaretskii wrote:

> To me, an IDE is not a set of functionalities.  It's a coherent
> application that provides an IDE-like look-and-feel, and all the
> related functions already integrated and ready for me to be used.
> That includes window-layout, btw, because configuring Emacs windows
> for IDE-like behavior is an exceedingly complex task, one that's
> impossible without good command of ELisp.  Not something I'd offer a
> user whose only wish is to build a project in some language we
> support.

While I agree that working with windows in Emacs is often more trouble 
than it should be, I don't think that offering a fixed layout like ECB 
is the answer: it doesn't anticipate the needs of commands like vc-dir, 
and it doesn't solve all problems anyway.

Rather than that, we should provide more consistent guidelines for 
window behavior, like whether a command should use a new window, reuse 
an existing one, etc, and try harder not to destroy a layout the user 
created. Maybe include a more accessible alternative to winner-mode 
(which is a lifesaver, but is more of a kludge than a user-friendly 
solution).

Re: IDE

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 1648 bytes --]

On 10/10/2015 02:23 PM, Dmitry Gutov wrote:
> On 10/10/2015 09:58 PM, Eli Zaretskii wrote:
> 
>> To me, an IDE is not a set of functionalities.  It's a coherent
>> application that provides an IDE-like look-and-feel, and all the
>> related functions already integrated and ready for me to be used.
>> That includes window-layout, btw, because configuring Emacs windows
>> for IDE-like behavior is an exceedingly complex task, one that's
>> impossible without good command of ELisp.  Not something I'd offer a
>> user whose only wish is to build a project in some language we
>> support.
> 
> While I agree that working with windows in Emacs is often more trouble
> than it should be, I don't think that offering a fixed layout like ECB
> is the answer: it doesn't anticipate the needs of commands like vc-dir,
> and it doesn't solve all problems anyway.
> 
> Rather than that, we should provide more consistent guidelines for
> window behavior, like whether a command should use a new window, reuse
> an existing one, etc, and try harder not to destroy a layout the user
> created. Maybe include a more accessible alternative to winner-mode
> (which is a lifesaver, but is more of a kludge than a user-friendly
> solution).

A pet peeve of mine is pop-to-buffer. I've been tempted on a few
occasions to make pop-to-buffer equivalent to display-buffer and use
buffer-display customization for all the differences.

I'm also a heavy winner user, but I'd appreciate the ability to "dock"
certain windows and buffers within them. (I haven't found the dedicated
window functionality useful, but _somebody_ must, right?)


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: IDE

[Eric Ludlam]

On 10/10/2015 02:31 PM, John Wiegley wrote:
> If we have a single paradigm for "determining interesting details about the
> buffer, and near point", with the ability to refine the query based on what is
> need, optionally cache results, etc., then the competing libraries we have
> today could share functionality. The present day `all-completions` function is
> too spare to fit this bill, so it's less of an IDE feature in my book and more
> just a Lisp library function.

In CEDET, the function / command `semantic-analyze-current-context' 
provides an output that has lots of details about the buffer near point. 
  Not just what the cursor is on, but if it is a chain of symbols such 
as dereferencing struct pointers, and in many cases, it figures out the 
data type of the symbol the cursor is on.  It also handles in-buffer 
caching, etc and plenty of performance tweaking is available.

This is independent of the functions that perform completions.

Eric

Re: IDE

[John Wiegley]

>>>>> Eric Ludlam <eric@siege-engine.com> writes:

> In CEDET, the function / command `semantic-analyze-current-context' provides
> an output that has lots of details about the buffer near point. Not just
> what the cursor is on, but if it is a chain of symbols such as dereferencing
> struct pointers, and in many cases, it figures out the data type of the
> symbol the cursor is on. It also handles in-buffer caching, etc and plenty
> of performance tweaking is available.

My preference is for each core feature to have an extremely simple and light
interface (taking indentation as an ideal), so that it can also be used for
non-IDE tasks we haven't imagined yet. From what I know about CEDET to date,
it is much more complex than this objective.

When I squint, I see ctags, imenu, pcomplete, helm, company, hippie-expand,
flycheck, and more, all starting to look somewhat alike: They each act upon
information relevant to the buffer in some way. Where they differ is in how
they derive this information, and how the user interacts with it. Can we
provide a common, low-level interface for this style of functionality? A
common API against which we can implement both data-gathering backends, and
display front-ends? And with an emphasis on efficiency and low-latency!

We need to harness the power of multiplication, so that every new backend
makes every frontend automatically more powerful, and vice versa. This will
also help us better leverage our existing base of contributors.

John

Re: IDE

[raman]

"John Wiegley" <johnw@newartisans.com> writes:

Well said. Let's focus on functionality -- and the rest -- including a
multiplicity of visual look-and-feel setups will follow along with
everything else.>>>>>> martin rudalics <rudalics@gmx.at> writes:
>
>> I never use side windows so I can't tell whether they still work. I have
>> written a frame-tabs.el package based on side windows but I don't use that
>> either. At the time I installed the side windows functions I also added a
>> texinfo section but Stefan later asked me to remove it. That info does not
>> reflect later changes to the code so it might be outdated. You have to live
>> with the doc-strings which should be fairly accurate.
>
> We should define what we want from a "more IDE" Emacs. For example, I do not
> want window-layout functionality. That's a presentation aspect of IDEs that's
> entirely separate from what I want from them.
>
> Right now we have a pretty nice infrastructure for things like indenting code.
> That is, there are standard keybindings (TAB, C-M-\), a standard per-buffer
> override variable (indent-line-function), a standard command
> (indent-according-to-mode), and ways for packages like Yasnippet to override
> the meaning of TAB without ruining functionality.
>
> I think that what an "IDE is" has little to do with what it looks like. Emacs
> being a better IDE, to me, means making IDE-like functionality a first-class
> citizen, as we do with indenting. This would provide a core for fancy display
> modules to build on top of.
>
> I also don't think core Emacs should *provide* this functionality, since
> that's impossible, given how many different languages and use cases there are.
> It's more about giving developers a common API to base their work on, so that
> we maximize collaboration between them.
>
> Here is a list of functionality I think should be first-class, structurally
> speaking (that is, an API will exist, but it may not do anything until a
> contributor implements the functionality for their language). The ones with
> a * mean areas where we're already succeeding to some degree:
>
>   * indentation (see above)
>     reformatting
>   * syntax highlighting (font-lock)
>     help at point
>     documentation lookup (sadly, fewer projects use Info these days)
>   ? completion
>     refactoring
>     semantic editing (for example, paredit)
>   * compilation (M-x compile)
>     live compilation (flymake/flycheck)
>   * REPL (comint)
>     running tests
>   * debugging (GUD)
>   * version control (VC)
>     profiling
>     code coverage
>     app interaction
>
> The reason I don't have a * next to completion, despite having so many things
> capable of doing it (company, auto-complete, ido, hippie-expand, helm, ivy,
> etc., etc.), is that there are too MANY ways to do it. This is where I think
> proper IDE support could help.
>
> If we have a single paradigm for "determining interesting details about the
> buffer, and near point", with the ability to refine the query based on what is
> need, optionally cache results, etc., then the competing libraries we have
> today could share functionality. The present day `all-completions` function is
> too spare to fit this bill, so it's less of an IDE feature in my book and more
> just a Lisp library function. For example, I've written nearly the same
> backend code for at least 4 different completion/lookup packages, and each
> time I wonder how we could do better here. The differences are so minimal.
>
> To reiterate: I think Emacs becomes more of an IDE when it provides the
> backbone of an IDE, and not when it looks like one. We have some pieces of
> that backbone already, which have avoided fragmentation in those areas, but we
> need more. A standardized way to do tooltips, popups, visual completion
> selection (or at least the structure), REPLs that interact with buffer
> contents, etc., would give us a foundation to move to the next step.
>
> John
>

-- 

Re: IDE

[Dmitry Gutov]

Hi Eric,

On 10/10/2015 07:48 PM, Eric Ludlam wrote:

> I had always intended CEDET to be a baseline for IDE like features.
> Looking just at tagging files, those familiar with it's internals
> recognize that it can use external tools as weak as ctags or GLOBAL, and
> can use a more powerful external tool for parsing as well, such as using
> JAVAC for decompiling .jar files into tags.

It sounds good if all I have is a parser/tagger.

But if I already have an external tool that can give me a set of 
completions at a given position in a given buffer, and also provides a 
go-to-definition feature, what's the advantage of going through 
Semantic? SRecode support?

Not to mention that we have tools like Jedi that only (mostly) do these 
two things, but don't provide dumps of the syntax tree.

> As a backup, it also has
> in-buffer parsing for when you've only half-written your code, or when
> no external tool is available.

That assumes we also have a Semantic grammar for the said language. And 
then, we'll need to duplicate whatever logic is used by the external 
tool to disambiguate between same-named methods in different classes.

If that's even possible.

> The same philosophy is throughout CEDET.  Unfortunately, those who are
> interested in tooling who brush casually past CEDET only see that people
> complain that it doesn't always complete right, or that the C++ part is
> hard to setup

The example we've been given in one of the previous discussions of C++ 
support is that Semantic can't tell the difference between different 
methods with the same name? If Z#foo returns type A, and T#foo returns 
type B, it's been shown that Semantic doesn't know which is the type 
'x.foo'.

Given this, how would we expect it to properly support languages with 
type inference like Scala or Go? Not to mention dynamic languages which 
normally have no type annotations on methods, and often use more 
complicated algorithms for accurate code completion.

> If we look at the two starting key features listed for IDEs,
> "completion" and "refactoring", the basics are all there.  CEDET has a
> completion engine, but it is only as good as the input data.

It's limited by the input data, sure, but it also has to know what to do 
with it.

If only implementing accurate code completion was as easy as writing a 
language parser.

> CEDET also has 'srecode' which is about
> generating code.  This bit is trickier, as the intention is you could
> write one highlevel tool that might extract tag data from your file,
> permute the language-independent tags, and then write them back out with
> srecode.  There are tests showing this working, but they are simple and
> proof of concept and not stuff someone would depend on day-to-day.

SRecode should be in a better position, but I imagine it would also fail 
often enough in dynamic languages.

And in any case, one has to teach Semantic about scoping rules for each 
given language, for SRecode to only rename, say, a variable 'foo' but 
not function 'foo', or only the variable 'bar', but not the variable 
'bar' up the scope that's being shadowed.

Re: IDE

[Andreas Röhler]

Am 10.10.2015 um 23:23 schrieb Dmitry Gutov:
> On 10/10/2015 09:58 PM, Eli Zaretskii wrote:
>
>> To me, an IDE is not a set of functionalities.  It's a coherent
>> application that provides an IDE-like look-and-feel, and all the
>> related functions already integrated and ready for me to be used.
>> That includes window-layout, btw, because configuring Emacs windows
>> for IDE-like behavior is an exceedingly complex task, one that's
>> impossible without good command of ELisp.  Not something I'd offer a
>> user whose only wish is to build a project in some language we
>> support.
>
> While I agree that working with windows in Emacs is often more trouble 
> than it should be, I don't think that offering a fixed layout like ECB 
> is the answer: it doesn't anticipate the needs of commands like 
> vc-dir, and it doesn't solve all problems anyway.
>
> Rather than that, we should provide more consistent guidelines for 
> window behavior, like whether a command should use a new window, reuse 
> an existing one, etc, and try harder not to destroy a layout the user 
> created. Maybe include a more accessible alternative to winner-mode 
> (which is a lifesaver, but is more of a kludge than a user-friendly 
> solution).
>
>

AFAIU Emacs in past time changed too fast, introduced new features or 
new function without an appropriate discussion, broke backward code too 
fast.

Got the impression developers spend a lot of time with fruits of their 
colleages, which just weren't in time to sell it yet.

That's rather an impression than established knowledge.

In case its true, a policy focused on branches while feature-freeze 
being the normal state might help.

Re: IDE

[Dmitry Gutov]

On 10/10/2015 05:25 PM, Eli Zaretskii wrote:

> It's standard software engineering practice, why should you ask for
> its justification?

I'm asking for details. Again, to further the discussion.

"Let's unify features X, Y and Z" is not necessarily the standard 
practice. It all depends on the exact features and how they are used.

I've also given a few general reasons why a "big design" might be 
suboptimal: more engineering effort needed, the result is likely to turn 
out to be less flexible, and a significant number of users might prefer 
to use only some of the parts.

For instance, because we don't provide support for some feature in their 
environment (e.g. refactoring for Ruby), or because they already have a 
satisfactory solution for the feature in question. And as you know, some 
users dislike changing their habits.

> They look nicer.  I don't know how to explain better.

I would expect something akin to a normal bug report (enumerating things 
we're missing). But never mind.

> The other IDEs use something similar to a tooltip, or a drop-down menu
> with different fonts and colors.

You can already customize colors and fonts user for the Company popup. 
But if you end up using fonts with different dimensions, of course, that 
would result in jagged display.

I'd be happy to use a "native" tooltip in Emacs, but didn't have time to 
fully investigate it yet. However, there's a third-party package called 
pos-tip which tries to provide a "show tooltip" interface based on 
x-show-tip. From trying it out, I have the following complaints about 
x-show-tip capabilities:

- It's background rendering is inconsistent. As an example, the first 
time I evaluate (tooltip-show "abc") in an Emacs session, the background 
is yellow-ish. The next time, and after that, the background is black.
- Is there a way to show several tooltips at once? To display different 
elements of the completion UI side by side.
- If a tooltip is displayed, and I Alt-Tab to another program's window, 
the tooltip remains on top. This is by far the most annoying one.

Re: IDE

[martin rudalics]

[-- Attachment #1: Type: text/plain, Size: 698 bytes --]

 > - It's background rendering is inconsistent. As an example, the first
 > time I evaluate (tooltip-show "abc") in an Emacs session, the
 > background is yellow-ish. The next time, and after that, the
 > background is black.

This is the reason I'm always using ‘x-show-tip’.

 > - Is there a way to show several tooltips at once? To display different elements of the completion UI side by side.

AFAICT no.  You'd have to put them into the same tooltip.

 > - If a tooltip is displayed, and I Alt-Tab to another program's window, the tooltip remains on top. This is by far the most annoying one.

Very annoying, indeed.  See my solution for this in
‘eldoc-tooltip-mode’.

martin

[-- Attachment #2: eldoc-tooltip.el --]
[-- Type: application/emacs-lisp, Size: 21418 bytes --]

Re: IDE

[Dmitry Gutov]

On 10/10/2015 08:52 PM, martin rudalics wrote:

> Cedet does have ‘semantic-displayor-tooltip-mode’.

Could you explain how to try it out?

Re: IDE

[martin rudalics]

> Could you explain how to try it out?

No. I just noticed it when I started to use tooltips for ElDoc.

martin

Re: IDE

[Dmitry Gutov]

On 10/11/2015 01:25 PM, martin rudalics wrote:
>> Could you explain how to try it out?
>
> No. I just noticed it when I started to use tooltips for ElDoc.

Too bad.

I tried (setq semantic-complete-inline-analyzer-displayor-class 
'semantic-displayor-tooltip), with no apparent result.

Re: IDE

[Dmitry Gutov]

Thanks! I've tried your package out, and it's pretty nice. Especially 
the part where you carefully try to position the popup so it doesn't 
hide the code.

On 10/11/2015 01:17 PM, martin rudalics wrote:
>  > - It's background rendering is inconsistent. As an example, the first
>  > time I evaluate (tooltip-show "abc") in an Emacs session, the
>  > background is yellow-ish. The next time, and after that, the
>  > background is black.
>
> This is the reason I'm always using ‘x-show-tip’.

That doesn't fix the problem. Even with your package, it looked fine 
with greenish background for a while, then I switched away from Emacs, 
did some reading in the web browser. Then switched back to Emacs, and 
the background is black now.

>  > - Is there a way to show several tooltips at once? To display
> different elements of the completion UI side by side.
>
> AFAICT no.  You'd have to put them into the same tooltip.

I think I'll hold off on trying on integrate it in Company until this 
feature is implemented.

For feature parity with Intellij IDEA and MS VS, we should be able to 
display the list of completions and documentation for the currently 
selected completion in two separate popups:

https://i-msdn.sec.s-msft.com/dynimg/IC797655.jpeg
https://www.jetbrains.com/img/webhelp/idea/constructors_docs_in_completion.png

In Company (as well as the IDEA interface), "show documentation" is a 
separate action which can take some perceptible time, so we can't 
include the documentation in the same popup as completions.

>  > - If a tooltip is displayed, and I Alt-Tab to another program's
> window, the tooltip remains on top. This is by far the most annoying one.
>
> Very annoying, indeed.  See my solution for this in
> ‘eldoc-tooltip-mode’.

focus-out-hook? That'll be good enough, thanks.

Re: IDE

[martin rudalics]

 > That doesn't fix the problem. Even with your package, it looked fine
 > with greenish background for a while, then I switched away from Emacs,
 > did some reading in the web browser. Then switched back to Emacs, and
 > the background is black now.

Are you by any chance using GTK tooltips?  They are a pain.

 >>  > - Is there a way to show several tooltips at once? To display
 >> different elements of the completion UI side by side.
 >>
 >> AFAICT no.  You'd have to put them into the same tooltip.
 >
 > I think I'll hold off on trying on integrate it in Company until this
 > feature is implemented.

Maybe we should just use simple frames with all decorations removed.
The timeout to unshow the frame is not useful anyway and to remove the
overhead for frame creation it would suffice to make the frame invisible
as long as it's not needed.

martin

Re: IDE

[Dmitry Gutov]

On 10/11/2015 02:38 PM, martin rudalics wrote:

> Are you by any chance using GTK tooltips?  They are a pain.

I've built Emacs with GTK 3, if that's what you mean. The configure 
script does that by default here.

> Maybe we should just use simple frames with all decorations removed.

How do I do that?

> The timeout to unshow the frame is not useful anyway and to remove the
> overhead for frame creation it would suffice to make the frame invisible
> as long as it's not needed.

The timeout is not useful indeed.

Re: IDE

[martin rudalics]

 >> Are you by any chance using GTK tooltips?  They are a pain.
 >
 > I've built Emacs with GTK 3, if that's what you mean. The configure script does that by default here.

Try setting ‘x-gtk-use-system-tooltips’ to nil.

 >> Maybe we should just use simple frames with all decorations removed.
 >
 > How do I do that?

By making a frame without minibuffer, title, scroll, tool, menu bars and
borders.  Maybe it can't be done in Lisp alone.

martin

Re: IDE

[David Engster]

Dmitry Gutov writes:
> On 10/11/2015 01:25 PM, martin rudalics wrote:
>>> Could you explain how to try it out?
>>
>> No. I just noticed it when I started to use tooltips for ElDoc.
>
> Too bad.
>
> I tried (setq semantic-complete-inline-analyzer-displayor-class
> 'semantic-displayor-tooltip), with no apparent result.

Make sure you call semantic-complete-analyze-inline to display
completions.

Personally, I don't use the tooltip displayor. I don't know if this is a
general problem or one with the implementation in CEDET, but I find them
too slow, and they don't play well with keyboard navigation.

-David

Re: IDE

[Dmitry Gutov]

On 10/11/2015 03:16 PM, David Engster wrote:

>> I tried (setq semantic-complete-inline-analyzer-displayor-class
>> 'semantic-displayor-tooltip), with no apparent result.
>
> Make sure you call semantic-complete-analyze-inline to display
> completions.

Tried that. No dice.

> Personally, I don't use the tooltip displayor. I don't know if this is a
> general problem or one with the implementation in CEDET, but I find them
> too slow, and they don't play well with keyboard navigation.

I thought tooltips don't affect the keyboard input?

Re: IDE

[David Engster]

martin rudalics writes:
>>> Are you by any chance using GTK tooltips?  They are a pain.
>>
>> I've built Emacs with GTK 3, if that's what you mean. The configure
>> script does that by default here.
>
> Try setting ‘x-gtk-use-system-tooltips’ to nil.
>
>>> Maybe we should just use simple frames with all decorations
>>> removed.
>>
>> How do I do that?
>
> By making a frame without minibuffer, title, scroll, tool, menu bars
> and
> borders.  Maybe it can't be done in Lisp alone.

Here's what I use in my doc-present package[1] for displaying slides:

(with-selected-frame
    (make-frame '((minibuffer . nil)
		  (left-fringe . 0)
		  (right-fringe . 0)
		  (menu-bar-lines . 0)
		  (internal-border-width . 0)
		  (vertical-scroll-bars . nil)
		  (unsplittable . t)
		  (cursor-type . nil)
		  (tool-bar-lines . 0)))
  (pop-to-buffer (get-buffer-create "*test*"))
  (setq mode-line-format nil))

The only thing remaining are the decorations from the window manager. I
don't think you can get rid of those from within Emacs?

-David

Re: IDE

[David Engster]

Dmitry Gutov writes:
> On 10/11/2015 03:16 PM, David Engster wrote:
>
>>> I tried (setq semantic-complete-inline-analyzer-displayor-class
>>> 'semantic-displayor-tooltip), with no apparent result.
>>
>> Make sure you call semantic-complete-analyze-inline to display
>> completions.
>
> Tried that. No dice.

Works for me from 'emacs -Q' (emacs 24.5, lucid toolkit):

- M-x semantic-mode
- eval '(setq semantic-complete-inline-analyzer-displayor-class 'semantic-displayor-tooltip)'
- Load empty file 'test.c' and insert

void foo();
void foo2();

int main()
{
  fo
}

- Put cursor behind 'fo' and do M-x semantic-complete-analyze-inline.

>> Personally, I don't use the tooltip displayor. I don't know if this
>> is a
>> general problem or one with the implementation in CEDET, but I find
>> them
>> too slow, and they don't play well with keyboard navigation.
>
> I thought tooltips don't affect the keyboard input?

I mean I cannot choose completions with the keyboard with up/down. But
this is a problem with how it is done in CEDET, which was never my cup
of tea, but I also didn't work on it because as I said, I found it to be
too slow to display in the first place (that was years ago though, and I
guess it also depends on the toolkit, plus I often times worked over a
remote X connection).

-David

Re: IDE

[martin rudalics]

 > The only thing remaining are the decorations from the window manager. I
 > don't think you can get rid of those from within Emacs?

 From Lisp I suppose only via the tooltip interface.  Also one has to
make sure that the frame always stays on top of the Z-order.

martin

Re: IDE

[Dmitry Gutov]

On 10/11/2015 03:37 PM, David Engster wrote:

> Works for me from 'emacs -Q' (emacs 24.5, lucid toolkit):

Thanks, this sequence works for me too, even on master, but only with 
'-Q'. So something in my config is at fault.

>> I thought tooltips don't affect the keyboard input?
>
> I mean I cannot choose completions with the keyboard with up/down. But
> this is a problem with how it is done in CEDET, which was never my cup

Right, CEDET would have to handle keyboard input somehow, for it to work 
better.

Re: IDE

[Przemysław Wojnowski]

W dniu 10.10.2015 o 11:40, Eli Zaretskii pisze:
> Come to think of that, even coming up with a list of such features
> would be real progress, as I don't think we have that, or ever had.

I can "step forward and work" conceptually here. :-)

One can start with a question:
What is purpose of an IDE (except for making many for a company behind it)?
The simple answer is: to help programmers do their work productively.

The answer tells two things:
1. Who is the target: programmers not average computer user
It could be possibly constrained to e.g. programmers working on projects, not 
single file programs.
2. How to optimize product backlog: maximization of productivity.

Both these things help to select features.
For example project-oriented programmers _have to have_ project support, whereas 
"single file" programmers may not care about it.
Features/bugs that impact their productivity have higher priority.

Let's say that we would target "project programmers", then we could ask:
1. What features would make such programmers productive?
2. What other environments offer that attract them?

IMHO the answer is: it depends on target language(s).

I cannot say what makes a Ruby programmer productive, but I can list a few 
things important for Java programmers. Others, having experience with other 
languages, may list _must have_ features for their environments.

For (project-oriented/enterprise) Java the features are:
1. Project support
IDE has to know the boundaries of a project - where are sources, tests, libs, 
resources/assets (additional files used in an app in runtime), docs - and what 
are relations between them. Also it has to know how to work with project's build 
tool (be it Maven, Gradle, etc.).
A programmer joining a project (99% of cases) should be able to open/import it 
and start work. Every Java IDE have that.

2. Code completion
 From whole project, used libraries, and resources

3. Jumping around the project code and resources.
Jumping to around the project code and used libraries. Another thing is jumping 
to/from resources (for example aspects can be defined in an XML file and IDE 
could allow to jump to matching classes).

4. Finding usages of fields, methods, types.
Helps to wrap head around project.

5. Automatic compilation and showing compilation erros/warnings.
Tightens development feedback loop.

6. Running the tests (current, selected, all)
Tighten development feedback loop.

7. Debugging
A must, especially for legacy code, so, basically 99% of projects. :-)

8. Running the app

9. Basic refactoring.
I do refactor _a lot_ and in my experience the most important refactoring is 
Extract Method. Others, while helpful, are less often used, compared to the EM.
One variation of Extract Method is "EM with finding duplicates", which works 
like this:
- ask user for a method name,
- find all occurrences of selected code in the buffer
- ask user if she wants to replace all occurrences with the call to the new method.
This is fantastic feature that Intellij has and helps to remove a lot of 
duplicated code.

10. Showing documentation (tooltip, etc.)
Maybe EWW could be used to show javadoc in a small window?


Of course, part of these things (e.g. build tool support) are language dependent 
and should not be in Emacs core.

Also some of the features are already scattered in different elpa libraries, but 
lack integration with others.

IMHO people having experience in other languages could list features that make 
them productive and maybe we will be able to find a set of core, absolutely must 
have, IDE features. :-)

IMHO the list of core IDE features will be fairly small and doable.

Sorry for long email.

Cheers,
Przemysław

Re: IDE

[Jean-Christophe Helary]

> On Oct 11, 2015, at 22:18, Przemysław Wojnowski <esperanto@cumego.com> wrote:
> 
> W dniu 10.10.2015 o 11:40, Eli Zaretskii pisze:
>> Come to think of that, even coming up with a list of such features
>> would be real progress, as I don't think we have that, or ever had.
> 
> I can "step forward and work" conceptually here. :-)
> 
> One can start with a question:
> What is purpose of an IDE (except for making many for a company behind it)?
> The simple answer is: to help programmers do their work productively.

Productivity for code *writers* could be defined by how many correct code strings they output in a given amount of time.

Anything that

1) increase the volume
2) increase the correctness

of the strings increases productivity.

Jean-Christophe Helary 

Re: IDE

[Przemysław Wojnowski]

W dniu 11.10.2015 o 15:26, Jean-Christophe Helary pisze:
> Productivity for code *writers* could be defined by how many correct code strings they output in a given amount of time.
>
> Anything that
>
> 1) increase the volume
> 2) increase the correctness
>
> of the strings increases productivity.

I like jokes, but adding your "list of features" would be great too. ;-)

Re: IDE

[Jean-Christophe Helary]

> On Oct 11, 2015, at 22:34, Przemysław Wojnowski <esperanto@cumego.com> wrote:
> 
> W dniu 11.10.2015 o 15:26, Jean-Christophe Helary pisze:
>> Productivity for code *writers* could be defined by how many correct code strings they output in a given amount of time.
>> 
>> Anything that
>> 
>> 1) increase the volume
>> 2) increase the correctness
>> 
>> of the strings increases productivity.
> 
> I like jokes, but adding your "list of features" would be great too. ;-)

It was not intended as a joke :) But if we start by defining what we mean by "productivity" in the context of code writing, then it's easier to see what can be on the feature list and what does not need to be.

Also, the above definition applies equally to any (non literary) writer. So instead of thinking IDE, you could think IWE and get perspectives that you would not have it you just focused on Emacs vs other IDEs.

Jean-Christophe 

Re: IDE

[Óscar Fuentes]

Jean-Christophe Helary <jean.christophe.helary@gmail.com> writes:

> Productivity for code *writers* could be defined by how many correct
> code strings they output in a given amount of time.
>
> Anything that
>
> 1) increase the volume
> 2) increase the correctness
>
> of the strings increases productivity.

I reckon you are a Java code writer.

<ducks>

Re: IDE

[Przemysław Wojnowski]

W dniu 11.10.2015 o 15:41, Jean-Christophe Helary pisze:
[...]
> It was not intended as a joke :) But if we start by defining what we mean by "productivity" in the context of code writing, then it's easier to see what can be on the feature list and what does not need to be.
Note one thing: I have written "to help programmers do their work productively".
It doesn't have to be "writing code". It can be analyzing code, learning it, 
refactoring, deleting code, running app/tests/debugging, etc.

Actually in many legacy systems writing code is minor activity.

In different projects productivity may be defined differently, but, where money 
counts, usually it's defined as "deliver software on time". Nobody care how many 
lines is that - the less, the better. If its oneliner and does the job then do 
it. :-)

Re: IDE

[Przemysław Wojnowski]

I forgot to mention one thing about Java projects: very often they are 
multi-language. Main code in Java, tests in e.g. Groovy, resources mix of HTML, 
JavaScript, XML, JSON. So, IDE needs to have configurations for different 
languages in the same project.

Re: IDE

[Jean-Christophe Helary]

> On Oct 11, 2015, at 22:59, Óscar Fuentes <ofv@wanadoo.es> wrote:
> 
> Jean-Christophe Helary <jean.christophe.helary@gmail.com> writes:
> 
>> Productivity for code *writers* could be defined by how many correct
>> code strings they output in a given amount of time.
>> 
>> Anything that
>> 
>> 1) increase the volume
>> 2) increase the correctness
>> 
>> of the strings increases productivity.
> 
> I reckon you are a Java code writer.

No, I'm a translator :)

Jean-Christophe 

Re: IDE

[Jean-Christophe Helary]

> On Oct 11, 2015, at 23:05, Przemysław Wojnowski <esperanto@cumego.com> wrote:
> 
> W dniu 11.10.2015 o 15:41, Jean-Christophe Helary pisze:
> [...]
>> It was not intended as a joke :) But if we start by defining what we mean by "productivity" in the context of code writing, then it's easier to see what can be on the feature list and what does not need to be.
> Note one thing: I have written "to help programmers do their work productively".
> It doesn't have to be "writing code". It can be analyzing code, learning it, refactoring, deleting code, running app/tests/debugging, etc.

That's what I'd put under "correctness".

> Actually in many legacy systems writing code is minor activity.

In which case your focus is on "correctness".

> In different projects productivity may be defined differently, but, where money counts, usually it's defined as "deliver software on time". Nobody care how many lines is that - the less, the better. If its oneliner and does the job then do it. :-)

Volume itself is a function of the expressiveness of the language. But even in a very expressive language, writing the most and most correct code in the smallest amount of time can be used as a definition for productivity and to identify basic features that you'd want to achieve that.

But the definition of productivity itself does not matter as long as you have one that allows you to specify what features are required and what features are not.

Jean-Christophe

Re: IDE

[Eli Zaretskii]

> Cc: adatgyujto@gmail.com, emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Sun, 11 Oct 2015 07:38:31 +0300
> 
> On 10/10/2015 07:48 PM, Eric Ludlam wrote:
> 
> > I had always intended CEDET to be a baseline for IDE like features.
> > Looking just at tagging files, those familiar with it's internals
> > recognize that it can use external tools as weak as ctags or GLOBAL, and
> > can use a more powerful external tool for parsing as well, such as using
> > JAVAC for decompiling .jar files into tags.
> 
> It sounds good if all I have is a parser/tagger.
> 
> But if I already have an external tool that can give me a set of 
> completions at a given position in a given buffer, and also provides a 
> go-to-definition feature, what's the advantage of going through 
> Semantic?

One advantage that comes to mind is that you don't depend on an
external tool whose development is beyond our control.

Re: IDE

[David Kastrup]

Eli Zaretskii <eliz@gnu.org> writes:

>> Cc: adatgyujto@gmail.com, emacs-devel@gnu.org
>> From: Dmitry Gutov <dgutov@yandex.ru>
>> Date: Sun, 11 Oct 2015 07:38:31 +0300
>> 
>> On 10/10/2015 07:48 PM, Eric Ludlam wrote:
>> 
>> > I had always intended CEDET to be a baseline for IDE like features.
>> > Looking just at tagging files, those familiar with it's internals
>> > recognize that it can use external tools as weak as ctags or GLOBAL, and
>> > can use a more powerful external tool for parsing as well, such as using
>> > JAVAC for decompiling .jar files into tags.
>> 
>> It sounds good if all I have is a parser/tagger.
>> 
>> But if I already have an external tool that can give me a set of 
>> completions at a given position in a given buffer, and also provides a 
>> go-to-definition feature, what's the advantage of going through 
>> Semantic?
>
> One advantage that comes to mind is that you don't depend on an
> external tool whose development is beyond our control.

Well, we'd certainly like to depend on GCC.

-- 
David Kastrup

Re: IDE

[Eli Zaretskii]

> Cc: adatgyujto@gmail.com, emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Sun, 11 Oct 2015 13:10:10 +0300
> 
> On 10/10/2015 05:25 PM, Eli Zaretskii wrote:
> 
> > It's standard software engineering practice, why should you ask for
> > its justification?
> 
> I'm asking for details. Again, to further the discussion.

I don't understand what details you expect me to produce.

> "Let's unify features X, Y and Z" is not necessarily the standard 
> practice.

I didn't say anything even close.  I did suggest to _decide_ which
features we'd like to be in an Emacs IDE.

> I've also given a few general reasons why a "big design" might be 
> suboptimal

High-level design is not "big design".  Quite the contrary: it
generally leaves out all of the details, and documents only the main
design decisions.  Like which features will be included and which
definitely excluded, whether "fixed-window" appearance should be part
of it or not, etc.  You can call this "guidelines" if "design" sounds
too much.  Developers will benefit from such guidelines because they
will know what to expect and how to design their components.

This is all standard SE practice.

> the result is likely to turn out to be less flexible

A good tool strikes a fine balance between "flexible enough" and "too
flexible".  The latter more often than not means the tool is complex
and hard to set up.

> a significant number of users might prefer to use only some of the
> parts.

That's OK, and I see no problem with that.

> > The other IDEs use something similar to a tooltip, or a drop-down menu
> > with different fonts and colors.
> 
> You can already customize colors and fonts user for the Company popup. 
> But if you end up using fonts with different dimensions, of course, that 
> would result in jagged display.

Right.  Which is why tooltips or pop-up menus are better: they don't
suffer from these problems.

> From trying it out, I have the following complaints about x-show-tip
> capabilities:
> 
> - It's background rendering is inconsistent. As an example, the first 
> time I evaluate (tooltip-show "abc") in an Emacs session, the background 
> is yellow-ish. The next time, and after that, the background is black.

This could be the result of the recent changes by Ken, to make
tooltips less "voracious", to use Martin's term.  Trying in an older
Emacs will tell.

FWIW, I don't see anything like that here (on MS-Windows).

> - Is there a way to show several tooltips at once? To display different 
> elements of the completion UI side by side.

No, you need to lump them all together, or use menus.

> - If a tooltip is displayed, and I Alt-Tab to another program's window, 
> the tooltip remains on top. This is by far the most annoying one.

Martin told you how to solve this, I think.

Re: IDE

[Eli Zaretskii]

> Cc: adatgyujto@gmail.com, emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Sun, 11 Oct 2015 14:02:34 +0300
> 
> For feature parity with Intellij IDEA and MS VS, we should be able to 
> display the list of completions and documentation for the currently 
> selected completion in two separate popups:
> 
> https://i-msdn.sec.s-msft.com/dynimg/IC797655.jpeg
> https://www.jetbrains.com/img/webhelp/idea/constructors_docs_in_completion.png

The first one looks like a popup menu, the second like a small frame,
is that right?

Re: IDE

[Eli Zaretskii]

> Date: Sun, 11 Oct 2015 13:38:35 +0200
> From: martin rudalics <rudalics@gmx.at>
> CC: adatgyujto@gmail.com, emacs-devel@gnu.org
> 
> Maybe we should just use simple frames with all decorations removed.

How is that different from tooltips?  They are such frames, AFAIK.

Re: IDE

[Eli Zaretskii]

> From: David Kastrup <dak@gnu.org>
> Cc: Dmitry Gutov <dgutov@yandex.ru>,  emacs-devel@gnu.org,  adatgyujto@gmail.com,  eric@siege-engine.com
> Date: Sun, 11 Oct 2015 17:23:10 +0200
> 
> Well, we'd certainly like to depend on GCC.

Or on GDB, or on any other flagship part of the GNU Project.

But I was talking about the rest of them.

Re: IDE

[Eli Zaretskii]

> From: David Engster <deng@randomsample.de>
> Cc: Dmitry Gutov <dgutov@yandex.ru>,  Eli Zaretskii <eliz@gnu.org>,  adatgyujto@gmail.com,  emacs-devel@gnu.org
> Date: Sun, 11 Oct 2015 14:27:02 +0200
> 
> Here's what I use in my doc-present package[1] for displaying slides:
> 
> (with-selected-frame
>     (make-frame '((minibuffer . nil)
> 		  (left-fringe . 0)
> 		  (right-fringe . 0)
> 		  (menu-bar-lines . 0)
> 		  (internal-border-width . 0)
> 		  (vertical-scroll-bars . nil)
> 		  (unsplittable . t)
> 		  (cursor-type . nil)
> 		  (tool-bar-lines . 0)))
>   (pop-to-buffer (get-buffer-create "*test*"))
>   (setq mode-line-format nil))
> 
> The only thing remaining are the decorations from the window manager. I
> don't think you can get rid of those from within Emacs?

We could easily extend make-frame to allow that, by defining new
attributes.

Re: IDE

[Eli Zaretskii]

> Cc: adatgyujto@gmail.com, emacs-devel@gnu.org
> From: Przemysław Wojnowski <esperanto@cumego.com>
> Date: Sun, 11 Oct 2015 15:18:17 +0200
> 
> For (project-oriented/enterprise) Java the features are:
> 1. Project support
> IDE has to know the boundaries of a project - where are sources, tests, libs, 
> resources/assets (additional files used in an app in runtime), docs - and what 
> are relations between them. Also it has to know how to work with project's build 
> tool (be it Maven, Gradle, etc.).
> A programmer joining a project (99% of cases) should be able to open/import it 
> and start work. Every Java IDE have that.
> 
> 2. Code completion
>  From whole project, used libraries, and resources
> 
> 3. Jumping around the project code and resources.
> Jumping to around the project code and used libraries. Another thing is jumping 
> to/from resources (for example aspects can be defined in an XML file and IDE 
> could allow to jump to matching classes).
> 
> 4. Finding usages of fields, methods, types.
> Helps to wrap head around project.
> 
> 5. Automatic compilation and showing compilation erros/warnings.
> Tightens development feedback loop.
> 
> 6. Running the tests (current, selected, all)
> Tighten development feedback loop.
> 
> 7. Debugging
> A must, especially for legacy code, so, basically 99% of projects. :-)
> 
> 8. Running the app
> 
> 9. Basic refactoring.
> I do refactor _a lot_ and in my experience the most important refactoring is 
> Extract Method. Others, while helpful, are less often used, compared to the EM.
> One variation of Extract Method is "EM with finding duplicates", which works 
> like this:
> - ask user for a method name,
> - find all occurrences of selected code in the buffer
> - ask user if she wants to replace all occurrences with the call to the new method.
> This is fantastic feature that Intellij has and helps to remove a lot of 
> duplicated code.
> 
> 10. Showing documentation (tooltip, etc.)

I think you forgot profiling.

Re: IDE

[Przemysław Wojnowski]

W dniu 11.10.2015 o 18:04, Eli Zaretskii pisze:
[...]
>
> I think you forgot profiling.

I don't know you and, most probably, we're in different cultures, but if that
was a sarcasm, then you are waisting your time and mine. It doesn't hurt me
(Bible: 1-Cor,13,11), but rather the project by not focusing on getting the job
done. I'm not complaining that no one does the job, but offer a help (even small 
one) and try to improve the project.

Cheers,
Przemysław

Re: IDE

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org
> From: Przemysław Wojnowski <esperanto@cumego.com>
> Date: Sun, 11 Oct 2015 19:05:38 +0200
> 
> W dniu 11.10.2015 o 18:04, Eli Zaretskii pisze:
> [...]
> >
> > I think you forgot profiling.
> 
> I don't know you and, most probably, we're in different cultures, but if that
> was a sarcasm

It wasn't.  AFAIR, a profiler is indeed an important part of at least
one popular Jave IDE.

Re: IDE

[David Kastrup]

Eli Zaretskii <eliz@gnu.org> writes:

>> Cc: emacs-devel@gnu.org
>> From: Przemysław Wojnowski <esperanto@cumego.com>
>> Date: Sun, 11 Oct 2015 19:05:38 +0200
>> 
>> W dniu 11.10.2015 o 18:04, Eli Zaretskii pisze:
>> [...]
>> >
>> > I think you forgot profiling.
>> 
>> I don't know you and, most probably, we're in different cultures, but if that
>> was a sarcasm
>
> It wasn't.  AFAIR, a profiler is indeed an important part of at least
> one popular Jave IDE.

Associating the output of gprof and gcov (and perf) with the
corresponding sources (possibly also selecting traced areas) and
allowing editing based on the results would certainly fit what
developers have to do.  Leafing through a profiled call graph while
displaying the corresponding source files in a parallel window: clear
IDE functionality.

Not sure why that would be sarcastic.

-- 
David Kastrup

Re: IDE

[Eric Ludlam]

On 10/11/2015 12:38 AM, Dmitry Gutov wrote:
> Hi Eric,
>
> On 10/10/2015 07:48 PM, Eric Ludlam wrote:
>
>> I had always intended CEDET to be a baseline for IDE like features.
>> Looking just at tagging files, those familiar with it's internals
>> recognize that it can use external tools as weak as ctags or GLOBAL, and
>> can use a more powerful external tool for parsing as well, such as using
>> JAVAC for decompiling .jar files into tags.
>
> It sounds good if all I have is a parser/tagger.
>
> But if I already have an external tool that can give me a set of 
> completions at a given position in a given buffer, and also provides a 
> go-to-definition feature, what's the advantage of going through 
> Semantic? SRecode support?

CEDET's tools are set up in layers.  As an language support author you 
can start with a series of overrides for writing a lexer or parser 
yourself.  If you have a tool that parses already (such as exuberent 
CTags), you can instead just start at the high level tagging parser and 
skip all the lower level stuff.

If you have an existing completion engine for cases where you happen to 
have an interpreter with completion, or something else, you can just 
override the completion engine directly.  The srecode tool does this, 
and there is an experimental clang version in CEDET's repository as well.

Having overrides at all 3 levels is the best, since different tools ask 
for different features, but it is not necessary as there is backup 
implementation for the higher level features.

> Not to mention that we have tools like Jedi that only (mostly) do 
> these two things, but don't provide dumps of the syntax tree.
>
>> As a backup, it also has
>> in-buffer parsing for when you've only half-written your code, or when
>> no external tool is available.
>
> That assumes we also have a Semantic grammar for the said language. 
> And then, we'll need to duplicate whatever logic is used by the 
> external tool to disambiguate between same-named methods in different 
> classes.
>
> If that's even possible.
>

Of course.  For strongly typed languages the problems are deterministic 
so it's possible.  For dynamic languages, they usually provide something 
you can ask.  It's just a matter of integration.

I enjoy working on puzzles like this, but for me it is a matter of free 
time.  Kids, job, etc get in the way.

>> The same philosophy is throughout CEDET. Unfortunately, those who are
>> interested in tooling who brush casually past CEDET only see that people
>> complain that it doesn't always complete right, or that the C++ part is
>> hard to setup
>
> The example we've been given in one of the previous discussions of C++ 
> support is that Semantic can't tell the difference between different 
> methods with the same name? If Z#foo returns type A, and T#foo returns 
> type B, it's been shown that Semantic doesn't know which is the type 
> 'x.foo'.
>
> Given this, how would we expect it to properly support languages with 
> type inference like Scala or Go? Not to mention dynamic languages 
> which normally have no type annotations on methods, and often use more 
> complicated algorithms for accurate code completion.

Semantic has the information to differentiate between two methods of 
different types.  There are still a few short-cuts in the completion 
engine  where it could do a little more to disambiguate.  There is a 
function to '-select-best-tag' in the analyzer, but at the end, it just 
grabs the first element from the list of possibilities.  If you have a 
particular use case in mind, it would good to have a simple example that 
shows what it is and maybe it will be easy to update.

>> If we look at the two starting key features listed for IDEs,
>> "completion" and "refactoring", the basics are all there.  CEDET has a
>> completion engine, but it is only as good as the input data.
>
> It's limited by the input data, sure, but it also has to know what to 
> do with it.
>
> If only implementing accurate code completion was as easy as writing a 
> language parser.

This is a matter of time and desire.  For the code I've been writing 
lately (arduino hobby stuff), it has been more than sufficient. David 
was interested in handling templates and was able to add support for 
that.  If someone wants it to handle something new, the data is there to 
do it.

>> CEDET also has 'srecode' which is about
>> generating code.  This bit is trickier, as the intention is you could
>> write one highlevel tool that might extract tag data from your file,
>> permute the language-independent tags, and then write them back out with
>> srecode.  There are tests showing this working, but they are simple and
>> proof of concept and not stuff someone would depend on day-to-day.
>
> SRecode should be in a better position, but I imagine it would also 
> fail often enough in dynamic languages.

I'm not quite sure what the intention is here.  Srecode is just template 
expansion, with mechanisms to enable multiple layers of templates, and 
support for "application" templates and user override templates for 
customization.

What's missing is someone taking the time to start building the logic to 
use semantic to pull buffers apart, and srecode to re-write the code.

> And in any case, one has to teach Semantic about scoping rules for 
> each given language, for SRecode to only rename, say, a variable 'foo' 
> but not function 'foo', or only the variable 'bar', but not the 
> variable 'bar' up the scope that's being shadowed.
>

There is a fair amount of logic for supporting language scopes. There is 
almost nothing about running filters for "all the calls to symbol FOO 
that has some particular feature".  The symref tool has a comment in it 
that says "do the filtering here", but no one has gotten around to it.

My assumption is that it would be easier to start there than start from 
scratch, unless you just hijack some external tool.

Eric

RE: IDE

[Drew Adams]

[-- Attachment #1: Type: text/plain, Size: 1554 bytes --]

> For feature parity with Intellij IDEA and MS VS, we should be able
> to display the list of completions and documentation for the
> currently selected completion in two separate popups:
> https://i-msdn.sec.s-msft.com/dynimg/IC797655.jpeg
> https://www.jetbrains.com/img/webhelp/idea/constructors_docs_in_completion.png

FWIW, Icicles has long provided this, just by using *Completions*
for the completions and *Help* for the current-completion doc.

You can of course use separate frames for these.  There is also an
option that moves the *Completions* frame to the display edge when
you request candidate help or you act otherwise on a candidate.

Users control when/if candidate help is displayed (by key).
It is not just thrown in their face systematically.

Here is a screenshot of one possible layout:
http://www.emacswiki.org/emacs/Icicles_-_A_Propos_d'Apropos#CompFrameMovedLeft

FWIW, I also consider it a plus, not a minus, that the window-mgr
"decorations" are present.  Users can move the frames, iconify
them, shrink/enlarge them - do whatever they want with them,
including in some cases on the fly, by key.

They are ordinary Emacs frames, not just hard-coded popups
that exhibit only some developer's idea of the best behavior.

And you'll notice that the *Completions* mode line is made use
of; it provides brief help on the current candidate even when
*Help* is not shown.

I do agree that things like a menu-bar are best removed, by
default (as shown).  But users can always add them, if they
prefer.

[-- Attachment #2: drew-emacs-icicle-Comp-moved-left.png --]
[-- Type: image/png, Size: 38480 bytes --]

Re: IDE

[John Wiegley]

>>>>> Przemysław Wojnowski <esperanto@cumego.com> writes:

> I cannot say what makes a Ruby programmer productive, but I can list a few
> things important for Java programmers. Others, having experience with other
> languages, may list _must have_ features for their environments.

I presented a list of candidate IDE features a few days ago. If you could
present your list in terms of additions or subtractions from that one, it
would make things much easier. I believe I already covered everything that you
proposed, but I'd like your confirmation of that.

John

Re: IDE

[martin rudalics]

 > How is that different from tooltips?  They are such frames, AFAIK.

Yes.  Just that we do not expose them to Lisp.

martin

Re: IDE

[Przemysław Wojnowski]

W dniu 11.10.2015 o 19:15, Eli Zaretskii pisze:
>> Cc: emacs-devel@gnu.org
>> From: Przemysław Wojnowski <esperanto@cumego.com>
>> Date: Sun, 11 Oct 2015 19:05:38 +0200
>>
>> W dniu 11.10.2015 o 18:04, Eli Zaretskii pisze:
>> [...]
>>>
>>> I think you forgot profiling.
>>
>> I don't know you and, most probably, we're in different cultures, but if that
>> was a sarcasm
>
> It wasn't.  AFAIR, a profiler is indeed an important part of at least
> one popular Jave IDE.

In such case I'm sorry for going too far. :-)

Anyway, I haven't forgotten about a profiler. In enterprise Java it doesn't
have to be a part of an IDE and in most cases an external product is used -
AFAIK it's true for other JVM languages too.

That's why people having experience in other environments should write what
makes them productive, in their environments. Maybe for C/C++ programmer a
profiler is a must. I don't know.

To be clear that we're on the same page. We are in the brainstorming phase now,
collecting ideas that will be evaluated later.

Re: IDE

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Yes. ECB uses lots of advices to achieve what it does. I think somebody
  > (Martin?) worked on a 'window group' feature which would make this
  > easier.

I used to have a rule of not installing anything in Emacs that put
advice on some other part of Emacs.  When module A wants module B to
do something special, rather than having A advise B, it's better for
mainenance to make B run a hook and have A set up that hook.

The hook is cleaner because, when you look at the code of B, you see
it runs the hook, and you you know to check what's in the hook.  If
advice is used, there is nothing in B to warn you that the function
does something you can't see.

Thus, whenever I installed a package which used advice, I added hooks
and changed it to use them instead.

Advice is for users to use, not for Emacs to use.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: IDE

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > I indeed think that we should have infrastructure to turn on a major
  > mode in a region of a buffer.

I have tried to encourage progress in this direction for 15 years.

  > I'm not sure we should use text properties or overlays for that,
  > though.  The region could be part of the command that turns on the
  > mode with region limits stored in markers.

The two are not necessarily alternatives.  They might be two
different levels of functionality.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: IDE

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > "Let's unify features X, Y and Z" is not necessarily the standard 
  > practice. It all depends on the exact features and how they are used.

I agree with this point.

Lots of subtle details of how Emacs commands behave in various
situations are convenient for some users.  Users have chosen their
usage based on those details.  A simple and clean scheme to
make things uniform can be very elegant, and yet make many users
unhappy.


-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: IDE

[John Wiegley]

>>>>> Richard Stallman <rms@gnu.org> writes:

> Lots of subtle details of how Emacs commands behave in various situations
> are convenient for some users. Users have chosen their usage based on those
> details. A simple and clean scheme to make things uniform can be very
> elegant, and yet make many users unhappy.

I do agree with this. Any uniform core must allow for the idiosyncratic
behaviors people have come to rely upon. If we find this cannot be done,
I'd give up on the attempt.

What I want to explore is whether things can be better. We have great
unification in some areas, and I've a sense there are more opportunities
if we can build the right layers of abstraction.

John

Re: IDE

[Dmitry Gutov]

On 10/11/2015 06:08 PM, Eli Zaretskii wrote:

> One advantage that comes to mind is that you don't depend on an
> external tool whose development is beyond our control.

You still depend on an external tool for other things.

Even if that wasn't true, having to re-implement everything yourself, 
for each language, is not an advantage.

Certainly not for a relatively small project like Emacs.

Re: IDE

[Dmitry Gutov]

On 10/11/2015 06:25 PM, Eli Zaretskii wrote:

>> https://i-msdn.sec.s-msft.com/dynimg/IC797655.jpeg
>> https://www.jetbrains.com/img/webhelp/idea/constructors_docs_in_completion.png
>
> The first one looks like a popup menu, the second like a small frame,
> is that right?

Both pictures show similar configurations: a menu with completions and a 
small frame with documentation.

In the second one the frame just overlaps the menu, for some reason (I 
couldn't find a better screenshot on a short notice).

Re: IDE

[Dmitry Gutov]

On 10/11/2015 06:23 PM, Eli Zaretskii wrote:

> High-level design is not "big design".  Quite the contrary: it
> generally leaves out all of the details, and documents only the main
> design decisions.  Like which features will be included and which
> definitely excluded, whether "fixed-window" appearance should be part
> of it or not, etc.  You can call this "guidelines" if "design" sounds
> too much.

Sounds good to me. Yes, "guidelines" is a more familiar term for that.

>> a significant number of users might prefer to use only some of the
>> parts.
>
> That's OK, and I see no problem with that.

So we need to make sure that they can.

>> - It's background rendering is inconsistent. As an example, the first
>> time I evaluate (tooltip-show "abc") in an Emacs session, the background
>> is yellow-ish. The next time, and after that, the background is black.
>
> This could be the result of the recent changes by Ken, to make
> tooltips less "voracious", to use Martin's term.  Trying in an older
> Emacs will tell.

No, that's a fairly old problem. And specific to GTK, IIUC.

>> - Is there a way to show several tooltips at once? To display different
>> elements of the completion UI side by side.
>
> No, you need to lump them all together, or use menus.

Neither option is good enough.

>> - If a tooltip is displayed, and I Alt-Tab to another program's window,
>> the tooltip remains on top. This is by far the most annoying one.
>
> Martin told you how to solve this, I think.

Yup.

Re: IDE

[Dmitry Gutov]

On 10/11/2015 09:10 PM, Drew Adams wrote:

> FWIW, Icicles has long provided this, just by using *Completions*
> for the completions and *Help* for the current-completion doc.

Indeed, by default Company just pops a window for documentation. But 
many users have taken to using a third-party frontend that displays the 
documentation using pos-tip.

I don't want them to have to choose between that and displaying 
completion in a tooltip. They should be able to have both.

> Users control when/if candidate help is displayed (by key).
> It is not just thrown in their face systematically.

Many users like it to be displayed after a delay automatically.

> FWIW, I also consider it a plus, not a minus, that the window-mgr
> "decorations" are present.  Users can move the frames, iconify
> them, shrink/enlarge them - do whatever they want with them,
> including in some cases on the fly, by key.

Well, I don't consider it a plus.

Re: IDE

[John Yates]

[-- Attachment #1: Type: text/plain, Size: 1293 bytes --]

On Tue, Oct 6, 2015 at 8:18 PM, Richard Stallman <rms@gnu.org> wrote:

> Emacs with GUD is an IDE.


Wow Richard!  I get that you probably have never used an IDE.  Have you
ever had one demo-ed for you?  Please assure me that your grasp of modern
IDE functionality extends beyond what that statement suggests.  I can never
recall a time when anyone working in the tools and development environment
areas would have concurred that Emacs+GUD constitutes an IDE.


> It has a big advantage compared with other IDEs:

when you see a source file, you're editing it with Emacs.
>

To what are you contrasting Emacs+GUD?  GDB with only a command-line
interface?  GDB with TUI?  For decades source display integrated with
visual indications of the locations of breakpoints, and highlighting the
current execution line has been the absolute minimum ante for debugging
support within an IDE.

If it falls short compared with other IDEs, I think this would be a
> great area for improvement of Emacs.
>

I use GUD + gdb-many-windows.  I put up with its quirks, shortcoming and
idiosyncrasies because I am an old geezer devoted to emacs.  And I would
love to see an improved debugging environment.  But no amount of
improvement along those lines will give Emacs any stronger claim to being
an IDE.

/john

[-- Attachment #2: Type: text/html, Size: 2311 bytes --]

Re: IDE

[Eli Zaretskii]

> Date: Sun, 11 Oct 2015 18:23:55 -0400
> From: John Yates <john@yates-sheets.org>
> Cc: Emacs developers <emacs-devel@gnu.org>, Dmitry Gutov <dgutov@yandex.ru>
> 
> I use GUD + gdb-many-windows. I put up with its quirks, shortcoming and
> idiosyncrasies because I am an old geezer devoted to emacs. And I would love to
> see an improved debugging environment. But no amount of improvement along those
> lines will give Emacs any stronger claim to being an IDE.

I don't understand what made you worked out.  All Richard was saying
that a debugger front-end is an important part of an IDE.  I'm sure
you agree.  Yes, much more is needed to make it a more complete IDE.

Re: IDE

[John Yates]

[-- Attachment #1: Type: text/plain, Size: 1101 bytes --]

On Sun, Oct 11, 2015 at 10:45 PM, Eli Zaretskii <eliz@gnu.org> wrote:

> All Richard was saying that a debugger front-end is an important part of
> an IDE.


Eli,

With all due respect, while you may interpret Richard's words in that
manner, that is not what he wrote.  I quoted at the top of my posting
Richard's very first sentence that started this extended thread.  Here it
is again in its entirety.:

   > Emacs with GUD is an IDE.

Given the amount of influence Richard will exert over efforts to fashion
Emacs into something approximating a modern IDE I believe it is reasonable
to wonder how familiar he is (or is willing to become) with such
technologies.  When we had the long thread some while back about supporting
completion and refactoring I got the sense that Richard was unfamiliar with
the functionality and user experience of modern IDEs.  IIRC he committed to
seeking some outside guidance which might have included becoming more
familiar with the current state of typical IDEs.  If that has yet to happen
I wonder how John's anticipated f2f discussion with Richard will go.

/john

[-- Attachment #2: Type: text/html, Size: 1827 bytes --]

Re: IDE

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 1935 bytes --]

On 10/12/2015 02:45 AM, John Yates wrote:
> On Sun, Oct 11, 2015 at 10:45 PM, Eli Zaretskii <eliz@gnu.org
> <mailto:eliz@gnu.org>> wrote:
> 
>     All Richard was saying that a debugger front-end is an important
>     part of an IDE.
> 
> 
> Eli,
> 
> With all due respect, while you may interpret Richard's words in that
> manner, that is not what he wrote.  I quoted at the top of my posting
> Richard's very first sentence that started this extended thread.  Here
> it is again in its entirety.:
> 
>    > Emacs with GUD is an IDE.
> 
> Given the amount of influence Richard will exert over efforts to fashion
> Emacs into something approximating a modern IDE I believe it is
> reasonable to wonder how familiar he is (or is willing to become) with
> such technologies.  When we had the long thread some while back about
> supporting completion and refactoring I got the sense that Richard was
> unfamiliar with the functionality and user experience of modern IDEs. 
> IIRC he committed to seeking some outside guidance which might have
> included becoming more familiar with the current state of typical IDEs. 
> If that has yet to happen I wonder how John's anticipated f2f discussion
> with Richard will go.

Well, Emacs with GUD is integrated with GDB, is for development, and is
an environment. That this combination still lacks some features of other
IDEs doesn't matter, since some heavyweight IDEs lack the features of
some others. (Is an IDE without autocompletion an IDE? Is an IDE without
automated refactoring tools an IDE? If not, which tools does it need?)

"IDE" isn't really a distinct category of program. There's a continuum
all the way from ed the standard editor to IntelliJ. We want to move
Emacs toward the latter, but there's no distinct point at which Emacs
will stop being a text editor and start being an IDE; according to some
people, it's already over that line.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: IDE

[Oleh Krehel]

Dmitry Gutov <dgutov@yandex.ru> writes:

> For feature parity with Intellij IDEA and MS VS, we should be able to
> display the list of completions and documentation for the currently
> selected completion in two separate popups:
>
> https://i-msdn.sec.s-msft.com/dynimg/IC797655.jpeg
> https://www.jetbrains.com/img/webhelp/idea/constructors_docs_in_completion.png

I like the first style a lot more. The second looks a lot like the ugly
mess of Eclipse.

Here's what's currently possible in Emacs for C++:

- show function arguments and docstring in an overlay:
    http://oremacs.com/download/fa-do-comments.png
- complete class member at point:
    http://oremacs.com/download/function-args-qt.png
- jump to tag in directory:
    http://oremacs.com/download/function-args-boost.png

The latter two can be done with powerful regexp-based completion, which
MS VS likely still doesn't include.

What's missing, compared to the MS VS picture:

- Candidate completion is in the minibuffer instead of at-point.
- The docstring (only the comment part) is shown separately.

The first part is just Emacs' style of doing things: we usually enter
stuff in the minibuffer, so it makes sense for completion to display
there.  The second part is arguably unnecessary: I usually just jump to
definition of symbol rather than look at the docstring inline.

What's missing, in my opinion, is only faster and more precise parser
(CEDET, GCC etc). For example, currently `semantic-fetch-tags' parses
public/private/protected labels as tags, instead of applying these
properties to actual tags. If that were so, it would be very easy to add
a public/private/protected icon to each tag, just like MS VS does it.

Another example is the QT code: it's a popular LGPL C++ framework that's
currently hard to setup for CEDET.
For instance, `#include <QtGui/QPushButton>` is a plain file without an 
extension with only this code inside:

    #include "qpushbutton.h"

Since the extension isn't recognized, it's not parsed by CEDET.  And I
have to write `#include "qpushbutton.h"` in my application instead of
the more preferred `#include <QtGui/QPushButton>`, because that way I
get tag completion.

These small things are what the competing IDE have been incrementally
improving over the last 20 years. I think we have a serviceable
foundation for C++ completion, it only needs to be polished *a lot*
more.

I also think that the way to do it right would be to integrate with GCC
for code parsing, for reasons of speed and precision.  As I mentioned
before, CEDET is usable, but it can't be as fast and as precise as
GCC. Add to that that the language standard is updated every 5 years or
so with new syntax. GCC has the people to update the parser
accordingly. Doing so for CEDET would be a duplication of effort, and we
don't have the people to do it anyway.

Could someone explain to me if making GCC the dependency of Emacs would
be a good idea, from technical and freedom point of view?  Personally, I
wouldn't care if Emacs executable would get inflated a bit more, if that
meant access to true IDE features, which are only possible with a
precise and fast parser.

    Oleh

Re: IDE

[Dmitry Gutov]

On 10/12/2015 02:05 PM, Oleh Krehel wrote:

> I like the first style a lot more. The second looks a lot like the ugly
> mess of Eclipse.

I agree, the Microsoft IDE looks slicker here, but the examples are 
basically the same, in that they use separate frames for the completion 
list and documentation, not the same one.

Eclipse uses a combined one, last I checked.

> The first part is just Emacs' style of doing things: we usually enter
> stuff in the minibuffer, so it makes sense for completion to display

A lot of users are fine with that, but I think we should do better.

Ivy also could use a popup to display completions, in the future. 
Ideally, I'd prefer something like Sublime/Atom popup at the top of the 
screen that you get after pressing C-p.

> there.  The second part is arguably unnecessary: I usually just jump to
> definition of symbol rather than look at the docstring inline.

You'd think so, but displaying the docstring automatically, like 
Auto-Complete does (as well as certain IDEs), has been a common request 
for a while. And now, https://github.com/expez/company-quickhelp is 
pretty popular.

> CEDET is usable, but it can't be as fast and as precise as
> GCC. Add to that that the language standard is updated every 5 years or
> so with new syntax. GCC has the people to update the parser
> accordingly. Doing so for CEDET would be a duplication of effort, and we
> don't have the people to do it anyway.

Agree.

> Could someone explain to me if making GCC the dependency of Emacs would
> be a good idea, from technical and freedom point of view?  Personally, I
> wouldn't care if Emacs executable would get inflated a bit more, if that
> meant access to true IDE features, which are only possible with a
> precise and fast parser.

Having the whole GCC as a dependency might be problematic, but that's 
not the #1 problem. AFAIK, GCC currently has no "code completion" 
feature anywhere.

Re: IDE

[Eric Ludlam]

On 10/11/2015 08:56 AM, Dmitry Gutov wrote:
> On 10/11/2015 03:37 PM, David Engster wrote:
>
>> Works for me from 'emacs -Q' (emacs 24.5, lucid toolkit):
>
> Thanks, this sequence works for me too, even on master, but only with
> '-Q'. So something in my config is at fault.
>
>>> I thought tooltips don't affect the keyboard input?
>>
>> I mean I cannot choose completions with the keyboard with up/down. But
>> this is a problem with how it is done in CEDET, which was never my cup
>
> Right, CEDET would have to handle keyboard input somehow, for it to work
> better.

Back when I added that, there was no way for me to get keyboard 
navigation to work with the tooltip on my linux.  There were menus 
available that did handle keyboard nav depending on toolkit, but those 
blocked typing, so I couldn't use them for the inline display that might 
popup automatically while you type.

I have found that some of the other completion UIs have worked much 
better than the ones I put together, so I had since focused on just 
getting the completions as correct as I can.

Eric

Re: IDE

[Dmitry Gutov]

On 10/11/2015 04:18 PM, Przemysław Wojnowski wrote:

> For (project-oriented/enterprise) Java the features are:
> 1. Project support
> IDE has to know the boundaries of a project - where are sources, tests,
> libs, resources/assets (additional files used in an app in runtime),
> docs - and what are relations between them. Also it has to know how to
> work with project's build tool (be it Maven, Gradle, etc.).
> A programmer joining a project (99% of cases) should be able to
> open/import it and start work. Every Java IDE have that.

Emacs master now has project.el, which will be a step in this direction.

We can extend that API to contain information about where certain 
resources are, but for each type of projects there will be a different 
set of resources. While we could have a jump-to-test command (tests 
should be common), we can't easily have a commands for each kind of 
"jump to the web template for this controller" action.

So I'm not sure how to solve this. Have a "jump-to-related-file" 
command, which will prompt for the type of the resource when invoked?

Support for build tools seems more straightforward, someone should just 
collect an overview of how users interact with different ones, like 
Make, Maven, Gradle, Rake, etc, to firmly establish the common ground.

> 2. Code completion
>  From whole project, used libraries, and resources

Unless CEDET delivers on that front, we're unlikely to have completion 
for Java in the near future in the core. But there are third-party 
packages that provide it.

> 3. Jumping around the project code and resources.
> Jumping to around the project code and used libraries. Another thing is
> jumping to/from resources (for example aspects can be defined in an XML
> file and IDE could allow to jump to matching classes).

Do you mean "jump to the thing at point"? That sounds complicated, and 
support for different "things" will have to be implemented separately.

> 4. Finding usages of fields, methods, types.
> Helps to wrap head around project.

That's within the purview of xref, and up to CEDET or external tools to 
implement. But you can try xref-find-references now, for a 
quick-and-dirty Grep-based solution.

> 5. Automatic compilation and showing compilation erros/warnings.
> Tightens development feedback loop.

Flycheck. Sadly, it's not in Emacs.

> 6. Running the tests (current, selected, all)
> Tighten development feedback loop.

Not sure which facility would be most fitting. A project *could* include 
metadata about how to run its tests better, but then the resulting 
buffer would need new compilation-error-regexp-alist entries, and/or an 
ability to interact with a debugger/REPL if it's triggered during the 
test run.

> One variation of Extract Method is "EM with finding duplicates", which
> works like this:
> - ask user for a method name,
> - find all occurrences of selected code in the buffer
> - ask user if she wants to replace all occurrences with the call to the
> new method.
> This is fantastic feature that Intellij has and helps to remove a lot of
> duplicated code.

That sounds useful indeed.

Re: IDE

[Dmitry Gutov]

On 10/12/2015 02:45 PM, Eric Ludlam wrote:

> There were menus
> available that did handle keyboard nav depending on toolkit, but those
> blocked typing, so I couldn't use them for the inline display that might
> popup automatically while you type.

Indeed. That's still the case, so using a menu for that is not an option.

Re: IDE

[Eric Ludlam]

On 10/10/2015 07:20 PM, John Wiegley wrote:
>>>>>> Eric Ludlam <eric@siege-engine.com> writes:
>
>> In CEDET, the function / command `semantic-analyze-current-context' provides
>> an output that has lots of details about the buffer near point. Not just
>> what the cursor is on, but if it is a chain of symbols such as dereferencing
>> struct pointers, and in many cases, it figures out the data type of the
>> symbol the cursor is on. It also handles in-buffer caching, etc and plenty
>> of performance tweaking is available.
>
> My preference is for each core feature to have an extremely simple and light
> interface (taking indentation as an ideal), so that it can also be used for
> non-IDE tasks we haven't imagined yet. From what I know about CEDET to date,
> it is much more complex than this objective.

That is because the task is complex.

> When I squint, I see ctags, imenu, pcomplete, helm, company, hippie-expand,
> flycheck, and more, all starting to look somewhat alike: They each act upon
> information relevant to the buffer in some way. Where they differ is in how
> they derive this information, and how the user interacts with it. Can we
> provide a common, low-level interface for this style of functionality? A
> common API against which we can implement both data-gathering backends, and
> display front-ends? And with an emphasis on efficiency and low-latency!

The primary difference between your list and CEDET is that those other 
tools focus on picking up a current symbol, or perhaps a substring, and 
it is up to the next layer to figure out more about it.  I agree that 
that could be simplified across the variosu tools, but AFAIK the 
'thing-at-pt' stuff is that common interface.  IDEs know more about the 
buffer than just some symbol and the major-mode of the current buffer.

Things like dabbrev work pretty well finding similar strings that often 
have the feel of being 'smart', but that only works if you've typed it 
in before.

If you want a stronger set of smart behaviours at point, you will need 
to raise your standard to include more derived data.

> We need to harness the power of multiplication, so that every new backend
> makes every frontend automatically more powerful, and vice versa. This will
> also help us better leverage our existing base of contributors.

This I agree with.

Eric

Re: IDE

[Oleh Krehel]

Dmitry Gutov <dgutov@yandex.ru> writes:

> Ivy also could use a popup to display completions, in the
> future. Ideally, I'd prefer something like Sublime/Atom popup at the
> top of the screen that you get after pressing C-p.

Eventually, yes. But it's hard for me to figure out how to make the text
and the overlays work together while the text is being updated.  Perhaps
I should try Sublime/Atom at least once, just to see how it's done
there.

>> Could someone explain to me if making GCC the dependency of Emacs would
>> be a good idea, from technical and freedom point of view?  Personally, I
>> wouldn't care if Emacs executable would get inflated a bit more, if that
>> meant access to true IDE features, which are only possible with a
>> precise and fast parser.
>
> Having the whole GCC as a dependency might be problematic, but that's
> not the #1 problem. AFAIK, GCC currently has no "code completion"
> feature anywhere.

There's no need for a specific "code completion" feature. Take this code
for example:

    #include "qapplication.h"
    #include "qfont.h"
    #include "qpushbutton.h"
    
    int main(int argc, char *argv[]) {
        QApplication app(argc, argv);
        QPushButton quit("Quit");
        quit.
            
        return 0;
    }

Here's what (semantic-fetch-tags) returns:

    (("qapplication.h"
      include
      nil
      nil
      #<overlay from 1 to 26 in main.cc>)
     ("qfont.h"
      include
      nil
      nil
      #<overlay from 27 to 45 in main.cc>)
     ("qpushbutton.h"
      include
      nil
      nil
      #<overlay from 46 to 70 in main.cc>)
     ("main"
      function
      (:arguments (("argc"
                    variable
                    (:type "int")
                    (reparse-symbol arg-sub-list)
                    #<overlay from 81 to 90 in main.cc>)
                   ("argv"
                    variable
                    (:pointer 1
                              :dereference 1
                              :type "char")
                    (reparse-symbol arg-sub-list)
                    #<overlay from 91 to 104 in main.cc>))
                  :type "int")
      nil
      #<overlay from 72 to 205 in main.cc>))

A similar data structure *has* to be somewhere in the GCC innards: it's
a first step for compilation. In addition, this information is used to
point out compilation errors/warnings.
This data structure would know all defined functions/variables/types.
For instance, it would know that the class QPushButton is defined in
qpushbutton.h at line 57, and it inherits from QAbstractButton.

It only remains to write a small wrapper that dumps all the AST included
into `file_name` up to the point `offset`:

    void gcc_ast_for_file (char* file_name, int offset, void* out)

I hope that I'm not being naive here and it's as simple to do as that.

    Oleh

Re: IDE

[Óscar Fuentes]

Oleh Krehel <ohwoeowho@gmail.com> writes:

[snip]

> It only remains to write a small wrapper that dumps all the AST included
> into `file_name` up to the point `offset`:
>
>     void gcc_ast_for_file (char* file_name, int offset, void* out)
>
> I hope that I'm not being naive here and it's as simple to do as that.

As mentioned on other message of mine, this is what a GCC frontend
developer has to say (search for the comments of mlopezibanez):

https://lwn.net/Articles/629259/

Basically what he says is: "No, it is not simple at all to get that
information from GCC, you'll better use Clang."

Re: IDE

[Dmitry Gutov]

On 10/12/2015 03:37 PM, Oleh Krehel wrote:

> Eventually, yes. But it's hard for me to figure out how to make the text
> and the overlays work together while the text is being updated.  Perhaps
> I should try Sublime/Atom at least once, just to see how it's done
> there.

Not sure how looking at Atom would help, but you should absolutely try 
it. It's Free Software, after all.

> Here's what (semantic-fetch-tags) returns:
>
> ...
> A similar data structure *has* to be somewhere in the GCC innards: it's
> a first step for compilation. In addition, this information is used to
> point out compilation errors/warnings.
> This data structure would know all defined functions/variables/types.
> For instance, it would know that the class QPushButton is defined in
> qpushbutton.h at line 57, and it inherits from QAbstractButton.

You have to find out what the type of the call target is, first.

In your example, it's rather easy: "QPushButton quit" is right on the 
previous line.

But what if we're at the end of a chain of calls, like

app->window->resolve_handler(bar).|

?

And resolve_handler is overloaded, and its return type is dependent on 
the type of its argument?

What if `bar' is declared as `auto' (see C++11)? Or `app' itself?

I'm fairly sure an experienced C++ tooling developer could add other 
complicated examples.

Re: IDE

[Oleh Krehel]

Dmitry Gutov <dgutov@yandex.ru> writes:

> Not sure how looking at Atom would help, but you should absolutely try
> it. It's Free Software, after all.

Thanks for mentioning that. I wasn't sure that was the case.

>> Here's what (semantic-fetch-tags) returns:
>>
>> ...
>> A similar data structure *has* to be somewhere in the GCC innards: it's
>> a first step for compilation. In addition, this information is used to
>> point out compilation errors/warnings.
>> This data structure would know all defined functions/variables/types.
>> For instance, it would know that the class QPushButton is defined in
>> qpushbutton.h at line 57, and it inherits from QAbstractButton.
>
> You have to find out what the type of the call target is, first.
>
> In your example, it's rather easy: "QPushButton quit" is right on the
> previous line.
>
> But what if we're at the end of a chain of calls, like
>
> app->window->resolve_handler(bar).|
>
> ?
>
> And resolve_handler is overloaded, and its return type is dependent on
> the type of its argument?

CEDET already resolves these complicated chains pretty well, as long as
it's got the correct AST.

> What if `bar' is declared as `auto' (see C++11)? Or `app' itself?

This would be harder, but still very doable, even within CEDET.

Now, there are several issues standing in the way of getting the correct
AST:

1. Finding where the relevant headers are located on the file
system. This means that the AST parser should hook into the currently
used build system, and correctly see which #ifdef switches apply where.
Only GCC or its likes have access to this info always. To reiterate, if
the program compiles with GCC, the exact same switches must be passed to
the AST parser, so that it parses exactly the same headers in exactly
the same way.

2. Parsing the newly introduced language features. Here CEDET and
cc-mode are behind because of the shear complexity of the full C++
syntax. However, GCC does it very well.

I've been working on function-args.el to extend CEDET C++ support for my
uses for around 2 years now. The most common error that I have to work
around is "Type definition not found", which happens either because the
header path wasn't resolved, or the wrong #ifdef path was taken.

And, of course, there's the issue of Elisp speed. While jumping to a C
tag in emacs/src, I collect the tags from 190 files in that
directory. They are all pre-parsed by CEDET, totaling to 19460 tags. It
takes more than a full second to just merge these 190 lists retrieved
for each file into a single list. It makes me think that maybe that list
should be built in an external process.
On the other hand, having the full info in Elisp data structures would
ease the application development significantly.

    Oleh

Re: IDE

[Oleh Krehel]

Óscar Fuentes <ofv@wanadoo.es> writes:

> As mentioned on other message of mine, this is what a GCC frontend
> developer has to say (search for the comments of mlopezibanez):
>
> https://lwn.net/Articles/629259/
>
> Basically what he says is: "No, it is not simple at all to get that
> information from GCC, you'll better use Clang."

The words of a single frontend developer made unofficially on someone's
blog post aren't very reassuring.  I'd rather hear the official response
of the current GCC head maintainer.

    Oleh

Re: IDE

[Dmitry Gutov]

On 10/12/2015 05:08 PM, Oleh Krehel wrote:

> Thanks for mentioning that. I wasn't sure that was the case.

It's basically MIT license, see 
https://directory.fsf.org/wiki/Atom#tab=Details.

> CEDET already resolves these complicated chains pretty well, as long as
> it's got the correct AST.

Semantic has some known issues in this general area, exactly because 
it's hard. Clang, on the other hand, resolves them well.

>> What if `bar' is declared as `auto' (see C++11)? Or `app' itself?
>
> This would be harder, but still very doable, even within CEDET.

You mean something would have to implement that. Good luck.

Another possible complication is using function templates in that method 
call chain.

> 1. Finding where the relevant headers are located on the file
> system. This means that the AST parser should hook into the currently
> used build system, and correctly see which #ifdef switches apply where.
> Only GCC or its likes have access to this info always. To reiterate, if
> the program compiles with GCC, the exact same switches must be passed to
> the AST parser, so that it parses exactly the same headers in exactly
> the same way.

This is non-trivial, because the switches are usually constructed by the 
build tool at run time. But see:

https://github.com/Sarcasm/irony-mode#compilation-database

> 2. Parsing the newly introduced language features. Here CEDET and
> cc-mode are behind because of the shear complexity of the full C++
> syntax. However, GCC does it very well.

Note that any CEDET feature that handles target type resolution is also 
subject to having to deal with new C++ language features.

> I've been working on function-args.el to extend CEDET C++ support for my
> uses for around 2 years now. The most common error that I have to work
> around is "Type definition not found", which happens either because the
> header path wasn't resolved, or the wrong #ifdef path was taken.

It could simply mean that until things work well on this level, you're 
not going to pay attention to the more subtle problems.

RE: IDE

[Drew Adams]

> I agree, the Microsoft IDE looks slicker here, but the examples
> are basically the same, in that they use separate frames for
> the completion list and documentation, not the same one.

Whether separate frames are used or not is not so important.
It is important, though, to be _able_ to show the completions
without necessarily showing the help for the current one.
IOW, separate display can be useful, whether or not separate
frames are used for that.

The Icicles approach lets you show the complete help for the
current candidate (systematically or on demand), but it does
not mandate that.  It happens to use separate windows (or
frames), but that is not important here, IMO - the same effect
could be provided with a single frame.

Icicles also automatically (by default - option turns on/off)
shows short help in the mode line (of `*Completions*'),
systematically.  Maybe that's all you had in mind, a one-liner?

If so, then users would be missing the ability to show the
complete help for the current candidate (again, systematically
or on demand).  Being able to do that is a big help, IMO.

Summary:

1. Users should be able to show candidates without also the help.
2. Whether the same or separate frames or windows are used is
   not important.  (It could be important to a given user, and
   it could be made a user choice.)
3. For simple, one-liner reminder help, the mode-line suffices.
4. Users should be able to show the complete help for the current
   candidate.  One-liner help is no substitute for this.

> > The first part is just Emacs' style of doing things: we usually
> > enter stuff in the minibuffer, so it makes sense for completion
> to display
> 
> A lot of users are fine with that, but I think we should do better.

What is better, and why?  Please don't gloss over this.

I would not argue that minibuffer input is always better than other
methods (e.g. at-point cycling).  But it has different pros & cons.

In particular, (1) it allows actual (and complete) editing, and
(2) it provides its own keymap, for completion features or cycling
features or candidate-action features.  (And users can easily
customize that keymap.)

Those are pros.  A con is that the minibuffer and `*Completions*'
are not necessarily displayed close to point.  That's a display
question that could be addressed in various ways.

> You'd think so, but displaying the docstring automatically, like
> Auto-Complete does (as well as certain IDEs), has been a common
> request for a while. And now, https://github.com/expez/company-
> quickhelp is pretty popular.

Does it display the complete doc string?  If not, I'd say users
are missing out.  If yes, and this is done systematically, I'd
say that users are being force-fed.  They should have a choice.

For systematic display (but it should still be "offable"),
one-line help is fine.  But it's not a replacement for being
able to see the whole doc string.

RE: IDE

[Drew Adams]

> > 3. Jumping around the project code and resources.
> > Jumping to around the project code and used libraries. Another
> > thing is jumping to/from resources (for example aspects can be
> > defined in an XML file and IDE could allow to jump to matching
> > classes).
> 
> Do you mean "jump to the thing at point"? That sounds complicated,
> and support for different "things" will have to be implemented
> separately.

Sounds like good ol' Emacs TAGS, to me (or across-project Imenu).
Of course, the limiting factor is TAGS files that support such
"things".  But the infrastructure is there for it.  People have
been using Emacs TAGS files to "jump to the [definition of the]
thing at point" for 40 years.

> > 4. Finding usages of fields, methods, types.
> > Helps to wrap head around project.
> 
> That's within the purview of xref, and up to CEDET or external
> tools to implement. But you can try xref-find-references now,
> for a quick-and-dirty Grep-based solution.

TAGS files are typically for definitions, but they can be for
anything, including "usages of fields, methods, types".  You
could have different TAGS files for each of these "usages",
and use (search) them selectively or together.

What has not been done is write the code to create such TAGS
files, AFAIK.  (Maybe Semantic helps here?)

Re: IDE

[Dmitry Gutov]

On 10/12/2015 05:39 PM, Drew Adams wrote:

> Whether separate frames are used or not is not so important.
> It is important, though, to be _able_ to show the completions
> without necessarily showing the help for the current one.
> IOW, separate display can be useful, whether or not separate
> frames are used for that.

If you're forced to use the same frame, you're forced to make the 
documentation pane and the completions menu the same height (or width), 
or just have an empty rectangle somewhere. That's not ideal.

Having separate display would also need work in that case.

> What is better, and why?  Please don't gloss over this.

Not having to always look at the minibuffer when entering stuff.

> Those are pros.  A con is that the minibuffer and `*Completions*'
> are not necessarily displayed close to point.  That's a display
> question that could be addressed in various ways.

Right.

> Does it display the complete doc string?  If not, I'd say users
> are missing out.  If yes, and this is done systematically, I'd
> say that users are being force-fed.  They should have a choice.

The user can disable the minor mode.

Re: IDE

[Tom]

Drew Adams <drew.adams <at> oracle.com> writes:
> 
> TAGS files are typically for definitions, but they can be for
> anything, including "usages of fields, methods, types".  You
> could have different TAGS files for each of these "usages",
> and use (search) them selectively or together.

What if different objects have fields or methods of the same name?

E.g. there is field called 'name' in lots of classes and I want to
find all usage of a name field, but only with certain object types
and there is code like:

   obj->name

?


TAGS files have considerable limitiations compared to techniques 
which actually understand the code. 

RE: IDE

[Drew Adams]

> > Whether separate frames are used or not is not so important.
> > It is important, though, to be _able_ to show the completions
> > without necessarily showing the help for the current one.
> > IOW, separate display can be useful, whether or not separate
> > frames are used for that.
> 
> If you're forced to use the same frame, you're forced to make the
> documentation pane and the completions menu the same height (or
> width), or just have an empty rectangle somewhere. That's not ideal.

I agree.  I personally use separate frames for them, and the
frames are fit to the content they display.

My point was that what is important is the ability to separate
their display - not couple them in a hardcoded way.  Whether
separate frames are used for that is a separate question, and
less important.  (But yes, I too prefer separate frames.)

Separate frames allow not only elimination of wasted space, for
the reason you gave.  They can also be overlapped, which can be
useful for additional space savings.  IOW, sometimes you don't
need to see all of the contents of each frame, and you want to
additionally see other stuff on your display at the same time.
Frames allow great flexibility for screen real estate.

> > What is better, and why?  Please don't gloss over this.
> 
> Not having to always look at the minibuffer when entering stuff.

If what you are entering is simple then you don't have to look
at it.  And if not then you still need to look at the place where
you are inputting/editing the pattern to complete.

I agree that it can be handy for that to be point in the main
buffer.  But a con in that scenario is the attendant "busyness"
of that area: input pattern editing + output completions & help
display.  And if you allow not just completion and help, but
also actions on the current candidate (a la Helm or Icicles)
then the display can become even busier in that area.

And for a complex input pattern, which might be multiline,
using a separate editing buffer (the minibuffer) is a plus, IMO.

> > Does it display the complete doc string?  If not, I'd say users
> > are missing out.  If yes, and this is done systematically, I'd
> > say that users are being force-fed.  They should have a choice.
> 
> The user can disable the minor mode.

What does that mean?  Does it (can it) display the complete doc
string?

RE: IDE

[Drew Adams]

> > TAGS files are typically for definitions, but they can be for
> > anything, including "usages of fields, methods, types".  You
> > could have different TAGS files for each of these "usages",
> > and use (search) them selectively or together.
> 
> What if different objects have fields or methods of the same name?
> 
> E.g. there is field called 'name' in lots of classes and I want to
> find all usage of a name field, but only with certain object types
> and there is code like:
> 
>    obj->name

That's why I mentioned creation of TAGS files that make such
distinctions.  Separate files for separate such usages, for
example.  (The files can be mixed-and-matched when used.)

A TAGS file is just an index.  It can index anything you like -
not just function and variable definitions.  Of course, as I said,
code to write such sophisticated TAGS files would need to be written.

> TAGS files have considerable limitiations compared to techniques
> which actually understand the code.

TAGS files are written by programs that "actually understand
the code".

The understanding of the existing tags-file-creation programs
is not up to the task.  Granted.  But a program that does
understand the code in a deeper way could write better TAGS
files.  That was my point.

IOW, the answer is not there yet, but TAGS files were designed
for precisely this use, and they have been used for it for a
long time.  What's needed is code that writes the TAGS files
needed today, for the languages and contexts used today.

Re: IDE

[Dmitry Gutov]

On 10/12/2015 06:02 PM, Drew Adams wrote:

> My point was that what is important is the ability to separate
> their display - not couple them in a hardcoded way.  Whether
> separate frames are used for that is a separate question, and
> less important.  (But yes, I too prefer separate frames.)

You should read the source code of Company. One can plug in a different 
frontend, and have different visualization as a result.

> Separate frames allow not only elimination of wasted space, for
> the reason you gave.  They can also be overlapped, which can be
> useful for additional space savings.  IOW, sometimes you don't
> need to see all of the contents of each frame, and you want to
> additionally see other stuff on your display at the same time.
> Frames allow great flexibility for screen real estate.

My frames are always fullscreen.

> And for a complex input pattern, which might be multiline,
> using a separate editing buffer (the minibuffer) is a plus, IMO.

Sure.

>> The user can disable the minor mode.
>
> What does that mean?  Does it (can it) display the complete doc
> string?

Everything is fine there, and the user is in control. Like I said, we 
have a separate command to display the doc in a new buffer.

Re: IDE

[Dmitry Gutov]

On 10/11/2015 08:37 PM, Eric Ludlam wrote:
> If you have a tool that parses already (such as exuberent
> CTags), you can instead just start at the high level tagging parser and
> skip all the lower level stuff.

Right.

> If you have an existing completion engine for cases where you happen to
> have an interpreter with completion, or something else, you can just
> override the completion engine directly.

You haven't answered the question about the advantage of doing it this 
way. If I override the completion engine directly, what main benefits of 
using CEDET are left? I mean, are they worth working on defining a 
grammar for the language, and keeping it up-to-date. A grammar can take 
a lot of effort by itself.

> The srecode tool does this,
> and there is an experimental clang version in CEDET's repository as well.

srecode overrides the completion engine? Why?

> Having overrides at all 3 levels is the best, since different tools ask
> for different features, but it is not necessary as there is backup
> implementation for the higher level features.

I agree that, in general, having overridable implementations is pretty 
great.

> Of course.  For strongly typed languages the problems are deterministic
> so it's possible.  For dynamic languages, they usually provide something
> you can ask.  It's just a matter of integration.
>
> I enjoy working on puzzles like this, but for me it is a matter of free
> time.  Kids, job, etc get in the way.

My impression is that nobody has solved this puzzle for any of the 
dynamic languages that CEDET aims to support. Which is not a definitive 
evidence either way, but it's pretty discouraging for a person 
interested in implementing better support for a dynamic language they're 
interested in.

> Semantic has the information to differentiate between two methods of
> different types.  There are still a few short-cuts in the completion
> engine  where it could do a little more to disambiguate.  There is a
> function to '-select-best-tag' in the analyzer, but at the end, it just
> grabs the first element from the list of possibilities.  If you have a
> particular use case in mind, it would good to have a simple example that
> shows what it is and maybe it will be easy to update.

Try this: https://gist.github.com/dgutov/19c45ef43d1c90b96483

I should get "a" and "b" as completions, but instead I get "c" and "d".

> This is a matter of time and desire.  For the code I've been writing
> lately (arduino hobby stuff), it has been more than sufficient. David
> was interested in handling templates and was able to add support for
> that.  If someone wants it to handle something new, the data is there to
> do it.

Yes, templates are pretty important. And complex. I don't know if 
re-implementing the code to understand them in Elisp is the way to go.

> What's missing is someone taking the time to start building the logic to
> use semantic to pull buffers apart, and srecode to re-write the code.

All right. So we could try to use Srecode interface for refactoring.

Re: IDE

[Eli Zaretskii]

> Date: Mon, 12 Oct 2015 05:45:43 -0400
> From: John Yates <john@yates-sheets.org>
> Cc: Richard Stallman <rms@gnu.org>, Emacs developers <emacs-devel@gnu.org>, Brief Busters <dgutov@yandex.ru>
> 
>     All Richard was saying that a debugger front-end is an important part of an
>     IDE.
> 
> Eli,
> 
> With all due respect, while you may interpret Richard's words in that manner,
> that is not what he wrote. I quoted at the top of my posting Richard's very
> first sentence that started this extended thread. Here it is again in its
> entirety.:
> 
> > Emacs with GUD is an IDE.

I'll let it to Richard to tell which interpretation, yours or mine, is
closer to what he meant.

> Given the amount of influence Richard will exert over efforts to fashion Emacs
> into something approximating a modern IDE I believe it is reasonable to wonder
> how familiar he is (or is willing to become) with such technologies.

We don't usually ask people here to present their credentials before
they are allowed to speak up on some issue.  Nor do I think we should.

FWIW, I never saw Richard speaking about something without knowing the
issues, or asking some experts.

> When we had the long thread some while back about supporting
> completion and refactoring I got the sense that Richard was
> unfamiliar with the functionality and user experience of modern
> IDEs. IIRC he committed to seeking some outside guidance which might
> have included becoming more familiar with the current state of
> typical IDEs. If that has yet to happen I wonder how John's
> anticipated f2f discussion with Richard will go.

I suggest we leave that to the 2 participants ;-)

Re: IDE

[Eli Zaretskii]

> From: Oleh Krehel <ohwoeowho@gmail.com>
> Date: Mon, 12 Oct 2015 13:05:02 +0200
> Cc: martin rudalics <rudalics@gmx.at>, Eli Zaretskii <eliz@gnu.org>,
> 	adatgyujto@gmail.com, emacs-devel@gnu.org
> 
> > For feature parity with Intellij IDEA and MS VS, we should be able to
> > display the list of completions and documentation for the currently
> > selected completion in two separate popups:
> >
> > https://i-msdn.sec.s-msft.com/dynimg/IC797655.jpeg
> > https://www.jetbrains.com/img/webhelp/idea/constructors_docs_in_completion.png
> 
> I like the first style a lot more. The second looks a lot like the ugly
> mess of Eclipse.
> 
> Here's what's currently possible in Emacs for C++:
> 
> - show function arguments and docstring in an overlay:
>     http://oremacs.com/download/fa-do-comments.png
> - complete class member at point:
>     http://oremacs.com/download/function-args-qt.png
> - jump to tag in directory:
>     http://oremacs.com/download/function-args-boost.png

Talking just about the visual appearance of these, IMO using overlays
limits us in our choice of faces, and fonts.  So I think it's better
to use tooltips or menus or maybe specialized small frames (which are
just tooltips with some limitations lifted and with automatic hide
action removed).

> The latter two can be done with powerful regexp-based completion, which
> MS VS likely still doesn't include.

This aspect is orthogonal to the visual appearance, I think.

> What's missing, compared to the MS VS picture:
> 
> - Candidate completion is in the minibuffer instead of at-point.
> - The docstring (only the comment part) is shown separately.
> 
> The first part is just Emacs' style of doing things: we usually enter
> stuff in the minibuffer, so it makes sense for completion to display
> there.  The second part is arguably unnecessary: I usually just jump to
> definition of symbol rather than look at the docstring inline.

IMO, we should support both the "traditional" Emacs style, and the
style users expect because other IDEs provide them.

> Another example is the QT code: it's a popular LGPL C++ framework that's
> currently hard to setup for CEDET.
> For instance, `#include <QtGui/QPushButton>` is a plain file without an 
> extension with only this code inside:
> 
>     #include "qpushbutton.h"
> 
> Since the extension isn't recognized, it's not parsed by CEDET.

Good C++ support indeed requires to be able to DTRT with
extension-less header files.  It would be good to add such a feature,
e.g. via magic-mode-alist and some creative regexp or match function.

> Could someone explain to me if making GCC the dependency of Emacs would
> be a good idea, from technical and freedom point of view?

You mean, invoke the compiler as part of some command?  No problem at
all (we actually do that already in a couple of commands).

Re: IDE

[Eli Zaretskii]

> Cc: martin rudalics <rudalics@gmx.at>, Eli Zaretskii <eliz@gnu.org>,
>  adatgyujto@gmail.com, emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Mon, 12 Oct 2015 14:47:53 +0300
> 
>     There were menus
>     available that did handle keyboard nav depending on toolkit, but those
>     blocked typing, so I couldn't use them for the inline display that might
>     popup automatically while you type.
> 
> Indeed. That's still the case, so using a menu for that is not an option.

I don't see why you would conclude this.  Imagine that each keystroke
pops down the menu and then immediately pops it up with the updated
display -- what is the problem with that in this scenario?

Re: IDE

[Eli Zaretskii]

> From: Oleh Krehel <ohwoeowho@gmail.com>
> Cc: martin rudalics <rudalics@gmx.at>,  Eli Zaretskii <eliz@gnu.org>,  adatgyujto@gmail.com,  emacs-devel@gnu.org
> Date: Mon, 12 Oct 2015 14:37:26 +0200
> 
> > Having the whole GCC as a dependency might be problematic, but that's
> > not the #1 problem. AFAIK, GCC currently has no "code completion"
> > feature anywhere.
> 
> There's no need for a specific "code completion" feature. Take this code
> for example:
> 
>     #include "qapplication.h"
>     #include "qfont.h"
>     #include "qpushbutton.h"
>     
>     int main(int argc, char *argv[]) {
>         QApplication app(argc, argv);
>         QPushButton quit("Quit");
>         quit.
>             
>         return 0;
>     }
> 
> Here's what (semantic-fetch-tags) returns:
> 
>     (("qapplication.h"
>       include
>       nil
>       nil
>       #<overlay from 1 to 26 in main.cc>)
>      ("qfont.h"
>       include
>       nil
>       nil
>       #<overlay from 27 to 45 in main.cc>)
>      ("qpushbutton.h"
>       include
>       nil
>       nil
>       #<overlay from 46 to 70 in main.cc>)
>      ("main"
>       function
>       (:arguments (("argc"
>                     variable
>                     (:type "int")
>                     (reparse-symbol arg-sub-list)
>                     #<overlay from 81 to 90 in main.cc>)
>                    ("argv"
>                     variable
>                     (:pointer 1
>                               :dereference 1
>                               :type "char")
>                     (reparse-symbol arg-sub-list)
>                     #<overlay from 91 to 104 in main.cc>))
>                   :type "int")
>       nil
>       #<overlay from 72 to 205 in main.cc>))
> 
> A similar data structure *has* to be somewhere in the GCC innards: it's
> a first step for compilation. In addition, this information is used to
> point out compilation errors/warnings.

See the documentation of the various -fdump-rtl-* switches to GCC.  Or
maybe you want the -dx switch.

Re: IDE

[Eli Zaretskii]

> From: Óscar Fuentes <ofv@wanadoo.es>
> Date: Mon, 12 Oct 2015 15:08:46 +0200
> 
> As mentioned on other message of mine, this is what a GCC frontend
> developer has to say (search for the comments of mlopezibanez):
> 
> https://lwn.net/Articles/629259/
> 
> Basically what he says is: "No, it is not simple at all to get that
> information from GCC, you'll better use Clang."

I'd urge interested and motivated individuals not to take one man's
opinions for more (or less) than it is.

I could fill several hours telling how many many opinions about what
could and what couldn't be done for bidi I heard before I decided to
sit down and do it.

My advice: read what others say, investigate the issues, then make up
your own mind.

Re: IDE

[Dmitry Gutov]

On 10/12/2015 06:55 PM, Eli Zaretskii wrote:

> I don't see why you would conclude this.  Imagine that each keystroke
> pops down the menu and then immediately pops it up with the updated
> display -- what is the problem with that in this scenario?

If that can work quickly enough, that minus ones problem.

But our menus are rather limited in terms of decoration. You asked for a 
popup with pictures and different fonts, I don't think we'd support it 
that way.

Re: IDE

[Eli Zaretskii]

> Cc: eric@siege-engine.com, deng@randomsample.de, rudalics@gmx.at,
>  adatgyujto@gmail.com, emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Mon, 12 Oct 2015 19:21:43 +0300
> 
> But our menus are rather limited in terms of decoration. You asked for a 
> popup with pictures and different fonts, I don't think we'd support it 
> that way.

OK, then I guess having small frames is the way to go.  But then we'd
need to come up with some alternative for text-mode terminals.

Re: IDE

[Dmitry Gutov]

On 10/12/2015 07:58 PM, Eli Zaretskii wrote:

> OK, then I guess having small frames is the way to go.  But then we'd
> need to come up with some alternative for text-mode terminals.

In terminal, we won't be able to use the decorations anyway (right?). So 
if the Lisp interface for graphical-mode menus is the same as for 
terminal menus, we can create a menu-based popup implementation anyway.

In that case, since it'll work everywhere, it would be a good first 
step, while small frames can be left for later.

Re: IDE

[Eli Zaretskii]

> Cc: eric@siege-engine.com, deng@randomsample.de, rudalics@gmx.at,
>  adatgyujto@gmail.com, emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Mon, 12 Oct 2015 20:26:44 +0300
> 
> On 10/12/2015 07:58 PM, Eli Zaretskii wrote:
> 
> > OK, then I guess having small frames is the way to go.  But then we'd
> > need to come up with some alternative for text-mode terminals.
> 
> In terminal, we won't be able to use the decorations anyway
> (right?).

On a TTY, there are no decorations, of course.  Worse,a frame you want
to display completely obscures all the other frames, which is
unacceptable for this kind of features.

> So if the Lisp interface for graphical-mode menus is the same as for
> terminal menus, we can create a menu-based popup implementation
> anyway.

Yes, the interface is the same: x-popup-menu.

> In that case, since it'll work everywhere, it would be a good first 
> step, while small frames can be left for later.

Right.

Re: IDE

[Steinar Bang]

>>>>> Eli Zaretskii <eliz@gnu.org>:

> IMO, we should support both the "traditional" Emacs style, and the
> style users expect because other IDEs provide them.

As a long-time emacs user that uses IDEs these days, I don't need emacs
to look like an IDE to go back to using it for all programming.

I need the functionality I'm missing when I'm back in emacs:
 - run tests interactively and get easy-to-navigate feedback from
   failing tests
 - completion (I plan to check out company mode that I learned about
   from this thread)
 - navigation, most used:
  - go to definition of a symbol
  - go to usages of a symbol
  - go to derived symbols (in languages with inheritance)
  - go base symbols (ditto)
 - for refactoring, renaming support would cover >90% of what I use,
   other refactoring support I use, are:
    - copy/cut/paste of methods and other class members from a outline
      (maybe that's already supported these days?)
    - move members up and down in class hierarchies (I don't use this
      all that much, but it's nice to have when I need it)

That's all I can think about, offhand.

Re: IDE

[Steinar Bang]

>>>>> Eli Zaretskii <eliz@gnu.org>:

> To me, an IDE is not a set of functionalities.  It's a coherent
> application that provides an IDE-like look-and-feel, and all the
> related functions already integrated and ready for me to be used.
> That includes window-layout, btw, because configuring Emacs windows
> for IDE-like behavior is an exceedingly complex task, one that's
> impossible without good command of ELisp.

Ah, there we differ.  I don't care about the layout and would feel
constrained in that layout.

But I do want the functionalities.

Re: IDE

[Eli Zaretskii]

> From: Steinar Bang <sb@dod.no>
> Date: Mon, 12 Oct 2015 20:12:22 +0200
> 
> >>>>> Eli Zaretskii <eliz@gnu.org>:
> 
> > To me, an IDE is not a set of functionalities.  It's a coherent
> > application that provides an IDE-like look-and-feel, and all the
> > related functions already integrated and ready for me to be used.
> > That includes window-layout, btw, because configuring Emacs windows
> > for IDE-like behavior is an exceedingly complex task, one that's
> > impossible without good command of ELisp.
> 
> Ah, there we differ.  I don't care about the layout and would feel
> constrained in that layout.
> 
> But I do want the functionalities.

The interesting question is what the youngsters want (or expect) these
days.  You and me are sold long ago.

Re: IDE

[David Kastrup]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Steinar Bang <sb@dod.no>
>> Date: Mon, 12 Oct 2015 20:12:22 +0200
>> 
>> >>>>> Eli Zaretskii <eliz@gnu.org>:
>> 
>> > To me, an IDE is not a set of functionalities.  It's a coherent
>> > application that provides an IDE-like look-and-feel, and all the
>> > related functions already integrated and ready for me to be used.
>> > That includes window-layout, btw, because configuring Emacs windows
>> > for IDE-like behavior is an exceedingly complex task, one that's
>> > impossible without good command of ELisp.
>> 
>> Ah, there we differ.  I don't care about the layout and would feel
>> constrained in that layout.
>> 
>> But I do want the functionalities.
>
> The interesting question is what the youngsters want (or expect) these
> days.  You and me are sold long ago.

So what?  I'm more interested in things that make Emacs better than
things that made Visual C++ better.  It's one thing to put lipstick on a
pig, but an octopus might not even have a good place for it.

-- 
David Kastrup

Re: IDE

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

It seems that Przemysław took offense at

  > I think you forgot profiling.

because the words "you forgot" suggested criticism of him.

It would be better (though less colorful) to say the same point with

  > I think we need profiling too.

because that is less likely to be misunderstood.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: IDE

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > What I want to explore is whether things can be better. We have great
  > unification in some areas, and I've a sense there are more opportunities
  > if we can build the right layers of abstraction.

I agree it is worth trying.  Sometimes it does manage to fly, and in those
cases it is a substantial improvement.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: IDE

[John Wiegley]

>>>>> Eric Ludlam <eric@siege-engine.com> writes:

>> From what I know about CEDET to date, it is much more complex than this
>> objective.

> That is because the task is complex.

The task is, but the solution needn't be, if structured well.

I'm not willing right now to say "CEDET is the answer", mostly because I've
had bad experiences trying to use it over the years. I want to return to
basics, define what is wanted, and then ask what layers are needed to get
there.

> The primary difference between your list and CEDET is that those other tools
> focus on picking up a current symbol, or perhaps a substring, and it is up
> to the next layer to figure out more about it. I agree that that could be
> simplified across the variosu tools, but AFAIK the 'thing-at-pt' stuff is
> that common interface. IDEs know more about the buffer than just some symbol
> and the major-mode of the current buffer.

thing-at-pt is likely a piece of the puzzle, but a small piece.

> If you want a stronger set of smart behaviours at point, you will need to
> raise your standard to include more derived data.

Correct.

I'll compile a more in-depth proposal for this idea, within the context of a
larger plan for IDE functionality -- if it falls to me to do so.

John

Re: IDE

[Przemysław Wojnowski]

W dniu 2015-10-12 13:46, Dmitry Gutov napisał(a):
>> For (project-oriented/enterprise) Java the features are:
>> 1. Project support
[...]
> Emacs master now has project.el, which will be a step in this
> direction.
I have seen that thread, but unfortunately haven't read it, yet.
It's waiting for the nearest holiday. :-|

> We can extend that API to contain information about where certain
> resources are, but for each type of projects there will be a different
> set of resources. While we could have a jump-to-test command (tests
> should be common), we can't easily have a commands for each kind of
> "jump to the web template for this controller" action.
IMHO defining relations between project elements should be delegated
to each type of project. For example Java Project knows where are
sources/tests/resources and can setup that using Project API.

Moreover in one project (lets call it Meta Project) there
should be a way to configure a set of language specific
subprojects, each one having its own backend(s) setup
(for code completion, docs, etc.).
A backend would be chosen by a mode in the current buffer (region?).

For example in a common "Java" webapp, the Meta Project setup could be:
{ languages:
   { java: {backend: classpath-indexer, build-tool: gradle, 
other-options: ...}
     javascript: {backend: tags-backend, build-tool: npm, ...}
     groovy: ...} }

> Support for build tools seems more straightforward, someone should
> just collect an overview of how users interact with different ones,
> like Make, Maven, Gradle, Rake, etc, to firmly establish the common
> ground.
IMHO better approach would be to provide an API that could be
used by build tool specific plugins to add build tasks
(in a Command Pattern manner). Such registered commands could be
presented to a user in some uniform form. For example:
In Maven plugin:
(build-api-add-command
   {name: "compile", command: function-ref})

When user selects a command the unified build tools runner does:
(defun build-api-run (command)
   (apply command.function-ref))

Of course, there can be more than one build tool in a project,
so, if windows/menus were presented, the user would see for each of them 
or maybe depending on current buffer's mode.
But the point is to provide an API not an implementation.

>> 2. Code completion
>>  From whole project, used libraries, and resources
>
> Unless CEDET delivers on that front, we're unlikely to have completion
> for Java in the near future in the core. But there are third-party
> packages that provide it.

IMHO, as many other things, this should be delegated to external tools 
that specializes in that. And CEDET should allow to use a set of 
external backends (at the same time).

One thing I'm not sure is whether CEDET should be used. IMHO yes, but 
AFAIR I've seen different voices on the list. So, not sure about that.

>> 3. Jumping around the project code and resources.
>> Jumping to around the project code and used libraries. Another thing
>> is
>> jumping to/from resources (for example aspects can be defined in an
>> XML file and IDE could allow to jump to matching classes).
>
> Do you mean "jump to the thing at point"? That sounds complicated, and
> support for different "things" will have to be implemented separately.
Yes, it would need to be implemented separately by each project type, 
because only project types know how to add semantic to different resources.
For example a Java project seeing Spring in dependencies may assume (or 
ask user) that applicationContext.xml file in actually Spring 
Application context, hence would know which elements are class names and 
then could use Java backend from the same project to find class' source.

>> 4. Finding usages of fields, methods, types.
>> Helps to wrap head around project.
>
> That's within the purview of xref, and up to CEDET or external tools
> to implement. But you can try xref-find-references now, for a
> quick-and-dirty Grep-based solution.
How to teach a backend (xref, CEDET, etc.) what my project is?
IMHO even good TAGS backend would be a good start if I, as an Emacs 
user, wouldn't need to configure it for a week and fight with each 
package to use it. This is where IDE steps in - it integrates.

>> 6. Running the tests (current, selected, all)
>> Tighten development feedback loop.
>
> Not sure which facility would be most fitting. A project *could*
> include metadata about how to run its tests better, but then the
> resulting buffer would need new compilation-error-regexp-alist
> entries,
Depending on language environment there may be different options. In 
Java world most build tools have plugins to run tests (single, group, 
all, etc.). So, Emacs Unified Test Runner could delegate that task to 
the tool and display the result (clearly the result can be displayed in 
many ways - other window being the simplest one, but the problem has 
been solved by MVX patterns decades ago).
Again, the test runner could provide an API for project types to 
register language specific runner Commands - this way a Java file may 
have, for example, two runners BuildToolRunner (that calls build tool 
command) and DirectJavaRunner (that instantiates and runs the test 
directly).

In the places where an API would be exposed I would see an EIEIO 
interface to make it explicit.

 > [...] and/or an ability to interact with a debugger/REPL if it's
 > triggered during the test run.
I have to think about that.


------------
To clarify, I've just listed features that I know are expected from a 
Java IDE. Some of them have implementations in Emacs/packages or 
external tools used by them. The point is that an IDE should _integrate_ 
that tools/packages, not end users.

At this moment whenever Emacs user tries to use a language she has to 
find a blog post that describes how to configure Emacs to be able to do 
something.

Re: IDE

[Dmitry Gutov]

On 10/13/2015 12:54 AM, Przemysław Wojnowski wrote:

>> Emacs master now has project.el, which will be a step in this
>> direction.
> I have seen that thread, but unfortunately haven't read it, yet.
> It's waiting for the nearest holiday. :-|

There have been several threads about it.

> IMHO defining relations between project elements should be delegated
> to each type of project. For example Java Project knows where are
> sources/tests/resources and can setup that using Project API.

Yes. So, what would do we do about the set of available commands?

> Moreover in one project (lets call it Meta Project) there
> should be a way to configure a set of language specific
> subprojects, each one having its own backend(s) setup
> (for code completion, docs, etc.).
> A backend would be chosen by a mode in the current buffer (region?).

I'm not sure about all that, but code completion and docs are outside of 
project.el's jurisdiction anyway.

> IMHO better approach would be to provide an API that could be
> used by build tool specific plugins to add build tasks
> (in a Command Pattern manner). Such registered commands could be
> presented to a user in some uniform form. For example:
> In Maven plugin:
> (build-api-add-command
>    {name: "compile", command: function-ref})

Mmm, yes. Allowing some code outside of the project definition to define 
the tasks and the way to run them might be better.

On the other hand, often the project definition file and the build file 
would be the same. So the project backend would have to read it anyway.

> Of course, there can be more than one build tool in a project,
> so, if windows/menus were presented, the user would see for each of them
> or maybe depending on current buffer's mode.

A build file (or several) is usually related to the project as a whole, 
so picking based on the current file might be suboptimal. On the other 
hand, we could present tasks from all build files combined.

> But the point is to provide an API not an implementation.

Sure. Please feel free to have a stab at defining it.

> IMHO, as many other things, this should be delegated to external tools
> that specializes in that. And CEDET should allow to use a set of
> external backends (at the same time).

Proposals which external tools exactly the Emacs core can use, will be 
welcome.

> How to teach a backend (xref, CEDET, etc.) what my project is?

An xref backend can call (project-current). But something (probably a 
minor mode) would have to set the appropriate xref backend.

> IMHO even good TAGS backend would be a good start if I, as an Emacs
> user, wouldn't need to configure it for a week and fight with each
> package to use it. This is where IDE steps in - it integrates.

The etags backend is currently the default one.

> Depending on language environment there may be different options. In
> Java world most build tools have plugins to run tests (single, group,
> all, etc.). So, Emacs Unified Test Runner could delegate that task to
> the tool and display the result (clearly the result can be displayed in
> many ways - other window being the simplest one, but the problem has
> been solved by MVX patterns decades ago).

Of course it would have to call the tool. That doesn't solve the 
questions of error highlighting and debugger integration.

> Again, the test runner could provide an API for project types to
> register language specific runner Commands - this way a Java file may
> have, for example, two runners BuildToolRunner (that calls build tool
> command) and DirectJavaRunner (that instantiates and runs the test
> directly).

I'd like to look at specific proposals for how this would look inside 
project.el, or some other, new file.

> To clarify, I've just listed features that I know are expected from a
> Java IDE. Some of them have implementations in Emacs/packages or
> external tools used by them. The point is that an IDE should _integrate_
> that tools/packages, not end users.

Sure, that would be better.

> At this moment whenever Emacs user tries to use a language she has to
> find a blog post that describes how to configure Emacs to be able to do
> something.

That might still be the case afterwards, to some extent, but at least 
after she's configured Emacs, the experience will be more consistent, 
and similar between different languages.

Re: IDE

[Lluís]

Eli Zaretskii writes:
[...]
>> For C/C++, the community has Irony and Rtags, both based on libclang. If 
>> libclang is unacceptable for you, you probably know a more appropriate 
>> mailing list to bring that up at.

> Let's not reiterate past discussions: you forget CEDET.

Just thinking out loud: it seems to me that many people forget that CEDET is,
from my understanding, a framework for writing tools first, and a set of such
example tools later.


> And if anyone _really_ cares about supporting C/C++, they should be
> working with and on GCC's libcc1, which is available for quite some
> time already.

If this is the libgcc1 you mean [1], I'm not sure it's suitable for code
completion. Instead, GCC should be modified to hook into the frontend parser or
the generic AST and then parsing that, which is no small feat. Fortunately,
hooking is already possible using GCC plugins [2].

[1] http://gcc.gnu.org/onlinedocs/gccint/Libgcc.html
[2] http://www.codesynthesis.com/~boris/blog/2010/05/03/parsing-cxx-with-gcc-plugin-part-1/

With this, it's a relatively easy (but time-consuming) task to build an external
tool that parses files on-demand. The ideal would be some kind of persistent
daemon+database, as was discussed in the CEDET list quite some time ago, but
that's an entirely different story.


[...]
>> Would you expect the programs mentioned above to become a part of Emacs? 

> I expect to see a coherent, orchestrated effort towards having an IDE
> mode in Emacs.  I don't see it, certainly not in discussions on this
> list.  I also don't see any commits that would provide evidence of
> such an effort.

> If such activities happen somewhere else, I would suggest their
> participants to come here and work with and within the core.  For
> starters, I don't imagine they would succeed without some significant
> changes and additions in the core infrastructure.  The place to
> discuss that is here.

I think that things are happening outside (completion, automatic project
detection, etc) because there is no common goal on what features should be
available and through what interface.

This, and that giving an opinion on these topics is way much less work than
actually implementing them (and I include myself on the first group of
non-implementors).


Cheers,
  Lluis

-- 
"And it's much the same thing with knowledge, for whenever you learn
something new, the whole world becomes that much richer."
-- The Princess of Pure Reason, as told by Norton Juster in The Phantom
Tollbooth

Re: IDE

[John Wiegley]

>>>>> Lluís  <xscript@gmx.net> writes:

> Eli Zaretskii writes:
> [...]
>>> For C/C++, the community has Irony and Rtags, both based on libclang. If
>>> libclang is unacceptable for you, you probably know a more appropriate
>>> mailing list to bring that up at.

>> Let's not reiterate past discussions: you forget CEDET.

CEDET first came out in 2003. If it were the answer to our present questions,
we would not be asking them.

I'm willing to hear how CEDET can offer solutions to issues we've brought up,
but I won't curtail the discussion "because CEDET".

I'm also OK with reiteration, because I wasn't present during those past
discussions, and I want to know what is most relevant today.

John

Re: IDE

[David Kastrup]

"John Wiegley" <johnw@newartisans.com> writes:

>>>>>> Lluís  <xscript@gmx.net> writes:
>
>> Eli Zaretskii writes:
>> [...]
>>>> For C/C++, the community has Irony and Rtags, both based on libclang. If
>>>> libclang is unacceptable for you, you probably know a more appropriate
>>>> mailing list to bring that up at.
>
>>> Let's not reiterate past discussions: you forget CEDET.
>
> CEDET first came out in 2003. If it were the answer to our present
> questions, we would not be asking them.

But since it did come out in 2003, we really should be asking _why_ it
isn't the answer to our present questions, in order to avoid the effort
of creating CEDET2 and CEDET3.

> I'm willing to hear how CEDET can offer solutions to issues we've
> brought up, but I won't curtail the discussion "because CEDET".

I don't think the idea is to curtail it but rather to _shape_ it.  If we
decide we need $x and CEDET provides $x, then either we haven't fully
figured out the details of the $x we need or CEDET does something wrong
when providing it.  Figuring out either will hopefully save us time in
arriving at something actually doing what we want.

-- 
David Kastrup

Re: IDE

[John Wiegley]

>>>>> David Kastrup <dak@gnu.org> writes:

> But since it did come out in 2003, we really should be asking _why_ it isn't
> the answer to our present questions, in order to avoid the effort of
> creating CEDET2 and CEDET3.

I certainly do want to avoid CEDET2.

> I don't think the idea is to curtail it but rather to _shape_ it. If we
> decide we need $x and CEDET provides $x, then either we haven't fully
> figured out the details of the $x we need or CEDET does something wrong when
> providing it. Figuring out either will hopefully save us time in arriving at
> something actually doing what we want.

I will not approach this by asking how CEDET can be improved to meet the needs
of an Emacs IDE. That is the most likely path leading to CEDET2.

Emacs' needs as an IDE should be considered on their own, as I've said before.
Any or all existing methodologies can be taken into account, but none deserve
preference until an architecture begins to take shape.

John

Re: IDE

[Eric Ludlam]

On 10/12/2015 11:18 AM, Dmitry Gutov wrote:
> On 10/11/2015 08:37 PM, Eric Ludlam wrote:
>> If you have a tool that parses already (such as exuberent
>> CTags), you can instead just start at the high level tagging parser and
>> skip all the lower level stuff.
>
> Right.
>
>> If you have an existing completion engine for cases where you happen to
>> have an interpreter with completion, or something else, you can just
>> override the completion engine directly.
>
> You haven't answered the question about the advantage of doing it this 
> way. If I override the completion engine directly, what main benefits 
> of using CEDET are left? I mean, are they worth working on defining a 
> grammar for the language, and keeping it up-to-date. A grammar can 
> take a lot of effort by itself.

The primary reason is that having tag information in a buffer so you can 
access it quickly is helpful.

The reason you'd want a tag-level parser at all is to provide:
1) a database of tags in the buffer, plus positional information
2) a database of tags across a project to search through
3) a standard way of knowing where you are in relation to other tags

Simple things like showing the function you are editing, highlighting 
tags with various features in different ways,or knowing what class the 
method you are in are handy and quick little features that can be built 
generically on top of CEDET, but which require piles of code to do 
individually without that type of support. imenu, etags, ctags, global, 
ident, etc all exist because it is useful, but none of those tools get 
bound into a buffer, so their level of usefulness is limited to "jump to 
a location" instead of handy inline features.

>> The srecode tool does this,
>> and there is an experimental clang version in CEDET's repository as 
>> well.
>
> srecode overrides the completion engine? Why?
>

Because it is a multi-mode buffer, so sometimes you want to complete 
srecode symbols, and sometimes you want to complete from a different 
language.

Eric

Re: IDE

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > So what?  I'm more interested in things that make Emacs better than
  > things that made Visual C++ better.  It's one thing to put lipstick on a
  > pig, but an octopus might not even have a good place for it.

This is true as a general statement, but in general I hope we can
find ways to integrate into Emacs the useful or appealing features
of other IDEs.  Let's at least try to find a way to make them fit,
before we dismiss the idea.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: IDE

[Eric Ludlam]

On 10/12/2015 07:05 AM, Oleh Krehel wrote:
> What's missing, in my opinion, is only faster and more precise parser
> (CEDET, GCC etc). For example, currently `semantic-fetch-tags' parses
> public/private/protected labels as tags, instead of applying these
> properties to actual tags. If that were so, it would be very easy to add
> a public/private/protected icon to each tag, just like MS VS does it.

The parser saves the buffer as close as it can - that allows it to be 
regenerated later.  If you use the 'semantic-format-*' functions, such 
as the uml version, it will identify the protection and use the right 
symbology.  If you are writing the code that calls the formatter, you 
need to specify the parent tag.

> Another example is the QT code: it's a popular LGPL C++ framework that's
> currently hard to setup for CEDET.
> For instance, `#include <QtGui/QPushButton>` is a plain file without an
> extension with only this code inside:
>
>      #include "qpushbutton.h"
>
> Since the extension isn't recognized, it's not parsed by CEDET.  And I
> have to write `#include "qpushbutton.h"` in my application instead of
> the more preferred `#include <QtGui/QPushButton>`, because that way I
> get tag completion.

You can solve this by adding the qt include directory to 
auto-mode-alist.  There is a workaround posted in emacswiki roughly like 
this:

(setq qt4-base-dir "/usr/include/qt4")
(setq qt4-gui-dir (concat qt4-base-dir "/QtGui"))
(semantic-add-system-include qt4-base-dir 'c++-mode)
(semantic-add-system-include qt4-gui-dir 'c++-mode)
(add-to-list 'auto-mode-alist (cons qt4-base-dir 'c++-mode))

There are a few extra steps for Qt preprocessor symbols and more, but 
the above lets you avoid the no extension problem.

> Could someone explain to me if making GCC the dependency of Emacs would
> be a good idea, from technical and freedom point of view?  Personally, I
> wouldn't care if Emacs executable would get inflated a bit more, if that
> meant access to true IDE features, which are only possible with a
> precise and fast parser.

There are folks using CEDET without gcc on their system, or at least, 
they've needed configuration help with alternate compilers.

Eric

Re: IDE

[Eric Ludlam]

On 10/12/2015 05:54 PM, Przemysław Wojnowski wrote:
> IMHO defining relations between project elements should be delegated
> to each type of project. For example Java Project knows where are
> sources/tests/resources and can setup that using Project API.
>
> Moreover in one project (lets call it Meta Project) there
> should be a way to configure a set of language specific
> subprojects, each one having its own backend(s) setup
> (for code completion, docs, etc.).
> A backend would be chosen by a mode in the current buffer (region?).
>
> For example in a common "Java" webapp, the Meta Project setup could be:
> { languages:
>    { java: {backend: classpath-indexer, build-tool: gradle,
> other-options: ...}
>      javascript: {backend: tags-backend, build-tool: npm, ...}
>      groovy: ...} }
>
>> Support for build tools seems more straightforward, someone should
>> just collect an overview of how users interact with different ones,
>> like Make, Maven, Gradle, Rake, etc, to firmly establish the common
>> ground.
> IMHO better approach would be to provide an API that could be
> used by build tool specific plugins to add build tasks
> (in a Command Pattern manner). Such registered commands could be
> presented to a user in some uniform form. For example:
> In Maven plugin:
> (build-api-add-command
>    {name: "compile", command: function-ref})
>
> When user selects a command the unified build tools runner does:
> (defun build-api-run (command)
>    (apply command.function-ref))
>
> Of course, there can be more than one build tool in a project,
> so, if windows/menus were presented, the user would see for each of them
> or maybe depending on current buffer's mode.
> But the point is to provide an API not an implementation.

This is how EDE (a part of CEDET) is setup and works.  There are 
"projects", and in projects there are "targets".  There are project 
build commands, and target build commands.  Each project or target can 
have language specific features for setting up CEDET's parsers.

There is a set of different base classes for projects, and many 
specializations for various flavors of java projects such as maven and 
ant, C++ projects, lisp projects, and more.

Many folks besides myself have built support for different kinds of 
projects, so extending to new types isn't too hard.

Eric

Re: IDE

[Eric Ludlam]

On 10/13/2015 09:02 AM, Lluís wrote:
>> Let's not reiterate past discussions: you forget CEDET.
> Just thinking out loud: it seems to me that many people forget that CEDET is,
> from my understanding, a framework for writing tools first, and a set of such
> example tools later.

Thanks for the reminder.  I forgot to bring that part up in this 
iteration of the conversation.

>> >And if anyone_really_  cares about supporting C/C++, they should be
>> >working with and on GCC's libcc1, which is available for quite some
>> >time already.
> If this is the libgcc1 you mean [1], I'm not sure it's suitable for code
> completion. Instead, GCC should be modified to hook into the frontend parser or
> the generic AST and then parsing that, which is no small feat. Fortunately,
> hooking is already possible using GCC plugins [2].
>
> [1]http://gcc.gnu.org/onlinedocs/gccint/Libgcc.html
> [2]http://www.codesynthesis.com/~boris/blog/2010/05/03/parsing-cxx-with-gcc-plugin-part-1/
>
> With this, it's a relatively easy (but time-consuming) task to build an external
> tool that parses files on-demand. The ideal would be some kind of persistent
> daemon+database, as was discussed in the CEDET list quite some time ago, but
> that's an entirely different story.

Yes, I remember those discussions. I was able to put in examples of an 
external parser by integrating exuberent ctags which is a bit weak for 
CEDET needs, but was never able to get any help figuring out how to make 
gcc or any other compiler provide the data I wanted.  I tried some stuff 
with using debug symbols and gdb, but that was a flop.  The ectags 
example showed marked performance improvement by delegating parsing 
outside the Emacs process.

A second area was an external database for looking up symbols.  I pulled 
in GNU Global and a couple others as proof of concept examples, but 
never got around to anything custom, as that was a bit bigger task.  The 
lack of an external parser to use with it made the project more daunting.

Eric

Re: IDE

[Eric Ludlam]

On 10/13/2015 12:28 PM, David Kastrup wrote:
> "John Wiegley" <johnw@newartisans.com> writes:
>
>>>>>>> Lluís  <xscript@gmx.net> writes:
>>
>>> Eli Zaretskii writes:
>>> [...]
>>>>> For C/C++, the community has Irony and Rtags, both based on libclang. If
>>>>> libclang is unacceptable for you, you probably know a more appropriate
>>>>> mailing list to bring that up at.
>>
>>>> Let's not reiterate past discussions: you forget CEDET.
>>
>> CEDET first came out in 2003. If it were the answer to our present
>> questions, we would not be asking them.
>
> But since it did come out in 2003, we really should be asking _why_ it
> isn't the answer to our present questions, in order to avoid the effort
> of creating CEDET2 and CEDET3.

Based on the many emails I've seen on the topic, I suspect the answer is:

* It is hard to configure (ie - setting up project files,
   include paths, or whatever.)
* Specific implementations are incomplete (ie - c++ || other parser is
   imperfect, the project system doesn't implement some feature, etc)
* It is compared against better staffed tools

>> I'm willing to hear how CEDET can offer solutions to issues we've
>> brought up, but I won't curtail the discussion "because CEDET".
>
> I don't think the idea is to curtail it but rather to _shape_ it.  If we
> decide we need $x and CEDET provides $x, then either we haven't fully
> figured out the details of the $x we need or CEDET does something wrong
> when providing it.  Figuring out either will hopefully save us time in
> arriving at something actually doing what we want.

My main concern is about folks claiming CEDET is complicated (which it 
is) then oversimplifying the problem space to kick off some new thing 
which will likely end up just as complicated.

I know I thought the problem space seemed simple when I started.  I 
might not have started if I'd known how big it is.

Eric

Re: IDE

[John Wiegley]

>>>>> Eric Ludlam <eric@siege-engine.com> writes:

> My main concern is about folks claiming CEDET is complicated (which it is)
> then oversimplifying the problem space to kick off some new thing which will
> likely end up just as complicated.

> I know I thought the problem space seemed simple when I started. I might not
> have started if I'd known how big it is.

I appreciate your experience and point of view in this, Eric, and definitely
want you to be a part of this conversation. This same phenomenon happens with
build tools: there's a new one almost every year, because everyone thinks
they're trivial, yet no one has ever fully solved the problem.

John

Re: IDE

[Steinar Bang]

>>>>> Richard Stallman <rms@gnu.org>:

> This is true as a general statement, but in general I hope we can find
> ways to integrate into Emacs the useful or appealing features of other
> IDEs.  Let's at least try to find a way to make them fit, before we
> dismiss the idea.

The IDE features I miss most in emacs, is:
 - Auto complete
 - Navigation (definition of symbol, usages of symbol)
 - Renaming support
 - Outline cut/copy/paste

All of these features can be implemented in emacs without needing to
support (or enforce) an IDE-like layout.

What I don't like in other IDEs is the need to use a mouse to switch between
buffers and change focus between IDE windows, and this is a place where
emacs shines: I never _need_ to use the mouse

My languages are currently Java and Python (the latter I use emacs for
already.  The former I use emacs for formatting cleanup, and commit, and
large scale text subistitution 

Re: IDE

[David Kastrup]

Eric Ludlam <eric@siege-engine.com> writes:

> On 10/13/2015 12:28 PM, David Kastrup wrote:
>> "John Wiegley" <johnw@newartisans.com> writes:
>>
>>>>>>>> Lluís  <xscript@gmx.net> writes:
>>>
>>>> Eli Zaretskii writes:
>>>> [...]
>>>>>> For C/C++, the community has Irony and Rtags, both based on libclang. If
>>>>>> libclang is unacceptable for you, you probably know a more appropriate
>>>>>> mailing list to bring that up at.
>>>
>>>>> Let's not reiterate past discussions: you forget CEDET.
>>>
>>> CEDET first came out in 2003. If it were the answer to our present
>>> questions, we would not be asking them.
>>
>> But since it did come out in 2003, we really should be asking _why_ it
>> isn't the answer to our present questions, in order to avoid the effort
>> of creating CEDET2 and CEDET3.
>
> Based on the many emails I've seen on the topic, I suspect the answer is:
>
> * It is hard to configure (ie - setting up project files,
>   include paths, or whatever.)
> * Specific implementations are incomplete (ie - c++ || other parser is
>   imperfect, the project system doesn't implement some feature, etc)
> * It is compared against better staffed tools

I got rid of it because it tended to eat all my CPU repeatedly digging
through buffers and files in the background.  I don't want some tool to
go treasure-hunting for hours in my directories without concrete cause,
then restart for inscrutable reasons.

It had its own idea of projects not matching the projects I was working
with, and it's an absolute no-go for Emacs to meddle with project
organization: I want to be able to jump in with Emacs into any project
without any pre- or post-configuration.

Maybe that's a decisive difference between what people got to expect
from an IDE and I expect from Emacs: if someone develops stuff in Visual
C++, everybody in the project is expected to use the project
organization tools of the Visual C++ IDE.  But I don't want my choice of
Emacs as an editor bleed all over a project.

Now you'll say that EDE (or Semantic, or whatever other component) is
entirely optional but it's hard to figure out just what the relations of
the various parts of CEDET are.  If you want to just work with the code
you have and not get stuff messed up, at some point of time it's easier
to just forego the whole inscrutable package and simplify one's life.

Again, that's a main difference to what a normal IDE is doing: it tends
to focus on a small set of languages and does them well when I buy into
the IDE, and I can use IDE features as needed.

But my buy-in was to Emacs.  I don't want to buy into a competing
framework CEDET.  If I want completion, I enable an option or package
for it, and I don't want it to come with a host of things I have no idea
how to keep from messing with my work environment.  It's nice when there
is some framework with which major mode writers can easily provide a lot
of functionality commonly expected of IDEs.  But CEDET appears to be
mainly a user choice, and it leaves the user with the job of maintaining
Emacs/CEDET integrity for his workflows.

And it did not particularly help that seminal parts of CEDET like its
parser generators were kept out of Emacs for very, very long: you needed
to install a third-party CEDET in order to even be able to maintain some
Emacs-internal modes.

-- 
David Kastrup

Re: IDE

[Dmitry Gutov]

On 10/14/2015 06:16 AM, Eric Ludlam wrote:

> My main concern is about folks claiming CEDET is complicated (which it
> is) then oversimplifying the problem space to kick off some new thing
> which will likely end up just as complicated.

My already-stated impression is that it's over-specialized and tightly 
coupled.

Not saying that the problem domain is easy, but being able to use 
different pieces of the solution separately would go a long way towards 
alleviating the complaint that certain other parts are incomplete.

Especially if it were easier to swap in different solutions for some of 
those parts (and do entirely without some others), and do that in not 
too many lines, all as part of the user's configuration.

Re: IDE

[Dmitry Gutov]

On 10/14/2015 05:45 AM, Eric Ludlam wrote:

> This is how EDE (a part of CEDET) is setup and works.  There are
> "projects", and in projects there are "targets".  There are project
> build commands, and target build commands.  Each project or target can
> have language specific features for setting up CEDET's parsers.

Is there a particular reason to have the notion of "target" in the 
project API? If the need is to simply disambiguate commands with the 
same name, the commands could be prefixed with the target name, e.g. 
"release:compile", "release:test".

> There is a set of different base classes for projects, and many
> specializations for various flavors of java projects such as maven and
> ant, C++ projects, lisp projects, and more.

What do you do if several different project types use the same build 
system (and so the logic to parse the build targets is the same)?

What would you do if a certain project type can be used with different 
build systems? Create an inheriting sub-type for each of them?

That approach looks worrying if we get several varying pieces of 
behavior like that: for example, different build tools and different 
test frameworks.

Re: IDE

[Eric Ludlam]

On 10/14/2015 04:09 AM, David Kastrup wrote:
> Eric Ludlam <eric@siege-engine.com> writes:
>
>> On 10/13/2015 12:28 PM, David Kastrup wrote:
>> Based on the many emails I've seen on the topic, I suspect the answer 
>> is: * It is hard to configure (ie - setting up project files, include 
>> paths, or whatever.) * Specific implementations are incomplete (ie - 
>> c++ || other parser is imperfect, the project system doesn't 
>> implement some feature, etc) * It is compared against better staffed 
>> tools 
> I got rid of it because it tended to eat all my CPU repeatedly digging
> through buffers and files in the background.  I don't want some tool to
> go treasure-hunting for hours in my directories without concrete cause,
> then restart for inscrutable reasons.
>
> It had its own idea of projects not matching the projects I was working
> with, and it's an absolute no-go for Emacs to meddle with project
> organization: I want to be able to jump in with Emacs into any project
> without any pre- or post-configuration.
Thanks for the diverse feedback.

The need to move your files into some new structure is something I've 
always avoided.  There is a "do it for you" project structure if you 
don't care, and several other types that just uses what you have, and 
can detect a bunch of variants without leaving its own files behind.

>
> Maybe that's a decisive difference between what people got to expect
> from an IDE and I expect from Emacs: if someone develops stuff in Visual
> C++, everybody in the project is expected to use the project
> organization tools of the Visual C++ IDE.  But I don't want my choice of
> Emacs as an editor bleed all over a project.
>
> Now you'll say that EDE (or Semantic, or whatever other component) is
> entirely optional but it's hard to figure out just what the relations of
> the various parts of CEDET are.  If you want to just work with the code
> you have and not get stuff messed up, at some point of time it's easier
> to just forego the whole inscrutable package and simplify one's life

The puzzle for me here is that while the different pieces are 
technically independent, the more complex tasks, such as completion, 
depend on the other tools doing their job.  Good smart completion 
depends on a knowledge of a project's structure to find headers (C/C++), 
and it also depends on rummaging around in your files to find the needed 
symbols.

AFAIK, every smart completion engine out there has the same dependency.  
There are plenty of others that don't, like Global which just finds 
what's there and makes the most of it, but it isn't smart completion.

I suspect what you'd really like is to say "yeah, I'd like some smart 
completion with a side of API doc", and have an auto-configure thingy do 
the rest.  Sounds great!  To make that happen though, we need Emacs to 
be taught how to detect your files and rummage through them to make it 
happen.  If you work on code of a style I or other contributors never 
worked on, it probably isn't in CEDET.

Pulling in external tools like gcc, clang, whatever to do the work is a 
great way to make that happen as it pushes the CPU work off of Emacs' 
thread, and in some cases brings knowledge of the project along with 
it.  Doing that type of integration can be done with CEDET's framework, 
or independent of it.  I am not advocating to not do that type of 
integration, but to consider doing it in CEDET's framework because:

a) it will be easier than starting from scratch
b) doesn't preclude other types of integration later

On 10/14/2015 06:47 AM, Dmitry Gutov wrote:
> My already-stated impression is that it's over-specialized and tightly 
> coupled.
>

There are definitely dependencies.  I don't think it is 
over-specialized, but perhaps overly-generalized.  Every layer was set 
up so new languages, modes, projects, whatever can be slotted into the 
system.  The tendency is that many are not complete which lends itself 
to disappointment.  This is not uncommon in Emacs. There are lots of 
modes floating around with no indentation, poor syntax tables and 
incomplete highlighting.

> Not saying that the problem domain is easy, but being able to use 
> different pieces of the solution separately would go a long way 
> towards alleviating the complaint that certain other parts are 
> incomplete.
>

I agree, this would be nice.

> Especially if it were easier to swap in different solutions for some 
> of those parts (and do entirely without some others), and do that in 
> not too many lines, all as part of the user's configuration.

It is possible to swap in different solutions (under the CEDET 
framework) but in many cases, there is currently only one solution.

In these conversations it is hard to distinguish if the (usually valid) 
criticism are about CEDET the framework, or about various 
implementations under CEDET.

It is also hard since I don't really have time to address the issues raised.

Eric

Re: IDE

[Alexis]

Dmitry Gutov <dgutov@yandex.ru> writes:

> On 10/14/2015 05:45 AM, Eric Ludlam wrote:
>
>> This is how EDE (a part of CEDET) is setup and works.  There 
>> are "projects", and in projects there are "targets".  There are 
>> project build commands, and target build commands.  Each 
>> project or target can have language specific features for 
>> setting up CEDET's parsers.
>
> Is there a particular reason to have the notion of "target" in 
> the  project API?

i've been wondering this too; in a post to this list in August:

    https://lists.gnu.org/archive/html/emacs-devel/2015-08/msg00278.html

i wrote:
 
> Eric Ludlam <address@hidden> writes: 
> 
>> I'm also interested in how to resolve some of the 
>> misconceptions about EDE. Perhaps looking at what prevents it 
>> from being 'on by default', which I'm sure is a long list, is a 
>> good way to move it in the right direction, even if it never is 
>> on by default. 
> 
> Well, say i'm developing a small Perl5 project - one main 
> module, supported by two helper modules, all of which are pure 
> Perl. When i look at the overview of EDE in the Emacs manual: 
> 
>    https://www.gnu.org/software/emacs/manual/html_node/emacs/EDE.html 
> 
> and read:  
>    A project may contain one or more targets. A target can be an 
>    object file, executable program, or some other type of file, 
>    which is “built” from one or more of the files in the 
>    project. 
> 
> i think to myself: "Okay, so EDE might be able to help me build 
> (say) a .tar.gz when i feel the project is ready for release and 
> distribution, but it's not relevant prior to that stage, since 
> the source .pm files /are/ the executable files. There's nothing 
> to 'build' during development and testing, so it seems i have no 
> use for EDE during those phases." 
> 
> i would also guess that, similarly, EDE might not be that useful 
> when developing in languages such as ELisp, Python or Ruby. 
> 
> Am i wrong? What would using EDE get me in such contexts?


Alexis.

Re: IDE

[Stephen Leake]

David Kastrup <dak@gnu.org> writes:

> Eric Ludlam <eric@siege-engine.com> writes:
>
>> On 10/13/2015 12:28 PM, David Kastrup wrote:
>>> "John Wiegley" <johnw@newartisans.com> writes:
>>>
>>>>>>>>> Lluís  <xscript@gmx.net> writes:
>>>>
>>>>> Eli Zaretskii writes:
>>>>> [...]
>>>>>>> For C/C++, the community has Irony and Rtags, both based on libclang. If
>>>>>>> libclang is unacceptable for you, you probably know a more appropriate
>>>>>>> mailing list to bring that up at.
>>>>
>>>>>> Let's not reiterate past discussions: you forget CEDET.
>>>>
>>>> CEDET first came out in 2003. If it were the answer to our present
>>>> questions, we would not be asking them.
>>>
>>> But since it did come out in 2003, we really should be asking _why_ it
>>> isn't the answer to our present questions, in order to avoid the effort
>>> of creating CEDET2 and CEDET3.
>>
>> Based on the many emails I've seen on the topic, I suspect the answer is:
>>
>> * It is hard to configure (ie - setting up project files,
>>   include paths, or whatever.)
>> * Specific implementations are incomplete (ie - c++ || other parser is
>>   imperfect, the project system doesn't implement some feature, etc)
>> * It is compared against better staffed tools
>
> I got rid of it because it tended to eat all my CPU repeatedly digging
> through buffers and files in the background.  I don't want some tool to
> go treasure-hunting for hours in my directories without concrete cause,
> then restart for inscrutable reasons.
>
> It had its own idea of projects not matching the projects I was working
> with, and it's an absolute no-go for Emacs to meddle with project
> organization: I want to be able to jump in with Emacs into any project
> without any pre- or post-configuration.
>
> Maybe that's a decisive difference between what people got to expect
> from an IDE and I expect from Emacs: if someone develops stuff in Visual
> C++, everybody in the project is expected to use the project
> organization tools of the Visual C++ IDE.  But I don't want my choice of
> Emacs as an editor bleed all over a project.

That means CEDET needs to recognize your Visual C++ project, just like
the Visual C++ IDE does. CEDET does not currently support this.

> Now you'll say that EDE (or Semantic, or whatever other component) is
> entirely optional but it's hard to figure out just what the relations of
> the various parts of CEDET are.  If you want to just work with the code
> you have and not get stuff messed up, at some point of time it's easier
> to just forego the whole inscrutable package and simplify one's life.

You seem to be implying that something in CEDET was changing things on
the disk without your permission; is that what you are actually saying?

> Again, that's a main difference to what a normal IDE is doing: it tends
> to focus on a small set of languages and does them well when I buy into
> the IDE, and I can use IDE features as needed.

It's more than just the language; it's also the build tools and cross
reference tools, and the associated configuration files.

-- 
-- Stephe

Re: IDE

[David Kastrup]

Stephen Leake <stephen_leake@stephe-leake.org> writes:

> David Kastrup <dak@gnu.org> writes:
>>
>> I got rid of it because it tended to eat all my CPU repeatedly digging
>> through buffers and files in the background.  I don't want some tool to
>> go treasure-hunting for hours in my directories without concrete cause,
>> then restart for inscrutable reasons.
>>
>> It had its own idea of projects not matching the projects I was working
>> with, and it's an absolute no-go for Emacs to meddle with project
>> organization: I want to be able to jump in with Emacs into any project
>> without any pre- or post-configuration.
>>
>> Maybe that's a decisive difference between what people got to expect
>> from an IDE and I expect from Emacs: if someone develops stuff in Visual
>> C++, everybody in the project is expected to use the project
>> organization tools of the Visual C++ IDE.  But I don't want my choice of
>> Emacs as an editor bleed all over a project.
>
> That means CEDET needs to recognize your Visual C++ project, just like
> the Visual C++ IDE does. CEDET does not currently support this.

Uh, no?  I don't think I ever used Visual C++.  Projects I work on use
Makefiles if anything.  Or some other build infrastructure.

So if CEDET needs project information, its first idea of getting it
should be to look at some toplevel Makefiles and, if it finds them, ask
GNU Make for dependencies and stuff and probably look at a few standard
Make targets.  That's what to expect in GNU projects, the most important
clientele.

But I don't think it should ever get foraging for stuff on its own.

>> Now you'll say that EDE (or Semantic, or whatever other component) is
>> entirely optional but it's hard to figure out just what the relations
>> of the various parts of CEDET are.  If you want to just work with the
>> code you have and not get stuff messed up, at some point of time it's
>> easier to just forego the whole inscrutable package and simplify
>> one's life.
>
> You seem to be implying that something in CEDET was changing things on
> the disk without your permission; is that what you are actually
> saying?

No.  I was saying

>> I got rid of it because it tended to eat all my CPU repeatedly
>> digging through buffers and files in the background.  I don't want
>> some tool to go treasure-hunting for hours in my directories without
>> concrete cause, then restart for inscrutable reasons.

Is there any reason to assume I mean something different when I write
stuff like that?

>> Again, that's a main difference to what a normal IDE is doing: it
>> tends to focus on a small set of languages and does them well when I
>> buy into the IDE, and I can use IDE features as needed.
>
> It's more than just the language; it's also the build tools and cross
> reference tools, and the associated configuration files.

Whatever.  I wrote why I got rid of CEDET, you answer that the world is
difficult for an IDE.  That's nice but irrelevant.  I'm fine with only
requiring those services of an IDE which it can provide without being
painful to me.  If I don't get any obvious way to choose, if it's "take
it all or leave it", then the probability is that I'll settle on the
"leave it" option.

-- 
David Kastrup

Re: IDE

[Dmitry Gutov]

On 10/14/2015 03:14 PM, Alexis wrote:

>> i would also guess that, similarly, EDE might not be that useful when
>> developing in languages such as ELisp, Python or Ruby.

That has been my impression as well.

Re: IDE

[Eric Ludlam]

On 10/14/2015 07:42 AM, Dmitry Gutov wrote:
> On 10/14/2015 05:45 AM, Eric Ludlam wrote:
>
>> This is how EDE (a part of CEDET) is setup and works.  There are
>> "projects", and in projects there are "targets".  There are project
>> build commands, and target build commands.  Each project or target can
>> have language specific features for setting up CEDET's parsers.
>
> Is there a particular reason to have the notion of "target" in the 
> project API? If the need is to simply disambiguate commands with the 
> same name, the commands could be prefixed with the target name, e.g. 
> "release:compile", "release:test".

Historically, the 'targets' matched the makefile targets, and was used 
to generate a Makefile from a configuration.

For the other projects, the targets simply group source code together so 
if you use the 'compile' key sequence, you get something appropriate for 
that group of source files.  Sometime it is very simple so that .texi or 
.cpp execute different compile steps. Sometimes there is no difference.

>> There is a set of different base classes for projects, and many
>> specializations for various flavors of java projects such as maven and
>> ant, C++ projects, lisp projects, and more.
>
> What do you do if several different project types use the same build 
> system (and so the logic to parse the build targets is the same)?
>
> What would you do if a certain project type can be used with different 
> build systems? Create an inheriting sub-type for each of them?
>
> That approach looks worrying if we get several varying pieces of 
> behavior like that: for example, different build tools and different 
> test frameworks.
> .
>

It's fine for several project systems to do the same thing.  They could 
share some implementation or not, depending the way any set of lisp 
programs might.  Many shared behaviours are pushed up in the class 
hierarchy.  For example, handling include paths, java classpath, etc.  
For most things like 'compile', the similar code is about appending the 
string "make " with some target, or whatever it might be, so it isn't 
too deep.

There are some very simple projects in EDE, such as "emacs" which just 
knows how to find emacs.c and mark that directory tree as a project, and 
it knows how to assemble some compile and debug commands.  It also knows 
how to setup C preprocessor symbols and include paths.  Other project 
types are very complex, such as those that let you configure a project 
using 'customize' and then generate Makefiles for you.   That was a bit 
of a stretch.

Some are in between, such as the 'android' project type that finds the 
AndroidManifest.xml file, tags that tree, and queries your android SDK 
for build tools and paths to set everything up for you.

In one of these threads someone noted that it should be easy to setup 
and transparent.  When someone's project matches a supported type, all 
you need to do is turn on EDE and it then sets up the Semantic details 
for you.  Most C/C++ projects are not as magical, and need a more hands 
on approach, and that's where things get tricky.

Eric

Re: IDE

[Dmitry Gutov]

On 10/14/2015 01:29 AM, Eric Ludlam wrote:

> Simple things like showing the function you are editing, highlighting
> tags with various features in different ways,or knowing what class the
> method you are in are handy and quick little features that can be built
> generically on top of CEDET, but which require piles of code to do
> individually without that type of support. imenu, etags, ctags, global,
> ident, etc all exist because it is useful, but none of those tools get
> bound into a buffer, so their level of usefulness is limited to "jump to
> a location" instead of handy inline features.

Not sure what you mean by "bound into a buffer", but IMenu, in general, 
only requires a few regexps, and once you have those in place, 
which-func-mode can show up in which "tag" you are.

And ctags can be used for "a database of tags across a project". You 
require it either way, since only open buffers are parsed by Semantic.

 > 3) a standard way of knowing where you are in relation to other tags

How does that help?

> Because it is a multi-mode buffer, so sometimes you want to complete
> srecode symbols, and sometimes you want to complete from a different
> language.

Makes sense, thanks.

Re: IDE

[Eric Ludlam]

On 10/14/2015 09:53 AM, Dmitry Gutov wrote:
> On 10/14/2015 03:14 PM, Alexis wrote:
>
>>> i would also guess that, similarly, EDE might not be that useful when
>>> developing in languages such as ELisp, Python or Ruby.
>
> That has been my impression as well.

I use EDE with my Elisp projects and find it useful for 2 things:

* Simplify compiling elisp (when it matters)
* scoping of symref calls

Other lispy things supported with CEDET don't depend on the EDE part.

For languages that need ot fire up an external interpreter, it could 
provide a simple way to know where to start the interpreter so you can 
ask it questions.  That is similar to the other project.el project.

Eric

Re: IDE

[Dmitry Gutov]

On 10/14/2015 03:05 PM, Eric Ludlam wrote:

> The puzzle for me here is that while the different pieces are
> technically independent, the more complex tasks, such as completion,
> depend on the other tools doing their job.  Good smart completion
> depends on a knowledge of a project's structure to find headers (C/C++),
> and it also depends on rummaging around in your files to find the needed
> symbols.

It seems what we need is a set of "juncture points", which define how 
separate systems/tools should communicate. That's what xref and 
project.el are about, for finding locations and project information 
respectively; everything else in there is mostly for convenience.

> There are definitely dependencies.  I don't think it is
> over-specialized, but perhaps overly-generalized.

Could be both. But by over-specialized, I mean that a lot of stuff in 
e.g. ede-project doesn't make sense for many projects. targets, for 
example. And mailinglist/web-site-url/ftp-site?..

I'm also under impression that EDE projects are always one directory 
deep (and for each subdirectory, you need subprojects). Which reflects 
the common Makefile usage, but not how projects are structured under 
many other build systems.

> It is possible to swap in different solutions (under the CEDET
> framework) but in many cases, there is currently only one solution.

Yet still, to even become pluggable, a piece of functionality has to buy 
into the CEDET fundamentals, like using EIEIO, mode-locals, and CEDET's 
base classes.

Re: IDE

[Dmitry Gutov]

On 10/15/2015 03:14 AM, Eric Ludlam wrote:

> Historically, the 'targets' matched the makefile targets, and was used
> to generate a Makefile from a configuration.
>
> For the other projects, the targets simply group source code together so
> if you use the 'compile' key sequence, you get something appropriate for
> that group of source files.  Sometime it is very simple so that .texi or
> .cpp execute different compile steps. Sometimes there is no difference.

That doesn't sound portable: with flexibly scriptable build tools (e.g. 
Rake, and Ant, I imagine), an external tool reading the build file won't 
be able to determine, in general, which files or groups of files a task 
is related to (unless it executes the task and watches the disk for 
changes, which is not tenable).

> It's fine for several project systems to do the same thing.  They could
> share some implementation or not, depending the way any set of lisp
> programs might.  Many shared behaviours are pushed up in the class
> hierarchy.  For example, handling include paths, java classpath, etc.
> For most things like 'compile', the similar code is about appending the
> string "make " with some target, or whatever it might be, so it isn't
> too deep.

It's nice when things can fit into the single-inheritance hierarchy.

But consider if there are several axes of customization. For example, if 
we have Java, Scala and Groovy based projects, and each sort can use 
Ant, Maven or, say, SBT.

Will that be 9 project definitions that someone will have to type out?

If we decouple the "build system" from "project", however, that would be 
just 6 definitions, and one could add to either set without having to 
create multiple new definitions.

So, for project.el it seems appropriate to add either a variable 
(project-build-system) or a hook similar to project-find-functions.

Does a lot of code in EDE *really* need to know what build system a 
project uses?

> Some are in between, such as the 'android' project type that finds the
> AndroidManifest.xml file, tags that tree, and queries your android SDK
> for build tools and paths to set everything up for you.

That's one of the easier cases, conceptually, because there are no 
alternative choices for build tools.

Re: IDE

[Elias Mårtenson]

[-- Attachment #1: Type: text/plain, Size: 685 bytes --]

On 15 October 2015 at 12:21, Dmitry Gutov <dgutov@yandex.ru> wrote:

> On 10/15/2015 03:14 AM, Eric Ludlam wrote:
>
> Some are in between, such as the 'android' project type that finds the
>>
> AndroidManifest.xml file, tags that tree, and queries your android SDK
>> for build tools and paths to set everything up for you.
>>
>
> That's one of the easier cases, conceptually, because there are no
> alternative choices for build tools.
>

For Android, there is at least three that I can think of: Old-style pure
Ant builds, IntelliJ IDEA builds and Gradle builds.

All these have different behaviours. In particular, they all have different
ways of generating R.java.

Regards,
Elias

[-- Attachment #2: Type: text/html, Size: 1352 bytes --]

Re: IDE

[Dmitry Gutov]

On 10/15/2015 08:07 AM, Elias Mårtenson wrote:

> For Android, there is at least three that I can think of: Old-style pure
> Ant builds, IntelliJ IDEA builds and Gradle builds.

Very well, I stand corrected. 
https://developer.android.com/sdk/installing/studio-build.html gave the 
impression that Gradle is the only one, or at least the current blessed 
solution.

I wonder if CEDET supports more than one of them.

Re: IDE

[Eric Ludlam]

On 10/14/2015 11:16 PM, Dmitry Gutov wrote:
> On 10/14/2015 01:29 AM, Eric Ludlam wrote:
>
>> Simple things like showing the function you are editing, highlighting
>> tags with various features in different ways,or knowing what class the
>> method you are in are handy and quick little features that can be built
>> generically on top of CEDET, but which require piles of code to do
>> individually without that type of support. imenu, etags, ctags, global,
>> ident, etc all exist because it is useful, but none of those tools get
>> bound into a buffer, so their level of usefulness is limited to "jump to
>> a location" instead of handy inline features.
>
> Not sure what you mean by "bound into a buffer", but IMenu, in 
> general, only requires a few regexps, and once you have those in 
> place, which-func-mode can show up in which "tag" you are.

CEDET will store tags into a set of overlays in the buffer.  That means 
figuring out what tag the cursor is in is as fast as asking for what 
overlay the cursor is in.

Imenu stores it's tags in a list, so you need to scan the list to figure 
it out. Imenu's tags are also weak, so the elisp knows very little about 
the tag, only where it is, and enough to queue the reader.

> And ctags can be used for "a database of tags across a project". You 
> require it either way, since only open buffers are parsed by Semantic.
>

Yes.  There are other tools that do different pieces of what CEDET does.

> > 3) a standard way of knowing where you are in relation to other tags
>
> How does that help?
>

* It lets you 'copy' a tag, and 'yank' it somewhere else.
* It provides an accurate 'beginning of defun', 'end of defun',
   'narrow to defun'
* srecode can programmatically insert new tags between other tags
   using a hueristic.
* It can figure out where to place or find a header comment.
* You can decorate the tags accurately
* Provides a starting symbol for some commands, such as symref.
* Adding folding of tags to a buffer is pretty easy (though that
   contribution didn't get a release. :( )
* The stuff imenu / which-func does but with more options such as
   breadcrumb type format.
* You can parse the local context more quickly determining nesting
   context (ie - method in a class) for decoding symbols (like "this")
* There's a mode that tracks what tag you are in as you edit so you can
   jump through your history by name.
* From a self-dependency point of view, it enables incremental
   reparsing of a buffer.

-Eric

Re: IDE

[Eric Ludlam]

On 10/14/2015 11:40 PM, Dmitry Gutov wrote:
> On 10/14/2015 03:05 PM, Eric Ludlam wrote:
>
>> The puzzle for me here is that while the different pieces are
>> technically independent, the more complex tasks, such as completion,
>> depend on the other tools doing their job.  Good smart completion
>> depends on a knowledge of a project's structure to find headers (C/C++),
>> and it also depends on rummaging around in your files to find the needed
>> symbols.
>
> It seems what we need is a set of "juncture points", which define how 
> separate systems/tools should communicate. That's what xref and 
> project.el are about, for finding locations and project information 
> respectively; everything else in there is mostly for convenience.

Indeed.  That is a similarity.

>> There are definitely dependencies.  I don't think it is
>> over-specialized, but perhaps overly-generalized.
>
> Could be both. But by over-specialized, I mean that a lot of stuff in 
> e.g. ede-project doesn't make sense for many projects. targets, for 
> example. And mailinglist/web-site-url/ftp-site?..
>

There is indeed a bunch of extra not so useful things for all classes in 
the baseclasses.  I would not claim it could not be improved.

Some, such as 'version' and 'name' seem trivial, but become more 
convenient when you start browsing through your open projects.

> I'm also under impression that EDE projects are always one directory 
> deep (and for each subdirectory, you need subprojects). Which reflects 
> the common Makefile usage, but not how projects are structured under 
> many other build systems.
>

Different projects handle this differently.  The original set regarding 
both Automake and Makefile based projects are as you describe.

Last year I simplified a bunch of systems regarding project detection 
and directory hierarchy.  At this point, the 'simplest' project as far 
as setup is the 'generic' project type, one of which finds a project 
based on your VC root (git, cvs, and a few others). It is one project 
covering all the subdirectories.  You can 'customize' the project and 
just fill in the blanks for your build system, include-path and others, 
or just ignore it all.

>> It is possible to swap in different solutions (under the CEDET
>> framework) but in many cases, there is currently only one solution.
>
> Yet still, to even become pluggable, a piece of functionality has to 
> buy into the CEDET fundamentals, like using EIEIO, mode-locals, and 
> CEDET's base classes.
>

Yes.  The structure I started building used traditional emacsy tools for 
handling structure and buffer local behaviours.  The broader the tool 
became, the more of a PITA it was managing all the random lists of 
behaviour differences, and random list references, so I put in missing 
infrastructure.

You will note other tools that wrangle data now use defstruct (now part 
of emacs) or seem to create their own data structure with their own 
accessors, so building such tools is not unusual.  I just tried to make 
mine more generally useful.

Eric

Re: IDE

[Eric Ludlam]

On 10/15/2015 12:21 AM, Dmitry Gutov wrote:
> It's fine for several project systems to do the same thing.  They could
>> share some implementation or not, depending the way any set of lisp
>> programs might.  Many shared behaviours are pushed up in the class
>> hierarchy.  For example, handling include paths, java classpath, etc.
>> For most things like 'compile', the similar code is about appending the
>> string "make " with some target, or whatever it might be, so it isn't
>> too deep.
>
> It's nice when things can fit into the single-inheritance hierarchy.
>
> But consider if there are several axes of customization. For example, 
> if we have Java, Scala and Groovy based projects, and each sort can 
> use Ant, Maven or, say, SBT.
>
> Will that be 9 project definitions that someone will have to type out?
>
> If we decouple the "build system" from "project", however, that would 
> be just 6 definitions, and one could add to either set without having 
> to create multiple new definitions.
>
> So, for project.el it seems appropriate to add either a variable 
> (project-build-system) or a hook similar to project-find-functions.
>
> Does a lot of code in EDE *really* need to know what build system a 
> project uses?
>

EDE, the high level framework, doesn't care about the build system.

Any particular project type may or may not care about the build system.  
Some do because they are the build system.  Some do because they look in 
the config files to try to extract some handy nuggets of information.  
Some do because the build system leaves a file behind that can be 
detected as the root of the project.

Several just let you configure a text string that says how to fork off 
the 'compile' command and leave it at that.

For your example above someone who is familiar with those tools would 
pick the best way to detect a project (maybe by build system like ant, 
or by some other means) and also pick the best way to extract whatever 
data is needed, such as a command to pass to 'compile', and hopefully a 
classpath.  It might be 6 independent types that share a lot of code or 
baseclasses, or maybe one hybrid. I don't think it really matters.

In the end, EDE the framework provides a common set of:
* commands a user can use to interact with the project such as compiling.
* A place to put logic that helps other tools that have project
   dependencies such as include paths, classpath, or project root detection.

The individual EDE projects then layer on bonus features, such as 
creating a specialized project (such as Android or Automake) from 
scratch, handling configurations, and a few other random things.

Eric

Re: IDE

[Eric Ludlam]

On 10/15/2015 01:16 AM, Dmitry Gutov wrote:
> On 10/15/2015 08:07 AM, Elias Mårtenson wrote:
>
>> For Android, there is at least three that I can think of: Old-style pure
>> Ant builds, IntelliJ IDEA builds and Gradle builds.
>
> Very well, I stand corrected. 
> https://developer.android.com/sdk/installing/studio-build.html gave 
> the impression that Gradle is the only one, or at least the current 
> blessed solution.
>
> I wonder if CEDET supports more than one of them.
>
I added support for the original command line SDK that used ant for Linux.

I haven't upgraded to android studio and haven't hacked my phone in a 
while so can't say much more.  I don't know if anyone else really uses 
Emacs for
android as I've gotten no feedback here.

Eric

Re: IDE

[Przemysław Wojnowski]

W dniu 15.10.2015 o 15:18, Eric Ludlam pisze:
> EDE, the high level framework, doesn't care about the build system.
>
> Any particular project type may or may not care about the build system. Some do
> because they are the build system.  Some do because they look in the config
> files to try to extract some handy nuggets of information. Some do because the
> build system leaves a file behind that can be detected as the root of the project.
>
[...]

> In the end, EDE the framework provides a common set of:
> * commands a user can use to interact with the project such as compiling.
> * A place to put logic that helps other tools that have project
>    dependencies such as include paths, classpath, or project root detection.

IMHO this is very good start for project support, even though EDE is not
perfect. It is much better to refine it than to throw away and reinvent the
wheel, which may not have a happy end.

Anyway, I'll try it.

Thanks for explanations,
Przemysław

Re: IDE

[Dmitry Gutov]

On 10/15/2015 04:18 PM, Eric Ludlam wrote:

> Any particular project type may or may not care about the build system.
> Some do because they are the build system.  Some do because they look in
> the config files to try to extract some handy nuggets of information.
> Some do because the build system leaves a file behind that can be
> detected as the root of the project.

Then we might as well treat the build-file-as-project-root-marker and 
build-file-as-source-of-build-tasks as two unrelated things, in the API. 
And in certain cases both implementations can delegate to the same code.

> For your example above someone who is familiar with those tools would
> pick the best way to detect a project (maybe by build system like ant,
> or by some other means) and also pick the best way to extract whatever
> data is needed, such as a command to pass to 'compile', and hopefully a
> classpath.  It might be 6 independent types that share a lot of code or
> baseclasses, or maybe one hybrid. I don't think it really matters.

It matters if to *really* add support for a new build tool, the author 
has to add X new project definitions.

Re: IDE

[Dmitry Gutov]

On 10/15/2015 04:08 PM, Eric Ludlam wrote:

> Indeed.  That is a similarity.

Not an accidental one.

> Some, such as 'version' and 'name' seem trivial, but become more
> convenient when you start browsing through your open projects.

I'm not saying there aren't useful, just not essential.

> Different projects handle this differently.  The original set regarding
> both Automake and Makefile based projects are as you describe.

I see. And the distinction is managed by the 
ede-find-subproject-for-directory implementation.

> You can 'customize' the project and
> just fill in the blanks for your build system, include-path and others,
> or just ignore it all.

How does one customize an instance of the generic project?

> You will note other tools that wrangle data now use defstruct (now part
> of emacs) or seem to create their own data structure with their own
> accessors, so building such tools is not unusual.  I just tried to make
> mine more generally useful.

Sure.

Re: IDE

[Eric Ludlam]

On 10/15/2015 05:03 PM, Dmitry Gutov wrote:
> On 10/15/2015 04:08 PM, Eric Ludlam wrote:
>
>> You can 'customize' the project and
>> just fill in the blanks for your build system, include-path and others,
>> or just ignore it all.
>
> How does one customize an instance of the generic project?
If you open a project that keeps a save file, which would be either the 
Makefile or Automakefile project, or any of the 'generic' project types, 
then you can do:

M-x customize-project


If you want to try it out, you would need:

(global-ede-mode 1)
(ede-enable-generic-projects)

then visit something in a version-control system (not emacs, that has 
it's own project type)

Eric

Re: IDE

[Przemysław Wojnowski]

W dniu 15.10.2015 o 22:31, Dmitry Gutov pisze:
> On 10/15/2015 04:18 PM, Eric Ludlam wrote:
>
>> Any particular project type may or may not care about the build system.
>> Some do because they are the build system.  Some do because they look in
>> the config files to try to extract some handy nuggets of information.
>> Some do because the build system leaves a file behind that can be
>> detected as the root of the project.
>
> Then we might as well treat the build-file-as-project-root-marker and
> build-file-as-source-of-build-tasks as two unrelated things, in the API. And in
> certain cases both implementations can delegate to the same code.
>
>> For your example above someone who is familiar with those tools would
>> pick the best way to detect a project (maybe by build system like ant,
>> or by some other means) and also pick the best way to extract whatever
>> data is needed, such as a command to pass to 'compile', and hopefully a
>> classpath.  It might be 6 independent types that share a lot of code or
>> baseclasses, or maybe one hybrid. I don't think it really matters.
>
> It matters if to *really* add support for a new build tool, the author has to
> add X new project definitions.
>
IIUC someone developing an EDE support (a plugin?) for a build tool can provide 
as many as s/he wants, right? For example for a build tool a developer may 
provide only two definitions: clean and build.

I think that two more features would be helpful:
- composition of tasks (e.g. run "clean" before "build")
- allow user to add own tasks (could be customizations of default tasks, like 
"compile --with-debug-symbols").

Correct me if I'm wrong, but EDE (xref and project.el) are open for 
modifications, right? It not "take it as it is or leave it".

BTW Why EDE (and CEDET) are in two different repositories (and projects?).
One is in Emacs sources, another here http://sourceforge.net/projects/cedet/ ?

Re: IDE

[Przemysław Wojnowski]

W dniu 15.10.2015 o 14:57, Eric Ludlam pisze:
> * It lets you 'copy' a tag, and 'yank' it somewhere else.
> * It provides an accurate 'beginning of defun', 'end of defun',
>    'narrow to defun'
> * srecode can programmatically insert new tags between other tags
>    using a hueristic.
> * It can figure out where to place or find a header comment.
> * You can decorate the tags accurately
> * Provides a starting symbol for some commands, such as symref.
> * Adding folding of tags to a buffer is pretty easy (though that
>    contribution didn't get a release. :( )
> * The stuff imenu / which-func does but with more options such as
>    breadcrumb type format.
> * You can parse the local context more quickly determining nesting
>    context (ie - method in a class) for decoding symbols (like "this")
> * There's a mode that tracks what tag you are in as you edit so you can
>    jump through your history by name.
> * From a self-dependency point of view, it enables incremental
>    reparsing of a buffer.
>
> -Eric
>

IMHO Semantic + SRecode combo, even with information from only one buffer, is a 
great fit for implementation of many local refactorings: Extract Method, Extract 
Var/Const, Inline temp, etc. (see here: http://www.refactoring.com/catalog/)

Re: IDE

[Dmitry Gutov]

On 10/16/2015 05:40 AM, Eric Ludlam wrote:

> If you open a project that keeps a save file, which would be either the
> Makefile or Automakefile project, or any of the 'generic' project types,
> then you can do:
>
> M-x customize-project (...)

That's very nice, thank you. We might reuse that code for in project.el 
sometime.

Re: IDE

[Dmitry Gutov]

On 10/16/2015 10:39 AM, Przemysław Wojnowski wrote:

> IIUC someone developing an EDE support (a plugin?) for a build tool can
> provide as many as s/he wants, right? For example for a build tool a
> developer may provide only two definitions: clean and build.

No, they'll have to extend every relevant project type (thus creating 
new types), and in each of them, define clean and build. Or something 
along these lines.

Any user-defined types will miss out, because the author of the build 
tool support is not aware of them.

> I think that two more features would be helpful:
> - composition of tasks (e.g. run "clean" before "build")

Don't build tools do that?

> - allow user to add own tasks (could be customizations of default tasks,
> like "compile --with-debug-symbols").

The command line could be made editable when the "call build tool" 
command is invoked with a prefix.

> Correct me if I'm wrong, but EDE (xref and project.el) are open for
> modifications, right? It not "take it as it is or leave it".

Probably. Not sure what you're really asking. For modifications by whom 
and how?

> BTW Why EDE (and CEDET) are in two different repositories (and projects?).
> One is in Emacs sources, another here
> http://sourceforge.net/projects/cedet/ ?

CEDET is in Emacs sources as well (although is has some omissions). See 
lisp/cedet.

Re: IDE

[Przemysław Wojnowski]

W dniu 16.10.2015 o 12:27, Dmitry Gutov pisze:
> On 10/16/2015 10:39 AM, Przemysław Wojnowski wrote:
>
>> IIUC someone developing an EDE support (a plugin?) for a build tool can
>> provide as many as s/he wants, right? For example for a build tool a
>> developer may provide only two definitions: clean and build.
>
> No, they'll have to extend every relevant project type (thus creating new
> types), and in each of them, define clean and build. Or something along these
> lines.
>
> Any user-defined types will miss out, because the author of the build tool
> support is not aware of them.
IIUC you suggest to use the Bridge Pattern to separate Project Type and Build 
Tool abstractions and allow to exchange them independently, right?
IMHO this is good design and maybe EDE is already built is such way (I'll have
to use and read EDE first to get better understanding of it.)

(One thing Design Patterns give, except solutions to common design problems, is 
a common vocabulary to facilitate communication. So instead of writing 
elaborates on design, one can simply say "We could apply The Bridge between X 
and Y abstractions", etc.)

>> I think that two more features would be helpful:
>> - composition of tasks (e.g. run "clean" before "build")
>
> Don't build tools do that?
Some do. But even then a user may be willing to run them in a different way.
For example Maven doesn't run "clean" phase by default, when one runs other
phases (compile/package/install), so if an user wants to run "clean install"
she has to execute two separate commands. But that can be easily defined as one
compound command (clean-compile).

BTW in the Command Pattern, commands implement a common interface, so an
executor doesn't even know what is executed, except that it is a command, which
can be a Composite (pattern) Command that implements the same interface.

>> - allow user to add own tasks (could be customizations of default tasks,
>> like "compile --with-debug-symbols").
>
> The command line could be made editable when the "call build tool" command is
> invoked with a prefix.
Yes, that would be a good addition too. But sooner or later a use has a
workflow, a set of commands that she runs every time and it would be good to
allow to define in a separate, easily accessible command.

>> Correct me if I'm wrong, but EDE (xref and project.el) are open for
>> modifications, right? It not "take it as it is or leave it".
>
> Probably. Not sure what you're really asking. For modifications by whom and how?
By contributors. Sometime projects have very restrictive attitude towards
new contributions and just reject most of proposals if they are not perfect up
front and doesn't solve all their problems at once.

>> BTW Why EDE (and CEDET) are in two different repositories (and projects?).
>> One is in Emacs sources, another here
>> http://sourceforge.net/projects/cedet/ ?
>
> CEDET is in Emacs sources as well (although is has some omissions). See lisp/cedet.
Yes, so when one would like change EDE to, lets say, apply the Bridge pattern,
which one should be used?

Re: IDE

[Dmitry Gutov]

On 10/16/2015 02:17 PM, Przemysław Wojnowski wrote:

> IIUC you suggest to use the Bridge Pattern to separate Project Type and
> Build Tool abstractions and allow to exchange them independently, right?

Possibly. But I'm not sure yet how it'll end up looking in Elisp, and in 
the API. For all I know, the Project Type doesn't need to reference the 
Build Tool at all, in the general case. Certain commands would just 
reference both.

So we might be served better with separate notions of "current project" 
and "current build tool".

> (One thing Design Patterns give, except solutions to common design
> problems, is a common vocabulary to facilitate communication. So instead
> of writing elaborates on design, one can simply say "We could apply The
> Bridge between X and Y abstractions", etc.)

Yes and no. Details matter.

> For example Maven doesn't run "clean" phase by default, when one runs other
> phases (compile/package/install), so if an user wants to run "clean
> install"
> she has to execute two separate commands. But that can be easily defined
> as one
> compound command (clean-compile).

It should be easy to write as a user-defined command anyway.

> BTW in the Command Pattern, commands implement a common interface, so an
> executor doesn't even know what is executed, except that it is a
> command, which
> can be a Composite (pattern) Command that implements the same interface.

I know all these words as well. :) If you'd like a stab at the 
implementation (or just defining the API), be my guest.

If you'd like to use composite commands, my first question would be how 
a user would form them (that requires a UI). I think at this stage we 
should just be satisfied that this is doable, and can be delayed almost 
indefinitely (that's the beauty of the Composite pattern).

> Yes, that would be a good addition too. But sooner or later a use has a
> workflow, a set of commands that she runs every time and it would be
> good to
> allow to define in a separate, easily accessible command.

a) We can have command history, b) The user could define new commands. 
Certain tools could provide some pre-defined ones as well.

> By contributors. Sometime projects have very restrictive attitude towards
> new contributions and just reject most of proposals if they are not
> perfect up
> front and doesn't solve all their problems at once.

We try to "hone" the proposals to our general liking. On the flip side, 
that can result in long discussions and, at times, contributor 
dissatisfaction.

>> CEDET is in Emacs sources as well (although is has some omissions).
>> See lisp/cedet.
> Yes, so when one would like change EDE to, lets say, apply the Bridge
> pattern,
> which one should be used?

I'll let Eric answer this one.

Re: IDE

[Dmitry Gutov]

On 10/15/2015 03:57 PM, Eric Ludlam wrote:

> CEDET will store tags into a set of overlays in the buffer.  That means
> figuring out what tag the cursor is in is as fast as asking for what
> overlay the cursor is in.

I see. But you have to keep that info up-to-date as well, and that 
becomes less fast, because you implement it in Elisp. If we're comparing 
to an external program.

> Imenu stores it's tags in a list, so you need to scan the list to figure
> it out. Imenu's tags are also weak, so the elisp knows very little about
> the tag, only where it is, and enough to queue the reader.

All true. But we have other facilities as well. For instance, the modes 
which use SMIE for indentation can implement extraction of similar 
information, in a more accurate way.

If we were able to easily substitute in SMIE-based "current tag" 
implementation instead of using Wisent, that would be a plus.

> Yes.  There are other tools that do different pieces of what CEDET does.

I mean that you can't *really* use Semantic for "jump to a tag in the 
project", because one doesn't usually like to open all project files at 
once. But if the "daemon" proposal sees any development, maybe...

> * It lets you 'copy' a tag, and 'yank' it somewhere else.
> * It provides an accurate 'beginning of defun', 'end of defun',
>    'narrow to defun'

Emacs usually provides fairly accurate definitions of these as well.

> * srecode can programmatically insert new tags between other tags
>    using a hueristic.

I suppose it could use beginning-of-defun-function as well.

> * Provides a starting symbol for some commands, such as symref.

I wonder if it's ideal: in IntellijIDEA, say, you can click on any of 
the method's uses and to list the other references. With your scheme, 
however, one has to jump to its definition first.

For someone user to the former, it's counter-intuitive: you move point 
to 'bar' in foo.bar(), and instead semantic-symref suggests asking for 
uses of whatever function you're currently inside. If you don't actually 
read the prompt fully and just press y (or yes), the result will be 
puzzling.

> * The stuff imenu / which-func does but with more options such as
>    breadcrumb type format.

A new generic API could use a more detailed format as well.

> * You can parse the local context more quickly determining nesting
>    context (ie - method in a class) for decoding symbols (like "this")

Yes, you can usually resolve what 'this' is (and, consequently, 
this.foo()). But not what foo.bar() is, in the general case. Not in a 
duck-typed language anyway.

So, that's a problem. If we could, using semantic-symref could be made 
more natural.

> * There's a mode that tracks what tag you are in as you edit so you can
>    jump through your history by name.

That's pretty cool.

Re: IDE

[Dmitry Gutov]

On 10/16/2015 01:00 PM, Przemysław Wojnowski wrote:

> IMHO Semantic + SRecode combo, even with information from only one
> buffer, is a great fit for implementation of many local refactorings:
> Extract Method, Extract Var/Const, Inline temp, etc. (see here:
> http://www.refactoring.com/catalog/)

In theory, it could be. But from what I've read from various mailing 
list postings, Semantic grammars often skip over the method contents. 
And that where most of the code lives (which you want to refactor).

Re: IDE

[Lluís]

Przemysław Wojnowski writes:

> W dniu 16.10.2015 o 12:27, Dmitry Gutov pisze:
>> On 10/16/2015 10:39 AM, Przemysław Wojnowski wrote:
>> 
>>> IIUC someone developing an EDE support (a plugin?) for a build tool can
>>> provide as many as s/he wants, right? For example for a build tool a
>>> developer may provide only two definitions: clean and build.
>> 
>> No, they'll have to extend every relevant project type (thus creating new
>> types), and in each of them, define clean and build. Or something along these
>> lines.
>> 
>> Any user-defined types will miss out, because the author of the build tool
>> support is not aware of them.
> IIUC you suggest to use the Bridge Pattern to separate Project Type and Build
> Tool abstractions and allow to exchange them independently, right?
> IMHO this is good design and maybe EDE is already built is such way (I'll have
> to use and read EDE first to get better understanding of it.)

Hmmm, I think I should read about that pattern more closely, but AFAIK this is
not what EDE uses. EDE is built around a class hierarchy of ede-project that
provides all the information and actions you might need for each type of project
and all possible operations and information sources on it.

But this bridge pattern does seem to match what I had in mind, though (I made up
the names just now):

* service-type: A type of service (one instance per type), like build, find root
  directory, include search paths, compilation flags, etc.

* service: A specific implementation of a service type, like using vc-dir for
  finding the root directory of a project.

* service-collection: A set of service implementations of the same service
  type. This allows aggregating results from each service into a single answer.

* project-type: A type of project defines the service collections it provides
  for each service type it knows about. An example could be an Emacs project
  (knows how to auto-detect it, things like the build system it uses, etc.), and
  Android project, a Linux kernel project, an autotools project, etc.

* project: A specific project built from a combination of project types. For
  example, an autotools project with some customized project type that overrides
  some of it. This would be what other tools use to interact with each of the
  services supported by the project.

This scheme is something I've had on the back of my mind for quite some time in
order to make EDE easier to develop for and for users to add persistent
customizations on top of existing project knowledge.


Thanks,
  Lluis

-- 
"And it's much the same thing with knowledge, for whenever you learn
something new, the whole world becomes that much richer."
-- The Princess of Pure Reason, as told by Norton Juster in The Phantom
Tollbooth

Re: IDE

[John Wiegley]

>>>>> Dmitry Gutov <dgutov@yandex.ru> writes:

> My already-stated impression is that it's over-specialized and tightly
> coupled.
>
> Not saying that the problem domain is easy, but being able to use different
> pieces of the solution separately would go a long way towards alleviating
> the complaint that certain other parts are incomplete.
>
> Especially if it were easier to swap in different solutions for some of
> those parts (and do entirely without some others), and do that in not too
> many lines, all as part of the user's configuration.

You've taken the reply right out of my mouth, Dmitry. David's response was
also very much in line with my thinking. As I said before, if CEDET were the
answer to our questions, we wouldn't still be asking them.

If, later in our design discussions, CEDET has experiences, code, and
perspectives to offer, this could be of tremendous value. I hope Eric will be
avail of us his time then, and share those nuggets with us.

John

Re: IDE

[Xue Fuqiao]

On Fri, Oct 16, 2015 at 3:39 PM, Przemysław Wojnowski
<esperanto@cumego.com> wrote:
> BTW Why EDE (and CEDET) are in two different repositories (and projects?).
> One is in Emacs sources, another here http://sourceforge.net/projects/cedet/
> ?

The same reason(s) for CC mode, Gnus, Org, Tramp, etc.

Some reasons I imagine are:

* It can leave bleeding edge features on its own repo (not necessary in
  the next Emacs release)
* It can have its own bug tracker (if its authors don't like Debbugs)
* It can include code not distributed with Emacs for some reasons (for
  example, a new author doesn't want to sign or haven't signed the
  copyright assignment yet)
* It can use a VCS Emacs is not using (for example, Gnus and Org was
  using Git before the Emacs Git transition)
* It can include compatibility helpers for XEmacs and other emacsen

On Fri, Oct 16, 2015 at 7:17 PM, Przemysław Wojnowski
<esperanto@cumego.com> wrote:
> Yes, so when one would like change EDE to, lets say, apply the Bridge
> pattern,
> which one should be used?

Both are OK, I think.  IIRC the SourceForge repo merges changes from
Emacs master from time to time.

Re: IDE

[Eric Ludlam]

On 10/16/2015 03:39 AM, Przemysław Wojnowski wrote:
>> It matters if to *really* add support for a new build tool, the 
>> author has to
>> add X new project definitions.
>>
> IIUC someone developing an EDE support (a plugin?) for a build tool 
> can provide as many as s/he wants, right? For example for a build tool 
> a developer may provide only two definitions: clean and build.

EDE's framework starts with no assumptions about anything other than a 
basic "compile", and letting the project implementation populate it.

Some projects, such as the android and arduino ones, add new commands 
such as 'upload to device' which make no sense in other projects.  Those 
projects end up self consistent as far as keybindings.  Until there are 
lots of projects with different ideas, there isn't much in the way of 
guidelines.

> I think that two more features would be helpful:
> - composition of tasks (e.g. run "clean" before "build")
> - allow user to add own tasks (could be customizations of default 
> tasks, like "compile --with-debug-symbols").
>

Project implementations can have as many configurations or custom build 
commands as they like.

> Correct me if I'm wrong, but EDE (xref and project.el) are open for 
> modifications, right? It not "take it as it is or leave it".

Indeed.

> BTW Why EDE (and CEDET) are in two different repositories (and 
> projects?).
> One is in Emacs sources, another here 
> http://sourceforge.net/projects/cedet/ ?
> . 

I maintain CEDET, but I cannot get a permanent release from my company, 
so I have to keep getting it updated.  By having a separate repository I 
can keep going without an active release without getting Emacs legally 
dirty.

Eric

Re: IDE

[Eric Ludlam]

On 10/16/2015 07:17 AM, Przemysław Wojnowski wrote:
>>> BTW Why EDE (and CEDET) are in two different repositories (and 
>>> projects?).
>>> One is in Emacs sources, another here
>>> http://sourceforge.net/projects/cedet/ ?
>>
>> CEDET is in Emacs sources as well (although is has some omissions). 
>> See lisp/cedet.
> Yes, so when one would like change EDE to, lets say, apply the Bridge 
> pattern,
> which one should be used?
> . 

Good question.  The recent EIEIO changes in Emacs made the two 
repositories really hard to merge, and EDE is steeped in EIEIO.

David Engster has been doing the merges and might have a good idea. I'm 
just not that good with git. ;(

Eric

Re: IDE

[Eric Ludlam]

On 10/16/2015 09:05 AM, Dmitry Gutov wrote:
> On 10/15/2015 03:57 PM, Eric Ludlam wrote:
>
>> CEDET will store tags into a set of overlays in the buffer.  That means
>> figuring out what tag the cursor is in is as fast as asking for what
>> overlay the cursor is in.
>
> I see. But you have to keep that info up-to-date as well, and that 
> becomes less fast, because you implement it in Elisp. If we're 
> comparing to an external program.
>

The magic of overlays is that the overlays keep the bounds of the tag up 
to date for you.  Once you stop editing, and incremental parser fixes up 
any weird things you might have done, such as hacking a tag in half, 
adding new tags, etc.  Because it is incremental, is typically 
instantaneous.

>> Imenu stores it's tags in a list, so you need to scan the list to figure
>> it out. Imenu's tags are also weak, so the elisp knows very little about
>> the tag, only where it is, and enough to queue the reader.
>
> All true. But we have other facilities as well. For instance, the 
> modes which use SMIE for indentation can implement extraction of 
> similar information, in a more accurate way.
>
I am not familiar with SMIE as an acronym, but a google search makes it 
look like someone made a parser generator, in which case there is no 
real difference except that SMIE is tuned for indentation, and wisent is 
tuned for parsing tags.

> If we were able to easily substitute in SMIE-based "current tag" 
> implementation instead of using Wisent, that would be a plus.

The problem is the same (ie - go write a parser to find tags), except 
the SMIE tagger isn't implemented yet.

>> Yes.  There are other tools that do different pieces of what CEDET does.
>
> I mean that you can't *really* use Semantic for "jump to a tag in the 
> project", because one doesn't usually like to open all project files 
> at once. But if the "daemon" proposal sees any development, maybe...
>

If by "open all project files" you mean pull file contents into a buffer 
and leave it open, then that is not what semantic does.  The data is 
indeed resident in memory, but files stay closed unless the user asks to 
jump there.  The first time you make a request that needs searching, it 
will open files to parse them, but then it closes the file.  Later, it 
refers only the the cached data, not to the files.

>> * It lets you 'copy' a tag, and 'yank' it somewhere else.
>> * It provides an accurate 'beginning of defun', 'end of defun',
>>    'narrow to defun'
>
> Emacs usually provides fairly accurate definitions of these as well.

The difference is switching from "fairly" to "very", and mode authors 
would not need to go and write all those beginning-of-defun type overrides.

>
>> * Provides a starting symbol for some commands, such as symref.
>
> I wonder if it's ideal: in IntellijIDEA, say, you can click on any of 
> the method's uses and to list the other references. With your scheme, 
> however, one has to jump to its definition first.
>

There is always a desire to go two ways "where is the symbol under 
cursor", and "who uses the function I'm in".  I was describing the 
later.  The former is important, and not solved by my suggestion above.

>> * You can parse the local context more quickly determining nesting
>>    context (ie - method in a class) for decoding symbols (like "this")
>
> Yes, you can usually resolve what 'this' is (and, consequently, 
> this.foo()). But not what foo.bar() is, in the general case. Not in a 
> duck-typed language anyway.
>

CEDET/Semantic usually gets foo.bar() syntax correct, unless you are 
saying something else.  Can you elaborate?

> So, that's a problem. If we could, using semantic-symref could be made 
> more natural.

Yes.  There is a spot in symref waiting with a comment for that nugget 
of code to be written.  My primary concern is that of performance since 
that data is derived, not cached.

Eric

Re: IDE

[Dmitry Gutov]

On 10/17/2015 05:39 AM, Eric Ludlam wrote:

> The magic of overlays is that the overlays keep the bounds of the tag up
> to date for you.  Once you stop editing, and incremental parser fixes up
> any weird things you might have done, such as hacking a tag in half,
> adding new tags, etc.  Because it is incremental, is typically
> instantaneous.

Is that still true if you're editing at the beginning of the file?

> I am not familiar with SMIE as an acronym, but a google search makes it
> look like someone made a parser generator, in which case there is no
> real difference except that SMIE is tuned for indentation, and wisent is
> tuned for parsing tags.

My point is, Semantic grammars don't help with writing indentation code 
(could they?), and we already have SMIE grammars for several languages. 
It would make sense to be able to make do only with one or the other, 
not have them both for the same language.

>> If we were able to easily substitute in SMIE-based "current tag"
>> implementation instead of using Wisent, that would be a plus.
>
> The problem is the same (ie - go write a parser to find tags), except
> the SMIE tagger isn't implemented yet.

That's less work than writing a grammar, and it could be reusable (at 
least partially) across languages.

> If by "open all project files" you mean pull file contents into a buffer
> and leave it open, then that is not what semantic does.  The data is
> indeed resident in memory, but files stay closed unless the user asks to
> jump there.  The first time you make a request that needs searching, it
> will open files to parse them, but then it closes the file.  Later, it
> refers only the the cached data, not to the files.

I mean that you have to use etags at some point anyway. Or gtags, or 
id-utils, etc.

> The difference is switching from "fairly" to "very", and mode authors
> would not need to go and write all those beginning-of-defun type overrides.

True enough.

>> I wonder if it's ideal: in IntellijIDEA, say, you can click on any of
>> the method's uses and to list the other references. With your scheme,
>> however, one has to jump to its definition first.
>
> There is always a desire to go two ways "where is the symbol under
> cursor", and "who uses the function I'm in".  I was describing the
> later.  The former is important, and not solved by my suggestion above.

I was also describing the latter. Although, I suppose, my way could be 
implemented on top of what Semantic does now, as long as it can "jump to 
the definition" accurately.

> CEDET/Semantic usually gets foo.bar() syntax correct, unless you are
> saying something else.  Can you elaborate?

In the few languages it properly supports now? Maybe it does, most of 
the time (although not in the problem example I gave in this discussion: 
https://gist.github.com/dgutov/19c45ef43d1c90b96483; no matter the 
argument tee is called with, `C-c , J' jumps to the first function with 
that name).

But what about duck typed languages? If a method foo calls bar on its 
argument tee, we don't know the type of tee, all we know that it has a 
method bar.

>> So, that's a problem. If we could, using semantic-symref could be made
>> more natural.
>
> Yes.  There is a spot in symref waiting with a comment for that nugget
> of code to be written.  My primary concern is that of performance since
> that data is derived, not cached.

Not sure what you mean here (which part you believe to have to be 
written yet, or which data we're talking about).

But in any case, symref can afford to be a little slow when prompting 
for a symbol name, just as long the slowness is not proportional to the 
size of the repository (or the multiplier is small, at least).

Re: IDE

[Dmitry Gutov]

On 10/16/2015 07:47 PM, Lluís wrote:

> * service-type: A type of service (one instance per type), like build, find root
>    directory, include search paths, compilation flags, etc.
>
> * service: A specific implementation of a service type, like using vc-dir for
>    finding the root directory of a project.

This sounds good. The implementations could be different (e.g. Java-ish 
Service Locator [0]), but in the absence of other requirements the 
closest Emacs idiomatic structure would be a hook (list of functions) 
for each service type. Like project-find-functions, for example.

Note that the objects the latter currently returns provide both "find 
root" and "search path" services. Thus far it seemed to make sense. Can 
one, and would one want to create different search-path services working 
with the same find-root service? That's an interesting question, and I'd 
love to see a practical example.

More generally, if a package P registers services A, B and C, if B needs 
some information from A, would we expect them to go through P-internal 
channels, or the public interface? For instance, most services might 
like to know the project root (let's call that service type R).

We could standardize the dependencies like that. In the simplest case, 
all services depend on project-root (except for itself), and we'll pass 
the current project-root instance to each functions in their hooks. Then 
an element of service-b-functions will only return non-nil if that 
implementation of B can work with the given kind of project-root (or 
simply "project"; we might include some more info into that instance as 
well).

On the flip side, if there are no B implementations that can work with 
the currently detected R service r1, but there's a certain b2 in 
service-b-functions that would work with r2 (currently shadowed by r1), 
we may be missing out.

> * service-collection: A set of service implementations of the same service
>    type. This allows aggregating results from each service into a single answer.

I suppose you can get this from the same service-b-functions hook, if 
you treat it in a different way.

> * project-type: A type of project defines the service collections it provides
>    for each service type it knows about. An example could be an Emacs project
>    (knows how to auto-detect it, things like the build system it uses, etc.), and
>    Android project, a Linux kernel project, an autotools project, etc.

Having to declare each combination of service-types that a project can 
provide still seems unnecessarily limiting (and tedious). But that's 
what project-types would be for, right?

How would a client even use those declarations? If we were to go that 
way, it'd be better to simply have a project instance have a method that 
returns a list of all services it supports. Or, even simpler, return a 
nil or raise a not-implemented error in each of the generic methods it 
doesn't implement.

Next, suppose a project instance can tell whether it provides a "build 
tool" service. What does a "call build task" command does? Looks for the 
current project and gives up if it doesn't provide the required service, 
or specifically asks the locator for a project implementation that does 
provide that service?

In the latter case, we potentially buy more functionality, at the cost 
of having different commands disagree about what the current project 
actually is.

> Thanks,
>    Lluis
>

[0] http://martinfowler.com/articles/injection.html#ADynamicServiceLocator

Re: IDE

[Eli Zaretskii]

> From: "John Wiegley" <johnw@newartisans.com>
> Date: Fri, 16 Oct 2015 15:58:57 -0700
> 
> >>>>> Dmitry Gutov <dgutov@yandex.ru> writes:
> 
> > My already-stated impression is that it's over-specialized and tightly
> > coupled.
> >
> > Not saying that the problem domain is easy, but being able to use different
> > pieces of the solution separately would go a long way towards alleviating
> > the complaint that certain other parts are incomplete.
> >
> > Especially if it were easier to swap in different solutions for some of
> > those parts (and do entirely without some others), and do that in not too
> > many lines, all as part of the user's configuration.
> 
> You've taken the reply right out of my mouth, Dmitry. David's response was
> also very much in line with my thinking. As I said before, if CEDET were the
> answer to our questions, we wouldn't still be asking them.

Could it be that we don't understand the answer?

I'd suggest to be very careful with such conclusions.  They can only
be valid when based on a detailed analysis of what is and isn't in
CEDET, and on good knowledge and understanding of its design and
implementation.  My impression so far is that neither is particularly
true, and my evidence is the number of times Eric and David Engster
described some CEDET features that came as a surprise to us.

I'm quite sure CEDET has collected and expressed in code a lot of
experience and solutions to many problems that arise in the context of
building an IDE.  It's OK to discard that, if we sure that's the
proverbial 1st variant everyone throws away, but we need first to be
sure we know what we are discarding.

Re: IDE

[David Kastrup]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: "John Wiegley" <johnw@newartisans.com>
>> Date: Fri, 16 Oct 2015 15:58:57 -0700
>> 
>> >>>>> Dmitry Gutov <dgutov@yandex.ru> writes:
>> 
>> > My already-stated impression is that it's over-specialized and tightly
>> > coupled.
>> >
>> > Not saying that the problem domain is easy, but being able to use different
>> > pieces of the solution separately would go a long way towards alleviating
>> > the complaint that certain other parts are incomplete.
>> >
>> > Especially if it were easier to swap in different solutions for some of
>> > those parts (and do entirely without some others), and do that in not too
>> > many lines, all as part of the user's configuration.
>> 
>> You've taken the reply right out of my mouth, Dmitry. David's response was
>> also very much in line with my thinking. As I said before, if CEDET were the
>> answer to our questions, we wouldn't still be asking them.
>
> Could it be that we don't understand the answer?
>
> I'd suggest to be very careful with such conclusions.  They can only
> be valid when based on a detailed analysis of what is and isn't in
> CEDET, and on good knowledge and understanding of its design and
> implementation.

If we wanted that, we would be using vi.  The amount of functionality
available with Emacs is so diversified that it needs to fall apart into
pieces that can be comprehended on their own with reasonable effort.
Actually, I am being cheeky here: a morning with vi documentation will
get you set.  A week with CEDET documentation won't.

An IDE is principally a tool that is supposed to make things easier for
the programmer and let him figure out fewer things on his own rather
than more.  CEDET didn't do that for me.

I'm old and experienced enough that I have the arrogance to claim that
something that is too hard for me to understand and/or use effectively
after putting in a reasonable amount of effort is not a generally useful
tool.

> My impression so far is that neither is particularly true, and my
> evidence is the number of times Eric and David Engster described some
> CEDET features that came as a surprise to us.

Is that supposed to be a good thing?

> I'm quite sure CEDET has collected and expressed in code a lot of
> experience and solutions to many problems that arise in the context of
> building an IDE.  It's OK to discard that, if we sure that's the
> proverbial 1st variant everyone throws away, but we need first to be
> sure we know what we are discarding.

I am not qualified to have much of an opinion about the substance, the
raw functionality implemented in CEDET.  That's probably 90% of the
work, and redoing that will not likely make a decisive difference with
regard to the remaining 10% that are for putting that functionality at
the fingertips of the programmer.

My refactoring work tends to be done using stuff like

git grep '^MAKE_[_A-Z]*SCHEME_CALLBACK[_A-Z]* ' lily|sed -n 's/^\([^:]*\):MAKE_[_A-Z]*SCHEME_CALLBACK[_A-Z]* (\([^,]*\), \([^,]*\), \([^)]*\)).*$/\1 \2 \3 \4/p' |
    while read file class name args
    do
	argname=$(sed -n "s/^$class::$name (SCM \\?\\([^,)]*\\)[,)].*\$/\\1/p" $file)
	sed -i "/^\\(class\\|struct\\) $class\\( {\\)\\?\$/,/^}/s/^\\( *DECLARE_\\)SCHEME\\(_CALLBACK ($name, (\\)SCM[^,)]*\\(, \\)\\?/\\1MEMBER\\2/" $(git grep -l "^\\(class\\|struct\\) $class\\( \\|\$\\)") &&
	    case "$argname" in
		[a-zA-Z]*)
		    thisexpr=$(echo "unsmob<[_a-zA-Z]*> *($argname)"
			       sed -n "/^$class::$name (/,/^}/s/^ *\\([_a-zA-Z]\\+\\) *\\* *\\([_a-zA-Z]\\+\\) *= *unsmob *<\\1> ($argname);\$/\\2/p" $file |
				   while read alias
				   do
				       sed -i "/^$class::$name (/,/^}/{/^ *\\([_a-zA-Z]\\+\\) *\\* *$alias *= *unsmob *<\\1> ($argname);\$/d;}" $file
				       echo $alias
				   done)
		    thisexpr=$(echo "$thisexpr"|sed -n '1h;1!H;${x;s/\n/\\|/g;p}')
		    sed -i "/^$class::$name (/,/^}/{s/\\b\\($thisexpr\\)\\b/this/g;s/\\bthis *-> *//g;s/\\*this\\.//g;}" $file
		    sed -i "s/^\\(MAKE_[_A-Z]*\\)SCHEME\\(_CALLBACK[_A-Z]* ($class, $name,\\)/\\1MEMBER\\2/
s/^\\($class::$name (\\)SCM \\?[^,)]*\\(, \\)\\?\\(.*\\)\$/\\1\\3/" $file
		    ;;
		*)
		    sed -i "s/^\\(MAKE_[_A-Z]*\\)SCHEME\\(_CALLBACK[_A-Z]* ($class, $name,\\)/\\1MEMBER\\2/
s/^\\($class::$name (\\)SCM \\?[^,)]*\\(, \\)\\?\\(.*\\)\$/\\1\\3/" $file
	esac
#	sed "/^MAKE_SCHEME_CALLBACK ($class, $name, $args)/d;

#	echo $file $class $name $args $argname $argtype $param
    done

Yes, this is for real, bulk rewriting certain types of static member
functions into proper member functions, changing certain constructs
systematically while doing that.  Doing that kind of stuff requires
knowing about 5 pages of sed programming and shell programming that
mostly worked already 20 years ago (quoting constructs have become more
robust).

I don't relish doing stuff in that manner.  It's not the kind of job an
automated tool could do easily, but an automated tool could at least
find the code that needs changing and possibly even offer some way to do
replacements that are a bit more syntactically conscious than sed.

I've had bulk changes touching several thousands of lines and I prefer
investing two days into creating a one-shot automated tool than 6 hours
doing everything manually.

I'm pretty sure that somewhere between those two-days sed scripting and
6 hours of completely manual work there must be a sweet spot where an
IDE/refactoring/thing-aware tool tied into Emacs should save both time
and nerves.

M-x grep RET works a lot smoother than M-! grep and the kind of
difference between the two are where Emacs shines, providing the editor
connection to technology.

-- 
David Kastrup

Re: IDE

[David Engster]

Eli Zaretskii writes:
>> From: "John Wiegley" <johnw@newartisans.com>
>> Date: Fri, 16 Oct 2015 15:58:57 -0700
>> 
>
>> >>>>> Dmitry Gutov <dgutov@yandex.ru> writes:
>> 
>> > My already-stated impression is that it's over-specialized and tightly
>> > coupled.
>> >
>> > Not saying that the problem domain is easy, but being able to use different
>> > pieces of the solution separately would go a long way towards alleviating
>> > the complaint that certain other parts are incomplete.
>> >
>> > Especially if it were easier to swap in different solutions for some of
>> > those parts (and do entirely without some others), and do that in not too
>> > many lines, all as part of the user's configuration.
>> 
>> You've taken the reply right out of my mouth, Dmitry. David's response was
>> also very much in line with my thinking. As I said before, if CEDET were the
>> answer to our questions, we wouldn't still be asking them.
>
> Could it be that we don't understand the answer?
>
> I'd suggest to be very careful with such conclusions.  They can only
> be valid when based on a detailed analysis of what is and isn't in
> CEDET, and on good knowledge and understanding of its design and
> implementation.  My impression so far is that neither is particularly
> true, and my evidence is the number of times Eric and David Engster
> described some CEDET features that came as a surprise to us.
>
> I'm quite sure CEDET has collected and expressed in code a lot of
> experience and solutions to many problems that arise in the context of
> building an IDE.  It's OK to discard that, if we sure that's the
> proverbial 1st variant everyone throws away, but we need first to be
> sure we know what we are discarding.

Actually, Eric rewrote Semantic once already...

From the discussion so far, I think the main issue at least w.r.t. to
Semantic is: do you actually want Semantic's tag-based system, or more
general: do you want quick access to AST information in your buffer?

If I understand Dmitry correctly, he is not really interested in that,
as for dynamic languages, the AST information is usually missing
important information (unless you bother to implement a complete
frontend). He'd rather call external binaries for complicated stuff like
completion, and use simpler tools (like pure regexp-based parsing) for
stuff like font-locking, navigation, folding, etc. Of course, you can
hook external binaries into Semantic pretty easily, but I can understand
Dmitry that if he does not need the rest of Semantic, why should he
bother?

Now, I think having AST information in your buffer is great, and I don't
like depending on external binaries if I don't have to, because I want
as much as possible in Emacs Lisp. For me, that's what Emacs is about
and why I still use it in the first place.

-David

Re: IDE

[Eric Ludlam]

On 10/16/2015 11:06 PM, Dmitry Gutov wrote:
> On 10/17/2015 05:39 AM, Eric Ludlam wrote:
>
>> The magic of overlays is that the overlays keep the bounds of the tag up
>> to date for you.  Once you stop editing, and incremental parser fixes up
>> any weird things you might have done, such as hacking a tag in half,
>> adding new tags, etc.  Because it is incremental, is typically
>> instantaneous.
>
> Is that still true if you're editing at the beginning of the file?
>

Yes.

>> I am not familiar with SMIE as an acronym, but a google search makes it
>> look like someone made a parser generator, in which case there is no
>> real difference except that SMIE is tuned for indentation, and wisent is
>> tuned for parsing tags.
>
> My point is, Semantic grammars don't help with writing indentation 
> code (could they?), and we already have SMIE grammars for several 
> languages. It would make sense to be able to make do only with one or 
> the other, not have them both for the same language.
>

Ah, I think I see what you are poking at.

There is nothing about semantic the framework that prevents using 
something like SMIE as a tagging engine.  It seems likely that if some 
language not supported by CEDET today has a good SMIE grammar, and if 
SMIE could be adapted to produce compatible tags (which isn't really 
that hard to do), and if SMIE has an entry point that allows parsing a 
tag set out of a region, then it would integrate into the semantic 
framework at the best level.

The difference is that the parser generators in Semantic today have 
optimizations for skipping parts of the buffer while tagging.  This lets 
it parse whole files more quickly, and more robustly.  Since it uses 
syntax tables to do that, I'll guess SMIE can do it, but I'm not that 
familiar with it.

Also, wisent type grammars could easily dump out data you need for 
indentation since actions can do anything you can write in lisp.  No one 
has tried to do it though.

>
>> If by "open all project files" you mean pull file contents into a buffer
>> and leave it open, then that is not what semantic does.  The data is
>> indeed resident in memory, but files stay closed unless the user asks to
>> jump there.  The first time you make a request that needs searching, it
>> will open files to parse them, but then it closes the file. Later, it
>> refers only the the cached data, not to the files.
>
> I mean that you have to use etags at some point anyway. Or gtags, or 
> id-utils, etc.
>

Have to, no.  Want to, probably. Those tools provide nice short-cuts 
when doing simple name lookup.

The mechanism in use is that there are detailed taggers, like those 
currently written for Semantic using wisent type parsers, and weak 
taggers, like GNU Global.

Once a buffer is open, the detailed tagger runs and is truth.  When a 
search occurs, a process of pulling all relevant databases together is 
started.  This includes tags from the detailed taggers already cached, 
and high level 'omniscient' taggers that are usually weak, but have 
scanned the whole project.  The output of the search includes a database 
table, and tags found in that table.  The tags are additionally refined 
based on whatever the layered criteria is, and when the code decides to 
work with a detailed tag from the search, it forces it to be real, and 
pulls it into a buffer if necessary, and any problems with the weak 
tagger is resolved.

The key here is that the detailed tagger is needed to do complex tasks, 
but weak taggers are great to integrate in due to the speed advantage.  
Having a mix gives you the best possible results.

>
>> CEDET/Semantic usually gets foo.bar() syntax correct, unless you are
>> saying something else.  Can you elaborate?
>
> In the few languages it properly supports now? Maybe it does, most of 
> the time (although not in the problem example I gave in this 
> discussion: https://gist.github.com/dgutov/19c45ef43d1c90b96483; no 
> matter the argument tee is called with, `C-c , J' jumps to the first 
> function with that name).
>

I missed that before.  The answer here is that:

C-c , J RET

has two results.  Since that is a prompt based query and only the string 
is available once it is up.  If instead you put the cursor on "tee" and do

M-x semantic-ia-fast-jump RET

it will go to the right spot.

For the keybinding you were using, the workflow is:

C-c , J tee TAB

shows you where it would jump, then

TAB

refines to the next possible target.  Hit RET when it shows you the 
place you do what to jump to.

It needs more typing since the default isn't on the prompt.  Perhaps 
that is a possible improvement.

Can all this be improved - sure.  I agree it would be nicer to mix the 
two workflows better.  Most of my time goes into the parsers etc.

> But what about duck typed languages? If a method foo calls bar on its 
> argument tee, we don't know the type of tee, all we know that it has a 
> method bar.
>

In the workflow above, you can just keep pressing TAB to pick the one 
you want.

Eric

Re: IDE

[Dmitry Gutov]

On 10/17/2015 03:00 PM, David Engster wrote:
> Eli Zaretskii writes:
>> I'm quite sure CEDET has collected and expressed in code a lot of
>> experience and solutions to many problems that arise in the context of
>> building an IDE.  It's OK to discard that, if we sure that's the
>> proverbial 1st variant everyone throws away, but we need first to be
>> sure we know what we are discarding.

I wouldn't want to discard Semantic, as long as it works better than the 
new solution, in the domain that it's actually targeting. And we should 
learn from its implementation either way.

>  From the discussion so far, I think the main issue at least w.r.t. to
> Semantic is: do you actually want Semantic's tag-based system, or more
> general: do you want quick access to AST information in your buffer?

Not really the main issue. Whether the AST is saved in overlays (and 
thus not refreshed as long as buffer is not modified), for me is an open 
question.

But not many of the existing code assistance tools that are currently 
available, provide that info, at least for dynamic languages. So it 
would be more productive to focus on the things they do provide, and 
grow the internal IDE APIs from that.

Whenever we have a tool that does provide AST dumps at will, or Semantic 
supports the language in question well, it would of course be great to 
use it.

> If I understand Dmitry correctly, he is not really interested in that,
> as for dynamic languages, the AST information is usually missing
> important information (unless you bother to implement a complete
> frontend).

Not sure what you mean exactly by "complete frontend", but it'll often 
be missing anyway. If the current method takes 'foo' as an argument, and 
the method is public, and the method contains 'foo.bar()', often the 
best thing we can tell about 'foo' is that its class contains a method 
called 'bar'.

So at least the Semantic database would have to be extended to work 
meaningfully with tags like that.

> He'd rather call external binaries for complicated stuff like
> completion, and use simpler tools (like pure regexp-based parsing) for
> stuff like font-locking, navigation, folding, etc.

I mentioned SMIE as well. It's a bit more advanced than that.

> Of course, you can
> hook external binaries into Semantic pretty easily, but I can understand
> Dmitry that if he does not need the rest of Semantic, why should he
> bother?

Being able to mix and match would be best, of course.

> Now, I think having AST information in your buffer is great, and I don't
> like depending on external binaries if I don't have to, because I want
> as much as possible in Emacs Lisp. For me, that's what Emacs is about
> and why I still use it in the first place.

My biggest issue with Semantic is that it tries to implement type 
analysis in Elisp, and still hasn't gotten it right even for simple C++ 
code (no classes or templates). Allow me to repeat myself with this: 
https://gist.github.com/dgutov/19c45ef43d1c90b96483

Suggested completions should depend on the argument tee receives ("a" 
and "b" for i, "c" and "d" for l), but they don't.

And I think re-implementing type analysis for each language in Elisp is 
untenable. Correct me if I'm wrong, but you cannot generalize that 
logic: different languages have different type systems, and for each one 
you'd have to write the analysis from scratch.

Type analysis is often at least as complex as parsing (if not more). I'm 
not in any hurry to rewrite e.g. Tern in Elisp, and then keep it up-to-date.

And speaking of more personal reasons, I already have written a code 
assistance tool for Ruby, in Ruby (which is not a Lisp, but still a 
pretty okay language), and smooth integration with whatever APIs we 
choose would be nice.

Re: IDE

[Stephen Leake]

Eric Ludlam <ericludlam@gmail.com> writes:

> Also, wisent type grammars could easily dump out data you need for
> indentation since actions can do anything you can write in lisp.  No
> one has tried to do it though.

Not quite true; I tried for Ada mode. But I ended up rewriting the
parser to be generalized LALR; see
http://stephe-leake.org/emacs/ada-mode/emacs-ada-mode.html, and the
ada-mode package in Gnu ELPA.

That parser is missing the incremental features of wisent, mostly
because I ran out of steam.

I had also tried SMIE before that; it was not powerful enough for Ada.

-- 
-- Stephe

Re: IDE

[Dmitry Gutov]

On 10/17/2015 03:45 PM, Eric Ludlam wrote:

> There is nothing about semantic the framework that prevents using
> something like SMIE as a tagging engine.  It seems likely that if some
> language not supported by CEDET today has a good SMIE grammar, and if
> SMIE could be adapted to produce compatible tags (which isn't really
> that hard to do), and if SMIE has an entry point that allows parsing a
> tag set out of a region, then it would integrate into the semantic
> framework at the best level.

With SMIE, though, it might make sense to optimize the process and only 
ask it for the current tag (and maybe the nearby ones), because it would 
otherwise be slower (no incremental parsing, or anything like that). So 
a straightforward conversion might not be optimal.

But from the main examples you gave of tag usage, knowing the current 
tag is by far the most important.

> The difference is that the parser generators in Semantic today have
> optimizations for skipping parts of the buffer while tagging.  This lets
> it parse whole files more quickly, and more robustly.  Since it uses
> syntax tables to do that, I'll guess SMIE can do it, but I'm not that
> familiar with it.

Skipping strings and comments - probably. Method bodies - unlikely 
(except in C-like languages: skipping over a pair of braces is easy).

> Also, wisent type grammars could easily dump out data you need for
> indentation since actions can do anything you can write in lisp.  No one
> has tried to do it though.

Someone should try and tell us about it.

> The mechanism in use is that there are detailed taggers, like those
> currently written for Semantic using wisent type parsers, and weak
> taggers, like GNU Global.
>
> Once a buffer is open, the detailed tagger runs and is truth.  When a
> search occurs, a process of pulling all relevant databases together is
> started.  This includes tags from the detailed taggers already cached,
> and high level 'omniscient' taggers that are usually weak, but have
> scanned the whole project.

Well, that's my point: if you don't use etags, to have all project files 
indexed you have to open each of them at some point.

Some languages have import systems where you have to declare all file's 
dependencies at its top (and so you can know which files to reindex if 
you're only interested in this file; although even that could be a lot). 
But in Ruby and pre-ES6 JavaScript, you don't usually have that.

> The output of the search includes a database
> table, and tags found in that table.  The tags are additionally refined
> based on whatever the layered criteria is, and when the code decides to
> work with a detailed tag from the search, it forces it to be real, and
> pulls it into a buffer if necessary, and any problems with the weak
> tagger is resolved.

Yup.

> The key here is that the detailed tagger is needed to do complex tasks,
> but weak taggers are great to integrate in due to the speed advantage.
> Having a mix gives you the best possible results.

If the program generating TAGS is smart enough, the result can be pretty 
accurate already (and contain qualified method name). Eli has recently 
made some improvements to that effect to etags.

> If instead you put the cursor on "tee" and do
>
> M-x semantic-ia-fast-jump RET
>
> it will go to the right spot.

Indeed it does, thanks. But it goes to the second definition 
irrespective of which argument is passed to tee. Which brings us back to 
the completion problem I reported with that snippet.

> For the keybinding you were using, the workflow is:

Right, why doesn't semantic-ia-fast-jump have a default keybinding? I 
was only looking for the appropriate command in the active keymaps.

> C-c , J tee TAB
>
> shows you where it would jump, then
>
> TAB

Oh, so it uses something more advanced than completing-read. That's 
unexpected.

>> But what about duck typed languages? If a method foo calls bar on its
>> argument tee, we don't know the type of tee, all we know that it has a
>> method bar.
>
> In the workflow above, you can just keep pressing TAB to pick the one
> you want.

I would want semantic-ia-fast-jump to work with them, of course. As well 
as semantic-symref.

Re: IDE

[David Engster]

Dmitry Gutov writes:
> (except in C-like languages: skipping over a pair of braces is easy).

After the pre-processor has run over it, yes...

#ifdef WHATEVER
struct something : public foo {
#else
struct something : public bar {
#
 // body
}

This stuff is more common than one would think...

-David

Re: IDE

[Dmitry Gutov]

On 10/17/2015 05:41 PM, David Engster wrote:
> Dmitry Gutov writes:
>> (except in C-like languages: skipping over a pair of braces is easy).
>
> After the pre-processor has run over it, yes...

Fair point. But whatever your solution for that is, it's unlikely to 
help to quickly jump over do ... end blocks in Ruby.

Re: IDE

[David Engster]

Dmitry Gutov writes:
> On 10/17/2015 03:00 PM, David Engster wrote:
>> Eli Zaretskii writes:
>>> I'm quite sure CEDET has collected and expressed in code a lot of
>>> experience and solutions to many problems that arise in the context of
>>> building an IDE.  It's OK to discard that, if we sure that's the
>>> proverbial 1st variant everyone throws away, but we need first to be
>>> sure we know what we are discarding.
>
> I wouldn't want to discard Semantic, as long as it works better than
> the new solution, in the domain that it's actually targeting. And we
> should learn from its implementation either way.
>
>>  From the discussion so far, I think the main issue at least w.r.t. to
>> Semantic is: do you actually want Semantic's tag-based system, or more
>> general: do you want quick access to AST information in your buffer?
>
> Not really the main issue. Whether the AST is saved in overlays (and
> thus not refreshed as long as buffer is not modified), for me is an
> open question.

Well, all I can say is it works well.

>> If I understand Dmitry correctly, he is not really interested in that,
>> as for dynamic languages, the AST information is usually missing
>> important information (unless you bother to implement a complete
>> frontend).
>
> Not sure what you mean exactly by "complete frontend", but it'll often
> be missing anyway. If the current method takes 'foo' as an argument,
> and the method is public, and the method contains 'foo.bar()', often
> the best thing we can tell about 'foo' is that its class contains a
> method called 'bar'.
>
> So at least the Semantic database would have to be extended to work
> meaningfully with tags like that.

The database wouldn't be much of a problem, I think. It's just
incredibly hard, if not impossible, to parse such a dynamic language
statically. For Javascript, I wrote a small semanticdb backend to query
a running Firefox process through mozrepl. That worked pretty well for
getting completions on stuff like jQuery.

>> Of course, you can
>> hook external binaries into Semantic pretty easily, but I can understand
>> Dmitry that if he does not need the rest of Semantic, why should he
>> bother?
>
> Being able to mix and match would be best, of course.

I fully agree. The above mentioned mozrepl-binding worked together with
our small Javascript parser.

>> Now, I think having AST information in your buffer is great, and I don't
>> like depending on external binaries if I don't have to, because I want
>> as much as possible in Emacs Lisp. For me, that's what Emacs is about
>> and why I still use it in the first place.
>
> My biggest issue with Semantic is that it tries to implement type
> analysis in Elisp, and still hasn't gotten it right even for simple
> C++ code (no classes or templates). Allow me to repeat myself with
> this: https://gist.github.com/dgutov/19c45ef43d1c90b96483

It's a matter of someone implementing overloads. And such a
implementation could be used from any other languages that has
overloading on function arguments, which is pretty common.

> And I think re-implementing type analysis for each language in Elisp
> is untenable. Correct me if I'm wrong, but you cannot generalize that
> logic: different languages have different type systems, and for each
> one you'd have to write the analysis from scratch.

There are a lot of similarities in C-like languages. Also, any
OOP-language will have something like classes, parents, methods,
attributes. But yes, type inference and dynamic languages make things
more complicated. Querying an external REPL or some tool that analyzes
the code would often be necessary.

> Type analysis is often at least as complex as parsing (if not
> more).

For C++11, it has become more complex, which is why I will indeed ask an
external tool for type information. Since such a tool will have to build
the AST anyway, it will provide that as well.

> And speaking of more personal reasons, I already have written a code
> assistance tool for Ruby, in Ruby (which is not a Lisp, but still a
> pretty okay language), and smooth integration with whatever APIs we
> choose would be nice.

Yes, of course.

To be honest, I mostly lost track about what is actually discussed
here. I just took offense at John's "if CEDET was the answer we wouldn't
still be asking questions" and avoiding a "CEDET2", like the CEDET we
have in core is a failure.

CEDET tries to walk a narrow path, trying to provide IDE-like features
without Emacs actually becoming a "typical IDE". The IDEs out there have
it easier, as they usually force you into organizing your projects in a
certain way, and they usually target only one language (or language
family).

-David

Re: IDE

[Przemysław Wojnowski]

W dniu 17.10.2015 o 10:39, David Kastrup pisze:
[...]
> My refactoring work tends to be done using stuff like
 >[...]
With all due respect, please spend a morning with a modern IDE (like
Intellij, Eclipse) to learn how they make programmers productive.
Especially look at their refactoring features. It's a different league.

Re: IDE

[David Kastrup]

Przemysław Wojnowski <esperanto@cumego.com> writes:

> W dniu 17.10.2015 o 10:39, David Kastrup pisze:
> [...]
>> My refactoring work tends to be done using stuff like
>>[...]
> With all due respect, please spend a morning with a modern IDE (like
> Intellij, Eclipse) to learn how they make programmers productive.

Not really interested.  This refactoring was necessitated by stuff being
implemented with a contorted C macro system.  Any refactoring system
able to do this out of the box would be too complicated to learn.

> Especially look at their refactoring features. It's a different
> league.

To sed?  Uh, that's not a particularly interesting contest since sed has
no refactoring features at all.  Multi-line replacements are noisome
enough.  At any rate, I am not interested in using other tools here.
Scripting means I can work incrementally and put the respective
responsible script into the commit message.

Which means that redoing that particular refactoring when branches are
rebased or my commit takes a week for reviewing while other changes pour
in is then trivial.

-- 
David Kastrup

Re: IDE

[Eli Zaretskii]

> From: David Kastrup <dak@gnu.org>
> Cc: John Wiegley <johnw@newartisans.com>,  emacs-devel@gnu.org
> Date: Sat, 17 Oct 2015 10:39:51 +0200
> 
> > Could it be that we don't understand the answer?
> >
> > I'd suggest to be very careful with such conclusions.  They can only
> > be valid when based on a detailed analysis of what is and isn't in
> > CEDET, and on good knowledge and understanding of its design and
> > implementation.
> 
> If we wanted that, we would be using vi.

Since vi isn't part of Emacs, this is irrelevant.

> I'm old and experienced enough that I have the arrogance to claim that
> something that is too hard for me to understand and/or use effectively
> after putting in a reasonable amount of effort is not a generally useful
> tool.

I'm older, but until now there wasn't a single thing I wanted to
understand and/or use effectively that I couldn't.  I guess our idea
of "reasonable amount" differs.

> > My impression so far is that neither is particularly true, and my
> > evidence is the number of times Eric and David Engster described some
> > CEDET features that came as a surprise to us.
> 
> Is that supposed to be a good thing?

Not if we pretend to claim we know what CEDET is.

> M-x grep RET works a lot smoother than M-! grep and the kind of
> difference between the two are where Emacs shines, providing the editor
> connection to technology.

That's fine.  IDE isn't for everyone.

Re: IDE

[Lluís]

Dmitry Gutov writes:

> On 10/16/2015 07:47 PM, Lluís wrote:
>> * service-type: A type of service (one instance per type), like build, find root
>> directory, include search paths, compilation flags, etc.
>> 
>> * service: A specific implementation of a service type, like using vc-dir for
>> finding the root directory of a project.

> This sounds good. The implementations could be different (e.g. Java-ish Service
> Locator [0]), but in the absence of other requirements the closest Emacs
> idiomatic structure would be a hook (list of functions) for each service
> type. Like project-find-functions, for example.

> Note that the objects the latter currently returns provide both "find root" and
> "search path" services. Thus far it seemed to make sense. Can one, and would one
> want to create different search-path services working with the same find-root
> service? That's an interesting question, and I'd love to see a practical
> example.

I didn't think of the "find path" functionality, but it does seem to make sense
to fold it into the "root path" service type.


> More generally, if a package P registers services A, B and C, if B needs some
> information from A, would we expect them to go through P-internal channels, or
> the public interface? For instance, most services might like to know the project
> root (let's call that service type R).

> We could standardize the dependencies like that. In the simplest case, all
> services depend on project-root (except for itself), and we'll pass the current
> project-root instance to each functions in their hooks. Then an element of
> service-b-functions will only return non-nil if that implementation of B can
> work with the given kind of project-root (or simply "project"; we might include
> some more info into that instance as well).

> On the flip side, if there are no B implementations that can work with the
> currently detected R service r1, but there's a certain b2 in service-b-functions
> that would work with r2 (currently shadowed by r1), we may be missing out.

Exactly. I started prototyping a few interfaces where all requests go through
the public interfaces. A project-type acts similarly to your "service locator"
link, and projects are simply a service locator that manage multiple
project-type instances grouped by the project.

More specifically, I was thinking about the following for deciding how to manage
results when information comes from different service implementations given a
service-type: for each public method of a service type, have a prioritized list
of method implementations, plus a property that tells us how to merge their
results. Possible properties would be returning the first non-error result from
the possible method instances or returning a concatenation of all non-error
results from all method instances.


>> * service-collection: A set of service implementations of the same service
>> type. This allows aggregating results from each service into a single answer.

> I suppose you can get this from the same service-b-functions hook, if you treat
> it in a different way.

While prototyping it, I actually decided it makes sense to implement this
functionality on the service-type. It should know how to merge results from
multiple services, and every service must declare what service-type it provides.


>> * project-type: A type of project defines the service collections it provides
>> for each service type it knows about. An example could be an Emacs project
>> (knows how to auto-detect it, things like the build system it uses, etc.), and
>> Android project, a Linux kernel project, an autotools project, etc.

> Having to declare each combination of service-types that a project can provide
> still seems unnecessarily limiting (and tedious). But that's what project-types
> would be for, right?

Exactly. My idea was that common project types would be written once declaring
all the service types they provide and through what service
implementations. In order to make projects easily customizable, there should
also be a project-type that can be dynamically constructed from some user
settings (some call in init.el or values on directory variables). Then you
simply overlay the custom project-type on top of some other project-type that is
auto-detected or specified by the user.


> How would a client even use those declarations? If we were to go that way, it'd
> be better to simply have a project instance have a method that returns a list of
> all services it supports. Or, even simpler, return a nil or raise a
> not-implemented error in each of the generic methods it doesn't implement.

I'm not sure I follow, but maybe I responded you on my previous paragraphs.


> Next, suppose a project instance can tell whether it provides a "build tool"
> service. What does a "call build task" command does? Looks for the current
> project and gives up if it doesn't provide the required service, or specifically
> asks the locator for a project implementation that does provide that service?

I'd say both. My idea is that all project-types provide an auto-detection
function; given a path, tell me if it conforms to this project-type. Then, if we
do not know of any project instance for that path, build a new one that contains
instances of all project-types that match with it.


> In the latter case, we potentially buy more functionality, at the cost of having
> different commands disagree about what the current project actually is.

> [0] http://martinfowler.com/articles/injection.html#ADynamicServiceLocator

Building a project from all matching project-types provides a clean solution to
this. But to be honest, I'm not sure if it makes much sense beyond taking a
single auto-detected project type plus an optional overlaid project-type that
contains user customizations.

For example, we could have a fallback "project-type-generic", a makefile-based
"project-type-make" and a very specific "project-type-linux". Should we overlay
all of them or just take the most "specific" one? (aka linux).

In the overlaying case, there should be a way to override decisions from other
project-types. This can get hairy, since in the beginning I said that each
per-service method implementation of a service-type method follows some property
(e.g., first match or concatenate), so now we want to override this. Should the
override be per method? Per service? At the entire project-type level?

In the second case, we can simply assign some priority number and take the
highest-priority project-type, and overlay the user-customizable project-type as
a special case.

I really am not sure about the project-type and project part.


Cheers,
  Lluis

-- 
"And it's much the same thing with knowledge, for whenever you learn
something new, the whole world becomes that much richer."
-- The Princess of Pure Reason, as told by Norton Juster in The Phantom
Tollbooth

Re: IDE

[Dmitry Gutov]

On 10/17/2015 06:26 PM, David Engster wrote:

> The database wouldn't be much of a problem, I think. It's just
> incredibly hard, if not impossible, to parse such a dynamic language
> statically.

Maybe not the database, but the "tag" value is used in many places in 
Semantic. I'd want to be able to use them for, say, JavaScript as well. 
For instance, in semantic-symref, not just for completions.

> For Javascript, I wrote a small semanticdb backend to query
> a running Firefox process through mozrepl. That worked pretty well for
> getting completions on stuff like jQuery.

I'll have to check it out later.

> It's a matter of someone implementing overloads. And such a
> implementation could be used from any other languages that has
> overloading on function arguments, which is pretty common.

I'm sure it can be implemented in Elisp, just surprised it hasn't been 
taken care of since a similar example came came up 6 (or 12?) months 
ago, in a Clang-related discussion.

> There are a lot of similarities in C-like languages. Also, any
> OOP-language will have something like classes, parents, methods,
> attributes. But yes, type inference and dynamic languages make things
> more complicated. Querying an external REPL or some tool that analyzes
> the code would often be necessary.

C-like languages - probably, but every one also has tiny peculiarities 
which have to be handled in a different way.

But Emacs users are a diverse bunch, and for many the draw is in being 
able to use many different languages, including very young ones. So a 
general IDE solution should anticipate having to handle different type 
systems.

>> Type analysis is often at least as complex as parsing (if not
>> more).
>
> For C++11, it has become more complex, which is why I will indeed ask an
> external tool for type information. Since such a tool will have to build
> the AST anyway, it will provide that as well.

I'm glad we agree on this. But Java has also become more complex (since 
1.5, maybe?), and Java 8 added lambdas, which allow omitting argument 
types. Not sure about things about Objective-C land.

But otherwise, if it'll be common to use an external tool for type 
resolution, and even parsing the buffer contents, maybe at some point 
you'll deprecate Wisent grammars. Or use, say, a very simplified format 
for them, only what's strictly necessary to find the names and 
boundaries of tags.

> To be honest, I mostly lost track about what is actually discussed
> here. I just took offense at John's "if CEDET was the answer we wouldn't
> still be asking questions" and avoiding a "CEDET2", like the CEDET we
> have in core is a failure.

Personally, I'm happy discussing and figuring out the current state of 
affairs, so that our common understanding goes further than "CEDET 
didn't live up to its promises" or "don't throw out CEDET, we've got 
nothing better".

The EDE subthread also brought up some ideas for project.el.

> CEDET tries to walk a narrow path, trying to provide IDE-like features
> without Emacs actually becoming a "typical IDE". The IDEs out there have
> it easier, as they usually force you into organizing your projects in a
> certain way, and they usually target only one language (or language
> family).

That's not really true. IntelliJ IDEA supports a big swath of languages, 
and my colleagues use it successfully for our "non-standard" Ruby 
projects (no Rails), and also with JS and different markup languages. 
But of course you have different challenges with C++, and the JetBrains 
team has more manpower anyway.

But I'd be ecstatic to even have a consistent UI for features that VS 
Code (smaller cousin of Visual Studio) touts here: 
https://code.visualstudio.com/docs/editor/editingevolved

Re: IDE

[Przemysław Wojnowski]

W dniu 17.10.2015 o 22:19, Dmitry Gutov pisze:
[...]
>> There are a lot of similarities in C-like languages. Also, any
>> OOP-language will have something like classes, parents, methods,
>> attributes. But yes, type inference and dynamic languages make things
>> more complicated. Querying an external REPL or some tool that analyzes
>> the code would often be necessary.
>
> C-like languages - probably, but every one also has tiny peculiarities which
> have to be handled in a different way.
>
> But Emacs users are a diverse bunch, and for many the draw is in being able to
> use many different languages, including very young ones. So a general IDE
> solution should anticipate having to handle different type systems.
I think it is not a problem. An IDE could switch (or enable) language backend
depending on current language. For C-like (or maybe statically typed in
general) languages (covering most of programming world) could use Semantic, for
other languages maybe something other (like tern for JS).

[...]
> The EDE subthread also brought up some ideas for project.el.
Does that mean that you don't want to reuse EDE, but reimplement everything
from scratch? Don't you think it would be better to reuse what already is and
just change parts of it to be more flexible?

>> CEDET tries to walk a narrow path, trying to provide IDE-like features
>> without Emacs actually becoming a "typical IDE". The IDEs out there have
>> it easier, as they usually force you into organizing your projects in a
>> certain way, and they usually target only one language (or language
>> family).
>
> That's not really true. IntelliJ IDEA supports a big swath of languages, and my
> colleagues use it successfully for our "non-standard" Ruby projects (no Rails),
> and also with JS and different markup languages. But of course you have
> different challenges with C++, and the JetBrains team has more manpower anyway.
Intellij's support for JavaScript is not so great. Compared to Java it is poor.

Re: IDE

[Dmitry Gutov]

On 10/18/2015 01:03 AM, Przemysław Wojnowski wrote:

> I think it is not a problem. An IDE could switch (or enable) language
> backend
> depending on current language. For C-like (or maybe statically typed in
> general) languages (covering most of programming world) could use
> Semantic, for
> other languages maybe something other (like tern for JS).

It would be better if we could support a set of common operations for 
both static and dynamic languages. If those were implemented by Semantic 
by static languages, more power to it.

>> The EDE subthread also brought up some ideas for project.el.
> Does that mean that you don't want to reuse EDE, but reimplement everything
> from scratch? Don't you think it would be better to reuse what already
> is and
> just change parts of it to be more flexible?

Have you looked at lisp/progmodes/project.el yet?

> Intellij's support for JavaScript is not so great. Compared to Java it
> is poor.

Compared to most other JS editors, it's pretty great. Or so I'm told.

Re: IDE

[Przemysław Wojnowski]

W dniu 18.10.2015 o 00:07, Dmitry Gutov pisze:
> On 10/18/2015 01:03 AM, Przemysław Wojnowski wrote:
>
>> I think it is not a problem. An IDE could switch (or enable) language
>> backend
>> depending on current language. For C-like (or maybe statically typed in
>> general) languages (covering most of programming world) could use
>> Semantic, for
>> other languages maybe something other (like tern for JS).
>
> It would be better if we could support a set of common operations for both
> static and dynamic languages. If those were implemented by Semantic by static
> languages, more power to it.
I don't think it is possible, because languages are very different and their
surrounding tooling is very different.

>>> The EDE subthread also brought up some ideas for project.el.
>> Does that mean that you don't want to reuse EDE, but reimplement everything
>> from scratch? Don't you think it would be better to reuse what already
>> is and
>> just change parts of it to be more flexible?
>
> Have you looked at lisp/progmodes/project.el yet?
Yes. And I've looked at EDE at SF 
(http://sourceforge.net/p/cedet/git/ci/master/tree/lisp/cedet/ede/). There's 
support for many types of projects, build tools, etc. There are even some tests.

What's the point of reimplementing that from scratch?

Re: IDE

[Dmitry Gutov]

On 10/18/2015 01:28 AM, Przemysław Wojnowski wrote:

> I don't think it is possible, because languages are very different and
> their
> surrounding tooling is very different.

The meanings of "go to definition", "find references" and "complete text 
at point" are very much the same across languages. Some refactorings, too.

> Yes. And I've looked at EDE at SF
> (http://sourceforge.net/p/cedet/git/ci/master/tree/lisp/cedet/ede/).
> There's support for many types of projects, build tools, etc. There are
> even some tests.

A more idiomatic, Emacs-y API that isn't tied to CEDET, that's friendly 
to third-party packages, and doesn't ask its caller to know the 
intricacies of EDE's object hierarchies.

You're welcome to work on making EDE more flexible, though. We can 
compare progress later.

> What's the point of reimplementing that from scratch?

We can reuse code from EDE, if it fits.

Re: IDE

[Dmitry Gutov]

On 10/17/2015 08:18 PM, Lluís wrote:

> I didn't think of the "find path" functionality, but it does seem to make sense
> to fold it into the "root path" service type.

You mean "find search path", right? Another name if it is "include 
path", and that's different from the executable path.

I think I've formed an idea how separating it from the find-root(s) 
service would be useful. Currently, we have this awkward 
project-search-path-function, which can be defined in a major mode (we 
want this kind of functionality to be major mode-providable).

But if it were a separate service, that would be tidier. Introduce 
project-search-path-functions, which would take the current project as 
an argument and return an object that contains the search path (and, as 
a bonus, maybe knows how to resolve a relative import string).

Then ruby-mode won't have to define a separate project type. It'll just 
add to project-search-path-functions that checks that there is a Gemfile 
somewhere above the current directory and below the project root, and 
use the information from it. The fact that returned object can also 
resolve imports is nice, because I wasn't sure where to put that behavior.

In general, though, this might be problematic when a language L will 
simply read its search path from an environment variable, and doesn't 
require a presence of any "bundle" file. Does it provide its search-path 
to absolutely any project? We might need a "list of project languages" 
accessor on the "root" service instances. User-customizable, of course.

>> On the flip side, if there are no B implementations that can work with the
>> currently detected R service r1, but there's a certain b2 in service-b-functions
>> that would work with r2 (currently shadowed by r1), we may be missing out.
>
> Exactly. I started prototyping a few interfaces where all requests go through
> the public interfaces. A project-type acts similarly to your "service locator"
> link, and projects are simply a service locator that manage multiple
> project-type instances grouped by the project.

You mean different project-types are like entirely different collections 
of services? Why call them project-types at all? I'm lost.

Isn't a project a collection of services (of different service-types)?

> More specifically, I was thinking about the following for deciding how to manage
> results when information comes from different service implementations given a
> service-type: for each public method of a service type, have a prioritized list
> of method implementations, plus a property that tells us how to merge their
> results.

We're unlikely to have too many service-types. No need for a property, 
merging could be done by a respective utility function (like 
`project-current', but for other services). Priority comes from the 
order of the elements in a hook.

> Possible properties would be returning the first non-error result from
> the possible method instances or returning a concatenation of all non-error
> results from all method instances.

Those are the two main options. Not sure if we'll ever really need 
anything else.

> While prototyping it, I actually decided it makes sense to implement this
> functionality on the service-type. It should know how to merge results from
> multiple services, and every service must declare what service-type it provides.

How about we reframe the description in terms of hooks? service-type is 
a hook variable, services are elements in it. You can call the elements 
of a hook in a sequence until one returns non-nil, or you can call all 
of them and combine.

>> Having to declare each combination of service-types that a project can provide
>> still seems unnecessarily limiting (and tedious). But that's what project-types
>> would be for, right?
>
> Exactly. My idea was that common project types would be written once declaring
> all the service types they provide and through what service
> implementations.

So, why?

Take a look at xref-find-regexp. It looks up the current project, takes 
its roots (service 1) and its search-path (service 2). If either of the 
services can be undefined in the current project, what would 
xref-find-regexp be supposed to do?

And what is the use of several common project types? Could you give an 
example?

> I'm not sure I follow, but maybe I responded you on my previous paragraphs.

How would a random command that needs certain info from services x, y 
and z, use this API?

>> Next, suppose a project instance can tell whether it provides a "build tool"
>> service. What does a "call build task" command does? Looks for the current
>> project and gives up if it doesn't provide the required service, or specifically
>> asks the locator for a project implementation that does provide that service?
>
> I'd say both. My idea is that all project-types provide an auto-detection
> function; given a path, tell me if it conforms to this project-type.

In Emacs parlance, a "path" is a list of directories (see exec-path or 
compilation-search-path).

The call-build-type needs the build-tool service. What project-type does 
it use, and why?

> Then, if we
> do not know of any project instance for that path, build a new one that contains
> instances of all project-types that match with it.

What if we already have a project instance for that directory, but it 
doesn't provide that service?

>> [0] http://martinfowler.com/articles/injection.html#ADynamicServiceLocator
>
> Building a project from all matching project-types provides a clean solution to
> this. But to be honest, I'm not sure if it makes much sense beyond taking a
> single auto-detected project type plus an optional overlaid project-type that
> contains user customizations.

I don't think user customizations should be a project-type. Rather, they 
can be applied via .dir-locals.el, inside each service-type utility 
function.

> For example, we could have a fallback "project-type-generic", a makefile-based
> "project-type-make" and a very specific "project-type-linux". Should we overlay
> all of them or just take the most "specific" one? (aka linux).

The one that comes first in the hook. Hook entries from minor modes 
usually come before the hook entries from major modes, and it's user's 
responsibility not to enable several minor modes at once that share the 
same responsibility.

Re: IDE

[John Wiegley]

>>>>> Eli Zaretskii <eliz@gnu.org> writes:

> I'm quite sure CEDET has collected and expressed in code a lot of experience
> and solutions to many problems that arise in the context of building an IDE.
> It's OK to discard that, if we sure that's the proverbial 1st variant
> everyone throws away, but we need first to be sure we know what we are
> discarding.

I'm not suggesting we discard experiences. What I'm saying is: it doesn't make
sense to proceed by looking at CEDET, and then asking what should be changed.

CEDET is like a hammer. When it was made, the problem looked like nail.

Today, the problem might be a screw (is it? do we know?). We're not going to
arrive at the best answer by asking ourselves how a hammer can be changed to
meet the needs of a screw. It deserves looking at the problem anew.

It doesn't mean we throw out the hammer. Maybe we do have a nail, maybe we
don't. The point is: If we make technical assumptions before learning what we
want to end up with, we're going to arrive at something shaped more by those
assumptions than by our needs.

So unless there are other features I should bear in mind, I'm going to turn my
attention away from CEDET now and back to the IDE vision I'd like everyone's
help with, once there is more to say.

John

Re: IDE

[Steinar Bang]

>>>>> David Kastrup <dak@gnu.org>:

> M-x grep RET works a lot smoother than M-! grep and the kind of
> difference between the two are where Emacs shines, providing the editor
> connection to technology.

M-x rgrep is even nicer, particularily in maven Java projects (with
deeply nested directory hierarchies).

My own mass edits tends to be done in a combination of emacs keyboard
macros run over rgrep hits.

And the mass edits tend to be done in emacs, rather than the IDE I'm
officially using at that time...

(Side note: If I had a task that would require 6 hour manual editing I
would probably script it in perl (I am trying to replace my use of perl
with python, but I keep having relapses...))

Re: IDE

[Przemysław Wojnowski]

W dniu 18.10.2015 o 00:37, Dmitry Gutov pisze:
> On 10/18/2015 01:28 AM, Przemysław Wojnowski wrote:
>
>> I don't think it is possible, because languages are very different and
>> their
>> surrounding tooling is very different.
>
> The meanings of "go to definition", "find references" and "complete text at
> point" are very much the same across languages.
Yes, and EDE could be changed to provide such API.

> Some refactorings, too.
And some are not and will never be, because languages are too different. So
IDE's support for them will be different or crippled to the lowest denominator
as you seem to suggest.

>> Yes. And I've looked at EDE at SF
>> (http://sourceforge.net/p/cedet/git/ci/master/tree/lisp/cedet/ede/).
>> There's support for many types of projects, build tools, etc. There are
>> even some tests.
>
> A more idiomatic, Emacs-y
What does that mean? Is it described somewhere what is idiomatic in Emacs Lisp?

> API that isn't tied to CEDET, that's friendly to
> third-party packages, and doesn't ask its caller to know the intricacies of
> EDE's object hierarchies.
Yes. This part would need to be changed in EDE.

> You're welcome to work on making EDE more flexible, though. We can compare
> progress later.
I don't waste resources on duplicated efforts just to show my ego. Have many 
other things to work on.

>> What's the point of reimplementing that from scratch?
>
> We can reuse code from EDE, if it fits.
Depending on what you means by "it fits", but if that's possible, it's ok for
me.

My only concern is that EDE is something that already works, but needs 
improvement. Creating, more or less, the same from scratch is IMHO reinventing 
the wheel, which may not even end being as round as EDE is.

Also reusing EDE maybe would "reuse" developers that were/are working on it and
hence Emacs would have another contributors for free. Not doing so may mean
that some devs will work on EDE and some on project.el, duplicating many things
and loosing steam by lone work. Future? Two incomplete packages that do mostly
the same, but unusable for users - the most common scenario in open source
world (see JDEE, malabar-mode, eclim).

Re: IDE

[Dmitry Gutov]

On 10/18/2015 12:08 PM, Przemysław Wojnowski wrote:

>> The meanings of "go to definition", "find references" and "complete
>> text at
>> point" are very much the same across languages.
> Yes, and EDE could be changed to provide such API.

You probably mean Semantic.

>> Some refactorings, too.
> And some are not and will never be, because languages are too different. So
> IDE's support for them will be different or crippled to the lowest
> denominator as you seem to suggest.

You're just arguing for the sake of arguing here. We could have a common 
UI for implementing refactoring commands, and some of those commands 
could be different for different languages. Or something along these lines.

>> A more idiomatic, Emacs-y
> What does that mean? Is it described somewhere what is idiomatic in
> Emacs Lisp?

In the manual, maybe. I wouldn't know.

Point is, we can have a better, friendlier and more general API. And 
when we know that it makes sense, we can also change the existing code 
to fit it.

> I don't waste resources on duplicated efforts just to show my ego. Have
> many other things to work on.

Working on EDE can be more than an ego trip: maybe you'll stumble on a 
good, practical flexible design (which isn't out of the question, given 
a lot of code you'd have to keep working). And if EDE is more flexible, 
most likely we could reuse pieces of it more easily as well.

Did you actually intend to do some work on project support, or was that 
just backseat driving?

>> We can reuse code from EDE, if it fits.
> Depending on what you means by "it fits", but if that's possible, it's
> ok for me.

Build tool support code should fit the bill when project.el supports 
build tools (or maybe we'll separate that into build-tool.el).

> My only concern is that EDE is something that already works, but needs
> improvement. Creating, more or less, the same from scratch is IMHO
> reinventing the wheel, which may not even end being as round as EDE is.

Projectile already works, too. And eproject. And ENSIME, and Eclim. The 
two last examples also use their own code to manage projects, and I 
suspect it would be easier to ask them to implement a bridge to 
project.el, rather than migrate to EDE.

> Also reusing EDE maybe would "reuse" developers that were/are working on
> it and

Nope. David and Eric are pretty busy, and won't magically become active 
Emacs contributors just because of that.

> hence Emacs would have another contributors for free. Not doing so may mean
> that some devs will work on EDE and some on project.el, duplicating many
> things
> and loosing steam by lone work.

project.el is foremost an API, so it needs better design proposals, 
rather than more teamwork, right now. Improving EDE design would also 
help there.

> the same, but unusable for users - the most common scenario in open source
> world (see JDEE, malabar-mode, eclim).

I wish wish we could say that each of those projects has got *something* 
working really well, so it would make sense to talk about combining them.

Re: IDE

[David Engster]

Dmitry Gutov writes:
> On 10/17/2015 06:26 PM, David Engster wrote:
>
>> The database wouldn't be much of a problem, I think. It's just
>> incredibly hard, if not impossible, to parse such a dynamic language
>> statically.
>
> Maybe not the database, but the "tag" value is used in many places in
> Semantic. I'd want to be able to use them for, say, JavaScript as
> well. For instance, in semantic-symref, not just for completions.

In dynamic languages, you would mostly have tags with no specific type.

>> For Javascript, I wrote a small semanticdb backend to query
>> a running Firefox process through mozrepl. That worked pretty well for
>> getting completions on stuff like jQuery.
>
> I'll have to check it out later.

It is only upstream, and I wouldn't be surprised if it is broken by now
due to changes in mozrepl (this is why I don't like depending on
external binaries). I don't code Javascript anymore.

>> It's a matter of someone implementing overloads. And such a
>> implementation could be used from any other languages that has
>> overloading on function arguments, which is pretty common.
>
> I'm sure it can be implemented in Elisp, just surprised it hasn't been
> taken care of since a similar example came came up 6 (or 12?) months
> ago, in a Clang-related discussion.

Well, it's not like a bunch of people are hacking on the C
parser. Several things happened:

- Emacs switched to git, so CEDET had to switch as well, which broke my
  merge setup. I still need to repair it, which is extremely boring
  work.

- CEDET does not work with Emacs25 due to large changes in
  EIEIO. Someone has to fix that as well. Again: extremely boring work.

- This gcc-AST discussion.

- New kid.

- In general, hacking Emacs isn't as much fun as it used to be.

>> There are a lot of similarities in C-like languages. Also, any
>> OOP-language will have something like classes, parents, methods,
>> attributes. But yes, type inference and dynamic languages make things
>> more complicated. Querying an external REPL or some tool that analyzes
>> the code would often be necessary.
>
> C-like languages - probably, but every one also has tiny peculiarities
> which have to be handled in a different way.

Which is why Semantic allows to tweak pretty much any part through
mode-local overrides.

> But Emacs users are a diverse bunch, and for many the draw is in being
> able to use many different languages, including very young ones. So a
> general IDE solution should anticipate having to handle different type
> systems.

Of course. That's the goal of CEDET. If it had just focused on C/C++, it
would be simpler.

>>> Type analysis is often at least as complex as parsing (if not
>>> more).
>>
>> For C++11, it has become more complex, which is why I will indeed ask an
>> external tool for type information. Since such a tool will have to build
>> the AST anyway, it will provide that as well.
>
> I'm glad we agree on this. But Java has also become more complex
> (since 1.5, maybe?), and Java 8 added lambdas, which allow omitting
> argument types.

Yes. If people want to hook in Eclim or whatever, that's fine be me. But
it's always the same thing: Those people are *only* iterested in Java,
and they don't want to bother with the bigger picture. This is exactly
why we have a plethora of solutions for different languages at the
moment.

> Not sure about things about Objective-C land.

It's deserted and people move to Swift land.

> But otherwise, if it'll be common to use an external tool for type
> resolution, and even parsing the buffer contents, maybe at some point
> you'll deprecate Wisent grammars. Or use, say, a very simplified
> format for them, only what's strictly necessary to find the names and
> boundaries of tags.

Maybe.

>> CEDET tries to walk a narrow path, trying to provide IDE-like features
>> without Emacs actually becoming a "typical IDE". The IDEs out there have
>> it easier, as they usually force you into organizing your projects in a
>> certain way, and they usually target only one language (or language
>> family).
>
> That's not really true. IntelliJ IDEA supports a big swath of
> languages, and my colleagues use it successfully for our
> "non-standard" Ruby projects (no Rails), and also with JS and
> different markup languages. But of course you have different
> challenges with C++, and the JetBrains team has more manpower anyway.

Yes, and do you know how the Jetbrains guys achieve this? They have an
extensive framework for writing grammars, lexers, etc. Those guys are
weird!

> But I'd be ecstatic to even have a consistent UI for features that VS
> Code (smaller cousin of Visual Studio) touts here:
> https://code.visualstudio.com/docs/editor/editingevolved

Ever tried to load some random make-based C++ project into Visual C++?

-David

Re: IDE

[David Kastrup]

David Engster <deng@randomsample.de> writes:

> Well, it's not like a bunch of people are hacking on the C
> parser. Several things happened:
>
> - Emacs switched to git, so CEDET had to switch as well, which broke my
>   merge setup. I still need to repair it, which is extremely boring
>   work.
>
> - CEDET does not work with Emacs25 due to large changes in
>   EIEIO. Someone has to fix that as well. Again: extremely boring work.
>
> - This gcc-AST discussion.
>
> - New kid.
>
> - In general, hacking Emacs isn't as much fun as it used to be.

To me it seems like changing the last item would likely have the most
long-term impact while likely being the most elusive target.

-- 
David Kastrup

Re: IDE

[Dmitry Gutov]

On 10/18/2015 02:56 PM, David Engster wrote:

> In dynamic languages, you would mostly have tags with no specific type.

I'd say 50/50. Calls with implicit target can roughly be expected to 
have the current class (which can be determined lexically) as the 
target. Similarly for calls on 'this' or 'super'.

Add to this calls on variables that have been assigned a value 
instantiated in the current lexical scope (or, say, method), or the 
results of standard library calls that we've analyzed in advance, where 
we can determine the type of output. This kind of analysis would be 
better left to an external tool, though.

> It is only upstream, and I wouldn't be surprised if it is broken by now
> due to changes in mozrepl (this is why I don't like depending on
> external binaries). I don't code Javascript anymore.

Yes, it didn't seem to work last time I checked (a while ago). The 
implementation should be interesting anyway.

> Well, it's not like a bunch of people are hacking on the C
> parser. Several things happened:

It's not like I'm blaming anyone, really. But it leaves an impression of 
CEDET being more of a research project.

> - In general, hacking Emacs isn't as much fun as it used to be.

Many of us can sympathize, I'm sure.

> Yes, and do you know how the Jetbrains guys achieve this? They have an
> extensive framework for writing grammars, lexers, etc. Those guys are
> weird!

I'm sure that's not all they have. They have completion, and they have 
(at least some kind of) refactorings using the same interface across 
products. That hints at flexible coupling between components.

Or maybe not. EDE-like structure might work for them as well.

>> But I'd be ecstatic to even have a consistent UI for features that VS
>> Code (smaller cousin of Visual Studio) touts here:
>> https://code.visualstudio.com/docs/editor/editingevolved
>
> Ever tried to load some random make-based C++ project into Visual C++?

It probably won't work. But so what? It's great that you have a solution 
for this in CEDET, but it shouldn't impose particular constraints on 
what a project API should look like. At least I don't see why or how it 
should.

My point was about user-facing features anyway. If we only provide them 
initially for languages with simpler/standardized dependency management, 
that would be a big step forward already.

Re: IDE

[Eli Zaretskii]

> From: John Wiegley <johnw@newartisans.com>
> Date: Sat, 17 Oct 2015 22:23:30 -0700
> 
> >>>>> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > I'm quite sure CEDET has collected and expressed in code a lot of experience
> > and solutions to many problems that arise in the context of building an IDE.
> > It's OK to discard that, if we sure that's the proverbial 1st variant
> > everyone throws away, but we need first to be sure we know what we are
> > discarding.
> 
> I'm not suggesting we discard experiences. What I'm saying is: it doesn't make
> sense to proceed by looking at CEDET, and then asking what should be changed.
> 
> CEDET is like a hammer. When it was made, the problem looked like nail.
> 
> Today, the problem might be a screw (is it? do we know?). We're not going to
> arrive at the best answer by asking ourselves how a hammer can be changed to
> meet the needs of a screw. It deserves looking at the problem anew.
> 
> It doesn't mean we throw out the hammer. Maybe we do have a nail, maybe we
> don't. The point is: If we make technical assumptions before learning what we
> want to end up with, we're going to arrive at something shaped more by those
> assumptions than by our needs.

I wasn't aware that the IDE landscape might have changed in such a
significant way recently.  This discussion seems to focus on details,
which seems to indicate no such radical changes happened.  But I'm not
an expert, so maybe you are right.

I did suggest up-thread to come up with a list of main features we
think an Emacs IDE should have.  If that is what you have in mind, I
obviously agree.

In any case, CEDET is not an EDE, AFAIK.  It is an infrastructure and
a set of tools for building an IDE.  IOW, it's neither a hammer nor a
screwdriver, but something that allows us to make one or the other (or
something else entirely).  So it could very well be a good basis for
an Emacs IDE.

Re: IDE

[David Engster]

Dmitry Gutov writes:
>> Well, it's not like a bunch of people are hacking on the C
>> parser. Several things happened:
>
> It's not like I'm blaming anyone, really. But it leaves an impression
> of CEDET being more of a research project.

It leaves the impression of an understaffed project.

> I'm sure that's not all they have. They have completion, and they have
> (at least some kind of) refactorings using the same interface across
> products. That hints at flexible coupling between components.

It's not a secret:

http://www.jetbrains.org/intellij/sdk/docs/reference_guide/custom_language_support.html

I haven't looked at it in detail, but it looks familiar to me.

>> Ever tried to load some random make-based C++ project into Visual C++?
>
> It probably won't work. But so what? It's great that you have a
> solution for this in CEDET,

We don't. Makefiles are way too complex. The best solution IMHO is using
compilation databases, and EDE upstream has support for them. Of course,
it's another LLVM thingy, so I'm not sure if I'm allowed to merge it.

> but it shouldn't impose particular constraints on what a project API
> should look like. At least I don't see why or how it should.

My point is that people say they want an Emacs to be an "IDE", but at
the same time they don't want to be restricted in their Emacs usage, and
rightfully so. A typical IDE like Visual C++ forces you into using their
project system, otherwise nothing will work. So people want
IDE-features, but at the same time they want all the freedom Emacs
allows. And that's what CEDET tries to support.

Eric started with EDE as a project system similar to other IDEs, that
means you can start a project from scratch with EDE and it will manage
the build for you by generating a Makefile. While technically
impressive, I never liked this kind of project system, and honestly, I
think we should probably drop that part from EDE. But you are not forced
to use it, because Eric also added a custom project detection that is
very flexible. But if you manage the build yourself, you have to specify
everything manually (include paths, used compiler, etc.).

It doesn't help that Emacs is a very conservative piece of software. A
good example was already given: C++ includes without an extension. By
default, Emacs will open such files in fundamental mode. The GNU libc
actually has local variables comments in their includes just for that
reason! But of course, Qt does not have those, so Emacs cannot parse
those files by default, and people complain about Semantic not parsing
anything. Any other C++ IDE will make that decision for you, but guess
what would happen if we simply modified people's auto-mode-alist?

-David

Re: IDE

[David Kastrup]

David Engster <deng@randomsample.de> writes:

> Dmitry Gutov writes:
>>> Well, it's not like a bunch of people are hacking on the C
>>> parser. Several things happened:
>>
>> It's not like I'm blaming anyone, really. But it leaves an impression
>> of CEDET being more of a research project.
>
> It leaves the impression of an understaffed project.

Which is the rule rather than the exception for Free Software projects.
Even when done under corporate veil, the expectation for Free Software
(partly due to the claims of the Open Source movement) is very much that
many people may pitch in.

Most projects are neither prepared to deal with significantly fewer
people pitching in than expected (the more likely case) as they are for
significantly more people pitching in.

The key provision for either possibility is a modular architecture where
people can work on multiple fronts without either depending too much on
each other for their individual progress, nor getting in each other's
hair.

I think that if CEDET fell apart into more independently accessible,
usable, and changeable parts, it might gather more buy-in on its
independent components.  Of course, having a separate repository that
has diverged from the versions in upstream Emacs does not exactly help
with having contributors become active on just their favorite parts of
it.

-- 
David Kastrup

Re: IDE

[John Wiegley]

>>>>> Eli Zaretskii <eliz@gnu.org> writes:

> I wasn't aware that the IDE landscape might have changed in such a
> significant way recently. This discussion seems to focus on details, which
> seems to indicate no such radical changes happened. But I'm not an expert,
> so maybe you are right.
> 
> I did suggest up-thread to come up with a list of main features we think an
> Emacs IDE should have. If that is what you have in mind, I obviously agree.

That's great, Eli. I don't know if the landscape has really changed or not,
only that I'd love to take a step back together and survey the field: if for
no other reason than to help us feel we've reached our conclusions on the same
footing. Who knows, we may end up where we are now; except that given the
current level of disgruntlement, I'd be surprised by that outcome as well.

My list of main features from a previous message (and this is just a draft,
subject to change) is:

  indentation (see above)
  reformatting
  syntax highlighting (font-lock)
  help at point
  documentation lookup (sadly, fewer projects use Info these days)
  completion
  refactoring
  semantic editing (for example, paredit)
  compilation (M-x compile)
  live compilation (flymake/flycheck)
  REPL (comint)
  running tests
  debugging (GUD)
  version control (CV)
  profiling
  code coverage
  app interaction

I'll refine this shortly and come back with a better list, and then we can
start new threads for each sub-area, and discover what expertise we already
have in those areas within the community.

BTW- I used to work at Borland on their C++Builder IDE, and I've worked full
time on Java J2EE projects using IntelliJ, so that is the basis for some of my
opinions about IDEs.

> In any case, CEDET is not an EDE, AFAIK. It is an infrastructure and a set
> of tools for building an IDE. IOW, it's neither a hammer nor a screwdriver,
> but something that allows us to make one or the other (or something else
> entirely). So it could very well be a good basis for an Emacs IDE.

It could be! I'm pretty sure it will come up in discussions again shortly,
with some valuable experiences to add, and perhaps even code.

John

Re: IDE

[Achim Gratz]

David Engster writes:
> It doesn't help that Emacs is a very conservative piece of software. A
> good example was already given: C++ includes without an extension. By
> default, Emacs will open such files in fundamental mode.

If I use /usr/bin/file on such an include, it happily tells me it's been
looking at "C++ source, ASCII text".  So instead of insisting on a known
extension to determine the major mode, Emacs could check what the file
mode is supposed to be in its absence before falling back to fundamental
mode.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Factory and User Sound Singles for Waldorf Blofeld:
http://Synth.Stromeko.net/Downloads.html#WaldorfSounds

Re: IDE

[Lluís]

Dmitry Gutov writes:

> On 10/17/2015 08:18 PM, Lluís wrote:
>> I didn't think of the "find path" functionality, but it does seem to make sense
>> to fold it into the "root path" service type.

> You mean "find search path", right? Another name if it is "include path", and
> that's different from the executable path.

No, I mean something as simple as concatenating the project's root with a
non-absolute path given as an argument. Search or include paths are specific to
the organization of the project at hand (e.g., on a C project not every
directory should be searched for an include), but could be implemented on top of
the "find path" functionality.


> I think I've formed an idea how separating it from the find-root(s) service
> would be useful. Currently, we have this awkward project-search-path-function,
> which can be defined in a major mode (we want this kind of functionality to be
> major mode-providable).

> But if it were a separate service, that would be tidier. Introduce
> project-search-path-functions, which would take the current project as an
> argument and return an object that contains the search path (and, as a bonus,
> maybe knows how to resolve a relative import string).

> Then ruby-mode won't have to define a separate project type. It'll just add to
> project-search-path-functions that checks that there is a Gemfile somewhere
> above the current directory and below the project root, and use the information
> from it. The fact that returned object can also resolve imports is nice, because
> I wasn't sure where to put that behavior.

> In general, though, this might be problematic when a language L will simply read
> its search path from an environment variable, and doesn't require a presence of
> any "bundle" file. Does it provide its search-path to absolutely any project? We
> might need a "list of project languages" accessor on the "root" service
> instances. User-customizable, of course.

That's one of my concerns with the project concept. As you pointed out,
behaviour can change, for example, based on the mode of the file you're working
on. This can be the case for projects that contain code in multiple languages,
so that each can be treated slightly differently.

The question is thus, for example, how do we fit the major-mode variable into
this scheme? The problem becomes multi-dimensional, and I'm not sure how to
specify that. Should some service-types be dependant on the major-mode, such
that a project can have mutiple services for the same type but for different
major-modes?

If that sounds reasonable, then why not also account for the path of the current
file? The same project might have different file search paths for different
files...


>>> On the flip side, if there are no B implementations that can work with the
>>> currently detected R service r1, but there's a certain b2 in service-b-functions
>>> that would work with r2 (currently shadowed by r1), we may be missing out.
>> 
>> Exactly. I started prototyping a few interfaces where all requests go through
>> the public interfaces. A project-type acts similarly to your "service locator"
>> link, and projects are simply a service locator that manage multiple
>> project-type instances grouped by the project.

> You mean different project-types are like entirely different collections of
> services? Why call them project-types at all? I'm lost.

> Isn't a project a collection of services (of different service-types)?

Both are collections of services. I was thinking of the project as a collection
of project-types too, so that you can combine multiple of them.


[...]
>> While prototyping it, I actually decided it makes sense to implement this
>> functionality on the service-type. It should know how to merge results from
>> multiple services, and every service must declare what service-type it provides.

> How about we reframe the description in terms of hooks? service-type is a hook
> variable, services are elements in it. You can call the elements of a hook in a
> sequence until one returns non-nil, or you can call all of them and combine.

I was thinking of service-types as providing multiple methods (hook variables),
one per function it exposes.


>>> Having to declare each combination of service-types that a project can provide
>>> still seems unnecessarily limiting (and tedious). But that's what project-types
>>> would be for, right?
>> 
>> Exactly. My idea was that common project types would be written once declaring
>> all the service types they provide and through what service
>> implementations.

> So, why?

> Take a look at xref-find-regexp. It looks up the current project, takes its
> roots (service 1) and its search-path (service 2). If either of the services can
> be undefined in the current project, what would xref-find-regexp be supposed to
> do?

> And what is the use of several common project types? Could you give an example?

Suppose you have project-type T1 that provides service 1 (S1) but not S2. As you
said, xref-find-regexp will fail.

Now, there can be some best-effort project-type T0 that provides a brain-dead S2
that looks at all sub-directories. If you create your project overlaying T0 and
T1, you can always get some best-effort functionality in case you're not using
any well-known project type.

Note that some other project-type might provide a more intelligent S2 that, for
example, takes information from a compilation database.


>> I'm not sure I follow, but maybe I responded you on my previous paragraphs.

> How would a random command that needs certain info from services x, y and z, use
> this API?

Does the previous paragraph answer it now?


>>> Next, suppose a project instance can tell whether it provides a "build tool"
>>> service. What does a "call build task" command does? Looks for the current
>>> project and gives up if it doesn't provide the required service, or specifically
>>> asks the locator for a project implementation that does provide that service?
>> 
>> I'd say both. My idea is that all project-types provide an auto-detection
>> function; given a path, tell me if it conforms to this project-type.

> In Emacs parlance, a "path" is a list of directories (see exec-path or
> compilation-search-path).

> The call-build-type needs the build-tool service. What project-type does it use,
> and why?

Suppose these three project-types:

* project-type-scons
* project-type-make
* project-type-linux

Each provides only the build-tool service, and you open a file from the linux
kernel. The type project-type-scons does not apply, but the other two do. If we
can assign some priority to them, we will end up with project-type-linux.


>> Then, if we
>> do not know of any project instance for that path, build a new one that contains
>> instances of all project-types that match with it.

> What if we already have a project instance for that directory, but it doesn't
> provide that service?

I'd say there's no way out of this one. What troubles me the most is nesting,
where each sub-directory might need to behave differently (e.g., have files for
different programming languages). Would then each file or directory have a
potentially different project instance? Or should the project abstraction
account for different services on different paths?


>>> [0] http://martinfowler.com/articles/injection.html#ADynamicServiceLocator
>> 
>> Building a project from all matching project-types provides a clean solution to
>> this. But to be honest, I'm not sure if it makes much sense beyond taking a
>> single auto-detected project type plus an optional overlaid project-type that
>> contains user customizations.

> I don't think user customizations should be a project-type. Rather, they can be
> applied via .dir-locals.el, inside each service-type utility function.

I'm still not sure about this one.


>> For example, we could have a fallback "project-type-generic", a makefile-based
>> "project-type-make" and a very specific "project-type-linux". Should we overlay
>> all of them or just take the most "specific" one? (aka linux).

> The one that comes first in the hook. Hook entries from minor modes usually come
> before the hook entries from major modes, and it's user's responsibility not to
> enable several minor modes at once that share the same responsibility.

I'd prefer the system to be able to auto-detect the proper type whenever
possible, instead of relying on the user establishing order for each possible
directory.


Thanks,
  Lluis

-- 
"And it's much the same thing with knowledge, for whenever you learn
something new, the whole world becomes that much richer."
-- The Princess of Pure Reason, as told by Norton Juster in The Phantom
Tollbooth

Re: IDE

[David Kastrup]

Achim Gratz <Stromeko@nexgo.de> writes:

> David Engster writes:
>> It doesn't help that Emacs is a very conservative piece of software. A
>> good example was already given: C++ includes without an extension. By
>> default, Emacs will open such files in fundamental mode.
>
> If I use /usr/bin/file on such an include, it happily tells me it's been
> looking at "C++ source, ASCII text".  So instead of insisting on a known
> extension to determine the major mode, Emacs could check what the file
> mode is supposed to be in its absence before falling back to fundamental
> mode.

Emacs can do that.

magic-mode-alist is a variable defined in ‘files.el’.
Its value is nil

  This variable may be risky if used as a file-local variable.

Documentation:
Alist of buffer beginnings vs. corresponding major mode functions.
Each element looks like (REGEXP . FUNCTION) or (MATCH-FUNCTION . FUNCTION).
After visiting a file, if REGEXP matches the text at the beginning of the
buffer, or calling MATCH-FUNCTION returns non-nil, ‘normal-mode’ will
call FUNCTION rather than allowing ‘auto-mode-alist’ to decide the buffer’s
major mode.

If FUNCTION is nil, then it is not called.  (That is a way of saying
"allow ‘auto-mode-alist’ to decide for these files.")

[back]




-- 
David Kastrup

Re: IDE

[Achim Gratz]

David Kastrup writes:
> Achim Gratz <Stromeko@nexgo.de> writes:
>
>> David Engster writes:
>>> It doesn't help that Emacs is a very conservative piece of software. A
>>> good example was already given: C++ includes without an extension. By
>>> default, Emacs will open such files in fundamental mode.
>>
>> If I use /usr/bin/file on such an include, it happily tells me it's been
>> looking at "C++ source, ASCII text".  So instead of insisting on a known
>> extension to determine the major mode, Emacs could check what the file
>> mode is supposed to be in its absence before falling back to fundamental
>> mode.
>
> Emacs can do that.
>
> magic-mode-alist is a variable defined in ‘files.el’.
> Its value is nil
>
>   This variable may be risky if used as a file-local variable.
>
> Documentation:
> Alist of buffer beginnings vs. corresponding major mode functions.
> Each element looks like (REGEXP . FUNCTION) or (MATCH-FUNCTION . FUNCTION).
> After visiting a file, if REGEXP matches the text at the beginning of the
> buffer, or calling MATCH-FUNCTION returns non-nil, ‘normal-mode’ will
> call FUNCTION rather than allowing ‘auto-mode-alist’ to decide the buffer’s
> major mode.
>
> If FUNCTION is nil, then it is not called.  (That is a way of saying
> "allow ‘auto-mode-alist’ to decide for these files.")
>
> [back]

Well, that does the opposite of what I described: it doesn't check
auto-mode-alist at all when it matches.  I want auto-mode-alist to take
precedence and only if it doesn't know any better than
"fundamental-mode" should it consult some other mechanism.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Wavetables for the Waldorf Blofeld:
http://Synth.Stromeko.net/Downloads.html#BlofeldUserWavetables

Re: IDE

[David Kastrup]

Achim Gratz <Stromeko@nexgo.de> writes:

> David Kastrup writes:
>> Achim Gratz <Stromeko@nexgo.de> writes:
>>
>>> David Engster writes:
>>>> It doesn't help that Emacs is a very conservative piece of software. A
>>>> good example was already given: C++ includes without an extension. By
>>>> default, Emacs will open such files in fundamental mode.
>>>
>>> If I use /usr/bin/file on such an include, it happily tells me it's been
>>> looking at "C++ source, ASCII text".  So instead of insisting on a known
>>> extension to determine the major mode, Emacs could check what the file
>>> mode is supposed to be in its absence before falling back to fundamental
>>> mode.
>>
>> Emacs can do that.
>>
>> magic-mode-alist is a variable defined in ‘files.el’.

[...]

> Well, that does the opposite of what I described: it doesn't check
> auto-mode-alist at all when it matches.  I want auto-mode-alist to take
> precedence and only if it doesn't know any better than
> "fundamental-mode" should it consult some other mechanism.

Ok, how about a different approach using auto-mode-alist?
auto-mode-alist contains several patterns including directories, so one
could match on /include/[a-zA-Z-]+\' or similar.  That's somewhat crude
(and non C++ programmers might protest the results) but at least for a
C++ programmer it seems like a reasonable default setting.  Another
possibility for particular projects would be to use directory variables.

-- 
David Kastrup

Re: IDE

[Dmitry Gutov]

On 10/18/2015 08:12 PM, David Engster wrote:

> http://www.jetbrains.org/intellij/sdk/docs/reference_guide/custom_language_support.html

Thanks a lot, that should be very helpful. I have their Community 
Edition installed and try it out from time to time, but I haven't looked 
into their documentation, and this looks pretty good.

> We don't. Makefiles are way too complex. The best solution IMHO is using
> compilation databases, and EDE upstream has support for them. Of course,
> it's another LLVM thingy, so I'm not sure if I'm allowed to merge it.

I really think we should move CEDET to ELPA. It can allow you to keep 
the users up-to-date more easily, and would make cedet-devel-load.el 
unnecessary. The standard of what we're "allowed" to distribute, is also 
a bit more lax there, in practice.

But if it turns out to be a problem, as long as the compilation database 
is pluggable, you could extract it into a new package and let users 
install it from MELPA, until the policy here gets with the times.

> My point is that people say they want an Emacs to be an "IDE", but at
> the same time they don't want to be restricted in their Emacs usage, and
> rightfully so. A typical IDE like Visual C++ forces you into using their
> project system, otherwise nothing will work. So people want
> IDE-features, but at the same time they want all the freedom Emacs
> allows. And that's what CEDET tries to support.

Yes. The "project system" I'd expect from Emacs is pretty different. A 
lot of users have lived with a "lightweight" one (Projectile) for years, 
so I try to approach the subject in project.el also from that direction.

> Eric started with EDE as a project system similar to other IDEs, that
> means you can start a project from scratch with EDE and it will manage
> the build for you by generating a Makefile. While technically
> impressive, I never liked this kind of project system, and honestly, I
> think we should probably drop that part from EDE.

I'd say drop it, but you might have users depending on it.

> But you are not forced
> to use it, because Eric also added a custom project detection that is
> very flexible. But if you manage the build yourself, you have to specify
> everything manually (include paths, used compiler, etc.).

But you can also create a specialized project implementation that knows 
how to read e.g. include paths from a project file, right?

> It doesn't help that Emacs is a very conservative piece of software. A
> good example was already given: C++ includes without an extension. By
> default, Emacs will open such files in fundamental mode. The GNU libc
> actually has local variables comments in their includes just for that
> reason! But of course, Qt does not have those, so Emacs cannot parse
> those files by default, and people complain about Semantic not parsing
> anything. Any other C++ IDE will make that decision for you, but guess
> what would happen if we simply modified people's auto-mode-alist?

I always thought that magic-mode-alist takes action only after no 
matches in auto-mode-alist have been found. Maybe that's a worthwhile 
change to make.

Or add a new, third variable, meta-magic-mode-alist, that would take 
action after auto-mode-alist, when the fundamental mode is picked.

Re: IDE

[Dmitry Gutov]

On 10/18/2015 09:21 PM, Lluís wrote:

> No, I mean something as simple as concatenating the project's root with a
> non-absolute path given as an argument. Search or include paths are specific to
> the organization of the project at hand (e.g., on a C project not every
> directory should be searched for an include), but could be implemented on top of
> the "find path" functionality.

Then, UUIC, maybe that would be called "find relative path". Similar to 
https://atom.io/docs/api/v1.0.19/Project#instance-relativizePath

> That's one of my concerns with the project concept. As you pointed out,
> behaviour can change, for example, based on the mode of the file you're working
> on. This can be the case for projects that contain code in multiple languages,
> so that each can be treated slightly differently.

We can't rely on the current file being the deciding factor, however. At 
least some users would expect that if we're in a given project (say, in 
an open Dired buffer, or in a YAML config, or in a Compilation buffer), 
and the project contains Ruby files, the search for references to `each' 
would still use the Ruby search path.

> The question is thus, for example, how do we fit the major-mode variable into
> this scheme? The problem becomes multi-dimensional, and I'm not sure how to
> specify that. Should some service-types be dependant on the major-mode, such
> that a project can have mutiple services for the same type but for different
> major-modes?

The language can be passed as a second argument to 
project-search-path-functions elements. But then the project would need 
to know the set of languages to pass in.

As long as we prohibit search-path from depending on the current buffer, 
any parameters need to passed in explicitly (except one, see below).

> If that sounds reasonable, then why not also account for the path of the current
> file? The same project might have different file search paths for different
> files...

Yes. Add the "directory" parameter (or pass it implicitly by binding 
`default-directory')? The list is getting longer...

> I was thinking of service-types as providing multiple methods (hook variables),
> one per function it exposes.

Nothing can contain multiple hook variables. Hook variables are global. 
The values returned from them, however, could define several methods.

> Now, there can be some best-effort project-type T0 that provides a brain-dead S2
> that looks at all sub-directories. If you create your project overlaying T0 and
> T1, you can always get some best-effort functionality in case you're not using
> any well-known project type.

At which point does a project become "overlaying" T0 and T1? Would that 
be a tricky configuration a user would have to perform in a file inside 
the root directory of each of their projects?

> Note that some other project-type might provide a more intelligent S2 that, for
> example, takes information from a compilation database.

Right.

>> How would a random command that needs certain info from services x, y and z, use
>> this API?
>
> Does the previous paragraph answer it now?

Not really. I "create" a project? Could you rephrase that in terms of 
the project.el API?

Here's an example: the command calls `project-current', which returns 
the current detected value of project. Then it passes it to 
`project-current-search-path', which consults 
project-search-path-functions and returns an object describing the 
search-path, which the command uses with another method.

Something will "create" a project (the user, somehow, or a minor mode), 
but that's of no interest to the API.

> What troubles me the most is nesting,
> where each sub-directory might need to behave differently (e.g., have files for
> different programming languages).

I worry more about directories with files for different programming 
languages all inside them. At some point, some code will either have to 
prompt the user, guess somehow, or ask for search-path for all the 
languages, and then combine the results.

> Would then each file or directory have a
> potentially different project instance? Or should the project abstraction
> account for different services on different paths?

Maybe we should decide that search-path must be the same for all 
directories inside a project. Otherwise, we'd get different results for 
commands called from different directories, sometimes with no apparent 
reason.

Still unsure about it.

>> The one that comes first in the hook. Hook entries from minor modes usually come
>> before the hook entries from major modes, and it's user's responsibility not to
>> enable several minor modes at once that share the same responsibility.
>
> I'd prefer the system to be able to auto-detect the proper type whenever
> possible, instead of relying on the user establishing order for each possible
> directory.

How would having numeric priorities help? They'd still be static and 
apply to all directories.

Also note that for project types supplied for Emacs, we'd be able to put 
them into hooks in the proper order. With third-party code, however, the 
order might be wrong sometimes, but we also have no guarantee that their 
authors would assign correct priorities.

Re: IDE

[David Engster]

Eric Ludlam writes:
> On 10/16/2015 07:17 AM, Przemysław Wojnowski wrote:
>>>> BTW Why EDE (and CEDET) are in two different repositories (and
>>>> projects?).
>>>> One is in Emacs sources, another here
>
>>>> http://sourceforge.net/projects/cedet/ ?
>>>
>>> CEDET is in Emacs sources as well (although is has some
>>> omissions). See lisp/cedet.
>> Yes, so when one would like change EDE to, lets say, apply the
>> Bridge pattern,
>> which one should be used?
>> . 
>
> Good question.  The recent EIEIO changes in Emacs made the two
> repositories really hard to merge, and EDE is steeped in EIEIO.
>
> David Engster has been doing the merges and might have a good
> idea. I'm just not that good with git. ;(

At the moment it is best to work in the Emacs repository.

-David

Re: IDE

[Eric Ludlam]

On 10/17/2015 10:25 AM, Dmitry Gutov wrote:
> On 10/17/2015 03:45 PM, Eric Ludlam wrote:
>
>> There is nothing about semantic the framework that prevents using
>> something like SMIE as a tagging engine.  It seems likely that if some
>> language not supported by CEDET today has a good SMIE grammar, and if
>> SMIE could be adapted to produce compatible tags (which isn't really
>> that hard to do), and if SMIE has an entry point that allows parsing a
>> tag set out of a region, then it would integrate into the semantic
>> framework at the best level.
>
> With SMIE, though, it might make sense to optimize the process and 
> only ask it for the current tag (and maybe the nearby ones), because 
> it would otherwise be slower (no incremental parsing, or anything like 
> that). So a straightforward conversion might not be optimal.
>
> But from the main examples you gave of tag usage, knowing the current 
> tag is by far the most important.

There are many reasons why having all the tags in the current buffer is 
useful.  The simplest of which are things like imenu - the menu part, 
speedbar, or ECB listings of all tags so you can jump to them.  There is 
a command for jumping to a tag in a buffer by name with completion.  
There's an isearch flavor that searched only tag names in the current 
buffer.

Of course, the most important is that if you have all the detailed tags 
from a file that is closed, you don't need to open the file again if you 
need to reference it.

>> The mechanism in use is that there are detailed taggers, like those
>> currently written for Semantic using wisent type parsers, and weak
>> taggers, like GNU Global.
>>
>> Once a buffer is open, the detailed tagger runs and is truth. When a
>> search occurs, a process of pulling all relevant databases together is
>> started.  This includes tags from the detailed taggers already cached,
>> and high level 'omniscient' taggers that are usually weak, but have
>> scanned the whole project.
>
> Well, that's my point: if you don't use etags, to have all project 
> files indexed you have to open each of them at some point.
>

Sure - something does have to open them.  If not emacs, then something will.

> Some languages have import systems where you have to declare all 
> file's dependencies at its top (and so you can know which files to 
> reindex if you're only interested in this file; although even that 
> could be a lot). But in Ruby and pre-ES6 JavaScript, you don't usually 
> have that.
>

Using imports and includes, the semantic framework tracks what the 
dependencies are, and will reparse files that change outside of Emacs.  
It also will only open files listed as dependencies.

It will also, in idle time, pull in things it suspects you might want to 
open later, but that is to make visiting those things faster.  It is 
easy to turn off, and we could turn it off by default if it is causing 
problems.

>
>> The key here is that the detailed tagger is needed to do complex tasks,
>> but weak taggers are great to integrate in due to the speed advantage.
>> Having a mix gives you the best possible results.
>
> If the program generating TAGS is smart enough, the result can be 
> pretty accurate already (and contain qualified method name). Eli has 
> recently made some improvements to that effect to etags.
>

If it can produce the needed data, then it would be simple enough to 
swap it in as an external parser.  Since Emacs has full control over 
etags, it could be adapted to handle regions, and thus fit in just fine.

>> If instead you put the cursor on "tee" and do
>>
>> M-x semantic-ia-fast-jump RET
>>
>> it will go to the right spot.
>
> Indeed it does, thanks. But it goes to the second definition 
> irrespective of which argument is passed to tee. Which brings us back 
> to the completion problem I reported with that snippet.
>

You are right, I missed that.  I've started a new test suite around this 
missing feature and will look into it.

>> For the keybinding you were using, the workflow is:
>
> Right, why doesn't semantic-ia-fast-jump have a default keybinding? I 
> was only looking for the appropriate command in the active keymaps.
>

I've run out of steam trying to design the perfect set of keybindings.  
Everyone seems to like something different.  ;)

>> C-c , J tee TAB
>>
>> shows you where it would jump, then
>>
>> TAB
>
> Oh, so it uses something more advanced than completing-read. That's 
> unexpected.
>

Yes.  I wrote my own so that the completion engine could weed out the 
maximum number of possible completions and speed up the process.  Some 
of the tricks others have done that do take all possible completions in 
their UIs I think have obsoleted that advantage.

But also, I was looking for ways to allow selection of different methods 
of the same name that was easy and quick, and this is what I came up with.


Eric

Re: IDE

[Lluís]

Dmitry Gutov writes:

> On 10/18/2015 09:21 PM, Lluís wrote:
[...]
>> If that sounds reasonable, then why not also account for the path of the current
>> file? The same project might have different file search paths for different
>> files...

> Yes. Add the "directory" parameter (or pass it implicitly by binding
> `default-directory')? The list is getting longer...

That's the point I wanted to raise. How flexible should the underlying
infrastructure be? And based on what should it be flexible?


>> I was thinking of service-types as providing multiple methods (hook variables),
>> one per function it exposes.

> Nothing can contain multiple hook variables. Hook variables are global. The
> values returned from them, however, could define several methods.

I think I did not explain myself well. I meant having one hook variable for each
public method of each service type.


>> Now, there can be some best-effort project-type T0 that provides a brain-dead S2
>> that looks at all sub-directories. If you create your project overlaying T0 and
>> T1, you can always get some best-effort functionality in case you're not using
>> any well-known project type.

> At which point does a project become "overlaying" T0 and T1? Would that be a
> tricky configuration a user would have to perform in a file inside the root
> directory of each of their projects?

If every project-type T* has a probing function that tells you if it's
applicable to your current project, then you simply overlay all project types
that match (or support) your project.

IMHO, how to expose this to the user is a question that needs answering later


[...]
>>> How would a random command that needs certain info from services x, y and z, use
>>> this API?
>> 
>> Does the previous paragraph answer it now?

> Not really. I "create" a project? Could you rephrase that in terms of the
> project.el API?

> Here's an example: the command calls `project-current', which returns the
> current detected value of project. Then it passes it to
> `project-current-search-path', which consults project-search-path-functions and
> returns an object describing the search-path, which the command uses with
> another method.

> Something will "create" a project (the user, somehow, or a minor mode), but
> that's of no interest to the API.

Ok, so the first time you call `project-current', the system can check all
project-types it knows about and see which ones apply to your file. From that a
project object object is created overlaying the matched project-types. Next time
you request `project-current' a cached value can be returned, or maybe looking
at the value of some parent directory.


>> What troubles me the most is nesting,
>> where each sub-directory might need to behave differently (e.g., have files for
>> different programming languages).

> I worry more about directories with files for different programming languages
> all inside them. At some point, some code will either have to prompt the user,
> guess somehow, or ask for search-path for all the languages, and then combine
> the results.

>> Would then each file or directory have a
>> potentially different project instance? Or should the project abstraction
>> account for different services on different paths?

> Maybe we should decide that search-path must be the same for all directories
> inside a project. Otherwise, we'd get different results for commands called from
> different directories, sometimes with no apparent reason.

> Still unsure about it.

Ok, so it seems that we have two kinds of operations:

* global: an operation that behaves the same regardless of where you're
  performing the request from

* local: an operation that can behave differently depending on the path you're
  performing the request from, depending on the major-mode, depending on some
  environment variable, or depending on some other arbitrary condition

Would it ever make sense for a service method to provide both kinds of
operations? (i.e., discriminate through an argument)


>>> The one that comes first in the hook. Hook entries from minor modes usually come
>>> before the hook entries from major modes, and it's user's responsibility not to
>>> enable several minor modes at once that share the same responsibility.
>> 
>> I'd prefer the system to be able to auto-detect the proper type whenever
>> possible, instead of relying on the user establishing order for each possible
>> directory.

> How would having numeric priorities help? They'd still be static and apply to
> all directories.

> Also note that for project types supplied for Emacs, we'd be able to put them
> into hooks in the proper order. With third-party code, however, the order might
> be wrong sometimes, but we also have no guarantee that their authors would
> assign correct priorities.

That's true, the user always needs some "backdoor" to customize behavior, since
there's no global behavior that can fit all cases.


Cheers,
  Lluis

-- 
"And it's much the same thing with knowledge, for whenever you learn
something new, the whole world becomes that much richer."
-- The Princess of Pure Reason, as told by Norton Juster in The Phantom
Tollbooth

Re: IDE

[Dmitry Gutov]

On 10/19/2015 04:04 PM, Lluís wrote:

> That's the point I wanted to raise. How flexible should the underlying
> infrastructure be? And based on what should it be flexible?

Well, yeah. These are the practical questions, decisions on which will 
shape the result.

Having limitations isn't bad, as long as the user will still be able to 
enjoy useful behavior, one way or another.

Option one: we allow search-path to depend on default-directory (in 
addition to the current project). Then the user can employ VC as a 
backend for projects, and yet have several different applications inside 
that repository. I'm describing this on the basis of one of the projects 
we have at work.

Option two: we disallow search-path depending on default-directory. 
Then, for the same project, VC can't be the backend, and the "project" 
will have to be split into several (VC can't be used as a backend 
heere). Which will make at least as much sense. It will require a 
special project type for Ruby, based on the presence of Gemfile.

> I think I did not explain myself well. I meant having one hook variable for each
> public method of each service type.

Why aggregate them into service types, then? Just call each "public 
method" (or hook) a separate service.

> If every project-type T* has a probing function that tells you if it's
> applicable to your current project, then you simply overlay all project types
> that match (or support) your project.
>
> IMHO, how to expose this to the user is a question that needs answering later

The "overlaying" process would also need some explanation. And 
consideration.

Suppose there are several implementations of the same service-type, and 
the results they return (like search-path elements) overlap. How would 
you produce the resulting list? (delete-dups (apply #'append ...))?

That would assume that there can't be "bad" (less precise) elements. For 
the "find references" service (if we had one), you would want to only 
keep the precise results, if available, a not mash them together with 
the output of Grep.

> Ok, so the first time you call `project-current', the system can check all
> project-types it knows about and see which ones apply to your file. From that a
> project object object is created overlaying the matched project-types. Next time
> you request `project-current' a cached value can be returned, or maybe looking
> at the value of some parent directory.

Caching is incidental. And I still don't know what "overlaying" is. In 
solid details, I mean.

> Ok, so it seems that we have two kinds of operations:
>
> * global: an operation that behaves the same regardless of where you're
>    performing the request from
>
> * local: an operation that can behave differently depending on the path you're
>    performing the request from, depending on the major-mode, depending on some
>    environment variable, or depending on some other arbitrary condition
>
> Would it ever make sense for a service method to provide both kinds of
> operations? (i.e., discriminate through an argument)

It might. The "find included file" operation would probably want to 
consider the type of the current file.

The environment variable can be looked up from the service itself. 
Arbitrary conditions are fine, as long they're not dependent on the 
current buffer too much.

> That's true, the user always needs some "backdoor" to customize behavior, since
> there's no global behavior that can fit all cases.

The user would always be able to rearrange the elements in a hook, or 
add their own function (service implementation), maybe composing 
existing ones.

Re: IDE

[Dmitry Gutov]

On 10/19/2015 02:51 PM, Eric Ludlam wrote:

> There are many reasons why having all the tags in the current buffer is
> useful.  The simplest of which are things like imenu - the menu part,
> speedbar, or ECB listings of all tags so you can jump to them.  There is
> a command for jumping to a tag in a buffer by name with completion.
> There's an isearch flavor that searched only tag names in the current
> buffer.

All right. But I would say that IMenu was conceived as a poor cousin to 
tagging. After all, if we have up-to-date information about tags in the 
current project, we can jump anywhere, not just in the current file.

My point is, we could have a more limited "protocol" to be used when we 
don't have a list of tags for the current file (and "jump to xxx" 
command is implemented via overrides, using an external tool). And a 
"full" protocol for when the tags are available.

> Using imports and includes, the semantic framework tracks what the
> dependencies are, and will reparse files that change outside of Emacs.
> It also will only open files listed as dependencies.

Right. It's quite nifty. I was just pointing out that it won't work for 
some languages.

> If it can produce the needed data, then it would be simple enough to
> swap it in as an external parser.  Since Emacs has full control over
> etags, it could be adapted to handle regions, and thus fit in just fine.

I suppose so. ctags (not in the Emacs repo) supports more file formats, 
though.

> You are right, I missed that.  I've started a new test suite around this
> missing feature and will look into it.

Thank you.

> I've run out of steam trying to design the perfect set of keybindings.
> Everyone seems to like something different.  ;)

I understand the difficulty, but surely you'd want to advertise 
semantic-ia-fast-jump? It's more precise, and thus, more powerful.

> But also, I was looking for ways to allow selection of different methods
> of the same name that was easy and quick, and this is what I came up with.

We could use some mix of the current xref behavior, with this one, for 
xref-find-definitions. xref is static, but shows everything. This one is 
dynamic, but shows only one option at a time.

Re: IDE

[Eric Ludlam]

On 10/17/2015 09:21 AM, Dmitry Gutov wrote:
> And speaking of more personal reasons, I already have written a code
> assistance tool for Ruby, in Ruby (which is not a Lisp, but still a
> pretty okay language), and smooth integration with whatever APIs we
> choose would be nice.

We should poke at this.  Can you share a little about what your tool 
does?  Then perhaps we hypothesize about the efficacy of integrating 
into CEDET as an example of integrating external tools.

Eric

Re: IDE

[Eric Ludlam]

On 10/18/2015 05:42 AM, Dmitry Gutov wrote:
>>> Some refactorings, too.
>> And some are not and will never be, because languages are too
>> different. So
>> IDE's support for them will be different or crippled to the lowest
>> denominator as you seem to suggest.
>
> You're just arguing for the sake of arguing here. We could have a common
> UI for implementing refactoring commands, and some of those commands
> could be different for different languages. Or something along these lines.

Here is an example of a refactoring toolset that Tu Do started using 
CEDET/Semantic as a starting point.

https://github.com/tuhdo/semantic-refactor

Eric

Re: IDE

[Eric Ludlam]

On 10/18/2015 01:28 PM, David Kastrup wrote:
> I think that if CEDET fell apart into more independently accessible,
> usable, and changeable parts, it might gather more buy-in on its
> independent components.  Of course, having a separate repository that
> has diverged from the versions in upstream Emacs does not exactly help
> with having contributors become active on just their favorite parts of
> it.

There are many examples where different developers with none to minimal 
help have successfully:

* Added new language parsers
* created new EDE project types and detection schemes
* Added support for new code generation steps.
* built whole new language independent tools using APIs from semantic

The core parts of CEDET is all about allowing extension by having well 
defined interfaces, and clear places where your custom thing can diverge 
and grow.

Many of the contributed pieces are in the upstream "contrib" area when 
they couldn't not provide assignment, and some are part of CEDET when we 
do have assignments.

I think it is instead more fun and easier to go off and whip up your own 
thing wrapping up some external tool than it is to become part of 
something else.

Also, I'm not much of a verbose advocate, which is probably what is 
needed here.

Eric

Re: IDE

[Dmitry Gutov]

On 10/20/2015 04:03 AM, Eric Ludlam wrote:

> We should poke at this.  Can you share a little about what your tool
> does?  Then perhaps we hypothesize about the efficacy of integrating
> into CEDET as an example of integrating external tools.

It probably wouldn't gain too much from CEDET integration.

It has:

- Server, serving JSON over HTTP, with a RPC-like protocol.

- Emacs client in the shape of a minor mode. It defines a 
completion-at-point-functions element and a keymap with essentially 
three commands: "jump to the definition", "jump back", "show 
documentation for the method at point".

- To determine the current context (which would be similar to "current 
tag" in Semantic), it calls ruby-add-log-current-method and parses the 
returned string. I've improved that function in ruby-mode a year or two 
ago, and it's pretty accurate.

Robe is also pretty hacky compared to most other similar tools:

- The server loads itself into and runs from an REPL buffer. We either 
expect the user to already have such a REPL running (with the project 
loaded), or offer to launch one automatically (which fails in certain 
configurations).

- It doesn't support multiple projects in the same Emacs session.

- It misses some trivial opportunities to infer the type of a local 
variable. That would be my first priority to work on... when I deal with 
all that project and xref stuff in the core, I guess.

Lives here: https://github.com/dgutov/robe/

Re: IDE

[Lluís]

Dmitry Gutov writes:

> On 10/19/2015 04:04 PM, Lluís wrote:
[...]
>> I think I did not explain myself well. I meant having one hook variable for each
>> public method of each service type.

> Why aggregate them into service types, then? Just call each "public method" (or
> hook) a separate service.

Just a matter of keeping tightly-related functionality together.


[...]
>> Ok, so the first time you call `project-current', the system can check all
>> project-types it knows about and see which ones apply to your file. From that a
>> project object object is created overlaying the matched project-types. Next time
>> you request `project-current' a cached value can be returned, or maybe looking
>> at the value of some parent directory.

> Caching is incidental. And I still don't know what "overlaying" is. In solid
> details, I mean.

Ok, a more specific example. Assume the service-type "service-type-symbols" that
provides a method to jump to symbols, and we're editing sources of the linux
kernel. Now, we have the following service implementations for that
service-type:

* service-symbols-cscope: Use external database.
* service-symbols-wisent: Parse source files, requires an implementation of
  "service-type-search-paths".

And we also have the following project-types:

* project-type-generic:
  * match: Return t if we can detect a project root using VC
  * service-types:
    * service-type-root: service-root-vc

* project-type-generic-c:
  * match: It's unclear when to activate it
  * service-types:
    * service-type-symbols: service-symbols-cscope, service-symbols-wisent

* project-type-linux:
  * match: We see a root directory for a Linux project
  * service-types:
    * service-type-root: service-root-linux
    * service-type-search-paths: service-search-paths-linux

If we assume all project types match, the resulting "overlaid" project for Linux
would have the following service-types:

* service-type-root: service-root-linux
* service-type-search-paths: service-search-paths-linux
* service-type-symbols: service-symbols-cscope, service-symbols-wisent

If we were not editing some Linux directory:

* service-type-root: service-root-vc
* service-type-symbols: service-symbols-cscope


Cheers,
  Lluis

-- 
"And it's much the same thing with knowledge, for whenever you learn
something new, the whole world becomes that much richer."
-- The Princess of Pure Reason, as told by Norton Juster in The Phantom
Tollbooth

Re: IDE

[Steinar Bang]

>>>>> Dmitry Gutov <dgutov@yandex.ru>:

> Option one: we allow search-path to depend on default-directory (in
> addition to the current project). Then the user can employ VC as a
> backend for projects, and yet have several different applications
> inside that repository. I'm describing this on the basis of one of the
> projects we have at work.

> Option two: we disallow search-path depending on
> default-directory. Then, for the same project, VC can't be the
> backend, and the "project" will have to be split into several (VC
> can't be used as a backend heere). Which will make at least as much
> sense. It will require a special project type for Ruby, based on the
> presence of Gemfile.

With maven, I might do something like this:
 project/
   pom.xml
   .git/
   module1/
     pom.xml
     .git/
   module2/
     pom.xml
     .git/

The same projects in an eclipse workspace, might look something like
this:
 workspace/
   module1/
     pom.xml
     .git/
   module2/
     pom.xml
     .git/

Or perhaps something like this (if the parent also has common settings I
would like easily editable).
 workspace/
   project/
     pom.xml
     .git/
   module1/
     pom.xml
     .git/
   module2/
     pom.xml
     .git/

If the project was to be only edited with emacs, I would go for the top
layout, however if I was to edit the same projects in both emacs and
eclipse, it should handle the latter two layouts as well (though the
bottom one doesn't work too well with command line maven).

Re: IDE

[Dmitry Gutov]

On 10/20/2015 04:07 AM, Eric Ludlam wrote:

> Here is an example of a refactoring toolset that Tu Do started using
> CEDET/Semantic as a starting point.
>
> https://github.com/tuhdo/semantic-refactor

It's promising, but if we're talking about a UI, I would expect 
something like a preview of the operation. That would be most useful for 
things like rename.

That's not there yet, but it does support "rename local variable" and 
"extract method", so anyone sorely missing refactoring in Emacs (in C or 
C++), should check it out. AFAICS, it's definitely not getting enough 
feedback.

Re: IDE

[Dmitry Gutov]

On 10/20/2015 03:37 PM, Lluís wrote:

> And we also have the following project-types:
>
> * project-type-generic:
>    * match: Return t if we can detect a project root using VC
>    * service-types:
>      * service-type-root: service-root-vc

I see. This looks more like a classical structure, with one "project" 
objects, two attributes in there, and the second one containing a map 
with list values.

We could indeed interpret those values as hooks, but that question is 
less important.

The main benefit of this structure is familiarity to anyone who's done a 
little OOP.

What I was imagining, since we're talking about services:

project-types:
   - project-type-generic:
     * match
     * root
   - project-type-generic-c:
     * match
     * root

search-paths-types:
   - search-paths-linux (project)
     * search-paths
     * resolve-include-path <-- maybe
   - search-paths-c++ (project)
     * likewise
     * likewise

symbols-list-types:
   - symbols-list-cscope (project, search-paths)
   - symbols-list-wisent (project, search-paths)

The upside is that I can write a new package called magic-8-ball, which 
would add an element at the beginning of symbols-list-types, and that 
element will have a chance to be used in every kind of project. Now just 
the projects that were explicitly written with magic-8-ball in mind. 
That adds flexibility.

The downside is that indirection adds complexity as well, and could turn 
out to be mostly unused.

Re: IDE

[Dmitry Gutov]

Hi Steinar,

On 10/20/2015 06:23 PM, Steinar Bang wrote:
> With maven, I might do something like this: (A)
>   project/
>     pom.xml
>     .git/
>     module1/
>       pom.xml
>       .git/
>     module2/
>       pom.xml
>       .git/
>
> The same projects in an eclipse workspace, might look something like
> this: (B)
>   workspace/
>     module1/
>       pom.xml
>       .git/
>     module2/
>       pom.xml
>       .git/
>
> Or perhaps something like this (if the parent also has common settings I
> would like easily editable): (C)
>   workspace/
>     project/
>       pom.xml
>       .git/
>     module1/
>       pom.xml
>       .git/
>     module2/
>       pom.xml
>       .git/
>
> If the project was to be only edited with emacs, I would go for the top
> layout, however if I was to edit the same projects in both emacs and
> eclipse, it should handle the latter two layouts as well (though the
> bottom one doesn't work too well with command line maven).

That doesn't help, by itself. Surely we want to support all of these 
directory structures. The question is what each of them would translate 
to in the project API.

Consider A. It could be considered one project, but then it would have 
certain attributes dependent on the current directory. Take classpath, 
for example. Could we consider it to be constant value across the whole 
project, or would we have, for certain operations (like "find the class 
named foo.bar.Bar"), different values of classpath in module1 and 
module2? The build-tool behavior would certainly depend on the current 
directory, but could we say that all other important project attributes 
are kind of the same for project, module1 and module2?

Another option for A is to promote module1 and module2 to whole 
projects. But then, do we track project dependencies now? If I call a 
command "search for occurrences of 'xyz' in project", does it also 
search in module1 and module2? Or similarly for "list all files in 
project", does that include files in module1? That would be useful with 
Helm, for "jump to a project file". Does "list all files in project", 
when called in module1, include files from the parent project, and from 
module2?

B doesn't look too different, except we apparently don't have a 
top-level pom-file.

I don't understand C. Is module1 still inside project? Is it still a 
dependency? Do we treat it differently WRT to questions I've asked for 
the option A?

Re: IDE

[Eric Ludlam]

On 10/19/2015 08:56 PM, Dmitry Gutov wrote:
> On 10/19/2015 02:51 PM, Eric Ludlam wrote:
>
>> There are many reasons why having all the tags in the current buffer is
>> useful.  The simplest of which are things like imenu - the menu part,
>> speedbar, or ECB listings of all tags so you can jump to them. There is
>> a command for jumping to a tag in a buffer by name with completion.
>> There's an isearch flavor that searched only tag names in the current
>> buffer.
>
> All right. But I would say that IMenu was conceived as a poor cousin 
> to tagging. After all, if we have up-to-date information about tags in 
> the current project, we can jump anywhere, not just in the current file.
>
It was an example, and there are plenty of cases where you want to 
restrict your destination to be local simply because completing on a 
zillion options is slow, regardless of the tool used to create said list.

> My point is, we could have a more limited "protocol" to be used when 
> we don't have a list of tags for the current file (and "jump to xxx" 
> command is implemented via overrides, using an external tool). And a 
> "full" protocol for when the tags are available.
>

That might work in a simple case, but for current tag to truly be 
accurate, you need to parse the whole buffer from start to point so you 
don't get confused by text in strings, text in comments, nested function 
syntax, and #ifdef type syntax.  Once you do that, you might as well 
just parse the whole buffer and cache it.

>
>> I've run out of steam trying to design the perfect set of keybindings.
>> Everyone seems to like something different.  ;)
>
> I understand the difficulty, but surely you'd want to advertise 
> semantic-ia-fast-jump? It's more precise, and thus, more powerful.
>

Yes, there are some guidelines on what functions there are but I haven't 
tied much together into minor modes or keybindings.  We used to have 
'senator' mode which we updated with new stuff, but the Emacs merge has 
gotten that a bit confused.

As an added bonus, all the semantic-ia-* functions were originally meant 
to be examples on how to use the infrastructure for someone who wanted 
build interface functions, so they lack a bit of polish and haven't been 
promoted into keymaps.

It would be great if someone interested in the user interface side could 
help out getting the right interface together on top of some of the 
semantic concepts.

Eric

Re: IDE

[Eric Ludlam]

On 10/20/2015 07:28 AM, Dmitry Gutov wrote:
> On 10/20/2015 04:03 AM, Eric Ludlam wrote:
>
>> We should poke at this.  Can you share a little about what your tool
>> does?  Then perhaps we hypothesize about the efficacy of integrating
>> into CEDET as an example of integrating external tools.
>
> It probably wouldn't gain too much from CEDET integration.

Ruby support tooling itself would not benefit from CEDET integration, 
but tools built on CEDET would gain Ruby support, and that improves the 
diversity of Ruby related tools available.

> It has:
>
> - Server, serving JSON over HTTP, with a RPC-like protocol.
>
> - Emacs client in the shape of a minor mode. It defines a 
> completion-at-point-functions element and a keymap with essentially 
> three commands: "jump to the definition", "jump back", "show 
> documentation for the method at point".
>

Inside this feature you must have a way to query for the location of a 
particular symbol, and convert a symbol into a doc reference.

In CEDET speak, the tooling that converts a symbol into a location would 
be a special kind of Semantic Database.  This could be tied to a project 
or tied in as a system database depending on what it is querying.  Or 
one of each.  The job of such databases is to provide a search for tag 
by name, completion substring, regexp, and for languages that care, type.

The output is a list of matches as faux tags.  If an application wanted 
to know more about the symbol, it would pull in the reference file, and 
extract real tag data using whatever parser is available.

This would enable Semantic's jump to tag system to be as accurate as yours.

I am not sure what your doc piece might be like.  There is some limited 
support for finding doc strings, but usually it just looks for comments 
preceding a tag.

> - To determine the current context (which would be similar to "current 
> tag" in Semantic), it calls ruby-add-log-current-method and parses the 
> returned string. I've improved that function in ruby-mode a year or 
> two ago, and it's pretty accurate.

We used to have a way to tweak add-log but I think the mechanism is for 
an older version of Emacs where we needed to use advice.  It would make 
sense to update this if there is a better way.

> Robe is also pretty hacky compared to most other similar tools:
>
> - The server loads itself into and runs from an REPL buffer. We either 
> expect the user to already have such a REPL running (with the project 
> loaded), or offer to launch one automatically (which fails in certain 
> configurations).
>
> - It doesn't support multiple projects in the same Emacs session.
>

An EDE type project for ruby (whatever that looks like) would provide a 
place to hang project specific REPL buffers as needed.

> - It misses some trivial opportunities to infer the type of a local 
> variable. That would be my first priority to work on... when I deal 
> with all that project and xref stuff in the core, I guess.

I'm not sure which code bit you are referencing here.  If you do your 
tag parsing with a semantic grammar, then you can optionally use that 
same grammar to parse function bodies, and thus make detecting local 
variable types a bit easier.   I'm speculating however as I am not 
familiar with Ruby.

There is a wisent based grammar for Ruby in the 'contrib' area that 
seems straightforward.  It would probably not be much of a stretch to 
build one with the right releases to include in Emacs, solve this one 
problem, and then get support from other CEDET tools.

Eric

Re: IDE

[Dmitry Gutov]

On 10/21/2015 06:13 AM, Eric Ludlam wrote:

> Ruby support tooling itself would not benefit from CEDET integration,
> but tools built on CEDET would gain Ruby support, and that improves the
> diversity of Ruby related tools available.

If CEDET support won't improve Robe, and if the said support will amount 
to writing a Wisent grammar, it just becomes a separate task.

Which I don't object to doing, in principle, but it'll go to the bottom 
of the pile, in terms of priority.

> Inside this feature you must have a way to query for the location of a
> particular symbol, and convert a symbol into a doc reference.

In both cases, I often have to prompt the user (with completing-read) to 
disambiguate between classes that define the method with the given name.

> The output is a list of matches as faux tags.  If an application wanted
> to know more about the symbol, it would pull in the reference file, and
> extract real tag data using whatever parser is available.

So as faux tags, I could return all methods with the same name from 
different classes?

> This would enable Semantic's jump to tag system to be as accurate as yours.

Would that be semantic-complete-jump or semantic-ia-fast-jump?

> I am not sure what your doc piece might be like.  There is some limited
> support for finding doc strings, but usually it just looks for comments
> preceding a tag.

It's a struct, containing both the arguments list, and a doc string.

I display it specially, though: do a little lightweight parsing, and 
highlight the Ruby examples parts with ruby-mode font-lock rules, and C 
sources with c-mode. Plus highlighted function signature at the top and 
a button to go to the source.

> We used to have a way to tweak add-log but I think the mechanism is for
> an older version of Emacs where we needed to use advice.  It would make
> sense to update this if there is a better way.

ruby-add-log-current-method is the value of 
add-log-current-defun-function, but I don't know if you'd be able to use 
it for different languages: the format doesn't seem to be particularly 
standardized.

> An EDE type project for ruby (whatever that looks like) would provide a
> place to hang project specific REPL buffers as needed.

How? Using which major mode? I current use inf-ruby for that (not 
available in Emacs, for copyright assignment reasons). So it seems I'd 
have to add multi-REPL support for it first.

The conceptually hard question is what do you do with external files 
that can be referenced from different projects? Suppose you have two 
open projects, each with its own REPL, you made a jump to an external 
file from project1. The you want to "jump to definition" on some method 
call in there. How do you pick the right REPL to ask for info?

Suppose, after the first jump, you saved the reference to the right 
project in a buffer-local variable, so you can refer to it for the 
second jump. What if I want to do the next jump not from the same exact 
file, but from its neighbor? As a user, I can be confident that both 
files must be referenced by the project, but there will be no 
buffer-local value to use.

Finding an idiomatic approach to that would be great.

>> - It misses some trivial opportunities to infer the type of a local
>> variable. That would be my first priority to work on... when I deal
>> with all that project and xref stuff in the core, I guess.
>
> I'm not sure which code bit you are referencing here.  If you do your
> tag parsing with a semantic grammar, then you can optionally use that
> same grammar to parse function bodies, and thus make detecting local
> variable types a bit easier.   I'm speculating however as I am not
> familiar with Ruby.

I don't know how much work would that be. Ruby doesn't have anything 
close to official, up-to-date BNF grammar. And it's pretty complex.

At the moment, I'm doing context parsing in Elisp, so fix for the 
"trivial opportunities" might also be in Elisp, at first, with a few 
simple regexp searches.

However, I'm seriously considering moving that part (and more) to Ruby, 
so that authors of integration with other editors won't have to redo 
that work:

- There's a feature request for Vim support.
- Someone actually implemented an Atom plugin already (not at all 
popular so far, but that doesn't matter).

I'd also be able to reuse an existing parser. There is one that's been 
gaining in popularity over the last years.

> There is a wisent based grammar for Ruby in the 'contrib' area that
> seems straightforward.  It would probably not be much of a stretch to
> build one with the right releases to include in Emacs, solve this one
> problem, and then get support from other CEDET tools.

Since that grammar has been outside of Emacs for so long, I always 
assumed that the author vanished, and obtaining the release is impossible.

Re: IDE

[Lluís]

Dmitry Gutov writes:

> On 10/20/2015 03:37 PM, Lluís wrote:
>> And we also have the following project-types:
>> 
>> * project-type-generic:
>> * match: Return t if we can detect a project root using VC
>> * service-types:
>> * service-type-root: service-root-vc

> I see. This looks more like a classical structure, with one "project" objects,
> two attributes in there, and the second one containing a map with list values.

> We could indeed interpret those values as hooks, but that question is less
> important.

> The main benefit of this structure is familiarity to anyone who's done a little
> OOP.

> What I was imagining, since we're talking about services:

> project-types:
>   - project-type-generic:
>     * match
>     * root
>   - project-type-generic-c:
>     * match
>     * root

> search-paths-types:
>   - search-paths-linux (project)
>     * search-paths
>     * resolve-include-path <-- maybe
>   - search-paths-c++ (project)
>     * likewise
>     * likewise

> symbols-list-types:
>   - symbols-list-cscope (project, search-paths)
>   - symbols-list-wisent (project, search-paths)

> The upside is that I can write a new package called magic-8-ball, which would
> add an element at the beginning of symbols-list-types, and that element will
> have a chance to be used in every kind of project. Now just the projects that
> were explicitly written with magic-8-ball in mind. That adds flexibility.

> The downside is that indirection adds complexity as well, and could turn out to
> be mostly unused.

Aha, I see we were imagining different ways of structuring the concepts. What I
don't understand is what you mean with the parenthesis you add to some of the
elements of your "service-type hooks". Are these dependencies between services?

Now, how do you auto-detect what services to use on your currently open project?
I.e., how do you auto-detect what service implementations must be used?

Maybe I wasn't explicit enough in my case, but project-types are the only ones
that provide project detection, and therefore dictate the service
implementations to use on your project.


Cheers,
  Lluis

-- 
"And it's much the same thing with knowledge, for whenever you learn
something new, the whole world becomes that much richer."
-- The Princess of Pure Reason, as told by Norton Juster in The Phantom
Tollbooth

Re: IDE

[Lluís]

Dmitry Gutov writes:

> Hi Steinar,
> On 10/20/2015 06:23 PM, Steinar Bang wrote:
>> With maven, I might do something like this:
>>  project/
>>    pom.xml
>>    .git/
>>    module1/
>>      pom.xml
>>      .git/
>>    module2/
>>      pom.xml
>>      .git/
>>
>> The same projects in an eclipse workspace, might look something like
>> this:
>>  workspace/
>>    module1/
>>      pom.xml
>>      .git/
>>    module2/
>>      pom.xml
>>      .git/
>>
>> Or perhaps something like this (if the parent also has common settings I
>> would like easily editable).
>>  workspace/
>>    project/
>>      pom.xml
>>      .git/
>>    module1/
>>      pom.xml
>>      .git/
>>    module2/
>>      pom.xml
>>      .git/
>>
>> If the project was to be only edited with emacs, I would go for the top
>> layout, however if I was to edit the same projects in both emacs and
>> eclipse, it should handle the latter two layouts as well (though the
>> bottom one doesn't work too well with command line maven).

> That doesn't help, by itself. Surely we want to support all of these directory
> structures. The question is what each of them would translate to in the project
> API.

> Consider A. It could be considered one project, but then it would have certain
> attributes dependent on the current directory. Take classpath, for
> example. Could we consider it to be constant value across the whole project, or
> would we have, for certain operations (like "find the class named foo.bar.Bar"),
> different values of classpath in module1 and module2? The build-tool behavior
> would certainly depend on the current directory, but could we say that all other
> important project attributes are kind of the same for project, module1 and
> module2?

> Another option for A is to promote module1 and module2 to whole projects. But
> then, do we track project dependencies now? If I call a command "search for
> occurrences of 'xyz' in project", does it also search in module1 and module2? Or
> similarly for "list all files in project", does that include files in module1?
> That would be useful with Helm, for "jump to a project file". Does "list all
> files in project", when called in module1, include files from the parent
> project, and from module2?

> B doesn't look too different, except we apparently don't have a top-level
> pom-file.

> I don't understand C. Is module1 still inside project? Is it still a dependency?
> Do we treat it differently WRT to questions I've asked for the option A?

Ok, so what if we let project-types define project nesting?

Let's suppose example A has a "flat" configuration, while example B has a
nested one:

* service-search-paths-X-Y
  A "search path" service where X tells us whether paths are searched
  recursively, and where searches start at path Y.

* project-type-A:
  * children: nil
  * service-types
    * service-type-search-paths: service-search-paths-t-project/

* project-type-B:
  * children: project-type-B-m1 project-type-B-m2
  * service-types
    * service-type-search-paths: service-search-paths-nil-project/

* project-type-B-m1:
  * children: nil
  * service-types
    * service-type-search-paths: service-search-paths-nil-project/module1/

* project-type-B-m2:
  * children: nil
  * service-types
    * service-type-search-paths: service-search-paths-nil-project/module2/

Note that the same can be applied to things like whether cross-module searches
must work. For example, this would activate them for path searches:

    * service-type-search-paths: service-search-paths-nil-project/module1/ service-search-paths-nil-project/module2/

In most cases, it probably makes more sense to construct this nesting
programmatically by adding some logic during project auto-detection (e.g., read
some configuration file that is part of the project).


Thanks,
  Lluis

-- 
"And it's much the same thing with knowledge, for whenever you learn
something new, the whole world becomes that much richer."
-- The Princess of Pure Reason, as told by Norton Juster in The Phantom
Tollbooth

Re: IDE

[Dmitry Gutov]

On 10/21/2015 05:40 PM, Lluís wrote:

> Aha, I see we were imagining different ways of structuring the concepts. What I
> don't understand is what you mean with the parenthesis you add to some of the
> elements of your "service-type hooks". Are these dependencies between services?

Yes. And they're also function arguments. So the search-paths services 
would be implemented something like this:

(defun search-paths (project)
   (run-hook-with-args-until-success
    'search-path-functions
    project))

And let the others look like this:

(defun current-project (dir)
   (run-hook-with-args-until-success
    'project-find-functions
    dir))

(defun symbols-list (project search-paths)
   (run-hook-with-args-until-success
    'symbols-list-functions
    project
    search-paths))

Then a command which wants to use the symbols list will have to call all 
three of these functions.

> Now, how do you auto-detect what services to use on your currently open project?
> I.e., how do you auto-detect what service implementations must be used?

Each service implementation decides whether it's eligible, based on the 
PROJECT argument. It can ask the project for its root, or it can even 
check its struct/eieio/etc type, if it only want to work with certain 
kind of projects (for instance, when some project-find-functions and 
symbols-list-functions come from the same package, such as EDE).

> Maybe I wasn't explicit enough in my case, but project-types are the only ones
> that provide project detection, and therefore dictate the service
> implementations to use on your project.

Right. In this case, if I want to provide an search-paths implementation 
(maybe inside ruby-mode package), I also have to create a new project 
implementation, or look for an existing one to hook into somehow (that 
will most likely require some code modification).

Whereas, really, I can provide a rather small search-paths 
implementation that would fit the vast majority of Ruby projects, and 
would only look for Gemfile.lock in a parent directory, and depend on 
its contents (and maybe a few other file that signify the version of 
Ruby that's being used). That would nicely compose with Projectile as 
the project implementation, for instance.

Re: IDE

[Eric Ludlam]

On 10/21/2015 06:54 AM, Dmitry Gutov wrote:
> On 10/21/2015 06:13 AM, Eric Ludlam wrote:
>
>> Ruby support tooling itself would not benefit from CEDET integration,
>> but tools built on CEDET would gain Ruby support, and that improves the
>> diversity of Ruby related tools available.
>
> If CEDET support won't improve Robe, and if the said support will 
> amount to writing a Wisent grammar, it just becomes a separate task.
>
> Which I don't object to doing, in principle, but it'll go to the 
> bottom of the pile, in terms of priority.
I don't think it is about if CEDET improves ROBE.  I looked at ROBE on 
github, and think there is some core piece of it regarding talking to 
ruby that can be integrated as an extension of CEDET. You could then use 
the company-mode, auto-complete, plus many other tools that already use 
CEDET APIs, and could delete that code from robe.

In other words, your ruby experience would have more options without 
having to add them to robe yourself.

It is speculative to suggest that writing a ruby wisent parser would 
take less time than to write all the same types of extensions already on 
top of CEDET in Robe since I'm not sure which subset of features is the 
useful subset for something like ruby.

>> Inside this feature you must have a way to query for the location of a
>> particular symbol, and convert a symbol into a doc reference.
>
> In both cases, I often have to prompt the user (with completing-read) 
> to disambiguate between classes that define the method with the given 
> name.
>

In this case, things like semantic-complete-jump you could use the TAB 
technique to disambiguate, and semantic-ia-complete-jump might be able 
to distinguish depending on what sort of type information is available.

There is at least one contribution on top of cedet that do similar 
disambiguation using fancy uis.  I think it copied a textmate feature.

>> The output is a list of matches as faux tags.  If an application wanted
>> to know more about the symbol, it would pull in the reference file, and
>> extract real tag data using whatever parser is available.
>
> So as faux tags, I could return all methods with the same name from 
> different classes?

Yes.  You would provide what you do know (full class names, fields, 
methods, different file locations, etc) which could be initially 
reasoned on.  Sometimes no more is needed.  Other times those files 
might be opened to get more data using a better wisent parser.

The semantic javap extension pulls java tags from Jar files, and does 
not need to claim they are temporary tags because the data provided is 
as-good as the java parser.  If your ruby system is similar, then the 
system would not need to extract additional data from the source file.

>> This would enable Semantic's jump to tag system to be as accurate as 
>> yours.
>
> Would that be semantic-complete-jump or semantic-ia-fast-jump?
>

Both, plus other similar functions different people have written.

>> An EDE type project for ruby (whatever that looks like) would provide a
>> place to hang project specific REPL buffers as needed.
>
> How? Using which major mode? I current use inf-ruby for that (not 
> available in Emacs, for copyright assignment reasons). So it seems I'd 
> have to add multi-REPL support for it first.
>
> The conceptually hard question is what do you do with external files 
> that can be referenced from different projects? Suppose you have two 
> open projects, each with its own REPL, you made a jump to an external 
> file from project1. The you want to "jump to definition" on some 
> method call in there. How do you pick the right REPL to ask for info?
>

In C++ and Java, the common files would be 'system' includes.  The 
naming is a bit C centric, but the basic idea is that there are system 
includes projects refer to, and those in turn aren't in a project, but 
refer to system paths to continue navigating between themselves.

If you instead refer to a library of files unrelated to the system, then 
ideally those would have their own project structure around them to 
enable correct navigation.

Of course, the above examples for C don't have external REPL buffers to 
maintain, but might use independent GNU Global databases which are less 
heavy weight than an external process-per-project.

> Suppose, after the first jump, you saved the reference to the right 
> project in a buffer-local variable, so you can refer to it for the 
> second jump. What if I want to do the next jump not from the same 
> exact file, but from its neighbor? As a user, I can be confident that 
> both files must be referenced by the project, but there will be no 
> buffer-local value to use.
>

I'm not sure I followed this.  The shared code refers back to project1 
in some way?  This sounds like messy code you wouldn't want to write so 
I'm not sure I understand what you are trying to do.

Standard emacs mark-ring stuff is usually sufficient for traveling back.

> Finding an idiomatic approach to that would be great.
>
>>> - It misses some trivial opportunities to infer the type of a local
>>> variable. That would be my first priority to work on... when I deal
>>> with all that project and xref stuff in the core, I guess.
>>
>> I'm not sure which code bit you are referencing here.  If you do your
>> tag parsing with a semantic grammar, then you can optionally use that
>> same grammar to parse function bodies, and thus make detecting local
>> variable types a bit easier.   I'm speculating however as I am not
>> familiar with Ruby.
>
> I don't know how much work would that be. Ruby doesn't have anything 
> close to official, up-to-date BNF grammar. And it's pretty complex.
>
> At the moment, I'm doing context parsing in Elisp, so fix for the 
> "trivial opportunities" might also be in Elisp, at first, with a few 
> simple regexp searches.
>
> However, I'm seriously considering moving that part (and more) to 
> Ruby, so that authors of integration with other editors won't have to 
> redo that work:
>
> - There's a feature request for Vim support.
> - Someone actually implemented an Atom plugin already (not at all 
> popular so far, but that doesn't matter).
>
> I'd also be able to reuse an existing parser. There is one that's been 
> gaining in popularity over the last years.
>

Semantic doesn't demand it's parsers be in wisent, or even in Emacs at 
all.  If you have a nice Ruby grammar in Ruby, and you can convert it's 
internal data into lisp-like syntax, pulling it into the Semantic system 
is pretty easy.  What you would loose is dynamic reparsing without 
having to save, thus it may be a bit quirky.

>> There is a wisent based grammar for Ruby in the 'contrib' area that
>> seems straightforward.  It would probably not be much of a stretch to
>> build one with the right releases to include in Emacs, solve this one
>> problem, and then get support from other CEDET tools.
>
> Since that grammar has been outside of Emacs for so long, I always 
> assumed that the author vanished, and obtaining the release is 
> impossible.
>

Indeed, I suggested it to simply show someone made a useful grammar that 
looks relatively simple.

Eric

Re: IDE

[Dmitry Gutov]

On 10/22/2015 01:52 AM, Eric Ludlam wrote:

> I don't think it is about if CEDET improves ROBE.  I looked at ROBE on
> github, and think there is some core piece of it regarding talking to
> ruby that can be integrated as an extension of CEDET. You could then use
> the company-mode, auto-complete,

Well, that part doesn't make sense. I couldn't care less about 
auto-complete, and company-mode already supports 
completion-at-point-functions. Semantic does, too. 
completion-at-point-functions is a thinner dependency, and its API is at 
least as rich as what Semantic provides.

> plus many other tools that already use
> CEDET APIs, and could delete that code from robe.

Being able to delete any considerable amount of code is doubtful. 
Speaking of "many other tools", a compelling example would help. IMenu 
already works adequately thanks to ruby-mode, and which-func-mode works 
as well.

> In other words, your ruby experience would have more options without
> having to add them to robe yourself.

I get that idea, of course. An extra file with a bridge to Semantic 
wouldn't be the worst thing to have.

> It is speculative to suggest that writing a ruby wisent parser would
> take less time than to write all the same types of extensions already on
> top of CEDET in Robe since I'm not sure which subset of features is the
> useful subset for something like ruby.

Therein lies the problem: I've been trying out CEDET for a while, and 
I'm still not sure which extensions I'd want to reimplement. TAB 
completion in semantic-complete-jump is kinda nice, but it has a 
downside compared to the xref interface as well.

> In this case, things like semantic-complete-jump you could use the TAB
> technique to disambiguate, and semantic-ia-complete-jump might be able
> to distinguish depending on what sort of type information is available.

At first I was going to say it might be a solid (if small) improvement, 
but... there might be a mismatch:

- Simply searching for methods with the given name, like 
semantic-complete-jump, iterating through all matches, is wasteful IMO. 
It's much better to use the context if you can (or allow the user to 
input the fully qualified method name, as an extra feature).

- Even when the context is used, Robe can still return multiple matches. 
So if semantic-ia-complete-jump can't deal with that (and display all 
matches, for instance), it's not a good fit either.

> There is at least one contribution on top of cedet that do similar
> disambiguation using fancy uis.  I think it copied a textmate feature.

I'd love to take a look.

> The semantic javap extension pulls java tags from Jar files, and does
> not need to claim they are temporary tags because the data provided is
> as-good as the java parser.  If your ruby system is similar, then the
> system would not need to extract additional data from the source file.

Likewise, Robe would return a list of more-or-less exact tags.

Ruby runtimes keep the location of every defined method.

> In C++ and Java, the common files would be 'system' includes.  The
> naming is a bit C centric, but the basic idea is that there are system
> includes projects refer to, and those in turn aren't in a project, but
> refer to system paths to continue navigating between themselves.

That can be a system include, or it can be a library in a Git checkout 
that, say, two projects are using (it's included in the build by both of 
them).

> If you instead refer to a library of files unrelated to the system, then
> ideally those would have their own project structure around them to
> enable correct navigation.

a) There might not be a sufficient project structure (as with the system 
includes).
b) The project structure of that external dependency might not be 
enough. Here's what should be a clearer analogy:

Let's say that my projects P1 and P2 include the library lib_foo. It 
contains something like this:

class IFoo {
   public:
     virtual void bar();
};

class Tee {
   public:
     int qux(IFoo& foo) {
       foo.bar();
     }
}

Now, somewhere in P1 there's a snippet of code like:

   tee = new Tee();
   tee.qux(new P1Foo());

To follow the control flow, I move cursor to the qux call, jump to its 
definition (thus moving into lib_foo), and then I need to find the 
definition of bar that qux is calling (or rather, all of its overrides).

But only looking in lib_foo is insufficient for that: P1Foo is defined 
inside P1, and lib_foo, by itself, doesn't know anything about P1. It's 
P1 that depends on it.

So a seemingly obvious solution would be, when jumping to lib_foo, to 
store the fact that we came from P1 this time around. But there's no 
obvious place to keep it (directory-local storage?), and there might be 
other difficulties I haven't accounted for.

> Of course, the above examples for C don't have external REPL buffers to
> maintain, but might use independent GNU Global databases which are less
> heavy weight than an external process-per-project.

Yes, I believe you're going to have the same problem with etags, gtags 
or etc, if you try to use it in the described scenario.

> I'm not sure I followed this.  The shared code refers back to project1
> in some way?  This sounds like messy code you wouldn't want to write so
> I'm not sure I understand what you are trying to do.

I want to keep track of which project I'm working in right now. For code 
navigation forward, not back, like described above.

> Standard emacs mark-ring stuff is usually sufficient for traveling back.

Mark rings are highly inadequate, but that's beside the point.

> Semantic doesn't demand it's parsers be in wisent, or even in Emacs at
> all.  If you have a nice Ruby grammar in Ruby, and you can convert it's
> internal data into lisp-like syntax, pulling it into the Semantic system
> is pretty easy.  What you would loose is dynamic reparsing without
> having to save, thus it may be a bit quirky.

Why would I have to sacrifice reparsing without saving? I could sent the 
current buffer contents to the completion server, e.g. over HTTP. That's 
actually common practice now.

> Indeed, I suggested it to simply show someone made a useful grammar that
> looks relatively simple.

Simply redoing it to clear the copyright status won't be enough. Last I 
checked (some 3 years ago), the grammar didn't work. I've also seen 
complaints to that effect somewhere on the web.

Re: IDE

[Eric Ludlam]

On 10/21/2015 07:57 PM, Dmitry Gutov wrote:
> On 10/22/2015 01:52 AM, Eric Ludlam wrote:
>
>> I don't think it is about if CEDET improves ROBE.  I looked at ROBE on
>> github, and think there is some core piece of it regarding talking to
>> ruby that can be integrated as an extension of CEDET. You could then use
>> the company-mode, auto-complete,
>
> Well, that part doesn't make sense. I couldn't care less about 
> auto-complete, and company-mode already supports 
> completion-at-point-functions. Semantic does, too. 
> completion-at-point-functions is a thinner dependency, and its API is 
> at least as rich as what Semantic provides.
>

I assumed auto-complete and company were important because the ROBE 
documentation talked about how it supported those tools.

I'm not sure what is relevant here as far as listing features. There are 
12 minor modes that come with semantic doing various diverse things.  
Things like idle-summary-mode will show you an api summary of the symbol 
under point.  Breadcrumbs shows you a more oo version of where the 
cursor is.  Outside of those modes, the core analyzer and completion 
engines both use database backends to figure out what is going on near 
cursor and do completion.  There is COGRE which can pull data from the 
buffer, reference some databases, and pop up a uml diagram of your code 
which is a bit over-the-top. There are random utilities like semantic 
refactor that would need to be extended to new langauges but would gain 
base support from cedet.  There are browsers like speedbar and ECB that 
pull in data and let you navigate the structure of the code.

Of course, if none of those things are important, then you would have no 
reason to want to integrate w/ CEDET.

>
>> There is at least one contribution on top of cedet that do similar
>> disambiguation using fancy uis.  I think it copied a textmate feature.
>
> I'd love to take a look.
>
It is called eassist.  Looks like I was remembering it from 2006, so not 
sure how it will work with recent Emacs changes.

>
>> If you instead refer to a library of files unrelated to the system, then
>> ideally those would have their own project structure around them to
>> enable correct navigation.
>
> a) There might not be a sufficient project structure (as with the 
> system includes).
> b) The project structure of that external dependency might not be 
> enough. Here's what should be a clearer analogy:
>
> Let's say that my projects P1 and P2 include the library lib_foo. It 
> contains something like this:
>
> class IFoo {
>   public:
>     virtual void bar();
> };
>
> class Tee {
>   public:
>     int qux(IFoo& foo) {
>       foo.bar();
>     }
> }
>
> Now, somewhere in P1 there's a snippet of code like:
>
>   tee = new Tee();
>   tee.qux(new P1Foo());
>
> To follow the control flow, I move cursor to the qux call, jump to its 
> definition (thus moving into lib_foo), and then I need to find the 
> definition of bar that qux is calling (or rather, all of its overrides).
>
> But only looking in lib_foo is insufficient for that: P1Foo is defined 
> inside P1, and lib_foo, by itself, doesn't know anything about P1. 
> It's P1 that depends on it.
>
> So a seemingly obvious solution would be, when jumping to lib_foo, to 
> store the fact that we came from P1 this time around. But there's no 
> obvious place to keep it (directory-local storage?), and there might 
> be other difficulties I haven't accounted for.
>

Ah, I see.  I hadn't considered a workflow like this.  I can see how 
that would be useful.  I haven't thought about how to solve that 
problem.  I suspect the tooling would be custom where it knows more 
about the language, and can infer intent based on things like 'virtual' 
or the prospect of there being overrides.


>
>> Semantic doesn't demand it's parsers be in wisent, or even in Emacs at
>> all.  If you have a nice Ruby grammar in Ruby, and you can convert it's
>> internal data into lisp-like syntax, pulling it into the Semantic system
>> is pretty easy.  What you would loose is dynamic reparsing without
>> having to save, thus it may be a bit quirky.
>
> Why would I have to sacrifice reparsing without saving? I could sent 
> the current buffer contents to the completion server, e.g. over HTTP. 
> That's actually common practice now.
>
That sounds good.


Eric

Re: IDE

[Dmitry Gutov]

On 10/22/2015 04:35 AM, Eric Ludlam wrote:

> I assumed auto-complete and company were important because the ROBE
> documentation talked about how it supported those tools.

Oh, right. Sorry.

> I'm not sure what is relevant here as far as listing features. There are
> 12 minor modes that come with semantic doing various diverse things.
> Things like idle-summary-mode will show you an api summary of the symbol
> under point.

Robe already supports eldoc-mode. If idle-summary-mode has some UI 
advantages, I believe they should be folded into eldoc-mode as well.

> There is COGRE which can pull data from the
> buffer, reference some databases, and pop up a uml diagram of your code
> which is a bit over-the-top.

COGRE sounds good, but I imagine it'll need more support than just 
dumping the AST.

And I can't get it to do anything (here's the documentation for 
automatic generation: 
http://www.randomsample.de/cedetdocs/cogre/Semantic-Support.html#Semantic-Support, 
and trying to interact with the editor manually leads to "eieio-oref: 
eieio-oref called on a class: cogre-class", etc).

It's also not in Emacs, for some reason.

> It is called eassist.  Looks like I was remembering it from 2006, so not
> sure how it will work with recent Emacs changes.

I've taken a look, and it works okay as an alternative to IMenu, but 
jumping to method at point doesn't work (shows an error).

> Ah, I see.  I hadn't considered a workflow like this.  I can see how
> that would be useful.  I haven't thought about how to solve that
> problem.  I suspect the tooling would be custom where it knows more
> about the language, and can infer intent based on things like 'virtual'
> or the prospect of there being overrides.

The tooling needs to be smart enough, of course.

Java IDEs support workflows like that, and I think they're more 
important than speedbar or showing breadcrumbs (*).

Just to let you know where my priorities are. Here's the current plan 
for that feature, by the way: 
https://github.com/nonsequitur/inf-ruby/issues/76#issuecomment-150182991

(*) Other things that the users ask for is fuzzy completion, showing 
completions right away after dot (Robe isn't fast enough for that) and 
working over TRAMP inside a Vagrant environment. It doesn't seem like 
CEDET integration will help with any of those.

Re: IDE

[Eric Ludlam]

On 10/22/2015 07:17 AM, Dmitry Gutov wrote:
> On 10/22/2015 04:35 AM, Eric Ludlam wrote:
>
>> I'm not sure what is relevant here as far as listing features. There are
>> 12 minor modes that come with semantic doing various diverse things.
>> Things like idle-summary-mode will show you an api summary of the symbol
>> under point.
>
> Robe already supports eldoc-mode. If idle-summary-mode has some UI 
> advantages, I believe they should be folded into eldoc-mode as well.
>

Semantic's idle process handling makes sure databases are synchronized 
once, then enables tools to run after, so it is more about the 
scheduling of different tools that use semantic.

In addition, by going through the semantic version, there are a range of 
different formatting options to use that the user can select from.  That 
doesn't require idle-summary-mode, but is a side effect of using 
semantic to extract contextual information.

Also, eldoc only supported Emacs Lisp and no extensions when I wrote the 
semantic idle handler.  Other features of idle-summary-mode would port 
fine between either.

>> There is COGRE which can pull data from the
>> buffer, reference some databases, and pop up a uml diagram of your code
>> which is a bit over-the-top.
>
> COGRE sounds good, but I imagine it'll need more support than just 
> dumping the AST.
>
> And I can't get it to do anything (here's the documentation for 
> automatic generation: 
> http://www.randomsample.de/cedetdocs/cogre/Semantic-Support.html#Semantic-Support, 
> and trying to interact with the editor manually leads to "eieio-oref: 
> eieio-oref called on a class: cogre-class", etc).
>
> It's also not in Emacs, for some reason.

It was deemed optional when Yidong merged CEDET into Emacs.  You will 
probably need to use Emacs24 to make it work.  To try it out do this:

1) Install graphviz (it uses that for the layout engine)
2) Start emacs 24
3) Use CEDET from it's git repository
4) M-x find-library RET cogre RET
5) find cogre-element-peer in the code
6) M-x cogre-uml-quick-class RET

should get you something to play with.

> (*) Other things that the users ask for is fuzzy completion, showing 
> completions right away after dot (Robe isn't fast enough for that) and 
> working over TRAMP inside a Vagrant environment. It doesn't seem like 
> CEDET integration will help with any of those.
>

CEDET is a framework that provides an abstraction for connecting 
different tools that need to talk about hard problems together.  The 
problems it solves are related to project information, abstracting 'tag' 
information down to something Lisp programs can reason on, and 
abstracting code generation into a scheme that can allow lisp programs 
to support multiple languages.  CEDET doesn't have 'fuzzy completion' 
but it can feed a fuzzy completion engine.  CEDET doesn't do anything 
special with TRAMP, but someone could use CEDET to bind that workflow 
into the common workflow.  When thinking about CEDET, it isn't about a 
bullet list of user facing features but about how it can enable someone 
working on said feature to have their work leveraged to a wider audience.

Eric

Re: IDE

[Louis Höfler]

Hello everyone,
I followed that topic with much interest.

I developed a starting point of a IDE implementation in emacs.
http://scm.mathematek.de/index.cgi/emacside/index
It is not nearly finished, but I use it for my daily work.
Most ide's I started to use have tons of options but don't do
the core things well.
For me it is important to have a ide which:
1. Does not break
2. Does not change

I even do not need colored editor output, I just want a working IDE.
An IDE is in my point of view, a toolset which allows me to
compile things fast, and do repeating processes without much time loss.
I dont like IDE environments where you need 200 Dialogs to reach
functions.
My package currently can do compilations, and I started to implement
a emacsserver API, so that you can send a compilation configuration to
a remote Computer, which does the compilation.
If you want to have more functions arround that, you can just hack some
elisp code arround each executed command with hooks, so you can get full
debugger functionalities.
Running gdb after a compilation for example, throws me directly into the
debugger environment, where I can set breakpoints etc.

This is a out-of-the-need 1-man project. So you can't expect it to be as 
complete
and functional as other packages with many contributors.

For me personally, the simple things are the ones I continue to use 
nowadays.
The complex things are those who break from time to time.
Zero administration is one of the biggest concerns if I start to use 
software.


Greetings, Louis

Am 22.10.2015 um 14:58 schrieb Eric Ludlam:
> On 10/22/2015 07:17 AM, Dmitry Gutov wrote:
>> On 10/22/2015 04:35 AM, Eric Ludlam wrote:
>>
>>> I'm not sure what is relevant here as far as listing features. There 
>>> are
>>> 12 minor modes that come with semantic doing various diverse things.
>>> Things like idle-summary-mode will show you an api summary of the 
>>> symbol
>>> under point.
>>
>> Robe already supports eldoc-mode. If idle-summary-mode has some UI 
>> advantages, I believe they should be folded into eldoc-mode as well.
>>
>
> Semantic's idle process handling makes sure databases are synchronized 
> once, then enables tools to run after, so it is more about the 
> scheduling of different tools that use semantic.
>
> In addition, by going through the semantic version, there are a range 
> of different formatting options to use that the user can select from.  
> That doesn't require idle-summary-mode, but is a side effect of using 
> semantic to extract contextual information.
>
> Also, eldoc only supported Emacs Lisp and no extensions when I wrote 
> the semantic idle handler.  Other features of idle-summary-mode would 
> port fine between either.
>
>>> There is COGRE which can pull data from the
>>> buffer, reference some databases, and pop up a uml diagram of your code
>>> which is a bit over-the-top.
>>
>> COGRE sounds good, but I imagine it'll need more support than just 
>> dumping the AST.
>>
>> And I can't get it to do anything (here's the documentation for 
>> automatic generation: 
>> http://www.randomsample.de/cedetdocs/cogre/Semantic-Support.html#Semantic-Support, 
>> and trying to interact with the editor manually leads to "eieio-oref: 
>> eieio-oref called on a class: cogre-class", etc).
>>
>> It's also not in Emacs, for some reason.
>
> It was deemed optional when Yidong merged CEDET into Emacs.  You will 
> probably need to use Emacs24 to make it work.  To try it out do this:
>
> 1) Install graphviz (it uses that for the layout engine)
> 2) Start emacs 24
> 3) Use CEDET from it's git repository
> 4) M-x find-library RET cogre RET
> 5) find cogre-element-peer in the code
> 6) M-x cogre-uml-quick-class RET
>
> should get you something to play with.
>
>> (*) Other things that the users ask for is fuzzy completion, showing 
>> completions right away after dot (Robe isn't fast enough for that) 
>> and working over TRAMP inside a Vagrant environment. It doesn't seem 
>> like CEDET integration will help with any of those.
>>
>
> CEDET is a framework that provides an abstraction for connecting 
> different tools that need to talk about hard problems together. The 
> problems it solves are related to project information, abstracting 
> 'tag' information down to something Lisp programs can reason on, and 
> abstracting code generation into a scheme that can allow lisp programs 
> to support multiple languages.  CEDET doesn't have 'fuzzy completion' 
> but it can feed a fuzzy completion engine.  CEDET doesn't do anything 
> special with TRAMP, but someone could use CEDET to bind that workflow 
> into the common workflow. When thinking about CEDET, it isn't about a 
> bullet list of user facing features but about how it can enable 
> someone working on said feature to have their work leveraged to a 
> wider audience.
>
> Eric
>

Re: IDE

[Evgeniy Dushistov]

On Wed, Oct 21, 2015 at 06:52:55PM -0400, Eric Ludlam wrote:
> 
> Semantic doesn't demand it's parsers be in wisent, or even in Emacs at all.
> If you have a nice Ruby grammar in Ruby, and you can convert it's internal
> data into lisp-like syntax, pulling it into the Semantic system is pretty
> easy.  What you would loose is dynamic reparsing without having to save,
> thus it may be a bit quirky.
> 

Is any documentation about this:
- format of data that Semantic understand
- which hooks should be used to give this information to semantic,
so it can use it to colorization, completition, tags navigation etc
?


For example, if I write such code
#include <string>
int main() { std::string str; }

and call semantic-ia-fast-jump on "std::string" ede/cedet/semantic
tell me "Could not find `string'. Jump to std?"
It works in more simple case for std::vector,
but fails with std::string.

But if I ask emacs via rtags-find-symbol, all works fine and
I see
  typedef basic_string<char>    string;   

So I think it would be good integrate
rtags(https://github.com/Andersbakken/rtags) (or some another existing
daemon) for such complex language as c++ and cedet.

But I can not find any clear description for external parsers usage?

-- 
/Evgeniy

Re: IDE

[David Engster]

Evgeniy Dushistov writes:
> On Wed, Oct 21, 2015 at 06:52:55PM -0400, Eric Ludlam wrote:
>> 
>> Semantic doesn't demand it's parsers be in wisent, or even in Emacs at all.
>> If you have a nice Ruby grammar in Ruby, and you can convert it's internal
>> data into lisp-like syntax, pulling it into the Semantic system is pretty
>> easy.  What you would loose is dynamic reparsing without having to save,
>> thus it may be a bit quirky.
>> 
>
> Is any documentation about this:
> - format of data that Semantic understand

See chapter 1 in the Semantic AppDev manual:

  http://www.randomsample.de/cedetdocs/semantic-appdev/

Running M-x bovinate in a C file will already give you a good impression.

> - which hooks should be used to give this information to semantic, so
> it can use it to colorization, completition, tags navigation etc ?

Hooks are not well suited for this task, since you want to extend or
override functionality depending on the current major-mode. For this,
the mode-local package is used, so one should get familiar with it
first. The most important functions to override are listed in the
Semantic AppDev manual (see for instance chapters 7 and 12). It is also
good to look into a Semantic backend for a language which you know
reasonably well to see how it is done. Last not least, you can ask on
cedet-devel.

> So I think it would be good integrate
> rtags(https://github.com/Andersbakken/rtags) (or some another existing
> daemon) for such complex language as c++ and cedet.

AFAIK, rtags solely uses libclang, which does not give you direct access
to the AST, so I wouldn't know how rtags could be used to export the
semantic tags of the current buffer. You could hook rtags into Semantic
purely for the end-user features, like completion/references.

This is also why I will use libtooling for C++, which lets you walk the
AST.

-David

Re: IDE

[Evgeniy Dushistov]

On Fri, Oct 23, 2015 at 04:55:06PM +0200, David Engster wrote:
> Evgeniy Dushistov writes:
> > On Wed, Oct 21, 2015 at 06:52:55PM -0400, Eric Ludlam wrote:
> >> 
> >> Semantic doesn't demand it's parsers be in wisent, or even in Emacs at all.
> >> If you have a nice Ruby grammar in Ruby, and you can convert it's internal
> >> data into lisp-like syntax, pulling it into the Semantic system is pretty
> >> easy.  What you would loose is dynamic reparsing without having to save,
> >> thus it may be a bit quirky.
> >> 
> >
> > Is any documentation about this:
> > - format of data that Semantic understand
> 
> See chapter 1 in the Semantic AppDev manual:
> 
>   http://www.randomsample.de/cedetdocs/semantic-appdev/
> 

Thanks, I used 'info semantic' and have no idea that there is also
semantic-appdev


> > So I think it would be good integrate
> > rtags(https://github.com/Andersbakken/rtags) (or some another existing
> > daemon) for such complex language as c++ and cedet.
> 
> AFAIK, rtags solely uses libclang, which does not give you direct access
> to the AST, so I wouldn't know how rtags could be used to export the
> semantic tags of the current buffer. You could hook rtags into Semantic
> purely for the end-user features, like completion/references.
> 
> This is also why I will use libtooling for C++, which lets you walk the
> AST.

Why do you think that libclang not give access to AST?
I used it from python like this:

def callexpr_visitor(node, parent, parser):
# your code here
    for c in node.get_children():
        callexpr_visitor(c, node, parser)
    return 2 # means continue visiting recursively

tu = TranslationUnit.from_source(fname, args)
callexpr_visitor(tu.cursor, None, parser)

and callexpr_visitor  visit each node in AST.
Because of clang's python API just part of C/C++ API,
then it should be possible in rtags to have access to AST.


-- 
/Evgeniy

Re: IDE

[David Engster]

Evgeniy Dushistov writes:
> Because of clang's python API just part of C/C++ API,
> then it should be possible in rtags to have access to AST.

You are right, the stuff that is available through libclang could be
exported as Semantic tags. But last I checked lots of stuff was not
accessible through libclang as it was only exported as so called
"unexposed statements".

-David

------------------------------------------------------------------------------

Re: IDE

[Nix]

[Catching up on this thread...]

On 12 Oct 2015, Drew Adams verbalised:

>> > 3. Jumping around the project code and resources.
>> > Jumping to around the project code and used libraries. Another
>> > thing is jumping to/from resources (for example aspects can be
>> > defined in an XML file and IDE could allow to jump to matching
>> > classes).
>> 
>> Do you mean "jump to the thing at point"? That sounds complicated,
>> and support for different "things" will have to be implemented
>> separately.
>
> Sounds like good ol' Emacs TAGS, to me (or across-project Imenu).
> Of course, the limiting factor is TAGS files that support such
> "things".  But the infrastructure is there for it.  People have
> been using Emacs TAGS files to "jump to the [definition of the]
> thing at point" for 40 years.

btw, recent GNU GLOBAL has now shifted to using a SQLite database for
its tags files, which makes it hugely more extensible, in theory, and
also makes it possible that Emacs could (once the modules code lands so
we could write a glue layer to SQLite) directly extract info from it.

Raw TAGS files are more or less unsuitable for anything but C and Lisp,
and are pretty poor even for that (e.g. you can only jump from uses to
definitions, the definition can only be in one place...)

-- 
NULL && (void)

Re: IDE

[Eli Zaretskii]

> From: Nix <nix@esperi.org.uk>
> Cc: Dmitry Gutov <dgutov@yandex.ru>,
>         Przemysław Wojnowski
>  <esperanto@cumego.com>,
>         Eli Zaretskii <eliz@gnu.org>, adatgyujto@gmail.com,
>         emacs-devel@gnu.org
> Emacs: The Awakening
> Date: Sat, 24 Oct 2015 15:17:10 +0100
> 
> Raw TAGS files are more or less unsuitable for anything but C and Lisp,
> and are pretty poor even for that (e.g. you can only jump from uses to
> definitions, the definition can only be in one place...)

Raw TAGS files are not supposed to be used for anything but
definitions.

For references, you are supposed to use ID-Utils or something similar,
which use a different format of their DB.

Re: IDE

[Nix]

On 24 Oct 2015, Eli Zaretskii outgrape:

>> From: Nix <nix@esperi.org.uk>
>> Cc: Dmitry Gutov <dgutov@yandex.ru>,
>>         Przemysław Wojnowski
>>  <esperanto@cumego.com>,
>>         Eli Zaretskii <eliz@gnu.org>, adatgyujto@gmail.com,
>>         emacs-devel@gnu.org
>> Emacs: The Awakening
>> Date: Sat, 24 Oct 2015 15:17:10 +0100
>> 
>> Raw TAGS files are more or less unsuitable for anything but C and Lisp,
>> and are pretty poor even for that (e.g. you can only jump from uses to
>> definitions, the definition can only be in one place...)
>
> Raw TAGS files are not supposed to be used for anything but
> definitions.
>
> For references, you are supposed to use ID-Utils or something similar,
> which use a different format of their DB.

Two different tools, for more or less identical jobs except that one is
one->many and the other is many->one? (In particular, the hard part
isn't the data structure, but the parsing.)

That strikes me as really, really ugly.

-- 
NULL && (void)

Re: IDE

[Eli Zaretskii]

> From: Nix <nix@esperi.org.uk>
> Cc: drew.adams@oracle.com, dgutov@yandex.ru, esperanto@cumego.com,
>         adatgyujto@gmail.com, emacs-devel@gnu.org
> Emacs: a Lisp interpreter masquerading as ... a Lisp interpreter!
> Date: Sat, 24 Oct 2015 17:29:31 +0100
> 
> On 24 Oct 2015, Eli Zaretskii outgrape:
> 
> >> From: Nix <nix@esperi.org.uk>
> >> Cc: Dmitry Gutov <dgutov@yandex.ru>,
> >>         Przemysław Wojnowski
> >>  <esperanto@cumego.com>,
> >>         Eli Zaretskii <eliz@gnu.org>, adatgyujto@gmail.com,
> >>         emacs-devel@gnu.org
> >> Emacs: The Awakening
> >> Date: Sat, 24 Oct 2015 15:17:10 +0100
> >> 
> >> Raw TAGS files are more or less unsuitable for anything but C and Lisp,
> >> and are pretty poor even for that (e.g. you can only jump from uses to
> >> definitions, the definition can only be in one place...)
> >
> > Raw TAGS files are not supposed to be used for anything but
> > definitions.
> >
> > For references, you are supposed to use ID-Utils or something similar,
> > which use a different format of their DB.
> 
> Two different tools, for more or less identical jobs except that one is
> one->many and the other is many->one? (In particular, the hard part
> isn't the data structure, but the parsing.)
> 
> That strikes me as really, really ugly.

Are we still talking about an Emacs IDE?  If so, there's only one
tool: Emacs.  What happens behind the scenes is of interest to us
developers, but the user doesn't need to know, or even suspect.

Right?

RE: IDE

[Drew Adams]

> > Sounds like good ol' Emacs TAGS, to me (or across-project Imenu).
> > Of course, the limiting factor is TAGS files that support such
> > "things".  But the infrastructure is there for it.  People have
> > been using Emacs TAGS files to "jump to the [definition of the]
> > thing at point" for 40 years.
> 
> btw, recent GNU GLOBAL has now shifted to using a SQLite database for
> its tags files, which makes it hugely more extensible, in theory, and
> also makes it possible that Emacs could (once the modules code lands so
> we could write a glue layer to SQLite) directly extract info from it.
> 
> Raw TAGS files are more or less unsuitable for anything but C and Lisp,
> and are pretty poor even for that (e.g. you can only jump from uses to
> definitions, the definition can only be in one place...)

I don't think this is something inherent in TAGS files.

You can write a TAGS file for any kind of "definitions".
And I put that in quotes because such a "definition" can
really mean anything at all.  A TAGS file is just an index
into a document or a set of documents.

The fact that a program might not exist yet for creating
useful TAGS files for some language does not change this.

And the same thing you say about TAGS could be said about,
say, Imenu:  Until/unless someone writes the code needed
to use Imenu in a particular mode, it does not support that
mode.  That's not a problem with or limitation of Imenu.
It's just a lack of interest in writing the requisite
support for it for that mode/language.

It's also not clear to me what you mean by "the definition
can only be in one place".  AFAIK, you can have multiple
definitions of ("defining" locations for) the same thingy
in a single TAGS file.

And certainly you can have multiple such in a _set_ of
TAGS files.  And part of the use of TAGS files is
searching across multiple TAGS files.  TAGS files are
composable: they can be used together.

Please correct me if I'm mistaken.  I'm no expert on
TAGS files.

(Also, it is welcome that SQLite and other indexing and
querying means also be made available to Emacs.  Emacs
is not limited to TAGS or Imenu or ...  Tomorrow you
might use a SQL database with SQL/JSON indexing and
querying.  Who knows?)

RE: IDE

[Drew Adams]

> > Raw TAGS files are more or less unsuitable for anything but C and Lisp,
> > and are pretty poor even for that (e.g. you can only jump from uses to
> > definitions, the definition can only be in one place...)
> 
> Raw TAGS files are not supposed to be used for anything but
> definitions.

Where "definition" can be whatever you want, AFAIK.  So unless I'm
mistaken about this, I don't think your statement is very meaningful.

A TAGS file is just an index.  You can index whatever you like,
AFAIK.

RE: IDE

[Drew Adams]

> > For references, you are supposed to use ID-Utils or something similar,
> > which use a different format of their DB.
> 
> Two different tools, for more or less identical jobs except that one is
> one->many and the other is many->one? (In particular, the hard part
> isn't the data structure, but the parsing.)

What prevents someone from creating a TAGS file that includes
"references" as (additions forms of) "definitions"?  How is
adding references different in principle from adding, say,
handling of defstructs to a program that previously only
handled defuns and defvars?

You can index pretty much anything.  You could presumably
even create a full-text index and write it out as a TAGS
file, if you were up to it.

(But I agree: the hard part is the parsing.  The TAGS file
data structure is not the problem.)

Re: IDE

[Dmitry Gutov]

On 10/24/2015 05:25 PM, Eli Zaretskii wrote:

> Raw TAGS files are not supposed to be used for anything but
> definitions.
>
> For references, you are supposed to use ID-Utils or something similar,
> which use a different format of their DB.

Why id-utils, really? AFAIK, Global supports different kinds of searches 
(definitions, references and others), *and* it can plug into ctags, 
which extends its list of supported languages (not sure if that limits 
the files scanned that way to "find definitions" search only).

Seems like a natural solution to try to use in Emacs. ggtags, in GNU 
ELPA, already has an Emacs Lisp interface for it.

Re: IDE

[Eli Zaretskii]

> Date: Sat, 24 Oct 2015 10:00:25 -0700 (PDT)
> From: Drew Adams <drew.adams@oracle.com>
> Cc: dgutov@yandex.ru, esperanto@cumego.com, adatgyujto@gmail.com,
>         emacs-devel@gnu.org
> 
> > > Raw TAGS files are more or less unsuitable for anything but C and Lisp,
> > > and are pretty poor even for that (e.g. you can only jump from uses to
> > > definitions, the definition can only be in one place...)
> > 
> > Raw TAGS files are not supposed to be used for anything but
> > definitions.
> 
> Where "definition" can be whatever you want, AFAIK.

"Definition" in this context means the implementation.  There's only
one implementation, but there might be many references
(a.k.a. "calls").

> A TAGS file is just an index.  You can index whatever you like,
> AFAIK.

An index where the key is the symbol itself can only hold one instance
of every symbol.

Re: IDE

[Eli Zaretskii]

> Cc: drew.adams@oracle.com, esperanto@cumego.com, adatgyujto@gmail.com,
>  emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Sat, 24 Oct 2015 20:02:56 +0300
> 
> On 10/24/2015 05:25 PM, Eli Zaretskii wrote:
> 
> > Raw TAGS files are not supposed to be used for anything but
> > definitions.
> >
> > For references, you are supposed to use ID-Utils or something similar,
> > which use a different format of their DB.
> 
> Why id-utils, really? AFAIK, Global supports different kinds of searches 
> (definitions, references and others), *and* it can plug into ctags, 
> which extends its list of supported languages (not sure if that limits 
> the files scanned that way to "find definitions" search only).

We already support both, don't we?  So there's no contradiction.

My point was that the format of the database is not really important,
certainly not to the user.

Re: IDE

[Dmitry Gutov]

On 10/24/2015 08:00 PM, Drew Adams wrote:

> (But I agree: the hard part is the parsing.  The TAGS file
> data structure is not the problem.)

It *is* a problem as well. You basically have to re-generate the file 
each time you reparse the project (--append has its drawbacks), and GNU 
Global optimizes that (someone familiar with SQL databases should easily 
imagine a way to do that).

Further, now we're forced to parse the TAGS file and perform filtering 
in Elisp, which can be performed faster in an external program. Global 
can help there as well (we don't really need FFI support, unless Global 
command line interface is found too limiting).

Re: IDE

[Dmitry Gutov]

On 10/24/2015 08:11 PM, Eli Zaretskii wrote:

> We already support both, don't we?

For find-references, let's say yes. But not for find-definition, unless 
you include the GNU ELPA package.

> My point was that the format of the database is not really important,
> certainly not to the user.

A user might like fast project re-scanning, I imagine.

Re: IDE

[Eli Zaretskii]

> Cc: nix@esperi.org.uk, drew.adams@oracle.com, esperanto@cumego.com,
>  adatgyujto@gmail.com, emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Sat, 24 Oct 2015 20:15:11 +0300
> 
> On 10/24/2015 08:11 PM, Eli Zaretskii wrote:
> 
> > We already support both, don't we?
> 
> For find-references, let's say yes. But not for find-definition, unless 
> you include the GNU ELPA package.

Why cannot we bundle it?

> > My point was that the format of the database is not really important,
> > certainly not to the user.
> 
> A user might like fast project re-scanning, I imagine.

If etags is not fast enough, then the user can switch the back-end.

Once again, the issue is not the format of the database.  That's
immaterial.

RE: IDE

[Drew Adams]

> > Where "definition" can be whatever you want, AFAIK.
> 
> "Definition" in this context means the implementation.  There's only
> one implementation, but there might be many references
> (a.k.a. "calls").

That just assumes that you index only the "implementation".

In principle, a TAGS file could be created (including on the
fly) from not only "the implementation" files but also the
"calls" files.

> > A TAGS file is just an index.  You can index whatever you like,
> > AFAIK.
> 
> An index where the key is the symbol itself can only hold one instance
> of every symbol.

Is a TAGS file limited to symbols?  I didn't think so.

And I definitely have TAGS files that have multiple entries
for the same symbol definition.  The definitions are from
different source files, but they are in the same TAGS file
(in different sections, separated by form-feed chars).

For example:

^L
frame-cmds-OLD.el,1980
(defun iconify-everything ()\x7ficonify-everything\x01298,11152
...
^L
frame-cmds.el,1890
(defun iconify-everything ()\x7ficonify-everything\x01141,5218

RE: IDE

[Drew Adams]

> > (But I agree: the hard part is the parsing.  The TAGS file
> > data structure is not the problem.)
> 
> It *is* a problem as well. You basically have to re-generate the file
> each time you reparse the project (--append has its drawbacks), and GNU
> Global optimizes that (someone familiar with SQL databases should easily
> imagine a way to do that).
>
> Further, now we're forced to parse the TAGS file and perform filtering
> in Elisp, which can be performed faster in an external program. Global
> can help there as well (we don't really need FFI support, unless Global
> command line interface is found too limiting).

I'm not contrasting TAGS with Gnu Global.  You are free to do
that.  I am not arguing in favor of TAGS over other indexing
and querying mechanisms.

The TAGS file feature defines an index format and an index
query mechanism.  That's all.  How and when the content of a
given index gets generated or updated is a different question.

And that generation/updating involves parsing, which has
been acknowledged to be the hard part.

Everything you say in support of claiming that the data
structure "*is* a problem" is, in fact statements about
the problem of parsing, not the data structure format.

If you want to argue that any use of *files* to hold the
index structure is problematic then do so explicitly.

Even then, that does not invalidate the TAGS index
structure.  It need not be stored on disk, in principle.

Re: IDE

[Eli Zaretskii]

> Date: Sat, 24 Oct 2015 10:37:47 -0700 (PDT)
> From: Drew Adams <drew.adams@oracle.com>
> Cc: nix@esperi.org.uk, dgutov@yandex.ru, esperanto@cumego.com,
>         adatgyujto@gmail.com, emacs-devel@gnu.org
> 
> > > Where "definition" can be whatever you want, AFAIK.
> > 
> > "Definition" in this context means the implementation.  There's only
> > one implementation, but there might be many references
> > (a.k.a. "calls").
> 
> That just assumes that you index only the "implementation".

No, it's a definition of "definition" in this context.

> Is a TAGS file limited to symbols?  I didn't think so.

The "tags" are symbols, yes.

> And I definitely have TAGS files that have multiple entries
> for the same symbol definition.  The definitions are from
> different source files, but they are in the same TAGS file
> (in different sections, separated by form-feed chars).
> 
> For example:
> 
> ^L
> frame-cmds-OLD.el,1980
> (defun iconify-everything ()\x7ficonify-everything\x01298,11152
> ...
> ^L
> frame-cmds.el,1890
> (defun iconify-everything ()\x7ficonify-everything\x01141,5218

These are two different symbols, because the file name is (implicitly)
part of it.  There can be at most one definition per file, but many
references.

Anyway, what's this thread about, exactly?

RE: IDE

[Drew Adams]

> > And I definitely have TAGS files that have multiple entries
> > for the same symbol definition.  The definitions are from
> > different source files, but they are in the same TAGS file
> > (in different sections, separated by form-feed chars).
> >
> > For example:
> > ^L
> > frame-cmds-OLD.el,1980
> > (defun iconify-everything ()\x7ficonify-everything\x01298,11152
> > ...
> > ^L
> > frame-cmds.el,1890
> > (defun iconify-everything ()\x7ficonify-everything\x01141,5218
> 
> These are two different symbols, because the file name is (implicitly)
> part of it.  There can be at most one definition per file, but many
> references.

Now you've changed the kind of "file" being talked about.
You are now presumably saying that there can be only one
definition for a given term per _source_ file, not per _TAGS_
file.

The question being discussed was whether you could have multiple
"definitions" of a term in the same TAGS file.  And AFAICT you
can.

And a fortiori, you can have multiple definitions of a given
term in a set of multiple TAGS files, which is part of the
design for querying tags.

Re: IDE

[Dmitry Gutov]

On 10/24/2015 08:42 PM, Drew Adams wrote:

> I'm not contrasting TAGS with Gnu Global.  You are free to do
> that.  I am not arguing in favor of TAGS over other indexing
> and querying mechanisms.

Emacs doesn't have any real abstraction over TAGS files. etags.el 
basically operates on its contents.

> The TAGS file feature defines an index format and an index
> query mechanism.  That's all.  How and when the content of a
> given index gets generated or updated is a different question.

The index doesn't even say what kind of hit it is. Is it a definition? 
Is it a reference? Is it both? Like, a method override.

> And that generation/updating involves parsing, which has
> been acknowledged to be the hard part.

There are several parts, of varying difficulties.

> Everything you say in support of claiming that the data
> structure "*is* a problem" is, in fact statements about
> the problem of parsing, not the data structure format.

Nothing I've said about the file format yet is concerned with parsing.

> If you want to argue that any use of *files* to hold the
> index structure is problematic then do so explicitly.

Flat files, yes. Not any kind of files, obviously.

Re: IDE

[Dmitry Gutov]

On 10/24/2015 08:20 PM, Eli Zaretskii wrote:

> Why cannot we bundle it?

We "can" do anything. That's not an interesting question.

I believe Nix's suggestion was to use Global as the primary indexing 
solution (that role still belongs to etags.el, at least in part).

I think that might be a worthy change.

> Once again, the issue is not the format of the database.  That's
> immaterial.

Database format can have a real performance impact.

RE: IDE

[Drew Adams]

> > I'm not contrasting TAGS with Gnu Global.  You are free to do
> > that.  I am not arguing in favor of TAGS over other indexing
> > and querying mechanisms.
> 
> Emacs doesn't have any real abstraction over TAGS files. etags.el
> basically operates on its contents.

Yes, and?

> > The TAGS file feature defines an index format and an index
> > query mechanism.  That's all.  How and when the content of a
> > given index gets generated or updated is a different question.
> 
> The index doesn't even say what kind of hit it is. Is it a definition?
> Is it a reference? Is it both? Like, a method override.

It lets you index a location in a source file, associating it
with a name (symbol).  That's all it does.  No one is saying
that TAGS is the most general indexing system, or that it is
adequate for all indexing needs.

It is one tool among many possible index-and-query tools.

> > And that generation/updating involves parsing, which has
> > been acknowledged to be the hard part.
> 
> There are several parts, of varying difficulties.
> 
> > Everything you say in support of claiming that the data
> > structure "*is* a problem" is, in fact statements about
> > the problem of parsing, not the data structure format.
> 
> Nothing I've said about the file format yet is concerned
> with parsing.

Please read what you wrote, which you've elided.  The problems
you mentioned were about (1) having to "re-generate the file
each time you reparse the project" and (2) being "forced to
parse the TAGS file and perform filtering in Elisp".

I guess you could argue that #2 as a problem derives from
the TAGS data structure, but nothing specific was said about
what that problem is.  #1 seems clearly to be about parsing
the source files.

> > If you want to argue that any use of *files* to hold the
> > index structure is problematic then do so explicitly.
> 
> Flat files, yes. Not any kind of files, obviously.

You might want to elaborate, if there is something important
there.

But again.  No one is arguing that TAGS files are the only
or the "best" indexing feature.  It would be silly to do
so.  They, like Imenu, remain a useful feature for Emacs.

And it is unfair, I think, to point to current deficiencies
in support for a language as proof, by itself, that the Emacs
TAGS feature is problematic for that language.

There can be other ways in which it is not ideal, but current
lack of someone having written support for this or that language
is not, in itself, a reason.  The same holds for Imenu.

There are no doubt languages for which TAGS or Imenu is not
sufficient.  But just because a given language currently has
no support for creating a TAGS file or an Imenu menu is not a
sufficient reason for concluding that TAGS or Imenu is
inadequate for that language.  Some other, specific reasons
need to be given (your mention of methods, for example).

Re: IDE

[Eli Zaretskii]

> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Sat, 24 Oct 2015 21:15:18 +0300
> Cc: nix@esperi.org.uk, esperanto@cumego.com, adatgyujto@gmail.com,
> 	drew.adams@oracle.com, emacs-devel@gnu.org
> 
> > Once again, the issue is not the format of the database.  That's
> > immaterial.
> 
> Database format can have a real performance impact.

Yes, but the issue is performance, not the database format.

Re: IDE

[Dmitry Gutov]

On 10/24/2015 09:59 PM, Eli Zaretskii wrote:

>> Database format can have a real performance impact.
>
> Yes, but the issue is performance, not the database format.

In a situation like that, someone else might have called lower 
performance a "symptom", and not "the issue".

Re: IDE

[Andreas Röhler]

  On 18.10.2015 07:23, John Wiegley wrote:
>>>>>> Eli Zaretskii<eliz@gnu.org>  writes:
>> I'm quite sure CEDET has collected and expressed in code a lot of experience
>> and solutions to many problems that arise in the context of building an IDE.
>> It's OK to discard that, if we sure that's the proverbial 1st variant
>> everyone throws away, but we need first to be sure we know what we are
>> discarding.
> I'm not suggesting we discard experiences. What I'm saying is: it doesn't make
> sense to proceed by looking at CEDET, and then asking what should be changed.
>
> CEDET is like a hammer. When it was made, the problem looked like nail.
>
> Today, the problem might be a screw (is it? do we know?). We're not going to
> arrive at the best answer by asking ourselves how a hammer can be changed to
> meet the needs of a screw. It deserves looking at the problem anew.
>
> It doesn't mean we throw out the hammer. Maybe we do have a nail, maybe we
> don't. The point is: If we make technical assumptions before learning what we
> want to end up with, we're going to arrive at something shaped more by those
> assumptions than by our needs.
>
> So unless there are other features I should bear in mind, I'm going to turn my
> attention away from CEDET now and back to the IDE vision I'd like everyone's
> help with, once there is more to say.
>
> John
>

"hammer" seems a suitable abstraction for the moment :)

IMO the difficulty is twofold:

- CEDET tries to achieve things, which require detailed knowledge of the 
goal-languages - hardly to expect WRT several hundreds in use meanwhile.

- it's not written in plain Emacs Lisp, thus extending CEDET requires 
another special knowledge, which makes shrink the circle of would-be 
developers again.

Re: IDE

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

How about making a separate thread for this discussion of tags files
and the like.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: IDE

[Steinar Bang]

>>>>> Eli Zaretskii <eliz@gnu.org>:

> An index where the key is the symbol itself can only hold one instance
> of every symbol.

This will come as a shock to all the search engines out there.

(unless you were talking about TAGS files specifically...?)

Re: IDE

[Eli Zaretskii]

> From: Steinar Bang <sb@dod.no>
> Date: Mon, 26 Oct 2015 18:32:36 +0100
> 
> >>>>> Eli Zaretskii <eliz@gnu.org>:
> 
> > An index where the key is the symbol itself can only hold one instance
> > of every symbol.
> 
> This will come as a shock to all the search engines out there.
> 
> (unless you were talking about TAGS files specifically...?)

Of course, I was.  That's what this thread is about.

Re: IDE

[Andreas Schwab]

Eli Zaretskii <eliz@gnu.org> writes:

>> >>>>> Eli Zaretskii <eliz@gnu.org>:
>> 
>> > An index where the key is the symbol itself can only hold one instance
>> > of every symbol.
>> 
>> This will come as a shock to all the search engines out there.
>> 
>> (unless you were talking about TAGS files specifically...?)
>
> Of course, I was.  That's what this thread is about.

But TAGS files can handle multiple values for the same key just well.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: IDE

[Eli Zaretskii]

> From: Andreas Schwab <schwab@linux-m68k.org>
> Date: Mon, 26 Oct 2015 21:04:15 +0100
> Cc: Steinar Bang <sb@dod.no>, emacs-devel@gnu.org
> 
> TAGS files can handle multiple values for the same key just well.

If they do, they will not be able to distinguish between the
definition and the references.  All the values are equal.

Re: IDE

[Óscar Fuentes]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Andreas Schwab <schwab@linux-m68k.org>
>> Date: Mon, 26 Oct 2015 21:04:15 +0100
>> Cc: Steinar Bang <sb@dod.no>, emacs-devel@gnu.org
>> 
>> TAGS files can handle multiple values for the same key just well.
>
> If they do, they will not be able to distinguish between the
> definition and the references.  All the values are equal.

I use etags-select for displaying multiple entries for the same key. The
line corresponding to each value is shown, so you can locate what you
want reasonably fast. An example:

Finding tag: push-back

In: /home/oscar/dev/idb/lp0/lib/prelude.lp0
1 [push-back] (defmacro push-back (v a b &rest)
2 [push-back] (defun push-back (v e)

In: /home/oscar/dev/idb/lp0/lib/list.lp0
3 [push-back] (defun push-back (list e)
4 [push-back] (defun push-back (list)

Re: IDE

[Dmitry Gutov]

On 10/26/2015 10:27 PM, Óscar Fuentes wrote:

> I use etags-select for displaying multiple entries for the same key. The
> line corresponding to each value is shown, so you can locate what you
> want reasonably fast.

That still means that the tool you are using doesn't know the kinds of 
these references.

And the subject of this thread is "IDE". Modern IDEs do know stuff.

Re: IDE

[Eli Zaretskii]

> From: Óscar Fuentes <ofv@wanadoo.es>
> Date: Mon, 26 Oct 2015 21:27:56 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> From: Andreas Schwab <schwab@linux-m68k.org>
> >> Date: Mon, 26 Oct 2015 21:04:15 +0100
> >> Cc: Steinar Bang <sb@dod.no>, emacs-devel@gnu.org
> >> 
> >> TAGS files can handle multiple values for the same key just well.
> >
> > If they do, they will not be able to distinguish between the
> > definition and the references.  All the values are equal.
> 
> I use etags-select for displaying multiple entries for the same key. The
> line corresponding to each value is shown, so you can locate what you
> want reasonably fast.

How fast it is depends on the number of references.  It is not
reasonable to ask the user to examine all the references one by one in
order to find the single definition.  "Go to definition" should be
instantaneous, if we want to satisfy users.

References is another matter: there users expect a list from which to
choose.

> An example:
> 
> Finding tag: push-back
> 
> In: /home/oscar/dev/idb/lp0/lib/prelude.lp0
> 1 [push-back] (defmacro push-back (v a b &rest)
> 2 [push-back] (defun push-back (v e)
> 
> In: /home/oscar/dev/idb/lp0/lib/list.lp0
> 3 [push-back] (defun push-back (list e)
> 4 [push-back] (defun push-back (list)

Your example includes only definitions, and a small number at that.
It doesn't scale.

Re: IDE

[Andreas Schwab]

Eli Zaretskii <eliz@gnu.org> writes:

> If they do, they will not be able to distinguish between the
> definition and the references.  All the values are equal.

A TAGS file does not know anything.  It is just an index, whose
interpretation is up to the application.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: IDE

[Óscar Fuentes]

Dmitry Gutov <dgutov@yandex.ru> writes:

> On 10/26/2015 10:27 PM, Óscar Fuentes wrote:
>
>> I use etags-select for displaying multiple entries for the same key. The
>> line corresponding to each value is shown, so you can locate what you
>> want reasonably fast.
>
> That still means that the tool you are using doesn't know the kinds of
> these references.
>
> And the subject of this thread is "IDE". Modern IDEs do know stuff.

I was addressing the specific sub-topic about TAGS.

For knowing stuff you need tools which are as sophisticated as the
language they are working with, and there is no medium term solution for
that requirement as far as Emacs core is concerned on the C++ realm,
probably Java as well. There are external Emacs packages that are on
track for solving this problem, and an increasing number of features are
being implemented around those external packages. That makes this
discussion about IDEs on Emacs core to look like idle chatting.

Re: IDE

[Óscar Fuentes]

Eli Zaretskii <eliz@gnu.org> writes:

>> >> TAGS files can handle multiple values for the same key just well.
>> >
>> > If they do, they will not be able to distinguish between the
>> > definition and the references.  All the values are equal.
>> 
>> I use etags-select for displaying multiple entries for the same key. The
>> line corresponding to each value is shown, so you can locate what you
>> want reasonably fast.
>
> How fast it is depends on the number of references.  It is not
> reasonable to ask the user to examine all the references one by one in
> order to find the single definition.  "Go to definition" should be
> instantaneous, if we want to satisfy users.

My example shows multiple definitions of the same key taken from a TAGS
file, which proves that you can store there multiple values for the same
key and solve a real problem.

> References is another matter: there users expect a list from which to
> choose.

For references, TAGS is of little help. Grep is probably better.
Obviously, pointing to a data member or function and asking for the
references to *that* data member or function is something that is beyond
what Emacs core will provide on a realistic timeframe. People interested
on adding that feature to Emacs core would be more productive directing
their effort to GCC than to Emacs.

Re: IDE

[Dmitry Gutov]

On 10/27/2015 12:09 AM, Óscar Fuentes wrote:

>> And the subject of this thread is "IDE". Modern IDEs do know stuff.
>
> I was addressing the specific sub-topic about TAGS.

That subtopic started exactly with complaint that TAGS only allow for 
"jump to definition". Then someone argued that no, you can store 
anything what you like in them, we discussed that a bit, and here you've 
come to repeat the beginning: "For references, TAGS is of little help".

> For knowing stuff you need tools which are as sophisticated as the
> language they are working with, and there is no medium term solution for
> that requirement as far as Emacs core is concerned on the C++ realm,
> probably Java as well.

There can be some middle ground. From what I've seen of GNU Global 
(admittedly not too much), it's better for both "find definitions" and 
"find references" than TAGS.

Even in C++ and Java it's not too hard to implement a reasonably 
accurate parser that will index definitions. Recognizing references 
accurately is more of a problem (for now we indeed use Grep or similar 
tools).

> There are external Emacs packages that are on
> track for solving this problem, and an increasing number of features are
> being implemented around those external packages.

What Emacs can do is provide a common interface those external packages 
to hook into. Like progmodes/xref.el, for example.

> That makes this
> discussion about IDEs on Emacs core to look like idle chatting.

It is idle chatting, for the most part. No discussions of the code in 
Emacs master, no patches, etc.

Re: IDE

[Óscar Fuentes]

Dmitry Gutov <dgutov@yandex.ru> writes:

> On 10/27/2015 12:09 AM, Óscar Fuentes wrote:
>
>>> And the subject of this thread is "IDE". Modern IDEs do know stuff.
>>
>> I was addressing the specific sub-topic about TAGS.
>
> That subtopic started exactly with complaint that TAGS only allow for
> "jump to definition". Then someone argued that no, you can store
> anything what you like in them, we discussed that a bit, and here
> you've come to repeat the beginning: "For references, TAGS is of
> little help".

Sorry, I jumped into the middle of the thread.

>> For knowing stuff you need tools which are as sophisticated as the
>> language they are working with, and there is no medium term solution for
>> that requirement as far as Emacs core is concerned on the C++ realm,
>> probably Java as well.
>
> There can be some middle ground.

If by "middle ground" you refer to something that gives the right result
90% of the time when there are external packages that gives the right
result 100% of the time, that middle ground is a waste of time by the
Emacs core hackers.

> From what I've seen of GNU Global
> (admittedly not too much), it's better for both "find definitions" and
> "find references" than TAGS.
>
> Even in C++ and Java it's not too hard to implement a reasonably
> accurate parser that will index definitions. Recognizing references
> accurately is more of a problem (for now we indeed use Grep or similar
> tools).

Definitions can be tricky in C++, if you wish to distinguish the case
where multiple namespaces or classes defines the same keyword and you
expect from Emacs to jump to the correct definition, deduces from the
context. In that case you need a parser that is good enough to act as a
compiler front-end.

>> There are external Emacs packages that are on
>> track for solving this problem, and an increasing number of features are
>> being implemented around those external packages.
>
> What Emacs can do is provide a common interface those external
> packages to hook into. Like progmodes/xref.el, for example.

Trying to find a common ground on current use cases is difficult enough.
Anticipating future requirements is almost impossible. Good luck with
that.

[snip]

Re: IDE

[Dmitry Gutov]

On 10/27/2015 01:06 AM, Óscar Fuentes wrote:

> If by "middle ground" you refer to something that gives the right result
> 90% of the time when there are external packages that gives the right
> result 100% of the time, that middle ground is a waste of time by the
> Emacs core hackers.

A Grep-based implementation is fairly easy to do, so not too much time 
is wasted. On the other hand, it can exercise the new APIs and provide 
the necessary feedback.

> Definitions can be tricky in C++, if you wish to distinguish the case
> where multiple namespaces or classes defines the same keyword and you
> expect from Emacs to jump to the correct definition, deduces from the
> context. In that case you need a parser that is good enough to act as a
> compiler front-end.

"jump to definition" is about accurately recognizing references. But if 
you're jumping to foo.bar(), and Emacs doesn't know the type of 'foo', 
at least it can show you the definitions of all methods named 'bar', and 
you'll be able to choose for yourself. Using etags-select, for instance.

> Trying to find a common ground on current use cases is difficult enough.
> Anticipating future requirements is almost impossible. Good luck with
> that.

That's defeatism. Doesn't it bother you that every third-party package 
uses a (sometimes subtly) different set of key bindings, and a different 
way to present the same kinds of information (definitions, references, 
documentation, etc)?

Re: IDE

[Óscar Fuentes]

Dmitry Gutov <dgutov@yandex.ru> writes:

[snip]

>> Definitions can be tricky in C++, if you wish to distinguish the case
>> where multiple namespaces or classes defines the same keyword and you
>> expect from Emacs to jump to the correct definition, deduces from the
>> context. In that case you need a parser that is good enough to act as a
>> compiler front-end.
>
> "jump to definition" is about accurately recognizing references. But
> if you're jumping to foo.bar(), and Emacs doesn't know the type of
> 'foo', at least it can show you the definitions of all methods named
> 'bar', and you'll be able to choose for yourself. Using etags-select,
> for instance.

For not-so-big C++ projects, that can be as useless as displaying all
the uses of "bar" on the code base.

>> Trying to find a common ground on current use cases is difficult enough.
>> Anticipating future requirements is almost impossible. Good luck with
>> that.
>
> That's defeatism. Doesn't it bother you that every third-party package
> uses a (sometimes subtly) different set of key bindings, and a
> different way to present the same kinds of information (definitions,
> references, documentation, etc)?

If there is a common pattern, by all means, implement it (on a way that
provides for those subtle differences when they make sense.) But those
packages still need to maintain compatibility with older Emacsen.
Distributing the library on GNU ELPA instead of the Emacs core would
make things better for the external packages, though. But then, what's
the difference from developing the library outside of Emacs core and
working directly with the authors of those external packages? If the
library or API is developed on emacs-devel, is it acceptable to design
it by the requirements of rtags, for instance?

Re: IDE

[Dmitry Gutov]

On 10/27/2015 01:40 AM, Óscar Fuentes wrote:

> For not-so-big C++ projects, that can be as useless as displaying all
> the uses of "bar" on the code base.

Technically, it must be at least an order-of-magnitude better.

And there are other languages that we want to support, too. It's not 
just C++ out there.

> If there is a common pattern, by all means, implement it (on a way that
> provides for those subtle differences when they make sense.)

This kind of feature requests will have to be more detailed.

> But those
> packages still need to maintain compatibility with older Emacsen.

Not necessarily: they can have some duplication in the code, and work 
both ways.

> Distributing the library on GNU ELPA instead of the Emacs core would
> make things better for the external packages, though.

That's one solution, yes.

> But then, what's
> the difference from developing the library outside of Emacs core and
> working directly with the authors of those external packages?

GNU ELPA is still a part of Emacs core, organizationally. Anyway, I 
don't understand the question.

> If the
> library or API is developed on emacs-devel, is it acceptable to design
> it by the requirements of rtags, for instance?

Why not? As long as it's not *just* the requirements of rtags that are 
considered.

Re: IDE

[Eric Ludlam]

On 10/26/2015 07:06 PM, Óscar Fuentes wrote:
>>> There are external Emacs packages that are on
>>> >>track for solving this problem, and an increasing number of features are
>>> >>being implemented around those external packages.
>> >
>> >What Emacs can do is provide a common interface those external
>> >packages to hook into. Like progmodes/xref.el, for example.
 >
> Trying to find a common ground on current use cases is difficult enough.
> Anticipating future requirements is almost impossible. Good luck with
> that.

CEDET/Semantic already does this.  It can use itself, Global, idutils, 
or cscope and convert the output into a common semantic tag 
infrastructure.  It has a common searching mechanism so you just write 
one bit of code to find the symbol you want (via semanticdb) or 
references you want (via semantic-symref) and it will work fine no 
matter how the user may have set it up.

While there has certainly been debate here about if people should be 
writing parsers in elisp and how smart its smart completion is, but the 
above interfaces are indeed generic, rich, and provides the raw answers 
from those external tools with a common output format.

Eric

Re: IDE

[Nikolaus Rath]

On Oct 26 2015, Eric Ludlam <eric@siege-engine.com> wrote:
> On 10/26/2015 07:06 PM, Óscar Fuentes wrote:
>>>> There are external Emacs packages that are on
>>>> >>track for solving this problem, and an increasing number of features are
>>>> >>being implemented around those external packages.
>>> >
>>> >What Emacs can do is provide a common interface those external
>>> >packages to hook into. Like progmodes/xref.el, for example.
>>
>> Trying to find a common ground on current use cases is difficult enough.
>> Anticipating future requirements is almost impossible. Good luck with
>> that.
>
> CEDET/Semantic already does this.  It can use itself, Global, idutils,
> or cscope and convert the output into a common semantic tag
> infrastructure.  It has a common searching mechanism so you just write
> one bit of code to find the symbol you want (via semanticdb) or
> references you want (via semantic-symref) and it will work fine no
> matter how the user may have set it up.

Unfortunately, at least for me the "one bit of code" was not at all
obvious after reading the CEDET documentation. So while I believe that
any of Global/idutils/cscope would be good enough for the majority of my
use-cases, I wasn't able to make CEDET use any of them (or maybe CEDET
uses them, but I'm not actually invoking cedet with M-.?).

My impression that CEDET/Semantic doesn't lack functionality, but
end-user documentation.


Best,
-Nikolauss
-- 
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

             »Time flies like an arrow, fruit flies like a Banana.«

Re: IDE

[Eli Zaretskii]

> From: Andreas Schwab <schwab@linux-m68k.org>
> Cc: sb@dod.no,  emacs-devel@gnu.org
> Date: Mon, 26 Oct 2015 22:14:41 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > If they do, they will not be able to distinguish between the
> > definition and the references.  All the values are equal.
> 
> A TAGS file does not know anything.  It is just an index, whose
> interpretation is up to the application.

The application just interprets what's in the file.  If the file
doesn't provide any indication that one of the entries is a
definition, the application will not know which one is.

Re: IDE

[Eli Zaretskii]

> From: Óscar Fuentes <ofv@wanadoo.es>
> Date: Mon, 26 Oct 2015 23:16:38 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> >> TAGS files can handle multiple values for the same key just well.
> >> >
> >> > If they do, they will not be able to distinguish between the
> >> > definition and the references.  All the values are equal.
> >> 
> >> I use etags-select for displaying multiple entries for the same key. The
> >> line corresponding to each value is shown, so you can locate what you
> >> want reasonably fast.
> >
> > How fast it is depends on the number of references.  It is not
> > reasonable to ask the user to examine all the references one by one in
> > order to find the single definition.  "Go to definition" should be
> > instantaneous, if we want to satisfy users.
> 
> My example shows multiple definitions of the same key taken from a TAGS
> file, which proves that you can store there multiple values for the same
> key and solve a real problem.

You are taking this sub-thread out of context.  Its context is that
TAGS cannot support both definitions and references because there's no
indication which one is which.

> > References is another matter: there users expect a list from which to
> > choose.
> 
> For references, TAGS is of little help. Grep is probably better.

ID-Utils is even better.  Etc., etc.  And that's exactly what this
sub-thread is about.

> Obviously, pointing to a data member or function and asking for the
> references to *that* data member or function is something that is beyond
> what Emacs core will provide on a realistic timeframe. People interested
> on adding that feature to Emacs core would be more productive directing
> their effort to GCC than to Emacs.

We are not talking about the Emacs core here (which does know how to
do what you say).  We are talking about TAGS and the functionality
that can be built only based on TAGS, without any other databases.

Re: IDE

[Eli Zaretskii]

> From: Nikolaus Rath <Nikolaus@rath.org>
> Date: Mon, 26 Oct 2015 20:01:06 -0700
> 
> My impression that CEDET/Semantic doesn't lack functionality, but
> end-user documentation.

Detailed bug reports about missing or incomplete CEDET documentation
are welcome.  Patches to improve that documentation are even more
welcome.

Re: IDE

[Nikolaus Rath]

On Oct 27 2015, Eli Zaretskii <eliz@gnu.org> wrote:
>> From: Nikolaus Rath <Nikolaus@rath.org>
>> Date: Mon, 26 Oct 2015 20:01:06 -0700
>> 
>> My impression that CEDET/Semantic doesn't lack functionality, but
>> end-user documentation.
>
> Detailed bug reports about missing or incomplete CEDET documentation
> are welcome.

It's hard to provide details about undocumented functionality, if one
doesn't know which functionality the code provides (since it's not
documented).

The best I can do is this: judging from this thread, Semantic offers a
lot of things that are not enabled when one just follows the "Using
Semantic" section of the info manual. There should be documentation
about how to enable and use those features.

> Patches to improve that documentation are even more
> welcome.

Naturally.


Best,
-Nikolaus

-- 
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

             »Time flies like an arrow, fruit flies like a Banana.«

Re: IDE

[Marcus Harnisch]

Eli Zaretskii <eliz@gnu.org> writes:

> "Definition" in this context means the implementation.  There's only
> one implementation, [...]

FWIW, in my day job I am spending most of my time in an aspect
oriented language. While there is an identifiable “master” definition,
having all aspects show up as definitions is far more useful. In many
cases (e.g vendor libraries) the “master” definitions are invisible,
yet I want to be able to refer to my own extensions as definition
points.

Cheers
Marcus

Re: IDE

[Oleh Krehel]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Dmitry Gutov <dgutov@yandex.ru>
>> Date: Sat, 24 Oct 2015 21:15:18 +0300
>> Cc: nix@esperi.org.uk, esperanto@cumego.com, adatgyujto@gmail.com,
>> 	drew.adams@oracle.com, emacs-devel@gnu.org
>> 
>> > Once again, the issue is not the format of the database.  That's
>> > immaterial.
>> 
>> Database format can have a real performance impact.
>
> Yes, but the issue is performance, not the database format.

If I understood correctly what you mean by the database format, it
matters to me. The TAGS files are too simplistic, they don't understand
the language, either C and especially C++.  On the other hand, have a
look that these Semantic tags entries for e.g. etags.c:

("regex.h" include (:system-flag t) nil [4626 4644])
("CTAGS" variable (:constant-flag t) nil [4934 4939])
("streq" function
         (:typemodifiers ("static")
                         :arguments 
                         ( ("s" variable
                                (:pointer 1
                                          :constant-flag t
                                          :type "char")
                                (reparse-symbol arg-sub-list) [4973 4987])
                           ("t" variable
                                (:pointer 1
                                          :constant-flag t
                                          :type "char")
                                (reparse-symbol arg-sub-list) [4988 5002]))                  
                         :type "bool")
         nil [4954 5035])

This is a lot of useful information in a readable and usable format.
The only problem is that it's a little slow to parse: 190 files in
emacs/src take around 30 seconds for a full reparse. But then, all this
info is kept and is re-parsed only on timestamp changes.

I did a caching optimization for `moo-jump-local' from function-args
package. When called without update it takes <0.1s to bring up all 19356
semantic tags. The update (through a call with "C-u") takes ~3 seconds +
reparse time for any out-of-date file.

My point is that because `moo-jump-local' uses semantic, it's a lot more
precise than e.g. `ggtags-find-definition' that gives only the names of
9956 tags, with no semantic information.

Compare:

    MAX_PARAGRAPH_SEARCH
    x_parse_color

to:

#include <dispextern.h>                                                                           xterm.h
#include <termhooks.h>                                                                            xterm.h
#define BLACK_PIX_DEFAULT                                                                         xterm.h
#define WHITE_PIX_DEFAULT                                                                         xterm.h
#define STANDARD_EVENT_SET                                                                        xterm.h
x_bitmap_record                                                                                   xterm.h
  Pixmap pixmap                                                                                   xterm.h
  bool have_mask                                                                                  xterm.h
  Pixmap mask                                                                                     xterm.h
  char* file                                                                                      xterm.h
  int refcount                                                                                    xterm.h
  int height                                                                                      xterm.h
  int width                                                                                       xterm.h
  int depth                                                                                       xterm.h
color_name_cache_entry                                                                            xterm.h
  color_name_cache_entry* next                                                                    xterm.h
  XColor rgb                                                                                      xterm.h
  char* name                                                                                      xterm.h
Status x_parse_color (frame *f, const char *color_name, XColor *color);                           xterm.h

In my opinion, the tags format of semantic is very good, much better
than plain TAGS. Maybe some work needs to be done to make them generate
faster/more precise, e.g. make GCC output these tags files.
    

Re: IDE

[Óscar Fuentes]

Eli Zaretskii <eliz@gnu.org> writes:

[snip]

>> Obviously, pointing to a data member or function and asking for the
>> references to *that* data member or function is something that is beyond
>> what Emacs core will provide on a realistic timeframe. People interested
>> on adding that feature to Emacs core would be more productive directing
>> their effort to GCC than to Emacs.
>
> We are not talking about the Emacs core here (which does know how to
> do what you say).

Does it? AFAIK it has a framework that does that as long as something
else provides the required information. This does not qualify as
*knowing* how to do the necessary inferences.

I mention this just in case some user might get the impression that
Emacs has that feature.

[snip]

Re: IDE

[Steinar Bang]

>>>>> Dmitry Gutov <dgutov@yandex.ru>:

> Consider A. It could be considered one project, but then it would have
> certain attributes dependent on the current directory. Take classpath,
> for example.

(Classpath in maven is actually independent of the directory layout of
the checked out projects, but I won't get into that now)

> Could we consider it to be constant value across the whole project, or
> would we have, for certain operations (like "find the class named
> foo.bar.Bar"), different values of classpath in module1 and module2?

The classpath will be per project/module, and for each project it can
differ for compilation, runtime, test execution.

A *brief* explanation of how the class path is built (for the initiated:
by "identity" below, I mean the "maven coordinates" consisting of
groupId/artifactId/version):
 - Each pom.xml file, have:
  - an identity
  - the identity of a parent pom.xml file
  - a list of dependencies (including the scope of the dependency,
    eg. "test")
 - The combined dependency list of a pom.xml is:
  - The dependencies in the dependency list
  - The transitive dependencies of the dependencies in the dependency
    list
  - The combined dependencies of the parent pom.xml (which includes that
    pom.xml file's dependencies and transitive dependencies and those of
    its parent in turn)
 - Maven will download all dependencies and install them in the maven
   local cache (by default the maven local cache resides in
   $HOME/.m2/repository/ )
 - Maven will create the appropriate classpath for compile, test, run,
   etc. consisting of jar files in the maven local cache

The contents of the maven local cache can be both downloaded
dependencies and artifacts (e.g. jar files) built by other local maven
projects.

(That means that as long as the projects are built in dependency order,
they can in *theory* be be checked out separately and built separately,
parent, and dependencies to other projects can be resolved against the
local cache.  In practice, however, it is simpler to organize the maven
project in the recommended hierarchy and let maven handle built order
and stuff)

> The build-tool behavior would certainly depend on the current
> directory, but could we say that all other important project
> attributes are kind of the same for project, module1 and module2?

Depends on what you mean with important project attributes...?

Dependencies are resolved individually for module1 and module2, but
there can be a common set of dependencies in project (provided "project"
is the parent pom.xml of "module1" and "module2").

If "project" is the parent pom.xml of the two modules, you can set
properties in "project" that are shared by module1 and module2.  You can
also share plugin configurations in this way.

> Another option for A is to promote module1 and module2 to whole
> projects. But then, do we track project dependencies now?

The same way eclipse maven support ("m2e") does, perhaps...?
Even projects that should have been in a hierarchy are checked out
separately, and if "module1" is a dependency of "module2",

Ie. if we have only
 workspace/
   module1/
     pom.xml
     .git/
   module2/
     pom.xml
     .git/

eclipse m2e will still recognize that "module1" contains the dependency
of "module2" and build them in order (they will show up as project
dependencies in the project classpath in eclipse).

This is because each pom.xml file has a (hopefully) unique identity
consisting of groupId/artifactId/version.

> If I call a command "search for occurrences of 'xyz' in project", does
> it also search in module1 and module2? Or similarly for "list all
> files in project", does that include files in module1?

"project" is defined by the existence of a pom.xml file, for both
eclipse and maven, so "project" in eclipse would mean either "module1"
and "module2".

What it would mean if "project" was included into the workspace is a
little more unclear (there is an "enclosing projects" option in eclipse
search, but I don't know what this means").

The default search scope in eclipse is "workspace", which is the
projects seen in eclipse's package explorer.

> That would be useful with Helm, for "jump to a project file". Does
> "list all files in project", when called in module1, include files
> from the parent project, and from module2?

> B doesn't look too different, except we apparently don't have a
> top-level pom-file.

In this case, if the top level pom file isn't a parent of either module
it doesn't have to be there (it could be a build-only file).  If the top
level pom _is_ a dependency of the modules, it could be resolved against
the maven local cache (if not found there, the build would break).

> I don't understand C. Is module1 still inside project? Is it still a
> dependency? Do we treat it differently WRT to questions I've asked for
> the option A?

In eclipse, C would be treated identical to A (and look the same in a
workspace), but A would build from the command line, and C would not
(actually: would be cumbersome to build).

So C is just a theoretical possibility, and could be disregarded (though
I have done similar things during early m2e experimentations).

Re: IDE

[Andreas Schwab]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Andreas Schwab <schwab@linux-m68k.org>
>> Cc: sb@dod.no,  emacs-devel@gnu.org
>> Date: Mon, 26 Oct 2015 22:14:41 +0100
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> > If they do, they will not be able to distinguish between the
>> > definition and the references.  All the values are equal.
>> 
>> A TAGS file does not know anything.  It is just an index, whose
>> interpretation is up to the application.
>
> The application just interprets what's in the file.  If the file
> doesn't provide any indication that one of the entries is a
> definition, the application will not know which one is.

There are producers and consumers.  They just have to agree on the
format.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: IDE

[Eli Zaretskii]

> From: Nikolaus Rath <Nikolaus@rath.org>
> Cc: emacs-devel@gnu.org
> Date: Mon, 26 Oct 2015 21:02:43 -0700
> 
> On Oct 27 2015, Eli Zaretskii <eliz@gnu.org> wrote:
> >> From: Nikolaus Rath <Nikolaus@rath.org>
> >> Date: Mon, 26 Oct 2015 20:01:06 -0700
> >> 
> >> My impression that CEDET/Semantic doesn't lack functionality, but
> >> end-user documentation.
> >
> > Detailed bug reports about missing or incomplete CEDET documentation
> > are welcome.
> 
> It's hard to provide details about undocumented functionality, if one
> doesn't know which functionality the code provides (since it's not
> documented).

I guess I've misunderstood you, sorry.  You said:

> Unfortunately, at least for me the "one bit of code" was not at all
> obvious after reading the CEDET documentation. So while I believe that
> any of Global/idutils/cscope would be good enough for the majority of my
> use-cases, I wasn't able to make CEDET use any of them (or maybe CEDET
> uses them, but I'm not actually invoking cedet with M-.?).

My interpretation of that, perhaps incorrect, was that you tried to
use some functionality and tried to find documentation about it, but
either the documentation was inadequate or completely missing.  Based
on that interpretation, I suggested that you report what you tried to
figure out from the documentation, but couldn't.

> The best I can do is this: judging from this thread, Semantic offers a
> lot of things that are not enabled when one just follows the "Using
> Semantic" section of the info manual. There should be documentation
> about how to enable and use those features.

A list if such issues that lack proper documentation would also be
useful, if you cannot provide a more detailed information.  TIA.

Re: IDE

[Eli Zaretskii]

> From: Oleh Krehel <ohwoeowho@gmail.com>
> Cc: Dmitry Gutov <dgutov@yandex.ru>,  nix@esperi.org.uk,  esperanto@cumego.com,  adatgyujto@gmail.com,  drew.adams@oracle.com,  emacs-devel@gnu.org
> Date: Tue, 27 Oct 2015 09:21:40 +0100
> 
> >> > Once again, the issue is not the format of the database.  That's
> >> > immaterial.
> >> 
> >> Database format can have a real performance impact.
> >
> > Yes, but the issue is performance, not the database format.
> 
> If I understood correctly what you mean by the database format, it
> matters to me. The TAGS files are too simplistic, they don't understand
> the language, either C and especially C++.  On the other hand, have a
> look that these Semantic tags entries for e.g. etags.c:
> 
> ("regex.h" include (:system-flag t) nil [4626 4644])
> ("CTAGS" variable (:constant-flag t) nil [4934 4939])
> ("streq" function
>          (:typemodifiers ("static")
>                          :arguments 
>                          ( ("s" variable
>                                 (:pointer 1
>                                           :constant-flag t
>                                           :type "char")
>                                 (reparse-symbol arg-sub-list) [4973 4987])
>                            ("t" variable
>                                 (:pointer 1
>                                           :constant-flag t
>                                           :type "char")
>                                 (reparse-symbol arg-sub-list) [4988 5002]))                  
>                          :type "bool")
>          nil [4954 5035])
> 
> This is a lot of useful information in a readable and usable format.
> The only problem is that it's a little slow to parse: 190 files in
> emacs/src take around 30 seconds for a full reparse. But then, all this
> info is kept and is re-parsed only on timestamp changes.
> 
> I did a caching optimization for `moo-jump-local' from function-args
> package. When called without update it takes <0.1s to bring up all 19356
> semantic tags. The update (through a call with "C-u") takes ~3 seconds +
> reparse time for any out-of-date file.
> 
> My point is that because `moo-jump-local' uses semantic, it's a lot more
> precise than e.g. `ggtags-find-definition' that gives only the names of
> 9956 tags, with no semantic information.

The point I was trying to make is that for users all this is
unimportant.  All they want is functionality: performance, accuracy,
minimum false positives, etc.  How this affects the database format is
something they expect us the developers to figure out.

And even if you take the POV of a developer, first there should be
requirements: performance, accuracy, supported languages, etc.  Only
after that we get to design and implementation, where we select the
database which can enable the functionality and support the
requirements.  Note as you explain above why TAGS might not be
appropriate _because_ it cannot support certain important
functionality:

> In my opinion, the tags format of semantic is very good, much better
> than plain TAGS. Maybe some work needs to be done to make them generate
> faster/more precise, e.g. make GCC output these tags files.

And that's my point exactly: first there's functionality we want, and
only after that comes the selection of the database.

Re: IDE

[Eli Zaretskii]

> From: Óscar Fuentes <ofv@wanadoo.es>
> Date: Tue, 27 Oct 2015 13:24:37 +0100
> 
> >> Obviously, pointing to a data member or function and asking for the
> >> references to *that* data member or function is something that is beyond
> >> what Emacs core will provide on a realistic timeframe. People interested
> >> on adding that feature to Emacs core would be more productive directing
> >> their effort to GCC than to Emacs.
> >
> > We are not talking about the Emacs core here (which does know how to
> > do what you say).
> 
> Does it? AFAIK it has a framework that does that as long as something
> else provides the required information. This does not qualify as
> *knowing* how to do the necessary inferences.

The framework has a couple of back-ends that enable this.

> I mention this just in case some user might get the impression that
> Emacs has that feature.

It does.

Re: IDE

[Eli Zaretskii]

> From: Andreas Schwab <schwab@linux-m68k.org>
> Cc: sb@dod.no,  emacs-devel@gnu.org
> Date: Tue, 27 Oct 2015 18:39:17 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> From: Andreas Schwab <schwab@linux-m68k.org>
> >> Cc: sb@dod.no,  emacs-devel@gnu.org
> >> Date: Mon, 26 Oct 2015 22:14:41 +0100
> >> 
> >> Eli Zaretskii <eliz@gnu.org> writes:
> >> 
> >> > If they do, they will not be able to distinguish between the
> >> > definition and the references.  All the values are equal.
> >> 
> >> A TAGS file does not know anything.  It is just an index, whose
> >> interpretation is up to the application.
> >
> > The application just interprets what's in the file.  If the file
> > doesn't provide any indication that one of the entries is a
> > definition, the application will not know which one is.
> 
> There are producers and consumers.  They just have to agree on the
> format.

I was talking about the _existing_ format.  It goes without saying
that it can be (almost trivially) extended to support additional
functionality.  But then it won't be TAGS format, strictly speaking.

Re: IDE

[Óscar Fuentes]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Óscar Fuentes <ofv@wanadoo.es>
>> Date: Tue, 27 Oct 2015 13:24:37 +0100
>> 
>> >> Obviously, pointing to a data member or function and asking for the
>> >> references to *that* data member or function is something that is beyond
>> >> what Emacs core will provide on a realistic timeframe. People interested
>> >> on adding that feature to Emacs core would be more productive directing
>> >> their effort to GCC than to Emacs.
>> >
>> > We are not talking about the Emacs core here (which does know how to
>> > do what you say).
>> 
>> Does it? AFAIK it has a framework that does that as long as something
>> else provides the required information. This does not qualify as
>> *knowing* how to do the necessary inferences.
>
> The framework has a couple of back-ends that enable this.

Which ones?

>> I mention this just in case some user might get the impression that
>> Emacs has that feature.
>
> It does.

Maybe if you work on C and the expression at hand is not very
complicated. For C++, definitively not.

Re: IDE

[Nikolaus Rath]

On Oct 27 2015, Eli Zaretskii <eliz@gnu.org> wrote:
>> From: Nikolaus Rath <Nikolaus@rath.org>
>> Cc: emacs-devel@gnu.org
>> Date: Mon, 26 Oct 2015 21:02:43 -0700
>> 
>> On Oct 27 2015, Eli Zaretskii <eliz@gnu.org> wrote:
>> >> From: Nikolaus Rath <Nikolaus@rath.org>
>> >> Date: Mon, 26 Oct 2015 20:01:06 -0700
>> >> 
>> >> My impression that CEDET/Semantic doesn't lack functionality, but
>> >> end-user documentation.
>> >
>> > Detailed bug reports about missing or incomplete CEDET documentation
>> > are welcome.
>> 
>> It's hard to provide details about undocumented functionality, if one
>> doesn't know which functionality the code provides (since it's not
>> documented).
>
> I guess I've misunderstood you, sorry.  You said:
>
>> Unfortunately, at least for me the "one bit of code" was not at all
>> obvious after reading the CEDET documentation. So while I believe that
>> any of Global/idutils/cscope would be good enough for the majority of my
>> use-cases, I wasn't able to make CEDET use any of them (or maybe CEDET
>> uses them, but I'm not actually invoking cedet with M-.?).
>
> My interpretation of that, perhaps incorrect, was that you tried to
> use some functionality and tried to find documentation about it, but
> either the documentation was inadequate or completely missing.

Well, I followed the instructions in the manual but (as far as I can
tell) didn't get support for Global/idutils/cscope. At the time I didn't
suspect anything was wrong - until I read here that there ought to be
such support. Thus my conclusion that the "obvious bit of code" isn't
all that obvious (in both necessity and content).

> Based on that interpretation, I suggested that you report what you
> tried to figure out from the documentation, but couldn't.

I would have expected the documentation to tell me that I need to do
something in order to get Global et al support (and ideally also *what*
I need to do).


Best,
-Nikolaus

-- 
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

             »Time flies like an arrow, fruit flies like a Banana.«

Re: IDE

[Dmitry Gutov]

On 10/22/2015 03:58 PM, Eric Ludlam wrote:

> Semantic's idle process handling makes sure databases are synchronized
> once, then enables tools to run after, so it is more about the
> scheduling of different tools that use semantic.

Cool. But I guess we could just say I was thinking of 
'semantic-idle-summary-idle-function, and not semantic-idle-summary-mode 
itself.

> In addition, by going through the semantic version, there are a range of
> different formatting options to use that the user can select from.  That
> doesn't require idle-summary-mode, but is a side effect of using
> semantic to extract contextual information.

The formatting options depend on using some rich structure like Semantic 
tags. So it would make sense to port that only after we standardize on 
some tag format universally across Emacs (are Semantic tags optimal? I 
don't know).

> It was deemed optional when Yidong merged CEDET into Emacs.  You will
> probably need to use Emacs24 to make it work.  To try it out do this:

Step 6 is broken for me.

> 1) Install graphviz (it uses that for the layout engine)
> 2) Start emacs 24
> 3) Use CEDET from it's git repository
> 4) M-x find-library RET cogre RET
> 5) find cogre-element-peer in the code
> 6) M-x cogre-uml-quick-class RET

I only get a "Class:" prompt, without a default value or completions. 
Typing "cogre-element-peer" gives me "Could not find class ...", even 
though cogre is obviously loaded. That's in Emacs 24.5.

> When thinking about CEDET, it isn't about a
> bullet list of user facing features but about how it can enable someone
> working on said feature to have their work leveraged to a wider audience.

People working on said features should be encouraged, of course. 
Unfortunately, the two more interesting projects that I've seen utilize 
CEDET are language-specific:

- SRefactor only does anything useful for C/C++.
- Oleh Krehel's function-args even mentions C/C++ in its summary.

Perhaps, if there were more broadly applicable examples, it would lead 
to broader adoption. Maybe we should wonder why prefer making tools for 
CEDET that only target C and C++.

Re: IDE

[Dmitry Gutov]

On 10/21/2015 05:52 PM, Lluís wrote:

>> I don't understand C. Is module1 still inside project? Is it still a dependency?
>> Do we treat it differently WRT to questions I've asked for the option A?
>
> Ok, so what if we let project-types define project nesting?

Every new thing, like projects being allowed to have children (or 
modules; are modules different from projects?), or paths being possibly 
non-recursive, raises complexity of the API, and makes it less 
straightforward to use it.

That's why I asking questions: which commands people would want to see 
implemented, that would consume information about project structure, and 
how they would expect the said commands to behave WRT to nesting, 
submodules, etc.

For example, if when we're working on a submodule we don't *really* need 
to know that we're inside a bigger project (or at least don't need to 
impart that information to most project-related commands), we can avoid 
the notion of nesting in the API, and just ask any project 
implementation to return the "module" we're currently in as the current 
project.

And a lot of languages don't have the same kind of modules that 
Maven-based Java projects use. Would the notion 'children' be only 
useful for Java projects?

> In most cases, it probably makes more sense to construct this nesting
> programmatically by adding some logic during project auto-detection (e.g., read
> some configuration file that is part of the project).

Yes, of course. A project implementation like that would probably read 
pom.xml (or equivalent), and construct the nesting based on that.

Re: IDE

[Eric Ludlam]

On 10/27/2015 10:16 PM, Dmitry Gutov wrote:
> On 10/22/2015 03:58 PM, Eric Ludlam wrote:
>
>> In addition, by going through the semantic version, there are a range of
>> different formatting options to use that the user can select from.  That
>> doesn't require idle-summary-mode, but is a side effect of using
>> semantic to extract contextual information.
>
> The formatting options depend on using some rich structure like 
> Semantic tags. So it would make sense to port that only after we 
> standardize on some tag format universally across Emacs (are Semantic 
> tags optimal? I don't know).
>

What is optimal?  For me, it was about pulling all the datatype 
information out of the syntax, and differentiating between different tag 
classes, and having something that was Emacs friendly, and extensible to 
any language, no matter how strange.  For me, Semantic's tag structure 
is pretty good.  It has 5 slots:

* The name, as 'car' so you can use assoc and other friendly things.
* The class of the tag is next, as all languages have different kinds of 
tags.
* plist of attributes, extensible to use anything, but with some common 
attribute names defined for use via API.
* plist of properties, or bits of data needed by the tooling, which is 
arbitrary and extensible for whatever a tool might need.
* Buffer binding information - either the [start end] of the tag, or an 
overlay.

That makes the tag extensible for almost any situation, and keeps TAG 
data and OPERATIONAL data separate, saveable, and searchable.

>> It was deemed optional when Yidong merged CEDET into Emacs.  You will
>> probably need to use Emacs24 to make it work.  To try it out do this:
>
> Step 6 is broken for me.
>
>> 1) Install graphviz (it uses that for the layout engine)
>> 2) Start emacs 24
>> 3) Use CEDET from it's git repository
>> 4) M-x find-library RET cogre RET
>> 5) find cogre-element-peer in the code
>> 6) M-x cogre-uml-quick-class RET
>
> I only get a "Class:" prompt, without a default value or completions. 
> Typing "cogre-element-peer" gives me "Could not find class ...", even 
> though cogre is obviously loaded. That's in Emacs 24.5.

Did you enable semantic-mode?  The buffer would need to be parsed for 
the data to be available, and for the prompt to have completion.

>> When thinking about CEDET, it isn't about a
>> bullet list of user facing features but about how it can enable someone
>> working on said feature to have their work leveraged to a wider 
>> audience.
>
> People working on said features should be encouraged, of course. 
> Unfortunately, the two more interesting projects that I've seen 
> utilize CEDET are language-specific:
>
> - SRefactor only does anything useful for C/C++.
> - Oleh Krehel's function-args even mentions C/C++ in its summary.
>
> Perhaps, if there were more broadly applicable examples, it would lead 
> to broader adoption. Maybe we should wonder why prefer making tools 
> for CEDET that only target C and C++.
> .

This is a good point.  I always have thought about language independence 
and extensibility, but most folks are just trying to solve a problem and 
may not be interested in extending to anything else.

I've built many little tools over the years, but perhaps they are 
observed as parts of CEDET core, and not as examples.  The semantic-ia 
functions were supposed to be examples for how to use the 
infrastructure, but users have ended up using them as their interface 
and they became more complex.

Eric

Re: IDE

[Filipp Gunbin]

On 27/10/2015 18:28 +0100, Steinar Bang wrote:

>>>>>> Dmitry Gutov <dgutov@yandex.ru>:
>
>> Could we consider it to be constant value across the whole project, or
>> would we have, for certain operations (like "find the class named
>> foo.bar.Bar"), different values of classpath in module1 and module2?
>
> The classpath will be per project/module, and for each project it can
> differ for compilation, runtime, test execution.
>
> A *brief* explanation of how the class path is built (for the initiated:
> by "identity" below, I mean the "maven coordinates" consisting of
> groupId/artifactId/version):
>  - Each pom.xml file, have:
>   - an identity
>   - the identity of a parent pom.xml file
>   - a list of dependencies (including the scope of the dependency,
>     eg. "test")
>  - The combined dependency list of a pom.xml is:
>   - The dependencies in the dependency list
>   - The transitive dependencies of the dependencies in the dependency
>     list
>   - The combined dependencies of the parent pom.xml (which includes that
>     pom.xml file's dependencies and transitive dependencies and those of
>     its parent in turn)
>  - Maven will download all dependencies and install them in the maven
>    local cache (by default the maven local cache resides in
>    $HOME/.m2/repository/ )
>  - Maven will create the appropriate classpath for compile, test, run,
>    etc. consisting of jar files in the maven local cache
>

I've been following this long thread with interest and there's one
thing that keeps bothering me - should we duplicate the logic of build
tool in Emacs IDE-like features?

In my `javaimp' package (available in GNU Elpa) Emacs calls Maven to get
classpath, then scans all jars in it (it takes them from local Maven
cache and reads what classes are inside).  This is done only once given
that pom.xml doesn't change (and if it changes this is repeated).

This requires some time to wait for Maven to finish and output the
classpath.  But jar scanning is inevitable, I guess, and takes more time
(both are cached then).

I'm hoping to implement something like this for Gradle, but I didn't get
to it yet and I don't know whether it can output classpath like Maven
can.

Btw, there are subtleties between different Maven versions in dealing
with dependency merging when what is called an `effective pom' is built
(that is, when pom hierarchy is merged to produce effective settings for
the current module).  I remember that initially Maven 3 was outputting a
slightly wrong tree with `mvn dependency:tree' rather than it was
actually using during build.

This looks a lot like the GCC AST problem.

Filipp

Re: IDE

[Lluís]

Dmitry Gutov writes:

> On 10/21/2015 05:52 PM, Lluís wrote:
>>> I don't understand C. Is module1 still inside project? Is it still a dependency?
>>> Do we treat it differently WRT to questions I've asked for the option A?
>> 
>> Ok, so what if we let project-types define project nesting?

> Every new thing, like projects being allowed to have children (or modules; are
> modules different from projects?), or paths being possibly non-recursive, raises
> complexity of the API, and makes it less straightforward to use it.

> That's why I asking questions: which commands people would want to see
> implemented, that would consume information about project structure, and how
> they would expect the said commands to behave WRT to nesting, submodules, etc.

> For example, if when we're working on a submodule we don't *really* need to know
> that we're inside a bigger project (or at least don't need to impart that
> information to most project-related commands), we can avoid the notion of
> nesting in the API, and just ask any project implementation to return the
> "module" we're currently in as the current project.

> And a lot of languages don't have the same kind of modules that Maven-based Java
> projects use. Would the notion 'children' be only useful for Java projects?

That's right. I see "modules" and "projects" as the same thing in terms of
services. The only point where it *might* make sense to expose nesting in the
interface is to define a project that uses the services of some other project in
a parent directory.

Internally, it would probably make sense to be aware of nesting, but I agree
that exposing it on the interface adds complexity that is better avoided.

Cheers,
  Lluis


-- 
"And it's much the same thing with knowledge, for whenever you learn
something new, the whole world becomes that much richer."
-- The Princess of Pure Reason, as told by Norton Juster in The Phantom
Tollbooth

Re: IDE

[Dmitry Gutov]

On 10/28/2015 01:39 PM, Eric Ludlam wrote:

> Did you enable semantic-mode?  The buffer would need to be parsed for
> the data to be available, and for the prompt to have completion.

Yes, that was the problem, thanks. It works, and looks kinda nice.

> This is a good point.  I always have thought about language independence
> and extensibility, but most folks are just trying to solve a problem and
> may not be interested in extending to anything else.

One way to encourage wider applicability is to offer smaller, maybe 
less-functional, but more generic APIs.

> I've built many little tools over the years, but perhaps they are
> observed as parts of CEDET core, and not as examples.  The semantic-ia
> functions were supposed to be examples for how to use the
> infrastructure, but users have ended up using them as their interface
> and they became more complex.

Maybe it would be better if they were more polished and 
fully-functional, instead.

Re: IDE

[Steinar Bang]

>>>>> Filipp Gunbin <fgunbin@fastmail.fm>:

> I've been following this long thread with interest and there's one
> thing that keeps bothering me - should we duplicate the logic of build
> tool in Emacs IDE-like features?

If you mean: "should we re-implement things like maven in lisp?" then my
answer is: "no, not necessarily"

But it's useful to know how the interesting parts of the build config is
created and what it means (in this case how the classpath is set up when
multiple pom.xml files are involved).

> In my `javaimp' package (available in GNU Elpa) Emacs calls Maven to get
> classpath, then scans all jars in it (it takes them from local Maven
> cache and reads what classes are inside).

Sounds interesting, I will check it out.

> This is done only once given that pom.xml doesn't change (and if it
> changes this is repeated).

Have you given any thoughts to triggering this when the pom.xml is saved
and/or changed on disk?

> This requires some time to wait for Maven to finish and output the
> classpath.  But jar scanning is inevitable, I guess, and takes more time
> (both are cached then).

Do you cache this only in memory or on disk? (sounds like might be a
good idea to cache this information in an S-expression file, eg. pom.el,
toghether with each pom.xml and maintain the pom.el in a make-like
fashion...?) 

Re: IDE

[Nix]

On 27 Oct 2015, Nikolaus Rath said:

> On Oct 27 2015, Eli Zaretskii <eliz@gnu.org> wrote:
>> My interpretation of that, perhaps incorrect, was that you tried to
>> use some functionality and tried to find documentation about it, but
>> either the documentation was inadequate or completely missing.
>
> Well, I followed the instructions in the manual but (as far as I can
> tell) didn't get support for Global/idutils/cscope. At the time I didn't
> suspect anything was wrong - until I read here that there ought to be
> such support. Thus my conclusion that the "obvious bit of code" isn't
> all that obvious (in both necessity and content).
>
>> Based on that interpretation, I suggested that you report what you
>> tried to figure out from the documentation, but couldn't.
>
> I would have expected the documentation to tell me that I need to do
> something in order to get Global et al support (and ideally also *what*
> I need to do).

The manual says

,----[ 7. Miscellaneous Commands ]
|    EDE can use external tools to help with file finding.  To do this,
| customize `ede-locate-setup-options'.
| 
|  -- Variable: ede-locate-setup-options
|      List of locate objects to try out by default.  Listed in order of
|      preference.  If the first item cannot be used in a particular
|      project, then the next one is tried.  It is always assumed that
|      "ede-locate-base" is at end of the list.
`----

It is true that 'Miscellaneous Commands' is not a very good place for
this, and it doesn't note that this is not only used for C-c . f (which
I have never used, and would never use, since it doesn't have any
completions at all so feels like something from a previous era), but for
all file searches done by EDE, Semantic, etc. None of the available
options are documented at all: you have to dig through the source code.

FWIW, I got it to work via

(setq ede-locate-setup-options '(ede-locate-global ede-locate-locate ede-locate-base))
;;...
(semantic-mode 1)
(semanticdb-enable-gnu-global-databases 'c-mode)
(semanticdb-enable-gnu-global-databases 'c++-mode)

but whether all of these are actually necessary or not (in particular,
whether the latter two options are redundant with part of the first) I
have no real idea.

-- 
NULL && (void)

Re: IDE

[Filipp Gunbin]

On 28/10/2015 16:57 +0100, Steinar Bang wrote:

>>>>>> Filipp Gunbin <fgunbin@fastmail.fm>:
>> This is done only once given that pom.xml doesn't change (and if it
>> changes this is repeated).
>
> Have you given any thoughts to triggering this when the pom.xml is saved
> and/or changed on disk?

When a set of completions is prepared for an interactive command (like
`javaimp-add-import'), pom files' timestamps are checked and if they
changed then the information stored by the package is updated.  This
works only for current and one parent pom up the hierarchy for now, for
others it's better to do `javaimp-forget-all-visited-modules' manually.

The same is done for jar files: when a jar file classes are added to a
set of completion alternatives, its timestamp is checked - useful
because often we work with SNAPSHOT versions of the current module's
siblings and wish to read the latest information from them.

Current module's classes are added by scanning current source directory,
this is rather dumb scanning - no code parsing, just file names.

javaimp does not support importing of inner classes, but usually for me
it's not a problem because it's enough to import the top class in a
file.


>> This requires some time to wait for Maven to finish and output the
>> classpath.  But jar scanning is inevitable, I guess, and takes more time
>> (both are cached then).
>
> Do you cache this only in memory or on disk? (sounds like might be a
> good idea to cache this information in an S-expression file, eg. pom.el,
> toghether with each pom.xml and maintain the pom.el in a make-like
> fashion...?)

Currently only in memory, but I'm thinking of doing a "flush" to a
file.

My workflow is like that: when I "open" a project, I call
`javaimp-maven-visit-root' on a top-level project and it loads
information on it and its child projects (mvn help:effective-pom).  When
I call `javaimp-add-import' from one of the projects' files, it
recognizes which module needs to be scanned and invokes Maven on that
module (mvn dependency:build-classpath AFAIR), then scans jars which
require scanning and suggests completion alternatives.  So it's all on
demand (have to wait a bit during the first time, though).  This is to
eliminate constant indexing / scanning / updating which I hated in
Idea/Eclipse.

Filipp

Re: IDE

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > If you mean: "should we re-implement things like maven in lisp?" then my
  > answer is: "no, not necessarily"

Quoth the maven, "implement me nevermore."
;-)

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: IDE

[Oleh Krehel]

Dmitry Gutov <dgutov@yandex.ru> writes:

> - Oleh Krehel's function-args even mentions C/C++ in its summary.
>
> Perhaps, if there were more broadly applicable examples, it would lead
> to broader adoption. Maybe we should wonder why prefer making tools
> for CEDET that only target C and C++.

I targeted C++ because that's a language that I use a lot and I needed
support for.

Dynamic languages like JavaScript/Ruby/Python/Elisp/CL/Clojure/Scheme
don't need to rely on Semantic for tags info, they can just get it from
the REPL. It's still a choice, and both things can work and cooperate.

However, for static languages like C++ Semantic is the only choice for
getting the tag metadata. Which other popular language is in the static
camp? Only Java, the rest I label as hipster, no offense.  I guess some
good progress could be made by extending Semantic for Java, however it's
hard to find Emacs people who are enthusiastic about that language. And
the amount of work to be done would have to be enormous to trump the
popular Java IDE.

    Oleh

Re: IDE

[Dmitry Gutov]

On 10/29/2015 01:12 PM, Oleh Krehel wrote:

> I targeted C++ because that's a language that I use a lot and I needed
> support for.

Makes sense. Why make it C++-specific, though?

> Dynamic languages like JavaScript/Ruby/Python/Elisp/CL/Clojure/Scheme
> don't need to rely on Semantic for tags info, they can just get it from
> the REPL. It's still a choice, and both things can work and cooperate.

C++ can get it from rtags or irony-mode. And "just get it from the REPL" 
is a big simplification.

> However, for static languages like C++ Semantic is the only choice for
> getting the tag metadata. Which other popular language is in the static
> camp? Only Java, the rest I label as hipster, no offense.

a) Why is the dynamic/static distinction important for function-args? 
Ruby and Python, for instance, have function signatures that don't look 
too different from C++/Java ones.

b) There's Scala, and the fairly popular ENSIME. They are working on 
Java support, by the way: https://github.com/ensime/ensime-server/issues/345

Re: IDE

[Oleh Krehel]

Dmitry Gutov <dgutov@yandex.ru> writes:

> On 10/29/2015 01:12 PM, Oleh Krehel wrote:
>
>> I targeted C++ because that's a language that I use a lot and I needed
>> support for.
>
> Makes sense. Why make it C++-specific, though?

It works for C as well. I was using a lot of C++ templates at that time.
And Semantic didn't display them nicely.

>> Dynamic languages like JavaScript/Ruby/Python/Elisp/CL/Clojure/Scheme
>> don't need to rely on Semantic for tags info, they can just get it from
>> the REPL. It's still a choice, and both things can work and cooperate.
>
> C++ can get it from rtags or irony-mode. And "just get it from the
> REPL" is a big simplification.

Well, I know that the mentioned lisps can really get this info from the
REPL. Of course, the restriction is that the code needs to be loaded.
The same applies to Jedi and mozrepl/tern, I guess. Not sure if there's
any good similar tooling for Ruby.

>> However, for static languages like C++ Semantic is the only choice for
>> getting the tag metadata. Which other popular language is in the static
>> camp? Only Java, the rest I label as hipster, no offense.
>
> a) Why is the dynamic/static distinction important for function-args?
> Ruby and Python, for instance, have function signatures that don't
> look too different from C++/Java ones.

The thing is that function-args adds some functionality that would be
missing otherwise for C++.  Ruby and Python already have some of this
functionality by extracting it from the REPL. Of course, it would be
nice to make it work everywhere, but it's not urgent if the gap is
already filled by something else, e.g. Jedi.

> b) There's Scala, and the fairly popular ENSIME. They are working on
> Java support, by the way:
> https://github.com/ensime/ensime-server/issues/345

That's nice. But somehow I don't see why anyone would not just use
Clojure if you need a JVM hosted language.

Re: IDE

[Dmitry Gutov]

On 10/29/2015 01:37 PM, Oleh Krehel wrote:

> The thing is that function-args adds some functionality that would be
> missing otherwise for C++.

I'm not exactly sure what you're referring to, but shouldn't that go 
straight into CEDET?

> That's nice. But somehow I don't see why anyone would not just use
> Clojure if you need a JVM hosted language.

If only everyone worked on greenfield projects, liked Lisp and had 
freedom to choose it at the workplace.

Re: IDE

[Oleh Krehel]

Dmitry Gutov <dgutov@yandex.ru> writes:

> On 10/29/2015 01:37 PM, Oleh Krehel wrote:
>
>> The thing is that function-args adds some functionality that would be
>> missing otherwise for C++.
>
> I'm not exactly sure what you're referring to, but shouldn't that go
> straight into CEDET?

I'm referring to the display of complex C++ types, some inheritance of
members from one namespace into another, some heuristics for
symbol-at-point completion, the ability to jump-to-tag in a whole
directory instead of a single file etc.  These are mostly minor
improvements here and there. Since I make these small improvements often
and don't have write access to CEDET, I keep them in a separate package.

>> That's nice. But somehow I don't see why anyone would not just use
>> Clojure if you need a JVM hosted language.
>
> If only everyone worked on greenfield projects, liked Lisp and had
> freedom to choose it at the workplace.

Oh well, can't have it all I guess:)

Re: IDE

[Dmitry Gutov]

On 10/29/2015 02:56 PM, Oleh Krehel wrote:

> I'm referring to the display of complex C++ types, some inheritance of
> members from one namespace into another, some heuristics for
> symbol-at-point completion, the ability to jump-to-tag in a whole
> directory instead of a single file etc.  These are mostly minor
> improvements here and there. Since I make these small improvements often
> and don't have write access to CEDET

There never was a prohibition to committing changes to CEDET in the 
Emacs repository, and now according to the recent discussions on the 
CEDET mailing list, all CEDET development is going to move to Emacs.

So, commit away?

Re: IDE

[Eric Ludlam]

On 10/29/2015 09:13 AM, Dmitry Gutov wrote:
> On 10/29/2015 02:56 PM, Oleh Krehel wrote:
>
>> I'm referring to the display of complex C++ types, some inheritance of
>> members from one namespace into another, some heuristics for
>> symbol-at-point completion, the ability to jump-to-tag in a whole
>> directory instead of a single file etc.  These are mostly minor
>> improvements here and there. Since I make these small improvements often
>> and don't have write access to CEDET
>
> There never was a prohibition to committing changes to CEDET in the 
> Emacs repository, and now according to the recent discussions on the 
> CEDET mailing list, all CEDET development is going to move to Emacs.
>
> So, commit away?
>
I read through the function-args readme briefly today, and it looks like 
some nice stuff.

I can see why some of the pieces would be c++ specific.

The usability improvements in this tool are really nice.  If you have an 
assignment w/ the FSF, I would have been happy to provide write access 
and have help getting patches into CEDET, or add features.   Your 
function args package notes that it has improved versions of CEDET 
functions.  I rarely had time to focus on usability of keybindings and 
such, as I usually have my hands full keeping parsers and other 
infrastructure up to date.   If someone has interests in interfaces and 
what kind of data is useful while editing, I'll always be glad to pull 
in those changes, and help keep things language neutral where applicable.

Eric

Re: IDE

[Oleh Krehel]

Hi Eric,

Eric Ludlam <ericludlam@gmail.com> writes:

> I read through the function-args readme briefly today, and it looks
> like some nice stuff.
>
> I can see why some of the pieces would be c++ specific.
>
> The usability improvements in this tool are really nice.  If you have
> an assignment w/ the FSF, I would have been happy to provide write
> access and have help getting patches into CEDET, or add features.
> Your function args package notes that it has improved versions of
> CEDET functions.  I rarely had time to focus on usability of
> keybindings and such, as I usually have my hands full keeping parsers
> and other infrastructure up to date.   If someone has interests in
> interfaces and what kind of data is useful while editing, I'll always
> be glad to pull in those changes, and help keep things language
> neutral where applicable.

I have an assignment and commit access to Emacs.  Should I wait until
CEDET gets a git branch in Emacs until committing?
Or just commit to the CEDET that's already in Emacs core and hope that
it will magically be pulled to the base repository?
I could also commit directly to CEDET if I had write access, but I think
CEDET doesn't use git, and that's the only thing I (know how to) use.

    Oleh

Re: IDE

[Xue Fuqiao]

On Fri, Oct 30, 2015 at 5:04 PM, Oleh Krehel <ohwoeowho@gmail.com> wrote:
> but I think CEDET doesn't use git, and that's the only thing I (know
> how to) use.

CEDET uses Git now.  You can browse the repository here:
http://sourceforge.net/p/cedet/git/ci/master/tree/

Related information:

* http://sourceforge.net/p/cedet/mailman/message/33106122/
* http://sourceforge.net/p/cedet/mailman/message/33188785/
* http://sourceforge.net/p/cedet/mailman/message/33209931/

Re: IDE

[Oleh Krehel]

Xue Fuqiao <xfq.free@gmail.com> writes:

> On Fri, Oct 30, 2015 at 5:04 PM, Oleh Krehel <ohwoeowho@gmail.com> wrote:
>> but I think CEDET doesn't use git, and that's the only thing I (know
>> how to) use.
>
> CEDET uses Git now.  You can browse the repository here:
> http://sourceforge.net/p/cedet/git/ci/master/tree/

Thanks, Xue. I was using http://git.code.sf.net/p/cedet/git for a long
time now. I thought it was only a Git mirror though.

Eric, could I possibly get write access there? I promise not commit
anything too strange without asking first.

Re: IDE

[Dmitry Gutov]

Hi Steinar,

Sorry for the late response.

On 10/27/2015 07:28 PM, Steinar Bang wrote:
> (Classpath in maven is actually independent of the directory layout of
> the checked out projects, but I won't get into that now)

I meant that either we have several values: project-project, 
project-module1, project-module2, and each of those reports a different 
value of classpath, or we only have one value, project, and the accessor 
function project-classpath takes a additional argument, DIR, and can 
return different values, depending on which module, internally, DIR 
belongs to.

Note that classpath itself is a pretty JVM-centric attribute. It points 
to jars, as well as directories (the latter less often), and reading the 
contents of jars, if we want to, will have to be handled specially. 
Overall, classpath will be more relevant for features that particularly 
target the JVM.

More language agnostic attributes, which I'm more worried about, are 
sets of directories related to the project. We can search those 
directories, and we can list files in them (and implement, based on 
that, the feature "jump to file in the project", with completion).

We have currently selected two attributes like that: "project roots", 
directories which contain the code of the current project, and "search 
path" (which would probably be more appropriately named as "library 
roots"). Those directories contain the code that the current project 
refers to.

For C or C++ the closest analog would be the include path. For Ruby or 
Python - the directories in the current language installation where the 
libraries are installed.

For JVM languages, the classpath seems to be the closest thing, but it 
contains compiled code. The same jars can also contain sources (I 
think?), but searching through them, as well as listing contained files, 
or visiting them, is tricker. We should probably consider that for a 
later stage.

> (That means that as long as the projects are built in dependency order,
> they can in *theory* be be checked out separately and built separately,
> parent, and dependencies to other projects can be resolved against the
> local cache.  In practice, however, it is simpler to organize the maven
> project in the recommended hierarchy and let maven handle built order
> and stuff)

This is all great, but as long as Maven knows how to build the project, 
is there a reason for the project API to know these details?

> Depends on what you mean with important project attributes...?

The full set will probably emerge incrementally, over time, as 
developers who want to use the project API complain that it covers too 
little.

> If "project" is the parent pom.xml of the two modules, you can set
> properties in "project" that are shared by module1 and module2.  You can
> also share plugin configurations in this way.

All right. So either the project API will handle of property 
inheritance, or the implementations will need to do that.

The latter imposes some difficulties on how the project values are 
constructed, but since the multi-module project model seem to be 
prevalent in the Maven world only, I think I'd rather have a simpler API.

> eclipse m2e will still recognize that "module1" contains the dependency
> of "module2" and build them in order (they will show up as project
> dependencies in the project classpath in eclipse).

So I take it the dependencies are mostly important for the "build 
everything" command.

> The default search scope in eclipse is "workspace", which is the
> projects seen in eclipse's package explorer.

That's very interesting. Maybe we'll have a notion of a workspace (the 
set of the currently opened projects), as well as manual 
"open/visit/close project" commands. Like, in the second version.

Re: IDE

[Dmitry Gutov]

On 10/28/2015 02:34 PM, Filipp Gunbin wrote:

> I've been following this long thread with interest and there's one
> thing that keeps bothering me - should we duplicate the logic of build
> tool in Emacs IDE-like features?

Probably not. We could conceivably want to provide the access to some 
information, if someone wanted to implement an Emacs-y editor for the 
project files. That's ways off to the future, though.

Re: IDE

[Steinar Bang]

>>>>> Dmitry Gutov <dgutov@yandex.ru>:

> Probably not. We could conceivably want to provide the access to some
> information, if someone wanted to implement an Emacs-y editor for the
> project files. That's ways off to the future, though.

(If we're talking maven then nxml works file.  I believe I have an RNG
schema for maven lying around somewhere...)

Re: IDE

[Dmitry Gutov]

On 11/01/2015 10:10 PM, Steinar Bang wrote:

> (If we're talking maven then nxml works file.

Cool, I'm all for lightweight approaches. It was just the best example 
of using dependency information that I could think of right away.

> I believe I have an RNG
> schema for maven lying around somewhere...)

If we want to get fancier, Intellij IDEA also provides completion for 
e.g. artifact ids. But that's neither here nor there.

Re: IDE

[Eric Ludlam]

On 10/31/2015 07:40 AM, Oleh Krehel wrote:
> Xue Fuqiao <xfq.free@gmail.com> writes:
>
>> On Fri, Oct 30, 2015 at 5:04 PM, Oleh Krehel <ohwoeowho@gmail.com> wrote:
>>> but I think CEDET doesn't use git, and that's the only thing I (know
>>> how to) use.
>> CEDET uses Git now.  You can browse the repository here:
>> http://sourceforge.net/p/cedet/git/ci/master/tree/
> Thanks, Xue. I was using http://git.code.sf.net/p/cedet/git for a long
> time now. I thought it was only a Git mirror though.
>
> Eric, could I possibly get write access there? I promise not commit
> anything too strange without asking first.
> .
>
Source Forge wouldn't let me edit user permissions this morning.  :(

If you email me your SF id, I can add you when SF next allows me to.

I think you might be better off checking directly into Emacs however.  
We need to do a final merge from the CEDET repository into Emacs proper, 
after which we will probably abandon the separate repository.  Keeping 
them in sync has been a problem, so we will probably come up with an 
alternate branching strategy.

For function-args proper, lets consider how to bring it into the CEDET 
space.  For example, there is this line in the commentary:

;; * `moo-complete' -- a c++-specific version of 
`semantic-ia-complete-symbol'. Perhaps we can figure out how to either 
merge the two, or add the right way to extend -ia-complete-jump in that 
way for any language.

Thanks!
Eric

Re: IDE

[joakim]

Eric Ludlam <ericludlam@gmail.com> writes:

> On 10/27/2015 10:16 PM, Dmitry Gutov wrote:
>> On 10/22/2015 03:58 PM, Eric Ludlam wrote:
>>
>>> In addition, by going through the semantic version, there are a range of
>>> different formatting options to use that the user can select from.  That
>>> doesn't require idle-summary-mode, but is a side effect of using
>>> semantic to extract contextual information.
>>
>> The formatting options depend on using some rich structure like
>> Semantic tags. So it would make sense to port that only after we
>> standardize on some tag format universally across Emacs (are
>> Semantic tags optimal? I don't know).
>>
>
> What is optimal?  For me, it was about pulling all the datatype
> information out of the syntax, and differentiating between different
> tag classes, and having something that was Emacs friendly, and
> extensible to any language, no matter how strange.  For me, Semantic's
> tag structure is pretty good.  It has 5 slots:
>
> * The name, as 'car' so you can use assoc and other friendly things.
> * The class of the tag is next, as all languages have different kinds
> of tags.
> * plist of attributes, extensible to use anything, but with some
> common attribute names defined for use via API.
> * plist of properties, or bits of data needed by the tooling, which is
> arbitrary and extensible for whatever a tool might need.
> * Buffer binding information - either the [start end] of the tag, or
> an overlay.
>
> That makes the tag extensible for almost any situation, and keeps TAG
> data and OPERATIONAL data separate, saveable, and searchable.
>
>>> It was deemed optional when Yidong merged CEDET into Emacs.  You will
>>> probably need to use Emacs24 to make it work.  To try it out do this:
>>
>> Step 6 is broken for me.
>>
>>> 1) Install graphviz (it uses that for the layout engine)
>>> 2) Start emacs 24
>>> 3) Use CEDET from it's git repository
>>> 4) M-x find-library RET cogre RET
>>> 5) find cogre-element-peer in the code
>>> 6) M-x cogre-uml-quick-class RET
>>
>> I only get a "Class:" prompt, without a default value or
>> completions. Typing "cogre-element-peer" gives me "Could not find
>> class ...", even though cogre is obviously loaded. That's in Emacs
>> 24.5.
>
> Did you enable semantic-mode?  The buffer would need to be parsed for
> the data to be available, and for the prompt to have completion.
>
>>> When thinking about CEDET, it isn't about a
>>> bullet list of user facing features but about how it can enable someone
>>> working on said feature to have their work leveraged to a wider
>>> audience.
>>
>> People working on said features should be encouraged, of
>> course. Unfortunately, the two more interesting projects that I've
>> seen utilize CEDET are language-specific:
>>
>> - SRefactor only does anything useful for C/C++.
>> - Oleh Krehel's function-args even mentions C/C++ in its summary.

I'm not sure this is related, but I've done many little project specific
tools using parts of Cedet. For instance SRecode, and Cogre, and
Semantic. Since the things I made were tools for one-off tasks, I didn't
bother to publish them. I think a lot of use-cases are like this, and a
lot of people use various parts of Cedet like this.

Now you will say that this is a thread about making an IDE, but I still
think this use-case deserves mentioning. 

>>
>> Perhaps, if there were more broadly applicable examples, it would
>> lead to broader adoption. Maybe we should wonder why prefer making
>> tools for CEDET that only target C and C++.
>> .
>
> This is a good point.  I always have thought about language
> independence and extensibility, but most folks are just trying to
> solve a problem and may not be interested in extending to anything
> else.
>
> I've built many little tools over the years, but perhaps they are
> observed as parts of CEDET core, and not as examples.  The semantic-ia
> functions were supposed to be examples for how to use the
> infrastructure, but users have ended up using them as their interface
> and they became more complex.
>
> Eric
>
>

-- 
Joakim Verona

Re: IDE

[Oleh Krehel]

Eric Ludlam <ericludlam@gmail.com> writes:

> On 10/31/2015 07:40 AM, Oleh Krehel wrote:
>> Eric, could I possibly get write access there? I promise not commit
>> anything too strange without asking first.

> If you email me your SF id, I can add you when SF next allows me to.

I actually don't have a Source Forge account. Does Source Forge allow to
simply upload id_rsa.pub to allow access, like Github or Savannah?

> I think you might be better off checking directly into Emacs however.
> We need to do a final merge from the CEDET repository into Emacs
> proper, after which we will probably abandon the separate repository.
> Keeping them in sync has been a problem, so we will probably come up
> with an alternate branching strategy.

This is also fine with me. Only depends on how long until the final
merge is done.

> For function-args proper, lets consider how to bring it into the CEDET
> space.  For example, there is this line in the commentary:
>
> ;; * `moo-complete' -- a c++-specific version of
> `semantic-ia-complete-symbol'. Perhaps we can figure out how to either
> merge the two, or add the right way to extend -ia-complete-jump in
> that way for any language.

It's very C++ specific (namespaces, smart pointers, templates etc) to
generalize. I think a dispatch can be made on `major-mode' that forwards
`semantic-ia-complete-symbol' to `moo-complete' for c++-mode (and maybe
c-mode).

It may be possible to simplify `moo-complete' a lot depending on how
much better we can make the parser in terms of understanding C++.  A lot
of the code in `moo-complete' are "dumb" heuristics to come up with at
least something when `semantic-ia-complete-symbol' fails.

However, e.g. semantic-directory.el could be generalized to any
language. It may overlap a bit with the built-in CEDET functionality,
but I just couldn't find a good function that returns a joined list of
tags for a list of files, with all tags up-to-date, even if one of the
files was modified very recently.

Also the function arguments overlays could be generalized, making a good
alternative to Eldoc for all languages supported by semantic.