What IDE features are in CEDET?

What IDE features are in CEDET?

[Eric M. Ludlam]

There are a couple threads going about IDE code features, what does it
all mean, and what role does CEDET fill in all this.  Let me try to
answer the last question.

My long term goal for CEDET is wider than just intellisense.  The
intellisense that currently works in the CEDET/Semantic piece is a
small part of where I'd like to go that has turned into a big project.
I want to see Emacs provide round-trip code to UML and back again
interface that works for any language, where refactoring can occur w/
a UML diagram editor that re-writes your code.  Anyone familiar with
"Together" which I think is now owned by IBM will know what I mean.

This background will hopefully make the parts in CEDET more logically
connected.

If we batch up various IDE like features and list their requirements,
you need a couple things up front:

1) Project management -
   Emacs needs to know how a project is laid out so it can find the
   files that contain the code that you want to operate on.  This is
   the CEDET/EDE part.

   EDE supports automake directly (no extra work needed by users), or
   you can build projects via menus/keybindings that builds
   [auto]makefiles for you.

   I'd label EDE as needing testing, but otherwise sufficient for many
   tasks.  CEDET uses EDE for its own build infrastructure.

2) Code parsing / Code structure database
   Emacs needs to understand the code you want to work with, and that
   parsing infrastructure needs to translate any language into a
   generic form that tools can manipulate without knowing the source
   language.  This is the CEDET/Semantic part.

   There are pre-written parsers for C++, python, java, and others.
   There are also pre-written tools against the parser output
   implementing eldoc like behavior, plus smart completion.

   I also have a database system that catalogs the structure of your
   project, and keeps that data persistent.  There is an abstract
   layer so that the all-Emacs Lisp solution can be replaced with
   anything, such as beanshell (java) or ebrowse (C++).

   I'd say that the Semantic core is stable, the core languages with
   parsers are pretty good.  The C++ parser really needs to be ported
   from the old (semantic version 1) parser to the new bison based
   parser David Ponce wrote, and the gcc-xml thing I read about
   recently is another neat option that could be used.

3) Object system
   Writing the above needs some sort of OO infrastructure, so 
   CEDET/EIEIO provides a CLOS like layer in Emacs.

   EIEIO has been stable for me for a long time, and has a bunch of
   unit tests to keep it that way.


To follow up with the code generation/UML diagram thing, you need a
couple more features.


4) Code Generation
   To go in and re-write someone's code, you need a way to provide
   code generation.  Your refactoring tool will probably want to have
   a language independent core, so you need an abstraction layer
   between code-specific constructs and what you would code to in
   Lisp.  CEDET/Semantic provides the data structure for representing
   code at a high level (functions, variables, types, etc) and
   CEDET/SRecode provides a template authoring and cataloging tool.

   With SRecode as found in CEDET/CVS now, you can parse a Java File
   W/ Semantic, then write out the output from the semantic parser into a
   C++ file and get something that is mostly correct.

   I have not yet delved into writing code (ie if (..) { } else {}) 
   part, just the structure (ie functions, variables, types, etc).

   I'd say that the SRecode core is in good shape, but the language
   specific templates need a lot of help.

5) UML editing.
   UML editing isn't really needed for most coding tasks, but it's on
   my list for round trip graphic editing.  The CEDET/COGRE part has
   only the most rudimentary parts needed.


Ok, sounds good.  What's missing?

* These tools do not delve far into the code part, just the declarations.
  The Semantic parsing infrastructure can certainly handle it, I just
  haven't gone there.

* The number of languages supported is still meager.

* UI related features.  I seem to spend a lot of time on
  infrastructure, and less where the user actually types stuff in.
  Resolving where to bind the functionality into the typical Emacs
  user's workflow needs help.

  In addition, the data generated by Semantic could do a lot more, but
  no one has written the tooling for it.  The good news is that if
  someone wants to build dependency tree diagrams, subclass warnings,
  method override highlighting, or whatever, you can write it once for
  any language that supports the feature you want and has a parsing
  infrastructure associated with it.

* Parsers for indenting / coloring
  I get this question alot.  Can a Semantic parser be used for
  indentation or coloring?  Answer: yes.  I just haven't done it.  The
  more interesting question is, can a single parser provide data about
  the code, indentation, and coloring at the same time without making
  it too hard to maintain? 


There are a lot of little tools in CEDET beyond what I discussed
above, but these are the big features.

Eric

-- 
          Eric Ludlam:                       eric@siege-engine.com
   Siege: www.siege-engine.com          Emacs: http://cedet.sourceforge.net

Re: What IDE features are in CEDET?

[Lennart Borgman (gmail)]

Eric M. Ludlam wrote:
> 2) Code parsing / Code structure database
...
>    I'd say that the Semantic core is stable, the core languages with
>    parsers are pretty good.  The C++ parser really needs to be ported
>    from the old (semantic version 1) parser to the new bison based
>    parser David Ponce wrote, and the gcc-xml thing I read about
>    recently is another neat option that could be used.

There are some good parsers that need some changes to fit into the 
framework:

- nxml-mode XML parser.
- Steve Yegge's javascript parser.

It think that some parser also should be rewritten to work with parts of 
a buffer so that all the languages for building XHTML (like PHP etc) can 
be supported. A lot of programming is done with such languages today and 
IMO it also has a democratic value since frameworks for exchanging 
information (Drupal, WordPress etc) are built in those languages.


> * Parsers for indenting / coloring
>   I get this question alot.  Can a Semantic parser be used for
>   indentation or coloring?  Answer: yes.  I just haven't done it.  The
>   more interesting question is, can a single parser provide data about
>   the code, indentation, and coloring at the same time without making
>   it too hard to maintain? 

I am not sure what the benefits with a single parser would be. Instead I 
believe that a framework that cooperates with Semantic in some way and 
gives it chunks with different major modes would have benefits. (I hope 
mumamo can provide this.)

Re[2]: What IDE features are in CEDET?

[Eric M. Ludlam]

>>> "Lennart Borgman (gmail)" <lennart.borgman@gmail.com> seems to think that:
>Eric M. Ludlam wrote:
>> 2) Code parsing / Code structure database
>...
>>    I'd say that the Semantic core is stable, the core languages with
>>    parsers are pretty good.  The C++ parser really needs to be ported
>>    from the old (semantic version 1) parser to the new bison based
>>    parser David Ponce wrote, and the gcc-xml thing I read about
>>    recently is another neat option that could be used.
>
>There are some good parsers that need some changes to fit into the 
>framework:
>
>- nxml-mode XML parser.

I use and like nxml mode, but I'm not really sure what the integration
path here is.  If you suspect nxml-mode should use one of the
parser-generator frameworks, then I would have to disagree.  If you
suspect the existing nxml parser output could be translated into the
same style of output as Semantic's tag structures, that can work, but
I'm not entirely sure what would be done with the results since xml
is flexible enough to cause every schema to need it's own translator
into Semantic tag format.

>- Steve Yegge's javascript parser.

Steve has expressed displeasure with the CEDET infrastructure.  In
addition, the semantic parser doesn't meet his goals.  He has some
good concepts that would be worthwhile in Semantic though, such as
handling colors and indentation, and allowing incremental first-time
parsing.

Semantic has incremental parsing also, but only have a full pass has
been done.

>It think that some parser also should be rewritten to work with parts of 
>a buffer so that all the languages for building XHTML (like PHP etc) can 
>be supported. A lot of programming is done with such languages today and 
>IMO it also has a democratic value since frameworks for exchanging 
>information (Drupal, WordPress etc) are built in those languages.

There was a long thread on this topic in the cedet-devel@sf.net
mailing list.  Semantic's mode targeting is done using
"mode-local.el" which is part of CEDET, which allows simple context
switching between language features in a single buffer.

>> * Parsers for indenting / coloring
>>   I get this question alot.  Can a Semantic parser be used for
>>   indentation or coloring?  Answer: yes.  I just haven't done it.  The
>>   more interesting question is, can a single parser provide data about
>>   the code, indentation, and coloring at the same time without making
>>   it too hard to maintain? 
>
>I am not sure what the benefits with a single parser would be. Instead I 
>believe that a framework that cooperates with Semantic in some way and 
>gives it chunks with different major modes would have benefits. (I hope 
>mumamo can provide this.)

The benefit to a single parser for multiple jobs against a single
language in a single buffer is that you only need to lex it and run
through the parser once.  You are talking about multiple languages in
a single buffer which is different.  Multiple parsers is fine in that
case since each character in the buffer would only be lex'd up once.

Eric

-- 
          Eric Ludlam:                       eric@siege-engine.com
   Siege: www.siege-engine.com          Emacs: http://cedet.sourceforge.net

Re: What IDE features are in CEDET?

[Lennart Borgman (gmail)]

Eric M. Ludlam wrote:
> If you
> suspect the existing nxml parser output could be translated into the
> same style of output as Semantic's tag structures, that can work, but
> I'm not entirely sure what would be done with the results since xml
> is flexible enough to cause every schema to need it's own translator
> into Semantic tag format.

I no nothing about the details. What I thought was that Semantic (or 
perhaps more the user interface part of it) could ask nxml-mode for the 
state, completione possibilities etc, whenever needed.

> Semantic has incremental parsing also, but only have a full pass has
> been done.

That reminds me of that Emacs font-lock can do a second pass where 
context is taken into account. (If I have understood things correctly.)

> There was a long thread on this topic in the cedet-devel@sf.net
> mailing list.  Semantic's mode targeting is done using
> "mode-local.el" which is part of CEDET, which allows simple context
> switching between language features in a single buffer.

Seems like a needed feature.

> You are talking about multiple languages in
> a single buffer which is different.

Yes, but sometimes you can have view the buffer content either way. For 
example a buffer containing a php file can be viewed as either just 
consisting of php code (which of course is a valid view) or as some part 
containing xhtml and some containing php script code.

The advantages/disadvanteges is of course those you have mentioned. I 
mention this just to make the concepts more clear.

Re[2]: What IDE features are in CEDET?

[Eric M. Ludlam]

>>> "Lennart Borgman (gmail)" <lennart.borgman@gmail.com> seems to think that:
>Eric M. Ludlam wrote:
>> If you
>> suspect the existing nxml parser output could be translated into the
>> same style of output as Semantic's tag structures, that can work, but
>> I'm not entirely sure what would be done with the results since xml
>> is flexible enough to cause every schema to need it's own translator
>> into Semantic tag format.
>
>I no nothing about the details. What I thought was that Semantic (or 
>perhaps more the user interface part of it) could ask nxml-mode for the 
>state, completione possibilities etc, whenever needed.

Ah, that is certainly possible.  I'd be happy to help someone get
setup for doing that if I know what they are looking for.

Eric

-- 
          Eric Ludlam:                       eric@siege-engine.com
   Siege: www.siege-engine.com          Emacs: http://cedet.sourceforge.net

Re: What IDE features are in CEDET?

[Richard M Stallman]

    My long term goal for CEDET is wider than just intellisense.

The question I asked is about current status, not goals.
We were shown a list of common IDE capabilities.
Which of those can CEDET currently do?
Which of those can CEDET not currently do?

It may be that I could determine the answer from your message
if I knew the field thoroughly, but I don't and I can't.

    I want to see Emacs provide round-trip code to UML and back again
    interface that works for any language, where refactoring can occur w/
    a UML diagram editor that re-writes your code.

Do you mean to implement this INSIDE Emacs, or just have Emacs use
external programs to do it?

Re[2]: What IDE features are in CEDET?

[Eric M. Ludlam]

>>> Richard M Stallman <rms@gnu.org> seems to think that:
>    My long term goal for CEDET is wider than just intellisense.
>
>The question I asked is about current status, not goals.
>We were shown a list of common IDE capabilities.
>Which of those can CEDET currently do?
>Which of those can CEDET not currently do?
>
>It may be that I could determine the answer from your message
>if I knew the field thoroughly, but I don't and I can't.

Ok.  CEDET provides smart-completion, and the smart help (like eldoc)
that was discussed now.  CEDET is part-way to providing a range of the
refactoring features listed earlier, but will not do them now.  CEDET
provides the data needed by ECB for all the browsing features that
were discussed.

>    I want to see Emacs provide round-trip code to UML and back again
>    interface that works for any language, where refactoring can occur w/
>    a UML diagram editor that re-writes your code.
>
>Do you mean to implement this INSIDE Emacs, or just have Emacs use
>external programs to do it?

Inside Emacs, with external programs optional.

Eric

-- 
          Eric Ludlam:                       eric@siege-engine.com
   Siege: www.siege-engine.com          Emacs: http://cedet.sourceforge.net

Re: What IDE features are in CEDET?

[Richard M Stallman]

It is clear that CEDET already handles enough languages to be very
useful.  That's the crucial question for the issue at hand, which
is installing CEDET in Emacs.

Support for more languages is useful to add, but doesn't affect
what we want to do with CEDET now.

Re: What IDE features are in CEDET?

[Richard M Stallman]

    >    I want to see Emacs provide round-trip code to UML and back again
    >    interface that works for any language, where refactoring can occur w/
    >    a UML diagram editor that re-writes your code.
    >
    >Do you mean to implement this INSIDE Emacs, or just have Emacs use
    >external programs to do it?

    Inside Emacs, with external programs optional.

Is a part of this already implemented in Cedet?
If so, which part?

Re[2]: What IDE features are in CEDET?

[Eric M. Ludlam]

>>> Richard M Stallman <rms@gnu.org> seems to think that:
>    >    I want to see Emacs provide round-trip code to UML and back again
>    >    interface that works for any language, where refactoring can occur w/
>    >    a UML diagram editor that re-writes your code.
>    >
>    >Do you mean to implement this INSIDE Emacs, or just have Emacs use
>    >external programs to do it?
>
>    Inside Emacs, with external programs optional.
>
>Is a part of this already implemented in Cedet?
>If so, which part?

The short answer:

CEDET has a parser (code to ..) and a UML editor (uml) and a template
manager (.. and back again) and all three can work with any language.

Longer answer:

CEDET has a package called COGRE (connected graph editor) that will
draw and allow editing of UML diagrams in an Emacs buffer.  Emacs is
not an ideal environment for drawing diagrams, but it is passable and
quick.

CEDET also has a tool I called SRecode that is a template manager.  I
started with Tempo, but it did not have the feature set I needed to
manage complex templates.  SRecode is used to create code, and can
already generate a whole compilable program in C++ from a Lisp data
structure.  I have not combined CORGE (the UML diagram editor) to
SRecode (template manager) to create code from UML yet.

CEDET also has the Semantic tools which has been more broadly
discussed.  It will parse code into a structure that can be
represented by UML.  I did write a program that will generate a very
simple UML diagram from the Semantic data, but it is more proof of
concept than anything else.  I have a different program that will
create a dot program for drawing broader UML diagrams.  DOT requires
an external tool that creates diagrams.  If you have DOT installed, it
would be possible to generate an image to display in Emacs, though I
have not written that.


Thus, you can parse a program, and you can edit UML, and you can
generate code with CEDET, but the three have not been combined to
provide the full tool.

For more detail, there is my previous email.

Does this make more sense now?

Eric