Project detection and configuration (Was: IDE)

Project detection and configuration (Was: IDE)

[Lluís]

Hi!

As Eric pointed out, most of the user-facing features of an IDE depend on some
underlying support to detect and configure projects.

Now, it is not clear what "detect and configure" really means, but it seems that
EDE (CEDET's project management component) is not fully satisfying or simple
enough to extend/configure at this point in time.

I've formed my own opinions on how this could be improved, but I would like to
hear what others think before discussing how to do it. So, what is required from
"project management"? A few things quickly come into mind:

* Repository
** Detect the root directory that conforms a checked-out repository
** Ignore repository-specific files on other components
** Interact with repo-management tools?

* Project information
** Name, version, homepage, etc

* Build system
** This one seems rather tricky to provide a unified interface
** Detect autotools, hand-made makefiles, ant, maven, linux, etc.
** Build modes of a project (e.g., debug vs release)
** Generate files (targets) of a project
** Create release tarballs?
** Sometimes identifies the root directory of the project

* Source file locations
** Detect the "MIME" of every file on the project
** Information from the build system usually helps identifying files
** Detect changes to files

* Per-source compilation flags
** Necessary for code-analysis tools
** Can be extracted from the build system on some very narrow cases

* Code analysis
** Use CEDET's parsers
*** Needs to know where are the relevant sources, and their interdependence
** Use external tools
*** Need to know how to access their information
*** Need to know how to update their information?

* Auto-detection
** All functionality implementations should support as much auto-configuration
   as possible

* Code navigation, refactoring and other high-level operations
** I'm leaving these out of the "project management" discussion for now, since I
   think they can be implemented on top of the rest

* Let users customize each of the above
** It should be easy to customize existing functionality (e.g., using dir-local
   vars, a simple alist, etc.)
** It should be reasonably easy to compose different implementations of a
   functionality (e.g., use some project auto-detection and on top overlay some
   functionality for a specific external dependency - finding the headers for
   external library dependencies comes into mind -)
** It should be reasonably easy to write new functionality for each of these
   tasks; made easier if composition is easy


What else?

PS: Note that some functionalities are repeated on that list.


Thanks,
  Lluis

-- 
"And it's much the same thing with knowledge, for whenever you learn
something new, the whole world becomes that much richer."
-- The Princess of Pure Reason, as told by Norton Juster in The Phantom
Tollbooth

Re: Project detection and configuration

[John Wiegley]

>>>>> Lluís  <xscript@gmx.net> writes:

> I've formed my own opinions on how this could be improved, but I would like
> to hear what others think before discussing how to do it. So, what is
> required from "project management"?

Great list, Lluís! I agree that making "projects" a more well-defined entity
is part of the IDE picture.

At the moment I use Magit and Projectile, which rely on version control to
define the scope and contents of a project. I only bring this up to say that I
like "zero configuration", so that Emacs can be used to navigate projects
without requiring the user to define those contents manually until they have a
special need that can't be inferred.

John

Re: Project detection and configuration (Was: IDE)

[Evgeniy Dushistov]

On Wed, Oct 14, 2015 at 05:58:21PM +0200, Lluís wrote:
> 
> As Eric pointed out, most of the user-facing features of an IDE depend on some
> underlying support to detect and configure projects.
> 
> Now, it is not clear what "detect and configure" really means, but it seems that
> EDE (CEDET's project management component) is not fully satisfying or simple
> enough to extend/configure at this point in time.
> 
> I've formed my own opinions on how this could be improved, but I would like to
> hear what others think before discussing how to do it. So, what is required from
> "project management"? A few things quickly come into mind:
> 
> * Repository
> ** Detect the root directory that conforms a checked-out repository
> ** Ignore repository-specific files on other components
> ** Interact with repo-management tools?
> 

"magit" detect repo root automatically, you just call "magit-status"
in buffer mapped to file/directory inside repo and it show right stuff,
may be just create "vc-status" that call "vc-dir" with relevant dir?

Anyway why do something about this, "vc-" and "magit-" good enough,
is any IDE do better stuff using information about project?

> * Project information
> ** Name, version, homepage, etc
> 

IMHO, never used this information in any IDE, except name.

> * Build system

Actually if we talk about C/C++ open source project this is a key for
all other features.

Almost all C/C++ open source project have makefile/cmake file/scons etc,
and thats all. IDE must parse them and exctract information about:

- how to compile exactly one source file (to get auto completion,
  refactoring, right syntax highlight, warning/errors during typing [like flymake] etc)
- what executable files was created (to make possible click Debug/Run,
  to run valgrind with it, excract gcov information etc)

And cedet have framework for this, it can parse makefile/cmake, plus
you can easily extend it to support new build system, writing something
like this:
(defclass my-ede-cpp-root-project (ede-cpp-root-project eieio-instance-tracker)
  (
   (custom_build_cmd :initarg :custom_build_cmd
		 :initform ""
		 :type string
		 :custom string
		 :documentation "Set this field to shell command for build your project")
   (path_to_exe :initarg :path_to_exe
		:initfrom ""
		:type string
		:custom string
		:documentation "This field hold path to your project")
   )
  "My class for ede projects"
  :method-invocation-order :depth-first
)

So it would be great to finish makefile/cmake support in EDE, plus
some easy way to get from EDE information about exact file: includes, defines, plus
flags like -std=c++11

-- 
/Evgeniy

Re: Project detection and configuration

[Lluís]

Evgeniy Dushistov writes:

> On Wed, Oct 14, 2015 at 05:58:21PM +0200, Lluís wrote:
>> 
>> As Eric pointed out, most of the user-facing features of an IDE depend on some
>> underlying support to detect and configure projects.
>> 
>> Now, it is not clear what "detect and configure" really means, but it seems that
>> EDE (CEDET's project management component) is not fully satisfying or simple
>> enough to extend/configure at this point in time.
>> 
>> I've formed my own opinions on how this could be improved, but I would like to
>> hear what others think before discussing how to do it. So, what is required from
>> "project management"? A few things quickly come into mind:
>> 
>> * Repository
>> ** Detect the root directory that conforms a checked-out repository
>> ** Ignore repository-specific files on other components
>> ** Interact with repo-management tools?
>> 

> "magit" detect repo root automatically, you just call "magit-status"
> in buffer mapped to file/directory inside repo and it show right stuff,
> may be just create "vc-status" that call "vc-dir" with relevant dir?

> Anyway why do something about this, "vc-" and "magit-" good enough,
> is any IDE do better stuff using information about project?

Well, I put the interaction piece with a question mark because I'm not sure
there is anything that the "project management" piece should do, beside maybe
starting up the appropriate repository management tool (e.g., magit or vc).


>> * Project information
>> ** Name, version, homepage, etc
>> 

> IMHO, never used this information in any IDE, except name.

Yeah, I'm not sure how useful these might be either. Maybe it's just my mind
reminding me that EDE already has that information.


>> * Build system

> Actually if we talk about C/C++ open source project this is a key for
> all other features.

> Almost all C/C++ open source project have makefile/cmake file/scons etc,
> and thats all. IDE must parse them and exctract information about:

> - how to compile exactly one source file (to get auto completion,
>   refactoring, right syntax highlight, warning/errors during typing [like flymake] etc)
> - what executable files was created (to make possible click Debug/Run,
>   to run valgrind with it, excract gcov information etc)

> And cedet have framework for this, it can parse makefile/cmake, plus
> you can easily extend it to support new build system, writing something
> like this:
> (defclass my-ede-cpp-root-project (ede-cpp-root-project eieio-instance-tracker)
>   (
>    (custom_build_cmd :initarg :custom_build_cmd
> 		 :initform ""
> 		 :type string
> 		 :custom string
> 		 :documentation "Set this field to shell command for build your project")
>    (path_to_exe :initarg :path_to_exe
> 		:initfrom ""
> 		:type string
> 		:custom string
> 		:documentation "This field hold path to your project")
>    )
>   "My class for ede projects"
>   :method-invocation-order :depth-first
> )

> So it would be great to finish makefile/cmake support in EDE, plus
> some easy way to get from EDE information about exact file: includes, defines, plus
> flags like -std=c++11

Actually, I'm more interested in analyzing what is necessary rather than what
existing code we can use to make it happen. Also, I just sent a mail describing
a possible approach to let such features to be developed independently from the
project concept:

  http://lists.gnu.org/archive/html/emacs-devel/2015-10/msg01302.html


Thanks,
  Lluis

-- 
"And it's much the same thing with knowledge, for whenever you learn
something new, the whole world becomes that much richer."
-- The Princess of Pure Reason, as told by Norton Juster in The Phantom
Tollbooth

Re: Project detection and configuration (Was: IDE)

[Dmitry Gutov]

On 10/14/2015 06:58 PM, Lluís wrote:

> * Repository
> ** Detect the root directory that conforms a checked-out repository
> ** Ignore repository-specific files on other components
> ** Interact with repo-management tools?

You might want to look at lisp/progmodes/project.el in the current 
master, if you haven't already.

> * Project information
> ** Name, version, homepage, etc

I think these are very low priority.

> * Build system
> ** This one seems rather tricky to provide a unified interface
> ** Detect autotools, hand-made makefiles, ant, maven, linux, etc.
> ** Build modes of a project (e.g., debug vs release)
> ** Generate files (targets) of a project
> ** Create release tarballs?

I'd rather have a flat list of tasks, so that different tasks could do 
many of these things.

Do we really need to be able to know, programmatically, that a certain 
task is related to the "debug" build? Otherwise, it would simply be to 
the build tool integration to include (or not) "debug" in the returned 
task name.

> ** Sometimes identifies the root directory of the project

When it does, a project implementation will have to handle this. Using 
`locate-dominating-file' or `file-exists-p' is not hard.

Re: Project detection and configuration

[Lluís]

Dmitry Gutov writes:

> On 10/14/2015 06:58 PM, Lluís wrote:
>> * Repository
>> ** Detect the root directory that conforms a checked-out repository
>> ** Ignore repository-specific files on other components
>> ** Interact with repo-management tools?

> You might want to look at lisp/progmodes/project.el in the current master, if
> you haven't already.

I wasn't aware of it, thanks.


>> * Project information
>> ** Name, version, homepage, etc

> I think these are very low priority.

Completely true.


>> * Build system
>> ** This one seems rather tricky to provide a unified interface
>> ** Detect autotools, hand-made makefiles, ant, maven, linux, etc.
>> ** Build modes of a project (e.g., debug vs release)
>> ** Generate files (targets) of a project
>> ** Create release tarballs?

> I'd rather have a flat list of tasks, so that different tasks could do many of
> these things.

> Do we really need to be able to know, programmatically, that a certain task is
> related to the "debug" build? Otherwise, it would simply be to the build tool
> integration to include (or not) "debug" in the returned task name.

Hmmmm, it's not that a task is debug only, but that it's not the same a task in
debug mode than in release mode. For example, you can have a debug and a release
build of the same program, each on its own out-of-tree build
directory. Therefore a make command would differ based on the target *and* the
build mode.

Still, I'm not sure there's a clean way to handle this without hardcoding the
difference between task and build mode. And I'm not even sure this makes sense
on all types of builds.


>> ** Sometimes identifies the root directory of the project

> When it does, a project implementation will have to handle this. Using
> `locate-dominating-file' or `file-exists-p' is not hard.

Right. My point is that using a repository root is a sane default most of the
time, but sometimes it might be necessary to use project-specific information
for that. For example, imagine some linux kernel checkout (which can be properly
identified) that is stored on some sub-directory of a larger repository.

Still, the root directory is not a useful piece of information per se, but it
can be handy to derive other information like paths to look for other files.


Cheers,
  Lluis

-- 
"And it's much the same thing with knowledge, for whenever you learn
something new, the whole world becomes that much richer."
-- The Princess of Pure Reason, as told by Norton Juster in The Phantom
Tollbooth

Re: Project detection and configuration

[David Kastrup]

Lluís <xscript@gmx.net> writes:

> Hi!
>
> As Eric pointed out, most of the user-facing features of an IDE depend on some
> underlying support to detect and configure projects.
>
> Now, it is not clear what "detect and configure" really means, but it seems that
> EDE (CEDET's project management component) is not fully satisfying or simple
> enough to extend/configure at this point in time.
>
> I've formed my own opinions on how this could be improved, but I would
> like to hear what others think before discussing how to do it. So,
> what is required from "project management"?

For me the most important thing is not to do things behind my back based
on secondguessing which are hard to control/configure.

-- 
David Kastrup

Re: Project detection and configuration

[Lluís]

David Kastrup writes:

> Lluís <xscript@gmx.net> writes:
>> Hi!
>> 
>> As Eric pointed out, most of the user-facing features of an IDE depend on some
>> underlying support to detect and configure projects.
>> 
>> Now, it is not clear what "detect and configure" really means, but it seems that
>> EDE (CEDET's project management component) is not fully satisfying or simple
>> enough to extend/configure at this point in time.
>> 
>> I've formed my own opinions on how this could be improved, but I would
>> like to hear what others think before discussing how to do it. So,
>> what is required from "project management"?

> For me the most important thing is not to do things behind my back based
> on secondguessing which are hard to control/configure.

Right, customizing the behaviour should be as simple as possible.


Cheers,
  Lluis

-- 
"And it's much the same thing with knowledge, for whenever you learn
something new, the whole world becomes that much richer."
-- The Princess of Pure Reason, as told by Norton Juster in The Phantom
Tollbooth

Re: Project detection and configuration

[John Wiegley]

>>>>> Lluís  <xscript@gmx.net> writes:

> I've formed my own opinions on how this could be improved, but I would like
> to hear what others think before discussing how to do it. So, what is
> required from "project management"? A few things quickly come into mind:

Thank you, Lluís, I've saved your list locally so that I can refer to it as we
proceed with our design discussions.

John