progmodes/project.el and search paths

progmodes/project.el and search paths

[Eric Ludlam]

[-- Attachment #1: Type: text/plain, Size: 1384 bytes --]

I have a some thoughts related to the project.el project being discussed 
in the mailing list.  When it was first proposed as a set of core 
functions to generically describe a project, I thought it could be a 
good idea, since there are a several 'projects' out there, such as EDE, 
projectile, jdee, and a few others I know less about.  It would save 
some tools the trouble of knowing how to integrate all those 
possibilities if all they needed was some random piece of project metadata.

project.el and the current discussion seems to revolve almost entirely 
around search paths and xref.  EDE does project management, but has not 
needed to track search paths the way project.el does.  It certainly has 
ways to find files, and has include paths and such needed by source 
code, but not generic search paths.  I think project.el should be recast 
as defining search paths for search tools.  EDE could then plug itself 
in to provide some paths if asked.  I think calling it a "project" is 
overstating what project.el does.

In the meantime, attached is a small patch to enable EDE to provide 
roots to project.el.  The new function works for me with 24.3, but I 
didn't try it in project.el

Perhaps all the work you're doing could just hang off ede since it only 
takes 2 lines of code, and if it is missing some sort of pruning tricks, 
it could just be added to ede.

Eric

[-- Attachment #2: project.el.patch --]
[-- Type: text/x-patch, Size: 1040 bytes --]

diff --git a/lisp/progmodes/project.el b/lisp/progmodes/project.el
index d849f93..3fd1358 100644
--- a/lisp/progmodes/project.el
+++ b/lisp/progmodes/project.el
@@ -27,7 +27,8 @@
 
 (require 'cl-generic)
 
-(defvar project-find-functions (list #'project-try-vc
+(defvar project-find-functions (list #'project-try-ede
+                                     #'project-try-vc
                                      #'project-ask-user)
   "Special hook to find the project containing a given directory.
 Each functions on this hook is called in turn with one
@@ -99,6 +100,13 @@ an element of `project-search-path'."
                               (vc-call-backend backend 'root dir)))))
     (and root (cons 'vc root))))
 
+(defun project-try-ede (dir)
+  "Use EDE to determine if DIR is a project directory."
+  (when (featurep 'ede)
+    (let* ((eproj (ede-current-project dir))
+           (eroot (ede-project-root-directory eproj)))
+      (cons 'ede eroot))))
+
 (cl-defmethod project-roots ((project (head vc)))
   (list (cdr project)))
 

Re: progmodes/project.el and search paths

[Dmitry Gutov]

Hi Eric,

On 08/02/2015 04:52 PM, Eric Ludlam wrote:

> project.el and the current discussion seems to revolve almost entirely
> around search paths and xref.  EDE does project management, but has not
> needed to track search paths the way project.el does.  It certainly has
> ways to find files, and has include paths and such needed by source
> code, but not generic search paths.  I think project.el should be recast
> as defining search paths for search tools.  EDE could then plug itself
> in to provide some paths if asked.  I think calling it a "project" is
> overstating what project.el does.

The intention is to provide a generic API that other packages can use. 
The roots and search-path looked to be the most important things to know 
about a project, so far.

If you can propose better ways to do the same things, or something else 
worth adding, please do.

> In the meantime, attached is a small patch to enable EDE to provide
> roots to project.el.  The new function works for me with 24.3, but I
> didn't try it in project.el

I've implemented EDE support similarly to your patch initially, but then 
Stefan opined it's better to dispatch on the actual EIEIO class. This 
implementation lives near the bottom of ede.el in Emacs master. Please 
take a look and try it sometime.

> Perhaps all the work you're doing could just hang off ede since it only
> takes 2 lines of code, and if it is missing some sort of pruning tricks,
> it could just be added to ede.

project.el defines a generic API. EDE is one implementation.

One that a lot of Emacs users still avoid using (hard to tell exactly 
why; probably because it doesn't match every kind of project). 
Projectile seems to the most popular "project implementation" out there.

Anyway, the suggestion to replace an agnostic API with a specific 
implementation doesn't make a lot of sense.

Re: progmodes/project.el and search paths

[Eric Ludlam]

On 08/02/2015 01:17 PM, Dmitry Gutov wrote:
> Hi Eric,
>
> On 08/02/2015 04:52 PM, Eric Ludlam wrote:
>
>> project.el and the current discussion seems to revolve almost entirely
>> around search paths and xref.  EDE does project management, but has not
>> needed to track search paths the way project.el does.  It certainly has
>> ways to find files, and has include paths and such needed by source
>> code, but not generic search paths.  I think project.el should be recast
>> as defining search paths for search tools.  EDE could then plug itself
>> in to provide some paths if asked.  I think calling it a "project" is
>> overstating what project.el does.
>
> The intention is to provide a generic API that other packages can use. 
> The roots and search-path looked to be the most important things to 
> know about a project, so far.
>
> If you can propose better ways to do the same things, or something 
> else worth adding, please do.

I apologize for not being up on the full thread.  It was very long and I 
didn't really get what the goals were.

One of my hopes for Emacs is that it will keep tabs on what I'm up to to 
provide useful context relevant feedback when I ask for it. My 
experience has been that tools that depend on a project to ask what the 
current project is do so a lot.  As such, I cache the current project, 
and current root project as buffer local variables.  Those project 
structures then have relevant meta data as well.  Another tactic is 
initializing each buffer into a project as it is created.  That moves 
search time into a different timing that is less noticeable than 
on-demand when the user makes a request.

Also, identifying projects should be a service of the core project 
system.  Dependent projects need to provide detection data without 
having Emacs load the full project definition.  That way you can have 
lots of projects types without forcing Emacs to load piles of code that 
you don't need.  My opinion is that all the projects should be available 
for detection without the user having to custom add anything about which 
they want.  Externally supplied projects (those not part of core emacs) 
can have a user provide a small snippet for detection or could do a 
trick with autoloads.

By way of example, EDE's autoloader does this well, and is wrapped in a 
complex test suite to make sure things don't get loaded when detecting 
projects of a different type.  I re-wrote most of it last year and it is 
much better than it was previously.  Use of 'locate-dominating-file' was 
a big win.

ede/detect.el is relatively independent of most of EDE's other 
functionality.  It does depend on ede/auto.el which defines a data 
structure for detecting projects.  Note that the important class is 
ede-project-autoload which provides basic matching data.  For more 
complex matching tasks, ede-project-autoload-dirmatch provides a little 
more.  Then over in ede/generic.el, you will find 
ede-enable-generic-projects, which has a simplified method for creating 
autoloads for the most basic of project types, including various vc 
types.  Your vc detector looks like it uses some nicer infrastructure 
than the simple file system matcher in generic.  The point is simply a 
simplified matcher construct.

If you'd like to adopt EDE's detector and rename bits of it 'project', 
that would work well for EDE (obviously) and would get you a very 
robust, flexible, and tested project detection system.  I don't think 
you can re-use EDE's caching without wholesale adoption of ede since it 
depends on ede's project classes.

Anyway, all that is just about matching projects.  Projects then also 
have meta data.  Things like compilation configuration, compile 
commands, release commands, include paths, classpaths, and whatnot. I 
don't know what you intend with project meta data.  If there was a 
thread about that, I missed it, sorry.

>> In the meantime, attached is a small patch to enable EDE to provide
>> roots to project.el.  The new function works for me with 24.3, but I
>> didn't try it in project.el
>
> I've implemented EDE support similarly to your patch initially, but 
> then Stefan opined it's better to dispatch on the actual EIEIO class. 
> This implementation lives near the bottom of ede.el in Emacs master. 
> Please take a look and try it sometime.

Thanks for the pointer.  My Emacs25 didn't compile when I pulled from 
git today, so I didn't get to try it or go very far beyond reading the 
latest in project.el.

For your implementation in ede.el, you are welcome to use what I wrote.  
If EDE is active, you can just reach out with those functions to get the 
same directory names.  For a subset of projects you can get other roots 
for different targets too.  I wasn't sure what the use case was for that 
was.

>> Perhaps all the work you're doing could just hang off ede since it only
>> takes 2 lines of code, and if it is missing some sort of pruning tricks,
>> it could just be added to ede.
>
> project.el defines a generic API. EDE is one implementation.

EDE used to be an implementation of a project management system that 
could edit Makefiles and Automakefiles.  The demands of other tools like 
semantic forced it to switch to a generic system that can detect 
arbitrary patterns of files that indicate projects created an maintained 
by other systems.  It then wraps those with detection code, and when 
possible pushes useful features such as compile commands, debugging, 
configuration options and more out to the user.

I think EDE has a generic API for defining a project.  It probably looks 
big on the outside if you just read the doc where it starts with 
"ede-new", but for most cases, if you enable EDE, it will find your 
Emacs src, Linux src, Cmake based build, etc, and it will magically 
provide that to Semantic's parser, to location functions (if enabled) 
and to taggers like global (if enabled.)   The reason it looks big is 
because those are the things that were needed while I was working on 
other tools that needed project support.  I would guess your system will 
grow to a similar state if you keep adding features needed by tools that 
want a project.  It is a really big task.

> One that a lot of Emacs users still avoid using (hard to tell exactly 
> why; 

Someone pointed out on the cedet mailing list recently that the doc (and 
probably the website) is too focused on how EDE can create makefiles 
which most people don't want.  Most questions on my mailing list ask how 
to get some other tool like smart completion in semantic working, and 
when they find out that they just need to turn on EDE, and not use the 
full project management system, they are pretty happy.

> probably because it doesn't match every kind of project). Projectile 
> seems to the most popular "project implementation" out there.

Projectile is a nice interface for moving around between files.  It 
happens to have a project detector in it.  Most of projectile is nice 
stuff that could fit easily on top of EDE's project detection without 
overlapping features of either tool.  Projectile has definitions for 
projects EDE does not have because various CEDET developers haven't 
worked with those tools.  The opposite is probably true too.

> Anyway, the suggestion to replace an agnostic API with a specific 
> implementation doesn't make a lot of sense.

If EDE is considered specific because it is called ede, or because it 
uses eieio, that would be too bad.  It's purpose is to generically 
detect projects such that anyone could create a new project definition, 
find project metadata, and make it accessible to other tools.  This has 
already happened many times.  Supporting EDE is more difficult than just 
providing a dominating file name because it also wants to find extra 
meta data.  I think that is a pretty generic task also.

One way EDE could be made more generic is to make the 'target' concept 
optional.  Simple ede projects also have to define targets as part of 
the detection and matching system, so making them optional would 
simplify creating new projects types in EDE.

A second way for EDE to be more generic is to better support 
ede/generic.el, or perhaps have something like ede/generic.el that 
leaves off all the extra meta-data handling.

Another way EDE could be made more generic is by having built-in tools 
like the 'compile' command ask EDE how to compile, or by having flymake 
use EDE to autodetect how to run make, or perhaps etags could ask EDE 
where to run etags from, and a host of other options.  I had pulled 
those features out into ede named features to keep things independent 
when CEDET was an independent tool.  I don't see why naming conventions 
couldn't change to make things better.

Eric

Re: progmodes/project.el and search paths

[David Engster]

Dmitry Gutov writes:
> On 08/02/2015 04:52 PM, Eric Ludlam wrote:
>> Perhaps all the work you're doing could just hang off ede since it only
>> takes 2 lines of code, and if it is missing some sort of pruning tricks,
>> it could just be added to ede.
>
> project.el defines a generic API. EDE is one implementation.

From what I see in project.el, this is an API for defining a set of
directories. I'm not saying that Emacs does not need such an API, but I
would not call this a "project API". What about things like setting up
toolchains (compiler, linker, debugger), configurations (debug/release),
support for external build systems, setting up environment variables and
pre-processor symbols, and so on? I mean, this is the bread and butter
of projects in pretty much any IDE. Will you add this to project,el
eventually?  Otherwise, calling EDE an implementation of your generic
project API is pretty daring, IMHO.

-David

Re: progmodes/project.el and search paths

[Dmitry Gutov]

On 08/03/2015 04:49 PM, David Engster wrote:

>  From what I see in project.el, this is an API for defining a set of
> directories. I'm not saying that Emacs does not need such an API, but I
> would not call this a "project API".

These are the traits common to most projects that we've identified so 
far. Ones that a third-party package will almost always be able to use.

> What about things like setting up
> toolchains (compiler, linker, debugger), configurations (debug/release),
> support for external build systems, setting up environment variables and
> pre-processor symbols, and so on?

What linker and pre-processor? Seriously, you're talking to a Ruby 
developer here.

What aspects of "setting up a debugger" would you consider generalizable 
over different project systems? Same question about "support for 
external build systems".

Do you need help setting up environment variables? If that comes down to 
editing a list, then I suppose "environment variables" could become a 
standard piece of project metadata. Especially if someone out there is 
interested in writing a env var editor.

> I mean, this is the bread and butter
> of projects in pretty much any IDE. Will you add this to project,el
> eventually?

If people such as yourself help identify more pieces of knowledge that 
apply to most projects out there, I'd be happy to add them.

> Otherwise, calling EDE an implementation of your generic
> project API is pretty daring, IMHO.

Well, as you've put it, this is obviously false.

EDE is *an* implementation. Like some class can implement some (often 
small) interface. This metaphor should be familiar to a C++ programmer.

Aside from that, EDE is a project system obviously geared towards C/C++ 
development.

Re: progmodes/project.el and search paths

[David Engster]

Dmitry Gutov writes:
> On 08/03/2015 04:49 PM, David Engster wrote:
>
>>  From what I see in project.el, this is an API for defining a set of
>> directories. I'm not saying that Emacs does not need such an API, but I
>> would not call this a "project API".
>
> These are the traits common to most projects that we've identified so
> far. Ones that a third-party package will almost always be able to
> use.

As I've said: I'm not saying it's not useful.

>> What about things like setting up
>> toolchains (compiler, linker, debugger), configurations (debug/release),
>> support for external build systems, setting up environment variables and
>> pre-processor symbols, and so on?
>
> What linker and pre-processor? Seriously, you're talking to a Ruby
> developer here.

Just because Ruby does not have them, project.el should not have support
for different toolchains?

> What aspects of "setting up a debugger" would you consider
> generalizable over different project systems?

A debugger needs to know where the executable is and how it should be
called.

> Same question about "support for external build systems".

If you say "compile this project", you will usually not call the
compiler directly but run some external build systems. Depending on the
current configuration (debug/release), it must be called differently.

> Do you need help setting up environment variables?

Yes. I might need to set things like CFLAGS dependend on the current
configuration. Or I might want to cross-compile, in which case I need to
set ARCH and CROSS_COMPILE.

>> of projects in pretty much any IDE. Will you add this to project,el
>> eventually?
>
> If people such as yourself help identify more pieces of knowledge that
> apply to most projects out there, I'd be happy to add them.

I've mentioned a few. If you add support for this, your simple API will
grow, and I wouldn't be surprised if in the end it looks a bit like EDE.

> Aside from that, EDE is a project system obviously geared towards
> C/C++ development.

It can (and does) support other languages.

-David

Re: progmodes/project.el and search paths

[Eli Zaretskii]

> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Mon, 3 Aug 2015 17:09:29 +0300
> Cc: Emacs Development <emacs-devel@gnu.org>,
> 	Eric Ludlam <eric@siege-engine.com>
> 
> > What about things like setting up
> > toolchains (compiler, linker, debugger), configurations (debug/release),
> > support for external build systems, setting up environment variables and
> > pre-processor symbols, and so on?
> 
> What linker and pre-processor? Seriously, you're talking to a Ruby 
> developer here.

I sincerely hope the results of this development will support more
than just Ruby.  Most languages today, even those that are "normally"
interpreted, have some kind of compiler.  And even Ruby might grow a
compiler some day.  I think supporting that is a no-brainer.

Re: progmodes/project.el and search paths

[Dmitry Gutov]

On 08/03/2015 05:27 PM, David Engster wrote:

>> What linker and pre-processor? Seriously, you're talking to a Ruby
>> developer here.
>
> Just because Ruby does not have them, project.el should not have support
> for different toolchains?

I'm not sure yet how to fit the toolchain-specialized aspects best. 
Maybe we'll have a set of methods that are optional to implement. Or 
different "types" of projects the consumer will have to "cast" the 
project object to.

In any case, you'll need to present a use case first. If that 
information will only be consumed by functions inside EDE itself, 
there's not much use in making it a part of the general API.

>> What aspects of "setting up a debugger" would you consider
>> generalizable over different project systems?
>
> A debugger needs to know where the executable is and how it should be
> called.

Just for your reference, the ways I tend to use the debugger in Ruby 
projects are quite different from calling it with a path to the executable.

Or consider some Java web application project. To launch it in debug 
mode, you pass an argument to the application launcher, and then an 
application (maybe the IDE itself) connects to it via a socket.

> If you say "compile this project", you will usually not call the
> compiler directly but run some external build systems. Depending on the
> current configuration (debug/release), it must be called differently.

Is there something there to be automated in Emacs, that will save the 
user a lot of effort, compared to 'make env=DEBUG' in the console?

> Yes. I might need to set things like CFLAGS dependend on the current
> configuration. Or I might want to cross-compile, in which case I need to
> set ARCH and CROSS_COMPILE.

So, the API would not only list the variables, but also somehow describe 
the possible values.

>> If people such as yourself help identify more pieces of knowledge that
>> apply to most projects out there, I'd be happy to add them.
>
> I've mentioned a few.

I hope you can see that there's more design work to be done that just 
mentioning them.

> If you add support for this, your simple API will
> grow, and I wouldn't be surprised if in the end it looks a bit like EDE.

I guess we'll have to see. But for now, compare the 140-line long 
project.el and 12000 lines in ede.el and ede/*.el combined.

>> Aside from that, EDE is a project system obviously geared towards
>> C/C++ development.
>
> It can (and does) support other languages.

It contains a lot of stuff that's useless from a Ruby developer's 
standpoint (or Python, or Elisp, or R), and it doesn't compensate that 
with a lot of features.

Re: progmodes/project.el and search paths

[Dmitry Gutov]

On 08/03/2015 06:09 PM, Eli Zaretskii wrote:

> I sincerely hope the results of this development will support more
> than just Ruby.

Obviously.

> Most languages today, even those that are "normally"
> interpreted, have some kind of compiler.  And even Ruby might grow a
> compiler some day.  I think supporting that is a no-brainer.

For all I know, when talking about "supporting a compiler", you and 
David have entirely different use cases in mind.

It'll need a concrete design proposal.

Re: progmodes/project.el and search paths

[Eli Zaretskii]

> Cc: deng@randomsample.de, emacs-devel@gnu.org, eric@siege-engine.com
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Mon, 3 Aug 2015 18:16:44 +0300
> 
> > Most languages today, even those that are "normally"
> > interpreted, have some kind of compiler.  And even Ruby might grow a
> > compiler some day.  I think supporting that is a no-brainer.
> 
> For all I know, when talking about "supporting a compiler", you and 
> David have entirely different use cases in mind.

I don't think so, because everything David said made perfect sense to
me.

Re: progmodes/project.el and search paths

[Stephen Leake]

Eric Ludlam <ericludlam@gmail.com> writes:

> I apologize for not being up on the full thread.  It was very long and
> I didn't really get what the goals were.

I still don't get what the goals are ;).

> If you'd like to adopt EDE's detector and rename bits of it 'project',
> that would work well for EDE (obviously) and would get you a very
> robust, flexible, and tested project detection system.  I don't think
> you can re-use EDE's caching without wholesale adoption of ede since
> it depends on ede's project classes.

That makes sense to me. In the context of project.el, this detection
would return both the "project backend" (the elisp code to load) and
any relevant "project file", if the backend uses one.

I agree the result of that detection should be cached, rather than
recomputed each time it is needed.

I prefer to cache it in a global variable; that way, I have only one
project active at a time, for all buffers. So if I'm debugging the elisp
code that implements Ada mode, I get elisp search and completion paths
from Ada mode buffers, as well as elisp buffers.

> Anyway, all that is just about matching projects.  Projects then also
> have meta data.  Things like compilation configuration, compile
> commands, release commands, include paths, classpaths, and whatnot.

I think that level of metadata belongs on a backend, not the core. Some
projects don't need all of that data (elisp and Ada mode for example).

> EDE used to be an implementation of a project management system that
> could edit Makefiles and Automakefiles.  The demands of other tools
> like semantic forced it to switch to a generic system that can detect
> arbitrary patterns of files that indicate projects created an
> maintained by other systems.  It then wraps those with detection code,
> and when possible pushes useful features such as compile commands,
> debugging, configuration options and more out to the user.

I'll see if I can implement Ada, JDEE, and elisp projects as EDE
backends; that will allow a detailed comparison to the project backends
I'm also writing.

> I think EDE has a generic API for defining a project. It probably
> looks big on the outside if you just read the doc where it starts with
> "ede-new", but for most cases, if you enable EDE, it will find your
> Emacs src, Linux src, Cmake based build, etc, and it will magically
> provide that to Semantic's parser, to location functions (if enabled)
> and to taggers like global (if enabled.)

Ada mode has its own parser and xref tool, and never uses CMake. So at
first glance it seems like there is too much in the EDE core.

Similarly for JDEE.

> The reason it looks big is because those are the things that
> were needed while I was working on other tools that needed project
> support. I would guess your system will grow to a similar state if you
> keep adding features needed by tools that want a project. It is a
> really big task.

Right. And we should leverage the work that has gone into EDE.

I suspect we are really talking about refactoring EDE so the core is
smaller.

That is really just an optimization, and may not be worth it, if the
overhead the present core imposes is not noticeable.

>> One that a lot of Emacs users still avoid using (hard to tell
>> exactly why;
>
> Someone pointed out on the cedet mailing list recently that the doc
> (and probably the website) is too focused on how EDE can create
> makefiles which most people don't want.

Yes; that is the first doc I saw (I don't remember where now, sorry).
The intro at
https://www.gnu.org/software/emacs/manual/html_node/emacs/EDE.html is
better. Although it could mention that it is possible to add new project
types (I guess that's just assumed with Emacs :).

>> probably because it doesn't match every kind of project). Projectile
>> seems to the most popular "project implementation" out there.
>
> Projectile is a nice interface for moving around between files.  It
> happens to have a project detector in it.  Most of projectile is nice
> stuff that could fit easily on top of EDE's project detection without
> overlapping features of either tool.  Projectile has definitions for
> projects EDE does not have because various CEDET developers haven't
> worked with those tools.  The opposite is probably true too.

Which is one reason to have an "independent" project core; perhaps the
projectile team will be more amenable to adopting a core API that isn't
named "ede"?

>> Anyway, the suggestion to replace an agnostic API with a specific
>> implementation doesn't make a lot of sense.
>
> If EDE is considered specific because it is called ede, or because it
> uses eieio, that would be too bad.

Right. "EDE" is "Emacs Development Environment"; that's a good generic
name.

eieio is apparently a little more problematic; some people are still
alergic to object oriented programming. For example, projects.el and
xref.el use several random function variables, in addition to
dispatching via cl-defmethod.

There are also restrictions for elisp, since it is preloaded. I think
the current rule is cl-lib can be preloaded, but not eieio?

> It's purpose is to generically
> detect projects such that anyone could create a new project
> definition, find project metadata, and make it accessible to other
> tools.  This has already happened many times.  Supporting EDE is more
> difficult than just providing a dominating file name because it also
> wants to find extra meta data.  I think that is a pretty generic task
> also.

If it _insists_ on collecting meta data that is simply not defined (for
example "the location of the CMake file in this Ada project"), and can't
handle either nil or signal as a result, that's a problem.

> One way EDE could be made more generic is to make the 'target' concept
> optional.  Simple ede projects also have to define targets as part of
> the detection and matching system, so making them optional would
> simplify creating new projects types in EDE.

That would help; anything to reduce the burden of getting a new project
backend working is good.


> Another way EDE could be made more generic is by having built-in tools
> like the 'compile' command ask EDE how to compile, or by having
> flymake use EDE to autodetect how to run make, or perhaps etags could
> ask EDE where to run etags from, and a host of other options.  I had
> pulled those features out into ede named features to keep things
> independent when CEDET was an independent tool.  I don't see why
> naming conventions couldn't change to make things better.

Yes. xref is going the same root as EDE took; there is xref-find-regexp.
I agree it would be better (if projects are enabled) for grep to use
project information for the search path.

--
-- Stephe

Re: progmodes/project.el and search paths

[Stephen Leake]

David Engster <deng@randomsample.de> writes:

> From what I see in project.el, this is an API for defining a set of
> directories. I'm not saying that Emacs does not need such an API, but I
> would not call this a "project API". What about things like setting up
> toolchains (compiler, linker, debugger), 

Ada mode uses AdaCore gprbuild, which uses a project file that defines
all that stuff. No help from Emacs needed. And so far, the only thing
Emacs needs from the project file is the source path, for
compilation-source-path.

> configurations (debug/release), support for external build systems,

To support a build system, you provide a project backend.

I assume I could provide an EDE backend for gprbuild, and another for
gradle (for Android Java projects). 

> setting up environment variables and pre-processor symbols, 

No environment variables for Ada, Java only needs CLASSPATH, which is
set from the JDEE project file (so that could use an EDE feature?). No
preprocessor in Ada or Java.

> and so on? I mean, this is the bread and butter of projects in pretty
> much any IDE. 

I suspect there may be self-selection going on here; people/project
backends that need all of those features use EDE, and new project
backends get written.

But for people/projects that don't need all of those things, EDE seems
overwhelming.

So it seems like we are discussing refactoring EDE, and adding some
features it doesn't currently have.

-- 
-- Stephe

Re: progmodes/project.el and search paths

[Stephen Leake]

David Engster <deng@randomsample.de> writes:

> Dmitry Gutov writes:
>> On 08/03/2015 04:49 PM, David Engster wrote:
>>
>>>  From what I see in project.el, this is an API for defining a set of
>>> directories. I'm not saying that Emacs does not need such an API, but I
>>> would not call this a "project API".
>>
>> These are the traits common to most projects that we've identified so
>> far. Ones that a third-party package will almost always be able to
>> use.
>
> As I've said: I'm not saying it's not useful.
>
>>> What about things like setting up
>>> toolchains (compiler, linker, debugger), configurations (debug/release),
>>> support for external build systems, setting up environment variables and
>>> pre-processor symbols, and so on?
>>
>> What linker and pre-processor? Seriously, you're talking to a Ruby
>> developer here.
>
> Just because Ruby does not have them, project.el should not have support
> for different toolchains?

It depends on whether there are really common features across several
toolchains.

I guess the top-level API for these things in EDE is common across
several toolchains, but I'm not clear; you've talked about some things
are in EDE 'only for Java', or only for some other backend.

I agree the core API should allow all possible backends, but there's a
trade-off. If the core includes stuff that only one or a few backends
need, then that imposes extra overhead (if only by overloading the
implementer understanding of the API).

>> What aspects of "setting up a debugger" would you consider
>> generalizable over different project systems?
>
> A debugger needs to know where the executable is and how it should be
> called.

gud does that, but if a new debugger needs to be added, perhaps gud
should consult the project system first, rather than having a separate
way to add a debugger.

> If you say "compile this project", you will usually not call the
> compiler directly but run some external build systems. Depending on the
> current configuration (debug/release), it must be called differently.

Right. I do that by editing Makefiles directly; M-x compile is always
"make -r <target>" for me. I've always felt that anything else just gets
in the way. Clearly I'm in the minority on this one :(>.

Although I've recently started working on JDEE, and it doesn't have a
Makefile, just an ant build.xml. So I'm trying to force myself to not
add a Makefile. We'll see. So far it's very frustrating trying to track
down exactly what ant is doing; must easier with a makefile.

>> Do you need help setting up environment variables?
>
> Yes. I might need to set things like CFLAGS dependend on the current
> configuration. Or I might want to cross-compile, in which case I need to
> set ARCH and CROSS_COMPILE.

These seem very backend specific. How much of this is really in the EDE
core, and how much in some "make/gcc backend"?

-- 
-- Stephe

Re: progmodes/project.el and search paths

[Eli Zaretskii]

> From: Stephen Leake <stephen_leake@stephe-leake.org>
> Date: Mon, 03 Aug 2015 11:35:11 -0500
> 
> > A debugger needs to know where the executable is and how it should be
> > called.
> 
> gud does that

No, it doesn't.  It leaves it to the user to type the exact file name
every single time.

Re: progmodes/project.el and search paths

[Dmitry Gutov]

On 08/03/2015 06:29 PM, Eli Zaretskii wrote:

> I don't think so, because everything David said made perfect sense to
> me.

IIUC, (and I'm mostly guessing), you want the project to report on the 
program a debugger should run.

To do that, we can introduce a project-attributes dict, and it may, or 
may not, include a key `debugger-program'.

This looks like a low-cost, low-benefit proposal. Anything else?

Re: progmodes/project.el and search paths

[Eli Zaretskii]

> Cc: deng@randomsample.de, emacs-devel@gnu.org, eric@siege-engine.com
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Mon, 3 Aug 2015 22:01:17 +0300
> 
> IIUC, (and I'm mostly guessing), you want the project to report on the 
> program a debugger should run.

David mentioned more than just that.

Re: progmodes/project.el and search paths

[Dmitry Gutov]

On 08/03/2015 10:19 PM, Eli Zaretskii wrote:

> David mentioned more than just that.

Unfortunately, I couldn't make out any other specific proposals.

Re: progmodes/project.el and search paths

[Stephen Leake]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Stephen Leake <stephen_leake@stephe-leake.org>
>> Date: Mon, 03 Aug 2015 11:35:11 -0500
>> 
>> > A debugger needs to know where the executable is and how it should be
>> > called.
>> 
>> gud does that
>
> No, it doesn't.  It leaves it to the user to type the exact file name
> every single time.

Ah; I was confusing "the debugger exectuable" and "the user executable
being debugged".

It does remember the last file debugged, so it's not "every single time"
:).

In addition, when I build things in makefiles, gud does a good job of
guessing the correct user executable, and the others are available thru
file completion (since they are all in the same directory).

I don't see how a project can do better; there will still be a list of
possible executables; the most recently compiled isn't necessarily the
one I want to debug.

-- 
-- Stephe

Re: progmodes/project.el and search paths

[David Engster]

Stephen Leake writes:
> In addition, when I build things in makefiles, gud does a good job of
> guessing the correct user executable, and the others are available thru
> file completion (since they are all in the same directory).

Not for me. Executables are all over the place and I always have to make
sure I switch to the correct directory first. Gud frequently guesses
wrong, probably because we have a very complicated parallel build.
Also, I frequently have to switch between debugging locally and remotely
on the target platform through gdbserver.

> I don't see how a project can do better; there will still be a list of
> possible executables; the most recently compiled isn't necessarily the
> one I want to debug.

Well, let's take Eclipse. It automatically scans the tree for binaries
which you can quickly select for debugging. You can define different
debug configurations where you can define the debugger to use, how it
should be called and with which init file, if and where it should stop
execution on startup, which executable it should debug and what
arguments and environment it should provide, and if maybe this
executable should be copied to a target system and debugged through
gdbserver...

And that's just what *I* regularly use.

-David

Re: progmodes/project.el and search paths

[Stefan Monnier]

> From what I see in project.el, this is an API for defining a set of
> directories.  I'm not saying that Emacs does not need such an API, but I
> would not call this a "project API".  What about things like setting up
> toolchains (compiler, linker, debugger), configurations (debug/release),
> support for external build systems, setting up environment variables and
> pre-processor symbols, and so on?

The point of view is very different: project.el is meant to provide to
the rest of Emacs the information needed to perform operations that
require knowledge about "the current project".

So the API is designed as a kind of "intersection (rather than union) of
all possible project systems".

Currently the "rest of Emacs" part that uses project knowledge is:
- text search within the project
- xref thingies
We could/should/will extend the API if there's a need for it in order to
implement further generic functionality.  I can imagine a few such
generic functions:
- "make"
- "run"
- flymake
I don't think any of those requires exposing concepts like toolchains or
compilers in the API (tho any particular implementation of that API
may/will probably be tailored to some particular toolchain/compiler
and/or will need to be taught bout the toolchain/compiler setup).


        Stefan

Re: progmodes/project.el and search paths

[David Engster]

Dmitry Gutov writes:
> On 08/03/2015 05:27 PM, David Engster wrote:
>
>>> What linker and pre-processor? Seriously, you're talking to a Ruby
>>> developer here.
>>
>> Just because Ruby does not have them, project.el should not have support
>> for different toolchains?
>
> I'm not sure yet how to fit the toolchain-specialized aspects
> best. Maybe we'll have a set of methods that are optional to
> implement.

You mean like ede-compile-project?

> In any case, you'll need to present a use case first.

I hit a key, the project gets compiled by calling some build command
based on the currently selected configuration.

> If that information will only be consumed by functions inside EDE
> itself, there's not much use in making it a part of the general API.

I don't understand this.

>>> What aspects of "setting up a debugger" would you consider
>>> generalizable over different project systems?
>>
>> A debugger needs to know where the executable is and how it should be
>> called.
>
> Just for your reference, the ways I tend to use the debugger in Ruby
> projects are quite different from calling it with a path to the
> executable.
>
> Or consider some Java web application project. To launch it in debug
> mode, you pass an argument to the application launcher, and then an
> application (maybe the IDE itself) connects to it via a socket.

Yes, the notion of "debugging the project" varies wildly; same goes for
"running the executable". You can also add remote debugging on a target
platform through a debug server. Why not try to support it all?

>> If you say "compile this project", you will usually not call the
>> compiler directly but run some external build systems. Depending on the
>> current configuration (debug/release), it must be called differently.
>
> Is there something there to be automated in Emacs, that will save the
> user a lot of effort, compared to 'make env=DEBUG' in the console?

Every switch away from Emacs is a distraction. These are things I have
to do *many* times a day. Besides, I need the Make output in a
compilation buffer, so that I can jump to errors, etc.

>> Yes. I might need to set things like CFLAGS dependend on the current
>> configuration. Or I might want to cross-compile, in which case I need to
>> set ARCH and CROSS_COMPILE.
>
> So, the API would not only list the variables, but also somehow
> describe the possible values.

The user can switch between different configurations, and dependend on
the current configuration a certain set of environment variables is used
when the build system is called.

>>> If people such as yourself help identify more pieces of knowledge that
>>> apply to most projects out there, I'd be happy to add them.
>>
>> I've mentioned a few.
>
> I hope you can see that there's more design work to be done that just
> mentioning them.

Of course, but since I'm quite happy with EDE, that's not an itch I need
to scratch. You volunteered to create a project API, and I simply state
what I would expect from such an API to be usable for me, as an EDE
user. I don't want to sound so negative, and I surely don't want to
discourage your efforts. A simple API that defines a certain set of
directories based on the current buffer could cover a lot of what users
need. While I still think that EDE could support this, it has the huge
drawback that it cannot be automatically loaded at startup because of
EIEIO.

>> If you add support for this, your simple API will
>> grow, and I wouldn't be surprised if in the end it looks a bit like EDE.
>
> I guess we'll have to see. But for now, compare the 140-line long
> project.el and 12000 lines in ede.el and ede/*.el combined.

You are comparing what is essentially an interface definition, covering
a tiny amount of the EDE API, against the whole EDE suite. That factor
is hardly surprising.

I'm the first to say that EDE needs a serious overhaul. I also wouldn't
object to simply require that the user already has a working build
through whatever build system she chooses (make, ninja, ant, gradle,
rake, you name it), which would mean we could drop a lot of stuff like
the automatic Makefile generation.

>>> Aside from that, EDE is a project system obviously geared towards
>>> C/C++ development.
>>
>> It can (and does) support other languages.
>
> It contains a lot of stuff that's useless from a Ruby developer's
> standpoint (or Python, or Elisp, or R), and it doesn't compensate that
> with a lot of features.

So if those features are only needed by compiled languages they don't
count? This indeed will keep things simple...

-David

Re: progmodes/project.el and search paths

[David Engster]

Stefan Monnier writes:
>> From what I see in project.el, this is an API for defining a set of
>> directories.  I'm not saying that Emacs does not need such an API, but I
>> would not call this a "project API".  What about things like setting up
>> toolchains (compiler, linker, debugger), configurations (debug/release),
>> support for external build systems, setting up environment variables and
>> pre-processor symbols, and so on?
>
> The point of view is very different: project.el is meant to provide to
> the rest of Emacs the information needed to perform operations that
> require knowledge about "the current project".

And what constitutes "the current project", exactly? Only the list of
directories associated with it?

> So the API is designed as a kind of "intersection (rather than union) of
> all possible project systems".

That intersection is tiny.

-David

Re: progmodes/project.el and search paths

[Dmitry Gutov]

Sorry for the late reply.

On 08/03/2015 04:19 AM, Eric Ludlam wrote:

> I apologize for not being up on the full thread.  It was very long and I
> didn't really get what the goals were.

I would be lying if I said I got everything figured out. One "unified 
project interface" proposal surfaced a while back, and eventually 
fizzled out. I decided to revisit the subject when I was writing a 
couple of commands that needed some information about the current 
project, and here we are.

Those turned out to be pretty useful, by the way.

> One of my hopes for Emacs is that it will keep tabs on what I'm up to to
> provide useful context relevant feedback when I ask for it.

That sounds like something Flymake or Flycheck would do. Or a completion 
system.

> My
> experience has been that tools that depend on a project to ask what the
> current project is do so a lot.  As such, I cache the current project,
> and current root project as buffer local variables.  Those project
> structures then have relevant meta data as well.  Another tactic is
> initializing each buffer into a project as it is created.  That moves
> search time into a different timing that is less noticeable than
> on-demand when the user makes a request.

IME, (vc-call-backend backend 'root dir) takes a negligible amount of 
time, so we might as well call it once per command. Of course, it might 
be different when using Tramp, but both Tramp and VC do some caching of 
their own. Other project implementations are free to employ their own 
caching methods (might be buffer-local, or if the implementation 
considers that only one project might be enabled at a time, simply in 
its own variables).

> Also, identifying projects should be a service of the core project
> system.  Dependent projects need to provide detection data without
> having Emacs load the full project definition.  That way you can have
> lots of projects types without forcing Emacs to load piles of code that
> you don't need.

Each project-find-functions element is free to only contain the bare 
necessary logic to identify its project type, and autoload the rest only 
as required.

> By way of example, EDE's autoloader does this well, and is wrapped in a
> complex test suite to make sure things don't get loaded when detecting
> projects of a different type.  I re-wrote most of it last year and it is
> much better than it was previously.  Use of 'locate-dominating-file' was
> a big win.

 From a quick glance, it defines some restrictions on how a project 
could be detected, and then adds some reusable logic to optimize that. 
I'm not really clear on what those restrictions are. Maybe we'll want 
them, maybe not, we'll have to see.

> Anyway, all that is just about matching projects.  Projects then also
> have meta data.  Things like compilation configuration, compile
> commands, release commands, include paths, classpaths, and whatnot. I
> don't know what you intend with project meta data.  If there was a
> thread about that, I missed it, sorry.

Probably a generic function called project-atrributes that returns a 
hash. Some keys could be standardized.

includes path and class path roughtly correspond to project-search-path.

> Thanks for the pointer.  My Emacs25 didn't compile when I pulled from
> git today, so I didn't get to try it or go very far beyond reading the
> latest in project.el.

That's odd, I didn't have any particular problems compiling it recently.

> For your implementation in ede.el, you are welcome to use what I wrote.

I'm not sure I understand. There's an implementation there already. Is 
your code better? How?

> I think EDE has a generic API for defining a project.  It probably looks
> big on the outside if you just read the doc where it starts with
> "ede-new",

There is a lot to read there. One of the value propositions of 
project.el is that it's easier to grasp.

Another distinction is that project.el is not an API for defining a 
project. It's an API for doing something with an already-defined project.

> but for most cases, if you enable EDE, it will find your
> Emacs src, Linux src, Cmake based build, etc, and it will magically
> provide that to Semantic's parser, to location functions (if enabled)
> and to taggers like global (if enabled.)   The reason it looks big is
> because those are the things that were needed while I was working on
> other tools that needed project support.  I would guess your system will
> grow to a similar state if you keep adding features needed by tools that
> want a project.  It is a really big task.

Maybe. So far, out of that list, the "information for taggers" seems 
like something that could be exposed in the API. The information that 
Semantic needs is probably not generic enough.

The rest are just project types, with different source directories.

> Someone pointed out on the cedet mailing list recently that the doc (and
> probably the website) is too focused on how EDE can create makefiles
> which most people don't want.  Most questions on my mailing list ask how
> to get some other tool like smart completion in semantic working, and
> when they find out that they just need to turn on EDE, and not use the
> full project management system, they are pretty happy.

That's good. In my experience, the CEDET tools are quite dependent on 
each other. Semantic doesn't work for the languages I use, so a lot of 
value proposition goes out of the window.

>> probably because it doesn't match every kind of project). Projectile
>> seems to the most popular "project implementation" out there.
>
> Projectile is a nice interface for moving around between files.

Some other commands it defines:

projectile-run-shell-command-in-root
projectile-find-tag
projectile-multi-occur
projectile-replace
projectile-regenerate-tags
projectile-grep
projectile-test-project
projectile-run-project

I figure most of them could be re-implemented on top of a generic API.

> Projectile has definitions for
> projects EDE does not have because various CEDET developers haven't
> worked with those tools.  The opposite is probably true too.

Probably. Projectile is also 2675 lines long, in total.

> If EDE is considered specific because it is called ede, or because it
> uses eieio, that would be too bad.  It's purpose is to generically
> detect projects such that anyone could create a new project definition,
> find project metadata, and make it accessible to other tools.  This has
> already happened many times.  Supporting EDE is more difficult than just
> providing a dominating file name because it also wants to find extra
> meta data.  I think that is a pretty generic task also.

All I know for sure is that I find EDE's code hard to read (and CEDET as 
a whole, to be honest), and there are not too many third-party packages 
that have decided to build on EDE. Maybe a simpler API will sway a 
larger crowd.

> One way EDE could be made more generic is to make the 'target' concept
> optional.  Simple ede projects also have to define targets as part of
> the detection and matching system, so making them optional would
> simplify creating new projects types in EDE.

It's pretty hard to take something out of an existing API, or a class 
hierarchy. And you can say these methods are optional, but they still 
take up space, and possible consumers are still going to see them.

> Another way EDE could be made more generic is by having built-in tools
> like the 'compile' command ask EDE how to compile, or by having flymake
> use EDE to autodetect how to run make, or perhaps etags could ask EDE
> where to run etags from, and a host of other options.  I had pulled
> those features out into ede named features to keep things independent
> when CEDET was an independent tool.  I don't see why naming conventions
> couldn't change to make things better.

These sounds like good possible applications for project.el.

Re: progmodes/project.el and search paths

[Dmitry Gutov]

On 08/04/2015 01:15 AM, David Engster wrote:

> And what constitutes "the current project", exactly? Only the list of
> directories associated with it?

That definition probably varies between different technologies. And we 
look for the common ground.

> That intersection is tiny.

That's a matter of opinion. I find it more interesting than a lot of the 
other stuff you've mentioned.

Re: progmodes/project.el and search paths

[Dmitry Gutov]

On 08/03/2015 07:16 PM, Stephen Leake wrote:

 > Which is one reason to have an "independent" project core; perhaps the
> projectile team will be more amenable to adopting a core API that isn't
> named "ede"?

I'm sure any reservations developers have about EDE are not related to 
the name.

> There are also restrictions for elisp, since it is preloaded. I think
> the current rule is cl-lib can be preloaded, but not eieio?

eieio is not preloaded, but cl-generic already is. That means you can 
use the generic functions and cl-structs, but not EIEIO classes, in the 
preloaded code.

Re: progmodes/project.el and search paths

[Dmitry Gutov]

On 08/04/2015 12:35 AM, David Engster wrote:

> You mean like ede-compile-project?

I suppose.

And here we find out that EDE took over the project- namespace, a while 
ago. That's not nice.

> I hit a key, the project gets compiled by calling some build command
> based on the currently selected configuration.

Sounds fine to me. The respective method should probably return the 
command to compile, not perform compilation itself.

>> If that information will only be consumed by functions inside EDE
>> itself, there's not much use in making it a part of the general API.
>
> I don't understand this.

If the only functions that would use the information are included in 
CEDET, and there's no plans to extract them, there's no point in making 
them go through the project.el API, since they already know they're 
dealing with EDE.

> Yes, the notion of "debugging the project" varies wildly; same goes for
> "running the executable". You can also add remote debugging on a target
> platform through a debug server. Why not try to support it all?

I'm not particularly against it, just saying it's not easy to do in both 
generic and useful way.

>> So, the API would not only list the variables, but also somehow
>> describe the possible values.
>
> The user can switch between different configurations, and dependend on
> the current configuration a certain set of environment variables is used
> when the build system is called.

Okay, the complications seem to outweigh the benefits here.

> Of course, but since I'm quite happy with EDE, that's not an itch I need
> to scratch. You volunteered to create a project API, and I simply state
> what I would expect from such an API to be usable for me, as an EDE
> user.

The API is not for a user. It's for other packages. If you're happily 
using EDE, and CEDET, and not much else, then it's unlikely to make your 
life better.

> You are comparing what is essentially an interface definition, covering
> a tiny amount of the EDE API, against the whole EDE suite. That factor
> is hardly surprising.

Yes, and that's the point. The interface definition should be clear and 
as short as possible.

> So if those features are only needed by compiled languages they don't
> count? This indeed will keep things simple...

How useful are those things for Haskell? Or ML?

Initially yet, we'll probably avoid those (if only because they're not a 
priority for me personally), but generally useful 
compiled-language-related things can be added later on.

Re: progmodes/project.el and search paths

[Eli Zaretskii]

> From: Stephen Leake <stephen_leake@stephe-leake.org>
> Date: Mon, 03 Aug 2015 16:07:17 -0500
> 
> In addition, when I build things in makefiles, gud does a good job of
> guessing the correct user executable

It does?  For me, it always suggests config.bat as the default when I
want to debug Emacs.

> I don't see how a project can do better; there will still be a list of
> possible executables; the most recently compiled isn't necessarily the
> one I want to debug.

A list of executables I may wish to debug is much shorter than the
list of all executable files.

Re: progmodes/project.el and search paths

[Stefan Monnier]

> And what constitutes "the current project", exactly?

That's for the backend(s) to decide.

> Only the list of directories associated with it?

Not necessarily, tho so far it's the only thing that a backend needs to expose.

>> So the API is designed as a kind of "intersection (rather than union) of
>> all possible project systems".
> That intersection is tiny.

Do you find this to be a problem?  If so, why?
This is not meant to replace/reimplement EDE, but only to provide
a common API between different systems, EDE being hopefully one of them.


        Stefan

Re: progmodes/project.el and search paths

[David Engster]

Stefan Monnier writes:
>>> So the API is designed as a kind of "intersection (rather than union) of
>>> all possible project systems".
>> That intersection is tiny.
>
> Do you find this to be a problem?  If so, why?

It's a very limited view of a project, so other packages who want to
work on a project cannot depend on much information unless they ask one
backend directly.

> This is not meant to replace/reimplement EDE, but only to provide
> a common API between different systems, EDE being hopefully one of them.

OK. Let's see how it evolves.

-David

Re: progmodes/project.el and search paths

[David Engster]

Dmitry Gutov writes:
> On 08/04/2015 12:35 AM, David Engster wrote:
>
>> You mean like ede-compile-project?
>
> I suppose.
>
> And here we find out that EDE took over the project- namespace, a
> while ago. That's not nice.

That can be changed.

> The API is not for a user. It's for other packages. If you're happily
> using EDE, and CEDET, and not much else, then it's unlikely to make
> your life better.

OK.

-David

Re: progmodes/project.el and search paths

[Stephen Leake]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> This is not meant to replace/reimplement EDE, but only to provide
> a common API between different systems, EDE being hopefully one of
> them.

It would help if the header comment in project.el said something along
those lines.

There is a big difference between:

"provide an interface between projects and the rest of Emacs"

and

"provide a core API for implementing projects".

The current project.el header is unclear as to which it is intended to
be.

I was assuming projects.el was the latter, and thus in competition with
EDE, projectile, etc.

If it's really the former, that puts a much different light on things;
it should be providing wrappers/adaptors for EDE, projectile, etc, so
grep, compile etc can use project information.


In particular, there should be no "xref-find-regexp"; instead, "grep"
and similar commands should be enhanced to optionally use project
functions to get the search path.

-- 
-- Stephe

Re: progmodes/project.el and search paths

[Eric Ludlam]

On 08/03/2015 11:09 AM, Eli Zaretskii wrote:
>> >What linker and pre-processor? Seriously, you're talking to a Ruby
>> >developer here.
> I sincerely hope the results of this development will support more
> than just Ruby.  Most languages today, even those that are "normally"
> interpreted, have some kind of compiler.  And even Ruby might grow a
> compiler some day.  I think supporting that is a no-brainer.
> .
>

To flip things around, what would it take to have CEDET support ruby?

I don't know anything about it, but if you just need to identify the 
root of some vc identified directory tree, then it already does.

Eric

Re: progmodes/project.el and search paths

[Eric Ludlam]

On 08/03/2015 06:47 PM, Dmitry Gutov wrote:
>> My
>> experience has been that tools that depend on a project to ask what the
>> current project is do so a lot.  As such, I cache the current project,
>> and current root project as buffer local variables.  Those project
>> structures then have relevant meta data as well.  Another tactic is
>> initializing each buffer into a project as it is created.  That moves
>> search time into a different timing that is less noticeable than
>> on-demand when the user makes a request.
>
> IME, (vc-call-backend backend 'root dir) takes a negligible amount of 
> time, so we might as well call it once per command. Of course, it 
> might be different when using Tramp, but both Tramp and VC do some 
> caching of their own. Other project implementations are free to employ 
> their own caching methods (might be buffer-local, or if the 
> implementation considers that only one project might be enabled at a 
> time, simply in its own variables). 

That is true now for project.el.  Once you support 20+ project 
definitions as in EDE where vc is just one of them, all that negligible 
time adds up.

Asking all 20+ project definitions to do their own caching is something 
EDE used to do, and it was a pain to maintain.  I recommend against it.

Eric

Re: progmodes/project.el and search paths

[Eli Zaretskii]

> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Tue, 4 Aug 2015 02:21:08 +0300
> Cc: Eric Ludlam <eric@siege-engine.com>,
> 	Emacs Development <emacs-devel@gnu.org>
> 
> >> So, the API would not only list the variables, but also somehow
> >> describe the possible values.
> >
> > The user can switch between different configurations, and dependend on
> > the current configuration a certain set of environment variables is used
> > when the build system is called.
> 
> Okay, the complications seem to outweigh the benefits here.

I really think we could benefit from presenting some kind of goal or
vision for the project.el development.  AFAICS, it started as a way to
standardize finding a "project root", then moved to finding "project
directories", but now it seems we are discussing a much wider scope,
something like infrastructure for project definition and possibly also
IDEs?  If it's the latter, then I won't expect to see arguments about
"complications outweighing benefits" in reference to features that are
basic for any IDE that is worth its menus.

So maybe we should step back a notch and decide anew where is
project.el going, and what is and isn't in its scope?

Re: progmodes/project.el and search paths

[Dmitry Gutov]

On 08/04/2015 02:52 PM, Eric Ludlam wrote:

> That is true now for project.el.  Once you support 20+ project
> definitions as in EDE where vc is just one of them, all that negligible
> time adds up.

I honestly expect that project-find-functions, for each individual user, 
will almost never reach even 10 items, let alone 20.

> Asking all 20+ project definitions to do their own caching is something
> EDE used to do, and it was a pain to maintain.  I recommend against it.

You were able to do caching thanks to allowing some restrictions on how 
the current project depends on the current buffer (basically, along the 
lines of locate-dominating-file).

Just a day or two ago, Stephen Leake posted that he was writing a 
project implementation that's active everywhere, as soon as the user 
visits the project.

Caching is probably still possible, even in view of that, but there are 
more things to consider.

Re: progmodes/project.el and search paths

[Dmitry Gutov]

On 08/04/2015 02:48 PM, Eric Ludlam wrote:

> To flip things around, what would it take to have CEDET support ruby?

Semantic support seems paramount (semantic-symref depends on it).

However, it seems to be a losing proposition: writing a grammar for Ruby 
is hard enough (there's no official grammar for the current version of 
Ruby). IIRC, there was some incomplete version of grammar for it in 
CEDET upstream, which doesn't work (at all? well? I don't remember 
exactly), and even if someone were to spend the effort on fixing it:

- It won't be included in Emacs, unless that someone wrote the grammar 
from scratch.
- It won't help with code completion, which is considered the primary 
goal of Semantic (AFAICT), because the latter lacks facilities to handle 
type inference, or types in dynamic languages in general.

> I don't know anything about it, but if you just need to identify the
> root of some vc identified directory tree, then it already does.

Yes, thanks, I don't need to load a 12000-lines project to do that.

Re: progmodes/project.el and search paths

[Dmitry Gutov]

On 08/04/2015 12:40 PM, Stephen Leake wrote:

> It would help if the header comment in project.el said something along
> those lines.
>
> There is a big difference between:
>
> "provide an interface between projects and the rest of Emacs"
>
> and
>
> "provide a core API for implementing projects".
>
> The current project.el header is unclear as to which it is intended to
> be.

Would adding this help?

diff --git a/lisp/progmodes/project.el b/lisp/progmodes/project.el
index d849f93..16578f1 100644
--- a/lisp/progmodes/project.el
+++ b/lisp/progmodes/project.el
@@ -22,6 +22,10 @@
  ;; This file contains generic infrastructure for dealing with
  ;; projects, and a number of public functions: finding the current
  ;; root, related project directories, search path, etc.
+;;
+;; The goal is to make it easy for Lisp programs to operate on the
+;; current project, without having to know which package handles
+;; detection of that project type, parsing its config files, etc.

  ;;; Code:

> If it's really the former, that puts a much different light on things;
> it should be providing wrappers/adaptors for EDE, projectile, etc, so
> grep, compile etc can use project information.

"Should" is a strong word. There's wrapper for EDE living in 
lisp/cedet/ede.el already. Projectile "should" include its own wrapper 
as well.

> In particular, there should be no "xref-find-regexp"; instead, "grep"
> and similar commands should be enhanced to optionally use project
> functions to get the search path.

xref-find-regexp can still be useful: it'll search a user-specified 
directory and present the results using the xref interface.

At the moment, xref-find-regexp offers both types of searches. To search 
in an arbitrary directory, you preface it with C-u.

Re: progmodes/project.el and search paths

[Dmitry Gutov]

On 08/04/2015 04:43 PM, Eli Zaretskii wrote:

> I really think we could benefit from presenting some kind of goal or
> vision for the project.el development.

The vision is: identify general pieces of information that a interesting 
Lisp program can use without worrying too much about which kind of a 
project it is, and add them to the API in a way that's both easy to use 
and (hopefully) implement.

Simply identifying pieces of information doesn't help until you have an 
application in mind. Which, at this point, will often imply "intend to 
implement that application yourself". There's no good way to verify 
usefulness of a piece of API aside from creating at least an 
quick-and-dirty, example consumer.

> AFAICS, it started as a way to
> standardize finding a "project root", then moved to finding "project
> directories", but now it seems we are discussing a much wider scope,
> something like infrastructure for project definition and possibly also
> IDEs?

Much of the recent discussion centered on the question of whether EDE is 
good enough. David also questioned whether the scope (of type-agnostic 
project operations) is a wide one.

> If it's the latter, then I won't expect to see arguments about
> "complications outweighing benefits" in reference to features that are
> basic for any IDE that is worth its menus.

It was a quick judgment: David doesn't seem interested on working on 
extracting some general behavior out of it. I'm not captivated either.

In any case, that was either a weak proposal, or a very incomplete one.

Re: progmodes/project.el and search paths

[Eli Zaretskii]

> Cc: deng@randomsample.de, eric@siege-engine.com, emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Tue, 4 Aug 2015 21:05:31 +0300
> 
> The vision is: identify general pieces of information that a interesting 
> Lisp program can use without worrying too much about which kind of a 
> project it is

So project.el is supposed to allow extracting information from
projects, but is not supposed to help in defining a project?

Re: progmodes/project.el and search paths

[Dmitry Gutov]

On 08/04/2015 09:16 PM, Eli Zaretskii wrote:

> So project.el is supposed to allow extracting information from
> projects, but is not supposed to help in defining a project?

If I understand you right, it is not. Project implementations don't, in 
general, consume the project API. It's not really for them.

Re: progmodes/project.el and search paths

[Eli Zaretskii]

> Cc: deng@randomsample.de, eric@siege-engine.com, emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Tue, 4 Aug 2015 21:41:18 +0300
> 
> On 08/04/2015 09:16 PM, Eli Zaretskii wrote:
> 
> > So project.el is supposed to allow extracting information from
> > projects, but is not supposed to help in defining a project?
> 
> If I understand you right, it is not.

It is not what?  There are 2 parts to that sentence, which one are you
refuting (or confirming)?

If you are saying that both parts of what I said are correct, then why
did you and Stephen spend so many time discussing how to define those
pieces of information for a project?  It shouldn't be part of the job;
instead, you should look at what information is provided as part of
defining a project by different means out there, for example, in a
Makefile.

E.g., there are a couple of standard ways to specify in a Makefile the
directories where the project's files live.  Other significant parts
of a project's information in a Makefile are recipes to build the
project, to install/uninstall it, to clean the tree, etc.  Also,
various standard commands, like compilation command, link command, a
command that creates a library and a manual, etc.

These are all clear candidates for "project information an interested
Lisp program would like to extract", no?

Re: progmodes/project.el and search paths

[João Távora]

On Tue, Aug 4, 2015 at 8:23 PM, Eli Zaretskii <eliz@gnu.org> wrote:
>> On 08/04/2015 09:16 PM, Eli Zaretskii wrote:

> E.g., there are a couple of standard ways to specify in a Makefile the
> directories where the project's files live.  Other significant parts
> of a project's information in a Makefile are recipes to build the
> project, to install/uninstall it, to clean the tree, etc.  Also,
> various standard commands, like compilation command, link command, a
> command that creates a library and a manual, etc.
>
> These are all clear candidates for "project information an interested
> Lisp program would like to extract", no?

If I may go out on possibly a very long limb here, and answering
your specific question my understanding of project.el is that it helps in
writing a "project implementation backend". It is that backend that
understands Makefiles and supplies these bits of information, via
project.el's API to interested Lisp programs, like xref, perhaps
"M-x compile" or some other project-wide command.

I'd welcome that, provided it becomes popular enough that some useful
backends actually do get written.

But I may be misreading the whole discussion, it's indeed become a
bit confusing.

-- 
João Távora

Re: progmodes/project.el and search paths

[Stephen Leake]

Dmitry Gutov <dgutov@yandex.ru> writes:

>
> Would adding this help?
>
> diff --git a/lisp/progmodes/project.el b/lisp/progmodes/project.el
> index d849f93..16578f1 100644
> --- a/lisp/progmodes/project.el
> +++ b/lisp/progmodes/project.el
> @@ -22,6 +22,10 @@
>  ;; This file contains generic infrastructure for dealing with
>  ;; projects, and a number of public functions: finding the current
>  ;; root, related project directories, search path, etc.
> +;;
> +;; The goal is to make it easy for Lisp programs to operate on the
> +;; current project, without having to know which package handles
> +;; detection of that project type, parsing its config files, etc.

Better.

I suggest adding "and Emacs core functions", to emphasize that grep etc
should use the project facilities, and also mention existing backends:

;; The goal is to make it easy for Lisp programs, and Emacs core
;; functions, to operate on the current project, without having to know
;; which package handles detection of that project type, parsing its
;; config files, etc. Existing project packages should provide thin
;; wrappers around their functions to implement this API.

> There's wrapper for EDE living in
> lisp/cedet/ede.el already.

That's a start.

> Projectile "should" include its own wrapper as well.

Yes.

>> In particular, there should be no "xref-find-regexp"; instead, "grep"
>> and similar commands should be enhanced to optionally use project
>> functions to get the search path.
>
> xref-find-regexp can still be useful: it'll search a user-specified
> directory and present the results using the xref interface.
>
> At the moment, xref-find-regexp offers both types of searches. To
> search in an arbitrary directory, you preface it with C-u.

I thought you agreed that the compilation-mode output of grep was better
for this than the xref display.

grep currently searches a user-specified directory; it could offer the
choice of project search path via C-u 1 or some such. And we could add a
grep-project command so the choice shows up in command completion.

grep-project could get filename patterns from the project, as ede almost
provides.

That would require an additional wrapper in ede, for project-file-pattern.

--
-- Stephe

Re: progmodes/project.el and search paths

[Dmitry Gutov]

On 08/04/2015 10:49 PM, Stephen Leake wrote:

> I suggest adding "and Emacs core functions", to emphasize that grep etc
> should use the project facilities, and also mention existing backends:

I don't particularly like that. We're obviously talking about only some 
of the "core functions" (and it's debatable whether Grep is one of 
them). At least "Lisp programs" is a superset of them.

> I thought you agreed that the compilation-mode output of grep was better
> for this than the xref display.

You stated that. I never agreed. I just mentioned that it has an 
advantage (each result is displayed as soon as the process finds it), 
but we should be able to implement it for xref as well, using generators.

Re: progmodes/project.el and search paths

[Dmitry Gutov]

On 08/04/2015 10:23 PM, Eli Zaretskii wrote:

> It is not what?  There are 2 parts to that sentence, which one are you
> refuting (or confirming)?

Confirming both, more or less.

> If you are saying that both parts of what I said are correct, then why
> did you and Stephen spend so many time discussing how to define those
> pieces of information for a project?  It shouldn't be part of the job;
> instead, you should look at what information is provided as part of
> defining a project by different means out there, for example, in a
> Makefile.

Some of that discussion was misunderstandings, some of it was arguing 
about semantics, and some of it was arguing about what's the best way to 
present a certain information in the API. There are many ways to present 
the same information.

We've certainly discussed different project files and information 
contained (or missing) in them. Not Makefiles, though.

> E.g., there are a couple of standard ways to specify in a Makefile the
> directories where the project's files live.  Other significant parts
> of a project's information in a Makefile are recipes to build the
> project, to install/uninstall it, to clean the tree, etc.  Also,
> various standard commands, like compilation command, link command, a
> command that creates a library and a manual, etc.
>
> These are all clear candidates for "project information an interested
> Lisp program would like to extract", no?

I can't easily answer that. What's the use for "command that creates a 
manual" (even setting aside the fact that not all projects have that)? 
Calling it in M-x project-create-manual, a command that only does that 
one thing? Not very interesting, IMHO.

Re: progmodes/project.el and search paths

[Eric Ludlam]

[-- Attachment #1: Type: text/plain, Size: 2925 bytes --]

On 08/02/2015 01:17 PM, Dmitry Gutov wrote:
> If you can propose better ways to do the same things, or something 
> else worth adding, please do. 

AND

On 08/04/2015 09:43 AM, Eli Zaretskii wrote:
> I really think we could benefit from presenting some kind of goal or
> vision for the project.el development.  AFAICS, it started as a way to
> standardize finding a "project root", then moved to finding "project
> directories", but now it seems we are discussing a much wider scope,
> something like infrastructure for project definition and possibly also
> IDEs?  If it's the latter, then I won't expect to see arguments about
> "complications outweighing benefits" in reference to features that are
> basic for any IDE that is worth its menus.
>
> So maybe we should step back a notch and decide anew where is
> project.el going, and what is and isn't in its scope?

Hi Dmitry,

I was tempted to start replying to multiple sub-threads in multiple 
emails, but I suspect there is nuance I am not expressing well, so I've 
opted to go back to one of your original questions to engage in a more 
positive way.

I scraped through some old threads, old code, and my memory to put 
together the attached org file.  The intent is to answer your first 
question about things worth adding to a generic project.  It also 
happens to be something that can answer Eli's question, so I quoted him too.

When the generic project was first proposed, I thought it was a good 
idea where either EDE could be adapted to be a base, or EDE could adapt 
to a new base.  I wasn't too picky, but I clearly had many assumptions 
about what it was that did not match.

EDE's story is one where it started out specific, but became more 
generic because I needed to provide services to external tools I was 
writing, and there was just a lot of code donations and demand for more 
project definitions.

Emacs has a lot of tools that could use a project service, and sometimes 
all the tool needs is a small push of convenience, and not full project 
management system.  A generic project system will make it easy to add 
small conveniences in lots of places you may not expect because tool 
authors will feel safe using it.

The attached is my experience of things I either needed in EDE's base 
classes to implement a project, or that some tool needed, and what I 
learned along the way.  You might think it is too EDE specific, but I 
put in only the most basic stuff, and put in things I kind of wish I had 
time to add to EDE.

I am sure there are entries you consider out of scope.  In my mind 
projects need a centralizing force to bring consistency in the same way 
comint brought some very basic consistency to shell buffers.  We should 
not skimp in approaching this goal.

I consider this file incomplete, but I hope it proves useful as a 
starting point.  I'd be happy to periodically update it for a little 
while if it is useful.

Eric

[-- Attachment #2: generic_project.txt --]
[-- Type: text/plain, Size: 12119 bytes --]

Eric's notes for Generic Project

* Definitions
** Project Directory
 A top level directory in which source code for a project resides.
** Project
 A collection of Meta Data and Services that describe how code in a Project
 Directory is organized, versioned, compiled, etc.
** Project Meta Data
 A piece of information that describes some aspect of the project.
 Also: `slot' on a 'Project' in CLOS parlance
** Project Service
 Functionality provided by a generic project interface
 Also: 'method' on a 'Project' in CLOS parlance
** Project Client
 A feature that depends on either the Project Directory, Project, or a
 Project Service, usually developed independently of project implementations.
** Project Implementation
 An implementation of a concrete project that provides actual
 functionality, some of which is provided via the generic project
 service.
** Thing
 Something the generic project should handle that doesn't fit into
 one of the above categories... yet.

* Service: Current Project Directory
Provides external tools a way to identify the top-most directory of the current project.

** Features:
*** Performance; For a buffer:
As fast as a buffer local variable ref
*** Performance; For a file or directory name: 
First time lookup should take advantage of any known open projects.
*** Nested Projects
There should be ways for there to be projects in projects.
**** Example: Having a set of small projects for testing in a larger project

** Project Client Examples
*** Identify a TAGS file to use for navigating a source tree
*** Starting point for search tools
*** Location of Manifests, AUTHOR, and other dist files

* Service: Current Project
Tools that need to know about some meta-data of the current project
will need a way to refer to the current project.  The current project
needs to be some 'thing' that provides project features.

** Features
*** Project Meta Data
Projects can have meta data.  There should be some standard meta-data,
and meta data optionally provided by specific project implementations.
*** Project Services
Projects can have behavior and provide services.  There should be
some basic standard services, and optionally provided services
specific to project implementations.
*** Project Commands
Services for users to execute when they are managing their project.
There should be some basic standard commands, and optionally provided
commands specific to project implementations.

Project Commands may seem optional from a generic project point of view.
They are primarily helpful as a homogenizing force between disparate
project implementations which would be a good thing in the long run,
the way comint made all shell buffers very similar and easy to move
between.
*** Menu/Toolbar/Other UI Services
Project implementations should have a standard way to integrate into
the Development menu where users expect project commands to exist.
*** Extensible
The thing used to represent a generic project should be extensible so
project implementations can implement abstract services, or add new
ones.
*** Sensible Defaults
The generic project thing should have sensible defaults for meta data
and specific services so a project implementation doesn't need to
implement them all.

** Example Features
*** Version number of project
Useful when distinguishing between different local repositories you
might be working on.
*** Compile
Integration with the `compile' command, a standard action for many
compiled programs.  A more generic term that would work for running
tests in an interpreted environment may be preferred.

* Service: Project Detection
All project management systems need a way to 'detect' a project the
first time a user visits a file.  Many of those systems are the same,
and involve walking up and down directory trees looking for
dominating files.

** Features:
*** Dominating File Detection
A core way to identify projects that use a dominating file, such as a
top level build.xml, or a AndroidManifest.xml.
*** Distributed dominating file detection
A core way to identify projects that use the same dominating file in
all directories in a project, such as Makefile, CVS.
*** Nested Projects
There should be ways for there to be projects in projects.
*** Declarative detection features
When an project implementation integrates with the generic project
system, it should be able to 'declare' how to detect a project of its
preferred type.  That way the source code for the project
implementation will only be loaded into Emacs when a project of that
type is detect.
*** Many project types detectable w/out user configuration
Users should have all available project types automatically detected
without having to fiddle with their .emacs file.  ELPA project
implementations should have a way to add themselves to the detection
list without loading their code.

** Project Detection Declaration features 
Features that a declaration of how to detect should provide
*** How to detect
A declarative way to express the desired detection mechanism.
*** How to load the project code to support the project
A function symbol and/or a symbol to require that will enable a
project implementation to be created to support the detected project.
*** Project Loading safety
Some project types might need to load elisp-code-like structures to
bring in meta data.  If this is potentially unsafe, the project loader
should ask the user, or consult a configuration, before allowing it.
*** Detection ordering
Some project types are a subset of other project types.  Priority of
project detection/loading should be a part of the detection declaration.
**** Example:
Makefile based project could exist with a .git repository.  The
Makefile project is more specific than the .git repository, and
should be selected first.

** Tool Example
*** A project manager (JDEE, EDE, etc)
These tools need to detect projects that belong to them, but EDE might
identify an ANT project, and JDEE might identify its .prj file, and
they would attempt to co-exist without a mediating generic project
service.  When a tool asks for the current project, there should be
only one.

* Service: Project Commands
Emacs already has a wide range of useful IDE like tools, such as
`compile' and `gud'.  These are generic, and require the user to type
in any specific options needed for their project.

Generic projects should provide integration with those tools,
and likely multiple variants regarding the project in question.  The
project service should allow project implementations to augment these
core features with variants.

** Features
*** Compile integration
Projects implementations should be able to provide default compile
arguments based on project meta data.  Multiple variations should be
available such as `whole project' or `just this thing in my buffer'.
*** GUD integration
Provide default debugger commands, and arguments to GUD supported
debugger commands.
*** Update meta-data
Emacs is not the only project system out there, so users should have
a way to force a project implementation to refresh itself from
on-disk changes.
*** Project browsing
Start users preferred file browsing project client on the project.
**** Examples: ECB, Speedbar, Dired, external executable

* Service: Keybinding / Menu / Toolbar management
Project implementations should have a standard way to integrate into
the Development menu where users expect project commands to exist.
User commands provided should be consistent between types of projects.

The purpose of this is to provide a homogenizing force so that
different project implementations will gain a similar look and feel
so users can move between different projects, and enjoy consistent
keybindings and UI elements.

** Features
*** Organized project options
Tools should be logically organized in whichever UI elements are
used to express it.
*** Default project commands
There should be some default project commands, like 'compile' built in.
*** Room for custom options
There should be an easy way for a project implementation to add new
items into the logical groupings provided.

** Examples
*** Arduino: upload to dev board
Arduino project implementation needs to add an entry to the menu
system to upload a project onto the dev board, separate from
compiling to a binary.
*** JDEE vs EDE
Compile commands are different between JDEE and EDE.  If a standard
way of invoking the disparate commands were available, users could
work in C++ and Java and use the same keybindings or menus in each.

* Service: Project Local Variables
Tools like dir-locals provide some level of this feature.  Generic
projects could help enable project local variables.

* Meta Data: Version
Projects should provide a way to extract version numbers, and make
them readily available so users can distinguish between multiple
versions of the same code base.

* Meta Data: Arbitrary
Project implementations should have a way to provide their own meta
data to external commands, and have a way to negotiate for commonality
between project implementations that support the same concept.

* Meta Data: Client Data
Project clients that develop a user service based on generic project
data should have a way to provide their own data to a project where
the original project implementation author may not know what is
needed.

** Examples
*** Flymake
Flymake should have a way to ask for meta data about compilation, and
if the project implementation doesn't support it, there should be no
errors, and Flymake shouldn't need to know anything specific about
all implementations that might support it.
*** User Option for client
The project client 'projectile' can toggle between code and test, but
the project implementation author didn't account for that.
Projectile will need to associate user supplied meta-data about test
locations with the project.

Puzzle - persistence?

* Service: Finding Files
Finding files comes in a couple different flavors.
1) User wants to visit a file via a 'short name', and the project can
   find it.
2) Code often refers to other files using a short name, and refers to
   an 'include path' concept in the project to find it.

** Features:
*** Enable a project client to provide user tools to 'jump' to a file.
*** Enable language specific 'include path' concept
Useful when multiple files with the same name exist, and are
distinguished by location.
** Examples
*** C++ include path
The semantic smart completion engine depends on include paths to find
headers which provide definitions to complete on.
*** Java Classpath
Semantic depends on a java classpath to find jar files from which it
extracts symbol information used for smart completion.
*** cscope or GNU Global integration as project service
These tools track not just references, but files in a project, and
have easy ways to convert a short file name into a full path name.
*** project client to jump from 'include' statement in C++ to the file
*** project client to jump between a src file (xml, etc) to generated output.

* Command: Load a project;   Service: Known unopened projects
The generic project system should have a handy way to provide a
completion list of known projects a user might choose to load.

Project lists might be provided by recording all projects a user may
have previously opened, or via scan of areas relevant to the user.

** Examples:
*** desktop
A project client might associated a working set of buffers to load upon
opening a project.

* Thing: Multiple activities on one project
Projects usually have multiple activities in them.  At the most basic
there are 'get it ready to use' support files, such as compilation
system, INSTALL doc, etc, and the source code.  Full project
documentation, and multi-feature projects have multiple build
targets, and complex build systems.

The generic project should consider supporting concepts for multiple
activities in one project.

** Examples
*** `compile' knows which thing to compile in a project
*** Language specificity
Operations in a particular language restricted to other files of that
language.
*** Code & Doc
Enable a project client to support navigating between code and doc.

Re: progmodes/project.el and search paths

[Eli Zaretskii]

> Cc: deng@randomsample.de, eric@siege-engine.com, emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Tue, 4 Aug 2015 23:15:16 +0300
> 
> > E.g., there are a couple of standard ways to specify in a Makefile the
> > directories where the project's files live.  Other significant parts
> > of a project's information in a Makefile are recipes to build the
> > project, to install/uninstall it, to clean the tree, etc.  Also,
> > various standard commands, like compilation command, link command, a
> > command that creates a library and a manual, etc.
> >
> > These are all clear candidates for "project information an interested
> > Lisp program would like to extract", no?
> 
> I can't easily answer that. What's the use for "command that creates a 
> manual" (even setting aside the fact that not all projects have that)? 
> Calling it in M-x project-create-manual, a command that only does that 
> one thing?

Yes, for starters.  Building the whole project from scratch, including
the manuals, is another.  Finding out whether the project _has_ a
manual building instructions is yet another.

> Not very interesting, IMHO.

I don't understand this criterion.  I think the criterion should be
"is this useful".

Re: progmodes/project.el and search paths

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Tue, 4 Aug 2015 20:40:08 +0100
> Cc: Dmitry Gutov <dgutov@yandex.ru>, emacs-devel <emacs-devel@gnu.org>, deng@randomsample.de, 
> 	Eric Ludlam <eric@siege-engine.com>
> 
> If I may go out on possibly a very long limb here, and answering
> your specific question my understanding of project.el is that it helps in
> writing a "project implementation backend".

No, I don't think it does that.  I think it allows other Lisp programs
to extract information from a project system that is already built.

Re: progmodes/project.el and search paths

[Stephen Leake]

Dmitry Gutov <dgutov@yandex.ru> writes:

> On 08/04/2015 10:49 PM, Stephen Leake wrote:
>
>> I suggest adding "and Emacs core functions", to emphasize that grep etc
>> should use the project facilities, and also mention existing backends:
>
> I don't particularly like that. We're obviously talking about only
> some of the "core functions" (and it's debatable whether Grep is one
> of them). At least "Lisp programs" is a superset of them.
>
>> I thought you agreed that the compilation-mode output of grep was better
>> for this than the xref display.
>
> You stated that. I never agreed. I just mentioned that it has an
> advantage (each result is displayed as soon as the process finds it),
> but we should be able to implement it for xref as well, using
> generators.

Ok, so xref-find-regexp uses the project search path by default,
and the xref display; it uses a user-specified root dir with C-u.

Do you aggree it would be good if rgrep supported an option to use the
project search path? Possibly only via the command grep-project (or
project-grep)? 

-- 
-- Stephe

Re: progmodes/project.el and search paths

[Stephen Leake]

Eli Zaretskii <eliz@gnu.org> writes:

>> Cc: deng@randomsample.de, eric@siege-engine.com, emacs-devel@gnu.org
>> From: Dmitry Gutov <dgutov@yandex.ru>
>> Date: Tue, 4 Aug 2015 23:15:16 +0300
>> 
>> > E.g., there are a couple of standard ways to specify in a Makefile the
>> > directories where the project's files live.  Other significant parts
>> > of a project's information in a Makefile are recipes to build the
>> > project, to install/uninstall it, to clean the tree, etc.  Also,
>> > various standard commands, like compilation command, link command, a
>> > command that creates a library and a manual, etc.
>> >
>> > These are all clear candidates for "project information an interested
>> > Lisp program would like to extract", no?
>> 
>> I can't easily answer that. What's the use for "command that creates a 
>> manual" (even setting aside the fact that not all projects have that)? 
>> Calling it in M-x project-create-manual, a command that only does that 
>> one thing?
>
> Yes, for starters.  Building the whole project from scratch, including
> the manuals, is another.  Finding out whether the project _has_ a
> manual building instructions is yet another.
>
>> Not very interesting, IMHO.
>
> I don't understand this criterion.  I think the criterion should be
> "is this useful".

Apparently EDE has the notion of "targets" for this sort of thing. So
the EDE users find it useful, for actually running the targets.

The question for project.el is whether is would be useful for _other_
(non-EDE) elisp programs; we already have EDE to create and run the
targets.

I can imagine a full-blown Configuration Management application (think
IBM ClearCase) that manages the created binaries as well as the sources;
it would want a list of the targets of a project, to ensure they were
tracked in the CM database. But such a thing would probably not be
written on top of project.el; it would be more like a new EDE project
type, or a complete replacement for EDE.

Short of that, I can't think of a non-EDE elisp program that would want
to know about targets. There might be one someday; we can extend
project.el then.

There are probably other features in EDE that some non-EDE programs
would want to know about; I'm still learning about EDE.

-- 
-- Stephe

Re: progmodes/project.el and search paths

[Dmitry Gutov]

On 08/05/2015 05:49 AM, Eli Zaretskii wrote:

> Yes, for starters.  Building the whole project from scratch, including
> the manuals, is another.  Finding out whether the project _has_ a
> manual building instructions is yet another.

Project could define a way to build it whole, for instance, without 
giving out specific information about every goal of build process.

Or exposing it all, in a standardized structure, without, however, 
having the list of specific targets a project can support (manuals, etc) 
pre-defined.

>> Not very interesting, IMHO.
>
> I don't understand this criterion.  I think the criterion should be
> "is this useful".

Just a criterion for what I spend time on.

Do you really consider this on the same of usefulness as, say, 
xref-find-references?

Re: progmodes/project.el and search paths

[Dmitry Gutov]

On 08/05/2015 09:02 AM, Stephen Leake wrote:

> Ok, so xref-find-regexp uses the project search path by default,
> and the xref display; it uses a user-specified root dir with C-u.

That's a bit awkward, though. While it's not very surprising, there's 
still nothing in this command's name to indicate that it uses the 
current project by default.

> Do you aggree it would be good if rgrep supported an option to use the
> project search path? Possibly only via the command grep-project (or
> project-grep)?

I think we should have project-find-regexp (or project-rgrep) which uses 
the xref interface. grep-project will be functionally redundant.

Re: progmodes/project.el and search paths

[Stefan Monnier]

> It's a very limited view of a project, so other packages who want to
> work on a project cannot depend on much information unless they ask one
> backend directly.

Yes, it's very limited, simply because there hasn't be any other such
"other package" which required something else.  As mentioned I can see
things like "flymake" imposing extra requirements (mostly a new method
to compile some file(s) and return the corresponding compilation
messages).


        Stefan

Re: progmodes/project.el and search paths

[Eli Zaretskii]

> From: Stephen Leake <stephen_leake@stephe-leake.org>
> Date: Wed, 05 Aug 2015 01:18:54 -0500
> 
> >> What's the use for "command that creates a manual" (even setting
> >> aside the fact that not all projects have that)?  Calling it in
> >> M-x project-create-manual, a command that only does that one
> >> thing?
> >
> > Yes, for starters.  Building the whole project from scratch, including
> > the manuals, is another.  Finding out whether the project _has_ a
> > manual building instructions is yet another.
> >
> >> Not very interesting, IMHO.
> >
> > I don't understand this criterion.  I think the criterion should be
> > "is this useful".
> 
> Apparently EDE has the notion of "targets" for this sort of thing. So
> the EDE users find it useful, for actually running the targets.
> 
> The question for project.el is whether is would be useful for _other_
> (non-EDE) elisp programs; we already have EDE to create and run the
> targets.

If EDE finds this useful, what doubt is there that other Lisp programs
will?  How many examples do we need before we decide that a specific
trait of a project is "useful"?  I say one is enough (sometimes more
than enough).

> Short of that, I can't think of a non-EDE elisp program that would want
> to know about targets. There might be one someday; we can extend
> project.el then.

That's exactly my problem: I find such an approach surprising, to say
the least, when building such basic infrastructure.  When you build
infrastructure, you don't wait for clients to come requesting basic
features.  You are supposed to study the use cases and the entities
you want to support, and provide support for at least the
standard/basic aspects and traits of those entities.  I think there
should be small doubt that access to project documentation, including
its build command, is one of these basics, because every project has
that.

Re: progmodes/project.el and search paths

[Eli Zaretskii]

> Cc: deng@randomsample.de, eric@siege-engine.com, emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Wed, 5 Aug 2015 12:42:07 +0300
> 
>     Yes, for starters.  Building the whole project from scratch, including
>     the manuals, is another.  Finding out whether the project _has_ a
>     manual building instructions is yet another.
> 
> Project could define a way to build it whole, for instance, without giving out specific information about every goal of build process.

It could, but why would it want to?  It sounds silly to have to
rebuild everything when you only need to rebuild some (small) part.

> Or exposing it all, in a standardized structure, without, however, having the list of specific targets a project can support (manuals, etc) pre-defined.

Again, sounds silly, given how most, if not all, modern build tools
work.

But (shrug) not a catastrophe: the corresponding attribute will simply
be nil, or the command to build everything.

IOW, I see no problem here, and no reasons to consider this unfit for
the API.

>         Not very interesting, IMHO.
> 
>     I don't understand this criterion.  I think the criterion should be
>     "is this useful".
> 
> Just a criterion for what I spend time on.

I don't think you have this luxury when you work on infrastructure.

E.g., company.el is "not very interesting" for me, but I still try
very hard to fix every issue in the display engine that you or your
users report.

> Do you really consider this on the same of usefulness as, say, xref-find-references?

Yes.  They are both no-brainers to have in the infrastructure that
AFAIU you are trying to provide.  It might even be much more useful to
a Lisp program that only cares about the manual of a project.

Re: progmodes/project.el and search paths

[Dmitry Gutov]

On 08/05/2015 06:23 PM, Eli Zaretskii wrote:

> Again, sounds silly, given how most, if not all, modern build tools
> work.

That is counter to my experience. The modern build tools work in a 
variety of different ways.

> But (shrug) not a catastrophe: the corresponding attribute will simply
> be nil, or the command to build everything.

And I don't understand what you mean here.

> IOW, I see no problem here, and no reasons to consider this unfit for
> the API.

It's fit. As soon as we're sure someone is going to write a Lisp program 
making use of this part of the API. And by "sure", I mean we have at 
least a proof-of-concept patch for a non-trivial piece of functionality.

> I don't think you have this luxury when you work on infrastructure.

I don't think anyone here has the authority to tell me what to spend 
time on.

> E.g., company.el is "not very interesting" for me, but I still try
> very hard to fix every issue in the display engine that you or your
> users report.

You don't work on adding new features to it, however.

> Yes.  They are both no-brainers to have in the infrastructure that
> AFAIU you are trying to provide.  It might even be much more useful to
> a Lisp program that only cares about the manual of a project.

It makes sense to go for features with bigger impact first.

Re: progmodes/project.el and search paths

[Dmitry Gutov]

On 08/05/2015 06:08 PM, Eli Zaretskii wrote:

> If EDE finds this useful, what doubt is there that other Lisp programs
> will?  How many examples do we need before we decide that a specific
> trait of a project is "useful"?  I say one is enough (sometimes more
> than enough).

A trait should be useful *and* fairly common.

Do you see someone in a hurry to rewrite EDE on top of project.el? I 
don't. So we have zero potential consumers so far.

>> Short of that, I can't think of a non-EDE elisp program that would want
>> to know about targets. There might be one someday; we can extend
>> project.el then.
>
> That's exactly my problem: I find such an approach surprising, to say
> the least, when building such basic infrastructure.

On the other hand, it wastes less time on speculative features, and the 
API is almost guaranteed to turn out better in the end.

> I think there
> should be small doubt that access to project documentation, including
> its build command, is one of these basics, because every project has
> that.

I don't even know what "access to project documentation" means. Is it 
just the build command? Or the paths of resulting files? Or their 
formats? Or an index? Or a way to display them?

Re: progmodes/project.el and search paths

[Eli Zaretskii]

> Cc: deng@randomsample.de, eric@siege-engine.com, emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Wed, 5 Aug 2015 18:31:35 +0300
> 
> On 08/05/2015 06:23 PM, Eli Zaretskii wrote:
> 
> > Again, sounds silly, given how most, if not all, modern build tools
> > work.
> 
> That is counter to my experience. The modern build tools work in a 
> variety of different ways.

You mean, some of them don't support specifying how to build
documentation?  Which ones?

> > But (shrug) not a catastrophe: the corresponding attribute will simply
> > be nil, or the command to build everything.
> 
> And I don't understand what you mean here.

An attribute can exist even if with some projects its value will be
nil or a trivial "build-all" command.  IOW, even if some projects
won't have any non-trivial value for this attribute, it's not an
argument against the attribute's existence.

> > IOW, I see no problem here, and no reasons to consider this unfit for
> > the API.
> 
> It's fit. As soon as we're sure someone is going to write a Lisp program 
> making use of this part of the API. And by "sure", I mean we have at 
> least a proof-of-concept patch for a non-trivial piece of functionality.

I don't think it's correct to wait for specific requests for every
single sub-feature, when developing infrastructure like this one.
Some basic traits should be supported just because they are basic,
before someone asks.

> > I don't think you have this luxury when you work on infrastructure.
> 
> I don't think anyone here has the authority to tell me what to spend 
> time on.

I didn't tell you anything except my views on this.

> > E.g., company.el is "not very interesting" for me, but I still try
> > very hard to fix every issue in the display engine that you or your
> > users report.
> 
> You don't work on adding new features to it, however.

Yes, I do.  But that's besides the point, isn't it?

> > Yes.  They are both no-brainers to have in the infrastructure that
> > AFAIU you are trying to provide.  It might even be much more useful to
> > a Lisp program that only cares about the manual of a project.
> 
> It makes sense to go for features with bigger impact first.

If this is just a matter of priority, I'm okay with whatever the
decision is.  I was under the impression that the mere need is being
questioned.

Re: progmodes/project.el and search paths

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Wed, 5 Aug 2015 18:36:08 +0300
> 
> On 08/05/2015 06:08 PM, Eli Zaretskii wrote:
> 
> > If EDE finds this useful, what doubt is there that other Lisp programs
> > will?  How many examples do we need before we decide that a specific
> > trait of a project is "useful"?  I say one is enough (sometimes more
> > than enough).
> 
> A trait should be useful *and* fairly common.

It's common, just look at any IDE out there.

> Do you see someone in a hurry to rewrite EDE on top of project.el? I 
> don't. So we have zero potential consumers so far.

Like I said, I consider this kind of arguments irrelevant for
developing infrastructure.  Either it's important to develop
infrastructure for accessing project traits, or it isn't.  If it is
(and I gather you decided it is), then the infrastructure should IMO
be reasonably complete.

> >> Short of that, I can't think of a non-EDE elisp program that would want
> >> to know about targets. There might be one someday; we can extend
> >> project.el then.
> >
> > That's exactly my problem: I find such an approach surprising, to say
> > the least, when building such basic infrastructure.
> 
> On the other hand, it wastes less time on speculative features, and the 
> API is almost guaranteed to turn out better in the end.

Not IME.  What I saw more than once is unfinished infrastructure that
no one wanted to continue developing, because the person(s) who
started moved on without arriving at some reasonably complete
component.

> > I think there
> > should be small doubt that access to project documentation, including
> > its build command, is one of these basics, because every project has
> > that.
> 
> I don't even know what "access to project documentation" means. Is it 
> just the build command? Or the paths of resulting files? Or their 
> formats? Or an index? Or a way to display them?

There are a lot of IDEs out there to learn from and get answers to all
these questions very quickly.

IME, doing that kind of research is always part of any infrastructure
development.  It sounds like you would prefer waiting till someone
comes up and teaches you/us all these answers, or presents clear
requirements that allow you to formulate those answers, but IME that
seldom happens, because people who already went through that research,
or gained the answers from personal experience, are a few and far
in-between, and they are unlikely to line up to help us, of all the
projects.  I went through such an experience myself years ago, when I
started to work on bidirectional display.

Anyway, I think my views on this are clear, so I think it's time for
me to stop arguing about this.

Re: progmodes/project.el and search paths

[David Engster]

Dmitry Gutov writes:
> On 08/05/2015 06:08 PM, Eli Zaretskii wrote:
>
>> If EDE finds this useful, what doubt is there that other Lisp programs
>> will?  How many examples do we need before we decide that a specific
>> trait of a project is "useful"?  I say one is enough (sometimes more
>> than enough).
>
> A trait should be useful *and* fairly common.
>
> Do you see someone in a hurry to rewrite EDE on top of project.el?

What does that mean? Isn't project.el always at the top, fed by various
project backends?

-David

Re: progmodes/project.el and search paths

[Dmitry Gutov]

On 08/05/2015 07:45 PM, David Engster wrote:

> What does that mean? Isn't project.el always at the top, fed by various
> project backends?

Sorry for being unclear.

Probably not EDE's core, but maybe the integrations that build on it 
(like ede/speedbar, for instance), or parts of CEDET that use EDE.

Re: progmodes/project.el and search paths

[Dmitry Gutov]

On 08/05/2015 07:16 PM, Eli Zaretskii wrote:

>> That is counter to my experience. The modern build tools work in a
>> variety of different ways.
>
> You mean, some of them don't support specifying how to build
> documentation?  Which ones?

I didn't mean building docs in particular, but yes, kinda. You can build 
docs with Rake (and in different formats, depending on which ones you 
want), but there's no standard task name for that.

> I don't think it's correct to wait for specific requests for every
> single sub-feature, when developing infrastructure like this one.
> Some basic traits should be supported just because they are basic,
> before someone asks.

I guess so. When we get around to the "building" aspect of projects, 
we'll try to make it reasonably complete.

Without any build-related commands in Emacs, though, we'll still fall 
short of providing the complete editing experience you seem to want. And 
those would be exactly the kind of consumers I'm looking in order to 
grow the project API.

>>> E.g., company.el is "not very interesting" for me, but I still try
>>> very hard to fix every issue in the display engine that you or your
>>> users report.
>>
>> You don't work on adding new features to it, however.
>
> Yes, I do.  But that's besides the point, isn't it?

Adding new features to Company, I mean.

Re: progmodes/project.el and search paths

[Stephen Leake]

Dmitry Gutov <dgutov@yandex.ru> writes:

> On 08/05/2015 09:02 AM, Stephen Leake wrote:
>
>> Do you aggree it would be good if rgrep supported an option to use the
>> project search path? Possibly only via the command grep-project (or
>> project-grep)?
>
> I think we should have project-find-regexp (or project-rgrep) which
> uses the xref interface. grep-project will be functionally redundant.

Since they are the same function, why is the name project-find-regexp
better than the name grep-project?

The latter will be _much_ easier to discover for the current grep user;
M-x gre<tab> will show it.

The purpose of project.el is to provide access to project information to
other elisp programs; it is _not_ to provide user level programs.

'grep' is a user level elisp program that could use some project
information. If it is possible to provide user control over that via
prefixes or prompts, then the name doesn't have to change. If that's
too complicated, grep-project seems like the best choice; similar to
grep-find.

'project-find-regexp' is a user level elisp program; project.el is not
supposed to provide those.

-- 
-- Stephe

Re: progmodes/project.el and search paths

[Stephen Leake]

Dmitry Gutov <dgutov@yandex.ru> writes:

> On 08/05/2015 05:49 AM, Eli Zaretskii wrote:
>
>>> Not very interesting, IMHO.
>>
>> I don't understand this criterion.  I think the criterion should be
>> "is this useful".
>
> Just a criterion for what I spend time on.

Since we are discussing core Emacs functionality, you should be clearer
about this. There is a big difference between:

- that would be a useful core feature, but I don't want to work on it

and

- that would not be a useful core feature.


> Do you really consider this on the same of usefulness as, say,
> xref-find-references?

Personally, I use 'make' much more than I use cross reference tools, so
yes, build is more useful.

For Emacs core, they are both important, which is why we have 'compile'
and 'xref-find-*'.

For project.el, this fits into "provide build related project
information to other elisp programs" in a couple of ways:

- 'compile' tries to provide tab completion on Makefile targets; it
doesn't do it well (it misses targets in #include <makefile>).
Presumably EDE can do better, so project.el should provide
project-build-targets, so compile can use that to provide the completion
list. That would then automatically work for ant, gradle, etc projects
as those are added to EDE or another project implementation.

- 'compile' also tries to guess the build command; it could use
project-build-command.

-- 
-- Stephe

Re: progmodes/project.el and search paths

[Stephen Leake]

Dmitry Gutov <dgutov@yandex.ru> writes:

> Do you see someone in a hurry to rewrite EDE on top of project.el? I
> don't. So we have zero potential consumers so far.

I think Eli's point is valid here.

EDE has an internal "project info" data structure and API. It's possibly
not very welll separated from the rest of the EDE code.

A future replacement for EDE (or a rewrite/refactoring of EDE) could use
the standard Emacs project API, and thus gain instant access to all new
project implementations.

So we should at least plan on supporting that level of functionality.

-- 
-- Stephe

Re: progmodes/project.el and search paths

[Stephen Leake]

David Engster <deng@randomsample.de> writes:

> Dmitry Gutov writes:
>> On 08/05/2015 06:08 PM, Eli Zaretskii wrote:
>>
>>> If EDE finds this useful, what doubt is there that other Lisp programs
>>> will?  How many examples do we need before we decide that a specific
>>> trait of a project is "useful"?  I say one is enough (sometimes more
>>> than enough).
>>
>> A trait should be useful *and* fairly common.
>>
>> Do you see someone in a hurry to rewrite EDE on top of project.el?
>
> What does that mean? Isn't project.el always at the top, fed by various
> project backends?

As I understand it, EDE provides both project backends and build
functions that use information from the project backends.

A future refactoring of EDE could use the Emacs project.el API to
define the interface between the two.


-- 
-- Stephe

Re: progmodes/project.el and search paths

[Dmitry Gutov]

On 08/06/2015 10:43 AM, Stephen Leake wrote:

> Since we are discussing core Emacs functionality, you should be clearer
> about this. There is a big difference between:

I doesn't seem particularly useful, and I don't want to work on it. But 
there's no reason to assume that I'm going to veto it (do I even have a 
veto power?), if someone's going to implement it in a clean way.

> Personally, I use 'make' much more than I use cross reference tools, so
> yes, build is more useful.

You can still run 'make' in the console, but cross-referencing needs an 
editor to function adequately.

> - 'compile' tries to provide tab completion on Makefile targets; it
> doesn't do it well (it misses targets in #include <makefile>).
> Presumably EDE can do better, so project.el should provide
> project-build-targets, so compile can use that to provide the completion
> list. That would then automatically work for ant, gradle, etc projects
> as those are added to EDE or another project implementation.

If project-build-targets just returns a list of strings, that would be 
insufficient to run the build. So we need a more informative structure.

name, description, full command to run. Maybe also cram running build 
targets in different environments in there somehow.

> - 'compile' also tries to guess the build command; it could use
> project-build-command.

Maybe all required info will be in project-build-targets. Or we'll just 
have project-default-target, which will return a simple string.

Re: {Spam?} Re: progmodes/project.el and search paths

[Eric Ludlam]

On 08/05/2015 09:42 AM, Stefan Monnier wrote:
>> It's a very limited view of a project, so other packages who want to
>> work on a project cannot depend on much information unless they ask one
>> backend directly.
> Yes, it's very limited, simply because there hasn't be any other such
> "other package" which required something else.  As mentioned I can see
> things like "flymake" imposing extra requirements (mostly a new method
> to compile some file(s) and return the corresponding compilation
> messages).
>

I would think that backend projects can also depend on services from a 
generic project system.  Comint provides a generic way to interact with 
a shell command.  Using comint is easier than starting from scratch and 
makes no assumptions by providing for a wide range of features that 
might be needed by sub shells.  A side effect is that every subordinate 
shell buffer in Emacs is consistent, configuring options in comint works 
everywhere, and keybindings are the same. Another side effect is that a 
you can write services that work with comint that thus work in many 
different shells.  Wouldn't it be cool if users could write things that 
feel like project management backend features that work with multiple 
back-ends.

It would be nice if the generic base project were the same.  Any project 
backend you use would have similar keybindings, menus, detection 
mechanisms, file location systems, etc because it will be easier to use 
the base support than invent something new.

Eric

Re: progmodes/project.el and search paths

[Stephen Leake]

Dmitry Gutov <dgutov@yandex.ru> writes:

> On 08/06/2015 10:43 AM, Stephen Leake wrote:
>
>> Since we are discussing core Emacs functionality, you should be clearer
>> about this. There is a big difference between:
>
> I doesn't seem particularly useful, and I don't want to work on it.
> But there's no reason to assume that I'm going to veto it (do I even
> have a veto power?), if someone's going to implement it in a clean
> way.
>
>> Personally, I use 'make' much more than I use cross reference tools, so
>> yes, build is more useful.
>
> You can still run 'make' in the console, 

`next-error' is a very powerful Emacs feature; nothing equivalent is
available in a console.

-- 
-- Stephe

Re: {Spam?} Re: progmodes/project.el and search paths

[Stefan Monnier]

> It would be nice if the generic base project were the same.  Any project
> backend you use would have similar keybindings, menus, detection mechanisms,
> file location systems, etc because it will be easier to use the base support
> than invent something new.

Could be nice, but I think it's a different beast from project.el.
I actually have the impression that what you describe already exists and
is called EDE ;-)


        Stefan

Re: progmodes/project.el and search paths

[Dmitry Gutov]

On 08/06/2015 05:27 PM, Stephen Leake wrote:

>> You can still run 'make' in the console,
>
> `next-error' is a very powerful Emacs feature; nothing equivalent is
> available in a console.

Point taken.

Still, I wouldn't want to settle on a version of project-build-tasks 
over-specialized for, say, for C/C++ Makefile-based projects. The best 
way to do that is by implementing a flexible "run task" command.

M-x compile could use the API for its default value, but I don't think 
it should be the focus when adding the new feature.

Re: {Spam?} Re: progmodes/project.el and search paths

[Eric Ludlam]

On 08/06/2015 07:10 PM, Stefan Monnier wrote:
>> It would be nice if the generic base project were the same.  Any project
>> backend you use would have similar keybindings, menus, detection mechanisms,
>> file location systems, etc because it will be easier to use the base support
>> than invent something new.
> Could be nice, but I think it's a different beast from project.el.
> I actually have the impression that what you describe already exists and
> is called EDE ;-)
>

Hmmm.   Now I'm back to the beginning of the thread.

Perhaps project.el is really project-root.el.  I think that would 
clarify a lot, and would continue to be helpful to the tools that only 
need to know about the root.  A clear intent would prevent 
misconceptions from causing the base to grow into something unintended.

I'm also interested in how to resolve some of the misconceptions about 
EDE.  Perhaps looking at what prevents it from being 'on by default', 
which I'm sure is a long list, is a good way to move it in the right 
direction, even if it never is on by default.

Eric

Re: {Spam?} Re: progmodes/project.el and search paths

[David Engster]

Eric Ludlam writes:
> On 08/06/2015 07:10 PM, Stefan Monnier wrote:
>>> It would be nice if the generic base project were the same.  Any project
>>> backend you use would have similar keybindings, menus, detection mechanisms,
>>> file location systems, etc because it will be easier to use the base support
>>> than invent something new.
>> Could be nice, but I think it's a different beast from project.el.
>> I actually have the impression that what you describe already exists and
>> is called EDE ;-)
>>
>
> Hmmm.   Now I'm back to the beginning of the thread.
>
> Perhaps project.el is really project-root.el.  I think that would
> clarify a lot, and would continue to be helpful to the tools that only
> need to know about the root.  A clear intent would prevent
> misconceptions from causing the base to grow into something
> unintended.

Despite what may have been said, project.el is *not* meant to be some
kind of "base class" for different project systems (that was at least
what I initially thought, and which you seem to think as well). The way
I see it, project.el is more like an adapter, although one that
delegates the actual implementation, so something like an "abstract
adapter". Dmitry and Stefan are saying that this adapter should only
provide the information that is common to all possible project systems,
which pretty much means "the list of files/directories that are part of
this project". While I agree this will already be very useful, as it
allows you to use xref, grep, ido-find-file, etc. on the current
project. I also think this is a very small feature for claiming the
whole project- namespace.

> I'm also interested in how to resolve some of the misconceptions about
> EDE.  Perhaps looking at what prevents it from being 'on by default',

Last time we discussed this it boiled down to "we cannot preload EIEIO".

-David

Re: {Spam?} Re: progmodes/project.el and search paths

[Alexis]

Eric Ludlam <ericludlam@gmail.com> writes:

> I'm also interested in how to resolve some of the misconceptions 
> about EDE.  Perhaps looking at what prevents it from being 'on 
> by default', which I'm sure is a long list, is a good way to 
> move it in the right direction, even if it never is on by 
> default.

Well, say i'm developing a small Perl5 project - one main module, 
supported by two helper modules, all of which are pure Perl. When 
i look at the overview of EDE in the Emacs manual:

    https://www.gnu.org/software/emacs/manual/html_node/emacs/EDE.html

and read:

    A project may contain one or more targets. A target can be an 
    object file, executable program, or some other type of file, 
    which is “built” from one or more of the files in the project.

i think to myself: "Okay, so EDE might be able to help me build 
(say) a .tar.gz when i feel the project is ready for release and 
distribution, but it's not relevant prior to that stage, since the 
source .pm files /are/ the executable files. There's nothing to 
'build' during development and testing, so it seems i have no use 
for EDE during those phases."

i would also guess that, similarly, EDE might not be that useful 
when developing in languages such as ELisp, Python or Ruby.

Am i wrong? What would using EDE get me in such contexts?


Alexis.

Re: {Spam?} Re: progmodes/project.el and search paths

[Dmitry Gutov]

On 08/07/2015 02:43 PM, David Engster wrote:

> Despite what may have been said, project.el is *not* meant to be some
> kind of "base class" for different project systems (that was at least
> what I initially thought, and which you seem to think as well). The way
> I see it, project.el is more like an adapter, although one that
> delegates the actual implementation, so something like an "abstract
> adapter".

In OOP terms, it most closely resembles an "interface".

> Dmitry and Stefan are saying that this adapter should only
> provide the information that is common to all possible project systems,
> which pretty much means "the list of files/directories that are part of
> this project".

Not necessarily. As has been touched on in a related discussion, it 
should be possible, for example, to define a fairly agnostic 
project-build-targets (or "tasks", not sure which is the more common term).

I'm sure the list can be continued.

Re: {Spam?} Re: progmodes/project.el and search paths

[David Engster]

Dmitry Gutov writes:
> On 08/07/2015 02:43 PM, David Engster wrote:
>
>> Despite what may have been said, project.el is *not* meant to be some
>> kind of "base class" for different project systems (that was at least
>> what I initially thought, and which you seem to think as well). The way
>> I see it, project.el is more like an adapter, although one that
>> delegates the actual implementation, so something like an "abstract
>> adapter".
>
> In OOP terms, it most closely resembles an "interface".

EDE already has an interface. project.el allows to use (a small part of)
that interface through another, hence 'adapter'.

>> Dmitry and Stefan are saying that this adapter should only
>> provide the information that is common to all possible project systems,
>> which pretty much means "the list of files/directories that are part of
>> this project".
>
> Not necessarily. As has been touched on in a related discussion, it
> should be possible, for example, to define a fairly agnostic
> project-build-targets (or "tasks", not sure which is the more common
> term).
>
> I'm sure the list can be continued.

Yes, but your usual answer so far was that you want a concrete package
that needs it, so there's a chicken/egg problem here.

-David

Re: {Spam?} Re: progmodes/project.el and search paths

[Dmitry Gutov]

On 08/07/2015 03:40 PM, David Engster wrote:

> EDE already has an interface. project.el allows to use (a small part of)
> that interface through another, hence 'adapter'.

Well, it's a different interface. We can have several.

An adapter, in this case, is the project.el backend for EDE, which does 
little more than delegation.

> Yes, but your usual answer so far was that you want a concrete package
> that needs it, so there's a chicken/egg problem here.

I disagree with that assessment: we need "concrete" consumers in Emacs 
anyway (at least some basic implementations).

Not too long ago, there was a discussion on emacs-devel with people 
complaining that Emacs lacks a lot of IDE-like features.

If certain important features of project.el are only used by third-party 
code (or, even worse, are still waiting for someone to use them), I'm 
sure the complaints will continue.

Re: progmodes/project.el and search paths

[Stephen Leake]

Dmitry Gutov <dgutov@yandex.ru> writes:

> On 08/06/2015 05:27 PM, Stephen Leake wrote:
>
>>> You can still run 'make' in the console,
>>
>> `next-error' is a very powerful Emacs feature; nothing equivalent is
>> available in a console.
>
> Point taken.
>
> Still, I wouldn't want to settle on a version of project-build-tasks
> over-specialized for, say, for C/C++ Makefile-based projects. The best
> way to do that is by implementing a flexible "run task" command.

That's what 'compile' _is_; it runs any command in a subprocess. How
much more flexible can you get?

-- 
-- Stephe

Re: progmodes/project.el and search paths

[Dmitry Gutov]

On 08/06/2015 10:25 AM, Stephen Leake wrote:

>> I think we should have project-find-regexp (or project-rgrep) which
>> uses the xref interface. grep-project will be functionally redundant.
>
> Since they are the same function, why is the name project-find-regexp
> better than the name grep-project?

project-rgrep sounds just as good to me (M-x grep is flat).

Would grep-project, in your view, use the xref interface? If not, 
they're not the same either.

> The latter will be _much_ easier to discover for the current grep user;
> M-x gre<tab> will show it.

On the other hand, M-x project-<tab> won't. And the user won't be able 
to discover all core commands that work on the current project.

Both problems should be solved by making execute-extended-command use a 
new completion category that would perform laxer matching.

> The purpose of project.el is to provide access to project information to
> other elisp programs; it is _not_ to provide user level programs.

It's the purpose of the API. Nothing says we can't provide some commands 
that use the API together with it.

We could move them to a yet-another package is that's the consensus, but 
there is value in keeping them together.

Not every project command is going to have a "normal" counterpart.

Re: progmodes/project.el and search paths

[Dmitry Gutov]

On 08/07/2015 05:10 PM, Stephen Leake wrote:

> That's what 'compile' _is_; it runs any command in a subprocess. How
> much more flexible can you get?

Well, if you're sure about that.

To continue this discussion, we'll need a proof-of-concept patch with 
new method(s) in project.el, a EDE-based implementation (I guess?) and 
the changes to the compile command that make use of the new information 
and at least provide completion for task names.

Maybe also allows switching between different environments to compile in 
(test/development/debug/release/etc).

Re: progmodes/project.el and search paths

[Nix]

On 3 Aug 2015, Stephen Leake told this:
> I prefer to cache it in a global variable; that way, I have only one
> project active at a time, for all buffers.

That doesn't work very well for those of us working on multi-component
projects. If I'm working in the Linux kernel I do *not* want identifier
search or file lookup wandering into glibc or Emacs, even though they
share some identifiers (with completely divergent definitions) and even
though I may have quite a lot of buffers open on those projects.

(I have fairly often worked on things that affect glibc, binutils, GCC,
and the kernel all at once. Those are different projects with distinct
roots, identifier sets, and tags tables...)

-- 
NULL && (void)

Re: progmodes/project.el and search paths

[Stephen Leake]

Nix <nix@esperi.org.uk> writes:

> On 3 Aug 2015, Stephen Leake told this:
>> I prefer to cache it in a global variable; that way, I have only one
>> project active at a time, for all buffers.
>
> That doesn't work very well for those of us working on multi-component
> projects. 

Fine; of course this behavior should be user-configurable.

At the same time, I work on multi-component projects all the time, and
one active project works for me.

> If I'm working in the Linux kernel I do *not* want identifier
> search or file lookup wandering into glibc or Emacs, even though they
> share some identifiers (with completely divergent definitions) and even
> though I may have quite a lot of buffers open on those projects.

> (I have fairly often worked on things that affect glibc, binutils, GCC,
> and the kernel all at once. Those are different projects with distinct
> roots, identifier sets, and tags tables...)

But if you are in glibc, looking at an identifier for a function that
happens to be defined in the linux kernel, you want to be able to find
the corresponding definition; the linux kernel is a subproject of glibc
(which is a subproject of higher level projects like Emacs). 

So the Emacs project tools should reflect that. For example, file name
completion must handle duplicate file names. That's a problem right now
in Emacs for elisp files; there are both lisp/cedet/ede/dired.el and
lisp/dired.el in Emacs core. 'load' and 'require' handle this by having
only lisp/cedet in load-path, not lisp/cedet/ede. So if I do file name
completion with just load-path, I have to know to type ede/d<tab> when
searching for ede/dired.el, which is _not_ friendly. But if I do file
name completion with an expanded load path that includes cedet/ede, it
shows only one dired.el (it happens to be ede/dired.el). Which is also
not friendly; I'll be filing a bug report on this later today.

For identifier search, you can use a cross reference tool that
understands name overloading; AdaCore asistant is one - it uses cross
reference information output by the compiler, which obviously handles
the duplicate names correctly. 

The multi-component projects I work on are in Ada, using the AdaCore
tools. Both make multi-component projects easier to work with, and Emacs
Ada mode understands the full project structure. Recently we added a
large C++ component to one product, and it suffered from some of the
above problems. So part of the solution is using a better programming
language (not always a choice, but it will never happen if we don't ask
for it!).

In addition, sometimes the user wants to search all subprojects,
sometimes a subset. So we need a good way of specifying subsets of
subprojects at search time. Emacs Ada mode doesn't support this yet (I
just put up with the unwanted hits in searches; better than false
negaitives). Hardcoding the searchable subset to only the top-level
project is overly restrictive.

-- 
-- Stephe

Re: progmodes/project.el and search paths

[David Engster]

Stephen Leake writes:
> Nix <nix@esperi.org.uk> writes:
>
>> On 3 Aug 2015, Stephen Leake told this:
>>> I prefer to cache it in a global variable; that way, I have only one
>>> project active at a time, for all buffers.
>>
>> That doesn't work very well for those of us working on multi-component
>> projects. 
>
> Fine; of course this behavior should be user-configurable.
>
> At the same time, I work on multi-component projects all the time, and
> one active project works for me.
>
>> If I'm working in the Linux kernel I do *not* want identifier
>> search or file lookup wandering into glibc or Emacs, even though they
>> share some identifiers (with completely divergent definitions) and even
>> though I may have quite a lot of buffers open on those projects.
>
>> (I have fairly often worked on things that affect glibc, binutils, GCC,
>> and the kernel all at once. Those are different projects with distinct
>> roots, identifier sets, and tags tables...)
>
> But if you are in glibc, looking at an identifier for a function that
> happens to be defined in the linux kernel, you want to be able to find
> the corresponding definition; the linux kernel is a subproject of
> glibc (which is a subproject of higher level projects like Emacs).

I must say I find this to be a very unusual view. Just because a program
depends on libc, the libc is not a "subproject". Instead, you have a
dependency on a library which has a fixed API. The only thing you need
are the declarations of the API and the needed type definitions, which
are given through the system headers, but those hardly qualify as a
"subproject"; they simply state how libc functions must be called. I
don't care how printf is implemented; in fact, I might have to switch to
musl or ulibc if my target system can't handle glibc's size. You can of
course have the full glibc and Emacs sources loaded as projects at the
same time, but they are separate projects on the same level in the
project hierarchy.

> So the Emacs project tools should reflect that. For example, file name
> completion must handle duplicate file names. That's a problem right now
> in Emacs for elisp files; there are both lisp/cedet/ede/dired.el and
> lisp/dired.el in Emacs core.

This is a different problem. You are within one project here; of course
a project system should be able to handle this.

-David

Re: progmodes/project.el and search paths

[Stephen Leake]

David Engster <deng@randomsample.de> writes:

> Stephen Leake writes:
>>
>> But if you are in glibc, looking at an identifier for a function that
>> happens to be defined in the linux kernel, you want to be able to find
>> the corresponding definition; the linux kernel is a subproject of
>> glibc (which is a subproject of higher level projects like Emacs).
>
> I must say I find this to be a very unusual view. Just because a program
> depends on libc, the libc is not a "subproject". 

We are arguing about the definition of the word "subproject".

The Oxford English Dictonary doesn't define it.

Wiktionary says: 

   A project within a larger project.

I agree libc is not "within" Emacs, so apparently I'm using the word
wrong.

> Instead, you have a
> dependency on a library which has a fixed API. 

"dependency" is fine with me. 

"fixed API"; only sometimes. If I have control over both the top level
project and the dependency, it is sometimes advantageous to change the
API to better implement some feature.

> The only thing you need are the declarations of the API and the needed
> type definitions, 

Sometimes. There were _many_ times at NASA when I wished we had paid the
contractor to deliver the full source code, rather than an API and user
guide.

Certainly in Emacs the doc string is often not enough, and I need to
read the body of the function to find out what's actually going on.

I often feel the need to do that in Android, but I haven't developed the
search/cross-reference skills to be able to yet.

> which are given through the system headers, 

Not all libraries are "system", but libraries do provide headers, yes.

Whether and where a library is "installed" is a packaging/distribution
detail; it should not impact this discussion.

> I don't care how printf is implemented; 

There are some libraries you trust (hopefully the ones you are paying
for), and others not so much.

>> So the Emacs project tools should reflect that. For example, file name
>> completion must handle duplicate file names. That's a problem right now
>> in Emacs for elisp files; there are both lisp/cedet/ede/dired.el and
>> lisp/dired.el in Emacs core.
>
> This is a different problem. 

Ok, now we are getting somewhere; this is one example of the kind of
problems I'm trying to address.

> You are within one project here; 

No, 'cedet' is a library as far as I'm concerned; it has an API, I try
to treat it as a black box. 

The fact that it is in Emacs core is just a packaging issue; it doesn't
change the logical structure.

The fact that Emacs locate-library searches within it just indicates
that Emacs got it right from my point of view :).

ELPA packages are also libraries; they are included in load-path when I
need them, and not otherwise.

The same problem occurs with any language library projects; if I'm
working on top_project, and need to read the impelementation for a
function that happens to be in library_project_1, they both need to be
in the file completion search path.

Same for identifier completion, find parent/child declarations, etc.

-- 
-- Stephe

Re: progmodes/project.el and search paths

[David Engster]

Stephen Leake writes:
> David Engster <deng@randomsample.de> writes:
>
>> Stephen Leake writes:
>>>
>>> But if you are in glibc, looking at an identifier for a function that
>>> happens to be defined in the linux kernel, you want to be able to find
>>> the corresponding definition; the linux kernel is a subproject of
>>> glibc (which is a subproject of higher level projects like Emacs).
>>
>> I must say I find this to be a very unusual view. Just because a program
>> depends on libc, the libc is not a "subproject". 
>
> We are arguing about the definition of the word "subproject".
>
> The Oxford English Dictonary doesn't define it.
>
> Wiktionary says: 
>
>    A project within a larger project.
>
> I agree libc is not "within" Emacs, so apparently I'm using the word
> wrong.

Actually, EDE has support for subprojects. For instance, EDE itself is a
subproject of CEDET, just like Semantic, SRecode, and Cogre. Together,
they form CEDET. While this is nice in theory, this generates more
problems than it is worth. For instance, EDE will generate a recursive
Makefile for this, which I'm really not very fond of...

>> The only thing you need are the declarations of the API and the needed
>> type definitions, 
>
> Sometimes. There were _many_ times at NASA when I wished we had paid the
> contractor to deliver the full source code, rather than an API and user
> guide.
>
> Certainly in Emacs the doc string is often not enough, and I need to
> read the body of the function to find out what's actually going on.
>
> I often feel the need to do that in Android, but I haven't developed the
> search/cross-reference skills to be able to yet.

We are in violent agreement here. Of course you should *have* the
source. But I would not put it into a subproject. I want a flat project
hierarchy.

>>> So the Emacs project tools should reflect that. For example, file name
>>> completion must handle duplicate file names. That's a problem right now
>>> in Emacs for elisp files; there are both lisp/cedet/ede/dired.el and
>>> lisp/dired.el in Emacs core.
>>
>> This is a different problem. 
>
> Ok, now we are getting somewhere; this is one example of the kind of
> problems I'm trying to address.
>
>> You are within one project here; 
>
> No, 'cedet' is a library as far as I'm concerned; it has an API, I try
> to treat it as a black box. 

Sorry, but CEDET is not a library. You might think that the lisp/cedet
directory is CEDET, but that's not true (unfortunately, as it makes
merging pretty tedious). The files are scattered all over the Emacs
sources, which is also why it doesn't make sense to treat lisp/cedet as
a "subproject". The grammars are in admin/grammars, SRecode files are in
etc/srecode, documentation is in doc/misc, EIEIO is in
lisp/emacs-lisp. They are all part of CEDET (though EIEIO is now
maintained in Emacs). Originally there was even more (speedbar, for
instance).

> The fact that it is in Emacs core is just a packaging issue; it doesn't
> change the logical structure.

First and foremost, we simply have files with the same name scattered
across the Emacs source tree. dired.el, custom.el, speedbar.el,
shell.el, and many more. This is a very common thing in larger projects,
and a project system must be able to handle this.

Technically, since lisp/cedet/ede is not in load-path, those two are
different files (dired.el and ede/dired.el). Also, all the symbols
defined in those files live in different namespaces (dired- and ede-,
resp.). So really, the only issue here is when you say "give me the file
dired.el from the Emacs project", in which case the project system
should tell you that it exists twice and asks you which one to load. But
it doesn't matter when you search for symbols.

> The same problem occurs with any language library projects; if I'm
> working on top_project, and need to read the impelementation for a
> function that happens to be in library_project_1, they both need to be
> in the file completion search path.

Say I really want to see the implementation of 'foo' in glibc. I simply
load the glibc project into my workspace (to use the Eclipse lingo); it
is not a subproject, but simply another project parallel to all the
others I have loaded. I can now jump from some Emacs C source file to
the 'foo' definition.

My guess your issue is that another library might have defined a symbol
'foo' as well, and its project is also loaded in my workspace - then it
depends on the #include's in your source file. So here, a project system
must ask something like Semantic for the search path of the current
file.

-David

Re: progmodes/project.el and search paths

[Stephen Leake]

David Engster <deng@randomsample.de> writes:

> Stephen Leake writes:
>> The same problem occurs with any language library projects; if I'm
>> working on top_project, and need to read the impelementation for a
>> function that happens to be in library_project_1, they both need to be
>> in the file completion search path.
>
> Say I really want to see the implementation of 'foo' in glibc. I simply
> load the glibc project into my workspace (to use the Eclipse lingo); it
> is not a subproject, but simply another project parallel to all the
> others I have loaded. I can now jump from some Emacs C source file to
> the 'foo' definition.

Do you have to load glibc manually? I would think that would be done for
you by the declared dependency.

So Eclipse implicitly adds all the projects to a single "source search
path".

That is what I'm asking for in Emacs project-related searches.

load-path does that for elisp. compilation-search-path does that for
most language modes.

In Ada mode lingo, that group of projects is one hierarchical project;
the project definition files define the project hierarchy. Ada mode
supports having several distict hierarchical projects loaded, but only
one active.

So that is the source of my confusion.

> My guess your issue is that another library might have defined a symbol
> 'foo' as well, and its project is also loaded in my workspace - then it
> depends on the #include's in your source file. So here, a project system
> must ask something like Semantic for the search path of the current
> file.

Depends on the use case; the user might want to see:

- "all (overloaded) definitions of "foo", in all projects "

    Perhaps to check whether current occurance could be using a
    different definition. Or to see if the different definitions could
    be combined, or further refactored.

    This is more useful in languages that explicitly support overloaded
    symbols (Ada has many definitions of "Put" in its standard library).

- "the definition for _this_ use of "foo", in some project "

    To see what it actually does.

- "all methods for this generic function"

    Similar to the first use case, but limited to one generic function.
    
So the question now is: what does EDE do for these use cases?

Ie, what functions does the user invoke in each case, and what is the
result?


The definition of xref-find-definitions is agnostic with respect to
these use cases; the "identifier" returned by xref--read-identifier can
be:

- a simple string for the first use case

- a more complex structure (in text properties) that identifies this
  occurance for the second and third use case

In addition, the backend can use the extra info or ignore it.

But the xref-find-definitions UI doesn't allow the user to express a
desire for one these uses.

In any case, the "source search path" used by xref-find-definitions
should include all of the relevant projects, as defined by the project
dependencies. 

--
-- Stephe

Re: progmodes/project.el and search paths

[Nix]

On 9 Aug 2015, David Engster outgrape:

> Stephen Leake writes:
>> Nix <nix@esperi.org.uk> writes:
>>> On 3 Aug 2015, Stephen Leake told this:
>>>> I prefer to cache it in a global variable; that way, I have only one
>>>> project active at a time, for all buffers.
>>>
>>> That doesn't work very well for those of us working on multi-component
>>> projects. 
>>
>> Fine; of course this behavior should be user-configurable.
>>
>> At the same time, I work on multi-component projects all the time, and
>> one active project works for me.
>>
>>> If I'm working in the Linux kernel I do *not* want identifier
>>> search or file lookup wandering into glibc or Emacs, even though they
>>> share some identifiers (with completely divergent definitions) and even
>>> though I may have quite a lot of buffers open on those projects.
>>
>>> (I have fairly often worked on things that affect glibc, binutils, GCC,
>>> and the kernel all at once. Those are different projects with distinct
>>> roots, identifier sets, and tags tables...)
>>
>> But if you are in glibc, looking at an identifier for a function that
>> happens to be defined in the linux kernel, you want to be able to find
>> the corresponding definition; the linux kernel is a subproject of
>> glibc (which is a subproject of higher level projects like Emacs).
>
> I must say I find this to be a very unusual view.

In this specific case, I'd call it strictly incorrect. I picked those
examples for a reason: they're an example of two sorta-related projects
which define identical identifiers which relate to *different things* in
each project: the kernel is in effect a freestanding environment and
*does not* depend on glibc (indeed, the opposite is true, glibc depends
on the kernel, even though it cannot resolve any symbols in it and no
identifier lookups should ever traverse from the one to the other).

e.g. the kernel defines many of the *printf() functions, and so does
glibc -- but they define them in different places. If something inside
glibc references snprintf(), the identifier lookup should proceed within
glibc alone: if something inside the kernel references snprintf(), it
had better find the copy in the kernel's lib/vsprintf.c.

(Meanwhile, there are a lot of projects that really *do* depend on each
other and symbol resolution in one can look up symbols in the other: it
would be nice if identifier search could follow that, but it is very far
from crucial.)

>> So the Emacs project tools should reflect that. For example, file name
>> completion must handle duplicate file names. That's a problem right now
>> in Emacs for elisp files; there are both lisp/cedet/ede/dired.el and
>> lisp/dired.el in Emacs core.
>
> This is a different problem. You are within one project here; of course
> a project system should be able to handle this.

Yes indeed.

-- 
NULL && (void)

Re: progmodes/project.el and search paths

[Dmitry Gutov]

Hi Eric,

Sorry for the late reply. I wasn't really sure what kind of response is 
expected.

In short, I like your vision as a whole, but whether each particular 
element of it can apply to project.el, should be investigated separately.

On some of them:

* I wonder what's the main use of having hierarchical projects vs. 
simply tracking several currently open projects (or project roots). And 
how the consumers of the API would take advantage of that distinction.

* Project services seem to be the proper way to handle optional traits 
of significant size that some projects have, and others don't.

* "Project Loading safety" are probably out of scope. I'd expect the 
majority of implementations to be based around loading a project build 
(or settings) file of pre-existing format. The popular IDEs don't ask 
the user whether they want to load the project file. On the other hand, 
some project types might need to be enabled/visited explicitly.

* WRT "Detection ordering", the current project-find-functions structure 
might be enough. Any Makefile project implementation will most likely go 
before the VC project anyway (unless the respective add-hook form is 
told to APPEND, for some reason). And it's hard for us to expect for 
third-party implementations to be able to declare priorities between 
themselves.

* "Update meta-data" might be handled by each respective implementation, 
in conjunction with auto-revert-mode.

* "Project browsing" shouldn't need much more than project.el exposes 
already.

* "Project Local Variables" look very similar to project metadata.

* The "'include path' concept" looks very similar to 
project-search-path. This might be the main purpose for the 
project-search-path entries that lie within project-roots.

* WRT cscope, etc, integration, I think the main unsolved problem we 
currently have is that they aren't re-run automatically. Nor do we have 
a command that would do that for the user on demand.

* "Load a project" seems incompatible with the project-find-functions 
approach. Which is unfortunate.

* "navigating between code and doc", as stated, doesn't seem 
particularly useful: if you're reading the source code of a function, 
the docstring is usually right above. Jumping from the doc to the source 
is better, but this is closer to the domain of xref.

* "Language specificity" might be useful, but in practical terms it 
sounds like adding an optional (?) "language" argument to 
project-search-path and any similar method that we add later. Which 
complicates implementations. Or just have an Emacs-global repository of 
file name patterns corresponding to languages? Not sure.

* "commands specific to project implementations" could also be added by 
the minor modes supplied by the implementations' packages.

On 08/05/2015 04:29 AM, Eric Ludlam wrote:

> Emacs has a lot of tools that could use a project service, and sometimes
> all the tool needs is a small push of convenience, and not full project
> management system.  A generic project system will make it easy to add
> small conveniences in lots of places you may not expect because tool
> authors will feel safe using it.

That's the idea, yes.

> The attached is my experience of things I either needed in EDE's base
> classes to implement a project, or that some tool needed, and what I
> learned along the way.  You might think it is too EDE specific,

I wouldn't say the majority of items in the list are EDE specific.

> I consider this file incomplete, but I hope it proves useful as a
> starting point.  I'd be happy to periodically update it for a little
> while if it is useful.

Were we using a nice issue tracker, I'd have asked to create a tracking 
issue (or would've created one myself), but in the present situation, I 
guess you'll have to resend it with updates.

But in any case, I'd say technical proposals on specific features, as 
well as patches, would provide more value here, rather than maintaining 
a high-level overview of things we might get around to someday.

Re: progmodes/project.el and search paths

[Eric Ludlam]

Hi Dmitry,

Thanks for the thoughtful reply.

On 08/11/2015 04:01 PM, Dmitry Gutov wrote:
> Hi Eric,
>
> Sorry for the late reply. I wasn't really sure what kind of response 
> is expected.
>
> In short, I like your vision as a whole, but whether each particular 
> element of it can apply to project.el, should be investigated separately.
>
> On some of them:
>
> * I wonder what's the main use of having hierarchical projects vs. 
> simply tracking several currently open projects (or project roots). 
> And how the consumers of the API would take advantage of that 
> distinction.
>

I doubt consumers of the project API would care.  Nested projects is 
simply something the detection system should enable.  If there is a 
centralized way of scanning for projects, it would need to support 
this.  If each project system detects projects its own way, then it 
wouldn't matter at the core level.

My hypothesis is that project detection is tricky, and new custom 
projects would pop up more frequently if there were infrastructure for 
detecting them.  If project.el were to provide that infrastructure, it 
would need to support nested projects.

> * Project services seem to be the proper way to handle optional traits 
> of significant size that some projects have, and others don't.

> * "Project Loading safety" are probably out of scope. I'd expect the 
> majority of implementations to be based around loading a project build 
> (or settings) file of pre-existing format. The popular IDEs don't ask 
> the user whether they want to load the project file. On the other 
> hand, some project types might need to be enabled/visited explicitly.
>

Every project is indeed different.  A system that needs to load lisp 
code needs to protect against ill formatted lisp code.  Putting this in 
a core system will help implementations from all creating the same 
latent bug.  If project.el isn't going to provide detection services, 
then it won't need this feature.

> * WRT "Detection ordering", the current project-find-functions 
> structure might be enough. Any Makefile project implementation will 
> most likely go before the VC project anyway (unless the respective 
> add-hook form is told to APPEND, for some reason). And it's hard for 
> us to expect for third-party implementations to be able to declare 
> priorities between themselves.

Project.el is providing a list of ways for projects to present 
themselves.  At the moment however, projectile v EDE v your vc detector 
could all claim ownership of the same project.  At the moment it 
probably doesn't matter because since all three provide different 
services based on their personal consideration of 'root'. Any tool 
needing to know only the root would be happy, but if there were some 
extra service to be provided, a priority between the 3 would be needed.  
If project.el were to provide the detection service as well, then the 3 
would need to prioritize.

In EDE, a project adds itself to the list of possible projects. Each 
project classifies itself by a term that places itself in one of three 
bins, and hopefully those bins don't conflict. ;)

Anyway, EDE's bins are:

'unique - something like the Emacs or Linux project - detect these first.

  no flag provide - One of the specialized build system projects like 
Automake support, or Maven.

'generic - Something that provides very little overall support, such as 
detection by VC, or existence of SCons. - Always at the end.

This is pretty easy to self bin by.

> * "Update meta-data" might be handled by each respective 
> implementation, in conjunction with auto-revert-mode.

Right.  Each project implementation needs to update their data from the 
file system their own way.  If there are generic development menus, it 
would be helpful for the user to have one way to update project meta 
data regardless of which type of implementation is active.

> * "Project browsing" shouldn't need much more than project.el exposes 
> already.
>
Yup.  I was trying to be complete.

> * "Project Local Variables" look very similar to project metadata.

They are a type of meta data that let users provide extended behavior to 
the project.

> * The "'include path' concept" looks very similar to 
> project-search-path. This might be the main purpose for the 
> project-search-path entries that lie within project-roots.
>

I will need to look at your search path concept more.  Are there 
multiple paths for different purposes?

> * WRT cscope, etc, integration, I think the main unsolved problem we 
> currently have is that they aren't re-run automatically. Nor do we 
> have a command that would do that for the user on demand.
>

Several of these tools have an 'update the current tags file' type mode 
you can get at.  I don't recall if I wrapped that up with a handy 
function or not.

> * "Load a project" seems incompatible with the project-find-functions 
> approach. Which is unfortunate.
>
> * "navigating between code and doc", as stated, doesn't seem 
> particularly useful: if you're reading the source code of a function, 
> the docstring is usually right above. Jumping from the doc to the 
> source is better, but this is closer to the domain of xref.

Just pulling out examples, some were from threads on the ML.

The key concept is that a project is rarely just one single activity.  
Usually there are doc activities, code activities, build system 
activities.  Some projects have Java activities, and native C++ 
activities.  All these different things are for a common goal, and often 
there is a common lingo between them.  I represent that lingo as 'tags' 
in Semantic, and it enables you to 'copy' a tag which includes the meta 
data around the name and 'paste' it into a buffer of a different 
language.  The only useful one there can turn a lisp tag into a texi 
@defun block, but you could imagine a way to update your java native 
methods bidirectionally.   The project knows how to link these systems 
together, such as finding the right doc for your Lisp code.

This is an advanced maneuver that would be great for a tool author to be 
able to do, and is more likely if that kind of association can be easily 
made.

> * "Language specificity" might be useful, but in practical terms it 
> sounds like adding an optional (?) "language" argument to 
> project-search-path and any similar method that we add later. Which 
> complicates implementations. Or just have an Emacs-global repository 
> of file name patterns corresponding to languages? Not sure.
>

Language specificity was key for me because of the way mode-local works, 
and how each language had customized behavior for the parsing tools.  I 
tended to focus on language specific accurate behaviors, and not generic 
potentially inaccurate features.  For example, smart completion vs dabbrevs.

As noted somewhere, symref had a way of converting modes to patterns.  I 
had to ask around, but eventually got something pretty good.

[...]
>> I consider this file incomplete, but I hope it proves useful as a
>> starting point.  I'd be happy to periodically update it for a little
>> while if it is useful.
>
> Were we using a nice issue tracker, I'd have asked to create a 
> tracking issue (or would've created one myself), but in the present 
> situation, I guess you'll have to resend it with updates.
>
> But in any case, I'd say technical proposals on specific features, as 
> well as patches, would provide more value here, rather than 
> maintaining a high-level overview of things we might get around to someday

My goal is to be more helpful than adversarial, but at the time I didn't 
have a clear picture where project.el would fit, and if there were 
anything I had to deal with in the future.

I'd be glad if I could stop maintaining parts of EDE because it became 
part of some core system.   It is still unclear to me where the 
boundaries lie.  I'm not sure where a patch would be helpful yet.  My 
hope is that my document would help you define the bounds of project.el 
based in a set of terms we can both agree to.  It would then be obvious 
how I could help by donating pieces of EDE as needed.

My recommendation for you is to perhaps not reply to individual 
clarifications above, but perhaps define where you think project.el fits 
into the whole using whatever you like from my doc, and that would help 
us focus the conversation on the boundary, and not on random things I 
invented that aren't close to the boundary.

Thanks
Eric

project terminology

[Stephen Leake]

Eric Ludlam <ericludlam@gmail.com> writes:

> On 08/11/2015 04:01 PM, Dmitry Gutov wrote:
>>
>> * I wonder what's the main use of having hierarchical projects vs.
>> simply tracking several currently open projects (or project roots).
>> And how the consumers of the API would take advantage of that
>> distinction.
>>
>
> I doubt consumers of the project API would care.  Nested projects is
> simply something the detection system should enable.  

I'm puzzled by this terminology; I don't think "nested" means the same
thing as "hierarchical"; nested is a subset of hierarchical.

For example, here are two possible project layouts on the disk:

layout 1:

/core
    # math and IO utils

/my_proj
    # project file includes dependency on /core


layout 2:

/my_proj
    # no dependency in project file
    /core
        # math and IO utils


Other than the layout change, the code is the same (the build scripts
that specify paths etc are slightly different).

I would call both of these "hierarchical", but only layout 2 is
"nested".

Apparently, the term "subproject" is also appropriate for core in layout
2.

In any case, it would be good to have the option to treat "my_proj" as a
single project, that "includes" "core" in any operations that involve
searching for things in "the current project".

As for distinguishing hiearchical from a simple set of open projects,
that's what the project dependency declarations are for. They don't
necessarily need to be visible to project.el; some backends may handle
that entirely on their own (the Ada mode backend does, for example).

> A system that needs to load lisp
> code needs to protect against ill formatted lisp code.  Putting this
> in a core system will help implementations from all creating the same
> latent bug.  If project.el isn't going to provide detection services,
> then it won't need this feature.

There is a difference between detecting the presence of a project, and
using that project.

An extremely simple way to detect projects is to ask the user to specify
a project location. Another way is to look for a special file name.

Once the project file is found, then it must be loaded. But the
detection step (whether ask user or some other method), does not require
loading the elisp file (or parsing the project file in general).

In the above, I've assumed "project detection" meant "deciding whether
there is a project in a disk directory".

In other circustances, "project detection" means "pick which of the
available projects is appropriate for the current action".

It's not always clear from context which meaning is meant. Perhaps we
could use the term "project selection" for the latter.

In project.el, project-find is used for project selection, but the
current implementations of it do project detection as the selection
algorithm. Other choices are possible; the "project-explicit"
implementation I proposed separates detection from selection.

I think EDE is similar, in that it tends to use project detection as
part of the selection algorithm.

-- 
-- Stephe

Re: progmodes/project.el and search paths

[Dmitry Gutov]

On 08/12/2015 03:49 AM, Eric Ludlam wrote:

> Every project is indeed different.  A system that needs to load lisp
> code needs to protect against ill formatted lisp code.  Putting this in
> a core system will help implementations from all creating the same
> latent bug.  If project.el isn't going to provide detection services,
> then it won't need this feature.

Probably not, at least not at first.

> In EDE, a project adds itself to the list of possible projects. Each
> project classifies itself by a term that places itself in one of three
> bins, and hopefully those bins don't conflict. ;)

Not to dismiss the detailed approach, but the user could reorder the 
elements in project-find-functions as well, if the default order is not 
to their liking.

What you're describing would be an improvement, but not of the 
revolutionary kind. And I'm not really sure how to fit the 
classification together with project-find-functions.

> Right.  Each project implementation needs to update their data from the
> file system their own way.  If there are generic development menus, it
> would be helpful for the user to have one way to update project meta
> data regardless of which type of implementation is active.

I'd rather user never had to do that at all.

>> * "Project Local Variables" look very similar to project metadata.
>
> They are a type of meta data that let users provide extended behavior to
> the project.

That sounds fine in theory, but we need a more concrete application. 
Your Projectile example, for instance, would work just as well if it 
didn't set the "project local variables" via the project API, and just 
stored the information in its internal storage.

> I will need to look at your search path concept more.  Are there
> multiple paths for different purposes?

No, just this one. Adding other paths will need more justification.

> Several of these tools have an 'update the current tags file' type mode
> you can get at.  I don't recall if I wrapped that up with a handy
> function or not.

That would be nice. But then you don't need any extra information about 
the current project to do that, just knowing how to find the file and/or 
call the tool.

In that case, it shouldn't need anything new in the project API. Not 
even how to find the root, maybe (just use locate-dominating-file, 
especially when there are several tags files in the project).

> The key concept is that a project is rarely just one single activity.
> Usually there are doc activities, code activities, build system
> activities.  Some projects have Java activities, and native C++
> activities.  All these different things are for a common goal, and often
> there is a common lingo between them.  I represent that lingo as 'tags'
> in Semantic, and it enables you to 'copy' a tag which includes the meta
> data around the name and 'paste' it into a buffer of a different
> language.

That sounds like it needs a parser to work well. xref tries to handle it 
as "identifier at point", but we'll have to see whether the different 
implementations will include context information in it (via text 
properties).

> The only useful one there can turn a lisp tag into a texi
> @defun block, but you could imagine a way to update your java native
> methods bidirectionally.   The project knows how to link these systems
> together, such as finding the right doc for your Lisp code.

For one thing, writing documentation separately from sources is an alien 
idea to me (and for other Ruby, Java or JS programmers, I think). So 
this kind of information may need to be exposed as an optional service.

> Language specificity was key for me because of the way mode-local works,
> and how each language had customized behavior for the parsing tools.  I
> tended to focus on language specific accurate behaviors, and not generic
> potentially inaccurate features.  For example, smart completion vs
> dabbrevs.

I like smart completion, but by tying it to projects we'd discount the 
less accurate implementations that are still useful. On the other hand, 
a package that implements project handling can add its own, accurate 
completion functions as well.

> As noted somewhere, symref had a way of converting modes to patterns.  I
> had to ask around, but eventually got something pretty good.

It didn't work well for Ruby files initially, so I had to add the 
respective entries to semantic-symref-filepattern-alist. I'm sure there 
are languages out there that are still not covered.

> I'd be glad if I could stop maintaining parts of EDE because it became
> part of some core system.   It is still unclear to me where the
> boundaries lie.  I'm not sure where a patch would be helpful yet.  My
> hope is that my document would help you define the bounds of project.el
> based in a set of terms we can both agree to.  It would then be obvious
> how I could help by donating pieces of EDE as needed.

My understanding of EDE is if there are tools that would be worth 
porting, they would need to be rewritten on top of more general methods, 
and would probably need rethinking.

> My recommendation for you is to perhaps not reply to individual
> clarifications above, but perhaps define where you think project.el fits
> into the whole using whatever you like from my doc, and that would help
> us focus the conversation on the boundary, and not on random things I
> invented that aren't close to the boundary.

I'll try to do something like that in the foreseeable future. But 
really, from where I'm standing, the question is not how to decide the 
list of features, but how to implement each particular feature in a way 
that would make sense for as many project types as possible.

Re: progmodes/project.el and search paths

[David Engster]

Stephen Leake writes:
> David Engster <deng@randomsample.de> writes:
>
>> Stephen Leake writes:
>>> The same problem occurs with any language library projects; if I'm
>>> working on top_project, and need to read the impelementation for a
>>> function that happens to be in library_project_1, they both need to be
>>> in the file completion search path.
>>
>> Say I really want to see the implementation of 'foo' in glibc. I simply
>> load the glibc project into my workspace (to use the Eclipse lingo); it
>> is not a subproject, but simply another project parallel to all the
>> others I have loaded. I can now jump from some Emacs C source file to
>> the 'foo' definition.
>
> Do you have to load glibc manually?

Yes. You can either create a separate project for glibc and add this to
your workspace, or you add the sources of glibc directly to your project
as a special "library source". The latter is what you usually do, unless
you are a glibc developer.

> I would think that would be done for you by the declared dependency.

Since you only need the headers for compilation and the library object
for linking, there's no need for Eclipse to force you to add the
sources. It also cannot add them automatically, because they are usually
not installed in the first place. My current Emacs binary depends on 73
libraries, and AFAIK I have sources for exactly none of them installed.

> So Eclipse implicitly adds all the projects to a single "source search
> path".

Not implicitly; you have to configure that.

> That is what I'm asking for in Emacs project-related searches.
>
> load-path does that for elisp. compilation-search-path does that for
> most language modes.
>
> In Ada mode lingo, that group of projects is one hierarchical project;
> the project definition files define the project hierarchy. Ada mode
> supports having several distict hierarchical projects loaded, but only
> one active.
>
> So that is the source of my confusion.

I see.

>> My guess your issue is that another library might have defined a symbol
>> 'foo' as well, and its project is also loaded in my workspace - then it
>> depends on the #include's in your source file. So here, a project system
>> must ask something like Semantic for the search path of the current
>> file.
>
> Depends on the use case; the user might want to see:
>
> - "all (overloaded) definitions of "foo", in all projects "
>
>     Perhaps to check whether current occurance could be using a
>     different definition. Or to see if the different definitions could
>     be combined, or further refactored.
>
>     This is more useful in languages that explicitly support overloaded
>     symbols (Ada has many definitions of "Put" in its standard library).
>
> - "the definition for _this_ use of "foo", in some project "
>
>     To see what it actually does.
>
> - "all methods for this generic function"
>
>     Similar to the first use case, but limited to one generic function.
>
> So the question now is: what does EDE do for these use cases?

AFAIK, you can only search in the current project. However, your use
cases involve more than just a project system. If you want to search for
overloads or methods, you need a parser as well. This also applies to
the most important feature (at least for me): show all definitions of
'foo' that are currently in scope. EDE has no notions of symbols, scope,
overloading, methods, etc.

It gets much more complicated when you're not only interested in
definitions, but any occurance of a symbol. For instance, you want all
places where 'foo' gets called, but not any 'foo', but the method 'foo'
from class 'bar' and not from any other class.

> The definition of xref-find-definitions is agnostic with respect to
> these use cases; the "identifier" returned by xref--read-identifier can
> be:
>
> - a simple string for the first use case
>
> - a more complex structure (in text properties) that identifies this
>   occurance for the second and third use case
>
> In addition, the backend can use the extra info or ignore it.
>
> But the xref-find-definitions UI doesn't allow the user to express a
> desire for one these uses.

I can't really say much about xref since I haven't really worked with it
yet. The semantic-symref feature is also pretty limited in this
regard. So in the end, I often ended up calling grep or global
directly. All this stuff is highly dependent on the language you're
using. 

> In any case, the "source search path" used by xref-find-definitions
> should include all of the relevant projects, as defined by the project
> dependencies. 

I agree.

-David

Re: progmodes/project.el and search paths

[Stephen Leake]

David Engster <deng@randomsample.de> writes:

> Stephen Leake writes:
>> Depends on the use case; the user might want to see:
>>
>> - "all (overloaded) definitions of "foo", in all projects "
>>
>>     Perhaps to check whether current occurance could be using a
>>     different definition. Or to see if the different definitions could
>>     be combined, or further refactored.
>>
>>     This is more useful in languages that explicitly support overloaded
>>     symbols (Ada has many definitions of "Put" in its standard library).
>>
>> - "the definition for _this_ use of "foo", in some project "
>>
>>     To see what it actually does.
>>
>> - "all methods for this generic function"
>>
>>     Similar to the first use case, but limited to one generic function.
>>
>> So the question now is: what does EDE do for these use cases?
>
> AFAIK, you can only search in the current project. However, your use
> cases involve more than just a project system. If you want to search for
> overloads or methods, you need a parser as well. 

Yes. 

> This also applies to the most important feature (at least for me):
> show all definitions of 'foo' that are currently in scope. EDE has no
> notions of symbols, scope, overloading, methods, etc.

Yes, I'm beginning to realize that. Semantic provides those things. And
it can use different backends, so it can serve as the Emacs standard
interface to such things (although I'd want to make significant changes
to use it for Ada mode).

In that sense, Semantic competes with xref in the same way EDE competes
with project.el.

> It gets much more complicated when you're not only interested in
> definitions, but any occurance of a symbol. For instance, you want all
> places where 'foo' gets called, but not any 'foo', but the method 'foo'
> from class 'bar' and not from any other class.

Right; that's name resolution; best done by the compiler. So hopefully
it can output cross reference information (or provide access to it in
some way).

> I can't really say much about xref since I haven't really worked with it
> yet. The semantic-symref feature is also pretty limited in this
> regard. So in the end, I often ended up calling grep or global
> directly. 

Right.

> All this stuff is highly dependent on the language you're using.

The backend may be, but at the API level, there are some things that all
object oriented languages support:

- Search for all implementations of this method.

    That's the language used by elisp cl-generic and eieio defclass. I
    just made this the default behavior for xref-find-definitions.
    Although that's not tested on eieio classes; I'll add that to my
    list.
    
    In Ada, it's called "functions that override primitive operations".

    In C++, it's "functions that override a virtual function".

    The terminology is different, but the underlying notion of runtime
    dispatch is the same.

    We might want to customize the menu entry depending on the current
    programming language.
    
- Search for the parents of this type.

    "Type" is the Ada word; other languages use "class" here.

- Search for all functions that overload this one.

    Ada and C++ (at least) support this.
    
- Search for all references to this function.

In C, these searches are simpler; the first three only return one
result.

-- 
-- Stephe

Re: progmodes/project.el and search paths

[David Engster]

Stephen Leake writes:
> David Engster <deng@randomsample.de> writes:
>> This also applies to the most important feature (at least for me):
>> show all definitions of 'foo' that are currently in scope. EDE has no
>> notions of symbols, scope, overloading, methods, etc.
>
> Yes, I'm beginning to realize that. Semantic provides those things. And
> it can use different backends, so it can serve as the Emacs standard
> interface to such things (although I'd want to make significant changes
> to use it for Ada mode).
>
> In that sense, Semantic competes with xref in the same way EDE competes
> with project.el.

There is some overlap, especially in the GUI area for presenting
references. But at least I would be happy to give that up in favor of
xref, since CEDET was always planned to be more on the backend side of
things, and getting the GUI right is nasty, and maintaining it takes a
lot of time I'd rather spend elsewhere. So I'm happy that Dmitry is
working on this (also with maintaining company-mode, which IMO should
become part of core Emacs rather sooner than later).

With respect to project.el, I'm still not sure where this is going.

>> It gets much more complicated when you're not only interested in
>> definitions, but any occurance of a symbol. For instance, you want all
>> places where 'foo' gets called, but not any 'foo', but the method 'foo'
>> from class 'bar' and not from any other class.
>
> Right; that's name resolution; best done by the compiler.

Yeah well, we had that discussion. I hope I find the time to work on
that. Currently, there are more urgent issue, like getting CEDET
mergeable again after the switch to git and the extensive EIEIO
refactoring...

>> All this stuff is highly dependent on the language you're using.
>
> The backend may be, but at the API level, there are some things that all
> object oriented languages support:
>
> - Search for all implementations of this method.
>
>     That's the language used by elisp cl-generic and eieio defclass. I
>     just made this the default behavior for xref-find-definitions.
>     Although that's not tested on eieio classes; I'll add that to my
>     list.
>     
>     In Ada, it's called "functions that override primitive operations".
>
>     In C++, it's "functions that override a virtual function".
>
>     The terminology is different, but the underlying notion of runtime
>     dispatch is the same.
>
>     We might want to customize the menu entry depending on the current
>     programming language.

As always, it is more complicated in C++ (overload vs override,
namespaces, argument dependent lookup, etc.), but currently I don't
worry too much about details. Getting the usual dispatch right would
cover a lot of ground.

> - Search for the parents of this type.
>
>     "Type" is the Ada word; other languages use "class" here.
>
> - Search for all functions that overload this one.
>
>     Ada and C++ (at least) support this.
>     
> - Search for all references to this function.

Yes.

> In C, these searches are simpler; the first three only return one
> result.

Aah, the joys of ANSI C. Our parser is actually pretty good for
that. :-)

-David

Re: progmodes/project.el and search paths

[Nix]

On 12 Aug 2015, David Engster verbalised:

> Stephen Leake writes:
>> - Search for all implementations of this method.
>>
>>     That's the language used by elisp cl-generic and eieio defclass. I
>>     just made this the default behavior for xref-find-definitions.
>>     Although that's not tested on eieio classes; I'll add that to my
>>     list.
>>     
>>     In Ada, it's called "functions that override primitive operations".
>>
>>     In C++, it's "functions that override a virtual function".
>>
>>     The terminology is different, but the underlying notion of runtime
>>     dispatch is the same.
>>
>>     We might want to customize the menu entry depending on the current
>>     programming language.
>
> As always, it is more complicated in C++ (overload vs override,
> namespaces, argument dependent lookup, etc.), but currently I don't
> worry too much about details. Getting the usual dispatch right would
> cover a lot of ground.

Quite. Given that name lookup can be literally arbitrarily complicated
in C++, the only way to get it guaranteed right there involves writing a
C++ compiler in elisp or compiler assistance -- and even that is kind of
hopeless, because of course CEDET has to routinely work with
uncompilable and only-partially-written buffers. (This is a fairly
strong argument *against* relying on the compiler, in fact -- we'll
always want something like Semantic, because we can tell it to be less
pedantic in its parsing, recover as much as possible from every parse
error, and so on.)

>> In C, these searches are simpler; the first three only return one
>> result.
>
> Aah, the joys of ANSI C. Our parser is actually pretty good for
> that. :-)

Now, of course, we have C11's _Generic to make things more interesting
:)

-- 
NULL && (void)

Re: progmodes/project.el and search paths

[Dmitry Gutov]

On 08/13/2015 02:53 PM, Nix wrote:

> Quite. Given that name lookup can be literally arbitrarily complicated
> in C++, the only way to get it guaranteed right there involves writing a
> C++ compiler in elisp or compiler assistance -- and even that is kind of
> hopeless, because of course CEDET has to routinely work with
> uncompilable and only-partially-written buffers.

There are compilers written with editing and incomplete code in mind. 
Clang, of course, performs some at least error recovery, or else it 
would be useless for code completion.

> (This is a fairly
> strong argument *against* relying on the compiler, in fact -- we'll
> always want something like Semantic, because we can tell it to be less
> pedantic in its parsing, recover as much as possible from every parse
> error, and so on.)

Can we?

I'm guessing Semantic has at most one level of being "less pedantic", 
and implementing several of them, as well as support for those in 
grammars, wouldn't be trivial.

Re: progmodes/project.el and search paths

[Eric Ludlam]

On 08/13/2015 08:05 AM, Dmitry Gutov wrote:
> On 08/13/2015 02:53 PM, Nix wrote:

>> (This is a fairly
>> strong argument *against* relying on the compiler, in fact -- we'll
>> always want something like Semantic, because we can tell it to be less
>> pedantic in its parsing, recover as much as possible from every parse
>> error, and so on.)
>
> Can we?
>
> I'm guessing Semantic has at most one level of being "less pedantic",
> and implementing several of them, as well as support for those in
> grammars, wouldn't be trivial.

While Semantic does have an implementation of Bison as it's core for 
some languages, it is quite unlike most compilers.  You can think of it 
has having 4 ways of handling errors the grammar author doesn't have to 
put effort into:

1) Lexical 'block' symbols

The lexer can identify blocks, which would normally be handled by the 
parser.  This is because Emacs has built in block matching that is quite 
good.  This lets the parser completely ignore anything in { code } 
blocks or ( paramater lists ) unless it is important.  As long as your 
block indicators match up, it doesn't matter how crappy your code is.

2) Parsers don't check types, etc as they go, and are flexible about the 
symbols they accept.

This means you can get irrational tags.  This is for a combination of 
handling incomplete code, speed (less to do) and because most folks just 
go an run their compiler at some point anyway.

3) Iterative parsing, so each invocation of the parser only executes 
across a single tag.

This means that if parsing fails, we just move forward one lexical token 
and try again.  This lets the parser just wash over anything it doesn't get.

4) Incremental parser

Emacs watches you edit, tracks your changes, and matches it up against 
previously parsed tags.  If things are 'broken' in the code you edit, 
but you didn't touch the boundaries of the other nearby tags, it lets it 
be until next time.  This means all the previously captured data is 
available for use, including the active broken tag you might be editing.


It also has the typical bison error handling, but that tends not to be 
necessary.

In summary; Semantic can extract useful data out of pretty badly trashed 
code, and track your changes while you edit in a robust way.

Eric

Re: progmodes/project.el and search paths

[Dmitry Gutov]

On 08/14/2015 02:52 PM, Eric Ludlam wrote:

> 2) Parsers don't check types, etc as they go, and are flexible about the
> symbols they accept.
>
> This means you can get irrational tags.  This is for a combination of
> handling incomplete code, speed (less to do) and because most folks just
> go an run their compiler at some point anyway.

Doesn't that mean that code completion at the end of a chain of calls 
can be inaccurate (with e.g. lots of false positives), even if code is 
otherwise valid?

> 3) Iterative parsing, so each invocation of the parser only executes
> across a single tag.
>
> This means that if parsing fails, we just move forward one lexical token
> and try again.  This lets the parser just wash over anything it doesn't
> get.

AFAIK, parsers in general try to ignore errors, or else you wouldn't be 
able to get any useful information, after compilation fails, about any 
errors after the first one.

> 4) Incremental parser
>
> Emacs watches you edit, tracks your changes, and matches it up against
> previously parsed tags.  If things are 'broken' in the code you edit,
> but you didn't touch the boundaries of the other nearby tags, it lets it
> be until next time.  This means all the previously captured data is
> available for use, including the active broken tag you might be editing.

That sounds useful.

> It also has the typical bison error handling, but that tends not to be
> necessary.
>
> In summary; Semantic can extract useful data out of pretty badly trashed
> code, and track your changes while you edit in a robust way.

It seems I stand corrected, thanks. However, it would be interesting to 
do a comparison on some real file, against a widely used compiler, such 
as Clang.

Re: progmodes/project.el and search paths

[Eric Ludlam]

On 08/14/2015 06:30 PM, Dmitry Gutov wrote:
> On 08/14/2015 02:52 PM, Eric Ludlam wrote:
>
>> 2) Parsers don't check types, etc as they go, and are flexible about the
>> symbols they accept.
>>
>> This means you can get irrational tags.  This is for a combination of
>> handling incomplete code, speed (less to do) and because most folks just
>> go an run their compiler at some point anyway.
>
> Doesn't that mean that code completion at the end of a chain of calls 
> can be inaccurate (with e.g. lots of false positives), even if code is 
> otherwise valid?

While the parser generally ignores data types, it does record the 
declarations.  When smart completion is invoked, only the types 
currently relevant are resolved in order to make the completions 
accurate.  Thus:

Parsing phase: Just record the data as declared, don't check.

Completion phase: Look up data types as needed for symbols in the 
completion string.

If point of fact, Semantic's tagging parser doesn't parse the bodies of 
functions, and just skips over them.  The body is only parsed if a 
completion is needed since the symbols within are locally scoped and 
don't affect other tags.

On a side note, I've had a lot of complaints that completion was too 
pedantic, and if it can't find a pedantic answer, perhaps it could let 
loose a little and find a vaguely close answer.

>>
>>> In summary; Semantic can extract useful data out of pretty badly 
>>> trashed
>>> code, and track your changes while you edit in a robust way.
>>
>> It seems I stand corrected, thanks. However, it would be interesting 
>> to do a comparison on some real file, against a widely used compiler, 
>> such as Clang.
>>

That would be interesting.  What would the goal be?  There are lots of 
things Semantic doesn't do, nor was intended for.   Like many Emacs 
tools, the Semantic parser is a bit liberal in the assumptions it makes 
to get a good enough answer.

Eric

Re: progmodes/project.el and search paths

[Eli Zaretskii]

> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Sat, 15 Aug 2015 01:30:41 +0300
> Cc: Stephen Leake <stephen_leake@stephe-leake.org>, emacs-devel@gnu.org
> 
> > 3) Iterative parsing, so each invocation of the parser only executes
> > across a single tag.
> >
> > This means that if parsing fails, we just move forward one lexical token
> > and try again.  This lets the parser just wash over anything it doesn't
> > get.
> 
> AFAIK, parsers in general try to ignore errors, or else you wouldn't be 
> able to get any useful information, after compilation fails, about any 
> errors after the first one.

Compiler parsers indeed do that, but AFAIK not at lexical token
granularity.  I think they throw away much larger portions of the
parse tree before trying to resync.  IME, when a compiler reports an
error, it also frequently reports additional bogus errors about
unrelated symbols.

Re: progmodes/project.el and search paths

[Dmitry Gutov]

Hi Eric,

On 08/05/2015 04:29 AM, Eric Ludlam wrote:

> The attached is my experience of things I either needed in EDE's base
> classes to implement a project, or that some tool needed, and what I
> learned along the way.  You might think it is too EDE specific, but I
> put in only the most basic stuff, and put in things I kind of wish I had
> time to add to EDE.

I've put a list of TODO items into project.el. It's less ambitious than 
your list, but hopefully it covers the essentials that we (together with 
the community) will be able to consistently implement across the board.

I have omitted certain useful things like "jump from 'include' statement 
in C++ to the file" and "navigating between code and doc" because they 
seem to require separate (though related) facilities that we're yet to 
design.