Project systems (again)

Project systems (again)

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 6256 bytes --]

I'd like to include a good, minimal "project system" in Emacs 24.5 and
turn it on by default.

A "project system" is a set of commands and infrastructure functions for
finding and operating on groups of related files. Some features we want
to provide are running compile and shell commands on the project root
(as opposed to the directory in which a particular file lives), using
completion of easily locate a file in a project, and performing vc
operations on the project as a whole.

We should ship a minimal core and leave tasks like aggressive caching,
build system generation, customize-style project changes, source
indexing, and so on to other packages. That said, the core should
provide a way for different modules that provide "project" functionality
to talk to each other. Users should be able to easily customize how the
system works, or turn it off completely.

Implementation-wise, each project should be a lisp object we store in a
`project' buffer-local variable. When we first visit a file, we'll run a
hook to find the right project object, then store it in `project'.  (A
bit like `vc-handled-backends').

Each project object could just be a simple lisp closure that accepts the
function itself, a request symbol and a variable number of arguments. If
a project function doesn't know how to handle a particular request, it
should forward all its arguments to another project function or to a
default project function (say, `project-perform-default') that we
provide. This way, we allow simple kinds of project reuse and support
ad-hoc extensibility without the heft and conceptual complexity of an
object system.

If we visit two files in the same project, it might be useful for both
buffers to have the same value of `project'. We can make that work, but
when the last buffer that refers to a particular project is killed, we
should forget the project object entirely. Accumulating state as Emacs
opens and closes many projects makes it hard to reason about the system.

At the infrastructure level, project objects provide services. The two
services that interest me are requesting paths of a particular kind in
relation to a particular file and requesting a single path in relation
to a particular file.

The first might be something like this: project-paths PROJECT &key FILE
KIND. `project-paths' would return a completion table. We'd define some
well-known KIND values --- e.g., 'source-files, 'all-files,
'include-paths --- and let individual project implementations figure out
how to provide this service. For example, semantic might want to know
all the C include paths for a file, so it would call `project-paths'
with FILE being the name of the file it's about to parse and KIND being
'c-include-paths. An automake project could inspect generated makefiles
to determine the correct include paths for a given file. If a project
doesn't understand a KIND, it should just pass the request to its parent
project or to `project-perform-default'.

In stock project types, we should provide 'all-files and 'source-files
KIND values, making a good guess for the latter. Most of the time, we
can use vc to very quickly find the list of files --- just like
projectile does.

We should also provide something like this: project-path PROJECT &key
FILE KIND. This function should return a single path, or signal if we
can't find that path. This facility makes it easy to build "just give me
the damn project root" functionality, sure, but it also lets us provide
other kinds of relevant paths. (Imagine KIND being 'generated-file, say.)

Projects should also provide a user-accessible general-purpose plist,
since these things come in handy in all sorts of situations.

We also need to provide a good user interface. We should clone a subset
of Projectile's interface --- except that we'd want to use a
non-user-reserved prefix (say, C-x ,) instead of C-c p. In particular,
we need commands for finding a file in a project (with completion --- so
if you were working in Emacs, you'd be able to just type "lisp.h"),
running commands in the project root, running find-dired, and so on.

We should avoid aggressive caching in the project system. Modern systems
are very fast and we shouldn't complicate the implementation (and
debugging) for the sake of corner cases.

Performance should be fine. In the common case, the runtime overhead
will be a few additional calls to locate-dominating-file --- beyond
these calls, you only pay for what you use. The most expensive operation
is providing a completion table of all the files in a project: vc should
be able to make this operation very fast in practice, since vc-backed
projects are very common these days. We can also refer to GNU global,
cscope, and so on files as available.

EDE already provides some of the functionality I've described above, but
I don't like the way it does it. It's very complicated and embeds
uncommon features into the core. EDE's reliance on EIEIO and its large
feature-set create difficulties dumping the system with Emacs. And have
you tried actually creating a new EDE project type? It's surprisingly
and disappointingly difficult. Ad-hoc extensibility with EDE is hard
because of its EIEIO use, and I don't think EIEIO is buying us anything,
really.

EDE also includes many features that are not generally useful (like
makefile generation) and that complicate the codebase. EDE being
developed out-of-tree makes it hard to fix these issues. I'd rather have
a minimal and more idiomatically Emacs-ish core facility that EDE can
build on. EDE is also not currently integrated with VC, and it doesn't
provide nice user interfaces (ede-find-file, for example, doesn't do
completion.)

Projectile (https://github.com/bbatsov/projectile) is another popular
project system; its design is a bit closer to what I describe above.
It's not included in Emacs though, and it's not factored into an
extensible core like I described above. (In particular, it's written in
terms of project root files rather than abstract project-finding
functions in a function list.) I also don't know whether the author
would be interested in contributing the code to Emacs.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 901 bytes --]

Re: Project systems (again)

[Eli Zaretskii]

> Date: Thu, 17 Apr 2014 14:52:44 -0700
> From: Daniel Colascione <dancol@dancol.org>
> 
> I'd like to include a good, minimal "project system" in Emacs 24.5 and
> turn it on by default.

IMO, this would be a very good progress, thanks.

> EDE already provides some of the functionality I've described above, but
> I don't like the way it does it. It's very complicated and embeds
> uncommon features into the core. EDE's reliance on EIEIO and its large
> feature-set create difficulties dumping the system with Emacs. And have
> you tried actually creating a new EDE project type? It's surprisingly
> and disappointingly difficult. Ad-hoc extensibility with EDE is hard
> because of its EIEIO use, and I don't think EIEIO is buying us anything,
> really.
> 
> EDE also includes many features that are not generally useful (like
> makefile generation) and that complicate the codebase. EDE being
> developed out-of-tree makes it hard to fix these issues. I'd rather have
> a minimal and more idiomatically Emacs-ish core facility that EDE can
> build on. EDE is also not currently integrated with VC, and it doesn't
> provide nice user interfaces (ede-find-file, for example, doesn't do
> completion.)

FWIW, I'd prefer that you work with EDE developers to improve and
extend what they have; starting from scratch (or almost from scratch)
sounds like waste of effort, especially since some of the EDE is
already in Emacs.  I find the Makefile generation feature useful
(e.g., when you need to develop on one platform, then build the same
code and run it on another, where the full development environment is
not necessarily installed).  In any case, I don't see how unused
features could get in your way too much, unless their design is wrong.

Re: Project systems (again)

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 3284 bytes --]

On 04/17/2014 11:37 PM, Eli Zaretskii wrote:
>> Date: Thu, 17 Apr 2014 14:52:44 -0700
>> From: Daniel Colascione <dancol@dancol.org>
>>
>> I'd like to include a good, minimal "project system" in Emacs 24.5 and
>> turn it on by default.
> 
> IMO, this would be a very good progress, thanks.
> 
>> EDE already provides some of the functionality I've described above, but
>> I don't like the way it does it. It's very complicated and embeds
>> uncommon features into the core. EDE's reliance on EIEIO and its large
>> feature-set create difficulties dumping the system with Emacs. And have
>> you tried actually creating a new EDE project type? It's surprisingly
>> and disappointingly difficult. Ad-hoc extensibility with EDE is hard
>> because of its EIEIO use, and I don't think EIEIO is buying us anything,
>> really.
>>
>> EDE also includes many features that are not generally useful (like
>> makefile generation) and that complicate the codebase. EDE being
>> developed out-of-tree makes it hard to fix these issues. I'd rather have
>> a minimal and more idiomatically Emacs-ish core facility that EDE can
>> build on. EDE is also not currently integrated with VC, and it doesn't
>> provide nice user interfaces (ede-find-file, for example, doesn't do
>> completion.)
> 
> FWIW, I'd prefer that you work with EDE developers to improve and
> extend what they have;

If EIEIO can't be preloaded (or, equivalent morally, autoloaded on
find-file), there's no point in pursuing EDE improvements. An EIEIO-less
EDE would be an EDE rewrite anyway. Plus, I don't think the problem
space really warrants a complex object system: conventional elisp idioms
are adequate.

> starting from scratch (or almost from scratch)
> sounds like waste of effort, especially since some of the EDE is
> already in Emacs. 

I really don't want to start from scratch, but I think it's the best
option. A project system is one of those systems for which the hard part
isn't the coding, but agreeing on having a single interface to the code.
I think we need something much simpler than what exists.

> I find the Makefile generation feature useful
> (e.g., when you need to develop on one platform, then build the same
> code and run it on another, where the full development environment is
> not necessarily installed).

There's no reason that a system like that couldn't be built on top of a
minimal project architecture, or why you couldn't use EDE's current
implementation. (I hope it's easy to see how EDE could plug into a more
generic system.)

>  In any case, I don't see how unused
> features could get in your way too much, unless their design is wrong.

I find the abstractions in EDE to be much more confusing than they are
useful. For something that, at its core, ought to be very simple, there
are too many concepts --- target, project, sub-project, config, project
placeholder, too much shared state, and too few opportunities for ad-hoc
customization. The system feels specialized for a project based on
nested autoconf files that build C and C++, and the documentation
reflects that. I understand that EDE started simple and grew
functionality, but this functionality belongs in separate layers, not
mingled into the core.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 884 bytes --]

Re: Project systems (again)

[Eli Zaretskii]

> Date: Fri, 18 Apr 2014 00:07:20 -0700
> From: Daniel Colascione <dancol@dancol.org>
> CC: emacs-devel@gnu.org
> 
> > FWIW, I'd prefer that you work with EDE developers to improve and
> > extend what they have;
> 
> If EIEIO can't be preloaded (or, equivalent morally, autoloaded on
> find-file), there's no point in pursuing EDE improvements. An EIEIO-less
> EDE would be an EDE rewrite anyway.

I don't know enough about this: why couldn't EIEIO be autoloaded?

> Plus, I don't think the problem space really warrants a complex
> object system: conventional elisp idioms are adequate.

Since EDE is already there, I think this is a moot point.  (I believe
Eric explained why this design was chosen a while ago.)

> > starting from scratch (or almost from scratch)
> > sounds like waste of effort, especially since some of the EDE is
> > already in Emacs. 
> 
> I really don't want to start from scratch, but I think it's the best
> option. A project system is one of those systems for which the hard part
> isn't the coding, but agreeing on having a single interface to the code.
> I think we need something much simpler than what exists.

Then how about asking the EDE developers to provide an "easy-ede"
layer which would conceal the complexity for those situations where
the corresponding power is not needed?

> I find the abstractions in EDE to be much more confusing than they are
> useful. For something that, at its core, ought to be very simple, there
> are too many concepts --- target, project, sub-project, config, project
> placeholder, too much shared state, and too few opportunities for ad-hoc
> customization. The system feels specialized for a project based on
> nested autoconf files that build C and C++, and the documentation
> reflects that. I understand that EDE started simple and grew
> functionality, but this functionality belongs in separate layers, not
> mingled into the core.

I see your point.  However, EDE was added to Emacs with the intent
that it serves as basis for developing features such as what you have
in mind.  If there's a reasonably practical way of basing your project
on EDE, I think we should explore that possibility first, even if it
requires more work on the EDE infrastructure side, because not doing
so would waste the effort of integrating EDE into Emacs.

Re: Project systems (again)

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 3260 bytes --]

On 04/18/2014 12:50 AM, Eli Zaretskii wrote:
>> Date: Fri, 18 Apr 2014 00:07:20 -0700
>> From: Daniel Colascione <dancol@dancol.org>
>> CC: emacs-devel@gnu.org
>>
>>> FWIW, I'd prefer that you work with EDE developers to improve and
>>> extend what they have;
>>
>> If EIEIO can't be preloaded (or, equivalent morally, autoloaded on
>> find-file), there's no point in pursuing EDE improvements. An EIEIO-less
>> EDE would be an EDE rewrite anyway.
> 
> I don't know enough about this: why couldn't EIEIO be autoloaded?

Stefan was against it the last time this issue came up. There are
namespace cleanliness issues as well as deeper issues of functional
appropriateness.

>> Plus, I don't think the problem space really warrants a complex
>> object system: conventional elisp idioms are adequate.
> 
> Since EDE is already there, I think this is a moot point.  (I believe
> Eric explained why this design was chosen a while ago.)

It's there, but it hasn't seen wide use. The existence of non-EDE
projects like Projectile should tell us that EDE, as it stands today,
isn't well-suited to being a default project system.

>>> starting from scratch (or almost from scratch)
>>> sounds like waste of effort, especially since some of the EDE is
>>> already in Emacs. 
>>
>> I really don't want to start from scratch, but I think it's the best
>> option. A project system is one of those systems for which the hard part
>> isn't the coding, but agreeing on having a single interface to the code.
>> I think we need something much simpler than what exists.
> 
> Then how about asking the EDE developers to provide an "easy-ede"
> layer which would conceal the complexity for those situations where
> the corresponding power is not needed?

That's what I'm proposing, except that I imagine EDE can sit on top of
that layer, not that that layer could sit on top of EDE.

>> I find the abstractions in EDE to be much more confusing than they are
>> useful. For something that, at its core, ought to be very simple, there
>> are too many concepts --- target, project, sub-project, config, project
>> placeholder, too much shared state, and too few opportunities for ad-hoc
>> customization. The system feels specialized for a project based on
>> nested autoconf files that build C and C++, and the documentation
>> reflects that. I understand that EDE started simple and grew
>> functionality, but this functionality belongs in separate layers, not
>> mingled into the core.
> 
> I see your point.  However, EDE was added to Emacs with the intent
> that it serves as basis for developing features such as what you have
> in mind.  If there's a reasonably practical way of basing your project
> on EDE, I think we should explore that possibility first, even if it
> requires more work on the EDE infrastructure side, because not doing
> so would waste the effort of integrating EDE into Emacs.

I was afraid of an argument of this form --- it's just the sunken costs
fallacy, isn't it? The fact that effort has been put into EDE shouldn't
influence our evaluations of present alternatives. I don't think using
EDE as a base would reduce the amount of needed work. I actually think
it would increase it.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 884 bytes --]

Re: Project systems (again)

[Eli Zaretskii]

> Date: Fri, 18 Apr 2014 00:58:54 -0700
> From: Daniel Colascione <dancol@dancol.org>
> CC: emacs-devel@gnu.org
> 
> > I see your point.  However, EDE was added to Emacs with the intent
> > that it serves as basis for developing features such as what you have
> > in mind.  If there's a reasonably practical way of basing your project
> > on EDE, I think we should explore that possibility first, even if it
> > requires more work on the EDE infrastructure side, because not doing
> > so would waste the effort of integrating EDE into Emacs.
> 
> I was afraid of an argument of this form --- it's just the sunken costs
> fallacy, isn't it?

It is only a fallacy if the decision to include EDE was mistaken.

> The fact that effort has been put into EDE shouldn't influence our
> evaluations of present alternatives.

Indeed, it shouldn't.  But EDE is one such alternative, and IMO we
should give some weight to past decisions, because AFAIK they are
generally taken after some careful consideration.  IOW, we should not
discard them without discussing and re-evaluating them.

> I don't think using EDE as a base would reduce the amount of needed
> work. I actually think it would increase it.

If you are right that using EDE increases the amount of work, then I
think we should remove EDE from Emacs, because it would be just
ballast then.  But I suggest to hear from the EDE developers, and
hopefully also from Chong (CC'ed), who AFAIR worked on including EDE,
before we reach such conclusions.

Re: Project systems (again)

[Dmitry Gutov]

Eli Zaretskii <eliz@gnu.org> writes:

> FWIW, I'd prefer that you work with EDE developers to improve and
> extend what they have; starting from scratch (or almost from scratch)
> sounds like waste of effort, especially since some of the EDE is
> already in Emacs.

I disagree: this wouldn't be starting from scratch.

Projectile is one such existing system that people use, and a generic
interface would allow for better integration with other packages, even
those that choose to be project-management-system-agnostic, which is
generally a good choice.

Personally, I use Projectile already, and switching to EDE wouldn't be
an improvement for my use. As a bonus, its primary developer (Bozhidar
Batsov) already has copyright assignment on file, so it wouldn't be
inconceivable to add it to GNU ELPA soon or eventually. But of course,
there will be other contributors to take care of.

Re: Project systems (again)

[Stefan Monnier]

>> FWIW, I'd prefer that you work with EDE developers to improve and
>> extend what they have;

It's important to take EDE into account, indeed.  But I think what we
need at this point is a core feature that will probably be used all the
time (i.e. needs to be preloaded because it's called even when the user
doesn't request any specific action, kind of like vc-hooks.el).

So one way to look at it, is to try and extract some core part of EDE
that can be moved into the core.  But another is to design a new system,
making sure that EDE can be "retargetted" on top of that new system.

Either way is fine by me.

It's perfectly fine to autoload EIEIO when the user requests
a project-related command, but preloading it is not an option right now
(it may be an option in the future, but that would first mean using
cl-lib instead of cl, and obviously it would mean preloading cl-lib).


        Stefan

Re: Project systems (again)

[Alex Ott]

Hello Daniel

Daniel Colascione  at "Fri, 18 Apr 2014 00:07:20 -0700" wrote:
 DC> On 04/17/2014 11:37 PM, Eli Zaretskii wrote:
 >>  In any case, I don't see how unused
 >> features could get in your way too much, unless their design is wrong.

 DC> I find the abstractions in EDE to be much more confusing than they are
 DC> useful. For something that, at its core, ought to be very simple, there
 DC> are too many concepts --- target, project, sub-project, config, project
 DC> placeholder, too much shared state, and too few opportunities for ad-hoc
 DC> customization. The system feels specialized for a project based on
 DC> nested autoconf files that build C and C++, and the documentation
 DC> reflects that. I understand that EDE started simple and grew
 DC> functionality, but this functionality belongs in separate layers, not
 DC> mingled into the core.

I did some development to support simple projects mostly for projects with one central project file in the top of the project (I call it single-root projects) - there is a primitive support for Maven, Leiningen, Ant, Rebar there.

If you interested, you can look to the code at https://github.com/alexott/cedet/tree/devel/lisp/cedet/ede

I plan to merge it into CEDET's trunk when the more functionality will be implemented

-- 
With best wishes, Alex Ott
http://alexott.blogspot.com/        http://alexott.net/
http://alexott-ru.blogspot.com/
Skype: alex.ott

Re: Project systems (again)

[Eric M. Ludlam]

On 04/18/2014 03:58 AM, Daniel Colascione wrote:
>> >  Then how about asking the EDE developers to provide an "easy-ede"
>> >  layer which would conceal the complexity for those situations where
>> >  the corresponding power is not needed?
> That's what I'm proposing, except that I imagine EDE can sit on top of
> that layer, not that that layer could sit on top of EDE.

This thread has been pretty high level which I think has made the 
conversation less productive.  Daniel was asking detailed questions on 
the CEDET mailing list earlier and raised some good points there, which 
are related to his above statement.  From what I remember, the core 
issue is this;

EDE started out as a wrapper for automake, and detected those projects 
where all the subdirs of a project had a Makefile.am.  At least the 
projects I was using all did.

This resulted in a project detection system that didn't support projects 
where the key-file that was searched for (ie - Makefile.am) only showed 
up in one place, such as an .ant or Manifest file.

When support for projects of that nature were added, what I quickly 
found was that walking up the directory tree searching for them really 
hammered the auto-mounter on networked file systems.  What is there now 
is a bit of a hybrid that avoids walking up the directory tree.

The next issue was related to performance.  I spent a lot of time 
profiling and creating caches to make it fast enough to be used as the 
back-end for Semantic's auto-completion.  Wrangling large tables of tags 
organized by project required a lot from EDE's file management.

The result is something that is a bit challenging to extend if you don't 
want to follow one of the existing patterns, or if you try to add in the 
missing feature of detecting a project with a single root key-file from 
a sub-directory.

I would be interested in ideas on how to better support projects with a 
single root key-file.  If there were a simple service that only detected 
the root of a tree based on a key-file, or any other useful and similar 
mechanism identifying a project root, it would be pretty easy to update 
EDE to use it as one of it's mechanisms.  EDE is not dependent on any 
single way to do it.  If said system were very fast, then I could 
probably drop some of the cache-heavy and hard to debug code in EDE.

Also, the EDE project types (ie - Automake, Arduino, Android, etc) are 
independent from the detection system.  Once a project is detected, then 
the high-level project is created to represent it and operates 
independently.

Eric

Re: Project systems (again)

[Bozhidar Batsov]

[-- Attachment #1: Type: text/plain, Size: 1726 bytes --]

On 18 April 2014 17:03, Dmitry Gutov <dgutov@yandex.ru> wrote:

> Eli Zaretskii <eliz@gnu.org> writes:
>
> > FWIW, I'd prefer that you work with EDE developers to improve and
> > extend what they have; starting from scratch (or almost from scratch)
> > sounds like waste of effort, especially since some of the EDE is
> > already in Emacs.
>
> I disagree: this wouldn't be starting from scratch.
>
> Projectile is one such existing system that people use, and a generic
> interface would allow for better integration with other packages, even
> those that choose to be project-management-system-agnostic, which is
> generally a good choice.
>
> Personally, I use Projectile already, and switching to EDE wouldn't be
> an improvement for my use. As a bonus, its primary developer (Bozhidar
> Batsov) already has copyright assignment on file, so it wouldn't be
> inconceivable to add it to GNU ELPA soon or eventually. But of course,
> there will be other contributors to take care of.
>
>
Btw, Projectile was the reason I signed the copyright assignment in the
first place. Stefan approached me a few years back
about including it into ELPA, but for some reason we never actually got to
doing this.

Projectile is not without its quirks, but I'm fairly certain its one of the
best project management options around. What's most important, however, is
that it's pretty well battle tested - thousands of users have been using it
over the last 3 years and have submitted numerous bug reports, feature
requests and patches. Starting for scratch would reset the counter on all
(most) of that. I'm certainly biased, but I think focusing more effort on
improving Projectile makes more sense than implementing an alternative
solution.

[-- Attachment #2: Type: text/html, Size: 2253 bytes --]

Re: Project systems (again)

[Stefan Monnier]

> When support for projects of that nature were added, what I quickly found
> was that walking up the directory tree searching for them really hammered
> the auto-mounter on networked file systems.

Currently, we walk up the tree several times per find-file for VC's
backend detection.  So this is a "solved" problem (use
locate-dominating-file, which should really be named
locate-dominating-dir, but it's not worth the trouble renaming it).


        Stefan

Re: Project systems (again)

[Stefan Monnier]

> Btw, Projectile was the reason I signed the copyright assignment in the
> first place.  Stefan approached me a few years back
> about including it into ELPA, but for some reason we never actually got to
> doing this.

Indeed, I'd still be very happy to see it included in GNU ELPA.


        Stefan

Re: Project systems (again)

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 1865 bytes --]

On 04/19/2014 01:55 AM, Bozhidar Batsov wrote:
> Btw, Projectile was the reason I signed the copyright assignment in the
> first place. Stefan approached me a few years back
> about including it into ELPA, but for some reason we never actually got
> to doing this.  

ELPA isn't adequate. I want something in the Emacs core and enabled by
default.

> Projectile is not without its quirks, but I'm fairly certain its one of
> the best project management options around.

Agreed, but that doesn't make it suitable as-is.

> What's most important,
> however, is that it's pretty well battle tested - thousands of users
> have been using it over the last 3 years and have submitted numerous bug
> reports, feature requests and patches. Starting for scratch would reset
> the counter on all (most) of that.

There's no reason we can't learn from all the effort, just as we can
learn from what the EDE people have done.

> I'm certainly biased, but I think
> focusing more effort on improving Projectile makes more sense than
> implementing an alternative solution.

Being "battle tested" is overrated: DOS was "battle tested" too.
Projectile is a good package. That said, it's only about 2,000 lines long.

The hard part about a project system isn't locating and manipulating
packages, but integrating the system into external tools. Right now,
Projectile is a monolith and doesn't have any internal abstractions of
layering or extensibility. (See `projectile-project-vcs'.) It's not
possible to use it as a base for EDE, for example, nor to extend it with
the ability to answer arbitrary questions about files in the project
(which is what you'd need for something like semantic's UI.)

Projectile has some good ideas, particularly with respect to the UI, but
the needed backend modifications would amount to a rewrite anyway.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 884 bytes --]

Re: Project systems (again)

[Eric M. Ludlam]

[-- Attachment #1: Type: text/plain, Size: 1582 bytes --]

On 04/19/2014 10:26 AM, Stefan Monnier wrote:
>> When support for projects of that nature were added, what I quickly found
>> was that walking up the directory tree searching for them really hammered
>> the auto-mounter on networked file systems.
>
> Currently, we walk up the tree several times per find-file for VC's
> backend detection.  So this is a "solved" problem (use
> locate-dominating-file, which should really be named
> locate-dominating-dir, but it's not worth the trouble renaming it).

Thanks Stefan,

I'd only heard of this function recently, and was unsure if it was 
appropriate for EDE.   In particular, I see in reading the code that it 
tries to avoid searching for files around root, above home directories, 
and matching a particular regexp, so that is good.

I tried some timing tests with it in my CEDET repository (4 deep 
directory) looping 100 times to see how it competes.

Detect detect-with-ldf took 0.0166 seconds
Detect detect-with-ldf-ede-simple took 0.0442 seconds
Detect detect-with-ldf-ede-full took 0.2112 seconds

Detect detect-with-ede took 0.0000 seconds
Detect detect-with-ede-file-detect took 0.0016 seconds
Detect detect-with-ede-no-buffer-cache took 0.1612 seconds
Detect detect-with-ede-buffer-init-hook took 0.2805 seconds

See attached code for details.

It did pretty well considering the added feature.

There are some strange hacks in the EDE autoloader due to not navigating 
up a directory tree.  I'll see if I can take advantage of this function 
and simplify & enhance some of EDE's detection: the last 2 above timings.

Eric

[-- Attachment #2: tpdm.el --]
[-- Type: text/x-emacs-lisp, Size: 3188 bytes --]

;; Test speeds of different project detect machanisms.

(require 'ede)

(defun find-ede-proj-with-ldf-predicate (ldf-file)
  "Predicate for finding EDE projects with `locate-dominating-file'.
Argument LDF-FILE is the current directory file from LDF."
  (let ((types ede-project-class-files)
	(ret nil))
    ;; Loop over all types, loading in the first type that we find.
    (while (and types (not ret))
      (if (ede-dir-to-projectfile-simple (car types) ldf-file)
	  (progn
	    ;; We found one!  Require it now since we will need it.
	    ;;(require (oref (car types) file))
	    (setq ret (car types)))
	(setq types (cdr types))))
    ret))

(defmethod ede-dir-to-projectfile-simple ((this ede-project-autoload) dir)
  "Simplified version of `ede-dir-to-projectfile'."
    (let* ((d (file-name-as-directory dir))
	   (pf (oref this proj-file))
	   (f (when (stringp pf) (expand-file-name pf (or root d))))
	   )
      (when (and f (file-exists-p f))
	f)))

(defun find-ede-proj-with-ldf-predicate-full (ldf-file)
  "Predicate for finding EDE projects with `locate-dominating-file'.
Argument LDF-FILE is the current directory file from LDF."
  (let ((types ede-project-class-files)
	(ret nil))
    ;; Loop over all types, loading in the first type that we find.
    (while (and types (not ret))
      (if (ede-dir-to-projectfile (car types) ldf-file)
	  (progn
	    ;; We found one!  Require it now since we will need it.
	    ;;(require (oref (car types) file))
	    (setq ret (car types)))
	(setq types (cdr types))))
    ret))

(defun detect-with-ldf ()
  "Detect using `locate-dominating-file'."
  (locate-dominating-file (buffer-file-name) "INSTALL"))

(defun detect-with-ldf-ede-simple ()
  "Detect using `locate-dominating-file' using a simplifed EDE predicate."
  (locate-dominating-file (buffer-file-name) 'find-ede-proj-with-ldf-predicate))

(defun detect-with-ldf-ede-full ()
  "Detect using `locate-dominating-file' using an EDE predicate."
  (locate-dominating-file (buffer-file-name) 'find-ede-proj-with-ldf-predicate-full))

(defun detect-with-ede ()
  "Detect using EDE's project detector."
  (ede-current-project))

(defun detect-with-ede-file-detect ()
  "Detect using EDE's file based project detector."
  (ede-directory-project-p default-directory))

(defun detect-with-ede-no-buffer-cache ()
  "Detect using EDE's file detect that uses no buffer cache info."
  (ede-directory-get-open-project default-directory 'ROOT))

(defun detect-with-ede-buffer-init-hook ()
  "Detect using EDE's buffer initialization hook."
  (ede-initialize-state-current-buffer))
  ;(ede-toplevel-project default-directory))

(defun detection-speed ()
  "Try out different detection schemes."
  (interactive)
  (dolist (DM '(detect-with-ldf
		detect-with-ldf-ede-simple
		detect-with-ldf-ede-full
		detect-with-ede
		detect-with-ede-file-detect
		detect-with-ede-no-buffer-cache
		detect-with-ede-buffer-init-hook
		))
    (let* ((start (current-time))
	   (index 100)
	   (out
	    (while (> index 0)
	      (funcall DM)
	      (setq index (1- index))))
	   (end (current-time)))
      (message "Detect %S took %.4f seconds"
	       DM
	       (float-time (time-subtract end start))))))