A unified project root interface

A unified project root interface

[Jorgen Schaefer]

Hello!

A growing number of extensions out there deal with "projects" as
opposed to "single files", that is, a collection of files that belong
together in some form or another. The usual way to define a project is
via the "project root", a directory so that all files under that
directory are considered part of "the project".

This is not exactly difficult to write, which has lead to the situation
where pretty much every extension that uses projects defines their own
code to find and set the project root. Examples of this are
find-file-in-project, projectile, elpy, flymake, ropemacs, nose, etc.
Now, if an user wants to use two or more of those extensions together,
she needs to "synchronize" the different ideas of project root manually.
Which gets more complicated the more extensions that define their own
project roots.

It would be really useful if there was a single standard way to define
the project root, so that extensions can just use that standard way
without each and every one of them writing the same code over and over
again. Sadly, as the code is not exactly difficult and so many projects
have already used their own way, it's highly unlikely that a random
third-party library would simply "emerge" as the standard way of doing
things. Which makes me believe the only way to solve this is via a bit
of a top-down decision to include a library in Emacs and declare it as
"the default".

So this is my somewhat verbose request to say "yes, let's do this" and
pick a library to provide the functionality. The library itself doesn't
have to be big at all, in theory a single variable `project-root'
that everyone is encouraged to use would be enough. Adding some
basic functionality to this would be helpful, though. If we keep the
functionality to a minimum, this lets other extensions use it without
being annoyed at the features they drag in.

Suitable libraries exist:

https://github.com/jorgenschaefer/project-el (my own proposal for this
problem, GPLed, happy to assign copyright / should already have.)

https://github.com/nex3/project-el (unsure about the status, it's on
marmalade, though.)

There are probably others.


Does this sound sensible?
Are there other possible approaches?
What's the next step to solve this problem?


Regards,
	-- Jorgen

Re: A unified project root interface

[Fabian Ezequiel Gallina]

2013/3/9 Jorgen Schaefer <forcer@forcix.cx>:
>
> It would be really useful if there was a single standard way to define
> the project root, so that extensions can just use that standard way
> without each and every one of them writing the same code over and over
> again. Sadly, as the code is not exactly difficult and so many projects
> have already used their own way, it's highly unlikely that a random
> third-party library would simply "emerge" as the standard way of doing
> things. Which makes me believe the only way to solve this is via a bit
> of a top-down decision to include a library in Emacs and declare it as
> "the default".
>

There's a standard way of defining variable values for files inside a
parent folder (read project). This mechanism is the Directory Local
variables [M-: (info "(emacs)Directory Variables")]. And it works, and
works good, in fact python.el makes use of this a lot and is able to
even to set the virtualenv for all your project files (without messing
with external others) thanks to it[0].

Normally I would consider the .dir-locals.el file location as the
project root, but then perhaps the user needs to override some values
in a sub-folder. So in that case I think it might be a good idea to
introduce an additional standard name for .dir-locals.el files that
are ought to be used only at the project root. I propose the name of
.project-locals.el. With this detecting if a file belongs to a project
and what's the project root is as easy as to find the
.project-locals.el.

>
> So this is my somewhat verbose request to say "yes, let's do this" and
> pick a library to provide the functionality. The library itself doesn't
> have to be big at all, in theory a single variable `project-root'
> that everyone is encouraged to use would be enough. Adding some
> basic functionality to this would be helpful, though. If we keep the
> functionality to a minimum, this lets other extensions use it without
> being annoyed at the features they drag in.
>

With something like the .project-locals.el, the only thing missing is
some UI for people fearing associations lists. I think, the best way
to go is a similar interface of what you get with customize-group
(hopefully most of its code could be borrowed).

[0] See Directory Local Variables section at: http://bit.ly/ZzUxxL



Regards,
Fabián

Re: A unified project root interface

[Stefan Monnier]

> So this is my somewhat verbose request to say "yes, let's do this" and
> pick a library to provide the functionality.

Agreed.  There are already different such things distributed with Emacs:
.dir-locals.el is one of them.  CEDET has its own notion.
As mentioned .dir-local.el often acts as a project root, but it can also
appear in sub-directories of a project, so it's not sufficient.
It should at least be augmented so that some .dir-locals.el can be
flagged as "non-root".

If the only need is to figure out the "root directory of the project",
then maybe a simple solution is:

   (defvar project-root-predicate
       (lambda () (file-exists-p ".dir-locals.el")))
   (defun project-root ()
     (locate-dominating-file default-directory project-root-predicate))

where packages can (add-function :until-before (default-value
project-root-predicate) <mypredicate>) to recognize other "tell tale
sign" of a project root (e.g. a Tupfile).

   
        Stefan

Re: A unified project root interface

[Jorgen Schaefer]

On Sun, 10 Mar 2013 00:38:12 -0500
Stefan Monnier <monnier@iro.umontreal.ca> wrote:

> > So this is my somewhat verbose request to say "yes, let's do this"
> > and pick a library to provide the functionality.
> 
> Agreed.
>
> [...]
>
> If the only need is to figure out the "root directory of the project",

That is I think the minimum requirement, yes.

> then maybe a simple solution is:
> 
>    (defvar project-root-predicate
>        (lambda () (file-exists-p ".dir-locals.el")))
>    (defun project-root ()
>      (locate-dominating-file default-directory
> project-root-predicate))
> 
> where packages can (add-function :until-before (default-value
> project-root-predicate) <mypredicate>) to recognize other "tell tale
> sign" of a project root (e.g. a Tupfile).

This would indeed be the minimal set to help with the problem. I think
it's a bit *too* minimal, because it fails for most common cases, i.e.
those where the project root is not defined yet. Every package trying
to use it will need to extend it with some way to define or set the
project root before this code can be used at all. The project.el files I
linked to (and most projects) by default look for VCS markers to find
the project root if nothing else is specified. This might be trivial
enough to add by default, too.

The only added functionality of the project.el files I linked to is
that they allow the user to override the project root (using M-x
project-set-root), which in turn makes it worthwhile to have the
project root as a buffer-local variable.

Regards,
	-- Jorgen

Re: A unified project root interface

[Lluís]

Stefan Monnier writes:

>> So this is my somewhat verbose request to say "yes, let's do this" and
>> pick a library to provide the functionality.

> Agreed.  There are already different such things distributed with Emacs:
> .dir-locals.el is one of them.  CEDET has its own notion.
> As mentioned .dir-local.el often acts as a project root, but it can also
> appear in sub-directories of a project, so it's not sufficient.
> It should at least be augmented so that some .dir-locals.el can be
> flagged as "non-root".

> If the only need is to figure out the "root directory of the project",
> then maybe a simple solution is:

>    (defvar project-root-predicate
>        (lambda () (file-exists-p ".dir-locals.el")))
>    (defun project-root ()
>      (locate-dominating-file default-directory project-root-predicate))

> where packages can (add-function :until-before (default-value
> project-root-predicate) <mypredicate>) to recognize other "tell tale
> sign" of a project root (e.g. a Tupfile).

CEDET provides this in two ways (as part of the EDE subsystem):

* If a Project.ede file exists, that's the root (similar to .dir-locals.el in
  this context)

* If signs of a "project-like" structure exist (e.g., (auto)makefiles, scons,
  java, etc), uses system-specific knowledge to automatically detect the project
  root.


I'm commenting this because, first, managing projects is the purpose of EDE
(although it tries to do more than just identifying their root) and it's
integrated in Emacs; and second, because the auto-detection could help in making
the process simpler and, in the best case, auto-magical.


Lluis

-- 
 "And it's much the same thing with knowledge, for whenever you learn
 something new, the whole world becomes that much richer."
 -- The Princess of Pure Reason, as told by Norton Juster in The Phantom
 Tollbooth

Re: A unified project root interface

[Eric M. Ludlam]

On 03/11/2013 02:57 PM, Lluís wrote:
> Stefan Monnier writes:
>
>>> So this is my somewhat verbose request to say "yes, let's do this" and
>>> pick a library to provide the functionality.
>
>> Agreed.  There are already different such things distributed with Emacs:
>> .dir-locals.el is one of them.  CEDET has its own notion.
>> As mentioned .dir-local.el often acts as a project root, but it can also
>> appear in sub-directories of a project, so it's not sufficient.
>> It should at least be augmented so that some .dir-locals.el can be
>> flagged as "non-root".
>
>> If the only need is to figure out the "root directory of the project",
>> then maybe a simple solution is:
>
>>     (defvar project-root-predicate
>>         (lambda () (file-exists-p ".dir-locals.el")))
>>     (defun project-root ()
>>       (locate-dominating-file default-directory project-root-predicate))
>
>> where packages can (add-function :until-before (default-value
>> project-root-predicate)<mypredicate>) to recognize other "tell tale
>> sign" of a project root (e.g. a Tupfile).
>
> CEDET provides this in two ways (as part of the EDE subsystem):
>
> * If a Project.ede file exists, that's the root (similar to .dir-locals.el in
>    this context)
>
> * If signs of a "project-like" structure exist (e.g., (auto)makefiles, scons,
>    java, etc), uses system-specific knowledge to automatically detect the project
>    root.
>
>
> I'm commenting this because, first, managing projects is the purpose of EDE
> (although it tries to do more than just identifying their root) and it's
> integrated in Emacs; and second, because the auto-detection could help in making
> the process simpler and, in the best case, auto-magical.

I, of course, agree with Lluis.  EDE is already setup to automatically 
find projects as was requested.  Adding new projects through the 
'generic' system is a pretty simple prospect.

I know some people think of EDE as being this big thing for creating 
your Makefiles, but its most common use really is to just tag the root 
of a project, and hang some configuration features for semantic auto 
completion.  Adding new detectable projects to EDE would be a win all 
around.

Eric

Re: A unified project root interface

[Jorgen Schaefer]

On Tue, 12 Mar 2013 19:28:28 -0400
"Eric M. Ludlam" <eric@siege-engine.com> wrote:

> I know some people think of EDE as being this big thing for creating 
> your Makefiles, but its most common use really is to just tag the
> root of a project, and hang some configuration features for semantic
> auto completion.  Adding new detectable projects to EDE would be a
> win all around.

Also extracting the project part so you can use it without depending on
all of EDE. Right now, that seems either not doable or badly
documented, as none of the project-based extensions I have seen actually
uses that functionality.

Regards,
	-- Jorgen

Re: A unified project root interface

[Eric M. Ludlam]

On 03/12/2013 07:42 PM, Jorgen Schaefer wrote:
> On Tue, 12 Mar 2013 19:28:28 -0400
> "Eric M. Ludlam"<eric@siege-engine.com>  wrote:
>
>> I know some people think of EDE as being this big thing for creating
>> your Makefiles, but its most common use really is to just tag the
>> root of a project, and hang some configuration features for semantic
>> auto completion.  Adding new detectable projects to EDE would be a
>> win all around.
>
> Also extracting the project part so you can use it without depending on
> all of EDE. Right now, that seems either not doable or badly
> documented, as none of the project-based extensions I have seen actually
> uses that functionality.
>
> Regards,
> 	-- Jorgen

It's not clear to me what you mean.  When a project is detected, it 
instantiates an instance of that project for that directory.  It doesn't 
load any other projects but the one it detected.  Thus, if it finds and 
Emacs project, it loads that project type which is pretty small.  It 
doesn't load the big Makefile generating project.

If you mean that "all of EDE" represents the core EDE project classes, 
then no, you can't use the project detection piece without that.  The 
project detection code first needs to look up a project.  If a project 
is already open, it will instead find it in the list of currently open 
projects without using the file system based search.

Eric

Re: A unified project root interface

[David Engster]

Eric M. Ludlam writes:
> On 03/11/2013 02:57 PM, Lluís wrote:
>> CEDET provides this in two ways (as part of the EDE subsystem):
>>
>> * If a Project.ede file exists, that's the root (similar to .dir-locals.el in
>>    this context)
>>
>> * If signs of a "project-like" structure exist (e.g., (auto)makefiles, scons,
>>    java, etc), uses system-specific knowledge to automatically
>> detect the project
>>    root.
>> I'm commenting this because, first, managing projects is the purpose of EDE
>> (although it tries to do more than just identifying their root) and it's
>> integrated in Emacs; and second, because the auto-detection could help in making
>> the process simpler and, in the best case, auto-magical.
>
> I, of course, agree with Lluis.  EDE is already setup to automatically
> find projects as was requested.  Adding new projects through the
> generic' system is a pretty simple prospect.

Yes, but we cannot envision all the kinds of projects people would want
and create those for them beforehand. And defining own projects with
ede-generic is simple, but not simple enough for end users. Let's take
the example for defining those simple projects in project-roots.el,
given by Sudish Joseph in this thread:

	("Perl Project"
         :root-contains-files ("t" "lib")
         :on-hit (lambda (p) (message (car p))))

You can create a project like this with ede-generic, but then you have
to write a little defclass inheriting from ede-generic, call
ede-generic-new-autoloader, and to actually *do* something when the
project is loaded you have to define methods like
`ede-generic-setup-configuration'. This is just too much boilerplate for
such a simple thing like the above. Also, even most Emacs developers are
not familiar with the CLOS-like syntax that's needed to define those
things.

There's no doubt that EDE can do all what's needed, but is has to be
wrapped in something that's easier to use, at least for simple stuff
like what project-roots.el does.

-David

Re: A unified project root interface

[Sudish Joseph]

David Engster <deng@randomsample.de> writes:
> You can create a project like this with ede-generic, but then you have
> to write a little defclass inheriting from ede-generic, call
> ede-generic-new-autoloader, and to actually *do* something when the
> project is loaded you have to define methods like
> `ede-generic-setup-configuration'. This is just too much boilerplate for
> such a simple thing like the above. Also, even most Emacs developers are
> not familiar with the CLOS-like syntax that's needed to define those
> things.
>
> There's no doubt that EDE can do all what's needed, but is has to be
> wrapped in something that's easier to use, at least for simple stuff
> like what project-roots.el does.

The simplest - and for me, only - use is to ask a buffer which project
it belongs to, if any.  Everything else in my usage can be derived from
that.  This is how vc works, too, I think, and ibuffer-vc can aggregate
buffers by vc-roots handily.

Buffers and buffer-locals provide a simple and emacsy form of extensible
object and attributes, so if there's more data associated with a project
it'd be nice to expose them as such as opposed or in addition to having
a dedicated project api.  This fits with .dir-locals as well.

Most project convenience functions I need turn out to be general
directory and file functions.  For e.g., "all files in current project"
could be as simple as
	(all-files-in project-root)
where the former is generic functionality and the latter is a buffer
local.

-Sudish

PS: Some more examples of how I find having a simple buffer-local
project-root handy - I use these every day:

;; project-details is a buffer-local of
;; (matching-rule-name . project-root-dir)
(defun sj/project-root-dir ()
  (when (project-root-fetch)
    (cdr project-details)))

;; this auto-restricts recursive ack searches to current project, where
;; project is defined dynamically - vc roots, etc.
(setq ack-mode-root-directory-function 'sj/project-root-dir)


;; A couple of anything.el sources that automatically use the current
;; buffer's project root

;; Files from current project root, if any
(defconst sj/anything-source-project-root-files
  '((name . "Project Files")
    (init . (lambda ()
	      (setq anything-project-root project-details)))
    (candidates . (lambda ()
		    (project-root-file-find-process anything-pattern)))
    (candidate-transformer . sj/anything-file-candidate-filter)
    (requires-pattern . 2)
    (volatile)
    (type . file)))

;; A source that's aware of the MVC structure of a Rails project.
(defconst sj/anything-rails-files
  '((name . "Rails")
    (init . (lambda ()
	      (setq anything-project-root project-details)))
    (candidates . (lambda ()
		    (when (equal "Rails Project" (car anything-project-root))
		      (rails-list-project-files (cdr anything-project-root)))))
    (type . file)))

Re: A unified project root interface

[Eric M. Ludlam]

On 03/13/2013 02:03 PM, David Engster wrote:
> Eric M. Ludlam writes:
>> On 03/11/2013 02:57 PM, Lluís wrote:
>>> CEDET provides this in two ways (as part of the EDE subsystem):
>>>
>>> * If a Project.ede file exists, that's the root (similar to .dir-locals.el in
>>>     this context)
>>>
>>> * If signs of a "project-like" structure exist (e.g., (auto)makefiles, scons,
>>>     java, etc), uses system-specific knowledge to automatically
>>> detect the project
>>>     root.
>>> I'm commenting this because, first, managing projects is the purpose of EDE
>>> (although it tries to do more than just identifying their root) and it's
>>> integrated in Emacs; and second, because the auto-detection could help in making
>>> the process simpler and, in the best case, auto-magical.
>>
>> I, of course, agree with Lluis.  EDE is already setup to automatically
>> find projects as was requested.  Adding new projects through the
>> generic' system is a pretty simple prospect.
>
> Yes, but we cannot envision all the kinds of projects people would want
> and create those for them beforehand. And defining own projects with
> ede-generic is simple, but not simple enough for end users. Let's take
> the example for defining those simple projects in project-roots.el,
> given by Sudish Joseph in this thread:
>
> 	("Perl Project"
>           :root-contains-files ("t" "lib")
>           :on-hit (lambda (p) (message (car p))))
>
> You can create a project like this with ede-generic, but then you have
> to write a little defclass inheriting from ede-generic, call
> ede-generic-new-autoloader, and to actually *do* something when the
> project is loaded you have to define methods like
> `ede-generic-setup-configuration'. This is just too much boilerplate for
> such a simple thing like the above. Also, even most Emacs developers are
> not familiar with the CLOS-like syntax that's needed to define those
> things.
>
> There's no doubt that EDE can do all what's needed, but is has to be
> wrapped in something that's easier to use, at least for simple stuff
> like what project-roots.el does.

Ok, I can buy that.  EDE was never simple.  I just know that if an 
"official" project mechanism is built, I'll have some work to do. ;)

In particular, if a system for detecting projects is developed, it would 
be great if EDE could stop detecting projects (ie - I delete some code), 
and instead use a hook to say "hey, let me know if you found a project", 
after which it can do its bit of magic for projects it also knows about.

Also be aware that I ran into a lot of performance issues I had to work 
through with EDE project detection.  Once other code starts asking where 
it is in a project, it's amazing how often that stuff gets called.  A 
project cache is a super handy thing, and was one of the key performance 
improvements I made back when i was trying to speed up smart completion.

Eric

Re: A unified project root interface

[David Engster]

Eric M. Ludlam writes:
> On 03/13/2013 02:03 PM, David Engster wrote:
>> There's no doubt that EDE can do all what's needed, but is has to be
>> wrapped in something that's easier to use, at least for simple stuff
>> like what project-roots.el does.
>
> Ok, I can buy that.  EDE was never simple.  I just know that if an
> "official" project mechanism is built, I'll have some work to do. ;)

I think we don't need another project mechanism in Emacs. EDE and
dir-locals should be enough. What we have to do however is to develop at
least one very basic project type in EDE which can be used similar to
project-root.el. I think what Sudish Joseph explained in this thread is
pretty typical:

- Recognizing projects based on some files or directories which are in
  the current directory or further up the tree.

- The ability to ask Emacs of what project the current buffer is part
  of.

- Automatic setting of variables based on the current project.

The user should be able to configure this in a variable, similar to how
project-root.el does it. For example:

("Perl Project"
   :root-contains ("t" "lib")
   :on-load 'some-function 
   :variable-settings '((tab-width . 4)
                        (fill-column . 79)))

This is easy to implement with EDE. I can work on this, but I'd like to
finish my five other things which are 90% finished first...

> Also be aware that I ran into a lot of performance issues I had to
> work through with EDE project detection.  Once other code starts
> asking where it is in a project, it's amazing how often that stuff
> gets called.  A project cache is a super handy thing, and was one of
> the key performance improvements I made back when i was trying to
> speed up smart completion.

Yes, I know. This is why I think EDE should be the framework on which
those things are built.

-David

Re: A unified project root interface

[Jorgen Schaefer]

On Sat, 16 Mar 2013 15:18:50 +0100
David Engster <deng@randomsample.de> wrote:

> I think we don't need another project mechanism in Emacs. EDE and
> dir-locals should be enough. What we have to do however is to develop
> at least one very basic project type in EDE which can be used similar
> to project-root.el.

Unless you create this in a way that is usable without requiring (or
understanding) EDE, I fear that it will not be used by many
other packages. And that's the issue I started this thread for.

The issue is not that we would not have a way to define a project. We
do. Dozens of them, actually. Every package has their own. It's so
trivial to do for 90% of all use cases that most packages just write
their own code.

If we do not provide a way that is simpler to use than "I'll just
quickly write my own", the original problem will not go away. No matter
how much we say "but we have a solution".

Regards,
	-- Jorgen

Re: A unified project root interface

[Phil Hagelberg]

Jorgen Schaefer writes:

> Unless you create this in a way that is usable without requiring (or
> understanding) EDE, I fear that it will not be used by many other
> packages. And that's the issue I started this thread for.

Speaking as the author of one of the most widely-used project-centric
libraries[1], I agree with Jorgen. I would be happy to adopt a mechanism
that ships with Emacs as long as it doesn't involve CLOS.

My personal vote is for a way to provide a list of project-defining
predicates, but a .dir-locals.el-based approach that didn't have false
positives for .dir-locals nested inside project subdirectories would be
fine too.

thanks,
Phil

[1] - https://github.com/technomancy/find-file-in-project/blob/master/find-file-in-project.el

Re: A unified project root interface

[David Engster]

Jorgen Schaefer writes:
> On Sat, 16 Mar 2013 15:18:50 +0100
> David Engster <deng@randomsample.de> wrote:
>
>> I think we don't need another project mechanism in Emacs. EDE and
>> dir-locals should be enough. What we have to do however is to develop
>> at least one very basic project type in EDE which can be used similar
>> to project-root.el.
>
> Unless you create this in a way that is usable without requiring (or
> understanding) EDE, I fear that it will not be used by many
> other packages. And that's the issue I started this thread for.
>
> The issue is not that we would not have a way to define a project. We
> do. Dozens of them, actually. Every package has their own. It's so
> trivial to do for 90% of all use cases that most packages just write
> their own code.

I think the main issue is that every "language community" has a
different view of what a "project" entails, so existing solutions are
often missing something crucial, or things that should be easy to do are
too cumbersome. It's the same reason why almost every new popular
programming language sooner or later comes along with its own build tool
(Rake, SCons, Maven, Ant, Leiningen, CMake, and so on). It's always
better to have something specifically aimed at the tool-chain you're
using. In my opinion, this is the reason why so many people develop
their own project definitions in Emacs: they want something with
precisely the functionality they need, nothing more or less.

EDE has developed into a framework for writing such specialized projects
in. If you want to *develop* such a specialized project type for your
tool-chain, you indeed need to understand the inner workings of EDE,
which mostly means to first learn EIEIO syntax, which I guess is what
most people don't like about it. However, if you just want to *query* a
project object for something, all you need to know is 'oref', and even
that could be wrapped in helper functions if needed.

It is entirely possible to write a project type so that it is very
simple to use for end users. I think our C/C++ project wrapper is a good
example for this, where you define a project like this:

   (ede-cpp-root-project "NAME"
         :file "~/myproject/Makefile"
         :include-path '( "/include" "../include" "/c/include" )
         :system-include-path '( "/usr/include/c++/3.2.2/" )
         :spp-table '( ("OS_GNU_LINUX" . "1") ))

I'd argue this is already similar to how projects are defined in
project-root.el. This is all a user has to use, and every file loaded
from that project will have a buffer local ede-object, from which you
can easily extract those attributes. This project type was deliberately
created for people who don't want to explicitly mess with targets,
linkers, compilers, object files, dependency generation, 'clean' rules,
and so on. My suggestion was to create something similar, but more
generic and not explicitly aimed at C/C++. We already have a 'generic'
project type, which IMO comes close.

-David

Re: A unified project root interface

[Jorgen Schaefer]

On Sat, 16 Mar 2013 23:59:39 +0100
David Engster <deng@randomsample.de> wrote:

> I think our C/C++ project wrapper is a
> good example for this, where you define a project like this:
> 
>    (ede-cpp-root-project "NAME"
>          :file "~/myproject/Makefile"
>          :include-path '( "/include" "../include" "/c/include" )
>          :system-include-path '( "/usr/include/c++/3.2.2/" )
>          :spp-table '( ("OS_GNU_LINUX" . "1") ))
> 
> I'd argue this is already similar to how projects are defined in
> project-root.el.

I have no idea what project-root.el is or why anyone brought it up. The
two project libraries I linked in the first post of this thread and
described therein do not require such cumbersome extra effort.

In the normal case, I do not want to define a project at all.

As an extension author, I just want to call (project-root) in my
extension and have it magically return a directory that's the root
directory of the project.

As a user, I just want to Have It Work[tm] when opening a file without
me defining anything. At most, if all possible heuristics fail, I can
live with being asked for the project root directory.

Everything more than this, and I'll just ignore that "feature" because
it's cumbersome to use.

Every further feature is "nice to have", and only useful if it does not
require more effort than the above.

And every requirement for "project" the above does not cover (like,
say, multiple project roots) is a fringe use I can happily live without,
as apparently can a bunch of other extension authors.

The problem I raised in this thread is that too many packages write
their own solution because using any of the possible standard solutions
is too cumbersome. Saying "there is a cumbersome standard solution"
does not solve the problem.

Regards,
	-- Jorgen

Re: A unified project root interface

[David Engster]

[-- Attachment #1: Type: text/plain, Size: 3143 bytes --]

Jorgen Schaefer writes:
> On Sat, 16 Mar 2013 23:59:39 +0100
> David Engster <deng@randomsample.de> wrote:
>
>> I think our C/C++ project wrapper is a
>> good example for this, where you define a project like this:
>> 
>>    (ede-cpp-root-project "NAME"
>>          :file "~/myproject/Makefile"
>>          :include-path '( "/include" "../include" "/c/include" )
>>          :system-include-path '( "/usr/include/c++/3.2.2/" )
>>          :spp-table '( ("OS_GNU_LINUX" . "1") ))
>> 
>> I'd argue this is already similar to how projects are defined in
>> project-root.el.
>
> I have no idea what project-root.el is or why anyone brought it up. The
> two project libraries I linked in the first post of this thread and
> described therein do not require such cumbersome extra effort.

This is just a matter of providing default project types, which are
activated at startup. You do not have to explicitly specify projects
like this. EDE already ships with such default projects, for example for
detecting the Emacs source, or the Linux kernel. It is very easy to do
something similar for version controlled files (see below), similar to
what your project.el does.

project-root.el is just a slightly more generic version than your
project.el. Most importantly, it allows to define different types of
projects, so that a package author can for example add some
functionality only for Perl projects, excluding all the others.

> In the normal case, I do not want to define a project at all.

Fair enough, but while this may be the "normal" case, the ability to
define own project types is not an exotic demand, IMO. I like how
project-root handles this.

> As an extension author, I just want to call (project-root) in my
> extension and have it magically return a directory that's the root
> directory of the project.

In EDE, you have a buffer-local variable `ede-object-root-project',
which holds the root project. I can be queried through methods, for
instance

(ede-project-root-directory ede-object-root-project)

will get you the root directory.

> The problem I raised in this thread is that too many packages write
> their own solution because using any of the possible standard solutions
> is too cumbersome. Saying "there is a cumbersome standard solution"
> does not solve the problem.

I do not say that.

To hopefully prove my point, I've hacked together a small EDE project
for detecting files under git/hg/bzr version control, very similar to
how you do it in your project.el (I've dropped the CVS/.svn detection
for the moment).

Just load Emacs with

emacs -Q -f global-ede-mode -l vcs-root.el

and load some file under a git/hg/bzr versioned directory. There is an
example function `my-get-project-root' to print the project's root.

While you might not like the CLOS-like syntax, I hope it can be seen
that this is not difficult and fairly short (most of it is just
boilerplate), and you directly profit from stuff like caching that EDE
does (as Eric has already mentioned in this thread).

Just to be clear: I do *not* expect that users write this stuff;
instead, Emacs should provide projects like this as defaults.

-David


[-- Attachment #2: vcs-root.el --]
[-- Type: application/emacs-lisp, Size: 1643 bytes --]

Re: A unified project root interface

[Jorgen Schaefer]

On Sun, 17 Mar 2013 18:40:13 +0100
David Engster <deng@randomsample.de> wrote:

> > The problem I raised in this thread is that too many packages write
> > their own solution because using any of the possible standard
> > solutions is too cumbersome. Saying "there is a cumbersome standard
> > solution" does not solve the problem.
> 
> I do not say that.

Great. I'm glad I misunderstood, and thank you for correcting me.

> To hopefully prove my point, I've hacked together a small EDE project
> for detecting files under git/hg/bzr version control, very similar to
> how you do it in your project.el (I've dropped the CVS/.svn detection
> for the moment).

> [...]

> and load some file under a git/hg/bzr versioned directory. There is an
> example function `my-get-project-root' to print the project's root.

Opening a file in a subdirectory of a repository root does not
associate it with a project. I suspect this is a bug in this "proof of
concept" implementation more so than a conceptual problem, though :-)

Thank you for providing an example implementation.

> Just to be clear: I do *not* expect that users write this stuff;
> instead, Emacs should provide projects like this as defaults.

If I understand this right, this looks good.

I'm a bit unsure about requiring EDE, CEDET, etc. for this - it's not
unlikely that people will go "meh" and not use it because of that. (I'm
sorry to say so, but the complexity of CEDET is a recurring theme on the
Emacs IRC channel.)

So in the end, what we need for trivial implementation:

- Provide a default "simple project" that auto-detects the root via
  VCS markers

- Define a (project-root) that simply returns (and
  ede-object-root-project (ede-project-root-directory
  ede-object-root-project))

- Define a (project-set-root DIR) that does (oset this :file DIR) for
  the current project. If there is no current project, this should
  create a "simple project" for that directory so other uses of
  (project-root) will find it.

- Ask authors of extensions to use (ede-minor-mode 1) in their mode
  function and simply use that function in their modes.

Does this sound sensible?

Regards,
	-- Jorgen

Re: A unified project root interface

[David Engster]

[-- Attachment #1: Type: text/plain, Size: 2939 bytes --]

Jorgen Schaefer writes:
> On Sun, 17 Mar 2013 18:40:13 +0100
> David Engster <deng@randomsample.de> wrote:
>> To hopefully prove my point, I've hacked together a small EDE project
>> for detecting files under git/hg/bzr version control, very similar to
>> how you do it in your project.el (I've dropped the CVS/.svn detection
>> for the moment).
>
>> [...]
>
>> and load some file under a git/hg/bzr versioned directory. There is an
>> example function `my-get-project-root' to print the project's root.
>
> Opening a file in a subdirectory of a repository root does not
> associate it with a project. I suspect this is a bug in this "proof of
> concept" implementation more so than a conceptual problem, though :-)

I've attached a new version which should hopefully work better.

> I'm a bit unsure about requiring EDE, CEDET, etc. for this - it's not
> unlikely that people will go "meh" and not use it because of that. (I'm
> sorry to say so, but the complexity of CEDET is a recurring theme on the
> Emacs IRC channel.)

I'd like to stress again that querying EDE for stuff like the current
root project directory does not require knowledge of EIEIO/CLOS; if you
find that it does, then please report this and we fix it.

Yes, functions like `ede-project-root-directory' are actually methods,
but why does it matter? Just call them like any other function. What
*is* a real problem at the moment is that `describe-function' does not
say anything meaningful about them, but I'm in the process of fixing
that. If it's still seen as too cumbersome after that, then OK, let's
wrap'em. :-)

I'm willing to code the necessary stuff on the EDE side of things; if it
turns out too complicated to use for package maintainers, I have no
problem throwing it away.

> So in the end, what we need for trivial implementation:
>
> - Provide a default "simple project" that auto-detects the root via
>   VCS markers
>
> - Define a (project-root) that simply returns (and
>   ede-object-root-project (ede-project-root-directory
>   ede-object-root-project))

That's for the maintainers to decide. I have a hunch they'd like to have
an ede- prefix...

> - Define a (project-set-root DIR) that does (oset this :file DIR) for
>   the current project. If there is no current project, this should
>   create a "simple project" for that directory so other uses of
>   (project-root) will find it.

This can be done. I will need a bit of time though, since I really need
to do another CEDET merge round with current trunk first. I think I'll
be able to come up with something in the coming weeks.

> - Ask authors of extensions to use (ede-minor-mode 1) in their mode
>   function and simply use that function in their modes.

Not sure if it's a good idea to enable EDE behind the user's back; I
think they should enable it in their init file if they want to have
project support. But IMO that's a detail; let's cross that bridge when
we get there.

-David


[-- Attachment #2: vcs-root.el --]
[-- Type: application/emacs-lisp, Size: 1881 bytes --]

Re: A unified project root interface

[John Yates]

On Mon, Mar 18, 2013 at 6:50 PM, David Engster <deng@randomsample.de> wrote:
>> - Define a (project-root) that simply returns (and
>>   ede-object-root-project (ede-project-root-directory
>>   ede-object-root-project))
>
> That's for the maintainers to decide. I have a hunch they'd like to have
> an ede- prefix...
...
>> - Ask authors of extensions to use (ede-minor-mode 1) in their mode
>>   function and simply use that function in their modes.
>
> Not sure if it's a good idea to enable EDE behind the user's back; I
> think they should enable it in their init file if they want to have
> project support.

I took the sense of the OP to be that the concept of a project-root
deserved to become part of the core set of emacs concepts unrelated to
optional packages e.g. much as file and directory local variables.
Once the concept gets pulled into the core extension author simply
assume existence of project-root functionality without needing to have
to enable in any particular way.

/john

Re: A unified project root interface

[David Engster]

John Yates writes:
> I took the sense of the OP to be that the concept of a project-root
> deserved to become part of the core set of emacs concepts unrelated to
> optional packages e.g. much as file and directory local variables.
> Once the concept gets pulled into the core extension author simply
> assume existence of project-root functionality without needing to have
> to enable in any particular way.

OK, if this is something which should be available with emacs -Q right
at startup, then EDE is out. It indeed is too big for that.

-David

Re: A unified project root interface

[Eric M. Ludlam]

On 03/19/2013 03:18 AM, David Engster wrote:
> John Yates writes:
>> I took the sense of the OP to be that the concept of a project-root
>> deserved to become part of the core set of emacs concepts unrelated to
>> optional packages e.g. much as file and directory local variables.
>> Once the concept gets pulled into the core extension author simply
>> assume existence of project-root functionality without needing to have
>> to enable in any particular way.
>
> OK, if this is something which should be available with emacs -Q right
> at startup, then EDE is out. It indeed is too big for that.

Hi David,

If you look at ede/base.el at the baseclass ede-project-placeholder, you 
will see bits from an old implementation of a project cache.  It used to 
save just the root directories between emacs sessions, and the 
placeholder was used in the project cache to speed up detection.  The 
goal was a "Recent Projects" menu item to load projects from.  I never 
got that far though.

It should be possible to revive that old bit of code and have a mode of 
EDE that only creates placeholders for detected projects, and only loads 
the project if the user asks, calls 'ede-compile', or some other 
explicit action.

In this scenario, I'm sure it would be possible to trim ede/base.el, and 
ede.el to be minimal, and provide just the root project detection.  CLOS 
would still be involved, but the result would be pretty light-weight.

I'd be happy to help figure that out if it would be useful.

Eric

Re: A unified project root interface

[Stefan Monnier]

>> I took the sense of the OP to be that the concept of a project-root
>> deserved to become part of the core set of emacs concepts unrelated to
>> optional packages e.g. much as file and directory local variables.
>> Once the concept gets pulled into the core extension author simply
>> assume existence of project-root functionality without needing to have
>> to enable in any particular way.
> OK, if this is something which should be available with emacs -Q right
> at startup, then EDE is out.  It indeed is too big for that.

It should be possible to write the code such that the project-root part
of EDE doesn't require loading too much code.


        Stefan

Re: A unified project root interface

[David Engster]

Stefan Monnier writes:
>>> I took the sense of the OP to be that the concept of a project-root
>>> deserved to become part of the core set of emacs concepts unrelated to
>>> optional packages e.g. much as file and directory local variables.
>>> Once the concept gets pulled into the core extension author simply
>>> assume existence of project-root functionality without needing to have
>>> to enable in any particular way.
>> OK, if this is something which should be available with emacs -Q right
>> at startup, then EDE is out.  It indeed is too big for that.
>
> It should be possible to write the code such that the project-root part
> of EDE doesn't require loading too much code.

Yes, but no matter how small, EDE depends on EIEIO, so this would mean
that EIEIO is loaded right at startup. I thought this would be
considered a showstopper, even when we clean up its namespace use.

-David

Re: A unified project root interface

[Stefan Monnier]

>>>> I took the sense of the OP to be that the concept of a project-root
>>>> deserved to become part of the core set of emacs concepts unrelated to
>>>> optional packages e.g. much as file and directory local variables.
>>>> Once the concept gets pulled into the core extension author simply
>>>> assume existence of project-root functionality without needing to have
>>>> to enable in any particular way.
>>> OK, if this is something which should be available with emacs -Q right
>>> at startup, then EDE is out.  It indeed is too big for that.
>> It should be possible to write the code such that the project-root part
>> of EDE doesn't require loading too much code.
> Yes, but no matter how small, EDE depends on EIEIO, so this would mean
> that EIEIO is loaded right at startup.  I thought this would be
> considered a showstopper, even when we clean up its namespace use.

Right, preloading EIEIO is not what I was planning to do, indeed.
IOW, I think the challenge is to extract the "project-root" part of EDE
in such a way that it doesn't require preloading as much code.
That might require splitting EIEIO, or some other approach, maybe,
I don't know.


        Stefan

Re: A unified project root interface

[Leo Liu]

On 2013-03-20 11:21 +0800, Stefan Monnier wrote:
> Right, preloading EIEIO is not what I was planning to do, indeed.
> IOW, I think the challenge is to extract the "project-root" part of EDE
> in such a way that it doesn't require preloading as much code.
> That might require splitting EIEIO, or some other approach, maybe,
> I don't know.

I personally would like a package not using any of pseudo-CLOS features
unless it is ABSOLUTELY necessary. There are projects with far larger
scale and beautifully built before EIEIO came along.

Leo

Re: A unified project root interface

[joakim]

Leo Liu <sdl.web@gmail.com> writes:

> On 2013-03-20 11:21 +0800, Stefan Monnier wrote:
>> Right, preloading EIEIO is not what I was planning to do, indeed.
>> IOW, I think the challenge is to extract the "project-root" part of EDE
>> in such a way that it doesn't require preloading as much code.
>> That might require splitting EIEIO, or some other approach, maybe,
>> I don't know.
>
> I personally would like a package not using any of pseudo-CLOS features
> unless it is ABSOLUTELY necessary. There are projects with far larger
> scale and beautifully built before EIEIO came along.

Just so another view is represented then:

I think EIEIO is pretty neat. The EDE feature is also prety neat. It has
most features you need in a project root fromework already. What it
lacks is ease of use.

If we rewrite EDE using some other method, it will eventually turn into
EDE if its going to cater for the same use-cases.

IMHO if one would like to work on reducing the footprint of Emacs, which
is a worthy goal, greater gains could be made implementing lazy loading
of C libraries on platforms where its possible.

> Leo
>

-- 
Joakim Verona

Re: A unified project root interface

[David Engster]

Leo Liu writes:
> On 2013-03-20 11:21 +0800, Stefan Monnier wrote:
>> Right, preloading EIEIO is not what I was planning to do, indeed.
>> IOW, I think the challenge is to extract the "project-root" part of EDE
>> in such a way that it doesn't require preloading as much code.
>> That might require splitting EIEIO, or some other approach, maybe,
>> I don't know.
>
> I personally would like a package not using any of pseudo-CLOS features
> unless it is ABSOLUTELY necessary. There are projects with far larger
> scale and beautifully built before EIEIO came along.

That's a straw man argument. No one here is claiming that EIEIO is
needed to build large scale packages. EDE just happens to use it.

-David

Re: A unified project root interface

[David Engster]

Stefan Monnier writes:
>>>>> I took the sense of the OP to be that the concept of a project-root
>>>>> deserved to become part of the core set of emacs concepts unrelated to
>>>>> optional packages e.g. much as file and directory local variables.
>
>>>>> Once the concept gets pulled into the core extension author simply
>>>>> assume existence of project-root functionality without needing to have
>>>>> to enable in any particular way.
>>>> OK, if this is something which should be available with emacs -Q right
>>>> at startup, then EDE is out.  It indeed is too big for that.
>>> It should be possible to write the code such that the project-root part
>>> of EDE doesn't require loading too much code.
>> Yes, but no matter how small, EDE depends on EIEIO, so this would mean
>> that EIEIO is loaded right at startup.  I thought this would be
>> considered a showstopper, even when we clean up its namespace use.
>
> Right, preloading EIEIO is not what I was planning to do, indeed.
> IOW, I think the challenge is to extract the "project-root" part of EDE
> in such a way that it doesn't require preloading as much code.
> That might require splitting EIEIO, or some other approach, maybe,
> I don't know.

I don't think that's a practical approach. In EDE, a project *is* a
class.

I would however like to ask if such a 'project-root' feature is
absolutely needed right at startup time. I'm not sure about the overhead
involved when loading files, which might be annoying to people who don't
need it. OTOH, we already have vc-find-file-hook enabled by default,
which I guess renders such questions moot...

BTW, I was wondering if the vc package couldn't just provide the current
root; it felt kind of silly to implement scanning for vcs markers, when
'vc' actually already did that. This way, we could implement a
'project-root' function, which first checks if there's an EDE project,
and if not, it could return the root directory from 'vc' if the current
file is under version control.

-David

Re: A unified project root interface

[Stefan Monnier]

>> That might require splitting EIEIO, or some other approach, maybe,
>> I don't know.
> I don't think that's a practical approach. In EDE, a project *is* a
> class.

[ Actually, there are various reasons to split EIEIO, one of them would
  be to try and fix the eval-and-compile mess.  ]

> I would however like to ask if such a 'project-root' feature is
> absolutely needed right at startup time. I'm not sure about the overhead
> involved when loading files, which might be annoying to people who don't
> need it. OTOH, we already have vc-find-file-hook enabled by default,
> which I guess renders such questions moot...

Right, one possible approach is to try and delay the use of
project-root, as is done for VC: have a preloaded ede-hooks.el file
which is just enough to try and detect projects that use EDE, and then
only load EDE if/when opening a file in such a project.


        Stefan

Re: A unified project root interface

[Davis Herring]

> Right, one possible approach is to try and delay the use of
> project-root, as is done for VC: have a preloaded ede-hooks.el file
> which is just enough to try and detect projects that use EDE, and then
> only load EDE if/when opening a file in such a project.

If we have VCS-based EDE project autovivification, this means that any
file under version control will load EDE.  Is that sufficiently far from
"always" to warrant the effort?

Davis

-- 
This product is sold by volume, not by mass.  If it appears too dense or
too sparse, it is because mass-energy conversion has occurred during
shipping.

Re: A unified project root interface

[Stefan Monnier]

>> Right, one possible approach is to try and delay the use of
>> project-root, as is done for VC: have a preloaded ede-hooks.el file
>> which is just enough to try and detect projects that use EDE, and then
>> only load EDE if/when opening a file in such a project.
> If we have VCS-based EDE project autovivification, this means that any
> file under version control will load EDE.

No, not necessarily.


        Stefan

Re: A unified project root interface

[Stefan Monnier]

> If we have VCS-based EDE project autovivification, this means that any
> file under version control will load EDE.

In order to find a solution to the problem, you shouldn't try to find
the reasons why it will fail, but instead find how you can make it work
(including how to rephrase the problem so it can be solved).


        Stefan

Re: A unified project root interface

[Bruce Korb]

On 03/20/13 09:14, Davis Herring wrote:
>> Right, one possible approach is to try and delay the use of
>> project-root, as is done for VC: have a preloaded ede-hooks.el file
>> which is just enough to try and detect projects that use EDE, and ...

All this is really nice for projects that allow editor hint files
to be incorporated into the project.  Otherwise, no.  Some projects
I work on absolutely insist on not having any editor hints anywhere
at all.  After all, there is one and only one correct way to format
your code, so learn it, use it and keep the hints out of the source.

Maybe just make this easier, since it doesn't work for me:

http://www.emacswiki.org/ProjectSettings
http://www.gnu.org/software/emacs/manual/html_node/emacs/Directory-Variables.html

Re: A unified project root interface

[Stefan Monnier]

>>> Right, one possible approach is to try and delay the use of
>>> project-root, as is done for VC: have a preloaded ede-hooks.el file
>>> which is just enough to try and detect projects that use EDE, and ...
> All this is really nice for projects that allow editor hint files
> to be incorporated into the project.  Otherwise, no.

The text you quoted does not assume you make any Emacs-specific changes
to your project.


        Stefan

Re: A unified project root interface

[David Engster]

Stefan Monnier writes:
>>> That might require splitting EIEIO, or some other approach, maybe,
>>> I don't know.
>> I don't think that's a practical approach. In EDE, a project *is* a
>> class.
>
> [ Actually, there are various reasons to split EIEIO, one of them would
>   be to try and fix the eval-and-compile mess.  ]

What do you mean by "splitting EIEIO"? Into what parts?

>> I would however like to ask if such a 'project-root' feature is
>> absolutely needed right at startup time. I'm not sure about the overhead
>> involved when loading files, which might be annoying to people who don't
>> need it. OTOH, we already have vc-find-file-hook enabled by default,
>> which I guess renders such questions moot...
>
> Right, one possible approach is to try and delay the use of
> project-root, as is done for VC: have a preloaded ede-hooks.el file
> which is just enough to try and detect projects that use EDE, and then
> only load EDE if/when opening a file in such a project.

This won't work; you cannot detect whether a file is part of a project
without the project's definition (which comes in the form of a class, as
I've already written). It's a chicken/egg thing. Yes, there are projects
which save their state in a file 'Project.ede'; these could be detected,
but that's a special case.

So again, if such a 'project-root' feature is needed at startup, EDE is
out. Separating EDE from EIEIO would mean rewriting it.

Instead, I'd vote for a very simple approach which at least takes EDE
projects into account, among other things. Roughly like this:

(defun project-root-ede (file)
  (when (and (featurep' ede)
             (with-current-buffer file
               ede-object-root-project))
    ... return EDE root ... ))

(defun project-root-vc (file)
  (when (vc-file-registered file)
    ... return root from VC ... ))

(defvar project-root-detect-functions 
          '(project-root-ede project-root-vc))

(defun project-root ()
   (run-hook-with-args-until-success
       project-root-detect-functions
       (buffer-file-name (current-buffer))))

-David

Re: A unified project root interface

[Stefan Monnier]

>> [ Actually, there are various reasons to split EIEIO, one of them would
>> be to try and fix the eval-and-compile mess.  ]
> What do you mean by "splitting EIEIO"? Into what parts?

Sorry, I mostly meant "split eieio.el".  What parts?  Well, that's
a good question.  I think if I knew, I'd have done that already ;-)
Maybe a first split would be "all the code in eieio.el up until the end
of the first big eval-and-compile".
At least, if the aim is to get rid of those nasty eval-and-compile.

> This won't work; you cannot detect whether a file is part of a project
> without the project's definition (which comes in the form of a class, as
> I've already written). It's a chicken/egg thing.

I don't think so.  The first part of the detection is to look for the
tell-tale files (.bzr, .dir-locals, Tupfile, you name it).

The project definitions can setup some side table that can be preloaded
and used by a function that looks for those files.

Some of those files will be frequent enough that we may also want to
preload some additional predicate that checks whether we really need to
load "the rest".


        Stefan

Re: A unified project root interface

[Eric M. Ludlam]

On 03/20/2013 01:47 PM, Stefan Monnier wrote:
>>> [ Actually, there are various reasons to split EIEIO, one of them would
>>> be to try and fix the eval-and-compile mess.  ]
>> What do you mean by "splitting EIEIO"? Into what parts?
>
> Sorry, I mostly meant "split eieio.el".  What parts?  Well, that's
> a good question.  I think if I knew, I'd have done that already ;-)
> Maybe a first split would be "all the code in eieio.el up until the end
> of the first big eval-and-compile".
> At least, if the aim is to get rid of those nasty eval-and-compile.

This email, and browsing in eieio.el in the CEDET repository gave me an 
a-ha moment.  I think I know how to "split" eieio such that the 
eval-and-compile elements are fixed, and have a much nicer eieio.el in 
the process.  My initial experiments show I'm on the right track.

David, is there something that needs merging from Emacs to CEDET I need 
to worry about before making massive structural changes in EIEIO?

>> This won't work; you cannot detect whether a file is part of a project
>> without the project's definition (which comes in the form of a class, as
>> I've already written). It's a chicken/egg thing.
>
> I don't think so.  The first part of the detection is to look for the
> tell-tale files (.bzr, .dir-locals, Tupfile, you name it).
>
> The project definitions can setup some side table that can be preloaded
> and used by a function that looks for those files.
>
> Some of those files will be frequent enough that we may also want to
> preload some additional predicate that checks whether we really need to
> load "the rest".

I think all David is trying to say is that the EDE project detection 
system uses EIEIO classes to hold the match data.  For example, the 
class instance contains the name of the key file to look for that 
identifies the project.

If the goal is to use EDE's detection scheme and data, but not load EDE 
project classes, then we are all set already since the EDE project 
classes are not directly involved in detecting the projects.  If the 
goal is to not use EIEIO at all, then we'd end up just using a plist 
other random data structure instead of using EIEIO to do it.  This 
wouldn't be a big deal because AFAIK, these classes aren't subclassed, 
so impact would be small.

In both cases above, we'd need a simple mechanism to disable the loading 
of EDE project classes, and using a placeholder in the project cache 
instead.

Alternately, perhaps I can find out why EIEIO isn't acceptable, and fix 
that instead.

Eric

Re: A unified project root interface

[Stefan Monnier]

> My initial experiments show I'm on the right track.
> David, is there something that needs merging from Emacs to CEDET I need to
> worry about before making massive structural changes in EIEIO?

The EIEIO in Emacs has my renaming patch (which fixes some of the
namespace issues, tho it's only a first step).

Maybe it would make sense to make this change on Emacs's side only.
And to split EIEIO from CEDET (you can distribute an "old EIEIO" for
users of CEDET on older Emacsen that came without EIEIO).

> If the goal is to use EDE's detection scheme and data, but not load EDE
> project classes, then we are all set already since the EDE project classes
> are not directly involved in detecting the projects.  If the goal is to not
> use EIEIO at all, then we'd end up just using a plist other random data
> structure instead of using EIEIO to do it.  This wouldn't be a big deal
> because AFAIK, these classes aren't subclassed, so impact would be small.

I'mnot sure I understand exactly what you're saying, but IIUC you're
saying that there's not much to do to delay EDE loading, and we could
even delay loading of EIEIO with a few code changes.

That sounds promising and confirms my intuition.

> In both cases above, we'd need a simple mechanism to disable the loading of
> EDE project classes, and using a placeholder in the project cache instead.

Sot sure what you're saying here.

> Alternately, perhaps I can find out why EIEIO isn't acceptable, and fix
> that instead.

Preloading EIEIO and EDE is not something I'm ready to do.  Even just
preloading EIEIO is not something I'm ready to do either.

I haven't started to think about why, but I'm pretty sure EDE is out of
the question for now.  For EIEIO, maybe we could take a look at what it
would entail to see under which circumstances it would be acceptable,
but first it'd need to be namespace clean.

From what you say above, it's probably easier to change EDE to not use
EIEIO for the data used for project-detection, before EDE is
actually loaded.


        Stefan

Re: A unified project root interface

[Eric M. Ludlam]

On 03/20/2013 11:27 PM, Stefan Monnier wrote:
>> My initial experiments show I'm on the right track.
>> David, is there something that needs merging from Emacs to CEDET I need to
>> worry about before making massive structural changes in EIEIO?
>
> The EIEIO in Emacs has my renaming patch (which fixes some of the
> namespace issues, tho it's only a first step).
>
> Maybe it would make sense to make this change on Emacs's side only.
> And to split EIEIO from CEDET (you can distribute an "old EIEIO" for
> users of CEDET on older Emacsen that came without EIEIO).

Hmmm, well, I managed to split EIEIO to no longer use eval-and-compile, 
and it is passing my unit tests, but not my integration tests.  I made 
CEDET branch with this change in it in case someone wants to play with it.

http://cedet.bzr.sourceforge.net/bzr/cedet/code/eieiosplit/files/head%3A/lisp/eieio/

Getting the integration tests to pass is more than I have time for 
tonight though.

>> If the goal is to use EDE's detection scheme and data, but not load EDE
>> project classes, then we are all set already since the EDE project classes
>> are not directly involved in detecting the projects.  If the goal is to not
>> use EIEIO at all, then we'd end up just using a plist other random data
>> structure instead of using EIEIO to do it.  This wouldn't be a big deal
>> because AFAIK, these classes aren't subclassed, so impact would be small.
>
> I'mnot sure I understand exactly what you're saying, but IIUC you're
> saying that there's not much to do to delay EDE loading, and we could
> even delay loading of EIEIO with a few code changes.

Correct

> That sounds promising and confirms my intuition.
>
>> Alternately, perhaps I can find out why EIEIO isn't acceptable, and fix
>> that instead.
>
> Preloading EIEIO and EDE is not something I'm ready to do.  Even just
> preloading EIEIO is not something I'm ready to do either.

Is this stating that the goal is for project detection to be part of 
Lisp code that is dumped with Emacs?

> I haven't started to think about why, but I'm pretty sure EDE is out of
> the question for now.  For EIEIO, maybe we could take a look at what it
> would entail to see under which circumstances it would be acceptable,
> but first it'd need to be namespace clean.
>
>> From what you say above, it's probably easier to change EDE to not use
> EIEIO for the data used for project-detection, before EDE is
> actually loaded.

If we are talking about what gets dumped out with Emacs, I agree.

Eric

Re: A unified project root interface

[Stefan Monnier]

>> Preloading EIEIO and EDE is not something I'm ready to do.  Even just
>> preloading EIEIO is not something I'm ready to do either.
> Is this stating that the goal is for project detection to be part of Lisp
> code that is dumped with Emacs?

Yes.


        Stefan

Re: A unified project root interface

[Eric M. Ludlam]

On 03/21/2013 12:07 AM, Eric M. Ludlam wrote:
> Hmmm, well, I managed to split EIEIO to no longer use eval-and-compile,
> and it is passing my unit tests, but not my integration tests.  I made
> CEDET branch with this change in it in case someone wants to play with it.
>
> http://cedet.bzr.sourceforge.net/bzr/cedet/code/eieiosplit/files/head%3A/lisp/eieio/
>
>
> Getting the integration tests to pass is more than I have time for
> tonight though.

I found the fix for the integration tests, and have been running on this 
branch this evening.  I also removed the eval-and-compile from the 
defclass macro.  All builds and tests seem to run fine.

I did some basic dev with it, and seems to be working fine.

If anyone has comments on this branch, or thoughts that will make it 
better, let me know.

Eric

EIEIO split (was: A unified project root interface)

[David Engster]

Eric M. Ludlam writes:
> On 03/21/2013 12:07 AM, Eric M. Ludlam wrote:
>> Hmmm, well, I managed to split EIEIO to no longer use eval-and-compile,
>> and it is passing my unit tests, but not my integration tests.  I made
>> CEDET branch with this change in it in case someone wants to play with it.
>>
>> http://cedet.bzr.sourceforge.net/bzr/cedet/code/eieiosplit/files/head%3A/lisp/eieio/
>>
>>
>> Getting the integration tests to pass is more than I have time for
>> tonight though.
>
> I found the fix for the integration tests, and have been running on
> this branch this evening.  I also removed the eval-and-compile from
> the defclass macro.  All builds and tests seem to run fine.
>
> I did some basic dev with it, and seems to be working fine.
>
> If anyone has comments on this branch, or thoughts that will make it
> better, let me know.

I think this is a very good change and should be merged, since this will
also fix bug #13813 on our side.

-David

Re: A unified project root interface

[David Engster]

Eric M. Ludlam writes:
W> On 03/20/2013 01:47 PM, Stefan Monnier wrote:
>> Sorry, I mostly meant "split eieio.el".  What parts?  Well, that's
>> a good question.  I think if I knew, I'd have done that already ;-)
>> Maybe a first split would be "all the code in eieio.el up until the end
>> of the first big eval-and-compile".
>> At least, if the aim is to get rid of those nasty eval-and-compile.
>
> This email, and browsing in eieio.el in the CEDET repository gave me
> an a-ha moment.  I think I know how to "split" eieio such that the
> eval-and-compile elements are fixed, and have a much nicer eieio.el in
> the process.  My initial experiments show I'm on the right track.
>
> David, is there something that needs merging from Emacs to CEDET I
> need to worry about before making massive structural changes in EIEIO?

No, it's all already merged to upstream, including the renames by
Stefan.

> If the goal is to use EDE's detection scheme and data, but not load
> EDE project classes, then we are all set already since the EDE project
> classes are not directly involved in detecting the projects.  If the
> goal is to not use EIEIO at all, then we'd end up just using a plist
> other random data structure instead of using EIEIO to do it.  This
> wouldn't be a big deal because AFAIK, these classes aren't subclassed,
> so impact would be small.

Let's talk examples, then. Say we have

 (ede-project-autoload "vcs-root"
		       :name "VCS ROOT"
		       :file 'vcs-root
		       :proj-file 'ede-check-for-vcs-dirs
		       :proj-root 'ede-vcs-root-dir
		       :proj-root-dirmatch "NONE"
		       :class-sym 'ede-vcs-root-project
		       :load-type 'ede-vcs-root-load
		       :new-p nil
		       :safe-p t)

So from that we'd take the list of slots, which is practically a plist,
and would write something akin to ede-dir-to-projectfile, but using
plist-get/set instead of oref/oset?

And what would happen when `ede-check-for-vcs-dirs' returns t?  Would
that load EDE then, or would we try to go on to provide the basic
functionality (like getting the root) with a class-less version?

-David

Re: A unified project root interface

[Eric M. Ludlam]

On 03/21/2013 12:32 PM, David Engster wrote:
> Eric M. Ludlam writes:
> W>  On 03/20/2013 01:47 PM, Stefan Monnier wrote:
>>> Sorry, I mostly meant "split eieio.el".  What parts?  Well, that's
>>> a good question.  I think if I knew, I'd have done that already ;-)
>>> Maybe a first split would be "all the code in eieio.el up until the end
>>> of the first big eval-and-compile".
>>> At least, if the aim is to get rid of those nasty eval-and-compile.
>>
>> This email, and browsing in eieio.el in the CEDET repository gave me
>> an a-ha moment.  I think I know how to "split" eieio such that the
>> eval-and-compile elements are fixed, and have a much nicer eieio.el in
>> the process.  My initial experiments show I'm on the right track.
>>
>> David, is there something that needs merging from Emacs to CEDET I
>> need to worry about before making massive structural changes in EIEIO?
>
> No, it's all already merged to upstream, including the renames by
> Stefan.

Great!  Thanks.

>> If the goal is to use EDE's detection scheme and data, but not load
>> EDE project classes, then we are all set already since the EDE project
>> classes are not directly involved in detecting the projects.  If the
>> goal is to not use EIEIO at all, then we'd end up just using a plist
>> other random data structure instead of using EIEIO to do it.  This
>> wouldn't be a big deal because AFAIK, these classes aren't subclassed,
>> so impact would be small.
>
> Let's talk examples, then. Say we have
>
>   (ede-project-autoload "vcs-root"
> 		       :name "VCS ROOT"
> 		       :file 'vcs-root
> 		       :proj-file 'ede-check-for-vcs-dirs
> 		       :proj-root 'ede-vcs-root-dir
> 		       :proj-root-dirmatch "NONE"
> 		       :class-sym 'ede-vcs-root-project
> 		       :load-type 'ede-vcs-root-load
> 		       :new-p nil
> 		       :safe-p t)
>
> So from that we'd take the list of slots, which is practically a plist,
> and would write something akin to ede-dir-to-projectfile, but using
> plist-get/set instead of oref/oset?

Yes, basically.  Seems odd to swap one data type form for another, but 
there it is.  Since the class name is used as a function, no code using 
it will need to change.  Perhaps.

> And what would happen when `ede-check-for-vcs-dirs' returns t?  Would
> that load EDE then, or would we try to go on to provide the basic
> functionality (like getting the root) with a class-less version?

I think this will be a bit of a challenge.  The project detection, and 
then project hash are the key important pieces.  If the goal is to get 
something that will be dumped w/ Emacs, and Fast, we'd need to start by 
refactoring ede/files.el to split out the parts that track the 
directory-to-project associations from all the misc EDE related file 
finding routines.

Once that is working, I had originally thought we could use the 
ede-project-placeholder as a starting point since it was designed to be 
swapped out for a real project later, but that is still an EIEIO class.

Instead, probably another small list left as a buffer-local variable 
that can be tracked.  I imagine it would hold the root of the project, 
and some misc :key describing the nature of the matched project.  That 
would be used instead of ede-object for quick project detection.  I 
think it would also need a couple new hooks, including 
ede-new-project-detected-hook.  EDE would use that to actually load a 
project instead of doing so directly from the auto-loader.

Another option is to hand-craft an EIEIO object vector.  They are all 
just vectors, so if we had an EIEIO class that had a small number of 
slots (smaller than the existing placeholder object) it could be created 
by hand with (vector ... ) and used to track projects.  Once EIEIO was 
loaded, more complex code could just keep using it, but as an object.  I 
wouldn't do that if there were more than 2 or 3 slots in it though.  As 
code changed, the indices might change, and maintenance would be a pain. 
  As for a transition strategy though, it might make things easier.

As for ede-check-for-vcs-dirs, it would need to provide data to the 
auto-loader that doesn't need to be called to detect the project.  There 
is a bit of logic that says "If the :file isn't loaded yet, try this 
other thing".  That way if it is loaded, it can use one of the handy 
functions which often have caches of their own for detection and makes 
things faster.

On the flip side, perhaps this is an opportunity to simplify.  There is 
a bunch of historical baggage in the loader.  I'll bet there are some 
options that could be fixed/removed, simplifying the whole thing.

Anyway, the key is that the autoloader could be recycled into something 
EIEIO free, and a simple hook would give EDE what it needs to keep 
going, without it's current custom autoloader, and it might be possible 
to keep old registration fcns working in the process.

Eric

Re: A unified project root interface

[David Engster]

Eric M. Ludlam writes:
> On 03/21/2013 12:32 PM, David Engster wrote:
>> And what would happen when `ede-check-for-vcs-dirs' returns t?  Would
>> that load EDE then, or would we try to go on to provide the basic
>> functionality (like getting the root) with a class-less version?
>
> I think this will be a bit of a challenge.  The project detection, and
> then project hash are the key important pieces.  If the goal is to get
> something that will be dumped w/ Emacs, and Fast, we'd need to start
> by refactoring ede/files.el to split out the parts that track the
> directory-to-project associations from all the misc EDE related file
> finding routines.

Frankly, I don't think this is the right approach. To cite from Jorgen's
initial post which started this whole thread:

"The usual way to define a project is via the "project root", a
directory so that all files under that directory are considered part of
"the project"."

As he also stresses, this is *not* a complex problem to solve, quite the
contrary.

Therefore, I don't think it is necessary or even desirable to rewrite
the whole EDE project autodetection into a class-less version, which
then by all means tries to delay loading the "real" EDE as long as
possible. It is not necessary that an 'emacs -Q' can detect all kinds of
projects from Linux kernel trees to the Emacs source. If people need
this, they should just put 'global-ede-mode' in their init file (which
takes about 0.1 seconds on my machine) and use the full-fledged EDE.

> Instead, probably another small list left as a buffer-local variable
> that can be tracked.  I imagine it would hold the root of the project,
> and some misc :key describing the nature of the matched project.  That
> would be used instead of ede-object for quick project detection.

[...]

> On the flip side, perhaps this is an opportunity to simplify.  There
> is a bunch of historical baggage in the loader.  I'll bet there are
> some options that could be fixed/removed, simplifying the whole thing.
>
> Anyway, the key is that the autoloader could be recycled into
> something EIEIO free, and a simple hook would give EDE what it needs
> to keep going, without it's current custom autoloader, and it might be
> possible to keep old registration fcns working in the process.

I'm wondering: what is the advantage of having this class-less EDE
autoloader, compared to a new, simple package which is able to detect
simple projects and which can be overridden by EDE?

My problem is not that I don't think this couldn't work. I'm pretty sure
it would. However, it sounds like way too much work for something which
should be really simple, and which has not much benefit for EDE as a
framework; in fact, I wonder if it doesn't even raise its complexity.

-David

Re: A unified project root interface

[Eric M. Ludlam]

On 03/22/2013 04:30 PM, David Engster wrote:
> Eric M. Ludlam writes:
>> On 03/21/2013 12:32 PM, David Engster wrote:
>>> And what would happen when `ede-check-for-vcs-dirs' returns t?  Would
>>> that load EDE then, or would we try to go on to provide the basic
>>> functionality (like getting the root) with a class-less version?
>>
>> I think this will be a bit of a challenge.  The project detection, and
>> then project hash are the key important pieces.  If the goal is to get
>> something that will be dumped w/ Emacs, and Fast, we'd need to start
>> by refactoring ede/files.el to split out the parts that track the
>> directory-to-project associations from all the misc EDE related file
>> finding routines.
>
> Frankly, I don't think this is the right approach. To cite from Jorgen's
> initial post which started this whole thread:
>
> "The usual way to define a project is via the "project root", a
> directory so that all files under that directory are considered part of
> "the project"."
>
> As he also stresses, this is *not* a complex problem to solve, quite the
> contrary.
>
> Therefore, I don't think it is necessary or even desirable to rewrite
> the whole EDE project autodetection into a class-less version, which
> then by all means tries to delay loading the "real" EDE as long as
> possible. It is not necessary that an 'emacs -Q' can detect all kinds of
> projects from Linux kernel trees to the Emacs source. If people need
> this, they should just put 'global-ede-mode' in their init file (which
> takes about 0.1 seconds on my machine) and use the full-fledged EDE.

To be plain, I agree.  What Jorgen described is not the EDE project 
loader.  Here's the deal though.  EDE's project loader was once what 
Jorgen described, without all the classes and stuff.  It just detected 
the project, found the root, and moved on.

Different projects need different features though, and things evolved. 
If a simple project system is put into Emacs core, it will satisfy the 
primary use case.  Then over time the case+1 will evolve it until it is 
just as messy as the EDE one.

Case:  simple.el is 6000+ lines of code.  Everything from set-variable 
(pretty simple) to compose-mail-other-frame (what ??).

If this project concept is created as a simple thing, that's fine, but 
EDE won't be able to use it, though it could contribute.   If that's the 
overall story where simple uses need the simple project, and EDE is used 
when you need more, that seems like a fine compromise, but it won't 
simplify the plethora of project projects.  Of course, that's a good 
thing too.

Eric

Re: A unified project root interface

[Jorgen Schaefer]

On Sat, 23 Mar 2013 13:10:20 -0400
"Eric M. Ludlam" <eric@siege-engine.com> wrote:

> If this project concept is created as a simple thing, that's fine,
> but EDE won't be able to use it, though it could contribute.   If
> that's the overall story where simple uses need the simple project,
> and EDE is used when you need more, that seems like a fine
> compromise, but it won't simplify the plethora of project projects.

Let's say we create two primary API functions

- (project-root)
- (project-set-root DIR)

so that PROJECT-ROOT returns the current project root if set, and if
not, sets it to something a simple hook returns. Then, if EDE's project
system simply calls (project-set-root DIR) when it loads, we end up
with a upwards-compatible system so that some extensions can just use
P-R to find the project root, and will continue to work when EDE is in
use. If they need more functionality than P-R, they can simply
transition over to using EDE.

Am I missing something?

	-- Jorgen

Re: A unified project root interface

[Dmitry Gutov]

Jorgen Schaefer <forcer@forcix.cx> writes:

> On Sat, 23 Mar 2013 13:10:20 -0400
> "Eric M. Ludlam" <eric@siege-engine.com> wrote:
>
>> If this project concept is created as a simple thing, that's fine,
>> but EDE won't be able to use it, though it could contribute.   If
>> that's the overall story where simple uses need the simple project,
>> and EDE is used when you need more, that seems like a fine
>> compromise, but it won't simplify the plethora of project projects.
>
> Let's say we create two primary API functions
>
> - (project-root)
> - (project-set-root DIR)

I think it would work better if insted of the second function, we'll
have a variable `project-root-functions', along the lines of
`completion-at-point-functions'.

It will contain some simple function by default (that will look for
.dir-locals.el, for example), and EDE, when loaded, can push its own
function on top.

Re: A unified project root interface

[Pascal J. Bourguignon]

Dmitry Gutov <dgutov@yandex.ru> writes:

> Jorgen Schaefer <forcer@forcix.cx> writes:
>
>> On Sat, 23 Mar 2013 13:10:20 -0400
>> "Eric M. Ludlam" <eric@siege-engine.com> wrote:
>>
>>> If this project concept is created as a simple thing, that's fine,
>>> but EDE won't be able to use it, though it could contribute.   If
>>> that's the overall story where simple uses need the simple project,
>>> and EDE is used when you need more, that seems like a fine
>>> compromise, but it won't simplify the plethora of project projects.
>>
>> Let's say we create two primary API functions
>>
>> - (project-root)
>> - (project-set-root DIR)
>
> I think it would work better if insted of the second function, we'll
> have a variable `project-root-functions', along the lines of
> `completion-at-point-functions'.

Not instead, but as hooks:

(defvar *current-project-dir* nil)
(defvar *resign-current-project-hook* '())
(defvar *become-current-project-hook* '())

(defun project-set-root (dir)
   (when *current-project-dir*
     (run-hook-with-args *resign-current-project-hook* *current-project-dir*))
   (do-default-project-set-root dir) 
   (assert (eq *current-project-dir* dir))
   (run-hook-with-args *become-current-project-hook* *current-project-dir*))


> It will contain some simple function by default (that will look for
> .dir-locals.el, for example), and EDE, when loaded, can push its own
> function on top.

Indeed, (do-default-project-set-root dir) could just set 
*current-project-dir*, and allt the rest could be done in
*become-current-project-hook*.

-- 
__Pascal Bourguignon__                     http://www.informatimago.com/
A bad day in () is better than a good day in {}.

Re: A unified project root interface

[Dmitry Gutov]

"Pascal J. Bourguignon" <pjb@informatimago.com> writes:

> Dmitry Gutov <dgutov@yandex.ru> writes:
>
>> Jorgen Schaefer <forcer@forcix.cx> writes:
>>
>>> On Sat, 23 Mar 2013 13:10:20 -0400
>>> "Eric M. Ludlam" <eric@siege-engine.com> wrote:
>>>
>>>> If this project concept is created as a simple thing, that's fine,
>>>> but EDE won't be able to use it, though it could contribute.   If
>>>> that's the overall story where simple uses need the simple project,
>>>> and EDE is used when you need more, that seems like a fine
>>>> compromise, but it won't simplify the plethora of project projects.
>>>
>>> Let's say we create two primary API functions
>>>
>>> - (project-root)
>>> - (project-set-root DIR)
>>
>> I think it would work better if insted of the second function, we'll
>> have a variable `project-root-functions', along the lines of
>> `completion-at-point-functions'.
>
> Not instead, but as hooks:
>
> (defvar *current-project-dir* nil)
> (defvar *resign-current-project-hook* '())
> (defvar *become-current-project-hook* '())

The presence of hooks would be independent of what I'm suggesting.

The idea is that `project-root' calls all functions in
`project-root-functions', and as soon as one of them returns a
meaningful value, `project-root' (being the caller) uses it to set the
value of `current-project-dir', and then probably caches it.

The difference is like between "push" and "pull" designs.

With "push" design, it harder to understand which functions and under
which circumstances should call `project-set-root'.

> (defun project-set-root (dir)
>    (when *current-project-dir*
>      (run-hook-with-args *resign-current-project-hook* *current-project-dir*))
>    (do-default-project-set-root dir) 
>    (assert (eq *current-project-dir* dir))
>    (run-hook-with-args *become-current-project-hook* *current-project-dir*))
>
>
>> It will contain some simple function by default (that will look for
>> .dir-locals.el, for example), and EDE, when loaded, can push its own
>> function on top.
>
> Indeed, (do-default-project-set-root dir) could just set 
> *current-project-dir*, and allt the rest could be done in
> *become-current-project-hook*.

Imagine you're writing a third-party library that would like to know the
project root of the current buffer. What function should it call? Just
`project-root'? Who is going to call `do-default-project-set-root'?

Re: A unified project root interface

[Jorgen Schaefer]

On Sun, 24 Mar 2013 08:25:38 +0400
Dmitry Gutov <dgutov@yandex.ru> wrote:

> The idea is that `project-root' calls all functions in
> `project-root-functions', and as soon as one of them returns a
> meaningful value, `project-root' (being the caller) uses it to set the
> value of `current-project-dir', and then probably caches it.

I did a quick implementation of my current ideas on the topic:

https://raw.github.com/jorgenschaefer/project-el/master/project.el

Quick run-down:

- If you only want to know the project root, you call (project-root)
  and never have to worry about anything else.
- This will call functions in `project-guess-root-functions' until one
  returns a non-nil value. This value is cached for future calls.
- The default functions use the current EDE project if any, or VC
  directory markers, or ask the user; can be extended as needed.
- When an extension provides a way to change the project root, they
  can call `project-set-root', which will run
  `project-root-changed-hook' so that other extensions (e.g. EDE) can
  intercept that and adjust themselves accordingly. Or even throw an
  error.

Other extensions should be able to build on that.

Regards,
	-- Jorgen

Re: A unified project root interface

[Jorgen Schaefer]

On Sun, 24 Mar 2013 11:13:10 +0100
Jorgen Schaefer <forcer@forcix.cx> wrote:

> I did a quick implementation of my current ideas on the topic:
> 
> https://raw.github.com/jorgenschaefer/project-el/master/project.el
> 
> Quick run-down:
> 
> - If you only want to know the project root, you call (project-root)
>   and never have to worry about anything else.
> - This will call functions in `project-guess-root-functions' until one
>   returns a non-nil value. This value is cached for future calls.
> - The default functions use the current EDE project if any, or VC
>   directory markers, or ask the user; can be extended as needed.
> - When an extension provides a way to change the project root, they
>   can call `project-set-root', which will run
>   `project-root-changed-hook' so that other extensions (e.g. EDE) can
>   intercept that and adjust themselves accordingly. Or even throw an
>   error.
> 
> Other extensions should be able to build on that.

As the discussion died down: What would be the next step here?

Regards,
	-- Jorgen

Re: A unified project root interface

[Eric M. Ludlam]

On 04/06/2013 09:25 AM, Jorgen Schaefer wrote:
> On Sun, 24 Mar 2013 11:13:10 +0100
> Jorgen Schaefer<forcer@forcix.cx>  wrote:
>
>> I did a quick implementation of my current ideas on the topic:
>>
>> https://raw.github.com/jorgenschaefer/project-el/master/project.el
>>
>> Quick run-down:
>>
>> - If you only want to know the project root, you call (project-root)
>>    and never have to worry about anything else.
>> - This will call functions in `project-guess-root-functions' until one
>>    returns a non-nil value. This value is cached for future calls.
>> - The default functions use the current EDE project if any, or VC
>>    directory markers, or ask the user; can be extended as needed.
>> - When an extension provides a way to change the project root, they
>>    can call `project-set-root', which will run
>>    `project-root-changed-hook' so that other extensions (e.g. EDE) can
>>    intercept that and adjust themselves accordingly. Or even throw an
>>    error.
>>
>> Other extensions should be able to build on that.
>
> As the discussion died down: What would be the next step here?

I am probably too close to EDE to be very helpful in solving the wider 
problem, but I can offer a branch in the CEDET repository as a testing 
ground so that any changes needed in EDE can be easily made and tested 
in our test suites.

We can remove it from CEDET for maintenance elsewhere once the initial 
development is done.  We can keep new tests around in CEDET to make sure 
EDE stays compatible.


Eric

Re: A unified project root interface

[David Engster]

Eric M. Ludlam writes:
> On 04/06/2013 09:25 AM, Jorgen Schaefer wrote:
>> On Sun, 24 Mar 2013 11:13:10 +0100
>> Jorgen Schaefer<forcer@forcix.cx>  wrote:
>>
>
>>> I did a quick implementation of my current ideas on the topic:
>>>
>>> https://raw.github.com/jorgenschaefer/project-el/master/project.el
>>>
>>> Quick run-down:
>>>
>>> - If you only want to know the project root, you call (project-root)
>>>    and never have to worry about anything else.
>>> - This will call functions in `project-guess-root-functions' until one
>>>    returns a non-nil value. This value is cached for future calls.
>>> - The default functions use the current EDE project if any, or VC
>>>    directory markers, or ask the user; can be extended as needed.
>>> - When an extension provides a way to change the project root, they
>>>    can call `project-set-root', which will run
>>>    `project-root-changed-hook' so that other extensions (e.g. EDE) can
>>>    intercept that and adjust themselves accordingly. Or even throw an
>>>    error.
>>>
>>> Other extensions should be able to build on that.
>>
>> As the discussion died down: What would be the next step here?
>
> I am probably too close to EDE to be very helpful in solving the wider
> problem, but I can offer a branch in the CEDET repository as a testing
> ground so that any changes needed in EDE can be easily made and tested
> in our test suites.
>
> We can remove it from CEDET for maintenance elsewhere once the initial
> development is done.  We can keep new tests around in CEDET to make
> sure EDE stays compatible.

I don't think there are any changes necessary in EDE. All Jorgen's code
does is to determine the project's root directory for the current file,
and EDE is one possibility to provide this information, among others.

I don't have a problem with this approach, besides a small hunch that
soon people will add more features to 'project-root', and one day it may
look a bit like EDE. Unsurprisingly, I still think for general project
definitions EDE is the right choice.

-David

Re: A unified project root interface

[Daniel Colascione]

On 03/24/2013 03:13 AM, Jorgen Schaefer wrote:
> On Sun, 24 Mar 2013 08:25:38 +0400
> Dmitry Gutov <dgutov@yandex.ru> wrote:
>
>> The idea is that `project-root' calls all functions in
>> `project-root-functions', and as soon as one of them returns a
>> meaningful value, `project-root' (being the caller) uses it to set the
>> value of `current-project-dir', and then probably caches it.
>
> I did a quick implementation of my current ideas on the topic:
>
> https://raw.github.com/jorgenschaefer/project-el/master/project.el

After the feature freeze, can we please get something like this (or a 
refactored auto autoloaded EIEIO and EDE-light) into Emacs?

Re: A unified project root interface

[Jorgen Schaefer]

On Wed, 20 Mar 2013 17:34:09 +0100
David Engster <deng@randomsample.de> wrote:

> Instead, I'd vote for a very simple approach which at least takes EDE
> projects into account, among other things. Roughly like this:
> 
> (defun project-root-ede (file)
>   (when (and (featurep' ede)
>              (with-current-buffer file
>                ede-object-root-project))
>     ... return EDE root ... ))
> 
> (defun project-root-vc (file)
>   (when (vc-file-registered file)
>     ... return root from VC ... ))
> 
> (defvar project-root-detect-functions 
>           '(project-root-ede project-root-vc))
> 
> (defun project-root ()
>    (run-hook-with-args-until-success
>        project-root-detect-functions
>        (buffer-file-name (current-buffer))))

This sounds like an excellent idea. If you guarantee the hook to be run
in the file's buffer, you do not even need the argument.

Happy to write the code etc. if this is deemed the way to go :-)

Regards,
	-- Jorgen

Re: A unified project root interface

[Jorgen Schaefer]

On Mon, 18 Mar 2013 23:50:23 +0100
David Engster <deng@randomsample.de> wrote:

> Yes, functions like `ede-project-root-directory' are actually methods,
> but why does it matter?

Nothing. I don't have a problem with CLOS (I think you might be
confusing me with another person who mentioned a distaste for CLOS). A
method for me has a complexity of a function with one required
argument. Which is, a higher complexity than a function with no
arguments. ;-)

> I'm willing to code the necessary stuff on the EDE side of things; if
> it turns out too complicated to use for package maintainers, I have no
> problem throwing it away.

This is the tricky part. I do not know what is "too complex", and it's
difficult to predict this. The only way is to try and provide a very
simple API and see if it gets adopted.

As to EDE/CEDET's complexity, I hope that "oh no, the memory usage" or
"oh no, the load time" isn't a *huge* concern if the API is simple
enough.

It would be great if EDE has such a simple entry point to allow for
expansion and better integration between packages. That is, you can use
this API without knowing anything else, and when you need more, you can
stumble into the CEDET manual and go "woah, it's full of stars". ;-)

My original idea was to provide an API that CEDET/EDE can build upon,
too, but I don't really care either way.

> This can be done. I will need a bit of time though, since I really
> need to do another CEDET merge round with current trunk first. I
> think I'll be able to come up with something in the coming weeks.

Thank you for your willingness to support this!

> > - Ask authors of extensions to use (ede-minor-mode 1) in their mode
> >   function and simply use that function in their modes.
> 
> Not sure if it's a good idea to enable EDE behind the user's back; I
> think they should enable it in their init file if they want to have
> project support. But IMO that's a detail; let's cross that bridge when
> we get there.

Yes, it's a detail. I'm trying to replicate the current use case in
e.g. find-file-in-project where the user simply has to install ffip as
a package and can use the function without having to configure anything
else.

Regards,
	-- Jorgen

Re: A unified project root interface

[joakim]

David Engster <deng@randomsample.de> writes:

> Jorgen Schaefer writes:
>> On Sat, 16 Mar 2013 15:18:50 +0100
>> David Engster <deng@randomsample.de> wrote:
>>
>>> I think we don't need another project mechanism in Emacs. EDE and
>>> dir-locals should be enough. What we have to do however is to develop
>>> at least one very basic project type in EDE which can be used similar
>>> to project-root.el.
>>
>> Unless you create this in a way that is usable without requiring (or
>> understanding) EDE, I fear that it will not be used by many
>> other packages. And that's the issue I started this thread for.
>>
>> The issue is not that we would not have a way to define a project. We
>> do. Dozens of them, actually. Every package has their own. It's so
>> trivial to do for 90% of all use cases that most packages just write
>> their own code.
>
> I think the main issue is that every "language community" has a
> different view of what a "project" entails, so existing solutions are
> often missing something crucial, or things that should be easy to do are
> too cumbersome. It's the same reason why almost every new popular
> programming language sooner or later comes along with its own build tool
> (Rake, SCons, Maven, Ant, Leiningen, CMake, and so on). It's always
> better to have something specifically aimed at the tool-chain you're
> using. In my opinion, this is the reason why so many people develop
> their own project definitions in Emacs: they want something with
> precisely the functionality they need, nothing more or less.
>
> EDE has developed into a framework for writing such specialized projects
> in. If you want to *develop* such a specialized project type for your
> tool-chain, you indeed need to understand the inner workings of EDE,
> which mostly means to first learn EIEIO syntax, which I guess is what
> most people don't like about it. However, if you just want to *query* a
> project object for something, all you need to know is 'oref', and even
> that could be wrapped in helper functions if needed.
>
> It is entirely possible to write a project type so that it is very
> simple to use for end users. I think our C/C++ project wrapper is a good
> example for this, where you define a project like this:
>
>    (ede-cpp-root-project "NAME"
>          :file "~/myproject/Makefile"
>          :include-path '( "/include" "../include" "/c/include" )
>          :system-include-path '( "/usr/include/c++/3.2.2/" )
>          :spp-table '( ("OS_GNU_LINUX" . "1") ))
>
> I'd argue this is already similar to how projects are defined in
> project-root.el. This is all a user has to use, and every file loaded
> from that project will have a buffer local ede-object, from which you
> can easily extract those attributes. This project type was deliberately
> created for people who don't want to explicitly mess with targets,
> linkers, compilers, object files, dependency generation, 'clean' rules,
> and so on. My suggestion was to create something similar, but more
> generic and not explicitly aimed at C/C++. We already have a 'generic'
> project type, which IMO comes close.

I would just like to voice my support for EDE. It is a good base for
further facilities.

The critique is also correct though, that it has been too complicated to
setup a very simple project type. That should be something that we that
are interested in CEDET should continue to work on. It has all the
facilities one might ask for with a project base framework in
Emacs. (That doesnt mean that others can't keep on working on their own
competing solutions of course, in the time honoured tradition of
Emacs-hackers)

>
> -David
>

-- 
Joakim Verona

Re: A unified project root interface

[Sudish Joseph]

Using .dir-locals as the definition of a project is eminently
reasonable, but it'd be good if projects and their roots could also be
computed dynamically or definitely implicitly from the context of an
action.

Computable roots:

- VC root directories define the most natural project roots in many
  cases.  This is by far the common case for me: look for the .git/
  directory.
- File and directory existence checks.  Rails projects contain these
  directories at their root: app, config, db.  Perl used to contain t
  and lib, though it's been a while.

It'd be good if Emacs didn't require or even write any context of its
own into these directories like most IDEs do - there's no reason to.

Implicit roots:

- Emacsclient should generate an implicit project root if given a
  directory as an argument.
- Drag and drop a directory into Emacs. This used to be one of
  Textmate's killer features - easy project creation.

Emacs doesn't need to define all this, just provide standard hooks to
let the user or external packages define this.  What would be great
would be if the grep interfaces and such could look for and use project
roots if one is found.

Here's the definition of projects I'm using right now:

(setq project-roots
      `(("Rails Project"
         :root-contains-files ("app" "config" "db" "lib" "script" "test")
         :on-hit (lambda (p) (message (car p))))
	("Emacs config"
         :path-matches ,(format "\\(%s\\)" sj/emacs-base-dir)
	 :on-hit (lambda (p) (message (car p))))
	("Perl Project"
         :root-contains-files ("t" "lib")
         :on-hit (lambda (p) (message (car p))))
	("Git Project"
         :root-contains-files (".git")
         :on-hit (lambda (p) (message (car p))))))

That's for project-root.el - selected because it was the only one I
found at the time.

The default .git rule turns out to be the most useful and catches most
projects.

Here's an example of a package that groups buffers by vc root that could
do an even better job with a standardized project interface in Emacs
itself: https://github.com/purcell/ibuffer-vc

Cheers,
-Sudish

Re: A unified project root interface

[Dmitry Gutov]

Sudish Joseph <sudish@gmail.com> writes:

> Here's the definition of projects I'm using right now:
>
> (setq project-roots
>       `(("Rails Project"
>          :root-contains-files ("app" "config" "db" "lib" "script" "test")
>          :on-hit (lambda (p) (message (car p))))
> 	("Emacs config"
>          :path-matches ,(format "\\(%s\\)" sj/emacs-base-dir)
> 	 :on-hit (lambda (p) (message (car p))))
> 	("Perl Project"
>          :root-contains-files ("t" "lib")
>          :on-hit (lambda (p) (message (car p))))
> 	("Git Project"
>          :root-contains-files (".git")
>          :on-hit (lambda (p) (message (car p))))))

This looks quite similar to eproject configuration:

https://github.com/jrockway/eproject/wiki/ProjectTypes

> Here's an example of a package that groups buffers by vc root that could
> do an even better job with a standardized project interface in Emacs
> itself: https://github.com/purcell/ibuffer-vc

Indeed. As another example, if I were to start working on some
programming-language specific feature, like code completion, I'd need a
way to enumerate files and/or buffers that belong to the same project.

So, to add to the list of the required features, the project root
mechanism will need a way to enumerate all files in the project (or
maybe all files of a certain type), and a way to ignore files and
directories, for example to improve performance of the enumerator
function.

Re: A unified project root interface

[Ted Zlatanov]

On Sun, 10 Mar 2013 00:38:12 -0500 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 

SM> As mentioned .dir-local.el often acts as a project root, but it can also
SM> appear in sub-directories of a project, so it's not sufficient.
SM> It should at least be augmented so that some .dir-locals.el can be
SM> flagged as "non-root".

I think it's simple to add a "root" property to .dir-locals.el.  If it
exists, stop searching.  It's backwards compatible (unlike a non-root
flag, IIUC) and doesn't require new libraries or variables, just a guard
in the search code.

Ted