Re: progmodes/project.el and search paths

[Eric Ludlam <ericludlam@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 2925 bytes --]

On 08/02/2015 01:17 PM, Dmitry Gutov wrote:
> If you can propose better ways to do the same things, or something 
> else worth adding, please do. 

AND

On 08/04/2015 09:43 AM, Eli Zaretskii wrote:
> I really think we could benefit from presenting some kind of goal or
> vision for the project.el development.  AFAICS, it started as a way to
> standardize finding a "project root", then moved to finding "project
> directories", but now it seems we are discussing a much wider scope,
> something like infrastructure for project definition and possibly also
> IDEs?  If it's the latter, then I won't expect to see arguments about
> "complications outweighing benefits" in reference to features that are
> basic for any IDE that is worth its menus.
>
> So maybe we should step back a notch and decide anew where is
> project.el going, and what is and isn't in its scope?

Hi Dmitry,

I was tempted to start replying to multiple sub-threads in multiple 
emails, but I suspect there is nuance I am not expressing well, so I've 
opted to go back to one of your original questions to engage in a more 
positive way.

I scraped through some old threads, old code, and my memory to put 
together the attached org file.  The intent is to answer your first 
question about things worth adding to a generic project.  It also 
happens to be something that can answer Eli's question, so I quoted him too.

When the generic project was first proposed, I thought it was a good 
idea where either EDE could be adapted to be a base, or EDE could adapt 
to a new base.  I wasn't too picky, but I clearly had many assumptions 
about what it was that did not match.

EDE's story is one where it started out specific, but became more 
generic because I needed to provide services to external tools I was 
writing, and there was just a lot of code donations and demand for more 
project definitions.

Emacs has a lot of tools that could use a project service, and sometimes 
all the tool needs is a small push of convenience, and not full project 
management system.  A generic project system will make it easy to add 
small conveniences in lots of places you may not expect because tool 
authors will feel safe using it.

The attached is my experience of things I either needed in EDE's base 
classes to implement a project, or that some tool needed, and what I 
learned along the way.  You might think it is too EDE specific, but I 
put in only the most basic stuff, and put in things I kind of wish I had 
time to add to EDE.

I am sure there are entries you consider out of scope.  In my mind 
projects need a centralizing force to bring consistency in the same way 
comint brought some very basic consistency to shell buffers.  We should 
not skimp in approaching this goal.

I consider this file incomplete, but I hope it proves useful as a 
starting point.  I'd be happy to periodically update it for a little 
while if it is useful.

Eric

[-- Attachment #2: generic_project.txt --]
[-- Type: text/plain, Size: 12119 bytes --]

Eric's notes for Generic Project

* Definitions
** Project Directory
 A top level directory in which source code for a project resides.
** Project
 A collection of Meta Data and Services that describe how code in a Project
 Directory is organized, versioned, compiled, etc.
** Project Meta Data
 A piece of information that describes some aspect of the project.
 Also: `slot' on a 'Project' in CLOS parlance
** Project Service
 Functionality provided by a generic project interface
 Also: 'method' on a 'Project' in CLOS parlance
** Project Client
 A feature that depends on either the Project Directory, Project, or a
 Project Service, usually developed independently of project implementations.
** Project Implementation
 An implementation of a concrete project that provides actual
 functionality, some of which is provided via the generic project
 service.
** Thing
 Something the generic project should handle that doesn't fit into
 one of the above categories... yet.

* Service: Current Project Directory
Provides external tools a way to identify the top-most directory of the current project.

** Features:
*** Performance; For a buffer:
As fast as a buffer local variable ref
*** Performance; For a file or directory name: 
First time lookup should take advantage of any known open projects.
*** Nested Projects
There should be ways for there to be projects in projects.
**** Example: Having a set of small projects for testing in a larger project

** Project Client Examples
*** Identify a TAGS file to use for navigating a source tree
*** Starting point for search tools
*** Location of Manifests, AUTHOR, and other dist files

* Service: Current Project
Tools that need to know about some meta-data of the current project
will need a way to refer to the current project.  The current project
needs to be some 'thing' that provides project features.

** Features
*** Project Meta Data
Projects can have meta data.  There should be some standard meta-data,
and meta data optionally provided by specific project implementations.
*** Project Services
Projects can have behavior and provide services.  There should be
some basic standard services, and optionally provided services
specific to project implementations.
*** Project Commands
Services for users to execute when they are managing their project.
There should be some basic standard commands, and optionally provided
commands specific to project implementations.

Project Commands may seem optional from a generic project point of view.
They are primarily helpful as a homogenizing force between disparate
project implementations which would be a good thing in the long run,
the way comint made all shell buffers very similar and easy to move
between.
*** Menu/Toolbar/Other UI Services
Project implementations should have a standard way to integrate into
the Development menu where users expect project commands to exist.
*** Extensible
The thing used to represent a generic project should be extensible so
project implementations can implement abstract services, or add new
ones.
*** Sensible Defaults
The generic project thing should have sensible defaults for meta data
and specific services so a project implementation doesn't need to
implement them all.

** Example Features
*** Version number of project
Useful when distinguishing between different local repositories you
might be working on.
*** Compile
Integration with the `compile' command, a standard action for many
compiled programs.  A more generic term that would work for running
tests in an interpreted environment may be preferred.

* Service: Project Detection
All project management systems need a way to 'detect' a project the
first time a user visits a file.  Many of those systems are the same,
and involve walking up and down directory trees looking for
dominating files.

** Features:
*** Dominating File Detection
A core way to identify projects that use a dominating file, such as a
top level build.xml, or a AndroidManifest.xml.
*** Distributed dominating file detection
A core way to identify projects that use the same dominating file in
all directories in a project, such as Makefile, CVS.
*** Nested Projects
There should be ways for there to be projects in projects.
*** Declarative detection features
When an project implementation integrates with the generic project
system, it should be able to 'declare' how to detect a project of its
preferred type.  That way the source code for the project
implementation will only be loaded into Emacs when a project of that
type is detect.
*** Many project types detectable w/out user configuration
Users should have all available project types automatically detected
without having to fiddle with their .emacs file.  ELPA project
implementations should have a way to add themselves to the detection
list without loading their code.

** Project Detection Declaration features 
Features that a declaration of how to detect should provide
*** How to detect
A declarative way to express the desired detection mechanism.
*** How to load the project code to support the project
A function symbol and/or a symbol to require that will enable a
project implementation to be created to support the detected project.
*** Project Loading safety
Some project types might need to load elisp-code-like structures to
bring in meta data.  If this is potentially unsafe, the project loader
should ask the user, or consult a configuration, before allowing it.
*** Detection ordering
Some project types are a subset of other project types.  Priority of
project detection/loading should be a part of the detection declaration.
**** Example:
Makefile based project could exist with a .git repository.  The
Makefile project is more specific than the .git repository, and
should be selected first.

** Tool Example
*** A project manager (JDEE, EDE, etc)
These tools need to detect projects that belong to them, but EDE might
identify an ANT project, and JDEE might identify its .prj file, and
they would attempt to co-exist without a mediating generic project
service.  When a tool asks for the current project, there should be
only one.

* Service: Project Commands
Emacs already has a wide range of useful IDE like tools, such as
`compile' and `gud'.  These are generic, and require the user to type
in any specific options needed for their project.

Generic projects should provide integration with those tools,
and likely multiple variants regarding the project in question.  The
project service should allow project implementations to augment these
core features with variants.

** Features
*** Compile integration
Projects implementations should be able to provide default compile
arguments based on project meta data.  Multiple variations should be
available such as `whole project' or `just this thing in my buffer'.
*** GUD integration
Provide default debugger commands, and arguments to GUD supported
debugger commands.
*** Update meta-data
Emacs is not the only project system out there, so users should have
a way to force a project implementation to refresh itself from
on-disk changes.
*** Project browsing
Start users preferred file browsing project client on the project.
**** Examples: ECB, Speedbar, Dired, external executable

* Service: Keybinding / Menu / Toolbar management
Project implementations should have a standard way to integrate into
the Development menu where users expect project commands to exist.
User commands provided should be consistent between types of projects.

The purpose of this is to provide a homogenizing force so that
different project implementations will gain a similar look and feel
so users can move between different projects, and enjoy consistent
keybindings and UI elements.

** Features
*** Organized project options
Tools should be logically organized in whichever UI elements are
used to express it.
*** Default project commands
There should be some default project commands, like 'compile' built in.
*** Room for custom options
There should be an easy way for a project implementation to add new
items into the logical groupings provided.

** Examples
*** Arduino: upload to dev board
Arduino project implementation needs to add an entry to the menu
system to upload a project onto the dev board, separate from
compiling to a binary.
*** JDEE vs EDE
Compile commands are different between JDEE and EDE.  If a standard
way of invoking the disparate commands were available, users could
work in C++ and Java and use the same keybindings or menus in each.

* Service: Project Local Variables
Tools like dir-locals provide some level of this feature.  Generic
projects could help enable project local variables.

* Meta Data: Version
Projects should provide a way to extract version numbers, and make
them readily available so users can distinguish between multiple
versions of the same code base.

* Meta Data: Arbitrary
Project implementations should have a way to provide their own meta
data to external commands, and have a way to negotiate for commonality
between project implementations that support the same concept.

* Meta Data: Client Data
Project clients that develop a user service based on generic project
data should have a way to provide their own data to a project where
the original project implementation author may not know what is
needed.

** Examples
*** Flymake
Flymake should have a way to ask for meta data about compilation, and
if the project implementation doesn't support it, there should be no
errors, and Flymake shouldn't need to know anything specific about
all implementations that might support it.
*** User Option for client
The project client 'projectile' can toggle between code and test, but
the project implementation author didn't account for that.
Projectile will need to associate user supplied meta-data about test
locations with the project.

Puzzle - persistence?

* Service: Finding Files
Finding files comes in a couple different flavors.
1) User wants to visit a file via a 'short name', and the project can
   find it.
2) Code often refers to other files using a short name, and refers to
   an 'include path' concept in the project to find it.

** Features:
*** Enable a project client to provide user tools to 'jump' to a file.
*** Enable language specific 'include path' concept
Useful when multiple files with the same name exist, and are
distinguished by location.
** Examples
*** C++ include path
The semantic smart completion engine depends on include paths to find
headers which provide definitions to complete on.
*** Java Classpath
Semantic depends on a java classpath to find jar files from which it
extracts symbol information used for smart completion.
*** cscope or GNU Global integration as project service
These tools track not just references, but files in a project, and
have easy ways to convert a short file name into a full path name.
*** project client to jump from 'include' statement in C++ to the file
*** project client to jump between a src file (xml, etc) to generated output.

* Command: Load a project;   Service: Known unopened projects
The generic project system should have a handy way to provide a
completion list of known projects a user might choose to load.

Project lists might be provided by recording all projects a user may
have previously opened, or via scan of areas relevant to the user.

** Examples:
*** desktop
A project client might associated a working set of buffers to load upon
opening a project.

* Thing: Multiple activities on one project
Projects usually have multiple activities in them.  At the most basic
there are 'get it ready to use' support files, such as compilation
system, INSTALL doc, etc, and the source code.  Full project
documentation, and multi-feature projects have multiple build
targets, and complex build systems.

The generic project should consider supporting concepts for multiple
activities in one project.

** Examples
*** `compile' knows which thing to compile in a project
*** Language specificity
Operations in a particular language restricted to other files of that
language.
*** Code & Doc
Enable a project client to support navigating between code and doc.