Unified project interface

Unified project interface

[Dmitry Gutov]

Hi all,

A while ago, there was floated a "unified project root" proposal. I'd 
like to resume that discussion.

Unfortunately, the code proposed back them doesn't serve the immediate 
need I have in mind: basically, I need that function to return a *list* 
of directories, because both etags and elisp xref backends operate on 
multiple directories, via tags-table-list and load-path respectively.

If we have it, we'll be able to untie `xref-find-regexp' from xref 
backends, because really there's nothing language-specific to this command.

Now, before adding this feature to the core, are there other related 
general questions we want to be able to answer? Depending on that, a 
"project" might be better implemented as a cl-struct, providing 
specialized implementations for those several generic methods.

Re: Unified project interface

[Stephen Leake]

Dmitry Gutov <dgutov@yandex.ru> writes:

> A while ago, there was floated a "unified project root" proposal. I'd
> like to resume that discussion.
>
> Unfortunately, the code proposed back them doesn't serve the immediate
> need I have in mind: basically, I need that function to return a
> *list* of directories, because both etags and elisp xref backends
> operate on multiple directories, via tags-table-list and load-path
> respectively.
>
> If we have it, we'll be able to untie `xref-find-regexp' from xref
> backends, because really there's nothing language-specific to this
> command.

The list of directories that xref-find-regexp needs to search is not
the project root; it is the list of directories included by the project
source code.

For that you need a function like "project-source-directories", which
could be customized for each project backend.

I handle this for ada-mode by defining a project file syntax. A project
file defines a list of source directories, among other things. The list
of source directories then gets stored in compilation-search-path.

For ada-mode, project-source-directories would just return
compilation-search-path.

The ada-mode project file can be anywhere; in my projects, it is usually
_not_ at the "project root directory", but down one or two layers in
build/ or build/release/.

I could add "prj_root" to the project file syntax; then ada-mode could
provide a "unified project root" backend that knows what the current
project file is, and returns the value from prj_root. But that would not
be useful in ada-mode; that's why it's not there yet.

The notion of "project root directory" is more useful for configuration
management; typically a config management "project" consists of the tree
of files under one root. But ada-mode knowns nothing about config
management; that's vc's job.

Hmm - vc could query for the current project root, and ada-mode would
provide the answer. That might be useful, and a good reason to have a
unified project interface.

> Now, before adding this feature to the core, are there other related
> general questions we want to be able to answer? Depending on that, a
> "project" might be better implemented as a cl-struct, providing
> specialized implementations for those several generic methods.

There are lots of project meta-data that the ada-mode project file syntax
provides; compiler options, object directory, case exceptions, etc. Some
of those might be common with other projects.

Getting a basic infrastructure in place first is a good idea; we can
always promote things that turn out to be common among many backends.

In any case, I believe a cl-struct is the way to go.

I don't think we should try to unify the xref and project backends, but
every major mode will probably want to provide both.

-- 
-- Stephe

Re: Unified project interface

[Dmitry Gutov]

On 06/04/2015 05:40 PM, Stephen Leake wrote:

> The list of directories that xref-find-regexp needs to search is not
> the project root; it is the list of directories included by the project
> source code.

Not exactly. xref-find-regexp is interested in all kinds of files, not 
just source files. That includes, say, the README file at the top of the 
project.

> For that you need a function like "project-source-directories", which
> could be customized for each project backend.

However, you raise an interesting point. Whereas xref-find-references 
would like to search in all load-path directories, maybe it would be 
enough if xref-find-regexp only searches inside the current "project 
root", for some definition of that notion.

I wonder how we could make it work this way. Make "project root" an 
orthogonal feature?

> The ada-mode project file can be anywhere; in my projects, it is usually
> _not_ at the "project root directory", but down one or two layers in
> build/ or build/release/.

We can't rely on every Elisp project declaring a "project root" in the 
same way, though.

> Hmm - vc could query for the current project root, and ada-mode would
> provide the answer. That might be useful, and a good reason to have a
> unified project interface.

VC is one option for this. Or the project may be not registered in any 
VCS yet, and the root would be determined based on, say, the presence of 
configure.ac. Again, this suggests that notions of "source directories" 
and "project root" can be somewhat orthogonal.

On the other hand, after the project root is determined, it might want 
to add some new element(s) to the source directories list.

> There are lots of project meta-data that the ada-mode project file syntax
> provides; compiler options, object directory, case exceptions, etc. Some
> of those might be common with other projects.

Right. We'll need those pieces of metadata that are useful to more than 
one subsystem, and can be employed in a general way. Though a generic 
metadata storage might be useful as well: thus, if some minor mode has 
read the project file and parsed the compiler options, it can set the 
related metadata, so that code completion and linter could use it 
without waiting for Emacs core to standardize it.

> I don't think we should try to unify the xref and project backends, but
> every major mode will probably want to provide both.

Mostly those major modes that can launch and interact with a smart 
external process, I think. The rest (except for Elisp) will leave it to 
minor modes.

Re: Unified project interface

[Stephen Leake]

Dmitry Gutov <dgutov@yandex.ru> writes:

> On 06/04/2015 05:40 PM, Stephen Leake wrote:
>
>> The list of directories that xref-find-regexp needs to search is not
>> the project root; it is the list of directories included by the project
>> source code.
>
> Not exactly. xref-find-regexp is interested in all kinds of files, not
> just source files. That includes, say, the README file at the top of
> the project.

No problem; make sure that directory is in project-source-directories.

README is a perfectly valid source file; it is written in the "plain
text" language (for which no compiler is needed ;).

I have no problem with projects including more than one language in the
sources; most of mine have Ada, Texinfo, LaTeX, text, elisp, and make
source code. Which is one reason why project and xref cannot be merged
(xref must be language-specific, even for tags).

It might make sense to parameterize project-source-directories
with a language (or major-mode) name, to get the corresponding subset.

>> For that you need a function like "project-source-directories", which
>> could be customized for each project backend.
>
> However, you raise an interesting point. Whereas xref-find-references
> would like to search in all load-path directories, 

'load-path' is an elisp notion. For other languages "source-path" is
more appropriate.

> maybe it would be enough if xref-find-regexp only searches inside the
> current "project root", for some definition of that notion.

No, xref-find-regexp should search project-source-directories.

Most real projects include other projects, so limiting the search to
only the top project is wrong in general (although that might be a
useful option in some use cases).

In addition, there might be a directory under project root that should
_not_ be searched; the object file directory for ada-mode, for example.

project-source-directories should return the union of the source
directories for all of the included projects. That's what load-path is
for elisp, and what compilation-search-path is for ada-mode and other
language modes.

For elisp, project-source-directories should simply return load-path.

> I wonder how we could make it work this way. Make "project root" an
> orthogonal feature?

If you mean "orthogonal to source directories", then yes, that is what I
am suggesting.

>> The ada-mode project file can be anywhere; in my projects, it is usually
>> _not_ at the "project root directory", but down one or two layers in
>> build/ or build/release/.
>
> We can't rely on every Elisp project declaring a "project root" in the
> same way, though.

We can make it a requirement in order to use the general tool. But first
we have to justify it; ada-mode has never needed that notion; neither
has elisp. In my experience, only config management needs it.

>> Hmm - vc could query for the current project root, and ada-mode would
>> provide the answer. That might be useful, and a good reason to have a
>> unified project interface.
>
> VC is one option for this. Or the project may be not registered in any
> VCS yet, and the root would be determined based on, say, the presence
> of configure.ac. Again, this suggests that notions of "source
> directories" and "project root" can be somewhat orthogonal.

Not just "somewhat"; "completely" :).

It also points out that "project root" is poorly defined; I think that
is because it's not at all clear when and why we need it.

> On the other hand, after the project root is determined, it might want
> to add some new element(s) to the source directories list.

"it" is what here ? Can you give an example?

>> There are lots of project meta-data that the ada-mode project file syntax
>> provides; compiler options, object directory, case exceptions, etc. Some
>> of those might be common with other projects.
>
> Right. We'll need those pieces of metadata that are useful to more
> than one subsystem, and can be employed in a general way. Though a
> generic metadata storage might be useful as well: thus, if some minor
> mode has read the project file and parsed the compiler options, it can
> set the related metadata, so that code completion and linter could use
> it without waiting for Emacs core to standardize it.

ada-mode uses a plist to represent the project metadata, and has examples
of minor modes adding to the plist.

-- 
-- Stephe

Re: Unified project interface

[Stephen Leake]

Stephen Leake <stephen_leake@stephe-leake.org> writes:

> Dmitry Gutov <dgutov@yandex.ru> writes:
>
>> On 06/04/2015 05:40 PM, Stephen Leake wrote:
>>
>>> The list of directories that xref-find-regexp needs to search is not
>>> the project root; it is the list of directories included by the project
>>> source code.
>>
>> Not exactly. xref-find-regexp is interested in all kinds of files, not
>> just source files. That includes, say, the README file at the top of
>> the project.
>
> No problem; make sure that directory is in project-source-directories.
>
> README is a perfectly valid source file; it is written in the "plain
> text" language (for which no compiler is needed ;).
>
> I have no problem with projects including more than one language in the
> sources; most of mine have Ada, Texinfo, LaTeX, text, elisp, and make
> source code. Which is one reason why project and xref cannot be merged
> (xref must be language-specific, even for tags).
>
> It might make sense to parameterize project-source-directories
> with a language (or major-mode) name, to get the corresponding subset.

There are a couple of use cases here that require different functions:

1) You are editing a LaTeX file containing documentation for a Python
module implemented in Python and C++. You want to find all mentions/references
related to feature "foo", in LaTeX, Python, and C++ source files. So you
invoke project-find-regexp, which searches for *.tex, *.py, *.hhp, *.cpp
in project-source-directories.

2) You are refactoring a C++ function foo::bar, and want to find all
references to bar* for comparison. You invoke xref-find-regexp, which
searches xref-source-directories.

There are a couple of implementation options here:

- implement project-source-directories as the union of
  xref-source-directories for all backends listed in a project file

- implement xref-source-directories as a subset of
  project-source-directories for each backend.

I'm inclined to the latter; it's closer to what ada-mode does now.

-- 
-- Stephe

Re: Unified project interface

[Dmitry Gutov]

On 06/05/2015 04:03 PM, Stephen Leake wrote:

> 2) You are refactoring a C++ function foo::bar, and want to find all
> references to bar* for comparison. You invoke xref-find-regexp, which
> searches xref-source-directories.

You'll call xref-find-references for that. Its default implementation 
will use a similar set of directories to xref-find-regexp, but it can be 
limited to the proper "source directories".

Sorry, I'll comment on the rest later.

Re: Unified project interface

[Bozhidar Batsov]

[-- Attachment #1: Type: text/plain, Size: 1133 bytes --]

Can you explain in simple terms what exactly are the use cases for the
functionality? Being able to find definitions and grep in projects? I
recall having some discussion a while back, but the details are really
vague in my head.

On 4 June 2015 at 13:43, Dmitry Gutov <dgutov@yandex.ru> wrote:

> Hi all,
>
> A while ago, there was floated a "unified project root" proposal. I'd like
> to resume that discussion.
>
> Unfortunately, the code proposed back them doesn't serve the immediate
> need I have in mind: basically, I need that function to return a *list* of
> directories, because both etags and elisp xref backends operate on multiple
> directories, via tags-table-list and load-path respectively.
>
> If we have it, we'll be able to untie `xref-find-regexp' from xref
> backends, because really there's nothing language-specific to this command.
>
> Now, before adding this feature to the core, are there other related
> general questions we want to be able to answer? Depending on that, a
> "project" might be better implemented as a cl-struct, providing specialized
> implementations for those several generic methods.
>

[-- Attachment #2: Type: text/html, Size: 1479 bytes --]

Re: Unified project interface

[Dmitry Gutov]

On 06/06/2015 01:20 PM, Bozhidar Batsov wrote:
> Can you explain in simple terms what exactly are the use cases for the
> functionality?

That's what this thread is about: to discuss different use cases for the 
unified project interface.

> Being able to find definitions and grep in projects? I
> recall having some discussion a while back, but the details are really
> vague in my head.

xref-find-regexp will use it (always, I think). xref-find-references 
will also use it, at least the default implementation.

I haven't considered, thus far, creating a default xref-find-definitions 
implementation along these lines.

Re: Unified project interface

[Eric Ludlam]

On 06/04/2015 07:43 AM, Dmitry Gutov wrote:
> Hi all,
>
> A while ago, there was floated a "unified project root" proposal. I'd
> like to resume that discussion.
>
[...]

>
> Now, before adding this feature to the core, are there other related
> general questions we want to be able to answer? Depending on that, a
> "project" might be better implemented as a cl-struct, providing
> specialized implementations for those several generic methods.

In that old thread, I had proposed using EDE either as a supplier to a 
generic interface, or as a way to build said interface.  Several people 
pointed out flaws in EDE's project detection scheme.   Since that time I 
have accounted for those recommendations and added a severe test suite 
for detecting projects more cleanly.  Those changes were merged into 
Emacs a while back.

As an aside, thanks for the advice back then.

In general, if there is a proposed generic project root type I'll be 
happy to enabled EDE to supply data to it.  Alternately, if there is an 
interest in an EDE-lite where key functionality such as detection and 
providing a root path is pulled up into something that doesn't turn on 
all the other unwanted EDE features, I'd be happy to advise or help.

For an example, in 25.x, in cedet/ede/generic.el, look for 
`ede-enable-generic-projects' for some simple definitions.  I'm sure the 
generic project type in EDE is overkill for what is wanted, so I'm 
assuming there would need to be a stripped down version.

As a reminder, ede uses CLOS, so there is a base object representing the 
project, and you can run methods against it to get information which 
would be simple to expand to any type of desired generic interface.

Eric

Re: Unified project interface

[Dmitry Gutov]

Hi Eric,

On 06/06/2015 03:32 PM, Eric Ludlam wrote:

> In general, if there is a proposed generic project root type I'll be
> happy to enabled EDE to supply data to it.

That's good.

> Alternately, if there is an
> interest in an EDE-lite where key functionality such as detection and
> providing a root path is pulled up into something that doesn't turn on
> all the other unwanted EDE features, I'd be happy to advise or help.
>
> For an example, in 25.x, in cedet/ede/generic.el, look for
> `ede-enable-generic-projects' for some simple definitions.  I'm sure the
> generic project type in EDE is overkill for what is wanted, so I'm
> assuming there would need to be a stripped down version.

While it looks pretty decent, I'm among those'd prefer to see CEDET 
moved out to GNU ELPA.

And anyway, it would be better to have the smallest possible API that, 
say, Projectile could implement without relying on EDE being present and 
loaded.

But any lessons EDE can teach us about making that API better, would be 
great to have.

> As a reminder, ede uses CLOS, so there is a base object representing the
> project, and you can run methods against it to get information which
> would be simple to expand to any type of desired generic interface.

I imagine the new API will work along the same principle, only using 
cl-generic instead of eieio.

Does EDE store information about paths outside of the current project 
root? Like linked projects, or C include directories, or load-path 
elements (if EDE supports Elisp projects)?

Does it allow to quickly grep across all of them, or use 
semantic-symref, again, across all of them?

Re: Unified project interface

[Eli Zaretskii]

> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Sat, 6 Jun 2015 21:44:12 +0300
> Cc: Jorgen Schaefer <contact@jorgenschaefer.de>,
> 	Bozhidar Batsov <bozhidar@batsov.com>
> 
> I'm among those'd prefer to see CEDET moved out to GNU ELPA.

If we want the IDE capabilities of Emacs become better, this should
never happen.

Re: Unified project interface

[Dmitry Gutov]

On 06/06/2015 10:28 PM, Eli Zaretskii wrote:

> If we want the IDE capabilities of Emacs become better, this should
> never happen.

IME, CEDET is fairly interdependent, and Semantic's approach doesn't 
seem to be useful enough for, say, code completion in dynamic languages.

I see the path forward as different CEDET features growing into more 
flexible APIs that different packages can implement.

Like semantic-symref now has a generic counterpart of 
xref-find-references, and while the latter has a simplistic grep-based 
implementation, CEDET can define its own xref backend, to implement the 
latter through the former.

Re: Unified project interface

[Dmitry Gutov]

On 06/05/2015 01:08 PM, Stephen Leake wrote:

> No problem; make sure that directory is in project-source-directories.

How would that work, in technical terms? Which entity will "make sure" 
of that? Will a project-backend "source directories" implementation 
always cl-call-next-method? Will the default impl simply return the 
value of some variable, assigned by a major mode?

> README is a perfectly valid source file; it is written in the "plain
> text" language (for which no compiler is needed ;).

I see a few problems with that:

- README has no source code (it doesn't affect the logic of any 
program). "sources" already has a common meaning, and at least in one 
proprietary editor it's just one a subset of all project files (see the 
nomenclature here: 
https://www.jetbrains.com/idea/help/content-root.html). Further, if we 
can separate them, we could make sure xref-find-references only searches 
in the "proper" source directories.

- There's no direct way to find the directory holding the README when 
you only have the value of load-path or compilation-search-path (because 
it's quite often one level above one of the path elements). So it would 
make sense to have a different name (and a different function/slot/etc) 
for directories that might be reached that way.

> I have no problem with projects including more than one language in the
> sources; most of mine have Ada, Texinfo, LaTeX, text, elisp, and make
> source code. Which is one reason why project and xref cannot be merged
> (xref must be language-specific, even for tags).

xref should take the language into account, but it should work across 
them as well. For instance, a Ruby method call in an *.html.erb template 
file should get recognized as a reference to that method, and its "find 
definition" should lead to the corresponding *.rb file.

> It might make sense to parameterize project-source-directories
> with a language (or major-mode) name, to get the corresponding subset.

That's probably not necessary: after all, only the default 
xref-find-references implementation will use project-source-directories 
in a naive way. A smarter xref backend should determine itself which 
directories to search.

> 'load-path' is an elisp notion. For other languages "source-path" is
> more appropriate.

I didn't mean to suggest to call anything generic a "load path".

> No, xref-find-regexp should search project-source-directories.

Maybe even project-directories (project-roots?), if we agree to separate 
those notions.

> Most real projects include other projects, so limiting the search to
> only the top project is wrong in general (although that might be a
> useful option in some use cases).

Maybe when called with C-u. It's not easy to allow to specify multiple 
directories that way anyway.

> In addition, there might be a directory under project root that should
> _not_ be searched; the object file directory for ada-mode, for example.

project-ignore-directories should be a thing. Probably in the format of 
grep-find-ignored-directories (and we might have a function 
corresponding to grep-find-ignored-files). Trouble is, 
projectile-ignored-directories seems to be more flexible already, so 
maybe that will have to be accompanied with an upgrade to how 
grep-find-ignored-directories are handled.

> We can make it a requirement in order to use the general tool. But first
> we have to justify it; ada-mode has never needed that notion; neither
> has elisp. In my experience, only config management needs it.

Like mentioned above, for example, the distinction would help with the 
default xref-find-references implementation.

>> On the other hand, after the project root is determined, it might want
>> to add some new element(s) to the source directories list.
>
> "it" is what here ? Can you give an example?

Some package that defines that root-finding logic. Not every major mode 
knows where the relevant source directories are, or it might not know 
some of them.

If we've opened one of the files in a third-party Elisp project that 
hasn't been loaded yet (so not in load-path), a project implementation 
can still add an element or several to project-source-directories, so 
that xref-find-references returns somewhat useful results anyway.

Or, say, for Ruby: while the major mode might conceivably be taught 
about paths where to find system-wide libraries, the locations of source 
files in a given project depend on the project's type, and that must be 
something a project implementation knows better. Even if it'll simply 
add the root to project-source-directories.

> ada-mode uses a plist to represent the project metadata, and has examples
> of minor modes adding to the plist.

Cool. Then it must be useful indeed.

Re: Unified project interface

[Dmitry Gutov]

On 06/05/2015 04:03 PM, Stephen Leake wrote:

> 1) You are editing a LaTeX file containing documentation for a Python
> module implemented in Python and C++. You want to find all mentions/references
> related to feature "foo", in LaTeX, Python, and C++ source files. So you
> invoke project-find-regexp, which searches for *.tex, *.py, *.hhp, *.cpp
> in project-source-directories.

project-find-regexp? Do you imagine we'll need a separate command like that?

> 2) You are refactoring a C++ function foo::bar, and want to find all
> references to bar* for comparison. You invoke xref-find-regexp, which
> searches xref-source-directories.

Like mentioned, it'll hopefully be xref-find-references.

> There are a couple of implementation options here:
>
> - implement project-source-directories as the union of
>    xref-source-directories for all backends listed in a project file

This sounds like you envision a one-backend-per-language kind of 
situation. I think that there should be just one backend there, set in a 
minor mode, that knows enough about this mixed project's structure, to 
know where to search.

> - implement xref-source-directories as a subset of
>    project-source-directories for each backend.
>
> I'm inclined to the latter; it's closer to what ada-mode does now.

Could you explain what that means exactly? How is it implemented?

Re: Unified project interface

[Stephen Leake]

Dmitry Gutov <dgutov@yandex.ru> writes:

> On 06/05/2015 04:03 PM, Stephen Leake wrote:
>
>> 2) You are refactoring a C++ function foo::bar, and want to find all
>> references to bar* for comparison. You invoke xref-find-regexp, which
>> searches xref-source-directories.
>
> You'll call xref-find-references 

That handles a specific identifier, not all identifiers macthing "bar.*"
(to use proper regexp syntax, rather than shell glob).

> for that. Its default implementation
> will use a similar set of directories to xref-find-regexp, but it can
> be limited to the proper "source directories".

How will you specify two (or more?) sets of directories for xref?

-- 
-- Stephe

Re: [SPAM UNSURE] Re: Unified project interface

[Stephen Leake]

Dmitry Gutov <dgutov@yandex.ru> writes:

> On 06/05/2015 04:03 PM, Stephen Leake wrote:
>
>> 1) You are editing a LaTeX file containing documentation for a Python
>> module implemented in Python and C++. You want to find all mentions/references
>> related to feature "foo", in LaTeX, Python, and C++ source files. So you
>> invoke project-find-regexp, which searches for *.tex, *.py, *.hhp, *.cpp
>> in project-source-directories.
>
> project-find-regexp? Do you imagine we'll need a separate command like
> that?

Yes, that's the point. That's how you distinuish between multi-language
and single-language searches.

>> There are a couple of implementation options here:
>>
>> - implement project-source-directories as the union of
>>    xref-source-directories for all backends listed in a project file
>
> This sounds like you envision a one-backend-per-language kind of
> situation. 

Yes, for xref.

> I think that there should be just one backend there, set in
> a minor mode, that knows enough about this mixed project's structure,
> to know where to search.

I don't see how that would be better; that minor mode would end up doing
what I'm suggesting the "project" code do.

>> - implement xref-source-directories as a subset of
>>    project-source-directories for each backend.
>>
>> I'm inclined to the latter; it's closer to what ada-mode does now.
>
> Could you explain what that means exactly? How is it implemented?

I don't have a working example of this precise structure now. What I'm
imagining is a "project file" that lists source directories without
regard to language. To be concrete, imagine a project with the following
structure:

/root
    src
        *.c files

    doc
        *.tex files

    build
        foo.prj -- the project file
        *.make files


foo.prj contains (among other things):

src_dir=../src
src_dir=../doc
src_dir=.

So project-source-directories is the list ("root/src" "root/doc"
"root/build").

If we are in a .c buffer, and we request xref-source-directories, the
code looks thru project-source-directories for directories that contain
c files, and returns only ("root/src"). This could be an expensive
check, so we'd want to cache the results.

Alternately, foo.prj could have:

language=c
language=tex
language=make

src_dir(c)=../src
src_dir(tex)=../doc
src_dir(make)=.

(You could get fancier and use "latex" as a language name, and have
another statement that maps that to ".tex". Which is needed for C and
Ada anyway, since they use two file extensions; .h/.c, .ads/.adb).

That imposes the requirement on the user that they know what files are
where.

Note that the directories do not have to be language exclusive; they are
here just for clarity.

-- 
-- Stephe

Re: Unified project interface

[Stephen Leake]

Dmitry Gutov <dgutov@yandex.ru> writes:

> On 06/05/2015 01:08 PM, Stephen Leake wrote:
>
>> No problem; make sure that directory is in project-source-directories.
>
> How would that work, in technical terms? Which entity will "make sure"
> of that? 

The user; they are the only ones that know what "the project" is.

> Will a project-backend "source directories" implementation
> always cl-call-next-method? 

I don't understand this.

> Will the default impl simply return the value of some variable,
> assigned by a major mode?

default implementation of what function?

>> README is a perfectly valid source file; it is written in the "plain
>> text" language (for which no compiler is needed ;).
>
> I see a few problems with that:
>
> - README has no source code (it doesn't affect the logic of any
> program). 

It's the other way around; there is no object code corresponding to the
README source code.  But that's not very significant.

> "sources" already has a common meaning, 

What is that? I take it to mean "any human-readable file". Which thus
includes README, Makefile, .git/config, etc.

> and at least in one proprietary editor it's just one a subset of all
> project files (see the nomenclature here:
> https://www.jetbrains.com/idea/help/content-root.html). 

That gives me 404

> Further, if we can separate them, we could make sure
> xref-find-references only searches in the "proper" source directories.

What do you mean by "proper"?

We need it to search what the user wants to search; the user needs to be
able to specify that clearly and easily.

> - There's no direct way to find the directory holding the README when
> you only have the value of load-path or compilation-search-path

If the user specifies compilation-search-path (directly or indirectly),
and the user wants to be able to search README, then the user must
include the directory containing README in compilation-search-path.

Is that a problem?

> So it would make sense to have a different name (and a different
> function/slot/etc) for directories that might be reached that way.

I don't follow; can you give more details, and a clear example?

>> I have no problem with projects including more than one language in the
>> sources; most of mine have Ada, Texinfo, LaTeX, text, elisp, and make
>> source code. Which is one reason why project and xref cannot be merged
>> (xref must be language-specific, even for tags).
>
> xref should take the language into account, but it should work across
> them as well. For instance, a Ruby method call in an *.html.erb
> template file should get recognized as a reference to that method, and
> its "find definition" should lead to the corresponding *.rb file.

Ah. Now you are talking about one file that mixes source in two
languages. Yes, that is a problem. There have been discussions here of
multi-mode files. I don't know how that would work in practice.

I have projects that mix Ada and C++; the project level cross reference
facility handles both, using the current xref API. That works because
gcc generates the same cross reference info for those two langauages; I
doubt there is something similar for Ruby and html.

>> It might make sense to parameterize project-source-directories
>> with a language (or major-mode) name, to get the corresponding subset.
>
> That's probably not necessary: after all, only the default
> xref-find-references implementation will use
> project-source-directories in a naive way. A smarter xref backend
> should determine itself which directories to search.

I don't see how that is at all possible, especially in the face of
hierarchical projects.

Only the user knows what a "project" is; they write the hierachical
project files that tell the system what the project structure is.

>> In addition, there might be a directory under project root that should
>> _not_ be searched; the object file directory for ada-mode, for example.
>
> project-ignore-directories should be a thing. Probably in the format
> of grep-find-ignored-directories (and we might have a function
> corresponding to grep-find-ignored-files). Trouble is,
> projectile-ignored-directories seems to be more flexible already, so
> maybe that will have to be accompanied with an upgrade to how
> grep-find-ignored-directories are handled.
>
>> We can make it a requirement in order to use the general tool. But first
>> we have to justify it; ada-mode has never needed that notion; neither
>> has elisp. In my experience, only config management needs it.
>
> Like mentioned above, for example, the distinction would help with the
> default xref-find-references implementation.

You seem to be missing my point; I see no need for project-root, outside
of configuration management. So I don't see why xref should care about
"root" at all.

>>> On the other hand, after the project root is determined, it might want
>>> to add some new element(s) to the source directories list.
>>
>> "it" is what here ? Can you give an example?
>
> Some package that defines that root-finding logic. Not every major
> mode knows where the relevant source directories are, or it might not
> know some of them.

As I've said, none of the languages I deal with care about a "root"
directory. Only git, mtn, cvs care.

Hmm. Java imposes something like a root by forcing the directory
structure to match the hiearchical package naming. But referencing other
package hierarchies complicates that, just as in Ada and C++.

> If we've opened one of the files in a third-party Elisp project that
> hasn't been loaded yet (so not in load-path), a project implementation
> can still add an element or several to project-source-directories, so
> that xref-find-references returns somewhat useful results anyway.

What does that have to do with "root"?

The third-party elisp file will presumably have some paths in "(require
...)" forms that will define a set of directories to be added to
project-source-directories. 

> Or, say, for Ruby: while the major mode might conceivably be taught
> about paths where to find system-wide libraries, the locations of
> source files in a given project depend on the project's type, and that
> must be something a project implementation knows better. Even if it'll
> simply add the root to project-source-directories.

What "root"? Is that the directory containing the Ruby file that we
opened? Why is that a "root" as opposed to just some source directory?

-- 
-- Stephe

Re: Unified project interface

[Dmitry Gutov]

On 06/08/2015 04:24 AM, Stephen Leake wrote:

> That handles a specific identifier, not all identifiers macthing "bar.*"
> (to use proper regexp syntax, rather than shell glob).

I suppose that kind of command could be implemented on top of the 
current xref API: first, find all identifiers that match this regexp 
(the "apropos" action, or maybe just use the 
identifier-completion-table), then find all references to each of them.

Or probably have a new generic method, do the above in the default 
implementation and allow backends to override it.

> How will you specify two (or more?) sets of directories for xref?

Considering we don't currently specify even *one* set of directories 
xref can use (that is one of the goals of this discussion)... How about 
the same way as I'll specify that one, only twice?

Two variables, two generic methods, or so on.

Re: Unified project interface

[Eli Zaretskii]

> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Tue, 9 Jun 2015 21:16:35 +0300
> 
> On 06/08/2015 04:24 AM, Stephen Leake wrote:
> 
> > That handles a specific identifier, not all identifiers macthing "bar.*"
> > (to use proper regexp syntax, rather than shell glob).
> 
> I suppose that kind of command could be implemented on top of the 
> current xref API: first, find all identifiers that match this regexp 
> (the "apropos" action, or maybe just use the 
> identifier-completion-table), then find all references to each of them.
> 
> Or probably have a new generic method, do the above in the default 
> implementation and allow backends to override it.

Both etags and id-utils support regexp search, so a separate method
sounds like a better idea, IMO.

Re: Unified project interface

[Dmitry Gutov]

On 06/09/2015 09:21 PM, Eli Zaretskii wrote:

> Both etags and id-utils support regexp search, so a separate method
> sounds like a better idea, IMO.

etags doesn't support search for references, which is what this 
subthread is about. id-utils does, though.

Re: Unified project interface

[Eli Zaretskii]

> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Tue, 9 Jun 2015 21:49:52 +0300
> Cc: stephen_leake@stephe-leake.org, emacs-devel@gnu.org
> 
> On 06/09/2015 09:21 PM, Eli Zaretskii wrote:
> 
> > Both etags and id-utils support regexp search, so a separate method
> > sounds like a better idea, IMO.
> 
> etags doesn't support search for references, which is what this 
> subthread is about.

Depends on the program that created the database, I guess.

Re: [SPAM UNSURE] Re: Unified project interface

[Dmitry Gutov]

On 06/08/2015 04:35 AM, Stephen Leake wrote:

> Yes, that's the point. That's how you distinuish between multi-language
> and single-language searches.

That is inadequate. An ideal "project" implementation should be smart 
enough to figure out whether to look for the symbol's definition in the 
files with the same extension as the current one, or in files with some 
other extensions as well.

Elisp is lucky to have all its environment at hand. In many cases, for 
decent "find definitions" result, we'll need to know the project 
structure first anyway.

>> I think that there should be just one backend there, set in
>> a minor mode, that knows enough about this mixed project's structure,
>> to know where to search.
>
> I don't see how that would be better; that minor mode would end up doing
> what I'm suggesting the "project" code do.

Maybe. Why wouldn't that be better?

> I don't have a working example of this precise structure now. What I'm
> imagining is a "project file" that lists source directories without
> regard to language. To be concrete, imagine a project with the following
> structure:

That looks decent, but if you mean that xref-source-directories would 
take the whole set of source directories defined for that project and 
only take those from it that correspond to the current language, that 
seems unnecessarily limiting to me (unless there's a way it can be very 
smart about that, and the notion of "current language" will be 
generalized to possibly mean an arbitrary set of major modes and file 
extensions).

Further, I don't want to have xref-source-directories as a separate 
variable or a function, unless we have to.

The idea is to add a new feature to Emacs that deals with projects, and 
make xref use *that*.

Re: Unified project interface

[Dmitry Gutov]

On 06/08/2015 04:59 AM, Stephen Leake wrote:

> The user; they are the only ones that know what "the project" is.

Let's cut out the user out of the discussion and just say that the 
project system knows stuff. Because it can know everything we want to 
ask the user; we just need to figure out the right set of questions.

>> Will a project-backend "source directories" implementation
>> always cl-call-next-method?
>
> I don't understand this.

If project-source-directories is a generic method, maybe we can say that 
its implementation should almost always (unless it really knows what 
it's doing) call the next applicable implementation.

That implementation could be set up to dispatch based on the value of 
major-mode.

>> Will the default impl simply return the value of some variable,
>> assigned by a major mode?
>
> default implementation of what function?

project-source-directories.

>> "sources" already has a common meaning,
>
> What is that? I take it to mean "any human-readable file". Which thus
> includes README, Makefile, .git/config, etc.

Maybe there's value in allowing this semantic split to (sometimes?) go 
along different lines, but the two main directions I want to see 
commands work are these:

- Search all files that belong to any of the projects we're currently 
interested in.

- Search only some of those files; ones that are likely programmatically 
(semantically?) related to the one we're editing right now.

The latter could have a decent-enough implementation using Grep, but it 
would be better if we could only search directories that are related.

Maybe it's indeed an excessive requirement, though. After all, backends 
can provide their own implementations of "find references" already.

>> and at least in one proprietary editor it's just one a subset of all
>> project files (see the nomenclature here:
>> https://www.jetbrains.com/idea/help/content-root.html).
>
> That gives me 404

That link still works for me. Maybe your email client mangles it somehow?

> We need it to search what the user wants to search; the user needs to be
> able to specify that clearly and easily.

I intend to allow the user to specify the exact directory for 
xref-find-regexp to search, when called with a prefix. However, allowing 
to input an arbitrary set of directories is obviously non-trivial (if we 
want to have an interface that's not annoying).

> If the user specifies compilation-search-path (directly or indirectly),
> and the user wants to be able to search README, then the user must
> include the directory containing README in compilation-search-path.
>
> Is that a problem?

I mentioned compilation-search-path only as an example, though I didn't 
expect that you might suggest putting the project root into 
compilation-search-path.

Anyway, emacs-lisp-mode doesn't set compilation-search-path. And 
hopefully you're not suggesting to put project roots into load-path, 
just to be able to search READMEs.

>> So it would make sense to have a different name (and a different
>> function/slot/etc) for directories that might be reached that way.
>
> I don't follow; can you give more details, and a clear example?

If my java project is in ~/vc/fooberz, then project-directories would be 
'("~/vc/fooberz"), and project-source-directories would be 
'("~/vc/fooberz/src"). That seems obvious enough, so maybe I don't 
understand the question.

> Ah. Now you are talking about one file that mixes source in two
> languages. Yes, that is a problem. There have been discussions here of
> multi-mode files.

It's targentially related, but the difficulties in displaying and 
editing multi-mode files are entirely different from the subject of this 
discussion.

> I have projects that mix Ada and C++; the project level cross reference
> facility handles both, using the current xref API. That works because
> gcc generates the same cross reference info for those two langauages; I
> doubt there is something similar for Ruby and html.

Maybe, or maybe not; but that's not very relevant. If the tool *can* be 
written, the API is manageable. In the meantime, we can use Grep.

>> That's probably not necessary: after all, only the default
>> xref-find-references implementation will use
>> project-source-directories in a naive way. A smarter xref backend
>> should determine itself which directories to search.
>
> I don't see how that is at all possible, especially in the face of
> hierarchical projects.

I suppose all parts of the "hierarchical project" will employ the same 
project definition logic, possibly enabled by the same minor mode. And 
that minor mode might as well implement the specialized xref backend 
that will know where to search.

> Only the user knows what a "project" is; they write the hierachical
> project files that tell the system what the project structure is.

See the beginning of this email.

> You seem to be missing my point; I see no need for project-root, outside
> of configuration management. So I don't see why xref should care about
> "root" at all.

Or you're missing mine.

The distinction I'm (or was) suggesting is not between the root and 
everything else, but between "source directories" and "other directories 
the user might want to search in", and we'll often want to see the 
project's root directory inside the latter list (when not in both).

>>>> On the other hand, after the project root is determined, it might want
>>>> to add some new element(s) to the source directories list.
>>>
>>> "it" is what here ? Can you give an example?
>>
>> Some package that defines that root-finding logic. Not every major
>> mode knows where the relevant source directories are, or it might not
>> know some of them.
>
> As I've said, none of the languages I deal with care about a "root"
> directory. Only git, mtn, cvs care.

Allow me to repeat: "it" is the package that defines the root-finding 
logic. Defines the logic that determines the bounds of the current (and 
possibly linked) project(s).

The act of find the project root directory may sometimes be skipped in 
some odd cases, but otherwise, it'll usually be the first step to 
determining the current project.

> Hmm. Java imposes something like a root by forcing the directory
> structure to match the hiearchical package naming. But referencing other
> package hierarchies complicates that, just as in Ada and C++.

So there'll be several roots. Or project-directories. No big deal.

>> If we've opened one of the files in a third-party Elisp project that
>> hasn't been loaded yet (so not in load-path), a project implementation
>> can still add an element or several to project-source-directories, so
>> that xref-find-references returns somewhat useful results anyway.
>
> What does that have to do with "root"?

The root directory usually, a) contains data about the project 
structure, b) includes files we want to search.

> The third-party elisp file will presumably have some paths in "(require
> ...)" forms that will define a set of directories to be added to
> project-source-directories.

First of all, those forms specify path relative to *any* load-path 
element. We can't add every permutation. Second, I was describing the 
case where the requires load-path element weren't even set yet.

Third, either way it won't include the files that depend on the current 
one. Those are important, too.

>> Or, say, for Ruby: while the major mode might conceivably be taught
>> about paths where to find system-wide libraries, the locations of
>> source files in a given project depend on the project's type, and that
>> must be something a project implementation knows better. Even if it'll
>> simply add the root to project-source-directories.
>
> What "root"? Is that the directory containing the Ruby file that we
> opened?

It's some parent directory of the currently opened file. Probably not 
the direct parent. It's important because it contains files related to 
the current one.

 > Why is that a "root" as opposed to just some source directory?

That's a weird question. It's a root of the project. It *is* also one of 
the source directories, in this example.

Re: Unified project interface

[Steinar Bang]

>>>>> Dmitry Gutov <dgutov@yandex.ru>:
> On 06/08/2015 04:59 AM, Stephen Leake wrote:

>>> and at least in one proprietary editor it's just one a subset of all
>>> project files (see the nomenclature here:
>>> https://www.jetbrains.com/idea/help/content-root.html).

>> That gives me 404

Could be the ")" or ")." at the end of the URL...?  

It's always good to use a space after the end of the URL (Gnus managed
to extract the URL without the trailing ").", though...).

Re: Unified project interface

[Dmitry Gutov]

On 06/10/2015 01:31 AM, Dmitry Gutov wrote:

> If project-source-directories is a generic method, maybe we can say that
> its implementation should almost always (unless it really knows what
> it's doing) call the next applicable implementation.
>
> That implementation could be set up to dispatch based on the value of
> major-mode.

I've pushed that to the branch 'scratch/project', please take a look.

The split between "project directories" and "project source directories" 
is not 100% necessary, but I think it can be useful, provided the 
semantic difference between the two is clear.

What I want to do with project-source-directories here, is to allow a 
major mode to provide a default list (or, actually, a function that 
would compute it, because `load-path' can change at runtime).

This way, unless a project implementation explicitly overrides it, Elisp 
authors will have the whole load-path used for searching.

Maybe using a specialized implementation using &context is not the best 
thing to do there. Alternatively, we can introduce a new variable (like 
`project-source-directories-function'), and refer to it in the default 
`project-source-directories' implementation, as well as its docstring.

Re: Unified project interface

[Bozhidar Batsov]

[-- Attachment #1: Type: text/plain, Size: 1824 bytes --]

I finally took a quick look at the current code (and the discussion so far)
and I have a few remarks:

* even within source directories it's useful to be able to exclude certain
subdirs from project-level operations. How is this going to be handled?
* it's not 100% clear to me what are major mode writes supposed to provide
as "project implementations"
* In 24.4 a `vc-root-dir` was added. Seems it overlaps a bit with
`project.el`
* how can Projectile (or a similar package) leverage `project.el`?

On 8 July 2015 at 03:25, Dmitry Gutov <dgutov@yandex.ru> wrote:

> On 06/10/2015 01:31 AM, Dmitry Gutov wrote:
>
>  If project-source-directories is a generic method, maybe we can say that
>> its implementation should almost always (unless it really knows what
>> it's doing) call the next applicable implementation.
>>
>> That implementation could be set up to dispatch based on the value of
>> major-mode.
>>
>
> I've pushed that to the branch 'scratch/project', please take a look.
>
> The split between "project directories" and "project source directories"
> is not 100% necessary, but I think it can be useful, provided the semantic
> difference between the two is clear.
>
> What I want to do with project-source-directories here, is to allow a
> major mode to provide a default list (or, actually, a function that would
> compute it, because `load-path' can change at runtime).
>
> This way, unless a project implementation explicitly overrides it, Elisp
> authors will have the whole load-path used for searching.
>
> Maybe using a specialized implementation using &context is not the best
> thing to do there. Alternatively, we can introduce a new variable (like
> `project-source-directories-function'), and refer to it in the default
> `project-source-directories' implementation, as well as its docstring.
>
>

[-- Attachment #2: Type: text/html, Size: 2418 bytes --]

Re: Unified project interface

[Dmitry Gutov]

On 07/11/2015 04:43 PM, Bozhidar Batsov wrote:
> I finally took a quick look at the current code (and the discussion so
> far) and I have a few remarks:
>
> * even within source directories it's useful to be able to exclude
> certain subdirs from project-level operations. How is this going to be
> handled?

A new function (or two?) will be needed for that. Is there a particular 
reason Projectile has both ignored-directories and ignored-files?

I'm also undecided on what it (or they) should return. Should that be 
just a list of shell globs? That would integrate fine with grep-find.

It also has both grep-find-ignored-directories and 
grep-find-ignored-directories. I wonder if that was done for performance.

> * it's not 100% clear to me what are major mode writes supposed to
> provide as "project implementations"

major modes aren't really supposed to. Since the current project is 
dependent on the directory and not a specific files, it's best left to 
minor modes.

An implementation is a new project-find-functions element and a set of 
cl-defmethod forms to go with it. Maybe also a cl-struct definition for 
the project structure, but as you can see from the current project.el 
contents, you can do without it.

> * In 24.4 a `vc-root-dir` was added. Seems it overlaps a bit with
> `project.el`

Only if the project is based on VC. As you can see project-try-vc uses 
that backend action, if not the function itself.

> * how can Projectile (or a similar package) leverage `project.el`?

Do something along these lines:

(defun project-try-projectile ()
   (when (projectile-project-root) 'projectile)

(add-hook 'project-find-functions #'project-try-projectile)

(cl-defmethod project-root (project (eql projectile))
   (projectile-project-root))

...and so on.

The first difficulty will be that cl-generic (and hence, cl-defmethod) 
are unavailable in the previous Emacs versions. Maybe put all this in a 
separate file and (require ...) it conditionally on the Emacs version.

Re: Unified project interface

[Dmitry Gutov]

On 07/11/2015 05:17 PM, Dmitry Gutov wrote:

> I'm also undecided on what it (or they) should return. Should that be
> just a list of shell globs? That would integrate fine with grep-find.

I've pushed a single-function glob-based implementation.

Re: Unified project interface

[Bozhidar Batsov]

[-- Attachment #1: Type: text/plain, Size: 2512 bytes --]

On 11 July 2015 at 17:17, Dmitry Gutov <dgutov@yandex.ru> wrote:

> On 07/11/2015 04:43 PM, Bozhidar Batsov wrote:
>
>> I finally took a quick look at the current code (and the discussion so
>> far) and I have a few remarks:
>>
>> * even within source directories it's useful to be able to exclude
>> certain subdirs from project-level operations. How is this going to be
>> handled?
>>
>
> A new function (or two?) will be needed for that. Is there a particular
> reason Projectile has both ignored-directories and ignored-files?
>
> I'm also undecided on what it (or they) should return. Should that be just
> a list of shell globs? That would integrate fine with grep-find.
>
> It also has both grep-find-ignored-directories and
> grep-find-ignored-directories. I wonder if that was done for performance.


Might have been an oversight on my part. Right now nothing comes to mind
and I'll have to consult the source to remember. Originally there was only
the option to ignore folders, this much I remember.


>
>
>  * it's not 100% clear to me what are major mode writes supposed to
>> provide as "project implementations"
>>
>
> major modes aren't really supposed to. Since the current project is
> dependent on the directory and not a specific files, it's best left to
> minor modes.
>

e.g. Rails mode or something like this?


>
> An implementation is a new project-find-functions element and a set of
> cl-defmethod forms to go with it. Maybe also a cl-struct definition for the
> project structure, but as you can see from the current project.el contents,
> you can do without it.
>
>  * In 24.4 a `vc-root-dir` was added. Seems it overlaps a bit with
>> `project.el`
>>
>
> Only if the project is based on VC. As you can see project-try-vc uses
> that backend action, if not the function itself.
>
>  * how can Projectile (or a similar package) leverage `project.el`?
>>
>
> Do something along these lines:
>
> (defun project-try-projectile ()
>   (when (projectile-project-root) 'projectile)
>
> (add-hook 'project-find-functions #'project-try-projectile)
>
> (cl-defmethod project-root (project (eql projectile))
>   (projectile-project-root))
>
> ...and so on.
>
> The first difficulty will be that cl-generic (and hence, cl-defmethod) are
> unavailable in the previous Emacs versions. Maybe put all this in a
> separate file and (require ...) it conditionally on the Emacs version.'
>

Yeah, backward compatibility is problematic. I guess those can't be
backported to the `cl-lib` package in ELPA?

[-- Attachment #2: Type: text/html, Size: 3842 bytes --]

Re: Unified project interface

[Dmitry Gutov]

On 07/13/2015 11:49 AM, Bozhidar Batsov wrote:

> Might have been an oversight on my part. Right now nothing comes to mind
> and I'll have to consult the source to remember. Originally there was only
> the option to ignore folders, this much I remember.

I've looked at the code, and you have two functions that returns globs 
to ignore: projectile-paths-to-ignore and projectile-patterns-to-ignore.

They're split along different lines, though (rooted vs non-rooted 
patterns). I'm not sure how useful that would be.

And projectile-ignored-files/directories return actual file names inside 
the project, that's not what I want here either.

For now, it seems you'll implement `project-ignores' by using the value 
returned by `projectile-parse-dirconfig-file' (with a slight change: 
replacing / at bos with ./)

> e.g. Rails mode or something like this?

Yep. ENSIME, Malabar, etc. Or Projectile.

> Yeah, backward compatibility is problematic. I guess those can't be
> backported to the `cl-lib` package in ELPA?

Well, maybe if there's enough of a demand someone will do it.

It's not there yet, and I'm not sure how much work it'll require.

Re: Unified project interface

[Dmitry Gutov]

Let's maybe continue this discussion?

On 06/05/2015 01:08 PM, Stephen Leake wrote:

> No, xref-find-regexp should search project-source-directories.
>
> Most real projects include other projects, so limiting the search to
> only the top project is wrong in general (although that might be a
> useful option in some use cases).

I suppose, and maybe I was wrong to suggest a distinction between 
"source" directories and the rest of them. However, here's one which 
could be better: directories to search vs. directories to modify.

Like, I might want to search the full system includes path, in a C 
project, but when performing a replacement operation (such as 
xref-query-replace), I should probably limit myself to the directories 
with code I actually own. At the moment, it's the distinction I imagine 
for project-search-path vs project-directories.

> In addition, there might be a directory under project root that should
> _not_ be searched; the object file directory for ada-mode, for example.

We have that in project-ignores now.

> We can make [root] a requirement in order to use the general tool. But first
> we have to justify it; ada-mode has never needed that notion; neither
> has elisp. In my experience, only config management needs it.
 > ...
> It also points out that "project root" is poorly defined; I think that
> is because it's not at all clear when and why we need it.

Here's the first use case I know of:

There's a third-party project called rspec-mode. It allows one to switch 
between a source file and its test, and to run tests in a compilation 
buffer. For both purposes, it needs to determine the directory which to 
build the relative paths against, and in which to run the 'rspec' 
program. Currently, it does a locate-dominating-file like routine 
(looking for Gemfile or Rakefile), but it could switch to the project 
API in the future.

You're probably aware of similar tools.

So, now we have both project-root and project-directories. However, the 
relationship between them looks murky, from the consumer's standpoint. 
Like, must the former always be included in the latter?

I'm considering removing project-root, and copying a part of Atom's API 
[0]: adding a function project-relativize-path, which will, given an 
arbitrary directory, if it's inside project-directories, split its file 
name in two and return a cons, with one of project-directories in the 
car, and the relative file name against it in the cdr.

Then instead of project-root, a consumer will use (car 
(project-relativize-path default-directory)).

Thoughts?

[0] https://atom.io/docs/api/v1.0.2/Project#instance-relativizePath

Re: Unified project interface

[Stephen Leake]

Dmitry Gutov <dgutov@yandex.ru> writes:

> Let's maybe continue this discussion?
>
> On 06/05/2015 01:08 PM, Stephen Leake wrote:
>
>> No, xref-find-regexp should search project-source-directories.
>>
>> Most real projects include other projects, so limiting the search to
>> only the top project is wrong in general (although that might be a
>> useful option in some use cases).
>
> I suppose, and maybe I was wrong to suggest a distinction between
> "source" directories and the rest of them. However, here's one which
> could be better: directories to search vs. directories to modify.
>
> Like, I might want to search the full system includes path, in a C
> project, but when performing a replacement operation (such as
> xref-query-replace), I should probably limit myself to the directories
> with code I actually own. 

Yes, that's a good use case.

> At the moment, it's the distinction I
> imagine for project-search-path vs project-directories.

I would suggest the terminology "main project" vs "included projects"
for this; the "main project" is the code you own; the "included
projects" are all the others (included the system includes).

So this would be an argument to xref-find-regexp;

(defun xref-find-regexp (regexp &optional not-included-projects)
"Search all files in all project directories for REGEXP.
If NOT-INCLUDED-PROJECTS is non-nil (default nil), search only the main
project. Otherwise, search the main project and included projects."

To implement this, project-search-path might want to be an alist of:

(PROJECT-NAME <search-path>)

where the first element is the main project and the rest are the
included ones.

>> In addition, there might be a directory under project root that should
>> _not_ be searched; the object file directory for ada-mode, for example.
>
> We have that in project-ignores now.

Right. That's not consistent with current things like load-path. I
should think at some point in the implementation of any "search project"
function you will want an explicit list of directories (and maybe files)
to search, which will be the union of project-search-path minus
project-ignores. So it seems simpler to specify that list directly.

The user interface (the syntax of the project file) could be structured
as includes and ignores (excludes).

>> We can make [root] a requirement in order to use the general tool. But first
>> we have to justify it; ada-mode has never needed that notion; neither
>> has elisp. In my experience, only config management needs it.
>> ...
>> It also points out that "project root" is poorly defined; I think that
>> is because it's not at all clear when and why we need it.
>
> Here's the first use case I know of:
>
> There's a third-party project called rspec-mode. It allows one to
> switch between a source file and its test, and to run tests in a
> compilation buffer. For both purposes, it needs to determine the
> directory which to build the relative paths against, and in which to
> run the 'rspec' program. Currently, it does a locate-dominating-file
> like routine (looking for Gemfile or Rakefile), but it could switch to
> the project API in the future.

Ok, makes sense.

> You're probably aware of similar tools.

monotone, git, subversion ... :).

Recently I've been playing with Google's Android Studio (shudder; only
to make Emacs do what it does, I promise :). It defines a project root
by a dominating file.

On the other hand, AdaCore GPS (an Ada-specific IDE) defines a project
by a project file that specifies various lists of directories (source,
object, executable); there is no root directory. The project file itself
can be anywhere, it doesn't have to be in one of the listed directories.
That's what Emacs Ada mode does as well.

> So, now we have both project-root and project-directories. However,
> the relationship between them looks murky, from the consumer's
> standpoint. Like, must the former always be included in the latter?

I think they are disjoint concepts. Nothing wrong with keeping both;
some tools will need them, some won't.

> I'm considering removing project-root, and copying a part of Atom's
> API [0]: adding a function project-relativize-path, which will, given
> an arbitrary directory, if it's inside project-directories, split its
> file name in two and return a cons, with one of project-directories in
> the car, and the relative file name against it in the cdr.

If project-directories contains all directories (not just root ones),
this is just (file-name-directory path) . (file-name-non-directory
path).

> Then instead of project-root, a consumer will use (car
> (project-relativize-path default-directory)).

I don't follow; that means the value of 'root directory' changes
depending on where the current file is.

Unless project-directories contains only one directory, which would then
be the root.

Given the variable project-root, it makes sense to define
project-relativize-path to return the portion of the path that is
relative to project-root, and error if there is no such portion (ie an
absolute path outside the root tree). Monotone has such a function in
the C++ code.

-- 
-- Stephe

Re: Unified project interface

[Eli Zaretskii]

> From: Stephen Leake <stephen_leake@stephe-leake.org>
> Date: Fri, 24 Jul 2015 19:55:09 -0500
> 
> I would suggest the terminology "main project" vs "included projects"
> for this; the "main project" is the code you own; the "included
> projects" are all the others (included the system includes).

IMO, system header files are not the same as "included projects",
because the system headers are immutable.

Re: Unified project interface

[Dmitry Gutov]

Hi Stephen,

Sorry for the delay; I had to think on it.

On 07/25/2015 03:55 AM, Stephen Leake wrote:

> I would suggest the terminology "main project" vs "included projects"
> for this; the "main project" is the code you own; the "included
> projects" are all the others (included the system includes).

There's sense in this, but is "an included project you don't own, which 
is not in system includes" a significant use case?

In my own experience, either the projects on which the current one 
depends are installed in system directories, and thus are 
indistinguishable from system includes (such as Ruby gems, installed via 
"gem install"), or the linked project's tree lives near the current one, 
and we can consider it a part of the main project, for the purposes of 
editing.

Because even if the second project is a third-party dependency, 
sometimes you want to make changes to it anyway, and the user might get 
miffed if xref-query-replace prohibits that.

> So this would be an argument to xref-find-regexp;
>
> (defun xref-find-regexp (regexp &optional not-included-projects)
> "Search all files in all project directories for REGEXP.
> If NOT-INCLUDED-PROJECTS is non-nil (default nil), search only the main
> project. Otherwise, search the main project and included projects."

It should probably be a new command, because otherwise specifying that 
second argument interactively would be a pain (C-u already makes 
xref-find-regexp ask a bunch of questions).

> To implement this, project-search-path might want to be an alist of:
>
> (PROJECT-NAME <search-path>)
>
> where the first element is the main project and the rest are the
> included ones.

Maybe do it differently?

If we do have linked projects (and the separation between 
main/included), some generic function, probably named differently, would 
return a list of those project instances. Then the caller will ask each 
instance for its root (or the directories).

There's no need for the main project's search-path entry there, 
xref-find-regexp can just use its root (or the directories) as well.

>> We have that in project-ignores now.
>
> Right. That's not consistent with current things like load-path. I

Indeed. But maybe supporting ignores for project-search-path is not a 
big priority. There's not much to ignore in /usr/include, for instance. 
The directories in load-path also usually don't contain weird files or 
build artefacts (and when they do, it's not like elisp-search-path 
really can account for those). So maybe a good first approximation would 
be to simply not apply the ignores list to the project-search-path 
elements outside of the project tree.

If that's not good enough, project-ignores can accept a directory as its 
argument (it must be among those returned by project-directories, or 
project-search-path), and return the appropriate ignores list.

This will definitely help if each element in project-directories is a 
separate (e.g. Git) repository, with its own .gitignore.

> should think at some point in the implementation of any "search project"
> function you will want an explicit list of directories (and maybe files)
> to search, which will be the union of project-search-path minus
> project-ignores. So it seems simpler to specify that list directly.

I'm not sure about that. It seems then the implementation will have to 
walk the whole project tree itself, and collect all directories that 
don't match the ignores (because .gitignore syntax allows not-anchored 
entries for directories). And if we're to support ignoring files (and 
obvious use case), it'll become a list of files, which can grow pretty 
big. I think it's better to delegate this job to 'find', as long as that 
doesn't force us to compromise on functionality.

> The user interface (the syntax of the project file) could be structured
> as includes and ignores (excludes).

.gitignore also supports whitelisting entries (which override the 
excludes). I haven't gotten around to writing support for those.

>> You're probably aware of similar tools.
>
> monotone, git, subversion ... :).

Not sure about the other two, but Git doesn't need to be told the 
project root. It can be called from any directory, and works with 
relative paths quite well.

> Recently I've been playing with Google's Android Studio (shudder; only
> to make Emacs do what it does, I promise :). It defines a project root
> by a dominating file.

Of course. Why else one would be playing with it? ;)

In this case, project-directories can return a list with one element.

> On the other hand, AdaCore GPS (an Ada-specific IDE) defines a project
> by a project file that specifies various lists of directories (source,
> object, executable); there is no root directory. The project file itself
> can be anywhere, it doesn't have to be in one of the listed directories.
> That's what Emacs Ada mode does as well.

In this case, it seems, the relevant project implementation will be 
based on user explicitly telling it where the project file lives (by 
calling a particular command, with a prompt), and then the project will 
be enabled globally, for the whole Emacs session.

project-directories then will return some of the directories specified 
in the project file (except the object ones, maybe, if there's nothing 
we can edit there), but probably (?) not the one where the project file 
lies, if it's somewhere separately.

Would there be any significant value in project-root returning the 
directory of the project file?

> I think they are disjoint concepts. Nothing wrong with keeping both;
> some tools will need them, some won't.

That would mean documenting the distinction, so that implementers know 
what to do. I'm increasingly less clear on it.

> If project-directories contains all directories (not just root ones),
> this is just (file-name-directory path) . (file-name-non-directory
> path).

It would be. But I expect project-directories to only include the roots. 
IOW, no project-directories element should be a subdirectory of another.

>> Then instead of project-root, a consumer will use (car
>> (project-relativize-path default-directory)).
>
> I don't follow; that means the value of 'root directory' changes
> depending on where the current file is.

That's correct. If the file is inside the "main" project directory, it 
will use the main project root. If the file is in another directory, 
it's in a "linked" body of code (which would often be called a project 
as well), and its root is used.

Atom has taken this approach: each element in 
atom.project.getDirectories() corresponds to a Git repository. See the 
link I sent previously, and also https://github.com/atom/atom/pull/3691.

This may look wrong for certain configurations (your AdaCore example 
comes to mind), but would the concept of root be even useful in that case?

> Given the variable project-root, it makes sense to define
> project-relativize-path to return the portion of the path that is
> relative to project-root, and error if there is no such portion (ie an
> absolute path outside the root tree). Monotone has such a function in
> the C++ code.

Sure. But if project-root is not among project-directories, I'm not sure 
when, and for what cases, that directory can be used.

Re: Unified project interface

[Dmitry Gutov]

On 07/25/2015 10:29 AM, Eli Zaretskii wrote:

> IMO, system header files are not the same as "included projects",
> because the system headers are immutable.

AFAICT, that's exactly what Stephen meant by "included projects": 
projects, whose code we're not allowed to change.

Re: Unified project interface

[Eli Zaretskii]

> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Sun, 26 Jul 2015 05:12:44 +0300
> Cc: emacs-devel@gnu.org
> 
> On 07/25/2015 10:29 AM, Eli Zaretskii wrote:
> 
> > IMO, system header files are not the same as "included projects",
> > because the system headers are immutable.
> 
> AFAICT, that's exactly what Stephen meant by "included projects": 
> projects, whose code we're not allowed to change.

Then the name is misleading.

Re: Unified project interface

[Stephen Leake]

Dmitry Gutov <dgutov@yandex.ru> writes:

> Hi Stephen,
>
> Sorry for the delay; I had to think on it.
>
> On 07/25/2015 03:55 AM, Stephen Leake wrote:
>
>> I would suggest the terminology "main project" vs "included projects"
>> for this; the "main project" is the code you own; the "included
>> projects" are all the others (included the system includes).
>
> There's sense in this, but is "an included project you don't own,
> which is not in system includes" a significant use case?

Yes; any third party library that you should not edit, because the
sources are controlled upstream.

> In my own experience, either the projects on which the current one
> depends are installed in system directories, and thus are
> indistinguishable from system includes (such as Ruby gems, installed
> via "gem install"), 

Ok. I don't see the notion of "system directories" as important here;
the important point is whether the libraries are read-only or
read/write. We should not rely on where they happen to reside to decide
that; sometimes I edit "system directories" when working on patches for
upstream. 

Or just ignore the read-only issue in the project code; file permissions
are good enough.

>> should think at some point in the implementation of any "search project"
>> function you will want an explicit list of directories (and maybe files)
>> to search, which will be the union of project-search-path minus
>> project-ignores. So it seems simpler to specify that list directly.
>
> I'm not sure about that. It seems then the implementation will have to
> walk the whole project tree itself, and collect all directories that
> don't match the ignores (because .gitignore syntax allows not-anchored
> entries for directories). 

Yes. You do that once when the project is opened, and cache the result
in the variable project-search-path. Then all code that searches
projects is simpler; it deals with a simple list of directories, plus a
list of file extensions (just as locate-file does).

>> The user interface (the syntax of the project file) could be structured
>> as includes and ignores (excludes).
>
> .gitignore also supports whitelisting entries (which override the
> excludes). I haven't gotten around to writing support for those.

Right. .gitignore is part of "the project file" in this sense.

>> Recently I've been playing with Google's Android Studio (shudder; only
>> to make Emacs do what it does, I promise :). It defines a project root
>> by a dominating file.
>
> Of course. Why else one would be playing with it? ;)
>
> In this case, project-directories can return a list with one element.

No, Emacs needs to read the Studio project file and extract the search
path; it's just another project file syntax.

>> On the other hand, AdaCore GPS (an Ada-specific IDE) defines a project
>> by a project file that specifies various lists of directories (source,
>> object, executable); there is no root directory. The project file itself
>> can be anywhere, it doesn't have to be in one of the listed directories.
>> That's what Emacs Ada mode does as well.
>
> In this case, it seems, the relevant project implementation will be
> based on user explicitly telling it where the project file lives (by
> calling a particular command, with a prompt), and then the project
> will be enabled globally, for the whole Emacs session.

Yes. No different from any other Emacs project; the user has to tell
Emacs where it is. If that's a directory, Emacs searches for project
file extensions it knows about, and reads that. If it's a file, no
searching needed.

> Would there be any significant value in project-root returning the
> directory of the project file?

Emacs should trust what the project file says; it has an explicit list of
"source code directories". Emacs should not care whether that contains
the project file itself or not; that's up to the user. 

Since the external tools that use the project file can report errors in
its syntax, I usually include the project file in a directory that is in
the source path. But that's my choice, not Emacs's.

>> I think they are disjoint concepts. Nothing wrong with keeping both;
>> some tools will need them, some won't.
>
> That would mean documenting the distinction, so that implementers know
> what to do. I'm increasingly less clear on it.

project-root; a single absolute directory, usually containing
significant files for external tools (ie .git/), often the root of the
source tree.

    note that the _only_ requirement here is that it is a single
    absolute directory; the rest is just suggestions.

project-search-path; a list of directories, containing files related to
the project. This will normally include project-root. It may also
contain included projects.

    this might want to be split into project-source-search-path,
    project-object-search-path, but since Emacs is mostly interested in
    editing sources, only the source path is interesting.
    
-- 
-- Stephe

Re: Unified project interface

[Stephen Leake]

Dmitry Gutov <dgutov@yandex.ru> writes:

> On 07/25/2015 10:29 AM, Eli Zaretskii wrote:
>
>> IMO, system header files are not the same as "included projects",
>> because the system headers are immutable.
>
> AFAICT, that's exactly what Stephen meant by "included projects":
> projects, whose code we're not allowed to change.

No, _some_ included projects are read-only; others are read-write. I
don't think the distinction is useful at the project level; file
permissions handle it.

-- 
-- Stephe

Re: Unified project interface

[Dmitry Gutov]

On 07/26/2015 02:22 PM, Stephen Leake wrote:

> Yes; any third party library that you should not edit, because the
> sources are controlled upstream.

Is it common to have its sources checked out in a directory nearby?

> Ok. I don't see the notion of "system directories" as important here;
> the important point is whether the libraries are read-only or
> read/write. We should not rely on where they happen to reside to decide
> that; sometimes I edit "system directories" when working on patches for
> upstream.

There is a difference, from my perspective: usually, there are no object 
files in system libraries, no build files, or .gitignore files.

> Or just ignore the read-only issue in the project code; file permissions
> are good enough.

Maybe ignoring that is fine (are we backtracking on the separation of 
the directories for the purposes of search and edit operations?), but 
file permissions won't help in a lot of cases.

In Ruby and Python communities, for instance, it's common to install the 
language runtime, with all libraries, inside the developer's home 
directory (using RVM/rbenv for Ruby, or virtualenv for Python). And I 
might edit some of files there for debugging purposes, but if I forget 
about those edits later, only bad can come of it. Likewise, I usually 
wouldn't want a global replace operation to touch them.

>> I'm not sure about that. It seems then the implementation will have to
>> walk the whole project tree itself, and collect all directories that
>> don't match the ignores (because .gitignore syntax allows not-anchored
>> entries for directories).
>
> Yes. You do that once when the project is opened, and cache the result

And then the user creates a new dir (or switches to a different branch, 
revision, etc), and the cache goes out of sync. How do we handle that?

On the other hand, delegating the scan of the filesystem to 'find' seems 
performant enough. Someone correct me if I'm wrong, but most of the time 
is spent in Grep anyway.

> in the variable project-search-path. Then all code that searches
> projects is simpler; it deals with a simple list of directories, plus a
> list of file extensions (just as locate-file does).

locate-file seems to use a whitelist of suffixes, which is distinctly 
less powerful than with what's offered by .gitignore. I think keeping 
compatibility with it (and similar .bzrignore and .hgignore) is valuable.

> No, Emacs needs to read the Studio project file and extract the search
> path; it's just another project file syntax.

But according to you, the "search path" will be just the directories 
inside the project's tree, right? That can be determined solely on the 
location of that file.

What will "included projects" look like, in an average Android project?

> Yes. No different from any other Emacs project; the user has to tell
> Emacs where it is. If that's a directory, Emacs searches for project
> file extensions it knows about, and reads that. If it's a file, no
> searching needed.

The VC project implementation is currently based on the current buffer's 
directory. No need to explicitly specify it, Emacs find the repo root 
without user intervention.

If you've ever tried Projectile, it works in a similar fashion. Android 
projects could be detected like that as well.

> Emacs should trust what the project file says; it has an explicit list of
> "source code directories". Emacs should not care whether that contains
> the project file itself or not; that's up to the user.

Will this miss all the other directories in the project, such as 
documentation, maybe? One would expect xref-find-regexp to search them, too.

> Since the external tools that use the project file can report errors in
> its syntax, I usually include the project file in a directory that is in
> the source path. But that's my choice, not Emacs's.

One would expect xref-find-regexp to search in the project file, too.

> project-root; a single absolute directory, usually containing
> significant files for external tools (ie .git/), often the root of the
> source tree.

Would we have project-root mainly for the purpose of xref-find-regexp 
being able to search in the project file?

If project-root is not in project-search-path, can there be other, 
unrelated files in that directory? How would we make sure 
xref-find-regexp ignores them?

Seems like any other tool that would use it would be specific to that 
kind of project (such as project file checker). Maybe the path to the 
project file should be set via metadata, using a key like ada-project-file.

>      note that the _only_ requirement here is that it is a single
>      absolute directory; the rest is just suggestions.

That's a problem, IMO. If project-root is loosely defined, it's not 
useful to third-party code that doesn't know much more about your kind 
of project.

And as per below, we're be including source directories from both the 
current project and the included projects into project-search-path, 
right? Seemingly on equal rights. But then, each of those projects 
likely has a .gitignore, a build file, and so on, but project-root will 
only return the location of project files for the current project? That 
seems inconsistent.

> project-search-path; a list of directories, containing files related to
> the project. This will normally include project-root. It may also
> contain included projects.

So, this tries to combine project-search-path and project-directories, 
as they're defined now? From the previous email, I imagined a 
project-specific project-search-path plus an accessor like 
project-dependencies, which would contain a list of project objects 
corresponding to the included projects.

>      this might want to be split into project-source-search-path,
>      project-object-search-path, but since Emacs is mostly interested in
>      editing sources, only the source path is interesting.

Right. I'd call it project-directories, though.

Re: Unified project interface

[Stephen Leake]

Dmitry Gutov <dgutov@yandex.ru> writes:

> In Ruby and Python communities, for instance, it's common to install
> the language runtime, with all libraries, inside the developer's home
> directory (using RVM/rbenv for Ruby, or virtualenv for Python). And I
> might edit some of files there for debugging purposes, but if I forget
> about those edits later, only bad can come of it. Likewise, I usually
> wouldn't want a global replace operation to touch them.

Then it's not "global", is it? It's "operate on this set of projects,
exclude that set". So you need a way to specify sets of projects.

>>> I'm not sure about that. It seems then the implementation will have to
>>> walk the whole project tree itself, and collect all directories that
>>> don't match the ignores (because .gitignore syntax allows not-anchored
>>> entries for directories).
>>
>> Yes. You do that once when the project is opened, and cache the result
>
> And then the user creates a new dir (or switches to a different
> branch, revision, etc), and the cache goes out of sync. How do we
> handle that?

project-cache-refresh. Only the user knows when it should be done; they
may want to keep the slightly out of date cache because something isn't
finished with the edits yet.

>> No, Emacs needs to read the Studio project file and extract the search
>> path; it's just another project file syntax.
>
> But according to you, the "search path" will be just the directories
> inside the project's tree, right? 

No, I have been explicitly saying that is not the case; the search path
is whatever the project file says it is.

> What will "included projects" look like, in an average Android
> project?

A list of project names. Depending on the project manager, that implies
a list of project files in one syntax or another, each of which
specifies a set of project directories (explicitly or implicitly).

>> Yes. No different from any other Emacs project; the user has to tell
>> Emacs where it is. If that's a directory, Emacs searches for project
>> file extensions it knows about, and reads that. If it's a file, no
>> searching needed.
>
> The VC project implementation is currently based on the current
> buffer's directory. No need to explicitly specify it, Emacs find the
> repo root without user intervention.

Yes.

> If you've ever tried Projectile, it works in a similar fashion.
> Android projects could be detected like that as well.

Maybe. Depends on the project manager software. Some have a dominating
file, some do it some other way.

Why do we care? Just support whatever method the project manger backend
requires. project-find-project-file is a dispatching function on the backend.

>> Emacs should trust what the project file says; it has an explicit list of
>> "source code directories". Emacs should not care whether that contains
>> the project file itself or not; that's up to the user.
>
> Will this miss all the other directories in the project, such as
> documentation, maybe? One would expect xref-find-regexp to search
> them, too.

Trust what the project file says. The user told Emacs to use the project
file; it should not try to be smarter.

If the user wants documentation in the project, then it is in the
project file. If the project manager tool is brain-dead and can't handle
documentation, the the user can switch tools.

>> Since the external tools that use the project file can report errors in
>> its syntax, I usually include the project file in a directory that is in
>> the source path. But that's my choice, not Emacs's.
>
> One would expect xref-find-regexp to search in the project file, too.

Only if I tell it to by including it in the project file. If I tell it
to _not_ search the project file, it should do that.

>> project-root; a single absolute directory, usually containing
>> significant files for external tools (ie .git/), often the root of the
>> source tree.
>
> Would we have project-root mainly for the purpose of xref-find-regexp
> being able to search in the project file?

No. We should have project-root for tools that need a root directory.
xref-find-regexp needs a search path, not a root directory.

> If project-root is not in project-search-path, can there be other,
> unrelated files in that directory? How would we make sure
> xref-find-regexp ignores them?

xref-find-regexp searches project-search-path. project-search-path is
set by reading the project files. The project files are provided by the
user. Voila, xref-find-regexp does _exactly_ what the user wants, no
more, no less.


-- 
-- Stephe

Re: Unified project interface

[John Yates]

[-- Attachment #1: Type: text/plain, Size: 1640 bytes --]

On Sun, Jul 26, 2015 at 2:57 PM, Stephen Leake <
stephen_leake@stephe-leake.org> wrote:

> > And then the user creates a new dir (or switches to a different
> > branch, revision, etc), and the cache goes out of sync. How do we
> > handle that?
>
> project-cache-refresh. Only the user knows when it should be done; they
> may want to keep the slightly out of date cache because something isn't
> finished with the edits yet.


This seems to me to be "rationalizing a zit".  I conjecture that you only
advocate such a UI because you imagine a not-particularly performant
implementation.  If there were no cost to keeping the cache current
(equivalent to stating that a naive user never would _never_ need to learn
of that cache)  would you still advocate such a UI?

In small projects the cost of a simplistic brute-force change determination
is likely to be entirely acceptable.  On systems that support emacs'
file-notify watching every directory within a project should be able to
keep cache refresh overhead to negligible levels (potentially improved by
only watch the modifiable portions of the project).

In the exceedingly rare case where a user needs to keep work hidden from
project searches he can either modify the project definition to blacklist
temporarily the areas he wants excluded.  Alternatively we could provide a
toggle for very advanced users to suppress cache refersh.

Bottom line: start by getting the model of state and the UI right.  After
that if the performance in corner cases needs help then give advanced users
tools to tune the system.  But please do not foist a user-maintained cache
on casual users.

/john

[-- Attachment #2: Type: text/html, Size: 2190 bytes --]

Re: Unified project interface

[Dmitry Gutov]

On 07/27/2015 02:56 AM, John Yates wrote:

> If there were no cost to keeping the cache
> current (equivalent to stating that a naive user never would _never_
> need to learn of that cache)  would you still advocate such a UI?

I think the word "cache" itself implies a storage that can get stale.

> Bottom line: start by getting the model of state and the UI right.
> After that if the performance in corner cases needs help then give
> advanced users tools to tune the system.  But please do not foist a
> user-maintained cache on casual users.

Right. Even if there's a cache mechanism, I'd very much like to see its 
invalidation by automatic means (via file-notify, or similar), at least 
by default.

Anyway, I'd prefer to implement as many operations as feasible without 
needing that cache. If the search through the project tree spends an 
order of magnitude more time searching than listing the files, there's 
not point in caching the latter.

Re: Unified project interface

[Stephen Leake]

John Yates <john@yates-sheets.org> writes:

> On Sun, Jul 26, 2015 at 2:57 PM, Stephen Leake <
> stephen_leake@stephe-leake.org> wrote:
>
>> > And then the user creates a new dir (or switches to a different
>> > branch, revision, etc), and the cache goes out of sync. How do we
>> > handle that?
>>
>> project-cache-refresh. Only the user knows when it should be done; they
>> may want to keep the slightly out of date cache because something isn't
>> finished with the edits yet.
>
>
> This seems to me to be "rationalizing a zit".  I conjecture that you only
> advocate such a UI because you imagine a not-particularly performant
> implementation.  If there were no cost to keeping the cache current
> (equivalent to stating that a naive user never would _never_ need to learn
> of that cache)  would you still advocate such a UI?

The use case is this:

You have checked out a fully working workspace, in order to do some
refactoring.

Emacs reads the project file and caches everything. In particular,
xref-find-definition works nicely.

Now you start to make changes. Some of the files have incorrect syntax,
or don't compile because of undefined names.

If you attempt to refresh the cache at this point, you will lose _all_
xref, because of the syntax and name errors.

On the other hand, the existing cache allows most xref commands to still
work. So you keep it for a while.


This assumes the mechanism that refreshes the cache is sensitive to such
issues. This is true for the Ada xref backend; it uses the xref info
output by the compiler, so files must compile. If the file fails to
compile for any reason, there is no xref info available.

It is also true that the user time penalty for refreshing the cache is
significant for Ada mode, and grows with the size of the project.

> In small projects the cost of a simplistic brute-force change determination
> is likely to be entirely acceptable.  

Yes, that's true, especially if the cache builder handles syntax errors
gracefully, and is very fast. So that should be one option.

> In the exceedingly rare case where a user needs to keep work hidden from
> project searches he can either modify the project definition to blacklist
> temporarily the areas he wants excluded.  

That's a different use case than I'm talking about, and one I've never
run across; can you elaborate?

I guess that's one way to implement "find-replace in this subset of the
project files".

> Alternatively we could provide a toggle for very advanced users to
> suppress cache refresh.

Yes, that's what I'd like.

> Bottom line: start by getting the model of state and the UI right.  After
> that if the performance in corner cases needs help then give advanced users
> tools to tune the system.  But please do not foist a user-maintained cache
> on casual users.

Yes, good approach.

-- 
-- Stephe

Re: Unified project interface

[Dmitry Gutov]

On 07/27/2015 02:12 PM, Stephen Leake wrote:

> Emacs reads the project file and caches everything. In particular,
> xref-find-definition works nicely.
>
> Now you start to make changes. Some of the files have incorrect syntax,
> or don't compile because of undefined names.
>
> If you attempt to refresh the cache at this point, you will lose _all_
> xref, because of the syntax and name errors.

I thought you were talking about caching the list of files, not their 
contents? The former won't help you with keeping xref-find-definitions 
working.

The latter is a whole different (complex) discussion, largely unrelated 
to the project API. We've touched on it in the thread about hidden buffers.

> On the other hand, the existing cache allows most xref commands to still
> work. So you keep it for a while.

You may be thinking of some cached data somewhere in the implementation 
of the xref backend. That's a valid approach, but neither the project 
API, nor xref API should know anything about it. It's an implementation 
detail.

> This assumes the mechanism that refreshes the cache is sensitive to such
> issues. This is true for the Ada xref backend; it uses the xref info
> output by the compiler, so files must compile. If the file fails to
> compile for any reason, there is no xref info available.

You can keep the xref buffer with the results of the last search, for a 
while, but as you continue editing files, those entries will also begin 
to point at wrong locations.

The exact point at which they stop working will depend on the specific 
implementation of xref-location-marker, though. xref-etags-location is 
designed to be quite resilient, for instance.

Re: Unified project interface

[Dmitry Gutov]

On 07/26/2015 09:57 PM, Stephen Leake wrote:

> Then it's not "global", is it? It's "operate on this set of projects,
> exclude that set". So you need a way to specify sets of projects.

Maybe the "global" is not the right word, but I'm quite confident that 
if I ask an average user what a "search across project" action should 
do, it would replace in the current project, (maybe) in the included 
ones, and definitely not in the installed libraries. Even if they're 
writable.

Even if we add a "choose which projects to touch" interface, it's good 
to have a reasonable default.

>> But according to you, the "search path" will be just the directories
>> inside the project's tree, right?
>
> No, I have been explicitly saying that is not the case; the search path
> is whatever the project file says it is.

Sorry, I've replied to that part before reading the rest, and then 
forgot about it.

> A list of project names. Depending on the project manager, that implies
> a list of project files in one syntax or another, each of which
> specifies a set of project directories (explicitly or implicitly).

Could you give an example?

Since they're referred only by names, this sounds like a list of 
dependencies on the installed packages. Those we usually don't want to 
edit, and I'd classify them as system includes.

> Trust what the project file says. The user told Emacs to use the project
> file; it should not try to be smarter.

So, xref-find-regexp won't call project-root. Got it.

> If the user wants documentation in the project, then it is in the
> project file. If the project manager tool is brain-dead and can't handle
> documentation, the the user can switch tools.

In that case, will the user add the documentation directory to the 
search-path section of the project file?

> No. We should have project-root for tools that need a root directory.
> xref-find-regexp needs a search path, not a root directory.

It could include the root directory in the search path, if it's not 
already there. But it seems you don't want to do that. Okay.

> xref-find-regexp searches project-search-path. project-search-path is
> set by reading the project files. The project files are provided by the
> user. Voila, xref-find-regexp does _exactly_ what the user wants, no
> more, no less.

Very well.

To sum up, I see a strong case for project-search-path as you described 
it (maybe under a different name; like project-directories, as it is 
now), and so far, no real case for project-root.

Further, the system search path should be separate, because it's usually 
specified like that in project files already, in my experience.

And if someone were to create, say, an ECB panel showing the current 
project layout, they might appreciate having the information about these 
directories being different.

If you (or someone else) don't like using "project search path" for the 
system libraries path, feel free to suggest another name.

Re: Unified project interface

[Dmitry Gutov]

On 07/27/2015 04:00 PM, Dmitry Gutov wrote:

> Maybe "global" is not the right word, but I'm quite confident that
> if I ask an average user what a "search across project" action should
                                    ^ replace

Re: Unified project interface

[Stephen Leake]

Dmitry Gutov <dgutov@yandex.ru> writes:

> On 07/26/2015 09:57 PM, Stephen Leake wrote:
>
>> A list of project names. Depending on the project manager, that implies
>> a list of project files in one syntax or another, each of which
>> specifies a set of project directories (explicitly or implicitly).
>
> Could you give an example?

Ada, gradle.

> Since they're referred only by names, this sounds like a list of
> dependencies on the installed packages. Those we usually don't want to
> edit, and I'd classify them as system includes.

I don't know who this "we" is, but I usually structure a large project
as a main with several lower level libraries, all of which I maintain,
and I edit them all together to implement new functionality in main. So
that has to be a choice.

>> Trust what the project file says. The user told Emacs to use the project
>> file; it should not try to be smarter.
>
> So, xref-find-regexp won't call project-root. Got it.

Actually, 'xref-find-regexp' should be named 'project-find-regexp' (or
prj-find-regexp). Keep 'xref' for strictly cross-reference stuff;
'project' is for more general project stuff.

Either that, or drop 'project' and just call it all 'xref'.

>> If the user wants documentation in the project, then it is in the
>> project file. If the project manager tool is brain-dead and can't handle
>> documentation, the the user can switch tools.
>
> In that case, will the user add the documentation directory to the
> search-path section of the project file?

Yes, except most backend project files don't have things called
"search-path", since they are not for editors; it would be
"source-path".

> To sum up, I see a strong case for project-search-path as you
> described it (maybe under a different name; like project-directories,
> as it is now), and so far, no real case for project-root.

Ok, we are converging.

-- 
-- Stephe

Re: Unified project interface

[Stephen Leake]

Stephen Leake <stephen_leake@stephe-leake.org> writes:

> Dmitry Gutov <dgutov@yandex.ru> writes:
>
>> On 07/26/2015 09:57 PM, Stephen Leake wrote:
>>
>>> A list of project names. Depending on the project manager, that implies
>>> a list of project files in one syntax or another, each of which
>>> specifies a set of project directories (explicitly or implicitly).
>>
>> Could you give an example?
>
> Ada, gradle.
>
>> Since they're referred only by names, this sounds like a list of
>> dependencies on the installed packages. Those we usually don't want to
>> edit, and I'd classify them as system includes.
>
> I don't know who this "we" is, but I usually structure a large project
> as a main with several lower level libraries, all of which I maintain,
> and I edit them all together to implement new functionality in main. So
> that has to be a choice.
>
>>> Trust what the project file says. The user told Emacs to use the project
>>> file; it should not try to be smarter.
>>
>> So, xref-find-regexp won't call project-root. Got it.
>
> Actually, 'xref-find-regexp' should be named 'project-find-regexp' (or
> prj-find-regexp). Keep 'xref' for strictly cross-reference stuff;
> 'project' is for more general project stuff.
>
> Either that, or drop 'project' and just call it all 'xref'.
>
>>> If the user wants documentation in the project, then it is in the
>>> project file. If the project manager tool is brain-dead and can't handle
>>> documentation, the the user can switch tools.
>>
>> In that case, will the user add the documentation directory to the
>> search-path section of the project file?
>
> Yes, except most backend project files don't have things called
> "search-path", since they are not for editors; it would be
> "source-path".
>
>> To sum up, I see a strong case for project-search-path as you
>> described it (maybe under a different name; like project-directories,
>> as it is now), and so far, no real case for project-root.
>
> Ok, we are converging.

I just realized project.el is in emacs master; I thought it was only in
the experimental branch.

So some specific change proposals:

- Rename 'project-directories' to 'project-root-directories' or
  'project-roots'. The current project root should always be first in
  the list.

- 'project-search-path' should not include 'project-root-directories'.

- 'elisp-search-path' should not include 'package-user-dir';
  package-user-dir is not an elisp source directory, and load-path
  already includes the subdirs of it, if packages are initialized.
  
-- 
-- Stephe

Re: Unified project interface

[Dmitry Gutov]

On 07/28/2015 04:21 AM, Stephen Leake wrote:

>> Could you give an example?
>
> Ada, gradle.

I meant an example (or several) of those declarations from an actual 
project file.

Here an excerpt from an imaginary Gemfile:

gem 'foo', '~> 1.2.3'
gem 'bar', path: '../bar'

Having the latter kind of declaration might not be considered the best 
idea, but it's often nice to be able to do that. Anyway, I'm sure there 
are corresponding configurations in other ecosystems that are more 
idiomatic.

> I don't know who this "we" is, but I usually structure a large project
> as a main with several lower level libraries, all of which I maintain,
> and I edit them all together to implement new functionality in main. So
> that has to be a choice.

IME, aside from a few libraries one (person/team/company) maintains 
themselves, there's also a long list of external dependencies.

In the example above, 'bar' will be one of project-directory-roots, and 
'foo' will only be in project-search-path.

> Actually, 'xref-find-regexp' should be named 'project-find-regexp' (or
> prj-find-regexp). Keep 'xref' for strictly cross-reference stuff;

Sounds good to me. I'll make that change as soon as xref has a suitable 
public interface for that.

> 'project' is for more general project stuff.

It's also used in the xref-find-references implementations in elisp-mode 
and etags, which I'm inclined to converge into one default 
implementation that uses the info from the project package. I think 
that's fine.

> Either that, or drop 'project' and just call it all 'xref'.

project is foremost intended to be used for third-party code. It's a 
feature that's been requested for a while.

Re: Unified project interface

[Dmitry Gutov]

On 07/28/2015 02:05 PM, Stephen Leake wrote:

> I just realized project.el is in emacs master; I thought it was only in
> the experimental branch.

It would be cleaner to have it separate, but being able to dogfood it 
now is quite handy.

> So some specific change proposals:
>
> - Rename 'project-directories' to 'project-root-directories' or
>    'project-roots'.

Either is fine with me, but note that y having "root" in the name we 
give up on adding a special "project-root" function in the future, one 
that you described earlier.

 > The current project root should always be first in the list.

> - 'project-search-path' should not include 'project-root-directories'.

I say we document it like that (or say that it's "allowed not to 
include"), but the consumers will still take care of the duplication 
(call project--prune-directories). That's easier on the implementations.

> - 'elisp-search-path' should not include 'package-user-dir';
>    package-user-dir is not an elisp source directory, and load-path
>    already includes the subdirs of it, if packages are initialized.

What's the harm? It's an optimization: this way, we only call 'find' 
once per elpa dir, instead of doing it for each installed package. Not 
sure what's the overhead on that, but it seems wasteful.

Re: Unified project interface

[Stephen Leake]

Dmitry Gutov <dgutov@yandex.ru> writes:

> On 07/28/2015 02:05 PM, Stephen Leake wrote:
>
>> - Rename 'project-directories' to 'project-root-directories' or
>>    'project-roots'.
>
> Either is fine with me, but note that y having "root" in the name we
> give up on adding a special "project-root" function in the future, one
> that you described earlier.

No, I'm happy with this:

'project-root'; find the root directory of the current project.

'project-roots'; return a list of related project roots.

That's what renaming 'project-directories' to 'project-roots' gives us.
Although some will object to the small name difference between
'project-root' and 'project-roots'.

>> - 'project-search-path' should not include 'project-root-directories'.
>
> I say we document it like that (or say that it's "allowed not to
> include"), but the consumers will still take care of the duplication
> (call project--prune-directories). That's easier on the
> implementations.

Are you agreeing that the default implementation should not include
project-root-directories?

The point is that if you only customize by setting
project-search-path-function, you can't exclude project-directories.

And I thought we agreed that project-search-path is disjoint from
project-directories.

>> - 'elisp-search-path' should not include 'package-user-dir';
>>    package-user-dir is not an elisp source directory, and load-path
>>    already includes the subdirs of it, if packages are initialized.
>
> What's the harm? It's an optimization: this way, we only call 'find'
> once per elpa dir, instead of doing it for each installed package. Not
> sure what's the overhead on that, but it seems wasteful.

As with all such optimizations, it can only be justified by actual
measurement.

But I missed the fact that the default project-search-dirs uses
'project--prune-directories' to throw away the subdirs; so while
'elisp-search-dirs' contains redundant directories,
'project-search-dirs' does not.

I find it confusing mixing two customization mechanisms; I started
another thread on that.

It would be less confusing if the default definition of
project-search-path did _not_ call project--prune-directories, but
elisp-search-path _did_. Then project--prune-directories should be
renamed without the '--'.

The doc string for project-search-path says "source directories", not
"source root directories". That needs to be fixed.


--
-- Stephe

Re: Unified project interface

[Stephen Leake]

Dmitry Gutov <dgutov@yandex.ru> writes:

> On 07/28/2015 04:21 AM, Stephen Leake wrote:
>
>>> Could you give an example?
>>
>> Ada, gradle.

> I meant an example (or several) of those declarations from an actual
> project file.

From an Ada project file:

aggregate project Books_Agg is
   for Project_Path use
     ("../../../org.stephe_leake.sal/build/release",
      "../../../org.stephe_leake.gtk/build/release",
      "../../../org.stephe_leake.makerules");

   for Project_Files use ("books.gpr");

end Books_Agg;

with "gtkada";
with "gnatcoll_sqlite";
with "sal";
with "sal_gtk";
with "standard_common";
project Books is
   for Source_Dirs use ("../../source", "../../test");

...

end Books;


Project_Path says where to find projects (there is also an implicit
system location); 'with "...";' says what projects to include. In this
case, "gtkada" and "gnatcoll_sqlite" are in the system location, and are
read only; sal, sal_gtk and standard_common read/write. But the Ada
project tool (as provided by the compiler vendor) makes no distinction
between "system" and "user" libraries; more precisely, it relies on the
file system read/write permissions to do that.


From an Android gradle file:

dependencies {

    compile project(':licensesdialoglibrary')
    compile project(':quickScroll')
    compile project(':velocityviewpagerlibrary')
    compile project(':dragsortlistviewlibrary')
    compile project(':viewpagerindicatorlibrary')
    compile project(':circularImageView')
    compile project(':picasso')
    compile files('libs/android-async-http-1.4.2-66-g4b6eb97.jar')
    compile files('libs/commons-io-2.4.jar')
    compile files('libs/commons-lang3-3.1.jar')
    compile files('libs/commons-logging.jar')
    compile files('libs/dashclock-api-r2.0.jar')
    compile files('libs/google-http-client-1.16.0-rc.jar')
    compile files('libs/google-http-client-android-1.16.0-rc.jar')
    compile files('libs/jaudiotagger-2.0.4-20111207.115108-15.jar')
    compile files('libs/libGoogleAnalyticsServices.jar')
    compile files('libs/universal-image-loader-1.9.3-with-sources.jar')
    compile files('libs/com.haarman.listviewanimations-2.6.0.jar')
    compile files('libs/nineoldandroids-2.4.0.jar')
    compile files('libs/renderscript-v8.jar')
    compile 'com.android.support:support-v4:+'
    compile 'com.google.android.gms:play-services:+'
}

Some of these are read/write, some are read-only; gradle does not have a
mechanism to indicate that.


> Here an excerpt from an imaginary Gemfile:
>
> gem 'foo', '~> 1.2.3'
> gem 'bar', path: '../bar'
>
> Having the latter kind of declaration might not be considered the best
> idea, but it's often nice to be able to do that. Anyway, I'm sure
> there are corresponding configurations in other ecosystems that are
> more idiomatic.

The point is that the abstract project API must handle all of these. In
the case of Ada, there is a command-line tool that parses the project
file and provides the info that Emacs needs.

>> I don't know who this "we" is, but I usually structure a large project
>> as a main with several lower level libraries, all of which I maintain,
>> and I edit them all together to implement new functionality in main. So
>> that has to be a choice.
>
> IME, aside from a few libraries one (person/team/company) maintains
> themselves, there's also a long list of external dependencies.

Yes, we are agreeing; some dependencies are read/write, some are read-only.

How many of each, and whether they are "system" or not, are minor
issues.

> In the example above, 'bar' will be one of project-directory-roots,
> and 'foo' will only be in project-search-path.

Why? What effect does that have on what you can do with 'foo' vs 'bar'?

How can you tell that just from the gem file?

I'm guessing you are saying "foo is maintained by the same team as the
main project (ie read/write), bar is maintained elsewhere (ie
read-only)".

So then files in project-directories are read-only, files in
project-search-path are read/write.

But the current default implementation of project-search-path _inludes_
project-directories, so this is inconsistent.

And we've said that project-directories includes the main project root,
so that's also inconsistent.

So I don't understand what you are doing here.

>> Actually, 'xref-find-regexp' should be named 'project-find-regexp' (or
>> prj-find-regexp). Keep 'xref' for strictly cross-reference stuff;
>
> Sounds good to me. I'll make that change as soon as xref has a
> suitable public interface for that.

I don't understand; this requires deleting things from xref, not adding
to it.

The current implementation of xref-find-regexp uses project-current,
xref-collect-matches, and xref--show-xrefs. 

If you make the implementation of project-find-regexp use xref
facilities, then I understand your comment. I was hoping for more
separation.

I suggest renaming 'xref--show-xrefs' to 'project-show-locations', and
have it take a list of locations, not implicitly call
xref-find-function. Also move all of the location stuff to project; it's
more general than xref.

xref-collect-matches uses grep; it can simply be moved to
project-collect-matches, along with the xref-*grep functions it uses.
The doc string needs to mention subdirs.

-- 
-- Stephe

Re: Unified project interface

[Dmitry Gutov]

On 07/28/2015 06:45 PM, Stephen Leake wrote:

> 'project-roots'; return a list of related project roots.
>
> That's what renaming 'project-directories' to 'project-roots' gives us.
> Although some will object to the small name difference between
> 'project-root' and 'project-roots'.

If project-roots does not include project-root (so we limit the notion 
of "current project" to just one directory tree), it would be better to 
rename that to project-related-projects, and return project instances in 
there.

This way, the consumer can inquire about each related project's 
dependencies, as well as call project-ignores on each of them.

But that becomes more complex, because then you'd expect the consumer to 
call project-related-projects on each related project ask well, and 
traverse the whole tree, handle loops, etc.

I think it'll be better to incorporate all related projects into the 
notion of the current project, return all project roots on equal rights, 
give project-ignores an argument (one of the roots), and introduce 
project-relativize which, like described previously, will return (ROOT . 
RELATIVE-PATH).

In this scheme, project-root is not needed, because we want to treat all 
roots somewhat equally, and also because project-root shouldn't use 
default-directory, or buffer-file-name.

> Are you agreeing that the default implementation should not include
> project-root-directories?

Yes.

> The point is that if you only customize by setting
> project-search-path-function, you can't exclude project-directories.
>
> And I thought we agreed that project-search-path is disjoint from
> project-directories.

Like I said, "allowed not to include". Not matter if emacs-lisp-mode 
sets project-search-path-function, or provides a specialized 
implementation of project-search-path, it doesn't know the current 
project roots, so it can't exclude them.

> As with all such optimizations, it can only be justified by actual
> measurement.

It's an easy one, though.

> It would be less confusing if the default definition of
> project-search-path did _not_ call project--prune-directories, but
> elisp-search-path _did_.

Some caller up the stack will have to call project--prune-directories 
anyway, because we don't want to guarantee that search-path and 
project-roots don't mix (see above). However, we could provide a public 
function in project.el that will combine both. I having hard time coming 
up with the name, though: project-full-search-path doesn't sound too good.

 > Then project--prune-directories should be> renamed without the '--'.

I've been waiting for a better name: there are several 
-prune-directories function in Emacs already, in different packages, all 
with different semantics.

> The doc string for project-search-path says "source directories", not
> "source root directories". That needs to be fixed.

All right. To me, the two sound equivalent, and I meant it like that.

Re: Unified project interface

[Dmitry Gutov]

On 07/28/2015 07:15 PM, Stephen Leake wrote:

>  From an Ada project file:

Thank you.

> Project_Path says where to find projects (there is also an implicit
> system location); 'with "...";' says what projects to include. In this
> case, "gtkada" and "gnatcoll_sqlite" are in the system location, and are
> read only; sal, sal_gtk and standard_common read/write. But the Ada
> project tool (as provided by the compiler vendor) makes no distinction
> between "system" and "user" libraries; more precisely, it relies on the
> file system read/write permissions to do that.

That's unfortunate. But you do specify "custom" locations for sal, 
sal_gtk and standard_common in the beginning, right? Then a smart enough 
tool could learn that information.

In the meantime, I guess, the Ada projects will have to classify all 
dependencies together as search-path.

>
>  From an Android gradle file:
>
> dependencies {
>
>      compile project(':licensesdialoglibrary')
>      compile files('libs/android-async-http-1.4.2-66-g4b6eb97.jar')
>      compile 'com.google.android.gms:play-services:+'
> }
>
> Some of these are read/write, some are read-only; gradle does not have a
> mechanism to indicate that.

jars are obviously build artefacts. Doesn't the difference between 
compile '...' and compile project('...') signify something, though?

>> Here an excerpt from an imaginary Gemfile:
>>
>> gem 'foo', '~> 1.2.3'
>> gem 'bar', path: '../bar'
>>
> The point is that the abstract project API must handle all of these. In
> the case of Ada, there is a command-line tool that parses the project
> file and provides the info that Emacs needs.

I'm assuming that the build tools in other ecosystems can likewise 
separate dependencies into search-path and full-on projects. Or, at 
least, that most of them can, and this separation is meaningful.

And that there's no separation along different lines that's more useful 
than that.

> Yes, we are agreeing; some dependencies are read/write, some are read-only.
>
> How many of each, and whether they are "system" or not, are minor
> issues.

Actually, neither "system", nor "read/write" terminology strikes me as 
optimal. "search path" vs. "projects" sounds better: one could edit a 
file inside the system search path, with purpose, and we're should allow 
that operation, but not optimize the interface for it.

>> In the example above, 'bar' will be one of project-directory-roots,
>> and 'foo' will only be in project-search-path.
>
> Why? What effect does that have on what you can do with 'foo' vs 'bar'?
>
> How can you tell that just from the gem file?

path: '/..' says that 'bar' is a project residing in a nearby directory, 
and it's probably version-controlled. The fact that I have a Git 
checkout of 'bar' is a strong indicator that I'm maintaining that 
project (though not a guarantee, of course).

It makes sense to perform read/write operations on version-controlled 
directories.

'foo', on the other hand, resides in 
~/.rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/foo-1.2.3/, and any 
edits I make to it are hard to track or push to the development repo.

> I'm guessing you are saying "foo is maintained by the same team as the
> main project (ie read/write), bar is maintained elsewhere (ie
> read-only)".

The opposite of that.

> So then files in project-directories are read-only, files in
> project-search-path are read/write.

Ditto.

> But the current default implementation of project-search-path _inludes_
> project-directories, so this is inconsistent.

Is allowed to include.

> And we've said that project-directories includes the main project root,
> so that's also inconsistent.

Why?

> So I don't understand what you are doing here.

read/write operations will use project-roots; read-only operations will 
use project-roots combined with project-search-path. I though we've 
already settled on that.

> I don't understand; this requires deleting things from xref, not adding
> to it.

xref-find-regexp uses things from xref, such as the output buffer, its 
major mode, etc. They're not exposed publicly in a proper fashion, yet.

> The current implementation of xref-find-regexp uses project-current,
> xref-collect-matches, and xref--show-xrefs.

Right. So it uses both xref.el and project.el.

> If you make the implementation of project-find-regexp use xref
> facilities, then I understand your comment.

Exactly.

> I was hoping for more separation.

What kind?

> I suggest renaming 'xref--show-xrefs' to 'project-show-locations', and

Why project-? The procedure of displaying the locations doesn't have 
much to do with the project API. The xref API, however, is very much 
related.

> have it take a list of locations, not implicitly call
> xref-find-function.

The difficulty with that had been the necessity to track the buffers 
that xref-find-function creates, in order to be able to delete each of 
them that haven't been used otherwise, in xref-quit.

The parallel thread about hidden/temporary buffers should result in 
solving that in a different way, and then I'll do just that.

> xref-collect-matches uses grep; it can simply be moved to
> project-collect-matches, along with the xref-*grep functions it uses.
> The doc string needs to mention subdirs.

I think there's value in providing function, in xref, that returns a 
list of matches for a given regexp in a given directory, as xref-items. 
It's not, by itself, project-related, and other xref backends might use 
it for cheap xref-find-references implementations.

Re: Unified project interface

[Stephen Leake]

Dmitry Gutov <dgutov@yandex.ru> writes:

> On 07/28/2015 06:45 PM, Stephen Leake wrote:
>
> I think it'll be better to incorporate all related projects into the
> notion of the current project, return all project roots on equal
> rights, 

That is certainly what Ada mode does, and gradle.

> give project-ignores an argument (one of the roots), and
> introduce project-relativize which, like described previously, will
> return (ROOT . RELATIVE-PATH).
>
> In this scheme, project-root is not needed, because we want to treat
> all roots somewhat equally, and also because project-root shouldn't
> use default-directory, or buffer-file-name.

So you would keep project-roots, but not project-root. I can live with
that (Ada mode needs neither :).

> Not matter if emacs-lisp-mode
> sets project-search-path-function, or provides a specialized
> implementation of project-search-path, it doesn't know the current
> project roots, so it can't exclude them.

How will the elisp backend define project-roots?

Since it doesn't know anything about project-roots, it should either
throw an error, or return nil.

On the other hand, (project-search-path (current-project)) in a .el
buffer on my machine returns:

("c:/Projects/emacs/master-build-mingw64/lisp/"
 "c:/Projects/emacs/master/"
 "c:/Projects/emacs_stephe.main/emacs_stephe/"
 "c:/Projects/emacs_stephe.main/emacs_stephe_site_lisp/"
 "c:/Projects/org.emacs.ada-mode/"
 "c:/Projects/org.emacs.dvc/lisp/"
 "c:/home/stephe/.emacs.d/elpa/")

Those are all 'project-roots' as far as I can tell; they each represent
a logically and physically separate collection of elisp source files
(except that master-build-mingw64/lisp/ actually has no elisp source files).

So for elisp, it seems we are definining project-search-path to be
project-roots.

Which goes a long way to explaining why this has been so confusing.

On the other hand, "(project-directories (project-current))" returns the
single root that contains the current .el file; the same as
"(project-root (project-current))". Why does project-directories not
return a list?

Currently, the only code that uses project-directories is
xref-find-regexp, and it concats it with project-search-path, so it
confuses the two.

Ada mode has no need of project-roots. It seems elisp also has no need
of project-roots. JDE (Java Development Environment) has no need of
project-roots. 

>> It would be less confusing if the default definition of
>> project-search-path did _not_ call project--prune-directories, but
>> elisp-search-path _did_.
>
> Some caller up the stack will have to call project--prune-directories
> anyway, because we don't want to guarantee that search-path and
> project-roots don't mix (see above). 

The top-level generic API should be agnostic about that; only the
backend knows precisely what project-search-path and project-roots are.
In elisp and Ada, project-roots is simply undefined.

Ada mode wants search-path to be flat, _not_ containing roots. So it
does _not_ what something to call project--prune-directories.

> However, we could provide a
> public function in project.el that will combine both. 

Why?

I still have not seen an actual use case for project-roots; it seems the
simplest thing is to simply drop it.

>> The doc string for project-search-path says "source directories", not
>> "source root directories". That needs to be fixed.
>
> All right. To me, the two sound equivalent, and I meant it like that.

The difference is whether recursion into subdirectories is required (in
this case, allowed); that is always worth documenting; it is normally an
option in search functions (think grep vs grep-find;
semantic-symref-find-references-by-name has a :scope argument).

-- 
-- Stephe

Re: Unified project interface

[Dmitry Gutov]

On 07/29/2015 04:36 AM, Stephen Leake wrote:

> So you would keep project-roots, but not project-root. I can live with
> that (Ada mode needs neither :).

It's fine if some projects define project-roots to nil. Unfortunately, 
as I imagine it, xref-query-replace won't work for them.

Of course, we could add a preference (turned off by default) that 
xref-query-replace walks the matches in the search-path, too, not just 
in project-roots.

> How will the elisp backend define project-roots?

It won't. Creating a project backend for Elisp seems pretty much 
impossible (or rather useless), from where I'm standing.

That's why emacs-lisp-mode sets project-search-path-function instead: 
the VC project backend uses it automatically.

> Since it doesn't know anything about project-roots, it should either
> throw an error, or return nil.

But that's exactly the problem: if I'm working on a third-party Elisp 
project, it has its own repository, there's no guarantee that its root 
directory is added to load-path. If all Elisp code is in the lisp 
subdirectory (and it's in load-path), if I call xref-find-regexp, it 
would miss all files on the level above.

> On the other hand, (project-search-path (current-project)) in a .el
> buffer on my machine returns:
>
> ("c:/Projects/emacs/master-build-mingw64/lisp/"
>   "c:/Projects/emacs/master/"
>   "c:/Projects/emacs_stephe.main/emacs_stephe/"
>   "c:/Projects/emacs_stephe.main/emacs_stephe_site_lisp/"
>   "c:/Projects/org.emacs.ada-mode/"
>   "c:/Projects/org.emacs.dvc/lisp/"
>   "c:/home/stephe/.emacs.d/elpa/")
>
> Those are all 'project-roots' as far as I can tell; they each represent
> a logically and physically separate collection of elisp source files
> (except that master-build-mingw64/lisp/ actually has no elisp source files).

No, these are just search-path. We don't know where the roots of most of 
these "projects" are.

> On the other hand, "(project-directories (project-current))" returns the
> single root that contains the current .el file; the same as
> "(project-root (project-current))". Why does project-directories not
> return a list?

It returns a one-element list.

> Currently, the only code that uses project-directories is
> xref-find-regexp, and it concats it with project-search-path, so it
> confuses the two.

Right. Sorry, that must indeed be confusing. xref-query-replace does not 
discriminate against project-search-path yet.

> Ada mode has no need of project-roots. It seems elisp also has no need
> of project-roots. JDE (Java Development Environment) has no need of
> project-roots.

A common Java project usually has one root: the directory containing its 
project file (build.xml, pom.xml, or so on).

Even so the project implementation could provide a way to open several 
projects together, if explicitly told so by the user. Then it would 
return them all in project-roots.

>> Some caller up the stack will have to call project--prune-directories
>> anyway, because we don't want to guarantee that search-path and
>> project-roots don't mix (see above).
>
> The top-level generic API should be agnostic about that; only the
> backend knows precisely what project-search-path and project-roots are.
> In elisp and Ada, project-roots is simply undefined.

So?

The caller would do what xref-find-regexp does now:

(project--prune-directories
   (nconc
     (project-roots proj)
     (project-search-path proj)))

> Ada mode wants search-path to be flat, _not_ containing roots. So it
> does _not_ what something to call project--prune-directories.

It wouldn't hurt anyway.

>> However, we could provide a
>> public function in project.el that will combine both.
>
> Why?

So that the caller doesn't need to call project--prune-directories itself.

> I still have not seen an actual use case for project-roots; it seems the
> simplest thing is to simply drop it.

I have provided a few, along the way of this discussion.

Re: Unified project interface

[Stephen Leake]

Dmitry Gutov <dgutov@yandex.ru> writes:

> On 07/28/2015 07:15 PM, Stephen Leake wrote:
>
>>  From an Ada project file:
>
> Thank you.
>
>> Project_Path says where to find projects (there is also an implicit
>> system location); 'with "...";' says what projects to include. In this
>> case, "gtkada" and "gnatcoll_sqlite" are in the system location, and are
>> read only; sal, sal_gtk and standard_common read/write. But the Ada
>> project tool (as provided by the compiler vendor) makes no distinction
>> between "system" and "user" libraries; more precisely, it relies on the
>> file system read/write permissions to do that.
>
> That's unfortunate. 

Why? it is what it is; it works.

> But you do specify "custom" locations for sal,
> sal_gtk and standard_common in the beginning, right? Then a smart
> enough tool could learn that information.

I don't know what you mean by "learn"; the user writes the project file,
the tools read it.

Emacs uses one tool to read the project file, and extract a flat
project-search-path (the concatenation of all the Source_Path
attributes).

> In the meantime, I guess, the Ada projects will have to classify all
> dependencies together as search-path.

Yes, that's what I've been saying; it's a valid use case.

That is also what elisp is doing now.

>>  From an Android gradle file:
>>
>> dependencies {
>>
>>      compile project(':licensesdialoglibrary')
>>      compile files('libs/android-async-http-1.4.2-66-g4b6eb97.jar')
>>      compile 'com.google.android.gms:play-services:+'
>> }
>>
>> Some of these are read/write, some are read-only; gradle does not have a
>> mechanism to indicate that.
>
> jars are obviously build artefacts. Doesn't the difference between
> compile '...' and compile project('...') signify something, though?

Yes, but I'm not sure what. I haven't used this enough yet.

>>> Here an excerpt from an imaginary Gemfile:
>>>
>>> gem 'foo', '~> 1.2.3'
>>> gem 'bar', path: '../bar'
>>>
>> The point is that the abstract project API must handle all of these. In
>> the case of Ada, there is a command-line tool that parses the project
>> file and provides the info that Emacs needs.
>
> I'm assuming that the build tools in other ecosystems can likewise
> separate dependencies into search-path and full-on projects. Or, at
> least, that most of them can, and this separation is meaningful.

I know of no tool that does that; I do _not_ make that assumption. Where
did you get the idea it was meaningful?

>>> In the example above, 'bar' will be one of project-directory-roots,
>>> and 'foo' will only be in project-search-path.
>>
>> Why? What effect does that have on what you can do with 'foo' vs 'bar'?
>>
>> How can you tell that just from the gem file?
>
> path: '/..' says that 'bar' is a project residing in a nearby
> directory,

Ok

> and it's probably version-controlled. The fact that I have
> a Git checkout of 'bar' is a strong indicator that I'm maintaining
> that project (though not a guarantee, of course).

That's extra information, _not_ in the gem file.

> 'foo', on the other hand, resides in
> ~/.rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/foo-1.2.3/, 

I guess that's what ">.1.2.3" means? It's some ruby-defined library
directory in your home directory. Similar to a "system library"?

> and any edits I make to it are hard to track or push to the
> development repo.

Ok, since it's like a system library.

>> I'm guessing you are saying "foo is maintained by the same team as the
>> main project (ie read/write), bar is maintained elsewhere (ie
>> read-only)".
>
> The opposite of that.

Ok.

>> So then files in project-directories are read-only, files in
>> project-search-path are read/write.
>
> Ditto.

Which is _not_ what the current code does; xref-query-replace (an edit
function) operates on the result of xref-find-regexp (among other
things), which uses (nconc project-search-path project-directories).

>> But the current default implementation of project-search-path _inludes_
>> project-directories, so this is inconsistent.
>
> Is allowed to include.

I'm not talking about the doc string; I'm talking about the actual code,
which is what elisp uses. But you've agreed to change that.

> read/write operations will use project-roots; read-only operations
> will use project-roots combined with project-search-path. I though
> we've already settled on that.

News to me.

If that is the intended use of these two functions, then they should be
named project-path-read-write and project-path-read-only; that will be
_much_ clearer. It can still be up to the backend to decide whether they
are flat or require additional recursion.

I've never found the need for that distinction in Ada mode.

>> I suggest renaming 'xref--show-xrefs' to 'project-show-locations', and
>
> Why project-? The procedure of displaying the locations doesn't have
> much to do with the project API. The xref API, however, is very much
> related.

Locations are tied other things besides cross-reference; cross-reference
is just one way of producing a list of locations.

find-regexp is another way.

compilation errors is another way.

All of those could use a common location API. 'project' is one place to
put it; it could be in its own 'location' namespace.

Currently, 'grep-find' and 'compilation' share the same location API.
Which is why I suggested that xref also share it. But I do like the
current xref location display, so maybe there is a case for them to
remain distinct. 

They serve different purposes; the xref display makes it easy to choose
one location from a small list; the compilation display makes it easier
to step through all of them in sequence, and it works even for very long
lists.

There should at least there should be this choice of display; the
location data structure could probably be shared (the compilation
location structure contains more info).

project-find-regexp should use the same location display API as grep-find,
since they serve very similar purposes; the user will want to step thru
all of them. Or maybe not, in some particular use case.

(grep-find could eventually be made obsolete in favor of
project-find-regexp).

>> have it take a list of locations, not implicitly call
>> xref-find-function.
>
> The difficulty with that had been the necessity to track the buffers
> that xref-find-function creates, in order to be able to delete each of
> them that haven't been used otherwise, in xref-quit.
>
> The parallel thread about hidden/temporary buffers should result in
> solving that in a different way, and then I'll do just that.

Good.

>> xref-collect-matches uses grep; it can simply be moved to
>> project-collect-matches, along with the xref-*grep functions it uses.
>> The doc string needs to mention subdirs.

> I think there's value in providing function, in xref, that returns a
> list of matches for a given regexp in a given directory, as
> xref-items. 

That's just reimplementing Emacs grep to use the alternate location API; not
worth it.

> It's not, by itself, project-related,

Nor is it xref-related.

> and other xref backends might use it for cheap xref-find-references
> implementations.

After adding path and/or directory recursion support, for searching
project-search-path.

I hope not; better to say "xref not implemented; use Emacs grep". That will
motivate people to implement a real xref backend, as well as immediately
explaining why the xref is not accurate.


It would be nice if <menu-bar> <tools> <grep> defaulted to the
identifier at point.

-- 
-- Stephe

Re: Unified project interface

[Dmitry Gutov]

On 07/29/2015 05:27 AM, Stephen Leake wrote:

> Why? it is what it is; it works.

It gives us less information than it could, about the project configuration.

>> But you do specify "custom" locations for sal,
>> sal_gtk and standard_common in the beginning, right? Then a smart
>> enough tool could learn that information.
>
> I don't know what you mean by "learn"; the user writes the project file,
> the tools read it.

Figure out somehow, based on the project file and the contents of the 
file system, hopefully without asking the user more questions.

For instance, even if the project file gives no direct indication of 
which search-path elements are "proper" projects, some Elisp code could 
look inside each of those directories (and one level above, maybe) and 
see if there are project files in any of them. And those where there 
are, promote to project-roots as well.

That might be harder for Ada, but should be totally doable for 
Java-based projects (if build.gradle doesn't provide that info).

> Yes, that's what I've been saying; it's a valid use case.
>
> That is also what elisp is doing now.

Elisp is also relying on runtime information. It's not reliable enough, 
IMO, to use as the basis of xref-query-replace, for instance. Like I 
mentioned in the other email, the project I've opened right now isn't 
necessarily in load-path.

AdaCore and Gradle are in better positions than that.

>> jars are obviously build artefacts. Doesn't the difference between
>> compile '...' and compile project('...') signify something, though?
>
> Yes, but I'm not sure what. I haven't used this enough yet.

That might be worth investigating.

>> I'm assuming that the build tools in other ecosystems can likewise
>> separate dependencies into search-path and full-on projects. Or, at
>> least, that most of them can, and this separation is meaningful.
>
> I know of no tool that does that; I do _not_ make that assumption. Where
> did you get the idea it was meaningful?

The IDEs usually separate the "current project" (one root with its 
subtree) from the system dependencies. Especially in the Java world, 
where there's just too much code to look at.

Multi-project builds also exist, but they seem to be usually constrained 
to the "several projects inside one parent directory" configuration, see 
"7.3. Multi-project Java build" in 
https://docs.gradle.org/current/userguide/tutorial_java_projects.html.

Also, here's a screenshot of a "project drawer" in a popular IDE: 
http://blog.pivotal.io/wp-content/uploads/2012/02/intellij_modules.png

It showcases a separation of the current project's "modules" from 
"external libraries".

>> and it's probably version-controlled. The fact that I have
>> a Git checkout of 'bar' is a strong indicator that I'm maintaining
>> that project (though not a guarantee, of course).
>
> That's extra information, _not_ in the gem file.

It's a good assumption. Even if it's not version controlled, it's most 
likely a working directory anyway, and the developer's just being 
ignorant about version control.

>> 'foo', on the other hand, resides in
>> ~/.rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/foo-1.2.3/,
>
> I guess that's what ">.1.2.3" means? It's some ruby-defined library
> directory in your home directory. Similar to a "system library"?

Exactly like a system library.

> Which is _not_ what the current code does; xref-query-replace (an edit
> function) operates on the result of xref-find-regexp (among other
> things), which uses (nconc project-search-path project-directories).

Indeed. But that shouldn't be surprising: we are discussing a change 
proposal, and xref-find-regexp, xref-query-replace, and other related 
code you can see in master, had all been written before the current 
iteration of this thread.

>>> But the current default implementation of project-search-path _inludes_
>>> project-directories, so this is inconsistent.
>>
>> Is allowed to include.
>
> I'm not talking about the doc string; I'm talking about the actual code,
> which is what elisp uses. But you've agreed to change that.

Ah, yes, of course. I intend to change that.

But consider this: elisp-search-path is not ideal (the current project 
could be not in load-path). So if project-search-path only consults 
project-search-path-function, by default, what will the default 
implementation of xref-find-references use?

Just (project-search-path (project-current)), or

  (project-prune-directories
    (append
      (project-roots proj)
      (project-search-path proj)))

?

The latter moves the same logic into the caller. The former will give 
the user false negatives, in the absence of a smart, language-aware 
project implementation.

> If that is the intended use of these two functions, then they should be
> named project-path-read-write and project-path-read-only; that will be
> _much_ clearer. It can still be up to the backend to decide whether they
> are flat or require additional recursion.

Err, that sounds silly. Why would you want not to recurse?

The current xref-collect-matches implementation unconditionally 
recurses, and it doesn't seem particularly useful to create a variant 
that doesn't.

> Locations are tied other things besides cross-reference; cross-reference
> is just one way of producing a list of locations.
>
> find-regexp is another way.
>
> compilation errors is another way.
>
> All of those could use a common location API. 'project' is one place to
> put it; it could be in its own 'location' namespace.

I suppose you're right. But until we have a unified location API (which 
I wouldn't count on appearing anytime soon), xref is the better place.

xref-location lives there, and it's the best purpose-agnostic location 
class that we have now.

> Currently, 'grep-find' and 'compilation' share the same location API.
> Which is why I suggested that xref also share it. But I do like the
> current xref location display, so maybe there is a case for them to
> remain distinct.

But there's not much API there, is there? To obtain the list of matches 
from the Grep or Compilation output, one has to re-search-forward in 
that buffer.

>> I think there's value in providing function, in xref, that returns a
>> list of matches for a given regexp in a given directory, as
>> xref-items.
>
> That's just reimplementing Emacs grep to use the alternate location API; not
> worth it.

I think it's useful enough.

>> It's not, by itself, project-related,
>
> Nor is it xref-related.

It's location-related, so to speak. Which live in xref, for now.

> After adding path and/or directory recursion support, for searching
> project-search-path.

Not sure what you mean here.

> I hope not; better to say "xref not implemented; use Emacs grep". That will
> motivate people to implement a real xref backend, as well as immediately
> explaining why the xref is not accurate.

Why would using M-x grep be better? I think it's better to provide an 
imperfect implementation that showcases the available features 
(bindings, commands, presentation).

That will better motivate third-party developers to improve on other 
aspects, such as precision. That's how most open-source projects get 
their contributors, anyway: when a feature works almost, but not 
exactly, right.

And one shouldn't discount Grep-based searched out of hand. For 
instance, a simple xref-find-definitions implementations that searches a 
set of directories for "def funcname", would be a lot better that the 
absolute nothing we provide out of the box to users who don't use etags.

> It would be nice if <menu-bar> <tools> <grep> defaulted to the
> identifier at point.

M-x report-emacs-bug? That seems unrelated.

xref display and multiple locations, was: Re: Unified project interface

[Dmitry Gutov]

On 07/29/2015 05:27 AM, Stephen Leake wrote:

> Currently, 'grep-find' and 'compilation' share the same location API.
> Which is why I suggested that xref also share it. But I do like the
> current xref location display, so maybe there is a case for them to
> remain distinct.
>
> They serve different purposes; the xref display makes it easy to choose
> one location from a small list; the compilation display makes it easier
> to step through all of them in sequence, and it works even for very long
> lists.

I've come to believe that it's a mistake. The only xref command where 
we're likely to only be only in one location is xref-find-definitions, 
and in ideal implementations, it's likely not to show the xref buffer at 
all, jumping straight to the sole result.

So in the average case, pressing RET, or clicking on an xref element, 
should display the location, but keep the xref buffer displayed as well. 
We just need a way to use the current behavior in xref-find-definitions 
(or implement a yet-another selection interface for it).

> There should at least there should be this choice of display; the
> location data structure could probably be shared (the compilation
> location structure contains more info).

The current customization point is xref-show-xrefs-function. There are 
no different implementations yet.

One nice feature of M-x grep, by the way, is that it shows the matches 
immediately as it finds them (whereas the current xref commands are 
forced to wait until the output ends, and collect the whole list).

One way to reach parity there would be to teach xref to accept 
generators (as implemented in generator.el), as well as regular lists.

> (grep-find could eventually be made obsolete in favor of
> project-find-regexp).

Here's hoping.

Re: Unified project interface

[Stephen Leake]

Dmitry Gutov <dgutov@yandex.ru> writes:

> On 07/29/2015 05:27 AM, Stephen Leake wrote:
>
>>> But you do specify "custom" locations for sal,
>>> sal_gtk and standard_common in the beginning, right? Then a smart
>>> enough tool could learn that information.
>>
>> I don't know what you mean by "learn"; the user writes the project file,
>> the tools read it.
>
> Figure out somehow, based on the project file and the contents of the
> file system, hopefully without asking the user more questions.

No need to "figure it out"; the search path for project files is right
there in the file:

   for Project_Path use
     ("../../../org.stephe_leake.sal/build/release",
      "../../../org.stephe_leake.gtk/build/release",
      "../../../org.stephe_leake.makerules");

That's where the project files for sal, sal_gtk, standard_common are
found.

This is interpreted relative to the directory the project file is found
in.

> For instance, even if the project file gives no direct indication of
> which search-path elements are "proper" projects, some Elisp code
> could look inside each of those directories (and one level above,
> maybe) and see if there are project files in any of them. And those
> where there are, promote to project-roots as well.

> That might be harder for Ada, but should be totally doable for
> Java-based projects (if build.gradle doesn't provide that info).

For Ada project files, everything is crystal clear; anything mentioned
in "with" is a proper project; nothing else can be. No searching
involved.

Same for gradle; every build.gradle file defines a proper project;
nothing else does.

> But consider this: elisp-search-path is not ideal (the current project
> could be not in load-path).

I don't understand; the user sets load-path. If the user wants to use
project-* and xref-* functions with the current project, why is it not
in load-path?

That's just bad user configuration; no amount of elisp intelligence can
fix that.

xref-find-definitions for elisp relies on the code being loaded, so
using it without the code being loaded makes no sense.

>> It can still be up to the backend to decide whether they
>> are flat or require additional recursion.
>
> Err, that sounds silly. Why would you want not to recurse?

1) Because many Emacs functions don't recurse. For example, I'd like to add:

(defun project-find-file ()
  "Prompt for a file name, find it on project-search-path."
  (interactive)
  (let ((filename
	 (completing-read
	  "file: "
	  (apply-partially
	   'locate-file-completion-table
	   (project-search-path (project-current))
	   nil))))

    (find-file (locate-file filename (project-search-path (project-current)) nil))))

This works fine with "load-path" instead of "project-search-path". But
it won't work unless project-search-path is flat;
locate-file-completion-table and locate-file do not recurse.

2) As I've mentioned before, not recursing allows computing the set of
   included/excluded directories once, instead of computing it again on
   each recursion.

3) Because some tools provide and expect a flat search path (AdaCore
   tools, gcc include path, emacs load-path, compilation-search-path,
   ff-search-directories).

    It is not at all clear how to convert a flat path into a recursive path
    with ignores; the solution you have for elisp load-path is a kludge,
    and could easily be wrong for some users.
    
> The current xref-collect-matches implementation unconditionally
> recurses, and it doesn't seem particularly useful to create a variant
> that doesn't.

The backends I'm aware of require flat paths (no recursion). So unless
some plausible backend actually requires recursion, it makes more sense
to assume recursion is not required.

>> Locations are tied other things besides cross-reference; cross-reference
>> is just one way of producing a list of locations.
>>
>> find-regexp is another way.
>>
>> compilation errors is another way.
>>
>> All of those could use a common location API. 'project' is one place to
>> put it; it could be in its own 'location' namespace.
>
> I suppose you're right. But until we have a unified location API
> (which I wouldn't count on appearing anytime soon), xref is the better
> place.
>
> xref-location lives there, and it's the best purpose-agnostic location
> class that we have now.

So move it to "location" instead of "xref". The longest journey starts
with one step ...

--
-- Stephe

Re: Unified project interface

[Dmitry Gutov]

On 07/30/2015 11:17 AM, Stephen Leake wrote:

> For Ada project files, everything is crystal clear; anything mentioned
> in "with" is a proper project; nothing else can be. No searching
> involved.

Fantastic, then Ada gives you all you need to implement both 
project-search-path and project-roots.

> I don't understand; the user sets load-path. If the user wants to use
> project-* and xref-* functions with the current project, why is it not
> in load-path?

It's not always the current practice, AFAIK. And adding it to load-path 
interactively is not something we should force the users to do.

Consider that when xref-find-references doesn't find anything, it will 
say "nothing found", without further explanations, leaving the user to 
puzzle out why and what. We should err on the side of showing too much, 
rather than too little.

> xref-find-definitions for elisp relies on the code being loaded, so
> using it without the code being loaded makes no sense.

One can also load the code without adding the file to load-path. For 
instance, via eval-buffer.

> 1) Because many Emacs functions don't recurse. For example, I'd like to add:
>
> (defun project-find-file ()
>    "Prompt for a file name, find it on project-search-path."
>    (interactive)
>    (let ((filename
> 	 (completing-read
> 	  "file: "
> 	  (apply-partially
> 	   'locate-file-completion-table
> 	   (project-search-path (project-current))
> 	   nil))))
>
>      (find-file (locate-file filename (project-search-path (project-current)) nil))))
>
> This works fine with "load-path" instead of "project-search-path". But
> it won't work unless project-search-path is flat;
> locate-file-completion-table and locate-file do not recurse.

I guess you'd need to implement a "flattening" function for that, 
indeed. I don't really see myself using this command, though.

> 2) As I've mentioned before, not recursing allows computing the set of
>     included/excluded directories once, instead of computing it again on
>     each recursion.

And I countered that, for many commands, computing that set is unlikely 
to be the bottleneck. And cache invalidation is not user-friendly.

> 3) Because some tools provide and expect a flat search path (AdaCore
>     tools, gcc include path, emacs load-path, compilation-search-path,
>     ff-search-directories).

That's not really true.

There are many directories inside /usr/include, but C_INCLUDE_PATH won't 
mention most of them. Instead, #include <postgresql/pg_config> will use 
/usr/include/postgresql/pg_config.h, as long as /usr/include is in 
C_INCLUDE_PATH.

There's only one directory in Emacs load-path with "cedet" in its name, 
but (require 'semantic/db) will load lisp/cedet/semantic/db.el.

You're right about compilation-search-path and ff-search-directories, 
though.

It seems to be true for the Ada source search path as well, but since 
/usr/lib/gcc/x86_64-linux-gnu/4.9/adainclude/ has no subdirectories, the 
point is moot. Traversing it recursively wouldn't make anything worse.

>      It is not at all clear how to convert a flat path into a recursive path
>      with ignores; the solution you have for elisp load-path is a kludge,
>      and could easily be wrong for some users.

I'd like to see one of those users first.

>> xref-location lives there, and it's the best purpose-agnostic location
>> class that we have now.
>
> So move it to "location" instead of "xref". The longest journey starts
> with one step ...

Until it's used in at least two different places, creating a separate 
package for it is, I think, premature.

Re: Unified project interface

[Stephen Leake]

Dmitry Gutov <dgutov@yandex.ru> writes:

> On 07/30/2015 11:17 AM, Stephen Leake wrote:
>
>> I don't understand; the user sets load-path. If the user wants to use
>> project-* and xref-* functions with the current project, why is it not
>> in load-path?
>
> It's not always the current practice, AFAIK.

I still don't understand why. But I've proposed
`project-add-search-path' (below and in another email), so that can be
used to handle this case.

> And adding it to load-path interactively is not something we should
> force the users to do.

Yes, if the user doesn't want a file in load-path, they don't put it
there. But then they will have to use project-add-search-path instead.

We have to expect some help from the user to indicate what's in the
project!

> Consider that when xref-find-references doesn't find anything, it will
> say "nothing found", without further explanations, leaving the user to
> puzzle out why and what. We should err on the side of showing too
> much, rather than too little.

Sorry, I can't guess what you have in mind that the user is doing here.
What is the user searching for, what did they do to specify
project-search-path, and what do they expect xref-find-references to
show?

>> 1) Because many Emacs functions don't recurse. For example, I'd like to add:
>>
>> (defun project-find-file ()
>>    "Prompt for a file name, find it on project-search-path."
>>    (interactive)
>>    (let ((filename
>> 	 (completing-read
>> 	  "file: "
>> 	  (apply-partially
>> 	   'locate-file-completion-table
>> 	   (project-search-path (project-current))
>> 	   nil))))
>>
>>      (find-file (locate-file filename (project-search-path (project-current)) nil))))
>>
>> This works fine with "load-path" instead of "project-search-path". But
>> it won't work unless project-search-path is flat;
>> locate-file-completion-table and locate-file do not recurse.
>
> I guess you'd need to implement a "flattening" function for that,
> indeed.

If project-search-path is recursive, yes. Which is certainly possible to
implement here; I propose a "recursive" project flag below. If that's
set, this function can use directory-files and directories in
project-ignores to do the recursion.

The predicate for locate-file-completion-table (set to nil above) needs
to handle the files in project-ignores as well.

> I don't really see myself using this command, though.

That doesn't mean it's not a valid use case; you need to consider all
users.

I often want to look at project.el (or xref.el, ada-mode.el, etc); I
bind this function (currently using load-path) to M-F12, so M-F12
proj <ret> finds it for me easily.

The Ada user group requested this function for Ada projects (it has a
similar implementation).

>> 2) As I've mentioned before, not recursing allows computing the set of
>>     included/excluded directories once, instead of computing it again on
>>     each recursion.
>
> And I countered that, for many commands, computing that set is
> unlikely to be the bottleneck. And cache invalidation is not
> user-friendly.

Which is faster/friendlier depends on the details, so we'll have to wait
for actual implementations. All I'm asking is that we do not make design
decisions at the generic project API level that rule out this approach.

>> 3) Because some tools provide and expect a flat search path (AdaCore
>>     tools, gcc include path, emacs load-path, compilation-search-path,
>>     ff-search-directories).
>
> That's not really true.
>
> There are many directories inside /usr/include, but C_INCLUDE_PATH
> won't mention most of them. Instead, #include <postgresql/pg_config>
> will use /usr/include/postgresql/pg_config.h, as long as /usr/include
> is in C_INCLUDE_PATH.

Yes. So if the user wants to search all relevant includes for some
declaration, they have to treat C_INCLUDE_PATH as recursive. But the
compiler (which _defines_ C_INCLUDE_PATH) does _not_ treat it as
recursive. So it's confused.

> There's only one directory in Emacs load-path with "cedet" in its
> name, but (require 'semantic/db) will load lisp/cedet/semantic/db.el.

Similar situation: `load' does not treat `load-path' as recursive, but
text searches need to. I had not considered that.

I think cedet is the only exception to the 'flat load-path' rule (at
least in core emacs); I hope it goes away, and is not copied by other
projects.

I would provide a special syntax to handle this in
project-add-search-path:

(defun project-add-search-path (project path &optional make-recursive)
  "Add PATH (a list of directories) to the search path for PROJECT.

If a directory in PATH ends in \"/**\", also add all directories under
that directory.

If MAKE-RECURSIVE is non-nil, the full project search path is pruned so
that only highest-level directories are present, and all uses of
`project-search-path' recurse into subdirectories. In this case, it is
likely you will also need to specify ignored directories with
`project-add-ignores'.

If MAKE-RECURSIVE is nil, uses of `project-search-path' no not recurse
into subdirectories."
... )

The Ada project files support the ** sytnax.


User projects can have directory structures for which recursive
search from the vc root is wrong:

myproj/
    .git/
    .gitignore

    build/
        Makefile
        ada_project.gpr
        *.exe
        obj/
            # contains object files for ada files

    lisp/
        # elisp files for emacs-25+

    lisp-emacs-24/
        # elisp files defining emacs-25 functions

    lisp-emacs-23/

    ada/
        # ada files

The elisp project (depending on current Emacs version) will only want to
search one of:

(lisp)
(lisp lisp-emacs-24)
(lisp lisp-emacs-23)

The Ada project will only want to search the "ada" and "build"
directories (for the Makefile and Ada project file).

There will be cases where the user will want to search all files; then a
separate project that uses project-add-search-path to add all dirs would
be appropriate.

No search should include the "obj" directory.

This could be handled if the user can explicitly specify project ignores
(not just rely on .gitignore etc; git does _not_ ignore the lisp-emacs-*
and ada directories). More complicated structures are harder to handle
with ignores; you'll end up with a long flat list of ignores, so it
might be simpler to have a flat list of includes instead.

> You're right about compilation-search-path and ff-search-directories,
> though.
>
> It seems to be true for the Ada source search path as well, but since
> /usr/lib/gcc/x86_64-linux-gnu/4.9/adainclude/ has no subdirectories,
> the point is moot. Traversing it recursively wouldn't make anything
> worse.

User Ada projects can easily have directory structures for which
recursive search from the vc root would be wrong; see above.

It would be surprising to an Ada user (used to flat search paths), if the
Emacs project silently started using recursive search paths.

>>      It is not at all clear how to convert a flat path into a recursive path
>>      with ignores; the solution you have for elisp load-path is a kludge,
>>      and could easily be wrong for some users.
>
> I'd like to see one of those users first.

Me, and many other teams at NASA - structures similar to the above,
where the non-included directories can also be release version,
debug/release options, target hardware/os option, compiler
vendor/version, etc.

The jdee project has lisp and java directories.

The kludge is in the root implementation of project-search-path, so it
applies to all project backends that only customize
project-search-path-function.


I have no objection to allowing some project backends to use recursive
path/ignore dirs; I just want to also allow some backends to use
non-recursive path/ignore files. Or the user can choose, via
project-add-search-path above.


>>> xref-location lives there, and it's the best purpose-agnostic location
>>> class that we have now.
>>
>> So move it to "location" instead of "xref". The longest journey starts
>> with one step ...
>
> Until it's used in at least two different places, creating a separate
> package for it is, I think, premature.

Right. We now have those two places; xref-find-definitions,
project-find-regexp.

Unless project-find-regexp uses the compile location data structure,
which would probably be better.

Although it would be even better if project-find-regexp uses the compile
location data structure wrapped in a location-compile class, derived
from the root location class, renamed from xref-location.

--
-- Stephe

Re: Unified project interface

[Dmitry Gutov]

On 07/31/2015 07:13 PM, Stephen Leake wrote:

> I often want to look at project.el (or xref.el, ada-mode.el, etc); I
> bind this function (currently using load-path) to M-F12, so M-F12
> proj <ret> finds it for me easily.

To reach the same result, I currently either use ido-switch-buffer (with 
ido-use-virtual-buffers set to t) or projectile-find-file (which asks 
the user to choose between files in the project, based on file names 
relative to the root). The key thing here is substring matching: I also 
simply type "project." and see lisp/progmodes/project.el as the only 
completion.

Using relative paths also helps with navigating to files with the same 
name inside different directories.

> Which is faster/friendlier depends on the details, so we'll have to wait
> for actual implementations. All I'm asking is that we do not make design
> decisions at the generic project API level that rule out this approach.

We do have to make some decisions, though. For instance, supporting both 
recursive and non-recursive elements in search-path will require a 
separate code path for each function that uses it.

As such, it will require a separate code path in xref-collect-matches 
and an alternative for xref--rgrep-command.

>> There are many directories inside /usr/include, but C_INCLUDE_PATH
>> won't mention most of them. Instead, #include <postgresql/pg_config>
>> will use /usr/include/postgresql/pg_config.h, as long as /usr/include
>> is in C_INCLUDE_PATH.
>
> Yes. So if the user wants to search all relevant includes for some
> declaration, they have to treat C_INCLUDE_PATH as recursive. But the
> compiler (which _defines_ C_INCLUDE_PATH) does _not_ treat it as
> recursive. So it's confused.

We need to search files in subdirectories as well, and that is the key 
distinction.

The fact that compiler treats the roots in a special way, however, can 
be important. Because of that, the list of roots contains information 
that can be lost after a naive translation to a list of "flat" 
directories. For instance, if you have the roots, you can search them 
linearly and tell whether there's a file corresponding to 
'postgresql/pg_config' inside one of them.

With a flat list, we could only give an imprecise answer to that 
question, and the search might have to perform extra work.

> I think cedet is the only exception to the 'flat load-path' rule (at
> least in core emacs); I hope it goes away, and is not copied by other
> projects.

Even though it's not very popular, it's unlikely to go away. But in any 
case, the corresponding path variables in all other ecosystems that I 
know of (Ruby, Python, Java, Node.js) follow in the footsteps of C and 
Emacs.

And the principle of translating qualified package (or module, or class) 
names to file names is quite pervasive.

Ada is an exception, rather than the rule.

> I would provide a special syntax to handle this in
> project-add-search-path:
>
> (defun project-add-search-path (project path &optional make-recursive)
>    "Add PATH (a list of directories) to the search path for PROJECT.
>
> If a directory in PATH ends in \"/**\", also add all directories under
> that directory.
>
> If MAKE-RECURSIVE is non-nil, the full project search path is pruned so
> that only highest-level directories are present, and all uses of
> `project-search-path' recurse into subdirectories. In this case, it is
> likely you will also need to specify ignored directories with
> `project-add-ignores'.

This description explains the purpose, but doesn't describe the 
resulting data structure that would hold that information. The ones that 
come to mind are rather ugly.

> If MAKE-RECURSIVE is nil, uses of `project-search-path' no not recurse
> into subdirectories."
> ... )
>
> The Ada project files support the ** sytnax.

The /** syntax is rather nifty. But why do you propose to have it and 
MAKE-RECURSIVE at the same time? We could just interpret /** as the 
recursive marker. Then the result could be a list of strings.

> This could be handled if the user can explicitly specify project ignores
> (not just rely on .gitignore etc; git does _not_ ignore the lisp-emacs-*
> and ada directories). More complicated structures are harder to handle
> with ignores; you'll end up with a long flat list of ignores, so it
> might be simpler to have a flat list of includes instead.

I've just started to write about this... Yes indeed, I think every flat 
structure can be translated into path/to/dir, plus path/to/dir/*/ in the 
ignores. The translation function shouldn't take too much work to write.

There's no need to produce the list of ignores manually. The Ada project 
backend could do that for the user. And if the list is long... first, 
you can use wildcards. Second, at least xref-collect-matches won't have 
to handle the flat case specially. Nor will the other consumers of the 
project API.

> It would be surprising to an Ada user (used to flat search paths), if the
> Emacs project silently started using recursive search paths.

The Ada project implementation can provide a friendlier interface/set of 
customizations/etc for Ada users. Different communities are used to 
different things. Any default would be surprising to someone.

>>>       It is not at all clear how to convert a flat path into a recursive path
>>>       with ignores; the solution you have for elisp load-path is a kludge,
>>>       and could easily be wrong for some users.
>>
>> I'd like to see one of those users first.
>
> Me, and many other teams at NASA - structures similar to the above,
> where the non-included directories can also be release version,
> debug/release options, target hardware/os option, compiler
> vendor/version, etc.

There can be different opinions about this. As a project maintainer, I'd 
rather be able to find all references to the function foo, no matter the 
target OS of the code, or the targeted compiler version.

> The jdee project has lisp and java directories.

The Java class path is also "recursive".

> The kludge is in the root implementation of project-search-path, so it
> applies to all project backends that only customize
> project-search-path-function.

Please, stop calling major modes that do that "project backends". 
Customizing project-search-path-function does not constitute one.

A real backend can always override project-search-path (and smart ones 
should).

>>> So move it to "location" instead of "xref". The longest journey starts
>>> with one step ...
>>
>> Until it's used in at least two different places, creating a separate
>> package for it is, I think, premature.
>
> Right. We now have those two places; xref-find-definitions,
> project-find-regexp.

Both still use xref for display. The location values aren't consumed by 
anything else. So that's just one place.

> Although it would be even better if project-find-regexp uses the compile
> location data structure wrapped in a location-compile class, derived
> from the root location class, renamed from xref-location.

You're welcome to explore this direction with code.

Per-language project-search-path, was: Re: Unified project interface

[Dmitry Gutov]

On 07/31/2015 07:13 PM, Stephen Leake wrote:

> The elisp project (depending on current Emacs version) will only want to
> search one of:
>
> (lisp)
> (lisp lisp-emacs-24)
> (lisp lisp-emacs-23)
>
> The Ada project will only want to search the "ada" and "build"
> directories (for the Makefile and Ada project file).

By the way, do you think project-search-path should have a major-mode 
(or language) argument? Or maybe depend on it implicitly, via the 
current buffer?

ede-source-paths dispatches on both PROJECT and MODE; to reuse it (in 
the ede project backend), we'd have to produce the mode argument.

The caveats with the given choices are:

- Users can use different major modes for the same files. Normally, 
project support shouldn't affect that.

- We have no registry of languages, AFAIK. And different languages can 
be spelled differently (C++ vs Cpp, Fortran vs F77, Objective-C vs ObjC).

- Just prescribing to use the current buffer in whatever way the project 
backend wants could be the most flexible, but it doesn't solve either 
difficulty.

We could ask a similar question for project-ignores as well, though I'd 
rather not go that far. It needs a DIRECTORY argument already.

Re: Unified project interface

[Stephen Leake]

Dmitry Gutov <dgutov@yandex.ru> writes:

> On 07/31/2015 07:13 PM, Stephen Leake wrote:
>
>> Which is faster/friendlier depends on the details, so we'll have to wait
>> for actual implementations. All I'm asking is that we do not make design
>> decisions at the generic project API level that rule out this approach.
>
> We do have to make some decisions, though. For instance, supporting
> both recursive and non-recursive elements in search-path will require
> a separate code path for each function that uses it.
>
> As such, it will require a separate code path in xref-collect-matches
> and an alternative for xref--rgrep-command.

Or they could abort; that would be ok while we are experimenting. It may
be that one or the other will turn out to be The Right Way.

A backend that uses flat paths doesn't have to use those functions. Or,
the people writing that backend can modify them if that seems better
than putting similar code elsewhere. That would be part of the cost of
that design decision.

>> I would provide a special syntax to handle this in
>> project-add-search-path:
>>
>> (defun project-add-search-path (project path &optional make-recursive)
>>    "Add PATH (a list of directories) to the search path for PROJECT.
>>
>> If a directory in PATH ends in \"/**\", also add all directories under
>> that directory.
>>
>> If MAKE-RECURSIVE is non-nil, the full project search path is pruned so
>> that only highest-level directories are present, and all uses of
>> `project-search-path' recurse into subdirectories. In this case, it is
>> likely you will also need to specify ignored directories with
>> `project-add-ignores'.
>>
>> If MAKE-RECURSIVE is nil, uses of `project-search-path' no not recurse
>> into subdirectories."
>> ... )
>>
>> The Ada project files support the ** sytnax.
>>
> This description explains the purpose, but doesn't describe the
> resulting data structure that would hold that information. The ones
> that come to mind are rather ugly.

I have no idea what data structures come to mind for you.

This could be implemented with a root projects type:

(cl-defstruct (projects)
  user-search-path ;; list of directories added to project-search-path
  user-ignores ;; list of ignores added to project-ignores
  recursive-search-path
  ;; if non-nil, project-search-path is treated as recursive in all usees
  ...
  )

Is that "ugly"?

> The /** syntax is rather nifty. But why do you propose to have it and
> MAKE-RECURSIVE at the same time? We could just interpret /** as the
> recursive marker. Then the result could be a list of strings.

Because you never need both. If you are used to using recursive paths,
adding ** in the path to tell the code that will seem odd. So it seems
best to have an explicit flag.

It might be better to specify the recursive flag in the project
constructor; then this function would check it, not set it.

>> This could be handled if the user can explicitly specify project ignores
>> (not just rely on .gitignore etc; git does _not_ ignore the lisp-emacs-*
>> and ada directories). More complicated structures are harder to handle
>> with ignores; you'll end up with a long flat list of ignores, so it
>> might be simpler to have a flat list of includes instead.
>
> I've just started to write about this... Yes indeed, I think every
> flat structure can be translated into path/to/dir, plus path/to/dir/*/
> in the ignores. The translation function shouldn't take too much work
> to write.

The reverse is also true; it is possible to convert recursive
path/ignores into flat path/ignores.

> There's no need to produce the list of ignores manually. 

Only if you ignore the use cases I posted. There are situations where
the algorithm will not do what the user wants.

The fact that you don't like them does not mean they are not valid.

Would it be such a problem to provide 'project-add-ignores'?

>> Me, and many other teams at NASA - structures similar to the above,
>> where the non-included directories can also be release version,
>> debug/release options, target hardware/os option, compiler
>> vendor/version, etc.
>
> There can be different opinions about this. 

Yes, precisely. And Emacs core should not be enforcing any opinions!

>>>> So move it to "location" instead of "xref". The longest journey starts
>>>> with one step ...
>>>
>>> Until it's used in at least two different places, creating a separate
>>> package for it is, I think, premature.
>>
>> Right. We now have those two places; xref-find-definitions,
>> project-find-regexp.
>
> Both still use xref for display. The location values aren't consumed
> by anything else. So that's just one place.

They both display locations. The code that displays locations should
also be in the locations namespace, not the xref one.

-- 
-- Stephe

Re: Per-language project-search-path, was: Re: Unified project interface

[Stephen Leake]

Dmitry Gutov <dgutov@yandex.ru> writes:

> On 07/31/2015 07:13 PM, Stephen Leake wrote:
>
>> The elisp project (depending on current Emacs version) will only want to
>> search one of:
>>
>> (lisp)
>> (lisp lisp-emacs-24)
>> (lisp lisp-emacs-23)
>>
>> The Ada project will only want to search the "ada" and "build"
>> directories (for the Makefile and Ada project file).
>
> By the way, do you think project-search-path should have a major-mode
> (or language) argument? Or maybe depend on it implicitly, via the
> current buffer?

My experience is always with a project file; the user explicitly selects
which project file is active. That can be multi-language or
single-language.

So I'm not familiar with the project system trying to guess a project on
the fly, as the current code does.

As you point out, there are use cases for using the major mode to
influence the project, and others for not doing that.

I don't know which would be better as the default; the only general
solution is to require a project file.

> ede-source-paths dispatches on both PROJECT and MODE; to reuse it (in
> the ede project backend), we'd have to produce the mode argument.

Ede also uses the (require 'path/file) style; not surprising, since it
was written by the CEDET team.

I don't see any implementations of ede-source-paths, and only one use of
it, so it's hard to say what the purpose of the mode parameter is.


-- 
-- Stephe

Re: Unified project interface

[Stephen Leake]

Stephen Leake <stephen_leake@stephe-leake.org> writes:

> Dmitry Gutov <dgutov@yandex.ru> writes:
>
>> The /** syntax is rather nifty. But why do you propose to have it and
>> MAKE-RECURSIVE at the same time? We could just interpret /** as the
>> recursive marker. Then the result could be a list of strings.

I read this wrong.

You _only_ need /** if the paths are _not_ recursive, so this cannot
serve as the recursive flag.

-- 
-- Stephe

Re: Unified project interface

[Dmitry Gutov]

On 08/01/2015 12:50 PM, Stephen Leake wrote:

>> As such, it will require a separate code path in xref-collect-matches
>> and an alternative for xref--rgrep-command.
>
> Or they could abort; that would be ok while we are experimenting. It may
> be that one or the other will turn out to be The Right Way.

That's not helpful. Of course we want the API be straightforward and 
usable for as many callers as possible. It's the main goal.

I'm not sure how The Right Way would supposedly present itself.

> A backend that uses flat paths doesn't have to use those functions. Or,
> the people writing that backend can modify them if that seems better
> than putting similar code elsewhere. That would be part of the cost of
> that design decision.

It's a high cost.

If the author of such backend intends to reimplement or rewrite every 
feature like that, they can just as well not bother with implementing 
the Project API. As you can see, there's not much code in project.el, so 
they won't miss much if they don't value interoperability.

> This could be implemented with a root projects type:
>
> (cl-defstruct (projects)
>    user-search-path ;; list of directories added to project-search-path
>    user-ignores ;; list of ignores added to project-ignores
>    recursive-search-path
>    ;; if non-nil, project-search-path is treated as recursive in all usees
>    ...
>    )

That doesn't tell me what values project-search-path would return, for 
every kind of configuration. That's the important part.

> Is that "ugly"?

Depends on how you feel about double negatives. There should be just one 
way to express whether a given directory should be traversed recursively.

(cl-defstruct (projects)
   flat-search-path
   recursive-search-path

would be one option for the implementation, but what would it mean for 
the API? If we have both project-flat-search-path and 
project-recursive-search-path as accessors, there's high probability 
that some functions will only look up and handle one of them. That's bad.

>> The /** syntax is rather nifty. But why do you propose to have it and
>> MAKE-RECURSIVE at the same time? We could just interpret /** as the
>> recursive marker. Then the result could be a list of strings.
>
> Because you never need both. If you are used to using recursive paths,
> adding ** in the path to tell the code that will seem odd. So it seems
> best to have an explicit flag.

The user could specify the paths in any way they like, are used to, etc. 
What's important is what the project backend returns in project-search-path.

>> I've just started to write about this... Yes indeed, I think every
>> flat structure can be translated into path/to/dir, plus path/to/dir/*/
>> in the ignores. The translation function shouldn't take too much work
>> to write.
>
> The reverse is also true; it is possible to convert recursive
> path/ignores into flat path/ignores.

Not without drawbacks. First, it's more computationally intensive (Emacs 
would have to traverse the actual directory tree). Second, like I 
mentioned before, that loses information: in particular, we won't know 
which directories were the root ones.

>> There's no need to produce the list of ignores manually.
>
> Only if you ignore the use cases I posted. There are situations where
> the algorithm will not do what the user wants.

What algorithm? I don't see why you can't write an algorithm that will 
both handle flat directories via ignores, as well as honor the user's 
wishes.

> The fact that you don't like them does not mean they are not valid.
>
> Would it be such a problem to provide 'project-add-ignores'?

I don't understand why you would state this question here? It doesn't 
seem relevant.

>>> Me, and many other teams at NASA - structures similar to the above,
>>> where the non-included directories can also be release version,
>>> debug/release options, target hardware/os option, compiler
>>> vendor/version, etc.
>>
>> There can be different opinions about this.
>
> Yes, precisely. And Emacs core should not be enforcing any opinions!

Err, some opinions are worth being enforced. But anyway, the Ada project 
backend can behave any way you wish, in this respect.

>> Both still use xref for display. The location values aren't consumed
>> by anything else. So that's just one place.
>
> They both display locations. The code that displays locations should
> also be in the locations namespace, not the xref one.

What's left for xref, then? Most of the code there deals with displaying 
xrefs in the buffer.

Re: Unified project interface

[Dmitry Gutov]

On 08/01/2015 01:51 PM, Stephen Leake wrote:

> You _only_ need /** if the paths are _not_ recursive, so this cannot
> serve as the recursive flag.

It would serve as "recursive flag" for each particular directory.

Of course, in this approach the directories are flat by default.

Re: Per-language project-search-path, was: Re: Unified project interface

[Dmitry Gutov]

On 08/01/2015 01:43 PM, Stephen Leake wrote:

> My experience is always with a project file; the user explicitly selects
> which project file is active. That can be multi-language or
> single-language.
>
> So I'm not familiar with the project system trying to guess a project on
> the fly, as the current code does.

I'm not discussing a particular project backend here, just the API. And 
you can use the user-selected backend everywhere, if there's an 
appropriate element at the head of project-find-functions.

If I recall, you've mentioned a concept of project file listing 
different search-paths for different languages. Without 
project-search-path taking the language into account, you'll have to 
append all paths together.

And what about jdee? Would you treat lisp/.dir-locals.el as the project 
file for Elisp, and ask the user to select it?

> As you point out, there are use cases for using the major mode to
> influence the project, and others for not doing that.

I've only listed the implementation difficulties. If there are any 
particular problems with the concept of dispatching on the language, I'd 
like to hear about those.

> I don't know which would be better as the default; the only general
> solution is to require a project file.

How does that translate to API requirements?

> Ede also uses the (require 'path/file) style; not surprising, since it
> was written by the CEDET team.

It does. This doesn't seem relevant.

> I don't see any implementations of ede-source-paths, and only one use of
> it, so it's hard to say what the purpose of the mode parameter is.

That's true, unfortunately. The only implementations I could find are in 
malabar-mode (another third-party project attempting to provide a Java 
editing environment).

Eric, could you tell if we're missing anything? Are there other places 
where ede-source-paths is defined or used?

Any reason why symref isn't using it?

Re: Unified project interface

[Stephen Leake]

Dmitry Gutov <dgutov@yandex.ru> writes:

> On 08/01/2015 12:50 PM, Stephen Leake wrote:
>
>> This could be implemented with a root projects type:
>>
>> (cl-defstruct (projects)
>>    user-search-path ;; list of directories added to project-search-path
>>    user-ignores ;; list of ignores added to project-ignores
>>    recursive-search-path
>>    ;; if non-nil, project-search-path is treated as recursive in all usees
>>    ...
>>    )
>
> That doesn't tell me what values project-search-path would return, for
> every kind of configuration. That's the important part.

(cl-defgeneric project-search-path (project)
  "..."
  (project--prune-directories
   (nconc (project-user-search-path project)
          (funcall project-search-path-function)
          ;; Include these, because we don't know any better.
          ;; But a specialized implementation may include only some of
          ;; the project's subdirectories, if there are no source
          ;; files at the top level.
          (project-directories project))))

>> Is that "ugly"?
>
> Depends on how you feel about double negatives. There should be just
> one way to express whether a given directory should be traversed
> recursively.

There is; the boolean flag recursive-search-path.

> (cl-defstruct (projects)
>   flat-search-path
>   recursive-search-path
>

I'm guessing that here "flat-search-path" is a list of directories,
intended to be used non-recursively. A comment would really help;
guessing is dangerous in this discussion.

And then "recursive-search-path" is a list of directories, intended to
be used recursively.

So they contain redundant information; they are each a cache of the
result of the current `project-search-path' (with and without pruning).

> would be one option for the implementation, but what would it mean for
> the API? If we have both project-flat-search-path and
> project-recursive-search-path as accessors, there's high probability
> that some functions will only look up and handle one of them. That's
> bad.

Yes, it would be nice if we could agree that either flat or recursive is
the baseline; any function that needs the other can convert.

So far, I think we have these arguments on each side:

- elisp load-path is already mostly flat

    it needs to be expanded to handle path/file style requires

- Ada project paths are flat

    This is actually AdaCore project paths; they support C++ and C as well.

- Some other project paths are recursive

- Converting between them (while also converting the corresponding
  includes) is possible
  
- Using find-grep and similar external programs allows recursive search
  to display first results immediately
  
>>> I've just started to write about this... Yes indeed, I think every
>>> flat structure can be translated into path/to/dir, plus path/to/dir/*/
>>> in the ignores. The translation function shouldn't take too much work
>>> to write.
>>
>> The reverse is also true; it is possible to convert recursive
>> path/ignores into flat path/ignores.
>
> Not without drawbacks. First, it's more computationally intensive
> (Emacs would have to traverse the actual directory tree). 

You have to traverse the tree at some point; either during the use of
the recursive path, or to convert the path.

Converting it once is cheaper overall.

But, as I've said before, this is an optimization issue. If it turns out
that the performance either way is acceptable, it's not important.

> Second, like
> I mentioned before, that loses information: in particular, we won't
> know which directories were the root ones.

If that is useful information, obviously it can be saved. That's what
project-roots is for.

>>> There's no need to produce the list of ignores manually.
>>
>> Only if you ignore the use cases I posted. There are situations where
>> the algorithm will not do what the user wants.
>
> What algorithm? 

The algorithm that automatically produces the list of ignores.

> I don't see why you can't write an algorithm that will
> both handle flat directories via ignores, as well as honor the user's
> wishes.

The user has to have a way of expressing those wishes. That's what
project-add-ignores is for.

>> The fact that you don't like them does not mean they are not valid.
>>
>> Would it be such a problem to provide 'project-add-ignores'?
>
> I don't understand why you would state this question here? It doesn't
> seem relevant.

I'll take that as agreeing to add the functions project-add-search-path
and project-add-ignores to project.el

Given all of the above, I'll make an attempt to write Ada and JDEE
project backends, using recursive paths. We'll see how that goes.

-- 
-- Stephe

Re: Unified project interface

[Dmitry Gutov]

On 08/01/2015 05:15 PM, Stephen Leake wrote:

> (cl-defgeneric project-search-path (project)
>    "..."
>    (project--prune-directories
>     (nconc (project-user-search-path project)
>            (funcall project-search-path-function)
>            ;; Include these, because we don't know any better.
>            ;; But a specialized implementation may include only some of
>            ;; the project's subdirectories, if there are no source
>            ;; files at the top level.
>            (project-directories project))))

That doesn't use recursive-search-path, and doesn't allow to treat only 
some elements recursively. Which would be natural.

And if we allowed /**, the recursive-search-path flag would be redundant.

>> (cl-defstruct (projects)
>>    flat-search-path
>>    recursive-search-path
>>
>
> I'm guessing that here "flat-search-path" is a list of directories,
> intended to be used non-recursively.
> ...
> And then "recursive-search-path" is a list of directories, intended to
> be used recursively.

Yes.

> So they contain redundant information; they are each a cache of the
> result of the current `project-search-path' (with and without pruning).

This is backwards. There wouldn't be project-search-path function at 
all, and neither flat-search-path, nor recursive-search-path would be 
the result of pruning.

> Yes, it would be nice if we could agree that either flat or recursive is
> the baseline; any function that needs the other can convert.

That's the direction I'm inclined to take, too.

> So far, I think we have these arguments on each side:
>
> - elisp load-path is already mostly flat

For out purposes, it's plainly recursive, even if that feature isn't in 
use in most projects. That's highly unlikely to ever change.

>      it needs to be expanded to handle path/file style requires

Same as any other language with "recursive" search path that I know of.

> - Ada project paths are flat
>
>      This is actually AdaCore project paths; they support C++ and C as well.
>
> - Some other project paths are recursive

Most of them, AFAIK. I've given several examples; there have been no 
further counter-examples mentioned.

> - Converting between them (while also converting the corresponding
>    includes) is possible

I think you mean the corresponding ignores. And converting a certain 
class of ignores from recursive to flat paths seems non-trivial.

> - Using find-grep and similar external programs allows recursive search
>    to display first results immediately

That's a good point, and I think it hasn't been stated clearly before.

>> Second, like
>> I mentioned before, that loses information: in particular, we won't
>> know which directories were the root ones.
>
> If that is useful information, obviously it can be saved. That's what
> project-roots is for.

project-roots are different. This would have to be a separate accessor, 
because the search path roots outside of the project are just as (if not 
more) important.

>> I don't see why you can't write an algorithm that will
>> both handle flat directories via ignores, as well as honor the user's
>> wishes.
>
> The user has to have a way of expressing those wishes. That's what
> project-add-ignores is for.

The user will express their wishes via the project file and .gitignore.

There's nothing to discuss there; obviously, a project implementation 
will have access to all information the user will give it.

> I'll take that as agreeing to add the functions project-add-search-path
> and project-add-ignores to project.el

Not really. I don't think we should mess with a project backend's 
internals. So unless we see a use case for third-party code to modify 
project's search-path and ignores, these functions should stay out of 
the API.

> Given all of the above, I'll make an attempt to write Ada and JDEE
> project backends, using recursive paths. We'll see how that goes.

Hopefully, you'll make that attempt anyway.

Re: Per-language project-search-path, was: Re: Unified project interface

[Stephen Leake]

Dmitry Gutov <dgutov@yandex.ru> writes:

> On 08/01/2015 01:43 PM, Stephen Leake wrote:
>
> If I recall, you've mentioned a concept of project file listing
> different search-paths for different languages. Without
> project-search-path taking the language into account, you'll have to
> append all paths together.

I don't know what you mean by "all paths" here; one project file defines
one source path.

In the case of a multilanguage project, that search path includes files
in all languages.

I often mix languages in one directory anyway; the file extension
indicates the language, so there's no need for it in the directory name.


> And what about jdee? Would you treat lisp/.dir-locals.el as the
> project file for Elisp, and ask the user to select it?

I think you are asking

    "what project would you use while editing elisp files that implement
    jdee"

not

    "what project file does jdee use for Java projects".

Yes, serious users will want to explicitly select the active project.

If I'm debugging jdee, I want the search path to be elisp load-path,
whether I'm in a notes.text file, a jdee .el file, or a test .java file.
So I don't want Emacs to guess what the active project is, nor determine
the search path based on the mode.

When I'm testing jdee's use of the project API, I will explicitly select
a Java project as the active project.

I've just written this:

(cl-defstruct (elisp-project)
  ;; no slots yet
  )

(cl-defmethod project-search-path ((type elisp-project))
  (project--prune-directories load-path))

(cl-defmethod project-root ((type elisp-project))
  ;; no meaningful root
  nil)

;; more overrides to come

(setq project-explicit-prj (make-elisp-project))

See below for project-explicit-prj.

In my mind, there is only one elisp project; all files currently on
load-path.

There will be a similar cl-defstruct for jdee-project.

>> I don't know which would be better as the default; the only general
>> solution is to require a project file.
>
> How does that translate to API requirements?

- Provide a mechanism for the user to explicitly set a project file.

    For example (now in my local project-patches.el):

    (defvar project-explicit-prj nil
      "Project explicitly set by user.")

    (defun project-explicit-prj (dir)
      "Return the project the user has set in `project-explicit-prj'."
      project-explicit-prj)

    (setq project-find-functions 'project-explicit-prj)

    A menu or completion list of previously selected projects would be
    very useful here (I'm working on that).


- Provide a way for functions to require a particular type of project file

    This is mostly there; all functions that only work with projects of
    some type should be cl-defmethods for that type.

--
-- Stephe

Re: Unified project interface

[Stephen Leake]

Dmitry Gutov <dgutov@yandex.ru> writes:

>> I'll take that as agreeing to add the functions project-add-search-path
>> and project-add-ignores to project.el
>
> Not really. I don't think we should mess with a project backend's
> internals. 

Well, yes; these should not be user level functions. But it would be
nice if any backend could use them to build the project object.

As you keep saying, "elisp does not have a project backend", so the
user is forced to kludges for elisp projects.

>> Given all of the above, I'll make an attempt to write Ada and JDEE
>> project backends, using recursive paths. We'll see how that goes.
>
> Hopefully, you'll make that attempt anyway.

Yes. I'll write what I need, and if there are functions/data structures
used in both JDEE and Ada project backends, I'll propose them as
additions to the project API.

It turns out I'm also writing a more complete elisp backend as well.

-- 
-- Stephe

Re: Unified project interface

[Dmitry Gutov]

On 08/01/2015 10:04 PM, Stephen Leake wrote:

> Yes. I'll write what I need, and if there are functions/data structures
> used in both JDEE and Ada project backends, I'll propose them as
> additions to the project API.

Great. That should be the way to go.

Please be advised: I've added a second argument to project-ignores, like 
discussed before.

> It turns out I'm also writing a more complete elisp backend as well.

Maybe we could add it to Emacs as well.

Re: Per-language project-search-path, was: Re: Unified project interface

[Dmitry Gutov]

On 08/01/2015 09:57 PM, Stephen Leake wrote:

> I often mix languages in one directory anyway; the file extension
> indicates the language, so there's no need for it in the directory name.

All right. That's fine if you don't need it.

>> And what about jdee? Would you treat lisp/.dir-locals.el as the
>> project file for Elisp, and ask the user to select it?
>
> I think you are asking
>
>      "what project would you use while editing elisp files that implement
>      jdee"
>
> ...
>
> Yes, serious users will want to explicitly select the active project.

Okay. I could add that different languages can have interdependencies 
(like templates can references to "normal" source code), so it's not 
always easy to separate them. But I guess you can use their search paths 
together, and rely on filtering by extension.

> If I'm debugging jdee, I want the search path to be elisp load-path,
> whether I'm in a notes.text file, a jdee .el file, or a test .java file.
> So I don't want Emacs to guess what the active project is, nor determine
> the search path based on the mode.

Fair enough.

>      (setq project-find-functions 'project-explicit-prj)

When the package is ready, this should be added via add-hook, when a 
certain minor mode is enabled.

>
>      A menu or completion list of previously selected projects would be
>      very useful here (I'm working on that).

That minor mode could provide a relevant command.

> - Provide a way for functions to require a particular type of project file
>
>      This is mostly there; all functions that only work with projects of
>      some type should be cl-defmethods for that type.

That's fine, but it's better is more functions are universal.

Re: Per-language project-search-path, was: Re: Unified project interface

[Eric Ludlam]

Hi,

Sorry for a long silence- it was a busy month. ;)

On 08/01/2015 10:12 AM, Dmitry Gutov wrote:
> On 08/01/2015 01:43 PM, Stephen Leake wrote:
>
>> My experience is always with a project file; the user explicitly selects
>> which project file is active. That can be multi-language or
>> single-language.
>>
>> So I'm not familiar with the project system trying to guess a project on
>> the fly, as the current code does.
>
> I'm not discussing a particular project backend here, just the API. 
> And you can use the user-selected backend everywhere, if there's an 
> appropriate element at the head of project-find-functions.

WRT EDE as an existing model to consider ...

EDE will detect projects based on a key file, or you can ask it to load 
a project file with the ede command.  It defines a base implementation 
of what a project is for handling menus and keybindings, and depends on 
different project backends to implement features used by the menus and 
keybindings.

EDE does not have a list of 'find' type functions.  Instead, each 
project type declares a small structure with functions for identifying 
the dominating file, which classes to implement, and for some complex 
cases, how to identify where projects are based on the contents of some 
config file.  For example, Arduino has a config file that says where 
your sketches are.  I used the structures because the allow project 
detection without autoloading the supporting back-end until that project 
type is actually used.

> If I recall, you've mentioned a concept of project file listing 
> different search-paths for different languages. Without 
> project-search-path taking the language into account, you'll have to 
> append all paths together.

EDE was not originally intended to be a way to search for files, but 
this feature was needed for semantic to find headers and referenced 
files using short names.  It does this with 'ede-expand-filename'. For 
projects that include an exhaustive list of files as part of the 
configuration (such as Automake) searches just go through the known 
files using major-mode to filter if needed.

Other projects that do not have all the files ready, such as EDE's 
android backend instead implement a method for `ede-expand-filename' 
that indicates where to start searching based on simple patterns, such 
as the java code in src, and resource xml files in res.  Using methods 
is important so backend authors can do checking before blindly creating 
path names.

For particularly large projects with a lot of source files, you can bind 
in an external tools such as Global or locate to help you find files 
when the basics don't work out.

> I've only listed the implementation difficulties. If there are any 
> particular problems with the concept of dispatching on the language, 
> I'd like to hear about those.

EDE had to manage the concept of derived modes via the mode-local tool 
in order to handle cases where C++ was like C with a few more details.  
If you intend to dispatch behavior based on mode, I strongly recommend 
mode local as a way to make it declarative and traceable.
>
>> I don't see any implementations of ede-source-paths, and only one use of
>> it, so it's hard to say what the purpose of the mode parameter is.
>
> That's true, unfortunately. The only implementations I could find are 
> in malabar-mode (another third-party project attempting to provide a 
> Java editing environment).
>
> Eric, could you tell if we're missing anything? Are there other places 
> where ede-source-paths is defined or used?

ede-source-paths is implemented for several java flavors, such as 
ant/maven and android.  It is then used by semantic's javap database 
backend, and SRecode's template based code generator.

It works well for java because class names and path names line up 
nicely.  It hasn't been used for anything else, even though the comment 
mentions C++.

I suspect what your source-paths concept better matches EDE's targets 
concept.  EDE breaks a project into targets, matching a Makefile style 
target if that metadata is available.  Each target represents some set 
of source files, usually all of the same mode.

EDE also tracks include paths, which are locations where you might 
include code from some other location, matching the C/C++ concept. It 
works ok for some other languages, but primarily it is for that. Java 
has it's own classpath notion to better match what it does.

> Any reason why symref isn't using it?

Symref style searching, which I assume is like your xref concept, 
doesn't line up with the various paths in EDE, as EDE doesn't really 
need those types of paths for anything.  Even the semantic tool, which 
used to deal with lists of project roots dropped the concept in favor of 
EDE's project detection as it was superior for all use cases.

Every EDE project marks the top directory of a source tree, with 
.ede-ignore enabling breaks for nested projects.  Using that top path, 
you can use find/grep, GNU Global, and various other tools to do your 
searching for you.  The nice thing when you start using Global or CScope 
is that your config for the tool in question does the mode type 
filtering for you.

My experience with EDE and other CEDET pieces is that I need to do 
symref type searches quite rarely.  Usually the various tag based 
searches do a fine job getting me the data I need, the exception being 
finding uses of some function.  In that case, symref, along with Global 
or cscope is nice because it does a good job organizing the results so 
it is easy to find what you want.  I had intended symref to be where 
various refactoring tools would be based, as it allows you to select 
various hits, and perform operations on only selected hits.  I stopped 
developing code as my job before I got to those features.

Eric

Re: Per-language project-search-path, was: Re: Unified project interface

[Nix]

On 2 Aug 2015, Eric Ludlam stated:
> Every EDE project marks the top directory of a source tree, with
> .ede-ignore enabling breaks for nested projects. Using that top path,
> you can use find/grep, GNU Global, and various other tools to do your
> searching for you. The nice thing when you start using Global or
> CScope is that your config for the tool in question does the mode type
> filtering for you.

Allow me to interject here that this feature had me applauding when I
discovered it. It's an example of helpful automagic: if you have a large
project, you're going to want a tagging system anyway: having Semantic
automatically exploit it as soon as it appears is just gravy.

Speaking personally, I don't need an extra project system: as far as I'm
concerned Emacs already has one. I wrote one, as everyone does, and then
threw it away because I never use it. EDE and GNU GLOBAL are sufficient,
once you figure out how to use them: they know what files are part of
the project, how to build it, how to find things in it with a degree of
context-sensitivity that xref can only dream of (though its
multi-backend feature means that presumably it can learn to *use*
Semantic to provide context-sensitive search), and can handle things
like projects with handwritten Makefiles that don't match its automagic
project-root- searching features easily:

(ede-cpp-root-project "DTrace-CTF" :file "/usr/src/libdtrace-ctf/Makerules"
    :include-path '("/include" "/build")
    :system-include-path '("/usr/include")
    :spp-table '(("_LITTLE_ENDIAN" . "") ("_GNU_SOURCE" . "")))

That, combined with a GTAGS file, gives me most of EDE's features, and
all the features proposed for the project system. For projects like
Emacs or the Linux kernel or anything using Automake I don't even need
that much. EDE is an underappreciated jewel. Most of this project
discussion and the invasive code piling into modes all across Emacs
feels to me like an attempt to reimplement part of EDE, but without
things like the grammar-sensitive searching that makes EDE so capable.

(What I'd really like is something like the el-search pcase-based
searcher Michael Heerdegen recently proposed, only for everything
supported by Semantic, but with most language grammars not being very
amenable to ML-style pattern-matching I doubt this is possible...)

-- 
NULL && (void)

Re: Per-language project-search-path, was: Re: Unified project interface

[Michael Heerdegen]

Nix <nix@esperi.org.uk> writes:

> (What I'd really like is something like the el-search pcase-based
> searcher Michael Heerdegen recently proposed, only for everything
> supported by Semantic, but with most language grammars not being very
> amenable to ML-style pattern-matching I doubt this is possible...)

I agree that would be very useful.  But I don't know much about
Semantic, and the info pages seem to be outdated.

Is there some function in semantic that parses the expression at point
and returns a parse tree represented as (Lisp) list?  The outdated info
page mentions ast.el which seems to do that, but I can't find it
anymore.

I guess one could try the following, more or less: go through the buffer
and parse it on the fly to ast expressions.  The search command prompts
for a "pattern" using the syntax of the buffer's language plus
placeholders.  The input is parsed and converted to a an ast as well.
Then the matching could be done in Elisp like el-search does.


Michael.

Re: Per-language project-search-path, was: Re: Unified project interface

[Dmitry Gutov]

On 08/02/2015 11:57 AM, Nix wrote:

> Allow me to interject here that this feature had me applauding when I
> discovered it. It's an example of helpful automagic: if you have a large
> project, you're going to want a tagging system anyway: having Semantic
> automatically exploit it as soon as it appears is just gravy.

If EDE works for you, that's great. You should continue using it.

> Speaking personally, I don't need an extra project system: as far as I'm
> concerned Emacs already has one.

The point is other tools can use some information about your current 
project without being aware that you're using EDE. Or something else.

> the project, how to build it, how to find things in it with a degree of
> context-sensitivity that xref can only dream of (though its
> multi-backend feature means that presumably it can learn to *use*
> Semantic to provide context-sensitive search)

Indeed, it can. *And* it's also useful for languages which Semantic 
doesn't support.

> Most of this project
> discussion and the invasive code piling into modes all across Emacs
> feels to me like an attempt to reimplement part of EDE, but without
> things like the grammar-sensitive searching that makes EDE so capable.

What code piling?

Re: Per-language project-search-path, was: Re: Unified project interface

[Eric Ludlam]

On 08/02/2015 01:14 PM, Michael Heerdegen wrote:
> Nix <nix@esperi.org.uk> writes:
>
>> (What I'd really like is something like the el-search pcase-based
>> searcher Michael Heerdegen recently proposed, only for everything
>> supported by Semantic, but with most language grammars not being very
>> amenable to ML-style pattern-matching I doubt this is possible...)
> I agree that would be very useful.  But I don't know much about
> Semantic, and the info pages seem to be outdated.

Most of the doc is about using some of the user facing tools.  The 
sections about the lex stage should be good.

> Is there some function in semantic that parses the expression at point
> and returns a parse tree represented as (Lisp) list?  The outdated info
> page mentions ast.el which seems to do that, but I can't find it
> anymore.
>
> I guess one could try the following, more or less: go through the buffer
> and parse it on the fly to ast expressions.  The search command prompts
> for a "pattern" using the syntax of the buffer's language plus
> placeholders.  The input is parsed and converted to a an ast as well.
> Then the matching could be done in Elisp like el-search does.

You can use the lexer from semantic to get the list of parsed tokens.  
The current parsers output tags instead of an AST.  Making an AST would 
be handy for langauge sensitive tooling past tagging parsers, but would 
be a bit slow.  Large parts of the buffer aren't parsed at all right now 
as the parser skips over large { bracketed } blocks of code which keeps 
thing faster.

It may be that the specifics you are looking for are part of the local 
context parser, which includes functions such as -get-local-variables, 
-get-local-arguments, and -current-symbol, which gets the whole symbols 
in a dotted expression.   This is in semantic/ctxt.el.

Eric

Re: Per-language project-search-path, was: Re: Unified project interface

[Michael Heerdegen]

Eric Ludlam <ericludlam@gmail.com> writes:

> It may be that the specifics you are looking for are part of the local
> context parser, which includes functions such as -get-local-variables,
> -get-local-arguments, and -current-symbol, which gets the whole
> symbols in a dotted expression.   This is in semantic/ctxt.el.

Thanks for the explanations.

I gave these functions a quick try, evaluating them via M-:
(the-function), with not much luck.  In Elisp, all calls always returned
nil.  In a C file, I even got an error:

Debugger entered--Lisp error: (wrong-type-argument syntax-table-p nil)
  set-syntax-table(nil)
  (progn (set-syntax-table semantic-lex-syntax-table) (save-excursion (cond ((looking-at "\\w\\|\\s_") (forward-sexp 1)) ((looking-at fieldsep1) nil) ((save-excursion (and (condition-case nil (progn ... ... t) (error nil)) (looking-at fieldsep1))) (setq symlist (list "")) (forward-sexp -1) (while (looking-at "\\s(") (forward-sexp -1)) (forward-sexp 1))) (setq end (point)) (condition-case nil (while (save-excursion (forward-char -1) (looking-at "\\w\\|\\s_")) (forward-sexp -1) (setq symlist (cons (buffer-substring-no-properties (point) end) symlist)) (let ((cp (point))) (forward-sexp -1) (forward-sexp 1) (if (<= cp (point)) (error nil))) (if (looking-at fieldsep) (progn (forward-sexp -1) (while (and ... ...) (forward-sexp -1)) (forward-sexp 1) (setq end (point))) (error nil))) (error nil)))
  symlist)
  (unwind-protect (progn (set-syntax-table semantic-lex-syntax-table) (save-excursion (cond ((looking-at "\\w\\|\\s_") (forward-sexp 1)) ((looking-at fieldsep1) nil) ((save-excursion (and (condition-case nil ... ...) (looking-at fieldsep1))) (setq symlist (list "")) (forward-sexp -1) (while (looking-at "\\s(") (forward-sexp -1)) (forward-sexp 1))) (setq end (point)) (condition-case nil (while (save-excursion (forward-char -1) (looking-at "\\w\\|\\s_")) (forward-sexp -1) (setq symlist (cons (buffer-substring-no-properties ... end) symlist)) (let ((cp ...)) (forward-sexp -1) (forward-sexp 1) (if (<= cp ...) (error nil))) (if (looking-at fieldsep) (progn (forward-sexp -1) (while ... ...) (forward-sexp 1) (setq end ...)) (error nil))) (error nil))) symlist) (save-current-buffer (set-buffer #:b
 uffer) (set-syntax-table #:table)))
  (let ((#2=#:table (syntax-table)) (#1=#:buffer (current-buffer))) (unwind-protect (progn (set-syntax-table semantic-lex-syntax-table) (save-excursion (cond ((looking-at "\\w\\|\\s_") (forward-sexp 1)) ((looking-at fieldsep1) nil) ((save-excursion (and ... ...)) (setq symlist (list "")) (forward-sexp -1) (while (looking-at "\\s(") (forward-sexp -1)) (forward-sexp 1))) (setq end (point)) (condition-case nil (while (save-excursion (forward-char -1) (looking-at "\\w\\|\\s_")) (forward-sexp -1) (setq symlist (cons ... symlist)) (let (...) (forward-sexp -1) (forward-sexp 1) (if ... ...)) (if (looking-at fieldsep) (progn ... ... ... ...) (error nil))) (error nil))) symlist) (save-current-buffer (set-buffer #1#) (set-syntax-table #2#))))
  (with-syntax-table semantic-lex-syntax-table (save-excursion (cond ((looking-at "\\w\\|\\s_") (forward-sexp 1)) ((looking-at fieldsep1) nil) ((save-excursion (and (condition-case nil (progn ... ... t) (error nil)) (looking-at fieldsep1))) (setq symlist (list "")) (forward-sexp -1) (while (looking-at "\\s(") (forward-sexp -1)) (forward-sexp 1))) (setq end (point)) (condition-case nil (while (save-excursion (forward-char -1) (looking-at "\\w\\|\\s_")) (forward-sexp -1) (setq symlist (cons (buffer-substring-no-properties (point) end) symlist)) (let ((cp (point))) (forward-sexp -1) (forward-sexp 1) (if (<= cp (point)) (error nil))) (if (looking-at fieldsep) (progn (forward-sexp -1) (while (and ... ...) (forward-sexp -1)) (forward-sexp 1) (setq end (point))) (error nil))) (error nil))) symlis
 t)
  (let* ((fieldsep1 (mapconcat (lambda (a) (regexp-quote a)) semantic-type-relation-separator-character "\\|")) (fieldsep (concat "[ 	\n.]*\\(" fieldsep1 "\\)[ 	\n.]*\\(\\w\\|\\s_\\)")) (case-fold-search semantic-case-fold) (symlist nil) end) (with-syntax-table semantic-lex-syntax-table (save-excursion (cond ((looking-at "\\w\\|\\s_") (forward-sexp 1)) ((looking-at fieldsep1) nil) ((save-excursion (and (condition-case nil ... ...) (looking-at fieldsep1))) (setq symlist (list "")) (forward-sexp -1) (while (looking-at "\\s(") (forward-sexp -1)) (forward-sexp 1))) (setq end (point)) (condition-case nil (while (save-excursion (forward-char -1) (looking-at "\\w\\|\\s_")) (forward-sexp -1) (setq symlist (cons (buffer-substring-no-properties ... end) symlist)) (let ((cp ...)) (forward-sexp -1) (f
 orward-sexp 1) (if (<= cp ...) (error nil))) (if (looking-at fieldsep) (progn (forward-sexp -1) (while ... ...) (forward-sexp 1) (setq end ...)) (error nil))) (error nil))) symlist))
  (save-excursion (if point (goto-char point)) (let* ((fieldsep1 (mapconcat (lambda (a) (regexp-quote a)) semantic-type-relation-separator-character "\\|")) (fieldsep (concat "[ 	\n.]*\\(" fieldsep1 "\\)[ 	\n.]*\\(\\w\\|\\s_\\)")) (case-fold-search semantic-case-fold) (symlist nil) end) (with-syntax-table semantic-lex-syntax-table (save-excursion (cond ((looking-at "\\w\\|\\s_") (forward-sexp 1)) ((looking-at fieldsep1) nil) ((save-excursion (and ... ...)) (setq symlist (list "")) (forward-sexp -1) (while (looking-at "\\s(") (forward-sexp -1)) (forward-sexp 1))) (setq end (point)) (condition-case nil (while (save-excursion (forward-char -1) (looking-at "\\w\\|\\s_")) (forward-sexp -1) (setq symlist (cons ... symlist)) (let (...) (forward-sexp -1) (forward-sexp 1) (if ... ...)) (if (looking
 -at fieldsep) (progn ... ... ... ...) (error nil))) (error nil))) symlist)))
  semantic-ctxt-current-symbol-default()
  ...

because semantic-lex-syntax-table is bound to nil which
`set-syntax-table' doesn't like.


Michael.

Re: Per-language project-search-path, was: Re: Unified project interface

[Dmitry Gutov]

Hi Erc,

On 08/02/2015 05:29 AM, Eric Ludlam wrote:

> EDE does not have a list of 'find' type functions.  Instead, each
> project type declares a small structure with functions for identifying
> the dominating file, which classes to implement, and for some complex

"which classes to implement" - what does that mean? Is this something 
interesting, in the context of this discussion?

> EDE was not originally intended to be a way to search for files, but
> this feature was needed for semantic to find headers and referenced
> files using short names.

I think it's quite natural for a project to be able to list the 
directories where related files could be found. For instance, so that 
someone could implement a find-file-in-project command. Or search across 
those files.

> It does this with 'ede-expand-filename'. For
> projects that include an exhaustive list of files as part of the
> configuration (such as Automake) searches just go through the known
> files using major-mode to filter if needed.

This seems useful enough, although maybe I'd untie the expand-filename 
logic from a given project implementation and make it dependent on the 
major mode. For instance, no matter if it's a common Java project or an 
Android project, whether it's based on Ant, or Maven, or Gradle, the 
logic for finding the file based on the class name is the same: you just 
need the list of directories to look in.

Which project-search-path might correspond to.

> For particularly large projects with a lot of source files, you can bind
> in an external tools such as Global or locate to help you find files
> when the basics don't work out.

Global seems fine to search inside the project (for certain languages, 
at least), but what about the system libraries? Headers, etc. If they're 
used in the current project, being able to search in them would be 
useful, too.

> EDE had to manage the concept of derived modes via the mode-local tool
> in order to handle cases where C++ was like C with a few more details.
> If you intend to dispatch behavior based on mode, I strongly recommend
> mode local as a way to make it declarative and traceable.

That's interesting, but it doesn't sidestep the issue of different users 
possibly using alternative major modes for the same files.

So far as I see it, in most cases the project implementation could 
easily write something ad-hoc like that that would be good enough. IOW, 
I'm more concerned about the basics right now.

> ede-source-paths is implemented for several java flavors, such as
> ant/maven and android.  It is then used by semantic's javap database
> backend, and SRecode's template based code generator.

I see, thank you. I've only been looking in CEDET inside Emacs.

> It works well for java because class names and path names line up
> nicely.  It hasn't been used for anything else, even though the comment
> mentions C++.

In Ruby, class and module names also often map to file paths. Only by 
convention, though, so there's some variability there. The same happens, 
to certain degrees, in other languages too.

> I suspect what your source-paths concept better matches EDE's targets
> concept.  EDE breaks a project into targets, matching a Makefile style
> target if that metadata is available.  Each target represents some set
> of source files, usually all of the same mode.

Maybe? I'm not sure how the concept of targets maps to dynamic 
languages, for example.

> EDE also tracks include paths, which are locations where you might
> include code from some other location, matching the C/C++ concept. It
> works ok for some other languages, but primarily it is for that. Java
> has it's own classpath notion to better match what it does.

Indeed. project-search-path, as it's currently imagined, is a merge of 
ede-source-paths and ede-system-include-path.

So xref-find-regexp and xref-find-references will, by default, search 
the includes path too. Something that symref doesn't do.

>> Any reason why symref isn't using it?
>
> Symref style searching, which I assume is like your xref concept,
> doesn't line up with the various paths in EDE, as EDE doesn't really
> need those types of paths for anything.  Even the semantic tool, which
> used to deal with lists of project roots dropped the concept in favor of
> EDE's project detection as it was superior for all use cases.

Yes, we also try to reuse semantic-symref-find-references-by-name in the 
default implementation of xref-find-references.

The two main drawbacks now are: it doesn't handle the project's ignored 
directories (which should be relevant for symref/grep.el), and 
semantic-symref-filepattern-alist needs to have correct entries for 
every language.

> Every EDE project marks the top directory of a source tree, with
> .ede-ignore enabling breaks for nested projects.  Using that top path,
> you can use find/grep, GNU Global, and various other tools to do your
> searching for you.  The nice thing when you start using Global or CScope
> is that your config for the tool in question does the mode type
> filtering for you.

I suppose. As long as the tool supports all kinds of operators (search 
for references, regexp, listing files), and supports your target 
language. Though it probably won't help with related code outside of 
your project's tree anyway.

Maybe xref-find-references should also always search all project-roots 
anyway (while filtering by file names and ignoring non-interesting 
directories), and project-search-path should focus on "source roots" 
(inside and outside of the project tree) exclusively. So even if the 
project has a build file written in the same language as the rest of it 
(and it lies in the root directory), project-search-path still would 
only include, say, project/lib and project/app.

The main use case for those entries, then, would be locating the file 
for a given class name. Which could serve as a basis for a naive 
find-definition implementation.

> My experience with EDE and other CEDET pieces is that I need to do
> symref type searches quite rarely.  Usually the various tag based
> searches do a fine job getting me the data I need, the exception being
> finding uses of some function.

"Finding uses" is an operation I have to do quite frequently.

>I had intended symref to be where
> various refactoring tools would be based, as it allows you to select
> various hits, and perform operations on only selected hits.

Yup, support for renaming is something I'm interested in as well. It's 
generally accepted that Emacs's refactoring support is lacking/nonexistent.

Re: Per-language project-search-path, was: Re: Unified project interface

[Nix]

On 3 Aug 2015, Dmitry Gutov spake thusly:

> On 08/02/2015 11:57 AM, Nix wrote:
>> Most of this project
>> discussion and the invasive code piling into modes all across Emacs
>> feels to me like an attempt to reimplement part of EDE, but without
>> things like the grammar-sensitive searching that makes EDE so capable.
>
> What code piling?

There seem to be rather a lot of references to project-* landing in
unrelated files (language modes, etc). Or so it seems to me, anyway.
(But it's much less invasive than, say, the curved-quotes stuff, and
much more useful ;) )

-- 
NULL && (void)

Re: Per-language project-search-path, was: Re: Unified project interface

[Dmitry Gutov]

On 08/03/2015 01:24 PM, Nix wrote:

> There seem to be rather a lot of references to project-* landing in
> unrelated files (language modes, etc). Or so it seems to me, anyway.
   ^
   └ citation needed

Re: Per-language project-search-path, was: Re: Unified project interface

[Eric Ludlam]

On 08/02/2015 07:39 PM, Michael Heerdegen wrote:
> Eric Ludlam <ericludlam@gmail.com> writes:
>
>> It may be that the specifics you are looking for are part of the local
>> context parser, which includes functions such as -get-local-variables,
>> -get-local-arguments, and -current-symbol, which gets the whole
>> symbols in a dotted expression.   This is in semantic/ctxt.el.
> Thanks for the explanations.
>
> I gave these functions a quick try, evaluating them via M-:
> (the-function), with not much luck.  In Elisp, all calls always returned
> nil.  In a C file, I even got an error:
>
> Debugger entered--Lisp error: (wrong-type-argument syntax-table-p nil)
>    set-syntax-table(nil)
>    (progn (set-syntax-table semantic-lex-syntax-table) (save-excursion (cond ((looking-at "\\w\\|\\s_")
> because semantic-lex-syntax-table is bound to nil which
> `set-syntax-table' doesn't like.
>

Everything in the semantic system needs `semantic-mode' enabled. That 
sets up all the parsing context needed for the various tools, and keeps 
it up to date.  There is a lot of little features needed to handle 
parsing, so the mode enables the hooks to get it setup as you load 
files, and then the tagging parser parsers you buffers, which the 
context parser uses to identify the outer bounds of different contexts.

Once you do that, you could try 'semantic-analyze-current-context' which 
will give you some high level data about where point is.  The local 
-ctxt- functions can be used to do something unrelated smart completion.

Eric

Re: Per-language project-search-path, was: Re: Unified project interface

[Nix]

On 3 Aug 2015, Dmitry Gutov verbalised:

> On 08/03/2015 01:24 PM, Nix wrote:
>
>> There seem to be rather a lot of references to project-* landing in
>> unrelated files (language modes, etc). Or so it seems to me, anyway.
>   ^
>   └ citation needed

None available because it was predicated on a bunch of misunderstandings
of how tightly things like xref tie into it. :) sorry!

-- 
NULL && (void)